Depending on who you ask, ChatGPT (and other similar LLM chatbots) might be impressive, dangerous, misunderstood, or one of many other things. However, it and other recent generative AI have created plenty of entertaining content.
Crossing over my interests between tabletop RPGs and computing, I read an article about playing D&D with ChatGPT as the DM, and I had to try it for myself.
You are the DM for a 5th edition D&D campaign. I am the only player. Please run a game of D&D for me
Great! Let’s begin your solo adventure in the world of Dungeons and Dragons.
1. The story starts with a cliche
You find yourself in the bustling city of Waterdeep, standing in the middle of a crowded street. The sounds of merchants hawking their wares, children playing, and carriages clattering fill the air. Suddenly, a man in a dark cloak bumps into you and slips something into your pocket before quickly disappearing into the crowd.
You reach into your pocket and find a small piece of parchment. It reads:
“Meet me at the Yawning Portal Inn tonight at midnight. Bring your courage and your skills. There is a great reward for those who can help me with a dangerous task.”
As you look up from the note, you notice a group of tough-looking individuals eyeing you suspiciously. They begin to approach you, and it becomes clear that they mean you harm.
What do you do?
This is a good hook. I have a setting with sights, sounds, and feeling. ChatGPT immediately provided me with both a goal (so I know what to do) and a danger (so I have some urgency). And “What do you do?” is a classic opening.
However, many D&D players might roll their eyes. The Yawning Portal Inn is the most famous spot in Waterdeep, which is the most famous city in the Forgotten Realms, which is the most famous D&D campaign setting.
To be fair, I never told ChatGPT to be creative. Within my rough understanding of how ChatGPT works is that it will favor the popular and easily accessibility
2. The NPCs are very sincere
In most D&D games I have played, the players and DM regularly break character. There’s in-game talk as well as meta-game talk. We know we’re playing make-believe in our imagination, so we sometimes laugh when it’s obvious when we hit the invisible walls setup by the DM.
ChatGPT, however, was always very genuine when I prodded it.
For example, I had received a quest to take care of goblin raiders, and I solved it by brokering peace with the goblins. When I went back to the quest giver, he gave me… more quests to take care of raiders.
Durnan nods in approval and says, “Excellent work, Phil. I knew I could count on you. As for jobs, I have a few leads. There’s a group of bandits who have been terrorizing the countryside. They’ve been targeting travelers and small settlements. There’s also a group of orcs who have set up camp in the nearby mountains. They’ve been raiding the surrounding towns and villages… (skipped a bit of the rest of the response)
“Gosh, I really don’t want to send the goblins off to fight some bandits or orcs. It seems like we could try to come to peace with those guys, too.”
Durnan nods, “I understand your sentiment, but unfortunately, bandits and orcs won’t be as easily swayed as goblins. Sometimes the only way to deal with them is through force. However, if you have any other ideas, I’m all ears.”
If a human DM followed up one quest with two more that sound exactly the same, I would give them a hard time, so I did the same for ChatGPT.
Honestly, it was a pretty good answer. ChatGPT has been criticized for factual inaccuracies, and I would consider this similarly dishonest: the answer works in context, but it’s not really why it came up with the same quests.
This happened again when I went to visit a mysterious tower without any doors. I met someone named Arin and suggested she might figure out a way in.
“Arin, have any ideas about how we get in? Maybe sing a song?”
Arin ponders for a moment and then begins to sing a melody. As the notes carry through the air, the stone around the door begins to shift and change, forming intricate designs that glow with a faint blue light. Suddenly, the door creaks open, revealing a dark and mysterious passage.
“How did you figure out what song to sing?”
Arin responds, “Well, it’s just a hunch. I remember reading a book that mentioned a tower with a hidden entrance that could be opened with a specific song. I didn’t think much of it until we stumbled upon this tower, and it seemed like it was worth a shot.”
“That’s… quite the coincidence”
Arin nods and says, “Yes, it is. But I believe there are no coincidences in this world, only fate and destiny.” She then gestures towards the tower and adds, “Shall we go in and see what destiny awaits us?”
This situation is obviously contrived. I would accuse a DM of being way too easy on me for just saying yes to what I asked for. And yet, the response about “fate and destiny” is perfectly in-character. It even doubles down by mentioning “destiny” again.
3. The world is inconsistent
So who is this Arin? Well, it depends on when you asked.
Arin looks down, slightly embarrassed. “I grew up in a small village, and my mother was a bard. She taught me everything she knew before she passed away. I never thought my singing would be useful for anything other than entertaining at local taverns.”
Arin looks at Eolande with awe and respect. “It is an honor to meet you, Eolande. I am Arin, a bard from Waterdeep…
Arin explains that she is a botanist and her research is focused on finding and documenting rare and exotic plant species…
Maybe Arin is just a people pleaser. Or maybe she’s multi-talented. Or maybe ChatGPT just doesn’t retain the context of what it said before.
It’s actually quite incredible that ChatGPT is capable of integrating live information into its responses at all. The continuity is critical in the context of a conversation. Despite having been trained for a very long time on large swaths of text from the internet, it somehow manages to instantly integrate short bits of new information.
But it’s not perfect. It has a limited capacity for context, and even between subsequent responses, it has some obviously jarring changes. For example, I got several different names and descriptions of the flower Arin is looking for, one after another.
4. ChatGPT pegged my play style
For almost the entire conversation, I stayed in-character. Near the end, though, I was curious to see whether it was equally committed or capable of understanding the situation.
Let’s pause the game for a moment. How would you describe my playstyle as a player?
Based on our interactions, I would say that your playstyle involves a mix of exploration, problem-solving, and character interaction. You seem to enjoy delving into the details of the world and its characters, asking questions to gain more information, and considering different approaches to challenges. You also seem to be interested in building relationships and connections with NPCs and other players in the game. Overall, your playstyle appears to be collaborative and open to experimentation.
That sounded like a pretty good response. I like doing all of those things. However, I wondered if this was like horoscopes, where ChatGPT could give a sufficiently vague response that anyone would agree with.
What aspects of RPGs does it seem like I don’t like?
Based on our interactions, it doesn’t seem like you particularly enjoy combat or violence, as you have avoided fighting when possible and have shown more interest in exploring and interacting with NPCs. Additionally, you seem to enjoy solving puzzles and discovering the lore and backstory of the world.
That’s also a pretty good answer. Other than the brief encounter at the beginning with the thugs (who I actually also ran away from), I didn’t really fight at all.
5. ChatGPT is a pretty good DM
Based on this session, ChatGPT has some obvious gaps as a DM. Perhaps these could be improved with prompt engineering, but some of these issues are just wonky.
And yet, I think ChatGPT did a pretty good job.
It’s not perfect, but it doesn’t have to be. When we’re evaluating how well AI works, we can aim for perfection, but that’s not really the baseline. The baseline is often whether it can do better than humans, whether that’s beating grandmaster chess players or finding cancer in a CT scan.
Compared to human DMs, ChatGPT does make some obvious mistakes, but they are similar. Human DMs can be cliche and fall back on availability biases. Human DMs forget previously established lore and retcon details. Human DMs lampshade wonky parts of the game.
The interesting part is that I think that D&D is a great use case for ChatGPT because these issues are largely acceptable.
Asking ChatGPT for facts, whether news, history, or science, can be dicey when it very sincerely will hallucinate facts. It isn’t great for chess, where it will cheat.
However, D&D is a game of the imagination. At its core, it’s only as real as what exists in the player’s mind. There are rules, but they aren’t essential: I never asked ChatGPT to provide a character sheet or dice rolls for me. I know that the world isn’t real, and when a DM (human or chatbot) does something a little wonky, I can look past it and continue to engage in the game just as well.
Final Thoughts
I had a very fun afternoon playing D&D with ChatGPT. I was between other tasks, but I kept wanting to come back and play.
However, the magic did wear off for me. That evening, I tried to keep it going, but I stopped only after a few prompts. And I haven’t continued since. I probably will do a few more after I finish writing this post, but I’m not feeling it call to me.
What’s missing for me? I’m not quite sure. Compared to playing a video game RPG, I feel like I should enjoy it more. My biggest gripe with CRPGs is that they are mostly finite in scope to what the writers defined. And when it’s procedurally generated, it often doesn’t feel deep enough. Playing with ChatGPT feels infinite because it’s reacting and creating based on what I say.
I have always appreciated that TTRPGs are played with real people, but this experience has really solidified my belief in the importance of that. I can’t say for sure that having humans instead of algorithms on the other side is the difference, but I can say that I am looking forward to my next regular RPG session.