Some Recent Failed Side Projects

The Big Brother Coronavirus RPG

I recently read that the participants in Big Brother Germany were locked in the house and had the Coronavirus pandemic news broken to them by the staff. Strange situations make interesting premises for a casual, one-shot roleplaying games.

Here’s the concept. All of the players are contestants on a reality TV show set in isolation like Big Brother or Survivor. The game starts when they receive news (directly or through surprise exposure) that a calamity, like a global pandemic, has occurred, and they need to get out. Since this is reality TV, each character has a crush and a rival from among the other characters. The twist is that with this news, each player also has an assigned secret agenda. Some are surprisingly rational for reality TV contestants, and they believe that the calamity is real, so they need to escape and find the host. However, others believe it’s a hoax and part of the show, so they are still trying to “win” the show perhaps by permanently eliminating others. Commence the drama.

The system had 3 inspirations: Reality Shows, Lasers & Feelings, and Bang!. I liked the flavor of the shows, the simple roleplaying mechanics of Lasers & Feelings, and the hidden agenda of Bang to really reinforce reality show paranoia. I was working out the mechanics, but they became too prescriptive and felt more like a board game of Bang! or Mafia rather than a roleplaying game like D&D.

It seemed like a really good idea for the first hour, but it didn’t quite come together as I was hoping. In the end, I figured that it would take more balancing and play testing than the novelty of the concept was worth, so I let it go.

However, let me know if you were really tickled and want to playtest. The rules still exist across a few pages of my Moleskine but are likely somewhat wonky. A deep love of Reality TV shows would presumably help because I frankly don’t that much about it

Augmented Reality Battlemaps

When I went to WWDC 2019, I saw the presentation about RealityComposer, Apple’s technology for easily creating Augmented Reality (AR) apps. It looked awesome, and I immediately began to explore the AR apps. In my mind, AR needs to use both the physical world and virtual world. I tested several AR games in the App Store, but most didn’t take advantage of AR: they were cool games but would work as just virtual games.

Of course, I gravitated towards apps for D&D. Players and DMs like the automation of digital tools to track stats and characters, but they also like the physicality of props like maps, miniatures, and dice. Could AR merge that?

My first idea was to create AR maps. Artists and tools create beautiful digital battlemaps images, but they are hard to use in the real world. You either need a large projected image or TV on your surface or to print out massive images to work with miniatures.

With AR maps, you could let your phone draw the battlemaps digitally onto your surface with your miniatures on top. You wouldn’t see anything in real life (and would likely need a grid to align everything), but by peering through your phone, you could see it all integrated.

This project actually took me awhile to get started because I knew very little about the technology. Not only am I unfamiliar with AR, I also haven’t done any graphics or much iOS programming either. Eventually, I paid a token amount for a tutorial subscription that got me over the hump to build a prototype.

Then, I ran into a fundamental problem: occlusion.

It isn’t as simple as just lining up the map in 3 dimensional space on the table. The app also needs to cut out the areas for the miniatures on top of it so that those parts of the camera image appear on top of the map.

It turns out that’s really hard. One of the big features of ARKit 3 was People Occlusion to solve this. Unfortunately, miniatures aren’t literally people and could not be occluded. As such, when I managed to draw the map on top of the table, it covered up the miniature, too.

Can you see my coffee mug and miniature on there? I tried to tone down the opacity so that it could balance it, but it just didn’t work. Although the map is correctly positioned flat against the surface, it looks like the map is “floating” above the objects,

In theory, I could also apply movie magic by using a physical green screen and draw only where the map should be. It might work, but I didn’t obviously see anything in the ARKit library to do masking.

Julie suggested that I could also use AR to keep track of statuses or stats for miniatures. Rather than writing down the hit points for that monster, the app could show and track for each miniature on the screen. Unfortunately, recognizing virtual objects unfortunately requires scans of the objects, and miniatures are quite small and often appear similar.

The technology isn’t quite there yet, but it might be coming soon. Apple just announced yesterday that ARKit 3.5 uses the new iPad Pro’s LiDAR technology for better depth tracking and people occlusion. It still isn’t what I need, but I will keep an eye on future development.

StarCraft 2 Game Summaries

Through Spawning Tool, I have access to a high quality dataset of tens of thousands of StarCraft 2 replays. Since those replays were automatically generated by the StarCraft 2 game client, they’re quite accurate and relatively clean.

Of course, large datasets are intrinsically intimidating to people: there are thousands of hours of game footage in there. To make the replays more accessible on the site, I occasionally wrote up game summaries for “featured replays”, and I wondered whether it was possible to do this automatically.

This concept isn’t new: other companies have applied Natural Language Generation (NLG) techniques to convert structured data (e.g. financial spreadsheets, baseball box scores) into natural language (e.g. written reports, game summaries). Why not do the same thing for StarCraft?

My first attempt to do it like baseball turned out to be too laborious. You can generate baseball summaries from box scores and scorekeeping sheets that already break down the game into hits, swings, and discrete actions. StarCraft replays are more like GPS on each player and the ball, so I determined that a rules based approach would be too cumbersome.

Next, I thought I could use Deep Learning to learn the complexity for me. To be specific, I could use a recurrent neural network (RNN) to consume the game state continuously and generated the text as it went. Although I liked this architecture, I would need a lot of training data. Although I had the raw replay input, I also needed target output.

My clever idea was to take transcripts of casted StarCraft games and match that up with time stamps in games. Although there are plenty of casted professional StarCraft games, there isn’t a reliable way to get an audio transcript. I was able to find a way to use the auto-generated English captions off of YouTube videos, but the highest quality casts from the official Blizzard tournament (WCS) turned those off.

And even a few thousand transcripts might not be enough given the range of human language. When I thought about the architecture, I not only had to train the RNN to accurately encode game state, but I would also need another model to convert that game state into regular speech trained on the text. This project just got too big, so I gave up