TNM Post-Mortem pt. 3: What Went Wrong?

Here’s the third and final part of my post-mortem of The Nameless Mod. I’ve chosen to cut it up into three posts because it’s very long. Two days ago, I posted the introduction. Yesterday, I gave you the five things that went right. Today, I’m wrapping it up with five things that went wrong. You can also download the whole thing minus illustrations as a PDF

1. We spent no time on pre-production

In stark contrast to Deus Ex’s luxurious six months of pre-production, we literally had none. Actual game assets were being produced from day one; before we’d even decided what the game was going to be about or what sort of story we would write, artists were churning out character textures and level designers were building maps. A month or two of recklessly disorganised brainstorming segued into a confused, hurried documentation process where we struggled to keep up with our own ideas. Since the project was entirely anarchistic in the beginning, nobody had to wait for permission to start pumping out assets, and little to no coordination took place between different designers or artists.

In and of itself, “don’t eliminate pre-production” may seem like a pretty obvious piece of advice, but what we were actually trying to do, in hind-sight, was iterative design: We were experimenting with the engine and the tools at our disposal, testing the capabilities of our team, and throwing our every idea at the game with no sense of restraint in order to see what worked and what didn’t. The approach could’ve worked out a lot better than it did if we’d made a conscious effort to evaluate the decisions we made and deliberately pick out the elements we wanted. Our team size and structure was even quite suitable for an iterative design model; we had around ten highly dedicated people working on it back then, and there was no management or budget to worry about, so we were free to mess around and distil the useful ideas from the poor.

Unfortunately we made the major mistake of not eliminating anything at all. Everything people created and every idea anybody had went straight into the design documents. In that way, we managed to waste what was actually a pretty healthy pre-production process because we treated it as full production. A lot of the bad ideas and decisions that made it into the game back then, we managed to replace or fix later down the line, but some of it can still be found in the final product because it’s simply too time-consuming to fix. File that one under “how not to do Agile development”.

2. We didn’t understand all the code

They say those who don’t know history are doomed to repeat it, but sometimes you know going into a project that you’re going to face some major problems, and there’s very little you can do about it. The sometimes makeshift nature of parts of Deus Ex’s code was one such problem we faced. Now, far be it from me to criticise the Deus Ex programmers – they did an amazing amount of work and unlike us, they were working under deadlines. But by their own admission, the artificial intelligence in Deus Ex wasn’t as well designed as it should have been and suffered even further for being constructed precariously on top of the Unreal Engine code.

To us, the problem wasn’t just that we didn’t want to mess too much with the AI code, and that we never had access to Unreal’s source code to fix some of the more persistent bugs we encountered, but that we didn’t fully understand how the code was meant to be used in the design. Our AI problems are the most illustrative example of that, but unfortunately not the only one. Deus Ex’s AI, it turns out, is pretty stable when it just has to be friendly or when it just has to be hostile. When you start mixing friendly and hostile (and perhaps even neutral, if you hate yourself) behaviours in the same area, the illusion begins to fall apart. What we did was to mix friendly, hostile, and neutral characters in the same levels, and then switch their alliances around depending on the player’s previous choices. It blew up in our faces, and it never stopped blowing up.

I suppose there were two ways we could have solved this problem: One was to sit down and rewrite all of Deus Ex’s AI code as early as possible. This wasn’t an option due to our extremely limited programming resources, so the remaining solution was to look very carefully at how Deus Ex’s characters had their AI set up and how the different settings were used, and then stick to that; but once we finally realised that, implementing such a solution was tantamount to scrapping everything and starting over from scratch.

Some of our levels really pushed the capabilities of the creaky old Unreal Engine. We had to cut almost half of this one and replace all its background lights with ambient "zone lighting" to make it stop crashing; and to be honest, it's still too big.

3. We had to compromise on quality

By the time TNM was released, our standards were pretty high. We screened our actors thoroughly for audio quality, we rebuilt entire levels dangerously late in development if there were too many problems with them, and we’d long since stopped implementing any feature if we didn’t feel that we could take it all the way. But of course, beggars can’t be choosers, and we sometimes had to compromise because we couldn’t pay people. The two areas where I think this is most obvious in the final product are level design and voice-over. For the voice-over, it was a hard and bitter struggle to secure solid actors for the major characters, but I think we largely made it. In terms of minor characters, the picture looks a little different: Many of them were recorded by ourselves or our girlfriends or by random fans. Even our good actors occasionally had sub-standard recording quality, which becomes obvious far more often than it should.

In terms of level design, the problems were technical as well as aesthetic. Some of our level designers were quite good at producing pretty architecture, but didn’t seem to grasp the design principles that made Deus Ex’s missions so exciting. Others had brilliant ideas and a great understanding of design, but their levels looked boxy and uninteresting. Some levels that were otherwise nice and well designed had been constructed entirely off-grid, causing substantial technical issues later on, and it’s painfully apparent that we never had an art director or a concept artist, making our levels vary erratically in style and tone. Far too late did we understand the concept of dividing level design tasks amongst a designer and an environmental artist, and even then, we never had the resources to do it in the first place.

Though this problem was emphasised because we were forced to make do with the generosity of volunteers, I imagine that too short a supply of talented team members who meet your standards of quality is a problem even commercial projects face, and I suspect that if I had a fool-proof solution for it, I could use it as the basis for a whole new article. In the future, we aim to conquer the problem by counting on a small team of only the most skilled and dedicated rather than designing a project that demands the participation of 50 contributors of varying talent and enthusiasm. In short, we’ll embrace the indie ethos and stop pretending we’re an AAA studio.

4. We didn’t test enough

We tried though. We gradually brought over twenty testers in to play the game starting several years before we were finished, and set up an online bug tracking system for them to report back to us with. We set up a forum specifically for the testers, and we sent out guide documents to everybody detailing how to install the test build, what sort of bugs they could expect, how to report them, etc. We evaluated our bug database and set up priority fix lists towards the end of the project to make sure all the really problematic bugs were fixed before release. When TNM was released, it had no major bugs that we knew about.

And yet the first journalist we sent the release candidate out to, before the download link was publicised, found a bug in the kill counting code which would close off an entire map prematurely due to a recent fix between the two last release candidates. We then spent an entire week fixing bugs in some sort of infernal post-crunch, followed by another month of slower, steady fixes. By the second patch, we’ve fixed around 240 problems, and we’ve more or less brought TNM into the state we thought it was in when we released.

There are three things we could’ve done to improve our quality assurance. First and most importantly, we should have been more careful to test our own work during development. Far too often, features and assets went into the game without any testing. In a game as heavily dependent on emergent gameplay as Deus Ex, it’s critically important to apply very thorough testing to everything you implement: If we had tested each of our new additions in combination with as many different factors as we could think of, that would’ve significantly reduced the amount of combinations that slipped through the cracks. Secondly, we should have set up some auto-testing: One of our programmers coded up auto-testing scripts a couple of months before we were done, which was far too late – we only managed to apply it twice before release. Since our plot had so many branches and variations along the way, being able to fast-forward through certain common branches would’ve been a great help earlier in the project.

Finally, we should have managed our testers more strictly. Unfortunately I’m not sure most of our testers would’ve responded very well to tighter management, but I suspect five dedicated and well managed testers would’ve been far more useful than a score of testers left mostly to their own devices. We tried to get our testers to commit to particular kinds of playthroughs, hoping we’d get all the branches covered, but it never worked because most people who volunteered to test weren’t interested in committing to anything beyond playing the game and writing down the major bugs they found. In summary, our primary mistake was that we skipped the organised alpha testing and went straight to beta testing.

One of our design tenets was that every mission should have some sort of gimmick or setpiece to set them apart from each other. For example, our final mission takes place on a space station, leading to much new oxygen- or zero-gravity-related gameplay.

5. Our project is a PR nightmare

The final and perhaps the hardest problem we faced was making people understand that our game wasn’t shit. We’d spent a lot of time making sure that people wouldn’t be alienated by our game and that we had plenty of depth and internal consistency to keep everybody interested, but we’d completely underestimated what a huge turn-off the basic idea of the mod was to so many people. Every site where we published our trailer, and every forum thread where people began to discuss the mod, one sentiment would immediately surface like a knee-jerk reflex: What an idiotic concept. Why would anybody spend seven years working on this fan-boy circle-jerk of a game?

We were pretty crest-fallen: We’d gone to such lengths to make sure TNM was a game, not just a joke, and many people wouldn’t even give it a chance because they immediately assumed the worst. But to make matters worse, we came to realise that we’d frontloaded all the Internet references, the fan culture and the memes and the in-jokes right in the first mission of the game. Part of this was unavoidable: The first mission served by necessity to introduce the player to our setting, so all the opaque references and Internet semiotics were presented to you immediately. Once out of the first introductory hub area, the setting would quickly slip into the back seat to leave room for the plot itself, but too many people seemed to never reach it, having lost all interest long before then. Since The Nameless Mod is free to download, we have no demo, but in terms of convincing people to invest their time in playing through TNM, that introduction area is all we have, and it seems to be doing a rather poor job.

Perhaps our greatest mistake was to tell people that The Nameless Mod was inspired by a real community that existed on the Internet at one point in time. I suspect people in general would be a lot more susceptible to our quirky cyberspace setting if they thought we’d just invented it as a Snow Crash-esque sci-fi take on the Internet, because then they wouldn’t associate us with the reviled genre of “forum fan fiction” to begin with, and once playing the game, they wouldn’t be expecting in-jokes everywhere. Much of the feedback we’ve received has implied that people constantly see in-jokes and obscure references when by far most of the game’s fiction was either invented specifically to suit the plot or the setting or twisted so far out of its original shape that it no longer bears any resemblance to the events or the people it was inspired by.

A well-known games journalist graciously defending our concept wrote: “no wonder everyone makes games with space marines being gruff”. I’d like to think people are generally open to new concepts, but I’ll admit that our premise and our setting do us no favours. Not, however, because TNM is too weird – there have been many successful games far stranger than The Nameless Mod and our world is quite recognisable when it comes down to it. But at the same time it reminds people of a genre of fiction which is almost never executed well. If we’d managed to set ourselves further apart from that genre with all the PR material we sent out, and if we’d done more to change people’s expectations before firing up the game, I think we’d had a much easier sell.

In Closing

In some ways, The Nameless Mod is a triumph for Agile development and iterative design processes. Not only did we learn how to make the game we were trying to make, we learned almost everything we now know about game development and project management. We learned these things by trial and error, and our errors pushed us to seek out the information we needed and apply it to our own work.

It’s unfortunate that we wasted so much time messing up and making bad design decisions. Part of this was because we didn’t manage feature creep as well as we could have, and part of it was because we got carried away with perfectionism, constantly recreating old assets to match our new standards. If we’d known what we were doing right from the start, TNM could probably have been released in 2006.

It’s been an incredible project though, and it’s changed some of our lives substantially. I think we ultimately came out on top of most of our mistakes, and we’re very excited to move on to the next challenge. With all the feedback we’ve received on The Nameless Mod, hopefully we won’t repeat any of the mistakes we’ve made. And now we have this post-mortem to help us remember them.