Cry Beta and Let Loose the QA Team

Modding is a hobby that carries its own reward. It demands an excrutiating amount of work, but if you’re into it, it feels more like fun than duty and at the end of the day you have a product to show for it. Hopefully a product you can be proud of.

As the project draws to a close, the rewards become greater and more numerous. In the beginning, you get a kick out of seeing your dialogue implemented, your characters animated, your world lit and ready to explore. Few feelings compare to the pride of hearing an actor interpret lines that you have written, however. And finally, letting people play the mod and hearing what they have to say about it is an amazing experience.

Also painful. But thrills like that don’t come for free, you understand.

Beta tester response to The Nameless Mod has generally been positive. Some testers felt compelled to play the mod for 5 hours at a time, tearing through the missions. Some testers are still mucking about in the first mission, poking it relentlessly to see how many different solutions it has (spoiler: A ridiculous, unbelievable amount). A few testers have been bogged down by our still annoyingly unstable first hub level, but they seem determined to defy the crashes and give the game a proper testing.

As an aside, how embarrassing is it that our first hub level crashes all the time? The rest of the game is almost completely stable (with the odd path finding error still needing to be found and fixed), but what good is that if most players give up because all the crashes are in the first mission? We’ve done some drastic things to eliminate them (the map used to be approximately 30% larger than it is now), but clearly not drastic enough. Methinks it’s time to give the ambient lighting in that map a complete overhaul.

Some annoying mistakes made it into the beta build. Predictably, almost all are results of last-minute fixes made during our short crunch before the beta deadline. Fixes beget new bugs, and though most of these are small mistakes, easy to fix, the sheer size of our builds (Beta 1 was 583 MB) makes it impractical to put out a new build until we’ve accumulated a greater amount of fixes.

Our game is structured into 5 “missions” (a Deus Ex term – more accurate nouns would be “chapters” or “sections”). Our first chapter makes up ca. 40% of the game, and has been tested thoroughly. Second chapter is tiny but notoriously buggy – we’ll see how many bugs persist in the beta 1 build – and the third is another enormous chapter, a revisit to the locales of chapter 1. Chapters 4 and 5 are small but brimming with special-case scripting and complex events and systems. These last two chapters have barely been tested at all in alpha, and I fear we may be flooded with bug reports when our diligent beta testers reach it.

Our testing method is aimless and far from thorough. We have no coordination between the testers to ensure that every corner and every plot branch of the mod is tested. We’re running a very real risk that certain more obscure branches may never be tested at all – that some choices are simply too unlikely to be made until a larger population of players gets their hands on the game. If this had been a real game, some form of automated unit testing would have been indispensible, but if I suggested any such thing to Shane or Nick at this point, they would strangle me to death with my own mouse cord. As it stands, we must cross our fingers and pray that everything is tested properly, and then patch after release if we turn out to be mistaken.

As the testing carries on, my ego swells to formerly unheard of sizes. It seems we’ve done it right, but of course we’re standing on the shoulder of giants. What TNM has taught us, essentially, is how best to design Deus Ex.