GDC 09: Valve on Playtesting

I always try to catch Valve sessions at GDC, because almost everyone recognizes how far ahead of the curve they are when it comes to game development processes. Valve’s Mike Ambinder gave a talk this morning about how Valve playtests its games, which is very involved but boils down to a very simple process: prototype -> playtest -> collect feedback -> iterate -> repeat until the game is fun.

I won’t attempt to recount the whole session here, in which Ambinder ran through seven types of testing and data collection, ranging from basic observation to surveys and even emotional and biomechanical feedback (not many companies outside Valve can afford a camera setup to track players’ eye movements). A big part of the talk covered the downsides of each type of testing, and where they can often produce false results (people often can’t tell you why they do the things they do, which makes it more important to watch what they do than listen to what they say) and how to avoid them.

I ran into several other developers while leaving the session, who all admitted they were there because Valve seems to do this kind of stuff better than anyone else. To which I usually agree, but still wonder: what went wrong with TF2’s Scout update?