The Value of (and difference between) PLAY TESTING and FOCUS TESTING

Got this post about FOCUS TESTS from blog reader Daniel in the comments section of another post. I thought it was worth addressing the topic in a post of its own. So here goes:

Here's what Daniel wrote:

David, I know you've had experience with focus tests, I've read your piece on when you were doing a live blog during a focus test for one of your games. I don't understand why this happens in the industry.
Why have games become so watered down? Is it because COD4 sold millions of copies that other publishers want the piece of the pie and think their games have to be designed in similar fashion of watering them down?
I just don't understand the need to make games dumber and more accessible. The challenges in-game are what make the games entertaining. I guess if I wanted to just pass time I'd play Solitaire but that is pushing it since it's more challenging than COD4. :)

My thoughts on this:

- Firstly, Play Tests and Focus Tests are different things. A Play Test is when you put the game in front of players who have never played before so the team can learn about nuts and bolts production/gameplay issues. A Play Test teaches you things like: where the game is confusing, too hard, or too easy. It lets you know if a particular puzzle's solution is too obtuse or if the campaign's difficulty ramps at a good pace. Here's just a few of the questions we've used Play Tests to help us answer about the new TWISTED METAL: 

- did placing that shotgun pick up near the movie theater doors motivate players to drive near the doors and thus increase the chances of players noticing the doors and thus motivating players to bust down those doors and thus discover there is a whole room behind those doors that many players were never finding? 

- which driving control scheme should be our default (the classic and much loved- but very, very dated- controls of TM1,2,and TMB OR a more modern control system ((gas on the shoulders)) that- while modern and expected and much more pick up and play friendly- is not as shooter/car combat friendly as the classic control scheme)?

- Has the VOICE OVER NARRATION and on-screen/HUD text we've put in the game since our LAST Play Test significantly aided brand new players in understanding the rules for our NUKE game mode?

Also: Play Tests bring up brand new (and often times crucial) problems with the game that you and the team never saw coming (either because it's impossible to foresee every problem and/or because after you've been playing and living with the game for 18 months straight, the team gets used to annoyances and folds dealing with those annoyances INTO the play experience…but a new player who is trying out your game for the first time- and who is not PAID to play and live with the game for over a year- will not be so adaptable ). Here's a few of the problems with the new Twisted Metal that we never would have considered fixing/addressing had it not been for some invaluable Play Tests:

- Many players are confused about how to select the vehicle's ALTERNATIVE SPECIAL WEAPON. We assumed the big 'S' with the TRIANGLE ICON above it would make it clear that hitting the triangle when you had SPECIAL selected would cause the SPECIAL to change to ALT SPECIAL. But no such luck. 

- A great deal of the meatier gameplay (i.e. the remote bomb has 2 stages but you have to let the bomb 'cook' in order for the bomb to reach the triple damage of stage 2; if you fire the shotgun very close into the FRONT window of an enemy vehicle, you get a mega damaging POINT BLANK attack) is not being well communicated to players so they have no idea there is as much depth and substance to the gameplay as there really is.

So- as I imagine you can tell- I'm a HUGE fan of Play Tests. I would not want to make a commercial game without them. 

So that's a Play Test.

But a Focus Test is usually a marketing department tool used to test higher level concepts. Things such as: is the core idea of the game marketable, culturally relevant, and commercial? Is the main character unique enough to be memorable but not so unique that he/she/it is not relatable? How does the game you just played compare to similar games in the genre (better/worse/the same) and if you were only going to buy one of them in the next year, what would it take for you to choose OUR game? 

A Focus Test can be run at anytime in the game's life cycle and very different- and very valuable data- can be mined based on when you run the test. For example: when Twisted Metal was just in design phase several years back, marketing did a focus test to figure out how relevant the brand still was as well as how relevant the genre of car combat still was. That particular test told us that car combat was dead to gamers UNLESS it was the TWISTED METAL brand. When it was 'TWISTED METAL car combat' the tests showed us that the genre was still very exciting to action fans. That test also showed us that many gamers really, really, really wanted to see the TWISTED METAL brand revived/back in the gaming space. So- needless to say- that was a dang good Focus Test for all of us at Eat Sleep Play! 

That test also told us that the majority of gamers loved the MP of Twisted Metal BUT still had a very high demand for the Twisted characters and stories and the general TWISTED METAL world. We used that same focus test to see if gamers were more warm to a $39.99 multiplayer-only Twisted Metal Blu Ray game (ala WARHAWK and SOCOM:CONFRONTATION) OR if they wanted a fully featured $59.99 Twisted Metal with Multiplayer and single player campaign modes. 

So that Focus Test went down in 2008 maybe? Start of 2009? But so we did another marketing driven Focus Test just a few months ago (I think Jan 2011) where we let players play a post E3 build of the game and Sony Marketing used that particular Focus Test data to help determine things such as: which features are players most excited about, what levels did they enjoy the most, and which modes and weapons did they like and choose most often. The point of this new data was to aid marketing in knowing where to focus the marketing message when it comes to tv, print, and internet ads as well as general PR opportunities. 

Again, great, great data and a great value to the game.

A lot of people knee jerk at the words PLAY TEST and FOCUS TEST and automatically assume they are evil and they are killing art/emotion/individuality (in games or movies or television or music). 

But at least part of the knee jerk is the gamer not understanding the difference between Play Test and Focus Test. I would imagine if gamers understood that PLAY TESTING was- in essence- mining data to make the game live up to the team's vision, there would be very few folks who would oppose it (except for auteur based art games that are really intended to be a singular vision and playability takes a backseat to the game's message). 

But I think I get where the other part of the 'play/focus tests are evil' knee jerk comes from. It's probably motivated by the times when the data from those tests has been used to undermine the intent and goal of a game (or movie or tv show) in order to make it more palpable to the masses. Does this happen? I'm sure it does…hell, I know it does! But that's not always a bad thing. Hell, in most cases it's not a bad thing. But just like there's a popular myth that ALL corporations are bad and evil, there's one that says MARKETING IS BAD and the people who work in marketing are stupid, unartistic sheep. This is not near true from my experience. Hell, the marketing manager for the new Twisted Metal is MORE of a gamer than I AM! 

The deal is, the main reason the vast majority of people consume entertainment is to be distracted from their everyday lives. They want a break from the norm, or something to do with their friends on a Friday night…or just something to kill time with. Be it a mass market shooter or a stupid sitcom or the latest loud, exploding summer blockbuster. That's just the way it is. Can those things also have depth and meaning and meat on the bones? Oh hell yes and when they do, they make even MORE money (God of War tried to be that; a sitcom that is funny and dumb but still has heart does that; a big blockbuster like THE DARK KNIGHT RETURNS does that). But when you are spending the kind of money it takes to lure in the mainstream (be it production costs or marketing costs), there is value in casting the net wide. And Focus AND Play tests help content creators and content marketers get a sense of which way the wind is blowing before they spend much money as well as WHILE they are spending a shit ton of money. 

The less cash you spend, the less people you need to worry about appealing to. And I love pure art pieces, pure commercial pieces, and I especially love those pieces that blend the two. And in all mediums, there is a place and a need for both.I think in games especially, our industry does a great job of making games all across that pure art/pure business spectrum. 

Do I think it's super cool when tons of money is spent on pure art/vision pieces? Oh totally! It's super cool when that happens! But it's rare and it SHOULD be rare. And getting upset that- as games have become super expensive to make- game companies have decided to use Play and Focus Tests to help cast the net wider in order to increase the chance of having a hit, shows a lack of understanding (or a lack of acknowledgement) that games is a business as well as an entertainment/art form. 

Also- when it comes to games like CALL OF DUTY (Daniel's specific example in his comment): COD campaigns have difficulty settings so you can play campaign in super hard mode if you like (at least I think they do…I only play on medium). And while I know his point is the MP aspect, the reality is COD has a shite ton of settings in unranked games so there is nothing it to stop you from setting up a super hard core game if you like. And hell, if you want MORE hardcore than what the unranked COD host settings offer, there are games out there like OPERATION FLASHPOINT that fit the hardcore bill. That game is not near as pretty or polished on the production value end (pretty and polish tends to = lotsa cash) BUT gameplay is what immerses you anyway once the visuals wear off and so those games are being made for folks looking for a more hardcore experience.

David