Saturday, April 02, 2011

The Value of (and difference between) PLAY TESTING and FOCUS TESTING

Got this post about FOCUS TESTS from blog reader Daniel in the comments section of another post. I thought it was worth addressing the topic in a post of its own. So here goes:

Here's what Daniel wrote:

David, I know you've had experience with focus tests, I've read your piece on when you were doing a live blog during a focus test for one of your games. I don't understand why this happens in the industry.
Why have games become so watered down? Is it because COD4 sold millions of copies that other publishers want the piece of the pie and think their games have to be designed in similar fashion of watering them down?
I just don't understand the need to make games dumber and more accessible. The challenges in-game are what make the games entertaining. I guess if I wanted to just pass time I'd play Solitaire but that is pushing it since it's more challenging than COD4. :)

My thoughts on this:

- Firstly, Play Tests and Focus Tests are different things. A Play Test is when you put the game in front of players who have never played before so the team can learn about nuts and bolts production/gameplay issues. A Play Test teaches you things like: where the game is confusing, too hard, or too easy. It lets you know if a particular puzzle's solution is too obtuse or if the campaign's difficulty ramps at a good pace. Here's just a few of the questions we've used Play Tests to help us answer about the new TWISTED METAL: 
- did placing that shotgun pick up near the movie theater doors motivate players to drive near the doors and thus increase the chances of players noticing the doors and thus motivating players to bust down those doors and thus discover there is a whole room behind those doors that many players were never finding? 
- which driving control scheme should be our default (the classic and much loved- but very, very dated- controls of TM1,2,and TMB OR a more modern control system ((gas on the shoulders)) that- while modern and expected and much more pick up and play friendly- is not as shooter/car combat friendly as the classic control scheme)?
- Has the VOICE OVER NARRATION and on-screen/HUD text we've put in the game since our LAST Play Test significantly aided brand new players in understanding the rules for our NUKE game mode?
Also: Play Tests bring up brand new (and often times crucial) problems with the game that you and the team never saw coming (either because it's impossible to foresee every problem and/or because after you've been playing and living with the game for 18 months straight, the team gets used to annoyances and folds dealing with those annoyances INTO the play experience…but a new player who is trying out your game for the first time- and who is not PAID to play and live with the game for over a year- will not be so adaptable ). Here's a few of the problems with the new Twisted Metal that we never would have considered fixing/addressing had it not been for some invaluable Play Tests:
- Many players are confused about how to select the vehicle's ALTERNATIVE SPECIAL WEAPON. We assumed the big 'S' with the TRIANGLE ICON above it would make it clear that hitting the triangle when you had SPECIAL selected would cause the SPECIAL to change to ALT SPECIAL. But no such luck. 
- A great deal of the meatier gameplay (i.e. the remote bomb has 2 stages but you have to let the bomb 'cook' in order for the bomb to reach the triple damage of stage 2; if you fire the shotgun very close into the FRONT window of an enemy vehicle, you get a mega damaging POINT BLANK attack) is not being well communicated to players so they have no idea there is as much depth and substance to the gameplay as there really is.
So- as I imagine you can tell- I'm a HUGE fan of Play Tests. I would not want to make a commercial game without them. 
So that's a Play Test.
But a Focus Test is usually a marketing department tool used to test higher level concepts. Things such as: is the core idea of the game marketable, culturally relevant, and commercial? Is the main character unique enough to be memorable but not so unique that he/she/it is not relatable? How does the game you just played compare to similar games in the genre (better/worse/the same) and if you were only going to buy one of them in the next year, what would it take for you to choose OUR game? 
A Focus Test can be run at anytime in the game's life cycle and very different- and very valuable data- can be mined based on when you run the test. For example: when Twisted Metal was just in design phase several years back, marketing did a focus test to figure out how relevant the brand still was as well as how relevant the genre of car combat still was. That particular test told us that car combat was dead to gamers UNLESS it was the TWISTED METAL brand. When it was 'TWISTED METAL car combat' the tests showed us that the genre was still very exciting to action fans. That test also showed us that many gamers really, really, really wanted to see the TWISTED METAL brand revived/back in the gaming space. So- needless to say- that was a dang good Focus Test for all of us at Eat Sleep Play! 
That test also told us that the majority of gamers loved the MP of Twisted Metal BUT still had a very high demand for the Twisted characters and stories and the general TWISTED METAL world. We used that same focus test to see if gamers were more warm to a $39.99 multiplayer-only Twisted Metal Blu Ray game (ala WARHAWK and SOCOM:CONFRONTATION) OR if they wanted a fully featured $59.99 Twisted Metal with Multiplayer and single player campaign modes. 
So that Focus Test went down in 2008 maybe? Start of 2009? But so we did another marketing driven Focus Test just a few months ago (I think Jan 2011) where we let players play a post E3 build of the game and Sony Marketing used that particular Focus Test data to help determine things such as: which features are players most excited about, what levels did they enjoy the most, and which modes and weapons did they like and choose most often. The point of this new data was to aid marketing in knowing where to focus the marketing message when it comes to tv, print, and internet ads as well as general PR opportunities. 

Again, great, great data and a great value to the game.

A lot of people knee jerk at the words PLAY TEST and FOCUS TEST and automatically assume they are evil and they are killing art/emotion/individuality (in games or movies or television or music). 
But at least part of the knee jerk is the gamer not understanding the difference between Play Test and Focus Test. I would imagine if gamers understood that PLAY TESTING was- in essence- mining data to make the game live up to the team's vision, there would be very few folks who would oppose it (except for auteur based art games that are really intended to be a singular vision and playability takes a backseat to the game's message). 
But I think I get where the other part of the 'play/focus tests are evil' knee jerk comes from. It's probably motivated by the times when the data from those tests has been used to undermine the intent and goal of a game (or movie or tv show) in order to make it more palpable to the masses. Does this happen? I'm sure it does…hell, I know it does! But that's not always a bad thing. Hell, in most cases it's not a bad thing. But just like there's a popular myth that ALL corporations are bad and evil, there's one that says MARKETING IS BAD and the people who work in marketing are stupid, unartistic sheep. This is not near true from my experience. Hell, the marketing manager for the new Twisted Metal is MORE of a gamer than I AM! 
The deal is, the main reason the vast majority of people consume entertainment is to be distracted from their everyday lives. They want a break from the norm, or something to do with their friends on a Friday night…or just something to kill time with. Be it a mass market shooter or a stupid sitcom or the latest loud, exploding summer blockbuster. That's just the way it is. Can those things also have depth and meaning and meat on the bones? Oh hell yes and when they do, they make even MORE money (God of War tried to be that; a sitcom that is funny and dumb but still has heart does that; a big blockbuster like THE DARK KNIGHT RETURNS does that). But when you are spending the kind of money it takes to lure in the mainstream (be it production costs or marketing costs), there is value in casting the net wide. And Focus AND Play tests help content creators and content marketers get a sense of which way the wind is blowing before they spend much money as well as WHILE they are spending a shit ton of money. 
The less cash you spend, the less people you need to worry about appealing to. And I love pure art pieces, pure commercial pieces, and I especially love those pieces that blend the two. And in all mediums, there is a place and a need for both.I think in games especially, our industry does a great job of making games all across that pure art/pure business spectrum. 
Do I think it's super cool when tons of money is spent on pure art/vision pieces? Oh totally! It's super cool when that happens! But it's rare and it SHOULD be rare. And getting upset that- as games have become super expensive to make- game companies have decided to use Play and Focus Tests to help cast the net wider in order to increase the chance of having a hit, shows a lack of understanding (or a lack of acknowledgement) that games is a business as well as an entertainment/art form. 
Also- when it comes to games like CALL OF DUTY (Daniel's specific example in his comment): COD campaigns have difficulty settings so you can play campaign in super hard mode if you like (at least I think they do…I only play on medium). And while I know his point is the MP aspect, the reality is COD has a shite ton of settings in unranked games so there is nothing it to stop you from setting up a super hard core game if you like. And hell, if you want MORE hardcore than what the unranked COD host settings offer, there are games out there like OPERATION FLASHPOINT that fit the hardcore bill. That game is not near as pretty or polished on the production value end (pretty and polish tends to = lotsa cash) BUT gameplay is what immerses you anyway once the visuals wear off and so those games are being made for folks looking for a more hardcore experience.



Kilrahi said...

I have a question on your perspective on Betas sent out for the public to test.

I've always assumed that it provided invaluable data to the game designers. Especially with games that are online, as Twisted Metal is, because, if nothing else, you often find some gamebreaking bugs. They seem to be the industry norm, and it seems like the games that didn't do it suffered (TMHO being a good example).

However, recently you've backed away from saying this new Twisted Metal will have a beta. As a gamer, on one level I'm totally cool with it. I like surprises, and there is something to be said about experiencing things on day one.

Yet, on another level, it seems like a mistake. Especially since I feel there are some longtime fans who could probably spot some potential problems and their feedback would be useful.

What's your perspective on betas exactly? Do you feel that, unlike play testing and focus testing, there really isn't much of value that comes from them?

David Jaffe said...


Betas are super important an TM will have one for sure. I was saying I didn't want to do a public beta let alone a glorified demo that CALLS itself a public beta but is really just a demo.

To me, I'd rather us keep the beta small-comparatively speaking- because we don't need a mega sized crowd in order for this to be a real, honest to goodness, FUNCTIONAL BETA with lots of folks commenting and giving advice on tuning and bugs. Also, frankly, I don't want to give too much of the game away and/or burn people out before the game launches so when it does launch, they are over it and don't want to play. So it's a balance.

But yes, we ARE doing a BETA and I think they are very valuable. But again, a REAL ONE. So much of what passes as BETAS today or so not BETAS.


Frosty said...

Glad to hear about the updated driving controls :p as much as I enjoyed the old memories of TM replaying through some of the older ones and being as used to newer driving games the old controls just felt so alien.

Small time betas are definitely the way to go, if it's just a "beta" too many companies these days seem to confuse the words "beta" with "demo". While some other companies still run traditional betas but only for bugs and not feedback. Both are important to listen to just check how bad a AAA game like Final Fantasy XIV is going because they didn't listen to said feedback during alpha and beta stages.

Tony said...

David, I have a request that I need you guys to hear

First, I would like to point out that am a hardcore TM fan. And when I say hardcore I mean I've been crazy about TM since I was 9 years old (am 23 now)

When I got TM black I was shaking from excitement returning home from the store to play it for the first time.. I remember playing the game for hours thinking how strange and unnatural the controls and game mechanics of the cars felt. It reminded me of TM1&2.

but I thought, maybe its because the game is new and am not used to this one yet. By the time I do, am gonna start to enjoy the game. It took me 3 weeks playing the game hours everyday when I realized, it wasn’t my fault.. it was the game. I called my cousin, whom he and I have played the game since we were kids (especially TM3) told me the same thing. He bought the game the day it came out like I did. And I believe he told me he was so disappointed. He said TB black was in his words “unplayable”..

as I watched the gameplay demo in E3 and some of the gameplay videos on YouTube, I read a comment from someone who was making fun of the game. He was basically saying how ridiculous it is that in 2011 someone could come up with a game where a car could turn 360 degrees in half a second. I felt mad, I wanted to defend the game so bad. But he was right. The game controls and game mechanics of TM black was what made the game such a disappointment to me and my cousin (and am sure A LOT of other people).

Till this very day, I still think TM3 was the best of the series. It had a much more natural feel to it. And I still enjoy playing it every now and then with my cousin.

Now, I feel like you probably think there’s an advantage to making the new TM similar to TM black (and TM1&2) in that respect, otherwise you wouldn’t keep doing this and you wouldn’t keep making the car mechanics look and feel unnatural to both people who are playing and people who are watching it played by others. But am telling you man , there’s NO advantage.

I write this without much hope that I could change your mind about this, or that I could in anyway sway you to my side. but I beg you, al least consider what am saying.
Or maybe the new TM could be something in between TMB & TM3..just think about it. am craving for this one to be the work of art that we -diehard fans- have been waiting for what feels like a very long time..

much love

David Jaffe said...

Tony- your post is appreciated and I can assure you, we hear you. And you may be surprised to know that EVERY TIME we make a new TM, Scott and I are like, 'let's make the cars drive more realistically!'...and while we could up the realism quotient with- literally- the adjusting of about 2-5 tweakers- we never go down that road. And we have not for this game either.

Here's why: In TM- like in any other shooter and/or fighter- you are always either on offense, defense, or in 'prep' mode (collecting weapons, going for health, racing to an area of the map you want to be in for this reason or that).

You are in offense AT LEAST 33% of the time and realistically speaking- for most players- that number probably lives at around 70%. Because of that- and because of the speeds of the vehicles- there is a lot of 'jousting', much like the gameplay in a fighter plane game. So there is a lot of rushing at an enemy and unloading as you go past then needing to flip a 180 super fast and do it again...this repeats for 2-5 times until someone who was on offense decides, 'screw this- I'm gonna die! And then switches to defensive play'.

So that 180 being super fast is REALLY IMPORTANT because when we have the cars turning more realistically, it's super frustrating. The reason you didn't mind it so much in TM1/2 is that the frame rate of the game lived at around 10-20 frames/second most times versus the solid 30fps (and hell, I think TMB maybe have even run at 60) of the later games.

Acceleration is another super unrealistically tweaked attribute. That's because once you do your 180 flip, you want- as a player- to get back up to speed asap and realistic acceleration makes that process way too slow.

We've tried slowing the game down and while YES, the cars feel more realistic AND the game- on some levels- is actually more readable (which I LOVE)- the game just feels too slow and sluggish. If our vehicles didn't look like cars that needed to go fast and if players didn't demand big levels PERHAPS- like MARIO KART BATTLE MODE- we could slow things down and still be ok with the core play but that just isn't TWISTED METAL.

I can tell you that SOME of the vehicles WILL drive slower and turn slower and accelerate slower. We're hoping for a big differentiation between the vehicles- bigger than TMB- and so if you want a car that drives more like the TM2 vehicles, there may be 1-2 that fits that description (but we've not locked tuning yet so it's really hard to say).

Either way- we hope you give the game a try when it comes out and hope we can sell you on Twisted's brand on mayhem and depth this time around. It really is- to us- the best one we've ever made and we hope you at least give us a shot via a rental so you can decide if this new TM is right for you (and your cousin!) :)...

Again, thanks for the detailed comment and for stopping by the blog. Much appreciated!


Tony said...

yeah well, it was worth a shot :)

you know, I remember my cousin telling me that it made him laugh the first time his car was upside down in TMB when he flipped it back on its wheels again by pressing right/left!

I think the reason we weren't impressed is cuz even though the game was better in a lot of ways and it had great graphics compared to the previous games, is cuz it still felt like a major downgrade!..which made us feel sad :(

its interesting you say its THAT easy to make everything more realistic. I know little about game developing so I dont know how feasible this is, but can the game be the way you want it to be (by default) but for the rest of us haters we can still play the game more "realistically" by checking a box or something in game options? :D

also, you guys need to put a lot more weapons in this one. Am serious!..I felt like TMB had very few weapons compared to the previous ones (although I've never counted weapons in any of them) I think thats because the Reticle, SAT, and Zoomy felt like the same weapon (because it was the same pickup)

I love you and everything you've done over the course of your career. And of course I'll be getting this one the day it comes out


maartyrr said...

I really don't think the hardcore hardcore fans would be burned out from beta testing..hell we still play tm2pc and tmbo to this day more than twice a week. I do agree with Kil, about using our feedback for your info, since we would be the ones who play the game for many years. Things such as finding overpowering weapons/cars/level exploits, glitches (breaking though level walls and being in areas we shouldn't be) etc I would say come easier to us more so than others.

I can see the whole picture as to why some players don't get some of the concept and depth. There are certain levels of thinking when it comes to all different types of gaming. With TM2, the more skilled players would notice that bombs do more damage to opponents when they're in the air. Or in TMB a smarter player would use roadkill special to bluff an attack and force the other cars to draw shields. There are hundreds of well thought out chess like strategies from both games - some of it accidental discovery and others were intentionally tried.

I do believe that a lot of the problem areas would be found and resloved through a closed beta. Oh and I'm all for keeping it pure arcadey driving- that's what made TM stand out from games like V8, and what made TM fun all these years

Julian Brogden said...

I totally agree with you about Betas and demos.

I tend to avoid demo's because I like the entire experience being fresh, and demo's not only can leave a bad/inaccurate impression of the full game but also somewhat spoils the experience. It's like Christmas. I don't want to know what i'm getting before Christmas day lol. I think small scale Beta is a good idea to expose the kinks but keep the experience fresh for masses.

And about play testing, I don't really know who would complain about them but I can totally appreciate the necessity of them. I'm always curious about games that launch with horrible mechanics thinking, "how is it not obvious this game plays horribly?" lol..

Needless to say great post.

Unknown said...

coach outlet
coach outlet
ray ban sunglasses outlet
louis vuitton outlet stores
ralph lauren polo
cheap jordan shoes
michael kors outlet
louis vuitton handbags
adidas ultra boost
christian louboutin outlet
rolex watches
coach outlet
kate spade handbags
michael kors canada
adidas originals
cheap jordans
louboutin pas cher
louis vuitton outlet
oakley sunglasses
louis vuitton outlet
kate spade handbags
mont blanc pens
cheap oakley sunglasses
abercrombie outlet
gucci handbags
vans shoes
michael kors outlet
nike nfl jerseys
michael kors outlet
michael kors outlet
toms shoes
ray ban sunglasses
hollister outlet
nike air max 90
jordan shoes
cheap jordan shoes
michael kors handbags
cheap oakley sunglasses
michael kors outlet

Anonymous said...

i0q16g2i18 m5h93x4h29 r2l47u1s30 r9f05b7s20 q1m78y8g76 y8g64j7a21