Author: mmittner

I have been working on a research project at EA this semester. The goal of this project has been to see if we can predict what types of games people will like based on their in-game behavior. Yes, the goal was that general. As the designer on the project it was pretty much up to me to come up with a plan of action as to how we might accomplish this. I thought a lot about the way people play together, and looked into everything from the work of Chris Bell to this huge Gamasutra article on player types. I’m not going to go into player types too much as you can read the article, which reviews them extensively, if you are super interested. However, what I am going to touch on is our process, the models we choose, and how we have been using them thus far.

Many people may be familiar with Bartle types. These were originally adapted for analyzing players of MUDs, but have since been adapted for multiple uses. GamerDNA.com uses a pretty good Bartle Test to analyze what type of player you are based on your behavior in MMORPGS. But, basically Bartle breaks down players into four categories (sometimes eight, but for our most commonly and for our purposes four): Killers, Achievers, Socializers, and Explorers. But, Bartle doesn’t apply to everything and we needed something a little more complex.

If you know Richard Bartle, you may also be familiar with Nicole Lazzaro. She breaks players down by why they play. Again, she puts players into four categories by the type of fun they are pursuing: Serious, Hard (Fiero), People, and Easy.

These two models of types correlate with each other very well. For example, an achiever who prefers to pursue leveling, achievements, equipment, and customization; these players are are drawn to fiero. I mean by this statement that they gain pleasure from overcoming difficult situations, like puzzles, acting on difficult problem in the world (whether these are combat challenges or perhaps building challenges) and will gain a sense of satisfaction from the achievements of a mentee or partner in game. Below is a map of how these groups correlate.

So we have developed a structured way of observing players. First, we created a Dead Space 3 level which serves as a microcosm for the whole game. What do I mean by that? Well, we only have about an hour with each player to see if we can predict their player type, and see if our prediction about their player type will accurately allow us to predict whether they will like another game. In this case we are trying to predict if they will like Army of Two: The 40^th Day. So we get 40 minutes with Dead Space and 15 minutes with Army of Two. In order to observe as much as possible in 40 minutes we needed to have a lot of the normal game’s possible activities shoved into a shorter experience. For example, we needed to have more opportunities for customization, exploration, frustrating combat, and puzzles. This way, we could see what a player would do quickly. We are trying to automate this process, so in the future a system will have the whole game over which to collect data, but again, we only have 40 minutes.

At the end of the test, after we have made our prediction about what their type is, and whether they will like Army of Two, we give them a Bartle test to get a boolean on whether we are hitting close to the mark with our predictions. We also ask them some questions, amongst these is whether they like Army of Two or not.

First off, we have determined there are a few things we thought would be good predictors that aren’t. For example: time spent in EVA (floating around in outer space) is not a good determiner of whether someone is an Explorer since some people can’t orient themselves well in a zero-g environment. Also, the way people play is highly influenced by player types they are with. Explorers tend to have a negative experience when playing with Killers, as Killers will try to drive the experience quickly through a level. Socializers on the other hand will go with the flow and can interact well with either of those types (usually, we did have a gross exception).

On the whole we have found that our process gives us good results. We can predict a player’s type over 65% of the time so that it correlates with the first Bartel type returned on a Bartle test administered at the end of the testing session. Also, we had hypothesized that Achiever and Socializer player types would like Army of Two the most and this has proven to be true for the most part. We can predict whether a player will like Army of two with about 68% accuracy going by this hard and fast rule. Socializers almost always like Army of Two once they play it, because they are attracted to the co-op aspects, and especially like the camaraderie system. Achievers on the other hand like Army of Two only about half of the time. This may be because they sometimes do not have enough time to get into the customization systems available to them, and don’t play enough to experience any fiero. Trust me, having played the game, there are definitely opportunities to experience fiero. However, our predictions about who will not like the game are spot on. We have only had one explorer who liked the game at all, and they said, “It was okay, I might play it if my friend had it.” Our killers like it less than 20% of the time, this is probably because they can’t complete the game all by themselves or force the experience to move forward without their partner.

We have a lot more testing to do, and it remains to be seen whether we can get our automated system working to make these predictions. But, if nothing else results from this project, I am happy that we came up with a way to predict this much. I feel that the way we predict player types in Dead Space 3 can easily be transferred to a wide variety of games from platformers to open world RPGs.

More to come as we wrap up this project.

Update Post Project May 2013:

I just wanted to say that we were able to reassess our criteria (looking for Socializers, Achiever Socializers, and Killer Socializers specifically) for predicting who would like certain games, specifically Army of Two based on the results of our testing. We found that we can now predict with 75% accuracy whether or not a player will like AO2 based on how they played a 40 minute session of our DS3 testing level. Most of the numbers I have given in this blog are rounded off, that one is not. Amazingly when we crunched the numbers it actually came out to a flat 75% (where as before our accuracy rate was actually 68.4% ). I will link our postmortem documentation when I get back to San Francisco.

 

Prior to working on my current project at EA I did a project for Oglebay’s Shrader Environmental Education Center in Wheeling West Virginia. We called this project SEECQUEL, because it was the second project done by a CMU team for Oglebay (the first project having been called SEEC). SEECQUEL was a location based entertainment project which challenged us to teach kids about Nature in the environment while using technology. It might sound simple, but getting kids to put the focus on Nature while out bumming around with a piece of interesting tech is difficult. Turn that around and upside down too: asking kids to use a piece of tech that is boring and bland, to learn something, while they could be playing in the mud is equally difficult. We had to find that nice middle ground where kids were engaged by our experience without being “taken out” of nature by it. We did this by providing the children an augmented reality device which scaled back to a tool as they progressed through the experience. At the beginning kids could use an Android tablet to “see like a bug” but by the end the tablet was being used to get kids to answer questions or take a picture. It worked out really well, and while I could go in to all the specifics of the design decisions that resulted in our success on this front I’m not going to. I’d rather talk about a few of our serious problems (because let’s face it that is where we learned the most) and how we addressed them.

What follows are some important lessons learned that I think anyone designing for kids can learn from. Some may seem obvious, and maybe I should have realized these things right off.

Lesson 1: Children will naturally help each other (well, sometimes girls won’t).

We had this activity where we were trying to teach kids how to determine whether a tree in the forest was older than the ones around it. To do this a child had to hold up the tablet and point it at the tree, and then touch the screen in several places. If done alone, this meant holding a tablet steady one handed while doing the main part of the activity, a difficult task for anyone let alone a 8 to 12 year old. The experience was designed primarily for groups of 4 to 6 kids from schools visiting the Schrader Center, and we wanted to give the children instructions to have their peers help them hold up the tablet. However, we were already having trouble delivering instructions to the kids that they would consistently pay attention to (more on that later), so the less instructions we had to give the better. We had a number of ideas on how to solve this problem but were not able to implement any of them before our first mechanics playtest. In retrospect we saved ourselves some time by not prioritizing this UI change higher. It turned out that our testers saw the problem and immediately started helping each other with this activity. Partly because the one holding the tablet would say it was a little hard, and partly because all the children wanted to be involved and in this case it mean being close to the shiny object. With these positive results from our first playtest we patted ourselves on the back for being such geniuses and not over designing.

Of course it couldn’t last; toward the end of the project as everything was nice and refined we tested with a number of groups of kids towards the older end of our age range made up of exclusively girls. I bet you can guess what happened. That’s right, they didn’t even consider helping each other and this was almost completely consistent across all the groups. I talked to the teachers of some of these girls and they told me that this behavior stemmed from a fear of doing something wrong. I won’t pretend to understand child psychology at that level; however we had already instituted a “try again” button. We did this because we had learned that, even with the help of peers, kids of course didn’t perform the task correctly 100% of the time. Sometimes kids even trolled each other, imagine that. So, it was a simple matter to add a suggestion, after they had tried a couple times, that they might collaborate. This got some of the girls to collaborate, and some groups just ignored it. So, there is another lesson there, you can’t get everyone.

Lesson 2: Kids do pay attention, to themselves.

We had this virtual avatar named A.B. Brooks who guided the whole experience. He was based on this early naturalist who helped to found Oglebay. He tells the kids at the beginning that he will be taking them on a quest and teaching them to be “Junior Naturalists” through the completion of a number of tasks. We originally had planned to not only have him give lessons at each activity location but also provide all instructions. We quickly learned that if we wanted the kids to actually know what they were supposed to do at the activity locations that we could not do this. At first we thought it was Brooks’ voice. We thought maybe that his deep tones (which we had to pitch up a bit just to play better out of the tablet speakers) were not grabbing their attention well. Or, maybe they couldn’t hear him. But, neither of these things were the case. The kids seemed to like him, said they liked him in the testing surveys, and would listen to him every time he started talking, so they would get part of the general lesson they were supposed to learn at each location. However, I say “part of the general lesson” because our lessons were too long by far at that point. The problem was that the younger the child was the shorter the amount of time they would spend focused on Brooks each time he started speaking. It’s like Saturday morning cartoons. As an adult if you ever watch children’s cartoons for our target age range you might think they are super spastic, but the truth is that makers of cartoons have really figured out how to peg a kid’s average attention span and fit as much action into scenes that fit that amount of time. Even the commercials are shorter. Some people might say that kids’ attention spans are conditioned by cartoons and other media to be like this. Maybe that is the case, but we were working with children who HAD to play outside and had strict rules about watching cartoons. Wheeling is supposed to be one of the best places in the US to raise active children. So, maybe not, all I know is the behavior was consistent. We found that if we kept whatever Brooks said to 30 seconds or less then all the children would pay attention. Anything longer was pushing it, but sometimes we had to, however we never exceeded 45 seconds. Do you have any idea how hard it is to teach about the Mohs Hardness Scale in less than 45 seconds?

Well, maybe we could teach children something significant about tree leaves or rock hardness in 30 seconds but we certainly couldn’t do that and give them instructions on how to use the tablet in different ways at each location. Since at each podium the kids visited the use of the tablet changed and they needed an introduction to how the tablet would be used there we had to figure out how to solve this problem. We struggled WAY too long with this when the solution was so elegant and simple when it finally occurred to me I tried to kick myself in my own butt. From the very beginning we had the kids read the instructions to each other. When they got to a new location they were told to pass the tablet to a certain child (identified by a totem animal we assigned them at the beginning of the game, that thankfully was something we got right from the get go), and this child would be asked to read out loud to the others. It was perfect, it helped build teams, kids would pay attention to each other as they saw it as a duty each had to take on, and if someone wasn’t a particularly good reader others would help them.

Lesson 3: Theming can get you into trouble. But, without the theming things would be less interesting.

I wanted to theme nature quest in two ways both of which caused some problems for us. First off I wanted Nature Quest to play much like a MMORPG Quest chain you might do with a group. I wanted it to have a lot of game like elements: an achievement system, an interactive map, a quest finder arrow in the HUD, and side quests. This made it way too scopey, partly it was just too many elements to put into the experience in the time allotted for development. But, on top of this, it was not elegant enough. Location based entertainment isn’t like a video game. People aren’t totally immersing themselves in a virtual world; we didn’t have a tutorial period at the beginning we can run them through to teach them the ins and outs of how to “be” in our experience. Giving them a complex HUD would have just given them something to focus a lot of energy and attention on, and worked counter to our goal of getting them involved with nature. No one cares about virtual achievements they never see again. Side quests were just a way to split up the team, some children would be interested and others wouldn’t  And what happens when a side quest wants them to take pictures of flora and fauna, and they simply couldn’t because these things happened to not be around at the time? The point of the experience was to make kids feel like they were learning and succeeding every step of the way, failing a side quest would just discourage them. So we cut it down to the essentials, it was still a quest, they were still on a journey, and there were still some perceived stakes. They even got a singular achievement at the end (a Junior Naturalist Badge) that they could take away. But, every task was repeatable so even if there was a knowledge test they could succeed provided they were willing to learn and keep trying. It worked out much better, and the most satisfying moment for me was when one of the children of an employee at the center told me, “ I’ve been down these trails millions of times! But, I never noticed these rocks, you know, because I wasn’t thinking about it like I am now.” It meant we had done our job and he had become more involved with the environment as a result.

The other theming I wanted was to have a consistent aesthetic to the whole experience. I wanted everything to look like it came from within the life time of A.B. Brooks. I wanted the podium signs to look like they came out of a naturalist’s notebook with drawings of plants and animals rather than photos. I wanted the UI to use icons that were period specific and all the text to be on parchment that looked like it might have been torn from a naturalist’s notebook. Overall this worked out really well, and a lot of credit has to go to Daniel Aum for just being an amazing person to work with and making that vision come together with his illustrations. However, we had one issue which turned out to be a bit of a problem. When adults think of a camera we still remember single-lens reflex cameras as being the only thing around. So an old plate camera which has very much the same look to the lens in front doesn’t throw us for a loop. However, when the modern child thinks of a camera they think of an iphone. We had three buttons you could use almost all the time in our interface: a magnifying glass for zooming in on things, a map for bringing up the map of the trail, and a plate camera to activate the on-board camera. Sadly, only about a third of kids were able to identify it as a camera when prompted to use the camera during certain activities. Even when told outright by Brooks that it was a camera they didn’t seem to process this information. We ended up having to put each button against a notebook background and labels under each before the children would process that it was indeed a camera. We could have removed the images but that looked too bland. Or, we could have used a more modern looking image, but that would have broken with the theming. So we settled for this and I think it was the least elegant part of the whole interface. But, it worked in the end so I guess I shouldn’t knock it.

I could talk about other things we messed up and fixed along the way. Like using Unity at all, and hitting the uncanny valley too hard early on with our avatar. But, this is already might be the longest blog I have ever written so I will refrain. If you want to know more you can always ask questions in the comments. I would like to take a moment to thank one of the best teams I have ever worked with: Garret Kimball, Daniel Aum, Prateek Gudihal, and Emmanuel Eytan.