Video icon 64
Learn to Code. Start your free trial today.
Advertisement

How to Research Ideas and Test Assumptions, Before You Develop Anything

by
Student iconAre you a student? Get a yearly Tuts+ subscription for $45 →

Before you spend years of your life making a game because you think it will sell, make sure it's something worth making! You're driven to explore game design because you love it, right? Your time (and the time of your players) is precious. Here are three experiments you can do to make sure your ideas are worth that time.


You've Got Ideas

You've got game ideas. You probably have a whole shoebox full of them you've been adding to since you were a kid! You come up with them in your sleep, on the bus, in the shower. Heck, coming up with ideas for great games is probably what got you Googling game development in the first place.

But finding the idea worth developing next - now that's where it gets a bit tricky. If you knew where to start you wouldn't be reading this article; you'd be working on your game! But since you are reading this, I assume that you're stuck on what to do next.

What I Do

How great ideas get made and how creative people succeed (against all odds) is what I've spent the last four years researching and experimenting to figure out. In that time I've helped teams of artists and programmers make their first games at the Academy of Interactive Entertainment and I've coached startup games companies around Australia through finding a niche, making a better experience for the player and finding (and managing) awesome people to help make their games.

These are the techniques we've used to get to our first paying customers in five months, win games compettions and choose the games worth building a business around. If you're really curious, and want to know more, Guy Blomberg from yugstar.com got most of my life story in a podcast and the work I've done growing the Sydney game development scene was featured on Gamasutra here.

But this article isn't about me or any other developers, it's about you.

What You'll Do

You know deep down that it's going to take a lot of time to make something awesome. It took Halfbrick six weeks to make Fruit Ninja with a team of seasoned developers, and that was after years working on contract games for overseas publishers. By now you've probably read that Rovio made something like fifty games before getting to Angry Birds - and even then they had the advantage of being one of the very first game developers on a brand new platform!

Everything you've ever learned how to do took time, right? You know it's going to take a lot of trial and error before you really get the hang of making a great game that people enjoy and will invest their time into - because time is precious! It's a finite resource and once you spend it it's gone forever.

Since the last thing you want is to regret the time you spent making stuff that you didn't learn from or didn't enjoy making, focusing your time on increasing those learning opportunities is absolutely crucial.

So I'm going to ask you to carefully choose which projects you work on. Which ideas will give you the best bang for your temporal buck?

These need to be ideas that:

  • You'll enjoy working on (worth all the time and effort you're going to put into it)
  • Fill a need in the market (you're not wasting anyone else's time)
  • You'll learn heaps from making (so the next game you make is even better than the last one!)

Over the course of two articles I'll give you techniques you can use to narrow down that huge list of potential games in that shoebox of yours, and spend your time getting closer to making those games we'll remember you for!


Because Beginnings Are Important!

If this were an article on building a house we'd talk about picking the right spot, carefully selecting your base materials, checking and double checking all our measurements and then focusing on building strong foundations before even thinking about the fancy details of what it's going to look like.

And as pleasant as it is to imagine life as a house owner, to be able to say "yes, I designed and built my own house!", it's important to set that fantasy aside and just focus on getting those foundations set up right - otherwise you'll never get to that point.

Now, although this article is not about building a house (a good thing, because I don't know the first thing about construction) it is about starting out right with good, solid foundations, before building anything at all.


Okay, So What Are We Testing For?

There's a whole lot of variables that go into what makes a game successful: from the game itself to whatever's happening in the world when you release it. It gets even more messy later on when we start asking questions like what platform to publish on, how much to sell it for, when to publish, and so on.

I say messy because all of that is subject to change. New platforms get developed all the time (like that OUYA thing and Steam on TV). Another game, released just a month before yours, could completely change the standard business model for your genre. A new subscription service could totally blow away free-to-play and if those new Google glasses get fashionable then Augmented Reality games could trump the next social/mobile/casual game .

Plus there's this whole life thing: it just keeps happening! People fall in love, have kids, get new jobs. Markets collapse, personal lives implode, people have medical emergencies... You can't plan for that. You can't control that.

The only constant, the only thing that you can absolutely guarantee will happen, is that time will pass.

When you spend your time on something, it is gone. So what you choose to spend time on is the only variable here you can actually control. And that's what we're testing for. We're aiming to answer two questions about time - yours and your player's:

  1. Will I enjoy spending most of my time, probably for years, working on this?
  2. Will other people choose to spend their time playing this, as opposed to all the other things they could be doing?

Stand Back, We're Going to Try Science...

Remember learning about the scientific method back in school? We are going to get back into that - but now that you're all grown up, you can get away with making it as silly and fun as you like (which we totally did in my incubator for a month during our Test Tube Games competitions - we had lab coats and everything!).

How to Research Ideas and Test Assumptions (Before You Develop Anything)

Photo credit: Darin McClure on Flickr

The whole point of going through the scientific method is to test if what seemed like "a good idea at the time" really is or not - that way you don't end up spending time, money and energy pursuing silly things.

Here's a quick refresher on how this system of testing works:

  1. Ask a question (like, "Will people play this for longer than 5 minutes?")
  2. Do background research (are people spending time on similar things?)
  3. Construct a hypothesis ("I suspect people will spend longer than 5 minutes on this because...")
  4. Test your hypothesis by doing an experiment (or two, or three...)
  5. Analyse your data - both personal feedback and statistical data (were you right or wrong?)
  6. Communicate your results (share it with the team, blog about it, whatever)

Don't worry I'm not going to ask you to be all super strict about how you do this stuff!

The developers I've coached and I have taken the core philosophy behind the scientific method and applied it to three kinds of tests we can do to determine if we've got a game idea worth spending time on (for ourselves and our players):

  • Test Type 1: Comparative Research
  • Test Type 2: Experiential Testing
  • Test Type 3: The (Semi) Public Prototype

You just need a few things to do this right:

  1. A way to keep track of your results (Moleskine notebooks and a wordpress blog worked fine during Test Tube Games).
  2. Access to the Internet (most cafes have Wi-Fi these days and your local library probably does too).
  3. An open mind and willingness to be proven wrong. (Because I promise you will be proven wrong! That's just being human; we get stuff wrong a lot of the time and that is why the scientific method is such a useful system: it protects us from investing in what seemed like a really really good idea at the time.)

The anatomy of a good test:

  • It needs to be low cost (if not free!)
  • It needs to be fast (maximum learning in minimum time)
  • It needs to provide good data and real evidence ("yeah that could be cool I guess" isn't very useful)

Let's look at the three types of test...


Test Type 1: Comparative Research

Estimated Time Cost: Anywhere between a few hours of Googling to a few days of reading/research.

This is the fastest test you can do, it won't give you completely accurate results but you'll at least be able to validate that it's something you're interested in (and seriously, if you can't maintain motivation for a few hours of Googling then it's probably not worth developing further!). You'll also have an opportunity to validate whether or not people are spending time on this kind of stuff already.

Example: A friend of mine at Gamebook Adventures recently asked me to send through a pitch of a game idea I've been talking about. It's a situational problem solving game set in a boy's imagination that blends a few different things I'm really interested in:

  • Oldschool mystery "whodunnit" puzzle books (I loved these as a kid!)
  • Illusion, imagination and parallel fantasy worlds (think Neverwhere, Peter Pan, Harry Potter).
  • Conscious and subconscious decision making (because the human brain is weird and fascinating!)
  • How people learn new ways of looking at and interacting with situations.

First Question to Answer

I think I'll enjoy working on a game that combines all these things - but will they actually hold my interest and keep me motivated through what could be years of development?

Easy tests: Start by Googling for general knowledge on each of these things, then find the most highly rated books on Amazon in each subject and actually read them! Start making things (in my case sketching out situations). If I have the desire to actually finish those books and sketch more scenes and I want to keep doing more... I've probably got enough interest to keep me motivated through prototyping (and if I'm still hungry for more after prototyping then it's probably worth developing further!).

Second Question to Answer

Will other people find this stuff nifty? Nifty enough to spend time playing a game around it, instead of going out, catching up with friends, doing homework or playing one of the many games they bought during Steam sales?

Easy tests: Find out whether people are buying and spending time on similar things. Sure enough, mystery detective books are still popular and there's a bunch of them selling on Amazon. People are also buying 'brain teaser' books and games - increasing perception and critical thinking is something that people are both demonstrating desire to do and a willingness to pay for the experience (if it's good!).

And of course Harry Potter, Peter Pan and Neil Gaiman's Neverwhere are still capturing people's imaginations. (I almost bought Neverwhere again the other day - I keep giving my copies away to people I think will enjoy it!)

Conclusion

While this kind of "market testing" doesn't provide any real evidence that this particular game is worth spending time on, it does answer a few questions about who might be interested in the idea, and whether I'm interested enough that I'll be motivated to spend however many years it takes to make it awesome.

If I want real evidence that this game is worth developing further, I would run an experiment similar to the "Experiential Testing" Red Knight Games did in the next experiment...


Test Type 2: Experiential Testing

Estimated Time Cost: Anywhere from a few days of pen-and-paper prototyping to a week of live experience testing.

This is the next best thing to actually prototyping your core mechanics - it's fast, really easy to tweak and this experiment will help you figure out exactly what those core mechanics should be! Plus there's just something creatively awesome about working with your hands off the computer.

The idea here is to take the core experience of the game you want to create, and create that experience as quickly and cheaply as possible within a day or two (ideally so you can run that experience a few times and tweak things before you sit down to code). Most often this experience is non-digital (unless you're a super l33t programmer) and what people tend to call it is "Pen and Paper Prototyping".

What do I mean by "core experience"? It is the feeling you want your player to walk away with, what you want them to describe to their friends when they recommend your game. Your core mechanics are the most direct and straightforward mechanism for interacting with and engaging in this experience (do they push, pull, jump, talk, walk, write, draw, dance, shoot portals?)

The reason you really can prototype this experience non-digitally without actually having to code the game first, is the fact that most digital games simulate reality anyway - so you can usually simulate your digital gameplay within the real world. If a kid can make up a Knights vs Monsters adventure game with a stick and a trashcan lid, you can simulate gameplay using... well, whatever you want to use!

A Real Example

So, let's say you want to make a very zen game around the art of origami - the core experience you're after is relaxing minimalism as you learn the art of origami without distraction. This creates a tricky design question: how exactly do you produce this experience without distracting visual feedback like popups and text?

That experience of teaching someone Origami in a very minimal, non-flashy way, is exactly what Gonzalo Araya of Red Knight Games wanted to test before asking his programmer to hack together a prototype (his programmer was busy finishing off their last game at the time). I asked them to film the test so we could share what Experiential Testing looks like with you, and they asked me to be their guinea pig test subject (because I'm a total and complete n00b when it comes to Origami!). Check it out:

First Question to Answer

What is the Core Experience you want to create for your players? What feeling do you want them to walk away with?

Second Question to Answer

Only answer this after you've answered the first question and you know what feeling/understanding/experience you want the player to have:

What is the most direct way to produce (and learn from producing) that experience, right now?

Another Example

Dan Graf, a designer friend of mine currently at Halfbrick over in Australia, has been working on a beautiful game world in his spare time for many years. What stops him from taking it into develoment is the immensity of the project! To make what he has in mind would require a great deal of time and money to produce: interaction between players is critical to his core experience and the moment you say "online multiplayer" you can double or triple the amount of work involved in getting that kind of game feeling right.

Until he's answered the key assumptions "would I enjoy developing this?" and "would other people enjoy spending their time on this?" he's not comfortable asking people to invest their time into his idea.

Possible Test: The most direct way to produce the experience he wants to deliver, right now, would be to design a one-off live-action-roleplay game (the closest non-digital equivalent to an MMO) and submit it to the Australian roleplaying festivals Gen Con and Eye Con. Running his game at a festival gives him a chance to see how players respond to his world and see his core engagement mechanics in action. This kind of "live experience" testing would give him tons of data that he can use to refine his idea before working on a public prototype.


Test Type 3: The (Semi) Public Prototype

Estimated Time Cost: Between one and two weeks per iteration (less if you're a super l33t programmer).

Finally - and this one's classic - spend a week or two prototyping your gameplay and get people to play it!

How to Research Ideas and Test Assumptions (Before You Develop Anything)

It is very important not to spend more than two weeks on each prototype - any more than that and your natural (and wonderful) desire to complete a project will kick in and you'll accidentally switch from an experimental mindset into "developing" an untested idea.

You can't help yourself from wanting to finish something and do it right. That two week limit keeps things fast and loose and helps you focus on experimenting and testing so you an maximize how much you learn in the minimum amount of time.

Whom you get to playtest is totally dependent on the game you're making. You may only have a few close peers play it (if you can trust them to be absolutely honest in their feedback), or you may desire a larger pool of data and feedback from a public release of your prototype online (if it won't spoil the experience for your players later).

Going back to that "figure out your foundations" building-a-house analogy we had before, this prototype is all about getting to that core experience: the feel of play, not the look of it; we're not worrying about the details yet, just how those mechanics feel.

(And anyone who says you need finished art before a player will enjoy the game evidently never played Dungeons and Dragons, Pong , Zork or any of the original "golden age" era games.)

Getting your mechanics working and feeling great answers the first question of "will I enjoy spending my time on this?", but you have to get people playing it if you're going to be able to answer the second question about whether or not it's worth their time.

An Example

Matt Cabanag, one of the founders of Silver Nova Software and a published researcher in artificial intelligence, sought to craft a new control scheme for real time strategic gameplay that enables the intense tactical experience you get from playing chess - that hope that if you pay attention and act fast in response to moments of weakness you can win, even against a more skilled player.

Before Silver Nova Software could flesh out the rest of the game or the game world they were interested in creating (with the award winning talents of fiction writer Christian Read), they had to test if they could deliver that core experience from Matt's control scheme alone. If the core mechanics, on their own, produce the experience he's after for his players, then whatever game world they build onto those foundations will be enjoyable at the core.

First test: Can they produce a control scheme (core mechanics) that produces that fluid, real time tactical experience at all?

First experiment and public prototype #1: Their first prototype was focused entirely on getting the feel of those core mechanics right to produce the core experience they were after. Through internal playtesting, external playtesting and online playtesting they gathered survey results and data from in-game analytics to learn as much as they could about the player experience, and use that information to guide the next iteration.

Their Results: After many consecutive weeks of that Design > Build > Playtest > Learn cycle, Matt and his team had a control scheme that consistently produced the feeling they were after. Finally they were ready to answer their second big question:

Second test: Will people want to invest time on a game built around this core experience?

Public prototype #2: Before committing themselves to a few years of development, the Silver Nova Software guys needed to validate their assumption that total strangers would spend time on a game like this. So they fleshed out the prototype into a five minute "mini-game" they could demo at Supernova, a comic and games convention in Sydney. How long people played for and whether they submitted their email address for updates on the game would provide the data Silver Nova needed to move forward with this game (or start testing the next one). As almost a lark, Matt put up a sign asking for Kickstarter-style donations to support development in return for "first fan" perks.

Their Results: Success! People enjoyed the game enough to play it again, even after five minutes, and frequently provided their email address to receive updates on the game. As a bonus, the team even made a few hundred dollars that weekend from donations! Total strangers were willing to donate time and money to a mini-game based on the core experience and core engagement mechanics Silver Nova Software set out to create. They've committed themselves to another year of development - you can track their progress here.


Which Type of Test Is Best?

Embrace the awesome power of "AND"!

Test 1 leads into Test 2. Do the Comparative Research then start Experiential Testing.

Test 2 leads into Test 3. If the Experiential Testing goes well and you want to proceed into public prototyping you can build on the testing you've already done, prototyping to answer the design questions you weren't able to answer in "live" play.

And if an idea makes it all the way to Test 3 and you can easily say a resounding and empathic "Yes!" to:

  1. Will I enjoy spending most of my time, for years, developing this?
  2. Will others choose to spend their time on this, as opposed to spending that time on other things?

...then you've got something worth developing!

Once you have that game worth spending time on (yours and your players') then you can start looking into the engine infrastructure, art, audio and business models that make the most sense for your particular game.

Remember, your time is precious and once you spend it it's gone. And that's why these tests exist: to give you the opportunity to make sure you're spending your time on something meaningful before investing more time (and money) into development.

So good luck, get testing, tell us about your experiences in the comments! And if you've got any questions (or you just like the cut of my jib and want to read more) you'll find me writing at indiebits.com and tweeting as @indiepieces.

Thank you for reading! In the next article we'll get into my three favorite methods for generating new ideas that take into account all the things that make you uniquely you (your skills, interests and background) so that you're generating new, original experiences, from the very beginning.

Advertisement