building better learning games: leveraging game design and user testing for results

© 2009 Enspire Learning and D2D Fund, Inc.

Building Better Learning Games: Leveraging Game Design and User Testing for Results

April 9, 2009Austin, TX

- 2 -© 2009 Enspire Learning and D2D Fund, Inc.

Today’s Agenda

Introductions

Casual video games and learning needs

Case Study: “Celebrity Calamity”

User-testing and Game Development: Lessons Learned

Questions and Answers


Introductions

Ben Katz, Enspire Learning, Lead Game Developer

Nick Maynard, D2D Fund, Director of Innovation

Kristy Bowden, Skillpoint Alliance, Director of Digital Media Council


Enspire Learning

Founded in 2001 by Bjorn Billhardt and based in Austin, Texas, Enspire creates:

Learning Games

Simulations

E-Learning Solutions“Rise of the Shadow Specters”

Simulation E-Learning Simulation


Doorways to Dreams Fund (www.d2dfund.org) was co-founded in 2000 by Harvard Business School Professor and Associate Dean, Peter Tufano; D2D is

A non-profit 501(c)3 headquartered in Roxbury, Massachusetts,

Whose mission is to increase savings and asset-building for poor and working poor Americans,

And our corporate analogues are an out-sourced:

– R&D lab

– New products group

– Product management team

Doorways to Dreams (D2D)

http://www.d2dfund.org/


Skillpoint Alliance – Digital Media Council

Skillpoint Alliance is a 501(c)3 nonprofit organization that builds partnerships with industry, education and community to support economic growth in Central Texas.

Skillpoint’s Digital Media Council focuses on the digital media and creativetechnology fields in the region, bringing together industry executives and educational and community leaders to engage in joint problem-solving on challenging local workforce and education issues.


Today’s Agenda

Introductions






Games at 50,000 Feet

While this is not new to many (nor comprehensive), games come in a variety of shapes and sizes:

Console Games: Halo (XBOX), Super Smash Brothers (Wii)

MMORPG (Massively Multi-player Online Role Playing Game): World of Warcraft(Adults), Club Penguin (Kids)

Simulations: Flight Simulator, The SIMS (a strategic life simulation game)

ARGs (Alternate Reality Games): World Without Oil– From Wikipedia: An interactive narrative that uses the real world as a platform, often involving

multiple media and game elements, to tell a story that may be affected by participants' ideas or actions.

But there is another whole genre….


Casual Games

Do you play any of these games?


Casual Games

Bejeweled

Tetris

Solitaire


Casual Game Demographics

In 2007, 61% of online game play was either puzzle/board/game show/trivia/card games or downloadable

version of games like Bejeweled and Diner Dash


Casual Game Demographics

Who is playing casual games?

Gender– 63% Women

– 37% Men

Age– 71% are 25-54

• 21% are 25-34

– 17% are 55+

Families– 46% have children

Experience online– 90% have been on the Web longer than

three years


Casual Game Design

Casual video games share these design principals:

Low barriers to entry– Very few screens to go through before start

– Few or “as you play” instructions

Forgiving– No punishment if you do something wrong

– Rewards error, oftentimes

Short play times– During a coffee break, can have meaningful play for 10 minutes

Highly re-playable

Depth of game play is revealed gradually (“accessible” to everyone)

Themes are often non-violent

Skywire 2


Casual Game Development

Casual games make it easy to:

Build a prototype and iterate quickly and efficiently

Closely match game mechanics to key learning objectives

Deploy games online in Flash (and eventually on web portals such as Kongregate)

Diner Dash


Today’s Agenda

Introductions






Why are we here?


Financial Entertainment

“D2D’s vision is financial entertainment. Taking cues from business and entertainment, we need to work with and for consumers in the development of engaging new media that teaches them how to better manage their money.”

As a starting point, we envision a library of casual video games, each teaching simple financial lessons; these include:

Use of credit and debit cards;

Personal budgeting;

Basic calculations, like compound interest;

Saving for retirement;

Student and other loans; and

Awareness of expensive pitfalls (i.e., pay-day lending, check cashing)


Why financial literacy casual games?

To date, efforts in this space have been unsuccessful at finding the right balance between fun and education:

Very complex simulations that can seem like Quicken in a video game

Flashy graphics and audio surrounding a multi-choice quiz type presentation of facts

Brand marketing packaged as an video game


First Prototype Game: Celebrity Calamity

While focused on fun, the game’s explicit learning objectives include:

Paying more than the minimum credit card payment;

Minimizing credit card finance charges;

Avoiding all fees including bank overdraft, credit card late payment, and credit card over-limit; and

Making good annual percentage rate (APR) choices.

The game also includes a number of implicit learning objectives, such as raising awareness of spending behavior and the value of saving money.

Game Trailer:

www.youtube.com/watch?v=oX2tr4RHvYM

Prototype Game:

www.celebritycalamity.com/

http://www.youtube.com/watch?v=oX2tr4RHvYM

http://www.celebritycalamity.com/


Celebrity Calamity: Lessons Learned

Know your audience.

Games are a popular form of digital media.

Listening to people’s needs and preferences is important.

Evaluation is important.


Know your Audience.

We are creating games for a financially vulnerable market:

Low to moderate income females

18-32 years old

Single or Married with children

Household income less than $40,000

Education of at least high school or GED

Ethnically diverse

This segment faces a number of important challenges:

Very low levels of financial literacy

Peer effects do not work in their favor

Little or no savings

Limited opportunities to receive financial education


Games are a very popular form of digital media.

Digital media offers the potential to marry fun and education:

72% of Americans are playing video games;

The rates of play are highest among those under the age of 35;

Casual video games are the fastest growing segment in the game industry; and

The casual segment is dominated by female players


Listening to people’s needs and preferences is important.

Celebrity Calamity was created in partnership with users:

Game played and tested at three key development milestones with audience;

Individual, focus-group, and observational feedback;

Extension of the game development process timeline to integrate feedback into design.


Celebrity Calamity Game Development Process

FirstPlayable

Alpha Beta Final

At each development milestone, :

Fun. Are players enjoying the game? Do they want to continue playing (if the game was fully developed)? What components of play are most enjoyable?

Learning needs. What do players know about the teaching topics? What don’t players know? What aspects of the game connect with real life?

Assessment. Are players engaged? What level of enthusiasm due they show for the content? Do we observe increases in self-confidence?

Test Test Test


Game Development Process: Additional Benefits

This development partnership also:

Brings together digital media companies with a financially vulnerable audience;

Exposes low-to-moderate income women to the digital media industry;

Raises awareness about the importance of digital media’s potential impact on societal issues; and

Highlights the digital media industry.


Evaluation is Important.

D2D has completed preliminary testing of Celebrity Calamity, and we are pursuing more rigorous evaluation given the positive initial results

Early evaluation was incorporated in the product development process at Beta

Preliminary formal testing has been completed with 44 LMI women in three cities: Austin, TX, Tulsa, OK, and Providence, RI

Additional play-testing has been completed with high school students in Dorchester, MA and an older Spanish-speaking audience in Lawrence, MA


Preliminary Game Efficacy Testing: Self-Confidence

Table 1: Self-Confidence (Pre and Post, % Change)

Rate your degree of confidence in the following: Pre Post %

Change bad spending habits 3.50 3.89 11%

Identify needs and wants separately. 4.02 4.30 7%

Avoid impulse and emotional spending. 3.23 3.93 22%

Pay off new charges on credit cards every month. 3.16 3.84 21%

Pay more than the minimum amount on credit card bills. 3.16 4.09 29%

Avoid late fees, over-limit fees, and overdraft fees. 3.23 4.05 25%

Save money regularly. 3.14 4.14 32%

Manage my finances. 3.57 4.09 15%

Perform assigned job-related tasks. 4.23 4.43 5%

Follow directions from my boss/supervisor. 4.66 4.61 -1%


Preliminary Game Efficacy Testing: Knowledge

Table 2: Knowledge (Pre and Post, % Change)

Knowledge Area: Pre Post %

Annual Percentage Rate (APR) 55% 86% 58%

Finance Charges 48% 82% 71%

Pay More Than the Minimum 82% 93% 14%

Interest Compounding 61% 64% 4%


Qualitative Feedback from the Audience

Enthusiasm: “Can I have a copy of the game? Where can I buy it?”

Engagement: “I’ve got the hang of it now, I can do this.”

Education: “I’ve had credit cards for a while now, but I did not know what annual percentage rate, APR, was, until now.”

Empowerment: “It helped me to better understand how to manage money.”


Celebrity Calamity: Lessons Learned

Know your audience.– While casual video games are quite popular among women, in general, identifying a

specific demographic allows for focused innovation.

Games are a popular form of digital media. – Despite the “Digital Divide,” lower-income young women have grown up playing video

games are actively engaged with digital media.

Listening to people’s needs and preferences is important.– By incorporating the end-user into game development, D2D has created a

“participatory learning” platform for financial education.

Evaluation is important.– Assessment and feedback are built in throughout the product development processes

as well as more formally once game development is complete.


Celebrity Calamity Next Steps

1. More rigorous evaluation of Celebrity Calamity;

2. Making additional casual games (second game in development); and

3. Testing distribution strategies along with associated social marketing techniques; target channels include: – Employers

– Community Colleges

– Grassroots Organizations (non-profits)

– Financial Institutions


Testing Partnerships

If you want to join a growing list of national employers and organizations testing distribution of Celebrity Calamity, please contact Nick Maynard, [email protected]

We are interested in game deployments that would help us learn the following:

• How many employees/individuals play the game? For how long do they play? What do we know about them demographically (e.g., age, gender)? Where do they play the game? Do other members of their household play the game?

• Which marketing approaches work best? Which frame is most effective for driving play? Does content associated with game characters engage the audience?

• Are there other educational materials that can be combined with the game to improve impact with specific audiences?

mailto:[email protected]


Today’s Agenda

Introductions






User-Testing: Lessons Learned

Test early, test often– Can’t make game in a vacuum

– Test at key deliverables, and then iterate

Usability testing first, then efficacy testing later on– Early prototypes more focused on creating best user experience

– Focus on knowledge and confidence changes after usability of game is assured

Collect as much good data as possible– Likert scale based confidence questions

– Knowledge questions

– In-game data collection for future analysis

– Avoid focus groups until end of session


Game Design: Lessons Learned

Games you like aren’t necessarily games your target demographic likes– Game designers aren’t like everyone else. They’re weird.

– Interface elements we take for granted aren’t understood by everyone (ex. Click to Continue, Slider)

Prototype early and iterate often– Avoid heavy multimedia development until later on

– Game appearance isn’t as important as game mechanics, especially at an early stage

Game goals must be immediately apparent to players– This is not the place for subtlety

– Win and lose states must be reasonable and fair

– Learning objectives must be closely matched to success in gameplay


Education: Lessons Learned

Discipline in the scope of teaching content– Fun can quickly be overwhelmed by rush to include teaching points

– If teaching points are not relevant to game play, players will miss them

Repetition, repetition, repetition– The second celebrity in Celebrity Calamity and APR

• Character talks about APR• APR changes every round• “Calamities” are tied closely to APR impacts (e.g., cash advances)• APR offers dropped with the money

– Multiple touch points in the game design

Players can learn from failure– Games allow for changing strategy and trying new things

– Scoring must reinforce learning points clearly


Question and Answer


Presenter Contact Information

Ben Katz, Enspire Learning: [email protected]

Nick Maynard, D2D Fund: [email protected]

Kristy Bowden, Skillpoint Alliance: [email protected]


