battlestorm game evaluation summary

Evaluation conducted by: Cause Communications, Network Impact

Summary of

Evaluation Findings

Games as a Tool for

Engagement

Playing games can bring together fundamental

aspects of psychology, sociology and technology to

engage people for social change.

72% of American households play computer and

video games.

Have seen the rise of games in popular culture with

video games and reality television competitions

Games engage the public’s imagination

Why fund social impact

games?

Knight Foundation aims to increase the ability of individuals to engage in change

Seeking innovative ways to make that happen

Research on digital games shows they can improve learning and influence behavior

Knight wanted to see if the same was true for games that took place in the real-world, with real-people, in real-time

Why fund social impact

games?

Funded two pilot projects: – Battlestorm in Biloxi, MS,

– Macon Money in Macon, GA

Macon Money – Knight partnered with local organizations and residents on a game that fosters new connections between people and with local businesses by creating a local currency to spend in participating stores and restaurants.

Connecting residents and

spurring economic growth

Macon Money

Goal: Engaging youth and their families

in hurricane preparedness with activities

focused on youth as leaders.

A new way to engage kids around hurricane preparedness and get them interested – Give them a new language to use to talk about

hurricanes

Influence the behavior of parents and community

Empower youth to act safely and take responsibility during dangerous storms and their aftermath

Why the Mississippi

Gulf Coast?

In the Gulf Coast Region – levels of preparedness had not increased in past 5 years

Could tap into Boys and Girls Clubs’ afterschool programming

Committed to recovery and resiliency through previous grantmaking efforts – Gulf Coast Community Foundation, Habitat for Humanity, Knight Nonprofit Center

IP Center

East Biloxi

Qatar Center Hancock

County

Forest

Heights

Partners and Cities

What can be learned about

potential uses of real world

social games?

How did the game operate

as a tool for community

engagement?

How can this game be used

as a template for other

communities or issues?

Advisory Board: Consulted on

Game from Soup to Nuts

James Paul Gee, Arizona State University, Digital Literacies, Situational Learning

Beth Kolko, Univ of Washington , Berkman Center, Digital Games Research Group

Tracy Fullerton, USC, Electronic Arts Games Innovation Lab,

Ben Stokes, USC, Co-founder, Games for Change

New game: combination of

dodgeball and freeze tag

493 players total in 5

communities on Gulf Coast

10 - 14 year-old boys and

girls, after school at Boys and

Girls Clubs

March – May, 2011

Iterative Approach to

Game Design

Tap into community character Intensive

development process for designing the game

that started by understanding the community

Fun is paramount! Kids have to want to play

the game

Tap into existing local networks for players,

team support and big event

Make hurricane prep a public, social effort.

Iterative Approach to

Game Design

How to Play

Youth from different neighborhoods play on teams representing the “Town” against Navy See Bees representing the “Hurricane”

Town teams can earn tokens as people in their community upload photos of hurricane prep kits to the game website

Tokens can be cashed in to buy extra powers like “Shelter” (which goves teams a safe zone on the court) or “Flashlight” (which removes a Hurricane player)

Months of practice culminate in a tournament

Community

Involvement

Community Involvement

Big Event tournament -- youth played against Team Hurricane of Navy Seabees

Photos of family’s and friend’s prep kits uploaded to game website give “power boost” to

Attendees took home a prep kit from watching the tournament and the winning team took an additional 500 kits home to their neighborhood to hand out

How it worked:

4 Potential Pathways

for Change and Impact

Pathways for Change

Players would learned about hurricane prep from

game which would also prepare them for future

learning

Game would leverage youth as “superconductors”

for information for their families and communities

Parents of players would become more

knowledgeable and prepared

Game would trigger conversation, catharsis, healing

Pathway 1:

Knowledge Gain and Prep for

Future Learning

Least effective results (although important to note that game was not designed around learning or knowledge gain)

Surveys confirmed that players learned little about hurricane characteristics or behavior

Game did not prime kids to learn about hurricane prep more easily in the future

Knowledge gains limited

Knowledge Gain and Prep for

Future Learning

Three exceptions…

1. Linking game strategy to specific concepts was

connected to learning retention

2. Small but significant increase in knowledge of

hurricane prep kits items

3. Higher quality of questions asked about hurricanes

and hurricane prep post-play

Pathway 2:

Youth as “Superconductors”

of Information

Graphic Slide here

Youth as

“Superconductors”

of Information

Youth as “Superconductors”

of Information

Research has shown that conversations can

lead directly to action, which is why this potential

pathway is so important.

Game was a trigger for conversation and

preparation (disaster prep needs a trigger)

Youth as “Superconductors” –

Talking with Parents

68% of Battlestorm players started/continued

talking with parents about hurricanes vs. only

38% in control group

One third of parents reported learning something

new about hurricane prep from their child who

played the game



Example: After a conversation with his daughter, one parent put their family’s important paperwork in plastic bags

Example: One player interviewed his grandparents to make sure they had all of the materials they needed to make a prep kit

Example: a girl from East Biloxi told her father that she was playing a game about hurricanes, which reminded him to update the family’s flood insurance


Sharing with Friends

After the game…

– 44% of players had spoken with a friend about how to be prepared for a hurricane vs. only 23% at baseline

– 40% of players had spoken with a friend about what goes into a hurricane kit vs. only 10% at baseline


Sharing with Friends

Pathway 3:

Parents More Knowledgeable

and Prepared

Parents More Knowledgeable

and Prepared

25% of parents learned something new from watching the game

Families of players had more hurricane kit items after the game

After the game… – 65% of parents had spoken with their kids about how to be

prepared, what goes into a prep kit, and about having a family communication plan

– 59% of parents had spoken with kids about a family evacuation plan

Pathway 4:

Conversation as Catharsis


Talking about games allowed families to process

experiences and talk about hurricanes in an

empowered way

– More than half of Battlstorm parents surveyed confirmed

that their family was directly effected by Katrina

– Of those, nearly 1 in 5 said their children continue to

experience symptoms caused by the storm

– 75% of players’ parents believe that games like

Battlestorm can help children withemotional or

psychological problems related to hurricanes

Challenges:

Submit Your Kit

Challenges:

Submit your Kit

33% of families received kits at Big Event rather

than through self-preparation

Kit assembly component not promoted adequately

with instructors and families

Digital divide did play a small role

Additional training on how to take pictures of kits

and how to upload photos to the website could have

helped this component

Challenges:

Community

Partner

Engagement

Challenges:

Community Partner Engagement

Partner contributions could have been more

strategically integrated or coordinated

Participation in the Big Event could have been

leveraged more effectively in community/with

partners

Needed more explicit explanation of game’s goals

and purpose with audiences such as instructors,

parents, and partners

Challenges:

Community Partner Engagement

Positives:

– Partners did see opportunity to promote

preparedness AND physical activity at the

same time

– Partners saw value of engaging young people

as levers for community change

Best Practices: Set Up

Games require significant local staff who have both time and talent!

Multi-stakeholder approach is key in game development and implementation (e.g., Game designers, foundation, community leaders, community partners)

Don’t leave partnerships to chance - Intra-organizational partnerships need orchestration and coordination

Best Practices: Game

Design

Games can be a catalyst for awareness,

attitudinal, and/or behavior change

Balance fun and Education - Fun is paramount,

but tie context and concepts explicitly not implicitly

Link game strategies to learning outcomes

Be thoughtful about conveying the social purpose of

game to players and partners

Best Practices:

Implementation

Best Practices:

Implementation

Evaluators can play an important role in

mid-course corrections; monitoring

implementation is critical

Strategic communication can be key to

engaging players, partners and community

in game and Big Event

Replication

Recommendations

Customize for other types of disaster preparedness; not “generic hazard”

Create a discussion guide for instructors to bridge in-game and out-of-game experience

Consider integrating into schools vs. after-school programs

Improve “Submit Your Kit” mechanism with early outreach and greater visual clarity; game mechanic may still be valid

Replication

Recommendations

Improve Big Event coordination, logistics, and

promotion; focus as much on spectator experience

as on player experience to engage community

Think about follow-up communications that can

leverage increased learning preparedness from

game

– Could include a scavenger hunt element for kit creation

that is done as part of team

Battlestorm In Your

Community

Documentation designed to help anyone

interested in creating Battlestorm in their

community can be found at this link

http://battlestormgame.org/battlestorm-in-your-

community/

http://battlestormgame.org/battlestorm-in-your-community/







More on Knight and Social

Impact Games

Please visit: Knightfoundation.org/games/

Evaluation and

Assessment Methodology

Pre- and post-game survey to players with a control group

Post-game focus group with players

Pre- and post-game survey to players’ parents/adults care-givers

Participation observation of game practices and Big Event

Exit survey to Big Event audience

Baseline and follow-up interviews with community partners

Interviews with game advisory board and Area / Code

Surveys as part of a process to test the game’s effect on “preparation for learning”

Questions?