formative evaluation report zoom on the webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 executive...

33
Formative Evaluation Report ZOOM on the Web (Drawing by Sara, age 8) May 5, 1998 Prepared By: Shannon Beltz Erik Blankinship Rachel Garb Marnie Oakes Harvard Graduate School of Education Prepared For: WGBH Interactive

Upload: others

Post on 12-Jul-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

Formative Evaluation Report

ZOOM on the Web

(Drawing by Sara, age 8)

May 5, 1998

Prepared By:

Shannon BeltzErik Blankinship

Rachel GarbMarnie Oakes

Harvard Graduate School of Education

Prepared For:

WGBH Interactive

Page 2: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

Table of Contents

Executive Summary ..............................................................................................................2

Project Description ...............................................................................................................3

Research Goals.....................................................................................................................4

Subject Demographics..........................................................................................................5

Methodology ........................................................................................................................7

Findings................................................................................................................................9

Drawings ........................................................................................................................10

Surfing............................................................................................................................16

Focus Groups.................................................................................................................20

Questionnaires................................................................................................................22

Recommendations...............................................................................................................29

About the Researchers ........................................................................................................31

Page 3: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

2

Executive Summary

To assist the developers of the new ZOOM Web site for kids aged 8-12, our formative

evaluation research focused on three main areas: context, content, and community.

Context addresses issues of navigation, visual appeal, and the fit with the ZOOM TV show.

Content is about making the Web site more fun for kids. Community building is

creating a fun, sustainable “kids-only” online environment within the ZOOM Web site.

Our research involved 21 children in the target age range and our methodology involved a

series of activities. First, the kids viewed a portion of the ZOOM pilot TV show. Next, they

drew their ideas of what the ZOOM Web site should look. The kids then spent some time

surfing the ZOOM Web site. Afterwards, they shared their ideas about the site in a focus

group discussion. Finally the kids filled out a paper questionnaire.

In general, the subjects were excited about the ZOOM concept, and were attracted to the

pilot TV show. Their enthusiasm is reflected in creative drawings of their ZOOM Web site

ideas. However, the initial version of the ZOOM Web Site did not meet their expectations.

Throughout the surfing activity, the focus group discussions, and the questionnaire

responses, kids expressed their disappointment.

Our report details suggestions from the kids for improving and personalizing their ZOOM

Web site experience. Based on their exposure to the Pilot TV show and their visits to other

Web sites, kids want to see real people, more pictures, and more games to play on the site.

Kids had a strong reaction to the amount of on-screen text and the unwieldy submission

process for their ideas.

Based on the findings of the study, we offer the following recommendations:

• Clarify ZOOM terminology and links.• Provide more opportunities for kids to personalize their ZOOM experience by

submitting their own drawings, games, jokes and web site links.• Make the submission process for all ideas more intuitive and accessible from

anywhere on the site.• Develop the ZOOM on TV area to include more information about the cast and

more news and information about the show.• Offer a richer variety of media on the Web site, such as video and audio clips.

Page 4: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

3

Project Description

A Web site, intended for children 8-12 years old, is currently under development by

WGBH Interactive. The ZOOM Web site is a complement to the new ZOOM television

series with content by kids, for kids , displayed online. During the initial phase, a 30 minute

pilot of the new show will be airing nationwide; material on the site will be designed for the

sole purpose of prompting kid submissions for this new series. Once a sufficient number

of submissions has been generated and the pilot stops airing, the developers intend to

expand the initial site design and underlying technology to create a kids’ online community.

ZOOM’s Web presence is expected to complement and support the interest and enthusiasm

generated by the new TV show.

The ZOOM Web site is being positioned as one component of the “ZOOMuniverse.” In

the short term, while the ZOOM pilot is airing nationwide, the material on the site is

designed to be experienced, experimented with, and expanded upon to prompt additional

submissions for the new TV series.

This initial version of the ZOOM Web site is intended to:

• Feature highly produced kid submissions• Offer ZOOMgoodies such as a ZOOMscreensaver and ZOOMwallpaper• Promote ZOOM on TV with a description of the pilot, streamed video of the 30

minute pilot and a comprehensive schedule of pilot airings on the many PBSstations nationwide;

• Solicit submissions by displaying contributions that other ZOOMers have sent in;and, provide an electronic collection mechanism for text based contributions whichwill be entered into web page forms and sent to the ZOOM staff via e-mail.

Once this first set of goals is successfully met, the developers intend to expand the initial

site design and underlying technology to create an online kids’ community. This second

phase will be realized through the implementation of a backend database for user profiles

and submissions, allowing personalization of the user experience. Automated collection,

storage and dynamic display of submissions are also planned.

By the time the show airs in January 1999, WGBH expects the site to be a cool, safe place

for kids to hang online–a place where they can: share their ideas, display their work to their

peers, find fun things to do, stretch their thinking, help one another solve problems, and find

friends in the virtual world of ZOOM on the Web.

Page 5: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

4

Research Goals

Our research was geared to meet the initial concerns and questions raised by the developers.

The main topic areas are listed below:

• Does the intended audience think the ZOOM Web site contains content that’s fun?• Do users have any navigation problems?• Does use of the ZOOM Web site match the ZOOM TV viewing experience?• Does the ZOOM Web site provide users with content and activities that they want

from the Web for this program?• All ideas for this program come from kids who watch ZOOM on TV. However, the

presentation of kids’ ideas on the ZOOM Web site is created by adults. Are kidsaware (from the appearance of the Web site) that it is created by adults?

• Is it necessary that the ZOOM Web site maintain the illusion that adults have noinvolvement in ZOOM?

• Are there other issues of breadth and depth that need to be addressed?

The researchers organized these topics, and related questions, into a framework of three

“C’s.” The three C’s are: Context, Content, and Community. Listed below are the

overarching questions we considered in the development and design of our data collection

process.

Context

• Do kids feel the Web site fits well with the ZOOM TV experience?• Do kids find the Web site easy to navigate?• Do kids find the Web site visually appealing?

Content

• Do kids find the content fun?• Do kids like the way the content is organized?• Where do kids like to go on the Web for content?

Community

• Do kids prefer a kids-only community over a kids-and-adults community?• Do kids want to participate in the Web site community?• Do kids find it easy to participate?

Page 6: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

5

Subject Demographics

General Background Information

A total of 21 subjects participated in this formative evaluation study. The majority of the

children were recruited from Cambridge and Somerville, MA areas. Their age and gender

breakdown are as follows:

• Age range: 7 to 12 years• Average age: 9 years, 4 months• Gender: 17 girls; 4 boys

Internet Experience

Most of the subjects use the Internet on a regular basis, and their experience varies, as

shown in Figure 1, below. For 14% of the participants, visiting the ZOOM Web site was

their first exposure to the Internet.

How long have you been using the Internet?

0 1 2 3 4 5 6

More than 2 years

1 to 2 years

6 to 11 months

3 to 5 months

Less than 3 months

Never before today

# Responses

Figure 1

As shown in Figure 2, most of the subjects said they would be accessing the ZOOM Web

site from their home. None of the participants chose School as a point of entry for the site.

Page 7: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

6

From where would you visit the ZOOM Web site?

0 1 2 3 4 5 6 7 8 9 1 0

Friend's House

Library

School

Home

# Reponses

Figure 2

We also asked the subjects to name their favorite places on the Internet. Sites most often

mentioned were Beanie Babies (12 subjects), Nickelodeon (4 subjects), and Titanic (3

subjects). Other favorite sites include Winnie the Pooh, Lego, MTV, BET, NFL, PBS,

Iditarod, and GamePro.

Page 8: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

7

Methodology

Our research methodology consisted of a series of five activities, outlined below:

Activity 1: Viewing of the TV Pilot

The subjects watched highlights from the ZOOM TV pilot, provided by WGBH.

Activity 2: Drawings of the Web Site

The researchers explained to the subjects that the ZOOM TV show has a companion Web

site under development. Subjects were asked to draw pictures of what they thought the Web

site should look like. Examples of these drawings, along with explanations from the

children, can be found beginning on page 10.

Activity 3: Surfing the ZOOM Web Site

In this activity, subjects visited the ZOOM Web site for 15 minutes. In each research

session, subjects were divided into two random groups: Subjects in Group A were asked to

rate each page they visited, according to the scale below:

1Zucks!

2Not Good

3OK

4Good

5Zuper!

Highlights of Group A’s ratings can be found beginning on page 16.

Subjects in Group B were asked to surf on their own. Their reactions were collected in later

activities. Navigational histories (surfing patterns) were collected from every subject’s

computer.

Activity 4: Focus Group

All subjects were brought together for an informal, open discussion about the ZOOM Web

site. Subjects shared their feelings about the look, navigation, and content of the site.

Researchers started off some general discussions with these three framing questions:

• “What did you think of the ZOOM Web site?”• “Would you contribute your ideas to the ZOOM Web site?• “What changes, if any, would you make to the ZOOM Web site?

Page 9: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

8

Key findings from the focus groups can be found beginning on page 20.

Activity 5: Questionnaire

Subjects were given paper surveys to fill out individually. Questions were designed to

capture feedback about the overall ZOOM Web site surfing experience, and subject

demographics. These findings are detailed throughout this report. Highlights of the web

related feedback can be found beginning on page 22.

All data collection was conducted on site at the Harvard Graduate School of Education.

Research sessions each lasted approximately two hours.

Page 10: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

9

Findings

From the research activities, we collected a variety of quantitative and qualitative data.

• Drawings of the ZOOM Web site and common elements found within thesedrawings (page 10).

• Data on the frequency of visits to particular areas of the site, and subjects’ ratingsfor most frequently visited areas (page 16).

• Excerpts from focus group discussions (page 20).• Key findings from the questionnaire (page 22).

Page 11: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

10

Drawings

Before introducing the subjects to the ZOOM Web site, we asked them to draw what the

ZOOM Web site should look like based on what they saw on the ZOOM TV pilot. The

drawings indicated trends in terms of the content and functionality they expected. Figure 3

lists the most common elements featured in the children’s drawings.

ZOOM Web Site Ideas from Kids' Drawings

0 2 4 6 8 1 0 1 2

QT Video

Food

Drawing

Help

Chat

ZOOMTV Show

Jokes

Meet the Cast

Questions

E-mail

Ideas

People

Games

Feat

ures

Frequency of Appearance

Figure 3

The following pages contain some drawings, illustrating the children’s expectations and

ideas before they viewed the ZOOM Web site. Comments from the subjects’ own

explanation of their pictures and how their Web site would work are included.

Page 12: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

11

Child: “They have these twovideos, if they wanna play thevideo they press play, and ifthey wanna stop the video, atthat point, they press stop. Thisis the volume. And they have thesame thing over here. And then,you can break it apart and turninto a puzzle.”

Here the child’s drawingfeatures her suggestions forareas where kids can send inideas, learn more about theshow, and play games. She alsodrew a “ZOOM Food” area,where kids can satisfy theirvirtual appetite.

Child: “You click on Games, ifyou wanna play anygames...Jokes to send injokes...Talk to the castmembers...”

Page 13: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

12

Researcher: "What kind of giftswould you want to go get–ifyou click on the flower, whatkind of gifts could you get?"

Child: "Well, like a ZOOM T-shirt, a ZOOM hat..."

This drawing illustrates thechild’s expectations for seeingpeople “have fun” on the Website, and the ability to watchsome of the ZOOM show.

Child: "...and if you click onthis, like, two 'O's' in the middleof the "ZOOM", you coulddesign your own ZOOM, like,Web site, like we did, and youcould design your own ZOOMPlace..."

Page 14: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

13

Along with the ability to playgames, send in questions and e-mail, and see cast members, thischild’s ZOOM site features theability to learn ZOOM songs.

Child: "...and there's a placewhere you can chat with otherpeople...and talk with themabout your ideas and stuff likethat..."

Researcher: "People that areyour friends? Or people fromthe show...or who would youwant to talk to?"

Child: "Other people who areon the Web."

Child: "...and you can click ontime, and it will tell you whattime it is...you can go intoGames like they have on theshow...you can send in yourideas in e-mail...you can getinformation on the show likewhat channel it's on andstuff...you can send in yourquestions, comments... andthere's a joke box..."

Page 15: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

14

Child: "It has games andactivities that you can play, andthey're from the TV show...ifyou wanna send in an idea, ifyou wanna meet the castmembers, you can...and behindthe scenes, you can write to theZOOM people and... Help andQuestions, if you have anyquestions, and I made it alldifferent colors...

Researcher: "Who are theZOOM People?"

Child: "Um...people who makethe film."

Researcher: "The kids on theshow?"

Child: "Yeah."

Child: "and um...I made aperson standing up, with aZOOM shirt on, and they'rejumping up and down saying,"ZOOM" , 'cause they like it,ZOOM the Web site."

Researcher: "And who is thisperson?"

Child: "one of the castmembers."

Page 16: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

15

This child wrote ZOOMbackwards in the center of thescreen because, she said, that iswhat she would see from“inside of the computer.”

Here the child drew a re-creation of the “milk crate”ZOOMDo that was done by thekids on the pilot show.

Page 17: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

16

Surfing

Data reporting from the surfing activity is focused on the areas within “C’mon and

ZOOM”, which constitute the bulk of Web site’s content. Figure 4 lists the total hits for

each area (the number of times subjects visited pages within each of these content areas).

C'mon and ZOOM: # Hits in Each Category

0 2 4 6 8 1 0 1 2 1 4 1 6 1 8

ZOOMvid

ZOOMa cum laude

CaféZOOM

ZOOMdo

ZOOMtech

ZOOMgame

ZOOMlit

ZOOMzinger

ZOOMsci

ZOOPs

ZOOMplayhouse

ZOOMfunny

Area

# Hits

Figure 4

Figure 5 shows the ratings for the areas that were visited the most times during the activity.

The rating scale is 1 to 5, with 5 being the most fun (“Zuper!”) and 1 being the least fun

(“Zucks!”). Ratings for areas that received fewer than 10 hits are not included in this chart.

These infrequently visited areas were: CaféZOOM, ZOOMa cum laude, ZOOMdo,

ZOOMtech, and ZOOMvid.

Page 18: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

17

Figure 5

As we observed the subjects visiting the ZOOM Web site, we recorded their comments, on a

page-to-page basis. Following are excerpts from when the subjects volunteered comments

about content and/or navigation:

Site Area Comments

Home “How do I do anything?”

“Boring. This doesn’t do anything.”

ZOOMplayhouse “What do I do?”

“It doesn’t look like a playhouse. I give it a 2, because when you say‘playhouse’ you think there’s gonna be, well, you would put a housethere would be kids playing inside, but this just gives you items. Imean, there’s somebody down there raising their hand, but that’s allyou see.”

Page 19: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

18

Site Area Comments

ZOOMfunny “How do I get back?”

“No fair. What if you have no arms? I don’t like this joke.”

ZOOMzinger “I can’t find any games...”

Send it to ZOOM “I don’t get this!”

“There’s nothing to do but look.”

ZOOMsci “They tell you how to make stuff.”

Download “Should I get it?”

Search “I think it’s okay that you can type in whatever you want to find out,but I really don’t like that it has the same picture that was on before[referring to the graphic of red-silhouette head sticking out sideways]...” “I think the picture should be different.”

“They should not have the same picture.”

ZOOMgame “What are they doing? I can’t understand what they’re doing.”

“Where’s the sound? I don’t want to play this game...It depends onhow much you like the game.”

“The Cracker Whistle zucks. Because a lot of people know how to doit already.”

“This is the Cup Game. I would give it a Zuper. “

“I think they should make the children talk.”

ZOOMZinger “Oh, that’s where they stand on it...” (recalls activity from TV pilot)

“I can do that.”

ZOOMlit “They don’t tell anything about it. They should have a voice say it soyou don’t have to read every word--and pictures with kids. Make it soit makes sense. You could type in your friends’ names. You couldmake your own show on the computer, with your own friends, but itwould have to make sense.”

“I don’t know what this is...[reads it]...oh, it’s water. I bet a kid didit.”

“I give it a 5 because, you might think it’s about a Flamingo, and itIS!”

“Maybe they should add in comic books, you could print it out so youcould read it later.”

ZOOMgoodies “You don’t really see anything. You only see words.”

“There’s nothing really here. When you think of goodies, you think ofthings that would be good, like games, this is just something that,nothing’s on it, it’s just telling you that we have goody stuff, but youdon’t really see anything. Just words. We thought there was going tobe pictures on it.”

Page 20: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

19

Site Area Comments

ZOOM on TV “You think you’re gonna see people talking and stuff, but it just sayswhen it’s on and stuff.”

“I rate it a 1, because when you go to ZOOM TV, you think it’s goingto be something where you see everybody [from the cast] talking, butthis is just telling you what time it’s on and what channel it’s on.”

Page 21: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

20

Focus Groups

We conducted focus groups on four different occasions. Key trends emerged in these

informal discussions among the children and are highlighted below:

About the Experience

Many times the children indicated that areas of the site did not “make sense.” Often this

was tied to the naming convention of the particular page. The names of different areas of the

ZOOM Web site conjured some positive and creative images for the children, and they were

often confused or disappointed when their expectations were not met. These discrepancies

(e.g. clicking on a certain link, and arriving at a different result) are detailed in the Surfing

section of this document on page 16.

About the Design

Overall, subjects wanted to see more images, including pictures of cast members. They also

expressed a desire for a greater variety of colors throughout the site. The children said there

was too much blue. In addition to requesting more pictures, they wanted less text, and many

times said they didn’t want to read everything on each page.

About Games

Two game related themes were evident in the focus group discussions. First, the children

were excited about the prospect of sending in their own game ideas to ZOOM, and finding

each other’s games on the Web. Second, they were disappointed that many of the Games

featured on the site were static instructions, rather than games they could play right there on

the ZOOM Web site.

About Jokes

Jokes were a popular topic in the focus groups. Every single participant voted that they

would like to send in their jokes to the Web site.

About Submissions

Some children expressed confusion about how to submit their jokes, and said that it should

be easier to send in your ideas. Participants who were able to figure out the submission

process spent a lot of time on the activity, and were very excited about it.

About the TV show vs. the Web site

Page 22: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

21

The ZOOM TV pilot sparked a lot of the creativity and imagination found in the children’s

drawings. Their comments were very positive about the TV show and the overall ZOOM

experience until they saw the Web site. Many of the subjects were looking for interactive

experiences and pictures of real people (i.e. cast members, ZOOM Guests, other kids).

About their “Wish List”

When discussing how they would change the Web site, the children most often mentioned

wanting games, pictures of people and additional graphics, and the ability to draw. There

was a big emphasis on being able to personalize their ZOOM Web site experience by

submitting their own pictures, drawings and jokes, and by chatting with other kids.

Page 23: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

22

Questionnaires

We administered a paper survey with 17 questions. The children answered these

individually; our quantitative and qualitative findings are included in the topics below:

The Fun Factor

One of the key goals driving this research study was to determine if kids think the ZOOM

Web site is fun. The questionnaire prompted subjects to provide their opinion on the “fun

factor,” both in quantitative and qualitative terms.

Figure 6 shows subjects’ responses to the question, “How fun is the ZOOM Web site?”

You can see that the majority of the participants indicated that the site was “Not fun at all”

for them.

How fun is the ZOOM Web site?

0 1 2 3 4 5 6 7 8

Not fun at all

Not so fun

Can't decide

Kind of fun

Totally fun

# Responses

Figure 6

After that question, subjects were given an opportunity to provide written feedback on what

parts of the ZOOM Web site they liked the best and least. The following table lists their

responses.

Page 24: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

23

Parts Liked Best Parts Liked Least

“I like how there are movements andpictures that come to life.”

“The part where they show you poems,plays, and stories with drawings...”

“I liked the part where you could submit aplay.”

“The jokes and the playhouse, because theywere fun.”

“I like the part when the kids are tell[ing]you what happen[ed] to them.”

“The parts where they tell you how to makethings.”

“Feedback page.”

“The videos on the computer.”

“The most fun for me was the page thatma[d]e the [balloon]car, and the jokes.”

Note: When asked the question, “Whatparts did you like best?”, nine childrenprovided negative responses, such as “Ididn’t like anything” and “None.”

“When we had a page error.”

“Wait for the Web site.”

“Everything.”

“All of the parts.”

“All of them.”

“The part when they are singing.”

“Game. Because you only could read howto play them; you could not play them.”

“The Games, because you didn’t get to playthem.”

“The Games, because you could not playthem on the Internet.”

“They should have some games you canplay. I didn’t like that, and they should letyou draw.”

The questionnaire included an area where subjects could write in suggestions for making

the ZOOM Web site more fun. The following is a list of the children’s suggestions.

• “Add more color, pictures, games, cartoons and more interesting stuff.”• “Brighter colors, more options.”• “Put brighter colors, more pictures, and sound effects.”• “Add more pictures, drawings, colors, and games you can play.”• “You should add brighter colors, let them play the games.”• “An interactive game.”• “More colors and more pictures, and web sites.”• “More pictures, drawings, people color and fun activities.”• “People could send in their drawings. You could have a radio on the Web site.”• “By putting more games on it.”• “More games. More Stuff to do.”• “Make kids talk. Have the web site [be] more fun.”• “Make games and have it make sense.”• “More games.”• “[add] Dogs, Kids and Ferrets.”• “Have more ways to make cool stuff like the cars, to make stuff like that, and some

of the movie things.”• “Playing games”

Page 25: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

24

• “Yes, they should [have] more color and pictures and when you want to send in ajoke, they should make the directions more clear.”

TV Show Vs. The Web Site

In this section of the questionnaire, subjects were asked, “Does the ZOOM Web Site

remind you of the ZOOM [Pilot] TV Show that you saw?” As shown in Figure 7, the

overwhelming majority (90%) of the subjects responded no, while only 10% replied yes.

Does the ZOOM Web site remind you of the ZOOM TV show?

Yes10%

No90%

Figure 7

Those who felt that the Web site did not remind them of the TV show provided the

following reasons:

About People:• “Because nobody that is on the show is on the web site.”• “Because the web site really just has color and shadow people”• “Because it did not have kids playing.”• “It didn’t have any of the characters that they had on the show.”• “We did not see the people on the show on the Web site.”

Page 26: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

25

• “Because it doesn’t show the people that were on it.”

About Design:• “The web site is kind of boring in some ways, and it needs more color and

pictures”• “Because the color and environment is different [from the TV show].”• “Because pictures are more on the video than on the web.”

About Activities/Games:• “Because there’s activities on the TV [show].”• “Because I think it’s not like the TV show. Also, because I think the web site

stinks.”

In General:• “It sucks so bad.”• “Because the TV show is better.”• “Because nobody is tell[ing] you to make this or try this.”• “Because it has a lot of different things that weren’t on the show.”• “Because it really didn’t tell you much about the show, and it didn’t tell you about

the cast, or about the [original 1970’s] show before they put it back on.”

Reasons given by children who felt the Web site reminded them of the TV show were:• “Because it showed the milk carton.”• “They showed you a lot of the same things they did on TV.”

Although we did not formally collect data on the children’s opinions of the TV show,

children told us throughout the research activities that they liked the pilot.

Made By Kids?

In Figure 8, we have charted the subjects’ thinking about who actually made the ZOOM

Web site. This question was designed to address the Producers’ concerns outlined in our

“Community” research goals (Are kids aware–from the appearance of the Web site–that it

is created by adults? Is it necessary that the ZOOM Web site maintain the illusion that

adults have no involvement in ZOOM?) The majority of the subjects (52%) indicated that

the site was not made by kids.

Page 27: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

26

Do you think the ZOOM Web site was made by kids?

Yes38%

No52%

Undecided10%

Figure 8

For subjects who responded “No”, we asked the additional question, “Who do you think

made the ZOOM Web site?” Listed below are their comments:

• “Half [by kids] and Half [by adults].”• “The producers.”• “Adults.”• “Adults. Because, I think if it was made by kids, there would be more pictures on

it.”• “The makers of the show.”• “The Director made a script!”• “Older adults and TV Stars.”• “People who made the TV Show.”• “Grown ups.”• “A big fat head with hands.”• “Some grownups made it.”• “Grownups and Kids.”

Page 28: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

27

Target Age Range

In Figure 9, we gauged the subjects’ perception of the ZOOM Web site’s age

appropriateness. The findings fall squarely within the producers’ targeted range of 8 to 12

years of age.

How old do you think the kids are who are visiting the ZOOM Web site?

0

2

4

6

8

10

12

14

16

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Age

# R

espo

nses

Figure 9

Page 29: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

28

Appeal

To measure the subjects’ overall feelings and continued interest, we asked: “Would you

like to visit the ZOOM Web site again?” As shown in Figure 10, 30% of the subjects

responded that they would probably or definitely visit again, while 50% probably or

definitely will not return.

Would you like to visit the ZOOM Web site again?

0 1 2 3 4 5 6

Definitely not

Probably not

Can't decide

Probably yes

Definitely yes

# Responses

Figure 10

Page 30: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

29

Recommendations

In addition to the children’s’ suggestions reported in our questionnaire, we have organized

our formative evaluation findings below under our three main categories of Context, Content

and Community (see the Research Goals section).

Context

The following is a list of recommendations relating to the context of the Web site. These

suggestions address issues of navigation, visual appeal, and the fit with the ZOOM TV

show.

• Incorporate Home page links and/or rename these features suggested in the kids’drawings:

Games People Meet the Cast

E-mail Questions Help

ZOOM TV Show Chat

Submit Your Drawings Submit Your Ideas

• Alternate the color backgrounds or offer more visual breaks from the ZOOM bluebackdrop.

• Make the submission process for all ideas more intuitive, and accessible fromanywhere on the site.

• Clarify the names of the areas of the Web site to eliminate confusion.• Have brighter colors; more variety of color.• In addition to using ZOOM terms for links to other pages, also include visual icons

that help kids get a sense of what the content will be.• Have more pictures/images.• Decrease repetitious graphics (i.e., the red silhouette head, etc.)• Reduce the amount of text on each page.• To the extent possible, offer more interactive media experiences with video and

sound.

Content

The following is a list of suggestions for making the content of the Web site more fun for

kids.

• Provide a balance between games the kids can get instructions for, and games theycan actually play on the site.

• Develop the ZOOM on TV area to include more information about the cast andmore news and information about the show; offer more video clips/segments fromthe show.

• Offer opportunity for kids to make their own ZOOM web pages.• Offer an area for kids to send for ZOOM memorabilia/merchandise.• Develop the Help area. It should be more visible, accessible and user-friendly.

Page 31: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

30

• To increase traffic to the least visited areas of the site (ZOOMtech, ZOOMdo,CaféZOOM, ZOOMa cum laude), clarify their titles and develop more engagingcontent.

• Eliminate silhouette “shadow people”; replace with real photos of cast members orother kids.

• Provide more opportunities for kids to personalize their ZOOM experience bysubmitting their own drawings, games, jokes and web site links.

Community

The following is a list of suggestions for building an online community within the ZOOM

Web site.

• Offer the opportunity to correspond with the ZOOM TV show cast members bye-mail.

• Offer area for chat experience with other visitors and cast members, creating more ofan online ZOOM community.

• Invite kids to link their own web pages to the ZOOM community.• Offer an area for kids to draw and send in their own artwork to share with other

“ZOOMers.”

Page 32: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

31

About the Researchers

Shannon Beltz is an Ed.M. graduate student in Harvard’s Technology in Education (TIE)

program. Her background includes research and project development with the Children’s

Television Workshop, Fox Kids Network, and the Learning Channel. Recently, Shannon’s

instructional literacy Web site was unveiled at a South Boston High School, connecting English

students across the country via the Internet.

Erik Blankinship is an Ed.M. graduate candidate who will continue his Ph.D. studies next

year at MIT’s Media Lab. He is an animator, working on a film with funding from the Jim

Henson foundation. Currently, he is a Computer Clubhouse volunteer at Boston’s Computer

Museum. Erik is also an Eagle Scout, and recipient of the first ever cinematography merit

badge.

Rachel Garb is an Ed.M. student at Harvard’s Graduate School of Education, focusing her

studies on the design of educational technologies. Her experience includes six years in Silicon

Valley, producing Web sites, multimedia CD-ROMs, and documentation for a variety of

companies, including America Online, IBM, and Netscape Communications.

Marnie Oakes is also an Ed.M. student, creating partnerships between school and

business communities, and information technology School-to-Career pathways. Her

degree work is linked between Harvard’s Graduate School of Education, MIT’s

Technology & Policy Program, and the Kennedy School of Government. Marnie’s past

experience is in Marketing and Human Resources, and building volunteer coalitions in the

Washington, DC area.

Contact Information

Shannon Beltz [email protected] 617/497-7118

Erik Blankinship [email protected] 781/758-0325

Page 33: Formative Evaluation Report ZOOM on the Webalumni.media.mit.edu/~erikb/papers/zoom.pdf2 Executive Summary To assist the developers of the new ZOOM Web site for kids aged 8-12, our

32

Rachel Garb [email protected] 617/868-3203

Marnie Oakes [email protected] 617/868-2801

c/o Dr. Art Johnson [email protected] 617/495-3541