Web-based Sound and Music Games with Activitiesfor STEM Education

Travis M. Doll, Raymond Migneco, and Youngmoo E. KimMusic and Entertainment Technology Lab

Department of Electrical and Computer EngineeringDrexel University, Philadelphia, PA, U.S.A.

{tdoll, rmigneco, ykim}@drexel.edu

Abstract—It is widely believed that computer games have greatpotential as a compelling and effective tool for science, technology,engineering, and mathematics (STEM) education, particularlyfor younger students in grades K-12 (ages 5-18). In previouswork we created several prototype games that allow students touse sound and acoustics simulations to explore general scientificconcepts. These games were developed using Adobe Flash andare accessible through a standard web browser, making themparticularly well-suited for education environments where customsoftware may be difficult to install. Pilot testing our games withlocal K-12 classrooms has demonstrated potential benefits interms of captivating student interest, but has also revealed mul-tiple avenues for improvement and refinement. We have recentlydeveloped a unified development platform that overcomes priorarchitectural limitations to enable dynamic sound processing andsynthesis within Flash applications to improve responsiveness andcreate rich interactive experiences. Since they are constructed inActionScript, the relatively straightforward scripting languageof Flash, these games can be easily extended or adapted totarget highly specific lessons and concepts. In this paper, wedetail several specific lesson plans for mathematics and scienceeducation developed using customized versions of our previousgames.

I. INTRODUCTION

Video games are integrated into the daily lives of stu-dents, particularly children in grades K-12. For some time,researchers have discussed the potential benefits of designingspecific games for education. For science, technology, en-gineering, and mathematics (STEM) education in particular,video game platforms provide unique possibilities for simu-lating, exploring, visualizing, and experimenting with difficulttheoretical concepts. To this end, we have previously createdseveral web games that allow students to use sound andmusic simulations to explore general concepts in acoustics andmathematics [1]. Based on this prior work, we have createda general framework for the rapid development of media-rich interactive games in Flash. Since these games are writtenusing an approachable scripting language, ActionScript, theycan be readily extended or adapted for the teaching of highlyspecific curricular goals, for example, the relationship betweendistance, time, and velocity.

Our previously developed games investigate the perceptionof sound through the manipulation of speech and music.The first, titled Hide & Speak, explores the “cocktail partyproblem” [2], i.e., our ability to focus on and understandone voice in a crowded room (a problem that still eludes

computational solutions). The second game, Tone Benderinvestigates musical instrument timbre (the sound qualities thatdifferentiate instrument sounds). Based on the frameworks ofthese games, we have since developed extensions of their coreactivities that utilize interactive exploration and feedback toteach students specific mathematical and scientific lessons.These web-based lessons are designed as interfaces withminimal complexity so they may be easily used by studentswithout prior instruction.

To maintain widespread accessibility, our lessons requireonly internet access through a web browser and run inde-pendently of external applications. This paper will discussthe specific learning objectives and procedures of severalexample activities as well as their implementation in theclassroom. We will demonstrate the potential of this platformto enhance student learning and to offer fine-grained data onstudent progress and assessment. The remainder of the paperis structured as follows: In Section II, we present relatedprior work on similar types of educational games. SectionIII briefly describes the development platform used for ourexisting Flash games. In Sections IV and V, we present Hide& Speak, a voice identification game, and lesson plans derivedfrom modifications to the original game. Sections VI andVII describe Tone Bender, a musical instrument game, andits associated lesson plans. Finally, we present discussion inSection VIII and conclusions and future work in Section IX.

II. BACKGROUND

Since the introduction of personal computers, educatorshave sought ways of integrating digital games into K-12 cur-ricula to augment traditional teaching methods. Some studiesincorporating video games have reported improved spellingand reading skills [3] and increased student motivation [4]at elementary grade levels (K-2). Effective integration ofcomputer games at all levels of education, however, has provento be difficult because of the inherent challenges in designinga learning activity that is fun and engaging for an increasinglydemanding audience, and some so-called “edutainment” soft-ware is neither entertaining nor educational [5]. But recentwork by Shaffer and colleagues suggests that video gamesand computer simulations offer an ideal medium for introduc-ing new collections of skills and knowledge (the theory ofepistemic frames), valuing innovation over rote learning, into

education curricula [6]. Similarly, Dede advocates a “learning-through-doing” approach using experiential simulations inwhich students learn by interacting with a virtual environment.Furthermore, he believes that advances in technology havemade the development of such simulations a reality, to thepoint where they will soon be available on demand over theInternet [7].

There are three common approaches in which computergames are integrated with educational curricula [8]. The firstengages the students to design and build their own gamesunder the premise that they will learn desired concepts as theydevelop the game. But few K-12 teachers have the backgroundand experience to teach programming and game development.The second approach uses games designed by professionaldesigners, who can incorporate the sophisticated graphics andcontrols found in modern games to engage players. Thesegames require significant development time, and commercialgame studios are hesitant to embark on such projects dueto the limited success of previous educational games (andthere is no guarantee that the resulting games will be fun forstudents). The third approach integrates commercial-off-the-shelf (COTS) games that facilitate selected learning objectivesinto the classroom [9]. These games are readily available,designed for entertainment, and have credibility with studentsoutside of the school environment, but are limited in what theycan teach and may be expensive to obtain for a large numberof students.

Our web-based collaborative games offer a hybrid approachthat addresses many of these issues. The activities incorporatean inherent design phase, which allows students to participatein the development of portions of the game played by otherstudents. By working in partnership with schools and teachersthrough existing outreach programs, such as the NSF GK-12 program [10], our games are collaboratively designed andtested by teachers and graduate students. This facilitates rapidprototyping using the university’s laboratory resources, pilottesting through partner schools, and eventual integration intoexisting K-12 curricula.

III. DEVELOPMENT PLATFORM

Our goal is to develop highly interactive educational gamesand lessons, and from early on we focused on web-basedplatforms because of their potential to reach the broadest pos-sible audience. Additionally, an application that runs entirelywithin a web browser overcomes the common impediments ofinstalling custom software at educational institutions. Theseconstraints led us to adopt the Adobe Flash platform, thedominant platform for web-based game development. Flashallows programmers to author games on a cross-platformarchitecture and provides tools for easily implementing richgraphics, animation and user interface controls.

Although Flash provides a straightforward avenue for de-ploying media-rich applications on the web, its support forsound and music has been limited only to playback of pre-recorded clips. The sound and music games we envisioned,however, required real-time dynamic audio capabilities that

were unavailable at the time (Flash version 9). We initiallydeveloped a hybrid Flash-Java implementation [1], which pro-vided a mechanism for overcoming some of the limitations inFlash at the expense of interface fluidity. The release of Flashversion 10 in October, 2008 [11] and the public preview of theAdobe Alchemy project in December, 2008 [12], provided apotential solution for these problems, and we have since devel-oped a new architecture that enables uninterrupted, interactiveaudio processing directly within the lesson environment, whichgreatly improves the user experience. This new architecture isdetailed in [13] and [14] and briefly summarized below.

Flash ApplicationActionScript

GUI

Audio

ALF

Convolution

FFT

Features

Audio

Parameters

Fig. 1. Flash and Alchemy architecture.

Unlike previous versions, Flash 10 makes it possible todynamically generate and output audio within the Flash frame-work. This functionality is asynchronous, allowing sound toplay without blocking the main application thread. The AdobeAlchemy project provides the ability for C/C++ code to be di-rectly compiled for the ActionScript Virtual Machine (AVM2),greatly increasing performance for computationally intensiveprocesses. We have tested the performance of the Alchemycompiled C code with results that are significantly fasterthan other implementations [13]. With these tools, it is nowpossible to develop Flash-based applications that incorporatedynamic audio generation and playback capabilities withoutthe need for an external interface for computation-intensivesignal processing applications.

The Alchemy framework enables a relatively straightfor-ward implementation of standard C code into Flash projects,thus existing signal processing libraries written in C can beincorporated as well. C code is compiled by the Alchemy-supplied GNU Compiler Collection resulting in an SWCfile, an archive containing a library of C functions, whichis accessible in Flash via ActionScript function calls. Anintegrated application is created by simply including thisarchive, which we call the Audio processing Library for Flash(ALF), within the Flash project, producing a standard SWF(Flash executable) file when built. All of the games and lessonspresented here were developed using this architecture.

IV. HIDE & SPEAK

Hide & Speak employs a simulated acoustic room environ-ment that is designed to demonstrate the well-known “cocktailparty” phenomenon, which is our ability to isolate a voice ofinterest from other sounds, essentially filtering out backgroundsounds from an audio mixture [15]. The interface allows aplayer to explore the effects that source location and acousticenvironments have on the sound quality of a voice and theintelligibility of speech.

This game consists of two complementary componentsin which players create room simulations through guidedexploration and then listen to configurations created by otherplayers. In the creation activity, called Hide the Spy, theplayer begins with a single target voice with the objectiveof modifying the number and location of interfering voicesuntil the target is barely comprehensible. The player may alsoalter the amount of reverberation and adjust the position ofthe listener. As the player is designing the room, the difficultyof the room space is determined by the signal-to-interferersplus noise ratio (SINR), where the target voice is the signaland the other sources are the interferers. The room’s difficultydetermines the potential point value for the creator, whichis awarded only when another player is able to successfullyidentify the target voice within the room in the listening phase.

The listening activity, Find the Spy, is a straightforwardquiz where a player is asked, given a isolated sample of thetarget’s voice, to determine whether the target is present withina mixture of voices. The audio mixtures are randomly chosenfrom configurations created in the Hide the Spy component.The listening player is awarded points for a correct responsebased on the room difficulty, and a fraction of the possiblepoints is deducted for an incorrect response.

The game interface is designed to be highly interactive sothat players can easily explore the effects of modifying soundsource locations, listener location and reverberation intensitiesin real-time. The graphical layout of the game represents anacoustic space to provide a visual and auditory correlationbetween the room configuration and the resulting sound.

Fig. 2. Hide the Spy interface of Hide & Speak for creating rooms.

Fig. 3. Find the Spy interface of Hide & Speak for listening to rooms.

Fig. 4. Optional view of room configuration in Find the Spy.

V. LESSONS BASED ON HIDE & SPEAK FRAMEWORK

Hide & Speak provides a framework for the rapid de-velopment of lessons targeting the physics of sound andproperties of acoustic waves for K-12 education. Due to itsorigins as a collaborative game, network communication isalready established, which enables all data to be saved forteachers and instructors to view students’ progress and analyzeperformance. Next, we detail four specific lesson plans wehave developed using the Hide & Speak framework.

A. Wave Propagation

1) Overview: This lesson explores how waves propagatethrough a medium (air) and the time required for wavesto travel from one point to another. As an example ofwave propagation, the lesson interface provides an interactivedemonstration of sound waves radiating from a point sourceand eventually reaching an observer as the waves spreadthroughout the room. The demonstration is interactive, andthe user can modify the positions of the source and theobserver and alter room dimensions to experience a varietyof auditory conditions. After sufficient exploration of thewave propagation phenomenon, the student performs distancemeasurements and calculations to determine the amount of

Fig. 5. Interface for the wave propagation lesson.

time taken for a wave to travel between two points based onthe speed of propagation of the medium.

2) Educational Objectives: At the conclusion of this activ-ity, the student should be able to perform the following tasks:

• Calculate the time required for a wave to travel from aone point to another with a specified propagation speed.

• Calculate the speed of a propagating wave given the traveltime and the user measured distance between two points.

• Demonstrate an understanding of the relationship be-tween distance, time, and speed.

• Measure distances using a virtual tape measure andperform the necessary unit conversions for calculations.

• Provide a qualitative explanation of how waves travelfrom one location to another and describe the typicalcircular (spherical) propagation pattern.

3) Procedure: At the beginning of the lesson, the student ispresented with an an animated introduction on wave propaga-tion in the context of speech and sound that includes relevantterminology and visual demonstrations of the discussed con-cepts. Next, the student enters the guided exploration phase ofthe room environment simulator, where they can investigatewave propagation more thoroughly. This interface providesthe user with the ability to adjust the positions of the wavesource, destination, and room dimensions, yielding potentiallyunlimited configurations. The speed of the animated wavepropagation can also be varied to better visualize the under-lying phenomenon.

In the next component, the student is asked to determinethe relationship between the time required for a sound waveto travel from a sound source to an observer and the distancebetween the two, essentially deriving the speed of sound. Theuser begins this task by first measuring the travel time from thesource to the listener and then measuring the distance betweenthe sound source and the observer using a virtual tape measure.After three simulations, the student is asked to compare theresults and to compute the speed of sound in air.

As a follow-up task, the student must predict the wavetravel time based on the speed of sound. Therefore, they mustmeasure the distance traveled, and solve for the propagation

time. A series of randomly generated rooms are presentedto provide a variety of parameters to the student beforethis section is complete to obtain sufficient practice of theseconcepts. Some measurements will be presented in differentunits, requiring a conversion in order to match the originalunits in the presented equations. During this process, studentswill also compute the speed of sound in different units.

At the end of the activity, the student is asked several com-prehension questions that involve synthesis of the knowledge,which are described below.

4) Assessment: The wave propagation lesson incorporateswithin-lesson and post-lesson assessments in the form of mul-tiple choice questions, rate calculations, and unit conversions.The student must correctly apply the concepts developed inthe introduction section and perform distance measurementsand unit conversions in order to successfully complete alltasks. Several random variations of sound source and observerpositions and room sizes are provided for repetition to ensurethe student has sufficiently learned the calculation process.

At the conclusion of the activity, the student is asked todemonstrate synthesis of their knowledge by designing con-figurations that match a particular specification (propagationtime between two locations). After a few variations of this, thestudent will be asked to design multiple configurations to meetthe same specification (equivalent distances from the source).This leads to a final assessment of the student’s knowledge ofwave propagation being circular, or spherical, which they areasked to demonstrate by drawing freehand using the interface.These drawings are stored in the game database, where theycan be evaluated by a teacher.

B. Reflection Angles of Waves

Fig. 6. Interface for the reflection angles of waves lesson.

1) Overview: When sound waves propagate throughout aroom, the intended path of a wave is interrupted by surfaces,modifying its direction. In this lesson, students investigatein the phenomenon of wave reflection to establish the directrelationship between the angle at which the wave approachesand reflects from a surface (i.e., the equivalence of the anglesof incidence and reflection). This concept is illustrated, using

sound waves as an example, through an interactive acousticroom environment that allows the user to visualize the pathsof sound waves as they reflect off of walls. The studenthas the ability to directly manipulate the room environmentby changing the sound source and listener locations and thedimensions of the room.

2) Educational Objectives: At the conclusion of this lesson,we expect the student to be able to perform the following:

• Measure angles in degrees using a virtual protractor (i.e.,angles of incidence and reflection).

• Calculate the angle of reflection of an incoming wave.• Trace the path of a wave as it propagates throughout the

room, reflecting from surfaces.• Calculate the time taken for a reflected wave to reach a

destination.3) Procedure: At the beginning of the lesson, the student is

presented with an an animated introduction on the reflection ofwaves in the context of speech and sound, which includes rel-evant terminology and visual demonstrations of the discussedconcepts. Then, the user proceeds to a guided investigation ofwave reflections using the simulated room environment. Theinterface offers the ability to modify the positions of the soundsource, listener, and overall room dimensions, as well as thespeed of the animation. After the user has sufficiently exploredthe concepts in the simulator, they proceed to the next sectionto apply the general knowledge gained from this section tospecific situations.

In next component of the lesson, the student is presentedwith a reflecting wave that creates several angles with miss-ing labels that the student must determine. Several differentrandomly generated scenarios of reflecting sound waves arepresented to give the user practice with measuring angles.Variations of this setup require the user to determine unknownangles in the figures with only some of angles given, forexample:

• The angle of incidence is given, but the angle of reflectionmust be determined.

• Only the angle consisting of the incidence path andreflecting surface is shown and all others are not.

• No angles are provided, and the student must measurethe angles with a digital protractor.

The student is also asked to label angles and paths with theproper terminology learned in the introduction when applica-ble.

Next, the user is provided with a situation where a soundwave must be reflected from the wall and with the goal ofreaching the observer. A virtual protractor is provided formaking measurements needed to draw the incidence path fromthe sound source to the wall so that the reflecting path reachesthe observer. Precision is important in this component becausethe user receives an accuracy rating based on how close thesound wave is to the observer after one reflection from thewall. This can be extended to include two or three reflectionsfrom the walls to achieve a higher level of difficulty. Uponsuccessfully drawing the sound path the user must finish the

exercise by calculating all angles constructed by the soundwave.

4) Assessment: As the student proceeds through the lesson,their comprehension of the material is periodically tested.The assessment includes different types of exercises: multiplechoice questions, labeling of key components, angle calcu-lations and having the user draw reflecting sound waves tomeet certain criteria. Knowledge of terminology is evaluatedby having the student label components of the room environ-ment when handling calculations of angles. The combinationquestions that ask the user to measure angles with the virtualprotractor and calculate angles mathematically test both theirknowledge of the terminology and their ability to compute therequired angles. In the final stage, the user is required to reflecta wave from the wall, which synthesizes all of the presentedmaterial for the lesson.

C. Wave Energy Conservation

Fig. 7. Interface for the wave energy conservation lesson.

1) Overview: As sound waves propagate throughout aroom, they encounter surfaces that absorb some of the energyof the wave as they are reflected. In this lesson, studentsexplore the underlying physics behind this phenomenon, i.e.,when a wave reflects off of a boundary, some of the wave’senergy may be transferred to the reflecting surface, reducingthe remaining energy of the wave. To illustrate this concept,the student investigates energy absorption by the reflectingsurfaces in a simulated acoustic room environment.

The interface for this lesson allows the student to listento the audio of a wave and provides a visual representationof the sound wave losing energy as it travels throughout theroom. This is accomplished by decreasing the color intensityfor the path of the sound wave as it travels through the room,which are accompanied with changes in audio intensity (asmeasured at different locations). The combination of the visualand auditory cues provides an illustration of the effects ofwave reflection at boundaries. The interactive interface allowsstudents to directly alter the environment by changing thesource location, direction of the wave path, and the materialsof the room wall surfaces.

2) Educational Objectives: After completing this lesson,the student should be able perform the following tasks:

• Explain why waves do not continue on forever.• Describe how sound energy is transferred to the reflecting

surface.• Create an ordered list of room materials in terms of

energy absorbing ability.• Design a room that encourages or discourages continuing

sound wave reflection.3) Procedure: At the beginning of the lesson, the student

is presented with an an animated introduction on energyconservation in the context of speech and sound, including rel-evant terminology and visual demonstrations of the discussedconcepts. Afterwards, the user proceeds to a guided explo-ration of energy conservation in waves using an acoustic roomenvironment simulator. The interface provides a means formodifying the wave source position, energy absorbing qualitiesof the walls (material composition), and the dimensions of theroom. When the user has gained adequate knowledge regardingthe sound absorption strengths of the wall materials, severalassessment questions are presented, such as ordering thematerials in terms of their ability to attenuate the propagationof sound waves at the room boundaries.

In the next section of the lesson, several examples ofchanges in sound wave intensity are presented and a studentchooses from a set of three materials to decide which onemight be the material of the room walls. The energy absorbingcharacteristics of the materials will not be similar enough toraise confusion, but the task will still require knowledge fromthe previous section.

Building upon previous sections, the final component re-quires the student to design a room with under a fixed budgetto meet specific criteria, such as:

• A sound booth for recording music.• A conference room for small group meetings.• A concert hall with significant echo.

A short description of each room type along with an audioexample is provided in the event that the student is not awareof the purpose of these rooms.

4) Assessment: Throughout this lesson on energy conser-vation in waves, the student’s comprehension is assessed usingmultiple choice questions, ordering exercises, and room build-ing simulations. All of these methods encompass the materialpresented in each section while incorporating information builtupon previous sections. The exploration phase provides thestudent with an opportunity to investigate the properties ofeach room material before being required to order the materialsrelative to their energy absorbing properties. Lastly, the studentis required to construct a room meeting a specified room style,which requires the application of all of the concepts fromprevious sections.

D. Wave Triangulation

1) Overview: Detecting the originating direction of a soundis a task that humans perform instinctively and quite accu-

Fig. 8. Interface for the wave triangulation lesson.

rately. In this lesson, students explore the physics and mathe-matics behind triangulation along with specific demonstrationsusing sounds. The process of determining the angle of arrivalof a wave requires information from two receivers, morespecifically the time of arrival of the sound at each receiver.The difference in arrival times between the two receivers isthe key component in determining the line of sight (from themid-point between the receivers) to the location from whichthe wave emanated. The lesson interface allows for control ofsound source and observer positioning with real-time interac-tive audio feedback. A representation of the arriving wavesis illustrated in the diagram, visually synchronized with theaudio to establish a direct connection between the two. Inthis lesson, students are exposed to the mathematics necessaryto determine the angle of arrival and the estimated distancebetween the source and observer.

2) Educational Objectives: After completing this lesson,the student should be able perform the following tasks:

• Determine the direction (angle) of an incoming wavefrom two measurements.

• Understand the meaning of the time axis in a 2-dimensional graph.

• Calculate the speed of an incoming wave.3) Procedure: At the beginning of the lesson, the student is

presented with an an animated introduction on determining thedirection of a sound, including relevant terminology and audio-visual demonstrations of the discussed concepts. The studentnext enters a guided exploration of sound triangulation usinga simulated room environment where the user can experiencethe effects of moving the wave source and listener about theroom. The traveling waves are simulated by drawing the pathsof the waves as they leave the source traveling towards thereceivers (ears). A timer will be provided for each ear to trackthe amount of time taken to travel from the source to receiver.

In the next section, the student completes several small tasksto develop additional knowledge regarding sound directivity.In the first task, the student estimates the general direction ofarrival of the wave given either time graphs or travel timesfor the waves by selecting the quadrant from which the sound

emanated. After selecting the quadrant the true source positionwill be revealed and the user must determine the angle for theline of sight. Similar to the previous section, the line of sightis the mid-point between the receivers, in this case the centerof the head. The student then measures the angle of the lineof sight and calculates several other angles created by wavepaths in the diagram.

Then in the second task, the user will be provided with thesound source and observer locations along with the simulatedaudio. The user must choose between a set of predefined timegraphs for each ear to represent the current source and observerconfiguration. The predefined set of time graphs would onlyinclude audio signals that start near the beginning, middle andend of the time axis to keep the task reasonable. Similarly inthe third task, the user is provided with the source and observerpositions, but must create a representation of the configurationjust using the starting the onset times of the signals. Thismeans the set of time graphs would now be numerical valuespertaining to the arrival time of the audio signals.

4) Assessment: Throughout this lesson on sound triangu-lation the student’s level of comprehension is assessed, firstusing a few multiple choice questions, but mostly by havingthe user interact with the room environment to completetasks based on lesson content. In the first guided exploration,the user develops knowledge of the components involved intriangulating the location of a sound source, and the secondinterface consists of tasks that require the user to applytriangulation in specific scenarios, such as determining theapproaching angle of a wave based on the time of arrivalof the wave at each receiver using both time graphs andmeasurements of elapsed time.

VI. TONE BENDER

Tone Bender explores the perception of a musical instru-ment’s characteristic qualities, known as timbre, though usermodifications of the sound. It has been shown that a significantfactor in the perception of timbre is the relative distribution ofthe instrument’s sound energy over time and frequency [16],[17]. Tone Bender consists of two separate components inwhich players generate their own musical instrument soundsthrough interactive exploration and then listen to instrumentsounds created by others with the goal of identifying theinstrument correctly.

In the creation interface, the player’s objective is to makewidespread acoustic modifications to a given instrument whilestill maintaining its identifiability. The player is able to in-dependently modify the time and frequency components ofthe instrument audio, which are displayed visually in separatewindows: the instrument’s loudness over time is given in onewindow and the relative energy of the instrument’s overtonesare shown in another window. A score is provided to the playerbased on the signal-to-noise ratio (SNR) between the originalinstrument and their modified version such that an instrumentwith a lower SNR values is potentially worth more points if themodified instrument is correctly identified by another player.

The listening interface is an instrument identification quiz,which provides the means for evaluating the modified instru-ment sounds from the first game component. During the lis-tening process, the user is also allowed to view the amplitudeand frequency characteristics of the instrument to assist withidentifying the current instrument. It should be noted thatresponses to both the instrument and the instrument familyare sought. Points are awarded to both the listener and creatorof the modified instrument sound upon correct classification.

Fig. 9. The creation interface for Tone Bender.

Fig. 10. The listening interface for Tone Listener.

VII. LESSONS BASED ON TONE BENDER FRAMEWORK

As with Hide & Speak, Tone Bender provides a frameworkfor the rapid development of lessons in mathematics andacoustics, though clearly targeting a different set of con-cepts. Tone Bender is particularly well-suited to explainingtrigonometric functions and parameters (e.g., amplitude andfrequency). Two lesson plans developed using this gameplatform are described below.

A. Effects of Amplitude Over Time

1) Overview: This lesson demonstrates the connection be-tween the perceived loudness of a sound and its graphical rep-resentation by drawing on graph reading skills and basic rate-change knowledge in mathematics. In particular, the referenceaudio signals used in this activity will be drawn from musicalinstrument sounds, since different instruments possess ampli-tude characteristics that vary with time in numerous ways.

Fig. 11. The interface for the effects of amplitude over time lesson.

Examples of such amplitude characteristics include the rapidlyrising and slow decaying nature of plucked string instrumentsand the steady, sustained nature of wind instruments. Theseexamples correlate well with their graphical representations ina two-dimensional space, since the points of rapid increaseand/or decrease can be clearly identified.

2) Education Objectives: Upon completion of this lessons,the student should be able to complete the following tasks:

• Demonstrate an understanding of rate change for lineswith different types of slope (no slope, positive andnegative)

• Determine inflection points in the data plot of a 2-D graphgiven the variables of the x and y axes

• Produce a graph of amplitude versus time representingthe variation in the loudness of a given the description ofan audio signal

3) Procedure: The lesson begins with an introductiondemonstrating relevant terminology pertaining to amplitudeof signals and loudness of musical instruments. The useris provided with several plots indicating how the loudnessvaries over time for different musical instrument families (i.e.plucked strings, wind excited instruments and bowed strings).Each plot is accompanied by audio feedback to providean auditory correlation with the visual representation of theinstrument’s loudness. As the audio for a particular plot isplayed, a vertical scrub tool is displayed. The use of the scrubtool accentuates the relationship between increasing slope inthe loudness curve and increasing perceived loudness. Thepurpose of this exercise is to provide a basic understandingof how loudness can be visualized as a plot of a dependent(amplitude) and independent (time) variable.

After completing the first lesson, the student is tested withthe a perception situation, which requires selecting an audiosignal that best represents a given plot of amplitude versustime. The reference audio signals are distinctly different innature such that the distinction is clear, but requires carefulobservation by the student.

Following this exercise, the student is required to demon-strate anticipatory knowledge on the time-varying nature loud-ness. The student is provided with the description of an audiosignal in words, and is required to draw out its loudness

characteristics. The student’s response is compared with thetrue signal and an accuracy rating is returned. An exampleis shown in Figure 11 where the reference loudness curve ofan instrument is shown in green and the student’s estimate isin blue. This accuracy rating can be analyzed to determine ifthe student grasps the concept of line slope if their amplitudeenvelope follows a general, expected contour for the givenmusical instrument.

4) Assessment: The first part of this activity consists ofa tutorial to provide the student with basic knowledge oninstrument loudness and to enforce concepts of graph reading.This is assessed by determining if a student can reproducethe perceived loudness on an instrument by replicating thesignificant points of change in that instrument’s loudness dis-tribution. Likewise, the student’s inferential skills are evaluatedwhen they are required to match an actual audio signal to aprovided loudness curve. By comparing the results of thesetwo activities, the students’ understanding of time-varyingamplitude can be assessed by checking for consistency in theirresponses.

B. Effects of Overtones in Sounds

Fig. 12. The interface for the effects of overtones in sounds lesson.

1) Overview: In this lesson, students will investigate howsounds can be decomposed into a set of weighted frequency(spectral) components. More specifically, we describe thegeneral contour of these weights as a “spectral envelope”, in-dicating the distribution of a sound’s energy across frequency,which is related to the perceived timbre of the sound. Tofurther illustrate this concept, sounds from musical instrumentsare examined by manipulating the strengths of the differentfrequency components to effect changes in timbre.

2) Educational Objectives: Upon completion of this lesson,the student should be able to perform the following tasks:

• Correlate audio samples with mathematical graphs of thesound’s spectral content

• Analyze a 2-D graph given the variables of the x and yaxes

• Explain that musical sounds can be decomposed intoindividual tones, each with a specific frequency

• Illustrate how the distribution of energy over frequencyaffects the timbre of a musical sound

3) Procedure: The lesson begins with an introduction tothe frequency components of a sound and a description ofthe spectral envelope, including definitions of the relevantterminology and short audio demonstrations of the underlyingconcepts. Following the introduction, the user is presentedwith a guided exploration of changing timbre through themanipulation of the spectral envelope. This includes such tasksas:

• Creating a frequency distribution that contains only stronglow frequencies

• Picking between three spectral envelope graphs that bestrepresent an audio example

• Drawing the spectral envelope based on an audio exampleAfter completing all tasks in the guided exploration, the

student is presented with a different interface asking themto draw the spectral envelope of a reference sound. Severaliterations of this type of problem are provided to ensureunderstanding of the material.

4) Assessment: At the end of the introduction, the ma-terial is tested using multiple choice questions that verifythe user’s comprehension of the basic lesson content. In theguided exploration, we assess the students’ understanding ofovertones and distribution of energy in frequency, by askingthe users to match the audio of a reference musical with oneof three spectral envelopes that they can choose from. Thesespectral envelopes differ significantly, including sounds withits’ energy concentrated in the lower frequency regions, onewith its’ energy distributed evenly across all frequencies andan envelope with the energy focused at higher frequencies.Based on the students’ selections, we can determine if theycan correlate a visual representation of the frequency envelopewith the timbre of the sound they are hearing. This providesverification that the students’ grasp the main overall conceptof the lesson.

VIII. DISCUSSION / CONCLUSION

The development of the collaborative games that providea framework for the lessons presented have gone throughcountless iterations that consisted of more informative userperformance feedback, complete re-designing of the interfaces,game flow changes and overhauling the audio processing ca-pabilities. Since previous versions of the games, the interfaceshave become more appealing, user-friendly and intuitive withadequate gameplay feedback to eliminate the most commonuncertainties.

Numerous testing iterations of the collaborative games havebeen performed through the student authors who are part ofNSF Graduate Fellowships, including NSF GK-12 and NSFDiscovery K-12 programs. The participating school includingMartha Washington Elementary and Girard Academic MusicProgram (GAMP) as part of NSF GK-12 Program, Creativeand Performing Arts High School (CAPA) as part of DK-12, and various other outreach programs. As a result of theseevaluations, the qualitative results will aid in further devel-opment of current and future lessons to increase enjoyment,educational content and intuitiveness of the lessons.

As a result of observing the students engaged in the collab-orative games, we noticed that the students were encounteringa disconnect between modifications they were making to theinterface and the audio presented. Therefore, we enhancedthe games to include real-time dynamic audio that providesinstantaneous feedback with the audio as adjustments are madeto the parameters in the game. For example in Hide & Speak,moving a sound source around the room modifies the audioplayback as the user is changing the position of the source.Previously, updates only occurred when the audio looped backthrough the audio, repeating the previous audio with new audiomodifications. This has created a better learning environmentand helped to remove any disconnect between the interfaceactions and audio playback. The real-time dynamic audiocapability has been included in the presented lessons to enablea fully interactive environment.

With the many iterations developing the web-based game,we feel that we have a suitable framework for rapidly develop-ing lessons about sound and acoustics relating the mathematicsand science concepts in the K-12 curriculum.

IX. FUTURE WORK

The presented lessons are part of a pilot study where theHide & Speak and Tone Bender frameworks are extended toeducate students on mathematics and science related topics.The lessons are currently being tested at local K-12 schoolsby the student authors through the previously mentionedprograms. Performing the lessons in the classroom will providequalitative feedback that will help to improve the quality andintuitiveness of the lessons, as occurred with the initial games.As the lessons are performed, analysis of data collected fromstudent responses in each lesson will provide information onhow well the lessons educate the students at the various topics.

As the development of the lessons progresses, the packageof lessons will be expanded to include more topics in the K-12curriculum and more customizations added to existing lessonsbased on student feedback. An essential customization that willbe added for the teachers and instructors to monitor studentperformance is access to student responses and completionlevels for each of the lessons.

ACKNOWLEDGMENT

This work is supported by NSF grants DGE-0538476 andDRL-0733284.

REFERENCES

[1] Y. E. Kim, T. M. Doll, and R. V. Migneco, “Collaborative onlineactivities for acoustics education and psychoacoustic data collection,”IEEE transactions on learning technologies, 2009, preprint.

[2] E. C. Cherry, “Some experiments on the recognition of speech, with oneand with two ears,” The Journal of the Acoustical Society of America,vol. 25, no. 5, pp. 975–979, 1953.

[3] F. S. Din and J. Caleo, “Playing computer games versus better learning,”in Paper presented at the 70th Annual Conference of the Eastern Edu-cational Research Association, Clearwater, FL, February 16-19 2000.

[4] R. Rosas, M. Nussbaum, M. Cumsille, V. Marianov, M. Correa, P. Flores,V. Grau, F. Lagos, X. Lopez, V. Lopex, P. Rodriguez, and M. Salinas,“Beyond nintendo: Design and assessment of educational video gamesfor first and second grade students,” Computers & Education, vol. 40,pp. 71–94, 2003.

[5] S. Papert, “Does easy do it? children, games, and learning,” GameDeveloper, p. 88, June 1998.

[6] D. W. Shaffer, How Computer Games Help Children Learn. PalgraveMacMillan, New York, NY, 2006.

[7] C. Dede, “Emerging technologies and distributed learning,” AmericanJournal of Distance Education, vol. 10, no. 2, pp. 4–36, 1996.

[8] R. Van Eck, “Digital game-based learning: It’s not just the digital nativeswho are restless,” EDUCAUSE Review, vol. 41, no. 2, pp. 16–30, 2006.

[9] K. Squire, “Video games in education,” International Journal of Intel-ligent Simulations and Gaming, vol. 2, no. 1, 2003.

[10] National Science Foundation (NSF), “Graduate TeachingFellows in K-12 Education (GK-12).” [Online]. Available:http://www.nsf.gov/funding/pgm summ.jsp?pims id=5472& org=DGE

[11] Adobe, “Flash player 10.” [Online]. Available: http://labs.adobe.com/technologies/flashplayer10/

[12] Adobe Labs, “Alchemy.” [Online]. Available: http://labs.adobe.com/technologies/alchemy/

[13] T. M. Doll, R. Migneco, J. J. Scott, and Y. E. Kim, “An audio DSPtoolkit for rapid application development in flash,” Submitted to IEEEInternational Workshop on Multimedia Signal Processing, 2009.

[14] R. Migneco, T. M. Doll, J. J. Scott, C. Hahn, P. J. Diefenbach, and Y. E.Kim, “An audio processing library for game development in Flash,”Submitted to the International IEEE Consumer Electronics Society’sGames Innovations Conference, 2009.

[15] S. Haykin and Z. Chen, “The Cocktail Party Problem,” Neural Compu-tation, vol. 17, no. 9, pp. 1875–1902, Sep. 2005.

[16] H. Helmholtz and A. J. Ellis (trans), Die Lehre von den Tonempfindungenals physiologische Grundlage fur die Theorie der Musik (On theSensations of Tone as a Physiological Basis for the Theory of Music),2nd (English) ed. London: Longmans, Green, 1885.

[17] M. Clark, D. Luce, R. Abrams, H. Schlossberg, and J. Rome, “Prelimi-nary experiments on the aural significance of parts of tones of orchestralinstruments and on choral tones.” Journal of the Audio EngineeringSociety, vol. 11, no. 1, pp. 45–54, 1963.