The Design of a Web-Based Multimedia Sport Instructional System

Kosuke Takano Department of Computer and Information Sciences

Kanagawa Institute of Technology Atsugi, Kanagawa, Japan

takano@ic.kanagawa-it.ac.jp

Kin Fun Li, Mark Graham Johnson Department of Electrical and Computer Engineering

University of Victoria Victoria, BC, Canada

kinli@uvic.ca, mgrahamj@uvic.ca

Abstract—With the advances in computer technology, many systems have been developed for educational and instructional purposes. In particular, a computer-based system is very attractive in sports instruction and training as compared to the traditional human coaching approach, saving time, space, and cost. Among several important technical issues to be resolved in such sports instructional systems, motion or gesture capture and the organization of the database are the major ones. Of course, the most successful implementations are the systems from the gaming industry. In this work, we briefly discuss these commercial technologies and systems. A prototype system for learning tennis is introduced. Various techniques for motion capture are described and discussed. An event-based motion detecting and matching algorithm for capturing tennis swings from people of different size is presented. Finally, a framework for a Web-based generic sport system is presented.

Keywords-gaming; multimedia instructional systems; gesture capture; sports learning; Web-based education

I. COMPUTER-BASED INSTRUCTIONAL SYSTEMS With the advances in computer technology, many

systems have been developed for educational and instructional purposes. In particular, a computer-based system is very attractive in sports instruction and training as compared to the traditional human coaching approach, saving time, space, and cost. A person learning a new sport or upgrading his/her skill in a familiar sport would be able to do so without the presence of a coach. In many sports such as tennis and golf, an experienced coach is difficult to engage for the desired lesson time, not to mention that it is extremely expensive to hire a private tutor in sports. In addition, in many countries around the world, land is a premium asset making sports facilities costly and difficult to reserve; however, the use of a computer-based instructional system would eliminate these inconveniences.

There are many existing instructional systems in the literature, either proposed or implemented, each focusing on some specific aspects of a sports instructional system. For instances, a proposal to use fuzzy logic and expert system approach can be found in [7], and several patent applications with [2] as one of the cases that emphasize the general purpose nature of such systems.

Among several important technical issues to be resolved in a sports instructional system, capturing the motion or gesture and organizing an efficient database are the major ones. Of course, the most successful implementations are the systems from the gaming industry. In this work, the technologies and systems of Microsoft [8], Nintendo [9], and Sony [10] are briefly discussed. A prototype system for learning tennis is introduced. Techniques for motion capture are explored. A Web-based generic sport system framework derived from our prototype system is presented.

II. GESTURE AND MOTION DETECTION DEVICES Motion tracking is an actively researched area [3, 4, 11],

fuelled by the commercial success of the three major video game developers (i.e., Microsoft, Nintendo, and Sony). All three have released, at the time of this writing, consoles with motion tracking capabilities.

The recently introduced (September 2010) Sony PlayStation Move controller has a multi-color light source that, together with a Web cam, provides data of the controller’s movement in the 3D space. Furthermore, accelerometer and inertial sensor within the handheld controller supply additional data to increase positional sensing accuracy [10].

Microsoft released (November 2010) Kinect for its Xbox 360 console is another Web cam based system; however, unlike Sony’s Move controller, Kinect provides a device-less human machine interface using gesture and voice [8].

The Nintendo Wii (November 2006) remote controller, or the Wiimote, uses accelerometers and infrared detectors (together with the LED sensor bar) to sense the controller in the 3D space [9]. The add-on MotionPlus attachment is capable of supplying three axes of orientation change (or angular velocity) through its built-in gyroscopes.

Of the three, the oldest Nintendo Wiimote is of particular interest as it presents a low-cost yet reasonably accurate means to capture motion or gesture data [1, 5]. Furthermore, the Wiimote can be set up easily and connected to a computer via a USB Bluetooth transceiver. However, the Wiimote has several limitations. The major one being the small “waggling” motions, intentional or not, could

2011 Workshops of International Conference on Advanced Information Networking and Applications

978-0-7695-4338-3/11 $26.00 © 2011 IEEE

DOI 10.1109/WAINA.2011.141

650

sometimes be interpreted as full movements. Though, this deficiency can be corrected by various means [13].

III. AN INSTRUCTIONAL SYSTEM FOR TENNIS

A. Background and Motivation To provide sports instruction at schools or recreational

facilities, learning materials using multimedia content can be developed in order to assist the learners to understand a sport’s playing motions. Ideally, having an instructor showing the actual playing motion is the best way to convey to a learner the proper movement technique. However, for example, when a school teacher has to instruct sports to his/her students in a physical education class, the teacher does not always have the necessary or sufficient skill to show the appropriate playing motions. This is usually the case since the teacher often is not necessarily a good player of that particular sport. Therefore, it would be useful and effective to show the playing motions in a multimedia format, especially using video material. In addition, showing the playing motions of the world’s top players in a video motivates the learners and entices them to practice further.

For a proof of concept of our ideas, we have implemented a tennis instruction system. In the sport of tennis, a racket swing, as shown in Figure 1, is one of the most basic play motions. It is therefore beneficial to the learner to leverage a tennis video database for racket swing motions, when learning how to play tennis. Using a 3D acceleration sensor to capture a learner’s motion, a video search subsystem can find the closest match in a tennis motion video database, as shown in Figure 2.

[Forehand Drive]

[Forehand Slice]

Figure 1. Examples of Racket Swing Motion in Tennis

Figure 2. A Snapshot of Our Prototype Tennis Instruction System

There are numerous tennis swings that can be classified

into various types of motions such as forehand stroke, backhand stroke, forehand volley, backhand volley, smash, and serve. Furthermore, even among the basic strokes, there are several subcategories such as flat, drive, and slice, as shown in Table 1. In addition, it is not uncommon that, in a

tennis match, the players use various ‘anomalistic’, ‘creative’, and ‘non-traditional’ swinging motions. Finally, a large number of adjectives such as strongly, gingerly, etc., can be used to describe each unique type of swings.

With such a variety of swings, no doubt that it is a daunting task for database designers to create the commonly

651

employed term-based index using somewhat subjective phrases such as ‘strongly up-and-forward’ and ‘angularly upward’ to represent the swinging motions and their corresponding videos. In addition, it is almost impossible to have a unified descriptive phrase for each type of swings intuitive and agreeable to all people so that it can be used to search the corresponding video in the database.

In order to address such a problem, our system leverages 3D data of swing motions, in this case as obtained from a 3D acceleration sensor. The Nintendo Wiimote is chosen as the sensor due to its versatility, software availability, and low cost. The 3D acceleration data of swing motions are tagged as metadata to the corresponding swing motions in the tennis videos. A learner can intuitively search for and retrieve the video of a particular swing motion by imitating the desired swing.

TABLE I. EXAMPLES OF TENNIS SWING DESCRIPTION

Stroke Type Description (from Wikipedia)[Forehand Drive]

Topspin (Drive) is a property of a shot where the ball rotates as if rolling in the same direction as it is moving. Topspin on a shot imparts a downward force that causes the ball to drop, due to its interaction with the air. It can be generated by hitting the ball with an up-and-forward swing, with the racquet facing below the direction it is moving.

[Forehand Slice]

Slice is a shot such that the ball rotates backwards (as though rolling back towards the player) after it is hit. The trajectory of the shot involves an upward force that lifts the ball (see Magnus effect). While a normal hit bounces forward, when backspin shots bounce, they tend to bounce off the sides or even bounce up.

B. System Overview The proposed instructional system searches a particular

tennis video according to the movement of a swing as captured by the 3D acceleration sensor. Thus, users can find specific tennis videos by swinging their rackets imitating the swing motion of interest. The system allows users to find, intuitively, specific tennis videos by only swinging their rackets, rather than a text-based or mouse-click-based input.

1) Process to Create Motion Metadata for Tennis Video

Our tennis instructional system attaches motion metadata captured by the Wiimote 3D acceleration sensor to each tennis video in the following steps, as shown in Figure 3:

Step-1: The database creator, preferably a decent tennis player, performs the actually racket swing motion using the Wiimote according to content of each tennis video.

Step-2: A motion metadata extraction module captures and extracts the 3D acceleration data of the arm-motion in Step-1. Step-3: The motion metadata extraction module registers the 3D acceleration data, linked to the corresponding tennis video, into the motion metadata database.

2) Retrieval Steps for Tennis Video

Our tennis instructional system retrieves a tennis video according to the input swing motion of the racket using the Wiimote 3D acceleration sensor in the following steps, as shown in Figure 4:

Step-1: A learner performs the actual racket swing motion using the Wiimote in order to search for the desired tennis video.

Tennis Movie DB

Motion Metadata DB

Motion Metadata Registration IF

Wii Remote

Motion Metadata Extraction Module

Step-1

Step-2

Step-3

Figure 3. Creation of Motion Metadata

Tennis Movie DB

Motion Metadata DB

Search IF

Wii Remote

Motion Metadata Extraction Module

Step-1

Step-2

Step-3

Similarity ComputationModule

Step-4

Figure 4. Retrieval of Tennis Video

652

Step-2: The motion metadata extraction module extracts the 3D acceleration data of the arm-motion in Step-1. The motion metadata extracted is used as a query for the tennis video database. Step-3: A similarity computation module computes the similarity of the 3D acceleration data between the query created in Step-2 and the motion metadata associated with each tennis video. Step-4: A search user interface (IF) shows the ranking results based on the similarity values obtained in Step-3, and the user has the option of playing the video corresponds to the highest ranking swing.

IV. A WEB-BASED INSTRUCTIONAL SYSTEM The system introduced in Section III is ideal for individual users who have the instructional system installed on their own PCs. A more elegant and efficient approach is to have a Web-based system in which users can communicate to a centralized Web database with minimal specialized hardware and software. Obviously the USB transceiver used in the tennis instructional system is a good choice for motion input capture, and the input device Wiimote is relatively inexpensive.

The idea behind a Web-based system is that it can be made available to the general public. This would enable many users, independent of their physical locations, to utilize and enjoy the system; however, this great benefit comes with a new problem. Each user is of different size and therefore the features of a swing as introduced in Section III may be skewed. Without having motion accurately detected, captured, and normalized, the matching to the motion metadata database would not work accurately. To resolve this problem, event detection technique has been investigated and utilized, where sequence of events can be captured and normalized to the size of the user. This technique also eliminates the “waggling” motion issue as discussed in Section II.

Wiimotelib is a library of open source code [12] for using and integrating the Wiimote functions into a C# program. The most recent beta version has incorporated some MotionPlus support which we have modified and tailored to our needs.

A. Fine-Tuning Motion Capture The most commonly used motion matching approach is

the point-to-point correspondence where the captured motion data is compared to the baseline or model motions using feature vectors. This method is simple though it has two main disadvantages. First, the relative timing when comparing two feature vectors is a problematic issue due to the large variation of the size of a motion. For example, people with longer arms would produce larger swings than people with shorter arms. Furthermore, a large number of

model swings for each categorized size must be made available, necessitates a large database and complex analysis.

An event-based matching approach is proposed that breaks down a motion into events when the type of movement changes. With the beginning to the end of a swing as the reference time line, instances of changes can be derived from the sequence of event occurrences. The periods between events become proportional to the reference time line thus allowing a single swing model of a known type to represent swings of different size. The added advantages of an event-based approach include a manageable database size and simplified analysis.

The Wiimote with the MotionPlus provides six data points from each sample: the three axes of angular velocity and three axes of acceleration [13]. Thus, each event Ei can be represented by the three-tuple: Gyro-velocity �vi, Gyro-slope �si, and Acceleration-time Ati:

Ei = (�vi, �si, Ati) Using Gxi, Gyi, and Gzi as the values captured at each

sampling by the gyroscopes, we have, using the x-axis data as an example,

Gyro-velocity �vxi = average (Gxi-n/2 + … + Gxi-1 + Gxi + Gxi+1 + … +

Gxi+n/2) This quantity can then be classified into positive (+), negative (-), and approximately-0 (~0), showing the direction of the rotation.

Gyro-slope �sxi = function [(Gxi+2 – Gxi+1), (Gxi+3 – Gxi+2), …, (Gxi+n –

Gxi+n-1)]

This quantity indicates trends of increasing, decreasing, flat (i.e., zero) and undefined (i.e., the slope is changing). Details of how the Gyro-velocity and the Gyro-slope are derived can be found in [6].

The acceleration data include gravity and it is difficult to remove this component from the actual data. The squared sum of the 3-axis acceleration data, however, is found to be Gaussian for many motions and therefore this quantity is being used as a relative timing reference:

Ati = (Axi + Ayi + Azi)2

Since there are three possible Gyro-velocity’s and four

Gyro-slope’s, therefore the twelve total combinations of these two quantities constitute the set of events types (e.g., ~0 and increasing, + and decreasing). A change of event type during the swing marks a time reference and can be used to establish a time interval between two events. Proportional to the entire swing, this reference time line compensates for swings of shorter or longer arms.

For each swing, a sequence of events is detected and time references are established with the corresponding Ati values.

653

This concise representation of swing motions makes the size of the database manageable and the computation tractable.

To reduce the impact of acceleration jitter, accelerations below a certain threshold are ignored [13]. Moreover, to reduce the effect of noise in the captured motion data, a filter must be used to obtain a clean smooth signal for further processing. Various filters were investigated in our experiments. Although a low-pass filter in the form of

[�f(n) + (1-�)f(n-1)]

was found to produce reasonable results, one hundred passes of a triangular filter

[((f(n-2) + 2f(n-1) + 3f(n) + 2f(n+1) + f(n+2)) / 9] seems to give the cleanest and most consistent signal. As an illustration, Figures 5 and 6 show the Pre- and Post-filtered Acceleration-time Ati and the Gyro-velocity �vxi along the x-axis of a forehand swing, respectively. In both figures, the vertical axis shows the captured or processed input data and the horizontal axis shows the instances of the Acceleration-time of an event change relative to the unit time of a complete swing.

B. Event-Based Approach Validation To illustrate the feasibility of the event-based matching

approach, preliminary testing and validation have been carried out. The objectives are, first, to capture and process the swings by people of different size such that the data are normalized and can be compared on the same scale, and second, to examine the data sets of different swings are indeed distinguishable for subsequent processing.

Several volunteers used the Wiimote and performed different swing motions. Of particular interest are the data of Goel and Jessie. Goel is a six-foot one-inch tall athletic male while Jessie is a five-foot two-inch female who is not physically inclined.

Figure 7 shows the x-axis Gyro-Velocity �vxi data while Figure 8 shows the z-axis Gyro-Velocity �vzi comparison of both subjects’ forehand and overhead swings. It has been observed that for each swing, there is a dominant axis which shows the prominent motion of that swing. For the forehand swing, the x-axis is the dominant axis while the z-axis is dominant in an overhead swing. The motions, hence the data, on the ‘secondary’ axes, seem to be more difficult to interpret though they may be meaningful and useful in certain applications.

It can be seen in both figures that the two subjects produce similar data envelopes on the dominant axis of the respective swings. This indicates the data sets are normalized on the same scale for people of different size. Furthermore, the data envelopes of the forehand swing on its dominant x-axis are clearly distinguishable to that of the overhead swing on its dominant z-axis. Once the dominant axes of different types of swings are identified, the data of the dominant axes can be used for matching or categorizing swing motion. Using only the dominant axis facilitates the matching

process, economizes the database, and potentially improves result quality.

The preliminary results as illustrated are very encouraging. The value of the event-based matching approach is demonstrated and it warrants further investigation. Further details of motion data filtering and event detection can be found in [6].

A Web-based tennis swing database can be established using the above event representation and the process as discussed in Section III. Similarly, a learner’s swing action is captured as a sequence of events and can be used to find a match in the database using similarity computation as described in Section III.

V. A FRAMEWORK FOR A GENERIC WEB-BASED INSTRUCTIONAL SYSTEM

The Web-based tennis instruction system can be further refined and extended to a generic Web-based instructional system as shown in Figure 9.

Currently, several video databases, such as YouTube, are available on the Web freely for public viewing. In many of these Web video databases, there exist abundant video contents, for a variety of sports including tennis, golf, badminton and fencing, etc., However, these legacy video databases accept only keyword-based query as described in Section III and do not provide any search interface that can leverage a user’s timed 3D motion as obtained from a sensor device.

Using our approach, databases of motion metadata can be constructed independently and linked to corresponding video contents stored in the legacy video databases. In our motion representation scheme, the timed 3D motion data are compressed in the form of event-based data as described in Section IV. A transfer protocol of the timed 3D motion data can easily be implemented in the Web environment instead of using raw data directly obtained from the sensor device. For example, in a Web-based tennis instructional system, a user can intuitively obtain the desired tennis video from a legacy video database such as YouTube with an event-based 3D motion query, simply by swinging a sensor device like the Wiimote controller.

In addition, our Web-based sports instructional system can also be applied to other sports domains such as golf, badminton, baseball, and fencing, etc., in which instructions for 3D motions using a club, racket, bat, and foil are essential in improving a learner’s skill.

VI. VALIDATION AND CONCLUSION Currently, we are validating and streamlining the event-based motion input capture technique and the matching process into a motion metadata database. Specifically, the sequence of events representing a particular type of swing should be applicable to learners of different physical size, and the sequences of events representing all the swings in the universal motion metadata set should be sufficiently distinguishable for the subsequent matching purpose.

654

References

[1] Bradshaw, D., and K. Ng, “Tracking Conductors Hand Movements Using Multiple Wiimotes”, International Conference on Automated Solutions for Cross Media Content and Multi-Cannel Distribution, pp. 93-99, 2008.

[2] Diehl, G.M., J.f. Lynch, and D. Bastone, Patent application title “Sports Instruction system and Method,”, available at http://www.faqs.org/patents/app/20080293023, October 20, 2010.

[3] Hay, S., J. Newman, and R. Harle, “Optical Tracking Using Commodity Hardware”, IEEE International Sympoisum on Mixed and Augmented Reality, Cambridge, UK, pp. 159-160, 2008.

[4] Hoffman, M., P. Varcholik, and J.J. LaViola Jr., “Breaking the Status Quo: Improving 3D Gesture Recognition with Spatially Convenient Input Devices”, IEEE Virtual Reality, pp. 59-66, 2010.

[5] Lee, J.C., “Hacking the Nintendo Wii Remote”, IEEE Pervasive Computing, Volume 7, Issue 3, pp. 39-45, 2008.

[6] Li, K.F., and M.G. Johnson, “Capturing Motion Data with the Wiimote”, NEWS Technical Report, Department of Electrical and Computer Engineering, University of Victoria, Canada, August 2010.

[7] Lo, C.-Y., H.-I. Chang, and Y.-T. Chang, “Research on Recreational Sports Instruction Using an Expert System”, LNCS 5820, J. Liu et al. editors, pp. 250-262, 2009.

[8] Microsoft XBOX, available at http://www.xbox.com, October 20, 2010.

[9] Nintendo Wii System, available at http://wii.com, October 20, 2010. [10] Sony PlayStation, available at http://www.playstation.com, October

20, 2010. [11] Wang, Y., et al., “Using Human Body Gestures as Inputs for Gaming

via Depth Analysis”, IEEE International Conferene on Multimedia and Expo, pp. 993-996, 2008.

[12] Wiimotelib, available at http://wiimotelib.codeplex.com, October 20, 2010.

[13] Wingrave, C.A., et al., “The Wiimote and Beyond: Spatially Convient Devices for 3D User Interfaces”, IEEE Computer Graphics and Applications, pp. 71-85, Volume 30, Issue 2, 2010.

Figure 5. Acceleration Data of a Forehand Swing

655

Figure 6. Gyro-Velocity Data of a Forehand Swing

Figure 7. Gyro-Velocity �vxi Comparison

656

Figure 8. Gyro-Velocity �vzi Comparison

TennisVideo�DB

Legacy�Video�Databases�on�the�Web

Motion�metadata�databases��are�associated�with�each�video�database.

Web

...Golf�Video�DB

Gesture�based

Video�DB

Tennis�Motion�Metadata�DB

Golf�Motion�Metadata�DB

Motion�Metadata�DB

Results

Query�using�event�based�motion�data�

User�inputs�query�using�sensor�device�representing� timed�3D�motion.

Figure 9. A Generic Web-Based Instructional System

657