Vocal JoystickA New Dimension in Human-Machine

Interaction

ET 2 PresentationGroup 3

Jeremy Moody, Carrie Chudy

Introduction

A new technology is emerging in the software area of information technology.

This “emerging technology” is called Vocal Joystick and will bring new light to the way we operate computers.

What Is It?

Vocal Joystick is software that will enable individuals with motor impairments to control objects on a computer screen and ultimately electro-mechanical instruments simply by using vocal parameters.

www.sciencentral.com/articles/view.php3?type=article&article_id=218393030

Who Uses It?

Vocal Joystick’s primary user is anyone with a physical impairment that limits the use of their arms and hands.

“There are many people who have perfect use of their voice who don’t have use of their hands and arms.” -Jeffrey Bilmes, Creator

However, Vocal Joystick software will be increasingly useful for everyday computer operators.

How Does It Work?

Vocal Joystick utilizes three main components• Acoustic signal processing• Pattern recognition• Motion control

Translates verbal cues into directional movements.

Acoustic Signal Processing

• The signal processing module extracts short-term acoustic features– Energy– Autocorrelation coefficients– Linear prediction coefficients– Mel frequency cepstral coefficients (MFCC)

• Signal conditioning and analysis techniques are needed for accurate estimation of these features

Pattern Recognition

• The features are then directed into the pattern recognition module where energy smoothing, pitch and formant tracking, vowel classification and discrete sound recognition take place.

• This stage involves statistical learning techniques such as neural networks and dynamic Bayesian networks.

Motion Control

• Energy, pitch, vowel quality and discrete sound become acoustic parameters to be transformed into direction, speed and other motion related parameters.

• These motion control parameters are then used to launch corresponding actions.

Engine Diagram

Vowel Corpus

The Vocal Joystick vowel corpus represents the sounds used to control the movements of the “mouse”.

Vocalic signals are used with manipulations of pitch and intensity for variation.

Vocal Corpus (cont’d)

• Detects sounds 100 times a second

• Users are able to transition smoothly from one vowel to another unlike in speech recognition.

• Louder sounds make the cursor move faster.

Applications

• Browsing the Web• Drawing on a screen• Controlling a cursor• Operating a robotic arm

• In the future, Vocal Joystick is optimistic about its use to control an electronic wheelchair.

Conclusion

Vocal Joystick is an amazingly beneficial “Emerging Technology” that will have a major impact on the Information Technology world.

With simply a microphone, a computer with a standard sound card and a user who can vocalize, anyone can utilize Vocal Joystick!

References

• http://www.sciencentral.com/articles/view.php3?type=article&article_id=218393030

• http://seattletimes.nwsource.com/html/businesstechnology/2008231288_btjoystick06.html

• http://ssli.ee.washington.edu/vj/

References (cont’d)

• http://ssli.ee.washington.edu/vj/engine.htm

• http://ssli.ee.washington.edu/vj/corpus.html

• http://www.yubanet.com/artman/publish/article_67481.shtml