april 23, 2005 mitsubishi electric research laboratories

April 23, 2005

Mitsubishi Electric Research Laboratories

James L. FrankelSlide 1

Combadge: A Voice Messaging Device for the Masses

Berkeley UNIDO ConferenceInformation & Communications Technology (ICT) Workshop

April 23, 2005

James L. FrankelMitsubishi Electric Research Laboratories

Cambridge, Massachusetts

April 23, 2005



Combadge

A speech-enabled communications device Functionality: Two-way voice messaging

with simple spoken commands and a one-button interface.

Platform: Basis for new handheld research

Goal: Bring state-of-the-art wireless communication and services to the less-wealthy in the world with a simple, low-cost device.

Advantages: Offers new services, yet is unimposing and non-intrusive, with low device and low ongoing infrastructure costs.

Contact: [email protected]

April 23, 2005



Asynchronous Operation (1 of 2)

Users decide when to listen and respondMessages are sent to and from device when connected

Device can be very smallHas no display

Requires only one button

Need not reach from mouth to ear

In the future, it will be feasible to be packaged in a watch

Voice interface makes Combadge usable by illiterate users

Can use better compressionNo need for real-time compression

Can fully utilize available spectrum (packet switched)

April 23, 2005



Asynchronous Operation (2 of 2)

Graceful degradation of service during network overload

Users less aware of dead spots in networkFunctional without any connectivity

Messages are cached in the Combadge

All functions that don’t require communication are useable

Reduces peak power demand, allowing much longer battery lifeSpeech recognition, compression and radio not used simultaneously

Can operate radio less frequently (it's like voice IM, not a phone)

Can use Internet for cheap global connectivity (like e-mail or IP telephony)

Makes group messaging easy

April 23, 2005



SimpleSingle button, push-to-talk: no keypad, no display

Reduced manufacture cost and reduced power used

Simple interface using speech, e.g.:“New message for Peter""Play New", "Reply"

Talk immediately: no waiting for a dial tone, for someone to answer, or for a menuAfter adding another Combadge to the phonebook, there are no phone numbers to memorize

Everyone is identified by spoken name (or nickname)For children, restrictions applied on adding new Combadges

Optionally, no messages from people you don’t know

April 23, 2005



Customer Base

Appeal to new users:

The less-privileged and less-educated in the world (including developing countries)

Designed for illiterate users

Lower cost device

Lower cost service

The cost conscious, such as youth (ages 8-14) and the elderly

Those irritated or intimidated by cell phones

Use cellular networks, but create a low bandwidth, low cost service

Use 802.11a/b/g for campus or village/town/city connectivity

Can use DakNet-like network for transport

April 23, 2005



Interaction with Services and Other Devices

Open-ended opportunity to create new services, providing simple spoken interfaces to the entire digital universe

“Weather for Boston”

“Market price for rice”

“Calendar: Am I free Friday afternoon?”

“Traffic on the Mass. Pike”

Voice control of devices“House: Turn garage lights on”

“HVAC: Set living room temperature to 20 degrees Celsius”

Integration with e-mail, telephones, voice mail, etc.

April 23, 2005



Hardware (Introduction)

Hardware component is code-named “Dilithium”

Back side of main board

April 23, 2005



Hardware (Introduction)

Front side of main board

April 23, 2005



Hardware (Daughterboard)

Daughterboard

April 23, 2005



Hardware (Case Components)

Some Case Components

April 23, 2005



Hardware (In Case)

Dilithium in Case

April 23, 2005



Assembled Combadge

April 23, 2005



Combadge In Use

April 23, 2005



Hardware (1 of 4)Processor is Intel XScale StrongARM running at 206 MHz

Moving to Intel XScale at 400 to 624 MHz and faster

MemorySDRAM: 64 Mbytes; Flash: 64 Mbytes

Integrated GSM/GPRS Modem for Wide-area NetworkingOn-board SIM Socket

Optional Daughterboard Provides One or Two Compact Flash (CF) Slots

802.11b Local Area NetworkingMany Other CF Peripherals (Ethernet, CF Memory Cards, Additional I/O Ports, CF Disk Drives)

Two On-board SiSonic Silicon-MEMS MicrophonesOn-microphone preampCan perform active noise cancellation

April 23, 2005



Hardware (2 of 4)Flexible CODEC sampling rates

11.025, 22.05, 44.1 (CD), 8 (telephony), 16, 32, and 48 KHz

LED’sTwo banks of blue LED’s under the translucent side buttonsTwo bi-color LED’s on frontOne LED for bi-directional communication using LEDComm

Two-axis AccelerometerGesture detection

Vibrator (for silent new message indication)JTAG ConnectionUSB PortSerial Port with on-board RS232 driversTwo Stereo 2.5mm Phone Jacks for Audio In and Audio Out

April 23, 2005



Hardware (3 of 4)Pushbuttons

Left and Right Push-to-Talk

Power On

Reset (Accessible through hole)

Real-time Clock

Dense component packing; Small overall size

Heavy use of BGA componentsProcessor, Four memory chips, and CPLD

Design of caseSolidWorks

SLA Master (Stereolithography)

Limited-run Rubber Molds

April 23, 2005



Hardware (4 of 4)Hardware Revisions

Rev. 1Fabricated one deviceThis device has had a fruitful lifeStill functional today

Rev. 2Fabricated five devicesThese are the devices in the demo

Rev. 3Power management hardware addedReal-time clock addedGround planes to attenuate audio noise addedFabricated twenty-five devices to date

XScale Revision (StrongARM has been discontinued)

April 23, 2005



Software (1 of 5)

InitializationJTAG Programming Utility

Initializes Flash memory using JTAG interface to StrongARM

Boot LoaderFirst Program running on StrongARM

Initializes memory and I/O devices

Provides debugging tools

Loads Operating System

Linux Operating SystemWe ported Linux 2.4.19 to Dilithium

Started with the Compaq “Familiar” Linux port

April 23, 2005



Software (2 of 5)Linux Porting Issues

Our New Dilithium Architecture

New Flash memory chips

Custom Device Drivers

Accelerometer, buttons, LED’s

Combadge Voice-Messaging ApplicationInitial development on iPAQ PDA running Linux

Developed in Python, C, C++, and Shell Scripts

Voice RecognitionTwo Recognizers (Using SDX from SpeechWorks/ScanSoft):

One for speaker-independent tokens

One for speaker-dependent name tags such as the name given to phonebook entry

April 23, 2005



Software (3 of 5)Grammar used for Combadge commands

Play new messages; Play again; Play next; Play previousNew message for <name>ReplyCreate contactPhonebookStatus all; Status ID; Status connection; Status messages; …Profile normal; Profile meeting; Profile silentVolume 1; Volume 9; Volume off; …Delete contact <name>; Delete all contactsShutdown; Restart; Configure MERL; Configure adhoc; Configure GPRS; …Version; Utility ping

April 23, 2005



Software (4 of 5)Combadge application complexities

Heavily multi-threadedBarge in capabilityExtensive loggingGraceful handling of exceptional events

Power-down components when not usedAmplifierGSM/GPRS modem802.11b interface

More work is needed to cause Combadge to sleep to extend battery life when device is inactiveAudio messages are now PCM files; will transition to WAV filesGateway from voicemail system at MERL to Combadge

April 23, 2005



Software (5 of 5)Voice messages are delivered using SMTP and IMAP

A custom “cbd” protocol is used to communicate from the Combadge to a “cbd” serverThe “cbd” server actually sends messages via SMTP and gets messages via IMAPSMTP is also used directly by the Combadge to verify valid phonebook entry addresses (using VRFY)

The Combadge application does the management of three categories of messages

Recorded to be sent, but not yet sent to serverReceived from server, but not yet heardReceived from server and already heard

The Combadge maintains a cache of messages in its own memoryCombadge is fully-functional without any connection to a network

April 23, 2005



Deployment ConnectionsU. C. Berkeley

Eric Brewer

Divya Ramachandran, Graduate Student

Voice recognition for Tamil

Integration with Berkeley’s network transport for intermittent connectivity and long-distance 802.11b

Deployment in Tamil Nadu in India

Media Lab at MITSMART Group – EKG information transmission in ER or disaster situation

Mike Best – Potential developing world deployments

World Bank

April 23, 2005



Server Environment

Server runs Linux with dhcpd, sendmail, imap (invoked by xinetd), and cbd (the Combadge server daemon)

April 23, 2005



Research Directions (1 of 3)User studies in developing world deploymentsUser studies in deployments in urban/suburban settings in the United StatesInvestigate mesh networking

Combadge as an infrastructure-less voice messaging consumer appliance (like a walkie-talkie/FRS/GMRS)Forward messages through other Combadges toward the destinationAttention needed to patterns of physical location of Combadge over time (i.e., usual weekday daytime location, usual weekend daytime location, usual nighttime location)Utilize connection to Internet when present

April 23, 2005



Research Directions (2 of 3)Develop services for Combadge users

Traffic reportingWeather informationSchedule/appointmentsStock quotes

Continue to Integrate with other Communication ParadigmsTelephone

Speech synthesisE-mailPagers

April 23, 2005



Research Directions (3 of 3)Develop as an audio home appliance remote control

Audio and video systems

Security system

HVAC

Audio interface to use as an MP3 player

Utilize Dilithium platform for other MERL projectsMicrophone and audio processing server

April 23, 2005



CreditsEarly work

Barry Perlman

David Anderson

Current workDaniel Bromberg

April 23, 2005



Questions and Discussion

april 23, 2005 mitsubishi electric research laboratories

Documents