Good Morning!Good Morning!

• How many times have you tried to get How many times have you tried to get ahold of someone and failed?ahold of someone and failed?• Need to keep tryingNeed to keep trying• Need to find alternate waysNeed to find alternate ways

• How many times have you forgotten an How many times have you forgotten an important appointment?important appointment?

With VERA you can…With VERA you can…

• Schedule wake-ups for yourself and Schedule wake-ups for yourself and othersothers

• Schedule message delivery to yourself Schedule message delivery to yourself and othersand others

• Schedule recurring callsSchedule recurring calls• Send “urgent” messages with return Send “urgent” messages with return

receiptreceipt• Automatically try multiple contacts for Automatically try multiple contacts for

a persona person• Automatically try again laterAutomatically try again later

Voice Enabled Reminding AgentVoice Enabled Reminding AgentVera-OUT: UlaVera-OUT: Ulaşş Bardak, Scott Judy, Vasco Pedro Bardak, Scott Judy, Vasco PedroVera-IN: Tal Blum, Jeongwoo Ko, Ryosuke MiyataVera-IN: Tal Blum, Jeongwoo Ko, Ryosuke Miyata

OverviewOverview

• Overall ArchitectureOverall Architecture• VeraOUT ArchitectureVeraOUT Architecture

• SkyperSkyper• Dialog Manager InternalsDialog Manager Internals• Web interfaceWeb interface

• DemoDemo• Pros and ConsPros and Cons• Future WorkFuture Work

Overall Vera Architecture

Web Interface

Vera-OUT**

*Vera-IN by Tal *Vera-IN by Tal Blum, Jeongwoo Blum, Jeongwoo Ko, and Ryosuke Ko, and Ryosuke MiyataMiyata

Vera-IN*

DB

Skype

Skype

InternetInternet

VoIP VoIP ContactsContacts

**Vera-OUT by **Vera-OUT by Ulas Bardak, Scott Ulas Bardak, Scott Judy, and Vasco Judy, and Vasco PedroPedro

SkypeSwitching Center

PSTN

Vera Server Audio

Out

Out

Mic

Mic

Monitor SpeakersMonitor Speakersfor User Speechfor User Speech

Monitor SpeakersMonitor Speakersfor Synthesizedfor SynthesizedSpeechSpeech InternetInternet

CepstralSpeech

Synthesizer

Sphinx SpeechRecongizer

Sound Card #1(Windows Default

Audio) Sound Card #2(Skype Audio)

VoIPSkype

Software

VeraOUT Building Blocks

Vera-OUTVera-OUT

DM SystemDM System

VeraTM

VoIP

Skyper

VeraDM

DB Backend

Web Interface

Vera-IN

OtherComponents

VeraOUT Building Blocks

Vera-OUTVera-OUT

DM SystemDM System

VeraTM

VoIP

Skyper

VeraDM

DB Backend

Web Interface

Vera-IN

OtherComponents

Dialog Management Internals

Lang. Understand.PHOENIX/HELIOS

Dialog Manag.RavenClaw

Lang. GenerationROSETTA

RecognitionSPHINX

SynthesisTHETA

From Dan Bohus’ talk on Ravenclaw

Vera Backend

Dialog Management Internals

Lang. Understand.PHOENIX/HELIOS

Dialog Manag.RavenClaw

Vera Backend

Lang. GenerationROSETTA

HUB

RecognitionSPHINX

SynthesisTHETA

From Dan Bohus’ talk on Ravenclaw

PROCESSMONITOR

SPHINX

Dialog Management Internals

Dialog Manag.RavenClaw

Vera Backend

Lang. GenerationROSETTA

HUB

Lang. Understand.PHOENIX/HELIOS

RecognitionServer

SynthesisTHETA

Multiple, paralleldecoders

Task ManagerVeraTM

DB Access

Text I/OTTYServer

ParsingPHOENIX

ConfidenceHELIOS

From Dan Bohus’ talk on Ravenclaw

DateTime

Skyper

Vera-OUT Architecture

Vera-OUTVera-OUT

DM SystemDM System

VeraTM

VoIP

Skyper

VeraDM

DB Backend

Web Interface

Vera-IN

OtherComponents

Database

Vera-OUT Building Blocks

Vera-OUTVera-OUT

DM SystemDM System

VeraTM

VoIP

Skyper

VeraDM

DB Backend

Web Interface

Vera-IN

OtherComponents

Web Interface

VERA-OUT DemoVERA-OUT Demo

We showed a demo at this point in the presentation.

Here are some recorded sample conversations:

RavenClaw Agent Tree

E:GetContextE:GetContext A:GetUserA:GetUser

A:IsUserA:IsUser R:SpeakWithUserR:SpeakWithUser R:WaitForUserR:WaitForUser

A:VeraOutA:VeraOut

R:LogoutR:Logout

I:DeliverMessageI:DeliverMessage

A:RightUserA:RightUser

E:SetCompletedE:SetCompleted

A:WrongUserA:WrongUser

I:CallLaterI:CallLater E:CallNextE:CallNext A:DoTaskA:DoTask

I:WakeUpI:WakeUp I:WakeUpSystemI:WakeUpSystem

E:CheckTaskIDE:CheckTaskID

I:MessageDeliveredI:MessageDeliveredI:MessageNotDeliveredI:MessageNotDelivered

I:InformMessageI:InformMessage A:messageHandlingA:messageHandling

E:setSuspendE:setSuspend

R:getUserStringR:getUserString

E:setRightUserE:setRightUser

E:setWrongUserE:setWrongUser

E:setNotHereE:setNotHere

R:WaitForSpeechR:WaitForSpeech

I:PlayMessageI:PlayMessage

R:getTaskDoneStringR:getTaskDoneString

I:RepeatMessageI:RepeatMessage

E:SetMessagePlayedE:SetMessagePlayed

VERA-Out User TestsVERA-Out User Tests

• A user pool of around 10 fully A user pool of around 10 fully cooperating userscooperating users

• Used a relatively simpler dialogueUsed a relatively simpler dialogue

• Problems:Problems:• Non-native speakers have a lot of troubleNon-native speakers have a lot of trouble• Background noise and booming on the Background noise and booming on the

phone can affect the systemphone can affect the system• Speakers get very confused when the Speakers get very confused when the

system repeats itself with no clarification.system repeats itself with no clarification.

Vera-Out User Test OutcomeVera-Out User Test Outcome

• Increased the dialog cooperationIncreased the dialog cooperation• Clarification on what’s being asked forClarification on what’s being asked for• Stating what is expectedStating what is expected

• Added more coverage for speech Added more coverage for speech recognition grammar.recognition grammar.

• Added more coverage for dialog Added more coverage for dialog manager.manager.

VERA-OUT AdvantagesVERA-OUT Advantages

• Skype is inxexpensiveSkype is inxexpensive• Skype has worldwide reach.Skype has worldwide reach.• No phone line or special equipment required No phone line or special equipment required

(except for an extra sound card).(except for an extra sound card).• Privacy is preservedPrivacy is preserved

• No numbers are shared with other users.No numbers are shared with other users.• No calls made to a number once the user removes it.No calls made to a number once the user removes it.

• Ability to call someone at a time when you may Ability to call someone at a time when you may not be free yourself.not be free yourself.

• PersistencePersistence• Ability to confirm delivery of messages.Ability to confirm delivery of messages.• Support for recurring tasksSupport for recurring tasks

VERA-OUT IssuesVERA-OUT Issues

• Skype does cost money when phones are Skype does cost money when phones are called. What’s the best way to do called. What’s the best way to do accounting / charge the right user?accounting / charge the right user?

• Skype calls show up as “unknown” on caller Skype calls show up as “unknown” on caller ID.ID.

• Being socially appropriate:Being socially appropriate:• Where’s the line between “trying hard enough Where’s the line between “trying hard enough

to make contact” and “harassment”?to make contact” and “harassment”?• How late/early should VERA call? (time zones?)How late/early should VERA call? (time zones?)• Can others override this in an emergency?Can others override this in an emergency?• Who decides what an emergency is?Who decides what an emergency is?

VERA-OUT Issues (ctd.)VERA-OUT Issues (ctd.)

• Should users be able to cancel all the calls Should users be able to cancel all the calls that others have set up for them?that others have set up for them?

• Handling failure to make contact.Handling failure to make contact.• Calling other contacts.Calling other contacts.• Calling back later (after 1 hour)Calling back later (after 1 hour)• More work needed.More work needed.

• Answering machinesAnswering machines• How to handle someone who’s not How to handle someone who’s not

expecting a call from a computerexpecting a call from a computer

VERA-OUT Future WorkVERA-OUT Future Work

• Better dialog to confirm user identity Better dialog to confirm user identity • Blocked users / Allowed usersBlocked users / Allowed users• Calls to multiple usersCalls to multiple users• Getting user contacts from the Skype Getting user contacts from the Skype

directorydirectory• Ability to forward and reply to Ability to forward and reply to

messages.messages.

VERA-OUT Future Work (ctd.)VERA-OUT Future Work (ctd.)

• Setting up meetings with multiple Setting up meetings with multiple participants.participants.

1.1. Call VERA-IN or use the web interface to specify Call VERA-IN or use the web interface to specify available times and target meeting participants.available times and target meeting participants.

2.2. VERA-OUT calls everyone and finds common VERA-OUT calls everyone and finds common times.times.

3.3. VERA-OUT gets final approval from the initiator.VERA-OUT gets final approval from the initiator.

4.4. VERA-OUT calls Roomline to book a room???? VERA-OUT calls Roomline to book a room????

5.5. VERA-OUT informs everyone of the result.VERA-OUT informs everyone of the result.

VERA-OUT ConclusionVERA-OUT Conclusion

• With Vera, reach the people you need to reach, when you want to reach them!

Special Thanks!

• Our professors:• Alan Black• Alex Rudnicky

• RavenClaw experts:• Dan Bohus• Jahanzeb Sherwani• Antoine Raux

• Our longsuffering office-mates:• Yee-Man (Betty) Cheng• Yanjun Qi

VERA-OUTVERA-OUT

The EndThe End