Tele-Robotics DemonstrationTele-Robotics Demonstration(Rover Tele-operation)(Rover Tele-operation)

JUSTSAP-PISCES Symposium 2008JUSTSAP-PISCES Symposium 2008

November 11, 2008November 11, 2008

2

AgendaAgenda

Demonstration overview

Technology and Products of remote control

platform

Let’s enjoy “Rover Tele-operation Time Trial”

3

Demonstration overview Demonstration overview

Let's experience the effect of "latency cancLet's experience the effect of "latency cancel technology" for teleoperation.el technology" for teleoperation.

Let's experience the effect of "latency cancLet's experience the effect of "latency cancel technology" for teleoperation.el technology" for teleoperation.

RoverWireless LANAccess pointInternet simulate PC

Operation PC

PISCES

Simulate data transmission delay

in Internet

InternetInternetInternetInternet

Japan/USA

4

Technology and ProductsTechnology and Products

You can develop You can develop advancedadvanced remote control remote control system system easilyeasily and at a and at a low costlow cost..

TTeleelePPresence over resence over IPIP TTeleelePPresence over resence over IPIP

Platform for the remote control Platform for the remote control system system which requires the digital which requires the digital video video transmitted by wireless transmitted by wireless

LAN and internet.LAN and internet.

5

Technology and ProductsTechnology and Products

Advanced remote control system Operation from everywhere of Internet available places

High operativity using Motion JPEG video

Easily and low cost I/F for using COTS equipments (WirelessLAN, Internet, PC, game con

troller, etc.)

Software Library for easy system development by users

All-in-one platform for video-based remote control system

6

Remote control system using TPIPRemote control system using TPIP

Video & Sound data、 Sensor dataVideo & Sound data、 Sensor data

Control command data、 Sound dataControl command data、 Sound data

internetinternetinternetinternet

PC

controller

Target SystemTarget System

Wireless LAN card

WireLess LANAccess point

Bidirectional WireLess LANIEEE802.11a/b/g

NTSCCamer

a

SensorMicrophon

e

Speaker

Motorcontrol

input

output

TPIP UnitTPIP Unit

TPIP LibraryTPIP Library

7

Minimization of “Video transmission delay”Minimization of “Video transmission delay”

NTSC video capture

JPEGEncoder

JPEG compression

Wireless LANIEEE802.11g20Mbsp 15Kbyte(1/60)

Decoder&

rendering

JPEGdecode & rendering

PC monitorRefresh Rate:60Hz

VGA/QVGA 30fps

NTSC Camera

+ Internet

33ms

Processing timeProcessing time

20ms

6ms

20ms

17ms

TPIPTPIP Total delay time

96ms96ms≒100ms≒100msWeb Camera Total delay time

500ms500ms

Network Camera

VGA 15fpsQVGA 30fps

8

High-speedHigh-speed image compressionimage compression

30 frames/sec. (NTSC -> Motion JPEG)

No re-transmissionNo re-transmissionCommunicationCommunication UDP/IP

High-speedHigh-speedimage renderingimage rendering

High-speed Motion JPEG decoding, image processing middleware

Resolution VGA(640 x 480),QVGA(320 x 240)

Frame rate 30 frame/sec

Video transmission delay Less than 100ms

Performance

Minimization of “Video transmission delay”Minimization of “Video transmission delay”

9

Removal of “Transmission delay influence”Removal of “Transmission delay influence”

1sec delay1sec delay 　　

moves during delay time

4km/h4km/h （（ ex. walkex. walk ））

10km/h10km/h （（ ex. model carex. model car ））

HawaiiHawaii JapanJapan

6sec? delay6sec? delay 　　EarthEarth MoonMoon

HawaiiHawaii

2.8m

MoonMoon

HawaiiHawaii

MoonMoon

5700km5700km

384400km384400km

16.8m

6.6m

1.1m

10

received imagereceived image 　　 captured at “captured at “ T T ””

corrected imagecorrected image 　　 view at view at T+ΔTT+ΔT

Steering angleSteering angle

speedspeed

Image transmissionImage transmissiondelaydelay

ΔTΔT ：： Image transmission delayImage transmission delay

specify a red frame partspecify a red frame partby three data by three data

PointPoint TT

T+ΔTT+ΔT

Removal of “Transmission delay influence”Removal of “Transmission delay influence”

11

Correction of “Transmission delay fluctuation”Correction of “Transmission delay fluctuation”

Remove the video frame intervals fluctuation which dRemove the video frame intervals fluctuation which degrades operativity without image bufferingegrades operativity without image buffering

Delay time ： t1

Ｔ１ Ｔ２ Ｔ３

Ｔ４

Time Stamp

（ t1=“Time now” -T1 ）

SendSendimageimage

Ｔ１

Ｔ２

Ｔ３ Ｔ４

Ｔ１

Ｔ２

Ｔ３

Ｔ４

ReceiveReceiveimageimage

DisplayDisplayimageimage

A fluctuation by the transmission delay

occurs

display images at same interval by time stamps

12

Let’s enjoy “Rover Tele-operation Time Let’s enjoy “Rover Tele-operation Time Trial”Trial”

Start and GoalStart and Goal

First First GateGate

Second Second GateGate

Latency cancel technology

1st lap OFF

2nd lap ON

13

An Introduction “TPIP Unit”

14

Image processing boardImage processing boardImage processing boardImage processing board

Control boardControl boardControl boardControl board

The feature of TPIP unitThe feature of TPIP unit

Value remote control platformValue remote control platform

All-in-one platform for video-based remote control systemAll-in-one platform for video-based remote control system((Image processing board, control board, software libraryImage processing board, control board, software library))

Various input/output for remote control systemVarious input/output for remote control system

TPIP UnitTPIP UnitTPIP UnitTPIP Unit

Input image : NTSC

Output image : MotionJPEG (VGA/QVGA 30fps)

Sound input/output : Audio

Wireless Communication : IEEE802.11a/b/g

Digital I/O RC servo I/F

DC motor I/F

Pulse input

Analog input CAN I/F

RS232 I/F

15

TPIP Unit

Hardware

16

TPIP Unit TPIP Unit SpecificationsSpecifications

Vout

V1

V2

V3

pw

RS232

SH250MHz

S1

S2

S3

S4

S5

S6

S7

S8

S9

S10

WirelwssLAN(11a,11g)

Servo

NTSCCAMERA

Voltage

Encoder

V in

RS232

pw

Ethernet

WirelessLANCard

DC

6～

16

V SensorAnalog Input

JPEGchip

Motor SH4240MHz

Image processing boardImage processing boardTarget control boardTarget control board

P1

P2

A1

RS232

CAN I/F

RS232

Other unit

Other unit

5V

A3

A2

A4

A5

A7

A6

A8

I1I3I2I4

O1

O3

O2

O4

O５

O７ O

６O

８

SensorDigital INPUT

Switch

Audio

Mikespeaker

Vout NTSCdisplay

Item Specifications / performance

Communication

LANWireless ： IEEE802.11a,g,b (PC card) × １Cable ： 10baseT / 100baseTX × １

Image Image transmit

VGA/QVGA 30fps, delay ： Less than 100ms（ Remove IP network delay time ）

Camera I/F Max ３（ change signal ）

SoundSound

transmitMonaural recording bidirectional transmit（ Sampling Rate:16/8KHz,44.1KHz)

Board size 105×84×60mm

Item Specifications / performance

I/O RC servo I/F Max 　 10ch

DC motor I/F 1ch

Pulse input Encoder Input 2ch

Analog input 8ch

Digital I/O Input 4ch / output 8ch (TTL)

CAN I/F 1ch

RS232 I/F 2ch

17

Compatibility with the radio control system Compatibility with the radio control system

Image processingboard

R/C receiver

Servo

Servo Motor

amp

Motor

amp

Radio control system for hobby

Servo

Servo Motor

Motor

amp

Target controlboard

TPIP unitTPIP unit

DC MotorDC MotorOutputOutput

RC servoRC servoOutputOutput

11a

CameraCameraInputInput

SensorSensorInputInput

RC partsRC partsusableusable

RC servoRC servoOutputOutput

replaceablereplaceable

Two-wayTwo-wayDigital control commandDigital control commandVideo & Sensor dataVideo & Sensor data

One-wayOne-wayAnalog control Analog control command onlycommand only

DC MotorDC MotorOutputOutput

18

TPIP Unit

Software

19

SoundSoundcontrolcontrol

TPIP libraryTPIP library

OS : LinuxOS : Linux

JPEG compression boardJPEG compression board

Motor control boardMotor control board

JPEG device driverJPEG device driver

Real-Time Image transmitReal-Time Image transmitSoftwareSoftware

UDP / IPUDP / IP

S/W

H/W

Internet

TPIP UnitTPIP UnitTPIP UnitTPIP Unit

ControllerControllerSoftwareSoftware

PC/AT compatible machinePC/AT compatible machine

OS : Windows XPOS : Windows XPUDP / IPUDP / IP

JPEG decodeJPEG decodeSoftwareSoftware

Real-TimeReal-TimeImage transmitImage transmit

SoftwareSoftware

High-speedHigh-speedrenderingrendering

User programUser program

DLL DLL (Dynamic Link Library)(Dynamic Link Library)

TPIP LibraryTPIP LibraryTPIP LibraryTPIP Library

20

TPIP libraryTPIP library

USBUSB

DisplayDisplay

WirelessWirelessLANLAN

RS232RS232OtherOther

controllercontroller

Controller inputController inputfunctionfunction

Image control / transmitImage control / transmitfunctionfunction

SIO CommunicationSIO Communicationfunctionfunction

Graphic controlGraphic controlfunctionfunction

UDP functionUDP function

UDP-SIO communicationUDP-SIO communicationfunctionfunction

Base libraryBase library apply libraryapply library

TPIP libraryTPIP library

TPIP UnitTPIP Unit

21

UniversityUniversityTechnicalTechnicaljunior collegejunior college

ResearchResearchlaboratorieslaboratories

SanritzAutomationSanritzAutomation

TechnicalTechnical informationinformation

TPIP Library TPIP Library source codesource code

CommunityCommunity

ProductProductinformationinformation

Research &Research &developmentdevelopment

resultsresults

offer & technical supportoffer & technical support

archivearchive

Research and development(school) Research and development(school) ：： 26 members26 members

Research and development(Enterprise):Research and development(Enterprise):

11 members11 members

RRC Student member : 23 membersRRC Student member : 23 members

Student member : 9 membersStudent member : 9 members

Activities themeActivities theme・・ Robot ControlRobot Control・・ OpenCV Image data processinOpenCV Image data processingg　　　　　　　　　　　　　　　　　　　　　　 etc…etc…

TPIP User CommunityTPIP User Community

22

And…And…・・ TPIP Platform customizeTPIP Platform customize・・ TPIP Systems construction supportTPIP Systems construction support

Image processing boardImage processing board（（ compression & transmitcompression & transmit ））

\140,000\140,000

TPIP Platform PACKAGETPIP Platform PACKAGE\200,000\200,000

academic price \100,000academic price \100,000

Price in JapanPrice in Japan

Tele-Operation Rover PACKAGETele-Operation Rover PACKAGE\2,500,000\2,500,000

academic price \1,500,000academic price \1,500,000Remote controlRemote control

TargetTarget(Rover)(Rover)

TPIP UnitTPIP Unit

TPIP libraryTPIP library

TPIP UnitTPIP Unit

TPIP libraryTPIP library

Image processing boardImage processing board

TPIP libraryTPIP library

You can choose TPIP PlatformYou can choose TPIP Platform