computer vision technologies for remote collaboration using physical whiteboards, projectors and...

Computer Vision Technologies for Remote Collaboration Using Physical Whiteboards, Projectors and Cameras

Zhengyou ZhangZhengyou Zhang

Microsoft ResearchMicrosoft Researchmailto: mailto: zhang@microsoft.comzhang@microsoft.com

http://http://research.microsoft.comresearch.microsoft.com/~zhang//~zhang/

Motivations

WB provides a large shared space WB provides a large shared space for the participants to focus their attention and for the participants to focus their attention and

express their ideas spontaneouslyexpress their ideas spontaneously Many meetings use whiteboard heavily Many meetings use whiteboard heavily

brainstorming sessions, lectures, project planning brainstorming sessions, lectures, project planning meetings, patent disclosures, etc meetings, patent disclosures, etc

WB is ubiquitous in knowledge workers’ officesWB is ubiquitous in knowledge workers’ offices

→ → Whiteboard will still exist for long timeWhiteboard will still exist for long time

Motivations (cont’d)

Difficulties: Difficulties: – Content is hard to archive or shareContent is hard to archive or share– Busy in note taking, instead of spending time Busy in note taking, instead of spending time

sharing and absorbing ideas sharing and absorbing ideas Need tools to increase productivitiesNeed tools to increase productivities

– Don’t lose any sparksDon’t lose any sparks– Share with othersShare with others– Keep for later reviewKeep for later review– Integrate with your office documentIntegrate with your office document

Motivations (cont’d)

Projector Projector – A great tool for presentationA great tool for presentation– Not so convenient for discussionsNot so convenient for discussions

Whiteboard-Whiteboard-Camera SystemWhiteboard-Whiteboard-Camera System– Whiteboard: Projecting surface (Output) and Whiteboard: Projecting surface (Output) and

Writing surface (Input)Writing surface (Input)– Seamless integration of computer Seamless integration of computer

presentation with whiteboard discussionspresentation with whiteboard discussions– Enable remote collaboration on a shared Enable remote collaboration on a shared

workspaceworkspace

Whiteboard Scenarios

Note taking with a high-res digital cameraNote taking with a high-res digital camera– Whiteboard It!Whiteboard It!

Note taking with a low-res PC cameraNote taking with a low-res PC camera– Whiteboard ScanningWhiteboard Scanning

Real-time collaborationReal-time collaboration– Live WhiteboardLive Whiteboard

Meeting (WB + Audio / Video) ArchivingMeeting (WB + Audio / Video) Archiving– Whiteboard ArchivingWhiteboard Archiving

♥ Whiteboard It! ♥

Convert WB Content into an Electronic DocumentConvert WB Content into an Electronic Document

Overview of the System

♥ Whiteboard Scanning ♥

Get a High-res Scan With an Inexpensive Get a High-res Scan With an Inexpensive PC CameraPC Camera

How to do it?How to do it?– Take multiple pictures with overlapTake multiple pictures with overlap– Stitch them Stitch them automaticallyautomatically into a high-res image into a high-res image– Image enhancement (Image enhancement (Whiteboard It!Whiteboard It!))

System Overview

Example

+

+

Document Scanning w/ Low-Res camera

♥ Live Whiteboard ♥

Real-time distributed collaborationReal-time distributed collaboration

Whiteboard

DisplayMic (array)

Table Projector

Speaker

PC

Meeting Room/Office

Windows Communications Desktop

TabletPC

TabletPC Wire or wireless

Camera

Offices

A typical image sequence

Segmentation of the person and WB background is neededSegmentation of the person and WB background is needed

Demo

Integration with Windows MessengerIntegration with Windows Messenger

Whiteboard update

Annotations

Windows Messenger

Windows MessengerT120

Local WB client Remote WB client

USB 2

Video

Real-Time WB Processing

Video

Whiteboard changes

♥ Whiteboard Archiving ♥

Capture whiteboard content + audio/Video Capture whiteboard content + audio/Video – high-resolution digital still camera high-resolution digital still camera

Produce key framesProduce key frames– A KF usually corresponds to a major topicA KF usually corresponds to a major topic– Print as notes, or cut & paste into documents Print as notes, or cut & paste into documents

Record time stamps of each strokeRecord time stamps of each stroke Efficient meeting browsingEfficient meeting browsing

– Key frames to navigate between sectionsKey frames to navigate between sections– Strokes to bring up the audio at the moment Strokes to bring up the audio at the moment

when they were writtenwhen they were written

A Typical System Installation

Example: Sample Images

Key Frame Extraction

Number of strokesNumber of strokes

0

100

200

300

400

500

600

700

800

900

1000

1 21 41 61 81 101 121 141 161 181

Chapter 1 Chapter 2

Key Frame 1

Key Frame 2

Browsing Interface: Demo

Key Frame Thumbnails

Future Strokes

Current Strokes

Raw Image

VCR & Timeline Control

♥ Projector-Whiteboard-Camera System ♥

Communi-cation

Network

Video Mixer

Presentation

Video Mixer

P

Visual Echo Cancellation

Wri t i

ngs

A

A AnnotationA

P P

P

W

W

Local Meeting Room

Remote Room

Whiteboard

Camera

ProjectorDisplay

Remote Room

P+A

P+A+W

P+W

Visual Echo Cancellation

PrerequisitesPrerequisites– Geometric calibrationGeometric calibration– Color calibrationColor calibration

recover writings W

albedo estimation

& color

clustering

Color Lookup Table(given by color calibration)

Color Mapping & Geometric Warping

estimated visual echo E

captured image I

Homography H (given by geometric calibration)

VEC Results

Conclusions

Tools to use whiteboard and projectorTools to use whiteboard and projector Whiteboard It!Whiteboard It! for anyone who has a for anyone who has a

digital camera and it is easy to usedigital camera and it is easy to use Whiteboard ScanningWhiteboard Scanning for anyone who for anyone who

has a PC camera and it is easy to usehas a PC camera and it is easy to use Whiteboard ArchivingWhiteboard Archiving and and StreamingStreaming

for meeting rooms and collaboration for meeting rooms and collaboration Integration of Integration of ProjectorProjector and and WhiteboardWhiteboard

for seamless remote collaborationfor seamless remote collaboration

Acknowledgements

Thanks to Li-wei He, Zicheng Liu & Thanks to Li-wei He, Zicheng Liu & Hanning Zhou for contributionsHanning Zhou for contributions