MP3: Multi-view Surveillance System

Instructor: Klara NahrstedtApril 20, 2012

CS414

MP3: Multi-view Surveillance System

Window for server 1

Window for server 2

• Server– Capture Audio and Video at a fixed rate– Video: 30 fps, Audio: 8000Hz

• Client– Requests for Video and/or Audio– Works in Two modes: Active Mode and Passive Mode

• Active Mode: Media type: Audio, VideoVideo Rate: 15 to 25 fpsAudio Rate: 8000HzVideo Resolution: 640X480

• Passive Mode: Media type: VideoVideo Rate: 10 fpsVideo Resolution: 320X240

Client Behavior• Client can request active mode from Server 1

and passive mode from Server 2• Client can request active mode from Server 2

and passive mode from Server 1• Client can request active mode from both

Server 1 and Server 2• Client can request passive mode from both

Server 1 and Server 2

Client User Interface: Example

SWITCH SWITCH

Video window for Server 1

Connect Edit

Video window for Server 2

Computation window for Server 2Computation window for Server 1

Resource Admission at Client

• User defines streaming MODE: active or passive• Client performs resource admission

Server 1 Client

Control Channel

MODE

User

Resource Admission at Client

• Client– Available Application Bandwidth ABN

– Application Frame Size, MN = ?

– Application Frame Rate, RN = ?– Audio Bandwidth = 8000 * 16– Request Bandwidth• Active Mode: BN = (MN * RN) + 8000 * 16

• Passive Mode: BN = (MN * RN) = (MN * 10)

resource.txt

Optimistic Allocation

8000Hz Audio Signal

How to define RN for active mode?

fps: 10

[fps:15-25]

Resource Admission at Client

• Client– Available Application Bandwidth ABN

– Application Frame Size, MN = ?

– Application Frame Rate, RN = ?– Request Bandwidth

• Active Mode: BN = (MN * RN) + Audio Bandwidth

resource.txt

Show your computation on the Screen

ABN = (MN*RN) + Audio Bandwidthif RN > 25, RN = 25if RN < 15, REJECT requestelse RN

Resource Negotiation with Server

• Information sent to server includes– Requested Frame Rate (FPS) for video, – Video Resolution, and – Media type (audio, video)

Server 1 Client

FPS, ResolutionMedia types

MODE

Resource Admission at Server

• Server– Available Application Bandwidth ABN – Application Frame Size, MN = ?– Application Frame Rate, RN = [received from client]– Audio Bandwidth = 8000 * 16– Request Bandwidth

• Active Mode: BN = (MN * RN) + 8000 * 16• Passive Mode: BN = (MN * RN) = (MN * 10)

– Admitted if BN <= ABN

– Else Rejected

resource.txt

Optimistic Allocation

Video Data Flow Architecture

RTP

Pack

et

Rate Control

Network

Video Source Video Filter Media Encoder Video Muxer App Sink

RTP

Pack

et App SourceDecode Bin2Video Sink

Synchronization

Payload

Media Type Timestamp RTP headerID

UDP Packet Payload

Segment

Merge

Server-Side

Client-Side

Rate Control: QoS Enforcement

• You can build leaky bucket• You can implement token bucket• You can use gstreamer rate control• You can implement your own

method

Sleep (100) => 10 fps

You can simply read using a loopWhile(true){ SendPacket(); Thread.sleep(100);}

Resource Reservation: Client

– Available Application Bandwidth ABN

– Used Bandwidth for Server 1 B1

– Available Bandwidth ABN = ABN –B1

– Admission is successful for B2 if B2 <= ABN

resource.txt

Server 1 60 Kbps

Server 2 35 KBps

Resource Table

Video Data Flow Architecture using RtpBin

RTP

Pack

et

Rate Control

Network

Video Source Video Filter Media Encoder Video Muxer App Sink

RTP

Pack

et

App SourceDecode Bin2Video Sink

Synchronization

Segment

Merge

Server-Side

Client-Side

RtpBinUdpSinkUdpSrc

Implementation Using RtpBin @ Server

Video Source Video FilterTee Video Scale Scale Filter

Video Rate Rate Filter Video Encoder RTP Payload

RTP Bin

UDP Sink

Queue

UDP SinkUDP Source

RTCP

RTCP

RtpBin combines •RtpSession•RtpSsrcDemux•RtpPtDesmux•RtpJitterBuffer

Implementation Using RtpBin @ Client

Video SinkVideo DecoderRTP DePayload

RTP Bin

UDP Source

UDP SourceUDP Sink

RTCP

RTCP

rtpBin.connect(new Element.PAD_ADDED() {public void padAdded(Element element, Pad pad) {

if (pad.getCaps().toString().contains("video")) {//LINK TO VIDEO SINK via VIDEO MONITOR

} else if (pad.getCaps().toString().contains("audio")) {

// LINK TO AUDIO SINK via AUDIO MONITOR}}

});

Internals of RtpBin

http://gstreamer.freedesktop.org/documentation/rtp.html

Evaluation Process for MP3• Case I: – Server 1 is requested to send stream in ACTIVE mode– Server 2 is requested to send stream in PASSIVE mode– Client has resource to support 25 fps in ACTIVE mode

• Case II: – Server 1 is requested to send stream in ACTIVE mode– Server 2 is requested to send stream in PASSIVE mode– SWITCH streaming mode of Server 1 and Server 2– Client has resource to support 25 fps in ACTIVE mode

Evaluation Process for MP3• Case III: – Server 1 is requested to send streams in PASSIVE mode– Server 2 is requested to send streams in ACTIVE mode– Modify resource.txt so that ACTIVE mode use 15fps– Low interference in streaming

• Case IV: – Server 1 is requested to send streams in PASSIVE mode– Server 2 is requested to send streams in ACTIVE mode– SWITCH mode of Server 1 and Server 2– Client has resources to support 15fps in ACTIVE mode

Evaluation Process for MP3• Case V: – Server 1 is requested to send streams in ACTIVE mode– Server 2 is requested to send streams in PASSIVE mode– Modify resource.txt so that ACTIVE mode use 25fps– Low interference in streaming

• Case VI: – Server 1 is requested to send streams in ACTIVE mode– Server 2 is requested to send streams in PASSIVE mode– Update resource.txt so that client has enough available

bandwidth to support two ACTIVE modes– SWITCH mode of Server 2 into ACTIVE mode– By default, no audio playback from Server 2; however

client should receive audio streams from Server 2

Evaluation Process for MP3• Case VII: – Server 1 is requested to send streams in ACTIVE mode– Server 2 is requested to send streams in ACTIVE mode– MUTE Server 1 audio playback and UNMUTE Server 2

audio playback– No interference in streaming– Client has enough resource to support two ACTIVE mode

• Case VIII: – Server 1 is requested to send streams in ACTIVE mode– Server 2 is requested to send streams in ACTIVE mode– SWITCH mode of Server 1 into PASSIVE mode

Evaluation Process for MP3• Case IX: – Server 1 is requested to send streams in PASSIVE mode– Server 2 is requested to send streams in ACTIVE mode– Modify resource.txt so that admission for ACTIVE mode is

not successful– Streaming from Server 2 turns off, no interference in

Server 1 streaming

• Case X: – Server 1 is requested to streams send in ACTIVE mode– Modify resource.txt so that Server 2 can be requested in

ACTIVE mode with 15 fps– SWITCH between modes of Server 1 and Server 2

Evaluation Process of MP3• For All Cases:

– Cases are designed to run continuously one after another process without restarting the program or starting from the beginning

– Maintain audio-visual synchronization in ACTIVE mode– Show the computation of resource admission– Show the current frame rate, bandwidth usage, and

network jitter

Demonstration and Interview

• You must use at least 3 PCs/ laptops for the demo• If you want to use wireless/ wired with only your

laptops, please setup the demo in SC3401• If you want to use wired demo with lab PCs, please

setup your demo at SC0216• Demo date: April 30th • Demo time:– SC3401 @ 5pm – SC0216 @ 6pm

Additional Features for MP3?• It is highly recommended that you build an

additional feature in MP3 to SURPRISE US!• Bonus Point: 20• Bonus point will be awarded based on the

amount of surprise!