webrtc hacks: lessons learned

PowerPoint Presentation

WebRTC Hacks: Lessons LearnedTwilio Signal 2016

What would Macguyver do..with WebRTC?

Chad Wallace Hart Consulting

https://c1.staticflickr.com/1/658/21755244141_c6e6826585_b.jpg1

Hi!CHAD HARTIndependent Consultant, Chief Editor, webrtcHacks

https://webrtcHacks.com@webrtcHacks

@[email protected]

WebRTC Hacks - Twilio Signal 2016

#May 25, 2016


I am an independent analyst, product management & marketing consultant, and advisor focusing on RTC, especially WebRTC.Personally, I care a lot about where technology and products are headed. I focus at lot on trying to understand the future and luckily I was able to do this often career wise as an industry analyst, doing business intelligence, and strategic product planning.

I also run a blog for webrtc developers webrtcHacks. It was great to see so many WebRTC hacks during the hackathon. It is amazing to see how far WebRTC has come, even on my own blog.webrtcHacks is pretty specific about being for developers our content is generally pretty technical and not something you would care about unless you are at least involved in WebRTC development. We have grown from a few thousand pageviews a month in 2013 to 30 to 40,000 now.

2

May 25, 2016WebRTC Hacks - Twilio Signal 20163


Lots of definitions of Hack3



This is probably my favorite. Quick or inelegant solution 4



Hack #1: Motion Detecting Baby Monitor

WebRTC Hacks - Twilio Signal 2016https://webrtchacks.com/baby-motion-detector/

#May 25, 2016


September 2013

My first hack6

Detecting motion

WebRTC Hacks - Twilio Signal 2016https://github.com/ReallyGood/js-motion-detection

#May 25, 2016


Using the HTML5 Canvas to detect motion

7

Hack #1: Motion Detecting Baby MonitorWebRTC Hacks - Twilio Signal 2016https://webrtchacks.com/baby-motion-detector/

#May 25, 2016


September 2013

My first hack8

Lessons LearnedSDKs & cloud services make WebRTC super simpleUse HTML5 Canvas on your media streamYou dont need to know what youre doingWebRTC Hacks - Twilio Signal 20169

#May 25, 2016


September 20139

Hack #2: Build your own phone company with WebRTC and a weekend

WebRTC Hacks - Twilio Signal 2016https://webrtchacks.com/own-phoneco-with-webrtc/

#May 25, 2016


January 7, 2014

What would it take to make my own personal VoIP service10

Build your own WebRTC ServerBack-end StackServer LogicWebRTC Hacks - Twilio Signal 2016

#May 25, 2016


EasyRTC for WebRTC signalingExpress & NodeAWS Ubuntu image from Bitnami

11

Lessons Learned: Building out a Back-endSignaling: easier than expected Cloud setup wasnt so badDont skip SSL for HTTPSDont skip TURN serverDont get detached from UX


#May 25, 2016


12

Hack #3: How to train a dog with JavaScriptWebRTC Hacks - Twilio Signal 2016

https://webrtchacks.com/javascript-dog-trainer/

#May 25, 2016


November 201413

WebRTC Hacks - Twilio Signal 2016This is Levy

#May 25, 2016


I had a bull dogHe liked to sleep on the couch14

WebRTC Hacks - Twilio Signal 2016Levy liked our old couch

#May 25, 2016



We got a new couch.Levy wasnt welcome here.

#May 25, 2016


But them we bought a new couchSleeping on the couch was a no-no

16


Levy didnt care

#May 25, 2016


Fun with my kids toys (& IoT)


#May 25, 2016


Tessel.ioElenco Snap Circuits rover

18

Quick proof-of-concept

WebRTC Hacks - Twilio Signal 2016Could I mix WebRTC & IoT to train my dog?

#May 25, 2016


WebRTC Hacks - Twilio Signal 2016Not fun: getting organizedSignaling ServerWeb server IoT deviceBrowser-based monitor appBrowser-based viewing appMMS API

#May 25, 2016


Lots of nodesSignaling server from OSS Simple WebRTCNode for my web servertessel-based rover also running nowLight weight monitor app that does motion detectionBrowser app that does recordingTwilio for MMS20

Fun: seeing it workWebRTC Hacks - Twilio Signal 2016

https://webrtchacks.com/javascript-dog-trainer/MMS Alert

#May 25, 2016


Fun: seeing it workWebRTC Hacks - Twilio Signal 2016https://webrtchacks.com/javascript-dog-trainer/

#May 25, 2016


Lessons LearnedNew features for free: recording MMS is easier than writing an app to get mobile notificationsApple makes things difficultMore things means more opportunities for short RTC sessionsWebRTC Hacks - Twilio Signal 2016

#May 25, 2016


23

Hack 4: ThirdEye: WebRTC + Computer vision


#May 25, 2016


June 201524

Start boring..


#May 25, 2016


Adding computer visionWebRTC Hacks - Twilio Signal 2016

#May 25, 2016


26


#May 25, 2016


Solving real problems


#May 25, 2016


Pretend developers can win

#May 25, 2016


Other lessons learnedGathering context from real time streams isnt so hardDont make the signaling harder than it needs to beSomeone really needs to make a wifi WebRTC cameraWebRTC Hacks - Twilio Signal 2016

#May 25, 2016


Thinking about machines & RTC 2015 Chad Wallace Hart ConsultingPerson to person calling because of machinesMachine makes a call when something happensSend machine real time audio/video for processingMachines send other machines audio/video for real time processingWhole new categories of interactions that will drive RTC growth

#May 25, 2016


P2P: something happens get someone on the phone to talk about itM2P: dog trainer example where you get calledP2M: computer vision example where you just want to process the streamLooking forward to the day when AI bots talk to other AI bots using real time voice & video31

My Next Hack: Adding WebRTC to Toys?


#May 25, 2016


Raspberry Pi

32


#May 25, 2016


33


WebRTC hacks in the wild


https://upload.wikimedia.org/wikipedia/commons/9/90/Flickr_-_brewbooks_-_Palm_Jungle_-_Hawaii_Tropical_Botanical_Garden.jpg34

The browser as a real time media processorCanvasWeb AudioWebRTC Hacks - Twilio Signal 2016

http://www.html5rocks.com/en/tutorials/webaudio/games/http://diveinto.html5doctor.com/canvas.htmlMediaStream = canvas.captureStream(framerate)var microphone = context.createMediaStreamSource(stream);var peer = context.createMediaStreamDestination();

#May 25, 2016


Media stream = canvas.captureStream(framerate)https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/webrtc-integration.html35

Hacks that are new: make your own conferencing server

WebRTC Hacks - Twilio Signal 2016https://webrtchacks.com/web-audio-conference/

#May 25, 2016


Hacks we dont like


https://webrtchacks.com/dear-ny-times/

#May 25, 2016



http://randomwalker.info/publications/OpenWPM_1_million_site_tracking_measurement.pdf

#May 25, 2016


http://randomwalker.info/publications/OpenWPM_1_million_site_tracking_measurement.pdf

This is a real problem with more than 700 of the top 1M sites using WebRTC trackingStill small vs. other forms38

Hacking a hack the WebRTC notifierWebRTC Hacks - Twilio Signal 2016

https://webrtchacks.com/webrtc-notify/

#May 25, 2016


Han-Ka39

Production Hack: skip the P2P


https://webrtchacks.com/slack-webrtc-slacking/

#May 25, 2016


Retrospective: long, but not impossible jump from a hack to realitySecurityTURN serviceElasticityRedundancyVersioningComplianceMulti-partyMonitoringActive testSecurityiOSAndroidInternet ExplorerSafariUX timingBandwidth checksDevice checksFailure recoveryWebRTC Hacks - Twilio Signal 2016Server-sideClient-side

#May 25, 2016



WebRTC!https://upload.wikimedia.org/wikipedia/commons/2/26/USAF_EOD_explosion.jpg


https://upload.wikimedia.org/wikipedia/commons/2/26/USAF_EOD_explosion.jpg42