kranky geek 2015 - decisions & considerations in building your webrtc app
TRANSCRIPT
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
RUNNING RTCOperational lessons learned at Twilio.
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
TWILIO?
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
Communication Channels
T W I L I O C O M M U N I C AT I O N S P L AT F O R M
VO I C E S M S M M S
W E B M O B I L E
Your Business Applications
I P M ESS A G I N G V I D E O
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
750,000 Developers & Businesses
50 BILLION
API Interactions a year
O P E R AT I N G AT S C A L E
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
TWILIO AND WEBRTC?
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
2011 2012 2013 2014 Present
TWILIO CLIENT 1.1: WEBRTC
TWILIO CLIENT: WEBRTC CALL VOLUME
WEBRTC RELEASED IN CHROME 23
TWILIO CLIENT 1.0: FLASH
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
2011 2012 2013 2014 Present
TWILIO CLIENT 1.1: WEBRTC
TWILIO CLIENT 1.2: GLOBAL LOW LATENCY
TWILIO CLIENT: WEBRTC CALL VOLUME
WEBRTC RELEASED IN CHROME 23
WEBRTC RELEASED IN FIREFOX 22
TWILIO CLIENT 1.0: FLASH
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
2011 2012 2013 2014 Present
TWILIO CLIENT 1.1: WEBRTC
TWILIO CLIENT 1.2: GLOBAL LOW LATENCY
TWILIO CLIENT: WEBRTC CALL VOLUME
WEBRTC RELEASED IN CHROME 23
WEBRTC RELEASED IN FIREFOX 22
TWILIO CLIENT 1.0: FLASH
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
COMMON USE CASES
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
COMMON USE CASES
CRM and SaaS application integration
Call center / customer service
Sharing economy
Education / expertise on-‐demand
Recruiting
Anonymous calling
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
A HANDFUL OF OUR WEBRTC CUSTOMERS
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
WEBRTC COMPLEXITY DOES NOT INCREASE LINEARLY.
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
Signaling Server
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
Signaling Server
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
Signaling Server
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
Signaling Server
TURN Server
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
TURN Server
Signaling Server Media Server
Conference State
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
TURN Server
Signaling Server Media Server SIP Gateway
Conference State
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
TURN Server
Signaling Server SIP Gateway
Conference State
Media Server
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
TURN Server
Signaling Server SIP Gateway
Conference State
PSTN GatewayMedia Server
Carrier
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
TS SC PM CDC1
DC5
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
DC3
TS SC PM C
DC7
DC6DC1
DC5
DC2
DC4
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
SOMETHING WILL BREAK.
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
Stuff you can’t control
Random host failure
Network degradation
Spurious network traffic
Stuff you can control
Resource management
Operational mishaps
Bugs / crashes
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
Tens of failures / week.
100% up-‐time two quarters running.
99.95% up-‐time for the year.
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
TEST CONSTANTLY.
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
END-TO-END TESTS. CONTINUALLY TEST THE PRODUCT THE
SAME WAY THE CUSTOMER WILL.
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
END-TO-END TESTING
node-‐webrtcsignaling
media
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
END-TO-END TESTING
node-‐webrtcsignaling
media
Original approach:
-‐ Test app built using node-‐webrtc.
-‐ Acted like a full-‐fledged browser client.
-‐ Ran a suite of tests at high frequency (some tests multiple times / min, other tests every few minutes).
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
END-TO-END TESTING
node-‐webrtcsignaling
media
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
END-TO-END TESTING
node-‐webrtcsignaling
media
Hard to maintain:
-‐ Difficult to keep up-‐to-‐date with WebRTC changes.
-‐ Build challenges (CentOS 6 on AWS).
-‐ (node-‐webrtc has since removed MediaStream API)
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
END-TO-END TESTING
node signaling
mediarobocallerssignaling
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
END-TO-END TESTING
node signaling
media
Revised approach:
-‐ Frequent testing for service monitoring through simple node websocket client.
-‐ Automated browser regression testing and “thick” scenario testing with robocallers.
robocallerssignaling
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
Rollbar
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
END-TO-END TESTING
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
END-TO-END TESTING
Manual testing tools:
-‐ Automated tests won’t catch everything.
-‐ Simple, reliably manual testing tools for rapidly running standard scenarios are invaluable.
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
End-‐to-‐end tests
Anomaly detection
Host monitoring (Nagios)
Manual tools
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
End-‐to-‐end tests
Anomaly detection
Host monitoring (Nagios)
Manual tools
Layered monitoring
-‐ Event aggregated in Rollbar (and assorted other tools).
-‐ Alerts delivered through PagerDuty.
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
PLAN FOR FAILURE, PLAN FOR RECOVERY.
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
Load Balancer A Load Balancer B
Gateway A Gateway B Gateway C
DNS distributionwss://www.mywebrtcservice.com
Registrar
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
Load Balancer A Load Balancer B
Gateway A Gateway B Gateway C
Client
Registrar
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
Load Balancer A Load Balancer B
Gateway A Gateway B Gateway C
Client
Registrar
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
Load Balancer A Load Balancer B
Gateway A Gateway B Gateway C
Client
Registrar
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
x Many
Load Balancer A Load Balancer B
Gateway A Gateway B Gateway C
Client
Registrar
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
x 10,000,000
Load Balancer A Load Balancer B
Gateway A Gateway B Gateway C
Client
Registrar
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
x 10,000,000
Load Balancer A Load Balancer B
Gateway A Gateway B Gateway C
Client
Registrar
Rate limit here to protect downstream services
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
x 10,000,000
Load Balancer A Load Balancer B
Gateway A Gateway B Gateway C
Client
Registrar
Be careful about locking here
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
EXPECT LOUSY NETWORKS.
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
Network Link Conditioner
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
https://github.com/vadymy/networkThrottler
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
.GETSTATS() IS YOUR FRIEND.
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
pc.getStats()
AMAZON KINESIS
MONITORING
REDSHIFT
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
SOME OF THE STATS WE CAPTURE…
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
audioInputLevel
audioOutputLevel
browser
jitter
rtt
received_packets
sent_packets
received_packets_lost
SOME OF THE STATS WE CAPTURE…
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
SOME OF THE PATTERNS WE LOOK FOR…
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
High percentage of packets lost
Excessive jitter or rtt
Calls > 5 seconds with no packets sent but > 0 audio-‐input level (or the inverse; evidence of one-‐way audio)
…look for patterns by Twilio account, endpoint identity,
browser version, etc.
SOME OF THE PATTERNS WE LOOK FOR…
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
OPERATIONAL EXCELLENCE IS A CULTURE ISSUE.
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
SOME CULTURAL TOOLS
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
SOME CULTURAL TOOLS
Write the SLA first
Build it, operate it
Five-‐Whys Analysis
Fire Drills
Runbooks
Post-‐Mortems
Time to Recovery Reports
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
COMMUNICATIONS TOOLS ARE ONLY AS GOOD AS THEIR AVAILABILITY.
© 2009-2015 TWILIO, INC. ALL RIGHTS [email protected] PRODUCT MANAGER, SDKSROB BRAZIER
ROB BRAZIER [email protected]