building a great web api - qcon san francisco · twilio cloud communications building a great web...
TRANSCRIPT
![Page 1: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/1.jpg)
twilioCLOUD COMMUNICATIONS
BUILDING A
GREAT WEB APINOVEMBER 17, 2011, QCON
EVAN COOKE @emcookeCO-FOUNDER & CTO
HTTP
![Page 2: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/2.jpg)
One simple goalfor building great
Web APIs...
![Page 3: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/3.jpg)
5Minutes
![Page 4: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/4.jpg)
It should take no more than 5 minutes for a
developer to perform a useful task using your API for the first time
![Page 5: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/5.jpg)
What is a Web API?
• HTTP/HTTPS endpoint
• JSON/XML
• Hosted API/Web service
• Example: Amazon S3
Definition for today:
![Page 6: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/6.jpg)
![Page 7: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/7.jpg)
Twilio
Developer
End User
Carriers Inbound CallsOutbound Calls
Mobile/Browser VoIPVoice
SMS
PhoneNumbers
Send To/From Phone Numbers
Short Codes
Dynamically Buy Phone Numbers
Web service APIs to automate Voice and SMS phone communications
![Page 8: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/8.jpg)
![Page 9: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/9.jpg)
Why 5 Minutes?• Clear value proposition
• Fast signup
• Simple API
• Efficient quickstarts
• Concise accessible docs
• Easy authentication (basic vs. oauth)
• Debuggable
![Page 10: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/10.jpg)
API Adoption Funnel
Hear about API Try it
Try it
Build it
Production Use Buy it
Traction
Good APIsPromoteAdoption(tell friends)
![Page 11: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/11.jpg)
API Adoption Funnel
Hear about API 5-MinuteGoal
Traction
Try it
Try it
Build it
Buy itProduction Use
![Page 12: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/12.jpg)
5-minute APIsAPI Design
![Page 13: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/13.jpg)
5-minute APIsAPI Design
Counter Example
![Page 14: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/14.jpg)
• HTTP API to analyze large (MBs) media files
• High tps ~100,000/day
• POST data to the API
• API synchronously returns analysis of media (could be minutes/hours later)
Media processing API
You APIMedia
Analysis
![Page 15: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/15.jpg)
• Control and data in the same request (100 MB every request)
• Unclear error conditions. Can you resend request?
• Synchronously wait for response
• No request history/billing information
Media processing API
You API???
![Page 16: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/16.jpg)
Media processing API
POST http://api.vendor.com/API/Process?key=2hkh&mode=bob&filter=yeah
Body is 100MB of binary data
1 Original Request
ProblemCan we safely retry the request on
a transient failure?
![Page 17: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/17.jpg)
Media processing API
POST http://api.vendor.com/API/Process?key=2hkh&mode=bob&filter=yeah&token=TK123
Body is 100MB of binary data
Add a token allowing us to safely retry2
ProblemExposed to transient failures if the processing takes minutes/hours?
![Page 18: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/18.jpg)
Media processing API
POST http://api.vendor.com/API/Process?key=2hkh&mode=bob&filter=yeah&token=TK123&cbUrl=http%3A%2F%2Fmyserver.com%2Fresponse
Body is 100MB of binary data
Add webhook url to asynchronously respond3
ProblemRequest is huge and fragile when
data is included with control cmd?
![Page 19: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/19.jpg)
Media processing API
POST http://api.vendor.com/API/ProcessBody
key=2hkhmode=bobfilter=yeahtoken=TK123cbUrl=http://myserver.com/responsemediaUrl=http://s3.com/media.mov
Async fetch media & move POST params to body4
![Page 20: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/20.jpg)
Media processing API
POST http://api.vendor.com/v1/MediaBody
key=2hkhmode=bobfilter=yeahtoken=TK123cbUrl=http://myserver.com/responsemediaUrl=http://s3.com/media.mov
ResponseURI http://api.vendor.com/v1/Media/MD123
Version API and make URL more RESTful5
![Page 21: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/21.jpg)
5-minute APIsAPI Design
![Page 22: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/22.jpg)
Idempotency• Idempotence is the property of certain
operations that they can be applied multiple times without changing the result.
• Request failures happen. Provide users a safe way to retry requests
• Example:
POST /BankAccount/Funds
{‘value’: 1000, ‘token’: ‘TX123’}
![Page 23: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/23.jpg)
![Page 24: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/24.jpg)
Self-documentation• Sweat what and how you name. Naming is
is the best documentation.
• Example:
GET /Users/ID123GET /Users/ID123/FriendsGET /Users/ID123/Photos
![Page 25: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/25.jpg)
![Page 26: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/26.jpg)
RESTfulness• Adherence to REST object model and
verbs provides a clean way to expose business logic
• Create
• Fetch
• Modify
• Remove
POST /Users
GET /Users/ID123
PUT/POST /Users/ID123
DELETE /Users/ID123
![Page 27: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/27.jpg)
![Page 28: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/28.jpg)
Versioning• Do it. Remember you will be stuck
supporting old API versions (indefinitely?)
• Examples
GET /api/v1/blagGET /api/20101206/blag
GET /api/1/blag
![Page 29: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/29.jpg)
![Page 30: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/30.jpg)
Statefulness• When possible, offload the work of keeping
state/history because it’s hard!
• Examples
POST /CallsGET /Calls/CA123GET /Calls/CA123/Recordings/RE123
![Page 31: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/31.jpg)
![Page 32: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/32.jpg)
Predictability• The API should do what users expect
• Examples
<Play>music.mp3</Play>
<Play>music.aiff</Play>
![Page 33: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/33.jpg)
![Page 34: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/34.jpg)
Responsiveness• When a response is available immediately,
use a synchronous response. Otherwise, consider an asynchronous webhook rather then polling
• Examples:
POST /Object/TransformsBody cbUrl=http://b.com/response
![Page 35: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/35.jpg)
![Page 36: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/36.jpg)
• Idempotency
• Self-documentation
• RESTfulness
• Versioning
• Statefulness
• Predictability
• Responsiveness
Key Ideas
![Page 37: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/37.jpg)
5-minute APIs“Good”
![Page 38: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/38.jpg)
What makes a good API?
• Easy to Learn
• Easy to use (even without documentation)
• Hard to Misuse
• Easy to read and maintain code that uses it
• Sufficiently powerful to satisfy requirements
• Easy to extend
• Appropriate to audienceHow to Design a Good API and Why it MattersJoshua Bloch, Google
![Page 39: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/39.jpg)
Human-centric
![Page 40: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/40.jpg)
![Page 41: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/41.jpg)
Saleforceauth Campaigns from
Adword API
![Page 42: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/42.jpg)
Dynamic phone numberprovisioning via Twilio API
![Page 43: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/43.jpg)
Saleforce CRM data
Google Maps API
Saleforce app APIs
Twilio Client API
![Page 44: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/44.jpg)
![Page 45: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/45.jpg)
Simplicity
![Page 46: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/46.jpg)
twilioCLOUD COMMUNICATIONS
DEPLOYING YOUR WEB API
HTTP
![Page 47: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/47.jpg)
Infrastructure• 100’s of prod hosts in continuous operation
• 80+ service types running in prod
- Python(Twisted/GEvent), PHP, Java
• 50+ prod database servers
- MySQL, Redis
• Prod deployments several times/day across 7 engineering teams
![Page 48: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/48.jpg)
1000x
WebsiteContent
CMS
100x
WebsiteCode
PHP/Rubyetc.
10x
RESTAPI
Python/Javaetc.
1x
Big DBSchema
SQL
Log
Scal
e
DeploymentFrequency(Risk)
4 buckets
![Page 49: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/49.jpg)
WebsiteContent
One Click
WebsiteCode
One ClickCI Tests
RESTAPI
One Click
CI TestsHuman Sign-off
Big DBSchema
Human Assisted Click
CI TestsHuman Sign-off
DeploymentProcesses
![Page 50: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/50.jpg)
Cluster automation via boxconfig
![Page 51: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/51.jpg)
Cluster automation via boxconfig
• Build and deployment system - boot entire Twilio stack with one key press
• Host configuration - versioned code & config
• Host orchestration - load balancing
• Monitoring and alerting - nagios
• Multi-datacenter deployment & analytics
![Page 52: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/52.jpg)
Cluster automation via boxconfig
Boxconfig
Vanilla Linux Host(cloud/colo)
Provision
Base (AMI)
SVN/git
role role role Start Roles
S3Fetch
![Page 53: BUILDING A GREAT WEB API - QCon San Francisco · twilio cloud communications building a great web api november 17, 2011, qcon evan cooke @emcooke co-founder & cto h t t p](https://reader030.vdocuments.net/reader030/viewer/2022041211/5dd0c10bd6be591ccb62885c/html5/thumbnails/53.jpg)
twilioEvan Cooke@emcooke