one billion notes as 'small data' (dave engberg)
TRANSCRIPT
One Billion Notes as ‘Small Data’
Dave Engberg, CTO
Evernote Overview
Launched in 2008
“Freemium” subscription service
Mobile and desktop
33.3 million registered accounts
1.12 billion notes
1.6 billion unique attachments
Evernote Product Family
Food
Clearly
Peek
Hello
Skitch
Penultimate
Worldwide Reach
Last 30 days
Accounts created 1.9 million
Distinct users 11 million
Active clients 1337
New notes 77 million
Note edits 152 million
HTTP(S) requests 14 billion
Image OCR 90 million
Architecture philosophy
Only optimize for your application’stop 1-2 challenges
Evernote CPU
Evernote service requirements
ComponentComponent SizeSize Avg LoadAvg Load Peak LoadPeak Load
CPU Low Medium
Bandwidth
Latency
File storage
Metadata
Network load
Smooth traffic (no bursts)
Nearly symmetric
Sync clients: not latency-sensitive
100 TB in, 180 TB out per month
Evernote service requirements
ComponentComponent SizeSize Avg LoadAvg Load Peak LoadPeak Load
CPU Low Medium
Bandwidth Medium Medium
Latency Low Low
File storage
Metadata
File Storage
2.8 billion attachments
1.6 billion unique files
374 TB, de-duplicated
Effectively permanent
High redundancy required (stored 3x)
Evernote service requirements
ComponentComponent SizeSize Avg LoadAvg Load Peak LoadPeak Load
CPU Low Medium
Bandwidth Medium Medium
Latency Low Low
File storage High Low Medium
Metadata
Metadata
Strong ACID transactional DB < 10 TB Peak riops: 350 Peak wiops: 50
Near-realtime search < 10 TB Peak riops: 800 Peak wiops: 500
Evernote service requirements
ComponentComponent SizeSize Avg LoadAvg Load Peak LoadPeak Load
CPU Low Medium
Bandwidth Medium Medium
Latency Low Low
File storage High Low Medium
Metadata Low Medium High
Sharded architecture
Hardware:SuperMicro 1U2x L5630 CPU96 GB RAM6x 300GB Intel SSDLSI RAID 5 (+spare)~$8,000
Hardware:SuperMicro 4U1x L5630 CPU12 GB RAM24x 3TB (Seagate)LSI RAID 6 (x3)~$12,000
Software:TomcatJava 6MySQL 5.1DRBDXenDebian stable
Software:Apachemod_davDebian stable
265 shards
Around 400 Linux servers overall
Tiers of a cloud
Cloud Provider Strengths
Applications with bursts:- bandwidth- storage- compute
CPU-bound applications
Applications with low or fixed storage,low-medium iops
Fewer operations staff
Evernote’s Service
Consistent network usage
Consistent compute usage
File storage grows indefinitely: users * time
Random iops bound
CPU + Metadata comparison
Evernote shard
440 GB usable / VMsysbench: 5000 rwiops$8000
Over 4 years:$166/month
200,000 users/shard:$0.01/user/year
AWS EC2 + EBS
High-Memory 2XL:$300/month
Provisioned IOPS EBS volume (max 1000 iops):$155/month
Total: $455/month
File storage
Evernote WebDAV
WebDAV server, 54 TB$12,100
Over 4 years:$250/month$4.70/TB/month
With triple redundancy:$14/TB/month
AWS S3
$99/TB/month
Networking comparison
Evernote managed
$5/Mbps/month
~800Mbps peak
$4,000/month
AWS networking (out)
~$0.08/GB/month
180TB/month
$13,700/month