architecting a cloud native internet archive
TRANSCRIPT
© 2016 NodeSource1
© 2016 NodeSource2
© 2016 NodeSource3
Welcome
Ross Kukulinski
Currently PM @ NodeSource
Node.js Evangelism WG
Introduction to CoreOS - O’Reilly Media
@RossKukulinski on Twitter/GitHub
© 2016 NodeSource4
What does “Cloud Native” mean?
© 2016 NodeSource
Cloud Native
5
At its root, Cloud Native is structuring teams, culture and technology to utilize
automation and architectures to manage complexity and unlock velocity.
Joe Beda - @jbeda
© 2016 NodeSource6
© 2016 NodeSource7
© 2016 NodeSource8
© 2016 NodeSource9
© 2016 NodeSource
Internet Archive
10
S I M P L E D E S I G N G O A L S • The system should use only commodity equipment • The system should not rely on commercial software • The system should not require a PhD to implement or maintain • The system should be as simple as possible
© 2016 NodeSource11
© 2016 NodeSource12
© 2016 NodeSource
Buzzword Bingo!
14
Current implementation • React & webpack
• Public-facing REST API (Swagger Documentation)
• Node.js microservice architecture
• Message queues for internal APIs
• 100% Containerized from day 1 - Docker & Kubernetes
• Cloud Storage APIs
• RethinkDB
• CI/CD using CircleCI
© 2016 NodeSource15
© 2016 NodeSource
Lessons Learned
16
• It’s fun to experiment recklessly
• Kubernetes is great for developers
• Design docs for discussion & reference materials
• Ship early & ship often
• Containers scale independent of your cloud provider
• Microservices shift complexity around, don’t resolve it
• Documenting & versioning streaming APIs is difficult
© 2016 NodeSource
Open source tooling
17
• Containerizing stateful services is hard!
• github.com/rosskukulinski/kubernetes-rethinkdb-cluster
• It’s ok to have opinions
• npmjs.com/package/k8s-scripts
© 2016 NodeSource
Future
18
What’s next?
• More tests… always more tests…
• Ephemeral environments for PR testing
• Language agnostic message bus
• Replace Redis with NATS.io
• Streaming API documentation & validation
© 2016 NodeSource19
© 2016 NodeSource
Resources
20
Jaffe, E. & Kirkpatrick, S. (2009). Architecture of the internet archive. Proceeedings of SYSTOR 2009: The Israeli Experimental Systems Conference on - SYSTOR ’09. doi:10.1145/1534530.1534545
http://web.archive.org/web/20140828081906/http://iwaw.europarchive.org/07/IWAW2007_tofel.pdf
http://archive-access.sourceforge.net/projects/wayback/
http://www.motherjones.com/media/2014/05/internet-archive-wayback-machine-brewster-kahle
https://blog.archive.org
https://sourceforge.net/projects/archive-crawler
