go on gae (go israel meetup)
TRANSCRIPT
Go Israel First Meetup of 2015
Go on GAE
Or Hiltch / CTO Streamrail
Intro
• How we are using GAE• GAE PaaS in a nutshell• Go specific GAE goodness• Dev and (sort of) CI process on GAE - Small Live Demo• Pitfalls• My wisthlist for GAE• Q&A
How we are using GAE
• whoami• StreamRail develops client side SDKs for mobile and web that
accelerate content• We do cool P2P shit, neat graph theory stuff (web & native
mobile, so no Go here, yet)• Clients are ad networks and publishers that want to get a speed
boost• Our backend is 100% Go, but most of it is not actually GAE
GAE - PaaS in a nutshell
• Exists since 2008, used by SnapChat, Khan Academy (python), Secret (go) and lots of others
• Strict PaaS Philosophy (sometimes good, sometimes bad)• Service for everything, don’t care about under the hood -
Memcache, Cron, DB, Task queue, Blobstore, Auth (OpenID, OAuth) and more
• Cannot do outside the box stuff. If you can’t do it inside the box (yet), don’t use it
• Easy sandbox model (app containers gets shipped and scheduled on instances with autoscaling), strong vendor lock-in (logging, sending emails, and even performing http requests is via a GAE API).
GAE - PaaS in a nutshell
• Almost free to begin with (both $ and time wise)• Not just the dev, you can achieve CI with GAE with ease• Some PaaS providers don’t get it right (AWS Beanstalk-
configure ElastiCache - private security groups, VPCs, subnets, cache nodes, etc,
• That part of your app that needs to “just work”, not your main thing (your product is an app for web or mobile, not something like a search engine)
How does a GAE Go app look like?
• app.yaml (app name, version, routes, runtime, etc)• cron.yaml (cron jobs schedules)• crossdomain.xml (CORS settings)
How does a GAE Go app look like?• We like to work with GitHub as source control• github.com/streamrail/appengine holds all packages related to
GAE• We like to reflect the package paths on every dev’s file system
toogithub.com├── streamrail│ ├── appengine│ │ ├── admin│ │ ├── analytics│ │ ├── build│ │ ├── go-bitly-gae│ │ ├── go-gae-bigquery│ │ ├── main│ │ ├── qa│ │ ├── resource│ │ ├── swarm│ │ ├── twilio-gae│ │ ├── util│ │ └── website
• action needs to be performed during the request - parallelize (note memcache and datastore APIs)
Concurrency Example
• RAM:Highest speed - O(1µs)Purged most frequently, not horizontal (per instance cache)May overflow instance memory (can’t store gigs)
• Memcache:High speed - O(1ms)Horizontal , Limited to 1 MB per objectMay be purged without noticeUse large dedicated memcache! (more $$$ but much better)
• DatastoreSlowest of the bunch - O(20ms)HorizontalPersistent (not purged)
A Word About Caching on GAE
• Don’t forget - GAE will kill outstanding API calls once req returned
Control API Calls
• Use the delay package to defer work that can be deferred
Deferring Work
Continuous Integration & Delivery
• GAE does not use Git publishing like Heroku, but it’s easily achieved if you have a CI server
• Simple “goapp deploy” to deploy a version• “goapp test” - like go test, but establishes almost complete GAE
test env on CI server / dev machine (inc. http, memcache, datastore, etc.)
• Create subversions of your app and traffic split before rolling out
• Easy rollback in one click• We have connected GAE to Circle CI to test and deploy
Circle.yaml file
Traffic Splitting
Live Streaming to BQ• Prev solution (downloading logs/datastore using Remote API and
inserting to DB) failed• No need to store (and maintain!) the logs anywhere (GCS, S3,
etc.)• Apache logs exist anyway as backup on GAE• Near real time data stream (in-memory buffering is used to
reduce contention)• BQ awesome, connects to Google Spreadsheets• Useful packages we’ve written for this:
https://github.com/streamrail/go-gae-remote https://github.com/streamrail/go-gae-bigqueryhttps://github.com/streamrail/bq-schema https://github.com/streamrail/gapps-bqutil
Streaming from a handler
Validating schema
Binding BQ to Spreadsheets
Pitfalls & Bad Use Cases
“If you do nothing, you can scale infinitely” - Scott Hanselman• Long-lived connections• Serving static content, CDN style• No WebSockets solution (Channel API crap)• Datastore is annoying• No SSL for versioned apps• Limited caching options
General Pitfalls
• GOMAXPROCS=1 is set (force one “active” thread)• Go GAE instance allows 10 concurrent “blocking” (non-active)
threads• Not very cost effective• Sandboxed - no C imports etc.
• Firebase integration, reasonable pricing• More features (long lived connections, sticky sessions, web
sockets, wider selection of DB and caching)• Better performance (global distribution, multithreading for Go)• Git publishing (heroku style!)• Better out of the box monitoring (alerts, schematic logging)
My Wishlist for GAE
