go on gae (go israel meetup)

Go Israel First Meetup of 2015

Go on GAE

Or Hiltch / CTO Streamrail

Intro

• How we are using GAE• GAE PaaS in a nutshell• Go specific GAE goodness• Dev and (sort of) CI process on GAE - Small Live Demo• Pitfalls• My wisthlist for GAE• Q&A

How we are using GAE

• whoami• StreamRail develops client side SDKs for mobile and web that

accelerate content• We do cool P2P shit, neat graph theory stuff (web & native

mobile, so no Go here, yet)• Clients are ad networks and publishers that want to get a speed

boost• Our backend is 100% Go, but most of it is not actually GAE

GAE - PaaS in a nutshell

• Exists since 2008, used by SnapChat, Khan Academy (python), Secret (go) and lots of others

• Strict PaaS Philosophy (sometimes good, sometimes bad)• Service for everything, don’t care about under the hood -

Memcache, Cron, DB, Task queue, Blobstore, Auth (OpenID, OAuth) and more

• Cannot do outside the box stuff. If you can’t do it inside the box (yet), don’t use it

• Easy sandbox model (app containers gets shipped and scheduled on instances with autoscaling), strong vendor lock-in (logging, sending emails, and even performing http requests is via a GAE API).

GAE - PaaS in a nutshell

• Almost free to begin with (both $ and time wise)• Not just the dev, you can achieve CI with GAE with ease• Some PaaS providers don’t get it right (AWS Beanstalk-

configure ElastiCache - private security groups, VPCs, subnets, cache nodes, etc,

• That part of your app that needs to “just work”, not your main thing (your product is an app for web or mobile, not something like a search engine)

How does a GAE Go app look like?

• app.yaml (app name, version, routes, runtime, etc)• cron.yaml (cron jobs schedules)• crossdomain.xml (CORS settings)

How does a GAE Go app look like?• We like to work with GitHub as source control• github.com/streamrail/appengine holds all packages related to

GAE• We like to reflect the package paths on every dev’s file system

toogithub.com├── streamrail│   ├── appengine│   │   ├── admin│   │   ├── analytics│   │   ├── build│   │   ├── go-bitly-gae│   │   ├── go-gae-bigquery│   │   ├── main│   │   ├── qa│   │   ├── resource│   │   ├── swarm│   │   ├── twilio-gae│   │   ├── util│   │   └── website

• action needs to be performed during the request - parallelize (note memcache and datastore APIs)

Concurrency Example

• RAM:Highest speed - O(1µs)Purged most frequently, not horizontal (per instance cache)May overflow instance memory (can’t store gigs)

• Memcache:High speed - O(1ms)Horizontal , Limited to 1 MB per objectMay be purged without noticeUse large dedicated memcache! (more $$$ but much better)

• DatastoreSlowest of the bunch - O(20ms)HorizontalPersistent (not purged)

A Word About Caching on GAE

• Don’t forget - GAE will kill outstanding API calls once req returned

Control API Calls

• Use the delay package to defer work that can be deferred

Deferring Work

Continuous Integration & Delivery

• GAE does not use Git publishing like Heroku, but it’s easily achieved if you have a CI server

• Simple “goapp deploy” to deploy a version• “goapp test” - like go test, but establishes almost complete GAE

test env on CI server / dev machine (inc. http, memcache, datastore, etc.)

• Create subversions of your app and traffic split before rolling out

• Easy rollback in one click• We have connected GAE to Circle CI to test and deploy

Circle.yaml file

Traffic Splitting

Live Streaming to BQ• Prev solution (downloading logs/datastore using Remote API and

inserting to DB) failed• No need to store (and maintain!) the logs anywhere (GCS, S3,

etc.)• Apache logs exist anyway as backup on GAE• Near real time data stream (in-memory buffering is used to

reduce contention)• BQ awesome, connects to Google Spreadsheets• Useful packages we’ve written for this:

https://github.com/streamrail/go-gae-remote https://github.com/streamrail/go-gae-bigqueryhttps://github.com/streamrail/bq-schema https://github.com/streamrail/gapps-bqutil

Streaming from a handler

Validating schema

Binding BQ to Spreadsheets

Pitfalls & Bad Use Cases

“If you do nothing, you can scale infinitely” - Scott Hanselman• Long-lived connections• Serving static content, CDN style• No WebSockets solution (Channel API crap)• Datastore is annoying• No SSL for versioned apps• Limited caching options

General Pitfalls

• GOMAXPROCS=1 is set (force one “active” thread)• Go GAE instance allows 10 concurrent “blocking” (non-active)

threads• Not very cost effective• Sandboxed - no C imports etc.

• Firebase integration, reasonable pricing• More features (long lived connections, sticky sessions, web

sockets, wider selection of DB and caching)• Better performance (global distribution, multithreading for Go)• Git publishing (heroku style!)• Better out of the box monitoring (alerts, schematic logging)

My Wishlist for GAE

THANK YOU!We’re hiring - jobs@streamrail.com