big data and html5 (devcon tlv 2012)

Big Data And HTML5@DevCon TLV

Ido GreenDeveloper Advocatehttp://plus.ly/greenido@greenido DevCon TLV

September 5th 2012#devcon

http://plus.ly/greenido



http://twitter.com/!#/greenido



●The power of HTML5 APIs●Modern Web Apps

●Google Cloud Endpoints●Live Code & Demo

●Big Query●Q&A

Agenda

#devcon

Modern Web Applications

Modern Web Applications

● Self Contained & Functional

● "Offline First"

● Client Side Architecture & MVC Frameworks

● Device Aware / 60fps

#devcon

http://goo.gl/5cJwH

#devcon

http://goo.gl/5cJwH

http://goo.gl/5cJwH

http://goo.gl/Y6AG1

#devcon

http://goo.gl/Y6AG1

http://goo.gl/Y6AG1

Client Side Frameworks

● Ember.js - Don't waste time making trivial choices

● Backbone.js - Gives structure to web applications by providing models with binding, collections and views

● Angular.js - AngularJS lets you extend HTML vocabulary for your application

More: addyosmani.github.com/todomvc/

#devcon

http://emberjs.com/

http://emberjs.com/

http://documentcloud.github.com/backbone/

http://angularjs.org/

http://addyosmani.github.com/todomvc/

● Airplane, road trip, deserted island

● Flaky connections (e.g. cafes, car)

● Better performance

● Consolidates the concept of permanent app you will have always available

* We will use: Lawnchair for our demo.

Offline - Why?

#devcon

http://brian.io/lawnchair/

● Storing assets: AppCache

● Storing data: localStorage, IndexedDB, File API.

● Offline first:○ Pretend that there's no internet connection○ Implement a sync layer that works only when

online.

Offline - How?

navigator.onLine & window.(ononline|onoffline)

#devcon

Do More For Your Users

Web Intents is a framework for client-side service discovery and inter-application communication

#devcon

Google Chrome Frame

Chrome Frame is an open source plug-in that seamlessly brings Google Chrome's open web technologies and speedy JavaScript engine to IE

<meta http-equiv="X-UA-Compatible" content="chrome=1">

X-UA-Compatible: chrome=1

#devcon

https://developers.google.com/chrome/chrome-frame/

Modern Web Apps and The Server Conundrum

All modern web apps have to deal with a "home" server○ Offload Computation○ Sharing and Collaboration

But who wants to run a server○ Spikey traffic ○ Client Server communication○ Serialization○ OAuth Dance

#devcon

● Make it easy to build

● Easy to run and manage

● Easy to scale

● Free to get started, pay for what you use

● Not just for web serving, but also data persistence

Google App Engine To The Rescue

#devcon

Google Cloud Endpoints:

Business Logic

APIs for Mobile and Web Backends Made Easy!

Storage(DataStore, SQL, Drive, etc)

Web APIs

End

poin

ts

Trusted Tester

#devcon

Demo

https://github.com/greenido/backbone-bira

#devcon



Google Cloud Endpoints for Beer!

Business Logic

Beer Rating and Review Application

StorageNoSql Datastore

End

poin

ts

Access Control, Sort, Filter

Client UI for managing Beers

#devcon

Demo!Beers On!https://birra-io2012.appspot.com/

https://birra-io2012.appspot.com/

https://birra-io2012.appspot.com/

Load Test Results - From laptop wifi

% ab -n9000 -c100 http://birra-io2012.appspot.com/

Percentage of the requests served within a certain time (ms) 50% 263 66% 321 75% 370 80% 390 90% 449 95% 536 98% 649 99% 715 100% 3858 (longest request)

> 90% of r

equests b

elow

1/2 a se

c

#devcon

Load Test Results - From Compute Engine

% ab -n1000000 -c10 http://birra-io2012.appspot.com/

Percentage of the requests served within a certain time (ms) 50% 6 66% 6 75% 6 80% 6 90% 7 95% 7 98% 11 99% 17 100% 3019 (longest request)

> 95% less

than 7m

s

#devcon

Screen shot of admin console

> 6k QPS!!

How BIG is big?

1 million1 million1 million1 million1 million1 million1 million1 million1 million1 million

10 million rows?





100 million rows?

1 million

1 million1 million 1 million

1 million1 million1 million1 million1 million1 million1 million1 million1 million




1 million1 million1 million1 million

1 million1 million1 million1 million1 million

1 million1 million1 million1 million1 million1 million1 million1 million1 million


1 million1 million1 million1 million1 million1 million1 million

1 million

1 million1 million1 million


1 million

1 million

1 million1 million1 million









720 million rows!

1 million

Big Data at Google

70+ hours100 million gigabytes425 million users

#devcon

BigQuery gives you this power

Store data with reliability, redundancy and consistency

Go from data to meaning

Quickly!

At scale & secure way...

#devcon

How are developers using it?

Game and social media analytics

Advertising campaign optimization

Sensor data analysis

Infrastructure monitoring

#devcon

Big Data at Google - Finding slow servers

SELECT count(*) AS count, source_machine AS machineFROM product.product_log.liveWHERE elapsed_time > 4000GROUP BY source_machineORDER BY count DESC

Result in ~20 seconds!#devcon

Google Cloud Storage

Upload your Data

It's An API

#devcon

BigQuery

Your data is loaded

Google Cloud Storage

GCE

#devcon

Load your data into BigQuery"jobReference":{ "projectId":"605902584318"}, "configuration":{ "load":{ "destinationTable":{ "projectId":"605902584318", "datasetId":"my_dataset", "tableId":"widget_sales"}, "sourceUris":[ "gs://widget-sales-data/2012080100.csv"], "schema":{ "fields":[{ "name":"widget", "type":"string"}, ...

POST https://www.googleapis.com/bigquery/v2/projects/605902584318/jobs

#devcon

https://www.googleapis.com/bigquery/v2/projects/605902584318/jobs

Query Away!

"jobReference":{ "projectId":"605902584318"}, "query":"SELECT TOP(widget, 50), COUNT(*) AS sale_count FROM widget_sales", "maxResults":100, "apiVersion":"v2"}

POST https://www.googleapis.com/bigquery/v2/projects/605902584318/jobs

#devcon

●Java●Python●.NET●PHP●JavaScript●Apps Script●More ...

Libraries

#devcon

Libraries - Example JavaScript query

var request = gapi.client.bigquery.jobs.query({ 'projectId': project_id, 'timeoutMs': '30000', 'query': 'SELECT state, AVG(mother_age) AS theav FROM [publicdata:samples.natality] WHERE year=2000 AND ever_born=1 GROUP BY state ORDER BY theav DESC;'});

request.execute(function(response) { console.log(response); $.each(response.result.rows, function(i, item) { ...

#devcon

BigQuery UI

bigquery.cloud.google.com#devcon

Prepare your data / Batch Jobs

●App Engine MapReduce●Commercial ETL tools

● Pervasive● Informatica● Talend

●UNIX command-line

Don't need interactive queries for some jobs?● priority: "BATCH"

#devcon

Google Spreadsheets

Google Visualization API

#devcon

Commercial visualization tools

#devcon

Key Take Aways

Building modern applications with HTML5 and App Engine

AppEngine makes for easy deployment at scale

Use Big Query to 'feel' your data

Build interactivity by leveraging Cloud Endpoints

Exploit Modern Browser Features:○ Offline○ Geo○ Web Intents

#devcon

Toda & Questions?

Ido GreenDeveloper RelationsGoogle Chrome Platformplus.ly/greenido

App:: http://birra-io2012.appspot.com/

Code: https://github.com/greenido/Slides: ido-green.appspot.com

Access: http://endpoints-trusted-tester.appspot.com

#devcon



http://birra-io2012.appspot.com/



https://github.com/greenido/

http://ido-green.appspot.com

http://endpoints-trusted-tester.appspot.com

big data and html5 (devcon tlv 2012)

Technology

percentage

big data

requests served

longest request

devcon

data

query

big