rubynation visualizing big data on small devices

Download RubyNation Visualizing Big Data on Small Devices

Post on 27-Jan-2015

103 views

Category:

Technology

0 download

Embed Size (px)

DESCRIPTION

 

TRANSCRIPT

  • 1. Copyright 2014 Intridea Inc. All rights reserved. Visualizing Big Data on Small Devices Tom Zeng Director of Engineering tom@intridea.com @tomzeng www.linkedin.com/in/tomzeng

2. Copyright 2014 Intridea Inc. All rights reserved. Agenda Introduction Front End - HTML5/Bootstrap, Backbone/CoeeScript, D3, MapBox Backend - Rails, MongoDB Big Data Processing - Hadoop, Hive, Pig Showcase - Mobile and Data Visualization Related Projects Q & A 3. Copyright 2014 Intridea Inc. All rights reserved. Intridea - Rails, UX/Data Visualization, Mobile, Big Data, e-commerce American Bible Society (ABS http://www.americanbible.org/) - partners with Bible publishers Provides API access to 539 Bible versions in 242 languages The usage of the APIs is tracked at the verse level, along with ip location, timestamp, and duration 530 million view logs/year(12-13 data), 1.5 mil/day, each view log has packed about 12 bible views Amounts to 5-6 billion Bible views each year ABS asked Intridea to build the dashboard app Scripture Analytics (http://www.scriptureanalytics.com) Introduction 4. Copyright 2014 Intridea Inc. All rights reserved. BY WORKING REMOTELY 9,816 Hours Saved Annually ACROSS THE US & OVERSEAS 30+ Employees FOUNDED & STARTED IN 2007 Washington D.C. We Make ! # ON GITHUB Open Source Software $ 5. Copyright 2014 Intridea Inc. All rights reserved. Major Open Source Contributions OmniAuth is a flexible authentication system utilizing Rack middleware. OmniAuth An opinionated micro-framework for creating REST-like APIs in Ruby. Grape Hashie is a simple collection of useful Hash extensions. Hashie A Ruby wrapper for the OAuth 2.0 protocol. oauth2 A symbol font that makes it easy to create a map of the U.S. with HTML/CSS. Stately A generic swappable back-end for JSON handling. Multi_JSON Mission control dashboard for your distributed teams. Houston github.com/intridea 6. Copyright 2014 Intridea Inc. All rights reserved. INTRIDEA Simplified and modernized application experience for ADP Application engineering for BusinessWeek.com Worlds first 100% web based Point of Sale system 7. Copyright 2014 Intridea Inc. All rights reserved. Where people go to make a dierence with their investment capital. Simplifying ADPs core business: Payroll Engineering for the most-traicked wedding planning solution 8. Copyright 2014 Intridea Inc. All rights reserved. 9. Copyright 2014 Intridea Inc. All rights reserved. ABS Scripture Analytics Query Requirements Visualizations 10. Copyright 2014 Intridea Inc. All rights reserved. Public and private dashboards visualizing Bible reading across the Internet Up to the minute dashboards showing what Bible verses are being read when, and where, all over the globe. 11. Copyright 2014 Intridea Inc. All rights reserved. Mobile Desktop Tablet Responsive Web App www.scriptureanalytics.com 12. Copyright 2014 Intridea Inc. All rights reserved. Front End User Interface Single Page Application using Backbone.js CoeeScript (Ruby like, Jasmine in CoeeScript similar to RSpec) D3 for Data Visualization Twitter Bootstrap for Responsive UI Packery for Responsive Layout - http://packery.metafizzy.co/ Mapbox for Map Rendering - https://www.mapbox.com/ 13. Copyright 2014 Intridea Inc. All rights reserved. Backend Servers/Services Rails on Ruby Application mostly as the API server MongoDB as the data store/cache Mongoid for Active Record like queries MongoDB Aggregation Framework for complex queries Pulling data periodically from S3 to populate the Mongo database Local R&D Hadoop and Mongo clusters for data exploration 14. Copyright 2014 Intridea Inc. All rights reserved. MongoDB Document oriented, schema free, JSON format Very high data read and write throughput Rich query capabilities (aggregation framework), flexible indexes Scale with auto-sharded replica sets Map/Reduce in JavaScript 15. Copyright 2014 Intridea Inc. All rights reserved. Hadoop/Pig/Hive/Impala Hadoop cluster (AWS Elastic Map/Reduce on-demand) to process and store data in S3 Pig to parse, transform, geo-code data Hive to query data and generate aggregated JSON reports Impala is similar to Hive (but much fast than the older version of Hive), used for ETL ! 16. Copyright 2014 Intridea Inc. All rights reserved. Elastic Map/Reduce Hadoop Cluster - On Demand Processing 17. Copyright 2014 Intridea Inc. All rights reserved. Elastic Map/Reduce Hadoop Cluster - Terminated when done 18. Copyright 2014 Intridea Inc. All rights reserved. Cloudera CDH4 - on local 10-node cluster 19. Copyright 2014 Intridea Inc. All rights reserved. Cloudera CDH4 - Streaming Data into Hive Table 20. Copyright 2014 Intridea Inc. All rights reserved. Pig Sample Query 21. Copyright 2014 Intridea Inc. All rights reserved. Hive Sample Query 22. Copyright 2014 Intridea Inc. All rights reserved. Hive Query Results - Bible views by City 23. Copyright 2014 Intridea Inc. All rights reserved. Hive Query Results - Most popular verse before Mothers day 28 Her children show their appreciation, and her husband praises her. 24. Copyright 2014 Intridea Inc. All rights reserved. Hive Query Results - Most popular verse on Mothers day 28 Her children show their appreciation, and her husband praises her. 25. Copyright 2014 Intridea Inc. All rights reserved. Hive Query Results - Most popular verse after Mothers day 28 Her children show their appreciation, and her husband praises her. 26. Copyright 2014 Intridea Inc. All rights reserved. MongoDB Aggregation Framework Example http://docs.mongodb.org/manual/core/aggregation-pipeline/ 27. Copyright 2014 Intridea Inc. All rights reserved. MongoDB Aggregation Framework Example 28. Copyright 2014 Intridea Inc. All rights reserved. ABS Data Processing using Hadoop and MongoDB 29. Copyright 2014 Intridea Inc. All rights reserved. Analyzing Twitter using Hadoop and MongoDB 30. Copyright 2014 Intridea Inc. All rights reserved. Mobile and Data Visualization Project Showcase ADP BLiNQ PEW Templeton - Global Religious Futures Cato Institute - HumanProgress ! ! 31. Copyright 2014 Intridea Inc. All rights reserved. Redefining ADPs touch and desktop experiences ADP processes one out of every six paychecks in the United States. Were bringing payroll into the decade of touch. 32. Copyright 2014 Intridea Inc. All rights reserved. ADP TLM 33. Copyright 2014 Intridea Inc. All rights reserved. ADP HCR 34. Copyright 2014 Intridea Inc. All rights reserved. INTRIDEA 35. Copyright 2014 Intridea Inc. All rights reserved. Dashboard insights for 600 of the worlds largest advertisers Analytics, planning, and flight management for social advertising campaigns and brand engagement. 36. Copyright 2014 Intridea Inc. All rights reserved. INTRIDEA INTRIDEA 37. Copyright 2014 Intridea Inc. All rights reserved. INTRIDEA INTRIDEA 38. Copyright 2014 Intridea Inc. All rights reserved. INTRIDEA INTRIDEA 39. Copyright 2014 Intridea Inc. All rights reserved. Analyzing religious change and its impact on societies around the world Interactive website exploring the patterns and trends in religions across the globe 40. Copyright 2014 Intridea Inc. All rights reserved. PEW Global Research PEW Global Research 41. Copyright 2014 Intridea Inc. All rights reserved. PEW Global Research PEW Global Research 42. Copyright 2014 Intridea Inc. All rights reserved. PEW Global Research PEW Global Research 43. Copyright 2014 Intridea Inc. All rights reserved. Human advancement to a higher stage Human Progress seeks to document changes in living standards in the past and present while explaining and exploring the best ways to improve conditions for people. 44. Copyright 2014 Intridea Inc. All rights reserved. INTRIDEA humanprogress.org 45. Gracias Merci Danke Thank You Copyright 2014 Intridea Inc. All rights reserved. Tom Zeng Director of Engineering tom@intridea.com @tomzeng www.linkedin.com/in/tomzeng