data, big data and real time analytics for connected devices

24
Srinath Perera, Ph.D. Director of Research WSO2 Inc. Data, Big Data and real time analytics for Connected Devices

Upload: srinath-perera

Post on 19-Nov-2014

467 views

Category:

Technology


3 download

DESCRIPTION

Internet of things paints a vivid picture of a possible reality that is both fascinating and imposing. However, few talk about the sensing and decision making infrastructure--the brain--that must be present with those devices. Underline decision framework needs to collect data, analyze them, compare and contrast with all data, and draw conclusions and arrive at decisions before humans at the other end notice the lag. In talk will start with IoT reference architecture and will discuss Complex Event Processing (CEP) coupled with Lambda architecture as a underline decision framework for underline IoT scenario while drawing examples from several real-world scenarios. You will learn about design choices in building an IoT architecture, CEP, Hive, and Lambda architecture. Topics to be covered: The relationship between IoT and data, big data, and real-time analytics Design choices in building an IoT architecture, CEP, Hive, and Lambda architecture

TRANSCRIPT

  • 1. Outline IOT Why Bigdata? A referenceArchitecture Technologies UsecasesPhoto by John Trainoron Flickrhttp://www.flickr.com/photos/trainor/2902023575/, Licensed under CC

2. Internet of Things Currently physical worldand software worlds aredetached Internet of thingspromises to bridge this It is about sensors andactuators everywhere In your fridge, in yourblanket, in your chair, inyour carpet.. Yes even inyour socks Google IO pressure mats 3. IoT Usecases Smart Home - energy optimization (e.g. controltemperature), hostspot reporting, home surveillance,smart lighting, perimeter checks for pets, kids Smart health - personal tracker, in home care Agriculture - water based on moisture level, pestcontrol, live stock management Smart city - waste management, parking, traffic,pollution monitoring, smart bridges/ constructions -put lot of sensors in to concrete . Smart buildings - energy, surveillance, elevators, Smart retail, smart logistics, smart manufacturing etc. 4. Data Processing Tools Landscape 5. Lot of focus on hardware side,making connection etc. What ittake to make good decisions basedon data? 6. IoT Usecases by Decision Models 7. Decision Model for IoT 8. Design Considerations Edge processing - local processing onsite for efficiencyand HA Last mile how to push actions. How to carry out aaction that takes time and avoid conflicts Integration with the world - Calendar, understandingfor the context, using other services and data Most decisions falls into broad classes - prediction,anomaly detection, optimization Support for context - who, current time, where,calendar, habits, interests, weather, who he is with,current pending actions 9. Edge Processing First solutions suggestdecisions are placed inthe cloud Have to send all datato cloud? May be toomuch What if cloud orconnection failed? Latency However, bettermodels and decisionare possible using datafrom many sites *WSO2 CEP can alsorun in the edge 10. Smart Energy as an Example 11. Taking Human Out of the loop Need to be done carefully as we do notforesee possible outcomes Gradually with fine grain control Provide alarms, and give potential actions Ask user to confirm actions Only automate selected actions e.g. once userOK every time, let user control at each actionlevel. 12. Tools for Building Decisionsystems for IoT 13. What you need from tools? Realtime processing Process streams without storing Temporal queries Low latency Complex Event Processing Systems are a great match (e.g.WSO2 CEP) Batch processing Basic analytics, MapReduce or Spark Decision system Reason with many facts Derive inferences Often done with a rule based system (e.g. WSO2 RulesServer/ Drools, Prolog) 14. WSO2 Complex Event Processor 15. Business Activity Monitor 16. Lambda Architecture 17. Other Products useful in IoT context WSO2 Business Rule Server support ruleexecution based on Drools WSO2 ESB for integrating with other systems WSO2 User Engagement Server (UES) to builddashboards WSO2 EMM for device management WSO2 API Manager to expose APIs 18. Case study: Smart Energy DEBS (Distributed Event Based Systems)is a premier academic conference, whichpost yearly event processing challenge Smart Home electricity data: 2000sensors, 40 houses, 4 Billion events WSO2 CEP based solution is one of thefour finalists (Others Dresden Universityof Technology and Fraunhofer Institute(Germany), and Imperial College London) We posted fastest single node solutionmeasured (400K events/sec) and close toone million distributed throughput. 19. Case study: Realtime Soccer AnalyticsVideo in https://www.youtube.com/watch?v=nRI6buQ0NOM 20. Conclusions Understanding IoT IoT architecture and design IoT Decision systems Tools and design choices Conclusion 21. Questions?