accumulo summit 2015: preventing bugs: how phemi put accumulo to work in the field [geo]

of 26 /26
Preventing Bugs Putting Accumulo To Work In The Field Russ Weeks, PHEMI Orchard

Author: accumulo-summit

Post on 15-Jul-2015




3 download

Embed Size (px)


  • Preventing BugsPutting Accumulo To Work In The Field

    Russ Weeks, PHEMI


  • Precision Agriculture

  • Sensors


  • Why We Accumulo

    Continuous Aggregations

    Time-series Data

    Spatiotemporal Data

  • Farming is Risky!

  • Adrian Baggaley, FruitForum

  • Horticultural Sciences Department, University of Florida

  • USDA

  • Aggregation Schema

  • Time-Series Data

    Sensor messages arriving in constant, high-velocity stream

    Common use case: Show me the last 2 weeks of data for sensor X

  • No problem!

    Row CF CQ ValM0 msg timestamp 2014/07/01T19:30

    M0 msg sensor_id 1138

    M0 msg msg_id 1138:2014/07/01T1930

    M0 dev battery 3.3v

    M0 env temp 20.3c

    M0 env windspd 2.96m/s

  • Lexicode + Transpose

    Row CF CQ Valmsg:msg_id:1138:2014/07/01T2020 M842 1

    msg:msg_id:1138:2014/07/01T2010 M767 1

    msg:msg_id:1138:2014/07/01T2000 M684 1

    msg:msg_id:1138:2014/07/01T1950 M592 1

    msg:msg_id:1138:2014/07/01T1940 M356 1

    msg:msg_id:1138:2014/07/01T1930 M0 1

  • But my old database could dereference those message IDs automatically!

    All I had to do was this join, see

  • Time-Series Index

    Row CF CQ Val(sensor_id:1138,2014/07/01T2020,battery,3.3v) [] M842 1(sensor_id:1138,2014/07/01T2020,windspd,1.83m/s) [] M842 1(sensor_id:1138,2014/07/01T2020,temp,18.6c) [] M842 1(sensor_id:1138,2014/07/01T2010,battery,3.2v) [] M767 1(sensor_id:1138,2014/07/01T2010,windspd,1.51m/s) [] M767 1(sensor_id:1138,2014/07/01T2010,temp,18.8c) [] M767 1

  • Spatiotemporal

  • Map Tiles: OpenStreetMap

  • Pos. 0 Pos. 212-1

  • Spatiotemporal

  • OK, But Why Not HBase?

  • Iterators are awesome

    Customer is interested in cell-level security

    Future applications may involve controlled data-sharing

  • Thanks!