indexing the real world · ocean data management sensor data discovery engine from a small buoy to...
TRANSCRIPT
![Page 1: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/1.jpg)
Rainer Sternfeld, CE September 2014
Indexing the real world
March, 2015
André Karpištšenko Research Lead, Planet OS
![Page 2: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/2.jpg)
2 March 2015
Planet OS Presence
Tallinn
Rio de Janeiro
Washington DC
HoustonLos Angeles
Sunnyvale
![Page 3: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/3.jpg)
3 March 2015
Sensor Data Discovery EngineOcean Data Management
From a small buoy to Big Data
Data BuoysMarket: $2 billion
Competitors: 100+ producers Scalability: poor to limited
2008 2012 2014
Market: $5 billion Competitors: 25+
Scalability: good but slow
Market: $100+ billion Competitors: 15+
Scalability: very scalable and fast
![Page 4: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/4.jpg)
4 March 2015
Data
Time
Trends in Sensor Data
“By 2020, 40% of all data ever collected by human kind will be generated by sensors.”
Hewlett Packard:
![Page 5: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/5.jpg)
5 March 2015
Map of all devices on the Internet
August 2, 2014
![Page 6: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/6.jpg)
6 March 2015
Consolidate dataflows, organize and make sense of your data
Access your data with 3rd party tools and systems
One interface to search and discover your local and remote data
Securely exchange, acquire or sell datasets
Build advanced domain specific solutions without hassle
Industrial IoT Platform for Real-World Sensor Networks designed for ocean, land, air and space data
![Page 7: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/7.jpg)
7 January 2015
Planet OS is a Data Discovery Engine designed for sensor and machine data
![Page 8: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/8.jpg)
January 20158
/
![Page 9: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/9.jpg)
9 March 2015
Case #1: NOAA Real-time weather and climate data + forecasts
Image Credit: NASA
They are working hard on a challenge presented by:
• Tens of thousands of devices deployed in the ocean, on land, and space
• Tens of terabytes coming in every day
• 700 scattered “web services” (FTPs, flat files, Thredds servers, no APIs)
• Breaking connections (all.the.time.)
• 25,000 employees
• Critical data for the government, the industry, and citizens
What if all this data would be easily discoverable and machine-readable?
![Page 10: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/10.jpg)
10 March 2015
Example to case #1: Marinexplore.org 40,000+ data streams from 33 organizations of open data
Advanced Data Discovery Raster data / heat map overlays Access with third party applications
Raster data / quiver plots Graph Monitor Build custom datasets
![Page 11: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/11.jpg)
11 January 2015
3D situational awareness
![Page 12: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/12.jpg)
12 March 2015
Professional Services Experience: Data IntegrationImage Credit: NASA
• 42,000+ data streams from 33 organizations • Work with up to 100 years of historic data • Buy single datasets or subscribe to vendor deliveries
• Ask Planet OS about: • More data sources for your needs • Custom APIs to data • On-demand alerts if something you need is found
KEY BENEFITS
Devices Data Products
• 21,134 devices like the WMO Global Telecommunications System (GTS) buoys, discus buoys, weather buoys,
profiling buoys, drifting buoys, moored buoys etc.
• 5,041 research vessels measuring temperature, salinity, CO2, analyzing important hydrographic and biological
parameters, collecting ADCP profiles and other parameters in the ocean.
• 3,842 operational Argo floats collecting CTD datasets.
• 969 common buoys, for example buoys supporting marine operations.
• 931 fixed stations, for example stations related to US Meteorological Aerodrome Report.
• 126 Teledyne SLOCUM gliders.
• 21 tagged animals by University of St. Andrews.
• 13 station streams from U.S. HF-Radar Network (CODAR SeaSonde)
• 11 Ships of Opportunity, for example the Baltic Operational Oceanographic System related ships.
• 7 Liquid Robotics Wave Gliders, configurable platforms designed to support a wide variety of sensor payloads.
• 2 satellites EUMETSAT ASCAT microwave scatterometer and Aquarius/SAC-D.
• NOAA NCEP WAVEWATCH III, GFS and Global CFSR Reanalysis Hindcasts.
• ETOPO1 Global Relief Model, including both bedrock and ice surface, a 1 arc-minute global relief model.
• The Group for High-Resolution Sea Surface Temperature (GHRSST) Blended Global 1-km Resolution SST.
• NOAA International Comprehensive Ocean-Atmosphere Data Set (ICOADS) SST, Sea Level Pressure and Air Temp.
• NOAA Ocean Surface Current Analyses - Realtime (OSCAR) Third Degree Sea Surface Velocity.
• Daily near-real-time sea height anomaly (SHA) data from TOPEX, ERS-2 and GFO altimeters.
• Hybrid Coordinate Ocean Model (HYCOM) + NCODA Southeast United States and Global Analysis
• KLIWAS North Sea Climatology - Hydrographic Climatology.
• Baltic Sea Physics Reanalysis from DMI and SMIH (1990-2009), Baltic Sea - Sea Ice concentration and Thickness
• Ocean Colour Sea Surface Level-3 Chlorophyll-A Concentration
• Global Ocean Biogeochemistry Analysis: nitrate, phosphate, dissolved oxygen, chlorophyll, phytoplankton
• Mediterranean Sea High Resolution and Ultra High Resolution Sea Surface Temperature Analysis.
• Ocean Optics Products for daily observations and monthly averages reprocessed for the 1997-2010 time period.
• Global Ocean OSTIA SST and Sea Ice (Re)Analysis
• Ifremer Sea Surface Temperature L3 observations
• Sea surface winds from scatterometer, namely descending and ascending swaths.
• NOAA/NCDC Blended daily 0.25-degree Sea Surface Winds.
![Page 13: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/13.jpg)
13 March 2015
Example to Case #2: Bravante Helping to deliver offshore data reports 80% faster
Problems and challenges:
• How to disseminate information to 5 end-customers onshore in parallel?
• How to engage specialists and mangers with the same tool?
• How not to change anything in the equipment on the vessel?
• How to improve speed and quality of the environmental baseline
surveys?
What if all this data would be available in the cloud?
![Page 14: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/14.jpg)
14 March 2015
R/V Jean Charcot
Interactive reporting
Case #2: Bravante Helping to deliver offshore data reports 80% faster
![Page 15: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/15.jpg)
15 March 2015
Robotic ocean-borne sensor platforms
increase productivity
LIQUID ROBOTICS WAVE GLIDER
![Page 16: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/16.jpg)
16 March 2015
A decade of growth in marine acoustics sensor data
summer year-round
2004 2014
2K 400Kn < 10 n > 10002D 4D6 weeks 6 months
sample rate# of sensorsdimensionstime spanactivity
PGS SURVEY VESSEL, 12KM STREAMERS
![Page 17: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/17.jpg)
17 March 2015
Satellites are getting smaller and cheaper.
150 launched since 2011 (3x of the market estimate)
SPIRE, A SAN FRANCISCO STARTUP BUILDING NON-IMAGING LOW-ORBIT NANOSATELLITES USING RF SENSORS
![Page 18: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/18.jpg)
18 March 2015
Unmanned vehicles are estimated to grow 10x in 10 years
Image Credit: Northrop Grumman
![Page 19: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/19.jpg)
19 March 2015
Supported data types and formats
19
VIDEO SONARS (ADCP)
SPATIO-TEMPORAL INDEXING
TIME-SERIES
DATA TYPES
VECTORS
ARRAYS
RASTERS
INSTRUMENTS
SATELLITES
IN-SITU DEVICES
MODELS
HF RADARS
SEISMIC
M
M
M Metadata-level
![Page 20: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/20.jpg)
High-Level Platform Architecture
UIUI
Private and Public Data
Access Securely Anywhere
Connect Storages
UI Integrated Tools
Data Services
Secure Roles & Projects
Exchange Data
ETL
Diverse Formats
PUBLIC
Run Analysis
Vertica, Postgres, S3, EBS Python/Cython, numpy/scipy GDAL, C++ OPeNDAP, NetCDF4 Web GIS, Leaflet, d3, Django
20 January 2015
![Page 21: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/21.jpg)
21 January 2015
StreamFlow is a Dataflow Consolidation Engine
Transformation
Aggregation
Indexing
Data Segments
Incoming Datasets Configuration Channels
Data fetching
Computations
Input Streams
Data Streams
Collection
Plugins
Derived streams
Original Data Formats
Plugins
![Page 22: Indexing the real world · Ocean Data Management Sensor Data Discovery Engine From a small buoy to Big Data Data Buoys Market: $2 billion Competitors: 100+ producers Scalability:](https://reader034.vdocuments.net/reader034/viewer/2022051410/603abcfb875fe671b82a65ec/html5/thumbnails/22.jpg)
22 March 2015
Traditional Data Value Chain is Missing Data Exchange
SATELLITES DRONES CARS BUOYS OIL PLATFORMS SHIPS TRACTORS LIGHTING
PHYSICAL CHEMICAL BIOLOGICAL OPTICAL NON-OPTICAL (RF)
TIME-SERIES RASTER ARRAYS VECTORS SEISMIC VIDEO ACOUSTIC
QA/QA OUTLIER DETECTION MODELING DATA FUSION DATA LOGISTICS VISUALIZATION ENHANCEMENT DERIVATIVES
HARDWARE SENSORS RAW DATA SOFTWARE ANALYTICS
CUSTOMER APPLICATIONS
ENERGY WEATHER AGRICULTURE TRANSPORTATION INSURANCE TELECOMMS LIGHTING HEALTHCARE
DATA EXCHANGE LIVES HERE