how the weather company monetizes weather: the big data challenge (bdt208) | aws re:invent 2013
Post on 07-Dec-2014
1.458 Views
Preview:
DESCRIPTION
TRANSCRIPT
How the Weather Company Monetizes Weather, the Original Big Data Problem Sathish Gaddipati, VP - Data Management, The Weather Company Raja Selvaraj, Manager - Data and System Engineering, The Weather Company
November 14, 2013
WHERE THE WORLD GETS ITS WEATHER
#1 MOST DISTRIBUTED Cable
Network*
100M+ App Downloads* #2 All Time Most Downloaded iPad App**
47.2M Unduplicated Monthly Uniques***
61.6M Unduplicated Monthly Uniques
(weather.com + wunderground)*
72% visit 2x or more Daily**
TV
DIRECT CONSUMER
ONLINE & APPS
B 2 B 2 C
B 2 B
GOV’T PRODUCTS
*Nielsen, October 2012.
*comScore, Oct 2012 **Omniture, July 2012
*Various app stores, TWC only, **App Store Rewind, 2011 ***Omniture Site Catalyst, August 2012. TWC Mobile & wunderground
Did You Know? TWC Weather Services for Businesses World’s Leading Weather Media Company • - Power 500+ broadcast channels world-wide
World’s Leading Aviation Weather Service • - Support 50,000+ flights daily with embedded
forecasters, systems and tools • - 85% of major US airlines. 30% of Top 100
global.
World’s Leading Energy Forecaster • - Over 300 power trading clients worldwide • - Hourly, daily, and seasonal trading
Fastest Growing Insurance Weather Service • - 7 of the top 10 P&C insurance customers • - Connecting Marketing, Underwriting, & Claims
with systems & tools
WEATHER IS CHANGING • 2012 was the hottest in recorded
history for the US • Last time January was colder than
normal was 1983 • In July 2012 61% of the country was in
drought conditions • 9.2MM acres charred in 2012; third
highest in 13 years
Source: Time.com; Climate Central
Weather changes… So must IT • In the next 12 months we will:
• Increase our data points by 1,400% • Increase our data velocity 300% • Increase our number of deployments
by 2,000% • Increase our number of TV feeds
from 1 to 7 • Increase the amount of video content
by 100% • Increase the amount of social
content by 500%
5
• 13 data centers to 4 • 75% of workloads to IaaS by
building a cloud abstraction toolkit
• Move to agile development teams
• New TV Transmission System (retire 2, build 1)
• 575 APIs and 20 data platforms to 1 platform
• 2500+ new localization servers • New HR, Email, Finance,
Inventory, Expense, etc. • New Ad Stack • New Video Asset
Management Huge
pro
ject
s req
uire
d to
supp
ort t
hese
bu
sines
s nee
ds
The Weather Company’s Core
Science
Data Technology
WDS
Insurance
Retail
Weather Data Services - Use Cases
Tornado and flood forecasts Weather warnings Historical weather trends
Wind Energy
Wind speed forecast Historical wind speeds
Precipitation forecast Temperature forecast Extreme weather forecast
Max. Premium Rate Min. Claims
PPC Maintenance
Inventory management Distribution
WDS
Media
Ad Targeting
Hourly forecasts Daily forecast Current conditions
Temperature forecast Historical trends
Real-time conditions Forecasts Customer location
Weather data Weather content
Commodity bidding Demand forecasting
Impression serving Improved Ad. Exchange
Energy Exchange
Weather Data Service - Use Cases
WDS
Mobile Apps
Hospitality
Hourly forecasts Daily forecast Current conditions
Weather forecast Historical forecast Watches and warnings
Current conditions Forecasts Airline delays/ cancellations
Weather data Weather content
Local weather National weather
Room rates Revenue optimization
Gov
Weather Data Service - Use Cases
WDS
Hourly forecasts Daily forecast Current conditions Historical trends
Historical data Consumer spend vs. weather conditions
Air turbulence and wind speeds Weather forecasts Current conditions
Weather data Weather content
Business impact Consumer behavior
Optimal Routes Flight Schedules
O&O Products
Weather Analytics
Airlines
Weather Data Service - Use Cases
11
ACCURACY THROUGH SUPERIOR DATA
Global Lightning Detection Network
Proprietary Radar
Algorithms
Industry Best Forecast Modeling
State-of-the-Science Forecast
Technologies PhD 220+ Full-time
Meteorologists
Robust weather data APIs
Largest collection of worldwide forecast data
Proprietary weather data analytics
APIs Network of 30,000+ Weather Sensors
Number 1 in forecasting accuracy per ForecastWatch in the US and Europe for
the last 6 months… 5 points better than NWS Forecast
We defined an architecture that could support the needs of all of our divisions. • API interfaces • IaaS abstraction • Shardable data • Global scale
1. Reduce time to deploy and market new data sets
2. Reduce operating cost of data services
3. Centralize data services across the company
4. Provide visibility of data access
5. Provides data governance process and framework
6. Serves world’s best weather forecast across all products
7. Low latency, highly scalable APIs
8. Secured access to data
9. Centralized and scalable architecture
10. Consistent “rich” content across platforms
Top 10 Objectives of Weather Data Services Platform
Weather Data Services Platform
1. Distribute thousands of gridded binary files to multiple locations across globe within 5 minutes
2. Serve more than billion data services API requests/day
3. Metering and authentication of API calls with low latency
4. Process multiple TBs of data every day
5. Ensure business continuity
6. Leverage data caching
7. Store petabytes of historical data
8. Meshing weather data with consumer behavior and derive analytics
9. Build flexible data ingestion platform to manage 100s of data feeds from external parties
10.Maintain above systems within OPEX budget
Top 10 Computing Challenges
Data Store
Manageability
Availability
Performance
Replication
Capacity
DATA STORES – WHAT WE WANTED
Historical weather data
Raster/Image data
Analytics platform
Severe weather data
DATA STORES – WHAT WE USE
Public S
ubnet
AZ1
AZ2
Pub
lic
Sub
net
Priv
ate
Sub
net
AZ2
Private S
ubnet
AZ1
VPC VPC
Amazon – Region 1 Amazon – Region 2
RIAK REPLICATION @AWS
NEXT GENERATION – WEATHER FORECASTING SYSTEMS
Amazon Redshift
WeatherFX
CreativeFX
AudienceFX
App N
… TWC
Private Data
Center EDW
Partner EDW
Platform
Amazon Redshift
Amazon S3
TWC’s Redshift Cluster now contains: 600+ million records of weather forecast data since March 2005 700+ million records of current conditions observation data since April 2004 11+ billion records of Int. Civil Aviation Org. (ICAO) observation data Since 1970 TBs of storm event data from from 1950 - 2012 Weather sentiment survey data
Weather’s Big Data Analytics Platform for Enterprises and Government
~1 Billion records Bulk loaded in 36
minutes!
Contact Us Sathish Gaddipati – sathish.gaddipati@weather.com Raja Selvaraj – raja.selvaraj@weather.com
Please give us your feedback on this presentation
As a thank you, we will select prize winners daily for completed surveys!
BDT208
top related