how the weather company monetizes weather: the big data challenge (bdt208) | aws re:invent 2013
DESCRIPTION
(Presented by Basho) This session will discuss the transformation of the most widely distributed cable TV network in the United States, building on one of the world's most visited digital properties, to create a world class Big Data platform. Architects, CTOs, CIOs, IT Director, and development managers will learn how to run highly scalable analytics workloads on Amazon EC2 and Amazon EMR for complex, real-time analysis of large data sets. All while decreasing time to results and increasing business agility. Bryson Koehler, EVP & CIO of The Weather Company, will discuss architecture, technology choices, performance results and business benefits realized as part of their use of AWS services to host an exciting set of weather.com solutions and generate new revenue streams. Weather impacts over 30% of the global GDP daily and is the source of vast amounts of data collection. The Weather Company is the leader in weather forecasting and is bringing the world's most accurate forecasting capabilities alive in a full suite of data APIs built fully on Infrastructure as a Service platforms, including AWS and next generation products like Basho Riak, Hadoop, and Dasein. This session will discuss how the application of these technologies help keep people safe and helps businesses plan and become more profitable, thanks to the latest intersection of consumer behavior and weather forecasting and reporting.TRANSCRIPT
How the Weather Company Monetizes Weather, the Original Big Data Problem Sathish Gaddipati, VP - Data Management, The Weather Company Raja Selvaraj, Manager - Data and System Engineering, The Weather Company
November 14, 2013
WHERE THE WORLD GETS ITS WEATHER
#1 MOST DISTRIBUTED Cable
Network*
100M+ App Downloads* #2 All Time Most Downloaded iPad App**
47.2M Unduplicated Monthly Uniques***
61.6M Unduplicated Monthly Uniques
(weather.com + wunderground)*
72% visit 2x or more Daily**
TV
DIRECT CONSUMER
ONLINE & APPS
B 2 B 2 C
B 2 B
GOV’T PRODUCTS
*Nielsen, October 2012.
*comScore, Oct 2012 **Omniture, July 2012
*Various app stores, TWC only, **App Store Rewind, 2011 ***Omniture Site Catalyst, August 2012. TWC Mobile & wunderground
Did You Know? TWC Weather Services for Businesses World’s Leading Weather Media Company • - Power 500+ broadcast channels world-wide
World’s Leading Aviation Weather Service • - Support 50,000+ flights daily with embedded
forecasters, systems and tools • - 85% of major US airlines. 30% of Top 100
global.
World’s Leading Energy Forecaster • - Over 300 power trading clients worldwide • - Hourly, daily, and seasonal trading
Fastest Growing Insurance Weather Service • - 7 of the top 10 P&C insurance customers • - Connecting Marketing, Underwriting, & Claims
with systems & tools
WEATHER IS CHANGING • 2012 was the hottest in recorded
history for the US • Last time January was colder than
normal was 1983 • In July 2012 61% of the country was in
drought conditions • 9.2MM acres charred in 2012; third
highest in 13 years
Source: Time.com; Climate Central
Weather changes… So must IT • In the next 12 months we will:
• Increase our data points by 1,400% • Increase our data velocity 300% • Increase our number of deployments
by 2,000% • Increase our number of TV feeds
from 1 to 7 • Increase the amount of video content
by 100% • Increase the amount of social
content by 500%
5
• 13 data centers to 4 • 75% of workloads to IaaS by
building a cloud abstraction toolkit
• Move to agile development teams
• New TV Transmission System (retire 2, build 1)
• 575 APIs and 20 data platforms to 1 platform
• 2500+ new localization servers • New HR, Email, Finance,
Inventory, Expense, etc. • New Ad Stack • New Video Asset
Management Huge
pro
ject
s req
uire
d to
supp
ort t
hese
bu
sines
s nee
ds
The Weather Company’s Core
Science
Data Technology
WDS
Insurance
Retail
Weather Data Services - Use Cases
Tornado and flood forecasts Weather warnings Historical weather trends
Wind Energy
Wind speed forecast Historical wind speeds
Precipitation forecast Temperature forecast Extreme weather forecast
Max. Premium Rate Min. Claims
PPC Maintenance
Inventory management Distribution
WDS
Media
Ad Targeting
Hourly forecasts Daily forecast Current conditions
Temperature forecast Historical trends
Real-time conditions Forecasts Customer location
Weather data Weather content
Commodity bidding Demand forecasting
Impression serving Improved Ad. Exchange
Energy Exchange
Weather Data Service - Use Cases
WDS
Mobile Apps
Hospitality
Hourly forecasts Daily forecast Current conditions
Weather forecast Historical forecast Watches and warnings
Current conditions Forecasts Airline delays/ cancellations
Weather data Weather content
Local weather National weather
Room rates Revenue optimization
Gov
Weather Data Service - Use Cases
WDS
Hourly forecasts Daily forecast Current conditions Historical trends
Historical data Consumer spend vs. weather conditions
Air turbulence and wind speeds Weather forecasts Current conditions
Weather data Weather content
Business impact Consumer behavior
Optimal Routes Flight Schedules
O&O Products
Weather Analytics
Airlines
Weather Data Service - Use Cases
11
ACCURACY THROUGH SUPERIOR DATA
Global Lightning Detection Network
Proprietary Radar
Algorithms
Industry Best Forecast Modeling
State-of-the-Science Forecast
Technologies PhD 220+ Full-time
Meteorologists
Robust weather data APIs
Largest collection of worldwide forecast data
Proprietary weather data analytics
APIs Network of 30,000+ Weather Sensors
Number 1 in forecasting accuracy per ForecastWatch in the US and Europe for
the last 6 months… 5 points better than NWS Forecast
We defined an architecture that could support the needs of all of our divisions. • API interfaces • IaaS abstraction • Shardable data • Global scale
1. Reduce time to deploy and market new data sets
2. Reduce operating cost of data services
3. Centralize data services across the company
4. Provide visibility of data access
5. Provides data governance process and framework
6. Serves world’s best weather forecast across all products
7. Low latency, highly scalable APIs
8. Secured access to data
9. Centralized and scalable architecture
10. Consistent “rich” content across platforms
Top 10 Objectives of Weather Data Services Platform
Weather Data Services Platform
1. Distribute thousands of gridded binary files to multiple locations across globe within 5 minutes
2. Serve more than billion data services API requests/day
3. Metering and authentication of API calls with low latency
4. Process multiple TBs of data every day
5. Ensure business continuity
6. Leverage data caching
7. Store petabytes of historical data
8. Meshing weather data with consumer behavior and derive analytics
9. Build flexible data ingestion platform to manage 100s of data feeds from external parties
10.Maintain above systems within OPEX budget
Top 10 Computing Challenges
Data Store
Manageability
Availability
Performance
Replication
Capacity
DATA STORES – WHAT WE WANTED
Historical weather data
Raster/Image data
Analytics platform
Severe weather data
DATA STORES – WHAT WE USE
Public S
ubnet
AZ1
AZ2
Pub
lic
Sub
net
Priv
ate
Sub
net
AZ2
Private S
ubnet
AZ1
VPC VPC
Amazon – Region 1 Amazon – Region 2
RIAK REPLICATION @AWS
NEXT GENERATION – WEATHER FORECASTING SYSTEMS
Amazon Redshift
WeatherFX
CreativeFX
AudienceFX
App N
… TWC
Private Data
Center EDW
Partner EDW
Platform
Amazon Redshift
Amazon S3
TWC’s Redshift Cluster now contains: 600+ million records of weather forecast data since March 2005 700+ million records of current conditions observation data since April 2004 11+ billion records of Int. Civil Aviation Org. (ICAO) observation data Since 1970 TBs of storm event data from from 1950 - 2012 Weather sentiment survey data
Weather’s Big Data Analytics Platform for Enterprises and Government
~1 Billion records Bulk loaded in 36
minutes!
Contact Us Sathish Gaddipati – [email protected] Raja Selvaraj – [email protected]
Please give us your feedback on this presentation
As a thank you, we will select prize winners daily for completed surveys!
BDT208