insight:intelligent synthesis and real time response using massive streaming of heterogeneous data -...
TRANSCRIPT
INSIGHT: Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
INSIGHTDermot Kinane (21st January 2015)
1
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Overview
Dublin City Council Introduction and background leading into traffic use-case
• Local Authority with a number of responsibilities
Current Situation Facilities
• Control Room (24/7)• Radio Station• AA Road Watch/Dublin Bus/Gardai• Incident Management Centre
Systems Limitations INSIGHT in Use
Next steps with INSIGHT
2
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Objective
INSIGHT
Intelligent Synthesis and Real-time Response Using Massive Streaming of Heterogeneous
Data
3
Develop an adaptive, scalable and dependable, real-time infrastructure for improving our ability to identify and cope with emergencies
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Motivation
Currently, little infrastructure in place to take advantage of data provided by sensors or humans
Improve understanding, prediction and warning of emergencies through real-time processing of data streams including social data
Guiding more informed decisions leading to effective response
4
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Overview
Dublin City Council Introduction and background leading into traffic use-case
• Local Authority with a number of responsibilities
Current Situation Facilities
• Control Room (24/7)• Radio Station• AA Road Watch/Dublin Bus/Gardai• Incident Management Centre
Systems Limitations INSIGHT in Use
Next steps with INSIGHT
5
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Traffic Control Room
6
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Traffic Control Room
7
Operations team (24/7)
Radio station
AA Roadwatch
Dublin Bus
Gardai
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Incident Management Centre
8
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Systems in use
SCATS (Sydney Coordinated Adaptive Traffic System)
CCTV
TRIPS
DPTIM
FMS
RTPI
9
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Unusual/Unpredicted Congestion
10
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Unusual Congestion and its Effects (M50 Fire)
11
Tuesday 30th September 2014~14:30
Lots of data – needs to be integrated to tell the full story
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Unusual Congestion and its Effects (SCATS)
12
Auburn - Blanchardstown
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Unusual Congestion and its Effects (SCATS)
13
Naas Road – Heuston Station
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Unusual Congestion and its Effects (SCATS)
14
Rathfarnham Road (inbound)
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Unusual Congestion and its Effects (SCATS)
15
Samuel Beckett Bridge
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Unusual Congestion and its Effects (bus)
16
12:00:00 AM
12:05:00 AM
12:10:00 AM
12:15:00 AM
12:20:00 AM
12:25:00 AM
12:30:00 AM
12:35:00 AM
12:40:00 AM
12:45:00 AM
12:50:00 AM
12:55:00 AM
1:00:00 AM
1:05:00 AM
1:10:00 AM
1:15:00 AM
29.09.2014
30.09.2014
St. John’s Road
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Unusual Congestion and its Effects (bus)
17
Chapelizod
0:00:00
0:05:00
0:10:00
0:15:00
0:20:00
0:25:00
0:30:00
0:35:00
0:40:00
29.09.2014
30.09.2014
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Unusual Congestion and its Effects (bus)
18
Harold’s Cross
12:00:00 AM
12:05:00 AM
12:10:00 AM
12:15:00 AM
12:20:00 AM
12:25:00 AM
12:30:00 AM
12:35:00 AM
12:40:00 AM
29.09.2014
30.09.2014
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Unusual Congestion and its Effects (Tweets)
19
Live Drive Tweets
Impressions – average 45,206 – fire 165,346
Replies – average 19 – fire 132
0
20
40
60
80
100
120
140
160
180
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Unusual Congestion and its Effects (Tweets)
20
M50 fire
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Unusual Congestion and its Effects (Results)
21
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Limitations
Manual operation
We find out about all serious incidents… eventually
Platform doesn’t exist to integrate heterogeneous streams
Opportunities…
22
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Challenges
Handling heterogeneity
Handling large datasets
Handling uncertainty
Real-time processing
23
Partners
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Data Integration & Availability
24
- Working with DCC, eircom & IBM, a virtual private cloud (VPC) solution was proposed. Using Amazon Web Services (AWS), an IPSEC VPN to the DCC firewall was created.
- Using AWS brought the following advantages:- Compute services based on pay-as-you-use rates- 24x7 management of servers, up to OS layer included- Regular back-up schedule- All operating system licenses built into price- High resilience as it is running over 2 Availability Zones (AZ)
- Bus data retrieval is simply from a data storage area in AWS S3 (Storage Bucket)
- SCATS data stream provided in JSON
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Solutions
25
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
INSIGHT Architecture
26
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
INSIGHT in action (symbols & examples)
27
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
INSIGHT in action
28
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
INSIGHT in action
29
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
INSIGHT in action
30
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
INSIGHT in action
31
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
INSIGHT in action
32
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
INSIGHT in action
33
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
INSIGHT in action
34
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
INSIGHT in action
35
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Crowdsourcing
36
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Overview
Dublin City Council Introduction and background leading into traffic use-case
• Local Authority with a number of responsibilities
Current Situation Facilities
• Control Room (24/7)• Radio Station• AA Road Watch/Dublin Bus/Gardai• Incident Management Centre
Systems Limitations INSIGHT in Use
Next steps with INSIGHT
37
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Design of Validation Experiments
Description/Validation of Experiments/Next Steps
How DCC intend to put the INSIGHT software to test
• 1st phase - Radio Station to be test bed
• To verify that events are happening
• False positives/missed events
• 2nd phase – Control Room operator(s)
• All INSIGHT partners have been invited to work from the Dublin City Council Traffic Control Room when the INSIGHT beta is released
Plan for testing crowdsourcing app in place
Extended datasets
38
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Summary
Currently a very manual operation
INSIGHT very relevant to DCC
Intelligent synthesis
Real-time response
Streaming heterogeneous data
Confidence level
Accurate deployment of resources
Opportunity to add value to DCC operation (& other cities)
Further information
http://www.insight-ict.eu/
39
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Backup slides
40
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Events & Notifications
41
Prototype – City Case
Root
Anomaly
Traffic Jam
Low Speed High DelayHigh Degree of Saturation
Flood
Normal
SCATS ISA High Degree of Saturation
Twitter ISA Flood Traffic Tweet (Traffic or Flood Related)
Bus ISA Low Speed, High Delays Noisy Sensors
Bus Reports Extremely High Speed Bus Moving but Reports that is at Bus
Stop Bus seems to be far away from the stop
it reports to be at
Crowdsourcing Tasks (questions) are sent
to the users
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
Second Year Demo Architecture
26
Prototype – City Case
Intelligent Synthesis and Real Time Response using Massive Streaming of Heterogeneous Data
❑http://195.134.71.40:8000/html/visualization/