tfl hackathon solution 4-3-comp - mandrel - as presented · tfl hackathon solution 4-3-comp -...
TRANSCRIPT
Summary Slides …
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 2
Challenge 1 / Answer 3 ….... • Data mapping to a common geography • Within TfL our most granular view of our network is defined by the Ordinance Survey Integrated TOID (Topological IdenPfier) Network (OS ITN). • TOID = Unique ID of feature on the Ordinance Survey map.
• However, it not common that our data sets can map directly onto this. • Really?
• It is our vision that data fusion can enable a single view of performance which takes input from all available datasets and enable us to produce a consistent view of our network. • You bet! Our vision too. You asked …
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 3
Tiered Topology Model Mandrel “The InformaOon Architects”
Challenge 1 In 1933 Harry Beck revoluPonized maps. In 2015 we have to leave the map again… • Edges & Vertex
• Unique Id • Hierarchy
• Topology per Mode • Physical &
Logical
• Linked Topologies • Across Modes
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 4
1933
2015
Unconnected Mapping Data So Which Sensor relates To Which Bus Route & Are We On A Bus Lane?
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 5
Unconnected Mapping Data – Over OSM So Which Sensor relates To Which Bus Route & Are We On A Bus Lane?
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 6
Logical Model With Hierarchy Of Routes Per Transport Mode Links Down To Physical Model(s)
Bus 1 Stop at X,Y
(Vertex)
Bus 2 Stop at X,Y
(Vertex)
Bus 3 Stop at X,Y
(Vertex) Route Leg 2 (Edge)
Route Leg 1 (Edge)
Route (Vertex) Route Has Leg
(Edge) Route Has Leg(Edge)
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 7
Bus 3 Stop at X,Y
(Vertex)
Road Signals & Sensors (Vertex)
Road SecPons (Edge), Waypoints (Vertex), Bus Lanes
& JuncPons
We can now tell which Sensor relates To Which Bus Route & If We Are We On A Bus Lane
More Hierarchy Links Physical & Logical Views Two Physical Stops Either Side Of Road
Bus 1a Stop at X,Y
(Vertex)
Bus 2a Stop at X,Y
(Vertex)
Route
End
Bus 1a Stop at X,Y
(Vertex)
Route Leg 1a (Edge)
Bus 2a Stop at X,Y
(Vertex)
Route Leg 2a (Edge)
Marble Arch Bus “Stop” (Vertex)
Has Bus Stop (Edge)
Has Bus Stop (Edge)
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 8
Now Link Modes With Summary InformaOon
Different parts of the hierarchy are needed to map between modes
Logical model of Tube
journey ends at staPon
Physical model of staPon
has many exits
StaPon exit has walking route to
nearest bus stop of 5 min
Bus route runs from bus stop to bus stop near
home
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 9
Full Slide Deck …
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 10
Challenge 1 – An Unexpected Answer? • Data mapping to a common geography
• Within TfL our most granular view of our network is defined by the Ordinance Survey Integrated TOID (Topological IdenPfier) Network (OS ITN).
• However, it not common that our data sets can map directly onto this.
• It is our vision that data fusion can enable a single view of performance which takes input from all available datasets and enable us to produce a consistent view of our network.
• Our answer is a vision • In 1933 Harry Beck produced the beauPful
design for the London Underground. • It was simple, a topology, not a map. • For AnalyPcs 3.0 we have to take the idea
further breaking the data away from being directly dependent physical map locaPons.
• Adopt a vision to refine and link all of the reference data in shape files to a set of linked Pered topologies covering the TfL network and beyond.
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 11
Unconnected Mapping Data So Which Sensor relates To Which Bus Route & Are We On A Bus Lane?
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 12
Unconnected Mapping Data – Over OSM So Which Sensor relates To Which Bus Route & Are We On A Bus Lane?
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 13
AnalyOcs 2.0 Early Big Data IntegraOon Pair-‐wise Linkage Of Data Sources
Find a way to smash data
together
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 14
AnalyOcs 3.0 Coherent reference data model Joins mulOple data sources
Single Reference Data Model. Tiered Linked Topologies.
Slow changing reference data:
staPons, routes, ….
High volume data. E.g. sensor and trip data.
PosiPvely Link related enPtles. Not just rely on similar map locaPons.
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 15
How Do We Know You Need This? • We wanted to do clever things at the Hackathon but …. • We spent too much Pme trying to merge
GIS and reference informaPon and tear apart Shape files • Not recommended in R or Python
• Other network based industries have gone through exactly the same pain. • For example the mobile phone networks
are (too late) desperately trying to build unified topology models of their mulP-‐layered mulP-‐Pered mulP-‐technology networks.
• New big data tooling is built to handle data network model. • E.g. Spark’s GraphX
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 16
Some Detailed DirecOon
• Here is what we know interpreted from other industries ….
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 17
Everything Becomes Directed Line Graphs Stops (Vertex) & Route Leg (Edges)
Bus Route (One DirecPon)
Bus 1 Stop at X,Y
(Vertex)
Bus 2 Stop at X,Y
(Vertex)
Bus 3 Stop at X,Y
(Vertex) Route Leg 2 (Edge)
Route Leg 1 (Edge)
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 18
Introduce Hierarchy In Logical Model More VerOces and Edges Build Routes
Bus 1 Stop at X,Y
(Vertex)
Bus 2 Stop at X,Y
(Vertex)
Bus 3 Stop at X,Y
(Vertex) Route Leg 2 (Edge)
Route Leg 1 (Edge)
Route (Vertex) Route
Has Leg (Edge)
Route Has Leg (Edge) 15/11/15 © 2015 Mandrel Systems Limited, All Rights
Reserved 19
More Hierarchy Links Physical & Logical Views Two Physical Stops Either Side Of Road
Bus 1a Stop at X,Y
(Vertex)
Bus 2a Stop at X,Y
(Vertex)
Route
End
Bus 1a Stop at X,Y
(Vertex)
Route Leg 1a (Edge)
Bus 2a Stop at X,Y
(Vertex)
Route Leg 2a (Edge)
Marble Arch Bus “Stop” (Vertex)
Has Bus Stop (Edge)
Has Bus Stop (Edge)
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 20
Route Legs Link To Map Physical Topology E.g. road Segments, Waypoints, Sensors Etc. (Very Simplified Drawing)
Route Leg 1 (Edge) Bus 1
Stop at X,Y
(Vertex)
Bus 2 Stop at X,Y
(Vertex)
Road Sensor 23423 (Vertex)
Road SecPons (Edge),
Waypoints (Vertex)
& JuncPons
For each transport mode you will have
one or more linked
physical and logical models.
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 21
Now Link Modes With Summary InformaOon
Different parts of the hierarchy are needed to map between modes
Logical model of Tube
journey ends at staPon
Physical model of staPon
has many exits
StaPon exit has walking route to
nearest bus stop of 5 min
Bus route runs from bus stop to bus stop near
home
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 22
Robust and PracOcable Model Only when all the model is built can you add in mulPple data sets…
…to map mulP-‐modal journeys
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 23
Step One … Add Your Own Unique IDs Everywhere
• Everything Has a Unique Numeric ID • So Anything can Link to Anything In Either
DirecPon • Bus Stop To Tube Exit • MulPple Tube Exit to Tube StaPon • Tube StaPon To Railway StaPon • Railway StaPon To Plaporms • Railway StaPon To Taxi rank
• Enables mulP-‐mode in single topology • Some things may have a locaPon or link to a map
TOID • But can’t just link everything via a OS map enPty
ID (TOID)
• Extra Nodes & Edges Build Hierarchy • A sensor is on a road segment • MulPple road segments make up a journey leg • MulPple journey legs make up a route run
Bus 1 Stop at X,Y
(Vertex)
Tube StaPon Exit X,Y (Vertex)
Walk: 3 Min (Edge)
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 24
Step Two … Consistent Time and Agribute Structure
• You have at least 7 different representaPons of Time • To simplify linkage seqle on a single reference representaPon of Pme and Pme intervals
• All your different data sources have an underlying structure • Use it to make searching and linking feasible and Pmely
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 25
To ConOnue …
• TacPcally • Please do release as much reference data as you can.
• Be kind to Web developers and also release shape files as GeoJson with Lat/Long.
• Strategically
• Build the coherent model. • Please involve us.
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 26
For More InformaOon
• Let’s Have A Chat • mary-‐[email protected] • [email protected]
• See Also: • Processing Graphs: Spark GraphX
• hqp://spark.apache.org/docs/latest/graphx-‐programming-‐guide.html
15/11/15 © 2015 Mandrel Systems Limited, All Rights Reserved 27