loginext big data workshop nasscom
TRANSCRIPT
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
1 What?
2 Why?
3 Where?
4 How?
AGENDA
1
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
WHAT?
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
How Much Data You Consume & Produce Everyday?
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
How Much Data You Consume & Produce Everyday?
1 MB
5 MB 0.5 MB
0.5 MB
0.5 MB
1 MB
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
Big Data Facts
200 Million Emails 1.8 Million FB Likes
Every Minute…
5
300K Tweets 48 hours YouTube Videos
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
The Mindset Th4Vsindset
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
WHY?
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
The Value
COLLECT
STORE
ANALYZE
VISUALIZE
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
Be daring, be different, be
impractical, be anything that
will assert integrity of
purpose and imaginative
vision against the play-it-
safers, the creatures of the
commonplace, the slaves of
the ordinary.
Data Around Us…
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
Be daring, be different, be
impractical, be anything that
will assert integrity of
purpose and imaginative
vision against the play-it-
safers, the creatures of the
commonplace, the slaves of
the ordinary.
Make Things Smarter…
• Critical Alerts
• Technical Problems
• Service Alerts
• Headlight On Warning
• Location Information
• Towing Alerts
• Crash Alerts
• Over Speeding Alerts
• Fuel Economy
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
Be daring, be different, be
impractical, be anything that
will assert integrity of
purpose and imaginative
vision against the play-it-
safers, the creatures of the
commonplace, the slaves of
the ordinary.
Analytics
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
Be daring, be different, be
impractical, be anything that
will assert integrity of
purpose and imaginative
vision against the play-it-
safers, the creatures of the
commonplace, the slaves of
the ordinary.
Analytics
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
Visualization
https://housing.com/dsl/heatmaps/mumbai/rent
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
14
What is the Maximum Speed?
Day Speed (kmph) Day Speed (kmph) Day Speed (kmph)
1 30 11 60 21 90
2 117 12 115 22 60
3 45 13 35 23 63
4 90 14 121 24 68
5 60 15 40 25 79
6 63 16 0 26 108
7 68 17 0 27 40
8 79 18 30 28 123
9 108 19 42 29 70
10 110 20 45 30 83
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
15
How Many Days the Speed was Above 100?
Day Speed (kmph) Day Speed (kmph) Day Speed (kmph)
1 30 11 60 21 90
2 117 12 115 22 60
3 45 13 35 23 63
4 90 14 121 24 68
5 60 15 40 25 79
6 63 16 0 26 108
7 68 17 0 27 40
8 79 18 30 28 123
9 108 19 42 29 70
10 110 20 45 30 83
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
16
How Many Days Did You Spend at Home?
Day Speed (kmph) Day Speed (kmph) Day Speed (kmph)
1 30 11 60 21 90
2 117 12 115 22 60
3 45 13 35 23 63
4 90 14 121 24 68
5 60 15 40 25 79
6 63 16 0 26 108
7 68 17 0 27 40
8 79 18 30 28 123
9 108 19 42 29 70
10 110 20 45 30 83
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
3 Seconds...
0
20
40
60
80
100
120
140
160
180
200
220
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
Cars
Sports
WHERE?
Manufacturing
Politics
Education
E-commerce
Health Care
Logistics
Homes
18
Retail
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
19
What we see!!
Lot of Data
?
What it is!!
Complex Chain
Big Data & Logistics
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
20
Saves 60 Million USD
Big Data & Logistics
https://www.youtube.com/watch?v=ZHqgpVsUfhI
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
21
Big Data & Logistics
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
22
Predictive & Proactive
• Know Where The Shipments Are
• Know How to Deliver Better & Cheaper
• Make It Easier To Digest For Your Field-Team
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
Case Study: Last Mile Delivery
Real-time Increased Visibility & Customer Satisfaction
Reduction in Delivery Times
15 %
Increase in Resource Utilization 7%
Reduction in Delays & Errors 12 %
100,000+
5000+
250,000+ KMS
23
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
BREAK
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
HOW?
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
BIG DATA LANDSCAPE
Source: hEp://www.forbes.com/sites/davefeinleib/2012/07/24/big-‐data-‐trends/
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
DATA ARCHITECTURE
Source: hEp://www.tricoresolu3ons.com/resources/blog/best-‐prac3ces-‐in-‐bi-‐think-‐big-‐start-‐small-‐ensuring-‐a-‐successful-‐bi-‐big-‐da
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
DATA ARCHITECTURE
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
STORAGE MongoDB
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
Data Sources
“In God We Trust, All Others Bring Data” – W.E. Deming
Mobile Apps for Delivery Boys
Flight Data
Matchbox Sized Smart Tags
External Sensors & ERP Data
Train Data
Traffic & Weather Data
30
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
• Easy & Dynamic Scalability
• Flexible Schema
• Semi-structured & Unstructured Data
• Impedance Mismatch Between Application and Database
• Relationship Between Data Not Always Required in Database
• Better Performance
• Low Cost
• Auto-Sharding
• Distributed Query Support
• Integrated Caching
Why NoSQL?
The Demands of Your Database Are Changing
31
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
NoSQL - Scalability
32 Source: hEp://www.couchbase.com/nosql-‐resources/what-‐is-‐no-‐sql
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
NoSQL - Scalability
33 Source: hEp://www.couchbase.com/nosql-‐resources/what-‐is-‐no-‐sql
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
NoSQL – Schemaless
34
Storage Is Not A Concern Any More
Source: hEp://www.couchbase.com/nosql-‐resources/what-‐is-‐no-‐sql
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
NoSQL Types
KEY- VALUE
WE OFFER THE BEST THERE IS 35
GRAPH COLUMN DOCUMENT
OrientDB
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
Why MongoDB?
36
• Document Storage
• BSON - Binary JSON
• Easy Import & Querying (mongoimport -f filename.json)
• Friendly with JS
• Super Easy Installation
• Querying
• Rich JS Based Query Syntax
• Allows Deep Nested Queries
• Very Powerful For Analysis
“MongoDB (from "humongous") is an open-source document database, and the leading
NoSQL database” – mongodb.org
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
MongoDB – Geospatial Features
37
• GeoJSON Objects
• Point
• LineString
• Polygon
• MultiPoint
• MultiLineString
• MultiPolygon
• GeometryCollection
• Legacy Coordinate Pairs
“MongoDB offers a number of indexes and query mechanisms to handle
geospatial information”
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
MongoDB – Query Operations
38
• Inclusion - $geoWithin
{ <location field>: { $geoWithin: { $geometry: { type: <"Polygon”> , coordinates: [ <coordinates> ] } } } }
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
MongoDB – Query Operations
39
• Intersection- $geoIntersects
{ <location field>: { $geoIntersects: { $geometry: { type: "Polygon” , coordinates: [ <coordinates> ] } } } }
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
MongoDB – Query Operations
40
• Proximity- $near
{ $near: { $geometry: { type: "Point" , coordinates: [ <longitude> , <latitude> ] }, $maxDistance: <distance in meters>, $minDistance: <distance in meters> } }
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
MongoDB – Talent
41 Source: hEp://blogs.the451group.com/informa3on_management/2014/10/01/nosql-‐linkedin-‐skills-‐index-‐september-‐2014/
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
42
LogiNext - All In One
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
Case Study: Avg. E-Commerce Company
43
100,000+ Shipments Per Day
10,000+ PIN Codes Served
300+ Hubs Across India
Shipments Get Delayed
Hands Change For Each Delivery
40%
8%
6 hrs
Of The Time Goes Without Visibility Into Location of Package
7
Saved Per Month
Idle Time for Packages
7 Lacs INR
20%
Business
Challenges Results
More Accuracy in Tracking SLAs
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
Share Your Use Case
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
ONLINE
MySQL
MongoDB
OFFLINE
Hadoop
EDW
Big Data Management
45
Real-time Low-latency High availability
Long-running High-latency Availability Low Priority
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
46
Big Data Management
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
VISUALIZATION TECHNIQUES
Tag Cloud
47
Heat Map
Spatial Information Flow Network Graphs
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
VISUALIZATION TOOLS & TECHS
48
• Tools
• Tableau
• SAS
• Maps
• Google Maps
• OpenStreetMaps
• CartoDB
• MapMyIndia
• Charts
• D3.js
• HighCharts
• GoogleCharts
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
CHALLENGES
Insufficient Budget
49
Adoption
Technology
Technical Skills
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
OPPORTUNITY
DATA iS GROWING
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
Please don’t hesitate
Questions
Copyright © LogiNext Solu3ons Inc. All Rights Reserved. Visit www.loginextsolu3ons.com/termsofuse for more details.
Thank You
for your precious time and attention