Download - Managing your Assets with Big Data Tools
![Page 1: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/1.jpg)
Managing your
Assets with Big Data Tools
Karthigai Muthu, MachinePulse
![Page 2: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/2.jpg)
Big Data value proposition
Big Data Technology Stack
Agenda
![Page 3: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/3.jpg)
Source: Wikipedia
Hype Cycle for Emerging Technologies
![Page 4: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/4.jpg)
Sources of data
25+ TBs oflog data
every day 2+
billion
people
on the
Web
by end
2011
30 billion
RFID tags
today
(1.3B in
2005)
4.6
billion
camera
phones
world
wide
100s of
millions of
GPS
enabled
devices
sold
annually
76 million smart
meters in 2009…
200M by 2014
12+ TBsof tweet data
every day
? T
Bs o
fd
ata
eve
ry d
ay
![Page 5: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/5.jpg)
What makes Data Big
Characteristics Description Attributes Drivers
Volume The amount of data generated or
intensify that must be ingested,
analyzed and managed to make
decision based on complete data
analysis
Exabyte (EB)
Zettabyte (ZB)
Yottabyte (YB)
Increase in data sources
Higher resolution sensors
Scalable infrastructure
Velocity How fast the data is being
produced and changed and the
speed at which is transformed into
insight
Batch
Near real time
Real time and Streams
Rapid feedback loop
Improved throughput connectivity
Competitive advantage
Pre-computed information
Variety The degree of diversity of data from
sources both inside and outside an
organization
Degree of structure
Complexity
M2M/IoT
Social Media
Genomics
Video and Mobile
Veracity The quality and provenance of data Consistency
Completeness
Ambiguity
Integrity
Cost
Need of traceability and justification
![Page 6: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/6.jpg)
Big Data’s Greatest Power: Predictive Analytics
![Page 7: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/7.jpg)
What’s driving Big Data
- Ad-hoc querying and reporting
- Data mining techniques
- Structured data, typical sources
- Small to mid-size datasets
- Optimizations and predictive analytics
- Complex statistical analysis
- All types of data, and many sources
- Very large datasets
- More of a real-time
![Page 8: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/8.jpg)
Big Data:
Batch Processing &
Distributed Data StoreHadoop/Spark;
HBase/Cassandra/MongoDB
BI Reporting
OLAP &
Data warehouse
Business Objects, SAS,
Informatica, Cognos
other SQL Reporting
Tools
Interactive
Business
Intelligence &
In-memory
RDBMSQlikView, Tableau,HANA
Big Data:
Real Time &
Single View
Graph Databases
The Evolution of Business
Intelligence
1990’s 2000’s
2010’s
Speed
Scale
Scale
Speed
![Page 9: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/9.jpg)
Solving business problem with big data
![Page 10: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/10.jpg)
Formulation of big data strategy
People31%
intent20%
Data16%
Tools33%
![Page 11: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/11.jpg)
Companies Market share in Big Data
![Page 12: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/12.jpg)
Big Data Investments
![Page 13: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/13.jpg)
Priority for big data across industry
![Page 14: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/14.jpg)
Are you aware the risk of not implementing Big Data in
your company
![Page 15: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/15.jpg)
Big data changed connected things to Internet
of Everything(IoE)
![Page 16: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/16.jpg)
How the industry can leverage from big data
![Page 17: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/17.jpg)
Challenges in implementing the big data
![Page 18: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/18.jpg)
Returns of Investment(ROI)
![Page 19: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/19.jpg)
Merge
Optimize
Respond
Empower
How do companies get MORE from big data
![Page 20: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/20.jpg)
Are you planning to launch your new product.
![Page 21: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/21.jpg)
Customer 360`
Customer
Social
Media
Gaming
Entertain
Banking
Finance
Our
Known
History
Purchase
![Page 22: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/22.jpg)
Real-Time Analytics/Decision Requirement
Customer
Influence
Behavior
Product
Recommendations
that are Relevant
& Compelling
Friend Invitations
to join a
Game or Activity
that expands
business
Preventing Fraud
as it is Occurring
& preventing more
proactively
Learning why Customers
Switch to competitors
and their offers; in
time to Counter
Improving the
Marketing
Effectiveness of a
Promotion while it
is still in Play
![Page 23: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/23.jpg)
IoT+Big Data = IoE(Internet-of-Everything)
![Page 24: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/24.jpg)
Big Data is a factor that will, to a large extent, determine the future
growth rate in the M2M industry
M2M will connect increasingly more nodes that will provide data from
endpoints.
Data will be more granular, more frequent, and more accurate, with
bigger data sets or even live data streams
Large volume of endpoint connections IPv4 addressing scheme can’t
accommodate everything(sensors, smart phones, smart factories, smart
grids, smart vehicles, controllers, meters ) that it requires IPv6
IoE= Convergence of IoT, Big Data Analytics ,Cloud Computing and
other technologies is collectively called as Internet of Everything
Role of Big Data in M2M/IoT
![Page 25: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/25.jpg)
Meeting the need for speed
Data understanding
Maintaining data quality
Displaying the meaningful result
Challenges of Big Data in M2M/IoT
![Page 26: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/26.jpg)
IoT/M2M Applications..
![Page 27: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/27.jpg)
Personal IoT: the scope is a single person, such as a smartphone
equipped with GPS sensor or a fitness device that measures the heart
rate. This is one of the fastest growing, consumer-oriented areas of IoT.
Group IoT: the scope is a fairly small group of people, such as a family
in a smart house, co-workers in a van or a group of tourists. This is one
of the most challenging areas and is still in its early phase.
Community IoT: the scope is a large group of people, potentially
thousands and more; usually this is in a public infrastructure context,
such as smart cities or smart roads. This is a young and potentially
promising IoT area.
Industrial IoT: the scope can be within an organization (smart factory)
or between organizations (retailer supply chain). This is arguably the
most established and mature part of IoT.
Big Data Use Cases – IoT/M2M
![Page 28: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/28.jpg)
Agriculture - sensors can be deployed on farm machinery in order to provide data about
the equipment, soil temperature, moisture, etc.
Buildings/Smart Homes - Building sensors be used to help facility managers become
more proactive about ensuring that their buildings operate at peak efficiency.
Communities – Smart cities make use of parking space availability systems, intelligent
traffic monitoring systems, intelligent highways, weather-adaptive street lighting, and
more.
Healthcare – Infant monitors, smart diapers, pills with ingestible sensors are just some of
the IOT-based devices.
Manufacturing – factories with sensors can improve operations, product quality, and
decrease safety hazards.
Smartphones – can control everything from door locks, thermostats, light bulbs, vacuum
cleaners, and more.
Utilities – smart water meters can be used to reduce water leaks. Smart electric grids
can adjust rates depending on usage.
Wearables – Smart watches, fitness trackers and health monitors may become primary
source for human-related data, and can also be used in sports, retail, travel and
manufacturing.
Big Data Use cases – IoT/M2M
![Page 29: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/29.jpg)
1. Device Maintenance:
a. Time for next patch upgrade
b. Energy management
c. Inventory management and track replacement
2. Proactive Healthcare:
Capture and analyze real time data from medical monitors to predict
potential health problems before patients manifest clinical signs of
infection.
3. Monetize Machine Data:
a. Monitor performance, usage and capacity details to uncover up-sell
and cross-sell opportunities
b. Maximize the lifespan and performance of high value medical assets
Benefits of Big Data Analytics in M2M/IoT
![Page 30: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/30.jpg)
4. Optimize Support Operations:
a. Reduce MTTR and support escalations
b. Preempt failures with proactive support
c. Troubleshoot with accurate information
d. Proactive consultation to customers on approaching
expiry dates
Benefits of Big Data Analytics cont..
![Page 31: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/31.jpg)
Big Data Analytics Stack
![Page 32: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/32.jpg)
Lamda Architecture
![Page 33: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/33.jpg)
![Page 34: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/34.jpg)
Batch processing
- Gathering of data and processing as a group at one time.
- Jobs run to completion
- Data might be out of date
Real-time processing
- Processing of data that takes place as the information is being
entered.
- Run for ever
Batch vs. Real-Time processing
![Page 35: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/35.jpg)
Apache Storm is a free and
open source distributed real-time
computation system.
Storm makes it easy to reliably
process unbounded streams of data,
doing for real-time processing what
Hadoop did for batch processing
Storm
![Page 36: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/36.jpg)
Stream Processing
Fast
Scalable
Fault Tolerant
Reliable
Storm Is
![Page 37: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/37.jpg)
Tuple
![Page 38: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/38.jpg)
Streams
![Page 39: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/39.jpg)
Spouts
![Page 40: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/40.jpg)
Bolts
![Page 41: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/41.jpg)
Topologies
![Page 42: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/42.jpg)
Reliable Processing
![Page 43: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/43.jpg)
Reliable Processing
![Page 44: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/44.jpg)
Groupings are used to decide to which task in the
subscribing bolt (group) a tuple is sent.
Possible Groupings:
- Shuffle
- Fields
- All
- Global
- None
- Direct
- Local or Shuffle
Stream Grouping
![Page 45: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/45.jpg)
Storm Cluster View
![Page 46: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/46.jpg)
Fault Tolerance
![Page 47: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/47.jpg)
Fault Tolerance
![Page 48: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/48.jpg)
Fault Tolerance
![Page 49: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/49.jpg)
Fault Tolerance
![Page 50: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/50.jpg)
Fault Tolerance
![Page 51: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/51.jpg)
Parallelism
![Page 52: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/52.jpg)
Parallelism
![Page 53: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/53.jpg)
Apache Storm Real-time -Use cases
Segment Prevent Use Cases Optimize Use Cases
Financial Services Securities fraud
Operational risks & compliance
violations
Order routing
Pricing
Telecom Security breaches
Network outages
Bandwidth allocation
Customer service
Retails Shrinkage
Stock outs
Offers
Pricing
Manufacturing Preventative maintenance
Quality assurance
Supply chain optimization
Reduced plant downtime
Transportation Driver monitoring
Predictive maintenance
Routes
Pricing
Web Application failures
Operational Issues
Personalized content
![Page 54: Managing your Assets with Big Data Tools](https://reader033.vdocuments.net/reader033/viewer/2022042817/55a614871a28ab04338b45ee/html5/thumbnails/54.jpg)
The End