cto perspective on capturing the potential of big data in a service provider
TRANSCRIPT
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
1/17
CTO Perspective on Capturing tPotential of Big Data in a ServicProviderJrgen UrbanskiBoard Member Big Data & Analytics of BITKOM (German IT Industry Views and opinions expressed are not necessarily those of his emplo
T-Systems, the enterprise arm of Deutsche Telekom.
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
2/17
Big Data 1956
IBM 305
RAMAC5 MB!
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
3/17
Volume
Velocity
Variety
Big Data 1956
IBM 305
RAMAC5 MB!
Big Data 2013
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
4/17
Big Data Use Cases for Telcos
Business Intelligence Marketing & Sales Service & Operat
! 360 degree view ofcustomer value
! Targeted TV advertising,serving the mostprofitable TV ad to
individual viewers basedon analysis of their
viewing behavior (e.g.,what ads prompt theuser to switch channels)
! Personalized marketingcampaigns, integratinganalytics across various
marketing channels
! Big data as a product
! Network maintenance anoperational intelligence for planning and customer exp
! Analytics and archiving of c(CDRs) on Hadoop for com
disputes, and congestion m
! Real time analytics of CDRand in-memory technologyon pre-paid services (e.g.,
! Security analytics (e.g., intr! M2M device telemetry anal
security and assisted living
! Log analytics for customer
! Enterprise datawarehouse offload
- Data landing zone- Active archive- Enterprise-wide data
lake augmenting theEDW
! Mainframe offload
1 3
2
= HighN
* A large mobile carrier might reach 1 billion new CDRs, ingesting 20 TB per day
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
5/17
The Situation
! Many EDWs are at capacity! Running out of budget before running out of
relevant data
! Older data archived in the dark, notavailable for exploration
Enterprise Data Warehouse Offload
Operational (44%)
ETL Processing (42%)
Analytics (11%)
DATAWAREHOUSE
Storage & Process
HADOOP
1
Operational (50%
Analytics (50%)
DATAWAREHOUS
The Solution
Cost is
1/10th
! Hadoop for data storage and pparse, cleanse, apply structuretransform
! Free EDW for valuable queries! Retain all data for analysis!
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
6/17
! Europes leading real estatemarketplace with data on...
! 1m properties listed currently! 20m properties cumulative! 6m saved searches! Geographical coordinates! Enriched by socio-demographic data on
19m properties
! Team! Product manager! Data scientists! 2 scrum teams
Data Products: ImmobilienScout (a DT subsidiary)2
! Market Navigator serv! Supports realtors in acquiri
customers
! Local market analysis helpssetting for rent and buy
! Integrates third-party data! Functionality includes! Price heat maps and trendi! Demand- and supply-side i! Local area information! Comparable transactions
The Situation The Solution
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
7/17
Turning Big Data into Products!2
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
8/17
! Analyze node utilization against customer experience indicators to secorrelates to experience
! Increase of dropped packets (IPDR data)! Increase of calls into the call center for customers associated to the
(customer experience data)
! Increase of requests to drop service (work order data)
Network Maintenance and Upgrades to Improve the
Customer Experience The Situation3
Approach
Challenge!
Poor visibility into how cable network congestion affects churn, and wnetwork upgrades produce the most incremental revenue
Hypothesis! Nodes considered to be causing customer experience issues can be p
maintenance and upgrade based on the value of the customers serve
node
Source: Zaloni project for a large cable provider
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
9/17
! Analysis that integratessubscriber and network
node data to seecorrelations betweennetwork congestion andcustomer experience
! 11 different data sources! 4m subscriber records, 12m
work orders, 9m calls, 42mIPDRs1, 20m Tivoli NPMs2
! Finding:Only smallnumber of nodes areresponsible for majority ofthe negative customerexperience
3
Source: Zaloni project for a large cable provider
1 IPDR = Internet Protocol Detail Record, provides information about IP-based service usage, usually to inform OSS and2 NPM = NetView Performance Monitor Messages.
NetworkNode
TNMPCMTS
Performanc
CompetitiveSpendData
SubscriberSubscriber
MasterSubscriber
Record
MarketingDemo-
graphics
CallerExperience
WorkOrders
Products
Equipment
Network Maintenance and Upgrades to Improve the
Customer Experience The Solution
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
10/17
Approach to Execution
Implications:
! Technology st! Target archite! Vendor select! New processe! New skills!
Privacy consid! Etc.
Architecture& Design Proof ofConcept Pilot
Production
Implemen-tation
Training1
1 Architect, Developer, Data Science and Admin
Agile
learningon eachproject
Programmaticsteering
Supply-side requirements:
! Higher capital efficiency! Lower upfront investment! Faster time to value! More rapid innovation! Market-facing differentiation!
Security & compliance
+A
= HighN
B
C
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
11/17
Technology Strategy: From Data Puddles to Data Lak
AVOID:
Systems separated by workloadtype due to contention
BigData
BU1
BigData
BU2
BigData
BU3
GOAL:
Platform that natively sumixed workloads as share
Big DataTransactions, Interactions, Obs
Refine Explore
Batch Interactive
A
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
12/17
Target Architecture: Modular against a Fragmented E
Physical Infrastructure
Data Processing
Batch ProcessingStreaming & Complex
Event ProcessingSearch
DataIntegration &Governance
Extract,Transform,
Load
Real Time &Batch
Ingestion
DataConnectors
Life CycleManagement
DEnc
Data I& Mten
IdenAc
Mana
Data
CusGat
1 Includes key value, document, graph and object data bases.
Application
Data Mining& Predictive
Geo-spatialVideo &Audio
Web &Social Media
Text &Semantics
OLAP
Presentation
AdvancedVisualization
ClientsReal-TimeMonitoring
Reports &Dashboards
Data Management
DistributedStorage &
Processing
(Scale-out)Relational DB
NoSQL DB1In-memory
DB(MPP) EDW
B
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
13/17
Hadoop Is the Foundation for Much of the Innovation
Physical Infrastructure
Data Processing
Batch ProcessingStreaming & Complex
Event ProcessingSearch2
DataIntegration &Governance
Extract,Transform,
Load
Real Time &Batch
Ingestion
DataConnectors
Life CycleManagement
DEnc
Data I& Mten
IdenAc
Mana
Data
CusGat
1 Includes key value, document, graph and object data bases.2 Solr and Lucene open source projects, also applicable outside Hadoop.
Application
Data Mining& Predictive
Geo-spatialVideo &Audio
Web &Social Media
Text &Semantics
OLAP
Hadoop Projects & Ecosystem Ad
Presentation
AdvancedVisualization
ClientsReal-TimeMonitoring
Reports &Dashboards
Data Management
DistributedStorage &
Processing
(Scale-out)Relational DB
NoSQL DB1In-memory
DB(MPP) EDW
Store first, ask
questions later (HDFS)
Parallel scale-out
processing (MapReduce)
Much cheaper
storage
Any data type,
including unstructured
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
14/17
Vendor Selection ConsiderationsCategory Select requirements
DataManagement
! Support of batch, interactive, online & streaming use cases! Full data lifecycle management
Operations! Rolling upgrades without service disruption and fallback capability! Support for end-to-end management and automation frameworks (e.g., P
Security! Granular role-based access control via AD, LDAP, Kerberos! Tenant, data, network and namespace separation in all services! Auditability
Infrastructure! Deployment flexibility: virtual on-premise environments, public clouds, ap
Strategic Fit
! Relevance of ISV ecosystem (notably EDW & BI) that is certified! Avoidance of vendor lock-in (open source vs. proprietary)
EXTC
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
15/17
! In 3 years, 50% of new data for enterpriseworkloads will land on Hadoop! Big Data can deliver value in every function of a telco! Big Data has a high return-on-investment, if you master the learning curve! Operators who embrace Hadoop today will see their business performance
away from those who are late to join the new world of Big Data
Our perspective
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
16/17
! Start Small, Grow Tall: Debunking Three Big Data Myths (Link)! WIRED Innovation Insights! October 4, 2013! Enterprises dont need petabytes of data, a small army of data scientists, not even a big budget to get a me
start with Big Data -- thanks to Hadoop.
! Hadoops Second Generation Offers More To Enterprises (Link)! Information Week! October 2, 2013! The first Hadoop tools weren't easy to deploy or manage. But the second-wave tools deliver great advances
! Hadoop! Coming soon to an enterprise data warehouse near you (Link)! TDWI! June 2013! Deutsche Telekoms perspective on how the open-source Hadoop ecosystem delivers powerful innovation
databases and business intelligence at a fraction of the cost of legacy systems.
! Been there, forked that: What the Unix-Linux schism can teach us about Hadoops futu! GigaOm! June 26, 2013! Concerned about proprietary and expensive forks of Hadoop, T-Systems Juergen Urbanski explains how to
are buying an open version of Hadoop or something you might later regret.
Further Reading
-
8/14/2019 CTO Perspective on Capturing the Potential of Big Data in a Service Provider
17/17
CTO Perspective on Capturing tPotential of Big Data in a ServicProviderJrgen UrbanskiBoard Member Big Data & Analytics of BITKOM (German IT Industry Views and opinions expressed are not necessarily those of his emplo
T-Systems, the enterprise arm of Deutsche Telekom.