global big data conference hyderabad-2aug2013- finance/manufacturing use cases
Post on 21-Oct-2014
3.698 views
DESCRIPTION
Financial institutions today are under intense pressure to provide more value add to the customers, reduce IT costs and also grow year to year. This challenge has been further complicated by huge amounts of data being generated as well as mandatory federal compliances in place. Similarly, Manufacturing industry today also is facing the challenge to process huge amount of data in real time and predict failures as early as possible to reduce cost and increase production efficiency. The session will cover some high level Big Data use cases applicable to financial and manufacturing domain and how big data technologies are being used successfully to solve these challenges using some examples in credit card/banking industry in financial domain and semi-conductor production in manufacturing domain.TRANSCRIPT
1
Finance and Manufacturing Big Data Use Cases and Solutions
Sanjay SharmaPrincipal Architect
August 2013
Impetus Big Data Services
2
Big Data Platform Implementation Operations and Visualization
Business Analyticsand Data Science
Solution Architecture, POC and Production planning
Technology strategy, Use Case development & Validation
BUSINESS PROCESS
MANAGEMENT
Assessment
Solution Modeling
Solution Analysis
People, Process,
Technology Impact
Analyze & Optimize
Objectives & Strategy Model
© 2013 Impetus Technologies
Big Data : Value Drivers
© 2013 Impetus Technologies
Financial World: Key Challenges
Business Challenges
• Fraud• Regulatory
Compliances• Customer Insights• Risk Handling
Technical Challenges
• Reduce IT costs• Do more with
LESS• Unstructured data
© 2013 Impetus Technologies
Financial World: Some Use Cases
Fraud Detection and
Analysis
Risk Management
Customer Insights/Mana
gement
Micro targeting
Trade/Payments Analytics
Long Term ‐Storage & Analytics
New Opportunities
Reputational Risk
Marketing Campaigns
© 2013 Impetus Technologies
Financial World: Technical Solution Challenges
• Regulatory Compliances• Machine Learning based supervised and
unsupervised analytics
Large Data Storage and Advanced
Analytics
• Mash up structured and unstructured data• Enrich TX with NLP/Text analyticsUnstructured Data
• Customer level Profiling/ Recommendations
• Transaction/Trade Ticker level analytics
Individual Level Analytical Processing
© 2013 Impetus Technologies
Big Data Solution: Building Blocks
© 2013 Impetus Technologies
Big Data Use Case Solution: Example
• Major Credit Card Company• Traditionally Oracle/ DB2 + SAS +
Informatica• Huge data– Existing solutions/ infrastructure
inadequate to meet new business requirements and data growth cost effectively
– Successfully revamping to Big Data Platform
© 2013 Impetus Technologies
Big Data Use Case Solution: Example
Step 1: Hadoop POC with Sentiment
Analysis
Step 2: Hadoop Distribution Selection
Step 3: Production-ized Recommendation
Engine
Step 4: Production-ized Negative Fraud
Detection ($100+mlln)Step 5: …
Hive
Hadoop Platform
HBase
Oozie
Mahout + NLP
Transactions
Unstructured Sources
(Email, Surveys, CRM notes, Social Feeds etc.)
Sqoop
Flume
MapRed
BI/ Visualization
Analytics- SAS/R
Real Time DeliveryRDBMS/API
RDBMS/MPP
Informatica
ETL/ELTL
Advantages= PB scale= ETL and Analytical Offloading= Prescriptive Analytics than Predictive
© 2013 Impetus Technologies
Manufacturing Domain: Challenges
Business Challenges
• Early preventive maintenance/repair• Real time/ near real time
actionable response• Improve productivity/Margin
Recovery• Reduce wastes, improve
efficiency• Improve Yield
• Supply Chain• Optimize supply chain
Technical Challenges
• High Ingestion Rates• Sensor/ tool data with sub
second ingestion requirements
• Millions of read/writes per second
• Complex log formats• Semi-structured data
• Huge amount of data• TB/PB of data storage for
deeper analytics
© 2013 Impetus Technologies
Manufacturing Domain: Architectural Building Blocks
Real Time AnalyticsStreaming Analytics + CEP
Hadoop
Advanced Analytics
Rules and Alerts Machine Learning
Loopback for RuleEnhancement/ Enrichment
NoSQL + Search
Machine Data
© 2013 Impetus Technologies
Machine Data
Ingestion Engine
(Real time + Batch
components)
Real Time Processing
Engine(CEP/Analytics/
Rule Engine)
Real Time Data
Storage Engine
(Store + Indexing/Searc
h)
Business Process Engine(Business Process+ Rule management)
Kafka/ Storm Storm + Esper Cassandra + Solr
JBoss Drools/jBPM
Manufacturing Domain: Data Ingestion/ Streaming – Customer Example
© 2013 Impetus Technologies
HBase + Elastic SearchSpark Streaming
Hadoop
Hive/PIG/Shark
Rules and Alerts Machine Learning
Loopback for RuleEnhancement/ Enrichment
NoSQL + Search
Machine Data
Manufacturing Domain: Reference Architecture-Cloud– Customer Example
© 2013 Impetus Technologies
• Software Solutions and Services Company• Leader in Innovation led Technology services• 17 years of customer success, 1500 people
across US/ India• Big Data, Enterprise Mobility, Test and
Performance Engineering, Carrier Grade Large Systems
• Vendor neutral, open source contributor with Big data accelerator products
Impetus Technologies
