best practices monitoring multi-tenant hadoop
TRANSCRIPT
Confidential2
TRUSTEDby over 10,000
companies as their big data app platform
BACKEDby top Silicon Valley
investors True Ventures, Rembrandt
VP, Bain Capital
FOUNDED in 2008, with
headquarters in San Francisco
Confidential
WHY NOW?
3
As Big Data applications become the engine driving innovation and new revenue models, they graduate to 1st class citizens demanding higher standards of quality, reliability, and manageability
Confidential
PERFORMANCE MANAGEMENT FOR BIG DATA APPLICATIONS
your big data apps
MONITOR to resolve
issues fasterbig data apps
more effectively
MANAGE
COLLABORATE
Confidential 7Confidential & Proprietary
ACHIEVING OPERATIONAL EXCELLENCE
• Performance monitoring and visibility for all big data applications- See everything in one place with a single view of all data applications real-time performance
metrics across the entire enterprise• Instantly understand the business context of a problem
- Knowing you have a problem is one thing but understanding the end to end & downstream dependencies, the quality and service guarantees, who is impacted, the owner(s) and the business criticality of the failure is game changing
• Easily collaborate across teams to resolve issues faster- Multi-tenant architectures are inherently chaotic. Accelerate issue resolution by delivering the right
information to the appropriate individuals and teams so everyone is informed and can work together to resolution
• Manage the business of big data- Get the operational visibility and controls you need to effectively manage priorities, policies,
capacity, compliance and utilization.
Balancing a fully operational data processing operation and customer demands is challenging
1
2
3
4
Confidential 8Confidential & Proprietary
PERFORMANCE MONITORING & VISIBILITY
Comprehensive view of all your data processing execution Fully visualize your entire data pipeline
See all successful, failed, pending processes
…quickly segment your aggregate view by app name, team , cluster etc…
Confidential 9Confidential & Proprietary
PERFORMANCE MONITORING & VISIBILITY
Comprehensive view of all your data processing execution
Fully visualize your entire data pipeline
Join Operations Imbedded Hive Flows
Functions, Filters and Groupings
ResultsSource SYNC
Confidential 10Confidential & Proprietary
PERFORMANCE MONITORING & VISIBILITY
Pinpoint bottlenecks and identify causes
EXECUTING WAITING
Detailed Performance Stats
See how all apps consume resources as they run
Confidential 11Confidential & Proprietary
PERFORMANCE MONITORING & VISIBILITY
Pinpoint bottlenecks and identify causes See how all apps consume resources as they run
Confidential 12Confidential & Proprietary
PERFORMANCE MONITORING & VISIBILITY
View Application and Unit of Work Details Surface Important Metrics and Information
See roll-up metrics on applications
Support monitoring Hive Server applications
See the status of each Unit of Work including HQL statements. Drill down to
get UoW details.
Confidential 13Confidential & Proprietary
PERFORMANCE MONITORING & VISIBILITY
View Application and Unit of Work Details Surface Important Metrics and Information
Counters Properties Environment
Confidential 14Confidential & Proprietary
ACHIEVING OPERATIONAL EXCELLENCE
• Performance monitoring and visibility for all big data applications- See everything in one place with a single view of all data applications real-time performance
metrics across the entire enterprise• Instantly understand the business context of a problem
- Knowing you have a problem is one thing but understanding the end to end & downstream dependencies, the quality and service guarantees, who is impacted, the owner(s) and the business criticality of the failure is game changing
• Easily collaborate across teams to resolve issues faster- Multi-tenant architectures are inherently chaotic. Accelerate issue resolution by delivering the right
information to the appropriate individuals and teams so everyone is informed and can work together to resolution
• Manage the business of big data- Get the operational visibility and controls you need to effectively manage priorities, policies,
capacity, compliance and utilization.
Balancing a fully operational data processing operation and customer demands is challenging
1
2
3
4
Confidential 15Confidential & Proprietary
UNDERSTANDING THE CONTEXT OF THE PROBLEM
Quickly identify the operational and business context
Quickly and easily identify execution errors
Not all problems are created equal
App Name Owner Organization Cluster A or B Privacy Level Production or Dev Custom Tags More …
Confidential 16Confidential & Proprietary
UNDERSTANDING THE CONTEXT OF THE PROBLEM
Quickly identify the operational and business context
Quickly and easily identify execution errors
Confidential 17Confidential & Proprietary
ACHIEVING OPERATIONAL EXCELLENCE
• Performance monitoring and visibility for all big data applications- See everything in one place with a single view of all data applications real-time performance
metrics across the entire enterprise• Instantly understand the business context of a problem
- Knowing you have a problem is one thing but understanding the end to end & downstream dependencies, the quality and service guarantees, who is impacted, the owner(s) and the business criticality of the failure is game changing
• Easily collaborate across teams to resolve issues faster- Multi-tenant architectures are inherently chaotic. Accelerate issue resolution by delivering the right
information to the appropriate individuals and teams so everyone is informed and can work together to resolution
• Manage the business of big data- Get the operational visibility and controls you need to effectively manage priorities, policies,
capacity, compliance and utilization.
Balancing a fully operational data processing operation and customer demands is challenging
1
2
3
4
Confidential 18Confidential & Proprietary
NURTURE A CULTURE OF OPERATIONAL EXCELLENCE
Ensure that business, development, IT operations can collaborate seamlessly when it matters
Confidential 19Confidential & Proprietary
COLLABORATIONShare and set alerts with any custom views and analytics
Share privately with an existing team or publically for all users to reference.
Confidential 20Confidential & Proprietary
COLLABORATIONCreate JIRA issues with views and data for quickly
collaborating to resolve performance problemsIntegrate alerts with popular notification platforms
like HipChat, PagerDuty, & Nagios
With one click, create a Jira issue with a link to this view
Confidential 21Confidential & Proprietary
COLLABORATIONCreate JIRA issues with views and data for quickly
collaborating to resolve performance problemsIntegrate alerts with popular notification platforms
like HipChat, PagerDuty, & Nagios
Automatically send app status notifications via webhooks or JMX
Confidential 22Confidential & Proprietary
ACHIEVING OPERATIONAL EXCELLENCE
• Performance monitoring and visibility for all big data applications- See everything in one place with a single view of all data applications real-time performance
metrics across the entire enterprise• Instantly understand the business context of a problem
- Knowing you have a problem is one thing but understanding the end to end & downstream dependencies, the quality and service guarantees, who is impacted, the owner(s) and the business criticality of the failure is game changing
• Easily collaborate across teams to resolve issues faster- Multi-tenant architectures are inherently chaotic. Accelerate issue resolution by delivering the right
information to the appropriate individuals and teams so everyone is informed and can work together to resolution
• Manage the business of big data- Get the operational visibility and controls you need to effectively manage priorities, policies,
capacity, compliance and utilization.
Balancing a fully operational data processing operation and customer demands is challenging
1
2
3
4
Confidential 23Confidential & Proprietary
MANAGING THE BUSINESS OF BIG DATA
Powerful Search& Custom Views
Monitor and analyze business critical data processes
Easily detect apps that violate SLA’s and policies
Views can be just for you, for a team, public or private, and categorized under general status or application specific
Quickly find and filter what you are looking for and save as a custom view
`
Confidential 24Confidential & Proprietary
MANAGING THE BUSINESS OF BIG DATA
Powerful Search& Custom Views
Monitor and analyze business critical data processes
Easily detect apps that violate SLA’s and policies
Metrics for all apps on the production cluster that failed to execute in under 30 minutes…
…or all apps that use more than their allotment of mappers
Confidential 25Confidential & Proprietary
MANAGING THE BUSINESS OF BIG DATA
Powerful Search& Custom Views
Monitor and analyze business critical data processes
Easily detect apps that violate SLA’s and policies
Measure and compare key analytics and metrics of application performance trends over
any period of time…
…Set SLA thresholds, observe and detect performance anomalies
Confidential 26Confidential & Proprietary
MANAGING THE BUSINESS OF BIG DATA
Segment performance by team, by department or custom tags for role-based views, chargeback models, and capacity planning
Reporting & Planning
Marketing
Sales
Compliance
Data science team
QA cluster
Production cluster
Performance of all apps owned by the Data Scientist team
Confidential 27Confidential & Proprietary
MANAGING THE BUSINESS OF BIG DATA
Segment performance by team, by department or custom tags for role-based views, chargeback models, and capacity planning
Reporting & Planning
Deeper analytics
Compliance
Chargeback reporting
Capacity planning
Any custom search filter can be saved and exported for reporting, to generate an audit
artifact, or to drop into your favorite spreadsheet
Confidential 28Confidential & Proprietary
MANAGING THE BUSINESS OF BIG DATAVisualize Lineage - See exactly how each app ingests,
manipulates and outputs dataFurther inspect lineage by detecting apps that write to,
or read from, a given dataset
Confidential 29Confidential & Proprietary
MANAGING THE BUSINESS OF BIG DATAVisualize Lineage - See exactly how each app ingests,
manipulates and outputs dataFurther inspect lineage by detecting apps that write to,
or read from, a given dataset
Show all apps that interact with the dataset in “rain.txt”
Confidential 30Confidential & Proprietary
MORE INFORMATION…
Visit us at www.driven.io