denodo datafest 2016: what’s new in denodo platform – demo and roadmap
TRANSCRIPT
![Page 1: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/1.jpg)
O C T O B E R 1 8 , 2 0 1 6 S A N F R A N C I S C O B A Y A R E A , C A
#DenodoDataFest
RAPID, AGILE DATA STRATEGIESFor Accelerating Analytics, Cloud, and Big Data Initiatives.
![Page 2: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/2.jpg)
What’s New in Denodo Platform
Dr. Alberto Pan
Denodo, CTO
![Page 3: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/3.jpg)
Agenda1. Performance
2.Self-Service
3.Managing Large Deployments
4.Connectivity
5.Q & A
![Page 4: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/4.jpg)
What’s New
Some Recent and Upcoming Features
![Page 5: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/5.jpg)
Main Areas
Dynamic Query Optimizer for Big Data (Denodo 6)
Incremental queries (Denodo 6 Updates)
Embedded in-memory fabric (Denodo 7)
New Information Self-Service Tool (Denodo 6)
Information Self-service: Glossary and Collaboration Features (Denodo 7)
▪ Tighter integration with Data Governance and
Data Modeling Tools (Denodo 7)
Workload Management: Denodo Resource Manager (Denodo 6)
Monitoring and Diagnostic Tool (Denodo 6 Updates)
Solution Manager (Denodo 7)
New VDP Admin Tool (Denodo 6)
GIT Support (Denodo 6)
▪ Support for new data sources and publishing formats (continuous work)
▪ New Data Types (Denodo 7)
Performance in BigData Scenarios
Security, Governance and Self-service
Enterprise Wide Deployments
Connectivity and Data Transformation
![Page 6: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/6.jpg)
Move Processing to the Data
Process the data where it resides
Process the data locally where it resides
DV System combines partial results
Minimizes network traffic
Leverages specialized data sources
![Page 7: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/7.jpg)
7
How to Choose the Best Execution Plan?
Cost-Based Optimization in Data Virtualization
Data statistics to estimate size of intermediate result sets
Data Source Indexes (and other physical structures)
Execution Model of data sources: e.g. Parallel Databases VS Hadoop clusters VS Relational Databases
Features of data sources (e.g. number of processing cores in parallel database or Hadoop Cluster)
Data Transfer rate
Must take into account:
![Page 8: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/8.jpg)
8
Denodo has done extensive testing using queries from the standard benchmarking test
TPC-DS* and the following scenario
Compares the performance of a federated approach in Denodo with an MPP system where
all the data has been replicated via ETL
Customer Dim.2 M rows
Sales Facts290 M rows
Items Dim.400 K rows
* TPC-DS is the de-facto industry standard benchmark for measuring the performance of decision support solutions including, but not limited to, Big Data systems.
vs.Sales Facts290 M rows
Items Dim.400 K rows
Customer Dim.2 M rows
Denodo 6.0 ArchitecturePerformance Comparison – Logical Data Warehouse vs. Physical Data Warehouse
![Page 9: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/9.jpg)
9
Denodo 6.0 Architecture
Query DescriptionReturned
RowsTime Netezza
Time Denodo (Federated Oracle,
Netezza & SQL Server)
Optimization Technique (automatically selected)
Total sales by customer 1,99 M 20.9 sec. 21.4 sec. Full aggregation push-down
Total sales by customer and year between 2000 and 2004
5,51 M 52.3 sec. 59.0 sec Full aggregation push-down
Total sales by item brand 31,35 K 4.7 sec. 5.0 sec. Partial aggregation push-down
Total sales by item where sale price less than current
list price17,05 K 3.5 sec. 5.2 sec On the fly data movement
Performance Comparison – Logical Data Warehouse vs. Physical Data Warehouse
![Page 10: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/10.jpg)
10
Incremental QueriesNew Caching Mode for SaaS Data Sources
Merge cached data with delta changes from the data source
Real-time results with minimum latency
Data source needs to provide a way to obtain the delta changes
Get Leads Changed / Added since 1:00AM
CACHELeads updated at 1:00AM
Up-to-date Leads data
![Page 11: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/11.jpg)
Full Cache – Incremental queriesConfiguration
1. Cached data
3. Merged based on PK
2. New data from source
11
![Page 12: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/12.jpg)
12
Parallel In-Memory FabricEmbedded in-memory fabric fully integrated with cost optimization (Denodo 7)
Embedded in-memory fabric
MPP processing of costly local processing operations
External in-memory fabrics supported
Integrated with cost-based optimization
![Page 13: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/13.jpg)
Main Areas
Dynamic Query Optimizer for Big Data (Denodo 6)
Incremental queries (Denodo 6 Updates)
Embedded in-memory fabric (Denodo 7)
New Information Self-Service Tool (Denodo 6)
Information Self-service: Glossary and Collaboration Features (Denodo 7)
▪ Tighter integration with Data Governance and
Data Modeling Tools (Denodo 7)
Workload Management: Denodo Resource Manager (Denodo 6)
Monitoring and Diagnostic Tool (Denodo 6 Updates)
Solution Manager (Denodo 7)
New VDP Admin Tool (Denodo 6)
GIT Support (Denodo 6)
▪ Support for new data sources and publishing formats (continuous work)
▪ New Data Types (Denodo 7)
Performance in BigData Scenarios
Security, Governance and Self-service
Enterprise Wide Deployments
Connectivity and Data Transformation
![Page 14: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/14.jpg)
14
Information Discovery and Self-Service (1)
Graphically Expose Data Views to Business Users
Search and Query Data andMetadata
Browse data associations
Transform and combine views
Publish results to Denodo or your favourite reporting tool
Find more details at: datavirtualization.bloghttp://www.datavirtualizationblog.com/data-exploration-and-self-service-bi-welcome-to-the-dataweb/
![Page 15: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/15.jpg)
15
Information Discovery and Self-Service (2)
Browse associations between data views
![Page 16: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/16.jpg)
16
Information Discovery and Self-Service (3)
Inspect Data Lineage
![Page 17: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/17.jpg)
17
Information Discovery and Self-Service (4)
Search Content in All Views
![Page 18: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/18.jpg)
18
Information Discovery and Self-Service (and 5)
Query, Combine and Transform Data Views
![Page 19: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/19.jpg)
19
Information Self-Service Tool: 6.0 UpdatesEnhancements in 6.0 Updates
Support for Solr, Elastic Search in Global Search
See folders structure
See web services
Improved metadata search And Support for specifying field descriptions
![Page 20: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/20.jpg)
20
Information Self-Service Tool: Denodo 7 (1)Extended metadata and Components Catalog
Categorized/Tagged catalog of data components to associate views and business terms
Extended metadata fields
Ability to Edit Metadata
![Page 21: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/21.jpg)
21
Information Self-Service Tool: Denodo 7 (and 2)
Governance and Collaboration Features
Publish / share new components to the catalog
Governance:
- Approval process
- Stewards
Public and private comments
![Page 22: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/22.jpg)
Main Areas
Dynamic Query Optimizer for Big Data (Denodo 6)
Incremental queries (Denodo 6 Updates)
Embedded in-memory fabric (Denodo 7)
New Information Self-Service Tool (Denodo 6)
Information Self-service: Glossary and Collaboration Features (Denodo 7)
▪ Tighter integration with Data Governance and
Data Modeling Tools (Denodo 7)
Workload Management: Denodo Resource Manager (Denodo 6)
Monitoring and Diagnostic Tool (Denodo 6 Updates)
Solution Manager (Denodo 7)
New VDP Admin Tool (Denodo 6)
GIT Support (Denodo 6)
▪ Support for new data sources and publishing formats (continuous work)
▪ New Data Types (Denodo 7)
Performance in BigData Scenarios
Security, Governance and Self-service
Enterprise Wide Deployments
Connectivity and Data Transformation
![Page 23: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/23.jpg)
23
Denodo Resource Manager
Controlled Resource Allocation
1 Defines a rule that will be triggered for “app1” and users with the role “reporting”
2 For those request that fulfill the rule, if the CPU usage is greater than 85%, will apply the following:• Reduce thread priority• Reduce the number of concurrent requests• Limit the number of queued queries
![Page 24: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/24.jpg)
24
Monitor current state of servers and clusters
Inspect sessions, queries(with real-time trace), connections,...
Inspect data sources activity, cache load processes and content,...
Monitoring and Diagnostic Tool (1)Graphical Monitoring and Diagnosing of Servers and Clusters
Go back in time to the moment where a problem happened
Diagnose root cause of the problem
![Page 25: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/25.jpg)
25
Monitoring and Diagnosing Tool (2)Graphical Monitoring and Diagnosing of Servers and Clusters
State: Summary of the state of the server/environment
Resources: physical resources (memory, cpu,…)
Requests: including real-time execution trace
Session: Currently opened sessions, including client application
Cache: cache load processes, cache contents,...
Datasources: pools state, active requests,...
Threads: priorities, CPU usage,...
Errors: Inspect logged errors and warnings
… and many others
Filter and sort information by any criteria
![Page 26: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/26.jpg)
26
Monitoring and Diagnosing Tool (3)
Automatic Alerts (Denodo 6.0 Updates)
Server down
Data Source or Cache Down
% CPU Usage
Connection Pool full
…
Alerts (Visual / E-Mail):
![Page 27: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/27.jpg)
27
Monitoring and Diagnostic Tool (and 4)Pre-defined Reports (Denodo 7)
Pre-defined graphical usage reports)
• Workload breakdown by application
• Most used views
• Requests per Data Source
• …
![Page 28: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/28.jpg)
28
28
Denodo Solution ManagerMake it easier to manage large Deployments (Denodo 7)
Catalog of all elements of a Denodo deployment
Manage licenses configuration, logs and extensions
Automate migrations
Integrated governance workflows
![Page 29: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/29.jpg)
29
Automate Migration Between Environments
Overview of the Migration Process in Denodo 7 (Simplified)
S11
denodo-prd-1
S21
denodo-prd-2
S12
S22
S13
S23SolutionManager
Properties DB
Developers Migration Admins
Development
Production
1. SelectElementsto Migrate
2. ValidateRevision
VCS
4. Deploy Revision
5. Save full VQL after Revision
Load Balancer
3. RegisterRevision
![Page 30: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/30.jpg)
Main Areas
Dynamic Query Optimizer for Big Data (Denodo 6)
Incremental queries (Denodo 6 Updates)
Embedded in-memory fabric (Denodo 7)
New Information Self-Service Tool (Denodo 6)
Information Self-service: Glossary and Collaboration Features (Denodo 7)
▪ Tighter integration with Data Governance and
Data Modeling Tools (Denodo 7)
Workload Management: Denodo Resource Manager (Denodo 6)
Monitoring and Diagnosing Tool (Denodo 6 Updates)
Solution Manager (Denodo 7)
New VDP Admin Tool (Denodo 6)
GIT Support (Denodo 6)
▪ Support for new data sources and publishing formats (continuous work)
▪ New Data Types (Denodo 7)
Performance in BigData Scenarios
Security, Governance and Self-service
Enterprise Wide Deployments
Connectivity and Data Transformation
![Page 31: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/31.jpg)
Multiple Tabs
MultipleDatabases
New VDP Admin Tool (1)
31
![Page 32: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/32.jpg)
New VDP Admin Tool (and 2)
Collapsable Work Areas
32
![Page 33: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/33.jpg)
33
New adapters for Spark, Redshift and Snowflake (already available), Presto DB (Q1 2017), Neo4j (Denodo 7)
New adapters for Denodo in IBM Cognos and Looker (already available), Tableau (Q4 2016)
Extended set of geospatial functions and GeoJSON support (Denodo7)
Continuous work on transformation functions
Connectivity:
Other Enhancements
Transformation / Integration:
![Page 34: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/34.jpg)
Q&A
![Page 35: Denodo DataFest 2016: What’s New in Denodo Platform – Demo and Roadmap](https://reader031.vdocuments.net/reader031/viewer/2022030221/5884afc91a28ab76798b6785/html5/thumbnails/35.jpg)
Thank you!
© Copyright Denodo Technologies. All rights reservedUnless otherwise specified, no part of this PDF file may be reproduced or utilized in any for or by any means, electronic or mechanical, including photocopying and microfilm, without prior the written authorization from Denodo Technologies.
O C T O B E R 1 8 , 2 0 1 6 S A N F R A N C I S C O B A Y A R E A , C A
#DenodoDataFest