www.synerzip.com data virtualization & information as a service (iaas) by anil allewar senior...

www.synerzip.comwww.synerzip.com

Data Virtualization&

Information As A Service (IaaS)

By Anil AllewarSenior Solutions Architect - Synerzip

About Me!!

Confidential 2

Anil Allewar

Senior Solutions Architect @ Synerzip

Technology Evangelist & speaker

Core interests: JEE, EAI, EII

• Use cases

Agenda

• What does it mean?

• Implementation Frameworks

• Demo

• Questions?

• Architecture explained

Why it makes sense?

Confidential 4

Use Cases

Data Warehouse

Financial Data

OLTP Data

3rd Party Data

Data Mart

Web Service 1

Web Service 2

Legacy Data

Custom Program

Excel files

Confidential 5

Traditional Data Integration

Enterprise Information System

Source System

Confidential 6

Business Applications

Problems with ETL

Confidential 7

More than 1 copy of data for staging

Intermediate data => Errors

Lead time to add new source

Domain knowledge for mapping

Batch Process => No real time data

Problems with DBMS consolidation

Confidential 8

Alternate approach => Single EIS (say RDBMS)

Extensive changes to existing apps

Might not satisfy everyone’s requiremets

• Use cases

Agenda

• Demo

• Questions?

Data Virtualization & Federation

Confidential 10

Single API to access data

Only metadata stored at virtualization layer

Real time access without

copying/moving dataFederate data across hetero/homogenous

sources

Data Virtualization

Confidential 11

• Use cases

Agenda

• Demo

• Questions?

Architecture

Confidential 13

UserApplication

Connector 1

Connector 2

RUNTIME & QUERY ENGINE

VirtualDatabase

Translator 1

Translator 2

• Use cases

Agenda

• Demo

• Questions?

Vendors

• Commercial Products– Composite Software

• http://www.compositesw.com/data-virtualization/

– Denodo• http://www.denodo.com/en/product/overview.php?n=h

– IBM• http://www-03.ibm.com/software/products/en/ibminfofedeserv

– Informatica• http://www.informatica.com/us/data-virtualization/

– Red Hat• http://www.redhat.com/products/jbossenterprisemiddleware/data-virtualizati

• Open Source– Jboss Teiid

• http://teiid.jboss.org/

Confidential 15

Selected Platform – JBoss Teiid

Confidential 16

Open Source

Number of relational/NoSQL/ERP/CRM data stores

JEE standards

Add custom EIS support using JEE

components

Active & responsive community Synerzip contribution: Defect

discovery, root cause analysis, feature verification

Teiid Components

• Virtual Database– container for components used to integrate

data from multiple data sources• Source Models

– structure and characteristics of physical data sources

• View Models– structure and characteristics of abstract structures you want to expose

to your applications

• Teiid Designer– Eclipse based UI to dynamically discover data

source objects and apply data federation– Generate virtual database from 1 or more

sources

Confidential 17

Teiid Components

• Translator– Provides abstraction later between Teiid Query

Engine and source system– Convert Teiid SQL commands to source specific

execution commands– Convert result data from source system to Teiid

specific format• Resource Adapter

– Provides connectivity to the physical data source

– Integration provided through Java Connector Architecture (JCA) API

Confidential 18

Teiid – Supported EIS

• Amazon SimpleDB• Apache Accumulo• Apache SOLR• Cassandra• File• Google Spreadsheet• JPA• LDAP• Excel – as file• SalesForce

• JDBC– MS access, DB2, derby, excel-

odbc, greenplum, h2 , hive(for accessing Hadoop), oracle, teradata and most RDBMS

• MongoDB• Object• OData• OLAP• Web Services• SAP Netweaver Gateway

Confidential 19

Performance Characteristics

• Access same data using Oracle and Teiid drivers

– Retrieval times comparable when accessing tables having no Blobs

Confidential 20

0 0 0 1 13693

505,005

505,007

720,850

859,457

1,003,798

1,103,9180

10,000

15,000

20,000

25,000No. of rows Vs Time: No Blobs

Oracle-JDBCTeiid-JDBC

No. of rows

Performance Characteristics

– Teiid slower when accessing Blob data• Can be tuned

Confidential 21

0 0 2 42 21,804 32,531 185,4540

10,000

15,000

20,000

25,000

30,000

No. of rows Vs Time: Blobs

Oracle-JDBCTeiid-JDBCm

No. of rows

• Use cases

Agenda

• Demo

• Questions?

Confidential 23

JDBC Client JD

BC API

RDBMS Resource Adapter

MongoDB Resource Adapter

TEIID RUNTIME & QUERY ENGINE

Federated VDB

mySQL Translator

MongoDB Translator mySQL

Demo-Steps• Pre-requisites

– mySQL server 5.5+ installed– MongoDB 2.4.x+ installed

• Steps– Load the mySql and MongoDB database with sample

data– Setup environment – JBoss, Eclipse– Create Teiid project in Eclipse using Teiid designer

• Import source model using JDBC• Create the virtual model and federate data from the

source model• Create a virtual database (VDB) and deploy to JBoss

– Access data using JDBC client or through browser using OData

Confidential 24

Demo – Scenario

Confidential 25

FederatedData

Demo – Connection Profile

Confidential 26

Demo – Source Model

Confidential 27

Demo - Source Model Generation

Confidential 28

Demo – Map Source To View

Confidential 29

Demo - Association

Confidential 30

Demo – Data Federation

Confidential 31

Demo – Source Code

• Source code–https://github.com/Synerzip/JBoss-Teiid

–Contains• Configuration files• Instructions• “How-to” videos• VDBs, source models and view models

Confidential 32

Conclusion

• Data Virtualization and Federation is a rapidly emerging technology that solves traditional BI/ETL problems.

• It provides lower time to market, distributes data across the enterprise as a service and provides real time access to enterprise data.

Confidential 33

• Use cases

Agenda

• Demo

• Questions?

Contact Me

• anil.allewar@synerzip.com

Confidential 35

www.synerzip.comwww.synerzip.com• 3

Questions?

www.synerzip.comHemant Elhence

hemant@synerzip.com469.322.0349

Synerzip in a Nutshell1. Software product development partner for

small/mid-sized technology companies• Exclusive focus on small/mid-sized technology companies,

typically venture-backed companies in growth phase• By definition, all Synerzip work is the IP of its respective

clients• Deep experience in full SDLC – design, dev, QA/testing,

deployment2. Dedicated team of high caliber software

professionals for each client• Seamlessly extends client’s local team, offering full

transparency• Stable teams with very low turn-over• NOT just “staff augmentation”, but provide full mgmt

support3. Actually reduces risk of development/delivery

• Experienced team - uses appropriate level of engineering discipline

• Practices Agile development – responsive, yet disciplined4. Reduces cost – dual-shore team, 50% cost

advantage5. Offers long term flexibility – allows (facilitates)

taking offshore team captive – aka “BOT” option

Our Clients

Call Us for a Free Consultation!

Hemant Elhence hemant@synerzip.com

469.322.0349

Thanks!

www.synerzip.com data virtualization & information as a service (iaas) by anil allewar senior...

copyingmoving data

intermediate data

copy of data

real time data slide

eii slide

confidential15 slide

confidential4 slide

feature verification

Documents

the total economic impact™ of microsoft azure iaas iaas...

agile2016 conference top 10 presented by synerzip

iaas en gwab

www.synerzip.com serving software development/qa needs of...

iaas + csb

iaas ordering guide

www.synerzip.com cross platform mobile applications by rohit...

agile testing concepts and techniques - synerzip

azure iaas

managing iaas resources

iaas pricing models

slicing pie eureka!. win a signed copy:...

buy, mphasis, synerzip, artisan | competitive intelligence...

www.synerzip.com responsive web design nikhil j deshpande...

www.synerzip.com agile approach @synerzip agile/iterative...

www.synerzip.com qa/testing @synerzip october 2008

failsafe iaas

qa/testing @synerzip

iaas azure_vs_amazon

cloud computing iaas