Integrating a Heterogeneous Environment using XML

Sandeepan BanerjeeDirector, Oracle Server Technologies

The Problem

Multiple domain-specific applications

– Manufacturing, Inventory, Supply Chain, Financial, …

Information is trapped within these applications

How does an organization get a consolidated view of its information – in real time ?

CRM ApplicationFinancial

Application

Contract Management

Application

SynthesizedInformation

Excel Files on Disk

E-Mail and DocumentRepositories

Technical Challenges Domain-specific information

– Replication does not make sense

Independent operation of applications – Access to information has to be in real-time

Different access method for each application– Each application has its own protocol and access method

What architecture can best accommodate my present and future needs ?

– Complexity: Avoid the n by m matrix– Flexibility: Add new sources easily – Time to market: Within days, not months

What is XML-based Data Integration?

XML-based Data Integration or Enterprise Information Integration (EII):

Create aggregated views using XQuery Get information from diverse sources in XML Consume synthesized information

Database Web Service

J2EETM CA

JDBC HTTP

XQuery Engine

Order Tracking

Shipment Tracking

XML Data Integration Example

Synthesized Information

EIS

Parts Inventory

XML-based Data Integration

J2EETM Platform

Technologies Involved

Why XML?– Different data formats

Why XQuery ?– Declarative way to query

XML documents

Why J2EETM ? – Standards-based

infrastructure platform

Why XML Database ?– Native XML storage– XML data management – Performance optimizations

XML Database

XQuery Engine

Comparison with Existing Technologies

Difference Similarity

Application Integration

More about data pumping and synchronization among systems

Involves data adapters, data translation and transformation

Data Warehousing

Explicit ETL steps required; large data volume; batch loading, not real-time

Optional cache pre-population step similar to the “loading” step

Traditional Report Generation

SQL & relation-based vs. XQuery & XML-based

Cannot handle non-relational sources

Query-based

XQuery Status

• XQuery is emerging as the consensus ‘native’ query language for XML

• Expected W3C Recommendation in late 2005• Oracle 10gR2 is the first mainstream commercial database release to support XQuery

• Plan to release under an event in 10gR2

XQuery Example

Assume a document – emp.xml <empset> <emp empno=“21” ename=“SCOTT” salary=“120000”/>

<emp empno=“22” ename=“JONES” salary=“344000”/> </empset>

To get the names of employees with salary > 200000for $i in document(‘emp.xml’)/empsetlet $j = 200000where $i/@salary > $jreturn $i/@ename

Result (attribute node) JONES

Another ExampleCustomer/Address/ZipShipTo/Address/ZipSupplier/Location/AddressForTaxCalculation/ZipCustomer/Address/Work/City/ZipCustomer/Address/Home/City/ZipShipper/DropOffLocation/US/California/SFO/Zip+ any other zip nested to any unanticipated level

count ( for $i in doc("contacts.xml")/Contact where $i//zip eq 94065 return $i)

Example III: Auction/Bids Say we have 3 heterogeneous data sources that yield their

contents as XML– Item Description comes from a Filesystem– User Information comes from LDAP/DB– Bid Information comes from an App Server

-- For all bicycles, list the item number, description, highest bid (if any), ordered by item no.

for $i in doc("items.xml")//item_tuple

let $b := doc("bids.xml")//bid_tuple[itemno = $i/itemno]

where contains($i/description, "Bicycle") order by $i/itemno

return <item_tuple> { $i/itemno} { $i/description }

<high_bid>{ max($b/bid) }</high_bid> </item_tuple>

XQuery and SQL

Existing relational applications will continue to use SQL, and Oracle will remain the industry’s best implementation of SQL

New applications based on XML will use XQuery, and Oracle will be the industry’s best implementation of XQuery

Oracle will support XQuery both in the database and in the mid-tier

– Use the mid-tier engine when you want to query non-database sources

Intelligent ‘query pushdown’ from mid-tier to db when possible

Can mix-and-match XQuery and SQL in same query

XQuery is different from SQL

Navigation-oriented (using Xpath expressions) Different type system (XMLSchema based simple types) Identity-based (XML Node identities and document

order) Namespace aware name-resolution (functions, variables,

element creation) XML-Item based vs Row-based Results are heterogeneous sequences Does not have all SQL extensions (e.g. data

warehousing etc..)

XQuery Mid-tier Architecture

XQuery Java

Engine

XQJ

API

DriverDB

Drivers

JDBC Driver

Java XMLType

XQueryXQueryXXPath

SQL + XQuery orXQueryX

Other Data sources

XQuery DB Architecture

Mid-Tier XQuery Engine

User

XQuery Aware SQL Engine

SQ

L C

om

piler

Optimizer, Execution Engine

Normalization

XQuery Type check

Compiler

XQuery ParserXMLQuery, XMLTable

SQL

XQJ API

XQuery

SQLX/XPath

D E M O N S T R A T I O N

XML Query

Oracle XML Database (XML DB)

Native XML storage – Available since Oracle Database Release 9.2

Inherits RDBMS features: Security, Transaction,… XML-specific features

– XML indexing, XPath & XSLT support, XML schema validation, XML partial update

Supports SQL/XML– Allows blending relational and XML data operations

Leveraging Oracle XML DB

XML DB can be an XQuery source – Can define XML views of relational data– XQuery engine can rewrite query into SQL/XML

XML DB could also be used for caching– Efficient storage & indexing for large data sets

Can leverage security framework of XML DB– For both source and cache

Datasources

Databases – Relational+ XML Views, Object-Relational, CLOB, Compact

XML (future) Mid-tier sources

– files, cache, JCA datasources Bind (an existing DOM) xmldatasrc – Oracle language addition Datasource API

– initialize– describe– execute – Fetch

iAS XDS Architecture

XDS Client API’s

EJB JSP Tags Web Service

XML Data Synthesis

Cached XML Data Source

Cached XML Data Source

Applications using XDS

e.g. Portals, Reports etc

Query

Builder

Tool

Meta-data

Repository

Ora

cle

En

terprise

Man

ager

XML DB

JCache

Filesystem

J2EE Security Framewor

kXDS Cache

Security

XDS

Caching

Service

XML Data source adaptors CCI-XML

Web

Services

J2CA EAI

SAP

Oracle

AppsJMS RDBMS

Files HTTP

Web

Cache

Java

Functions

XQuery Subsystem

XQ4J/JXQI

XQuery Engine

XMLDataSource modules

XQuery Result

In-Memory

Stored Query

Example – XDS usage

User registers webservice as datasource XDS creates an XQuery module automatically User Query for querying webservice

import module namespace wss=“datasrc/stockws”; for $i in wss:getcompanies()return wss:get_stock_price($i/name)

wss – namespace prefix for the loaded moduleJNDI lookup to get datasource implementationXDS adapters implement datasource

All Your Searches!

Search with any language – SQL/XQuery/XSL Search anywhere - mid tier or backend Search anything -any XML/relational content Search everything

– XML visualization of all data (backend)– XML based adapters provide XML content for all data in mid-

tier (XDS) Search any form – text based/structured Search any size - Scalable solution Search any time - Unbreakable solution

More Information

XML in general– http://www.oracle.com/technology/tech/xml/index.html

XML Query– http://www.oracle.com/technology/tech/xml/xquery/index.html

XML DB– http://www.oracle.com/technology/tech/xml/xmldb/index.html

AQ&Q U E S T I O N SQ U E S T I O N S

A N S W E R SA N S W E R S