oracle berkeley db in life sciences: embedded databases for better

<Insert Picture Here>

Oracle Berkeley DB in Life Sciences:Embedded Databases for better, faster results

© 2007 Oracle Corporation – Proprietary and Confidential 2

Gregory BurdProduct Manager

Berkeley DB, JE, and XML


Agenda

• Berkeley DB Overview• Uses of Berkeley DB• Programming Berkeley DB• Q&A


What is Berkeley DB?

• Reliable, scalable, flexible, fast and transactional data management• Library that links directly into your application• Enterprise database functionality with:

• A fraction of the footprint• Hands-off administration• Programmatic configuration and management• Ability to select which features you use

• Highly portable, even to esoteric platforms• Data management hidden within an application

An engine for your specialized data management needs.


Storage Engine, not a full RDBMSRDBMS Berkeley DB

Your Application Code

ODBC, JDBC, OCI, etc.

TCP/IP

TCP/IP

SQL Parser

Your Application CodeQuery Optimizer

Berkeley DBStorage Manager

• Runs in-process – no IPC• Direct, indexed access – simple function-call API• Application-native storage – no SQL


Relational Databases

sql> select * from Employees


Embeddable Databases

Application


Fundamental Concepts of DB

• Fast indexed or sequential lookup• API is get/put, or through cursors

• Concurrent access• Threads, processes, both

• Transactions• ACID, recoverable, nested, etc.

key data

key data

key data

key data

Conceptual View of Data

Data organized in specialized indexed structures on disk, cached into memory


Berkeley DB, a Family of Products

Three different products.

Berkeley DB

Berkeley DB XML

Berkeley DBJava Edition

a layer on top of DB

functionally equivalent to DB


Agenda



Commercial Applications

Financial Services

Service Providers

Content Mgmt

Storage & Sys Mgmt

Enterprise Infra/EAISecurityNetworking

Telecom Applicationsinfrastructure

Devices/ Appliances

Berkeley DB

http://www.emc.com/


Open Source Applications

Berkeley DB

All versionsof Linux

All versionsof BSD UNIX

Apache Web Server, Directory, et al

LDAP Directory

Productivity suite

Chandler email/PIM

Website traffic analysis

Kerberos Network Authentication

DatabaseSubversion version

control system

Red Hat Package Manager

Internet search

Text editorMail server (MTA)

Mail server (MTA)

SquidGuardspam blocker

Spam blocker

Perl

Python

GNU C library

PHP

EmailApplicationsInfrastructure Programming Languages

ToolsOperatingSystems


• Queries are known in advance• Performance matters• Queries are (relatively) simple• Footprint matters• You are resource constrained• Data has natural structure you want to preserve• The system must run without administrator support• Build once, deploy with confidence 1000s of times

Berkeley DB

The right answer when:


Agenda



Terminology

• Environment• Key Data Pair• Database• Access method• Cursors

• A Database (in relational)• Specified as a directory• Referenced by a DB_ENV handle

• Opaque byte strings• Encapsulates pointer and length• Referenced by DBT handle

• A position in a database• Used for sequential access• Referenced by a DBC

• A collection of key/data pairs• Share index(es) and sort order• No fixed schema• Referenced by DB handle

• An indexing structure• One of: hash, btree, recno

or queue


Application Architecture

• Create/Open/Recover environment• Open database handles• Spawn off utility threads• Spawn off worker threads

• Begin transaction• Operations on databases• Commit/abort transaction

• Close database handles• Close environment


Sample APIs

• Create/Open/Recover an environment• Create/Open a database• Transactional read-modify-write• Sequential database traversal


Open Environment

if ((ret = db_env_create(&dbenv, 0)) != 0)

goto err;

if ((ret = dbenv->set_cachesize(

dbenv, gigabytes, bytes, ncaches)) != 0)

goto err;

/* Other configuration here. */

flags = DB_CREATE |DB_INIT_TXN | DB_INIT_LOCK | DB_RECOVER;

if ((ret = dbenv->open(dbenv, HOME, flags, 0)) != 0)

goto err;


Create/Open Database

if ((ret = db_create(&dbp, dbenv, 0)) != 0)

goto err;

if ((ret = dbp->set_pagesize(dbp, 2048)) != 0)

goto err;

/* More database configuration here. */

if ((ret = dbp->open(dbp,

“file.db”, “db1”, DB_BTREE, DB_CREATE, 0)) != 0)

goto err;


Transactional RMW

if ((ret = dbenv->txn_begin(dbenv, NULL, &txn, 0)) != 0)

goto err;

if ((dbp->get(dbp, txn, keydbt, datadbt, DB_RMW)) != 0)

goto abort_txn;

/* Modify data */

if ((dbp->put(dbp, txn, keydbt, datadbt, 0)) != 0)

goto abort_txn;

if ((ret = txn->commit(txn, 0)) != 0)

goto err;


Sequential Traversal

if ((ret = dbp->cursor(dbp, txn, &dbc, 0)) != 0)

goto abort_txn;

while ((ret = dbc->c_get(

dbc, keydbt, datadbt, DB_NEXT)) == 0)

/* Process record. */

if (ret != 0 && ret != DB_NOTFOUND)

(void)txn->abort(txn, 0);

if ((ret = txn->commit(txn, 0)) != 0)

goto err;


Outline

• Berkeley DB Overview• Functionality• Terminology• Programming with Berkeley DB• Configuring and administering Berkeley DB


Flexibility

• Disk-based or in-memory execution• Four degrees of isolation• Commit to disk, file system, or memory• Configure for small footprint: 350 KB• Synchronous vs asynchronous replication• Designated master or elected master


Administration

• Command line or application-embedded• All the utilities you expect:

• Dump, load, verify, salvage• Recover• Checkpoint and log file archival and removal• Hot backup• Deadlock detection• Statistics


The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remain at the sole discretion of Oracle.


For More Information

http://search.oracle.com

Berkeley DB

orhttp://www.oracle.com/database/berkeley-db


Berkeley DB

Four different feature sets.

DS CDS TDS HA

Berkeley DB

Berkeley DB XML

CDS TDS


inherits features sets from DB

implements similar feature sets


Feature Sets

• Data Store• One writer or multiple readers• Blindingly fast storage and retrieval

• Concurrent Data Store• Single writer and multiple readers• Designed for read-mostly environments

• Transactional Data Store• High concurrency• Recovery from application, software, and system failures

• High Availability• Replication for failover, performance, and scalability• When downtime is unacceptable


Berkeley DB

Many different interfaces.

C++ API Java API

Java Collections API Direct Persistence(DPL)Berkeley DB XML

C++ API Java API Java APIC API

DS CDS TDS HA

Berkeley DB

CDS TDS



Berkeley DB Replication

• Failover• on a device, from card to card• within an application, from one machine to another

• Clusters• one system, the master, manages writes• the rest, the replicas, service reads• the master delivers changes to the replicas automatically• on failure, a new master is selected

• Scalability• when one machine can’t keep up• when load balancing is required

• Highly configurable• Not Oracle RAC/Grid


Use Berkeley DB when …

• You are developing an application.• You need data storage.• You are about to write your own: btree, hash table,

binary tree.• You find yourself calling: open, read/write, close, and

you aren’t reading a simple byte stream.• You need: high performance, high availability,

recovery, …


Agenda

• Berkeley DB Overview• Case Studies• Q&A


Berkeley DB for Life Sciences

• Electronic lab notebooks (ELN)• Chemical structure databases• Toxicology chemistry• Medicinal chemistry• Genomic data storage and mining• Combinatorial Chemistry• Study data management and mining• Clinical trial data storage and analysis


Why would it be better than relational?

• Complex data models, intrinsically non-relational• User defined encoding, schema optimizes representation• Tailored indexing with cross referencing for fast access• Encrypted data on disk for secure record management• Setup and forget, no constant administration required• XML and XQuery for ad-hoc access to semi-structured data• XML, meta-data, and key/value data managed in same location • Large database (256TB)• Large data values (4GB)• Transactional, protected, recoverable data• Distributed transactions (XA) for integration with other systems• Replication across cheap hardware for concurrent processing


Agenda

• Berkeley DB Overview• Case Studies• Q&A


What about SQL?

• “I thought one used an RDBMS and SQL to manage data.”

• You do, but• One size doesn’t fit all• No single solution is always the right answer.

• So, when is Berkeley DB right for you?


SQL and BDB—Similarities

RecoveryRecovery

IterationRange lookup

Secondary Index or Secondary Database

Secondary Index

Keyed lookupIndex lookup

Database KeyPrimary IndexConcurrencyConcurrency

ACID TransactionsACID TransactionsBerkeley DBSQL


SQL and BDB—Differences

Embedded in application address space.

Client/Server

Utility threads spawned directly from your application

Database administration

Berkeley DB APISQL Data Definition Language (DDL)

Hard-coded into applicationQuery processing and optimization

Berkeley DB APISQL Data Manipulation Language (DML)

Berkeley DBSQL


Berkeley DB

• Need to offload processing to the edge• Correctness, performance, ACID features important• In-house solution (if any) has reached its limits• Resource consolidation (servers or developers) within the

organization• Add new and enhanced capabilities to the edge servers/devices• Natural fit due to data model, simplicity, no admin, ...• Create a recurring revenue stream by adding new functionality

with each new release (upgrade the edge)

A Natural fit when:


https://www.google.com/accounts/


Authentication challenge for

login:password:


http://www.google.com/calendar


Google’s Requirements

• Build a business critical edge service• Simple problem, but of infinite value• Must be lights out 24/7/365, nine-nines• Any failure would be disastrous and public• Any performance issues would cost money• Must scale out world wide to billions of users• Billions of requests, no mistakes



Google’s Options

• Option 1: SQL•select password from Userswhere user_name = ‘greg’

•update Usersset password = ‘tiger’where user_name = ‘greg’

• Option 2: Build their own BTREE• Google frequently builds their own infrastructure

• Option 3: Berkeley DB•db->get(user, password);•db->set(user, password);


Google’s Second Decision

• Option 1: Build• Most critical infrastructure systems are built in-house• There was no apparent open source solution at the time• There are very smart database people on staff

• Option 2: Buy• There was no apparent solution on the commercial market• There was no apparent open source solution available

• They were building a solution in-house• a replicated, transactional, btree-based database

• Then Google discovered Berkeley DB/HA• they abandoned their internal project


Berkeley DB/HA in Production



“Berkeley DB was 20 times faster than other databases. It has the operational speed of a main memory database, the startup and shut down speed of a disk-resident database, and does not have the overhead of a client-server inter-process communication.”Ray van Tassle, Senior Staff Engineer, Motorola

What Motorola is Saying about DB


Motorola Wireless Network Gateway

• Requirement: Motorola’s Wireless Network Gateway (WNG) needed carrier-class performance, scalability and reliability. In order to meet 99.999% uptime requirements, the data manager needed to have theability to automatically restart after a failure with very fast startup and shutdown speed.

• Solution: Motorola selected Berkeley DB to store configuration data such as radio ID and user information for the WNG. Berkeley DB met Motorola’s requirements for high performance and ability to operate unattended and recover quickly from system crashes.


Motorola A760 A780 E680 Smart Phones

• Requirement: Motorola’s Linux-based smart phones needed a data management component to be fast, flexible, reliable and designed to operate within highly constrained environments.

• Solution: Motorola selected best-of-breed open source components including Oracle’s Berkeley DB, Montavista Linux and Trolltech Qt. Over 3 million of these devices have been shipped.

Motorola A760

Motorola A768

Motorola E680

Motorola A780


“When we switched to Sleepycat we never looked back. It gave us high performance, small footprint and a set of well-targeted features that let us take our solution to a new level.”Anton Okmianski, Senior Software Engineer, Cisco

What Cisco is Saying about DB


Cisco Systems Broadband Provisioning Registrar

• Requirement: Cisco’s Broadband Provisioning Registrar (BPR) needed to manage up to 5 million networked devices and 150 configuration change transactions/second. BPR needed an embedded data managerwhich was fast, scalable, reliable and cost-effective.

• Solution: Berkeley DB was used to replace an object-oriented database in BPR. The result was faster, more reliable and saved Cisco $50,000/CPU.


“Speed, reliability and ease-of-use are important considerations for telecommunication infrastructure equipment. We're pleased that Sleepycat continues to improve Berkeley DB for the telecommunications industry. Openwave relies on Berkeley DB as the core message store in Email Mx, a messaging platform that handles more than 1.5 billion messages a day.Rich Wong, SVP Products and Solutions Group, Openwave

What Openwave is Saying about DB


Openwave Mobile Messaging

• Requirement: Openwave customers demanded a high performance, low TCO solution

• Solution: Openwave uses Berkeley DB for the carrier-class message store and LDAP directory that are the basis for all of its messaging products, including Email Mx, MMSC and IP Voicemail. By eliminating the operator’s database licensing costs, extra hardware costs and DBA costs, the Berkeley DB based solution provided lower acquisition and operating costs for Openwave’s customers.


“When we did the analysis, Berkeley DB proved to be highly sophisticated while remaining cost effective.”Ralph Anzarouth, Director Systems and Network Management, Mitel

What Mitel is Saying about DB


Mitel Networks VOIP Platforms

• Requirement: Mitel’s flagship VOIP products, the 3100 and 3300 Integrated Communications Platforms, are all-in-one, IP PBX solutions for enterprises. Both used internally-developed data managers, which were unable to meet Mitel’s requirements for the next releases of those products.

• Solution: Berkeley DB replaced the homegrown data managers for configuration data, user profile management and tracking records. Berkeley DB enabled Mitel to slash 9 months off the production schedule and focus scarce engineering resources on core competencies.


Alcatel Policy Router

• Requirement: Alcatel’s Fort Knox Policy Router needed a fast, reliable and fault-tolerant data storage for the policy information used for making routing decisions.

• Solution: Berkeley DB is the storage engine that allows the box to make policy decisions at the speed demanded by high-performance, networked deployments.


LogicaCMG Mobile Messaging

• Projects that use Berkeley DB• MMSC (Multi-Media Messaging System)

• Highest ranked in Quality of Service • Highest ranked in Response times

• End-to-end message delivery• MMSC latency

• WSB (Wireless Service Broker)• WAP-based services• Video/Audio streaming

• UOne (Unified communications/messaging)• Single message store for

• Voice• Fax• Email• MMS• Video


IONA Technologies

• Relies on Berkeley DB to provide all high availability features• Berkeley DB stores all internal/service data• Locator, Naming, Trader services

Locator

CORBAServer

CORBAServer

CORBAServer

Locator

CORBAServer

CORBAServer

CORBAServer

Locator

CORBAServer

CORBAServer

CORBAServer

NamingServer

TraderServer

NamingServer

TraderServer


Berkeley DB Customer

Centera content-aware storage:• Conventional file access• Share duplicate blocks and files to reduce cost

Berkeley DB is used to store:• Device configuration and logs• File metadata• Hashes and reference counts• ACLs for administration

Value:• Scalability• Performance and reliability• Embeddability


“Berkeley DB Java Edition is the optimal choice for an internal database for Business Events, because it provides fast, transactional persistence in a pure Java package.”Matt Quinn, VP Product Strategy, TIBCO

What TIBCO is Saying about JE


TIBCO BusinessEvents UsesBerkeley DB Java Edition

• TIBCO BusinessEvents• Monitors disparate systems for interesting

activity• Correlates business and IT events based on

rules

• Berkeley DB Java Edition stores:• Rules identifying “interesting activity” and

“business events”• Event descriptions• Log for audit and reporting• System state saved every 20-30 seconds

• Why Berkeley DB Java Edition?• High throughput and reliability• Scalability• Integration with Java runtime

“Berkeley DB Java Edition is the optimal choice for an internal database for BusinessEvents, because it provides fast, transactional persistence in a pure Java package.”

Matt QuinnVP Product Strategy

TIBCO


General Dynamics

Command Post of the Future (CPOF)

• CoMotion Visualization Software –collaborative visualization software for decision-making

• Applications:• Battlefield decision support in Iraq• Supply-chain visibility• Patient records• Financial monitoring

• Berkeley DB Java Edition provides the repository for CoMotion

• Why Berkeley DB Java Edition?• High performance• Reliability• Lightweight and pure Java


“The Sleepycat database allows us to crawl and rank all those URLs efficiently. Because the crawler is written entirely in Java, we needed a highly scalable, very fast, 100% pure Java database engine.”Gordon Mohr, Chief Architect, Internet Archive

What Internet Archive is Saying about JE


Internet Archive –Heritrix and the WayBack Machine

• Internet Archive (www.archive.org)• Library that stores historical copies of the entire Web• Over 50 billion URLs and growing

• Berkeley DB Java Edition is the repository for Heritrix, the webcrawler for Internet Archive

• Very large crawls (tens of millions of URLs)bounded by RAM

• Berkeley DB Java Edition used to persistdata to disk with minimal performance penalty

• Why Berkeley DB Java Edition?• Very high scalability• Very high performance• Simple Java Collections API• Pure Java

http://www.archive.org/


FareLogix

FLX Platform:• Integrate travel products from suppliers• Web-based booking and confirmation

Berkeley DB XML is used to store :• XML objects aggregated from suppliers• Business rules for bookings• ACLs and identity

Value:• Scalability• Performance and reliability• Replication• XML support


Autodesk

Autodesk’s MapGuide product is a leading GIS application,

the data lives in Berkeley DB XML.


Juniper Networks

The Juniper NetScreen Security Managerhas to determine what is a threat, and what isn’t.

the resulting threat reports live in Berkeley DB XML.


Wall Street & Financial Markets

• Bloomberg• Database infrastructure sitting beneath suite of in-house

applications.• Uses Berkeley DB-HA to maintain perfectly consistent replicated

databases.• NASDAQ

• Stock trading application• 1500 inserts/second

• Toronto Stock Exchange• Stock matching

oracle berkeley db in life sciences: embedded databases for better

Documents