oracle nosql database -- big data bellevue meetup - 02-18-15
TRANSCRIPT
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
NoSQL in the Real World
Dave Segleau Dir. Product Management Feb. 18, 2015
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
2
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Agenda
Why NoSQL?
Oracle NoSQL Database
Use cases
What are the NoSQL Challenges?
3
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Why NoSQL?
1. The V’s of Big Data
– More Data, More Queries, Low Latency Operations (SLAs) Driving Business Innovation
2. Cost of Operations
– Cost effective horizontally scalable simple operations on simple data, Requires less HW, Lower license costs than traditional RDBMS databases
3. Time to Market
– Developer-centric Data Management, Data Variety, Rapid Evolution Shorter Time to Market
4
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Where is NoSQL used?
5
Simple Data Management
Globally Distributed,
“Always On” data
Competitive Advantages of
“Fast Data”
Lower TCO, commodity
HW scale-out
ERP
EAM
Inventory Control
Accting & Payroll
Process Mgmt
Business Analytics
CRM
…
Driver
Application Real Time Event
Processing
Web-Scale Personalization &
Xaction Processing
Aggregation & Consolidation
…
Mobile Data Management
Time Series & Sensor Data Mgmt
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
The NoSQL Landscape
NoSQL
Columnar & Key/Value
• Keyspaces, Tables & Records
• Key-based access
• Limited Transactions
• Broad set of use cases
Document
• Collections
• Document-based access
• JSON & XML
• “Objects as documents” use cases
Graph
• Interconnected graphs
• Relatedness-based access
• Properties and Graphs, RDF
• Specific use cases
• Developer- centric APIs • Flexible schemas • Partitioned/sharded data
• Horizontally scalable • High Availability via Replication • Integrated with Hadoop
6
What we hear from our customers: • There is no “one size fits all NoSQL solution” • Customers use multiple NoSQL products
• NoSQL and SQL (RDBMS) must work together • Application version 2.0 is the one that really works
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Agenda
Why NoSQL?
Oracle NoSQL Database
Use cases
What are the NoSQL Challenges?
7
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
What is Oracle NoSQL Database
8
101100101001001001101010101011100101010100100101
Reliable Flexible Fast Simple
advanced Key-Value database designed as cost effective, high performance solution for simple operations on collections of data with built in high availability and elastic
scale-out.
less is more
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL Database NoSQL for Developers and IT
9
Setup, Admin, API & Integration
Built-in HA, Predictable Performance
Parallel Access & Scale-out
Simple:
Fast:
Flexible:
Reliable:
Flexible schema & Agile development
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
• Features
– Elastic
– BASE Operations
– Tables / JSON / Binary
– Online management
• Differentiators
– ACID transactions
– Online rolling upgrades
– Streaming large object support
– Oracle technology integrated
– Engineered Systems and Commodity HW
Enterprise Ready
10
Application
Storage Nodes Datacenter B
Storage Nodes Datacenter A
Application
NoSQL DB Driver
Application
NoSQL DB Driver
Application – Data Center Support
– Secondary Indexes
– Secure Access
– Flexible schema
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Architecture (logical) Linear scaling and replication
11
• Elastic Auto Sharding
(split, add, contract)
Store
Shard
M
Shard
R
R
R R
Application
NoSQL Driver
M
• Writes to elected
node with flexible
durability
• Reads from any
node in system
Shard
R
R
M
Expand and Rebalance
Shard
M
R
R • Auto re-balance of
data on expansion
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 12
Oracle Big Data Management System
Data Warehouse Data Reservoir +
Oracle Big Data Connectors
Oracle Big Data SQL
Oracle Advanced Analytics
Oracle
Database
Oracle Spatial & Graph
Cloudera Hadoop
Oracle R Distribution
Oracle NoSQL Database
Oracle Industry Models
Oracle GoldenGate
Oracle Data Integrator
Oracle Event Processing
Oracle Event Processing
Apache Flume
Oracle GoldenGate
Oracle Advanced Analytics
Oracle Database
Oracle Spatial & Graph
Oracle Industry Models
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
• Query NoSQL data from Oracle Database and Hive
• Access NoSQL data from Hadoop and Spark for DW and analytics
• Share data with Oracle Coherence for extensible in-memory cache grid
• Persist history & event streams for processing with Oracle Event Processing
• Store & query RDF data using Oracle RDF for NoSQL
• Replicate changes in Oracle Database to NoSQL DB using Oracle Golden Gate
• Monitor your NoSQL cluster using Oracle Enterprise Manager
13
Enterprise ready -- Integrated out of the box
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Agenda
Why NoSQL?
Oracle NoSQL Database
Use cases
What are the NoSQL Challenges?
14
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Web-Scale Personalization & Xaction Processing • Direct-to-Consumer web services
• Real-time transaction capture
• High volume, Low latency
• Retail, Financial Svcs, Insurance
Time Series & Sensor Data Mgmt • Efficient, scalable data persistence
• Requires analytics and visualization
• Scalability, Cost management
• Utilities, Oil & Gas, Manufacturing, Financial Svcs
15
Real Time Event Processing • Scalable container for rules and history
• Provides context to interpret events
• Low latency, Cost trade-offs
• Credit Card Svcs, Content Recommend Engines
Aggregation & Consolidation • Disparate data sources, integrated data
• Customer Service, 360 View
• Business Innovation from legacy systems
• Insurance, Banking, Healthcare, Entertainment
Common NoSQL Use Cases
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Finance
• Large credit card network provider
• Fraud rule and history management
• 6,500 ops/sec, 85% update on read
Manufacturing
• Large airline manufacturer
• Developing next generation planes
• 750K key inserts/sec, 6TB/hour
16
Telecoms
• Large mobile telecom operator
• Mobile personalized advertizing
• 24,000 ops/sec, 100% update on read
Customer Loyalty
• Operationalize analytics to NoSQL Customer Portal
• Addressed processing conflicts
• 20M+ customers in multiple countries
Oracle NoSQL Database customer success…
……many more
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Agenda
Why NoSQL?
Oracle NoSQL Database
Use cases
What are the NoSQL Challenges?
17
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Getting Started -- What’s Easy?
• Initial Installation and Deployment Simple out of the box experience
• Developer-centric APIs Flexible documents and key/value pairs
• Data Ingestion and Query Fast and simple
18
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Getting Real – What are the First Challenges?
• Too Many Choices Dozens of NoSQL products
• Data Modeling Model affects behavior, specially performance
• Functional Integration How do I do X? Analytics is a hot topic
19
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Deploying to Production -- What’s Hard?
• Scalability, Reliability & Support Buyer beware
• Manageability at Scale Common “hidden” cost
• Continuous Integration Integration with RDBMS and other systems
20
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Enterprise NoSQL
Text
Core Database
Functionality
Application Developer
Friendly
Enterprise-grade
Software
• Predictable, low latency
• Highly available
• Highly scalable
• Administrator friendly (IT)
• Strong Integration
• Enterprise-grade support
• Multiple APIs
• Simple, flexible schemas
• Flexible data model
2/18/2015 21
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Benefits
Scalable multi-channel customer portal
Predictable performance for all operations
Reduced time to market
Easy application evolution
Coupon redemption, Vendor recommendation
NoSQL for Customer Loyalty
Objectives
Scalable customer loyalty portal
New multi-channel consumer model
Improve operational efficiency
Solution
Personalized multi-channel coupon generation and redemption
Cross-promote affiliated vendors
Scale system with customers and participating retailers
NoSQL DB Driver Application
Retail Partners
Customer Profiles
End Customers
Available Coupons Market Segmentation
27
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Financial Services coordinated theft prevention
NoSQL for Fraud Scoring
Objectives
Solution
Benefits
Application Data Ingestion
Tran
sact
ion
Au
tho
riza
tio
n
Pro
cess
or
Combine data sources for complex scoring
Detect, alert analyst with low latency
Handle burst seasonal transaction volumes
Oracle Coherence cluster for real time transaction object management
Oracle NoSQL Database for fraud model and customer profile management
Oracle Database for statistics and fraud modeling-related data
Simple data model, ACID transactions
Scalability, Reliability, Low Latency
Elasticity of sharded data repositories
Easy configuration and administration
NoSQL DB Driver
28
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Objectives
Solution
Benefits
• Increase scalability of data storage •Deliver higher concurrency analytic data access •Scale data loading independently from analysis •Commercial support for mission critical system
•Oracle NoSQL database for high speed storage and range based extraction of time series data.
•Oracle NoSQL database for agile schema, replaced HDF5 storage format, kept analysis client program
•Oracle Big Data Appliance for efficient manageability and lowest TCO
•Hadoop post processing and RDBMS connectivity to Enterprise systems
• Improve scale of storage for flight test sensor data • Increase concurrency of access to data for analysis • Improve system availability for analysts by allowing
simultaneous data ingestion and analysis
Big Data Appliance
NoSQL DB Driver
Event Ingestion and Extraction
NoSQLDB/ Oracle RDBMS
Hadoop/ Oracle RDBMS
Oracle or Any third parties
SQL/Data Analytics Tools
NoSQL for Sensor Event Storage & Processing Large scale sensor data capture and analysis
29
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Product Recommendation Platform
30
Elastic Data Store
Recommendation Rule Engine
Source Systems
Admin
Customers
Corporate Web Page Recommendation Platform
Rules
Customer Info
Recommend
User Info, Billing Info, etc
Request Respond Login
User ID
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL Database Technical Features
31
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Smart Topology Management
• Automated Resource Planning
• Storage nodes have indication of “capacity”
• Allocates replicas per storage node
• Intelligent Master/Replica load balance
• Ensures distribution of replicas
• Efficient use of system resources
• Reduces operator-caused configuration errors
32
Application
Smart Topology Driver
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Data Center Support
• Availability Zones
• Flexible configuration
• Primary Zones
– Durability guarantees
– Low latency writes, HA
• 2nd’ary Read-Only Zones
– Asynchronous replication
– Analytic workloads
– Report generation
• Topology Aware Client Driver
• Provides business continuity and distributed workload management
33
DC1 DC2 DC3
PrimaryZones
Reports
Batch Analytics
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Configurable CAP
• Greater Flexibility
– Configurable • Durability per operation
• Consistency per operation
– ACID by default
– Transaction scope is single API call
– Records share same shard key
– Multiple operations supported
34
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Large Object Support
• Parallel Streaming Interfaces
– Efficient storage and retrieval of large objects
– Client side streaming interface for low memory consumption
– Server side splitting and distribution of object chunks across nodes for better read/write latency
– Automatic partial LOB detection
35
Large
Object
No
SQL
DB
Dri
ver
Ap
plic
atio
n
Shard 2
Shard N
Shard 1
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 36
Flexible Data Model
Oracle NoSQL Database
advanced key-value database
1. KV API, Application specific opaque values
Primary key & Shard key Automatic Data sharding & local indexing BASE & ACID Transactions
1. KV API, Application specific opaque values 2. JSON API, JSON Structures 1. Application specific opaque values 2. JSON API, JSON Structures 3. Table API, Tables
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Secondary Indexes
37
Pkey A Value A
F1 F2 F3 F4 F5 F6
Pkey B Value B
F1 F2 F3 F4 F5 F6 Skey A Pkey A
F5 F6 F1 F2 F3
Skey B Pkey B
F5 F6 F1 F2 F3
Primary Records
Secondary Indexes
Shard 1
Writes
Reads
Shard 2
Shard N
Index on (F4)
Skey A Pkey A
F4 F1 F2 F3
Skey B Pkey B
F4 F1 F2 F3
Index on (F5, F6)
• Data local b-tree
• Broader low latency use cases
• Low cardinality indexing
• Consistent indexes
• Simple & Compound indexes
• Simple & Complex data types
• Range and equality searches
• Store wide sorted —Parallel scans
—Ordered results
—Array indexing
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Security
• Data Access Protection
– Configurable enforcement
– Authentication • User/Password
• Configurable client time-outs
• Oracle Wallet integration
• Internal components self-authenticate
– Encryption over the wire • All channels SSL encrypted
– Authorization • Read/Write Access
• Administration
– Future integration with Oracle Products
38
Store
Shard
M
Shard
R
R
Shard
R
R R
NoSQL Driver
M R
M
Username Password
SSL
SSL SSL
Application
Read/Write or Admin
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Online Rolling Upgrades
• Linear time impact for upgrade – Admin commands available to describe
safe upgrade order – Scripted available hands-free upgrade
experience – Read/Write availability throughout the
upgrade process
200 Node version upgrade in 15 minutes
39
0
2.5
5
7.5
10
12.5
15
17.5
72 (24x3) 144 (48x3) 216 (72x3)
Tim
e t
o U
pgr
ade
(m
in)
Total Nodes (Shards x Rep. Factor)
Online Rolling Upgrade
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Developer and Admin Tools
• Standards based tooling
– SNMP / JMX metrics
– Oracle Enterprise Manager
– Cloud ready HTML5 browser admin
– Command line interface • Scripting
• Query prototyping
• Data load
– Easy to use developer API • Java, C, REST
• R, JRuby, Jython community drivers
• Coming: Javascript, Python
40
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Developer Centric APIs • Java API for Key-Value, JSON and Table data models
• C API for Key-Value & JSON
• C API for Table data model
– Implementing using a new Thrift, JNI-less protocol
– Additional Thrift-based APIs to follow
• REST API available through Oracle Rest Data Services
– Deployments using Oracle Web Logic Server, Glassfish, Tomcat
– Directs Web Service calls to NoSQL DB
– Marshals data returned into JSON format
– Full CRUD operations across NoSQL Cluster
• Client-side DDL API
– Create and Alter tables and schemas inside an application
41