1 copyright © 2013, oracle and/or its affiliates. all...

41

Upload: others

Post on 05-Jun-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

1 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 2: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

The following is intended to outline our general product

direction. It is intended for information purposes only, and

may not be incorporated into any contract. It is not a

commitment to deliver any material, code, or functionality,

and should not be relied upon in making purchasing

decisions. The development, release, and timing of any

features or functionality described for Oracle's products

remains at the sole discretion of Oracle.

2 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 3: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Application Development

Big Data & Data Warehousing

Consolidation

Database as a Service

Data Optimization

High Availability

In-Memory

Performance & Scalability

Security & Compliance

Released on June 25th 2013

3 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 4: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Engineered for Clouds and Big Data

Page 5: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Customer Initiatives

5 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Big Data

Database as a Service

Cloud

5 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 6: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Database Consolidation on CloudsTraditional consolidation methods

Virtual Machines Clustered Databases

Co

nso

lida

tio

nD

en

sity

Schema Consolidation

Share Servers, OS & DatabaseShare Servers Share Servers & OS

6 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 7: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Co

nso

lida

tio

nD

en

sity

Oracle MultitenantSimplifies consolidation, enables Database as a Service

Pluggable DatabasesVirtual Machines Clustered Databases

Share Servers, OS & DatabaseShare Servers & OSShare Servers

7 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 8: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Oracle Database ArchitectureRequires memory, processes and database files

System Resources

8 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 9: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

New Multitenant ArchitectureMemory and processes required at container level only

System Resources

9 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 10: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Oracle Multitenant for ConsolidationMore efficient utilization of system resources

System Resources

10 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 11: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Oracle Multitenant for Test and DevelopmentFast, flexible copy and snapshot of pluggable databases

12 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 12: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Oracle Multitenant for ProvisioningFast Provisioning, Snapshot Clones

25

20

15

10

5

0

Time Taken to Provision New Database

Non CDB PDB Clone PDB using Copy-on-Write File

System

13 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 13: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

RAC, Data Guard

Oracle Multitenant for Database as a ServicePick from standard sizes and service levels

GOLD

Data Guard

SILVER

BRONZE Backups✔

14 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 14: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Demo

Self-Service

15 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

v

Provisioning

Page 15: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Oracle Multitenant for Database as a ServiceTrivially migrate tiers as databases become more mission critical

GOLD RAC, Data Guard

SILVER

BRONZE✔

✔Data Guard

16 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Backups

Page 16: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Delivering Database as a ServiceOracle Enterprise Manager, Oracle Multitenant and Oracle Exadata

17 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 17: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Engineered for Database as a Service

18 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 18: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

19 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 19: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

IDC Converged & Integrated Systems Survey Results

Rate how likely is it that your organization will utilize converged systems over the next three years?

2012 2013

0 = N o t a t a ll L ik e ly 1 . 0 %

1 1 . 0 %

2 2 .9 %

3 1 . 3 %

4 3 .9 %

5 4 .5 %

6 4 .5 %

1 2 .7 %

6 .2 %

1 . 6 %

6 0 .4 %

0 % 2 0 % 4 0 % 6 0 % 8 0 %

8

9

1 0 = E x t re m e ly L ik e ly

N = 308

Base = All Respondents

Source: Source: IDC Converged and Integrated Systems End-User

Survey, July, 2013

0 = N o t a t a ll L ik e ly 1 3 .7 %

1 2 .7 %

2 1 . 3 %

3 2 . 3 %

4 4 .0 %

5 1 4 .3 %

6 1 6 .3 %

7 1 3 .0 %

8 8 . 7 %

9 2 . 3 %

1 0 = E x t re m e ly L ik e ly 2 1 .3 %

0 % 2 0 % 4 0 %

N = 300

Base = All Respondents

Source: IDC Converged Systems Survey, July 2012

20 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

7

6 0 % 8 0 %

Page 20: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

What Customers ThinkAbout Engineered Systems for Oracle Databases.

“Standardizing our database services and configurations has yielded benefits

across many dimensions.” Andy Wottenhofer, University of Minnesota

“Oracle Database Appliance enables us to provide a single system solution that's

affordable and simple to deploy.” Luigi Giuri, Wirex

“There are a lot less people involved to support the Oracle Exadata system.” Alex

Mann, Garmin

“Oracle Exadata achieved a large reduction in administration time along with huge

storage savings costs and greatly improved performance.” Eric Zonneveld, KPN

21 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 21: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

INTRODUCING

ORACLE DATABASE BACKUP LOGGING RECOVERY APPLIANCE

22 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 22: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Database Backup and RecoveryTypical customer challenges.

“We take regular backups, but have no confidence that backups will work when

needed.” Global Payments Company

“With close to 1000 databases to protect, standardization and automation of

backup/recovery is very important to us.” Pharmacy Benefit Management Company

“Trying to reduce our backup windows has traditionally been a problem for us

especially with our larger databases.” Global Research Company

23 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 23: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Database Protection Without CompromiseOracle Database Backup Logging Recovery Appliance

000s of

Databases

Validated and

compressed changes

Redo and

change data

Replicate changes

offsite to the Cloud

24 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 24: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

What Customers ThinkAbout the Oracle Database Backup Logging Recovery Appliance.

“This new appliance can solve our recovery problems for us.” Global Payments

Company

“It will allow us to simplify the DBA and storage management disconnect.” Global

Oil Company

“With this appliance the database backup process is dramatically simplified, and

much shorter backup windows.” Global Research Company

25 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 25: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Customer Initiatives

26 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Database as a Service

Cloud Big Data

26 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 26: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Engineered for Big Data & Data Warehousing

Oracle

Exadata

Oracle

Oracle Big Data

Appliance

Acquire Analyze

Exalytics

DecideOrganize

27 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 27: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Big Data

The Customer ’s View

29 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

v

Page 28: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

“Find transactions that suggest fraudulent activity”

• Recognize event patterns in sequence

• Simplify and scale analysis of Big Data

• Find answers as quickly as possible

Big Data AnalyticsScalable discovery of business events

30 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 29: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Simplifying Big Data AnalyticsSQL Pattern Matching with Oracle Database 12c

• Clickstream logs:

– sessionization, search behaviour

• Business transactions:

– fraud detection, stock analysis

• Sensor data:

– Automated observations and detections

Select * from

Transactions MATCH_RECOGNIZE (

PATTERN(S X{2,4} Y)

DEFINE S AS (type = T),

X AS (loc =

PREV(loc)), Y AS (loc

!= PREV(loc))

)

Ascendin

gO

rder

Possible fraudulent

banking transactions

defined as regular

expression

31 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 30: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

return (status ? 0 : 1);}

public static void main(String[] args) throws Exception {int exitCode = ToolRunner.run(new FraudFinder(), args);//int exitCode = 0; if (exitCode == 0) {

exitCode = ToolRunner.run(new AggregateJob(), args);}System.exit(exitCode);

}}

package fraudfinder;

import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; import org.apache.hadoop.util.Tool;

public class AggregateJob extends Configured implements Tool {@Overridepublic int run(String[] args) throws Exception {

boolean status = false;

Configuration conf = this.getConf();

Job job = new Job(conf, "aggregator");

job.setJarByClass(AggregateJob.class);

job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class);

job.setMapperClass(AggregateMapper.class); job.setReducerClass(AggregateReducer.class);

job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class);

FileInputFormat.setInputPaths(job, new Path(args[1])); FileOutputFormat.setOutputPath(job, new Path(args[1] + "_aggregate"));

status = job.waitForCompletion(true);

select count(*) number_of_occurences, sum(fraudulant_amount) amount_defrauded,

fraud_occured_at_loc

from (select * from transactions)

MATCH_RECOGNIZE

( PARTITION BY ACCOUNT_ID ORDER BY TRANS_TIME

MEASURES

x.trans_time as last_trans_time,

y.trans_time as time_of_fraud,

y.amount as fraudulant_amount,

x.location_id as previous_location,

y.location_id as fraud_occured_at_loc,

ROUND(FRAUD_ANALYSIS.DISTANCE_RATIO(Y.LOCATION_ID, X.LOCATION_ID, Y.TRANS_TIME,

X.TRANS_TIME),2) as fraud_likleyhood

ONE ROW PER MATCH

PATTERN (S X{2,4} Y)

DEFINE S as (trans_type='T'),

X as (trans_type='T' and S.LOCATION_ID = X.LOCATION_ID ),

Y as (trans_type='T'

AND Y.LOCATION_ID != X.LOCATION_ID

)

)

where fraud_likleyhood > 1

group by fraud_occured_at_loc

order by number_of_occurences desc

18 Lines of SQL

Using SQL Pattern MatchingFewer lines of code required and runs over 50x faster

return (status ? 0 : 1);}

}

package fraudfinder;

import fraudfinder.locutil.DistanceRatioCalculator; import java.util.ArrayList;

public class PatternMatcher {private ArrayList<PatternElement> patternFifo = new ArrayList<PatternElement>(); private PatternElement xPE;private PatternElement yPE; private boolean match; private int matchXCount= 0;private int maxMatchXCount = Integer.MAX_VALUE;

public void addPatternElement(PatternElement element) {

32int patt

Cern

oLpenygrtihg

=h

pta©tter

2nF0ifo1.s3iz,eO();

racle and/or its affiliates. All rights reserved.match = false;if (!element.isTransTypeT()) {

clearPattern();

650+ Lines of Java Map Reduce

Page 31: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

INTRODUCING

ORACLE DATABASE IN-MEMORY OPTION

34 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 32: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Oracle Database In-MemoryGoals

100X Faster Queries: Real-Time Analytics

• Get instantaneous query results

• Querying OLTP database or data warehouse

2X Increase Transaction Processing Rates

• Insert rows 3 to 4X faster

35 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 33: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Optimizing Transaction and Query PerformanceRow Format Databases versus Column Format Databases

Row

• Transactions run faster on row format

– Insert or query a sales order

– Fast processing few rows, many columns

ORDER

SALES

Column

• Analytics run faster on column format

– Report on sales totals by state

– Fast accessing few columns, many rows

SALES

S

T

A

T

E

36 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 34: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

• Simultaneously active and

transactionally consistent

• Simultaneously active and

transactionally consistent

Dual Format In-Memory DatabaseBOTH row and column in-memory formats for same data/table

Sales

Column

Memory

Sales

Row

Memory

AnalyticsOLTP

• 100X Faster Analytics &

reporting: column format

• 2X Faster OLTP: row format

• 100X Faster Analytics &

reporting: column format

• 2X Faster OLTP: row format

FormatFormat

37 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 35: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Demo

Oracle Database v

In-Memory

38 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 36: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

lnd!ex

8 RunTraditionalR u n 12'c ln,Mernory

6i!Parallel

Traditional In-Memory 12c In-Memory Option

180Million Rows Scanned I Sec

12473Million Rows Scanned I Sec

170,_

0

I12eIn-Memory

Option

iDl l_

0

·uo

TraditionalIn-Memory 'Q)

l l_

0

25-AUG 13 WBB>f..S 31·AUG 13

Page 37: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Oracle Database In-Memory is Trivial to DeployNo changes required to existing applications

1. Configure Memory Capacity• inmemory_area = XXXX GB

2. Configure tables or partitions to be in memory• alter table | partition … inmemory;

3. Later Drop analytic indexes to speed up OLTP

40 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 38: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Oracle Database In-Memory OptionLeading edge In-Memory technology

• Seamlessly integrated into Oracle Database 12c

• Delivers extreme performance for

– Analytics and ad-hoc reporting on live data

– Enterprise OLTP and Data Warehousing

– Scale-up and scale-out

• Trivial to deploy for all applications and customers

41 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 39: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

Engineered for Clouds and Big Data

42 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Big Data

Database as a Service

Cloud

42 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 40: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

43 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

Page 41: 1 Copyright © 2013, Oracle and/or its affiliates. All …nyoug.org/Presentations/2013/Winter/Hardie-NYOUG-2013...SQLPattern Matching with Oracle Database 12c •Clickstream logs:

44 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.