40621435 123217 oracle streams step by step ppt
TRANSCRIPT
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
1/30
Oracle Streams: Step by Step
What is Oracle Streams? What is Advanced Queues (AQ)?
What is Change Data Capture (CDC)? What is a Logical Change Record (LCR)? How Streams Works How do you configure Streams, AQ and CDC?
How do you use Streams in your programs? Adding records automatically Adding record programmatically Pulling records off the queue
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
2/30
Who am I
Lewis R Cunningham Oracle ACE
Certified PL/SQL Developer Author Blogger Database Geek
Member of SOUG, IOUG, ODTUG, ACM
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
3/30
What is Oracle Streams?
Streams enables data and event movement Movement may be within or between
databases Can implement replication Data and events can be captured
explicitly: programatically Implicitly: redo log
Performant Log mining reduces overhead Downstreams mining reduces it even more
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
4/30
What is Oracle Streams?, cont'd
Streams Features Distinct capture/propagation/apply services Message Queueing (via AQ) Publish Subscribe Rules based Capture/Transform/Apply Database integration Easy configuration
Flexible Inline transformations
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
5/30
What is Oracle AQ?
Oracle's messaging solution Queue based
Persistent queues protect data Buffered, non-persistent queues offer maximum
performance
Oracle provides all of the maintenance code Secure Queue Level Access Control Messages Types
Raw XML Object
Anydata
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
6/30
What is Oracle AQ?, cont'd
Publish/Subscribe Multi-consumer Multi-message types
RAC integration API support
OCI (C), OCCI (C++) Java JMS
PL/SQL OO40 (COM) Internet Data Access (IDAP) ODBC via Heterogeneous Gateways
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
7/30
What is Change Data Capture?
CDC efficiently identifies changed data andmake it available for downstrean use
CDC enables efficient warehouse loading No flat files Direct data movement
Changes are queued Multiple subscribers are allowed Subscribers can receive changed data Data may be transformed Changes can include DDL and DML
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
8/30
LCR = Logical Change Record Committed records generate an LCR to the
redo log An LCR shows the difference between an
existing record and the new record For DDL, the record is a database object For DML, the record is table data
What is an LCR?
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
9/30
What is an LCR?
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
10/30
0
How it works
Sourc t s T r t t s
T
l
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
11/30
11
Data Changes
Sourc t s T r t t s
T
l
og
Writ r
t
i ul tion
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
12/30
12
DB Writes to OS File
Sourc t s T rget t se
T
le
Log
Writer
LOG
Writes To
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
13/30
13
Streams Takes Over
Source t se T rget t se
T
le
Log
Writer
OLOG
Streamsapture
Capture rocess
eads edoand
tracts t eLogical
Change ecord
(LCR)
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
14/30
14
Streams Queues The LCR
Source atabase Target atabase
Table
Log
Writer
REDOLOG
StreamsCapture
Streams
ueue
Streams Capture rocess
rites theLCR toa local OR
REMOTEStreams ueue
Table
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
15/30
15
LCR Propagates
SourceDatabase Target Database
Table
Log
Writer
REDOLOG StreamsCapture
Streams
ueue
Streams Capture rocess
rites theLCR toa local OR
REMOTEStreams ueue
Table
Streams
ueue
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
16/30
16
Changes Are Applied
SourceDatabase Target Database
Table
Log
Writer
REDOLOG StreamsCapture
Streams
ueue
Streams Capture rocess
rites theLCR toa local OR
REMOTEStreams ueue
Table
Streams
ueue
Streamsppl
Streams ppl
canalso transform
forone record to
many, add
columns, etc.
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
17/30
17
Oracle StreamsOverview - US IFS
Target Database
SourceDatabase Target Database
Table
Log
Writer
REDOLOG StreamsCapture
Streams
ueue
Streams Capture rocess
rites theLCR toa local OR
REMOTEStreams ueue
Table
Streams
ueue
Streamspply
Table
Table
Table
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
18/30
Configuring Streams
The Scenario Capture changes to the Employee table in 10g
ORCL instance) Send changes to Employee_audit in 9i
(SECOND instance) We will start with capture and apply in 10g Once the capture and apply is working, we will
modify the stream to send the data to 9i
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
19/30
Configuring Streams 10g
Create Streams Admin Setup CDC Queue Create Capture Rule Instantiate SCN Create Apply Rule Add Supplemental Logging Create DML Procedure Create DML Handlers Set Parameters Start Apply Start Capture
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
20/30
Configuring Streams 9i
Create AQ Table Create AQ Queue Start AQ Queue Create Propagate on 10g Create Dequeue Procedure Dequeue Transactions
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
21/30
Configuring Streams
St
fi
Sou r
r
t
!
ml
"
#
$
im
Se tup CreateStreams % &
mi
CreateStreams % &
m i
CREATE '
SE RStreams ;
( RANT ) 0 ATOStreams ;
BEG IN
)
B1
S_STREAMS_AUTH.GRANT_ADM IN_PRIVI2
EG E(
rantee3
> 'streams ',
rant_pri4
ileges3
> true);
END ;
2 CDC SetupCDC5
ueue
BEG IN
DBMS_STREAMS_ADM .SET_UP_QUEUE(
6
ueue_table 3 > 'streams .streams_ent_queue_table',
queue_name3
> 'streams .streams_ent_queue' );
END ;
3 CDC CreateCaptureRule
BEG IN
DBMS_STREAMS_ADM .ADD_TAB2
E_RULES(
table_name3
> '
es_ods.ent2',
streams_t7
pe3
> '
apture',
streams_name 3 > '
apture_ent2',
queue_name 3 > 'streams .streams_ent_queue',inc lude_dml
3> true,
inc lude_ddl3
>
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
22/30
Configuring Streams
4 CDC Instantiate SCN
DECLARE
l_scn number;
BEGIN
l_scn := DBMS_FLASHBACK.GET_SYSTEM_CHANGE_NUMBER
();
DBMS_AP PLY_ADM.S ET_TABLE_INSTANTIATION_SCN (
source_object_name =8
'ces_ods.ent2',
source_database_name =8
'lbces.world',
instantiation_scn =8
l_scn
5 CDC Create Apply Rule
DECLARE
entity_rule_name_dml9
ARCHAR2(30);
entity_rule_name_ddl9
ARCHAR2(30);
BEGIN
DBMS_STREAMS_ADM.ADD_TABLE_RULES(
table_name = 8 'ces_ods.ent2',
streams_type =8
'apply',
streams_name = 8 'apply_ent2',
6 CDC
Add Any S umplemental
Logging
BEGINDBMS_CAPTURE_ADM.INCLUDE_EXTRA_ATTRIBUTE(
capture_name = 8 'capture_ent2',
attribute_name =8
'username',
include =8
true);
END ;
/
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
23/30
Configuring Streams
Create DML Handler Proc CREATE OR REPLACE PROCEDURE emp_dml_handler(in_any
IN ANYDATA) ISlcr SYS.LCR$_ROW_RECORD;
rc PLS_INTEGER;command VARCHAR2(30);old_values SYS.LCR$_ROW_LIST;
BEGIN-- Access the LCRrc := in_any.GETOBJECT(lcr);-- Get the object command type
command := lcr.GET_COMMAND_TYPE();
-- I am inserting the XML equivalent of the LCRinto the monitoring table.insert into streams_monitor (txt_msg)
values (command ||DBMS_STREAMS.CONVERT_LCR_TO_XML(in_any) );
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
24/30
Configuring Streams
Create DML Handler (cont'd) -- Set the command_type in the row LCR to INSERT
lcr.SET_COMMAND_TYPE('INSERT');-- Set the object_name in the row LCR to EMP_DEL
lcr.SET_OBJECT_NAME('EMPLOYEE_AUDIT');
-- Set the new values to the old values forupdate and deleteIF command IN ('DELETE', 'UPDATE') THEN
-- Get the old values in the row LCR
old_values := lcr.GET_VALUES('old');-- Set the old values in the row LCR to thenew values in the row LCR
lcr.SET_VALUES('new', old_values);-- Set the old values in the row LCR to NULLlcr.SET_VALUES('old', NULL);
END IF;
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
25/30
Configuring Streams
Create DML Handler (cont'd) -- Add a SYSDATE for upd_date
lcr.ADD_COLUMN('new', 'UPD_DATE',ANYDATA.ConvertDate(SYSDATE));
-- Add a user columnlcr.ADD_COLUMN('new', 'user_name',
lcr.GET_EXTRA_ATTRIBUTE('USERNAME') );-- Add an action columnlcr.ADD_COLUMN('new', 'ACTION',
ANYDATA.ConvertVarChar2(command));
-- Make the changeslcr.EXECUTE(true);commit;END;/
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
26/30
Configuring Streams
8 CDC Create DML handler@
AE
BIN
DA
MS_APPLY_ADM.SET_DML_HANDLER(
object_na C e => 'ce@
_od@
.entD
' E
object_type => 'TAA
LE 'E
operat ion_naC
e => 'INSERT'E
error_handler => fal@
e,
u@
er_procedure => '@
treaC
@
.ce@
_dC
l_handler',
apply_d
9 CDC
Set Any Additional
Para C eter@
AE
BIN
DA
MS_APPLY_ADM.SET_PARAMETER(
apply_naC
e => 'apply_entD
',
paraC
eter => 'di@
able_on_error',
value => 'n' F ;
END ;
G
H
CDC Start Apply
AE
BIN
DA
MS_APPLY_ADM.START_APPLY(
apply_naC
e => 'apply_entD
'F;
END ;
G GCDC Start Capture
AE
BIN
DA
MS_CAPTURE_ADM.START_CAPTURE(
capture_naC
e => 'capture_entD
'F;
END ;
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
27/30
Configuring Streams
12 AQ CeateAQ QTable CreateAQ QTable
BEG IN
DBMS_AQADM.CREATE_QUEUE_TABLE(
queue_tableI
> 'lrc_src_ent3_mc_t',
queue_payload_type I > 'sys.P
mltype ',
multiple_consumersI
> TRUE,
compatibleI
> '8.1');
END ;
13 AQ CreateAQ QQ CreateAQ QQ
BEG IN
DBMS_AQADM.CREATE_QUEUE(
queue_name I > 'streams .lrc_src_ent3_mc_q ',
queue_tableI
> 'streams .lrc_src_ent3_mc_t');
END ;
14AQ Start AQ Q Start AQ Q
BEG IN
DBMS_AQADM.START_QUEUE (
queue_nameI
> 'streams .lrc_src_ent3_mc_q ');
END ;
15 AQ CreatePropSchedule
BEG IN
DBMS_STREAMS_ADM .ADD_TABLE_PROPAGATION_RULES(
table_nameI
> 'ces_ods.ent2',
streams_name I > 'ces_to_gems 2',
source_queue_nameI
> 'streams .lrc_src_ent3_mc_q ',
destination_queue_nameI
>
'streams [email protected]
orld',
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
28/30
Configuring Streams
Create a DEQUEUE proc in 9i CREATE OR REPLACE PROCEDURE emp_dq (consumer IN
VARCHAR2) AS msg ANYDATA; row_lcr SYS.LCR$_ROW_RECORD; num_var pls_integer; more_messages BOOLEAN := true; navigation VARCHAR2(30); BEGIN navigation := 'FIRST MESSAGE'; WHILE (more_messages) LOOP DBMS_OUTPUT.PUT_LINE('Looping thru messages');
BEGIN DBMS_STREAMS_MESSAGING.DEQUEUE( queue_name => 'strmadmin.streams_queue', streams_name => consumer, payload => msg, navigation => navigation, wait => DBMS_STREAMS_MESSAGING.NO_WAIT);
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
29/30
Configuring Streams
Create a DEQUEUE proc in 9i IF msg.GETTYPENAME() = 'SYS.LCR$_ROW_RECORD' THEN num_var := msg.GetObject(row_lcr); DBMS_OUTPUT.PUT_LINE(row_lcr.GET_COMMAND_TYPE || ' row LCR dequeued'); END IF; navigation := 'NEXT MESSAGE'; COMMIT; EXCEPTION WHEN SYS.DBMS_STREAMS_MESSAGING.ENDOFCURTRANS THEN navigation := 'NEXT TRANSACTION'; WHEN DBMS_STREAMS_MESSAGING.NOMOREMSGS THEN
more_messages := false; DBMS_OUTPUT.PUT_LINE('No more messages.'); WHEN OTHERS THEN RAISE; END; END LOOP; END; /
-
8/8/2019 40621435 123217 Oracle Streams Step by Step PPT
30/30
Summary
Streams only looks complex
It's conceptually simple and is implementedvia a few procedures
Questions?
Thank you for coming