sequences
Post on 12-Oct-2015
37 Views
Preview:
DESCRIPTION
TRANSCRIPT
-
5/21/2018 Sequences
1/64
Sequences:
Job Sequencer now supports:
Checkpoint/Restart
Looping
Expression Support
User Variables Stage
Automatically Handling Aborts
Terminator Stage ~ aborting a sequence
Enhanced Variable Support Enhancements to other activity stages:
WaitForFile Stage
Notify Stage
Checkpoint/re-start mechanism for Sequences:
Set/unset via a job property Compiler generates different code.
Continued
-
5/21/2018 Sequences
2/64
Sequences
If a job fails within a sequence, the sequence can be restarted and will skip tothe pointing of failure before starting execution:
Successfully run jobs are not restarted Failed (aborted) jobs are reset & run Activities occurring beyond the original point of failure are run when sequence
reaches them.
If a sequence is in the "Aborted / Restartable" state: "Run" will restart it from the checkpoint
"Reset" will clear the checkpoint information so the next "Run" will start theentire sequence again Recompiling the sequence clears checkpoint information too
-
5/21/2018 Sequences
3/64
Checkpoints:
Checkpointing must be enabled at design time:
Project wide setting to enable checkpointing in all new sequences.
Job level property to override the project setting for a given sequence.
Activity (stage) level property to allow disabling of checkpointing for a specific job runor activity (routine, command etc.) within a sequence.
-
5/21/2018 Sequences
4/64
When SEQUENCE runs:
JOB1fails
JOB1OK, JOB2fails
JOB1OK, JOB2OK
JOB3fails
JOB1OK, JOB2OK
JOB3OK
When SEQUENCE is restarted:
Reset JOB1 and run it again
Skip JOB1, reset JOB2 and run it again
Skip JOB1 & JOB2, reset JOB3 and run it again
Start at JOB1 again.
Method of execution:
-
5/21/2018 Sequences
5/64
Step1 :This is how a sequence of jobs looks like.
Recovery and Re-Startability :
-
5/21/2018 Sequences
6/64
Step2 :Go to Job Activity Stage's Properties and check "Add checkpoints so sequence is re-startable on failure" option.
-
5/21/2018 Sequences
7/64
Step3 :For example, let the "Loan Management" job alone is erroneous. After rectifying,check the "Do not checkpoint" option so as to skip the "Customer Management" from re-execution.
-
5/21/2018 Sequences
8/64
Step4 :The "Customer Management" is skipped, the "Loan Management" alonere-executed.
"CustomerManagement"
is skipped.
"Loan
Management"is re-executed.
-
5/21/2018 Sequences
9/64
Job Sequencer Loop Constructs:
Loop Variable available to activities within the loop
startloopname.$Counter
Contains loop iteration number or current list item
Loops may be nested
each End must point to one Start
2 counter variables are available in the inner loop
-
5/21/2018 Sequences
10/64
Loop Activities :
-
5/21/2018 Sequences
11/64
Loop Property :
-
5/21/2018 Sequences
12/64
Loop Activity Log :
-
5/21/2018 Sequences
13/64
Job after a successful run :
-
5/21/2018 Sequences
14/64
Expression Support:
Transformer-like expression syntax added to:
Job activity ~ setting job parameter values
Routine activity ~ setting routine argument values
Triggers ~ defining custom conditions
Context-menu help available when editing expressions, similar to Server jobtransformer + activity variables
Call Routines/Transforms/DSMacros
Do Arithmetic/Comparisons/If...Then...Else etc.
All other single-line fields:
Accept either straight text, or job parameters/activity variables surrounded by "#", orcombination
Have button to help select from variables available at that point
E.g. Command Activity, Notification (SendMail), WaitForFile
-
5/21/2018 Sequences
15/64
-
5/21/2018 Sequences
16/64
Job Sequencer User Variables:
-
5/21/2018 Sequences
17/64
Continued
User Variables
-
5/21/2018 Sequences
18/64
User Variables
-
5/21/2018 Sequences
19/64
Job Sequencer Terminator Stage:
Easy way to abort a sequence cleanly:
Optionally send STOPs to all running jobs
Optionally wait for all jobs to finish
Log final abort message
Can have any number of these in different branches of a sequence:
Only one will get executed of course
-
5/21/2018 Sequences
20/64
Terminator Activity :
A terminator stage can be placed in a job sequence to ensure that the sequence is stoppedcleanly if certain situations arise. You can have multiple terminator activities and can placethem anywhere in the sequence. They are connected to other stages by triggers, which specifywhen a terminator will be invoked.
-
5/21/2018 Sequences
21/64
Terminator Property :
-
5/21/2018 Sequences
22/64
Example job :
-
5/21/2018 Sequences
23/64
Handling Exceptions (Automatic):
Whathappen toWarnings?
Automaticallyaborts when an
error is detected.
Triggers"Executed OK"
-
5/21/2018 Sequences
24/64
Handling Exceptions with Triggers:
-
5/21/2018 Sequences
25/64
Handling Exceptions (Best Practice):
Warnings are Explicitly HandledOr
Set Warning Threshold to 1
AutomaticException
Handler
Uses variabels:$ErrSource
$ErrNumber$ErrMessage
ControlParallel
Streams
-
5/21/2018 Sequences
26/64
Parameter Management Strategies:
Standard parameter management:
Works fine if you are organized.
Naming standards a must. Establish early. Parent job propagates to child job.
Set default values in director.
Utilize templates to manage frequently used parameter sets.
Utilize environment variables for project wide values.
Password management: Encrypt
Must use Standard parameter management for security.
Set default values in director.
Or use encrypted environment variables per project.
-
5/21/2018 Sequences
27/64
Standard Parameter Management:
Job Parameters.
Sequence.
Environment variables.
Encrypted environment variables.
St d d P t M t
-
5/21/2018 Sequences
28/64
A sequence of two jobs FileSet and DataSet.
Continued
Standard Parameter Management
Standard Parameter Management
-
5/21/2018 Sequences
29/64
Continued
Standard Parameter Management
Standard Parameter Management
-
5/21/2018 Sequences
30/64
Enter the value expression using Insert Parameter Value:
Continued
Standard Parameter Management
Standard Parameter Management
-
5/21/2018 Sequences
31/64
Insert Parameter value with the same procedure to the next job activity also
Compile and run the job sequence
Insert Parameter val
Continued
Standard Parameter Management
Standard Parameter Management
-
5/21/2018 Sequences
32/64
View Log File:
Continued
Standard Parameter Management
A f I d F db k & A di
-
5/21/2018 Sequences
33/64
Aspects of Integrated Feedback & Audit:
Tools and methods:
Getting row counts: numRowsModified = DSGetLinkInfo(JobHandle, ActiveStageName,
LinkName, DSJ.LINKROWCOUNT)
Automatic Sequence Features Log Warnings for Activities that finish with status other than okay. Log Report Messages after each job run
Escalate with DSJobController ControllerHandle = DSAttachJob (DSJobController, DSJ.ERRFATAL ) EventError =DSLogEvent (ControllerHandle, DSJ.LOGWARNING,
MessgeTxt)
Stats from the command line : dsjobrepot
DSMakeJobReport
Other tools (for reading files, invoking OS commands and parsing output) Call DSExecute(ShellType, Command, Output, SystemReturnCode)
R Ti M H dli
-
5/21/2018 Sequences
34/64
Run Time Message Handling:
Runtime Message Customization:
Allows the user to customize how runtime messages are handled in DataStage
Based on definition of "Message Handlers"
Specify the action to be taken if a particular message is generated
Customization is based on the message ID
Three customization actions are supported:
Suppress message [Informational & Warning messages only] Demote message [from Warning to Informational]
Promote message [from Informational to Warning]
Message Handlers can be applied to whole projects or to individual jobs.
A Message Handler is a named file containing the customization rules.
Continued
Runtime Message Handling
-
5/21/2018 Sequences
35/64
To define a Message Handler: From Director log view, select the message to handle and then select "Job->Add rule
to message handler" menu option or right-click & select "Add rule to messagehandler" on the context menu.
Message can be added to a specific message handler file, a project handler or localhandler for this job.
Action is dependent on message type.
Types of Message Handlers: Local runtime message handler
Applies to the current job, and takes effect next time the job is run. Set in Job Properties->Defaults tab Gets compiled into the job and is exported with the executable Applied before any project level handler
Project message handler A Message Handler file which is associated with the current project Set in the Administrator client on the Project Properties->Parallel tab Applied after the local runtime handler (if enabled)
The combination of the two levels (both optional) defines the message handling for agiven job.
g g
Continued
Runtime Message Handling
-
5/21/2018 Sequences
36/64
Managing Message Handlers: A management tools enables the user to Create, View, Edit and Delete message handler
files. Available from a "Message Handlers" option on the Manager and Director Tools menus for
Parallel enabled projects only.
Managing Handler Manager: Displays handler rules in either the local runtime handler, the project handler or the
selected handler.
Rules can be edited * the handler saved (overwriting or "as new").
g g
Continued
M H dl
-
5/21/2018 Sequences
37/64
Message Handlers:
Continued
Message Handlers
-
5/21/2018 Sequences
38/64
Director > Job > Add rule to message handler
Message Handlers
Continued
Message Handlers
-
5/21/2018 Sequences
39/64
Click previous till message text appears, select the action to be done:
Message Handlers
Continued
Message Handlers
-
5/21/2018 Sequences
40/64
Click Add Rule and click OK:
Message Handlers
Continued
Message Handlers
-
5/21/2018 Sequences
41/64
Close and run the job View the log file:
Message Handlers
Continued
Message Handlers
-
5/21/2018 Sequences
42/64
Creating new message handler:
Director > Tools > Message Handler Management
Message Handlers
Continued
Message Handlers
-
5/21/2018 Sequences
43/64
Click New and click Save Enter a name:
Message Handlers
Continued
Message Handlers
-
5/21/2018 Sequences
44/64
Designer > View > JobProperties > Defaults > select the name given:
g
Continued
Message Handlers
-
5/21/2018 Sequences
45/64
Open Add rule to message handler:
Select Add rule to message handler
Select the given name from message handler
Click Add rule
Click OK, Close
Compile and run the job
g
Continued
Message Handlers
-
5/21/2018 Sequences
46/64
View director log:
g
Continued
Nested Condition:
-
5/21/2018 Sequences
47/64
Nested Condition:
Continued
Nested Condition
-
5/21/2018 Sequences
48/64
Continued
Nested Condition
-
5/21/2018 Sequences
49/64
Continued
Log file:
Debugging Changes:
-
5/21/2018 Sequences
50/64
Debugging Changes:
Debugging helpers:
Option to append row number and file name to row being processed, option to read only
(n) number of rows and first line is columns names
Routine Activities for Unlimited Flexibility:
-
5/21/2018 Sequences
51/64
Routine Activities for Unlimited Flexibility:
Combine repetitive operations.
Reuse.
More parameter control.
Take advantage of graph walking:
Checkpoint
Sub flow selection
Return status of flow
Routine Activity Stage :
-
5/21/2018 Sequences
52/64
Routine - DSJobReport :
Routine Activity Stage :
-
5/21/2018 Sequences
53/64
RoutineDSSendMail :
-
5/21/2018 Sequences
54/64
E-Mail Notification Property :
-
5/21/2018 Sequences
55/64
-
5/21/2018 Sequences
56/64
ExecDOS Routine :
Step1:
-
5/21/2018 Sequences
57/64
Step2:
-
5/21/2018 Sequences
58/64
Log Display:
-
5/21/2018 Sequences
59/64
Step1:
ExecTCL Routine :
-
5/21/2018 Sequences
60/64
Step2:
-
5/21/2018 Sequences
61/64
Log Display:
-
5/21/2018 Sequences
62/64
ExecSH : ExecSHSilent :
-
5/21/2018 Sequences
63/64
ExecSH Property : ExecSHSilent Property:
-
5/21/2018 Sequences
64/64
ExecSH Log : ExecSHSilent Log:
top related