ibm system z and z/os java batch solutions...© 2013 ibm corporation ibm americas advanced technical...

© 2013 IBM CorporationIBM Advanced Technical Skills

IBM System z and z/OS

Java Batch SolutionsVersion: September 10, 2013

© 2013 IBM CorporationIBM Americas Advanced Technical SkillsGaithersburg, MD2

Version: September 10, 2013

This page intentionally left blank



Agenda / Table of Contents

Material Available Online …

Overview

Java on z/OS

JVM Launcher Solutions

WebSphere Java Batch

Java Batch in CICS, DB2

Batch and Decision Management

Open Standard Java Batch



This Presentation Available Online

Overview …

ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101783

PDF in Chart FormatShowing the charts as you're seeing them here

PDF in Speaker Notes FormatSame charts, but printed so the speaker notes associated with each chart appear.



OverviewSetting the stage for the discussion of modern batch solutions



Batch (or Bulk) Processing

Examples of bulk processing …

Many definitions exist ... they all have in common the relative lack of human interaction, and the expectation of results at a future time rather than immediately:

Request

Response

Request

Response

In general:● Interaction is one-for-one ... that

is, request with matching response

● Expectation is for response to follow request in a near-immediate span time frame

Online Processing

Submit

Complete

Start

EndProcess

In general:● Interaction is one-for-many... that is, initial request

results in many results from processing

● Expectation is for results to finish within some determined non-immediate time frame

Bulk Processing



Some Examples of Batch (or Bulk) Processing

What's Behind This? …

Just to set some context for the upcoming discussion

Month-End Tax or Fee Calculation and BillingCustomer records are processed with tax or other calculations processed against activity. This may be part of a larger process of calculating amount owed and formulating billing.

Period-End Statements and ReportsAn example is investment portfolio results and reporting.

Data TransformationRaw data records are processed with goal of transforming some aspect of the data content or layout. Result is a second set of data with the intended formatting.

Data AnalysisData records are analyzed to determine trends or patterns. Data mining to find new potential markets is one example. Analyzing vast quantities of seismic data is another.

Point is these are activities where processing in bulk is a better way to maximize efficiency of data access and computer resources. The

completion time is determined but not immediate.



What's Behind This?

Approaches to Batch Modernization …

Every business has different motivators. The common ones we've seen:

Batch Window CompressionThe window in time for batch processing is shrinking. There's a need to better manage online and batch processing concurrently within the same system.

Java Skills and Common ToolingJava skills are more common that traditional programming skills. Further, Java tooling for online work is powerful and capable of being used for batch programming as well.

GP cycles and Offload to Specialty EnginesCost pressures are creating a need to explore ways of offloading processing cycles from GP to specialty engines such as zAAP or zAAP-on-zIIP.

Other motivators may exist. The key is that these motivators are real and they are driving exploration of modern batch.



Approaches to "Batch Modernization"

Framework for discussion …

Generally speaking, we see two basic approaches:

Preserve Existing ... Java Batch for NewLeave existing batch processes as they are today, but as new requirements come up then engineer them into the Java batch model

Re-Engineer Some Existing Batch ProcessesIdentify existing non-Java as candidates and then re-engineer them to operate in a Java batch environment

Typical starting approach: identify batch processes with fewer interdependencies and then work out from there

What we don't see is a "rip-and-replace" strategy. That's too costly and too risky.

Always a reasoned incremental approach.



Framework for Discussion That Follows

Java on z/OS …

The following diagram illustrates the points we will touch on as we review the modern batch technologies available on System z and z/OS:

Modern Batch Technology

What it isAn explanation of its intended purpose

How it worksAn explanation of its functions and features

DifferentiatorsA discussion of how the technology compares to others discussed, and the particular value this technology offers

How it takes advantage of System zAn explanation of System z and z/OS exploitation that may occur with the technology

How it integrates with existing processesAn exploration of integration and usage patterns when other components in your overall architecture are involved



Java on z/OSA story of taking advantage of the platform



IBM Java on z/OS

Specialty Engines …

It's important to understand that while the Java APIs are industry standards, the implementation below the APIs becomes increasingly platform-aware:

ProgrammingAPIs

High-Level JVM Functions

Low-Level JVM Functions

SDK conforms to the accepted standards● IBM SDK provides all the required APIs according to the

specification at the level being discussed● IBM z/OS SDK provides additional APIs to take advantage

of z/OS platform specific functions (such as SAF security)

JVM Functions common across IBM SDKs● The JVM is entirely IBM's ... first delivered in 2005● Many features: generational GC, shared classes● High-level JVM functions common across IBM Java

System z and z/OS functions● Takes specific advantage of platform, including exploitation

of new CISC instructions available with new System z: z10, z196, EC12

● Big Decimal, Large Page, Out of Order execution, transactional execution, flash paging ... equals performance

● Work with z/OS dispatcher to offload to specialty engines



System z Specialty Engines

Timeline …

Specialty engines provide additional processing capacity with an attractive financial profile: lower acquisition cost, not counted towards software license charges

Java Work1 DDF and DB2 Utilities2 Other Work

z/OSDispatcher

zAAP Processors zIIP Processors General Processors

zAAP - System z Application Assist ProcessorOffload of Java and XML parsing work.

zIIP - System z Integrated Information ProcessorCertain DB2 work and XML parsing services.

Note 2 -- Plus other work, see http://www.ibm.com/systems/z/hardware/features/ziip/

Note 1 -- See http://www.ibm.com/systems/z/hardware/features/zaap/

zAAP-on-zIIPA means of more efficiently using specialty engines by defining only a pool of zIIP processors and allowing eligible zAAP work to run on the zIIPs3.

zAAP-on-zIIP

Note 3 -- EC12 planned to be the last system that supports zAAP; after that, zAAP-on-zIIP will be the offload mechanism

Note: simplified representation of offload mechanism. Various tuning and control parameters exist.

IFL - Integrated Facility for LinuxFor running z/VM and Linux. Does not apply to z/OS, but plays strong role in Linux for System z



IBM System z Java Product Timeline

Performance …

This shows the level of commitment to Java and Java on System z:

SDK 1.1.1● First OS/390 Java product● Followed by 1.1.4 and 1.1.6● 31-bit

SDK 1.1.8● 31-bit

SDK 1.3.1● 31-bit● z/OS ● Linux for System z

SDK 1.4● 31-bit z/OS● 64-bit Linux for System z

SDK 1.4.2● 64-bit z/OS● First use of IBM J9 VM

and JIT Technology

SDK 5● 31-bit and 64-bit● z/OS and Linux for System z● z9 exploitation

SDK 6.0.0● Java SE 6 APIs● 31-bit and 64-bit● z/OS and Linux for System z● z10 exploitation

SDK 6.0.1● Java SE 6 APIs● 31-bit and 64-bit● z/OS and Linux for System z● z196 exploitation● Enhanced JZOS and z/OS security

SDK 7● Java SE 7 APIs● 31-bit and 64-bit● z/OS and Linux for System z● More z196 exploitation● zEC12 exploitation● OpenJDK

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012



IBM z/OS Java Performance

Summary …

The story is one of improvement in hardware and Java software

zEC12 SDK 7 SR3

zEC12 SDK 7 SR1

z196 SDK 7 SR1z196 SDK 6 SR8

z10 SDK 6 SR4

z10 SDK 6 GMz9 SDK 5 SR5

CPU Intensive Multi-threaded Workload on 16-way Machine

Better scaling slope leading up to CPU

over-commit

Around 12X improvement

CPU intensive workload so scaling

past over-commit not expected

Four things going on: (1) faster hardware, (2) exploitation of machine instructions, (3) exploitation of machine features (Flash Express paging with large pages), and (4) increasing efficiency of the JVM function itself

Disclaimer: controlled environment with specific conditions during measured testing. Results may vary. This is not a promise of performance or results.

-Xaggressive and large page



Section Summary

JVM Launchers …

A very broad summary of the Java on z/OS section:

IBM Java is fully compliant with Java specificationsProgramming APIs compliant with the specification

IBM Java on z/OS has programming extensionsProvide you option to code apps to extensions for added functionality

IBM Java has IBM JVM, GC and JIT TechnologyBased on decades of experience in operating systems

IBM Java on z/OS takes advantage of System zWhich manifests as performance improvements over time

IBM continues to invest in Java on z/OSCommitment to the platform is evident



JVM Launcher SolutionsBPXBATCH, JZOS and the Batch Runtime



What is a "JVM Launcher?"

Three JVM Launchers on z/OS …

It's simply a means of initiating a JVM start and the running of Java programs in it:

Four questions arise:

1. What JVM launcher technologies are available on z/OS?

2. What does each provide in addition to the JVM?

3. Why would one be used over another?

4. If not JVM launcher, what else might be considered?

Java Program

Java Virtual Machine

Instantiation JVM tear-downProgram Execution

Launch the JVM



Three JVM Launcher Technologies We'll Explore

BPXBATCH …

For the purposes of this discussion we'll focus on three:

BPXBATCHProvides a UNIX shell environment to JCL. From the UNIX shell you may then issue a "java" command to launch the JVM and run the specified program.For purposes of comparison, consider this equivalent to invoking "java" from a shell script or manually from a command line.

JZOSProvides a more comprehensive JVM launcher utility that overcomes some of the shortcomings of BPXBATCH. Includes a set of class libraries and JNI native to access z/OS functions and services.

z/OS Batch RuntimeProvides a functional subset of the WebSphere Java Batch environment for running Java batch programs. Includes such things as transactional management, integration with COBOL, and transactional integrity between Java, COBOL and DB2.



BPXBATCH (and BPXBATSL)

JZOS …

Provides access to a UNIX shell environment from which you may instantiate a JVM and run a program. It works:

http://publib.boulder.ibm.com/infocenter/zos/v1r13/topic/com.ibm.zos.r13.bpxa500/bpxbatr.htm

//BPXBATCH JOB ...//STEP EXEC PGM=BPXBATCH,// PARM='SH java HelloWorld'//STDIN DD DUMMY//STDPARM DD DUMMY//STDERR DD SYSOUT=*//STDOUT DD SYSOUT=*//STDENV DD *CLASSPATH=/u/myclasses/*//*

JVM instantiated from shell with java command, with Java class file specified.

Parameters to Java program may be specified trailing class name in PARM=' ' or in location specified by STDPARM DD.

If STDIN DD specified, it must be a z/OS UNIX file.

STDERR and STDOUT may be JES spool, or UNIX file, or MVS data set

BPXBATSL is like BPXBATCH, but it allows child processes to be fork/exec'd within the same address space. (Restrictions apply, so see InfoCenter article below.)

It provides a JVM with some basic access to z/OS DD statements. It has limitations with respect to other z/OS services. That's why

JZOS was developed and IBM provides that with z/OS.



Overview of JZOS

JZOS Toolkit …

JZOS is technology originally developed by Dovetail Technologies and acquired by IBM. It is now supplied as part of z/OS.

http://www.ibm.com/systems/z/os/zos/tools/java/products/jzos/overview.html

JVM Launcher

Toolkit

JZOSLaunch a JVM and run a Java program

● One address space; all accounting data attributes to the job or started task● Mix Java and non-Java steps in same JCL● Pass condition codes between Java steps, or between Java and non-Java● Access MVS data set via DD statements in Java

Toolkit provides Java classes and native libraries● Methods for allocating, deleting, and renaming MVS data sets● Interface for reading and writing sequential data sets in record mode● Methods for issuing single and multi-line WTOs● Interface for invoking DFSORT and directing I/O into/from Java applications.● Java APIs for submitting z/OS batch jobs from Java● Java APIs that provide interface for invoking IDCAMS (VSAM)● Java APIs for many z/OS functions:

Obtaining job/step/user names, process ids, etc. Writing SMF records Reading environment variables

A more flexible JVM launcher mechanism along with a set of class and native libraries which provide Java programs access to z/OS services



JZOS Toolkit Usable Outside JZOS Launcher

JZOS Launcher …

The JZOS JVM launcher and the JZOS toolkit are not tightly coupled. The toolkit services may be used by other Java Batch mechanisms:


z/OS Batch Runtime

Toolkit Classpath and Libpath updates

z/OS Services Offered by JZOS Toolkit

We've not yet covered WebSphere Java Batch or z/OS Batch Runtime

Key point is JZOS toolkit and those offerings are not mutually exclusiveAnd as we'll see, the z/OS Batch Runtime in fact uses the JZOS JVM launcher as well



z/OS JZOS JVM Launcher

Multi-Step Jobs …

It comes with the z/OS SDK installation. Sample JCL is provided along with the JZOS load module. Invocation involves customizing sample JCL:

/usr/lpp/java/J7.0/mvstools JVMLDM70 /samples /jcl JVMJCL70 JVMPRC70

Note: JZOS is specific to the Java version and the bit-mode. Here SDK 7 31-bit is shown.

● Load module -- copy out to LIBRARY data set

● Calling JCL -- this JCL calls the proc and passes variables

● Procedure -- this receives variables and invokes JZOS

JCLJCLNames the Java class to invoke

Sets up the environment variables

Provides STDIN DD (optional)

Passes arguments (optional)

PRCPRCReceives Java class and arguments

Calls JZOS load module and specifies the Java class file and any passed-in arguments

Defines standard DD statements

Starts address space, instantiates the JVM,

loads the class, invokes the main()

method and supplies any arguments



JZOS and Multi-Step Batch Jobs

Batch Runtime Overview …

JZOS allows you to mix Java and non-Java as steps in a larger batch job. Condition codes set by each step available to other steps in the job.

//MYBATCH JOB ...//STEP1 EXEC PGM=COBOL1 : (step DD cards)

//STEP2 EXEC PROC=JVMPRC70,// JAVACLS='com.ibm.MyClass1' : (step DD cards)

//STEP3 EXEC PGM=ASSEMBLER : (step DD cards)

//STEP2 EXEC PROC=JVMPRC70,// JAVACLS='com.ibm.MyClass2' : (step DD cards)

One job

Multiple steps

Mixed programs

One address space

Same WLM policyService Class (goals)Reporting Class (statistics)

Accounting data consolidated under one job

COBOLCOBOL

JavaJava

AssemblerAssembler

JavaJava



z/OS Batch Runtime Overview

z/OS 2.1 enhancements …

This is function supplied with z/OS 1.13 and higher. It provides a means of integrating Java and COBOL with DB2 JDBC T2 connection shared and TX coordination:

Java Program

DB2 z/OS

COBOL Program● Start TX● Call DB2● Invoke COBOL● Finish TX

● Business Logic● Call DB2● Business Logic● Return

Batch Runtime Transaction and Execution Services

RRS Shared Attach

z/OS 1.13 Batch Runtime

JCLJCL

Submit

Or initialize with COBOL and have it call Java ... it works in either case

The other JVM Launcher technologies provide nothing in the way of TX services. If your code

has TX requirements, then you write custom code.

With the z/OS Batch Runtime the TX services provided and help coordinate between Java and COBOL when using JDBC Type 2 to local DB2.

z/OS 2.1 adds PL/I



z/OS 2.1 - Enhancements to Batch Runtime

JZOS and z/OS Batch Runtime …

The Batch Runtime with z/OS 2.1 comes with enhancements:

z/OS 1.13 z/OS 2.1

Java main() Programs

Single Step Jobs

COBOL

DB2 and JDBC T2

Java main() Programs

Single Step and Multi-Step Jobs

WebSphere Java Batch programsExceptions:● Java SE runtime, not Java EE● No WAS functionality (EJB, MDB, JCA)● No WLM classification, no SMF 120.9● Parallel Job Manager not available

COBOL

PL/I

DB2 and JDBC T2

Transactional VSAM

More on WebSphere Java Batch coming up



JZOS and z/OS Batch Runtime

Integration with Enterprise Schedulers …

The two are not mutually exclusive. In fact, the Batch Runtime makes use of JZOS to launch the JVM and load the Batch Runtime function:

com.ibm.zos.batch.container.BCDBatchContainer

BCDPROC

BCDBATCH

JZOSLauncher

JZOSLauncher

z/OS Address Space

31-Bit Java Virtual Machine

z/OS Batch Runtime

JavaCOBOL

JZOSToolkit

JZOSToolkit

z/OS Servicesz/OS Services

General Requirements:● IBM 31-bit SDK for z/OS, V6.0.1

Must be 31-bit because COBOL is a 31-bit language

● IBM Enterprise COBOL Version 4.2

● DB2 V9 with PTF UK62190 for JDBC 3.0 specification, or PTF UK62191 for JDBC 4.0 specification

● DB2 V10 with PTF UK62141 for JDBC 3.0 specification, or PTF UK62145 for JDBC 4.0 specification level

Calling JCL / procedure similar to JZOS; Batch Container

supplies samples of its own

The JZOS toolkit may also be used in conjunction with the

Batch Container



Integration with Enterprise Schedulers

JVM Launcher Positioning …

All three JVM launcher technologies discussed may be initiated via submission of JCL. That becomes the integration point with enterprise schedulers:

Enterprise Scheduler Function

Job JCL

Java Program

Java Virtual Machine

JES SpoolOr elsewhere if JCL defines other than spool

Submit

In many ways this is business as usual ... to the Enterprise Scheduler function the submitted JCL job is the batch job. The invocation of a JVM

to run Java is a detail not of concern to the scheduler function



JVM Launcher Positioning

Summary …

The following provides a high-level positioning of the alternatives discussed so far:

JVM Bit-mode

Batch Services

DB2 Access

TX Coordination

Programming Model

BPXBATCH JZOS Batch Runtime

Address Spaces

31-bit or 64-bit 31-bit or 64-bit 31-bit

Multiple(one with BPXBATSL)

One One

NoneProvides a JVM but no

other extra-JVM services

SomeToolkit provides z/OS

services. Batch services such as checkpoint not provided

Yesz/OS 2.1 - WebSphere Java

Batch batch services (checkpoint, data stream, etc.)

JDBC T2 or T4Access code is all

custom to the application

JDBC T2 or T4Access code is all

custom to the application

JDBC T2 onlyTX coordination requires RRSAF connect to DB2

z/OS 2.1 - tVSAM as well

NoneAny TX must be custom

implementation

NoneAny TX must be custom

implementation

YesProvided by Batch

Runtime and coordination with RRS

Java main()Standalone Java

Java main()Standalone Java

WebSphere Java Batchz/OS 2.1 - Based on subset of

WebSphere Java Batch model; runs in J2SE not J2EE, and no

WAS classes available.



Section Summary

WebSphere Java Batch …

A very broad summary of the JVM Launcher section:

JVM Launchers instantiate JVMs and run programGenerally an "application per JVM" model

We showed three JVM Launcher ModelsThey range from simple (BPXBATCH) to more functional (Batch Runtime)

Which you use depends on your business needsBPXBATCH may work fine for simple ad hoc functions; JZOS provides a more

flexible launching model with z/OS access services; Batch Runtime addresses the need for Java/COBOL interaction with coordinated transaction support.

Potential downside: cost of JVM start and stopOne potential downside of JVM launchers is the overhead associated with starting and stopping the JVM. May not be an issue if cycles are few; tends to become an

issue when the number of start/stop cycles increases.



WebSphere Java BatchJava Batch inside WebSphere Application Server



WebSphere Java Batch Overview

Two Delivery Mechanisms …

The following is a very high-level overview of the WebSphere Java Batch function:

WebSphere AppServer Instance

EJB Container

Web Container

Batch Container

Java Batch Applications

IBM RAD Tooling Support

Dispatcher Function

z/OS Operating System

System z HW Platform

Job Submission Interfaces

xJCLxJCL

Job Properties Declaration File

Key Points:● This JVM stays up for as long as the WAS server stays up

Days or more ... avoids overhead of JVM instantiation in environments where batch invocation is high

● All the services of WAS z/OS are available to the batch applications running thereTransaction, data access, Parallel Sysplex and clustering, WLM, SAF, SMF

● The batch container provides valuable batch services that keep batch logic focused on the business and away from writing custom middleware codeMulti-step jobs, Batch Data Stream Framework, Checkpoint, Skip record, Step retry, Parallel Job Manager

TechDoc: ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101783

● Browser● Command Line● SOAP web services● RMI● MDB



At Present Two Delivery Mechanisms

Container Services …

The "WebSphere Java Batch" function is delivered in two ways:

WebSphere Application Server

Version 7 or Version 8WebSphere Application Server

Version 8.5

Compute Grid V8WebSphere Java Batch Product

Add to WAS ("Augment") WebSphere Java BatchFunctionally equal to CG V8

For those who have a standardized commitment to

WAS V7 or V8 but desire WebSphere Java Batch function

WebSphere Java Batch function provided in WAS V8.5 product

Note how both are functionally identical



Overview of Batch Container Services

Job Submission and the JMC …

The following provides a summary. The WP101783 Techdoc has many more details:

Batch job dispatching and management system● Separates job invocation and management from the

instantiation of the JVM● Provides job control: stop, restart, cancel, suspend● Multiple interfaces: Web, SOAP, RMI, command, MDB

Data record read and write support services● Abstracts low-level stream handling from business logic● Framework passes data to batch logic as Java object

Checkpoint and job restart services● Checkpoint tracking abstracted from batch logic

and performed as function of container● Commit and rollback a function of the container● Job restart from last checkpoint

Job resiliency services● Ability to skip and track records where data read

or write operation throws exception● Ability to retry job steps when unhandled

exception is thrown● Ability to utilize defined call-out points in batch

job execution to invoke custom Listener code

Parallel job management and execution services● Develop and deploy applications to run segments in

parallel based on your designation of data partitioning● Ability to programmatically partition data ranges based on

passed-in parameters● Container manages top-job and all sub-jobs to completion

COBOL Container Support● Invoke COBOL from Java● JNI stub code provided with container● Call stub generator utility to create call stubs and data

parameter code



Job Submission and Monitoring

CI Model …

The Job Management Console (JMC) is the browser interface. It allows you to submit and monitor the jobs:

Job State Flow

Jobs go through a series of states in this model. The interface to the

Dispatcher Function provides you control to submit, view, cancel, stop,

suspend, restart or remove jobs



High-Level: "Compute Intensive" Model

Transactional Batch Model …

This provides a simple execution model suitable for (a) computational work, or (b) a means of wrappering existing Java main() programs to run:

:

public class MyCIWork implements com.ibm.websphere.ci.CIWork {

:

public void run() {

Your batch code goes here

}

}

Batch Controller BeanStateless session bean provided by container framework

Success if no Exceptions

Thrown

Dispatch

Very simple model to develop and run

Very few batch runtime services available● No data read or write patterns to utilize ... roll your own● No checkpoint processing ... roll your own● No commit and rollback ... roll your own

Use tooling to wrapper Java main() into this framework and it will run in WebSphere Java Batch

Useful model for some types of applications.

Transactional Model may be better suited to your needs ...



High-Level: "Transactional Batch" Model

xJCL …

This is what people usually think of when they think of "batch." At the heart of this is the job step code, which encapsulates the data record business logic:


Dispatch

:public class MyJobStep implements BatchJobStepInterface { : public void createJobStep() { } public int destroyJobStep() { } public int processJobStep() { Step logic for a data record here

if record not null: return BatchConstants.STEP_CONTINUE; if record null: return BatchConstants.STEP_COMPLETE; }}

Controller bean invokes job step in iterative loop until STEP_COMPLETE returned

Your job step logic would process data record

according to business needs



Job Declaration File - xJCL

Checkpoint processing …

Conceptually, xJCL is just like normal // JCL -- it describes the job to be run and the context in which the job is to operate. The difference is xJCL is written in XML:

<?xml version="1.0" encoding="UTF-8" ?><job name="MyJob" ... "> <substitution-props> <prop name="inputDataStream" value="/tmp/input-text.txt" /> <prop name="outputDataStream" value="/tmp/output-text.txt" /> <prop name="checkPoint" value="10" /> </substitution-props>

<job-step name="MyStep1"> <classname>com.ibm.ws.batch.MyStep1</classname> <batch-data-streams> <bds> <logical-name>inputStream</logical-name> <props> <prop name="PATTERN_IMPL_CLASS" value="com.ibm.websphere.batch.EchoReader" /> <prop name="FILENAME" value="${inputDataStream}" /> </props> </bds> </batch-data-streams> </job-step> : <job-step ...> </job-step></job>

Note: this is a trimmed version of actual xJCL

Comparable to the JCL "JOB" card. It sets job-level information along with some substitution properties

Substitution properties may be overridden at submission time.

Comparable to the JCL step. It names the Java class that

implements the step function.

The "Batch Data Stream" implementation is specified. In this

example it defines the class that implements the input stream.

If job consists of more steps they are specified in sequence.

This tells WebSphere Java Batch what to run and how to run it



Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Checkpoint Processing

Batch data stream framework …

Checkpoint commit and rollback is a function of the container. It relies on the transaction manager function of WAS itself for this:

Java Batch Application

Batch Container

xJCL where checkpoint declared

WAS Transaction Manager

Commit

Commit

Last good checkpoint value persisted

Checkpoint interval (record or time) specified in the xJCL

This is a function of the batch container, not your application code

As checkpoint intervals are reached, container commits and records the checkpoint attained

In the event of a failure, job may be restarted at the last good checkpoint

Set the checkpoint interval based on your knowledge of balance between recoverability and efficiency



Job Step Class

High-Level: "Batch Data Stream Framework"

Skip Record …

This provides a way of abstracting the data read and write logic away from the batch step code. Several Batch Data Stream patterns are provided:


Dispatch

Batch Data Stream ClassLogic to read or write a record of

data to the data store

Data passed as Java object to and from the BDS implementation

Version 8.5 InfoCenter: search on cgrid_btchpttrnover

Supplied "patterns" for:● Read and write using JDBC● Read and write JDBC and stored procedures● Read and write byte data from file● Read and write text file● Read and write to z/OS data set● Read and write using JPA

Options:● Take existing pattern and

customize to your needs● Create your own custom pattern● Not use BDS but rather code data

record read and write handling as part of batch job step



Skip Record Processing

Multi-step jobs …

Provides a container-managed way of tolerating data read or write errors so the job itself may continue on. Information about data errors may be logged.

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record


Batch Container

Batch Data Stream Framework Code

Exception!

Exception!

xJCL where skip record policy declared

xJCL tells BDSF:

● How many data read or write exceptions to consume

● What exceptions to consider for skiprecord processing

● Alternatively, what exceptions to exclude from skip-record processing

Objective is to provide mechanism to survive the odd data exception rather than stop the whole job

The BDSF implementation code consumes the exception rather than surfacing it up to the batch step code

A "listener" may be called to log information about the data record where exception occurredMore on this coming up

When the skip limit is reached then further exceptions are exposed up to the batch step implementationWhere it might result in job going into a restartable state. Normal checkpoint restart processing would occur.

Technical Flyer at: http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101783

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101783



High-Level: Multi-Step Jobs

Retry step …

A job may consist of multiple steps. The job step execution for each is what was shown earlier. Job steps are executed in sequence according to xJCL:


Dispatch

Job Step Class

Job Step Class

Job Step Class

<job ... <job-step ... > </job-step>

<job-step ... > </job-step> : <job-step ... > </job-step></job>

xJCL

STEP_COMPLETE

STEP_COMPLETE

STEP_COMPLETE

Job may consist of one to many stepsMay be mixed CI, TX, Parallel

Steps are executed in sequential order according to xJCL

Supplied results algorithm allows for passing step results to next step

Conditional step processing supported

The construction of WebSphere Java Batch jobs begins to look very much like traditional batch with "Jobs" and "Steps". This is the one of

the benefits of a container-managed Java batch environment.



Retry Step Processing

Batch Listeners …

Provides a way of retrying a job step in the event of an exception thrown. If successful on retry then the job continues and your processing completes.

Job Step● Java class● Input / Output● Other Properties


Batch Job

xJCL tells Container:● How many step retries may be attempted● What exceptions to consider for retry-step

processing● Alternatively, what exceptions to exclude

from retry-step processing● Whether to process a delay before

attempting a retry of the step

If exception, retry step n

times

This is at a higher level from Skip Record. This is at the "invoke batch step" level. This provides a way to retry the step for exceptions.

Batch container falls back to last good checkpoint and restarts from there

A "retry-step listener" may be called so you can perform custom action upon retry-step processingMore on "batch listeners" coming up

When limit reached, then job goes into restartable state. Normal restart from checkpoint processing.


Provides a way to survive unexpected transient

exceptions through step retry




Batch "Listeners"

Parallel Job Manager …

These are callout points where your custom "listener" code will be called when key events occur. The callouts are managed by the batch container:



Batch Job

Job Listener

Retry Step Listener

Skip Record Listener

Job Listener● Callouts occur:

Start of the job; Start of each step; End of each step; End of job● Register your code to container with property in xJCL● Use this to perform any special setup or cleanup actions at those

points in the lifecycle of a batch job

Retry-Step Listener● Callouts occur:

When the exception is thrown; When the retry is attempted● Register your listener with code in application createJobStep()

method● Use this to take action at these points, such as logging

information about the exception and the point in the processing where it occurred

Skip-Record Listener● Callouts occur:

On skipped read or skipped write operation● Register your listener with code in application createJobStep()

method● Use this to take action at these points, such as logging

information about the exception and the record skipped





Parallel Job Manager

COBOL Container …

This provides a mechanism for partitioning batch activity into parallel jobs based on ranges of data you designate:

Job Dispatching

Function

Batch Endpoint

PJM

Parameterizer

Batch Appl

Batch Endpoint

Batch Appl

Batch Endpoint

Batch Appl

Top-Job xJCL

Sub-Job xJCL

Top-job and job number

Sub-jobs ... top-job number and each sub-job number

State of each job

<job ...<run instances="multiple" jvm="multiple"> <prop name="parallel.jobcount" value="3" /> </props></run> :

Job step seen as PJM eligible based on "multiple" specified in xJCL

PJM function receives "top job" xJCL and invokes Parameterizer to perform your custom data range segmentation

Job Dispatcher then dispatches "sub jobs" to eligible endpoints for execution. Each is passed the data select parameters generated by parameterizer

Job Dispatcher coordinates execution of top job and sub jobs as logical entity

Not all jobs are well suited for PJM, but those that are may benefit greatly

Parallel either in multiple endpoints, on multiple threads, or both.



Java Batch Integration with COBOL

Enterprise Schedulers …

This is similar in concept to the z/OS Batch Runtime, but with some differences. The fundamental objectives is the same: low overhead integration Java and COBOL:


Batch Container

COBOL Container JNI Code

COBOL Container

COBOL Module

PDSE or USS Directory

WAS z/OS Servant Region Address Space

STEPLIB or LIBPATH

Call Stub Generator

IBM RAD

Call Stubs

COBOL Container manifests as separate LE Enclave inside the WAS z/OS servant address spaceKeeps COBOL/Java separate, so environments stay clean

The JNI code for Java/COBOL exchanges providedKey piece of plumbing provided, which relieves you of writing complex JNI

Call Stub Generator builds Java call stubs and data bindingsBased on what's in COBOL source

Provides a very efficient means of calling COBOL modules as part of a Java Batch solution. The JNI code and Call Stub Generator make using this solution relatively simple.

Share JDBC T2 with TXSimilar to what Batch Runtime provides



Integration with Enterprise Schedulers

WLM Classification …

One of the interfaces on the Job Dispatcher functions is MDB. This allows a client program to interact with Java Batch and integrate that system with Ent. Schedulers:

Dispatcher Function

MDB Interface

Output

Input

Enterprise Scheduler Function

"WSGRID"Utility

WSGRID JCL

JES SpoolOr elsewhere if JCL defines other than spool

Submit

Exact same Java Batch processing as before

● Messaging provider either MQ or SIBus component of WAS● WSGRID submits job request to dispatcher● Job output from WebSphere Java Batch flows back to WSGRID● WSGRID stays active for duration of Java Batch job

Enterprise Scheduler sees "batch job" as the WSGRID invocation. That stays up for life of Java Batch job. Job output from WebSphere Java batch flows back to WSGRID. Return code from Java Batch flows back to WGRID.



WLM Classification

SMF 120.9 …

The Dispatcher Function on z/OS has the ability to assign a WLM Transaction Class value to a submitted job. That allows mapping to Service Class and Reporting Class:

Dispatcher Function

xJCL

Control Region

Servant Region(s)

Batch Application

WLM

WLM

Classify Work

DispatchSubmit

If (rule) then TC=tclass● Job class from xJCL● Jobname● Application Name● Submitter ID● other criteria

TCLASS

Assigned Service Class and/or Reporting Class

Two use-cases for this:

Assign separate WLM Service Classes

Have jobs dispatched into separate servant regions where WLM can manage to the goal

Assign separate WLM Report Classes

This allows categories of Java Batch jobs to have system resource information accumulated under the report class for RMF reporting



Job Usage Information -- SMF 120.9 on z/OS

Positioning …

In addition to capturing job statistics in a relational table (all platforms), on z/OS the SMF facility can be used to capture batch job information.

Job Number

Submitter ID

Total CPU used

Job state

Node/Server

Job start time

Last update time

Accounting string

Job Number

Submitter ID

Total CPU used

GP Time used

Job state

Node/Server

Job start time

Last update time

Accounting string

All Platforms z/OS SMF 120.9

The information provided is very similar between "all platforms" and "z/OS SMF 120.9"

SMF 120.9 augments CPU information with GP usage. You can derive offload: Total - GP = Offload

If z/OS SMF is part of other job accounting and chargeback purposes, then SMF 120.9 for WebSphere Java Batch is simply part of that.

An example of taking advantage of the platform services



Positioning WebSphere Java Batch

Summary …


JVM ModelBatch Runtime WebSphere Java Batch

COBOL Invocation Both WaysJava invoke COBOL or COBOL

invoke Java; z/OS 2.1 adds PL/I as wel. You write the JNI glue code

One WayJava invoke COBOL but not

the other way around. JNI glue code supplied with runtime.

Bit-mode 31-bitExpectation is it will be

used with COBOL

31-bit or 64-bit31-bit if COBOL container;

64-bit possible if not

JVM LauncherImplies JVM instantiation and tear-

down for each invocation

Long Running JVMBuilt on WAS server model where

servers stay up for as long as needed

Data Access Full Java EEAny data access protocol supported by Java EE WAS itself (JDBC, JCA,

JMS, WOLA, etc.)

JDBC and tVSAMIn z/OS 2.1 support for JDBC T2

and transactional VSAM.

Batch Services Full Java BatchEverything Batch Runtime has

plus Dispatcher and PJM

Subset of Java BatchIn z/OS 1.13 TX services; in z/OS 2.1 very close to full Java Batch, but no PJM

Programming Model WebSphere Java BatchFull Java Batch programming

model. Ability to wrapper Java main(). Full Java EE runtime.

WebSphere Java BatchIn z/OS 2.1 the support is based on Java

Batch. Difference: no PJM. Also, only JDBC T2 and tVSAM, no other Java EE access.



Section Summary

CICS, DB2 …

A very broad summary of the WebSphere Java Batch solution:

WebSphere Java Batch represents a full-function Java Batch framework and runtime model

Much more batch function than BPXBATCH or JZOS

It is built on WebSphere Application ServerWhich means all the functional value statements of WAS accrue to the Java Batch environment. On z/OS, all the "Why WAS z/OS" value statements accrue as well.

Integration with Enterprise SchedulersIt provides a messaging-based interface to the Dispatcher function so Enterprise

Scheduler functions can treat WebSphere Java Batch as any other batch job. This allows WebSphere Java Batch to be part of larger orchestrated batch flows.



Java Batch in CICS, DB2Exploring how Java Batch is implemented in those key data systems



WebSphere Java Batch and Access to CICS

Feature Pack for Modern Batch …

Imagine a WebSphere Java Batch application needing access to CICS for data required to complete the batch job. How can that CICS access be accomplished?


EJB Container

Web Container

Batch Container


CICS Program

CICS Region

Dispatcher FunctionxJCLxJCL


What options exist for this interaction?

Web ServicesUbiquitous, open-standard protocol that spans platforms

Overhead per invocation somewhat high, particularly if transaction required. Not a good solution if latency per invocation a concern.

CTG JCAProven WAS-to-CICS interaction pattern.

WAS-to-CICS global TX

If EXCI, then 32K limit

If IPIC, then TCP stack code path is invoked

WOLAHighly efficient WAS-to-CICS interaction pattern

WAS-to-CICS global TX

Same LPAR required



CICS TS Feature Pack for Modern Batch V1.0

Programming Model …

This allows a CICS region to become an job endpoint for a WebSphere Java Batch dispatching server. Puts batch logic much closer to the CICS data:



FP Modern Batch 1.0

Java Batch Application CICS

Program

JCICS Interface and Services

CICS Region TS4.2 or TS5.1

XML file in CICS Feature Pack setup lets that batch container communicate with the WAS Dispatcher Function host and port. That lets the Dispatcher know the

endpoint is present and what batch applications are deployed.

The application in the CICS batch container has a programming model very similar to that used by WebSphere Java Batch. Benefit is access to JCICS services and

very close proximity to the CICS program the batch application requires.

The batch job management is maintained in WebSphere Java Batch. The dispatching may take place to either WAS endpoints or CICS endpoints or both.

WAS V8.5 or aboveAny Platform

Communicates availability and applications deployed in container

xJCL submitted with application in CICS container gets dispatched

xJCL submitted with application in WAS container gets dispatched



Programming Model for CICS Modern Batch FP

Positioning …

The programming model is based on the WebSphere Java Batch model. There are a few restrictions plus some added functions:

Essential Programming ModelEverything discussed in the earlier section on WebSphere Java Batch applies to the CICS Modern Batch FP as well, except Java EE functions and those things noted below.

Some Batch Data Stream Patterns Not SupportedThe following patterns of WebSphere Java Batch are not supported in the CICS Modern Batch Feature Pack environment:

JDBCReader and JDBCWriterLocalJDBCReader and LocalJDBCWriter

See below ... Feature Pack provides its own JDBC patterns

Additional Batch Data Stream Patterns ProvidedThe CICS Modern Batch Feature Pack adds the following BDS patterns:

VsamKsdsReaderImpl -- read from a VSAM KSDS as input data to the job.

VsamKsdsUpdaterImpl -- update in a VSAM KSDS file.

CICSJDBCReader -- retrieve data from a database using a Java Database Connectivity (JDBC) connection.

CICSJDBCWriter -- write data to a database using a Java Database Connectivity (JDBC) connection.

In addition, the CICS Modern Batch Feature Pack provides access to the full set of JCICS APIs



Positioning CICS Modern Batch Feature Pack

DB2 Stored Procedures …


WebSphere Java Batch Modern Batch FPJVM Model Long Running JVM

Built on WAS server model where servers stay up for as long as needed

Long Running JVMBuilt on CICS JVM server model; JVM

stays up for as long as needed

Programming Model WebSphere Modern BatchCI or Transactional Batch

WebSphere Modern BatchCI or Transactional Batch

Multi-Step YesMulti-step jobs supported

YesMulti-step jobs supported

Job Dispatching Job Dispatching FunctionJob dispatching is handled

through the WebSphere Java Batch "scheduler" interface

Job Dispatching FunctionJob dispatching is handled

through the WebSphere Java Batch "scheduler" interface

Parallel Job Manager YesSupports WebSphere Java Batch

PJM interfaces and operation

YesSupports WebSphere Java Batch

PJM interfaces and operation

Enterprise Schedulers YesIntegration through WSGRID and

MDB interfaces to dispatcher function of WebSphere Java Batch

YesIntegration through WSGRID and

MDB interfaces to dispatcher function of WebSphere Java Batch



High Level: DB2 Stored Procedures Written in Java

Summary …

Stored procedures allow for compact and efficient SQL calls from Java. Writing them in Java allows the execution of the stored procedure to be offloadable:

DB2 for z/OSV7 or above


EJB Container

Web Container

Batch Container

Java Batch Applications

z/OS Operating System

System z HW Platform

Java source for DB2 Stored Procedure

Compiled Java classDB2 Stored Procedure

WLMWLM

StoredProcedure

Address spaces established by WLM to run JVM and Stored Procedures

NUMTCB in proc determines how many JVMs per address space; JVMs single threaded and re-used

CREATE PROCEDURESQLAPPL ENVJCL Proc

Reference: http://www.redbooks.ibm.com/abstracts/sg247604.html

Details behind all this, obviously ... key point is DB2 supports Java as a language for Stored Procedures (User Defined Functions as

well) and those may be called by batch processes

This shows WebSphere Java Batch, but any batch environment capable of calling SP in DB2

may exercise Java-based Stored Procedures



Section Summary

Decision Management …

A very broad summary of the CICS and DB2 section:

The CICS Modern Batch Feature Pack extends WebSphere Java Batch management/execution realm

It allows CICS to participate as a WebSphere Java Batch endpoint server.

A CICS endpoint puts batch logic closer to CICSWhat work should run in the CICS endpoint is a function of the batch job and the

nature / intensity of its CICS interactions

DB2 Stored Procedures and JavaNot really "batch" in our normal sense, but it can part of a larger batch process.

Stored procedures can make batch execution more efficient, and Java stored procedures allow for further exploitation of Java on z/OS



Batch and Decision ManagementIntegrating Java Batch with WODM



Batch Step Decision Logic -- In Step or Outside Step?

Operational Decision Manager …

Imagine a batch job that reads data records and acts upon those records based on a set of decisions (or rules). Where would you prefer those rules be?

Job

Step

Read

Process

Write

Rules

Step

End

Decision logic written within the batch step

Job

Step

Read

Process

Write

Step

End

Rules Execution

Server

Call

Result

Rules Designer Tooling

Rules Repository

Rules

All else equal, it provides greater flexibility of design,

change and execution to externalize the rules



IBM Operational Decision Manager

RES Options …

Provides real-time handling of business rule execution and situational awareness via correlation of business events received:

Decision Model:Given a snapshot view of data, determines best course of action at a specific point in a process or application

Main Purpose:To automate a decision based on a combination of factors such as business policies, regulations, and best practices.

Example:If the customer is not primary card holder and age is less than 21 then freeze account and notify primary card holder

Time-Based Pattern Detection:Correlating events as data is in motion

Main Purpose:To determine what of interest is transpiring and coordinate one or more responses by other systems or generate alerts to people

Example:If more than 2 customer withdraws in an ATM are done in the same day and the 2 ATMs are from 2 foreign countries then Investigate possible fraud. Reduce cash redraw max amount to 100$

Business Rules Business Events



High Level: Rules Execution Server Options

JSR 352 …

The Rules Execution Server (RES) has a few means of placement in z/OS architecture:

zRES

zRES

RES on WAS for z

COBOL

Java

COBOL

IMS

COBOL

Batch

CICS Region

WebSphere Application Server for z/OS

Started Task

The Rules Execution Server may run either in a CICS region, as a

standalone started task, or inside the Java EE runtime of WAS z/OS

Our focus is on how a Java batch application in WebSphere Java Batch may access the RES, and it's by running it in WAS as well.

Invocation patterns on WAS include EJB and remote EJB.

Depending on proximity of Java Batch to RES, one of those two

would be optimalOther interfaces: Web Services and JMS/MDB

WO

LA



Open Standard Java BatchA look at the JSR 352 specification



JSR 352 - "Batch Applications for the Java Platform"

Generic Diagram …

The V1.0 final release is dated April 18th of 2013. IBM served as specification lead on this JSR. It is available for download from the web at the following URL:

http://jcp.org/aboutJava/communityprocess/final/jsr352/index.html

The document has a very nice section titled "Domain Language of Batch" which provides

an overview of key concepts and key

terminologyAs well as detailed sections on the specification interfaces and other details of the specification

If interested in this JSR, download and review is encouraged



Generic Diagram for Domain Language of JSR 352

Comparison …

The diagram is generic but useful in that it provides a few key conceptual building blocks on which the rest of the specification relies:

JobOperator Job Step

ItemReader

ItemProcessor

ItemWriter

JobRepository

1 * 1

1

1

1

A job has one to many steps

Retrieval of input, one item at a time

Business processing for an item

Output of the step -- item or chunk

Interface to manage aspects of job processing

Holds information about jobs currently running and jobs that have run in the past

Concepts here are applicable to batch of any flavor -- historic COBOL, WebSphere Java Batch, JSR 352.



JSR 352 Comparison to WebSphere Java Batch

Possible Integration …

A good deal of functional similarity with a few points where each has differential:

From Chris Vignola's IMPACT 2013 Presentation, TAW-1443



Theoretical JSR 352/WebSphere Batch Integration

References …

A topology picture showing both models working cooperatively:

From Chris Vignola's IMPACT 2013 Presentation, TAW-1443

Common Batch Container

WebSphere Java Batch Applications



JSR 352 Java Batch Applications

JSLJSL

JSR 352 JSL

Common Job Repository

Common Job Logging

This implies no promise of future delivery ... that said, the model shown above is well within the possible given the nature of JSR 352 and the

similarities with the WebSphere Java Batch model.

WebSphere Java Batch as well as JSR 352

JobOperator function



ReferencesA guide to reference information for further details



References

http://www.ibm.com/developerworks/systems/library/es-java-batchz.html

JZOS vs. BPXBATCH

http://www.ibmsystemsmag.com/mainframe/trends/whatsnew/java_compiler/

SDK 7 features with EC12

http://www.ibm.com/systems/z/os/zos/tools/java/products/jzos/overview.htmlJZOS

z/OS 2.1 announcement with information on Batch Runtime improvements

http://www.ibm.com/common/ssi/rep_ca/3/897/ENUS213-013/ENUS213-013.PDF


http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101783

CICS Transaction Server Feature Pack for Modern Batch V1.0

http://www-01.ibm.com/software/htp/cics/batch/

ibm system z and z/os java batch solutions...© 2013 ibm corporation ibm americas advanced technical...

Documents