batch processing - a&bp cc

Batch ProcessingORA/J

Agenda

▪

▪

▪

▪

▪

Batch principles

Characteristics

▪

▪

▪

▪

Batch processing ≠ Scheduling

Batch processing Scheduling

Process large volumes of data Scheduling tasks

eg. Spring Batch, JEE eg. Quartz, Obsidian

Things to consider

▪

▪

▪

▪

▪▪

Essential building blocks

Categorize based on responsibility

▪

▪

▪

▪

▪

▪

Categorize based on input source

▪

▪

▪

Guidelines

▪

▪

▪

▪

▪

Transaction Management

ACID principle

▪

▪

▪

▪

Transactions are handled at the step level

▪

▪

Transaction management in steps (tasklets)

Step 1

Step 2

Step 3

Transaction

Transaction

Transaction x nn

Example: chunk-oriented step

<step id="importProductsStep"> <tasklet> <chunk reader="reader" writer="writer" commit-interval="100" /> <transaction-attributes isolation="READ_UNCOMMITTED" /> </tasklet> </step>

Transaction

Transaction management in listeners

▪

▪

▪

▪ ChunkListener

beforeChunk afterChunk

Transactional reader

<step id="importProductsStep"> <tasklet> <chunk reader="reader" writer="writer" commit-interval="100" reader-transactional-queue="true"/> <transaction-attributes isolation="READ_UNCOMMITTED" /> </tasklet></step>

Transaction management patterns

▪

Transaction management patterns

▪

Transaction management patterns

▪ …

▪

▪

Global transactions implementation

Drawbacks to global transactions

▪

▪▪

▪

▪

▪▪

Shared resource transaction pattern

▪

Best effort pattern with JMS

What can go wrong with JMS?

▪

▪

Best effort pattern with JMS

comm

it

star

t

com

mit

start

Chunk localtransaction

JMS localtransaction

timeline

Commits are synchronized

<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate"> <property name="connectionFactory" ref="connectionFactory" /> <property name="defaultDestination" ref="orderQueue" /> <property name="receiveTimeout" value="100" /> <property name="sessionTransacted" value="true" /></bean>

JMS failure will result in a duplicate message

comm

it

star

t

com

mit

start

Chunk localtransaction

JMS localtransaction

timeline

Commits are synchronized

failed

Handling duplicate messages with manual detection

▪

▪

Handling duplicate messages with manual detection

Handling duplicate messages with idempotency

Conclusions

▪

▪

▪

▪

Execution flows

Execution flows

▪

▪

Standard flow

Conditional flow - example

Conditional flow

Conditional flow

▪

▪

▪

ExitStatus vs BatchStatus

▪

…

▪

▪

▪

Custom Exitstatus

Custom Batchstatus

▪

▪▪▪

▪

▪

Custom Batchstatus

Parallel execution

▪

▪

Parallel execution - types

▪

▪

▪

Parallel execution - Multithreaded

▪

▪

▪

Parallel execution - Remote chunking

▪

▪

▪

▪

▪

▪

Parallel execution - Step partitioning

▪

▪

▪

▪

Parallel execution - Step partitioning

Parallel execution - Step partitioning example

Demo: Speeder problem

Introduction

▪

▪

▪

Introduction

▪

▪

Introduction

▪

▪▪▪▪▪

Generating input

▪

▪

▪

Reading the input

Reading the input

▪

▪▪

▪

▪

Processing the data

Processing the lines we just read

▪

▪ KmPerHourCalculator

▪ null

Writing the data

Writing the Fines

▪

▪ BeanWrapperFieldExtractor

DelimitedLineAggregator

Adding it all together

▪

▪ BatchConfiguration

▪▪

▪

▪ Step

▪

▪

Listening in

●

●

●

○○○

●

Gathering statistics

▪

▪

▪ StepExecutionContext

JobExecutionContext

Gathering Statistics

Spring Batch and JEE

JSR-352

▪

▪

▪

▪

JSR-352 Enterprise

▪

▪

JSR-352 spring

▪

▪

▪

org.springframework.batch.core.

SkipListener#onSkipInWrite(S item, Throwable t)

javax.batch.api.chunk.listener.

SkipWriteListener#onSkipWriteItem(List<Object> items,

Exception ex)

JSR-352 Spring vs. JEE

Questions?