batch processing
TRANSCRIPT
Batch ProcessingORA/J
Agenda
▪
▪
▪
▪
▪
Batch principles
Characteristics
▪
▪
▪
▪
Batch processing ≠ Scheduling
Batch processing Scheduling
Process large volumes of data Scheduling tasks
eg. Spring Batch, JEE eg. Quartz, Obsidian
Things to consider
▪
▪
▪
▪
▪▪
Essential building blocks
Categorize based on responsibility
▪
▪
▪
▪
▪
▪
Categorize based on input source
▪
▪
▪
Guidelines
▪
▪
▪
▪
▪
Transaction Management
ACID principle
▪
▪
▪
▪
Transactions are handled at the step level
▪
▪
Transaction management in steps (tasklets)
Step 1
Step 2
Step 3
Transaction
Transaction
Transaction x nn
Example: chunk-oriented step
<step id="importProductsStep"> <tasklet> <chunk reader="reader" writer="writer" commit-interval="100" /> <transaction-attributes isolation="READ_UNCOMMITTED" /> </tasklet> </step>
Transaction
Transaction management in listeners
▪
▪
▪
▪ ChunkListener
beforeChunk afterChunk
Transactional reader
<step id="importProductsStep"> <tasklet> <chunk reader="reader" writer="writer" commit-interval="100" reader-transactional-queue="true"/> <transaction-attributes isolation="READ_UNCOMMITTED" /> </tasklet></step>
Transaction management patterns
▪
Transaction management patterns
▪
Transaction management patterns
▪ …
▪
▪
Global transactions implementation
Drawbacks to global transactions
▪
▪▪
▪
▪
▪▪
Shared resource transaction pattern
▪
Best effort pattern with JMS
What can go wrong with JMS?
▪
▪
Best effort pattern with JMS
comm
it
star
t
com
mit
start
Chunk localtransaction
JMS localtransaction
timeline
Commits are synchronized
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate"> <property name="connectionFactory" ref="connectionFactory" /> <property name="defaultDestination" ref="orderQueue" /> <property name="receiveTimeout" value="100" /> <property name="sessionTransacted" value="true" /></bean>
JMS failure will result in a duplicate message
comm
it
star
t
com
mit
start
Chunk localtransaction
JMS localtransaction
timeline
Commits are synchronized
failed
Handling duplicate messages with manual detection
▪
▪
Handling duplicate messages with manual detection
Handling duplicate messages with idempotency
Conclusions
▪
▪
▪
▪
Execution flows
Execution flows
▪
▪
Standard flow
Conditional flow - example
Conditional flow
Conditional flow
▪
▪
▪
ExitStatus vs BatchStatus
▪
…
▪
▪
▪
Custom Exitstatus
Custom Batchstatus
▪
▪▪▪
▪
▪
Custom Batchstatus
Parallel execution
▪
▪
Parallel execution - types
▪
▪
▪
Parallel execution - Multithreaded
▪
▪
▪
Parallel execution - Remote chunking
▪
▪
▪
▪
▪
▪
Parallel execution - Step partitioning
▪
▪
▪
▪
Parallel execution - Step partitioning
Parallel execution - Step partitioning example
Demo: Speeder problem
Introduction
▪
▪
▪
Introduction
▪
▪
Introduction
▪
▪▪▪▪▪
Generating input
▪
▪
▪
Reading the input
Reading the input
▪
▪▪
▪
▪
Processing the data
Processing the lines we just read
▪
▪ KmPerHourCalculator
▪ null
Writing the data
Writing the Fines
▪
▪ BeanWrapperFieldExtractor
DelimitedLineAggregator
Adding it all together
▪
▪ BatchConfiguration
▪▪
▪
▪ Step
▪
▪
Listening in
●
●
●
○○○
●
Gathering statistics
▪
▪
▪ StepExecutionContext
JobExecutionContext
Gathering Statistics
Spring Batch and JEE
JSR-352
▪
▪
▪
▪
JSR-352 Enterprise
▪
▪
JSR-352 spring
▪
▪
▪
org.springframework.batch.core.
SkipListener#onSkipInWrite(S item, Throwable t)
javax.batch.api.chunk.listener.
SkipWriteListener#onSkipWriteItem(List<Object> items,
Exception ex)
JSR-352 Spring vs. JEE
Questions?