file processing - process execution solution

31
PROCESS EXECUTION AUTOMATICS, INDIVIDUALS AND MASSIVES Abimael Desales López 22/10/2015

Upload: abimael-desales-lopez

Post on 16-Apr-2017

221 views

Category:

Software


0 download

TRANSCRIPT

Page 1: File Processing - Process Execution Solution

PROCESS EXECUTION

AUTOMATICS, INDIVIDUALS AND MASSIVES

Abimael Desales López 22/10/2015

Page 2: File Processing - Process Execution Solution

Processes

Segment

bootstrap

Individual

Massive

Automatic

Individual

Massive

Automatic

Pro

cess

su

bty

pe

s

Page 3: File Processing - Process Execution Solution

Automatic Process

Automatic

Ve

rify

pre

con

dit

ion

s

Quartz

Controller Executor

Scheduler

Ge

t p

aram

ete

rs

Ve

rify

Mat

rixc

oex

iste

nc

Exe

cute

Bu

sin

ess

Ru

les

Cal

cula

tio

n o

f t

hre

ads

Thro

wn

Th

read

s

Mak

e c

alcu

lati

on

s

Wri

te o

utp

ut

file

s

Du

mp

to

dat

abas

e

DB

DB

Page 4: File Processing - Process Execution Solution

Automatic processing it will carry out through a component whosemission is verify execution times by the means of scheduler that it willhave maped the time and frecuency of each process in data base, andfurther it will have the ability to build the expression that feed toquartz to make the scheduling of task being performed.

It plans split the automatic processing in two coarse components thatit will execute each one their corresponding function, the (fisrt)Controller it will verify every input of the process and it will decide if itgoing to execute, fulfilling its pre-conditions, defining the commitfrecuency and number of threads in parallel to execute. The (second)Executor is responsible to execute the process business logic.

Automatic Process

Page 5: File Processing - Process Execution Solution

Individual Processing

Process

Bootstrap

Individual

Sub

typ

es

of

Pro

cess

F6s f5ds9f f9sf55 f6sd5f f6sd5lm ñlkF5d f5d6xf d5f6d 5fd6s f6ds5f6 wa

Result

DB

Get

par

amet

ers

Ve

rifi

yC

on

vive

nce

Mat

rix

Exe

cute

Bu

sin

ess

Ru

les

Mak

e C

alcu

lati

on

s

Typing

Processing

DB

Page 6: File Processing - Process Execution Solution

In the individual processing typing the data in screen and accepting it;to begin the processing the system it will create a ticket, will assign it aticket number to the request (if and only if the validation rules havebeen successes and the required precondition is ok), it will get theprocess parameters, if applies their execution this will be carried out,in other way the ticket is automaticly canceled. The business rules willbe get and applyed in predefined order for the process. If allprocessing was executed successfully and the business rules are okthen it begin the dump of information to database, inserting and/orupdating registries in the respective tables of the process, and finallyupdating the ticket status to PROCESSED.

Individual Processing

Page 7: File Processing - Process Execution Solution

Massive Processing

Segment

Bootstrap

Massive

Sub

typ

es

of

Pro

cess

es

CAMEL

TXT

XLS

CSV

F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65dDsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f585fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf26df5s6df6ds5f5sd6f5s65f6sd5f65ds6f65ds56sd5f6f5d6g5df5gdf6df6h5gfh5Df65f5d65gd6fg56df5gdf6gdf5gd6fgf5dgffd6g5df6g5df5gdf5g651df5g6f5d

F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65dDsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f585fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf26df5s6df6ds5f5sd6f5s65f6sd5f65ds6f65ds56sd5f6f5d6g5df5gdf6df6h5gfh5Df65f5d65gd6fg56df5gdf6gdf5gd6fgf5dgffd6g5df6g5df5gdf5g65df5g62f5d

F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65dDsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f585fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf26df5s6df6ds5f5sd6f5s65f6sd5f65ds6f65ds56sd5f6f5d6g5df5gdf6df6h5gfh5Df65f5d65gd6fg56df5gdf6gdf5gd6fgf5dgffd6g5df6g5df5gdf51g65df5g6f5d

OK File

FAILED File

OK File

FAILED File

OK File

FAILED File

CAMEL

Result of Processing

DB

Processing

Export

Export

Export

Controller Executor

Page 8: File Processing - Process Execution Solution

Massive Processing

Confirm

F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65dDsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f585fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf26df5s6df6ds5f5sd6f5s65f6sd5f65ds6f65ds56sd5f6f5d6g5df5gdf6df6h5gfh5Df65f5d65gd6fg56df5gdf6gdf5gd6fgf5dgffd6g5df6g5df5gdf5g651df5g6f5d

OK_File

CAMEL

Processing

Controller Executor

Cal

cula

tio

n o

f th

read

s

Thro

wn

Th

read

s

Def

ine

co

mm

its

Get

th

e F

ile

Cre

ate

gen

eri

c la

you

tDB

Thread 2Thread 3

Thread 4

Thread n-1

Thread 5

Thread n

Thread 1

Commit x Thread and saved of processing data tokeep the trace to reprocess the file in case offailure

Updating to ARCH_PROCESSED

Here it will execute final insert

Page 9: File Processing - Process Execution Solution

Massive Processing

In the files massive processing highlight three generales steps:

• File load and format recognition.

• File processing, upon their format, and dump in database thesummary of the process.

• Dump in database of each file records (or lines) processed.

To carry out the first two steps it is intended to exploit the features ofthe framework Apache Camel, and apply the procedure made in theproof of concept.

Page 10: File Processing - Process Execution Solution

The tasks that will be carry out through Camel in the first stage of themassive processing are below:

• Routing with dependence of file format

• Get the file registries for the processing

• Validation of structure of the file

• Locate the suitable place to apply the business rules.

• Depending on the result, on the rules/validations enforcementdecide wheter the registry will be write in a OK file or in a FAILED file.

• Routing the outcoming files to a specified route.

Massive Processing

Page 11: File Processing - Process Execution Solution

The task to be carry out by Camel in the second phase of the massiveprocessing (since that is already integrated the OK file) will be thefollowing:

• Obtaining the OK file

• Receiving as a parameter the number of threads to create.

• Do the parcial Commits towards database by each thread.

• Saved trail processed and not processed blocks

• Positioning in the appropiate place in the file to run only a block ofthe same, or no secuential blocks as effect the start of processing offailed registries.

• Writing to database the summary of failed records of the file.

• Summary of cuantity by columns when is required.

Massive Processing

Page 12: File Processing - Process Execution Solution

Structure of the Processes Manager

The Process Manager it will be a component composedof another 3 main components, the Scheduler, theController and the Executor.

Page 13: File Processing - Process Execution Solution

13Design Guidelines

10/22/2015

Solution Design Processing

Quartz Job

(Modulo)

Web App Controller ExecutorJob Business Logic

Business Tables

Processes Manager Components Application ComponentsApplication

Java Classes(thread)

Java Classes (Process)

B/L Java Classes

Call

Call

Repository

BusinessApplication DB

Shell Script Specific Tasks

Call

Shell Unix

►Run

Web App

Web AppRestringedAdmin

SuperAdmin

OS

JVM

OS

JVM

Executionengine

Executionengine

OS

Controller Executor

Trámite1

Trámite2

Trámite3

Trámite4

Assign Job IDand register to queue

Get and executeJob

Business DB

Block 1 Block 2 Block 3

Page 14: File Processing - Process Execution Solution

Flow from loading the file

Web Sytem

Up

load

File

Bea

nu

plo

adFi

le(…

)

Upload File toServer

Create FileStructure

Create Ticket

Get the tickets to Process

Camel classmove file toprocessing

path and routeit to rightmethod

ProcessedFileinsert()

Get theShedule

Put the OK and FAILED

files

ProcessedFileService

update()

TicketServiceupdate()

Process registries byCamel (apply

Business Rules)

Up

load

File

Bea

n

Page 15: File Processing - Process Execution Solution

Structure of the Processes Manager

Scheduler Responsabilities:• Schedule the execution times of the process, if are there some

window for modify de execution times, have a funcionality that iscapable to traslate date-time to Quartz expression and back again.

• Create a Job Group by process.

• Start each process to the time of day that is scheduled inautomatic case, or at once in massive e individual cases, this is topass the control flow to the Controller.

Page 16: File Processing - Process Execution Solution

Approach of the Scheduler

We look that a Job is scheduled, which is only responsible to invoke to one Manager, althoughthis could change if necessary invoking something more. Manager is responsible for orchestrating their services and these ones to their corresponding Daos.

Page 17: File Processing - Process Execution Solution

Structure of the Processes Manager

Responsabilities of the Scheduler:• It will be created so many Job classes as different types of services

you have to run, namely, services will can be grouped in a singleJob so that these conform transactional units or database access,each Job must implement the org.quartz.Job interface.

• The package structure will be as follows:

– For the orchestator classescom.axxi.gestor.procesos.planificador

– For the Job classes:com.axxi.gestor.procesos.planificador.job

– For the job launchers (classes with main method) if required:com.axxi.gestor.procesos.planificador.launcher

Page 18: File Processing - Process Execution Solution

Structure of the Processes Manager

package com.axxi.quartz;

import org.quartz.Job;

import org.quartz.JobDataMap;

import org.quartz.JobExecutionContext;

import org.quartz.JobExecutionException;

// Otros imports

public class ControllerJob implements Job {

// Services injected by Spring should be injected by setter injection

private IProcessXManager processXManager;

public void setProcessXManager(IProcessXManager processXManager){

this.processXManager=processXManager;

}

public void execute(JobExecutionContext arg0) throws JobExecutionException {

//Get JobDataMap from the context

JobDataMap dataMap = arg0.getMergedJobDataMap();

processX.orchest(datamap.getWrappedMap());

}

}

Page 19: File Processing - Process Execution Solution

Structure of the Processes Manager

Responsabilities of the Scheduler:• Inside each Job in the execute(…) method it should get the

context’s JobDataMap parameters and set them to the service.JobDataMap dataMap = arg0.getMergedJobDataMap();

Solicitud solicitud = (Solicitud) dataMap.get("solicitud");

ArchivoProcesado archProcesado = (ArchivoProcesado) dataMap.get("archProcesado");

• In the Job class it will be injected the service by Spring throughsetter injection.

• The Manager, Service and Dao clases still work the same way as inthe web project, injecting the instances in the Job, Manager andService classes respectively, by Spring.

Page 20: File Processing - Process Execution Solution

Manager Example

public class TramiteManager implements ITramiteManager {

// Get the instance by Spring

private ISolicitudService solicitudService;

private IArchivoProcesadoService fileProcessedService;

private IVerificadorPrecondicionesService verifyPreconditions ;

//Setters …

public void orchest(Map<String, Object> parameters) {

Solicitud solicitud = (Solicitud) parameters.get("solicitud");

FileProcessed fileProcessed = (FileProcessed) parameters.get(“fileProcessed");

Integer xParameter = (Integer) parametros.get("xParameter");

solicitudService.createSolicitud(solicitud);

fileProcessedService.createFileProcessed(fileProcessed);

verifyPreconditions.verifyPreconditionsAfiliacion(xParameter);

}

}

Page 21: File Processing - Process Execution Solution

Structure of the Processes Manager

Responsabilities of the Scheduler:• It will be created one Scheduler class for massive processing, one

for individual processing and one for automatic processing, eachone must have as minimum one method by each process type.– MassiveScheduler.java

– IndividualScheduler.java

– AutomaticScheduler.java

• Each class must implement one specific interface with therequired methods for the functionality.

Page 22: File Processing - Process Execution Solution

Structure of the Processes Manager

Responsabilities of the Scheduler:• It must create one properties file with the mapping of groups with

the processes name.#Mapping of execution groups with processes

Grupo1=Affiliation

Grupo2=Collection

Grupo3=ABCAdministrative

Grupo4=Unification

Grupo5=Withdraw

Grupo6=ReinvestmentResources

Grupo7=RGPE

Grupo8=Yield

Grupo9=Comission

Grupo10=Loan

Page 23: File Processing - Process Execution Solution

Structure of the Processes Manager

Responsabilities of the Scheduler:• Must have one orchestrator class by each process type (massive,

automatic, individual) by way of pseudo-workflow to execute theflows.

• Each one of the orchestrator class must be within of the nextpackage:

com.xxx.manager.processes.scheduler.launcher

• In order to maintain there should be one service to build arequest and one processed file of each process type since eachprocess has specific values for their request.

Page 24: File Processing - Process Execution Solution

Structure of the Processes Manager

Responsabilities of the Controller:

• Verify the preconditions to start the process.

• Determine if the process goes forward in function of the matrix ofcoexistence.

• Get the parameters to start the process, this is, the layout, and maybesome specific execution parameters, as the ID of the process, etc.

• Give way to the Executor so it get the file and begin their processing,passing it the entire set of parameters obtained.

• When applicable, depending of the process, make the calculations ofthreads, and launching threads.

• Control the threads processing and commits, saving the track of theprocessed registries, when applicable the processing for threads.

• Having ability to restart.

Page 25: File Processing - Process Execution Solution

Structure of the Processes Manager

Responsabilities of the Controller:

For the phase of final processing:

• Getting file to be processed with the corresponding parameters (i.e.enterprise, product, process y subtype), ticket number.

• Invocation to generic layout maker class.

• Determination of the number of commits.

• Calculation of execution threads –if is staticly assgned, this step is notnecessary.

• Launching and control of threads, unexecuted blocks recovery.

• Allow recovery of certain blocks of the file.

Page 26: File Processing - Process Execution Solution

Structure of the Processes Manager

Responsabilities of the Executor:

For the phase of final processing:

• Receive the parameters get in the Controller.

• Start the flow of Camel processing.

• In case of to be interrupted the flow of Camel, send one flag to theController that the process was interrupted in certain point or incertain block that includes since the X registry to the Y registry(may be writing to database), so that this can reprocess the file.

• Notify the conclusión of the processing to the Controller

• Get a generic service that records and updates the request and thetable of processed files.

Page 27: File Processing - Process Execution Solution

Structure of the Processes Manager

Responsabilities of the Executor:

For the phase of final processing:

• Receiv as parameters the generic layout, the ticket file number,the frecuency of commits.

• Invoke to the class of each process so that it executes the insert tothe database.

• Return the control of the flow to the Controller so that thisregister the block of processed information.

• Invoke to the generic service so that it updates the process state.

Page 28: File Processing - Process Execution Solution

Structure of the Processes Manager

Responsabilities of the Executor:

For the phase of final processing:

• It must be created one transactional method for each process toinsert to database the lists of particular layouts corresponding toeach process, each one of this methods must be developed by theowner responsible of that process logic, can’t be a genericfunctionality. The method will receiv one list of Map from theCamel class to get registries. Within this method it will be invokedone method that assemble the generic layouts.

• The method must return a Map holding three data:

– 1.- Position of initial registry

– 2.- Position of final registry

– Flag indicating if it can to do the insert (True or False)

Page 29: File Processing - Process Execution Solution

Structure of the Processes Manager

Responsabilities of the Executor:

For the phase of final processing:

• It must be created one method for the assembly of the genericlayout that will receiv one Map conteining the field ID - value ofthe field, and the last data of the Map will be the number(position) of the registry in the original file to track what is beinginserted, and we can be aware not inserted records to try therework/restart. This method must return one bean with thereceived data of the specific layout, but conteining all possiblegeneric fields.

Page 30: File Processing - Process Execution Solution

Structure of the Processes Manager

Responsabilities of the Executor:

For the phase of final processing:

• The required methods will have the next suggested signs:LayoutGenericoRecaudacion armarLayoutRecaudacion(Map<String, String> layGenerico);

@Transactional

Map <String, Object> insertRecaudacion(List<Map<String, String>> listaLayouts){

LayoutGenericoRecaudacion layGenerico;

for(Map<String, String> map : listaLayouts){

layGenerico = armarLayoutGenerico(map);

//Processing …

}

}

Page 31: File Processing - Process Execution Solution

THANKS

Abimael Desales LópezSoftware ArchitectIbermática México

[email protected]@gmail.com

Feel free to contact me!