file processing - process execution solution
Post on 16-Apr-2017
221 Views
Preview:
TRANSCRIPT
PROCESS EXECUTION
AUTOMATICS, INDIVIDUALS AND MASSIVES
Abimael Desales López 22/10/2015
Processes
Segment
bootstrap
Individual
Massive
Automatic
Individual
Massive
Automatic
Pro
cess
su
bty
pe
s
Automatic Process
Automatic
Ve
rify
pre
con
dit
ion
s
Quartz
Controller Executor
Scheduler
Ge
t p
aram
ete
rs
Ve
rify
Mat
rixc
oex
iste
nc
Exe
cute
Bu
sin
ess
Ru
les
Cal
cula
tio
n o
f t
hre
ads
Thro
wn
Th
read
s
Mak
e c
alcu
lati
on
s
Wri
te o
utp
ut
file
s
Du
mp
to
dat
abas
e
DB
DB
Automatic processing it will carry out through a component whosemission is verify execution times by the means of scheduler that it willhave maped the time and frecuency of each process in data base, andfurther it will have the ability to build the expression that feed toquartz to make the scheduling of task being performed.
It plans split the automatic processing in two coarse components thatit will execute each one their corresponding function, the (fisrt)Controller it will verify every input of the process and it will decide if itgoing to execute, fulfilling its pre-conditions, defining the commitfrecuency and number of threads in parallel to execute. The (second)Executor is responsible to execute the process business logic.
Automatic Process
Individual Processing
Process
Bootstrap
Individual
Sub
typ
es
of
Pro
cess
F6s f5ds9f f9sf55 f6sd5f f6sd5lm ñlkF5d f5d6xf d5f6d 5fd6s f6ds5f6 wa
Result
DB
Get
par
amet
ers
Ve
rifi
yC
on
vive
nce
Mat
rix
Exe
cute
Bu
sin
ess
Ru
les
Mak
e C
alcu
lati
on
s
Typing
Processing
DB
In the individual processing typing the data in screen and accepting it;to begin the processing the system it will create a ticket, will assign it aticket number to the request (if and only if the validation rules havebeen successes and the required precondition is ok), it will get theprocess parameters, if applies their execution this will be carried out,in other way the ticket is automaticly canceled. The business rules willbe get and applyed in predefined order for the process. If allprocessing was executed successfully and the business rules are okthen it begin the dump of information to database, inserting and/orupdating registries in the respective tables of the process, and finallyupdating the ticket status to PROCESSED.
Individual Processing
Massive Processing
Segment
Bootstrap
Massive
Sub
typ
es
of
Pro
cess
es
CAMEL
TXT
XLS
CSV
F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65dDsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f585fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf26df5s6df6ds5f5sd6f5s65f6sd5f65ds6f65ds56sd5f6f5d6g5df5gdf6df6h5gfh5Df65f5d65gd6fg56df5gdf6gdf5gd6fgf5dgffd6g5df6g5df5gdf5g651df5g6f5d
F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65dDsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f585fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf26df5s6df6ds5f5sd6f5s65f6sd5f65ds6f65ds56sd5f6f5d6g5df5gdf6df6h5gfh5Df65f5d65gd6fg56df5gdf6gdf5gd6fgf5dgffd6g5df6g5df5gdf5g65df5g62f5d
F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65dDsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f585fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf26df5s6df6ds5f5sd6f5s65f6sd5f65ds6f65ds56sd5f6f5d6g5df5gdf6df6h5gfh5Df65f5d65gd6fg56df5gdf6gdf5gd6fgf5dgffd6g5df6g5df5gdf51g65df5g6f5d
OK File
FAILED File
OK File
FAILED File
OK File
FAILED File
CAMEL
Result of Processing
DB
Processing
Export
Export
Export
Controller Executor
Massive Processing
Confirm
F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65dDsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f585fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf26df5s6df6ds5f5sd6f5s65f6sd5f65ds6f65ds56sd5f6f5d6g5df5gdf6df6h5gfh5Df65f5d65gd6fg56df5gdf6gdf5gd6fgf5dgffd6g5df6g5df5gdf5g651df5g6f5d
OK_File
CAMEL
Processing
Controller Executor
Cal
cula
tio
n o
f th
read
s
Thro
wn
Th
read
s
Def
ine
co
mm
its
Get
th
e F
ile
Cre
ate
gen
eri
c la
you
tDB
Thread 2Thread 3
Thread 4
Thread n-1
Thread 5
Thread n
Thread 1
Commit x Thread and saved of processing data tokeep the trace to reprocess the file in case offailure
Updating to ARCH_PROCESSED
Here it will execute final insert
Massive Processing
In the files massive processing highlight three generales steps:
• File load and format recognition.
• File processing, upon their format, and dump in database thesummary of the process.
• Dump in database of each file records (or lines) processed.
To carry out the first two steps it is intended to exploit the features ofthe framework Apache Camel, and apply the procedure made in theproof of concept.
The tasks that will be carry out through Camel in the first stage of themassive processing are below:
• Routing with dependence of file format
• Get the file registries for the processing
• Validation of structure of the file
• Locate the suitable place to apply the business rules.
• Depending on the result, on the rules/validations enforcementdecide wheter the registry will be write in a OK file or in a FAILED file.
• Routing the outcoming files to a specified route.
Massive Processing
The task to be carry out by Camel in the second phase of the massiveprocessing (since that is already integrated the OK file) will be thefollowing:
• Obtaining the OK file
• Receiving as a parameter the number of threads to create.
• Do the parcial Commits towards database by each thread.
• Saved trail processed and not processed blocks
• Positioning in the appropiate place in the file to run only a block ofthe same, or no secuential blocks as effect the start of processing offailed registries.
• Writing to database the summary of failed records of the file.
• Summary of cuantity by columns when is required.
Massive Processing
Structure of the Processes Manager
The Process Manager it will be a component composedof another 3 main components, the Scheduler, theController and the Executor.
13Design Guidelines
10/22/2015
Solution Design Processing
Quartz Job
(Modulo)
Web App Controller ExecutorJob Business Logic
Business Tables
Processes Manager Components Application ComponentsApplication
Java Classes(thread)
Java Classes (Process)
B/L Java Classes
Call
Call
Repository
BusinessApplication DB
Shell Script Specific Tasks
Call
Shell Unix
►Run
Web App
Web AppRestringedAdmin
SuperAdmin
OS
JVM
OS
JVM
Executionengine
Executionengine
OS
Controller Executor
Trámite1
Trámite2
Trámite3
Trámite4
Assign Job IDand register to queue
Get and executeJob
Business DB
Block 1 Block 2 Block 3
Flow from loading the file
Web Sytem
Up
load
File
Bea
nu
plo
adFi
le(…
)
Upload File toServer
Create FileStructure
Create Ticket
Get the tickets to Process
Camel classmove file toprocessing
path and routeit to rightmethod
ProcessedFileinsert()
Get theShedule
Put the OK and FAILED
files
ProcessedFileService
update()
TicketServiceupdate()
Process registries byCamel (apply
Business Rules)
Up
load
File
Bea
n
Structure of the Processes Manager
Scheduler Responsabilities:• Schedule the execution times of the process, if are there some
window for modify de execution times, have a funcionality that iscapable to traslate date-time to Quartz expression and back again.
• Create a Job Group by process.
• Start each process to the time of day that is scheduled inautomatic case, or at once in massive e individual cases, this is topass the control flow to the Controller.
Approach of the Scheduler
We look that a Job is scheduled, which is only responsible to invoke to one Manager, althoughthis could change if necessary invoking something more. Manager is responsible for orchestrating their services and these ones to their corresponding Daos.
Structure of the Processes Manager
Responsabilities of the Scheduler:• It will be created so many Job classes as different types of services
you have to run, namely, services will can be grouped in a singleJob so that these conform transactional units or database access,each Job must implement the org.quartz.Job interface.
• The package structure will be as follows:
– For the orchestator classescom.axxi.gestor.procesos.planificador
– For the Job classes:com.axxi.gestor.procesos.planificador.job
– For the job launchers (classes with main method) if required:com.axxi.gestor.procesos.planificador.launcher
Structure of the Processes Manager
package com.axxi.quartz;
import org.quartz.Job;
import org.quartz.JobDataMap;
import org.quartz.JobExecutionContext;
import org.quartz.JobExecutionException;
// Otros imports
public class ControllerJob implements Job {
// Services injected by Spring should be injected by setter injection
private IProcessXManager processXManager;
public void setProcessXManager(IProcessXManager processXManager){
this.processXManager=processXManager;
}
public void execute(JobExecutionContext arg0) throws JobExecutionException {
//Get JobDataMap from the context
JobDataMap dataMap = arg0.getMergedJobDataMap();
processX.orchest(datamap.getWrappedMap());
}
}
Structure of the Processes Manager
Responsabilities of the Scheduler:• Inside each Job in the execute(…) method it should get the
context’s JobDataMap parameters and set them to the service.JobDataMap dataMap = arg0.getMergedJobDataMap();
Solicitud solicitud = (Solicitud) dataMap.get("solicitud");
ArchivoProcesado archProcesado = (ArchivoProcesado) dataMap.get("archProcesado");
• In the Job class it will be injected the service by Spring throughsetter injection.
• The Manager, Service and Dao clases still work the same way as inthe web project, injecting the instances in the Job, Manager andService classes respectively, by Spring.
Manager Example
public class TramiteManager implements ITramiteManager {
// Get the instance by Spring
private ISolicitudService solicitudService;
private IArchivoProcesadoService fileProcessedService;
private IVerificadorPrecondicionesService verifyPreconditions ;
//Setters …
public void orchest(Map<String, Object> parameters) {
Solicitud solicitud = (Solicitud) parameters.get("solicitud");
FileProcessed fileProcessed = (FileProcessed) parameters.get(“fileProcessed");
Integer xParameter = (Integer) parametros.get("xParameter");
solicitudService.createSolicitud(solicitud);
fileProcessedService.createFileProcessed(fileProcessed);
verifyPreconditions.verifyPreconditionsAfiliacion(xParameter);
}
}
Structure of the Processes Manager
Responsabilities of the Scheduler:• It will be created one Scheduler class for massive processing, one
for individual processing and one for automatic processing, eachone must have as minimum one method by each process type.– MassiveScheduler.java
– IndividualScheduler.java
– AutomaticScheduler.java
• Each class must implement one specific interface with therequired methods for the functionality.
Structure of the Processes Manager
Responsabilities of the Scheduler:• It must create one properties file with the mapping of groups with
the processes name.#Mapping of execution groups with processes
Grupo1=Affiliation
Grupo2=Collection
Grupo3=ABCAdministrative
Grupo4=Unification
Grupo5=Withdraw
Grupo6=ReinvestmentResources
Grupo7=RGPE
Grupo8=Yield
Grupo9=Comission
Grupo10=Loan
Structure of the Processes Manager
Responsabilities of the Scheduler:• Must have one orchestrator class by each process type (massive,
automatic, individual) by way of pseudo-workflow to execute theflows.
• Each one of the orchestrator class must be within of the nextpackage:
com.xxx.manager.processes.scheduler.launcher
• In order to maintain there should be one service to build arequest and one processed file of each process type since eachprocess has specific values for their request.
Structure of the Processes Manager
Responsabilities of the Controller:
• Verify the preconditions to start the process.
• Determine if the process goes forward in function of the matrix ofcoexistence.
• Get the parameters to start the process, this is, the layout, and maybesome specific execution parameters, as the ID of the process, etc.
• Give way to the Executor so it get the file and begin their processing,passing it the entire set of parameters obtained.
• When applicable, depending of the process, make the calculations ofthreads, and launching threads.
• Control the threads processing and commits, saving the track of theprocessed registries, when applicable the processing for threads.
• Having ability to restart.
Structure of the Processes Manager
Responsabilities of the Controller:
For the phase of final processing:
• Getting file to be processed with the corresponding parameters (i.e.enterprise, product, process y subtype), ticket number.
• Invocation to generic layout maker class.
• Determination of the number of commits.
• Calculation of execution threads –if is staticly assgned, this step is notnecessary.
• Launching and control of threads, unexecuted blocks recovery.
• Allow recovery of certain blocks of the file.
Structure of the Processes Manager
Responsabilities of the Executor:
For the phase of final processing:
• Receive the parameters get in the Controller.
• Start the flow of Camel processing.
• In case of to be interrupted the flow of Camel, send one flag to theController that the process was interrupted in certain point or incertain block that includes since the X registry to the Y registry(may be writing to database), so that this can reprocess the file.
• Notify the conclusión of the processing to the Controller
• Get a generic service that records and updates the request and thetable of processed files.
Structure of the Processes Manager
Responsabilities of the Executor:
For the phase of final processing:
• Receiv as parameters the generic layout, the ticket file number,the frecuency of commits.
• Invoke to the class of each process so that it executes the insert tothe database.
• Return the control of the flow to the Controller so that thisregister the block of processed information.
• Invoke to the generic service so that it updates the process state.
Structure of the Processes Manager
Responsabilities of the Executor:
For the phase of final processing:
• It must be created one transactional method for each process toinsert to database the lists of particular layouts corresponding toeach process, each one of this methods must be developed by theowner responsible of that process logic, can’t be a genericfunctionality. The method will receiv one list of Map from theCamel class to get registries. Within this method it will be invokedone method that assemble the generic layouts.
• The method must return a Map holding three data:
– 1.- Position of initial registry
– 2.- Position of final registry
– Flag indicating if it can to do the insert (True or False)
Structure of the Processes Manager
Responsabilities of the Executor:
For the phase of final processing:
• It must be created one method for the assembly of the genericlayout that will receiv one Map conteining the field ID - value ofthe field, and the last data of the Map will be the number(position) of the registry in the original file to track what is beinginserted, and we can be aware not inserted records to try therework/restart. This method must return one bean with thereceived data of the specific layout, but conteining all possiblegeneric fields.
Structure of the Processes Manager
Responsabilities of the Executor:
For the phase of final processing:
• The required methods will have the next suggested signs:LayoutGenericoRecaudacion armarLayoutRecaudacion(Map<String, String> layGenerico);
@Transactional
Map <String, Object> insertRecaudacion(List<Map<String, String>> listaLayouts){
LayoutGenericoRecaudacion layGenerico;
for(Map<String, String> map : listaLayouts){
layGenerico = armarLayoutGenerico(map);
//Processing …
}
}
THANKS
Abimael Desales LópezSoftware ArchitectIbermática México
a.desales@ibermatica.comIt.adesales@gmail.com
Feel free to contact me!
top related