file processing - batch process execution

Download File Processing - Batch Process Execution

Post on 17-Feb-2017

407 views

Category:

Software

0 download

Embed Size (px)

TRANSCRIPT

  • PROCESS EXECUTION

    AUTOMATICS, INDIVIDUALS AND MASSIVES

    Abimael Desales Lpez 22/10/2015

  • Processes

    Segment

    bootstrap

    Individual

    Massive

    Automatic

    Individual

    Massive

    Automatic

    Pro

    cess

    su

    bty

    pe

    s

  • Automatic Process

    Automatic

    Ve

    rify

    pre

    con

    dit

    ion

    s

    Quartz

    Controller Executor

    Scheduler

    Ge

    t p

    aram

    ete

    rs

    Ve

    rify

    Mat

    rixc

    oex

    iste

    nc

    Exe

    cute

    Bu

    sin

    ess

    Ru

    les

    Cal

    cula

    tio

    n o

    f t

    hre

    ads

    Thro

    wn

    Th

    read

    s

    Mak

    e c

    alcu

    lati

    on

    s

    Wri

    te o

    utp

    ut

    file

    s

    Du

    mp

    to

    dat

    abas

    e

    DB

    DB

  • Automatic processing it will carry out through a component whosemission is verify execution times by the means of scheduler that it willhave maped the time and frecuency of each process in data base, andfurther it will have the ability to build the expression that feed toquartz to make the scheduling of task being performed.

    It plans split the automatic processing in two coarse components thatit will execute each one their corresponding function, the (fisrt)Controller it will verify every input of the process and it will decide if itgoing to execute, fulfilling its pre-conditions, defining the commitfrecuency and number of threads in parallel to execute. The (second)Executor is responsible to execute the process business logic.

    Automatic Process

  • Individual Processing

    Process

    Bootstrap

    Individual

    Sub

    typ

    es

    of

    Pro

    cess

    F6s f5ds9f f9sf55 f6sd5f f6sd5lm lkF5d f5d6xf d5f6d 5fd6s f6ds5f6 wa

    Result

    DB

    Get

    par

    amet

    ers

    Ve

    rifi

    yC

    on

    vive

    nce

    Mat

    rix

    Exe

    cute

    Bu

    sin

    ess

    Ru

    les

    Mak

    e C

    alcu

    lati

    on

    s

    Typing

    Processing

    DB

  • In the individual processing typing the data in screen and accepting it;to begin the processing the system it will create a ticket, will assign it aticket number to the request (if and only if the validation rules havebeen successes and the required precondition is ok), it will get theprocess parameters, if applies their execution this will be carried out,in other way the ticket is automaticly canceled. The business rules willbe get and applyed in predefined order for the process. If allprocessing was executed successfully and the business rules are okthen it begin the dump of information to database, inserting and/orupdating registries in the respective tables of the process, and finallyupdating the ticket status to PROCESSED.

    Individual Processing

  • Massive Processing

    Segment

    Bootstrap

    Massive

    Sub

    typ

    es

    of

    Pro

    cess

    es

    CAMEL

    TXT

    XLS

    CSV

    F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65dDsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f585fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf26df5s6df6ds5f5sd6f5s65f6sd5f65ds6f65ds56sd5f6f5d6g5df5gdf6df6h5gfh5Df65f5d65gd6fg56df5gdf6gdf5gd6fgf5dgffd6g5df6g5df5gdf5g651df5g6f5d

    F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65dDsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f585fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf26df5s6df6ds5f5sd6f5s65f6sd5f65ds6f65ds56sd5f6f5d6g5df5gdf6df6h5gfh5Df65f5d65gd6fg56df5gdf6gdf5gd6fgf5dgffd6g5df6g5df5gdf5g65df5g62f5d

    F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65dDsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f585fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf26df5s6df6ds5f5sd6f5s65f6sd5f65ds6f65ds56sd5f6f5d6g5df5gdf6df6h5gfh5Df65f5d65gd6fg56df5gdf6gdf5gd6fgf5dgffd6g5df6g5df5gdf51g65df5g6f5d

    OK File

    FAILED File

    OK File

    FAILED File

    OK File

    FAILED File

    CAMEL

    Result of Processing

    DB

    Processing

    Export

    Export

    Export

    Controller Executor

  • Massive Processing

    Confirm

    F6sdf5ds9fd8f9sf55sdf6sd5f6s5f6sd5F5dsf5d6xf56d5f6ds5fd6s5f6ds5f65dDsfd4sfdsfds56f5ds6f5ds6f5d6f5d6f585fdsf52fd6f2sf2d6s2fd6s2f6ds2f6d2Fds56f5df5s6f2d6s2fd6s2f6ds2f6dsf26df5s6df6ds5f5sd6f5s65f6sd5f65ds6f65ds56sd5f6f5d6g5df5gdf6df6h5gfh5Df65f5d65gd6fg56df5gdf6gdf5gd6fgf5dgffd6g5df6g5df5gdf5g651df5g6f5d

    OK_File

    CAMEL

    Processing

    Controller Executor

    Cal

    cula

    tio

    n o

    f th

    read

    s

    Thro

    wn

    Th

    read

    s

    Def

    ine

    co

    mm

    its

    Get

    th

    e F

    ile

    Cre

    ate

    gen

    eri

    c la

    you

    tDB

    Thread 2Thread 3

    Thread 4

    Thread n-1

    Thread 5

    Thread n

    Thread 1

    Commit x Thread and saved of processing data tokeep the trace to reprocess the file in case offailure

    Updating to ARCH_PROCESSED

    Here it will execute final insert

  • Massive Processing

    In the files massive processing highlight three generales steps:

    File load and format recognition.

    File processing, upon their format, and dump in database thesummary of the process.

    Dump in database of each file records (or lines) processed.

    To carry out the first two steps it is intended to exploit the features ofthe framework Apache Camel, and apply the procedure made in theproof of concept.

  • The tasks that will be carry out through Camel in the first stage of themassive processing are below:

    Routing with dependence of file format

    Get the file registries for the processing

    Validation of structure of the file

    Locate the suitable place to apply the business rules.

    Depending on the result, on the rules/validations enforcementdecide wheter the registry will be write in a OK file or in a FAILED file.

    Routing the outcoming files to a specified route.

    Massive Processing

  • The task to be carry out by Camel in the second phase of the massiveprocessing (since that is already integrated the OK file) will be thefollowing:

    Obtaining the OK file

    Receiving as a parameter the number of threads to create.

    Do the parcial Commits towards database by each thread.

    Saved trail processed and not processed blocks

    Positioning in the appropiate place in the file to run only a block ofthe same, or no secuential blocks as effect the start of processing offailed registries.

    Writing to database the summary of failed records of the file.

    Summary of cuantity by columns when is required.

    Massive Processing

  • Structure of the Processes Manager

    The Process Manager it will be a component composedof another 3 main components, the Scheduler, theController and the Executor.

  • 13Design Guidelines

    10/22/2015

    Solution Design Processing

    Quartz Job

    (Modulo)

    Web App Controller ExecutorJob Business Logic

    Business Tables

    Processes Manager Components Application ComponentsApplication

    Java Classes(thread)

    Java Classes (Process)

    B/L Java Classes

    Call

    Call

    Repository

    BusinessApplication DB

    Shell Script Specific Tasks

    Call

    Shell Unix

    Run

    Web App

    Web AppRestringedAdmin

    SuperAdmin

    OS

    JVM

    OS

    JVM

    Executionengine

    Executionengine

    OS

    Controller Executor

    Trmite1

    Trmite2

    Trmite3

    Trmite4

    Assign Job IDand register to queue

    Get and executeJob

    Business DB

    Block 1 Block 2 Block 3

  • Flow from loading the file

    Web Sytem

    Up

    load

    File

    Bea

    nu

    plo

    adFi

    le(

    )

    Upload File toServer

    Create FileStructure

    Create Ticket

    Get the tickets to Process

    Camel classmove file toprocessing

    path and routeit to rightmethod

    ProcessedFileinsert()

    Get theShedule

    Put the OK and FAILED

    files

    ProcessedFileService

    update()

    TicketServiceupdate()

    Process registries byCamel (apply

    Business Rules)

    Up

    load

    File

    Bea

    n

  • Structure of the Processes Manager

    Scheduler Responsabilities: Schedule the execution times of the process, if are there some

    window for modify de execution times, have a funcionality that iscapable to traslate date-time to Quartz expression and back again.

    Create a Job Group by process.

    Start each process to the time of day that is scheduled inautomatic case, or at once in massive e individual cases, this is topass the control flow to the Controller.

  • Approach of the Scheduler

    We look that a Job is scheduled, which is only responsible to invoke to one Manager, althoughthis could change if necessary invoking something more. Manager is responsible for orchestrating their services and these ones to their corresponding Daos.

  • Structure of the Processes Manager

    Responsabilities of the Scheduler: It will be created so many Job classes as different types of services

    you have to run, namely, services will can be grouped in a singleJob so that these conform transactional units or database access,each Job must implement the org.quartz.Job interface.

    The package structure will be as follows:

    For the orchesta

Recommended

View more >