parallel processing with crm business transactions

2 v1.0

mailto:[email protected]

© SAP Global Delivery 2009

3 v1.0

1. PURPOSE .................................................................................................................................. 4

2. ABSTRACT ................................................................................................................................. 5

3. DESCRIPTION ............................................................................................................................ 6

3.1. PARALLEL PROCESSING DRIVER PROGRAM ........................................................................... 6

3.2. REMOTE ENABLE FUNCTION MODULE .................................................................................... 8

3.3. AN SPECIFIC SCENARIO: UPDATING SERVICE CONTRACTS ...................................................... 9

4. RESULTS ................................................................................................................................. 18

5. CONCLUSIONS ......................................................................................................................... 19

6. SOURCE .................................................................................................................................. 20

4 v1.0

1. Purpose This document brings a detailed knowledge on how to use parallel processing approach to update

CRM Business Transactions.


5 v1.0

2. Abstract Customer need to do a mass update for any Business Transaction. Performance has to be considered as

is one of the key requirements.

Working with large amount of data using a unique local process can lead to very large time for

processing (typically client says when is too expensive). Parallel processing using asynchronous remote

function calls (aRFC) can be the solution.

When updating CRM Business Transaction through interface reports using aRFC there are common steps

which always apply.

Usually the solution consists in following objects:

- Parallel Processing Driver Program: obtains data based on selection screen entries, splits the whole

data into packages, manages parallel processing coordination calling aRFC, show log after entire

processing ends.

- Remote enable Function module: receives GUIDs or IDs and do all necessary logic to get business

transaction updated: typically includes calling function modules CRM_ORDER_READ,

CRM_ORDER_MAINTAIN, CRM_ORDER_SAVE. Besides returns log table for changed transactions.

6 v1.0

3. Description

3.1. Parallel processing driver program

Selection Screen

Apart from necessary fields to data selection from DB (depending purely on the requirement)

user should be able to switch parallel execution on or off, and (if switched on) to configure

details of parallel processing. Making parallelism optional is useful when the workload is small

and parallel processing doesn’t make sense.

“Batch size” is the quantity of Business Transactions which together make a package. Supposing this

field set to 100, and the whole data consisting of 1000 business transaction ID, then data will be split

into 10 packages.

“Max # of jobs” indicates the maximum quantity of parallel processes running simultaneously.

“Server group” is required when calling remote function module for asynchronous processing.

Last two parameters help handling when remote function module results unsuccessful:

“Max. # of Attempt” is the maximum consecutive times that a FM recall can fail before cancelling

program (forcing program cancelation helps avoiding dead-lock).

“Max. wait time per attempt” is how many seconds to wait before recalling a FM after failing.

Data selection

This section is really very simple as there is no difference comparing to a simple report.

Data to be processed first has to be selected from database filtering accordingly to selection screen

parameters.

The important thing here is to select data into an internal table which will be used then to split data into

packages using a LOOP ENDLOOP logic calling remote function call from there.


7 v1.0

Parallel processing logic

After declaring data, the report calls the ‘SPBT_INITIALIZE’ function module. This call lets the program

check that the parallel processing group is valid and that resources are available.

The heart of the report is a WHILE loop in which the function module that is to be processed in parallel is

called (CALL FUNCTION STARTING NEW TASK DESTINATION IN GROUP CALLING

‘FORM_WHEN_TASK_END’ ON END OF TASK).

After call have to check results:

- In sy-subrc is 0 then task was created and FM is currently running.

- If sy-subrc is 1 or 2 (system_failure or communication_failure) then task could not be created and this has to be handled. Here come into play parameters declared in section 3.1 related to error handling. Variable which takes count of consecutive failures is increased by 1 and compared to “Max. # of Attempt”, exiting program in case they are the same.

After trying to call function again, program waits for “Max. wait time per attempt” seconds.

Then should reprocess task with all transactions involved.

- In sy-subrc are 3 this is directly related to lack of free parallel process. In this case the only thing to do is waiting until current processes end so new task can begin processing. For this purpose the same logic that previous case is done.

For exceptions 1 and 2, as these are related directly to server availability additionally could exclude

specific server from group calling function 'SPBT_DO_NOT_USE_SERVER'.

WHILE logic repeats until all transactions were processed.

As part of task management, job must wait until all of the parallel processing tasks have been

completed. To do this, the program uses the WAIT keyword to wait until the number of completed

parallel processing tasks is equal to the number of tasks that were created. Independently of this WAIT;

the callback form ‘FORM_WHEN_TASK_ENDS’ (used in remote function call) is triggered. The callback

form keeps track of the number of completed parallel processing tasks, so that the WAIT condition can

be correctly evaluated.

Additionally, this routine can receive result from remote function execution like a log table.

8 v1.0

3.2. Remote enable function module

Interface

The aim is to receive as less data as possible to avoid expensive communication time. After processing

data further a log table is returned with execution results.

A possible approach would be to receive transaction’ GUIDs necessary to call CRM_ORDER_READ (most

of times this call is a must in order to obtain further data about transaction):

After processing further according to logic (logging all important information in log structure), at the end

of execution log table has to be returned:

Logic

Again, this section is very simple as there are no additional considerations but only related to functional

requirement.

A typical update scenario in CRM involves calling function ‘CRM_ORDER_READ’ obtaining tables

according to requested logic, modify calling ‘CRM_ORDER_MAINTAIN’, saving changes with a call to

‘CRM_ORDER_SAVE’ and finally persisting data with 'BAPI_TRANSACTION_COMMIT'.

Considering that this FM runs on and independent LUW (logical unit of work) then in case that any

exception occurs in function calls it has to be caught and logged so the caller program (here Parallel

processing driver program) can be aware of this and take corrective action (action will depend on

requirement, perhaps doing nothing but reporting to user is enough).


9 v1.0

3.3. An specific scenario: updating service contracts

Requirement introduction

Customer requires a report in order to mass update all service contracts entered in screen adding a new

item line only for those contracts which pass all validations logic inflicted by user.

Existing more than 100 thousand contracts in the system and being execution time a key factor of

success, using parallel processing is an excellent idea and can lead to the success of our development.

Parallel processing driver program

Selection screen

This selection screen let user decides for which process type, including specific product id at item level,

even specifying contracts id (if blank so include all which apply restricting only by other parameters).

Different options for running basically include a control report (no update to contract is done), and 3

update options each of them involving specific validations.

Last tab includes parallel processing related parameters as explained in section 3.1.

Data selection

This part of the program is required to obtain all the contracts which applied according to filtering

parameters.

Depending on system, this could vary. A valid example where is:

10 v1.0

SELECT DISTINCT guid

FROM crmv_order_items

INTO CORRESPONDING FIELDS OF TABLE it_hdr_guid

WHERE process_type IN s_cnt_ty

AND ordered_prod = p_prd_id.

IF sy-subrc IS NOT INITIAL.

MESSAGE e027(zcrm_error) WITH text-003.

ENDIF.

*crmv_order_items is a view from database.

Is important to select into a table (it_hdr_guid). This will be use later for parallel processing logic using a

while structure.

Parallel processing logic

This section includes the main part of the solution. Consist of 2 routines, one splitting data into tasks and

calling remote function in parallel and other one executing when task ends taking over the execution

result of each one of them.

*&---------------------------------------------------------------------*

*& Form F_PARALEL_PROCESS

*&---------------------------------------------------------------------

FORM f_paralel_process CHANGING pv_error TYPE char1

pv_error_from TYPE i.

* Calling function SPBT_INITIALIZE to initialize the Server Call

* and Checking for preliminary requisite

CALL FUNCTION 'SPBT_INITIALIZE'

EXPORTING

group_name = lv_server_group

* importing

* max_pbt_wps = max_pbt_wps

* free_pbt_wps = free_pbt_wps

EXCEPTIONS

invalid_group_name = 1

internal_error = 2

pbt_env_already_initialized = 3

currently_no_resources_avail = 4

no_pbt_resources_found = 5

cant_init_different_pbt_groups = 6

OTHERS = 7.

* SAP Standard Messages

IF sy-subrc EQ 0. "all is o.k.

ELSE.


11 v1.0

*HANDLE Error

ENDIF.

* No. of contracts to be passed to the aRFC call

DESCRIBE TABLE it_hdr_guid LINES lv_nof_cntracts.

* Incrementing the higher index to define the batch

lv_index_high = lv_index_high + p_batch.

lv_index_low = 1.

lv_run_ok = c_x.

WHILE lv_index_low LE lv_nof_cntracts.

IF lv_run_ok IS NOT INITIAL.

REFRESH lt_hdr_guid_tmp.

* If high index exceed number of contracts adjust to then append correct

lines

IF lv_index_high GT lv_nof_cntracts.

lv_index_high = lv_nof_cntracts.

ENDIF.

APPEND LINES OF it_hdr_guid FROM lv_index_low

TO lv_index_high

TO lt_hdr_guid_tmp.

* Create a unique task name

CLEAR lv_taskname.

* incrementing Counter to get unique task id

lv_counter = lv_counter + 1 . lv_taskname = lv_counter.

SHIFT lv_taskname LEFT DELETING LEADING space.

CONCATENATE c_prefix lv_taskname INTO lv_taskname.

lw_task-taskname = lv_taskname.

lw_task-index_from = lv_index_low.

lw_task-index_to = lv_index_high.

APPEND lw_task TO it_tasks.

* Wait until number of processes is less than maximum entered in Screen

WAIT UNTIL gv_current_jobs LT p_maxjob.

ENDIF.

* aRFC Call

CALL FUNCTION 'ZCRM_ESM_RESET_CONT_ARFC'

STARTING NEW TASK lv_taskname DESTINATION IN GROUP lv_server_group

PERFORMING f_return_info ON END OF TASK

EXPORTING

i_product_guid = v_prod_guid " Product GUID

i_year = p_curr_y " Processing year

12 v1.0

i_mode = l_ex_mode " Execution Mode (Control, Re

set, Add, Init)

i_contract_sel = v_id_sel " Check if contract were spec

ified

TABLES

t_contract_guid = lt_hdr_guid_tmp " Contracts

EXCEPTIONS

system_failure = 1

communication_failure = 2

resource_failure = 3

OTHERS = 4.

CASE sy-subrc .

WHEN 0 .

lv_run_ok = c_x.

* increment no. of jobs currently being Executed by the System

gv_current_jobs = gv_current_jobs + 1.

* assigning the index_high to index_low to increment value

lv_index_low = lv_index_high + 1.

lv_index_high = lv_index_high + p_batch.

CLEAR lv_res_fail.

WHEN 1 OR 2.

"Remove server from further consideration for

"parallel processing tasks in this program.

CALL FUNCTION 'SPBT_GET_PP_DESTINATION' "#EC *

IMPORTING

rfcdest = lv_rfcdest

EXCEPTIONS

OTHERS = 1.

lv_no_use_serv = lv_rfcdest.

"Then remove from list of available servers.

CALL FUNCTION 'SPBT_DO_NOT_USE_SERVER' "#EC *

EXPORTING

server_name = lv_no_use_serv

EXCEPTIONS

invalid_server_name = 1

no_more_resources_left = 2 "No servers left in group.

pbt_env_not_initialized_yet = 3

OTHERS = 4.

IF lv_res_fail LT p_maxres.

WAIT UP TO p_w_res SECONDS.

lv_res_fail = lv_res_fail + 1.

CLEAR lv_run_ok.

ELSE.


13 v1.0

pv_error = c_x.

pv_error_from = lv_index_low.

* In case there are other tasks still running

WAIT UNTIL gv_current_jobs EQ 0.

RETURN.

ENDIF.

WHEN 3.




CLEAR lv_run_ok.

ELSE.

pv_error = c_x.




RETURN.

ENDIF.

WHEN OTHERS.




CLEAR lv_run_ok.

ELSE.

pv_error = c_x.




RETURN.

ENDIF.

ENDCASE.

ENDWHILE.


ENDFORM. " F_PARALEL_PROCESS

*&---------------------------------------------------------------------

*& Form F_RETURN_INFO

*&---------------------------------------------------------------------

FORM f_return_info USING p_taskname TYPE clike.

DATA: lt_log TYPE ztt_cntr_reset_log,

lw_task TYPE ty_tasklog.

RECEIVE RESULTS FROM FUNCTION 'ZCRM_ESM_RESET_CONT_ARFC'

IMPORTING

et_log = lt_log

14 v1.0

EXCEPTIONS

system_failure = 1

communication_failure = 2

OTHERS = 4.

* decrementing the no. of current jobs counter

after the arfc session has been executed.

gv_current_jobs = gv_current_jobs - 1.

IF sy-subrc IS INITIAL.

APPEND LINES OF lt_log TO it_log.

ELSE.

READ TABLE it_tasks WITH KEY taskname = p_taskname

INTO lw_task.

APPEND LINES OF it_hdr_guid

FROM lw_task-index_from

TO lw_task-index_to

TO it_log_err.

ENDIF.

* Delete from tasks table

DELETE it_tasks WHERE taskname EQ p_taskname.

ENDFORM. " F_RETURN_INFO

Finally, the log table is shown for each contract previously processed.


15 v1.0

Remote enable function module

Interface

Necessary input to do corresponding validations is imported:

Always need to have contracts GUID to process. These are the contracts contained in task currently

executing:

Once function finishes processing result is returned to main program:

Logic

Function logic is the continuation of the main program. At this time contract GUID is available and

further processing is intended to be done here. In this way heaviest time-consuming logic is performed

for a limited number of contracts (in this example 100 per task) and running at the same time (in this

example 3 maximum).

FUNCTION zcrm_esm_reset_cont_arfc .

*"---------------------------------------------------------------------

*"*"Local Interface:

16 v1.0

*" IMPORTING

*" VALUE(I_PRODUCT_GUID) TYPE CRMT_OBJECT_GUID

*" VALUE(I_YEAR) TYPE SA_GJAHR

*" VALUE(I_MODE) TYPE CHAR1

*" VALUE(I_CONTRACT_SEL) TYPE CHAR1 OPTIONAL

*" EXPORTING

*" VALUE(ET_LOG) TYPE ZTT_CNTR_RESET_LOG

*" TABLES

*" T_CONTRACT_GUID STRUCTURE ZCRMT_OBJECT_GUID_FLAT

*"--------------------------------------------------------------------

DATA: lw_guid TYPE zcrmt_object_guid_flat.

REFRESH: it_log,

it_hdr_guid,

it_orderadm_h,

it_orderadm_i,

it_customer_h,

it_appointment,

it_partner,

it_status,

it_schedlin_i,

it_cumulated_i,

it_doc_flow.

LOOP AT t_contract_guid INTO lw_guid.

INSERT lw_guid-guid INTO TABLE it_hdr_guid.

ENDLOOP.

v_prd_gu = i_product_guid.

v_curr_y = i_year.

v_id_sel = i_contract_sel.

* Fill initial values

PERFORM f_fill_ini.

v_curr_year = v_curr_y.

v_prev_year = v_curr_year - 1.

PERFORM f_ret_contract_info.

* Order internal tables to optimize access

PERFORM f_order_i_tables.

PERFORM f_proccess.

PERFORM f_save_objects.

*Copy log table to export parameter

et_log = it_log.


17 v1.0

ENDFUNCTION.

Basically ‘F_RET_CONTRACT_INFO’ routine gets additional data calling function ‘CRM_ORDER_READ’.

‘F_PROCCESS’ calls ‘CRM_ORDER_MAINTAIN’; ‘F_SAVE_OBJECTS’ calls ‘CRM_ORDER_SAVE’ and

‘BAPI_TRANSACTION_COMMIT’.

Along these calls any relevant message is logged and finally the log table is returned.

18 v1.0

4. Results After implementing parallel processing processing time decreased considerably.

In a real production environment with 30k contracts it took about 8 hours to process in a normal report

(without parallelism) for full cycle modifications (CRM_ORDER_READ, CRM_ORDER_MAINTAIN,

CRM_ORDER_SAVE).

Once parallel processing logic was add to report, the same operations took about 2,5 hours. This means

less than the third part of original time.

It is recommended when measuring response times to ensure that environment variables remain the

same so that conclusions are right (i.e. users currently processing, server processes available, etc.).


19 v1.0

5. Conclusions Parallel processing using asynchronic remote function calls is recommended to be used when a large

amount of data has to be processed and time is a valuable resource. Being simple to implement, is an

alternative that can lead to development success.

Quicksteps for enabling parallel processing in a typical report:

1. Include parameters related to parallel processing in selection-screen of main report. Always make this run mode optional.

2. Create an RFC Function module in SE37 which has to be called from within the WHILE…ENDWHILE loop. Parameters that are normally passed to report via the selection screen should now be passed again to this Function Module.

3. Inside the WHILE…ENDWHILE loop in the main program, split the Table GUID Sequence into Work Packets by means of counters and make the ARFC calls by passing the Table GUID sequences along with the selection screen parameters and any other data necessary to process further.

4. Once all the tasks are returned, write code to read the records from the log table and call the method for showing the Run Log from within the program.

For a deeper look at the code refer to files in section 6 (based on example from section 5).

20 v1.0

6. Source

parallel processing with crm business transactions

Documents