saldre on dm365 - ti e2e community

SALDRE on DM365

User Guide

Literature Number: TBD May 2011

IMPORTANT NOTICE

Texas Instruments Incorporated and its subsidiaries (TI) reserve the right to make corrections, modifications, enhancements, improvements, and other changes to its products and services at any time and to discontinue any product or service without notice. Customers should obtain the latest relevant information before placing orders and should verify that such information is current and complete. All products are sold subject to TI’s terms and conditions of sale supplied at the time of order acknowledgment.

TI warrants performance of its hardware products to the specifications applicable at the time of sale in accordance with TI’s standard warranty. Testing and other quality control techniques are used to the extent TI deems necessary to support this warranty. Except where mandated by government requirements, testing of all parameters of each product is not necessarily performed.

TI assumes no liability for applications assistance or customer product design. Customers are responsible for their products and applications using TI components. To minimize the risks associated with customer products and applications, customers should provide adequate design and operating safeguards.

TI does not warrant or represent that any license, either express or implied, is granted under any TI patent right, copyright, mask work right, or other TI intellectual property right relating to any combination, machine, or process in which TI products or services are used. Information published by TI regarding third-party products or services does not constitute a license from TI to use such products or services or a warranty or endorsement thereof. Use of such information may require a license from a third party under the patents or other intellectual property of the third party, or a license from TI under the patents or other intellectual property of TI.

Reproduction of information in TI data books or data sheets is permissible only if reproduction is without alteration and is accompanied by all associated warranties, conditions, limitations, and notices. Reproduction of this information with alteration is an unfair and deceptive business practice. TI is not responsible or liable for such altered documentation.

Resale of TI products or services with statements different from or beyond the parameters stated by TI for that product or service voids all express and any implied warranties for the associated TI product or service and is an unfair and deceptive business practice. TI is not responsible or liable for any such statements.

Following are URLs where you can obtain information on other Texas Instruments products and application solutions: Products Applications

Amplifiers amplifier.ti.com Audio www.ti.com/audio

Data Converters dataconverter.ti.com Automotive www.ti.com/automotive

DSP dsp.ti.com Broadband www.ti.com/broadband

Interface interface.ti.com Digital Control www.ti.com/digitalcontrol

Logic logic.ti.com Military www.ti.com/military

Power Mgmt power.ti.com Optical Networking www.ti.com/opticalnetwork

Microcontrollers microcontroller.ti.com Security www.ti.com/security

Telephony www.ti.com/telephony

Video & Imaging www.ti.com/video

Wireless www.ti.com/wireless

Mailing Address: Texas Instruments

Post Office Box 655303 Dallas, Texas 75265

Copyright © 2010-12, Texas Instruments Incorporated

iii

Preface

Read This First

About This Manual The scene adaptive local dynamic range enhancement (SAL-DRE) algorithm improves the dynamic range of a scene by adaptive tone mapping such that the contents in both the dark and bright regions are visible. The purpose of SAL-DRE is to modify the brightness/contrast of a video/image in such a way that it looks more similar to how a human would see the actual scene.

Intended Audience

This document is intended for system engineers who want to integrate TI’s algorithms with other software to build a multimedia system based on the DM365 platform.

This document assumes that you are fluent in the C language, have a good working knowledge of Digital Signal Processing (DSP), digital signal processors, and DSP applications.

Read This First

iv

Related Documentation

You can use the following documents to supplement this user guide:

TBD

Text Conventions

The following conventions are used in this document:

Text inside back-quotes (‘‘) represents pseudo-code.

Program source code, function and macro names, parameters, and command line commands are shown in a mono-spaced font.

File names, paths are shown in italic.

Product Support

When contacting TI for support on this codec, please quote the product name (SALDRE algorithm on DM365) and version number. The version number of the codec is included in the Title of the Release Notes that accompanies this codec.

Read This First

v

Trademarks

Code Composer Studio and eXpressDSP are trademarks of Texas Instruments.

All trademarks are the property of their respective owners.

Software Copyright

Software Copyright © 2006 Texas Instruments Inc.

vi

Contents

Read This First ................................................................................................................ iii

About This Manual .....................................................................................................iii Intended Audience .....................................................................................................iii Related Documentation ............................................................................................. iv Text Conventions ...................................................................................................... iv Product Support ........................................................................................................ iv Trademarks ................................................................................................................ v Software Copyright..................................................................................................... v

Contents ..........................................................................................................................vi 1.1 Overview .........................................................................................................0-1 1.2 Data flow .........................................................................................................0-2 1.3 Execution flow .................................................................................................0-3 1.4 APIs 0-8 1.5 External dependencies ..................................................................................0-12 1.6 Quality tuning.................................................................................................0-12

1.6.1 Auto-exposure ................................................................................................. 0-13 1.6.2 Local flickering prevention ............................................................................... 0-13

1.7 Performance considerations ..........................................................................0-13

Revision History

Version Date Comments 1.3 May 5, 2011 • Updated example code p.0-4 and p.0-5 1.2 April 14, 2011 • Updated SALDRE_create() prototype with explanation on

new parameter temporalAverWinSize. • Updated Quality tuning section • Updated Performance section

1.1 Jan 16, 2011 Initial document

vii

This page is intentionally left blank

1

1.1 Overview The scene adaptive local dynamic range enhancement (SAL-DRE) algorithm improves the dynamic range of a scene by adaptive tone mapping such that the contents in both the dark and bright regions are visible. The purpose of SAL-DRE is to modify the brightness/contrast of a video/image in such a way that it looks more similar to how a human would see the actual scene. The dynamic range ratio of the scene in the figure below is 100 and the SAL-DRE algorithm enhances the dark regions while preserving the bright regions.

Examples of processed images through the SAL-DRE algorithm are shown below.

Unprocessed Processed

Contents

0-2

1.2 Data flow

The SALDRE algorithm must operate on raw data obtained by the ISIF. The capture data flow used must be ISIF->DDR->IPIPE depicted in the following figure:

The SALDRE algorithm simply operates between the ISIF and IPIPE altering the data flow to: ISIF->DDR->SALDRE->DDR->IPIPE:

Sensor ISIF

H3A

DDR

IPIPE

DDR DDR

BSC

RSZ2

VSTAB (IMCOP)

2A (ARM)

DDR

NF (IMCOP)

DDR

SW OSD (ARM)

1536x864 (20% more than 1280x720 for VS in H and V direction)

1536x864 YUV420

WxH YUV420

WxH YUV420

AEWB control to Sensor, IPIPE, ISIF

RSZ1

DDR

DDR

LDC

768x432 YUV420

Face Info

(W*1.2) x (H*1.2) YUV420 (From RSZx) WxH = 1280x720, 320x180, 640x360

DDR

RSZ1

DDR

394x216 YUV420

Contents

0-3

The IPNC can enable an alternate data flow ISIF->IPIPE which is to pass the output of the ISIF directly to the IPIPE. This data flow is the default one and consumes less DDR bandwidth. However it cannot be used in conjunction with SALDRE since there is no access to the raw data (the IPIPE outputs directly YUV data).

The latest release IPNC 2.5 automatically switches to the ISIF->DDR->IPIPE data flow whenever the user enables SALDRE through the GUI. If SALDRE was previously disabled, the system restarts the AVServer in order to switch from the default ISIF->IPIPE data flow.

To integrate SALDRE into a prior release, such as 2.0, one must edit the file av_capture\application\ipnc\av_server\src\common\avServerUi.c and modifies the function UI_setConfig() such that config->captureRawInMode is initialized to AVSERVER_CAPTURE_RAW_IN_MODE_DDR_IN whenever SALDRE is enabled.

1.3 Execution flow

The SALDRE algorithm is implemented by the library av_capture\framework\alg\lib\alg_saldre_ti.a .

The library contains the following APIs:

1. SALDRE_create()

Sensor ISIF

H3A

DDR

IPIPE

DDR DDR

BSC

RSZ2

VSTAB (IMCOP)

2A (ARM)

DDR

NF (IMCOP)

DDR

SW OSD (ARM)

1536x864 (20% more than 1280x720 for VS in H and V direction)

1536x864 YUV420

WxH YUV420

WxH YUV420

AEWB control to Sensor, IPIPE, ISIF

RSZ1

DDR

DDR

LDC

768x432 YUV420

Face Info

(W*1.2) x (H*1.2) YUV420 (From RSZx) WxH = 1280x720, 320x180, 640x360

DDR RSZ1

DDR

394x216 YUV420

SALDRE

DDR

Contents

0-4

2. SALDRE_func0()

3. SALDRE_func1()

4. SALDRE_setupFunc2()

5. SALDRE_startFunc2()

6. SALDRE_waitFunc2()

7. SALDRE_endFunc2()

As it can be noted, there is a creation function called SALDRE_create(), which just needs to be called once during system initialization and three processing functions func0, func1 and func2. Out of these three functions, only func1 executes on the ARM, the other two use the DM368 SOC’s hardware accelerators. Note also that func1 processes the output of func0. To take advantage of parallel processing between ARM and the accelerators, func2 can run in parallel with func1 but this involves a more complicated processing flow with ping-pong buffering on the buffers. A simple sequential schedule can first be implemented using the following pseudo code:

byte *srcAddr; Byte dstBuffer[2*WITH*HEIGHT]; SALDRE_create(); while(isThereNewFrameFromCaptureDriver== TRUE) { srcAddr= getFrameAddrFromCaptureDriver(); SALDRE_func0(srcAddr, func0output, …); SALDRE_func1(func0output, 0, …); SALDRE_setupFunc2(srcAddr, dstBuffer, …); SALDRE_startFunc2(); SALDRE_waitFunc2(); SALDRE_end_func2(); IPIPE_RSZ(dstBuffer); }

As this pseudo code shows, all the processing is sequential: the frame obtained from the getFrameAddrFromCaptureDriver() function gets passed along to the other functions in a sequential manner. The function IPIPE_RSZ() processes the raw bayer frame pointed by dstBuffer, resize it, enhances it and converts it into YUV format. It runs one the IPIPE accelerator, which is different than the accelerator used by SALDRE. So another level of parallelism can be added.

Now to take advantage of parallelism between IPIPE, SALDRE accelerator and ARM, a different schedule needs to be implemented, which involves ping-pong buffering for both input and output buffers and pipeline fill-up. One thing to note is that func0 and func1 must be ran sequentially to each other since func1 processes the output of func0. Here is such a pseudo code reflecting new concurrent schedule:

Byte *srcAddr[2];

Byte *dstBuffer[2];

Byte *func0output[2];

Byte *weightBuffer;

Contents

0-5

/* Buffers must be allocated using CMEM so they are continuous */

outputBufSize= SALDRE_getOutputBufferSize(WIDTH, HEIGHT, WIDTH);

dstBuffer[0]= OSA_cmemAlloc(outputBufSize, 32);

dstBuffer[1]= OSA_cmemAlloc(outputBufSize, 32);

func0output= OSA_cmemAlloc((2*WIDTH*HEIGHT)/256, 32);

weightBuffer= OSA_cmemAlloc(2*WIDTH*HEIGHT, 32);

SALDRE_create(WIDTH, HEIGHT, temporalAverWinSize);

saldreInBufferIndex= 0;

saldreOutBufferIndex= 0;

srcAddr[0]= getFrameAddrFromCaptureDriver();

SALDRE_runFunc0(srcAddr[0], func0output, WIDTH, HEIGHT, STRIDE);

SALDRE_runFunc1(func0output, 0, WIDTH, HEIGHT, 0, 128, 64, 6144, 4096);

saldreInBufferIndex= 1;

frame_no= 1;

while(isThereNewFrameFromCaptureDriver== TRUE) {

if (frame_no != 1) IPIPE_RSZ_thread_start(dstBuffer[saldreOutBufferIndex^1]);

srcAddr[saldreInBufferIndex]= getFrameAddrFromCaptureDriver();

SALDRE_runFunc0(srcAddr[saldreInBufferIndex], func0output WIDTH, HEIGHT, STRIDE);

SALDRE_setupFunc2(srcAddr[saldreInBufferIndex^1], dstBuffer[saldreOutBufferIndex], weightBuffer, NULL, WIDTH, HEIGHT, STRIDE, STRIDE);

SALDRE_startFunc2();

SALDRE_func1(func0output, saldreInBufferIndex, WIDTH, HEIGHT, 0, 128, 64, 6144, 4096);

SALDRE_waitFunc2();

SALDRE_end_func2();

saldreInBufferIndex ^=1;

Contents

0-6

saldreOutBufferIndex^= 1;

if (frame_no != 1)

IPIPE_RSZ_thread_wait();

frame_no++;

}

SALDRE_setupFunc2(srcAddr[saldreInBufferIndex^1], dstBuffer[saldreOutBufferIndex], weightBuffer, NULL, WIDTH, HEIGHT, STRIDE, STRIDE);

SALDRE_startFunc2();

SALDRE_waitFunc2();

SALDRE_end_func2();

IPIPE_RSZ_thread_start(dstBuffer[saldreOutBufferIndex]);

IPIPE_RSZ_thread_wait();

srcAddr and dstBuffer are now two entries arrays to implement ping-pong buffering of input and output buffers.

We first fill the pipeline by grabbing the very first frame. This frame is processed by SALDRE_func0 and then SALDRE_func1 . The input buffer flag saldreInBufferIndex is then flipped from 0 to 1 before going to the while() loop which is the steady state processing in which func1 is ran parallel to func2.

In that loop, we first run the IPIPE_RSZ_thread_start() on a frame previously processed by func2. If it is the first iteration, this step must be skipped since the very first frame grabbed hasn’t had the opportunity to be processed yet. Note this IPIPE_RSZ_thread_start() is slightly different than IPIPE_RSZ() used in the sequential schedule because it must switch to a different thread that will run IPIPE_RSZ(). This is to make the call non-blocking. We assume we run on Linux environment so at this point, two threads will run in parallel: the IPIPE_RSZ thread which mainly calls the IPIPE accelerator and the SALDRE thread.

Next in the SALDRE thread, a new frame is grabbed and processed by func0() and func1(). Note that the execution of func1() happens during the start() and wait() calls of func2, effectively achieving parallelism between ARM and SALDRE’s accelerator. Lastly, we wait for completion of the IPIPE_RSZ_thread() at the end of the innermost loop. Basically we need to ensure that all the accelerators have finished their processing before going to the next loop iteration.

A variation of this concurrent schedule is implemented in the IPNC in files av_capture/application/ipnc/av_server/src/video/videoCaptureThr.c and av_capture/application/ipnc/av_server/src/video/videoSaldreThr.c .

The difference between the actual IPNC implementation and above pseudo-code is that the processing flow on IPNC system is split into two threads:

- A SALDRE thread implemented in videoSaldreThr.c. Its implementation is very close to the pseudo code, except that it doesn’t grab any frames and call IPIPE_RSZ_thread(). These two actions are carried out by the video capture thread.

Contents

0-7

- A video capture thread implemented in videoCaptureThr.c: this thread is responsible of grabbing a new frame, call IPIPE_RSZ function and call the SALDRE thread. Note that IPIPE_RSZ doesn’t need a separate thread as it executes inside the video capture thread. Parallelism between SALDRE and IPIPE_RSZ is achieved because SALDRE runs in a different thread than the video capture thread, which contains the IPIPE_RZS function.

Please refer to function VIDEO_captureTskRunDdrIn(). In it, look at the following code section:

if (gAVSERVER_config.aewb_config.saldreEnable == FALSE){

VIDEO_captureRszFunc(pRawBufInfo->virtAddr, pOutBufInfo, rszBufId, numStreams);

}

else {

saldreOutputBufAddr= VIDEO_saldreGetOutputBufAddr();

OSA_semSignal(&g_saldreSemStart);

VIDEO_captureRszFunc(saldreOutputBufAddr, pOutBufInfo, rszBufId, numStreams);

OSA_semWait(&g_saldreSemDone, OSA_TIMEOUT_FOREVER);

}

On the case SALDRE is enabled, the video capture thread gets the output address of the last frame processed by SALDRE and then it triggers the SALDRE thread by signaling a semaphore. From this point of time, the SALDRE thread executes in parallel with the video capture thread. The video capture thread goes on calling the IPIPE_RSZ function, which is VIDEO_captureRszFunc(). A semaphore wait must be called to synchronize all the processing.

Note that waiting for IPIPE completion is implicitly done inside VIDEO_captureRszFunc().

Frames are continuously processed because the function VIDEO_captureTskRunDdrIn() is called inside a while(1) loop in function VIDEO_captureTaskMain(). Additionally, the system maintains a pool of input buffers in order to prevent frame drop in case . The number of input buffers is set by the macro symbol VIDEO_NUM_BUF in file video.h . For optimum performance, please set it 4 otherwise frame rate won’t be the best the system can achieve. Default value of 3 is to allow higher resolution such as 2 MegaPixels, 3 MegaPixels without encountering any memory allocation error. If your target resolution is 720P or 1080P, you can set it to 4.

1.4 APIs

Following are descriptions of the SALDRE APIs.

║ Name

SALDRE_create() – Create the algorithm ║ Synopsis

Contents

0-8

Int32 SALDRE_create(Uint16 maxWidth, Uint16 maxHeight, Uint16 temporalAverWinSize) ║ Arguments

Uint16 width /*maximum width in number of pixels of any Bayer frame that will be processed. */ Uint16 height /*maximum height in number of rows of any Bayer frame that will be processed */ Uint16 temporalAverWinSize /* size of the temporal averaging window in number of frames */

║ Return Value 0: success -1: error

║ Description

Initialize the SALDRE algorithm. Must be called before any other SALDRE functions. To prevent local flickering effects due to sensor noise, the SALDRE algorithm performs some temporal averaging. The parameter temporalAverWinSize sets the number of frames that are averaged over time. The greater this number is, the more averaging is done and the more stable the image is. But the scene doesn't adapt so quickly to sudden illumination changes or spatially large content changes. So if the camera is pointing to a mostly static scene with small motion areas, then a large value such as 16 can be assigned to temporalAverWinSize. Otherwise use smaller values such as 2, which is the default one set in the IPNC codebase. Refer to file videoSaldreThr.c in which SALDRE_create() is called by VIDEO_saldreCreate().

║ Name

SALDRE_getOutputBufferSize

║ Synopsis Int32 SALDRE_getOutputBufferSize(Uint16 roiWidth, Uint16 roiHeight, Uint16 outputStride) ║ Arguments

Uint16 roiWidth /* ROI width of the frame in pixel */ Uint16 roiWeight /* ROI height of the frame in pixel */ Uint16 dstStride /* stride of the output buffer in pixels, usually equal to roiWidth */

║ Return Value sizeOutput in bytes of frame buffer for holding the output

║ Description

This function returns the minimum number of bytes that need to be allocated to hold one frame output by SALDRE_func2(). The input parameters correspond to the region interest's pixel width and pixel height of the frame to process as well as the stride of the output frame. Most of the time the returned size value is equal to 2*dstStride*roiHeight but for some case such as 720P resolution, it is a larger value because the actual output height may be greater than roiHeight.

Later, when calling SALDRE_setupFunc2(), the application must make sure that the arguments imgHeight and dstStride for this function satisfy the expression: 2*imgHeight*dstStride= sizeOutput .

║ Name

SALDRE_runFunc0

║ Synopsis

Contents

0-9

Int32 SALDRE_runFunc0(Uint8* srcAddr, Uint8* dstAddr, Uint16 imgWidth, Uint16 imgHeight, Uint16 srcStride) ║ Arguments

Uint8* srcAddr /* pointer to the raw Bayer frame to process of size 2* srcStride *imgHeight bytes */ Uint8* dstAddr /* pointer to buffer of size 2*imgWidth*Height/256 bytes */ Uint16 imgWidth /* width in number of pixels of the Bayer frame to process */ Uint16 imgHeight /* height in number of rows of the Bayer frame to process */ Uint16 srcStride /* stride in number of pixels of the Bayer frame to process */


║ Description

This function collects some statistics about Bayer frame pointed by srcAddr and write them into the buffer pointed by dstAddr of size 2*imgWidth*Height/256 bytes.

║ Name

SALDRE_runFunc1

║ Synopsis Int32 SALDRE_runFunc1(Uint8 *dstAddrFromFunc0, Uint16 index, Uint16 imgWidth, Uint16 imgHeight, Uint32 config, Int16 GBEStrength, Int16 GCEStrength, Int16 LBEStrength, Int16 LCEStrength) ║ Arguments

Uint8* dstAddrFromFunc0/* pointer to the statistics buffer generated by SALDRE_runFunc0 */ Uint16 index /* if sequential processing implemented, always 0, if concurrent processing implemented, alternate between 0 or 1 */ Uint16 imgWidth /* width in number of pixels of the Bayer frame to process */ Uint16 imgHeight /* height in number of rows of the Bayer frame to process */ Uint32 config /* Ignored for now, set to 0 */

Int16 GBEStrength /*global brightness strength, can be 64, 128, 256 */ Int16 GCEStrength /*global contrast strength, can be 64, 128, 256 */ Int16 LBEStrength /*local brightness strength, can be 4096, 6144, 8192 */ Int16 LCEStrength /*local contrast strength, should be set to 4096 */


║ Description

This function generates tone curves from the statistics generated by SALDRE_runFunc0

Contents

0-10

║ Name

SALDRE_setupFunc2

║ Synopsis ║ Int32 SALDRE_setupFunc2(Uint8 *srcAddr, Uint8*dstAddr, Uint8* weight, Uint8 *toneCurves, Uint16 imgWidth, Uint16 imgHeight, Uint16 strideSrc, Uint16 strideDst) ║ Arguments

Uint8 *srcAddr /* pointer to the raw Bayer frame to process of size 2*strideSrc *imgHeight bytes */ Uint8 *dstAddr /* pointer to the destination buffer of size 2*strideDst *imgHeight bytes */ Uint8 *weight /* pointer to buffer of size 2*imgWidth*Height */ Uint8 *toneCurves /* set to NULL to use tone curves generated by SALDRE_runFunc1()*/ Uint16 imgWidth /* width in number of pixels of the Bayer frame to process */ Uint16 imgHeight /* height in number of rows of the Bayer frame to process */ Uint16 srcStride /* stride in number of pixels of the Bayer frame to process */ Uint16 dstStride /* stride in number of pixels of the destination buffer */


║ Description

Setup the function that will enhance the dynamic range of the source frame, using the tone curves generated by SALDRE_runFunc1() if toneCurves set to NULL. If toneCurves is different than NULL, it must point to valid toneCurves table. Contact TI for details on format of this table.

The application must make sure that the arguments imgHeight and dstStride for this function satisfy the expression: 2*imgHeight*dstStride= sizeOutput where sizeOutput was returned by SALDRE_getOutputBufferSize().

║ Name

SALDRE_startFunc2

║ Synopsis void SALDRE_startFunc2() ║ Arguments ║ Return Value

0: success -1: error

║ Description

Start execution of the function setup by SALDRE_setupFunc2(). Upon return of the function, the SALDRE processing has started in the background.

Contents

0-11

║ Name

SALDRE_waitFunc2

║ Synopsis void SALDRE_waitFunc2() ║ Arguments ║ Return Value


║ Description

Wait for the completion of the function started by SALDRE_startFunc2(). Upon return of the function, the SALDRE processing has completed.

║ Name

SALDRE_endFunc2

║ Synopsis void SALDRE_endFunc2() ║ Arguments ║ Return Value


║ Description

De-initialize the SALDRE function setup by SALDRE_setupFunc2() . Must be called at the end of each frame processing, before processing next frame.

║ Name

SALDRE_resetFunc2

║ Synopsis void SALDRE_resetFunc2() ║ Arguments ║ Return Value


║ Description

This function needs to be called after every time the AV_server is restarted due to change of resolution, codec, enabling/disabling SALDRE.

1.5 External dependencies

The SALDRE library calls the following functions defined in av_capture\framework\alg\src\alg_aewb_control.c:

ALG_aewbGetAEValues()

Contents

0-12

ALG_aewbGetAWBGains() ALG_aewbGetRgb2Rgb()

These functions access some image statistics maintained by the TI 2A or Appro 2A driver. If the system uses a custom driver, these functions will have to be re-implemented.

1.6 Quality tuning

The SALDRE algorithm doesn't require much tuning as long as the functions ALG_aewbGetAEValues(), ALG_aewbGetAWBGains() and ALG_aewbGetRgb2Rgb() are correctly implemented. Some tuning concerning auto-exposure and local flickering prevention may be done per the following guidance.

1.6.1 Auto-exposure

For TI-2A, in case bright regions become saturated after SALDRE is applied, lower the image pipe gain by changing in

av_capture\framework\alg\src\alg_aewb.c

the aeDynamicParams.ipipeGainRange[i].max value to some value lower than 4092.

1.6.2 Local flickering prevention

To prevent local flickering effects due to sensor noise, the SALDRE algorithm performs some temporal averaging. The parameter temporalAverWinSize of SALDRE_create() sets the number of frames that are averaged over time. The greater this number is, the more averaging is done and the more stable the image is. But the scene doesn't adapt so quickly to sudden illumination changes or spatially large content changes. So if the camera is pointing to a mostly static scene with small motion areas, then a large value such as 16 can be assigned to temporalAverWinSize. Otherwise use smaller values such as 2, which is the default one set in the IPNC codebase. Refer to file videoSaldreThr.c in which SALDRE_create() is called by VIDEO_saldreCreate().

1.7 Performance considerations

The following lists timing information for each function in micro-second for 720P resolution:

func0 time_frm = 6500


func2 setup time_frm = 5546


Total frame processing time (func0 + setup func2 + max(func1, func2)= 37046

As you see the total frame processing time exceeds 33 ms, which indicates that the algorithm cannot run at 30 fps. However it is possible to improve its speed by executing SALDRE_func0(), SALDRE_func1() every other N frames while SALDRE_setupFunc2(), SALDRE_startFunc2(), SALDRE_waitFunc2() and SALDRE_endFunc2() must be executed every frame. This way we can achieve > 30 fps.

To see how such a technique is implemented please refer to the implementation of function VIDEO_saldreCreate() and VIDEO_saldreRun() in file videoSaldreThr.c, which can be

Contents

0-13

summarized as follow: from videoCaptureThr.c, VIDEO_saldreCreate(Uint16 frameSkip) is called with argument frameSkip set to the number of frames to skip between consecutive calls of SALDRE_func0() and SALDRE_func1(). Then VIDEO_saldreRun() updates the internal counter numframes which triggers the execution of SALDRE_func0() and SALDRE_func1() whenever the condition !(numframes % frameSkip)) becomes true.

When value frameSkip=0 is passed to VIDEO_saldreCreate(), there isn't enough processing power to run SALDRE at 30 fps and ARM load shoots up to 90%. But when the value frameSkip=1 is passed to VIDEO_saldreCreate(), frame rate becomes 30 fps and ARM load is approximately 58 %. To reduce ARM load further, you can increase frameSkip at the expense of some lag when big illumination changes occur in the scene. If the scene is mostly static, this may not be an issue. No adverse effects have been observed when frameSkip=1 . But if you increase frameSkip to 2, 3, etc, then you will want to decrease temporalAverWinSize parameter, which is passed to SALDRE_create() in file videoSaldtreThr.c .

In addition, make sure that the number of input buffers is set by the macro symbol VIDEO_NUM_BUF in file video.h . For optimum performance, please set it to 4 otherwise frame rate won’t be the best the system can achieve. Default value of 3 is to allow higher resolution such as 2 MegaPixels, 3 MegaPixels without encountering any memory allocation error. If your target resolution is 720P or 1080P, you can set it 4.

saldre on dm365 - ti e2e community

Documents