Independent Study

1

A REPORT ON MY WORK ON THE

BEAGLEBOARD XM (Angstrom OS),

WHICH INCLUDES IMAGE PROCESSING,

OPTIMIZATION AND

WORK WITH LEOPARD CAMERA.

Kaushik Kanetkar______

Independent Study

2

Contents:

1. Porting Angstrom to Beagle Xm 3

2. Getting the code to work alongwith ffmpeg on Beagle 6

3. Work on Leopard camera 7

4. My time-lapse example 9

5. Remote debugging using GDB 10

6. Image processing and optimization 11

1.Neon instructions

7. Porting Ubuntu O.S on the Beagleboard 18

Independent Study

3

Porting Angstrom on to Beagle Xm:

Following are the steps to load Angstrom kernel on your Beagleboard Xm:

Preparing the SD card:

Plug in the SD card on your host system and wait for it to mount. You can check

by the command df -h . Look for the line which starts with /dev/sdXX, it will

generally be mounted in /media/

Make sure the size of the SD card is enough.

Now un-mount the SD card using umount /dev/sdXX

For Angstrom to run you will need to have a sd-card with two partitions; one fat partition to hold the boot files and then a ext3 partition with the root file system on it.

Make 2 partitions p1:Fat32 (0x0c) and other p2: ext3 (0x83) partition.

Make p1 bootable and keep around 50-70 Mb space.

To make the two partitions you can either use the GUI based application: Disk Utility. The other way would be to use the script omap-mkcard.sh in the folder ‘ ‘ by running the following commands: chmod +x omap3-mkcard.sh sudo ./omap3-mkcard.sh /dev/sdX sync

Remove the SD card and plug it in after 2-3 seconds. You will find two partitions /media/boot and /media/Angstrom.

The next step would be to generate a build. You can use the rootangstrom.tar.gz in the folder, or else you can generate your custom build at http://narcissus.angstrom-distribution.org/.

Select: BeagleBoard (the original BeagleBoard and BeagleBoard-xM both use the same build); simple; and console only. Now in the “additional packages” section, make sure you select the following: Toolchain

Native u-boot mkimage

All kernel modules

Ffmpeg

Bootloader Files (x-load/u-boot/scripts) in the “Platform specific packages

section”

Independent Study

4

Now click on ‘build me!’ which will generate a “*.tar.gz”. Download this “*.tar.gz”

in your working directory.

There are only three files that are mandatory for boot partition. The following

lines will extract the files from the download build and copy those to the boot

partition on the sd-card.

tar –xvf [your-file].tar.gz

cp boot/MLO-* /media/boot/MLO

cp boot/uImage-* /media/boot/uImage

cp boot/u-boot-*.bin /media/boot/u-boot.bin

sync

Now we have to copy/extract the filesystem to your SD card.

cd /media/Ansgtrom

sudo tar –xvf [path to you download file].tar.gz

sync

Make it safe to remove your SD card.

umount /media/boot

umount /media Angstrom

Remove the SD card and place it inside the card case on your Beagleboard.

Connect the Serial (BeagleBoardxM) to USB(Host PC) converter and configure the minicom connection.

Start minicom on your host machine in configuration mode. As root: # minicom -o -s -w

· A menu of configuration should appear. Use the Down-arrow key to scroll down and select the Serial port setup option, and press Enter.

· Verify that the listed serial port is the same one that is connected to the target board. If it is not, press A, and enter the correct device. This is /dev/ttyUSB0 on most Linux distributions.

· Set the Bps/Par/Bits option by pressing the letter E and using the next menu toset the appropriate values. You press the key that corresponds to the value 115200, and then press Enter.

· Set Hardware flow control to No using the F key.

· Set Software flow control to No using the G key.

Independent Study

5

· Press Enter to return to the main configuration menu, and then press Esc to exit this menu.

Reset the board, and wait for a moment. If you do not see output from the board, press Enter several times until you see the prompt. If you do not see any output from the board, and have verified that the serial terminal connection is setup correctly, contact your board vendor.

When you boot Angstrom for the first time on the beagleboard, it will take some time. Be patient. Once it is done , you will be asked for a beagleboard login: (enter ‘root’)

Run the following command:

ifconfig usb0 192.168.1.10 (gives an ip address to your usb)

Now run the following command on your host:

ifconfig eth0 192.168.1.4 (configures a static ip address)

From your beagleboard see if you can ping your host.

ping 192.168.1.4

If you are able to ping, then you are all set and ready to work with Angstrom on

beagleboard and to transfer files between your beagle and the host system !

Independent Study

6

Timelapse on the Beagleboard :

Build the code_to_capture_frames.c code on the Beagleboard using: arm-angstrom-linux-gnueabi-gcc code_to_capture_frames.c –lm –pthread –lrt

Run the executable ./a.out file

The leopard camera will now capture images and store it in the YUV folder.

The next step is to convert the YUV’s into PPM’s and encode them into a video.

You have two options:

1. On the Beagleboard:

Build the buildvideo.c code using

gcc buildvideo.c –lm

Now run the ./a.out output file and specify the no. of frames you want to

use in forming the video.

./a.out <no.of frames>

2. On your host system (Ubuntu):

Transfer the YUV folder from Beagleboard to the host system using scp.

scp –r yuvs/ hostname@host_ip_address:/home/

Build the buildvideo.c code and run it as above.

Run the video on VLC player.

Independent Study

7

Leopard Camera:

Refer to the following document regarding Video4Linux in Srinivas’ report and the link :

http://v4l2spec.bytesex.org/spec/

I have made a few changes to his code to optimize the use of Leopard camera.

Firstly, I will just go through some of the important ioctl’s for getting images from the

Leopard camera.

ioctl(fd, VIDIOC_REQBUFS, &req);

-This allocates the number of buffers to be allocated.

ioctl(fd, VIDIOC_QUERYBUF, &buffer);

-It queries the status of the buffers. You need to make sure that all the allocated buffers

are queried.

ioctl(fd, VIDIOC_QBUF, &buffer);

-This ioctl call is used the enqueue a buffer in the driver’s incoming queue.

ioctl(fd, VIDIOC_DQBUF, &buffer);

-This ioctl is used to dequeue a filled buffer from the driver’s outgoing queue.

ioctl(fd, VIDIOC_STREAMON, &type);

ioctl(fd, VIDIOC_STREAMOFF, &type);

-The VIDIOC_STREAMON and VIDIOC_STREAMOFF are used to start and stop the

capture.

Note: While using the VIDIOC_STREAMON there is a high chance that it might prompt

an error that the device is busy. So make sure there is a ‘while’ loop such that wait till

the ioctl stops returning -1 (error value), and then only proceed.

Independent Study

8

It takes a lot of time for the VIDIOC_DQBUF, VIDIOC_QBUF calls to execute, especially if

only one buffer is used.

I added more buffers in the queue ( a maximum of 32 can be allocated ).

Adding more buffers brought down the time for capturing and also brought the frame

capturing rate up.

The following was included after all the settings related to MMAP and STREAMON were

done:

{buffer.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;

buffer.memory = V4L2_MEMORY_MMAP;

buffer.index = j

//make sure this value of ‘j’ is the value of buffer number which you are currently using.

ioctl(fd, VIDIOC_DQBUF, &buffer);

----getting the yuv-----

ioctl(fd, VIDIOC_QBUF, &buffer);

}

With this, I was able to get a better frame rate, by making use of buffers.

Future Scope:

While aiming to achieve a higher frame rate, it gives an error in the VIDIOC_QBUF or

VIDIOC_DQBUF.

If somehow, this error can be avoided by making optimum use of the maximum amount

of buffers available, then, a stable and a better frame rate can be achieved.

*Special packages to click screenshots*

Mplayer:

Angstrom has an application called ‘mplayer’ which can be used on the Beagleboard, to

view and click images based on the frame rate.

I used this application on the console to click images.

Independent Study

9

Time-lapse example :

1. I have created a time-lapse focusing on a digital clock. The frequency is 1Hz.

The ppms folder contains the images clicked by the Leopard camera at every second.

2. The other time-lapse I created was the melting of ice-cubes.

http://www.youtube.com/watch?v=tsxUPD3gP9w&feature=g-upl

Independent Study

10

Remote debugging using GDB:

Get the following cross-compiler tools (arm-none-linux-gnueabi-gcc) from

http://www.codesourcery.com/sgpp/lite/arm/portal/release1600

Using command line:

Now, for example the ip address of my system (host) is 192.168.1.4 and the ip address

of the target (Beagle) is 192.168.1.14

With the given Angstrom kernel, you indeed have gdbserver installed on your target.

To start gdbserver on the target you need to specify a port number which you will listen

and then connect through and the executable to debug. The port can be anything so

long as it’s not a reserved port e.g. 21,80.

I am using the port number 2345.

gdbserver test:2345 a.out

This creates the pid and starts listening on port 2345.

Run the gdb command from the host system:

arm-none-linux-gnueabi-gdb ./a.out

Once gdb starts, run the following:

(gdb) target remote 192.168.1.14:2345

Independent Study

11

Image processing and optimization:

I used an image from the Leopard cam (pic-5.ppm), having a resolution of 640x480.

I applied the sharpen algorithm on it without any optimization or mulit-threading.

This was the result I got using the arm-angstrom-linux-gnueabi-gcc compiler for ARM.

Then, I used some optimizations mentioned in the following paper by Prof.Sam Siewert

http://software.intel.com/en-us/articles/using-intel-streaming-simd-extensions-and-intel-integrated-

performance-primitives-to-accelerate-algorithms/

These are the Intel SIMD instructions for gcc optimization.

It provided the following speed-up.

Generally we would want the processor to speed up more. So ARM does have a number

of instruction sets, one of them being the NEON instruction set.

NEON is included in all Cortex-A8 devices but is optional in Cortex-A9 devices. Lets see

how the NEON provides better optimization.

Independent Study

12

NEON Instructions :

The Advanced SIMD extension (aka NEON or "MPE" Media Processing Engine) is a

combined 64- and 128-bit single instruction multiple data (SIMD) instruction set that

provides standardized acceleration for media and signal processing applications. NEON

can execute MP3 audio decoding on CPUs running at 10 MHz and can run

the GSM adaptive multi-rate (AMR) speech codec at no more than 13 MHz. It features a

comprehensive instruction set, separate register files and independent execution

hardware. NEON supports 8-, 16-, 32- and 64-bit integer and single-precision (32-bit)

floating-point data and operates in SIMD operations for handling audio and video

processing as well as graphics and gaming processing. In NEON, the SIMD supports up to

16 operations at the same time. Devices such as the ARM Cortex-A8 and Cortex-A9

support 128-bit vectors but will execute with just 64 bits at a time, whereas newer

Cortex-A15 devices can execute 128 bits at once.

NEON Instructions perform “Packed SIMD” processing:

Registers are considered as vectors of elements of the same data type

Data types can be: signed/unsigned 8-bit, 16-bit, 32-bit, 64-bit, single prec. float

Instructions perform the same operation in all lanes

As it can be seen, for adding R1 to R2, it performs the single instruction of adding on

multiple data (four lanes) thus providing speedup.

Independent Study

13

Using the NEON optimization:

-- In order to instruct the compiler to produce NEON you should use the following

compile flags:

-mfpu=neon

-- To specify which floating-point ABI to use, permissible values are: `soft', `softfp' and `hard'. Specifying `soft' causes GCC to generate output containing library calls for floating-point operations. `softfp' allows the generation of code using hardware floating-point instructions, but still uses the soft-float calling conventions. `hard' allows generation of floating-point instructions and uses FPU-specific calling conventions. -mfloat-abi=name -- To specify the name of the target ARM processor. GCC uses this name to determine what kind of instructions it can emit when generating assembly code.

-mcpu=cortex-a8

-- If you want the compiler to attempt to vectorize your integer / floating point code for

the NEON you should add: -ftree-vectorize to your flags.

-ftree-vectorize

Using all these alongwith the –O3 optimization, this was the result I got on the same

single threaded sharpen code on the 640x480 image.

The NEON instructions provided a speedup of around 3X!

Independent Study

14

Multi-threading:

Multi – threading does give better results especially when higher amount of processing

is involved. Threading helps in speeding up the process. I have written a code to

modify/process any sized image into any desired number of thread blocks, and get the

necessary speedup.

This was the result, when the image was sharpened using single-threading.

When multiple threads are used to process the image at its different locations, the

following result is observed on the same 640x480 image. (Using 4x3 threads)

Multithreading using 8x6 threads:

Thus, there is a speed up when 3x4 threads are used as compared to 6x8 threads.

Independent Study

15

Following are the results of some image processing done on the images captured by the

Leopard camera.

Some other transformations:

1. Centroid

<->

Thus, the neon provides a huge speedup in performance.

Independent Study

16

2. Boost

<->

The following output shows the speedup in performance using NEON.

Independent Study

17

Image upscaling:

I wrote a code to convert 320x240 images to 640x480 images.

The method I used was nearest neighbor averaging. Now for example, we have to

transform a 2x2 to a 4x4.

The new added pixel will be the average of the neighboring pixels.

Once, the neighboring pixels method was carried out, I applied the averaging filter, to

smoothen out edges (if any)

Similarly, for a conversion from 640x480 to 1280x720, the neighboring pixels

method is used, where the ratio is taken into account.

640/1280 = (1/2)

480/720 = (2/3)

Every new pixel in the up scaled image corresponds to the (1/2)th location in the

original images’ width while, to the (2/3)rd location in its height.

Independent Study

18

THIS IS A TUTORIAL TO HELP YOU

PORT THE

“UBUNTU”

OPERATING SYSTEM

ON THE BEAGLEBOARD xM

Independent Study

19

Porting UBUNTU on the BeagleBoard:

Following are the steps to port another operating system, Ubuntu, on the

BeagleBoard Xm.

Steps:

Make your home directory to store all the files for the Ubuntu operating system.

(root files)

Plug in your micro SD card onto your host laptop system.

Now we need to get the armel:

wget http://ynezz.ibawizard.net/beagleboard/natty/ubuntu-11.04-r1-minimal-

armel.tar.xz

md5sum ubuntu-11.04-r1-minimal-armel.tar.xz

tar –xvf ubuntu-11.04-r1-minimal-armel.tar.xz

Instead of the above commands, you can use the files in my folder ‘Ubuntu-

make’.

Now, enter the following:

cd ubuntu-11.04-r1-minimal-armel.tar.xz

sudo ./setup_sdcard.sh --mmc /dev/sdb --uboot beagle_bx --rootfstype ext3

Independent Study

20

This process includes formatting your SD card, transferring boot files into the

boot partition and the rootfilesystem onto the other partition of the memory

card.

Independent Study

21

When it is done, remove the SD card, and plug it back after 2-3 seconds.

You will see two partitions, as listed below.

Remove the SD card from the card holder and plug it in the Beagleboard and

switch on the power.

Connect the Serial (BeagleBoardxM) to USB(Host PC) converter and configure the minicom connection.

Start minicom on your host machine in configuration mode. As root: #minicom -o -s -w

· A menu of configuration should appear. Use the Down-arrow key to scroll down and select the Serial port setup option, and press Enter.

· Verify that the listed serial port is the same one that is connected to the target board. If it is not, press A, and enter the correct device. This is /dev/ttyUSB0 on most Linux distributions.

Independent Study

22

· Set the Bps/Par/Bits option by pressing the letter E and using the next menu toset the appropriate values. You press the key that corresponds to the value 115200, and then press Enter.

· Set Hardware flow control to No using the F key.

· Set Software flow control to No using the G key.

· Press Enter to return to the main configuration menu, and then press Esc to exit this menu.

Reset the board, and wait for a moment. If you do not see output from the board, press Enter several times until you see the prompt. If you do not see any output from the board, and have verified that the serial terminal connection is setup correctly, contact your board vendor.

You will see something like this when the kernel boots.

Once, it has booted, it will ask for a login, and a password.

The login is ‘ubuntu’, and the password is ‘temppwd’.

Independent Study

23

It is necessary that you update your kernel using the following command.

To get the Ethernet working, enter the following command.

sudo dhclient eth0

You can check your assigned ip-address using ifconfig –a

Independent Study

24

References:

1. Nisheeth Bhatt’s IS

http://ecee.colorado.edu/~ecen5623/ecen/labs/Linux/IS/Report.pdf

2. Srinivas Panchpakesan’s work on the Leopard camera.