Android for Computer Vision Apps

CS 202-1-4171

Computer Science Department, BGU

Advanced Topic in Computational Vision

Mini-Project in Mobile Vision

1

Camera –The Basics

• Capturing images and video through the CameraAPI or camera Intent.

• Relevant Classes:

– Camera: API for controlling device camera.

– SurfaceView: Present live camera preview to the user

– Intent: Capture images\videos without directly using the camera object.

2

Camera – Manifest Declaration• Camera Permission:

• Camera Features:

• Storage Permission:

• Audio Recording Permission:

3

<uses-permission android:name="android.permission.CAMERA" />

<uses-feature android:name="android.hardware.autofocus " />

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

<uses-permission android:name="android.permission.RECORD_AUDIO" />

Using Existing Camera Apps

• Invoking a camera intent:

–Compose a Camera Intent

• MediaStore.ACTION_IMAGE_CAPTURE

• MediaStore.ACTION_VIDEO_CAPTURE

– Start the Camera Intent

• startActivityForResult()

–Receive the Intent Result

• onActivityResult()

4

Image Capture Intent

private static final int CAPTURE_IMAGE_ACTIVITY_REQUEST_CODE = 100;private Uri fileUri;

@Overridepublic void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);setContentView(R.layout.main);

// create Intent to take a picture and return control to the calling applicationIntent intent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);

// create a file to save the image fileUri = getOutputMediaFileUri(MEDIA_TYPE_IMAGE); intent.putExtra(MediaStore.EXTRA_OUTPUT, fileUri); // set the image file name

// start the image capture IntentstartActivityForResult(intent, CAPTURE_IMAGE_ACTIVITY_REQUEST_CODE);

}

5

Video Capture Contentprivate static final int CAPTURE_VIDEO_ACTIVITY_REQUEST_CODE = 200;private Uri fileUri;

@Overridepublic void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);setContentView(R.layout.main);//create new IntentIntent intent = new Intent(MediaStore.ACTION_VIDEO_CAPTURE);

// create a file to save the video fileUri = getOutputMediaFileUri(MEDIA_TYPE_VIDEO); intent.putExtra(MediaStore.EXTRA_OUTPUT, fileUri); // set the image file name// set the video image quality to high

intent.putExtra(MediaStore.EXTRA_VIDEO_QUALITY, 1); // start the Video Capture IntentstartActivityForResult(intent, CAPTURE_VIDEO_ACTIVITY_REQUEST_CODE);

}

6

Receiving camera intent resultprivate static final int CAPTURE_IMAGE_ACTIVITY_REQUEST_CODE = 100;private static final int CAPTURE_VIDEO_ACTIVITY_REQUEST_CODE = 200;

@Overrideprotected void onActivityResult(int requestCode, int resultCode, Intent data) {

if (requestCode == CAPTURE_IMAGE_ACTIVITY_REQUEST_CODE) {if (resultCode == RESULT_OK) {

// Image captured and saved to fileUri specified in the IntentToast.makeText(this, "Image saved to:\n" + data.getData(), Toast.LENGTH_LONG).show();

} else if (resultCode == RESULT_CANCELED) {// User cancelled the image capture

} else {// Image capture failed, advise user

}}

if (requestCode == CAPTURE_VIDEO_ACTIVITY_REQUEST_CODE) {if (resultCode == RESULT_OK) {

// Video captured and saved to fileUri specified in the IntentToast.makeText(this, "Video saved to:\n" +

data.getData(), Toast.LENGTH_LONG).show();} else if (resultCode == RESULT_CANCELED) {

// User cancelled the video capture} else {

// Video capture failed, advise user}

}}

7

8

Example: Hello Camera

• Create an application that launches the built-in Camera and use the results for displaying the captured image.

Example: Hello Camera

• Make a LinearLayout with

– a TextView,

– a Button, and

– an ImageView

• Create a handler for the OnClick event of the Button

9

10

Launch Camera

Intent cameraIntent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);

Creating an Intent, as follows, within your button click handler:

startActivityForResult(cameraIntent, CAMERA_PIC_REQUEST);

Launch camera activity and asking for result:

private static final int CAMERA_PIC_REQUEST = 1;

Define within your application as the request code returned by the Camera image capture Intent

Handling result from Camera

11

protected void onActivityResult(int requestCode, int resultCode, Intent data){

if (requestCode == CAMERA_PIC_REQUEST) {

// display imageBitmap thumbnail = (Bitmap) data.getExtras().get("data");ImageView imageTakePic = (ImageView)

findViewById(R.id.imageViewPhone); imageTakePic.setImageBitmap(thumbnail);

}}

Adding onActivityResult() in your Activity:

Enforcing Device Requirement

12

<?xml version="1.0" encoding="utf-8"?><manifest xmlns:android="http://schemas.android.com/apk/res/android"

package="ucab.test.camera"android:versionCode="1"android:versionName="1.0">

<uses-sdk android:minSdkVersion="7" />

<uses-feature android:name="android.hardware.camera"></uses-feature>

<application android:icon="@drawable/icon" android:label="@string/app_name"><activity android:name=".AndroidCameraActivity"

android:label="@string/app_name"><intent-filter>

::

Application wants a cameraIt is not enforced by Android

Building a Camera App

• Detect and Access Camera

• Create a Preview Class

• Build a Preview Layout

• Setup Listeners for Capture

• Capture and Save Files

• Release the Camera

13

Detecting Camera Hardware

• PackageManager.hasSystemFeature()

• Camera.getNumberOfCameras()

14

Accessing the Camera/** A safe way to get an instance of the Camera object. */

public static Camera getCameraInstance(){Camera c = null;try {

c = Camera.open(); // attempt to get a Camera instance

}catch (Exception e){

// Camera is not available (in use or does not exist)}return c; // returns null if camera is unavailable

}15

Creating a Preview Class

/** A basic Camera preview class */public class CameraPreview extends SurfaceView implements SurfaceHolder.Callback {

private SurfaceHolder mHolder;private Camera mCamera;

public CameraPreview(Context context, Camera camera) {super(context);mCamera = camera;

// Install a SurfaceHolder.Callback so we get notified when the// underlying surface is created and destroyed.mHolder = getHolder();mHolder.addCallback(this);

}

16

Creating a Preview Class

public void surfaceCreated(SurfaceHolder holder) {// The Surface has been created, tell the camera where to draw the preview.

try {mCamera.setPreviewDisplay(holder);mCamera.startPreview();

} catch (IOException e) {Log.d(TAG, "Error setting camera preview: " + e.getMessage());

}}

public void surfaceDestroyed(SurfaceHolder holder) {// empty. Take care of releasing the Camera preview in your activity.

}

17

Placing Preview in a Layout<?xml version="1.0" encoding="utf-8"?><LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"

android:orientation="horizontal"android:layout_width="fill_parent"android:layout_height="fill_parent">

<FrameLayoutandroid:id="@+id/camera_preview"android:layout_width="fill_parent"android:layout_height="fill_parent"android:layout_weight="1"/>

<Buttonandroid:id="@+id/button_capture"android:text="Capture"android:layout_width="wrap_content"android:layout_height="wrap_content"android:layout_gravity="center"/>

</LinearLayout> 18

Activity for the Camera Viewpublic class CameraActivity extends Activity {

private Camera mCamera;private CameraPreview mPreview;

@Overridepublic void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);setContentView(R.layout.main);

// Create an instance of CameramCamera = getCameraInstance();

// Create our Preview view and set it as the content of our activity.mPreview = new CameraPreview(this, mCamera);FrameLayout preview = (FrameLayout)

findViewById(id.camera_preview);preview.addView(mPreview);

}} 19

Setup Listeners

// Add a listener to the Capture buttonButton captureButton = (Button) findViewById(id.button_capture);

captureButton.setOnClickListener(new View.OnClickListener() {@Overridepublic void onClick(View v) {

// get an image from the cameramCamera.takePicture(null, null, mPicture);

}}

);

20

Camera.takePicture()private PictureCallback mPicture = new PictureCallback() {

@Overridepublic void onPictureTaken(byte[] data, Camera camera) {

File pictureFile = getOutputMediaFile(MEDIA_TYPE_IMAGE);if (pictureFile == null){

Log.d(TAG, "Error creating media file, check storage permissions: " +e.getMessage());

return;}

try {FileOutputStream fos = new FileOutputStream(pictureFile);fos.write(data);fos.close();

} catch (FileNotFoundException e) {Log.d(TAG, "File not found: " + e.getMessage());

} catch (IOException e) {Log.d(TAG, "Error accessing file: " + e.getMessage());

}}

};

21

Camera Features

• Face Detection

• Focus Areas

• Multiple Cameras

• Zoom

• GPS Data

• Scene Mode

• Using Camera.Parameters Object by Calling the getParameters() Method.

22

OpenCV

• OpenCV (Open Source Computer Vision) is a library of programming functions for real time computer vision.

• Over 2500 optimized algorithms written in C with C\C++ interface.

• Wrappers for langauage C#, Phyton and Java.

• Free for both academic and commercial use.

23

http://opencv.willowgarage.com/wiki/

OpenCV

24

OpenCV

• OpenCV library is packed as a ready-for-use Android Library Project (OpenCV 2.3.1 Android).

• Download OpenCV 2.3.1 Android bin

• Put your App+OpenCV in a single workplace

• Add reference to OpenCV library

• Import OpenCV project

25

http://sourceforge.net/projects/opencvlibrary/files/opencv-android

OpenCV+NDK

• OpenCV in C++ (without OpenCV Java API)

• Integrate native code within your Android application (NDK).

• Access the native code via the Java Native Interface (JNI).

26

http://opencv.itseez.com/doc/tutorials/introduction/android_binary_package/android_binary_package_using_with_NDK.html#android-binary-package-with-ndk

org.opencv.core

• public Mat(int rows, int cols, int type)

• public Point(double x, double y)

• public Point3(double x, double y, double z)

• public Size(double width, double height)

• public Rect(int x, int y, int width, int height)

27

org.opencv.imgproc

• Canny(Mat image, Mat edges, double threshold1, double threshold2)

• GaussianBlur(Mat src, Mat dst, Size ksize, double sigma1)

• cvtColor(Mat src, Mat dst, int code)

• resize(Mat src, Mat dst, Size dsize)28

Org.opencv.features2d

• SIFT

• SURF

• HARRIS

• FAST

• STAR

29

Org.opencv.ml

• CvSVM()

• CvKNearest()

• CvNormalBayesClassifier()

• CvANN_MLP

30

Org.opencv.objdetect

• Cascade Classifier

• HOG Descriptor

31

Org.opencv.video

• BackgroundSubtractorMOG

• Kalman filter

32

Projects (just Ideas)

• Optical Character Recognition (OCR) for License Plate/Business Card/Text Translation

• Face/Object Recognition

• Sudoku/Puzzle Solver

• Gesture Recognition

• Depth Estimation

• Vehicle Safety

• Augmented Reality

33