creating a face-recognising security cam with a raspberry pi part 2

7/28/2019 Creating a Face-recognising Security Cam With a Raspberry Pi Part 2

1/11

Through the Interface

September 19, 201 2

Creating a face-recognising security cam with a Raspberry Pi Part 2

In the first post in this series, we introduced the idea behind the Facecam, a Facebook-enabled security camera

running on a Raspberry Pi device.

Over the next three posts were going to look in more detail at the systems co mponents. In this post, were gointo start by looking at the co mponent the only one that doesnt actually run on the RaspberryPi that

do wnloads the information from Faceboo k to build the facial reco gnition database to be used by the dev ice.

As mentioned last time, Iv e coded this as a .NET application actually by ex tending the WinForms sample

provided with the Facebo ok C# SDK in order to dedicate the Raspberry Pi to performing the security cam

function. Im not actually going to share the co de for this application, as its really a work in progress. The co de

itself isnt very co mplicated the Facebo ok piece is mostly ex ecuting FQL, which allows y ou to use a SQL-like

syntax to query the Facebo ok Graph API and Ill be descr ibing the techniques it uses should people want to g

ahead and roll their own.

Before loo king at the behaviour of the application itself, its important to understand more about this databasethat its going to help build.

The facial reco gnition co de Iv e adopted is based on work by Shervin Emami and uses the Eigenface algorithm t

co mpare detec ted faces against a database. To understand more about Eigenface, this article pro vides a

thorough explanation.

Shervins OnlineFaceRec sample makes use o fOpenCV for this process. Ive borrowed co de from this sample fo

use on the Raspberry Pi, but today s application simply builds a file and folder structure that can be used by the

OnlineFaceRec application to train the database (and essentially build the facedata.xml file well transfer acros

to the Raspberry Pi).

There are two main parts of this proc ess: downloading the friend data from Facebo ok and then parsing that daton disk in order to create a training script to be passed to OnlineFaceRec.

These two functions are reflected by the basic UI of the .NET application:

Clicking on the first button lets us log in to Facebo ok and accept the permissions requested by the .NET
http://through-the-interface.typepad.com/.a/6a00d83452464869e2017d3bff2450970c-pihttp://through-the-interface.typepad.com/.a/6a00d83452464869e2017d3bff2450970c-pihttp://through-the-interface.typepad.com/.a/6a00d83452464869e2017d3bff2450970c-pihttp://through-the-interface.typepad.com/.a/6a00d83452464869e2017d3bff2450970c-pihttp://through-the-interface.typepad.com/.a/6a00d83452464869e2017d3bff2450970c-pihttp://through-the-interface.typepad.com/.a/6a00d83452464869e2017d3bff2450970c-pihttp://through-the-interface.typepad.com/.a/6a00d83452464869e2017d3bff2450970c-pihttp://through-the-interface.typepad.com/.a/6a00d83452464869e2017d3bff2450970c-pihttp://through-the-interface.typepad.com/.a/6a00d83452464869e2017d3bff2450970c-pihttp://www.shervinemami.info/faceRecognition.htmlhttp://en.wikipedia.org/wiki/Eigenfacehttp://through-the-interface.typepad.com/.a/6a00d83452464869e2017d3bff2450970c-pihttp://www.cognotics.com/opencv/servo_2007_series/part_4/index.htmlhttp://en.wikipedia.org/wiki/Eigenfacehttp://www.shervinemami.info/faceRecognition.htmlhttps://developers.facebook.com/docs/reference/api/https://developers.facebook.com/docs/reference/fql/http://csharpsdk.org/https://github.com/facebook-csharp-sdk/facebook-winforms-samplehttp://through-the-interface.typepad.com/through_the_interface/2012/09/creating-a-face-recognising-security-cam-with-a-raspberry-pi-part-1.htmlhttp://through-the-interface.typepad.com/through_the_interface/


2/11

application. Which reminds me: to create an application that makes use of the Facebook A PI, youll need to

register it with Facebook and include the AppId in y our co de. As y ou c an see below, I c alled my app

FriendExtractor:

Once were lo gged in, we see the main dialog which will help us drive the download proc ess:

Clicking on the Populate Friends button pulls down the list of the current users friends from Faceboo k and

populates the list on the left:
http://through-the-interface.typepad.com/.a/6a00d83452464869e2017744ae6a65970d-pihttp://through-the-interface.typepad.com/.a/6a00d83452464869e2017c31d0a3da970b-pihttp://facebook.com/developers


3/11

Y ou can se e the list has checkbo xes asso ciated with each item. Originally the implementation didnt allo w

selection o f friends to include, which ended in a huge amount of data being downloaded (it turns out I hav e

friends who seem to do no thing other than upload their pictures to Facebo ok and have 1 000+ tagged images u

there). Running OnlineFaceRec to train the database with the results ended up in a 400 MB facedata.xml file

be ing c reated. Giv en the fact the R-Pi has 25 6MB of RAM< I figured that was a no n-starter. :-)

I therefore enabled selec tion of friends that are actually likely to v isit our home or that I may end up demoin

this to at AU which greatly reduc es the amount of probably redundant data that gets downloaded and

processed. The list of selected friends gets saved in a simple text file and reloaded automatically.

I also added the ability to start do wnloading from a particular friend, to allow incremental additions/updates.

The first time its run, its certainly wo rth starting from the beginning, of co urse, but y ou may want to restart

from a friend further down the list a later point.

When y ou selec t a friend o n the le ft, the Start Download from Selected butto n should c ome aliv e. Pushing it

does whats written on the box :
http://through-the-interface.typepad.com/.a/6a00d83452464869e2017c31d0a3fa970b-pi


4/11

Once c ompleted, y oull have a folder full of photos of you and y our friends. The folder structure contains one

level of indirection to use the user I D as friends can v ery well have the same name (theres a teenager named

Kean Walmsley who lives in Canada and friended me o n Facebo ok some time ago, for instance) and inside tha

folder y oull find a collec tion of images:
http://through-the-interface.typepad.com/.a/6a00d83452464869e2017744ae6a93970d-pi


5/11

Looking closely, abo ve, y oull see there are both .jpg and .pgm images. The .jpg files are those downloaded

directly from Faceboo k nothing surprising there. The .pgm files are c reated by the .NET application using

following algorithm:

For each tagged photo o f a user

Use OpenCV face detection to find the faces in the image

For this I used Emgu CV a .NET wrapper for OpenCV along with OpenCVs face detection featur

that looks for haar-like features

Compare the tag location with the faces returned. For the best match

Take a greyscale copy of the cropped area

Resize it to 50 x 5 0

Equalize the histogram to get c onsistent brightness and contrast

Save that file to the .pgm format

Regarding the quality o f the photos downloaded from Facebook: frankly, it varies. Some pho tos are po orly

tagged, and some are tagged well enough but the face c annot easily be ex tracted (which sometimes means

someone elses gets picked up in its place). A nd then some pho tos simply arent well suited to facedetection/ex traction, so the results end up being just plain strange.

Iv e found that some kind of manual scrubbing process between the download and the script creation/database

training helps the quality of the data a great deal. Ive used a tool called Ir fanView to do this: y ou can searc h a

set of sub-folders for *.pgm files and transfer the results acro ss into Thumbnail View, and c an then proc eed to

delete the ugly pics from there. Of course y ou could do the same thing in Explo rer if you had a shell extension

that can prev iew .pgm files, but I wasnt able to find one. A nd IrfanView seems to wo rk pretty well.

Heres a v iew of some shots o f Sco tt Sheppard and Shaan Hurley , for instance:
http://www.irfanview.net/http://opencv.willowgarage.com/wiki/FaceDetection/http://www.emgu.com/http://through-the-interface.typepad.com/.a/6a00d83452464869e2017d3bff24c3970c-pi


6/11

In time I expec t a bespoke tool one that remembers the pics youve deleted before, to save yo u having to

repeat that task each time is the way to go. But thats for another time.

Once this is done, y ou should hav e a fairly c lean set of normalised (i.e. of the same size and with similar

br ightness/ contrast) files that can be used to generate a training script for the OnlineFac eRec tool.

The second button o n our main dialog runs some co de that copies the .pgm files to a separate folder and create

a training script listing them all:

privatevoid btnCreateScript_Click(object sender, EventArgs e)

{

conststring root = "Z:\\fb_photos\\"; conststring destRoot = "Z:\\rp_photos\\";

SortedList checkedFriends =

InfoDialog.LoadCheckedFriends();

// Get the list of PGM files in our source directory

DirectoryInfo di = newDirectoryInfo(root);

var files = di.GetFiles("*.pgm", SearchOption.AllDirectories);

StringBuilder sb = newStringBuilder(),

cur = newStringBuilder();

// We'll maintain an index for the persons and an

// old user ID, so we can tell when we have changed

// to a new person's data

int person = 0;

string ouid = "";

// Maintain a counter of the number of photos for

// the current person (we only write out 2+ photos,

// as otherwise they don't train the database),

// as well as a list of their paths
http://through-the-interface.typepad.com/.a/6a00d83452464869e2017d3bff24f6970c-pi


7/11

int photosForCurrent = 0;

List filesForCurrent = newList();

foreach (FileInfo fi in files)

{

var path = fi.DirectoryName;

var info = path.Split(new[] { '\\' });

if (info.Length == 4)

{

string name = info[2], uid = info[3];

if (ouid != uid)

{

// We have a new user

ouid = uid;

// If the last user had more than 2 photos...

if (photosForCurrent > 2)

{

// Let's add them to the training script and

// copy the files to the destination

person++;

sb.Append(cur.ToString());

foreach (string file in filesForCurrent)

{

string destFile =

destRoot +

Utils.RemoveAccents(file.Substring(root.Length));

Directory.CreateDirectory(

Path.GetDirectoryName(destFile)

);

File.Copy(file, destFile, true);

}

}

// Reset the string, photos and image file info

// for the current user

cur.Clear();

photosForCurrent = 0;

filesForCurrent.Clear();

}

long id = Int64.Parse(uid);

if (checkedFriends.ContainsKey(id))

{

// Add the information on the current photo to

// the current user's string

cur.Append(

String.Format(

"{0} {1} {2}\r\n",

person, Utils.RemoveAccents(name),

"./" +

Utils.RemoveAccents(fi.FullName).

Substring(root.Length).Replace('\\', '/')

)

);


8/11

// Do the same for the photos to copy across to

// the destination

photosForCurrent++;

filesForCurrent.Add(fi.FullName);

}

}

}

// Don't forget to tidy up and copy the last user's

// training string and files across

sb.Append(cur.ToString());

foreach (string file in filesForCurrent)

{

string destFile =

destRoot + file.Substring(root.Length);

Directory.CreateDirectory(

Path.GetDirectoryName(destFile)

);

File.Copy(file, destFile, true);

}

// Finally we write out the string to the training file

File.WriteAllText(

destRoot + "training.txt", sb.ToString()

);

}

In case y oure wondering, Ive kept the two main pieces of the process separate primarily so I can co mbine my

wifes friends with my own be fore tr aining the database. :-)

Now its a simple matter of running OnlineFaceRec with the training script:

The output of this tool is pretty interesting: aside from the v ery importantfacedata.xmlfile (the database well

use on the Raspberry Pi), we also two files that are created mainly to demonstrate the principle at work.

Firstly we have the average face which is the absolute av erage of all the pictures of your friends, and tends to

be a bland, slightly andro gy nous and some might say idealised image o f a human fac e:
http://through-the-interface.typepad.com/.a/6a00d83452464869e2017744ae6af1970d-pi


9/11

And seo ndly we hav e an image o f the Eigenfac es themse lv es, which map the differences from this av erage imag
http://through-the-interface.typepad.com/.a/6a00d83452464869e2017c31d0a4c9970b-pihttp://through-the-interface.typepad.com/.a/6a00d83452464869e2017d3bff2526970c-pi


10/11
http://through-the-interface.typepad.com/.a/6a00d83452464869e2017c31d0a4c9970b-pi


11/11

Creating a face-

recognising

Creating a face-

recognising

Waiting for my

Raspberry Pi

Creating a motion-

detecting security

Neither of these images needs to b e transferred acro ss to the Raspberry Pi their data is captured in the XML

database (which stands at around 28 MB for the selected subset of my friends).

In the next post, well look at the implementation of a co mponent that makes use o f this XML data to perform

facial reco gnition on images captured by o ur motion detecto r.

You m igh t also like:
http://through-the-interface.typepad.com/.a/6a00d83452464869e2017c31d0a4c9970b-pihttp://through-the-interface.typepad.com/through_the_interface/2012/08/creating-a-motion-detecting-security-cam-with-a-raspberry-pi-part-1.htmlhttp://through-the-interface.typepad.com/through_the_interface/2012/08/waiting-for-my-raspberry-pi.htmlhttp://through-the-interface.typepad.com/through_the_interface/2012/10/creating-a-face-recognising-security-cam-with-a-raspberry-pi-part-4.htmlhttp://through-the-interface.typepad.com/through_the_interface/2012/09/creating-a-face-recognising-security-cam-with-a-raspberry-pi-part-1.html

creating a face-recognising security cam with a raspberry pi part 2

Documents