writing apps that can see: getting data from coreimage to computer vision - 360idev 2013...

Report

Post on 09-May-2015

3.312 Views

Category:

Technology

0 Downloads

Preview:

Click to see full reader

DESCRIPTION

Apps that Can See: Getting Data from CoreImage to Computer Vision You hear about the app that can solve a Rubik's Cube after you take pictures of each side? How about the one that can import Sudoku puzzles by letting you take a picture of a puzzle? Wouldn't it be cool if your apps could get information that way? Come and learn how to integrate image processing in your apps. We'll start with Apple's built-in CoreImage libraries, and then move on to open-source C++ libraries that let you detect colors, shapes, letters and numbers. We'll go through quite a bit of sample code that you can take with you to use in your apps. We'll also talk about practical tips and experiences, like how to debug and troubleshoot code when things aren't working the way you expected.

TRANSCRIPT

#360iDev@CarlBrwn

Apps That Can SeeCarl Brown

Missing Last Vowel -I blame Flickr/Tumblr

1Monday, September 9, 13

@CarlBrwn #360iDev

Computer Vision?Turning Photos Into Useable Data

Image Analysis

Colors

Shapes

Numbers & Letters

2Monday, September 9, 13

@CarlBrwn #360iDev

Word Lens

http://www.youtube.com/watch?v=h2OfQdYrHRs

3Monday, September 9, 13

@CarlBrwn #360iDev

Word Lens

http://www.youtube.com/watch?v=h2OfQdYrHRs

3Monday, September 9, 13

@CarlBrwn #360iDev

Today's TalkHow quick and easy it is to get actionable info from photos

4Monday, September 9, 13

@CarlBrwn #360iDev

Today's TalkHow quick and easy it is to get actionable info from photos

Demo Code with Source

4Monday, September 9, 13

@CarlBrwn #360iDev

Today's TalkHow quick and easy it is to get actionable info from photos

Demo Code with Source

Not about HOW OpenCV works

4Monday, September 9, 13

@CarlBrwn #360iDev

Today's TalkHow quick and easy it is to get actionable info from photos

Demo Code with Source

Not about HOW OpenCV works

"OpenCV by Example"

4Monday, September 9, 13

@CarlBrwn #360iDev

CubeCheater &Sudoku Grab

http://www.youtube.com/watch?v=HNwx0nbgm7M http://www.youtube.com/watch?v=wt96OomJY9A

5Monday, September 9, 13

@CarlBrwn #360iDev

Both have had Legal Problems

http://cubecheater.efaller.com http://sudokugrab.blogspot.com/2010/12/sudoku-grab-removed-from-sale-in-japan.html

6Monday, September 9, 13

#360iDev@CarlBrwn

Based onCode From:github.com/carlbrown/OpenCVDemo360iDev2013

7Monday, September 9, 13

@CarlBrwn #360iDev

About the CodeExplicit and Redundant to make it easier to follow/explain

8Monday, September 9, 13

@CarlBrwn #360iDev

About the CodeExplicit and Redundant to make it easier to follow/explain

Not production ready - for teaching purposes

8Monday, September 9, 13

@CarlBrwn #360iDev

About the CodeExplicit and Redundant to make it easier to follow/explain

Not production ready - for teaching purposes

Minimum Viable Interface

8Monday, September 9, 13

@CarlBrwn #360iDev

About the CodeExplicit and Redundant to make it easier to follow/explain

Not production ready - for teaching purposes

Minimum Viable Interface

Don't hate, okay?

8Monday, September 9, 13

#360iDev@CarlBrwn

Face CounterDemo 1/3

9Monday, September 9, 13

#360iDev@CarlBrwn

Core ImageOption 1

10Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- coreImageButtonPressed:

[CIImage imageWithCGImage:]Detect

11Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- coreImageButtonPressed:

[CIImage imageWithCGImage:][CIDetector detectorOfType:! ! ! ! CIDetectorTypeFace]

Detect

11Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- coreImageButtonPressed:

[CIImage imageWithCGImage:][CIDetector detectorOfType:! ! ! ! CIDetectorTypeFace]NSArray *features= [detector ! ! ! ! ! ! featuresInImage:]

Detect

11Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- coreImageButtonPressed:

[CIImage imageWithCGImage:][CIDetector detectorOfType:! ! ! ! CIDetectorTypeFace]NSArray *features= [detector ! ! ! ! ! ! featuresInImage:]for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)

Detect

11Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- coreImageButtonPressed:

for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)

Debugging

12Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- coreImageButtonPressed:

for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.bounds

Debugging

12Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- coreImageButtonPressed:

for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.boundsframe *= contentScaleFactor

Debugging

12Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- coreImageButtonPressed:

for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.boundsframe *= contentScaleFactor//flip to CIImage Coords

Debugging

12Monday, September 9, 13

@CarlBrwn #360iDev

Coordinate Systems

UIView has (0,0) at Upper Left

Core Image has (0,0) Lower Left

(Like OS X)

13Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- coreImageButtonPressed:

for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.boundsframe *= contentScaleFactor//flip to CIImage Coords

Debugging

14Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- coreImageButtonPressed:

for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.boundsframe *= contentScaleFactor//flip to CIImage CoordsfaceView.layer.borderWidth = 4;

Debugging

14Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- coreImageButtonPressed:

for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.boundsframe *= contentScaleFactor//flip to CIImage CoordsfaceView.layer.borderWidth = 4;addSubview:faceView

Debugging

14Monday, September 9, 13

#360iDev@CarlBrwn

Core ImageOption 1

15Monday, September 9, 13

#360iDev@CarlBrwn

OpenCVOption 2

16Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- openCVButtonPressed:

convert to cv::MatDetect

17Monday, September 9, 13

@CarlBrwn #360iDev

Convert to cv::MatCGImageGetColorSpace(image.CGImage)

cv::Mat cvMat()

CGBitmapContextCreate(cvMat.data,...)

CGContextDrawImage(ctx,r,image.CGImage)

CGContextRelease()

18Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- openCVButtonPressed:

convert to cv::MatDetect

19Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- openCVButtonPressed:

convert to cv::Matcv::cvtColor( CV_BGR2GRAY )

Detect

19Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- openCVButtonPressed:

convert to cv::Matcv::cvtColor( CV_BGR2GRAY )cv::equalizeHist()

Detect

19Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- openCVButtonPressed:

convert to cv::Matcv::cvtColor( CV_BGR2GRAY )cv::equalizeHist()cv::face_cascade.\! ! ! ! ! ! detectMultiScale()

Detect

19Monday, September 9, 13

@CarlBrwn #360iDev

Cascade Classifierhttp://docs.opencv.org/doc/tutorials/objdetect/cascade_classifier/cascade_classifier.html

20Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- openCVButtonPressed:

convert to cv::Matcv::cvtColor( CV_BGR2GRAY )cv::equalizeHist()face_cascade.detectMultiScale()

Detect

21Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- openCVButtonPressed:

convert to cv::Matcv::cvtColor( CV_BGR2GRAY )cv::equalizeHist()face_cascade.detectMultiScale()for( int i = 0; ! ! ! ! i < faces.size(); i++ )

Detect

21Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- openCVButtonPressed:

for( int i = 0; ! ! ! ! i < faces.size(); i++ )

Debugging

22Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- openCVButtonPressed:

for( int i = 0; ! ! ! ! i < faces.size(); i++ )CGRectMake(faces[i].frame)

Debugging

22Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- openCVButtonPressed:

for( int i = 0; ! ! ! ! i < faces.size(); i++ )CGRectMake(faces[i].frame)frame *= contentScaleFactor

Debugging

22Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- openCVButtonPressed:

for( int i = 0; ! ! ! ! i < faces.size(); i++ )CGRectMake(faces[i].frame)frame *= contentScaleFactorfaceView.layer.borderWidth = 4;

Debugging

22Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- openCVButtonPressed:

for( int i = 0; ! ! ! ! i < faces.size(); i++ )CGRectMake(faces[i].frame)frame *= contentScaleFactorfaceView.layer.borderWidth = 4;addSubview:faceView

Debugging

22Monday, September 9, 13

#360iDev@CarlBrwn

OpenCVOption 2

23Monday, September 9, 13

#360iDev@CarlBrwn

CubeFaceGrabberDemo 2/3

24Monday, September 9, 13

#360iDev@CarlBrwn

DetectEdgesdetectFacesPressed:

25Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- detectFacesPressed:

convert to cv::MatDetect

26Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- detectFacesPressed:

convert to cv::Mat

cv::Canny()

Detect

26Monday, September 9, 13

@CarlBrwn #360iDev

Canny27Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- detectFacesPressed:

convert to cv::Mat

cv::Canny()

Detect

28Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- detectFacesPressed:

convert to cv::Mat

cv::Canny()cv::HoughLinesP()

Detect

28Monday, September 9, 13

@CarlBrwn #360iDev

HoughLinesP29Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- detectFacesPressed:

convert to cv::Mat

cv::Canny()cv::HoughLinesP()

Detect

30Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- detectFacesPressed:

convert to cv::Mat

cv::Canny()cv::HoughLinesP()for( size_t i = 0; ! ! ! i < lines.size(); i++ )

Detect

30Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- detectFacesPressed:

convert to cv::Mat

cv::Canny()cv::HoughLinesP()for( size_t i = 0; ! ! ! i < lines.size(); i++ )

if (LineIntersect(l, l2))

Detect

30Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- detectFacesPressed:

convert to cv::Mat

cv::Canny()cv::HoughLinesP()for( size_t i = 0; ! ! ! i < lines.size(); i++ )

if (LineIntersect(l, l2))//Measure Distance to Image corners

Detect

30Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- detectFacesPressed:

for( size_t i = 0; ! ! ! i < lines.size(); i++ )

Debugging

31Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- detectFacesPressed:

for( size_t i = 0; ! ! ! i < lines.size(); i++ )cv::line() //Draw

Debugging

31Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- detectFacesPressed:

for( size_t i = 0; ! ! ! i < lines.size(); i++ )cv::line() //Draw//Check for bounding corners

Debugging

31Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- detectFacesPressed:

for( size_t i = 0; ! ! ! i < lines.size(); i++ )cv::line() //Draw//Check for bounding cornerscv::circle() //Draw

Debugging

31Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- detectFacesPressed:

for( size_t i = 0; ! ! ! i < lines.size(); i++ )cv::line() //Draw//Check for bounding cornerscv::circle() //Drawconvert to UIImage //Display

Debugging

31Monday, September 9, 13

@CarlBrwn #360iDev

Convert to UIImage

[NSData dataWithBytes:cvMat.data length:]

CGDataProviderCreateWithCFData()

CGImageCreate()

[[UIImage alloc] initWithCGImage:]

CGXXXRelease()

32Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- detectFacesPressed:

for( size_t i = 0; ! ! ! i < lines.size(); i++ )cv::line() //Draw//Check for bounding cornerscv::circle() //Drawconvert to UIImage //Display

Debugging

33Monday, September 9, 13

#360iDev@CarlBrwn

DetectEdgesdetectFacesPressed:

34Monday, September 9, 13

#360iDev@CarlBrwn

UnskewgetAndApplyTransformPressed:

35Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- getAndApplyTransformPressed:

convert to cv::Mat

36Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- getAndApplyTransformPressed:

convert to cv::MatgetPerspectiveTransform()

36Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- getAndApplyTransformPressed:

convert to cv::MatgetPerspectiveTransform()warpPerspective(transform)

36Monday, September 9, 13

@CarlBrwn #360iDev

Map the Corners circled on the Left

To the Corners of the Images on the Right

37Monday, September 9, 13

@CarlBrwn #360iDev

Map the Corners circled on the Left

To the Corners of the Images on the Right

37Monday, September 9, 13

@CarlBrwn #360iDev

Map the Corners circled on the Left

To the Corners of the Images on the Right

37Monday, September 9, 13

@CarlBrwn #360iDev

Map the Corners circled on the Left

To the Corners of the Images on the Right

37Monday, September 9, 13

@CarlBrwn #360iDev

Map the Corners circled on the Left

To the Corners of the Images on the Right

37Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- getAndApplyTransformPressed:

convert to cv::MatgetPerspectiveTransform()warpPerspective(transform)

38Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- getAndApplyTransformPressed:

convert to cv::MatgetPerspectiveTransform()warpPerspective(transform)convert to UIImage //Display

38Monday, September 9, 13

#360iDev@CarlBrwn

UnskewgetAndApplyTransformPressed:

39Monday, September 9, 13

#360iDev@CarlBrwn

ExtractColorsextractColorsPressed:

40Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractColorsPressed:

convert to cv::Matcv::cvtColor(CV_RGB2HSV)

Detect

41Monday, September 9, 13

@CarlBrwn #360iDev

HSV Space

Hue/Saturation/Value

Easier to get Color out

http://en.wikipedia.org/wiki/HSV_color_space42Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractColorsPressed:

convert to cv::Matcv::cvtColor(CV_RGB2HSV)

Detect

43Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractColorsPressed:

convert to cv::Matcv::cvtColor(CV_RGB2HSV)for (int hSlice;;) { ! ! for (int vSlice;;) {

Detect

43Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractColorsPressed:

convert to cv::Matcv::cvtColor(CV_RGB2HSV)for (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()

Detect

43Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractColorsPressed:

convert to cv::Matcv::cvtColor(CV_RGB2HSV)for (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)

Detect

43Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractColorsPressed:

convert to cv::Matcv::cvtColor(CV_RGB2HSV)for (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)avgColor=cv::mean(roi)

Detect

43Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractColorsPressed:

convert to cv::Matcv::cvtColor(CV_RGB2HSV)for (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)avgColor=cv::mean(roi)//Check Saturation and Hue

Detect

43Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractColorsPressed:

for (int hSlice;;) { ! ! for (int vSlice;;) {

Debugging

44Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractColorsPressed:

for (int hSlice;;) { ! ! for (int vSlice;;) { roi = avgColor //Fill

Debugging

44Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractColorsPressed:

for (int hSlice;;) { ! ! for (int vSlice;;) { roi = avgColor //FillputText(roi, [color cString]

Debugging

44Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractColorsPressed:

for (int hSlice;;) { ! ! for (int vSlice;;) { roi = avgColor //FillputText(roi, [color cString]putText(roi, stringWithFormat:@"%.0f/%.0f/%.0f")

Debugging

44Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractColorsPressed:

for (int hSlice;;) { ! ! for (int vSlice;;) { roi = avgColor //FillputText(roi, [color cString]putText(roi, stringWithFormat:@"%.0f/%.0f/%.0f")

convert to UIImage //Display

Debugging

44Monday, September 9, 13

#360iDev@CarlBrwn

ExtractColorsextractColorsPressed:

45Monday, September 9, 13

#360iDev@CarlBrwn

SudokuGrabberDemo 3/3

46Monday, September 9, 13

#360iDev@CarlBrwn

FindEdgesfindPuzzlePressed:

47Monday, September 9, 13

@CarlBrwn #360iDev

Just like Cube

48Monday, September 9, 13

#360iDev@CarlBrwn

UnSkewPuzzlesquarePressed:

49Monday, September 9, 13

@CarlBrwn #360iDev

Just like Cube

50Monday, September 9, 13

#360iDev@CarlBrwn

ExtractTextextractPressed:

51Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractPressed:

Tesseract* tesseract =Setup

52Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractPressed:

Tesseract* tesseract =initWithDataPath:@"tessdata" language:@"eng"

Setup

52Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractPressed:

Tesseract* tesseract =initWithDataPath:@"tessdata" language:@"eng"tesseract setVariableValue:@"0123456789"forKey:@"tessedit_char_whitelist"

Setup

52Monday, September 9, 13

@CarlBrwn #360iDev

tesseract-ocrhttp://code.google.com/p/tesseract-ocr/

53Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractPressed:

convert to cv::MatSlice

54Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractPressed:

convert to cv::Matfor (int hSlice;;) { ! ! for (int vSlice;;) {

Slice

54Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractPressed:

convert to cv::Matfor (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()

Slice

54Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractPressed:

convert to cv::Matfor (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)

Slice

54Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractPressed:

convert to cv::Matfor (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)cv::cvtColor(CV_RGB2GRAY);

Slice

54Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractPressed:

convert to cv::Matfor (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)cv::cvtColor(CV_RGB2GRAY);cv::threshold(roi_gray)

Slice

54Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractPressed:

convert slice to UIImage Detect

55Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractPressed:

convert slice to UIImage tesseract setImage:imageToParse

Detect

55Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractPressed:

convert slice to UIImage tesseract setImage:imageToParsetesseract recognize

Detect

55Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractPressed:

[[UILabel alloc]initWithFrame:CGRectMake()]

Debugging

56Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractPressed:

[[UILabel alloc]initWithFrame:CGRectMake()] labelsetText:recognizedString

Debugging

56Monday, September 9, 13

@CarlBrwn #360iDev

ViewController.mm- extractPressed:

[[UILabel alloc]initWithFrame:CGRectMake()] labelsetText:recognizedStringself.viewaddSubview:recognizedLabel

Debugging

56Monday, September 9, 13

#360iDev@CarlBrwn

ExtractTextextractPressed:

57Monday, September 9, 13

#360iDev@CarlBrwn

Based onCode From:github.com/carlbrown/OpenCVDemo360iDev2013

Questions?

58Monday, September 9, 13

top related

mobile marketing apps: getting prepared to drive conversions
Technology
deep learning on ios #360idev
Technology
getting toys to play with apps--past present & future
Technology
gpu programming 360idev
Technology
getting to grips with sharepoint 2013 apps - chris o'brien
Documents
automating your workflow - daniel pfeiffer at 360idev 2016
Software
getting started on packaging apps with open build service
Technology
mobile apps - getting your idea across
Design
getting googly with google apps for education
Documents
getting started with serverless apps
Documents
progressive web apps - overview & getting started
Technology
getting started vmware apps
Technology
getting started with progressive web apps
Technology
building apps with phonefare : getting started
Technology
[apps electronics]forrest m. mims - getting started in...
Documents
getting started: designing html5 web apps
Technology
getting started developing universal windows platform (uwp)...
Technology
360idev - gregory kennedy
Technology
google apps script: getting the most out of google apps
Technology
pydio white and grey labelled apps - getting started
Technology