writing apps that can see: getting data from coreimage to computer vision - 360idev 2013...

#360iDev@CarlBrwn

Apps That Can SeeCarl Brown

Missing Last Vowel -I blame Flickr/Tumblr

1Monday, September 9, 13

@CarlBrwn #360iDev

Computer Vision?Turning Photos Into Useable Data

Image Analysis

Colors

Shapes

Numbers & Letters


@CarlBrwn #360iDev

Word Lens

http://www.youtube.com/watch?v=h2OfQdYrHRs


@CarlBrwn #360iDev

Today's TalkHow quick and easy it is to get actionable info from photos


@CarlBrwn #360iDev


Demo Code with Source


@CarlBrwn #360iDev



Not about HOW OpenCV works


@CarlBrwn #360iDev



Not about HOW OpenCV works

"OpenCV by Example"


@CarlBrwn #360iDev

CubeCheater &Sudoku Grab

http://www.youtube.com/watch?v=HNwx0nbgm7M http://www.youtube.com/watch?v=wt96OomJY9A


@CarlBrwn #360iDev

Both have had Legal Problems

http://cubecheater.efaller.com http://sudokugrab.blogspot.com/2010/12/sudoku-grab-removed-from-sale-in-japan.html


#360iDev@CarlBrwn

Based onCode From:github.com/carlbrown/OpenCVDemo360iDev2013


@CarlBrwn #360iDev

About the CodeExplicit and Redundant to make it easier to follow/explain


@CarlBrwn #360iDev


Not production ready - for teaching purposes


@CarlBrwn #360iDev



Minimum Viable Interface


@CarlBrwn #360iDev



Minimum Viable Interface

Don't hate, okay?


#360iDev@CarlBrwn

Face CounterDemo 1/3


#360iDev@CarlBrwn

Core ImageOption 1


@CarlBrwn #360iDev

ViewController.mm- coreImageButtonPressed:

[CIImage imageWithCGImage:]Detect


@CarlBrwn #360iDev


[CIImage imageWithCGImage:][CIDetector detectorOfType:! ! ! ! CIDetectorTypeFace]

Detect


@CarlBrwn #360iDev


[CIImage imageWithCGImage:][CIDetector detectorOfType:! ! ! ! CIDetectorTypeFace]NSArray *features= [detector ! ! ! ! ! ! featuresInImage:]

Detect


@CarlBrwn #360iDev


[CIImage imageWithCGImage:][CIDetector detectorOfType:! ! ! ! CIDetectorTypeFace]NSArray *features= [detector ! ! ! ! ! ! featuresInImage:]for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)

Detect


@CarlBrwn #360iDev


for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)

Debugging


@CarlBrwn #360iDev


for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.bounds

Debugging


@CarlBrwn #360iDev


for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.boundsframe *= contentScaleFactor

Debugging


@CarlBrwn #360iDev


for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.boundsframe *= contentScaleFactor//flip to CIImage Coords

Debugging


@CarlBrwn #360iDev

Coordinate Systems

UIView has (0,0) at Upper Left

Core Image has (0,0) Lower Left

(Like OS X)


@CarlBrwn #360iDev


for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.boundsframe *= contentScaleFactor//flip to CIImage Coords

Debugging


@CarlBrwn #360iDev


for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.boundsframe *= contentScaleFactor//flip to CIImage CoordsfaceView.layer.borderWidth = 4;

Debugging


@CarlBrwn #360iDev


for(CIFaceFeature* faceFeature !! ! ! ! ! ! ! in features)frame = faceFeature.boundsframe *= contentScaleFactor//flip to CIImage CoordsfaceView.layer.borderWidth = 4;addSubview:faceView

Debugging


#360iDev@CarlBrwn

Core ImageOption 1


#360iDev@CarlBrwn

OpenCVOption 2


@CarlBrwn #360iDev

ViewController.mm- openCVButtonPressed:

convert to cv::MatDetect


@CarlBrwn #360iDev

Convert to cv::MatCGImageGetColorSpace(image.CGImage)

cv::Mat cvMat()

CGBitmapContextCreate(cvMat.data,...)

CGContextDrawImage(ctx,r,image.CGImage)

CGContextRelease()


@CarlBrwn #360iDev




@CarlBrwn #360iDev


convert to cv::Matcv::cvtColor( CV_BGR2GRAY )

Detect


@CarlBrwn #360iDev


convert to cv::Matcv::cvtColor( CV_BGR2GRAY )cv::equalizeHist()

Detect


@CarlBrwn #360iDev


convert to cv::Matcv::cvtColor( CV_BGR2GRAY )cv::equalizeHist()cv::face_cascade.\! ! ! ! ! ! detectMultiScale()

Detect


@CarlBrwn #360iDev

Cascade Classifierhttp://docs.opencv.org/doc/tutorials/objdetect/cascade_classifier/cascade_classifier.html


@CarlBrwn #360iDev


convert to cv::Matcv::cvtColor( CV_BGR2GRAY )cv::equalizeHist()face_cascade.detectMultiScale()

Detect


@CarlBrwn #360iDev


convert to cv::Matcv::cvtColor( CV_BGR2GRAY )cv::equalizeHist()face_cascade.detectMultiScale()for( int i = 0; ! ! ! ! i < faces.size(); i++ )

Detect


@CarlBrwn #360iDev


for( int i = 0; ! ! ! ! i < faces.size(); i++ )

Debugging


@CarlBrwn #360iDev


for( int i = 0; ! ! ! ! i < faces.size(); i++ )CGRectMake(faces[i].frame)

Debugging


@CarlBrwn #360iDev


for( int i = 0; ! ! ! ! i < faces.size(); i++ )CGRectMake(faces[i].frame)frame *= contentScaleFactor

Debugging


@CarlBrwn #360iDev


for( int i = 0; ! ! ! ! i < faces.size(); i++ )CGRectMake(faces[i].frame)frame *= contentScaleFactorfaceView.layer.borderWidth = 4;

Debugging


@CarlBrwn #360iDev


for( int i = 0; ! ! ! ! i < faces.size(); i++ )CGRectMake(faces[i].frame)frame *= contentScaleFactorfaceView.layer.borderWidth = 4;addSubview:faceView

Debugging


#360iDev@CarlBrwn

OpenCVOption 2


#360iDev@CarlBrwn

CubeFaceGrabberDemo 2/3


#360iDev@CarlBrwn

DetectEdgesdetectFacesPressed:


@CarlBrwn #360iDev

ViewController.mm- detectFacesPressed:



@CarlBrwn #360iDev


convert to cv::Mat

cv::Canny()

Detect


@CarlBrwn #360iDev

Canny27Monday, September 9, 13

@CarlBrwn #360iDev


convert to cv::Mat

cv::Canny()

Detect


@CarlBrwn #360iDev


convert to cv::Mat

cv::Canny()cv::HoughLinesP()

Detect


@CarlBrwn #360iDev

HoughLinesP29Monday, September 9, 13

@CarlBrwn #360iDev


convert to cv::Mat

cv::Canny()cv::HoughLinesP()

Detect


@CarlBrwn #360iDev


convert to cv::Mat

cv::Canny()cv::HoughLinesP()for( size_t i = 0; ! ! ! i < lines.size(); i++ )

Detect


@CarlBrwn #360iDev


convert to cv::Mat


if (LineIntersect(l, l2))

Detect


@CarlBrwn #360iDev


convert to cv::Mat


if (LineIntersect(l, l2))//Measure Distance to Image corners

Detect


@CarlBrwn #360iDev


for( size_t i = 0; ! ! ! i < lines.size(); i++ )

Debugging


@CarlBrwn #360iDev


for( size_t i = 0; ! ! ! i < lines.size(); i++ )cv::line() //Draw

Debugging


@CarlBrwn #360iDev


for( size_t i = 0; ! ! ! i < lines.size(); i++ )cv::line() //Draw//Check for bounding corners

Debugging


@CarlBrwn #360iDev


for( size_t i = 0; ! ! ! i < lines.size(); i++ )cv::line() //Draw//Check for bounding cornerscv::circle() //Draw

Debugging


@CarlBrwn #360iDev


for( size_t i = 0; ! ! ! i < lines.size(); i++ )cv::line() //Draw//Check for bounding cornerscv::circle() //Drawconvert to UIImage //Display

Debugging


@CarlBrwn #360iDev

Convert to UIImage

[NSData dataWithBytes:cvMat.data length:]

CGDataProviderCreateWithCFData()

CGImageCreate()

[[UIImage alloc] initWithCGImage:]

CGXXXRelease()


@CarlBrwn #360iDev


for( size_t i = 0; ! ! ! i < lines.size(); i++ )cv::line() //Draw//Check for bounding cornerscv::circle() //Drawconvert to UIImage //Display

Debugging


#360iDev@CarlBrwn

DetectEdgesdetectFacesPressed:


#360iDev@CarlBrwn

UnskewgetAndApplyTransformPressed:


@CarlBrwn #360iDev

ViewController.mm- getAndApplyTransformPressed:

convert to cv::Mat


@CarlBrwn #360iDev


convert to cv::MatgetPerspectiveTransform()


@CarlBrwn #360iDev


convert to cv::MatgetPerspectiveTransform()warpPerspective(transform)


@CarlBrwn #360iDev

Map the Corners circled on the Left

To the Corners of the Images on the Right


@CarlBrwn #360iDev


convert to cv::MatgetPerspectiveTransform()warpPerspective(transform)


@CarlBrwn #360iDev


convert to cv::MatgetPerspectiveTransform()warpPerspective(transform)convert to UIImage //Display


#360iDev@CarlBrwn

UnskewgetAndApplyTransformPressed:


#360iDev@CarlBrwn

ExtractColorsextractColorsPressed:


@CarlBrwn #360iDev

ViewController.mm- extractColorsPressed:

convert to cv::Matcv::cvtColor(CV_RGB2HSV)

Detect


@CarlBrwn #360iDev

HSV Space

Hue/Saturation/Value

Easier to get Color out

http://en.wikipedia.org/wiki/HSV_color_space42Monday, September 9, 13

@CarlBrwn #360iDev


convert to cv::Matcv::cvtColor(CV_RGB2HSV)

Detect


@CarlBrwn #360iDev


convert to cv::Matcv::cvtColor(CV_RGB2HSV)for (int hSlice;;) { ! ! for (int vSlice;;) {

Detect


@CarlBrwn #360iDev


convert to cv::Matcv::cvtColor(CV_RGB2HSV)for (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()

Detect


@CarlBrwn #360iDev


convert to cv::Matcv::cvtColor(CV_RGB2HSV)for (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)

Detect


@CarlBrwn #360iDev


convert to cv::Matcv::cvtColor(CV_RGB2HSV)for (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)avgColor=cv::mean(roi)

Detect


@CarlBrwn #360iDev


convert to cv::Matcv::cvtColor(CV_RGB2HSV)for (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)avgColor=cv::mean(roi)//Check Saturation and Hue

Detect


@CarlBrwn #360iDev


for (int hSlice;;) { ! ! for (int vSlice;;) {

Debugging


@CarlBrwn #360iDev


for (int hSlice;;) { ! ! for (int vSlice;;) { roi = avgColor //Fill

Debugging


@CarlBrwn #360iDev


for (int hSlice;;) { ! ! for (int vSlice;;) { roi = avgColor //FillputText(roi, [color cString]

Debugging


@CarlBrwn #360iDev


for (int hSlice;;) { ! ! for (int vSlice;;) { roi = avgColor //FillputText(roi, [color cString]putText(roi, stringWithFormat:@"%.0f/%.0f/%.0f")

Debugging


@CarlBrwn #360iDev


for (int hSlice;;) { ! ! for (int vSlice;;) { roi = avgColor //FillputText(roi, [color cString]putText(roi, stringWithFormat:@"%.0f/%.0f/%.0f")

convert to UIImage //Display

Debugging


#360iDev@CarlBrwn

ExtractColorsextractColorsPressed:


#360iDev@CarlBrwn

SudokuGrabberDemo 3/3


#360iDev@CarlBrwn

FindEdgesfindPuzzlePressed:


@CarlBrwn #360iDev

Just like Cube


#360iDev@CarlBrwn

UnSkewPuzzlesquarePressed:


@CarlBrwn #360iDev

Just like Cube


#360iDev@CarlBrwn

ExtractTextextractPressed:


@CarlBrwn #360iDev

ViewController.mm- extractPressed:

Tesseract* tesseract =Setup


@CarlBrwn #360iDev


Tesseract* tesseract =initWithDataPath:@"tessdata" language:@"eng"

Setup


@CarlBrwn #360iDev


Tesseract* tesseract =initWithDataPath:@"tessdata" language:@"eng"tesseract setVariableValue:@"0123456789"forKey:@"tessedit_char_whitelist"

Setup


@CarlBrwn #360iDev

tesseract-ocrhttp://code.google.com/p/tesseract-ocr/


@CarlBrwn #360iDev


convert to cv::MatSlice


@CarlBrwn #360iDev


convert to cv::Matfor (int hSlice;;) { ! ! for (int vSlice;;) {

Slice


@CarlBrwn #360iDev


convert to cv::Matfor (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()

Slice


@CarlBrwn #360iDev


convert to cv::Matfor (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)

Slice


@CarlBrwn #360iDev


convert to cv::Matfor (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)cv::cvtColor(CV_RGB2GRAY);

Slice


@CarlBrwn #360iDev


convert to cv::Matfor (int hSlice;;) { ! ! for (int vSlice;;) {cv::Rect r()Mat cv::roi(r)cv::cvtColor(CV_RGB2GRAY);cv::threshold(roi_gray)

Slice


@CarlBrwn #360iDev


convert slice to UIImage Detect


@CarlBrwn #360iDev


convert slice to UIImage tesseract setImage:imageToParse

Detect


@CarlBrwn #360iDev


convert slice to UIImage tesseract setImage:imageToParsetesseract recognize

Detect


@CarlBrwn #360iDev


[[UILabel alloc]initWithFrame:CGRectMake()]

Debugging


@CarlBrwn #360iDev


[[UILabel alloc]initWithFrame:CGRectMake()] labelsetText:recognizedString

Debugging


@CarlBrwn #360iDev


[[UILabel alloc]initWithFrame:CGRectMake()] labelsetText:recognizedStringself.viewaddSubview:recognizedLabel

Debugging


#360iDev@CarlBrwn

ExtractTextextractPressed:


#360iDev@CarlBrwn

Based onCode From:github.com/carlbrown/OpenCVDemo360iDev2013

Questions?


writing apps that can see: getting data from coreimage to computer vision - 360idev 2013...

Technology