OpenCV 3.0 Gary Bradski

Chief Scientist, Perception and AI at Magic Leap CEO, OpenCV.org

OpenCV at glance

• BSD license, 10M downloads, 500K+ lines of code

• Huge community involvement, automated patch testing and integration process

• Runs everywhere

SSE, NEON, IPP, OpenCL, CUDA, OpenCV4Tegra, …

core, imgproc, objdetect …

OpenCV HAL

OpenCV

face, text, rgbd, … OpenCV Contrib

Bindings: Python, Java

Samples, Apps, Solutions

• Find more at http://opencv.org (user)

• Or http://code.opencv.org (developer)

Recent Stats NOTE: This is only for source forge. Many more downloads come from Git and many more come on Unix distros.

OpenCV History

Willow Support

OpenCV Foundation

Intel Support

Google Summer of Code

Nvidia Support

Renewed Intel Support

Magic Leap

OpenCV Algorithm Modules Overview

5

Image Processing

Object recognition Machine learning

Transforms

Calibration Features VSLAM

Fitting Optical Flow Tracking

Depth, Pose Normals, Planes,

3D Features

Computational Photography

CORE: Data structures, Matrix math, Exceptions etc

Segmentation

HighGUI: I/O, Interface

Recent Contributions 2013

https://www.youtube.com/watch?v=_TTtN4frMEA

Recent Contributions 2014

https://www.youtube.com/watch?v=3f76HCHJJRA

OpenCV 3.0 at glance

• Mostly compatible with OpenCV 2.x; OpenCV 1.x C API is deprecated and partially removed

• Highlights:

– even more modular and extendible

– very stable API tailored for a long-term support

– decent out-of-box performance thanks to IPP, OpenCL (T-API) and NEON instructions

– lot’s of new functionality!

Aug’14 Nov’14 Dec’14 May’15

3.0 alpha 3.0 beta 3.0rc 3.0 3.1

Q3’15

Goal of 3.0: make a better OpenCV 2.0, cleanup API, get better performance (with T-API, IPP, NEON), shift to modular structure and enable user contributions. – Vadim Pisarevsky

OpenCV => OpenCV + opencv_contrib

OpenCV 2.x

OpenCV 3.x contributions

OpenCV 3.x

• OpenCV 3.0 includes mature and clean algorithms that are fully supported

• A separate contribution repository is for new CV algorithms that people want to share: http://github.com/itseez/opencv_contrib

• Patches to the contrib repository are tested as well by our buildbot to ensure integrity!

Using opencv_contrib The modules in contrib have the same structure as the standard ones:

$ cmake –D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules …

opencv/

modules/

core/

include/, doc/, src/, test/, …

CMakeLists.txt

imgproc

…

opencv_contrib/modules

text/

include/, doc/, src/, test/, …

CMakeLists.txt

…

Path to the contrib modules can be passed to cmake to build them

together with OpenCV:

New-style C++ API • All the high-level vision algorithms (face detectors, optical flow estimators,

stereo matchers etc.) are declared as pure abstract classes, similarly to interfaces in Java

• Smart pointers to instances are retrieved using special “factory” methods

• All the implementation details are hidden

• The saved space in headers is taken up instead by the verbose Doxygen comments in order to generate an always up-to-date reference manual. class StereoMatcher : public Algorithm // Algorithm is used as a base

{

public: // common methods for all stereo matchers

virtual void compute(InputArray left, InputArray right,

OutputArray disparity) = 0;

…

};

class StereoBM : public StereoMatcher

{

public: // specific methods for particular algorithm

virtual void setUniquenessRatio(int ratio) = 0;

…

// factory method

static Ptr<StereoBM> create(…);

};

…

Ptr<StereoBM> matcher = StereoBM::create(…);

matcher->compute(left, right, disparity);

Transparent API (T-API) for HW Specializations

• single API entry for each function/algorithm – no specialized cv::Canny, ocl::Canny, gpu::Canny etc.

• uses dynamically loaded OpenCL runtime if available; otherwise falls back to CPU code. Dispatching is at runtime, no recompilation needed!

• minimal or no changes in user code

– CPU-only processing – no changes should be required

• includes the following key components:

– new data structure UMat

– simple and robust mechanism for async processing

– bonus: very convenient API for implementing custom OpenCL kernels

UMat • UMat is new type of array that wraps clmem when OpenCL is available • Replacing Mat with UMat is often the only change needed

#include "opencv2/opencv.hpp"

using namespace cv;

int main(int argc, char** argv)

{

Mat img, gray;

img = imread(argv[1], 1);

imshow("original", img);

cvtColor(img, gray,

COLOR_BGR2GRAY);

GaussianBlur(gray, gray,

Size(7, 7), 1.5);

Canny(gray, gray, 0, 50);

imshow("edges", gray);

waitKey();

return 0;

}

#include "opencv2/opencv.hpp"

using namespace cv;

int main(int argc, char** argv)

{

UMat img, gray;

img = imread(argv[1]).getUMat();

imshow("original", img);

cvtColor(img, gray,

COLOR_BGR2GRAY);

GaussianBlur(gray, gray,

Size(7, 7), 1.5);

Canny(gray, gray, 0, 50);

imshow("edges", gray);

waitKey();

return 0;

}

Transparent API: under the hood bool _ocl_cvtColor(InputArray src, OutputArray dst, int code) {

static ocl::ProgramSource oclsrc(“//cvtcolor.cl source code\n …”);

UMat src_ocl = src.getUMat(), dst_ocl = dst.getUMat();

if (code == COLOR_BGR2GRAY) {

// get the kernel; kernel is compiled only once and cached

ocl::Kernel kernel(“bgr2gray”, oclsrc, <compile_flags>);

// pass 2 arrays to the kernel and run it

return kernel.args(src, dst).run(0, 0, false);

} else if(code == COLOR_BGR2YUV) { … }

return false; // OpenCL function does not have to support all modes

}

void _cpu_cvtColor(const Mat& src, Mat& dst, int code) { … }

// transparent API dispatcher function

void cvtColor(InputArray src, OutputArray dst, int code) {

dst.create(src.size(), …);

if (useOpenCL(src, dst) && _ocl_cvtColor(src, dst, code)) return;

// getMat() uses zero-copy if available; and with SVM it’s no op

Mat src_cpu = src.getMat();

Mat dst_cpu = dst.getMat();

_cpu_cvtColor(src_cpu, dst_cpu, code);

}

OpenCV+OpenCL execution model

• One queue and one OpenCL device per CPU thread • Different CPU threads can share a device, but use different queues. • OpenCL kernels are executed asynchronously • cv::ocl::finish() puts the barrier in the current CPU thread. • It’s rarely needed to call cv::ocl::finish() manually.

…

ocl::Queue

ocl::Device

ocl::Queue ocl::Queue

ocl::Device

…

…

ocl::Context

CPU threads

IPP + OpenCV = v. fast OpenCV

• Intel gave us and our users free (as in “beer”) and royalty-free subset of IPP 8.x (IPPICV).

• IPPICV is linked into OpenCV at compile stage and replaces the corresponding low-level C code.

• Our buildbot ensures that all the tests pass

New Functionality and other improvements

• Results from 20+ successful projects from GSoC 2013, 2014:

– Computational photography, Text detection, Object Tracking, Matlab bindings etc.

• 1000+ Pull Requests @ github (200+ PR’s between alpha & beta)

• 18 new OpenCV modules! (mostly in opencv_contrib)

OpenCV QA Contribution/patch workflow: see OpenCV wiki

build.opencv.org: buildbot with 50+ builders

pullrequest.opencv.org: tests each pullrequest github.com/itseez/opencv

OpenCV test suite

• GoogleTest-based + set of Python scripts

• Thousands of unit tests

• Accuracy tests

• Performance tests

python ../modules/ts/misc/summary.py core*.xml -f "add:.*C4" -u s

Geometric mean

Name of Test core core core

posix posix posix

x64 x64 x64

6693M 6695 6695

2011-09-08--13-13-41 2011-09-08--13-30-06 2011-09-08--13-30-06

vs

core

posix

x64

6693M

2011-09-08--13-13-41

core_arithm__add::Size_MatType::(127x61, 8UC4) 0.000 s 0.000 s 1.00

core_arithm__add::Size_MatType::(1280x720, 8UC4) 0.004 s 0.004 s 0.99

core_arithm__add::Size_MatType::(1920x1080, 8UC4) 0.009 s 0.009 s 1.02

core_arithm__add::Size_MatType::(640x480, 8UC4) 0.001 s 0.001 s 1.00

OpenVX (Khronos HAL)

OpenCV was one of the key contributors to the new Khronos accelerated vision API: OpenVX

(Hardware Acceleration Library)

The HAL + Accelerators • opencv_hal - IPP-like, fastcv-like low-level API to accelerate

OpenCV for different platforms. To be extended in 3.X…

• As stated, opencv_ocl module (OpenCL acceleration) is now used in other modules to give transparent acceleration (T-API)

• Universal Umat structure now can be used instead of cv::Mat and OclMat.

• OpenVX, as it matures, may be used as one of the HW accelerators

SSE, NEON, IPP, OpenCL, CUDA, OpenCV4Tegra, …

core, imgproc, objdetect …

OpenCV HAL

OpenCV

face, text, rgbd, … OpenCV Contrib

Bindings: Python, Java

Samples, Apps, Solutions

New from Google Summer of Code 2015

• Deep network optimized execution and interoperability to existing libraries

• Stereo matching improvements

• Projection mapping

• Improved Camera Calibration

• Better AR fiducial support

• Improvements to text detection and tracking

• Neon optimization

Other Initiatives: CVPR State of the Art Vision Challenge

State of the Art Vision Challenge at CVPR 2015 Our aim is to make available state of the art vision in OpenCV. We thus ran a vision challenge to meet or exceed the state of the art in various areas. We will present the results. The contest details are available at: http://code.opencv.org/projects/opencv/wiki/VisionChallenge Prizes: 1. Win: $1000; Submit code: $3000 2. Win: $1000; Submit code: $3000 3. Win: $1000; Submit code: $3000 4. Win: $1000; Submit code: $3000 5. Win: $1000; Submit code: $3000

Other Initiatives: People’s Choice: Best Paper

People’s Choice: Best paper We will tally the people’s vote for best paper/paper you’d most like to see implemented. We’ll present the histogram of results which is an indication of the algorithms people are interested in overall and then list the 5 top winners. Prizes will be awarded in two stages: A modest award for winning and a larger award for presenting the code w/in 5 months as a pull request to OpenCV as Detailed here: http://code.opencv.org/projects/opencv/wiki/How_to_contribute Prizes: 1. Win: $500; Submit code: $6000 2. Win: $300; Submit code: $4000 3. Win: $100; Submit code: $3000 4. Win: $50; Submit code: $3000 5. Win: $50; Submit code: $3000

Functional Language Exploration

• Proliferation of new hardware makes it hard to support code.

– Let the compiler port to different hardware using a no-side effects functional language “NUML.”

– Can compile NUML to C++, C, Java and Python.

• NUML is an array/image comprehending functional language. – https://github.com/vpisarev/numl/tree/alt_syntax/src

Learning OpenCV 3.0

• Out with 3.0/CVPR … or late summer!

For Version 3.0

Other news: Intel is now supporting OpenCV Optimizations • https://software.intel.com/en-us/opencv

OpenCV release candidate is now out! • http://opencv.org/opencv-3-0-rc1.html

27

Photo: Gary Bradski

Questions?

http://youtu.be/LE7aiONMjK4