diy- computer vision with gwt
TRANSCRIPT
Alberto Software Developer & Linux Sysadmin
C/C++ ( since 80’s :/ )GWT ( since 1.6 )
Javascript & Java
[email protected]+AlbertoMancini
FrancescaWeb and mobile developer,software engineer and architect
C/C++, CFD, HPCGWT since 2009
“with a passion for clean code and fine tuned details”
[email protected]@francescatosi+FrancescaTosi
Devfest’s CodeProject
A couple of years(**) ago we started working on a pet project trying to explore how mixing modern WebAPIs and an effective programming approach(*) can enable us to develop complex client-side applications running in a browser.
(*)read: GWT, i.e. a typed language compiled into js
(**)for the 2012’s GDG DevFest season
Anybody knows
Browsers are a good
“execution environnement”
for evoluted web-pages ….
Browsers are
enough ?
I Mean ...
C’mon we are in 2015:
browsers are certainly more than enough
for any kind of application !
GoogleDocs, Inbox, Plus, Scala vm in js, JsDOSBox…
BETting
The right tool(*) is enough to let us to develop
- complex - compute-intensive
browser-based applications ?
(*) GWT of course :)
The RIGHT TOOL
Should (at least) let us:● master the (quite tricky) web platform● code safely … in a type-safe language
(but provide native-like flexibility and performance)
● reuse existing code
… and of course produce a fast and reliable application
NO,no way
Our forecast was
Target
We were working on Augmented Reality and we had an expertize on it so we started trying to realize an AR webApp in order to show that this kind of applications is way too-complex for the performance of a browser.
Augmented Reality (browser based)
Marker BasedAugmented Reality
Marker Based AR
Essentially composed of 3 steps
● Marker detection and identification(finding the marker in a picture)
● Computation of the marker pose(det. six degrees of freedom w.r.t camera, f.i.)
● Syntetic objects collocation (object space to camera space transf)
Marker Based AR
NyARToolkit, do not reinvent the wheel
NyARToolKit is an ARToolKit built with 100% pure Java.
[great performance, featurefull, GPL]
http://nyatla.jp/nyartoolkit/wp/
GWT-NyARToolkit
… given an image we get a 3x4 (3x3 + a translation vector actually) matrix that identifies the marker in the camera space
nyar.update(i_sensor); ← ‘push’ an image
found = nyar.isExistMarker(marker_id); ← query if a marker is detected
mm = nyar.getMarkerMatrix(marker_id); ← compute the collocation matrix
https://code.google.com/p/gwt-nyartoolkit/http://jooink.blogspot.com
Browser Based AR
<video/>
<canvas/>
Ny
HTTP
?
?
GWT … glue
Acquiring Video (and eventually Audio)
WebRTC
Acquiring Video (and eventually Audio)
Acquisition of Audio/Video stream via simple js api.
Why WebRTC ?
Why WebRTC ?Plugins for IE
e.g.Temasys provides ad ad hoc craftedreplacement ofadapter.js
Audio/Video
Acqusition Audio/Video stream via simple js apis.
navigator.getUserMedia(constraints,successCallBack,errorCallBack)
WebRTC with GWT
GWT (Elemental at the beginning)
At the beginning elemental, but right now (using jnsi and now jsinterop) we are able to do all the operation without dependencies GWT.
https://code.google.com/p/elemental-getusermedia-demo
https://github.com/jooink/gwt-getusermedia
Browser Based AR
<video/>
the video stream
can (has to be) be played in a <video/> tag
“native” (jsni)
video.src = URL.createObjectURL(stream);
Browser Based AR
<canvas/>
<video/>
canvasCtx2D.drawImage(video, ...);
Browser Based AR
<canvas/>
<video/>
canvasCtx2D.drawImage(video, ...);
canvasCtx2D.getImageData(sx, sy, sw, sh)
Browser Based AR
<video/>Ny
<canvas/>
Browser Based AR
<video/>
<canvas/>
Ny
3D models to be shown on the marker
after the recognition
Browser Based AR
<video/>
<canvas/>
Ny
HTTP
WebGLWe had to reinvent the wheel :(
three.js, BabylonJS
parallax (GWT !)
awesome shading rendering and lighting
tend to mix webgl-interaction with 3D graphics
WebGL, gwt-webgl
<video/>Ny
<canvas/>
https://code.google.com/p/gwt-webgl
based on Elemental with some modifications
damn thin wrapper around webgl standard
Tiny3D
● 3D geometry (4x4 & affine matrices)
● Lambertian and Phong lighting
● Polygons, Polyhedra, normals etc..Wavefront OBJ import
Browser Based AR
<video/>
<canvas/>
Ny
HTTP
Browsers are enough!
GWT is more than enough!
mGWT
GWT Widgets & morefor mobile
FirefoxOS
Web developerspreferredmobile OS?
B2G
Browser Based AR
<video/>
<canvas/>
Ny
HTTP
Browser Based AR
<video/>
<canvas/>
Ny
HTTP
Computer Vision
Computer Vision
As told by Alberto, after a while we needed to have some more complex features on this tool for image recognition, in order to have more information on the image we are working with.
Computer Vision
Computer Vision
There exists a lot of CV libs (complete, tested and fine tuned) for example:
- OpenCV- VLFeat- VLX- …
C/C++
Computer Vision
Computer Vision
There exists a lot of CV libs (complete, tested and fine tuned) for example:
- OpenCV- VLFeat- VLX- …
- BoofCV
C/C++
Java
Computer Vision
Computer Vision by BoofCV
There is a great library for Computer Vision interely written in Pure Java: BoofCV → we decided to adapt BoofCV for GWT.
BoofCV
BoofCV is a complete & complex project
BoofCV is an opensource Java library for computer vision & robotics applications.
BoofCV is organized into several packages: image processing, features, geometric vision, calibration, recognition, visualize and IO.
BoofCV
Written by Peter Abeles
www.boofcv.org
Link to the source code:
https://github.com/lessthanoptimal/BoofCV
BoofCV
Adapting an entire big & complex library is not is a funny game.
BoofCV depends on other Java libs, so we must adapt all the dependencies
BoofCV
- DDogLeg: Java numerical lib for non-linear optimization, robust model-fitting, polynomial root finding, sorting and more;
- GeoRegression: open source java geometry lib for scientific computing;
- EJML: dense matrix linear algebra library written in Java
BoofCV- GWT-DDogLeg:
https://github.com/jooink/gwt-ddogleg
http://gwtddogleg.appspot.com/
- GWT-GeoRegression: https://github.com/jooink/gwt-georegressionhttp://gwtgeoregression.appspot.com/
- GWT-EJML: https://github.com/jooink/gwt-ejml
http://gwtejml.appspot.com/
EJML
- EJML: dense matrix linear algebra library written in Java
https://github.com/lessthanoptimal/ejml
GWT-EJML
GWT-EJML
is an adapter-project around EJML that let us to use the original library:
- provides .gwt.xml files - supersources many classes - contains running examples
GWT-EJML
GWT-EJML
published on github
https://github.com/jooink/gwt-ejml
GWT-EJML
org.EJML.gwt.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.6.1.rc1//EN" "http://google-web-toolkit.googlecode.com/svn/tags/2.6.1.rc1/distro-source/core/src/gwt-module.dtd">
<module>
<source path="ejml"/>
</module>
GWT-EJML
.gwt.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0//EN"
"http://gwtproject.org/doctype/2.7.0/gwt-module.dtd">
<module rename-to='gwtejml'>
<inherits name='org.EJML'/>
...
<super-source path="jre" />
...
</module>
GWT-EJML
The Strategy
we decided to use the SuperSourcing in order to change/adapt the class of the basic lib and leave the ejml project unchanged
→ the maintainability is easier
GWT-EJML
SuperSourcing
We added the super source path in the gwt.xml config file
<super-source path="jre" />
and we created the file to modify in that directory (using the right path of the file in the main project).
GWT-EJML
.gwt.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0//EN"
"http://gwtproject.org/doctype/2.7.0/gwt-module.dtd">
<module rename-to='gwtejml'>
<inherits name='org.EJML'/>
...
<super-source path="jre" />
...
</module>
GWT-EJML
Supersourced parts of JRE
java.io.ByteArrayOutputStream
public class ByteArrayOutputStream extends OutputStream {
public ByteArrayOutputStream() { … }
public void write(int arg0) { … }
}
GWT-EJML
Supersourced parts of EJMLorg.ejml.ops.MatrixIO;
public class MatrixIO { … }
org.ejml.alg.dense.decomposition
.eig.symm.SymmetricQREigenHelper
org.ejml.alg.dense.decomposition
.svd.implicitqr.SvdImplicitQrAlgorithm
java.io.PrintStream
printf()
GWT-EJML
Supersourced parts of EJML
java.io.PrintStream does not have printf() in the gwt emulated jre
We could not supersource the PrintStream because already exist on emulated jre and we cannot super-sourcing just a method.
GWT-EJML
Link
https://github.com/jooink/gwt-ejml
http://gwtejml.appspot.com/
DDogLeg
- DDOGLEG: Java numerical lib for non-linear optimization, robust model-fitting, polynomial root finding, sorting and more;
https://github.com/lessthanoptimal/ddogleg
GWT-DDogLeg
.gwt.xml <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0//EN"
"http://gwtproject.org/doctype/2.7.0/gwt-module.dtd">
<module rename-to='gwtddogleg'>
<inherits name='org.DDOGLEG'/>
<inherits name='com.jooink.gwtejml.gwtEJML_lib'/>
...
<super-source path="jre" />
...
</module>
GWT-DDogLeg
Supersourced parts of DDOGLEG:
java.lang.OutOfMemoryError
Unchecked Exception …
GWT-DDogLeg
Supersourced parts of DDOGLEG:
org.ddogleg.struct.FastQueue
org.ddogleg.struct.FastQueueList
→ Reflection (not supported by gwt-jre)
GWT-DDogLeg
Link
https://github.com/jooink/gwt-ddogleg
http://gwtddogleg.appspot.com/
GEOREGRESSION
- GEOREGRESSION: open source java geometry lib for scientific computing;
https://github.com/lessthanoptimal/georegression
GWT-GEOREGRESSION
- GWT-GEOREGRESSION: minor modifications
GWT-GEOREGRESSION
.gwt.xml <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0//EN" "http://gwtproject.org/doctype/2.7.0/gwt-module.dtd">
<module rename-to='gwtgeoregression'>
<inherits name='georegression.GEOREGRESSION'/>
<inherits name='com.jooink.gwtejml.gwtEJML_lib'/>
<inherits name='com.jooink.gwtddogleg.gwtDDogLeg_lib'/>
...
</module>
GWT-GEOREGRESSION
Link
https://github.com/jooink/gwt-georegression
http://gwtgeoregression.appspot.com/
BoofCV
- BOOFCV: fast computer vision library written entirely in Java;
https://github.com/lessthanoptimal/boofcv
.gwt.xml <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Google Inc.//DTD Google Web Toolkit 2.7.0//EN" "http://gwtproject.org/doctype/2.7.0/gwt-module.dtd">
<module rename-to='gwtboofcv'>
<inherits name='boofcv.BOOFCV'/>
<inherits name='com.jooink.gwtejml.gwtEJML_lib'/>
<inherits name='com.jooink.gwtddogleg.gwtDDogLeg_lib'/>
<inherits name='com.jooink.gwtGeoRegression.gwtGeoRegression_lib'/>
...
<super-source path="jre" />
...
</module>
BoofCV
GWT-BOOFCV:
● Reflection
● Image managment
BoofCV
GWT-BOOFCV:
● Reflection: ○ used mostly to handle Arrays of primitive
types○ probably unneeded upstream
working with Peter to understand if can be avoided through, for instance, a few of factories.
● Image managment
BoofCV
Reflection:
java.lang.reflect.Array
Array.newInstance( … )
Array.getLength( … )
...
BoofCV
GWT-BOOFCV:
● Reflection
● Image managment
BoofCV
Image managment:
of course we had to work-out the different point of view about Images:
Browser → (<img/> <video/>)
BoofCV → array of values
→ we used <canvas/> (also in this case);
canvas.getContext2d().getImageData( … );
BoofCV
EXAMPLES:
- Interest Point detection
- Interest point association
BoofCV
EXAMPLE 1
Interest Point detection:
ImageFloat32
DetectDescribePoint<ImageFloat32,SurfFeature>detDesc.detect(boofImage);
BoofCV
- Interest point
detection
http://static.jooink.com/experiments/boofcv-gwt/interest-points/
BoofCV
- Interest point
association
Questions? Contact us :
Francesca TosiR&D at Jooink [email protected]
Alberto ManciniDev at Jooink [email protected]
Thanks !Please rate this presentation http://gwtcreate.com/agenda