426 lecture 4: ar developer tools
Post on 28-Jan-2015
129 Views
Preview:
DESCRIPTION
TRANSCRIPT
COSC 426: Augmented Reality
Mark Billinghurst
mark.billinghurst@hitlabnz.org
August 1st 2012
Lecture 4: AR Developer Tools
Low Level AR Libraries
Low Level AR Libraries ARToolKit
Marker based tracking
FLARToolKit Flash version of ARToolKit
SSTT Simple Spatial Template Tracking
Opira Robust Natural Feature Tracking
What is ARToolKit?
Marker Tracking Library for AR applications Open Source, Multi-platform (Linux, Windows, MacOS)
Overlays 3D virtual objects on real markers Uses single tracking marker Determines camera pose information (6 DOF)
ARToolKit Websites http://www.hitl.washington.edu/artoolkit/
http://artoolkit.sourceforge.net/
How it Works
ARToolKit Software ARToolKit version: 2.65 or later Currently two license models
Open Source (GPL): ARToolKit 2.72 Commercial (ARToolWorks): ARToolKit 4.0
OS: Linux, Windows, MacOS X, iPhone/Android Programming language: C Related software
ARToolKit Professional: Commercial version ARToolKitPlus: Advanced version NyARToolkit: Java and C# version FLARToolKit: Flash version
ARToolKit Family
ARToolKit ARToolKit NFT
ARToolKit (Symbian)
NyToolKit - Java, C#, - Android, WM
JARToolKit (Java)
FLARToolKit (Flash)
FLARManager (Flash)
ARToolKit Plus
ARToolKit Structure
Three key libraries: AR32.lib – ARToolKit image processing functions ARgsub32.lib – ARToolKit graphics functions ARvideo.lib – DirectShow video capture class
DirectShow
ARvideo.lib
Additional Software
ARToolKit just provides tracking For an AR application you’ll need more software
High level rendering library Open VRML, Open Inventor, osgART, etc
Audio Library Fmod, etc
Peripheral support
What does ARToolKit Calculate? Position of makers in the camera coordinates Pose of markers in the camera coordinates Output format
3x4 matrix format to represent the transformation matrix from the marker coordinates to the camera coordinates
Coordinate Systems
From Marker To Camera Rotation & Translation
TCM : 4x4 transformation matrix from marker coord. to camera coord.
AR Application Development
An ARToolKit Application Initialization
Load camera and pattern parameters Main Loop
Step1. Image capture and display Step2. Marker detection Step3. Marker identification Step4. Getting pose information Step5. Object Interactions/Simulation Step6. Display virtual objects
End Application Camera shut down
Sample ARToolKit Applications
Ex. 1: Simple video display Ex. 2: Detecting a marker Ex. 3: Using pattern Ex. 4: Getting a 3D information Ex. 5: Virtual object overlay
Ex 1: Simple Video Display
Program : sample1.c Key points
Loop structure Video image handling Camera parameter handling Window setup Mouse and keyboard handling
Sample1.c – video initialization Configure the video input
vconf = <video configuration string>
Start video capture arVideoCapStart();
In init(), open the video arVideoOpen( vconf ); arVideoInqSize(&xsize, &ysize);
When finished, close the video path arVideoCapStop(); arVideoClose();
Changing Image Size
For input capture "vconf = “videoWidth=320,videoHeight=240";
Note – the camera must support this image size
For display argInit( &cparam, 1.5, 0, 0, 0, 0 );
The second parameter means zoom ratio for display image size related to input image.
Graphics handling: libARgsub
Set up and clean up the graphics window void argInit( ARParam *cparam, double zoom, int fullFlag, int xwin, int ywin, int hmd_flag );
void argCleanup( void );
cparam: camera parameter zoom: zoom ratio fullFlag: 0: normal, 1: full screen mode Xwin, ywin: create small window for debug hmd_flag: 0: normal, 1: optical see-through mode
Sample1.c Main Function main()!{!!init();!!argMainLoop( mouseEvent, !!
!keyEvent, mainLoop); !}!
Graphics handling: libARgsub Go into the iterative cycle
void argMainLoop( void (*mouseFunc)(int btn,int state,int x,int y), void (*keyFunc)(unsigned char key, int x, int y), void (*mainFunc)(void)
);
Swap buffers void argSwapBuffers( void );
Sample1.c - mainLoop Function if( dataPtr = (ARUint8 *) arVideoGetImage()) == NULL ) {
arUtilSleep(2); return;
} argDrawMode2D(); argDispImage(dataPtr, 0, 0 ); arVideoCapNext(); argSwapBuffers();
Image capture: libARvideo
Return the pointer for captured image ARUint8 *arVideoGetImage( void );
Pixel format and byte size are defined in config.h #define AR_PIX_FORMAT_BGR #define AR_PIX_SIZE 3
Graphics handling: libARgsub Set the window for 2D drawing
void argDrawMode2D( void );
Set the window for 3D drawing void argDrawMode3D( void );
void argDraw3dCamera( int xwin, int ywin );
Display image void argDispImage( ARUint8 *image, int xwin, int ywin );
Ex. 2: Detecting a Marker Program : sample2.c Key points
Threshold value Important external variables arDebug – keep thresholded image arImage – pointer for thresholded image arImageProcMode – use 50% image for image
processing - AR_IMAGE_PROC_IN_FULL - AR_IMAGE_PROC_IN_HALF
Sample2.c – marker detection
/* detect the markers in the video frame */ if(arDetectMarker(dataPtr, thresh, &marker_info, &marker_num) < 0 ) { cleanup(); exit(0); } for( i = 0; i < marker_num; i++ ) { argDrawSquare(marker_info[i].vertex,0,0); }
Sample2.c – marker_info structure typedef struct {
int area; int id; int dir; double cf; double pos[2]; double line[4][3]; double vertex[4][2];
} ARMarkerInfo; !
Ex. 3: Using a Pattern
Program : sample3.c Key points
Pattern files loading Structure of marker
information - Region features - Pattern Id, direction - Certainty factor
Marker identification
Making a pattern template Use of utility program:
mk_patt.exe Show the pattern Put the corner of red line
segments on the left-top vertex of the marker
Pattern stored as a template in a file
1:2:1 ratio determines the pattern region used
Sample3.c – Pattern File Loading
int patt_id; char *patt_name = “Data/kanjiPatt”
/* load pattern file */ if(patt_id=arLoadPatt (patt_name) < 0) {
printf ("Pattern file load error !! \n"); exit(0);
}
Checking for known patterns /* check for known patterns */
k = -1; for( i = 0; i < marker_num; i++ ) { if( marker_info[i].id == patt_id) { /* you've found a pattern */ printf("Found pattern: %d \n",patt_id); if( k == -1 ) k = i;
else /* make sure you have the best pattern
(highest confidence factor) */ if( marker_info[k].cf < marker_info[i].cf )
k = i; } }
Ex. 4 – Getting 3D information
Program : sample4.c Key points
Definition of a real marker Transformation matrix
- Rotation component - Translation component
Sample4.c – Transformation matrix
double marker_center[2] = {0.0, 0.0}; double marker_width = 80.0; double marker_trans[3][4];
arGetTransMat(&marker_info[i], marker_center, marker_width, marker_trans);
Finding the Camera Position This function sets transformation matrix from marker
to camera into marker_trans[3][4]." arGetTransMat(&marker_info[k], marker_center,
marker_width, marker_trans);
You can see the position information in the values of marker_trans[3][4]."" Xpos = marker_trans[0][3]; Ypos = marker_trans[1][3]; Zpos = marker_trans[2][3];
ARToolKit Coordinate Frame
Ex. 5- Virtual Object Display
Program : sample5.c Key points
OpenGL parameter setting Setup of projection matrix Setup of modelview matrix
Appending your own OpenGL code
Set the camera parameters to OpenGL Projection matrix. argDrawMode3D(); argDraw3dCamera( 0, 0 );
Set the transformation matrix from the marker to the camera to the OpenGL ModelView matrix. argConvGlpara(marker_trans, gl_para); glMatrixMode(GL_MODELVIEW); glLoadMatrixd( gl_para );
After calling these functions, your OpenGL objects are drawn in the real marker coordinates.
3D CG Model Rendering ARToolKit does not have a function to handle
3D CG models. 3rd party CG rendering software should be
employed. OpenVRML OpenSceneGraph etc
Loading Multiple Patterns
Sample File: LoadMulti.c Uses object.c to load
Object Structure typedef struct { char name[256]; int id; int visible; double marker_coord[4][2]; double trans[3][4]; double marker_width; double marker_center[2]; } ObjectData_T;
Finding Multiple Transforms
Create object list ObjectData_T *object;
Read in objects - in init( ) read_ObjData( char *name, int *objectnum );
Find Transform – in mainLoop( ) for( i = 0; i < objectnum; i++ ) { ..Check patterns ..Find transforms for each marker }
Drawing Multiple Objects
Send the object list to the draw function draw( object, objectnum );
Draw each object individually for( i = 0; i < objectnum; i++ ) { if( object[i].visible == 0 ) continue; argConvGlpara(object[i].trans, gl_para); draw_object( object[i].id, gl_para); }
Limitations of ARToolKit Partial occlusions cause tracking failure Affected by lighting and shadows Tracking range depends on marker size Performance depends on number of markers
cf artTag, ARToolKitPlus Pose accuracy depends on distance to marker Pose accuracy depends on angle to marker
ARToolKit in the World
Hundreds of projects Large research community
FLARToolKit Flash AS3 Version of the ARToolKit
(was ported from NyARToolkit the Java Version of the ARToolkit)
enables augmented reality in the Browser uses Papervision3D for as 3D Engine available at http://saqoosha.net/ dual license, GPL and commercial license
FLARToolkit
Papervision 3D
Adobe Flash
AR Application Components
private function mainEnter(e:Event):void { /* Capture video frame*/ capture.draw(vid);
/* Detect marker */ if (detector.detectMarkerLite(raster, 80) && detector.getConfidence() > 0.5) { //Get the transfomration matrix for the current marker position detector.getTransformMatrix(trans);
//Translates and rotates the mainContainer so it looks right mainContainer.setTransformMatrix(trans);
//Render the papervision scene renderer.render(); } }
FLARToolKit Examples
Papervision 3D http://www.papervision3d.org/ Flash-based 3D-Engine Supports
import of 3D Models texturing animation scene graph
alternatives: Away3d, Sandy,…
Source Packages „Original“ FLARToolkit (Libspark, Saqoosha) (
http://www.libspark.org/svn/as3/FLARToolKit/trunk/ )
Start-up-guides Saqoosha (http://saqoosha.net/en/flartoolkit/start-up-guide/ )
Miko Haapoja (http://www.mikkoh.com/blog/?p=182 )
„Frameworks“ Squidder MultipleMarker – Example (
http://www.squidder.com/2009/03/06/flar-how-to-multiple-instances-of-multiple-markers/ )
FLARManager (http://words.transmote.com/wp/flarmanager/ )
Other Languages NyARToolKit
http://nyatla.jp/nyartoolkit/wp/ AS3, Java, C#, Processing, Unity, etc
openFrameworks http://www.openframeworks.cc/ https://sites.google.com/site/ofauckland/examples/8-artoolkit-example
Support for other libraries - Kinect, Audio, Physics, etc
void testApp::update(){ //capture video and detect markers mov.update(); if (mov.isFrameNew()) { img.setFromPixels(mov.getPixels(), ofGetWidth(), ofGetHeight()); gray = img; tracker.setFromPixels(gray.getPixels()); } } //-------------------------------------------------------------- void testApp::draw(){ //draw AR objects ofSetColor(0xffffff); mov.draw(0, 0); for (int i=0; i<tracker.markers.size(); i++) { ARMarkerInfo &m = tracker.markers[i]; tracker.loadMarkerModelViewMatrix(m); ofSetColor(255, 255, 0, 100); ofCircle(0,0,25); ofSetColor(0); ofDrawBitmapString(ofToString(m.id),0,0); } }
AR Tools
experiences
applications
tools
components
Sony CSL © 2004
Building Compelling AR Experiences
Tracking, Display
Authoring
AR Authoring Software Libraries
osgART, Studierstube, MXRToolKit Plug-ins to existing software
DART (Macromedia Director), mARx, Unity, Stand Alone
AMIRE, BuildAR, etc Next Generation
iaTAR (Tangible AR)
mARx Plug-in
3D Studio Max Plug-in Can model and view AR content at the same time
BuildAR
http://www.buildar.co.nz/ Stand alone application Visual interface for AR model viewing application Enables non-programmers to build AR scenes
Plug-ins for 3D authoring tools
Unity - Game development tool (www.unity3d.com) Esperient Creator – General interactive 3D authoring
(www.esperient.com)
Metaio Design Complete commercial authoring platform
http://www.metaio.com/software/design/ Offers viewer and editor tools Drag and drop tools 3D model import
Total Immersion D’Fusion Studio Complete commercial authoring platform
http://www.t-immersion.com/ Multi-platform Markerless tracking Scripting Face tracking Finger tracking Kinect support
Others AR-Media
http://www.inglobetechnologies.com/ Google sketch-up plug-in
LinceoVR http://linceovr.seac02.it/ AR/VR authoring package
Libraries JARToolKit, MXRToolKit, ARLib, Goblin XNA
OSGART Programming Library Integration of ARToolKit with a High-Level
Rendering Engine (OpenSceneGraph) OSGART= OpenSceneGraph + ARToolKit
Supporting Geometric + Photometric Registration
osgART:Features
C++ (but also Python, Lua, etc). Multiple Video Input supports:
Direct (Firewire/USB Camera), Files, Network by ARvideo, PtGrey, CVCam, VideoWrapper, etc.
Benefits of Open Scene Graph Rendering Engine, Plug-ins, etc
Advanced Authoring: iaTAR (Lee 2004)
Immersive AR Authoring Using real objects to create AR applications
More Information • Mark Billinghurst
– mark.billinghurst@hitlabnz.org • Websites
– www.hitlabnz.org
top related