philipp nagele (cto, wikitude) an insider deep-dive into the wikitude sdk
TRANSCRIPT
Deep-Dive into the Wikitude SDK
2
Who is talking
Philipp Nagele
● CTO of Wikitude (Product & Technology)
● MSc in Computer System Engineering from Halmstad University in Sweden
● Previous companies:
- 3united/VeriSign Inc.
- T-Mobile Austria
Agenda for today
●Who is Wikitude
●Architecture of the Wikitude SDK
●A deeper look at the JavaScript API and the Plugins API
v v v
Wikitude AR at a glance.
First AR app in the world in
2008
10,000 apps powered by
Wikitude SDK
3D tracking
technology
v
REFERENCES
5
HQ office in Salzburg, Austria
6
AR creator and CMSCloud Recognition
Wikitude AR Products & Tools
Connect your own CMS
Mock-ups only
… understand what you are looking at
Mock-ups only
… make the World your playground
….augment books
Mock-ups only
… augment print, ads, catalogues, packaging
… virtually mount products
Maintenance
Platform agnostic technology
15
AR/VR wearables TabletsSmartphones
multi platform technology core based on C++
Tailored SDKs for Smart Glasses
Partnership since 2014, optimized SDK for BT-200, full documentation →BT-300 in pipeline
Partnership since 2016 optimized SDK for R-7
Partnership since 2014, SDK for M100, full documentation →M300 in pipeline
Native API JavaScript API
Easy access to AR mobile development
Xamarin
PhoneGapCordova
titaniumunity
Android
iOS
Wikitude SDK Architecture
2D On-DeviceReco &
Tracking
2D Cloud Reco
3D Reco & Tracking
Unity3D
Unity Plugin
Plugins API
Custom CV plugin
Custom camera plugins
Your App
Your App Cordova
PluginTitanium Module
Xamarin Componen
t
Your App
Cordova
Titanium
Xamarin
Your App
Your App
JavaScript APINative API
Your App
3D Rendering
Augmen-tations LBS
Plugin Manager
OpenGL ES2 Rendering
ARMv7 NEON
optimizations
ARMv8 optimization
sCamera
GPU optimization
s
OpenCL
Metal APIIMU
Hardware Abstraction Hardware Optimizations
C++ Layer
Java/Obj C
Wikitude JS SDK Java API
Wikitude JS SDK has a very slim Java API
• ArchitectView• Lifecycle events (onCreate, onPause, onResume, …
methods)• Load Architect Worlds
Everything else is optional• ArchitectUrlListener• ArchitectWorldLoadedListener• CaptureScreenCallback
Architect view composition
20
OpenGL ES 2 view* Renders camera stream* Renders augmentations
Web view* Loads main .html file* Transparent background
JavaScript Augmented Reality Experience
21
Architect Worlds consist of:• .html file(s)• .js file(s)• .css file(s)
• Defines your augmented reality experience• Use the Wikitude JS API provided within the Architect
view context<script src=“https://wikitude.com/libs/architect.js"></script>
• Loaded by the Wikitude SDKs Architect view
Architect world
22
Javascript execution triggers actions in the underlying C++ layer
Types of augmentations
23
Augmentations are represented by AR.Drawable subclasses
• Images - AR.ImageResource / AR.ImageDrawable• Videos (also with alpha channel) - AR.VideoDrawable• 3D models - AR.Model• Web views - AR.HTMLDrawable• Labels - AR.Label
Wikitude JavaScript API
24
Model• AR.GeoLocation• AR.RelativeLocation• AR.Tracker• AR.ImageResource
View• AR.ImageDrawable• AR.VideoDrawable• AR.Model• AR.Label
Controller• AR.GeoObject• AR.Trackable2D• AR.PropertyAnimation• AR.AnimationGroup
Geo AR
25
How to define e.g. a point of interest (POI)
• Define where the augmentation is placed by creating AR.GeoLocations (latitude/longitude coordinates)
• Define the augmentation by creating AR.Drawables
• Combine location and drawables using AR.GeoObjects
= The same geo object is displayed at different locations
POI CPOI A
POI A
Poi B
Pokémon Go like Geo AR
26
How to build a Pokémon Go like AR experience:
Pokémon Go like Geo AR
27
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon• Relative to the current user position using AR.RelativeLocation
//a relative location being 12 meters south, 20 meters west and 13 meters lower than current position of the uservar relativeLocation = new AR.RelativeLocation(null, -12, -20, -13);
Pokémon Go like Geo AR
28
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon• Relative to the current user position using AR.RelativeLocation• Absolute to the user using AR.GeoLocation
//latitude, longitude, altitudevar location = new AR.GeoLocation(47.77317, 13.069929, 320.);
Pokémon Go like Geo AR
29
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon• Relative to the current user position using AR.RelativeLocation• Absolute to the user using AR.GeoLocation
• AR.Model object to render the 3D model of a Pokémon
//create a new Modelvar model = new AR.Model(“models/pokemon.wt3”);
Pokémon Go like Geo AR
30
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon• Relative to the current user position using AR.RelativeLocation• Absolute to the user using AR.GeoLocation
• AR.Model object to render the 3D model of a Pokémon• Can be animated through .fbx defined animations
//We want to start an animation inside a modelvar model = new AR.Model(“models/pokemon.wt3");var modelAnim = new AR.ModelAnimation(model, "hit"); //start hit-animation defined inside model
Pokémon Go like Geo AR
31
How to build a Pokémon Go like AR experience:
• Define the position of a Pokémon• …
• AR.Model object to render the 3D model of a Pokémon• …
• AR.GeoObject to actually bring the model onto the screen
// a GeoObject with drawables set on creation timevar geoObject = new AR.GeoObject(location, { drawables : { cam: [model] //the drawable representing the GeoObject in the camera view }});
Pokémon Go like Geo AR
32
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger• trigger custom events when the Pokémon becomes visible in the
camera
geoObject.onEnterFieldOfVision = function() { /* your custom behaviour e.g. start a certain model animation */};geoObject.onExitFieldOfVision = function() { /* your custom behaviour */};
Pokémon Go like Geo AR
33
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger• trigger custom events when the Pokémon becomes visible in the
camera• Define a direction indicator drawable for your Pokémon
• Helps users to find the Pokémon around them
var imageResource = new AR.ImageResource(“assets/pokemonIndicator.png”);var pokemonIndicator = new AR.ImageDrawable(imageResource, 0.2);
// a GeoObject with drawables set on creation timevar geoObject = new AR.GeoObject(location, { drawables : { cam: [model], //the drawable representing the GeoObject in the camera view indicator: pokemonIndicator }});
Pokémon Go like Geo AR
34
GeoObjects & HTML content
Menu barjQuery support
Geo-located POIs
Multiple Drawables- 3 images- 3 text labels
Web View
Augmented Reality View
Pokémon Go like Geo AR
36
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger• trigger custom events when the Pokémon becomes visible in the
camera• Define a direction indicator drawable for your Pokémon
• Helps users to find the Pokémon around them• Use the AR.Radar component to permanently show the position of
a Pokémon in the AR scene
Pokémon Go like Geo AR
37
Pokémon Go like Geo AR
38
How to build an even better Pokémon Go like AR experience:
• Use AR.GeoObjects onEnter/ExitFieldOfVision trigger• trigger custom events when the Pokémon becomes visible in the
camera• Define a direction indicator drawable for your Pokémon
• Helps users to find the Pokémon around them• Use the AR.Radar to permanently show the position of a Pokémon in
the AR scene• Capture a screenshot after you caught a Pokémon using the
corresponding ArchitectView Java API.
Image Recognition
Augment target images using JavaScript API capabilities (Drawables, Sounds, Animations,Videos, 3D Models)
var kitchenTracker = new AR.Tracker("Kitchen.wtc"); var overlay = new AR.VideoDrawable("howdens.mp4", 0.65); var trackable2DObject = new AR.Trackable2DObject(
kitchenTracker: "Kitchen", { drawables:
{ cam: overlay }});
Wikitude SDK Architecture
2D On-DeviceReco &
Tracking
2D Cloud Reco
3D Reco & Tracking
Unity3D
Unity Plugin
Plugins API
Custom CV plugin
Custom camera plugins
Your App
Your App Cordova
PluginTitanium Module
Xamarin Componen
t
Your App
Cordova
Titanium
Xamarin
Your App
Your App
JavaScript APINative API
Your App
3D Rendering
Augmen-tations LBS
Plugin Manager
OpenGL ES2 Rendering
ARMv7 NEON
optimizations
ARMv8 optimization
sCamera
GPU optimization
s
OpenCL
Metal APIIMU
Hardware Abstraction Hardware Optimizations
C++ Layer
Java/Obj C
41
Plugins API in detail
Custom Frame
Provider
Image Frame
Standard Frame
Provider
Wikitude CV EngineRendering
Custom CV Plugin
Plugins API
class Plugin { public: … void cameraFrameAvailable(const Frame&; cameraFrame_); void update(const vector<RecognizedTarget> recognizedTargets_); void startRender(); void endRender …};
1234567
class RecognizedTarget { public: const string& getIdentifier() const; const Mat4& getModelViewMatrix() const; const Mat4& getProjectionMatrix() const; const float getDistanceToCamera() const;};
Use-Cases for Plugins API
● Run own computer-vision related code in parallel to Wikitude SDK-OCR Engine-Barcode Reader-Face Recognition● Own the camera stream and the camera rendering-Custom effects and shaders-Remote camera stream with local processing
….architecture, real estate….. ….gaming, filming industry….. ….indoor navigation, retail…..
New algorithms and concepts
3D Tracking / SLAMObject Reco & Tracking
See more.At Booth B6