Low latency rendering for mobile augmented reality

W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen

Delft University of TechnologyUbiCom – http://www.ubicom.tudelft.nl

System sketch

Way finding

Maintenance, assistance

How to do this?

1. System overview

2. Why low latency3. Low-latency rendering4. Low-latency tracking

5. Status of project

1. System overview

InternetBasestationBackboneMobile

unitBase

stationMass

storageComputepower

Mobile unit off-loads processes to backbone

Camera serves 2 purposesViewing through camera of another user Tracking the position of the user

Mobile unit hardware17GHz, IR

TransmissionCompressAnalysisvideo outTrackingDecom-press

Render

2. Latency requirements

Accurate positioning requires low latencyMost demanding situation: head rotation

Alignment error depends on lag and rotational speed

See-throughdisplayReal objectMisalignedvirtual object

Humans can detect <8ms

But hardly any literature on relation with task.

Estimated for mentioned tasks:Max. error 0.5˚ @ 50˚/s latency required ≤ 10 ms

VR: several sources of latency

Source of latency               typical3D Rendering 30-200 msTracking 15-40 msDisplay @50Hz 20 ms

Fastest reported: 17 (33)ms latency

Our goal: 10 msRendering + Display 8msTracking 2ms

Std. Voodoo 3D game card

Render just ahead of raster beam

4 partitionsgive latency5-10 ms

20msDisplaying here

Rendering hereTo be cleared and rendered

3. Low-latency rendering

~ hundred texture-mapped polygons Simplify virtual world

In backbone, using LOD polygon reduction polygon-to-texture techniques

Feasible: relatively small #virtual objects

ApplicationdatabasesNetworkresourcesComputestationsimplified sceneBasestation

Latency-layered system

complexvirtualobjects

simplevirtualobjects

displaylistDisplay>200HzHeadsetBackbone~10Hz<1HzPositionfrom inertial tracker

Position from camera and prediction

Usermovement~10ms~100ms~1s

PrototypeKURT Realtime Linux to• schedule renderer at 200Hz• in sync with displayLatency measured: 4.2-8.8ms (6.35 avg)

TrackerSimilar latency-layered techniques as rendering, using• inertial tracker in low-latency loop• camera feature tracker in outer loop

Prototypeoff-the-shelf componentsNot yet wireless (tracker & power cables)

Rendering system• “VRML2” scene graphs• changed animation:

no 200Hz scene graph traversal-> compile @10Hz to display list @ 200Hz-> tags in displaylist for last-minute updates

5. Status of project

Tracking• Proto: Intersense IS600 (inertial+acoustic)• Working on optical tracking with artificial feducials• Later: natural feducials

Future• Fast & simple hardware for rendering, combining well with video transmission• Task - latency relation • Applications• Incremental scene simplification

ConclusionAccurate alignment for ARLatency-layered system architectureFirst-ever <10ms rendering (no prediction)

www.ubicom.tudelft.nl