mipi devcon bangalore 2017: imaging systems design for mixed reality scenarios
TRANSCRIPT
Prasanna KrishnaswamyIntelPlatform Architect
Imaging Systems Design for Mixed Reality Scenarios
©2017MIPIAlliance,Inc.
Agenda• MixedReality(MR)– Introduction• KeyUseCases• MRSystemDesignOverview• ImagingSensorsforMR• UseCaseDecomposition• Opportunities
2
Intel
©2017MIPIAlliance,Inc.
MixedReality- Introduction
3
Intel
• ExperiencesthatoverlaygraphicsonvideostreamsofphysicalworldareAugmentedReality
• ExperiencesthatoccludeyourviewtopresentadigitalexperienceareVirtualReality
• ExperiencesenabledbetweenthesetwoextremesareMixedReality,whichisblendingofthephysical
worldanddigitalworld
DiagramsfromMicrosoftMixedRealityAcademy
https://developer.microsoft.com/en-us/windows/mixed-reality/academy
©2017MIPIAlliance,Inc.
KeyUseCases
4
Intel
ExperienceVectorsOpticalSeeThroughvsVideoSeeThroughHMD
AllDayvsLimitedTimeWiredvsWireless
©2017MIPIAlliance,Inc.
SystemDesignOverview
5
Intel
2MP-13MP
FE Global Shutter
FE Global Shutter
Active/Passive
Stereo
Data Coalescing
and/or Compression
6DOF and Dense SLAM
Decode + RGB Dewarp + Depth
Stitching
GPU / ISPCVA
GPU/CVA
Gesture Detection
Face/Person/Object Detection and Tracking
ActivitySemantic
SegmentationMaterial
Understanding
CPU/CVA
GPU/CVA
GPU/CVA
3D Reconstruction
OS Services or SDK Plugins
Holo APIUnity Plugins
GPU
MR Application
CompressionGPU/ISP/CVAAcceleration
Active/Passive
Stereo
MIPI/USB
MonocularvsBinocularSLAM
WorkloadPartitioning
GPU/CVAAcceleration
MetadataContainers
IMUI2C
MRCameraSystemDesign
ImagePreProcessingPipe
InsideOut6DOF
Realtimereconstruction
MRWorkloadDistribuitonandRepresentation
Vid
eo S
ee
Thro
ugh
Cam
era
Trac
king
Ca
mer
asD
epth
Ca
mer
as
MIPI/USB
MIPI/USB
MIPI/USB
MIPI/USB
©2017MIPIAlliance,Inc.
ImageSensorsforMR
6
Intel
TypeofSensor Application TypicalSpecification forMIPICSI-2℠ImagingSensors
SystemDesignConsiderations
TrackingSensor Insideout 6DOF WFOV,Fish Eye,GlobalShutter,Monochrome,>120fps,>720p
WiderFOVthan displaytotrackinperiphery
DepthSensor 3DReconstruction,Semantic,Gestures
ActivevsPassive,Upto1080p/30fps,0.3m-5m,fewmmerror
Independent oftrackingsensor
RGBSensor Texture,SeeThroughMode
Upto 13MP/30fpsforhightextureMatchDisplayResolution&fpsforSeeThroughMode(trending4k/120fps)
ConfigurableISPprocessingforsimultaneoushigh
qualitytextureandvideoseethrough
EventSensor Inside outTracking,ObjectTracking
QVGA @2kfps,FOVmatchingdepthcamera
AugmentedwithRGBandDepthcamera
HyperSpectral orThermalSensors
FastClassification RGB-NIR/LWIR Sensor,QVGA@30fps Lowpower ROIoreventgeneration
©2017MIPIAlliance,Inc.
MIPICSI-2SM AdvantagesinMR
7
Intel
• AllowsflexibleISP,VisionAcceleratorandHostSOCcombinationsinMRenablingsegmentationandmobility
VPU->VisionProcessingUnit,typicallyconsistsofCVandNeuralNetworkaccelerators
AllinOneMRHMDwithvisionprocessingdistributedbetweenahostandavision
processingunit
ISP&VPU SOC
MIPIR
x
MIPITx
MIPIR
x
SOCwithISPM
IPIR
x
AllinOneMRHMDwithallvisionprocessingon
hostMRHMDtetheredtoahostPC
MIPIToHost
ConverterSOC
MIPIR
x
HMD Host
USB3orTBT
CSI-2
CSI-2
CSI-2
CSI-2
CSI-2
CSI-2
CSI-2
CSI-2
CSI-2
CSI-2
©2017MIPIAlliance,Inc.
MIPICSI-2AdvantagesinMR
8
Intel
• TotalpowerconsumptiononHMD,especiallyforopticalseethroughorAIOHMDwithMIPICSI-2
ISP&VPU SOC
MIPIR
x
MIPITx
MIPIR
x
CSI-2CameraType Typical Power
DepthCamera 300mW - 1.5W
RGBCamera 300mw- 1W
TrackingCamera 300- 500mw
TotalpowerdissipatedonaAIOHMDcanbeanywherefrom10-25W.Camerapowerstartstoplayasignificantportionofthesame
CSI-2
CSI-2
CSI-2
©2017MIPIAlliance,Inc.
MIPICSI-2AdvantagesinMR
9
Intel
• Easiersynchronizationbetweensensors
SWSynchronizationbetweencamerasandI3CSensorHub
TimeStampingbetweenMIPIframessynchronizedwithhostorSOCHWtimeand
sensorhub
SOCSOC
I2C/I3C℠
HWSync
HWSynchronization
Master
Slave
Slave
CSI-2
CSI-2CSI-2
SPI/MIPII3CIMU
I2C/MIPII3C
HWSyn
c
©2017MIPIAlliance,Inc.
MIPICSI-2FutureOpportunitiesforMR
10
Intel
• CustomcameramodulesforMR– DifferentcombinationsthatcanbeintegratedintoMRHMD’s
• MetadataContainers– StandardizedMetadataforMRvisionanalyticsfromonboardHMDforconsumptionbyhost
• MIPItoUSB/TBT/Wifi ReferenceDesignsforMRHeadsets– EnablesuserstobuyoffmarketlowercostHMD’stobeusedwithPC’sincreasingadoption
• LowpowereventsensingforMR– Standardizedeventtriggersfromeventsensorstofeedintoothersensorsforbetterpower
management• AdoptionofBeyondVisibleCameras
– AdoptionofMIPIbasedThermal,HyperspectralsensorsforMRusecases
©2017MIPIAlliance,Inc.
NOTICESANDDISCLAIMERS• Intel,theIntellogo,Intel® aretrademarksofIntelCorporation oritssubsidiariesintheU.S.and/orothercountries.• *Othernamesandbrandsmaybeclaimedasthepropertyofothers.• Inteltechnologies’featuresandbenefitsdependonsystemconfigurationandmayrequireenabledhardware,softwareorservice
activation.Performancevariesdependingonsystemconfiguration.Nocomputersystemcanbeabsolutelysecure.Checkwithyoursystemmanufacturerorretailerorlearnmoreatintel.com.
• Inteltechnologiesmayrequireenabledhardware,specificsoftware,orservicesactivation.Checkwithyoursystemmanufacturerorretailer.
• Nolicense(expressorimplied,byestoppelorotherwise)toanyintellectualpropertyrightsisgrantedbythisdocument.
11
Intel®
