full body 3d scanning team kippa sam calabrese, abhishek gandhi, changyin zhou {smc2171, asg2160,...
TRANSCRIPT
Full Body 3D ScanningFull Body 3D ScanningTeam KIPPA
Sam Calabrese, Abhishek Gandhi, Changyin Zhou
{smc2171, asg2160, cz2166}@columbia.edu
OutlineOutline
BackgroundBackgroundMotivationMotivationOur PlanOur PlanData CaptureData CaptureData ProcessingData ProcessingResult ComparisonResult ComparisonDiscussionDiscussion
BackgroundBackground Laser Scanner using TOF
High precision, long range, slow Laser Scanner using triangulation method
High precision, smaller range, occlusion problem, slow Image-based method using triangulation method
(motion, stereo, focus/defocus, shading…) Relatively low precision and resolution Pose assumption to the surface Fast (can be real-time)
.. With Structured Light Improve the precision and resolution Indoor
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
MotivationMotivation
Lots of people dream to have an accurate 3D Lots of people dream to have an accurate 3D model of themselvesmodel of themselves
Lots of applications with the real 3D model Lots of applications with the real 3D model (Animation, Augmented Realistic and even (Animation, Augmented Realistic and even Clothes Design…)Clothes Design…)
DifficultiesDifficulties Laser scanner – too slow to scan a live person Laser scanner – too slow to scan a live person
(moving and non-rigid)(moving and non-rigid) Image-based method – not enough resolution and Image-based method – not enough resolution and
precision (even with structured light)precision (even with structured light) Laser may hurt the eyesLaser may hurt the eyes
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
Our PlanOur Plan
Image-based method to model the headImage-based method to model the headLaser scanner to capture the bodyLaser scanner to capture the body
Proper experiment settings to minimize the Proper experiment settings to minimize the model movement during the scanningmodel movement during the scanning
Proper post-process to tolerant slight Proper post-process to tolerant slight movementmovement
Software to merge them togetherSoftware to merge them togetherMap skins and do animation afterwardMap skins and do animation afterward
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
Data Capture - HeadData Capture - Head
FaceGen: www.facegen.comUse face symmetric, a large set of 3D Face models
(gender, ages, races..)
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
Data Capture -BodyData Capture -Body A professional model + a large private room + 4 hoursA professional model + a large private room + 4 hours Tripods to rest the arms (10 cm lower than the shoulders)Tripods to rest the arms (10 cm lower than the shoulders) Mark the feet positionMark the feet position A camera to track the movementA camera to track the movement Scanner is as high as the shoulderScanner is as high as the shoulder The distance to scanner ranges from 2.5m to 3mThe distance to scanner ranges from 2.5m to 3m
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
Data Capture –Body (cont’d)Data Capture –Body (cont’d) Four scan points (Frontal/Back x Left/Right) Four scan points (Frontal/Back x Left/Right) Ten targets around for further registrationTen targets around for further registration Precision set to 2mmPrecision set to 2mm ≈≈10 min for each scan, but a long time to move the scanner10 min for each scan, but a long time to move the scanner 90 – 130 K vertices for each range data90 – 130 K vertices for each range data
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
Data Capture –Body (cont’d)Data Capture –Body (cont’d)
Registration with CycloneRegistration with Cyclone The body movement causes The body movement causes
big problemsbig problems we need more stronger non-we need more stronger non-
rigid mesh merging methodsrigid mesh merging methods
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
MeshLabMeshLab
Initially used to Initially used to convert PTX to PLYconvert PTX to PLY
Problems arose with Problems arose with the original PTX files the original PTX files from Cyclonefrom Cyclone
Used MeshLab to Used MeshLab to reorientreorient
Used again to clean Used again to clean data and resurface data and resurface after VRip surfacingafter VRip surfacing
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
ScanalyzeScanalyze
Used to re-register the Used to re-register the Range scans following Range scans following the initial errorsthe initial errors
Prepares for VRip directly Prepares for VRip directly by saving out .conf by saving out .conf and .xf files which VRip and .xf files which VRip reads to orient the reads to orient the different scans to each different scans to each other while maintaining other while maintaining their original orientationtheir original orientation
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
VRipVRip
Used to create a surface from the registered point cloudUsed to create a surface from the registered point cloud Uses view direction and ramp weights to get confidence Uses view direction and ramp weights to get confidence
about each vertexabout each vertex Sampled at .002m in each direction per voxelSampled at .002m in each direction per voxel Used ramp weight of .004m with slight increase in Used ramp weight of .004m with slight increase in
standard weights standard weights
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
PlyCrunch and Meshlab (again)PlyCrunch and Meshlab (again)
PlyCrunch is Packaged with VRipPlyCrunch is Packaged with VRipDecimated the Mesh from 3.5 million Decimated the Mesh from 3.5 million
polygons to just over 10,000polygons to just over 10,000Full of holes because Plycrunch deleted Full of holes because Plycrunch deleted
Triangles, but left proper vertices.Triangles, but left proper vertices.
Meshlab used to clean and resurface Meshlab used to clean and resurface resulting mesh, filled most of the holes resulting mesh, filled most of the holes
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
3dsMax3dsMax
Used to finalize mesh by capping the remaining Used to finalize mesh by capping the remaining holes and smoothing the result.holes and smoothing the result.
Used to attach the head from facegenUsed to attach the head from facegen Attached a skin shader for Mental Ray, one of Attached a skin shader for Mental Ray, one of
the built in Renderers which used Sub-Surface the built in Renderers which used Sub-Surface Scattering for realism, modified to match the Scattering for realism, modified to match the texture from FaceGentexture from FaceGen
Rigged with Biped objectRigged with Biped object Animated with stock Motion Capture DataAnimated with stock Motion Capture Data Rendered into animationsRendered into animations
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
3dsMax3dsMaxBackground | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
Result ComparisonResult Comparison
We compared our output with that of two We compared our output with that of two professional companies :professional companies : Headus – based in AustraliaHeadus – based in Australia AvatarMe (initially developed by University of Surrey)AvatarMe (initially developed by University of Surrey)
Criteria for Comparison :Criteria for Comparison : Resolution of the outputResolution of the output Accuracy of scanned dataAccuracy of scanned data Error prevention during scanningError prevention during scanning Cost/Ease of setupCost/Ease of setup
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
Resolution of the outputResolution of the output
We made way for Quality via QuantityWe made way for Quality via Quantity KIPPAKIPPA
Number of Polygons in original scanNumber of Polygons in original scan Scan 1 : 168,855Scan 1 : 168,855 Scan 2 : 179,868Scan 2 : 179,868 Scan 3 : 174,686Scan 3 : 174,686 Scan 4 : 101,010Scan 4 : 101,010 Total number of points : 500,000Total number of points : 500,000
Data Density : 2mmData Density : 2mm
OthersOthers Data Density 4 mm (Headus)Data Density 4 mm (Headus) Average number of points : 300,000Average number of points : 300,000
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
Resolution of the outputResolution of the output
AvatarMeAvatarMe KIPPA KIPPA
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
Accuracy of Scanned DataAccuracy of Scanned Data(smoothness, details captured)(smoothness, details captured)
Used more than just one good software –Used more than just one good software –
Vrip- 3,858,996
MeshLab-3,858,783
PlyCruch-10,019
MeshLab-15,056
3ds Max-15,080
93,108
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
Error prevention during scanning &Error prevention during scanning &Cost/Ease of setupCost/Ease of setup
KIPPAKIPPA We didn’t have a pre-defined model/shape against We didn’t have a pre-defined model/shape against
which to map our data.(explain)which to map our data.(explain) Cheap/Simple set-upCheap/Simple set-up
Separate scanning for head and bodySeparate scanning for head and body Limited body movement below the neckLimited body movement below the neck
OthersOthersWBX Platform(explain)WBX Platform(explain)
(movable platform, cables, etc.)(movable platform, cables, etc.)http://www.cyberware.com/documentation/digisize/www/info/http://www.cyberware.com/documentation/digisize/www/info/
WBXPlatform.htmlWBXPlatform.html
Background | Motivation | Our Plan | Data Capture | Data Processing | Result Comparison
Discussion & ConclusionDiscussion & Conclusion Reduction and dealing with motion…Reduction and dealing with motion… Repositioning the scanner was difficult…Repositioning the scanner was difficult… Problems during transformation…Problems during transformation… Efficient use of markers…Efficient use of markers… Ignored hair…Ignored hair… Missed details of hands and toes…Missed details of hands and toes…
To conclude, To conclude,
3D human scan was a very interesting problem to deal with. We 3D human scan was a very interesting problem to deal with. We managed to get a satisfying output in a very efficient manner.managed to get a satisfying output in a very efficient manner.
It was enjoyable learning and using softwares such as Vrip, It was enjoyable learning and using softwares such as Vrip, Cyclone, MeshLab, PlyCrunch and 3ds Max.Cyclone, MeshLab, PlyCrunch and 3ds Max.
References & ResourcesReferences & Resources ALLEN, P., 2007. 3d photography 2007 fall. Class notes on Active 3D Sensing. ALOIMONOS, Y., AND SPETSAKIS, M. 1989. A unified theory of structure from motion. BERALDIN, J., BLAIS, F., COURNOYER, L., GODIN, G., AND RIOUX, M. 2000. Active 3D Sensing. Modelli
E Metodi per lo studio e la conservazione dell’architettura storica, 22–46. BLAIS, F., PICARD, M., AND GODIN, G. 2004. Accurate 3d acquisition of freely moving objects. In 3DPVT04,
422–429. BLAIS, F. 2004. Review of 20 years of range sensor development. Journal of Electronic Imaging 13, 231. DHOND, U., AND AGGARWAL, J. 1989. Structure from stereo-a review. Systems, Man and Cybernetics,
IEEE Transactions on 19, 6, 1489–1510. DU, H., ZOU, D., AND CHEN, Y. Q. 2007. Relative epipolar motion of tracked features for correspondence in
binocular stereo. In IEEE International Conference on Computer Vision (ICCV). NAYAR, S., WATANABE, M., AND NOGUCHI, M. 1996. Realtime focus range sensor. IEEE Transactions on
Pattern Analysis and Machine Intelligence 18, 12, 1186–1198. RUSINKIEWICZ, S., HALL-HOLT, O., AND LEVOY, M. 2002. Real-time 3D model acquisition. Proceedings of
the 29th annual conference on Computer graphics and interactive techniques, 438–446. SCHECHNER, Y., AND KIRYATI, N. 2000. Depth from Defocus vs. Stereo: How Different Really Are They?
International Journal of Computer Vision 39, 2, 141–162. WATANABE, M., AND NAYAR, S. 1998. Rational Filters for Passive Depth from Defocus. International
Journal of Computer Vision 27, 3, 203–225. ZHANG, R., TSAI, P., CRYER, J., AND SHAH, M. 1999. Shape from shading: A survey. IEEE Transactions
on Pattern Analysis and Machine Intelligence 21, 8, 690–706. Facegen: http://www.facegen.com, Meshlab: http://meshlab.sourceforge.net/ Vrip, Scanalyze, Plycrunch: http://graphics.stanford.edu/software/vrip/, 3Ds Max: www.autodesk.com/3dsmax Prometheus : http://personal.ee.surrey.ac.uk/Personal/A.Hilton/research/PrometheusResults/index.htmlPrometheus : http://personal.ee.surrey.ac.uk/Personal/A.Hilton/research/PrometheusResults/index.html Headus : Headus : www.www.headusheadus.com/au/3D_.com/au/3D_scanscans/index.html s/index.html TC Square : http://www.tc2.com/what/bodyscan/index.htmlTC Square : http://www.tc2.com/what/bodyscan/index.html Cornell University Body Scan : http://www.bodyscan.human.cornell.edu/scene0037.htmlCornell University Body Scan : http://www.bodyscan.human.cornell.edu/scene0037.html
Thanks to Prof. Allen, Karan, Matei and Paul for your kindly help and support.
Special thanks to our model, Daniel, for his professional, passion and great cooperation.
Any Questions…Any Questions…
~Thank You
(team KIPPA)