Model-Driven 3D Content Creation as

Variation

Model-Driven 3D Content Creation as

VariationHao (Richard) Zhang – 张皓

GrUVi Lab, Simon Fraser University (SFU)

Talk @ HKUST, 04/21/11

Hao (Richard) Zhang – 张皓GrUVi Lab, Simon Fraser University (SFU)

Talk @ HKUST, 04/21/11

TAUTAU ZJUZJUNUDTNUDTSFUSFU

3D content creation

Inspiration a readily usable digital 3D modelInspiration a readily usable digital 3D model

Inspiration?Inspiration?

Realistic reconstruction

• Inspiration = real-world data

[Nan et al., SIGGRAPH 2010][Nan et al., SIGGRAPH 2010]

Creative inspiration

•Creation of novel 3D shapes

• Inspiration = design concept, mental

picture, …

sketchsketch

High demand in VFX, games, simulation, VR, …

High demand in VFX, games, simulation, VR, …

3D content creation is hard

•2D-to-3D: an ill-posed problem

▫Shape from shading, sketch-based modeling, …

•Creation from scratch is hard: job for skilled

artistsOne of the most central problems in

graphics; One of the most discussed at SIG’10 panel

One of the most central problems in graphics; One of the most discussed at

SIG’10 panel

Usable 3D content even harder

•Models created are meant for subsequent

use

•Creation of readily usable 3D models

Usable 3D content even harder

•Models created are meant for subsequent use

•Creation of readily usable 3D models

•Higher-level information beyond low-level mesh

▫Part or segmentation information

▫Structural relations between parts

▫Correspondence to relevant models, etc.Hard shape analysis problems!Hard shape analysis problems!

Key: model reuse

•Reuse existing 3D models and associated

info

•Model-driven approach: creation is driven

by or based on existing (pre-analyzed)

models

Key: model reuse

•Reuse existing 3D models and associated

info

•Model-driven approach: creation is driven

by or based on existing (pre-analyzed)

models

•Two primary modes of reuse:

▫New creation via part composition

Key: model reuse

•Reuse existing 3D models and associated info

•Model-driven approach: creation is driven by

or based on existing (pre-analyzed) models

•Two primary modes of reuse:

▫New creation via part composition

▫New creation as variation or modification of

existing model(s), e.g., a warp or a

deformation

Modeling by example

•New models composed by parts retrieved

from an existing data repository

•Key: retrieve relevant parts

•Many variants …

[Funkhouser et al., SIGGRAPH 2004][Funkhouser et al., SIGGRAPH 2004]

Pros and cons

•Pros:▫Significant deviation from existing models

▫Exploratory modeling via part suggestions

[Chaudhuri & Koltun., SIG Asia 2010][Chaudhuri & Koltun., SIG Asia 2010]

Pros and cons

•Pros:▫Significant deviation from existing models

▫Exploratory modeling with part suggestions

•Cons:▫Are models composed by parts readily

usable?

Pros and cons

•Pros:▫Significant deviation from existing models

▫Exploratory modeling with part suggestions

•Cons:▫Are models composed by parts readily

usable? structure lost by part composition; how to stitch?

Pros and cons

•Pros:▫Significant deviation from existing models

▫Exploratory modeling with part suggestions

•Cons:▫Are models composed by parts readily

usable? structure lost by part composition; how to stitch?

▫Does part exploration always reflect user design intent?

Model-driven creation as variation

•New creation as variation of existing model(s)

Enrich a set; generate “more of

the same” …

Enrich a set; generate “more of

the same” …

Photo-inspired 3D model creation

Photo-inspired 3D model creation

Inspiration = photographsInspiration = photographs

Inspiration = a model set

Inspiration = a model set

Model-driven creation as variation

•New creation as variation of existing model(s)

Enrich a set; generate “more of

the same” …

Enrich a set; generate “more of

the same” …

Inspiration = a model set

Inspiration = a model set

Style-Content Separation by Anisotropic Part Scales

Kai Xu1,2, Honghua Li2, Hao Zhang2, Daniel Cohen-Or3

Yueshan Xiong2, and Zhi-Quan Cheng2

1Simon Fraser Universtiy 2National Univ. of Defense Tech. 3Tel-Aviv University

Motivation

•Enrich a set of 3D models with their derivatives

Set belongs to the same family or class

Set belongs to the same family or class

Variations in shape parts in the set

Geometric or content difference

Part proportion (= style) difference

?

Style transfer as a derivative

Part proportion style

?

Style transfer as a derivative

Part proportion style

Difficulty with style transfer

•Style transfer needs part correspondence

•Part correspondence is difficult

▫Unsupervised problem

▫Both content and style variations

Variations can be significant!

Work at part and OBB level

Parts enclosed and characterized by tight oriented bounding boxes (OBBs)Parts enclosed and characterized by tight oriented bounding boxes (OBBs)

Style content separation

•To address both shape variations in the set▫Separate treatment of “style” and “content”

Style 1

Style 2

Style 3

ContentContent

Sty

leS

tyle

Style transfer as a derivative

•Creation = filling in the style-content table

Style vs. content

•Fundamental to human perception

Content Style

Language Words Accents

Text Letters Fonts

Human face Identities Expressions

Style content separation

•Previous works on faces, motion, etc.

▫Prerequisite: data correspondence

▫Correspondence dealt with independently

▫Correspondence itself is the very challenge!

Our approach

• One particular style:

Anisotropic part scales or part proportions

Our approach

• One particular style:

Anisotropic part scales or part proportions

• The approach:

Style-content separation with style

clustering in a correspondence-free way

Algorithm overview

•Pipeline

Style clustering Co-segmentation Inter-style part correspondence

Contentclassification

Anisotropic part scales

•Measure style distance between two shapes

Anisotropic part scales

•Measure style distance between two shapes

Part OBB graphs of

given segmentatio

n

Anisotropic part scales

•Measure style distance between two shapes

Computestyle

signatures

……

Part OBB graphs of

given segmentatio

n

Anisotropic part scales

•Measure style distance between two shapes

……

Part OBB graphs of

given segmentatio

n

Euclideandistance

Computestyle

signatures

Style distance issues

•Unknown segmentation

•Unknown correspondence

?

?

Style distance

•Search over all part compositions and part counts

……

……

Style distance

•For each part count, find minimal distance

……

……

A good signature will return min distance across all part counts to reflect corresponding part decompositions …

Correspondence-free style signature

Binary relations: difference of part scales between adjacent OBBs

Use Laplacian graph spectra:

OBB graph

Correspondence-free style signature

Unary attributes: anisotropy of parts

Use Laplacian graph spectra:

OBB graph linear planar spherical

Graph spectra is permutation-free

Style clustering

•Spectral clustering using style distances

Pipeline

Style clustering Co-segmentation Inter-style part correspondence

Contentclassification

Co-segmentation

•Approach:▫Consistent segmentation [Golovinskiy & Funkhouser,

SMI 09]

▫ Initial guess: global alignment (ICP)

[Golovinskiy & Funkhouser 09]

Co-segmentation

•Approach:▫Consistent segmentation [Golovinskiy & Funkhouser, SMI 09]

▫ Initial guess: global alignment (ICP)

•We co-segment within a style cluster▫Removing non-homogeneous part scaling from analysis

[Golovinskiy & Funkhouser 09]

Co-segmentation

•Approach:▫Consistent segmentation [Golovinskiy & Funkhouser, SMI 09]

▫ Initial guess: global alignment (ICP)

•We co-segment within a style cluster▫Removing non-homogeneous part scaling from analysis

[Golovinskiy & Funkhouser 09] After style separation

Pipeline

Style clustering Co-segmentation Inter-style part correspondence

Contentclassification

Inter-style part correspondence

•Approach: deform-to-fit

▫Deformation-driven correspondence [Zhang et al., SGP 08]

▫Consider common interactions between OBBs

1D-to-1D 1D-to-2D 2D-to-2D 2D-to-3D

Inter-style part correspondence

•Deform-to-fit: appropriate deformation energy

Pruned priority-driven search

Pipeline

Style clustering Co-segmentation Inter-style part correspondence

Contentclassification

Content classification

•Use Light Field Descriptor [Chen et al. 2003]

•Compare corresponding parts

Part-level LFD Global LFD

Synthesis by style transfer

•OBBs are scaled

•Underlying geometry via space

deformationcontent

style style transfer

Results: hammers

Results: goblets

Results: humanoids

Results: chairs

Pros and cons

•Pros:▫Automatic generation of many variations

▫Unsupervised

▫Deals with anisotropic part scales

▫Variation = part scaling: structure preservation

Pros and cons

•Pros:▫Automatic generation of many variations

▫Unsupervised

▫Deals with anisotropic part scales

▫Variation = part scaling: structure preservation

•Cons:▫Rely on sufficiently good initial segmentations

▫Variation does not create new content

Interesting future work

•Learn and synthesize with generic styles

Model-driven creation as variation

•New creation as variation of existing model(s)

Photo-inspired 3D model creation

Photo-inspired 3D model creation

Inspiration = photographsInspiration = photographs

Photo-inspired 3D modeling

Photo-Inspired Model-Driven 3D Object Modeling

Kai Xu1,2, Hanlin Zheng4, Hao Zhang2, Daniel Cohen-Or3

Ligang Liu4, and Yueshan Xiong2

1NUDT 2SFU 3TAU 4ZJU Conditionally acceptedConditionally accepted

Overview

Input: single photograph + pre-analyzed datasetInput: single photograph + pre-analyzed dataset

Overview

1. Model-driven labelled segmentation of photographed object

1. Model-driven labelled segmentation of photographed object

Overview

2. Choosing of a candidate model from the database

2. Choosing of a candidate model from the database

Overview

3. Silhouette-constrained deform-to-fit of candidate

3. Silhouette-constrained deform-to-fit of candidate

Overview

OutputOutput

Structure preservation

•Any higher-level structural info in the candidate

models is preserved during deform-to-fit

▫Symmetry relations

▫Part-level correspondence in the set

▫Controller structures [Zheng et al. @ HKUST, EG 11]

Structure preservation

•Any higher-level structural info in the candidate

models is preserved during deform-to-fit

▫Symmetry relations

▫Part-level correspondence in the set

▫Controller structures [Zheng et al. @ HKUST, EG 11]

•Structures also serve to constrain deformation of

candidate model

Controller representations

•Controllers: cuboids and generalized cylinders

•Relations: symmetry, proximity, etc.

Fitting primitivesFitting primitives

Controller representations

•Controllers: cuboids and generalized cylinders

•Relations: symmetry, proximity, etc.

Fitting primitivesFitting primitives

Deformation of controllers

photophoto

Controller primitivesController primitives

Deformation of controllers

photophoto candidate modelcandidate model

Controller primitivesController primitives

Deformation of controllers

Result of silhouette-driven deform-to-fit

Result of silhouette-driven deform-to-fit

photophoto candidate modelcandidate model

Structure preservation at work

symmetrysymmetry

Structure preservation at work

symmetrysymmetry

proximityproximity

Structure preservation at work

symmetrysymmetry

proximityproximity

optimizationoptimization

Structure preservation at work

symmetrysymmetry

proximityproximity

optimizationoptimization

outputoutput

Short videoShort video

Results

•Guidance in single view but coherent 3D results

Results

The Google chair challenge

Not just chairs …

Pros and cons

•Pros:▫Photos: immensely rich source of inspiration

▫Silhouette-driven deformation

▫Variation is less “intrusive” to retain high-level info of source model readily usable

Pros and cons

•Pros:▫Photos: immensely rich source of inspiration

▫Silhouette-driven deformation

▫Variation is less “intrusive” to retain high-level info of source model more readily usable

•Cons▫Variation does not create new structures

Future work

•Photo-inspired model deformation only a start

•Further model refinement, e.g., via sketches

Future work

•Photo-inspired model deformation only a start

•Further model refinement, e.g., via sketches

•Model-driven structure modification

Future work

•Photo-inspired model deformation only a start

•Further model refinement, e.g., via sketches

•Model-driven structure modification

•Other inspirations for 3D content creation

▫Sketch-inspired model variation

Future work

•Photo-inspired model deformation only a start

•Further model refinement, e.g., via sketches

•Model-driven structure modification

•Other inspirations for 3D content creation

▫Sketch-inspired model variation

•Style transfer with unknown style in a set

Thank you, 谢谢

TAUTAU ZJUZJUNUDTNUDTSFUSFU