Slide 1

ShadowDraw Real-Time User Guidance for Freehand Drawing Larry Zitnick, Michael Cohen Microsoft Research Yong Jae Lee U. of Texas at Austin Slide 2 Drawing Subject 1 Subject 2 We need help.. Slide 3 Tracing Difficult to find photos Limits creativity Slide 4 Drawing a face Learn a set of rules Requires significant training Slide 5 Our Idea: ShadowDraw ShadowDraw Slide 6 Creating shadows = Shadow Shadow gives gist of many images simultaneously + Collection of images Slide 7 Creating shadows What if the only input is the users drawing? Shadow Drawing Requires partial matching and dynamic updates Slide 8 Speed Real-time updating of shadows is critical 0.1 seconds feels right 0.5 seconds useful 2 seconds useless Slide 9 Overview User Drawing For each sub-window Histogram ( ID x d x x d y ) Top 100 Fine Alignment Spatial Scoring Shadow Verify Query time (online) Database ImageEdges For each sub-window Sketch, Img_ID, x, y . Sketch, Img_ID, x, y Min Hash Database (offline) Slide 10 Database Slide 11 Efficient (sub-linear) retrieval Stores image index and offset Inverted file structure: [Zitnick, ECCV 2010] Edge descriptor: Position Orientation Linear length Database ImageEdges For each sub-window Sketch, Img_ID, x, y . Sketch, Img_ID, x, y Min Hash Database (offline) Slide 12 Matching Second stage: Refine alignment and compute blending weights First stage: Use inverted file structure to find candidate set User Drawing For each sub-window Histogram ( ID x d x x d y ) Top 100 Fine Alignment Spatial Scoring Shadow Verify Query time (online) Slide 13 Fine alignment Hough transform (3D)Break into three 1D problems: YY XX XX YY S (scale) SS User Drawing For each sub-window Histogram ( ID x d x x d y ) Top 100 Fine Alignment Spatial Scoring Shadow Verify Slide 14 Blending weights = Shadows are the composite of many images * * + Pen Strokes Top matchesWeights Slide 15 Blending weights = * * + Shadows are the composite of many images Top matches Weights Pen Strokes Slide 16 Rendering + Pen Strokes Pen PositionShadow * + = Output Higher contrast near pen position Main focus is users drawing Slide 17 Related work Interactive drawing interfaces Teddy [Igarashi et al., 1999] Fluid Sketches [Arvo and Novins, 2000] 3D drawing system [Igarashi and Hughes, 2000] iCanDraw [Dixon et al., 2010] Drawing studies Where do people draw lines? [Cole et al., 2008] Slide 18 User studies 30,000 images, 20 categories 16 drawers (8 men, 8 women), 8 evaluators 5 objects (shoe, face, bicycle, butterfly, rabbit) Slide 19 User studies Good drawers With ShadowDraw Without ShadowDraw Slide 20 User studies Bad drawers With ShadowDraw Without ShadowDraw Slide 21 User studies Average drawers With ShadowDraw Without ShadowDraw Slide 22 User studies Average drawers With ShadowDraw Without ShadowDraw Slide 23 User studies Significant improvement for Average group Slide 24 User studies Improvement for most object categories Rabbit is control variable Slide 25 User studies After training all users improved: Poor skillGood skill Subjects personal style is maintained! Slide 26 Future work Matching against drawings and not photographs Temporal strokes Color, shading, etc. Drawing priors Not future work: Changing peoples strokes Slide 27 Live demo