Walk-Through Illustrations:Walk-Through Illustrations:Frame-Coherent Pen-and-Frame-Coherent Pen-and-

Ink Style in a Game EngineInk Style in a Game Engine

Bert GreudenbergBert Greudenberg

Maic MasuchMaic Masuch

Thomas StrothotteThomas Strothotte

It Looks All CrappyIt Looks All Crappy

Cartman: Cartman: Aw, screw it. Aw, screw it. It probably isn't all It probably isn't all that good anyway. that good anyway.

Kyle: Kyle: Cartman! What Cartman! What are you talking are you talking about?! You LOVE about?! You LOVE Terrance and Phillip!Terrance and Phillip!

Cartman: Cartman: Yeah, but the Yeah, but the animation's all crappy animation's all crappy — it probably can't — it probably can't sustain itself over sustain itself over ninety minutes.ninety minutes.

Who wants crappy Who wants crappy animation?animation?

• Tired of the gothic space dungeons?Tired of the gothic space dungeons?

• Tired of “realistic” animation that Tired of “realistic” animation that fools you for five seconds?fools you for five seconds?

• You need a changeYou need a change

• A non-photorealistic change!A non-photorealistic change!

3D Game Engines3D Game Engines

• 3D game engines have 3D game engines have made huge advancesmade huge advances

• Engines primarily still Engines primarily still focus on photorealistic focus on photorealistic imagesimages

• Game images are Game images are technically non technically non photorealisticphotorealistic

NPR Here to StayNPR Here to Stay

• Research has turned Research has turned to NPR of still to NPR of still imagesimages

• Great advances in Great advances in Toon, Painterly, and Toon, Painterly, and Pen & Ink (i.e. South Pen & Ink (i.e. South Park)Park)

• However, animating However, animating NPR or real-time NPR or real-time rendering 2 rendering 2 problems occur.problems occur.

What’s the Problem?What’s the Problem?

• Maintenance of frame-to-frame Maintenance of frame-to-frame coherencecoherence– Hidden Surface RemovalHidden Surface Removal

•Objects appear and disappear correctlyObjects appear and disappear correctly

• Ability of a 3D engine to render NP Ability of a 3D engine to render NP images in real timeimages in real time

Solution!Solution!

• Create an engine that facilitates a Create an engine that facilitates a hybrid rendering pipelinehybrid rendering pipeline

• Allows specification of the rendering Allows specification of the rendering style on a per object basisstyle on a per object basis

More to followMore to follow

Computer Games and NPRComputer Games and NPR

• No games achieves No games achieves true photorealismtrue photorealism

• Rendering style Rendering style can be called can be called photorealisticphotorealistic– Shapes, spatial Shapes, spatial

relationships, relationships, surface detailssurface details

NPR Video GamesNPR Video Games

• Very few games Very few games incorporate NPR incorporate NPR elementselements

• Examples:Examples:– Cartoon RenderingCartoon Rendering– Pen & InkPen & Ink

• Broken ArrowBroken Arrow

Cartoon RenderingCartoon Rendering

• Shading Model in Shading Model in 2D cartoon2D cartoon– Games only use 2D Games only use 2D

data.data.– Visually appealing, Visually appealing,

but not that but not that interactiveinteractive

• Monkey Island IIIMonkey Island III

• Broken SwordBroken Sword

Monkey Island IIIMonkey Island III

Pen and Ink RenderingPen and Ink Rendering

• NPRQuake by GleicherNPRQuake by Gleicher

• Intercepts the calls to Intercepts the calls to graphics librariesgraphics libraries

• Replaces them with Replaces them with different drawing different drawing primitivesprimitives

• Uses core GLQuake Uses core GLQuake engine and maintains engine and maintains gameplaygameplay

NPRQuake SketchNPRQuake Sketch

NPRQuake Blue-PrintNPRQuake Blue-Print

NPRQuake Brush-StrokeNPRQuake Brush-Stroke

Problems with NPRQuakeProblems with NPRQuake

• Achieves non-photorealistic lookAchieves non-photorealistic look

• Style elements do not preserve Style elements do not preserve frame coherenceframe coherence– Outlines of objects vary from frame to Outlines of objects vary from frame to

frameframe– Results in lively but disturbing Results in lively but disturbing

experienceexperience

How to Fix It?How to Fix It?

• Few systems deal with frame to Few systems deal with frame to frame coherence.frame coherence.

• Many systems can create Pen & Ink Many systems can create Pen & Ink illustrationsillustrations– But, probability is used to model the line But, probability is used to model the line

deviations of hand drawingsdeviations of hand drawings•Not suitable for real time rendering in Not suitable for real time rendering in

animationsanimations

Non-Photorealistic Visualization Non-Photorealistic Visualization In Game EngineIn Game Engine

• NP imaging allows the NP imaging allows the user to simplify an user to simplify an image by omitting image by omitting details to control the details to control the viewer’s attentionviewer’s attention– Emphasize, Emphasize,

Deemphasize certain Deemphasize certain elements of a sceneelements of a scene

• Aim for pen-and-ink Aim for pen-and-ink illustrationsillustrations– Silhouettes and creases Silhouettes and creases

are clearly drawnare clearly drawn– Surface detail added Surface detail added

through hatchingthrough hatching

Hybrid Rendering PipelineHybrid Rendering Pipeline

• Able to specify rendering style of each Able to specify rendering style of each objectobject– Gain additional freedom in expression for the Gain additional freedom in expression for the

visualizationvisualization• Photorealistic Images – use normal texture Photorealistic Images – use normal texture

mapping and shadingmapping and shading• NP Images – consist of modified shading, NP Images – consist of modified shading,

an outline, and NP textures (ink maps or an outline, and NP textures (ink maps or hatch maps)hatch maps)

• Hybrid – combine photorealistic rendering Hybrid – combine photorealistic rendering and NP drawing stylesand NP drawing styles

NP Elements in 3D Game NP Elements in 3D Game EngineEngine

• Properties of 3D game engineProperties of 3D game engine– Provide player with highly immersive Provide player with highly immersive

experienceexperience•Enhace speed, realism, and image qualityEnhace speed, realism, and image quality

– Preprocess as much data as possiblePreprocess as much data as possible– Let hardware handle most of the per-Let hardware handle most of the per-

frame loadframe load•Mip-Mapping, anti-aliasingMip-Mapping, anti-aliasing

NP Elements in 3D Game NP Elements in 3D Game EngineEngine

• Non-Photorealism uses image space Non-Photorealism uses image space elementselements– Pixels generated on the screen are not a Pixels generated on the screen are not a

strict projection of the 3D scene objectsstrict projection of the 3D scene objects– A triangle outline rendered as lines has A triangle outline rendered as lines has

a constant width independent of the a constant width independent of the actual distance of the triangle to the actual distance of the triangle to the viewerviewer

OutliningOutlining

• SilhouettesSilhouettes– Drawn to enhance the visual separation of objects from Drawn to enhance the visual separation of objects from

each other and backgroundeach other and background– Silhouette edges need to be determined every frameSilhouette edges need to be determined every frame– Non-Convex edges are never drawn (second)Non-Convex edges are never drawn (second)– Sharp Edges are always drawn (third)Sharp Edges are always drawn (third)– Only smooth convex edges are drawn (fourth)Only smooth convex edges are drawn (fourth)– Result with hidden lines removedResult with hidden lines removed

HatchingHatching

• Textures are supposed to become smaller with Textures are supposed to become smaller with increasing distanceincreasing distance

• HatchingHatching– shade by drawing or etching fine parallel or perpendicular linesshade by drawing or etching fine parallel or perpendicular lines– Pen & Ink hatching textures cause serious moire´e patternsPen & Ink hatching textures cause serious moire´e patterns– In motion this produces even worse artifacts than still imagesIn motion this produces even worse artifacts than still images

• Prevents frame coherencePrevents frame coherence

How to Fix This?How to Fix This?

• Common antidoteCommon antidote– Mip-mappingMip-mapping– Unfortunately, normal filtering process applied to large Unfortunately, normal filtering process applied to large

white areas and thin black lines of hatching texture white areas and thin black lines of hatching texture cause lines to vanishcause lines to vanish

– Visible structure is lostVisible structure is lost

Hatch MapsHatch Maps

• Normal mip-mappingNormal mip-mapping– Lines are black and well Lines are black and well

separated at the top levelseparated at the top level– Become lighter and Become lighter and

closer at lower levelscloser at lower levels

• Hatch MapHatch Map– Equally space black Equally space black

hatch lines for all mip-hatch lines for all mip-map levelsmap levels

– Only very lowest levels Only very lowest levels are gray values used to are gray values used to maintain tonemaintain tone

Hatch MapsHatch Maps

• Unfortunately, all over tone is not Unfortunately, all over tone is not continuouscontinuous

• Abrupt switch between hatch-map levelsAbrupt switch between hatch-map levels

• Use Tri-linear FilteringUse Tri-linear Filtering

Ink MapsInk Maps

• Hatching is a mechanical processHatching is a mechanical process• Hybrid PipelineHybrid Pipeline

– Photorealistic: Mip-MappingPhotorealistic: Mip-Mapping– Non-Photorealistic: Ink MapNon-Photorealistic: Ink Map

• Adding details to surfaces is artistically challengingAdding details to surfaces is artistically challenging• Construct ink mapsConstruct ink maps

– Smooth animation achieved with carefully crafted ink mapsSmooth animation achieved with carefully crafted ink maps– Frame Coherence is maintained using tri-linear filteringFrame Coherence is maintained using tri-linear filtering

What They AchievedWhat They Achieved

• Part of a medieval palace created for the Part of a medieval palace created for the UNREAL engineUNREAL engine

What they AchievedWhat they Achieved

SpecificationsSpecifications

• 3D engine runs on:3D engine runs on:– 750 MHz CPU750 MHz CPU– GeForce2 ultra graphics cardGeForce2 ultra graphics card

• Even on unoptimized models, engine Even on unoptimized models, engine runs at interactive speeds, i.e. 25 fpsruns at interactive speeds, i.e. 25 fps

And Now….Zelda!!!And Now….Zelda!!!

• The Legend of Zelda: The Wind The Legend of Zelda: The Wind WakersWakers

• Right now, this is the pinnacle of NPR Right now, this is the pinnacle of NPR in game engines.in game engines.– First game to max out hardware of First game to max out hardware of

Nintendo GamecubeNintendo Gamecube– Combines great use of interactive 3D Combines great use of interactive 3D

and NPRand NPR

ScreenshotsScreenshots

MoreMore

Zelda Video!!Zelda Video!!

Thank YouThank You

Any Questions?Any Questions?