CS 354Texture Mapping

Mark KilgardUniversity of TexasFebruary 23, 2012

CS 354 2

Today’s material

In-class quiz Lecture topics

Texture mapping Course work

Schedule your Project 1 demos with Randy Reading

Chapter 5, pages 257-296 on Lighting Homework #3

Available on the course web site; announced on Piazza http://www.cs.utexas.edu/~mjk/teaching/cs354_s12/hw3.pdf

Transforms, blending, compositing, color spaces Due Tuesday, February 28 at beginning of class

CS 354 3

My Office Hours

Tuesday, before class Painter (PAI) 5.35 8:45 a.m. to 9:15

Thursday, after class ACE 6.302 11:00 a.m. to 12:00

CS 354 4

Last time, this time

Last lecture, we discussed Finish off compositing Color representation

This lecture Texture mapping

CS 354 5

Daily Quiz

1. Which is not a Porter-Duff compositing mode?

a) Src-Overb) Pass-Throughc) Dst-Overd) Cleare) XOR

2. Multiple choice: Sample patterns for anti-aliasing generally result in better image quality when the sample pattern has

a) more samples on an orthogonal gridb) fewer samples on a jittered gridc) more samples on a jittered gridd) fewer samples on an orthogonal gride) just one sample

3. True or False: The human eye has cones that detect red light, green light, and blue light.

4. Which abbreviation does not refer to a color space?

a) RGBb) XYZc) CYMKd) YIQe) NTSCf) HSL

On a sheet of paper• Write your EID, name, and date• Write #1, #2, #3, #4 followed by its answer

CS 354 6

Texture Supplies Detail to Rendered Scenes

Without texture

With texture

CS 354 7

Textures Make Graphics Pretty

Texture → detail,

detail → immersion,

immersion → fun

Unreal Tournament

Microsoft Flight Simulator X

Sacred 2

CS 354 8

Textured Polygonal Models

++

ResultResult

Key-frameKey-framemodelmodelgeometrygeometry DecalDecal

skinskin

CS 354 9

Multiple Textures forInvolved Shading

Key-frameKey-framemodelmodelgeometrygeometry

DecalDecalskinskintexturetexture

BumpBumpskinskintexturetexture

GlossGlossskinskintexturetexture

++

CS 354 10

Shaders Often Combine Multiple Textures

(modulate)

=

lightmaps onlylightmaps only

decal onlydecal only

combined scenecombined scene* Id Software’s Quake 2

circa 1997

CS 354 11

Projected Texturing for Shadow Mapping

Depth map from light’s point of viewis re-used as a texture andre-projected into eye’s view to generate shadows

lightposition

without shadows with shadows

“what thelight sees”

CS 354 12

Shadow Mapping Explained

Planar distance from lightPlanar distance from light Depth map projected onto sceneDepth map projected onto scene

≤≤ ==lesslessthanthan

True “un-shadowed” True “un-shadowed” region shown greenregion shown green

equalsequals

CS 354 13

Texture’s Not All Fun and Games

Volume rendering of fluid turbulence,

Lawrence Berkley National Lab

Automotive design, RTT

Seismic visualization, Landmark Graphics

CS 354 14

Texture in the Context of theOpenGL Graphics Pipeline

vertexprocessing

rasterization& fragment

coloring

texture rasteroperations

framebuffer

pixelunpack

pixelpack

vertexpuller

clientmemory

pixeltransfer

glReadPixels / glCopyPixels / glCopyTex{Sub}Image

glDrawPixelsglBitmapglCopyPixels

glTex{Sub}ImageglCopyTex{Sub}Image

glDrawElementsglDrawArrays

selection / feedback / transform feedback

glVertex*glColor*glTexCoord*etc.

blendingdepth testingstencil testingaccumulation

storageaccess

operations

Image (Pixel) Processing

Geometry (Vertex) Processing

CS 354 15

Simple Texture Mapping

glBegin(GL_TRIANGLES); glTexCoord2f(0, 0); glVertex2f(-0.8, 0.8);

glTexCoord2f(1, 0); glVertex2f(0.8, 0.8);

glTexCoord2f(0.5, 1); glVertex2f(0.0, -0.8);glEnd();

++

glTexCoord2f

like glColor4f

but sets “current”

texture coordinate

instead of color

glMultiTexCoord2f

takes texture unit parameter so

glMultiTexCoord2f(GL_TEXTURE0, s,t) same as glTexCoord2f(s,t)

ST = (0,0)

ST = (1,1)

CS 354 16

Texture CoordinatesAssigned at Each Vertex

XYZ = (0,-0.8)ST = (0.5,1)

XYZ = (0.8,0.8)ST = (1,0)

XYZ = (-0.8,0.8)ST = (0,0)

CS 354 17

Loose Ends of Texture Setup

Texture object specification

Fixed-function texture binding and enabling

static const GLubyte

myDemonTextureImage[3*(128*128)] = {

/* RGB8 image data for a mipmapped 128x128 demon texture */

#include "demon_image.h"

};

/* Tightly packed texture data. */

glPixelStorei(GL_UNPACK_ALIGNMENT, 1);

glBindTexture(GL_TEXTURE_2D, 666);

/* Load demon decal texture with mipmaps. */

gluBuild2DMipmaps(GL_TEXTURE_2D, GL_RGB8,

128, 128, GL_RGB, GL_UNSIGNED_BYTE, myDemonTextureImage);

glTexParameteri(GL_TEXTURE_2D,

GL_TEXTURE_MIN_FILTER,

GL_LINEAR_MIPMAP_LINEAR);

glActiveTexture(GL_TEXTURE0);

glTexEnvi(GL_TEXTURE_ENV,

GL_TEXTURE_ENV_MODE, GL_REPLACE);

glEnable(GL_TEXTURE_2D);

glBindTexture(GL_TEXTURE_2D, 666);

gluBuild2DMipmaps

calls glTexImage2D on image, then down-samples iteratively 64x64, 32x32, 16x16, 8x8, 4x4, 2x1, and 1x1 images

(called mipmap chain)

CS 354 18

What happens at every fragment when texturing?

A basic operation called a “texture fetch”

Seems pretty simple…

Given1. An image

2. A position Return the

color of image at position

Fetch at

(s,t) = (0.6, 0.25)

T a

xis

S axis0.0

1.0

0.0

1.0

RGBA Result is

0.95,0.4,0.24,1.0)

CS 354 19

Texture Coordinates Associated with Transformed Vertices

Interpolated over rasterized primitives

parametric coordinates

texture coordinates

world coordinateswindow coordinates

CS 354 20

Texture Resources

Textures are loaded prior to rendering Stored on the GPU in video memory

For speed and bandwidth reasons Each texture is an “object”

Texture object = parameters + texture images In OpenGL, each texture has a GLuint “name” Games may have many thousands of textures

When rendering with a texture Must first bind the texture object

glBindTexture in OpenGL Multiple parallel texture units allow multiple texture

objects to be bound and accessed by shaders

CS 354 21

Where do texture coordinates come from?

Assigned ad-hoc by artist Tedious! Has gift wrapping problem

Computed based on XYZ position Texture coordinate generation (“texgen”) Hard to map to “surface space” Function maps (x,y,z) to (s,t,r,q)

From bi-varite parameterization of geometry Good when geometry

is generated from patches So (u,v) of patch maps to

(x,y,z) and (s,t)

[PTex]

CS 354 22

What’s so hard about a texture fetch?

Filtering Poor quality results if you just return the closest color sample

in the image Bilinear filtering + mipmapping needed

Complications Wrap modes, formats, compression, color spaces, other

dimensionalities (1D, 3D, cube maps), etc. Gotta be quick

Applications desire billions of fetches per second What’s done per-fragment in the shader, must be done per-

texel in the texture fetch—so 8x times as much work! Essentially a miniature, real-time re-sampling kernel

GeForce 480 capable of 42,000,000,000 per second! *

* 700 Mhz clock × 15 Streaming Multiprocessors × 4 fetches per clock

CS 354 23

Anatomy of a Texture Fetch

Filteredtexel vector

TexelSelection

TexelCombination

Texel offsets

Texel data

Texture images

Combination parameters

Texture parameters

CS 354 24

Texture Fetch Functionality (1) Texture coordinate processing

Projective texturing (OpenGL 1.0) Cube map face selection (OpenGL 1.3) Texture array indexing (OpenGL 2.1) Coordinate scale: normalization (ARB_texture_rectangle)

Level-of-detail (LOD) computation Log of maximum texture coordinate partial derivative (OpenGL 1.0) LOD clamping (OpenGL 1.2) LOD bias (OpenGL 1.3) Anisotropic scaling of partial derivatives (SGIX_texture_lod_bias)

Wrap modes Repeat, clamp (OpenGL 1.0) Clamp to edge (OpenGL 1.2), Clamp to border (OpenGL 1.3) Mirrored repeat (OpenGL 1.4) Fully generalized clamped mirror repeat (EXT_texture_mirror_clamp) Wrap to adjacent cube map face (ARB_seamless_cube_map) Region clamp & mirror (PlayStation 2)

CS 354 25

Wrap Modes

Texture image is defined in [0..1]x[0..1] region What happens outside that region? Texture wrap modes say

texture

s

t

GL_CLAMPwrapping

GL_REPEATwrapping

CS 354 26

Projective Texturing Homogenous coordinates support projection

Similar to (x/w,y/w,z/w) But (s/q,t/q,r/q) instead Also used in shadow mapping

Source: Wolfgang [99]Source: Wolfgang [99]

CS 354 27

Cube Map Textures

Instead of one 2D images Six 2D images arranged

like the faces of a cube +X, -X, +Y, -Y, +Z, -Z

Indexed by 3D (s,t,r) un-normalized vector Instead of 2D (s,t) Where on the cube images

does the vector “poke through”?

That’s the texture result

CS 354 28

Environment Mapping viaTexture Cube Maps

Access textureAccess textureby surface reflectionby surface reflectionvectorvector

CS 354 29

More Cube Mapping

CS 354 30

Dynamic Cube Map Textures

Rendered sceneRendered scene

Dynamically Dynamically createdcreatedcube map imagecube map image

Image credit:“Guts” GeForce 2 GTS demo,Thant Thessman

CS 354 31

Texture Arrays

Multiple skins packed in texture array Motivation: binding to one multi-skin texture

array avoids texture bind per objectTexture array index

0 1 2 3 4

0

1

234

Mip

map

lev

el i

nd

ex

CS 354 32

Texture Fetch Functionality (2) Filter modes

Minification / magnification transition (OpenGL 1.0) Nearest, linear, mipmap (OpenGL 1.0) 1D & 2D (OpenGL 1.0), 3D (OpenGL 1.2), 4D (SGIS_texture4D) Anisotropic (EXT_texture_filter_anisotropic) Fixed-weights: Quincunx, 3x3 Gaussian

Used for multi-sample resolves Detail texture magnification (SGIS_detail_texture) Sharpen texture magnification (SGIS_sharpen_texture) 4x4 filter (SGIS_texture_filter4) Sharp-edge texture magnification (E&S Harmony) Floating-point texture filtering (ARB_texture_float, OpenGL 3.0)

CS 354 33

Pre-filtered Image Versions Base texture image is say 256x256

Then down-sample 128x128, 64x64, 32x32, all the way down to 1x1

Trick: When sampling the texture, pixel the mipmap level with the closest mapping of pixel to texel size

Why? Hardware wants to sample just a small (1 to 8) number of samples for every fetch—and want constant time access

CS 354 34

Mipmap Texture Filtering

E. Angel and D. Shreiner: Interactive Computer Graphics 6E © Addison-Wesley 2012

pointsampling

mipmapped pointsampling

mipmapped linear filtering

linear filtering

CS 354 35

Anisotropic Texture Filtering Standard (isotropic) mipmap LOD selection

Uses magnitude of texture coordinate gradient (not direction) Tends to spread blurring at shallow viewing angles

Anisotropic texture filtering considers gradients direction Minimizes blurring

Isotropic Anisotropic

CS 354 36

Texture Fetch Functionality (3)

Texture formats Uncompressed

Packing: RGBA8, RGB5A1, etc. (OpenGL 1.1) Type: unsigned, signed (NV_texture_shader) Normalized: fixed-point vs. integer (OpenGL 3.0)

Compressed DXT compression formats (EXT_texture_compression_s3tc) 4:2:2 video compression (various extensions) 1- and 2-component compression (EXT_texture_compression_latc,

OpenGL 3.0) Other approaches: IDCT, VQ, differential encoding, normal maps,

separable decompositions Alternate encodings

RGB9 with 5-bit shared exponent (EXT_texture_shared_exponent) Spherical harmonics Sum of product decompositions

CS 354 37

Texture Fetch Functionality (4)

Pre-filtering operations Gamma correction (OpenGL 2.1)

Table: sRGB / arbitrary Shadow map comparison (OpenGL 1.4)

Compare functions: LEQUAL, GREATER, etc. (OpenGL 1.5) Needs “R” depth value per texel

Palette lookup (EXT_paletted_texture) Thresh-holding

Color key Generalized thresh-holding

CS 354 38

Color Space Decoding During the Texture Fetch for sRGB

Problem: PC display devices have non-linear (sRGB) display gamut Color shading, filtering, and blending with linear math looks bad

Conventional

rendering(uncorrect

ed color)

Gamma correct(sRGB rendered)

Softer andmore natural

Unnaturallydeep facial

shadows

NVIDIA’s Adriana GeForce 8 Launch Demo

CS 354 39

Texture Fetch Functionality (5) Optimizations

Level-of-detail weighting adjustments Mid-maps (extra pre-filtered levels in-between existing levels)

Unconventional uses Bitmap textures for fonts with large filters (Direct3D 10) Rip-mapping Non-uniform texture border color Clip-mapping (SGIX_clipmap) Multi-texel borders Silhouette maps (Pardeep Sen’s work)

Shadow mapping Sharp piecewise linear magnification

CS 354 40

Phased Data Flow

Must hide long memory read latency between Selection and Combination phases

Memoryreads for samples

FIFOing of combination parameters

Filteredtexel vector

TexelSelection

TexelCombination

Texel offsets

Texel data

Texture images

Combination parameters

Texture parameters

Texture coordinate

vector

CS 354 41

What really happens?

Let’s consider a simple tri-linear mip-mapped 2D projective texture fetch

Logically one shader instructionfloat4 color = tex2Dproj(decalSampler, st);

TXP o[COLR], f[TEX3], TEX2, 2D;

Logically Texel selection Texel combination

How many operations are involved?

Assembly instruction

(NV_fragment_program)

High-level language

statement (Cg/HLSL)

CS 354 42

Medium-Level Dissectionof a Texture Fetch

Converttexel

coordsto

texeloffsets

integer /fixed-point

texelcombination

texel offsets

texel data

texture images

combinationparameters

interpolatedtexture

coords vector

texture parameters

Converttexturecoords

totexel

coords

filteredtexelvector

texel coords floor /

frac

integercoords & fractional

weights

floating-pointscaling

andcombination

integer /fixed-pointtexelintermediates

CS 354 43

Interpolation First we need to interpolate (s,t,r,q) This is the f[TEX3] part of the TXP instruction Projective texturing means we want (s/q, t/q)

And possible r/q if shadow mapping In order to correct for perspective, hardware actually interpolates

(s/w, t/w, r/w, q/w) If not projective texturing, could linearly interpolate inverse w (or 1/w)

Then compute its reciprocal to get w Since 1/(1/w) equals w

Then multiply (s/w,t/w,r/w,q/w) times w To get (s,t,r,q)

If projective texturing, we can instead Compute reciprocal of q/w to get w/q Then multiple (s/w,t/w,r/w) by w/q to get (s/q, t/q, r/q)

Observe projective texturing is same cost as perspective correction

CS 354 44

Interpolation Operations

Ax + By + C per scalar linear interpolation 2 MADs

One reciprocal to invert q/w for projective texturing Or one reciprocal to invert 1/w for perspective texturing

Then 1 MUL per component for s/w * w/q Or s/w * w

For (s,t) means 4 MADs, 2 MULs, & 1 RCP (s,t,r) requires 6 MADs, 3 MULs, & 1 RCP

All floating-point operations

CS 354 45

Texture Space Mapping

Have interpolated & projected coordinates Now need to determine what texels to fetch

Multiple (s,t) by (width,height) of texture base level Could convert (s,t) to fixed-point first

Or do math in floating-point Say based texture is 256x256 so

So compute (s*256, t*256)=(u,v)

CS 354 46

Mipmap Level-of-detail Selection

Tri-linear mip-mapping means compute appropriate mipmap level

Hardware rasterizes in 2x2 pixel entities Typically called quad-pixels or just quad Finite difference with neighbors to get change in u

and v with respect to window space Approximation to ∂u/∂x, ∂u/∂y, ∂v/∂x, ∂v/∂y Means 4 subtractions per quad (1 per pixel)

Now compute approximation to gradient length p = max(sqrt((∂u/∂x)2+(∂u/∂y)2),

sqrt((∂v/∂x)2+(∂v/∂y)2))one-pixel separation

CS 354 47

Level-of-detail Bias and Clamping

Convert p length to power-of-two level-of-detail and apply LOD bias λ = log2(p) + lodBias

Now clamp λ to valid LOD range λ’ = max(minLOD, min(maxLOD, λ))

CS 354 48

Determine Mipmap Levels andLevel Filtering Weight

Determine lower and upper mipmap levels b = floor(λ’)) is bottom mipmap level t = floor(λ’+1) is top mipmap level

Determine filter weight between levels w = frac(λ’) is filter weight

CS 354 49

Determine Texture Sample Point

Get (u,v) for selected top and bottom mipmap levels Consider a level l which could be either level t or b

With (u,v) locations (ul,vl)

Perform GL_CLAMP_TO_EDGE wrap modes uw = max(1/2*widthOfLevel(l),

min(1-1/2*widthOfLevel(l), u)) vw = max(1/2*heightOfLevel(l),

min(1-1/2*heightOfLevel(l), v)) Get integer location (i,j) within

each level (i,j) = ( floor(uw* widthOfLevel(l)),

floor(vw* ) )border

edge

s

t

CS 354 50

Determine Texel Locations

Bilinear sample needs 4 texel locations (i0,j0), (i0,j1), (i1,j0), (i1,j1)

With integer texel coordinates i0 = floor(i-1/2) i1 = floor(i+1/2) j0 = floor(j-1/2) j1 = floor(j+1/2)

Also compute fractional weights for bilinear filtering a = frac(i-1/2) b = frac(j-1/2)

CS 354 51

Determine Texel Addresses

Assuming a texture level image’s base pointer, compute a texel address of each texel to fetch Assume bytesPerTexel = 4 bytes for RGBA8 texture

Example addr00 = baseOfLevel(l) +

bytesPerTexel*(i0+j0*widthOfLevel(l)) addr01 = baseOfLevel(l) +

bytesPerTexel*(i0+j1*widthOfLevel(l)) addr10 = baseOfLevel(l) +

bytesPerTexel*(i1+j0*widthOfLevel(l)) addr11 = baseOfLevel(l) +

bytesPerTexel*(i1+j1*widthOfLevel(l)) More complicated address schemes are needed for

good texture locality!

CS 354 52

Initiate Texture Reads

Initiate texture memory reads at the 8 texel addresses addr00, addr01, addr10, addr11 for the upper level addr00, addr01, addr10, addr11 for the lower level

Queue the weights a, b, and w Latency FIFO in hardware makes these weights

available when texture reads complete

CS 354 53

Phased Data Flow

Must hide long memory read latency between Selection and Combination phases

Memoryreads for samples

FIFOing of combination parameters

Filteredtexel vector

TexelSelection

TexelCombination

Texel offsets

Texel data

Texture images

Combination parameters

Texture parameters

Texture coordinate

vector

CS 354 54

Texel Combination When texels reads are returned, begin filtering

Assume results are Top texels: t00, t01, t10, t11 Bottom texels: b00, b01, b10, b11

Per-component filtering math is tri-linear filter RGBA8 is four components

result = (1-a)*(1-b)*(1-w)*b00 + (1-a)*b*(1-w)*b*b01 + a*(1-b)*(1-w)*b10 + a*b*(1-w)*b11 + (1-a)*(1-b)*w*t00 + (1-a)*b*w*t01 + a*(1-b)*w*t10 + a*b*w*t11;

24 MADs per component, or 96 for RGBA Lerp-tree could do 14 MADs per component, or 56 for RGBA

CS 354 55

Total Texture Fetch Operations

Interpolation 6 MADs, 3 MULs, & 1 RCP (floating-point)

Texel selection Texture space mapping

2 MULs (fixed-point) LOD determination (floating-point)

1 pixel difference, 2 SQRTs, 4 MULs, 1 LOG2 LOD bias and clamping (fixed-point)

1 ADD, 1 MIN, 1 MAX Level determination and level weighting (fixed-point)

1 FLOOR, 1 ADD, 1 FRAC Texture sample point

4 MAXs, 4 MINs, 2 FLOORs (fixed-point) Texel locations and bi-linear weights

8 FLOORs, 4 FRACs, 8 ADDs (fixed-point) Addressing

16 integer MADs (integer) Texel combination

56 fixed-point MADs (fixed-point)

Assuming a fixed-point RGBAtri-linear mipmap filtered projective texture fetch

CS 354 56

Intel’s Larrabee Design Recognized the Texture Fetch’s Complexity

Original intended to be amulti-core x86-basedgraphics architecture

Texture filtering still most commonly uses 8-bit color components, which can be filtered more efficiently in dedicated logic than in the 32-bit wide VPU lanes.

Efficiently selecting unaligned 2x2 quads to filter requires a specialized kind of pipelined gather logic.

Loading texture data into the VPU for filtering requires an impractical amount of register file bandwidth.

On-the-fly texture decompression is dramatically more efficient in dedicated hardware than in CPU code.”

—Larrabee: A Many-Core x86 Architecture for Visual Computing [2008]

“Larrabee includes texture filter logic because this operation cannot be efficiently performed in software on the cores. Our analysis shows that software texture filtering on our cores would take 12x to 40x longer than our fixed function logic, depending on whether decompression is required. There are four basic reasons:

CS 354 57

Take Away Information Texture mapping “bridges” geometry processing

and image processing The GPU texture fetch is about two orders of

magnitude more complex than the most complex CPU instruction And texture fetches are extremely common Dozens of billions of texture fetches are expected by

modern GPU applications Texturing is not just a graphics thing

Using CUDA, you can access textures from within your compute- and bandwidth-intensive parallel kernels

CS 354 58

Next Lecture

Lighting computations How can simulate how light interacts with surface appearance?

As usual, expect a short quiz on today’s lecture

Assignments Schedule your Project 1 demos with Randy Reading

Chapter 5, pages 257-296 on Lighting Homework #3

Available on the course web site; announced on Piazza http://www.cs.utexas.edu/~mjk/teaching/cs354_s12/hw3.pdf

Transforms, blending, compositing, color spaces Due Tuesday, February 28 at beginning of class