Practical Implementation of High Dynamic Range

RenderingMasaki Kawase

BUNKASHA GAMESBUNKASHA PUBLISHING CO.,LTD

http://www.bunkasha-games.comhttp://www.daionet.gr.jp/~masa/

Agenda• What can be done with HDR?• Dynamic Range• Implementation on DX8 hardware• Implementation on DX9 hardware• Multiple Gaussian Filters• HDR in games• References

What can be done with HDR?• Dazzling light • Dazzling reflection• Fresnel reflection

– bright reflection in the normal direction• Exposure control effects• Realistic depth-of-field effects• Realistic motion blur

Dazzling Light

Dazzling Reflection

HDR Fresnel

Bright reflectionoff low-reflectance surfaces

Exposure Control

HDR Depth-of-Field

Future perspective

HDR Motion Blur

Future perspective

Dynamic Range• The ratio of the greatest value to the smallest val

ue that can be represented• Displayable image

– 28 Low dynamic range (LDR)• Frame buffer of absolute luminance

– Render the scene in absolute luminance space– >232 represents all luminances directly

• Frame buffer of relative luminance– Apply exposure scaling during rendering– >215~16dark regions are not important

HDR Frame Buffers• For glare generation• When rendering with relative luminances:

– Ideally, more than 215~16

– In games• 212~13 (4,000~10,000) is acceptable

HDR Environment Maps• Very important for representing:

– Realistic specular reflection– Dazzling specular reflection

• Specular reflectance of nonmetals– Reflectance in the normal direction is typically

less than 4%– Bright light remains bright after such low

reflection• To maintain dazzles after reflection of ~1-4%

– Dynamic range of more than 10,000 or 20,000 is necessary

Implementation on DX8 Hardware

• We have no choices• Pixel Shader 1.x

– Integer operations only• HDR buffer formats

– Low-precision buffers only– Use the alpha channel as luminance

information– Fake it to achieve believable appearance

• Accurate calculation is not feasible

Fake HDR Pixel ShaderGlossy reflection material

ps_1_1

tex t0tex t1tex t2

mad r0.rgb, v0, t2, v1 // Scale the primary diffuse color by // shadow/light map, and add the result of // other per-vertex lighting+mul t0.a, v1.a, t0.a // Scale the specular reflectance // by gloss map

mul r0.rgb, t0, r0 // Modulate diffuse color with decal texture+mul r0.a, t0.a, t1.a // r0.a = specular reflectance * envmap luminance

mul t1.rgb, t1, c0 // Modulate envmap with specular color+mul r1.a, r0.a, t1.a // Envmap brightness parameter // r1.a = specular reflectance * envmap luminance * gloss map

lrp r0.rgb, t0.a, t1, r0 // Reflect the envmap by specular reflectance

mul r1.a, r1.a, c0.a // Envmap brightness parameter // r1.a = specular reflectance * envmap luminance * gloss map * Clamp(gloss * 2, 0, 1)

lrp r0.rgb, r1.a, t1, r0 // Output color // Interpolate the envmap color and the // result of LDR computation, based on // the envmap brightness parameter (r1.a)+lrp r0.a, r1.a, t1.a, r0.a // Output luminance information

// v0.rgb : Diffuse color of primary light// v1.rgb : Color for other lights/ambient// pre-scaled by (exposure * 0.5)//// v1.a : Specular reflectance (Fresnel)//// t0.rgb : Decal texture (for diffuse)// t0.a : Gloss map (for specular)// t1.rgb : Envmap color// t1.a : Envmap luminance

// t2.rgb : Shadow/light map//// c0.rgb : Specular color// c0.a : Clamp(gloss * 2, 0, 1)

Generating Displayable Image

• Extract high-luminance regions

Threshold : ~0.4-0.5

• Generate glare– Reference:

• Kawase, Masaki, “Frame Buffer Postprocessing Effects in DOUBLE-S.T.E.A.L (Wreckless)”

• Generate a displayable image– Calculate the luminance from the frame buffer

– Add the result of glare generation to the luminance 216... 2 aBufferrgbBufferrgbBufferLumexposed

aBufferThresholdrgbBufferGlaresrc .

161.

Notes on DX8 Implementation

• Accurate calculation is not feasible– How to make it believable by faking– Based on appearance rather than

theory

Implementation on DX9 Hardware

• There are currently many limitations– Choose implementations accordingly

• Pixel Shader– Pixel Shader 2.0 or later– Pixel Shader 1.x

• Buffer formats for HDR– High-precision integer/float buffers– Low-precision integer buffers

Issues with High-Precision Buffers

• Memory usage– At least twice as much memory as the

conventional full-color buffer is needed• Limitations

– Alpha blending cannot be used– Texture filtering cannot be used with

floating-point formats• Some systems don’t support them• The situation is not good…

Use Low-Precision Buffers• Make use of low-precision buffers

– A8R8G8B8 / A2R10G10B10 etc.– Low memory consumption– Alpha blending can be used

Compression with Tone Mapping

• Render directly to displayable format• Nonlinear color compression

– Effectively wide dynamic range– Reference:

• Reinhard, Erik, Mike Stark, Peter Shirley, and Jim Ferwerda, “Photographic Tone Reproduction for Digital Images”

• The alpha channel is not used– Can be used for any other purpose

1),(),(

),(

yxLum

yxLumyxBuffer

exposed

exposed

Environment Map Formats• Relatively low resolution• Alpha channel/blending is not very important• Use the 16-bit integer format if enough

memory storage is available– Treat it as having an interval of [0, 256] or [0,

512]– Texture filtering can be used

• In the future– Do it all with A16B16G16R16F

Low-Precision Environment Maps• Use them when:

– High-precision buffers are not supported, or– Memory storage is limited

• If the fill-rate of your system is relatively low– Use the same format as used in DX8 fake HDR

• If the fill-rate is high enough:– Nonlinear color compression

• Similar to tone mapping– Store exponents into the alpha channel

• More accurate operations are possible• Using it just as a scale factor is not enough

– Even the DX8 fake HDR has a much bigger impact

Color Compression• Similar to tone mapping• Encode when rendering to an environment map

Offset : luminance curve controlling factor (~2-4)• A bigger offset means:

– High-luminance regions have higher resolutions– Low-luminance regions have Lower resolutions

• Decode when rendering to a frame buffer– From the environment map fetched

: a small value to avoid divide-by-zero

OffsetyxLumyxLum

yxBufferexposed

exposed

),(),(

),(

),(1),(),(

yxBufferOffsetyxBufferyxLumexposed

δ

Color Compression• Use carefully

– Mach banding may become noticeable on reflections of large area light sources

• e.g. Light sky

E8R8G8B8• Store a common exponent for RGB into the alpha c

hannel• Use a base of 1.04 to 1.08

offset : ~64-128

– Base=1.04 means dynamic range of ~23,000 (1.04256)

• A bigger base value means:– Higher dynamic range– Lower resolution (Mach banding becomes noticeable)

offsetαBufferexposed basergbBufferLum 256..

E8R8G8B8 Encoding (HLSL)

// a^n = b#define LOG(a, b) ( log((b)) / log((a)) )

#define EXP_BASE (1.06)#define EXP_OFFSET (128.0)

// Pixel Shader (6 instruction slots)// rgb already exposure-scaledfloat4 EncodeHDR_RGB_RGBE8(in float3 rgb){

// Compute a common exponentfloat fLen = dot(rgb.rgb, 1.0) ;float fExp = LOG(EXP_BASE, fLen) ;

float4 ret ;ret.a = (fExp + EXP_OFFSET) / 256 ;ret.rgb = rgb / fLen ;

return ret ;}

// More accurate encoding#define EXP_BASE (1.04)#define EXP_OFFSET (64.0)

// Pixel Shader (13 instruction slots)float4 EncodeHDR_RGB_RGBE8(in float3 rgb){

float4 ret ;

// Compute a common exponent// based on the brightest color channelfloat fLen = max(rgb.r, rgb.g) ;fLen = max(fLen, rgb.b) ;float fExp = floor( LOG(EXP_BASE, fLen) ) ;

float4 ret ;ret.a = clamp( (fExp + EXP_OFFSET) / 256, 0.0, 1.0 ) ;ret.rgb = rgb / pow(EXP_BASE, ret.a * 256 - EXP_OFFSET) ;

return ret ;}

E8R8G8B8 Decoding// Pixel Shader (5 instruction slots)float3 DecodeHDR_RGBE8_RGB(in float4 rgbe){

float fExp = rgbe.a * 256 - EXP_OFFSET ;float fScale = pow(EXP_BASE, fExp) ;

return (rgbe.rgb * fScaler) ;}

• Encoding/decoding should be done using partial-precision instructions– Rounding errors inherent in the texture

format are much bigger

// If R16F texture format is available,// you can use texture to convert alpha to scale factorfloat3 DecodeHDR_RGBE8_RGB(in float4 rgbe){

// samp1D_Exp: 1D float texture of 256x1// pow(EXP_BASE, uCoord * 256 - EXP_OFFSET)float fScale = tex1D(samp1D_Exp, rgbe.a).r ;

return (rgbe.rgb * fScale) ;}

Rendering with Tone Mapping

Glossy reflection material float4 PS_GlossReflect(PS_INPUT_GlossReflect vIn) : COLOR0{

float4 vDecalMap = tex2D(samp2D_Decal, vIn.tcDecal) ;float3 vLightMap = tex2D(samp2D_LightMap, vIn.tcLightMap) ;

float3 vDiffuse = vIn.cPrimaryDiffuse * vLightMap + vIn.cOtherDiffuse ;vDiffuse *= vDecalMap ;

// HDR-decoding of environment mapfloat3 vSpecular = DecodeHDR_RGBE8_RGB( texCUBE(sampCUBE_EnvMap, vIn.tcReflect) ) ;float3 vRoughSpecular = texCUBE(sampCUBE_DullEnvMap, vIn.tcReflect) ;

float fReflectance = tex2D( samp2D_Fresnel, vIn.tcFresnel ).a ;fReflectance *= vDecalMap.a ;

vSpecular = lerp(vSpecular, vRoughSpecular, fShininess) ;float3 vLum = lerp(vDiffuse, vSpecular, fReflectance) ;

// HDR tone-mapping encodingfloat4 vOut ;vOut.rgb = vLum / (vLum + 1.0) ;vOut.a = 0.0 ;

return vOut ;}

struct PS_INPUT_GlossReflect{ float2 tcDecal : TEXCOORD0 ; float3 tcReflect : TEXCOORD1 ; float2 tcLightMap : TEXCOORD2 ; float2 tcFresnel : TEXCOORD3 ;

// Exposure-scaled lighting results

// Use TEXCOORD to avoid clamping float3 cPrimaryDiffuse : TEXCOORD6 ; float3 cOtherDiffuse : TEXCOORD7 ;} ;

Generating Displayable Image

• Extract high-luminance regions

Threshold : ~0.5-0.8• Divide by (1 - Threshold) to normalize

• Generate glare– Use an integer buffer to apply texture filtering

• Hopefully, a float buffer with filtering…• Generate a displayable image

– Add the glare to the frame buffer

0,1

.maxThreshold

ThresholdrgbBufferGlaresrc

Notes on DX9 Implementation

• High-precision buffers– Consumes a lot of memory– No blending capability

• Low-precision buffers– Pixel shaders are expensive

• Consider fake techniques like DX8– High performance– Low memory consumption– Very effective

Multiple Gaussian Filters• Bloom generation• A single Gaussian filter does not give

very good results– Small effective radius– Not sharp enough around the light position

• Composite multiple Gaussian filters– Use Gaussian filters of different radii– Larger but sharper glare becomes possible

Multiple Gaussian Filters

origin thefrom distance

22 yxr 222

22

)4()2(

)5.0()25.0(

1684

2rrr

rr

eee

ee

2re

Original image 2re222

22

)4()2(

)5.0()25.0(

1684

2rrr

rr

eee

ee

Multiple Gaussian Filters

Multiple Gaussian Filters• A filter of large radius is very expensive• Make use of downscaled buffers

– A large radius means a strong low-pass filter• Apply a blur filter to a low-res version of the image and

magnify it by bilinear filtering The error is unnoticeable• Change the image resolution rather than the filter radius

– 1/4 x 1/4 (1/16 the cost)– 1/8 x 1/8 (1/64 the cost)– 1/16 x 1/16 (1/256 the cost)– 1/32 x 1/32 (1/1024 the cost)– …

• Even a large filter of several hundred pixels square can be applied very quickly

Applying Gaussian Filters to Downscaled Buffers

1/32 x 1/32 (32x48 pixels)1/64 x 1/64 (16x12 pixels)

1/4 x 1/4 (256x192 pixels)1/8 x 1/8 (128x96 pixels)1/16 x 1/16 (64x48 pixels)

Bilinear Filtering and Composition• Magnify them using bilinear

filtering and composite the results– The error is almost unrecognizable

+ +

+ =

Good Enough!

Notes on Filter• Use high-precision formats for low-res buff

ers• Don’t take too many samples

– Very expensive, especially for high-res images

HDR in Games• Should be appealing rather than accurate

– Accuracy is not important for players– Even an inaccurate scene can be appealing

• Cost and performance– The key is to understand the effects of HDR– Devise a fake technique that is fast enough and

produces believable results• Accurate HDR rendering is still hard in games

– Use HDR only for effects that give a large impact• Use sprites if you want to generate glare only for the

sun, which is much faster and gives high quality results

HDR in Games• Integer formats have very limited

dynamic range – Render in relative luminance space– Apply exposure scaling during rendering– High precision is not needed for the dark

regions after exposure scaling• As those regions remain dark in the final

image– Carefully choose the range to maximize

effective precision

HDR in Games• Future perspective

– All operations can be done in float– Effective use of HDR

• Depth-of-field with the shape of aperture stop• Motion blur with high luminances not clamped

References• Reinhard, Erik, Mike Stark, Peter Shirley, and Jim Ferwerda,

“Photographic Tone Reproduction for Digital Images”• Mitchell, Jason L., “Real-Time 3D Scene Post-Processing”• DirectX 9.0 SDK Summer 2003 Update, “HDRLighting Sample”• Debevec, Paul E., “Paul Debevec Home Page”• Kawase, Masaki, “Frame Buffer Postprocessing

Effects in DOUBLE-S.T.E.A.L (Wreckless)”