## Water drop 2b – Dynamic rain and its effects

Version : 1.1 – Living blog – First version was 3 january 2013

This is the second post of a series about simulating rain and its effect on the world in game. As it is a pretty big post, I split it in two parts a and b:

Water drop 1 – Observe rainy world
Water drop 2a – Dynamic rain and its effects
Water drop 2b – Dynamic rain and its effects
Water drop 3a – Physically based wet surfaces
Water drop 3b – Physically based wet surfaces
Water drop 4a – Reflecting wet world
Water drop 4b – Reflecting wet world

Directly following the part a let’s continue with other rain effects:

Water influence on material, water accumulation and puddles

In the observation post we see that a wet surface is darker and brighter depends on its material properties. The influence of water on a material is complex and will be discuss in detail in the next post : Water drop 3 – Physically based wet surfaces. For now we will follow the guideline define by Nakamae et al in “A Lighting Model Aiming at Drive Simulators” [1]. When it rains water accumulate in ground crack, gap and deformation. With sufficient precipitation puddles can appear and stay a long time even after it stop to rain (as highlight by the observation post). To define the different states of the ground surfaces [1] introduce the following classification:

Type 1: a dry region
Type 2: a wet region; i.e., the region where the road surface is wet but no water gathers
Type 3: a drenched region; i.e., the region where water remains to some extent but no puddles are formed, or the region of the margin of a puddle
Type 4: a puddle region

When a surface is wet (type 2), the paper suggest to apply a reflection coefficient of 5 to 10 on the specular and 0.1 to 0.3 on the diffuse. In the pseudo code of this section, we will represent this water influence by a function DoWetProcess. This function take a percentage of wetting strength under the form of a shader variable we will call wet level. When wet level is 0, the surface is dry, when 1 it is wet. This value is different from the raindrop intensity of the previous sections. Wet level variable is increase when the rain starts and takes some time to decrease it after it stops. Allowing simulating some drying. Here is a simple pseudo code:

void DoWetProcess(inout float3 Diffuse, inout float Gloss, float WetLevel)
{
// Water influence on material BRDF
Diffuse    *= lerp(1.0, 0.3, WetLevel);
// Not the same boost factor than the paper
Gloss       = min(Gloss * lerp(1.0, 2.5, WetLevel), 1.0);
}

// Water influence on material BRDF
DoWetProcess(Diffuse, Gloss, WetLevel);

Note that’s there is no change apply on the normal, when the surface is wet, we simply use the original normal. Here is shot with an environment map apply:

For puddles (type 4), the paper suggest a two layers reflection model as the photos of real rainy world show us. For now we keep it simple and just use the water BRDF parameters coupled with the diffuse attenuation of a wet surface. For the margin region of puddles (type 3) a simple weighting function between the two previous models is proposed. Here we lerp between the current material BRDF parameters (wet or not) and result of type 4 BRDF parameters to simulate the presence of accumulate water. Puddles placement need to be control by the artists and we use the alpha channel of vertex colors of a mesh for this purpose.  We provide a tool to our artists to paint vertex color in the editor directly on the mesh instance (more exactly Unreal Engine 3 provides tools). The blend weight of our lerping method is defined by the value of the vertex’s color alpha channel: 0 mean puddle and 255 mean no puddle (the default value for vertex color is often white opaque). Pseudo code:

AccumulatedWater = VertexColor.a;

// Type 2 : Wet region
DoWetProcess(Diffuse, Gloss, WetLevel);

// Apply accumulated water effect
// When AccumulatedWater is 1.0 we are in Type 4
// so full water properties, in between we are in Type 3
// Water is smooth
Gloss    = lerp(Gloss, 1.0, AccumulatedWater);
// Water F0 specular is 0.02 (based on IOR of 1.33)
Specular = lerp(Specular, 0.02, AccumulatedWater);
N        = lerp(N, float3(0, 0, 1), AccumulatedWater);

View in editor: no puddle, puddle paint with vertex color’s tools, puddle

Puddle paint can be seen in action at the begin of the puddles, heightmap and ripples youtube video.

## Water drop 2a – Dynamic rain and its effects

Version : 1.3 – Living blog – First version was 27 december 2012

This is the second post of a series about simulating rain and its effect on the world in game. As it is a pretty big post, I split it in two parts a and b:

Water drop 1 – Observe rainy world
Water drop 2a – Dynamic rain and its effects
Water drop 2b – Dynamic rain and its effects
Water drop 3a – Physically based wet surfaces
Water drop 3b – Physically based wet surfaces
Water drop 4a – Reflecting wet world
Water drop 4b – Reflecting wet world

In the first water drop we have seen several rain effects. To immerse the player in a rainy world, we need to support a lot of them. The major reference for rain city environment rendering is the “Toy Shop” demo from ATI which has been widely covered by Natalya Tatarchuck at many conferences [2][3][4]. However, even if the demo was available in late 2005, all the techniques described can’t easily fit in a PS3/XBOX360 playable game environment. In this second water drop, I want to share the work me and others at Dontnod have done around these rain effects for “Remember Me“. This post is the result of our researches. We will not only discuss about what we implemented but also about theory and other approaches. For this post, I invited my co-workers Antoine Zanuttini, Laury Michel and Orson Favrel to write some words, so this is a collaborative post :). We focused on rainy urban environment and we described different rain effects one by one. Our engine (Unreal engine 3) is a forward renderer but ideas here could also be applied in a deferred renderer.

## Rain Effects

Rain splashes  / Falling drops splashes

In the real world, when a falling drop hits a surface a splash is generated. Rain, or water flow on large heights like rooftop or tree, can generate falling drops, the behavior is the same in both cases. We  will focus on raindrops in a first time. Rain splashes can be simulated easily in a game by spawning a water splash particle when the stretched particle representing the raindrop collides with the scene. Tracking every particles colliding with a scene can be costly. With so many raindrops creating water splashes, it is hard to distinguish which rain drop is causing a specific rain splash. Based on this fact and for performance reasons it is simpler to have two independent systems to manage raindrops and rain splashes. Most games collide a bunch of random rays starting from the top of the world downward with a simple geometry representation of the scene then generate water splashes particles at the origin of the collisions [1][2]. As an optimization, the water splashes are only generated close to the screen. Another simple solution when you have complex geometry that you can’t simply approximate is to manually put an emitter of water splash particles following geometry boundaries. The pattern will not be as random as other water splashes but the effect will be there.

We tried another approach. Instead of trying to collide some rays with the world, we can simply render a depth map view from the top in the rain direction. The depth map gives all the information we require to emit a water splash particle at a random position in the world respecting the scene geometry. The steps of our approach are :
- Render a depth map
- Transfer depth map from GPU to CPU memory
- Use the depth map to generate random positions following the world geometry
- Emit the water splash at the generated positions

To render the depth map, we link a dummy location in front of the current camera but at a little higher,  then render the world geometry from this point of view. All standard shadow map optimizations apply here (Any culling method, Z double speed,  to render masked after opaque, having a stream with position only, not rendering  too small objects, forcing lod mesh etc…). As not all parts of the world need to generate rain splashes, we added an extra meshes tagging method for our artists to specify if a mesh needs to be rendered in the depth map. We also allow a mesh to be only rendered in the depth map and not in the normal scene. This is useful when you have translucent objects like glass which should stop rain but can’t render in opaque depth map or to approximate a lot of meshes by a single less complex mesh. To ease the debugging we added a special visualization mode in our editor to only see objects relevant to the rain splash.

The precision of world generated positions from this depth map depends on the resolution and the size of the frustum of the depth map. With a 256×256 depth map and a 20m x 20m orthogonal frustum we get world cells of 7.8cm² at the height taken from the depth map. The rasterizer will rule the height store in the depth map. This means that if you get an object in a cell of 7.8cm² with large height disparities, chances are the water splash will be spawned at a wrong height. This is a tradeoff between memory and performance.
To render the depth map we can use either an orthogonal or a perspective matrix. We haven’t found any usage for perspective matrix but in the following I will suppose that we can have both. Moreover, on console or DX10 and above, we can access the depth buffer, so we will use this functionality. On PC DX9 we store the depth value in the alpha channel of a color buffer. For consistency with other platforms the depth value is stored in normalized coordinate device. In case of a perspective projection, a reversed floating depth value is used to increase precision. Here is the PC DX9 pseudo code for this encoding:

## Water drop 1 – Observe rainy world

Version : 1.2 – Living blog – First version was 10 December 2012

This post is the first of a series about simulating dynamic rain and its effect on the world. All in the context of games:

Water drop 1 – Observe rainy world
Water drop 2a – Dynamic rain and its effects
Water drop 2b – Dynamic rain and its effects
Water drop 3a – Physically based wet surfaces
Water drop 3b – Physically based wet surfaces
Water drop 4a – Reflecting wet world
Water drop 4b – Reflecting wet world

In several games today there are dynamic weather effects. The most popular weather effect is rain. Rain has sadly often no effect on the gameplay but it has on the visual. Rain in real life has a lot of impact on the appearance of the world. The goal of this series is to describe technique, both technical and artistic, to be able to render a world rainy mood. By wet world, I mean not only world under rain, but also world after it stop raining. Let’s begin this series by an observation of the real-life wet world. As always, any feedbacks or comments are welcome.

## Real wet world

The first thing I have done when I started to study this topic for my game “Remember Me” is to make a lot of references. All pictures are programmer’s photography with low camera :). I should advice that’s I focus on moderate rain in urban environment not rain forest or other heavy rain. Let’s share some result (click for high res version).

The first thing everybody notice when it’s raining in the night is the long stretched highlight reflection of the bright light sources:

But this is not restricted to the night (and even not restricted to wet surface, it is only more visible with wet surfaces):

Highlight reflection vary with the roughness of the underlying surface:

The highlights get dimmer when the surface is rougher (This is energy conservation):

Highlights size depends on view angle. The anisotropic reflection seems to follow a Blinn-Phong behavior (Also Blinn-Phong model don’t allow to strech so much):

## Siggraph 2012 and Game Connection 2012 talk : Local Image-based Lighting With Parallax-corrected Cubemap

Here is the slides of me and my co-worker Antoine Zanuttini’s talk at Siggraph 2012 and Game Connection 2012 : “Local Image-based Lighting With Parallax-corrected Cubemap”

The two talk are similar but the Game Connection talk are more recent and contain little update. This is the one to get.

Game Connection 2012:

A short sum up can be seen on the official Game Connection 2012 website : Local Image-based Lighting With Parallax-corrected Cubemap

Powerpoint 2007:  Parallax_corrected_cubemap-GameConnection2012.pptx
PDF: Parallax_corrected_cubemap-GameConnection2012.pdf
Accompanying video of the slides:

Siggraph 2012:

A short sum up can be seen on the official Siggraph 2012 website : Fast Realistic Lighting

Powerpoint 2007: Parallax_corrected_cubemap-Siggraph2012.pptx
PDF: Parallax_corrected_cubemap-Siggraph2012.pdf
Accompanying video of the slides: http://www.youtube.com/watch?v=Bvar6X0dUGs

Description:

The talk describes a new approach of local image-based lighting. It consist in mixing several local parallax-corrected cubemaps and use the result to light scene objects.
The advantages are accurate local lighting, no lighting seams and smooth lighting transitions.
The content of this talk is also describe in the post : Image-based Lighting approaches and parallax-corrected cubemap.

## Image-based Lighting approaches and parallax-corrected cubemap

Version : 1.28 – Living blog – First version was 2 December 2011
This post replace and update a previous post name “Tips, tricks and guidelines for specular cubemap” with information contain in the Siggraph 2012 talk “Local Image-based Lighting With Parallax-corrected Cubemap” available here.

Image-based lighting (IBL) is common in game today to simulate ambient lighting and it fit perfectly with physically based rendering (PBR). The cubemap parameterization is the main use due to its hardware efficiency and this post is focus on ambient specular lighting with cubemap. There is many different specular cubemaps usage in game and this post will describe most of them and include the description of a new approach to simulate ambient specular lighting with  local image-based Lighting. Even if most part of this post are dedicated to tools setup for this new approach, they can easily be reuse in other context. The post will first describe different IBL strategies for ambient specular lighting, then give some tips on saving specular cubemap memory. Second, it will detail an algorithm to gather nearest cubemaps from a point of interest (POI), calculate each cubemap’s contribution  and efficiently blend these cubemaps on the GPU. It will finish by several algorithms to parallax-correct a cubemap and the application for the local IBL with parallax-cubemap approach. I use the term specular cubemap to refer to both classic specular cubemap and prefiltered mipmapped radiance environment map. As always, any feedbacks or comments are welcomed.

## IBL strategies for ambient specular lighting

In this section I will discuss several strategies of cubemap usage for ambient specular lighting. The choice of the right method to use depends on the game context and engine architecture. For clarity, I need to introduce some definitions:

I will divide cubemaps in two categories:
- Infinite cubemaps: These cubemaps are used as a representation of infinite distant lighting, they have no location. They can be generated with the game engine or authored by hand. They are perfect for representing low frequency lighting scene like outdoor lighting (i.e the light is rather smooth across the level) .
- Local cubemaps: These cubemaps have a location and represent finite environment lighting.They are mostly generating with game engine based on a sample location in the level. The generated lighting is only right at the location where the cubemap was generated, all other locations must be approximate. More, as cubemap represent an infinite box by definition, there is parallax issue (Reflected objects are not at the right position) which require tricks to be compensated. They are used for middle and high frequency lighting scene like indoor lighting. The number of local cubemap required to match lighting condition of a scene increase with the lighting  complexity (i.e if you have a lot of different lights affecting a scene, you need to sample the lighting at several location to be able to simulate the original lighting condition).

And as we often need to blend multiple cubemap,  I will define different cubemap blending method :
- Sampling K cubemaps in the main shader and do a weighted sum. Expensive.
- Blending cubemap on the CPU and use the resulted cubemap in the shader. Expensive depends on the resolution and required double buffering resources to avoid GPU stall.
- Blending cubemap on the GPU and use the resulted cubemap in the shader. Fast.
- Only with a deferred or light-prepass engine: Apply K cubemaps by weighted additive blending. Each cubemap bounding volume is rendered to the screen and normal+roughness from G-Buffer is used to sample the cubemap.

In all strategies describe below, I won’t talk about visual fidelity but rather about problems and advantages.

Object based cubemap
Each object is linked to a local cubemap. Objects take the nearest cubemap placed in the level and use it as specular ambient light source. This is the way adopted by Half Life 2 [1] for their world specular lighting.
Background objects will be linked at their nearest cubemaps offline and dynamic objects will do dynamic queries at runtime. Cubemaps can have a range to not affect objects outside their boundaries, they can affect background objects only, dynamic objects only or both.

The main problem with object based cubemap is lighting seams between adjacent objects using different cubemaps.
Here is a screenshot (click for full res)

## AMD Cubemapgen for physically based rendering

Version : 1.66 – Living blog – First version was 4 September 2011

AMD Cubemapgen is a useful tool which allow cubemap filtering and mipchain generation. Sadly, AMD decide to stop the support of it. However it has been made open source  [1] and has been upload on Google code repository  [2] to be improved by community. With some modification, this tool is really useful for physically based rendering because it allow to generate an irradiance environment map (IEM) or a prefiltered mipmaped radiance environment map (PMREM).  A PMREM is an environment map (in our case a cubemap) where each mipmap has been filtered by a cosine power lobe of decreasing cosine power value. This post describe such improvement I made for Cubemapgen and few others.

This post will first describe the new features added to Cubemapgen, then for interested (and advanced) readers, I will talk about theory behind the modification and go into some implementation details.

## The modified Cubemapgen

The current improvements are under the form of new options accessible in the interface:

(click for full rez)

- Use Multithread : Allow to use all hardware threads available on the computer. If uncheck, use the default behavior of Cubemapgen. However new features are unsupported with the default behavior.
- Irradiance Cubemap : Allow a fast computation of an irradiance cubemap. When checked, no other filter or option are take in account. An irradiance cubemap can be get without this option by setting a cosine filter with a Base angle filter of 180 which is a really slow process. Only the base cubemap is affected by this option, the following mipmap use a cosine filter with some default values but these mipmaps should not be used.
- Cosine power filter : Allow to specify a cosine power lobe filter as current filter. It allow to filter the cubemap with a cosine power lobe. You must select this filter to generate a PMREM.
- MipmapChain : Only available with Cosine power filter. Allow to select which mode to use to generate the specular power values used to generate each PMREM’s mipmaps.
- Power drop on mip, Cosine power edit box : Only available with the Drop mode of MipmapChain. Use to generate specular power values used for each PMREM’s mipmaps. The first mipmap will use the cosine power edit box value as cosine power for the cosine power lobe filter. Then the cosine power will be scale by power drop on mip to process the next mipmap and once again this new cosine power will be scale for the next mipmap until all mipmap are generated. For sample, settings 2048 as cosine power edit box and 0.25 as power drop on mip, you will generate a PMREM with each mipmap respectively filtered by cosine power lobe of 2048, 512, 128, 32, 8, 2…
- Num Mipmap, Gloss scale, Gloss bias : Only available with the Mipmap mode of MipmapChain. Use to generate specular power values used for each PMREM’s mipmaps.  The value of Num mipmap, Gloss scale and Gloss bias will be used to generate a specular power value for each mipmap.
- Lighting model: This option should be use only with cosine power filter. The choice of the lighting model depends on your game lighting equation. The goal is that the filtering better match your in game lighting.
- Exclude Base : With Cosine power filter, allow to not process the base mimap of the PMREM.
- Warp edge fixup: New edge fixup method which do not used Width based on NVTT from Ignacio Castaño.
- Bent edge fixup: New edge fixup method which do not used Width based on TriAce CEDEC 2011 presentation.
- Strecht edge fixup, FixSeams: New edge fixup method which do not used Width based on NVTT from Ignacio Castaño. FixSeams allow to display PMREM generated with Edge fixup’s Stretch method without seams.

All modification are available in command line (Print usage for detail with “ModifiedCubemapgen.exe – help”).

Here is a comparison between irradiance map generated with cosine filter of 180 and the option irradiance cubemap (Which use spherical harmonic(SH) for fast processing): Read more of this post

## Spherical Gaussian approximation for Blinn-Phong, Phong and Fresnel

Spherical Gaussian approximation for lighting calculation is not new to the graphic community[4][5][6][7][8] but its use has  recently been adopted by the game developer community [1][2][3].
Spherical Gaussian (SG) is a type of spherical radial basis function (SRBF) [8] which can be used to approximate spherical lobes with Gaussian-like function.
This post will describe how SG can be use to approximate Blinn-Phong lighting, Phong lighting and Fresnel.

Why care about SG approximation ?
In the context of realtime rendering for games, the SG approximation allows to save a few instructions when performing lighting calculations. For modern graphics cards, saving few ALU in a shader is not always beneficial, but for older hardware like one can find in the PS3, every GPU cycle counts. This is less true for XBOX360 GPU, which performs better with arithmetic instructions, though this optimization can still be beneficial. It can also be used to schedule instructions in a different (low loaded) pipe on SPUs to increase performance [2].

Part of the work presented here credits to Matthew Jones (from Criterion Games) [9].

## Different spherical radial basis function

The post talk about SG, but it is good to know that there is several different types of SRBF found in graphics papers.
For completeness, I will talk quickly about SG and von Mises-Fisher (vMF) as this can be confusing.

SG definition can be found in [4]:
$G(v; p,\lambda,\mu)=\mu e^{\lambda(v.p -1)}$ where p ∈ 𝕊2 is the lobe axis, λ ∈ (0,+∞) is the lobe sharpness,
and μ ∈ ℝ is the lobe amplitude (μ ∈ ℝ3 for RGB color).

vMF is detailed in [8]:
$\gamma(n.\mu;\theta) = \frac{k}{4\pi \sinh(k)} e^{k(n.\mu)}$ with inverse width k and central direction μ. vMFs are normalized to integrate
to 1. Note: the notation from the paper [8] doesn’t match the notation from above.
The paper presents an approximation of this formula for k > 2 (which is almost always the case) : $\gamma(n.\mu;\theta)\approx \frac{k}{2\pi} e^{-k(1-n.\mu)}$

As you can see, the formulation stays the same for a lobe sharpness > 2, the difference lies only in the normalization constant for vMF, which we will omit. In the following, I will only refer to the SG function.

## Approximate Blinn-Phong with SG

An approximation of the Blinn-Phong model with SG is provided in the supplementary material of [4]:
$D(h)=(h.n)^k \approx e^{-k(1-(h.n))}$

Here is the comparison of the accuracy of the approximation for low specular power (<10), medium (25) and high( > 50)