Transforming normals
From CGAFaq
In 3D, the orientation of a plane in space can be given by a vector perpendicular to the plane, a "normal vector" or "normal" for short. Often it is convenient to keep that vector of unit length, or "normalized"; be careful of the different meanings of "normality". A smooth surface has a plane tangent to each point, and by extension a normal to that plane is called a "surface normal". Graphics code also cheats by associating artificial normal vectors with the vertices of polygonal models to simulate the reflection properties of curved surfaces; these are called "vertex normals".
The "orientation" of a plane has two meanings, both of which usually apply. Aside from the rotational tilting and turning meaning (geometric orientation), there is also the sense of "side" (topological orientation). A closed convex surface made of polygons has two sides, an inside and an outside, and normals can be assigned to the polygons in such a way that they all consistently point outside. This is often desirable for shading and culling.
When a model is defined in one coordinate system and used in another, as is commonly done, it may be necessary to transform normals also. If the change of point coordinates is effected by means of a rotation plus a translation, one simply rotates the normals as well (with no translation). Other coordinate changes need more care.
Although it is possible to use projective transformations to map a model into world coordinates, ordinarily they are used only for viewing. It is usually a mistake to apply perspective to a normal, as shading and culling are best done in world coordinates for correct results. Also perspective may be computed using degenerate matrices which are not invertible, though that need not be the case. For the importance of this, see below.
The combination of a linear transformation and a translation is called an affine transformation, and is performed as a matrix multiplication plus a vector addition:
- p' = A(p) = Lp + t.
When the model-to-world point transform is affine, the proper way to transform normals is with the transpose of the inverse of L.
- n' = (L−1)T n
However that is not enough.
If L includes scaling effects, a unit normal in model space will usually transform to a non-unit normal, which can cause problems for shaders and other code. This may need correcting after the normal is transformed.
If L includes reflection, the inside-outside orientation of the normal is reversed. This, too, can cause problems, and may need correcting. The determinant of L will be negative in this case.
When a complicated distortion is used, it must be approximated differently at each point in space by a linear transform made up of partial derivatives, the Jacobian. The matrix for the Jacobian replaces L in the equation for transforming normals.
Why use the transposed inverse? Write the dot product of column vectors n and v as a matrix product nT v. Write vector v as a difference of points, q-p. Let p, q, and thus v lie in the desired plane, and let n be normal to it. Vectors at right angles have zero dot product.
- nT v = 0
The transform v' of v is
v' = (Lq+t)-(Lp+t) = (Lq-Lp)+(t-t) = L(q-p) = Lv
The transform n' of n will remain normal if it satisfies
- n'T v' = nT v
Let n' = Mn for some M. Then the requirement is
n'T v' = (Mn)T (Lv) = nT (MT L) v = nT v
This holds if
- MT L = I
where I is the identity. Right multiplying by the inverse L−1 and transposing both sides gives, as claimed,
- M = (L−1)T
When L is a rotation, L−1 = LT, so M = L. When L has no inverse it will still have an "adjoint" to substitute for for orthogonality purposes, differing only by a scale factor.
