Sprite Renderer - Custom Skinned Shader

First off, after taking a few weeks off I went back to work this week, and stuff has been happening. Unfortunately, most of it is utility work so it’s not exactly exciting stuff to show or write about.

Second, today’s post is going to be a little different. It’s going to be a tutorial or guide on how to use XNA’s SkinnedEffect with custom shaders. I needed to do this for my sprite renderer, and it was pretty tricky. I couldn’t find any ready made code, and I’ve been requested to post on the subject. The image included here is what I did with this for my sprite rendering utility. So, here goes…

Custom shaders with SkinnedEffect for XNA 4

If you want to use animated models in XNA without much fuss, it provides the SkinnedEffect class. To use this you also need the SkinnedModelPipeline from XNA’s skinning sample. It works great, provides basic three point lighting, your choice of vertex or pixel lighting, materials and textures (although you need to assign a blank texture for any untextured model to work) and is generally great to work with.

Until you want to throw your own custom shaders into the mix. Though the HLSL and code behind the SkinnedEffect class are available in their stock effects download, the HLSL is incredibly convoluted and hard to understand even for those well versed in writing shaders. Worse, the effect code used by the SkinnedEffect can’t be easily replaced, so inheriting its code is right out.

You could write your own implementation of SkinnedEffect and your own shader for use with skinned models from the ground up, but if that’s so easy you were probably not looking at XNA’s built ins to begin with. To retain all of SkinnedEffect’s built in functionality, we’ll have to reimplement it from the stock effects code. This will guide you through the process of doing so, and making the HLSL involved easier to customize while retaining as much of the original functionality as desired. Warning: this is long. You can just skip to the sample download but I encourage you to follow along so you know how things work.

This tutorial assumes a familiarity with HLSL, basic coding concepts, and a familiarity with the SkinnedModelPipeline from the skinning sample project.

Step 1: Creating a CustomSkinnedEffect class

To start we’ll simply copy and paste the code from SkinnedEffect.cs, change the namespace, and change the class name to CustomSkinnedEffect:

#region Using Statements
using System;
using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Graphics;
#endregion

namespace StockEffects
{
    /// <summary>
    /// Custom effect for rendering skinned character models.
    /// </summary>
    public class CustomSkinnedEffect : Effect, IEffectMatrices, IEffectLights, IEffectFog
    {
        public const int MaxBones = 72;

        ...

SkinnedEffect also calls functions from the static EffectHelpers class and uses EffectDirtyFlags members. Just copying EffectHelpers.cs and changing the namespace would suffice, but CustomSkinnedEffect needs to be customizable so we’ll integrate the helper functions into our code. That way if you subclass CustomSkinnedEffect, you can change what these functions do easily and thereby modify SkinnedEffect’s built-in functionality. First, copy the EffectDirtyFlags enum:

    [Flags]
    public enum EffectDirtyFlags
    {
        WorldViewProj = 1,
        World = 2,
        EyePosition = 4,
        MaterialColor = 8,
        Fog = 16,
        FogEnable = 32,
        AlphaTest = 64,
        ShaderIndex = 128,
        All = -1
    }

Next, copy the following functions and make them non-static functions of CustomSkinnedEffect. I’ve included the code below, but it’s a lot with only minor changes to make the functions non-static, so feel free to skip over it.

        #region EffectHelpers

        ///<summary>
        /// Sets up the standard key/fill/back lighting rig.
        /// </summary>
        public virtual Vector3 EnableDefaultLighting(DirectionalLight light0, DirectionalLight light1, DirectionalLight light2)
        {
            // Key light.
            light0.Direction = new Vector3(-0.5265408f, -0.5735765f, -0.6275069f);
            light0.DiffuseColor = new Vector3(1, 0.9607844f, 0.8078432f);
            light0.SpecularColor = new Vector3(1, 0.9607844f, 0.8078432f);
            light0.Enabled = true;

            // Fill light.
            light1.Direction = new Vector3(0.7198464f, 0.3420201f, 0.6040227f);
            light1.DiffuseColor = new Vector3(0.9647059f, 0.7607844f, 0.4078432f);
            light1.SpecularColor = Vector3.Zero;
            light1.Enabled = true;

            // Back light.
            light2.Direction = new Vector3(0.4545195f, -0.7660444f, 0.4545195f);
            light2.DiffuseColor = new Vector3(0.3231373f, 0.3607844f, 0.3937255f);
            light2.SpecularColor = new Vector3(0.3231373f, 0.3607844f, 0.3937255f);
            light2.Enabled = true;

            // Ambient light.
            return new Vector3(0.05333332f, 0.09882354f, 0.1819608f);
        }

        /// <summary>
        /// Lazily recomputes the world+view+projection matrix and
        /// fog vector based on the current effect parameter settings.
        /// </summary>
        public virtual EffectDirtyFlags SetWorldViewProjAndFog(EffectDirtyFlags dirtyFlags,
                                                                ref Matrix world, ref Matrix view, ref Matrix projection, ref Matrix worldView,
                                                                bool fogEnabled, float fogStart, float fogEnd,
                                                                EffectParameter worldViewProjParam, EffectParameter fogVectorParam)
        {
            // Recompute the world+view+projection matrix?
            if ((dirtyFlags & EffectDirtyFlags.WorldViewProj) != 0)
            {
                Matrix worldViewProj;

                Matrix.Multiply(ref world, ref view, out worldView);
                Matrix.Multiply(ref worldView, ref projection, out worldViewProj);

                worldViewProjParam.SetValue(worldViewProj);

                dirtyFlags &= ~EffectDirtyFlags.WorldViewProj;
            }

            if (fogEnabled)
            {
                // Recompute the fog vector?
                if ((dirtyFlags & (EffectDirtyFlags.Fog | EffectDirtyFlags.FogEnable)) != 0)
                {
                    SetFogVector(ref worldView, fogStart, fogEnd, fogVectorParam);

                    dirtyFlags &= ~(EffectDirtyFlags.Fog | EffectDirtyFlags.FogEnable);
                }
            }
            else
            {
                // When fog is disabled, make sure the fog vector is reset to zero.
                if ((dirtyFlags & EffectDirtyFlags.FogEnable) != 0)
                {
                    fogVectorParam.SetValue(Vector4.Zero);

                    dirtyFlags &= ~EffectDirtyFlags.FogEnable;
                }
            }

            return dirtyFlags;
        }

        /// <summary>
        /// Sets a vector which can be dotted with the object space vertex position to compute fog amount.
        /// </summary>
        public virtual void SetFogVector(ref Matrix worldView, float fogStart, float fogEnd, EffectParameter fogVectorParam)
        {
            if (fogStart == fogEnd)
            {
                // Degenerate case: force everything to 100% fogged if start and end are the same.
                fogVectorParam.SetValue(new Vector4(0, 0, 0, 1));
            }
            else
            {
                // We want to transform vertex positions into view space, take the resulting
                // Z value, then scale and offset according to the fog start/end distances.
                // Because we only care about the Z component, the shader can do all this
                // with a single dot product, using only the Z row of the world+view matrix.

                float scale = 1f / (fogStart - fogEnd);

                Vector4 fogVector = new Vector4();

                fogVector.X = worldView.M13 * scale;
                fogVector.Y = worldView.M23 * scale;
                fogVector.Z = worldView.M33 * scale;
                fogVector.W = (worldView.M43 + fogStart) * scale;

                fogVectorParam.SetValue(fogVector);
            }
        }

        /// <summary>
        /// Lazily recomputes the world inverse transpose matrix and
        /// eye position based on the current effect parameter settings.
        /// </summary>
        public virtual EffectDirtyFlags SetLightingMatrices(EffectDirtyFlags dirtyFlags, ref Matrix world, ref Matrix view,
                                                             EffectParameter worldParam, EffectParameter worldInverseTransposeParam, EffectParameter eyePositionParam)
        {
            // Set the world and world inverse transpose matrices.
            if ((dirtyFlags & EffectDirtyFlags.World) != 0)
            {
                Matrix worldTranspose;
                Matrix worldInverseTranspose;

                Matrix.Invert(ref world, out worldTranspose);
                Matrix.Transpose(ref worldTranspose, out worldInverseTranspose);

                worldParam.SetValue(world);
                worldInverseTransposeParam.SetValue(worldInverseTranspose);

                dirtyFlags &= ~EffectDirtyFlags.World;
            }

            // Set the eye position.
            if ((dirtyFlags & EffectDirtyFlags.EyePosition) != 0)
            {
                Matrix viewInverse;

                Matrix.Invert(ref view, out viewInverse);

                eyePositionParam.SetValue(viewInverse.Translation);

                dirtyFlags &= ~EffectDirtyFlags.EyePosition;
            }

            return dirtyFlags;
        }

        /// <summary>
        /// Sets the diffuse/emissive/alpha material color parameters.
        /// </summary>
        public virtual void SetMaterialColor(bool lightingEnabled, float alpha,
                                              ref Vector3 diffuseColor, ref Vector3 emissiveColor, ref Vector3 ambientLightColor,
                                              EffectParameter diffuseColorParam, EffectParameter emissiveColorParam)
        {
            // Desired lighting model:
            //
            //     ((AmbientLightColor + sum(diffuse directional light)) * DiffuseColor) + EmissiveColor
            //
            // When lighting is disabled, ambient and directional lights are ignored, leaving:
            //
            //     DiffuseColor + EmissiveColor
            //
            // For the lighting disabled case, we can save one shader instruction by precomputing
            // diffuse+emissive on the CPU, after which the shader can use DiffuseColor directly,
            // ignoring its emissive parameter.
            //
            // When lighting is enabled, we can merge the ambient and emissive settings. If we
            // set our emissive parameter to emissive+(ambient*diffuse), the shader no longer
            // needs to bother adding the ambient contribution, simplifying its computation to:
            //
            //     (sum(diffuse directional light) * DiffuseColor) + EmissiveColor
            //
            // For futher optimization goodness, we merge material alpha with the diffuse
            // color parameter, and premultiply all color values by this alpha.

            if (lightingEnabled)
            {
                Vector4 diffuse = new Vector4();
                Vector3 emissive = new Vector3();

                diffuse.X = diffuseColor.X * alpha;
                diffuse.Y = diffuseColor.Y * alpha;
                diffuse.Z = diffuseColor.Z * alpha;
                diffuse.W = alpha;

                emissive.X = (emissiveColor.X + ambientLightColor.X * diffuseColor.X) * alpha;
                emissive.Y = (emissiveColor.Y + ambientLightColor.Y * diffuseColor.Y) * alpha;
                emissive.Z = (emissiveColor.Z + ambientLightColor.Z * diffuseColor.Z) * alpha;

                diffuseColorParam.SetValue(diffuse);
                emissiveColorParam.SetValue(emissive);
            }
            else
            {
                Vector4 diffuse = new Vector4();

                diffuse.X = (diffuseColor.X + emissiveColor.X) * alpha;
                diffuse.Y = (diffuseColor.Y + emissiveColor.Y) * alpha;
                diffuse.Z = (diffuseColor.Z + emissiveColor.Z) * alpha;
                diffuse.W = alpha;

                diffuseColorParam.SetValue(diffuse);
            }
        }
        #endregion

Now we’ve got a working copy of SkinnedEffect we can start customizing it to do what we want it to. First, we’ll change the default constructor from this:

        /// <summary>
        /// Creates a new SkinnedEffect with default parameter settings.
        /// </summary>
        public SkinnedEffect(GraphicsDevice device)
            : base(device, Resources.SkinnedEffect)

To this:

        /// <summary>
        /// Creates a new CustomSkinnedEffect with default parameter settings.
        /// </summary>
        public CustomSkinnedEffect(Effect effect)
            : base(effect)

This will let us create a CustomSkinnedEffect instance by passing in any already loaded effect, and set up as usual. This copy will replace SkinnedEffect’s shader code.

Next we also want to copy parameters from existing SkinnedEffect instances. Fortunately SkinnedEffect already has a constructor that copies itself:

        /// <summary>
        /// Creates a new SkinnedEffect by cloning parameter settings from an existing instance.
        /// </summary>
        protected SkinnedEffect(SkinnedEffect cloneSource)
            : base(cloneSource)
        {
            CacheEffectParameters(cloneSource);

            preferPerPixelLighting = cloneSource.preferPerPixelLighting;
            fogEnabled = cloneSource.fogEnabled;

            world = cloneSource.world;
            view = cloneSource.view;
            projection = cloneSource.projection;

            diffuseColor = cloneSource.diffuseColor;
            emissiveColor = cloneSource.emissiveColor;
            ambientLightColor = cloneSource.ambientLightColor;

            alpha = cloneSource.alpha;

            fogStart = cloneSource.fogStart;
            fogEnd = cloneSource.fogEnd;

            weightsPerVertex = cloneSource.weightsPerVertex;
        }

Our first order of business is to change “SkinnedEffect cloneSource” to “CustomSkinnedEffect cloneSource”, to maintain identical usage as SkinnedEffect. Secondly we’ll copy this entire function and make a new one we’ll call CopyFromSkinnedEffect:

        /// <summary>
        /// Copies parameters from an existing SkinnedEffect instance.
        /// </summary>
        public void CopyFromSkinnedEffect(SkinnedEffect cloneSource)
        {
            CacheEffectParameters(cloneSource);

            preferPerPixelLighting = cloneSource.preferPerPixelLighting;
            fogEnabled = cloneSource.fogEnabled;

            world = cloneSource.world;
            view = cloneSource.view;
            projection = cloneSource.projection;

            diffuseColor = cloneSource.diffuseColor;
            emissiveColor = cloneSource.emissiveColor;
            ambientLightColor = cloneSource.ambientLightColor;

            alpha = cloneSource.alpha;

            fogStart = cloneSource.fogStart;
            fogEnd = cloneSource.fogEnd;

            weightsPerVertex = cloneSource.weightsPerVertex;
        }

This is what we’ll call on the effects that come with models loaded from the SkinnedModelPipeline, but more on that later. First, calling CacheEffectParameters on a SkinnedEffect unfortunately doesn’t work because we couldn’t subclass SkinnedEffect, so we’ll copy that function in its entirety and rename it to “CacheEffectParametersFromSkinnedEffect”. We’ll also change the fields light0, light1 and light2 to the properties DirectionalLight0, DirectionalLight1 and DirectionalLight2:

        /// <summary>
        /// Looks up shortcut references to our effect parameters.
        /// </summary>
        void CacheEffectParametersFromSkinnedEffect(SkinnedEffect cloneSource)
        {
            textureParam = Parameters["Texture"];
            diffuseColorParam = Parameters["DiffuseColor"];
            emissiveColorParam = Parameters["EmissiveColor"];
            specularColorParam = Parameters["SpecularColor"];
            specularPowerParam = Parameters["SpecularPower"];
            eyePositionParam = Parameters["EyePosition"];
            fogColorParam = Parameters["FogColor"];
            fogVectorParam = Parameters["FogVector"];
            worldParam = Parameters["World"];
            worldInverseTransposeParam = Parameters["WorldInverseTranspose"];
            worldViewProjParam = Parameters["WorldViewProj"];
            bonesParam = Parameters["Bones"];
            shaderIndexParam = Parameters["ShaderIndex"];

            light0 = new DirectionalLight(Parameters["DirLight0Direction"],
                                          Parameters["DirLight0DiffuseColor"],
                                          Parameters["DirLight0SpecularColor"],
                                          (cloneSource != null) ? cloneSource.DirectionalLight0 : null);

            light1 = new DirectionalLight(Parameters["DirLight1Direction"],
                                          Parameters["DirLight1DiffuseColor"],
                                          Parameters["DirLight1SpecularColor"],
                                          (cloneSource != null) ? cloneSource.DirectionalLight1 : null);

            light2 = new DirectionalLight(Parameters["DirLight2Direction"],
                                          Parameters["DirLight2DiffuseColor"],
                                          Parameters["DirLight2SpecularColor"],
                                          (cloneSource != null) ? cloneSource.DirectionalLight2 : null);
        }

Unfortunately this doesn’t yet copy all pertinent information so we need to add some extra code to our CopyFromSkinnedEffect function:

            Texture = cloneSource.Texture;
            SpecularColor = cloneSource.SpecularColor;
            SpecularPower = cloneSource.SpecularPower;
            FogColor = cloneSource.FogColor;

            eyePositionParam.SetValue(cloneSource.Parameters["EyePosition"].GetValueVector3());
            fogVectorParam.SetValue(cloneSource.Parameters["FogVector"].GetValueVector4());
            worldInverseTransposeParam.SetValue(cloneSource.Parameters["WorldInverseTranspose"].GetValueMatrix());
            bonesParam.SetValue(cloneSource.Parameters["Bones"].GetValueMatrixArray(MaxBones));
            shaderIndexParam.SetValue(cloneSource.Parameters["ShaderIndex"].GetValueInt32());

The final function looks like this:

        /// <summary>
        /// Copies parameters from an existing SkinnedEffect instance.
        /// </summary>
        public void CopyFromSkinnedEffect(SkinnedEffect cloneSource)
        {
            CacheEffectParametersFromSkinnedEffect(cloneSource);

            preferPerPixelLighting = cloneSource.PreferPerPixelLighting;
            fogEnabled = cloneSource.FogEnabled;

            world = cloneSource.World;
            view = cloneSource.View;
            projection = cloneSource.Projection;

            diffuseColor = cloneSource.DiffuseColor;
            emissiveColor = cloneSource.EmissiveColor;
            ambientLightColor = cloneSource.AmbientLightColor;

            alpha = cloneSource.Alpha;

            fogStart = cloneSource.FogStart;
            fogEnd = cloneSource.FogEnd;

            weightsPerVertex = cloneSource.WeightsPerVertex;

            Texture = cloneSource.Texture;
            SpecularColor = cloneSource.SpecularColor;
            SpecularPower = cloneSource.SpecularPower;
            FogColor = cloneSource.FogColor;

            eyePositionParam.SetValue(cloneSource.Parameters["EyePosition"].GetValueVector3());
            fogVectorParam.SetValue(cloneSource.Parameters["FogVector"].GetValueVector4());
            worldInverseTransposeParam.SetValue(cloneSource.Parameters["WorldInverseTranspose"].GetValueMatrix());
            bonesParam.SetValue(cloneSource.Parameters["Bones"].GetValueMatrixArray(MaxBones));
            shaderIndexParam.SetValue(cloneSource.Parameters["ShaderIndex"].GetValueInt32());
        }

We’ve now completed our CustomSkinnedEffect class, which is utterly useless without custom HLSL code to use it with. Yay!

Step 2: Customizing the HLSL code

All the relevant shader code is unfortunately ridiculously obtuse and hard to understand, let alone modify. There’s a lot of duplicate code, and there is an array of nine vertex shaders involved. This is because, as Shawn Hargreaves has pointed out, Stock Effect is production code, not a tutorial. We’re going to fix that.

First copy over all the relevant files: Common.fxh, Lighting.fxh, Macros.fxh, SkinnedEffect.fx (rename this to SkinnedEffect.fxh) and Structures.fxh. That gives us our base to work from. To minimize working inside XNA’s stock code, we’ll create our own base effect file called CustomSkinnedEffect.fx. We’ll start by adding our vertex shader output structs, which are compatible with the ones used by XNA, and including the original shader file:

struct CustomVSOutput
{
    float4 Diffuse    : COLOR0;
    float4 Specular   : COLOR1;
    float2 TexCoord   : TEXCOORD0;
    float4 PositionPS : SV_Position;
};

struct CustomVSOutputPixelLighting
{
    float2 TexCoord   : TEXCOORD0;
    float4 PositionWS : TEXCOORD1;
    float3 NormalWS   : TEXCOORD2;
    float4 Diffuse    : COLOR0;
    float4 PositionPS : SV_Position;
};

#include "SkinnedEffect.fxh"

Next we’re going to move everything below the last pixel shader function inside SkinnedEffect.fxh into our own effect file. We’ll leave the stock vertex and pixel shader functions for reference. We’ll then modify the code we just moved to point to our own functions:

VertexShader VSArray[9] =
{
    compile vs_2_0 CustomVL1(), //VSSkinnedVertexLightingOneBone(),
    compile vs_2_0 CustomVL2(),
    compile vs_2_0 CustomVL4(),

    compile vs_2_0 CustomLight1(),
    compile vs_2_0 CustomLight2(),
    compile vs_2_0 CustomLight4(),

    compile vs_2_0 CustomPixelLighting1(),
    compile vs_2_0 CustomPixelLighting2(),
    compile vs_2_0 CustomPixelLighting4(),
};

int VSIndices[18] =
{
    0,      // vertex lighting, one bone
    0,      // vertex lighting, one bone, no fog
    1,      // vertex lighting, two bones
    1,      // vertex lighting, two bones, no fog
    2,      // vertex lighting, four bones
    2,      // vertex lighting, four bones, no fog

    3,      // one light, one bone
    3,      // one light, one bone, no fog
    4,      // one light, two bones
    4,      // one light, two bones, no fog
    5,      // one light, four bones
    5,      // one light, four bones, no fog

    6,      // pixel lighting, one bone
    6,      // pixel lighting, one bone, no fog
    7,      // pixel lighting, two bones
    7,      // pixel lighting, two bones, no fog
    8,      // pixel lighting, four bones
    8,      // pixel lighting, four bones, no fog
};

PixelShader PSArray[3] =
{
    compile ps_2_0 CustomPS(),
    compile ps_2_0 CustomPSNoFog(),
    compile ps_2_0 CustomPSPixelLighting(),
};

int PSIndices[18] =
{
    0,      // vertex lighting, one bone
    1,      // vertex lighting, one bone, no fog
    0,      // vertex lighting, two bones
    1,      // vertex lighting, two bones, no fog
    0,      // vertex lighting, four bones
    1,      // vertex lighting, four bones, no fog

    0,      // one light, one bone
    1,      // one light, one bone, no fog
    0,      // one light, two bones
    1,      // one light, two bones, no fog
    0,      // one light, four bones
    1,      // one light, four bones, no fog

    2,      // pixel lighting, one bone
    2,      // pixel lighting, one bone, no fog
    2,      // pixel lighting, two bones
    2,      // pixel lighting, two bones, no fog
    2,      // pixel lighting, four bones
    2,      // pixel lighting, four bones, no fog
};

int ShaderIndex = 0;

Technique SkinnedEffect
{
    Pass
    {
        VertexShader = (VSArray[VSIndices[ShaderIndex]]);
        PixelShader  = (PSArray[PSIndices[ShaderIndex]]);
    }
}

Next we’ll actually make the functions we’re pointing to, right below our #include statement:

#include "SkinnedEffect.fxh"

// Vertex shader: vertex lighting, one bone.
CustomVSOutput CustomVL1(VSInputNmTxWeights vin)
{
    Skin(vin, 1);
    return CustomVS(vin);
}

// Vertex shader: vertex lighting, two bones.
CustomVSOutput CustomVL2(VSInputNmTxWeights vin)
{
    Skin(vin, 2);
    return CustomVS(vin);
}

// Vertex shader: vertex lighting, four bones.
CustomVSOutput CustomVL4(VSInputNmTxWeights vin)
{
    Skin(vin, 4);
    return CustomVS(vin);
}

// Vertex shader: one light, one bone.
CustomVSOutput CustomLight1(VSInputNmTxWeights vin)
{
    Skin(vin, 1);
    return CustomVSLight(vin);
}

// Vertex shader: one light, two bones.
CustomVSOutput CustomLight2(VSInputNmTxWeights vin)
{
    Skin(vin, 2);
    return CustomVSLight(vin);
}

// Vertex shader: one light, four bones.
CustomVSOutput CustomLight4(VSInputNmTxWeights vin)
{
    Skin(vin, 4);
    return CustomVSLight(vin);
}

// Vertex shader: pixel lighting, one bone.
CustomVSOutputPixelLighting CustomPixelLighting1(VSInputNmTxWeights vin)
{
    Skin(vin, 1);
    return CustomVSPixelLighting(vin);
}

// Vertex shader: pixel lighting, two bones.
CustomVSOutputPixelLighting CustomPixelLighting2(VSInputNmTxWeights vin)
{
    Skin(vin, 2);
    return CustomVSPixelLighting(vin);
}

// Vertex shader: pixel lighting, four bones.
CustomVSOutputPixelLighting CustomPixelLighting4(VSInputNmTxWeights vin)
{
    Skin(vin, 4);
    return CustomVSPixelLighting(vin);
}

Because SkinnedEffect.fxh duplicates code in all three categories of vertex shader (vertex lighting, one light, pixel lighting), we simply only call the skinning function here for the correct number of bones, then call another vertex shader function which implements the shared code. That leaves us with only three vertex shaders to customize, giving a much better overview.

We need to implement these functions though, as well as the three pixel shaders. Since we have a hot mess of included files already, let’s make it easy on ourselves and keep all the code we actually want to customize inside separate files as well. We’ll call these “CustomVS.fxh” and “CustomPS.fxh”. Include these inside our base effect file:

#include "SkinnedEffect.fxh"
#include "CustomVS.fxh"
#include "CustomPS.fxh"

Next we’ll work on the vertex shaders. These will simply share the common code of SkinnedEffect.fxh, although I’ve inlined some of the called functions for clarity. Put these in CustomVS.fxh:

CustomVSOutput CustomVS(VSInputNmTxWeights vin)
{
    CustomVSOutput output;

    float4 pos_ws = mul(vin.Position, World);
    float3 eyeVector = normalize(EyePosition - pos_ws.xyz);
    float3 worldNormal = normalize(mul(vin.Normal, WorldInverseTranspose));

    ColorPair lightResult = ComputeLights(eyeVector, worldNormal, 3);

    output.PositionPS = mul(vin.Position, WorldViewProj);
    output.Diffuse = float4(lightResult.Diffuse, DiffuseColor.a);
    output.Specular = float4(lightResult.Specular, ComputeFogFactor(vin.Position));

    output.TexCoord = vin.TexCoord;

    return output;
}

CustomVSOutput CustomVSLight(VSInputNmTxWeights vin)
{
    CustomVSOutput output;

    float4 pos_ws = mul(vin.Position, World);
    float3 eyeVector = normalize(EyePosition - pos_ws.xyz);
    float3 worldNormal = normalize(mul(vin.Normal, WorldInverseTranspose));

    ColorPair lightResult = ComputeLights(eyeVector, worldNormal, 1);

    output.PositionPS = mul(vin.Position, WorldViewProj);
    output.Diffuse = float4(lightResult.Diffuse, DiffuseColor.a);
    output.Specular = float4(lightResult.Specular, ComputeFogFactor(vin.Position));

    output.TexCoord = vin.TexCoord;

    return output;
}

CustomVSOutputPixelLighting CustomVSPixelLighting(VSInputNmTxWeights vin)
{
    CustomVSOutputPixelLighting output;

    output.PositionPS = mul(vin.Position, WorldViewProj);
    output.PositionWS = float4(mul(vin.Position, World).xyz, ComputeFogFactor(vin.Position));
    output.NormalWS = normalize(mul(vin.Normal, WorldInverseTranspose));

    output.Diffuse = float4(1, 1, 1, DiffuseColor.a);
    output.TexCoord = vin.TexCoord;

    return output;
}

If you’re familiar with HLSL code this should all look pretty familiar. Now whenever we want to customize the effect, we just go into this file and change the shaders. Next up is “CustomPS.fxh” which is also a copy and paste job:

// Pixel shader: vertex lighting.
float4 CustomPS(CustomVSOutput pin) : SV_Target0
{
    float4 color = SAMPLE_TEXTURE(Texture, pin.TexCoord) * pin.Diffuse;

    AddSpecular(color, pin.Specular.rgb);
    ApplyFog(color, pin.Specular.w);

    return color;
}

// Pixel shader: vertex lighting, no fog.
float4 CustomPSNoFog(CustomVSOutput pin) : SV_Target0
{
    float4 color = SAMPLE_TEXTURE(Texture, pin.TexCoord) * pin.Diffuse;

    AddSpecular(color, pin.Specular.rgb);

    return color;
}

// Pixel shader: pixel lighting.
float4 CustomPSPixelLighting(CustomVSOutputPixelLighting pin) : SV_Target0
{
    float4 color = SAMPLE_TEXTURE(Texture, pin.TexCoord) * pin.Diffuse;

    float3 eyeVector = normalize(EyePosition - pin.PositionWS.xyz);
    float3 worldNormal = normalize(pin.NormalWS);

    ColorPair lightResult = ComputeLights(eyeVector, worldNormal, 3);

    color.rgb *= lightResult.Diffuse;

    AddSpecular(color, lightResult.Specular);
    ApplyFog(color, pin.PositionWS.w);

    return color;
}

And we’re done! Isn’t that nice and tidy?

Step 3: Implementation

Now all you need to do to replace the SkinnedEffects on a skinned model (named “currentModel”) from the SkinnedModelPipeline is to load your model, load your effect (named “customEffect”) as normal, and then do this:

foreach (ModelMesh mesh in currentModel.Meshes)
{
    foreach (ModelMeshPart part in mesh.MeshParts)
    {
        SkinnedEffect skinnedEffect = part.Effect as SkinnedEffect;
        if (skinnedEffect != null)
        {
            // Create new custom skinned effect from our base effect
            CustomSkinnedEffect custom = new CustomSkinnedEffect(customEffect);
            custom.CopyFromSkinnedEffect(skinnedEffect);

            part.Effect = custom;
        }
    }
}

Lastly, don’t forget when iterating your model to iterate CustomSkinnedEffect instances, not SkinnedEffect:

// Render the skinned mesh.
foreach (ModelMesh mesh in currentModel.Meshes)
{
    foreach (CustomSkinnedEffect effect in mesh.Effects)
    {
        effect.SetBoneTransforms(bones);

        effect.View = view;
        effect.Projection = projection;

        effect.EnableDefaultLighting();

        effect.SpecularColor = new Vector3(0.25f);
        effect.SpecularPower = 16;
    }

    mesh.Draw();
}

So if we want to add a basic (kind of ugly) toon shader effect we can simply add the following line of code to all three pixel shaders:

color = round(color * 5) / 5;

And hey presto, Fanny’s your aunt, Bob’s your uncle!

SkinnedEffect with custom shader sample screenshot

Sample project: CustomSkinnedEffect_Sample.zip