Archived: Tutorial: Custom shaders with SkinnedEffect for XNA 4
First off, after taking a few weeks off I went back to work this week, and stuff has been happening. Unfortunately, most of it is utility work so it’s not exactly exciting stuff to show or write about.
Second, today’s post is going to be a little different. It’s going to be a tutorial or guide on how to use XNA’s SkinnedEffect with custom shaders. I needed to do this for my sprite renderer, and it was pretty tricky. I couldn’t find any ready made code, and I’ve been requested to post on the subject. The image included here is what I did with this for my sprite rendering utility. So, here goes…
Custom shaders with SkinnedEffect for XNA 4
If you want to use animated models in XNA without much fuss, it provides the SkinnedEffect class. To use this you also need the SkinnedModelPipeline from XNA’s skinning sample. It works great, provides basic three point lighting, your choice of vertex or pixel lighting, materials and textures (although you need to assign a blank texture for any untextured model to work) and is generally great to work with.
Until you want to throw your own custom shaders into the mix. Though the HLSL and code behind the SkinnedEffect class are available in their stock effects download, the HLSL is incredibly convoluted and hard to understand even for those well versed in writing shaders. Worse, the effect code used by the SkinnedEffect can’t be easily replaced, so inheriting its code is right out.
You could write your own implementation of SkinnedEffect and your own shader for use with skinned models from the ground up, but if that’s so easy you were probably not looking at XNA’s built ins to begin with. To retain all of SkinnedEffect’s built in functionality, we’ll have to reimplement it from the stock effects code. This will guide you through the process of doing so, and making the HLSL involved easier to customize while retaining as much of the original functionality as desired. Warning: this is long. You can just skip to the sample download but I encourage you to follow along so you know how things work.
This tutorial assumes a familiarity with HLSL, basic coding concepts, and a familiarity with the SkinnedModelPipeline from the skinning sample project.
Step 1: Creating a CustomSkinnedEffect class
To start we’ll simply copy and paste the code from SkinnedEffect.cs, change the namespace, and change the class name to CustomSkinnedEffect:
SkinnedEffect also calls functions from the static EffectHelpers class and uses EffectDirtyFlags members. Just copying EffectHelpers.cs and changing the namespace would suffice, but CustomSkinnedEffect needs to be customizable so we’ll integrate the helper functions into our code. That way if you subclass CustomSkinnedEffect, you can change what these functions do easily and thereby modify SkinnedEffect’s built-in functionality. First, copy the EffectDirtyFlags enum:
Next, copy the following functions and make them non-static functions of CustomSkinnedEffect. I’ve included the code below, but it’s a lot with only minor changes to make the functions non-static, so feel free to skip over it.
Now we’ve got a working copy of SkinnedEffect we can start customizing it to do what we want it to. First, we’ll change the default constructor from this:
This will let us create a CustomSkinnedEffect instance by passing in any already loaded effect, and set up as usual. This copy will replace SkinnedEffect’s shader code.
Next we also want to copy parameters from existing SkinnedEffect instances. Fortunately SkinnedEffect already has a constructor that copies itself:
Our first order of business is to change “SkinnedEffect cloneSource” to “CustomSkinnedEffect cloneSource”, to maintain identical usage as SkinnedEffect. Secondly we’ll copy this entire function and make a new one we’ll call CopyFromSkinnedEffect:
This is what we’ll call on the effects that come with models loaded from the SkinnedModelPipeline, but more on that later. First, calling CacheEffectParameters on a SkinnedEffect unfortunately doesn’t work because we couldn’t subclass SkinnedEffect, so we’ll copy that function in its entirety and rename it to “CacheEffectParametersFromSkinnedEffect”. We’ll also change the fields light0, light1 and light2 to the properties DirectionalLight0, DirectionalLight1 and DirectionalLight2:
Unfortunately this doesn’t yet copy all pertinent information so we need to add some extra code to our CopyFromSkinnedEffect function:
The final function looks like this:
We’ve now completed our CustomSkinnedEffect class, which is utterly useless without custom HLSL code to use it with. Yay!
Step 2: Customizing the HLSL code
All the relevant shader code is unfortunately ridiculously obtuse and hard to understand, let alone modify. There’s a lot of duplicate code, and there is an array of nine vertex shaders involved. This is because, as Shawn Hargreaves has pointed out, Stock Effect is production code, not a tutorial. We’re going to fix that.
First copy over all the relevant files: Common.fxh, Lighting.fxh, Macros.fxh, SkinnedEffect.fx (rename this to SkinnedEffect.fxh) and Structures.fxh. That gives us our base to work from. To minimize working inside XNA’s stock code, we’ll create our own base effect file called CustomSkinnedEffect.fx. We’ll start by adding our vertex shader output structs, which are compatible with the ones used by XNA, and including the original shader file:
Next we’re going to move everything below the last pixel shader function inside SkinnedEffect.fxh into our own effect file. We’ll leave the stock vertex and pixel shader functions for reference. We’ll then modify the code we just moved to point to our own functions:
Next we’ll actually make the functions we’re pointing to, right below our #include statement:
Because SkinnedEffect.fxh duplicates code in all three categories of vertex shader (vertex lighting, one light, pixel lighting), we simply only call the skinning function here for the correct number of bones, then call another vertex shader function which implements the shared code. That leaves us with only three vertex shaders to customize, giving a much better overview.
We need to implement these functions though, as well as the three pixel shaders. Since we have a hot mess of included files already, let’s make it easy on ourselves and keep all the code we actually want to customize inside separate files as well. We’ll call these “CustomVS.fxh” and “CustomPS.fxh”. Include these inside our base effect file:
Next we’ll work on the vertex shaders. These will simply share the common code of SkinnedEffect.fxh, although I’ve inlined some of the called functions for clarity. Put these in CustomVS.fxh:
If you’re familiar with HLSL code this should all look pretty familiar. Now whenever we want to customize the effect, we just go into this file and change the shaders. Next up is “CustomPS.fxh” which is also a copy and paste job:
And we’re done! Isn’t that nice and tidy?
Step 3: Implementation
Now all you need to do to replace the SkinnedEffects on a skinned model (named “currentModel”) from the SkinnedModelPipeline is to load your model, load your effect (named “customEffect”) as normal, and then do this:
Lastly, don’t forget when iterating your model to iterate CustomSkinnedEffect instances, not SkinnedEffect:
So if we want to add a basic (kind of ugly) toon shader effect we can simply add the following line of code to all three pixel shaders:
And hey presto, Fanny’s your aunt, Bob’s your uncle!