"SemiToon": Shader Portfolio Piece, could use a critique

Rigging
#1

Hey guys!

I think this may actually be the first time I’ve posted something I’ve worked on! So I noticed my portfolio was really light on shader work. So today I whipped this up. I call it SemiToon!

It’s a bit inspired by TF2, and how it’s not quite traditionally Toon Shaded, but definitely Non-Photo-Real. At it’s heart, SemiToon takes traditional lighting calculations, and picks a place to clamp them, and fades between the clamped regions. These clamps produce a toon-like effect. If you set the ToonMinimum and ToonMaximum parameters to 0 and 1, respectively, you get something very close to a traditional Lambert/Phong. If you make the numbers very close together, you will get a hard-edge toon look. The awesome part is, if you really wanted to, you could dynamically change these parameters to make something change from non-toon to toon and back again before your very eyes. Ambient Color is masked to only areas that receive no light, so that you can get dynamic colored shadow effects. I didn’t put much love into the Lighting Accumulation ( 2 Points, just to prove that it works. ) because it seems that every Engine does it differently these days.

The shader comes in 3 flavors.
REALLY EXPENSIVE ( Transparent with an Outline )
Pretty Expensive ( Opaque with an Outline )
Spry ( Opaque, No Outline )

If you use the outline, you want to close out your pass for every group you want outlined. ( FX Composer will batch them all into one set of passes by default, so if you use that, the outlines might look funny when they overlap. )

The Toon Outline was a HUGE pain to get correct with no artifacts. I’ve never done that before. I just fumbled through a bunch of Stencil Tricks until it worked. How do you guys do things like that?

I’d love to hear what you guys think about SemiToon! I’d also really welcome critiques about coding style or things going on in the code. Tips and Tricks are welcome to! A lot of this Toon-Shading stuff is new ground for me, and I really just did it by feel.


// +--------------------------------------------+
//   SemiToon: by Lithium
// +--------------------------------------------+
// +--------------------------------------------+
//   Semantics
// +--------------------------------------------+

float4x4 WorldInverseTranspose : WorldInverseTranspose  < string UIWidget="None"; >;
float4x4 WorldViewProject      : WorldViewProjection    < string UIWidget="None"; >;
float4x4 ViewProject           : ViewProjection         < string UIWidget="None"; >;
float4x4 World                 : World                  < string UIWidget="None"; >;
float4x4 ViewInverse           : ViewInverse            < string UIWidget="None"; >;

// +--------------------------------------------+
//   Parameters
// +--------------------------------------------+

// +--------------------------------------------+
//   Frame
float3 AmbientLight : Ambient <
	string UIName =  "Ambient Lighting";
	string UIWidget = "Color";
> = {0.1f, 0.1f, 0.1f};

// +--------------------------------------------+
//   Bound
bool UsePoint0
<
	string UIName = "Use Point Light 0";
> = true;

float3 Point0Pos : POSITION
<
	string Object = "Point Light 0";
	string UIName =  "Lamp 0 Position";
	string Space = "World";
> = {0.5f,2.0f,1.25f};

float3 Point0Col : COLOR <
	string Object = "Point Light 0";
	string UIName =  "Lamp 0 Color";
	string UIWidget = "Color";
> = {1.0f,1.0f,1.0f};

float Point0Constant : CONSTANTATTENUATION <
	string Object = "Point Light 0";
	string UIName = "Lamp 0 Constant Attenuation";
	string UIWidget ="slider";
> = 1;

float Point0Linear : LINEARATTENUATION <
	string Object = "Point Light 0";
	string UIName = "Lamp 0 Linear Attenuation";
	string UIWidget ="slider";
> = 0.05;

float Point0Quadratic : QUADRATICATTENUATION <
	string Object = "Point Light 0";
	string UIName = "Lamp 0 Linear Attenuation";
	string UIWidget ="slider";
> = 0;

bool UsePoint1
<
	string UIName = "Use Point Light 1";
> = true;

float3 Point1Pos : POSITION
<
	string Object = "Point Light 1";
	string UIName =  "Lamp 1 Position";
	string Space = "World";
> = {0.5f,2.0f,1.25f};

float3 Point1Col : COLOR <
	string Object = "Point Light 1";
	string UIName =  "Lamp 1 Color";
	string UIWidget = "Color";
> = {1.0f,1.0f,1.0f};

float Point1Constant : CONSTANTATTENUATION <
	string Object = "Point Light 1";
	string UIName = "Lamp 1 Constant Attenuation";
	string UIWidget ="slider";
> = 1;

float Point1Linear : LINEARATTENUATION <
	string Object = "Point Light 1";
	string UIName = "Lamp 1 Linear Attenuation";
	string UIWidget ="slider";
> = 0.05;

float Point1Quadratic : QUADRATICATTENUATION <
	string Object = "Point Light 1";
	string UIName = "Lamp 1 Linear Attenuation";
	string UIWidget ="slider";
> = 0;

// +--------------------------------------------+
//   Vertex Properties

float BorderThickness <
	string UIName = "Border Thickness";
	string UIWidget = "slider";
	float  UIMin = 0.0f;
	float  UIMax = 10.0f;
	float  UIStep = .01f;
> = .05f;

float4 BorderColor <
	string UIName = "Border Color";
	string UIWidget = "Color";
> = { 0, 0, 0, 1 };

// +--------------------------------------------+
//   Surface Properties
float4 DiffuseColor : DIFFUSE <
	string UIName = "Diffuse Color";
	string UIWidget = "Color";
> = {0.5f, 0.5f, 0.5f, .1f};

texture DiffuseMap <
	string ResourceName = "";
	string UIName = "Diffuse Texture";
	string ResourceType = "2D";
>;

sampler2D DiffuseMapSampler = sampler_state {
	Texture = <DiffuseMap>;
	MinFilter = Linear;
	MagFilter = Linear;
	MipFilter = Linear;
	AddressU = Wrap;
	AddressV = Wrap;
};

float DiffuseToonMin <
	string UIName = "Toon Diffuse Minimum";
	string UIWidget = "slider";
	float  UIMin = 0.0f;
	float  UIMax = 1.0f;
	float  UIStep = .01f;
> = 0.1f;

float DiffuseToonMax <
	string UIName = "Toon Diffuse Maximum";
	string UIWidget = "slider";
	float  UIMin  = 0.0f;
	float  UIMax  = 1.0f;
	float  UIStep = .01f;
> = 0.25f;

texture NormalMap  <
	string ResourceName = "";
	string UIName = "Normal Texture";
	string ResourceType = "2D";
>;

sampler2D NormalMapSampler = sampler_state {
	Texture = <NormalMap>;
	MinFilter = Linear;
	MagFilter = Linear;
	MipFilter = Linear;
	AddressU = Wrap;
	AddressV = Wrap;
};
	
// +--------------------------------------------+
//   Specular
float3 SpecularColor : SPECULAR <
	string UIName = "Specular Color";
	string UIWidget = "Color";
> = {0.8, 0.8f, 1.0f};

texture SpecularMap <
	string ResourceName = "";
	string UIName = "Specular Texture";
	string ResourceType = "2D";
>;

sampler2D SpecularMapSampler = sampler_state {
	Texture = <SpecularMap>;
	MinFilter = Linear;
	MagFilter = Linear;
	MipFilter = Linear;
	AddressU = Wrap;
	AddressV = Wrap;
};

float SpecularIntensity <
	string UIWidget = "slider";
	float UIMin = 0.0;
	float UIMax = 1.0;
	float UIStep = 0.01;
	string UIName =  "Specular Intensity";
> = 0.5;

float SpecularPower : SpecularPower <
	string UIWidget = "slider";
	float UIMin = 1.0;
	float UIMax = 128.0;
	float UIStep = 1.0;
	string UIName =  "Specular Power";
> = 5.0;

float SpecularToonMin <
	string UIName = "Toon Specular Minimum";
	string UIWidget = "slider";
	float  UIMin = 0.0f;
	float  UIMax = 1.0f;
	float  UIStep = .5;
> = 0.5f;

float SpecularToonMax <
	string UIName = "Toon Specular Maximum";
	string UIWidget = "slider";
	float  UIMin  = 0.0f;
	float  UIMax  = 1.0f;
	float  UIStep = .65f;
> = 0.55f;

// +--------------------------------------------+
//   Interop
// +--------------------------------------------+

struct appdata
{
	float3 Position : POSITION;
	float4 UV       : TEXCOORD0;
	float4 Normal   : NORMAL0;
	float4 Tangent  : TANGENT0;
	float4 Binormal : BINORMAL0;
};

struct vertexOutput
{
	float4 HPos         : POSITION;
	float2 UV           : TEXCOORD0;
	float3 Normal       : TEXCOORD1;
	float3 Tangent      : TEXCOORD2;
	float3 Binormal     : TEXCOORD3;
	float3 Eye          : TEXCOORD4;
	float3 Light0       : TEXCOORD5;
	float3 Light1       : TEXCOORD6;
	
};
 
// +--------------------------------------------+
//	 VERTEX
// +--------------------------------------------+

vertexOutput SemiToonVS( appdata IN )
{
	vertexOutput OUT;
	
	// Pass Through
	OUT.UV = float2( IN.UV.x, 1 - IN.UV.y );
	
	// World Space
	OUT.Normal   = mul(IN.Normal, WorldInverseTranspose ).xyz;
	OUT.Tangent  = mul(IN.Tangent, WorldInverseTranspose ).xyz;
	OUT.Binormal = mul(IN.Binormal, WorldInverseTranspose ).xyz;
	
	float4 MPos  = float4( IN.Position.xyz, 1 );
	float4 WPos  = mul( MPos, World );
	
	OUT.Light0 = Point0Pos - WPos;
	OUT.Light1 = Point1Pos - WPos;
	OUT.Eye = ViewInverse[3].xyz - WPos.xyz;
	
	// Screen Space
	OUT.HPos = mul( MPos, WorldViewProject );
	
	return OUT;
}

float4 BorderVS( appdata IN ) : POSITION
{
	// Do Border in World so units remain sensible.
	float3 WPos = mul( float4( IN.Position.xyz, 1 ), World ).xyz;
	float3 Norm = mul( IN.Normal.xyz, WorldInverseTranspose ).xyz;
	
	WPos += Norm * BorderThickness;
	
	float4 HPos = mul( float4( WPos, 1), ViewProject );
	return HPos;
}

float4 DebugVS( appdata IN ) : POSITION
{
	return mul( float4( IN.Position.xyz, 1), WorldViewProject );
}

// +--------------------------------------------+
//	 PIXEL UTILITY
// +--------------------------------------------+

float3 NormalFromTex2D( sampler2D normalSampler, float2 uv )
{
	return ( tex2D( normalSampler, uv ) * 2 - 1 );
}

struct FragData
{
	float3 Normal;
	float3 Eye;
};

FragData Frag( float3 Normal, float3 Eye )
{
	FragData OUT;
	
	OUT.Normal = Normal;
	OUT.Eye = normalize( Eye );
	
	return OUT;
}

struct Light
{
	float3 Position;
	float3 Direction;
	float3 Color;
	float  Attenuation;
};

Light PointLight( float3 pos, float3 vec, float3 col, float constFall, bool linFall, float quadFall  )
{
	Light OUT;
	
	OUT.Position = pos;
	OUT.Direction = normalize( vec );
	OUT.Color = col;
	
	float mag = length( vec );
	float atten = constFall + linFall * mag + quadFall * mag * mag;
	OUT.Attenuation = saturate( 1 / atten );
	
	return OUT;
}

float Accumulate( FragData frag, Light light, inout float3 diffuse, inout float3 specular )
{
	// Lambertian Diffuse
	float  LdotN = dot( light.Direction, frag.Normal );
	
	// Phong Specular
	float3 R     = normalize( 2.0f * frag.Normal * LdotN - light.Direction );
	float  RdotV = dot( R, frag.Eye);
	float  phong = pow( saturate( RdotV ), SpecularPower );
	
	// Toon Modification
	float toonDiff = smoothstep( DiffuseToonMin, DiffuseToonMax, LdotN ) ;
	float toonSpec = smoothstep( SpecularToonMin, SpecularToonMax, phong ) ;
	
	// Apply Light Factors
	toonDiff *= light.Attenuation;
	toonSpec *= light.Attenuation;
	diffuse  += toonDiff * light.Color;           // Lambert
	specular += toonSpec * light.Color;           // Blinn
	
	// Return the lighting amount.  Useful for masking.
	return toonDiff;
}


// +--------------------------------------------+
//	 PIXEL
// +--------------------------------------------+

float4 SemiToonPS(vertexOutput IN) : COLOR
{
	float3x3 TBN = float3x3( IN.Tangent, IN.Binormal, IN.Normal );
	float3 Normal = NormalFromTex2D( NormalMapSampler, IN.UV );
	
	FragData frag = Frag( mul( Normal, TBN ), IN.Eye );
	
	float3 diffTerm = 0;
	float3 specTerm = 0;
	float lit = 0;
	
	if( UsePoint0 )
	{
		Light light0 = PointLight( Point0Pos, IN.Light0, Point0Col, Point0Constant, Point0Linear, Point0Quadratic );
		lit += Accumulate( frag, light0, diffTerm, specTerm );
	}
	if( UsePoint1 )
	{
		Light light1 = PointLight( Point1Pos, IN.Light1, Point1Col, Point1Constant, Point1Linear, Point1Quadratic );
		lit += Accumulate( frag, light1, diffTerm, specTerm );
	}
	
	// Clamp lighting Values.
	diffTerm = saturate( diffTerm );
	specTerm = saturate( specTerm * SpecularIntensity );
	lit      = saturate( lit );
	
	diffTerm += ( 1 - lit ) * AmbientLight;
	
	float4 diffTex = tex2D( DiffuseMapSampler, IN.UV );
	float4 specTex = tex2D( SpecularMapSampler, IN.UV );
	
	diffTerm *= DiffuseColor * diffTex.rgb;
	specTerm *= SpecularColor * specTex.rgb;
	
	return float4( diffTerm + specTerm, diffTex.a);
}

float4 BorderPS ( float4 Position : POSITION ) : Color
{
	return float4( BorderColor );
}

float4 DebugPS ( float4 Position : POSITION ) : Color
{
	return float4( 1, 0, 0, 1);
}


// +--------------------------------------------+
//	 Technique
// +--------------------------------------------+

technique transparent
{
	pass depth
	{	// Depth pre-pass so there isn't overlap in the transparency
		VertexShader = compile vs_2_a DebugVS();
		PixelShader = compile ps_2_a DebugPS();
		
		ColorWriteEnable = false;
		
		ZEnable = true;
		ZWriteEnable = true;
		CullMode= cw;
	}
	pass body
	{	// Draw the Toon Shaded Body
		VertexShader = compile vs_2_a SemiToonVS();
		PixelShader = compile ps_2_a SemiToonPS();
		
		ColorWriteEnable = Red | Green | Blue | Alpha;
		
		// Create a Mask for where the shape was
		StencilEnable = true;
		StencilRef = 1;
		StencilFunc = ALWAYS;
		StencilPass = REPLACE;
		
		AlphaBlendEnable = true;
		SrcBlend = SRCALPHA;
		DestBlend = INVSRCALPHA;
		
		ZEnable = true;
		ZFunc = EQUAL;
		ZWriteEnable = false;		
		CullMode= cw;
	}
	pass outline
	{	// Draw The border
		VertexShader = compile vs_2_a BorderVS();
		PixelShader = compile ps_2_a BorderPS();
		
		// Draw only where the mask is not.
		StencilEnable = true;
		StencilRef = 1;
		StencilFunc = NOTEQUAL;
		StencilPass = REPLACE;
		
		AlphaBlendEnable = true;
		SrcBlend = SRCALPHA;
		DestBlend = INVSRCALPHA;
		
		ZEnable = true;
		ZWriteEnable = false;
		ZFunc = LESSEQUAL;
		
		CullMode= cw;
	}
	pass cleanup
	{	// Erase the stencil.  Write Expanded shape to ZBuffer
		VertexShader = compile vs_2_a BorderVS();
		PixelShader = compile ps_2_a BorderPS();
		
		ColorWriteEnable = false;
		
		// Reset Mask
		StencilEnable = true;
		StencilFunc = ALWAYS;
		StencilPass = ZERO;

		ZEnable = true;
		ZWriteEnable = true;		
		CullMode= cw;
	}	
}

technique opaque
{
	pass body
	{	// Draw the Toon Shaded Body
		VertexShader = compile vs_2_a SemiToonVS();
		PixelShader = compile ps_2_a SemiToonPS();
		
		ColorWriteEnable = Red | Green | Blue | Alpha;
		
		// Create a Mask for where the shape was
		StencilEnable = true;
		StencilRef = 1;
		StencilFunc = ALWAYS;
		StencilPass = REPLACE;
		
		ZEnable = true;
		ZWriteEnable = true;		
		CullMode= cw;
	}
	pass outline
	{	// Draw The border
		VertexShader = compile vs_2_a BorderVS();
		PixelShader = compile ps_2_a BorderPS();
		
		// Draw only where the mask is not.
		StencilEnable = true;
		StencilRef = 1;
		StencilFunc = NOTEQUAL;
		StencilPass = REPLACE;
		
		AlphaBlendEnable = true;
		SrcBlend = SRCALPHA;
		DestBlend = INVSRCALPHA;
		
		ZEnable = true;
		ZWriteEnable = true;
		ZFunc = LESSEQUAL;
		
		CullMode= cw;
	}
	pass cleanup
	{	// Erase the stencil.  Write Expanded shape to ZBuffer
		VertexShader = compile vs_2_a BorderVS();
		PixelShader = compile ps_2_a BorderPS();
		
		ColorWriteEnable = false;
		
		// Reset Mask
		StencilEnable = true;
		StencilFunc = ALWAYS;
		StencilPass = ZERO;

		ZEnable = true;
		ZFunc = EQUAL;
		ZWriteEnable = false;		
		CullMode= cw;
	}	
}

technique simple
{
	pass p0
	{
		VertexShader = compile vs_2_a SemiToonVS();
		PixelShader = compile ps_2_a SemiToonPS();
		ZEnable = true;
		ZWriteEnable = true;		
		CullMode= cw;
	}
}
#2

I like the renders you put out - very cool. I haven’t had the spare time to go test it out, but as soon as I do I’ll have more feedback. I like the look of it though.

~Alex

#3

I’m glad you like the renders! I’m not very good at presenting my tech well. I really wish I had something cool I could show this on, because I think you can get a look unlike many others using this. It’s a very exciting shader to me.

Some cool things that may not be obviously cool:
-There is a border where the donut and the teapot meet. You can’t do this with naive normal flipping.
-There are no artifacts like with normal flipping, you don’t see interior backfaces. The border is cleanly on the outside of objects.
-For the transparent one, because of the early depth pass, there are no backfaces, doubling, or other artifacts. It is very clean.
-Because it is doing the color-stepping on the light, not the output color, you can get more scene cohesion with full color depth than with traditional toon methods. In other words, you can do scene composition with light, not textures, in a traditional next-gen way.

As far as bugs, I may need to normalize the Normal/Tangent/Binormal. It doesn’t appear to like Scaling right now.