Maya HLSL Shader

[chronic]

I'd like to share with you a real-time HLSL Shader for Maya’s Viewport that I wrote as part of my exploration into the wonderful world of HLSL.
Try it out, and if you like it, let me know!

UPDATED: lcNextGenShader_1.15.fx (http://media.leocov.com/blog/hlsl/lcNextGenShader_1.15.fx)

My Original Blog Post (http://blog.leocov.com/2009/11/lcnextgenshaderfx-and-lchlsltoolsmel.html)

lcNextGenShader_1.1.fx

Download: lcNextGenShader_1.1.fx (http://media.leocov.com/blog/hlsl/lcNextGenShader_1.1.fx)
Sample Cube Map: default_env_map.dds (http://media.leocov.com/blog/hlsl/default_env_map.dds)

http://media.leocov.com/blog/hlsl/lcNextGenShader_exampleT.jpg (http://media.leocov.com/blog/hlsl/lcNextGenShader_example.jpg)

http://media.leocov.com/blog/hlsl/lcNextGenShader_interfaceT.jpg (http://media.leocov.com/blog/hlsl/lcNextGenShader_interface.jpg)

Techniques:

NextGen_SM3 – full shading (requires shader model 3 capable graphics card)

Diffuse_Texture_Only – only the diffuse texture or color, without any other effects

Normal_Map_As_Color – only the normal map color values (helps to find seams)

Specular_Only_SM3 – only the specular effects, light and specular map contributions (requires shader model 3 capable graphics)

Parameters:

backFaceLighting – for single sided polygon objects, the back sides will be lit correctly

ambientColor – a color ‘added’ on top of the final effect

diffuseColor – the Lambert diffuse shading color

specularColor – the Phong specular shading color

glossiness – controls the size of the Phong specular highlight

useDiffuseTexture – use the diffuse texture from ‘diffuseMap’ instead of diffuseColor

transparencyInDiffuse – use the diffuseMap’s alpha channel for transparency

specularInDiffuse – use the diffuseMap’s alpha channel for specular masking

diffuseMap – a diffuse texture

useNormalTexture – use a tangent space normal map texture

specularInNormal – use the normalMap’s alpha channel for specular masking

normalPower – intensity of the normal map effect, 0=no effect

flipGreen – invert the green channel of the normal map (default = true)

normalMap – a tangent space normal map texture

useSpecularMap – use a specularMap for specular coloring and masking

specularMap – a specular texture for highlight coloring and masking

useAmbEnv – use cubeEnvMap for ambient environment lighting – cube map is forced down in mip levels – make sure to use a cube map processed by ATI CubeMapGen (http://developer.amd.com/gpu/cubemapgen/pages/default.aspx) for proper filtering

ambPower – intensity of the ambient environment lighting – also multiplied with the ambientColor

useReflEnv - use cubeEnvMap for environment reflections

reflPower – intensity of the environment reflections (has a slight hardcoded Fresnel effect)

cubeEnvMap – a cube map – preferably one processed by ATI CubeMapGen for proper filtering

useGlowTexture – use glowMap for self illumination effects

glowPower – the intensity of the self illumination effect

glowMap – a texture that is ‘added’ on top of the final shader result – self illumination/incandescence

useLightFalloff – use falloff on the point lights

decayScale – scale of the light falloff for the point lights

Light1 is a directional light, lights 2 and 3 are point lights


Note: There are some issues with overlapping transparency planes in Maya’s viewport when using realtime shaders – unfortunately this seems to be a problem with Maya and not the HLSL code itself – so there is no fix available.

[Giometric]

Cool stuff! :D Really nice work on this, of course in typical fashion, even after using it for like 30 minutes tops I already have complaints :P

You might want to specify on your blog site or a readme or something that some file formats for textures don't work at all; I didn't know and thought I was doing something wrong for a while, hehe. As of right now I can see that PSDs and TIFFs for sure don't work, they just show up as black. I've only gotten TGAs and DDS files to display. Is that a limitation of how the shader works or is it a bug? If it is a limitation, which other file types work?

Another thing is the backface lighting works great (good thing too, the model I'm testing with looks horrible without it :lol: ), but it does cause some strange stuff around the model's edges. Again not sure if this is a bug or a limitation in the way the shader works.

Attached is a screenshot of what I'm talking about in case it's of any use. Thanks again!

EDIT: Almost forgot to mention, I'm on Maya 2008, if that makes any difference.

[chronic]

Texture compatability is limited by the HLSL/DirectX runtime env - tga and dds are going to be the best option although other basic formats like bmp and jpg might also work. I definatly know that psd's dont work and using a layered psd will crash maya.

backface lighting is a tricky thing, and the edge bleed is a well known artifact of the technique i am using, its partialy a result of low polygon density combined with the way face normals work at steep angles. unfortunaly there is no way around it except to use backface lighting only on single sided objects and have all solid objects given a different shader with that option turned off. unconfirmed: it may be less noticeable on triangulated meshes - but i hate triangulating my mesh :(

[Giometric]

I see.. well having to use dds or tga isn't too bad, just something I figure should be mentioned so people aren't giving up on your shader right off the bat because they don't know about that.

I've actually never had problems in Maya using PSDs really, other than occasionally the layer blend modes don't get calculated exactly the same as in Photoshop (I think this is more an issue with older versions of Maya and trying to use much newer Photoshop files). And of course Mental Ray won't render them, but that's Mental Ray's fault. I like using 'em for the small speedup in texturing workflow I get, paint a bit, Ctrl-S, reload the file in Maya, and right away I can see the changes. When I'm done is usually when I save out a TIFF or TGA or whatever is needed. I just hate having to Save As over and over in Photoshop, hehe.

Sucks about the edge bleeding thing, sadly triangulating the mesh has no effect at all on it. Not that I would want to do that until the very end either :lol:

[chronic]

UPDATE

Download: lcNextGenShader_1.15.fx (http://media.leocov.com/blog/hlsl/lcNextGenShader_1.15.fx)

Update to the shader includes:

Fresnel attributes (fresnel no longer hard coded and now affects both cubemap reflections and specularity)

frBias – changes how far across the surface the effect occurs

frETA – bumps up the intensity in the ‘center’ area

frPower – Power/intensity multiplier on the entire effect

-

My texturing workflow has always been to CTRL+SHIFT+S to 'Save As' a .tga or whatever format