Maya API: MFnMesh.numColors() is broken?

MFnMesh.numColors() returns the number of colors per-polygon per-vertex. Or so the documentation says.

However I’ve seen this function return all sorts of crazy numbers and I’m not quite sure what’s going on here. The code is below.

What I’m trying to achieve is to apply a function on each vertex face. The function does some magic based on position data. The issue I get is when I try to APPLY these colors. MFnMesh.setColors() will give you a kInvalidParameter error if MFnMesh.numColors() is larger than the length of your MColorArray. And since numColors() goes bananas, it doesn’t matter how I iterate over my geometry. On a simple prism (two triangle end pieces, three quad sides) I get numColors() = 67

import maya.OpenMaya as om

# Get selection
selection = om.MSelectionList()
component = om.MObject()
mDagPath = om.MDagPath()
om.MGlobal.getActiveSelectionList(selection)
selection.getDagPath(0, mDagPath, component)
meshFn = om.MFnMesh(mDagPath)

# This one will lie to us. I've got all kinds of weird numbers from this one...
print("numColors: %s" % meshFn.numColors())

color_array = om.MColorArray()
color_ids = om.MIntArray()

# The iterators are great. They don't lie to us. They will go thru the geo as it is.
mItFaceVtx = om.MItMeshFaceVertex(mDagPath)
while not mItFaceVtx.isDone():

    # Bal bla bla - Running some func for determining color based on it's position here...

    # ...but for now, use dummy data
    color = om.MColor()
    color.r = 0.2
    color.g = 0.3
    color.b = 0.4

    color_array.append(color)

    # Getting the color index of the current vertex face. Unfortunately getColorIndex requires an Int &
    intRef = om.MScriptUtil().asIntPtr()
    mItFaceVtx.getColorIndex(intRef, "my_color_set")
    # ...okay we got the pointer, now we want it as an Int
    id = om.MScriptUtil(intRef).asInt()
    print("id: %s" % id)
    color_ids.append(id)
    print("color_ids are now of len(): %s" % len(color_ids))

    mItFaceVtx.next()

meshFn.setColors(color_array, "my_color_set")  # ...and since numColors() is crazy, we get kInvalidParameter quite often here
meshFn.assignColors(color_ids, "my_color_set")
meshFn.setCurrentColorSetName("my_color_set")

I would also like to point out that the colors are all scrambled and I don’t understand why. It’s regardless of how I iterate. I’ve tried iterating over meshes, polygons (faces), vertices, vertex faces… and never does the color get in the correct order. The only time it has been working was when I studied Faranell’s example here: Maya API Vertex Colors
…looping thru the indices of MeshFn.getVertices (vertexCount and vertexList)

But in my case I can’t do that as I need to perform an operation based on the position of a vertex (say: color things that are Y+ red, Y- blue, X+ black, X- white, Z+ green, etc)

Things I’ve tried:
MItMeshFaceVertex iterator, position() and getColorIndex()
MItMeshPolygon, getVertices() followed by looping thru the vertices and doing point() to get the vertex position.
MItMeshVertex using position() and getColorIndices()
…all three gave me scrambled results

I know I struggled with something similar while back so looked it up, my case might have an easier time seeing each shell will be a flat color so won’t have any “discontinous” values (Modo jargon but might still apply here)

    mesh = om.MFnMesh( dagPath )

    colors = om.MColorArray()
    vertexIds = om.MIntArray()
    for i in range( mesh.numVertices):
        colors.append( Color )
        vertexIds.append( i )

    mesh.setVertexColors( colors, vertexIds )

Hope this helps

Thank you.
This did work!

I’m a tiny bit bothered by not knowing how to color things when looping thru vertex faces though :confused:

Example, run this code on a cube centered at origin (it’s a very basic experiment: color all vertices black or white if they are above or below Y=0):

import maya.OpenMaya as om
import pymel.core as pm

my_set = "colorSet1"

# Get selection
selection = om.MSelectionList()
component = om.MObject()
mDagPath = om.MDagPath()
om.MGlobal.getActiveSelectionList(selection)
selection.getDagPath(0, mDagPath, component)
meshFn = om.MFnMesh(mDagPath)

print("numColors: %s" % meshFn.numColors())
meshFn.clearColors()
print("numColors after clear: %s" % meshFn.numColors())

# Count vertices
# space = om.MSpace.kObject
space = om.MSpace.kWorld
vertex_count = om.MIntArray()
vertex_list = om.MIntArray()
vertex_positions = om.MFloatPointArray()
meshFn.getVertices(vertex_count, vertex_list)
meshFn.getPoints(vertex_positions, space)

color_array = om.MColorArray()
# meshFn.getVertexColors(color_array, my_set, om.MColor(0, 0, 0, 1))
print("color_array.length() before: %s" % color_array.length())

# Iterate
i = 0
color_ids = om.MIntArray()

mItVtx = om.MItMeshVertex(mDagPath)
while not mItVtx.isDone():

    vertex_indices = om.MIntArray()
    mItVtx.getColorIndices(vertex_indices, my_set)
    print("vertex_indices: %s" % vertex_indices)  # Why does this give us an array with 3 values on a cube with only quads?

    # Get position
    position = mItVtx.position(space)

    # Create color and append to MColorArray
    color = om.MColor()
    if position.y > 0.0:
        color.r = 1.0
        color.g = 1.0
        color.b = 1.0
    else:
        color.r = 0.0
        color.g = 0.0
        color.b = 0.0

    for j in vertex_indices:
        color_array.append(color)
        color_ids.append(i)
        print("face: %s --- color_index: %s --- position.y: %s --- color value: %s" % (mItVtx.index(), i, position.y, color.r))
        i += 1

    mItVtx.next()

print("color_array.length() after: %s" % color_array.length())

meshFn.setColors(gradient_colors, my_set)
meshFn.assignColors(color_ids, my_set)
meshFn.setCurrentColorSetName(my_set)
# Why are the colors scrambled?

First off, running your code above gave me this error:
# Error: NameError: file <maya console> line 66: name 'gradient_colors' is not defined #

Seems like you still had an old variable stored in memory that you used. I guess it should have been color_array?

Anyway.
MeshFn.getVertices() gives back 2 arrays.
The first one is the amount of vertices that each polygon is composed of. So for a cube that’s [4, 4, 4, 4, 4, 4] meaning that each polygon consists out of 4 vertices/faceVertices.
The second array is are the (object relative) indices that belong to those polygon.
[0, 1, 3, 2, 2, 3, 5, 4, 4, 5, 7, 6, 6, 7, 1, 0, 1, 7, 5, 3, 6, 0, 2, 4]

So practically you could read the second array as this:
[[0, 1, 3, 2], [2, 3, 5, 4], [4, 5, 7, 6], [6, 7, 1, 0], [1, 7, 5, 3], [6, 0, 2, 4]]

I believe I remember that numColors was giving me a headache too. I’m not exactly sure anymore but I believe numColors (and setColors), can store more colors than you have in vertices. Its just an internally stored array of colors, of which some colors can be used and other aren’t.
That is the reason you still have to use meshFn.assignColors() after, as that will truly assign the vertexColor to each vertex.

This why in the end I went with getFaceVertexColors(), which always gave me the data I was looking for for each polygon.

Yep you are correct, color_array is the proper var name.
Yes I followed your example with MeshFn.getVertices() and this one works great for the color set / channel -transferring that you did earlier. But if you need to run a component-method (like getting the position or normal of a vertex) then I don’t know how to access a component of MeshFn thru just the per-triangle-per-vertex -indices. :frowning:

numColors() only seem to do the right thing if I run MeshFn.clearColors() before I attempt to do anything. Maybe the array is stored in memory and not cleared properly, causing you to get more colors in it than is actually physically possible on your mesh.

getFaceVertexColors() and setFaceVertexColors() seem like a good choice. A colleague of mine recommended the same.

I think you might be over-complicating this a little bit. All you really need here is the position of each vertex to do a check and figure out what color to apply. If you’re using MItMeshVertex, you don’t have to get the vertex positions ahead of time unless you need them for other reasons.

That being said, the Maya API is a tricky beast, and it’s particularly difficult to learn in python since there are multiple ways to do similar things and most of them aren’t very obvious.

As for the goal of setting vertex colors to white above the origin and black below, this is how I normally do it.

import maya.OpenMaya as om
import pymel.core as pm

def getDAG(node_name):
    selection_list = om.MSelectionList()
    selection_list.add(node_name)
    m_dag = om.MDagPath()
    selection_list.getDagPath(0, m_dag)
    return m_dag

if __name__ == '__main__':

    # get first node from selection and turn on display colors
    first_selected = pm.selected()[0]
    first_selected.displayColors.set(True)

    # get dag path and attach mesh Fn set
    dag_path = getDAG(first_selected.nodeName())
    mesh_fn = om.MFnMesh(dag_path)

    new_vertex_colors = om.MColorArray()

    # loop through and determine vertex colors
    it_vtx = om.MItMeshVertex(dag_path)
    while not it_vtx.isDone():
        vert_position = it_vtx.position()

        if vert_position.y >= 0:
            color = om.MColor(1.0, 1.0, 1.0, 1.0)
        else:
            color = om.MColor(0.0, 0.0, 0.0, 1.0)

        new_vertex_colors.append(color)
        it_vtx.next()

    # you have to build this yourself, since the api can't convert a
    # python list to an MIntArray
    vertex_index_list = om.MIntArray()
    [vertex_index_list.append(j) for j in range(mesh_fn.numVertices())]

    # apply new vertex colors to the mesh
    # (None is an optional dag modifier. You usually don't need it)
    mesh_fn.setVertexColors(new_vertex_colors, vertex_index_list, None)

My last reply deals with per-vertex colors. I just want to clarify something

MFnMesh.SetVertexColors() and MFnMesh.SetFaceVertexColors() do pretty different things. From reading your posts it seems that what you’re looking for is MFnMesh.SetVertexColors().

I rarely ever have to do this, but if you need to deal specifically with face vertices, then it can get a little more complicated. It’s not the same thing as setting per-vertex colors, since you can actually assign a unique color for each face connected to a particular vertex.

Thank you for the reply.
I managed to get things working in the end.