Callback for any command executed? Or any UI created?

Coding
, , , ,
#1

Heyo good people!
So here’s a hypothetical.

Maybe, a studio has hundreds of tools, none of which has telemetry.
Maybe, those tools live as menu items, but might have been copied by artists to custom shelves, etc.
And maybe, the studio is moving to a new environment, and is deciding on priority of porting the tools, and could use telemetry at least for which tools are opened the most, so that they can prioritize with at least some information, without relying on asking hundreds of artists directly.

So maybe, I’m looking for a way to force telemetry on any tool that’s opened, without touching that tool’s script.
I have a simple function that i would run to gather the telemetry, but am struggling with finding the most effective and efficient way.

I can scrub all the menu items and the scripts they run, store in a dict.
Then, i would like to check what command has been run, and if it fits one of the ones in the scrubbed dict, then run my own function to register telemetry. The part I’m struggling with, is finding out when a command has been run (or potentially when a UI has been created).

I have come up with a couple of potential solutions, none of which make me particularly happy (in no particular order. also, including things that i consider dumb, but for the sake of completion…):

#1 - callback on idle event (ew?)
Never done this one, for obvious reasons, so not sure what the overhead would be, but me scared. This would work though, but… ewwww?

#2 - OpenMaya.MCommandMessage - doesn’t seem to work.
This doesn’t seem to run if a command doesn’t print anything into the Script Editor. When creating a tool from a menuItem, the command is not printed out, so the callback i register won’t run. Am i doing something wrong here? Or - is there a way to force a command being thrown into the Script Editor, for thie callback to work, when run from a menu item?

#3 - event listener for the whole maya, checking for clicks and what they produce.
Similarly to #1, i consider this to be a ‘not worth implementing’ because of so many things potentially wrong with this…

Bit more information:
Systems: Windows 10
Maya: Maya 2019 to 2022.5
Tools framework: mostly Qt, but cannot guarantee there isn’t any cmds. tools…

Is there anything that anyone could potentially recommend? I would be grateful for any advice on this.
Let me know if there’s not enough information.

Thank you!

#2

“easiest” would probably be to have an online db somewhere (mainly for ease of remot workers) and then have a py class that would handle “message” sending to it. It sill means you have to add calls IN your tooling functions to send the relevant data but that’s generally how I’ve worked with something like this.

We had a DB in the office and since everyone had to be on a vpn into the office anyways it worke fairly well. I built a python wrapper to send SOAP messages to the server (as it was a SOAP webservice aimed for the C++/C# tooling that everyone else was doing)
Once you have the messaging class made you just have to go through and add messages to your functions.

PERSONALLY I would say doing telemetry on “everything all the time” is a terrible way to deal with wanting to know what users are using and doing as you will end up with so much data that it will almost certainly become noise. Filtering it yourself by only adding messages about things you will care about is a much more usable metric
e.g. you send error reports when someone clicks a button and it fails unexpectedly.

By doing this instead of “all the data” you end up with a traceable way to send a message to the user and ask what they did that went wrong and can look at fixing is (this method for error sending can be hooked into by overriding sys.excepthook = handle_exceptions where handle_exceptions is the function you would want called to handle messaging the stack trace for you :slight_smile:

All in all, I do believe in this “behind the scenes” error/tool usage tracking as it means you can find more information about what has worked and what has not for the end users easier than trying to get it from them (especially errors as we know that most people ignore error pop ups and print outs…)

Also on the topic of menu items in the top menu - I’d probably only care about the tools/scripts internally made and you’re probably better off doing a “startup/open tool” message in the tool/script itself instead of trying to inject it into menu items (imho)

#3

I’ve written several such tools and frameworks in my long practice, and from my experience, I can tell that the best thing to do is “ask hundreds of artists directly”. :sunglasses:

1 Like
#4

something that you could do is create a decorator that handles the telemetry for you and add that to the python/ui commands you want to track.
the single decorator can handle all the telemetry and can easily be adjusted or removed when no longer necessary

1 Like
#5

“easiest” would probably be to have an online db somewhere (mainly for ease of remot workers) and then have a py class that would handle “message” sending to it. It sill means you have to add calls IN your tooling functions to send the relevant data but that’s generally how I’ve worked with something like this.

We had a DB in the office and since everyone had to be on a vpn into the office anyways it worke fairly well. I built a python wrapper to send SOAP messages to the server (as it was a SOAP webservice aimed for the C++/C# tooling that everyone else was doing)
Once you have the messaging class made you just have to go through and add messages to your functions.

PERSONALLY I would say doing telemetry on “everything all the time” is a terrible way to deal with wanting to know what users are using and doing as you will end up with so much data that it will almost certainly become noise. Filtering it yourself by only adding messages about things you will care about is a much more usable metric
e.g. you send error reports when someone clicks a button and it fails unexpectedly.

By doing this instead of “all the data” you end up with a traceable way to send a message to the user and ask what they did that went wrong and can look at fixing is (this method for error sending can be hooked into by overriding sys.excepthook = handle_exceptions where handle_exceptions is the function you would want called to handle messaging the stack trace for you :slight_smile:

All in all, I do believe in this “behind the scenes” error/tool usage tracking as it means you can find more information about what has worked and what has not for the end users easier than trying to get it from them (especially errors as we know that most people ignore error pop ups and print outs…)

Also on the topic of menu items in the top menu - I’d probably only care about the tools/scripts internally made and you’re probably better off doing a “startup/open tool” message in the tool/script itself instead of trying to inject it into menu items (imho)

oh, my message and goal seems to have gotten lost in translation somewhere. apologies if my explanation was not enough.

we have built a robust telemetry system, so the question was not for creating a telemetry or how to deal with it, but rather how to blanket-add it to any command that i specify is a target of this blanket approach - basically how to run a callback on a command executing.

i wholeheartedly agree with the idea of adding telemetry to tools in a conscientious way, and not this blanket nonsense.
Therefore, we are not considering this to be ‘the way’ to add telemetry - that would be very misguided - but are instead treating this as a last ditch resort of getting some data over the next couple of weeks to help us have a more informed opinion when starting porting.
Adding telemetry to hundreds of tools now, when they are managed by hundred+ people, is not an option that we would have the luxury of pursuing. Unfortunately…

I do appreciate the input though.

I have build a solution already, so i’ll post it separately, for anyone who would find this topic useful at some point in the future.

1 Like
#6

Yep, that’s the plan here two - this data is not the end all be all, so we do plan a two-pronged approach to this, and will do both telemetry as well as asking artists. wouldn’t want to put all our eggs in only one basket.

#7

Ah fair enough. I would also agree with Peerke88 on the decorator front. I had this as well and it does work pretty well :slight_smile:
Interested to see what you’ve come up with

#8

By the way… I got the best and most informative results when using “dialogue monitoring”. The background monitoring system tracks all(!) messages from the specified dialogues (tools), including built-in and third-party ones. This is the only way to enable “telemetry” not only for your own (in-house) tools, but for all tools, which is sometimes more important. Since you usually need to know which tools and methods are most often used (or not used) compared to other solutions.

Also, such monitoring helped me realize how often users use docked or floating versions of tools. The result has finally confirmed my fears that I was spending too much effort and time on providing “possible docking” for each of my tools.

Such a system has limitations because it does not allow you to track commands invoked without using the user interface. But those cases are usually few and special in the pipeline.

#9

I would be curious to hear a bit more about what you mean by this, if you’re willing to share?

#10

Time to share what i decided to go with:

  1. create a function that scrubs all internally created menus and their children, stores in a dict as key = command they run, and value = name of the tool (menuItem label). This dict is stored for the whole session of maya.
  2. create a callback function, that gets the last command that was run with mc.repeatLast(q = True), and then checks if that command exists in the dict that was created in step 1. If so, it throws out the needed info (menu name, command that was run, and some other metrics gathered by default such as dcc version, user, etc) into our telemetry system
  3. attached the callback created in step 2 using om.MCommandMessage.addCommandCallback(). This way has several things other people should keep in mind:
    a) It will get triggered tens of times, after a command is run. so in step 2 in the callback function, to not hamper the processing that much, i have implemented checking if the last command is same as the one before it, i ignore that trigger.
    b) this doesn’t trigger when a script is run from pressing a button in a tool. This was an issue for the goal of this mission, because we might have a tool or two that gathers other tools, as buttons. I have semi-solved this by catching when THAT tool is run, and then running an additional script that adds another action to each one of the clicked signals of the gathered buttons within that tool.
  4. I have then added the function to call all this, into the userSetup.py file, to make sure the callback is registered at the beginning of any Maya session.

This is about it for now. If there’s anything that someone suggests to do differently, don’t hesitate to let me know.

I stress again, that this is a hail mary attempt at some last minute basic telemetry. In no way is this considered to replace actual properly implemented telemetry, and in no way would i recommend this approach, if you have any other option.
For us, it wasn’t feasible at all, to consider touching each tool’s scripts, hence resulting in this situation.

Hope this eventually helps someone ^^

1 Like
#11

I have not tried it, but it occurs to me that you might (emphasis on the might) be able to just monkeypatch QPushButton . If you add a little shim which sends your telemetry squirt in addition do doing the usual thing you won’t need the whole dictionary thing. I think you’d need to stuff this in very early but it might be worth trying