Unit testing in Maya, anyway, works fine, though most of the time you’ll need to run your tests in a standalone maya to make sure that all of your code has what it needs to function. There are previous threads here.
What you’ll usually find is that the first time setup is painful because getting your nice clean isolated build machine to behave like your local maya install (with all the version issues, userSetup code, and correct startup settings) will reveal all of the weaknesses in your current setup. That’s good - it’s by design! Push through – when you can run a test process reliably from a clean machine you will have made your infrastructure much more robust.
Generally speaking you probably don’t need anything fancier than the vanlla
unittest module and a good naming convention for running the tests – I used to use Nose but in the end the name based system turned out to be simpler to maintain for me. YMMV of course, but it’s one less dependency to worry about on the build server.