Virtual environments, Python, and deployment

[cspears2002]

I have been curious as to what are the limits of a virtual environment. I do understand that the purpose of the environment is to install packages and modules without affecting your local Python install. Then you create the requirements.txt file using pip and use that file when deploying your code. You just run pip against the requirements file, and everything is installed.

However, I have heard of people using the virtual environment upon deployment. From the outside, the environment seems like an easy way to bundle necessary packages along with the code. On the flip side, many experienced developers argue against this. They must have a good reason, but I have to yet hear it.

Can someone explain why I should avoid this approach?

[bob.w]

The major argument is that virtual environments are a development tool, not a distribution or deployment tool. So using them for either isn’t guaranteed to ‘just work’.

My advice would be: package up your runtime, and all its dependencies and deploy that.
This way you have exact control over the environment your application is running in.

If you’re just writing libraries for other developers to consume, just deploy your code and the requirements.txt and let them worry about bundling it all.

I should also point out, that virtual enviroments do not work with every python deployment. For example you can not create a virtual enviroment from the python that ships with Maya, and I’m assuming 3ds Max. As they both bundle their standard library into a zip file, and virtualenv relies on those existing as loose files.
I believe the default python on MacOS suffered from this at one point as well (might still?)

[cspears2002]

Okay. This does make sense.

How do virtual environments work with custom code? For example, I wrote a logger that formats the logs just the way we like and puts them exactly where we want. My guess is that this will not work:

pip install my_awesome_logger

Sounds like you might have to actually copy code like that into the project itself if you are planning to write something that runs completely outside of your environment. If the tool is going to be in house anyway, you can probably get away with sys.path.append or something similar so the Python code finds the in-house packages.

[bob.w]

Well packaging your code to be installed with pip is a rather involved topic.
You need a setup.py file at the very least.

But as a TA, I’ve never really used pip as a deployment tool to my users.
Nor really relied on any of the builtin python packaging / distribution tools, except in a few cases the modules for packaging everything into a zip file.

Instead I’ve usually deployed them to users either by directly copying it from a shared location (network share), or through source control. Then depending on the environment relied on various methods to get my code onto python path for that enviroment.
For example in maya, using Maya.env or a custom module file (I heavily prefer the module file approach at this point), as it greatly helps simplify supporting multiple maya versions.

If its a fully standalone tool, you can look at something like py2exe, cx_freeze, pyinstaller and some other projects that will package the whole thing up into an exe.

So one reason that I avoid things like pip on the user side of things, is that pip much like virtualenv, is a developer tool. Not an end user tool.
I try to avoid situations where I have to treat my users as developers.

But again, if you are writing library code, for other developers, then yeah, pip and the other python machinery starts to make more sense.

[Theodox]

TLDR – pip, setup tools and the rest of the python deployment toolkit are developer-to-developer tools, optimized for letting individual coders experiment and try stuff out. They aren’t designed for – or very good at – supporting end users, particularly non technical end users.

Bob’s advice is totally sound: you vendor the approved environment as a whole and make sure that everybody gets it as a whole. That eliminates a whole class of “it worked on my machine” bugs, the worst and most time consuming ones to fix.

[cspears2002]

Okay. This makes a lot of sense. Thanks for the input!