Some history
The year is 1991. The Soviet Union has collapsed, Terminator 2 has been released, and the year as a whole smells like Teen Spirit. A Dutch programmer, Guido van Rossum, releases his new programming language on USENET, which he’s been developing as a spiritual successor to the ABC language for over a year. The selling point? “Remarkable power with very clear syntax”. Promoted as an alternative to shell scripting and Perl, it quickly outgrows its original scope.
Python comes with what we call the Python path, and while you usually have no
business reading it in your code, you can do so from Python’s sys module.
If you place code in the Python path, you can import it into your script. If
someone wants to share code with you, you can stick that code somewhere on the
Python path. Simple enough, right? You can simply copy the code into a
folder (let’s say site-packages) and you’re good to go.
Well, not so fast. Not all packages are merely source code. Some expect their
users to compile binaries. This all needed standardization. And so after nearly
a decade, distutils finally comes into being, released in Python 1.6. You
write some code in a file called setup.py, which imports distutils from
the standard library, and it bundles all your code with metadata into a single
directory, ready for you to import.
The next question is where to put your beautiful Python source distribution.
Perl has their own archive index, called the “Comprehensive Perl Archive
Network”. And thus is born the Cheese Shop, later known as the
Python Package Index. Just download your package from the
index, run the setup.py file, and your code is neatly installed into
site-packages. Well, sometimes it’s neat. Other times bad things might happen
to your computer. There can be a lot of moving parts in building the code, and
anything could go wrong.
Wouldn’t it be nice if developers could do all the nitty gritty compilation for
us, and bequeath us distributions with the binaries already built?
And so the successor to distutils arrives onto the scene in 2004:
setuptools, and with it, all your code and binaries bundled into a single
distribution, the egg. To install you run easy_install, and it finds a built
distribution, downloads it, and installs it from setup.py along with all the
dependencies. Not bad.
If you want to uninstall, all you have to do is… well, no. You have to do that manually. Go find all the dependencies you pulled in and remove those too. Not only that, but as the Python ecosystem grows larger and larger, you eventually, after installing too many packages with their own dependencies (and so on), inevitably undergo the rite of passage that is a broken Python installation.
pip arrives on the scene
Finally, we get pip, which might seem like it’s been around for a long time,
but for half of Python’s existence, there was no pip. Even after it is made
however, it is only preinstalled in Python from 2014 onwards. Prior to this,
things are bad. But pip does something new. To understand pip, you must
also understand its complement, virtualenv. Using these tools together
allows you to have a virtual Python environment with its own dependencies,
isolated from the dependencies of another project. Dependency resolution is
improved, and uninstallation is supported. Sounds like we’re done here…
Well, not quite. Remember those eggs containing precompiled binaries? Many
things could go wrong when installing them, depending on your system. Tools
like Anaconda help for a while, but they are far from a universal solution,
being bloated, less intuitive, and including non-free software. If a normal
Python installation involving pip and virtualenv were a car with faulty
headlights, Anaconda would be a combine harvester. You can see the road
now, but perhaps it would have been less trouble to simply fix the lights.
Hardly an elegant choice for the average Python developer. That’s not to say it
doesn’t solve a lot of problems, but if you aren’t a data scientist, it isn’t
always a viable solution.
So, in 2012, we transition from the egg to the wheel, the first benefit being
that setup.py is no longer run during installation. It can be installed by
merely unpacking its contents, including any binaries it might have. No longer
does every user of the distribution package need to compile anything, so long
as their particular system is catered to.
There remains another problem, however. Remember how I said that setup.py is
no longer run during installation. That didn’t mean you can throw it out
entirely. The developer still needs to run setup.py to build their wheel.
Yes, the user experience has improved a lot, but anyone who has ever had to
write, maintain, or even read a setup.py file remembers what nightmare fuel
these contain.
Attempts are made to use setup.cfg, but those do not work out. But finally,
in 2016, the new pyproject.toml standard appears. It’s sleek, semantic, and
intuitive. No longer do you require an abstract syntax tree to scrape metadata,
you just need to parse TOML. For the vast majority of Python developers, this
is what they need. While at first it was simply meant to be a configuration
file for different tools, developers quickly find it so useful and scalable,
that even build-systems come to rely on it with the subsequent introduction of
project metadata. Still writing setup.py files? What are you, a luddite?
Introducing the all new pyproject.toml file of 2020, batteries included.
Too many tools
To this day, setup.py is still supported, but its uncontested reign has
finally come to an end. And with that, we can at long last turn the page and
enjoy packaging done right… right? Wrong. You see, over the late 2010s and
early 2020s, new tools are been designed to deal with the many difficulties
that maintaining a package introduced. You want deterministic installations?
Use pipenv. You want your virtual environment to depend on a particular
version of Python? Use pyenv. You want a Python package installed with its
entry points exposed globally? Use pipx. You want a Cargo-like tool to manage
your packages? Use poetry. Remember the old adage?
there should be one, and preferably only one, obvious way to do it
It has been long forgotten. There seem to be many competing ways to do it.
To make matters worse, if someone new to Python uses a combination of these tools, without knowing what they were doing, they can dig themselves a very deep hole. The tools themselves are often slow to adapt to new standards, or only work for Linux users.
It’s now 2023, and Armin Ronacher has decided enough is enough. And so he builds, using Rust of all languages, a Python packaging tool that can, by clever delegation to other tools, create virtual environments with different Python versions, manage dependencies, build distributions, and even publish those distributions. Yours truly even gives a talk at Kiwi PyCon 2024 encouraging people to start using Rye. Sounds promising…
I am become death, destroyer of packaging tools
But it’s still 2023, and Charlie Marsh has decided that pip is slow. His team
of developers, also using Rust, build a tool to manage dependencies, called
uv. It’s fast. Seriously fast. As a nice bonus, Rye allows uv as a drop-in
replacement for pip. Python without pip? These are truly strange times.
By the middle of 2024, Charlie Marsh’s team have talked things over with Armin Ronacher, and they decide that Rye would make an excellent template for a new-and-improved uv, one which relies less on delegation and more resembled a complete product than any tool had prior. Finally, in October 2024, Rye goes into maintenance, and uv becomes the recommended package manager. And that brings us to today.
Packaging in Python has gone from being a dumpster fire to simple, fast, and
easy, which ultimately is what Python is about. We’re thankfully done with
eggs, with distutils, and with setup.py. It’s time for a new chapter in
Python’s history.
If you have not yet begun to use uv, visit the website and take a look. There’s plenty of good tutorials out there, so there’s really no excuse not to try. And if you do try, don’t be surprised if you never code in Python the same way again.
Python