I was reading the latest issue of Pycoder's Weekly and they mentioned Thomas Kluyver's pull request to implement PEP 518 in pip. The newsletter linked to the reddit thread announcing the landing of the PR, and being a co-author of the PEP I was curious to see what people had to say. I figured either people would understand what PEP 518 was about and thus be happy, or there would be angst due to a misunderstanding of what the change was about. In the end it was a mixed bag, but there was some obvious misunderstandings so I figured I would try and clear up what PEP 518 is about and what it leads to long-term for packaging in Python.

pyproject.toml

When you look at a package on PyPI you typically have at least two files: an sdist -- a.k.a. source distribution -- and a wheel (you can have multiple wheels if you have an extension module in your package). If you think of a wheel as the end product/artifact for your package since that's what pip ultimately installs, the steps from source to wheel is source → sdist → wheel. For most people the source → sdist step is typically just creating a zip file of your source through python setup.py sdist --formats=zip. To go from sdist → wheel, people and pip unzip the sdist and runs python setup.py bdist_wheel.

A key observation to make about the wheel-creating process is that pip is executing the setup.py file in the sdist. This is problematic for the key reason that you don't know what packages that file depends on. Since you can't reliably introspect Python source code to figure out its dependencies without executing it which will trigger all global-level imports, pip just assumes any setup.py file directly depends on setuptools and indirectly on wheel. This presents a nasty chicken-and-egg problem for any project that wants to use something other than setuptools to build a wheel as you have no way to tell pip that you want to use something else, and since pip assumes setuptools for using setup.py most projects don't even bother thinking about an alternative. How do you solve this problem?

The first step is to come up with a way to tell pip what packages your sdist depends on to be built. PEP 518 ended up introducing a file called pyproject.toml for this. For the common case that your project uses setuptools and 'wheel' to create a wheel, your pyproject.toml would be:

[build-system]
requires = ["setuptools", "wheel"]

But what if your project wanted to use flit instead of setuptools?

[build-system]
requires = ["flit"]

With PEP 518 implemented, pip can make sure that whatever your package needs to create a wheel will be install.

Probably the biggest complaint in that reddit thread seemed to revolve around why we needed yet another file for a project (I'm ignoring the complaints about choosing TOML over another configuration format as it's covered in the PEP and configuration file formats are like editors: you can never make everyone happy, so I'm not going to worrying about the fact that we don't). Most people listed setup.py, setup.cfg, and requirements.txt as other files a package typically has. But in actuality one of those files probably isn't appropriate and the other two will be by choice if the planned future for Python packaging comes together (PEP 518 is the first half of this, the other half is discussed later in this post).

First, requirements.txt and setup.py are for two separate use-cases and probably shouldn't be used together in most cases. You can read Donald Stufft's post on this, but basically setup.py is for specifying a library's dependencies while requirements.txt is for specifying your app's dependencies. In other words the former is for stuff you put on PyPI and the latter is for what you install on e.g. your server when you deploy your code. Now some people who develop packages like having a requirements.txt file to make it easier to install development dependencies, but in those cases I recommend defining an extras_require for dev in my setup.py, e.g.:

docs_require = ["sphinx"]
tests_require = ["pytest"]

setup(
    extras_require={
        "docs": docs_require,
        "tests": tests_require,
        "dev": docs_require + tests_require,
    }
)

You can then do pip install -e ".[dev]" to install your development dependencies and skip having a requirements.txt file.

As for the setup.cfg and setup.py files, having those files will be your choice by choosing to use setuptools. If other builders like flit catch on and they choose to store configuration data in pyproject.toml then you won't need either setup.cfg or setup.py unless you choose to continue to use setuptools (flit itself currently only has a flit.ini file, but there's a pull request to update flit to use pyproject.toml exclusively).

The (planned) future

So with PEP 518, pip now knows what your package must have available to run the builder you need to make a wheel. But to truly make this a front-end/back-end scenario like in compilers (where the builder is the back-end and pip is the front-end), you need a way to tell pip how to run your chosen back-end. That's where PEP 517 will (most likely) come into play.

Assuming PEP 517 gets accepted (and it's looking like there's a good chance that it will), the pyproject.toml file will let you specify how pip should execute your back-end. That means the python setup.py bdist_wheel step for building a wheel that was mentioned earlier stops being an assumption that pip makes and instead you would specify how to call your chosen back-end, e.g.:

[build-system]
requires = ["flit"]
build-backend = "flit.api:main"

This completely removes setuptools -- and distutils for that matter -- from being a requirement and instead makes it a choice, which I think is a good thing.