(This is the blog post version of my keynote from EuroPython 2025 in Prague, Czechia.)

We now have a lock file format specification. That might not sound like a big deal, but for me it took 4 years of active work to get us that specification. Part education, part therapy, this post is meant to help explain what make creating a lock file difficult and why it took so long to reach this point.

What goes into a lock file

A lock file is meant to record all the dependencies your code needs to work along with how to install those dependencies.

That involves The "how" is source trees, source distributions (aka sdists), and wheels. With all of these forms, the trick is recording the right details in order to know how to install code in any of those three forms. Luckily we already had the direct_url.json specification that just needed translation into TOML for source trees. As for sdists and wheels, it's effectively recording what an index server provides you when you look at a project's release.

The much trickier part is figuring what to install when. For instance, let's consider where your top-level, direct dependencies come from. In pyproject.toml there's project.dependencies for dependencies you always need for your code to run, project.optional-dependencies (aka extras), for when you want to offer your users the option to install additional dependencies, and then there's dependency-groups for dependencies that are not meant for end-users (e.g. listing your test dependencies).

But letting users control what is (not) installed isn't the end of things. There's also the specifiers you can add to any of your listed dependencies. They allow you to not only restrict what versions of things you want (i.e. setting a lower-bound and not setting an upper-bound if you can help it), but also when the dependency actually applies (e.g. is it specific to Windows?).

Put that all together and you end up with a graph of dependencies who edges dictate whether a dependency applies on some platform. If you manage to write it all out then you have multi-use lock files which are portable across platforms and whatever options the installing users selects, compared to single-use lock files that have a specific applicability due to only supporting a single platform and set of input dependencies.

Oh, and even getting the complete list of dependencies in either case is an NP-complete problem.

And it make makes things "interesting", I also wanted the file format to be written by software but readable by people, secure by default, fast to install, and allow the locker which write the lock file to be different from the installer that performs the install (and either be written in a language other than Python).

In the end, it all worked out (luckily); you can read the spec for all the nitty-gritty details about pylock.toml or watch the keynote where I go through the spec. But it sure did take a while to get to this point.

Why it took (over) 4 years

I'm not sure if this qualifies as the longest single project I have ever taken on for Python (rewriting the import system might still hold that record for me), but it definitely felt the most intense over a prolonged period of time.

The oldest record I have that I was thinking about this problem is a tweet from Feb 2019:

2019

That year there were 106 posts on discuss.python.org about a requirements.txt v2 proposal. It didn't come to any specific conclusion that I can recall, but it at least got the conversation started.

2020

The next year, the conversation continued and generated 43 posts. I was personally busy with PEP 621 and the [project] table in pyproject.toml.

2021

In January of 2021 Tzu-Ping Chung, Pradyun Gedam, and myself began researching how other language ecosystems did lock files. It culminated in us writing PEP 665 and posting it in July. That led to 359 posts that year.

The goal of PEP 665 was a very secure lock file which partially achieved that goal by only supporting wheels. With no source trees or sdists to contend with, it meant installation didn't involve executing a build back-end which can be slow, be indeterminate, and a security risk simply due to running more code. We wrote the PEP with the idea that any source trees or sdists would be built into wheels out-of-band so you could then lock against those wheels.

2022

In the end, PEP 665 was rejected in January of 2022, generating 106 posts on the subject both before and after the rejection. It turns out enough people had workflows dependent on sdists that they balked at having the added step of building wheels out-of-band. There was also some desire to also lock the build back-end dependencies.

2023

After the failure of PEP 665, I decided to try to tackle the problem again entirely on my own. I didn't want to drag other poor souls into this again and I thought that being opinionated might make things a bit easier (compromising to please everyone can lead to bad outcomes when a spec if large and complicated like I knew this would be).

I also knew I was going to need a proof-of-concept. That meant I needed code that could get metadata from an index server, resolve all the dependencies some set of projects needed (at least from a wheel), and at least know what I would install on any given platform. Unfortunately a lot of that didn't exist as some library on PyPI, so I had to write a bunch of it myself. Luckily I had already started the journey before with my mousebender project, but that only covered the metadata from an index server. I still needed to be able to read METADATA files from a wheel and do the resolution. The former Donald Stufft had taken a stab at and which I picked up and completed, leading to packaging.metadata. I then used resolvelib to create a resolver.

As such there were only 54 posts about lock files that were general discussion. The key outcome there was trying to lock for build back-ends confused people too much, and so I dropped that feature request from my thinking.

2024

Come 2024, I was getting enough pieces together to actually have a proof-of-concept. And then uv came out in February. That complicated things a bit as it did/planned to do things I had planned to help entice people to care about lock files. I also knew I couldn't keep up with the folks at Astral as I didn't get to work on this full-time as a job (although I did get a lot more time starting in September of 2024).

I also became a parent in April which initially gave me a chunk of time (babies for the first couple of months sleep a lot, so if gives you a bit of time). And so in July I posted the first draft of PEP 751. It was based on pdm.lock (which itself is based on poetry.lock). It covered sdists and wheels and was multi-use, all by recording the projects to install as a set which made installation fast.

But uv's popularity was growing and they had extra needs that PDM and Poetry– the other major participants in the PEP discussions --didn't. And do I wrote another draft where I pivoted from a set of projects to a graph of projects. But otherwise the original feature set was all there.

And then Hynek came by with what seemed like an innocuous request about making the version of a listed project optional instead of required (which was done because the version is required in PKG-INFO in sdists and METADATA in wheels).

Unfortunately the back-and-forth on that was enough to cause the Astral folks to want to scale the whole project back all the way to the requirements.txt v2 solution.

While I understood their reasoning and motivation, I would be lying if I said it wasn't disappointing. I felt we were extremely close up to that point in reaching an agreement on the PEP, and then having to walk back so much work and features did not exactly make me happy.

This was covered by 974 posts on discuss.python.org.

2025

But to get consensus among uv, Poetry, and PDM, I did a third draft of PEP 751. This went back to the set of projects to install, but was single-use only. I also became extremely stringent with timelines on when people could provide feedback as well as what would be required to add/remove anything. At this point I was fighting burn-out on this subject and my own wife had grown tired of the subject and seeing me feel dejected every time there was a setback. And so I set a deadline of the end of March to get things done, even if I had to drop features to make it happen.

And in February I thought we had reached and agreement on this third draft. But then Frost Ming, the maintainer of PDM, asked why did we drop multi-use lock files when they thought the opposition wasn't that strong?

And so, with another 150 posts and some very strict deadlines for feedback, we managed to bring back multi-use lock files and get PEP 751 accepted-- with no changes! -- on March 31.

2 PEPs and 6 years later ...

If you add in some ancillary discussions, the total number of posts on the subject of lock files since 2019 comes to over 1.8K. But as I write this post, less than 7 months since PEP 751 was accepted, PDM has already been updated to allow users to opt into using pylock.toml over pdm.lock (which shows that the lock file format works and meets the needs of at least one of the three key projects I tried to make happy). Uv and pip also have some form of support.

I will say, though, that I think I'm done with major packaging projects (work has also had me move on from working on packaging since April, so any time at this point would be my free time, which is scant when you have a toddler). Between pyproject.toml and pylock.toml, I'm ready to move on to the next area of Python where I think I could be the most useful.