Day 6 — Some PEP talk, or: What is a pyproject.toml?

17 August 2020 · recurse-center

Today I talked about PEPs 518 and 517 at RFCs We ❤️. I learned a lot from Brett Cannon's blog posts on the topic, and from reading the PEPs themselves. This post is a simpler version of everything.

First, let's briefly look at the history of Python packaging based on the timeline by the PyPA and The Hitchhiker's Guide to Packaging. To do that, run python -c "import time; time.travel(to_year='1998')" in your terminal, just kidding, that'd be a cool feature to have in the language though!

The year is 1998 and the question that plagues the Python community is "How do I build and share my Python code?". To talk about this problem, the distutils-sig mailing list is started at the 1998 International Python Conference, and development begins on distutils, a package that will solve this problem once and for all.

In 2000, distutils is added to the Python standary library. The changelog entry states that:

Greg Ward's "distutils" package is included: this will make installing, building and distributing third party packages much simpler.

distutils adds support for building and installing modules into an existing Python installation. It allows you to structure your Python project so that it has a setup.py, which lets you build a static snapshot (.tar.gz) of your project called a source distribution or sdist.


  from distutils.core import setup

  setup(
      name="awesome-library",
      version="1.0",
      # ...
  )

In 2004, setuptools is launched to overcome some limitations of distutils. It introduces the egg packaging format, and the ability to declare and install dependencies. It also adds the easy_install command-line tool which can let you install packages from the Python Package Index.


  from setuptools import setup

  setup(
      name="awesome-library",
      version="0.1.0",
      install_requires=[
          "requests>=2.0.0",
      ],
      # ...
  )

In 2008, pip is launched as an alternative to easy_install to overcome some of its limitations. Meanwhile, setuptools has become a vital part of the Python packaging ecosystem. But the development has slowed down. So the same year, distribute is launched as a fork of setuptools to create a more open project. One of the first big improvements it adds is Python 3 support.

In 2011, the Python Packaging Authority (PyPA) is created to take over the maintenance of pip. One of its other proposed names are "Ministry of Installation"!

In 2013, PyPA takes over maintenance of setuptools, and distribute is merged back into it. setuptools becomes the de facto choice for Python packaging. During this time, PEPs 425 and 427 are accepted. They specify the wheel packaging format, which is an improvement over the egg.

By 2014, pip and setuptools become the standard tools for Python packaging.

But there are two problems. The first problem, as PEP 518 states:

You can't execute a setup.py file without knowing its dependencies, but currently there is no standard way to know what those dependencies are in an automated fashion without executing the setup.py file where that information is stored. It's a catch-22 of a file not being runnable without knowing its own contents which can't be known programmatically unless you run the file.

Which brings us to the second problem of pip and setuptools being tightly coupled. pip assumes that setuptools is required in order to build a project, and injects setuptools into sys.modules before invoking setup.py. You can't even depend on a specific version of setuptools. Same goes for virtualenv which bootstraps setuptools into all new virtual environments that you create.

The problem with this, as PEP 518 states:

The problem with this, though, is it doesn't scale if another project began to gain traction in the community as setuptools has. It also prevents other projects from gaining traction due to the friction required to use it with a project when pip can't infer the fact that something other than setuptools is required.

This brings us to pyproject.toml, which is introduced by PEP 518 to solve the first problem of setup.py being an executable file.


  [build-system]
  # Minimum requirements for the build system to execute.
  requires = ["setuptools", "wheel"]

Using a pyproject.toml, you can list the build dependencies of your project in a declarative fashion. And tools like pip can use this to make sure that the dependencies are installed before building the project. It also does this in an isolated manner which leads to reproducible builds.

PEP 517 solves the second problem of pip and setuptools being tightly coupled. It says that, since wheels are based on a specification, pretty much all we need from a build backend (like setuptools) is to have some way to spit out standard-compliant wheels. So let's just give the build backend a standard interface which tools like pip can invoke.

It also defines this terminology of a build frontend and backend, kinda like compilers. A build frontend is a tool that users might run to build wheels from source code, which in most cases will be pip. The actual wheel building is done by the build backend, which can be setuptools, or anything else.


          + - - - - - +           + - - - - - - - +
          | frontend  |  -call->  |    backend    |
          |   (pip)   |           |   (anything)  |
          + - - - - - +           + - - - - - - - +

This is how you declare a build backend in a pyproject.toml. The format is the same as a setuptools entrypoint.


  [build-system]
  # Defined by PEP 518:
  requires = ["flit"]
  # Defined by this PEP:
  build-backend = "flit.api:main"

The following is the standard interface it talks about. All build backends must implement these mandatory hooks, which can then be invoked by the build frontend. There are also some other optional hooks which you can read about in PEP 517.


  def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
      ...

  def build_sdist(sdist_directory, config_settings=None):
      ...

These are the steps required to build a wheel:

Get the source code of the project.
Install the build system.
Execute the build system.

PEP 518 deals with step 2 and PEP 517 deals with step 3.

PEP 518 also added the tool section/table. It says that a pyproject.toml file may be used to store tool configuration details. The rule is that a project can use the tool.name section if, and only if, they own the entry for that name in the Python Package Index. This has been picked up by tools like black, flit and poetry.


  [tool.name]
  setting = "value"

flit, poetry and setuptools also share a lot of project metadata, which has led to PEP 621 being proposed earlier this year. The goal of this PEP is to standardize all the project metadata that can be stored in a pyproject.toml so it can be reused across tools. It proposes a project section/table to store this common metadata. Right now PEP 621 is in draft status but there's a good chance that it will get accepted. When that happens, you will be able to write your pyproject.toml like this:


  [build-system]
  requires = ["flit"]
  build-backend = "flit.api:main"

  [project]
  name = "awesome-library"
  version = "0.1.0"
  description = "An awesome library"
  readme = "README.md"
  authors = [
    {name = "Monty Python", email = "monty@python.org"}
  ]
  classifiers = [
    "Development Status :: 4 - Beta",
    "Programming Language :: Python"
  ]

PEP 518 and 517 are opt-in, so if you don't want all these features you can still use a setup.py. In that case, pip will default to the legacy behavior to build your project using setuptools.

I'm really excited to explore how wheel building works when writing Python extensions with C and Rust!

Vinayak Mehta

Day 6 — Some PEP talk, or: What is a pyproject.toml?