Day 39 — manylinux is awesome!

Today I watched these awesome talks on wheel building by Elana Hashman and Paul Kehrer. I also read PEPs 513, 571, 599, and 600 where I learned about manylinux, and how it solved a problem I used to face a lot! It's the problem of installing Python packages that contain C extensions, on Linux.

Some years ago, if you wanted to install cryptography on Linux, you'd have to first install libffi-dev and libssl-dev (or their equivalent) so that the C extension in cryptography is compiled against these shared libraries at install time. With manylinux, a package distributor can pre-compile C extensions against some shared libraries, and bundle them inside a Python wheel, so that you (the user) don't have to think about installing platform-specific dependencies!

Problem

PEP 513 mentions that distribution of Python extensions for Windows and macOS is straightforward. (I need to figure out how this is straightforward, and how to build Windows and macOS wheels!) But each Linux distro has its own package manager, and its own "latest" version of shared libraries. That's a problem because you can easily get version mismatches when you compile an extension against specific versions of some shared libraries on one Linux distro, and then install it on a different Linux distro which might have different versions for these shared libraries.

Solution

To solve this problem, the PyPA came up with a subset of shared libraries that are available on all major Linux distros. So that if you compile (and link) your extension against this subset, it is guaranteed to work on many Linux distros, hence the name! The PyPA put out a Docker image using which you can build manylinux1 (the platform tag) wheels for your extensions! They also put out a tool called auditwheel using which you can check if the manylinux1 wheels you built (with some other build process) are compliant or not, and also bundle shared libraries (not in the defined subset) into those wheels!

PEPs 571 and 599 defined the manylinux2010 and manylinux2014 platform tags, and updated the versions in the shared library subset defined for manylinux1. Wheels built for manylinux1 are automatically compatible with the new tags as these shared libraries (mostly glibc) follow a strictly backwards compatible symbol versioning scheme.

Shared library subset

According to the latest manylinux spec (manylinux2014), this is the shared library subset you can compile against:

  libc.so.6
  libdl.so.2
  libgcc_s.so.1
  libGL.so.1
  libglib-2.0.so.0
  libgobject-2.0.so.0
  libgthread-2.0.so.0
  libICE.so.6
  libm.so.6
  libnsl.so.1
  libpthread.so.0
  libresolv.so.2
  librt.so.1
  libSM.so.6
  libstdc++.so.6
  libutil.so.1
  libX11.so.6
  libXext.so.6
  libXrender.so.1

PEP 600

The tedious process of (1) writing a new PEP, and (2) updating the PyPI + pip codebases for each new manylinux platform tag, is not a good use of limited volunteer time, so PEP 600 defines a new scheme. Tags using the new scheme will look like manylinux_2_17_x86_64, or more generally manylinux_${GLIBCMAJOR}_${GLIBCMINOR}_${ARCH}.

The PEP mentions that this tag is a promise by the wheel creator that the wheel will work on any mainstream Linux distro that uses glibc version ${GLIBCMAJOR}.${GLIBCMINOR} or later, and where the ${ARCH} matches the return value from distutils.util.get_platform(). It also redefined the existing manylinux tags as aliases for this new style. For example:

• manylinux1_x86_64 -> manylinux_2_5_x86_64
• manylinux2010_x86_64 -> manylinux_2_12_x86_64

Because that's the minimum version of glibc in the manylinux1 and manylinux2010 specs! I've omitted a lot of details here, you should totally read the PEPs if you want to learn more.

Thank you PyPA for all your work <3