Day 32 — A Python C extension module!

Today I did some Yak shaving and made opep because opening/reading each PEP in its own browser tab (with all the extra information on the webpage) was getting tiring both for me and the browser.

I'd used the man command a lot yesterday to read some manual entries, so I thought it might be cool to have a similar interface for reading PEPs!

After getting bamboozled by SWIG's magic yesterday, I started looking for a way to write a Python C extension module from scratch (without SWIG handling all the boilerplate). And I found Ned Batchelder's awesome talk from PyCon 2009 — "A Whirlwind Excursion through Python C Extensions".

I want to learn how to do this so that I can (1) better interface with C code from Python, and then (2) package it for all platforms. Python is written in C and has a C API that we can use to pass things from our C code to our Python code. Ned mentioned that an amazing advantage to write against the same C API as the Python core developers is that their code can be our learning sample! Want to do something similar to a built-in function? Go find its source and learn how it was done!

This is what a "hello world" extension look like:


  // ext1.c: A sample C extension: one simple function

  #include "Python.h"

  // hello_world function.

  static PyObject*
  hello_world(PyObject *self, PyObject *args)
  {
      return Py_BuildValue("s", "Hello, world!");
  }

  // Module functions table.

  static PyMethodDef
  module_functions[] = {
      { "hello_world", hello_world, METH_VARARGS, "Say hello." },
      { NULL }
  };

  // This function is called to initialize the module.

  void
  initext1(void)
  {
      Py_InitModule3("ext1", module_functions, "A minimal module.");
  }

Everything is a PyObject! Here, the hello_world function returns a PyObject which is created using Py_BuildValue. In this case, it's just a string called "Hello, world!" so we pass "s" as the format specifier to Py_BuildValue. You can check out all the format specifiers in the "Parsing arguments and building values" section of Python's C API docs.

We also need to define a list of functions that the module exports. In that list, we specify (1) the name of the Python function, (2) the C function, (3) how the function should be called, and (4) a doc string. Ned mentioned that the whole thing is terminated with a sentinel structure { NULL } which is a common C idiom.

The last thing we need to define is an initext1 function to initialize our module. Ned mentioned that it's the "only symbol exported from the file while others are declared static". I need to find out what this means. What does a symbol look like? Can I cat on something to see a symbol?

The important thing to note with this initext1 function is that its name should follow the initMOD format (where MOD is the name of our extension module, import MOD is how we'll import it in Python). We can use Py_InitModule3 to initialize the module with its name, the function list we defined, and a doc string for the module.

After writing that C file, we need to just write a setup.py like this:


  from setuptools import setup, Extension

  setup(
      ext_modules=[
          Extension("ext1", sources=["ext1.c"]),
      ],
  )

And install the module using:


  $ python setup.py install

After that we can import our shiny module in Python!


  >>> import ext1
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  ImportError: <module>: undefined symbol: Py_InitModule3

Oh no! It raised an ImportError :( Turns out Py_InitModule3 is Python 2 syntax. The Python 3 equivalent of Py_InitModule is PyModule_Create.

Since I'm on Python 3, I had to replace void initext1 with the following code to fix the error:


  static struct PyModuleDef ext1 =
  {
      PyModuleDef_HEAD_INIT,
      "ext1",
      "A minimal module.",
      -1,
      module_functions
  };

  PyMODINIT_FUNC PyInit_ext1(void)
  {
      return PyModule_Create(&ext1);
  }

After reinstalling the module, everything worked like a charm!


  Python 3.8.5 (default, Jul 20 2020, 19:48:14)
  [GCC 7.5.0] on linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import ext1
  >>> ext1.hello_world()
  'Hello, world!'
  >>> ext1
  <module 'ext1' from '/home/vinayak/.virtualenvs/tempenv-620d303837a1d/lib/python3.8/site-packages/UNKNOWN-0.0.0-py3.8-linux-x86_64.egg/ext1.cpython-38-x86_64-linux-gnu.so'>

If Py_BuildValue makes it easy to convert C things into Python things, PyArg_ParseTuple makes it easy to convert Python things into C things. Ned goes on to define another function called string_peek which returns the ASCII value of the character at a particular index in the string.


  static PyObject* string_peek(PyObject *self, PyObject *args)
  {
      const char *pstr;
      int idx;

      if (!PyArg_ParseTuple(args, "si:string_peek", &pstr, &idx)) {
          return NULL;
      }

      int char_value = pstr[idx];

      return Py_BuildValue("i", char_value);
  }

If PyArg_ParseTuple returns false i.e. if it's not able to parse the arguments using the given format specifiers, then we need to return NULL to raise an exception and pass it up the call stack. We can see that if we try passing incorrect arguments to the string_peek function, it'll raise an exception just like any other Python function!


  >>> ext1.string_peek("whirlwind", 5)
  119
  >>> string_peek("whirlwind")
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: string_peek() takes exactly 2 arguments (1 given)
  >>> string_peek("whirlwind", "0")
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: an integer is required

I modified it slightly to return the character at a particular index instead of its ASCII value.


  const char char_value = pstr[idx];

  return Py_BuildValue("c", char_value);


  >>> import ext1
  >>> ext1.string_peek("whirlwind", 5)
  b'w'

There's still a problem with this function where if we ask for the 2000th character, the C code will happily read the contents of memory outside the actual string and return something.


  >>> ext1.string_peek("whirlwind", 2000)
  b'\x00'

To fix that, we can check if the index value lies within our string's length, and raise an exception if it doesn't. To raise an exception using Python's C API, we can call PyErr_SetString to set the error state with the exception type (PyExc_IndexError here), and a message string for the exception. And then finally return NULL to indicate to Python that an exception occurred.


  if (idx < 0 || idx >= strlen(pstr)) {
      PyErr_SetString(PyExc_IndexError, "peek index out of range");
      return NULL;
  }

If we pass in 2000 now, we'll see an exception!


  >>> import ext1
  >>> ext1.string_peek("whirlwind", 2000)
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  IndexError: peek index out of range

If you plan to write Python C extension modules, you need to be extremely careful about memory leaks! To do that you need to understand Python's reference counting model for memory management, how to deal with borrowed and owned references, and how to update reference counts for PyObjects so that Python's garbage collector can clean them up. You can watch Ned's talk for an intro on how to do that. He also goes into implementing a custom type using Python's C API!

Now I need to dive a bit deeper into how linkers and loaders work, and what "symbols" and .so files are.