Try to not use the Python C API directly

If there is one persnickety thing people ask of python-dev it is for the GIL to go away. People view it as this great bugaboo of performance that hinders CPython in a world where single-core chips are becoming extremely rare. Whether the GIL is really a performance killer as some people believe (it really depends on the type of work you want to run in parallel), the GIL's shadow does linger over CPython.

If you read the LWN writeup on the PyCon 2015 language summit entitled "Making Python more attractive" you will notice that the topic of removing the GIL in Python 3 did come up. But you will notice that there was one consistent theme in the discussions regarding the GIL: extension modules would need to change in some form. Even the least invasive suggestion of making sub-interpreters work could possibly require some extension modules to be reworked in order to be re-entrant. Other approaches that focused more on removing reference counting and switching to garbage collection required more changes that ranged from the mechanical -- e.g., adding a parameter on every function -- to even deeper changes that are not as mechanical.

The key point is that ridding CPython of the GIL requires at least "bending" the C API -- as Larry Hastings has explained it is having to tweak the murky corners of extension module code -- if not full-out changes of how extension modules work that will require code changes for everyone. To illustrate how the C API can hinder things, Larry told me a story about how PyPy has changed garbage collection schemes three times. Now if PyPy had a C API that they directly exposed to users like CPython has they would not have been able to change their garbage collection scheme without breaking extension modules (he told the story in his EuroPython 2015 GIL talk). It's this kind of inflexibility that the C API forces upon CPython that makes removing the GIL cleanly basically impossible.

What all of this means is that if you want to try and shield yourself from any potential bending of the C API that could occur in the future in order to get rid of the GIL then I would recommend not using CPython's C API directly and instead use a library that will assist you (this will also help you transition to Python 3). Which one you use will depend on why you want to use CPython's C API in the first place.

Calling C code

If you wanted to use CPython's C API in order to call other C code such as some C library, then I would recommend looking at CFFI. It's the most straight-forward for wrapping C code and it is explicitly designed for interfacing with C code. It also has the benefit of working with both PyPy and CPython.

You can also use Cython or SWIG to wrap the C code you want to interface with. In either case, though, PyPy compatibility won't be quite as good and it does require working in a DSL that some people might not like.

For performance

If you're using CPython's C API for performance then what you want to use will depend on whether you are using NumPy in your code. If you are then you should see if Numba fits your needs. It will apply a JIT to your NumPy code by doing nothing more than adding a decorator to some functions which is about as easy as it can get.

If your performance needs extend beyond Numba, then Cython will be your best bet. While you will need to write in Cython's hybrid language, it will probably be the easiest way to get better performance short of writing C code directly.

If writing C code for performance appeals to you then you can use CFFI or SWIG to wrap your hand-rolled C code. But please do consider Numba and/or Cython first as they will be easier to use for good performance gains.

Conclusion

The key takeaway I hope people have from this is that you should use a library/tool to help you interface Python with C code, whether it's for performance or FFI needs. While the C API for CPython helped it gain its reputation as a great glue language, that doesn't mean you need to continue to work with it directly with so many great projects out there to help make your life easier. So please consider one of the libraries/tools I mentioned the next time you start a new project that involves some C code and do consider tossing your hand-rolled code to make maintenance easier for you.