10 min read WebAssembly

Testing a Python project using the WASI build of CPython with pytest

As part of bringing Python to the browser via vscode.dev, I looked into what it looks like today (January 2023) to test a Python project that uses pytest with a WASI build of CPython (see my post on WebAssembly and its various platforms if you don't know what "WASI" means). It turns out not to be hard, but there are a few steps to making this all work and some of them may be a bit non-obvious, so I decided to write it all down for those who are interest in trying out some bleeding edge WebAssembly stuff with Python.

💡
Since WASI is a tier 3 platform for CPython as of this writing (January 2023), various details are subject to change in the future (hopefully for the better).

Step 1: get the project's source code

For my running example I'm going to use the packaging project. I'm one of the maintainers and I thought I wasn't going to have any install dependency issues since packaging has none. (Turns out I forgot about testing dependencies, but we will get to that. 😅)

git clone https://github.com/pypa/packaging.git && cd packaging
Clone and cd into a checkout of the packaging project

Due note that I changed the directory to the checkout in that command as the rest of the commands in this post will assume you're in the packaging/ directory you just checked out into for convenience (and as to why that's convenient, that will be explained below).

Step 2: get a WASI build of CPython

https://github.com/tiran/cpython-wasm-test/releases has some unofficial, pre-built WASI binaries that we can quickly grab for our use case today:

curl -O --location "https://github.com/tiran/cpython-wasm-test/releases/download/v3.11.0/Python-3.11.0-wasm32-wasi-16.zip"
💡
If you're not familiar with curl, -O means to save the file to the same name as the last part of the URL, and --location means to follow redirects.
💡
There is no Anaconda or conda-forge build of CPython for WASI that I'm aware of.

Once the zip file is downloaded you will want to extract the files. Now, if you happen to have bsdtar installed (on Fedora it's available in the bsdtar package, on Ubuntu it's included in the libarchive-tools apt package), you can extract the zip in-place with your project's code using the convenient --strip-components flag:

bsdtar -x -f Python-3.11.0-wasm32-wasi-16.zip --strip-components 1

If you don't have bsdtar and instead use unzip, you will want to copy the contents of the zip file into the directory storing your project's source code. Now you can technically have the WASI-related files live somewhere else, but this is a quick-and-dirty demo and this will simplify a step later on.

Step 3: install wasmtime

To run WebAssembly code compiled for WASI, you need a WASI runtime. In this instance we will use wasmtime since it's the one I'm the most familiar with, seems to have the most broad support, and the team was really nice to me when I file a bug against the project.

If you're a Homebrew user (macOS or Linux), it's available in the wasmtime formula.

brew install wasmtime

Once that's installed we can do a quick check that everything is working. The following command should print out wasi as the value for sys.platform:

wasmtime run --dir . python.wasm -- -c "import sys; print(sys.platform)"

If you drop everything past the -- it will launch the REPL.

You might be wondering what the --dir . is for? Well, it has to do with WASI's security model. The reason WebAssembly code requires a runtime even when compiled to native code is because it uses a capability-based security model. What that means is the WASI code only has access to what you give it, and that's all controlled by the runtime. So by saying --dir ., we are effectively telling wasmtime to give the WASI code access to the current directory. But by not specifying any other directories, the WASI code can only access the current directory and no other part of your file system! This is why some are viewing WASI as a way to move away from Docker as a containment/security strategy.

💡
The use of a runtime to control access to the operating system is why we are using WASI for VS Code for this project. By letting VS Code itself be a WASI runtime, we can make it so that things like file access pass through VS Code APIs. That lets code compiled for WASI have access to any files VS Code has access to, no matter where those files are (e.g. locally on disk, remotely on GitHub, etc.). Since the goal is to have Python work anywhere VS Code does, this gives us a really nice portability story regardless of whether you're running VS Code on your desktop or the web (and all the while working against standards as defined by WASI itself).

Remember earlier when I said it would be easier if you copied those files you unzipped into your checkout directory? The reason for that is this WASI build of CPython requires that lib/ directory to be mapped to /lib within the runtime. And while wasmtime has a --mapdir argument which you can use to control where directories get mapped to in the file system within the runtime (the host directory and guest directory, respectively), it's easier to just say --dir . as that automatically mapps the current directory to /. (There's an open bug about seeing if that restriction can be changed for CPython.)

Step 4: install your dependencies

Because WASI support for Python projects is a fairly new concept, there isn't an explicit way to know if a project is actually compatible with a WASI build of CPython (which can be due to various things, such as WASI not fully supporting sockets, threads, etc. right now). As such, the best you can do today to tell if a project is compatible with a WASI build of CPython is to only consider projects (and their dependencies) that have a pure Python wheel as potentially compatible with WASI (i.e. py3-none-any wheels).

Unfortunately, due to WASI lacking socket support, you can't just run pip using that WASI build of CPython you just downloaded and get the result you want. But luckily pip has enough command-line options for you tell it you only want pure Python wheels that are compatible with Python 3.11, to match the version we downloaded earlier (so you can technically run the command below with any version of Python that you want).

Another wrinkle at this point is there isn't a concept of a virtual environment with a WASI build of CPython. This makes some sense when you think about how wasmtime is the command you use to run Python, and so it doesn't operate like a symlink or copy of Python like you normally see for the python command in your virtual environment. That means we need to install the dependencies into the current directory. I also have PIP_REQUIRES_VIRTUALENV always set to make sure I never install into my Python interpeter directly, so I need to make sure to turn that off since there won't be a virtual environment.

In terms of dependencies to install, since packaging has no install dependencies, we just have to worry about installing its testing dependencies which are specified in tests/requirements.txt.

I'm also using the Python Launcher for Unix because I can. 😁

PIP_REQUIRE_VIRTUALENV=0 py -m pip install --target . --only-binary :all: --implementation py --abi none --platform any --python-version "3.11" -r tests/requirements.txt
Command to install pure Python wheels into the current directory for Python 3.11 using the tests/requirements.txt requirements file

Now unfortunately that command will fail:

ERROR: Could not find a version that satisfies the requirement coverage[toml]>=5.0.0 (from versions: none)
ERROR: No matching distribution found for coverage[toml]>=5.0.0

It turns out that (the latest) coverage.py 7.0.5 lacks a pure Python wheel. That means we need to comment out that requirement from tests/requirements.txt. Doing that will allow the rest of the dependencies to be installed successfully.

Step 5: run pytest

Now naively, I tried just running pytest in hopes it would work.

wasmtime --dir . python.wasm -- -m pytest tests/

Unfortunately, I wasn't that lucky:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/pytest/__main__.py", line 5, in <module>
    raise SystemExit(pytest.console_main())
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/_pytest/config/__init__.py", line 190, in console_main
    code = main()
           ^^^^^^
  File "/_pytest/config/__init__.py", line 148, in main
    config = _prepareconfig(args, plugins)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/_pytest/config/__init__.py", line 329, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/_pytest/helpconfig.py", line 103, in pytest_cmdline_parse
    config: Config = outcome.get_result()
                     ^^^^^^^^^^^^^^^^^^^^
  File "/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/_pytest/config/__init__.py", line 1058, in pytest_cmdline_parse
    self.parse(args)
  File "/_pytest/config/__init__.py", line 1346, in parse
    self._preparse(args, addopts=addopts)
  File "/_pytest/config/__init__.py", line 1248, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
           ^^^^^^^^^^^^^^^^^^^^
  File "/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/pluggy/_callers.py", line 34, in _multicall
    next(gen)  # first yield
    ^^^^^^^^^
  File "/_pytest/capture.py", line 141, in pytest_load_initial_conftests
    capman.start_global_capturing()
  File "/_pytest/capture.py", line 688, in start_global_capturing
    self._global_capturing = _get_multicapture(self._method)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/_pytest/capture.py", line 630, in _get_multicapture
    return MultiCapture(in_=FDCapture(0), out=FDCapture(1), err=FDCapture(2))
                            ^^^^^^^^^^^^
  File "/_pytest/capture.py", line 381, in __init__
    self.targetfd_save = os.dup(targetfd)
                         ^^^^^^^^^^^^^^^^
OSError: [Errno 58] Not supported

It turns out that pytest uses os.dup() in a couple of places and that function is not available in WASI. Luckily, if you set the --capture=no/-s flag to not capture stdout and --p no:faulthandler to turn off using faulthandler, you skip over the os.dup() calls. But even with those flags set, you get another error:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/_pytest/main.py", line 266, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File "/_pytest/config/__init__.py", line 1037, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File "/pluggy/_hooks.py", line 277, in call_historic
INTERNALERROR>     res = self._hookexec(self.name, self.get_hookimpls(), kwargs, False)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/_pytest/logging.py", line 533, in pytest_configure
INTERNALERROR>     config.pluginmanager.register(LoggingPlugin(config), "logging-plugin")
INTERNALERROR>                                   ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/_pytest/logging.py", line 567, in __init__
INTERNALERROR>     self.log_file_handler = _FileHandler(log_file, mode="w", encoding="UTF-8")
INTERNALERROR>                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/lib/python3.11/logging/__init__.py", line 1181, in __init__
INTERNALERROR>     StreamHandler.__init__(self, self._open())
INTERNALERROR>                                  ^^^^^^^^^^^^
INTERNALERROR>   File "/lib/python3.11/logging/__init__.py", line 1213, in _open
INTERNALERROR>     return open_func(self.baseFilename, self.mode,
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> FileNotFoundError: [Errno 44] No such file or directory: '/dev/null'

Remember how WASI's capability-based security model doesn't let code access things from the operating system you don't explicitly give it access to? This is a perfect example of that as the logging module wants access to /dev/null, but we didn't grant access to /dev, so it fails. So let's add it:

wasmtime --dir . --dir /dev python.wasm -- -m pytest -s -p no:faulthandler tests/

And that runs successfully! But there are test failures that shouldn't be failing ... 🤔

💡
I have opened an issue with pytest about requiring those flags due to os.dup() .

It turns out one of our test dependencies depends on packaging itself, so it installed it from PyPI while running the latest tests from the source checkout. 😅 Simply deleting the packaging directory and corresponding .dist-info directory removes that installed copy of the project.

But if you delete the packaging directory that py -m pip install --target . put there, you start getting import errors in the test code. That's due to the project using a src/ layout and thus hiding the code away from the top-level directory of the project (which, unfortunately in this case, it's designed to cause).

You might be wondering why I didn't do an editable install with pip by using -e ., but that actually won't work. For bootstrapping reasons, packaging uses flit-core as its build system and that uses a .pth file to implement editable installs. Unfortunately, .pth files only work when they are in site-packages. And how do you normally install something into site-packages? Via a virtual environment which we don't have. 😅

So, we have to fake an editable install the old-fashioned way: with a symlink.

ln -s src/packaging

And with that, everything works as expected (including the one test failure 😅).

Step 6: report/fix the bug(s)

It turns out one of the tests requires ctypes to exist even though the import is guarded so that it can fail. I've reported the issue so we can fix that.

Step 7: record that the project is WASI-compatible

As part of this work to see what it currently takes to test a project's compatibility with WASI, I realized there's no way for a project to declare on PyPI that it's compatible with WebAssembly via its pure Python wheel. To fix this I got some new classifiers for PyPI added related to WebAssembly:

  • Environment :: WebAssembly
  • Environment :: WebAssembly :: Emscripten
  • Environment :: WebAssembly :: WASI

This way projects can declare whether they are compatible with a WebAssembly build regardless of how it was built, compatible with an Emscripten build of Python (e.g. Pyodide), or with a WASI build (like we have been using here). These were added literally the day this post went out, so there's nothing under those classifiers yet, but hopefully someday there will be some projects listing their WebAssembly support. 🤞

Bonus: faster execution

wasmtime has an ahead-of-time (AOT) compiler which will take your WASI code and translates it to a native binary with a .cwasm file extension.

wasmtime compile python.wasm

You still need to use wasmtime to enforce the security model, but it does let code run faster compared to running the .wasm file directly. Also note you need to pass the --allow-precompiled flag to use your new python.cwasm file.

wasmtime --dir . --dir /dev --allow-precompiled python.cwasm -- -m pytest -s -p no:faulthandler tests/

Depending on what you're doing and how long your code runs, the results can be drastic or minimal. Thanks to the JIT that wasmtime has, the difference in running the packaging test suite is 46.3 seconds straight versus 44.5 seconds compiled (5% speedup). But if you just look at startup with -c "pass", it's 140ms straight and 17ms  compiled (88% speedup). Since the compliation is pretty fast, I don't see a reason not to do it.