Testing a Python project using the WASI build of CPython with pytest
As part of bringing Python to the browser via vscode.dev, I looked into what it looks like today (January 2023) to test a Python project that uses pytest with a WASI build of CPython (see my post on WebAssembly and its various platforms if you don't know what "WASI" means). It turns out not to be hard, but there are a few steps to making this all work and some of them may be a bit non-obvious, so I decided to write it all down for those who are interest in trying out some bleeding edge WebAssembly stuff with Python.
Step 1: get the project's source code
For my running example I'm going to use the
packaging project. I'm one of the maintainers and I thought I wasn't going to have any install dependency issues since
packaging has none. (Turns out I forgot about testing dependencies, but we will get to that. 😅)
Due note that I changed the directory to the checkout in that command as the rest of the commands in this post will assume you're in the
packaging/ directory you just checked out into for convenience (and as to why that's convenient, that will be explained below).
Step 2: get a WASI build of CPython
https://github.com/tiran/cpython-wasm-test/releases has some unofficial, pre-built WASI binaries that we can quickly grab for our use case today:
curl -O --location "https://github.com/tiran/cpython-wasm-test/releases/download/v3.11.0/Python-3.11.0-wasm32-wasi-16.zip"
-Omeans to save the file to the same name as the last part of the URL, and
--locationmeans to follow redirects.
Once the zip file is downloaded you will want to extract the files. Now, if you happen to have
bsdtar installed (on Fedora it's available in the
bsdtar package, on Ubuntu it's included in the
libarchive-tools apt package), you can extract the zip in-place with your project's code using the convenient
bsdtar -x -f Python-3.11.0-wasm32-wasi-16.zip --strip-components 1
If you don't have
bsdtar and instead use
unzip, you will want to copy the contents of the zip file into the directory storing your project's source code. Now you can technically have the WASI-related files live somewhere else, but this is a quick-and-dirty demo and this will simplify a step later on.
Step 3: install
To run WebAssembly code compiled for WASI, you need a WASI runtime. In this instance we will use
wasmtime since it's the one I'm the most familiar with, seems to have the most broad support, and the team was really nice to me when I file a bug against the project.
If you're a Homebrew user (macOS or Linux), it's available in the
brew install wasmtime
Once that's installed we can do a quick check that everything is working. The following command should print out
wasi as the value for
wasmtime run --dir . python.wasm -- -c "import sys; print(sys.platform)"
If you drop everything past the
-- it will launch the REPL.
You might be wondering what the
--dir . is for? Well, it has to do with WASI's security model. The reason WebAssembly code requires a runtime even when compiled to native code is because it uses a capability-based security model. What that means is the WASI code only has access to what you give it, and that's all controlled by the runtime. So by saying
--dir ., we are effectively telling
wasmtime to give the WASI code access to the current directory. But by not specifying any other directories, the WASI code can only access the current directory and no other part of your file system! This is why some are viewing WASI as a way to move away from Docker as a containment/security strategy.
Remember earlier when I said it would be easier if you copied those files you unzipped into your checkout directory? The reason for that is this WASI build of CPython requires that
lib/ directory to be mapped to
/lib within the runtime. And while
wasmtime has a
--mapdir argument which you can use to control where directories get mapped to in the file system within the runtime (the host directory and guest directory, respectively), it's easier to just say
--dir . as that automatically mapps the current directory to
/. (There's an open bug about seeing if that restriction can be changed for CPython.)
Step 4: install your dependencies
Because WASI support for Python projects is a fairly new concept, there isn't an explicit way to know if a project is actually compatible with a WASI build of CPython (which can be due to various things, such as WASI not fully supporting sockets, threads, etc. right now). As such, the best you can do today to tell if a project is compatible with a WASI build of CPython is to only consider projects (and their dependencies) that have a pure Python wheel as potentially compatible with WASI (i.e.
Unfortunately, due to WASI lacking socket support, you can't just run
pip using that WASI build of CPython you just downloaded and get the result you want. But luckily
pip has enough command-line options for you tell it you only want pure Python wheels that are compatible with Python 3.11, to match the version we downloaded earlier (so you can technically run the command below with any version of Python that you want).
Another wrinkle at this point is there isn't a concept of a virtual environment with a WASI build of CPython. This makes some sense when you think about how
wasmtime is the command you use to run Python, and so it doesn't operate like a symlink or copy of Python like you normally see for the
python command in your virtual environment. That means we need to install the dependencies into the current directory. I also have
PIP_REQUIRES_VIRTUALENV always set to make sure I never install into my Python interpeter directly, so I need to make sure to turn that off since there won't be a virtual environment.
In terms of dependencies to install, since
packaging has no install dependencies, we just have to worry about installing its testing dependencies which are specified in
I'm also using the Python Launcher for Unix because I can. 😁
Now unfortunately that command will fail:
ERROR: Could not find a version that satisfies the requirement coverage[toml]>=5.0.0 (from versions: none) ERROR: No matching distribution found for coverage[toml]>=5.0.0
It turns out that (the latest) coverage.py 7.0.5 lacks a pure Python wheel. That means we need to comment out that requirement from
tests/requirements.txt. Doing that will allow the rest of the dependencies to be installed successfully.
Step 5: run
Now naively, I tried just running
pytest in hopes it would work.
wasmtime --dir . python.wasm -- -m pytest tests/
Unfortunately, I wasn't that lucky:
Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/pytest/__main__.py", line 5, in <module> raise SystemExit(pytest.console_main()) ^^^^^^^^^^^^^^^^^^^^^ File "/_pytest/config/__init__.py", line 190, in console_main code = main() ^^^^^^ File "/_pytest/config/__init__.py", line 148, in main config = _prepareconfig(args, plugins) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/_pytest/config/__init__.py", line 329, in _prepareconfig config = pluginmanager.hook.pytest_cmdline_parse( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/pluggy/_hooks.py", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/pluggy/_manager.py", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/pluggy/_callers.py", line 55, in _multicall gen.send(outcome) File "/_pytest/helpconfig.py", line 103, in pytest_cmdline_parse config: Config = outcome.get_result() ^^^^^^^^^^^^^^^^^^^^ File "/pluggy/_result.py", line 60, in get_result raise ex.with_traceback(ex) File "/pluggy/_callers.py", line 39, in _multicall res = hook_impl.function(*args) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/_pytest/config/__init__.py", line 1058, in pytest_cmdline_parse self.parse(args) File "/_pytest/config/__init__.py", line 1346, in parse self._preparse(args, addopts=addopts) File "/_pytest/config/__init__.py", line 1248, in _preparse self.hook.pytest_load_initial_conftests( File "/pluggy/_hooks.py", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/pluggy/_manager.py", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/pluggy/_callers.py", line 60, in _multicall return outcome.get_result() ^^^^^^^^^^^^^^^^^^^^ File "/pluggy/_result.py", line 60, in get_result raise ex.with_traceback(ex) File "/pluggy/_callers.py", line 34, in _multicall next(gen) # first yield ^^^^^^^^^ File "/_pytest/capture.py", line 141, in pytest_load_initial_conftests capman.start_global_capturing() File "/_pytest/capture.py", line 688, in start_global_capturing self._global_capturing = _get_multicapture(self._method) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/_pytest/capture.py", line 630, in _get_multicapture return MultiCapture(in_=FDCapture(0), out=FDCapture(1), err=FDCapture(2)) ^^^^^^^^^^^^ File "/_pytest/capture.py", line 381, in __init__ self.targetfd_save = os.dup(targetfd) ^^^^^^^^^^^^^^^^ OSError: [Errno 58] Not supported
It turns out that
os.dup() in a couple of places and that function is not available in WASI. Luckily, if you set the
-s flag to not capture stdout and
--p no:faulthandler to turn off using
faulthandler, you skip over the
os.dup() calls. But even with those flags set, you get another error:
INTERNALERROR> Traceback (most recent call last): INTERNALERROR> File "/_pytest/main.py", line 266, in wrap_session INTERNALERROR> config._do_configure() INTERNALERROR> File "/_pytest/config/__init__.py", line 1037, in _do_configure INTERNALERROR> self.hook.pytest_configure.call_historic(kwargs=dict(config=self)) INTERNALERROR> File "/pluggy/_hooks.py", line 277, in call_historic INTERNALERROR> res = self._hookexec(self.name, self.get_hookimpls(), kwargs, False) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/pluggy/_manager.py", line 80, in _hookexec INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/pluggy/_callers.py", line 60, in _multicall INTERNALERROR> return outcome.get_result() INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/pluggy/_result.py", line 60, in get_result INTERNALERROR> raise ex.with_traceback(ex) INTERNALERROR> File "/pluggy/_callers.py", line 39, in _multicall INTERNALERROR> res = hook_impl.function(*args) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/_pytest/logging.py", line 533, in pytest_configure INTERNALERROR> config.pluginmanager.register(LoggingPlugin(config), "logging-plugin") INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/_pytest/logging.py", line 567, in __init__ INTERNALERROR> self.log_file_handler = _FileHandler(log_file, mode="w", encoding="UTF-8") INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/lib/python3.11/logging/__init__.py", line 1181, in __init__ INTERNALERROR> StreamHandler.__init__(self, self._open()) INTERNALERROR> ^^^^^^^^^^^^ INTERNALERROR> File "/lib/python3.11/logging/__init__.py", line 1213, in _open INTERNALERROR> return open_func(self.baseFilename, self.mode, INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> FileNotFoundError: [Errno 44] No such file or directory: '/dev/null'
Remember how WASI's capability-based security model doesn't let code access things from the operating system you don't explicitly give it access to? This is a perfect example of that as the
logging module wants access to
/dev/null, but we didn't grant access to
/dev, so it fails. So let's add it:
wasmtime --dir . --dir /dev python.wasm -- -m pytest -s -p no:faulthandler tests/
And that runs successfully! But there are test failures that shouldn't be failing ... 🤔
It turns out one of our test dependencies depends on
packaging itself, so it installed it from PyPI while running the latest tests from the source checkout. 😅 Simply deleting the
packaging directory and corresponding
.dist-info directory removes that installed copy of the project.
But if you delete the
packaging directory that
py -m pip install --target . put there, you start getting import errors in the test code. That's due to the project using a
src/ layout and thus hiding the code away from the top-level directory of the project (which, unfortunately in this case, it's designed to cause).
You might be wondering why I didn't do an editable install with
pip by using
-e ., but that actually won't work. For bootstrapping reasons,
flit-core as its build system and that uses a
.pth file to implement editable installs. Unfortunately,
.pth files only work when they are in
site-packages. And how do you normally install something into
site-packages? Via a virtual environment which we don't have. 😅
So, we have to fake an editable install the old-fashioned way: with a symlink.
ln -s src/packaging
And with that, everything works as expected (including the one test failure 😅).
Step 6: report/fix the bug(s)
It turns out one of the tests requires
ctypes to exist even though the import is guarded so that it can fail. I've reported the issue so we can fix that.
Step 7: record that the project is WASI-compatible
As part of this work to see what it currently takes to test a project's compatibility with WASI, I realized there's no way for a project to declare on PyPI that it's compatible with WebAssembly via its pure Python wheel. To fix this I got some new classifiers for PyPI added related to WebAssembly:
- Environment :: WebAssembly
- Environment :: WebAssembly :: Emscripten
- Environment :: WebAssembly :: WASI
This way projects can declare whether they are compatible with a WebAssembly build regardless of how it was built, compatible with an Emscripten build of Python (e.g. Pyodide), or with a WASI build (like we have been using here). These were added literally the day this post went out, so there's nothing under those classifiers yet, but hopefully someday there will be some projects listing their WebAssembly support. 🤞
Bonus: faster execution
wasmtime has an ahead-of-time (AOT) compiler which will take your WASI code and translates it to a native binary with a
.cwasm file extension.
wasmtime compile python.wasm
You still need to use
wasmtime to enforce the security model, but it does let code run faster compared to running the
.wasm file directly. Also note you need to pass the
--allow-precompiled flag to use your new
wasmtime --dir . --dir /dev --allow-precompiled python.cwasm -- -m pytest -s -p no:faulthandler tests/
Depending on what you're doing and how long your code runs, the results can be drastic or minimal. Thanks to the JIT that
wasmtime has, the difference in running the
packaging test suite is 46.3 seconds straight versus 44.5 seconds compiled (5% speedup). But if you just look at startup with
-c "pass", it's 140ms straight and 17ms compiled (88% speedup). Since the compliation is pretty fast, I don't see a reason not to do it.