One of the hardest parts of doing a new release of CPython is getting people to test it before the final release goes out. While CPython has a test suite and we try to do our best to have good code coverage (currently sitting at above 83%), nothing beats having people run CPython against real code because people use Python's features in ways we never anticipated all the time. But convincing people to download a e.g. release candidate of CPython and test it against their code is always difficult since it takes time to do so. But what if it was possible to test your code against the in-development branches of CPython without having to do so much as change a single configuration file? With Travis you can actually do just that and help test CPython!

Testing the next bugfix releases

In your configuration to run Travis with Python you probably have a configuration like the following:

language: python
cache: pip
python:
  - "3.6"
install: "pip install --upgrade -r requirements.txt"
script: pytest

That will run your pytest tests using Python 3.6 (while caching the wheels so that Travis runs faster and you don't constantly download the same wheels repeatedly). But did you know that Travis lets you run the in-development branches of Python 3.5 and 3.6? This lets you test the branches that will become e.g. 3.5.4 and 3.6.1. And adding them to your build matrix is really simple:

language: python
cache: pip
python:
  - "3.6"
  - "3.6-dev"
install: "pip install --upgrade -r requirements.txt"
script: pytest

By adding 3.6-dev you get to run your tests against the 3.6 branch of CPython's git repository. This is really helpful to the Python development team since bugfix branches should be very stable and more bug-free than the version of Python you're running in production, so if you find a bug in that branch it really should get fixed. This also helps you know if you're accidentally using some semantics that were buggy and got fixed as soon as we make the change to fix the bug. Everyone benefits!

Keeping CPython from making your CI fail

While we try to keep the bugfix branches bug-free, we are still human beings and mistakes can be made. If you're worried about a bug in CPython making your CI turn red, you can turn on some settings in your Travis configuration so that failures in the bugfix branches don't count against you (although if you can, please don't follow the rest of this subsection as it means there's less of a chance you will notice when your code will fail in the future or that your code has found a bug in CPython itself).

First, you can tell Travis when something is allowed to fail:

language: python
cache: pip
python:
  - "3.6"
  - "3.6-dev"

matrix:
  allow_failures:
    python: "3.6-dev"

install: "pip install --upgrade -r requirements.txt"
script: pytest

The allow_failures setting tells Travis that the matching matrix entries can fail. So if the 3.6-dev build fails your tests then you are still considered passing if 3.6 passes.

You can also make it so that as soon as the 3.6 build passes you are considered passing overall and Travis doesn't have to wait for 3.6-dev to finish:

language: python
cache: pip
python:
  - "3.6"
  - "3.6-dev"

matrix:
  fast_finish: true
  allow_failures:
    python: "3.6-dev"

install: "pip install --upgrade -r requirements.txt"
script: pytest

The fast_finish setting simply tells Travis that any build that is allowed to fail doesn't hold up Travis reporting the status of the overall CI run.

As I said at the beginning of this subsection, please don't switch fast_finish or allow_failures on if you can help it. Having Travis consider a 3.6-dev failure a testing failure will either actively tell you that your code will need updating for e.g. Python 3.6.1 before it's released publicly or it will let you know to report a bug at bugs.python.org against CPython itself. And if you don't use allow_failures then you can add 3.5-dev to your matrix to help test that branch as well.

Testing the next feature release

Other than 3.5-dev and 3.6-dev, Travis also supports 3.7-dev and nightly. Since Python 3.7 is under active development I can't guarantee it's stability or that things won't change from underneath you, so I'm not going to recommend testing against it yet. But when we announce 3.7.0b1, expect a plea from me to add 3.7-dev to your build matrix to help us test 3.7.0 before it's released publicly. I'm sure people will reshare this post when the time is right to do this, but when the time comes it would be fantastic if you could help us test the next feature release of CPython on top of the next bugfix releases. (And you could test against nightly, but that might be too bleeding edge.)

Testing on a regular basis

The last thing you can do to help out is to set up a testing cron job. By having Travis always run your CI daily, weekly, or monthly, you can make sure that your project works against the version(s) of Python that you have specified. This is helpful because if you are not committing code on a regular basis then you won't be triggering your CI to notice when a change in Python breaks your code. But by running your tests e.g. monthly you don't have to wait until someone send a new pull request to notice the breakage. A cron job is also helpful if you run tools like mypy unpinned in order to get better checks from new releases, as the cron job will make sure you pick up the new version on a regular basis.