Step 5 — Tests + CI

The minimum viable test suite

Two things stop a JOSS reviewer’s approval more often than anything else:

  1. “I can’t get it to install.”
  2. “There are no tests.”

Step 4 solved (1) for anyone with pip. This step solves (2).

The good news: JOSS does not ask for high test coverage. It asks that automated tests exist, that they run in CI, and that they exercise the parts of the package a user would actually hit. A well-chosen 20 tests is worth more than 200 tests that all check that 1 + 1 == 2.

Choose pytest

pytest is the de facto standard for scientific Python. It has no ceremony (no test classes needed), gives readable failure output, and integrates with everything.

If you already installed it as part of your dev extras (Step 3):

pip install -e ".[dev]"

Then a smoke test in tests/test_smoke.py:

"""One-second import + version smoke test."""
import yourthing


def test_import():
    assert yourthing is not None


def test_version_exists():
    assert hasattr(yourthing, "__version__")
    assert yourthing.__version__


def test_main_api_exposed():
    from yourthing import main   # or whatever your top-level API looks like
    assert callable(main.run) or hasattr(main, "run")

Three lines of test code catches the most common failure mode: broken imports after a refactor.

Run locally:

pytest -v

What to test — a triage

You do not need to test everything. You need to test:

  1. The things a user’s first import yourthing would touch — top-level imports, version, any auto-loaded config.
  2. The public functions your README’s example uses — because that example is the first thing anyone will try.
  3. Any function whose bug would silently produce wrong scientific results — a mis-scaled normalization, a wrong axis in a fusion step, a resampling method that quietly changes categorical values.
  4. The regression cases you already caught in development — every bug you fixed should get a test so it doesn’t come back.

You do not need to test:

  • Third-party libraries. Trust that numpy.mean returns the mean.
  • Trivial getters and setters.
  • Every combination of every input. Pick a representative sample.

geoai-datacubes at v0.1.0 shipped with ~85 tests across ~15 files. Roughly:

  • ~15 tests on the fusion pipeline (the scientific correctness case).
  • ~15 tests on the mission-registry configuration.
  • ~15 tests on the band-normalisation dispatch table.
  • ~10 tests on tile splitting and NaN handling.
  • ~10 tests on the file-format readers/writers.
  • ~20 tests on smaller helpers.

That coverage was enough to green-light JOSS review.

Add a coverage target — but don’t obsess

pytest-cov gives you a coverage percentage. It’s useful as a direction indicator (add tests to the least-covered file), not as a target (an 80%-covered project can still be poorly tested if the 80% is trivial code).

Aim for ≥ 50% coverage of your yourthing/ package, and no untested public function. Beyond that, extra coverage points are subject to diminishing returns.

pytest --cov=yourthing --cov-report=term-missing

GitHub Actions — run tests on every push

Create .github/workflows/tests.yml:

name: Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest]
        python: ["3.10", "3.11", "3.12"]
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python }}
      - name: Install
        run: |
          pip install --upgrade pip
          pip install -e ".[dev]"
      - name: Run tests
        run: pytest --cov=yourthing --cov-report=xml
      - uses: codecov/codecov-action@v4
        if: matrix.os == 'ubuntu-latest' && matrix.python == '3.11'
        with:
          files: coverage.xml
        continue-on-error: true

This runs on every push and every pull request. Six combinations (2 OS × 3 Python versions), so a matrix job. Coverage is uploaded to Codecov once for the canonical Python version.

TipWindows testing is rarely worth it for research code

The matrix above skips Windows. Most research users install via conda / mamba, which handles OS differences upstream, and Windows CI is where scientific-Python jobs go to die (GDAL, PROJ, PyTorch CUDA all have Windows-specific footguns). If a Windows user files an issue, add a Windows job then. Don’t front-load the pain.

The test badge

Once the tests workflow has run at least once, GitHub gives you a status badge you can paste into your README:

[![Tests](https://github.com/yourorg/yourthing/actions/workflows/tests.yml/badge.svg)](https://github.com/yourorg/yourthing/actions/workflows/tests.yml)

Green tests badge = signal to a JOSS reviewer that you take this seriously.

Slow / expensive tests: gate them

If your library has tests that download real data, hit remote APIs, or run for minutes, they should not run on every push. Two patterns:

Option 1 — pytest marker:

import pytest

@pytest.mark.slow
def test_end_to_end_fusion_pipeline():
    ...
# in tests.yml — skip slow tests by default
- run: pytest -m "not slow"

Slow tests then only run when you invoke pytest -m slow locally, or via a separate scheduled workflow (cron: "0 4 * * 0" — once a week at 4am).

Option 2 — separate workflow:

Put integration/smoke tests in a second workflow triggered on schedule: or workflow_dispatch: only. Keeps the primary tests workflow fast and green.

Checklist

Green CI + a green tests badge? Now let’s give strangers a one-command reproduction environment — on to Step 6 — Docker on GHCR.