Hardware tests¶

Audience: contributors with a real rig — or any subset of one (a single Watlow on a USB-serial adapter is enough to run one of the smoke tests). Scope: the hardware marker, the CAPA_HARDWARE_TESTS=1 opt-in, per-vendor connection env vars, the uniform shape every smoke test follows, and when to keep a test as hardware-marked vs. demote it to a sim-backed integration test.

Hardware tests live exclusively under tests/hardware/ and they are all pytest.mark.hardware. Default uv run pytest does not run them. They exist to catch the bugs that only manifest against real instruments: real serial timing, real Modbus framing edge cases, real V4L2 / FLIR camera quirks. They are not a substitute for unit tests against the stub adapters in tests/fixtures/ — those run on every push and catch most regressions before a rig is in the loop.

The opt-in gate¶

A single env var unlocks the entire tree:

$env:CAPA_HARDWARE_TESTS = "1"
uv run pytest tests/hardware

tests/hardware/init.py documents the gate; each smoke module also re-asserts it at module level:

pytestmark = [
    pytest.mark.hardware,
    pytest.mark.anyio,
    pytest.mark.skipif(
        os.environ.get("CAPA_HARDWARE_TESTS") != "1",
        reason="CAPA_HARDWARE_TESTS=1 required",
    ),
]

Unlike alicatlib's three-tier scheme (hardware / hardware_stateful / hardware_destructive), capa uses a single tier. Reasoning: every smoke test in tests/hardware/ is already designed to be non-destructive — the writes they perform are deliberate no-op echoes (set the heater's current setpoint to its current value, set the MFC setpoint to its current setpoint). The gas analyzer's smoke test writes nothing at all: no setting, no front-panel key, no calibration. If a destructive hardware test ever becomes necessary, add a second marker rather than weakening the meaning of hardware.

Per-vendor connection parameters¶

The opt-in env var alone is not enough — each smoke module also reads a small set of CAPA_TEST_<VENDOR>_* env vars that supply the connection parameters. If the vendor's primary env var is unset, the test calls pytest.skip(...) rather than failing, so it's safe to run pytest tests/hardware with only some instruments wired up.

Operator IDs default to "hw-test" and exist so the authorization gate has a real attributable identity in the bundle's event log — they have no rig-side meaning.

Example session running every wired-up smoke test:

$env:CAPA_HARDWARE_TESTS = "1"
$env:CAPA_TEST_WATLOW_PORT = "COM5"
$env:CAPA_TEST_ALICAT_PORT = "COM6"
$env:CAPA_TEST_SARTORIUS_PORT = "COM7"
$env:CAPA_TEST_FUJI_PORT = "COM8"
$env:CAPA_TEST_NIDAQ_DEVICE = "Dev1"
$env:CAPA_TEST_WEBCAM_DEVICE = "USB Video Device"
uv run pytest tests/hardware -v

Devices that aren't wired up will skip cleanly.

The shape every smoke test follows¶

Every vendor smoke module is built the same way. If you add a new device adapter and want to give it a hardware smoke test, mirror this shape exactly so future contributors can read one and understand all of them.

1. Open / identify / close¶

The cheapest check — does the serial port open, does the device respond to its identification query, does close() clean up.

class TestRealWatlow:
    async def test_open_identify_close(self) -> None:
        adapter = WatlowAdapter(name="heater", **_watlow_params())
        await adapter.open()
        try:
            assert adapter.device_info is not None
        finally:
            await adapter.close()

The try / finally is non-negotiable. A test that throws between open() and close() and leaves the port held breaks every subsequent test that wants that port.

2. Short headless freerun → sealed bundle¶

The integration-shaped check: build a minimal ExperimentConfig with one device, run run_headless for a few seconds, assert the bundle ended completed + sealed + ok and that both device_records/<vendor>.parquet and scalars.parquet were populated.

async def _go() -> Path:
    result = await run_headless(config, runs_root=tmp_path / "runs")
    assert result.bundle_path is not None
    assert result.run_status == "completed", result.exit_reason
    assert result.bundle_status == "sealed", result.exit_reason
    assert result.integrity_status == "ok", result.exit_reason
    return result.bundle_path

Each of the three assertions includes result.exit_reason as its message because the only context you'll have when this fails in three months is the pytest output.

3. Authorized no-op write (where applicable)¶

For controllers (not pure sensors), read the current setpoint, write the same value back, and read it again to check it did not move. Echo the setpoint itself, never the process value: writing the PV as the setpoint changes the controller. The physical state doesn't change, but the test exercises the Authorization + confirm=True round-trip through the real adapter — the path that goes wrong silently if anything in the authorization plumbing drifts. The test is automatically skipped if the connected device is a meter rather than a controller.

See authorization gates for why this check exists at all.

Fixture hygiene rules¶

These exist because hardware tests don't have the same isolation that pytest gives unit tests:

1. Always try / finally an open(). A leaked serial handle silently breaks the next test that wants the port.
2. Don't share an adapter across tests. Each test opens its own; modular state on the adapter object survives the test boundary otherwise.
3. No pytest.fixture(scope="session") for hardware connections. Some platforms misreport which test was running when an instrument-side error fires — per-test open isolates blame.
4. Tag any new hardware test with the same three-element pytestmark list (the hardware marker, anyio, and the skipif on CAPA_HARDWARE_TESTS). Don't rely on the package-level __init__.py alone; the per-module re-assertion is what gives you a useful skip reason in the pytest output.

When a hardware test should become a sim-backed integration test¶

If the bug a hardware test catches can be reproduced via a sim adapter, promote it to tests/integration/. Reasoning:

• CI never sees tests/hardware/ (it can't — no rig in the GitHub Actions runner). A bug only covered by a hardware test is effectively only covered when a contributor is sitting at the rig PC.
• Sim adapters are deliberately built to mirror the real adapter's emission shape (AlicatFrameField on a sim looks like one on a real Alicat). Most protocol-shape bugs reproduce on the sim.
• The remaining hardware-only failure surface — Modbus framing under real serial timing, FLIR's USB enumeration quirks, V4L2 input-format negotiation — is small and worth keeping in tests/hardware/. Everything else belongs in integration/.

A test should stay hardware-marked only if (a) reproducing it requires real-instrument timing or framing behaviour that sim can't fake, or (b) it is explicitly checking that the adapter physically opens and closes the underlying device handle.