Network replay for fabric testing — predictable, modular, fast

The problem it solves (why the 25-minute test failed)

A single fabric test today runs a long, stateful, internet-dependent chain:

boot workstation VM → cloud-init (apt install OVS+incus from mirrors) → incus admin init
→ fabric_up → fabric_launch (incus pulls images:debian/12 over the uplink) → DHCP → assert

Most of the wall-clock and nearly all of the non-determinism live in the network-facing parts: the Ubuntu mirror during cloud-init, and the Incus image server pull inside fabric_launch (images:debian/12 is hundreds of MB, fetched live, subject to upstream latency/availability). A test that has to pull an image over the network before it can assert anything is neither fast nor repeatable — that is the 25-minute failure.

Network replay = record the network-dependent interactions once, then replay them from a local, content-addressed fixture. It makes testing:

- predictable — same bytes every run; no mirror flakiness, no image-server outages, no DHCP race - fast — the inner loop reads from a local cache instead of the internet - modular — each network layer becomes an independent fixture, so layers can be tested in isolation

"The network" is four layers — replay each differently

| Layer | Live behavior | Deterministic? | Replay technique | |---|---|---|---| | Image acquisition | incus pulls images:debian/12 | ✗ slow, remote | local Incus image imported from an omni-git blob; or a local simplestreams mirror | | DHCP | dnsmasq hands a 10.1.0.x | ✗ address/timing vary | static lease (--dhcp-host=<mac>,<ip>) → the test instance always gets the same IP | | Provisioning HTTP | cloud-init apt from mirrors | ✗ remote | apt-cacher proxy, or bake packages into the base image (record once) | | Fabric mechanics | OVS bridge, port attach, NAT | ✓ local already | no replay needed — just a fast harness (idempotent fabric_up) |

The key realization: only the top three layers need replay. The OVS/bridge/NAT logic — the part that's actually novel here — is already local and deterministic; it's just trapped behind slow internet-facing prerequisites. Replay removes the prerequisites so the interesting layer can be tested on its own.

Architecture: record once, gate against the recording forever

This is the same discipline congruency's shadow gate already uses (a pure, deterministic regression gate run on every mint). Network replay extends it to the fabric.

- fabric_record (run occasionally, online): execute the real path once and snapshot - the pulled Incus image → an omni-git artifact (content-addressed, dedup'd, versioned) - the DHCP lease → a static mac→ip map - any cloud-init/apt HTTP → a replay cassette keyed by a fixture id. - fabric_replay (the test inner loop, offline): fabric_up, then - incus image import <omni-git blob> instead of pulling - dnsmasq --dhcp-host=<fixed-mac>,<fixed-ip> for the test instance - (optional) an HTTP replay proxy pinned on the uplink fabric_launch --replay <fixture> completes in seconds, deterministically.

omni-git is already the right fixture store — it is a content-addressed, deduplicating store of tarball artifacts (the same store used for VM DNA). Recorded images and network cassettes are exactly that shape: large, binary, versioned, diffable by hash. Recording a fixture is a jazz bakery ingest.

Modular test surface

Because each layer is its own fixture, tests decompose instead of running the whole chain:

1. Fabric mechanics — no VM image, no internet: assert fabric_up is idempotent, the bridge exists, 10.1.0.1 answers, a veth attaches as an OVS port. (network namespace or the bare VM) 2. DHCP — static-lease fixture: launch a stub instance, assert it gets the pinned 10.1.0.x. 3. Instance lifecycle — cached-image fixture: fabric_launch --replay, assert lease + route-out in seconds. 4. Routing/NAT — assert the MASQUERADE rule + ip_forward; the "out" hop can be a stub.

A failure in (1) is a real fabric regression; it can no longer be masked by a mirror timeout in the image-pull step, because that step isn't in the test path anymore.

Trade-offs (and how to keep them honest)

- Replay only tests what you recorded. It won't catch an upstream change (a new debian image that breaks provisioning). Mitigation: keep a periodic live fabric_record job as a slow outer canary that re-records and surfaces a fixture diff — replay is the fast inner loop, live is the weekly truth check. - Recording must be reproducible. Pin the non-deterministic inputs: fixed instance names, fixed MACs, fixed image ids/aliases. Otherwise the "recording" drifts. - Instances carry state. Use ephemeral instances from a golden base snapshot (incus snapshots / vm_snapshot) so every replayed start is identical.

Verdict

Yes — network replay makes fabric testing predictable and modular, and it is the correct fix for the 25-minute failure. The time and flakiness were dominated by the live image pull and the provisioning chain, not by the fabric logic. Replay collapses the internet-facing layers (image, apt, DNS) into local content-addressed fixtures — which omni-git already stores — and pinning DHCP makes addressing assertable. The fabric-mechanics layer is already local and deterministic and only needs a fast harness. Net effect: a seconds-long, offline, per-layer test suite, with a slow live canary kept on the side to catch upstream drift.

Smallest first step

Add fabric_launch(name, image=..., replay=<fixture_id>) to the MCP: on replay, incus image import from the omni-git blob and set a static dnsmasq lease before starting the instance. Record one fixture (debian/12 today) with jazz bakery ingest. That single change removes the image pull — the biggest chunk of the 25 minutes — and is the seed the rest of the layers grow from.