# mvirt 0.20.0 migrations: no wire: unchanged rev: 565f3d3de3e027be7445c16bbdd5efd3a425dc05 release: 0.20.0 The 2026-09-12 production outage RCA produced nine findings. Seven were real and are fixed here; two were wrong and are closed as such. The one that matters most is a cross-tenant correctness fault in the dataplane: two tenant networks that share an overlay address were being merged into one forwarding entry, so a share of every fresh flow — and on the same host, every flow — was delivered into a stranger's network. It was live on our own fleet for every address in 10.0.0.0/24 and it was what stopped a customer app's image pull. Nothing in this release migrates the store and nothing moves the wire. The per-network dataplane tables are node-local, so nodes need no agreement with each other or with the control plane. == The dataplane forwards per source network (#307) FORWARDING_V4/V6, ROUTES_V4/V6 and NAT_EGRESS were all keyed on an address alone, with no network discriminator, while the route mesh had been carrying `Destination.network_id` since ADR-0024 — the loader was simply discarding it. Two networks on one private prefix therefore collapsed into one entry: 10.0.0.0/32 count=2 hops=[10.40.0.3/vni106, 10.40.0.2/vni176] <- two tenants, one ECMP group Consequences measured on our fleet, all three now closed: * ECMP hashed a share of every NEW upstream connection into another tenant's network. The app's Envoy held one pooled connection, so a settled window measured 100% and any restart, reschedule or scale event re-rolled the dice — which is why it looked intermittent for a day. * ROUTES_V4 is consulted BEFORE the FIB and is single-valued, so same-host delivery was not a 50/50 hash but a deterministic mis-delivery to whichever tap wrote the key last. * NAT_EGRESS collapsed on the source subnet, so a pod egressed through another network's NAT owner and appeared on the internet as that tenant's public address — and its reply came back through a FIB entry shared with a stranger, which is what stalled the image pull. The fix keys every one of those lookups on the SOURCE network: an LPM trie per family whose key is `scope ‖ address` with the scope in the fixed high 32 bits, so each network owns a disjoint subtree. Scope 0 is reserved for the north-south scope (the border default, NAT/VIP public returns, public LB frontends — all globally unique by construction); the FIB resolves scoped-first and global-second, ROUTES is scope-exact, and NAT_EGRESS is keyed on the scope alone. The source scope comes from one new per-ifindex map written by the vNIC's own level function, published before the filter is attached; the border gateway veth gets the global scope explicitly rather than by absence. The two reverse paths that legitimately route into a DIFFERENT network — an ADR-0053 endpoint reply and a NAT owner's reply to its guest — carry their own scope in the value the level pass already rewrites. The invariant is SEPARATION, not rejection: two networks announcing one address are two entries. Refusing the second announce would trade a deterministic cross-tenant leak for a non-deterministic availability loss, and since IPAM deliberately hands out a prefix's network address, that collision is the steady state. Verified on a real kernel, not only in unit tests: `nix build .#bpf-programs` plus the netns suites (netns_overlay 14/14, netns_nat 3/3, netns_lb 5/5, netns_health 2/2) load both objects and drive real traffic. tc-egress grew 2746 -> 2975 instructions, tc-ingress 6064 -> 6398, and the largest stack frame moved tc_egress 312 -> 408 of 512; every program still verifies. An old userspace against a new object is refused at load with a message naming the cause, rather than degrading to a dataplane that silently installs nothing. Left deliberately: IP_TO_SGS still unions two tenants' identical addresses into one SG membership set (#314, a security finding), VIP_SNAT_V4 and the conntrack maps are the same family (#315), and resolving a scope handle back to a network id in a dump wants a surface of its own (#316). == A failed container says why (#298) Every pod on the fleet failed with `state: Failed, exitCode: 0, message: null` while mvirt-one had written the exact reason to the pod-VM's serial console on the hypervisor — the one place no customer can reach. `Container.message` had zero writers, so the null was structural; `exitCode: 0` was reported for a container that never ran, which also meant an `OnFailure` container that failed to START was read as a clean exit. The reason now reaches the tenant through the container's own stderr (a workload source, therefore cold-spooled and tenant-visible), through `Container.message` as a level-converged value, and through one lifecycle event per distinct reason; a pod can no longer sit at `Creating` while a container is terminally failed with a reason in hand, and mvirt-one no longer logs "started successfully" for a pod where nothing started. == Migrations are applied on becoming leader, not at boot (#309) This one broke reconcile fleet-wide during the 0.19.0 roll. Pending migrations were applied only at boot and only if that replica was leader at that instant; mvirt-1 booted as a follower, skipped them, and won an election 3.5 minutes later without re-checking, so every node's slice compile failed with `unknown table: network_settings` for nine minutes. In a healthy three-voter group a single restarting replica is never leader at its own boot, so a rolling restart could never reach the code path — production needed a coordinated cold start of all three control planes. The schema level is now a level-reconciled property: one pass reads the applied set (a local read, correct on a follower too) and, while this replica is the leader, takes the ADR-0069 D4 pre-migration backup once per pending set and applies. It runs at boot before the seeds and then on its own interval forever, observing leadership inside the pass rather than hooking the transition. A dead state machine is fatal and loud; a lost election is not. `replica-status.json` and `GET /v1/version` now carry `schemaLevel` and `pendingMigrations`. Deliberately NOT done: the leader-only loops are not held back while a migration is pending. Under the real outage only the slice compile failed per node, while liveness probing, certificate renewal, the external-prefix announcer and the log pipeline all kept running — holding them would have degraded more than the bug did, and would have kept node tunnels attached to a control plane serving nothing. == A crash-looping backend no longer takes the host dataplane down (#299) mvirt-ebpf aborted six times in one day on mvirt-1: smoltcp 0.11's AF_PACKET device panics unconditionally on any non-WouldBlock receive error, a tap that disappears yields ENETDOWN, and `panic = "abort"` turned that into SIGABRT for the whole daemon — every VM's and pod's datapath on the host, not just the probe. It was self-sustaining: the restart disturbed pod networking, pods were rescheduled, more taps churned. The probe transport now owns its own device over the bound fd and latches a dead device instead of panicking, and the prober set gained the level repair that the abort-plus-systemd cycle had been providing by accident. `panic = "abort"` is left alone on purpose; why, and what has to happen first, is #324. == Health gates the Maglev pool again (#306, partially) ADR-0030 HC6 and the ADR-0039 liveness gate had both become dead code with respect to the pool: the `LB_TARGET` announce moved into the daemon and was built from every pushed membership with no health filter, while three comments and the ADR still described the old arrangement. Health now gates the announce in the daemon, through the single existing HC8 predicate, with one mandatory addition: a pool the filter would EMPTY is announced in full. That fail-open is load-bearing since 0.19.0 made an empty pool withdraw the VIP — without it a bug in health probing could delete a customer's VIP from every peer's FIB. Honest limitation: that panic is evaluated over one host's own memberships, which is all a host can see, so in the one-replica-per-node topology we ship it fails open every time and health withdraws nothing. #313 carries what a pool-wide version needs, including the fleet-wide flap it has to avoid. #306 stays open for it, and the liveness half is #311. == A wedged certificate and a dark prefix explain themselves (#301, #300) Both certificates of the affected app sat at `acmeStep: "fleet-converging"` with `lastError: null` and `nextAttempt: null` while `app domains list` reported "DNS ok / CERT requested" — the gate was correct and correctly level-triggered, it was simply silent. It now names the blocking instance, but only after a grace budget: that hold is the NORMAL transit of every renewal, so announcing it immediately would have meant a false alarm per domain every two days and a fleet-wide config re-push on every pass. The app-status ladder also stops telling an operator to fix DNS that already resolves. An external prefix whose provider refuses every switch now reports a named verdict — the datapath and the provider disagreeing is `dark`, not `switching` — on the API and in the console, with one activity-log entry per change of error rather than per retry. Automatic reversion is NOT in this release: the design had four blocking defects, including a grace that measured the wrong interval and a revert that could create the dark-at-both-ends state in mirror image. That is #319, with the findings; the egress dependency itself is #320. == Smaller, same path * An app no longer adopts another app's pods when one name is a prefix of the other (#310). With apps `mirc` and `mirc-migrate` in one project, `mirc` reported two replicas of its one, and deleting it would have deleted the other app's pod, cascading a stranger's vNIC and containers. Four call sites now go through one ownership predicate that requires a slot index, not just the prefix. * The gateway agent's transport errors carry their cause (#308). `reqwest::Error`'s Display says only "error sending request for url (…)", so connect-refused, DNS, TLS and a read timeout printed identically — four such lines fired during the RCA and told us nothing. The agent image is pinned at m9. * A membership test bound fixed ports and flaked as `AddrInUse` whenever any process held one as an ephemeral port (#329). == Retracted Two findings from that RCA were wrong and are closed as such: IPAM handing out a prefix's network address is intended (#302), and a refused route-mesh subscription IS re-evaluated — `reassert_all` re-sends every scope every 30 seconds, so the grant direction already self-heals (#305). The latter also retracts a claim the RCA made about #304. Only the revocation direction is real, as #312. == How this was reviewed Every fix was implemented from a brief that an adversarial reviewer had already corrected, and four of those reviews changed the plan materially rather than decorating it: the #309 plan would have held the leader-only loops back while a migration was pending (which degrades more than the bug did), the #301 plan would have announced a hold that is the NORMAL transit of every renewal (one false alarm per domain every two days, plus a fleet-wide config re-push per pass), the #300 plan's automatic revert could have created the dark-at-both-ends state in mirror image, and the #306 plan's liveness gate failed closed on first sight — which, composed with 0.19.0's empty-pool rule, withdraws a live VIP. Each of those was cut back to the part that holds and the rest filed with its evidence. The finished diff then went through a six-dimension review — level-triggered discipline, eBPF datapath correctness, cross-fix interaction, rollback and mixed-version behaviour, test quality, and comment truthfulness plus disclosure — with every finding handed to an independent skeptic prompted to refute it. Twelve of twenty-five were refuted. Nothing was release-blocking. Two serious ones are filed rather than fixed, with the reasoning recorded in the code: #333 (an on-fabric client of a public LB VIP loses its replies — verified unreachable on our fleet, and now counted instead of silent) and #334 (this release lands only the leadership-gain half of ADR-0073 D5, so the next MIGRATING minor must not be rolled host-by-host until the quorum-of-upgraded rail exists). Every new test was proven red by reverting its production change and watching it fail. Three of those proofs found a problem with the test rather than the code and fixed the test. == Verification 1629 tests across mvdb/cplane/ebpf/node/one/vmm/cli; the images.toml consistency gate; clippy -D warnings; the console tsc + build; mvirt-gw-agent's own suite. The eBPF objects were built through nix and LOADED on a real kernel under root — netns_overlay 14/14, netns_nat 3/3, netns_lb 5/5, netns_health 2/2 — because a dataplane change that only passes unit tests has not been tested. The release machinery (dist/, the workflow, nix/, flake.lock) is byte-identical to what published 0.19.0, so this tag introduces no new build dependency. ## Upgrading Point your flake's `mvirt` input at this release and redeploy the control-plane host first, then each node in turn: ``` nix run .#colmena -- apply boot --reboot --on nix run .#colmena -- apply boot --reboot --on # one at a time ``` This release adds no migrations, so the control-plane host rolls back the same way a node does: boot the previous NixOS generation.