# mvirt 0.23.0

migrations: no
wire: unchanged
rev: 7a6f219eee6d7f8b3a13dd067fc07f2b27c90656

v0.23.0 — the raft transport carries what raft produces

Emergency release. A raft message larger than tonic's 4 MiB default cannot be
replicated, and the control-plane group then stops making progress permanently:
the entries are already in the leader's log, so every retry fails identically,
no commit lands, `ensure_linearizable` never completes, the leadership lease
goes stale, and leader-scoped serving refuses on every replica — the leader
included, pointing at itself. A leader change does not clear it, because
whichever replica wins the election holds the same un-replicable tail.

That took `api.mvirt.net` down fleetwide for four and a half hours on
2026-09-21 with all three daemons reporting `active (running)`, no pending
migrations, and no election in the journal (#352). Production cannot resume
without this release.

Control plane
- The raft gRPC listener and the pooled client both carry
  `MAX_RAFT_MESSAGE_BYTES` (256 MiB). On the listener it is applied before the
  authz interceptor wraps it — `InterceptedService` exposes no such knob, so
  wrapping first silently keeps the default. openraft batches up to 300 log
  entries per `AppendEntries` and one mvdb entry is a whole group-commit
  write-set, so ordinary traffic reached 4,545,161 bytes. A limit on this path
  is not a safety feature; the peers are mTLS-verified members (ADR-0059 P2).

Carried from 0.22.0 and the unrolled 0.21.0
- The raft transport no longer dials a fresh connection per leadership-confirm
  round, which had pinned 1005 dead sockets against a 1024 soft limit (#351).
- Both hand-written accept loops back off after a failed `accept`.
- The cplane unit sets `LimitNOFILE = 262144`; it previously set none.
- The host firewall no longer gates its served ports on the ingress NIC
  (#260, #346) — what silently expired four managed certificates on
  2026-09-19.
- A Task re-runs after its first failure; the log plane survives failover and
  restart; unplaceable replicated volume creates are refused up front; two
  stor-agent hydration races are closed.

Upgrading: this tag adds no migration, but a fleet coming from 0.20.0 still
crosses the two migrations 0.21.0 introduced (0138, 0139). Treat the roll as
migrating and use section C1a of docs/runbooks/release.md — host-by-host is
not a safe order on a multi-replica control plane.


## Upgrading

Point your flake's `mvirt` input at this release and redeploy the control-plane
host first, then each node in turn:

```
nix run .#colmena -- apply boot --reboot --on <control-plane host>
nix run .#colmena -- apply boot --reboot --on <node>   # one at a time
```

This release adds no migrations, so the control-plane host rolls back the same way
a node does: boot the previous NixOS generation.
