19 Releases 150 Tags

RSS Feed

• v0.11.1-beta.1 e20ed48948

  Compare

  fix(cluster): a durable FSM must not be rebuilt from a snapshot on every restart

  LastSkywalker released this 2026-09-13 17:11:02 +03:00

  A machine that moved from bare metal into a VM came back on its own dashboard
  twice: once as itself, and once as a dead card frozen at the uptime it had
  before the move. Meanwhile two of three nodes kept a host row the leader had
  deleted, and no amount of time made them agree again.

  Both come from treating our FSM as the textbook in-memory one. It is not — it
  is the SQLite database that just survived the restart.

  hashicorp/raft restores the newest snapshot on EVERY start and then replays the
  log after it, because an in-memory FSM has to be rebuilt. SQLiteRestorer obeyed:
  it truncated and re-imported every managed table, rewinding the live database.
  Snapshots here are rare (90s cadence, but only past 4096 new entries, and a
  steady-state cluster appends ~zero), so that rewind reached back four days — to
  a state where the machine was still bare metal — and the replay then ran its
  whole history forward against a live local collector that had since become a VM
  with a different SMBIOS UUID. The identity guard correctly read "a different
  machine" and created a second row, with a synthetic MAC because the live row
  held the real one. Hence the ghost.

  Snapshots now carry the index they were taken at, and a restore that would move
  this node BACKWARDS is skipped — the stream is drained and the local database
  kept. One genuinely ahead (a fresh joiner, a node behind its peers' log, a wiped
  database) still restores in full. Replay is bounded the same way, by a node-local
  watermark in raft_apply_state: it lives in the same file it describes, so
  restoring a DB backup rolls it back too.

  The divergence is the second half. Apply advanced the index whatever the applier
  returned, treating every failure as deterministic. A cascade delete that blew
  its budget on a multi-GB SQLite is not: that node kept the row its peers had
  dropped, the index moved on, and because the surviving row still owned the
  external_id and the MAC, the correction could never land either. Failures are
  now classified — a deadline, a locked database, a lost connection are retried in
  place, and one that outlives its retries freezes the watermark so the entry is
  re-applied on the next restart instead of being lost. While frozen the FSM also
  holds off snapshotting for 15 minutes, since a snapshot would move raft's own
  lastApplied past that entry and cancel the retry; the hold then expires, loudly,
  so a node nobody restarts cannot grow its log without bound. Deterministic
  failures still advance, as they must — every replica fails them identically.

  So the delete cannot blow its budget in the first place: the host row goes first
  in one small statement, the history after it, and whatever the purge does not
  reach is collected by a retention sweep of metric rows whose host is gone.

  Two smaller things the same machine surfaced. The disambiguating name suffix ran
  a MAC tail through the external-id shortener, so "SkyNAS [00:03]" arrived as
  "SkyNAS [03]" — a number matching nothing on the machine. And the move left
  every dashboard showing the old IP: NODE_STATS_IPV4 is pinned in the stack .env,
  compose injects it into the container, and an environment variable outranks the
  app's own .env, so writing the runtime config fixed nothing and a recreate put
  the stale value back. A re-attach now also updates the live process and hands
  the address to the controller for the stack .env (part of composeHash, not
  appHash, so nothing restarts), and the collector stops believing a pin the
  machine demonstrably no longer holds.

  Verified by tests, including the three restore outcomes, the retry/freeze paths
  and the row-first delete; and by booting a fresh install to confirm the new
  table migrates. NOT verified on a real cluster restart — that needs the rollout.

  Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)