19 Releases 150 Tags

RSS Feed

• v0.11.4-beta.0 0d84ea2671

  Compare

  fix(cluster): a restarting node announced its past as the present, reverting gateway routes

  LastSkywalker released this 2026-09-13 20:12:55 +03:00 | 55 commits to main since this release

  Route settings edited minutes earlier came back with their old values after an
  update — on every node at once, and repeatedly, so the same five routes were
  re-entered three times in an hour.

  The backfills are the culprit. Each one reads this node's LOCAL tables and
  republishes every row into the log, so that state configured while the node was
  standalone reaches the cluster. That is right in itself, but raft restores the
  newest snapshot into those very tables on every start and only re-applies the
  log tail afterwards — so during the replay they hold snapshot-age values. The
  gateway backfill runs 5 s after activation. On a multi-GB node the replay is
  nowhere near done by then, so it read rows from days ago and submitted them as
  new commands; the leader applied them, and every peer's current routes were
  overwritten with them. Nothing about it was leader-gated, so any member could do
  this to the whole cluster, and the node that had been offline longest did the
  most damage.

  All backfills now require the node to have applied everything it knows to be
  committed. The tick-driven ones (connectors, gateway routes and blocks) return
  and retry on the next 20 s pass; the user/host backfill runs once per activation,
  so it waits up to ten minutes and then skips with a warning rather than
  publishing from behind — a node still catching up after that is precisely the one
  whose rows must not be broadcast.

  This is older than the durable-FSM work, but that work is why it surfaced now:
  the snapshots on disk predate the applied-index field, so they cannot be skipped
  and every node in this cluster did a full rewind on the update restart. Once each
  node has written one snapshot in the new format the rewind stops happening at
  all — the gate is what makes the window harmless in the meantime, and for any
  node that legitimately restores later.

  Verified on the three real nodes: all three restarted at 16:58, the leader's
  backfill fired at 16:58:02 and the routes reverted; the tables now agree again
  across leader, LXC and NAS with the operator's values.

  Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)