The backup chain

The question

The lab keeps two kinds of backup. A nightly pull of eight hosts onto one machine, and a separate chain for the knowledge graph and its vector store, the two things that would hurt most: a local snapshot on the machine that runs the graph, an offsite copy on the backup machine, and since 21 August a third copy on a different physical box with its own disk. A health exporter watches all of it and reports green when a copy is fresh.

The question nobody had asked, in plain words, until the end of August: fresh according to what? A backup nobody has restored from is a belief, not a backup.

What was built

Three snapshot scripts with hard-linked generations, so that each night stores only what changed. A rotation that keeps seven generations. A health exporter that measures the age of the “current” pointer for fifteen datasets. And since 31 August, a messenger that raises an alarm on the lab’s internal bus when a run reports errors, throttled to one message every six hours, and a restore procedure that has been executed once.

What was measured

• 3 May: the rotation deleted the snapshot it had just written, because it sorted by modification time and hard-linked copies inherit the source’s timestamp. A seven-day gap, 26 April to 1 May, found afterwards.
• 7 June: a USB cable came loose for eighteen and a half hours. The backup machine kept running without its disk, and one script wrote its nightly copy onto the system disk instead. Hardened.
• 31 July: the graph had had no valid backup for 61 days. A database rename on 31 May had left the backup script copying old names into nothing, with errors silenced, reporting OK every night: 16 kilobytes instead of 1.2 gigabytes. Found, fixed, and a full restore rehearsed on a separate machine, service started, tools answering.
• 9 August: a dead source host produced six empty “generations” of four kilobytes each; the rotation counted them as real. The offsite vector copy was three days old, the code copy four, with no alarm.
• 23 August: the same rotation bug as in May, in the offsite script this time. The vector copy had been dead offsite for fourteen nights with zero errors reported, because the health exporter measured the age of a symbolic link that the script reset every night whether or not it had written anything.
• 27 to 31 August, the four days: the backup script treated exit code 24 from the copy tool as failure. For a live vector database that code is normal, it means files vanished during the run because the service compacts while it works. From 27 August the branch that advances the “current” anchor never ran again. The local tier lost its hard-link base and wrote a full 16 gigabyte copy every night instead of a delta, until the disk was nearly full. The offsite tier stored the same frozen 27 August snapshot three nights running under three new dates. The third tier did the same, and was missed in the first repair; the agent reported two tiers healthy when there were three.
• 30 August: a night watch reported a service failure on the graph machine, “no space left on device”, the direct consequence. Two sessions passed the finding on as an open item for two days without measuring it once; it had resolved itself on the 31st.
• 31 August: a sweep of 63 agents over the lab’s timers, hooks and reporting paths listed the backups as the most expensive unchecked item of all: a restore had never been attempted.
• 31 August: the fix, one line of logic and one of policy. The anchor is now the newest snapshot that actually exists on disk, not a link, and exit code 24 counts as success. Proven on a constructed failure with a negative control: a missing source still fails with code 23. The first version of the messenger would have reported into the void, to two mailboxes that did not exist; the bus rejected it, and the fix was a broadcast, not a forced override.
• 1 September, the restore: the graph, 1.6 gigabytes, pulled back from the offsite copy in 13 seconds and opened, 211 tables, 26,319 memories against 26,370 live, the difference being what was written since the snapshot. Two boot pins compared character for character, identical. The vectors, 16.4 gigabytes in 156 seconds, 374,332 rows against 374,261 live, dimension 2048, first values and a text sample identical. The third tier opened and checked for the first time ever; an SSH hook had been silently blocking root access to it, unnoticed until then.

What fell

• Age as a proxy for freshness. Three separate incidents, May, August and again in August, came from a pointer or a timestamp that looked new while the data behind it was not.
• Silence as success. The 61-day gap had errors redirected to nowhere; the four-day freeze had a normal exit code treated as fatal. Both reported OK.
• File checksums as the comparison between two tiers. On a live database two copies of the same size differ; content has to be compared, not files.
• “Both tiers healthy.” There were three.
• Passing a finding along instead of measuring it. Two days of an open item that had already closed.

What stands

A chain that has been restored from once, with the numbers on record, and a definition of “tested” that means opened and compared, not listed. A messenger on the bus. And a list of what is still belief: four of the six datasets in the chain have never been pulled back, the restore did not go into a running service, because stopping a service is the founder’s call, and the health exporter still measures the age of a link. The next failure of the same class will show up in a restore, not on a dashboard, unless that changes.