Daemon tick wedges on deleted-file row with NULL sha256 ("missing previous sha256") #23

New issue

Open

opened 2026-07-31 18:15:24 +00:00 by erik · 0 comments

erik commented

2026-07-31 18:15:24 +00:00

Owner

Symptom

A daemon root wedges on every tick with:

ERROR tick-end
  root=home device=server trigger=manifest-poll outcome=error
  error="upload: manifest: deleted file \"hi.mom\" missing previous sha256"

The tick fails uniformly and never recovers — the entire tick errors out each interval, so the root stops making progress.

Root cause

The error is the defense-in-depth invariant guard in deletedRowToTombstoneEntry (internal/upload/tombstone.go:68). A deleted row of type='file' is required to carry a sha256 so a valid tombstone can be built for it. The offending row had:

status = 'deleted'
type = 'file'
sha256 IS NULL
last_synced_manifest IS NOT NULL (inferred — see below)

That combination is supposed to be unreachable: the walker only classifies a row deleted when it already carries a committed baseline, and a baselined row always has a sha256. Rows that went new → deleted (never baselined) are swept by PurgeUnbaselinedDeleted.

The two sweepers miss each other. PurgeUnbaselinedDeleted only deletes deleted rows where last_synced_manifest IS NULL (internal/state/write.go:393). This row has a baseline pointer set and a NULL sha256, so the purge doesn't catch it and the tombstone builder rejects it — permanently. One malformed row wedges the whole root.

The file (hi.mom) existed only in the server's local .dropbear/state.sqlite. Verified against the R2 bucket: not in the laptop manifest, not in the server's published manifest (last written 2026-07-15), and no server tombstones exist at all. It is a ghost row that was never published anywhere — most likely produced by an earlier/buggy walk during development, and probably not reachable by current code.

Prescription (operator remediation)

Because the file exists nowhere and no tombstone is warranted, drop the poisoned row from the affected root's state DB. On the affected host:

systemctl --user stop dropbear.service
sqlite3 <root>/.dropbear/state.sqlite \
  "SELECT path,type,status,sha256,last_synced_manifest FROM files WHERE path='hi.mom';"
# confirm it's the file row with an empty sha256, then:
sqlite3 <root>/.dropbear/state.sqlite "DELETE FROM files WHERE path='hi.mom';"
systemctl --user start dropbear.service

Next tick goes green.

Suggested hardening (if not already impossible)

Suspected to be an impossible state under current code — needs confirmation. If it can still occur:

Widen PurgeUnbaselinedDeleted (or add a sibling sweep) to also drop deleted + type='file' + sha256 IS NULL rows — they can never produce a valid tombstone.
Make a tick skip/quarantine a single malformed row instead of failing the entire tick, so one bad row can't wedge a whole root.

If it is confirmed unreachable now, at minimum keep this as a known-recovery note for the invariant guard.

## Symptom A daemon root wedges on every tick with: ``` ERROR tick-end root=home device=server trigger=manifest-poll outcome=error error="upload: manifest: deleted file \"hi.mom\" missing previous sha256" ``` The tick fails uniformly and never recovers — the entire tick errors out each interval, so the root stops making progress. ## Root cause The error is the defense-in-depth invariant guard in `deletedRowToTombstoneEntry` (`internal/upload/tombstone.go:68`). A `deleted` row of `type='file'` is required to carry a `sha256` so a valid tombstone can be built for it. The offending row had: - `status = 'deleted'` - `type = 'file'` - `sha256 IS NULL` - `last_synced_manifest IS NOT NULL` (inferred — see below) That combination is supposed to be unreachable: the walker only classifies a row `deleted` when it already carries a committed baseline, and a baselined row always has a sha256. Rows that went `new` → `deleted` (never baselined) are swept by `PurgeUnbaselinedDeleted`. The two sweepers miss each other. `PurgeUnbaselinedDeleted` only deletes deleted rows where `last_synced_manifest IS NULL` (`internal/state/write.go:393`). This row has a baseline pointer set *and* a NULL sha256, so the purge doesn't catch it and the tombstone builder rejects it — permanently. One malformed row wedges the whole root. The file (`hi.mom`) existed **only** in the server's local `.dropbear/state.sqlite`. Verified against the R2 bucket: not in the laptop manifest, not in the server's published manifest (last written 2026-07-15), and no server tombstones exist at all. It is a ghost row that was never published anywhere — most likely produced by an earlier/buggy walk during development, and probably not reachable by current code. ## Prescription (operator remediation) Because the file exists nowhere and no tombstone is warranted, drop the poisoned row from the affected root's state DB. On the affected host: ```bash systemctl --user stop dropbear.service sqlite3 <root>/.dropbear/state.sqlite \ "SELECT path,type,status,sha256,last_synced_manifest FROM files WHERE path='hi.mom';" # confirm it's the file row with an empty sha256, then: sqlite3 <root>/.dropbear/state.sqlite "DELETE FROM files WHERE path='hi.mom';" systemctl --user start dropbear.service ``` Next tick goes green. ## Suggested hardening (if not already impossible) Suspected to be an impossible state under current code — needs confirmation. If it can still occur: 1. Widen `PurgeUnbaselinedDeleted` (or add a sibling sweep) to also drop `deleted` + `type='file'` + `sha256 IS NULL` rows — they can never produce a valid tombstone. 2. Make a tick skip/quarantine a single malformed row instead of failing the entire tick, so one bad row can't wedge a whole root. If it is confirmed unreachable now, at minimum keep this as a known-recovery note for the invariant guard.