More efficient manifests #10

Open
opened 2026-05-29 16:44:14 +00:00 by erik · 3 comments
Owner

At only a few entries, the manifest for each device is ~1KB. At only one thousand files, those manifests are ~1MB. dropbear should not update its manifest if there are no changes. Currently, it does that on every sync regardless.

While still early, we should consider alternate manifest file formats and/or compression.

Treat this as an investigation for now. No spec yet since we don't know which direction to go yet.

At only a few entries, the manifest for each device is ~1KB. At only one thousand files, those manifests are ~1MB. dropbear should not update its manifest if there are no changes. Currently, it does that on every sync regardless. While still early, we should consider alternate manifest file formats and/or compression. Treat this as an investigation for now. No spec yet since we don't know which direction to go yet.
agent-pm removed their assignment 2026-06-04 23:38:42 +00:00
Member

Brief — Developer

Phase: simple. Run fjx dev prompt simple for the cycle protocol.

<!-- pm:brief:dev --> # Brief — Developer **Phase:** simple. Run `fjx dev prompt simple` for the cycle protocol.
Member

Ledger for: agent-pm

Status: watching @ 2026-06-27T00:06:23.998Z
Briefs:

  • dev @ 2026-06-04T23:38:42Z

Activity

(none)


(PM manages this ledger; agent section unused)

<!-- agent-pm:ledger --> ## Ledger for: agent-pm Status: watching @ 2026-06-27T00:06:23.998Z Briefs: - [dev](#issuecomment-2585) @ 2026-06-04T23:38:42Z ### Activity _(none)_ --- _(PM manages this ledger; agent section unused)_
Member

Ledger for: agent-dev

Status: blocked @ 2026-06-27T00:14:38.192Z


Finding: not a simple change — recommend re-route to investigate

I read the issue as having two parts and investigated both against the current code.

Part 1 — "don't update the manifest if there are no changes" — already implemented

internal/upload/run.go already short-circuits the manifest republish when the tick has no work. runState.isNoOp() (run.go:147) returns true when there are zero uploads, downloads, already-present paths, deleted rows, applied tombstones, and reconciles. runUploadBlobs returns the errNoUploadWork sentinel in that case (run.go:309-311), and Run returns early on that sentinel (run.go:184-190) before runBuildAndPutManifest (run.go:194) ever runs. So in true steady state the manifest is not re-PUT.

git log -L :isNoOp:internal/upload/run.go shows this guard has existed since the sync subcommand was first implemented (0dbd501), so the issue's premise ("it does that on every sync regardless") does not match the current code for the no-op case.

Part 2 — the genuinely open design questions (the author's stated "investigation")

The issue body explicitly says: "Treat this as an investigation for now. No spec yet since we don't know which direction to go yet." The remaining asks are real but design-level, not a one-line fix:

  1. Partial-change ticks that don't actually change manifest content. isNoOp() is false (so the manifest is rebuilt and PUT) in cases where the manifest's content can still be byte-identical to what's stored — e.g. an applied-tombstone watermark advance, a reconcile, or an already-present download. Suppressing the PUT there requires detecting "content unchanged." That is non-trivial because the canonical manifest embeds CreatedAt: now (internal/upload/manifest.go:50, included in the sha256 over canonical bytes), so naive hash equality never holds. Options are (a) fetch the prior manifest and compare entries+applied-tombstones — but that downloads the ~1MB object the issue wants to stop moving, which is counterproductive, or (b) persist a content-hash (excluding CreatedAt) in local state and compare. (b) is a new abstraction / schema decision.
  2. Alternate manifest formats and/or compression — open-ended, no chosen direction.

This is not a simple one-line fix and should be re-routed to investigate (or propose) per the simple-phase contract. No code was changed this cycle.

Blocker: no handoff path for a no-code re-route

I cannot complete the prescribed handoff. The simple-phase escape hatch ("if the change isn't actually simple… stop and report it in the ledger rather than expanding scope — PM will re-route to propose or investigate") produces no code and therefore no PR, but the handoff command requires one.

Four checks:

  • fjx dev done --help: "Mark an issue's work ready for review: transition the issue to agent/review, assign the issue to agent-pm for routing, and assign the linked PR to $FJX_OWNER." No flag to hand off without a PR. fjx dev --help confirms the only other transitions are release (crash-recovery: adds agent/blocked, reassigns to $FJX_OWNER, appends a "crashed" entry — semantically wrong for a clean finding) and ledger.
  • Verbatim error from fjx dev done 10: No open PR links to issue #10 (looked for Closes #10/Fixes/Resolves/Refsin open PR bodies). Open the PR with one of those keywords in the body before runningfjx dev done.rg -i 'no open PR|without a PR|no-code|re-route' wiki/ returns nothing; no prior occurrence in the ledger.
  • Phase prompt re-read: re-checked the simple prompt step — it directs me to "stop and report it in the ledger rather than expanding scope — PM will re-route," i.e. report-only, no PR. That conflicts with fjx dev done's PR gate.
  • Classification: fjx gap (no-PR re-route handoff path missing for the documented simple-phase escape hatch).

Smallest unblock: either a fjx dev done mode that hands a no-code finding back to agent-pm without an open PR, or PM re-routes #10 to investigate/propose directly from this ledger. I have not changed labels/assignment beyond applying agent/blocked, to avoid routing around fjx.

<!-- agent-dev:ledger --> ## Ledger for: agent-dev Status: blocked @ 2026-06-27T00:14:38.192Z --- ## Finding: not a simple change — recommend re-route to `investigate` I read the issue as having two parts and investigated both against the current code. ### Part 1 — "don't update the manifest if there are no changes" — already implemented `internal/upload/run.go` already short-circuits the manifest republish when the tick has no work. `runState.isNoOp()` (run.go:147) returns true when there are zero uploads, downloads, already-present paths, deleted rows, applied tombstones, and reconciles. `runUploadBlobs` returns the `errNoUploadWork` sentinel in that case (run.go:309-311), and `Run` returns early on that sentinel (run.go:184-190) **before** `runBuildAndPutManifest` (run.go:194) ever runs. So in true steady state the manifest is not re-PUT. `git log -L :isNoOp:internal/upload/run.go` shows this guard has existed since the sync subcommand was first implemented (0dbd501), so the issue's premise ("it does that on every sync regardless") does not match the current code for the no-op case. ### Part 2 — the genuinely open design questions (the author's stated "investigation") The issue body explicitly says: *"Treat this as an investigation for now. No spec yet since we don't know which direction to go yet."* The remaining asks are real but design-level, not a one-line fix: 1. **Partial-change ticks that don't actually change manifest content.** `isNoOp()` is false (so the manifest is rebuilt and PUT) in cases where the manifest's *content* can still be byte-identical to what's stored — e.g. an applied-tombstone watermark advance, a reconcile, or an already-present download. Suppressing the PUT there requires detecting "content unchanged." That is non-trivial because the canonical manifest embeds `CreatedAt: now` (`internal/upload/manifest.go:50`, included in the sha256 over canonical bytes), so naive hash equality never holds. Options are (a) fetch the prior manifest and compare entries+applied-tombstones — but that downloads the ~1MB object the issue wants to stop moving, which is counterproductive, or (b) persist a content-hash (excluding `CreatedAt`) in local state and compare. (b) is a new abstraction / schema decision. 2. **Alternate manifest formats and/or compression** — open-ended, no chosen direction. This is not a simple one-line fix and should be re-routed to `investigate` (or `propose`) per the simple-phase contract. No code was changed this cycle. ## Blocker: no handoff path for a no-code re-route I cannot complete the prescribed handoff. The simple-phase escape hatch ("if the change isn't actually simple… stop and report it in the ledger rather than expanding scope — PM will re-route to `propose` or `investigate`") produces no code and therefore no PR, but the handoff command requires one. **Four checks:** - **`fjx dev done --help`**: "Mark an issue's work ready for review: transition the issue to agent/review, assign the issue to agent-pm for routing, and assign the linked PR to $FJX_OWNER." No flag to hand off without a PR. `fjx dev --help` confirms the only other transitions are `release` (crash-recovery: adds agent/blocked, reassigns to $FJX_OWNER, appends a "crashed" entry — semantically wrong for a clean finding) and `ledger`. - **Verbatim error** from `fjx dev done 10`: `No open PR links to issue #10 (looked for `Closes #10` / `Fixes` / `Resolves` / `Refs` in open PR bodies). Open the PR with one of those keywords in the body before running `fjx dev done`.` — `rg -i 'no open PR|without a PR|no-code|re-route' wiki/` returns nothing; no prior occurrence in the ledger. - **Phase prompt re-read**: re-checked the `simple` prompt step — it directs me to "stop and report it in the ledger rather than expanding scope — PM will re-route," i.e. report-only, no PR. That conflicts with `fjx dev done`'s PR gate. - **Classification**: fjx gap (no-PR re-route handoff path missing for the documented simple-phase escape hatch). **Smallest unblock:** either a `fjx dev done` mode that hands a no-code finding back to agent-pm without an open PR, or PM re-routes #10 to `investigate`/`propose` directly from this ledger. I have not changed labels/assignment beyond applying `agent/blocked`, to avoid routing around fjx.
Sign in to join this conversation.
No description provided.