v0.2 slice 1: remote head comparison + download path #1

Merged

erik merged 1 commit from v02-slice1-remote-head-download into main

2026-05-26 14:51:33 +00:00

erik commented

2026-05-26 14:45:15 +00:00

Owner

Summary

v0.2 slice 1 of 5: remote head comparison + download path. dropbear sync is no longer upload-only — it enumerates peer heads, fetches and verifies peer manifests, downloads missing blobs, and warns on conflicts. Conflict file generation lands in slice 3.

What's in

New packages. internal/blobio (atomic per-blob download + restore-tmp orphan cleanup, extracted from internal/restore). internal/download (peer enumeration, fetch+verify, decider, fail-fast worker pool).
internal/objectstore extended. List(ctx, prefix) iter.Seq2[ObjectInfo, error]; ObjectInfo.Key now populated by every method.
internal/syncplan extended. New download / conflict-skip actions, PeerDeviceID field on every action row, ↓/↑ summary line. JSON envelope is additive — schema_version stays at 1.
Sync engine extended. Download phase runs before upload; single end-of-run manifest reflects merged state; new exit codes 11 (hash mismatch) and 12 (peer-list failure); peers_skipped / conflicts_skipped counters in JSON. Real-time progress emits inline via Options.Progress instead of post-hoc CLI rendering — JSON mode stays silent until the final envelope.
Spec. Full OpenSpec proposal + design + delta specs in openspec/changes/v02-remote-head-download/.
Tests. 7 integration tests in internal/upload/integration_test.go covering two-roots convergence, cross-pull, conflict-skip, peer manifest tampering, peer-skip, empty peer set, and restore-tmp cleanup at sync entry.

What's out (intentional)

Tombstone propagation, conflict-file generation, and sync modes are all deferred to later slices. Conflicting paths print a stderr warning every run until slice 3 lands the resolution mechanism — this is by design.

Test plan

just validate green (format, vet, lint, race-tests at ~72% coverage, gitleaks clean, govulncheck 0 reachable)
Manual smoke against local MinIO: two roots with distinct device_ids cross-sync; conflict warning fires on divergence; bucket reflects expected manifest/head writes
CI on this PR green

Follow-up

Slice 2 (tombstones + safer deletes) is the next slice. Wiki's "Where We Are" updated.

## Summary v0.2 slice 1 of 5: remote head comparison + download path. `dropbear sync` is no longer upload-only — it enumerates peer heads, fetches and verifies peer manifests, downloads missing blobs, and warns on conflicts. Conflict file generation lands in slice 3. ## What's in - **New packages.** `internal/blobio` (atomic per-blob download + restore-tmp orphan cleanup, extracted from `internal/restore`). `internal/download` (peer enumeration, fetch+verify, decider, fail-fast worker pool). - **`internal/objectstore` extended.** `List(ctx, prefix) iter.Seq2[ObjectInfo, error]`; `ObjectInfo.Key` now populated by every method. - **`internal/syncplan` extended.** New `download` / `conflict-skip` actions, `PeerDeviceID` field on every action row, ↓/↑ summary line. JSON envelope is additive — `schema_version` stays at 1. - **Sync engine extended.** Download phase runs before upload; single end-of-run manifest reflects merged state; new exit codes 11 (hash mismatch) and 12 (peer-list failure); `peers_skipped` / `conflicts_skipped` counters in JSON. Real-time progress emits inline via `Options.Progress` instead of post-hoc CLI rendering — JSON mode stays silent until the final envelope. - **Spec.** Full OpenSpec proposal + design + delta specs in `openspec/changes/v02-remote-head-download/`. - **Tests.** 7 integration tests in `internal/upload/integration_test.go` covering two-roots convergence, cross-pull, conflict-skip, peer manifest tampering, peer-skip, empty peer set, and restore-tmp cleanup at sync entry. ## What's out (intentional) Tombstone propagation, conflict-file generation, and sync modes are all deferred to later slices. Conflicting paths print a stderr warning every run until slice 3 lands the resolution mechanism — this is by design. ## Test plan - [x] `just validate` green (format, vet, lint, race-tests at ~72% coverage, gitleaks clean, govulncheck 0 reachable) - [x] Manual smoke against local MinIO: two roots with distinct device_ids cross-sync; conflict warning fires on divergence; bucket reflects expected manifest/head writes - [x] CI on this PR green ## Follow-up Slice 2 (tombstones + safer deletes) is the next slice. Wiki's "Where We Are" updated.

erik added 1 commit

2026-05-26 14:45:15 +00:00

v0.2 slice 1: remote head comparison + download path

Validate / validate (push) Successful in 37s

Details

Validate / validate (pull_request) Successful in 37s

Details

315ca07715

Add bidirectional sync foundation: `dropbear sync` now enumerates peer heads under dropbear/roots/<root_id>/heads/, fetches and verifies peer manifests, decides per-entry actions (download / already-present / conflict-skip), and downloads missing blobs with the same atomic temp-then-rename guarantee restore uses. Conflicts are detected and warned to stderr — full conflict-file generation lands in slice 3.

New packages: internal/blobio (extracted from restore; shared per-blob atomic download primitive and restore-tmp orphan cleanup) and internal/download (peer-head enumeration, per-peer fetch+verify, entry merging with <multiple> sentinel for peer↔peer disagreement, per-entry decider, fail-fast bounded worker pool).

Extended internal/objectstore with List(ctx, prefix) iter.Seq2[ObjectInfo, error]; ObjectInfo.Key is now populated by every method. internal/syncplan gained `download` and `conflict-skip` actions, a PeerDeviceID field on every action, an additive JSON envelope still at schema_version 1, and a ↓/↑ summary line. internal/upload (the sync engine) now runs phases in download → upload order, writes one end-of-run manifest reflecting merged on-disk state, introduces exit codes 11 (hash mismatch) and 12 (peer-list failure), and reports peers_skipped / conflicts_skipped counters in JSON output. Progress reporting moved into the engines via Options.Progress (io.Writer), replacing post-hoc CLI rendering; JSON mode stays silent until the final envelope.

Spec lives in openspec/changes/v02-remote-head-download/{proposal,design,specs/,tasks}.md. Integration tests in internal/upload/integration_test.go cover two-roots convergence, cross-pull, conflict-skip, peer manifest tampering, peer-skip, empty peer set, and restore-tmp cleanup at sync entry.

\`just validate\` green.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

erik merged commit f496e26fad into main

2026-05-26 14:51:33 +00:00

erik deleted branch v02-slice1-remote-head-download

2026-05-26 14:51:33 +00:00

erik referenced this pull request from a commit

2026-05-26 14:51:34 +00:00

Merge pull request 'v0.2 slice 1: remote head comparison + download path' (#1) from v02-slice1-remote-head-download into main