tfks/dropbear
tfks/dropbear
4
0
Fork 0
Code Issues 5 Pull requests Projects Releases 3 Packages Wiki Activity Actions

Add microsecond precision to created_at and entry mtimes #9

New issue
Closed
opened 2026-05-27 22:52:35 +00:00 by erik · 6 comments
erik commented 2026-05-27 22:52:35 +00:00
Owner
Copy link

Manifest created_at, manifest object basenames, and per-entry mtime are all UTC truncated to one second. Two
devices publishing in the same wall-clock second produce manifests with identical created_at, so the
staleness guard in download.Decide cannot determine ordering when shas differ. Lab repro: d1 and d2 both
published hello.mom (different content) at 2026-05-27T21:54:06Z; the guard fell through to ActionDownload and
d1 was about to overwrite local content with d2's stale blob. The just-shipped guard fix turns these
collisions into .conflict-* files needing manual resolution — fine as a safety net, but at 1s daemon
intervals collisions are routine.

Proposal

Move all timestamps used for ordering to microsecond precision. Apply consistently to:

  1. created_at on Manifest and Tombstone. Producer formats with time.RFC3339Nano truncated to microsecond.
    Validation in internal/manifest/validate.go switches from regex (TimestampPattern) to
    time.Parse(time.RFC3339Nano, s) + UTC check — simpler, correct, no fractional-digit count to encode in a
    pattern. Drop TimestampPattern and the format constraint in the JSON Schema files under
    internal/manifest/schema/ (JSON Schema's built-in format: date-time already accepts fractional seconds).
  2. Manifest object basenames (manifestBasenameRegex at validate.go:28). Format becomes
    20260527T215406.123456Z_d1.json. Microsecond field is fixed-width, so lexical sort still matches
    chronological. Update the writer in internal/upload/manifest.go:40 and any reader that parses the basename.
  3. Per-entry mtime. Walker truncation in internal/fswalk/classify.go:19 drops Truncate(time.Second) →
    Truncate(time.Microsecond). APFS has nanosecond resolution; ext4 has nanosecond; macOS HFS+ is one-second but
    you're not running on HFS+. Cross-filesystem mtime shifts are an existing concern unchanged by this.
  4. Staleness guard. No code change needed — the existing same-CreatedAt → ConflictSkip fix stays as a final
    safety net for the now-rare microsecond collision.

Test plan

  • TestDecideCleanShaDifferStalenessGuard (already exists): add microsecond-differing CreatedAt cases.
  • Encode/decode round-trip a manifest with microsecond created_at.
  • Walker test: confirm mtime round-trips at microsecond precision; coarser-precision filesystems still
    classify clean when no other field changed.
  • Integration test: two daemons ticking on a 1s interval, both modifying the same file in the same second,
    ordered determinately by microsecond — no spurious conflict file.

Out of scope

  • Sub-second daemon tick scheduling.
  • Conflict-skip materialization path.
Manifest created_at, manifest object basenames, and per-entry mtime are all UTC truncated to one second. Two devices publishing in the same wall-clock second produce manifests with identical created_at, so the staleness guard in download.Decide cannot determine ordering when shas differ. Lab repro: d1 and d2 both published hello.mom (different content) at 2026-05-27T21:54:06Z; the guard fell through to ActionDownload and d1 was about to overwrite local content with d2's stale blob. The just-shipped guard fix turns these collisions into .conflict-* files needing manual resolution — fine as a safety net, but at 1s daemon intervals collisions are routine. ## Proposal Move all timestamps used for ordering to microsecond precision. Apply consistently to: 1. created_at on Manifest and Tombstone. Producer formats with time.RFC3339Nano truncated to microsecond. Validation in internal/manifest/validate.go switches from regex (TimestampPattern) to time.Parse(time.RFC3339Nano, s) + UTC check — simpler, correct, no fractional-digit count to encode in a pattern. Drop TimestampPattern and the format constraint in the JSON Schema files under internal/manifest/schema/ (JSON Schema's built-in format: date-time already accepts fractional seconds). 2. Manifest object basenames (manifestBasenameRegex at validate.go:28). Format becomes 20260527T215406.123456Z_d1.json. Microsecond field is fixed-width, so lexical sort still matches chronological. Update the writer in internal/upload/manifest.go:40 and any reader that parses the basename. 3. Per-entry mtime. Walker truncation in internal/fswalk/classify.go:19 drops Truncate(time.Second) → Truncate(time.Microsecond). APFS has nanosecond resolution; ext4 has nanosecond; macOS HFS+ is one-second but you're not running on HFS+. Cross-filesystem mtime shifts are an existing concern unchanged by this. 4. Staleness guard. No code change needed — the existing same-CreatedAt → ConflictSkip fix stays as a final safety net for the now-rare microsecond collision. ## Test plan - TestDecideCleanShaDifferStalenessGuard (already exists): add microsecond-differing CreatedAt cases. - Encode/decode round-trip a manifest with microsecond created_at. - Walker test: confirm mtime round-trips at microsecond precision; coarser-precision filesystems still classify clean when no other field changed. - Integration test: two daemons ticking on a 1s interval, both modifying the same file in the same second, ordered determinately by microsecond — no spurious conflict file. ## Out of scope - Sub-second daemon tick scheduling. - Conflict-skip materialization path.
agent-pm commented 2026-06-02 16:01:44 +00:00
Member
Copy link

Brief - Developer

Phase: simple. Run fjx dev prompt simple for the cycle protocol.

Context

  • Owner says the fjx issues that blocked the prior attempt are fixed; try the dev cycle again.
<!-- pm:brief:dev --> # Brief - Developer **Phase:** simple. Run `fjx dev prompt simple` for the cycle protocol. ## Context - Owner says the fjx issues that blocked the prior attempt are fixed; try the dev cycle again.
agent-pm commented 2026-06-02 16:01:45 +00:00
Member
Copy link

Ledger for: agent-pm

Status: closed @ 2026-06-25T04:24:06.845Z
Briefs:

Activity

  1. i9/closed : applied

(PM manages this ledger; agent section unused)

<!-- agent-pm:ledger --> ## Ledger for: agent-pm Status: closed @ 2026-06-25T04:24:06.845Z Briefs: - [dev](#issuecomment-2285) - [qa](#issuecomment-2921) ### Activity 1. i9/closed : applied --- _(PM manages this ledger; agent section unused)_
agent-dev commented 2026-06-04 23:40:08 +00:00
Member
Copy link

Ledger for: agent-dev

Status: review @ 2026-06-25T01:51:48.610Z
Refs: #16

Action summary

  • Addressed QA finding that typed manifest/tombstone timestamps were accepted with non-canonical wire precision because ManifestTime.UnmarshalJSON parsed RFC3339Nano and truncated before validation.
  • Changed ManifestTime.UnmarshalJSON to require the original JSON string to exactly match the canonical UTC microsecond layout before storing it.
  • Added regression coverage for manifest created_at, tombstone created_at, and tombstone entry deleted_at with nanosecond precision.

Tasks

  • Reproduce QA finding with a committed regression test for manifest created_at, tombstone created_at, and tombstone entry deleted_at.
    • Evidence: 53aadb4 adds TestDecodeRejectsNonCanonicalTypedTimestampPrecision.
  • Fix typed timestamp validation so non-canonical wire values are rejected instead of silently truncated.
    • Evidence: 53aadb4 makes ManifestTime.UnmarshalJSON reject non-canonical precision.
  • Run final validation once after committing code changes.
    • Evidence: just validate passed after commit 53aadb4.
  • Update ledger to review status and hand off with fjx dev done 9.

Validation

  • GOCACHE=$PWD/.cache/go-build GOMODCACHE=$PWD/.cache/gomod go vet ./... passed.
  • GOCACHE=$PWD/.cache/go-build GOMODCACHE=$PWD/.cache/gomod go test ./internal/manifest -count=1 passed.
  • just validate passed: vet, lint (0 issues.), race tests with coverage, gitleaks (no leaks found), govulncheck (0 vulnerabilities affecting code), and go.mod/go.sum drift check.

Notes

  • Initial plain go vet ./... failed because the default module cache path /usr/local/gopath/pkg is not writable in this environment; reran with workspace-local Go caches successfully.
  • One parallel focused test attempt collided with the simultaneous local module-cache download and failed on missing temporary zip files; the same focused test passed when rerun sequentially.
<!-- agent-dev:ledger --> ## Ledger for: agent-dev Status: review @ 2026-06-25T01:51:48.610Z Refs: #16 --- ## Action summary - Addressed QA finding that typed manifest/tombstone timestamps were accepted with non-canonical wire precision because `ManifestTime.UnmarshalJSON` parsed RFC3339Nano and truncated before validation. - Changed `ManifestTime.UnmarshalJSON` to require the original JSON string to exactly match the canonical UTC microsecond layout before storing it. - Added regression coverage for manifest `created_at`, tombstone `created_at`, and tombstone entry `deleted_at` with nanosecond precision. ## Tasks - [x] Reproduce QA finding with a committed regression test for manifest `created_at`, tombstone `created_at`, and tombstone entry `deleted_at`. - Evidence: `53aadb4` adds `TestDecodeRejectsNonCanonicalTypedTimestampPrecision`. - [x] Fix typed timestamp validation so non-canonical wire values are rejected instead of silently truncated. - Evidence: `53aadb4` makes `ManifestTime.UnmarshalJSON` reject non-canonical precision. - [x] Run final validation once after committing code changes. - Evidence: `just validate` passed after commit `53aadb4`. - [x] Update ledger to review status and hand off with `fjx dev done 9`. ## Validation - `GOCACHE=$PWD/.cache/go-build GOMODCACHE=$PWD/.cache/gomod go vet ./...` passed. - `GOCACHE=$PWD/.cache/go-build GOMODCACHE=$PWD/.cache/gomod go test ./internal/manifest -count=1` passed. - `just validate` passed: vet, lint (`0 issues.`), race tests with coverage, gitleaks (`no leaks found`), govulncheck (`0 vulnerabilities` affecting code), and go.mod/go.sum drift check. ## Notes - Initial plain `go vet ./...` failed because the default module cache path `/usr/local/gopath/pkg` is not writable in this environment; reran with workspace-local Go caches successfully. - One parallel focused test attempt collided with the simultaneous local module-cache download and failed on missing temporary zip files; the same focused test passed when rerun sequentially.
agent-dev removed their assignment 2026-06-04 23:40:22 +00:00
erik commented 2026-06-24 23:00:08 +00:00
Author
Owner
Copy link

@agent-pm, fjx issues are fixed. Have @agent-dev try again.

@agent-pm, fjx issues are fixed. Have @agent-dev try again.
agent-dev removed their assignment 2026-06-24 23:11:39 +00:00
agent-pm commented 2026-06-24 23:50:54 +00:00
Member
Copy link

Brief - QA

Phase: default. Run fjx qa prompt default for the cycle protocol.

Context

  • Developer handoff is PR #16: #16
  • Dev reported just check passed and GOTOOLCHAIN=go1.26.4+auto just validate passed twice; plain just validate failed only because the host Go toolchain was go1.26.3 with stdlib findings fixed in go1.26.4.
  • Focus the gap analysis on ordering semantics around microsecond created_at, manifest/tombstone validation, and per-entry mtime precision. The current implementation uses the single-manifest layout at devices/<device>/manifest.json, so the issue's timestamped manifest-basename item may not have an active code path.
<!-- pm:brief:qa --> # Brief - QA **Phase:** default. Run `fjx qa prompt default` for the cycle protocol. ## Context - Developer handoff is PR #16: https://git.tfks.net/tfks/dropbear/pulls/16 - Dev reported `just check` passed and `GOTOOLCHAIN=go1.26.4+auto just validate` passed twice; plain `just validate` failed only because the host Go toolchain was `go1.26.3` with stdlib findings fixed in `go1.26.4`. - Focus the gap analysis on ordering semantics around microsecond `created_at`, manifest/tombstone validation, and per-entry mtime precision. The current implementation uses the single-manifest layout at `devices/<device>/manifest.json`, so the issue's timestamped manifest-basename item may not have an active code path.
agent-pm removed their assignment 2026-06-24 23:51:01 +00:00
agent-qa commented 2026-06-24 23:52:22 +00:00
Member
Copy link

Ledger for: agent-qa

Status: fail @ 2026-06-25T01:34:00.138Z

Action summary

  • PR #16 head: 668c7d8 (Use canonical mtimes in manifest fixtures).
  • Forgejo Actions via fjx qa checks 16: validate.yaml run #25 failed for head 668c7d8; run URL https://git.tfks.net/tfks/dropbear/actions/runs/25.
  • fjx qa checks --help shows this CLI version only lists runs (Usage: fjx qa checks <id>), with no per-step log reader. I did not use the Forgejo API directly.
  • Local focused baseline: GOCACHE=$PWD/.cache/go-build GOMODCACHE=$PWD/.cache/gomod go test ./internal/manifest -count=1 passed.
  • Local standard check: GOCACHE=$PWD/.cache/go-build GOMODCACHE=$PWD/.cache/gomod just check passed (go vet ./..., golangci-lint run ./..., 0 issues.).
  • No committed coverage/summary.json is present in this worktree.

Gap analysis

  • The action/test gap is raw-vs-decoded timestamp validation. Entry mtime remains a raw *string, so validateTimestamp sees the original JSON and rejects non-canonical precision. Manifest created_at, tombstone created_at, and tombstone entry deleted_at decode into ManifestTime first.
  • ManifestTime.UnmarshalJSON parses RFC3339Nano and normalizes via NewManifestTime, which truncates to microseconds. Later Validate() calls .String() for those typed fields, so validation sees the canonicalized value rather than the wire value.
  • Ordering semantics in download.Decide compare timestamp strings. Fixed-width canonical microsecond strings preserve lexical ordering, but accepting and silently truncating non-canonical wire timestamps means peers can publish over-precise values that decode to a different ordering key than the bytes they sent.

Confirmed issues

  • Fail / medium: decode accepts non-canonical typed manifest and tombstone timestamps.
    • Code path: internal/manifest/time.go:75 parses RFC3339Nano, then internal/manifest/time.go:82 truncates through NewManifestTime.
    • Manifest validation then checks m.CreatedAt.String() at internal/manifest/validate.go:79, not the original JSON string.
    • Tombstone validation does the same for created_at at internal/manifest/validate.go:176 and deleted_at at internal/manifest/validate.go:200.
    • Reproduction test added temporarily and removed before handoff:
func TestQANonCanonicalTypedTimestamps(t *testing.T) {
    manifestRaw := []byte(`{"schema_version":1,"created_at":"2026-04-26T12:00:00.123456789Z","device_id":"macbook","entries":[],"parent_manifest":null,"applied_tombstones":{},"root_id":"travel"}`)
    if _, err := DecodeManifest(manifestRaw); err == nil {
        t.Error("DecodeManifest accepted non-canonical created_at")
    }

    tombstoneRaw := []byte(`{"schema_version":1,"created_at":"2026-04-26T12:00:00.123456789Z","device_id":"macbook","root_id":"travel","seq":1,"entries":[{"path":"a.txt","deleted_at":"2026-04-26T12:00:00.123456789Z","previous_type":"file","previous_sha256":"abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890"}]}`)
    if _, err := DecodeTombstone(tombstoneRaw); err == nil {
        t.Error("DecodeTombstone accepted non-canonical typed timestamp")
    }
}
  • Result:
--- FAIL: TestQANonCanonicalTypedTimestamps (0.00s)
    tmp_qa_precision_test.go:8: DecodeManifest accepted non-canonical created_at
    tmp_qa_precision_test.go:13: DecodeTombstone accepted non-canonical typed timestamp
FAIL
FAIL    git.tfks.net/tfks/dropbear/internal/manifest    0.008s

Untested assumptions

  • I did not run full just validate because the PR already has a failed remote validate.yaml run and the confirmed decode bug is sufficient to return the cycle.
  • I did not validate the inactive timestamped manifest-basename path; PM brief says the current implementation uses devices/<device>/manifest.json, so that issue item is not on an active code path.

Verdict

fail

<!-- agent-qa:ledger --> ## Ledger for: agent-qa Status: fail @ 2026-06-25T01:34:00.138Z --- ## Action summary - PR #16 head: `668c7d8` (`Use canonical mtimes in manifest fixtures`). - Forgejo Actions via `fjx qa checks 16`: `validate.yaml` run #25 failed for head `668c7d8`; run URL `https://git.tfks.net/tfks/dropbear/actions/runs/25`. - `fjx qa checks --help` shows this CLI version only lists runs (`Usage: fjx qa checks <id>`), with no per-step log reader. I did not use the Forgejo API directly. - Local focused baseline: `GOCACHE=$PWD/.cache/go-build GOMODCACHE=$PWD/.cache/gomod go test ./internal/manifest -count=1` passed. - Local standard check: `GOCACHE=$PWD/.cache/go-build GOMODCACHE=$PWD/.cache/gomod just check` passed (`go vet ./...`, `golangci-lint run ./...`, `0 issues.`). - No committed `coverage/summary.json` is present in this worktree. ## Gap analysis - The action/test gap is raw-vs-decoded timestamp validation. Entry `mtime` remains a raw `*string`, so `validateTimestamp` sees the original JSON and rejects non-canonical precision. Manifest `created_at`, tombstone `created_at`, and tombstone entry `deleted_at` decode into `ManifestTime` first. - `ManifestTime.UnmarshalJSON` parses RFC3339Nano and normalizes via `NewManifestTime`, which truncates to microseconds. Later `Validate()` calls `.String()` for those typed fields, so validation sees the canonicalized value rather than the wire value. - Ordering semantics in `download.Decide` compare timestamp strings. Fixed-width canonical microsecond strings preserve lexical ordering, but accepting and silently truncating non-canonical wire timestamps means peers can publish over-precise values that decode to a different ordering key than the bytes they sent. ## Confirmed issues - **Fail / medium:** decode accepts non-canonical typed manifest and tombstone timestamps. - Code path: `internal/manifest/time.go:75` parses RFC3339Nano, then `internal/manifest/time.go:82` truncates through `NewManifestTime`. - Manifest validation then checks `m.CreatedAt.String()` at `internal/manifest/validate.go:79`, not the original JSON string. - Tombstone validation does the same for `created_at` at `internal/manifest/validate.go:176` and `deleted_at` at `internal/manifest/validate.go:200`. - Reproduction test added temporarily and removed before handoff: ```go func TestQANonCanonicalTypedTimestamps(t *testing.T) { manifestRaw := []byte(`{"schema_version":1,"created_at":"2026-04-26T12:00:00.123456789Z","device_id":"macbook","entries":[],"parent_manifest":null,"applied_tombstones":{},"root_id":"travel"}`) if _, err := DecodeManifest(manifestRaw); err == nil { t.Error("DecodeManifest accepted non-canonical created_at") } tombstoneRaw := []byte(`{"schema_version":1,"created_at":"2026-04-26T12:00:00.123456789Z","device_id":"macbook","root_id":"travel","seq":1,"entries":[{"path":"a.txt","deleted_at":"2026-04-26T12:00:00.123456789Z","previous_type":"file","previous_sha256":"abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890"}]}`) if _, err := DecodeTombstone(tombstoneRaw); err == nil { t.Error("DecodeTombstone accepted non-canonical typed timestamp") } } ``` - Result: ```text --- FAIL: TestQANonCanonicalTypedTimestamps (0.00s) tmp_qa_precision_test.go:8: DecodeManifest accepted non-canonical created_at tmp_qa_precision_test.go:13: DecodeTombstone accepted non-canonical typed timestamp FAIL FAIL git.tfks.net/tfks/dropbear/internal/manifest 0.008s ``` ## Untested assumptions - I did not run full `just validate` because the PR already has a failed remote `validate.yaml` run and the confirmed decode bug is sufficient to return the cycle. - I did not validate the inactive timestamped manifest-basename path; PM brief says the current implementation uses `devices/<device>/manifest.json`, so that issue item is not on an active code path. ## Verdict fail
agent-qa removed their assignment 2026-06-24 23:55:36 +00:00
agent-dev removed their assignment 2026-06-25 00:11:33 +00:00
agent-dev removed their assignment 2026-06-25 01:14:23 +00:00
agent-qa removed their assignment 2026-06-25 01:34:07 +00:00
agent-dev removed their assignment 2026-06-25 01:51:58 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
add-daemon-lifecycle-control
v0.5.0
v0.4.3
v0.4.2
v0.4.1
v0.4.0
v0.3.0
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels
  
agent
blocked
 Needs human intervention

  
agent
new
 Waiting triage by PM

  
agent
review
 Awaiting next phase

  
agent
working
 Agent is actively implementing

  
complexity
high
 High complexity - use more capable LLM

  
complexity
low
 Low complexity - use simpler LLM

  
priority
high
 High priority

  
priority
low
 Low priority

  
priority
medium
 Medium priority

  
risk
high
 High risk — strict QA, no auto-release

  
risk
low
 Low risk — standard flow

  
risk
medium
 Medium risk — standard QA

  
type
bug
 Bug fix

  
type
chore
 Maintenance, deps, tooling

  
type
feature
 New feature — spec required

  
type
security
 Security change — spec + strict QA required

No labels
agent
blocked
agent
new
agent
review
agent
working
complexity
high
complexity
low
priority
high
priority
low
priority
medium
risk
high
risk
low
risk
medium
type
bug
type
chore
type
feature
type
security
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
agent-pm
4 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
tfks/dropbear#9
No description provided.