narratio/docs/roadmap/implementation.md

Workstream 8 Implementation Plan — Remote Publication And Artifact Locks

Status

Planned implementation for Workstream 8, Remote publication, per-artifact locks, and publish decomposition, in refactoring.md. Workstreams 1 through 7 are complete prerequisites. This plan does not enter Workstream 9's cleanup-stage work.

Authoritative Scope And Fixed Decisions

Workstream 8 in refactoring.md is authoritative. Its supporting records are H04, H15, H18, and H44 in audit.md, C05 in contract-audit.md, and their rows in audit-decisions.md. Read those cited sections rather than repeating the audits.

The following decisions are final:

• Keep publishStage.Run as the transaction owner. Split cohesive work into focused files within internal/stage; add no framework, distributed transaction, or third-party dependency.
• Replace static pipeline locks and the shared remote YAML manifest with one object per normalized stable source at {session_prefix}/locks/{sha256(source)}.lock. The digest is lowercase hexadecimal SHA-256 of the exact normalized source bytes.
• A lock is strict JSON with format_version: 1, source, optional reason, and an RFC 3339 UTC created_at. Define MaxRemoteLockObjectBytes = 64 << 10 centrally in its package. Reject unknown fields, trailing values, invalid/noncanonical fields, unsupported versions, oversize objects, and payload/key hash mismatches.
• internal/artifacts owns the payload, identity/key rules, and lock access; internal/adapters/storage owns object mechanics; internal/app owns commands; and internal/stage owns publish use.
• Extend storage.ObjectStore with idempotent deletion. Listing validates every reserved-prefix object and sorts by source. Add is conditional-create; --force makes one generation-matched replacement; remove is idempotent.
• Ignore old locks.yml objects—do not migrate or delete them. Remove their schema, merge/precedence/retry machinery, runtime injection, and static pipeline.publish.locks. Strict config decoding rejects that old field.
• The initial lock snapshot controls planning. An unlocked selected output publishes normally. A locked output carries its exact declaration from the prior current commit, if any. A locked output never published remains absent even if configured as required. A lock removed after the initial read may therefore cause one conservative carry.
• After immutable objects and the immutable commit are written, reread every initially unlocked relevant source. If any became locked, do not advance the pointer. Attempt objects may remain. A lock created after this final check may race the pointer write; do not add stronger transaction machinery.
• A carried published_output may reference an earlier run only within the same campaign/session. Preserve source, key, checksum, size, and generation exactly. Derive origin from its validated key; do not add an origin field or copy old bytes. Verify existence, size, and generation without downloading.
• Publish metadata and inspection distinguish exactly published, carried_locked, locked_unpublished, optional_missing, and unselected in one source-sorted publish_output_outcomes collection. Preserve upload counts and cleanup-consumed commit/pointer metadata until Workstream 9.
• Preserve transaction order: immutable object upload/verification, immutable commit, final lock revalidation, then exactly one current-pointer write.

Instructions For Every Stage

Before each stage:

1. Read docs/development.md, all documents under docs/policy/, this plan, Workstream 8, its cited audit sections, and the affected sections of ../internal/stage-publish.md, ../internal/storage.md, ../config.md, and ../operations.md.
2. Inspect affected production code and focused tests. Prefer the codebase knowledge graph for Go discovery/call tracing; use text search for schemas, config, CLI text, docs, fixtures, and removal checks.
3. Confirm worktree state and preserve unrelated changes. Do not implement an adjacent workstream.

During each stage, keep focused tests passing, use explicit package-private helpers instead of generic managers/frameworks, keep reads bounded and bodies closed, honor contexts, wrap errors with safe object identity, preserve determinism, and keep tests offline. Run gofmt on changed Go files and git diff --check. Commit only a completed stage with a short plain-English message. Update documentation alongside observable changes.

Stage 1 — Add Idempotent Object Deletion

Status: Completed

Goal

Add the sole storage primitive required to remove one independent lock.

Required Work

1. Add Delete(ctx, key) error to storage.ObjectStore. Absence succeeds; preserve existing key validation and contextual errors.
2. Extend the narrow S3 client port and backend using the provider delete-object operation. Propagate cancellation, permission, transport, and other errors.
3. Update every fake/implementation. The standard fake removes object and generation state, is idempotent, honors context, and permits injected-error tests consistently with its other operations.
4. Add no batch, recursive, or lock-specific storage API.

Verification And Exit Criteria

Test success, absence, cancellation, backend failures, and S3 request mapping. Run go test ./internal/adapters/storage, go test ./internal/artifacts ./internal/stage ./internal/app, and git diff --check. All implementations satisfy the narrow idempotent port.

Stage 2 — Define Independent Lock Objects

Status: Completed

Goal

Establish the complete lock contract before migrating callers.

Required Work

1. Add a cohesive internal/artifacts owner for payloads, size bound, canonical source validation, key construction, and strict encoding/decoding.
2. Reuse publish's canonical stable-source vocabulary. Reject noncanonical caller input rather than making a second identity. Validate the session prefix before joining the reserved locks/ path.
3. Implement exact lookup and prefix listing. Exact lookup returns an explicit present/absent result. Listing bounds every read, rejects unexpected names, verifies hashes, rejects duplicate sources, and sorts by source.
4. Implement conditional creation, one generation-matched forced replacement, and idempotent removal. A replacement conflict is reported for operator retry; do not add an internal retry loop.
5. Do not consult static config or locks.yml.

Verification And Exit Criteria

Table-test canonical keys, strict JSON, size limits, hash mismatches, absence, invalid prefix entries, ordering, add conflicts, force conflicts/replacement, and removal. Run go test ./internal/artifacts ./internal/adapters/storage and git diff --check. Each lock is independently manageable and validated.

Stage 3 — Permit Safe Cross-Run Declarations

Status: Completed

Goal

Permit exact older published declarations without weakening session isolation.

Required Work

1. Add a small key parser proving an immutable object belongs to the expected campaign/session and extracting its run. Reject malformed, mutable, traversal, cross-session, and empty-run keys.
2. Relax commit validation only for published_output: its key may use the current or an earlier run in the same session. Retain every other placement rule and the existing manifest version unless serialization changes.
3. Update restore, previous-state caching, current-output lookup, and status/artifact inspection to open the exact declared key and match by stable source instead of reconstructing a current-run path. Never scan prefixes.
4. Preserve duplicate-source and destination-collision enforcement when carried and current declarations coexist; derive visible origin from key.

Verification And Exit Criteria

Test same/earlier-run acceptance, malformed/mutable/cross-session rejection, exact-key reads, and source-based inspection. Run go test ./internal/artifacts ./internal/app ./internal/stage and git diff --check. Only safe same-session published-output carry is enabled.

Stage 4 — Separate Source And Output Planning

Status: Completed

Goal

Make source opening and output-rule resolution navigable without changing publish behavior.

Required Work

1. Move source discovery, metadata construction, safe opening, and validation from the aggregate publish file into one cohesive package-local owner.
2. Move output selection, source matching, required/optional rules, destination resolution, and collisions into another cohesive owner.
3. Keep publishStage.Run as caller/transaction owner with explicit inputs; avoid a mutable catch-all context. Do not integrate locks yet.
4. Remove only dead helpers/imports exposed by the mechanical split.

Verification And Exit Criteria

Run go test ./internal/stage -run Publish, go test ./internal/app -run 'Publish|Artifact|Status', and git diff --check. Existing behavior remains, with clear source/output file-level ownership.

Stage 5 — Separate Transfer And Commit Mechanics

Status: Completed

Goal

Finish decomposition while keeping the transaction visible in one place.

Required Work

1. Move copying/hashing, immutable upload, remote verification, and reusable prior-object verification into a transfer owner.
2. Move immutable commit and current-pointer construction, encoding, and writes into a commit owner.
3. Keep one explicit ordered path in publishStage.Run: plan, transfer, immutable commit, pre-pointer checks, pointer write. Document its durability boundary briefly.
4. Delete forwarding wrappers and dead helpers; do not add package-local interfaces without a second implementation.

Verification And Exit Criteria

Preserve checksum, size, generation, conditional-write, cancellation, and partial-failure tests. Run go test ./internal/stage -run Publish, go test ./internal/artifacts ./internal/adapters/storage, and git diff --check. The transaction remains easy to audit from Run.

Stage 6 — Model Locked Output Carry-Forward

Status: Completed

Goal

Represent every output outcome and build a coherent commit plan without I/O.

Required Work

1. Add one private output-plan/result type using the five fixed statuses. A participating output is a configured rule selected for this invocation; unselected rules are reported but do not enforce required-output presence.
2. Plan deterministically by source from selected rules, discovered sources, initial locks, and prior current commit.
3. Publish unlocked participants normally. Carry a locked prior declaration byte-for-field. Emit locked_unpublished without required-output failure when none exists. Preserve optional-missing and unselected separately.
4. Reject duplicate prior sources, invalid declarations, configured destination conflicts, and effective destination collisions. Never synthesize an old declaration.
5. Project one source-sorted metadata collection with source/status, destination, exact declared key, and derived origin where applicable; omit secrets and local absolute paths.

Verification And Exit Criteria

Pure table tests cover five states, exact carry, required locked absence, mixed snapshots, selection, ordering, and ambiguities. Run go test ./internal/stage -run 'Publish|Output' and git diff --check. One plan drives both declarations and metadata.

Stage 7 — Integrate Locks Into Publish

Status: Pending

Goal

Make independent locks authoritative while preserving the snapshot transaction.

Required Work

1. Read relevant locks at planning start and load/validate the prior current commit when a locked source may require carry-forward.
2. Use Stage 6's plan. Do not open, hash, or upload a carried or never-published locked source.
3. Before carrying, verify the exact prior object exists and matches declared size and generation without download; retain its declared checksum.
4. Upload/verify new objects, combine exact carried/new declarations, validate and write the immutable commit, then reread initially unlocked sources. If any is now locked, return a clear error without pointer write; otherwise write the pointer exactly once.
5. Remove Env.RevalidatePublishLocks, runner injection, and related plumbing once unused. Publish reads authoritative locks through artifact/storage.
6. Replace overlapping category arrays with publish_output_outcomes. Preserve upload counts, remote_commit_key, and current_commit_pointer_key for current cleanup behavior.

Verification And Exit Criteria

Test no locks, exact/mixed carry, required locked absence, invalid prior state, removed-after-initial lock, added-before-final lock, abandoned attempts, avoided locked-source work, and exactly one pointer write. Run go test ./internal/stage -run Publish, go test ./internal/runner ./internal/app, go test ./..., and git diff --check. Newly observed final locks prevent pointer advancement.

Stage 8 — Migrate Commands And Inspection

Status: Pending

Goal

Preserve the operator workflow over the independent-object model.

Required Work

1. Rewrite lock list/add/remove services and CLI wiring to use artifact lock operations while preserving command names and session selection.
2. List validated source-sorted locks. Add reports already locked unless --force; force performs one conditional replacement. Remove reports idempotent absence without error. Remove help/output references to precedence and shared manifests.
3. Reduce command composition to storage, session prefix, source, reason, clock where needed, and output. Remove config mutation and callbacks.
4. Make status/current-artifact inspection show five outcomes and identify carried sources by exact key and derived run. Fetch only declared objects.
5. Add command tests for exit behavior, conflicts, idempotent remove, deterministic output, malformed locks, and carried inspection.

Verification And Exit Criteria

Run go test ./internal/app ./internal/artifacts, go test ./cmd/..., and git diff --check. Operators retain simple, accurate lock and inspection commands.

Stage 9 — Remove Retired Lock Authorities

Status: Pending

Goal

Delete the obsolete control plane after all consumers migrate.

Required Work

1. Remove pipeline.publish.locks, PublishLockRule, central store types, locks.yml helpers, YAML I/O, merge/precedence logic, document replacement retries, injection fields, compatibility adapters, and dead tests/fixtures.
2. Do not retain migration readers. Remaining historical mentions must be clearly intentional prose.
3. If effective-config digest inputs change, advance its internal discriminator so prior successful state invalidates deliberately. Change no user-facing schema version unless repository policy requires it.
4. Add a representative strict-decode test proving the old config field is unknown, and search code/docs/examples/fixtures for all retired identifiers.

Verification And Exit Criteria

Run:

go test ./internal/config ./internal/artifacts ./internal/app ./internal/stage
go test ./...
git diff --check

Independent objects are the sole lock authority.

Stage 10 — Align Documentation And Close The Workstream

Status: Pending

Goal

Make the simplified implementation the fully documented repository state.

Required Work

1. Update canonical config, storage, stage, CLI, operations, and development docs for the payload/key, bounded validation, commands, carry behavior, race boundary, five outcomes, cross-run rule, and transaction sequence. Remove static-lock and locks.yml claims.
2. Document that old locks.yml is ignored and may be manually removed after upgrade verification; add no migration tool.
3. Remove stale comments, redundant wrappers/logic, dead fixtures, and obsolete metadata assertions exposed by final review.
4. Run all formatting, unit/integration, static-analysis, documentation, and policy checks required by docs/development.md. Record genuinely platform-only checks without weakening or silently skipping them.
5. Only after checks pass, mark this plan's stages and Workstream 8 completed. Leave Workstream 9 as the next pending scope.

Verification And Exit Criteria

At minimum run gofmt on changed Go, go test ./..., go vet ./..., go test ./internal/doccheck, git diff --check, and all repository scripts mandated by policy. Code, tests, docs, help, and examples agree, and no Workstream 9 implementation is included.

Open Questions

None. The audit decisions and fixed choices above make this plan decision-complete.