narratio/docs/roadmap/implementation.md

# Workstream 8 Implementation Plan — Remote Publication And Artifact Locks

## Status

Planned implementation for Workstream 8, **Remote publication, per-artifact
locks, and publish decomposition**, in [`refactoring.md`](refactoring.md).
Workstreams 1 through 7 are complete prerequisites. This plan does not enter
Workstream 9's cleanup-stage work.

## Authoritative Scope And Fixed Decisions

Workstream 8 in [`refactoring.md`](refactoring.md) is authoritative. Its
supporting records are H04, H15, H18, and H44 in [`audit.md`](audit.md), C05
in [`contract-audit.md`](contract-audit.md), and their rows in
[`audit-decisions.md`](audit-decisions.md). Read those cited sections rather
than repeating the audits.

The following decisions are final:

- Keep `publishStage.Run` as the transaction owner. Split cohesive work into
  focused files within `internal/stage`; add no framework, distributed
  transaction, or third-party dependency.
- Replace static pipeline locks and the shared remote YAML manifest with one
  object per normalized stable source at
  `{session_prefix}/locks/{sha256(source)}.lock`. The digest is lowercase
  hexadecimal SHA-256 of the exact normalized source bytes.
- A lock is strict JSON with `format_version: 1`, `source`, optional
  `reason`, and an RFC 3339 UTC `created_at`. Define
  `MaxRemoteLockObjectBytes = 64 << 10` centrally in its package. Reject
  unknown fields, trailing values, invalid/noncanonical fields, unsupported
  versions, oversize objects, and payload/key hash mismatches.
- `internal/artifacts` owns the payload, identity/key rules, and lock access;
  `internal/adapters/storage` owns object mechanics; `internal/app` owns
  commands; and `internal/stage` owns publish use.
- Extend `storage.ObjectStore` with idempotent deletion. Listing validates
  every reserved-prefix object and sorts by source. Add is conditional-create;
  `--force` makes one generation-matched replacement; remove is idempotent.
- Ignore old `locks.yml` objects—do not migrate or delete them. Remove their
  schema, merge/precedence/retry machinery, runtime injection, and static
  `pipeline.publish.locks`. Strict config decoding rejects that old field.
- The initial lock snapshot controls planning. An unlocked selected output
  publishes normally. A locked output carries its exact declaration from the
  prior current commit, if any. A locked output never published remains absent
  even if configured as required. A lock removed after the initial read may
  therefore cause one conservative carry.
- After immutable objects and the immutable commit are written, reread every
  initially unlocked relevant source. If any became locked, do not advance the
  pointer. Attempt objects may remain. A lock created after this final check
  may race the pointer write; do not add stronger transaction machinery.
- A carried `published_output` may reference an earlier run only within the
  same campaign/session. Preserve source, key, checksum, size, and generation
  exactly. Derive origin from its validated key; do not add an origin field or
  copy old bytes. Verify existence, size, and generation without downloading.
- Publish metadata and inspection distinguish exactly `published`,
  `carried_locked`, `locked_unpublished`, `optional_missing`, and
  `unselected` in one source-sorted `publish_output_outcomes` collection.
  Preserve upload counts and cleanup-consumed commit/pointer metadata until
  Workstream 9.
- Preserve transaction order: immutable object upload/verification, immutable
  commit, final lock revalidation, then exactly one current-pointer write.

## Instructions For Every Stage

Before each stage:

1. Read `docs/development.md`, all documents under `docs/policy/`, this
   plan, Workstream 8, its cited audit sections, and the affected sections of
   [`../internal/stage-publish.md`](../internal/stage-publish.md),
   [`../internal/storage.md`](../internal/storage.md),
   [`../config.md`](../config.md), and
   [`../operations.md`](../operations.md).
2. Inspect affected production code and focused tests. Prefer the codebase
   knowledge graph for Go discovery/call tracing; use text search for schemas,
   config, CLI text, docs, fixtures, and removal checks.
3. Confirm worktree state and preserve unrelated changes. Do not implement an
   adjacent workstream.

During each stage, keep focused tests passing, use explicit package-private
helpers instead of generic managers/frameworks, keep reads bounded and bodies
closed, honor contexts, wrap errors with safe object identity, preserve
determinism, and keep tests offline. Run `gofmt` on changed Go files and
`git diff --check`. Commit only a completed stage with a short plain-English
message. Update documentation alongside observable changes.

## Stage 1 — Add Idempotent Object Deletion

**Status:** Completed

### Goal

Add the sole storage primitive required to remove one independent lock.

### Required Work

1. Add `Delete(ctx, key) error` to `storage.ObjectStore`. Absence succeeds;
   preserve existing key validation and contextual errors.
2. Extend the narrow S3 client port and backend using the provider delete-object
   operation. Propagate cancellation, permission, transport, and other errors.
3. Update every fake/implementation. The standard fake removes object and
   generation state, is idempotent, honors context, and permits injected-error
   tests consistently with its other operations.
4. Add no batch, recursive, or lock-specific storage API.

### Verification And Exit Criteria

Test success, absence, cancellation, backend failures, and S3 request mapping.
Run `go test ./internal/adapters/storage`,
`go test ./internal/artifacts ./internal/stage ./internal/app`, and
`git diff --check`. All implementations satisfy the narrow idempotent port.

## Stage 2 — Define Independent Lock Objects

**Status:** Completed

### Goal

Establish the complete lock contract before migrating callers.

### Required Work

1. Add a cohesive `internal/artifacts` owner for payloads, size bound, canonical
   source validation, key construction, and strict encoding/decoding.
2. Reuse publish's canonical stable-source vocabulary. Reject noncanonical
   caller input rather than making a second identity. Validate the session
   prefix before joining the reserved `locks/` path.
3. Implement exact lookup and prefix listing. Exact lookup returns an explicit
   present/absent result. Listing bounds every read, rejects unexpected names,
   verifies hashes, rejects duplicate sources, and sorts by source.
4. Implement conditional creation, one generation-matched forced replacement,
   and idempotent removal. A replacement conflict is reported for operator
   retry; do not add an internal retry loop.
5. Do not consult static config or `locks.yml`.

### Verification And Exit Criteria

Table-test canonical keys, strict JSON, size limits, hash mismatches, absence,
invalid prefix entries, ordering, add conflicts, force conflicts/replacement,
and removal. Run `go test ./internal/artifacts ./internal/adapters/storage` and
`git diff --check`. Each lock is independently manageable and validated.

## Stage 3 — Permit Safe Cross-Run Declarations

**Status:** Completed

### Goal

Permit exact older published declarations without weakening session isolation.

### Required Work

1. Add a small key parser proving an immutable object belongs to the expected
   campaign/session and extracting its run. Reject malformed, mutable,
   traversal, cross-session, and empty-run keys.
2. Relax commit validation only for `published_output`: its key may use the
   current or an earlier run in the same session. Retain every other placement
   rule and the existing manifest version unless serialization changes.
3. Update restore, previous-state caching, current-output lookup, and
   status/artifact inspection to open the exact declared key and match by stable
   source instead of reconstructing a current-run path. Never scan prefixes.
4. Preserve duplicate-source and destination-collision enforcement when
   carried and current declarations coexist; derive visible origin from key.

### Verification And Exit Criteria

Test same/earlier-run acceptance, malformed/mutable/cross-session rejection,
exact-key reads, and source-based inspection. Run
`go test ./internal/artifacts ./internal/app ./internal/stage` and
`git diff --check`. Only safe same-session published-output carry is enabled.

## Stage 4 — Separate Source And Output Planning

**Status:** Completed

### Goal

Make source opening and output-rule resolution navigable without changing
publish behavior.

### Required Work

1. Move source discovery, metadata construction, safe opening, and validation
   from the aggregate publish file into one cohesive package-local owner.
2. Move output selection, source matching, required/optional rules, destination
   resolution, and collisions into another cohesive owner.
3. Keep `publishStage.Run` as caller/transaction owner with explicit inputs;
   avoid a mutable catch-all context. Do not integrate locks yet.
4. Remove only dead helpers/imports exposed by the mechanical split.

### Verification And Exit Criteria

Run `go test ./internal/stage -run Publish`,
`go test ./internal/app -run 'Publish|Artifact|Status'`, and
`git diff --check`. Existing behavior remains, with clear source/output
file-level ownership.

## Stage 5 — Separate Transfer And Commit Mechanics

**Status:** Completed

### Goal

Finish decomposition while keeping the transaction visible in one place.

### Required Work

1. Move copying/hashing, immutable upload, remote verification, and reusable
   prior-object verification into a transfer owner.
2. Move immutable commit and current-pointer construction, encoding, and writes
   into a commit owner.
3. Keep one explicit ordered path in `publishStage.Run`: plan, transfer,
   immutable commit, pre-pointer checks, pointer write. Document its durability
   boundary briefly.
4. Delete forwarding wrappers and dead helpers; do not add package-local
   interfaces without a second implementation.

### Verification And Exit Criteria

Preserve checksum, size, generation, conditional-write, cancellation, and
partial-failure tests. Run `go test ./internal/stage -run Publish`,
`go test ./internal/artifacts ./internal/adapters/storage`, and
`git diff --check`. The transaction remains easy to audit from `Run`.

## Stage 6 — Model Locked Output Carry-Forward

**Status:** Completed

### Goal

Represent every output outcome and build a coherent commit plan without I/O.

### Required Work

1. Add one private output-plan/result type using the five fixed statuses. A
   participating output is a configured rule selected for this invocation;
   unselected rules are reported but do not enforce required-output presence.
2. Plan deterministically by source from selected rules, discovered sources,
   initial locks, and prior current commit.
3. Publish unlocked participants normally. Carry a locked prior declaration
   byte-for-field. Emit `locked_unpublished` without required-output failure
   when none exists. Preserve optional-missing and unselected separately.
4. Reject duplicate prior sources, invalid declarations, configured destination
   conflicts, and effective destination collisions. Never synthesize an old
   declaration.
5. Project one source-sorted metadata collection with source/status,
   destination, exact declared key, and derived origin where applicable; omit
   secrets and local absolute paths.

### Verification And Exit Criteria

Pure table tests cover five states, exact carry, required locked absence, mixed
snapshots, selection, ordering, and ambiguities. Run
`go test ./internal/stage -run 'Publish|Output'` and `git diff --check`.
One plan drives both declarations and metadata.

## Stage 7 — Integrate Locks Into Publish

**Status:** Completed

### Goal

Make independent locks authoritative while preserving the snapshot transaction.

### Required Work

1. Read relevant locks at planning start and load/validate the prior current
   commit when a locked source may require carry-forward.
2. Use Stage 6's plan. Do not open, hash, or upload a carried or never-published
   locked source.
3. Before carrying, verify the exact prior object exists and matches declared
   size and generation without download; retain its declared checksum.
4. Upload/verify new objects, combine exact carried/new declarations, validate
   and write the immutable commit, then reread initially unlocked sources. If
   any is now locked, return a clear error without pointer write; otherwise
   write the pointer exactly once.
5. Remove `Env.RevalidatePublishLocks`, runner injection, and related plumbing
   once unused. Publish reads authoritative locks through artifact/storage.
6. Replace overlapping category arrays with `publish_output_outcomes`.
   Preserve upload counts, `remote_commit_key`, and
   `current_commit_pointer_key` for current cleanup behavior.

### Verification And Exit Criteria

Test no locks, exact/mixed carry, required locked absence, invalid prior state,
removed-after-initial lock, added-before-final lock, abandoned attempts, avoided
locked-source work, and exactly one pointer write. Run
`go test ./internal/stage -run Publish`,
`go test ./internal/runner ./internal/app`, `go test ./...`, and
`git diff --check`. Newly observed final locks prevent pointer advancement.

## Stage 8 — Migrate Commands And Inspection

**Status:** Completed

### Goal

Preserve the operator workflow over the independent-object model.

### Required Work

1. Rewrite lock list/add/remove services and CLI wiring to use artifact lock
   operations while preserving command names and session selection.
2. List validated source-sorted locks. Add reports already locked unless
   `--force`; force performs one conditional replacement. Remove reports
   idempotent absence without error. Remove help/output references to
   precedence and shared manifests.
3. Reduce command composition to storage, session prefix, source, reason,
   clock where needed, and output. Remove config mutation and callbacks.
4. Make status/current-artifact inspection show five outcomes and identify
   carried sources by exact key and derived run. Fetch only declared objects.
5. Add command tests for exit behavior, conflicts, idempotent remove,
   deterministic output, malformed locks, and carried inspection.

### Verification And Exit Criteria

Run `go test ./internal/app ./internal/artifacts`,
`go test ./cmd/...`, and `git diff --check`. Operators retain simple,
accurate lock and inspection commands.

## Stage 9 — Remove Retired Lock Authorities

**Status:** Completed

### Goal

Delete the obsolete control plane after all consumers migrate.

### Required Work

1. Remove `pipeline.publish.locks`, `PublishLockRule`, central store types,
   `locks.yml` helpers, YAML I/O, merge/precedence logic, document replacement
   retries, injection fields, compatibility adapters, and dead tests/fixtures.
2. Do not retain migration readers. Remaining historical mentions must be
   clearly intentional prose.
3. If effective-config digest inputs change, advance its internal discriminator
   so prior successful state invalidates deliberately. Change no user-facing
   schema version unless repository policy requires it.
4. Add a representative strict-decode test proving the old config field is
   unknown, and search code/docs/examples/fixtures for all retired identifiers.

### Verification And Exit Criteria

Run:

```sh
go test ./internal/config ./internal/artifacts ./internal/app ./internal/stage
go test ./...
git diff --check
```

Independent objects are the sole lock authority.

## Stage 10 — Align Documentation And Close The Workstream

**Status:** Pending

### Goal

Make the simplified implementation the fully documented repository state.

### Required Work

1. Update canonical config, storage, stage, CLI, operations, and development
   docs for the payload/key, bounded validation, commands, carry behavior, race
   boundary, five outcomes, cross-run rule, and transaction sequence. Remove
   static-lock and `locks.yml` claims.
2. Document that old `locks.yml` is ignored and may be manually removed after
   upgrade verification; add no migration tool.
3. Remove stale comments, redundant wrappers/logic, dead fixtures, and obsolete
   metadata assertions exposed by final review.
4. Run all formatting, unit/integration, static-analysis, documentation, and
   policy checks required by `docs/development.md`. Record genuinely
   platform-only checks without weakening or silently skipping them.
5. Only after checks pass, mark this plan's stages and Workstream 8 completed.
   Leave Workstream 9 as the next pending scope.

### Verification And Exit Criteria

At minimum run `gofmt` on changed Go, `go test ./...`, `go vet ./...`,
`go test ./internal/doccheck`, `git diff --check`, and all repository
scripts mandated by policy. Code, tests, docs, help, and examples agree, and no
Workstream 9 implementation is included.

## Open Questions

None. The audit decisions and fixed choices above make this plan
decision-complete.