diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index d67bf64..ad12598 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,524 +1,386 @@ -# Workstream 7 Implementation Plan — Configuration Structure And Diagnostics +# Workstream 8 Implementation Plan — Remote Publication And Artifact Locks ## Status -Completed implementation record for Workstream 7, **Configuration structure -and semantic diagnostics**, in [`refactoring.md`](refactoring.md). Workstreams -1 through 6 were prerequisites, and all stages below were completed in -numerical order. +Planned implementation for Workstream 8, **Remote publication, per-artifact +locks, and publish decomposition**, in [`refactoring.md`](refactoring.md). +Workstreams 1 through 7 are complete prerequisites. This plan does not enter +Workstream 9's cleanup-stage work. -The purpose of this workstream is to preserve Narratio's strict, deterministic, -operator-oriented configuration behavior while making the implementation -navigable by semantic domain. It also removes incidental composition metadata -and test expectations that turn private validation choreography into a -compatibility burden. This is an internal refactor with narrowly approved -diagnostic flexibility, not a configuration-language redesign. +## Authoritative Scope And Fixed Decisions -## Authoritative Scope And Decisions +Workstream 8 in [`refactoring.md`](refactoring.md) is authoritative. Its +supporting records are H04, H15, H18, and H44 in [`audit.md`](audit.md), C05 +in [`contract-audit.md`](contract-audit.md), and their rows in +[`audit-decisions.md`](audit-decisions.md). Read those cited sections rather +than repeating the audits. -The authoritative scope is Workstream 7 in [`refactoring.md`](refactoring.md). -Its supporting audit entries are H04, H14, H15, H17, H38, H39, H40, H41, and -H42 in [`audit.md`](audit.md), C01 and C06 in -[`contract-audit.md`](contract-audit.md), and the corresponding rows in -[`audit-decisions.md`](audit-decisions.md). +The following decisions are final: -The following implementation decisions are fixed: - -- Keep all configuration behavior in package `internal/config`. Do not create - subpackages, a generic configuration framework, a schema framework, a - validator registry, or a dependency-injection layer. -- Preserve the public configuration types and entry points. `Validate`, - `ValidatePipelineConfig`, `ValidateCampaignConfig`, all load APIs, effective - pipeline projections, and configuration-inspection commands retain their - current responsibilities. -- Keep `validate.go` as the short public and high-level orchestration owner. - Organize private validation into cohesive files for core pipeline/session, - adapters, publish/storage, Notarius, Scriptorium, and shared scalar/path - helpers. File names are private, but those ownership boundaries are the - intended final structure. -- Do not replace ordinary `error` returns with a new exported validation-error - hierarchy. No machine consumer currently requires one, and introducing one - would add a new compatibility surface. Field paths, source identities for - composition conflicts, wrapped causes where useful, and small semantic - phrases provide the stable diagnostic content tests should assert. -- Continue to fail fast. The workstream does not add error aggregation and does - not promise which of several independent invalid fields is reported first. - Traversal of maps must nevertheless be deterministic so identical input - produces identical behavior. -- Preserve strict single-document YAML decoding, duplicate- and unknown-field - rejection, additive imports with no duplicate ownership, one deterministic - profile selection, explicit-value overlay behavior, centralized defaults, - canonical party resolution, artifact-family expansion, and secret-free - effective digests. -- Preserve the current configuration schema. Campaign-owned - `campaign.inputs.party_file` and canonical artifact source IDs are current - contracts, not legacy fields. Removed player/session roster fields and old - artifact source spellings remain unsupported; add no compatibility aliases - or normalization branches for them. -- Treat Workstream 3's compatibility purge as authoritative. During this work, - remove any surviving production branch that actually accepts or translates - a retired configuration representation. A rejection supplied naturally by - strict decoding or the canonical source validator is sufficient; do not - maintain exhaustive legacy-form tests when representative canonical-boundary - tests already protect the rule. -- Preserve composition source information that is consumed by conflict - diagnostics or `config sources`. Remove only demonstrably unused node/field - metadata. In particular, source lists and semantic field paths remain; - redundant cached declaration-order indexes and copied line/column fields may - be removed when no consumer reads them because slice order and the YAML - parser already supply the required behavior. -- Preserve normalization as an all-or-nothing operation for each validator. - Notarius reference/output maps and scalar paths are installed on the config - only after all relevant checks succeed; a validation error must not leave a - partially normalized value behind. -- Keep `validateAudita` intact when moving it to the adapter-validation owner. - Its guards are cohesive; splitting them further would add names without - reducing complexity. -- Tests must protect semantic rules and observable normalized results, not - private helper calls, file placement, full error prose, punctuation, or one - historical validation call order. Exact strings remain appropriate only for - an intentionally documented CLI or machine-consumed contract. -- Do not enter Workstream 8 remote publication/lock decomposition, Workstream 9 - cleanup-stage work, or unrelated Workstream 10 cleanup. +- Keep `publishStage.Run` as the transaction owner. Split cohesive work into + focused files within `internal/stage`; add no framework, distributed + transaction, or third-party dependency. +- Replace static pipeline locks and the shared remote YAML manifest with one + object per normalized stable source at + `{session_prefix}/locks/{sha256(source)}.lock`. The digest is lowercase + hexadecimal SHA-256 of the exact normalized source bytes. +- A lock is strict JSON with `format_version: 1`, `source`, optional + `reason`, and an RFC 3339 UTC `created_at`. Define + `MaxRemoteLockObjectBytes = 64 << 10` centrally in its package. Reject + unknown fields, trailing values, invalid/noncanonical fields, unsupported + versions, oversize objects, and payload/key hash mismatches. +- `internal/artifacts` owns the payload, identity/key rules, and lock access; + `internal/adapters/storage` owns object mechanics; `internal/app` owns + commands; and `internal/stage` owns publish use. +- Extend `storage.ObjectStore` with idempotent deletion. Listing validates + every reserved-prefix object and sorts by source. Add is conditional-create; + `--force` makes one generation-matched replacement; remove is idempotent. +- Ignore old `locks.yml` objects—do not migrate or delete them. Remove their + schema, merge/precedence/retry machinery, runtime injection, and static + `pipeline.publish.locks`. Strict config decoding rejects that old field. +- The initial lock snapshot controls planning. An unlocked selected output + publishes normally. A locked output carries its exact declaration from the + prior current commit, if any. A locked output never published remains absent + even if configured as required. A lock removed after the initial read may + therefore cause one conservative carry. +- After immutable objects and the immutable commit are written, reread every + initially unlocked relevant source. If any became locked, do not advance the + pointer. Attempt objects may remain. A lock created after this final check + may race the pointer write; do not add stronger transaction machinery. +- A carried `published_output` may reference an earlier run only within the + same campaign/session. Preserve source, key, checksum, size, and generation + exactly. Derive origin from its validated key; do not add an origin field or + copy old bytes. Verify existence, size, and generation without downloading. +- Publish metadata and inspection distinguish exactly `published`, + `carried_locked`, `locked_unpublished`, `optional_missing`, and + `unselected` in one source-sorted `publish_output_outcomes` collection. + Preserve upload counts and cleanup-consumed commit/pointer metadata until + Workstream 9. +- Preserve transaction order: immutable object upload/verification, immutable + commit, final lock revalidation, then exactly one current-pointer write. ## Instructions For Every Stage Before each stage: -1. Read `docs/development.md`, every document under `docs/policy/`, this plan, - and Workstream 7 in [`refactoring.md`](refactoring.md). Read only the audit - sections and decision-register rows cited above; do not restart the audits. -2. Read [`../internal/configuration.md`](../internal/configuration.md) and the - affected portions of [`../config.md`](../config.md), then inspect the - production files and focused tests named by the stage. -3. Prefer the codebase knowledge graph for Go symbol discovery and call - tracing. Use text search for YAML fields, source IDs, error fragments, - examples, documentation, and removal verification. -4. Confirm the worktree state and preserve unrelated changes. Do not implement - an adjacent pending workstream merely because it shares a file. +1. Read `docs/development.md`, all documents under `docs/policy/`, this + plan, Workstream 8, its cited audit sections, and the affected sections of + [`../internal/stage-publish.md`](../internal/stage-publish.md), + [`../internal/storage.md`](../internal/storage.md), + [`../config.md`](../config.md), and + [`../operations.md`](../operations.md). +2. Inspect affected production code and focused tests. Prefer the codebase + knowledge graph for Go discovery/call tracing; use text search for schemas, + config, CLI text, docs, fixtures, and removal checks. +3. Confirm worktree state and preserve unrelated changes. Do not implement an + adjacent workstream. -During every stage: +During each stage, keep focused tests passing, use explicit package-private +helpers instead of generic managers/frameworks, keep reads bounded and bodies +closed, honor contexts, wrap errors with safe object identity, preserve +determinism, and keep tests offline. Run `gofmt` on changed Go files and +`git diff --check`. Commit only a completed stage with a short plain-English +message. Update documentation alongside observable changes. -- Keep the package compiling and focused tests passing. Move behavior before - redesigning the two complex validators so reviewable commits do not combine - unrelated mechanical and semantic changes. -- Use package-private functions with explicit inputs and return values. Avoid - stateful validator objects, forwarding-only wrappers, generic rule tables, - and abstractions whose only purpose is to reduce function length. -- Keep configuration mutation explicit and late. Do not mutate caller-owned - maps while validating a temporary normalized candidate. -- Retain deterministic order wherever config maps are traversed for validation, - normalization, canonical output, source reporting, or digest construction. -- Keep tests deterministic, offline, and independent of credentials or live - adapters. Prefer direct package behavior and real temporary files over mocks. -- Use `gofmt` on changed Go files, run the focused checks listed for the stage, - and run `git diff --check`. Mark a stage `Completed` only after its exit - criteria pass. +## Stage 1 — Add Idempotent Object Deletion -## Stage 1 — Separate Core And Shared Validation Ownership - -**Status:** Completed +**Status:** Pending ### Goal -Reduce `validate.go` to configuration validation entry points and readable -high-level sequencing, while giving core pipeline/session rules and reusable -primitive checks clear package-local owners. This stage is behavior-preserving. +Add the sole storage primitive required to remove one independent lock. ### Required Work -1. Keep these entry points and the visible pipeline-domain call sequence in - `internal/config/validate.go`: - - `Validate`; - - `ValidatePipelineConfig`; - - `ValidateCampaignConfig`; and - - the short `validatePipeline` coordinator. -2. Move core configuration checks into a cohesive private file such as - `validate_core.go`. This owner includes campaign validation, secrets, - spool/cache, session validation, resolved-party input validation, - session/campaign identifiers, and cross-pipeline/session rules. Keep - `publishUploadConfiguredForS3` with publish/storage in Stage 2 rather than - hiding publish policy in the core owner. -3. Move general validation primitives into a small private file such as - `validate_helpers.go`: environment-variable name checks, positive duration - parsing, safe relative paths, and path-within-root checks. Do not turn these - functions into a generic validation DSL. -4. Re-home the straightforward stage/adapter validators in a private adapter - owner such as `validate_adapters.go`: notification, normalize, trim, render, - WhisperX, Seriatim (including its output schema), and Audita. Move - `validateAudita` intact apart from import or formatting changes. -5. Preserve current nil handling, defaults, normalization, wrapped causes, - field identification, and high-level validation semantics. Do not use this - mechanical move to reorder domains or rewrite diagnostics. -6. Remove imports and dead private helpers left behind in `validate.go`. Do not - add facade functions to preserve former source-file placement; files are not - Go API boundaries. +1. Add `Delete(ctx, key) error` to `storage.ObjectStore`. Absence succeeds; + preserve existing key validation and contextual errors. +2. Extend the narrow S3 client port and backend using the provider delete-object + operation. Propagate cancellation, permission, transport, and other errors. +3. Update every fake/implementation. The standard fake removes object and + generation state, is idempotent, honors context, and permits injected-error + tests consistently with its other operations. +4. Add no batch, recursive, or lock-specific storage API. -### Tests And Verification +### Verification And Exit Criteria -- Existing core, campaign, session, duration, normalize, trim, render, - notification, and load/validate tests should normally pass unchanged. -- Add no tests that merely assert a function moved files. Add or repair a test - only if the move exposes an unprotected semantic rule. -- Run: +Test success, absence, cancellation, backend failures, and S3 request mapping. +Run `go test ./internal/adapters/storage`, +`go test ./internal/artifacts ./internal/stage ./internal/app`, and +`git diff --check`. All implementations satisfy the narrow idempotent port. - ```sh - go test ./internal/config - go test ./internal/app -run 'Config|Pipeline|Session' - git diff --check - ``` +## Stage 2 — Define Independent Lock Objects -### Exit Criteria - -- `validate.go` is a concise entry/coordinator file rather than a container for - unrelated domain implementations. -- Core, adapter, and shared helper ownership is obvious from the file layout, - and Audita remains one cohesive validator. -- No public behavior or serialized configuration shape changed. - -## Stage 2 — Separate Publish, Storage, Notarius, And Scriptorium Owners - -**Status:** Completed +**Status:** Pending ### Goal -Finish the domain-level split of the former aggregate validator before changing -the internal structure of Notarius or Scriptorium validation. +Establish the complete lock contract before migrating callers. ### Required Work -1. Move storage, publish output, and publish lock validation into a cohesive - private owner such as `validate_publish.go`. Move the associated helpers - with their consumers, including `ValidatePublishLockRules`, - `MergePublishLockRules`, configured output/source lookups, and - `publishUploadConfiguredForS3`. -2. Preserve the exported lock-rule helper signatures because current app and - stage code consume them. Do not redesign remote lock behavior; that belongs - to Workstream 8. -3. Move the existing Notarius validator into `validate_notarius.go` and its - Scriptorium counterpart, input-source validator, and exported dependency - validator into `validate_scriptorium.go`. At this stage, move these functions - without decomposing or semantically reordering them; Stages 3 and 4 own those - changes. -4. Keep source-vocabulary and collision helpers with the narrowest shared - domain owner. If both publish and Scriptorium need a small lookup derived - from Notarius/Scriptorium configuration, use one plainly named package - helper rather than duplicate maps or introduce a registry. -5. Ensure `validate.go` contains only entry points and high-level sequencing - after the move. Delete the emptied blocks and unused imports. -6. Search `internal/config` production code for acceptance or translation of - removed player/session roster fields and retired artifact source spellings. - Delete any actual compatibility branch found. Do not mistake current - campaign `party_file`, rejection through strict decoding, migration prose, - or a test fixture demonstrating rejection for an active compatibility path. +1. Add a cohesive `internal/artifacts` owner for payloads, size bound, canonical + source validation, key construction, and strict encoding/decoding. +2. Reuse publish's canonical stable-source vocabulary. Reject noncanonical + caller input rather than making a second identity. Validate the session + prefix before joining the reserved `locks/` path. +3. Implement exact lookup and prefix listing. Exact lookup returns an explicit + present/absent result. Listing bounds every read, rejects unexpected names, + verifies hashes, rejects duplicate sources, and sorts by source. +4. Implement conditional creation, one generation-matched forced replacement, + and idempotent removal. A replacement conflict is reported for operator + retry; do not add an internal retry loop. +5. Do not consult static config or `locks.yml`. -### Tests And Verification +### Verification And Exit Criteria -- Existing storage/publish, Notarius, Scriptorium, configuration loading, and - application configuration tests should pass without pinning file placement. -- Run: +Table-test canonical keys, strict JSON, size limits, hash mismatches, absence, +invalid prefix entries, ordering, add conflicts, force conflicts/replacement, +and removal. Run `go test ./internal/artifacts ./internal/adapters/storage` and +`git diff --check`. Each lock is independently manageable and validated. - ```sh - go test ./internal/config - go test ./internal/app ./internal/stage - git diff --check - ``` +## Stage 3 — Permit Safe Cross-Run Declarations -### Exit Criteria - -- Every validation domain named by Workstream 7 has one discoverable file-level - owner inside `internal/config`. -- There is no duplicate source/collision logic and no active retired-format - compatibility branch in configuration production code. -- Notarius and Scriptorium semantics remain unchanged pending their dedicated - stages. - -## Stage 3 — Decompose Notarius Validation And Normalization - -**Status:** Completed +**Status:** Pending ### Goal -Make `validateNotarius` a short transaction that validates top-level policy, -normalizes references, normalizes output lanes, and commits the resulting -configuration only after every check succeeds. +Permit exact older published declarations without weakening session isolation. ### Required Work -1. Keep `validateNotarius` responsible for nil/enablement behavior and the - top-level scalar contract: required binary, configuration path, pipeline ID, - positive timeout, working directory, at least one output, and the reference - binding limit. -2. Replace the validator's bespoke timeout parsing with the shared positive - duration helper when it preserves the same accepted values and field - identity. Do not embed duration policy in a Notarius-only branch. -3. Extract reference binding normalization into one private operation that: - - iterates raw selector keys in sorted order; - - normalizes selectors through `notariusref`; - - rejects normalized selector collisions; - - trims and validates each prepared-input source; and - - returns a newly allocated normalized map without mutating the input map. -4. Extract output normalization and collision validation into a separate - private operation that: - - builds the reserved built-in, configured Scriptorium, and previous-session - source set once; - - iterates raw output keys in sorted order; - - validates and normalizes configured keys and derived extraction source - IDs; - - rejects reserved-source, normalized-source, and lane-ID collisions; - - trims lane/media/schema/module scalar fields and checks required fields; - and - - returns a newly allocated normalized output map. -5. Normalize binary, paths, pipeline ID, timeout, and working directory into - local candidate values. Assign all scalar candidates, references, and - outputs back to `cfg` only after both helper operations succeed. Preserve - nil-versus-empty map behavior where it is observable in effective YAML or - digests. -6. Keep all helpers Notarius-specific and private. Do not move selector policy - out of `notariusref`, copy artifact source policy, or create a generalized - map-normalization framework. +1. Add a small key parser proving an immutable object belongs to the expected + campaign/session and extracting its run. Reject malformed, mutable, + traversal, cross-session, and empty-run keys. +2. Relax commit validation only for `published_output`: its key may use the + current or an earlier run in the same session. Retain every other placement + rule and the existing manifest version unless serialization changes. +3. Update restore, previous-state caching, current-output lookup, and + status/artifact inspection to open the exact declared key and match by stable + source instead of reconstructing a current-run path. Never scan prefixes. +4. Preserve duplicate-source and destination-collision enforcement when + carried and current declarations coexist; derive visible origin from key. -### Tests And Verification +### Verification And Exit Criteria -- Retain lean behavior tests for disabled configuration, required scalars, - timeout and reference-binding limits, selector normalization/collisions, supported - prepared sources, output normalization, reserved-source collisions, - duplicate lanes, and cross-use by Scriptorium/publish. -- Add or consolidate one test proving a failing reference or output candidate - leaves the original Notarius config unmodified. -- Add one determinism test that constructs equivalent reference/output maps in - different insertion orders and compares normalized success or equivalent - diagnostic classification. Do not assert which of multiple independent - invalid domains must win forever. -- Assert field paths and the smallest meaningful semantic fragment rather than - complete error sentences. -- Run: +Test same/earlier-run acceptance, malformed/mutable/cross-session rejection, +exact-key reads, and source-based inspection. Run +`go test ./internal/artifacts ./internal/app ./internal/stage` and +`git diff --check`. Only safe same-session published-output carry is enabled. - ```sh - go test ./internal/config -run 'Notarius|Scriptorium|Publish|Duration' - go test ./internal/stage -run 'Extract|Analyze' - git diff --check - ``` +## Stage 4 — Separate Source And Output Planning -### Exit Criteria - -- `validateNotarius` visibly sequences scalar validation, reference - normalization, output normalization/collision checks, and one final commit. -- Reference and output collision domains are separate, deterministic, and do - not partially mutate caller-owned configuration on failure. -- No Notarius behavior moved into a generic configuration abstraction. - -## Stage 4 — Decompose Deterministic Scriptorium Validation - -**Status:** Completed +**Status:** Pending ### Goal -Separate Scriptorium artifact declarations/outputs, dependencies/inputs, and -variables into readable validation phases with deterministic map traversal. +Make source opening and output-rule resolution navigable without changing +publish behavior. ### Required Work -1. Keep `validateScriptorium` as the transaction owner for nil behavior, - required binary, optional config path, stage timeout, ordered validation - phases, referenced-output checks, and the final dependency-graph check. -2. Produce one sorted list of configured artifact keys after validating their - canonical key syntax. Reuse that order across phases rather than ranging - over the artifact map independently. -3. Extract a declaration/output phase that validates enabled prompt and output - requirements, safe output paths, artifact timeout values, and duplicate - output ownership. Keep its collision map local to that phase. -4. Extract a dependency/input phase that: - - validates dependency entries and self-reference using declared slice - indexes for actionable field paths; - - iterates input maps by sorted input name; - - validates required source declarations through the existing canonical - input-source owner; - - requires configured-artifact sources to appear in `depends_on`; and - - returns or updates only the small referenced-artifact set needed by the - later referenced-output check. -5. Extract a variable phase that iterates variable names in sorted order, - rejects empty names, and accepts only the currently supported string and - boolean values. -6. Iterate the referenced-artifact set deterministically when checking required - outputs. Keep `ValidateScriptoriumArtifactDependencies` as the public graph - validator and preserve its cycle and dependency semantics. -7. It is acceptable for phase separation to change which of several unrelated - simultaneous mistakes is reported first. Within a given implementation, - equivalent maps must produce the same result. Do not add compatibility code - to reproduce the old mixed per-artifact call order. -8. Remove forwarding-only helpers and obsolete temporary sets after the split. - Keep `validateScriptoriumInputSource` cohesive unless a genuine duplicate - semantic owner emerges. +1. Move source discovery, metadata construction, safe opening, and validation + from the aggregate publish file into one cohesive package-local owner. +2. Move output selection, source matching, required/optional rules, destination + resolution, and collisions into another cohesive owner. +3. Keep `publishStage.Run` as caller/transaction owner with explicit inputs; + avoid a mutable catch-all context. Do not integrate locks yet. +4. Remove only dead helpers/imports exposed by the mechanical split. -### Tests And Verification +### Verification And Exit Criteria -- Retain semantic coverage for declaration keys, enabled requirements, output - path safety/collisions, timeouts, dependency existence/self/cycles, input - source classes, extraction references, required `depends_on`, referenced - outputs, and variable types. -- Add one map-insertion-order test that feeds equivalent Scriptorium maps in - different insertion orders and asserts equal validity or equivalent - diagnostics. Do not assert an arbitrary globally preferred independent-error - order. -- Consolidate table cases that repeat the same source-vocabulary rule through - different retired spellings when a representative canonical rejection test - already protects unsupported-source behavior. -- Run: +Run `go test ./internal/stage -run Publish`, +`go test ./internal/app -run 'Publish|Artifact|Status'`, and +`git diff --check`. Existing behavior remains, with clear source/output +file-level ownership. - ```sh - go test ./internal/config -run 'Scriptorium|ArtifactFamil|Notarius|Publish' - go test ./internal/stage -run 'Analyze' - go test ./internal/app -run 'Artifact|Config|Plan' - git diff --check - ``` +## Stage 5 — Separate Transfer And Commit Mechanics -### Exit Criteria - -- Scriptorium validation has explicit declaration/output, - dependency/input, variable, and graph phases. -- All map traversal that can affect validation or normalization is - deterministic. -- Tests protect fields and semantic rules without freezing the former mixed - validator choreography. - -## Stage 5 — Split Composition Envelope Phases And Remove Incidental Metadata - -**Status:** Completed +**Status:** Pending ### Goal -Separate composition-envelope shape decoding, cross-field semantics, and -root-only metadata removal while retaining the source-aware configuration model -that imports, profiles, diagnostics, and inspection commands require. +Finish decomposition while keeping the transaction visible in one place. ### Required Work -1. Refactor `splitPipelineCompositionEnvelope` in - `internal/config/pipeline_composition.go` into a short coordinator over - three explicit private phases: - - locate and decode the root `composition` mapping and its `imports`, - `default_profile`, and `profiles` shapes; - - validate cross-field semantics, including default-profile membership; and - - clone/remove the root-only envelope to produce the ordinary pipeline body. -2. Reuse `parsePipelineProfileDeclaration` and profile-name normalization. - Keep unknown-field rejection, non-empty profile declarations, imports and - profiles in declaration order, explicit-profile selection behavior, and - source-qualified shape/conflict diagnostics. -3. Do not mutate the parsed root document during validation. Return a distinct - body document only after the envelope is valid; the original document must - remain reusable by callers and tests. -4. Remove `compositionNode` and `compositionField` metadata that has no runtime - reader after the refactor. Specifically verify and remove redundant cached - `order`, `line`, and `column` fields and their clone/renumber assignments if - they remain write-only. Slice order remains the declaration-order - representation; YAML parser nodes remain the source of parse-time line and - column diagnostics. -5. Retain each node's semantic path and source list because merge conflict - reporting, ownership projection, effective digests, artifact-family source - projection, and `config sources` consume them. Do not reduce actionable - conflict diagnostics or delete provenance that has a real consumer. -6. Keep composition product-specific. Do not replace the tree with generic map - decoding or adopt a third-party composition library. +1. Move copying/hashing, immutable upload, remote verification, and reusable + prior-object verification into a transfer owner. +2. Move immutable commit and current-pointer construction, encoding, and writes + into a commit owner. +3. Keep one explicit ordered path in `publishStage.Run`: plan, transfer, + immutable commit, pre-pointer checks, pointer write. Document its durability + boundary briefly. +4. Delete forwarding wrappers and dead helpers; do not add package-local + interfaces without a second implementation. -### Tests And Verification +### Verification And Exit Criteria -- Test malformed envelope shapes, unknown fields, invalid profile declarations, - missing default membership, successful envelope removal, and absence of - caller-document mutation through the composition boundary. -- Preserve tests for explicit empty/false/zero/list/map values, additive - conflicts naming all claiming sources, overlay semantics, canonical YAML, - effective digest stability, profile selection, and `config sources` - ownership. These are semantic contracts, not incidental provenance. -- Remove assertions of deleted private cached metadata. Continue asserting - declaration behavior through ordered outputs rather than private index - fields. -- Run: +Preserve checksum, size, generation, conditional-write, cancellation, and +partial-failure tests. Run `go test ./internal/stage -run Publish`, +`go test ./internal/artifacts ./internal/adapters/storage`, and +`git diff --check`. The transaction remains easy to audit from `Run`. - ```sh - go test ./internal/config -run 'Composition|PipelineProfile|Effective|ArtifactFamil' - go test ./internal/app -run 'Config' - git diff --check - ``` +## Stage 6 — Model Locked Output Carry-Forward -### Exit Criteria - -- Envelope decoding, cross-field validation, and body production have separate - focused owners under one coordinator. -- Only provenance and ordering data with a demonstrated consumer remains. -- Imports, profiles, overlays, source reporting, and effective digests retain - deterministic behavior. - -## Stage 6 — Align Diagnostic Tests, Documentation, And Close Workstream 7 - -**Status:** Completed +**Status:** Pending ### Goal -Review the completed configuration refactor as one system, remove obsolete test -friction and documentation language, and close Workstream 7 with repository-wide -evidence. +Represent every output outcome and build a coherent commit plan without I/O. ### Required Work -1. Review every configuration test changed by Stages 1–5. Retain coverage of - schema strictness, normalization, semantic invariants, meaningful failure - categories, deterministic outcomes, composition source conflicts, effective - projections/digests, and representative assembled loads. -2. Rewrite assertions that pin complete diagnostic prose, punctuation, - formatting, private helper order, or an arbitrary independent-error winner. - Prefer a relevant field path plus one small semantic fragment. Use - `errors.Is`/`errors.As` only where an existing wrapped or typed cause is a - real contract; do not introduce types solely to change test style. -3. Delete redundant legacy-form enumerations whose only purpose is to preserve - knowledge of retired fields or spellings already rejected by the canonical - strict/source boundary. Retain representative tests proving unknown fields - and unsupported sources fail, and retain migration guidance tests only when - that guidance is an intentional user-facing contract. -4. Consolidate oversized table fixtures only where multiple cases protect the - same rule. Do not delete distinct risks merely to reduce test LOC, and do not - rewrite healthy semantic tests just because they use a small stable error - fragment. -5. Update [`../internal/configuration.md`](../internal/configuration.md) to - describe the implemented validation-domain ownership, deterministic phase - behavior, composition-envelope phases, and diagnostic guarantees without - listing private helper choreography. Update [`../config.md`](../config.md) - only if its user-visible current-behavior wording is inaccurate; no schema - change is expected. -6. Search current production, tests, examples, and current-behavior docs for - active acceptance of the retired configuration forms named by C01. Remove - actual compatibility code and stale current-behavior claims, while leaving - immutable audit/history documents unchanged. -7. Run the full repository validation from `docs/development.md`: +1. Add one private output-plan/result type using the five fixed statuses. A + participating output is a configured rule selected for this invocation; + unselected rules are reported but do not enforce required-output presence. +2. Plan deterministically by source from selected rules, discovered sources, + initial locks, and prior current commit. +3. Publish unlocked participants normally. Carry a locked prior declaration + byte-for-field. Emit `locked_unpublished` without required-output failure + when none exists. Preserve optional-missing and unselected separately. +4. Reject duplicate prior sources, invalid declarations, configured destination + conflicts, and effective destination collisions. Never synthesize an old + declaration. +5. Project one source-sorted metadata collection with source/status, + destination, exact declared key, and derived origin where applicable; omit + secrets and local absolute paths. - ```sh - go test ./... - go test -race ./... - go vet ./... - go build ./... - go test ./internal/doccheck - go test ./internal/config -run '^TestExamplesLoadAndValidate$' - ``` +### Verification And Exit Criteria -8. Run `go test -race -shuffle=on -count=3 ./internal/config ./internal/app` - to stress map-order independence and assembled configuration behavior. -9. After all checks pass, mark Stages 1–6 in this document `Completed` and mark - only Workstream 7 in [`refactoring.md`](refactoring.md) `Complete`. - Workstream 8 must remain the next pending workstream. +Pure table tests cover five states, exact carry, required locked absence, mixed +snapshots, selection, ordering, and ambiguities. Run +`go test ./internal/stage -run 'Publish|Output'` and `git diff --check`. +One plan drives both declarations and metadata. -### Exit Criteria +## Stage 7 — Integrate Locks Into Publish -- Configuration remains strict, deterministic, source-aware where useful, and - unchanged in public schema and normalized semantics. -- Validation code is organized by semantic owner; Notarius, Scriptorium, and - composition coordinators expose clear phases without generic frameworks. -- No active retired configuration compatibility path, write-only composition - metadata, or test dependency on incidental diagnostic choreography remains. -- Current documentation describes the implemented boundaries, all validation - commands pass, Workstream 7 is complete, and Workstream 8 remains pending. +**Status:** Pending + +### Goal + +Make independent locks authoritative while preserving the snapshot transaction. + +### Required Work + +1. Read relevant locks at planning start and load/validate the prior current + commit when a locked source may require carry-forward. +2. Use Stage 6's plan. Do not open, hash, or upload a carried or never-published + locked source. +3. Before carrying, verify the exact prior object exists and matches declared + size and generation without download; retain its declared checksum. +4. Upload/verify new objects, combine exact carried/new declarations, validate + and write the immutable commit, then reread initially unlocked sources. If + any is now locked, return a clear error without pointer write; otherwise + write the pointer exactly once. +5. Remove `Env.RevalidatePublishLocks`, runner injection, and related plumbing + once unused. Publish reads authoritative locks through artifact/storage. +6. Replace overlapping category arrays with `publish_output_outcomes`. + Preserve upload counts, `remote_commit_key`, and + `current_commit_pointer_key` for current cleanup behavior. + +### Verification And Exit Criteria + +Test no locks, exact/mixed carry, required locked absence, invalid prior state, +removed-after-initial lock, added-before-final lock, abandoned attempts, avoided +locked-source work, and exactly one pointer write. Run +`go test ./internal/stage -run Publish`, +`go test ./internal/runner ./internal/app`, `go test ./...`, and +`git diff --check`. Newly observed final locks prevent pointer advancement. + +## Stage 8 — Migrate Commands And Inspection + +**Status:** Pending + +### Goal + +Preserve the operator workflow over the independent-object model. + +### Required Work + +1. Rewrite lock list/add/remove services and CLI wiring to use artifact lock + operations while preserving command names and session selection. +2. List validated source-sorted locks. Add reports already locked unless + `--force`; force performs one conditional replacement. Remove reports + idempotent absence without error. Remove help/output references to + precedence and shared manifests. +3. Reduce command composition to storage, session prefix, source, reason, + clock where needed, and output. Remove config mutation and callbacks. +4. Make status/current-artifact inspection show five outcomes and identify + carried sources by exact key and derived run. Fetch only declared objects. +5. Add command tests for exit behavior, conflicts, idempotent remove, + deterministic output, malformed locks, and carried inspection. + +### Verification And Exit Criteria + +Run `go test ./internal/app ./internal/artifacts`, +`go test ./cmd/...`, and `git diff --check`. Operators retain simple, +accurate lock and inspection commands. + +## Stage 9 — Remove Retired Lock Authorities + +**Status:** Pending + +### Goal + +Delete the obsolete control plane after all consumers migrate. + +### Required Work + +1. Remove `pipeline.publish.locks`, `PublishLockRule`, central store types, + `locks.yml` helpers, YAML I/O, merge/precedence logic, document replacement + retries, injection fields, compatibility adapters, and dead tests/fixtures. +2. Do not retain migration readers. Remaining historical mentions must be + clearly intentional prose. +3. If effective-config digest inputs change, advance its internal discriminator + so prior successful state invalidates deliberately. Change no user-facing + schema version unless repository policy requires it. +4. Add a representative strict-decode test proving the old config field is + unknown, and search code/docs/examples/fixtures for all retired identifiers. + +### Verification And Exit Criteria + +Run: + +```sh +go test ./internal/config ./internal/artifacts ./internal/app ./internal/stage +go test ./... +git diff --check +``` + +Independent objects are the sole lock authority. + +## Stage 10 — Align Documentation And Close The Workstream + +**Status:** Pending + +### Goal + +Make the simplified implementation the fully documented repository state. + +### Required Work + +1. Update canonical config, storage, stage, CLI, operations, and development + docs for the payload/key, bounded validation, commands, carry behavior, race + boundary, five outcomes, cross-run rule, and transaction sequence. Remove + static-lock and `locks.yml` claims. +2. Document that old `locks.yml` is ignored and may be manually removed after + upgrade verification; add no migration tool. +3. Remove stale comments, redundant wrappers/logic, dead fixtures, and obsolete + metadata assertions exposed by final review. +4. Run all formatting, unit/integration, static-analysis, documentation, and + policy checks required by `docs/development.md`. Record genuinely + platform-only checks without weakening or silently skipping them. +5. Only after checks pass, mark this plan's stages and Workstream 8 completed. + Leave Workstream 9 as the next pending scope. + +### Verification And Exit Criteria + +At minimum run `gofmt` on changed Go, `go test ./...`, `go vet ./...`, +`go test ./internal/doccheck`, `git diff --check`, and all repository +scripts mandated by policy. Code, tests, docs, help, and examples agree, and no +Workstream 9 implementation is included. ## Open Questions -None. The audits, completed compatibility purge, current configuration -contracts, and fixed decisions above resolve the product and implementation -choices needed for a `gpt-5.6-terra` coding agent to execute this plan. +None. The audit decisions and fixed choices above make this plan +decision-complete.