eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 27 Wiki Activity

Plan remote publication refactoring

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-05 11:57:22 +00:00
parent a8b47f6185
commit 7f84fc5d1e
1 changed files with 308 additions and 446 deletions
Download Patch File Download Diff File Expand all files Collapse all files

754
docs/roadmap/implementation.md
View File

@@ -1,524 +1,386 @@
# Workstream 7 Implementation Plan — Configuration Structure And Diagnostics
# Workstream 8 Implementation Plan — Remote Publication And Artifact Locks
## Status
Completed implementation record for Workstream 7, **Configuration structure
and semantic diagnostics**, in [`refactoring.md`](refactoring.md). Workstreams
1 through 6 were prerequisites, and all stages below were completed in
numerical order.
Planned implementation for Workstream 8, **Remote publication, per-artifact
locks, and publish decomposition**, in [`refactoring.md`](refactoring.md).
Workstreams 1 through 7 are complete prerequisites. This plan does not enter
Workstream 9's cleanup-stage work.
The purpose of this workstream is to preserve Narratio's strict, deterministic,
operator-oriented configuration behavior while making the implementation
navigable by semantic domain. It also removes incidental composition metadata
and test expectations that turn private validation choreography into a
compatibility burden. This is an internal refactor with narrowly approved
diagnostic flexibility, not a configuration-language redesign.
## Authoritative Scope And Fixed Decisions
## Authoritative Scope And Decisions
Workstream 8 in [`refactoring.md`](refactoring.md) is authoritative. Its
supporting records are H04, H15, H18, and H44 in [`audit.md`](audit.md), C05
in [`contract-audit.md`](contract-audit.md), and their rows in
[`audit-decisions.md`](audit-decisions.md). Read those cited sections rather
than repeating the audits.
The authoritative scope is Workstream 7 in [`refactoring.md`](refactoring.md).
Its supporting audit entries are H04, H14, H15, H17, H38, H39, H40, H41, and
H42 in [`audit.md`](audit.md), C01 and C06 in
[`contract-audit.md`](contract-audit.md), and the corresponding rows in
[`audit-decisions.md`](audit-decisions.md).
The following decisions are final:
The following implementation decisions are fixed:
- Keep all configuration behavior in package `internal/config`. Do not create
subpackages, a generic configuration framework, a schema framework, a
validator registry, or a dependency-injection layer.
- Preserve the public configuration types and entry points. `Validate`,
`ValidatePipelineConfig`, `ValidateCampaignConfig`, all load APIs, effective
pipeline projections, and configuration-inspection commands retain their
current responsibilities.
- Keep `validate.go` as the short public and high-level orchestration owner.
Organize private validation into cohesive files for core pipeline/session,
adapters, publish/storage, Notarius, Scriptorium, and shared scalar/path
helpers. File names are private, but those ownership boundaries are the
intended final structure.
- Do not replace ordinary `error` returns with a new exported validation-error
hierarchy. No machine consumer currently requires one, and introducing one
would add a new compatibility surface. Field paths, source identities for
composition conflicts, wrapped causes where useful, and small semantic
phrases provide the stable diagnostic content tests should assert.
- Continue to fail fast. The workstream does not add error aggregation and does
not promise which of several independent invalid fields is reported first.
Traversal of maps must nevertheless be deterministic so identical input
produces identical behavior.
- Preserve strict single-document YAML decoding, duplicate- and unknown-field
rejection, additive imports with no duplicate ownership, one deterministic
profile selection, explicit-value overlay behavior, centralized defaults,
canonical party resolution, artifact-family expansion, and secret-free
effective digests.
- Preserve the current configuration schema. Campaign-owned
`campaign.inputs.party_file` and canonical artifact source IDs are current
contracts, not legacy fields. Removed player/session roster fields and old
artifact source spellings remain unsupported; add no compatibility aliases
or normalization branches for them.
- Treat Workstream 3's compatibility purge as authoritative. During this work,
remove any surviving production branch that actually accepts or translates
a retired configuration representation. A rejection supplied naturally by
strict decoding or the canonical source validator is sufficient; do not
maintain exhaustive legacy-form tests when representative canonical-boundary
tests already protect the rule.
- Preserve composition source information that is consumed by conflict
diagnostics or `config sources`. Remove only demonstrably unused node/field
metadata. In particular, source lists and semantic field paths remain;
redundant cached declaration-order indexes and copied line/column fields may
be removed when no consumer reads them because slice order and the YAML
parser already supply the required behavior.
- Preserve normalization as an all-or-nothing operation for each validator.
Notarius reference/output maps and scalar paths are installed on the config
only after all relevant checks succeed; a validation error must not leave a
partially normalized value behind.
- Keep `validateAudita` intact when moving it to the adapter-validation owner.
Its guards are cohesive; splitting them further would add names without
reducing complexity.
- Tests must protect semantic rules and observable normalized results, not
private helper calls, file placement, full error prose, punctuation, or one
historical validation call order. Exact strings remain appropriate only for
an intentionally documented CLI or machine-consumed contract.
- Do not enter Workstream 8 remote publication/lock decomposition, Workstream 9
cleanup-stage work, or unrelated Workstream 10 cleanup.
- Keep `publishStage.Run` as the transaction owner. Split cohesive work into
focused files within `internal/stage`; add no framework, distributed
transaction, or third-party dependency.
- Replace static pipeline locks and the shared remote YAML manifest with one
object per normalized stable source at
`{session_prefix}/locks/{sha256(source)}.lock`. The digest is lowercase
hexadecimal SHA-256 of the exact normalized source bytes.
- A lock is strict JSON with `format_version: 1`, `source`, optional
`reason`, and an RFC 3339 UTC `created_at`. Define
`MaxRemoteLockObjectBytes = 64 << 10` centrally in its package. Reject
unknown fields, trailing values, invalid/noncanonical fields, unsupported
versions, oversize objects, and payload/key hash mismatches.
- `internal/artifacts` owns the payload, identity/key rules, and lock access;
`internal/adapters/storage` owns object mechanics; `internal/app` owns
commands; and `internal/stage` owns publish use.
- Extend `storage.ObjectStore` with idempotent deletion. Listing validates
every reserved-prefix object and sorts by source. Add is conditional-create;
`--force` makes one generation-matched replacement; remove is idempotent.
- Ignore old `locks.yml` objects—do not migrate or delete them. Remove their
schema, merge/precedence/retry machinery, runtime injection, and static
`pipeline.publish.locks`. Strict config decoding rejects that old field.
- The initial lock snapshot controls planning. An unlocked selected output
publishes normally. A locked output carries its exact declaration from the
prior current commit, if any. A locked output never published remains absent
even if configured as required. A lock removed after the initial read may
therefore cause one conservative carry.
- After immutable objects and the immutable commit are written, reread every
initially unlocked relevant source. If any became locked, do not advance the
pointer. Attempt objects may remain. A lock created after this final check
may race the pointer write; do not add stronger transaction machinery.
- A carried `published_output` may reference an earlier run only within the
same campaign/session. Preserve source, key, checksum, size, and generation
exactly. Derive origin from its validated key; do not add an origin field or
copy old bytes. Verify existence, size, and generation without downloading.
- Publish metadata and inspection distinguish exactly `published`,
`carried_locked`, `locked_unpublished`, `optional_missing`, and
`unselected` in one source-sorted `publish_output_outcomes` collection.
Preserve upload counts and cleanup-consumed commit/pointer metadata until
Workstream 9.
- Preserve transaction order: immutable object upload/verification, immutable
commit, final lock revalidation, then exactly one current-pointer write.
## Instructions For Every Stage
Before each stage:
1. Read `docs/development.md`, every document under `docs/policy/`, this plan,
and Workstream 7 in [`refactoring.md`](refactoring.md). Read only the audit
sections and decision-register rows cited above; do not restart the audits.
2. Read [`../internal/configuration.md`](../internal/configuration.md) and the
affected portions of [`../config.md`](../config.md), then inspect the
production files and focused tests named by the stage.
3. Prefer the codebase knowledge graph for Go symbol discovery and call
tracing. Use text search for YAML fields, source IDs, error fragments,
examples, documentation, and removal verification.
4. Confirm the worktree state and preserve unrelated changes. Do not implement
an adjacent pending workstream merely because it shares a file.
1. Read `docs/development.md`, all documents under `docs/policy/`, this
plan, Workstream 8, its cited audit sections, and the affected sections of
[`../internal/stage-publish.md`](../internal/stage-publish.md),
[`../internal/storage.md`](../internal/storage.md),
[`../config.md`](../config.md), and
[`../operations.md`](../operations.md).
2. Inspect affected production code and focused tests. Prefer the codebase
knowledge graph for Go discovery/call tracing; use text search for schemas,
config, CLI text, docs, fixtures, and removal checks.
3. Confirm worktree state and preserve unrelated changes. Do not implement an
adjacent workstream.
During every stage:
During each stage, keep focused tests passing, use explicit package-private
helpers instead of generic managers/frameworks, keep reads bounded and bodies
closed, honor contexts, wrap errors with safe object identity, preserve
determinism, and keep tests offline. Run `gofmt` on changed Go files and
`git diff --check`. Commit only a completed stage with a short plain-English
message. Update documentation alongside observable changes.
- Keep the package compiling and focused tests passing. Move behavior before
redesigning the two complex validators so reviewable commits do not combine
unrelated mechanical and semantic changes.
- Use package-private functions with explicit inputs and return values. Avoid
stateful validator objects, forwarding-only wrappers, generic rule tables,
and abstractions whose only purpose is to reduce function length.
- Keep configuration mutation explicit and late. Do not mutate caller-owned
maps while validating a temporary normalized candidate.
- Retain deterministic order wherever config maps are traversed for validation,
normalization, canonical output, source reporting, or digest construction.
- Keep tests deterministic, offline, and independent of credentials or live
adapters. Prefer direct package behavior and real temporary files over mocks.
- Use `gofmt` on changed Go files, run the focused checks listed for the stage,
and run `git diff --check`. Mark a stage `Completed` only after its exit
criteria pass.
## Stage 1 — Add Idempotent Object Deletion
## Stage 1 — Separate Core And Shared Validation Ownership
**Status:** Completed
**Status:** Pending
### Goal
Reduce `validate.go` to configuration validation entry points and readable
high-level sequencing, while giving core pipeline/session rules and reusable
primitive checks clear package-local owners. This stage is behavior-preserving.
Add the sole storage primitive required to remove one independent lock.
### Required Work
1. Keep these entry points and the visible pipeline-domain call sequence in
`internal/config/validate.go`:
- `Validate`;
- `ValidatePipelineConfig`;
- `ValidateCampaignConfig`; and
- the short `validatePipeline` coordinator.
2. Move core configuration checks into a cohesive private file such as
`validate_core.go`. This owner includes campaign validation, secrets,
spool/cache, session validation, resolved-party input validation,
session/campaign identifiers, and cross-pipeline/session rules. Keep
`publishUploadConfiguredForS3` with publish/storage in Stage 2 rather than
hiding publish policy in the core owner.
3. Move general validation primitives into a small private file such as
`validate_helpers.go`: environment-variable name checks, positive duration
parsing, safe relative paths, and path-within-root checks. Do not turn these
functions into a generic validation DSL.
4. Re-home the straightforward stage/adapter validators in a private adapter
owner such as `validate_adapters.go`: notification, normalize, trim, render,
WhisperX, Seriatim (including its output schema), and Audita. Move
`validateAudita` intact apart from import or formatting changes.
5. Preserve current nil handling, defaults, normalization, wrapped causes,
field identification, and high-level validation semantics. Do not use this
mechanical move to reorder domains or rewrite diagnostics.
6. Remove imports and dead private helpers left behind in `validate.go`. Do not
add facade functions to preserve former source-file placement; files are not
Go API boundaries.
1. Add `Delete(ctx, key) error` to `storage.ObjectStore`. Absence succeeds;
preserve existing key validation and contextual errors.
2. Extend the narrow S3 client port and backend using the provider delete-object
operation. Propagate cancellation, permission, transport, and other errors.
3. Update every fake/implementation. The standard fake removes object and
generation state, is idempotent, honors context, and permits injected-error
tests consistently with its other operations.
4. Add no batch, recursive, or lock-specific storage API.
### Tests And Verification
### Verification And Exit Criteria
- Existing core, campaign, session, duration, normalize, trim, render,
notification, and load/validate tests should normally pass unchanged.
- Add no tests that merely assert a function moved files. Add or repair a test
only if the move exposes an unprotected semantic rule.
- Run:
Test success, absence, cancellation, backend failures, and S3 request mapping.
Run `go test ./internal/adapters/storage`,
`go test ./internal/artifacts ./internal/stage ./internal/app`, and
`git diff --check`. All implementations satisfy the narrow idempotent port.
```sh
go test ./internal/config
go test ./internal/app -run 'Config|Pipeline|Session'
git diff --check
```
## Stage 2 — Define Independent Lock Objects
### Exit Criteria
- `validate.go` is a concise entry/coordinator file rather than a container for
unrelated domain implementations.
- Core, adapter, and shared helper ownership is obvious from the file layout,
and Audita remains one cohesive validator.
- No public behavior or serialized configuration shape changed.
## Stage 2 — Separate Publish, Storage, Notarius, And Scriptorium Owners
**Status:** Completed
**Status:** Pending
### Goal
Finish the domain-level split of the former aggregate validator before changing
the internal structure of Notarius or Scriptorium validation.
Establish the complete lock contract before migrating callers.
### Required Work
1. Move storage, publish output, and publish lock validation into a cohesive
private owner such as `validate_publish.go`. Move the associated helpers
with their consumers, including `ValidatePublishLockRules`,
`MergePublishLockRules`, configured output/source lookups, and
`publishUploadConfiguredForS3`.
2. Preserve the exported lock-rule helper signatures because current app and
stage code consume them. Do not redesign remote lock behavior; that belongs
to Workstream 8.
3. Move the existing Notarius validator into `validate_notarius.go` and its
Scriptorium counterpart, input-source validator, and exported dependency
validator into `validate_scriptorium.go`. At this stage, move these functions
without decomposing or semantically reordering them; Stages 3 and 4 own those
changes.
4. Keep source-vocabulary and collision helpers with the narrowest shared
domain owner. If both publish and Scriptorium need a small lookup derived
from Notarius/Scriptorium configuration, use one plainly named package
helper rather than duplicate maps or introduce a registry.
5. Ensure `validate.go` contains only entry points and high-level sequencing
after the move. Delete the emptied blocks and unused imports.
6. Search `internal/config` production code for acceptance or translation of
removed player/session roster fields and retired artifact source spellings.
Delete any actual compatibility branch found. Do not mistake current
campaign `party_file`, rejection through strict decoding, migration prose,
or a test fixture demonstrating rejection for an active compatibility path.
1. Add a cohesive `internal/artifacts` owner for payloads, size bound, canonical
source validation, key construction, and strict encoding/decoding.
2. Reuse publish's canonical stable-source vocabulary. Reject noncanonical
caller input rather than making a second identity. Validate the session
prefix before joining the reserved `locks/` path.
3. Implement exact lookup and prefix listing. Exact lookup returns an explicit
present/absent result. Listing bounds every read, rejects unexpected names,
verifies hashes, rejects duplicate sources, and sorts by source.
4. Implement conditional creation, one generation-matched forced replacement,
and idempotent removal. A replacement conflict is reported for operator
retry; do not add an internal retry loop.
5. Do not consult static config or `locks.yml`.
### Tests And Verification
### Verification And Exit Criteria
- Existing storage/publish, Notarius, Scriptorium, configuration loading, and
application configuration tests should pass without pinning file placement.
- Run:
Table-test canonical keys, strict JSON, size limits, hash mismatches, absence,
invalid prefix entries, ordering, add conflicts, force conflicts/replacement,
and removal. Run `go test ./internal/artifacts ./internal/adapters/storage` and
`git diff --check`. Each lock is independently manageable and validated.
```sh
go test ./internal/config
go test ./internal/app ./internal/stage
git diff --check
```
## Stage 3 — Permit Safe Cross-Run Declarations
### Exit Criteria
- Every validation domain named by Workstream 7 has one discoverable file-level
owner inside `internal/config`.
- There is no duplicate source/collision logic and no active retired-format
compatibility branch in configuration production code.
- Notarius and Scriptorium semantics remain unchanged pending their dedicated
stages.
## Stage 3 — Decompose Notarius Validation And Normalization
**Status:** Completed
**Status:** Pending
### Goal
Make `validateNotarius` a short transaction that validates top-level policy,
normalizes references, normalizes output lanes, and commits the resulting
configuration only after every check succeeds.
Permit exact older published declarations without weakening session isolation.
### Required Work
1. Keep `validateNotarius` responsible for nil/enablement behavior and the
top-level scalar contract: required binary, configuration path, pipeline ID,
positive timeout, working directory, at least one output, and the reference
binding limit.
2. Replace the validator's bespoke timeout parsing with the shared positive
duration helper when it preserves the same accepted values and field
identity. Do not embed duration policy in a Notarius-only branch.
3. Extract reference binding normalization into one private operation that:
- iterates raw selector keys in sorted order;
- normalizes selectors through `notariusref`;
- rejects normalized selector collisions;
- trims and validates each prepared-input source; and
- returns a newly allocated normalized map without mutating the input map.
4. Extract output normalization and collision validation into a separate
private operation that:
- builds the reserved built-in, configured Scriptorium, and previous-session
source set once;
- iterates raw output keys in sorted order;
- validates and normalizes configured keys and derived extraction source
IDs;
- rejects reserved-source, normalized-source, and lane-ID collisions;
- trims lane/media/schema/module scalar fields and checks required fields;
and
- returns a newly allocated normalized output map.
5. Normalize binary, paths, pipeline ID, timeout, and working directory into
local candidate values. Assign all scalar candidates, references, and
outputs back to `cfg` only after both helper operations succeed. Preserve
nil-versus-empty map behavior where it is observable in effective YAML or
digests.
6. Keep all helpers Notarius-specific and private. Do not move selector policy
out of `notariusref`, copy artifact source policy, or create a generalized
map-normalization framework.
1. Add a small key parser proving an immutable object belongs to the expected
campaign/session and extracting its run. Reject malformed, mutable,
traversal, cross-session, and empty-run keys.
2. Relax commit validation only for `published_output`: its key may use the
current or an earlier run in the same session. Retain every other placement
rule and the existing manifest version unless serialization changes.
3. Update restore, previous-state caching, current-output lookup, and
status/artifact inspection to open the exact declared key and match by stable
source instead of reconstructing a current-run path. Never scan prefixes.
4. Preserve duplicate-source and destination-collision enforcement when
carried and current declarations coexist; derive visible origin from key.
### Tests And Verification
### Verification And Exit Criteria
- Retain lean behavior tests for disabled configuration, required scalars,
timeout and reference-binding limits, selector normalization/collisions, supported
prepared sources, output normalization, reserved-source collisions,
duplicate lanes, and cross-use by Scriptorium/publish.
- Add or consolidate one test proving a failing reference or output candidate
leaves the original Notarius config unmodified.
- Add one determinism test that constructs equivalent reference/output maps in
different insertion orders and compares normalized success or equivalent
diagnostic classification. Do not assert which of multiple independent
invalid domains must win forever.
- Assert field paths and the smallest meaningful semantic fragment rather than
complete error sentences.
- Run:
Test same/earlier-run acceptance, malformed/mutable/cross-session rejection,
exact-key reads, and source-based inspection. Run
`go test ./internal/artifacts ./internal/app ./internal/stage` and
`git diff --check`. Only safe same-session published-output carry is enabled.
```sh
go test ./internal/config -run 'Notarius|Scriptorium|Publish|Duration'
go test ./internal/stage -run 'Extract|Analyze'
git diff --check
```
## Stage 4 — Separate Source And Output Planning
### Exit Criteria
- `validateNotarius` visibly sequences scalar validation, reference
normalization, output normalization/collision checks, and one final commit.
- Reference and output collision domains are separate, deterministic, and do
not partially mutate caller-owned configuration on failure.
- No Notarius behavior moved into a generic configuration abstraction.
## Stage 4 — Decompose Deterministic Scriptorium Validation
**Status:** Completed
**Status:** Pending
### Goal
Separate Scriptorium artifact declarations/outputs, dependencies/inputs, and
variables into readable validation phases with deterministic map traversal.
Make source opening and output-rule resolution navigable without changing
publish behavior.
### Required Work
1. Keep `validateScriptorium` as the transaction owner for nil behavior,
required binary, optional config path, stage timeout, ordered validation
phases, referenced-output checks, and the final dependency-graph check.
2. Produce one sorted list of configured artifact keys after validating their
canonical key syntax. Reuse that order across phases rather than ranging
over the artifact map independently.
3. Extract a declaration/output phase that validates enabled prompt and output
requirements, safe output paths, artifact timeout values, and duplicate
output ownership. Keep its collision map local to that phase.
4. Extract a dependency/input phase that:
- validates dependency entries and self-reference using declared slice
indexes for actionable field paths;
- iterates input maps by sorted input name;
- validates required source declarations through the existing canonical
input-source owner;
- requires configured-artifact sources to appear in `depends_on`; and
- returns or updates only the small referenced-artifact set needed by the
later referenced-output check.
5. Extract a variable phase that iterates variable names in sorted order,
rejects empty names, and accepts only the currently supported string and
boolean values.
6. Iterate the referenced-artifact set deterministically when checking required
outputs. Keep `ValidateScriptoriumArtifactDependencies` as the public graph
validator and preserve its cycle and dependency semantics.
7. It is acceptable for phase separation to change which of several unrelated
simultaneous mistakes is reported first. Within a given implementation,
equivalent maps must produce the same result. Do not add compatibility code
to reproduce the old mixed per-artifact call order.
8. Remove forwarding-only helpers and obsolete temporary sets after the split.
Keep `validateScriptoriumInputSource` cohesive unless a genuine duplicate
semantic owner emerges.
1. Move source discovery, metadata construction, safe opening, and validation
from the aggregate publish file into one cohesive package-local owner.
2. Move output selection, source matching, required/optional rules, destination
resolution, and collisions into another cohesive owner.
3. Keep `publishStage.Run` as caller/transaction owner with explicit inputs;
avoid a mutable catch-all context. Do not integrate locks yet.
4. Remove only dead helpers/imports exposed by the mechanical split.
### Tests And Verification
### Verification And Exit Criteria
- Retain semantic coverage for declaration keys, enabled requirements, output
path safety/collisions, timeouts, dependency existence/self/cycles, input
source classes, extraction references, required `depends_on`, referenced
outputs, and variable types.
- Add one map-insertion-order test that feeds equivalent Scriptorium maps in
different insertion orders and asserts equal validity or equivalent
diagnostics. Do not assert an arbitrary globally preferred independent-error
order.
- Consolidate table cases that repeat the same source-vocabulary rule through
different retired spellings when a representative canonical rejection test
already protects unsupported-source behavior.
- Run:
Run `go test ./internal/stage -run Publish`,
`go test ./internal/app -run 'Publish|Artifact|Status'`, and
`git diff --check`. Existing behavior remains, with clear source/output
file-level ownership.
```sh
go test ./internal/config -run 'Scriptorium|ArtifactFamil|Notarius|Publish'
go test ./internal/stage -run 'Analyze'
go test ./internal/app -run 'Artifact|Config|Plan'
git diff --check
```
## Stage 5 — Separate Transfer And Commit Mechanics
### Exit Criteria
- Scriptorium validation has explicit declaration/output,
dependency/input, variable, and graph phases.
- All map traversal that can affect validation or normalization is
deterministic.
- Tests protect fields and semantic rules without freezing the former mixed
validator choreography.
## Stage 5 — Split Composition Envelope Phases And Remove Incidental Metadata
**Status:** Completed
**Status:** Pending
### Goal
Separate composition-envelope shape decoding, cross-field semantics, and
root-only metadata removal while retaining the source-aware configuration model
that imports, profiles, diagnostics, and inspection commands require.
Finish decomposition while keeping the transaction visible in one place.
### Required Work
1. Refactor `splitPipelineCompositionEnvelope` in
`internal/config/pipeline_composition.go` into a short coordinator over
three explicit private phases:
- locate and decode the root `composition` mapping and its `imports`,
`default_profile`, and `profiles` shapes;
- validate cross-field semantics, including default-profile membership; and
- clone/remove the root-only envelope to produce the ordinary pipeline body.
2. Reuse `parsePipelineProfileDeclaration` and profile-name normalization.
Keep unknown-field rejection, non-empty profile declarations, imports and
profiles in declaration order, explicit-profile selection behavior, and
source-qualified shape/conflict diagnostics.
3. Do not mutate the parsed root document during validation. Return a distinct
body document only after the envelope is valid; the original document must
remain reusable by callers and tests.
4. Remove `compositionNode` and `compositionField` metadata that has no runtime
reader after the refactor. Specifically verify and remove redundant cached
`order`, `line`, and `column` fields and their clone/renumber assignments if
they remain write-only. Slice order remains the declaration-order
representation; YAML parser nodes remain the source of parse-time line and
column diagnostics.
5. Retain each node's semantic path and source list because merge conflict
reporting, ownership projection, effective digests, artifact-family source
projection, and `config sources` consume them. Do not reduce actionable
conflict diagnostics or delete provenance that has a real consumer.
6. Keep composition product-specific. Do not replace the tree with generic map
decoding or adopt a third-party composition library.
1. Move copying/hashing, immutable upload, remote verification, and reusable
prior-object verification into a transfer owner.
2. Move immutable commit and current-pointer construction, encoding, and writes
into a commit owner.
3. Keep one explicit ordered path in `publishStage.Run`: plan, transfer,
immutable commit, pre-pointer checks, pointer write. Document its durability
boundary briefly.
4. Delete forwarding wrappers and dead helpers; do not add package-local
interfaces without a second implementation.
### Tests And Verification
### Verification And Exit Criteria
- Test malformed envelope shapes, unknown fields, invalid profile declarations,
missing default membership, successful envelope removal, and absence of
caller-document mutation through the composition boundary.
- Preserve tests for explicit empty/false/zero/list/map values, additive
conflicts naming all claiming sources, overlay semantics, canonical YAML,
effective digest stability, profile selection, and `config sources`
ownership. These are semantic contracts, not incidental provenance.
- Remove assertions of deleted private cached metadata. Continue asserting
declaration behavior through ordered outputs rather than private index
fields.
- Run:
Preserve checksum, size, generation, conditional-write, cancellation, and
partial-failure tests. Run `go test ./internal/stage -run Publish`,
`go test ./internal/artifacts ./internal/adapters/storage`, and
`git diff --check`. The transaction remains easy to audit from `Run`.
```sh
go test ./internal/config -run 'Composition|PipelineProfile|Effective|ArtifactFamil'
go test ./internal/app -run 'Config'
git diff --check
```
## Stage 6 — Model Locked Output Carry-Forward
### Exit Criteria
- Envelope decoding, cross-field validation, and body production have separate
focused owners under one coordinator.
- Only provenance and ordering data with a demonstrated consumer remains.
- Imports, profiles, overlays, source reporting, and effective digests retain
deterministic behavior.
## Stage 6 — Align Diagnostic Tests, Documentation, And Close Workstream 7
**Status:** Completed
**Status:** Pending
### Goal
Review the completed configuration refactor as one system, remove obsolete test
friction and documentation language, and close Workstream 7 with repository-wide
evidence.
Represent every output outcome and build a coherent commit plan without I/O.
### Required Work
1. Review every configuration test changed by Stages 1–5. Retain coverage of
schema strictness, normalization, semantic invariants, meaningful failure
categories, deterministic outcomes, composition source conflicts, effective
projections/digests, and representative assembled loads.
2. Rewrite assertions that pin complete diagnostic prose, punctuation,
formatting, private helper order, or an arbitrary independent-error winner.
Prefer a relevant field path plus one small semantic fragment. Use
`errors.Is`/`errors.As` only where an existing wrapped or typed cause is a
real contract; do not introduce types solely to change test style.
3. Delete redundant legacy-form enumerations whose only purpose is to preserve
knowledge of retired fields or spellings already rejected by the canonical
strict/source boundary. Retain representative tests proving unknown fields
and unsupported sources fail, and retain migration guidance tests only when
that guidance is an intentional user-facing contract.
4. Consolidate oversized table fixtures only where multiple cases protect the
same rule. Do not delete distinct risks merely to reduce test LOC, and do not
rewrite healthy semantic tests just because they use a small stable error
fragment.
5. Update [`../internal/configuration.md`](../internal/configuration.md) to
describe the implemented validation-domain ownership, deterministic phase
behavior, composition-envelope phases, and diagnostic guarantees without
listing private helper choreography. Update [`../config.md`](../config.md)
only if its user-visible current-behavior wording is inaccurate; no schema
change is expected.
6. Search current production, tests, examples, and current-behavior docs for
active acceptance of the retired configuration forms named by C01. Remove
actual compatibility code and stale current-behavior claims, while leaving
immutable audit/history documents unchanged.
7. Run the full repository validation from `docs/development.md`:
1. Add one private output-plan/result type using the five fixed statuses. A
participating output is a configured rule selected for this invocation;
unselected rules are reported but do not enforce required-output presence.
2. Plan deterministically by source from selected rules, discovered sources,
initial locks, and prior current commit.
3. Publish unlocked participants normally. Carry a locked prior declaration
byte-for-field. Emit `locked_unpublished` without required-output failure
when none exists. Preserve optional-missing and unselected separately.
4. Reject duplicate prior sources, invalid declarations, configured destination
conflicts, and effective destination collisions. Never synthesize an old
declaration.
5. Project one source-sorted metadata collection with source/status,
destination, exact declared key, and derived origin where applicable; omit
secrets and local absolute paths.
```sh
go test ./...
go test -race ./...
go vet ./...
go build ./...
go test ./internal/doccheck
go test ./internal/config -run '^TestExamplesLoadAndValidate$'
```
### Verification And Exit Criteria
8. Run `go test -race -shuffle=on -count=3 ./internal/config ./internal/app`
to stress map-order independence and assembled configuration behavior.
9. After all checks pass, mark Stages 1–6 in this document `Completed` and mark
only Workstream 7 in [`refactoring.md`](refactoring.md) `Complete`.
Workstream 8 must remain the next pending workstream.
Pure table tests cover five states, exact carry, required locked absence, mixed
snapshots, selection, ordering, and ambiguities. Run
`go test ./internal/stage -run 'Publish|Output'` and `git diff --check`.
One plan drives both declarations and metadata.
### Exit Criteria
## Stage 7 — Integrate Locks Into Publish
- Configuration remains strict, deterministic, source-aware where useful, and
unchanged in public schema and normalized semantics.
- Validation code is organized by semantic owner; Notarius, Scriptorium, and
composition coordinators expose clear phases without generic frameworks.
- No active retired configuration compatibility path, write-only composition
metadata, or test dependency on incidental diagnostic choreography remains.
- Current documentation describes the implemented boundaries, all validation
commands pass, Workstream 7 is complete, and Workstream 8 remains pending.
**Status:** Pending
### Goal
Make independent locks authoritative while preserving the snapshot transaction.
### Required Work
1. Read relevant locks at planning start and load/validate the prior current
commit when a locked source may require carry-forward.
2. Use Stage 6's plan. Do not open, hash, or upload a carried or never-published
locked source.
3. Before carrying, verify the exact prior object exists and matches declared
size and generation without download; retain its declared checksum.
4. Upload/verify new objects, combine exact carried/new declarations, validate
and write the immutable commit, then reread initially unlocked sources. If
any is now locked, return a clear error without pointer write; otherwise
write the pointer exactly once.
5. Remove `Env.RevalidatePublishLocks`, runner injection, and related plumbing
once unused. Publish reads authoritative locks through artifact/storage.
6. Replace overlapping category arrays with `publish_output_outcomes`.
Preserve upload counts, `remote_commit_key`, and
`current_commit_pointer_key` for current cleanup behavior.
### Verification And Exit Criteria
Test no locks, exact/mixed carry, required locked absence, invalid prior state,
removed-after-initial lock, added-before-final lock, abandoned attempts, avoided
locked-source work, and exactly one pointer write. Run
`go test ./internal/stage -run Publish`,
`go test ./internal/runner ./internal/app`, `go test ./...`, and
`git diff --check`. Newly observed final locks prevent pointer advancement.
## Stage 8 — Migrate Commands And Inspection
**Status:** Pending
### Goal
Preserve the operator workflow over the independent-object model.
### Required Work
1. Rewrite lock list/add/remove services and CLI wiring to use artifact lock
operations while preserving command names and session selection.
2. List validated source-sorted locks. Add reports already locked unless
`--force`; force performs one conditional replacement. Remove reports
idempotent absence without error. Remove help/output references to
precedence and shared manifests.
3. Reduce command composition to storage, session prefix, source, reason,
clock where needed, and output. Remove config mutation and callbacks.
4. Make status/current-artifact inspection show five outcomes and identify
carried sources by exact key and derived run. Fetch only declared objects.
5. Add command tests for exit behavior, conflicts, idempotent remove,
deterministic output, malformed locks, and carried inspection.
### Verification And Exit Criteria
Run `go test ./internal/app ./internal/artifacts`,
`go test ./cmd/...`, and `git diff --check`. Operators retain simple,
accurate lock and inspection commands.
## Stage 9 — Remove Retired Lock Authorities
**Status:** Pending
### Goal
Delete the obsolete control plane after all consumers migrate.
### Required Work
1. Remove `pipeline.publish.locks`, `PublishLockRule`, central store types,
`locks.yml` helpers, YAML I/O, merge/precedence logic, document replacement
retries, injection fields, compatibility adapters, and dead tests/fixtures.
2. Do not retain migration readers. Remaining historical mentions must be
clearly intentional prose.
3. If effective-config digest inputs change, advance its internal discriminator
so prior successful state invalidates deliberately. Change no user-facing
schema version unless repository policy requires it.
4. Add a representative strict-decode test proving the old config field is
unknown, and search code/docs/examples/fixtures for all retired identifiers.
### Verification And Exit Criteria
Run:
```sh
go test ./internal/config ./internal/artifacts ./internal/app ./internal/stage
go test ./...
git diff --check
```
Independent objects are the sole lock authority.
## Stage 10 — Align Documentation And Close The Workstream
**Status:** Pending
### Goal
Make the simplified implementation the fully documented repository state.
### Required Work
1. Update canonical config, storage, stage, CLI, operations, and development
docs for the payload/key, bounded validation, commands, carry behavior, race
boundary, five outcomes, cross-run rule, and transaction sequence. Remove
static-lock and `locks.yml` claims.
2. Document that old `locks.yml` is ignored and may be manually removed after
upgrade verification; add no migration tool.
3. Remove stale comments, redundant wrappers/logic, dead fixtures, and obsolete
metadata assertions exposed by final review.
4. Run all formatting, unit/integration, static-analysis, documentation, and
policy checks required by `docs/development.md`. Record genuinely
platform-only checks without weakening or silently skipping them.
5. Only after checks pass, mark this plan's stages and Workstream 8 completed.
Leave Workstream 9 as the next pending scope.
### Verification And Exit Criteria
At minimum run `gofmt` on changed Go, `go test ./...`, `go vet ./...`,
`go test ./internal/doccheck`, `git diff --check`, and all repository
scripts mandated by policy. Code, tests, docs, help, and examples agree, and no
Workstream 9 implementation is included.
## Open Questions
None. The audits, completed compatibility purge, current configuration
contracts, and fixed decisions above resolve the product and implementation
choices needed for a `gpt-5.6-terra` coding agent to execute this plan.
None. The audit decisions and fixed choices above make this plan
decision-complete.