eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 27 Wiki Activity

Plan remote publication refactoring

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-05 11:57:22 +00:00
parent a8b47f6185
commit 7f84fc5d1e
1 changed files with 308 additions and 446 deletions
Download Patch File Download Diff File Expand all files Collapse all files

748
docs/roadmap/implementation.md
View File

@@ -1,524 +1,386 @@
# Workstream 7 Implementation Plan — Configuration Structure And Diagnostics # Workstream 8 Implementation Plan — Remote Publication And Artifact Locks
## Status ## Status
Completed implementation record for Workstream 7, **Configuration structure Planned implementation for Workstream 8, **Remote publication, per-artifact
and semantic diagnostics**, in [`refactoring.md`](refactoring.md). Workstreams locks, and publish decomposition**, in [`refactoring.md`](refactoring.md).
1 through 6 were prerequisites, and all stages below were completed in Workstreams 1 through 7 are complete prerequisites. This plan does not enter
numerical order. Workstream 9's cleanup-stage work.
The purpose of this workstream is to preserve Narratio's strict, deterministic, ## Authoritative Scope And Fixed Decisions
operator-oriented configuration behavior while making the implementation
navigable by semantic domain. It also removes incidental composition metadata
and test expectations that turn private validation choreography into a
compatibility burden. This is an internal refactor with narrowly approved
diagnostic flexibility, not a configuration-language redesign.
## Authoritative Scope And Decisions Workstream 8 in [`refactoring.md`](refactoring.md) is authoritative. Its
supporting records are H04, H15, H18, and H44 in [`audit.md`](audit.md), C05
in [`contract-audit.md`](contract-audit.md), and their rows in
[`audit-decisions.md`](audit-decisions.md). Read those cited sections rather
than repeating the audits.
The authoritative scope is Workstream 7 in [`refactoring.md`](refactoring.md). The following decisions are final:
Its supporting audit entries are H04, H14, H15, H17, H38, H39, H40, H41, and
H42 in [`audit.md`](audit.md), C01 and C06 in
[`contract-audit.md`](contract-audit.md), and the corresponding rows in
[`audit-decisions.md`](audit-decisions.md).
The following implementation decisions are fixed: - Keep `publishStage.Run` as the transaction owner. Split cohesive work into
focused files within `internal/stage`; add no framework, distributed
- Keep all configuration behavior in package `internal/config`. Do not create transaction, or third-party dependency.
subpackages, a generic configuration framework, a schema framework, a - Replace static pipeline locks and the shared remote YAML manifest with one
validator registry, or a dependency-injection layer. object per normalized stable source at
- Preserve the public configuration types and entry points. `Validate`, `{session_prefix}/locks/{sha256(source)}.lock`. The digest is lowercase
`ValidatePipelineConfig`, `ValidateCampaignConfig`, all load APIs, effective hexadecimal SHA-256 of the exact normalized source bytes.
pipeline projections, and configuration-inspection commands retain their - A lock is strict JSON with `format_version: 1`, `source`, optional
current responsibilities. `reason`, and an RFC 3339 UTC `created_at`. Define
- Keep `validate.go` as the short public and high-level orchestration owner. `MaxRemoteLockObjectBytes = 64 << 10` centrally in its package. Reject
Organize private validation into cohesive files for core pipeline/session, unknown fields, trailing values, invalid/noncanonical fields, unsupported
adapters, publish/storage, Notarius, Scriptorium, and shared scalar/path versions, oversize objects, and payload/key hash mismatches.
helpers. File names are private, but those ownership boundaries are the - `internal/artifacts` owns the payload, identity/key rules, and lock access;
intended final structure. `internal/adapters/storage` owns object mechanics; `internal/app` owns
- Do not replace ordinary `error` returns with a new exported validation-error commands; and `internal/stage` owns publish use.
hierarchy. No machine consumer currently requires one, and introducing one - Extend `storage.ObjectStore` with idempotent deletion. Listing validates
would add a new compatibility surface. Field paths, source identities for every reserved-prefix object and sorts by source. Add is conditional-create;
composition conflicts, wrapped causes where useful, and small semantic `--force` makes one generation-matched replacement; remove is idempotent.
phrases provide the stable diagnostic content tests should assert. - Ignore old `locks.yml` objects—do not migrate or delete them. Remove their
- Continue to fail fast. The workstream does not add error aggregation and does schema, merge/precedence/retry machinery, runtime injection, and static
not promise which of several independent invalid fields is reported first. `pipeline.publish.locks`. Strict config decoding rejects that old field.
Traversal of maps must nevertheless be deterministic so identical input - The initial lock snapshot controls planning. An unlocked selected output
produces identical behavior. publishes normally. A locked output carries its exact declaration from the
- Preserve strict single-document YAML decoding, duplicate- and unknown-field prior current commit, if any. A locked output never published remains absent
rejection, additive imports with no duplicate ownership, one deterministic even if configured as required. A lock removed after the initial read may
profile selection, explicit-value overlay behavior, centralized defaults, therefore cause one conservative carry.
canonical party resolution, artifact-family expansion, and secret-free - After immutable objects and the immutable commit are written, reread every
effective digests. initially unlocked relevant source. If any became locked, do not advance the
- Preserve the current configuration schema. Campaign-owned pointer. Attempt objects may remain. A lock created after this final check
`campaign.inputs.party_file` and canonical artifact source IDs are current may race the pointer write; do not add stronger transaction machinery.
contracts, not legacy fields. Removed player/session roster fields and old - A carried `published_output` may reference an earlier run only within the
artifact source spellings remain unsupported; add no compatibility aliases same campaign/session. Preserve source, key, checksum, size, and generation
or normalization branches for them. exactly. Derive origin from its validated key; do not add an origin field or
- Treat Workstream 3's compatibility purge as authoritative. During this work, copy old bytes. Verify existence, size, and generation without downloading.
remove any surviving production branch that actually accepts or translates - Publish metadata and inspection distinguish exactly `published`,
a retired configuration representation. A rejection supplied naturally by `carried_locked`, `locked_unpublished`, `optional_missing`, and
strict decoding or the canonical source validator is sufficient; do not `unselected` in one source-sorted `publish_output_outcomes` collection.
maintain exhaustive legacy-form tests when representative canonical-boundary Preserve upload counts and cleanup-consumed commit/pointer metadata until
tests already protect the rule. Workstream 9.
- Preserve composition source information that is consumed by conflict - Preserve transaction order: immutable object upload/verification, immutable
diagnostics or `config sources`. Remove only demonstrably unused node/field commit, final lock revalidation, then exactly one current-pointer write.
metadata. In particular, source lists and semantic field paths remain;
redundant cached declaration-order indexes and copied line/column fields may
be removed when no consumer reads them because slice order and the YAML
parser already supply the required behavior.
- Preserve normalization as an all-or-nothing operation for each validator.
Notarius reference/output maps and scalar paths are installed on the config
only after all relevant checks succeed; a validation error must not leave a
partially normalized value behind.
- Keep `validateAudita` intact when moving it to the adapter-validation owner.
Its guards are cohesive; splitting them further would add names without
reducing complexity.
- Tests must protect semantic rules and observable normalized results, not
private helper calls, file placement, full error prose, punctuation, or one
historical validation call order. Exact strings remain appropriate only for
an intentionally documented CLI or machine-consumed contract.
- Do not enter Workstream 8 remote publication/lock decomposition, Workstream 9
cleanup-stage work, or unrelated Workstream 10 cleanup.
## Instructions For Every Stage ## Instructions For Every Stage
Before each stage: Before each stage:
1. Read `docs/development.md`, every document under `docs/policy/`, this plan, 1. Read `docs/development.md`, all documents under `docs/policy/`, this
and Workstream 7 in [`refactoring.md`](refactoring.md). Read only the audit plan, Workstream 8, its cited audit sections, and the affected sections of
sections and decision-register rows cited above; do not restart the audits. [`../internal/stage-publish.md`](../internal/stage-publish.md),
2. Read [`../internal/configuration.md`](../internal/configuration.md) and the [`../internal/storage.md`](../internal/storage.md),
affected portions of [`../config.md`](../config.md), then inspect the [`../config.md`](../config.md), and
production files and focused tests named by the stage. [`../operations.md`](../operations.md).
3. Prefer the codebase knowledge graph for Go symbol discovery and call 2. Inspect affected production code and focused tests. Prefer the codebase
tracing. Use text search for YAML fields, source IDs, error fragments, knowledge graph for Go discovery/call tracing; use text search for schemas,
examples, documentation, and removal verification. config, CLI text, docs, fixtures, and removal checks.
4. Confirm the worktree state and preserve unrelated changes. Do not implement 3. Confirm worktree state and preserve unrelated changes. Do not implement an
an adjacent pending workstream merely because it shares a file. adjacent workstream.
During every stage: During each stage, keep focused tests passing, use explicit package-private
helpers instead of generic managers/frameworks, keep reads bounded and bodies
closed, honor contexts, wrap errors with safe object identity, preserve
determinism, and keep tests offline. Run `gofmt` on changed Go files and
`git diff --check`. Commit only a completed stage with a short plain-English
message. Update documentation alongside observable changes.
- Keep the package compiling and focused tests passing. Move behavior before ## Stage 1 — Add Idempotent Object Deletion
redesigning the two complex validators so reviewable commits do not combine
unrelated mechanical and semantic changes.
- Use package-private functions with explicit inputs and return values. Avoid
stateful validator objects, forwarding-only wrappers, generic rule tables,
and abstractions whose only purpose is to reduce function length.
- Keep configuration mutation explicit and late. Do not mutate caller-owned
maps while validating a temporary normalized candidate.
- Retain deterministic order wherever config maps are traversed for validation,
normalization, canonical output, source reporting, or digest construction.
- Keep tests deterministic, offline, and independent of credentials or live
adapters. Prefer direct package behavior and real temporary files over mocks.
- Use `gofmt` on changed Go files, run the focused checks listed for the stage,
and run `git diff --check`. Mark a stage `Completed` only after its exit
criteria pass.
## Stage 1 — Separate Core And Shared Validation Ownership **Status:** Pending
**Status:** Completed
### Goal ### Goal
Reduce `validate.go` to configuration validation entry points and readable Add the sole storage primitive required to remove one independent lock.
high-level sequencing, while giving core pipeline/session rules and reusable
primitive checks clear package-local owners. This stage is behavior-preserving.
### Required Work ### Required Work
1. Keep these entry points and the visible pipeline-domain call sequence in 1. Add `Delete(ctx, key) error` to `storage.ObjectStore`. Absence succeeds;
`internal/config/validate.go`: preserve existing key validation and contextual errors.
- `Validate`; 2. Extend the narrow S3 client port and backend using the provider delete-object
- `ValidatePipelineConfig`; operation. Propagate cancellation, permission, transport, and other errors.
- `ValidateCampaignConfig`; and 3. Update every fake/implementation. The standard fake removes object and
- the short `validatePipeline` coordinator. generation state, is idempotent, honors context, and permits injected-error
2. Move core configuration checks into a cohesive private file such as tests consistently with its other operations.
`validate_core.go`. This owner includes campaign validation, secrets, 4. Add no batch, recursive, or lock-specific storage API.
spool/cache, session validation, resolved-party input validation,
session/campaign identifiers, and cross-pipeline/session rules. Keep
`publishUploadConfiguredForS3` with publish/storage in Stage 2 rather than
hiding publish policy in the core owner.
3. Move general validation primitives into a small private file such as
`validate_helpers.go`: environment-variable name checks, positive duration
parsing, safe relative paths, and path-within-root checks. Do not turn these
functions into a generic validation DSL.
4. Re-home the straightforward stage/adapter validators in a private adapter
owner such as `validate_adapters.go`: notification, normalize, trim, render,
WhisperX, Seriatim (including its output schema), and Audita. Move
`validateAudita` intact apart from import or formatting changes.
5. Preserve current nil handling, defaults, normalization, wrapped causes,
field identification, and high-level validation semantics. Do not use this
mechanical move to reorder domains or rewrite diagnostics.
6. Remove imports and dead private helpers left behind in `validate.go`. Do not
add facade functions to preserve former source-file placement; files are not
Go API boundaries.
### Tests And Verification ### Verification And Exit Criteria
- Existing core, campaign, session, duration, normalize, trim, render, Test success, absence, cancellation, backend failures, and S3 request mapping.
notification, and load/validate tests should normally pass unchanged. Run `go test ./internal/adapters/storage`,
- Add no tests that merely assert a function moved files. Add or repair a test `go test ./internal/artifacts ./internal/stage ./internal/app`, and
only if the move exposes an unprotected semantic rule. `git diff --check`. All implementations satisfy the narrow idempotent port.
- Run:
```sh ## Stage 2 — Define Independent Lock Objects
go test ./internal/config
go test ./internal/app -run 'Config|Pipeline|Session'
git diff --check
```
### Exit Criteria **Status:** Pending
- `validate.go` is a concise entry/coordinator file rather than a container for
unrelated domain implementations.
- Core, adapter, and shared helper ownership is obvious from the file layout,
and Audita remains one cohesive validator.
- No public behavior or serialized configuration shape changed.
## Stage 2 — Separate Publish, Storage, Notarius, And Scriptorium Owners
**Status:** Completed
### Goal ### Goal
Finish the domain-level split of the former aggregate validator before changing Establish the complete lock contract before migrating callers.
the internal structure of Notarius or Scriptorium validation.
### Required Work ### Required Work
1. Move storage, publish output, and publish lock validation into a cohesive 1. Add a cohesive `internal/artifacts` owner for payloads, size bound, canonical
private owner such as `validate_publish.go`. Move the associated helpers source validation, key construction, and strict encoding/decoding.
with their consumers, including `ValidatePublishLockRules`, 2. Reuse publish's canonical stable-source vocabulary. Reject noncanonical
`MergePublishLockRules`, configured output/source lookups, and caller input rather than making a second identity. Validate the session
`publishUploadConfiguredForS3`. prefix before joining the reserved `locks/` path.
2. Preserve the exported lock-rule helper signatures because current app and 3. Implement exact lookup and prefix listing. Exact lookup returns an explicit
stage code consume them. Do not redesign remote lock behavior; that belongs present/absent result. Listing bounds every read, rejects unexpected names,
to Workstream 8. verifies hashes, rejects duplicate sources, and sorts by source.
3. Move the existing Notarius validator into `validate_notarius.go` and its 4. Implement conditional creation, one generation-matched forced replacement,
Scriptorium counterpart, input-source validator, and exported dependency and idempotent removal. A replacement conflict is reported for operator
validator into `validate_scriptorium.go`. At this stage, move these functions retry; do not add an internal retry loop.
without decomposing or semantically reordering them; Stages 3 and 4 own those 5. Do not consult static config or `locks.yml`.
changes.
4. Keep source-vocabulary and collision helpers with the narrowest shared
domain owner. If both publish and Scriptorium need a small lookup derived
from Notarius/Scriptorium configuration, use one plainly named package
helper rather than duplicate maps or introduce a registry.
5. Ensure `validate.go` contains only entry points and high-level sequencing
after the move. Delete the emptied blocks and unused imports.
6. Search `internal/config` production code for acceptance or translation of
removed player/session roster fields and retired artifact source spellings.
Delete any actual compatibility branch found. Do not mistake current
campaign `party_file`, rejection through strict decoding, migration prose,
or a test fixture demonstrating rejection for an active compatibility path.
### Tests And Verification ### Verification And Exit Criteria
- Existing storage/publish, Notarius, Scriptorium, configuration loading, and Table-test canonical keys, strict JSON, size limits, hash mismatches, absence,
application configuration tests should pass without pinning file placement. invalid prefix entries, ordering, add conflicts, force conflicts/replacement,
- Run: and removal. Run `go test ./internal/artifacts ./internal/adapters/storage` and
`git diff --check`. Each lock is independently manageable and validated.
```sh ## Stage 3 — Permit Safe Cross-Run Declarations
go test ./internal/config
go test ./internal/app ./internal/stage
git diff --check
```
### Exit Criteria **Status:** Pending
- Every validation domain named by Workstream 7 has one discoverable file-level
owner inside `internal/config`.
- There is no duplicate source/collision logic and no active retired-format
compatibility branch in configuration production code.
- Notarius and Scriptorium semantics remain unchanged pending their dedicated
stages.
## Stage 3 — Decompose Notarius Validation And Normalization
**Status:** Completed
### Goal ### Goal
Make `validateNotarius` a short transaction that validates top-level policy, Permit exact older published declarations without weakening session isolation.
normalizes references, normalizes output lanes, and commits the resulting
configuration only after every check succeeds.
### Required Work ### Required Work
1. Keep `validateNotarius` responsible for nil/enablement behavior and the 1. Add a small key parser proving an immutable object belongs to the expected
top-level scalar contract: required binary, configuration path, pipeline ID, campaign/session and extracting its run. Reject malformed, mutable,
positive timeout, working directory, at least one output, and the reference traversal, cross-session, and empty-run keys.
binding limit. 2. Relax commit validation only for `published_output`: its key may use the
2. Replace the validator's bespoke timeout parsing with the shared positive current or an earlier run in the same session. Retain every other placement
duration helper when it preserves the same accepted values and field rule and the existing manifest version unless serialization changes.
identity. Do not embed duration policy in a Notarius-only branch. 3. Update restore, previous-state caching, current-output lookup, and
3. Extract reference binding normalization into one private operation that: status/artifact inspection to open the exact declared key and match by stable
- iterates raw selector keys in sorted order; source instead of reconstructing a current-run path. Never scan prefixes.
- normalizes selectors through `notariusref`; 4. Preserve duplicate-source and destination-collision enforcement when
- rejects normalized selector collisions; carried and current declarations coexist; derive visible origin from key.
- trims and validates each prepared-input source; and
- returns a newly allocated normalized map without mutating the input map.
4. Extract output normalization and collision validation into a separate
private operation that:
- builds the reserved built-in, configured Scriptorium, and previous-session
source set once;
- iterates raw output keys in sorted order;
- validates and normalizes configured keys and derived extraction source
IDs;
- rejects reserved-source, normalized-source, and lane-ID collisions;
- trims lane/media/schema/module scalar fields and checks required fields;
and
- returns a newly allocated normalized output map.
5. Normalize binary, paths, pipeline ID, timeout, and working directory into
local candidate values. Assign all scalar candidates, references, and
outputs back to `cfg` only after both helper operations succeed. Preserve
nil-versus-empty map behavior where it is observable in effective YAML or
digests.
6. Keep all helpers Notarius-specific and private. Do not move selector policy
out of `notariusref`, copy artifact source policy, or create a generalized
map-normalization framework.
### Tests And Verification ### Verification And Exit Criteria
- Retain lean behavior tests for disabled configuration, required scalars, Test same/earlier-run acceptance, malformed/mutable/cross-session rejection,
timeout and reference-binding limits, selector normalization/collisions, supported exact-key reads, and source-based inspection. Run
prepared sources, output normalization, reserved-source collisions, `go test ./internal/artifacts ./internal/app ./internal/stage` and
duplicate lanes, and cross-use by Scriptorium/publish. `git diff --check`. Only safe same-session published-output carry is enabled.
- Add or consolidate one test proving a failing reference or output candidate
leaves the original Notarius config unmodified.
- Add one determinism test that constructs equivalent reference/output maps in
different insertion orders and compares normalized success or equivalent
diagnostic classification. Do not assert which of multiple independent
invalid domains must win forever.
- Assert field paths and the smallest meaningful semantic fragment rather than
complete error sentences.
- Run:
```sh ## Stage 4 — Separate Source And Output Planning
go test ./internal/config -run 'Notarius|Scriptorium|Publish|Duration'
go test ./internal/stage -run 'Extract|Analyze'
git diff --check
```
### Exit Criteria **Status:** Pending
- `validateNotarius` visibly sequences scalar validation, reference
normalization, output normalization/collision checks, and one final commit.
- Reference and output collision domains are separate, deterministic, and do
not partially mutate caller-owned configuration on failure.
- No Notarius behavior moved into a generic configuration abstraction.
## Stage 4 — Decompose Deterministic Scriptorium Validation
**Status:** Completed
### Goal ### Goal
Separate Scriptorium artifact declarations/outputs, dependencies/inputs, and Make source opening and output-rule resolution navigable without changing
variables into readable validation phases with deterministic map traversal. publish behavior.
### Required Work ### Required Work
1. Keep `validateScriptorium` as the transaction owner for nil behavior, 1. Move source discovery, metadata construction, safe opening, and validation
required binary, optional config path, stage timeout, ordered validation from the aggregate publish file into one cohesive package-local owner.
phases, referenced-output checks, and the final dependency-graph check. 2. Move output selection, source matching, required/optional rules, destination
2. Produce one sorted list of configured artifact keys after validating their resolution, and collisions into another cohesive owner.
canonical key syntax. Reuse that order across phases rather than ranging 3. Keep `publishStage.Run` as caller/transaction owner with explicit inputs;
over the artifact map independently. avoid a mutable catch-all context. Do not integrate locks yet.
3. Extract a declaration/output phase that validates enabled prompt and output 4. Remove only dead helpers/imports exposed by the mechanical split.
requirements, safe output paths, artifact timeout values, and duplicate
output ownership. Keep its collision map local to that phase.
4. Extract a dependency/input phase that:
- validates dependency entries and self-reference using declared slice
indexes for actionable field paths;
- iterates input maps by sorted input name;
- validates required source declarations through the existing canonical
input-source owner;
- requires configured-artifact sources to appear in `depends_on`; and
- returns or updates only the small referenced-artifact set needed by the
later referenced-output check.
5. Extract a variable phase that iterates variable names in sorted order,
rejects empty names, and accepts only the currently supported string and
boolean values.
6. Iterate the referenced-artifact set deterministically when checking required
outputs. Keep `ValidateScriptoriumArtifactDependencies` as the public graph
validator and preserve its cycle and dependency semantics.
7. It is acceptable for phase separation to change which of several unrelated
simultaneous mistakes is reported first. Within a given implementation,
equivalent maps must produce the same result. Do not add compatibility code
to reproduce the old mixed per-artifact call order.
8. Remove forwarding-only helpers and obsolete temporary sets after the split.
Keep `validateScriptoriumInputSource` cohesive unless a genuine duplicate
semantic owner emerges.
### Tests And Verification ### Verification And Exit Criteria
- Retain semantic coverage for declaration keys, enabled requirements, output Run `go test ./internal/stage -run Publish`,
path safety/collisions, timeouts, dependency existence/self/cycles, input `go test ./internal/app -run 'Publish|Artifact|Status'`, and
source classes, extraction references, required `depends_on`, referenced `git diff --check`. Existing behavior remains, with clear source/output
outputs, and variable types. file-level ownership.
- Add one map-insertion-order test that feeds equivalent Scriptorium maps in
different insertion orders and asserts equal validity or equivalent
diagnostics. Do not assert an arbitrary globally preferred independent-error
order.
- Consolidate table cases that repeat the same source-vocabulary rule through
different retired spellings when a representative canonical rejection test
already protects unsupported-source behavior.
- Run:
```sh ## Stage 5 — Separate Transfer And Commit Mechanics
go test ./internal/config -run 'Scriptorium|ArtifactFamil|Notarius|Publish'
go test ./internal/stage -run 'Analyze'
go test ./internal/app -run 'Artifact|Config|Plan'
git diff --check
```
### Exit Criteria **Status:** Pending
- Scriptorium validation has explicit declaration/output,
dependency/input, variable, and graph phases.
- All map traversal that can affect validation or normalization is
deterministic.
- Tests protect fields and semantic rules without freezing the former mixed
validator choreography.
## Stage 5 — Split Composition Envelope Phases And Remove Incidental Metadata
**Status:** Completed
### Goal ### Goal
Separate composition-envelope shape decoding, cross-field semantics, and Finish decomposition while keeping the transaction visible in one place.
root-only metadata removal while retaining the source-aware configuration model
that imports, profiles, diagnostics, and inspection commands require.
### Required Work ### Required Work
1. Refactor `splitPipelineCompositionEnvelope` in 1. Move copying/hashing, immutable upload, remote verification, and reusable
`internal/config/pipeline_composition.go` into a short coordinator over prior-object verification into a transfer owner.
three explicit private phases: 2. Move immutable commit and current-pointer construction, encoding, and writes
- locate and decode the root `composition` mapping and its `imports`, into a commit owner.
`default_profile`, and `profiles` shapes; 3. Keep one explicit ordered path in `publishStage.Run`: plan, transfer,
- validate cross-field semantics, including default-profile membership; and immutable commit, pre-pointer checks, pointer write. Document its durability
- clone/remove the root-only envelope to produce the ordinary pipeline body. boundary briefly.
2. Reuse `parsePipelineProfileDeclaration` and profile-name normalization. 4. Delete forwarding wrappers and dead helpers; do not add package-local
Keep unknown-field rejection, non-empty profile declarations, imports and interfaces without a second implementation.
profiles in declaration order, explicit-profile selection behavior, and
source-qualified shape/conflict diagnostics.
3. Do not mutate the parsed root document during validation. Return a distinct
body document only after the envelope is valid; the original document must
remain reusable by callers and tests.
4. Remove `compositionNode` and `compositionField` metadata that has no runtime
reader after the refactor. Specifically verify and remove redundant cached
`order`, `line`, and `column` fields and their clone/renumber assignments if
they remain write-only. Slice order remains the declaration-order
representation; YAML parser nodes remain the source of parse-time line and
column diagnostics.
5. Retain each node's semantic path and source list because merge conflict
reporting, ownership projection, effective digests, artifact-family source
projection, and `config sources` consume them. Do not reduce actionable
conflict diagnostics or delete provenance that has a real consumer.
6. Keep composition product-specific. Do not replace the tree with generic map
decoding or adopt a third-party composition library.
### Tests And Verification ### Verification And Exit Criteria
- Test malformed envelope shapes, unknown fields, invalid profile declarations, Preserve checksum, size, generation, conditional-write, cancellation, and
missing default membership, successful envelope removal, and absence of partial-failure tests. Run `go test ./internal/stage -run Publish`,
caller-document mutation through the composition boundary. `go test ./internal/artifacts ./internal/adapters/storage`, and
- Preserve tests for explicit empty/false/zero/list/map values, additive `git diff --check`. The transaction remains easy to audit from `Run`.
conflicts naming all claiming sources, overlay semantics, canonical YAML,
effective digest stability, profile selection, and `config sources`
ownership. These are semantic contracts, not incidental provenance.
- Remove assertions of deleted private cached metadata. Continue asserting
declaration behavior through ordered outputs rather than private index
fields.
- Run:
```sh ## Stage 6 — Model Locked Output Carry-Forward
go test ./internal/config -run 'Composition|PipelineProfile|Effective|ArtifactFamil'
go test ./internal/app -run 'Config'
git diff --check
```
### Exit Criteria **Status:** Pending
- Envelope decoding, cross-field validation, and body production have separate
focused owners under one coordinator.
- Only provenance and ordering data with a demonstrated consumer remains.
- Imports, profiles, overlays, source reporting, and effective digests retain
deterministic behavior.
## Stage 6 — Align Diagnostic Tests, Documentation, And Close Workstream 7
**Status:** Completed
### Goal ### Goal
Review the completed configuration refactor as one system, remove obsolete test Represent every output outcome and build a coherent commit plan without I/O.
friction and documentation language, and close Workstream 7 with repository-wide
evidence.
### Required Work ### Required Work
1. Review every configuration test changed by Stages 1–5. Retain coverage of 1. Add one private output-plan/result type using the five fixed statuses. A
schema strictness, normalization, semantic invariants, meaningful failure participating output is a configured rule selected for this invocation;
categories, deterministic outcomes, composition source conflicts, effective unselected rules are reported but do not enforce required-output presence.
projections/digests, and representative assembled loads. 2. Plan deterministically by source from selected rules, discovered sources,
2. Rewrite assertions that pin complete diagnostic prose, punctuation, initial locks, and prior current commit.
formatting, private helper order, or an arbitrary independent-error winner. 3. Publish unlocked participants normally. Carry a locked prior declaration
Prefer a relevant field path plus one small semantic fragment. Use byte-for-field. Emit `locked_unpublished` without required-output failure
`errors.Is`/`errors.As` only where an existing wrapped or typed cause is a when none exists. Preserve optional-missing and unselected separately.
real contract; do not introduce types solely to change test style. 4. Reject duplicate prior sources, invalid declarations, configured destination
3. Delete redundant legacy-form enumerations whose only purpose is to preserve conflicts, and effective destination collisions. Never synthesize an old
knowledge of retired fields or spellings already rejected by the canonical declaration.
strict/source boundary. Retain representative tests proving unknown fields 5. Project one source-sorted metadata collection with source/status,
and unsupported sources fail, and retain migration guidance tests only when destination, exact declared key, and derived origin where applicable; omit
that guidance is an intentional user-facing contract. secrets and local absolute paths.
4. Consolidate oversized table fixtures only where multiple cases protect the
same rule. Do not delete distinct risks merely to reduce test LOC, and do not ### Verification And Exit Criteria
rewrite healthy semantic tests just because they use a small stable error
fragment. Pure table tests cover five states, exact carry, required locked absence, mixed
5. Update [`../internal/configuration.md`](../internal/configuration.md) to snapshots, selection, ordering, and ambiguities. Run
describe the implemented validation-domain ownership, deterministic phase `go test ./internal/stage -run 'Publish|Output'` and `git diff --check`.
behavior, composition-envelope phases, and diagnostic guarantees without One plan drives both declarations and metadata.
listing private helper choreography. Update [`../config.md`](../config.md)
only if its user-visible current-behavior wording is inaccurate; no schema ## Stage 7 — Integrate Locks Into Publish
change is expected.
6. Search current production, tests, examples, and current-behavior docs for **Status:** Pending
active acceptance of the retired configuration forms named by C01. Remove
actual compatibility code and stale current-behavior claims, while leaving ### Goal
immutable audit/history documents unchanged.
7. Run the full repository validation from `docs/development.md`: Make independent locks authoritative while preserving the snapshot transaction.
### Required Work
1. Read relevant locks at planning start and load/validate the prior current
commit when a locked source may require carry-forward.
2. Use Stage 6's plan. Do not open, hash, or upload a carried or never-published
locked source.
3. Before carrying, verify the exact prior object exists and matches declared
size and generation without download; retain its declared checksum.
4. Upload/verify new objects, combine exact carried/new declarations, validate
and write the immutable commit, then reread initially unlocked sources. If
any is now locked, return a clear error without pointer write; otherwise
write the pointer exactly once.
5. Remove `Env.RevalidatePublishLocks`, runner injection, and related plumbing
once unused. Publish reads authoritative locks through artifact/storage.
6. Replace overlapping category arrays with `publish_output_outcomes`.
Preserve upload counts, `remote_commit_key`, and
`current_commit_pointer_key` for current cleanup behavior.
### Verification And Exit Criteria
Test no locks, exact/mixed carry, required locked absence, invalid prior state,
removed-after-initial lock, added-before-final lock, abandoned attempts, avoided
locked-source work, and exactly one pointer write. Run
`go test ./internal/stage -run Publish`,
`go test ./internal/runner ./internal/app`, `go test ./...`, and
`git diff --check`. Newly observed final locks prevent pointer advancement.
## Stage 8 — Migrate Commands And Inspection
**Status:** Pending
### Goal
Preserve the operator workflow over the independent-object model.
### Required Work
1. Rewrite lock list/add/remove services and CLI wiring to use artifact lock
operations while preserving command names and session selection.
2. List validated source-sorted locks. Add reports already locked unless
`--force`; force performs one conditional replacement. Remove reports
idempotent absence without error. Remove help/output references to
precedence and shared manifests.
3. Reduce command composition to storage, session prefix, source, reason,
clock where needed, and output. Remove config mutation and callbacks.
4. Make status/current-artifact inspection show five outcomes and identify
carried sources by exact key and derived run. Fetch only declared objects.
5. Add command tests for exit behavior, conflicts, idempotent remove,
deterministic output, malformed locks, and carried inspection.
### Verification And Exit Criteria
Run `go test ./internal/app ./internal/artifacts`,
`go test ./cmd/...`, and `git diff --check`. Operators retain simple,
accurate lock and inspection commands.
## Stage 9 — Remove Retired Lock Authorities
**Status:** Pending
### Goal
Delete the obsolete control plane after all consumers migrate.
### Required Work
1. Remove `pipeline.publish.locks`, `PublishLockRule`, central store types,
`locks.yml` helpers, YAML I/O, merge/precedence logic, document replacement
retries, injection fields, compatibility adapters, and dead tests/fixtures.
2. Do not retain migration readers. Remaining historical mentions must be
clearly intentional prose.
3. If effective-config digest inputs change, advance its internal discriminator
so prior successful state invalidates deliberately. Change no user-facing
schema version unless repository policy requires it.
4. Add a representative strict-decode test proving the old config field is
unknown, and search code/docs/examples/fixtures for all retired identifiers.
### Verification And Exit Criteria
Run:
```sh ```sh
go test ./internal/config ./internal/artifacts ./internal/app ./internal/stage
go test ./... go test ./...
go test -race ./... git diff --check
go vet ./...
go build ./...
go test ./internal/doccheck
go test ./internal/config -run '^TestExamplesLoadAndValidate$'
``` ```
8. Run `go test -race -shuffle=on -count=3 ./internal/config ./internal/app` Independent objects are the sole lock authority.
to stress map-order independence and assembled configuration behavior.
9. After all checks pass, mark Stages 1–6 in this document `Completed` and mark
only Workstream 7 in [`refactoring.md`](refactoring.md) `Complete`.
Workstream 8 must remain the next pending workstream.
### Exit Criteria ## Stage 10 — Align Documentation And Close The Workstream
- Configuration remains strict, deterministic, source-aware where useful, and **Status:** Pending
unchanged in public schema and normalized semantics.
- Validation code is organized by semantic owner; Notarius, Scriptorium, and ### Goal
composition coordinators expose clear phases without generic frameworks.
- No active retired configuration compatibility path, write-only composition Make the simplified implementation the fully documented repository state.
metadata, or test dependency on incidental diagnostic choreography remains.
- Current documentation describes the implemented boundaries, all validation ### Required Work
commands pass, Workstream 7 is complete, and Workstream 8 remains pending.
1. Update canonical config, storage, stage, CLI, operations, and development
docs for the payload/key, bounded validation, commands, carry behavior, race
boundary, five outcomes, cross-run rule, and transaction sequence. Remove
static-lock and `locks.yml` claims.
2. Document that old `locks.yml` is ignored and may be manually removed after
upgrade verification; add no migration tool.
3. Remove stale comments, redundant wrappers/logic, dead fixtures, and obsolete
metadata assertions exposed by final review.
4. Run all formatting, unit/integration, static-analysis, documentation, and
policy checks required by `docs/development.md`. Record genuinely
platform-only checks without weakening or silently skipping them.
5. Only after checks pass, mark this plan's stages and Workstream 8 completed.
Leave Workstream 9 as the next pending scope.
### Verification And Exit Criteria
At minimum run `gofmt` on changed Go, `go test ./...`, `go vet ./...`,
`go test ./internal/doccheck`, `git diff --check`, and all repository
scripts mandated by policy. Code, tests, docs, help, and examples agree, and no
Workstream 9 implementation is included.
## Open Questions ## Open Questions
None. The audits, completed compatibility purge, current configuration None. The audit decisions and fixed choices above make this plan
contracts, and fixed decisions above resolve the product and implementation decision-complete.
choices needed for a `gpt-5.6-terra` coding agent to execute this plan.