narratio/docs/roadmap/implementation.md

Workstream 7 Implementation Plan — Configuration Structure And Diagnostics

Status

Completed implementation record for Workstream 7, Configuration structure and semantic diagnostics, in refactoring.md. Workstreams 1 through 6 were prerequisites, and all stages below were completed in numerical order.

The purpose of this workstream is to preserve Narratio's strict, deterministic, operator-oriented configuration behavior while making the implementation navigable by semantic domain. It also removes incidental composition metadata and test expectations that turn private validation choreography into a compatibility burden. This is an internal refactor with narrowly approved diagnostic flexibility, not a configuration-language redesign.

Authoritative Scope And Decisions

The authoritative scope is Workstream 7 in refactoring.md. Its supporting audit entries are H04, H14, H15, H17, H38, H39, H40, H41, and H42 in audit.md, C01 and C06 in contract-audit.md, and the corresponding rows in audit-decisions.md.

The following implementation decisions are fixed:

• Keep all configuration behavior in package internal/config. Do not create subpackages, a generic configuration framework, a schema framework, a validator registry, or a dependency-injection layer.
• Preserve the public configuration types and entry points. Validate, ValidatePipelineConfig, ValidateCampaignConfig, all load APIs, effective pipeline projections, and configuration-inspection commands retain their current responsibilities.
• Keep validate.go as the short public and high-level orchestration owner. Organize private validation into cohesive files for core pipeline/session, adapters, publish/storage, Notarius, Scriptorium, and shared scalar/path helpers. File names are private, but those ownership boundaries are the intended final structure.
• Do not replace ordinary error returns with a new exported validation-error hierarchy. No machine consumer currently requires one, and introducing one would add a new compatibility surface. Field paths, source identities for composition conflicts, wrapped causes where useful, and small semantic phrases provide the stable diagnostic content tests should assert.
• Continue to fail fast. The workstream does not add error aggregation and does not promise which of several independent invalid fields is reported first. Traversal of maps must nevertheless be deterministic so identical input produces identical behavior.
• Preserve strict single-document YAML decoding, duplicate- and unknown-field rejection, additive imports with no duplicate ownership, one deterministic profile selection, explicit-value overlay behavior, centralized defaults, canonical party resolution, artifact-family expansion, and secret-free effective digests.
• Preserve the current configuration schema. Campaign-owned campaign.inputs.party_file and canonical artifact source IDs are current contracts, not legacy fields. Removed player/session roster fields and old artifact source spellings remain unsupported; add no compatibility aliases or normalization branches for them.
• Treat Workstream 3's compatibility purge as authoritative. During this work, remove any surviving production branch that actually accepts or translates a retired configuration representation. A rejection supplied naturally by strict decoding or the canonical source validator is sufficient; do not maintain exhaustive legacy-form tests when representative canonical-boundary tests already protect the rule.
• Preserve composition source information that is consumed by conflict diagnostics or config sources. Remove only demonstrably unused node/field metadata. In particular, source lists and semantic field paths remain; redundant cached declaration-order indexes and copied line/column fields may be removed when no consumer reads them because slice order and the YAML parser already supply the required behavior.
• Preserve normalization as an all-or-nothing operation for each validator. Notarius reference/output maps and scalar paths are installed on the config only after all relevant checks succeed; a validation error must not leave a partially normalized value behind.
• Keep validateAudita intact when moving it to the adapter-validation owner. Its guards are cohesive; splitting them further would add names without reducing complexity.
• Tests must protect semantic rules and observable normalized results, not private helper calls, file placement, full error prose, punctuation, or one historical validation call order. Exact strings remain appropriate only for an intentionally documented CLI or machine-consumed contract.
• Do not enter Workstream 8 remote publication/lock decomposition, Workstream 9 cleanup-stage work, or unrelated Workstream 10 cleanup.

Instructions For Every Stage

Before each stage:

1. Read docs/development.md, every document under docs/policy/, this plan, and Workstream 7 in refactoring.md. Read only the audit sections and decision-register rows cited above; do not restart the audits.
2. Read ../internal/configuration.md and the affected portions of ../config.md, then inspect the production files and focused tests named by the stage.
3. Prefer the codebase knowledge graph for Go symbol discovery and call tracing. Use text search for YAML fields, source IDs, error fragments, examples, documentation, and removal verification.
4. Confirm the worktree state and preserve unrelated changes. Do not implement an adjacent pending workstream merely because it shares a file.

During every stage:

• Keep the package compiling and focused tests passing. Move behavior before redesigning the two complex validators so reviewable commits do not combine unrelated mechanical and semantic changes.
• Use package-private functions with explicit inputs and return values. Avoid stateful validator objects, forwarding-only wrappers, generic rule tables, and abstractions whose only purpose is to reduce function length.
• Keep configuration mutation explicit and late. Do not mutate caller-owned maps while validating a temporary normalized candidate.
• Retain deterministic order wherever config maps are traversed for validation, normalization, canonical output, source reporting, or digest construction.
• Keep tests deterministic, offline, and independent of credentials or live adapters. Prefer direct package behavior and real temporary files over mocks.
• Use gofmt on changed Go files, run the focused checks listed for the stage, and run git diff --check. Mark a stage Completed only after its exit criteria pass.

Stage 1 — Separate Core And Shared Validation Ownership

Status: Completed

Goal

Reduce validate.go to configuration validation entry points and readable high-level sequencing, while giving core pipeline/session rules and reusable primitive checks clear package-local owners. This stage is behavior-preserving.

Required Work

1. Keep these entry points and the visible pipeline-domain call sequence in internal/config/validate.go:
   • Validate;
   • ValidatePipelineConfig;
   • ValidateCampaignConfig; and
   • the short validatePipeline coordinator.
2. Move core configuration checks into a cohesive private file such as validate_core.go. This owner includes campaign validation, secrets, spool/cache, session validation, resolved-party input validation, session/campaign identifiers, and cross-pipeline/session rules. Keep publishUploadConfiguredForS3 with publish/storage in Stage 2 rather than hiding publish policy in the core owner.
3. Move general validation primitives into a small private file such as validate_helpers.go: environment-variable name checks, positive duration parsing, safe relative paths, and path-within-root checks. Do not turn these functions into a generic validation DSL.
4. Re-home the straightforward stage/adapter validators in a private adapter owner such as validate_adapters.go: notification, normalize, trim, render, WhisperX, Seriatim (including its output schema), and Audita. Move validateAudita intact apart from import or formatting changes.
5. Preserve current nil handling, defaults, normalization, wrapped causes, field identification, and high-level validation semantics. Do not use this mechanical move to reorder domains or rewrite diagnostics.
6. Remove imports and dead private helpers left behind in validate.go. Do not add facade functions to preserve former source-file placement; files are not Go API boundaries.

Tests And Verification

• Existing core, campaign, session, duration, normalize, trim, render, notification, and load/validate tests should normally pass unchanged.

• Add no tests that merely assert a function moved files. Add or repair a test only if the move exposes an unprotected semantic rule.

• Run:

  go test ./internal/config
  go test ./internal/app -run 'Config|Pipeline|Session'
  git diff --check
  

Exit Criteria

• validate.go is a concise entry/coordinator file rather than a container for unrelated domain implementations.
• Core, adapter, and shared helper ownership is obvious from the file layout, and Audita remains one cohesive validator.
• No public behavior or serialized configuration shape changed.

Stage 2 — Separate Publish, Storage, Notarius, And Scriptorium Owners

Status: Completed

Goal

Finish the domain-level split of the former aggregate validator before changing the internal structure of Notarius or Scriptorium validation.

Required Work

1. Move storage, publish output, and publish lock validation into a cohesive private owner such as validate_publish.go. Move the associated helpers with their consumers, including ValidatePublishLockRules, MergePublishLockRules, configured output/source lookups, and publishUploadConfiguredForS3.
2. Preserve the exported lock-rule helper signatures because current app and stage code consume them. Do not redesign remote lock behavior; that belongs to Workstream 8.
3. Move the existing Notarius validator into validate_notarius.go and its Scriptorium counterpart, input-source validator, and exported dependency validator into validate_scriptorium.go. At this stage, move these functions without decomposing or semantically reordering them; Stages 3 and 4 own those changes.
4. Keep source-vocabulary and collision helpers with the narrowest shared domain owner. If both publish and Scriptorium need a small lookup derived from Notarius/Scriptorium configuration, use one plainly named package helper rather than duplicate maps or introduce a registry.
5. Ensure validate.go contains only entry points and high-level sequencing after the move. Delete the emptied blocks and unused imports.
6. Search internal/config production code for acceptance or translation of removed player/session roster fields and retired artifact source spellings. Delete any actual compatibility branch found. Do not mistake current campaign party_file, rejection through strict decoding, migration prose, or a test fixture demonstrating rejection for an active compatibility path.

Tests And Verification

• Existing storage/publish, Notarius, Scriptorium, configuration loading, and application configuration tests should pass without pinning file placement.

• Run:

  go test ./internal/config
  go test ./internal/app ./internal/stage
  git diff --check
  

Exit Criteria

• Every validation domain named by Workstream 7 has one discoverable file-level owner inside internal/config.
• There is no duplicate source/collision logic and no active retired-format compatibility branch in configuration production code.
• Notarius and Scriptorium semantics remain unchanged pending their dedicated stages.

Stage 3 — Decompose Notarius Validation And Normalization

Status: Completed

Goal

Make validateNotarius a short transaction that validates top-level policy, normalizes references, normalizes output lanes, and commits the resulting configuration only after every check succeeds.

Required Work

1. Keep validateNotarius responsible for nil/enablement behavior and the top-level scalar contract: required binary, configuration path, pipeline ID, positive timeout, working directory, at least one output, and the reference binding limit.
2. Replace the validator's bespoke timeout parsing with the shared positive duration helper when it preserves the same accepted values and field identity. Do not embed duration policy in a Notarius-only branch.
3. Extract reference binding normalization into one private operation that:
   • iterates raw selector keys in sorted order;
   • normalizes selectors through notariusref;
   • rejects normalized selector collisions;
   • trims and validates each prepared-input source; and
   • returns a newly allocated normalized map without mutating the input map.
4. Extract output normalization and collision validation into a separate private operation that:
   • builds the reserved built-in, configured Scriptorium, and previous-session source set once;
   • iterates raw output keys in sorted order;
   • validates and normalizes configured keys and derived extraction source IDs;
   • rejects reserved-source, normalized-source, and lane-ID collisions;
   • trims lane/media/schema/module scalar fields and checks required fields; and
   • returns a newly allocated normalized output map.
5. Normalize binary, paths, pipeline ID, timeout, and working directory into local candidate values. Assign all scalar candidates, references, and outputs back to cfg only after both helper operations succeed. Preserve nil-versus-empty map behavior where it is observable in effective YAML or digests.
6. Keep all helpers Notarius-specific and private. Do not move selector policy out of notariusref, copy artifact source policy, or create a generalized map-normalization framework.

Tests And Verification

• Retain lean behavior tests for disabled configuration, required scalars, timeout and reference-binding limits, selector normalization/collisions, supported prepared sources, output normalization, reserved-source collisions, duplicate lanes, and cross-use by Scriptorium/publish.

• Add or consolidate one test proving a failing reference or output candidate leaves the original Notarius config unmodified.

• Add one determinism test that constructs equivalent reference/output maps in different insertion orders and compares normalized success or equivalent diagnostic classification. Do not assert which of multiple independent invalid domains must win forever.

• Assert field paths and the smallest meaningful semantic fragment rather than complete error sentences.

• Run:

  go test ./internal/config -run 'Notarius|Scriptorium|Publish|Duration'
  go test ./internal/stage -run 'Extract|Analyze'
  git diff --check
  

Exit Criteria

• validateNotarius visibly sequences scalar validation, reference normalization, output normalization/collision checks, and one final commit.
• Reference and output collision domains are separate, deterministic, and do not partially mutate caller-owned configuration on failure.
• No Notarius behavior moved into a generic configuration abstraction.

Stage 4 — Decompose Deterministic Scriptorium Validation

Status: Completed

Goal

Separate Scriptorium artifact declarations/outputs, dependencies/inputs, and variables into readable validation phases with deterministic map traversal.

Required Work

1. Keep validateScriptorium as the transaction owner for nil behavior, required binary, optional config path, stage timeout, ordered validation phases, referenced-output checks, and the final dependency-graph check.
2. Produce one sorted list of configured artifact keys after validating their canonical key syntax. Reuse that order across phases rather than ranging over the artifact map independently.
3. Extract a declaration/output phase that validates enabled prompt and output requirements, safe output paths, artifact timeout values, and duplicate output ownership. Keep its collision map local to that phase.
4. Extract a dependency/input phase that:
   • validates dependency entries and self-reference using declared slice indexes for actionable field paths;
   • iterates input maps by sorted input name;
   • validates required source declarations through the existing canonical input-source owner;
   • requires configured-artifact sources to appear in depends_on; and
   • returns or updates only the small referenced-artifact set needed by the later referenced-output check.
5. Extract a variable phase that iterates variable names in sorted order, rejects empty names, and accepts only the currently supported string and boolean values.
6. Iterate the referenced-artifact set deterministically when checking required outputs. Keep ValidateScriptoriumArtifactDependencies as the public graph validator and preserve its cycle and dependency semantics.
7. It is acceptable for phase separation to change which of several unrelated simultaneous mistakes is reported first. Within a given implementation, equivalent maps must produce the same result. Do not add compatibility code to reproduce the old mixed per-artifact call order.
8. Remove forwarding-only helpers and obsolete temporary sets after the split. Keep validateScriptoriumInputSource cohesive unless a genuine duplicate semantic owner emerges.

Tests And Verification

• Retain semantic coverage for declaration keys, enabled requirements, output path safety/collisions, timeouts, dependency existence/self/cycles, input source classes, extraction references, required depends_on, referenced outputs, and variable types.

• Add one map-insertion-order test that feeds equivalent Scriptorium maps in different insertion orders and asserts equal validity or equivalent diagnostics. Do not assert an arbitrary globally preferred independent-error order.

• Consolidate table cases that repeat the same source-vocabulary rule through different retired spellings when a representative canonical rejection test already protects unsupported-source behavior.

• Run:

  go test ./internal/config -run 'Scriptorium|ArtifactFamil|Notarius|Publish'
  go test ./internal/stage -run 'Analyze'
  go test ./internal/app -run 'Artifact|Config|Plan'
  git diff --check
  

Exit Criteria

• Scriptorium validation has explicit declaration/output, dependency/input, variable, and graph phases.
• All map traversal that can affect validation or normalization is deterministic.
• Tests protect fields and semantic rules without freezing the former mixed validator choreography.

Stage 5 — Split Composition Envelope Phases And Remove Incidental Metadata

Status: Completed

Goal

Separate composition-envelope shape decoding, cross-field semantics, and root-only metadata removal while retaining the source-aware configuration model that imports, profiles, diagnostics, and inspection commands require.

Required Work

1. Refactor splitPipelineCompositionEnvelope in internal/config/pipeline_composition.go into a short coordinator over three explicit private phases:
   • locate and decode the root composition mapping and its imports, default_profile, and profiles shapes;
   • validate cross-field semantics, including default-profile membership; and
   • clone/remove the root-only envelope to produce the ordinary pipeline body.
2. Reuse parsePipelineProfileDeclaration and profile-name normalization. Keep unknown-field rejection, non-empty profile declarations, imports and profiles in declaration order, explicit-profile selection behavior, and source-qualified shape/conflict diagnostics.
3. Do not mutate the parsed root document during validation. Return a distinct body document only after the envelope is valid; the original document must remain reusable by callers and tests.
4. Remove compositionNode and compositionField metadata that has no runtime reader after the refactor. Specifically verify and remove redundant cached order, line, and column fields and their clone/renumber assignments if they remain write-only. Slice order remains the declaration-order representation; YAML parser nodes remain the source of parse-time line and column diagnostics.
5. Retain each node's semantic path and source list because merge conflict reporting, ownership projection, effective digests, artifact-family source projection, and config sources consume them. Do not reduce actionable conflict diagnostics or delete provenance that has a real consumer.
6. Keep composition product-specific. Do not replace the tree with generic map decoding or adopt a third-party composition library.

Tests And Verification

• Test malformed envelope shapes, unknown fields, invalid profile declarations, missing default membership, successful envelope removal, and absence of caller-document mutation through the composition boundary.

• Preserve tests for explicit empty/false/zero/list/map values, additive conflicts naming all claiming sources, overlay semantics, canonical YAML, effective digest stability, profile selection, and config sources ownership. These are semantic contracts, not incidental provenance.

• Remove assertions of deleted private cached metadata. Continue asserting declaration behavior through ordered outputs rather than private index fields.

• Run:

  go test ./internal/config -run 'Composition|PipelineProfile|Effective|ArtifactFamil'
  go test ./internal/app -run 'Config'
  git diff --check
  

Exit Criteria

• Envelope decoding, cross-field validation, and body production have separate focused owners under one coordinator.
• Only provenance and ordering data with a demonstrated consumer remains.
• Imports, profiles, overlays, source reporting, and effective digests retain deterministic behavior.

Stage 6 — Align Diagnostic Tests, Documentation, And Close Workstream 7

Status: Completed

Goal

Review the completed configuration refactor as one system, remove obsolete test friction and documentation language, and close Workstream 7 with repository-wide evidence.

Required Work

1. Review every configuration test changed by Stages 1–5. Retain coverage of schema strictness, normalization, semantic invariants, meaningful failure categories, deterministic outcomes, composition source conflicts, effective projections/digests, and representative assembled loads.

2. Rewrite assertions that pin complete diagnostic prose, punctuation, formatting, private helper order, or an arbitrary independent-error winner. Prefer a relevant field path plus one small semantic fragment. Use errors.Is/errors.As only where an existing wrapped or typed cause is a real contract; do not introduce types solely to change test style.

3. Delete redundant legacy-form enumerations whose only purpose is to preserve knowledge of retired fields or spellings already rejected by the canonical strict/source boundary. Retain representative tests proving unknown fields and unsupported sources fail, and retain migration guidance tests only when that guidance is an intentional user-facing contract.

4. Consolidate oversized table fixtures only where multiple cases protect the same rule. Do not delete distinct risks merely to reduce test LOC, and do not rewrite healthy semantic tests just because they use a small stable error fragment.

5. Update ../internal/configuration.md to describe the implemented validation-domain ownership, deterministic phase behavior, composition-envelope phases, and diagnostic guarantees without listing private helper choreography. Update ../config.md only if its user-visible current-behavior wording is inaccurate; no schema change is expected.

6. Search current production, tests, examples, and current-behavior docs for active acceptance of the retired configuration forms named by C01. Remove actual compatibility code and stale current-behavior claims, while leaving immutable audit/history documents unchanged.

7. Run the full repository validation from docs/development.md:

   go test ./...
   go test -race ./...
   go vet ./...
   go build ./...
   go test ./internal/doccheck
   go test ./internal/config -run '^TestExamplesLoadAndValidate$'
   

8. Run go test -race -shuffle=on -count=3 ./internal/config ./internal/app to stress map-order independence and assembled configuration behavior.

9. After all checks pass, mark Stages 1–6 in this document Completed and mark only Workstream 7 in refactoring.md Complete. Workstream 8 must remain the next pending workstream.

Exit Criteria

• Configuration remains strict, deterministic, source-aware where useful, and unchanged in public schema and normalized semantics.
• Validation code is organized by semantic owner; Notarius, Scriptorium, and composition coordinators expose clear phases without generic frameworks.
• No active retired configuration compatibility path, write-only composition metadata, or test dependency on incidental diagnostic choreography remains.
• Current documentation describes the implemented boundaries, all validation commands pass, Workstream 7 is complete, and Workstream 8 remains pending.

Open Questions

None. The audits, completed compatibility purge, current configuration contracts, and fixed decisions above resolve the product and implementation choices needed for a gpt-5.6-terra coding agent to execute this plan.