From 7a0e2cb11b7d613551618df0ef64bb4e3b20a930 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 4 Sep 2026 23:37:02 +0000 Subject: [PATCH] Plan configuration refactoring --- docs/roadmap/implementation.md | 808 ++++++++++++++++++--------------- 1 file changed, 435 insertions(+), 373 deletions(-) diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 45d9769..786d4f4 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,431 +1,495 @@ -# Workstream 6 Implementation Plan — Per-Artifact Analyze Checkpointing +# Workstream 7 Implementation Plan — Configuration Structure And Diagnostics ## Status -Completed implementation record for Workstream 6, **Per-artifact analyze -checkpointing and recovery**, in [`refactoring.md`](refactoring.md). Workstreams -1 through 6 and all stages below are complete. Workstream 7 is the next pending -workstream; its implementation is outside this plan. +Implementation plan for Workstream 7, **Configuration structure and semantic +diagnostics**, in [`refactoring.md`](refactoring.md). Workstreams 1 through 6 +are complete and are prerequisites for this plan. All stages below are pending +and must be implemented in numerical order. -The purpose of this workstream is to make analyze a deterministic resumable -artifact queue backed only by the session manifest. If artifacts A and B -succeed, C fails, and D is not attempted, A and B must remain reusable, C must -be retried, and D must follow after C succeeds. The implementation must remove -the current end-of-stage analyze projection and rollback machinery rather than -building checkpointing alongside it as a second durable model. +The purpose of this workstream is to preserve Narratio's strict, deterministic, +operator-oriented configuration behavior while making the implementation +navigable by semantic domain. It also removes incidental composition metadata +and test expectations that turn private validation choreography into a +compatibility burden. This is an internal refactor with narrowly approved +diagnostic flexibility, not a configuration-language redesign. ## Authoritative Scope And Decisions -The authoritative product and architecture scope is Workstream 6, the -dependency relationships, compatibility intent, and consolidated target state -in [`refactoring.md`](refactoring.md). Supporting decisions are H04, H14, H15, -H36, H37, C02, and C03 in [`audit-decisions.md`](audit-decisions.md), with the -detailed rationale in H36 and H37 of [`audit.md`](audit.md) and Finding 3 of -[`contract-audit.md`](contract-audit.md). +The authoritative scope is Workstream 7 in [`refactoring.md`](refactoring.md). +Its supporting audit entries are H04, H14, H15, H17, H38, H39, H40, H41, and +H42 in [`audit.md`](audit.md), C01 and C06 in +[`contract-audit.md`](contract-audit.md), and the corresponding rows in +[`audit-decisions.md`](audit-decisions.md). -The following decisions are fixed for this implementation: +The following implementation decisions are fixed: -- Keep analyze as one fixed pipeline stage in `internal/stage`; per-artifact - work does not become dynamic pipeline stages or a generic workflow engine. -- Keep the session manifest's `analyze_artifacts` collection as the only durable - per-artifact authority. Do not introduce an invocation collection, journal, - sidecar, database, or replacement run manifest. -- Use the existing `stage.Env.ManifestStore` and the canonical session-manifest - path for checkpoints. Production execution already supplies that store and - holds the session lock for the entire `Stage.Run` call. Do not add a callback - framework or expose application lifecycle types to `internal/stage`. -- Checkpoint a copy-on-write candidate manifest. Validate and save the candidate - before installing its analyze collection into the live in-memory manifest. - A failed current-record checkpoint must never make newly materialized bytes - authoritative through a later terminal save. -- Persist the reconciled starting collection before the first adapter call when - it changes durable authority. This removes obsolete records and stales - scheduled or invalid current records before replacement work begins. -- For every successful artifact: validate and materialize its canonical output, - construct and validate its current record, update the runtime catalog, stale - transitive dependents when output identity changed, checkpoint the complete - session collection, and only then attempt the next artifact. -- For an artifact failure: retain earlier successful checkpoints, replace the - active artifact with a bounded failed record, stale its current transitive - dependents, and expose that collection on the live manifest. The application - runner remains responsible for marking aggregate analyze failed, invalidating - publish/notify, and atomically persisting the terminal manifest before the - command returns the error. -- If persistence of a successful-artifact checkpoint fails, stop immediately. - Keep the last durable collection, mark the active artifact failed in the live - collection without promoting its incidental output, and let the runner's - terminal failure save make that retry state durable when storage permits. -- Preserve the existing deterministic work planner, reconciliation, - fingerprints, evidence validation, dependency order, and current-output - identity comparison. File presence alone never creates authority. -- Preserve artifact selection and force behavior. With no `--artifacts`, the - effective configured set is targeted. With `--artifacts`, only the named - targets and their required dependency closure participate. Force reruns - explicit targets; a valid current prerequisite is reused unless it is itself - explicit or becomes non-current through dependency work. Unrelated valid - records are retained. -- On aggregate success, return ordinary `StageResult.Outputs` rebuilt in - deterministic normalized artifact-key order from all current session analyze - records. Per-artifact records retain their own logs, generated configuration - references, bounded Scriptorium provenance, and evidence; aggregate logs and - metadata continue to describe the current invocation. -- On aggregate failure, aggregate outputs remain empty. Earlier current - per-artifact records remain available to a later analyze run, but publish and - notify are invalidated by the existing application-stage dependency relation. -- A crash or cancellation after a successful checkpoint may leave aggregate - analyze `running` or later normalized to `interrupted`; the next ordinary run - must reconcile the durable collection, reuse valid completed artifacts, and - continue with the first failed, stale, or missing artifact. -- Keep the manifest schema and analyze-state contract version unchanged. This - work changes persistence timing and removes an internal projection transport; - it does not introduce a new serialized representation. -- Do not enter Workstream 7 configuration decomposition, Workstream 8 publish - locks/decomposition, or Workstream 9 cleanup-stage work. +- Keep all configuration behavior in package `internal/config`. Do not create + subpackages, a generic configuration framework, a schema framework, a + validator registry, or a dependency-injection layer. +- Preserve the public configuration types and entry points. `Validate`, + `ValidatePipelineConfig`, `ValidateCampaignConfig`, all load APIs, effective + pipeline projections, and configuration-inspection commands retain their + current responsibilities. +- Keep `validate.go` as the short public and high-level orchestration owner. + Organize private validation into cohesive files for core pipeline/session, + adapters, publish/storage, Notarius, Scriptorium, and shared scalar/path + helpers. File names are private, but those ownership boundaries are the + intended final structure. +- Do not replace ordinary `error` returns with a new exported validation-error + hierarchy. No machine consumer currently requires one, and introducing one + would add a new compatibility surface. Field paths, source identities for + composition conflicts, wrapped causes where useful, and small semantic + phrases provide the stable diagnostic content tests should assert. +- Continue to fail fast. The workstream does not add error aggregation and does + not promise which of several independent invalid fields is reported first. + Traversal of maps must nevertheless be deterministic so identical input + produces identical behavior. +- Preserve strict single-document YAML decoding, duplicate- and unknown-field + rejection, additive imports with no duplicate ownership, one deterministic + profile selection, explicit-value overlay behavior, centralized defaults, + canonical party resolution, artifact-family expansion, and secret-free + effective digests. +- Preserve the current configuration schema. Campaign-owned + `campaign.inputs.party_file` and canonical artifact source IDs are current + contracts, not legacy fields. Removed player/session roster fields and old + artifact source spellings remain unsupported; add no compatibility aliases + or normalization branches for them. +- Treat Workstream 3's compatibility purge as authoritative. During this work, + remove any surviving production branch that actually accepts or translates + a retired configuration representation. A rejection supplied naturally by + strict decoding or the canonical source validator is sufficient; do not + maintain exhaustive legacy-form tests when representative canonical-boundary + tests already protect the rule. +- Preserve composition source information that is consumed by conflict + diagnostics or `config sources`. Remove only demonstrably unused node/field + metadata. In particular, source lists and semantic field paths remain; + redundant cached declaration-order indexes and copied line/column fields may + be removed when no consumer reads them because slice order and the YAML + parser already supply the required behavior. +- Preserve normalization as an all-or-nothing operation for each validator. + Notarius reference/output maps and scalar paths are installed on the config + only after all relevant checks succeed; a validation error must not leave a + partially normalized value behind. +- Keep `validateAudita` intact when moving it to the adapter-validation owner. + Its guards are cohesive; splitting them further would add names without + reducing complexity. +- Tests must protect semantic rules and observable normalized results, not + private helper calls, file placement, full error prose, punctuation, or one + historical validation call order. Exact strings remain appropriate only for + an intentionally documented CLI or machine-consumed contract. +- Do not enter Workstream 8 remote publication/lock decomposition, Workstream 9 + cleanup-stage work, or unrelated Workstream 10 cleanup. ## Instructions For Every Stage Before each stage: 1. Read `docs/development.md`, every document under `docs/policy/`, this plan, - and Workstream 6 in [`refactoring.md`](refactoring.md). Read the decision - entries and audit sections named above; do not re-audit unrelated findings. -2. Read [`../internal/stage-analyze.md`](../internal/stage-analyze.md) and - [`../internal/manifest.md`](../internal/manifest.md), then inspect the + and Workstream 7 in [`refactoring.md`](refactoring.md). Read only the audit + sections and decision-register rows cited above; do not restart the audits. +2. Read [`../internal/configuration.md`](../internal/configuration.md) and the + affected portions of [`../config.md`](../config.md), then inspect the production files and focused tests named by the stage. -3. Prefer the codebase knowledge graph for Go symbol discovery and call tracing. - Use text search for serialized fields, documentation, fixtures, and removal - of projection terminology. -4. Confirm the worktree state and preserve unrelated changes. Do not begin a - later workstream merely because adjacent code is visible. +3. Prefer the codebase knowledge graph for Go symbol discovery and call + tracing. Use text search for YAML fields, source IDs, error fragments, + examples, documentation, and removal verification. +4. Confirm the worktree state and preserve unrelated changes. Do not implement + an adjacent pending workstream merely because it shares a file. During every stage: -- Keep the repository compiling and focused tests passing. A temporary private - delegation or projection bridge is permitted only where the next numbered - stage explicitly removes it; do not leave two durable authorities active. -- Preserve package ownership and acyclic dependency direction. Keep new types - private and cohesive; do not add public extension points. -- Keep all tests deterministic, offline, and independent of real Scriptorium or - paid APIs. Prefer the existing fake runner, real temporary filesystem, and a - small recording/failing manifest store only where checkpoint interaction is - itself the contract. -- Assert persisted manifest state, adapter calls, dependency order, and retry - behavior. Do not pin private helper choreography, complete error wording, or - incidental timestamps. -- Preserve bounded, non-secret errors and diagnostics. Never place prompt or - transcript content, variables containing secrets, or raw credentials in new - manifest fields or error assertions. -- Use `gofmt` on changed Go files, run the focused tests listed for the stage, +- Keep the package compiling and focused tests passing. Move behavior before + redesigning the two complex validators so reviewable commits do not combine + unrelated mechanical and semantic changes. +- Use package-private functions with explicit inputs and return values. Avoid + stateful validator objects, forwarding-only wrappers, generic rule tables, + and abstractions whose only purpose is to reduce function length. +- Keep configuration mutation explicit and late. Do not mutate caller-owned + maps while validating a temporary normalized candidate. +- Retain deterministic order wherever config maps are traversed for validation, + normalization, canonical output, source reporting, or digest construction. +- Keep tests deterministic, offline, and independent of credentials or live + adapters. Prefer direct package behavior and real temporary files over mocks. +- Use `gofmt` on changed Go files, run the focused checks listed for the stage, and run `git diff --check`. Mark a stage `Completed` only after its exit criteria pass. -## Stage 1 — Decompose The Single-Artifact Transaction +## Stage 1 — Separate Core And Shared Validation Ownership -**Status:** Completed +**Status:** Pending ### Goal -Make one artifact execution understandable and independently reviewable before -changing its persistence timing. This stage is behavior-preserving. +Reduce `validate.go` to configuration validation entry points and readable +high-level sequencing, while giving core pipeline/session rules and reusable +primitive checks clear package-local owners. This stage is behavior-preserving. ### Required Work -1. Refactor `executeAnalyzeArtifact` in `internal/stage/analyze.go` into one - short transaction owner plus private phases with explicit intermediate - values. Keep the existing `analyzeExecutionContext` and artifact plan, or - replace them with equally narrow private values; do not restore the former - broad parameter list. -2. Separate these concerns without changing their order or contracts: - - input identity resolution, omitted optional inputs, reused-current input - reporting, Scriptorium variables, timeout, canonical/run-local paths, and - request/diagnostic path preparation; - - optional render-debug invocation and validation; - - Scriptorium artifact invocation and adapter failure adaptation; - - authoritative output-path selection, bounded non-empty regular-file - validation, output-contract construction, canonical materialization, and - installed-checksum verification; and - - result/evidence construction, including provenance, logs, generated - configuration paths, metadata, output size, and reused artifacts. -3. Place the extracted code in focused analyze-owned files when that improves - navigation. Suggested responsibilities are preparation, adapter invocation, - and validated materialization/evidence; exact private filenames are not a - contract. -4. Keep Scriptorium request and response types inside the stage/adapter boundary. - Do not move Narratio artifact, manifest, path, or metadata policy into the - Scriptorium adapter. -5. Preserve the current run-local-before-canonical materialization boundary, - render-debug behavior, useful error context, workspace modes, centralized - size limit, and checksum comparison. Do not weaken validation or add a new - write path while moving code. -6. Retain existing behavior tests at their stable boundaries. Consolidate only - tests that become purely coupled to moved private helpers; do not add one - test per extracted function. +1. Keep these entry points and the visible pipeline-domain call sequence in + `internal/config/validate.go`: + - `Validate`; + - `ValidatePipelineConfig`; + - `ValidateCampaignConfig`; and + - the short `validatePipeline` coordinator. +2. Move core configuration checks into a cohesive private file such as + `validate_core.go`. This owner includes campaign validation, secrets, + spool/cache, session validation, resolved-party input validation, + session/campaign identifiers, and cross-pipeline/session rules. Keep + `publishUploadConfiguredForS3` with publish/storage in Stage 2 rather than + hiding publish policy in the core owner. +3. Move general validation primitives into a small private file such as + `validate_helpers.go`: environment-variable name checks, positive duration + parsing, safe relative paths, and path-within-root checks. Do not turn these + functions into a generic validation DSL. +4. Re-home the straightforward stage/adapter validators in a private adapter + owner such as `validate_adapters.go`: notification, normalize, trim, render, + WhisperX, Seriatim (including its output schema), and Audita. Move + `validateAudita` intact apart from import or formatting changes. +5. Preserve current nil handling, defaults, normalization, wrapped causes, + field identification, and high-level validation semantics. Do not use this + mechanical move to reorder domains or rewrite diagnostics. +6. Remove imports and dead private helpers left behind in `validate.go`. Do not + add facade functions to preserve former source-file placement; files are not + Go API boundaries. + +### Tests And Verification + +- Existing core, campaign, session, duration, normalize, trim, render, + notification, and load/validate tests should normally pass unchanged. +- Add no tests that merely assert a function moved files. Add or repair a test + only if the move exposes an unprotected semantic rule. +- Run: + + ```sh + go test ./internal/config + go test ./internal/app -run 'Config|Pipeline|Session' + git diff --check + ``` ### Exit Criteria -- `executeAnalyzeArtifact` reads as a short ordered transaction over explicit - phases and no longer mixes all preparation, invocation, materialization, and - evidence construction in one large function. -- Existing analyze request, render, validation, materialization, provenance, - and failure behavior is unchanged. -- `go test ./internal/stage ./internal/adapters/scriptorium` succeeds. -- `git diff --check` succeeds. +- `validate.go` is a concise entry/coordinator file rather than a container for + unrelated domain implementations. +- Core, adapter, and shared helper ownership is obvious from the file layout, + and Audita remains one cohesive validator. +- No public behavior or serialized configuration shape changed. -## Stage 2 — Establish Session-Only Analyze Execution State +## Stage 2 — Separate Publish, Storage, Notarius, And Scriptorium Owners -**Status:** Completed +**Status:** Pending ### Goal -Replace `analyzeStage.Run`'s parallel local collections with one private -session-oriented execution state that is ready to checkpoint, while retaining -the existing end-of-stage persistence boundary temporarily. +Finish the domain-level split of the former aggregate validator before changing +the internal structure of Notarius or Scriptorium validation. ### Required Work -1. Introduce a private analyze execution state/accumulator that owns: - - the shared execution context and deterministic work plan; - - the complete reconciled session artifact collection; - - prior current records needed for output-identity comparison; - - the evaluated/scheduled key set used for dependent invalidation; - - the mutable runtime artifact catalog; and - - invocation-only aggregate logs, generated configurations, generated - metadata, reused-artifact metadata, and its deduplication set. -2. Keep setup, reconciliation, work planning, adapter-availability validation, - deterministic iteration, and final `StageResult` assembly visible in - `analyzeStage.Run`. Move per-item state mutation and diagnostic collection - behind narrow methods on the private state; do not turn the accumulator into - a service locator. -3. Rename `analyzeWorkPlan.ProjectedRecords`, `projectedAnalyzeRecords`, and - related private terminology to describe the reconciled starting session - collection. It is the candidate durable state, not an invocation projection. - Preserve the planner's read-only behavior and existing classification. -4. Make the accumulator install its complete collection into the passed session - manifest on success and its failed/staled collection on error. During this - stage, continue returning the existing `AnalyzeStateProjection` as a - temporary compatibility bridge for the application runner; both views must - be derived from the same collection, and no additional save is introduced. -5. Add optional `ProducerRunID` provenance to `artifacts.Ref`. Update the - generic application result mapper to use that value when it is non-empty and - otherwise retain the current behavior of assigning the active run ID. This - is data needed by ordinary stage-result transport, not an analyze-specific - branch; cover both explicit and fallback behavior in the mapper tests. -6. Add a private helper that deterministically converts all current analyze - records into ordinary `artifacts.Ref` outputs. Preserve source ID, relative - path, contract, checksum, and producer run ID. Include valid current records - retained outside a partial selection; do not include stale, missing, or - failed records. Continue using the projection path for aggregate outputs - until Stage 3 switches ownership. -7. Rewrite stage tests to inspect the manifest collection mutated by the real - analyze stage rather than manually installing returned projections. Retain a - minimal assertion that the temporary projection matches that collection so - the transition cannot silently diverge. +1. Move storage, publish output, and publish lock validation into a cohesive + private owner such as `validate_publish.go`. Move the associated helpers + with their consumers, including `ValidatePublishLockRules`, + `MergePublishLockRules`, configured output/source lookups, and + `publishUploadConfiguredForS3`. +2. Preserve the exported lock-rule helper signatures because current app and + stage code consume them. Do not redesign remote lock behavior; that belongs + to Workstream 8. +3. Move the existing Notarius validator into `validate_notarius.go` and its + Scriptorium counterpart, input-source validator, and exported dependency + validator into `validate_scriptorium.go`. At this stage, move these functions + without decomposing or semantically reordering them; Stages 3 and 4 own those + changes. +4. Keep source-vocabulary and collision helpers with the narrowest shared + domain owner. If both publish and Scriptorium need a small lookup derived + from Notarius/Scriptorium configuration, use one plainly named package + helper rather than duplicate maps or introduce a registry. +5. Ensure `validate.go` contains only entry points and high-level sequencing + after the move. Delete the emptied blocks and unused imports. +6. Search `internal/config` production code for acceptance or translation of + removed player/session roster fields and retired artifact source spellings. + Delete any actual compatibility branch found. Do not mistake current + campaign `party_file`, rejection through strict decoding, migration prose, + or a test fixture demonstrating rejection for an active compatibility path. + +### Tests And Verification + +- Existing storage/publish, Notarius, Scriptorium, configuration loading, and + application configuration tests should pass without pinning file placement. +- Run: + + ```sh + go test ./internal/config + go test ./internal/app ./internal/stage + git diff --check + ``` ### Exit Criteria -- `analyzeStage.Run` coordinates one private session collection rather than - parallel record and diagnostic variables. -- Planning/reconciliation remains read-only, execution order and force/selection - behavior are unchanged, and stage tests primarily assert manifest state. -- Ordinary `StageResult.Outputs` can preserve a reused artifact's original - producer run identity without an analyze-specific application path. -- The existing application projection path still functions and no checkpoint - timing has changed yet. -- `go test ./internal/stage ./internal/app` succeeds. -- `git diff --check` succeeds. +- Every validation domain named by Workstream 7 has one discoverable file-level + owner inside `internal/config`. +- There is no duplicate source/collision logic and no active retired-format + compatibility branch in configuration production code. +- Notarius and Scriptorium semantics remain unchanged pending their dedicated + stages. -## Stage 3 — Cut Analyze Over To Durable Per-Artifact Checkpoints +## Stage 3 — Decompose Notarius Validation And Normalization -**Status:** Completed +**Status:** Pending ### Goal -Make each validated artifact completion durable before later artifact work and -stop using the end-of-stage projection transport for real analyze execution. +Make `validateNotarius` a short transaction that validates top-level policy, +normalizes references, normalizes output lanes, and commits the resulting +configuration only after every check succeeds. ### Required Work -1. Add a private analyze checkpoint operation in `internal/stage`. It must: - - require a non-nil session manifest, analyze stage record, and - `Env.ManifestStore` when analyze has state to persist; - - derive the canonical manifest path through `artifacts` helpers from the - resolved workspace, campaign, and session identity; - - validate the full candidate collection with - `manifest.ValidateAnalyzeArtifactCollection`; - - shallow-copy the manifest, clone the top-level stage map, copy the analyze - stage record, and deep-clone the candidate analyze collection so the save - cannot mutate live authority before success; - - preserve the aggregate analyze status (normally `running`) and every - unrelated manifest field; - - save the candidate through `Env.ManifestStore`; and - - only after success, install the candidate analyze version/collection and - the store-updated manifest timestamp into the live manifest. - Do not add a general manifest transaction framework or JSON round-trip clone. -2. Before invoking the first scheduled artifact, checkpoint the reconciled - starting collection when it differs from durable state. This must remove - records for removed configuration and stale any selected/forced/non-current - authority before replacement work begins. If this save fails, invoke no - adapter. -3. For each scheduled artifact, preserve this order: - - compute its execution fingerprint; - - execute the decomposed artifact transaction; - - construct and validate its current manifest record; - - prove the generated record can be added to the runtime catalog for use by - later dependents; - - stale unscheduled transitive dependents only when prior and replacement - output identities differ; - - checkpoint the complete updated session collection; and - - only then expose the result to a later artifact. -4. If a successful artifact's checkpoint fails, stop immediately and do not - call the next artifact. Build a failed record for the active artifact from - the last committed collection, with no output authority; stale its current - transitive dependents; and install that failure candidate in memory so the - runner's terminal save can persist retry state. Return the checkpoint - failure with artifact identity and manifest context. -5. Have real analyze execution return ordinary `StageResult.Outputs` built from - every current session record plus its existing aggregate diagnostics and - metadata. Stop returning `AnalyzeStateProjection` from `analyzeStage.Run`. - The legacy projection type and application handlers may remain only as dead - transitional code until Stage 4; add no new uses. -6. In `executeSelectedStage`, make a terminal save failure after any stage's - apparent success mark that stage failed and invalidate succeeded transitive - dependents before terminal persistence. This behavior currently occurs only - on the analyze-projection branch; it must remain correct when real analyze no - longer returns a projection and should be correct generically. -7. Update direct analyze fixtures so they supply a real temporary - `manifest.LocalStore`, complete campaign/run identity, an aggregate analyze - record in the state appropriate to the test, and an initially persisted - manifest. Keep unit-only planner/reconciliation tests free of persistence. -8. Add focused checkpoint tests at the stage/application boundary: - - the manifest store observes A current before B is invoked and A+B current - before C is invoked; - - a failed B checkpoint prevents C from being invoked; - - the failed B checkpoint never persists B as current merely because its - canonical file exists; and - - final aggregate success exposes outputs for all and only current records - in deterministic normalized artifact-key order, preserving each record's - producer run identity. +1. Keep `validateNotarius` responsible for nil/enablement behavior and the + top-level scalar contract: required binary, configuration path, pipeline ID, + positive timeout, working directory, at least one output, and the reference + binding limit. +2. Replace the validator's bespoke timeout parsing with the shared positive + duration helper when it preserves the same accepted values and field + identity. Do not embed duration policy in a Notarius-only branch. +3. Extract reference binding normalization into one private operation that: + - iterates raw selector keys in sorted order; + - normalizes selectors through `notariusref`; + - rejects normalized selector collisions; + - trims and validates each prepared-input source; and + - returns a newly allocated normalized map without mutating the input map. +4. Extract output normalization and collision validation into a separate + private operation that: + - builds the reserved built-in, configured Scriptorium, and previous-session + source set once; + - iterates raw output keys in sorted order; + - validates and normalizes configured keys and derived extraction source + IDs; + - rejects reserved-source, normalized-source, and lane-ID collisions; + - trims lane/media/schema/module scalar fields and checks required fields; + and + - returns a newly allocated normalized output map. +5. Normalize binary, paths, pipeline ID, timeout, and working directory into + local candidate values. Assign all scalar candidates, references, and + outputs back to `cfg` only after both helper operations succeed. Preserve + nil-versus-empty map behavior where it is observable in effective YAML or + digests. +6. Keep all helpers Notarius-specific and private. Do not move selector policy + out of `notariusref`, copy artifact source policy, or create a generalized + map-normalization framework. + +### Tests And Verification + +- Retain lean behavior tests for disabled configuration, required scalars, + timeout and reference-binding limits, selector normalization/collisions, supported + prepared sources, output normalization, reserved-source collisions, + duplicate lanes, and cross-use by Scriptorium/publish. +- Add or consolidate one test proving a failing reference or output candidate + leaves the original Notarius config unmodified. +- Add one determinism test that constructs equivalent reference/output maps in + different insertion orders and compares normalized success or equivalent + diagnostic classification. Do not assert which of multiple independent + invalid domains must win forever. +- Assert field paths and the smallest meaningful semantic fragment rather than + complete error sentences. +- Run: + + ```sh + go test ./internal/config -run 'Notarius|Scriptorium|Publish|Duration' + go test ./internal/stage -run 'Extract|Analyze' + git diff --check + ``` ### Exit Criteria -- Every successful scheduled artifact is durable before the next adapter call. -- A checkpoint failure stops later work and cannot promote incidental output. -- Real analyze execution no longer relies on an end-of-stage state projection. -- Aggregate success/failure persistence and dependent invalidation remain - application-owned and correct. -- `go test ./internal/stage ./internal/app ./internal/manifest` succeeds. -- `git diff --check` succeeds. +- `validateNotarius` visibly sequences scalar validation, reference + normalization, output normalization/collision checks, and one final commit. +- Reference and output collision domains are separate, deterministic, and do + not partially mutate caller-owned configuration on failure. +- No Notarius behavior moved into a generic configuration abstraction. -## Stage 4 — Complete Failure Recovery And Remove Projection Machinery +## Stage 4 — Decompose Deterministic Scriptorium Validation -**Status:** Completed +**Status:** Pending ### Goal -Complete the A/B/C/D recovery contract and delete the superseded projection, -validation, application, and rollback paths. +Separate Scriptorium artifact declarations/outputs, dependencies/inputs, and +variables into readable validation phases with deterministic map traversal. ### Required Work -1. On any artifact-specific failure—including fingerprinting, preparation, - render, adapter execution, result validation, materialization, evidence - construction, catalog admission, or checkpoint persistence—construct one - bounded failed record for the active key, clear its output authority, and - stale current transitive dependents. Preserve unrelated current records and - all earlier successfully checkpointed records. -2. Install the failed/staled collection on the live manifest before returning - the artifact error. The runner must then use its ordinary stage-error path to - mark aggregate analyze failed, clear aggregate outputs, invalidate publish - and notify, and persist the terminal session manifest. Join a terminal save - failure with the originating artifact/checkpoint error; never replace the - useful cause. -3. Ensure unattempted artifacts behave according to existing authority: - previously absent records remain absent/missing on reconciliation; unrelated - valid current records remain current; and a prior current record that - depends directly or transitively on the failed artifact becomes stale with - no output authority. -4. Remove `AnalyzeStateProjection` from `stage.StageResult`, delete - `internal/app/analyze_projection.go`, and remove all validation, cloning, - application, capture, rollback, and special output reconstruction that exists - solely for that projection. Remove fake projection stages and projection-only - tests; replace their meaningful risks with real analyze checkpoint/recovery - coverage. -5. Simplify `executeSelectedStage` to its generic result/error handling while - preserving analyze's live manifest collection across aggregate lifecycle - clearing. Do not add a type assertion, stage-name branch, callback, or new - result union to replace the projection special case. -6. Add a durable assembled-run regression for the approved scenario: - - configure deterministic A, B, C, and D work in dependency order; - - make A and B succeed, C fail, and verify D is not attempted; - - reload the manifest and verify A/B current, C failed, aggregate analyze - failed, and the correct missing/stale state for D; - - run again without force, verify only C and then D execute, and verify the - final aggregate and all selected records are current; and - - verify valid unrelated records and upstream transcript stages were never - invalidated or rerun. -7. Add focused variants for failure of the first and last artifact, cancellation - after at least one checkpoint, changed-output invalidation of an unselected - transitive dependent, and byte/contract-identical replacement preserving an - unselected dependent. Consolidate with existing incremental/failure tests so - each durable risk has one clear owner. +1. Keep `validateScriptorium` as the transaction owner for nil behavior, + required binary, optional config path, stage timeout, ordered validation + phases, referenced-output checks, and the final dependency-graph check. +2. Produce one sorted list of configured artifact keys after validating their + canonical key syntax. Reuse that order across phases rather than ranging + over the artifact map independently. +3. Extract a declaration/output phase that validates enabled prompt and output + requirements, safe output paths, artifact timeout values, and duplicate + output ownership. Keep its collision map local to that phase. +4. Extract a dependency/input phase that: + - validates dependency entries and self-reference using declared slice + indexes for actionable field paths; + - iterates input maps by sorted input name; + - validates required source declarations through the existing canonical + input-source owner; + - requires configured-artifact sources to appear in `depends_on`; and + - returns or updates only the small referenced-artifact set needed by the + later referenced-output check. +5. Extract a variable phase that iterates variable names in sorted order, + rejects empty names, and accepts only the currently supported string and + boolean values. +6. Iterate the referenced-artifact set deterministically when checking required + outputs. Keep `ValidateScriptoriumArtifactDependencies` as the public graph + validator and preserve its cycle and dependency semantics. +7. It is acceptable for phase separation to change which of several unrelated + simultaneous mistakes is reported first. Within a given implementation, + equivalent maps must produce the same result. Do not add compatibility code + to reproduce the old mixed per-artifact call order. +8. Remove forwarding-only helpers and obsolete temporary sets after the split. + Keep `validateScriptoriumInputSource` cohesive unless a genuine duplicate + semantic owner emerges. + +### Tests And Verification + +- Retain semantic coverage for declaration keys, enabled requirements, output + path safety/collisions, timeouts, dependency existence/self/cycles, input + source classes, extraction references, required `depends_on`, referenced + outputs, and variable types. +- Add one map-insertion-order test that feeds equivalent Scriptorium maps in + different insertion orders and asserts equal validity or equivalent + diagnostics. Do not assert an arbitrary globally preferred independent-error + order. +- Consolidate table cases that repeat the same source-vocabulary rule through + different retired spellings when a representative canonical rejection test + already protects unsupported-source behavior. +- Run: + + ```sh + go test ./internal/config -run 'Scriptorium|ArtifactFamil|Notarius|Publish' + go test ./internal/stage -run 'Analyze' + go test ./internal/app -run 'Artifact|Config|Plan' + git diff --check + ``` ### Exit Criteria -- A/B remain durable after C fails; an ordinary later run retries C and then D - without invoking A/B. -- Artifact failure and checkpoint failure leave no false current authority, and - publish/notify cannot remain successfully current after aggregate failure. -- No analyze projection type, field, helper, application branch, rollback path, - test fixture, or documentation claim remains. -- `go test ./internal/stage ./internal/app ./internal/manifest ./internal/artifacts` - succeeds. -- `git diff --check` succeeds. +- Scriptorium validation has explicit declaration/output, + dependency/input, variable, and graph phases. +- All map traversal that can affect validation or normalization is + deterministic. +- Tests protect fields and semantic rules without freezing the former mixed + validator choreography. -## Stage 5 — Verify Force Semantics, Align Documentation, And Close Workstream 6 +## Stage 5 — Split Composition Envelope Phases And Remove Incidental Metadata -**Status:** Completed +**Status:** Pending ### Goal -Verify the complete operational contract, remove superseded test/documentation -language, and leave Workstream 6 closed with repository-wide evidence. +Separate composition-envelope shape decoding, cross-field semantics, and +root-only metadata removal while retaining the source-aware configuration model +that imports, profiles, diagnostics, and inspection commands require. ### Required Work -1. Review existing force/selection/reconciliation tests and retain a lean set - that proves: - - an ordinary run reuses every valid current selected artifact; - - unqualified force reruns the full effective configured target set; - - selected force reruns the explicit targets, includes their dependency - closure, reuses valid current prerequisites, and leaves unrelated valid - artifacts untouched; - - stale, failed, missing, or dependency-invalid prerequisites execute in - deterministic topological order before their dependents; and - - an incidental canonical file without current manifest evidence is never - reused. -2. Replace projection-oriented test names, helpers, and assertions with - checkpoint/recovery language. Remove redundant tests whose only purpose was - to validate the retired transport or rollback choreography. Do not reduce - coverage of output validation, dependency invalidation, cancellation, - persistence failure, or assembled resume behavior. -3. Update the canonical current-behavior owners: - - `docs/internal/stage-analyze.md` for checkpoint order, resume behavior, - failure semantics, and the absence of projections; - - `docs/internal/manifest.md` for per-artifact checkpoint authority, - aggregate status, interruption recovery, and persistence failure; - - `docs/policy/architecture.md` only where its stage/persistence description - currently implies a single end-of-stage analyze transaction; and - - `docs/operations.md` and `docs/troubleshooting.md` only where operators - need to understand reuse after partial analyze failure or recover from a - manifest checkpoint error. - Do not duplicate CLI or configuration contracts, since syntax and fields do - not change. -4. Search production, tests, and current-behavior documentation for - `AnalyzeStateProjection`, `AnalyzeState`, `analyze projection`, - `projected records`, and `end-of-stage` language. Remove or revise every - reference tied to the retired mechanism; preserve unrelated planning uses of - the ordinary English word “projection” only when accurate. -5. Run the full repository validation from `docs/development.md`: +1. Refactor `splitPipelineCompositionEnvelope` in + `internal/config/pipeline_composition.go` into a short coordinator over + three explicit private phases: + - locate and decode the root `composition` mapping and its `imports`, + `default_profile`, and `profiles` shapes; + - validate cross-field semantics, including default-profile membership; and + - clone/remove the root-only envelope to produce the ordinary pipeline body. +2. Reuse `parsePipelineProfileDeclaration` and profile-name normalization. + Keep unknown-field rejection, non-empty profile declarations, imports and + profiles in declaration order, explicit-profile selection behavior, and + source-qualified shape/conflict diagnostics. +3. Do not mutate the parsed root document during validation. Return a distinct + body document only after the envelope is valid; the original document must + remain reusable by callers and tests. +4. Remove `compositionNode` and `compositionField` metadata that has no runtime + reader after the refactor. Specifically verify and remove redundant cached + `order`, `line`, and `column` fields and their clone/renumber assignments if + they remain write-only. Slice order remains the declaration-order + representation; YAML parser nodes remain the source of parse-time line and + column diagnostics. +5. Retain each node's semantic path and source list because merge conflict + reporting, ownership projection, effective digests, artifact-family source + projection, and `config sources` consume them. Do not reduce actionable + conflict diagnostics or delete provenance that has a real consumer. +6. Keep composition product-specific. Do not replace the tree with generic map + decoding or adopt a third-party composition library. + +### Tests And Verification + +- Test malformed envelope shapes, unknown fields, invalid profile declarations, + missing default membership, successful envelope removal, and absence of + caller-document mutation through the composition boundary. +- Preserve tests for explicit empty/false/zero/list/map values, additive + conflicts naming all claiming sources, overlay semantics, canonical YAML, + effective digest stability, profile selection, and `config sources` + ownership. These are semantic contracts, not incidental provenance. +- Remove assertions of deleted private cached metadata. Continue asserting + declaration behavior through ordered outputs rather than private index + fields. +- Run: + + ```sh + go test ./internal/config -run 'Composition|PipelineProfile|Effective|ArtifactFamil' + go test ./internal/app -run 'Config' + git diff --check + ``` + +### Exit Criteria + +- Envelope decoding, cross-field validation, and body production have separate + focused owners under one coordinator. +- Only provenance and ordering data with a demonstrated consumer remains. +- Imports, profiles, overlays, source reporting, and effective digests retain + deterministic behavior. + +## Stage 6 — Align Diagnostic Tests, Documentation, And Close Workstream 7 + +**Status:** Pending + +### Goal + +Review the completed configuration refactor as one system, remove obsolete test +friction and documentation language, and close Workstream 7 with repository-wide +evidence. + +### Required Work + +1. Review every configuration test changed by Stages 1–5. Retain coverage of + schema strictness, normalization, semantic invariants, meaningful failure + categories, deterministic outcomes, composition source conflicts, effective + projections/digests, and representative assembled loads. +2. Rewrite assertions that pin complete diagnostic prose, punctuation, + formatting, private helper order, or an arbitrary independent-error winner. + Prefer a relevant field path plus one small semantic fragment. Use + `errors.Is`/`errors.As` only where an existing wrapped or typed cause is a + real contract; do not introduce types solely to change test style. +3. Delete redundant legacy-form enumerations whose only purpose is to preserve + knowledge of retired fields or spellings already rejected by the canonical + strict/source boundary. Retain representative tests proving unknown fields + and unsupported sources fail, and retain migration guidance tests only when + that guidance is an intentional user-facing contract. +4. Consolidate oversized table fixtures only where multiple cases protect the + same rule. Do not delete distinct risks merely to reduce test LOC, and do not + rewrite healthy semantic tests just because they use a small stable error + fragment. +5. Update [`../internal/configuration.md`](../internal/configuration.md) to + describe the implemented validation-domain ownership, deterministic phase + behavior, composition-envelope phases, and diagnostic guarantees without + listing private helper choreography. Update [`../config.md`](../config.md) + only if its user-visible current-behavior wording is inaccurate; no schema + change is expected. +6. Search current production, tests, examples, and current-behavior docs for + active acceptance of the retired configuration forms named by C01. Remove + actual compatibility code and stale current-behavior claims, while leaving + immutable audit/history documents unchanged. +7. Run the full repository validation from `docs/development.md`: ```sh go test ./... @@ -436,27 +500,25 @@ language, and leave Workstream 6 closed with repository-wide evidence. go test ./internal/config -run '^TestExamplesLoadAndValidate$' ``` -6. Run `go test -race -shuffle=on -count=3 ./internal/stage ./internal/app` to - stress the changed recovery and persistence boundaries. Run the supported - release cross-build checks if an implementation change affects build-tagged - code; otherwise the repository build is sufficient for this workstream. -7. After all checks pass, mark Stages 1–5 in this document `Completed` and mark - only Workstream 6 in [`refactoring.md`](refactoring.md) `Complete`. - Workstream 7 must remain the next pending workstream. +8. Run `go test -race -shuffle=on -count=3 ./internal/config ./internal/app` + to stress map-order independence and assembled configuration behavior. +9. After all checks pass, mark Stages 1–6 in this document `Completed` and mark + only Workstream 7 in [`refactoring.md`](refactoring.md) `Complete`. + Workstream 8 must remain the next pending workstream. ### Exit Criteria -- Current documentation describes analyze as a session-manifest-backed, - per-artifact checkpointed queue and contains no retired projection behavior. -- The force, partial-selection, dependency, interruption, and persistence-error - cases are protected through stable behavioral boundaries. -- All repository validation and focused shuffled race checks pass. -- Workstream 6 is marked complete, Workstream 7 remains pending, and the - worktree contains no accidental generated or temporary files. -- `git diff --check` succeeds. +- Configuration remains strict, deterministic, source-aware where useful, and + unchanged in public schema and normalized semantics. +- Validation code is organized by semantic owner; Notarius, Scriptorium, and + composition coordinators expose clear phases without generic frameworks. +- No active retired configuration compatibility path, write-only composition + metadata, or test dependency on incidental diagnostic choreography remains. +- Current documentation describes the implemented boundaries, all validation + commands pass, Workstream 7 is complete, and Workstream 8 remains pending. ## Open Questions -None. The audits, approved Workstream 6 scope, existing selection contract, and -the fixed decisions above resolve the product and implementation choices needed -for the coding agent to execute this plan. +None. The audits, completed compatibility purge, current configuration +contracts, and fixed decisions above resolve the product and implementation +choices needed for a `gpt-5.6-terra` coding agent to execute this plan.