eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 27 Wiki Activity

Plan configuration refactoring

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-04 23:37:02 +00:00
parent 74ab9218ff
commit 7a0e2cb11b
1 changed files with 435 additions and 373 deletions
Download Patch File Download Diff File Expand all files Collapse all files

808
docs/roadmap/implementation.md
View File

@@ -1,431 +1,495 @@
# Workstream 6 Implementation Plan — Per-Artifact Analyze Checkpointing # Workstream 7 Implementation Plan — Configuration Structure And Diagnostics
## Status ## Status
Completed implementation record for Workstream 6, **Per-artifact analyze Implementation plan for Workstream 7, **Configuration structure and semantic
checkpointing and recovery**, in [`refactoring.md`](refactoring.md). Workstreams diagnostics**, in [`refactoring.md`](refactoring.md). Workstreams 1 through 6
1 through 6 and all stages below are complete. Workstream 7 is the next pending are complete and are prerequisites for this plan. All stages below are pending
workstream; its implementation is outside this plan. and must be implemented in numerical order.
The purpose of this workstream is to make analyze a deterministic resumable The purpose of this workstream is to preserve Narratio's strict, deterministic,
artifact queue backed only by the session manifest. If artifacts A and B operator-oriented configuration behavior while making the implementation
succeed, C fails, and D is not attempted, A and B must remain reusable, C must navigable by semantic domain. It also removes incidental composition metadata
be retried, and D must follow after C succeeds. The implementation must remove and test expectations that turn private validation choreography into a
the current end-of-stage analyze projection and rollback machinery rather than compatibility burden. This is an internal refactor with narrowly approved
building checkpointing alongside it as a second durable model. diagnostic flexibility, not a configuration-language redesign.
## Authoritative Scope And Decisions ## Authoritative Scope And Decisions
The authoritative product and architecture scope is Workstream 6, the The authoritative scope is Workstream 7 in [`refactoring.md`](refactoring.md).
dependency relationships, compatibility intent, and consolidated target state Its supporting audit entries are H04, H14, H15, H17, H38, H39, H40, H41, and
in [`refactoring.md`](refactoring.md). Supporting decisions are H04, H14, H15, H42 in [`audit.md`](audit.md), C01 and C06 in
H36, H37, C02, and C03 in [`audit-decisions.md`](audit-decisions.md), with the [`contract-audit.md`](contract-audit.md), and the corresponding rows in
detailed rationale in H36 and H37 of [`audit.md`](audit.md) and Finding 3 of [`audit-decisions.md`](audit-decisions.md).
[`contract-audit.md`](contract-audit.md).
The following decisions are fixed for this implementation: The following implementation decisions are fixed:
- Keep analyze as one fixed pipeline stage in `internal/stage`; per-artifact - Keep all configuration behavior in package `internal/config`. Do not create
work does not become dynamic pipeline stages or a generic workflow engine. subpackages, a generic configuration framework, a schema framework, a
- Keep the session manifest's `analyze_artifacts` collection as the only durable validator registry, or a dependency-injection layer.
per-artifact authority. Do not introduce an invocation collection, journal, - Preserve the public configuration types and entry points. `Validate`,
sidecar, database, or replacement run manifest. `ValidatePipelineConfig`, `ValidateCampaignConfig`, all load APIs, effective
- Use the existing `stage.Env.ManifestStore` and the canonical session-manifest pipeline projections, and configuration-inspection commands retain their
path for checkpoints. Production execution already supplies that store and current responsibilities.
holds the session lock for the entire `Stage.Run` call. Do not add a callback - Keep `validate.go` as the short public and high-level orchestration owner.
framework or expose application lifecycle types to `internal/stage`. Organize private validation into cohesive files for core pipeline/session,
- Checkpoint a copy-on-write candidate manifest. Validate and save the candidate adapters, publish/storage, Notarius, Scriptorium, and shared scalar/path
before installing its analyze collection into the live in-memory manifest. helpers. File names are private, but those ownership boundaries are the
A failed current-record checkpoint must never make newly materialized bytes intended final structure.
authoritative through a later terminal save. - Do not replace ordinary `error` returns with a new exported validation-error
- Persist the reconciled starting collection before the first adapter call when hierarchy. No machine consumer currently requires one, and introducing one
it changes durable authority. This removes obsolete records and stales would add a new compatibility surface. Field paths, source identities for
scheduled or invalid current records before replacement work begins. composition conflicts, wrapped causes where useful, and small semantic
- For every successful artifact: validate and materialize its canonical output, phrases provide the stable diagnostic content tests should assert.
construct and validate its current record, update the runtime catalog, stale - Continue to fail fast. The workstream does not add error aggregation and does
transitive dependents when output identity changed, checkpoint the complete not promise which of several independent invalid fields is reported first.
session collection, and only then attempt the next artifact. Traversal of maps must nevertheless be deterministic so identical input
- For an artifact failure: retain earlier successful checkpoints, replace the produces identical behavior.
active artifact with a bounded failed record, stale its current transitive - Preserve strict single-document YAML decoding, duplicate- and unknown-field
dependents, and expose that collection on the live manifest. The application rejection, additive imports with no duplicate ownership, one deterministic
runner remains responsible for marking aggregate analyze failed, invalidating profile selection, explicit-value overlay behavior, centralized defaults,
publish/notify, and atomically persisting the terminal manifest before the canonical party resolution, artifact-family expansion, and secret-free
command returns the error. effective digests.
- If persistence of a successful-artifact checkpoint fails, stop immediately. - Preserve the current configuration schema. Campaign-owned
Keep the last durable collection, mark the active artifact failed in the live `campaign.inputs.party_file` and canonical artifact source IDs are current
collection without promoting its incidental output, and let the runner's contracts, not legacy fields. Removed player/session roster fields and old
terminal failure save make that retry state durable when storage permits. artifact source spellings remain unsupported; add no compatibility aliases
- Preserve the existing deterministic work planner, reconciliation, or normalization branches for them.
fingerprints, evidence validation, dependency order, and current-output - Treat Workstream 3's compatibility purge as authoritative. During this work,
identity comparison. File presence alone never creates authority. remove any surviving production branch that actually accepts or translates
- Preserve artifact selection and force behavior. With no `--artifacts`, the a retired configuration representation. A rejection supplied naturally by
effective configured set is targeted. With `--artifacts`, only the named strict decoding or the canonical source validator is sufficient; do not
targets and their required dependency closure participate. Force reruns maintain exhaustive legacy-form tests when representative canonical-boundary
explicit targets; a valid current prerequisite is reused unless it is itself tests already protect the rule.
explicit or becomes non-current through dependency work. Unrelated valid - Preserve composition source information that is consumed by conflict
records are retained. diagnostics or `config sources`. Remove only demonstrably unused node/field
- On aggregate success, return ordinary `StageResult.Outputs` rebuilt in metadata. In particular, source lists and semantic field paths remain;
deterministic normalized artifact-key order from all current session analyze redundant cached declaration-order indexes and copied line/column fields may
records. Per-artifact records retain their own logs, generated configuration be removed when no consumer reads them because slice order and the YAML
references, bounded Scriptorium provenance, and evidence; aggregate logs and parser already supply the required behavior.
metadata continue to describe the current invocation. - Preserve normalization as an all-or-nothing operation for each validator.
- On aggregate failure, aggregate outputs remain empty. Earlier current Notarius reference/output maps and scalar paths are installed on the config
per-artifact records remain available to a later analyze run, but publish and only after all relevant checks succeed; a validation error must not leave a
notify are invalidated by the existing application-stage dependency relation. partially normalized value behind.
- A crash or cancellation after a successful checkpoint may leave aggregate - Keep `validateAudita` intact when moving it to the adapter-validation owner.
analyze `running` or later normalized to `interrupted`; the next ordinary run Its guards are cohesive; splitting them further would add names without
must reconcile the durable collection, reuse valid completed artifacts, and reducing complexity.
continue with the first failed, stale, or missing artifact. - Tests must protect semantic rules and observable normalized results, not
- Keep the manifest schema and analyze-state contract version unchanged. This private helper calls, file placement, full error prose, punctuation, or one
work changes persistence timing and removes an internal projection transport; historical validation call order. Exact strings remain appropriate only for
it does not introduce a new serialized representation. an intentionally documented CLI or machine-consumed contract.
- Do not enter Workstream 7 configuration decomposition, Workstream 8 publish - Do not enter Workstream 8 remote publication/lock decomposition, Workstream 9
locks/decomposition, or Workstream 9 cleanup-stage work. cleanup-stage work, or unrelated Workstream 10 cleanup.
## Instructions For Every Stage ## Instructions For Every Stage
Before each stage: Before each stage:
1. Read `docs/development.md`, every document under `docs/policy/`, this plan, 1. Read `docs/development.md`, every document under `docs/policy/`, this plan,
and Workstream 6 in [`refactoring.md`](refactoring.md). Read the decision and Workstream 7 in [`refactoring.md`](refactoring.md). Read only the audit
entries and audit sections named above; do not re-audit unrelated findings. sections and decision-register rows cited above; do not restart the audits.
2. Read [`../internal/stage-analyze.md`](../internal/stage-analyze.md) and 2. Read [`../internal/configuration.md`](../internal/configuration.md) and the
[`../internal/manifest.md`](../internal/manifest.md), then inspect the affected portions of [`../config.md`](../config.md), then inspect the
production files and focused tests named by the stage. production files and focused tests named by the stage.
3. Prefer the codebase knowledge graph for Go symbol discovery and call tracing. 3. Prefer the codebase knowledge graph for Go symbol discovery and call
Use text search for serialized fields, documentation, fixtures, and removal tracing. Use text search for YAML fields, source IDs, error fragments,
of projection terminology. examples, documentation, and removal verification.
4. Confirm the worktree state and preserve unrelated changes. Do not begin a 4. Confirm the worktree state and preserve unrelated changes. Do not implement
later workstream merely because adjacent code is visible. an adjacent pending workstream merely because it shares a file.
During every stage: During every stage:
- Keep the repository compiling and focused tests passing. A temporary private - Keep the package compiling and focused tests passing. Move behavior before
delegation or projection bridge is permitted only where the next numbered redesigning the two complex validators so reviewable commits do not combine
stage explicitly removes it; do not leave two durable authorities active. unrelated mechanical and semantic changes.
- Preserve package ownership and acyclic dependency direction. Keep new types - Use package-private functions with explicit inputs and return values. Avoid
private and cohesive; do not add public extension points. stateful validator objects, forwarding-only wrappers, generic rule tables,
- Keep all tests deterministic, offline, and independent of real Scriptorium or and abstractions whose only purpose is to reduce function length.
paid APIs. Prefer the existing fake runner, real temporary filesystem, and a - Keep configuration mutation explicit and late. Do not mutate caller-owned
small recording/failing manifest store only where checkpoint interaction is maps while validating a temporary normalized candidate.
itself the contract. - Retain deterministic order wherever config maps are traversed for validation,
- Assert persisted manifest state, adapter calls, dependency order, and retry normalization, canonical output, source reporting, or digest construction.
behavior. Do not pin private helper choreography, complete error wording, or - Keep tests deterministic, offline, and independent of credentials or live
incidental timestamps. adapters. Prefer direct package behavior and real temporary files over mocks.
- Preserve bounded, non-secret errors and diagnostics. Never place prompt or - Use `gofmt` on changed Go files, run the focused checks listed for the stage,
transcript content, variables containing secrets, or raw credentials in new
manifest fields or error assertions.
- Use `gofmt` on changed Go files, run the focused tests listed for the stage,
and run `git diff --check`. Mark a stage `Completed` only after its exit and run `git diff --check`. Mark a stage `Completed` only after its exit
criteria pass. criteria pass.
## Stage 1 — Decompose The Single-Artifact Transaction ## Stage 1 — Separate Core And Shared Validation Ownership
**Status:** Completed **Status:** Pending
### Goal ### Goal
Make one artifact execution understandable and independently reviewable before Reduce `validate.go` to configuration validation entry points and readable
changing its persistence timing. This stage is behavior-preserving. high-level sequencing, while giving core pipeline/session rules and reusable
primitive checks clear package-local owners. This stage is behavior-preserving.
### Required Work ### Required Work
1. Refactor `executeAnalyzeArtifact` in `internal/stage/analyze.go` into one 1. Keep these entry points and the visible pipeline-domain call sequence in
short transaction owner plus private phases with explicit intermediate `internal/config/validate.go`:
values. Keep the existing `analyzeExecutionContext` and artifact plan, or - `Validate`;
replace them with equally narrow private values; do not restore the former - `ValidatePipelineConfig`;
broad parameter list. - `ValidateCampaignConfig`; and
2. Separate these concerns without changing their order or contracts: - the short `validatePipeline` coordinator.
- input identity resolution, omitted optional inputs, reused-current input 2. Move core configuration checks into a cohesive private file such as
reporting, Scriptorium variables, timeout, canonical/run-local paths, and `validate_core.go`. This owner includes campaign validation, secrets,
request/diagnostic path preparation; spool/cache, session validation, resolved-party input validation,
- optional render-debug invocation and validation; session/campaign identifiers, and cross-pipeline/session rules. Keep
- Scriptorium artifact invocation and adapter failure adaptation; `publishUploadConfiguredForS3` with publish/storage in Stage 2 rather than
- authoritative output-path selection, bounded non-empty regular-file hiding publish policy in the core owner.
validation, output-contract construction, canonical materialization, and 3. Move general validation primitives into a small private file such as
installed-checksum verification; and `validate_helpers.go`: environment-variable name checks, positive duration
- result/evidence construction, including provenance, logs, generated parsing, safe relative paths, and path-within-root checks. Do not turn these
configuration paths, metadata, output size, and reused artifacts. functions into a generic validation DSL.
3. Place the extracted code in focused analyze-owned files when that improves 4. Re-home the straightforward stage/adapter validators in a private adapter
navigation. Suggested responsibilities are preparation, adapter invocation, owner such as `validate_adapters.go`: notification, normalize, trim, render,
and validated materialization/evidence; exact private filenames are not a WhisperX, Seriatim (including its output schema), and Audita. Move
contract. `validateAudita` intact apart from import or formatting changes.
4. Keep Scriptorium request and response types inside the stage/adapter boundary. 5. Preserve current nil handling, defaults, normalization, wrapped causes,
Do not move Narratio artifact, manifest, path, or metadata policy into the field identification, and high-level validation semantics. Do not use this
Scriptorium adapter. mechanical move to reorder domains or rewrite diagnostics.
5. Preserve the current run-local-before-canonical materialization boundary, 6. Remove imports and dead private helpers left behind in `validate.go`. Do not
render-debug behavior, useful error context, workspace modes, centralized add facade functions to preserve former source-file placement; files are not
size limit, and checksum comparison. Do not weaken validation or add a new Go API boundaries.
write path while moving code.
6. Retain existing behavior tests at their stable boundaries. Consolidate only ### Tests And Verification
tests that become purely coupled to moved private helpers; do not add one
test per extracted function. - Existing core, campaign, session, duration, normalize, trim, render,
notification, and load/validate tests should normally pass unchanged.
- Add no tests that merely assert a function moved files. Add or repair a test
only if the move exposes an unprotected semantic rule.
- Run:
```sh
go test ./internal/config
go test ./internal/app -run 'Config|Pipeline|Session'
git diff --check
```
### Exit Criteria ### Exit Criteria
- `executeAnalyzeArtifact` reads as a short ordered transaction over explicit - `validate.go` is a concise entry/coordinator file rather than a container for
phases and no longer mixes all preparation, invocation, materialization, and unrelated domain implementations.
evidence construction in one large function. - Core, adapter, and shared helper ownership is obvious from the file layout,
- Existing analyze request, render, validation, materialization, provenance, and Audita remains one cohesive validator.
and failure behavior is unchanged. - No public behavior or serialized configuration shape changed.
- `go test ./internal/stage ./internal/adapters/scriptorium` succeeds.
- `git diff --check` succeeds.
## Stage 2 — Establish Session-Only Analyze Execution State ## Stage 2 — Separate Publish, Storage, Notarius, And Scriptorium Owners
**Status:** Completed **Status:** Pending
### Goal ### Goal
Replace `analyzeStage.Run`'s parallel local collections with one private Finish the domain-level split of the former aggregate validator before changing
session-oriented execution state that is ready to checkpoint, while retaining the internal structure of Notarius or Scriptorium validation.
the existing end-of-stage persistence boundary temporarily.
### Required Work ### Required Work
1. Introduce a private analyze execution state/accumulator that owns: 1. Move storage, publish output, and publish lock validation into a cohesive
- the shared execution context and deterministic work plan; private owner such as `validate_publish.go`. Move the associated helpers
- the complete reconciled session artifact collection; with their consumers, including `ValidatePublishLockRules`,
- prior current records needed for output-identity comparison; `MergePublishLockRules`, configured output/source lookups, and
- the evaluated/scheduled key set used for dependent invalidation; `publishUploadConfiguredForS3`.
- the mutable runtime artifact catalog; and 2. Preserve the exported lock-rule helper signatures because current app and
- invocation-only aggregate logs, generated configurations, generated stage code consume them. Do not redesign remote lock behavior; that belongs
metadata, reused-artifact metadata, and its deduplication set. to Workstream 8.
2. Keep setup, reconciliation, work planning, adapter-availability validation, 3. Move the existing Notarius validator into `validate_notarius.go` and its
deterministic iteration, and final `StageResult` assembly visible in Scriptorium counterpart, input-source validator, and exported dependency
`analyzeStage.Run`. Move per-item state mutation and diagnostic collection validator into `validate_scriptorium.go`. At this stage, move these functions
behind narrow methods on the private state; do not turn the accumulator into without decomposing or semantically reordering them; Stages 3 and 4 own those
a service locator. changes.
3. Rename `analyzeWorkPlan.ProjectedRecords`, `projectedAnalyzeRecords`, and 4. Keep source-vocabulary and collision helpers with the narrowest shared
related private terminology to describe the reconciled starting session domain owner. If both publish and Scriptorium need a small lookup derived
collection. It is the candidate durable state, not an invocation projection. from Notarius/Scriptorium configuration, use one plainly named package
Preserve the planner's read-only behavior and existing classification. helper rather than duplicate maps or introduce a registry.
4. Make the accumulator install its complete collection into the passed session 5. Ensure `validate.go` contains only entry points and high-level sequencing
manifest on success and its failed/staled collection on error. During this after the move. Delete the emptied blocks and unused imports.
stage, continue returning the existing `AnalyzeStateProjection` as a 6. Search `internal/config` production code for acceptance or translation of
temporary compatibility bridge for the application runner; both views must removed player/session roster fields and retired artifact source spellings.
be derived from the same collection, and no additional save is introduced. Delete any actual compatibility branch found. Do not mistake current
5. Add optional `ProducerRunID` provenance to `artifacts.Ref`. Update the campaign `party_file`, rejection through strict decoding, migration prose,
generic application result mapper to use that value when it is non-empty and or a test fixture demonstrating rejection for an active compatibility path.
otherwise retain the current behavior of assigning the active run ID. This
is data needed by ordinary stage-result transport, not an analyze-specific ### Tests And Verification
branch; cover both explicit and fallback behavior in the mapper tests.
6. Add a private helper that deterministically converts all current analyze - Existing storage/publish, Notarius, Scriptorium, configuration loading, and
records into ordinary `artifacts.Ref` outputs. Preserve source ID, relative application configuration tests should pass without pinning file placement.
path, contract, checksum, and producer run ID. Include valid current records - Run:
retained outside a partial selection; do not include stale, missing, or
failed records. Continue using the projection path for aggregate outputs ```sh
until Stage 3 switches ownership. go test ./internal/config
7. Rewrite stage tests to inspect the manifest collection mutated by the real go test ./internal/app ./internal/stage
analyze stage rather than manually installing returned projections. Retain a git diff --check
minimal assertion that the temporary projection matches that collection so ```
the transition cannot silently diverge.
### Exit Criteria ### Exit Criteria
- `analyzeStage.Run` coordinates one private session collection rather than - Every validation domain named by Workstream 7 has one discoverable file-level
parallel record and diagnostic variables. owner inside `internal/config`.
- Planning/reconciliation remains read-only, execution order and force/selection - There is no duplicate source/collision logic and no active retired-format
behavior are unchanged, and stage tests primarily assert manifest state. compatibility branch in configuration production code.
- Ordinary `StageResult.Outputs` can preserve a reused artifact's original - Notarius and Scriptorium semantics remain unchanged pending their dedicated
producer run identity without an analyze-specific application path. stages.
- The existing application projection path still functions and no checkpoint
timing has changed yet.
- `go test ./internal/stage ./internal/app` succeeds.
- `git diff --check` succeeds.
## Stage 3 — Cut Analyze Over To Durable Per-Artifact Checkpoints ## Stage 3 — Decompose Notarius Validation And Normalization
**Status:** Completed **Status:** Pending
### Goal ### Goal
Make each validated artifact completion durable before later artifact work and Make `validateNotarius` a short transaction that validates top-level policy,
stop using the end-of-stage projection transport for real analyze execution. normalizes references, normalizes output lanes, and commits the resulting
configuration only after every check succeeds.
### Required Work ### Required Work
1. Add a private analyze checkpoint operation in `internal/stage`. It must: 1. Keep `validateNotarius` responsible for nil/enablement behavior and the
- require a non-nil session manifest, analyze stage record, and top-level scalar contract: required binary, configuration path, pipeline ID,
`Env.ManifestStore` when analyze has state to persist; positive timeout, working directory, at least one output, and the reference
- derive the canonical manifest path through `artifacts` helpers from the binding limit.
resolved workspace, campaign, and session identity; 2. Replace the validator's bespoke timeout parsing with the shared positive
- validate the full candidate collection with duration helper when it preserves the same accepted values and field
`manifest.ValidateAnalyzeArtifactCollection`; identity. Do not embed duration policy in a Notarius-only branch.
- shallow-copy the manifest, clone the top-level stage map, copy the analyze 3. Extract reference binding normalization into one private operation that:
stage record, and deep-clone the candidate analyze collection so the save - iterates raw selector keys in sorted order;
cannot mutate live authority before success; - normalizes selectors through `notariusref`;
- preserve the aggregate analyze status (normally `running`) and every - rejects normalized selector collisions;
unrelated manifest field; - trims and validates each prepared-input source; and
- save the candidate through `Env.ManifestStore`; and - returns a newly allocated normalized map without mutating the input map.
- only after success, install the candidate analyze version/collection and 4. Extract output normalization and collision validation into a separate
the store-updated manifest timestamp into the live manifest. private operation that:
Do not add a general manifest transaction framework or JSON round-trip clone. - builds the reserved built-in, configured Scriptorium, and previous-session
2. Before invoking the first scheduled artifact, checkpoint the reconciled source set once;
starting collection when it differs from durable state. This must remove - iterates raw output keys in sorted order;
records for removed configuration and stale any selected/forced/non-current - validates and normalizes configured keys and derived extraction source
authority before replacement work begins. If this save fails, invoke no IDs;
adapter. - rejects reserved-source, normalized-source, and lane-ID collisions;
3. For each scheduled artifact, preserve this order: - trims lane/media/schema/module scalar fields and checks required fields;
- compute its execution fingerprint; and
- execute the decomposed artifact transaction; - returns a newly allocated normalized output map.
- construct and validate its current manifest record; 5. Normalize binary, paths, pipeline ID, timeout, and working directory into
- prove the generated record can be added to the runtime catalog for use by local candidate values. Assign all scalar candidates, references, and
later dependents; outputs back to `cfg` only after both helper operations succeed. Preserve
- stale unscheduled transitive dependents only when prior and replacement nil-versus-empty map behavior where it is observable in effective YAML or
output identities differ; digests.
- checkpoint the complete updated session collection; and 6. Keep all helpers Notarius-specific and private. Do not move selector policy
- only then expose the result to a later artifact. out of `notariusref`, copy artifact source policy, or create a generalized
4. If a successful artifact's checkpoint fails, stop immediately and do not map-normalization framework.
call the next artifact. Build a failed record for the active artifact from
the last committed collection, with no output authority; stale its current ### Tests And Verification
transitive dependents; and install that failure candidate in memory so the
runner's terminal save can persist retry state. Return the checkpoint - Retain lean behavior tests for disabled configuration, required scalars,
failure with artifact identity and manifest context. timeout and reference-binding limits, selector normalization/collisions, supported
5. Have real analyze execution return ordinary `StageResult.Outputs` built from prepared sources, output normalization, reserved-source collisions,
every current session record plus its existing aggregate diagnostics and duplicate lanes, and cross-use by Scriptorium/publish.
metadata. Stop returning `AnalyzeStateProjection` from `analyzeStage.Run`. - Add or consolidate one test proving a failing reference or output candidate
The legacy projection type and application handlers may remain only as dead leaves the original Notarius config unmodified.
transitional code until Stage 4; add no new uses. - Add one determinism test that constructs equivalent reference/output maps in
6. In `executeSelectedStage`, make a terminal save failure after any stage's different insertion orders and compares normalized success or equivalent
apparent success mark that stage failed and invalidate succeeded transitive diagnostic classification. Do not assert which of multiple independent
dependents before terminal persistence. This behavior currently occurs only invalid domains must win forever.
on the analyze-projection branch; it must remain correct when real analyze no - Assert field paths and the smallest meaningful semantic fragment rather than
longer returns a projection and should be correct generically. complete error sentences.
7. Update direct analyze fixtures so they supply a real temporary - Run:
`manifest.LocalStore`, complete campaign/run identity, an aggregate analyze
record in the state appropriate to the test, and an initially persisted ```sh
manifest. Keep unit-only planner/reconciliation tests free of persistence. go test ./internal/config -run 'Notarius|Scriptorium|Publish|Duration'
8. Add focused checkpoint tests at the stage/application boundary: go test ./internal/stage -run 'Extract|Analyze'
- the manifest store observes A current before B is invoked and A+B current git diff --check
before C is invoked; ```
- a failed B checkpoint prevents C from being invoked;
- the failed B checkpoint never persists B as current merely because its
canonical file exists; and
- final aggregate success exposes outputs for all and only current records
in deterministic normalized artifact-key order, preserving each record's
producer run identity.
### Exit Criteria ### Exit Criteria
- Every successful scheduled artifact is durable before the next adapter call. - `validateNotarius` visibly sequences scalar validation, reference
- A checkpoint failure stops later work and cannot promote incidental output. normalization, output normalization/collision checks, and one final commit.
- Real analyze execution no longer relies on an end-of-stage state projection. - Reference and output collision domains are separate, deterministic, and do
- Aggregate success/failure persistence and dependent invalidation remain not partially mutate caller-owned configuration on failure.
application-owned and correct. - No Notarius behavior moved into a generic configuration abstraction.
- `go test ./internal/stage ./internal/app ./internal/manifest` succeeds.
- `git diff --check` succeeds.
## Stage 4 — Complete Failure Recovery And Remove Projection Machinery ## Stage 4 — Decompose Deterministic Scriptorium Validation
**Status:** Completed **Status:** Pending
### Goal ### Goal
Complete the A/B/C/D recovery contract and delete the superseded projection, Separate Scriptorium artifact declarations/outputs, dependencies/inputs, and
validation, application, and rollback paths. variables into readable validation phases with deterministic map traversal.
### Required Work ### Required Work
1. On any artifact-specific failure—including fingerprinting, preparation, 1. Keep `validateScriptorium` as the transaction owner for nil behavior,
render, adapter execution, result validation, materialization, evidence required binary, optional config path, stage timeout, ordered validation
construction, catalog admission, or checkpoint persistence—construct one phases, referenced-output checks, and the final dependency-graph check.
bounded failed record for the active key, clear its output authority, and 2. Produce one sorted list of configured artifact keys after validating their
stale current transitive dependents. Preserve unrelated current records and canonical key syntax. Reuse that order across phases rather than ranging
all earlier successfully checkpointed records. over the artifact map independently.
2. Install the failed/staled collection on the live manifest before returning 3. Extract a declaration/output phase that validates enabled prompt and output
the artifact error. The runner must then use its ordinary stage-error path to requirements, safe output paths, artifact timeout values, and duplicate
mark aggregate analyze failed, clear aggregate outputs, invalidate publish output ownership. Keep its collision map local to that phase.
and notify, and persist the terminal session manifest. Join a terminal save 4. Extract a dependency/input phase that:
failure with the originating artifact/checkpoint error; never replace the - validates dependency entries and self-reference using declared slice
useful cause. indexes for actionable field paths;
3. Ensure unattempted artifacts behave according to existing authority: - iterates input maps by sorted input name;
previously absent records remain absent/missing on reconciliation; unrelated - validates required source declarations through the existing canonical
valid current records remain current; and a prior current record that input-source owner;
depends directly or transitively on the failed artifact becomes stale with - requires configured-artifact sources to appear in `depends_on`; and
no output authority. - returns or updates only the small referenced-artifact set needed by the
4. Remove `AnalyzeStateProjection` from `stage.StageResult`, delete later referenced-output check.
`internal/app/analyze_projection.go`, and remove all validation, cloning, 5. Extract a variable phase that iterates variable names in sorted order,
application, capture, rollback, and special output reconstruction that exists rejects empty names, and accepts only the currently supported string and
solely for that projection. Remove fake projection stages and projection-only boolean values.
tests; replace their meaningful risks with real analyze checkpoint/recovery 6. Iterate the referenced-artifact set deterministically when checking required
coverage. outputs. Keep `ValidateScriptoriumArtifactDependencies` as the public graph
5. Simplify `executeSelectedStage` to its generic result/error handling while validator and preserve its cycle and dependency semantics.
preserving analyze's live manifest collection across aggregate lifecycle 7. It is acceptable for phase separation to change which of several unrelated
clearing. Do not add a type assertion, stage-name branch, callback, or new simultaneous mistakes is reported first. Within a given implementation,
result union to replace the projection special case. equivalent maps must produce the same result. Do not add compatibility code
6. Add a durable assembled-run regression for the approved scenario: to reproduce the old mixed per-artifact call order.
- configure deterministic A, B, C, and D work in dependency order; 8. Remove forwarding-only helpers and obsolete temporary sets after the split.
- make A and B succeed, C fail, and verify D is not attempted; Keep `validateScriptoriumInputSource` cohesive unless a genuine duplicate
- reload the manifest and verify A/B current, C failed, aggregate analyze semantic owner emerges.
failed, and the correct missing/stale state for D;
- run again without force, verify only C and then D execute, and verify the ### Tests And Verification
final aggregate and all selected records are current; and
- verify valid unrelated records and upstream transcript stages were never - Retain semantic coverage for declaration keys, enabled requirements, output
invalidated or rerun. path safety/collisions, timeouts, dependency existence/self/cycles, input
7. Add focused variants for failure of the first and last artifact, cancellation source classes, extraction references, required `depends_on`, referenced
after at least one checkpoint, changed-output invalidation of an unselected outputs, and variable types.
transitive dependent, and byte/contract-identical replacement preserving an - Add one map-insertion-order test that feeds equivalent Scriptorium maps in
unselected dependent. Consolidate with existing incremental/failure tests so different insertion orders and asserts equal validity or equivalent
each durable risk has one clear owner. diagnostics. Do not assert an arbitrary globally preferred independent-error
order.
- Consolidate table cases that repeat the same source-vocabulary rule through
different retired spellings when a representative canonical rejection test
already protects unsupported-source behavior.
- Run:
```sh
go test ./internal/config -run 'Scriptorium|ArtifactFamil|Notarius|Publish'
go test ./internal/stage -run 'Analyze'
go test ./internal/app -run 'Artifact|Config|Plan'
git diff --check
```
### Exit Criteria ### Exit Criteria
- A/B remain durable after C fails; an ordinary later run retries C and then D - Scriptorium validation has explicit declaration/output,
without invoking A/B. dependency/input, variable, and graph phases.
- Artifact failure and checkpoint failure leave no false current authority, and - All map traversal that can affect validation or normalization is
publish/notify cannot remain successfully current after aggregate failure. deterministic.
- No analyze projection type, field, helper, application branch, rollback path, - Tests protect fields and semantic rules without freezing the former mixed
test fixture, or documentation claim remains. validator choreography.
- `go test ./internal/stage ./internal/app ./internal/manifest ./internal/artifacts`
succeeds.
- `git diff --check` succeeds.
## Stage 5 — Verify Force Semantics, Align Documentation, And Close Workstream 6 ## Stage 5 — Split Composition Envelope Phases And Remove Incidental Metadata
**Status:** Completed **Status:** Pending
### Goal ### Goal
Verify the complete operational contract, remove superseded test/documentation Separate composition-envelope shape decoding, cross-field semantics, and
language, and leave Workstream 6 closed with repository-wide evidence. root-only metadata removal while retaining the source-aware configuration model
that imports, profiles, diagnostics, and inspection commands require.
### Required Work ### Required Work
1. Review existing force/selection/reconciliation tests and retain a lean set 1. Refactor `splitPipelineCompositionEnvelope` in
that proves: `internal/config/pipeline_composition.go` into a short coordinator over
- an ordinary run reuses every valid current selected artifact; three explicit private phases:
- unqualified force reruns the full effective configured target set; - locate and decode the root `composition` mapping and its `imports`,
- selected force reruns the explicit targets, includes their dependency `default_profile`, and `profiles` shapes;
closure, reuses valid current prerequisites, and leaves unrelated valid - validate cross-field semantics, including default-profile membership; and
artifacts untouched; - clone/remove the root-only envelope to produce the ordinary pipeline body.
- stale, failed, missing, or dependency-invalid prerequisites execute in 2. Reuse `parsePipelineProfileDeclaration` and profile-name normalization.
deterministic topological order before their dependents; and Keep unknown-field rejection, non-empty profile declarations, imports and
- an incidental canonical file without current manifest evidence is never profiles in declaration order, explicit-profile selection behavior, and
reused. source-qualified shape/conflict diagnostics.
2. Replace projection-oriented test names, helpers, and assertions with 3. Do not mutate the parsed root document during validation. Return a distinct
checkpoint/recovery language. Remove redundant tests whose only purpose was body document only after the envelope is valid; the original document must
to validate the retired transport or rollback choreography. Do not reduce remain reusable by callers and tests.
coverage of output validation, dependency invalidation, cancellation, 4. Remove `compositionNode` and `compositionField` metadata that has no runtime
persistence failure, or assembled resume behavior. reader after the refactor. Specifically verify and remove redundant cached
3. Update the canonical current-behavior owners: `order`, `line`, and `column` fields and their clone/renumber assignments if
- `docs/internal/stage-analyze.md` for checkpoint order, resume behavior, they remain write-only. Slice order remains the declaration-order
failure semantics, and the absence of projections; representation; YAML parser nodes remain the source of parse-time line and
- `docs/internal/manifest.md` for per-artifact checkpoint authority, column diagnostics.
aggregate status, interruption recovery, and persistence failure; 5. Retain each node's semantic path and source list because merge conflict
- `docs/policy/architecture.md` only where its stage/persistence description reporting, ownership projection, effective digests, artifact-family source
currently implies a single end-of-stage analyze transaction; and projection, and `config sources` consume them. Do not reduce actionable
- `docs/operations.md` and `docs/troubleshooting.md` only where operators conflict diagnostics or delete provenance that has a real consumer.
need to understand reuse after partial analyze failure or recover from a 6. Keep composition product-specific. Do not replace the tree with generic map
manifest checkpoint error. decoding or adopt a third-party composition library.
Do not duplicate CLI or configuration contracts, since syntax and fields do
not change. ### Tests And Verification
4. Search production, tests, and current-behavior documentation for
`AnalyzeStateProjection`, `AnalyzeState`, `analyze projection`, - Test malformed envelope shapes, unknown fields, invalid profile declarations,
`projected records`, and `end-of-stage` language. Remove or revise every missing default membership, successful envelope removal, and absence of
reference tied to the retired mechanism; preserve unrelated planning uses of caller-document mutation through the composition boundary.
the ordinary English word “projection” only when accurate. - Preserve tests for explicit empty/false/zero/list/map values, additive
5. Run the full repository validation from `docs/development.md`: conflicts naming all claiming sources, overlay semantics, canonical YAML,
effective digest stability, profile selection, and `config sources`
ownership. These are semantic contracts, not incidental provenance.
- Remove assertions of deleted private cached metadata. Continue asserting
declaration behavior through ordered outputs rather than private index
fields.
- Run:
```sh
go test ./internal/config -run 'Composition|PipelineProfile|Effective|ArtifactFamil'
go test ./internal/app -run 'Config'
git diff --check
```
### Exit Criteria
- Envelope decoding, cross-field validation, and body production have separate
focused owners under one coordinator.
- Only provenance and ordering data with a demonstrated consumer remains.
- Imports, profiles, overlays, source reporting, and effective digests retain
deterministic behavior.
## Stage 6 — Align Diagnostic Tests, Documentation, And Close Workstream 7
**Status:** Pending
### Goal
Review the completed configuration refactor as one system, remove obsolete test
friction and documentation language, and close Workstream 7 with repository-wide
evidence.
### Required Work
1. Review every configuration test changed by Stages 1–5. Retain coverage of
schema strictness, normalization, semantic invariants, meaningful failure
categories, deterministic outcomes, composition source conflicts, effective
projections/digests, and representative assembled loads.
2. Rewrite assertions that pin complete diagnostic prose, punctuation,
formatting, private helper order, or an arbitrary independent-error winner.
Prefer a relevant field path plus one small semantic fragment. Use
`errors.Is`/`errors.As` only where an existing wrapped or typed cause is a
real contract; do not introduce types solely to change test style.
3. Delete redundant legacy-form enumerations whose only purpose is to preserve
knowledge of retired fields or spellings already rejected by the canonical
strict/source boundary. Retain representative tests proving unknown fields
and unsupported sources fail, and retain migration guidance tests only when
that guidance is an intentional user-facing contract.
4. Consolidate oversized table fixtures only where multiple cases protect the
same rule. Do not delete distinct risks merely to reduce test LOC, and do not
rewrite healthy semantic tests just because they use a small stable error
fragment.
5. Update [`../internal/configuration.md`](../internal/configuration.md) to
describe the implemented validation-domain ownership, deterministic phase
behavior, composition-envelope phases, and diagnostic guarantees without
listing private helper choreography. Update [`../config.md`](../config.md)
only if its user-visible current-behavior wording is inaccurate; no schema
change is expected.
6. Search current production, tests, examples, and current-behavior docs for
active acceptance of the retired configuration forms named by C01. Remove
actual compatibility code and stale current-behavior claims, while leaving
immutable audit/history documents unchanged.
7. Run the full repository validation from `docs/development.md`:
```sh ```sh
go test ./... go test ./...
@@ -436,27 +500,25 @@ language, and leave Workstream 6 closed with repository-wide evidence.
go test ./internal/config -run '^TestExamplesLoadAndValidate$' go test ./internal/config -run '^TestExamplesLoadAndValidate$'
``` ```
6. Run `go test -race -shuffle=on -count=3 ./internal/stage ./internal/app` to 8. Run `go test -race -shuffle=on -count=3 ./internal/config ./internal/app`
stress the changed recovery and persistence boundaries. Run the supported to stress map-order independence and assembled configuration behavior.
release cross-build checks if an implementation change affects build-tagged 9. After all checks pass, mark Stages 1–6 in this document `Completed` and mark
code; otherwise the repository build is sufficient for this workstream. only Workstream 7 in [`refactoring.md`](refactoring.md) `Complete`.
7. After all checks pass, mark Stages 1–5 in this document `Completed` and mark Workstream 8 must remain the next pending workstream.
only Workstream 6 in [`refactoring.md`](refactoring.md) `Complete`.
Workstream 7 must remain the next pending workstream.
### Exit Criteria ### Exit Criteria
- Current documentation describes analyze as a session-manifest-backed, - Configuration remains strict, deterministic, source-aware where useful, and
per-artifact checkpointed queue and contains no retired projection behavior. unchanged in public schema and normalized semantics.
- The force, partial-selection, dependency, interruption, and persistence-error - Validation code is organized by semantic owner; Notarius, Scriptorium, and
cases are protected through stable behavioral boundaries. composition coordinators expose clear phases without generic frameworks.
- All repository validation and focused shuffled race checks pass. - No active retired configuration compatibility path, write-only composition
- Workstream 6 is marked complete, Workstream 7 remains pending, and the metadata, or test dependency on incidental diagnostic choreography remains.
worktree contains no accidental generated or temporary files. - Current documentation describes the implemented boundaries, all validation
- `git diff --check` succeeds. commands pass, Workstream 7 is complete, and Workstream 8 remains pending.
## Open Questions ## Open Questions
None. The audits, approved Workstream 6 scope, existing selection contract, and None. The audits, completed compatibility purge, current configuration
the fixed decisions above resolve the product and implementation choices needed contracts, and fixed decisions above resolve the product and implementation
for the coding agent to execute this plan. choices needed for a `gpt-5.6-terra` coding agent to execute this plan.