eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 27 Wiki Activity

Plan configuration refactoring

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-04 23:37:02 +00:00
parent 74ab9218ff
commit 7a0e2cb11b
1 changed files with 435 additions and 373 deletions
Download Patch File Download Diff File Expand all files Collapse all files

808
docs/roadmap/implementation.md
View File

@@ -1,431 +1,495 @@
# Workstream 6 Implementation Plan — Per-Artifact Analyze Checkpointing
# Workstream 7 Implementation Plan — Configuration Structure And Diagnostics
## Status
Completed implementation record for Workstream 6, **Per-artifact analyze
checkpointing and recovery**, in [`refactoring.md`](refactoring.md). Workstreams
1 through 6 and all stages below are complete. Workstream 7 is the next pending
workstream; its implementation is outside this plan.
Implementation plan for Workstream 7, **Configuration structure and semantic
diagnostics**, in [`refactoring.md`](refactoring.md). Workstreams 1 through 6
are complete and are prerequisites for this plan. All stages below are pending
and must be implemented in numerical order.
The purpose of this workstream is to make analyze a deterministic resumable
artifact queue backed only by the session manifest. If artifacts A and B
succeed, C fails, and D is not attempted, A and B must remain reusable, C must
be retried, and D must follow after C succeeds. The implementation must remove
the current end-of-stage analyze projection and rollback machinery rather than
building checkpointing alongside it as a second durable model.
The purpose of this workstream is to preserve Narratio's strict, deterministic,
operator-oriented configuration behavior while making the implementation
navigable by semantic domain. It also removes incidental composition metadata
and test expectations that turn private validation choreography into a
compatibility burden. This is an internal refactor with narrowly approved
diagnostic flexibility, not a configuration-language redesign.
## Authoritative Scope And Decisions
The authoritative product and architecture scope is Workstream 6, the
dependency relationships, compatibility intent, and consolidated target state
in [`refactoring.md`](refactoring.md). Supporting decisions are H04, H14, H15,
H36, H37, C02, and C03 in [`audit-decisions.md`](audit-decisions.md), with the
detailed rationale in H36 and H37 of [`audit.md`](audit.md) and Finding 3 of
[`contract-audit.md`](contract-audit.md).
The authoritative scope is Workstream 7 in [`refactoring.md`](refactoring.md).
Its supporting audit entries are H04, H14, H15, H17, H38, H39, H40, H41, and
H42 in [`audit.md`](audit.md), C01 and C06 in
[`contract-audit.md`](contract-audit.md), and the corresponding rows in
[`audit-decisions.md`](audit-decisions.md).
The following decisions are fixed for this implementation:
The following implementation decisions are fixed:
- Keep analyze as one fixed pipeline stage in `internal/stage`; per-artifact
work does not become dynamic pipeline stages or a generic workflow engine.
- Keep the session manifest's `analyze_artifacts` collection as the only durable
per-artifact authority. Do not introduce an invocation collection, journal,
sidecar, database, or replacement run manifest.
- Use the existing `stage.Env.ManifestStore` and the canonical session-manifest
path for checkpoints. Production execution already supplies that store and
holds the session lock for the entire `Stage.Run` call. Do not add a callback
framework or expose application lifecycle types to `internal/stage`.
- Checkpoint a copy-on-write candidate manifest. Validate and save the candidate
before installing its analyze collection into the live in-memory manifest.
A failed current-record checkpoint must never make newly materialized bytes
authoritative through a later terminal save.
- Persist the reconciled starting collection before the first adapter call when
it changes durable authority. This removes obsolete records and stales
scheduled or invalid current records before replacement work begins.
- For every successful artifact: validate and materialize its canonical output,
construct and validate its current record, update the runtime catalog, stale
transitive dependents when output identity changed, checkpoint the complete
session collection, and only then attempt the next artifact.
- For an artifact failure: retain earlier successful checkpoints, replace the
active artifact with a bounded failed record, stale its current transitive
dependents, and expose that collection on the live manifest. The application
runner remains responsible for marking aggregate analyze failed, invalidating
publish/notify, and atomically persisting the terminal manifest before the
command returns the error.
- If persistence of a successful-artifact checkpoint fails, stop immediately.
Keep the last durable collection, mark the active artifact failed in the live
collection without promoting its incidental output, and let the runner's
terminal failure save make that retry state durable when storage permits.
- Preserve the existing deterministic work planner, reconciliation,
fingerprints, evidence validation, dependency order, and current-output
identity comparison. File presence alone never creates authority.
- Preserve artifact selection and force behavior. With no `--artifacts`, the
effective configured set is targeted. With `--artifacts`, only the named
targets and their required dependency closure participate. Force reruns
explicit targets; a valid current prerequisite is reused unless it is itself
explicit or becomes non-current through dependency work. Unrelated valid
records are retained.
- On aggregate success, return ordinary `StageResult.Outputs` rebuilt in
deterministic normalized artifact-key order from all current session analyze
records. Per-artifact records retain their own logs, generated configuration
references, bounded Scriptorium provenance, and evidence; aggregate logs and
metadata continue to describe the current invocation.
- On aggregate failure, aggregate outputs remain empty. Earlier current
per-artifact records remain available to a later analyze run, but publish and
notify are invalidated by the existing application-stage dependency relation.
- A crash or cancellation after a successful checkpoint may leave aggregate
analyze `running` or later normalized to `interrupted`; the next ordinary run
must reconcile the durable collection, reuse valid completed artifacts, and
continue with the first failed, stale, or missing artifact.
- Keep the manifest schema and analyze-state contract version unchanged. This
work changes persistence timing and removes an internal projection transport;
it does not introduce a new serialized representation.
- Do not enter Workstream 7 configuration decomposition, Workstream 8 publish
locks/decomposition, or Workstream 9 cleanup-stage work.
- Keep all configuration behavior in package `internal/config`. Do not create
subpackages, a generic configuration framework, a schema framework, a
validator registry, or a dependency-injection layer.
- Preserve the public configuration types and entry points. `Validate`,
`ValidatePipelineConfig`, `ValidateCampaignConfig`, all load APIs, effective
pipeline projections, and configuration-inspection commands retain their
current responsibilities.
- Keep `validate.go` as the short public and high-level orchestration owner.
Organize private validation into cohesive files for core pipeline/session,
adapters, publish/storage, Notarius, Scriptorium, and shared scalar/path
helpers. File names are private, but those ownership boundaries are the
intended final structure.
- Do not replace ordinary `error` returns with a new exported validation-error
hierarchy. No machine consumer currently requires one, and introducing one
would add a new compatibility surface. Field paths, source identities for
composition conflicts, wrapped causes where useful, and small semantic
phrases provide the stable diagnostic content tests should assert.
- Continue to fail fast. The workstream does not add error aggregation and does
not promise which of several independent invalid fields is reported first.
Traversal of maps must nevertheless be deterministic so identical input
produces identical behavior.
- Preserve strict single-document YAML decoding, duplicate- and unknown-field
rejection, additive imports with no duplicate ownership, one deterministic
profile selection, explicit-value overlay behavior, centralized defaults,
canonical party resolution, artifact-family expansion, and secret-free
effective digests.
- Preserve the current configuration schema. Campaign-owned
`campaign.inputs.party_file` and canonical artifact source IDs are current
contracts, not legacy fields. Removed player/session roster fields and old
artifact source spellings remain unsupported; add no compatibility aliases
or normalization branches for them.
- Treat Workstream 3's compatibility purge as authoritative. During this work,
remove any surviving production branch that actually accepts or translates
a retired configuration representation. A rejection supplied naturally by
strict decoding or the canonical source validator is sufficient; do not
maintain exhaustive legacy-form tests when representative canonical-boundary
tests already protect the rule.
- Preserve composition source information that is consumed by conflict
diagnostics or `config sources`. Remove only demonstrably unused node/field
metadata. In particular, source lists and semantic field paths remain;
redundant cached declaration-order indexes and copied line/column fields may
be removed when no consumer reads them because slice order and the YAML
parser already supply the required behavior.
- Preserve normalization as an all-or-nothing operation for each validator.
Notarius reference/output maps and scalar paths are installed on the config
only after all relevant checks succeed; a validation error must not leave a
partially normalized value behind.
- Keep `validateAudita` intact when moving it to the adapter-validation owner.
Its guards are cohesive; splitting them further would add names without
reducing complexity.
- Tests must protect semantic rules and observable normalized results, not
private helper calls, file placement, full error prose, punctuation, or one
historical validation call order. Exact strings remain appropriate only for
an intentionally documented CLI or machine-consumed contract.
- Do not enter Workstream 8 remote publication/lock decomposition, Workstream 9
cleanup-stage work, or unrelated Workstream 10 cleanup.
## Instructions For Every Stage
Before each stage:
1. Read `docs/development.md`, every document under `docs/policy/`, this plan,
and Workstream 6 in [`refactoring.md`](refactoring.md). Read the decision
entries and audit sections named above; do not re-audit unrelated findings.
2. Read [`../internal/stage-analyze.md`](../internal/stage-analyze.md) and
[`../internal/manifest.md`](../internal/manifest.md), then inspect the
and Workstream 7 in [`refactoring.md`](refactoring.md). Read only the audit
sections and decision-register rows cited above; do not restart the audits.
2. Read [`../internal/configuration.md`](../internal/configuration.md) and the
affected portions of [`../config.md`](../config.md), then inspect the
production files and focused tests named by the stage.
3. Prefer the codebase knowledge graph for Go symbol discovery and call tracing.
Use text search for serialized fields, documentation, fixtures, and removal
of projection terminology.
4. Confirm the worktree state and preserve unrelated changes. Do not begin a
later workstream merely because adjacent code is visible.
3. Prefer the codebase knowledge graph for Go symbol discovery and call
tracing. Use text search for YAML fields, source IDs, error fragments,
examples, documentation, and removal verification.
4. Confirm the worktree state and preserve unrelated changes. Do not implement
an adjacent pending workstream merely because it shares a file.
During every stage:
- Keep the repository compiling and focused tests passing. A temporary private
delegation or projection bridge is permitted only where the next numbered
stage explicitly removes it; do not leave two durable authorities active.
- Preserve package ownership and acyclic dependency direction. Keep new types
private and cohesive; do not add public extension points.
- Keep all tests deterministic, offline, and independent of real Scriptorium or
paid APIs. Prefer the existing fake runner, real temporary filesystem, and a
small recording/failing manifest store only where checkpoint interaction is
itself the contract.
- Assert persisted manifest state, adapter calls, dependency order, and retry
behavior. Do not pin private helper choreography, complete error wording, or
incidental timestamps.
- Preserve bounded, non-secret errors and diagnostics. Never place prompt or
transcript content, variables containing secrets, or raw credentials in new
manifest fields or error assertions.
- Use `gofmt` on changed Go files, run the focused tests listed for the stage,
- Keep the package compiling and focused tests passing. Move behavior before
redesigning the two complex validators so reviewable commits do not combine
unrelated mechanical and semantic changes.
- Use package-private functions with explicit inputs and return values. Avoid
stateful validator objects, forwarding-only wrappers, generic rule tables,
and abstractions whose only purpose is to reduce function length.
- Keep configuration mutation explicit and late. Do not mutate caller-owned
maps while validating a temporary normalized candidate.
- Retain deterministic order wherever config maps are traversed for validation,
normalization, canonical output, source reporting, or digest construction.
- Keep tests deterministic, offline, and independent of credentials or live
adapters. Prefer direct package behavior and real temporary files over mocks.
- Use `gofmt` on changed Go files, run the focused checks listed for the stage,
and run `git diff --check`. Mark a stage `Completed` only after its exit
criteria pass.
## Stage 1 — Decompose The Single-Artifact Transaction
## Stage 1 — Separate Core And Shared Validation Ownership
**Status:** Completed
**Status:** Pending
### Goal
Make one artifact execution understandable and independently reviewable before
changing its persistence timing. This stage is behavior-preserving.
Reduce `validate.go` to configuration validation entry points and readable
high-level sequencing, while giving core pipeline/session rules and reusable
primitive checks clear package-local owners. This stage is behavior-preserving.
### Required Work
1. Refactor `executeAnalyzeArtifact` in `internal/stage/analyze.go` into one
short transaction owner plus private phases with explicit intermediate
values. Keep the existing `analyzeExecutionContext` and artifact plan, or
replace them with equally narrow private values; do not restore the former
broad parameter list.
2. Separate these concerns without changing their order or contracts:
- input identity resolution, omitted optional inputs, reused-current input
reporting, Scriptorium variables, timeout, canonical/run-local paths, and
request/diagnostic path preparation;
- optional render-debug invocation and validation;
- Scriptorium artifact invocation and adapter failure adaptation;
- authoritative output-path selection, bounded non-empty regular-file
validation, output-contract construction, canonical materialization, and
installed-checksum verification; and
- result/evidence construction, including provenance, logs, generated
configuration paths, metadata, output size, and reused artifacts.
3. Place the extracted code in focused analyze-owned files when that improves
navigation. Suggested responsibilities are preparation, adapter invocation,
and validated materialization/evidence; exact private filenames are not a
contract.
4. Keep Scriptorium request and response types inside the stage/adapter boundary.
Do not move Narratio artifact, manifest, path, or metadata policy into the
Scriptorium adapter.
5. Preserve the current run-local-before-canonical materialization boundary,
render-debug behavior, useful error context, workspace modes, centralized
size limit, and checksum comparison. Do not weaken validation or add a new
write path while moving code.
6. Retain existing behavior tests at their stable boundaries. Consolidate only
tests that become purely coupled to moved private helpers; do not add one
test per extracted function.
1. Keep these entry points and the visible pipeline-domain call sequence in
`internal/config/validate.go`:
- `Validate`;
- `ValidatePipelineConfig`;
- `ValidateCampaignConfig`; and
- the short `validatePipeline` coordinator.
2. Move core configuration checks into a cohesive private file such as
`validate_core.go`. This owner includes campaign validation, secrets,
spool/cache, session validation, resolved-party input validation,
session/campaign identifiers, and cross-pipeline/session rules. Keep
`publishUploadConfiguredForS3` with publish/storage in Stage 2 rather than
hiding publish policy in the core owner.
3. Move general validation primitives into a small private file such as
`validate_helpers.go`: environment-variable name checks, positive duration
parsing, safe relative paths, and path-within-root checks. Do not turn these
functions into a generic validation DSL.
4. Re-home the straightforward stage/adapter validators in a private adapter
owner such as `validate_adapters.go`: notification, normalize, trim, render,
WhisperX, Seriatim (including its output schema), and Audita. Move
`validateAudita` intact apart from import or formatting changes.
5. Preserve current nil handling, defaults, normalization, wrapped causes,
field identification, and high-level validation semantics. Do not use this
mechanical move to reorder domains or rewrite diagnostics.
6. Remove imports and dead private helpers left behind in `validate.go`. Do not
add facade functions to preserve former source-file placement; files are not
Go API boundaries.
### Tests And Verification
- Existing core, campaign, session, duration, normalize, trim, render,
notification, and load/validate tests should normally pass unchanged.
- Add no tests that merely assert a function moved files. Add or repair a test
only if the move exposes an unprotected semantic rule.
- Run:
```sh
go test ./internal/config
go test ./internal/app -run 'Config|Pipeline|Session'
git diff --check
```
### Exit Criteria
- `executeAnalyzeArtifact` reads as a short ordered transaction over explicit
phases and no longer mixes all preparation, invocation, materialization, and
evidence construction in one large function.
- Existing analyze request, render, validation, materialization, provenance,
and failure behavior is unchanged.
- `go test ./internal/stage ./internal/adapters/scriptorium` succeeds.
- `git diff --check` succeeds.
- `validate.go` is a concise entry/coordinator file rather than a container for
unrelated domain implementations.
- Core, adapter, and shared helper ownership is obvious from the file layout,
and Audita remains one cohesive validator.
- No public behavior or serialized configuration shape changed.
## Stage 2 — Establish Session-Only Analyze Execution State
## Stage 2 — Separate Publish, Storage, Notarius, And Scriptorium Owners
**Status:** Completed
**Status:** Pending
### Goal
Replace `analyzeStage.Run`'s parallel local collections with one private
session-oriented execution state that is ready to checkpoint, while retaining
the existing end-of-stage persistence boundary temporarily.
Finish the domain-level split of the former aggregate validator before changing
the internal structure of Notarius or Scriptorium validation.
### Required Work
1. Introduce a private analyze execution state/accumulator that owns:
- the shared execution context and deterministic work plan;
- the complete reconciled session artifact collection;
- prior current records needed for output-identity comparison;
- the evaluated/scheduled key set used for dependent invalidation;
- the mutable runtime artifact catalog; and
- invocation-only aggregate logs, generated configurations, generated
metadata, reused-artifact metadata, and its deduplication set.
2. Keep setup, reconciliation, work planning, adapter-availability validation,
deterministic iteration, and final `StageResult` assembly visible in
`analyzeStage.Run`. Move per-item state mutation and diagnostic collection
behind narrow methods on the private state; do not turn the accumulator into
a service locator.
3. Rename `analyzeWorkPlan.ProjectedRecords`, `projectedAnalyzeRecords`, and
related private terminology to describe the reconciled starting session
collection. It is the candidate durable state, not an invocation projection.
Preserve the planner's read-only behavior and existing classification.
4. Make the accumulator install its complete collection into the passed session
manifest on success and its failed/staled collection on error. During this
stage, continue returning the existing `AnalyzeStateProjection` as a
temporary compatibility bridge for the application runner; both views must
be derived from the same collection, and no additional save is introduced.
5. Add optional `ProducerRunID` provenance to `artifacts.Ref`. Update the
generic application result mapper to use that value when it is non-empty and
otherwise retain the current behavior of assigning the active run ID. This
is data needed by ordinary stage-result transport, not an analyze-specific
branch; cover both explicit and fallback behavior in the mapper tests.
6. Add a private helper that deterministically converts all current analyze
records into ordinary `artifacts.Ref` outputs. Preserve source ID, relative
path, contract, checksum, and producer run ID. Include valid current records
retained outside a partial selection; do not include stale, missing, or
failed records. Continue using the projection path for aggregate outputs
until Stage 3 switches ownership.
7. Rewrite stage tests to inspect the manifest collection mutated by the real
analyze stage rather than manually installing returned projections. Retain a
minimal assertion that the temporary projection matches that collection so
the transition cannot silently diverge.
1. Move storage, publish output, and publish lock validation into a cohesive
private owner such as `validate_publish.go`. Move the associated helpers
with their consumers, including `ValidatePublishLockRules`,
`MergePublishLockRules`, configured output/source lookups, and
`publishUploadConfiguredForS3`.
2. Preserve the exported lock-rule helper signatures because current app and
stage code consume them. Do not redesign remote lock behavior; that belongs
to Workstream 8.
3. Move the existing Notarius validator into `validate_notarius.go` and its
Scriptorium counterpart, input-source validator, and exported dependency
validator into `validate_scriptorium.go`. At this stage, move these functions
without decomposing or semantically reordering them; Stages 3 and 4 own those
changes.
4. Keep source-vocabulary and collision helpers with the narrowest shared
domain owner. If both publish and Scriptorium need a small lookup derived
from Notarius/Scriptorium configuration, use one plainly named package
helper rather than duplicate maps or introduce a registry.
5. Ensure `validate.go` contains only entry points and high-level sequencing
after the move. Delete the emptied blocks and unused imports.
6. Search `internal/config` production code for acceptance or translation of
removed player/session roster fields and retired artifact source spellings.
Delete any actual compatibility branch found. Do not mistake current
campaign `party_file`, rejection through strict decoding, migration prose,
or a test fixture demonstrating rejection for an active compatibility path.
### Tests And Verification
- Existing storage/publish, Notarius, Scriptorium, configuration loading, and
application configuration tests should pass without pinning file placement.
- Run:
```sh
go test ./internal/config
go test ./internal/app ./internal/stage
git diff --check
```
### Exit Criteria
- `analyzeStage.Run` coordinates one private session collection rather than
parallel record and diagnostic variables.
- Planning/reconciliation remains read-only, execution order and force/selection
behavior are unchanged, and stage tests primarily assert manifest state.
- Ordinary `StageResult.Outputs` can preserve a reused artifact's original
producer run identity without an analyze-specific application path.
- The existing application projection path still functions and no checkpoint
timing has changed yet.
- `go test ./internal/stage ./internal/app` succeeds.
- `git diff --check` succeeds.
- Every validation domain named by Workstream 7 has one discoverable file-level
owner inside `internal/config`.
- There is no duplicate source/collision logic and no active retired-format
compatibility branch in configuration production code.
- Notarius and Scriptorium semantics remain unchanged pending their dedicated
stages.
## Stage 3 — Cut Analyze Over To Durable Per-Artifact Checkpoints
## Stage 3 — Decompose Notarius Validation And Normalization
**Status:** Completed
**Status:** Pending
### Goal
Make each validated artifact completion durable before later artifact work and
stop using the end-of-stage projection transport for real analyze execution.
Make `validateNotarius` a short transaction that validates top-level policy,
normalizes references, normalizes output lanes, and commits the resulting
configuration only after every check succeeds.
### Required Work
1. Add a private analyze checkpoint operation in `internal/stage`. It must:
- require a non-nil session manifest, analyze stage record, and
`Env.ManifestStore` when analyze has state to persist;
- derive the canonical manifest path through `artifacts` helpers from the
resolved workspace, campaign, and session identity;
- validate the full candidate collection with
`manifest.ValidateAnalyzeArtifactCollection`;
- shallow-copy the manifest, clone the top-level stage map, copy the analyze
stage record, and deep-clone the candidate analyze collection so the save
cannot mutate live authority before success;
- preserve the aggregate analyze status (normally `running`) and every
unrelated manifest field;
- save the candidate through `Env.ManifestStore`; and
- only after success, install the candidate analyze version/collection and
the store-updated manifest timestamp into the live manifest.
Do not add a general manifest transaction framework or JSON round-trip clone.
2. Before invoking the first scheduled artifact, checkpoint the reconciled
starting collection when it differs from durable state. This must remove
records for removed configuration and stale any selected/forced/non-current
authority before replacement work begins. If this save fails, invoke no
adapter.
3. For each scheduled artifact, preserve this order:
- compute its execution fingerprint;
- execute the decomposed artifact transaction;
- construct and validate its current manifest record;
- prove the generated record can be added to the runtime catalog for use by
later dependents;
- stale unscheduled transitive dependents only when prior and replacement
output identities differ;
- checkpoint the complete updated session collection; and
- only then expose the result to a later artifact.
4. If a successful artifact's checkpoint fails, stop immediately and do not
call the next artifact. Build a failed record for the active artifact from
the last committed collection, with no output authority; stale its current
transitive dependents; and install that failure candidate in memory so the
runner's terminal save can persist retry state. Return the checkpoint
failure with artifact identity and manifest context.
5. Have real analyze execution return ordinary `StageResult.Outputs` built from
every current session record plus its existing aggregate diagnostics and
metadata. Stop returning `AnalyzeStateProjection` from `analyzeStage.Run`.
The legacy projection type and application handlers may remain only as dead
transitional code until Stage 4; add no new uses.
6. In `executeSelectedStage`, make a terminal save failure after any stage's
apparent success mark that stage failed and invalidate succeeded transitive
dependents before terminal persistence. This behavior currently occurs only
on the analyze-projection branch; it must remain correct when real analyze no
longer returns a projection and should be correct generically.
7. Update direct analyze fixtures so they supply a real temporary
`manifest.LocalStore`, complete campaign/run identity, an aggregate analyze
record in the state appropriate to the test, and an initially persisted
manifest. Keep unit-only planner/reconciliation tests free of persistence.
8. Add focused checkpoint tests at the stage/application boundary:
- the manifest store observes A current before B is invoked and A+B current
before C is invoked;
- a failed B checkpoint prevents C from being invoked;
- the failed B checkpoint never persists B as current merely because its
canonical file exists; and
- final aggregate success exposes outputs for all and only current records
in deterministic normalized artifact-key order, preserving each record's
producer run identity.
1. Keep `validateNotarius` responsible for nil/enablement behavior and the
top-level scalar contract: required binary, configuration path, pipeline ID,
positive timeout, working directory, at least one output, and the reference
binding limit.
2. Replace the validator's bespoke timeout parsing with the shared positive
duration helper when it preserves the same accepted values and field
identity. Do not embed duration policy in a Notarius-only branch.
3. Extract reference binding normalization into one private operation that:
- iterates raw selector keys in sorted order;
- normalizes selectors through `notariusref`;
- rejects normalized selector collisions;
- trims and validates each prepared-input source; and
- returns a newly allocated normalized map without mutating the input map.
4. Extract output normalization and collision validation into a separate
private operation that:
- builds the reserved built-in, configured Scriptorium, and previous-session
source set once;
- iterates raw output keys in sorted order;
- validates and normalizes configured keys and derived extraction source
IDs;
- rejects reserved-source, normalized-source, and lane-ID collisions;
- trims lane/media/schema/module scalar fields and checks required fields;
and
- returns a newly allocated normalized output map.
5. Normalize binary, paths, pipeline ID, timeout, and working directory into
local candidate values. Assign all scalar candidates, references, and
outputs back to `cfg` only after both helper operations succeed. Preserve
nil-versus-empty map behavior where it is observable in effective YAML or
digests.
6. Keep all helpers Notarius-specific and private. Do not move selector policy
out of `notariusref`, copy artifact source policy, or create a generalized
map-normalization framework.
### Tests And Verification
- Retain lean behavior tests for disabled configuration, required scalars,
timeout and reference-binding limits, selector normalization/collisions, supported
prepared sources, output normalization, reserved-source collisions,
duplicate lanes, and cross-use by Scriptorium/publish.
- Add or consolidate one test proving a failing reference or output candidate
leaves the original Notarius config unmodified.
- Add one determinism test that constructs equivalent reference/output maps in
different insertion orders and compares normalized success or equivalent
diagnostic classification. Do not assert which of multiple independent
invalid domains must win forever.
- Assert field paths and the smallest meaningful semantic fragment rather than
complete error sentences.
- Run:
```sh
go test ./internal/config -run 'Notarius|Scriptorium|Publish|Duration'
go test ./internal/stage -run 'Extract|Analyze'
git diff --check
```
### Exit Criteria
- Every successful scheduled artifact is durable before the next adapter call.
- A checkpoint failure stops later work and cannot promote incidental output.
- Real analyze execution no longer relies on an end-of-stage state projection.
- Aggregate success/failure persistence and dependent invalidation remain
application-owned and correct.
- `go test ./internal/stage ./internal/app ./internal/manifest` succeeds.
- `git diff --check` succeeds.
- `validateNotarius` visibly sequences scalar validation, reference
normalization, output normalization/collision checks, and one final commit.
- Reference and output collision domains are separate, deterministic, and do
not partially mutate caller-owned configuration on failure.
- No Notarius behavior moved into a generic configuration abstraction.
## Stage 4 — Complete Failure Recovery And Remove Projection Machinery
## Stage 4 — Decompose Deterministic Scriptorium Validation
**Status:** Completed
**Status:** Pending
### Goal
Complete the A/B/C/D recovery contract and delete the superseded projection,
validation, application, and rollback paths.
Separate Scriptorium artifact declarations/outputs, dependencies/inputs, and
variables into readable validation phases with deterministic map traversal.
### Required Work
1. On any artifact-specific failure—including fingerprinting, preparation,
render, adapter execution, result validation, materialization, evidence
construction, catalog admission, or checkpoint persistence—construct one
bounded failed record for the active key, clear its output authority, and
stale current transitive dependents. Preserve unrelated current records and
all earlier successfully checkpointed records.
2. Install the failed/staled collection on the live manifest before returning
the artifact error. The runner must then use its ordinary stage-error path to
mark aggregate analyze failed, clear aggregate outputs, invalidate publish
and notify, and persist the terminal session manifest. Join a terminal save
failure with the originating artifact/checkpoint error; never replace the
useful cause.
3. Ensure unattempted artifacts behave according to existing authority:
previously absent records remain absent/missing on reconciliation; unrelated
valid current records remain current; and a prior current record that
depends directly or transitively on the failed artifact becomes stale with
no output authority.
4. Remove `AnalyzeStateProjection` from `stage.StageResult`, delete
`internal/app/analyze_projection.go`, and remove all validation, cloning,
application, capture, rollback, and special output reconstruction that exists
solely for that projection. Remove fake projection stages and projection-only
tests; replace their meaningful risks with real analyze checkpoint/recovery
coverage.
5. Simplify `executeSelectedStage` to its generic result/error handling while
preserving analyze's live manifest collection across aggregate lifecycle
clearing. Do not add a type assertion, stage-name branch, callback, or new
result union to replace the projection special case.
6. Add a durable assembled-run regression for the approved scenario:
- configure deterministic A, B, C, and D work in dependency order;
- make A and B succeed, C fail, and verify D is not attempted;
- reload the manifest and verify A/B current, C failed, aggregate analyze
failed, and the correct missing/stale state for D;
- run again without force, verify only C and then D execute, and verify the
final aggregate and all selected records are current; and
- verify valid unrelated records and upstream transcript stages were never
invalidated or rerun.
7. Add focused variants for failure of the first and last artifact, cancellation
after at least one checkpoint, changed-output invalidation of an unselected
transitive dependent, and byte/contract-identical replacement preserving an
unselected dependent. Consolidate with existing incremental/failure tests so
each durable risk has one clear owner.
1. Keep `validateScriptorium` as the transaction owner for nil behavior,
required binary, optional config path, stage timeout, ordered validation
phases, referenced-output checks, and the final dependency-graph check.
2. Produce one sorted list of configured artifact keys after validating their
canonical key syntax. Reuse that order across phases rather than ranging
over the artifact map independently.
3. Extract a declaration/output phase that validates enabled prompt and output
requirements, safe output paths, artifact timeout values, and duplicate
output ownership. Keep its collision map local to that phase.
4. Extract a dependency/input phase that:
- validates dependency entries and self-reference using declared slice
indexes for actionable field paths;
- iterates input maps by sorted input name;
- validates required source declarations through the existing canonical
input-source owner;
- requires configured-artifact sources to appear in `depends_on`; and
- returns or updates only the small referenced-artifact set needed by the
later referenced-output check.
5. Extract a variable phase that iterates variable names in sorted order,
rejects empty names, and accepts only the currently supported string and
boolean values.
6. Iterate the referenced-artifact set deterministically when checking required
outputs. Keep `ValidateScriptoriumArtifactDependencies` as the public graph
validator and preserve its cycle and dependency semantics.
7. It is acceptable for phase separation to change which of several unrelated
simultaneous mistakes is reported first. Within a given implementation,
equivalent maps must produce the same result. Do not add compatibility code
to reproduce the old mixed per-artifact call order.
8. Remove forwarding-only helpers and obsolete temporary sets after the split.
Keep `validateScriptoriumInputSource` cohesive unless a genuine duplicate
semantic owner emerges.
### Tests And Verification
- Retain semantic coverage for declaration keys, enabled requirements, output
path safety/collisions, timeouts, dependency existence/self/cycles, input
source classes, extraction references, required `depends_on`, referenced
outputs, and variable types.
- Add one map-insertion-order test that feeds equivalent Scriptorium maps in
different insertion orders and asserts equal validity or equivalent
diagnostics. Do not assert an arbitrary globally preferred independent-error
order.
- Consolidate table cases that repeat the same source-vocabulary rule through
different retired spellings when a representative canonical rejection test
already protects unsupported-source behavior.
- Run:
```sh
go test ./internal/config -run 'Scriptorium|ArtifactFamil|Notarius|Publish'
go test ./internal/stage -run 'Analyze'
go test ./internal/app -run 'Artifact|Config|Plan'
git diff --check
```
### Exit Criteria
- A/B remain durable after C fails; an ordinary later run retries C and then D
without invoking A/B.
- Artifact failure and checkpoint failure leave no false current authority, and
publish/notify cannot remain successfully current after aggregate failure.
- No analyze projection type, field, helper, application branch, rollback path,
test fixture, or documentation claim remains.
- `go test ./internal/stage ./internal/app ./internal/manifest ./internal/artifacts`
succeeds.
- `git diff --check` succeeds.
- Scriptorium validation has explicit declaration/output,
dependency/input, variable, and graph phases.
- All map traversal that can affect validation or normalization is
deterministic.
- Tests protect fields and semantic rules without freezing the former mixed
validator choreography.
## Stage 5 — Verify Force Semantics, Align Documentation, And Close Workstream 6
## Stage 5 — Split Composition Envelope Phases And Remove Incidental Metadata
**Status:** Completed
**Status:** Pending
### Goal
Verify the complete operational contract, remove superseded test/documentation
language, and leave Workstream 6 closed with repository-wide evidence.
Separate composition-envelope shape decoding, cross-field semantics, and
root-only metadata removal while retaining the source-aware configuration model
that imports, profiles, diagnostics, and inspection commands require.
### Required Work
1. Review existing force/selection/reconciliation tests and retain a lean set
that proves:
- an ordinary run reuses every valid current selected artifact;
- unqualified force reruns the full effective configured target set;
- selected force reruns the explicit targets, includes their dependency
closure, reuses valid current prerequisites, and leaves unrelated valid
artifacts untouched;
- stale, failed, missing, or dependency-invalid prerequisites execute in
deterministic topological order before their dependents; and
- an incidental canonical file without current manifest evidence is never
reused.
2. Replace projection-oriented test names, helpers, and assertions with
checkpoint/recovery language. Remove redundant tests whose only purpose was
to validate the retired transport or rollback choreography. Do not reduce
coverage of output validation, dependency invalidation, cancellation,
persistence failure, or assembled resume behavior.
3. Update the canonical current-behavior owners:
- `docs/internal/stage-analyze.md` for checkpoint order, resume behavior,
failure semantics, and the absence of projections;
- `docs/internal/manifest.md` for per-artifact checkpoint authority,
aggregate status, interruption recovery, and persistence failure;
- `docs/policy/architecture.md` only where its stage/persistence description
currently implies a single end-of-stage analyze transaction; and
- `docs/operations.md` and `docs/troubleshooting.md` only where operators
need to understand reuse after partial analyze failure or recover from a
manifest checkpoint error.
Do not duplicate CLI or configuration contracts, since syntax and fields do
not change.
4. Search production, tests, and current-behavior documentation for
`AnalyzeStateProjection`, `AnalyzeState`, `analyze projection`,
`projected records`, and `end-of-stage` language. Remove or revise every
reference tied to the retired mechanism; preserve unrelated planning uses of
the ordinary English word “projection” only when accurate.
5. Run the full repository validation from `docs/development.md`:
1. Refactor `splitPipelineCompositionEnvelope` in
`internal/config/pipeline_composition.go` into a short coordinator over
three explicit private phases:
- locate and decode the root `composition` mapping and its `imports`,
`default_profile`, and `profiles` shapes;
- validate cross-field semantics, including default-profile membership; and
- clone/remove the root-only envelope to produce the ordinary pipeline body.
2. Reuse `parsePipelineProfileDeclaration` and profile-name normalization.
Keep unknown-field rejection, non-empty profile declarations, imports and
profiles in declaration order, explicit-profile selection behavior, and
source-qualified shape/conflict diagnostics.
3. Do not mutate the parsed root document during validation. Return a distinct
body document only after the envelope is valid; the original document must
remain reusable by callers and tests.
4. Remove `compositionNode` and `compositionField` metadata that has no runtime
reader after the refactor. Specifically verify and remove redundant cached
`order`, `line`, and `column` fields and their clone/renumber assignments if
they remain write-only. Slice order remains the declaration-order
representation; YAML parser nodes remain the source of parse-time line and
column diagnostics.
5. Retain each node's semantic path and source list because merge conflict
reporting, ownership projection, effective digests, artifact-family source
projection, and `config sources` consume them. Do not reduce actionable
conflict diagnostics or delete provenance that has a real consumer.
6. Keep composition product-specific. Do not replace the tree with generic map
decoding or adopt a third-party composition library.
### Tests And Verification
- Test malformed envelope shapes, unknown fields, invalid profile declarations,
missing default membership, successful envelope removal, and absence of
caller-document mutation through the composition boundary.
- Preserve tests for explicit empty/false/zero/list/map values, additive
conflicts naming all claiming sources, overlay semantics, canonical YAML,
effective digest stability, profile selection, and `config sources`
ownership. These are semantic contracts, not incidental provenance.
- Remove assertions of deleted private cached metadata. Continue asserting
declaration behavior through ordered outputs rather than private index
fields.
- Run:
```sh
go test ./internal/config -run 'Composition|PipelineProfile|Effective|ArtifactFamil'
go test ./internal/app -run 'Config'
git diff --check
```
### Exit Criteria
- Envelope decoding, cross-field validation, and body production have separate
focused owners under one coordinator.
- Only provenance and ordering data with a demonstrated consumer remains.
- Imports, profiles, overlays, source reporting, and effective digests retain
deterministic behavior.
## Stage 6 — Align Diagnostic Tests, Documentation, And Close Workstream 7
**Status:** Pending
### Goal
Review the completed configuration refactor as one system, remove obsolete test
friction and documentation language, and close Workstream 7 with repository-wide
evidence.
### Required Work
1. Review every configuration test changed by Stages 1–5. Retain coverage of
schema strictness, normalization, semantic invariants, meaningful failure
categories, deterministic outcomes, composition source conflicts, effective
projections/digests, and representative assembled loads.
2. Rewrite assertions that pin complete diagnostic prose, punctuation,
formatting, private helper order, or an arbitrary independent-error winner.
Prefer a relevant field path plus one small semantic fragment. Use
`errors.Is`/`errors.As` only where an existing wrapped or typed cause is a
real contract; do not introduce types solely to change test style.
3. Delete redundant legacy-form enumerations whose only purpose is to preserve
knowledge of retired fields or spellings already rejected by the canonical
strict/source boundary. Retain representative tests proving unknown fields
and unsupported sources fail, and retain migration guidance tests only when
that guidance is an intentional user-facing contract.
4. Consolidate oversized table fixtures only where multiple cases protect the
same rule. Do not delete distinct risks merely to reduce test LOC, and do not
rewrite healthy semantic tests just because they use a small stable error
fragment.
5. Update [`../internal/configuration.md`](../internal/configuration.md) to
describe the implemented validation-domain ownership, deterministic phase
behavior, composition-envelope phases, and diagnostic guarantees without
listing private helper choreography. Update [`../config.md`](../config.md)
only if its user-visible current-behavior wording is inaccurate; no schema
change is expected.
6. Search current production, tests, examples, and current-behavior docs for
active acceptance of the retired configuration forms named by C01. Remove
actual compatibility code and stale current-behavior claims, while leaving
immutable audit/history documents unchanged.
7. Run the full repository validation from `docs/development.md`:
```sh
go test ./...
@@ -436,27 +500,25 @@ language, and leave Workstream 6 closed with repository-wide evidence.
go test ./internal/config -run '^TestExamplesLoadAndValidate$'
```
6. Run `go test -race -shuffle=on -count=3 ./internal/stage ./internal/app` to
stress the changed recovery and persistence boundaries. Run the supported
release cross-build checks if an implementation change affects build-tagged
code; otherwise the repository build is sufficient for this workstream.
7. After all checks pass, mark Stages 1–5 in this document `Completed` and mark
only Workstream 6 in [`refactoring.md`](refactoring.md) `Complete`.
Workstream 7 must remain the next pending workstream.
8. Run `go test -race -shuffle=on -count=3 ./internal/config ./internal/app`
to stress map-order independence and assembled configuration behavior.
9. After all checks pass, mark Stages 1–6 in this document `Completed` and mark
only Workstream 7 in [`refactoring.md`](refactoring.md) `Complete`.
Workstream 8 must remain the next pending workstream.
### Exit Criteria
- Current documentation describes analyze as a session-manifest-backed,
per-artifact checkpointed queue and contains no retired projection behavior.
- The force, partial-selection, dependency, interruption, and persistence-error
cases are protected through stable behavioral boundaries.
- All repository validation and focused shuffled race checks pass.
- Workstream 6 is marked complete, Workstream 7 remains pending, and the
worktree contains no accidental generated or temporary files.
- `git diff --check` succeeds.
- Configuration remains strict, deterministic, source-aware where useful, and
unchanged in public schema and normalized semantics.
- Validation code is organized by semantic owner; Notarius, Scriptorium, and
composition coordinators expose clear phases without generic frameworks.
- No active retired configuration compatibility path, write-only composition
metadata, or test dependency on incidental diagnostic choreography remains.
- Current documentation describes the implemented boundaries, all validation
commands pass, Workstream 7 is complete, and Workstream 8 remains pending.
## Open Questions
None. The audits, approved Workstream 6 scope, existing selection contract, and
the fixed decisions above resolve the product and implementation choices needed
for the coding agent to execute this plan.
None. The audits, completed compatibility purge, current configuration
contracts, and fixed decisions above resolve the product and implementation
choices needed for a `gpt-5.6-terra` coding agent to execute this plan.