narratio/docs/internal/stage-analyze.md

Stage: analyze

Purpose

Reconcile configured Scriptorium artifacts, execute only required work in dependency order, and safely materialize validated outputs.

Inputs

• ordinary configured artifacts from pipeline.scriptorium.artifacts; canonical party artifact families have already expanded into this map during configuration resolution, including corresponding member dependencies and rewritten member-artifact input sources. Each artifact's dependency list is already the normalized union of explicit ordering edges and configured-artifact input edges.
• optional selected artifact keys supplied through the stage environment
• built-in, configured, extraction, and previous-session source references in artifact inputs

Supported source families:

• built-ins: narratio.transcript.*, narratio.bounds.session
• prepared stable inputs: narratio.input.players, narratio.input.party, narratio.input.glossary, narratio.input.spell_catalog
• configured artifacts: narratio.artifact.<key>
• extraction lanes: narratio.extraction.<key>
• previous-session cache: narratio.previous_session.artifact.<key>

Outputs

• one current per-artifact manifest record per validated materialized output
• stage metadata describing selected/generated/reused artifacts

Key Behavior

• when pipeline.scriptorium is absent or no configured artifact is executable, completes successfully and records explanatory metadata. Before returning, it reconciles and checkpoints the complete session collection: records removed from configuration are dropped, while valid records for configured but unselected artifacts remain authoritative. This is not an explicit self-skip: the session manifest records success, satisfies publish's prerequisite, and an ordinary later run reuses the result while the effective set remains empty. Enabling or selecting an artifact later makes missing versioned evidence non-resumable and schedules it without requiring force.
• builds a runtime artifact catalog containing built-ins, configured artifacts, and configured extraction lanes. Extraction availability is hydrated only from compatible successful extraction evidence.
• uses enabled configured artifacts by default. An explicit --artifacts selection is a one-invocation override that makes exactly the named configured artifacts explicit targets even when disabled. The work planner adds required configured prerequisites, reuses current ones, and schedules stale, missing, or otherwise non-current prerequisites before dependents.
• makes a non-executable configured artifact reusable only when its current manifest record and durable output pass the configured-artifact evidence contract; an incidental or stale canonical file is unavailable.
• validates selected artifact dependency order (cycle-safe topo ordering).
• consumes only the concrete dependency graph finalized by configuration; planning, prerequisite closure, execution, reuse, invalidation, and semantic evidence do not infer a second graph from inputs.
• resolves required/optional inputs per artifact source definition into an ordered semantic identity. Each identity records the configured input name, canonical source ID, required policy, explicit presence, source contract, checksum, size, and a source-based logical identity. Workspace paths and producer run IDs are excluded.
• orders input identities by configured input name independently of Go map iteration. Runtime adapter paths remain a separate execution-only map.
• omits an unavailable optional input from the adapter request while retaining explicit absence in its semantic identity; an unavailable required input fails.
• resolves prepared stable input sources through the shared manifest-authoritative identity resolver; it does not accept incidental files or fall back to campaign/session source paths.
• reuses checksums and sizes from validated prepared, extraction, and current configured-artifact evidence. Other resolved inputs are hashed as confined regular files with streaming reads and the central resolved-artifact size limit.
• owns a versioned SHA-256 fingerprint contract with one fixed-field canonical JSON payload and no map serialization. Configured artifacts are fingerprinted in deterministic dependency order.
• fingerprints the normalized artifact key, prompt and profile identifiers, effective render-debug behavior, session-relative output identity, sorted dependency keys, ordered input declarations and semantic identities, validated current dependency-output identities, and sorted effective Scriptorium variables (including Narratio's sticky session variable).
• provides read-only reconciliation that classifies each configured record as current, stale, missing, failed, or otherwise non-resumable, and separately identifies manifest records removed from current configuration. A record is current only when its fingerprint version and value match and its configured output still passes manifest-authoritative evidence validation.
• owns a read-only typed work planner. Its explicit targets are enabled artifacts by default or the exact normalized --artifacts selection when supplied. It closes targets over configured prerequisites, orders the closure topologically, reuses current members, and schedules every non-current member before its dependents.
• force applies only to explicit targets. A current prerequisite is reused unless it is itself an explicit forced target; disabled prerequisites may be rebuilt when required, while unrelated disabled artifacts are excluded.
• the work plan carries explicit targets, prerequisite-only work, deterministic execution and reuse lists, invalidated and removed records, and a cloned reconciled starting collection. Valid unrelated configured records survive reconciliation, removed records are omitted, and incidental files never become current without manifest-backed regeneration.
• implements aggregate resume validation by running the same read-only catalog, fingerprint reconciliation, and work planner used by execution. A succeeded aggregate record is reusable exactly when the selected closure schedules no artifact work and no removed configuration records require a checkpoint; stale unrelated records do not block a partial selection.
• exposes the typed artifact decision to session plan. Planning applies it to a cloned manifest after modeling earlier selected stage transitions, so aggregate run/skip and artifact execute/reuse decisions match the ordinary runner without creating durable state or invoking Scriptorium.
• executes only the work plan's scheduled entries. Manifest-validated current prerequisites remain available through the runtime catalog without invoking Scriptorium; newly produced prerequisites enter that catalog with the same contract, checksum, and size identity used for persisted current evidence.
• before the first adapter call, checkpoints the reconciled starting collection when it removed configuration records or cleared obsolete authority. After each successful artifact, checkpoints the complete collection before making that artifact available to the next scheduled dependent. Analyze supplies its typed update to the shared checkpoint persistence boundary and refreshes its working and committed collections only after the save succeeds.
• keeps adapter output in the invocation's run-local analyze directory until it is a safe, non-empty, bounded regular file with a calculated checksum and complete output contract. Canonical replacement uses the shared atomic file operation boundary and verifies that the installed checksum matches the validated run-local bytes.
• records each successful artifact's freshly computed fingerprint, canonical relative output path, contract, checksum, size, producer run ID, bounded Scriptorium provenance, logs, and generated configuration references in the analyze-owned manifest collection.
• preserves valid unrelated current records during partial execution. If a rebuilt output's bytes and contract are unchanged, unselected dependents may remain current. If that semantic identity changes, unselected transitive dependents become stale without being executed; dependents included in the invocation are evaluated in dependency order instead.
• returns aggregate outputs from every current session record in normalized-key order after the complete reconciled collection has been checkpointed; unrelated stale records do not make an otherwise successful partial invocation fail.
• resolves previous-session sources from local previous/ cache only.
• runs optional render-debug, then artifact execution.
• validates non-empty output files and materializes canonical outputs.

Failure Semantics

• required missing configured/previous-session inputs fail.
• missing required prepared stable input source includes prepare rerun guidance.
• missing required previous-session source includes prepare rerun guidance.
• missing required narratio.transcript.final_markdown or narratio.transcript.final_trimmed_markdown inputs includes render rerun guidance.
• dependency cycles or unavailable required dependencies fail.
• adapter validation failures fail stage.
• a scheduled artifact failure installs a retry collection on the live manifest with the active artifact marked failed, a bounded error, and no output authority. Current transitive dependents become stale without execution.
• earlier artifacts remain current only after their run-local output passed validation, canonical materialization, and a successful complete-collection manifest checkpoint; unattempted later artifacts retain their reconciled session state.
• unrelated current records survive a partial failure. Old canonical bytes for the failed artifact and newly materialized bytes whose checkpoint fails are incidental, not current evidence.
• a checkpoint failure stops the queue immediately. The in-memory retry collection starts from the last successful checkpoint, marks the active artifact failed without output authority, and stales its current transitive dependents.
• the runner's ordinary error path marks aggregate analyze failed, clears its aggregate outputs, invalidates publish, notify, and cleanup through the application dependency relation, and persists the live retry collection. Terminal-save errors are joined with the original artifact or checkpoint failure.

Invariants

• analyze performs no remote storage calls for previous-session source resolution.
• input-identity resolution is read-only: it does not invoke adapters, materialize outputs, update status, or create run-local files.
• fingerprints exclude timeouts, retries, timestamps, producer and Narratio run IDs, executable and config paths, workspace roots, diagnostic locations, and executable or private transitive configuration contents. A change that is visible only inside Scriptorium—such as a file privately loaded by its config path—requires an explicit forced regeneration.
• output provenance and metadata are deterministic per execution.
• a canonical file without current per-artifact manifest evidence is never promoted to current state.
• an ordinary retry after interruption or partial failure reuses every valid earlier checkpoint and schedules only failed, stale, missing, or otherwise non-current work in dependency order.

Related Contracts And Tests

• Configuration owns artifact fields and source-selection rules, including artifact families.
• CLI owns user-visible artifact selection.
• Scriptorium owns the subprocess contract.
• Implementation and tests: internal/stage/analyze.go, internal/stage/analyze_artifact_execution.go, internal/stage/analyze_execution_state.go, internal/stage/analyze_input_identity.go, internal/stage/analyze_fingerprint.go, internal/stage/analyze_reconciliation.go, internal/stage/analyze_plan.go, internal/stage/analyze_resume.go, internal/stage/analyze_test.go, internal/stage/analyze_checkpoint_test.go, internal/stage/analyze_input_identity_test.go, internal/stage/analyze_fingerprint_test.go, internal/stage/analyze_reconciliation_test.go, internal/stage/analyze_plan_test.go, internal/stage/analyze_incremental_execution_test.go, internal/stage/analyze_failure_test.go, internal/stage/analyze_resume_test.go, and internal/app/analyze_checkpoint_recovery_test.go.