narratio/docs/internal/manifest.md

Internal: Manifest

Purpose

Explain the session-progress model implemented by internal/manifest. Physical manifest placement belongs in Operations.

Session Manifest

manifest.Manifest records:

• required schema_version: narratio.session-manifest.v1
• identity (session_id, campaign, run_id)
• local path metadata (local_workdir, local_spool_dir)
• remote identity metadata (s3_bucket, s3_session_prefix, s3_run_prefix)
• inputs records
• durable artifacts records
• per-stage stages map
• an optional post_publish_cleanup obligation, which binds a committed run, remote commit identity, and each exact root-confined local target to its completion evidence

Session, campaign, and run identities in local and downloaded manifests must be portable opaque segments. Unsafe legacy identities are rejected with migration guidance rather than being normalized into a different workspace or remote namespace.

Every persisted session manifest has a non-nil stages map. Each entry is non-nil and its embedded name exactly matches its map key; loading never invents either marker. A missing or unsupported schema version, or incomplete stage identity, is unsupported local state: move it aside and regenerate it, or restore a verified current committed snapshot when that is authoritative.

Prepare records independent party and players input checksums. In canonical party mode, the party record retains its campaign source identity while the players record uses derived_from_party; raw roster content is never embedded in manifest metadata. Both records remain the durable authority for consumers of their prepared input source IDs.

The model admits these stage states:

• pending
• running
• succeeded
• failed
• skipped
• stale
• interrupted

Analyze-owned artifact state

Every analyze stage record carries analyze_state_version: 1 and an analyze_artifacts map keyed by normalized configured artifact key. The version is the authority marker: version 1 with no entries is a valid evaluated empty set. An absent or unsupported version is unsupported local state and is rejected rather than being treated as aggregate-only authority.

Each analyze artifact record has one disposition:

• current: the configured artifact is available and carries a versioned fingerprint plus a complete output record and separate output size;
• stale: the recorded semantic identity is no longer current;
• missing: no validated current result exists;
• failed: the attempted work failed and carries a bounded diagnostic; or
• unselected: the artifact was intentionally outside the evaluated set.

Records bind their normalized key and dependencies, fingerprint contract when evaluated, canonical session-relative output identity when current, producing Narratio run, update time, and bounded non-secret Scriptorium provenance and diagnostic paths. A current output includes its configured source ID, contract, checksum, and positive byte size. Non-current records cannot carry an output, so an older file is not advertised through stale, missing, failed, or unselected state.

Family-produced records additionally retain optional family and character_id provenance supplied by configuration resolution. These fields do not replace the concrete configured key or infer family membership from a name, so older records without them remain valid.

The session-stage collection is the reconciled authority across invocations. These records remain analyze-owned data inside the fixed stage; they are not dynamic stages or generic subtasks.

Analyze checkpoints one complete validated candidate collection after each successful artifact and before exposing that artifact to later work. It saves a detached manifest candidate and installs its version, collection, and updated timestamp into live state only after the store reports success. Aggregate stage outputs are ordinary result outputs rebuilt in configured-key order from current session records only.

Successful incremental execution replaces only evaluated artifact records and preserves valid unrelated current records. Rebuilt outputs are compared by bytes and contract: an unchanged identity permits an unselected dependent with the same recomputed fingerprint to remain current, while a changed identity removes output authority from every unselected transitive dependent by marking it stale. A partial analyze invocation can therefore succeed while unrelated configured records remain stale. Existing canonical files never create current records without validated execution and a successful checkpoint.

Aggregate analyze status is deliberately coarser than this collection. Resume validation may skip a succeeded aggregate record when the selected artifact closure is current even if unrelated records are stale. Conversely, a stale aggregate record may cross the ordinary runner boundary and perform zero Scriptorium calls when reconciliation proves every selected artifact current; ordinary successful stage completion then restores the aggregate status.

On an artifact error, analyze installs a complete retry collection on the live manifest before returning. The active record is failed without output, current transitive dependents are stale, unrelated current records survive, and only earlier checkpointed completions remain current. The runner's ordinary failure path marks aggregate analyze failed, clears aggregate outputs, invalidates delivery dependents, and saves that collection. A terminal session-save failure is joined with the original adapter, validation, filesystem, cancellation, or checkpoint cause.

If an artifact checkpoint fails, analyze rolls its working collection back to the last committed collection before constructing the active failed record. Newly materialized canonical bytes therefore remain incidental and cannot become current authority. A later ordinary run reconciles the durable records, reuses earlier current artifacts, and resumes with the failed or stale work. The same recovery applies after process interruption: the aggregate record may be non-successful, but every completed artifact checkpoint remains reusable.

Remote Commit Manifest

artifacts.RemoteCommitManifest is a separate, versioned remote snapshot contract. It is not a serialized session manifest and contains no local post-publication assertion such as current_pointer_written. A remote commit identifies one campaign, session, and run and declares its immutable artifact set. Each artifact has a typed source, immutable destination key, SHA-256 checksum, size, and storage generation.

current/commit-pointer.json is the sole mutable selector for the new contract. It identifies exactly one run-scoped runs/{run_id}/commit.json and binds that object by checksum, size, and generation. Readers strictly reject unknown fields, version mismatches, pointer/commit identity mismatches, and objects that do not match their declaration.

An absent commit pointer means no remote state is published. Republish to establish a current immutable snapshot.

Persistence Semantics

manifest.LocalStore:

• accepts only the exact supported session-manifest schema version;
• validates stage-map identity without synthesizing missing maps, records, or names;
• retains only representation-preserving collection normalization;
• writes through a sibling temporary file and atomic replacement visibility;
• updates updated_at on save.

Manifest replacement provides atomic visibility, not a crash- or power-loss durability guarantee. Callers reload after a returned save error before retrying. An analyze checkpoint error leaves the prior manifest as per-artifact authority. The runner then attempts to persist the bounded failed/stale retry collection; if that terminal save also fails, its error is joined with the originating failure and the next invocation must reconcile from the last readable manifest.

Execution Semantics

Planning and execution share one pure pre-execution decision over the recorded stage status, force selection, and normalized resume-validation result. That decision identifies run versus reuse and any required stale and dependent invalidation transition. Planning applies it only to a cloned manifest. Execution remains the sole owner of observed stage results and persistence.

For each selected stage, the application runner computes semantic evidence, performs any eligible resume validation, applies the pre-execution decision, and persists required invalidation before either reusing the existing success or beginning an attempt. The same one-stage execution owner marks the attempt running before invoking the stage and then records and persists its failed, skipped, or successful terminal state before a later stage may start. On success it records outputs, logs, generated configuration references, metadata, and—when the stage implements the optional contract—a versioned semantic-configuration fingerprint. Artifact records may include optional contract and external provenance objects; old manifests remain compatible when those fields are absent. A successful forced rerun marks only succeeded transitive dependent session-stage records stale. The application owns a fixed dependency relation distinct from execution order; dependents are returned in canonical order. Render and extract therefore never stale one another, while either can stale analyze, publish, and notify.

Starting an execution clears the current session-stage record's prior outputs, logs, generated configuration references, metadata, and semantic fingerprint. Failed and skipped transitions enforce the same clearing rule directly, while success repopulates only fields returned by the new result. Marking a record stale does not clear those details because resume validation and diagnosis may still require them before execution begins.

Aggregate lifecycle clearing deliberately preserves the analyze-owned per-artifact collection. This lets later reconciliation replace only evaluated entries without erasing unrelated current results. Other stages retain their existing aggregate-only lifecycle behavior and are forbidden from carrying the analyze-specific fields.

A stage may explicitly return a skipped disposition and stable reason. The runner persists that outcome in the session manifest, clears older outputs, logs, generated configuration references, and metadata, then applies any bounded details from the current skip and continues. This self-skip is distinct from deciding not to execute an already-succeeded stage and is reconsidered on later runs. Skipped results cannot contain outputs. An intentional self-skip records the current semantic fingerprint because it is a completed, reusable stage result; failed or interrupted work never promotes one.

When an already-succeeded stage is skipped, the session manifest deliberately retains its existing succeeded record because it remains the cross-invocation progress authority. If a stage supplies semantic configuration evidence, reuse first requires the persisted positive schema version and lowercase SHA-256 digest to match the current resolved stage semantics. Missing semantic evidence, malformed evidence, or a mismatch makes the stage and its fixed transitive dependents stale. The existing stage-specific resume validator runs only after this semantic check succeeds; both checks are required. Extraction and analyze have resume validators and may reject an otherwise eligible skip when their selected durable evidence is obsolete; the runner marks the aggregate record stale and executes it. Analyze's validator can still accept a partial selection when only unrelated artifact records are stale.

Implemented reuse coverage is deliberately split between aggregate semantic evidence and focused durable validators:

Work	Reuse authority	Focused owners
prepare	aggregate semantic fingerprint	prepare
transcribe	aggregate semantic fingerprint	transcribe, WhisperX
merge	aggregate semantic fingerprint	merge, Seriatim
polish	aggregate semantic fingerprint	polish, Audita
normalize	aggregate semantic fingerprint	normalize, Seriatim
trim	aggregate semantic fingerprint	trim, Scriptorium, Seriatim
render	aggregate semantic fingerprint	render, Seriatim
extract	aggregate semantic fingerprint plus reference/output validator	extract, Notarius
analyze artifacts	per-artifact fingerprint, reconciliation, and output validator	analyze, Scriptorium
publish	aggregate semantic fingerprint plus immediate lock/commit checks	publish, storage adapter
notify	aggregate delivery-mode fingerprint	pipeline overview, configuration

These contracts record resolved choices Narratio can observe, not operational runner tuning. External model, module, prompt, profile, and configuration-file contents that a tool privately loads remain outside the contract when their configured identifier is unchanged; operators must force the affected work after such a private content change.

The session manifest is the authoritative stage-progress ledger across invocations. It retains the most recently resolved invocation's bounded configuration provenance. It identifies the selected profile name and source (default or cli) plus the effective configuration digest, but never a raw secret or profile content. This provenance is informational: it does not participate in stage resume or cache decisions. A profile change therefore invalidates only stages whose semantic configuration changed. When a private external-tool model, module, prompt, or profile changes behind an unchanged configured identifier, use --force for the affected work.

session plan computes the same current fingerprint and applies the shared decision and invalidation rules to a cloned manifest. It predicts the runner decision without persisting session state. The semantic fingerprint owner hashes deterministic JSON from stage-owned typed structs; stage providers must exclude secrets, complete effective-configuration dumps, and operational values that cannot affect canonical results. Concrete coverage is owned by the focused stage and integration documents linked above.

Before an explicitly bounded execution starts after prepare, the application reads the session manifest and accepts only succeeded or skipped for every excluded canonical prefix stage. The first other status or absent record fails the request before layout mutation, adapter initialization, session-manifest writes, or run-directory creation. Excluded prefix records are not passed to resume validators. Records after the selected end are not prerequisites and may be made stale by selected work without being scheduled.

After a publish commits remotely, any configured local cleanup is first recorded as a session-manifest obligation before deletion begins. Each target becomes complete only after its confined deletion (or safe absence check) and a successful manifest save. An incomplete obligation is retried when publish executes again and retains the committed run and remote identity that authorized it; an invocation that does not execute publish does not perform cleanup.

Each invocation derives campaign, session, run, local-path, and remote-prefix metadata from the validated resolved configuration as one projection. A persisted session manifest must agree on campaign and session identity before execution; the current projection is refreshed for every invocation while stage progress, inputs, and durable artifacts remain session history.

For handled failures after the session identity transition, the runner records the failure on the session ledger and atomically attempts its save while preserving the originating error. Abrupt process death remains an accepted case where the last session transition can require operator interpretation.

Invariants

• stage resume/skip decisions are session-manifest driven.
• semantic fingerprint comparison precedes stage-specific resume validation.
• only successful and intentional-skipped results promote current semantic evidence.
• running, failed, and self-skipped stages do not retain result payloads from an earlier success.
• stale stages retain prior details until replacement execution starts.
• force reruns stale succeeded stages in the fixed dependency relation.
• the session manifest is the only durable local progress ledger.
• remote commitment is established by a verified current pointer and remote commit relationship, never by a mutable session-manifest boolean.

Implementation And Tests

• Models and transitions: internal/manifest/manifest.go
• Remote commit model and readers: internal/artifacts/remote_commit.go, internal/artifacts/current_state_commit.go,
• Persistence and validation: internal/manifest/store.go
• Package tests: internal/manifest/*_test.go
• Assembled execution behavior: internal/app/runner_test.go, internal/app/run_stage_test.go