narratio/docs/internal/manifest.md

Internal: Manifest

Purpose

Explain the session-progress and invocation-audit models implemented by internal/manifest. Physical manifest placement belongs in Operations.

Session Manifest

manifest.Manifest records:

• identity (session_id, campaign, run_id)
• local path metadata (local_workdir, local_spool_dir)
• remote identity metadata (s3_bucket, s3_session_prefix, s3_run_prefix)
• inputs records
• durable artifacts records
• per-stage stages map
• an optional post_publish_cleanup obligation, which binds a committed run, remote commit identity, and each exact root-confined local target to its completion evidence

Session, campaign, and run identities in local and downloaded manifests must be portable opaque segments. Unsafe legacy identities are rejected with migration guidance rather than being normalized into a different workspace or remote namespace.

The model admits these stage states:

• pending
• running
• succeeded
• failed
• skipped
• stale
• interrupted

Analyze-owned artifact state

The analyze stage record may carry analyze_state_version: 1 and an analyze_artifacts map keyed by normalized configured artifact key. The version is the authority marker: version 1 with no entries is a valid evaluated empty set, while an absent version is legacy aggregate-only state and provides no current configured-artifact evidence.

Each analyze artifact record has one disposition:

• current: the configured artifact is available and carries a versioned fingerprint plus a complete output record and separate output size;
• stale: the recorded semantic identity is no longer current;
• missing: no validated current result exists;
• failed: the attempted work failed and carries a bounded diagnostic; or
• unselected: the artifact was intentionally outside the evaluated set.

Records bind their normalized key and dependencies, fingerprint contract when evaluated, canonical session-relative output identity when current, producing Narratio run, update time, and bounded non-secret Scriptorium provenance and diagnostic paths. A current output includes its configured source ID, contract, checksum, and positive byte size. Non-current records cannot carry an output, so an older file is not advertised through stale, missing, failed, or unselected state.

The session-stage collection is the reconciled authority across invocations. The corresponding collection on an invocation's analyze stage record is an audit of only the artifacts evaluated or attempted by that run. These records remain analyze-owned data inside the fixed stage; they are not dynamic stages or generic subtasks.

The stage result contract has one analyze-specific projection boundary. On success, the runner validates and deep-copies the complete reconciled session collection and the invocation subset. Aggregate session outputs are rebuilt in configured-key order from current session records only; invocation outputs are limited to current records produced by that invocation's run ID. Ordinary stage outputs cannot accompany this projection, so there is one source of artifact authority.

Successful incremental execution replaces only evaluated artifact records and preserves valid unrelated current records. Rebuilt outputs are compared by bytes and contract: an unchanged identity permits an unselected dependent with the same recomputed fingerprint to remain current, while a changed identity removes output authority from every unselected transitive dependent by marking it stale. A partial analyze invocation can therefore succeed while unrelated configured records remain stale. Existing canonical files never create current records without validated execution and projection.

Analyze may return a projection together with an error. That restricted result cannot carry ordinary outputs, skip state, aggregate logs, generated configs, or metadata. The runner persists only the validated per-artifact collections, then marks the aggregate analyze and run state failed and invalidates delivery dependents conservatively. Unrelated current records survive because the session projection is complete. A malformed projection is not applied, and a failed session projection save restores the prior per-artifact authority before terminal failure persistence.

The incremental executor constructs this restricted projection at each scheduled artifact boundary. The active record is failed without output, current transitive dependents are stale, unrelated current records survive, and only earlier validated and materialized completions remain current in the invocation subset. Session failure state is persisted before invocation failure state. If either terminal save fails, its persistence error is joined with the original adapter, validation, or filesystem cause; a failed projection save does not turn incidental canonical bytes into manifest authority.

Run Manifest

manifest.RunManifest is created for each invocation and records:

• invocation identity and force flag
• requested stages
• per-stage action (run or skip)
• per-stage status
• overall run status (running, succeeded, failed)

Remote Commit Manifest

artifacts.RemoteCommitManifest is a separate, versioned remote snapshot contract. It is not a serialized session manifest and contains no local post-publication assertion such as current_pointer_written. A remote commit identifies one campaign, session, and run and declares its immutable artifact set. Each artifact has a typed source, immutable destination key, SHA-256 checksum, size, and storage generation.

current/commit-pointer.json is the sole mutable selector for the new contract. It identifies exactly one run-scoped runs/{run_id}/commit.json and binds that object by checksum, size, and generation. Readers strictly reject unknown fields, version mismatches, pointer/commit identity mismatches, and objects that do not match their declaration.

The reader retains a temporary, clearly isolated compatibility path for a coherent legacy current/manifest.json plus current/run_id.txt pair. That path is removable after migration and is never used to write new state.

Persistence Semantics

manifest.LocalStore:

• validates loaded documents;
• normalizes missing maps/stage records;
• writes through a sibling temporary file, syncing the completed file and destination directory after atomic replacement;
• updates updated_at on save.

If the operating system or filesystem cannot sync a directory, save returns an explicit error instead of claiming crash-durable replacement. A returned error after the rename can therefore leave the new manifest visible but not confirmed durable; callers must reload it before retrying.

Execution Semantics

The application runner marks an executing stage running and then succeeded or failed in both manifests, persisting each transition. On success it records outputs, logs, generated configuration references, and metadata. Artifact records may include optional contract and external provenance objects; old manifests remain compatible when those fields are absent. A successful forced rerun marks only succeeded transitive dependent session-stage records stale. The application owns a fixed dependency relation distinct from execution order; dependents are returned in canonical order. Render and extract therefore never stale one another, while either can stale analyze, publish, and notify.

Starting an execution clears the current session-stage record's prior outputs, logs, generated configuration references, and metadata. Failed and skipped transitions enforce the same clearing rule directly, while success repopulates only fields returned by the new result. Marking a record stale does not clear those details because resume validation and diagnosis may still require them before execution begins. Invocation run manifests remain immutable audit records of their own outcomes.

Aggregate lifecycle clearing deliberately preserves the analyze-owned per-artifact collection. This lets later reconciliation replace only evaluated entries without erasing unrelated current results. Other stages retain their existing aggregate-only lifecycle behavior and are forbidden from carrying the analyze-specific fields.

A stage may explicitly return a skipped disposition and stable reason. The runner persists that outcome in both manifests, clears older outputs for the session-stage record along with older logs, generated configuration references, and metadata, then applies any bounded details from the current skip and continues. This self-skip is distinct from deciding not to execute an already-succeeded stage and is reconsidered on later runs. Skipped results cannot contain outputs.

When an already-succeeded stage is skipped, the invocation run manifest records the skip action and reason. The session manifest deliberately retains its existing succeeded record because it remains the cross-invocation progress authority. Stages with a resume validator, currently extraction, may reject an otherwise eligible skip when the recorded durable result is obsolete; the runner marks it stale and executes it.

Session manifest is the authoritative stage-progress ledger across invocations. Run manifest is invocation-scoped audit state.

Before an explicitly bounded execution starts after prepare, the application reads the session manifest and accepts only succeeded or skipped for every excluded canonical prefix stage. The first other status or absent record fails the request before layout mutation, adapter initialization, session-manifest writes, or run-manifest creation. Excluded prefix records are not passed to resume validators. Records after the selected end are not prerequisites and may be made stale by selected work without being scheduled.

After a publish commits remotely, any configured local cleanup is first recorded as a session-manifest obligation before deletion begins. Each target becomes complete only after its confined deletion (or safe absence check) and a successful manifest save. An incomplete obligation is retried when publish executes again and retains the committed run and remote identity that authorized it; an invocation that does not execute publish does not perform cleanup.

Each invocation derives campaign, session, run, local-path, and remote-prefix metadata from the validated resolved configuration as one projection. A persisted session manifest must agree on campaign and session identity before execution; the current projection is refreshed for every invocation while stage progress, inputs, and durable artifacts remain session history.

For handled failures after an invocation record is created, the runner records the failure on the session ledger and persists it before persisting the failed run audit record. This preserves the resume authority while making a partial persistence disagreement visible. Abrupt process death remains an accepted case where a durable running record can require operator interpretation.

Invariants

• stage resume/skip decisions are session-manifest driven.
• running, failed, and self-skipped stages do not retain result payloads from an earlier success.
• stale stages retain prior details until replacement execution starts.
• force reruns stale succeeded stages in the fixed dependency relation.
• run manifest does not replace session manifest as progress authority.
• remote commitment is established by a verified current pointer and remote commit relationship, never by a mutable session-manifest boolean.

Implementation And Tests

• Models and transitions: internal/manifest/manifest.go, internal/manifest/run_manifest.go
• Remote commit model and readers: internal/artifacts/remote_commit.go, internal/artifacts/current_state_commit.go, internal/artifacts/current_state_legacy.go
• Persistence and validation: internal/manifest/store.go
• Package tests: internal/manifest/*_test.go
• Assembled execution behavior: internal/app/runner_test.go, internal/app/run_stage_test.go