narratio/docs/internal/manifest.md

# Internal: Manifest

## Purpose

Explain the session-progress and invocation-audit models implemented by
`internal/manifest`. Physical manifest placement belongs in
[Operations](../operations.md#local-state-layout).

## Session Manifest

`manifest.Manifest` records:

- identity (`session_id`, `campaign`, `run_id`)
- local path metadata (`local_workdir`, `local_spool_dir`)
- remote identity metadata (`s3_bucket`, `s3_session_prefix`, `s3_run_prefix`)
- `inputs` records
- durable `artifacts` records
- per-stage `stages` map
- an optional `post_publish_cleanup` obligation, which binds a committed run,
  remote commit identity, and each exact root-confined local target to its
  completion evidence

Session, campaign, and run identities in local and downloaded manifests must be
portable opaque segments. Unsafe legacy identities are rejected with migration
guidance rather than being normalized into a different workspace or remote
namespace.

The model admits these stage states:

- `pending`
- `running`
- `succeeded`
- `failed`
- `skipped`
- `stale`
- `interrupted`

### Analyze-owned artifact state

The `analyze` stage record may carry `analyze_state_version: 1` and an
`analyze_artifacts` map keyed by normalized configured artifact key. The
version is the authority marker: version 1 with no entries is a valid evaluated
empty set, while an absent version is legacy aggregate-only state and provides
no current configured-artifact evidence.

Each analyze artifact record has one disposition:

- `current`: the configured artifact is available and carries a versioned
  fingerprint plus a complete output record and separate output size;
- `stale`: the recorded semantic identity is no longer current;
- `missing`: no validated current result exists;
- `failed`: the attempted work failed and carries a bounded diagnostic; or
- `unselected`: the artifact was intentionally outside the evaluated set.

Records bind their normalized key and dependencies, fingerprint contract when
evaluated, canonical session-relative output identity when current, producing
Narratio run, update time, and bounded non-secret Scriptorium provenance and
diagnostic paths. A current output includes its configured source ID, contract,
checksum, and positive byte size. Non-current records cannot carry an output,
so an older file is not advertised through stale, missing, failed, or
unselected state.

The session-stage collection is the reconciled authority across invocations.
The corresponding collection on an invocation's `analyze` stage record is an
audit of only the artifacts evaluated or attempted by that run. These records
remain analyze-owned data inside the fixed stage; they are not dynamic stages
or generic subtasks.

The stage result contract has one analyze-specific projection boundary. On
success, the runner validates and deep-copies the complete reconciled session
collection and the invocation subset. Aggregate session outputs are rebuilt in
configured-key order from current session records only; invocation outputs are
limited to current records produced by that invocation's run ID. Ordinary
stage outputs cannot accompany this projection, so there is one source of
artifact authority.

Successful incremental execution replaces only evaluated artifact records and
preserves valid unrelated current records. Rebuilt outputs are compared by
bytes and contract: an unchanged identity permits an unselected dependent with
the same recomputed fingerprint to remain current, while a changed identity
removes output authority from every unselected transitive dependent by marking
it stale. A partial analyze invocation can therefore succeed while unrelated
configured records remain stale. Existing canonical files never create current
records without validated execution and projection.

Aggregate analyze status is deliberately coarser than this collection. Resume
validation may skip a succeeded aggregate record when the selected artifact
closure is current even if unrelated records are stale. Conversely, a stale
aggregate record may cross the ordinary runner boundary and perform zero
Scriptorium calls when reconciliation proves every selected artifact current;
the successful projection then restores the aggregate status.

Analyze may return a projection together with an error. That restricted result
cannot carry ordinary outputs, skip state, aggregate logs, generated configs,
or metadata. The runner persists only the validated per-artifact collections,
then marks the aggregate analyze and run state failed and invalidates delivery
dependents conservatively. Unrelated current records survive because the
session projection is complete. A malformed projection is not applied, and a
failed session projection save restores the prior per-artifact authority before
terminal failure persistence.

The incremental executor constructs this restricted projection at each
scheduled artifact boundary. The active record is failed without output,
current transitive dependents are stale, unrelated current records survive, and
only earlier validated and materialized completions remain current in the
invocation subset. Session failure state is persisted before invocation failure
state. If either terminal save fails, its persistence error is joined with the
original adapter, validation, or filesystem cause; a failed projection save
does not turn incidental canonical bytes into manifest authority.

## Run Manifest

`manifest.RunManifest` is created for each invocation and records:

- invocation identity and `force` flag
- requested stages
- per-stage action (`run` or `skip`)
- per-stage status
- overall run status (`running`, `succeeded`, `failed`)

## Remote Commit Manifest

`artifacts.RemoteCommitManifest` is a separate, versioned remote snapshot
contract. It is not a serialized session manifest and contains no local
post-publication assertion such as `current_pointer_written`. A remote commit
identifies one campaign, session, and run and declares its immutable artifact
set. Each artifact has a typed source, immutable destination key, SHA-256
checksum, size, and storage generation.

`current/commit-pointer.json` is the sole mutable selector for the new
contract. It identifies exactly one run-scoped `runs/{run_id}/commit.json` and
binds that object by checksum, size, and generation. Readers strictly reject
unknown fields, version mismatches, pointer/commit identity mismatches, and
objects that do not match their declaration.

The reader retains a temporary, clearly isolated compatibility path for a
coherent legacy `current/manifest.json` plus `current/run_id.txt` pair. That
path is removable after migration and is never used to write new state.

## Persistence Semantics

`manifest.LocalStore`:

- validates loaded documents;
- normalizes missing maps/stage records;
- writes through a sibling temporary file, syncing the completed file and
  destination directory after atomic replacement;
- updates `updated_at` on save.

If the operating system or filesystem cannot sync a directory, save returns an
explicit error instead of claiming crash-durable replacement. A returned error
after the rename can therefore leave the new manifest visible but not confirmed
durable; callers must reload it before retrying.

## Execution Semantics

The application runner marks an executing stage running and then succeeded or
failed in both manifests, persisting each transition. On success it records
outputs, logs, generated configuration references, metadata, and—when the
stage implements the optional contract—a versioned semantic-configuration
fingerprint. Artifact
records may include optional contract and external provenance objects; old
manifests remain compatible when those fields are absent. A successful forced
rerun marks only succeeded transitive dependent session-stage records stale.
The application owns a fixed dependency relation distinct from execution order;
dependents are returned in canonical order. Render and extract therefore never
stale one another, while either can stale analyze, publish, and notify.

Starting an execution clears the current session-stage record's prior outputs,
logs, generated configuration references, metadata, and semantic fingerprint.
Failed and skipped
transitions enforce the same clearing rule directly, while success repopulates
only fields returned by the new result. Marking a record stale does not clear
those details because resume validation and diagnosis may still require them
before execution begins. Invocation run manifests remain immutable audit
records of their own outcomes.

Aggregate lifecycle clearing deliberately preserves the analyze-owned
per-artifact collection. This lets later reconciliation replace only evaluated
entries without erasing unrelated current results. Other stages retain their
existing aggregate-only lifecycle behavior and are forbidden from carrying the
analyze-specific fields.

A stage may explicitly return a skipped disposition and stable reason. The
runner persists that outcome in both manifests, clears older outputs for the
session-stage record along with older logs, generated configuration references,
and metadata, then applies any bounded details from the current skip and
continues. This self-skip is distinct from deciding not to execute an
already-succeeded stage and is reconsidered on later runs. Skipped results
cannot contain outputs. An intentional self-skip records the current semantic
fingerprint because it is a completed, reusable stage result; failed or
interrupted work never promotes one.

When an already-succeeded stage is skipped, the invocation run manifest records
the `skip` action and reason. The session manifest deliberately retains its
existing succeeded record because it remains the cross-invocation progress
authority. If a stage supplies semantic configuration evidence, reuse first
requires the persisted positive schema version and lowercase SHA-256 digest to
match the current resolved stage semantics. Missing legacy evidence, malformed
evidence, or a mismatch makes the stage and its fixed transitive dependents
stale. The invocation skip copies the matched fingerprint for provenance but
does not rewrite session authority. The existing stage-specific resume
validator runs only after this semantic check succeeds; both checks are
required. Extraction and analyze have resume validators and may reject an
otherwise eligible skip when their selected durable evidence is obsolete; the
runner marks the aggregate record stale and executes it. Analyze's validator
can still accept a partial selection when only unrelated artifact records are
stale.

Implemented reuse coverage is deliberately split between aggregate semantic
evidence and focused durable validators:

| Work | Reuse authority | Focused owners |
| --- | --- | --- |
| prepare | aggregate semantic fingerprint | [prepare](stage-prepare.md) |
| transcribe | aggregate semantic fingerprint | [transcribe](stage-transcribe.md), [WhisperX](../integrations/whisperx.md) |
| merge | aggregate semantic fingerprint | [merge](stage-merge.md), [Seriatim](../integrations/seriatim.md) |
| polish | aggregate semantic fingerprint | [polish](stage-polish.md), [Audita](../integrations/audita.md) |
| normalize | aggregate semantic fingerprint | [normalize](stage-normalize.md), [Seriatim](../integrations/seriatim.md) |
| trim | aggregate semantic fingerprint | [trim](stage-trim.md), [Scriptorium](../integrations/scriptorium.md), [Seriatim](../integrations/seriatim.md) |
| render | aggregate semantic fingerprint | [render](stage-render.md), [Seriatim](../integrations/seriatim.md) |
| extract | aggregate semantic fingerprint plus reference/output validator | [extract](stage-extract.md), [Notarius](../integrations/notarius.md) |
| analyze artifacts | per-artifact fingerprint, reconciliation, and output validator | [analyze](stage-analyze.md), [Scriptorium](../integrations/scriptorium.md) |
| publish | aggregate semantic fingerprint plus immediate lock/commit checks | [publish](stage-publish.md), [storage adapter](adapters.md) |
| notify | aggregate delivery-mode fingerprint | [pipeline overview](overview.md), [configuration](../config.md#notifications) |

These contracts record resolved choices Narratio can observe, not operational
runner tuning. External model, module, prompt, profile, and configuration-file
contents that a tool privately loads remain outside the contract when their
configured identifier is unchanged; operators must force the affected work
after such a private content change.

Session manifest is the authoritative stage-progress ledger across invocations.
Run manifest is invocation-scoped audit state.

`session plan` computes the same current fingerprint and applies the same
comparison and invalidation rules to a cloned manifest. It predicts the runner
decision without persisting session or invocation state. The shared helper
hashes deterministic JSON from stage-owned typed structs; stage providers must
exclude secrets, complete effective-configuration dumps, and operational
values that cannot affect canonical results. Concrete coverage is owned by the
focused stage and integration documents linked above.

Before an explicitly bounded execution starts after `prepare`, the application
reads the session manifest and accepts only `succeeded` or `skipped` for every
excluded canonical prefix stage. The first other status or absent record fails
the request before layout mutation, adapter initialization, session-manifest
writes, or run-manifest creation. Excluded prefix records are not passed to
resume validators. Records after the selected end are not prerequisites and
may be made stale by selected work without being scheduled.

After a publish commits remotely, any configured local cleanup is first recorded
as a session-manifest obligation before deletion begins. Each target becomes
complete only after its confined deletion (or safe absence check) and a
successful manifest save. An incomplete obligation is retried when publish
executes again and retains the committed run and remote identity that authorized
it; an invocation that does not execute publish does not perform cleanup.

Each invocation derives campaign, session, run, local-path, and remote-prefix
metadata from the validated resolved configuration as one projection. A persisted
session manifest must agree on campaign and session identity before execution;
the current projection is refreshed for every invocation while stage progress,
inputs, and durable artifacts remain session history.

For handled failures after an invocation record is created, the runner records
the failure on the session ledger and persists it before persisting the failed
run audit record. This preserves the resume authority while making a partial
persistence disagreement visible. Abrupt process death remains an accepted case
where a durable running record can require operator interpretation.

## Invariants

- stage resume/skip decisions are session-manifest driven.
- semantic fingerprint comparison precedes stage-specific resume validation.
- only successful and intentional-skipped results promote current semantic
  evidence; invocation reuse copies evidence without replacing session state.
- running, failed, and self-skipped stages do not retain result payloads from
  an earlier success.
- stale stages retain prior details until replacement execution starts.
- force reruns stale succeeded stages in the fixed dependency relation.
- run manifest does not replace session manifest as progress authority.
- remote commitment is established by a verified current pointer and remote
  commit relationship, never by a mutable session-manifest boolean.

## Implementation And Tests

- Models and transitions: `internal/manifest/manifest.go`,
  `internal/manifest/run_manifest.go`
- Remote commit model and readers: `internal/artifacts/remote_commit.go`,
  `internal/artifacts/current_state_commit.go`,
  `internal/artifacts/current_state_legacy.go`
- Persistence and validation: `internal/manifest/store.go`
- Package tests: `internal/manifest/*_test.go`
- Assembled execution behavior: `internal/app/runner_test.go`,
  `internal/app/run_stage_test.go`