Internal: Manifest

Purpose

Explain the session-progress and invocation-audit models implemented by internal/manifest. Physical manifest placement belongs in Operations.

Session Manifest

manifest.Manifest records:

identity (session_id, campaign, run_id)
local path metadata (local_workdir, local_spool_dir)
remote identity metadata (s3_bucket, s3_session_prefix, s3_run_prefix)
inputs records
durable artifacts records
per-stage stages map

The model admits these stage states:

pending
running
succeeded
failed
skipped
stale
interrupted

Run Manifest

manifest.RunManifest is created for each invocation and records:

invocation identity and force flag
requested stages
per-stage action (run or skip)
per-stage status
overall run status (running, succeeded, failed)

Persistence Semantics

manifest.LocalStore:

validates loaded documents;
normalizes missing maps/stage records;
writes atomically via temp file + rename;
updates updated_at on save.

Execution Semantics

The application runner marks an executing stage running and then succeeded or failed in both manifests, persisting each transition. On success it records outputs, logs, generated configuration references, and metadata. Artifact records may include optional contract and external provenance objects; old manifests remain compatible when those fields are absent. A successful forced rerun marks only succeeded downstream session-stage records stale.

Starting an execution clears the current session-stage record's prior outputs, logs, generated configuration references, and metadata. Failed and skipped transitions enforce the same clearing rule directly, while success repopulates only fields returned by the new result. Marking a record stale does not clear those details because resume validation and diagnosis may still require them before execution begins. Invocation run manifests remain immutable audit records of their own outcomes.

A stage may explicitly return a skipped disposition and stable reason. The runner persists that outcome in both manifests, clears older outputs for the session-stage record along with older logs, generated configuration references, and metadata, then applies any bounded details from the current skip and continues. This self-skip is distinct from deciding not to execute an already-succeeded stage and is reconsidered on later runs. Skipped results cannot contain outputs.

When an already-succeeded stage is skipped, the invocation run manifest records the skip action and reason. The session manifest deliberately retains its existing succeeded record because it remains the cross-invocation progress authority. Stages with a resume validator, currently extraction, may reject an otherwise eligible skip when the recorded durable result is obsolete; the runner marks it stale and executes it.

Session manifest is the authoritative stage-progress ledger across invocations. Run manifest is invocation-scoped audit state.

Invariants

stage resume/skip decisions are session-manifest driven.
running, failed, and self-skipped stages do not retain result payloads from an earlier success.
stale stages retain prior details until replacement execution starts.
force reruns stale downstream succeeded stages.
run manifest does not replace session manifest as progress authority.

Implementation And Tests

Models and transitions: internal/manifest/manifest.go, internal/manifest/run_manifest.go
Persistence and validation: internal/manifest/store.go
Package tests: internal/manifest/*_test.go
Assembled execution behavior: internal/app/runner_test.go, internal/app/run_stage_test.go

3.7 KiB Raw Blame History