124 lines
5.1 KiB
Markdown
124 lines
5.1 KiB
Markdown
# Internal: Manifest
|
|
|
|
## Purpose
|
|
|
|
Explain the session-progress and invocation-audit models implemented by
|
|
`internal/manifest`. Physical manifest placement belongs in
|
|
[Operations](../operations.md#local-state-layout).
|
|
|
|
## Session Manifest
|
|
|
|
`manifest.Manifest` records:
|
|
|
|
- identity (`session_id`, `campaign`, `run_id`)
|
|
- local path metadata (`local_workdir`, `local_spool_dir`)
|
|
- remote identity metadata (`s3_bucket`, `s3_session_prefix`, `s3_run_prefix`)
|
|
- `inputs` records
|
|
- durable `artifacts` records
|
|
- per-stage `stages` map
|
|
|
|
Session, campaign, and run identities in local and downloaded manifests must be
|
|
portable opaque segments. Unsafe legacy identities are rejected with migration
|
|
guidance rather than being normalized into a different workspace or remote
|
|
namespace.
|
|
|
|
The model admits these stage states:
|
|
|
|
- `pending`
|
|
- `running`
|
|
- `succeeded`
|
|
- `failed`
|
|
- `skipped`
|
|
- `stale`
|
|
- `interrupted`
|
|
|
|
## Run Manifest
|
|
|
|
`manifest.RunManifest` is created for each invocation and records:
|
|
|
|
- invocation identity and `force` flag
|
|
- requested stages
|
|
- per-stage action (`run` or `skip`)
|
|
- per-stage status
|
|
- overall run status (`running`, `succeeded`, `failed`)
|
|
|
|
## Persistence Semantics
|
|
|
|
`manifest.LocalStore`:
|
|
|
|
- validates loaded documents;
|
|
- normalizes missing maps/stage records;
|
|
- writes through a sibling temporary file, syncing the completed file and
|
|
destination directory after atomic replacement;
|
|
- updates `updated_at` on save.
|
|
|
|
If the operating system or filesystem cannot sync a directory, save returns an
|
|
explicit error instead of claiming crash-durable replacement. A returned error
|
|
after the rename can therefore leave the new manifest visible but not confirmed
|
|
durable; callers must reload it before retrying.
|
|
|
|
## Execution Semantics
|
|
|
|
The application runner marks an executing stage running and then succeeded or
|
|
failed in both manifests, persisting each transition. On success it records
|
|
outputs, logs, generated configuration references, and metadata. Artifact
|
|
records may include optional contract and external provenance objects; old
|
|
manifests remain compatible when those fields are absent. A successful forced
|
|
rerun marks only succeeded downstream session-stage records stale.
|
|
|
|
Starting an execution clears the current session-stage record's prior outputs,
|
|
logs, generated configuration references, and metadata. Failed and skipped
|
|
transitions enforce the same clearing rule directly, while success repopulates
|
|
only fields returned by the new result. Marking a record stale does not clear
|
|
those details because resume validation and diagnosis may still require them
|
|
before execution begins. Invocation run manifests remain immutable audit
|
|
records of their own outcomes.
|
|
|
|
A stage may explicitly return a skipped disposition and stable reason. The
|
|
runner persists that outcome in both manifests, clears older outputs for the
|
|
session-stage record along with older logs, generated configuration references,
|
|
and metadata, then applies any bounded details from the current skip and
|
|
continues. This self-skip is distinct from deciding not to execute an
|
|
already-succeeded stage and is reconsidered on later runs. Skipped results
|
|
cannot contain outputs.
|
|
|
|
When an already-succeeded stage is skipped, the invocation run manifest records
|
|
the `skip` action and reason. The session manifest deliberately retains its
|
|
existing succeeded record because it remains the cross-invocation progress
|
|
authority. Stages with a resume validator, currently extraction, may reject an
|
|
otherwise eligible skip when the recorded durable result is obsolete; the
|
|
runner marks it stale and executes it.
|
|
|
|
Session manifest is the authoritative stage-progress ledger across invocations.
|
|
Run manifest is invocation-scoped audit state.
|
|
|
|
Each invocation derives campaign, session, run, local-path, and remote-prefix
|
|
metadata from the validated resolved configuration as one projection. A persisted
|
|
session manifest must agree on campaign and session identity before execution;
|
|
the current projection is refreshed for every invocation while stage progress,
|
|
inputs, and durable artifacts remain session history.
|
|
|
|
For handled failures after an invocation record is created, the runner records
|
|
the failure on the session ledger and persists it before persisting the failed
|
|
run audit record. This preserves the resume authority while making a partial
|
|
persistence disagreement visible. Abrupt process death remains an accepted case
|
|
where a durable running record can require operator interpretation.
|
|
|
|
## Invariants
|
|
|
|
- stage resume/skip decisions are session-manifest driven.
|
|
- running, failed, and self-skipped stages do not retain result payloads from
|
|
an earlier success.
|
|
- stale stages retain prior details until replacement execution starts.
|
|
- force reruns stale downstream succeeded stages.
|
|
- run manifest does not replace session manifest as progress authority.
|
|
|
|
## Implementation And Tests
|
|
|
|
- Models and transitions: `internal/manifest/manifest.go`,
|
|
`internal/manifest/run_manifest.go`
|
|
- Persistence and validation: `internal/manifest/store.go`
|
|
- Package tests: `internal/manifest/*_test.go`
|
|
- Assembled execution behavior: `internal/app/runner_test.go`,
|
|
`internal/app/run_stage_test.go`
|