223 lines
11 KiB
Markdown
223 lines
11 KiB
Markdown
# Internal: Manifest
|
|
|
|
## Purpose
|
|
|
|
Explain the session-progress and invocation-audit models implemented by
|
|
`internal/manifest`. Physical manifest placement belongs in
|
|
[Operations](../operations.md#local-state-layout).
|
|
|
|
## Session Manifest
|
|
|
|
`manifest.Manifest` records:
|
|
|
|
- identity (`session_id`, `campaign`, `run_id`)
|
|
- local path metadata (`local_workdir`, `local_spool_dir`)
|
|
- remote identity metadata (`s3_bucket`, `s3_session_prefix`, `s3_run_prefix`)
|
|
- `inputs` records
|
|
- durable `artifacts` records
|
|
- per-stage `stages` map
|
|
- an optional `post_publish_cleanup` obligation, which binds a committed run,
|
|
remote commit identity, and each exact root-confined local target to its
|
|
completion evidence
|
|
|
|
Session, campaign, and run identities in local and downloaded manifests must be
|
|
portable opaque segments. Unsafe legacy identities are rejected with migration
|
|
guidance rather than being normalized into a different workspace or remote
|
|
namespace.
|
|
|
|
The model admits these stage states:
|
|
|
|
- `pending`
|
|
- `running`
|
|
- `succeeded`
|
|
- `failed`
|
|
- `skipped`
|
|
- `stale`
|
|
- `interrupted`
|
|
|
|
### Analyze-owned artifact state
|
|
|
|
The `analyze` stage record may carry `analyze_state_version: 1` and an
|
|
`analyze_artifacts` map keyed by normalized configured artifact key. The
|
|
version is the authority marker: version 1 with no entries is a valid evaluated
|
|
empty set, while an absent version is legacy aggregate-only state and provides
|
|
no current configured-artifact evidence.
|
|
|
|
Each analyze artifact record has one disposition:
|
|
|
|
- `current`: the configured artifact is available and carries a versioned
|
|
fingerprint plus a complete output record and separate output size;
|
|
- `stale`: the recorded semantic identity is no longer current;
|
|
- `missing`: no validated current result exists;
|
|
- `failed`: the attempted work failed and carries a bounded diagnostic; or
|
|
- `unselected`: the artifact was intentionally outside the evaluated set.
|
|
|
|
Records bind their normalized key and dependencies, fingerprint contract when
|
|
evaluated, canonical session-relative output identity when current, producing
|
|
Narratio run, update time, and bounded non-secret Scriptorium provenance and
|
|
diagnostic paths. A current output includes its configured source ID, contract,
|
|
checksum, and positive byte size. Non-current records cannot carry an output,
|
|
so an older file is not advertised through stale, missing, failed, or
|
|
unselected state.
|
|
|
|
The session-stage collection is the reconciled authority across invocations.
|
|
The corresponding collection on an invocation's `analyze` stage record is an
|
|
audit of only the artifacts evaluated or attempted by that run. These records
|
|
remain analyze-owned data inside the fixed stage; they are not dynamic stages
|
|
or generic subtasks.
|
|
|
|
The stage result contract has one analyze-specific projection boundary. On
|
|
success, the runner validates and deep-copies the complete reconciled session
|
|
collection and the invocation subset. Aggregate session outputs are rebuilt in
|
|
configured-key order from current session records only; invocation outputs are
|
|
limited to current records produced by that invocation's run ID. Ordinary
|
|
stage outputs cannot accompany this projection, so there is one source of
|
|
artifact authority.
|
|
|
|
Analyze may return a projection together with an error. That restricted result
|
|
cannot carry ordinary outputs, skip state, aggregate logs, generated configs,
|
|
or metadata. The runner persists only the validated per-artifact collections,
|
|
then marks the aggregate analyze and run state failed and invalidates delivery
|
|
dependents conservatively. Unrelated current records survive because the
|
|
session projection is complete. A malformed projection is not applied, and a
|
|
failed session projection save restores the prior per-artifact authority before
|
|
terminal failure persistence.
|
|
|
|
## Run Manifest
|
|
|
|
`manifest.RunManifest` is created for each invocation and records:
|
|
|
|
- invocation identity and `force` flag
|
|
- requested stages
|
|
- per-stage action (`run` or `skip`)
|
|
- per-stage status
|
|
- overall run status (`running`, `succeeded`, `failed`)
|
|
|
|
## Remote Commit Manifest
|
|
|
|
`artifacts.RemoteCommitManifest` is a separate, versioned remote snapshot
|
|
contract. It is not a serialized session manifest and contains no local
|
|
post-publication assertion such as `current_pointer_written`. A remote commit
|
|
identifies one campaign, session, and run and declares its immutable artifact
|
|
set. Each artifact has a typed source, immutable destination key, SHA-256
|
|
checksum, size, and storage generation.
|
|
|
|
`current/commit-pointer.json` is the sole mutable selector for the new
|
|
contract. It identifies exactly one run-scoped `runs/{run_id}/commit.json` and
|
|
binds that object by checksum, size, and generation. Readers strictly reject
|
|
unknown fields, version mismatches, pointer/commit identity mismatches, and
|
|
objects that do not match their declaration.
|
|
|
|
The reader retains a temporary, clearly isolated compatibility path for a
|
|
coherent legacy `current/manifest.json` plus `current/run_id.txt` pair. That
|
|
path is removable after migration and is never used to write new state.
|
|
|
|
## Persistence Semantics
|
|
|
|
`manifest.LocalStore`:
|
|
|
|
- validates loaded documents;
|
|
- normalizes missing maps/stage records;
|
|
- writes through a sibling temporary file, syncing the completed file and
|
|
destination directory after atomic replacement;
|
|
- updates `updated_at` on save.
|
|
|
|
If the operating system or filesystem cannot sync a directory, save returns an
|
|
explicit error instead of claiming crash-durable replacement. A returned error
|
|
after the rename can therefore leave the new manifest visible but not confirmed
|
|
durable; callers must reload it before retrying.
|
|
|
|
## Execution Semantics
|
|
|
|
The application runner marks an executing stage running and then succeeded or
|
|
failed in both manifests, persisting each transition. On success it records
|
|
outputs, logs, generated configuration references, and metadata. Artifact
|
|
records may include optional contract and external provenance objects; old
|
|
manifests remain compatible when those fields are absent. A successful forced
|
|
rerun marks only succeeded transitive dependent session-stage records stale.
|
|
The application owns a fixed dependency relation distinct from execution order;
|
|
dependents are returned in canonical order. Render and extract therefore never
|
|
stale one another, while either can stale analyze, publish, and notify.
|
|
|
|
Starting an execution clears the current session-stage record's prior outputs,
|
|
logs, generated configuration references, and metadata. Failed and skipped
|
|
transitions enforce the same clearing rule directly, while success repopulates
|
|
only fields returned by the new result. Marking a record stale does not clear
|
|
those details because resume validation and diagnosis may still require them
|
|
before execution begins. Invocation run manifests remain immutable audit
|
|
records of their own outcomes.
|
|
|
|
Aggregate lifecycle clearing deliberately preserves the analyze-owned
|
|
per-artifact collection. This lets later reconciliation replace only evaluated
|
|
entries without erasing unrelated current results. Other stages retain their
|
|
existing aggregate-only lifecycle behavior and are forbidden from carrying the
|
|
analyze-specific fields.
|
|
|
|
A stage may explicitly return a skipped disposition and stable reason. The
|
|
runner persists that outcome in both manifests, clears older outputs for the
|
|
session-stage record along with older logs, generated configuration references,
|
|
and metadata, then applies any bounded details from the current skip and
|
|
continues. This self-skip is distinct from deciding not to execute an
|
|
already-succeeded stage and is reconsidered on later runs. Skipped results
|
|
cannot contain outputs.
|
|
|
|
When an already-succeeded stage is skipped, the invocation run manifest records
|
|
the `skip` action and reason. The session manifest deliberately retains its
|
|
existing succeeded record because it remains the cross-invocation progress
|
|
authority. Stages with a resume validator, currently extraction, may reject an
|
|
otherwise eligible skip when the recorded durable result is obsolete; the
|
|
runner marks it stale and executes it.
|
|
|
|
Session manifest is the authoritative stage-progress ledger across invocations.
|
|
Run manifest is invocation-scoped audit state.
|
|
|
|
Before an explicitly bounded execution starts after `prepare`, the application
|
|
reads the session manifest and accepts only `succeeded` or `skipped` for every
|
|
excluded canonical prefix stage. The first other status or absent record fails
|
|
the request before layout mutation, adapter initialization, session-manifest
|
|
writes, or run-manifest creation. Excluded prefix records are not passed to
|
|
resume validators. Records after the selected end are not prerequisites and
|
|
may be made stale by selected work without being scheduled.
|
|
|
|
After a publish commits remotely, any configured local cleanup is first recorded
|
|
as a session-manifest obligation before deletion begins. Each target becomes
|
|
complete only after its confined deletion (or safe absence check) and a
|
|
successful manifest save. An incomplete obligation is retried when publish
|
|
executes again and retains the committed run and remote identity that authorized
|
|
it; an invocation that does not execute publish does not perform cleanup.
|
|
|
|
Each invocation derives campaign, session, run, local-path, and remote-prefix
|
|
metadata from the validated resolved configuration as one projection. A persisted
|
|
session manifest must agree on campaign and session identity before execution;
|
|
the current projection is refreshed for every invocation while stage progress,
|
|
inputs, and durable artifacts remain session history.
|
|
|
|
For handled failures after an invocation record is created, the runner records
|
|
the failure on the session ledger and persists it before persisting the failed
|
|
run audit record. This preserves the resume authority while making a partial
|
|
persistence disagreement visible. Abrupt process death remains an accepted case
|
|
where a durable running record can require operator interpretation.
|
|
|
|
## Invariants
|
|
|
|
- stage resume/skip decisions are session-manifest driven.
|
|
- running, failed, and self-skipped stages do not retain result payloads from
|
|
an earlier success.
|
|
- stale stages retain prior details until replacement execution starts.
|
|
- force reruns stale succeeded stages in the fixed dependency relation.
|
|
- run manifest does not replace session manifest as progress authority.
|
|
- remote commitment is established by a verified current pointer and remote
|
|
commit relationship, never by a mutable session-manifest boolean.
|
|
|
|
## Implementation And Tests
|
|
|
|
- Models and transitions: `internal/manifest/manifest.go`,
|
|
`internal/manifest/run_manifest.go`
|
|
- Remote commit model and readers: `internal/artifacts/remote_commit.go`,
|
|
`internal/artifacts/current_state_commit.go`,
|
|
`internal/artifacts/current_state_legacy.go`
|
|
- Persistence and validation: `internal/manifest/store.go`
|
|
- Package tests: `internal/manifest/*_test.go`
|
|
- Assembled execution behavior: `internal/app/runner_test.go`,
|
|
`internal/app/run_stage_test.go`
|