Rewrite internal documentation for current stage and state contracts

This commit is contained in:
2026-05-23 12:57:59 +00:00
parent d723384888
commit 0299b128cf
15 changed files with 450 additions and 993 deletions

View File

@@ -1,88 +1,44 @@
# Stage: publish
## Purpose
Publish durable run/session state to object storage, then atomically advance remote current state.
Upload run/session outputs to object storage and atomically advance remote current state.
## Inputs and Outputs
Inputs:
- session manifest and prerequisite stage records
- run root contents under `runs/{run_id}/`
- publish output rules with artifact `source` IDs and publish `dest` paths (`pipeline.publish.outputs`)
- effective source-based publish locks from static config and remote session lock store
- session-level `previous/**` cache files when present
## Inputs
- successful prerequisite stages: `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`
- run root `runs/{run_id}/**`
- publish output rules (`pipeline.publish.outputs`)
- effective publish locks (static + remote merged lock set)
- local `previous/**` files when present
Outputs:
- uploaded run files under `{session_prefix}/runs/{run_id}/...`
- uploaded published outputs under `{session_prefix}/...`
- uploaded session previous-cache files under `{session_prefix}/previous/...` when present
- `{session_prefix}/current/manifest.json`
- `{session_prefix}/current/run_id.txt` written last
## Outputs
- uploaded run files under remote `runs/{run_id}/...` (excluding `audio/**`)
- uploaded selected publish outputs under session prefix
- uploaded `previous/**` files under session prefix when present
- uploaded `current/manifest.json`
- uploaded `current/run_id.txt` written last
## Boundaries
Owns:
- publish enable/disable gate behavior
- prerequisite stage success enforcement
- run file collection and upload (excluding `audio/`)
- publish output rule resolution and upload
- publish lock enforcement
- session previous-cache file collection/upload
- commit pointer publish order
## Key Behavior
- stage can self-skip when publish disabled or run upload disabled.
- validates prerequisite stage success and object-store availability.
- collects deterministic run file list plus run `manifest.json`.
- resolves publish output sources through runtime artifact catalog and manifest-aware resolution.
- selected artifact filter applies to configured artifact sources only.
- locked outputs are skipped intentionally (including required ones).
- optional missing outputs are skipped; required missing unlocked outputs fail.
- writes remote current manifest before current run pointer.
Does not own:
- stage execution before publish
- post-publish local cleanup policy execution (handled by app cleanup logic)
## Metadata Signals
Includes counts/lists for:
- run uploads
- published output uploads
- previous uploads
- skipped optional outputs
- skipped unselected outputs
- locked outputs
- current-state key paths
- `current_pointer_written`
## Config Fields Used
- `pipeline.publish.enabled`
- `pipeline.publish.upload_run`
- `pipeline.publish.outputs`
- `pipeline.publish.locks`
- `{session_prefix}/locks.yml` loaded by app orchestration before publish execution
- `pipeline.storage.s3.bucket`
- `pipeline.storage.s3.root_prefix`
- `pipeline.workspace.root`
- `session.campaign`
- `session.session_id`
## External Adapters Used
- Object storage backend (`env.ObjectStore`) for upload/list primitives.
## State and Manifest Behavior
- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
- Resolves bucket/prefix from manifest identity first, then config fallback.
- Uploads session `previous/**` files as durable session state when the local `previous/` directory exists.
- Skips top-level published output uploads for effective locked sources; run-local materialized outputs remain unchanged.
- When selected configured artifact keys are supplied, skips publish rules for unselected `narratio.artifact.<key>` sources; built-in transcript and bounds outputs still publish.
- Effective locks are the union of `pipeline.publish.locks` and remote `{session_prefix}/locks.yml`; static pipeline locks win on duplicate sources.
- Writes metadata including:
- upload counts/paths
- `previous_files_uploaded` and `previous_uploaded_paths`
- `published_files_uploaded` and `published_paths`
- `skipped_optional_outputs`
- `skipped_unselected_outputs`
- `locked_output_count` and `locked_outputs`
- `current_manifest_key`
- `current_run_id_key`
- `current_pointer_written`
- On skipped publish path, returns metadata with `skipped=true` and pointer not written.
## Skip and Resume Behavior
- Stage may self-skip (metadata skip) when publish disabled or run upload disabled.
- Runner-level skip also applies for previously succeeded stage unless forced.
## Failure Behavior
- Fails on missing prerequisite success, missing object store when required, missing run root, missing unlocked required output source, upload failures, or pointer write failures.
- Locked required outputs are intentional skips and do not fail publish.
- Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail.
## Tests to Inspect Before Changing
- `internal/stage/archive_test.go`
- `internal/app/post_archive_cleanup_test.go`
## Architectural Invariants
- Run upload excludes `audio/` subtree.
- Session `previous/**` is publishable durable input/provenance state, not run-local output.
- Ordinary `--force` does not override publish locks.
- Malformed or unreadable remote lock store fails publish-capable execution before output uploads.
- `current/manifest.json` uploads before `current/run_id.txt`.
- `current/run_id.txt` is the remote publish commit marker.
## Invariants
- `current/run_id.txt` is the remote commit marker and is written last.
- run upload excludes `audio/**`.
- publish locks are not overridden by `--force`.