Rewrite internal documentation for current stage and state contracts
This commit is contained in:
@@ -1,88 +1,44 @@
|
||||
# Stage: publish
|
||||
|
||||
## Purpose
|
||||
Publish durable run/session state to object storage, then atomically advance remote current state.
|
||||
Upload run/session outputs to object storage and atomically advance remote current state.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- session manifest and prerequisite stage records
|
||||
- run root contents under `runs/{run_id}/`
|
||||
- publish output rules with artifact `source` IDs and publish `dest` paths (`pipeline.publish.outputs`)
|
||||
- effective source-based publish locks from static config and remote session lock store
|
||||
- session-level `previous/**` cache files when present
|
||||
## Inputs
|
||||
- successful prerequisite stages: `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`
|
||||
- run root `runs/{run_id}/**`
|
||||
- publish output rules (`pipeline.publish.outputs`)
|
||||
- effective publish locks (static + remote merged lock set)
|
||||
- local `previous/**` files when present
|
||||
|
||||
Outputs:
|
||||
- uploaded run files under `{session_prefix}/runs/{run_id}/...`
|
||||
- uploaded published outputs under `{session_prefix}/...`
|
||||
- uploaded session previous-cache files under `{session_prefix}/previous/...` when present
|
||||
- `{session_prefix}/current/manifest.json`
|
||||
- `{session_prefix}/current/run_id.txt` written last
|
||||
## Outputs
|
||||
- uploaded run files under remote `runs/{run_id}/...` (excluding `audio/**`)
|
||||
- uploaded selected publish outputs under session prefix
|
||||
- uploaded `previous/**` files under session prefix when present
|
||||
- uploaded `current/manifest.json`
|
||||
- uploaded `current/run_id.txt` written last
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- publish enable/disable gate behavior
|
||||
- prerequisite stage success enforcement
|
||||
- run file collection and upload (excluding `audio/`)
|
||||
- publish output rule resolution and upload
|
||||
- publish lock enforcement
|
||||
- session previous-cache file collection/upload
|
||||
- commit pointer publish order
|
||||
## Key Behavior
|
||||
- stage can self-skip when publish disabled or run upload disabled.
|
||||
- validates prerequisite stage success and object-store availability.
|
||||
- collects deterministic run file list plus run `manifest.json`.
|
||||
- resolves publish output sources through runtime artifact catalog and manifest-aware resolution.
|
||||
- selected artifact filter applies to configured artifact sources only.
|
||||
- locked outputs are skipped intentionally (including required ones).
|
||||
- optional missing outputs are skipped; required missing unlocked outputs fail.
|
||||
- writes remote current manifest before current run pointer.
|
||||
|
||||
Does not own:
|
||||
- stage execution before publish
|
||||
- post-publish local cleanup policy execution (handled by app cleanup logic)
|
||||
## Metadata Signals
|
||||
Includes counts/lists for:
|
||||
- run uploads
|
||||
- published output uploads
|
||||
- previous uploads
|
||||
- skipped optional outputs
|
||||
- skipped unselected outputs
|
||||
- locked outputs
|
||||
- current-state key paths
|
||||
- `current_pointer_written`
|
||||
|
||||
## Config Fields Used
|
||||
- `pipeline.publish.enabled`
|
||||
- `pipeline.publish.upload_run`
|
||||
- `pipeline.publish.outputs`
|
||||
- `pipeline.publish.locks`
|
||||
- `{session_prefix}/locks.yml` loaded by app orchestration before publish execution
|
||||
- `pipeline.storage.s3.bucket`
|
||||
- `pipeline.storage.s3.root_prefix`
|
||||
- `pipeline.workspace.root`
|
||||
- `session.campaign`
|
||||
- `session.session_id`
|
||||
|
||||
## External Adapters Used
|
||||
- Object storage backend (`env.ObjectStore`) for upload/list primitives.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
|
||||
- Resolves bucket/prefix from manifest identity first, then config fallback.
|
||||
- Uploads session `previous/**` files as durable session state when the local `previous/` directory exists.
|
||||
- Skips top-level published output uploads for effective locked sources; run-local materialized outputs remain unchanged.
|
||||
- When selected configured artifact keys are supplied, skips publish rules for unselected `narratio.artifact.<key>` sources; built-in transcript and bounds outputs still publish.
|
||||
- Effective locks are the union of `pipeline.publish.locks` and remote `{session_prefix}/locks.yml`; static pipeline locks win on duplicate sources.
|
||||
- Writes metadata including:
|
||||
- upload counts/paths
|
||||
- `previous_files_uploaded` and `previous_uploaded_paths`
|
||||
- `published_files_uploaded` and `published_paths`
|
||||
- `skipped_optional_outputs`
|
||||
- `skipped_unselected_outputs`
|
||||
- `locked_output_count` and `locked_outputs`
|
||||
- `current_manifest_key`
|
||||
- `current_run_id_key`
|
||||
- `current_pointer_written`
|
||||
- On skipped publish path, returns metadata with `skipped=true` and pointer not written.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Stage may self-skip (metadata skip) when publish disabled or run upload disabled.
|
||||
- Runner-level skip also applies for previously succeeded stage unless forced.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing prerequisite success, missing object store when required, missing run root, missing unlocked required output source, upload failures, or pointer write failures.
|
||||
- Locked required outputs are intentional skips and do not fail publish.
|
||||
- Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/archive_test.go`
|
||||
- `internal/app/post_archive_cleanup_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Run upload excludes `audio/` subtree.
|
||||
- Session `previous/**` is publishable durable input/provenance state, not run-local output.
|
||||
- Ordinary `--force` does not override publish locks.
|
||||
- Malformed or unreadable remote lock store fails publish-capable execution before output uploads.
|
||||
- `current/manifest.json` uploads before `current/run_id.txt`.
|
||||
- `current/run_id.txt` is the remote publish commit marker.
|
||||
## Invariants
|
||||
- `current/run_id.txt` is the remote commit marker and is written last.
|
||||
- run upload excludes `audio/**`.
|
||||
- publish locks are not overridden by `--force`.
|
||||
|
||||
Reference in New Issue
Block a user