Add archive promotion locks

This commit is contained in:
2026-05-20 21:40:09 -05:00
parent 3aae4bbb12
commit 7111edeca4
9 changed files with 362 additions and 13 deletions

View File

@@ -154,6 +154,9 @@ archive:
- source: narratio.artifact.session_recap
dest: artifacts/session_recap.md
required: true
locks:
- source: narratio.artifact.session_recap
reason: Final recap was manually edited.
whisperx:
transcribe_url: "https://transcription.example.com/transcribe"
@@ -177,6 +180,7 @@ Operational notes:
- archive promotion is explicit and source-based via `archive.promote_artifacts`.
- `source` is required; `dest` is optional and derived when omitted.
- `archive.locks` skips top-level promotion overwrites for locked sources while preserving run-local uploads.
- Narratio does not auto-promote all generated analyze artifacts.
- `restore` reads the same config/campaign/session inputs and restore scope is bounded by committed archive current state.
@@ -205,6 +209,9 @@ Operational notes:
| `pipeline.archive.promote_artifacts[].source` | string | Yes (per rule) | none |
| `pipeline.archive.promote_artifacts[].dest` | string | No | derived from source |
| `pipeline.archive.promote_artifacts[].required` | bool | No | `true` |
| `pipeline.archive.locks[]` | list | No | empty |
| `pipeline.archive.locks[].source` | string | Yes (per lock) | none |
| `pipeline.archive.locks[].reason` | string | No | empty |
| `pipeline.whisperx.transcribe_url` | string | Yes | none |
| `pipeline.whisperx.language` | string | No | `en` |
| `pipeline.whisperx.timeout` | duration string | No | `30m` |
@@ -305,6 +312,8 @@ Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values:
- `narratio.bounds.session`
- `narratio.artifact.<configured_artifact_key>`
`pipeline.archive.locks[].source` accepts the same source values as `pipeline.archive.promote_artifacts[].source`.
Archive promotion destination rules:
- `dest` must be a clean relative path (not absolute, no traversal).
@@ -314,6 +323,14 @@ Archive promotion destination rules:
- configured sources derive from `pipeline.scriptorium.artifacts.<name>.output_path`;
- derivation failure is a config validation error.
Archive lock rules:
- locks are source-based and do not accept `dest`.
- duplicate lock sources are rejected.
- locked promotions are recorded as intentional skips in archive metadata.
- locked required promotions do not fail archive by default.
- ordinary `--force` reruns do not override locks.
Restore-related implications:
- restore remote identity requires archive S3 identity to resolve (`pipeline.storage.s3.bucket` and session prefix derivation inputs).

View File

@@ -8,6 +8,7 @@ Inputs:
- session manifest and prerequisite stage records
- run root contents under `runs/{run_id}/`
- promotion rules with artifact `source` IDs and archive `dest` paths (`archive.promote_artifacts`)
- source-based promotion locks (`archive.locks`)
- session-level `previous/**` cache files when present
Outputs:
@@ -23,6 +24,7 @@ Owns:
- Prerequisite stage success enforcement
- Run file collection and upload (excluding `audio/`)
- Promotion rule resolution and upload
- Promotion lock enforcement
- Session previous-cache file collection/upload
- Commit pointer publish order
@@ -34,6 +36,7 @@ Does not own:
- `pipeline.archive.enabled`
- `pipeline.archive.upload_run`
- `pipeline.archive.promote_artifacts`
- `pipeline.archive.locks`
- `pipeline.storage.s3.bucket`
- `pipeline.storage.s3.root_prefix`
- `pipeline.workspace.root`
@@ -47,9 +50,11 @@ Does not own:
- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
- Resolves bucket/prefix from manifest identity first, then config fallback.
- Uploads session `previous/**` files as durable session state when the local `previous/` directory exists.
- Skips top-level promotion uploads for sources listed in `pipeline.archive.locks`; run-local uploads still publish.
- Writes metadata including:
- upload counts/paths
- `previous_files_uploaded` and `previous_uploaded_paths`
- `locked_promotion_count` and `locked_promotions`
- `current_manifest_key`
- `current_run_id_key`
- `current_pointer_written`
@@ -60,7 +65,8 @@ Does not own:
- Runner-level skip also applies for previously succeeded stage unless forced.
## Failure Behavior
- Fails on missing prerequisite success, missing object store when required, missing run root, missing required promotion source, upload failures, or pointer write failures.
- Fails on missing prerequisite success, missing object store when required, missing run root, missing unlocked required promotion source, upload failures, or pointer write failures.
- Locked required promotions are intentional skips and do not fail archive.
- Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail.
## Tests to Inspect Before Changing
@@ -70,5 +76,6 @@ Does not own:
## Architectural Invariants
- Run upload excludes `audio/` subtree.
- Session `previous/**` is archiveable durable input/provenance state, not run-local output.
- Ordinary `--force` does not override archive locks.
- `current/manifest.json` uploads before `current/run_id.txt`.
- `current/run_id.txt` is the remote publish commit marker.

View File

@@ -132,6 +132,8 @@ Archive promotion is explicit and source-based:
- missing required promotion sources fail archive stage.
- missing optional promotion sources are skipped.
- invalid resolved artifacts fail archive stage.
- `archive.locks` skips top-level promotion overwrites for locked sources while run-local uploads still publish.
- locked required promotions are treated as intentional successful skips and are recorded in archive metadata.
## Resume, retry, restore, and safe rerun behavior
@@ -149,6 +151,7 @@ Restore conflict policy:
Forced reruns:
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
- ordinary `--force` does not override archive locks.
Safe rerun pattern:
1. rerun the changed stage with `--force`.

View File

@@ -169,13 +169,13 @@ When `prepare` materializes a remote session into `inputs/session.yml`, record t
- local checksum;
- downloaded temp/materialized path.
## Phase 3: Add Logical Archive Locks
## Phase 3: Add Logical Archive Locks (implemented)
Add source-based archive locks under `pipeline.archive.locks`. The current promotion system is already source-based, so the first implementation must not support destination-based locks.
Narratio supports source-based archive locks under `pipeline.archive.locks`. The promotion system is source-based; destination-based locks are not supported.
### Config Shape
Add lock entries:
Lock entries:
archive:
locks:
@@ -194,7 +194,7 @@ Validation rules:
### Archive Behavior
Archive must continue uploading complete run-local outputs under `runs/{run_id}/`.
Archive continues uploading complete run-local outputs under `runs/{run_id}/`.
Promotion behavior:
@@ -205,11 +205,11 @@ Promotion behavior:
5. Continue archive commit when all run-local uploads and all non-locked required promotions succeed.
6. Upload `current/manifest.json` and `current/run_id.txt` in the existing order, with `current/run_id.txt` last.
Ordinary `--force` must not override locks. Do not implement a lock-break override in this phase.
Ordinary `--force` does not override locks. A lock-break override remains out of scope.
### Metadata
Record locked promotion skips in archive metadata/reporting so operators can distinguish missing optional promotions from lock-protected promotions.
Archive records locked promotion skips in archive metadata/reporting so operators can distinguish missing optional promotions from lock-protected promotions.
Include: