Add archive promotion locks
This commit is contained in:
@@ -154,6 +154,9 @@ archive:
|
||||
- source: narratio.artifact.session_recap
|
||||
dest: artifacts/session_recap.md
|
||||
required: true
|
||||
locks:
|
||||
- source: narratio.artifact.session_recap
|
||||
reason: Final recap was manually edited.
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
@@ -177,6 +180,7 @@ Operational notes:
|
||||
|
||||
- archive promotion is explicit and source-based via `archive.promote_artifacts`.
|
||||
- `source` is required; `dest` is optional and derived when omitted.
|
||||
- `archive.locks` skips top-level promotion overwrites for locked sources while preserving run-local uploads.
|
||||
- Narratio does not auto-promote all generated analyze artifacts.
|
||||
- `restore` reads the same config/campaign/session inputs and restore scope is bounded by committed archive current state.
|
||||
|
||||
@@ -205,6 +209,9 @@ Operational notes:
|
||||
| `pipeline.archive.promote_artifacts[].source` | string | Yes (per rule) | none |
|
||||
| `pipeline.archive.promote_artifacts[].dest` | string | No | derived from source |
|
||||
| `pipeline.archive.promote_artifacts[].required` | bool | No | `true` |
|
||||
| `pipeline.archive.locks[]` | list | No | empty |
|
||||
| `pipeline.archive.locks[].source` | string | Yes (per lock) | none |
|
||||
| `pipeline.archive.locks[].reason` | string | No | empty |
|
||||
| `pipeline.whisperx.transcribe_url` | string | Yes | none |
|
||||
| `pipeline.whisperx.language` | string | No | `en` |
|
||||
| `pipeline.whisperx.timeout` | duration string | No | `30m` |
|
||||
@@ -305,6 +312,8 @@ Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values:
|
||||
- `narratio.bounds.session`
|
||||
- `narratio.artifact.<configured_artifact_key>`
|
||||
|
||||
`pipeline.archive.locks[].source` accepts the same source values as `pipeline.archive.promote_artifacts[].source`.
|
||||
|
||||
Archive promotion destination rules:
|
||||
|
||||
- `dest` must be a clean relative path (not absolute, no traversal).
|
||||
@@ -314,6 +323,14 @@ Archive promotion destination rules:
|
||||
- configured sources derive from `pipeline.scriptorium.artifacts.<name>.output_path`;
|
||||
- derivation failure is a config validation error.
|
||||
|
||||
Archive lock rules:
|
||||
|
||||
- locks are source-based and do not accept `dest`.
|
||||
- duplicate lock sources are rejected.
|
||||
- locked promotions are recorded as intentional skips in archive metadata.
|
||||
- locked required promotions do not fail archive by default.
|
||||
- ordinary `--force` reruns do not override locks.
|
||||
|
||||
Restore-related implications:
|
||||
|
||||
- restore remote identity requires archive S3 identity to resolve (`pipeline.storage.s3.bucket` and session prefix derivation inputs).
|
||||
|
||||
@@ -8,6 +8,7 @@ Inputs:
|
||||
- session manifest and prerequisite stage records
|
||||
- run root contents under `runs/{run_id}/`
|
||||
- promotion rules with artifact `source` IDs and archive `dest` paths (`archive.promote_artifacts`)
|
||||
- source-based promotion locks (`archive.locks`)
|
||||
- session-level `previous/**` cache files when present
|
||||
|
||||
Outputs:
|
||||
@@ -23,6 +24,7 @@ Owns:
|
||||
- Prerequisite stage success enforcement
|
||||
- Run file collection and upload (excluding `audio/`)
|
||||
- Promotion rule resolution and upload
|
||||
- Promotion lock enforcement
|
||||
- Session previous-cache file collection/upload
|
||||
- Commit pointer publish order
|
||||
|
||||
@@ -34,6 +36,7 @@ Does not own:
|
||||
- `pipeline.archive.enabled`
|
||||
- `pipeline.archive.upload_run`
|
||||
- `pipeline.archive.promote_artifacts`
|
||||
- `pipeline.archive.locks`
|
||||
- `pipeline.storage.s3.bucket`
|
||||
- `pipeline.storage.s3.root_prefix`
|
||||
- `pipeline.workspace.root`
|
||||
@@ -47,9 +50,11 @@ Does not own:
|
||||
- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
|
||||
- Resolves bucket/prefix from manifest identity first, then config fallback.
|
||||
- Uploads session `previous/**` files as durable session state when the local `previous/` directory exists.
|
||||
- Skips top-level promotion uploads for sources listed in `pipeline.archive.locks`; run-local uploads still publish.
|
||||
- Writes metadata including:
|
||||
- upload counts/paths
|
||||
- `previous_files_uploaded` and `previous_uploaded_paths`
|
||||
- `locked_promotion_count` and `locked_promotions`
|
||||
- `current_manifest_key`
|
||||
- `current_run_id_key`
|
||||
- `current_pointer_written`
|
||||
@@ -60,7 +65,8 @@ Does not own:
|
||||
- Runner-level skip also applies for previously succeeded stage unless forced.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing prerequisite success, missing object store when required, missing run root, missing required promotion source, upload failures, or pointer write failures.
|
||||
- Fails on missing prerequisite success, missing object store when required, missing run root, missing unlocked required promotion source, upload failures, or pointer write failures.
|
||||
- Locked required promotions are intentional skips and do not fail archive.
|
||||
- Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
@@ -70,5 +76,6 @@ Does not own:
|
||||
## Architectural Invariants
|
||||
- Run upload excludes `audio/` subtree.
|
||||
- Session `previous/**` is archiveable durable input/provenance state, not run-local output.
|
||||
- Ordinary `--force` does not override archive locks.
|
||||
- `current/manifest.json` uploads before `current/run_id.txt`.
|
||||
- `current/run_id.txt` is the remote publish commit marker.
|
||||
|
||||
@@ -132,6 +132,8 @@ Archive promotion is explicit and source-based:
|
||||
- missing required promotion sources fail archive stage.
|
||||
- missing optional promotion sources are skipped.
|
||||
- invalid resolved artifacts fail archive stage.
|
||||
- `archive.locks` skips top-level promotion overwrites for locked sources while run-local uploads still publish.
|
||||
- locked required promotions are treated as intentional successful skips and are recorded in archive metadata.
|
||||
|
||||
## Resume, retry, restore, and safe rerun behavior
|
||||
|
||||
@@ -149,6 +151,7 @@ Restore conflict policy:
|
||||
|
||||
Forced reruns:
|
||||
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
|
||||
- ordinary `--force` does not override archive locks.
|
||||
|
||||
Safe rerun pattern:
|
||||
1. rerun the changed stage with `--force`.
|
||||
|
||||
@@ -169,13 +169,13 @@ When `prepare` materializes a remote session into `inputs/session.yml`, record t
|
||||
- local checksum;
|
||||
- downloaded temp/materialized path.
|
||||
|
||||
## Phase 3: Add Logical Archive Locks
|
||||
## Phase 3: Add Logical Archive Locks (implemented)
|
||||
|
||||
Add source-based archive locks under `pipeline.archive.locks`. The current promotion system is already source-based, so the first implementation must not support destination-based locks.
|
||||
Narratio supports source-based archive locks under `pipeline.archive.locks`. The promotion system is source-based; destination-based locks are not supported.
|
||||
|
||||
### Config Shape
|
||||
|
||||
Add lock entries:
|
||||
Lock entries:
|
||||
|
||||
archive:
|
||||
locks:
|
||||
@@ -194,7 +194,7 @@ Validation rules:
|
||||
|
||||
### Archive Behavior
|
||||
|
||||
Archive must continue uploading complete run-local outputs under `runs/{run_id}/`.
|
||||
Archive continues uploading complete run-local outputs under `runs/{run_id}/`.
|
||||
|
||||
Promotion behavior:
|
||||
|
||||
@@ -205,11 +205,11 @@ Promotion behavior:
|
||||
5. Continue archive commit when all run-local uploads and all non-locked required promotions succeed.
|
||||
6. Upload `current/manifest.json` and `current/run_id.txt` in the existing order, with `current/run_id.txt` last.
|
||||
|
||||
Ordinary `--force` must not override locks. Do not implement a lock-break override in this phase.
|
||||
Ordinary `--force` does not override locks. A lock-break override remains out of scope.
|
||||
|
||||
### Metadata
|
||||
|
||||
Record locked promotion skips in archive metadata/reporting so operators can distinguish missing optional promotions from lock-protected promotions.
|
||||
Archive records locked promotion skips in archive metadata/reporting so operators can distinguish missing optional promotions from lock-protected promotions.
|
||||
|
||||
Include:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user