Compare commits
17 Commits
ea87c335d6
...
v1.0.0
| Author | SHA1 | Date | |
|---|---|---|---|
| 4491fb5ccd | |||
| 30b905765c | |||
| 03eac70881 | |||
| 0f7e6b979f | |||
| c366912586 | |||
| 9fe44cd00d | |||
| 094b0d2532 | |||
| 98649f4d81 | |||
| 8a559efd5b | |||
| 72deccb4e2 | |||
| 5620fc5bcf | |||
| be57e675e0 | |||
| 3971443831 | |||
| a6b0c33e9f | |||
| 96b886e711 | |||
| 7d584ee6cd | |||
| 572a112c31 |
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
Narratio is a stage-driven Go orchestrator for turning D&D session audio into polished transcripts and generated artifacts.
|
Narratio is a stage-driven Go orchestrator for turning D&D session audio into polished transcripts and generated artifacts.
|
||||||
|
|
||||||
It runs a deterministic workflow across `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`, and `publish`, with manifest-driven resume and restore support.
|
It runs a deterministic workflow across `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`, and `publish`, with manifest-driven continuation and restore support.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio run 2026-04-04
|
narratio run 2026-04-04
|
||||||
|
|||||||
36
docs/cli.md
36
docs/cli.md
@@ -13,7 +13,6 @@ This runs the canonical full pipeline for session `2026-04-04`.
|
|||||||
Top-level commands:
|
Top-level commands:
|
||||||
|
|
||||||
- `run <session_id>`: run full stage order.
|
- `run <session_id>`: run full stage order.
|
||||||
- `resume <session_id>`: continue from first non-succeeded stage.
|
|
||||||
- `run-stage <stage> <session_id>`: run one stage.
|
- `run-stage <stage> <session_id>`: run one stage.
|
||||||
- `analyze <session_id>`: force-run analyze.
|
- `analyze <session_id>`: force-run analyze.
|
||||||
- `publish <session_id>`: force-run publish.
|
- `publish <session_id>`: force-run publish.
|
||||||
@@ -40,14 +39,31 @@ Most session-aware commands accept:
|
|||||||
- `--campaign <id>`
|
- `--campaign <id>`
|
||||||
- `--campaign-file <campaign.yml>`
|
- `--campaign-file <campaign.yml>`
|
||||||
- `--session <session.yml>`
|
- `--session <session.yml>`
|
||||||
- `--previous-session-id <id>`
|
- `--session-id <session_id>`
|
||||||
|
- `--previous-session-id <session_id>`
|
||||||
|
|
||||||
Rules:
|
Rules:
|
||||||
|
|
||||||
- `--campaign` and `--campaign-file` are mutually exclusive.
|
- `--campaign` and `--campaign-file` are mutually exclusive.
|
||||||
- `--session` is not used by `session init`.
|
- `--session` is not used by `session init`.
|
||||||
|
- if both positional `<session_id>` and `--session-id` are provided, values must match.
|
||||||
- `clean --all` cannot be combined with campaign/session selectors.
|
- `clean --all` cannot be combined with campaign/session selectors.
|
||||||
|
|
||||||
|
## Session ID Input Rules
|
||||||
|
|
||||||
|
Session-aware commands accept one of these forms:
|
||||||
|
|
||||||
|
- positional session ID: `... <session_id>`
|
||||||
|
- compatibility flag: `... --session-id <session_id>`
|
||||||
|
|
||||||
|
When both are present, command parsing requires an exact match.
|
||||||
|
|
||||||
|
Commands with additional positionals keep their command-specific order:
|
||||||
|
|
||||||
|
- `run-stage <stage> <session_id>` or `run-stage <stage> --session-id <session_id>`
|
||||||
|
- `session locks add <session_id> <source>` or `session locks add --session-id <session_id> <source>`
|
||||||
|
- `session locks remove <session_id> <source>` or `session locks remove --session-id <session_id> <source>`
|
||||||
|
|
||||||
## Command Reference
|
## Command Reference
|
||||||
|
|
||||||
### `run`
|
### `run`
|
||||||
@@ -60,19 +76,9 @@ Behavior:
|
|||||||
|
|
||||||
- evaluates full stage order;
|
- evaluates full stage order;
|
||||||
- skips already-succeeded stages unless `--force` is set;
|
- skips already-succeeded stages unless `--force` is set;
|
||||||
|
- continues interrupted or partially completed sessions by running non-succeeded stages;
|
||||||
- writes session and run manifests.
|
- writes session and run manifests.
|
||||||
|
|
||||||
### `resume`
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio resume <session_id> [--force] [--artifacts <name[,name...]>] [...common config flags]
|
|
||||||
```
|
|
||||||
|
|
||||||
Behavior:
|
|
||||||
|
|
||||||
- when not forced, starts at first non-succeeded stage in manifest order;
|
|
||||||
- with `--force`, reevaluates the selected stage list as runnable.
|
|
||||||
|
|
||||||
### `run-stage`
|
### `run-stage`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -187,7 +193,7 @@ Options:
|
|||||||
Rules:
|
Rules:
|
||||||
|
|
||||||
- `--audio-dir` and `--audio-s3-prefix` are mutually exclusive.
|
- `--audio-dir` and `--audio-s3-prefix` are mutually exclusive.
|
||||||
- if campaign `session_template_file` is configured, `session init` renders it;
|
- if campaign `session_template_file` is configured, `session init` renders it.
|
||||||
- generated session YAML must be concrete (no unresolved `{{ ... }}` placeholders).
|
- generated session YAML must be concrete (no unresolved `{{ ... }}` placeholders).
|
||||||
|
|
||||||
### `session restore`
|
### `session restore`
|
||||||
@@ -236,7 +242,7 @@ Behavior:
|
|||||||
|
|
||||||
## `--artifacts` Selection Rules
|
## `--artifacts` Selection Rules
|
||||||
|
|
||||||
- accepted on `run`, `resume`, `run-stage`, `analyze`, and `publish`;
|
- accepted on `run`, `run-stage`, `analyze`, and `publish`;
|
||||||
- names must exist in `pipeline.scriptorium.artifacts`;
|
- names must exist in `pipeline.scriptorium.artifacts`;
|
||||||
- empty entries are invalid;
|
- empty entries are invalid;
|
||||||
- repeated names are deduplicated.
|
- repeated names are deduplicated.
|
||||||
|
|||||||
@@ -14,9 +14,6 @@ Primary adapters:
|
|||||||
- `storage.ObjectStore`
|
- `storage.ObjectStore`
|
||||||
- `notify.Sender`
|
- `notify.Sender`
|
||||||
|
|
||||||
Legacy compatibility boundary:
|
|
||||||
- `storage.Backend` remains in the storage adapter package and defaults to `NoopBackend`; current pipeline stages use `storage.ObjectStore`.
|
|
||||||
|
|
||||||
## Ownership
|
## Ownership
|
||||||
Adapters own:
|
Adapters own:
|
||||||
- HTTP/subprocess/SDK argument and transport details.
|
- HTTP/subprocess/SDK argument and transport details.
|
||||||
|
|||||||
@@ -1,9 +1,10 @@
|
|||||||
# Internal: Artifacts
|
# Internal: Artifacts
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define canonical artifact IDs, runtime catalog behavior, and source resolution rules for stage execution and publish output selection.
|
Define canonical artifact IDs, runtime catalog behavior, source resolution rules, and shared current-state mechanics used by app and previous-cache code.
|
||||||
|
|
||||||
## Built-in Source IDs
|
## Built-in Source IDs
|
||||||
|
|
||||||
- `narratio.transcript.base` -> `transcripts/base.json` (`merge`)
|
- `narratio.transcript.base` -> `transcripts/base.json` (`merge`)
|
||||||
- `narratio.transcript.polished` -> `transcripts/polished.json` (`polish`)
|
- `narratio.transcript.polished` -> `transcripts/polished.json` (`polish`)
|
||||||
- `narratio.transcript.final` -> `transcripts/final.json` (`normalize`)
|
- `narratio.transcript.final` -> `transcripts/final.json` (`normalize`)
|
||||||
@@ -11,58 +12,101 @@ Define canonical artifact IDs, runtime catalog behavior, and source resolution r
|
|||||||
- `narratio.bounds.session` -> `artifacts/session_bounds.json` (`trim`)
|
- `narratio.bounds.session` -> `artifacts/session_bounds.json` (`trim`)
|
||||||
|
|
||||||
## Configured and Previous-Session Sources
|
## Configured and Previous-Session Sources
|
||||||
- Configured artifact source ID: `narratio.artifact.<artifact_key>`
|
|
||||||
- Previous-session source ID: `narratio.previous_session.artifact.<artifact_key>`
|
|
||||||
|
|
||||||
Configured and previous-session source IDs are validated by strict regex rules.
|
- configured source ID format: `narratio.artifact.<artifact_key>`
|
||||||
|
- previous-session source ID format: `narratio.previous_session.artifact.<artifact_key>`
|
||||||
|
|
||||||
|
Both formats are validated by strict source-policy rules.
|
||||||
|
|
||||||
## Runtime Catalog
|
## Runtime Catalog
|
||||||
|
|
||||||
`ArtifactCatalog` tracks:
|
`ArtifactCatalog` tracks:
|
||||||
- `planned`: source registered for run context.
|
|
||||||
- `executable`: selected and enabled for analyze execution.
|
- `planned`: source registered for run context;
|
||||||
- `available`: local file exists and validated.
|
- `executable`: selected and enabled for analyze execution;
|
||||||
|
- `available`: local file exists and validates;
|
||||||
- `provenance`: availability source.
|
- `provenance`: availability source.
|
||||||
|
|
||||||
Current provenance values:
|
Current provenance values:
|
||||||
|
|
||||||
- `generated.current_analyze_run`
|
- `generated.current_analyze_run`
|
||||||
- `filesystem.disabled_artifact_output`
|
- `filesystem.disabled_artifact_output`
|
||||||
- `manifest.inputs.previous_cache`
|
- `manifest.inputs.previous_cache`
|
||||||
- `current_session.previous_cache`
|
- `current_session.previous_cache`
|
||||||
|
|
||||||
## Resolution Rules
|
## Resolution Rules
|
||||||
|
|
||||||
Built-ins:
|
Built-ins:
|
||||||
|
|
||||||
1. manifest producer outputs (when present)
|
1. manifest producer outputs (when present)
|
||||||
2. canonical session path fallback
|
2. canonical session-path fallback
|
||||||
|
|
||||||
Configured sources (`narratio.artifact.*`):
|
Configured sources (`narratio.artifact.*`):
|
||||||
|
|
||||||
- resolve only through runtime catalog availability.
|
- resolve only through runtime catalog availability.
|
||||||
|
|
||||||
Previous-session sources (`narratio.previous_session.artifact.*`):
|
Previous-session sources (`narratio.previous_session.artifact.*`):
|
||||||
- resolve only from local `previous/` cache state.
|
|
||||||
- prefer manifest-backed previous input paths.
|
- resolve only from local `previous/` cache state;
|
||||||
|
- prefer manifest-backed previous-input paths;
|
||||||
- fallback to existing previous-cache filesystem paths.
|
- fallback to existing previous-cache filesystem paths.
|
||||||
|
|
||||||
Validation by content type:
|
Validation by content type:
|
||||||
- transcript built-ins: JSON with top-level `segments` array.
|
|
||||||
- bounds built-in: valid JSON.
|
- transcript built-ins: JSON with top-level `segments` array;
|
||||||
|
- bounds built-in: valid JSON;
|
||||||
- configured/previous-session artifact files: non-empty text file.
|
- configured/previous-session artifact files: non-empty text file.
|
||||||
|
|
||||||
## Previous Requirement Collection
|
## Previous Requirement Collection
|
||||||
|
|
||||||
`CollectPreviousArtifactRequirements`:
|
`CollectPreviousArtifactRequirements`:
|
||||||
|
|
||||||
- scans enabled configured artifacts only;
|
- scans enabled configured artifacts only;
|
||||||
- extracts only canonical previous-session sources;
|
- extracts only canonical previous-session sources;
|
||||||
- deduplicates by artifact key;
|
- deduplicates by artifact key;
|
||||||
- merges required/optional (required wins);
|
- merges required and optional references (required wins);
|
||||||
- returns deterministic ordering and source locations.
|
- returns deterministic ordering and source locations.
|
||||||
|
|
||||||
|
## Current-State Helpers
|
||||||
|
|
||||||
|
Artifacts package owns shared remote current-state loading mechanics used by restore, status/validate checks, and previous-cache planning.
|
||||||
|
|
||||||
|
Core helpers:
|
||||||
|
|
||||||
|
- `LoadCurrentRunPointer`
|
||||||
|
- `LoadCurrentManifest`
|
||||||
|
- `LoadCurrentState`
|
||||||
|
- `ValidateCurrentStateIdentity`
|
||||||
|
|
||||||
|
Typed missing-state errors:
|
||||||
|
|
||||||
|
- `CurrentRunPointerMissingError` (`ErrCurrentRunPointerMissing`)
|
||||||
|
- `CurrentManifestMissingError` (`ErrCurrentManifestMissing`)
|
||||||
|
|
||||||
|
Identity validation supports caller-provided expectations:
|
||||||
|
|
||||||
|
- expected campaign;
|
||||||
|
- expected session ID;
|
||||||
|
- expected run ID, or pointer/manifest run-ID consistency check.
|
||||||
|
|
||||||
|
Caller policy is intentionally outside artifacts helpers:
|
||||||
|
|
||||||
|
- some callers fail on missing current state;
|
||||||
|
- some callers downgrade missing state to status/findings;
|
||||||
|
- some callers skip optional behavior when state is missing.
|
||||||
|
|
||||||
## Key Path Helpers
|
## Key Path Helpers
|
||||||
`internal/artifacts/paths.go` defines canonical helpers for:
|
|
||||||
|
`internal/artifacts/paths.go` and S3-key helpers define canonical helpers for:
|
||||||
|
|
||||||
- session/work/run paths;
|
- session/work/run paths;
|
||||||
- previous-cache paths;
|
- previous-cache paths;
|
||||||
- spool/cache paths;
|
- spool/cache paths;
|
||||||
- S3 key layout helpers for session/run/current pointers.
|
- S3 session/run/current-state key layout.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
- Source ID formats are stable contracts.
|
|
||||||
- Resolution is deterministic and manifest-aware.
|
- source ID formats are stable contracts;
|
||||||
- Previous-session source resolution does not call remote storage in `analyze`; remote hydration is `prepare` responsibility.
|
- artifact resolution is deterministic and manifest-aware;
|
||||||
|
- previous-session source resolution in `analyze` is local-only;
|
||||||
|
- remote current-state key construction remains centralized in artifacts helpers.
|
||||||
|
|||||||
@@ -1,65 +1,84 @@
|
|||||||
# Internal: Command Restore
|
# Internal: Command Restore
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Document the implemented `narratio session restore` command contract:
|
Define the implemented `narratio session restore` command contract:
|
||||||
|
|
||||||
- committed remote current-state discovery;
|
- committed remote current-state discovery;
|
||||||
- deterministic restore plan classification;
|
- deterministic restore planning;
|
||||||
- safe local install semantics;
|
- safe local install semantics;
|
||||||
- durable restore reporting.
|
- durable restore reporting.
|
||||||
|
|
||||||
## Discovery Contract
|
## Discovery Contract
|
||||||
Restore discovers remote committed state using:
|
|
||||||
- `current/run_id.txt` (required, non-empty)
|
|
||||||
- `current/manifest.json` (required, decodable)
|
|
||||||
|
|
||||||
Discovered manifest identity must match requested `session_id` and `campaign`.
|
Restore resolves remote committed state from the session publish current pointers:
|
||||||
|
|
||||||
## Plan Contract
|
- `current/run_id.txt` (required, non-empty);
|
||||||
Planner actions:
|
- `current/manifest.json` (required, decodable).
|
||||||
- `download`
|
|
||||||
- `skip_same`
|
Current-state discovery uses shared artifacts-level mechanics and validates identity against the resolved request config:
|
||||||
- `conflict`
|
|
||||||
|
- campaign must match;
|
||||||
|
- session ID must match.
|
||||||
|
|
||||||
|
Restore treats any missing or invalid remote current state as a command error.
|
||||||
|
|
||||||
|
## Planning Contract
|
||||||
|
|
||||||
|
Restore planner action kinds:
|
||||||
|
|
||||||
|
- `download`;
|
||||||
|
- `skip_same`;
|
||||||
|
- `conflict`.
|
||||||
|
|
||||||
|
Planner behavior:
|
||||||
|
|
||||||
Plan behavior:
|
|
||||||
- remote list scope is the resolved session prefix;
|
- remote list scope is the resolved session prefix;
|
||||||
- mapping to local paths is traversal-safe;
|
- remote-to-local mapping is traversal-safe;
|
||||||
- actions are sorted deterministically by local relative path.
|
- actions are sorted deterministically by local relative path.
|
||||||
|
|
||||||
Restore scope from current remote state:
|
Restore scope from current remote state:
|
||||||
- include `manifest.json`
|
|
||||||
- include `transcripts/**`
|
- include `manifest.json`;
|
||||||
- include `artifacts/**`
|
- include `transcripts/**`;
|
||||||
- include `audio/**` only with `--include-audio`
|
- include `artifacts/**`;
|
||||||
|
- include `audio/**` only with `--include-audio`.
|
||||||
|
|
||||||
Explicit exclusions from current remote state mapping:
|
Explicit exclusions from current remote state mapping:
|
||||||
- `current/**`
|
|
||||||
- `runs/**`
|
|
||||||
- `logs/**`
|
|
||||||
- `reports/**`
|
|
||||||
- `config/**`
|
|
||||||
- `inputs/**`
|
|
||||||
- `previous/**`
|
|
||||||
|
|
||||||
Previous-cache restore files are planned separately through `previouscache.BuildPlan` when configured previous-session requirements exist.
|
- `current/**`;
|
||||||
|
- `runs/**`;
|
||||||
|
- `logs/**`;
|
||||||
|
- `reports/**`;
|
||||||
|
- `config/**`;
|
||||||
|
- `inputs/**`;
|
||||||
|
- `previous/**`.
|
||||||
|
|
||||||
|
Previous-cache files are planned separately through `previouscache.BuildPlan` when configured previous-session requirements exist.
|
||||||
|
|
||||||
## Execution Contract
|
## Execution Contract
|
||||||
|
|
||||||
|
Execution order and safety:
|
||||||
|
|
||||||
- non-manifest downloads happen before manifest install;
|
- non-manifest downloads happen before manifest install;
|
||||||
- `manifest.json` is installed last;
|
- `manifest.json` installs last;
|
||||||
- downloads use sibling temp files + atomic rename;
|
- downloads use sibling temp files plus atomic rename;
|
||||||
- manifest replacement is validated before rename;
|
- manifest replacement is validated before rename;
|
||||||
- failed installs do not roll back previously written files.
|
- failed installs do not roll back files already written in the same execution.
|
||||||
|
|
||||||
Audio restore path:
|
Audio restore path:
|
||||||
|
|
||||||
- uses `audio.MaterializeS3Audio`;
|
- uses `audio.MaterializeS3Audio`;
|
||||||
- integrates spool and S3 audio cache paths;
|
- integrates spool and S3 audio cache paths;
|
||||||
- supports cache hit reuse without object redownload.
|
- supports cache-hit reuse without object redownload.
|
||||||
|
|
||||||
## Reporting Contract
|
## Reporting Contract
|
||||||
- dry-run: summary only (no writes).
|
|
||||||
|
- `--dry-run`: prints summary only; no local writes.
|
||||||
- non-dry-run: writes `reports/restore-latest.json`.
|
- non-dry-run: writes `reports/restore-latest.json`.
|
||||||
- report captures plan counts, action status, and execution failures.
|
- report includes plan counts, per-action status, and execution failures.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
- restore uses only committed remote current state as authority.
|
|
||||||
- `current/run_id.txt` is the remote commit marker.
|
- restore uses committed remote current state as authority;
|
||||||
- restore is a standalone command and does not run stages.
|
- `current/run_id.txt` is the remote publish commit marker;
|
||||||
|
- restore does not execute pipeline stages.
|
||||||
|
|||||||
@@ -29,9 +29,6 @@ S3 constructor behavior:
|
|||||||
- `Upload` streams local file and returns remote metadata.
|
- `Upload` streams local file and returns remote metadata.
|
||||||
- `Exists` maps not-found responses to `false`.
|
- `Exists` maps not-found responses to `false`.
|
||||||
|
|
||||||
## Legacy Compatibility Interface
|
|
||||||
`storage.Backend` (with `ArchiveRequest`) remains as compatibility surface with `NoopBackend`; it is not used by current stage execution.
|
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
- storage layer is stateless regarding manifest/stage progression.
|
- storage layer is stateless regarding manifest/stage progression.
|
||||||
- publish ordering semantics are owned by stage/app code, not storage adapters.
|
- publish ordering semantics are owned by stage/app code, not storage adapters.
|
||||||
|
|||||||
@@ -40,7 +40,7 @@ Run-local outputs are materialized back into canonical session paths before stag
|
|||||||
`artifacts.LocalStore` enforces single-writer session lock via `.lock` file (`ErrLockConflict` on contention).
|
`artifacts.LocalStore` enforces single-writer session lock via `.lock` file (`ErrLockConflict` on contention).
|
||||||
|
|
||||||
## Cleanup Semantics
|
## Cleanup Semantics
|
||||||
Automatic post-publish cleanup (`runPostArchiveCleanup`):
|
Automatic post-publish cleanup:
|
||||||
- only runs when publish actually executed and succeeded;
|
- only runs when publish actually executed and succeeded;
|
||||||
- requires `uploaded=true` and `current_pointer_written=true` metadata;
|
- requires `uploaded=true` and `current_pointer_written=true` metadata;
|
||||||
- respects `pipeline.spool.delete_audio_after_publish` and `pipeline.workspace.cleanup_after_publish`;
|
- respects `pipeline.spool.delete_audio_after_publish` and `pipeline.workspace.cleanup_after_publish`;
|
||||||
|
|||||||
@@ -4,33 +4,6 @@ Operator workflow for running, recovering, and publishing Narratio sessions.
|
|||||||
|
|
||||||
For command syntax, see [docs/cli.md](./cli.md). For field-level config, see [docs/config.md](./config.md).
|
For command syntax, see [docs/cli.md](./cli.md). For field-level config, see [docs/config.md](./config.md).
|
||||||
|
|
||||||
## Standard Session Workflow
|
|
||||||
|
|
||||||
1. Select pipeline/campaign/session config.
|
|
||||||
2. Validate session readiness:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio session validate 2026-04-04
|
|
||||||
```
|
|
||||||
|
|
||||||
3. (Optional) inspect stage decisions:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio session plan 2026-04-04
|
|
||||||
```
|
|
||||||
|
|
||||||
4. Run the pipeline:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio run 2026-04-04
|
|
||||||
```
|
|
||||||
|
|
||||||
5. Check state:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio session status 2026-04-04
|
|
||||||
```
|
|
||||||
|
|
||||||
## Campaign and Session Selection
|
## Campaign and Session Selection
|
||||||
|
|
||||||
Campaign selection priority:
|
Campaign selection priority:
|
||||||
@@ -63,7 +36,34 @@ narratio session init 2026-04-04 --remote --force
|
|||||||
|
|
||||||
If `campaign.yml` sets `session_template_file`, `session init` renders it. Template variables must resolve to concrete values.
|
If `campaign.yml` sets `session_template_file`, `session init` renders it. Template variables must resolve to concrete values.
|
||||||
|
|
||||||
## Stage Execution and Resume Behavior
|
## Standard Session Workflow
|
||||||
|
|
||||||
|
1. Select pipeline/campaign/session config.
|
||||||
|
2. Validate session readiness:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session validate 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
3. (Optional) inspect stage decisions:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session plan 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Run the pipeline:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
5. Check state:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session status 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
## Stage Execution and Continuation Behavior
|
||||||
|
|
||||||
Canonical stage order:
|
Canonical stage order:
|
||||||
|
|
||||||
@@ -80,7 +80,7 @@ Canonical stage order:
|
|||||||
Execution rules:
|
Execution rules:
|
||||||
|
|
||||||
- succeeded stages are skipped unless `--force` is set;
|
- succeeded stages are skipped unless `--force` is set;
|
||||||
- `resume` starts at first non-succeeded stage;
|
- `run` continues interrupted or partially completed sessions by running non-succeeded stages;
|
||||||
- force rerunning a succeeded upstream stage marks succeeded downstream stages as `stale`.
|
- force rerunning a succeeded upstream stage marks succeeded downstream stages as `stale`.
|
||||||
|
|
||||||
Single-stage execution:
|
Single-stage execution:
|
||||||
@@ -91,7 +91,7 @@ narratio run-stage normalize 2026-04-04 --force
|
|||||||
|
|
||||||
## Artifact Selection
|
## Artifact Selection
|
||||||
|
|
||||||
`--artifacts` can be used on `run`, `resume`, `run-stage`, `analyze`, and `publish`.
|
`--artifacts` can be used on `run`, `run-stage`, `analyze`, and `publish`.
|
||||||
|
|
||||||
Selection behavior:
|
Selection behavior:
|
||||||
|
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ Canonical contributor workflow and engineering conventions for implemented Narra
|
|||||||
## Repository layout
|
## Repository layout
|
||||||
|
|
||||||
- `cmd/narratio/`: CLI entrypoint.
|
- `cmd/narratio/`: CLI entrypoint.
|
||||||
- `internal/app/`: command handlers, plan/run/resume orchestration, cleanup gates, secrets loading.
|
- `internal/app/`: command handlers, run/stage orchestration, cleanup gates, secrets loading.
|
||||||
- `internal/config/`: strict YAML loading, defaults, and validation.
|
- `internal/config/`: strict YAML loading, defaults, and validation.
|
||||||
- `internal/stage/`: stage implementations and stage registry/order.
|
- `internal/stage/`: stage implementations and stage registry/order.
|
||||||
- `internal/adapters/`: external boundary adapters (WhisperX, Seriatim, Audita, Scriptorium, storage, notify).
|
- `internal/adapters/`: external boundary adapters (WhisperX, Seriatim, Audita, Scriptorium, storage, notify).
|
||||||
|
|||||||
@@ -1,556 +0,0 @@
|
|||||||
# Roadmap: Code Quality and Deduplication Audit
|
|
||||||
|
|
||||||
Status: Draft audit report
|
|
||||||
|
|
||||||
This report is a pre-1.0 implementation audit focused on high-confidence opportunities to simplify, centralize, or clarify Narratio before release. It is intentionally report-only: no refactors are included here.
|
|
||||||
|
|
||||||
The requested `docs/architecture.md` and `docs/development.md` paths do not exist in the current tree. This audit used the current policy documents at `docs/policy/architecture.md` and `docs/policy/development.md`, plus the current user, operator, and internal docs.
|
|
||||||
|
|
||||||
## 1. Executive Summary
|
|
||||||
|
|
||||||
Overall code quality is solid. The codebase has strong package boundaries in the important places: storage adapters expose a narrow object-store interface, AWS SDK types do not leak into app or stage logic, config loading is strict, and pipeline execution remains explicit and stage-driven. Recent pre-1.0 work has also produced useful central points for campaign/session config loading, secret-backed object-store creation, S3 audio caching, transcript artifact naming, local session paths, and S3 key construction.
|
|
||||||
|
|
||||||
The main release risk is not a large architectural flaw. It is policy drift from rapid feature growth. Several public-interface decisions now appear in more than one implementation path: artifact source interpretation, publish-output destination derivation, remote current-state inspection, cleanup safety checks, and session-oriented command parsing. Most of these are correct today, but a future bug fix would likely have to be made in multiple files.
|
|
||||||
|
|
||||||
Top three refactoring targets before 1.0:
|
|
||||||
|
|
||||||
1. Centralize artifact source and publish-output resolution across config validation, publish execution, status/artifacts output, restore, previous-cache hydration, and analyze input resolution.
|
|
||||||
2. Consolidate shared session-command flag parsing and config-loading context for run/resume/run-stage/analyze/publish/restore/clean/session helpers without introducing a generic command framework.
|
|
||||||
3. Finish the publish terminology cleanup internally so public `publish` behavior is not implemented through `archive`-named files, helpers, errors, and tests.
|
|
||||||
|
|
||||||
The codebase appears ready for a limited cleanup pass. No major architecture rewrite is warranted before 1.0.
|
|
||||||
|
|
||||||
## 2. High-Confidence Deduplication Opportunities
|
|
||||||
|
|
||||||
### Artifact Source and Publish Destination Policy Is Split Across Packages
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/config/validate.go`
|
|
||||||
- `internal/artifacts/artifact_resolver.go`
|
|
||||||
- `internal/artifacts/catalog.go`
|
|
||||||
- `internal/stage/archive.go`
|
|
||||||
- `internal/app/operator_helpers.go`
|
|
||||||
- `internal/previouscache/previouscache.go`
|
|
||||||
- `internal/stage/analyze.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
- Config validation accepts and derives destinations for `pipeline.publish.outputs[]` in `publishSourceKnown` and `derivePublishOutputDest`.
|
|
||||||
- Publish execution derives destinations again in `resolvePublishOutputDest`.
|
|
||||||
- Status and `artifacts list` derive destination display and remote checks in `helperPublishedOutputDest`.
|
|
||||||
- Previous-cache hydration reconstructs candidate artifact locations from manifest outputs, published paths, and configured Scriptorium paths in `artifactRelativePathCandidates`.
|
|
||||||
- Analyze resolves previous-session, built-in, and configured artifact sources separately in `resolveScriptoriumInput`.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
Artifact source IDs now define the public contract for analyze inputs, previous-session inputs, publish outputs, locks, status, artifacts listing, restore, and validation. When source interpretation is spread across these packages, it is easy for one path to accept, reject, or resolve a source differently from another.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Create one small artifact-source policy layer, likely in `internal/artifacts` or a dependency-light sibling of `internal/artifactmodel`, that can:
|
|
||||||
|
|
||||||
- classify source IDs as built-in, configured artifact, or previous-session configured artifact;
|
|
||||||
- validate a source against the current Scriptorium config;
|
|
||||||
- derive the default published destination for a source;
|
|
||||||
- normalize relative artifact destinations;
|
|
||||||
- return consistent display metadata for status and artifacts output.
|
|
||||||
|
|
||||||
Then update config validation, publish execution, helper commands, previous-cache planning, and analyze input resolution to call that policy instead of deriving partial answers locally.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- `internal/artifacts`: source classification, configured artifact validation, default destination derivation, relative destination normalization.
|
|
||||||
- `internal/config`: publish outputs and locks validate through the shared policy.
|
|
||||||
- `internal/stage`: publish output resolution preserves locked, optional, required, and selected-artifact behavior.
|
|
||||||
- `internal/app`: `artifacts list`, `status`, and locks use the same source rules as publish.
|
|
||||||
- `internal/previouscache`: previous-session source resolution still checks manifest outputs, published paths, and configured output paths in the intended order.
|
|
||||||
|
|
||||||
Risk level: Medium. The behavior is public, but a table-driven shared policy should reduce risk if introduced behind existing tests.
|
|
||||||
|
|
||||||
### Publish Terminology Cleanup Is Incomplete Internally
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/stage/archive.go`
|
|
||||||
- `internal/stage/archive_test.go`
|
|
||||||
- `internal/artifacts/archive_identity.go`
|
|
||||||
- `internal/app/post_archive_cleanup.go`
|
|
||||||
- `internal/app/remote_locks.go`
|
|
||||||
- `internal/app/operator_helpers.go`
|
|
||||||
- tests under `internal/app` and `internal/config`
|
|
||||||
- `internal/adapters/storage/archive.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
The public contract now uses `publish`, `published`, and `publish outputs`, but several internal names still use `archive`, `promotion`, or `promoted`. Examples include `archiveStage`, `ResolveArchiveSessionPrefix`, `ResolveArchiveCurrentStateKeys`, `runPostArchiveCleanup`, `staticArchiveLocks`, `normalizeArchiveRelativePath`, and test names such as `TestArchiveUploadsRunRecordPromotionsAndCurrentPointer`.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
This is mostly clarity risk, not current behavior risk. However, public docs and config now use publish terminology, while implementation and tests still use old names. This makes code review harder and increases the chance that future work reintroduces old config or command language.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Do a mechanical naming cleanup after artifact-source policy is centralized:
|
|
||||||
|
|
||||||
- rename `internal/stage/archive.go` to a publish-oriented file and rename `archiveStage` to `publishStage`;
|
|
||||||
- rename archive identity helpers to publish/current-state helpers while keeping S3 layout unchanged;
|
|
||||||
- rename post-archive cleanup helpers and tests to post-publish cleanup;
|
|
||||||
- update old comments and test failure messages that still say archive/promote when they mean publish/published;
|
|
||||||
- leave the immutable run-history path `runs/{run_id}` unchanged.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Existing `internal/stage`, `internal/app`, and `internal/artifacts` tests.
|
|
||||||
- A final term sweep for old terminology, allowing only historical roadmap references and adapter names that are intentionally retained.
|
|
||||||
|
|
||||||
Risk level: Low to Medium. Mostly mechanical, but broad enough to create churn.
|
|
||||||
|
|
||||||
### Session-Oriented CLI Parsing Is Repeated
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/run.go`
|
|
||||||
- `internal/app/resume.go`
|
|
||||||
- `internal/app/run_stage.go`
|
|
||||||
- `internal/app/restore.go`
|
|
||||||
- `internal/app/clean.go`
|
|
||||||
- `internal/app/operator_helpers.go`
|
|
||||||
- `internal/app/session_args.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
Many commands repeat the same flag setup and session ID handling:
|
|
||||||
|
|
||||||
- `--config`, `--campaign`, `--campaign-file`, `--session`, and `--previous-session-id`;
|
|
||||||
- positional session ID extraction;
|
|
||||||
- `--session-id` compatibility through `applyParsedSessionIDArg`;
|
|
||||||
- selected artifact parsing and validation for run/resume/analyze/publish/run-stage;
|
|
||||||
- load through `loadCommandConfig` followed by `config.Validate`.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
The command set has recently moved toward `narratio session <subcommand> <session_id>` and shorter top-level convenience commands. Repeated parser setup makes it easy for one command to miss a new flag, use a stale help string, or apply session ID precedence differently.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Keep command functions explicit, but add a small internal parser helper for common session-aware commands. Avoid a generic CLI framework. A good target is a helper that returns:
|
|
||||||
|
|
||||||
- common config flags;
|
|
||||||
- resolved positional/flag session ID;
|
|
||||||
- previous session override;
|
|
||||||
- optional selected configured artifacts;
|
|
||||||
- normalized command-specific positional validation.
|
|
||||||
|
|
||||||
`run-stage` can remain special because it has both stage and session positional arguments, but it should reuse the same common flag registration and selected-artifact parsing.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Existing app command tests for run, resume, run-stage, analyze, publish, restore, clean, and session subcommands.
|
|
||||||
- Focused tests for positional session ID vs `--session-id` mismatch, missing session ID, and unsupported `--artifacts` by command/stage.
|
|
||||||
|
|
||||||
Risk level: Medium. Refactor is local to app parsing but touches many public commands.
|
|
||||||
|
|
||||||
### Remote Current-State Discovery Is Reimplemented in Several Forms
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/restore_discovery.go`
|
|
||||||
- `internal/previouscache/previouscache.go`
|
|
||||||
- `internal/app/operator_helpers.go`
|
|
||||||
- `internal/stage/prepare_previous.go`
|
|
||||||
- `internal/app/remote_locks.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
Several paths check or download remote current state:
|
|
||||||
|
|
||||||
- restore discovers current run ID and current manifest, validates campaign/session identity, and decodes the manifest;
|
|
||||||
- previous-cache planning repeats current run pointer and manifest checks for the previous session;
|
|
||||||
- session validation checks previous current state with `Exists` calls;
|
|
||||||
- remote lock loading separately checks and downloads `locks.yml`;
|
|
||||||
- remote session fallback lists and downloads `session.yml`.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
These workflows are similar but not identical. Some need missing remote state to be an error, while status treats it as state. Still, the low-level sequence of key construction, `Exists`, temp download, decode, and campaign/session/run validation appears multiple times.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Extract narrow app-level or artifact-level helpers for remote session state objects, not a generic storage workflow engine. Candidate helpers:
|
|
||||||
|
|
||||||
- download object to temp safely;
|
|
||||||
- load current run pointer and manifest for a supplied session prefix;
|
|
||||||
- validate downloaded current manifest identity;
|
|
||||||
- represent missing current state as a typed error so status can downgrade it while restore/prepare fail.
|
|
||||||
|
|
||||||
Keep `storage.ObjectStore` as the boundary and keep S3 key construction in `internal/artifacts`.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- `internal/app`: restore current-state discovery, status missing-state behavior, session validate previous-state behavior.
|
|
||||||
- `internal/previouscache`: required vs optional previous artifact behavior with missing current pointers/manifests.
|
|
||||||
- `internal/app`: malformed remote lock/session data still fails closed where publish-capable execution requires it.
|
|
||||||
|
|
||||||
Risk level: Medium. The missing-state policy differs by caller, so the refactor should centralize mechanics and typed outcomes, not final command decisions.
|
|
||||||
|
|
||||||
### Safe Local Deletion Policy Is Duplicated
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/clean.go`
|
|
||||||
- `internal/app/post_archive_cleanup.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
Both `clean` and post-publish cleanup implement scoped deletion checks:
|
|
||||||
|
|
||||||
- reject empty roots/targets;
|
|
||||||
- resolve absolute paths;
|
|
||||||
- refuse root deletion;
|
|
||||||
- refuse deletion outside the configured root;
|
|
||||||
- refuse symlink deletion;
|
|
||||||
- handle missing targets as successful no-ops.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
Deletion policy is high-risk code. Even if the current implementations agree, future fixes should not need to be made twice.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Extract a small app-level cleanup safety helper, for example `cleanup_target.go`, with functions for:
|
|
||||||
|
|
||||||
- validating a scoped directory target;
|
|
||||||
- validating a scoped file target;
|
|
||||||
- validating removable children under a root.
|
|
||||||
|
|
||||||
Keep command-specific reporting in `clean.go` and manifest metadata handling in post-publish cleanup.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Move the existing focused unsafe-path tests to the shared helper.
|
|
||||||
- Preserve `clean` dry-run tests and post-publish cleanup eligibility tests.
|
|
||||||
|
|
||||||
Risk level: Low. This is a contained refactor with clear behavior preservation.
|
|
||||||
|
|
||||||
### Temp Object Download Helper Is Duplicated
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/restore_discovery.go`
|
|
||||||
- `internal/previouscache/previouscache.go`
|
|
||||||
- `internal/app/remote_locks.go`
|
|
||||||
- `internal/app/config_loader.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
Multiple call sites create a temp file, close it, download an object into it, and delete it on error or defer deletion. The app package has one `downloadObjectToTemp`, while `internal/previouscache` has another copy.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
Temp-download behavior affects cleanup, error wording, and future hardening. It is not worth abstracting all storage use, but this small operation is repeated enough to centralize.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Add a narrow helper close to the storage boundary. Options:
|
|
||||||
|
|
||||||
- `internal/adapters/storage` helper only if it does not learn Narratio session semantics;
|
|
||||||
- `internal/storageutil` if a small internal utility package is acceptable;
|
|
||||||
- app-level helper plus a previouscache dependency inversion if the team wants to avoid a new package.
|
|
||||||
|
|
||||||
The helper should not hide `ObjectStore`; it should only implement safe temp download mechanics.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- temp file cleanup on failed download;
|
|
||||||
- successful download returns a cleaned temp path;
|
|
||||||
- callers preserve their current contextual error messages.
|
|
||||||
|
|
||||||
Risk level: Low.
|
|
||||||
|
|
||||||
## 3. Medium-Confidence Opportunities
|
|
||||||
|
|
||||||
### Operator Helper Implementation Is Too Broad for One File
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/operator_helpers.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
This 1,100+ line file owns session validation, status, session init, artifacts listing, locks list/add/remove, lock-store mutation, artifact catalog rendering, remote output availability, finding formatting, local input validation, and template rendering.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
The code is not inherently wrong, and keeping helper commands in `internal/app` fits the architecture. The issue is discoverability and local coupling. Small changes to one helper command require navigating unrelated helper behavior.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Split by command or responsibility:
|
|
||||||
|
|
||||||
- `session_init.go`
|
|
||||||
- `session_validate.go`
|
|
||||||
- `status.go`
|
|
||||||
- `artifacts_list.go`
|
|
||||||
- `locks.go`
|
|
||||||
- `helper_findings.go`
|
|
||||||
- `helper_artifacts.go`
|
|
||||||
|
|
||||||
Do this only after higher-value policy centralization so the file split does not preserve duplicated logic under new names.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Existing `internal/app/operator_helpers_test.go` can be split later, but a file split alone should not require behavior changes.
|
|
||||||
|
|
||||||
Risk level: Low.
|
|
||||||
|
|
||||||
### Restore Planning Contains Its Own Remote-to-Local Path Policy
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/restore_plan.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
Restore maps remote session keys back to local session paths in `restoreLocalRelativePathForKey`, with explicit include/exclude rules for `current/`, `runs/`, `logs/`, `reports/`, `config/`, `inputs/`, `transcripts/`, `artifacts/`, `previous/`, and optional `audio/`.
|
|
||||||
|
|
||||||
Why it may be intentional:
|
|
||||||
|
|
||||||
Restore is the only command that should translate an entire remote session prefix into a local session subset. It has command-specific conflict and `--include-audio` semantics.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Do not generalize this immediately. If it changes again, move only the remote-key-to-local-restore-scope classifier into a small helper with table-driven tests. Leave restore action classification local to restore.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Restore scope tests for every included/excluded root.
|
|
||||||
- Audio-specific conflict behavior remains separate.
|
|
||||||
|
|
||||||
Risk level: Low.
|
|
||||||
|
|
||||||
### Manifest Output Scanning Is Repeated but Mostly Stage-Specific
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/artifacts/artifact_resolver.go`
|
|
||||||
- `internal/previouscache/previouscache.go`
|
|
||||||
- `internal/app/runner.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
Several call sites inspect manifest stage outputs or metadata to find artifact paths, published paths, run roots, or configured artifact outputs.
|
|
||||||
|
|
||||||
Why it may be intentional:
|
|
||||||
|
|
||||||
Manifest state has different meanings depending on caller: runtime artifact resolution, previous-cache reconstruction, and run summary construction are not the same policy.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Avoid a broad manifest-query abstraction before 1.0. Consider adding only narrow helpers for stable metadata reads, such as reading `published_paths` from the publish stage, if the previous-cache and restore paths continue to grow.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Existing manifest resolver tests plus previous-cache tests.
|
|
||||||
|
|
||||||
Risk level: Low.
|
|
||||||
|
|
||||||
### Command Output Formatting Could Be More Consistent
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/operator_helpers.go`
|
|
||||||
- `internal/app/restore_report.go`
|
|
||||||
- `internal/app/restore_plan.go`
|
|
||||||
- `internal/app/clean.go`
|
|
||||||
- `internal/app/plan.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
Status, session validate, artifacts list, locks, clean dry-run, restore dry-run, and plan all render text directly with `fmt.Fprintf`.
|
|
||||||
|
|
||||||
Why it may be intentional:
|
|
||||||
|
|
||||||
The output remains text-only and command-specific. A generic renderer would add complexity without much value.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Postpone unless user-facing inconsistencies become painful. A small findings renderer already exists for validation-style output; that is enough for now.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Snapshot-style output tests only for stable operator-facing lines that support workflows.
|
|
||||||
|
|
||||||
Risk level: Low.
|
|
||||||
|
|
||||||
## 4. Boundary and Responsibility Concerns
|
|
||||||
|
|
||||||
The major boundaries are healthy:
|
|
||||||
|
|
||||||
- `internal/adapters/storage` owns external storage implementation details.
|
|
||||||
- App code creates object stores through `newCommandObjectStore`, which loads filesystem secrets first.
|
|
||||||
- Stage code depends on `storage.ObjectStore`, not AWS SDK types.
|
|
||||||
- `internal/audio` correctly centralizes S3 audio cache materialization without making the storage adapter aware of cache policy.
|
|
||||||
- `internal/artifacts` owns most local paths and S3 keys.
|
|
||||||
|
|
||||||
Concerns to address:
|
|
||||||
|
|
||||||
- Artifact source policy is split between `internal/config`, `internal/artifacts`, `internal/stage`, `internal/app`, and `internal/previouscache`. This is the clearest boundary drift because source IDs are a shared public contract.
|
|
||||||
- `internal/config` currently derives default publish destinations. Validation should be able to call source policy, but the canonical mapping itself should live outside config.
|
|
||||||
- `internal/app/operator_helpers.go` owns artifact catalog rendering and remote published-output state. That is acceptable for formatting, but destination derivation and source classification should move out.
|
|
||||||
- `internal/stage/archive.go` implements the public `publish` stage. This does not violate boundaries, but it creates conceptual drift.
|
|
||||||
|
|
||||||
Recommended home for shared logic:
|
|
||||||
|
|
||||||
- Source classification and destination derivation: `internal/artifacts` or `internal/artifactmodel` plus a small adapter from Scriptorium config.
|
|
||||||
- Remote key construction: continue using `internal/artifacts`.
|
|
||||||
- Object-store initialization: keep in `internal/app`.
|
|
||||||
- Command parsing: keep in `internal/app`.
|
|
||||||
- Stage-specific execution policy: keep in `internal/stage`.
|
|
||||||
|
|
||||||
## 5. Path and Remote Key Construction Review
|
|
||||||
|
|
||||||
Local path construction is mostly centralized:
|
|
||||||
|
|
||||||
- `internal/artifacts/paths.go` owns session work roots, run roots, spool paths, previous-cache paths, and audio cache paths.
|
|
||||||
- Stage code often gets `artifacts.SessionPaths` and joins stage-local files from those roots, which is appropriate.
|
|
||||||
- The previous-cache redundant nested artifact path has already been addressed by `previousArtifactCacheRelativePath`.
|
|
||||||
|
|
||||||
Remote key construction is mostly centralized:
|
|
||||||
|
|
||||||
- `internal/artifacts/s3_keys.go` owns session prefixes, run prefixes, audio prefixes, `session.yml`, `locks.yml`, current manifest/run pointer keys, published output keys, and run-relative keys.
|
|
||||||
- App and stage code call these helpers rather than scattering full S3 key string concatenation.
|
|
||||||
|
|
||||||
Areas needing cleanup:
|
|
||||||
|
|
||||||
- `ResolveArchiveBucket`, `ResolveArchiveSessionPrefix`, `ResolveArchiveRunPrefix`, and `ResolveArchiveCurrentStateKeys` should be renamed to publish/current-state terminology.
|
|
||||||
- `normalizeArchiveRelativePath` exists in both `internal/stage/archive.go` and `internal/previouscache/previouscache.go`; `normalizeHelperArchiveRelativePath` exists in `internal/app/operator_helpers.go`. These should converge into one helper for clean relative artifact destination paths.
|
|
||||||
- `restore_plan.go` owns `normalizeRemoteKey` and remote key scope mapping. That may remain restore-specific, but it should be watched because it overlaps with S3 key normalization helpers.
|
|
||||||
- `downloadObjectToTemp` exists in more than one package and can be centralized.
|
|
||||||
|
|
||||||
## 6. Artifact/Catalog/Source Resolution Review
|
|
||||||
|
|
||||||
Artifact source handling has a strong foundation:
|
|
||||||
|
|
||||||
- Transcript source IDs and paths are centralized in `internal/artifactmodel/transcripts.go`.
|
|
||||||
- Runtime artifact registry and resolver live in `internal/artifacts/artifact_resolver.go`.
|
|
||||||
- Configured artifact source IDs are consistently formed by `artifacts.ConfiguredArtifactSourceID`.
|
|
||||||
- Previous-session source IDs are recognized by `artifacts.PreviousSessionArtifactName`.
|
|
||||||
- The runtime catalog supports built-ins, configured artifacts, selected artifact execution, and availability.
|
|
||||||
|
|
||||||
The remaining issue is that consumers still build their own partial views of this model:
|
|
||||||
|
|
||||||
- config validation validates and derives publish output destinations;
|
|
||||||
- publish execution resolves included outputs, skipped optional outputs, skipped unselected outputs, and locked outputs;
|
|
||||||
- status/artifacts list derives display destinations and remote published state;
|
|
||||||
- previous-cache planning reconstructs candidate remote paths from previous manifests and publish metadata;
|
|
||||||
- analyze input resolution has its own missing-source messages and previous-session behavior.
|
|
||||||
|
|
||||||
Recommendation:
|
|
||||||
|
|
||||||
Make artifact/source resolution the next cleanup target. The goal is not to create one all-purpose resolver. The goal is to centralize the public source vocabulary and destination derivation so each caller can keep its own policy for missing/required/locked behavior.
|
|
||||||
|
|
||||||
## 7. Config and Command-Loading Review
|
|
||||||
|
|
||||||
Config loading is generally consistent:
|
|
||||||
|
|
||||||
- `loadCommandConfig` is the main command path for pipeline, campaign, session, local discovery, and remote session fallback.
|
|
||||||
- `loadPipelineCampaignConfig` covers commands that create session config and therefore cannot load an existing session.
|
|
||||||
- `newCommandObjectStore` correctly centralizes secret-backed object-store creation.
|
|
||||||
- `config.LoadSessionBytesWithOptions` now rejects session templates outside `session init`, preserving strict concrete session loading.
|
|
||||||
|
|
||||||
Intentional differences:
|
|
||||||
|
|
||||||
- `session init` loads only pipeline and campaign because it creates `session.yml`.
|
|
||||||
- `clean --all` loads only pipeline because it is not session-specific.
|
|
||||||
- `status --manifest` remains a compatibility/local-manifest mode.
|
|
||||||
|
|
||||||
Likely accidental drift to clean up:
|
|
||||||
|
|
||||||
- Common flags and help strings are repeated across commands.
|
|
||||||
- Some helper-command messages still say archive where they now mean publish.
|
|
||||||
- `restore` uses `fs.SetOutput(out)` while most other command parsers discard flag package output and wrap errors themselves. This may be intentional for `--help`, but it is a difference worth documenting or standardizing.
|
|
||||||
- App tests and helper names still contain old archive/promotion terminology, making it harder to see which public contract is current.
|
|
||||||
|
|
||||||
## 8. Refactors to Avoid Before 1.0
|
|
||||||
|
|
||||||
Avoid these before release:
|
|
||||||
|
|
||||||
- A generic workflow engine or DAG abstraction. The explicit stage list is a core design choice and is working.
|
|
||||||
- A broad manifest query framework. Add narrow helpers only where repeated policy is clear.
|
|
||||||
- Moving secret loading into storage adapters. Secret loading is app orchestration policy and should stay out of adapters.
|
|
||||||
- Making storage adapters infer campaign/session/root-prefix semantics. They should continue to receive concrete keys.
|
|
||||||
- Replacing command functions with a generic CLI framework. Small shared flag parsers are enough.
|
|
||||||
- Generalizing all file copy/download behavior. S3 audio cache materialization is intentionally special; ordinary restore/download logic has different semantics.
|
|
||||||
- Adding compatibility aliases for old archive/promote or old transcript names during cleanup. The repo has intentionally made hard cutovers.
|
|
||||||
|
|
||||||
## 9. Recommended Implementation Sequence
|
|
||||||
|
|
||||||
1. Centralize relative artifact destination normalization and temp object download helpers.
|
|
||||||
- Scope: low-risk shared helpers for repeated mechanics.
|
|
||||||
- Tests: `internal/artifacts` or helper-package tests, plus existing app/stage tests.
|
|
||||||
|
|
||||||
2. Centralize artifact source and publish-output policy.
|
|
||||||
- Scope: source classification, source validation, default published destination derivation, destination normalization.
|
|
||||||
- Tests: `internal/artifacts`, `internal/config`, `internal/stage -run Publish`, `internal/app -run 'Artifacts|Status|Locks'`, `internal/previouscache`.
|
|
||||||
|
|
||||||
3. Finish publish terminology cleanup.
|
|
||||||
- Scope: rename archive-named files/helpers/tests/comments where they now mean publish; keep S3 layout stable.
|
|
||||||
- Tests: `go test ./internal/stage -v`, `go test ./internal/app -v`, `go test ./internal/artifacts -v`.
|
|
||||||
|
|
||||||
4. Consolidate session-aware command parsing.
|
|
||||||
- Scope: common config/session/artifact flag registration and session ID resolution; no public CLI behavior change.
|
|
||||||
- Tests: app command tests for run, resume, run-stage, analyze, publish, restore, clean, session helpers.
|
|
||||||
|
|
||||||
5. Extract remote current-state mechanics.
|
|
||||||
- Scope: shared helpers for current run pointer/manifest load and identity validation, with typed missing-state errors.
|
|
||||||
- Tests: restore discovery, previous-cache, status, session validate.
|
|
||||||
|
|
||||||
6. Split operator helper implementation by responsibility.
|
|
||||||
- Scope: file organization and small formatting/helper extraction only after policy deduplication.
|
|
||||||
- Tests: existing `internal/app` tests.
|
|
||||||
|
|
||||||
7. Sweep dead transitional terminology and stale tests.
|
|
||||||
- Scope: comments, test names, old strings, internal docs that still say archive/promote where publish is now canonical.
|
|
||||||
- Tests: final `rg` sweeps plus full test run.
|
|
||||||
|
|
||||||
## 10. Test Strategy
|
|
||||||
|
|
||||||
Focused package checks for cleanup work:
|
|
||||||
|
|
||||||
- `go test ./internal/artifacts -v`
|
|
||||||
- `go test ./internal/config -v`
|
|
||||||
- `go test ./internal/stage -run 'Analyze|Publish|Prepare|Restore' -v`
|
|
||||||
- `go test ./internal/app -run 'Run|RunStage|Analyze|Publish|Restore|Clean|Status|Artifacts|Locks|Session' -v`
|
|
||||||
- `go test ./internal/previouscache -v`
|
|
||||||
- `go test ./internal/adapters/storage -v`
|
|
||||||
- `go test ./internal/manifest -v`
|
|
||||||
|
|
||||||
Tests to add or strengthen during follow-up refactors:
|
|
||||||
|
|
||||||
- one table of valid/invalid artifact source IDs used by config validation, publish, locks, status, and analyze;
|
|
||||||
- one table of default published destination derivation for built-in and configured artifacts;
|
|
||||||
- relative destination normalization and path traversal rejection;
|
|
||||||
- shared remote current-state load outcomes: missing pointer, missing manifest, malformed manifest, campaign mismatch, session mismatch, run ID mismatch;
|
|
||||||
- shared cleanup safety helper behavior for files, directories, roots, symlinks, and outside-root paths;
|
|
||||||
- common session command parsing behavior for positional session IDs, `--session-id`, mismatch errors, and unsupported artifacts flags.
|
|
||||||
|
|
||||||
Full validation after each cleanup commit:
|
|
||||||
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Useful final searches:
|
|
||||||
|
|
||||||
- `rg -n "archive|promote|promoted|promotion" internal docs examples cmd`
|
|
||||||
- `rg -n "ResolveArchive|archiveStage|post_archive|staticArchive|normalizeArchive" internal`
|
|
||||||
- `rg -n "narratio.transcript.merged|narratio.transcript.full|narratio.transcript.trimmed" internal docs examples`
|
|
||||||
- `rg -n "previous_session_artifact|promote_artifacts|pipeline.archive" internal docs examples`
|
|
||||||
|
|
||||||
## 11. Appendix: Findings Not Worth Acting On
|
|
||||||
|
|
||||||
- Stage-local path joins for files inside a stage run directory are acceptable. They are local implementation details, not shared path policy.
|
|
||||||
- Direct `fmt.Fprintf` output in simple commands is acceptable. A generic renderer would likely obscure behavior.
|
|
||||||
- Restore's remote-session-prefix filtering is command-specific enough to stay local unless restore scope changes again.
|
|
||||||
- `session init` template rendering should remain separate from ordinary session loading. That separation is now a useful safety boundary.
|
|
||||||
- S3 audio cache materialization is already centralized in `internal/audio`; do not fold it into a generic downloader.
|
|
||||||
- Manifest-driven resume behavior should not be abstracted broadly. The explicit runner behavior is easier to audit.
|
|
||||||
@@ -1,294 +0,0 @@
|
|||||||
# Roadmap: Pre-1.0 Code Cleanup
|
|
||||||
|
|
||||||
Status: Planned
|
|
||||||
|
|
||||||
This roadmap turns the findings in `docs/roadmap/audit.md` into staged cleanup work for the 1.0 release. It is planning-only. Do not implement these refactors until a stage is explicitly selected for implementation.
|
|
||||||
|
|
||||||
The cleanup work must follow the policy documents under `docs/policy/`, especially these invariants:
|
|
||||||
|
|
||||||
- keep Narratio explicit and stage-driven;
|
|
||||||
- do not introduce a generic workflow engine, DAG abstraction, or generic CLI framework;
|
|
||||||
- keep external-system details behind adapters;
|
|
||||||
- do not move campaign/session/root-prefix semantics into storage adapters;
|
|
||||||
- keep AWS SDK types out of app and stage logic;
|
|
||||||
- keep path and remote key construction centralized;
|
|
||||||
- preserve manifest-driven run state;
|
|
||||||
- keep public CLI/config behavior stable unless a stage explicitly says it is an internal naming cleanup.
|
|
||||||
|
|
||||||
## Non-Goals
|
|
||||||
|
|
||||||
- Do not change public command syntax, config schema, S3 key layout, manifest schema, or artifact source IDs as part of this cleanup.
|
|
||||||
- Do not add compatibility aliases or migration logic.
|
|
||||||
- Do not rewrite stage execution, manifest state transitions, or adapter contracts.
|
|
||||||
- Do not generalize text output into a generic reporting framework.
|
|
||||||
- Do not fold S3 audio cache behavior into a generic downloader.
|
|
||||||
- Do not move secret loading into storage adapters.
|
|
||||||
|
|
||||||
## Stage 1: Shared Low-Risk Mechanics
|
|
||||||
|
|
||||||
Goal: remove duplicated mechanics that are easy to test and should not affect public behavior.
|
|
||||||
|
|
||||||
Implementation decisions:
|
|
||||||
|
|
||||||
- Add one shared helper for safe relative artifact destination normalization.
|
|
||||||
- It must reject empty paths, absolute paths, `.`, `..`, and traversal outside the artifact/session scope.
|
|
||||||
- It must normalize separators to slash-form for artifact and S3 destination logic.
|
|
||||||
- It must be dependency-light enough to be called from config validation, app helpers, publish execution, and previous-cache planning.
|
|
||||||
- Add one shared object-store temp download helper.
|
|
||||||
- It must take `context.Context`, `storage.ObjectStore`, a key, and a temp-file pattern.
|
|
||||||
- It must create and close the temp file before download, remove the temp file on failed download, and return a cleaned local path on success.
|
|
||||||
- It must not infer bucket, campaign, session, run, or root-prefix semantics.
|
|
||||||
- Extract shared cleanup target validation for local deletion.
|
|
||||||
- Cover scoped directory deletion, scoped file deletion, and removable children under a root.
|
|
||||||
- Preserve existing safety rules: reject empty roots/targets, root deletion, outside-root paths, symlinks, and wrong target types.
|
|
||||||
- Keep command-specific output in `clean` and manifest metadata handling in post-publish cleanup.
|
|
||||||
|
|
||||||
Expected callers:
|
|
||||||
|
|
||||||
- replace duplicate relative destination normalization in publish execution, helper command rendering, and previous-cache planning;
|
|
||||||
- replace duplicate temp download helpers in app and previous-cache code;
|
|
||||||
- replace duplicate scoped deletion validation in clean and post-publish cleanup.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- Add focused tests for destination normalization and path traversal rejection.
|
|
||||||
- Add temp download tests for success, failed download cleanup, and preserved contextual caller errors.
|
|
||||||
- Add shared cleanup validation tests for directories, files, symlinks, missing targets, root deletion, and outside-root targets.
|
|
||||||
- Run:
|
|
||||||
- `go test ./internal/artifacts -v`
|
|
||||||
- `go test ./internal/adapters/storage -v`
|
|
||||||
- `go test ./internal/app -run 'Clean|Post' -v`
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- duplicated low-level mechanics are removed;
|
|
||||||
- public behavior and output are unchanged;
|
|
||||||
- no stage, command, or config semantics move into storage adapters.
|
|
||||||
|
|
||||||
## Stage 2: Artifact Source and Published Output Policy
|
|
||||||
|
|
||||||
Goal: make artifact source IDs and published-output destination derivation a single shared policy.
|
|
||||||
|
|
||||||
Implementation decisions:
|
|
||||||
|
|
||||||
- Introduce `internal/artifactpolicy` as the shared source policy package.
|
|
||||||
- This package is the long-term home because it avoids config/artifacts import cycles.
|
|
||||||
- It may depend on dependency-light model packages, but it must not depend on app, stage, manifest stores, storage adapters, or downstream adapters.
|
|
||||||
- Centralize these behaviors in `internal/artifactpolicy`:
|
|
||||||
- classify source IDs as built-in, configured artifact, or previous-session configured artifact;
|
|
||||||
- parse configured artifact keys from `narratio.artifact.<key>`;
|
|
||||||
- parse previous-session artifact keys from `narratio.previous_session.artifact.<key>`;
|
|
||||||
- validate configured artifact sources against `pipeline.scriptorium.artifacts`;
|
|
||||||
- validate publish lock/output sources;
|
|
||||||
- derive default published destinations for built-in and configured artifact sources;
|
|
||||||
- normalize safe relative published-output destinations.
|
|
||||||
- Update callers to consume the shared policy:
|
|
||||||
- config validation for `publish.outputs` and `publish.locks`;
|
|
||||||
- publish-stage output resolution;
|
|
||||||
- status and `artifacts list` rendering;
|
|
||||||
- locks list/add/remove validation;
|
|
||||||
- analyze input source handling;
|
|
||||||
- previous-cache candidate planning.
|
|
||||||
- Preserve caller-specific policy at call sites.
|
|
||||||
- Required vs optional behavior remains in publish, analyze, restore, and previous-cache callers.
|
|
||||||
- Locked output behavior remains in publish.
|
|
||||||
- Text formatting remains in app commands.
|
|
||||||
- Manifest path scanning remains in artifact/previous-cache logic unless directly tied to source policy.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- Add `internal/artifactpolicy` table tests for source classification, configured artifact validation, previous-session parsing, default destination derivation, and destination normalization.
|
|
||||||
- Update `internal/config` tests so publish outputs and locks validate through the shared policy.
|
|
||||||
- Update `internal/stage` publish tests for selected, unselected, optional, required, and locked output behavior.
|
|
||||||
- Update `internal/app` tests for status, artifacts list, and locks.
|
|
||||||
- Update `internal/previouscache` tests for previous-session candidate ordering.
|
|
||||||
- Run:
|
|
||||||
- `go test ./internal/artifacts -v`
|
|
||||||
- `go test ./internal/config -v`
|
|
||||||
- `go test ./internal/stage -run 'Analyze|Publish' -v`
|
|
||||||
- `go test ./internal/app -run 'Artifacts|Status|Locks' -v`
|
|
||||||
- `go test ./internal/previouscache -v`
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- artifact source vocabulary and destination derivation are no longer reimplemented in config, app, stage, and previous-cache packages;
|
|
||||||
- every caller still owns its own missing/required/optional/locked decision;
|
|
||||||
- public behavior is unchanged.
|
|
||||||
|
|
||||||
## Stage 3: Publish Terminology Cleanup
|
|
||||||
|
|
||||||
Goal: align internal implementation names with the public publish contract.
|
|
||||||
|
|
||||||
Implementation decisions:
|
|
||||||
|
|
||||||
- Rename archive-named internal files, types, helpers, comments, and tests that now implement publish behavior.
|
|
||||||
- Replace names such as:
|
|
||||||
- `archiveStage` with `publishStage`;
|
|
||||||
- `ResolveArchiveSessionPrefix` with publish/current-state terminology;
|
|
||||||
- `ResolveArchiveRunPrefix` with publish/run-history terminology;
|
|
||||||
- `ResolveArchiveCurrentStateKeys` with current-state terminology;
|
|
||||||
- `runPostArchiveCleanup` with post-publish cleanup terminology;
|
|
||||||
- `staticArchiveLocks` with publish lock terminology.
|
|
||||||
- Keep the S3 layout stable:
|
|
||||||
- `{session_prefix}/runs/{run_id}/`;
|
|
||||||
- `{session_prefix}/current/manifest.json`;
|
|
||||||
- `{session_prefix}/current/run_id.txt`;
|
|
||||||
- `{session_prefix}/locks.yml`.
|
|
||||||
- Keep the public stage name `publish`.
|
|
||||||
- Keep old archive/promote references only where they are historical roadmap context or intentionally describe immutable run history.
|
|
||||||
|
|
||||||
Tests and checks:
|
|
||||||
|
|
||||||
- Run:
|
|
||||||
- `go test ./internal/stage -v`
|
|
||||||
- `go test ./internal/app -v`
|
|
||||||
- `go test ./internal/artifacts -v`
|
|
||||||
- `go test ./...`
|
|
||||||
- Run stale-term sweeps:
|
|
||||||
- `rg -n "archive|promote|promoted|promotion" internal docs examples cmd`
|
|
||||||
- `rg -n "ResolveArchive|archiveStage|post_archive|staticArchive|normalizeArchive" internal`
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- public publish behavior is no longer implemented through archive/promote names;
|
|
||||||
- remaining old terms are intentionally historical, test-fixture bucket names, or roadmap-only context;
|
|
||||||
- no config, CLI, manifest, or S3 layout changes are introduced.
|
|
||||||
|
|
||||||
## Stage 4: Session Command Parsing Consolidation
|
|
||||||
|
|
||||||
Goal: reduce command-loading drift while keeping command handlers explicit.
|
|
||||||
|
|
||||||
Implementation decisions:
|
|
||||||
|
|
||||||
- Add a small app-level parser helper for common session-aware commands.
|
|
||||||
- Centralize:
|
|
||||||
- common config flags: `--config`, `--campaign`, `--campaign-file`, `--session`;
|
|
||||||
- positional session ID handling;
|
|
||||||
- `--session-id` compatibility;
|
|
||||||
- `--previous-session-id`;
|
|
||||||
- optional selected-artifact parsing for commands that support it.
|
|
||||||
- Keep command handlers explicit and readable.
|
|
||||||
- Do not introduce a generic CLI framework.
|
|
||||||
- Treat these as intentional special cases:
|
|
||||||
- `session init` loads pipeline and campaign but not session;
|
|
||||||
- `clean --all` loads pipeline only;
|
|
||||||
- `status --manifest` remains local-manifest mode;
|
|
||||||
- `run-stage` keeps its stage-name positional handling but reuses common flag parsing where practical.
|
|
||||||
- Standardize flag help text where commands use the same semantics.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- Update app command tests for:
|
|
||||||
- positional session ID;
|
|
||||||
- `--session-id`;
|
|
||||||
- positional/flag mismatch;
|
|
||||||
- missing session ID;
|
|
||||||
- `--previous-session-id`;
|
|
||||||
- unsupported `--artifacts` by command/stage;
|
|
||||||
- unchanged behavior for `session init`, `clean --all`, and `status --manifest`.
|
|
||||||
- Run:
|
|
||||||
- `go test ./internal/app -run 'Run|RunStage|Analyze|Publish|Restore|Clean|Session' -v`
|
|
||||||
- `go test ./internal/app -v`
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- shared session flag/session ID behavior has one implementation;
|
|
||||||
- command handlers remain command-specific;
|
|
||||||
- public command syntax and output stay unchanged.
|
|
||||||
|
|
||||||
## Stage 5: Remote Current-State Mechanics
|
|
||||||
|
|
||||||
Goal: centralize remote current-state loading mechanics without hiding caller policy.
|
|
||||||
|
|
||||||
Implementation decisions:
|
|
||||||
|
|
||||||
- Extract narrow helpers for remote current state.
|
|
||||||
- Load current run pointer through `storage.ObjectStore`.
|
|
||||||
- Load and decode current manifest through `storage.ObjectStore`.
|
|
||||||
- Validate campaign, session, and run identity when requested by the caller.
|
|
||||||
- Return typed missing-state errors.
|
|
||||||
- Preserve caller policy:
|
|
||||||
- restore treats missing or invalid current state as an error;
|
|
||||||
- previous-cache hydration fails for required previous artifacts and skips optional missing artifacts;
|
|
||||||
- status reports missing remote state as state, not command failure;
|
|
||||||
- session validate emits findings and fails only for error findings.
|
|
||||||
- Keep all remote key construction in `internal/artifacts`.
|
|
||||||
- Keep object-store initialization in `internal/app`.
|
|
||||||
- Do not add storage adapter knowledge of campaigns, sessions, runs, root prefixes, current state, or manifests.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- Add helper tests for:
|
|
||||||
- missing current run pointer;
|
|
||||||
- missing current manifest;
|
|
||||||
- empty run pointer;
|
|
||||||
- malformed manifest;
|
|
||||||
- campaign mismatch;
|
|
||||||
- session mismatch;
|
|
||||||
- run ID mismatch.
|
|
||||||
- Update restore, previous-cache, status, and session validate tests to prove their caller-specific behavior is unchanged.
|
|
||||||
- Run:
|
|
||||||
- `go test ./internal/app -run 'Restore|Status|SessionValidate' -v`
|
|
||||||
- `go test ./internal/previouscache -v`
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- low-level remote current-state mechanics are shared;
|
|
||||||
- missing-state behavior remains caller-specific;
|
|
||||||
- storage adapter boundaries remain unchanged.
|
|
||||||
|
|
||||||
## Stage 6: Operator Helper File Split and Final Sweep
|
|
||||||
|
|
||||||
Goal: improve maintainability after shared policy and mechanics are already centralized.
|
|
||||||
|
|
||||||
Implementation decisions:
|
|
||||||
|
|
||||||
- Split the large operator helper implementation by command or responsibility.
|
|
||||||
- Suggested file grouping:
|
|
||||||
- session init;
|
|
||||||
- session validate;
|
|
||||||
- status;
|
|
||||||
- artifacts list;
|
|
||||||
- locks;
|
|
||||||
- helper findings;
|
|
||||||
- helper artifact rendering.
|
|
||||||
- Do not change command syntax, text output, config loading, remote loading, lock behavior, or artifact catalog behavior during the split.
|
|
||||||
- Keep output formatting text-only and command-specific unless a concrete inconsistency remains after the split.
|
|
||||||
- Update roadmap status notes after each completed stage.
|
|
||||||
|
|
||||||
Tests and checks:
|
|
||||||
|
|
||||||
- Run:
|
|
||||||
- `go test ./internal/app -v`
|
|
||||||
- `go test ./...`
|
|
||||||
- Final searches:
|
|
||||||
- `rg -n "archive|promote|promoted|promotion" internal docs examples cmd`
|
|
||||||
- `rg -n "narratio.transcript.merged|narratio.transcript.full|narratio.transcript.trimmed" internal docs examples`
|
|
||||||
- `rg -n "previous_session_artifact|promote_artifacts|pipeline.archive" internal docs examples`
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- operator helper code is easier to navigate;
|
|
||||||
- stale implementation terminology is removed or intentionally documented;
|
|
||||||
- no behavior changes are introduced by file organization.
|
|
||||||
|
|
||||||
## Overall Validation
|
|
||||||
|
|
||||||
After each implementation stage:
|
|
||||||
|
|
||||||
- run the focused tests listed for that stage;
|
|
||||||
- run `go test ./...`;
|
|
||||||
- run `git status --short`;
|
|
||||||
- update this roadmap to mark the completed stage implemented only after code, tests, and documentation are aligned.
|
|
||||||
|
|
||||||
## Assumptions
|
|
||||||
|
|
||||||
- This roadmap is a cleanup plan, not a feature plan.
|
|
||||||
- Stages may be implemented as separate prompts/commits.
|
|
||||||
- `internal/artifactpolicy` is the chosen home for shared source policy.
|
|
||||||
- Shared object-store temp download helpers must not learn Narratio session semantics.
|
|
||||||
- Public behavior must remain stable unless a stage explicitly says it is internal terminology cleanup.
|
|
||||||
@@ -1,105 +0,0 @@
|
|||||||
# Documentation Pass: Stage 1 Audit
|
|
||||||
|
|
||||||
Status: Completed (2026-05-23)
|
|
||||||
|
|
||||||
## Scope Reviewed
|
|
||||||
|
|
||||||
- All non-policy documentation files under `docs/`
|
|
||||||
- `README.md`
|
|
||||||
- Documentation references to maintained `examples/` files
|
|
||||||
- Documentation-related expectations in tests under `internal/**`
|
|
||||||
|
|
||||||
## File Inventory and Canonical Scope
|
|
||||||
|
|
||||||
| File | Intended audience | Canonical scope (per policy) | Primary source-of-truth anchors |
|
|
||||||
| --- | --- | --- | --- |
|
|
||||||
| `README.md` | Users, operators | Project orientation and links | `cmd/narratio`, `internal/app/commands.go`, docs index files |
|
|
||||||
| `docs/cli.md` | Users, operators | CLI syntax, flags, command workflows | `internal/app/*.go`, `internal/app/*_test.go` |
|
|
||||||
| `docs/config.md` | Operators, advanced users | Config discovery, schema, defaults, examples | `internal/config/*.go`, `internal/config/*_test.go`, `examples/*` |
|
|
||||||
| `docs/operations.md` | Operators | Run/resume/publish/restore/cleanup workflows | `internal/app/runner.go`, `internal/app/restore*.go`, `internal/stage/archive.go`, `internal/artifacts/*.go` |
|
|
||||||
| `docs/troubleshooting.md` | Operators | Failure diagnosis and safe fixes | `internal/app`, `internal/stage`, related tests |
|
|
||||||
| `docs/internal/README.md` | Developers, LLM coding agents | Internal docs index and scope boundaries | `docs/internal/*.md`, policy docs |
|
|
||||||
| `docs/internal/adapters.md` | Developers, LLM coding agents | Adapter boundaries and ownership | `internal/adapters/*`, `internal/stage/*` |
|
|
||||||
| `docs/internal/artifacts.md` | Developers, LLM coding agents | Artifact catalog and source resolution contracts | `internal/artifacts/*`, `internal/stage/analyze.go`, `internal/stage/prepare_previous.go` |
|
|
||||||
| `docs/internal/command-restore.md` | Developers, LLM coding agents | Restore command architecture and contracts | `internal/app/restore*.go`, `internal/app/restore*_test.go` |
|
|
||||||
| `docs/internal/manifest.md` | Developers, LLM coding agents | Session/run manifest contracts and transitions | `internal/manifest/*`, `internal/app/runner.go`, `internal/stage/*` |
|
|
||||||
| `docs/internal/stage-prepare.md` | Developers, LLM coding agents | Prepare stage IO and invariants | `internal/stage/prepare.go`, `internal/stage/prepare*_test.go` |
|
|
||||||
| `docs/internal/stage-transcribe.md` | Developers, LLM coding agents | Transcribe stage IO and invariants | `internal/stage/transcribe.go`, `internal/stage/transcribe_test.go` |
|
|
||||||
| `docs/internal/stage-merge.md` | Developers, LLM coding agents | Merge stage IO and invariants | `internal/stage/merge.go`, `internal/stage/merge_test.go` |
|
|
||||||
| `docs/internal/stage-polish.md` | Developers, LLM coding agents | Polish stage IO and invariants | `internal/stage/polish.go`, `internal/stage/polish_test.go` |
|
|
||||||
| `docs/internal/stage-normalize.md` | Developers, LLM coding agents | Normalize stage IO and invariants | `internal/stage/normalize.go`, `internal/stage/normalize_test.go` |
|
|
||||||
| `docs/internal/stage-trim.md` | Developers, LLM coding agents | Trim stage IO and invariants | `internal/stage/trim.go`, `internal/stage/trim_test.go` |
|
|
||||||
| `docs/internal/stage-analyze.md` | Developers, LLM coding agents | Analyze stage artifact execution and selection | `internal/stage/analyze.go`, `internal/stage/analyze_test.go` |
|
|
||||||
| `docs/internal/stage-publish.md` | Developers, LLM coding agents | Publish-stage commit/upload invariants | `internal/stage/archive.go`, `internal/stage/archive_test.go` |
|
|
||||||
| `docs/internal/storage.md` | Developers, LLM coding agents | Storage adapter contracts and semantics | `internal/adapters/storage/*`, `internal/app/object_store.go` |
|
|
||||||
| `docs/internal/workspace.md` | Developers, LLM coding agents | Local workspace/session/run path model | `internal/artifacts/*`, `internal/app/runner.go`, `internal/stage/run_local.go` |
|
|
||||||
| `docs/integrations/README.md` | Developers, LLM coding agents | Integration docs index | `docs/integrations/*.md` |
|
|
||||||
| `docs/integrations/audita.md` | Developers, integration maintainers | Audita adapter contract | `internal/adapters/audita/*`, `internal/stage/polish.go` |
|
|
||||||
| `docs/integrations/seriatim.md` | Developers, integration maintainers | Seriatim adapter contract | `internal/adapters/seriatim/*`, `internal/stage/merge.go`, `internal/stage/normalize.go`, `internal/stage/trim.go` |
|
|
||||||
| `docs/integrations/scriptorium.md` | Developers, integration maintainers | Scriptorium adapter contract | `internal/adapters/scriptorium/*`, `internal/stage/analyze.go`, `internal/stage/trim.go` |
|
|
||||||
| `docs/roadmap/documentation.md` | Developers, maintainers | Planning and implementation sequencing for documentation pass | N/A (planning artifact) |
|
|
||||||
| `docs/roadmap/documentation-stage1-audit.md` | Developers, maintainers | Stage-1 inventory and source-of-truth audit record | N/A (planning artifact) |
|
|
||||||
|
|
||||||
## Source-of-Truth Mapping Summary
|
|
||||||
|
|
||||||
- CLI behaviors and command names are grounded in `internal/app/commands.go` and command handlers in `internal/app/*.go`.
|
|
||||||
- Stage order and canonical stage names are grounded in `internal/stage/placeholders.go` (`prepare` -> `transcribe` -> `merge` -> `polish` -> `normalize` -> `trim` -> `analyze` -> `publish` -> `notify`).
|
|
||||||
- Publish behavior and current-pointer commit semantics are grounded in `internal/stage/archive.go`.
|
|
||||||
- Config schema/defaults/validation are grounded in `internal/config/*`.
|
|
||||||
- Local/remote paths, publish keys, and workspace layout are grounded in `internal/artifacts/*`.
|
|
||||||
- Restore behavior and report contracts are grounded in `internal/app/restore*.go`.
|
|
||||||
- Maintained examples and schema compatibility are grounded in `examples/*` plus `internal/config/load_validate_test.go` (`TestExamplesLoadAndValidate`).
|
|
||||||
|
|
||||||
## Findings
|
|
||||||
|
|
||||||
### Broken or stale references
|
|
||||||
|
|
||||||
1. `README.md` linked to non-existent files:
|
|
||||||
- `docs/development.md`
|
|
||||||
- `docs/architecture.md`
|
|
||||||
2. `docs/internal/README.md` and `docs/integrations/README.md` linked to non-existent path:
|
|
||||||
- `docs/documentation/policy.md`
|
|
||||||
|
|
||||||
Stage-1 fix applied:
|
|
||||||
- Updated those links to existing policy docs under `docs/policy/`.
|
|
||||||
|
|
||||||
### Stale terminology sweep
|
|
||||||
|
|
||||||
Sweep terms used: `archive`, `promote`, `promoted`, `promote_artifacts`, `run-stage archive`.
|
|
||||||
|
|
||||||
Findings:
|
|
||||||
- User-facing docs in scope did not show obvious stale command examples requiring immediate correction.
|
|
||||||
- Internal code and tests still contain historical `archive` identifiers while user-facing command/stage naming is `publish` (for example, `internal/stage/archive.go` type names). This is acceptable for now but should be normalized deliberately, not incidentally.
|
|
||||||
|
|
||||||
Stage-1 fix applied:
|
|
||||||
- Updated clearly stale publish-related wording in test expectation messages/comments:
|
|
||||||
- `internal/app/commands_test.go`
|
|
||||||
- `internal/app/operator_helpers_test.go`
|
|
||||||
|
|
||||||
### Example path validation
|
|
||||||
|
|
||||||
- All `examples/...` paths referenced from non-policy docs resolve to existing files.
|
|
||||||
- `internal/config/load_validate_test.go` includes `TestExamplesLoadAndValidate` and points to current example files.
|
|
||||||
|
|
||||||
### Roadmap leakage into current-behavior docs
|
|
||||||
|
|
||||||
- No obvious roadmap-only behavior leakage found in non-roadmap docs during this sweep.
|
|
||||||
|
|
||||||
### Duplicate content and scope drift
|
|
||||||
|
|
||||||
- No severe duplication requiring immediate rewrite in this stage.
|
|
||||||
- Existing docs still need full content rewrite for 1.0 readiness in later stages (user/operator first, then internal/integrations), as planned.
|
|
||||||
|
|
||||||
### Canonical-home inconsistency to resolve in rewrite stages
|
|
||||||
|
|
||||||
- Policy canonical-home language names `docs/architecture.md` and `docs/development.md`, while current repository stores those policy documents under `docs/policy/`.
|
|
||||||
- Stage 1 preserves repository behavior by fixing broken links to existing files. Later rewrite stages should converge canonical-home paths and references consistently across docs.
|
|
||||||
|
|
||||||
## Stage-1 Completion Check
|
|
||||||
|
|
||||||
Completed for this stage:
|
|
||||||
- Full non-policy file inventory with audience and scope mapping.
|
|
||||||
- Source-of-truth crosswalk to code/tests.
|
|
||||||
- Stale-term, link, and example-path sweeps.
|
|
||||||
- Documentation-related stale test wording corrections.
|
|
||||||
- Minimal fixes only; broad rewrites intentionally deferred.
|
|
||||||
@@ -1,237 +0,0 @@
|
|||||||
# Roadmap: 1.0 Documentation Pass
|
|
||||||
|
|
||||||
Status: Completed (2026-05-23)
|
|
||||||
|
|
||||||
## Goal
|
|
||||||
|
|
||||||
Prepare Narratio documentation for a 1.0 release by reviewing and rewriting
|
|
||||||
every non-policy document under `docs/` against the implemented codebase.
|
|
||||||
|
|
||||||
The finished documentation set should be accurate, concise, complete for its
|
|
||||||
audience, and compliant with:
|
|
||||||
|
|
||||||
- `docs/policy/documentation.md`
|
|
||||||
- `docs/policy/architecture.md`
|
|
||||||
- `docs/policy/development.md`
|
|
||||||
|
|
||||||
Do not modify files under `docs/policy/` during this pass.
|
|
||||||
|
|
||||||
Current behavior belongs in canonical docs. Future, planned, aspirational, or
|
|
||||||
unimplemented behavior belongs only under `docs/roadmap/`.
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
|
|
||||||
In scope:
|
|
||||||
|
|
||||||
- `docs/*.md`
|
|
||||||
- `docs/internal/*.md`
|
|
||||||
- `docs/integrations/*.md`
|
|
||||||
- `docs/roadmap/*.md`
|
|
||||||
- documentation references to files under `examples/`
|
|
||||||
- test expectation fixes when the documentation review exposes stale or
|
|
||||||
incorrect doc/example/path expectations
|
|
||||||
|
|
||||||
Out of scope:
|
|
||||||
|
|
||||||
- product/runtime code changes
|
|
||||||
- feature implementation
|
|
||||||
- edits under `docs/policy/`
|
|
||||||
- adding roadmap behavior to current-behavior docs before that behavior is
|
|
||||||
implemented
|
|
||||||
|
|
||||||
## Implementation Stages
|
|
||||||
|
|
||||||
### Stage 1: Inventory and Source-of-Truth Audit
|
|
||||||
|
|
||||||
Status: Completed (2026-05-23)
|
|
||||||
|
|
||||||
Create a file-by-file inventory of all non-policy docs before rewriting.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- List every non-policy documentation file and assign its intended audience.
|
|
||||||
- Identify each document's canonical scope using `docs/policy/documentation.md`.
|
|
||||||
- Compare docs against the current code and tests, especially:
|
|
||||||
- `internal/app`
|
|
||||||
- `internal/config`
|
|
||||||
- `internal/stage`
|
|
||||||
- `internal/artifacts`
|
|
||||||
- `examples`
|
|
||||||
- relevant tests under `internal/**`
|
|
||||||
- Record stale terminology, broken links, stale example paths, duplicate
|
|
||||||
content, and roadmap-only behavior that leaked into current-behavior docs.
|
|
||||||
- Record stale test expectations related to docs, examples, paths, or command
|
|
||||||
text.
|
|
||||||
- Do not rewrite content in this stage except obvious broken links or test
|
|
||||||
corrections needed to make documentation validation meaningful.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- The rewrite has a concrete file inventory and source-of-truth map.
|
|
||||||
- The team knows which docs are canonical and which should link elsewhere.
|
|
||||||
- Known stale terms and broken references are identified before broad edits.
|
|
||||||
|
|
||||||
### Stage 2: User and Operator Docs
|
|
||||||
|
|
||||||
Status: Completed (2026-05-23)
|
|
||||||
|
|
||||||
Rewrite the user-facing and operator-facing docs first.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Rewrite these docs as fresh, concise current-behavior references:
|
|
||||||
- `README.md`, if present
|
|
||||||
- `docs/cli.md`
|
|
||||||
- `docs/config.md`
|
|
||||||
- `docs/operations.md`
|
|
||||||
- `docs/troubleshooting.md`
|
|
||||||
- Verify every command, flag, config field, discovery rule, path, and workflow
|
|
||||||
against implemented behavior.
|
|
||||||
- Cover implemented 1.0 behavior, including:
|
|
||||||
- campaign registry selection;
|
|
||||||
- concrete session loading and template-driven `session init`;
|
|
||||||
- session-oriented helper commands;
|
|
||||||
- clean, restore, analyze, and publish workflows;
|
|
||||||
- artifact selection behavior;
|
|
||||||
- locks and published output behavior;
|
|
||||||
- workspace, spool, and cache behavior;
|
|
||||||
- secrets loading and S3-backed operation.
|
|
||||||
- Keep examples short and link to maintained files under `examples/` instead
|
|
||||||
of duplicating large config blocks.
|
|
||||||
- Fix tests only when they assert stale doc paths, example paths, command
|
|
||||||
names, or current-behavior text.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- User/operator docs are task-oriented and match actual CLI/config behavior.
|
|
||||||
- Current-behavior docs do not depend on roadmaps for normal usage.
|
|
||||||
- No current-behavior doc describes unimplemented roadmap items.
|
|
||||||
|
|
||||||
### Stage 3: Internal Developer Docs
|
|
||||||
|
|
||||||
Status: Completed (2026-05-23)
|
|
||||||
|
|
||||||
Rewrite implemented internal component docs after public docs stabilize.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Rewrite:
|
|
||||||
- `docs/internal/README.md`
|
|
||||||
- `docs/internal/adapters.md`
|
|
||||||
- `docs/internal/artifacts.md`
|
|
||||||
- `docs/internal/command-restore.md`
|
|
||||||
- `docs/internal/manifest.md`
|
|
||||||
- `docs/internal/stage-*.md`
|
|
||||||
- `docs/internal/storage.md`
|
|
||||||
- `docs/internal/workspace.md`
|
|
||||||
- Verify stage docs against current stage names, stage ordering, manifest
|
|
||||||
records, declared inputs/outputs, adapters, path helpers, storage behavior,
|
|
||||||
publish/current-state behavior, restore behavior, cache behavior, and
|
|
||||||
workspace cleanup.
|
|
||||||
- Keep implementation details in `docs/internal/`, not in user-facing docs.
|
|
||||||
- Avoid turning internal docs into duplicate config or CLI references; link to
|
|
||||||
canonical docs when needed.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Internal docs are accurate enough for developers and LLM coding agents to
|
|
||||||
change the system safely.
|
|
||||||
- Stage and adapter boundaries match `docs/policy/architecture.md`.
|
|
||||||
- Manifest, path, storage, and publish invariants are explicit and current.
|
|
||||||
|
|
||||||
### Stage 4: Integrations and Examples
|
|
||||||
|
|
||||||
Status: Completed (2026-05-23)
|
|
||||||
|
|
||||||
Review integration docs and maintained examples after core docs are rewritten.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Rewrite:
|
|
||||||
- `docs/integrations/README.md`
|
|
||||||
- `docs/integrations/audita.md`
|
|
||||||
- `docs/integrations/scriptorium.md`
|
|
||||||
- `docs/integrations/seriatim.md`
|
|
||||||
- Verify integration docs against current adapter contracts and expected
|
|
||||||
downstream tool behavior.
|
|
||||||
- Confirm every referenced example file exists.
|
|
||||||
- Confirm examples match current schema and command usage.
|
|
||||||
- Run or rely on example validation tests.
|
|
||||||
- Fix tests when they reference moved, renamed, or intentionally retired
|
|
||||||
examples.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Integration docs describe only implemented adapter expectations.
|
|
||||||
- Maintained examples are valid, secret-free, and linked from canonical docs.
|
|
||||||
- Example validation tests reflect the documented example set.
|
|
||||||
|
|
||||||
### Stage 5: Roadmap Cleanup and Final Sweep
|
|
||||||
|
|
||||||
Status: Completed (2026-05-23)
|
|
||||||
|
|
||||||
Clean up roadmap state and run final documentation validation.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Review `docs/roadmap/**` for implemented items that should be marked
|
|
||||||
implemented, retired, or left planned.
|
|
||||||
- Keep historical and planned behavior in roadmaps only.
|
|
||||||
- Run final link/path/term sweeps.
|
|
||||||
- Run validation commands:
|
|
||||||
- `go test ./internal/config -run TestExamplesLoadAndValidate -v`
|
|
||||||
- `go test ./internal/app -run TestExecute -v`
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- All non-policy docs are current for the 1.0 release.
|
|
||||||
- Roadmaps do not serve as required user/operator documentation.
|
|
||||||
- Tests pass after allowed documentation-related test expectation fixes.
|
|
||||||
|
|
||||||
## Required Checks
|
|
||||||
|
|
||||||
Run searches for stale terminology and references during the pass.
|
|
||||||
|
|
||||||
Stale terminology:
|
|
||||||
|
|
||||||
- `archive`
|
|
||||||
- `promote`
|
|
||||||
- `promoted`
|
|
||||||
- `promote_artifacts`
|
|
||||||
- legacy campaign path/discovery language
|
|
||||||
- old transcript names and paths
|
|
||||||
- removed CLI commands or aliases
|
|
||||||
|
|
||||||
Broken or stale references:
|
|
||||||
|
|
||||||
- missing local doc links;
|
|
||||||
- stale `examples/` paths;
|
|
||||||
- stale internal doc filenames;
|
|
||||||
- references to `docs/policy/**` as editable targets;
|
|
||||||
- command examples that no longer match the CLI.
|
|
||||||
|
|
||||||
Policy checks:
|
|
||||||
|
|
||||||
- Current-behavior docs mention only implemented behavior.
|
|
||||||
- Planned behavior appears only under `docs/roadmap/`.
|
|
||||||
- Docs do not expose raw secrets or recommend storing secrets in config.
|
|
||||||
- Docs use canonical homes:
|
|
||||||
- `docs/config.md` for config schema;
|
|
||||||
- `docs/cli.md` for command syntax;
|
|
||||||
- `docs/operations.md` for operator workflows;
|
|
||||||
- `docs/troubleshooting.md` for failure diagnosis;
|
|
||||||
- `docs/internal/` for implementation contracts;
|
|
||||||
- `docs/integrations/` for downstream tool integration notes;
|
|
||||||
- `docs/roadmap/` for future work.
|
|
||||||
|
|
||||||
## Assumptions
|
|
||||||
|
|
||||||
- `docs/policy/**` is read-only for this documentation pass.
|
|
||||||
- This pass is for 1.0 release readiness, not feature implementation.
|
|
||||||
- Product and runtime code changes are out of scope.
|
|
||||||
- Test fixes are in scope when they correct stale documentation, example, path,
|
|
||||||
command, or current-behavior expectations uncovered during the review.
|
|
||||||
- Roadmap files may remain as planning and historical records.
|
|
||||||
- Current-behavior docs must be sufficient for normal use without requiring
|
|
||||||
readers to consult roadmaps.
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
// Package storage declares archive/storage backend adapter boundaries.
|
|
||||||
package storage
|
|
||||||
|
|
||||||
import "context"
|
|
||||||
|
|
||||||
// TODO: implement remote storage/archive backends (S3/SFTP/etc.).
|
|
||||||
|
|
||||||
// Backend is the adapter boundary for archive/storage operations.
|
|
||||||
type Backend interface {
|
|
||||||
Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ArchiveItem describes one item to archive.
|
|
||||||
type ArchiveItem struct {
|
|
||||||
Kind string
|
|
||||||
LocalPath string
|
|
||||||
RemoteKey string
|
|
||||||
}
|
|
||||||
|
|
||||||
// ArchiveRequest describes one archive operation.
|
|
||||||
type ArchiveRequest struct {
|
|
||||||
SessionID string
|
|
||||||
ManifestPath string
|
|
||||||
Items []ArchiveItem
|
|
||||||
}
|
|
||||||
|
|
||||||
// ArchiveResult describes archive operation output.
|
|
||||||
type ArchiveResult struct {
|
|
||||||
Archived []ArchiveItem
|
|
||||||
Metadata map[string]any
|
|
||||||
}
|
|
||||||
@@ -10,23 +10,8 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// NoopBackend is a deterministic no-op archive/storage adapter.
|
// FakeBackend provides a deterministic in-memory object store for tests.
|
||||||
type NoopBackend struct{}
|
|
||||||
|
|
||||||
// Archive returns the requested items as archived with placeholder metadata.
|
|
||||||
func (n *NoopBackend) Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error) {
|
|
||||||
if err := ctx.Err(); err != nil {
|
|
||||||
return ArchiveResult{}, err
|
|
||||||
}
|
|
||||||
return ArchiveResult{Archived: append([]ArchiveItem(nil), req.Items...), Metadata: map[string]any{"placeholder": true}}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// FakeBackend captures archive requests and returns deterministic responses.
|
|
||||||
type FakeBackend struct {
|
type FakeBackend struct {
|
||||||
Requests []ArchiveRequest
|
|
||||||
Err error
|
|
||||||
Result ArchiveResult
|
|
||||||
|
|
||||||
Objects map[string]FakeObject
|
Objects map[string]FakeObject
|
||||||
Uploads []FakeUploadCall
|
Uploads []FakeUploadCall
|
||||||
Downloads []FakeDownloadCall
|
Downloads []FakeDownloadCall
|
||||||
@@ -50,25 +35,6 @@ type FakeDownloadCall struct {
|
|||||||
LocalPath string
|
LocalPath string
|
||||||
}
|
}
|
||||||
|
|
||||||
// Archive records request and returns configured response.
|
|
||||||
func (f *FakeBackend) Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error) {
|
|
||||||
if err := ctx.Err(); err != nil {
|
|
||||||
return ArchiveResult{}, err
|
|
||||||
}
|
|
||||||
f.Requests = append(f.Requests, req)
|
|
||||||
if f.Err != nil {
|
|
||||||
return ArchiveResult{}, f.Err
|
|
||||||
}
|
|
||||||
res := f.Result
|
|
||||||
if res.Archived == nil {
|
|
||||||
res.Archived = append([]ArchiveItem(nil), req.Items...)
|
|
||||||
}
|
|
||||||
if res.Metadata == nil {
|
|
||||||
res.Metadata = map[string]any{"fake": true}
|
|
||||||
}
|
|
||||||
return res, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// FakeObject is a deterministic fake object-store record.
|
// FakeObject is a deterministic fake object-store record.
|
||||||
type FakeObject struct {
|
type FakeObject struct {
|
||||||
Key string
|
Key string
|
||||||
|
|||||||
@@ -9,30 +9,6 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestFakeBackendCapturesRequestAndReturnsItems(t *testing.T) {
|
|
||||||
fake := &FakeBackend{}
|
|
||||||
req := ArchiveRequest{SessionID: "s1", Items: []ArchiveItem{{Kind: "artifact", LocalPath: "artifacts/log.md"}}}
|
|
||||||
|
|
||||||
res, err := fake.Archive(context.Background(), req)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Archive() error = %v", err)
|
|
||||||
}
|
|
||||||
if len(fake.Requests) != 1 || fake.Requests[0].SessionID != "s1" {
|
|
||||||
t.Fatalf("requests = %#v, want captured request", fake.Requests)
|
|
||||||
}
|
|
||||||
if len(res.Archived) != 1 {
|
|
||||||
t.Fatalf("archived len = %d, want 1", len(res.Archived))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFakeBackendError(t *testing.T) {
|
|
||||||
fake := &FakeBackend{Err: errors.New("boom")}
|
|
||||||
_, err := fake.Archive(context.Background(), ArchiveRequest{})
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("expected error, got nil")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFakeBackendListPrefixFiltering(t *testing.T) {
|
func TestFakeBackendListPrefixFiltering(t *testing.T) {
|
||||||
fake := &FakeBackend{}
|
fake := &FakeBackend{}
|
||||||
fake.SeedObject(FakeObject{Key: "dnd/campaigns/forsaken/audio/a.flac", Data: []byte("a")})
|
fake.SeedObject(FakeObject{Key: "dnd/campaigns/forsaken/audio/a.flac", Data: []byte("a")})
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ObjectStore is a remote object storage boundary used by future prepare/archive work.
|
// ObjectStore is a remote object storage boundary used by prepare, restore, and publish work.
|
||||||
//
|
//
|
||||||
// Key invariant:
|
// Key invariant:
|
||||||
// callers pass full bucket-relative object keys. Backend implementations do not
|
// callers pass full bucket-relative object keys. Backend implementations do not
|
||||||
|
|||||||
36
internal/adapters/storage/temp_download.go
Normal file
36
internal/adapters/storage/temp_download.go
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
package storage
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// DownloadObjectToTemp downloads an object into a temporary file and returns
|
||||||
|
// the cleaned local path.
|
||||||
|
func DownloadObjectToTemp(ctx context.Context, store ObjectStore, key, pattern string) (string, error) {
|
||||||
|
if store == nil {
|
||||||
|
return "", fmt.Errorf("object store is required")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(pattern) == "" {
|
||||||
|
return "", fmt.Errorf("temp file pattern is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
tmp, err := os.CreateTemp("", pattern)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("create temp file: %w", err)
|
||||||
|
}
|
||||||
|
path := tmp.Name()
|
||||||
|
if err := tmp.Close(); err != nil {
|
||||||
|
_ = os.Remove(path)
|
||||||
|
return "", fmt.Errorf("close temp file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := store.Download(ctx, key, path); err != nil {
|
||||||
|
_ = os.Remove(path)
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return filepath.Clean(path), nil
|
||||||
|
}
|
||||||
69
internal/adapters/storage/temp_download_test.go
Normal file
69
internal/adapters/storage/temp_download_test.go
Normal file
@@ -0,0 +1,69 @@
|
|||||||
|
package storage
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestDownloadObjectToTempSuccess(t *testing.T) {
|
||||||
|
store := &FakeBackend{}
|
||||||
|
store.SeedObject(FakeObject{Key: "sessions/a/current/run_id.txt", Data: []byte("run-123\n")})
|
||||||
|
|
||||||
|
path, err := DownloadObjectToTemp(context.Background(), store, "sessions/a/current/run_id.txt", "narratio-test-*.txt")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DownloadObjectToTemp() error = %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { _ = os.Remove(path) })
|
||||||
|
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile() error = %v", err)
|
||||||
|
}
|
||||||
|
if string(data) != "run-123\n" {
|
||||||
|
t.Fatalf("downloaded data = %q, want %q", string(data), "run-123\n")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDownloadObjectToTempFailedDownloadRemovesTempFile(t *testing.T) {
|
||||||
|
sentinel := errors.New("download failed")
|
||||||
|
store := &FakeBackend{DownloadErr: sentinel}
|
||||||
|
pattern := "narratio-test-fail-*.txt"
|
||||||
|
before, err := filepath.Glob(filepath.Join(os.TempDir(), "narratio-test-fail-*.txt"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Glob(before) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
path, err := DownloadObjectToTemp(context.Background(), store, "sessions/a/current/run_id.txt", pattern)
|
||||||
|
if !errors.Is(err, sentinel) {
|
||||||
|
t.Fatalf("DownloadObjectToTemp() error = %v, want %v", err, sentinel)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(path) != "" {
|
||||||
|
t.Fatalf("DownloadObjectToTemp() path = %q, want empty on failure", path)
|
||||||
|
}
|
||||||
|
after, err := filepath.Glob(filepath.Join(os.TempDir(), "narratio-test-fail-*.txt"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Glob(after) error = %v", err)
|
||||||
|
}
|
||||||
|
if len(after) != len(before) {
|
||||||
|
t.Fatalf("temp file count changed after failed download: before=%d after=%d", len(before), len(after))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDownloadObjectToTempCallerContextWrappingPreservesCause(t *testing.T) {
|
||||||
|
sentinel := errors.New("object missing")
|
||||||
|
store := &FakeBackend{DownloadErr: sentinel}
|
||||||
|
|
||||||
|
_, err := DownloadObjectToTemp(context.Background(), store, "sessions/a/current/run_id.txt", "narratio-test-*.txt")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("DownloadObjectToTemp() error = nil, want error")
|
||||||
|
}
|
||||||
|
err = fmt.Errorf("download run pointer failed: %w", err)
|
||||||
|
if !errors.Is(err, sentinel) {
|
||||||
|
t.Fatalf("wrapped error does not preserve sentinel cause: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -145,7 +145,7 @@ func TestHTTPClientDoesNotRetryOnNonRetryableStatus(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestHTTPClientInvalidJSONFailsAndDoesNotPromote(t *testing.T) {
|
func TestHTTPClientInvalidJSONFailsAndDoesNotInstallOutput(t *testing.T) {
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
_, _ = w.Write([]byte(`not-json`))
|
_, _ = w.Write([]byte(`not-json`))
|
||||||
}))
|
}))
|
||||||
|
|||||||
@@ -33,7 +33,7 @@ func TestExecuteRunStageArtifactsUnsupportedStageFails(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteRunStageArchivePropagatesSelectedArtifacts(t *testing.T) {
|
func TestExecuteRunStagePublishPropagatesSelectedArtifacts(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
|
|
||||||
@@ -120,7 +120,7 @@ func TestRunStageArtifactsDoesNotImplyForce(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResumeArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
|
func TestRunArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||||
@@ -135,16 +135,16 @@ func TestResumeArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
err := Resume(
|
err := Run(
|
||||||
context.Background(),
|
context.Background(),
|
||||||
[]string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
[]string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
||||||
&out,
|
&out,
|
||||||
)
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "has no remaining stages") {
|
if !strings.Contains(out.String(), "executed=0 skipped=9") {
|
||||||
t.Fatalf("output = %q, want no remaining stages", out.String())
|
t.Fatalf("output = %q, want all stages skipped", out.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -281,7 +281,7 @@ func TestExecuteAnalyzeMissingConfigUsesRunStageLoadingPath(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecutePublishForceRunsArchive(t *testing.T) {
|
func TestExecutePublishForceRunsPublish(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
|
|
||||||
|
|||||||
@@ -16,7 +16,6 @@ import (
|
|||||||
// Clean removes local workspace/spool state while preserving durable cache
|
// Clean removes local workspace/spool state while preserving durable cache
|
||||||
// state unless cache cleanup is explicitly requested.
|
// state unless cache cleanup is explicitly requested.
|
||||||
func Clean(ctx context.Context, args []string, out io.Writer) error {
|
func Clean(ctx context.Context, args []string, out io.Writer) error {
|
||||||
positionalSessionID, args := pullLeadingSessionID(args)
|
|
||||||
fs := flag.NewFlagSet("clean", flag.ContinueOnError)
|
fs := flag.NewFlagSet("clean", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
var flags commonConfigFlags
|
var flags commonConfigFlags
|
||||||
@@ -27,20 +26,8 @@ func Clean(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
fs.BoolVar(&all, "all", false, "clean all local session work/spool state")
|
fs.BoolVar(&all, "all", false, "clean all local session work/spool state")
|
||||||
fs.BoolVar(&dryRun, "dry-run", false, "print cleanup targets without deleting")
|
fs.BoolVar(&dryRun, "dry-run", false, "print cleanup targets without deleting")
|
||||||
fs.BoolVar(&clearCache, "clear-cache", false, "also clear durable S3 audio cache entries")
|
fs.BoolVar(&clearCache, "clear-cache", false, "also clear durable S3 audio cache entries")
|
||||||
if err := fs.Parse(args); err != nil {
|
if err := parseSessionAwareFlags("clean", fs, args, &flags.sessionID); err != nil {
|
||||||
return fmt.Errorf("clean: invalid flags: %w", err)
|
return err
|
||||||
}
|
|
||||||
if positionalSessionID == "" {
|
|
||||||
if err := applyParsedSessionIDArg("clean", fs, &flags.sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("clean: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("clean", positionalSessionID, &flags.sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if all {
|
if all {
|
||||||
return cleanAllLocal(flags, dryRun, clearCache, out)
|
return cleanAllLocal(flags, dryRun, clearCache, out)
|
||||||
@@ -182,26 +169,12 @@ func reportCleanRootChildren(out io.Writer, root, policy string, dryRun bool) er
|
|||||||
}
|
}
|
||||||
|
|
||||||
func cleanableRootChildren(root, policy string) (string, []string, error) {
|
func cleanableRootChildren(root, policy string) (string, []string, error) {
|
||||||
cleanRoot := strings.TrimSpace(root)
|
rootAbs, exists, err := validateCleanRoot(root, policy)
|
||||||
if cleanRoot == "" {
|
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
|
||||||
}
|
|
||||||
rootAbs, err := filepath.Abs(cleanRoot)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
return "", nil, err
|
||||||
}
|
}
|
||||||
info, err := os.Lstat(rootAbs)
|
if !exists {
|
||||||
if err != nil {
|
return rootAbs, nil, nil
|
||||||
if os.IsNotExist(err) {
|
|
||||||
return rootAbs, nil, nil
|
|
||||||
}
|
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: stat root %q: %w", policy, rootAbs, err)
|
|
||||||
}
|
|
||||||
if info.Mode()&os.ModeSymlink != 0 {
|
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: refusing to clean symlink root %q", policy, rootAbs)
|
|
||||||
}
|
|
||||||
if !info.IsDir() {
|
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: root %q is not a directory", policy, rootAbs)
|
|
||||||
}
|
}
|
||||||
entries, err := os.ReadDir(rootAbs)
|
entries, err := os.ReadDir(rootAbs)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -300,46 +273,7 @@ func reportCleanScopedFile(out io.Writer, root, target, policy string, dryRun bo
|
|||||||
}
|
}
|
||||||
|
|
||||||
func validateScopedFile(root, target, policy string) (scopedDir, error) {
|
func validateScopedFile(root, target, policy string) (scopedDir, error) {
|
||||||
cleanRoot := strings.TrimSpace(root)
|
return validateScopedTarget(root, target, policy, false)
|
||||||
cleanTarget := strings.TrimSpace(target)
|
|
||||||
if cleanRoot == "" {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
|
||||||
}
|
|
||||||
if cleanTarget == "" {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target path is required", policy)
|
|
||||||
}
|
|
||||||
rootAbs, err := filepath.Abs(cleanRoot)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
|
||||||
}
|
|
||||||
targetAbs, err := filepath.Abs(cleanTarget)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve target %q: %w", policy, cleanTarget, err)
|
|
||||||
}
|
|
||||||
rel, err := filepath.Rel(rootAbs, targetAbs)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: relative path from %q to %q: %w", policy, rootAbs, targetAbs, err)
|
|
||||||
}
|
|
||||||
if rel == "." {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete root directory %q", policy, rootAbs)
|
|
||||||
}
|
|
||||||
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete path outside root: root=%q target=%q", policy, rootAbs, targetAbs)
|
|
||||||
}
|
|
||||||
info, err := os.Lstat(targetAbs)
|
|
||||||
if err != nil {
|
|
||||||
if os.IsNotExist(err) {
|
|
||||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: false}, nil
|
|
||||||
}
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: stat target %q: %w", policy, targetAbs, err)
|
|
||||||
}
|
|
||||||
if info.Mode()&os.ModeSymlink != 0 {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, targetAbs)
|
|
||||||
}
|
|
||||||
if info.IsDir() {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is a directory", policy, targetAbs)
|
|
||||||
}
|
|
||||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: true}, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func cleanIsFlac(path string) bool {
|
func cleanIsFlac(path string) bool {
|
||||||
|
|||||||
82
internal/app/cleanup_targets.go
Normal file
82
internal/app/cleanup_targets.go
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
func validateScopedTarget(root, target, policy string, requireDir bool) (scopedDir, error) {
|
||||||
|
cleanRoot := strings.TrimSpace(root)
|
||||||
|
cleanTarget := strings.TrimSpace(target)
|
||||||
|
if cleanRoot == "" {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
||||||
|
}
|
||||||
|
if cleanTarget == "" {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: target path is required", policy)
|
||||||
|
}
|
||||||
|
|
||||||
|
rootAbs, err := filepath.Abs(cleanRoot)
|
||||||
|
if err != nil {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
||||||
|
}
|
||||||
|
targetAbs, err := filepath.Abs(cleanTarget)
|
||||||
|
if err != nil {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve target %q: %w", policy, cleanTarget, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
rel, err := filepath.Rel(rootAbs, targetAbs)
|
||||||
|
if err != nil {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: relative path from %q to %q: %w", policy, rootAbs, targetAbs, err)
|
||||||
|
}
|
||||||
|
if rel == "." {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete root directory %q", policy, rootAbs)
|
||||||
|
}
|
||||||
|
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete path outside root: root=%q target=%q", policy, rootAbs, targetAbs)
|
||||||
|
}
|
||||||
|
|
||||||
|
info, err := os.Lstat(targetAbs)
|
||||||
|
if err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: false}, nil
|
||||||
|
}
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: stat target %q: %w", policy, targetAbs, err)
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, targetAbs)
|
||||||
|
}
|
||||||
|
if requireDir && !info.IsDir() {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is not a directory", policy, targetAbs)
|
||||||
|
}
|
||||||
|
if !requireDir && info.IsDir() {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is a directory", policy, targetAbs)
|
||||||
|
}
|
||||||
|
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: true}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateCleanRoot(root, policy string) (string, bool, error) {
|
||||||
|
cleanRoot := strings.TrimSpace(root)
|
||||||
|
if cleanRoot == "" {
|
||||||
|
return "", false, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
||||||
|
}
|
||||||
|
rootAbs, err := filepath.Abs(cleanRoot)
|
||||||
|
if err != nil {
|
||||||
|
return "", false, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
||||||
|
}
|
||||||
|
info, err := os.Lstat(rootAbs)
|
||||||
|
if err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return rootAbs, false, nil
|
||||||
|
}
|
||||||
|
return "", false, fmt.Errorf("cleanup policy %s: stat root %q: %w", policy, rootAbs, err)
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 {
|
||||||
|
return "", false, fmt.Errorf("cleanup policy %s: refusing to clean symlink root %q", policy, rootAbs)
|
||||||
|
}
|
||||||
|
if !info.IsDir() {
|
||||||
|
return "", false, fmt.Errorf("cleanup policy %s: root %q is not a directory", policy, rootAbs)
|
||||||
|
}
|
||||||
|
return rootAbs, true, nil
|
||||||
|
}
|
||||||
103
internal/app/cleanup_targets_test.go
Normal file
103
internal/app/cleanup_targets_test.go
Normal file
@@ -0,0 +1,103 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestCleanValidateScopedDirAndFile(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
dirTarget := filepath.Join(root, "runs", "run-1")
|
||||||
|
fileTarget := filepath.Join(root, "cache", "a.flac")
|
||||||
|
if err := os.MkdirAll(dirTarget, 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(dirTarget) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.MkdirAll(filepath.Dir(fileTarget), 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(file parent) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(fileTarget, []byte("audio"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(fileTarget) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := validateScopedDir(root, dirTarget, "test.dir"); err != nil {
|
||||||
|
t.Fatalf("validateScopedDir() error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := validateScopedFile(root, fileTarget, "test.file"); err != nil {
|
||||||
|
t.Fatalf("validateScopedFile() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCleanValidateScopedTargetSafetyRules(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
outside := t.TempDir()
|
||||||
|
target := filepath.Join(root, "runs", "run-1")
|
||||||
|
if err := os.MkdirAll(target, 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(target) error = %v", err)
|
||||||
|
}
|
||||||
|
fileTarget := filepath.Join(root, "cache", "a.flac")
|
||||||
|
if err := os.MkdirAll(filepath.Dir(fileTarget), 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(file parent) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(fileTarget, []byte("audio"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(fileTarget) error = %v", err)
|
||||||
|
}
|
||||||
|
symlinkTarget := filepath.Join(root, "symlink")
|
||||||
|
if err := os.Symlink(target, symlinkTarget); err != nil {
|
||||||
|
t.Fatalf("Symlink() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := validateScopedDir(root, root, "test.root"); err == nil || !strings.Contains(err.Error(), "refusing to delete root directory") {
|
||||||
|
t.Fatalf("validateScopedDir(root) error = %v, want root deletion rejection", err)
|
||||||
|
}
|
||||||
|
if _, err := validateScopedDir(root, filepath.Join(outside, "x"), "test.outside"); err == nil || !strings.Contains(err.Error(), "outside root") {
|
||||||
|
t.Fatalf("validateScopedDir(outside) error = %v, want outside-root rejection", err)
|
||||||
|
}
|
||||||
|
if _, err := validateScopedDir(root, fileTarget, "test.file-as-dir"); err == nil || !strings.Contains(err.Error(), "is not a directory") {
|
||||||
|
t.Fatalf("validateScopedDir(file) error = %v, want not-a-directory rejection", err)
|
||||||
|
}
|
||||||
|
if _, err := validateScopedFile(root, target, "test.dir-as-file"); err == nil || !strings.Contains(err.Error(), "is a directory") {
|
||||||
|
t.Fatalf("validateScopedFile(dir) error = %v, want is-a-directory rejection", err)
|
||||||
|
}
|
||||||
|
if _, err := validateScopedDir(root, symlinkTarget, "test.symlink"); err == nil || !strings.Contains(err.Error(), "refusing to delete symlink path") {
|
||||||
|
t.Fatalf("validateScopedDir(symlink) error = %v, want symlink rejection", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCleanableRootChildrenRejectsSymlinkChild(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
realChild := filepath.Join(root, "runs")
|
||||||
|
if err := os.MkdirAll(realChild, 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(realChild) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.Symlink(realChild, filepath.Join(root, "link")); err != nil {
|
||||||
|
t.Fatalf("Symlink() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, _, err := cleanableRootChildren(root, "test.root.children")
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "refusing to delete symlink path") {
|
||||||
|
t.Fatalf("cleanableRootChildren() error = %v, want symlink rejection", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCleanValidateScopedTargetMissing(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
missingDir := filepath.Join(root, "runs", "missing")
|
||||||
|
got, err := validateScopedDir(root, missingDir, "test.missing")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("validateScopedDir(missing) error = %v", err)
|
||||||
|
}
|
||||||
|
if got.Exists {
|
||||||
|
t.Fatalf("validateScopedDir(missing).Exists = true, want false")
|
||||||
|
}
|
||||||
|
|
||||||
|
missingFile := filepath.Join(root, "cache", "missing.flac")
|
||||||
|
got, err = validateScopedFile(root, missingFile, "test.missing.file")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("validateScopedFile(missing) error = %v", err)
|
||||||
|
}
|
||||||
|
if got.Exists {
|
||||||
|
t.Fatalf("validateScopedFile(missing).Exists = true, want false")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -7,7 +7,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
var supportedCommands = []string{"run", "run-stage", "resume", "analyze", "publish", "clean", "session"}
|
var supportedCommands = []string{"run", "run-stage", "analyze", "publish", "clean", "session"}
|
||||||
|
|
||||||
// Execute dispatches CLI commands and returns a process exit code.
|
// Execute dispatches CLI commands and returns a process exit code.
|
||||||
func Execute(args []string, stdout, stderr io.Writer) int {
|
func Execute(args []string, stdout, stderr io.Writer) int {
|
||||||
@@ -24,8 +24,6 @@ func Execute(args []string, stdout, stderr io.Writer) int {
|
|||||||
switch cmd {
|
switch cmd {
|
||||||
case "run":
|
case "run":
|
||||||
err = Run(ctx, cmdArgs, stdout)
|
err = Run(ctx, cmdArgs, stdout)
|
||||||
case "resume":
|
|
||||||
err = Resume(ctx, cmdArgs, stdout)
|
|
||||||
case "run-stage":
|
case "run-stage":
|
||||||
err = RunStage(ctx, cmdArgs, stdout)
|
err = RunStage(ctx, cmdArgs, stdout)
|
||||||
case "analyze":
|
case "analyze":
|
||||||
|
|||||||
@@ -34,7 +34,6 @@ func TestExecuteValidCommands(t *testing.T) {
|
|||||||
{name: "run", args: []string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run: session 2026-05-03; executed=9 skipped=0; manifest="},
|
{name: "run", args: []string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run: session 2026-05-03; executed=9 skipped=0; manifest="},
|
||||||
{name: "session plan", args: []string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "prepare: skip\ntranscribe: skip\nmerge: skip\npolish: skip\nnormalize: skip\ntrim: skip\nanalyze: skip\npublish: skip\nnotify: skip"},
|
{name: "session plan", args: []string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "prepare: skip\ntranscribe: skip\nmerge: skip\npolish: skip\nnormalize: skip\ntrim: skip\nanalyze: skip\npublish: skip\nnotify: skip"},
|
||||||
{name: "session status", args: []string{"session", "status", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "Session: 2026-05-03"},
|
{name: "session status", args: []string{"session", "status", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "Session: 2026-05-03"},
|
||||||
{name: "resume", args: []string{"resume", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio resume: session 2026-05-03 has no remaining stages"},
|
|
||||||
{name: "run-stage", args: []string{"run-stage", "polish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run-stage: stage=polish executed=0 skipped=1 force=false; manifest="},
|
{name: "run-stage", args: []string{"run-stage", "polish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run-stage: stage=polish executed=0 skipped=1 force=false; manifest="},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -66,7 +65,7 @@ func TestExecuteMissingRequiredFlags(t *testing.T) {
|
|||||||
{name: "run missing session", args: []string{"run"}, want: "run: session_id is required"},
|
{name: "run missing session", args: []string{"run"}, want: "run: session_id is required"},
|
||||||
{name: "plan old top-level removed", args: []string{"plan"}, want: `unknown command: "plan"`},
|
{name: "plan old top-level removed", args: []string{"plan"}, want: `unknown command: "plan"`},
|
||||||
{name: "status old top-level removed", args: []string{"status"}, want: `unknown command: "status"`},
|
{name: "status old top-level removed", args: []string{"status"}, want: `unknown command: "status"`},
|
||||||
{name: "resume missing session", args: []string{"resume"}, want: "resume: session_id is required"},
|
{name: "resume removed", args: []string{"resume"}, want: `unknown command: "resume"`},
|
||||||
{name: "run-stage missing name", args: []string{"run-stage", "--config", "a", "--session", "b"}, want: "run-stage: expected stage name and session_id"},
|
{name: "run-stage missing name", args: []string{"run-stage", "--config", "a", "--session", "b"}, want: "run-stage: expected stage name and session_id"},
|
||||||
{name: "run-stage missing session", args: []string{"run-stage", "polish"}, want: "run-stage: expected stage name and session_id"},
|
{name: "run-stage missing session", args: []string{"run-stage", "polish"}, want: "run-stage: expected stage name and session_id"},
|
||||||
{name: "run missing config uses defaults", args: []string{"run", "2026-05-03", "--session", "session.yml"}, want: "run: no pipeline config path provided and no default pipeline config found; searched:"},
|
{name: "run missing config uses defaults", args: []string{"run", "2026-05-03", "--session", "session.yml"}, want: "run: no pipeline config path provided and no default pipeline config found; searched:"},
|
||||||
|
|||||||
@@ -4,7 +4,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
@@ -59,7 +58,7 @@ func loadCommandConfig(ctx context.Context, pipelineFlag, campaignFlag, campaign
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, missingSessionConfigError(discoveredSession.Searched, err.Error())
|
return nil, missingSessionConfigError(discoveredSession.Searched, err.Error())
|
||||||
}
|
}
|
||||||
sessionTempPath, err := downloadRemoteSessionConfig(ctx, store, remoteKey)
|
sessionTempPath, err := storage.DownloadObjectToTemp(ctx, store, remoteKey, "narratio-session-*.yml")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, missingSessionConfigError(discoveredSession.Searched, fmt.Sprintf("remote session %q download failed: %v", remoteKey, err))
|
return nil, missingSessionConfigError(discoveredSession.Searched, fmt.Sprintf("remote session %q download failed: %v", remoteKey, err))
|
||||||
}
|
}
|
||||||
@@ -134,21 +133,6 @@ func findRemoteSessionConfig(ctx context.Context, store storage.ObjectStore, ses
|
|||||||
return storage.ObjectInfo{}, fmt.Errorf("remote session %q not found", remoteKey)
|
return storage.ObjectInfo{}, fmt.Errorf("remote session %q not found", remoteKey)
|
||||||
}
|
}
|
||||||
|
|
||||||
func downloadRemoteSessionConfig(ctx context.Context, store storage.ObjectStore, remoteKey string) (string, error) {
|
|
||||||
f, err := os.CreateTemp("", "narratio-session-*.yml")
|
|
||||||
if err != nil {
|
|
||||||
return "", fmt.Errorf("create temp file: %w", err)
|
|
||||||
}
|
|
||||||
path := f.Name()
|
|
||||||
if err := f.Close(); err != nil {
|
|
||||||
return "", fmt.Errorf("close temp file %q: %w", path, err)
|
|
||||||
}
|
|
||||||
if err := store.Download(ctx, remoteKey, path); err != nil {
|
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
return filepath.Clean(path), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func s3BucketName(cfg *config.PipelineConfig) string {
|
func s3BucketName(cfg *config.PipelineConfig) string {
|
||||||
if cfg == nil || cfg.Storage.S3 == nil {
|
if cfg == nil || cfg.Storage.S3 == nil {
|
||||||
return ""
|
return ""
|
||||||
|
|||||||
136
internal/app/operator_artifact_rendering.go
Normal file
136
internal/app/operator_artifact_rendering.go
Normal file
@@ -0,0 +1,136 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
func buildHelperArtifactCatalog(cfg *config.Config) (*artifacts.ArtifactCatalog, error) {
|
||||||
|
catalog := artifacts.NewArtifactCatalog()
|
||||||
|
if err := catalog.RegisterBuiltIns(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
configured := map[string]artifacts.ConfiguredArtifactDefinition{}
|
||||||
|
if cfg.Pipeline.Scriptorium != nil {
|
||||||
|
for key, item := range cfg.Pipeline.Scriptorium.Artifacts {
|
||||||
|
configured[key] = artifacts.ConfiguredArtifactDefinition{Enabled: item.Enabled, OutputPath: item.OutputPath}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := catalog.RegisterConfiguredArtifacts(configured, nil); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return catalog, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeArtifactList(out io.Writer, cfg *config.Config, catalog *artifacts.ArtifactCatalog, locks *effectiveLocks, publishedRemoteState map[string]string) {
|
||||||
|
lockSet := lockSourceSet(locks.All)
|
||||||
|
fmt.Fprintln(out, "Built-in:")
|
||||||
|
for _, id := range []string{
|
||||||
|
artifacts.ArtifactTranscriptBase,
|
||||||
|
artifacts.ArtifactTranscriptPolished,
|
||||||
|
artifacts.ArtifactTranscriptFinal,
|
||||||
|
artifacts.ArtifactTranscriptFinalTrimmed,
|
||||||
|
artifacts.ArtifactBoundsSession,
|
||||||
|
} {
|
||||||
|
writeArtifactLine(out, id, lockSet)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Configured:")
|
||||||
|
for _, entry := range catalog.ListConfigured() {
|
||||||
|
writeArtifactLine(out, entry.SourceID, lockSet)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Previous-session:")
|
||||||
|
for _, req := range artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg)) {
|
||||||
|
fmt.Fprintf(out, "- %s required=%t\n", artifactpolicy.PreviousSessionSourceID(req.Name), req.Required)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Published:")
|
||||||
|
for _, rule := range cfg.Pipeline.Publish.Outputs {
|
||||||
|
writePublishedOutputLine(out, rule, catalog, lockSet, publishedRemoteState)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeArtifactLine(out io.Writer, source string, lockSet map[string]config.PublishLockRule) {
|
||||||
|
parts := []string{source}
|
||||||
|
if _, ok := lockSet[source]; ok {
|
||||||
|
parts = append(parts, "locked")
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||||
|
}
|
||||||
|
|
||||||
|
func writePublishedOutputLine(out io.Writer, rule config.PublishOutputRule, catalog *artifacts.ArtifactCatalog, lockSet map[string]config.PublishLockRule, remoteState map[string]string) {
|
||||||
|
source := strings.TrimSpace(rule.Source)
|
||||||
|
parts := []string{source}
|
||||||
|
if _, ok := lockSet[source]; ok {
|
||||||
|
parts = append(parts, "locked")
|
||||||
|
}
|
||||||
|
dest, showDest, err := helperPublishedOutputDest(rule, catalog)
|
||||||
|
if err != nil {
|
||||||
|
parts = append(parts, "remote=error")
|
||||||
|
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if showDest {
|
||||||
|
parts = append(parts, "dest="+dest)
|
||||||
|
}
|
||||||
|
if state := remoteState[publishedOutputRemoteStateKey(source, dest)]; state != "" {
|
||||||
|
parts = append(parts, state)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||||
|
}
|
||||||
|
|
||||||
|
func remotePublishedOutputAvailability(ctx context.Context, cfg *config.Config, store storage.ObjectStore, catalog *artifacts.ArtifactCatalog) map[string]string {
|
||||||
|
out := map[string]string{}
|
||||||
|
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
for _, rule := range cfg.Pipeline.Publish.Outputs {
|
||||||
|
source := strings.TrimSpace(rule.Source)
|
||||||
|
dest, _, err := helperPublishedOutputDest(rule, catalog)
|
||||||
|
if err != nil {
|
||||||
|
out[publishedOutputRemoteStateKey(source, "")] = "remote=error"
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
key := artifacts.S3PublishedOutputKey(sessionPrefix, dest)
|
||||||
|
if exists, err := store.Exists(ctx, key); err == nil && exists {
|
||||||
|
out[publishedOutputRemoteStateKey(source, dest)] = "remote=published"
|
||||||
|
} else if err != nil {
|
||||||
|
out[publishedOutputRemoteStateKey(source, dest)] = "remote=error"
|
||||||
|
} else {
|
||||||
|
out[publishedOutputRemoteStateKey(source, dest)] = "remote=missing"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func helperPublishedOutputDest(rule config.PublishOutputRule, catalog *artifacts.ArtifactCatalog) (string, bool, error) {
|
||||||
|
source := strings.TrimSpace(rule.Source)
|
||||||
|
normalized, err := artifactpolicy.ResolvePublishedDestination(source, rule.Dest, helperConfiguredOutputPathMap(catalog))
|
||||||
|
if err != nil {
|
||||||
|
return "", false, err
|
||||||
|
}
|
||||||
|
entry, ok := catalog.Lookup(source)
|
||||||
|
showDest := !ok || strings.TrimSpace(entry.CanonicalRelPath) != normalized
|
||||||
|
return normalized, showDest, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func helperConfiguredOutputPathMap(catalog *artifacts.ArtifactCatalog) map[string]string {
|
||||||
|
out := map[string]string{}
|
||||||
|
if catalog == nil {
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
for _, entry := range catalog.ListConfigured() {
|
||||||
|
if strings.TrimSpace(entry.ConfiguredKey) == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out[entry.ConfiguredKey] = strings.TrimSpace(entry.CanonicalRelPath)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func publishedOutputRemoteStateKey(source, dest string) string {
|
||||||
|
return strings.TrimSpace(source) + "\x00" + strings.TrimSpace(dest)
|
||||||
|
}
|
||||||
39
internal/app/operator_artifacts_list.go
Normal file
39
internal/app/operator_artifacts_list.go
Normal file
@@ -0,0 +1,39 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ArtifactsList lists effective artifact sources.
|
||||||
|
func ArtifactsList(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("artifacts list", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
var remote bool
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
fs.BoolVar(&remote, "remote", false, "inspect remote publish availability")
|
||||||
|
if err := parseSessionAwareFlags("artifacts list", fs, args, &flags.sessionID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("artifacts list: session_id is required")
|
||||||
|
}
|
||||||
|
cfg, store, locks, _, err := loadHelperContext(ctx, flags, remote)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("artifacts list: %w", err)
|
||||||
|
}
|
||||||
|
catalog, err := buildHelperArtifactCatalog(cfg)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("artifacts list: %w", err)
|
||||||
|
}
|
||||||
|
publishedRemoteState := map[string]string{}
|
||||||
|
if remote && store != nil {
|
||||||
|
publishedRemoteState = remotePublishedOutputAvailability(ctx, cfg, store, catalog)
|
||||||
|
}
|
||||||
|
writeArtifactList(out, cfg, catalog, locks, publishedRemoteState)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
140
internal/app/operator_findings.go
Normal file
140
internal/app/operator_findings.go
Normal file
@@ -0,0 +1,140 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
|
)
|
||||||
|
|
||||||
|
type finding struct {
|
||||||
|
Severity string
|
||||||
|
Category string
|
||||||
|
Message string
|
||||||
|
}
|
||||||
|
|
||||||
|
type findingError struct {
|
||||||
|
count int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e findingError) Error() string {
|
||||||
|
return fmt.Sprintf("%d validation error(s)", e.count)
|
||||||
|
}
|
||||||
|
|
||||||
|
func renderFindings(out io.Writer, campaign, sessionID string, findings []finding) error {
|
||||||
|
if campaign != "" || sessionID != "" {
|
||||||
|
fmt.Fprintf(out, "Campaign: %s\n", campaign)
|
||||||
|
fmt.Fprintf(out, "Session: %s\n\n", sessionID)
|
||||||
|
}
|
||||||
|
errorsCount := 0
|
||||||
|
for _, f := range findings {
|
||||||
|
if f.Severity == "ERROR" {
|
||||||
|
errorsCount++
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "%-5s %-10s %s\n", f.Severity, f.Category, f.Message)
|
||||||
|
}
|
||||||
|
if errorsCount > 0 {
|
||||||
|
return findingError{count: errorsCount}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func okFinding(category, msg string) finding { return finding{"OK", category, msg} }
|
||||||
|
func infoFinding(category, msg string) finding { return finding{"INFO", category, msg} }
|
||||||
|
func warnFinding(category, msg string) finding { return finding{"WARN", category, msg} }
|
||||||
|
func errorFinding(category, msg string) finding { return finding{"ERROR", category, msg} }
|
||||||
|
|
||||||
|
func sessionSourceSummary(cfg *config.Config) string {
|
||||||
|
source := cfg.SessionSource.Source
|
||||||
|
if source == "" {
|
||||||
|
source = "session_config"
|
||||||
|
}
|
||||||
|
if cfg.SessionSource.S3Key != "" {
|
||||||
|
return source + " " + cfg.SessionSource.S3Key
|
||||||
|
}
|
||||||
|
return source + " " + cfg.SessionPath
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateStableInputFindings(cfg *config.Config) []finding {
|
||||||
|
checks := inspectStableInputs(cfg)
|
||||||
|
out := make([]finding, 0, len(checks))
|
||||||
|
for _, check := range checks {
|
||||||
|
if check.Err != nil {
|
||||||
|
msg := check.Name + ": " + check.Err.Error()
|
||||||
|
if strings.TrimSpace(check.Path) != "" {
|
||||||
|
msg = fmt.Sprintf("%s missing: %v", check.Name, check.Err)
|
||||||
|
}
|
||||||
|
out = append(out, errorFinding("inputs", msg))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, okFinding("inputs", check.Name+": "+check.Path))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveHelperConfigRelativePath(input config.ResolvedInputFile) (string, error) {
|
||||||
|
if strings.TrimSpace(input.ConfigPath) == "" {
|
||||||
|
return "", fmt.Errorf("source config path is required")
|
||||||
|
}
|
||||||
|
path := strings.TrimSpace(input.Path)
|
||||||
|
if path == "" {
|
||||||
|
return "", fmt.Errorf("path is required")
|
||||||
|
}
|
||||||
|
if filepath.IsAbs(path) {
|
||||||
|
return filepath.Clean(path), nil
|
||||||
|
}
|
||||||
|
return filepath.Clean(filepath.Join(filepath.Dir(input.ConfigPath), path)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateLocalAudioFindings(cfg *config.Config) []finding {
|
||||||
|
check := inspectLocalAudioPresence(cfg)
|
||||||
|
if !check.Checked {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if check.Err != nil {
|
||||||
|
return []finding{errorFinding("audio", check.Err.Error())}
|
||||||
|
}
|
||||||
|
return []finding{okFinding("audio", fmt.Sprintf("%d local audio file(s)", len(check.Paths)))}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateRemoteAudioFinding(ctx context.Context, cfg *config.Config, store storage.ObjectStore) finding {
|
||||||
|
check := inspectRemoteAudioPresence(ctx, cfg, store)
|
||||||
|
if check.Err != nil {
|
||||||
|
return errorFinding("audio", check.Err.Error())
|
||||||
|
}
|
||||||
|
return okFinding("audio", fmt.Sprintf("%d remote .flac object(s)", len(check.Keys)))
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadLocalManifest(ctx context.Context, path string) (*manifest.Manifest, error) {
|
||||||
|
if _, err := os.Stat(path); err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
store := &manifest.LocalStore{}
|
||||||
|
return store.Load(ctx, path)
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStageStatuses(out io.Writer, m *manifest.Manifest) {
|
||||||
|
if m == nil || len(m.Stages) == 0 {
|
||||||
|
fmt.Fprintln(out, "stages: no stages recorded")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "stages:")
|
||||||
|
names := make([]string, 0, len(m.Stages))
|
||||||
|
for name := range m.Stages {
|
||||||
|
names = append(names, name)
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
for _, name := range names {
|
||||||
|
fmt.Fprintf(out, "- %s: %s\n", name, m.Stages[name].Status)
|
||||||
|
}
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -593,10 +593,57 @@ func TestExecuteLocksRequireSessionID(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteLocksMutationRejectsSessionIDMismatch(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
args []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "add mismatch",
|
||||||
|
args: []string{
|
||||||
|
"session", "locks", "add", "2026-05-03", "narratio.transcript.final_trimmed",
|
||||||
|
"--session-id", "2026-05-04",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "remove mismatch",
|
||||||
|
args: []string{
|
||||||
|
"session", "locks", "remove", "2026-05-03", "narratio.transcript.final_trimmed",
|
||||||
|
"--session-id", "2026-05-04",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute(tt.args, &stdout, &stderr)
|
||||||
|
if code == 0 {
|
||||||
|
t.Fatal("exit code = 0, want non-zero")
|
||||||
|
}
|
||||||
|
if !strings.Contains(stderr.String(), "does not match expected session id") {
|
||||||
|
t.Fatalf("stderr = %q, want session-id mismatch guidance", stderr.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteLocksCannotModifyStaticLocks(t *testing.T) {
|
func TestExecuteLocksCannotModifyStaticLocks(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
addStaticArchiveLockToPipelineConfig(t, pipelinePath, "narratio.transcript.final_trimmed")
|
addStaticPublishLockToPipelineConfig(t, pipelinePath, "narratio.transcript.final_trimmed")
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
var storeInitCalls int
|
var storeInitCalls int
|
||||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
@@ -683,10 +730,10 @@ func addSessionTemplateToCampaign(t *testing.T, campaignPath, templateFile strin
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteArtifactsListRemoteReportsPromotedAvailability(t *testing.T) {
|
func TestExecuteArtifactsListRemoteReportsPublishedAvailability(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
addPublishOutputsToPipeline(t, pipelinePath, `
|
||||||
outputs:
|
outputs:
|
||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
@@ -714,14 +761,14 @@ func TestExecuteArtifactsListRemoteReportsPromotedAvailability(t *testing.T) {
|
|||||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "narratio.transcript.final_trimmed remote=published") {
|
if !strings.Contains(stdout.String(), "narratio.transcript.final_trimmed remote=published") {
|
||||||
t.Fatalf("stdout = %q, want promoted remote availability", stdout.String())
|
t.Fatalf("stdout = %q, want published remote availability", stdout.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteArtifactsListRemoteUsesPromotionDestinations(t *testing.T) {
|
func TestExecuteArtifactsListRemoteUsesPublishOutputDestinations(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
addPublishOutputsToPipeline(t, pipelinePath, `
|
||||||
outputs:
|
outputs:
|
||||||
- source: narratio.transcript.final
|
- source: narratio.transcript.final
|
||||||
dest: transcripts/full.json
|
dest: transcripts/full.json
|
||||||
@@ -771,7 +818,7 @@ func TestExecuteArtifactsListRemoteUsesPromotionDestinations(t *testing.T) {
|
|||||||
func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
addPublishOutputsToPipeline(t, pipelinePath, `
|
||||||
outputs:
|
outputs:
|
||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
@@ -782,7 +829,7 @@ func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
|||||||
`)
|
`)
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03")
|
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03")
|
||||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
trimmedKey := artifacts.S3PublishedOutputKey(sessionPrefix, "transcripts/final.trimmed.json")
|
trimmedKey := artifacts.S3PublishedOutputKey(sessionPrefix, "transcripts/final.trimmed.json")
|
||||||
fullKey := artifacts.S3PublishedOutputKey(sessionPrefix, "transcripts/full.json")
|
fullKey := artifacts.S3PublishedOutputKey(sessionPrefix, "transcripts/full.json")
|
||||||
lockKey := artifacts.S3SessionLocksKey(sessionPrefix)
|
lockKey := artifacts.S3SessionLocksKey(sessionPrefix)
|
||||||
@@ -828,7 +875,7 @@ func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
|||||||
func TestExecuteStatusReportsRemoteArtifactCatalogErrorsWithoutFailing(t *testing.T) {
|
func TestExecuteStatusReportsRemoteArtifactCatalogErrorsWithoutFailing(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
addPublishOutputsToPipeline(t, pipelinePath, `
|
||||||
outputs:
|
outputs:
|
||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
@@ -861,9 +908,88 @@ func TestExecuteStatusReportsRemoteArtifactCatalogErrorsWithoutFailing(t *testin
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteArchiveLoadsRemoteLocks(t *testing.T) {
|
func TestExecuteStatusReportsMissingRemoteCurrentStateWithoutFailing(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidArchiveConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{
|
||||||
|
"session", "status", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "Remote publish: missing or unavailable: remote current run pointer missing") {
|
||||||
|
t.Fatalf("stdout = %q, want missing remote current-state line", stdout.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteStatusReportsPreviousStateReadinessWithoutFailing(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
|
replaceInFileOrFatal(t, pipelinePath, "source: narratio.artifact.session_recap", "source: narratio.previous_session.artifact.session_recap")
|
||||||
|
replaceInFileOrFatal(t, sessionPath, "session_id: 2026-05-03\n", "session_id: 2026-05-03\nprevious_session_id: 2026-04-26\n")
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{
|
||||||
|
"session", "status", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "Previous-session artifacts: unavailable: remote current run pointer missing") {
|
||||||
|
t.Fatalf("stdout = %q, want previous readiness unavailable line", stdout.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteSessionValidateReportsPreviousStateFindingAndReturnsFindingError(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
|
replaceInFileOrFatal(t, pipelinePath, "source: narratio.artifact.session_recap", "source: narratio.previous_session.artifact.session_recap")
|
||||||
|
replaceInFileOrFatal(t, sessionPath, "session_id: 2026-05-03\n", "session_id: 2026-05-03\nprevious_session_id: 2026-04-26\n")
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{
|
||||||
|
"session", "validate", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code == 0 {
|
||||||
|
t.Fatal("exit code = 0, want non-zero")
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "ERROR previous") {
|
||||||
|
t.Fatalf("stdout = %q, want previous finding error", stdout.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "remote current run pointer missing") {
|
||||||
|
t.Fatalf("stdout = %q, want missing run pointer finding", stdout.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stderr.String(), "validation error(s)") {
|
||||||
|
t.Fatalf("stderr = %q, want finding error summary", stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecutePublishLoadsRemoteLocks(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidPublishRunConfigFiles(t, workspaceRoot)
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
lockKey := artifacts.S3SessionLocksKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"))
|
lockKey := artifacts.S3SessionLocksKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"))
|
||||||
fake.SeedObject(storage.FakeObject{Key: lockKey, Data: []byte("locks:\n - source: narratio.transcript.final_trimmed\n reason: remote review\n")})
|
fake.SeedObject(storage.FakeObject{Key: lockKey, Data: []byte("locks:\n - source: narratio.transcript.final_trimmed\n reason: remote review\n")})
|
||||||
@@ -883,28 +1009,43 @@ func TestExecuteArchiveLoadsRemoteLocks(t *testing.T) {
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
promotedKey := artifacts.S3PublishedOutputKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"), "transcripts/final.trimmed.json")
|
publishedKey := artifacts.S3PublishedOutputKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"), "transcripts/final.trimmed.json")
|
||||||
if _, ok := fake.Objects[promotedKey]; ok {
|
if _, ok := fake.Objects[publishedKey]; ok {
|
||||||
t.Fatalf("locked promoted key %q was uploaded", promotedKey)
|
t.Fatalf("locked published key %q was uploaded", publishedKey)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func addArchivePromotionsToPipeline(t *testing.T, pipelinePath, archiveYAML string) {
|
func addPublishOutputsToPipeline(t *testing.T, pipelinePath, publishYAML string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
data, err := os.ReadFile(pipelinePath)
|
data, err := os.ReadFile(pipelinePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("read pipeline: %v", err)
|
t.Fatalf("read pipeline: %v", err)
|
||||||
}
|
}
|
||||||
updated := strings.Replace(string(data), " upload_run: false\n", " upload_run: false\n"+archiveYAML, 1)
|
updated := strings.Replace(string(data), " upload_run: false\n", " upload_run: false\n"+publishYAML, 1)
|
||||||
if updated == string(data) {
|
if updated == string(data) {
|
||||||
t.Fatalf("pipeline %q did not contain archive upload_run marker", pipelinePath)
|
t.Fatalf("pipeline %q did not contain publish upload_run marker", pipelinePath)
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(pipelinePath, []byte(updated), 0o644); err != nil {
|
if err := os.WriteFile(pipelinePath, []byte(updated), 0o644); err != nil {
|
||||||
t.Fatalf("write pipeline: %v", err)
|
t.Fatalf("write pipeline: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeValidArchiveConfigFiles(t *testing.T, workspaceRoot string) (string, string, string) {
|
func replaceInFileOrFatal(t *testing.T, path, old, new string) {
|
||||||
|
t.Helper()
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read %s: %v", path, err)
|
||||||
|
}
|
||||||
|
updated := strings.Replace(string(data), old, new, 1)
|
||||||
|
if updated == string(data) {
|
||||||
|
t.Fatalf("%s did not contain %q", path, old)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(path, []byte(updated), 0o644); err != nil {
|
||||||
|
t.Fatalf("write %s: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeValidPublishRunConfigFiles(t *testing.T, workspaceRoot string) (string, string, string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
data, err := os.ReadFile(pipelinePath)
|
data, err := os.ReadFile(pipelinePath)
|
||||||
@@ -941,7 +1082,7 @@ func writeValidArchiveConfigFiles(t *testing.T, workspaceRoot string) (string, s
|
|||||||
return pipelinePath, campaignPath, sessionPath
|
return pipelinePath, campaignPath, sessionPath
|
||||||
}
|
}
|
||||||
|
|
||||||
func addStaticArchiveLockToPipelineConfig(t *testing.T, pipelinePath, source string) {
|
func addStaticPublishLockToPipelineConfig(t *testing.T, pipelinePath, source string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
data, err := os.ReadFile(pipelinePath)
|
data, err := os.ReadFile(pipelinePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -954,7 +1095,7 @@ func addStaticArchiveLockToPipelineConfig(t *testing.T, pipelinePath, source str
|
|||||||
1,
|
1,
|
||||||
)
|
)
|
||||||
if updated == string(data) {
|
if updated == string(data) {
|
||||||
t.Fatalf("archive section not found in pipeline config")
|
t.Fatalf("publish section not found in pipeline config")
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(pipelinePath, []byte(updated), 0o644); err != nil {
|
if err := os.WriteFile(pipelinePath, []byte(updated), 0o644); err != nil {
|
||||||
t.Fatalf("write pipeline: %v", err)
|
t.Fatalf("write pipeline: %v", err)
|
||||||
|
|||||||
274
internal/app/operator_inspection.go
Normal file
274
internal/app/operator_inspection.go
Normal file
@@ -0,0 +1,274 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
type stableInputCheck struct {
|
||||||
|
Name string
|
||||||
|
Path string
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type localAudioCheck struct {
|
||||||
|
Checked bool
|
||||||
|
Paths []string
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type remoteAudioCheck struct {
|
||||||
|
Checked bool
|
||||||
|
Prefix string
|
||||||
|
Keys []string
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type previousArtifactReadiness struct {
|
||||||
|
Requirements []artifacts.PreviousArtifactRequirement
|
||||||
|
MissingID bool
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type remoteCurrentStateCheck struct {
|
||||||
|
State *RemoteCurrentState
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type effectiveLocksCheck struct {
|
||||||
|
Locks *effectiveLocks
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectStableInputs(cfg *config.Config) []stableInputCheck {
|
||||||
|
items := []struct {
|
||||||
|
name string
|
||||||
|
in config.ResolvedInputFile
|
||||||
|
}{
|
||||||
|
{name: "speakers", in: cfg.StableInputs.SpeakersFile},
|
||||||
|
{name: "autocorrect", in: cfg.StableInputs.AutocorrectFile},
|
||||||
|
{name: "glossary", in: cfg.StableInputs.GlossaryFile},
|
||||||
|
}
|
||||||
|
out := make([]stableInputCheck, 0, len(items))
|
||||||
|
for _, item := range items {
|
||||||
|
path, err := resolveHelperConfigRelativePath(item.in)
|
||||||
|
if err != nil {
|
||||||
|
out = append(out, stableInputCheck{Name: item.name, Err: err})
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(path); err != nil {
|
||||||
|
out = append(out, stableInputCheck{Name: item.name, Path: path, Err: err})
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, stableInputCheck{Name: item.name, Path: path})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectLocalAudioPresence(cfg *config.Config) localAudioCheck {
|
||||||
|
if cfg.Session.Inputs.AudioS3 != nil {
|
||||||
|
return localAudioCheck{}
|
||||||
|
}
|
||||||
|
|
||||||
|
sessionDir := filepath.Dir(cfg.SessionPath)
|
||||||
|
resolved, err := resolveLocalInspectionAudioPaths(sessionDir, cfg.Session.Inputs)
|
||||||
|
if err != nil {
|
||||||
|
return localAudioCheck{Checked: true, Err: err}
|
||||||
|
}
|
||||||
|
return localAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Paths: resolved,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectRemoteAudioPresence(ctx context.Context, cfg *config.Config, store storage.ObjectStore) remoteAudioCheck {
|
||||||
|
if cfg.Session.Inputs.AudioS3 == nil {
|
||||||
|
return remoteAudioCheck{}
|
||||||
|
}
|
||||||
|
if store == nil {
|
||||||
|
return remoteAudioCheck{Checked: true, Err: fmt.Errorf("storage backend is required for remote audio checks")}
|
||||||
|
}
|
||||||
|
|
||||||
|
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
audioPrefix := artifacts.S3AudioPrefix(sessionPrefix, cfg.Session.Inputs.AudioS3.Prefix)
|
||||||
|
objects, err := store.List(ctx, audioPrefix)
|
||||||
|
if err != nil {
|
||||||
|
return remoteAudioCheck{Checked: true, Prefix: audioPrefix, Err: err}
|
||||||
|
}
|
||||||
|
|
||||||
|
keys := make([]string, 0, len(objects))
|
||||||
|
seenBase := map[string]string{}
|
||||||
|
for _, obj := range objects {
|
||||||
|
key := strings.TrimSpace(obj.Key)
|
||||||
|
if key == "" || strings.HasSuffix(key, "/") || !isInspectionFlacPath(key) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
base := path.Base(key)
|
||||||
|
if prev, exists := seenBase[base]; exists && prev != key {
|
||||||
|
return remoteAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Prefix: audioPrefix,
|
||||||
|
Err: fmt.Errorf("duplicate s3 audio basename %q from %q and %q", base, prev, key),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
seenBase[base] = key
|
||||||
|
keys = append(keys, key)
|
||||||
|
}
|
||||||
|
sort.Strings(keys)
|
||||||
|
if len(keys) == 0 {
|
||||||
|
return remoteAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Prefix: audioPrefix,
|
||||||
|
Err: fmt.Errorf("no .flac files found under s3 audio prefix %q", audioPrefix),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return remoteAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Prefix: audioPrefix,
|
||||||
|
Keys: keys,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectPreviousArtifactReadiness(
|
||||||
|
ctx context.Context,
|
||||||
|
cfg *config.Config,
|
||||||
|
store storage.ObjectStore,
|
||||||
|
requirements []artifacts.PreviousArtifactRequirement,
|
||||||
|
) previousArtifactReadiness {
|
||||||
|
out := previousArtifactReadiness{
|
||||||
|
Requirements: append([]artifacts.PreviousArtifactRequirement(nil), requirements...),
|
||||||
|
}
|
||||||
|
if len(requirements) == 0 {
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(cfg.Session.PreviousSessionID) == "" {
|
||||||
|
out.MissingID = true
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
if store == nil {
|
||||||
|
out.Err = fmt.Errorf("previous-session artifacts cannot be checked because storage is unavailable")
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
prefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
||||||
|
if _, err := artifacts.LoadCurrentState(ctx, store, prefix, artifacts.CurrentStateValidation{
|
||||||
|
ExpectedSessionID: strings.TrimSpace(cfg.Session.PreviousSessionID),
|
||||||
|
ExpectedCampaign: strings.TrimSpace(cfg.Session.Campaign),
|
||||||
|
ValidateRunID: true,
|
||||||
|
}); err != nil {
|
||||||
|
out.Err = fmt.Errorf("remote %v", err)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectRemoteCurrentState(ctx context.Context, cfg *config.Config, store storage.ObjectStore) remoteCurrentStateCheck {
|
||||||
|
if store == nil {
|
||||||
|
return remoteCurrentStateCheck{}
|
||||||
|
}
|
||||||
|
current, err := discoverRemoteCurrentStateFn(ctx, cfg, store)
|
||||||
|
if err != nil {
|
||||||
|
return remoteCurrentStateCheck{Err: err}
|
||||||
|
}
|
||||||
|
return remoteCurrentStateCheck{State: current}
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectEffectiveLocks(ctx context.Context, cfg *config.Config, store storage.ObjectStore) effectiveLocksCheck {
|
||||||
|
locks, err := loadEffectiveLocks(ctx, cfg, store)
|
||||||
|
if err != nil {
|
||||||
|
return effectiveLocksCheck{Err: err}
|
||||||
|
}
|
||||||
|
return effectiveLocksCheck{Locks: locks}
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveLocalInspectionAudioPaths(sessionDir string, inputs config.SessionInputsConfig) ([]string, error) {
|
||||||
|
if len(inputs.AudioFiles) > 0 {
|
||||||
|
out := make([]string, 0, len(inputs.AudioFiles))
|
||||||
|
seenBase := map[string]string{}
|
||||||
|
for _, item := range inputs.AudioFiles {
|
||||||
|
resolved, err := resolveInspectionPath(sessionDir, item)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if !isInspectionFlacPath(resolved) {
|
||||||
|
return nil, fmt.Errorf("audio file %q must have .flac extension", resolved)
|
||||||
|
}
|
||||||
|
if err := requireInspectionFile(resolved, "audio file"); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
base := filepath.Base(resolved)
|
||||||
|
if prev, exists := seenBase[base]; exists && prev != resolved {
|
||||||
|
return nil, fmt.Errorf("duplicate audio basename %q from %q and %q", base, prev, resolved)
|
||||||
|
}
|
||||||
|
seenBase[base] = resolved
|
||||||
|
out = append(out, resolved)
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
audioDir, err := resolveInspectionPath(sessionDir, inputs.AudioDir)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
entries, err := os.ReadDir(audioDir)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read audio directory %q: %w", audioDir, err)
|
||||||
|
}
|
||||||
|
out := make([]string, 0, len(entries))
|
||||||
|
for _, entry := range entries {
|
||||||
|
if entry.IsDir() {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
full := filepath.Join(audioDir, entry.Name())
|
||||||
|
if !isInspectionFlacPath(full) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err := requireInspectionFile(full, "audio file"); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, full)
|
||||||
|
}
|
||||||
|
if len(out) == 0 {
|
||||||
|
return nil, fmt.Errorf("no .flac files found in audio directory %q", audioDir)
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveInspectionPath(baseDir, inputPath string) (string, error) {
|
||||||
|
pathValue := strings.TrimSpace(inputPath)
|
||||||
|
if pathValue == "" {
|
||||||
|
return "", fmt.Errorf("path is required")
|
||||||
|
}
|
||||||
|
if filepath.IsAbs(pathValue) {
|
||||||
|
return filepath.Clean(pathValue), nil
|
||||||
|
}
|
||||||
|
return filepath.Clean(filepath.Join(baseDir, pathValue)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireInspectionFile(path, label string) error {
|
||||||
|
info, err := os.Stat(path)
|
||||||
|
if err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return fmt.Errorf("%s %q does not exist", label, path)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("stat %s %q: %w", label, path, err)
|
||||||
|
}
|
||||||
|
if info.IsDir() {
|
||||||
|
return fmt.Errorf("%s %q is a directory", label, path)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func isInspectionFlacPath(path string) bool {
|
||||||
|
return strings.EqualFold(filepath.Ext(strings.TrimSpace(path)), ".flac")
|
||||||
|
}
|
||||||
172
internal/app/operator_locks.go
Normal file
172
internal/app/operator_locks.go
Normal file
@@ -0,0 +1,172 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Locks dispatches publish lock list and mutation helpers.
|
||||||
|
func Locks(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
|
||||||
|
switch args[0] {
|
||||||
|
case "add":
|
||||||
|
return LocksAdd(ctx, args[1:], out)
|
||||||
|
case "remove":
|
||||||
|
return LocksRemove(ctx, args[1:], out)
|
||||||
|
default:
|
||||||
|
return fmt.Errorf("locks: unknown subcommand %q", args[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return LocksList(ctx, args, out)
|
||||||
|
}
|
||||||
|
|
||||||
|
// LocksList lists effective publish locks.
|
||||||
|
func LocksList(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("locks", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
if err := parseSessionAwareFlags("locks", fs, args, &flags.sessionID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("locks: session_id is required")
|
||||||
|
}
|
||||||
|
cfg, _, locks, _, err := loadHelperContext(ctx, flags, true)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("locks: %w", err)
|
||||||
|
}
|
||||||
|
writeLocks(out, cfg, locks)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// LocksAdd adds or updates one remote lock.
|
||||||
|
func LocksAdd(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("locks add", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
var reason string
|
||||||
|
var force bool
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
fs.StringVar(&reason, "reason", "", "lock reason")
|
||||||
|
fs.BoolVar(&force, "force", false, "update existing remote lock")
|
||||||
|
source, err := parseSessionIDAndOnePositionalArg("locks add", "source id", fs, args, &flags.sessionID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("locks add: session_id is required")
|
||||||
|
}
|
||||||
|
cfg, store, locks, _, err := loadHelperContext(ctx, flags, true)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("locks add: %w", err)
|
||||||
|
}
|
||||||
|
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, "locks add"); err != nil {
|
||||||
|
return fmt.Errorf("locks add: %w", err)
|
||||||
|
}
|
||||||
|
if _, ok := lockSourceSet(locks.Static)[source]; ok {
|
||||||
|
return fmt.Errorf("locks add: source %q is locked by pipeline config and cannot be modified remotely", source)
|
||||||
|
}
|
||||||
|
remoteSet := lockSourceSet(locks.Remote)
|
||||||
|
if _, exists := remoteSet[source]; exists && !force {
|
||||||
|
return fmt.Errorf("locks add: remote lock for %q already exists; pass --force to update", source)
|
||||||
|
}
|
||||||
|
remoteSet[source] = config.PublishLockRule{Source: source, Reason: strings.TrimSpace(reason)}
|
||||||
|
remoteLocks := lockMapValues(remoteSet)
|
||||||
|
if _, err := config.ValidatePublishLockRules(remoteLocks, cfg.Pipeline.Scriptorium, "locks"); err != nil {
|
||||||
|
return fmt.Errorf("locks add: %w", err)
|
||||||
|
}
|
||||||
|
if err := uploadRemoteLockStore(ctx, store, locks.Key, &config.PublishLockStore{Locks: remoteLocks}); err != nil {
|
||||||
|
return fmt.Errorf("locks add: %w", err)
|
||||||
|
}
|
||||||
|
_, err = fmt.Fprintf(out, "narratio session locks add: locked %s\n", source)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// LocksRemove removes one remote lock.
|
||||||
|
func LocksRemove(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("locks remove", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
source, err := parseSessionIDAndOnePositionalArg("locks remove", "source id", fs, args, &flags.sessionID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("locks remove: session_id is required")
|
||||||
|
}
|
||||||
|
cfg, store, locks, _, err := loadHelperContext(ctx, flags, true)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("locks remove: %w", err)
|
||||||
|
}
|
||||||
|
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, "locks remove"); err != nil {
|
||||||
|
return fmt.Errorf("locks remove: %w", err)
|
||||||
|
}
|
||||||
|
remoteSet := lockSourceSet(locks.Remote)
|
||||||
|
if _, ok := remoteSet[source]; !ok {
|
||||||
|
if _, static := lockSourceSet(locks.Static)[source]; static {
|
||||||
|
return fmt.Errorf("locks remove: source %q is locked by pipeline config and cannot be unlocked remotely", source)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("locks remove: remote lock for %q does not exist", source)
|
||||||
|
}
|
||||||
|
delete(remoteSet, source)
|
||||||
|
remoteLocks := lockMapValues(remoteSet)
|
||||||
|
if err := uploadRemoteLockStore(ctx, store, locks.Key, &config.PublishLockStore{Locks: remoteLocks}); err != nil {
|
||||||
|
return fmt.Errorf("locks remove: %w", err)
|
||||||
|
}
|
||||||
|
_, err = fmt.Fprintf(out, "narratio session locks remove: unlocked %s\n", source)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeLocks(out io.Writer, cfg *config.Config, locks *effectiveLocks) {
|
||||||
|
if locks == nil || len(locks.All) == 0 {
|
||||||
|
fmt.Fprintln(out, "Publish locks: none")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Publish locks:")
|
||||||
|
published := map[string]config.PublishOutputRule{}
|
||||||
|
if cfg != nil && cfg.Pipeline != nil && cfg.Pipeline.Publish != nil {
|
||||||
|
for _, rule := range cfg.Pipeline.Publish.Outputs {
|
||||||
|
published[strings.TrimSpace(rule.Source)] = rule
|
||||||
|
}
|
||||||
|
}
|
||||||
|
staticSet := lockSourceSet(locks.Static)
|
||||||
|
for _, lock := range locks.All {
|
||||||
|
origin := "remote"
|
||||||
|
if _, ok := staticSet[lock.Source]; ok {
|
||||||
|
origin = "pipeline"
|
||||||
|
}
|
||||||
|
promo := "not-published"
|
||||||
|
if _, ok := published[lock.Source]; ok {
|
||||||
|
promo = "published"
|
||||||
|
}
|
||||||
|
reason := strings.TrimSpace(lock.Reason)
|
||||||
|
if reason == "" {
|
||||||
|
reason = "(no reason)"
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "- %s origin=%s %s reason=%s\n", lock.Source, origin, promo, reason)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func lockMapValues(in map[string]config.PublishLockRule) []config.PublishLockRule {
|
||||||
|
keys := make([]string, 0, len(in))
|
||||||
|
for key := range in {
|
||||||
|
keys = append(keys, key)
|
||||||
|
}
|
||||||
|
sort.Strings(keys)
|
||||||
|
out := make([]config.PublishLockRule, 0, len(keys))
|
||||||
|
for _, key := range keys {
|
||||||
|
item := in[key]
|
||||||
|
item.Source = key
|
||||||
|
item.Reason = strings.TrimSpace(item.Reason)
|
||||||
|
out = append(out, item)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
268
internal/app/operator_session_init.go
Normal file
268
internal/app/operator_session_init.go
Normal file
@@ -0,0 +1,268 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gopkg.in/yaml.v3"
|
||||||
|
)
|
||||||
|
|
||||||
|
// SessionInit creates a local or remote session.yml skeleton.
|
||||||
|
func SessionInit(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("session init", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var pipelinePath, campaignPath, campaignFilePath, sessionID, previousSessionID, date, title, output, audioS3Prefix, audioDir string
|
||||||
|
var remote, force bool
|
||||||
|
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||||
|
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
||||||
|
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
||||||
|
fs.StringVar(&sessionID, "session-id", "", "session identifier")
|
||||||
|
fs.StringVar(&previousSessionID, "previous-session-id", "", "previous session identifier")
|
||||||
|
fs.StringVar(&date, "date", "", "session date")
|
||||||
|
fs.StringVar(&title, "title", "", "session title")
|
||||||
|
fs.StringVar(&output, "output", "", "local output session.yml path")
|
||||||
|
fs.StringVar(&audioS3Prefix, "audio-s3-prefix", "", "session audio S3 prefix")
|
||||||
|
fs.StringVar(&audioDir, "audio-dir", "", "local audio directory")
|
||||||
|
fs.BoolVar(&remote, "remote", false, "write session.yml to S3 session prefix")
|
||||||
|
fs.BoolVar(&force, "force", false, "overwrite existing target")
|
||||||
|
if err := parseSessionAwareFlags("session init", fs, args, &sessionID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(sessionID) == "" {
|
||||||
|
return fmt.Errorf("session init: session_id is required")
|
||||||
|
}
|
||||||
|
if (strings.TrimSpace(output) == "") == !remote {
|
||||||
|
return fmt.Errorf("session init: specify exactly one target: --output <path> or --remote")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(audioDir) != "" && strings.TrimSpace(audioS3Prefix) != "" {
|
||||||
|
return fmt.Errorf("session init: --audio-dir and --audio-s3-prefix are mutually exclusive")
|
||||||
|
}
|
||||||
|
|
||||||
|
base, err := loadPipelineCampaignConfig(pipelinePath, campaignPath, campaignFilePath)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
input := sessionInitInput{
|
||||||
|
Campaign: config.CampaignID(base.Campaign),
|
||||||
|
CampaignPath: base.CampaignPath,
|
||||||
|
TemplateFile: base.Campaign.SessionTemplateFile,
|
||||||
|
SessionID: sessionID,
|
||||||
|
PreviousSessionID: previousSessionID,
|
||||||
|
Date: date,
|
||||||
|
Title: title,
|
||||||
|
AudioS3Prefix: audioS3Prefix,
|
||||||
|
AudioDir: audioDir,
|
||||||
|
}
|
||||||
|
data, err := buildSessionInitYAML(input)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
label := strings.TrimSpace(output)
|
||||||
|
if label == "" {
|
||||||
|
label = "remote session.yml"
|
||||||
|
}
|
||||||
|
sessionCfg, err := config.LoadSessionBytesWithOptions(label, data, config.SessionLoadOptions{
|
||||||
|
SessionID: sessionID,
|
||||||
|
PreviousSessionID: previousSessionID,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
cfg, err := config.Resolve(base.PipelinePath, base.Pipeline, base.CampaignPath, base.Campaign, label, sessionCfg, config.SessionSource{Source: "session_config", LocalPath: label})
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
if err := config.Validate(cfg); err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !remote {
|
||||||
|
if err := writeLocalFile(output, data, force); err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
_, err := fmt.Fprintf(out, "narratio session init: wrote %s\n", filepath.Clean(output))
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
store, err := newCommandObjectStore(ctx, cfg, nil)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
sessionPrefix := artifacts.S3SessionPrefix(base.Pipeline.Storage.S3.RootPrefix, config.CampaignID(base.Campaign), sessionID)
|
||||||
|
key := artifacts.S3SessionConfigKey(sessionPrefix)
|
||||||
|
exists, err := store.Exists(ctx, key)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: check remote session %q: %w", key, err)
|
||||||
|
}
|
||||||
|
if exists && !force {
|
||||||
|
return fmt.Errorf("session init: remote session %q already exists; pass --force to overwrite", key)
|
||||||
|
}
|
||||||
|
tmp, err := os.CreateTemp("", "narratio-session-init-*.yml")
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: create temp file: %w", err)
|
||||||
|
}
|
||||||
|
tmpPath := tmp.Name()
|
||||||
|
defer func() { _ = os.Remove(tmpPath) }()
|
||||||
|
if _, err := tmp.Write(data); err != nil {
|
||||||
|
_ = tmp.Close()
|
||||||
|
return fmt.Errorf("session init: write temp file: %w", err)
|
||||||
|
}
|
||||||
|
if err := tmp.Close(); err != nil {
|
||||||
|
return fmt.Errorf("session init: close temp file: %w", err)
|
||||||
|
}
|
||||||
|
if _, err := store.Upload(ctx, tmpPath, key, storage.UploadOptions{ContentType: "application/x-yaml; charset=utf-8"}); err != nil {
|
||||||
|
return fmt.Errorf("session init: upload remote session %q: %w", key, err)
|
||||||
|
}
|
||||||
|
_, err = fmt.Fprintf(out, "narratio session init: wrote s3://%s/%s\n", s3BucketName(base.Pipeline), key)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildSessionYAML(campaign, sessionID, previousSessionID, date, title, audioS3Prefix, audioDir string) ([]byte, error) {
|
||||||
|
if strings.TrimSpace(date) == "" && regexp.MustCompile(`^\d{4}-\d{2}-\d{2}$`).MatchString(strings.TrimSpace(sessionID)) {
|
||||||
|
date = strings.TrimSpace(sessionID)
|
||||||
|
}
|
||||||
|
type audioS3 struct {
|
||||||
|
Prefix string `yaml:"prefix"`
|
||||||
|
}
|
||||||
|
type inputs struct {
|
||||||
|
AudioDir string `yaml:"audio_dir,omitempty"`
|
||||||
|
AudioS3 *audioS3 `yaml:"audio_s3,omitempty"`
|
||||||
|
}
|
||||||
|
type sessionYAML struct {
|
||||||
|
Campaign string `yaml:"campaign"`
|
||||||
|
SessionID string `yaml:"session_id"`
|
||||||
|
PreviousSessionID string `yaml:"previous_session_id,omitempty"`
|
||||||
|
Date string `yaml:"date,omitempty"`
|
||||||
|
Title string `yaml:"title,omitempty"`
|
||||||
|
Inputs inputs `yaml:"inputs"`
|
||||||
|
}
|
||||||
|
in := inputs{AudioDir: strings.TrimSpace(audioDir)}
|
||||||
|
if in.AudioDir == "" {
|
||||||
|
prefix := strings.TrimSpace(audioS3Prefix)
|
||||||
|
if prefix == "" {
|
||||||
|
prefix = "audio/"
|
||||||
|
}
|
||||||
|
in.AudioS3 = &audioS3{Prefix: prefix}
|
||||||
|
}
|
||||||
|
data, err := yaml.Marshal(sessionYAML{
|
||||||
|
Campaign: strings.TrimSpace(campaign),
|
||||||
|
SessionID: strings.TrimSpace(sessionID),
|
||||||
|
PreviousSessionID: strings.TrimSpace(previousSessionID),
|
||||||
|
Date: strings.TrimSpace(date),
|
||||||
|
Title: strings.TrimSpace(title),
|
||||||
|
Inputs: in,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return data, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type sessionInitInput struct {
|
||||||
|
Campaign string
|
||||||
|
CampaignPath string
|
||||||
|
TemplateFile string
|
||||||
|
SessionID string
|
||||||
|
PreviousSessionID string
|
||||||
|
Date string
|
||||||
|
Title string
|
||||||
|
AudioS3Prefix string
|
||||||
|
AudioDir string
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildSessionInitYAML(in sessionInitInput) ([]byte, error) {
|
||||||
|
if strings.TrimSpace(in.TemplateFile) == "" {
|
||||||
|
return buildSessionYAML(in.Campaign, in.SessionID, in.PreviousSessionID, in.Date, in.Title, in.AudioS3Prefix, in.AudioDir)
|
||||||
|
}
|
||||||
|
templatePath := resolveSessionInitTemplatePath(in.CampaignPath, in.TemplateFile)
|
||||||
|
templateBytes, err := os.ReadFile(templatePath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read session template %q: %w", templatePath, err)
|
||||||
|
}
|
||||||
|
rendered, err := renderSessionInitTemplate(string(templateBytes), in)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("render session template %q: %w", templatePath, err)
|
||||||
|
}
|
||||||
|
return []byte(rendered), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveSessionInitTemplatePath(campaignPath, templateFile string) string {
|
||||||
|
templateFile = strings.TrimSpace(templateFile)
|
||||||
|
if filepath.IsAbs(templateFile) {
|
||||||
|
return filepath.Clean(templateFile)
|
||||||
|
}
|
||||||
|
return filepath.Clean(filepath.Join(filepath.Dir(campaignPath), templateFile))
|
||||||
|
}
|
||||||
|
|
||||||
|
var sessionInitTemplatePattern = regexp.MustCompile(`\{\{\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*\}\}`)
|
||||||
|
|
||||||
|
func renderSessionInitTemplate(content string, in sessionInitInput) (string, error) {
|
||||||
|
values := map[string]string{
|
||||||
|
"session_id": strings.TrimSpace(in.SessionID),
|
||||||
|
"previous_session_id": strings.TrimSpace(in.PreviousSessionID),
|
||||||
|
"date": strings.TrimSpace(in.Date),
|
||||||
|
"title": strings.TrimSpace(in.Title),
|
||||||
|
"audio_s3_prefix": strings.TrimSpace(in.AudioS3Prefix),
|
||||||
|
"audio_dir": strings.TrimSpace(in.AudioDir),
|
||||||
|
}
|
||||||
|
used := map[string]struct{}{}
|
||||||
|
unknown := map[string]struct{}{}
|
||||||
|
missing := map[string]struct{}{}
|
||||||
|
rendered := sessionInitTemplatePattern.ReplaceAllStringFunc(content, func(match string) string {
|
||||||
|
parts := sessionInitTemplatePattern.FindStringSubmatch(match)
|
||||||
|
if len(parts) < 2 {
|
||||||
|
return match
|
||||||
|
}
|
||||||
|
name := parts[1]
|
||||||
|
value, ok := values[name]
|
||||||
|
if !ok {
|
||||||
|
unknown[name] = struct{}{}
|
||||||
|
return match
|
||||||
|
}
|
||||||
|
used[name] = struct{}{}
|
||||||
|
if value == "" {
|
||||||
|
missing[name] = struct{}{}
|
||||||
|
return match
|
||||||
|
}
|
||||||
|
return value
|
||||||
|
})
|
||||||
|
if len(unknown) > 0 {
|
||||||
|
return "", fmt.Errorf("unsupported template variable(s): %s", sortedStringSet(unknown))
|
||||||
|
}
|
||||||
|
if len(missing) > 0 {
|
||||||
|
return "", fmt.Errorf("missing required template variable value(s): %s", sortedStringSet(missing))
|
||||||
|
}
|
||||||
|
unused := map[string]struct{}{}
|
||||||
|
for _, name := range []string{"previous_session_id", "date", "title", "audio_s3_prefix", "audio_dir"} {
|
||||||
|
if values[name] == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, ok := used[name]; !ok {
|
||||||
|
unused[name] = struct{}{}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(unused) > 0 {
|
||||||
|
return "", fmt.Errorf("unused template variable value(s): %s", sortedStringSet(unused))
|
||||||
|
}
|
||||||
|
return rendered, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func sortedStringSet(set map[string]struct{}) string {
|
||||||
|
items := make([]string, 0, len(set))
|
||||||
|
for item := range set {
|
||||||
|
items = append(items, item)
|
||||||
|
}
|
||||||
|
sort.Strings(items)
|
||||||
|
return strings.Join(items, ", ")
|
||||||
|
}
|
||||||
84
internal/app/operator_session_validate.go
Normal file
84
internal/app/operator_session_validate.go
Normal file
@@ -0,0 +1,84 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// SessionValidate performs a read-only session preflight.
|
||||||
|
func SessionValidate(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("session validate", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
if err := parseSessionAwareFlags("session validate", fs, args, &flags.sessionID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("session validate: session_id is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
findings := []finding{}
|
||||||
|
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||||
|
if err != nil {
|
||||||
|
findings = append(findings, errorFinding("config", err.Error()))
|
||||||
|
return renderFindings(out, "", "", findings)
|
||||||
|
}
|
||||||
|
if err := config.Validate(cfg); err != nil {
|
||||||
|
findings = append(findings, errorFinding("config", err.Error()))
|
||||||
|
} else {
|
||||||
|
findings = append(findings, okFinding("config", "resolved pipeline, campaign, and session config"))
|
||||||
|
}
|
||||||
|
findings = append(findings, okFinding("session", fmt.Sprintf("session source: %s", sessionSourceSummary(cfg))))
|
||||||
|
|
||||||
|
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
findings = append(findings, validateStableInputFindings(cfg)...)
|
||||||
|
findings = append(findings, validateLocalAudioFindings(cfg)...)
|
||||||
|
|
||||||
|
store, storeErr := objectStoreIfConfigured(ctx, cfg)
|
||||||
|
if storeErr != nil {
|
||||||
|
findings = append(findings, errorFinding("storage", storeErr.Error()))
|
||||||
|
}
|
||||||
|
if cfg.Session.Inputs.AudioS3 != nil {
|
||||||
|
if storeErr != nil {
|
||||||
|
findings = append(findings, errorFinding("audio", "remote audio cannot be checked because storage is unavailable"))
|
||||||
|
} else {
|
||||||
|
findings = append(findings, validateRemoteAudioFinding(ctx, cfg, store))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
requirements := artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg))
|
||||||
|
previous := inspectPreviousArtifactReadiness(ctx, cfg, store, requirements)
|
||||||
|
if len(previous.Requirements) == 0 {
|
||||||
|
findings = append(findings, okFinding("previous", "no previous-session artifacts required"))
|
||||||
|
} else if previous.MissingID {
|
||||||
|
findings = append(findings, errorFinding("previous", "previous_session_id is required by configured previous-session artifacts"))
|
||||||
|
} else if previous.Err != nil {
|
||||||
|
findings = append(findings, errorFinding("previous", previous.Err.Error()))
|
||||||
|
} else {
|
||||||
|
for _, req := range previous.Requirements {
|
||||||
|
findings = append(findings, okFinding("previous", fmt.Sprintf("%s required=%t", req.Name, req.Required)))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
locks := inspectEffectiveLocks(ctx, cfg, store)
|
||||||
|
if locks.Err != nil {
|
||||||
|
findings = append(findings, errorFinding("locks", locks.Err.Error()))
|
||||||
|
} else if len(locks.Locks.All) == 0 {
|
||||||
|
findings = append(findings, okFinding("locks", "no effective publish locks"))
|
||||||
|
} else {
|
||||||
|
for _, lock := range locks.Locks.All {
|
||||||
|
findings = append(findings, warnFinding("locks", fmt.Sprintf("%s locked: %s", lock.Source, strings.TrimSpace(lock.Reason))))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if paths.ManifestPath != "" {
|
||||||
|
findings = append(findings, infoFinding("workspace", "manifest path: "+paths.ManifestPath))
|
||||||
|
}
|
||||||
|
return renderFindings(out, cfg.Session.Campaign, cfg.Session.SessionID, findings)
|
||||||
|
}
|
||||||
166
internal/app/operator_status.go
Normal file
166
internal/app/operator_status.go
Normal file
@@ -0,0 +1,166 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Status reports effective local/remote session state.
|
||||||
|
func Status(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("status", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
if err := parseSessionAwareFlags("status", fs, args, &flags.sessionID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("status: session_id is required")
|
||||||
|
}
|
||||||
|
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("status: %w", err)
|
||||||
|
}
|
||||||
|
if err := config.Validate(cfg); err != nil {
|
||||||
|
return fmt.Errorf("status: %w", err)
|
||||||
|
}
|
||||||
|
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
fmt.Fprintf(out, "Session: %s\n", cfg.Session.SessionID)
|
||||||
|
fmt.Fprintf(out, "Campaign: %s\n", cfg.Session.Campaign)
|
||||||
|
fmt.Fprintf(out, "Workspace: %s\n", paths.Root)
|
||||||
|
fmt.Fprintf(out, "Session config: %s\n", sessionSourceSummary(cfg))
|
||||||
|
writeStatusStableInputs(out, inspectStableInputs(cfg))
|
||||||
|
writeStatusLocalAudio(out, inspectLocalAudioPresence(cfg))
|
||||||
|
|
||||||
|
if m, err := loadLocalManifest(ctx, paths.ManifestPath); err != nil {
|
||||||
|
fmt.Fprintf(out, "Local manifest: error: %v\n", err)
|
||||||
|
} else if m == nil {
|
||||||
|
fmt.Fprintln(out, "Local manifest: missing")
|
||||||
|
} else {
|
||||||
|
fmt.Fprintf(out, "Local manifest: %s\n", paths.ManifestPath)
|
||||||
|
writeStageStatuses(out, m)
|
||||||
|
}
|
||||||
|
|
||||||
|
store, storeErr := objectStoreIfConfigured(ctx, cfg)
|
||||||
|
if storeErr != nil {
|
||||||
|
fmt.Fprintf(out, "Remote publish: unavailable: %v\n", storeErr)
|
||||||
|
} else if store != nil {
|
||||||
|
current := inspectRemoteCurrentState(ctx, cfg, store)
|
||||||
|
if current.Err != nil {
|
||||||
|
fmt.Fprintf(out, "Remote publish: missing or unavailable: %v\n", current.Err)
|
||||||
|
} else {
|
||||||
|
fmt.Fprintf(out, "Remote publish: current run %s\n", current.State.RunID)
|
||||||
|
fmt.Fprintf(out, "Remote manifest: %s\n", current.State.CurrentManifestKey)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
writeStatusRemoteAudio(ctx, out, cfg, store, storeErr)
|
||||||
|
writeStatusPreviousArtifacts(out, inspectPreviousArtifactReadiness(
|
||||||
|
ctx,
|
||||||
|
cfg,
|
||||||
|
store,
|
||||||
|
artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg)),
|
||||||
|
))
|
||||||
|
|
||||||
|
lockChecks := inspectEffectiveLocks(ctx, cfg, store)
|
||||||
|
locks := lockChecks.Locks
|
||||||
|
lockErr := lockChecks.Err
|
||||||
|
if catalog, catalogErr := buildHelperArtifactCatalog(cfg); catalogErr != nil {
|
||||||
|
fmt.Fprintf(out, "Remote outputs: error: %v\n", catalogErr)
|
||||||
|
} else if storeErr == nil {
|
||||||
|
catalogLocks := locks
|
||||||
|
if lockErr != nil {
|
||||||
|
catalogLocks = &effectiveLocks{
|
||||||
|
Static: staticPublishLocks(cfg),
|
||||||
|
All: staticPublishLocks(cfg),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
publishedRemoteState := map[string]string{}
|
||||||
|
if store != nil {
|
||||||
|
publishedRemoteState = remotePublishedOutputAvailability(ctx, cfg, store, catalog)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Remote outputs:")
|
||||||
|
writeArtifactList(out, cfg, catalog, catalogLocks, publishedRemoteState)
|
||||||
|
}
|
||||||
|
if lockErr != nil {
|
||||||
|
fmt.Fprintf(out, "Publish locks: error: %v\n", lockErr)
|
||||||
|
} else {
|
||||||
|
writeLocks(out, cfg, locks)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Next actions:")
|
||||||
|
fmt.Fprintf(out, "- narratio session validate %s\n", cfg.Session.SessionID)
|
||||||
|
fmt.Fprintf(out, "- narratio session restore %s --dry-run\n", cfg.Session.SessionID)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStatusStableInputs(out io.Writer, checks []stableInputCheck) {
|
||||||
|
if len(checks) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for _, check := range checks {
|
||||||
|
if check.Err != nil {
|
||||||
|
if strings.TrimSpace(check.Path) != "" {
|
||||||
|
fmt.Fprintf(out, "Stable input %s: unavailable: %v\n", check.Name, check.Err)
|
||||||
|
} else {
|
||||||
|
fmt.Fprintf(out, "Stable input %s: unavailable: %s\n", check.Name, check.Err.Error())
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "Stable input %s: %s\n", check.Name, check.Path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStatusLocalAudio(out io.Writer, check localAudioCheck) {
|
||||||
|
if !check.Checked {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if check.Err != nil {
|
||||||
|
fmt.Fprintf(out, "Local audio: unavailable: %v\n", check.Err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "Local audio: %d file(s)\n", len(check.Paths))
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStatusRemoteAudio(ctx context.Context, out io.Writer, cfg *config.Config, store storage.ObjectStore, storeErr error) {
|
||||||
|
if cfg.Session.Inputs.AudioS3 == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if storeErr != nil {
|
||||||
|
fmt.Fprintf(out, "Remote audio: unavailable: %v\n", storeErr)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
check := inspectRemoteAudioPresence(ctx, cfg, store)
|
||||||
|
if check.Err != nil {
|
||||||
|
fmt.Fprintf(out, "Remote audio: unavailable: %v\n", check.Err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "Remote audio: %d .flac object(s)\n", len(check.Keys))
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStatusPreviousArtifacts(out io.Writer, readiness previousArtifactReadiness) {
|
||||||
|
if len(readiness.Requirements) == 0 {
|
||||||
|
fmt.Fprintln(out, "Previous-session artifacts: not required")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if readiness.MissingID {
|
||||||
|
fmt.Fprintln(out, "Previous-session artifacts: unavailable: previous_session_id is required by configured previous-session artifacts")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if readiness.Err != nil {
|
||||||
|
fmt.Fprintf(out, "Previous-session artifacts: unavailable: %v\n", readiness.Err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
names := make([]string, 0, len(readiness.Requirements))
|
||||||
|
for _, req := range readiness.Requirements {
|
||||||
|
names = append(names, fmt.Sprintf("%s(required=%t)", req.Name, req.Required))
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
fmt.Fprintf(out, "Previous-session artifacts: ready: %s\n", strings.Join(names, ", "))
|
||||||
|
}
|
||||||
@@ -7,7 +7,6 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"os"
|
"os"
|
||||||
"strings"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
@@ -17,46 +16,21 @@ import (
|
|||||||
|
|
||||||
// Plan validates configuration, prepares the local workdir, and prints stage order.
|
// Plan validates configuration, prepares the local workdir, and prints stage order.
|
||||||
func Plan(ctx context.Context, args []string, out io.Writer) error {
|
func Plan(ctx context.Context, args []string, out io.Writer) error {
|
||||||
positionalSessionID, args := pullLeadingSessionID(args)
|
|
||||||
fs := flag.NewFlagSet("plan", flag.ContinueOnError)
|
fs := flag.NewFlagSet("plan", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
|
|
||||||
var pipelinePath string
|
var flags commonConfigFlags
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var force bool
|
var force bool
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
if err := parseSessionAwareFlags("plan", fs, args, &flags.sessionID); err != nil {
|
||||||
return fmt.Errorf("plan: invalid flags: %w", err)
|
return err
|
||||||
}
|
}
|
||||||
if positionalSessionID == "" {
|
if flags.sessionID == "" {
|
||||||
if err := applyParsedSessionIDArg("plan", fs, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("plan: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("plan", positionalSessionID, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
|
||||||
return fmt.Errorf("plan: session_id is required")
|
return fmt.Errorf("plan: session_id is required")
|
||||||
}
|
}
|
||||||
cfg, err := loadCommandConfig(ctx, pipelinePath, campaignPath, campaignFilePath, sessionPath, config.SessionLoadOptions{
|
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||||
SessionID: sessionID,
|
|
||||||
PreviousSessionID: previousSessionID,
|
|
||||||
})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("plan: %w", err)
|
return fmt.Errorf("plan: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m *manifest.Manifest, executed []string) error {
|
func runPostPublishCleanup(ctx context.Context, env *Env, manifestPath string, m *manifest.Manifest, executed []string) error {
|
||||||
if env == nil || env.Config == nil || env.Config.Pipeline == nil || m == nil {
|
if env == nil || env.Config == nil || env.Config.Pipeline == nil || m == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -23,7 +23,7 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
sr := archiveStageRecordForCleanup(m, executed)
|
sr := publishStageRecordForCleanup(m, executed)
|
||||||
if sr == nil {
|
if sr == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -33,7 +33,7 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
|||||||
sr.Metadata["spool_cleanup_requested"] = spoolRequested
|
sr.Metadata["spool_cleanup_requested"] = spoolRequested
|
||||||
sr.Metadata["workdir_cleanup_requested"] = workRequested
|
sr.Metadata["workdir_cleanup_requested"] = workRequested
|
||||||
|
|
||||||
eligible, reason := archiveCleanupEligible(env.Config, sr)
|
eligible, reason := publishCleanupEligible(env.Config, sr)
|
||||||
if !eligible {
|
if !eligible {
|
||||||
sr.Metadata["cleanup_skipped"] = true
|
sr.Metadata["cleanup_skipped"] = true
|
||||||
sr.Metadata["cleanup_skipped_reason"] = reason
|
sr.Metadata["cleanup_skipped_reason"] = reason
|
||||||
@@ -96,7 +96,7 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveStageRecordForCleanup(m *manifest.Manifest, executed []string) *manifest.StageRecord {
|
func publishStageRecordForCleanup(m *manifest.Manifest, executed []string) *manifest.StageRecord {
|
||||||
if m == nil {
|
if m == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -117,7 +117,7 @@ func archiveStageRecordForCleanup(m *manifest.Manifest, executed []string) *mani
|
|||||||
return sr
|
return sr
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveCleanupEligible(cfg *config.Config, sr *manifest.StageRecord) (bool, string) {
|
func publishCleanupEligible(cfg *config.Config, sr *manifest.StageRecord) (bool, string) {
|
||||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Publish == nil {
|
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Publish == nil {
|
||||||
return false, "publish configuration is missing"
|
return false, "publish configuration is missing"
|
||||||
}
|
}
|
||||||
@@ -174,49 +174,7 @@ func removeRunScopedDir(root, target, policy string) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func validateScopedDir(root, target, policy string) (scopedDir, error) {
|
func validateScopedDir(root, target, policy string) (scopedDir, error) {
|
||||||
cleanRoot := strings.TrimSpace(root)
|
return validateScopedTarget(root, target, policy, true)
|
||||||
cleanTarget := strings.TrimSpace(target)
|
|
||||||
if cleanRoot == "" {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
|
||||||
}
|
|
||||||
if cleanTarget == "" {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target path is required", policy)
|
|
||||||
}
|
|
||||||
|
|
||||||
rootAbs, err := filepath.Abs(cleanRoot)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
|
||||||
}
|
|
||||||
targetAbs, err := filepath.Abs(cleanTarget)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve target %q: %w", policy, cleanTarget, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
rel, err := filepath.Rel(rootAbs, targetAbs)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: relative path from %q to %q: %w", policy, rootAbs, targetAbs, err)
|
|
||||||
}
|
|
||||||
if rel == "." {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete root directory %q", policy, rootAbs)
|
|
||||||
}
|
|
||||||
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete path outside root: root=%q target=%q", policy, rootAbs, targetAbs)
|
|
||||||
}
|
|
||||||
|
|
||||||
info, err := os.Lstat(targetAbs)
|
|
||||||
if err != nil {
|
|
||||||
if os.IsNotExist(err) {
|
|
||||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: false}, nil
|
|
||||||
}
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: stat target %q: %w", policy, targetAbs, err)
|
|
||||||
}
|
|
||||||
if info.Mode()&os.ModeSymlink != 0 {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, targetAbs)
|
|
||||||
}
|
|
||||||
if !info.IsDir() {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is not a directory", policy, targetAbs)
|
|
||||||
}
|
|
||||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: true}, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func asString(v any) string {
|
func asString(v any) string {
|
||||||
@@ -16,13 +16,13 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||||
)
|
)
|
||||||
|
|
||||||
type archiveSuccessStage struct {
|
type publishSuccessStage struct {
|
||||||
metadata map[string]any
|
metadata map[string]any
|
||||||
}
|
}
|
||||||
|
|
||||||
func (archiveSuccessStage) Name() string { return "publish" }
|
func (publishSuccessStage) Name() string { return "publish" }
|
||||||
func (archiveSuccessStage) Declares() stage.IODecl { return stage.IODecl{} }
|
func (publishSuccessStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||||
func (s archiveSuccessStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
func (s publishSuccessStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
||||||
md := map[string]any{
|
md := map[string]any{
|
||||||
"stage": "publish",
|
"stage": "publish",
|
||||||
"uploaded": true,
|
"uploaded": true,
|
||||||
@@ -43,12 +43,12 @@ func (notifyFailStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest
|
|||||||
return nil, errors.New("notify failed")
|
return nil, errors.New("notify failed")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupDisabledKeepsLocalDirs(t *testing.T) {
|
func TestPostPublishCleanupDisabledKeepsLocalDirs(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = false
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = false
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -57,12 +57,12 @@ func TestPostArchiveCleanupDisabledKeepsLocalDirs(t *testing.T) {
|
|||||||
assertExists(t, seed.localSourceAudio)
|
assertExists(t, seed.localSourceAudio)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupSpoolOnly(t *testing.T) {
|
func TestPostPublishCleanupSpoolOnly(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -71,12 +71,12 @@ func TestPostArchiveCleanupSpoolOnly(t *testing.T) {
|
|||||||
assertExists(t, seed.localSourceAudio)
|
assertExists(t, seed.localSourceAudio)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupWorkdirOnly(t *testing.T) {
|
func TestPostPublishCleanupWorkdirOnly(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = false
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = false
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -87,12 +87,12 @@ func TestPostArchiveCleanupWorkdirOnly(t *testing.T) {
|
|||||||
assertExists(t, seed.spoolAudioDir)
|
assertExists(t, seed.spoolAudioDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupBothPolicies(t *testing.T) {
|
func TestPostPublishCleanupBothPolicies(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -102,12 +102,12 @@ func TestPostArchiveCleanupBothPolicies(t *testing.T) {
|
|||||||
assertExists(t, seed.previousCachePath)
|
assertExists(t, seed.previousCachePath)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenArchiveFails(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenPublishFails(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
_, err := executeStages(context.Background(), cfg, []stage.Stage{failingStage{name: "publish", err: errors.New("archive failed")}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
_, err := executeStages(context.Background(), cfg, []stage.Stage{failingStage{name: "publish", err: errors.New("publish failed")}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||||
if err == nil || !strings.Contains(err.Error(), "stage \"publish\" failed") {
|
if err == nil || !strings.Contains(err.Error(), "stage \"publish\" failed") {
|
||||||
t.Fatalf("executeStages() error = %v, want publish failure", err)
|
t.Fatalf("executeStages() error = %v, want publish failure", err)
|
||||||
}
|
}
|
||||||
@@ -116,12 +116,12 @@ func TestPostArchiveCleanupNotRunWhenArchiveFails(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenArchiveSkipped(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenPublishSkipped(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{metadata: map[string]any{"skipped": true}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{metadata: map[string]any{"skipped": true}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -129,12 +129,12 @@ func TestPostArchiveCleanupNotRunWhenArchiveSkipped(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenCurrentPointerMissing(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenCurrentPointerMissing(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{metadata: map[string]any{"current_pointer_written": false}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{metadata: map[string]any{"current_pointer_written": false}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -142,13 +142,13 @@ func TestPostArchiveCleanupNotRunWhenCurrentPointerMissing(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenArchiveUploadDisabled(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenPublishUploadDisabled(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
cfg.Pipeline.Publish.UploadRun = boolPtr(false)
|
cfg.Pipeline.Publish.UploadRun = boolPtr(false)
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -156,12 +156,12 @@ func TestPostArchiveCleanupNotRunWhenArchiveUploadDisabled(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupWaitsUntilAllStagesSucceed(t *testing.T) {
|
func TestPostPublishCleanupWaitsUntilAllStagesSucceed(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
_, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}, notifyFailStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
_, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}, notifyFailStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||||
if err == nil || !strings.Contains(err.Error(), "stage \"notify\" failed") {
|
if err == nil || !strings.Contains(err.Error(), "stage \"notify\" failed") {
|
||||||
t.Fatalf("executeStages() error = %v, want notify failure", err)
|
t.Fatalf("executeStages() error = %v, want notify failure", err)
|
||||||
}
|
}
|
||||||
@@ -170,7 +170,7 @@ func TestPostArchiveCleanupWaitsUntilAllStagesSucceed(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupFailsOnUnsafePath(t *testing.T) {
|
func TestPostPublishCleanupFailsOnUnsafePath(t *testing.T) {
|
||||||
cfg, _ := cleanupFixtureConfig(t)
|
cfg, _ := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
||||||
@@ -186,25 +186,25 @@ func TestPostArchiveCleanupFailsOnUnsafePath(t *testing.T) {
|
|||||||
t.Fatalf("Save() error = %v", err)
|
t.Fatalf("Save() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||||
if err == nil || !strings.Contains(err.Error(), "refusing to delete path outside root") {
|
if err == nil || !strings.Contains(err.Error(), "refusing to delete path outside root") {
|
||||||
t.Fatalf("executeStages() error = %v, want safe-path failure", err)
|
t.Fatalf("executeStages() error = %v, want safe-path failure", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenOutputIsMissing(t *testing.T) {
|
||||||
cfg, seed, runID := archiveStageCleanupFixture(t)
|
cfg, seed, runID := publishStageCleanupFixture(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
cfg.Pipeline.Publish.Outputs = []config.PublishOutputRule{
|
cfg.Pipeline.Publish.Outputs = []config.PublishOutputRule{
|
||||||
{Source: "narratio.transcript.base", Dest: "transcripts/base.json", Required: boolPtr(true)},
|
{Source: "narratio.transcript.base", Dest: "transcripts/base.json", Required: boolPtr(true)},
|
||||||
}
|
}
|
||||||
|
|
||||||
archiveStageImpl, err := stage.Select("publish")
|
publishStageImpl, err := stage.Select("publish")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Select(publish) error = %v", err)
|
t.Fatalf("Select(publish) error = %v", err)
|
||||||
}
|
}
|
||||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveStageImpl}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishStageImpl}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||||
if err == nil || !strings.Contains(err.Error(), "required output source unavailable") {
|
if err == nil || !strings.Contains(err.Error(), "required output source unavailable") {
|
||||||
t.Fatalf("executeStages() error = %v, want required output source unavailable failure", err)
|
t.Fatalf("executeStages() error = %v, want required output source unavailable failure", err)
|
||||||
}
|
}
|
||||||
@@ -215,17 +215,17 @@ func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
|
|||||||
assertExists(t, artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
|
assertExists(t, artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
||||||
cfg, seed, _ := archiveStageCleanupFixture(t)
|
cfg, seed, _ := publishStageCleanupFixture(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
failKey := seed.sessionPrefix + "current/manifest.json"
|
failKey := seed.sessionPrefix + "current/manifest.json"
|
||||||
|
|
||||||
archiveStageImpl, err := stage.Select("publish")
|
publishStageImpl, err := stage.Select("publish")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Select(publish) error = %v", err)
|
t.Fatalf("Select(publish) error = %v", err)
|
||||||
}
|
}
|
||||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveStageImpl}, RunOptions{
|
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishStageImpl}, RunOptions{
|
||||||
Env: &Env{ObjectStore: &failKeyStore{delegate: &storage.FakeBackend{}, failKey: failKey}},
|
Env: &Env{ObjectStore: &failKeyStore{delegate: &storage.FakeBackend{}, failKey: failKey}},
|
||||||
})
|
})
|
||||||
if err == nil || !strings.Contains(err.Error(), "current manifest") {
|
if err == nil || !strings.Contains(err.Error(), "current manifest") {
|
||||||
@@ -236,17 +236,17 @@ func TestPostArchiveCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenCurrentPointerUploadFails(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenCurrentPointerUploadFails(t *testing.T) {
|
||||||
cfg, seed, _ := archiveStageCleanupFixture(t)
|
cfg, seed, _ := publishStageCleanupFixture(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
failKey := seed.sessionPrefix + "current/run_id.txt"
|
failKey := seed.sessionPrefix + "current/run_id.txt"
|
||||||
|
|
||||||
archiveStageImpl, err := stage.Select("publish")
|
publishStageImpl, err := stage.Select("publish")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Select(publish) error = %v", err)
|
t.Fatalf("Select(publish) error = %v", err)
|
||||||
}
|
}
|
||||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveStageImpl}, RunOptions{
|
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishStageImpl}, RunOptions{
|
||||||
Env: &Env{ObjectStore: &failKeyStore{delegate: &storage.FakeBackend{}, failKey: failKey}},
|
Env: &Env{ObjectStore: &failKeyStore{delegate: &storage.FakeBackend{}, failKey: failKey}},
|
||||||
})
|
})
|
||||||
if err == nil || !strings.Contains(err.Error(), "current run pointer") {
|
if err == nil || !strings.Contains(err.Error(), "current run pointer") {
|
||||||
@@ -320,7 +320,7 @@ func cleanupFixtureConfig(t *testing.T) (*config.Config, cleanupSeed) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, string) {
|
func publishStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
@@ -344,7 +344,7 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
|||||||
},
|
},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
writeArchiveFixtureRunFiles(
|
writePublishFixtureRunFiles(
|
||||||
t,
|
t,
|
||||||
seed.runWorkDir,
|
seed.runWorkDir,
|
||||||
artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID),
|
artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID),
|
||||||
@@ -367,7 +367,7 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
|||||||
return cfg, seed, runID
|
return cfg, seed, runID
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir, sessionRoot string) {
|
func writePublishFixtureRunFiles(t *testing.T, runWorkDir, sessionRoot string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
mustWriteFile(t, filepath.Join(runWorkDir, "prepare", "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
mustWriteFile(t, filepath.Join(runWorkDir, "prepare", "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
||||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
|
mustWriteFile(t, filepath.Join(runWorkDir, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
|
||||||
@@ -46,7 +46,7 @@ func loadRemoteLockStore(ctx context.Context, cfg *config.Config, store storage.
|
|||||||
if !exists {
|
if !exists {
|
||||||
return &config.PublishLockStore{}, key, nil
|
return &config.PublishLockStore{}, key, nil
|
||||||
}
|
}
|
||||||
tmp, err := downloadObjectToTemp(ctx, store, key, "narratio-locks-*.yml")
|
tmp, err := storage.DownloadObjectToTemp(ctx, store, key, "narratio-locks-*.yml")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, key, fmt.Errorf("download remote locks %q: %w", key, err)
|
return nil, key, fmt.Errorf("download remote locks %q: %w", key, err)
|
||||||
}
|
}
|
||||||
@@ -63,7 +63,7 @@ func loadRemoteLockStore(ctx context.Context, cfg *config.Config, store storage.
|
|||||||
}
|
}
|
||||||
|
|
||||||
func loadEffectiveLocks(ctx context.Context, cfg *config.Config, store storage.ObjectStore) (*effectiveLocks, error) {
|
func loadEffectiveLocks(ctx context.Context, cfg *config.Config, store storage.ObjectStore) (*effectiveLocks, error) {
|
||||||
staticLocks := staticArchiveLocks(cfg)
|
staticLocks := staticPublishLocks(cfg)
|
||||||
if store == nil {
|
if store == nil {
|
||||||
return &effectiveLocks{
|
return &effectiveLocks{
|
||||||
Static: staticLocks,
|
Static: staticLocks,
|
||||||
@@ -83,7 +83,7 @@ func loadEffectiveLocks(ctx context.Context, cfg *config.Config, store storage.O
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func staticArchiveLocks(cfg *config.Config) []config.PublishLockRule {
|
func staticPublishLocks(cfg *config.Config) []config.PublishLockRule {
|
||||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Publish == nil {
|
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Publish == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -27,23 +27,14 @@ func Restore(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
fs := flag.NewFlagSet("restore", flag.ContinueOnError)
|
fs := flag.NewFlagSet("restore", flag.ContinueOnError)
|
||||||
fs.SetOutput(out)
|
fs.SetOutput(out)
|
||||||
|
|
||||||
var pipelinePath string
|
var flags commonConfigFlags
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var dryRun bool
|
var dryRun bool
|
||||||
var force bool
|
var force bool
|
||||||
var includeAudio bool
|
var includeAudio bool
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.BoolVar(&dryRun, "dry-run", false, "plan restore actions without writing local files")
|
fs.BoolVar(&dryRun, "dry-run", false, "plan restore actions without writing local files")
|
||||||
fs.BoolVar(&force, "force", false, "overwrite local conflicts with remote state")
|
fs.BoolVar(&force, "force", false, "overwrite local conflicts with remote state")
|
||||||
fs.BoolVar(&includeAudio, "include-audio", false, "include archived session-level audio objects")
|
fs.BoolVar(&includeAudio, "include-audio", false, "include remote session-level audio objects")
|
||||||
fs.Usage = func() {
|
fs.Usage = func() {
|
||||||
_, _ = fmt.Fprintln(out, "Usage: narratio session restore <session_id> [--config <path>] [--campaign <id>] [--campaign-file <path>] [--session <path>] [--previous-session-id <value>] [--dry-run] [--force] [--include-audio]")
|
_, _ = fmt.Fprintln(out, "Usage: narratio session restore <session_id> [--config <path>] [--campaign <id>] [--campaign-file <path>] [--session <path>] [--previous-session-id <value>] [--dry-run] [--force] [--include-audio]")
|
||||||
_, _ = fmt.Fprintln(out)
|
_, _ = fmt.Fprintln(out)
|
||||||
@@ -57,25 +48,13 @@ func Restore(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
}
|
}
|
||||||
return fmt.Errorf("restore: invalid flags: %w", err)
|
return fmt.Errorf("restore: invalid flags: %w", err)
|
||||||
}
|
}
|
||||||
if positionalSessionID == "" {
|
if err := resolveParsedSessionID("restore", positionalSessionID, fs, &flags.sessionID); err != nil {
|
||||||
if err := applyParsedSessionIDArg("restore", fs, &sessionID); err != nil {
|
return err
|
||||||
return err
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("restore: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("restore", positionalSessionID, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
return fmt.Errorf("restore: session_id is required")
|
return fmt.Errorf("restore: session_id is required")
|
||||||
}
|
}
|
||||||
cfg, err := loadCommandConfig(ctx, pipelinePath, campaignPath, campaignFilePath, sessionPath, config.SessionLoadOptions{
|
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||||
SessionID: sessionID,
|
|
||||||
PreviousSessionID: previousSessionID,
|
|
||||||
})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("restore: %w", err)
|
return fmt.Errorf("restore: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ package app
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
@@ -12,7 +11,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
// RemoteCurrentState captures discovered committed remote archive state for one session.
|
// RemoteCurrentState captures discovered committed remote published current state for one session.
|
||||||
type RemoteCurrentState struct {
|
type RemoteCurrentState struct {
|
||||||
Bucket string
|
Bucket string
|
||||||
SessionPrefix string
|
SessionPrefix string
|
||||||
@@ -32,80 +31,24 @@ func discoverRemoteCurrentState(ctx context.Context, cfg *config.Config, store s
|
|||||||
return nil, fmt.Errorf("remote object store is required")
|
return nil, fmt.Errorf("remote object store is required")
|
||||||
}
|
}
|
||||||
|
|
||||||
bucket := artifacts.ResolveArchiveBucket(cfg, nil)
|
bucket := artifacts.ResolvePublishBucket(cfg, nil)
|
||||||
if strings.TrimSpace(bucket) == "" {
|
if strings.TrimSpace(bucket) == "" {
|
||||||
return nil, fmt.Errorf("archive bucket is required")
|
return nil, fmt.Errorf("publish bucket is required")
|
||||||
}
|
}
|
||||||
sessionPrefix, err := artifacts.ResolveArchiveSessionPrefix(cfg, nil)
|
sessionPrefix, err := artifacts.ResolvePublishSessionPrefix(cfg, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("resolve archive session prefix: %w", err)
|
return nil, fmt.Errorf("resolve publish session prefix: %w", err)
|
||||||
}
|
|
||||||
currentManifestKey, currentRunIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
|
||||||
|
|
||||||
exists, err := store.Exists(ctx, currentRunIDKey)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("check remote current run pointer %q: %w", currentRunIDKey, err)
|
|
||||||
}
|
|
||||||
if !exists {
|
|
||||||
return nil, fmt.Errorf("remote current run pointer missing: %q", currentRunIDKey)
|
|
||||||
}
|
|
||||||
|
|
||||||
runIDPath, err := downloadObjectToTemp(ctx, store, currentRunIDKey, "narratio-restore-current-run-id-*.txt")
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("download remote current run pointer %q: %w", currentRunIDKey, err)
|
|
||||||
}
|
|
||||||
defer func() { _ = os.Remove(runIDPath) }()
|
|
||||||
|
|
||||||
runIDData, err := os.ReadFile(runIDPath)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("read downloaded run pointer %q: %w", currentRunIDKey, err)
|
|
||||||
}
|
|
||||||
runID := strings.TrimSpace(string(runIDData))
|
|
||||||
if runID == "" {
|
|
||||||
return nil, fmt.Errorf("remote current run pointer %q is empty", currentRunIDKey)
|
|
||||||
}
|
|
||||||
|
|
||||||
exists, err = store.Exists(ctx, currentManifestKey)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("check remote current manifest %q: %w", currentManifestKey, err)
|
|
||||||
}
|
|
||||||
if !exists {
|
|
||||||
return nil, fmt.Errorf("remote current manifest missing: %q", currentManifestKey)
|
|
||||||
}
|
|
||||||
|
|
||||||
manifestPath, err := downloadObjectToTemp(ctx, store, currentManifestKey, "narratio-restore-current-manifest-*.json")
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("download remote current manifest %q: %w", currentManifestKey, err)
|
|
||||||
}
|
|
||||||
defer func() { _ = os.Remove(manifestPath) }()
|
|
||||||
|
|
||||||
manifestStore := &manifest.LocalStore{}
|
|
||||||
remoteManifest, err := manifestStore.Load(ctx, manifestPath)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("remote current manifest decode failed: %w", err)
|
|
||||||
}
|
}
|
||||||
|
currentManifestKey, currentRunIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
|
|
||||||
requestedSession := strings.TrimSpace(cfg.Session.SessionID)
|
requestedSession := strings.TrimSpace(cfg.Session.SessionID)
|
||||||
requestedCampaign := strings.TrimSpace(cfg.Session.Campaign)
|
requestedCampaign := strings.TrimSpace(cfg.Session.Campaign)
|
||||||
manifestSession := strings.TrimSpace(remoteManifest.SessionID)
|
current, err := artifacts.LoadCurrentState(ctx, store, sessionPrefix, artifacts.CurrentStateValidation{
|
||||||
manifestCampaign := strings.TrimSpace(remoteManifest.Campaign)
|
ExpectedSessionID: requestedSession,
|
||||||
|
ExpectedCampaign: requestedCampaign,
|
||||||
if manifestSession != requestedSession {
|
})
|
||||||
return nil, fmt.Errorf(
|
if err != nil {
|
||||||
"remote current manifest session_id %q does not match requested session_id %q",
|
return nil, fmt.Errorf("remote %w", err)
|
||||||
manifestSession,
|
|
||||||
requestedSession,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
if manifestCampaign == "" {
|
|
||||||
return nil, fmt.Errorf("remote current manifest campaign is required")
|
|
||||||
}
|
|
||||||
if manifestCampaign != requestedCampaign {
|
|
||||||
return nil, fmt.Errorf(
|
|
||||||
"remote current manifest campaign %q does not match requested campaign %q",
|
|
||||||
manifestCampaign,
|
|
||||||
requestedCampaign,
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return &RemoteCurrentState{
|
return &RemoteCurrentState{
|
||||||
@@ -113,27 +56,9 @@ func discoverRemoteCurrentState(ctx context.Context, cfg *config.Config, store s
|
|||||||
SessionPrefix: sessionPrefix,
|
SessionPrefix: sessionPrefix,
|
||||||
CurrentRunIDKey: currentRunIDKey,
|
CurrentRunIDKey: currentRunIDKey,
|
||||||
CurrentManifestKey: currentManifestKey,
|
CurrentManifestKey: currentManifestKey,
|
||||||
RunID: runID,
|
RunID: current.RunID,
|
||||||
SessionID: manifestSession,
|
SessionID: strings.TrimSpace(current.Manifest.SessionID),
|
||||||
Campaign: manifestCampaign,
|
Campaign: strings.TrimSpace(current.Manifest.Campaign),
|
||||||
Manifest: remoteManifest,
|
Manifest: current.Manifest,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func downloadObjectToTemp(ctx context.Context, store storage.ObjectStore, key, pattern string) (string, error) {
|
|
||||||
tmp, err := os.CreateTemp("", pattern)
|
|
||||||
if err != nil {
|
|
||||||
return "", fmt.Errorf("create temp file: %w", err)
|
|
||||||
}
|
|
||||||
path := tmp.Name()
|
|
||||||
if err := tmp.Close(); err != nil {
|
|
||||||
_ = os.Remove(path)
|
|
||||||
return "", fmt.Errorf("close temp file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := store.Download(ctx, key, path); err != nil {
|
|
||||||
_ = os.Remove(path)
|
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
return path, nil
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -102,7 +102,7 @@ func TestDiscoverRemoteCurrentStateSessionMismatchFails(t *testing.T) {
|
|||||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "wrong-session", cfg.Session.Campaign)})
|
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "wrong-session", cfg.Session.Campaign)})
|
||||||
|
|
||||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||||
if err == nil || !strings.Contains(err.Error(), "does not match requested session_id") {
|
if err == nil || !strings.Contains(err.Error(), "does not match expected session_id") {
|
||||||
t.Fatalf("error = %v, want session mismatch failure", err)
|
t.Fatalf("error = %v, want session mismatch failure", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -116,7 +116,7 @@ func TestDiscoverRemoteCurrentStateCampaignMismatchFails(t *testing.T) {
|
|||||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, "wrong-campaign")})
|
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, "wrong-campaign")})
|
||||||
|
|
||||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||||
if err == nil || !strings.Contains(err.Error(), "does not match requested campaign") {
|
if err == nil || !strings.Contains(err.Error(), "does not match expected campaign") {
|
||||||
t.Fatalf("error = %v, want campaign mismatch failure", err)
|
t.Fatalf("error = %v, want campaign mismatch failure", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -208,7 +208,7 @@ func restoreDiscoveryConfig() *config.Config {
|
|||||||
|
|
||||||
func restoreDiscoveryKeys(cfg *config.Config) (sessionPrefix, manifestKey, runIDKey string) {
|
func restoreDiscoveryKeys(cfg *config.Config) (sessionPrefix, manifestKey, runIDKey string) {
|
||||||
sessionPrefix = artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
sessionPrefix = artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
manifestKey, runIDKey = artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
manifestKey, runIDKey = artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
return sessionPrefix, manifestKey, runIDKey
|
return sessionPrefix, manifestKey, runIDKey
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/audio"
|
"gitea.maximumdirect.net/eric/narratio/internal/audio"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/fileops"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -114,10 +115,7 @@ func executeRestoreDownloadAction(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := os.Chmod(tmpPath, 0o644); err != nil {
|
if err := fileops.InstallDownloadedTempFile(tmpPath, safeLocalPath, 0o644); err != nil {
|
||||||
return fmt.Errorf("set file permissions: %w", err)
|
|
||||||
}
|
|
||||||
if err := os.Rename(tmpPath, safeLocalPath); err != nil {
|
|
||||||
return fmt.Errorf("install file atomically: %w", err)
|
return fmt.Errorf("install file atomically: %w", err)
|
||||||
}
|
}
|
||||||
removeTmp = false
|
removeTmp = false
|
||||||
|
|||||||
@@ -39,7 +39,7 @@ func TestExecuteRestoreNonDryRunRestoresDurableFiles(t *testing.T) {
|
|||||||
if stderr.Len() != 0 {
|
if stderr.Len() != 0 {
|
||||||
t.Fatalf("stderr = %q, want empty", stderr.String())
|
t.Fatalf("stderr = %q, want empty", stderr.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "Restored session archive for sample-campaign/2026-05-03") {
|
if !strings.Contains(stdout.String(), "Restored session state for sample-campaign/2026-05-03") {
|
||||||
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -427,7 +427,7 @@ func seedRestoreCommittedState(t *testing.T, fake *storage.FakeBackend, pipeline
|
|||||||
}
|
}
|
||||||
|
|
||||||
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
|
|
||||||
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
|
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
|
||||||
seedRestoreObject(fake, manifestKey, restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
|
seedRestoreObject(fake, manifestKey, restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
|
||||||
@@ -465,7 +465,7 @@ func seedRestorePreviousCurrent(t *testing.T, fake *storage.FakeBackend, cfg *co
|
|||||||
func seedRestorePreviousCurrentManifestOnly(t *testing.T, fake *storage.FakeBackend, cfg *config.Config) {
|
func seedRestorePreviousCurrentManifestOnly(t *testing.T, fake *storage.FakeBackend, cfg *config.Config) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
previousPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
previousPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
||||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(previousPrefix)
|
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(previousPrefix)
|
||||||
previousRunID := "20260426T010203Z-a1b2c3d4"
|
previousRunID := "20260426T010203Z-a1b2c3d4"
|
||||||
seedRestoreObject(fake, runIDKey, []byte(previousRunID+"\n"))
|
seedRestoreObject(fake, runIDKey, []byte(previousRunID+"\n"))
|
||||||
|
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package app
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
@@ -13,6 +14,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/pathsafe"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/previouscache"
|
"gitea.maximumdirect.net/eric/narratio/internal/previouscache"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -215,19 +217,17 @@ func joinWithinSessionRoot(sessionRoot, relative string) (string, error) {
|
|||||||
if strings.TrimSpace(sessionRoot) == "" {
|
if strings.TrimSpace(sessionRoot) == "" {
|
||||||
return "", fmt.Errorf("session root is required")
|
return "", fmt.Errorf("session root is required")
|
||||||
}
|
}
|
||||||
cleanRel := path.Clean(strings.TrimSpace(relative))
|
joined, err := pathsafe.JoinSlashRelativeUnderRoot(sessionRoot, filepath.ToSlash(strings.TrimSpace(relative)))
|
||||||
if cleanRel == "." || cleanRel == "" {
|
if err != nil {
|
||||||
return "", fmt.Errorf("relative path is required")
|
if errors.Is(err, pathsafe.ErrRelativePathRequired) {
|
||||||
|
return "", fmt.Errorf("relative path is required")
|
||||||
|
}
|
||||||
|
if errors.Is(err, pathsafe.ErrRelativePathEscape) || errors.Is(err, pathsafe.ErrRelativePathAbsolute) {
|
||||||
|
return "", fmt.Errorf("relative path escapes session root")
|
||||||
|
}
|
||||||
|
return "", fmt.Errorf("join relative path under session root: %w", err)
|
||||||
}
|
}
|
||||||
if cleanRel == ".." || strings.HasPrefix(cleanRel, "../") || strings.HasPrefix(cleanRel, "/") {
|
return joined, nil
|
||||||
return "", fmt.Errorf("relative path escapes session root")
|
|
||||||
}
|
|
||||||
abs := filepath.Clean(filepath.Join(sessionRoot, filepath.FromSlash(cleanRel)))
|
|
||||||
root := filepath.Clean(sessionRoot)
|
|
||||||
if abs != root && !strings.HasPrefix(abs, root+string(filepath.Separator)) {
|
|
||||||
return "", fmt.Errorf("resolved local path escapes session root")
|
|
||||||
}
|
|
||||||
return abs, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func buildPreviousCacheRestoreActions(
|
func buildPreviousCacheRestoreActions(
|
||||||
@@ -338,7 +338,7 @@ func classifyRestoreAction(
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return RestoreAction{}, fmt.Errorf("checksum local file: %w", err)
|
return RestoreAction{}, fmt.Errorf("checksum local file: %w", err)
|
||||||
}
|
}
|
||||||
remotePath, err := downloadObjectToTemp(ctx, store, action.RemoteKey, "narratio-restore-plan-remote-*.tmp")
|
remotePath, err := storage.DownloadObjectToTemp(ctx, store, action.RemoteKey, "narratio-restore-plan-remote-*.tmp")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return RestoreAction{}, fmt.Errorf("download remote object: %w", err)
|
return RestoreAction{}, fmt.Errorf("download remote object: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ func TestRestorePlanDefaultScope(t *testing.T) {
|
|||||||
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("# recap\n"))
|
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("# recap\n"))
|
||||||
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
|
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
|
||||||
seedRestoreObject(store, current.SessionPrefix+"runs/20260519T010203Z-a1b2/manifest.json", []byte("{}"))
|
seedRestoreObject(store, current.SessionPrefix+"runs/20260519T010203Z-a1b2/manifest.json", []byte("{}"))
|
||||||
seedRestoreObject(store, current.SessionPrefix+"logs/archive.log", []byte("log"))
|
seedRestoreObject(store, current.SessionPrefix+"logs/publish.log", []byte("log"))
|
||||||
|
|
||||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -325,7 +325,7 @@ func configureRestorePlanPreviousRequirement(cfg *config.Config, required bool)
|
|||||||
func restorePlanCurrentState(t *testing.T, cfg *config.Config) *RemoteCurrentState {
|
func restorePlanCurrentState(t *testing.T, cfg *config.Config) *RemoteCurrentState {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
sessionPrefix := artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
|
sessionPrefix := artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
return &RemoteCurrentState{
|
return &RemoteCurrentState{
|
||||||
Bucket: "test-bucket",
|
Bucket: "test-bucket",
|
||||||
SessionPrefix: sessionPrefix,
|
SessionPrefix: sessionPrefix,
|
||||||
|
|||||||
@@ -201,7 +201,7 @@ func writeRestoreSuccessSummary(out io.Writer, report *RestoreReport) error {
|
|||||||
if report == nil {
|
if report == nil {
|
||||||
return fmt.Errorf("restore report is required")
|
return fmt.Errorf("restore report is required")
|
||||||
}
|
}
|
||||||
if _, err := fmt.Fprintf(out, "Restored session archive for %s/%s\n", report.Campaign, report.SessionID); err != nil {
|
if _, err := fmt.Fprintf(out, "Restored session state for %s/%s\n", report.Campaign, report.SessionID); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if _, err := fmt.Fprintf(out, "Remote run: %s\n", report.RunID); err != nil {
|
if _, err := fmt.Fprintf(out, "Remote run: %s\n", report.RunID); err != nil {
|
||||||
|
|||||||
@@ -369,7 +369,7 @@ func TestExecuteRestoreNonDryRunForceExecutesPlan(t *testing.T) {
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "Restored session archive for sample-campaign/2026-05-03") {
|
if !strings.Contains(stdout.String(), "Restored session state for sample-campaign/2026-05-03") {
|
||||||
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
||||||
}
|
}
|
||||||
if stderr.Len() != 0 {
|
if stderr.Len() != 0 {
|
||||||
|
|||||||
@@ -1,128 +0,0 @@
|
|||||||
package app
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"flag"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"strings"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Resume continues execution from the first non-succeeded stage in the manifest.
|
|
||||||
func Resume(ctx context.Context, args []string, out io.Writer) error {
|
|
||||||
positionalSessionID, args := pullLeadingSessionID(args)
|
|
||||||
fs := flag.NewFlagSet("resume", flag.ContinueOnError)
|
|
||||||
fs.SetOutput(io.Discard)
|
|
||||||
|
|
||||||
var pipelinePath string
|
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var force bool
|
|
||||||
var selectedArtifacts artifactSelectionFlag
|
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.BoolVar(&force, "force", false, "force stage execution")
|
|
||||||
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute and publish (comma-separated or repeatable)")
|
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
|
||||||
return fmt.Errorf("resume: invalid flags: %w", err)
|
|
||||||
}
|
|
||||||
if positionalSessionID == "" {
|
|
||||||
if err := applyParsedSessionIDArg("resume", fs, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("resume: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("resume", positionalSessionID, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
|
||||||
return fmt.Errorf("resume: session_id is required")
|
|
||||||
}
|
|
||||||
cfg, err := loadCommandConfig(ctx, pipelinePath, campaignPath, campaignFilePath, sessionPath, config.SessionLoadOptions{
|
|
||||||
SessionID: sessionID,
|
|
||||||
PreviousSessionID: previousSessionID,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("resume: %w", err)
|
|
||||||
}
|
|
||||||
if err := config.Validate(cfg); err != nil {
|
|
||||||
return fmt.Errorf("resume: %w", err)
|
|
||||||
}
|
|
||||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("resume: invalid --artifacts: %w", err)
|
|
||||||
}
|
|
||||||
if err := validateSelectedArtifacts(cfg, normalizedArtifacts); err != nil {
|
|
||||||
return fmt.Errorf("resume: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
full := BuildFullPlan()
|
|
||||||
selected := full
|
|
||||||
if !force {
|
|
||||||
m, err := loadManifestIfPresent(ctx, cfg)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("resume: %w", err)
|
|
||||||
}
|
|
||||||
if m != nil {
|
|
||||||
start := firstNonSucceededIndex(full, m)
|
|
||||||
if start >= len(full) {
|
|
||||||
_, err := fmt.Fprintf(out, "narratio resume: session %s has no remaining stages\n", cfg.Session.SessionID)
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
selected = full[start:]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
summary, err := executeStagesFn(ctx, cfg, selected, RunOptions{
|
|
||||||
Force: force,
|
|
||||||
SelectedArtifacts: normalizedArtifacts,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("resume: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = fmt.Fprintf(
|
|
||||||
out,
|
|
||||||
"narratio resume: session %s; executed=%d skipped=%d; manifest=%s\n",
|
|
||||||
summary.SessionID,
|
|
||||||
len(summary.Executed),
|
|
||||||
len(summary.Skipped),
|
|
||||||
summary.ManifestPath,
|
|
||||||
)
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
func loadManifestIfPresent(ctx context.Context, cfg *config.Config) (*manifest.Manifest, error) {
|
|
||||||
path := artifacts.SessionManifestPathForCampaign(
|
|
||||||
cfg.Pipeline.Workspace.Root,
|
|
||||||
cfg.Session.Campaign,
|
|
||||||
cfg.Session.SessionID,
|
|
||||||
)
|
|
||||||
exists, err := fileExists(path)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("check manifest %q: %w", path, err)
|
|
||||||
}
|
|
||||||
if !exists {
|
|
||||||
return nil, nil
|
|
||||||
}
|
|
||||||
store := &manifest.LocalStore{}
|
|
||||||
m, err := store.Load(ctx, path)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("load manifest %q: %w", path, err)
|
|
||||||
}
|
|
||||||
return m, nil
|
|
||||||
}
|
|
||||||
@@ -5,55 +5,29 @@ import (
|
|||||||
"flag"
|
"flag"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"strings"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Run executes the pipeline plan and persists manifest state.
|
// Run executes the pipeline plan and persists manifest state.
|
||||||
func Run(ctx context.Context, args []string, out io.Writer) error {
|
func Run(ctx context.Context, args []string, out io.Writer) error {
|
||||||
positionalSessionID, args := pullLeadingSessionID(args)
|
|
||||||
fs := flag.NewFlagSet("run", flag.ContinueOnError)
|
fs := flag.NewFlagSet("run", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
|
|
||||||
var pipelinePath string
|
var flags commonConfigFlags
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var force bool
|
var force bool
|
||||||
var selectedArtifacts artifactSelectionFlag
|
var selectedArtifacts artifactSelectionFlag
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
||||||
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute and publish (comma-separated or repeatable)")
|
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute and publish (comma-separated or repeatable)")
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
if err := parseSessionAwareFlags("run", fs, args, &flags.sessionID); err != nil {
|
||||||
return fmt.Errorf("run: invalid flags: %w", err)
|
return err
|
||||||
}
|
}
|
||||||
if positionalSessionID == "" {
|
if flags.sessionID == "" {
|
||||||
if err := applyParsedSessionIDArg("run", fs, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("run: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("run", positionalSessionID, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
|
||||||
return fmt.Errorf("run: session_id is required")
|
return fmt.Errorf("run: session_id is required")
|
||||||
}
|
}
|
||||||
cfg, err := loadCommandConfig(ctx, pipelinePath, campaignPath, campaignFilePath, sessionPath, config.SessionLoadOptions{
|
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||||
SessionID: sessionID,
|
|
||||||
PreviousSessionID: previousSessionID,
|
|
||||||
})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("run: %w", err)
|
return fmt.Errorf("run: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,8 +1,12 @@
|
|||||||
package app
|
package app
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||||
)
|
)
|
||||||
@@ -43,13 +47,25 @@ func stageSucceeded(m *manifest.Manifest, name string) bool {
|
|||||||
return sr != nil && sr.Status == manifest.StatusSucceeded
|
return sr != nil && sr.Status == manifest.StatusSucceeded
|
||||||
}
|
}
|
||||||
|
|
||||||
func firstNonSucceededIndex(stages []stage.Stage, m *manifest.Manifest) int {
|
func loadManifestIfPresent(ctx context.Context, cfg *config.Config) (*manifest.Manifest, error) {
|
||||||
for i, s := range stages {
|
path := artifacts.SessionManifestPathForCampaign(
|
||||||
if !stageSucceeded(m, s.Name()) {
|
cfg.Pipeline.Workspace.Root,
|
||||||
return i
|
cfg.Session.Campaign,
|
||||||
}
|
cfg.Session.SessionID,
|
||||||
|
)
|
||||||
|
exists, err := fileExists(path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("check manifest %q: %w", path, err)
|
||||||
}
|
}
|
||||||
return len(stages)
|
if !exists {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
store := &manifest.LocalStore{}
|
||||||
|
m, err := store.Load(ctx, path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("load manifest %q: %w", path, err)
|
||||||
|
}
|
||||||
|
return m, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func canonicalStageNames() []string {
|
func canonicalStageNames() []string {
|
||||||
|
|||||||
@@ -8,18 +8,6 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestFirstNonSucceededIndex(t *testing.T) {
|
|
||||||
stages := BuildFullPlan()
|
|
||||||
m := manifest.New("2026-05-03", time.Now().UTC())
|
|
||||||
m.MarkStageSucceeded("prepare", time.Now().UTC(), nil)
|
|
||||||
m.MarkStageSucceeded("transcribe", time.Now().UTC(), nil)
|
|
||||||
|
|
||||||
got := firstNonSucceededIndex(stages, m)
|
|
||||||
if got != 2 {
|
|
||||||
t.Fatalf("firstNonSucceededIndex() = %d, want 2", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestDecideStageActions(t *testing.T) {
|
func TestDecideStageActions(t *testing.T) {
|
||||||
stages := BuildFullPlan()[:2]
|
stages := BuildFullPlan()[:2]
|
||||||
m := manifest.New("2026-05-03", time.Now().UTC())
|
m := manifest.New("2026-05-03", time.Now().UTC())
|
||||||
|
|||||||
@@ -23,19 +23,10 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
fs := flag.NewFlagSet("run-stage", flag.ContinueOnError)
|
fs := flag.NewFlagSet("run-stage", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
|
|
||||||
var pipelinePath string
|
var flags commonConfigFlags
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var force bool
|
var force bool
|
||||||
var selectedArtifacts artifactSelectionFlag
|
var selectedArtifacts artifactSelectionFlag
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
||||||
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute or publish (comma-separated or repeatable)")
|
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute or publish (comma-separated or repeatable)")
|
||||||
|
|
||||||
@@ -47,16 +38,21 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
case 2:
|
case 2:
|
||||||
stageName = strings.TrimSpace(fs.Arg(0))
|
stageName = strings.TrimSpace(fs.Arg(0))
|
||||||
positionalSessionID = strings.TrimSpace(fs.Arg(1))
|
positionalSessionID = strings.TrimSpace(fs.Arg(1))
|
||||||
|
case 1:
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("run-stage: expected stage name and session_id")
|
||||||
|
}
|
||||||
|
stageName = strings.TrimSpace(fs.Arg(0))
|
||||||
default:
|
default:
|
||||||
return fmt.Errorf("run-stage: expected stage name and session_id")
|
return fmt.Errorf("run-stage: expected stage name and session_id")
|
||||||
}
|
}
|
||||||
} else if fs.NArg() != 0 {
|
} else if fs.NArg() != 0 {
|
||||||
return fmt.Errorf("run-stage: unexpected positional arguments")
|
return fmt.Errorf("run-stage: unexpected positional arguments")
|
||||||
}
|
}
|
||||||
if err := applyPositionalSessionID("run-stage", positionalSessionID, &sessionID); err != nil {
|
if err := applyPositionalSessionID("run-stage", positionalSessionID, &flags.sessionID); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
return fmt.Errorf("run-stage: session_id is required")
|
return fmt.Errorf("run-stage: session_id is required")
|
||||||
}
|
}
|
||||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
||||||
@@ -70,12 +66,12 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
summary, err := runSingleStageCommand(ctx, singleStageCommand{
|
summary, err := runSingleStageCommand(ctx, singleStageCommand{
|
||||||
CommandName: "run-stage",
|
CommandName: "run-stage",
|
||||||
StageName: stageName,
|
StageName: stageName,
|
||||||
PipelinePath: pipelinePath,
|
PipelinePath: flags.pipelinePath,
|
||||||
CampaignPath: campaignPath,
|
CampaignPath: flags.campaignPath,
|
||||||
CampaignFilePath: campaignFilePath,
|
CampaignFilePath: flags.campaignFilePath,
|
||||||
SessionPath: sessionPath,
|
SessionPath: flags.sessionPath,
|
||||||
SessionID: sessionID,
|
SessionID: flags.sessionID,
|
||||||
PreviousSessionID: previousSessionID,
|
PreviousSessionID: flags.previousSessionID,
|
||||||
Force: force,
|
Force: force,
|
||||||
SelectedArtifacts: normalizedArtifacts,
|
SelectedArtifacts: normalizedArtifacts,
|
||||||
})
|
})
|
||||||
@@ -97,40 +93,18 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
|
|
||||||
// Analyze force-runs the analyze stage.
|
// Analyze force-runs the analyze stage.
|
||||||
func Analyze(ctx context.Context, args []string, out io.Writer) error {
|
func Analyze(ctx context.Context, args []string, out io.Writer) error {
|
||||||
positionalSessionID, args := pullLeadingSessionID(args)
|
|
||||||
fs := flag.NewFlagSet("analyze", flag.ContinueOnError)
|
fs := flag.NewFlagSet("analyze", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
|
|
||||||
var pipelinePath string
|
var flags commonConfigFlags
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var selectedArtifacts artifactSelectionFlag
|
var selectedArtifacts artifactSelectionFlag
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute during analyze (comma-separated or repeatable)")
|
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute during analyze (comma-separated or repeatable)")
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
if err := parseSessionAwareFlags("analyze", fs, args, &flags.sessionID); err != nil {
|
||||||
return fmt.Errorf("analyze: invalid flags: %w", err)
|
return err
|
||||||
}
|
}
|
||||||
if positionalSessionID == "" {
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
if err := applyParsedSessionIDArg("analyze", fs, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("analyze: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("analyze", positionalSessionID, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
|
||||||
return fmt.Errorf("analyze: session_id is required")
|
return fmt.Errorf("analyze: session_id is required")
|
||||||
}
|
}
|
||||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
||||||
@@ -141,12 +115,12 @@ func Analyze(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
summary, err := runSingleStageCommand(ctx, singleStageCommand{
|
summary, err := runSingleStageCommand(ctx, singleStageCommand{
|
||||||
CommandName: "analyze",
|
CommandName: "analyze",
|
||||||
StageName: "analyze",
|
StageName: "analyze",
|
||||||
PipelinePath: pipelinePath,
|
PipelinePath: flags.pipelinePath,
|
||||||
CampaignPath: campaignPath,
|
CampaignPath: flags.campaignPath,
|
||||||
CampaignFilePath: campaignFilePath,
|
CampaignFilePath: flags.campaignFilePath,
|
||||||
SessionPath: sessionPath,
|
SessionPath: flags.sessionPath,
|
||||||
SessionID: sessionID,
|
SessionID: flags.sessionID,
|
||||||
PreviousSessionID: previousSessionID,
|
PreviousSessionID: flags.previousSessionID,
|
||||||
Force: true,
|
Force: true,
|
||||||
SelectedArtifacts: normalizedArtifacts,
|
SelectedArtifacts: normalizedArtifacts,
|
||||||
})
|
})
|
||||||
@@ -166,40 +140,18 @@ func Analyze(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
|
|
||||||
// Publish force-runs the publish stage.
|
// Publish force-runs the publish stage.
|
||||||
func Publish(ctx context.Context, args []string, out io.Writer) error {
|
func Publish(ctx context.Context, args []string, out io.Writer) error {
|
||||||
positionalSessionID, args := pullLeadingSessionID(args)
|
|
||||||
fs := flag.NewFlagSet("publish", flag.ContinueOnError)
|
fs := flag.NewFlagSet("publish", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
|
|
||||||
var pipelinePath string
|
var flags commonConfigFlags
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var selectedArtifacts artifactSelectionFlag
|
var selectedArtifacts artifactSelectionFlag
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to publish (comma-separated or repeatable)")
|
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to publish (comma-separated or repeatable)")
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
if err := parseSessionAwareFlags("publish", fs, args, &flags.sessionID); err != nil {
|
||||||
return fmt.Errorf("publish: invalid flags: %w", err)
|
return err
|
||||||
}
|
}
|
||||||
if positionalSessionID == "" {
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
if err := applyParsedSessionIDArg("publish", fs, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("publish: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("publish", positionalSessionID, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
|
||||||
return fmt.Errorf("publish: session_id is required")
|
return fmt.Errorf("publish: session_id is required")
|
||||||
}
|
}
|
||||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
||||||
@@ -210,12 +162,12 @@ func Publish(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
summary, err := runSingleStageCommand(ctx, singleStageCommand{
|
summary, err := runSingleStageCommand(ctx, singleStageCommand{
|
||||||
CommandName: "publish",
|
CommandName: "publish",
|
||||||
StageName: "publish",
|
StageName: "publish",
|
||||||
PipelinePath: pipelinePath,
|
PipelinePath: flags.pipelinePath,
|
||||||
CampaignPath: campaignPath,
|
CampaignPath: flags.campaignPath,
|
||||||
CampaignFilePath: campaignFilePath,
|
CampaignFilePath: flags.campaignFilePath,
|
||||||
SessionPath: sessionPath,
|
SessionPath: flags.sessionPath,
|
||||||
SessionID: sessionID,
|
SessionID: flags.sessionID,
|
||||||
PreviousSessionID: previousSessionID,
|
PreviousSessionID: flags.previousSessionID,
|
||||||
Force: true,
|
Force: true,
|
||||||
SelectedArtifacts: normalizedArtifacts,
|
SelectedArtifacts: normalizedArtifacts,
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
func TestRunContinuesAfterCompletedStages(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||||
@@ -32,12 +32,12 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
|||||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
err := Resume(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
err := Run(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "executed=7 skipped=0") {
|
if !strings.Contains(out.String(), "executed=7 skipped=2") {
|
||||||
t.Fatalf("output = %q, want executed=7 skipped=0", out.String())
|
t.Fatalf("output = %q, want executed=7 skipped=2", out.String())
|
||||||
}
|
}
|
||||||
|
|
||||||
loaded, err := store.Load(context.Background(), manifestPath)
|
loaded, err := store.Load(context.Background(), manifestPath)
|
||||||
@@ -49,7 +49,7 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResumeNoRemainingStages(t *testing.T) {
|
func TestRunNoRemainingStagesRecordsSkippedStages(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||||
@@ -64,20 +64,20 @@ func TestResumeNoRemainingStages(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
err := Resume(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
err := Run(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "has no remaining stages") {
|
if !strings.Contains(out.String(), "executed=0 skipped=9") {
|
||||||
t.Fatalf("output = %q, want no remaining stages", out.String())
|
t.Fatalf("output = %q, want executed=0 skipped=9", out.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResumeForceRerunsSucceeded(t *testing.T) {
|
func TestRunForceRerunsSucceeded(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
_, _ = w.Write([]byte(`{"source":"resume-force-test","segments":[{"speaker":"alice"}]}`))
|
_, _ = w.Write([]byte(`{"source":"run-force-test","segments":[{"speaker":"alice"}]}`))
|
||||||
}))
|
}))
|
||||||
defer srv.Close()
|
defer srv.Close()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||||
@@ -93,9 +93,9 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
err := Resume(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--force"}, &out)
|
err := Run(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--force"}, &out)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "executed=9 skipped=0") {
|
if !strings.Contains(out.String(), "executed=9 skipped=0") {
|
||||||
t.Fatalf("output = %q, want forced full rerun", out.String())
|
t.Fatalf("output = %q, want forced full rerun", out.String())
|
||||||
@@ -166,7 +166,7 @@ func TestRunStageSkipAndForce(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunStageForceMarksDownstreamStaleAndResumeContinuesFromStale(t *testing.T) {
|
func TestRunStageForceMarksDownstreamStaleAndRunContinuesFromStale(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||||
@@ -203,12 +203,12 @@ func TestRunStageForceMarksDownstreamStaleAndResumeContinuesFromStale(t *testing
|
|||||||
}
|
}
|
||||||
|
|
||||||
out.Reset()
|
out.Reset()
|
||||||
err = Resume(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
err = Run(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "executed=5 skipped=0") {
|
if !strings.Contains(out.String(), "executed=5 skipped=4") {
|
||||||
t.Fatalf("output = %q, want resume to execute normalize..notify", out.String())
|
t.Fatalf("output = %q, want run to execute stale downstream stages", out.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -11,7 +11,6 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/notify"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/notify"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/scriptorium"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/scriptorium"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/seriatim"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/seriatim"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/whisperx"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/whisperx"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
@@ -83,9 +82,6 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
|||||||
if env.Scriptorium == nil {
|
if env.Scriptorium == nil {
|
||||||
env.Scriptorium = scriptorium.NewSubprocessRunner()
|
env.Scriptorium = scriptorium.NewSubprocessRunner()
|
||||||
}
|
}
|
||||||
if env.Storage == nil {
|
|
||||||
env.Storage = &storage.NoopBackend{}
|
|
||||||
}
|
|
||||||
if env.ObjectStore == nil && needsObjectStoreForRun(env.Config, stages) {
|
if env.ObjectStore == nil && needsObjectStoreForRun(env.Config, stages) {
|
||||||
objectStore, err := newCommandObjectStore(ctx, env.Config, nil)
|
objectStore, err := newCommandObjectStore(ctx, env.Config, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -96,7 +92,7 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
|||||||
if needsRemoteLocksForRun(env.Config, stages) {
|
if needsRemoteLocksForRun(env.Config, stages) {
|
||||||
locks, err := loadEffectiveLocks(ctx, env.Config, env.ObjectStore)
|
locks, err := loadEffectiveLocks(ctx, env.Config, env.ObjectStore)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("load remote archive locks: %w", err)
|
return nil, fmt.Errorf("load remote publish locks: %w", err)
|
||||||
}
|
}
|
||||||
applyEffectiveLocks(env.Config, locks.All)
|
applyEffectiveLocks(env.Config, locks.All)
|
||||||
}
|
}
|
||||||
@@ -234,14 +230,14 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
|||||||
env.Logger.Info("stage succeeded", "stage", s.Name())
|
env.Logger.Info("stage succeeded", "stage", s.Name())
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := runPostArchiveCleanup(ctx, env, manifestPath, m, executed); err != nil {
|
if err := runPostPublishCleanup(ctx, env, manifestPath, m, executed); err != nil {
|
||||||
failedAt := nowUTC()
|
failedAt := nowUTC()
|
||||||
runManifest.MarkFailed(failedAt, err.Error())
|
runManifest.MarkFailed(failedAt, err.Error())
|
||||||
syncRunManifestIdentityFromSession(m, runManifest)
|
syncRunManifestIdentityFromSession(m, runManifest)
|
||||||
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
|
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
|
||||||
return nil, fmt.Errorf("post-archive cleanup failed (%v) and run-manifest save failed (%v)", err, saveErr)
|
return nil, fmt.Errorf("post-publish cleanup failed (%v) and run-manifest save failed (%v)", err, saveErr)
|
||||||
}
|
}
|
||||||
return nil, fmt.Errorf("post-archive cleanup: %w", err)
|
return nil, fmt.Errorf("post-publish cleanup: %w", err)
|
||||||
}
|
}
|
||||||
completedAt := nowUTC()
|
completedAt := nowUTC()
|
||||||
runManifest.MarkSucceeded(completedAt)
|
runManifest.MarkSucceeded(completedAt)
|
||||||
|
|||||||
@@ -248,7 +248,7 @@ func TestNeedsObjectStoreForRunPrepareWithPreviousRequirements(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteStagesArchiveSkipsRequiredUnselectedConfiguredPromotion(t *testing.T) {
|
func TestExecuteStagesPublishSkipsRequiredUnselectedConfiguredOutput(t *testing.T) {
|
||||||
cfg := testConfig(t)
|
cfg := testConfig(t)
|
||||||
cfg.Pipeline.Storage.S3 = &config.StorageS3Config{
|
cfg.Pipeline.Storage.S3 = &config.StorageS3Config{
|
||||||
Bucket: "my-dnd-archive",
|
Bucket: "my-dnd-archive",
|
||||||
@@ -283,7 +283,7 @@ func TestExecuteStagesArchiveSkipsRequiredUnselectedConfiguredPromotion(t *testi
|
|||||||
t.Fatalf("Save manifest error = %v", err)
|
t.Fatalf("Save manifest error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
archiveStageImpl, err := stage.Select("publish")
|
publishStageImpl, err := stage.Select("publish")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Select(publish) error = %v", err)
|
t.Fatalf("Select(publish) error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -293,7 +293,7 @@ func TestExecuteStagesArchiveSkipsRequiredUnselectedConfiguredPromotion(t *testi
|
|||||||
cfg,
|
cfg,
|
||||||
[]stage.Stage{
|
[]stage.Stage{
|
||||||
selectedAnalyzeArtifactStage{expected: []string{"player_handout"}},
|
selectedAnalyzeArtifactStage{expected: []string{"player_handout"}},
|
||||||
archiveStageImpl,
|
publishStageImpl,
|
||||||
},
|
},
|
||||||
RunOptions{
|
RunOptions{
|
||||||
SelectedArtifacts: []string{"player_handout"},
|
SelectedArtifacts: []string{"player_handout"},
|
||||||
@@ -304,7 +304,7 @@ func TestExecuteStagesArchiveSkipsRequiredUnselectedConfiguredPromotion(t *testi
|
|||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
if len(summary.Executed) != 2 || summary.Executed[0] != "analyze" || summary.Executed[1] != "publish" {
|
if len(summary.Executed) != 2 || summary.Executed[0] != "analyze" || summary.Executed[1] != "publish" {
|
||||||
t.Fatalf("executed = %#v, want analyze and archive", summary.Executed)
|
t.Fatalf("executed = %#v, want analyze and publish", summary.Executed)
|
||||||
}
|
}
|
||||||
|
|
||||||
loadedManifest, err := store.Load(context.Background(), summary.ManifestPath)
|
loadedManifest, err := store.Load(context.Background(), summary.ManifestPath)
|
||||||
@@ -321,7 +321,7 @@ func TestExecuteStagesArchiveSkipsRequiredUnselectedConfiguredPromotion(t *testi
|
|||||||
t.Fatalf("skipped item = %#v, want object", skipped[0])
|
t.Fatalf("skipped item = %#v, want object", skipped[0])
|
||||||
}
|
}
|
||||||
if item["source"] != "narratio.artifact.session_recap" || item["dest"] != "artifacts/session_recap.md" || item["required"] != true {
|
if item["source"] != "narratio.artifact.session_recap" || item["dest"] != "artifacts/session_recap.md" || item["required"] != true {
|
||||||
t.Fatalf("skipped item = %#v, want required session_recap promotion", item)
|
t.Fatalf("skipped item = %#v, want required session_recap published output", item)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -427,10 +427,10 @@ func TestExecuteStagesPlaceholderSuccessUpdatesManifest(t *testing.T) {
|
|||||||
}
|
}
|
||||||
if name == "publish" {
|
if name == "publish" {
|
||||||
if sr.Metadata == nil || sr.Metadata["stage"] != "publish" {
|
if sr.Metadata == nil || sr.Metadata["stage"] != "publish" {
|
||||||
t.Fatalf("archive metadata missing stage=publish: %#v", sr.Metadata)
|
t.Fatalf("publish metadata missing stage=publish: %#v", sr.Metadata)
|
||||||
}
|
}
|
||||||
if sr.Metadata["skipped"] != true {
|
if sr.Metadata["skipped"] != true {
|
||||||
t.Fatalf("archive metadata missing skipped=true for test config without archive section: %#v", sr.Metadata)
|
t.Fatalf("publish metadata missing skipped=true for test config without publish section: %#v", sr.Metadata)
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -729,7 +729,7 @@ func TestExecuteStagesRunManifestRecordsSkippedStage(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteStagesRunLocalArtifactsAndCanonicalPromotion(t *testing.T) {
|
func TestExecuteStagesRunLocalArtifactsAndCanonicalSync(t *testing.T) {
|
||||||
cfg := testConfig(t)
|
cfg := testConfig(t)
|
||||||
stages := []stage.Stage{
|
stages := []stage.Stage{
|
||||||
BuildFullPlan()[0], // prepare
|
BuildFullPlan()[0], // prepare
|
||||||
@@ -778,7 +778,7 @@ func TestExecuteStagesRunLocalArtifactsAndCanonicalPromotion(t *testing.T) {
|
|||||||
}
|
}
|
||||||
for _, p := range canonicalChecks {
|
for _, p := range canonicalChecks {
|
||||||
if _, statErr := os.Stat(p); statErr != nil {
|
if _, statErr := os.Stat(p); statErr != nil {
|
||||||
t.Fatalf("canonical promoted artifact missing at %q: %v", p, statErr)
|
t.Fatalf("canonical published artifact missing at %q: %v", p, statErr)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -864,7 +864,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
|||||||
{name: "merge", env: &Env{Seriatim: &seriatim.FakeRunner{Err: errors.New("merge fail")}}},
|
{name: "merge", env: &Env{Seriatim: &seriatim.FakeRunner{Err: errors.New("merge fail")}}},
|
||||||
{name: "polish", env: &Env{Audita: &audita.FakeRunner{Err: errors.New("polish fail")}}},
|
{name: "polish", env: &Env{Audita: &audita.FakeRunner{Err: errors.New("polish fail")}}},
|
||||||
{name: "analyze", env: &Env{Scriptorium: &scriptorium.FakeRunner{RunErr: errors.New("analyze fail")}}},
|
{name: "analyze", env: &Env{Scriptorium: &scriptorium.FakeRunner{RunErr: errors.New("analyze fail")}}},
|
||||||
{name: "publish", env: &Env{ObjectStore: &storage.FakeBackend{UploadErr: errors.New("archive fail")}}},
|
{name: "publish", env: &Env{ObjectStore: &storage.FakeBackend{UploadErr: errors.New("publish fail")}}},
|
||||||
{name: "notify", env: &Env{Notifier: ¬ify.FakeSender{Err: errors.New("notify fail")}}},
|
{name: "notify", env: &Env{Notifier: ¬ify.FakeSender{Err: errors.New("notify fail")}}},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -961,13 +961,13 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
|||||||
runID := "20260516T010203Z-0a1b2c3d"
|
runID := "20260516T010203Z-0a1b2c3d"
|
||||||
runWorkDir := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
runWorkDir := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||||
if err := os.MkdirAll(filepath.Join(runWorkDir, "inputs"), 0o755); err != nil {
|
if err := os.MkdirAll(filepath.Join(runWorkDir, "inputs"), 0o755); err != nil {
|
||||||
t.Fatalf("mkdir archive inputs dir: %v", err)
|
t.Fatalf("mkdir publish inputs dir: %v", err)
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(filepath.Join(runWorkDir, "inputs", "session.yml"), []byte("session_id: 2026-05-03\n"), 0o644); err != nil {
|
if err := os.WriteFile(filepath.Join(runWorkDir, "inputs", "session.yml"), []byte("session_id: 2026-05-03\n"), 0o644); err != nil {
|
||||||
t.Fatalf("write archive fixture session.yml: %v", err)
|
t.Fatalf("write publish fixture session.yml: %v", err)
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(filepath.Join(runWorkDir, "manifest.json"), []byte("{}\n"), 0o644); err != nil {
|
if err := os.WriteFile(filepath.Join(runWorkDir, "manifest.json"), []byte("{}\n"), 0o644); err != nil {
|
||||||
t.Fatalf("write archive fixture manifest.json: %v", err)
|
t.Fatalf("write publish fixture manifest.json: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
||||||
@@ -981,7 +981,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
|||||||
seed.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
seed.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
||||||
}
|
}
|
||||||
if err := tc.env.ManifestStore.Save(context.Background(), manifestPathFor(cfg), seed); err != nil {
|
if err := tc.env.ManifestStore.Save(context.Background(), manifestPathFor(cfg), seed); err != nil {
|
||||||
t.Fatalf("seed archive manifest: %v", err)
|
t.Fatalf("seed publish manifest: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -41,3 +41,55 @@ func applyParsedSessionIDArg(command string, fs *flag.FlagSet, sessionID *string
|
|||||||
return fmt.Errorf("%s: unexpected positional arguments", command)
|
return fmt.Errorf("%s: unexpected positional arguments", command)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func resolveParsedSessionID(command, positionalSessionID string, fs *flag.FlagSet, sessionID *string) error {
|
||||||
|
if strings.TrimSpace(positionalSessionID) == "" {
|
||||||
|
return applyParsedSessionIDArg(command, fs, sessionID)
|
||||||
|
}
|
||||||
|
if fs.NArg() != 0 {
|
||||||
|
return fmt.Errorf("%s: unexpected positional arguments", command)
|
||||||
|
}
|
||||||
|
return applyPositionalSessionID(command, positionalSessionID, sessionID)
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseSessionAwareFlags(command string, fs *flag.FlagSet, args []string, sessionID *string) error {
|
||||||
|
positionalSessionID, args := pullLeadingSessionID(args)
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return fmt.Errorf("%s: invalid flags: %w", command, err)
|
||||||
|
}
|
||||||
|
return resolveParsedSessionID(command, positionalSessionID, fs, sessionID)
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseSessionIDAndOnePositionalArg(command, argName string, fs *flag.FlagSet, args []string, sessionID *string) (string, error) {
|
||||||
|
var positionalSessionID string
|
||||||
|
value := ""
|
||||||
|
if len(args) >= 2 && !isCLIFlagToken(args[0]) && !isCLIFlagToken(args[1]) {
|
||||||
|
positionalSessionID = strings.TrimSpace(args[0])
|
||||||
|
value = strings.TrimSpace(args[1])
|
||||||
|
args = append([]string(nil), args[2:]...)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return "", fmt.Errorf("%s: invalid flags: %w", command, err)
|
||||||
|
}
|
||||||
|
if value == "" {
|
||||||
|
switch fs.NArg() {
|
||||||
|
case 2:
|
||||||
|
positionalSessionID = strings.TrimSpace(fs.Arg(0))
|
||||||
|
value = strings.TrimSpace(fs.Arg(1))
|
||||||
|
case 1:
|
||||||
|
if strings.TrimSpace(*sessionID) == "" {
|
||||||
|
return "", fmt.Errorf("%s: expected session_id and %s", command, argName)
|
||||||
|
}
|
||||||
|
value = strings.TrimSpace(fs.Arg(0))
|
||||||
|
default:
|
||||||
|
return "", fmt.Errorf("%s: expected session_id and %s", command, argName)
|
||||||
|
}
|
||||||
|
} else if fs.NArg() != 0 {
|
||||||
|
return "", fmt.Errorf("%s: unexpected positional arguments", command)
|
||||||
|
}
|
||||||
|
if err := applyPositionalSessionID(command, positionalSessionID, sessionID); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return value, nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -71,15 +71,36 @@ func TestExecutePositionalSessionIDMismatchFails(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteSessionIDFlagFails(t *testing.T) {
|
func TestExecuteSessionIDFlagMismatchFails(t *testing.T) {
|
||||||
var stdout bytes.Buffer
|
var stdout bytes.Buffer
|
||||||
var stderr bytes.Buffer
|
var stderr bytes.Buffer
|
||||||
code := Execute([]string{"session", "status", "2026-05-03", "--session-id", "2026-05-04"}, &stdout, &stderr)
|
code := Execute([]string{"session", "status", "2026-05-03", "--session-id", "2026-05-04"}, &stdout, &stderr)
|
||||||
if code == 0 {
|
if code == 0 {
|
||||||
t.Fatal("exit code = 0, want non-zero")
|
t.Fatal("exit code = 0, want non-zero")
|
||||||
}
|
}
|
||||||
if !strings.Contains(stderr.String(), "flag provided but not defined: -session-id") {
|
if !strings.Contains(stderr.String(), "does not match expected session id") {
|
||||||
t.Fatalf("stderr = %q, want invalid --session-id flag", stderr.String())
|
t.Fatalf("stderr = %q, want positional/flag mismatch", stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteSessionIDFlagAcceptedWithoutPositional(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{
|
||||||
|
"session", "status",
|
||||||
|
"--session-id", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "Session: 2026-05-03") {
|
||||||
|
t.Fatalf("stdout = %q, want status output", stdout.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -141,8 +162,8 @@ func TestExecuteWorkflowCommandsAcceptPositionalSessionID(t *testing.T) {
|
|||||||
wantForce bool
|
wantForce bool
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "resume",
|
name: "run",
|
||||||
args: []string{"resume", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath},
|
args: []string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath},
|
||||||
wantStage: "prepare",
|
wantStage: "prepare",
|
||||||
wantForce: false,
|
wantForce: false,
|
||||||
},
|
},
|
||||||
@@ -212,7 +233,7 @@ func TestExecuteSessionSubcommandsAcceptPositionalSessionID(t *testing.T) {
|
|||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03")
|
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03")
|
||||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
fake.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
fake.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||||
fake.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "2026-05-03", "sample-campaign")})
|
fake.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "2026-05-03", "sample-campaign")})
|
||||||
var storeInitCalls int
|
var storeInitCalls int
|
||||||
|
|||||||
237
internal/artifactpolicy/policy.go
Normal file
237
internal/artifactpolicy/policy.go
Normal file
@@ -0,0 +1,237 @@
|
|||||||
|
package artifactpolicy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/pathsafe"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
SourceBoundsSession = "narratio.bounds.session"
|
||||||
|
|
||||||
|
configuredSourcePrefix = "narratio.artifact."
|
||||||
|
previousConfiguredSrcPrefix = "narratio.previous_session.artifact."
|
||||||
|
)
|
||||||
|
|
||||||
|
var configuredSourceRE = regexp.MustCompile(`^narratio\.artifact\.([a-z][a-z0-9_]*)$`)
|
||||||
|
var previousSourceRE = regexp.MustCompile(`^narratio\.previous_session\.artifact\.([a-z][a-z0-9_]*)$`)
|
||||||
|
|
||||||
|
var (
|
||||||
|
ErrUnsupportedScriptoriumInputSource = errors.New("unsupported scriptorium input source")
|
||||||
|
ErrInvalidPreviousSessionSource = errors.New("invalid previous-session source format")
|
||||||
|
)
|
||||||
|
|
||||||
|
type SourceKind string
|
||||||
|
|
||||||
|
const (
|
||||||
|
SourceKindBuiltIn SourceKind = "built_in"
|
||||||
|
SourceKindConfiguredArtifact SourceKind = "configured_artifact"
|
||||||
|
SourceKindPreviousArtifact SourceKind = "previous_session_configured_artifact"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Source describes one normalized artifact source identifier.
|
||||||
|
type Source struct {
|
||||||
|
ID string
|
||||||
|
Kind SourceKind
|
||||||
|
ConfiguredKey string
|
||||||
|
}
|
||||||
|
|
||||||
|
// ScriptoriumInputSourceDescriptor describes one validated Scriptorium input source.
|
||||||
|
type ScriptoriumInputSourceDescriptor struct {
|
||||||
|
Source Source
|
||||||
|
PreviousSession *PreviousSessionSourceDescriptor
|
||||||
|
}
|
||||||
|
|
||||||
|
// PreviousSessionSourceDescriptor describes one canonical previous-session input source.
|
||||||
|
type PreviousSessionSourceDescriptor struct {
|
||||||
|
SourceID string
|
||||||
|
ConfiguredKey string
|
||||||
|
ConfiguredSourceID string
|
||||||
|
}
|
||||||
|
|
||||||
|
// UnknownConfiguredArtifactError reports a source that references an undefined configured artifact key.
|
||||||
|
type UnknownConfiguredArtifactError struct {
|
||||||
|
ConfiguredKey string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *UnknownConfiguredArtifactError) Error() string {
|
||||||
|
return fmt.Sprintf("references unknown artifact %q", e.ConfiguredKey)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ConfiguredSourceID converts a configured artifact key into source id form.
|
||||||
|
func ConfiguredSourceID(key string) string {
|
||||||
|
return configuredSourcePrefix + strings.TrimSpace(key)
|
||||||
|
}
|
||||||
|
|
||||||
|
// PreviousSessionSourceID converts a configured artifact key into previous-session source id form.
|
||||||
|
func PreviousSessionSourceID(key string) string {
|
||||||
|
return previousConfiguredSrcPrefix + strings.TrimSpace(key)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseConfiguredSource extracts configured key from narratio.artifact.<key>.
|
||||||
|
func ParseConfiguredSource(source string) (string, bool) {
|
||||||
|
matches := configuredSourceRE.FindStringSubmatch(strings.TrimSpace(source))
|
||||||
|
if len(matches) != 2 {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
return matches[1], true
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParsePreviousSessionSource extracts configured key from narratio.previous_session.artifact.<key>.
|
||||||
|
func ParsePreviousSessionSource(source string) (string, bool) {
|
||||||
|
matches := previousSourceRE.FindStringSubmatch(strings.TrimSpace(source))
|
||||||
|
if len(matches) != 2 {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
return matches[1], true
|
||||||
|
}
|
||||||
|
|
||||||
|
// ClassifySource classifies a source id as built-in, configured, or previous-session configured.
|
||||||
|
func ClassifySource(source string) (Source, error) {
|
||||||
|
trimmed := strings.TrimSpace(source)
|
||||||
|
if trimmed == "" {
|
||||||
|
return Source{}, fmt.Errorf("artifact source is required")
|
||||||
|
}
|
||||||
|
if _, ok := artifactmodel.LookupRuntimeTranscriptArtifact(trimmed); ok {
|
||||||
|
return Source{ID: trimmed, Kind: SourceKindBuiltIn}, nil
|
||||||
|
}
|
||||||
|
if trimmed == SourceBoundsSession {
|
||||||
|
return Source{ID: trimmed, Kind: SourceKindBuiltIn}, nil
|
||||||
|
}
|
||||||
|
if key, ok := ParseConfiguredSource(trimmed); ok {
|
||||||
|
return Source{ID: trimmed, Kind: SourceKindConfiguredArtifact, ConfiguredKey: key}, nil
|
||||||
|
}
|
||||||
|
if key, ok := ParsePreviousSessionSource(trimmed); ok {
|
||||||
|
return Source{ID: trimmed, Kind: SourceKindPreviousArtifact, ConfiguredKey: key}, nil
|
||||||
|
}
|
||||||
|
return Source{}, fmt.Errorf("unsupported artifact source %q", source)
|
||||||
|
}
|
||||||
|
|
||||||
|
// DescribeScriptoriumInputSource classifies one input source and returns descriptor
|
||||||
|
// metadata used by config validation, analyze input resolution, and previous-cache planning.
|
||||||
|
func DescribeScriptoriumInputSource(source string) (ScriptoriumInputSourceDescriptor, error) {
|
||||||
|
trimmed := strings.TrimSpace(source)
|
||||||
|
if trimmed == "" {
|
||||||
|
return ScriptoriumInputSourceDescriptor{}, ErrUnsupportedScriptoriumInputSource
|
||||||
|
}
|
||||||
|
if strings.HasPrefix(trimmed, "narratio.previous_session.artifact") {
|
||||||
|
descriptor, err := DescribePreviousSessionSource(trimmed)
|
||||||
|
if err != nil {
|
||||||
|
return ScriptoriumInputSourceDescriptor{}, err
|
||||||
|
}
|
||||||
|
return ScriptoriumInputSourceDescriptor{
|
||||||
|
Source: Source{
|
||||||
|
ID: descriptor.SourceID,
|
||||||
|
Kind: SourceKindPreviousArtifact,
|
||||||
|
ConfiguredKey: descriptor.ConfiguredKey,
|
||||||
|
},
|
||||||
|
PreviousSession: &descriptor,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
classified, err := ClassifySource(trimmed)
|
||||||
|
if err != nil {
|
||||||
|
return ScriptoriumInputSourceDescriptor{}, ErrUnsupportedScriptoriumInputSource
|
||||||
|
}
|
||||||
|
return ScriptoriumInputSourceDescriptor{Source: classified}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// DescribePreviousSessionSource validates a canonical previous-session source id
|
||||||
|
// and returns both previous and configured-source vocabulary descriptors.
|
||||||
|
func DescribePreviousSessionSource(source string) (PreviousSessionSourceDescriptor, error) {
|
||||||
|
configuredKey, ok := ParsePreviousSessionSource(source)
|
||||||
|
if !ok {
|
||||||
|
return PreviousSessionSourceDescriptor{}, ErrInvalidPreviousSessionSource
|
||||||
|
}
|
||||||
|
return PreviousSessionSourceDescriptor{
|
||||||
|
SourceID: PreviousSessionSourceID(configuredKey),
|
||||||
|
ConfiguredKey: configuredKey,
|
||||||
|
ConfiguredSourceID: ConfiguredSourceID(configuredKey),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// PreviousSessionSourceDescriptorForConfiguredKey derives a previous-session source descriptor
|
||||||
|
// from a configured artifact key.
|
||||||
|
func PreviousSessionSourceDescriptorForConfiguredKey(configuredKey string) (PreviousSessionSourceDescriptor, error) {
|
||||||
|
return DescribePreviousSessionSource(PreviousSessionSourceID(configuredKey))
|
||||||
|
}
|
||||||
|
|
||||||
|
// ValidateInputConfiguredReference checks that configured/previous-session sources
|
||||||
|
// reference configured artifacts known to the current Scriptorium config.
|
||||||
|
func ValidateInputConfiguredReference(
|
||||||
|
descriptor ScriptoriumInputSourceDescriptor,
|
||||||
|
configured map[string]struct{},
|
||||||
|
) error {
|
||||||
|
switch descriptor.Source.Kind {
|
||||||
|
case SourceKindConfiguredArtifact, SourceKindPreviousArtifact:
|
||||||
|
if _, ok := configured[descriptor.Source.ConfiguredKey]; !ok {
|
||||||
|
return &UnknownConfiguredArtifactError{ConfiguredKey: descriptor.Source.ConfiguredKey}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ValidatePublishSource validates that a source is publish-compatible and references a known configured artifact.
|
||||||
|
func ValidatePublishSource(source string, configured map[string]string) (Source, error) {
|
||||||
|
classified, err := ClassifySource(source)
|
||||||
|
if err != nil {
|
||||||
|
return Source{}, fmt.Errorf("must be a built-in source id or narratio.artifact.<name>")
|
||||||
|
}
|
||||||
|
if classified.Kind == SourceKindPreviousArtifact {
|
||||||
|
return Source{}, fmt.Errorf("must be a built-in source id or narratio.artifact.<name>")
|
||||||
|
}
|
||||||
|
if classified.Kind == SourceKindConfiguredArtifact {
|
||||||
|
if configured == nil {
|
||||||
|
return Source{}, fmt.Errorf("configured artifact %q is not defined in pipeline.scriptorium.artifacts", classified.ConfiguredKey)
|
||||||
|
}
|
||||||
|
if _, ok := configured[classified.ConfiguredKey]; !ok {
|
||||||
|
return Source{}, fmt.Errorf("configured artifact %q is not defined in pipeline.scriptorium.artifacts", classified.ConfiguredKey)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return classified, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// DeriveDefaultPublishedDestination returns the default publish destination for one source.
|
||||||
|
func DeriveDefaultPublishedDestination(source Source, configured map[string]string) (string, error) {
|
||||||
|
switch source.Kind {
|
||||||
|
case SourceKindBuiltIn:
|
||||||
|
if spec, ok := artifactmodel.LookupRuntimeTranscriptArtifact(source.ID); ok {
|
||||||
|
return pathsafe.NormalizeRelativeDestination(spec.CanonicalRelPath)
|
||||||
|
}
|
||||||
|
if source.ID == SourceBoundsSession {
|
||||||
|
return pathsafe.NormalizeRelativeDestination("artifacts/session_bounds.json")
|
||||||
|
}
|
||||||
|
return "", fmt.Errorf("unsupported built-in source %q", source.ID)
|
||||||
|
case SourceKindConfiguredArtifact:
|
||||||
|
if configured == nil {
|
||||||
|
return "", fmt.Errorf("configured artifact %q is not defined in pipeline.scriptorium.artifacts", source.ConfiguredKey)
|
||||||
|
}
|
||||||
|
outputPath, ok := configured[source.ConfiguredKey]
|
||||||
|
if !ok {
|
||||||
|
return "", fmt.Errorf("configured artifact %q is not defined in pipeline.scriptorium.artifacts", source.ConfiguredKey)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(outputPath) == "" {
|
||||||
|
return "", fmt.Errorf("pipeline.scriptorium.artifacts.%s.output_path is empty", source.ConfiguredKey)
|
||||||
|
}
|
||||||
|
return pathsafe.NormalizeRelativeDestination(outputPath)
|
||||||
|
default:
|
||||||
|
return "", fmt.Errorf("publish destination cannot be derived from source %q", source.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResolvePublishedDestination validates and normalizes an explicit destination,
|
||||||
|
// or derives one when omitted.
|
||||||
|
func ResolvePublishedDestination(sourceID, explicitDest string, configured map[string]string) (string, error) {
|
||||||
|
source, err := ValidatePublishSource(sourceID, configured)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(explicitDest) != "" {
|
||||||
|
return pathsafe.NormalizeRelativeDestination(explicitDest)
|
||||||
|
}
|
||||||
|
return DeriveDefaultPublishedDestination(source, configured)
|
||||||
|
}
|
||||||
191
internal/artifactpolicy/policy_test.go
Normal file
191
internal/artifactpolicy/policy_test.go
Normal file
@@ -0,0 +1,191 @@
|
|||||||
|
package artifactpolicy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestClassifySource(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
source string
|
||||||
|
wantKind SourceKind
|
||||||
|
wantKey string
|
||||||
|
wantErrLike string
|
||||||
|
}{
|
||||||
|
{name: "built in transcript", source: "narratio.transcript.final_trimmed", wantKind: SourceKindBuiltIn},
|
||||||
|
{name: "built in bounds", source: "narratio.bounds.session", wantKind: SourceKindBuiltIn},
|
||||||
|
{name: "configured artifact", source: "narratio.artifact.session_recap", wantKind: SourceKindConfiguredArtifact, wantKey: "session_recap"},
|
||||||
|
{name: "previous session configured", source: "narratio.previous_session.artifact.session_recap", wantKind: SourceKindPreviousArtifact, wantKey: "session_recap"},
|
||||||
|
{name: "unsupported", source: "narratio.unknown", wantErrLike: "unsupported artifact source"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got, err := ClassifySource(tt.source)
|
||||||
|
if tt.wantErrLike != "" {
|
||||||
|
if err == nil || !strings.Contains(err.Error(), tt.wantErrLike) {
|
||||||
|
t.Fatalf("ClassifySource() error = %v, want like %q", err, tt.wantErrLike)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ClassifySource() error = %v", err)
|
||||||
|
}
|
||||||
|
if got.Kind != tt.wantKind {
|
||||||
|
t.Fatalf("ClassifySource().Kind = %q, want %q", got.Kind, tt.wantKind)
|
||||||
|
}
|
||||||
|
if got.ConfiguredKey != tt.wantKey {
|
||||||
|
t.Fatalf("ClassifySource().ConfiguredKey = %q, want %q", got.ConfiguredKey, tt.wantKey)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatePublishSource(t *testing.T) {
|
||||||
|
configured := map[string]string{"session_recap": "artifacts/session_recap.md"}
|
||||||
|
if _, err := ValidatePublishSource("narratio.artifact.session_recap", configured); err != nil {
|
||||||
|
t.Fatalf("ValidatePublishSource(configured) error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := ValidatePublishSource("narratio.previous_session.artifact.session_recap", configured); err == nil {
|
||||||
|
t.Fatal("ValidatePublishSource(previous) error = nil, want error")
|
||||||
|
}
|
||||||
|
if _, err := ValidatePublishSource("narratio.artifact.missing", configured); err == nil {
|
||||||
|
t.Fatal("ValidatePublishSource(missing configured) error = nil, want error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePublishedDestination(t *testing.T) {
|
||||||
|
configured := map[string]string{"session_recap": "artifacts/session_recap.md"}
|
||||||
|
|
||||||
|
got, err := ResolvePublishedDestination("narratio.transcript.final_trimmed", "", configured)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePublishedDestination(built-in) error = %v", err)
|
||||||
|
}
|
||||||
|
if got != "transcripts/final.trimmed.json" {
|
||||||
|
t.Fatalf("built-in destination = %q, want transcripts/final.trimmed.json", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
got, err = ResolvePublishedDestination("narratio.artifact.session_recap", "", configured)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePublishedDestination(configured) error = %v", err)
|
||||||
|
}
|
||||||
|
if got != "artifacts/session_recap.md" {
|
||||||
|
t.Fatalf("configured destination = %q, want artifacts/session_recap.md", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
got, err = ResolvePublishedDestination("narratio.transcript.final_trimmed", "published/../published/final.json", configured)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePublishedDestination(explicit) error = %v", err)
|
||||||
|
}
|
||||||
|
if got != "published/final.json" {
|
||||||
|
t.Fatalf("explicit destination = %q, want published/final.json", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePublishedDestinationRejectsTraversal(t *testing.T) {
|
||||||
|
configured := map[string]string{"session_recap": "artifacts/session_recap.md"}
|
||||||
|
_, err := ResolvePublishedDestination("narratio.transcript.final_trimmed", "../escape.txt", configured)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("ResolvePublishedDestination() error = nil, want traversal rejection")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDescribeScriptoriumInputSource(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
source string
|
||||||
|
wantKind SourceKind
|
||||||
|
wantKey string
|
||||||
|
wantPrev bool
|
||||||
|
wantErr error
|
||||||
|
wantErrLike string
|
||||||
|
}{
|
||||||
|
{name: "built in", source: "narratio.transcript.final_trimmed", wantKind: SourceKindBuiltIn},
|
||||||
|
{name: "configured", source: "narratio.artifact.session_recap", wantKind: SourceKindConfiguredArtifact, wantKey: "session_recap"},
|
||||||
|
{name: "previous", source: "narratio.previous_session.artifact.session_recap", wantKind: SourceKindPreviousArtifact, wantKey: "session_recap", wantPrev: true},
|
||||||
|
{name: "invalid previous", source: "narratio.previous_session.artifact.", wantErr: ErrInvalidPreviousSessionSource},
|
||||||
|
{name: "unsupported", source: "narratio.unknown", wantErr: ErrUnsupportedScriptoriumInputSource},
|
||||||
|
{name: "empty", source: " ", wantErr: ErrUnsupportedScriptoriumInputSource},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got, err := DescribeScriptoriumInputSource(tt.source)
|
||||||
|
if tt.wantErr != nil {
|
||||||
|
if !errors.Is(err, tt.wantErr) {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource() error = %v, want %v", err, tt.wantErr)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if tt.wantErrLike != "" {
|
||||||
|
if err == nil || !strings.Contains(err.Error(), tt.wantErrLike) {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource() error = %v, want like %q", err, tt.wantErrLike)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource() error = %v", err)
|
||||||
|
}
|
||||||
|
if got.Source.Kind != tt.wantKind {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource().Source.Kind = %q, want %q", got.Source.Kind, tt.wantKind)
|
||||||
|
}
|
||||||
|
if got.Source.ConfiguredKey != tt.wantKey {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource().Source.ConfiguredKey = %q, want %q", got.Source.ConfiguredKey, tt.wantKey)
|
||||||
|
}
|
||||||
|
if tt.wantPrev && got.PreviousSession == nil {
|
||||||
|
t.Fatal("DescribeScriptoriumInputSource().PreviousSession = nil, want descriptor")
|
||||||
|
}
|
||||||
|
if !tt.wantPrev && got.PreviousSession != nil {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource().PreviousSession = %#v, want nil", got.PreviousSession)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateInputConfiguredReference(t *testing.T) {
|
||||||
|
configured := map[string]struct{}{"session_recap": {}}
|
||||||
|
|
||||||
|
desc, err := DescribeScriptoriumInputSource("narratio.artifact.session_recap")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource(configured) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := ValidateInputConfiguredReference(desc, configured); err != nil {
|
||||||
|
t.Fatalf("ValidateInputConfiguredReference(configured) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
prevDesc, err := DescribeScriptoriumInputSource("narratio.previous_session.artifact.session_recap")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource(previous) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := ValidateInputConfiguredReference(prevDesc, configured); err != nil {
|
||||||
|
t.Fatalf("ValidateInputConfiguredReference(previous) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
missingDesc, err := DescribeScriptoriumInputSource("narratio.artifact.quest_log")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource(missing configured) error = %v", err)
|
||||||
|
}
|
||||||
|
err = ValidateInputConfiguredReference(missingDesc, configured)
|
||||||
|
var unknown *UnknownConfiguredArtifactError
|
||||||
|
if !errors.As(err, &unknown) || unknown.ConfiguredKey != "quest_log" {
|
||||||
|
t.Fatalf("ValidateInputConfiguredReference(missing configured) error = %v, want UnknownConfiguredArtifactError(quest_log)", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPreviousSessionSourceDescriptorForConfiguredKey(t *testing.T) {
|
||||||
|
got, err := PreviousSessionSourceDescriptorForConfiguredKey("session_recap")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("PreviousSessionSourceDescriptorForConfiguredKey() error = %v", err)
|
||||||
|
}
|
||||||
|
if got.SourceID != "narratio.previous_session.artifact.session_recap" {
|
||||||
|
t.Fatalf("SourceID = %q, want narratio.previous_session.artifact.session_recap", got.SourceID)
|
||||||
|
}
|
||||||
|
if got.ConfiguredSourceID != "narratio.artifact.session_recap" {
|
||||||
|
t.Fatalf("ConfiguredSourceID = %q, want narratio.artifact.session_recap", got.ConfiguredSourceID)
|
||||||
|
}
|
||||||
|
if got.ConfiguredKey != "session_recap" {
|
||||||
|
t.Fatalf("ConfiguredKey = %q, want session_recap", got.ConfiguredKey)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -6,10 +6,10 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"regexp"
|
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -40,8 +40,6 @@ const (
|
|||||||
|
|
||||||
// ErrSessionArtifactNotFound is returned when no readable artifact exists for a known ID.
|
// ErrSessionArtifactNotFound is returned when no readable artifact exists for a known ID.
|
||||||
var ErrSessionArtifactNotFound = errors.New("session artifact not found")
|
var ErrSessionArtifactNotFound = errors.New("session artifact not found")
|
||||||
var configuredArtifactSourceRE = regexp.MustCompile(`^narratio\.artifact\.([a-z][a-z0-9_]*)$`)
|
|
||||||
var previousSessionArtifactSourceRE = regexp.MustCompile(`^narratio\.previous_session\.artifact\.([a-z][a-z0-9_]*)$`)
|
|
||||||
|
|
||||||
type artifactContentKind string
|
type artifactContentKind string
|
||||||
|
|
||||||
@@ -107,43 +105,39 @@ func (e *SessionArtifactNotFoundError) Unwrap() error {
|
|||||||
|
|
||||||
// NormalizeSessionArtifactSource validates canonical artifact IDs.
|
// NormalizeSessionArtifactSource validates canonical artifact IDs.
|
||||||
func NormalizeSessionArtifactSource(source string) (string, error) {
|
func NormalizeSessionArtifactSource(source string) (string, error) {
|
||||||
normalized := strings.TrimSpace(source)
|
classified, err := artifactpolicy.ClassifySource(source)
|
||||||
if normalized == "" {
|
if err != nil {
|
||||||
return "", fmt.Errorf("artifact source is required")
|
|
||||||
}
|
|
||||||
if _, ok := artifactRegistry[normalized]; !ok {
|
|
||||||
return "", fmt.Errorf("unsupported artifact source %q", source)
|
return "", fmt.Errorf("unsupported artifact source %q", source)
|
||||||
}
|
}
|
||||||
return normalized, nil
|
if classified.Kind != artifactpolicy.SourceKindBuiltIn {
|
||||||
|
return "", fmt.Errorf("unsupported artifact source %q", source)
|
||||||
|
}
|
||||||
|
if _, ok := artifactRegistry[classified.ID]; !ok {
|
||||||
|
return "", fmt.Errorf("unsupported artifact source %q", source)
|
||||||
|
}
|
||||||
|
return classified.ID, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// IsConfiguredArtifactSource returns true when source is narratio.artifact.<name>.
|
// IsConfiguredArtifactSource returns true when source is narratio.artifact.<name>.
|
||||||
func IsConfiguredArtifactSource(source string) bool {
|
func IsConfiguredArtifactSource(source string) bool {
|
||||||
return configuredArtifactSourceRE.MatchString(strings.TrimSpace(source))
|
_, ok := artifactpolicy.ParseConfiguredSource(source)
|
||||||
|
return ok
|
||||||
}
|
}
|
||||||
|
|
||||||
// ConfiguredArtifactName extracts <name> from narratio.artifact.<name>.
|
// ConfiguredArtifactName extracts <name> from narratio.artifact.<name>.
|
||||||
func ConfiguredArtifactName(source string) (string, bool) {
|
func ConfiguredArtifactName(source string) (string, bool) {
|
||||||
matches := configuredArtifactSourceRE.FindStringSubmatch(strings.TrimSpace(source))
|
return artifactpolicy.ParseConfiguredSource(source)
|
||||||
if len(matches) != 2 {
|
|
||||||
return "", false
|
|
||||||
}
|
|
||||||
return matches[1], true
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// IsPreviousSessionArtifactSource returns true when source is narratio.previous_session.artifact.<name>.
|
// IsPreviousSessionArtifactSource returns true when source is narratio.previous_session.artifact.<name>.
|
||||||
func IsPreviousSessionArtifactSource(source string) bool {
|
func IsPreviousSessionArtifactSource(source string) bool {
|
||||||
_, ok := PreviousSessionArtifactName(source)
|
_, ok := artifactpolicy.ParsePreviousSessionSource(source)
|
||||||
return ok
|
return ok
|
||||||
}
|
}
|
||||||
|
|
||||||
// PreviousSessionArtifactName extracts <name> from narratio.previous_session.artifact.<name>.
|
// PreviousSessionArtifactName extracts <name> from narratio.previous_session.artifact.<name>.
|
||||||
func PreviousSessionArtifactName(source string) (string, bool) {
|
func PreviousSessionArtifactName(source string) (string, bool) {
|
||||||
matches := previousSessionArtifactSourceRE.FindStringSubmatch(strings.TrimSpace(source))
|
return artifactpolicy.ParsePreviousSessionSource(source)
|
||||||
if len(matches) != 2 {
|
|
||||||
return "", false
|
|
||||||
}
|
|
||||||
return matches[1], true
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ResolveSessionArtifact resolves a symbolic source to a readable local session artifact path.
|
// ResolveSessionArtifact resolves a symbolic source to a readable local session artifact path.
|
||||||
|
|||||||
@@ -4,6 +4,8 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -47,7 +49,7 @@ func NewArtifactCatalog() *ArtifactCatalog {
|
|||||||
|
|
||||||
// ConfiguredArtifactSourceID converts a configured artifact key into canonical source ID.
|
// ConfiguredArtifactSourceID converts a configured artifact key into canonical source ID.
|
||||||
func ConfiguredArtifactSourceID(key string) string {
|
func ConfiguredArtifactSourceID(key string) string {
|
||||||
return "narratio.artifact." + strings.TrimSpace(key)
|
return artifactpolicy.ConfiguredSourceID(key)
|
||||||
}
|
}
|
||||||
|
|
||||||
// RegisterBuiltIns registers built-in source definitions used by runtime artifact resolution.
|
// RegisterBuiltIns registers built-in source definitions used by runtime artifact resolution.
|
||||||
|
|||||||
203
internal/artifacts/current_state.go
Normal file
203
internal/artifacts/current_state.go
Normal file
@@ -0,0 +1,203 @@
|
|||||||
|
package artifacts
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
ErrCurrentRunPointerMissing = errors.New("current run pointer missing")
|
||||||
|
ErrCurrentManifestMissing = errors.New("current manifest missing")
|
||||||
|
)
|
||||||
|
|
||||||
|
type CurrentRunPointerMissingError struct {
|
||||||
|
Key string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *CurrentRunPointerMissingError) Error() string {
|
||||||
|
return fmt.Sprintf("%s: %q", ErrCurrentRunPointerMissing, e.Key)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *CurrentRunPointerMissingError) Unwrap() error {
|
||||||
|
return ErrCurrentRunPointerMissing
|
||||||
|
}
|
||||||
|
|
||||||
|
type CurrentManifestMissingError struct {
|
||||||
|
Key string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *CurrentManifestMissingError) Error() string {
|
||||||
|
return fmt.Sprintf("%s: %q", ErrCurrentManifestMissing, e.Key)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *CurrentManifestMissingError) Unwrap() error {
|
||||||
|
return ErrCurrentManifestMissing
|
||||||
|
}
|
||||||
|
|
||||||
|
type CurrentState struct {
|
||||||
|
SessionPrefix string
|
||||||
|
CurrentRunIDKey string
|
||||||
|
CurrentManifestKey string
|
||||||
|
RunID string
|
||||||
|
Manifest *manifest.Manifest
|
||||||
|
}
|
||||||
|
|
||||||
|
type CurrentStateValidation struct {
|
||||||
|
ExpectedCampaign string
|
||||||
|
ExpectedSessionID string
|
||||||
|
ExpectedRunID string
|
||||||
|
ValidateRunID bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func LoadCurrentRunPointer(ctx context.Context, store storage.ObjectStore, currentRunIDKey string) (string, error) {
|
||||||
|
if store == nil {
|
||||||
|
return "", fmt.Errorf("object store is required")
|
||||||
|
}
|
||||||
|
key := strings.TrimSpace(currentRunIDKey)
|
||||||
|
if key == "" {
|
||||||
|
return "", fmt.Errorf("current run pointer key is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
exists, err := store.Exists(ctx, key)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("check current run pointer %q: %w", key, err)
|
||||||
|
}
|
||||||
|
if !exists {
|
||||||
|
return "", &CurrentRunPointerMissingError{Key: key}
|
||||||
|
}
|
||||||
|
|
||||||
|
localPath, err := storage.DownloadObjectToTemp(ctx, store, key, "narratio-current-run-id-*.txt")
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("download current run pointer %q: %w", key, err)
|
||||||
|
}
|
||||||
|
defer func() { _ = os.Remove(localPath) }()
|
||||||
|
|
||||||
|
data, err := os.ReadFile(localPath)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("read downloaded current run pointer %q: %w", key, err)
|
||||||
|
}
|
||||||
|
runID := strings.TrimSpace(string(data))
|
||||||
|
if runID == "" {
|
||||||
|
return "", fmt.Errorf("current run pointer %q is empty", key)
|
||||||
|
}
|
||||||
|
return runID, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func LoadCurrentManifest(ctx context.Context, store storage.ObjectStore, currentManifestKey string) (*manifest.Manifest, error) {
|
||||||
|
if store == nil {
|
||||||
|
return nil, fmt.Errorf("object store is required")
|
||||||
|
}
|
||||||
|
key := strings.TrimSpace(currentManifestKey)
|
||||||
|
if key == "" {
|
||||||
|
return nil, fmt.Errorf("current manifest key is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
exists, err := store.Exists(ctx, key)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("check current manifest %q: %w", key, err)
|
||||||
|
}
|
||||||
|
if !exists {
|
||||||
|
return nil, &CurrentManifestMissingError{Key: key}
|
||||||
|
}
|
||||||
|
|
||||||
|
localPath, err := storage.DownloadObjectToTemp(ctx, store, key, "narratio-current-manifest-*.json")
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("download current manifest %q: %w", key, err)
|
||||||
|
}
|
||||||
|
defer func() { _ = os.Remove(localPath) }()
|
||||||
|
|
||||||
|
manifestStore := &manifest.LocalStore{}
|
||||||
|
m, err := manifestStore.Load(ctx, localPath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("current manifest decode failed: %w", err)
|
||||||
|
}
|
||||||
|
return m, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func LoadCurrentState(
|
||||||
|
ctx context.Context,
|
||||||
|
store storage.ObjectStore,
|
||||||
|
sessionPrefix string,
|
||||||
|
validation CurrentStateValidation,
|
||||||
|
) (*CurrentState, error) {
|
||||||
|
prefix := strings.TrimSpace(sessionPrefix)
|
||||||
|
if prefix == "" {
|
||||||
|
return nil, fmt.Errorf("session prefix is required")
|
||||||
|
}
|
||||||
|
currentManifestKey, currentRunIDKey := ResolveCurrentStateKeys(prefix)
|
||||||
|
runID, err := LoadCurrentRunPointer(ctx, store, currentRunIDKey)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
m, err := LoadCurrentManifest(ctx, store, currentManifestKey)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
state := &CurrentState{
|
||||||
|
SessionPrefix: prefix,
|
||||||
|
CurrentRunIDKey: currentRunIDKey,
|
||||||
|
CurrentManifestKey: currentManifestKey,
|
||||||
|
RunID: runID,
|
||||||
|
Manifest: m,
|
||||||
|
}
|
||||||
|
if err := ValidateCurrentStateIdentity(state, validation); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return state, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func ValidateCurrentStateIdentity(state *CurrentState, validation CurrentStateValidation) error {
|
||||||
|
if state == nil || state.Manifest == nil {
|
||||||
|
return fmt.Errorf("current state with manifest is required")
|
||||||
|
}
|
||||||
|
expectedSessionID := strings.TrimSpace(validation.ExpectedSessionID)
|
||||||
|
expectedCampaign := strings.TrimSpace(validation.ExpectedCampaign)
|
||||||
|
expectedRunID := strings.TrimSpace(validation.ExpectedRunID)
|
||||||
|
manifestSessionID := strings.TrimSpace(state.Manifest.SessionID)
|
||||||
|
manifestCampaign := strings.TrimSpace(state.Manifest.Campaign)
|
||||||
|
manifestRunID := strings.TrimSpace(state.Manifest.RunID)
|
||||||
|
|
||||||
|
if expectedSessionID != "" && manifestSessionID != expectedSessionID {
|
||||||
|
return fmt.Errorf(
|
||||||
|
"current manifest session_id %q does not match expected session_id %q",
|
||||||
|
manifestSessionID,
|
||||||
|
expectedSessionID,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
if expectedCampaign != "" {
|
||||||
|
if manifestCampaign == "" {
|
||||||
|
return fmt.Errorf("current manifest campaign is required")
|
||||||
|
}
|
||||||
|
if manifestCampaign != expectedCampaign {
|
||||||
|
return fmt.Errorf(
|
||||||
|
"current manifest campaign %q does not match expected campaign %q",
|
||||||
|
manifestCampaign,
|
||||||
|
expectedCampaign,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if expectedRunID == "" && validation.ValidateRunID {
|
||||||
|
expectedRunID = strings.TrimSpace(state.RunID)
|
||||||
|
}
|
||||||
|
if expectedRunID != "" {
|
||||||
|
if manifestRunID == "" {
|
||||||
|
return fmt.Errorf("current manifest run_id is required")
|
||||||
|
}
|
||||||
|
if manifestRunID != expectedRunID {
|
||||||
|
return fmt.Errorf(
|
||||||
|
"current run pointer %q references run %q but current manifest run_id is %q",
|
||||||
|
state.CurrentRunIDKey,
|
||||||
|
expectedRunID,
|
||||||
|
manifestRunID,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
140
internal/artifacts/current_state_test.go
Normal file
140
internal/artifacts/current_state_test.go
Normal file
@@ -0,0 +1,140 @@
|
|||||||
|
package artifacts
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestLoadCurrentStateMissingRunPointer(t *testing.T) {
|
||||||
|
store := &storage.FakeBackend{}
|
||||||
|
_, err := LoadCurrentState(context.Background(), store, testCurrentSessionPrefix(), CurrentStateValidation{})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("LoadCurrentState() error = nil, want missing run pointer error")
|
||||||
|
}
|
||||||
|
var missing *CurrentRunPointerMissingError
|
||||||
|
if !errors.As(err, &missing) {
|
||||||
|
t.Fatalf("errors.As(err, *CurrentRunPointerMissingError) = false; err=%v", err)
|
||||||
|
}
|
||||||
|
if !errors.Is(err, ErrCurrentRunPointerMissing) {
|
||||||
|
t.Fatalf("errors.Is(err, ErrCurrentRunPointerMissing) = false; err=%v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadCurrentStateMissingManifest(t *testing.T) {
|
||||||
|
store := &storage.FakeBackend{}
|
||||||
|
_, _, runIDKey := testCurrentStateKeys()
|
||||||
|
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||||
|
|
||||||
|
_, err := LoadCurrentState(context.Background(), store, testCurrentSessionPrefix(), CurrentStateValidation{})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("LoadCurrentState() error = nil, want missing manifest error")
|
||||||
|
}
|
||||||
|
var missing *CurrentManifestMissingError
|
||||||
|
if !errors.As(err, &missing) {
|
||||||
|
t.Fatalf("errors.As(err, *CurrentManifestMissingError) = false; err=%v", err)
|
||||||
|
}
|
||||||
|
if !errors.Is(err, ErrCurrentManifestMissing) {
|
||||||
|
t.Fatalf("errors.Is(err, ErrCurrentManifestMissing) = false; err=%v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadCurrentStateEmptyRunPointerFails(t *testing.T) {
|
||||||
|
store := &storage.FakeBackend{}
|
||||||
|
_, manifestKey, runIDKey := testCurrentStateKeys()
|
||||||
|
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte(" \n\t")})
|
||||||
|
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: testManifestJSON(t, "2026-05-03", "sample-campaign", "20260519T010203Z-a1b2c3d4")})
|
||||||
|
|
||||||
|
_, err := LoadCurrentState(context.Background(), store, testCurrentSessionPrefix(), CurrentStateValidation{})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "is empty") {
|
||||||
|
t.Fatalf("error = %v, want empty run pointer failure", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadCurrentStateMalformedManifestFails(t *testing.T) {
|
||||||
|
store := &storage.FakeBackend{}
|
||||||
|
_, manifestKey, runIDKey := testCurrentStateKeys()
|
||||||
|
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||||
|
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: []byte("{invalid json")})
|
||||||
|
|
||||||
|
_, err := LoadCurrentState(context.Background(), store, testCurrentSessionPrefix(), CurrentStateValidation{})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "current manifest decode failed") {
|
||||||
|
t.Fatalf("error = %v, want manifest decode failure", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadCurrentStateCampaignMismatchFails(t *testing.T) {
|
||||||
|
store := &storage.FakeBackend{}
|
||||||
|
seedCurrentState(t, store, "2026-05-03", "wrong-campaign", "20260519T010203Z-a1b2c3d4")
|
||||||
|
|
||||||
|
_, err := LoadCurrentState(context.Background(), store, testCurrentSessionPrefix(), CurrentStateValidation{
|
||||||
|
ExpectedCampaign: "sample-campaign",
|
||||||
|
})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "does not match expected campaign") {
|
||||||
|
t.Fatalf("error = %v, want campaign mismatch failure", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadCurrentStateSessionMismatchFails(t *testing.T) {
|
||||||
|
store := &storage.FakeBackend{}
|
||||||
|
seedCurrentState(t, store, "wrong-session", "sample-campaign", "20260519T010203Z-a1b2c3d4")
|
||||||
|
|
||||||
|
_, err := LoadCurrentState(context.Background(), store, testCurrentSessionPrefix(), CurrentStateValidation{
|
||||||
|
ExpectedSessionID: "2026-05-03",
|
||||||
|
})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "does not match expected session_id") {
|
||||||
|
t.Fatalf("error = %v, want session mismatch failure", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadCurrentStateRunIDMismatchFails(t *testing.T) {
|
||||||
|
store := &storage.FakeBackend{}
|
||||||
|
seedCurrentState(t, store, "2026-05-03", "sample-campaign", "different-run-id")
|
||||||
|
|
||||||
|
_, err := LoadCurrentState(context.Background(), store, testCurrentSessionPrefix(), CurrentStateValidation{
|
||||||
|
ValidateRunID: true,
|
||||||
|
})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "current manifest run_id") {
|
||||||
|
t.Fatalf("error = %v, want run mismatch failure", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func testCurrentSessionPrefix() string {
|
||||||
|
return S3SessionPrefix("dnd", "sample-campaign", "2026-05-03")
|
||||||
|
}
|
||||||
|
|
||||||
|
func testCurrentStateKeys() (sessionPrefix, manifestKey, runIDKey string) {
|
||||||
|
sessionPrefix = testCurrentSessionPrefix()
|
||||||
|
manifestKey, runIDKey = ResolveCurrentStateKeys(sessionPrefix)
|
||||||
|
return sessionPrefix, manifestKey, runIDKey
|
||||||
|
}
|
||||||
|
|
||||||
|
func seedCurrentState(t *testing.T, store *storage.FakeBackend, sessionID, campaign, manifestRunID string) {
|
||||||
|
t.Helper()
|
||||||
|
_, manifestKey, runIDKey := testCurrentStateKeys()
|
||||||
|
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||||
|
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: testManifestJSON(t, sessionID, campaign, manifestRunID)})
|
||||||
|
}
|
||||||
|
|
||||||
|
func testManifestJSON(t *testing.T, sessionID, campaign, runID string) []byte {
|
||||||
|
t.Helper()
|
||||||
|
now := time.Date(2026, 5, 19, 23, 0, 0, 0, time.UTC).Format(time.RFC3339Nano)
|
||||||
|
payload := map[string]any{
|
||||||
|
"session_id": sessionID,
|
||||||
|
"campaign": campaign,
|
||||||
|
"run_id": runID,
|
||||||
|
"created_at": now,
|
||||||
|
"updated_at": now,
|
||||||
|
"stages": map[string]any{},
|
||||||
|
}
|
||||||
|
data, err := json.Marshal(payload)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("marshal manifest payload: %v", err)
|
||||||
|
}
|
||||||
|
return append(data, '\n')
|
||||||
|
}
|
||||||
@@ -9,6 +9,9 @@ import (
|
|||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/fileops"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/pathsafe"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ErrLockConflict is returned when a session lock already exists.
|
// ErrLockConflict is returned when a session lock already exists.
|
||||||
@@ -101,7 +104,7 @@ func (s *LocalStore) copyInputWithPaths(paths SessionPaths, sessionID, srcPath,
|
|||||||
return Ref{}, fmt.Errorf("copy input: %w", err)
|
return Ref{}, fmt.Errorf("copy input: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := copyFileAtomic(srcPath, destAbs, 0o644); err != nil {
|
if err := fileops.CopyFileAtomic(srcPath, destAbs, 0o644); err != nil {
|
||||||
return Ref{}, fmt.Errorf("copy input %q -> %q: %w", srcPath, destAbs, err)
|
return Ref{}, fmt.Errorf("copy input %q -> %q: %w", srcPath, destAbs, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -145,45 +148,9 @@ func (s *LocalStore) WriteFileAtomic(path string, data []byte, perm os.FileMode)
|
|||||||
if strings.TrimSpace(path) == "" {
|
if strings.TrimSpace(path) == "" {
|
||||||
return fmt.Errorf("write file atomic: path is required")
|
return fmt.Errorf("write file atomic: path is required")
|
||||||
}
|
}
|
||||||
|
if err := fileops.WriteFileAtomic(path, data, perm); err != nil {
|
||||||
dir := filepath.Dir(path)
|
return fmt.Errorf("write file atomic: %w", err)
|
||||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
|
||||||
return fmt.Errorf("write file atomic: create parent dir %q: %w", dir, err)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
base := filepath.Base(path)
|
|
||||||
tmp, err := os.CreateTemp(dir, "."+base+".tmp-*")
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("write file atomic: create temp file: %w", err)
|
|
||||||
}
|
|
||||||
tmpName := tmp.Name()
|
|
||||||
removeTmp := true
|
|
||||||
defer func() {
|
|
||||||
if removeTmp {
|
|
||||||
_ = os.Remove(tmpName)
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
|
|
||||||
if _, err := tmp.Write(data); err != nil {
|
|
||||||
_ = tmp.Close()
|
|
||||||
return fmt.Errorf("write file atomic: write temp file: %w", err)
|
|
||||||
}
|
|
||||||
if err := tmp.Sync(); err != nil {
|
|
||||||
_ = tmp.Close()
|
|
||||||
return fmt.Errorf("write file atomic: sync temp file: %w", err)
|
|
||||||
}
|
|
||||||
if err := tmp.Close(); err != nil {
|
|
||||||
return fmt.Errorf("write file atomic: close temp file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := os.Chmod(tmpName, perm); err != nil {
|
|
||||||
return fmt.Errorf("write file atomic: chmod temp file: %w", err)
|
|
||||||
}
|
|
||||||
if err := os.Rename(tmpName, path); err != nil {
|
|
||||||
return fmt.Errorf("write file atomic: rename temp file: %w", err)
|
|
||||||
}
|
|
||||||
removeTmp = false
|
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -256,62 +223,21 @@ func (s *LocalStore) ReleaseSessionLock(lock *LockHandle) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func resolveInRoot(root, relative string) (string, error) {
|
func resolveInRoot(root, relative string) (string, error) {
|
||||||
rel := filepath.Clean(relative)
|
joined, err := pathsafe.JoinSlashRelativeUnderRoot(root, filepath.ToSlash(relative))
|
||||||
if rel == "." || rel == "" {
|
if err != nil {
|
||||||
|
switch {
|
||||||
|
case errors.Is(err, pathsafe.ErrRelativePathRequired):
|
||||||
|
return "", fmt.Errorf("relative destination path is required")
|
||||||
|
case errors.Is(err, pathsafe.ErrRelativePathAbsolute):
|
||||||
|
return "", fmt.Errorf("relative destination must not be absolute: %q", relative)
|
||||||
|
case errors.Is(err, pathsafe.ErrRelativePathEscape):
|
||||||
|
return "", fmt.Errorf("relative destination escapes root: %q", relative)
|
||||||
|
default:
|
||||||
|
return "", fmt.Errorf("resolve destination in root: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(joined) == "" {
|
||||||
return "", fmt.Errorf("relative destination path is required")
|
return "", fmt.Errorf("relative destination path is required")
|
||||||
}
|
}
|
||||||
if filepath.IsAbs(rel) {
|
return joined, nil
|
||||||
return "", fmt.Errorf("relative destination must not be absolute: %q", relative)
|
|
||||||
}
|
|
||||||
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
|
||||||
return "", fmt.Errorf("relative destination escapes root: %q", relative)
|
|
||||||
}
|
|
||||||
return filepath.Join(root, rel), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func copyFileAtomic(srcPath, dstPath string, perm os.FileMode) error {
|
|
||||||
src, err := os.Open(srcPath)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
defer src.Close()
|
|
||||||
|
|
||||||
if err := os.MkdirAll(filepath.Dir(dstPath), 0o755); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
dir := filepath.Dir(dstPath)
|
|
||||||
base := filepath.Base(dstPath)
|
|
||||||
tmp, err := os.CreateTemp(dir, "."+base+".tmp-*")
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
tmpName := tmp.Name()
|
|
||||||
removeTmp := true
|
|
||||||
defer func() {
|
|
||||||
if removeTmp {
|
|
||||||
_ = os.Remove(tmpName)
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
|
|
||||||
if _, err := io.Copy(tmp, src); err != nil {
|
|
||||||
_ = tmp.Close()
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if err := tmp.Sync(); err != nil {
|
|
||||||
_ = tmp.Close()
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if err := tmp.Close(); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if err := os.Chmod(tmpName, perm); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if err := os.Rename(tmpName, dstPath); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
removeTmp = false
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -75,9 +75,9 @@ func TestSessionPreviousPathsForCampaign(t *testing.T) {
|
|||||||
t.Fatalf("SessionPreviousArtifactPathForCampaign() = %q, want %q", artifactPath, wantArtifactPath)
|
t.Fatalf("SessionPreviousArtifactPathForCampaign() = %q, want %q", artifactPath, wantArtifactPath)
|
||||||
}
|
}
|
||||||
|
|
||||||
archiveRelativeArtifactPath := SessionPreviousArtifactPathForCampaign(root, "forsaken", "2026-04-19", "artifacts/session_recap.md")
|
previousRelativeArtifactPath := SessionPreviousArtifactPathForCampaign(root, "forsaken", "2026-04-19", "artifacts/session_recap.md")
|
||||||
if archiveRelativeArtifactPath != wantArtifactPath {
|
if previousRelativeArtifactPath != wantArtifactPath {
|
||||||
t.Fatalf("SessionPreviousArtifactPathForCampaign(archive-relative) = %q, want %q", archiveRelativeArtifactPath, wantArtifactPath)
|
t.Fatalf("SessionPreviousArtifactPathForCampaign(previous-relative) = %q, want %q", previousRelativeArtifactPath, wantArtifactPath)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -102,7 +102,7 @@ func TestSessionPreviousPathsFromSessionPaths(t *testing.T) {
|
|||||||
got = SessionPreviousArtifactPath(paths, "artifacts/session_recap.md")
|
got = SessionPreviousArtifactPath(paths, "artifacts/session_recap.md")
|
||||||
want = filepath.Join(paths.PreviousArtifactsDir, "session_recap.md")
|
want = filepath.Join(paths.PreviousArtifactsDir, "session_recap.md")
|
||||||
if got != want {
|
if got != want {
|
||||||
t.Fatalf("SessionPreviousArtifactPath(archive-relative) = %q, want %q", got, want)
|
t.Fatalf("SessionPreviousArtifactPath(previous-relative) = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -36,10 +37,11 @@ func CollectPreviousArtifactRequirements(
|
|||||||
inputNames := sortedScriptoriumInputKeys(artifactCfg.Inputs)
|
inputNames := sortedScriptoriumInputKeys(artifactCfg.Inputs)
|
||||||
for _, inputName := range inputNames {
|
for _, inputName := range inputNames {
|
||||||
inputCfg := artifactCfg.Inputs[inputName]
|
inputCfg := artifactCfg.Inputs[inputName]
|
||||||
previousName, ok := PreviousSessionArtifactName(inputCfg.Source)
|
descriptor, err := artifactpolicy.DescribeScriptoriumInputSource(inputCfg.Source)
|
||||||
if !ok {
|
if err != nil || descriptor.PreviousSession == nil {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
previousName := descriptor.PreviousSession.ConfiguredKey
|
||||||
|
|
||||||
location := fmt.Sprintf(
|
location := fmt.Sprintf(
|
||||||
"pipeline.scriptorium.artifacts.%s.inputs.%s.source",
|
"pipeline.scriptorium.artifacts.%s.inputs.%s.source",
|
||||||
|
|||||||
@@ -8,8 +8,8 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ResolveArchiveBucket resolves archive bucket identity with manifest-first precedence.
|
// ResolvePublishBucket resolves publish bucket identity with manifest-first precedence.
|
||||||
func ResolveArchiveBucket(cfg *config.Config, m *manifest.Manifest) string {
|
func ResolvePublishBucket(cfg *config.Config, m *manifest.Manifest) string {
|
||||||
if m != nil && strings.TrimSpace(m.S3Bucket) != "" {
|
if m != nil && strings.TrimSpace(m.S3Bucket) != "" {
|
||||||
return strings.TrimSpace(m.S3Bucket)
|
return strings.TrimSpace(m.S3Bucket)
|
||||||
}
|
}
|
||||||
@@ -19,8 +19,8 @@ func ResolveArchiveBucket(cfg *config.Config, m *manifest.Manifest) string {
|
|||||||
return strings.TrimSpace(cfg.Pipeline.Storage.S3.Bucket)
|
return strings.TrimSpace(cfg.Pipeline.Storage.S3.Bucket)
|
||||||
}
|
}
|
||||||
|
|
||||||
// ResolveArchiveSessionPrefix resolves archive session prefix with manifest-first precedence.
|
// ResolvePublishSessionPrefix resolves publish session prefix with manifest-first precedence.
|
||||||
func ResolveArchiveSessionPrefix(cfg *config.Config, m *manifest.Manifest) (string, error) {
|
func ResolvePublishSessionPrefix(cfg *config.Config, m *manifest.Manifest) (string, error) {
|
||||||
if m != nil && strings.TrimSpace(m.S3SessionPrefix) != "" {
|
if m != nil && strings.TrimSpace(m.S3SessionPrefix) != "" {
|
||||||
return strings.TrimSpace(m.S3SessionPrefix), nil
|
return strings.TrimSpace(m.S3SessionPrefix), nil
|
||||||
}
|
}
|
||||||
@@ -47,8 +47,8 @@ func ResolveArchiveSessionPrefix(cfg *config.Config, m *manifest.Manifest) (stri
|
|||||||
return sessionPrefix, nil
|
return sessionPrefix, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// ResolveArchiveRunPrefix resolves archive run prefix with manifest-first precedence.
|
// ResolvePublishRunPrefix resolves publish run prefix with manifest-first precedence.
|
||||||
func ResolveArchiveRunPrefix(cfg *config.Config, m *manifest.Manifest) (string, error) {
|
func ResolvePublishRunPrefix(cfg *config.Config, m *manifest.Manifest) (string, error) {
|
||||||
if m != nil {
|
if m != nil {
|
||||||
runPrefix := strings.TrimSpace(m.S3RunPrefix)
|
runPrefix := strings.TrimSpace(m.S3RunPrefix)
|
||||||
if runPrefix != "" {
|
if runPrefix != "" {
|
||||||
@@ -56,7 +56,7 @@ func ResolveArchiveRunPrefix(cfg *config.Config, m *manifest.Manifest) (string,
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
sessionPrefix, err := ResolveArchiveSessionPrefix(cfg, m)
|
sessionPrefix, err := ResolvePublishSessionPrefix(cfg, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", err
|
return "", err
|
||||||
}
|
}
|
||||||
@@ -71,7 +71,7 @@ func ResolveArchiveRunPrefix(cfg *config.Config, m *manifest.Manifest) (string,
|
|||||||
return S3RunPrefix(sessionPrefix, runID), nil
|
return S3RunPrefix(sessionPrefix, runID), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// ResolveArchiveCurrentStateKeys returns current pointer keys for a session prefix.
|
// ResolveCurrentStateKeys returns current pointer keys for a session prefix.
|
||||||
func ResolveArchiveCurrentStateKeys(sessionPrefix string) (manifestKey, runIDKey string) {
|
func ResolveCurrentStateKeys(sessionPrefix string) (manifestKey, runIDKey string) {
|
||||||
return S3CurrentManifestKey(sessionPrefix), S3CurrentRunPointerKey(sessionPrefix)
|
return S3CurrentManifestKey(sessionPrefix), S3CurrentRunPointerKey(sessionPrefix)
|
||||||
}
|
}
|
||||||
@@ -8,7 +8,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestResolveArchiveBucketPrefersManifestThenConfig(t *testing.T) {
|
func TestResolvePublishBucketPrefersManifestThenConfig(t *testing.T) {
|
||||||
cfg := &config.Config{
|
cfg := &config.Config{
|
||||||
Pipeline: &config.PipelineConfig{
|
Pipeline: &config.PipelineConfig{
|
||||||
Storage: config.StorageConfig{
|
Storage: config.StorageConfig{
|
||||||
@@ -17,15 +17,15 @@ func TestResolveArchiveBucketPrefersManifestThenConfig(t *testing.T) {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
if got := ResolveArchiveBucket(cfg, &manifest.Manifest{S3Bucket: "manifest-bucket"}); got != "manifest-bucket" {
|
if got := ResolvePublishBucket(cfg, &manifest.Manifest{S3Bucket: "manifest-bucket"}); got != "manifest-bucket" {
|
||||||
t.Fatalf("bucket = %q, want manifest-bucket", got)
|
t.Fatalf("bucket = %q, want manifest-bucket", got)
|
||||||
}
|
}
|
||||||
if got := ResolveArchiveBucket(cfg, &manifest.Manifest{}); got != "cfg-bucket" {
|
if got := ResolvePublishBucket(cfg, &manifest.Manifest{}); got != "cfg-bucket" {
|
||||||
t.Fatalf("bucket = %q, want cfg-bucket", got)
|
t.Fatalf("bucket = %q, want cfg-bucket", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResolveArchiveSessionPrefixPrefersManifestThenConfig(t *testing.T) {
|
func TestResolvePublishSessionPrefixPrefersManifestThenConfig(t *testing.T) {
|
||||||
cfg := &config.Config{
|
cfg := &config.Config{
|
||||||
Pipeline: &config.PipelineConfig{
|
Pipeline: &config.PipelineConfig{
|
||||||
Storage: config.StorageConfig{
|
Storage: config.StorageConfig{
|
||||||
@@ -39,17 +39,17 @@ func TestResolveArchiveSessionPrefixPrefersManifestThenConfig(t *testing.T) {
|
|||||||
}
|
}
|
||||||
m := &manifest.Manifest{S3SessionPrefix: "manifest/session/prefix/"}
|
m := &manifest.Manifest{S3SessionPrefix: "manifest/session/prefix/"}
|
||||||
|
|
||||||
got, err := ResolveArchiveSessionPrefix(cfg, m)
|
got, err := ResolvePublishSessionPrefix(cfg, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ResolveArchiveSessionPrefix() error = %v", err)
|
t.Fatalf("ResolvePublishSessionPrefix() error = %v", err)
|
||||||
}
|
}
|
||||||
if got != "manifest/session/prefix/" {
|
if got != "manifest/session/prefix/" {
|
||||||
t.Fatalf("session prefix = %q, want manifest/session/prefix/", got)
|
t.Fatalf("session prefix = %q, want manifest/session/prefix/", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
got, err = ResolveArchiveSessionPrefix(cfg, &manifest.Manifest{})
|
got, err = ResolvePublishSessionPrefix(cfg, &manifest.Manifest{})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ResolveArchiveSessionPrefix() error = %v", err)
|
t.Fatalf("ResolvePublishSessionPrefix() error = %v", err)
|
||||||
}
|
}
|
||||||
want := "dnd/campaigns/forsaken/sessions/2026-04-19/"
|
want := "dnd/campaigns/forsaken/sessions/2026-04-19/"
|
||||||
if got != want {
|
if got != want {
|
||||||
@@ -57,7 +57,7 @@ func TestResolveArchiveSessionPrefixPrefersManifestThenConfig(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResolveArchiveRunPrefixPrefersManifestThenDerived(t *testing.T) {
|
func TestResolvePublishRunPrefixPrefersManifestThenDerived(t *testing.T) {
|
||||||
cfg := &config.Config{
|
cfg := &config.Config{
|
||||||
Pipeline: &config.PipelineConfig{
|
Pipeline: &config.PipelineConfig{
|
||||||
Storage: config.StorageConfig{
|
Storage: config.StorageConfig{
|
||||||
@@ -74,9 +74,9 @@ func TestResolveArchiveRunPrefixPrefersManifestThenDerived(t *testing.T) {
|
|||||||
RunID: "20260516T010203Z-1a2b3c4d",
|
RunID: "20260516T010203Z-1a2b3c4d",
|
||||||
S3RunPrefix: "manifest/run/prefix/",
|
S3RunPrefix: "manifest/run/prefix/",
|
||||||
}
|
}
|
||||||
got, err := ResolveArchiveRunPrefix(cfg, m)
|
got, err := ResolvePublishRunPrefix(cfg, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ResolveArchiveRunPrefix() error = %v", err)
|
t.Fatalf("ResolvePublishRunPrefix() error = %v", err)
|
||||||
}
|
}
|
||||||
if got != "manifest/run/prefix/" {
|
if got != "manifest/run/prefix/" {
|
||||||
t.Fatalf("run prefix = %q, want manifest/run/prefix/", got)
|
t.Fatalf("run prefix = %q, want manifest/run/prefix/", got)
|
||||||
@@ -85,9 +85,9 @@ func TestResolveArchiveRunPrefixPrefersManifestThenDerived(t *testing.T) {
|
|||||||
m = &manifest.Manifest{
|
m = &manifest.Manifest{
|
||||||
RunID: "20260516T010203Z-1a2b3c4d",
|
RunID: "20260516T010203Z-1a2b3c4d",
|
||||||
}
|
}
|
||||||
got, err = ResolveArchiveRunPrefix(cfg, m)
|
got, err = ResolvePublishRunPrefix(cfg, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ResolveArchiveRunPrefix() error = %v", err)
|
t.Fatalf("ResolvePublishRunPrefix() error = %v", err)
|
||||||
}
|
}
|
||||||
want := "dnd/campaigns/forsaken/sessions/2026-04-19/runs/20260516T010203Z-1a2b3c4d/"
|
want := "dnd/campaigns/forsaken/sessions/2026-04-19/runs/20260516T010203Z-1a2b3c4d/"
|
||||||
if got != want {
|
if got != want {
|
||||||
@@ -95,12 +95,12 @@ func TestResolveArchiveRunPrefixPrefersManifestThenDerived(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResolveArchiveIdentityErrorsAreDeterministic(t *testing.T) {
|
func TestResolvePublishIdentityErrorsAreDeterministic(t *testing.T) {
|
||||||
cfgNoS3 := &config.Config{
|
cfgNoS3 := &config.Config{
|
||||||
Pipeline: &config.PipelineConfig{},
|
Pipeline: &config.PipelineConfig{},
|
||||||
Session: &config.SessionConfig{SessionID: "2026-04-19", Campaign: "forsaken"},
|
Session: &config.SessionConfig{SessionID: "2026-04-19", Campaign: "forsaken"},
|
||||||
}
|
}
|
||||||
_, err := ResolveArchiveSessionPrefix(cfgNoS3, &manifest.Manifest{})
|
_, err := ResolvePublishSessionPrefix(cfgNoS3, &manifest.Manifest{})
|
||||||
if err == nil || !strings.Contains(err.Error(), "pipeline.storage.s3 configuration is required") {
|
if err == nil || !strings.Contains(err.Error(), "pipeline.storage.s3 configuration is required") {
|
||||||
t.Fatalf("error = %v, want missing storage.s3", err)
|
t.Fatalf("error = %v, want missing storage.s3", err)
|
||||||
}
|
}
|
||||||
@@ -113,14 +113,14 @@ func TestResolveArchiveIdentityErrorsAreDeterministic(t *testing.T) {
|
|||||||
},
|
},
|
||||||
Session: &config.SessionConfig{SessionID: "2026-04-19", Campaign: "forsaken"},
|
Session: &config.SessionConfig{SessionID: "2026-04-19", Campaign: "forsaken"},
|
||||||
}
|
}
|
||||||
_, err = ResolveArchiveRunPrefix(cfg, &manifest.Manifest{})
|
_, err = ResolvePublishRunPrefix(cfg, &manifest.Manifest{})
|
||||||
if err == nil || !strings.Contains(err.Error(), "run id is required") {
|
if err == nil || !strings.Contains(err.Error(), "run id is required") {
|
||||||
t.Fatalf("error = %v, want missing run id", err)
|
t.Fatalf("error = %v, want missing run id", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResolveArchiveCurrentStateKeys(t *testing.T) {
|
func TestResolveCurrentStateKeys(t *testing.T) {
|
||||||
manifestKey, runIDKey := ResolveArchiveCurrentStateKeys("dnd/campaigns/forsaken/sessions/2026-04-19/")
|
manifestKey, runIDKey := ResolveCurrentStateKeys("dnd/campaigns/forsaken/sessions/2026-04-19/")
|
||||||
if manifestKey != "dnd/campaigns/forsaken/sessions/2026-04-19/current/manifest.json" {
|
if manifestKey != "dnd/campaigns/forsaken/sessions/2026-04-19/current/manifest.json" {
|
||||||
t.Fatalf("manifest key = %q", manifestKey)
|
t.Fatalf("manifest key = %q", manifestKey)
|
||||||
}
|
}
|
||||||
@@ -43,9 +43,9 @@ func TestS3KeyConstruction(t *testing.T) {
|
|||||||
t.Fatalf("manifest key = %q", manifestKey)
|
t.Fatalf("manifest key = %q", manifestKey)
|
||||||
}
|
}
|
||||||
|
|
||||||
promoted := S3PublishedOutputKey(sessionPrefix, "transcripts/final.trimmed.json")
|
publishedKey := S3PublishedOutputKey(sessionPrefix, "transcripts/final.trimmed.json")
|
||||||
if promoted != "dnd/campaigns/forsaken/sessions/2026-04-19/transcripts/final.trimmed.json" {
|
if publishedKey != "dnd/campaigns/forsaken/sessions/2026-04-19/transcripts/final.trimmed.json" {
|
||||||
t.Fatalf("promoted key = %q", promoted)
|
t.Fatalf("published key = %q", publishedKey)
|
||||||
}
|
}
|
||||||
|
|
||||||
runRelative := S3RunRelativeDestinationKey(runPrefix, `logs\whisperx.stdout.log`)
|
runRelative := S3RunRelativeDestinationKey(runPrefix, `logs\whisperx.stdout.log`)
|
||||||
|
|||||||
@@ -2,16 +2,14 @@ package audio
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"crypto/sha256"
|
|
||||||
"encoding/hex"
|
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/fileops"
|
||||||
)
|
)
|
||||||
|
|
||||||
// S3MaterializeRequest describes one S3-backed audio materialization.
|
// S3MaterializeRequest describes one S3-backed audio materialization.
|
||||||
@@ -61,7 +59,7 @@ func MaterializeS3Audio(ctx context.Context, req S3MaterializeRequest) (S3Materi
|
|||||||
if ok, err := validCachedAudio(cachePath, req.Object.Size); err != nil {
|
if ok, err := validCachedAudio(cachePath, req.Object.Size); err != nil {
|
||||||
return S3MaterializeResult{}, err
|
return S3MaterializeResult{}, err
|
||||||
} else if ok {
|
} else if ok {
|
||||||
checksum, err := copyFileAtomicWithChecksum(cachePath, req.DestPath, 0o644)
|
checksum, err := fileops.CopyFileAtomicWithChecksum(cachePath, req.DestPath, 0o644)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return S3MaterializeResult{}, fmt.Errorf("materialize cached audio %q: %w", cachePath, err)
|
return S3MaterializeResult{}, fmt.Errorf("materialize cached audio %q: %w", cachePath, err)
|
||||||
}
|
}
|
||||||
@@ -82,7 +80,7 @@ func MaterializeS3Audio(ctx context.Context, req S3MaterializeRequest) (S3Materi
|
|||||||
return S3MaterializeResult{}, fmt.Errorf("validate downloaded audio %q: %w", spoolPath, err)
|
return S3MaterializeResult{}, fmt.Errorf("validate downloaded audio %q: %w", spoolPath, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
checksum, err := copyFileAtomicWithChecksum(spoolPath, req.DestPath, 0o644)
|
checksum, err := fileops.CopyFileAtomicWithChecksum(spoolPath, req.DestPath, 0o644)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return S3MaterializeResult{}, fmt.Errorf("materialize downloaded audio %q: %w", filepath.Base(req.DestPath), err)
|
return S3MaterializeResult{}, fmt.Errorf("materialize downloaded audio %q: %w", filepath.Base(req.DestPath), err)
|
||||||
}
|
}
|
||||||
@@ -91,7 +89,7 @@ func MaterializeS3Audio(ctx context.Context, req S3MaterializeRequest) (S3Materi
|
|||||||
result.Downloaded = true
|
result.Downloaded = true
|
||||||
|
|
||||||
if result.CachePath != "" {
|
if result.CachePath != "" {
|
||||||
if _, err := copyFileAtomicWithChecksum(spoolPath, result.CachePath, 0o644); err != nil {
|
if _, err := fileops.CopyFileAtomicWithChecksum(spoolPath, result.CachePath, 0o644); err != nil {
|
||||||
return S3MaterializeResult{}, fmt.Errorf("populate audio cache %q: %w", result.CachePath, err)
|
return S3MaterializeResult{}, fmt.Errorf("populate audio cache %q: %w", result.CachePath, err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -164,60 +162,9 @@ func downloadObjectAtomic(ctx context.Context, store storage.ObjectStore, key, d
|
|||||||
if err := store.Download(ctx, key, tmpPath); err != nil {
|
if err := store.Download(ctx, key, tmpPath); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if err := os.Chmod(tmpPath, 0o644); err != nil {
|
if err := fileops.InstallDownloadedTempFile(tmpPath, destPath, 0o644); err != nil {
|
||||||
return fmt.Errorf("set temp file permissions: %w", err)
|
return err
|
||||||
}
|
|
||||||
if err := os.Rename(tmpPath, destPath); err != nil {
|
|
||||||
return fmt.Errorf("install downloaded file: %w", err)
|
|
||||||
}
|
}
|
||||||
removeTmp = false
|
removeTmp = false
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func copyFileAtomicWithChecksum(src, dst string, perm os.FileMode) (string, error) {
|
|
||||||
if strings.TrimSpace(src) == "" || strings.TrimSpace(dst) == "" {
|
|
||||||
return "", fmt.Errorf("source and destination paths are required")
|
|
||||||
}
|
|
||||||
in, err := os.Open(src)
|
|
||||||
if err != nil {
|
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
defer func() { _ = in.Close() }()
|
|
||||||
|
|
||||||
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
|
|
||||||
return "", fmt.Errorf("create destination directory: %w", err)
|
|
||||||
}
|
|
||||||
base := filepath.Base(dst)
|
|
||||||
tmp, err := os.CreateTemp(filepath.Dir(dst), "."+base+".tmp-*")
|
|
||||||
if err != nil {
|
|
||||||
return "", fmt.Errorf("create temp file: %w", err)
|
|
||||||
}
|
|
||||||
tmpPath := tmp.Name()
|
|
||||||
removeTmp := true
|
|
||||||
defer func() {
|
|
||||||
if removeTmp {
|
|
||||||
_ = os.Remove(tmpPath)
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
|
|
||||||
digest := sha256.New()
|
|
||||||
if _, err := io.Copy(io.MultiWriter(tmp, digest), in); err != nil {
|
|
||||||
_ = tmp.Close()
|
|
||||||
return "", fmt.Errorf("copy file: %w", err)
|
|
||||||
}
|
|
||||||
if err := tmp.Sync(); err != nil {
|
|
||||||
_ = tmp.Close()
|
|
||||||
return "", fmt.Errorf("sync temp file: %w", err)
|
|
||||||
}
|
|
||||||
if err := tmp.Close(); err != nil {
|
|
||||||
return "", fmt.Errorf("close temp file: %w", err)
|
|
||||||
}
|
|
||||||
if err := os.Chmod(tmpPath, perm); err != nil {
|
|
||||||
return "", fmt.Errorf("chmod temp file: %w", err)
|
|
||||||
}
|
|
||||||
if err := os.Rename(tmpPath, dst); err != nil {
|
|
||||||
return "", fmt.Errorf("install temp file: %w", err)
|
|
||||||
}
|
|
||||||
removeTmp = false
|
|
||||||
return hex.EncodeToString(digest.Sum(nil)), nil
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -153,7 +153,7 @@ storage:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestSpoolAndArchiveDefaults(t *testing.T) {
|
func TestSpoolAndPublishDefaults(t *testing.T) {
|
||||||
pipelinePath, sessionPath := writeConfigFiles(t, testPipelineBaseYAML, testSessionBaseYAML)
|
pipelinePath, sessionPath := writeConfigFiles(t, testPipelineBaseYAML, testSessionBaseYAML)
|
||||||
|
|
||||||
cfg, err := Load(pipelinePath, sessionPath)
|
cfg, err := Load(pipelinePath, sessionPath)
|
||||||
@@ -171,13 +171,13 @@ func TestSpoolAndArchiveDefaults(t *testing.T) {
|
|||||||
t.Fatalf("workspace.cleanup_after_publish = true, want false")
|
t.Fatalf("workspace.cleanup_after_publish = true, want false")
|
||||||
}
|
}
|
||||||
if cfg.Pipeline.Publish == nil {
|
if cfg.Pipeline.Publish == nil {
|
||||||
t.Fatal("archive should be initialized by defaults")
|
t.Fatal("publish should be initialized by defaults")
|
||||||
}
|
}
|
||||||
if cfg.Pipeline.Publish.Enabled == nil || !*cfg.Pipeline.Publish.Enabled {
|
if cfg.Pipeline.Publish.Enabled == nil || !*cfg.Pipeline.Publish.Enabled {
|
||||||
t.Fatalf("archive.enabled = %#v, want true", cfg.Pipeline.Publish.Enabled)
|
t.Fatalf("publish.enabled = %#v, want true", cfg.Pipeline.Publish.Enabled)
|
||||||
}
|
}
|
||||||
if cfg.Pipeline.Publish.UploadRun == nil || !*cfg.Pipeline.Publish.UploadRun {
|
if cfg.Pipeline.Publish.UploadRun == nil || !*cfg.Pipeline.Publish.UploadRun {
|
||||||
t.Fatalf("archive.upload_run = %#v, want true", cfg.Pipeline.Publish.UploadRun)
|
t.Fatalf("publish.upload_run = %#v, want true", cfg.Pipeline.Publish.UploadRun)
|
||||||
}
|
}
|
||||||
if len(cfg.Pipeline.Publish.Outputs) != 1 {
|
if len(cfg.Pipeline.Publish.Outputs) != 1 {
|
||||||
t.Fatalf("publish.outputs len = %d, want 1 default", len(cfg.Pipeline.Publish.Outputs))
|
t.Fatalf("publish.outputs len = %d, want 1 default", len(cfg.Pipeline.Publish.Outputs))
|
||||||
@@ -194,7 +194,7 @@ func TestSpoolAndArchiveDefaults(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchivePromotionValidation(t *testing.T) {
|
func TestPublishOutputValidation(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
ruleYML string
|
ruleYML string
|
||||||
@@ -278,7 +278,7 @@ publish:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchivePromotionLegacyTranscriptSourcesRejected(t *testing.T) {
|
func TestPublishOutputLegacyTranscriptSourcesRejected(t *testing.T) {
|
||||||
legacyTranscriptSources := []string{
|
legacyTranscriptSources := []string{
|
||||||
"narratio.transcript." + "merged",
|
"narratio.transcript." + "merged",
|
||||||
"narratio.transcript." + "full",
|
"narratio.transcript." + "full",
|
||||||
@@ -307,7 +307,7 @@ publish:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchivePromotionDerivesDestinationWhenOmitted(t *testing.T) {
|
func TestPublishOutputDerivesDestinationWhenOmitted(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
pipelineYML string
|
pipelineYML string
|
||||||
@@ -359,7 +359,7 @@ publish:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveLockValidation(t *testing.T) {
|
func TestPublishLockValidation(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
pipelineYML string
|
pipelineYML string
|
||||||
@@ -439,7 +439,7 @@ publish:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveLockLegacyTranscriptSourcesRejected(t *testing.T) {
|
func TestPublishLockLegacyTranscriptSourcesRejected(t *testing.T) {
|
||||||
legacyTranscriptSources := []string{
|
legacyTranscriptSources := []string{
|
||||||
"narratio.transcript." + "merged",
|
"narratio.transcript." + "merged",
|
||||||
"narratio.transcript." + "full",
|
"narratio.transcript." + "full",
|
||||||
@@ -467,7 +467,7 @@ publish:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveLockUnknownFieldFailsStrictDecode(t *testing.T) {
|
func TestPublishLockUnknownFieldFailsStrictDecode(t *testing.T) {
|
||||||
pipelineYAML := testPipelineBaseYAML + `
|
pipelineYAML := testPipelineBaseYAML + `
|
||||||
publish:
|
publish:
|
||||||
locks:
|
locks:
|
||||||
@@ -481,7 +481,7 @@ publish:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveLegacyFromToFailsStrictDecode(t *testing.T) {
|
func TestPublishLegacyFromToFailsStrictDecode(t *testing.T) {
|
||||||
pipelineYAML := testPipelineBaseYAML + `
|
pipelineYAML := testPipelineBaseYAML + `
|
||||||
publish:
|
publish:
|
||||||
outputs:
|
outputs:
|
||||||
@@ -495,7 +495,7 @@ publish:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveLockStoreBytesStrictDecodeAndValidation(t *testing.T) {
|
func TestPublishLockStoreBytesStrictDecodeAndValidation(t *testing.T) {
|
||||||
store, err := LoadPublishLockStoreBytes("locks.yml", []byte(`locks:
|
store, err := LoadPublishLockStoreBytes("locks.yml", []byte(`locks:
|
||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
reason: reviewed
|
reason: reviewed
|
||||||
@@ -524,7 +524,7 @@ func TestArchiveLockStoreBytesStrictDecodeAndValidation(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestMergeArchiveLockRulesStaticWins(t *testing.T) {
|
func TestMergePublishLockRulesStaticWins(t *testing.T) {
|
||||||
merged := MergePublishLockRules(
|
merged := MergePublishLockRules(
|
||||||
[]PublishLockRule{{Source: "narratio.transcript.final_trimmed", Reason: "static"}},
|
[]PublishLockRule{{Source: "narratio.transcript.final_trimmed", Reason: "static"}},
|
||||||
[]PublishLockRule{
|
[]PublishLockRule{
|
||||||
@@ -1,6 +1,7 @@
|
|||||||
package config
|
package config
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"net/url"
|
"net/url"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
@@ -8,7 +9,8 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/pathsafe"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Validate checks resolved configuration for required fields and parseable durations.
|
// Validate checks resolved configuration for required fields and parseable durations.
|
||||||
@@ -133,6 +135,7 @@ func validatePublish(cfg *PublishConfig, scriptorium *ScriptoriumConfig) error {
|
|||||||
if cfg == nil {
|
if cfg == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
configuredOutputs := scriptoriumOutputPathMap(scriptorium)
|
||||||
seenDest := map[string]struct{}{}
|
seenDest := map[string]struct{}{}
|
||||||
for i, item := range cfg.Outputs {
|
for i, item := range cfg.Outputs {
|
||||||
prefix := fmt.Sprintf("pipeline.publish.outputs[%d]", i)
|
prefix := fmt.Sprintf("pipeline.publish.outputs[%d]", i)
|
||||||
@@ -140,22 +143,29 @@ func validatePublish(cfg *PublishConfig, scriptorium *ScriptoriumConfig) error {
|
|||||||
if source == "" {
|
if source == "" {
|
||||||
return fmt.Errorf("%s.source is required", prefix)
|
return fmt.Errorf("%s.source is required", prefix)
|
||||||
}
|
}
|
||||||
if _, err := publishSourceKnown(source, scriptorium); err != nil {
|
if _, err := artifactpolicy.ValidatePublishSource(source, configuredOutputs); err != nil {
|
||||||
return fmt.Errorf("%s.source %q is unsupported: %w", prefix, item.Source, err)
|
return fmt.Errorf("%s.source %q is unsupported: %w", prefix, item.Source, err)
|
||||||
}
|
}
|
||||||
dest := strings.TrimSpace(item.Dest)
|
dest := strings.TrimSpace(item.Dest)
|
||||||
if dest == "" {
|
if dest == "" {
|
||||||
derivedDest, err := derivePublishOutputDest(source, scriptorium)
|
derivedDest, err := artifactpolicy.ResolvePublishedDestination(source, "", configuredOutputs)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("%s.dest is required when destination cannot be derived from %q: %w", prefix, source, err)
|
return fmt.Errorf("%s.dest is required when destination cannot be derived from %q: %w", prefix, source, err)
|
||||||
}
|
}
|
||||||
dest = derivedDest
|
dest = derivedDest
|
||||||
cfg.Outputs[i].Dest = derivedDest
|
cfg.Outputs[i].Dest = derivedDest
|
||||||
}
|
}
|
||||||
if err := validateRelativeSafePath(prefix+".dest", dest); err != nil {
|
normalizedDest, err := pathsafe.NormalizeRelativeDestination(dest)
|
||||||
return err
|
if err != nil {
|
||||||
|
switch {
|
||||||
|
case errors.Is(err, pathsafe.ErrRelativePathAbsolute):
|
||||||
|
return fmt.Errorf("%s.dest must be a relative path", prefix)
|
||||||
|
case errors.Is(err, pathsafe.ErrRelativePathEscape):
|
||||||
|
return fmt.Errorf("%s.dest must not contain path traversal", prefix)
|
||||||
|
default:
|
||||||
|
return fmt.Errorf("%s.dest must be non-empty", prefix)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
normalizedDest := filepath.ToSlash(filepath.Clean(dest))
|
|
||||||
if _, ok := seenDest[normalizedDest]; ok {
|
if _, ok := seenDest[normalizedDest]; ok {
|
||||||
return fmt.Errorf("%s.dest %q duplicates another publish output destination", prefix, dest)
|
return fmt.Errorf("%s.dest %q duplicates another publish output destination", prefix, dest)
|
||||||
}
|
}
|
||||||
@@ -173,6 +183,7 @@ func validatePublish(cfg *PublishConfig, scriptorium *ScriptoriumConfig) error {
|
|||||||
func ValidatePublishLockRules(locks []PublishLockRule, scriptorium *ScriptoriumConfig, label string) ([]PublishLockRule, error) {
|
func ValidatePublishLockRules(locks []PublishLockRule, scriptorium *ScriptoriumConfig, label string) ([]PublishLockRule, error) {
|
||||||
seenLocks := map[string]struct{}{}
|
seenLocks := map[string]struct{}{}
|
||||||
out := make([]PublishLockRule, 0, len(locks))
|
out := make([]PublishLockRule, 0, len(locks))
|
||||||
|
configuredOutputs := scriptoriumOutputPathMap(scriptorium)
|
||||||
if strings.TrimSpace(label) == "" {
|
if strings.TrimSpace(label) == "" {
|
||||||
label = "publish.locks"
|
label = "publish.locks"
|
||||||
}
|
}
|
||||||
@@ -182,7 +193,7 @@ func ValidatePublishLockRules(locks []PublishLockRule, scriptorium *ScriptoriumC
|
|||||||
if source == "" {
|
if source == "" {
|
||||||
return nil, fmt.Errorf("%s.source is required", prefix)
|
return nil, fmt.Errorf("%s.source is required", prefix)
|
||||||
}
|
}
|
||||||
if _, err := publishSourceKnown(source, scriptorium); err != nil {
|
if _, err := artifactpolicy.ValidatePublishSource(source, configuredOutputs); err != nil {
|
||||||
return nil, fmt.Errorf("%s.source %q is unsupported: %w", prefix, item.Source, err)
|
return nil, fmt.Errorf("%s.source %q is unsupported: %w", prefix, item.Source, err)
|
||||||
}
|
}
|
||||||
if _, ok := seenLocks[source]; ok {
|
if _, ok := seenLocks[source]; ok {
|
||||||
@@ -224,48 +235,15 @@ func MergePublishLockRules(staticLocks, remoteLocks []PublishLockRule) []Publish
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func publishSourceKnown(source string, scriptorium *ScriptoriumConfig) (string, error) {
|
func scriptoriumOutputPathMap(scriptorium *ScriptoriumConfig) map[string]string {
|
||||||
trimmed := strings.TrimSpace(source)
|
out := map[string]string{}
|
||||||
if _, ok := artifactmodel.LookupRuntimeTranscriptArtifact(trimmed); ok {
|
|
||||||
return "", nil
|
|
||||||
}
|
|
||||||
switch trimmed {
|
|
||||||
case "narratio.bounds.session":
|
|
||||||
return "", nil
|
|
||||||
}
|
|
||||||
matches := narratioArtifactSourceRE.FindStringSubmatch(trimmed)
|
|
||||||
if len(matches) != 2 {
|
|
||||||
return "", fmt.Errorf("must be a built-in source id or narratio.artifact.<name>")
|
|
||||||
}
|
|
||||||
artifactKey := matches[1]
|
|
||||||
if scriptorium == nil || len(scriptorium.Artifacts) == 0 {
|
if scriptorium == nil || len(scriptorium.Artifacts) == 0 {
|
||||||
return "", fmt.Errorf("configured artifact %q is not defined in pipeline.scriptorium.artifacts", artifactKey)
|
return out
|
||||||
}
|
}
|
||||||
if _, ok := scriptorium.Artifacts[artifactKey]; !ok {
|
for key, artifactCfg := range scriptorium.Artifacts {
|
||||||
return "", fmt.Errorf("configured artifact %q is not defined in pipeline.scriptorium.artifacts", artifactKey)
|
out[strings.TrimSpace(key)] = strings.TrimSpace(artifactCfg.OutputPath)
|
||||||
}
|
}
|
||||||
return artifactKey, nil
|
return out
|
||||||
}
|
|
||||||
|
|
||||||
func derivePublishOutputDest(source string, scriptorium *ScriptoriumConfig) (string, error) {
|
|
||||||
trimmed := strings.TrimSpace(source)
|
|
||||||
if spec, ok := artifactmodel.LookupRuntimeTranscriptArtifact(trimmed); ok {
|
|
||||||
return spec.CanonicalRelPath, nil
|
|
||||||
}
|
|
||||||
switch trimmed {
|
|
||||||
case "narratio.bounds.session":
|
|
||||||
return filepath.ToSlash(filepath.Join(PathArtifactsDirSegment, "session_bounds.json")), nil
|
|
||||||
}
|
|
||||||
artifactKey, err := publishSourceKnown(trimmed, scriptorium)
|
|
||||||
if err != nil {
|
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
artifactCfg := scriptorium.Artifacts[artifactKey]
|
|
||||||
outputPath := strings.TrimSpace(artifactCfg.OutputPath)
|
|
||||||
if outputPath == "" {
|
|
||||||
return "", fmt.Errorf("pipeline.scriptorium.artifacts.%s.output_path is empty", artifactKey)
|
|
||||||
}
|
|
||||||
return outputPath, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func validateSecrets(cfg *SecretsConfig) error {
|
func validateSecrets(cfg *SecretsConfig) error {
|
||||||
@@ -657,18 +635,12 @@ func publishUploadConfiguredForS3(pipeline *PipelineConfig) bool {
|
|||||||
var windowsAbsPathRE = regexp.MustCompile(`^[A-Za-z]:[\\/].*`)
|
var windowsAbsPathRE = regexp.MustCompile(`^[A-Za-z]:[\\/].*`)
|
||||||
var envVarNameRE = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
|
var envVarNameRE = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
|
||||||
var scriptoriumArtifactKeyRE = regexp.MustCompile(`^[a-z][a-z0-9_]*$`)
|
var scriptoriumArtifactKeyRE = regexp.MustCompile(`^[a-z][a-z0-9_]*$`)
|
||||||
var narratioArtifactSourceRE = regexp.MustCompile(`^narratio\.artifact\.([a-z][a-z0-9_]*)$`)
|
|
||||||
var narratioPreviousSessionArtifactSourceRE = regexp.MustCompile(`^narratio\.previous_session\.artifact\.([a-z][a-z0-9_]*)$`)
|
|
||||||
|
|
||||||
func validateScriptoriumInputSource(artifactName, inputName, source string, configuredArtifacts map[string]struct{}) (string, error) {
|
func validateScriptoriumInputSource(artifactName, inputName, source string, configuredArtifacts map[string]struct{}) (string, error) {
|
||||||
trimmedSource := strings.TrimSpace(source)
|
trimmedSource := strings.TrimSpace(source)
|
||||||
if isStaticSupportedScriptoriumInputSource(trimmedSource) {
|
descriptor, err := artifactpolicy.DescribeScriptoriumInputSource(trimmedSource)
|
||||||
return "", nil
|
if err != nil {
|
||||||
}
|
if errors.Is(err, artifactpolicy.ErrInvalidPreviousSessionSource) {
|
||||||
|
|
||||||
if strings.HasPrefix(trimmedSource, "narratio.previous_session.artifact") {
|
|
||||||
matches := narratioPreviousSessionArtifactSourceRE.FindStringSubmatch(trimmedSource)
|
|
||||||
if len(matches) != 2 {
|
|
||||||
return "", fmt.Errorf(
|
return "", fmt.Errorf(
|
||||||
"pipeline.scriptorium.artifacts.%s.inputs.%s.source %q must reference configured artifact key matching ^[a-z][a-z0-9_]*$",
|
"pipeline.scriptorium.artifacts.%s.inputs.%s.source %q must reference configured artifact key matching ^[a-z][a-z0-9_]*$",
|
||||||
artifactName,
|
artifactName,
|
||||||
@@ -676,21 +648,6 @@ func validateScriptoriumInputSource(artifactName, inputName, source string, conf
|
|||||||
source,
|
source,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
referenced := matches[1]
|
|
||||||
if _, ok := configuredArtifacts[referenced]; !ok {
|
|
||||||
return "", fmt.Errorf(
|
|
||||||
"pipeline.scriptorium.artifacts.%s.inputs.%s.source %q references unknown artifact %q",
|
|
||||||
artifactName,
|
|
||||||
inputName,
|
|
||||||
source,
|
|
||||||
referenced,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
return "", nil
|
|
||||||
}
|
|
||||||
|
|
||||||
matches := narratioArtifactSourceRE.FindStringSubmatch(trimmedSource)
|
|
||||||
if len(matches) != 2 {
|
|
||||||
return "", fmt.Errorf(
|
return "", fmt.Errorf(
|
||||||
"pipeline.scriptorium.artifacts.%s.inputs.%s.source %q is unsupported",
|
"pipeline.scriptorium.artifacts.%s.inputs.%s.source %q is unsupported",
|
||||||
artifactName,
|
artifactName,
|
||||||
@@ -698,29 +655,28 @@ func validateScriptoriumInputSource(artifactName, inputName, source string, conf
|
|||||||
source,
|
source,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
referenced := matches[1]
|
if err := artifactpolicy.ValidateInputConfiguredReference(descriptor, configuredArtifacts); err != nil {
|
||||||
if _, ok := configuredArtifacts[referenced]; !ok {
|
var unknownConfigured *artifactpolicy.UnknownConfiguredArtifactError
|
||||||
|
if errors.As(err, &unknownConfigured) {
|
||||||
|
return "", fmt.Errorf(
|
||||||
|
"pipeline.scriptorium.artifacts.%s.inputs.%s.source %q references unknown artifact %q",
|
||||||
|
artifactName,
|
||||||
|
inputName,
|
||||||
|
source,
|
||||||
|
unknownConfigured.ConfiguredKey,
|
||||||
|
)
|
||||||
|
}
|
||||||
return "", fmt.Errorf(
|
return "", fmt.Errorf(
|
||||||
"pipeline.scriptorium.artifacts.%s.inputs.%s.source %q references unknown artifact %q",
|
"pipeline.scriptorium.artifacts.%s.inputs.%s.source %q is unsupported",
|
||||||
artifactName,
|
artifactName,
|
||||||
inputName,
|
inputName,
|
||||||
source,
|
source,
|
||||||
referenced,
|
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
return referenced, nil
|
if descriptor.Source.Kind == artifactpolicy.SourceKindConfiguredArtifact {
|
||||||
}
|
return descriptor.Source.ConfiguredKey, nil
|
||||||
|
|
||||||
func isStaticSupportedScriptoriumInputSource(source string) bool {
|
|
||||||
if _, ok := artifactmodel.LookupRuntimeTranscriptArtifact(source); ok {
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
switch source {
|
|
||||||
case "narratio.bounds.session":
|
|
||||||
return true
|
|
||||||
default:
|
|
||||||
return false
|
|
||||||
}
|
}
|
||||||
|
return "", nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func validateEnvVarNameField(fieldName, value string) error {
|
func validateEnvVarNameField(fieldName, value string) error {
|
||||||
|
|||||||
128
internal/fileops/fileops.go
Normal file
128
internal/fileops/fileops.go
Normal file
@@ -0,0 +1,128 @@
|
|||||||
|
package fileops
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// WriteFileAtomic writes data to dst atomically via temp file + rename.
|
||||||
|
func WriteFileAtomic(dst string, data []byte, perm os.FileMode) error {
|
||||||
|
if strings.TrimSpace(dst) == "" {
|
||||||
|
return fmt.Errorf("destination path is required")
|
||||||
|
}
|
||||||
|
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
|
||||||
|
return fmt.Errorf("create destination directory: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
base := filepath.Base(dst)
|
||||||
|
tmp, err := os.CreateTemp(filepath.Dir(dst), "."+base+".tmp-*")
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("create temp file: %w", err)
|
||||||
|
}
|
||||||
|
tmpPath := tmp.Name()
|
||||||
|
removeTmp := true
|
||||||
|
defer func() {
|
||||||
|
if removeTmp {
|
||||||
|
_ = os.Remove(tmpPath)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
if _, err := tmp.Write(data); err != nil {
|
||||||
|
_ = tmp.Close()
|
||||||
|
return fmt.Errorf("write temp file: %w", err)
|
||||||
|
}
|
||||||
|
if err := tmp.Sync(); err != nil {
|
||||||
|
_ = tmp.Close()
|
||||||
|
return fmt.Errorf("sync temp file: %w", err)
|
||||||
|
}
|
||||||
|
if err := tmp.Close(); err != nil {
|
||||||
|
return fmt.Errorf("close temp file: %w", err)
|
||||||
|
}
|
||||||
|
if err := os.Chmod(tmpPath, perm); err != nil {
|
||||||
|
return fmt.Errorf("set temp file permissions: %w", err)
|
||||||
|
}
|
||||||
|
if err := os.Rename(tmpPath, dst); err != nil {
|
||||||
|
return fmt.Errorf("install temp file: %w", err)
|
||||||
|
}
|
||||||
|
removeTmp = false
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// CopyFileAtomic copies src to dst atomically via temp file + rename.
|
||||||
|
func CopyFileAtomic(src, dst string, perm os.FileMode) error {
|
||||||
|
_, err := CopyFileAtomicWithChecksum(src, dst, perm)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// CopyFileAtomicWithChecksum copies src to dst atomically and returns the SHA-256 checksum.
|
||||||
|
func CopyFileAtomicWithChecksum(src, dst string, perm os.FileMode) (string, error) {
|
||||||
|
if strings.TrimSpace(src) == "" || strings.TrimSpace(dst) == "" {
|
||||||
|
return "", fmt.Errorf("source and destination paths are required")
|
||||||
|
}
|
||||||
|
|
||||||
|
in, err := os.Open(src)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("open source file: %w", err)
|
||||||
|
}
|
||||||
|
defer func() { _ = in.Close() }()
|
||||||
|
|
||||||
|
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
|
||||||
|
return "", fmt.Errorf("create destination directory: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
base := filepath.Base(dst)
|
||||||
|
tmp, err := os.CreateTemp(filepath.Dir(dst), "."+base+".tmp-*")
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("create temp file: %w", err)
|
||||||
|
}
|
||||||
|
tmpPath := tmp.Name()
|
||||||
|
removeTmp := true
|
||||||
|
defer func() {
|
||||||
|
if removeTmp {
|
||||||
|
_ = os.Remove(tmpPath)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
digest := sha256.New()
|
||||||
|
if _, err := io.Copy(io.MultiWriter(tmp, digest), in); err != nil {
|
||||||
|
_ = tmp.Close()
|
||||||
|
return "", fmt.Errorf("copy file: %w", err)
|
||||||
|
}
|
||||||
|
if err := tmp.Sync(); err != nil {
|
||||||
|
_ = tmp.Close()
|
||||||
|
return "", fmt.Errorf("sync temp file: %w", err)
|
||||||
|
}
|
||||||
|
if err := tmp.Close(); err != nil {
|
||||||
|
return "", fmt.Errorf("close temp file: %w", err)
|
||||||
|
}
|
||||||
|
if err := os.Chmod(tmpPath, perm); err != nil {
|
||||||
|
return "", fmt.Errorf("set temp file permissions: %w", err)
|
||||||
|
}
|
||||||
|
if err := os.Rename(tmpPath, dst); err != nil {
|
||||||
|
return "", fmt.Errorf("install temp file: %w", err)
|
||||||
|
}
|
||||||
|
removeTmp = false
|
||||||
|
return hex.EncodeToString(digest.Sum(nil)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// InstallDownloadedTempFile installs a previously downloaded temp file at dst.
|
||||||
|
func InstallDownloadedTempFile(tmpPath, dst string, perm os.FileMode) error {
|
||||||
|
if strings.TrimSpace(tmpPath) == "" || strings.TrimSpace(dst) == "" {
|
||||||
|
return fmt.Errorf("temp and destination paths are required")
|
||||||
|
}
|
||||||
|
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
|
||||||
|
return fmt.Errorf("create destination directory: %w", err)
|
||||||
|
}
|
||||||
|
if err := os.Chmod(tmpPath, perm); err != nil {
|
||||||
|
return fmt.Errorf("set temp file permissions: %w", err)
|
||||||
|
}
|
||||||
|
if err := os.Rename(tmpPath, dst); err != nil {
|
||||||
|
return fmt.Errorf("install downloaded file: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
125
internal/fileops/fileops_test.go
Normal file
125
internal/fileops/fileops_test.go
Normal file
@@ -0,0 +1,125 @@
|
|||||||
|
package fileops
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestWriteFileAtomicOverwritesAndLeavesNoTempFile(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
dst := filepath.Join(root, "out", "value.txt")
|
||||||
|
|
||||||
|
if err := WriteFileAtomic(dst, []byte("one"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFileAtomic(first) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := WriteFileAtomic(dst, []byte("two"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFileAtomic(second) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := os.ReadFile(dst)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile() error = %v", err)
|
||||||
|
}
|
||||||
|
if string(data) != "two" {
|
||||||
|
t.Fatalf("file content = %q, want %q", string(data), "two")
|
||||||
|
}
|
||||||
|
|
||||||
|
assertNoMatchingTempFiles(t, filepath.Dir(dst), "."+filepath.Base(dst)+".tmp-")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWriteFileAtomicCleansTempFileOnInstallFailure(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
blockedPath := filepath.Join(root, "blocked")
|
||||||
|
if err := os.MkdirAll(blockedPath, 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(blockedPath) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err := WriteFileAtomic(blockedPath, []byte("data"), 0o644)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("WriteFileAtomic() error = nil, want install failure")
|
||||||
|
}
|
||||||
|
assertNoMatchingTempFiles(t, root, ".blocked.tmp-")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCopyFileAtomicWithChecksumMatchesDestination(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
src := filepath.Join(root, "source.txt")
|
||||||
|
dst := filepath.Join(root, "out", "copied.txt")
|
||||||
|
if err := os.WriteFile(src, []byte("copied-data"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(source) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
checksum, err := CopyFileAtomicWithChecksum(src, dst, 0o644)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("CopyFileAtomicWithChecksum() error = %v", err)
|
||||||
|
}
|
||||||
|
wantChecksum := "6e5c3f239e28cc315d57b2fcfc24169369c44a25802c0616a6d7081707fd24df"
|
||||||
|
if checksum != wantChecksum {
|
||||||
|
t.Fatalf("checksum = %q, want %q", checksum, wantChecksum)
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := os.ReadFile(dst)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(destination) error = %v", err)
|
||||||
|
}
|
||||||
|
if string(data) != "copied-data" {
|
||||||
|
t.Fatalf("destination content = %q, want %q", string(data), "copied-data")
|
||||||
|
}
|
||||||
|
assertNoMatchingTempFiles(t, filepath.Dir(dst), ".copied.txt.tmp-")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCopyFileAtomicCleansTempFileOnInstallFailure(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
src := filepath.Join(root, "source.txt")
|
||||||
|
if err := os.WriteFile(src, []byte("copied-data"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(source) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
blockedPath := filepath.Join(root, "blocked")
|
||||||
|
if err := os.MkdirAll(blockedPath, 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(blockedPath) error = %v", err)
|
||||||
|
}
|
||||||
|
err := CopyFileAtomic(src, blockedPath, 0o644)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("CopyFileAtomic() error = nil, want install failure")
|
||||||
|
}
|
||||||
|
assertNoMatchingTempFiles(t, root, ".blocked.tmp-")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestInstallDownloadedTempFileSetsPermissions(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
tmpPath := filepath.Join(root, ".payload.tmp")
|
||||||
|
dst := filepath.Join(root, "out", "payload.json")
|
||||||
|
if err := os.WriteFile(tmpPath, []byte("{\"ok\":true}\n"), 0o600); err != nil {
|
||||||
|
t.Fatalf("WriteFile(temp) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := InstallDownloadedTempFile(tmpPath, dst, 0o644); err != nil {
|
||||||
|
t.Fatalf("InstallDownloadedTempFile() error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(tmpPath); !os.IsNotExist(err) {
|
||||||
|
t.Fatalf("temp file still exists: stat err = %v", err)
|
||||||
|
}
|
||||||
|
info, err := os.Stat(dst)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Stat(destination) error = %v", err)
|
||||||
|
}
|
||||||
|
if info.Mode().Perm() != 0o644 {
|
||||||
|
t.Fatalf("destination mode = %o, want 644", info.Mode().Perm())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertNoMatchingTempFiles(t *testing.T, dir, prefix string) {
|
||||||
|
t.Helper()
|
||||||
|
entries, err := os.ReadDir(dir)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadDir(%q) error = %v", dir, err)
|
||||||
|
}
|
||||||
|
for _, e := range entries {
|
||||||
|
if strings.HasPrefix(e.Name(), prefix) {
|
||||||
|
t.Fatalf("unexpected temp file residue: %s", filepath.Join(dir, e.Name()))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
38
internal/pathsafe/relative_destination.go
Normal file
38
internal/pathsafe/relative_destination.go
Normal file
@@ -0,0 +1,38 @@
|
|||||||
|
package pathsafe
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"path"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
ErrRelativePathRequired = errors.New("relative path is required")
|
||||||
|
ErrRelativePathAbsolute = errors.New("relative path must not be absolute")
|
||||||
|
ErrRelativePathEscape = errors.New("relative path escapes root")
|
||||||
|
)
|
||||||
|
|
||||||
|
var windowsAbsPathRE = regexp.MustCompile(`^[a-zA-Z]:[\\/]`)
|
||||||
|
|
||||||
|
// NormalizeRelativeDestination validates and normalizes a relative destination
|
||||||
|
// path to slash-separated form.
|
||||||
|
func NormalizeRelativeDestination(relative string) (string, error) {
|
||||||
|
trimmed := strings.TrimSpace(relative)
|
||||||
|
if trimmed == "" {
|
||||||
|
return "", ErrRelativePathRequired
|
||||||
|
}
|
||||||
|
if strings.HasPrefix(trimmed, "/") || strings.HasPrefix(trimmed, `\`) || windowsAbsPathRE.MatchString(trimmed) {
|
||||||
|
return "", ErrRelativePathAbsolute
|
||||||
|
}
|
||||||
|
|
||||||
|
normalized := strings.ReplaceAll(trimmed, `\`, "/")
|
||||||
|
cleaned := path.Clean(normalized)
|
||||||
|
if cleaned == "." || cleaned == "" {
|
||||||
|
return "", ErrRelativePathRequired
|
||||||
|
}
|
||||||
|
if cleaned == ".." || strings.HasPrefix(cleaned, "../") {
|
||||||
|
return "", ErrRelativePathEscape
|
||||||
|
}
|
||||||
|
return cleaned, nil
|
||||||
|
}
|
||||||
118
internal/pathsafe/relative_destination_test.go
Normal file
118
internal/pathsafe/relative_destination_test.go
Normal file
@@ -0,0 +1,118 @@
|
|||||||
|
package pathsafe
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestNormalizeRelativeDestination(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
input string
|
||||||
|
want string
|
||||||
|
wantErr error
|
||||||
|
}{
|
||||||
|
{name: "simple", input: "artifacts/session_recap.md", want: "artifacts/session_recap.md"},
|
||||||
|
{name: "backslashes become slashes", input: `artifacts\session_recap.md`, want: "artifacts/session_recap.md"},
|
||||||
|
{name: "cleans dot segments", input: "artifacts/./session_recap.md", want: "artifacts/session_recap.md"},
|
||||||
|
{name: "reject empty", input: "", wantErr: ErrRelativePathRequired},
|
||||||
|
{name: "reject dot", input: ".", wantErr: ErrRelativePathRequired},
|
||||||
|
{name: "reject unix absolute", input: "/artifacts/session_recap.md", wantErr: ErrRelativePathAbsolute},
|
||||||
|
{name: "reject windows absolute", input: `C:\artifacts\session_recap.md`, wantErr: ErrRelativePathAbsolute},
|
||||||
|
{name: "reject traversal", input: "../artifacts/session_recap.md", wantErr: ErrRelativePathEscape},
|
||||||
|
{name: "reject traversal after clean", input: "a/../../session_recap.md", wantErr: ErrRelativePathEscape},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got, err := NormalizeRelativeDestination(tt.input)
|
||||||
|
if tt.wantErr != nil {
|
||||||
|
if !errors.Is(err, tt.wantErr) {
|
||||||
|
t.Fatalf("NormalizeRelativeDestination() error = %v, want %v", err, tt.wantErr)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NormalizeRelativeDestination() error = %v", err)
|
||||||
|
}
|
||||||
|
if got != tt.want {
|
||||||
|
t.Fatalf("NormalizeRelativeDestination() = %q, want %q", got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestJoinSlashRelativeUnderRoot(t *testing.T) {
|
||||||
|
root := filepath.Join(t.TempDir(), "session")
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
input string
|
||||||
|
want string
|
||||||
|
wantErr error
|
||||||
|
}{
|
||||||
|
{name: "valid relative", input: "artifacts/session_recap.md", want: filepath.Join(root, "artifacts", "session_recap.md")},
|
||||||
|
{name: "windows separators normalized", input: `artifacts\session_recap.md`, want: filepath.Join(root, "artifacts", "session_recap.md")},
|
||||||
|
{name: "reject empty", input: "", wantErr: ErrRelativePathRequired},
|
||||||
|
{name: "reject absolute", input: "/artifacts/session_recap.md", wantErr: ErrRelativePathAbsolute},
|
||||||
|
{name: "reject traversal", input: "../artifacts/session_recap.md", wantErr: ErrRelativePathEscape},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got, err := JoinSlashRelativeUnderRoot(root, tt.input)
|
||||||
|
if tt.wantErr != nil {
|
||||||
|
if !errors.Is(err, tt.wantErr) {
|
||||||
|
t.Fatalf("JoinSlashRelativeUnderRoot() error = %v, want %v", err, tt.wantErr)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("JoinSlashRelativeUnderRoot() error = %v", err)
|
||||||
|
}
|
||||||
|
if got != tt.want {
|
||||||
|
t.Fatalf("JoinSlashRelativeUnderRoot() = %q, want %q", got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSlashRelativeFromRoot(t *testing.T) {
|
||||||
|
root := filepath.Join(t.TempDir(), "session")
|
||||||
|
target := filepath.Join(root, "transcripts", "full.json")
|
||||||
|
|
||||||
|
got, err := SlashRelativeFromRoot(root, target)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SlashRelativeFromRoot() error = %v", err)
|
||||||
|
}
|
||||||
|
if got != "transcripts/full.json" {
|
||||||
|
t.Fatalf("SlashRelativeFromRoot() = %q, want transcripts/full.json", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
got, err = SlashRelativeFromRoot(root, `transcripts\full.json`)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SlashRelativeFromRoot(relative with windows separators) error = %v", err)
|
||||||
|
}
|
||||||
|
if got != "transcripts/full.json" {
|
||||||
|
t.Fatalf("SlashRelativeFromRoot(relative with windows separators) = %q, want transcripts/full.json", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSlashRelativeFromRootRejectsOutsideRoot(t *testing.T) {
|
||||||
|
root := filepath.Join(t.TempDir(), "session")
|
||||||
|
outside := filepath.Join(filepath.Dir(root), "outside", "file.txt")
|
||||||
|
|
||||||
|
_, err := SlashRelativeFromRoot(root, outside)
|
||||||
|
if !errors.Is(err, ErrRelativePathEscape) {
|
||||||
|
t.Fatalf("SlashRelativeFromRoot() error = %v, want %v", err, ErrRelativePathEscape)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestJoinSlashRelativeUnderRootRequiresRoot(t *testing.T) {
|
||||||
|
_, err := JoinSlashRelativeUnderRoot("", "artifacts/session_recap.md")
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "root path is required") {
|
||||||
|
t.Fatalf("JoinSlashRelativeUnderRoot() error = %v, want root-required error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
58
internal/pathsafe/root_scoped.go
Normal file
58
internal/pathsafe/root_scoped.go
Normal file
@@ -0,0 +1,58 @@
|
|||||||
|
package pathsafe
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// JoinSlashRelativeUnderRoot validates a slash-style relative path and resolves
|
||||||
|
// it under root. The returned path uses the host filepath separator.
|
||||||
|
func JoinSlashRelativeUnderRoot(root, relative string) (string, error) {
|
||||||
|
rootClean := filepath.Clean(strings.TrimSpace(root))
|
||||||
|
if rootClean == "." || rootClean == "" {
|
||||||
|
return "", fmt.Errorf("root path is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
normalized, err := NormalizeRelativeDestination(relative)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
|
||||||
|
joined := filepath.Clean(filepath.Join(rootClean, filepath.FromSlash(normalized)))
|
||||||
|
rel, err := filepath.Rel(rootClean, joined)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("resolve relative path under root: %w", err)
|
||||||
|
}
|
||||||
|
if rel == "." || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
||||||
|
return "", ErrRelativePathEscape
|
||||||
|
}
|
||||||
|
return joined, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// SlashRelativeFromRoot derives a slash-style relative path for target under
|
||||||
|
// root. Target may be absolute or relative to root.
|
||||||
|
func SlashRelativeFromRoot(root, target string) (string, error) {
|
||||||
|
rootClean := filepath.Clean(strings.TrimSpace(root))
|
||||||
|
if rootClean == "." || rootClean == "" {
|
||||||
|
return "", fmt.Errorf("root path is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
targetClean := filepath.Clean(strings.TrimSpace(target))
|
||||||
|
if targetClean == "." || targetClean == "" {
|
||||||
|
return "", ErrRelativePathRequired
|
||||||
|
}
|
||||||
|
if !filepath.IsAbs(targetClean) {
|
||||||
|
targetClean = filepath.Clean(filepath.Join(rootClean, targetClean))
|
||||||
|
}
|
||||||
|
|
||||||
|
rel, err := filepath.Rel(rootClean, targetClean)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("derive path relative to root: %w", err)
|
||||||
|
}
|
||||||
|
normalized, err := NormalizeRelativeDestination(filepath.ToSlash(rel))
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return normalized, nil
|
||||||
|
}
|
||||||
@@ -2,17 +2,19 @@ package previouscache
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
|
||||||
"path"
|
"path"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/pathsafe"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -90,86 +92,28 @@ func BuildPlan(
|
|||||||
campaign,
|
campaign,
|
||||||
previousSessionID,
|
previousSessionID,
|
||||||
)
|
)
|
||||||
currentManifestKey, currentRunIDKey := artifacts.ResolveArchiveCurrentStateKeys(previousSessionPrefix)
|
|
||||||
|
|
||||||
result := &Plan{}
|
result := &Plan{}
|
||||||
|
current, err := artifacts.LoadCurrentState(ctx, store, previousSessionPrefix, artifacts.CurrentStateValidation{
|
||||||
runPointerExists, err := store.Exists(ctx, currentRunIDKey)
|
ExpectedSessionID: previousSessionID,
|
||||||
|
ExpectedCampaign: campaign,
|
||||||
|
ValidateRunID: true,
|
||||||
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("check previous-session current run pointer %q: %w", currentRunIDKey, err)
|
var runPointerMissing *artifacts.CurrentRunPointerMissingError
|
||||||
}
|
var manifestMissing *artifacts.CurrentManifestMissingError
|
||||||
if !runPointerExists {
|
if errors.As(err, &runPointerMissing) || errors.As(err, &manifestMissing) {
|
||||||
if len(requiredNames) > 0 {
|
if len(requiredNames) > 0 {
|
||||||
return nil, fmt.Errorf("required previous-session artifacts unavailable: remote current run pointer missing: %q", currentRunIDKey)
|
return nil, fmt.Errorf("required previous-session artifacts unavailable: remote %w", err)
|
||||||
|
}
|
||||||
|
result.SkippedMissing = optionalNames
|
||||||
|
return result, nil
|
||||||
}
|
}
|
||||||
result.SkippedMissing = optionalNames
|
return nil, fmt.Errorf("load previous-session current state: %w", err)
|
||||||
return result, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
runIDTemp, err := downloadObjectToTemp(ctx, store, currentRunIDKey, "narratio-previous-run-id-*.txt")
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("download previous-session current run pointer %q: %w", currentRunIDKey, err)
|
|
||||||
}
|
|
||||||
defer func() { _ = os.Remove(runIDTemp) }()
|
|
||||||
|
|
||||||
runIDBytes, err := os.ReadFile(runIDTemp)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("read previous-session current run pointer %q: %w", currentRunIDKey, err)
|
|
||||||
}
|
|
||||||
previousRunID := strings.TrimSpace(string(runIDBytes))
|
|
||||||
if previousRunID == "" {
|
|
||||||
return nil, fmt.Errorf("previous-session current run pointer %q is empty", currentRunIDKey)
|
|
||||||
}
|
|
||||||
result.PreviousRunID = previousRunID
|
|
||||||
|
|
||||||
manifestExists, err := store.Exists(ctx, currentManifestKey)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("check previous-session current manifest %q: %w", currentManifestKey, err)
|
|
||||||
}
|
|
||||||
if !manifestExists {
|
|
||||||
if len(requiredNames) > 0 {
|
|
||||||
return nil, fmt.Errorf("required previous-session artifacts unavailable: remote current manifest missing: %q", currentManifestKey)
|
|
||||||
}
|
|
||||||
result.SkippedMissing = optionalNames
|
|
||||||
return result, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
manifestTemp, err := downloadObjectToTemp(ctx, store, currentManifestKey, "narratio-previous-manifest-*.json")
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("download previous-session current manifest %q: %w", currentManifestKey, err)
|
|
||||||
}
|
|
||||||
defer func() { _ = os.Remove(manifestTemp) }()
|
|
||||||
|
|
||||||
manifestStore := &manifest.LocalStore{}
|
|
||||||
previousManifest, err := manifestStore.Load(ctx, manifestTemp)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("decode downloaded previous-session manifest %q: %w", currentManifestKey, err)
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(previousManifest.SessionID) != previousSessionID {
|
|
||||||
return nil, fmt.Errorf(
|
|
||||||
"previous-session manifest session_id %q does not match configured previous_session_id %q",
|
|
||||||
strings.TrimSpace(previousManifest.SessionID),
|
|
||||||
previousSessionID,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(previousManifest.Campaign) != campaign {
|
|
||||||
return nil, fmt.Errorf(
|
|
||||||
"previous-session manifest campaign %q does not match current campaign %q",
|
|
||||||
strings.TrimSpace(previousManifest.Campaign),
|
|
||||||
campaign,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(previousManifest.RunID) == "" {
|
|
||||||
return nil, fmt.Errorf("previous-session manifest run_id is required")
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(previousManifest.RunID) != previousRunID {
|
|
||||||
return nil, fmt.Errorf(
|
|
||||||
"previous-session current run pointer %q references run %q but current manifest run_id is %q",
|
|
||||||
currentRunIDKey,
|
|
||||||
previousRunID,
|
|
||||||
strings.TrimSpace(previousManifest.RunID),
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
result.PreviousRunID = strings.TrimSpace(current.RunID)
|
||||||
|
currentManifestKey := current.CurrentManifestKey
|
||||||
|
previousManifest := current.Manifest
|
||||||
|
|
||||||
manifestRel, err := relativeToSession(paths, paths.PreviousManifestPath)
|
manifestRel, err := relativeToSession(paths, paths.PreviousManifestPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -188,7 +132,7 @@ func BuildPlan(
|
|||||||
if len(candidates) == 0 {
|
if len(candidates) == 0 {
|
||||||
if requirement.Required {
|
if requirement.Required {
|
||||||
return nil, fmt.Errorf(
|
return nil, fmt.Errorf(
|
||||||
"required previous-session artifact %q is unavailable in previous-session manifest/archive",
|
"required previous-session artifact %q is unavailable in previous-session manifest/published-state",
|
||||||
requirement.Name,
|
requirement.Name,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
@@ -214,7 +158,7 @@ func BuildPlan(
|
|||||||
if selectedRel == "" {
|
if selectedRel == "" {
|
||||||
if requirement.Required {
|
if requirement.Required {
|
||||||
return nil, fmt.Errorf(
|
return nil, fmt.Errorf(
|
||||||
"required previous-session artifact %q object missing from archive candidate keys",
|
"required previous-session artifact %q object missing from published candidate keys",
|
||||||
requirement.Name,
|
requirement.Name,
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
@@ -278,14 +222,18 @@ func artifactRelativePathCandidates(
|
|||||||
) []string {
|
) []string {
|
||||||
candidates := []string{}
|
candidates := []string{}
|
||||||
appendCandidate := func(v string) {
|
appendCandidate := func(v string) {
|
||||||
normalized, err := normalizeArchiveRelativePath(v)
|
normalized, err := pathsafe.NormalizeRelativeDestination(v)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
candidates = append(candidates, normalized)
|
candidates = append(candidates, normalized)
|
||||||
}
|
}
|
||||||
|
|
||||||
sourceID := artifacts.ConfiguredArtifactSourceID(artifactName)
|
sourceDescriptor, err := artifactpolicy.PreviousSessionSourceDescriptorForConfiguredKey(artifactName)
|
||||||
|
sourceID := artifactpolicy.ConfiguredSourceID(artifactName)
|
||||||
|
if err == nil {
|
||||||
|
sourceID = sourceDescriptor.ConfiguredSourceID
|
||||||
|
}
|
||||||
if rel, ok := manifestArtifactRelativePathBySourceID(previousManifest, sourceID); ok {
|
if rel, ok := manifestArtifactRelativePathBySourceID(previousManifest, sourceID); ok {
|
||||||
appendCandidate(rel)
|
appendCandidate(rel)
|
||||||
base := path.Base(rel)
|
base := path.Base(rel)
|
||||||
@@ -354,7 +302,7 @@ func deriveManifestRelativePath(previousManifest *manifest.Manifest, localPath s
|
|||||||
return "", false
|
return "", false
|
||||||
}
|
}
|
||||||
if !filepath.IsAbs(trimmed) {
|
if !filepath.IsAbs(trimmed) {
|
||||||
normalized, err := normalizeArchiveRelativePath(filepath.ToSlash(trimmed))
|
normalized, err := pathsafe.NormalizeRelativeDestination(filepath.ToSlash(trimmed))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", false
|
return "", false
|
||||||
}
|
}
|
||||||
@@ -365,11 +313,7 @@ func deriveManifestRelativePath(previousManifest *manifest.Manifest, localPath s
|
|||||||
if !ok {
|
if !ok {
|
||||||
return "", false
|
return "", false
|
||||||
}
|
}
|
||||||
rel, err := filepath.Rel(sessionRoot, trimmed)
|
normalized, err := pathsafe.SlashRelativeFromRoot(sessionRoot, trimmed)
|
||||||
if err != nil {
|
|
||||||
return "", false
|
|
||||||
}
|
|
||||||
normalized, err := normalizeArchiveRelativePath(filepath.ToSlash(rel))
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", false
|
return "", false
|
||||||
}
|
}
|
||||||
@@ -417,7 +361,7 @@ func manifestPublishedPaths(previousManifest *manifest.Manifest) []string {
|
|||||||
if !ok {
|
if !ok {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
normalized, err := normalizeArchiveRelativePath(asString)
|
normalized, err := pathsafe.NormalizeRelativeDestination(asString)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -426,31 +370,12 @@ func manifestPublishedPaths(previousManifest *manifest.Manifest) []string {
|
|||||||
return dedupeOrderedStrings(out)
|
return dedupeOrderedStrings(out)
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeArchiveRelativePath(rel string) (string, error) {
|
|
||||||
trimmed := strings.TrimSpace(rel)
|
|
||||||
if trimmed == "" {
|
|
||||||
return "", fmt.Errorf("relative path is required")
|
|
||||||
}
|
|
||||||
cleaned := filepath.ToSlash(filepath.Clean(filepath.FromSlash(trimmed)))
|
|
||||||
if cleaned == "." || cleaned == "" {
|
|
||||||
return "", fmt.Errorf("relative path is required")
|
|
||||||
}
|
|
||||||
if filepath.IsAbs(trimmed) || strings.HasPrefix(cleaned, "/") || cleaned == ".." || strings.HasPrefix(cleaned, "../") {
|
|
||||||
return "", fmt.Errorf("path must be a clean relative path")
|
|
||||||
}
|
|
||||||
return cleaned, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func relativeToSession(paths artifacts.SessionPaths, localPath string) (string, error) {
|
func relativeToSession(paths artifacts.SessionPaths, localPath string) (string, error) {
|
||||||
root := filepath.Clean(paths.Root)
|
root := filepath.Clean(paths.Root)
|
||||||
if strings.TrimSpace(root) == "" {
|
if strings.TrimSpace(root) == "" {
|
||||||
return "", fmt.Errorf("session root is required")
|
return "", fmt.Errorf("session root is required")
|
||||||
}
|
}
|
||||||
rel, err := filepath.Rel(root, filepath.Clean(localPath))
|
normalized, err := pathsafe.SlashRelativeFromRoot(root, localPath)
|
||||||
if err != nil {
|
|
||||||
return "", fmt.Errorf("resolve previous-cache relative path: %w", err)
|
|
||||||
}
|
|
||||||
normalized, err := normalizeArchiveRelativePath(filepath.ToSlash(rel))
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", fmt.Errorf("resolve previous-cache relative path: %w", err)
|
return "", fmt.Errorf("resolve previous-cache relative path: %w", err)
|
||||||
}
|
}
|
||||||
@@ -476,20 +401,3 @@ func dedupeOrderedStrings(values []string) []string {
|
|||||||
}
|
}
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func downloadObjectToTemp(ctx context.Context, store storage.ObjectStore, key, pattern string) (string, error) {
|
|
||||||
tmp, err := os.CreateTemp("", pattern)
|
|
||||||
if err != nil {
|
|
||||||
return "", fmt.Errorf("create temp file: %w", err)
|
|
||||||
}
|
|
||||||
path := tmp.Name()
|
|
||||||
if err := tmp.Close(); err != nil {
|
|
||||||
_ = os.Remove(path)
|
|
||||||
return "", fmt.Errorf("close temp file: %w", err)
|
|
||||||
}
|
|
||||||
if err := store.Download(ctx, key, path); err != nil {
|
|
||||||
_ = os.Remove(path)
|
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
return path, nil
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestBuildPlanResolvesPromotedArtifactFromPreviousManifest(t *testing.T) {
|
func TestBuildPlanResolvesPublishedArtifactFromPreviousManifest(t *testing.T) {
|
||||||
cfg, paths := previousCacheTestConfig(t)
|
cfg, paths := previousCacheTestConfig(t)
|
||||||
store := &storage.FakeBackend{}
|
store := &storage.FakeBackend{}
|
||||||
seedPreviousCurrent(t, store, cfg, previousManifestWithOutput(t, cfg, "artifacts/session_recap.md", []string{"artifacts/session_recap.md"}))
|
seedPreviousCurrent(t, store, cfg, previousManifestWithOutput(t, cfg, "artifacts/session_recap.md", []string{"artifacts/session_recap.md"}))
|
||||||
@@ -100,11 +100,71 @@ func TestBuildPlanValidatesPreviousManifestIdentity(t *testing.T) {
|
|||||||
_, err := BuildPlan(context.Background(), cfg, paths, []artifacts.PreviousArtifactRequirement{
|
_, err := BuildPlan(context.Background(), cfg, paths, []artifacts.PreviousArtifactRequirement{
|
||||||
{Name: "session_recap", Required: true},
|
{Name: "session_recap", Required: true},
|
||||||
}, store)
|
}, store)
|
||||||
if err == nil || !strings.Contains(err.Error(), "does not match configured previous_session_id") {
|
if err == nil || !strings.Contains(err.Error(), "does not match expected session_id") {
|
||||||
t.Fatalf("BuildPlan() error = %v, want identity validation error", err)
|
t.Fatalf("BuildPlan() error = %v, want identity validation error", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestBuildPlanMissingCurrentRunPointerSkipsOptionalRequirements(t *testing.T) {
|
||||||
|
cfg, paths := previousCacheTestConfig(t)
|
||||||
|
store := &storage.FakeBackend{}
|
||||||
|
|
||||||
|
plan, err := BuildPlan(context.Background(), cfg, paths, []artifacts.PreviousArtifactRequirement{
|
||||||
|
{Name: "session_recap", Required: false},
|
||||||
|
}, store)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("BuildPlan() error = %v", err)
|
||||||
|
}
|
||||||
|
if strings.Join(plan.SkippedMissing, ",") != "session_recap" {
|
||||||
|
t.Fatalf("SkippedMissing = %#v, want session_recap", plan.SkippedMissing)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildPlanMissingCurrentRunPointerFailsRequiredRequirements(t *testing.T) {
|
||||||
|
cfg, paths := previousCacheTestConfig(t)
|
||||||
|
store := &storage.FakeBackend{}
|
||||||
|
|
||||||
|
_, err := BuildPlan(context.Background(), cfg, paths, []artifacts.PreviousArtifactRequirement{
|
||||||
|
{Name: "session_recap", Required: true},
|
||||||
|
}, store)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), `required previous-session artifacts unavailable: remote current run pointer missing`) {
|
||||||
|
t.Fatalf("BuildPlan() error = %v, want required missing run pointer error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildPlanMissingCurrentManifestSkipsOptionalRequirements(t *testing.T) {
|
||||||
|
cfg, paths := previousCacheTestConfig(t)
|
||||||
|
store := &storage.FakeBackend{}
|
||||||
|
previousPrefix := artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
||||||
|
_, runIDKey := artifacts.ResolveCurrentStateKeys(previousPrefix)
|
||||||
|
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("previous-run\n")})
|
||||||
|
|
||||||
|
plan, err := BuildPlan(context.Background(), cfg, paths, []artifacts.PreviousArtifactRequirement{
|
||||||
|
{Name: "session_recap", Required: false},
|
||||||
|
}, store)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("BuildPlan() error = %v", err)
|
||||||
|
}
|
||||||
|
if strings.Join(plan.SkippedMissing, ",") != "session_recap" {
|
||||||
|
t.Fatalf("SkippedMissing = %#v, want session_recap", plan.SkippedMissing)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildPlanMissingCurrentManifestFailsRequiredRequirements(t *testing.T) {
|
||||||
|
cfg, paths := previousCacheTestConfig(t)
|
||||||
|
store := &storage.FakeBackend{}
|
||||||
|
previousPrefix := artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
||||||
|
_, runIDKey := artifacts.ResolveCurrentStateKeys(previousPrefix)
|
||||||
|
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("previous-run\n")})
|
||||||
|
|
||||||
|
_, err := BuildPlan(context.Background(), cfg, paths, []artifacts.PreviousArtifactRequirement{
|
||||||
|
{Name: "session_recap", Required: true},
|
||||||
|
}, store)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), `required previous-session artifacts unavailable: remote current manifest missing`) {
|
||||||
|
t.Fatalf("BuildPlan() error = %v, want required missing manifest error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func previousCacheTestConfig(t *testing.T) (*config.Config, artifacts.SessionPaths) {
|
func previousCacheTestConfig(t *testing.T) (*config.Config, artifacts.SessionPaths) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
@@ -134,7 +194,7 @@ func previousCacheTestConfig(t *testing.T) (*config.Config, artifacts.SessionPat
|
|||||||
func seedPreviousCurrent(t *testing.T, store *storage.FakeBackend, cfg *config.Config, m *manifest.Manifest) {
|
func seedPreviousCurrent(t *testing.T, store *storage.FakeBackend, cfg *config.Config, m *manifest.Manifest) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
previousPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
previousPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
||||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(previousPrefix)
|
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(previousPrefix)
|
||||||
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("previous-run\n")})
|
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("previous-run\n")})
|
||||||
data, err := marshalManifestForPreviousCacheTest(m)
|
data, err := marshalManifestForPreviousCacheTest(m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -143,7 +203,7 @@ func seedPreviousCurrent(t *testing.T, store *storage.FakeBackend, cfg *config.C
|
|||||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: data})
|
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: data})
|
||||||
}
|
}
|
||||||
|
|
||||||
func previousManifestWithOutput(t *testing.T, cfg *config.Config, rel string, promoted []string) *manifest.Manifest {
|
func previousManifestWithOutput(t *testing.T, cfg *config.Config, rel string, publishedPaths []string) *manifest.Manifest {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
m := manifest.New(cfg.Session.PreviousSessionID, time.Date(2026, 4, 26, 10, 0, 0, 0, time.UTC))
|
m := manifest.New(cfg.Session.PreviousSessionID, time.Date(2026, 4, 26, 10, 0, 0, 0, time.UTC))
|
||||||
m.Campaign = cfg.Session.Campaign
|
m.Campaign = cfg.Session.Campaign
|
||||||
@@ -154,11 +214,11 @@ func previousManifestWithOutput(t *testing.T, cfg *config.Config, rel string, pr
|
|||||||
{SourceID: "narratio.artifact.session_recap", LocalPath: filepath.Join(filepath.Dir(filepath.Dir(m.LocalWorkDir)), filepath.FromSlash(rel))},
|
{SourceID: "narratio.artifact.session_recap", LocalPath: filepath.Join(filepath.Dir(filepath.Dir(m.LocalWorkDir)), filepath.FromSlash(rel))},
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
if promoted != nil {
|
if publishedPaths != nil {
|
||||||
if m.Stages["publish"] == nil {
|
if m.Stages["publish"] == nil {
|
||||||
m.MarkStageSucceeded("publish", time.Date(2026, 4, 26, 10, 2, 0, 0, time.UTC), nil)
|
m.MarkStageSucceeded("publish", time.Date(2026, 4, 26, 10, 2, 0, 0, time.UTC), nil)
|
||||||
}
|
}
|
||||||
m.Stages["publish"].Metadata = map[string]any{"published_paths": promoted}
|
m.Stages["publish"].Metadata = map[string]any{"published_paths": publishedPaths}
|
||||||
}
|
}
|
||||||
return m
|
return m
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/scriptorium"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/scriptorium"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
@@ -549,12 +550,8 @@ func extractPlanNames(plans []analyzeArtifactExecutionPlan) []string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func configuredArtifactNameFromSourceID(sourceID string) string {
|
func configuredArtifactNameFromSourceID(sourceID string) string {
|
||||||
trimmed := strings.TrimSpace(sourceID)
|
name, _ := artifactpolicy.ParseConfiguredSource(sourceID)
|
||||||
const prefix = "narratio.artifact."
|
return name
|
||||||
if !strings.HasPrefix(trimmed, prefix) {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
return strings.TrimPrefix(trimmed, prefix)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func discoverProcessedTranscript(m *manifest.Manifest, paths artifacts.SessionPaths) (string, string, error) {
|
func discoverProcessedTranscript(m *manifest.Manifest, paths artifacts.SessionPaths) (string, string, error) {
|
||||||
@@ -631,7 +628,11 @@ func resolveScriptoriumInput(
|
|||||||
runtimeCatalog *artifacts.ArtifactCatalog,
|
runtimeCatalog *artifacts.ArtifactCatalog,
|
||||||
) (string, bool, *artifacts.ResolvedSessionArtifact, error) {
|
) (string, bool, *artifacts.ResolvedSessionArtifact, error) {
|
||||||
source := strings.TrimSpace(inputCfg.Source)
|
source := strings.TrimSpace(inputCfg.Source)
|
||||||
if artifacts.IsPreviousSessionArtifactSource(source) {
|
descriptor, describeErr := artifactpolicy.DescribeScriptoriumInputSource(source)
|
||||||
|
if describeErr != nil {
|
||||||
|
return "", false, nil, describeErr
|
||||||
|
}
|
||||||
|
if descriptor.Source.Kind == artifactpolicy.SourceKindPreviousArtifact {
|
||||||
resolved, err := artifacts.ResolvePreviousSessionArtifactWithCatalog(paths, m, source, runtimeCatalog)
|
resolved, err := artifacts.ResolvePreviousSessionArtifactWithCatalog(paths, m, source, runtimeCatalog)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
copy := resolved
|
copy := resolved
|
||||||
@@ -656,17 +657,13 @@ func resolveScriptoriumInput(
|
|||||||
return resolved.Path, true, ©, nil
|
return resolved.Path, true, ©, nil
|
||||||
}
|
}
|
||||||
if errors.Is(err, artifacts.ErrSessionArtifactNotFound) {
|
if errors.Is(err, artifacts.ErrSessionArtifactNotFound) {
|
||||||
if artifacts.IsConfiguredArtifactSource(source) {
|
if descriptor.Source.Kind == artifactpolicy.SourceKindConfiguredArtifact {
|
||||||
if inputCfg.Required {
|
if inputCfg.Required {
|
||||||
return "", false, nil, fmt.Errorf("configured artifact source %q is unavailable", source)
|
return "", false, nil, fmt.Errorf("configured artifact source %q is unavailable", source)
|
||||||
}
|
}
|
||||||
return "", false, nil, nil
|
return "", false, nil, nil
|
||||||
}
|
}
|
||||||
normalized, normalizeErr := artifacts.NormalizeSessionArtifactSource(source)
|
switch descriptor.Source.ID {
|
||||||
if normalizeErr != nil {
|
|
||||||
return "", false, nil, normalizeErr
|
|
||||||
}
|
|
||||||
switch normalized {
|
|
||||||
case artifacts.ArtifactTranscriptPolished:
|
case artifacts.ArtifactTranscriptPolished:
|
||||||
return "", false, nil, nil
|
return "", false, nil, nil
|
||||||
case artifacts.ArtifactTranscriptFinal:
|
case artifacts.ArtifactTranscriptFinal:
|
||||||
|
|||||||
@@ -255,7 +255,7 @@ func TestAnalyzeOmitsOptionalCanonicalPreviousRecapWhenUnavailable(t *testing.T)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestAnalyzeUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
func TestAnalyzeUsesRunLocalPathsAndMaterializesCanonical(t *testing.T) {
|
||||||
env, m, fake := setupAnalyzeEnv(t)
|
env, m, fake := setupAnalyzeEnv(t)
|
||||||
env.Config.Session.Campaign = "sample-campaign"
|
env.Config.Session.Campaign = "sample-campaign"
|
||||||
m.Campaign = "sample-campaign"
|
m.Campaign = "sample-campaign"
|
||||||
|
|||||||
@@ -156,7 +156,7 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
|
|||||||
SessionID: sessionID,
|
SessionID: sessionID,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("merge: promote merged transcript: %w", err)
|
return nil, fmt.Errorf("merge: materialize canonical base transcript: %w", err)
|
||||||
}
|
}
|
||||||
outputs := []artifacts.Ref{materializedMerged}
|
outputs := []artifacts.Ref{materializedMerged}
|
||||||
if reportEnabled {
|
if reportEnabled {
|
||||||
@@ -166,7 +166,7 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
|
|||||||
SessionID: sessionID,
|
SessionID: sessionID,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("merge: promote report: %w", err)
|
return nil, fmt.Errorf("merge: materialize canonical report: %w", err)
|
||||||
}
|
}
|
||||||
outputs = append(outputs, materializedReport)
|
outputs = append(outputs, materializedReport)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -289,7 +289,7 @@ func TestMergeStageFailsWhenNormalizedOutputInvalid(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestMergeStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
func TestMergeStageUsesRunLocalPathsAndMaterializesCanonical(t *testing.T) {
|
||||||
env, m := setupMergeEnv(t)
|
env, m := setupMergeEnv(t)
|
||||||
env.Config.Session.Campaign = "sample-campaign"
|
env.Config.Session.Campaign = "sample-campaign"
|
||||||
m.Campaign = "sample-campaign"
|
m.Campaign = "sample-campaign"
|
||||||
|
|||||||
@@ -139,7 +139,7 @@ func (normalizeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (
|
|||||||
SessionID: sessionID,
|
SessionID: sessionID,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("normalize: promote normalized transcript: %w", err)
|
return nil, fmt.Errorf("normalize: materialize canonical final transcript: %w", err)
|
||||||
}
|
}
|
||||||
outputs := []artifacts.Ref{materializedNormalized}
|
outputs := []artifacts.Ref{materializedNormalized}
|
||||||
if reportEnabled {
|
if reportEnabled {
|
||||||
@@ -149,7 +149,7 @@ func (normalizeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (
|
|||||||
SessionID: sessionID,
|
SessionID: sessionID,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("normalize: promote report: %w", err)
|
return nil, fmt.Errorf("normalize: materialize canonical report: %w", err)
|
||||||
}
|
}
|
||||||
outputs = append(outputs, materializedReport)
|
outputs = append(outputs, materializedReport)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -211,7 +211,7 @@ func TestNormalizeStageReportEnabledFailsWhenReportMissing(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestNormalizeStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
func TestNormalizeStageUsesRunLocalPathsAndMaterializesCanonical(t *testing.T) {
|
||||||
env, m, ser := setupNormalizeEnv(t)
|
env, m, ser := setupNormalizeEnv(t)
|
||||||
env.Config.Session.Campaign = "sample-campaign"
|
env.Config.Session.Campaign = "sample-campaign"
|
||||||
m.Campaign = "sample-campaign"
|
m.Campaign = "sample-campaign"
|
||||||
|
|||||||
@@ -75,7 +75,7 @@ func All() []Stage {
|
|||||||
normalizeStage{},
|
normalizeStage{},
|
||||||
trimStage{},
|
trimStage{},
|
||||||
analyzeStage{},
|
analyzeStage{},
|
||||||
archiveStage{},
|
publishStage{},
|
||||||
placeholderStage{name: "notify"},
|
placeholderStage{name: "notify"},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -96,7 +96,6 @@ func TestStagesReturnExpectedMetadata(t *testing.T) {
|
|||||||
Seriatim: sf,
|
Seriatim: sf,
|
||||||
Audita: af,
|
Audita: af,
|
||||||
Scriptorium: sc,
|
Scriptorium: sc,
|
||||||
Storage: st,
|
|
||||||
ObjectStore: st,
|
ObjectStore: st,
|
||||||
Notifier: nf,
|
Notifier: nf,
|
||||||
}
|
}
|
||||||
@@ -197,10 +196,10 @@ func TestStagesReturnExpectedMetadata(t *testing.T) {
|
|||||||
}
|
}
|
||||||
if s.Name() == "publish" {
|
if s.Name() == "publish" {
|
||||||
if result.Metadata["stage"] != "publish" {
|
if result.Metadata["stage"] != "publish" {
|
||||||
t.Fatalf("archive metadata = %#v, want stage=publish", result.Metadata)
|
t.Fatalf("publish metadata = %#v, want stage=publish", result.Metadata)
|
||||||
}
|
}
|
||||||
if result.Metadata["uploaded"] != true {
|
if result.Metadata["uploaded"] != true {
|
||||||
t.Fatalf("archive metadata = %#v, want uploaded=true", result.Metadata)
|
t.Fatalf("publish metadata = %#v, want uploaded=true", result.Metadata)
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -221,11 +220,8 @@ func TestStagesReturnExpectedMetadata(t *testing.T) {
|
|||||||
if len(sc.RunRequests) != 0 {
|
if len(sc.RunRequests) != 0 {
|
||||||
t.Fatalf("scriptorium run calls = %d, want 0 when scriptorium config is absent", len(sc.RunRequests))
|
t.Fatalf("scriptorium run calls = %d, want 0 when scriptorium config is absent", len(sc.RunRequests))
|
||||||
}
|
}
|
||||||
if len(st.Requests) != 0 {
|
|
||||||
t.Fatalf("storage archive calls = %d, want 0", len(st.Requests))
|
|
||||||
}
|
|
||||||
if _, ok := st.Objects["dnd/campaigns/sample-campaign/sessions/2026-05-03/runs/"+m.RunID+"/manifest.json"]; !ok {
|
if _, ok := st.Objects["dnd/campaigns/sample-campaign/sessions/2026-05-03/runs/"+m.RunID+"/manifest.json"]; !ok {
|
||||||
t.Fatalf("archive upload missing manifest key in fake object store")
|
t.Fatalf("publish upload missing manifest key in fake object store")
|
||||||
}
|
}
|
||||||
if len(nf.Requests) != 1 {
|
if len(nf.Requests) != 1 {
|
||||||
t.Fatalf("notify calls = %d, want 1", len(nf.Requests))
|
t.Fatalf("notify calls = %d, want 1", len(nf.Requests))
|
||||||
|
|||||||
@@ -155,7 +155,7 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
|||||||
SessionID: sessionID,
|
SessionID: sessionID,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("polish: promote processed transcript: %w", err)
|
return nil, fmt.Errorf("polish: materialize canonical polished transcript: %w", err)
|
||||||
}
|
}
|
||||||
outputs := []artifacts.Ref{materializedProcessed}
|
outputs := []artifacts.Ref{materializedProcessed}
|
||||||
if reportEnabled {
|
if reportEnabled {
|
||||||
@@ -165,7 +165,7 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
|||||||
SessionID: sessionID,
|
SessionID: sessionID,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("polish: promote report: %w", err)
|
return nil, fmt.Errorf("polish: materialize canonical report: %w", err)
|
||||||
}
|
}
|
||||||
outputs = append(outputs, materializedReport)
|
outputs = append(outputs, materializedReport)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -238,7 +238,7 @@ func TestPolishStageFailsWhenReportInvalid(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPolishStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
func TestPolishStageUsesRunLocalPathsAndMaterializesCanonical(t *testing.T) {
|
||||||
env, m := setupPolishEnv(t)
|
env, m := setupPolishEnv(t)
|
||||||
env.Config.Session.Campaign = "sample-campaign"
|
env.Config.Session.Campaign = "sample-campaign"
|
||||||
m.Campaign = "sample-campaign"
|
m.Campaign = "sample-campaign"
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import (
|
|||||||
"sort"
|
"sort"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/fileops"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/previouscache"
|
"gitea.maximumdirect.net/eric/narratio/internal/previouscache"
|
||||||
)
|
)
|
||||||
@@ -54,8 +55,35 @@ func hydratePreviousSessionArtifacts(
|
|||||||
if err := os.MkdirAll(filepath.Dir(record.LocalPath), 0o755); err != nil {
|
if err := os.MkdirAll(filepath.Dir(record.LocalPath), 0o755); err != nil {
|
||||||
return nil, fmt.Errorf("create previous-session path directory for %q: %w", record.LocalPath, err)
|
return nil, fmt.Errorf("create previous-session path directory for %q: %w", record.LocalPath, err)
|
||||||
}
|
}
|
||||||
if err := env.ObjectStore.Download(ctx, record.RemoteKey, record.LocalPath); err != nil {
|
|
||||||
return nil, fmt.Errorf("download previous-session object %q to %q: %w", record.RemoteKey, record.LocalPath, err)
|
base := filepath.Base(record.LocalPath)
|
||||||
|
tmp, err := os.CreateTemp(filepath.Dir(record.LocalPath), "."+base+".prepare-previous-*.tmp")
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("create previous-session temp file for %q: %w", record.LocalPath, err)
|
||||||
|
}
|
||||||
|
tmpPath := tmp.Name()
|
||||||
|
if err := tmp.Close(); err != nil {
|
||||||
|
_ = os.Remove(tmpPath)
|
||||||
|
return nil, fmt.Errorf("close previous-session temp file for %q: %w", record.LocalPath, err)
|
||||||
|
}
|
||||||
|
if err := func() error {
|
||||||
|
removeTmp := true
|
||||||
|
defer func() {
|
||||||
|
if removeTmp {
|
||||||
|
_ = os.Remove(tmpPath)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
if err := env.ObjectStore.Download(ctx, record.RemoteKey, tmpPath); err != nil {
|
||||||
|
return fmt.Errorf("download previous-session object %q to temp file: %w", record.RemoteKey, err)
|
||||||
|
}
|
||||||
|
if err := fileops.InstallDownloadedTempFile(tmpPath, record.LocalPath, 0o644); err != nil {
|
||||||
|
return fmt.Errorf("install previous-session object %q at %q: %w", record.RemoteKey, record.LocalPath, err)
|
||||||
|
}
|
||||||
|
removeTmp = false
|
||||||
|
return nil
|
||||||
|
}(); err != nil {
|
||||||
|
return nil, err
|
||||||
}
|
}
|
||||||
if record.Kind == preparePreviousInputKindArtifact {
|
if record.Kind == preparePreviousInputKindArtifact {
|
||||||
if err := requireNonEmptyFile(record.LocalPath, "previous-session artifact "+record.RequirementName); err != nil {
|
if err := requireNonEmptyFile(record.LocalPath, "previous-session artifact "+record.RequirementName); err != nil {
|
||||||
|
|||||||
@@ -193,7 +193,7 @@ func TestHydratePreviousSessionArtifactsDoesNotUseLocalPreviousWorkspaceState(t
|
|||||||
writeFile(t, filepath.Join(seed.PreviousSessionRoot, "artifacts", "session_recap.md"), "# local stale recap\n")
|
writeFile(t, filepath.Join(seed.PreviousSessionRoot, "artifacts", "session_recap.md"), "# local stale recap\n")
|
||||||
|
|
||||||
_, err := hydratePreviousSessionArtifacts(context.Background(), env, sessionPaths, requirements)
|
_, err := hydratePreviousSessionArtifacts(context.Background(), env, sessionPaths, requirements)
|
||||||
if err == nil || !strings.Contains(err.Error(), "object missing from archive") {
|
if err == nil || !strings.Contains(err.Error(), "object missing from published candidate keys") {
|
||||||
t.Fatalf("error = %v, want remote-object-missing failure", err)
|
t.Fatalf("error = %v, want remote-object-missing failure", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -273,7 +273,7 @@ func seedPreviousCurrentState(
|
|||||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||||
rootPrefix := strings.TrimSpace(env.Config.Pipeline.Storage.S3.RootPrefix)
|
rootPrefix := strings.TrimSpace(env.Config.Pipeline.Storage.S3.RootPrefix)
|
||||||
previousSessionPrefix := artifacts.S3SessionPrefix(rootPrefix, campaign, previousSessionID)
|
previousSessionPrefix := artifacts.S3SessionPrefix(rootPrefix, campaign, previousSessionID)
|
||||||
manifestKey, runPointerKey := artifacts.ResolveArchiveCurrentStateKeys(previousSessionPrefix)
|
manifestKey, runPointerKey := artifacts.ResolveCurrentStateKeys(previousSessionPrefix)
|
||||||
|
|
||||||
previousRunID := "20260510T010203Z-a1b2c3d4"
|
previousRunID := "20260510T010203Z-a1b2c3d4"
|
||||||
previousSessionRoot := filepath.Join(t.TempDir(), "work", campaign, previousSessionID)
|
previousSessionRoot := filepath.Join(t.TempDir(), "work", campaign, previousSessionID)
|
||||||
|
|||||||
@@ -13,18 +13,19 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
type archiveStage struct{}
|
type publishStage struct{}
|
||||||
type archiveUploadFile struct {
|
type publishUploadFile struct {
|
||||||
RelativePath string
|
RelativePath string
|
||||||
LocalPath string
|
LocalPath string
|
||||||
}
|
}
|
||||||
|
|
||||||
var archivePrerequisiteStages = []string{
|
var publishPrerequisiteStages = []string{
|
||||||
"prepare",
|
"prepare",
|
||||||
"transcribe",
|
"transcribe",
|
||||||
"merge",
|
"merge",
|
||||||
@@ -34,9 +35,9 @@ var archivePrerequisiteStages = []string{
|
|||||||
"analyze",
|
"analyze",
|
||||||
}
|
}
|
||||||
|
|
||||||
func (archiveStage) Name() string { return "publish" }
|
func (publishStage) Name() string { return "publish" }
|
||||||
|
|
||||||
func (archiveStage) Declares() IODecl {
|
func (publishStage) Declares() IODecl {
|
||||||
return IODecl{
|
return IODecl{
|
||||||
Inputs: []artifacts.Ref{
|
Inputs: []artifacts.Ref{
|
||||||
{Kind: "manifest", Category: "input", RelativePath: "manifest.json"},
|
{Kind: "manifest", Category: "input", RelativePath: "manifest.json"},
|
||||||
@@ -44,23 +45,23 @@ func (archiveStage) Declares() IODecl {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*StageResult, error) {
|
func (publishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*StageResult, error) {
|
||||||
if env == nil || env.Config == nil || env.Config.Pipeline == nil || env.Config.Session == nil {
|
if env == nil || env.Config == nil || env.Config.Pipeline == nil || env.Config.Session == nil {
|
||||||
return nil, fmt.Errorf("publish: resolved config must include pipeline and session")
|
return nil, fmt.Errorf("publish: resolved config must include pipeline and session")
|
||||||
}
|
}
|
||||||
|
|
||||||
if archiveDisabled(env) {
|
if publishDisabled(env) {
|
||||||
return &StageResult{
|
return &StageResult{
|
||||||
Metadata: map[string]any{
|
Metadata: map[string]any{
|
||||||
"stage": "publish",
|
"stage": "publish",
|
||||||
"skipped": true,
|
"skipped": true,
|
||||||
"archive_enabled": false,
|
"publish_enabled": false,
|
||||||
"audio_upload_skipped": true,
|
"audio_upload_skipped": true,
|
||||||
"current_pointer_written": false,
|
"current_pointer_written": false,
|
||||||
},
|
},
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
if archiveRunUploadDisabled(env) {
|
if publishRunUploadDisabled(env) {
|
||||||
return &StageResult{
|
return &StageResult{
|
||||||
Metadata: map[string]any{
|
Metadata: map[string]any{
|
||||||
"stage": "publish",
|
"stage": "publish",
|
||||||
@@ -79,7 +80,7 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
|||||||
return nil, fmt.Errorf("publish: remote object store backend is required when publish run upload is enabled")
|
return nil, fmt.Errorf("publish: remote object store backend is required when publish run upload is enabled")
|
||||||
}
|
}
|
||||||
|
|
||||||
runRoot, err := resolveArchiveRunRoot(env, m)
|
runRoot, err := resolvePublishRunRoot(env, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("publish: resolve run root: %w", err)
|
return nil, fmt.Errorf("publish: resolve run root: %w", err)
|
||||||
}
|
}
|
||||||
@@ -91,15 +92,15 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
|||||||
return nil, fmt.Errorf("publish: run root %q is not a directory", runRoot)
|
return nil, fmt.Errorf("publish: run root %q is not a directory", runRoot)
|
||||||
}
|
}
|
||||||
|
|
||||||
runPrefix, err := artifacts.ResolveArchiveRunPrefix(env.Config, m)
|
runPrefix, err := artifacts.ResolvePublishRunPrefix(env.Config, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("publish: resolve s3 run prefix: %w", err)
|
return nil, fmt.Errorf("publish: resolve s3 run prefix: %w", err)
|
||||||
}
|
}
|
||||||
sessionPrefix, err := artifacts.ResolveArchiveSessionPrefix(env.Config, m)
|
sessionPrefix, err := artifacts.ResolvePublishSessionPrefix(env.Config, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("publish: resolve s3 session prefix: %w", err)
|
return nil, fmt.Errorf("publish: resolve s3 session prefix: %w", err)
|
||||||
}
|
}
|
||||||
bucket := artifacts.ResolveArchiveBucket(env.Config, m)
|
bucket := artifacts.ResolvePublishBucket(env.Config, m)
|
||||||
if bucket == "" {
|
if bucket == "" {
|
||||||
return nil, fmt.Errorf("publish: resolve s3 bucket: bucket is required")
|
return nil, fmt.Errorf("publish: resolve s3 bucket: bucket is required")
|
||||||
}
|
}
|
||||||
@@ -108,21 +109,21 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
|||||||
return nil, fmt.Errorf("publish: run id is required")
|
return nil, fmt.Errorf("publish: run id is required")
|
||||||
}
|
}
|
||||||
|
|
||||||
manifestSource, err := resolveArchiveRunManifestSource(runRoot)
|
manifestSource, err := resolvePublishRunManifestSource(runRoot)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("publish: resolve run manifest source: %w", err)
|
return nil, fmt.Errorf("publish: resolve run manifest source: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
runFiles, err := collectArchiveRunFiles(runRoot, manifestSource)
|
runFiles, err := collectPublishRunFiles(runRoot, manifestSource)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("publish: collect run files: %w", err)
|
return nil, fmt.Errorf("publish: collect run files: %w", err)
|
||||||
}
|
}
|
||||||
sessionPaths := archiveSessionPaths(env, m)
|
sessionPaths := publishSessionPaths(env, m)
|
||||||
previousFiles, err := collectArchivePreviousFiles(sessionPaths.PreviousDir)
|
previousFiles, err := collectPublishPreviousFiles(sessionPaths.PreviousDir)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("publish: collect previous files: %w", err)
|
return nil, fmt.Errorf("publish: collect previous files: %w", err)
|
||||||
}
|
}
|
||||||
runtimeCatalog, err := buildArchiveRuntimeArtifactCatalog(sessionPaths, env.Config.Pipeline.Scriptorium)
|
runtimeCatalog, err := buildPublishRuntimeArtifactCatalog(sessionPaths, env.Config.Pipeline.Scriptorium)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("publish: build runtime artifact catalog: %w", err)
|
return nil, fmt.Errorf("publish: build runtime artifact catalog: %w", err)
|
||||||
}
|
}
|
||||||
@@ -148,12 +149,12 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
|||||||
}
|
}
|
||||||
|
|
||||||
publishedUploaded := make([]string, 0, len(publishOutputs))
|
publishedUploaded := make([]string, 0, len(publishOutputs))
|
||||||
for _, promotion := range publishOutputs {
|
for _, publishedOutput := range publishOutputs {
|
||||||
key := artifacts.S3PublishedOutputKey(sessionPrefix, promotion.Dest)
|
key := artifacts.S3PublishedOutputKey(sessionPrefix, publishedOutput.Dest)
|
||||||
if _, err := env.ObjectStore.Upload(ctx, promotion.LocalPath, key, storage.UploadOptions{}); err != nil {
|
if _, err := env.ObjectStore.Upload(ctx, publishedOutput.LocalPath, key, storage.UploadOptions{}); err != nil {
|
||||||
return nil, fmt.Errorf("publish: upload published output source %q to %q: %w", promotion.Source, key, err)
|
return nil, fmt.Errorf("publish: upload published output source %q to %q: %w", publishedOutput.Source, key, err)
|
||||||
}
|
}
|
||||||
publishedUploaded = append(publishedUploaded, promotion.Dest)
|
publishedUploaded = append(publishedUploaded, publishedOutput.Dest)
|
||||||
}
|
}
|
||||||
|
|
||||||
previousUploaded := make([]string, 0, len(previousFiles))
|
previousUploaded := make([]string, 0, len(previousFiles))
|
||||||
@@ -165,8 +166,8 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
|||||||
previousUploaded = append(previousUploaded, file.RelativePath)
|
previousUploaded = append(previousUploaded, file.RelativePath)
|
||||||
}
|
}
|
||||||
|
|
||||||
currentManifestKey, currentRunPointerKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
currentManifestKey, currentRunPointerKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
manifestTempPath, err := writeCurrentManifestSnapshot(m, archiveMetadataPreview(
|
manifestTempPath, err := writeCurrentManifestSnapshot(m, publishMetadataPreview(
|
||||||
bucket,
|
bucket,
|
||||||
runPrefix,
|
runPrefix,
|
||||||
sessionPrefix,
|
sessionPrefix,
|
||||||
@@ -249,7 +250,7 @@ type publishSkippedUnselectedOutput struct {
|
|||||||
Required bool
|
Required bool
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveDisabled(env *Env) bool {
|
func publishDisabled(env *Env) bool {
|
||||||
cfg := env.Config.Pipeline.Publish
|
cfg := env.Config.Pipeline.Publish
|
||||||
if cfg == nil {
|
if cfg == nil {
|
||||||
return true
|
return true
|
||||||
@@ -257,7 +258,7 @@ func archiveDisabled(env *Env) bool {
|
|||||||
return cfg.Enabled != nil && !*cfg.Enabled
|
return cfg.Enabled != nil && !*cfg.Enabled
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveRunUploadDisabled(env *Env) bool {
|
func publishRunUploadDisabled(env *Env) bool {
|
||||||
cfg := env.Config.Pipeline.Publish
|
cfg := env.Config.Pipeline.Publish
|
||||||
if cfg == nil {
|
if cfg == nil {
|
||||||
return true
|
return true
|
||||||
@@ -269,7 +270,7 @@ func validatePublishPrerequisites(m *manifest.Manifest) error {
|
|||||||
if m == nil {
|
if m == nil {
|
||||||
return fmt.Errorf("manifest is required")
|
return fmt.Errorf("manifest is required")
|
||||||
}
|
}
|
||||||
for _, stageName := range archivePrerequisiteStages {
|
for _, stageName := range publishPrerequisiteStages {
|
||||||
sr := m.Stages[stageName]
|
sr := m.Stages[stageName]
|
||||||
if sr == nil {
|
if sr == nil {
|
||||||
return fmt.Errorf("prerequisite stage %q has not succeeded", stageName)
|
return fmt.Errorf("prerequisite stage %q has not succeeded", stageName)
|
||||||
@@ -281,7 +282,7 @@ func validatePublishPrerequisites(m *manifest.Manifest) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func resolveArchiveRunRoot(env *Env, m *manifest.Manifest) (string, error) {
|
func resolvePublishRunRoot(env *Env, m *manifest.Manifest) (string, error) {
|
||||||
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
||||||
if sessionID == "" && m != nil {
|
if sessionID == "" && m != nil {
|
||||||
sessionID = strings.TrimSpace(m.SessionID)
|
sessionID = strings.TrimSpace(m.SessionID)
|
||||||
@@ -312,7 +313,7 @@ func resolveArchiveRunRoot(env *Env, m *manifest.Manifest) (string, error) {
|
|||||||
return canonical, nil
|
return canonical, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func resolveArchiveSessionRoot(env *Env, m *manifest.Manifest) (string, error) {
|
func resolvePublishSessionRoot(env *Env, m *manifest.Manifest) (string, error) {
|
||||||
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
||||||
if sessionID == "" && m != nil {
|
if sessionID == "" && m != nil {
|
||||||
sessionID = strings.TrimSpace(m.SessionID)
|
sessionID = strings.TrimSpace(m.SessionID)
|
||||||
@@ -330,7 +331,7 @@ func resolveArchiveSessionRoot(env *Env, m *manifest.Manifest) (string, error) {
|
|||||||
return filepath.Clean(artifacts.SessionWorkDirForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID)), nil
|
return filepath.Clean(artifacts.SessionWorkDirForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID)), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveSessionPaths(env *Env, m *manifest.Manifest) artifacts.SessionPaths {
|
func publishSessionPaths(env *Env, m *manifest.Manifest) artifacts.SessionPaths {
|
||||||
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
||||||
if sessionID == "" && m != nil {
|
if sessionID == "" && m != nil {
|
||||||
sessionID = strings.TrimSpace(m.SessionID)
|
sessionID = strings.TrimSpace(m.SessionID)
|
||||||
@@ -356,17 +357,18 @@ func resolvePublishOutputs(
|
|||||||
skippedOptionalOutputs := make([]string, 0)
|
skippedOptionalOutputs := make([]string, 0)
|
||||||
skippedUnselectedOutputs := make([]publishSkippedUnselectedOutput, 0)
|
skippedUnselectedOutputs := make([]publishSkippedUnselectedOutput, 0)
|
||||||
lockedOutputs := make([]publishLockedOutput, 0)
|
lockedOutputs := make([]publishLockedOutput, 0)
|
||||||
lockSet := archiveLockSet(locks)
|
lockSet := publishLockSet(locks)
|
||||||
selectedSet := archiveSelectedArtifactSet(selectedArtifactKeys)
|
selectedSet := publishSelectedArtifactSet(selectedArtifactKeys)
|
||||||
|
configuredOutputs := configuredOutputPathMapFromCatalog(catalog)
|
||||||
for _, rule := range rules {
|
for _, rule := range rules {
|
||||||
source := strings.TrimSpace(rule.Source)
|
source := strings.TrimSpace(rule.Source)
|
||||||
required := rule.Required == nil || *rule.Required
|
required := rule.Required == nil || *rule.Required
|
||||||
dest, err := resolvePublishOutputDest(rule, catalog)
|
dest, err := resolvePublishOutputDest(rule, configuredOutputs)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, nil, nil, nil, fmt.Errorf("source %q: %w", source, err)
|
return nil, nil, nil, nil, fmt.Errorf("source %q: %w", source, err)
|
||||||
}
|
}
|
||||||
if len(selectedSet) > 0 {
|
if len(selectedSet) > 0 {
|
||||||
if key, ok := artifacts.ConfiguredArtifactName(source); ok {
|
if key, ok := artifactpolicy.ParseConfiguredSource(source); ok {
|
||||||
if _, selected := selectedSet[key]; !selected {
|
if _, selected := selectedSet[key]; !selected {
|
||||||
skippedUnselectedOutputs = append(skippedUnselectedOutputs, publishSkippedUnselectedOutput{
|
skippedUnselectedOutputs = append(skippedUnselectedOutputs, publishSkippedUnselectedOutput{
|
||||||
Source: source,
|
Source: source,
|
||||||
@@ -422,7 +424,7 @@ func resolvePublishOutputs(
|
|||||||
return out, skippedOptionalOutputs, skippedUnselectedOutputs, lockedOutputs, nil
|
return out, skippedOptionalOutputs, skippedUnselectedOutputs, lockedOutputs, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveSelectedArtifactSet(selected []string) map[string]struct{} {
|
func publishSelectedArtifactSet(selected []string) map[string]struct{} {
|
||||||
if len(selected) == 0 {
|
if len(selected) == 0 {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -437,7 +439,7 @@ func archiveSelectedArtifactSet(selected []string) map[string]struct{} {
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveLockSet(locks []config.PublishLockRule) map[string]config.PublishLockRule {
|
func publishLockSet(locks []config.PublishLockRule) map[string]config.PublishLockRule {
|
||||||
out := make(map[string]config.PublishLockRule, len(locks))
|
out := make(map[string]config.PublishLockRule, len(locks))
|
||||||
for _, lock := range locks {
|
for _, lock := range locks {
|
||||||
source := strings.TrimSpace(lock.Source)
|
source := strings.TrimSpace(lock.Source)
|
||||||
@@ -451,37 +453,25 @@ func archiveLockSet(locks []config.PublishLockRule) map[string]config.PublishLoc
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
func resolvePublishOutputDest(rule config.PublishOutputRule, catalog *artifacts.ArtifactCatalog) (string, error) {
|
func resolvePublishOutputDest(rule config.PublishOutputRule, configured map[string]string) (string, error) {
|
||||||
dest := strings.TrimSpace(rule.Dest)
|
return artifactpolicy.ResolvePublishedDestination(rule.Source, rule.Dest, configured)
|
||||||
if dest == "" {
|
|
||||||
entry, ok := catalog.Lookup(strings.TrimSpace(rule.Source))
|
|
||||||
if !ok {
|
|
||||||
return "", fmt.Errorf("destination omitted and source is unknown")
|
|
||||||
}
|
|
||||||
dest = strings.TrimSpace(entry.CanonicalRelPath)
|
|
||||||
if dest == "" {
|
|
||||||
return "", fmt.Errorf("destination omitted and no canonical destination is available")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return normalizeArchiveRelativePath(dest)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func normalizeArchiveRelativePath(rel string) (string, error) {
|
func configuredOutputPathMapFromCatalog(catalog *artifacts.ArtifactCatalog) map[string]string {
|
||||||
trimmed := strings.TrimSpace(rel)
|
out := map[string]string{}
|
||||||
if trimmed == "" {
|
if catalog == nil {
|
||||||
return "", fmt.Errorf("relative path is required")
|
return out
|
||||||
}
|
}
|
||||||
cleaned := filepath.ToSlash(filepath.Clean(filepath.FromSlash(trimmed)))
|
for _, entry := range catalog.ListConfigured() {
|
||||||
if cleaned == "." || cleaned == "" {
|
if strings.TrimSpace(entry.ConfiguredKey) == "" {
|
||||||
return "", fmt.Errorf("relative path is required")
|
continue
|
||||||
|
}
|
||||||
|
out[entry.ConfiguredKey] = strings.TrimSpace(entry.CanonicalRelPath)
|
||||||
}
|
}
|
||||||
if filepath.IsAbs(trimmed) || strings.HasPrefix(cleaned, "/") || cleaned == ".." || strings.HasPrefix(cleaned, "../") {
|
return out
|
||||||
return "", fmt.Errorf("path must be a clean relative path")
|
|
||||||
}
|
|
||||||
return cleaned, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func buildArchiveRuntimeArtifactCatalog(
|
func buildPublishRuntimeArtifactCatalog(
|
||||||
paths artifacts.SessionPaths,
|
paths artifacts.SessionPaths,
|
||||||
scriptoriumCfg *config.ScriptoriumConfig,
|
scriptoriumCfg *config.ScriptoriumConfig,
|
||||||
) (*artifacts.ArtifactCatalog, error) {
|
) (*artifacts.ArtifactCatalog, error) {
|
||||||
@@ -548,8 +538,8 @@ func resolveConfiguredArtifactLocalPath(paths artifacts.SessionPaths, configured
|
|||||||
return filepath.Join(paths.Root, rel), nil
|
return filepath.Join(paths.Root, rel), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func collectArchiveRunFiles(runRoot, manifestPath string) ([]archiveUploadFile, error) {
|
func collectPublishRunFiles(runRoot, manifestPath string) ([]publishUploadFile, error) {
|
||||||
files := make([]archiveUploadFile, 0, 64)
|
files := make([]publishUploadFile, 0, 64)
|
||||||
err := filepath.WalkDir(runRoot, func(path string, d fs.DirEntry, walkErr error) error {
|
err := filepath.WalkDir(runRoot, func(path string, d fs.DirEntry, walkErr error) error {
|
||||||
if walkErr != nil {
|
if walkErr != nil {
|
||||||
return walkErr
|
return walkErr
|
||||||
@@ -563,7 +553,7 @@ func collectArchiveRunFiles(runRoot, manifestPath string) ([]archiveUploadFile,
|
|||||||
return fmt.Errorf("relative dir from %q to %q: %w", runRoot, path, err)
|
return fmt.Errorf("relative dir from %q to %q: %w", runRoot, path, err)
|
||||||
}
|
}
|
||||||
relDir = filepath.ToSlash(relDir)
|
relDir = filepath.ToSlash(relDir)
|
||||||
// Preserve existing behavior: audio is not uploaded in archive run record.
|
// Preserve existing behavior: audio is not uploaded in publish run record.
|
||||||
if relDir == "audio" || strings.HasPrefix(relDir, "audio/") {
|
if relDir == "audio" || strings.HasPrefix(relDir, "audio/") {
|
||||||
return filepath.SkipDir
|
return filepath.SkipDir
|
||||||
}
|
}
|
||||||
@@ -574,7 +564,7 @@ func collectArchiveRunFiles(runRoot, manifestPath string) ([]archiveUploadFile,
|
|||||||
return fmt.Errorf("relative path from %q to %q: %w", runRoot, path, err)
|
return fmt.Errorf("relative path from %q to %q: %w", runRoot, path, err)
|
||||||
}
|
}
|
||||||
rel = filepath.ToSlash(rel)
|
rel = filepath.ToSlash(rel)
|
||||||
files = append(files, archiveUploadFile{
|
files = append(files, publishUploadFile{
|
||||||
RelativePath: rel,
|
RelativePath: rel,
|
||||||
LocalPath: path,
|
LocalPath: path,
|
||||||
})
|
})
|
||||||
@@ -594,11 +584,11 @@ func collectArchiveRunFiles(runRoot, manifestPath string) ([]archiveUploadFile,
|
|||||||
if manifestInfo.IsDir() {
|
if manifestInfo.IsDir() {
|
||||||
return nil, fmt.Errorf("manifest path %q is a directory", manifestPath)
|
return nil, fmt.Errorf("manifest path %q is a directory", manifestPath)
|
||||||
}
|
}
|
||||||
files = append(files, archiveUploadFile{
|
files = append(files, publishUploadFile{
|
||||||
RelativePath: "manifest.json",
|
RelativePath: "manifest.json",
|
||||||
LocalPath: manifestPath,
|
LocalPath: manifestPath,
|
||||||
})
|
})
|
||||||
seen := map[string]archiveUploadFile{}
|
seen := map[string]publishUploadFile{}
|
||||||
for _, file := range files {
|
for _, file := range files {
|
||||||
seen[file.RelativePath] = file
|
seen[file.RelativePath] = file
|
||||||
}
|
}
|
||||||
@@ -613,7 +603,7 @@ func collectArchiveRunFiles(runRoot, manifestPath string) ([]archiveUploadFile,
|
|||||||
return files, nil
|
return files, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func collectArchivePreviousFiles(previousDir string) ([]archiveUploadFile, error) {
|
func collectPublishPreviousFiles(previousDir string) ([]publishUploadFile, error) {
|
||||||
previousDir = filepath.Clean(strings.TrimSpace(previousDir))
|
previousDir = filepath.Clean(strings.TrimSpace(previousDir))
|
||||||
if previousDir == "" {
|
if previousDir == "" {
|
||||||
return nil, fmt.Errorf("previous directory is required")
|
return nil, fmt.Errorf("previous directory is required")
|
||||||
@@ -629,7 +619,7 @@ func collectArchivePreviousFiles(previousDir string) ([]archiveUploadFile, error
|
|||||||
return nil, fmt.Errorf("previous path %q is not a directory", previousDir)
|
return nil, fmt.Errorf("previous path %q is not a directory", previousDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
files := make([]archiveUploadFile, 0, 16)
|
files := make([]publishUploadFile, 0, 16)
|
||||||
err = filepath.WalkDir(previousDir, func(path string, d fs.DirEntry, walkErr error) error {
|
err = filepath.WalkDir(previousDir, func(path string, d fs.DirEntry, walkErr error) error {
|
||||||
if walkErr != nil {
|
if walkErr != nil {
|
||||||
return walkErr
|
return walkErr
|
||||||
@@ -642,7 +632,7 @@ func collectArchivePreviousFiles(previousDir string) ([]archiveUploadFile, error
|
|||||||
return fmt.Errorf("relative path from %q to %q: %w", previousDir, path, err)
|
return fmt.Errorf("relative path from %q to %q: %w", previousDir, path, err)
|
||||||
}
|
}
|
||||||
rel = filepath.ToSlash(rel)
|
rel = filepath.ToSlash(rel)
|
||||||
files = append(files, archiveUploadFile{
|
files = append(files, publishUploadFile{
|
||||||
RelativePath: filepath.ToSlash(filepath.Join(config.PathPreviousDirSegment, rel)),
|
RelativePath: filepath.ToSlash(filepath.Join(config.PathPreviousDirSegment, rel)),
|
||||||
LocalPath: path,
|
LocalPath: path,
|
||||||
})
|
})
|
||||||
@@ -658,7 +648,7 @@ func collectArchivePreviousFiles(previousDir string) ([]archiveUploadFile, error
|
|||||||
return files, nil
|
return files, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func resolveArchiveRunManifestSource(runRoot string) (string, error) {
|
func resolvePublishRunManifestSource(runRoot string) (string, error) {
|
||||||
path := filepath.Join(filepath.Clean(runRoot), "manifest.json")
|
path := filepath.Join(filepath.Clean(runRoot), "manifest.json")
|
||||||
info, err := os.Stat(path)
|
info, err := os.Stat(path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -684,7 +674,7 @@ func directoryExists(path string) (bool, error) {
|
|||||||
return false, err
|
return false, err
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeCurrentManifestSnapshot(m *manifest.Manifest, archiveMetadata map[string]any) (string, error) {
|
func writeCurrentManifestSnapshot(m *manifest.Manifest, publishMetadata map[string]any) (string, error) {
|
||||||
if m == nil {
|
if m == nil {
|
||||||
return "", fmt.Errorf("manifest is required")
|
return "", fmt.Errorf("manifest is required")
|
||||||
}
|
}
|
||||||
@@ -717,7 +707,7 @@ func writeCurrentManifestSnapshot(m *manifest.Manifest, archiveMetadata map[stri
|
|||||||
now := time.Now().UTC()
|
now := time.Now().UTC()
|
||||||
clone.MarkStageSucceeded("publish", now, nil)
|
clone.MarkStageSucceeded("publish", now, nil)
|
||||||
if sr := clone.Stages["publish"]; sr != nil {
|
if sr := clone.Stages["publish"]; sr != nil {
|
||||||
sr.Metadata = archiveMetadata
|
sr.Metadata = publishMetadata
|
||||||
}
|
}
|
||||||
|
|
||||||
data, err := json.MarshalIndent(&clone, "", " ")
|
data, err := json.MarshalIndent(&clone, "", " ")
|
||||||
@@ -757,7 +747,7 @@ func writeCurrentRunIDPointer(runID string) (string, error) {
|
|||||||
return path, nil
|
return path, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveMetadataPreview(
|
func publishMetadataPreview(
|
||||||
bucket, runPrefix, sessionPrefix string,
|
bucket, runPrefix, sessionPrefix string,
|
||||||
runUploaded []string,
|
runUploaded []string,
|
||||||
publishedUploaded []string,
|
publishedUploaded []string,
|
||||||
@@ -17,11 +17,11 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestArchiveSkipsWhenDisabled(t *testing.T) {
|
func TestPublishSkipsWhenDisabled(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
env.Config.Pipeline.Publish.Enabled = boolPtr(false)
|
env.Config.Pipeline.Publish.Enabled = boolPtr(false)
|
||||||
|
|
||||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
result, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Run() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -29,15 +29,15 @@ func TestArchiveSkipsWhenDisabled(t *testing.T) {
|
|||||||
t.Fatalf("metadata = %#v, want skipped=true", result.Metadata)
|
t.Fatalf("metadata = %#v, want skipped=true", result.Metadata)
|
||||||
}
|
}
|
||||||
if len(env.ObjectStore.(*storage.FakeBackend).Uploads) != 0 {
|
if len(env.ObjectStore.(*storage.FakeBackend).Uploads) != 0 {
|
||||||
t.Fatalf("unexpected uploads when archive disabled")
|
t.Fatalf("unexpected uploads when publish disabled")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveSkipsRunUploadWhenDisabled(t *testing.T) {
|
func TestPublishSkipsRunUploadWhenDisabled(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
env.Config.Pipeline.Publish.UploadRun = boolPtr(false)
|
env.Config.Pipeline.Publish.UploadRun = boolPtr(false)
|
||||||
|
|
||||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
result, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Run() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -49,11 +49,11 @@ func TestArchiveSkipsRunUploadWhenDisabled(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveFailsWhenPrerequisiteNotSucceeded(t *testing.T) {
|
func TestPublishFailsWhenPrerequisiteNotSucceeded(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
m.Stages["trim"].Status = manifest.StatusFailed
|
m.Stages["trim"].Status = manifest.StatusFailed
|
||||||
|
|
||||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
_, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err == nil || !strings.Contains(err.Error(), `prerequisite stage "trim"`) {
|
if err == nil || !strings.Contains(err.Error(), `prerequisite stage "trim"`) {
|
||||||
t.Fatalf("Run() error = %v, want prerequisite failure", err)
|
t.Fatalf("Run() error = %v, want prerequisite failure", err)
|
||||||
}
|
}
|
||||||
@@ -62,11 +62,11 @@ func TestArchiveFailsWhenPrerequisiteNotSucceeded(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveUploadsRunRecordPromotionsAndCurrentPointer(t *testing.T) {
|
func TestPublishUploadsRunRecordPublishedOutputsAndCurrentPointer(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||||
|
|
||||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
result, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Run() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -94,10 +94,10 @@ func TestArchiveUploadsRunRecordPromotionsAndCurrentPointer(t *testing.T) {
|
|||||||
trimmedKey := sessionPrefix + "transcripts/final.trimmed.json"
|
trimmedKey := sessionPrefix + "transcripts/final.trimmed.json"
|
||||||
recapKey := sessionPrefix + "artifacts/session_recap.md"
|
recapKey := sessionPrefix + "artifacts/session_recap.md"
|
||||||
if _, ok := fake.Objects[trimmedKey]; !ok {
|
if _, ok := fake.Objects[trimmedKey]; !ok {
|
||||||
t.Fatalf("missing promoted key %q", trimmedKey)
|
t.Fatalf("missing published output key %q", trimmedKey)
|
||||||
}
|
}
|
||||||
if _, ok := fake.Objects[recapKey]; !ok {
|
if _, ok := fake.Objects[recapKey]; !ok {
|
||||||
t.Fatalf("missing promoted key %q", recapKey)
|
t.Fatalf("missing published output key %q", recapKey)
|
||||||
}
|
}
|
||||||
|
|
||||||
currentManifestKey := sessionPrefix + "current/manifest.json"
|
currentManifestKey := sessionPrefix + "current/manifest.json"
|
||||||
@@ -136,8 +136,8 @@ func TestArchiveUploadsRunRecordPromotionsAndCurrentPointer(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveUploadsPreviousCacheWhenPresent(t *testing.T) {
|
func TestPublishUploadsPreviousCacheWhenPresent(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||||
sessionRoot := artifacts.SessionWorkDirForCampaign(
|
sessionRoot := artifacts.SessionWorkDirForCampaign(
|
||||||
env.Config.Pipeline.Workspace.Root,
|
env.Config.Pipeline.Workspace.Root,
|
||||||
@@ -147,7 +147,7 @@ func TestArchiveUploadsPreviousCacheWhenPresent(t *testing.T) {
|
|||||||
writeStageTestFile(t, filepath.Join(sessionRoot, "previous", "manifest.json"), "{\"session_id\":\"2026-04-12\"}\n")
|
writeStageTestFile(t, filepath.Join(sessionRoot, "previous", "manifest.json"), "{\"session_id\":\"2026-04-12\"}\n")
|
||||||
writeStageTestFile(t, filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md"), "# previous recap\n")
|
writeStageTestFile(t, filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md"), "# previous recap\n")
|
||||||
|
|
||||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
result, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Run() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -155,20 +155,20 @@ func TestArchiveUploadsPreviousCacheWhenPresent(t *testing.T) {
|
|||||||
previousManifestKey := m.S3SessionPrefix + "previous/manifest.json"
|
previousManifestKey := m.S3SessionPrefix + "previous/manifest.json"
|
||||||
previousRecapKey := m.S3SessionPrefix + "previous/artifacts/session_recap.md"
|
previousRecapKey := m.S3SessionPrefix + "previous/artifacts/session_recap.md"
|
||||||
if _, ok := fake.Objects[previousManifestKey]; !ok {
|
if _, ok := fake.Objects[previousManifestKey]; !ok {
|
||||||
t.Fatalf("missing archived previous manifest key %q", previousManifestKey)
|
t.Fatalf("missing published previous manifest key %q", previousManifestKey)
|
||||||
}
|
}
|
||||||
if _, ok := fake.Objects[previousRecapKey]; !ok {
|
if _, ok := fake.Objects[previousRecapKey]; !ok {
|
||||||
t.Fatalf("missing archived previous artifact key %q", previousRecapKey)
|
t.Fatalf("missing published previous artifact key %q", previousRecapKey)
|
||||||
}
|
}
|
||||||
if result.Metadata["previous_files_uploaded"] != 2 {
|
if result.Metadata["previous_files_uploaded"] != 2 {
|
||||||
t.Fatalf("metadata previous_files_uploaded = %#v, want 2", result.Metadata["previous_files_uploaded"])
|
t.Fatalf("metadata previous_files_uploaded = %#v, want 2", result.Metadata["previous_files_uploaded"])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveToleratesMissingPreviousCache(t *testing.T) {
|
func TestPublishToleratesMissingPreviousCache(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
|
|
||||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
result, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Run() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -177,35 +177,35 @@ func TestArchiveToleratesMissingPreviousCache(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveUsesCustomPromotionRules(t *testing.T) {
|
func TestPublishUsesCustomOutputRules(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
env.Config.Pipeline.Publish.Outputs = []config.PublishOutputRule{
|
env.Config.Pipeline.Publish.Outputs = []config.PublishOutputRule{
|
||||||
{Source: "narratio.transcript.final_trimmed", Dest: "published/trimmed.json", Required: boolPtr(true)},
|
{Source: "narratio.transcript.final_trimmed", Dest: "published/trimmed.json", Required: boolPtr(true)},
|
||||||
{Source: "narratio.artifact.session_recap", Dest: "published/recap.md", Required: boolPtr(true)},
|
{Source: "narratio.artifact.session_recap", Dest: "published/recap.md", Required: boolPtr(true)},
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
_, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Run() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||||
if _, ok := fake.Objects[m.S3SessionPrefix+"published/trimmed.json"]; !ok {
|
if _, ok := fake.Objects[m.S3SessionPrefix+"published/trimmed.json"]; !ok {
|
||||||
t.Fatalf("missing custom promoted trimmed key")
|
t.Fatalf("missing custom published trimmed key")
|
||||||
}
|
}
|
||||||
if _, ok := fake.Objects[m.S3SessionPrefix+"published/recap.md"]; !ok {
|
if _, ok := fake.Objects[m.S3SessionPrefix+"published/recap.md"]; !ok {
|
||||||
t.Fatalf("missing custom promoted recap key")
|
t.Fatalf("missing custom published recap key")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveSkipsOptionalMissingPromotion(t *testing.T) {
|
func TestPublishSkipsOptionalMissingOutput(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
env.Config.Pipeline.Publish.Outputs = []config.PublishOutputRule{
|
env.Config.Pipeline.Publish.Outputs = []config.PublishOutputRule{
|
||||||
{Source: "narratio.transcript.final_trimmed", Dest: "transcripts/final.trimmed.json", Required: boolPtr(true)},
|
{Source: "narratio.transcript.final_trimmed", Dest: "transcripts/final.trimmed.json", Required: boolPtr(true)},
|
||||||
{Source: "narratio.transcript.base", Dest: "transcripts/base.json", Required: boolPtr(false)},
|
{Source: "narratio.transcript.base", Dest: "transcripts/base.json", Required: boolPtr(false)},
|
||||||
}
|
}
|
||||||
|
|
||||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
result, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Run() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -216,8 +216,8 @@ func TestArchiveSkipsOptionalMissingPromotion(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveSkipsRequiredUnselectedConfiguredPromotion(t *testing.T) {
|
func TestPublishSkipsRequiredUnselectedConfiguredOutput(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
env.SelectedArtifactKeys = []string{"player_handout"}
|
env.SelectedArtifactKeys = []string{"player_handout"}
|
||||||
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
|
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
|
||||||
Enabled: true,
|
Enabled: true,
|
||||||
@@ -226,15 +226,15 @@ func TestArchiveSkipsRequiredUnselectedConfiguredPromotion(t *testing.T) {
|
|||||||
}
|
}
|
||||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||||
|
|
||||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
result, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Run() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if _, ok := fake.Objects[m.S3SessionPrefix+"transcripts/final.trimmed.json"]; !ok {
|
if _, ok := fake.Objects[m.S3SessionPrefix+"transcripts/final.trimmed.json"]; !ok {
|
||||||
t.Fatalf("missing built-in promoted trimmed key")
|
t.Fatalf("missing built-in published trimmed key")
|
||||||
}
|
}
|
||||||
if _, ok := fake.Objects[m.S3SessionPrefix+"artifacts/session_recap.md"]; ok {
|
if _, ok := fake.Objects[m.S3SessionPrefix+"artifacts/session_recap.md"]; ok {
|
||||||
t.Fatalf("unexpected unselected recap promotion upload")
|
t.Fatalf("unexpected unselected recap published output upload")
|
||||||
}
|
}
|
||||||
skipped := result.Metadata["skipped_unselected_outputs"].([]map[string]any)
|
skipped := result.Metadata["skipped_unselected_outputs"].([]map[string]any)
|
||||||
if len(skipped) != 1 {
|
if len(skipped) != 1 {
|
||||||
@@ -251,8 +251,8 @@ func TestArchiveSkipsRequiredUnselectedConfiguredPromotion(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveSelectedConfiguredPromotionStillFailsWhenMissing(t *testing.T) {
|
func TestPublishSelectedConfiguredOutputStillFailsWhenMissing(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
env.SelectedArtifactKeys = []string{"session_recap"}
|
env.SelectedArtifactKeys = []string{"session_recap"}
|
||||||
sessionRoot := artifacts.SessionWorkDirForCampaign(
|
sessionRoot := artifacts.SessionWorkDirForCampaign(
|
||||||
env.Config.Pipeline.Workspace.Root,
|
env.Config.Pipeline.Workspace.Root,
|
||||||
@@ -263,26 +263,26 @@ func TestArchiveSelectedConfiguredPromotionStillFailsWhenMissing(t *testing.T) {
|
|||||||
t.Fatalf("remove recap: %v", err)
|
t.Fatalf("remove recap: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
_, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err == nil || !strings.Contains(err.Error(), `required output source unavailable: "narratio.artifact.session_recap"`) {
|
if err == nil || !strings.Contains(err.Error(), `required output source unavailable: "narratio.artifact.session_recap"`) {
|
||||||
t.Fatalf("Run() error = %v, want required selected output failure", err)
|
t.Fatalf("Run() error = %v, want required selected output failure", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveLockedSelectedPromotionSkipsAsLocked(t *testing.T) {
|
func TestPublishLockedSelectedOutputSkipsAsLocked(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
env.SelectedArtifactKeys = []string{"session_recap"}
|
env.SelectedArtifactKeys = []string{"session_recap"}
|
||||||
env.Config.Pipeline.Publish.Locks = []config.PublishLockRule{
|
env.Config.Pipeline.Publish.Locks = []config.PublishLockRule{
|
||||||
{Source: "narratio.artifact.session_recap", Reason: "reviewed"},
|
{Source: "narratio.artifact.session_recap", Reason: "reviewed"},
|
||||||
}
|
}
|
||||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||||
|
|
||||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
result, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Run() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if _, ok := fake.Objects[m.S3SessionPrefix+"artifacts/session_recap.md"]; ok {
|
if _, ok := fake.Objects[m.S3SessionPrefix+"artifacts/session_recap.md"]; ok {
|
||||||
t.Fatalf("unexpected locked recap promotion upload")
|
t.Fatalf("unexpected locked recap published output upload")
|
||||||
}
|
}
|
||||||
if result.Metadata["locked_output_count"] != 1 {
|
if result.Metadata["locked_output_count"] != 1 {
|
||||||
t.Fatalf("locked_output_count = %#v, want 1", result.Metadata["locked_output_count"])
|
t.Fatalf("locked_output_count = %#v, want 1", result.Metadata["locked_output_count"])
|
||||||
@@ -293,8 +293,8 @@ func TestArchiveLockedSelectedPromotionSkipsAsLocked(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveLockedUnselectedConfiguredPromotionSkipsAsUnselected(t *testing.T) {
|
func TestPublishLockedUnselectedConfiguredOutputSkipsAsUnselected(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
env.SelectedArtifactKeys = []string{"player_handout"}
|
env.SelectedArtifactKeys = []string{"player_handout"}
|
||||||
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
|
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
|
||||||
Enabled: true,
|
Enabled: true,
|
||||||
@@ -305,7 +305,7 @@ func TestArchiveLockedUnselectedConfiguredPromotionSkipsAsUnselected(t *testing.
|
|||||||
{Source: "narratio.artifact.session_recap", Reason: "reviewed"},
|
{Source: "narratio.artifact.session_recap", Reason: "reviewed"},
|
||||||
}
|
}
|
||||||
|
|
||||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
result, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Run() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -318,25 +318,25 @@ func TestArchiveLockedUnselectedConfiguredPromotionSkipsAsUnselected(t *testing.
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveSkipsLockedRequiredPromotionAndCommits(t *testing.T) {
|
func TestPublishSkipsLockedRequiredOutputAndCommits(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
env.Config.Pipeline.Publish.Locks = []config.PublishLockRule{
|
env.Config.Pipeline.Publish.Locks = []config.PublishLockRule{
|
||||||
{Source: "narratio.transcript.final_trimmed", Reason: "human reviewed"},
|
{Source: "narratio.transcript.final_trimmed", Reason: "human reviewed"},
|
||||||
}
|
}
|
||||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||||
|
|
||||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
result, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Run() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
trimmedKey := m.S3SessionPrefix + "transcripts/final.trimmed.json"
|
trimmedKey := m.S3SessionPrefix + "transcripts/final.trimmed.json"
|
||||||
if _, ok := fake.Objects[trimmedKey]; ok {
|
if _, ok := fake.Objects[trimmedKey]; ok {
|
||||||
t.Fatalf("locked promotion key %q should not be uploaded", trimmedKey)
|
t.Fatalf("locked published output key %q should not be uploaded", trimmedKey)
|
||||||
}
|
}
|
||||||
recapKey := m.S3SessionPrefix + "artifacts/session_recap.md"
|
recapKey := m.S3SessionPrefix + "artifacts/session_recap.md"
|
||||||
if _, ok := fake.Objects[recapKey]; !ok {
|
if _, ok := fake.Objects[recapKey]; !ok {
|
||||||
t.Fatalf("unlocked promotion key %q should be uploaded", recapKey)
|
t.Fatalf("unlocked published output key %q should be uploaded", recapKey)
|
||||||
}
|
}
|
||||||
runTrimmedKey := m.S3RunPrefix + "trim/outputs/transcripts/final.trimmed.json"
|
runTrimmedKey := m.S3RunPrefix + "trim/outputs/transcripts/final.trimmed.json"
|
||||||
if _, ok := fake.Objects[runTrimmedKey]; !ok {
|
if _, ok := fake.Objects[runTrimmedKey]; !ok {
|
||||||
@@ -365,7 +365,7 @@ func TestArchiveSkipsLockedRequiredPromotionAndCommits(t *testing.T) {
|
|||||||
locked[0]["required"] != true ||
|
locked[0]["required"] != true ||
|
||||||
locked[0]["local_path"] == "" ||
|
locked[0]["local_path"] == "" ||
|
||||||
locked[0]["provenance"] == "" {
|
locked[0]["provenance"] == "" {
|
||||||
t.Fatalf("locked promotion metadata = %#v", locked[0])
|
t.Fatalf("locked published output metadata = %#v", locked[0])
|
||||||
}
|
}
|
||||||
|
|
||||||
currentManifestKey := m.S3SessionPrefix + "current/manifest.json"
|
currentManifestKey := m.S3SessionPrefix + "current/manifest.json"
|
||||||
@@ -374,8 +374,8 @@ func TestArchiveSkipsLockedRequiredPromotionAndCommits(t *testing.T) {
|
|||||||
t.Fatalf("unmarshal current manifest: %v", err)
|
t.Fatalf("unmarshal current manifest: %v", err)
|
||||||
}
|
}
|
||||||
stages := current["stages"].(map[string]any)
|
stages := current["stages"].(map[string]any)
|
||||||
archive := stages["publish"].(map[string]any)
|
publishRecord := stages["publish"].(map[string]any)
|
||||||
meta := archive["metadata"].(map[string]any)
|
meta := publishRecord["metadata"].(map[string]any)
|
||||||
if meta["locked_output_count"] != float64(1) {
|
if meta["locked_output_count"] != float64(1) {
|
||||||
t.Fatalf("current manifest locked_output_count = %#v, want 1", meta["locked_output_count"])
|
t.Fatalf("current manifest locked_output_count = %#v, want 1", meta["locked_output_count"])
|
||||||
}
|
}
|
||||||
@@ -385,8 +385,8 @@ func TestArchiveSkipsLockedRequiredPromotionAndCommits(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveLockedRequiredMissingPromotionSucceeds(t *testing.T) {
|
func TestPublishLockedRequiredMissingOutputSucceeds(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
env.Config.Pipeline.Publish.Outputs = []config.PublishOutputRule{
|
env.Config.Pipeline.Publish.Outputs = []config.PublishOutputRule{
|
||||||
{Source: "narratio.transcript.base", Dest: "transcripts/base.json", Required: boolPtr(true)},
|
{Source: "narratio.transcript.base", Dest: "transcripts/base.json", Required: boolPtr(true)},
|
||||||
}
|
}
|
||||||
@@ -394,7 +394,7 @@ func TestArchiveLockedRequiredMissingPromotionSucceeds(t *testing.T) {
|
|||||||
{Source: "narratio.transcript.base", Reason: "manual merge is locked"},
|
{Source: "narratio.transcript.base", Reason: "manual merge is locked"},
|
||||||
}
|
}
|
||||||
|
|
||||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
result, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Run() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -402,22 +402,22 @@ func TestArchiveLockedRequiredMissingPromotionSucceeds(t *testing.T) {
|
|||||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||||
mergedKey := m.S3SessionPrefix + "transcripts/base.json"
|
mergedKey := m.S3SessionPrefix + "transcripts/base.json"
|
||||||
if _, ok := fake.Objects[mergedKey]; ok {
|
if _, ok := fake.Objects[mergedKey]; ok {
|
||||||
t.Fatalf("locked missing promotion key %q should not be uploaded", mergedKey)
|
t.Fatalf("locked missing published output key %q should not be uploaded", mergedKey)
|
||||||
}
|
}
|
||||||
if _, ok := fake.Objects[m.S3SessionPrefix+"current/run_id.txt"]; !ok {
|
if _, ok := fake.Objects[m.S3SessionPrefix+"current/run_id.txt"]; !ok {
|
||||||
t.Fatalf("current run pointer should be written for locked missing promotion")
|
t.Fatalf("current run pointer should be written for locked missing output")
|
||||||
}
|
}
|
||||||
locked := result.Metadata["locked_outputs"].([]map[string]any)
|
locked := result.Metadata["locked_outputs"].([]map[string]any)
|
||||||
if len(locked) != 1 {
|
if len(locked) != 1 {
|
||||||
t.Fatalf("locked_outputs = %#v, want one item", locked)
|
t.Fatalf("locked_outputs = %#v, want one item", locked)
|
||||||
}
|
}
|
||||||
if locked[0]["local_path"] != "" || locked[0]["provenance"] != "" {
|
if locked[0]["local_path"] != "" || locked[0]["provenance"] != "" {
|
||||||
t.Fatalf("locked missing promotion metadata = %#v, want empty local path/provenance", locked[0])
|
t.Fatalf("locked missing published output metadata = %#v, want empty local path/provenance", locked[0])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveLockDoesNotOverwriteExistingPromotion(t *testing.T) {
|
func TestPublishLockDoesNotOverwriteExistingOutput(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
env.Config.Pipeline.Publish.Locks = []config.PublishLockRule{
|
env.Config.Pipeline.Publish.Locks = []config.PublishLockRule{
|
||||||
{Source: "narratio.transcript.final_trimmed", Reason: "already published"},
|
{Source: "narratio.transcript.final_trimmed", Reason: "already published"},
|
||||||
}
|
}
|
||||||
@@ -425,40 +425,40 @@ func TestArchiveLockDoesNotOverwriteExistingPromotion(t *testing.T) {
|
|||||||
trimmedKey := m.S3SessionPrefix + "transcripts/final.trimmed.json"
|
trimmedKey := m.S3SessionPrefix + "transcripts/final.trimmed.json"
|
||||||
fake.SeedObject(storage.FakeObject{Key: trimmedKey, Data: []byte("previously published\n")})
|
fake.SeedObject(storage.FakeObject{Key: trimmedKey, Data: []byte("previously published\n")})
|
||||||
|
|
||||||
if _, err := (archiveStage{}).Run(context.Background(), env, m); err != nil {
|
if _, err := (publishStage{}).Run(context.Background(), env, m); err != nil {
|
||||||
t.Fatalf("Run() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
got := string(fake.Objects[trimmedKey].Data)
|
got := string(fake.Objects[trimmedKey].Data)
|
||||||
if got != "previously published\n" {
|
if got != "previously published\n" {
|
||||||
t.Fatalf("locked promotion object contents = %q, want existing object preserved", got)
|
t.Fatalf("locked published output object contents = %q, want existing object preserved", got)
|
||||||
}
|
}
|
||||||
for _, upload := range fake.Uploads {
|
for _, upload := range fake.Uploads {
|
||||||
if upload.Key == trimmedKey {
|
if upload.Key == trimmedKey {
|
||||||
t.Fatalf("locked promotion key %q was uploaded", trimmedKey)
|
t.Fatalf("locked published output key %q was uploaded", trimmedKey)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveFailsWhenRequiredPromotionMissing(t *testing.T) {
|
func TestPublishFailsWhenRequiredOutputMissing(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
env.Config.Pipeline.Publish.Outputs = []config.PublishOutputRule{
|
env.Config.Pipeline.Publish.Outputs = []config.PublishOutputRule{
|
||||||
{Source: "narratio.transcript.base", Dest: "transcripts/base.json", Required: boolPtr(true)},
|
{Source: "narratio.transcript.base", Dest: "transcripts/base.json", Required: boolPtr(true)},
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
_, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err == nil || !strings.Contains(err.Error(), "required output source unavailable") {
|
if err == nil || !strings.Contains(err.Error(), "required output source unavailable") {
|
||||||
t.Fatalf("Run() error = %v, want required output source unavailable failure", err)
|
t.Fatalf("Run() error = %v, want required output source unavailable failure", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveFailsWhenCanonicalRunRootMissing(t *testing.T) {
|
func TestPublishFailsWhenCanonicalRunRootMissing(t *testing.T) {
|
||||||
env, m, runRoot := archiveFixture(t)
|
env, m, runRoot := publishFixture(t)
|
||||||
if err := os.RemoveAll(runRoot); err != nil {
|
if err := os.RemoveAll(runRoot); err != nil {
|
||||||
t.Fatalf("remove run root: %v", err)
|
t.Fatalf("remove run root: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
_, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("expected missing run-root error, got nil")
|
t.Fatal("expected missing run-root error, got nil")
|
||||||
}
|
}
|
||||||
@@ -467,8 +467,8 @@ func TestArchiveFailsWhenCanonicalRunRootMissing(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveDoesNotWriteCurrentPointerWhenPromotionUploadFails(t *testing.T) {
|
func TestPublishDoesNotWriteCurrentPointerWhenOutputUploadFails(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||||
trimmedKey := m.S3SessionPrefix + "transcripts/final.trimmed.json"
|
trimmedKey := m.S3SessionPrefix + "transcripts/final.trimmed.json"
|
||||||
|
|
||||||
@@ -480,26 +480,26 @@ func TestArchiveDoesNotWriteCurrentPointerWhenPromotionUploadFails(t *testing.T)
|
|||||||
originalUpload := fake.Upload
|
originalUpload := fake.Upload
|
||||||
_ = originalUpload
|
_ = originalUpload
|
||||||
// Use UploadErr toggle by checking call sequence in postcondition.
|
// Use UploadErr toggle by checking call sequence in postcondition.
|
||||||
// First failure point is promotion upload; simulate by setting error immediately before promotion key write.
|
// First failure point is published output upload; simulate by setting error immediately before published output key write.
|
||||||
// We cannot hook FakeBackend per-key without changing public behavior; use dedicated backend wrapper instead.
|
// We cannot hook FakeBackend per-key without changing public behavior; use dedicated backend wrapper instead.
|
||||||
env.ObjectStore = &promotionFailingStore{delegate: fake, failKey: failingKey}
|
env.ObjectStore = &publishedOutputFailingStore{delegate: fake, failKey: failingKey}
|
||||||
|
|
||||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
_, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err == nil || !strings.Contains(err.Error(), "upload published output source") {
|
if err == nil || !strings.Contains(err.Error(), "upload published output source") {
|
||||||
t.Fatalf("Run() error = %v, want published output upload failure", err)
|
t.Fatalf("Run() error = %v, want published output upload failure", err)
|
||||||
}
|
}
|
||||||
if _, ok := fake.Objects[m.S3SessionPrefix+"current/run_id.txt"]; ok {
|
if _, ok := fake.Objects[m.S3SessionPrefix+"current/run_id.txt"]; ok {
|
||||||
t.Fatalf("unexpected current pointer write on promotion failure")
|
t.Fatalf("unexpected current pointer write on published output failure")
|
||||||
}
|
}
|
||||||
fake.UploadErr = origUploadErr
|
fake.UploadErr = origUploadErr
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveDoesNotWriteCurrentPointerWhenCurrentManifestUploadFails(t *testing.T) {
|
func TestPublishDoesNotWriteCurrentPointerWhenCurrentManifestUploadFails(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||||
env.ObjectStore = &promotionFailingStore{delegate: fake, failKey: m.S3SessionPrefix + "current/manifest.json"}
|
env.ObjectStore = &publishedOutputFailingStore{delegate: fake, failKey: m.S3SessionPrefix + "current/manifest.json"}
|
||||||
|
|
||||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
_, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err == nil || !strings.Contains(err.Error(), "current manifest") {
|
if err == nil || !strings.Contains(err.Error(), "current manifest") {
|
||||||
t.Fatalf("Run() error = %v, want current manifest upload failure", err)
|
t.Fatalf("Run() error = %v, want current manifest upload failure", err)
|
||||||
}
|
}
|
||||||
@@ -508,17 +508,17 @@ func TestArchiveDoesNotWriteCurrentPointerWhenCurrentManifestUploadFails(t *test
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestArchiveFailsWithoutObjectStore(t *testing.T) {
|
func TestPublishFailsWithoutObjectStore(t *testing.T) {
|
||||||
env, m, _ := archiveFixture(t)
|
env, m, _ := publishFixture(t)
|
||||||
env.ObjectStore = nil
|
env.ObjectStore = nil
|
||||||
|
|
||||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
_, err := publishStage{}.Run(context.Background(), env, m)
|
||||||
if err == nil || !strings.Contains(err.Error(), "object store") {
|
if err == nil || !strings.Contains(err.Error(), "object store") {
|
||||||
t.Fatalf("Run() error = %v, want object store backend failure", err)
|
t.Fatalf("Run() error = %v, want object store backend failure", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
|
func publishFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
root := t.TempDir()
|
root := t.TempDir()
|
||||||
@@ -549,7 +549,7 @@ func archiveFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
|
|||||||
m.S3Bucket = "my-dnd-archive"
|
m.S3Bucket = "my-dnd-archive"
|
||||||
m.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", campaign, sessionID)
|
m.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", campaign, sessionID)
|
||||||
m.S3RunPrefix = artifacts.S3RunPrefix(m.S3SessionPrefix, runID)
|
m.S3RunPrefix = artifacts.S3RunPrefix(m.S3SessionPrefix, runID)
|
||||||
for _, name := range archivePrerequisiteStages {
|
for _, name := range publishPrerequisiteStages {
|
||||||
m.MarkStageSucceeded(name, time.Date(2026, 5, 16, 1, 2, 3, 0, time.UTC), nil)
|
m.MarkStageSucceeded(name, time.Date(2026, 5, 16, 1, 2, 3, 0, time.UTC), nil)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -591,27 +591,27 @@ func archiveFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
|
|||||||
return env, m, runRoot
|
return env, m, runRoot
|
||||||
}
|
}
|
||||||
|
|
||||||
type promotionFailingStore struct {
|
type publishedOutputFailingStore struct {
|
||||||
delegate *storage.FakeBackend
|
delegate *storage.FakeBackend
|
||||||
failKey string
|
failKey string
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *promotionFailingStore) List(ctx context.Context, prefix string) ([]storage.ObjectInfo, error) {
|
func (s *publishedOutputFailingStore) List(ctx context.Context, prefix string) ([]storage.ObjectInfo, error) {
|
||||||
return s.delegate.List(ctx, prefix)
|
return s.delegate.List(ctx, prefix)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *promotionFailingStore) Download(ctx context.Context, key, localPath string) error {
|
func (s *publishedOutputFailingStore) Download(ctx context.Context, key, localPath string) error {
|
||||||
return s.delegate.Download(ctx, key, localPath)
|
return s.delegate.Download(ctx, key, localPath)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *promotionFailingStore) Upload(ctx context.Context, localPath, key string, opts storage.UploadOptions) (storage.ObjectInfo, error) {
|
func (s *publishedOutputFailingStore) Upload(ctx context.Context, localPath, key string, opts storage.UploadOptions) (storage.ObjectInfo, error) {
|
||||||
if strings.TrimSpace(key) == strings.TrimSpace(s.failKey) {
|
if strings.TrimSpace(key) == strings.TrimSpace(s.failKey) {
|
||||||
return storage.ObjectInfo{}, errors.New("forced upload failure")
|
return storage.ObjectInfo{}, errors.New("forced upload failure")
|
||||||
}
|
}
|
||||||
return s.delegate.Upload(ctx, localPath, key, opts)
|
return s.delegate.Upload(ctx, localPath, key, opts)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *promotionFailingStore) Exists(ctx context.Context, key string) (bool, error) {
|
func (s *publishedOutputFailingStore) Exists(ctx context.Context, key string) (bool, error) {
|
||||||
return s.delegate.Exists(ctx, key)
|
return s.delegate.Exists(ctx, key)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -9,6 +9,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/pathsafe"
|
||||||
)
|
)
|
||||||
|
|
||||||
type runStageLayout struct {
|
type runStageLayout struct {
|
||||||
@@ -92,18 +93,17 @@ func runLocalPathForCanonical(layout runStageLayout, sessionPaths artifacts.Sess
|
|||||||
if cleanCanonical == "" {
|
if cleanCanonical == "" {
|
||||||
return "", fmt.Errorf("canonical path is required")
|
return "", fmt.Errorf("canonical path is required")
|
||||||
}
|
}
|
||||||
rel, err := filepath.Rel(filepath.Clean(sessionPaths.Root), cleanCanonical)
|
rel, err := pathsafe.SlashRelativeFromRoot(sessionPaths.Root, cleanCanonical)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", fmt.Errorf("derive session-relative path for %q: %w", cleanCanonical, err)
|
return "", fmt.Errorf("derive session-relative path for %q: %w", cleanCanonical, err)
|
||||||
}
|
}
|
||||||
rel = filepath.Clean(rel)
|
if rel == config.PathPreviousDirSegment || strings.HasPrefix(rel, config.PathPreviousDirSegment+"/") {
|
||||||
if rel == "." || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
|
||||||
return "", fmt.Errorf("canonical path %q is outside session root %q", cleanCanonical, sessionPaths.Root)
|
|
||||||
}
|
|
||||||
if rel == config.PathPreviousDirSegment || strings.HasPrefix(rel, config.PathPreviousDirSegment+string(filepath.Separator)) {
|
|
||||||
return cleanCanonical, nil
|
return cleanCanonical, nil
|
||||||
}
|
}
|
||||||
localPath := filepath.Join(layout.OutputsDir, rel)
|
localPath, err := pathsafe.JoinSlashRelativeUnderRoot(layout.OutputsDir, rel)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("resolve run-local output path for %q: %w", cleanCanonical, err)
|
||||||
|
}
|
||||||
if err := os.MkdirAll(filepath.Dir(localPath), 0o755); err != nil {
|
if err := os.MkdirAll(filepath.Dir(localPath), 0o755); err != nil {
|
||||||
return "", fmt.Errorf("create run-local output parent for %q: %w", localPath, err)
|
return "", fmt.Errorf("create run-local output parent for %q: %w", localPath, err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -27,7 +27,6 @@ type Env struct {
|
|||||||
Seriatim seriatim.Runner
|
Seriatim seriatim.Runner
|
||||||
Audita audita.Runner
|
Audita audita.Runner
|
||||||
Scriptorium scriptorium.Runner
|
Scriptorium scriptorium.Runner
|
||||||
Storage storage.Backend
|
|
||||||
ObjectStore storage.ObjectStore
|
ObjectStore storage.ObjectStore
|
||||||
Notifier notify.Sender
|
Notifier notify.Sender
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -191,7 +191,7 @@ func TestTranscribeStageInvalidJSONFails(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestTranscribeStageUsesRunLocalOutputAndPromotesCanonical(t *testing.T) {
|
func TestTranscribeStageUsesRunLocalOutputAndMaterializesCanonical(t *testing.T) {
|
||||||
env, m := setupTranscribeEnv(t, []string{"alice.flac"})
|
env, m := setupTranscribeEnv(t, []string{"alice.flac"})
|
||||||
env.Config.Session.Campaign = "sample-campaign"
|
env.Config.Session.Campaign = "sample-campaign"
|
||||||
m.Campaign = "sample-campaign"
|
m.Campaign = "sample-campaign"
|
||||||
|
|||||||
@@ -106,7 +106,7 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
|||||||
SessionID: sessionID,
|
SessionID: sessionID,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("trim: promote trimmed transcript: %w", err)
|
return nil, fmt.Errorf("trim: materialize canonical final trimmed transcript: %w", err)
|
||||||
}
|
}
|
||||||
metadata["trim_action"] = "copy_disabled"
|
metadata["trim_action"] = "copy_disabled"
|
||||||
return &StageResult{
|
return &StageResult{
|
||||||
@@ -357,7 +357,7 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
|||||||
SessionID: sessionID,
|
SessionID: sessionID,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("trim: promote trimmed transcript: %w", err)
|
return nil, fmt.Errorf("trim: materialize canonical final trimmed transcript: %w", err)
|
||||||
}
|
}
|
||||||
materializedBounds, err := materializeRunLocalOutput(env.ArtifactStore, finalBoundsOutputPath, canonicalBoundsOutputPath, artifacts.Ref{
|
materializedBounds, err := materializeRunLocalOutput(env.ArtifactStore, finalBoundsOutputPath, canonicalBoundsOutputPath, artifacts.Ref{
|
||||||
Kind: "session_bounds",
|
Kind: "session_bounds",
|
||||||
@@ -365,7 +365,7 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
|||||||
SessionID: sessionID,
|
SessionID: sessionID,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("trim: promote session bounds: %w", err)
|
return nil, fmt.Errorf("trim: materialize canonical session bounds: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
return &StageResult{
|
return &StageResult{
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user