Compare commits
87 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 717451512a | |||
| 3ddb3a947b | |||
| c6632d5576 | |||
| ffc07922c7 | |||
| f3310d4d16 | |||
| 88cee96d8d | |||
| 2fece10215 | |||
| 0658f2f642 | |||
| a51228c803 | |||
| 4491fb5ccd | |||
| 30b905765c | |||
| 03eac70881 | |||
| 0f7e6b979f | |||
| c366912586 | |||
| 9fe44cd00d | |||
| 094b0d2532 | |||
| 98649f4d81 | |||
| 8a559efd5b | |||
| 72deccb4e2 | |||
| 5620fc5bcf | |||
| be57e675e0 | |||
| 3971443831 | |||
| a6b0c33e9f | |||
| 96b886e711 | |||
| 7d584ee6cd | |||
| 572a112c31 | |||
| ea87c335d6 | |||
| 7169ff04df | |||
| ef1f650bc0 | |||
| 0d02cb9fa0 | |||
| 0299b128cf | |||
| d723384888 | |||
| 54228055c8 | |||
| 23ed716450 | |||
| ab59bab044 | |||
| 71395bb076 | |||
| 79737edf79 | |||
| df2c765b7f | |||
| f050b9dd54 | |||
| 9c9cb54339 | |||
| 7657ec3ad6 | |||
| cee52aa092 | |||
| e920f3a8d5 | |||
| 591c529a09 | |||
| 7324c5a686 | |||
| d0936fb022 | |||
| 2aa074c5cf | |||
| 782d0cf3b9 | |||
| 083c01cfa0 | |||
| 2937696024 | |||
| b817a5b772 | |||
| 3022f20beb | |||
| ca1ded1821 | |||
| 3752f3ed28 | |||
| 870c2d69d5 | |||
| 135407ba7c | |||
| 228c348e42 | |||
| a813bd5a50 | |||
| d8f58dce31 | |||
| 7111edeca4 | |||
| 3aae4bbb12 | |||
| b29d8eeb50 | |||
| dffb432537 | |||
| 2dd38c7913 | |||
| bc2ade38d9 | |||
| 5be831eb13 | |||
| cae4d99a89 | |||
| e09dc0512d | |||
| ae82bc1ce0 | |||
| 01eb7aa1aa | |||
| 2ca700195c | |||
| 2b08c34539 | |||
| 79f1fc1e09 | |||
| 9c753270bd | |||
| b907cb01aa | |||
| 7824afd4a5 | |||
| 2a4e1e912c | |||
| dd03c09d75 | |||
| 5bc8e8683f | |||
| 648001a8fe | |||
| 6684774f52 | |||
| f3b63bd5e5 | |||
| 23d6470b0f | |||
| 128449040f | |||
| 02ab106ade | |||
| c128970f58 | |||
| d001baa660 |
18
README.md
18
README.md
@@ -1,22 +1,22 @@
|
||||
# narratio
|
||||
|
||||
Narratio is a Go orchestration application that turns D&D session audio into polished transcripts and generated session artifacts.
|
||||
Narratio is a stage-driven Go orchestrator for turning D&D session audio into polished transcripts and generated artifacts.
|
||||
|
||||
It coordinates transcription, merge/polish/normalize/trim processing, artifact generation, archive publishing, and resumable run state in one operator workflow.
|
||||
It runs a deterministic workflow across `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `render`, `analyze`, and `publish`, with manifest-driven continuation and restore support.
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04
|
||||
narratio run 2026-04-04
|
||||
```
|
||||
|
||||
This command requires discoverable `pipeline.yml` and `session.yml` files (or explicit `--config` and `--session` flags).
|
||||
This requires resolvable `pipeline.yml`, `campaign.yml`, and concrete `session.yml` (or explicit config flags).
|
||||
|
||||
## Documentation
|
||||
|
||||
- [Configuration](docs/config.md)
|
||||
- [CLI Reference](docs/cli.md)
|
||||
- [Operations and Recovery](docs/operations.md)
|
||||
- [Configuration](docs/config.md)
|
||||
- [Operations](docs/operations.md)
|
||||
- [Troubleshooting](docs/troubleshooting.md)
|
||||
- [Development Guide](docs/development.md)
|
||||
- [Architecture Principles](docs/architecture.md)
|
||||
- [Internal Component Contracts](docs/internal/README.md)
|
||||
- [Config Examples](examples/)
|
||||
- [Development Guide](docs/policy/development.md)
|
||||
- [Architecture Principles](docs/policy/architecture.md)
|
||||
- [Maintained Examples](examples/)
|
||||
|
||||
353
docs/cli.md
353
docs/cli.md
@@ -1,60 +1,89 @@
|
||||
# CLI
|
||||
# CLI Reference
|
||||
|
||||
## Shortest Useful Command
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04
|
||||
narratio run 2026-04-04
|
||||
```
|
||||
|
||||
This command uses default config discovery for `pipeline.yml` and `session.yml`; both files must be discoverable unless you pass explicit `--config` and `--session` paths.
|
||||
This runs the canonical full pipeline for session `2026-04-04`.
|
||||
|
||||
## Command Overview
|
||||
|
||||
Implemented commands:
|
||||
Top-level commands:
|
||||
|
||||
- `run`: execute pipeline stages and persist manifest state.
|
||||
- `plan`: validate config, prepare workspace layout, and print stage run/skip decisions.
|
||||
- `resume`: continue from first non-succeeded stage unless forced.
|
||||
- `status`: read and print stage statuses from an existing manifest.
|
||||
- `run-stage`: execute exactly one stage.
|
||||
- `run <session_id>`: run full stage order.
|
||||
- `run-stage <stage> <session_id>`: run one stage.
|
||||
- `analyze <session_id>`: force-run analyze.
|
||||
- `publish <session_id>`: force-run publish.
|
||||
- `clean <session_id>` or `clean --all`: remove local work/spool state.
|
||||
- `session <subcommand>`: session helper commands.
|
||||
|
||||
Unknown commands print usage and exit non-zero.
|
||||
Session subcommands:
|
||||
|
||||
For config semantics, see [docs/config.md](./config.md). For operator lifecycle and recovery, see [docs/operations.md](./operations.md).
|
||||
- `session init <session_id>`
|
||||
- `session plan <session_id>`
|
||||
- `session validate <session_id>`
|
||||
- `session status <session_id>`
|
||||
- `session restore <session_id>`
|
||||
- `session artifacts <session_id>`
|
||||
- `session locks <session_id>`
|
||||
- `session locks add <session_id> <source>`
|
||||
- `session locks remove <session_id> <source>`
|
||||
|
||||
## Complete Flag Reference
|
||||
## Common Config Flags
|
||||
|
||||
Most session-aware commands accept:
|
||||
|
||||
- `--config <pipeline.yml>`
|
||||
- `--campaign <id>`
|
||||
- `--campaign-file <campaign.yml>`
|
||||
- `--session <session.yml>`
|
||||
- `--session-id <session_id>`
|
||||
- `--previous-session-id <session_id>`
|
||||
|
||||
Rules:
|
||||
|
||||
- `--campaign` and `--campaign-file` are mutually exclusive.
|
||||
- `--session` is not used by `session init`.
|
||||
- if both positional `<session_id>` and `--session-id` are provided, values must match.
|
||||
- `clean --all` cannot be combined with campaign/session selectors.
|
||||
|
||||
## Session ID Input Rules
|
||||
|
||||
Session-aware commands accept one of these forms:
|
||||
|
||||
- positional session ID: `... <session_id>`
|
||||
- compatibility flag: `... --session-id <session_id>`
|
||||
|
||||
When both are present, command parsing requires an exact match.
|
||||
|
||||
Commands with additional positionals keep their command-specific order:
|
||||
|
||||
- `run-stage <stage> <session_id>` or `run-stage <stage> --session-id <session_id>`
|
||||
- `session locks add <session_id> <source>` or `session locks add --session-id <session_id> <source>`
|
||||
- `session locks remove <session_id> <source>` or `session locks remove --session-id <session_id> <source>`
|
||||
|
||||
## Command Reference
|
||||
|
||||
### `run`
|
||||
|
||||
- `--config <path>`: optional explicit `pipeline.yml` path.
|
||||
- `--session <path>`: optional explicit `session.yml` path.
|
||||
- `--session-id <value>`: session template variable value.
|
||||
- `--force`: force stage execution.
|
||||
- `--artifacts <names>`: analyze artifact keys to execute (repeatable or comma-separated).
|
||||
```bash
|
||||
narratio run <session_id> [--force] [--artifacts <name[,name...]>] [...common config flags]
|
||||
```
|
||||
|
||||
### `plan`
|
||||
Behavior:
|
||||
|
||||
- `--config <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--force`
|
||||
|
||||
### `resume`
|
||||
|
||||
- `--config <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--force`
|
||||
- `--artifacts <names>`: analyze artifact keys to execute (repeatable or comma-separated).
|
||||
- evaluates full stage order;
|
||||
- skips already-succeeded stages unless `--force` is set;
|
||||
- continues interrupted or partially completed sessions by running non-succeeded stages;
|
||||
- writes session and run manifests.
|
||||
|
||||
### `run-stage`
|
||||
|
||||
- `--config <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--force`
|
||||
- `--artifacts <names>`: analyze artifact keys to execute (repeatable or comma-separated).
|
||||
- positional `<stage>`: required stage name.
|
||||
```bash
|
||||
narratio run-stage <stage> <session_id> [--force] [--artifacts <name[,name...]>] [...common config flags]
|
||||
```
|
||||
|
||||
Valid stage names:
|
||||
|
||||
@@ -64,159 +93,189 @@ Valid stage names:
|
||||
- `polish`
|
||||
- `normalize`
|
||||
- `trim`
|
||||
- `render`
|
||||
- `analyze`
|
||||
- `archive`
|
||||
- `publish`
|
||||
- `notify`
|
||||
|
||||
### `status`
|
||||
Rules:
|
||||
|
||||
- `--manifest <path>`: required manifest path.
|
||||
- `--artifacts` is accepted only for `analyze` and `publish` stage targets.
|
||||
|
||||
## Command Reference
|
||||
|
||||
### `run`
|
||||
|
||||
Purpose:
|
||||
- Execute configured stages in canonical order.
|
||||
|
||||
Syntax:
|
||||
### `analyze`
|
||||
|
||||
```bash
|
||||
narratio run [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force] [--artifacts <name[,name...]>]
|
||||
narratio analyze <session_id> [--artifacts <name[,name...]>] [...common config flags]
|
||||
```
|
||||
|
||||
Success output:
|
||||
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
||||
|
||||
Common failure cases:
|
||||
- missing default config/session paths when flags omitted.
|
||||
- invalid template/rendered session mismatch.
|
||||
- unknown/invalid `--artifacts` value.
|
||||
- `--artifacts` with unknown configured artifact key.
|
||||
|
||||
### `plan`
|
||||
|
||||
Purpose:
|
||||
- Validate config, load secrets (if configured), prepare workdir, and print stage run/skip decisions.
|
||||
|
||||
Syntax:
|
||||
Equivalent to:
|
||||
|
||||
```bash
|
||||
narratio plan [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force]
|
||||
narratio run-stage analyze <session_id> --force [...common config flags]
|
||||
```
|
||||
|
||||
Success output includes:
|
||||
- `narratio plan: workdir prepared at <path>`
|
||||
- one line per stage (`<stage>: run|skip`)
|
||||
- `totals: run=<n> skip=<n>`
|
||||
|
||||
Common failure cases:
|
||||
- same config/session discovery and validation failures as `run`.
|
||||
- secrets directory read failures when `pipeline.secrets.env_dir` is configured.
|
||||
|
||||
### `resume`
|
||||
|
||||
Purpose:
|
||||
- Continue from session-manifest stage status.
|
||||
|
||||
Syntax:
|
||||
### `publish`
|
||||
|
||||
```bash
|
||||
narratio resume [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force] [--artifacts <name[,name...]>]
|
||||
narratio publish <session_id> [--artifacts <name[,name...]>] [...common config flags]
|
||||
```
|
||||
|
||||
Success output:
|
||||
- `narratio resume: session <session_id> has no remaining stages`
|
||||
- or `narratio resume: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
||||
|
||||
Common failure cases:
|
||||
- same discovery/template/validation failures as `run`.
|
||||
- manifest load errors when existing manifest is unreadable.
|
||||
- invalid or unknown artifact selections.
|
||||
|
||||
### `status`
|
||||
|
||||
Purpose:
|
||||
- Inspect one manifest file without executing stages.
|
||||
|
||||
Syntax:
|
||||
Equivalent to:
|
||||
|
||||
```bash
|
||||
narratio status --manifest <manifest.json>
|
||||
narratio run-stage publish <session_id> --force [...common config flags]
|
||||
```
|
||||
|
||||
Success output includes:
|
||||
- `session_id: <id>`
|
||||
- `updated_at: <timestamp>`
|
||||
- `stages:` entries (`- <stage>: <status>`)
|
||||
|
||||
Common failure cases:
|
||||
- missing `--manifest`.
|
||||
- unreadable or invalid manifest path.
|
||||
|
||||
### `run-stage`
|
||||
|
||||
Purpose:
|
||||
- Execute exactly one stage.
|
||||
|
||||
Syntax:
|
||||
### `clean`
|
||||
|
||||
```bash
|
||||
narratio run-stage [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force] [--artifacts <name[,name...]>] <stage>
|
||||
narratio clean <session_id> [--dry-run] [--clear-cache] [...common config flags]
|
||||
narratio clean --all [--dry-run] [--clear-cache] [--config <pipeline.yml>]
|
||||
```
|
||||
|
||||
Success output:
|
||||
- `narratio run-stage: stage=<name> executed=<n> skipped=<n> force=<true|false>; manifest=<path>`
|
||||
Behavior:
|
||||
|
||||
`--artifacts` behavior:
|
||||
- accepted only when `<stage>` is `analyze`.
|
||||
- names are normalized (trimmed, deduplicated, sorted).
|
||||
- unknown configured artifact keys fail.
|
||||
- session mode removes:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}`
|
||||
- `{spool.root}/{campaign}/{session_id}`
|
||||
- `--all` removes:
|
||||
- `{workspace.root}/work/*`
|
||||
- direct children under `{spool.root}`
|
||||
- cache remains unless `--clear-cache` is provided.
|
||||
|
||||
Common failure cases:
|
||||
- missing stage positional arg.
|
||||
- unknown stage name.
|
||||
- using `--artifacts` with any non-`analyze` stage.
|
||||
### `session plan`
|
||||
|
||||
```bash
|
||||
narratio session plan <session_id> [--force] [...common config flags]
|
||||
```
|
||||
|
||||
Validates config, prepares local workdir layout, and prints run/skip decisions for each stage.
|
||||
|
||||
### `session validate`
|
||||
|
||||
```bash
|
||||
narratio session validate <session_id> [...common config flags]
|
||||
```
|
||||
|
||||
Read-only preflight checks for config validity, required inputs, audio mode, previous-session requirements, publish outputs, and effective locks.
|
||||
|
||||
### `session status`
|
||||
|
||||
```bash
|
||||
narratio session status <session_id> [...common config flags]
|
||||
```
|
||||
|
||||
Prints local manifest state and, when storage is available, remote current-state and published-output status.
|
||||
|
||||
### `session init`
|
||||
|
||||
```bash
|
||||
narratio session init <session_id> --output ./session.yml [options]
|
||||
narratio session init <session_id> --remote [options]
|
||||
```
|
||||
|
||||
Required target selection:
|
||||
|
||||
- exactly one of:
|
||||
- `--output <path>`
|
||||
- `--remote`
|
||||
|
||||
Options:
|
||||
|
||||
- `--config <pipeline.yml>`
|
||||
- `--campaign <id>` or `--campaign-file <campaign.yml>`
|
||||
- `--previous-session-id <id>`
|
||||
- `--date <YYYY-MM-DD>`
|
||||
- `--title <text>`
|
||||
- `--audio-dir <path>`
|
||||
- `--audio-s3-prefix <prefix>`
|
||||
- `--force`
|
||||
|
||||
Rules:
|
||||
|
||||
- `--audio-dir` and `--audio-s3-prefix` are mutually exclusive.
|
||||
- if campaign `session_template_file` is configured, `session init` renders it.
|
||||
- generated session YAML must be concrete (no unresolved `{{ ... }}` placeholders).
|
||||
|
||||
### `session restore`
|
||||
|
||||
```bash
|
||||
narratio session restore <session_id> [--dry-run] [--force] [--include-audio] [...common config flags]
|
||||
```
|
||||
|
||||
Behavior:
|
||||
|
||||
- discovers committed remote current state;
|
||||
- plans local restores;
|
||||
- writes `reports/restore-latest.json` on execution;
|
||||
- blocks conflicting overwrites unless `--force` is set.
|
||||
|
||||
Default restore scope:
|
||||
|
||||
- `manifest.json`
|
||||
- `transcripts/**`
|
||||
- `artifacts/**`
|
||||
- `previous/**` when required by configured previous-session inputs
|
||||
|
||||
`audio/**` is included only with `--include-audio`.
|
||||
|
||||
### `session artifacts`
|
||||
|
||||
```bash
|
||||
narratio session artifacts <session_id> [--remote] [...common config flags]
|
||||
```
|
||||
|
||||
Lists effective built-in and configured artifact sources, publish rules, lock state, and optional remote published-state availability.
|
||||
|
||||
### `session locks`
|
||||
|
||||
```bash
|
||||
narratio session locks <session_id> [...common config flags]
|
||||
narratio session locks add <session_id> <source> [--reason <text>] [--force] [...common config flags]
|
||||
narratio session locks remove <session_id> <source> [...common config flags]
|
||||
```
|
||||
|
||||
Behavior:
|
||||
|
||||
- list mode merges static `pipeline.publish.locks` with remote `{session_prefix}/locks.yml`;
|
||||
- add/remove mutate only remote locks;
|
||||
- static locks from pipeline config cannot be removed by CLI commands.
|
||||
|
||||
## `--artifacts` Selection Rules
|
||||
|
||||
- accepted on `run`, `run-stage`, `analyze`, and `publish`;
|
||||
- names must exist in `pipeline.scriptorium.artifacts`;
|
||||
- empty entries are invalid;
|
||||
- repeated names are deduplicated.
|
||||
|
||||
Effects:
|
||||
|
||||
- filters analyze execution to selected configured artifacts;
|
||||
- filters publish rules that source `narratio.artifact.<name>`;
|
||||
- does not filter built-in transcript/bounds publish sources.
|
||||
|
||||
## Common Workflows
|
||||
|
||||
Default-discovery run:
|
||||
Run full pipeline:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04
|
||||
narratio run 2026-04-04
|
||||
```
|
||||
|
||||
Run only selected analyze artifacts:
|
||||
Dry-run restore plan:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04 --artifacts session_recap,player_handout
|
||||
narratio session restore 2026-04-04 --dry-run
|
||||
```
|
||||
|
||||
Resume with selected analyze artifacts:
|
||||
Generate a concrete session file from template/default structure:
|
||||
|
||||
```bash
|
||||
narratio resume --session-id 2026-04-04 --artifacts player_handout
|
||||
narratio session init 2026-04-04 --output ./session.yml --date 2026-04-04 --title "Session 12"
|
||||
```
|
||||
|
||||
Run only analyze stage with selected artifacts:
|
||||
Force publish only:
|
||||
|
||||
```bash
|
||||
narratio run-stage --session-id 2026-04-04 --artifacts player_handout analyze
|
||||
narratio publish 2026-04-04
|
||||
```
|
||||
|
||||
## Diagnostic / Recovery Commands
|
||||
|
||||
Inspect stage status:
|
||||
|
||||
```bash
|
||||
narratio status --manifest <manifest.json>
|
||||
```
|
||||
|
||||
Get manifest path from previous output:
|
||||
- `run`, `resume`, and `run-stage` print `manifest=<path>` on success.
|
||||
|
||||
## `--artifacts` and `--force`
|
||||
|
||||
- `--artifacts` filters which configured artifacts are executable when analyze runs.
|
||||
- `--artifacts` does not imply `--force`.
|
||||
- If analyze is already `succeeded` and `--force` is not set, runner-level skip still applies.
|
||||
|
||||
375
docs/config.md
375
docs/config.md
@@ -1,149 +1,139 @@
|
||||
# Configuration
|
||||
# Configuration Reference
|
||||
|
||||
## 1. Overview
|
||||
## Purpose
|
||||
|
||||
Narratio loads two YAML files:
|
||||
Narratio resolves three YAML documents:
|
||||
|
||||
- `pipeline.yml`: pipeline-level runtime configuration.
|
||||
- `session.yml`: per-session metadata and input selection.
|
||||
- `pipeline.yml`: pipeline/runtime settings
|
||||
- `campaign.yml`: campaign identity and stable input defaults
|
||||
- `session.yml`: session identity, metadata, and audio source selection
|
||||
|
||||
These commands load and validate both files before running:
|
||||
## Discovery and Selection
|
||||
|
||||
- `narratio run`
|
||||
- `narratio plan`
|
||||
- `narratio resume`
|
||||
- `narratio run-stage`
|
||||
### `pipeline.yml`
|
||||
|
||||
Behavior:
|
||||
When `--config` is omitted, search order is:
|
||||
|
||||
- strict YAML decode is enabled (`KnownFields(true)`): unknown fields fail.
|
||||
- session templates render before session YAML decode.
|
||||
- defaults are applied for optional pipeline fields.
|
||||
- validation enforces required fields, value formats, and cross-field constraints.
|
||||
1. `/usr/local/etc/narratio/pipeline.yml`
|
||||
2. `/etc/narratio/pipeline.yml`
|
||||
|
||||
## 2. Config file discovery
|
||||
### `campaign.yml`
|
||||
|
||||
Pipeline config lookup for `run`, `plan`, `resume`, and `run-stage`:
|
||||
Selection rules:
|
||||
|
||||
- If `--config <path>` is provided, that path is used.
|
||||
- If omitted, Narratio searches in order:
|
||||
1. `/usr/local/etc/narratio/pipeline.yml`
|
||||
2. `/etc/narratio/pipeline.yml`
|
||||
- First existing file wins.
|
||||
- if `--campaign-file` is set, use that path;
|
||||
- else if `--campaign <id>` is set, use `{pipeline.campaigns.root}/{id}/campaign.yml`;
|
||||
- else use `{pipeline.campaigns.root}/{pipeline.campaigns.default_campaign_id}/campaign.yml`.
|
||||
|
||||
## 3. Session file discovery and templating
|
||||
### `session.yml`
|
||||
|
||||
Session config lookup for `run`, `plan`, `resume`, and `run-stage`:
|
||||
When `--session` is omitted, local search order is:
|
||||
|
||||
- If `--session <path>` is provided, that path is used.
|
||||
- If omitted, Narratio searches in order:
|
||||
1. `./session.yml`
|
||||
2. `/usr/local/etc/narratio/session.yml`
|
||||
3. `/etc/narratio/session.yml`
|
||||
- First existing file wins.
|
||||
1. `/usr/local/etc/narratio/session.yml`
|
||||
2. `/etc/narratio/session.yml`
|
||||
|
||||
Template behavior:
|
||||
If local session discovery fails and a `session_id` is known, Narratio attempts remote session loading from:
|
||||
|
||||
- Supported placeholders:
|
||||
- `{{session_id}}`
|
||||
- `{{ session_id }}`
|
||||
- `--session-id <value>` supplies the placeholder value.
|
||||
- unresolved placeholders fail load.
|
||||
- if rendered `session_id` mismatches `--session-id`, load fails.
|
||||
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`
|
||||
|
||||
## 4. Minimal pipeline config
|
||||
using configured object storage.
|
||||
|
||||
## Validation and Merge Rules
|
||||
|
||||
- YAML decode is strict (`KnownFields(true)`): unknown fields fail load.
|
||||
- Session files must be concrete; unresolved `{{ ... }}` placeholders fail load.
|
||||
- Pipeline defaults are applied before validation.
|
||||
- Campaign and session identities must agree.
|
||||
- Stable files (`speakers_file`, `autocorrect_file`, `glossary_file`, `players_file`, `party_file`) resolve from session overrides when provided, otherwise from campaign defaults.
|
||||
- Exactly one audio mode must be configured in session input:
|
||||
- local (`audio_dir` or `audio_files`), or
|
||||
- S3 (`audio_s3.prefix`).
|
||||
|
||||
## Minimal Working Configuration
|
||||
|
||||
`pipeline.yml`
|
||||
|
||||
```yaml
|
||||
campaigns:
|
||||
root: /usr/local/share/narratio/campaigns
|
||||
default_campaign_id: sample-campaign
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
transcribe_url: https://transcription.example.com/transcribe
|
||||
```
|
||||
|
||||
Why this is sufficient:
|
||||
|
||||
- `whisperx.transcribe_url` is required.
|
||||
- `workspace.root` defaults to `/var/lib/narratio`.
|
||||
- optional sections (`seriatim`, `audita`, `archive`, `scriptorium`, `trim`, `normalize`, etc.) receive defaults or stay inactive.
|
||||
|
||||
## 5. Minimal session template
|
||||
`campaign.yml`
|
||||
|
||||
```yaml
|
||||
session_id: "{{ session_id }}"
|
||||
campaign: sample-campaign
|
||||
campaign_id: sample-campaign
|
||||
inputs:
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
players_file: ./players.yml
|
||||
party_file: ./party.yml
|
||||
```
|
||||
|
||||
`session.yml` (local audio)
|
||||
|
||||
```yaml
|
||||
session_id: 2026-05-03
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./examples/speakers.yml
|
||||
autocorrect_file: ./examples/autocorrect.yml
|
||||
glossary_file: ./examples/glossary.yml
|
||||
```
|
||||
|
||||
Usage:
|
||||
## Secrets Handling
|
||||
|
||||
```bash
|
||||
narratio run --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-05-03
|
||||
```
|
||||
- Do not place raw secrets in YAML.
|
||||
- Use env var names in config (for example `pipeline.audita.llm_api_key_env`).
|
||||
- Optionally load env files from `pipeline.secrets.env_dir`.
|
||||
- Commands that need storage/auth load filesystem secrets before constructing adapters.
|
||||
|
||||
## 6. Production-oriented config
|
||||
## Publish Configuration Summary
|
||||
|
||||
Publish rules live under `pipeline.publish`.
|
||||
|
||||
```yaml
|
||||
workspace:
|
||||
root: /var/lib/narratio/workspace
|
||||
cleanup_after_archive: true
|
||||
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: my-dnd-archive
|
||||
root_prefix: dnd
|
||||
region: us-east-1
|
||||
access_key_id_env: OBJECT_STORAGE_KEY_ID
|
||||
secret_access_key_env: OBJECT_STORAGE_KEY
|
||||
|
||||
spool:
|
||||
root: /var/spool/narratio
|
||||
delete_audio_after_archive: true
|
||||
|
||||
archive:
|
||||
publish:
|
||||
enabled: true
|
||||
upload_run: true
|
||||
promote_artifacts:
|
||||
- from: transcripts/trimmed.json
|
||||
to: transcripts/trimmed.json
|
||||
outputs:
|
||||
- source: narratio.transcript.final_trimmed
|
||||
dest: transcripts/final.trimmed.json
|
||||
required: true
|
||||
- from: artifacts/session_recap.md
|
||||
to: artifacts/session_recap.md
|
||||
- source: narratio.transcript.final_markdown
|
||||
dest: transcripts/final.md
|
||||
required: true
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
|
||||
scriptorium:
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
- source: narratio.transcript.final_trimmed_markdown
|
||||
dest: transcripts/final.trimmed.md
|
||||
required: true
|
||||
- source: narratio.artifact.session_recap
|
||||
dest: artifacts/session_recap.md
|
||||
required: true
|
||||
locks:
|
||||
- source: narratio.artifact.session_recap
|
||||
reason: manual post-publish edits
|
||||
```
|
||||
|
||||
Operational notes:
|
||||
Rules:
|
||||
|
||||
- archive promotion is explicit and path-based via `archive.promote_artifacts`.
|
||||
- Narratio does not auto-promote all generated analyze artifacts.
|
||||
- `outputs[].source` is required.
|
||||
- `outputs[].dest` may be omitted when derivable from source.
|
||||
- `outputs[].required` defaults to `true`.
|
||||
- static locks (`pipeline.publish.locks`) merge with remote locks (`{session_prefix}/locks.yml`), with static locks taking precedence on duplicates.
|
||||
|
||||
## 7. Full pipeline reference
|
||||
## Full Schema
|
||||
|
||||
| Path | Type | Required | Default |
|
||||
### Pipeline
|
||||
|
||||
| Field | Type | Required | Default / Rule |
|
||||
| --- | --- | --- | --- |
|
||||
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
|
||||
| `pipeline.workspace.cleanup_after_archive` | bool | No | `false` |
|
||||
| `pipeline.secrets.env_dir` | string | Conditional | none |
|
||||
| `pipeline.workspace.cleanup_after_publish` | bool | No | `false` |
|
||||
| `pipeline.campaigns.root` | string | No | `/usr/local/share/narratio/campaigns` |
|
||||
| `pipeline.campaigns.default_campaign_id` | string | No | empty |
|
||||
| `pipeline.secrets.env_dir` | string | No | empty |
|
||||
| `pipeline.storage.backend` | string | No | empty |
|
||||
| `pipeline.storage.bucket` | string | No | empty |
|
||||
| `pipeline.storage.prefix` | string | No | empty |
|
||||
| `pipeline.storage.s3.bucket` | string | Conditional | empty |
|
||||
| `pipeline.storage.s3.bucket` | string | Conditional | required for S3 session-audio and for publish upload when backend is `s3` |
|
||||
| `pipeline.storage.s3.root_prefix` | string | No | `dnd` |
|
||||
| `pipeline.storage.s3.region` | string | No | empty |
|
||||
| `pipeline.storage.s3.endpoint` | string | No | empty |
|
||||
@@ -151,21 +141,26 @@ Operational notes:
|
||||
| `pipeline.storage.s3.access_key_id_env` | string | No | `OBJECT_STORAGE_KEY_ID` |
|
||||
| `pipeline.storage.s3.secret_access_key_env` | string | No | `OBJECT_STORAGE_KEY` |
|
||||
| `pipeline.spool.root` | string | No | `/var/spool/narratio` |
|
||||
| `pipeline.spool.delete_audio_after_archive` | bool | No | `false` |
|
||||
| `pipeline.archive.enabled` | bool | No | `true` |
|
||||
| `pipeline.archive.upload_run` | bool | No | `true` |
|
||||
| `pipeline.archive.promote_artifacts[]` | list | No | trimmed + session_recap rules |
|
||||
| `pipeline.archive.promote_artifacts[].from` | string | Yes (per rule) | none |
|
||||
| `pipeline.archive.promote_artifacts[].to` | string | Yes (per rule) | none |
|
||||
| `pipeline.archive.promote_artifacts[].required` | bool | No | `true` |
|
||||
| `pipeline.whisperx.transcribe_url` | string | Yes | none |
|
||||
| `pipeline.spool.delete_audio_after_publish` | bool | No | `false` |
|
||||
| `pipeline.cache.root` | string | No | `/var/cache/narratio` |
|
||||
| `pipeline.cache.s3_audio` | bool | No | `true` |
|
||||
| `pipeline.publish.enabled` | bool | No | `true` |
|
||||
| `pipeline.publish.upload_run` | bool | No | `true` |
|
||||
| `pipeline.publish.outputs[]` | list | No | defaults to final trimmed JSON plus final and final-trimmed Markdown outputs |
|
||||
| `pipeline.publish.outputs[].source` | string | Yes (per rule) | must reference built-in or configured artifact source |
|
||||
| `pipeline.publish.outputs[].dest` | string | Conditional | derived if omitted and source supports derivation |
|
||||
| `pipeline.publish.outputs[].required` | bool | No | `true` |
|
||||
| `pipeline.publish.locks[]` | list | No | empty |
|
||||
| `pipeline.publish.locks[].source` | string | Yes (per lock) | must reference supported publish source |
|
||||
| `pipeline.publish.locks[].reason` | string | No | empty |
|
||||
| `pipeline.whisperx.transcribe_url` | string | Yes | valid URL |
|
||||
| `pipeline.whisperx.language` | string | No | `en` |
|
||||
| `pipeline.whisperx.timeout` | duration string | No | `30m` |
|
||||
| `pipeline.whisperx.timeout` | duration | No | `30m` |
|
||||
| `pipeline.whisperx.retries` | int | No | `3` |
|
||||
| `pipeline.whisperx.retry_delay` | duration string | No | `2s` |
|
||||
| `pipeline.whisperx.retry_delay` | duration | No | `2s` |
|
||||
| `pipeline.whisperx.concurrency` | int | No | `2` |
|
||||
| `pipeline.seriatim.binary` | string | No | `seriatim` |
|
||||
| `pipeline.seriatim.timeout` | duration string | No | `10m` |
|
||||
| `pipeline.seriatim.timeout` | duration | No | `10m` |
|
||||
| `pipeline.seriatim.output_schema` | string | No | `seriatim-intermediate` |
|
||||
| `pipeline.seriatim.coalesce_gap` | float | No | `3.0` |
|
||||
| `pipeline.seriatim.report` | bool | No | `true` |
|
||||
@@ -174,7 +169,7 @@ Operational notes:
|
||||
| `pipeline.seriatim.env.backchannel_max_duration` | float | No | unset |
|
||||
| `pipeline.seriatim.env.filler_max_duration` | float | No | unset |
|
||||
| `pipeline.audita.binary` | string | No | `audita` |
|
||||
| `pipeline.audita.timeout` | duration string | No | `3h` |
|
||||
| `pipeline.audita.timeout` | duration | No | `3h` |
|
||||
| `pipeline.audita.llm_api_key_env` | string | No | empty |
|
||||
| `pipeline.audita.modules[]` | list[string] | No | empty |
|
||||
| `pipeline.audita.base_url` | string | No | empty |
|
||||
@@ -188,117 +183,99 @@ Operational notes:
|
||||
| `pipeline.audita.output_schema` | string | No | empty |
|
||||
| `pipeline.audita.work_dir_retention` | string | No | empty |
|
||||
| `pipeline.audita.report` | bool | No | `true` |
|
||||
| `pipeline.normalize.output_path` | string | No | `transcripts/normalized.json` |
|
||||
| `pipeline.normalize.output_path` | string | No | `transcripts/final.json` |
|
||||
| `pipeline.normalize.output_schema` | string | No | `seriatim-intermediate` |
|
||||
| `pipeline.normalize.report` | bool | No | `true` |
|
||||
| `pipeline.trim.enabled` | bool | No | `false` |
|
||||
| `pipeline.trim.output_path` | string | Conditional | none |
|
||||
| `pipeline.trim.bounds.prompt_id` | string | Conditional | none |
|
||||
| `pipeline.trim.enabled` | bool | No | `true` |
|
||||
| `pipeline.trim.output_path` | string | No | `transcripts/final.trimmed.json` |
|
||||
| `pipeline.trim.bounds.prompt_id` | string | No | `dnd.session_bounds` |
|
||||
| `pipeline.trim.bounds.profile_id` | string | No | empty |
|
||||
| `pipeline.trim.bounds.transcript_input_name` | string | Conditional | none |
|
||||
| `pipeline.trim.bounds.output_path` | string | Conditional | none |
|
||||
| `pipeline.trim.bounds.timeout` | duration string | No | `10m` |
|
||||
| `pipeline.trim.bounds.transcript_input_name` | string | No | `transcript` |
|
||||
| `pipeline.trim.bounds.output_path` | string | No | `artifacts/session_bounds.json` |
|
||||
| `pipeline.trim.bounds.timeout` | duration | No | `10m` |
|
||||
| `pipeline.trim.bounds.render_debug` | bool | No | `false` |
|
||||
| `pipeline.trim.bounds.render_output_path` | string | Conditional | none |
|
||||
| `pipeline.trim.bounds.render_output_path` | string | Conditional | required when `render_debug` is true |
|
||||
| `pipeline.trim.seriatim.report` | bool | No | `false` |
|
||||
| `pipeline.render.enabled` | bool | No | `true` |
|
||||
| `pipeline.render.format` | string | No | `markdown` (only supported value) |
|
||||
| `pipeline.render.title` | string | No | empty (falls back to `session.title` when set) |
|
||||
| `pipeline.render.include_timestamps` | bool | No | `true` |
|
||||
| `pipeline.render.include_segment_ids` | bool | No | `true` |
|
||||
| `pipeline.render.include_metadata` | bool | No | `false` |
|
||||
| `pipeline.scriptorium.binary` | string | No | `scriptorium` |
|
||||
| `pipeline.scriptorium.config_path` | string | No | empty |
|
||||
| `pipeline.scriptorium.timeout` | duration string | No | `10m` |
|
||||
| `pipeline.scriptorium.timeout` | duration | No | `10m` |
|
||||
| `pipeline.scriptorium.render_debug` | bool | No | `false` |
|
||||
| `pipeline.scriptorium.artifacts` | map | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.enabled` | bool | No | `false` |
|
||||
| `pipeline.scriptorium.artifacts.<name>.depends_on[]` | list[string] | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.render_debug` | bool | No | unset |
|
||||
| `pipeline.scriptorium.artifacts.<name>.prompt_id` | string | Conditional | none |
|
||||
| `pipeline.scriptorium.artifacts.<name>.profile_id` | string | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.output_path` | string | Conditional | none |
|
||||
| `pipeline.scriptorium.artifacts.<name>.timeout` | duration string | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` | string | Conditional | none |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.artifact` | string | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.path` | string | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required` | bool | No | `false` |
|
||||
| `pipeline.scriptorium.artifacts.<name>.vars.<key>` | map value | No | empty |
|
||||
| `pipeline.analyzer.binary_path` | string | No | empty |
|
||||
| `pipeline.analyzer.timeout` | duration string | No | empty |
|
||||
| `pipeline.analyzer.artifacts.output_dir` | string | No | empty |
|
||||
| `pipeline.analyzer.artifacts.types[]` | list[string] | No | empty |
|
||||
| `pipeline.notification.backend` | string | No | empty |
|
||||
| `pipeline.notification.recipient` | string | No | empty |
|
||||
| `pipeline.notification.timeout` | duration string | No | empty |
|
||||
| `pipeline.notification.timeout` | duration | No | `30s` |
|
||||
|
||||
Scriptorium artifact-key and dependency rules:
|
||||
### Scriptorium Artifact Entries
|
||||
|
||||
- artifact keys must match `^[a-z][a-z0-9_]*$`.
|
||||
- enabled artifacts require `prompt_id` and `output_path`.
|
||||
- `output_path` must be relative, traversal-safe, and under `artifacts/`.
|
||||
- configured artifact input sources use `narratio.artifact.<name>`.
|
||||
- if input source references `narratio.artifact.<name>`, artifact `<name>` must exist and must be listed in `depends_on`.
|
||||
- every `depends_on` entry must be a configured artifact key.
|
||||
- self-dependency is rejected.
|
||||
- enabled dependency cycles are rejected.
|
||||
- any artifact referenced by `depends_on` or `narratio.artifact.<name>` source must define `output_path` (even if not enabled).
|
||||
For each `pipeline.scriptorium.artifacts.<name>`:
|
||||
|
||||
Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values:
|
||||
|
||||
- `previous_session_artifact`
|
||||
- `narratio.transcript.merged`
|
||||
- `narratio.transcript.polished`
|
||||
- `narratio.transcript.full`
|
||||
- `narratio.transcript.trimmed`
|
||||
- `narratio.bounds.session`
|
||||
- `narratio.artifact.<configured_artifact_key>`
|
||||
|
||||
## 8. Full session reference
|
||||
|
||||
| Path | Type | Required | Default |
|
||||
| Field | Type | Required | Rule |
|
||||
| --- | --- | --- | --- |
|
||||
| `session.session_id` | string | Yes | none |
|
||||
| `session.campaign` | string | Yes | none |
|
||||
| `session.date` | string | No | empty |
|
||||
| `session.title` | string | No | empty |
|
||||
| `session.inputs.audio_dir` | string | Conditional | empty |
|
||||
| `session.inputs.audio_files[]` | list[string] | Conditional | empty |
|
||||
| `session.inputs.audio_s3.prefix` | string | Conditional | none |
|
||||
| `session.inputs.speakers_file` | string | Yes | none |
|
||||
| `session.inputs.autocorrect_file` | string | Yes | none |
|
||||
| `session.inputs.glossary_file` | string | Yes | none |
|
||||
| `enabled` | bool | No | `false` if omitted |
|
||||
| `depends_on[]` | list[string] | No | must reference configured artifact keys; no self-reference; enabled graph must be acyclic |
|
||||
| `render_debug` | bool | No | per-artifact override |
|
||||
| `prompt_id` | string | Conditional | required when artifact is enabled |
|
||||
| `profile_id` | string | No | empty |
|
||||
| `output_path` | string | Conditional | required when enabled; also required when referenced by publish/output/input rules |
|
||||
| `timeout` | duration | No | artifact override |
|
||||
| `inputs` | map | No | input key names must be non-empty |
|
||||
| `vars` | map | No | values must be string or bool |
|
||||
|
||||
Audio-source rule:
|
||||
For each artifact input `pipeline.scriptorium.artifacts.<name>.inputs.<input_name>`:
|
||||
|
||||
- configure exactly one mode:
|
||||
- `audio_dir`, or
|
||||
- `audio_files` (at least one), or
|
||||
- `audio_s3.prefix`
|
||||
- `audio_s3` cannot be combined with local audio fields.
|
||||
| Field | Type | Required | Rule |
|
||||
| --- | --- | --- | --- |
|
||||
| `source` | string | Yes | built-in runtime source, prepared input source, `narratio.artifact.<name>`, or `narratio.previous_session.artifact.<name>` |
|
||||
| `artifact` | string | No | optional passthrough adapter field |
|
||||
| `path` | string | No | optional passthrough adapter field |
|
||||
| `required` | bool | No | optional input requirement |
|
||||
|
||||
## 9. Secrets
|
||||
### Campaign
|
||||
|
||||
Narratio supports filesystem-based secret injection via `pipeline.secrets.env_dir`.
|
||||
| Field | Type | Required | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| `campaign_id` | string | Yes | canonical campaign identity |
|
||||
| `session_template_file` | string | No | used by `session init` when set |
|
||||
| `inputs.speakers_file` | string | Yes | stable input default |
|
||||
| `inputs.autocorrect_file` | string | Yes | stable input default |
|
||||
| `inputs.glossary_file` | string | Yes | stable input default |
|
||||
| `inputs.players_file` | string | Yes | stable input default |
|
||||
| `inputs.party_file` | string | Yes | stable input default |
|
||||
|
||||
Behavior:
|
||||
### Session
|
||||
|
||||
- `env_dir` may be absolute or relative.
|
||||
- relative `env_dir` resolves from current working directory.
|
||||
- files with valid env-var names (`[A-Za-z_][A-Za-z0-9_]*`) are loaded.
|
||||
- values are loaded from file contents with trailing newline trimming.
|
||||
- existing process env vars are preserved.
|
||||
- invalid names and subdirectories are skipped.
|
||||
- missing/unreadable `env_dir` fails command execution.
|
||||
| Field | Type | Required in session file | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| `session_id` | string | Yes | must match CLI session target when provided |
|
||||
| `previous_session_id` | string | No | must not equal `session_id` |
|
||||
| `campaign` | string | No | filled from `campaign_id` during resolve if omitted |
|
||||
| `date` | string | No | metadata |
|
||||
| `title` | string | No | metadata |
|
||||
| `inputs.speakers_file` | string | No | overrides campaign stable input |
|
||||
| `inputs.autocorrect_file` | string | No | overrides campaign stable input |
|
||||
| `inputs.glossary_file` | string | No | overrides campaign stable input |
|
||||
| `inputs.players_file` | string | No | overrides campaign stable input |
|
||||
| `inputs.party_file` | string | No | overrides campaign stable input |
|
||||
| `inputs.audio_dir` | string | Conditional | local audio mode |
|
||||
| `inputs.audio_files[]` | list[string] | Conditional | local audio mode |
|
||||
| `inputs.audio_s3.prefix` | string | Conditional | S3 audio mode |
|
||||
|
||||
Guidance:
|
||||
Audio rules:
|
||||
|
||||
- do not put secret values directly in YAML.
|
||||
- configure env var names in config and provide values via env/secrets files.
|
||||
- configure local mode (`audio_dir` or `audio_files`) or S3 mode (`audio_s3.prefix`), not both.
|
||||
|
||||
## 10. Examples
|
||||
|
||||
Maintained examples:
|
||||
## Maintained Examples
|
||||
|
||||
- `examples/pipeline.minimal.yml`
|
||||
- `examples/pipeline.production.yml`
|
||||
- `examples/pipeline.full.annotated.yml`
|
||||
- `examples/session.template.yml`
|
||||
- `examples/campaigns/sample-campaign/campaign.yml`
|
||||
- `examples/session.local-audio.yml`
|
||||
- `examples/session.s3-audio.yml`
|
||||
|
||||
These examples are validated by `internal/config` tests.
|
||||
- `examples/session.template.yml`
|
||||
|
||||
@@ -1,15 +1,19 @@
|
||||
# Integration Documentation Index
|
||||
# Integrations Index
|
||||
|
||||
## Audience
|
||||
Developers and LLM coding agents changing Narratio's external integration contracts.
|
||||
Developers and coding agents changing Narratio's external integration boundaries.
|
||||
|
||||
## Scope
|
||||
Implemented-only reference notes for the external systems Narratio currently integrates with.
|
||||
`docs/integrations/` is the implementation-level reference for downstream tool adapter contracts.
|
||||
|
||||
## Integration Docs
|
||||
- `audita.md`: Audita adapter invocation and validation contract.
|
||||
- `seriatim.md`: Seriatim normalize/merge/trim adapter contract.
|
||||
- `scriptorium.md`: Scriptorium run/render adapter contract.
|
||||
These docs cover what Narratio expects from external tools and what each adapter guarantees back to stage code.
|
||||
|
||||
## Canonical Owner
|
||||
`docs/integrations/` is the canonical home for external integration reference notes per `docs/documentation/policy.md`.
|
||||
## Integration Contracts
|
||||
- `audita.md`: transcript polishing adapter (`audita process`).
|
||||
- `seriatim.md`: merge/normalize/trim/render adapter (`seriatim`).
|
||||
- `scriptorium.md`: artifact run/render adapter (`scriptorium run|render`).
|
||||
|
||||
## Related Canonical Docs
|
||||
- `docs/config.md`: operator-facing configuration reference.
|
||||
- `docs/internal/adapters.md`: shared adapter boundary and runner wiring.
|
||||
- `docs/internal/stage-*.md`: stage-specific integration usage.
|
||||
|
||||
@@ -1,66 +1,60 @@
|
||||
# Integration: audita
|
||||
# Integration: Audita
|
||||
|
||||
## Purpose
|
||||
Define Narratio's adapter contract for transcript polishing via Audita CLI subprocess execution.
|
||||
Define the Audita adapter contract used by the `polish` stage.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs (`audita.PolishRequest`):
|
||||
- merged transcript path
|
||||
- glossary path
|
||||
- output processed transcript path
|
||||
- optional report path (required when report enabled)
|
||||
- work dir
|
||||
- generated config path
|
||||
- stdout/stderr log paths
|
||||
- optional module/model/base URL and concurrency knobs
|
||||
## Adapter Boundary
|
||||
Interface:
|
||||
- `audita.Runner`
|
||||
- method: `Run(ctx, PolishRequest) (PolishResult, error)`
|
||||
|
||||
Outputs (`audita.PolishResult`):
|
||||
- processed transcript path
|
||||
- optional report path
|
||||
- generated config path
|
||||
- stdout/stderr log paths
|
||||
- exit code, duration, invoked binary
|
||||
- adapter metadata
|
||||
Primary implementation:
|
||||
- `internal/adapters/audita/SubprocessRunner`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Deterministic CLI argument construction for `audita process`
|
||||
- Environment bridging for API credentials
|
||||
- Invocation config emission
|
||||
- Output validation for processed transcript and report
|
||||
Execution mode:
|
||||
- subprocess invocation of `audita process`
|
||||
|
||||
Does not own:
|
||||
- Upstream/downstream stage orchestration
|
||||
- Credential sourcing policy beyond required env-var presence check
|
||||
## Request Contract
|
||||
`PolishRequest` carries:
|
||||
- required transcript/glossary/output/work-dir paths;
|
||||
- optional report path (required when report mode is enabled);
|
||||
- generated config and stdout/stderr log paths;
|
||||
- optional module/model/base-url/config/output-schema/concurrency settings.
|
||||
|
||||
## Config Fields Used
|
||||
Via `pipeline.audita.*` mapped in app/stage wiring:
|
||||
- `binary`, `timeout`, `llm_api_key_env`, `modules`, `base_url`, `model`
|
||||
- `transcript_description`, `config_path`, `output_schema`, `work_dir_retention`
|
||||
- `total_llm_concurrency`, `proposal_llm_concurrency`, `validation_model`, `validation_llm_concurrency`, `report`
|
||||
## Result Contract
|
||||
`PolishResult` returns:
|
||||
- processed transcript path;
|
||||
- optional report path;
|
||||
- work dir and generated-config/log paths;
|
||||
- exit code, duration, binary provenance;
|
||||
- adapter metadata map.
|
||||
|
||||
## External Adapters Used
|
||||
- Shared subprocess helper (`internal/adapters/subprocess`) to run CLI and capture logs.
|
||||
## Validation and Failure Semantics
|
||||
Construction fails for invalid static config values, including:
|
||||
- empty binary;
|
||||
- non-positive timeout;
|
||||
- invalid base URL;
|
||||
- invalid output schema;
|
||||
- invalid work-dir retention value;
|
||||
- invalid concurrency values.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- No direct manifest writes.
|
||||
- Stage-level metadata records adapter provenance and credential-present signal.
|
||||
- Generated invocation YAML is written when `GeneratedConfigPath` is provided.
|
||||
Run fails for:
|
||||
- missing required request paths;
|
||||
- missing required credential env var when configured (`llm_api_key_env`);
|
||||
- subprocess execution failure;
|
||||
- invalid processed transcript JSON (`segments` array required);
|
||||
- invalid report JSON when reporting is enabled.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Adapter has no skip/resume logic. Stage/runner controls this.
|
||||
Failure results still include output/log/config/exit metadata for diagnostics.
|
||||
|
||||
## Failure Behavior
|
||||
- Constructor validation fails on invalid binary/timeout/schema/concurrency/URL values.
|
||||
- Run fails on missing required paths, missing required credential env var, subprocess errors, invalid processed JSON shape, or invalid report JSON.
|
||||
- Failures preserve stdout/stderr paths in returned result metadata.
|
||||
## Deterministic Behavior
|
||||
- CLI args are built from runner config + request in a fixed order.
|
||||
- Generated invocation YAML (`audita.generated.v1`) is emitted when requested.
|
||||
- Manifest writes are stage-owned; adapter itself is stateless.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/adapters/audita/subprocess_test.go`
|
||||
- `internal/adapters/audita/fake_test.go`
|
||||
- `internal/stage/polish_test.go`
|
||||
## Config Mapping
|
||||
Config fields consumed through runner/stage wiring are under `pipeline.audita.*`.
|
||||
|
||||
## Architectural Invariants
|
||||
- Processed output must be valid JSON with top-level `segments` array.
|
||||
- When report is enabled, report output must be valid JSON.
|
||||
- If `llm_api_key_env` is configured, credential must be present in environment.
|
||||
Maintained example with Audita config:
|
||||
- `examples/pipeline.full.annotated.yml`
|
||||
- `examples/pipeline.production.yml`
|
||||
|
||||
@@ -1,64 +1,66 @@
|
||||
# Integration: scriptorium
|
||||
# Integration: Scriptorium
|
||||
|
||||
## Purpose
|
||||
Define Narratio's adapter contract for Scriptorium artifact generation and render-debug subprocess invocations.
|
||||
Define the Scriptorium adapter contract used by `analyze` and trim-bounds generation in `trim`.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `RunArtifactRequest`: binary, config path, prompt/profile IDs, input map, vars map, timeout, output path, logs/config paths, optional API env and working dir
|
||||
- `RenderArtifactRequest`: same core fields for render mode
|
||||
## Adapter Boundary
|
||||
Interface:
|
||||
- `scriptorium.Runner`
|
||||
- methods:
|
||||
- `RunArtifact(ctx, RunArtifactRequest)`
|
||||
- `RenderArtifact(ctx, RenderArtifactRequest)`
|
||||
|
||||
Outputs (`ArtifactResult`):
|
||||
- output path
|
||||
- stdout/stderr log paths
|
||||
- generated config path
|
||||
- exit code and duration
|
||||
- command mode (`run` or `render`)
|
||||
- prompt/profile provenance
|
||||
- validation failure signal
|
||||
- adapter metadata
|
||||
Primary implementation:
|
||||
- `internal/adapters/scriptorium/SubprocessRunner`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Deterministic CLI arg construction for `scriptorium run` and `scriptorium render`
|
||||
- Common request validation
|
||||
- Invocation config emission
|
||||
- Output existence/non-empty checks
|
||||
- Validation-failure mapping for run exit code 2
|
||||
Execution modes:
|
||||
- `scriptorium run`
|
||||
- `scriptorium render`
|
||||
|
||||
Does not own:
|
||||
- Artifact selection policy (`analyze` stage)
|
||||
- Bounds semantic validation (`trim` stage)
|
||||
## Request Contract
|
||||
Both request types carry:
|
||||
- binary/config/prompt/profile IDs;
|
||||
- input map and vars map;
|
||||
- output path;
|
||||
- timeout;
|
||||
- generated config + stdout/stderr log paths;
|
||||
- optional API-key env var name;
|
||||
- optional working directory.
|
||||
|
||||
## Config Fields Used
|
||||
Via `pipeline.scriptorium.*` and stage-level artifact config:
|
||||
- `binary`, `config_path`, `timeout`, `render_debug`
|
||||
- artifact-level `prompt_id`, `profile_id`, `timeout`, `inputs`, `vars`, `output_path`
|
||||
## Result Contract
|
||||
`ArtifactResult` returns:
|
||||
- output/log/generated-config paths;
|
||||
- exit code and duration;
|
||||
- command mode (`run` or `render`);
|
||||
- prompt/profile provenance;
|
||||
- `ValidationFailed` marker;
|
||||
- metadata map.
|
||||
|
||||
## External Adapters Used
|
||||
- Shared subprocess helper (`internal/adapters/subprocess`).
|
||||
## Validation and Failure Semantics
|
||||
Request validation fails for:
|
||||
- missing binary, prompt id, or output path;
|
||||
- non-positive timeout;
|
||||
- empty input/var names;
|
||||
- empty input path values;
|
||||
- missing required credential env var when `APIKeyEnv` is set.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- No direct manifest writes.
|
||||
- Stage metadata records adapter outputs and command mode.
|
||||
- Generated invocation YAML is written when requested.
|
||||
Run behavior:
|
||||
- subprocess errors propagate with context;
|
||||
- `run` exit code `2` is mapped to `ValidationFailed=true`;
|
||||
- successful subprocess still fails if output file is missing or empty.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Adapter has no skip/resume logic. Stage/runner controls execution.
|
||||
Render behavior:
|
||||
- subprocess errors propagate;
|
||||
- output file must exist and be non-empty.
|
||||
|
||||
## Failure Behavior
|
||||
- Request validation fails for missing binary/prompt/output, invalid timeout, invalid input/var names, or missing required API env var.
|
||||
- Subprocess errors bubble with command context.
|
||||
- `run` exit code 2 is treated as `ValidationFailed=true` and surfaced as error by calling stage.
|
||||
- Successful subprocess still fails if output file is missing/empty.
|
||||
## Deterministic Behavior
|
||||
- input and var maps are sorted into deterministic `--input` and `--var` CLI args.
|
||||
- generated invocation YAML (`scriptorium.generated.v1`) is emitted when requested.
|
||||
- adapter is stateless and does not own artifact-selection policy.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
- `internal/adapters/scriptorium/fake_test.go`
|
||||
- `internal/stage/analyze_test.go`
|
||||
- `internal/stage/trim_test.go`
|
||||
## Config Mapping
|
||||
Config fields consumed through runner/stage wiring are under `pipeline.scriptorium.*` plus per-artifact settings under `pipeline.scriptorium.artifacts.*`.
|
||||
|
||||
## Architectural Invariants
|
||||
- Both modes require explicit timeout > 0.
|
||||
- Input/var maps are sorted into deterministic CLI argument order.
|
||||
- Run-mode validation failures are represented explicitly, not silently skipped.
|
||||
Maintained examples with Scriptorium config:
|
||||
- `examples/pipeline.full.annotated.yml`
|
||||
- `examples/pipeline.production.yml`
|
||||
|
||||
@@ -1,60 +1,61 @@
|
||||
# Integration: seriatim
|
||||
# Integration: Seriatim
|
||||
|
||||
## Purpose
|
||||
Define Narratio's adapter contract for merge, normalize, and trim subprocess invocations of Seriatim.
|
||||
Define the Seriatim adapter contract used by `merge`, `normalize`, `trim`, and `render`.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `MergeRequest`: raw/normalized transcript inputs, output path, optional report, speaker/autocorrect paths, logs/config
|
||||
- `NormalizeRequest`: input transcript, output path, schema, optional report, timeout/log/config
|
||||
- `TrimRequest`: input transcript, output path, keep selector, timeout/log/config
|
||||
## Adapter Boundary
|
||||
Interface:
|
||||
- `seriatim.Runner`
|
||||
- methods:
|
||||
- `Run(ctx, MergeRequest)`
|
||||
- `Normalize(ctx, NormalizeRequest)`
|
||||
- `Trim(ctx, TrimRequest)`
|
||||
- `Render(ctx, RenderRequest)`
|
||||
|
||||
Outputs:
|
||||
- `MergeResult`, `NormalizeResult`, `TrimResult` with output paths, logs/config paths, exit code, duration, binary provenance, and metadata.
|
||||
Primary implementation:
|
||||
- `internal/adapters/seriatim/SubprocessRunner`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Validated deterministic CLI invocation construction
|
||||
- Optional env tuning propagation for merge
|
||||
- Invocation config file emission
|
||||
- JSON output validation
|
||||
Execution modes:
|
||||
- `seriatim merge`
|
||||
- `seriatim normalize`
|
||||
- `seriatim trim`
|
||||
- `seriatim render`
|
||||
|
||||
Does not own:
|
||||
- Transcript input selection/promotion logic (stage-owned)
|
||||
- Bounds computation (scriptorium/trim-stage-owned)
|
||||
## Request/Result Contracts
|
||||
- `MergeRequest`/`MergeResult`: multi-input merge to base transcript, optional report.
|
||||
- `NormalizeRequest`/`NormalizeResult`: transcript normalization with explicit schema.
|
||||
- `TrimRequest`/`TrimResult`: transcript trimming with required keep selector.
|
||||
- `RenderRequest`/`RenderResult`: transcript-to-markdown rendering with explicit format and render booleans.
|
||||
|
||||
## Config Fields Used
|
||||
Via `pipeline.seriatim.*` mapped in app/stage wiring:
|
||||
- `binary`, `timeout`, `output_schema`, `coalesce_gap`, `report`
|
||||
- `env.overlap_word_run_gap`
|
||||
- `env.overlap_word_run_reorder_window`
|
||||
- `env.backchannel_max_duration`
|
||||
- `env.filler_max_duration`
|
||||
Results include output/log/config paths, timing, exit code, and metadata.
|
||||
|
||||
## External Adapters Used
|
||||
- Shared subprocess helper (`internal/adapters/subprocess`).
|
||||
## Validation and Failure Semantics
|
||||
Runner construction validates:
|
||||
- binary presence;
|
||||
- timeout > 0;
|
||||
- supported output schema (`seriatim-minimal|seriatim-intermediate|seriatim-full`);
|
||||
- non-negative coalesce gap.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- No direct manifest writes.
|
||||
- Stage metadata consumes adapter result fields and preserves generated config/log references.
|
||||
Invocation fails on:
|
||||
- missing required request paths/inputs;
|
||||
- invalid normalize schema override;
|
||||
- unsupported render format;
|
||||
- subprocess failure;
|
||||
- invalid JSON outputs for merge/normalize/trim;
|
||||
- missing `segments` array for normalize/trim transcript outputs;
|
||||
- empty render output files.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Adapter has no skip/resume logic. Runner controls stage execution.
|
||||
When report paths are provided/enabled, report files must parse as JSON.
|
||||
|
||||
## Failure Behavior
|
||||
- Constructor fails for invalid binary/timeout/output-schema/coalesce-gap.
|
||||
- Merge fails on missing output path/inputs/report path (if enabled), subprocess errors, invalid merged output JSON, invalid report JSON.
|
||||
- Normalize fails on missing input/output, invalid schema, subprocess errors, invalid normalized output JSON shape, invalid report JSON.
|
||||
- Trim fails on missing input/output/keep selector, subprocess errors, invalid trimmed output JSON shape.
|
||||
## Deterministic Behavior
|
||||
- argument ordering is deterministic per command construction.
|
||||
- merge env overrides are explicit (`SERIATIM_*`) and only emitted when configured.
|
||||
- generated invocation YAML (`seriatim.generated.v1`) is emitted when requested.
|
||||
- adapter does not write manifests or choose stage inputs.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
- `internal/adapters/seriatim/fake_test.go`
|
||||
- `internal/stage/merge_test.go`
|
||||
- `internal/stage/normalize_test.go`
|
||||
- `internal/stage/trim_test.go`
|
||||
## Config Mapping
|
||||
Config fields consumed through runner/stage wiring are under `pipeline.seriatim.*` and `pipeline.render.*`.
|
||||
|
||||
## Architectural Invariants
|
||||
- Supported output schemas are limited to `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`.
|
||||
- Normalize/trim outputs must include `segments` arrays.
|
||||
- Merge/normalize/trim all route through deterministic subprocess invocation.
|
||||
Maintained examples with Seriatim config:
|
||||
- `examples/pipeline.full.annotated.yml`
|
||||
- `examples/pipeline.production.yml`
|
||||
|
||||
@@ -1,28 +1,45 @@
|
||||
# Internal Documentation Index
|
||||
|
||||
## Audience
|
||||
Developers and LLM coding agents changing Narratio internals.
|
||||
Developers and coding agents changing Narratio internals.
|
||||
|
||||
## Scope
|
||||
Implementation-accurate contracts for workspace/state, manifests, stages, artifact resolution, and adapter boundaries.
|
||||
`docs/internal/` documents implemented internal contracts: stage boundaries, manifest/state behavior, artifact resolution, restore behavior, storage boundaries, and workspace invariants.
|
||||
|
||||
## Component Docs
|
||||
- `adapters.md`: external adapter map, runtime wiring, and boundary ownership.
|
||||
- `storage.md`: remote storage backend contracts and object-store invariants.
|
||||
- `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics.
|
||||
- `artifacts.md`: built-in artifact registry, runtime artifact catalog, and source-resolution behavior.
|
||||
- `workspace.md`: local state model, manifests, run-local layout, promotion, and cleanup invariants.
|
||||
- `stage-prepare.md`: input materialization and provenance capture.
|
||||
- `stage-transcribe.md`: WhisperX transcript generation.
|
||||
- `stage-merge.md`: Seriatim normalization + merge.
|
||||
- `stage-polish.md`: Audita transcript polishing.
|
||||
- `stage-normalize.md`: post-polish normalization.
|
||||
- `stage-trim.md`: bounds-driven transcript trimming.
|
||||
- `stage-analyze.md`: dependency-ordered Scriptorium artifact generation for selected configured artifacts.
|
||||
- `stage-archive.md`: archive upload and current-pointer publish contract.
|
||||
User and operator behavior belongs in:
|
||||
- `docs/cli.md`
|
||||
- `docs/config.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/troubleshooting.md`
|
||||
|
||||
## External Integration Notes
|
||||
- `../integrations/README.md`: canonical location for external integration contracts (`audita.md`, `seriatim.md`, `scriptorium.md`).
|
||||
## Pipeline Stage Set
|
||||
Canonical stage order from `internal/stage.All()`:
|
||||
1. `prepare`
|
||||
2. `transcribe`
|
||||
3. `merge`
|
||||
4. `polish`
|
||||
5. `normalize`
|
||||
6. `trim`
|
||||
7. `render`
|
||||
8. `analyze`
|
||||
9. `publish`
|
||||
10. `notify` (placeholder)
|
||||
|
||||
## Canonical Owner
|
||||
`docs/internal/` is the canonical home for implemented internals per `docs/documentation/policy.md`.
|
||||
`notify` is currently a placeholder stage with optional notifier call behavior; it has no persisted pipeline outputs.
|
||||
|
||||
## Internal Component Docs
|
||||
- `adapters.md`: external adapter boundaries and default runtime wiring.
|
||||
- `artifacts.md`: canonical source IDs, runtime catalog behavior, and resolution rules.
|
||||
- `manifest.md`: session and run manifest contracts.
|
||||
- `storage.md`: object-store interface and S3 implementation behavior.
|
||||
- `workspace.md`: local session layout, run-local layout, and cleanup guardrails.
|
||||
- `command-restore.md`: restore discovery, planning, execution, and reporting.
|
||||
- `stage-prepare.md`
|
||||
- `stage-transcribe.md`
|
||||
- `stage-merge.md`
|
||||
- `stage-polish.md`
|
||||
- `stage-normalize.md`
|
||||
- `stage-trim.md`
|
||||
- `stage-render.md`
|
||||
- `stage-analyze.md`
|
||||
- `stage-publish.md`
|
||||
|
||||
@@ -1,79 +1,49 @@
|
||||
# Internal: Adapters
|
||||
|
||||
## Purpose
|
||||
Describe the external adapter boundaries used by Narratio stages and app orchestration, including default runtime wiring.
|
||||
Define external integration boundaries and default adapter wiring used by app/stage orchestration.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- Stage requests passed through adapter interfaces (for example transcription, merge/normalize/trim, polish, artifact generation, object-store operations, notifications).
|
||||
- Resolved config values used to construct default adapters.
|
||||
## Adapter Boundaries
|
||||
Narratio stage logic depends on adapter interfaces, not transport-specific details.
|
||||
|
||||
Outputs:
|
||||
- Adapter-specific result structs (paths, metadata, status/attempt info, duration/exit details).
|
||||
- Adapter errors returned to stage/app orchestration.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Transport/process/SDK details at system boundaries (`HTTP`, subprocess CLI invocation, AWS SDK calls).
|
||||
- Request/response contracts in `internal/adapters/*` packages.
|
||||
|
||||
Does not own:
|
||||
- Stage sequencing, skip/force/resume decisions.
|
||||
- Manifest transition logic.
|
||||
- Canonical workspace path policy.
|
||||
|
||||
## Config fields used
|
||||
Default wiring and adapter calls consume:
|
||||
- `pipeline.whisperx.*`
|
||||
- `pipeline.seriatim.*`
|
||||
- `pipeline.audita.*`
|
||||
- `pipeline.scriptorium.*`
|
||||
- `pipeline.storage.*` and `pipeline.archive.*` (object-store construction/gating)
|
||||
- `pipeline.notification.*` (sender boundary exists; placeholder behavior today)
|
||||
|
||||
## External adapters used
|
||||
Runtime env boundary fields (`internal/stage.Env`):
|
||||
Primary adapters:
|
||||
- `whisperx.Client`
|
||||
- `seriatim.Runner`
|
||||
- `audita.Runner`
|
||||
- `scriptorium.Runner`
|
||||
- `storage.ObjectStore`
|
||||
- `notify.Sender`
|
||||
- `analyzer.Runner`
|
||||
|
||||
Current execution usage:
|
||||
- Actively used by implemented stages: `WhisperX`, `Seriatim`, `Audita`, `Scriptorium`, `ObjectStore`, `Notifier`.
|
||||
- Present but not used by implemented stage set: `Analyzer`, legacy `storage.Backend`.
|
||||
## Ownership
|
||||
Adapters own:
|
||||
- HTTP/subprocess/SDK argument and transport details.
|
||||
- Backend-specific request/response mapping.
|
||||
|
||||
Default construction in app runner:
|
||||
- Auto-constructed when not injected: WhisperX HTTP client, Seriatim subprocess runner, Audita subprocess runner, Scriptorium subprocess runner, object store (only when needed), and `notify.NoopSender`.
|
||||
- Callers can inject test/fake implementations through `app.RunOptions.Env`.
|
||||
Adapters do not own:
|
||||
- stage ordering/skip/force logic;
|
||||
- manifest transitions;
|
||||
- canonical path policy.
|
||||
|
||||
## State and manifest behavior
|
||||
- Adapters do not directly mutate session/run manifests.
|
||||
- Stages and runner own manifest writes and stage status transitions.
|
||||
- Adapter outputs are persisted indirectly through stage result mapping (outputs/logs/generated configs/metadata).
|
||||
## Default Wiring
|
||||
`internal/app/runner.go` initializes default adapters when not injected:
|
||||
- WhisperX HTTP client from pipeline config.
|
||||
- Seriatim subprocess runner.
|
||||
- Audita subprocess runner.
|
||||
- Scriptorium subprocess runner.
|
||||
- Noop notifier (`notify.NoopSender`).
|
||||
- Object store only when required by selected stages/config.
|
||||
|
||||
## Skip and resume behavior
|
||||
- No adapter-level skip/resume semantics.
|
||||
- Skip/resume/force behavior is decided by app runner using manifest stage state.
|
||||
Object-store construction goes through `newCommandObjectStore`, which loads configured filesystem secrets before adapter initialization.
|
||||
|
||||
## Failure behavior
|
||||
- Adapter constructors validate config-derived values and fail early on invalid required inputs.
|
||||
- Adapter run-time failures are returned to stage code with boundary context and are recorded as stage failures by runner logic.
|
||||
- Subprocess adapters preserve stdout/stderr and generated-config paths to aid diagnosis.
|
||||
## Failure Semantics
|
||||
- Constructor errors fail stage execution setup early.
|
||||
- Runtime adapter errors propagate to stage code and then manifest failure handling.
|
||||
- Subprocess adapters persist stage logs/generated configs through stage-managed paths.
|
||||
|
||||
## Tests to inspect before changing
|
||||
## Test Surfaces
|
||||
- `internal/adapters/whisperx/http_test.go`
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
- `internal/adapters/audita/subprocess_test.go`
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
- `internal/adapters/storage/*_test.go`
|
||||
- `internal/adapters/notify/fake_test.go`
|
||||
- `internal/adapters/analyzer/fake_test.go`
|
||||
- `internal/app/runner_test.go`
|
||||
|
||||
## Architectural invariants
|
||||
- Stage code depends on adapter interfaces, not transport-specific implementation types.
|
||||
- External SDK-specific types remain inside adapter implementations.
|
||||
- Default app wiring must remain deterministic and overrideable via injected env dependencies.
|
||||
|
||||
@@ -1,87 +1,115 @@
|
||||
# Internal: Artifacts
|
||||
|
||||
## Purpose
|
||||
Define Narratio's artifact identity and resolution model for built-in transcript/bounds artifacts and runtime-configured analyze artifacts.
|
||||
Define canonical artifact IDs, runtime catalog behavior, source resolution rules, and shared current-state mechanics used by app and previous-cache code.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- artifact sources from config/runtime (`pipeline.scriptorium.artifacts.*.inputs.*.source`)
|
||||
- session paths and optional session manifest stage outputs
|
||||
- runtime artifact catalog state for configured artifact sources
|
||||
## Built-in Source IDs
|
||||
|
||||
Outputs:
|
||||
- resolved local artifact path and provenance (`ResolvedSessionArtifact`)
|
||||
- runtime catalog entries for planned/executable/available artifacts
|
||||
- validation errors for unsupported, missing, or invalid artifact sources
|
||||
- `narratio.transcript.base` -> `transcripts/base.json` (`merge`)
|
||||
- `narratio.transcript.polished` -> `transcripts/polished.json` (`polish`)
|
||||
- `narratio.transcript.final` -> `transcripts/final.json` (`normalize`)
|
||||
- `narratio.transcript.final_trimmed` -> `transcripts/final.trimmed.json` (`trim`)
|
||||
- `narratio.transcript.final_markdown` -> `transcripts/final.md` (`render`)
|
||||
- `narratio.transcript.final_trimmed_markdown` -> `transcripts/final.trimmed.md` (`render`)
|
||||
- `narratio.bounds.session` -> `artifacts/session_bounds.json` (`trim`)
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- built-in artifact registry and content validation rules
|
||||
- runtime artifact catalog for configured artifact source IDs
|
||||
- source resolution behavior for built-in and configured artifact sources
|
||||
## Configured and Previous-Session Sources
|
||||
|
||||
Does not own:
|
||||
- artifact generation (stages produce files)
|
||||
- manifest transition policy
|
||||
- archive promotion behavior
|
||||
- configured source ID format: `narratio.artifact.<artifact_key>`
|
||||
- previous-session source ID format: `narratio.previous_session.artifact.<artifact_key>`
|
||||
|
||||
## Config fields used
|
||||
- `pipeline.scriptorium.artifacts.<name>.enabled`
|
||||
- `pipeline.scriptorium.artifacts.<name>.output_path`
|
||||
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source`
|
||||
Both formats are validated by strict source-policy rules.
|
||||
|
||||
## External adapters used
|
||||
- none
|
||||
## Runtime Catalog
|
||||
|
||||
## State and manifest behavior
|
||||
Built-in registry entries:
|
||||
`ArtifactCatalog` tracks:
|
||||
|
||||
| Artifact ID | Canonical file | Producer stage | Output kind |
|
||||
| --- | --- | --- | --- |
|
||||
| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` |
|
||||
| `narratio.transcript.polished` | `transcripts/processed.json` | `polish` | `transcript_processed` |
|
||||
| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` |
|
||||
| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` |
|
||||
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` |
|
||||
- `planned`: source registered for run context;
|
||||
- `executable`: selected and enabled for analyze execution;
|
||||
- `available`: local file exists and validates;
|
||||
- `provenance`: availability source.
|
||||
|
||||
Runtime catalog entries include built-ins and configured `narratio.artifact.<name>` sources.
|
||||
Current provenance values:
|
||||
|
||||
Catalog states:
|
||||
- `planned`: source is registered and known for this run
|
||||
- `executable`: configured artifact is selected for analyze execution
|
||||
- `available`: artifact has a usable file path (generated this run or reused from disk)
|
||||
|
||||
Resolution behavior:
|
||||
- built-in sources resolve via manifest producer outputs first, then canonical fallback path
|
||||
- configured `narratio.artifact.<name>` sources resolve through runtime catalog availability
|
||||
- configured source lookup requires catalog context
|
||||
|
||||
Configured artifact provenance values:
|
||||
- `generated.current_analyze_run`
|
||||
- `filesystem.disabled_artifact_output`
|
||||
- `manifest.inputs.previous_cache`
|
||||
- `current_session.previous_cache`
|
||||
|
||||
Content validation:
|
||||
- transcript built-ins: JSON with top-level `segments` array
|
||||
- bounds built-in: valid JSON
|
||||
- configured artifacts: non-empty text file
|
||||
## Resolution Rules
|
||||
|
||||
## Skip and resume behavior
|
||||
- resolver and catalog have no direct skip/resume decisions
|
||||
- stage/runner skip-resume behavior consumes catalog/resolver results
|
||||
Built-ins:
|
||||
|
||||
## Failure behavior
|
||||
- unsupported source -> source validation error
|
||||
- known source unavailable -> `ErrSessionArtifactNotFound`
|
||||
- configured source without catalog -> resolution error
|
||||
- resolved file with invalid content -> validation error
|
||||
1. manifest producer outputs (when present)
|
||||
2. canonical session-path fallback
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/artifacts/artifact_resolver_test.go`
|
||||
- `internal/artifacts/catalog_test.go`
|
||||
- `internal/stage/analyze_test.go`
|
||||
- `internal/config/scriptorium_test.go`
|
||||
Configured sources (`narratio.artifact.*`):
|
||||
|
||||
## Architectural invariants
|
||||
- built-in IDs are static and registry-backed
|
||||
- configured artifact IDs are runtime-derived (`narratio.artifact.<name>`) and catalog-backed
|
||||
- built-in/source resolution remains deterministic and validation-gated
|
||||
- resolve only through runtime catalog availability.
|
||||
|
||||
Previous-session sources (`narratio.previous_session.artifact.*`):
|
||||
|
||||
- resolve only from local `previous/` cache state;
|
||||
- prefer manifest-backed previous-input paths;
|
||||
- fallback to existing previous-cache filesystem paths.
|
||||
|
||||
Validation by content type:
|
||||
|
||||
- transcript JSON built-ins: JSON with top-level `segments` array;
|
||||
- transcript Markdown built-ins: non-empty text file;
|
||||
- bounds built-in: valid JSON;
|
||||
- configured/previous-session artifact files: non-empty text file.
|
||||
|
||||
## Previous Requirement Collection
|
||||
|
||||
`CollectPreviousArtifactRequirements`:
|
||||
|
||||
- scans enabled configured artifacts only;
|
||||
- extracts only canonical previous-session sources;
|
||||
- deduplicates by artifact key;
|
||||
- merges required and optional references (required wins);
|
||||
- returns deterministic ordering and source locations.
|
||||
|
||||
## Current-State Helpers
|
||||
|
||||
Artifacts package owns shared remote current-state loading mechanics used by restore, status/validate checks, and previous-cache planning.
|
||||
|
||||
Core helpers:
|
||||
|
||||
- `LoadCurrentRunPointer`
|
||||
- `LoadCurrentManifest`
|
||||
- `LoadCurrentState`
|
||||
- `ValidateCurrentStateIdentity`
|
||||
|
||||
Typed missing-state errors:
|
||||
|
||||
- `CurrentRunPointerMissingError` (`ErrCurrentRunPointerMissing`)
|
||||
- `CurrentManifestMissingError` (`ErrCurrentManifestMissing`)
|
||||
|
||||
Identity validation supports caller-provided expectations:
|
||||
|
||||
- expected campaign;
|
||||
- expected session ID;
|
||||
- expected run ID, or pointer/manifest run-ID consistency check.
|
||||
|
||||
Caller policy is intentionally outside artifacts helpers:
|
||||
|
||||
- some callers fail on missing current state;
|
||||
- some callers downgrade missing state to status/findings;
|
||||
- some callers skip optional behavior when state is missing.
|
||||
|
||||
## Key Path Helpers
|
||||
|
||||
`internal/artifacts/paths.go` and S3-key helpers define canonical helpers for:
|
||||
|
||||
- session/work/run paths;
|
||||
- previous-cache paths;
|
||||
- spool/cache paths;
|
||||
- S3 session/run/current-state key layout.
|
||||
|
||||
## Invariants
|
||||
|
||||
- source ID formats are stable contracts;
|
||||
- artifact resolution is deterministic and manifest-aware;
|
||||
- previous-session source resolution in `analyze` is local-only;
|
||||
- remote current-state key construction remains centralized in artifacts helpers.
|
||||
|
||||
84
docs/internal/command-restore.md
Normal file
84
docs/internal/command-restore.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# Internal: Command Restore
|
||||
|
||||
## Purpose
|
||||
Define the implemented `narratio session restore` command contract:
|
||||
|
||||
- committed remote current-state discovery;
|
||||
- deterministic restore planning;
|
||||
- safe local install semantics;
|
||||
- durable restore reporting.
|
||||
|
||||
## Discovery Contract
|
||||
|
||||
Restore resolves remote committed state from the session publish current pointers:
|
||||
|
||||
- `current/run_id.txt` (required, non-empty);
|
||||
- `current/manifest.json` (required, decodable).
|
||||
|
||||
Current-state discovery uses shared artifacts-level mechanics and validates identity against the resolved request config:
|
||||
|
||||
- campaign must match;
|
||||
- session ID must match.
|
||||
|
||||
Restore treats any missing or invalid remote current state as a command error.
|
||||
|
||||
## Planning Contract
|
||||
|
||||
Restore planner action kinds:
|
||||
|
||||
- `download`;
|
||||
- `skip_same`;
|
||||
- `conflict`.
|
||||
|
||||
Planner behavior:
|
||||
|
||||
- remote list scope is the resolved session prefix;
|
||||
- remote-to-local mapping is traversal-safe;
|
||||
- actions are sorted deterministically by local relative path.
|
||||
|
||||
Restore scope from current remote state:
|
||||
|
||||
- include `manifest.json`;
|
||||
- include `transcripts/**`;
|
||||
- include `artifacts/**`;
|
||||
- include `audio/**` only with `--include-audio`.
|
||||
|
||||
Explicit exclusions from current remote state mapping:
|
||||
|
||||
- `current/**`;
|
||||
- `runs/**`;
|
||||
- `logs/**`;
|
||||
- `reports/**`;
|
||||
- `config/**`;
|
||||
- `inputs/**`;
|
||||
- `previous/**`.
|
||||
|
||||
Previous-cache files are planned separately through `previouscache.BuildPlan` when configured previous-session requirements exist.
|
||||
|
||||
## Execution Contract
|
||||
|
||||
Execution order and safety:
|
||||
|
||||
- non-manifest downloads happen before manifest install;
|
||||
- `manifest.json` installs last;
|
||||
- downloads use sibling temp files plus atomic rename;
|
||||
- manifest replacement is validated before rename;
|
||||
- failed installs do not roll back files already written in the same execution.
|
||||
|
||||
Audio restore path:
|
||||
|
||||
- uses `audio.MaterializeS3Audio`;
|
||||
- integrates spool and S3 audio cache paths;
|
||||
- supports cache-hit reuse without object redownload.
|
||||
|
||||
## Reporting Contract
|
||||
|
||||
- `--dry-run`: prints summary only; no local writes.
|
||||
- non-dry-run: writes `reports/restore-latest.json`.
|
||||
- report includes plan counts, per-action status, and execution failures.
|
||||
|
||||
## Invariants
|
||||
|
||||
- restore uses committed remote current state as authority;
|
||||
- `current/run_id.txt` is the remote publish commit marker;
|
||||
- restore does not execute pipeline stages.
|
||||
@@ -1,81 +1,57 @@
|
||||
# Internal: Manifest
|
||||
|
||||
## Purpose
|
||||
Describe Narratio's durable execution state model for session-level and run-level manifests, including lifecycle transitions and persistence behavior.
|
||||
Define durable session state (`manifest.json`) and invocation state (`runs/{run_id}/manifest.json`) contracts.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- Session identity and run identity from app orchestration.
|
||||
- Stage transition events and stage result payloads.
|
||||
## Session Manifest
|
||||
Path:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/manifest.json`
|
||||
|
||||
Outputs:
|
||||
- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`.
|
||||
- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`.
|
||||
Primary model (`manifest.Manifest`):
|
||||
- identity (`session_id`, `campaign`, `run_id`)
|
||||
- local path metadata (`local_workdir`, `local_spool_dir`)
|
||||
- remote identity metadata (`s3_bucket`, `s3_session_prefix`, `s3_run_prefix`)
|
||||
- `inputs` records
|
||||
- durable `artifacts` records
|
||||
- per-stage `stages` map
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Manifest schemas (`Manifest`, `RunManifest`, stage records, error records, input/artifact records).
|
||||
- Stage status/action transition methods.
|
||||
- Persistent store contract (`manifest.Store`) and local JSON store implementation.
|
||||
Stage status enum:
|
||||
- `pending`
|
||||
- `running`
|
||||
- `succeeded`
|
||||
- `failed`
|
||||
- `skipped`
|
||||
- `stale`
|
||||
- `interrupted`
|
||||
|
||||
Does not own:
|
||||
- Stage implementation details.
|
||||
- Path construction policy outside manifest file persistence calls.
|
||||
- CLI command behavior.
|
||||
## Run Manifest
|
||||
Path:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`
|
||||
|
||||
## Config fields used
|
||||
Manifest package itself does not read config directly.
|
||||
Run model (`manifest.RunManifest`):
|
||||
- invocation identity and `force` flag
|
||||
- requested stages
|
||||
- per-stage action (`run` or `skip`)
|
||||
- per-stage status
|
||||
- overall run status (`running`, `succeeded`, `failed`)
|
||||
|
||||
Manifest identity fields are populated by app/stage orchestration from:
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.storage.s3.*` (when archive/S3 identity is set)
|
||||
## Persistence Semantics
|
||||
`manifest.LocalStore`:
|
||||
- validates loaded documents;
|
||||
- normalizes missing maps/stage records;
|
||||
- writes atomically via temp file + rename;
|
||||
- updates `updated_at` on save.
|
||||
|
||||
## External adapters used
|
||||
- No external service adapters.
|
||||
- Uses local filesystem for persistence via `manifest.LocalStore`.
|
||||
## Execution Semantics
|
||||
Runner updates both manifests per stage transition:
|
||||
- mark running
|
||||
- mark succeeded/failed/skipped
|
||||
- persist logs/generated config refs and metadata
|
||||
|
||||
## State and manifest behavior
|
||||
Session manifest model:
|
||||
- Tracks durable per-session stage state and provenance (`pending`, `running`, `succeeded`, `failed`, `skipped`, `stale`, `interrupted`).
|
||||
- Stores resolved inputs, durable artifacts, stage logs/config refs, and stage metadata.
|
||||
Session manifest is the authoritative stage-progress ledger across invocations.
|
||||
Run manifest is invocation-scoped audit state.
|
||||
|
||||
Run manifest model:
|
||||
- Tracks one invocation (`run_id`) with requested stages and force mode.
|
||||
- Tracks per-stage action (`run` or `skip`) and per-stage status.
|
||||
- Tracks overall run status (`running`, `succeeded`, `failed`).
|
||||
|
||||
Persistence behavior:
|
||||
- Load validates required identity/timestamp fields and normalizes maps/records.
|
||||
- Save updates `updated_at` and writes JSON atomically (temp file + rename).
|
||||
- Session and run manifests are saved incrementally before/after stage transitions.
|
||||
|
||||
Relationship during execution:
|
||||
- Runner updates both manifests for every stage transition.
|
||||
- Session manifest is the durable pipeline-progress ledger.
|
||||
- Run manifest is invocation history and audit record.
|
||||
- Analyze stage outputs are persisted as `kind=scriptorium_artifact` with `source_id=narratio.artifact.<name>` for configured artifact identity.
|
||||
|
||||
## Skip and resume behavior
|
||||
- Resume and skip decisions are based on session-manifest stage statuses.
|
||||
- `--force` reruns selected stages and marks downstream succeeded stages as `stale` in session manifest.
|
||||
- Run manifest records whether each stage was executed or skipped in that invocation.
|
||||
|
||||
## Failure behavior
|
||||
- Stage failure marks both manifests failed for that stage and records error messages/timestamps.
|
||||
- Save failures are returned immediately and fail the command.
|
||||
- Invalid/malformed manifest files fail load with explicit validation/decode errors.
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/manifest/manifest_test.go`
|
||||
- `internal/manifest/run_manifest_test.go`
|
||||
- `internal/manifest/store_test.go`
|
||||
- `internal/app/runner_test.go`
|
||||
- `internal/app/run_control_test.go`
|
||||
- `internal/app/resume_run_stage_test.go`
|
||||
|
||||
## Architectural invariants
|
||||
- Session manifest is authoritative for stage progression across invocations.
|
||||
- Run manifest is invocation-scoped and never replaces session manifest as progress authority.
|
||||
- Manifest writes are atomic and deterministic (JSON + newline, temp rename pattern).
|
||||
## Invariants
|
||||
- stage resume/skip decisions are session-manifest driven.
|
||||
- force reruns stale downstream succeeded stages.
|
||||
- run manifest does not replace session manifest as progress authority.
|
||||
|
||||
@@ -1,84 +1,42 @@
|
||||
# Stage: analyze
|
||||
|
||||
## Purpose
|
||||
Execute selected configured Scriptorium artifacts in deterministic dependency order and promote successful outputs to canonical session artifact paths.
|
||||
Execute selected configured Scriptorium artifacts in dependency order and materialize outputs.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- configured artifact definitions from `pipeline.scriptorium.artifacts`
|
||||
- selected artifact filter from runtime (`--artifacts`) when provided
|
||||
- resolved artifact input sources declared per artifact (`inputs.*.source`)
|
||||
- optional previous-session file inputs (`previous_session_artifact`)
|
||||
## Inputs
|
||||
- configured artifacts from `pipeline.scriptorium.artifacts`
|
||||
- optional selected artifact filter (`--artifacts`)
|
||||
- built-in/configured/previous-session source references in artifact inputs
|
||||
|
||||
Outputs:
|
||||
- one promoted output file per executed configured artifact at that artifact's configured `output_path`
|
||||
- stage metadata containing generated artifact entries and reused disabled-artifact entries
|
||||
Supported source families:
|
||||
- built-ins: `narratio.transcript.*`, `narratio.bounds.session`
|
||||
- prepared stable inputs: `narratio.input.players`, `narratio.input.party`, `narratio.input.glossary`
|
||||
- configured artifacts: `narratio.artifact.<key>`
|
||||
- previous-session cache: `narratio.previous_session.artifact.<key>`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- runtime artifact catalog construction for analyze execution
|
||||
- selected-artifact planning and dependency ordering
|
||||
- per-artifact input resolution, var resolution, timeout/render-debug resolution
|
||||
- Scriptorium run/render invocation for each selected artifact
|
||||
- run-local output generation and canonical promotion
|
||||
## Outputs
|
||||
- one materialized output per executed configured artifact (`output_path`)
|
||||
- stage metadata describing selected/generated/reused artifacts
|
||||
|
||||
Does not own:
|
||||
- transcript generation/processing stages
|
||||
- archive promotion policy
|
||||
- per-artifact resume semantics
|
||||
## Key Behavior
|
||||
- skips with metadata when Scriptorium config is missing or no executable artifacts remain.
|
||||
- builds runtime artifact catalog (built-ins + configured artifacts).
|
||||
- marks non-executable configured artifacts as reusable when output files already exist.
|
||||
- validates selected artifact dependency order (cycle-safe topo ordering).
|
||||
- resolves required/optional inputs per artifact source definition.
|
||||
- resolves prepared stable input sources from `inputs/*.yml` materialized by `prepare`.
|
||||
- resolves previous-session sources from local `previous/` cache only.
|
||||
- runs optional render-debug, then artifact execution.
|
||||
- validates non-empty output files and materializes canonical outputs.
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.scriptorium.binary`
|
||||
- `pipeline.scriptorium.config_path`
|
||||
- `pipeline.scriptorium.timeout`
|
||||
- `pipeline.scriptorium.render_debug`
|
||||
- `pipeline.scriptorium.artifacts.<name>.*`
|
||||
- `enabled`
|
||||
- `depends_on`
|
||||
- `prompt_id`
|
||||
- `profile_id`
|
||||
- `timeout`
|
||||
- `output_path`
|
||||
- `render_debug`
|
||||
- `inputs`
|
||||
- `vars`
|
||||
## Failure Semantics
|
||||
- required missing configured/previous-session inputs fail.
|
||||
- missing required prepared stable input source includes prepare rerun guidance.
|
||||
- missing required previous-session source includes prepare rerun guidance.
|
||||
- missing required `narratio.transcript.final_markdown` or `narratio.transcript.final_trimmed_markdown` inputs includes render rerun guidance.
|
||||
- dependency cycles or unavailable required dependencies fail.
|
||||
- adapter validation failures fail stage.
|
||||
|
||||
## External Adapters Used
|
||||
- Scriptorium adapter:
|
||||
- optional `RenderArtifact` (render debug)
|
||||
- `RunArtifact` (artifact generation)
|
||||
|
||||
## State and Manifest Behavior
|
||||
- If `pipeline.scriptorium` is absent, stage returns success metadata with `skipped=true`.
|
||||
- If no artifacts are configured, stage returns success metadata with `skipped=true`.
|
||||
- If zero artifacts are executable after `enabled` + `--artifacts` filtering, stage returns success metadata with `skipped=true`.
|
||||
- Builds runtime catalog with built-ins and configured artifacts.
|
||||
- Non-executable configured artifacts are marked available only when their configured output file exists and is valid on disk.
|
||||
- Executes selected configured artifacts in topological order with deterministic tie-breaking.
|
||||
- For each generated artifact, records metadata fields including `name`, `source_id`, `output_kind`, `path`, `prompt_id`, `profile_id`, and `provenance`.
|
||||
- Reused disabled artifacts are recorded separately in `reused_artifacts` with provenance `filesystem.disabled_artifact_output`.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when analyze is already `succeeded` and `--force` is not set.
|
||||
- Analyze remains stage-scoped for resume/skip; there is no per-artifact resume state.
|
||||
- `--artifacts` filters which configured artifacts are executable when analyze runs; it does not imply `--force`.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on invalid dependency ordering, unavailable required configured inputs, invalid built-in input prerequisites, render/run adapter failures, validation-failed adapter results, or missing/empty outputs.
|
||||
- Required configured dependency missing from catalog availability fails clearly before invocation.
|
||||
- Optional missing inputs are omitted.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/analyze_test.go`
|
||||
- `internal/artifacts/catalog_test.go`
|
||||
- `internal/artifacts/artifact_resolver_test.go`
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Configured artifacts are identified by `narratio.artifact.<name>` source IDs.
|
||||
- Artifact-to-artifact references rely on explicit `depends_on` declarations validated in config.
|
||||
- Generated analyze outputs are treated uniformly as Scriptorium artifacts.
|
||||
- Successful outputs must exist and be non-empty before promotion.
|
||||
## Invariants
|
||||
- `analyze` performs no remote storage calls for previous-session source resolution.
|
||||
- output provenance and metadata are deterministic per execution.
|
||||
|
||||
@@ -1,68 +0,0 @@
|
||||
# Stage: archive
|
||||
|
||||
## Purpose
|
||||
Publish run records and promoted session artifacts to object storage, then atomically advance the remote current pointer.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- session manifest and prerequisite stage records
|
||||
- run root contents under `runs/{run_id}/`
|
||||
- promotion sources from session root (`archive.promote_artifacts`)
|
||||
|
||||
Outputs:
|
||||
- uploaded run files under `{session_prefix}/runs/{run_id}/...`
|
||||
- uploaded promoted artifacts under `{session_prefix}/...`
|
||||
- `{session_prefix}/current/manifest.json`
|
||||
- `{session_prefix}/current/run_id.txt` written last
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Archive enable/disable gate behavior
|
||||
- Prerequisite stage success enforcement
|
||||
- Run file collection and upload (excluding `audio/`)
|
||||
- Promotion rule resolution and upload
|
||||
- Commit pointer publish order
|
||||
|
||||
Does not own:
|
||||
- Stage execution before archive
|
||||
- Post-archive local cleanup policy execution (handled by app cleanup logic)
|
||||
|
||||
## Config Fields Used
|
||||
- `pipeline.archive.enabled`
|
||||
- `pipeline.archive.upload_run`
|
||||
- `pipeline.archive.promote_artifacts`
|
||||
- `pipeline.storage.s3.bucket`
|
||||
- `pipeline.storage.s3.root_prefix`
|
||||
- `pipeline.workspace.root`
|
||||
- `session.campaign`
|
||||
- `session.session_id`
|
||||
|
||||
## External Adapters Used
|
||||
- Object storage backend (`env.ObjectStore`) for upload/list primitives.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
|
||||
- Resolves bucket/prefix from manifest identity first, then config fallback.
|
||||
- Writes metadata including:
|
||||
- upload counts/paths
|
||||
- `current_manifest_key`
|
||||
- `current_run_id_key`
|
||||
- `current_pointer_written`
|
||||
- On skipped archive path, returns metadata with `skipped=true` and pointer not written.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Stage may self-skip (metadata skip) when archive disabled or run upload disabled.
|
||||
- Runner-level skip also applies for previously succeeded stage unless forced.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing prerequisite success, missing object store when required, missing run root, missing required promotion source, upload failures, or pointer write failures.
|
||||
- Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/archive_test.go`
|
||||
- `internal/app/post_archive_cleanup_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Run upload excludes `audio/` subtree.
|
||||
- `current/manifest.json` uploads before `current/run_id.txt`.
|
||||
- `current/run_id.txt` is the remote publish commit marker.
|
||||
@@ -1,63 +1,25 @@
|
||||
# Stage: merge
|
||||
|
||||
## Purpose
|
||||
Normalize per-speaker raw transcripts and merge them into one merged transcript via Seriatim.
|
||||
Normalize raw transcript inputs and merge into base transcript via Seriatim.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
## Inputs
|
||||
- `transcripts/raw/*.json`
|
||||
- `inputs/speakers.yml`
|
||||
- `inputs/autocorrect.yml`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/merged.json`
|
||||
- optional `artifacts/seriatim.report.json` (when report enabled)
|
||||
## Outputs
|
||||
- `transcripts/base.json`
|
||||
- optional `artifacts/seriatim.report.json`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Raw transcript discovery/validation
|
||||
- Per-input normalize calls to Seriatim
|
||||
- Final merge call to Seriatim
|
||||
- Run-local log/config/report path wiring
|
||||
- Promotion of merged/report outputs to canonical paths
|
||||
## Key Behavior
|
||||
- discovers and validates raw transcript inputs.
|
||||
- normalizes each raw transcript (`seriatim.Normalize`) into run-local scratch output.
|
||||
- merges normalized inputs (`seriatim.Run`) into base transcript.
|
||||
- validates merged transcript and optional report JSON.
|
||||
- materializes canonical outputs and records stage logs/generated configs.
|
||||
|
||||
Does not own:
|
||||
- Transcript polishing or downstream artifact generation
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.seriatim.binary`
|
||||
- `pipeline.seriatim.timeout`
|
||||
- `pipeline.seriatim.output_schema`
|
||||
- `pipeline.seriatim.coalesce_gap`
|
||||
- `pipeline.seriatim.report`
|
||||
- `pipeline.seriatim.env.*`
|
||||
|
||||
## External Adapters Used
|
||||
- Seriatim adapter:
|
||||
- `Normalize` for each raw input
|
||||
- `Run` for final merge
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads transcript inputs from transcribe stage outputs in manifest when present; falls back to canonical raw directory.
|
||||
- Writes run-local outputs/logs/config under `runs/{run_id}/merge/...` when enabled.
|
||||
- Promotes canonical merged transcript and optional report.
|
||||
- Records normalized-input provenance and adapter metadata in stage metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced rerun of this or upstream stages can stale downstream succeeded stages via runner invalidation.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid raw transcripts, missing speakers/autocorrect files, normalize failure, merge failure, invalid merged output JSON, or invalid report JSON when enabled.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/merge_test.go`
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Merge consumes normalized forms of each raw transcript.
|
||||
- Merged transcript must validate before promotion.
|
||||
- Report output is optional and gated by config.
|
||||
## Invariants
|
||||
- merge always consumes normalized forms of raw inputs.
|
||||
- base transcript must validate before stage success.
|
||||
- report output is config-gated.
|
||||
|
||||
@@ -1,56 +1,22 @@
|
||||
# Stage: normalize
|
||||
|
||||
## Purpose
|
||||
Normalize the processed transcript into a deterministic intermediate schema for trim and optionally emit a normalize report.
|
||||
Normalize polished transcript into final transcript using Seriatim.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `transcripts/processed.json`
|
||||
## Inputs
|
||||
- `transcripts/polished.json`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/normalized.json` (or configured normalize output path)
|
||||
## Outputs
|
||||
- `transcripts/final.json` (or configured normalize output path)
|
||||
- optional `artifacts/seriatim.normalize.report.json`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Processed transcript discovery/validation
|
||||
- Normalize request construction and invocation
|
||||
- Optional normalize report wiring
|
||||
- Promotion of normalized transcript and optional report
|
||||
## Key Behavior
|
||||
- resolves polished transcript from manifest outputs/canonical fallback.
|
||||
- applies `pipeline.normalize` config or default normalize config.
|
||||
- runs Seriatim normalize with configured timeout/binary.
|
||||
- validates normalized transcript and optional report.
|
||||
- materializes canonical outputs and records logs/generated configs.
|
||||
|
||||
Does not own:
|
||||
- Bounds detection or segment trimming
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.normalize.output_path`
|
||||
- `pipeline.normalize.output_schema`
|
||||
- `pipeline.normalize.report`
|
||||
- `pipeline.seriatim.binary`
|
||||
- `pipeline.seriatim.timeout`
|
||||
|
||||
## External Adapters Used
|
||||
- Seriatim adapter (`Normalize`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads processed transcript from polish outputs in manifest when present; falls back to canonical path.
|
||||
- Uses run-local output/report/log/config paths when run layout is enabled.
|
||||
- Promotes canonical normalized transcript and optional normalize report.
|
||||
- Records adapter/result metadata including source path selection.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced reruns can stale downstream succeeded stages.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid processed transcript, adapter error, invalid normalized output, or invalid report output when report enabled.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/normalize_test.go`
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Normalized output must validate as processed-transcript-compatible JSON (`segments` array required).
|
||||
- Default normalize config is applied when `pipeline.normalize` is unset.
|
||||
## Invariants
|
||||
- final transcript must validate as processed transcript JSON (`segments` array).
|
||||
- normalize defaults are applied when `pipeline.normalize` is unset.
|
||||
|
||||
@@ -1,69 +1,23 @@
|
||||
# Stage: polish
|
||||
|
||||
## Purpose
|
||||
Polish merged transcript with Audita and produce a processed transcript for downstream normalization/analyze.
|
||||
Run Audita polishing on base transcript and produce polished transcript.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `transcripts/merged.json`
|
||||
## Inputs
|
||||
- `transcripts/base.json`
|
||||
- `inputs/glossary.yml`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/processed.json`
|
||||
- optional `artifacts/audita.report.json` (when report enabled)
|
||||
## Outputs
|
||||
- `transcripts/polished.json`
|
||||
- optional `artifacts/audita.report.json`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Merged transcript discovery/validation
|
||||
- Audita invocation request construction
|
||||
- Run-local logs/config/work-dir/report wiring
|
||||
- Promotion of processed transcript and optional report
|
||||
## Key Behavior
|
||||
- resolves base transcript from merge outputs/canonical fallback.
|
||||
- invokes Audita with configured model/module/runtime options.
|
||||
- validates processed transcript structure (`segments` array required).
|
||||
- validates optional report JSON.
|
||||
- materializes canonical outputs; records logs/generated config and adapter metadata.
|
||||
|
||||
Does not own:
|
||||
- Upstream merge normalization
|
||||
- Downstream normalize/trim/analyze logic
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.audita.binary`
|
||||
- `pipeline.audita.timeout`
|
||||
- `pipeline.audita.llm_api_key_env`
|
||||
- `pipeline.audita.modules`
|
||||
- `pipeline.audita.base_url`
|
||||
- `pipeline.audita.model`
|
||||
- `pipeline.audita.transcript_description`
|
||||
- `pipeline.audita.config_path`
|
||||
- `pipeline.audita.output_schema`
|
||||
- `pipeline.audita.work_dir_retention`
|
||||
- `pipeline.audita.total_llm_concurrency`
|
||||
- `pipeline.audita.proposal_llm_concurrency`
|
||||
- `pipeline.audita.validation_model`
|
||||
- `pipeline.audita.validation_llm_concurrency`
|
||||
- `pipeline.audita.report`
|
||||
|
||||
## External Adapters Used
|
||||
- Audita adapter (`env.Audita.Run`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads merged transcript from merge manifest outputs when available; falls back to canonical merged path.
|
||||
- Uses run-local output/report/log/config/scratch paths when run layout is enabled.
|
||||
- Promotes canonical `transcripts/processed.json` and optional report.
|
||||
- Records adapter invocation metadata, credential presence signal, and output provenance in stage metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced rerun can stale downstream succeeded stages via runner invalidation.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid merged transcript, missing glossary, adapter error, invalid processed output shape (`segments` array required), or invalid report JSON when enabled.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/polish_test.go`
|
||||
- `internal/adapters/audita/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Processed transcript must contain a top-level `segments` array.
|
||||
- Report behavior is strictly config-gated.
|
||||
- Stage output canonicalization always ends at `transcripts/processed.json`.
|
||||
## Invariants
|
||||
- polished transcript schema validation is mandatory.
|
||||
- report output is config-gated.
|
||||
|
||||
@@ -1,74 +1,44 @@
|
||||
# Stage: prepare
|
||||
|
||||
## Purpose
|
||||
Materialize all required session inputs into canonical local workspace paths and record input provenance in the session manifest.
|
||||
Materialize canonical current-session inputs before processing stages.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `session.yml` (resolved session config)
|
||||
- `pipeline.resolved.yml` (materialized from resolved pipeline config)
|
||||
- `speakers.yml`
|
||||
- `autocorrect.yml`
|
||||
- `glossary.yml`
|
||||
## Inputs
|
||||
- resolved `campaign.yml`, `session.yml`, and pipeline config
|
||||
- stable input files (`speakers`, `autocorrect`, `glossary`, `players`, `party`)
|
||||
- audio source:
|
||||
- local (`session.inputs.audio_dir` or `session.inputs.audio_files`), or
|
||||
- S3 (`session.inputs.audio_s3.prefix`)
|
||||
- local `audio_dir`/`audio_files`, or
|
||||
- S3 `audio_s3.prefix`
|
||||
- enabled configured artifact input requirements for previous-session sources
|
||||
|
||||
Outputs:
|
||||
## Outputs
|
||||
- `inputs/campaign.yml`
|
||||
- `inputs/session.yml`
|
||||
- `inputs/pipeline.resolved.yml`
|
||||
- `inputs/speakers.yml`
|
||||
- `inputs/autocorrect.yml`
|
||||
- `inputs/glossary.yml`
|
||||
- `audio/*.flac` in session workdir
|
||||
- `manifest.Inputs` records with checksums and source metadata
|
||||
- `inputs/players.yml`
|
||||
- `inputs/party.yml`
|
||||
- `audio/*.flac`
|
||||
- optional `previous/manifest.json`
|
||||
- optional `previous/artifacts/**`
|
||||
- deterministic `manifest.inputs` entries (checksums + provenance)
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Input path resolution and validation
|
||||
- Local copy/materialization of configs and audio files
|
||||
- S3 audio download to run-scoped spool, then copy into work audio dir
|
||||
## Key Behavior
|
||||
- validates required config/store state.
|
||||
- enforces local audio vs S3 audio mutual exclusivity.
|
||||
- materializes S3 audio through spool/cache-aware logic.
|
||||
- scans enabled configured artifact inputs for `narratio.previous_session.artifact.*` requirements.
|
||||
- when previous requirements exist:
|
||||
- clears managed `previous/` state;
|
||||
- builds previous-cache remote plan;
|
||||
- downloads previous manifest/artifacts;
|
||||
- records previous inputs in `manifest.inputs`.
|
||||
|
||||
Does not own:
|
||||
- Transcript generation/processing
|
||||
- Archive publish behavior
|
||||
Required previous-session inputs fail when unavailable; optional missing inputs are skipped.
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `session.inputs.speakers_file`
|
||||
- `session.inputs.autocorrect_file`
|
||||
- `session.inputs.glossary_file`
|
||||
- `session.inputs.audio_dir`
|
||||
- `session.inputs.audio_files`
|
||||
- `session.inputs.audio_s3.prefix`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.spool.root`
|
||||
- `pipeline.storage.s3.bucket`
|
||||
- `pipeline.storage.s3.root_prefix`
|
||||
|
||||
## External Adapters Used
|
||||
- Object storage backend (`env.ObjectStore`) for S3 audio list/download when `audio_s3` is configured.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Ensures workspace layout exists.
|
||||
- Writes resolved config and input files to canonical `inputs/` paths.
|
||||
- Records all prepared inputs into `manifest.Inputs` (sorted deterministically by kind/path).
|
||||
- For S3 audio, records `S3Bucket`, `S3Key`, `S3Size`, `S3ETag`, and `SpoolPath` in each audio input record.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when stage already `succeeded` and `--force` is not set.
|
||||
- Stage itself is deterministic/idempotent for unchanged inputs (`copyFileIfChanged`, `writeBytesIfChanged`).
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing required files, invalid audio source combinations, no discoverable `.flac` files, duplicate audio basenames, missing object store for S3 mode, or S3 list/download failures.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/prepare_test.go`
|
||||
- `internal/app/session_cli_test.go`
|
||||
- `internal/config/load_validate_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive.
|
||||
- Audio files must be `.flac`.
|
||||
- Canonical `inputs/*` and `audio/*` paths are the durable source for downstream stages.
|
||||
## Invariants
|
||||
- only `prepare` hydrates canonical `previous/` cache state.
|
||||
- managed previous artifacts are stored under `previous/artifacts/**` without duplicate `artifacts/artifacts/` nesting.
|
||||
- `manifest.inputs` ordering is deterministic (`kind`, `path`).
|
||||
|
||||
44
docs/internal/stage-publish.md
Normal file
44
docs/internal/stage-publish.md
Normal file
@@ -0,0 +1,44 @@
|
||||
# Stage: publish
|
||||
|
||||
## Purpose
|
||||
Upload run/session outputs to object storage and atomically advance remote current state.
|
||||
|
||||
## Inputs
|
||||
- successful prerequisite stages: `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `render`, `analyze`
|
||||
- run root `runs/{run_id}/**`
|
||||
- publish output rules (`pipeline.publish.outputs`)
|
||||
- effective publish locks (static + remote merged lock set)
|
||||
- local `previous/**` files when present
|
||||
|
||||
## Outputs
|
||||
- uploaded run files under remote `runs/{run_id}/...` (excluding `audio/**`)
|
||||
- uploaded selected publish outputs under session prefix
|
||||
- uploaded `previous/**` files under session prefix when present
|
||||
- uploaded `current/manifest.json`
|
||||
- uploaded `current/run_id.txt` written last
|
||||
|
||||
## Key Behavior
|
||||
- stage can self-skip when publish disabled or run upload disabled.
|
||||
- validates prerequisite stage success and object-store availability.
|
||||
- collects deterministic run file list plus run `manifest.json`.
|
||||
- resolves publish output sources through runtime artifact catalog and manifest-aware resolution.
|
||||
- selected artifact filter applies to configured artifact sources only.
|
||||
- locked outputs are skipped intentionally (including required ones).
|
||||
- optional missing outputs are skipped; required missing unlocked outputs fail.
|
||||
- writes remote current manifest before current run pointer.
|
||||
|
||||
## Metadata Signals
|
||||
Includes counts/lists for:
|
||||
- run uploads
|
||||
- published output uploads
|
||||
- previous uploads
|
||||
- skipped optional outputs
|
||||
- skipped unselected outputs
|
||||
- locked outputs
|
||||
- current-state key paths
|
||||
- `current_pointer_written`
|
||||
|
||||
## Invariants
|
||||
- `current/run_id.txt` is the remote commit marker and is written last.
|
||||
- run upload excludes `audio/**`.
|
||||
- publish locks are not overridden by `--force`.
|
||||
29
docs/internal/stage-render.md
Normal file
29
docs/internal/stage-render.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# Stage: render
|
||||
|
||||
## Purpose
|
||||
Render Markdown transcript artifacts from normalized JSON transcripts via Seriatim.
|
||||
|
||||
## Inputs
|
||||
- `narratio.transcript.final` (`transcripts/final.json`)
|
||||
- `narratio.transcript.final_trimmed` (`transcripts/final.trimmed.json`)
|
||||
|
||||
## Outputs
|
||||
- `narratio.transcript.final_markdown` -> `transcripts/final.md`
|
||||
- `narratio.transcript.final_trimmed_markdown` -> `transcripts/final.trimmed.md`
|
||||
|
||||
## Key Behavior
|
||||
- uses `pipeline.render` settings (enabled/format/title/booleans).
|
||||
- resolves inputs manifest-first, then canonical fallback.
|
||||
- writes run-local outputs first, then materializes canonical session outputs.
|
||||
- records input provenance, output paths, adapter metadata, logs, and generated config refs.
|
||||
- skips with stage metadata when `pipeline.render.enabled=false`.
|
||||
|
||||
## Failure Semantics
|
||||
- missing normalized input fails with normalize rerun guidance.
|
||||
- missing trimmed input fails with trim rerun guidance.
|
||||
- adapter/subprocess failure fails stage.
|
||||
- empty render output files fail validation.
|
||||
|
||||
## Invariants
|
||||
- only `format: markdown` is supported.
|
||||
- render stage owns production of built-in Markdown transcript sources.
|
||||
@@ -1,58 +1,22 @@
|
||||
# Stage: transcribe
|
||||
|
||||
## Purpose
|
||||
Generate per-speaker raw transcripts from prepared audio using WhisperX.
|
||||
Generate raw per-speaker transcripts from prepared audio using WhisperX.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `audio/*.flac` prepared by `prepare`
|
||||
## Inputs
|
||||
- `audio/*.flac` from `prepare`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/raw/<speaker>.json` for each input audio file
|
||||
## Outputs
|
||||
- `transcripts/raw/<speaker>.json`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Discovering prepared audio inputs
|
||||
- Deriving speaker ids from audio basenames
|
||||
- Parallel WhisperX invocation with bounded concurrency
|
||||
- Validating produced JSON and promoting run-local outputs
|
||||
## Key Behavior
|
||||
- discovers prepared audio from manifest inputs or canonical audio directory.
|
||||
- derives speaker ID from `.flac` basename.
|
||||
- runs WhisperX with configured concurrency/retry settings.
|
||||
- validates each output as JSON.
|
||||
- writes run-local outputs then materializes canonical transcript outputs.
|
||||
|
||||
Does not own:
|
||||
- Transcript merge/polish/normalize/trim/analyze
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.whisperx.transcribe_url`
|
||||
- `pipeline.whisperx.language`
|
||||
- `pipeline.whisperx.timeout`
|
||||
- `pipeline.whisperx.retries`
|
||||
- `pipeline.whisperx.retry_delay`
|
||||
- `pipeline.whisperx.concurrency`
|
||||
|
||||
## External Adapters Used
|
||||
- WhisperX adapter (`env.WhisperX.Transcribe`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Uses run-local output paths under `runs/{run_id}/transcribe/outputs/...` when run layout is enabled.
|
||||
- Validates each generated transcript JSON before promotion.
|
||||
- Promotes canonical outputs to `transcripts/raw/*.json`.
|
||||
- Records per-file metadata (attempts/status/duration/output path) in stage metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies for previously succeeded stage unless forced.
|
||||
- On forced upstream reruns, downstream succeeded stages can be marked `stale` by runner logic.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails if no prepared audio exists, duplicate speaker basenames are detected, adapter output path mismatches expected path, any output JSON is invalid, or one worker fails.
|
||||
- Cancels in-flight workers after first terminal error.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/transcribe_test.go`
|
||||
- `internal/app/whisperx_wiring_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Speaker identity is derived from `.flac` basename and must be unique.
|
||||
- Every successful speaker output must be valid JSON before promotion.
|
||||
- Canonical raw transcript set is the only supported merge input surface.
|
||||
## Invariants
|
||||
- speaker basenames must be unique.
|
||||
- output path returned by adapter must match requested output path.
|
||||
- each successful output is validated before stage success.
|
||||
|
||||
@@ -1,75 +1,28 @@
|
||||
# Stage: trim
|
||||
|
||||
## Purpose
|
||||
Optionally trim the normalized transcript to session bounds; always produce a durable trimmed transcript.
|
||||
Produce a final-trimmed transcript. By default, the stage generates bounds and applies a bounds-driven trim.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `transcripts/normalized.json`
|
||||
## Inputs
|
||||
- `transcripts/final.json`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/trimmed.json` (or configured trim output path)
|
||||
## Outputs
|
||||
- `transcripts/final.trimmed.json` (or configured trim output path)
|
||||
- when trim enabled: `artifacts/session_bounds.json`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Trim-enabled switch behavior
|
||||
- Bounds generation via Scriptorium artifact run
|
||||
- Bounds validation against normalized transcript
|
||||
- Keep-selector derivation and Seriatim trim invocation
|
||||
- Copy-through behavior when disabled or bounds indicate unchanged transcript
|
||||
## Key Behavior
|
||||
When `trim.enabled=true`:
|
||||
- runs Scriptorium bounds artifact generation;
|
||||
- optionally runs render-debug output generation;
|
||||
- validates bounds payload against transcript;
|
||||
- derives keep selector;
|
||||
- either copies unchanged transcript or runs Seriatim trim;
|
||||
- validates trimmed transcript and materializes bounds output.
|
||||
|
||||
Does not own:
|
||||
- Upstream normalization
|
||||
- Downstream artifact analysis
|
||||
When `trim.enabled=false`:
|
||||
- copies normalized transcript to trimmed output.
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.trim.enabled`
|
||||
- `pipeline.trim.output_path`
|
||||
- `pipeline.trim.bounds.prompt_id`
|
||||
- `pipeline.trim.bounds.profile_id`
|
||||
- `pipeline.trim.bounds.timeout`
|
||||
- `pipeline.trim.bounds.output_path`
|
||||
- `pipeline.trim.bounds.transcript_input_name`
|
||||
- `pipeline.trim.bounds.render_debug`
|
||||
- `pipeline.trim.bounds.render_output_path`
|
||||
- `pipeline.seriatim.binary`
|
||||
- `pipeline.seriatim.timeout`
|
||||
- `pipeline.scriptorium.binary`
|
||||
- `pipeline.scriptorium.config_path`
|
||||
- `pipeline.scriptorium.timeout`
|
||||
|
||||
## External Adapters Used
|
||||
- Scriptorium adapter:
|
||||
- optional `RenderArtifact` for bounds debug render
|
||||
- `RunArtifact` for bounds output
|
||||
- Seriatim adapter:
|
||||
- `Trim` when bounds indicate trimming is required
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads normalized transcript from normalize manifest outputs when available; falls back to canonical path.
|
||||
- Uses run-local outputs/logs/reports/config/scratch paths when run layout is enabled.
|
||||
- Promotes canonical trimmed transcript; promotes session bounds when trim enabled.
|
||||
- Records bounds diagnostics, trim action, keep selector, and adapter metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced reruns can stale downstream succeeded stages.
|
||||
- When `trim.enabled=false`, stage still succeeds by copying normalized to trimmed output.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid normalized transcript.
|
||||
- With trim enabled, fails on missing adapters/config, bounds generation/validation errors, invalid bounds JSON, invalid range/segment ids, trim adapter failures, or invalid trimmed output.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/trim_test.go`
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Trim never falls back to processed transcript; normalized transcript is required input.
|
||||
- `session_bounds` output exists only for enabled trim path.
|
||||
- Render-debug artifacts are diagnostics and not declared stage outputs.
|
||||
## Invariants
|
||||
- normalized transcript is required input.
|
||||
- bounds output exists only in enabled trim path.
|
||||
- render-debug output is diagnostic and not a declared stage output.
|
||||
|
||||
@@ -1,71 +1,34 @@
|
||||
# Internal: Storage
|
||||
|
||||
## Purpose
|
||||
Document Narratio's remote storage backend contracts and implementations under `internal/adapters/storage`.
|
||||
Document remote object-store contracts and S3 implementation behavior.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- Resolved storage config (`pipeline.storage.*`).
|
||||
- Bucket-relative object keys and local file paths from stage/app orchestration.
|
||||
## Primary Contract
|
||||
`storage.ObjectStore` interface:
|
||||
- `List(ctx, prefix)`
|
||||
- `Download(ctx, key, localPath)`
|
||||
- `Upload(ctx, localPath, key, opts)`
|
||||
- `Exists(ctx, key)`
|
||||
|
||||
Outputs:
|
||||
- Listed/downloaded/uploaded object metadata (`ObjectInfo`).
|
||||
- Existence checks and storage-layer errors.
|
||||
Key invariant:
|
||||
- callers pass full bucket-relative keys;
|
||||
- storage implementations do not infer campaign/session/run prefixes.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Remote object-store interface and implementation details.
|
||||
- S3 client wiring and API calls.
|
||||
- Object key normalization and upload/download/list primitives.
|
||||
## Configuration
|
||||
`NewObjectStoreFromConfig` currently supports S3-backed stores from `pipeline.storage.*` config.
|
||||
|
||||
Does not own:
|
||||
- Session/run prefix semantics.
|
||||
- Archive commit order semantics.
|
||||
- Manifest updates.
|
||||
S3 constructor behavior:
|
||||
- requires configured bucket;
|
||||
- uses region/endpoint/path-style options when set;
|
||||
- resolves credentials from configured env var names (with defaults).
|
||||
|
||||
## Config fields used
|
||||
- `pipeline.storage.backend`
|
||||
- `pipeline.storage.s3.bucket`
|
||||
- `pipeline.storage.s3.region`
|
||||
- `pipeline.storage.s3.endpoint`
|
||||
- `pipeline.storage.s3.force_path_style`
|
||||
- `pipeline.storage.s3.access_key_id_env`
|
||||
- `pipeline.storage.s3.secret_access_key_env`
|
||||
## S3 Backend Behavior
|
||||
- normalizes object keys.
|
||||
- `List` paginates and returns normalized `ObjectInfo`.
|
||||
- `Download` writes local files with parent directory creation.
|
||||
- `Upload` streams local file and returns remote metadata.
|
||||
- `Exists` maps not-found responses to `false`.
|
||||
|
||||
## External adapters used
|
||||
Storage package contracts:
|
||||
- `ObjectStore` (active remote object-store boundary): `List`, `Download`, `Upload`, `Exists`.
|
||||
- `Backend` (archive request boundary): currently implemented with `NoopBackend` only.
|
||||
|
||||
Implementations:
|
||||
- `S3Backend`: AWS SDK-backed `ObjectStore` implementation.
|
||||
- `FakeBackend`: deterministic test `ObjectStore` and archive backend.
|
||||
- `NoopBackend`: deterministic no-op archive backend for compatibility wiring.
|
||||
|
||||
## State and manifest behavior
|
||||
- Storage implementations are stateless with respect to manifest/session lifecycle.
|
||||
- Caller supplies fully-qualified bucket-relative keys.
|
||||
- Storage layer does not infer campaign/session/run/root-prefix semantics.
|
||||
- Caller controls publish ordering; storage layer executes individual operations in the order invoked.
|
||||
|
||||
## Skip and resume behavior
|
||||
- No storage-level skip/resume behavior.
|
||||
- Skip/resume decisions are made by stage/app logic before storage calls occur.
|
||||
|
||||
## Failure behavior
|
||||
- `NewObjectStoreFromConfig` fails when no remote backend is configured or required S3 config is missing.
|
||||
- `S3Backend` constructor fails when required bucket is missing or AWS client setup fails.
|
||||
- CRUD operations return contextual errors (including not-found behavior via `Exists`).
|
||||
- Key normalization is applied before operations (`\\` to `/`, leading slash trimmed).
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/adapters/storage/factory_test.go`
|
||||
- `internal/adapters/storage/s3_backend_test.go`
|
||||
- `internal/adapters/storage/fake_test.go`
|
||||
- `internal/adapters/storage/keys_test.go`
|
||||
- `internal/adapters/storage/archive.go` + consumers in stage tests (`prepare`, `archive`)
|
||||
|
||||
## Architectural invariants
|
||||
- Callers pass full bucket-relative keys.
|
||||
- Storage backends must not prepend or infer narratio prefixes.
|
||||
- Remote transport details remain isolated to storage adapter implementations.
|
||||
## Invariants
|
||||
- storage layer is stateless regarding manifest/stage progression.
|
||||
- publish ordering semantics are owned by stage/app code, not storage adapters.
|
||||
|
||||
@@ -1,68 +1,57 @@
|
||||
# Workspace internals
|
||||
# Internal: Workspace
|
||||
|
||||
## Purpose
|
||||
Define the local durable and run-local workspace model used by stages, manifests, resume, and archive.
|
||||
Define local session layout, run-local stage layout, and cleanup guardrails.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `pipeline.workspace.root`
|
||||
- `session.campaign`
|
||||
- `session.session_id`
|
||||
- generated `run_id`
|
||||
## Canonical Session Layout
|
||||
Session root:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}`
|
||||
|
||||
Outputs:
|
||||
- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`
|
||||
- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`
|
||||
- Canonical durable session directories and run-local stage trees
|
||||
Core directories/files:
|
||||
- `inputs/`
|
||||
- `audio/`
|
||||
- `transcripts/`
|
||||
- `artifacts/`
|
||||
- `reports/`
|
||||
- `logs/`
|
||||
- `config/`
|
||||
- `current/`
|
||||
- `runs/`
|
||||
- `previous/`
|
||||
- `manifest.json`
|
||||
- `.lock`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Session-level path layout (`inputs/`, `audio/`, `transcripts/`, `artifacts/`, `reports/`, `logs/`, `config/`, `current/`, `runs/`)
|
||||
- Run-local stage sandbox layout under `runs/{run_id}/{stage}/`
|
||||
- Session lock acquisition/release (`.lock`)
|
||||
`previous/` reserved files:
|
||||
- `previous/manifest.json`
|
||||
- `previous/artifacts/**`
|
||||
|
||||
Does not own:
|
||||
- Stage business logic
|
||||
- Remote archive semantics (documented in `stage-archive.md`)
|
||||
- CLI argument parsing
|
||||
## Run-Local Stage Layout
|
||||
When run context is available, stages use:
|
||||
- `runs/{run_id}/{stage}/outputs/`
|
||||
- `runs/{run_id}/{stage}/logs/`
|
||||
- `runs/{run_id}/{stage}/reports/`
|
||||
- `runs/{run_id}/{stage}/config/`
|
||||
- `runs/{run_id}/{stage}/scratch/`
|
||||
|
||||
## Config Fields Used
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.workspace.cleanup_after_archive`
|
||||
- `pipeline.spool.root`
|
||||
- `pipeline.spool.delete_audio_after_archive`
|
||||
- `session.campaign`
|
||||
- `session.session_id`
|
||||
Run-local outputs are materialized back into canonical session paths before stage success.
|
||||
`previous/**` writes are never redirected to run-local output paths.
|
||||
|
||||
## External Adapters Used
|
||||
None directly in this subsystem. Stages may use object storage adapters and then write local outputs into this layout.
|
||||
## Locking
|
||||
`artifacts.LocalStore` enforces single-writer session lock via `.lock` file (`ErrLockConflict` on contention).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Session state is persisted in the session manifest (`manifest.Manifest`).
|
||||
- Invocation history is persisted per run in run manifests under `runs/{run_id}/manifest.json`.
|
||||
- During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and promoted to canonical session paths after stage success.
|
||||
- `manifest.Artifacts` entries record `ProducerRunID` for durable outputs.
|
||||
- For S3 audio sessions, `prepare` records spool/work paths and S3 provenance in `manifest.Inputs`.
|
||||
## Cleanup Semantics
|
||||
Automatic post-publish cleanup:
|
||||
- only runs when publish actually executed and succeeded;
|
||||
- requires `uploaded=true` and `current_pointer_written=true` metadata;
|
||||
- respects `pipeline.spool.delete_audio_after_publish` and `pipeline.workspace.cleanup_after_publish`;
|
||||
- refuses unsafe deletes (root delete, out-of-root delete, symlink paths).
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Skip/resume decisions are made in `internal/app` (`run_control.go`, `resume.go`) using stage status in the session manifest.
|
||||
- `--force` reruns selected stages and marks downstream previously-succeeded stages as `stale`.
|
||||
- Workspace layout is idempotent (`EnsureLayoutFor`) and reused across runs.
|
||||
Manual clean command:
|
||||
- `clean <session_id>` removes session work and spool subtree.
|
||||
- `clean --all` removes all workspace work and spool children.
|
||||
- durable cache is preserved unless `--clear-cache` is requested.
|
||||
|
||||
## Failure Behavior
|
||||
- Failures preserve manifests and run-local files for inspection.
|
||||
- Lock conflicts fail fast via `ErrLockConflict`.
|
||||
- Cleanup can fail post-archive; failure is recorded in archive stage metadata and returned by the run.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/artifacts/local_test.go`
|
||||
- `internal/stage/run_local_test.go`
|
||||
- `internal/app/run_control_test.go`
|
||||
- `internal/app/resume_run_stage_test.go`
|
||||
- `internal/app/post_archive_cleanup_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Session root is campaign-aware: `{workspace.root}/work/{campaign}/{session_id}`.
|
||||
- Run roots are always nested: `runs/{run_id}` under the session root.
|
||||
- Run-local output promotion must end in canonical session paths.
|
||||
- Cleanup only targets run-scoped directories and must never delete configured root directories.
|
||||
## Invariants
|
||||
- campaign-aware session root is mandatory.
|
||||
- manifest-driven stage state is durable across runs.
|
||||
- cleanup guardrails prevent destructive root/out-of-scope deletion.
|
||||
|
||||
@@ -1,149 +1,253 @@
|
||||
# Operations
|
||||
# Operations Guide
|
||||
|
||||
This guide describes the implemented operator lifecycle for Narratio.
|
||||
Operator workflow for running, recovering, and publishing Narratio sessions.
|
||||
|
||||
For field-level configuration, see [docs/config.md](./config.md). For full command/flag reference, see [docs/cli.md](./cli.md).
|
||||
For command syntax, see [docs/cli.md](./cli.md). For field-level config, see [docs/config.md](./config.md).
|
||||
|
||||
## Normal workflow (S3-first path)
|
||||
## Campaign and Session Selection
|
||||
|
||||
1. Upload session `.flac` files to object storage under the session audio prefix.
|
||||
2. Run Narratio:
|
||||
Campaign selection priority:
|
||||
|
||||
- `--campaign-file`
|
||||
- `--campaign`
|
||||
- `pipeline.campaigns.default_campaign_id`
|
||||
|
||||
Session source priority:
|
||||
|
||||
- `--session`
|
||||
- local default search paths
|
||||
- remote session object (S3) when local session file is not found and storage is configured
|
||||
|
||||
## Session Initialization
|
||||
|
||||
Use `session init` to generate a concrete session file for local or remote use.
|
||||
|
||||
Local file:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04
|
||||
narratio session init 2026-04-04 --output ./session.yml --date 2026-04-04 --title "Session 12"
|
||||
```
|
||||
|
||||
3. Read success output:
|
||||
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
||||
- use `manifest=<path>` with `status` for inspection.
|
||||
Remote session object:
|
||||
|
||||
Notes:
|
||||
- default config/session discovery applies unless `--config` and `--session` are passed.
|
||||
- S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access.
|
||||
```bash
|
||||
narratio session init 2026-04-04 --remote --force
|
||||
```
|
||||
|
||||
## Local filesystem layout and state artifacts
|
||||
If `campaign.yml` sets `session_template_file`, `session init` renders it. Template variables must resolve to concrete values.
|
||||
|
||||
Campaigns must provide stable input files for speakers, autocorrect, glossary, players, and party. Session files may override those paths for one session. The `prepare` stage materializes them under `inputs/`; configured Scriptorium artifacts can reference prepared `players`, `party`, and `glossary` files with `narratio.input.players`, `narratio.input.party`, and `narratio.input.glossary`.
|
||||
|
||||
## Standard Session Workflow
|
||||
|
||||
1. Select pipeline/campaign/session config.
|
||||
2. Validate session readiness:
|
||||
|
||||
```bash
|
||||
narratio session validate 2026-04-04
|
||||
```
|
||||
|
||||
3. (Optional) inspect stage decisions:
|
||||
|
||||
```bash
|
||||
narratio session plan 2026-04-04
|
||||
```
|
||||
|
||||
4. Run the pipeline:
|
||||
|
||||
```bash
|
||||
narratio run 2026-04-04
|
||||
```
|
||||
|
||||
5. Check state:
|
||||
|
||||
```bash
|
||||
narratio session status 2026-04-04
|
||||
```
|
||||
|
||||
## Stage Execution and Continuation Behavior
|
||||
|
||||
Canonical stage order:
|
||||
|
||||
1. `prepare`
|
||||
2. `transcribe`
|
||||
3. `merge`
|
||||
4. `polish`
|
||||
5. `normalize`
|
||||
6. `trim`
|
||||
7. `render`
|
||||
8. `analyze`
|
||||
9. `publish`
|
||||
10. `notify`
|
||||
|
||||
Execution rules:
|
||||
|
||||
- succeeded stages are skipped unless `--force` is set;
|
||||
- `run` continues interrupted or partially completed sessions by running non-succeeded stages;
|
||||
- force rerunning a succeeded upstream stage marks succeeded downstream stages as `stale`.
|
||||
|
||||
Single-stage execution:
|
||||
|
||||
```bash
|
||||
narratio run-stage normalize 2026-04-04 --force
|
||||
```
|
||||
|
||||
## Artifact Selection
|
||||
|
||||
`--artifacts` can be used on `run`, `run-stage`, `analyze`, and `publish`.
|
||||
|
||||
Selection behavior:
|
||||
|
||||
- validates names against `pipeline.scriptorium.artifacts`;
|
||||
- filters analyze execution to selected configured artifacts;
|
||||
- filters publish rules for `narratio.artifact.<name>` sources only;
|
||||
- does not suppress built-in transcript or bounds publish sources.
|
||||
|
||||
## Publish Workflow
|
||||
|
||||
Run publish only:
|
||||
|
||||
```bash
|
||||
narratio publish 2026-04-04
|
||||
```
|
||||
|
||||
Equivalent:
|
||||
|
||||
```bash
|
||||
narratio run-stage publish 2026-04-04 --force
|
||||
```
|
||||
|
||||
Publish commit model:
|
||||
|
||||
- uploads run files under `{session_prefix}/runs/{run_id}/`;
|
||||
- uploads configured published outputs;
|
||||
- uploads `previous/**` cache files when present;
|
||||
- writes `current/manifest.json`;
|
||||
- writes `current/run_id.txt` last.
|
||||
|
||||
`current/run_id.txt` is the remote current-state commit marker.
|
||||
|
||||
## Publish Locks
|
||||
|
||||
Lock sources:
|
||||
|
||||
- static locks in `pipeline.publish.locks`
|
||||
- mutable remote locks in `{session_prefix}/locks.yml`
|
||||
|
||||
Effective lock rules:
|
||||
|
||||
- static and remote locks are merged;
|
||||
- static locks win on source collisions;
|
||||
- locked outputs are intentional skips;
|
||||
- lock add/remove commands mutate only remote lock state.
|
||||
|
||||
Examples:
|
||||
|
||||
```bash
|
||||
narratio session locks 2026-04-04
|
||||
narratio session locks add 2026-04-04 narratio.artifact.session_recap --reason "manual edits" --force
|
||||
narratio session locks remove 2026-04-04 narratio.artifact.session_recap
|
||||
```
|
||||
|
||||
## Restore Workflow
|
||||
|
||||
Use restore when local durable session state is missing or stale and remote committed current state is authoritative.
|
||||
|
||||
Dry run:
|
||||
|
||||
```bash
|
||||
narratio session restore 2026-04-04 --dry-run
|
||||
```
|
||||
|
||||
Apply:
|
||||
|
||||
```bash
|
||||
narratio session restore 2026-04-04
|
||||
```
|
||||
|
||||
Default restore scope:
|
||||
|
||||
- `manifest.json`
|
||||
- `transcripts/**`
|
||||
- `artifacts/**`
|
||||
- `previous/**` when needed by configured previous-session artifact inputs
|
||||
|
||||
Optional:
|
||||
|
||||
- `--include-audio` to include `audio/**`
|
||||
- `--force` to overwrite local conflicts
|
||||
|
||||
Restore writes an execution report at `reports/restore-latest.json`.
|
||||
|
||||
## Local State Layout
|
||||
|
||||
Session root:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/`
|
||||
|
||||
Primary state:
|
||||
- `manifest.json`: session-level stage state.
|
||||
- `runs/{run_id}/manifest.json`: invocation-level state.
|
||||
- `.lock`: session lock while a run is active.
|
||||
- `{workspace.root}/work/{campaign}/{session_id}`
|
||||
|
||||
Canonical session directories:
|
||||
- `inputs/`
|
||||
- `audio/`
|
||||
- `transcripts/`
|
||||
- `artifacts/`
|
||||
- `reports/`
|
||||
- `logs/`
|
||||
- `config/`
|
||||
- `current/`
|
||||
- `runs/`
|
||||
Durable session paths:
|
||||
|
||||
Run-local stage directories:
|
||||
- `runs/{run_id}/{stage}/` with stage-local `outputs/`, `logs/`, `reports/`, `config/`, `scratch/`.
|
||||
- `manifest.json`
|
||||
- `inputs/**`
|
||||
- `audio/**`
|
||||
- `transcripts/**`
|
||||
- `artifacts/**`
|
||||
- `previous/**`
|
||||
- `reports/**`
|
||||
- `logs/**`
|
||||
- `config/**`
|
||||
- `runs/**`
|
||||
|
||||
Behavior:
|
||||
- directory creation is idempotent.
|
||||
- stage outputs are generally generated run-local first, then promoted to canonical paths on success.
|
||||
Run-local layout:
|
||||
|
||||
## Analyze artifact execution lifecycle
|
||||
- `runs/{run_id}/{stage}/outputs`
|
||||
- `runs/{run_id}/{stage}/logs`
|
||||
- `runs/{run_id}/{stage}/reports`
|
||||
- `runs/{run_id}/{stage}/config`
|
||||
- `runs/{run_id}/{stage}/scratch`
|
||||
|
||||
Analyze executes configured artifacts from `pipeline.scriptorium.artifacts`.
|
||||
Spool layout (runtime/transient):
|
||||
|
||||
Execution model:
|
||||
- executable set = enabled artifacts, filtered by `--artifacts` when provided.
|
||||
- artifact-to-artifact dependencies are declared via `depends_on`.
|
||||
- selected artifacts run in deterministic dependency order.
|
||||
- after each successful artifact run, output is promoted to configured canonical `output_path`.
|
||||
- `{spool.root}/{campaign}/{session_id}/{run_id}/...`
|
||||
- restore audio spool under `{spool.root}/{campaign}/{session_id}/restore/audio`
|
||||
|
||||
Configured artifact source reuse:
|
||||
- a non-executable configured artifact can satisfy inputs if its configured output file already exists and is valid.
|
||||
- reused configured artifact provenance is `filesystem.disabled_artifact_output`.
|
||||
Cache layout (durable S3 audio cache):
|
||||
|
||||
`--artifacts` behavior:
|
||||
- accepted on `run`, `resume`, and `run-stage analyze`.
|
||||
- filters analyze execution only; does not force stage rerun.
|
||||
- `{cache.root}/s3/{bucket}/...`
|
||||
|
||||
## Remote archive layout and publish contract
|
||||
## Cleanup
|
||||
|
||||
When archive is enabled and run upload is enabled, archive publishes under:
|
||||
|
||||
- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
- run prefix: `{session_prefix}/runs/{run_id}/`
|
||||
|
||||
Archive uploads:
|
||||
- run record files from run root (excluding `audio/`).
|
||||
- promoted files from explicit `archive.promote_artifacts` rules.
|
||||
|
||||
Publish order:
|
||||
1. upload `current/manifest.json`
|
||||
2. upload `current/run_id.txt` last
|
||||
|
||||
`current/run_id.txt` is the remote commit marker.
|
||||
|
||||
Archive promotion is explicit and path-based:
|
||||
- Narratio does not auto-promote all generated analyze artifacts.
|
||||
- missing required promotion sources fail archive stage.
|
||||
- missing optional promotion sources are skipped.
|
||||
|
||||
## Resume, retry, and safe rerun behavior
|
||||
|
||||
Default skip:
|
||||
- `run` and `run-stage` skip already-succeeded stages unless `--force` is set.
|
||||
|
||||
Resume:
|
||||
- `resume` starts at first non-succeeded stage.
|
||||
- `resume --force` runs full stage order.
|
||||
|
||||
Forced reruns:
|
||||
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
|
||||
|
||||
Safe rerun pattern:
|
||||
1. rerun the changed stage with `--force`.
|
||||
2. run `resume` to rebuild downstream stages.
|
||||
|
||||
## Cleanup behavior
|
||||
|
||||
Cleanup is considered only when archive stage executed and succeeded.
|
||||
|
||||
Cleanup toggles:
|
||||
- `pipeline.spool.delete_audio_after_archive=true` deletes run-scoped spool audio.
|
||||
- `pipeline.workspace.cleanup_after_archive=true` deletes run-scoped local run directory.
|
||||
|
||||
Cleanup eligibility gates:
|
||||
- archive enabled
|
||||
- archive run upload enabled
|
||||
- run record upload completed
|
||||
- current pointer write completed (`current/run_id.txt` written)
|
||||
|
||||
No cleanup for failed/incomplete/unarchived/archive-skipped runs.
|
||||
|
||||
## Failure and recovery playbooks
|
||||
|
||||
After failure, Narratio keeps:
|
||||
- session manifest
|
||||
- run manifest
|
||||
- run-local artifacts/logs/config/reports
|
||||
|
||||
Failed or incomplete runs remain local-only.
|
||||
|
||||
Recommended recovery:
|
||||
|
||||
1. inspect state:
|
||||
Session-scoped cleanup:
|
||||
|
||||
```bash
|
||||
narratio status --manifest <manifest-path>
|
||||
narratio clean 2026-04-04
|
||||
```
|
||||
|
||||
2. fix root cause (config/input/credentials/service availability).
|
||||
3. continue with `resume`, or targeted `run-stage --force` followed by `resume`.
|
||||
Global cleanup:
|
||||
|
||||
## Operational caveats
|
||||
```bash
|
||||
narratio clean --all
|
||||
```
|
||||
|
||||
- `status` requires explicit `--manifest`; there is no session-id lookup command.
|
||||
- local and S3 audio input modes are mutually exclusive.
|
||||
- archive publish requires upstream stages through `analyze` to be `succeeded`.
|
||||
- required promotion rules can fail when selected analyze artifacts did not generate a required file path.
|
||||
Dry-run and cache variants:
|
||||
|
||||
```bash
|
||||
narratio clean 2026-04-04 --dry-run --clear-cache
|
||||
narratio clean --all --dry-run --clear-cache
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- `clean` deletes work/spool session state;
|
||||
- cache is preserved unless `--clear-cache` is set;
|
||||
- automatic post-publish cleanup is gated by successful publish commit plus:
|
||||
- `pipeline.spool.delete_audio_after_publish=true`
|
||||
- `pipeline.workspace.cleanup_after_publish=true`
|
||||
|
||||
## Operational Caveats
|
||||
|
||||
- Local and S3 audio modes are mutually exclusive.
|
||||
- Publish requires prerequisite stages through `render` and `analyze` to be succeeded.
|
||||
- Markdown publish defaults require render outputs (`transcripts/final.md` and `transcripts/final.trimmed.md`).
|
||||
- Restore requires configured object storage and committed remote current state.
|
||||
- Storage-backed commands load filesystem secrets before object-store initialization.
|
||||
|
||||
@@ -19,7 +19,7 @@ It coordinates specialized downstream systems rather than reimplementing their d
|
||||
- Audita handles transcript correction and polishing.
|
||||
- Scriptorium handles prompt execution and generated artifacts.
|
||||
|
||||
Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and archive semantics.
|
||||
Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and publish semantics.
|
||||
|
||||
Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine.
|
||||
|
||||
@@ -80,7 +80,7 @@ It should record:
|
||||
- input and output refs;
|
||||
- logs and generated config refs;
|
||||
- checksums or provenance where useful;
|
||||
- non-secret adapter and archive metadata.
|
||||
- non-secret adapter and publish metadata.
|
||||
|
||||
Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress.
|
||||
|
||||
@@ -117,19 +117,19 @@ Narratio should not become a secondary configuration system for downstream tools
|
||||
|
||||
Local and remote paths are part of Narratio’s application contract.
|
||||
|
||||
Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and archive paths. Stages should avoid reconstructing canonical paths through scattered string concatenation.
|
||||
Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and publish/current paths. Stages should avoid reconstructing canonical paths through scattered string concatenation.
|
||||
|
||||
Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics.
|
||||
|
||||
## Archive Invariants
|
||||
## Publish Invariants
|
||||
|
||||
Archive behavior must preserve a clear commit boundary.
|
||||
Publish behavior must preserve a clear commit boundary.
|
||||
|
||||
A remote run is current only after the archive stage has successfully uploaded the run record, required promoted outputs, `current/manifest.json`, and finally `current/run_id.txt`.
|
||||
A remote run is current only after the publish stage has successfully uploaded the run record, required published outputs, `current/manifest.json`, and finally `current/run_id.txt`.
|
||||
|
||||
`current/run_id.txt` is the final remote commit marker and must be written last.
|
||||
|
||||
Failed, incomplete, skipped, or uncommitted archive attempts must not be presented as current remote state. Local cleanup is permitted only after successful archive commit and only when explicitly configured.
|
||||
Failed, incomplete, skipped, or uncommitted publish attempts must not be presented as current remote state. Local cleanup is permitted only after successful publish commit and only when explicitly configured.
|
||||
|
||||
## Security and Privacy
|
||||
|
||||
@@ -139,7 +139,7 @@ Rules:
|
||||
|
||||
- Do not store raw secrets in pipeline or session YAML.
|
||||
- Use environment variable names or secret-file references for secret handling.
|
||||
- Do not write raw secret values to manifests, logs, generated configs, or archive metadata.
|
||||
- Do not write raw secret values to manifests, logs, generated configs, or publish metadata.
|
||||
- Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive.
|
||||
- Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason.
|
||||
|
||||
@@ -177,7 +177,7 @@ Tests should cover:
|
||||
- stage success, failure, skip, and resume behavior;
|
||||
- adapter command construction;
|
||||
- fake storage behavior;
|
||||
- archive commit ordering;
|
||||
- publish commit ordering;
|
||||
- example config validity where practical.
|
||||
|
||||
Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs.
|
||||
@@ -6,7 +6,7 @@ Canonical contributor workflow and engineering conventions for implemented Narra
|
||||
## Repository layout
|
||||
|
||||
- `cmd/narratio/`: CLI entrypoint.
|
||||
- `internal/app/`: command handlers, plan/run/resume orchestration, cleanup gates, secrets loading.
|
||||
- `internal/app/`: command handlers, run/stage orchestration, cleanup gates, secrets loading.
|
||||
- `internal/config/`: strict YAML loading, defaults, and validation.
|
||||
- `internal/stage/`: stage implementations and stage registry/order.
|
||||
- `internal/adapters/`: external boundary adapters (WhisperX, Seriatim, Audita, Scriptorium, storage, notify).
|
||||
@@ -69,11 +69,13 @@ For design principles and invariants, see [docs/architecture.md](./architecture.
|
||||
2. Add or update command tests (`TestExecute` and command-specific tests).
|
||||
3. Update [docs/cli.md](./cli.md) and, if operator workflow changes, [docs/operations.md](./operations.md).
|
||||
|
||||
Remote-storage commands must obtain object storage through the app-level command object-store helper. Do not call `storage.NewObjectStoreFromConfig` directly from command handlers; the helper loads configured filesystem secrets before constructing the storage adapter.
|
||||
|
||||
### Add or modify stages/adapters
|
||||
|
||||
1. Implement stage behavior in `internal/stage` with clear input/output boundaries.
|
||||
2. Keep external transport/subprocess details in `internal/adapters`.
|
||||
3. Preserve manifest and promotion semantics expected by runner and archive logic.
|
||||
3. Preserve manifest and publish-output semantics expected by runner and publish logic.
|
||||
4. Add/update stage and adapter tests.
|
||||
5. Update internal component contracts in `docs/internal/`.
|
||||
|
||||
@@ -1,762 +0,0 @@
|
||||
# Roadmap: Runtime-Defined Scriptorium Artifacts
|
||||
|
||||
## Status
|
||||
|
||||
Implementation roadmap for a pre-release hard cutover.
|
||||
|
||||
## Purpose
|
||||
|
||||
Narratio currently treats artifact generation as a narrow `analyze` stage that supports a hard-coded `session_recap` artifact. This roadmap describes how to generalize artifact generation so operators can define Scriptorium-backed output artifacts at runtime through `pipeline.yml`.
|
||||
|
||||
The goal is to keep Narratio as a fixed pipeline orchestrator while making the artifact generation step configurable, composable, deterministic, and easy to regenerate selectively.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
Operators should be able to define artifacts such as session recaps, player handouts, NPC summaries, quest logs, entity maps, or other campaign-specific outputs without changing Narratio code.
|
||||
|
||||
A configured artifact is declared under:
|
||||
|
||||
```text
|
||||
pipeline.scriptorium.artifacts.<name>
|
||||
```
|
||||
|
||||
Each configured artifact becomes a canonical runtime artifact source ID:
|
||||
|
||||
```text
|
||||
narratio.artifact.<name>
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd_session.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
```
|
||||
|
||||
This artifact is addressable by later artifacts as:
|
||||
|
||||
```text
|
||||
narratio.artifact.session_recap
|
||||
```
|
||||
|
||||
A dependent artifact can then consume it explicitly:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
player_handout:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- session_recap
|
||||
prompt_id: dnd_session.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
```
|
||||
|
||||
## Resolved Design Decisions
|
||||
|
||||
The following decisions are settled for the initial implementation:
|
||||
|
||||
1. Configured artifact outputs must live under Narratio's internal artifact output directory, initially `artifacts/`.
|
||||
2. The artifact output directory should be defined as an internal default in `internal/config/defaults.go`, but no public configuration knob should be exposed yet.
|
||||
3. Artifact `output_path` should remain explicit in the initial implementation to avoid guessing file extensions or output formats.
|
||||
4. A disabled artifact may still be referenced as an input if its declared output already exists on disk and passes basic validation.
|
||||
5. A disabled artifact is not executable during the current analyze run.
|
||||
6. Artifact-to-artifact references require an explicit `depends_on` entry. Narratio should fail fast if the dependency declaration is missing.
|
||||
7. The manifest remains stage-oriented: `analyze` succeeds or fails as a full stage.
|
||||
8. Analyze-stage metadata may record per-artifact output details for provenance and later resolution, but not for intra-stage resume semantics.
|
||||
9. `--artifacts` should be added as a CLI filter for selective artifact generation.
|
||||
10. `--artifacts` does not imply `--force`; it only changes which configured artifacts are treated as executable when `analyze` actually runs.
|
||||
11. Because Narratio is still pre-release, the hard-coded `session_recap` behavior should be removed immediately rather than deprecated gradually.
|
||||
|
||||
## Scope
|
||||
|
||||
This roadmap covers:
|
||||
|
||||
- introducing a runtime artifact catalog;
|
||||
- generalizing configured Scriptorium artifact execution;
|
||||
- supporting `narratio.artifact.<name>` source IDs;
|
||||
- adding explicit artifact dependencies;
|
||||
- supporting disabled-but-resolvable artifact inputs;
|
||||
- adding selective artifact execution via `--artifacts`;
|
||||
- recording generated artifacts in analyze-stage metadata and/or manifest outputs;
|
||||
- removing hard-coded `session_recap` behavior;
|
||||
- updating tests and documentation.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This feature should not turn Narratio into a general workflow engine.
|
||||
|
||||
The initial implementation should not add:
|
||||
|
||||
- arbitrary shell-command artifacts;
|
||||
- arbitrary user-defined stages;
|
||||
- loops or conditional branching;
|
||||
- automatic archive promotion of generated artifacts;
|
||||
- semantic knowledge of particular artifact types;
|
||||
- per-artifact resume semantics within a successful or failed analyze stage;
|
||||
- automatic dependency inference without `depends_on`.
|
||||
|
||||
Narratio should continue to orchestrate a fixed pipeline. The configurable part is the set of Scriptorium artifact invocations performed during the `analyze` stage.
|
||||
|
||||
## Current State
|
||||
|
||||
Narratio already has several relevant pieces in place:
|
||||
|
||||
- `pipeline.scriptorium.artifacts` is modeled as a map of artifact definitions.
|
||||
- The Scriptorium adapter already accepts generic run/render requests.
|
||||
- The artifact resolver already understands canonical artifact source IDs.
|
||||
- The `analyze` stage already resolves inputs, optionally runs render-debug, invokes Scriptorium, verifies output, and records metadata.
|
||||
|
||||
The main limitation is that `analyze` currently treats `session_recap` as the only executable artifact and rejects other enabled artifact definitions.
|
||||
|
||||
## Target Architecture
|
||||
|
||||
### Runtime Artifact Catalog
|
||||
|
||||
Introduce a per-run artifact catalog that tracks built-in artifacts and configured artifacts.
|
||||
|
||||
Conceptually:
|
||||
|
||||
```text
|
||||
ArtifactCatalog
|
||||
├── built-in artifacts
|
||||
│ ├── narratio.transcript.merged
|
||||
│ ├── narratio.transcript.polished
|
||||
│ ├── narratio.transcript.full
|
||||
│ ├── narratio.transcript.trimmed
|
||||
│ └── narratio.bounds.session
|
||||
│
|
||||
└── configured artifacts
|
||||
├── narratio.artifact.session_recap
|
||||
├── narratio.artifact.player_handout
|
||||
└── narratio.artifact.npc_summary
|
||||
```
|
||||
|
||||
The catalog should distinguish between three states:
|
||||
|
||||
```text
|
||||
planned valid configured or built-in artifact known to Narratio
|
||||
available artifact has been produced or otherwise resolved
|
||||
executable configured artifact selected for execution in this analyze run
|
||||
```
|
||||
|
||||
Configured artifacts can be planned without being executable. This distinction is important for disabled artifacts and for `--artifacts` filtering.
|
||||
|
||||
### Configured Artifact Source IDs
|
||||
|
||||
Configured artifact keys map directly to source IDs:
|
||||
|
||||
```text
|
||||
pipeline.scriptorium.artifacts.<name>
|
||||
→ narratio.artifact.<name>
|
||||
```
|
||||
|
||||
`session_recap` should no longer be a special built-in analyze artifact. Instead, it is just a conventional configured artifact key:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd_session.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
```
|
||||
|
||||
`narratio.artifact.session_recap` remains valid only because `session_recap` is configured.
|
||||
|
||||
### Artifact Output Directory
|
||||
|
||||
Add an internal default artifact output directory, initially:
|
||||
|
||||
```text
|
||||
artifacts
|
||||
```
|
||||
|
||||
This default should live in `internal/config/defaults.go` or the existing equivalent defaults location.
|
||||
|
||||
For the initial implementation:
|
||||
|
||||
- expose no public config knob for the artifact output directory;
|
||||
- require each configured artifact to provide an explicit `output_path`;
|
||||
- validate that each configured artifact `output_path` is run-relative;
|
||||
- validate that each configured artifact `output_path` is under the internal artifact output directory;
|
||||
- reject output paths that escape the run workspace or use path traversal.
|
||||
|
||||
This preserves future configurability without forcing Narratio to guess output extensions or formats now.
|
||||
|
||||
### Enabled, Disabled, and Selected Artifacts
|
||||
|
||||
Configured artifacts should have three distinct execution states:
|
||||
|
||||
```text
|
||||
enabled by config artifact has enabled: true
|
||||
selected for execution artifact remains executable after --artifacts filtering
|
||||
disabled for execution artifact is not executable, but may be resolvable from disk
|
||||
```
|
||||
|
||||
Without `--artifacts`, all configured artifacts with `enabled: true` are selected for execution.
|
||||
|
||||
With `--artifacts`, only the named artifacts are selected for execution. All other configured artifacts are treated as disabled for the current analyze invocation, regardless of their configured `enabled` value.
|
||||
|
||||
Disabled artifacts may still be resolved as inputs if their configured `output_path` exists on disk and passes validation.
|
||||
|
||||
### Disabled Artifact Resolution
|
||||
|
||||
If artifact `B` references artifact `A`, and `A` is disabled for execution, Narratio should attempt to resolve `A` from disk.
|
||||
|
||||
This should succeed only when:
|
||||
|
||||
1. `A` is defined in `pipeline.scriptorium.artifacts`;
|
||||
2. `A` has a valid `output_path`;
|
||||
3. the output path exists in the current run workspace;
|
||||
4. the output is non-empty, or otherwise passes any available artifact-specific validation.
|
||||
|
||||
The resolved provenance should make the source clear, for example:
|
||||
|
||||
```text
|
||||
filesystem.disabled_artifact_output
|
||||
```
|
||||
|
||||
If the file does not exist or fails validation, the dependent artifact should fail before invoking Scriptorium.
|
||||
|
||||
Example error wording:
|
||||
|
||||
```text
|
||||
artifact player_handout requires narratio.artifact.session_recap, but session_recap is disabled for execution and artifacts/session_recap.md does not exist
|
||||
```
|
||||
|
||||
### Explicit Dependencies
|
||||
|
||||
Artifact-to-artifact references require explicit `depends_on` entries.
|
||||
|
||||
If artifact `B` has an input source of `narratio.artifact.A`, then `B.depends_on` must include `A`.
|
||||
|
||||
This should fail:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
player_handout:
|
||||
enabled: true
|
||||
prompt_id: dnd_session.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
```
|
||||
|
||||
This should pass:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
player_handout:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- session_recap
|
||||
prompt_id: dnd_session.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
```
|
||||
|
||||
`depends_on` values refer to configured artifact keys, not full source IDs.
|
||||
|
||||
Dependency validation should fail on:
|
||||
|
||||
- references to unknown artifact keys;
|
||||
- missing `depends_on` entries for artifact-to-artifact input references;
|
||||
- self-dependencies;
|
||||
- dependency cycles among executable artifacts.
|
||||
|
||||
Dependencies on disabled artifacts are permitted, but the disabled dependency must resolve from disk before the dependent artifact runs.
|
||||
|
||||
### Execution Order
|
||||
|
||||
The analyze stage should execute selected artifacts in dependency order.
|
||||
|
||||
Rules:
|
||||
|
||||
- selected artifacts are executable;
|
||||
- disabled artifacts are never executed;
|
||||
- selected artifacts may depend on other selected artifacts;
|
||||
- selected artifacts may depend on disabled artifacts if those disabled artifacts resolve from disk;
|
||||
- independent selected artifacts run in deterministic sorted-name order.
|
||||
|
||||
Use topological sorting over selected artifacts, while validating dependency references across the full configured artifact set.
|
||||
|
||||
### Input Resolution
|
||||
|
||||
Input resolution should use the artifact catalog and existing artifact resolver behavior.
|
||||
|
||||
For each configured artifact input:
|
||||
|
||||
- built-in sources resolve through existing resolver behavior;
|
||||
- `previous_session_artifact` preserves existing behavior;
|
||||
- `narratio.artifact.<name>` resolves through the runtime artifact catalog;
|
||||
- selected dependencies resolve after being produced earlier in the same analyze execution;
|
||||
- disabled dependencies resolve from their configured output path on disk;
|
||||
- optional missing inputs are omitted;
|
||||
- required missing inputs fail before Scriptorium is invoked.
|
||||
|
||||
### Analyze Stage Generalization
|
||||
|
||||
The `analyze` stage should become the generic Scriptorium artifact stage.
|
||||
|
||||
High-level flow:
|
||||
|
||||
1. Load configured Scriptorium artifacts.
|
||||
2. Apply the `--artifacts` filter, if present.
|
||||
3. If no artifacts are selected for execution, return success metadata with `skipped=true`.
|
||||
4. Build the runtime artifact catalog.
|
||||
5. Validate artifact names, output paths, source IDs, dependencies, selected artifacts, and required fields.
|
||||
6. Resolve any disabled dependencies that are required by selected artifacts.
|
||||
7. Sort selected artifacts by dependency order.
|
||||
8. For each selected artifact:
|
||||
- resolve configured inputs;
|
||||
- build the Scriptorium run request;
|
||||
- optionally run Scriptorium render-debug;
|
||||
- run Scriptorium;
|
||||
- fail on validation-failed result;
|
||||
- verify the output exists and is non-empty;
|
||||
- record artifact output metadata;
|
||||
- register `narratio.artifact.<name>` as available in the catalog.
|
||||
9. Return aggregate analyze-stage metadata containing all generated and reused artifacts relevant to the run.
|
||||
|
||||
The Scriptorium adapter should remain generic. It should not decide which artifacts run, how dependencies work, or how artifacts are registered.
|
||||
|
||||
### Manifest and Metadata
|
||||
|
||||
The manifest should remain stage-oriented.
|
||||
|
||||
This means:
|
||||
|
||||
- `analyze` succeeds or fails as a full stage;
|
||||
- if `analyze` has already succeeded and the user does not force it, the runner skips it as a full stage;
|
||||
- Narratio should not implement per-artifact resume in the first version.
|
||||
|
||||
However, analyze-stage metadata should still record artifact outputs for provenance and future resolution.
|
||||
|
||||
Recommended metadata shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"skipped": false,
|
||||
"artifacts": [
|
||||
{
|
||||
"name": "session_recap",
|
||||
"source_id": "narratio.artifact.session_recap",
|
||||
"output_kind": "scriptorium_artifact",
|
||||
"path": "artifacts/session_recap.md",
|
||||
"prompt_id": "dnd_session.session_recap",
|
||||
"profile_id": "local-gemma-31b",
|
||||
"provenance": "generated.current_analyze_run"
|
||||
},
|
||||
{
|
||||
"name": "player_handout",
|
||||
"source_id": "narratio.artifact.player_handout",
|
||||
"output_kind": "scriptorium_artifact",
|
||||
"path": "artifacts/player_handout.md",
|
||||
"prompt_id": "dnd_session.player_handout",
|
||||
"profile_id": "local-gemma-31b",
|
||||
"provenance": "generated.current_analyze_run"
|
||||
}
|
||||
],
|
||||
"reused_artifacts": [
|
||||
{
|
||||
"name": "session_recap",
|
||||
"source_id": "narratio.artifact.session_recap",
|
||||
"path": "artifacts/session_recap.md",
|
||||
"provenance": "filesystem.disabled_artifact_output"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The exact struct can differ from this example, but it should preserve:
|
||||
|
||||
- artifact name;
|
||||
- canonical source ID;
|
||||
- output path;
|
||||
- prompt/profile provenance for generated artifacts;
|
||||
- reused-vs-generated provenance.
|
||||
|
||||
### Resume and Force Behavior
|
||||
|
||||
Keep resume behavior stage-level.
|
||||
|
||||
Recommended semantics:
|
||||
|
||||
```text
|
||||
No --force, analyze already succeeded:
|
||||
runner skips analyze, regardless of --artifacts.
|
||||
|
||||
--force, no --artifacts:
|
||||
analyze regenerates all configured artifacts with enabled: true.
|
||||
|
||||
--force --artifacts player_handout:
|
||||
analyze treats only player_handout as executable.
|
||||
all other configured artifacts are disabled for execution.
|
||||
disabled dependencies may be reused from disk.
|
||||
|
||||
--artifacts player_handout on a not-yet-completed analyze stage:
|
||||
analyze runs only player_handout.
|
||||
disabled dependencies may be reused from disk.
|
||||
```
|
||||
|
||||
`--artifacts` should not imply `--force`. It is an execution filter, not a resume override.
|
||||
|
||||
### `--artifacts` CLI Flag
|
||||
|
||||
Add an `--artifacts` flag to commands that can execute or resume the analyze stage.
|
||||
|
||||
The flag should accept one or more configured artifact names. Internally, normalize values to a set of artifact keys.
|
||||
|
||||
Recommended behavior:
|
||||
|
||||
- validate all requested artifact names against `pipeline.scriptorium.artifacts`;
|
||||
- reject unknown artifact names before running stages;
|
||||
- treat requested artifacts as the only executable artifacts for the analyze stage;
|
||||
- treat all other configured artifacts as disabled for execution;
|
||||
- allow disabled artifacts to satisfy dependencies from disk as described above;
|
||||
- if `--artifacts` is used while executing a stage other than `analyze`, either reject it or ignore it with a clear validation error. Prefer rejection.
|
||||
|
||||
The exact CLI parsing style can follow Narratio's existing conventions. Both comma-separated and repeatable values are acceptable if the CLI package supports them cleanly, but the internal representation should be a set of artifact keys.
|
||||
|
||||
### Archive Behavior
|
||||
|
||||
Do not automatically archive every generated artifact.
|
||||
|
||||
Artifact generation and archive promotion should remain separate concerns. Operators should continue to use `archive.promote_artifacts` to decide which generated files should be promoted or uploaded.
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
archive:
|
||||
promote_artifacts:
|
||||
- from: artifacts/session_recap.md
|
||||
to: artifacts/session_recap.md
|
||||
required: true
|
||||
- from: artifacts/player_handout.md
|
||||
to: artifacts/player_handout.md
|
||||
required: false
|
||||
```
|
||||
|
||||
A later enhancement may add opt-in automatic promotion of configured artifacts, but explicit promotion should remain the default.
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Config Model and Defaults
|
||||
|
||||
Add or update the configured artifact model to include:
|
||||
|
||||
- `enabled`;
|
||||
- `depends_on`;
|
||||
- `prompt_id`;
|
||||
- `profile_id`;
|
||||
- `output_path`;
|
||||
- `timeout`;
|
||||
- `render_debug`;
|
||||
- `inputs`;
|
||||
- `vars`.
|
||||
|
||||
Add an internal default artifact output directory in `internal/config/defaults.go`, initially set to `artifacts`.
|
||||
|
||||
Validation rules:
|
||||
|
||||
- artifact names must match a conservative identifier pattern such as `^[a-z][a-z0-9_]*$`;
|
||||
- selected/executable artifacts require `prompt_id` and `output_path`;
|
||||
- configured artifacts that may be referenced while disabled require `output_path`;
|
||||
- configured artifact output paths must be run-relative;
|
||||
- configured artifact output paths must live under the internal artifact output directory;
|
||||
- configured artifact output paths must not escape the run workspace;
|
||||
- `narratio.artifact.<name>` input sources must refer to configured artifact keys;
|
||||
- any `narratio.artifact.<name>` input source must have a matching `depends_on` entry;
|
||||
- `depends_on` entries must refer to configured artifact keys;
|
||||
- dependencies must not contain self-references or executable cycles;
|
||||
- input names and var names must remain compatible with the Scriptorium adapter's validation rules;
|
||||
- unknown YAML fields must continue to fail strict decode.
|
||||
|
||||
Tests:
|
||||
|
||||
- valid single configured artifact;
|
||||
- valid multiple independent artifacts;
|
||||
- valid artifact-to-artifact dependency;
|
||||
- valid dependency on disabled artifact with output path;
|
||||
- invalid artifact name;
|
||||
- missing required fields;
|
||||
- output path outside `artifacts/`;
|
||||
- dependency on missing artifact;
|
||||
- missing `depends_on` for artifact input source;
|
||||
- self-dependency;
|
||||
- cycle detection;
|
||||
- typo in `narratio.artifact.<name>` source;
|
||||
- unknown YAML fields still fail strict decode.
|
||||
|
||||
### Phase 2: CLI Filtering
|
||||
|
||||
Add the `--artifacts` flag and carry the selected artifact set into the run execution options.
|
||||
|
||||
Implementation notes:
|
||||
|
||||
- parse values according to existing CLI conventions;
|
||||
- normalize to artifact key strings;
|
||||
- validate against configured artifact definitions after config load;
|
||||
- make the selected set available to the analyze stage;
|
||||
- reject use with commands or stages where analyze cannot run.
|
||||
|
||||
Tests:
|
||||
|
||||
- no `--artifacts` means all enabled artifacts are selected;
|
||||
- one requested artifact is selected;
|
||||
- multiple requested artifacts are selected;
|
||||
- unknown requested artifact fails;
|
||||
- `--artifacts` does not imply `--force`;
|
||||
- `--artifacts` with already-succeeded analyze stage is skipped unless forced;
|
||||
- `--artifacts` on unsupported stage command fails clearly.
|
||||
|
||||
### Phase 3: Runtime Artifact Catalog
|
||||
|
||||
Introduce an internal artifact catalog abstraction.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- register built-in artifact definitions;
|
||||
- register configured artifact definitions;
|
||||
- map configured artifact keys to `narratio.artifact.<name>` IDs;
|
||||
- track planned, available, and executable artifact states;
|
||||
- expose lookup by canonical source ID;
|
||||
- record generated provenance;
|
||||
- record disabled-from-disk provenance.
|
||||
|
||||
Keep the catalog narrow. It should not execute Scriptorium and should not understand prompt semantics.
|
||||
|
||||
Tests:
|
||||
|
||||
- built-in source lookup;
|
||||
- configured source registration;
|
||||
- duplicate/conflicting source handling;
|
||||
- planned but unavailable artifact lookup;
|
||||
- selected artifact state;
|
||||
- disabled artifact state;
|
||||
- registering an artifact as available after generation;
|
||||
- registering a disabled artifact as available from disk;
|
||||
- resolving a configured artifact from analyze metadata if that behavior is implemented.
|
||||
|
||||
### Phase 4: Resolver Integration
|
||||
|
||||
Update artifact resolution so configured artifact IDs are resolved through the runtime catalog.
|
||||
|
||||
Resolution behavior:
|
||||
|
||||
- built-in sources continue using existing resolver behavior;
|
||||
- configured artifact sources resolve from catalog availability/provenance;
|
||||
- selected configured artifacts become available after generation;
|
||||
- disabled configured artifacts may become available from disk;
|
||||
- missing optional configured artifact inputs are omitted;
|
||||
- missing required configured artifact inputs fail clearly.
|
||||
|
||||
Tests:
|
||||
|
||||
- configured artifact consumes a built-in transcript source;
|
||||
- configured artifact consumes another configured artifact produced earlier in the same analyze run;
|
||||
- configured artifact consumes a disabled artifact resolved from disk;
|
||||
- required disabled artifact missing on disk fails;
|
||||
- required configured artifact missing fails;
|
||||
- optional missing configured artifact is omitted;
|
||||
- reused artifact provenance is recorded distinctly from generated artifact provenance.
|
||||
|
||||
### Phase 5: Analyze Stage Generalization
|
||||
|
||||
Refactor `analyze` to execute selected configured artifacts.
|
||||
|
||||
Implementation notes:
|
||||
|
||||
- remove the hard-coded `session_recap` selection path;
|
||||
- remove the hard-coded rejection of non-`session_recap` artifacts;
|
||||
- preserve skip behavior when Scriptorium config is absent or no artifacts are selected;
|
||||
- build the runtime artifact catalog;
|
||||
- apply `--artifacts` filtering;
|
||||
- validate selected artifacts and their dependencies;
|
||||
- pre-resolve disabled dependencies from disk where required;
|
||||
- compute deterministic dependency order;
|
||||
- execute selected artifacts one at a time in dependency order;
|
||||
- keep render-debug behavior at global and artifact levels;
|
||||
- keep Scriptorium adapter invocation generic;
|
||||
- after each successful run, register the artifact as available in the catalog;
|
||||
- aggregate generated and reused artifact metadata.
|
||||
|
||||
Tests:
|
||||
|
||||
- no Scriptorium config skips;
|
||||
- empty artifact map skips;
|
||||
- no selected artifacts skips;
|
||||
- disabled artifacts do not run;
|
||||
- one selected artifact runs;
|
||||
- multiple independent artifacts run in deterministic order;
|
||||
- dependent selected artifact receives prior selected artifact as input;
|
||||
- dependent selected artifact receives disabled-from-disk artifact as input;
|
||||
- render-debug works for configured artifacts;
|
||||
- Scriptorium validation failure fails the stage;
|
||||
- missing required input fails the stage;
|
||||
- successful outputs are non-empty and recorded;
|
||||
- artifact filter executes only requested artifacts.
|
||||
|
||||
### Phase 6: Manifest and Stage Metadata
|
||||
|
||||
Update analyze-stage metadata and manifest output recording to support dynamic configured artifacts.
|
||||
|
||||
Recommended behavior:
|
||||
|
||||
- every generated configured artifact gets `source_id: narratio.artifact.<name>`;
|
||||
- every generated configured artifact gets a generic output kind such as `scriptorium_artifact`;
|
||||
- reused disabled artifacts are recorded separately from generated artifacts;
|
||||
- metadata is sufficient for debugging, provenance, and future resolver support;
|
||||
- metadata does not create per-artifact resume semantics.
|
||||
|
||||
Because this is a pre-release hard cutover, do not preserve a special legacy `session_recap` output kind unless a current internal test or archive path still requires it temporarily. Prefer updating tests and examples to treat `session_recap` as an ordinary configured artifact.
|
||||
|
||||
Tests:
|
||||
|
||||
- metadata records one generated configured artifact;
|
||||
- metadata records multiple generated configured artifacts;
|
||||
- metadata records reused disabled artifact provenance;
|
||||
- `session_recap` is recorded as a normal configured artifact;
|
||||
- manifest still treats `analyze` as a single succeeded or failed stage;
|
||||
- runner skip behavior remains stage-level.
|
||||
|
||||
### Phase 7: Archive and Promotion Review
|
||||
|
||||
Review archive behavior after dynamic artifacts are recorded.
|
||||
|
||||
Implementation notes:
|
||||
|
||||
- do not automatically promote every configured artifact;
|
||||
- keep `archive.promote_artifacts` explicit;
|
||||
- update default or example promotion rules to use configured `session_recap` output path;
|
||||
- ensure required promotion rules fail clearly when selected artifact generation did not produce a required file.
|
||||
|
||||
Tests:
|
||||
|
||||
- generated artifact can be promoted by explicit archive rule;
|
||||
- required archive promotion fails if selected artifact was not generated and no file exists;
|
||||
- optional archive promotion skips cleanly if file is absent;
|
||||
- hard cutover does not rely on hard-coded `session_recap` generation.
|
||||
|
||||
### Phase 8: Documentation and Examples
|
||||
|
||||
Status: complete.
|
||||
|
||||
Update documentation after the implementation is complete.
|
||||
|
||||
Recommended documentation changes:
|
||||
|
||||
- update `docs/config.md` with the generalized artifact configuration model;
|
||||
- update `docs/internal/artifacts.md` to describe the runtime artifact catalog;
|
||||
- update `docs/stages/analyze.md` to describe generic Scriptorium artifact generation;
|
||||
- update Scriptorium integration docs only if the adapter contract changes;
|
||||
- update full annotated pipeline examples;
|
||||
- add at least one example with multiple artifacts and one dependency;
|
||||
- document `--artifacts` behavior and its relationship to `--force`;
|
||||
- remove documentation stating that only `session_recap` is supported.
|
||||
|
||||
Documentation should make clear that:
|
||||
|
||||
- configured artifact source IDs use `narratio.artifact.<name>`;
|
||||
- `depends_on` uses artifact keys, not full source IDs;
|
||||
- artifact-to-artifact source references require explicit `depends_on`;
|
||||
- disabled artifacts can be reused from disk when required by selected artifacts;
|
||||
- `--artifacts` filters execution but does not imply `--force`;
|
||||
- archive promotion remains explicit;
|
||||
- per-artifact resume is not part of the initial implementation.
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
Because Narratio is pre-release, perform a hard cutover.
|
||||
|
||||
Required changes:
|
||||
|
||||
1. Remove the hard-coded `session_recap` analyze behavior.
|
||||
2. Require `session_recap` to be declared under `pipeline.scriptorium.artifacts.session_recap` if the operator wants a session recap.
|
||||
3. Treat `narratio.artifact.session_recap` as valid only when `session_recap` is a configured artifact key.
|
||||
4. Update config examples to show `session_recap` as a normal configured artifact.
|
||||
5. Update tests to stop assuming that `session_recap` is a built-in analyze artifact.
|
||||
6. Keep archive promotion explicit and path-based.
|
||||
|
||||
Example replacement config:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
config_path: /etc/scriptorium/config.yml
|
||||
timeout: 10m
|
||||
render_debug: false
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd_session.session_recap
|
||||
profile_id: local-gemma-31b
|
||||
output_path: artifacts/session_recap.md
|
||||
timeout: 20m
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
prior_recap:
|
||||
source: previous_session_artifact
|
||||
artifact: artifacts/session_recap.md
|
||||
required: false
|
||||
vars:
|
||||
artifact_title: Session Recap
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
The feature is complete when:
|
||||
|
||||
- operators can define more than one enabled Scriptorium artifact in `pipeline.yml`;
|
||||
- Narratio runs selected artifacts in deterministic dependency order;
|
||||
- configured artifacts are addressable as `narratio.artifact.<name>`;
|
||||
- one configured artifact can consume another configured artifact as an input;
|
||||
- artifact-to-artifact input references require explicit `depends_on`;
|
||||
- disabled artifacts can satisfy dependencies from existing on-disk outputs;
|
||||
- missing required disabled artifacts fail clearly;
|
||||
- optional missing inputs are omitted;
|
||||
- `--artifacts` can selectively execute valid configured artifact names;
|
||||
- `--artifacts` does not imply `--force`;
|
||||
- render-debug behavior works for all configured artifacts;
|
||||
- generated and reused artifacts are recorded in analyze-stage metadata;
|
||||
- `session_recap` is no longer hard-coded and works as a normal configured artifact;
|
||||
- archive promotion remains explicit;
|
||||
- tests cover config validation, dependency sorting, disabled artifact resolution, resolver behavior, CLI filtering, analyze execution, archive interactions, and metadata.
|
||||
|
||||
## Suggested Implementation Order
|
||||
|
||||
1. Config model, defaults, and validation.
|
||||
2. CLI parsing and propagation of `--artifacts` selection.
|
||||
3. Runtime artifact catalog.
|
||||
4. Resolver integration for configured artifacts.
|
||||
5. Analyze stage generalization.
|
||||
6. Stage metadata and manifest output recording.
|
||||
7. Archive behavior review.
|
||||
8. Documentation and examples.
|
||||
|
||||
This order keeps the most static pieces first, then moves into execution behavior once the configuration contract is explicit and well tested.
|
||||
@@ -1,289 +1,300 @@
|
||||
# Troubleshooting
|
||||
|
||||
## Purpose
|
||||
Canonical operator troubleshooting guide for recurring implemented Narratio failures.
|
||||
Operational diagnosis guide for common Narratio failures.
|
||||
|
||||
## Config file discovery failure
|
||||
## Config file not found
|
||||
|
||||
Symptom:
|
||||
- `run`, `plan`, `resume`, or `run-stage` fails with config/session not found.
|
||||
|
||||
Likely Cause:
|
||||
- `pipeline.yml` or `session.yml` is missing from discovery paths.
|
||||
- wrong working directory when relying on `./session.yml`.
|
||||
- command fails to resolve `pipeline.yml`, `campaign.yml`, or `session.yml`.
|
||||
|
||||
Likely causes:
|
||||
|
||||
- missing files in default search paths;
|
||||
- wrong campaign selection;
|
||||
- omitted explicit flags.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
pwd
|
||||
ls -l ./session.yml
|
||||
ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml
|
||||
narratio session plan 2026-04-04
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- pass explicit `--config` and `--session`.
|
||||
- or place files in documented discovery paths.
|
||||
Safe fix:
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/cli.md](./cli.md)
|
||||
- pass explicit `--config`, `--campaign` or `--campaign-file`, and `--session`.
|
||||
|
||||
## Session template rendering failure
|
||||
## Session template placeholders rejected
|
||||
|
||||
Symptom:
|
||||
- load fails with unresolved placeholder or `session_id` mismatch.
|
||||
|
||||
Likely Cause:
|
||||
- templated `session.yml` used without `--session-id`.
|
||||
- rendered `session_id` differs from passed `--session-id`.
|
||||
- load error says session file must be concrete or contains `{{ ... }}` placeholders.
|
||||
|
||||
Likely cause:
|
||||
|
||||
- using template content as runtime session config.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio plan --session ./session.yml --session-id 2026-04-04
|
||||
narratio session validate 2026-04-04 --session /path/session.yml
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- pass `--session-id` when template placeholders are present.
|
||||
- ensure rendered `session_id` matches intended run session id.
|
||||
Safe fix:
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- generate concrete session YAML with `narratio session init`.
|
||||
|
||||
## Strict YAML decode or validation failure
|
||||
## Strict decode or schema validation failure
|
||||
|
||||
Symptom:
|
||||
- config load fails with unknown field or validation error.
|
||||
|
||||
Likely Cause:
|
||||
- typo/stale field name.
|
||||
- missing required fields or invalid constraints.
|
||||
- unknown field / invalid value error during config load.
|
||||
|
||||
Likely cause:
|
||||
|
||||
- stale field name, typo, invalid enum, or invalid duration/path format.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio plan --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04
|
||||
narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- align fields/values to canonical config reference and examples.
|
||||
Safe fix:
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [examples/](../examples/)
|
||||
- align config with [docs/config.md](./config.md) and maintained files under `examples/`.
|
||||
|
||||
## `--artifacts` selection failure
|
||||
## Audio mode conflict
|
||||
|
||||
Symptom:
|
||||
- `run`/`resume`/`run-stage` fails with invalid or unknown artifact selection.
|
||||
|
||||
Likely Cause:
|
||||
- `--artifacts` contains blank names or unknown artifact keys.
|
||||
- `pipeline.scriptorium.artifacts` missing while using `--artifacts`.
|
||||
- validation fails on session audio configuration.
|
||||
|
||||
Likely cause:
|
||||
|
||||
- configured both local and S3 session audio inputs.
|
||||
|
||||
Safe fix:
|
||||
|
||||
- use local mode (`audio_dir` or `audio_files`) or S3 mode (`audio_s3.prefix`), not both.
|
||||
|
||||
## `--artifacts` selection error
|
||||
|
||||
Symptom:
|
||||
|
||||
- unknown artifact key or invalid `--artifacts` usage.
|
||||
|
||||
Likely causes:
|
||||
|
||||
- key not defined in `pipeline.scriptorium.artifacts`;
|
||||
- empty list entry (for example trailing comma);
|
||||
- `run-stage` used with non-`analyze`/`publish` target.
|
||||
|
||||
Safe fix:
|
||||
|
||||
- provide only configured keys and use `--artifacts` with supported commands/stages.
|
||||
|
||||
## Previous-session artifact input missing
|
||||
|
||||
Symptom:
|
||||
|
||||
- prepare/analyze fails due to missing required previous-session artifact cache input.
|
||||
|
||||
Likely causes:
|
||||
|
||||
- missing `session.previous_session_id`;
|
||||
- previous artifact not restored/published for source session.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio run --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts player_handout
|
||||
narratio session validate 2026-04-04
|
||||
narratio session status 2026-04-04
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- use configured artifact keys only.
|
||||
- ensure `pipeline.scriptorium.artifacts` is defined.
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## `run-stage --artifacts` on non-analyze stage
|
||||
|
||||
Symptom:
|
||||
- `run-stage` fails with `--artifacts is only supported for stage "analyze"`.
|
||||
|
||||
Likely Cause:
|
||||
- `--artifacts` was used with a non-`analyze` stage.
|
||||
|
||||
Diagnostics:
|
||||
Safe fix:
|
||||
|
||||
```bash
|
||||
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts session_recap polish
|
||||
narratio session restore 2026-04-04
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- use `--artifacts` only with `run-stage ... analyze`.
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
|
||||
## Configured artifact dependency/input validation failure
|
||||
|
||||
Symptom:
|
||||
- config validation fails for `depends_on`, `narratio.artifact.<name>` source, or artifact output path.
|
||||
|
||||
Likely Cause:
|
||||
- `narratio.artifact.<name>` source missing matching `depends_on` key.
|
||||
- dependency references unknown artifact key.
|
||||
- dependency self-reference or enabled dependency cycle.
|
||||
- artifact output path missing/invalid/outside `artifacts/` root.
|
||||
|
||||
Diagnostics:
|
||||
or rerun prepare after correcting session config:
|
||||
|
||||
```bash
|
||||
narratio plan --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04
|
||||
narratio run-stage prepare 2026-04-04 --force
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- ensure artifact-to-artifact inputs have explicit `depends_on` entries using artifact keys.
|
||||
- ensure referenced artifacts exist and define valid `output_path` values.
|
||||
- keep output paths relative and under `artifacts/`.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/internal/stage-analyze.md](./internal/stage-analyze.md)
|
||||
|
||||
## Required configured artifact input unavailable at analyze time
|
||||
|
||||
Symptom:
|
||||
- analyze fails because configured input source is unavailable.
|
||||
|
||||
Likely Cause:
|
||||
- required upstream configured artifact was not selected/executed this run.
|
||||
- non-executable dependency output file is missing or invalid on disk.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio status --manifest /path/to/manifest.json
|
||||
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts player_handout analyze
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- run analyze with needed artifacts selected.
|
||||
- or ensure dependency output file exists at configured path and is valid.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## Manifest/status path failure
|
||||
|
||||
Symptom:
|
||||
- `status` fails because manifest path is missing, unreadable, or invalid.
|
||||
|
||||
Likely Cause:
|
||||
- wrong manifest path.
|
||||
- manifest removed after cleanup.
|
||||
- `--manifest` omitted.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio status --manifest /path/to/manifest.json
|
||||
ls -l /path/to/manifest.json
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- use manifest path printed by `run`, `resume`, or `run-stage`.
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
|
||||
## Session lock conflict (`.lock`)
|
||||
|
||||
Symptom:
|
||||
- run fails with lock conflict for session workdir.
|
||||
|
||||
Likely Cause:
|
||||
- another Narratio process is running same session.
|
||||
- stale lock from interrupted prior run.
|
||||
- command fails acquiring session lock.
|
||||
|
||||
Likely causes:
|
||||
|
||||
- another process is running for the same session;
|
||||
- stale lock left by interrupted process.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
ls -l {workspace.root}/work/{campaign}/{session_id}/.lock
|
||||
cat {workspace.root}/work/{campaign}/{session_id}/.lock
|
||||
ps aux | grep narratio
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- wait for active run to finish.
|
||||
- if no process is active, remove only stale session `.lock` file.
|
||||
Safe fix:
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/internal/workspace.md](./internal/workspace.md)
|
||||
- wait for active process completion;
|
||||
- remove stale lock only after confirming no live process owns it.
|
||||
|
||||
## Secrets env-dir or credential-env failure
|
||||
## Restore conflict without `--force`
|
||||
|
||||
Symptom:
|
||||
- startup fails loading secrets directory, or stage fails due to missing credential env vars.
|
||||
|
||||
Likely Cause:
|
||||
- invalid `pipeline.secrets.env_dir` path/permissions.
|
||||
- required credential env var unset/empty.
|
||||
- restore fails with conflict count.
|
||||
|
||||
Likely cause:
|
||||
|
||||
- local durable files differ from remote restore sources.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio session restore 2026-04-04 --dry-run
|
||||
```
|
||||
|
||||
Safe fix:
|
||||
|
||||
- review conflicts;
|
||||
- rerun with `--force` only when remote state should overwrite local.
|
||||
|
||||
## Restore current-state discovery failure
|
||||
|
||||
Symptom:
|
||||
|
||||
- restore cannot find current pointer or current manifest.
|
||||
|
||||
Likely causes:
|
||||
|
||||
- no committed publish current state;
|
||||
- storage credentials or connectivity failure.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio session status 2026-04-04
|
||||
narratio session restore 2026-04-04 --dry-run
|
||||
```
|
||||
|
||||
Safe fix:
|
||||
|
||||
- resolve storage/auth issue;
|
||||
- republish from healthy local state if current pointer is missing.
|
||||
|
||||
## Publish output failure
|
||||
|
||||
Symptom:
|
||||
|
||||
- publish fails on missing required source, upload error, or commit write.
|
||||
|
||||
Likely causes:
|
||||
|
||||
- required source file not produced;
|
||||
- lock/state expectations mismatch;
|
||||
- remote storage failure.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio session artifacts 2026-04-04 --remote
|
||||
narratio session status 2026-04-04
|
||||
narratio run-stage publish 2026-04-04 --force
|
||||
```
|
||||
|
||||
Safe fix:
|
||||
|
||||
- regenerate missing sources by rerunning prerequisite stages;
|
||||
- correct publish source/destination rules;
|
||||
- retry after storage failure is resolved.
|
||||
|
||||
## Render markdown source missing
|
||||
|
||||
Symptom:
|
||||
|
||||
- analyze or publish fails because `narratio.transcript.final_markdown` or `narratio.transcript.final_trimmed_markdown` is unavailable.
|
||||
|
||||
Likely causes:
|
||||
|
||||
- render stage was not executed after transcript changes;
|
||||
- render stage failed before producing canonical markdown outputs.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio session status 2026-04-04
|
||||
narratio run-stage render 2026-04-04 --force
|
||||
```
|
||||
|
||||
Safe fix:
|
||||
|
||||
- rerun render and then retry downstream stage(s):
|
||||
|
||||
```bash
|
||||
narratio run-stage render 2026-04-04 --force
|
||||
narratio run-stage analyze 2026-04-04 --force
|
||||
```
|
||||
|
||||
## Secrets or storage credential failure
|
||||
|
||||
Symptom:
|
||||
|
||||
- object-store command fails at initialization/auth.
|
||||
|
||||
Likely causes:
|
||||
|
||||
- invalid `pipeline.secrets.env_dir`;
|
||||
- missing credential environment variables;
|
||||
- invalid S3 endpoint/bucket settings.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
ls -la /path/to/secrets_dir
|
||||
env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM'
|
||||
env | grep -E 'OBJECT_STORAGE|AWS|AUDITA|SCRIPTORIUM'
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- fix secrets directory and credential env vars.
|
||||
Safe fix:
|
||||
|
||||
- correct secret-file path and permissions;
|
||||
- provide required env vars;
|
||||
- keep secret values out of YAML.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## S3-audio prepare failure
|
||||
## S3 audio prepare failure
|
||||
|
||||
Symptom:
|
||||
- `prepare` fails in S3 mode (listing/downloading/no audio/backend error).
|
||||
|
||||
Likely Cause:
|
||||
- wrong `session.inputs.audio_s3.prefix`.
|
||||
- no `.flac` files at resolved prefix.
|
||||
- invalid/missing object-store credentials or backend config.
|
||||
- mixed local+S3 audio input config.
|
||||
- prepare fails listing/downloading session S3 audio.
|
||||
|
||||
Likely causes:
|
||||
|
||||
- incorrect `session.inputs.audio_s3.prefix`;
|
||||
- no matching `.flac` objects;
|
||||
- storage connectivity or permissions failure.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 prepare
|
||||
narratio run-stage prepare 2026-04-04 --force
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- configure exactly one audio source mode.
|
||||
- verify `.flac` files and storage access.
|
||||
Safe fix:
|
||||
|
||||
Links:
|
||||
- verify prefix contents and storage access;
|
||||
- keep session audio mode consistent.
|
||||
|
||||
## References
|
||||
|
||||
- [docs/cli.md](./cli.md)
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
|
||||
## Archive promotion/current-pointer failure
|
||||
|
||||
Symptom:
|
||||
- archive fails on required promotion source missing or pointer write failure.
|
||||
|
||||
Likely Cause:
|
||||
- required promoted file absent (including analyze outputs not generated for this run).
|
||||
- storage upload failed before `current/run_id.txt` commit marker write.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio status --manifest /path/to/manifest.json
|
||||
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 archive
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- rerun or resume upstream stages to generate required files.
|
||||
- adjust promotion rules to match files that must exist.
|
||||
- retry after storage issue is resolved.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
|
||||
- [docs/internal/stage-publish.md](./internal/stage-publish.md)
|
||||
|
||||
1
examples/campaigns/sample-campaign/autocorrect.yml
Normal file
1
examples/campaigns/sample-campaign/autocorrect.yml
Normal file
@@ -0,0 +1 @@
|
||||
[]
|
||||
8
examples/campaigns/sample-campaign/campaign.yml
Normal file
8
examples/campaigns/sample-campaign/campaign.yml
Normal file
@@ -0,0 +1,8 @@
|
||||
campaign_id: sample-campaign
|
||||
session_template_file: ./session.template.yml
|
||||
inputs:
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
players_file: ./players.yml
|
||||
party_file: ./party.yml
|
||||
1
examples/campaigns/sample-campaign/glossary.yml
Normal file
1
examples/campaigns/sample-campaign/glossary.yml
Normal file
@@ -0,0 +1 @@
|
||||
[]
|
||||
2
examples/campaigns/sample-campaign/party.yml
Normal file
2
examples/campaigns/sample-campaign/party.yml
Normal file
@@ -0,0 +1,2 @@
|
||||
- name: Example Hero
|
||||
type: pc
|
||||
2
examples/campaigns/sample-campaign/players.yml
Normal file
2
examples/campaigns/sample-campaign/players.yml
Normal file
@@ -0,0 +1,2 @@
|
||||
- name: Example Player
|
||||
role: player
|
||||
3
examples/campaigns/sample-campaign/session.template.yml
Normal file
3
examples/campaigns/sample-campaign/session.template.yml
Normal file
@@ -0,0 +1,3 @@
|
||||
session_id: "{{ session_id }}"
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
5
examples/campaigns/sample-campaign/speakers.yml
Normal file
5
examples/campaigns/sample-campaign/speakers.yml
Normal file
@@ -0,0 +1,5 @@
|
||||
match:
|
||||
- speaker: "Eric Rakestraw"
|
||||
match:
|
||||
- "Eric_Rakestraw"
|
||||
- "Eric"
|
||||
@@ -4,21 +4,18 @@
|
||||
workspace:
|
||||
# Optional: defaults to /var/lib/narratio.
|
||||
root: /var/lib/narratio/workspace
|
||||
# Optional: remove run-scoped workdir after successful archive commit.
|
||||
cleanup_after_archive: false
|
||||
# Optional: remove run-scoped workdir after successful publish commit.
|
||||
cleanup_after_publish: false
|
||||
|
||||
# Optional: local secret file loader (directory of ENV_VAR_NAME files).
|
||||
# secrets:
|
||||
# env_dir: ./secrets
|
||||
|
||||
storage:
|
||||
# Optional storage backend selector; use "s3" for archive + S3 audio workflows.
|
||||
# Optional storage backend selector; use "s3" for publish + S3 audio workflows.
|
||||
backend: s3
|
||||
# Compatibility fields retained in schema.
|
||||
bucket: ""
|
||||
prefix: ""
|
||||
s3:
|
||||
# Required when using S3 audio or S3 archive uploads.
|
||||
# Required when using S3 audio or S3 publish uploads.
|
||||
bucket: my-dnd-archive
|
||||
# Optional; defaults to "dnd".
|
||||
root_prefix: dnd
|
||||
@@ -30,26 +27,38 @@ storage:
|
||||
access_key_id_env: OBJECT_STORAGE_KEY_ID
|
||||
secret_access_key_env: OBJECT_STORAGE_KEY
|
||||
|
||||
campaigns:
|
||||
# Optional; defaults to /usr/local/share/narratio/campaigns.
|
||||
root: /usr/local/share/narratio/campaigns
|
||||
# Optional command default when --campaign is omitted.
|
||||
default_campaign_id: sample-campaign
|
||||
|
||||
spool:
|
||||
# Optional; defaults to /var/spool/narratio.
|
||||
root: /var/spool/narratio
|
||||
# Optional cleanup of run-scoped spool audio after successful archive commit.
|
||||
delete_audio_after_archive: false
|
||||
# Optional cleanup of run-scoped spool audio after successful publish commit.
|
||||
delete_audio_after_publish: false
|
||||
|
||||
archive:
|
||||
publish:
|
||||
# Optional booleans; defaults are true.
|
||||
enabled: true
|
||||
upload_run: true
|
||||
# Optional promotion rules; required files fail archive if missing.
|
||||
promote_artifacts:
|
||||
- from: transcripts/trimmed.json
|
||||
to: transcripts/trimmed.json
|
||||
# Optional publish output rules; sources use Narratio artifact source IDs.
|
||||
outputs:
|
||||
- source: narratio.transcript.final_trimmed
|
||||
dest: transcripts/final.trimmed.json
|
||||
required: true
|
||||
- from: artifacts/session_recap.md
|
||||
to: artifacts/session_recap.md
|
||||
- source: narratio.transcript.final_markdown
|
||||
dest: transcripts/final.md
|
||||
required: true
|
||||
- from: artifacts/player_handout.md
|
||||
to: artifacts/player_handout.md
|
||||
- source: narratio.transcript.final_trimmed_markdown
|
||||
dest: transcripts/final.trimmed.md
|
||||
required: true
|
||||
- source: narratio.artifact.session_recap
|
||||
dest: artifacts/session_recap.md
|
||||
required: true
|
||||
- source: narratio.artifact.player_handout
|
||||
dest: artifacts/player_handout.md
|
||||
required: false
|
||||
|
||||
whisperx:
|
||||
@@ -96,22 +105,21 @@ audita:
|
||||
|
||||
normalize:
|
||||
# Optional; defaults shown explicitly.
|
||||
output_path: transcripts/normalized.json
|
||||
output_path: transcripts/final.json
|
||||
output_schema: seriatim-intermediate
|
||||
report: true
|
||||
|
||||
trim:
|
||||
# Keep disabled unless bounds prompt integration is configured.
|
||||
enabled: false
|
||||
output_path: transcripts/trimmed.json
|
||||
# Optional; defaults shown explicitly.
|
||||
enabled: true
|
||||
output_path: transcripts/final.trimmed.json
|
||||
bounds:
|
||||
prompt_id: dnd.session_bounds
|
||||
profile_id: local-fast
|
||||
profile_id: ""
|
||||
transcript_input_name: transcript
|
||||
output_path: reports/session_bounds.json
|
||||
output_path: artifacts/session_bounds.json
|
||||
timeout: 10m
|
||||
render_debug: false
|
||||
render_output_path: reports/session_bounds.render.json
|
||||
seriatim:
|
||||
report: false
|
||||
|
||||
@@ -130,12 +138,19 @@ scriptorium:
|
||||
timeout: 10m
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
source: narratio.transcript.final_trimmed
|
||||
required: true
|
||||
previous_recap:
|
||||
source: previous_session_artifact
|
||||
artifact: session_recap
|
||||
path: ""
|
||||
source: narratio.previous_session.artifact.session_recap
|
||||
required: false
|
||||
players:
|
||||
source: narratio.input.players
|
||||
required: true
|
||||
party:
|
||||
source: narratio.input.party
|
||||
required: true
|
||||
glossary:
|
||||
source: narratio.input.glossary
|
||||
required: false
|
||||
vars:
|
||||
session_id: true
|
||||
@@ -160,21 +175,13 @@ scriptorium:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
source: narratio.transcript.final_trimmed
|
||||
required: true
|
||||
vars:
|
||||
session_id: true
|
||||
campaign_name: true
|
||||
output_kind: player_handout
|
||||
|
||||
analyzer:
|
||||
# Optional adapter settings.
|
||||
binary_path: ""
|
||||
timeout: 2m
|
||||
artifacts:
|
||||
output_dir: ""
|
||||
types: []
|
||||
|
||||
notification:
|
||||
# Optional notification settings.
|
||||
backend: ""
|
||||
|
||||
@@ -1,2 +1,6 @@
|
||||
campaigns:
|
||||
root: /usr/local/share/narratio/campaigns
|
||||
default_campaign_id: sample-campaign
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
workspace:
|
||||
root: /var/lib/narratio/workspace
|
||||
cleanup_after_archive: true
|
||||
cleanup_after_publish: true
|
||||
|
||||
storage:
|
||||
backend: s3
|
||||
@@ -11,22 +11,32 @@ storage:
|
||||
access_key_id_env: OBJECT_STORAGE_KEY_ID
|
||||
secret_access_key_env: OBJECT_STORAGE_KEY
|
||||
|
||||
campaigns:
|
||||
root: /usr/local/share/narratio/campaigns
|
||||
default_campaign_id: sample-campaign
|
||||
|
||||
spool:
|
||||
root: /var/spool/narratio
|
||||
delete_audio_after_archive: true
|
||||
delete_audio_after_publish: true
|
||||
|
||||
archive:
|
||||
publish:
|
||||
enabled: true
|
||||
upload_run: true
|
||||
promote_artifacts:
|
||||
- from: transcripts/trimmed.json
|
||||
to: transcripts/trimmed.json
|
||||
outputs:
|
||||
- source: narratio.transcript.final_trimmed
|
||||
dest: transcripts/final.trimmed.json
|
||||
required: true
|
||||
- from: artifacts/session_recap.md
|
||||
to: artifacts/session_recap.md
|
||||
- source: narratio.transcript.final_markdown
|
||||
dest: transcripts/final.md
|
||||
required: true
|
||||
- from: artifacts/player_handout.md
|
||||
to: artifacts/player_handout.md
|
||||
- source: narratio.transcript.final_trimmed_markdown
|
||||
dest: transcripts/final.trimmed.md
|
||||
required: true
|
||||
- source: narratio.artifact.session_recap
|
||||
dest: artifacts/session_recap.md
|
||||
required: true
|
||||
- source: narratio.artifact.player_handout
|
||||
dest: artifacts/player_handout.md
|
||||
required: false
|
||||
|
||||
whisperx:
|
||||
@@ -57,13 +67,10 @@ audita:
|
||||
report: true
|
||||
|
||||
normalize:
|
||||
output_path: transcripts/normalized.json
|
||||
output_path: transcripts/final.json
|
||||
output_schema: seriatim-intermediate
|
||||
report: true
|
||||
|
||||
trim:
|
||||
enabled: false
|
||||
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
config_path: /usr/local/etc/scriptorium/config.yml
|
||||
@@ -78,11 +85,19 @@ scriptorium:
|
||||
timeout: 10m
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
source: narratio.transcript.final_trimmed
|
||||
required: true
|
||||
previous_recap:
|
||||
source: previous_session_artifact
|
||||
artifact: session_recap
|
||||
source: narratio.previous_session.artifact.session_recap
|
||||
required: false
|
||||
players:
|
||||
source: narratio.input.players
|
||||
required: true
|
||||
party:
|
||||
source: narratio.input.party
|
||||
required: true
|
||||
glossary:
|
||||
source: narratio.input.glossary
|
||||
required: false
|
||||
vars:
|
||||
session_id: true
|
||||
@@ -103,14 +118,11 @@ scriptorium:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
source: narratio.transcript.final_trimmed
|
||||
required: true
|
||||
vars:
|
||||
session_id: true
|
||||
output_kind: player_handout
|
||||
|
||||
analyzer:
|
||||
timeout: 2m
|
||||
|
||||
notification:
|
||||
timeout: 30s
|
||||
|
||||
@@ -1,9 +1,5 @@
|
||||
session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
date: 2026-05-03
|
||||
title: Sample Session
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./examples/speakers.yml
|
||||
autocorrect_file: ./examples/autocorrect.yml
|
||||
glossary_file: ./examples/glossary.yml
|
||||
|
||||
@@ -1,10 +1,6 @@
|
||||
session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
date: 2026-05-03
|
||||
title: Sample Session
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
speakers_file: ./examples/speakers.yml
|
||||
autocorrect_file: ./examples/autocorrect.yml
|
||||
glossary_file: ./examples/glossary.yml
|
||||
|
||||
@@ -1,7 +1,3 @@
|
||||
session_id: "{{ session_id }}"
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./examples/speakers.yml
|
||||
autocorrect_file: ./examples/autocorrect.yml
|
||||
glossary_file: ./examples/glossary.yml
|
||||
|
||||
@@ -1,40 +0,0 @@
|
||||
package analyzer
|
||||
|
||||
import "context"
|
||||
|
||||
// NoopRunner is a deterministic no-op analyzer adapter.
|
||||
type NoopRunner struct{}
|
||||
|
||||
// Run returns the requested output path with placeholder metadata.
|
||||
func (n *NoopRunner) Run(ctx context.Context, req AnalyzeRequest) (AnalyzeResult, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return AnalyzeResult{}, err
|
||||
}
|
||||
return AnalyzeResult{ArtifactPath: req.OutputPath, Metadata: map[string]any{"placeholder": true}}, nil
|
||||
}
|
||||
|
||||
// FakeRunner captures analyze requests and returns deterministic responses.
|
||||
type FakeRunner struct {
|
||||
Requests []AnalyzeRequest
|
||||
Err error
|
||||
Result AnalyzeResult
|
||||
}
|
||||
|
||||
// Run records request and returns configured response.
|
||||
func (f *FakeRunner) Run(ctx context.Context, req AnalyzeRequest) (AnalyzeResult, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return AnalyzeResult{}, err
|
||||
}
|
||||
f.Requests = append(f.Requests, req)
|
||||
if f.Err != nil {
|
||||
return AnalyzeResult{}, f.Err
|
||||
}
|
||||
res := f.Result
|
||||
if res.ArtifactPath == "" {
|
||||
res.ArtifactPath = req.OutputPath
|
||||
}
|
||||
if res.Metadata == nil {
|
||||
res.Metadata = map[string]any{"fake": true}
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
@@ -1,31 +0,0 @@
|
||||
package analyzer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestFakeRunnerCapturesRequestAndReturnsPath(t *testing.T) {
|
||||
fake := &FakeRunner{}
|
||||
req := AnalyzeRequest{ArtifactType: "session-log", OutputPath: "artifacts/session-log.md"}
|
||||
|
||||
res, err := fake.Run(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(fake.Requests) != 1 || fake.Requests[0].ArtifactType != "session-log" {
|
||||
t.Fatalf("requests = %#v, want captured request", fake.Requests)
|
||||
}
|
||||
if res.ArtifactPath != req.OutputPath {
|
||||
t.Fatalf("artifact path = %q, want %q", res.ArtifactPath, req.OutputPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFakeRunnerError(t *testing.T) {
|
||||
fake := &FakeRunner{Err: errors.New("boom")}
|
||||
_, err := fake.Run(context.Background(), AnalyzeRequest{})
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
}
|
||||
@@ -1,28 +0,0 @@
|
||||
// Package analyzer declares the adapter contract for artifact analysis generation.
|
||||
package analyzer
|
||||
|
||||
import "context"
|
||||
|
||||
// TODO: implement analyzer integration once the analyzer contract is finalized.
|
||||
|
||||
// Runner is the adapter boundary for analyzer invocations.
|
||||
type Runner interface {
|
||||
Run(ctx context.Context, req AnalyzeRequest) (AnalyzeResult, error)
|
||||
}
|
||||
|
||||
// AnalyzeRequest describes one analyzer artifact generation request.
|
||||
type AnalyzeRequest struct {
|
||||
ArtifactType string
|
||||
ProcessedTranscriptPath string
|
||||
ContextReferences []string
|
||||
OutputPath string
|
||||
GeneratedConfigPath string
|
||||
StdoutLogPath string
|
||||
StderrLogPath string
|
||||
}
|
||||
|
||||
// AnalyzeResult describes analyzer output.
|
||||
type AnalyzeResult struct {
|
||||
ArtifactPath string
|
||||
Metadata map[string]any
|
||||
}
|
||||
@@ -14,7 +14,7 @@ func TestFakeRunnerCapturesRequestAndReturnsPath(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
req := PolishRequest{
|
||||
GeneratedConfigPath: filepath.Join(dir, "config", "audita.yml"),
|
||||
OutputProcessedPath: filepath.Join(dir, "transcripts", "processed.json"),
|
||||
OutputProcessedPath: filepath.Join(dir, "transcripts", "polished.json"),
|
||||
StdoutLogPath: filepath.Join(dir, "logs", "audita.stdout.log"),
|
||||
StderrLogPath: filepath.Join(dir, "logs", "audita.stderr.log"),
|
||||
}
|
||||
|
||||
@@ -52,9 +52,9 @@ func TestSubprocessRunnerSuccessArgsEnvAndValidation(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
req := PolishRequest{
|
||||
GeneratedConfigPath: filepath.Join(dir, "audita.generated.yml"),
|
||||
MergedTranscriptPath: filepath.Join(dir, "merged.json"),
|
||||
MergedTranscriptPath: filepath.Join(dir, "base.json"),
|
||||
GlossaryPath: filepath.Join(dir, "glossary.yml"),
|
||||
OutputProcessedPath: filepath.Join(dir, "processed.json"),
|
||||
OutputProcessedPath: filepath.Join(dir, "polished.json"),
|
||||
ReportPath: filepath.Join(dir, "audita.report.json"),
|
||||
WorkDir: filepath.Join(dir, "artifacts", "audita-work"),
|
||||
StdoutLogPath: filepath.Join(dir, "audita.stdout.log"),
|
||||
@@ -571,7 +571,7 @@ func mustAuditaRunner(t *testing.T, cfg SubprocessRunnerConfig) *SubprocessRunne
|
||||
func auditaReqForTest(t *testing.T, withReport bool) PolishRequest {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
merged := filepath.Join(dir, "merged.json")
|
||||
merged := filepath.Join(dir, "base.json")
|
||||
glossary := filepath.Join(dir, "glossary.yml")
|
||||
writeAuditaTestFile(t, merged, `{"segments":[]}`)
|
||||
writeAuditaTestFile(t, glossary, "terms: []\n")
|
||||
@@ -579,7 +579,7 @@ func auditaReqForTest(t *testing.T, withReport bool) PolishRequest {
|
||||
GeneratedConfigPath: filepath.Join(dir, "audita.generated.yml"),
|
||||
MergedTranscriptPath: merged,
|
||||
GlossaryPath: glossary,
|
||||
OutputProcessedPath: filepath.Join(dir, "processed.json"),
|
||||
OutputProcessedPath: filepath.Join(dir, "polished.json"),
|
||||
WorkDir: filepath.Join(dir, "artifacts", "audita-work"),
|
||||
StdoutLogPath: filepath.Join(dir, "audita.stdout.log"),
|
||||
StderrLogPath: filepath.Join(dir, "audita.stderr.log"),
|
||||
|
||||
@@ -31,7 +31,7 @@ func TestSubprocessRunnerRunSuccessBuildsDeterministicArgsAndCapturesLogs(t *tes
|
||||
ConfigPath: "/etc/scriptorium/config.yml",
|
||||
PromptID: "dnd.session_recap",
|
||||
ProfileID: "local-quality",
|
||||
InputPaths: map[string]string{"transcript": filepath.Join(dir, "processed.json"), "other": filepath.Join(dir, "other.md")},
|
||||
InputPaths: map[string]string{"transcript": filepath.Join(dir, "polished.json"), "other": filepath.Join(dir, "other.md")},
|
||||
Vars: map[string]string{"session_id": "2026-05-03", "campaign_name": "Icewind Dale"},
|
||||
OutputPath: filepath.Join(dir, "artifacts", "session_recap.md"),
|
||||
StdoutLogPath: filepath.Join(dir, "logs", "scriptorium.run.stdout.log"),
|
||||
@@ -180,7 +180,7 @@ func TestSubprocessRunnerRenderSuccess(t *testing.T) {
|
||||
req := RenderArtifactRequest{
|
||||
Binary: wrapper,
|
||||
PromptID: "dnd.session_recap",
|
||||
InputPaths: map[string]string{"transcript": filepath.Join(dir, "processed.json")},
|
||||
InputPaths: map[string]string{"transcript": filepath.Join(dir, "polished.json")},
|
||||
OutputPath: filepath.Join(dir, "artifacts", "session_recap.render.json"),
|
||||
StdoutLogPath: filepath.Join(dir, "logs", "scriptorium.render.stdout.log"),
|
||||
StderrLogPath: filepath.Join(dir, "logs", "scriptorium.render.stderr.log"),
|
||||
@@ -285,7 +285,7 @@ type scriptoriumHelperRecord struct {
|
||||
func runReqForTest(t *testing.T, binary string) RunArtifactRequest {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
transcriptPath := filepath.Join(dir, "processed.json")
|
||||
transcriptPath := filepath.Join(dir, "polished.json")
|
||||
writeScriptoriumFile(t, transcriptPath, `{"segments":[]}`)
|
||||
return RunArtifactRequest{
|
||||
Binary: binary,
|
||||
|
||||
@@ -68,6 +68,26 @@ func (n *NoopRunner) Normalize(ctx context.Context, req NormalizeRequest) (Norma
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Render returns the requested output path with placeholder metadata.
|
||||
func (n *NoopRunner) Render(ctx context.Context, req RenderRequest) (RenderResult, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return RenderResult{}, err
|
||||
}
|
||||
if err := materializeRenderPlaceholders(req); err != nil {
|
||||
return RenderResult{}, err
|
||||
}
|
||||
return RenderResult{
|
||||
OutputRenderedPath: req.OutputRenderedPath,
|
||||
StdoutLogPath: req.StdoutLogPath,
|
||||
StderrLogPath: req.StderrLogPath,
|
||||
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||
InvokedBinary: "noop",
|
||||
Format: req.Format,
|
||||
Title: req.Title,
|
||||
Metadata: map[string]any{"placeholder": true},
|
||||
}, nil
|
||||
}
|
||||
|
||||
// FakeRunner captures merge requests and returns deterministic responses.
|
||||
type FakeRunner struct {
|
||||
Requests []MergeRequest
|
||||
@@ -79,6 +99,9 @@ type FakeRunner struct {
|
||||
TrimRequests []TrimRequest
|
||||
TrimErr error
|
||||
TrimResult TrimResult
|
||||
RenderRequests []RenderRequest
|
||||
RenderErr error
|
||||
RenderResult RenderResult
|
||||
}
|
||||
|
||||
// Run records request and returns configured response.
|
||||
@@ -195,6 +218,46 @@ func (f *FakeRunner) Normalize(ctx context.Context, req NormalizeRequest) (Norma
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// Render records request and returns configured response.
|
||||
func (f *FakeRunner) Render(ctx context.Context, req RenderRequest) (RenderResult, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return RenderResult{}, err
|
||||
}
|
||||
f.RenderRequests = append(f.RenderRequests, req)
|
||||
if f.RenderErr != nil {
|
||||
return RenderResult{}, f.RenderErr
|
||||
}
|
||||
if err := materializeRenderPlaceholders(req); err != nil {
|
||||
return RenderResult{}, err
|
||||
}
|
||||
res := f.RenderResult
|
||||
if res.OutputRenderedPath == "" {
|
||||
res.OutputRenderedPath = req.OutputRenderedPath
|
||||
}
|
||||
if res.StdoutLogPath == "" {
|
||||
res.StdoutLogPath = req.StdoutLogPath
|
||||
}
|
||||
if res.StderrLogPath == "" {
|
||||
res.StderrLogPath = req.StderrLogPath
|
||||
}
|
||||
if res.GeneratedConfigPath == "" {
|
||||
res.GeneratedConfigPath = req.GeneratedConfigPath
|
||||
}
|
||||
if res.InvokedBinary == "" {
|
||||
res.InvokedBinary = "fake"
|
||||
}
|
||||
if res.Format == "" {
|
||||
res.Format = req.Format
|
||||
}
|
||||
if res.Title == "" {
|
||||
res.Title = req.Title
|
||||
}
|
||||
if res.Metadata == nil {
|
||||
res.Metadata = map[string]any{"fake": true}
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
func materializePlaceholders(req MergeRequest) error {
|
||||
if req.OutputMergedTranscriptPath != "" {
|
||||
if err := subprocess.WriteFileAtomic(req.OutputMergedTranscriptPath, []byte(`{"schema":"seriatim.intermediate.v1","segments":[]}`), 0o644); err != nil {
|
||||
@@ -301,3 +364,39 @@ func materializeNormalizePlaceholders(req NormalizeRequest) error {
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func materializeRenderPlaceholders(req RenderRequest) error {
|
||||
if req.OutputRenderedPath != "" {
|
||||
if err := subprocess.WriteFileAtomic(req.OutputRenderedPath, []byte("# Transcript\n\nRendered markdown placeholder.\n"), 0o644); err != nil {
|
||||
return fmt.Errorf("write rendered transcript %q: %w", req.OutputRenderedPath, err)
|
||||
}
|
||||
}
|
||||
if req.GeneratedConfigPath != "" {
|
||||
payload := map[string]any{
|
||||
"schema": "seriatim.generated.v1",
|
||||
"placeholder": true,
|
||||
"command": "render",
|
||||
"input_path": req.InputTranscriptPath,
|
||||
"output_path": req.OutputRenderedPath,
|
||||
"format": req.Format,
|
||||
"title": req.Title,
|
||||
"include_timestamps": req.IncludeTimestamps,
|
||||
"include_segment_ids": req.IncludeSegmentIDs,
|
||||
"include_metadata": req.IncludeMetadata,
|
||||
}
|
||||
if err := subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644); err != nil {
|
||||
return fmt.Errorf("write generated config %q: %w", req.GeneratedConfigPath, err)
|
||||
}
|
||||
}
|
||||
if req.StdoutLogPath != "" {
|
||||
if err := subprocess.WriteFileAtomic(req.StdoutLogPath, []byte("seriatim noop/fake render stdout placeholder\n"), 0o644); err != nil {
|
||||
return fmt.Errorf("write stdout log %q: %w", req.StdoutLogPath, err)
|
||||
}
|
||||
}
|
||||
if req.StderrLogPath != "" {
|
||||
if err := subprocess.WriteFileAtomic(req.StderrLogPath, []byte("seriatim noop/fake render stderr placeholder\n"), 0o644); err != nil {
|
||||
return fmt.Errorf("write stderr log %q: %w", req.StderrLogPath, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -14,7 +14,7 @@ func TestFakeRunnerCapturesRequestAndReturnsPath(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
req := MergeRequest{
|
||||
GeneratedConfigPath: filepath.Join(dir, "config", "seriatim.yml"),
|
||||
OutputMergedTranscriptPath: filepath.Join(dir, "transcripts", "merged.json"),
|
||||
OutputMergedTranscriptPath: filepath.Join(dir, "transcripts", "base.json"),
|
||||
StdoutLogPath: filepath.Join(dir, "logs", "seriatim.stdout.log"),
|
||||
StderrLogPath: filepath.Join(dir, "logs", "seriatim.stderr.log"),
|
||||
}
|
||||
@@ -57,8 +57,8 @@ func TestFakeRunnerTrimCapturesRequestAndReturnsPath(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
req := TrimRequest{
|
||||
GeneratedConfigPath: filepath.Join(dir, "config", "seriatim.trim.yml"),
|
||||
InputTranscriptPath: filepath.Join(dir, "transcripts", "processed.json"),
|
||||
OutputTrimmedPath: filepath.Join(dir, "transcripts", "trimmed.json"),
|
||||
InputTranscriptPath: filepath.Join(dir, "transcripts", "polished.json"),
|
||||
OutputTrimmedPath: filepath.Join(dir, "transcripts", "final.trimmed.json"),
|
||||
KeepSelector: "1-10",
|
||||
StdoutLogPath: filepath.Join(dir, "logs", "seriatim.trim.stdout.log"),
|
||||
StderrLogPath: filepath.Join(dir, "logs", "seriatim.trim.stderr.log"),
|
||||
@@ -105,8 +105,8 @@ func TestFakeRunnerNormalizeCapturesRequestAndReturnsPath(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
req := NormalizeRequest{
|
||||
GeneratedConfigPath: filepath.Join(dir, "config", "seriatim.normalize.yml"),
|
||||
InputTranscriptPath: filepath.Join(dir, "transcripts", "processed.json"),
|
||||
OutputNormalizedPath: filepath.Join(dir, "transcripts", "normalized.json"),
|
||||
InputTranscriptPath: filepath.Join(dir, "transcripts", "polished.json"),
|
||||
OutputNormalizedPath: filepath.Join(dir, "transcripts", "final.json"),
|
||||
OutputSchema: "seriatim-intermediate",
|
||||
ReportPath: filepath.Join(dir, "artifacts", "seriatim.normalize.report.json"),
|
||||
StdoutLogPath: filepath.Join(dir, "logs", "seriatim.normalize.stdout.log"),
|
||||
@@ -148,3 +148,58 @@ func TestFakeRunnerNormalizeError(t *testing.T) {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFakeRunnerRenderCapturesRequestAndReturnsPath(t *testing.T) {
|
||||
fake := &FakeRunner{}
|
||||
dir := t.TempDir()
|
||||
req := RenderRequest{
|
||||
GeneratedConfigPath: filepath.Join(dir, "config", "seriatim.render.yml"),
|
||||
InputTranscriptPath: filepath.Join(dir, "transcripts", "final.trimmed.json"),
|
||||
OutputRenderedPath: filepath.Join(dir, "transcripts", "final.trimmed.md"),
|
||||
Format: "markdown",
|
||||
Title: "Session render",
|
||||
IncludeTimestamps: true,
|
||||
IncludeSegmentIDs: false,
|
||||
IncludeMetadata: true,
|
||||
StdoutLogPath: filepath.Join(dir, "logs", "seriatim.render.stdout.log"),
|
||||
StderrLogPath: filepath.Join(dir, "logs", "seriatim.render.stderr.log"),
|
||||
}
|
||||
|
||||
res, err := fake.Render(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Render() error = %v", err)
|
||||
}
|
||||
if len(fake.RenderRequests) != 1 || fake.RenderRequests[0].GeneratedConfigPath == "" {
|
||||
t.Fatalf("render requests = %#v, want captured request", fake.RenderRequests)
|
||||
}
|
||||
if res.OutputRenderedPath != req.OutputRenderedPath {
|
||||
t.Fatalf("rendered path = %q, want %q", res.OutputRenderedPath, req.OutputRenderedPath)
|
||||
}
|
||||
if res.Format != req.Format {
|
||||
t.Fatalf("format = %q, want %q", res.Format, req.Format)
|
||||
}
|
||||
if res.Title != req.Title {
|
||||
t.Fatalf("title = %q, want %q", res.Title, req.Title)
|
||||
}
|
||||
|
||||
cfgData, err := os.ReadFile(req.GeneratedConfigPath)
|
||||
if err != nil {
|
||||
t.Fatalf("read generated config: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(cfgData), "command: render") {
|
||||
t.Fatalf("generated config = %q, want render command marker", string(cfgData))
|
||||
}
|
||||
for _, path := range []string{req.StdoutLogPath, req.StderrLogPath, req.OutputRenderedPath} {
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
t.Fatalf("expected file %q to exist: %v", path, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFakeRunnerRenderError(t *testing.T) {
|
||||
fake := &FakeRunner{RenderErr: errors.New("boom")}
|
||||
_, err := fake.Render(context.Background(), RenderRequest{})
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
// Package seriatim declares the adapter contract for transcript merge/normalize/trim execution.
|
||||
// Package seriatim declares the adapter contract for transcript merge/normalize/trim/render execution.
|
||||
package seriatim
|
||||
|
||||
import (
|
||||
@@ -6,11 +6,12 @@ import (
|
||||
"time"
|
||||
)
|
||||
|
||||
// Runner is the adapter boundary for seriatim merge/normalize/trim invocations.
|
||||
// Runner is the adapter boundary for seriatim merge/normalize/trim/render invocations.
|
||||
type Runner interface {
|
||||
Run(ctx context.Context, req MergeRequest) (MergeResult, error)
|
||||
Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error)
|
||||
Trim(ctx context.Context, req TrimRequest) (TrimResult, error)
|
||||
Render(ctx context.Context, req RenderRequest) (RenderResult, error)
|
||||
}
|
||||
|
||||
// MergeRequest describes a seriatim merge invocation.
|
||||
@@ -90,3 +91,33 @@ type TrimResult struct {
|
||||
KeepSelector string
|
||||
Metadata map[string]any
|
||||
}
|
||||
|
||||
// RenderRequest describes a seriatim render invocation.
|
||||
type RenderRequest struct {
|
||||
Binary string
|
||||
InputTranscriptPath string
|
||||
OutputRenderedPath string
|
||||
Format string
|
||||
Title string
|
||||
IncludeTimestamps bool
|
||||
IncludeSegmentIDs bool
|
||||
IncludeMetadata bool
|
||||
StdoutLogPath string
|
||||
StderrLogPath string
|
||||
GeneratedConfigPath string
|
||||
Timeout time.Duration
|
||||
}
|
||||
|
||||
// RenderResult describes a render output.
|
||||
type RenderResult struct {
|
||||
OutputRenderedPath string
|
||||
StdoutLogPath string
|
||||
StderrLogPath string
|
||||
GeneratedConfigPath string
|
||||
ExitCode int
|
||||
Duration time.Duration
|
||||
InvokedBinary string
|
||||
Format string
|
||||
Title string
|
||||
Metadata map[string]any
|
||||
}
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/subprocess"
|
||||
)
|
||||
@@ -384,6 +385,96 @@ func (r *SubprocessRunner) Normalize(ctx context.Context, req NormalizeRequest)
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Render executes Seriatim render with deterministic flags and validates non-empty text output.
|
||||
func (r *SubprocessRunner) Render(ctx context.Context, req RenderRequest) (RenderResult, error) {
|
||||
if r == nil {
|
||||
return RenderResult{}, fmt.Errorf("seriatim subprocess runner is nil")
|
||||
}
|
||||
if strings.TrimSpace(req.InputTranscriptPath) == "" {
|
||||
return RenderResult{}, fmt.Errorf("seriatim render input path is required")
|
||||
}
|
||||
if strings.TrimSpace(req.OutputRenderedPath) == "" {
|
||||
return RenderResult{}, fmt.Errorf("seriatim render output path is required")
|
||||
}
|
||||
format := strings.TrimSpace(req.Format)
|
||||
if format == "" {
|
||||
format = "markdown"
|
||||
}
|
||||
if format != "markdown" {
|
||||
return RenderResult{}, fmt.Errorf("seriatim render format %q is unsupported", req.Format)
|
||||
}
|
||||
|
||||
binary := r.binary
|
||||
if strings.TrimSpace(req.Binary) != "" {
|
||||
binary = strings.TrimSpace(req.Binary)
|
||||
}
|
||||
|
||||
timeout := r.timeout
|
||||
if req.Timeout < 0 {
|
||||
return RenderResult{}, fmt.Errorf("seriatim render timeout must be >= 0")
|
||||
}
|
||||
if req.Timeout > 0 {
|
||||
timeout = req.Timeout
|
||||
}
|
||||
|
||||
args := buildRenderArgs(req, format)
|
||||
if req.GeneratedConfigPath != "" {
|
||||
if err := writeRenderInvocationConfig(req, args, binary, timeout, format); err != nil {
|
||||
return RenderResult{}, fmt.Errorf("write seriatim render invocation config %q: %w", req.GeneratedConfigPath, err)
|
||||
}
|
||||
}
|
||||
|
||||
runRes, err := subprocess.Run(ctx, subprocess.RunRequest{
|
||||
Executable: binary,
|
||||
Args: args,
|
||||
Timeout: timeout,
|
||||
StdoutLogPath: req.StdoutLogPath,
|
||||
StderrLogPath: req.StderrLogPath,
|
||||
})
|
||||
if err != nil {
|
||||
return RenderResult{
|
||||
OutputRenderedPath: req.OutputRenderedPath,
|
||||
StdoutLogPath: req.StdoutLogPath,
|
||||
StderrLogPath: req.StderrLogPath,
|
||||
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||
ExitCode: runRes.ExitCode,
|
||||
Duration: runRes.Duration,
|
||||
InvokedBinary: binary,
|
||||
Format: format,
|
||||
Title: req.Title,
|
||||
}, fmt.Errorf("run seriatim render (binary=%q): %w", binary, err)
|
||||
}
|
||||
|
||||
if err := validateNonEmptyTextFile(req.OutputRenderedPath); err != nil {
|
||||
return RenderResult{
|
||||
OutputRenderedPath: req.OutputRenderedPath,
|
||||
StdoutLogPath: req.StdoutLogPath,
|
||||
StderrLogPath: req.StderrLogPath,
|
||||
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||
ExitCode: runRes.ExitCode,
|
||||
Duration: runRes.Duration,
|
||||
InvokedBinary: binary,
|
||||
Format: format,
|
||||
Title: req.Title,
|
||||
}, fmt.Errorf("validate seriatim rendered output %q: %w", req.OutputRenderedPath, err)
|
||||
}
|
||||
|
||||
return RenderResult{
|
||||
OutputRenderedPath: req.OutputRenderedPath,
|
||||
StdoutLogPath: req.StdoutLogPath,
|
||||
StderrLogPath: req.StderrLogPath,
|
||||
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||
ExitCode: runRes.ExitCode,
|
||||
Duration: runRes.Duration,
|
||||
InvokedBinary: binary,
|
||||
Format: format,
|
||||
Title: req.Title,
|
||||
Metadata: map[string]any{
|
||||
"adapter": "seriatim_subprocess",
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (r *SubprocessRunner) buildMergeArgs(req MergeRequest) []string {
|
||||
args := []string{"merge"}
|
||||
|
||||
@@ -480,6 +571,22 @@ func buildNormalizeArgs(req NormalizeRequest, outputSchema string) []string {
|
||||
return args
|
||||
}
|
||||
|
||||
func buildRenderArgs(req RenderRequest, format string) []string {
|
||||
args := []string{
|
||||
"render",
|
||||
"--input-file", req.InputTranscriptPath,
|
||||
"--output-file", req.OutputRenderedPath,
|
||||
"--format", format,
|
||||
"--include-timestamps=" + strconv.FormatBool(req.IncludeTimestamps),
|
||||
"--include-segment-ids=" + strconv.FormatBool(req.IncludeSegmentIDs),
|
||||
"--include-metadata=" + strconv.FormatBool(req.IncludeMetadata),
|
||||
}
|
||||
if strings.TrimSpace(req.Title) != "" {
|
||||
args = append(args, "--title", req.Title)
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
func writeTrimInvocationConfig(req TrimRequest, args []string, binary string, timeout time.Duration) error {
|
||||
payload := map[string]any{
|
||||
"schema": "seriatim.generated.v1",
|
||||
@@ -509,6 +616,24 @@ func writeNormalizeInvocationConfig(req NormalizeRequest, args []string, binary
|
||||
return subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644)
|
||||
}
|
||||
|
||||
func writeRenderInvocationConfig(req RenderRequest, args []string, binary string, timeout time.Duration, format string) error {
|
||||
payload := map[string]any{
|
||||
"schema": "seriatim.generated.v1",
|
||||
"command": "render",
|
||||
"binary": binary,
|
||||
"args": args,
|
||||
"timeout": timeout.String(),
|
||||
"input_path": req.InputTranscriptPath,
|
||||
"output_path": req.OutputRenderedPath,
|
||||
"format": format,
|
||||
"title": req.Title,
|
||||
"include_timestamps": req.IncludeTimestamps,
|
||||
"include_segment_ids": req.IncludeSegmentIDs,
|
||||
"include_metadata": req.IncludeMetadata,
|
||||
}
|
||||
return subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644)
|
||||
}
|
||||
|
||||
func validateJSONFile(path string) error {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
@@ -541,3 +666,20 @@ func validateJSONFileWithSegments(path string) error {
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateNonEmptyTextFile(path string) error {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read file: %w", err)
|
||||
}
|
||||
if len(data) == 0 {
|
||||
return fmt.Errorf("file is empty")
|
||||
}
|
||||
if !utf8.Valid(data) {
|
||||
return fmt.Errorf("file is not valid utf-8 text")
|
||||
}
|
||||
if strings.TrimSpace(string(data)) == "" {
|
||||
return fmt.Errorf("file has no non-whitespace content")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -50,7 +50,7 @@ func TestSubprocessRunnerSuccessWithReportArgsAndEnv(t *testing.T) {
|
||||
req := MergeRequest{
|
||||
GeneratedConfigPath: filepath.Join(dir, "seriatim.generated.yml"),
|
||||
InputTranscriptPaths: []string{filepath.Join(dir, "a.json"), filepath.Join(dir, "b.json")},
|
||||
OutputMergedTranscriptPath: filepath.Join(dir, "merged.json"),
|
||||
OutputMergedTranscriptPath: filepath.Join(dir, "base.json"),
|
||||
ReportPath: filepath.Join(dir, "seriatim.report.json"),
|
||||
SpeakersPath: filepath.Join(dir, "speakers.yml"),
|
||||
AutocorrectPath: filepath.Join(dir, "autocorrect.yml"),
|
||||
@@ -569,6 +569,156 @@ func TestSubprocessRunnerNormalizeInvalidReportJSONFails(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestSubprocessRunnerRenderSuccessInvocationAndProvenance(t *testing.T) {
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("helper wrapper script uses /bin/sh")
|
||||
}
|
||||
|
||||
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||
t.Setenv("SERIATIM_HELPER_MODE", "render_success")
|
||||
recordPath := filepath.Join(t.TempDir(), "record.json")
|
||||
t.Setenv("SERIATIM_HELPER_RECORD_PATH", recordPath)
|
||||
|
||||
wrapper := writeHelperWrapper(t)
|
||||
runner := mustRunner(t, wrapper, false)
|
||||
req := renderReqForTest(t)
|
||||
|
||||
res, err := runner.Render(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Render() error = %v", err)
|
||||
}
|
||||
if res.OutputRenderedPath != req.OutputRenderedPath {
|
||||
t.Fatalf("OutputRenderedPath = %q, want %q", res.OutputRenderedPath, req.OutputRenderedPath)
|
||||
}
|
||||
if res.Format != req.Format {
|
||||
t.Fatalf("Format = %q, want %q", res.Format, req.Format)
|
||||
}
|
||||
if res.Title != req.Title {
|
||||
t.Fatalf("Title = %q, want %q", res.Title, req.Title)
|
||||
}
|
||||
if res.InvokedBinary != wrapper {
|
||||
t.Fatalf("InvokedBinary = %q, want %q", res.InvokedBinary, wrapper)
|
||||
}
|
||||
if res.ExitCode != 0 {
|
||||
t.Fatalf("ExitCode = %d, want 0", res.ExitCode)
|
||||
}
|
||||
if res.Duration <= 0 {
|
||||
t.Fatalf("Duration = %s, want >0", res.Duration)
|
||||
}
|
||||
if res.Metadata == nil || res.Metadata["adapter"] != "seriatim_subprocess" {
|
||||
t.Fatalf("Metadata = %#v, want adapter marker", res.Metadata)
|
||||
}
|
||||
|
||||
if _, err := os.Stat(req.OutputRenderedPath); err != nil {
|
||||
t.Fatalf("rendered output missing: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(req.StdoutLogPath); err != nil {
|
||||
t.Fatalf("stdout log missing: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(req.StderrLogPath); err != nil {
|
||||
t.Fatalf("stderr log missing: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(req.GeneratedConfigPath); err != nil {
|
||||
t.Fatalf("generated config missing: %v", err)
|
||||
}
|
||||
|
||||
rec := readHelperRecord(t, recordPath)
|
||||
wantArgs := []string{
|
||||
"render",
|
||||
"--input-file", req.InputTranscriptPath,
|
||||
"--output-file", req.OutputRenderedPath,
|
||||
"--format", req.Format,
|
||||
"--include-timestamps=true",
|
||||
"--include-segment-ids=true",
|
||||
"--include-metadata=false",
|
||||
"--title", req.Title,
|
||||
}
|
||||
if strings.Join(rec.Args, "\n") != strings.Join(wantArgs, "\n") {
|
||||
t.Fatalf("args = %#v, want %#v", rec.Args, wantArgs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSubprocessRunnerRenderWithoutTitleOmitsTitleArg(t *testing.T) {
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("helper wrapper script uses /bin/sh")
|
||||
}
|
||||
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||
t.Setenv("SERIATIM_HELPER_MODE", "render_success")
|
||||
recordPath := filepath.Join(t.TempDir(), "record.json")
|
||||
t.Setenv("SERIATIM_HELPER_RECORD_PATH", recordPath)
|
||||
|
||||
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||
req := renderReqForTest(t)
|
||||
req.Title = ""
|
||||
if _, err := runner.Render(context.Background(), req); err != nil {
|
||||
t.Fatalf("Render() error = %v", err)
|
||||
}
|
||||
|
||||
rec := readHelperRecord(t, recordPath)
|
||||
for i := 0; i < len(rec.Args); i++ {
|
||||
if rec.Args[i] == "--title" {
|
||||
t.Fatalf("args = %#v, did not expect --title", rec.Args)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSubprocessRunnerRenderSubprocessFailure(t *testing.T) {
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("helper wrapper script uses /bin/sh")
|
||||
}
|
||||
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||
t.Setenv("SERIATIM_HELPER_MODE", "fail")
|
||||
t.Setenv("SERIATIM_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||
|
||||
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||
req := renderReqForTest(t)
|
||||
_, err := runner.Render(context.Background(), req)
|
||||
if err == nil {
|
||||
t.Fatal("Render() error = nil, want non-nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "run seriatim render") {
|
||||
t.Fatalf("error = %q, want subprocess context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestSubprocessRunnerRenderMissingOutputFails(t *testing.T) {
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("helper wrapper script uses /bin/sh")
|
||||
}
|
||||
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||
t.Setenv("SERIATIM_HELPER_MODE", "missing_output")
|
||||
t.Setenv("SERIATIM_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||
|
||||
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||
req := renderReqForTest(t)
|
||||
_, err := runner.Render(context.Background(), req)
|
||||
if err == nil {
|
||||
t.Fatal("Render() error = nil, want non-nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "validate seriatim rendered output") {
|
||||
t.Fatalf("error = %q, want output validation context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestSubprocessRunnerRenderEmptyOutputFails(t *testing.T) {
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("helper wrapper script uses /bin/sh")
|
||||
}
|
||||
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||
t.Setenv("SERIATIM_HELPER_MODE", "render_empty_output")
|
||||
t.Setenv("SERIATIM_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||
|
||||
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||
req := renderReqForTest(t)
|
||||
_, err := runner.Render(context.Background(), req)
|
||||
if err == nil {
|
||||
t.Fatal("Render() error = nil, want non-nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "file is empty") {
|
||||
t.Fatalf("error = %q, want empty-file validation", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestSubprocessRunnerConstructorValidation(t *testing.T) {
|
||||
_, err := NewSubprocessRunnerFromConfigValues("", "10m", "seriatim-intermediate", nil, true, EnvConfig{})
|
||||
if err == nil {
|
||||
@@ -702,6 +852,14 @@ func TestSeriatimSubprocessHelper(t *testing.T) {
|
||||
case "normalize_report_missing":
|
||||
writeSeriatimHelperFile(outputPath, `{"schema":"seriatim.intermediate.v1","segments":[]}`)
|
||||
os.Exit(0)
|
||||
case "render_success":
|
||||
writeSeriatimHelperFile(outputPath, "# Rendered transcript\n\nHello.\n")
|
||||
_, _ = os.Stdout.WriteString("seriatim helper render stdout\n")
|
||||
_, _ = os.Stderr.WriteString("seriatim helper render stderr\n")
|
||||
os.Exit(0)
|
||||
case "render_empty_output":
|
||||
writeSeriatimHelperFile(outputPath, "")
|
||||
os.Exit(0)
|
||||
default:
|
||||
_, _ = os.Stderr.WriteString(fmt.Sprintf("unknown helper mode %q\n", mode))
|
||||
os.Exit(2)
|
||||
@@ -732,7 +890,7 @@ func mergeReqForTest(t *testing.T, withReport bool) MergeRequest {
|
||||
req := MergeRequest{
|
||||
GeneratedConfigPath: filepath.Join(dir, "seriatim.generated.yml"),
|
||||
InputTranscriptPaths: []string{in1, in2},
|
||||
OutputMergedTranscriptPath: filepath.Join(dir, "merged.json"),
|
||||
OutputMergedTranscriptPath: filepath.Join(dir, "base.json"),
|
||||
StdoutLogPath: filepath.Join(dir, "seriatim.stdout.log"),
|
||||
StderrLogPath: filepath.Join(dir, "seriatim.stderr.log"),
|
||||
}
|
||||
@@ -745,11 +903,11 @@ func mergeReqForTest(t *testing.T, withReport bool) MergeRequest {
|
||||
func trimReqForTest(t *testing.T) TrimRequest {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
input := filepath.Join(dir, "processed.json")
|
||||
input := filepath.Join(dir, "polished.json")
|
||||
writeSeriatimFile(t, input, `{"schema":"seriatim.intermediate.v1","segments":[]}`)
|
||||
return TrimRequest{
|
||||
InputTranscriptPath: input,
|
||||
OutputTrimmedPath: filepath.Join(dir, "trimmed.json"),
|
||||
OutputTrimmedPath: filepath.Join(dir, "final.trimmed.json"),
|
||||
KeepSelector: "5-12",
|
||||
GeneratedConfigPath: filepath.Join(dir, "seriatim.trim.generated.yml"),
|
||||
StdoutLogPath: filepath.Join(dir, "seriatim.trim.stdout.log"),
|
||||
@@ -760,12 +918,12 @@ func trimReqForTest(t *testing.T) TrimRequest {
|
||||
func normalizeReqForTest(t *testing.T, withReport bool) NormalizeRequest {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
input := filepath.Join(dir, "processed.json")
|
||||
input := filepath.Join(dir, "polished.json")
|
||||
writeSeriatimFile(t, input, `{"schema":"audita.processed.v1","segments":[]}`)
|
||||
|
||||
req := NormalizeRequest{
|
||||
InputTranscriptPath: input,
|
||||
OutputNormalizedPath: filepath.Join(dir, "normalized.json"),
|
||||
OutputNormalizedPath: filepath.Join(dir, "final.json"),
|
||||
OutputSchema: "seriatim-intermediate",
|
||||
GeneratedConfigPath: filepath.Join(dir, "seriatim.normalize.generated.yml"),
|
||||
StdoutLogPath: filepath.Join(dir, "seriatim.normalize.stdout.log"),
|
||||
@@ -777,6 +935,25 @@ func normalizeReqForTest(t *testing.T, withReport bool) NormalizeRequest {
|
||||
return req
|
||||
}
|
||||
|
||||
func renderReqForTest(t *testing.T) RenderRequest {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
input := filepath.Join(dir, "final.trimmed.json")
|
||||
writeSeriatimFile(t, input, `{"schema":"seriatim.intermediate.v1","segments":[]}`)
|
||||
return RenderRequest{
|
||||
InputTranscriptPath: input,
|
||||
OutputRenderedPath: filepath.Join(dir, "final.trimmed.md"),
|
||||
Format: "markdown",
|
||||
Title: "Session 42",
|
||||
IncludeTimestamps: true,
|
||||
IncludeSegmentIDs: true,
|
||||
IncludeMetadata: false,
|
||||
GeneratedConfigPath: filepath.Join(dir, "seriatim.render.generated.yml"),
|
||||
StdoutLogPath: filepath.Join(dir, "seriatim.render.stdout.log"),
|
||||
StderrLogPath: filepath.Join(dir, "seriatim.render.stderr.log"),
|
||||
}
|
||||
}
|
||||
|
||||
func mustRunner(t *testing.T, binary string, report bool) *SubprocessRunner {
|
||||
t.Helper()
|
||||
coalesce := 3.0
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
// Package storage declares archive/storage backend adapter boundaries.
|
||||
package storage
|
||||
|
||||
import "context"
|
||||
|
||||
// TODO: implement remote storage/archive backends (S3/SFTP/etc.).
|
||||
|
||||
// Backend is the adapter boundary for archive/storage operations.
|
||||
type Backend interface {
|
||||
Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error)
|
||||
}
|
||||
|
||||
// ArchiveItem describes one item to archive.
|
||||
type ArchiveItem struct {
|
||||
Kind string
|
||||
LocalPath string
|
||||
RemoteKey string
|
||||
}
|
||||
|
||||
// ArchiveRequest describes one archive operation.
|
||||
type ArchiveRequest struct {
|
||||
SessionID string
|
||||
ManifestPath string
|
||||
Items []ArchiveItem
|
||||
}
|
||||
|
||||
// ArchiveResult describes archive operation output.
|
||||
type ArchiveResult struct {
|
||||
Archived []ArchiveItem
|
||||
Metadata map[string]any
|
||||
}
|
||||
@@ -10,25 +10,11 @@ import (
|
||||
"time"
|
||||
)
|
||||
|
||||
// NoopBackend is a deterministic no-op archive/storage adapter.
|
||||
type NoopBackend struct{}
|
||||
|
||||
// Archive returns the requested items as archived with placeholder metadata.
|
||||
func (n *NoopBackend) Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return ArchiveResult{}, err
|
||||
}
|
||||
return ArchiveResult{Archived: append([]ArchiveItem(nil), req.Items...), Metadata: map[string]any{"placeholder": true}}, nil
|
||||
}
|
||||
|
||||
// FakeBackend captures archive requests and returns deterministic responses.
|
||||
// FakeBackend provides a deterministic in-memory object store for tests.
|
||||
type FakeBackend struct {
|
||||
Requests []ArchiveRequest
|
||||
Err error
|
||||
Result ArchiveResult
|
||||
|
||||
Objects map[string]FakeObject
|
||||
Uploads []FakeUploadCall
|
||||
Objects map[string]FakeObject
|
||||
Uploads []FakeUploadCall
|
||||
Downloads []FakeDownloadCall
|
||||
|
||||
ListErr error
|
||||
DownloadErr error
|
||||
@@ -43,23 +29,10 @@ type FakeUploadCall struct {
|
||||
Options UploadOptions
|
||||
}
|
||||
|
||||
// Archive records request and returns configured response.
|
||||
func (f *FakeBackend) Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return ArchiveResult{}, err
|
||||
}
|
||||
f.Requests = append(f.Requests, req)
|
||||
if f.Err != nil {
|
||||
return ArchiveResult{}, f.Err
|
||||
}
|
||||
res := f.Result
|
||||
if res.Archived == nil {
|
||||
res.Archived = append([]ArchiveItem(nil), req.Items...)
|
||||
}
|
||||
if res.Metadata == nil {
|
||||
res.Metadata = map[string]any{"fake": true}
|
||||
}
|
||||
return res, nil
|
||||
// FakeDownloadCall captures one download invocation in call order.
|
||||
type FakeDownloadCall struct {
|
||||
Key string
|
||||
LocalPath string
|
||||
}
|
||||
|
||||
// FakeObject is a deterministic fake object-store record.
|
||||
@@ -130,6 +103,10 @@ func (f *FakeBackend) Download(ctx context.Context, key, localPath string) error
|
||||
if !ok {
|
||||
return fmt.Errorf("download object %q: %w", key, os.ErrNotExist)
|
||||
}
|
||||
f.Downloads = append(f.Downloads, FakeDownloadCall{
|
||||
Key: normalizeObjectKey(key),
|
||||
LocalPath: localPath,
|
||||
})
|
||||
|
||||
if err := os.MkdirAll(filepath.Dir(localPath), 0o755); err != nil {
|
||||
return fmt.Errorf("download object %q: create parent directory: %w", key, err)
|
||||
|
||||
@@ -9,30 +9,6 @@ import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestFakeBackendCapturesRequestAndReturnsItems(t *testing.T) {
|
||||
fake := &FakeBackend{}
|
||||
req := ArchiveRequest{SessionID: "s1", Items: []ArchiveItem{{Kind: "artifact", LocalPath: "artifacts/log.md"}}}
|
||||
|
||||
res, err := fake.Archive(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatalf("Archive() error = %v", err)
|
||||
}
|
||||
if len(fake.Requests) != 1 || fake.Requests[0].SessionID != "s1" {
|
||||
t.Fatalf("requests = %#v, want captured request", fake.Requests)
|
||||
}
|
||||
if len(res.Archived) != 1 {
|
||||
t.Fatalf("archived len = %d, want 1", len(res.Archived))
|
||||
}
|
||||
}
|
||||
|
||||
func TestFakeBackendError(t *testing.T) {
|
||||
fake := &FakeBackend{Err: errors.New("boom")}
|
||||
_, err := fake.Archive(context.Background(), ArchiveRequest{})
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFakeBackendListPrefixFiltering(t *testing.T) {
|
||||
fake := &FakeBackend{}
|
||||
fake.SeedObject(FakeObject{Key: "dnd/campaigns/forsaken/audio/a.flac", Data: []byte("a")})
|
||||
@@ -56,7 +32,7 @@ func TestFakeBackendDownload(t *testing.T) {
|
||||
fake.SeedObject(FakeObject{Key: "audio/a.flac", Data: []byte("audio-a")})
|
||||
|
||||
dst := filepath.Join(t.TempDir(), "nested", "a.flac")
|
||||
if err := fake.Download(context.Background(), "audio/a.flac", dst); err != nil {
|
||||
if err := fake.Download(context.Background(), `audio\a.flac`, dst); err != nil {
|
||||
t.Fatalf("Download() error = %v", err)
|
||||
}
|
||||
data, err := os.ReadFile(dst)
|
||||
|
||||
@@ -5,7 +5,7 @@ import (
|
||||
"time"
|
||||
)
|
||||
|
||||
// ObjectStore is a remote object storage boundary used by future prepare/archive work.
|
||||
// ObjectStore is a remote object storage boundary used by prepare, restore, and publish work.
|
||||
//
|
||||
// Key invariant:
|
||||
// callers pass full bucket-relative object keys. Backend implementations do not
|
||||
|
||||
36
internal/adapters/storage/temp_download.go
Normal file
36
internal/adapters/storage/temp_download.go
Normal file
@@ -0,0 +1,36 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// DownloadObjectToTemp downloads an object into a temporary file and returns
|
||||
// the cleaned local path.
|
||||
func DownloadObjectToTemp(ctx context.Context, store ObjectStore, key, pattern string) (string, error) {
|
||||
if store == nil {
|
||||
return "", fmt.Errorf("object store is required")
|
||||
}
|
||||
if strings.TrimSpace(pattern) == "" {
|
||||
return "", fmt.Errorf("temp file pattern is required")
|
||||
}
|
||||
|
||||
tmp, err := os.CreateTemp("", pattern)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("create temp file: %w", err)
|
||||
}
|
||||
path := tmp.Name()
|
||||
if err := tmp.Close(); err != nil {
|
||||
_ = os.Remove(path)
|
||||
return "", fmt.Errorf("close temp file: %w", err)
|
||||
}
|
||||
|
||||
if err := store.Download(ctx, key, path); err != nil {
|
||||
_ = os.Remove(path)
|
||||
return "", err
|
||||
}
|
||||
return filepath.Clean(path), nil
|
||||
}
|
||||
69
internal/adapters/storage/temp_download_test.go
Normal file
69
internal/adapters/storage/temp_download_test.go
Normal file
@@ -0,0 +1,69 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestDownloadObjectToTempSuccess(t *testing.T) {
|
||||
store := &FakeBackend{}
|
||||
store.SeedObject(FakeObject{Key: "sessions/a/current/run_id.txt", Data: []byte("run-123\n")})
|
||||
|
||||
path, err := DownloadObjectToTemp(context.Background(), store, "sessions/a/current/run_id.txt", "narratio-test-*.txt")
|
||||
if err != nil {
|
||||
t.Fatalf("DownloadObjectToTemp() error = %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = os.Remove(path) })
|
||||
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile() error = %v", err)
|
||||
}
|
||||
if string(data) != "run-123\n" {
|
||||
t.Fatalf("downloaded data = %q, want %q", string(data), "run-123\n")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDownloadObjectToTempFailedDownloadRemovesTempFile(t *testing.T) {
|
||||
sentinel := errors.New("download failed")
|
||||
store := &FakeBackend{DownloadErr: sentinel}
|
||||
pattern := "narratio-test-fail-*.txt"
|
||||
before, err := filepath.Glob(filepath.Join(os.TempDir(), "narratio-test-fail-*.txt"))
|
||||
if err != nil {
|
||||
t.Fatalf("Glob(before) error = %v", err)
|
||||
}
|
||||
|
||||
path, err := DownloadObjectToTemp(context.Background(), store, "sessions/a/current/run_id.txt", pattern)
|
||||
if !errors.Is(err, sentinel) {
|
||||
t.Fatalf("DownloadObjectToTemp() error = %v, want %v", err, sentinel)
|
||||
}
|
||||
if strings.TrimSpace(path) != "" {
|
||||
t.Fatalf("DownloadObjectToTemp() path = %q, want empty on failure", path)
|
||||
}
|
||||
after, err := filepath.Glob(filepath.Join(os.TempDir(), "narratio-test-fail-*.txt"))
|
||||
if err != nil {
|
||||
t.Fatalf("Glob(after) error = %v", err)
|
||||
}
|
||||
if len(after) != len(before) {
|
||||
t.Fatalf("temp file count changed after failed download: before=%d after=%d", len(before), len(after))
|
||||
}
|
||||
}
|
||||
|
||||
func TestDownloadObjectToTempCallerContextWrappingPreservesCause(t *testing.T) {
|
||||
sentinel := errors.New("object missing")
|
||||
store := &FakeBackend{DownloadErr: sentinel}
|
||||
|
||||
_, err := DownloadObjectToTemp(context.Background(), store, "sessions/a/current/run_id.txt", "narratio-test-*.txt")
|
||||
if err == nil {
|
||||
t.Fatal("DownloadObjectToTemp() error = nil, want error")
|
||||
}
|
||||
err = fmt.Errorf("download run pointer failed: %w", err)
|
||||
if !errors.Is(err, sentinel) {
|
||||
t.Fatalf("wrapped error does not preserve sentinel cause: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -145,7 +145,7 @@ func TestHTTPClientDoesNotRetryOnNonRetryableStatus(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestHTTPClientInvalidJSONFailsAndDoesNotPromote(t *testing.T) {
|
||||
func TestHTTPClientInvalidJSONFailsAndDoesNotInstallOutput(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
_, _ = w.Write([]byte(`not-json`))
|
||||
}))
|
||||
|
||||
@@ -46,7 +46,7 @@ func (f *artifactSelectionFlag) Normalize() ([]string, error) {
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func validateSelectedAnalyzeArtifacts(cfg *config.Config, selected []string) error {
|
||||
func validateSelectedArtifacts(cfg *config.Config, selected []string) error {
|
||||
if len(selected) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -9,36 +9,80 @@ import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||
)
|
||||
|
||||
func TestExecuteRunStageArtifactsNonAnalyzeFails(t *testing.T) {
|
||||
func TestExecuteRunStageArtifactsUnsupportedStageFails(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{"run-stage", "--config", pipelinePath, "--session", sessionPath, "--artifacts", "session_recap", "polish"},
|
||||
[]string{"run-stage", "polish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `run-stage: --artifacts is only supported for stage "analyze"`) {
|
||||
if !strings.Contains(stderr.String(), `run-stage: --artifacts is only supported for stages "analyze" and "publish"`) {
|
||||
t.Fatalf("stderr = %q, want stage-gating error", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRunStagePublishPropagatesSelectedArtifacts(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
var capturedStages []string
|
||||
var capturedArtifacts []string
|
||||
origExecuteStagesFn := executeStagesFn
|
||||
t.Cleanup(func() {
|
||||
executeStagesFn = origExecuteStagesFn
|
||||
})
|
||||
executeStagesFn = func(_ context.Context, _ *config.Config, stages []stage.Stage, opts RunOptions) (*RunSummary, error) {
|
||||
for _, s := range stages {
|
||||
capturedStages = append(capturedStages, s.Name())
|
||||
}
|
||||
capturedArtifacts = append([]string(nil), opts.SelectedArtifacts...)
|
||||
return &RunSummary{ManifestPath: filepath.Join(workspaceRoot, "manifest.json"), Executed: []string{"publish"}}, nil
|
||||
}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{
|
||||
"run-stage", "publish", "2026-05-03",
|
||||
"--config", pipelinePath,
|
||||
"--campaign-file", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--artifacts", "session_recap",
|
||||
},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if len(capturedStages) != 1 || capturedStages[0] != "publish" {
|
||||
t.Fatalf("captured stages = %#v, want [publish]", capturedStages)
|
||||
}
|
||||
if strings.Join(capturedArtifacts, ",") != "session_recap" {
|
||||
t.Fatalf("captured artifacts = %#v, want [session_recap]", capturedArtifacts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteUnknownArtifactsFailValidation(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{"run", "--config", pipelinePath, "--session", sessionPath, "--artifacts", "unknown_artifact"},
|
||||
[]string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "unknown_artifact"},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
@@ -52,7 +96,7 @@ func TestExecuteUnknownArtifactsFailValidation(t *testing.T) {
|
||||
|
||||
func TestRunStageArtifactsDoesNotImplyForce(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
@@ -65,7 +109,7 @@ func TestRunStageArtifactsDoesNotImplyForce(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
err := RunStage(
|
||||
context.Background(),
|
||||
[]string{"--config", pipelinePath, "--session", sessionPath, "--artifacts", "session_recap,session_recap", "analyze"},
|
||||
[]string{"analyze", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap,session_recap"},
|
||||
&out,
|
||||
)
|
||||
if err != nil {
|
||||
@@ -76,14 +120,14 @@ func TestRunStageArtifactsDoesNotImplyForce(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestResumeArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
|
||||
func TestRunArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
|
||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
|
||||
seed.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||
@@ -91,23 +135,273 @@ func TestResumeArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
err := Resume(
|
||||
err := Run(
|
||||
context.Background(),
|
||||
[]string{"--config", pipelinePath, "--session", sessionPath, "--artifacts", "session_recap"},
|
||||
[]string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
||||
&out,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("Resume() error = %v", err)
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "has no remaining stages") {
|
||||
t.Fatalf("output = %q, want no remaining stages", out.String())
|
||||
if !strings.Contains(out.String(), "executed=0 skipped=10") {
|
||||
t.Fatalf("output = %q, want all stages skipped", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func writeValidConfigFilesWithScriptoriumArtifacts(t *testing.T, workspaceRoot string) (string, string) {
|
||||
func TestExecuteAnalyzeForceRunsAnalyze(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
var capturedStages []string
|
||||
var capturedForce bool
|
||||
origExecuteStagesFn := executeStagesFn
|
||||
t.Cleanup(func() {
|
||||
executeStagesFn = origExecuteStagesFn
|
||||
})
|
||||
executeStagesFn = func(_ context.Context, _ *config.Config, stages []stage.Stage, opts RunOptions) (*RunSummary, error) {
|
||||
for _, s := range stages {
|
||||
capturedStages = append(capturedStages, s.Name())
|
||||
}
|
||||
capturedForce = opts.Force
|
||||
return &RunSummary{
|
||||
ManifestPath: filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json"),
|
||||
Executed: []string{"analyze"},
|
||||
}, nil
|
||||
}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{"analyze", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if len(capturedStages) != 1 || capturedStages[0] != "analyze" {
|
||||
t.Fatalf("captured stages = %#v, want [analyze]", capturedStages)
|
||||
}
|
||||
if !capturedForce {
|
||||
t.Fatal("captured force = false, want true")
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "narratio analyze: executed=1 skipped=0 force=true; manifest=") {
|
||||
t.Fatalf("stdout = %q, want analyze summary", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteAnalyzePropagatesSelectedArtifacts(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
var capturedArtifacts []string
|
||||
origExecuteStagesFn := executeStagesFn
|
||||
t.Cleanup(func() {
|
||||
executeStagesFn = origExecuteStagesFn
|
||||
})
|
||||
executeStagesFn = func(_ context.Context, _ *config.Config, _ []stage.Stage, opts RunOptions) (*RunSummary, error) {
|
||||
capturedArtifacts = append([]string(nil), opts.SelectedArtifacts...)
|
||||
return &RunSummary{ManifestPath: filepath.Join(workspaceRoot, "manifest.json"), Executed: []string{"analyze"}}, nil
|
||||
}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{
|
||||
"analyze",
|
||||
"2026-05-03",
|
||||
"--config", pipelinePath,
|
||||
"--campaign-file", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--artifacts", "player_handout,session_recap",
|
||||
},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if strings.Join(capturedArtifacts, ",") != "player_handout,session_recap" {
|
||||
t.Fatalf("captured artifacts = %#v, want sorted selected artifacts", capturedArtifacts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteAnalyzeUnknownArtifactFailsValidation(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{"analyze", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "unknown_artifact"},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `analyze: --artifacts includes unknown artifact "unknown_artifact"`) {
|
||||
t.Fatalf("stderr = %q, want unknown-artifact validation error", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteAnalyzeRejectsPositionalArgsAndForceFlag(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
args []string
|
||||
want string
|
||||
}{
|
||||
{name: "extra positional", args: []string{"analyze", "2026-05-03", "extra"}, want: "analyze: unexpected positional arguments"},
|
||||
{name: "force flag", args: []string{"analyze", "--force"}, want: "analyze: invalid flags: flag provided but not defined: -force"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(tc.args, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), tc.want) {
|
||||
t.Fatalf("stderr = %q, want %q", stderr.String(), tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteAnalyzeMissingConfigUsesRunStageLoadingPath(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"analyze", "2026-05-03"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "analyze: no pipeline config path provided and no default pipeline config found; searched:") {
|
||||
t.Fatalf("stderr = %q, want pipeline discovery error", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutePublishForceRunsPublish(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
var capturedStages []string
|
||||
var capturedForce bool
|
||||
var capturedArtifacts []string
|
||||
origExecuteStagesFn := executeStagesFn
|
||||
t.Cleanup(func() {
|
||||
executeStagesFn = origExecuteStagesFn
|
||||
})
|
||||
executeStagesFn = func(_ context.Context, _ *config.Config, stages []stage.Stage, opts RunOptions) (*RunSummary, error) {
|
||||
for _, s := range stages {
|
||||
capturedStages = append(capturedStages, s.Name())
|
||||
}
|
||||
capturedForce = opts.Force
|
||||
capturedArtifacts = append([]string(nil), opts.SelectedArtifacts...)
|
||||
return &RunSummary{
|
||||
ManifestPath: filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json"),
|
||||
Executed: []string{"publish"},
|
||||
}, nil
|
||||
}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{"publish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if len(capturedStages) != 1 || capturedStages[0] != "publish" {
|
||||
t.Fatalf("captured stages = %#v, want [publish]", capturedStages)
|
||||
}
|
||||
if !capturedForce {
|
||||
t.Fatal("captured force = false, want true")
|
||||
}
|
||||
if strings.Join(capturedArtifacts, ",") != "session_recap" {
|
||||
t.Fatalf("captured artifacts = %#v, want [session_recap]", capturedArtifacts)
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "narratio publish: executed=1 skipped=0 force=true; manifest=") {
|
||||
t.Fatalf("stdout = %q, want publish summary", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutePublishRejectsUnsupportedArgsAndFlags(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
args []string
|
||||
want string
|
||||
}{
|
||||
{name: "extra positional", args: []string{"publish", "2026-05-03", "extra"}, want: "publish: unexpected positional arguments"},
|
||||
{name: "force flag", args: []string{"publish", "--force"}, want: "publish: invalid flags: flag provided but not defined: -force"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(tc.args, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), tc.want) {
|
||||
t.Fatalf("stderr = %q, want %q", stderr.String(), tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutePublishUnknownArtifactFailsValidation(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{"publish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "unknown_artifact"},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `publish: --artifacts includes unknown artifact "unknown_artifact"`) {
|
||||
t.Fatalf("stderr = %q, want unknown-artifact validation error", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutePublishMissingConfigUsesRunStageLoadingPath(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"publish", "2026-05-03"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "publish: no pipeline config path provided and no default pipeline config found; searched:") {
|
||||
t.Fatalf("stderr = %q, want pipeline discovery error", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteUsageIncludesAnalyzeAndPublish(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(nil, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "analyze") {
|
||||
t.Fatalf("stderr = %q, want usage to include analyze", stderr.String())
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "publish") {
|
||||
t.Fatalf("stderr = %q, want usage to include publish", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func writeValidConfigFilesWithScriptoriumArtifacts(t *testing.T, workspaceRoot string) (string, string, string) {
|
||||
t.Helper()
|
||||
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
f, err := os.OpenFile(pipelinePath, os.O_APPEND|os.O_WRONLY, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("open pipeline config for append: %v", err)
|
||||
@@ -136,5 +430,5 @@ scriptorium:
|
||||
if _, err := f.WriteString(extra); err != nil {
|
||||
t.Fatalf("append scriptorium config: %v", err)
|
||||
}
|
||||
return pipelinePath, sessionPath
|
||||
return pipelinePath, campaignPath, sessionPath
|
||||
}
|
||||
|
||||
@@ -64,7 +64,7 @@ func TestArtifactSelectionFlagNormalize(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateSelectedAnalyzeArtifacts(t *testing.T) {
|
||||
func TestValidateSelectedArtifacts(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
cfg *config.Config
|
||||
@@ -114,7 +114,7 @@ func TestValidateSelectedAnalyzeArtifacts(t *testing.T) {
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := validateSelectedAnalyzeArtifacts(tt.cfg, tt.selected)
|
||||
err := validateSelectedArtifacts(tt.cfg, tt.selected)
|
||||
if tt.wantErr != "" {
|
||||
if err == nil {
|
||||
t.Fatalf("error = nil, want %q", tt.wantErr)
|
||||
|
||||
44
internal/app/campaign_config_path.go
Normal file
44
internal/app/campaign_config_path.go
Normal file
@@ -0,0 +1,44 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func resolveCampaignConfigPath(pipelineCfg *config.PipelineConfig, campaignIDFlag, campaignFileFlag string) (string, error) {
|
||||
campaignID := strings.TrimSpace(campaignIDFlag)
|
||||
campaignFile := strings.TrimSpace(campaignFileFlag)
|
||||
if campaignID != "" && campaignFile != "" {
|
||||
return "", fmt.Errorf("--campaign and --campaign-file are mutually exclusive")
|
||||
}
|
||||
if campaignFile != "" {
|
||||
return filepath.Clean(campaignFile), nil
|
||||
}
|
||||
if campaignID == "" && pipelineCfg != nil {
|
||||
campaignID = strings.TrimSpace(pipelineCfg.Campaigns.DefaultCampaignID)
|
||||
}
|
||||
if campaignID == "" {
|
||||
return "", fmt.Errorf("no campaign selected; pass --campaign <id> or set pipeline.campaigns.default_campaign_id")
|
||||
}
|
||||
if err := validateCampaignIDToken(campaignID); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if pipelineCfg == nil || strings.TrimSpace(pipelineCfg.Campaigns.Root) == "" {
|
||||
return "", fmt.Errorf("pipeline.campaigns.root is required to select campaign %q", campaignID)
|
||||
}
|
||||
return filepath.Clean(filepath.Join(pipelineCfg.Campaigns.Root, campaignID, "campaign.yml")), nil
|
||||
}
|
||||
|
||||
func validateCampaignIDToken(campaignID string) error {
|
||||
if filepath.IsAbs(campaignID) ||
|
||||
strings.Contains(campaignID, "/") ||
|
||||
strings.Contains(campaignID, `\`) ||
|
||||
campaignID == "." ||
|
||||
campaignID == ".." {
|
||||
return fmt.Errorf("campaign id %q must be a single path segment", campaignID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
84
internal/app/campaign_config_path_test.go
Normal file
84
internal/app/campaign_config_path_test.go
Normal file
@@ -0,0 +1,84 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func TestResolveCampaignConfigPathCampaignFileWins(t *testing.T) {
|
||||
explicit := filepath.Join(t.TempDir(), "custom-campaign.yml")
|
||||
got, err := resolveCampaignConfigPath(&config.PipelineConfig{}, "", explicit)
|
||||
if err != nil {
|
||||
t.Fatalf("resolveCampaignConfigPath() error = %v", err)
|
||||
}
|
||||
if got != explicit {
|
||||
t.Fatalf("path = %q, want explicit path %q", got, explicit)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveCampaignConfigPathUsesSelectedCampaignID(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
pipelineCfg := &config.PipelineConfig{}
|
||||
pipelineCfg.Campaigns.Root = dir
|
||||
|
||||
got, err := resolveCampaignConfigPath(pipelineCfg, "icewind", "")
|
||||
if err != nil {
|
||||
t.Fatalf("resolveCampaignConfigPath() error = %v", err)
|
||||
}
|
||||
want := filepath.Join(dir, "icewind", "campaign.yml")
|
||||
if got != filepath.Clean(want) {
|
||||
t.Fatalf("path = %q, want %q", got, filepath.Clean(want))
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveCampaignConfigPathUsesDefaultCampaignID(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
pipelineCfg := &config.PipelineConfig{}
|
||||
pipelineCfg.Campaigns.Root = dir
|
||||
pipelineCfg.Campaigns.DefaultCampaignID = "dilfs"
|
||||
|
||||
got, err := resolveCampaignConfigPath(pipelineCfg, "", "")
|
||||
if err != nil {
|
||||
t.Fatalf("resolveCampaignConfigPath() error = %v", err)
|
||||
}
|
||||
want := filepath.Join(dir, "dilfs", "campaign.yml")
|
||||
if got != filepath.Clean(want) {
|
||||
t.Fatalf("path = %q, want %q", got, filepath.Clean(want))
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveCampaignConfigPathRejectsCampaignIDAndFile(t *testing.T) {
|
||||
_, err := resolveCampaignConfigPath(&config.PipelineConfig{}, "dilfs", filepath.Join(t.TempDir(), "campaign.yml"))
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "mutually exclusive") {
|
||||
t.Fatalf("error = %q, want mutual exclusion", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveCampaignConfigPathRequiresCampaignSelection(t *testing.T) {
|
||||
_, err := resolveCampaignConfigPath(&config.PipelineConfig{}, "", "")
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "no campaign selected") {
|
||||
t.Fatalf("error = %q, want missing selection guidance", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveCampaignConfigPathRejectsPathLikeCampaignID(t *testing.T) {
|
||||
pipelineCfg := &config.PipelineConfig{}
|
||||
pipelineCfg.Campaigns.Root = t.TempDir()
|
||||
|
||||
_, err := resolveCampaignConfigPath(pipelineCfg, "../icewind", "")
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "single path segment") {
|
||||
t.Fatalf("error = %q, want path segment guidance", err.Error())
|
||||
}
|
||||
}
|
||||
281
internal/app/clean.go
Normal file
281
internal/app/clean.go
Normal file
@@ -0,0 +1,281 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
// Clean removes local workspace/spool state while preserving durable cache
|
||||
// state unless cache cleanup is explicitly requested.
|
||||
func Clean(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("clean", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var flags commonConfigFlags
|
||||
var all bool
|
||||
var dryRun bool
|
||||
var clearCache bool
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
fs.BoolVar(&all, "all", false, "clean all local session work/spool state")
|
||||
fs.BoolVar(&dryRun, "dry-run", false, "print cleanup targets without deleting")
|
||||
fs.BoolVar(&clearCache, "clear-cache", false, "also clear durable S3 audio cache entries")
|
||||
if err := parseSessionAwareFlags("clean", fs, args, &flags.sessionID); err != nil {
|
||||
return err
|
||||
}
|
||||
if all {
|
||||
return cleanAllLocal(flags, dryRun, clearCache, out)
|
||||
}
|
||||
return cleanSession(ctx, flags, dryRun, clearCache, out)
|
||||
}
|
||||
|
||||
func cleanSession(ctx context.Context, flags commonConfigFlags, dryRun, clearCache bool, out io.Writer) error {
|
||||
if strings.TrimSpace(flags.sessionID) == "" {
|
||||
return fmt.Errorf("clean: session_id is required unless --all is set")
|
||||
}
|
||||
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||
if err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
||||
return fmt.Errorf("clean: resolved pipeline and session config are required")
|
||||
}
|
||||
campaign := strings.TrimSpace(cfg.Session.Campaign)
|
||||
sessionID := strings.TrimSpace(cfg.Session.SessionID)
|
||||
if campaign == "" || sessionID == "" {
|
||||
return fmt.Errorf("clean: campaign and session_id are required")
|
||||
}
|
||||
|
||||
if dryRun {
|
||||
fmt.Fprintf(out, "Clean plan for %s/%s\n", campaign, sessionID)
|
||||
} else {
|
||||
fmt.Fprintf(out, "Cleaned %s/%s\n", campaign, sessionID)
|
||||
}
|
||||
|
||||
workDir := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, campaign, sessionID)
|
||||
spoolDir := artifacts.SessionSpoolDir(cfg.Pipeline.Spool.Root, campaign, sessionID)
|
||||
if err := reportCleanScopedDir(out, cfg.Pipeline.Workspace.Root, workDir, "clean.workspace.session", dryRun); err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
if err := reportCleanScopedDir(out, cfg.Pipeline.Spool.Root, spoolDir, "clean.spool.session", dryRun); err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
|
||||
if clearCache {
|
||||
if err := cleanSessionAudioCache(ctx, cfg, dryRun, out); err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
} else {
|
||||
fmt.Fprintln(out, "Cache: preserved")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func cleanAllLocal(flags commonConfigFlags, dryRun, clearCache bool, out io.Writer) error {
|
||||
if strings.TrimSpace(flags.campaignPath) != "" ||
|
||||
strings.TrimSpace(flags.campaignFilePath) != "" ||
|
||||
strings.TrimSpace(flags.sessionPath) != "" ||
|
||||
strings.TrimSpace(flags.sessionID) != "" ||
|
||||
strings.TrimSpace(flags.previousSessionID) != "" {
|
||||
return fmt.Errorf("clean: --all cannot be combined with --campaign, --campaign-file, --session, a session_id, or --previous-session-id")
|
||||
}
|
||||
resolvedPipelinePath, err := resolvePipelineConfigPath(flags.pipelinePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
pipelineCfg, err := config.LoadPipeline(resolvedPipelinePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
|
||||
if dryRun {
|
||||
fmt.Fprintln(out, "Clean plan for all local sessions")
|
||||
} else {
|
||||
fmt.Fprintln(out, "Cleaned all local sessions")
|
||||
}
|
||||
|
||||
workRoot := filepath.Join(pipelineCfg.Workspace.Root, config.PathWorkDirSegment)
|
||||
if err := reportCleanScopedDir(out, pipelineCfg.Workspace.Root, workRoot, "clean.workspace.all", dryRun); err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
if err := reportCleanRootChildren(out, pipelineCfg.Spool.Root, "clean.spool.all", dryRun); err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
|
||||
if clearCache {
|
||||
if err := cleanAllAudioCache(pipelineCfg, dryRun, out); err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
} else {
|
||||
fmt.Fprintln(out, "Cache: preserved")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func reportCleanScopedDir(out io.Writer, root, target, policy string, dryRun bool) error {
|
||||
dir, err := validateScopedDir(root, target, policy)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if dryRun {
|
||||
if dir.Exists {
|
||||
fmt.Fprintf(out, "Would delete: %s\n", dir.TargetAbs)
|
||||
} else {
|
||||
fmt.Fprintf(out, "Would skip missing: %s\n", dir.TargetAbs)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if !dir.Exists {
|
||||
fmt.Fprintf(out, "Missing: %s\n", dir.TargetAbs)
|
||||
return nil
|
||||
}
|
||||
if err := os.RemoveAll(dir.TargetAbs); err != nil {
|
||||
return fmt.Errorf("cleanup policy %s: remove %q: %w", policy, dir.TargetAbs, err)
|
||||
}
|
||||
fmt.Fprintf(out, "Deleted: %s\n", dir.TargetAbs)
|
||||
return nil
|
||||
}
|
||||
|
||||
func reportCleanRootChildren(out io.Writer, root, policy string, dryRun bool) error {
|
||||
rootAbs, entries, err := cleanableRootChildren(root, policy)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(entries) == 0 {
|
||||
if dryRun {
|
||||
fmt.Fprintf(out, "Would skip empty: %s\n", rootAbs)
|
||||
} else {
|
||||
fmt.Fprintf(out, "Empty: %s\n", rootAbs)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
for _, entry := range entries {
|
||||
if dryRun {
|
||||
fmt.Fprintf(out, "Would delete: %s\n", entry)
|
||||
continue
|
||||
}
|
||||
if err := os.RemoveAll(entry); err != nil {
|
||||
return fmt.Errorf("cleanup policy %s: remove %q: %w", policy, entry, err)
|
||||
}
|
||||
fmt.Fprintf(out, "Deleted: %s\n", entry)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func cleanableRootChildren(root, policy string) (string, []string, error) {
|
||||
rootAbs, exists, err := validateCleanRoot(root, policy)
|
||||
if err != nil {
|
||||
return "", nil, err
|
||||
}
|
||||
if !exists {
|
||||
return rootAbs, nil, nil
|
||||
}
|
||||
entries, err := os.ReadDir(rootAbs)
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("cleanup policy %s: read root %q: %w", policy, rootAbs, err)
|
||||
}
|
||||
out := make([]string, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
path := filepath.Join(rootAbs, entry.Name())
|
||||
info, err := os.Lstat(path)
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("cleanup policy %s: stat child %q: %w", policy, path, err)
|
||||
}
|
||||
if info.Mode()&os.ModeSymlink != 0 {
|
||||
return "", nil, fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, path)
|
||||
}
|
||||
out = append(out, path)
|
||||
}
|
||||
return rootAbs, out, nil
|
||||
}
|
||||
|
||||
func cleanSessionAudioCache(ctx context.Context, cfg *config.Config, dryRun bool, out io.Writer) error {
|
||||
if cfg.Session.Inputs.AudioS3 == nil {
|
||||
fmt.Fprintln(out, "Cache: skipped (session does not use audio_s3)")
|
||||
return nil
|
||||
}
|
||||
if cfg.Pipeline.Storage.S3 == nil || strings.TrimSpace(cfg.Pipeline.Storage.S3.Bucket) == "" {
|
||||
return fmt.Errorf("clear cache requires pipeline.storage.s3.bucket")
|
||||
}
|
||||
store, err := newCommandObjectStore(ctx, cfg, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("initialize object store for cache cleanup: %w", err)
|
||||
}
|
||||
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
audioPrefix := artifacts.S3AudioPrefix(sessionPrefix, cfg.Session.Inputs.AudioS3.Prefix)
|
||||
objects, err := store.List(ctx, audioPrefix)
|
||||
if err != nil {
|
||||
return fmt.Errorf("list s3 audio objects under %q: %w", audioPrefix, err)
|
||||
}
|
||||
count := 0
|
||||
for _, obj := range objects {
|
||||
key := strings.TrimSpace(obj.Key)
|
||||
if key == "" || strings.HasSuffix(key, "/") || !cleanIsFlac(key) {
|
||||
continue
|
||||
}
|
||||
cachePath, err := artifacts.S3AudioCachePath(cfg.Pipeline.Cache.Root, cfg.Pipeline.Storage.S3.Bucket, key)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
deleted, err := reportCleanScopedFile(out, cfg.Pipeline.Cache.Root, cachePath, "clean.cache.session", dryRun)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if deleted {
|
||||
count++
|
||||
}
|
||||
}
|
||||
if count == 0 {
|
||||
fmt.Fprintf(out, "Cache: no cached S3 audio files found for %s\n", audioPrefix)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func cleanAllAudioCache(cfg *config.PipelineConfig, dryRun bool, out io.Writer) error {
|
||||
if cfg.Storage.S3 == nil || strings.TrimSpace(cfg.Storage.S3.Bucket) == "" {
|
||||
return fmt.Errorf("clear cache requires pipeline.storage.s3.bucket")
|
||||
}
|
||||
namespaceDir, err := artifacts.S3AudioCacheNamespaceDir(cfg.Cache.Root, cfg.Storage.S3.Bucket, cfg.Storage.S3.RootPrefix)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return reportCleanScopedDir(out, cfg.Cache.Root, namespaceDir, "clean.cache.all", dryRun)
|
||||
}
|
||||
|
||||
func reportCleanScopedFile(out io.Writer, root, target, policy string, dryRun bool) (bool, error) {
|
||||
file, err := validateScopedFile(root, target, policy)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if dryRun {
|
||||
if file.Exists {
|
||||
fmt.Fprintf(out, "Would delete cache file: %s\n", file.TargetAbs)
|
||||
return true, nil
|
||||
}
|
||||
fmt.Fprintf(out, "Would skip missing cache file: %s\n", file.TargetAbs)
|
||||
return false, nil
|
||||
}
|
||||
if !file.Exists {
|
||||
fmt.Fprintf(out, "Missing cache file: %s\n", file.TargetAbs)
|
||||
return false, nil
|
||||
}
|
||||
if err := os.Remove(file.TargetAbs); err != nil {
|
||||
return false, fmt.Errorf("cleanup policy %s: remove %q: %w", policy, file.TargetAbs, err)
|
||||
}
|
||||
fmt.Fprintf(out, "Deleted cache file: %s\n", file.TargetAbs)
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func validateScopedFile(root, target, policy string) (scopedDir, error) {
|
||||
return validateScopedTarget(root, target, policy, false)
|
||||
}
|
||||
|
||||
func cleanIsFlac(path string) bool {
|
||||
return strings.EqualFold(filepath.Ext(path), ".flac")
|
||||
}
|
||||
255
internal/app/clean_test.go
Normal file
255
internal/app/clean_test.go
Normal file
@@ -0,0 +1,255 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
)
|
||||
|
||||
func TestExecuteCleanSessionDeletesWorkAndSpoolButPreservesCache(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
workDir := artifacts.SessionWorkDirForCampaign(workspaceRoot, "sample-campaign", "2026-05-03")
|
||||
spoolDir := artifacts.SessionSpoolDir(filepath.Join(workspaceRoot, "spool"), "sample-campaign", "2026-05-03")
|
||||
cachePath, err := artifacts.S3AudioCachePath(filepath.Join(workspaceRoot, "cache"), "test-bucket", "dnd/campaigns/sample-campaign/sessions/2026-05-03/audio/alice.flac")
|
||||
if err != nil {
|
||||
t.Fatalf("S3AudioCachePath() error = %v", err)
|
||||
}
|
||||
mustWriteTestFile(t, filepath.Join(workDir, "manifest.json"), "{}")
|
||||
mustWriteTestFile(t, filepath.Join(spoolDir, "run-1", "audio", "alice.flac"), "audio")
|
||||
mustWriteTestFile(t, cachePath, "cached-audio")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
cleanAssertMissing(t, workDir)
|
||||
cleanAssertMissing(t, spoolDir)
|
||||
cleanAssertExists(t, cachePath)
|
||||
if !strings.Contains(stdout.String(), "Cache: preserved") {
|
||||
t.Fatalf("stdout = %q, want cache preserved", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteCleanSessionDryRunDeletesNothing(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
workDir := artifacts.SessionWorkDirForCampaign(workspaceRoot, "sample-campaign", "2026-05-03")
|
||||
spoolDir := artifacts.SessionSpoolDir(filepath.Join(workspaceRoot, "spool"), "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workDir, "manifest.json"), "{}")
|
||||
mustWriteTestFile(t, filepath.Join(spoolDir, "run-1", "audio", "alice.flac"), "audio")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--dry-run"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
cleanAssertExists(t, workDir)
|
||||
cleanAssertExists(t, spoolDir)
|
||||
if !strings.Contains(stdout.String(), "Would delete:") {
|
||||
t.Fatalf("stdout = %q, want dry-run delete plan", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteCleanMissingSessionPathsSucceeds(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "Missing:") {
|
||||
t.Fatalf("stdout = %q, want missing path output", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteCleanSessionClearCacheRemovesOnlyS3AudioCache(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
if err := os.WriteFile(sessionPath, []byte(`session_id: 2026-05-03
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
`), 0o644); err != nil {
|
||||
t.Fatalf("write session: %v", err)
|
||||
}
|
||||
|
||||
audioKey := "dnd/campaigns/sample-campaign/sessions/2026-05-03/audio/alice.flac"
|
||||
fake := &storage.FakeBackend{}
|
||||
fake.SeedObject(storage.FakeObject{Key: audioKey, Data: []byte("audio")})
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||
|
||||
cacheRoot := filepath.Join(workspaceRoot, "cache")
|
||||
cachePath, err := artifacts.S3AudioCachePath(cacheRoot, "test-bucket", audioKey)
|
||||
if err != nil {
|
||||
t.Fatalf("S3AudioCachePath() error = %v", err)
|
||||
}
|
||||
otherCachePath, err := artifacts.S3AudioCachePath(cacheRoot, "test-bucket", "dnd/campaigns/other/sessions/2026-05-03/audio/bob.flac")
|
||||
if err != nil {
|
||||
t.Fatalf("S3AudioCachePath() error = %v", err)
|
||||
}
|
||||
mustWriteTestFile(t, cachePath, "cached-audio")
|
||||
mustWriteTestFile(t, otherCachePath, "other-audio")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--clear-cache"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
cleanAssertMissing(t, cachePath)
|
||||
cleanAssertExists(t, otherCachePath)
|
||||
if storeInitCalls != 1 {
|
||||
t.Fatalf("object store init calls = %d, want 1", storeInitCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteCleanLocalAudioClearCacheIsNoop(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--clear-cache"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "Cache: skipped (session does not use audio_s3)") {
|
||||
t.Fatalf("stdout = %q, want local audio cache no-op", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteCleanAllDeletesWorkAndSpoolContentsButPreservesCache(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, _, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
workRoot := filepath.Join(workspaceRoot, "work")
|
||||
spoolRoot := filepath.Join(workspaceRoot, "spool")
|
||||
cachePath := filepath.Join(workspaceRoot, "cache", "keep.txt")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "sample-campaign", "2026-05-03", "manifest.json"), "{}")
|
||||
mustWriteTestFile(t, filepath.Join(spoolRoot, "sample-campaign", "2026-05-03", "run-1", "audio", "alice.flac"), "audio")
|
||||
mustWriteTestFile(t, cachePath, "cache")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "--config", pipelinePath, "--all"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
cleanAssertMissing(t, workRoot)
|
||||
cleanAssertExists(t, spoolRoot)
|
||||
cleanAssertMissing(t, filepath.Join(spoolRoot, "sample-campaign"))
|
||||
cleanAssertExists(t, cachePath)
|
||||
}
|
||||
|
||||
func TestExecuteCleanAllClearCacheRemovesS3AudioNamespaceOnly(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, _, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
cacheRoot := filepath.Join(workspaceRoot, "cache")
|
||||
audioCachePath, err := artifacts.S3AudioCachePath(cacheRoot, "test-bucket", "dnd/campaigns/sample-campaign/sessions/2026-05-03/audio/alice.flac")
|
||||
if err != nil {
|
||||
t.Fatalf("S3AudioCachePath() error = %v", err)
|
||||
}
|
||||
otherCachePath, err := artifacts.S3AudioCachePath(cacheRoot, "test-bucket", "other-root/campaigns/sample-campaign/sessions/2026-05-03/audio/alice.flac")
|
||||
if err != nil {
|
||||
t.Fatalf("S3AudioCachePath() error = %v", err)
|
||||
}
|
||||
mustWriteTestFile(t, audioCachePath, "cached-audio")
|
||||
mustWriteTestFile(t, otherCachePath, "other-cache")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "--config", pipelinePath, "--all", "--clear-cache"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
cleanAssertMissing(t, audioCachePath)
|
||||
cleanAssertExists(t, otherCachePath)
|
||||
}
|
||||
|
||||
func TestExecuteCleanAllRejectsSessionScopedFlags(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--all"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "--all cannot be combined") {
|
||||
t.Fatalf("stderr = %q, want --all conflict", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanRequiresSessionID(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "session_id is required unless --all is set") {
|
||||
t.Fatalf("stderr = %q, want missing session-id", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanRejectsUnsafeTargets(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
outside := t.TempDir()
|
||||
if err := reportCleanScopedDir(&bytes.Buffer{}, root, filepath.Join(outside, "target"), "test.outside", false); err == nil {
|
||||
t.Fatal("outside target error = nil, want error")
|
||||
}
|
||||
if err := reportCleanScopedDir(&bytes.Buffer{}, root, root, "test.root", false); err == nil {
|
||||
t.Fatal("root target error = nil, want error")
|
||||
}
|
||||
filePath := filepath.Join(root, "file.txt")
|
||||
mustWriteTestFile(t, filePath, "file")
|
||||
if err := reportCleanScopedDir(&bytes.Buffer{}, root, filePath, "test.file", false); err == nil {
|
||||
t.Fatal("file target error = nil, want error")
|
||||
}
|
||||
symlinkPath := filepath.Join(root, "link")
|
||||
if err := os.Symlink(filepath.Join(root, "missing"), symlinkPath); err != nil {
|
||||
t.Fatalf("Symlink() error = %v", err)
|
||||
}
|
||||
if err := reportCleanScopedDir(&bytes.Buffer{}, root, symlinkPath, "test.symlink", false); err == nil {
|
||||
t.Fatal("symlink target error = nil, want error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestClearIsNotCommandAlias(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clear"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `unknown command: "clear"`) {
|
||||
t.Fatalf("stderr = %q, want unknown clear command", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func cleanAssertExists(t *testing.T, path string) {
|
||||
t.Helper()
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
t.Fatalf("expected %q to exist: %v", path, err)
|
||||
}
|
||||
}
|
||||
|
||||
func cleanAssertMissing(t *testing.T, path string) {
|
||||
t.Helper()
|
||||
if _, err := os.Stat(path); !os.IsNotExist(err) {
|
||||
t.Fatalf("expected %q to be missing, stat err=%v", path, err)
|
||||
}
|
||||
}
|
||||
82
internal/app/cleanup_targets.go
Normal file
82
internal/app/cleanup_targets.go
Normal file
@@ -0,0 +1,82 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func validateScopedTarget(root, target, policy string, requireDir bool) (scopedDir, error) {
|
||||
cleanRoot := strings.TrimSpace(root)
|
||||
cleanTarget := strings.TrimSpace(target)
|
||||
if cleanRoot == "" {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
||||
}
|
||||
if cleanTarget == "" {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target path is required", policy)
|
||||
}
|
||||
|
||||
rootAbs, err := filepath.Abs(cleanRoot)
|
||||
if err != nil {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
||||
}
|
||||
targetAbs, err := filepath.Abs(cleanTarget)
|
||||
if err != nil {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve target %q: %w", policy, cleanTarget, err)
|
||||
}
|
||||
|
||||
rel, err := filepath.Rel(rootAbs, targetAbs)
|
||||
if err != nil {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: relative path from %q to %q: %w", policy, rootAbs, targetAbs, err)
|
||||
}
|
||||
if rel == "." {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete root directory %q", policy, rootAbs)
|
||||
}
|
||||
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete path outside root: root=%q target=%q", policy, rootAbs, targetAbs)
|
||||
}
|
||||
|
||||
info, err := os.Lstat(targetAbs)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: false}, nil
|
||||
}
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: stat target %q: %w", policy, targetAbs, err)
|
||||
}
|
||||
if info.Mode()&os.ModeSymlink != 0 {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, targetAbs)
|
||||
}
|
||||
if requireDir && !info.IsDir() {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is not a directory", policy, targetAbs)
|
||||
}
|
||||
if !requireDir && info.IsDir() {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is a directory", policy, targetAbs)
|
||||
}
|
||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: true}, nil
|
||||
}
|
||||
|
||||
func validateCleanRoot(root, policy string) (string, bool, error) {
|
||||
cleanRoot := strings.TrimSpace(root)
|
||||
if cleanRoot == "" {
|
||||
return "", false, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
||||
}
|
||||
rootAbs, err := filepath.Abs(cleanRoot)
|
||||
if err != nil {
|
||||
return "", false, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
||||
}
|
||||
info, err := os.Lstat(rootAbs)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return rootAbs, false, nil
|
||||
}
|
||||
return "", false, fmt.Errorf("cleanup policy %s: stat root %q: %w", policy, rootAbs, err)
|
||||
}
|
||||
if info.Mode()&os.ModeSymlink != 0 {
|
||||
return "", false, fmt.Errorf("cleanup policy %s: refusing to clean symlink root %q", policy, rootAbs)
|
||||
}
|
||||
if !info.IsDir() {
|
||||
return "", false, fmt.Errorf("cleanup policy %s: root %q is not a directory", policy, rootAbs)
|
||||
}
|
||||
return rootAbs, true, nil
|
||||
}
|
||||
103
internal/app/cleanup_targets_test.go
Normal file
103
internal/app/cleanup_targets_test.go
Normal file
@@ -0,0 +1,103 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestCleanValidateScopedDirAndFile(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
dirTarget := filepath.Join(root, "runs", "run-1")
|
||||
fileTarget := filepath.Join(root, "cache", "a.flac")
|
||||
if err := os.MkdirAll(dirTarget, 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll(dirTarget) error = %v", err)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(fileTarget), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll(file parent) error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(fileTarget, []byte("audio"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile(fileTarget) error = %v", err)
|
||||
}
|
||||
|
||||
if _, err := validateScopedDir(root, dirTarget, "test.dir"); err != nil {
|
||||
t.Fatalf("validateScopedDir() error = %v", err)
|
||||
}
|
||||
if _, err := validateScopedFile(root, fileTarget, "test.file"); err != nil {
|
||||
t.Fatalf("validateScopedFile() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanValidateScopedTargetSafetyRules(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
outside := t.TempDir()
|
||||
target := filepath.Join(root, "runs", "run-1")
|
||||
if err := os.MkdirAll(target, 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll(target) error = %v", err)
|
||||
}
|
||||
fileTarget := filepath.Join(root, "cache", "a.flac")
|
||||
if err := os.MkdirAll(filepath.Dir(fileTarget), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll(file parent) error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(fileTarget, []byte("audio"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile(fileTarget) error = %v", err)
|
||||
}
|
||||
symlinkTarget := filepath.Join(root, "symlink")
|
||||
if err := os.Symlink(target, symlinkTarget); err != nil {
|
||||
t.Fatalf("Symlink() error = %v", err)
|
||||
}
|
||||
|
||||
if _, err := validateScopedDir(root, root, "test.root"); err == nil || !strings.Contains(err.Error(), "refusing to delete root directory") {
|
||||
t.Fatalf("validateScopedDir(root) error = %v, want root deletion rejection", err)
|
||||
}
|
||||
if _, err := validateScopedDir(root, filepath.Join(outside, "x"), "test.outside"); err == nil || !strings.Contains(err.Error(), "outside root") {
|
||||
t.Fatalf("validateScopedDir(outside) error = %v, want outside-root rejection", err)
|
||||
}
|
||||
if _, err := validateScopedDir(root, fileTarget, "test.file-as-dir"); err == nil || !strings.Contains(err.Error(), "is not a directory") {
|
||||
t.Fatalf("validateScopedDir(file) error = %v, want not-a-directory rejection", err)
|
||||
}
|
||||
if _, err := validateScopedFile(root, target, "test.dir-as-file"); err == nil || !strings.Contains(err.Error(), "is a directory") {
|
||||
t.Fatalf("validateScopedFile(dir) error = %v, want is-a-directory rejection", err)
|
||||
}
|
||||
if _, err := validateScopedDir(root, symlinkTarget, "test.symlink"); err == nil || !strings.Contains(err.Error(), "refusing to delete symlink path") {
|
||||
t.Fatalf("validateScopedDir(symlink) error = %v, want symlink rejection", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanableRootChildrenRejectsSymlinkChild(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
realChild := filepath.Join(root, "runs")
|
||||
if err := os.MkdirAll(realChild, 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll(realChild) error = %v", err)
|
||||
}
|
||||
if err := os.Symlink(realChild, filepath.Join(root, "link")); err != nil {
|
||||
t.Fatalf("Symlink() error = %v", err)
|
||||
}
|
||||
|
||||
_, _, err := cleanableRootChildren(root, "test.root.children")
|
||||
if err == nil || !strings.Contains(err.Error(), "refusing to delete symlink path") {
|
||||
t.Fatalf("cleanableRootChildren() error = %v, want symlink rejection", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanValidateScopedTargetMissing(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
missingDir := filepath.Join(root, "runs", "missing")
|
||||
got, err := validateScopedDir(root, missingDir, "test.missing")
|
||||
if err != nil {
|
||||
t.Fatalf("validateScopedDir(missing) error = %v", err)
|
||||
}
|
||||
if got.Exists {
|
||||
t.Fatalf("validateScopedDir(missing).Exists = true, want false")
|
||||
}
|
||||
|
||||
missingFile := filepath.Join(root, "cache", "missing.flac")
|
||||
got, err = validateScopedFile(root, missingFile, "test.missing.file")
|
||||
if err != nil {
|
||||
t.Fatalf("validateScopedFile(missing) error = %v", err)
|
||||
}
|
||||
if got.Exists {
|
||||
t.Fatalf("validateScopedFile(missing).Exists = true, want false")
|
||||
}
|
||||
}
|
||||
@@ -7,7 +7,7 @@ import (
|
||||
"strings"
|
||||
)
|
||||
|
||||
var supportedCommands = []string{"run", "plan", "status", "resume", "run-stage"}
|
||||
var supportedCommands = []string{"run", "run-stage", "analyze", "publish", "clean", "session"}
|
||||
|
||||
// Execute dispatches CLI commands and returns a process exit code.
|
||||
func Execute(args []string, stdout, stderr io.Writer) int {
|
||||
@@ -24,14 +24,16 @@ func Execute(args []string, stdout, stderr io.Writer) int {
|
||||
switch cmd {
|
||||
case "run":
|
||||
err = Run(ctx, cmdArgs, stdout)
|
||||
case "plan":
|
||||
err = Plan(ctx, cmdArgs, stdout)
|
||||
case "status":
|
||||
err = Status(ctx, cmdArgs, stdout)
|
||||
case "resume":
|
||||
err = Resume(ctx, cmdArgs, stdout)
|
||||
case "run-stage":
|
||||
err = RunStage(ctx, cmdArgs, stdout)
|
||||
case "analyze":
|
||||
err = Analyze(ctx, cmdArgs, stdout)
|
||||
case "publish":
|
||||
err = Publish(ctx, cmdArgs, stdout)
|
||||
case "session":
|
||||
err = Session(ctx, cmdArgs, stdout)
|
||||
case "clean":
|
||||
err = Clean(ctx, cmdArgs, stdout)
|
||||
default:
|
||||
fmt.Fprintf(stderr, "unknown command: %q\n\n", cmd)
|
||||
printUsage(stderr)
|
||||
|
||||
@@ -24,19 +24,17 @@ func TestExecuteValidCommands(t *testing.T) {
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
manifestPath := writeManifestPathForExecute(t)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
args []string
|
||||
wantOut string
|
||||
}{
|
||||
{name: "run", args: []string{"run", "--config", pipelinePath, "--session", sessionPath}, wantOut: "narratio run: session 2026-05-03; executed=9 skipped=0; manifest="},
|
||||
{name: "plan", args: []string{"plan", "--config", pipelinePath, "--session", sessionPath}, wantOut: "prepare: skip\ntranscribe: skip\nmerge: skip\npolish: skip\nnormalize: skip\ntrim: skip\nanalyze: skip\narchive: skip\nnotify: skip"},
|
||||
{name: "status", args: []string{"status", "--manifest", manifestPath}, wantOut: "session_id: 2026-05-03"},
|
||||
{name: "resume", args: []string{"resume", "--config", pipelinePath, "--session", sessionPath}, wantOut: "narratio resume: session 2026-05-03 has no remaining stages"},
|
||||
{name: "run-stage", args: []string{"run-stage", "--config", pipelinePath, "--session", sessionPath, "polish"}, wantOut: "narratio run-stage: stage=polish executed=0 skipped=1 force=false; manifest="},
|
||||
{name: "run", args: []string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run: session 2026-05-03; executed=10 skipped=0; manifest="},
|
||||
{name: "session plan", args: []string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "prepare: skip\ntranscribe: skip\nmerge: skip\npolish: skip\nnormalize: skip\ntrim: skip\nrender: skip\nanalyze: skip\npublish: skip\nnotify: skip"},
|
||||
{name: "session status", args: []string{"session", "status", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "Session: 2026-05-03"},
|
||||
{name: "run-stage", args: []string{"run-stage", "polish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run-stage: stage=polish executed=0 skipped=1 force=false; manifest="},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
@@ -64,13 +62,13 @@ func TestExecuteMissingRequiredFlags(t *testing.T) {
|
||||
args []string
|
||||
want string
|
||||
}{
|
||||
{name: "run missing flags", args: []string{"run"}, want: "run: no pipeline config path provided and no default pipeline config found; searched:"},
|
||||
{name: "plan missing flags", args: []string{"plan"}, want: "plan: no pipeline config path provided and no default pipeline config found; searched:"},
|
||||
{name: "status missing flags", args: []string{"status"}, want: "status: --manifest is required"},
|
||||
{name: "resume missing flags", args: []string{"resume"}, want: "resume: no pipeline config path provided and no default pipeline config found; searched:"},
|
||||
{name: "run-stage missing name", args: []string{"run-stage", "--config", "a", "--session", "b"}, want: "run-stage: expected exactly one stage name"},
|
||||
{name: "run-stage missing config flags", args: []string{"run-stage", "polish"}, want: "run-stage: no pipeline config path provided and no default pipeline config found; searched:"},
|
||||
{name: "run missing config uses defaults", args: []string{"run", "--session", "session.yml"}, want: "run: no pipeline config path provided and no default pipeline config found; searched:"},
|
||||
{name: "run missing session", args: []string{"run"}, want: "run: session_id is required"},
|
||||
{name: "plan old top-level removed", args: []string{"plan"}, want: `unknown command: "plan"`},
|
||||
{name: "status old top-level removed", args: []string{"status"}, want: `unknown command: "status"`},
|
||||
{name: "resume removed", args: []string{"resume"}, want: `unknown command: "resume"`},
|
||||
{name: "run-stage missing name", args: []string{"run-stage", "--config", "a", "--session", "b"}, want: "run-stage: expected stage name and session_id"},
|
||||
{name: "run-stage missing session", args: []string{"run-stage", "polish"}, want: "run-stage: expected stage name and session_id"},
|
||||
{name: "run missing config uses defaults", args: []string{"run", "2026-05-03", "--session", "session.yml"}, want: "run: no pipeline config path provided and no default pipeline config found; searched:"},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
@@ -94,12 +92,12 @@ func TestExecuteMissingRequiredFlags(t *testing.T) {
|
||||
|
||||
func TestExecuteRunStageUnknownFails(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := Execute([]string{"run-stage", "--config", pipelinePath, "--session", sessionPath, "unknown"}, &stdout, &stderr)
|
||||
code := Execute([]string{"run-stage", "unknown", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
@@ -108,16 +106,32 @@ func TestExecuteRunStageUnknownFails(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRunStageNormalizeIsAccepted(t *testing.T) {
|
||||
func TestExecuteRunStageArchiveAliasFails(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := Execute([]string{"run-stage", "--config", pipelinePath, "--session", sessionPath, "normalize"}, &stdout, &stderr)
|
||||
code := Execute([]string{"run-stage", "archive", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `unknown stage "archive"`) {
|
||||
t.Fatalf("stderr = %q, want unknown stage alias error", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRunStageNormalizeIsAccepted(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "polished.json"), `{"segments":[{"id":1}]}`)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := Execute([]string{"run-stage", "normalize", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
@@ -136,19 +150,19 @@ func TestExecuteRunStageTranscribeUsesConfiguredWhisperXServer(t *testing.T) {
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := Execute([]string{"run-stage", "--config", pipelinePath, "--session", sessionPath, "prepare"}, &stdout, &stderr)
|
||||
code := Execute([]string{"run-stage", "prepare", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("prepare exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
|
||||
code = Execute([]string{"run-stage", "--config", pipelinePath, "--session", sessionPath, "--force", "transcribe"}, &stdout, &stderr)
|
||||
code = Execute([]string{"run-stage", "transcribe", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--force"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("transcribe exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
@@ -188,6 +202,7 @@ func TestExecuteRunStagePolishLoadsCredentialFromSecretsDir(t *testing.T) {
|
||||
t.Setenv("GO_WANT_APP_AUDITA_HELPER", "1")
|
||||
|
||||
pipelinePath := filepath.Join(configDir, "pipeline.yml")
|
||||
campaignPath := writeAppTestCampaignConfig(t, configDir)
|
||||
sessionPath := filepath.Join(configDir, "session.yml")
|
||||
pipelineYAML := `workspace:
|
||||
root: ` + workspaceRoot + `
|
||||
@@ -202,8 +217,6 @@ seriatim:
|
||||
audita:
|
||||
binary: ` + auditaBinary + `
|
||||
llm_api_key_env: OPENROUTER_API_KEY
|
||||
analyzer:
|
||||
timeout: 20m
|
||||
notification:
|
||||
timeout: 10s
|
||||
`
|
||||
@@ -214,6 +227,8 @@ inputs:
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
players_file: ./players.yml
|
||||
party_file: ./party.yml
|
||||
`
|
||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||
t.Fatalf("write pipeline.yml: %v", err)
|
||||
@@ -234,12 +249,12 @@ inputs:
|
||||
})
|
||||
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", sessionID)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"schema":"seriatim-intermediate","segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "base.json"), `{"schema":"seriatim-intermediate","segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "[]\n")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"run-stage", "--config", pipelinePath, "--session", sessionPath, "--force", "polish"}, &stdout, &stderr)
|
||||
code := Execute([]string{"run-stage", "polish", sessionID, "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--force"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
@@ -252,6 +267,7 @@ func TestExecuteRunFailsWhenConfiguredSecretsDirMissing(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
configDir := t.TempDir()
|
||||
pipelinePath := filepath.Join(configDir, "pipeline.yml")
|
||||
campaignPath := writeAppTestCampaignConfig(t, configDir)
|
||||
sessionPath := filepath.Join(configDir, "session.yml")
|
||||
|
||||
pipelineYAML := `workspace:
|
||||
@@ -266,8 +282,6 @@ seriatim:
|
||||
binary: seriatim
|
||||
audita:
|
||||
binary: audita
|
||||
analyzer:
|
||||
timeout: 20m
|
||||
notification:
|
||||
timeout: 10s
|
||||
`
|
||||
@@ -278,6 +292,8 @@ inputs:
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
players_file: ./players.yml
|
||||
party_file: ./party.yml
|
||||
`
|
||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||
t.Fatalf("write pipeline.yml: %v", err)
|
||||
@@ -288,7 +304,7 @@ inputs:
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"run", "--config", pipelinePath, "--session", sessionPath}, &stdout, &stderr)
|
||||
code := Execute([]string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
@@ -305,24 +321,109 @@ func TestExecuteUsesDefaultPipelineConfigPathWhenConfigFlagOmitted(t *testing.T)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
originalDefaults := append([]string(nil), config.DefaultPipelineConfigSearchPaths...)
|
||||
config.DefaultPipelineConfigSearchPaths = []string{pipelinePath}
|
||||
defer func() {
|
||||
config.DefaultPipelineConfigSearchPaths = originalDefaults
|
||||
}()
|
||||
_ = campaignPath
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"run", "--session", sessionPath}, &stdout, &stderr)
|
||||
code := Execute([]string{"run", "2026-05-03", "--session", sessionPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "narratio run: session 2026-05-03; executed=9 skipped=0; manifest=") {
|
||||
if !strings.Contains(stdout.String(), "narratio run: session 2026-05-03; executed=10 skipped=0; manifest=") {
|
||||
t.Fatalf("stdout = %q, want successful run output", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteMissingCampaignConfigReportsRegistryPath(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
if err := os.Remove(campaignPath); err != nil {
|
||||
t.Fatalf("remove campaign config: %v", err)
|
||||
}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"run", "2026-05-03", "--config", pipelinePath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if stdout.Len() != 0 {
|
||||
t.Fatalf("stdout = %q, want empty", stdout.String())
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "load campaign config") {
|
||||
t.Fatalf("stderr = %q, want campaign discovery failure", stderr.String())
|
||||
}
|
||||
if !strings.Contains(stderr.String(), filepath.ToSlash(filepath.Join("campaigns", "sample-campaign", "campaign.yml"))) {
|
||||
t.Fatalf("stderr = %q, want campaign registry path", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteUsesPipelineDefaultCampaignID(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, _, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "status", "2026-05-03", "--config", pipelinePath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "Campaign: sample-campaign") {
|
||||
t.Fatalf("stdout = %q, want default campaign", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteCampaignIDSelectsRegistryCampaign(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
campaignRoot := filepath.Dir(filepath.Dir(campaignPath))
|
||||
otherDir := filepath.Join(campaignRoot, "icewind")
|
||||
mustWriteTestFile(t, filepath.Join(otherDir, "campaign.yml"), `campaign_id: icewind
|
||||
inputs:
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
players_file: ./players.yml
|
||||
party_file: ./party.yml
|
||||
`)
|
||||
mustWriteTestFile(t, filepath.Join(otherDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
||||
mustWriteTestFile(t, filepath.Join(otherDir, "autocorrect.yml"), "[]\n")
|
||||
mustWriteTestFile(t, filepath.Join(otherDir, "glossary.yml"), "[]\n")
|
||||
mustWriteTestFile(t, filepath.Join(otherDir, "players.yml"), "[]\n")
|
||||
mustWriteTestFile(t, filepath.Join(otherDir, "party.yml"), "[]\n")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "status", "2026-05-03", "--config", pipelinePath, "--campaign", "icewind", "--session", sessionPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "Campaign: icewind") {
|
||||
t.Fatalf("stdout = %q, want selected campaign", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRejectsCampaignIDAndCampaignFile(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "status", "2026-05-03", "--config", pipelinePath, "--campaign", "sample-campaign", "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "mutually exclusive") {
|
||||
t.Fatalf("stderr = %q, want mutually exclusive error", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteInvalidCommand(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
@@ -359,29 +460,42 @@ func TestExecuteMissingCommand(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func writeValidConfigFiles(t *testing.T, workspaceRoot string, transcribeURL ...string) (string, string) {
|
||||
func writeValidConfigFiles(t *testing.T, workspaceRoot string, transcribeURL ...string) (string, string, string) {
|
||||
t.Helper()
|
||||
|
||||
dir := t.TempDir()
|
||||
pipelinePath := filepath.Join(dir, "pipeline.yml")
|
||||
campaignRoot := filepath.Join(dir, "campaigns")
|
||||
campaignDir := filepath.Join(campaignRoot, "sample-campaign")
|
||||
campaignPath := filepath.Join(campaignDir, "campaign.yml")
|
||||
sessionPath := filepath.Join(dir, "session.yml")
|
||||
url := "https://example.com/transcribe"
|
||||
if len(transcribeURL) > 0 && strings.TrimSpace(transcribeURL[0]) != "" {
|
||||
url = transcribeURL[0]
|
||||
}
|
||||
seriatimBinary := writeSeriatimAppTestWrapper(t)
|
||||
scriptoriumBinary := writeScriptoriumAppTestWrapper(t)
|
||||
auditaBinary := writeAuditaAppTestWrapper(t)
|
||||
t.Setenv("GO_WANT_APP_SERIATIM_HELPER", "1")
|
||||
t.Setenv("GO_WANT_APP_SCRIPTORIUM_HELPER", "1")
|
||||
t.Setenv("GO_WANT_APP_AUDITA_HELPER", "1")
|
||||
t.Setenv("AUDITA_LLM_API_KEY", "test-audita-key")
|
||||
t.Setenv("PATH", filepath.Dir(scriptoriumBinary)+string(os.PathListSeparator)+os.Getenv("PATH"))
|
||||
|
||||
pipelineYAML := `workspace:
|
||||
root: ` + workspaceRoot + `
|
||||
campaigns:
|
||||
root: ` + campaignRoot + `
|
||||
default_campaign_id: sample-campaign
|
||||
cache:
|
||||
root: ` + filepath.Join(workspaceRoot, "cache") + `
|
||||
spool:
|
||||
root: ` + filepath.Join(workspaceRoot, "spool") + `
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: test-bucket
|
||||
archive:
|
||||
publish:
|
||||
enabled: true
|
||||
upload_run: false
|
||||
whisperx:
|
||||
@@ -398,36 +512,61 @@ seriatim:
|
||||
report: true
|
||||
audita:
|
||||
binary: ` + auditaBinary + `
|
||||
analyzer:
|
||||
timeout: 20m
|
||||
artifacts:
|
||||
output_dir: artifacts
|
||||
notification:
|
||||
timeout: 10s
|
||||
`
|
||||
|
||||
sessionYAML := `session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
`
|
||||
campaignYAML := `campaign_id: sample-campaign
|
||||
inputs:
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
players_file: ./players.yml
|
||||
party_file: ./party.yml
|
||||
`
|
||||
|
||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||
t.Fatalf("write pipeline config: %v", err)
|
||||
}
|
||||
if err := os.MkdirAll(campaignDir, 0o755); err != nil {
|
||||
t.Fatalf("create campaign dir: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(campaignPath, []byte(campaignYAML), 0o644); err != nil {
|
||||
t.Fatalf("write campaign config: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(sessionPath, []byte(sessionYAML), 0o644); err != nil {
|
||||
t.Fatalf("write session config: %v", err)
|
||||
}
|
||||
|
||||
mustWriteTestFile(t, filepath.Join(dir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
||||
mustWriteTestFile(t, filepath.Join(dir, "autocorrect.yml"), "[]\n")
|
||||
mustWriteTestFile(t, filepath.Join(dir, "glossary.yml"), "[]\n")
|
||||
mustWriteTestFile(t, filepath.Join(campaignDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
||||
mustWriteTestFile(t, filepath.Join(campaignDir, "autocorrect.yml"), "[]\n")
|
||||
mustWriteTestFile(t, filepath.Join(campaignDir, "glossary.yml"), "[]\n")
|
||||
mustWriteTestFile(t, filepath.Join(campaignDir, "players.yml"), "[]\n")
|
||||
mustWriteTestFile(t, filepath.Join(campaignDir, "party.yml"), "[]\n")
|
||||
mustWriteTestFile(t, filepath.Join(dir, "audio", "alice.flac"), "audio-bytes")
|
||||
|
||||
return pipelinePath, sessionPath
|
||||
return pipelinePath, campaignPath, sessionPath
|
||||
}
|
||||
|
||||
func writeAppTestCampaignConfig(t *testing.T, dir string) string {
|
||||
t.Helper()
|
||||
campaignPath := filepath.Join(dir, "campaign.yml")
|
||||
campaignYAML := `campaign_id: sample-campaign
|
||||
inputs:
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
players_file: ./players.yml
|
||||
party_file: ./party.yml
|
||||
`
|
||||
if err := os.WriteFile(campaignPath, []byte(campaignYAML), 0o644); err != nil {
|
||||
t.Fatalf("write campaign.yml: %v", err)
|
||||
}
|
||||
return campaignPath
|
||||
}
|
||||
|
||||
func writeManifestPathForExecute(t *testing.T) string {
|
||||
@@ -469,6 +608,60 @@ func writeSeriatimAppTestWrapper(t *testing.T) string {
|
||||
return path
|
||||
}
|
||||
|
||||
func writeScriptoriumAppTestWrapper(t *testing.T) string {
|
||||
t.Helper()
|
||||
exe, err := os.Executable()
|
||||
if err != nil {
|
||||
t.Fatalf("os.Executable() error = %v", err)
|
||||
}
|
||||
path := filepath.Join(t.TempDir(), "scriptorium")
|
||||
content := "#!/bin/sh\nexec \"" + exe + "\" -test.run=TestScriptoriumAppHelper -- \"$@\"\n"
|
||||
if err := os.WriteFile(path, []byte(content), 0o755); err != nil {
|
||||
t.Fatalf("WriteFile(%q): %v", path, err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func TestScriptoriumAppHelper(t *testing.T) {
|
||||
if os.Getenv("GO_WANT_APP_SCRIPTORIUM_HELPER") != "1" {
|
||||
return
|
||||
}
|
||||
|
||||
args := os.Args
|
||||
start := -1
|
||||
for i := range args {
|
||||
if args[i] == "--" {
|
||||
start = i + 1
|
||||
break
|
||||
}
|
||||
}
|
||||
if start < 0 || start >= len(args) {
|
||||
_, _ = os.Stderr.WriteString("missing -- args separator\n")
|
||||
os.Exit(2)
|
||||
}
|
||||
runArgs := args[start:]
|
||||
|
||||
outputPath := appSeriatimFlagValue(runArgs, "--out")
|
||||
if strings.TrimSpace(outputPath) == "" {
|
||||
outputPath = appSeriatimFlagValue(runArgs, "--output")
|
||||
}
|
||||
if strings.TrimSpace(outputPath) == "" {
|
||||
_, _ = os.Stderr.WriteString("missing output flag\n")
|
||||
os.Exit(2)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
|
||||
_, _ = os.Stderr.WriteString(fmt.Sprintf("mkdir output dir: %v\n", err))
|
||||
os.Exit(2)
|
||||
}
|
||||
if err := os.WriteFile(outputPath, []byte(`{"trim_action":"copy","warnings":[]}`), 0o644); err != nil {
|
||||
_, _ = os.Stderr.WriteString(fmt.Sprintf("write output: %v\n", err))
|
||||
os.Exit(2)
|
||||
}
|
||||
_, _ = os.Stdout.WriteString("scriptorium helper stdout\n")
|
||||
_, _ = os.Stderr.WriteString("scriptorium helper stderr\n")
|
||||
os.Exit(0)
|
||||
}
|
||||
|
||||
func TestSeriatimAppHelper(t *testing.T) {
|
||||
if os.Getenv("GO_WANT_APP_SERIATIM_HELPER") != "1" {
|
||||
return
|
||||
|
||||
141
internal/app/config_loader.go
Normal file
141
internal/app/config_loader.go
Normal file
@@ -0,0 +1,141 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
type pipelineCampaignConfig struct {
|
||||
PipelinePath string
|
||||
CampaignPath string
|
||||
Pipeline *config.PipelineConfig
|
||||
Campaign *config.CampaignConfig
|
||||
}
|
||||
|
||||
func loadCommandConfig(ctx context.Context, pipelineFlag, campaignFlag, campaignFileFlag, sessionFlag string, sessionOpts config.SessionLoadOptions) (*config.Config, error) {
|
||||
base, err := loadPipelineCampaignConfig(pipelineFlag, campaignFlag, campaignFileFlag)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if explicitSession := strings.TrimSpace(sessionFlag); explicitSession != "" {
|
||||
return config.LoadWithSessionOptions(base.PipelinePath, base.CampaignPath, explicitSession, sessionOpts)
|
||||
}
|
||||
|
||||
discoveredSession, err := discoverSessionConfigPathWithCandidates(config.DefaultSessionConfigSearchPaths)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if discoveredSession.Path != "" {
|
||||
return config.LoadWithSessionOptions(base.PipelinePath, base.CampaignPath, discoveredSession.Path, sessionOpts)
|
||||
}
|
||||
|
||||
sessionID := strings.TrimSpace(sessionOpts.SessionID)
|
||||
if sessionID == "" {
|
||||
return nil, missingSessionConfigError(discoveredSession.Searched, "remote session loading requires a session_id")
|
||||
}
|
||||
|
||||
sessionPrefix := artifacts.S3SessionPrefix(base.Pipeline.Storage.S3.RootPrefix, config.CampaignID(base.Campaign), sessionID)
|
||||
remoteKey := artifacts.S3SessionConfigKey(sessionPrefix)
|
||||
partialCfg := &config.Config{
|
||||
Pipeline: base.Pipeline,
|
||||
Campaign: base.Campaign,
|
||||
PipelinePath: base.PipelinePath,
|
||||
CampaignPath: base.CampaignPath,
|
||||
}
|
||||
store, err := newCommandObjectStore(ctx, partialCfg, nil)
|
||||
if err != nil {
|
||||
return nil, missingSessionConfigError(discoveredSession.Searched, fmt.Sprintf("remote session %q unavailable: %v", remoteKey, err))
|
||||
}
|
||||
|
||||
sessionInfo, err := findRemoteSessionConfig(ctx, store, sessionPrefix, remoteKey)
|
||||
if err != nil {
|
||||
return nil, missingSessionConfigError(discoveredSession.Searched, err.Error())
|
||||
}
|
||||
sessionTempPath, err := storage.DownloadObjectToTemp(ctx, store, remoteKey, "narratio-session-*.yml")
|
||||
if err != nil {
|
||||
return nil, missingSessionConfigError(discoveredSession.Searched, fmt.Sprintf("remote session %q download failed: %v", remoteKey, err))
|
||||
}
|
||||
sessionBytes, err := os.ReadFile(sessionTempPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read downloaded remote session %q: %w", sessionTempPath, err)
|
||||
}
|
||||
sessionCfg, err := config.LoadSessionBytesWithOptions("s3://"+s3BucketName(base.Pipeline)+"/"+remoteKey, sessionBytes, sessionOpts)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return config.Resolve(
|
||||
base.PipelinePath,
|
||||
base.Pipeline,
|
||||
base.CampaignPath,
|
||||
base.Campaign,
|
||||
sessionTempPath,
|
||||
sessionCfg,
|
||||
config.SessionSource{
|
||||
Source: "session_config.s3",
|
||||
LocalPath: sessionTempPath,
|
||||
S3Bucket: s3BucketName(base.Pipeline),
|
||||
S3Key: remoteKey,
|
||||
S3Size: sessionInfo.Size,
|
||||
S3ETag: sessionInfo.ETag,
|
||||
SpoolPath: sessionTempPath,
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
func loadPipelineCampaignConfig(pipelineFlag, campaignFlag, campaignFileFlag string) (*pipelineCampaignConfig, error) {
|
||||
resolvedPipelinePath, err := resolvePipelineConfigPath(pipelineFlag)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
pipelineCfg, err := config.LoadPipeline(resolvedPipelinePath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resolvedCampaignPath, err := resolveCampaignConfigPath(pipelineCfg, campaignFlag, campaignFileFlag)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
campaignCfg, err := config.LoadCampaign(resolvedCampaignPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if selectedID := strings.TrimSpace(campaignFlag); selectedID != "" && strings.TrimSpace(campaignFileFlag) == "" {
|
||||
if got := config.CampaignID(campaignCfg); got != selectedID {
|
||||
return nil, fmt.Errorf("campaign config %q invalid: campaign_id %q does not match selected campaign %q", resolvedCampaignPath, got, selectedID)
|
||||
}
|
||||
}
|
||||
return &pipelineCampaignConfig{
|
||||
PipelinePath: resolvedPipelinePath,
|
||||
CampaignPath: resolvedCampaignPath,
|
||||
Pipeline: pipelineCfg,
|
||||
Campaign: campaignCfg,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func findRemoteSessionConfig(ctx context.Context, store storage.ObjectStore, sessionPrefix, remoteKey string) (storage.ObjectInfo, error) {
|
||||
objects, err := store.List(ctx, sessionPrefix)
|
||||
if err != nil {
|
||||
return storage.ObjectInfo{}, fmt.Errorf("remote session %q list failed: %w", remoteKey, err)
|
||||
}
|
||||
for _, obj := range objects {
|
||||
if obj.Key == remoteKey {
|
||||
return obj, nil
|
||||
}
|
||||
}
|
||||
return storage.ObjectInfo{}, fmt.Errorf("remote session %q not found", remoteKey)
|
||||
}
|
||||
|
||||
func s3BucketName(cfg *config.PipelineConfig) string {
|
||||
if cfg == nil || cfg.Storage.S3 == nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(cfg.Storage.S3.Bucket)
|
||||
}
|
||||
21
internal/app/object_store.go
Normal file
21
internal/app/object_store.go
Normal file
@@ -0,0 +1,21 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func newCommandObjectStore(ctx context.Context, cfg *config.Config, logger *slog.Logger) (storage.ObjectStore, error) {
|
||||
if _, err := loadSecretsFromConfig(cfg, logger); err != nil {
|
||||
return nil, fmt.Errorf("load secrets from files: %w", err)
|
||||
}
|
||||
store, err := newObjectStoreFromConfigFn(ctx, cfg)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("initialize object store backend: %w", err)
|
||||
}
|
||||
return store, nil
|
||||
}
|
||||
165
internal/app/object_store_test.go
Normal file
165
internal/app/object_store_test.go
Normal file
@@ -0,0 +1,165 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func TestNewCommandObjectStoreLoadsSecretsBeforeFactory(t *testing.T) {
|
||||
accessKeyEnv := "NARRATIO_TEST_COMMAND_STORE_KEY_ID"
|
||||
secretKeyEnv := "NARRATIO_TEST_COMMAND_STORE_SECRET"
|
||||
restoreEnvAfterTest(t, accessKeyEnv, secretKeyEnv)
|
||||
|
||||
secretsDir := t.TempDir()
|
||||
mustWriteSecretFile(t, filepath.Join(secretsDir, accessKeyEnv), "loaded-key-id\n")
|
||||
mustWriteSecretFile(t, filepath.Join(secretsDir, secretKeyEnv), "loaded-secret\n")
|
||||
|
||||
cfg := commandObjectStoreTestConfig(secretsDir)
|
||||
fake := &storage.FakeBackend{}
|
||||
called := false
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
called = true
|
||||
if got := os.Getenv(accessKeyEnv); got != "loaded-key-id" {
|
||||
return nil, errors.New("access key was not loaded before object store init")
|
||||
}
|
||||
if got := os.Getenv(secretKeyEnv); got != "loaded-secret" {
|
||||
return nil, errors.New("secret key was not loaded before object store init")
|
||||
}
|
||||
return fake, nil
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
})
|
||||
|
||||
store, err := newCommandObjectStore(context.Background(), cfg, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("newCommandObjectStore() error = %v", err)
|
||||
}
|
||||
if store != fake {
|
||||
t.Fatalf("store = %#v, want fake backend", store)
|
||||
}
|
||||
if !called {
|
||||
t.Fatal("object store factory was not called")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewCommandObjectStorePreservesExistingEnv(t *testing.T) {
|
||||
accessKeyEnv := "NARRATIO_TEST_COMMAND_STORE_EXISTING_KEY_ID"
|
||||
secretKeyEnv := "NARRATIO_TEST_COMMAND_STORE_EXISTING_SECRET"
|
||||
t.Setenv(accessKeyEnv, "existing-key-id")
|
||||
t.Setenv(secretKeyEnv, "existing-secret")
|
||||
|
||||
secretsDir := t.TempDir()
|
||||
mustWriteSecretFile(t, filepath.Join(secretsDir, accessKeyEnv), "file-key-id\n")
|
||||
mustWriteSecretFile(t, filepath.Join(secretsDir, secretKeyEnv), "file-secret\n")
|
||||
|
||||
cfg := commandObjectStoreTestConfig(secretsDir)
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
if got := os.Getenv(accessKeyEnv); got != "existing-key-id" {
|
||||
return nil, errors.New("existing access key was overwritten")
|
||||
}
|
||||
if got := os.Getenv(secretKeyEnv); got != "existing-secret" {
|
||||
return nil, errors.New("existing secret key was overwritten")
|
||||
}
|
||||
return &storage.FakeBackend{}, nil
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
})
|
||||
|
||||
if _, err := newCommandObjectStore(context.Background(), cfg, nil); err != nil {
|
||||
t.Fatalf("newCommandObjectStore() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewCommandObjectStoreSecretErrorStopsFactory(t *testing.T) {
|
||||
cfg := commandObjectStoreTestConfig(filepath.Join(t.TempDir(), "missing"))
|
||||
called := false
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
called = true
|
||||
return &storage.FakeBackend{}, nil
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
})
|
||||
|
||||
_, err := newCommandObjectStore(context.Background(), cfg, nil)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if called {
|
||||
t.Fatal("object store factory was called after secret load failure")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "load secrets from files") {
|
||||
t.Fatalf("error = %q, want secret loading context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewCommandObjectStoreFactoryErrorIsContextual(t *testing.T) {
|
||||
cfg := commandObjectStoreTestConfig("")
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
return nil, errors.New("factory boom")
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
})
|
||||
|
||||
_, err := newCommandObjectStore(context.Background(), cfg, nil)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "initialize object store backend") || !strings.Contains(err.Error(), "factory boom") {
|
||||
t.Fatalf("error = %q, want factory context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func commandObjectStoreTestConfig(secretsDir string) *config.Config {
|
||||
cfg := &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Storage: config.StorageConfig{
|
||||
Backend: "s3",
|
||||
S3: &config.StorageS3Config{
|
||||
Bucket: "test-bucket",
|
||||
AccessKeyIDEnv: "NARRATIO_TEST_COMMAND_STORE_KEY_ID",
|
||||
SecretKeyEnv: "NARRATIO_TEST_COMMAND_STORE_SECRET",
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
if strings.TrimSpace(secretsDir) != "" {
|
||||
cfg.Pipeline.Secrets = &config.SecretsConfig{EnvDir: secretsDir}
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
|
||||
func restoreEnvAfterTest(t *testing.T, names ...string) {
|
||||
t.Helper()
|
||||
originals := make(map[string]string, len(names))
|
||||
present := make(map[string]bool, len(names))
|
||||
for _, name := range names {
|
||||
value, ok := os.LookupEnv(name)
|
||||
originals[name] = value
|
||||
present[name] = ok
|
||||
_ = os.Unsetenv(name)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
for _, name := range names {
|
||||
if present[name] {
|
||||
_ = os.Setenv(name, originals[name])
|
||||
} else {
|
||||
_ = os.Unsetenv(name)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
131
internal/app/operator_artifact_rendering.go
Normal file
131
internal/app/operator_artifact_rendering.go
Normal file
@@ -0,0 +1,131 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func buildHelperArtifactCatalog(cfg *config.Config) (*artifacts.ArtifactCatalog, error) {
|
||||
catalog := artifacts.NewArtifactCatalog()
|
||||
if err := catalog.RegisterBuiltIns(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
configured := map[string]artifacts.ConfiguredArtifactDefinition{}
|
||||
if cfg.Pipeline.Scriptorium != nil {
|
||||
for key, item := range cfg.Pipeline.Scriptorium.Artifacts {
|
||||
configured[key] = artifacts.ConfiguredArtifactDefinition{Enabled: item.Enabled, OutputPath: item.OutputPath}
|
||||
}
|
||||
}
|
||||
if err := catalog.RegisterConfiguredArtifacts(configured, nil); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return catalog, nil
|
||||
}
|
||||
|
||||
func writeArtifactList(out io.Writer, cfg *config.Config, catalog *artifacts.ArtifactCatalog, locks *effectiveLocks, publishedRemoteState map[string]string) {
|
||||
lockSet := lockSourceSet(locks.All)
|
||||
fmt.Fprintln(out, "Built-in:")
|
||||
for _, transcript := range artifacts.RuntimeTranscriptArtifacts() {
|
||||
writeArtifactLine(out, transcript.SourceID, lockSet)
|
||||
}
|
||||
writeArtifactLine(out, artifacts.ArtifactBoundsSession, lockSet)
|
||||
fmt.Fprintln(out, "Configured:")
|
||||
for _, entry := range catalog.ListConfigured() {
|
||||
writeArtifactLine(out, entry.SourceID, lockSet)
|
||||
}
|
||||
fmt.Fprintln(out, "Previous-session:")
|
||||
for _, req := range artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg)) {
|
||||
fmt.Fprintf(out, "- %s required=%t\n", artifactpolicy.PreviousSessionSourceID(req.Name), req.Required)
|
||||
}
|
||||
fmt.Fprintln(out, "Published:")
|
||||
for _, rule := range cfg.Pipeline.Publish.Outputs {
|
||||
writePublishedOutputLine(out, rule, catalog, lockSet, publishedRemoteState)
|
||||
}
|
||||
}
|
||||
|
||||
func writeArtifactLine(out io.Writer, source string, lockSet map[string]config.PublishLockRule) {
|
||||
parts := []string{source}
|
||||
if _, ok := lockSet[source]; ok {
|
||||
parts = append(parts, "locked")
|
||||
}
|
||||
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||
}
|
||||
|
||||
func writePublishedOutputLine(out io.Writer, rule config.PublishOutputRule, catalog *artifacts.ArtifactCatalog, lockSet map[string]config.PublishLockRule, remoteState map[string]string) {
|
||||
source := strings.TrimSpace(rule.Source)
|
||||
parts := []string{source}
|
||||
if _, ok := lockSet[source]; ok {
|
||||
parts = append(parts, "locked")
|
||||
}
|
||||
dest, showDest, err := helperPublishedOutputDest(rule, catalog)
|
||||
if err != nil {
|
||||
parts = append(parts, "remote=error")
|
||||
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||
return
|
||||
}
|
||||
if showDest {
|
||||
parts = append(parts, "dest="+dest)
|
||||
}
|
||||
if state := remoteState[publishedOutputRemoteStateKey(source, dest)]; state != "" {
|
||||
parts = append(parts, state)
|
||||
}
|
||||
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||
}
|
||||
|
||||
func remotePublishedOutputAvailability(ctx context.Context, cfg *config.Config, store storage.ObjectStore, catalog *artifacts.ArtifactCatalog) map[string]string {
|
||||
out := map[string]string{}
|
||||
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
for _, rule := range cfg.Pipeline.Publish.Outputs {
|
||||
source := strings.TrimSpace(rule.Source)
|
||||
dest, _, err := helperPublishedOutputDest(rule, catalog)
|
||||
if err != nil {
|
||||
out[publishedOutputRemoteStateKey(source, "")] = "remote=error"
|
||||
continue
|
||||
}
|
||||
key := artifacts.S3PublishedOutputKey(sessionPrefix, dest)
|
||||
if exists, err := store.Exists(ctx, key); err == nil && exists {
|
||||
out[publishedOutputRemoteStateKey(source, dest)] = "remote=published"
|
||||
} else if err != nil {
|
||||
out[publishedOutputRemoteStateKey(source, dest)] = "remote=error"
|
||||
} else {
|
||||
out[publishedOutputRemoteStateKey(source, dest)] = "remote=missing"
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func helperPublishedOutputDest(rule config.PublishOutputRule, catalog *artifacts.ArtifactCatalog) (string, bool, error) {
|
||||
source := strings.TrimSpace(rule.Source)
|
||||
normalized, err := artifactpolicy.ResolvePublishedDestination(source, rule.Dest, helperConfiguredOutputPathMap(catalog))
|
||||
if err != nil {
|
||||
return "", false, err
|
||||
}
|
||||
entry, ok := catalog.Lookup(source)
|
||||
showDest := !ok || strings.TrimSpace(entry.CanonicalRelPath) != normalized
|
||||
return normalized, showDest, nil
|
||||
}
|
||||
|
||||
func helperConfiguredOutputPathMap(catalog *artifacts.ArtifactCatalog) map[string]string {
|
||||
out := map[string]string{}
|
||||
if catalog == nil {
|
||||
return out
|
||||
}
|
||||
for _, entry := range catalog.ListConfigured() {
|
||||
if strings.TrimSpace(entry.ConfiguredKey) == "" {
|
||||
continue
|
||||
}
|
||||
out[entry.ConfiguredKey] = strings.TrimSpace(entry.CanonicalRelPath)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func publishedOutputRemoteStateKey(source, dest string) string {
|
||||
return strings.TrimSpace(source) + "\x00" + strings.TrimSpace(dest)
|
||||
}
|
||||
39
internal/app/operator_artifacts_list.go
Normal file
39
internal/app/operator_artifacts_list.go
Normal file
@@ -0,0 +1,39 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// ArtifactsList lists effective artifact sources.
|
||||
func ArtifactsList(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("artifacts list", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var flags commonConfigFlags
|
||||
var remote bool
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
fs.BoolVar(&remote, "remote", false, "inspect remote publish availability")
|
||||
if err := parseSessionAwareFlags("artifacts list", fs, args, &flags.sessionID); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(flags.sessionID) == "" {
|
||||
return fmt.Errorf("artifacts list: session_id is required")
|
||||
}
|
||||
cfg, store, locks, _, err := loadHelperContext(ctx, flags, remote)
|
||||
if err != nil {
|
||||
return fmt.Errorf("artifacts list: %w", err)
|
||||
}
|
||||
catalog, err := buildHelperArtifactCatalog(cfg)
|
||||
if err != nil {
|
||||
return fmt.Errorf("artifacts list: %w", err)
|
||||
}
|
||||
publishedRemoteState := map[string]string{}
|
||||
if remote && store != nil {
|
||||
publishedRemoteState = remotePublishedOutputAvailability(ctx, cfg, store, catalog)
|
||||
}
|
||||
writeArtifactList(out, cfg, catalog, locks, publishedRemoteState)
|
||||
return nil
|
||||
}
|
||||
140
internal/app/operator_findings.go
Normal file
140
internal/app/operator_findings.go
Normal file
@@ -0,0 +1,140 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
type finding struct {
|
||||
Severity string
|
||||
Category string
|
||||
Message string
|
||||
}
|
||||
|
||||
type findingError struct {
|
||||
count int
|
||||
}
|
||||
|
||||
func (e findingError) Error() string {
|
||||
return fmt.Sprintf("%d validation error(s)", e.count)
|
||||
}
|
||||
|
||||
func renderFindings(out io.Writer, campaign, sessionID string, findings []finding) error {
|
||||
if campaign != "" || sessionID != "" {
|
||||
fmt.Fprintf(out, "Campaign: %s\n", campaign)
|
||||
fmt.Fprintf(out, "Session: %s\n\n", sessionID)
|
||||
}
|
||||
errorsCount := 0
|
||||
for _, f := range findings {
|
||||
if f.Severity == "ERROR" {
|
||||
errorsCount++
|
||||
}
|
||||
fmt.Fprintf(out, "%-5s %-10s %s\n", f.Severity, f.Category, f.Message)
|
||||
}
|
||||
if errorsCount > 0 {
|
||||
return findingError{count: errorsCount}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func okFinding(category, msg string) finding { return finding{"OK", category, msg} }
|
||||
func infoFinding(category, msg string) finding { return finding{"INFO", category, msg} }
|
||||
func warnFinding(category, msg string) finding { return finding{"WARN", category, msg} }
|
||||
func errorFinding(category, msg string) finding { return finding{"ERROR", category, msg} }
|
||||
|
||||
func sessionSourceSummary(cfg *config.Config) string {
|
||||
source := cfg.SessionSource.Source
|
||||
if source == "" {
|
||||
source = "session_config"
|
||||
}
|
||||
if cfg.SessionSource.S3Key != "" {
|
||||
return source + " " + cfg.SessionSource.S3Key
|
||||
}
|
||||
return source + " " + cfg.SessionPath
|
||||
}
|
||||
|
||||
func validateStableInputFindings(cfg *config.Config) []finding {
|
||||
checks := inspectStableInputs(cfg)
|
||||
out := make([]finding, 0, len(checks))
|
||||
for _, check := range checks {
|
||||
if check.Err != nil {
|
||||
msg := check.Name + ": " + check.Err.Error()
|
||||
if strings.TrimSpace(check.Path) != "" {
|
||||
msg = fmt.Sprintf("%s missing: %v", check.Name, check.Err)
|
||||
}
|
||||
out = append(out, errorFinding("inputs", msg))
|
||||
continue
|
||||
}
|
||||
out = append(out, okFinding("inputs", check.Name+": "+check.Path))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func resolveHelperConfigRelativePath(input config.ResolvedInputFile) (string, error) {
|
||||
if strings.TrimSpace(input.ConfigPath) == "" {
|
||||
return "", fmt.Errorf("source config path is required")
|
||||
}
|
||||
path := strings.TrimSpace(input.Path)
|
||||
if path == "" {
|
||||
return "", fmt.Errorf("path is required")
|
||||
}
|
||||
if filepath.IsAbs(path) {
|
||||
return filepath.Clean(path), nil
|
||||
}
|
||||
return filepath.Clean(filepath.Join(filepath.Dir(input.ConfigPath), path)), nil
|
||||
}
|
||||
|
||||
func validateLocalAudioFindings(cfg *config.Config) []finding {
|
||||
check := inspectLocalAudioPresence(cfg)
|
||||
if !check.Checked {
|
||||
return nil
|
||||
}
|
||||
if check.Err != nil {
|
||||
return []finding{errorFinding("audio", check.Err.Error())}
|
||||
}
|
||||
return []finding{okFinding("audio", fmt.Sprintf("%d local audio file(s)", len(check.Paths)))}
|
||||
}
|
||||
|
||||
func validateRemoteAudioFinding(ctx context.Context, cfg *config.Config, store storage.ObjectStore) finding {
|
||||
check := inspectRemoteAudioPresence(ctx, cfg, store)
|
||||
if check.Err != nil {
|
||||
return errorFinding("audio", check.Err.Error())
|
||||
}
|
||||
return okFinding("audio", fmt.Sprintf("%d remote .flac object(s)", len(check.Keys)))
|
||||
}
|
||||
|
||||
func loadLocalManifest(ctx context.Context, path string) (*manifest.Manifest, error) {
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
store := &manifest.LocalStore{}
|
||||
return store.Load(ctx, path)
|
||||
}
|
||||
|
||||
func writeStageStatuses(out io.Writer, m *manifest.Manifest) {
|
||||
if m == nil || len(m.Stages) == 0 {
|
||||
fmt.Fprintln(out, "stages: no stages recorded")
|
||||
return
|
||||
}
|
||||
fmt.Fprintln(out, "stages:")
|
||||
names := make([]string, 0, len(m.Stages))
|
||||
for name := range m.Stages {
|
||||
names = append(names, name)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, name := range names {
|
||||
fmt.Fprintf(out, "- %s: %s\n", name, m.Stages[name].Status)
|
||||
}
|
||||
}
|
||||
131
internal/app/operator_helpers.go
Normal file
131
internal/app/operator_helpers.go
Normal file
@@ -0,0 +1,131 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
type commonConfigFlags struct {
|
||||
pipelinePath string
|
||||
campaignPath string
|
||||
campaignFilePath string
|
||||
sessionPath string
|
||||
sessionID string
|
||||
previousSessionID string
|
||||
}
|
||||
|
||||
func addCommonConfigFlags(fs *flag.FlagSet, flags *commonConfigFlags) {
|
||||
fs.StringVar(&flags.pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||
fs.StringVar(&flags.campaignPath, "campaign", "", "campaign ID")
|
||||
fs.StringVar(&flags.campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
||||
fs.StringVar(&flags.sessionPath, "session", "", "path to session.yml")
|
||||
fs.StringVar(&flags.sessionID, "session-id", "", "session identifier")
|
||||
fs.StringVar(&flags.previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
||||
}
|
||||
|
||||
func (f commonConfigFlags) sessionOptions() config.SessionLoadOptions {
|
||||
return config.SessionLoadOptions{
|
||||
SessionID: f.sessionID,
|
||||
PreviousSessionID: f.previousSessionID,
|
||||
}
|
||||
}
|
||||
|
||||
// Session dispatches session helper subcommands.
|
||||
func Session(ctx context.Context, args []string, out io.Writer) error {
|
||||
if len(args) == 0 {
|
||||
return fmt.Errorf("session: expected subcommand: init|validate|status|plan|restore|artifacts|locks")
|
||||
}
|
||||
switch args[0] {
|
||||
case "init":
|
||||
return SessionInit(ctx, args[1:], out)
|
||||
case "validate":
|
||||
return SessionValidate(ctx, args[1:], out)
|
||||
case "status":
|
||||
return Status(ctx, args[1:], out)
|
||||
case "plan":
|
||||
return Plan(ctx, args[1:], out)
|
||||
case "restore":
|
||||
return Restore(ctx, args[1:], out)
|
||||
case "artifacts":
|
||||
return ArtifactsList(ctx, args[1:], out)
|
||||
case "locks":
|
||||
return SessionLocks(ctx, args[1:], out)
|
||||
default:
|
||||
return fmt.Errorf("session: unknown subcommand %q", args[0])
|
||||
}
|
||||
}
|
||||
|
||||
// SessionLocks dispatches session-oriented publish lock list and mutation
|
||||
// helpers while preserving the existing lock implementations.
|
||||
func SessionLocks(ctx context.Context, args []string, out io.Writer) error {
|
||||
if len(args) > 0 && !isCLIFlagToken(args[0]) {
|
||||
switch args[0] {
|
||||
case "add":
|
||||
return LocksAdd(ctx, args[1:], out)
|
||||
case "remove":
|
||||
return LocksRemove(ctx, args[1:], out)
|
||||
}
|
||||
}
|
||||
return LocksList(ctx, args, out)
|
||||
}
|
||||
|
||||
// Artifacts dispatches artifact helper subcommands.
|
||||
func Artifacts(ctx context.Context, args []string, out io.Writer) error {
|
||||
if len(args) == 0 {
|
||||
return fmt.Errorf("artifacts: expected subcommand: list")
|
||||
}
|
||||
switch args[0] {
|
||||
case "list":
|
||||
return ArtifactsList(ctx, args[1:], out)
|
||||
default:
|
||||
return fmt.Errorf("artifacts: unknown subcommand %q", args[0])
|
||||
}
|
||||
}
|
||||
|
||||
func loadHelperContext(ctx context.Context, flags commonConfigFlags, needStore bool) (*config.Config, storage.ObjectStore, *effectiveLocks, *manifest.Manifest, error) {
|
||||
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||
if err != nil {
|
||||
return nil, nil, nil, nil, err
|
||||
}
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
return nil, nil, nil, nil, err
|
||||
}
|
||||
var store storage.ObjectStore
|
||||
if needStore {
|
||||
store, err = newCommandObjectStore(ctx, cfg, nil)
|
||||
if err != nil {
|
||||
return nil, nil, nil, nil, err
|
||||
}
|
||||
} else {
|
||||
store, _ = objectStoreIfConfigured(ctx, cfg)
|
||||
}
|
||||
locks, err := loadEffectiveLocks(ctx, cfg, store)
|
||||
if err != nil {
|
||||
return nil, nil, nil, nil, err
|
||||
}
|
||||
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
m, err := loadLocalManifest(ctx, paths.ManifestPath)
|
||||
if err != nil {
|
||||
return nil, nil, nil, nil, err
|
||||
}
|
||||
return cfg, store, locks, m, nil
|
||||
}
|
||||
|
||||
func objectStoreIfConfigured(ctx context.Context, cfg *config.Config) (storage.ObjectStore, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Storage.S3 == nil || strings.TrimSpace(cfg.Pipeline.Storage.S3.Bucket) == "" {
|
||||
return nil, nil
|
||||
}
|
||||
store, err := newCommandObjectStore(ctx, cfg, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return store, nil
|
||||
}
|
||||
1114
internal/app/operator_helpers_test.go
Normal file
1114
internal/app/operator_helpers_test.go
Normal file
File diff suppressed because it is too large
Load Diff
276
internal/app/operator_inspection.go
Normal file
276
internal/app/operator_inspection.go
Normal file
@@ -0,0 +1,276 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
type stableInputCheck struct {
|
||||
Name string
|
||||
Path string
|
||||
Err error
|
||||
}
|
||||
|
||||
type localAudioCheck struct {
|
||||
Checked bool
|
||||
Paths []string
|
||||
Err error
|
||||
}
|
||||
|
||||
type remoteAudioCheck struct {
|
||||
Checked bool
|
||||
Prefix string
|
||||
Keys []string
|
||||
Err error
|
||||
}
|
||||
|
||||
type previousArtifactReadiness struct {
|
||||
Requirements []artifacts.PreviousArtifactRequirement
|
||||
MissingID bool
|
||||
Err error
|
||||
}
|
||||
|
||||
type remoteCurrentStateCheck struct {
|
||||
State *RemoteCurrentState
|
||||
Err error
|
||||
}
|
||||
|
||||
type effectiveLocksCheck struct {
|
||||
Locks *effectiveLocks
|
||||
Err error
|
||||
}
|
||||
|
||||
func inspectStableInputs(cfg *config.Config) []stableInputCheck {
|
||||
items := []struct {
|
||||
name string
|
||||
in config.ResolvedInputFile
|
||||
}{
|
||||
{name: "speakers", in: cfg.StableInputs.SpeakersFile},
|
||||
{name: "autocorrect", in: cfg.StableInputs.AutocorrectFile},
|
||||
{name: "glossary", in: cfg.StableInputs.GlossaryFile},
|
||||
{name: "players", in: cfg.StableInputs.PlayersFile},
|
||||
{name: "party", in: cfg.StableInputs.PartyFile},
|
||||
}
|
||||
out := make([]stableInputCheck, 0, len(items))
|
||||
for _, item := range items {
|
||||
path, err := resolveHelperConfigRelativePath(item.in)
|
||||
if err != nil {
|
||||
out = append(out, stableInputCheck{Name: item.name, Err: err})
|
||||
continue
|
||||
}
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
out = append(out, stableInputCheck{Name: item.name, Path: path, Err: err})
|
||||
continue
|
||||
}
|
||||
out = append(out, stableInputCheck{Name: item.name, Path: path})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func inspectLocalAudioPresence(cfg *config.Config) localAudioCheck {
|
||||
if cfg.Session.Inputs.AudioS3 != nil {
|
||||
return localAudioCheck{}
|
||||
}
|
||||
|
||||
sessionDir := filepath.Dir(cfg.SessionPath)
|
||||
resolved, err := resolveLocalInspectionAudioPaths(sessionDir, cfg.Session.Inputs)
|
||||
if err != nil {
|
||||
return localAudioCheck{Checked: true, Err: err}
|
||||
}
|
||||
return localAudioCheck{
|
||||
Checked: true,
|
||||
Paths: resolved,
|
||||
}
|
||||
}
|
||||
|
||||
func inspectRemoteAudioPresence(ctx context.Context, cfg *config.Config, store storage.ObjectStore) remoteAudioCheck {
|
||||
if cfg.Session.Inputs.AudioS3 == nil {
|
||||
return remoteAudioCheck{}
|
||||
}
|
||||
if store == nil {
|
||||
return remoteAudioCheck{Checked: true, Err: fmt.Errorf("storage backend is required for remote audio checks")}
|
||||
}
|
||||
|
||||
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
audioPrefix := artifacts.S3AudioPrefix(sessionPrefix, cfg.Session.Inputs.AudioS3.Prefix)
|
||||
objects, err := store.List(ctx, audioPrefix)
|
||||
if err != nil {
|
||||
return remoteAudioCheck{Checked: true, Prefix: audioPrefix, Err: err}
|
||||
}
|
||||
|
||||
keys := make([]string, 0, len(objects))
|
||||
seenBase := map[string]string{}
|
||||
for _, obj := range objects {
|
||||
key := strings.TrimSpace(obj.Key)
|
||||
if key == "" || strings.HasSuffix(key, "/") || !isInspectionFlacPath(key) {
|
||||
continue
|
||||
}
|
||||
base := path.Base(key)
|
||||
if prev, exists := seenBase[base]; exists && prev != key {
|
||||
return remoteAudioCheck{
|
||||
Checked: true,
|
||||
Prefix: audioPrefix,
|
||||
Err: fmt.Errorf("duplicate s3 audio basename %q from %q and %q", base, prev, key),
|
||||
}
|
||||
}
|
||||
seenBase[base] = key
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
if len(keys) == 0 {
|
||||
return remoteAudioCheck{
|
||||
Checked: true,
|
||||
Prefix: audioPrefix,
|
||||
Err: fmt.Errorf("no .flac files found under s3 audio prefix %q", audioPrefix),
|
||||
}
|
||||
}
|
||||
return remoteAudioCheck{
|
||||
Checked: true,
|
||||
Prefix: audioPrefix,
|
||||
Keys: keys,
|
||||
}
|
||||
}
|
||||
|
||||
func inspectPreviousArtifactReadiness(
|
||||
ctx context.Context,
|
||||
cfg *config.Config,
|
||||
store storage.ObjectStore,
|
||||
requirements []artifacts.PreviousArtifactRequirement,
|
||||
) previousArtifactReadiness {
|
||||
out := previousArtifactReadiness{
|
||||
Requirements: append([]artifacts.PreviousArtifactRequirement(nil), requirements...),
|
||||
}
|
||||
if len(requirements) == 0 {
|
||||
return out
|
||||
}
|
||||
if strings.TrimSpace(cfg.Session.PreviousSessionID) == "" {
|
||||
out.MissingID = true
|
||||
return out
|
||||
}
|
||||
if store == nil {
|
||||
out.Err = fmt.Errorf("previous-session artifacts cannot be checked because storage is unavailable")
|
||||
return out
|
||||
}
|
||||
|
||||
prefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
||||
if _, err := artifacts.LoadCurrentState(ctx, store, prefix, artifacts.CurrentStateValidation{
|
||||
ExpectedSessionID: strings.TrimSpace(cfg.Session.PreviousSessionID),
|
||||
ExpectedCampaign: strings.TrimSpace(cfg.Session.Campaign),
|
||||
ValidateRunID: true,
|
||||
}); err != nil {
|
||||
out.Err = fmt.Errorf("remote %v", err)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func inspectRemoteCurrentState(ctx context.Context, cfg *config.Config, store storage.ObjectStore) remoteCurrentStateCheck {
|
||||
if store == nil {
|
||||
return remoteCurrentStateCheck{}
|
||||
}
|
||||
current, err := discoverRemoteCurrentStateFn(ctx, cfg, store)
|
||||
if err != nil {
|
||||
return remoteCurrentStateCheck{Err: err}
|
||||
}
|
||||
return remoteCurrentStateCheck{State: current}
|
||||
}
|
||||
|
||||
func inspectEffectiveLocks(ctx context.Context, cfg *config.Config, store storage.ObjectStore) effectiveLocksCheck {
|
||||
locks, err := loadEffectiveLocks(ctx, cfg, store)
|
||||
if err != nil {
|
||||
return effectiveLocksCheck{Err: err}
|
||||
}
|
||||
return effectiveLocksCheck{Locks: locks}
|
||||
}
|
||||
|
||||
func resolveLocalInspectionAudioPaths(sessionDir string, inputs config.SessionInputsConfig) ([]string, error) {
|
||||
if len(inputs.AudioFiles) > 0 {
|
||||
out := make([]string, 0, len(inputs.AudioFiles))
|
||||
seenBase := map[string]string{}
|
||||
for _, item := range inputs.AudioFiles {
|
||||
resolved, err := resolveInspectionPath(sessionDir, item)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !isInspectionFlacPath(resolved) {
|
||||
return nil, fmt.Errorf("audio file %q must have .flac extension", resolved)
|
||||
}
|
||||
if err := requireInspectionFile(resolved, "audio file"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
base := filepath.Base(resolved)
|
||||
if prev, exists := seenBase[base]; exists && prev != resolved {
|
||||
return nil, fmt.Errorf("duplicate audio basename %q from %q and %q", base, prev, resolved)
|
||||
}
|
||||
seenBase[base] = resolved
|
||||
out = append(out, resolved)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
audioDir, err := resolveInspectionPath(sessionDir, inputs.AudioDir)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
entries, err := os.ReadDir(audioDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read audio directory %q: %w", audioDir, err)
|
||||
}
|
||||
out := make([]string, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() {
|
||||
continue
|
||||
}
|
||||
full := filepath.Join(audioDir, entry.Name())
|
||||
if !isInspectionFlacPath(full) {
|
||||
continue
|
||||
}
|
||||
if err := requireInspectionFile(full, "audio file"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, full)
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil, fmt.Errorf("no .flac files found in audio directory %q", audioDir)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func resolveInspectionPath(baseDir, inputPath string) (string, error) {
|
||||
pathValue := strings.TrimSpace(inputPath)
|
||||
if pathValue == "" {
|
||||
return "", fmt.Errorf("path is required")
|
||||
}
|
||||
if filepath.IsAbs(pathValue) {
|
||||
return filepath.Clean(pathValue), nil
|
||||
}
|
||||
return filepath.Clean(filepath.Join(baseDir, pathValue)), nil
|
||||
}
|
||||
|
||||
func requireInspectionFile(path, label string) error {
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return fmt.Errorf("%s %q does not exist", label, path)
|
||||
}
|
||||
return fmt.Errorf("stat %s %q: %w", label, path, err)
|
||||
}
|
||||
if info.IsDir() {
|
||||
return fmt.Errorf("%s %q is a directory", label, path)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func isInspectionFlacPath(path string) bool {
|
||||
return strings.EqualFold(filepath.Ext(strings.TrimSpace(path)), ".flac")
|
||||
}
|
||||
172
internal/app/operator_locks.go
Normal file
172
internal/app/operator_locks.go
Normal file
@@ -0,0 +1,172 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
// Locks dispatches publish lock list and mutation helpers.
|
||||
func Locks(ctx context.Context, args []string, out io.Writer) error {
|
||||
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
|
||||
switch args[0] {
|
||||
case "add":
|
||||
return LocksAdd(ctx, args[1:], out)
|
||||
case "remove":
|
||||
return LocksRemove(ctx, args[1:], out)
|
||||
default:
|
||||
return fmt.Errorf("locks: unknown subcommand %q", args[0])
|
||||
}
|
||||
}
|
||||
return LocksList(ctx, args, out)
|
||||
}
|
||||
|
||||
// LocksList lists effective publish locks.
|
||||
func LocksList(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("locks", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var flags commonConfigFlags
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
if err := parseSessionAwareFlags("locks", fs, args, &flags.sessionID); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(flags.sessionID) == "" {
|
||||
return fmt.Errorf("locks: session_id is required")
|
||||
}
|
||||
cfg, _, locks, _, err := loadHelperContext(ctx, flags, true)
|
||||
if err != nil {
|
||||
return fmt.Errorf("locks: %w", err)
|
||||
}
|
||||
writeLocks(out, cfg, locks)
|
||||
return nil
|
||||
}
|
||||
|
||||
// LocksAdd adds or updates one remote lock.
|
||||
func LocksAdd(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("locks add", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var flags commonConfigFlags
|
||||
var reason string
|
||||
var force bool
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
fs.StringVar(&reason, "reason", "", "lock reason")
|
||||
fs.BoolVar(&force, "force", false, "update existing remote lock")
|
||||
source, err := parseSessionIDAndOnePositionalArg("locks add", "source id", fs, args, &flags.sessionID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(flags.sessionID) == "" {
|
||||
return fmt.Errorf("locks add: session_id is required")
|
||||
}
|
||||
cfg, store, locks, _, err := loadHelperContext(ctx, flags, true)
|
||||
if err != nil {
|
||||
return fmt.Errorf("locks add: %w", err)
|
||||
}
|
||||
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, "locks add"); err != nil {
|
||||
return fmt.Errorf("locks add: %w", err)
|
||||
}
|
||||
if _, ok := lockSourceSet(locks.Static)[source]; ok {
|
||||
return fmt.Errorf("locks add: source %q is locked by pipeline config and cannot be modified remotely", source)
|
||||
}
|
||||
remoteSet := lockSourceSet(locks.Remote)
|
||||
if _, exists := remoteSet[source]; exists && !force {
|
||||
return fmt.Errorf("locks add: remote lock for %q already exists; pass --force to update", source)
|
||||
}
|
||||
remoteSet[source] = config.PublishLockRule{Source: source, Reason: strings.TrimSpace(reason)}
|
||||
remoteLocks := lockMapValues(remoteSet)
|
||||
if _, err := config.ValidatePublishLockRules(remoteLocks, cfg.Pipeline.Scriptorium, "locks"); err != nil {
|
||||
return fmt.Errorf("locks add: %w", err)
|
||||
}
|
||||
if err := uploadRemoteLockStore(ctx, store, locks.Key, &config.PublishLockStore{Locks: remoteLocks}); err != nil {
|
||||
return fmt.Errorf("locks add: %w", err)
|
||||
}
|
||||
_, err = fmt.Fprintf(out, "narratio session locks add: locked %s\n", source)
|
||||
return err
|
||||
}
|
||||
|
||||
// LocksRemove removes one remote lock.
|
||||
func LocksRemove(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("locks remove", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var flags commonConfigFlags
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
source, err := parseSessionIDAndOnePositionalArg("locks remove", "source id", fs, args, &flags.sessionID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(flags.sessionID) == "" {
|
||||
return fmt.Errorf("locks remove: session_id is required")
|
||||
}
|
||||
cfg, store, locks, _, err := loadHelperContext(ctx, flags, true)
|
||||
if err != nil {
|
||||
return fmt.Errorf("locks remove: %w", err)
|
||||
}
|
||||
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, "locks remove"); err != nil {
|
||||
return fmt.Errorf("locks remove: %w", err)
|
||||
}
|
||||
remoteSet := lockSourceSet(locks.Remote)
|
||||
if _, ok := remoteSet[source]; !ok {
|
||||
if _, static := lockSourceSet(locks.Static)[source]; static {
|
||||
return fmt.Errorf("locks remove: source %q is locked by pipeline config and cannot be unlocked remotely", source)
|
||||
}
|
||||
return fmt.Errorf("locks remove: remote lock for %q does not exist", source)
|
||||
}
|
||||
delete(remoteSet, source)
|
||||
remoteLocks := lockMapValues(remoteSet)
|
||||
if err := uploadRemoteLockStore(ctx, store, locks.Key, &config.PublishLockStore{Locks: remoteLocks}); err != nil {
|
||||
return fmt.Errorf("locks remove: %w", err)
|
||||
}
|
||||
_, err = fmt.Fprintf(out, "narratio session locks remove: unlocked %s\n", source)
|
||||
return err
|
||||
}
|
||||
|
||||
func writeLocks(out io.Writer, cfg *config.Config, locks *effectiveLocks) {
|
||||
if locks == nil || len(locks.All) == 0 {
|
||||
fmt.Fprintln(out, "Publish locks: none")
|
||||
return
|
||||
}
|
||||
fmt.Fprintln(out, "Publish locks:")
|
||||
published := map[string]config.PublishOutputRule{}
|
||||
if cfg != nil && cfg.Pipeline != nil && cfg.Pipeline.Publish != nil {
|
||||
for _, rule := range cfg.Pipeline.Publish.Outputs {
|
||||
published[strings.TrimSpace(rule.Source)] = rule
|
||||
}
|
||||
}
|
||||
staticSet := lockSourceSet(locks.Static)
|
||||
for _, lock := range locks.All {
|
||||
origin := "remote"
|
||||
if _, ok := staticSet[lock.Source]; ok {
|
||||
origin = "pipeline"
|
||||
}
|
||||
promo := "not-published"
|
||||
if _, ok := published[lock.Source]; ok {
|
||||
promo = "published"
|
||||
}
|
||||
reason := strings.TrimSpace(lock.Reason)
|
||||
if reason == "" {
|
||||
reason = "(no reason)"
|
||||
}
|
||||
fmt.Fprintf(out, "- %s origin=%s %s reason=%s\n", lock.Source, origin, promo, reason)
|
||||
}
|
||||
}
|
||||
|
||||
func lockMapValues(in map[string]config.PublishLockRule) []config.PublishLockRule {
|
||||
keys := make([]string, 0, len(in))
|
||||
for key := range in {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
out := make([]config.PublishLockRule, 0, len(keys))
|
||||
for _, key := range keys {
|
||||
item := in[key]
|
||||
item.Source = key
|
||||
item.Reason = strings.TrimSpace(item.Reason)
|
||||
out = append(out, item)
|
||||
}
|
||||
return out
|
||||
}
|
||||
268
internal/app/operator_session_init.go
Normal file
268
internal/app/operator_session_init.go
Normal file
@@ -0,0 +1,268 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
// SessionInit creates a local or remote session.yml skeleton.
|
||||
func SessionInit(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("session init", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var pipelinePath, campaignPath, campaignFilePath, sessionID, previousSessionID, date, title, output, audioS3Prefix, audioDir string
|
||||
var remote, force bool
|
||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
||||
fs.StringVar(&sessionID, "session-id", "", "session identifier")
|
||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "previous session identifier")
|
||||
fs.StringVar(&date, "date", "", "session date")
|
||||
fs.StringVar(&title, "title", "", "session title")
|
||||
fs.StringVar(&output, "output", "", "local output session.yml path")
|
||||
fs.StringVar(&audioS3Prefix, "audio-s3-prefix", "", "session audio S3 prefix")
|
||||
fs.StringVar(&audioDir, "audio-dir", "", "local audio directory")
|
||||
fs.BoolVar(&remote, "remote", false, "write session.yml to S3 session prefix")
|
||||
fs.BoolVar(&force, "force", false, "overwrite existing target")
|
||||
if err := parseSessionAwareFlags("session init", fs, args, &sessionID); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(sessionID) == "" {
|
||||
return fmt.Errorf("session init: session_id is required")
|
||||
}
|
||||
if (strings.TrimSpace(output) == "") == !remote {
|
||||
return fmt.Errorf("session init: specify exactly one target: --output <path> or --remote")
|
||||
}
|
||||
if strings.TrimSpace(audioDir) != "" && strings.TrimSpace(audioS3Prefix) != "" {
|
||||
return fmt.Errorf("session init: --audio-dir and --audio-s3-prefix are mutually exclusive")
|
||||
}
|
||||
|
||||
base, err := loadPipelineCampaignConfig(pipelinePath, campaignPath, campaignFilePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
|
||||
input := sessionInitInput{
|
||||
Campaign: config.CampaignID(base.Campaign),
|
||||
CampaignPath: base.CampaignPath,
|
||||
TemplateFile: base.Campaign.SessionTemplateFile,
|
||||
SessionID: sessionID,
|
||||
PreviousSessionID: previousSessionID,
|
||||
Date: date,
|
||||
Title: title,
|
||||
AudioS3Prefix: audioS3Prefix,
|
||||
AudioDir: audioDir,
|
||||
}
|
||||
data, err := buildSessionInitYAML(input)
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
label := strings.TrimSpace(output)
|
||||
if label == "" {
|
||||
label = "remote session.yml"
|
||||
}
|
||||
sessionCfg, err := config.LoadSessionBytesWithOptions(label, data, config.SessionLoadOptions{
|
||||
SessionID: sessionID,
|
||||
PreviousSessionID: previousSessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
cfg, err := config.Resolve(base.PipelinePath, base.Pipeline, base.CampaignPath, base.Campaign, label, sessionCfg, config.SessionSource{Source: "session_config", LocalPath: label})
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
|
||||
if !remote {
|
||||
if err := writeLocalFile(output, data, force); err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
_, err := fmt.Fprintf(out, "narratio session init: wrote %s\n", filepath.Clean(output))
|
||||
return err
|
||||
}
|
||||
|
||||
store, err := newCommandObjectStore(ctx, cfg, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
sessionPrefix := artifacts.S3SessionPrefix(base.Pipeline.Storage.S3.RootPrefix, config.CampaignID(base.Campaign), sessionID)
|
||||
key := artifacts.S3SessionConfigKey(sessionPrefix)
|
||||
exists, err := store.Exists(ctx, key)
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: check remote session %q: %w", key, err)
|
||||
}
|
||||
if exists && !force {
|
||||
return fmt.Errorf("session init: remote session %q already exists; pass --force to overwrite", key)
|
||||
}
|
||||
tmp, err := os.CreateTemp("", "narratio-session-init-*.yml")
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: create temp file: %w", err)
|
||||
}
|
||||
tmpPath := tmp.Name()
|
||||
defer func() { _ = os.Remove(tmpPath) }()
|
||||
if _, err := tmp.Write(data); err != nil {
|
||||
_ = tmp.Close()
|
||||
return fmt.Errorf("session init: write temp file: %w", err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return fmt.Errorf("session init: close temp file: %w", err)
|
||||
}
|
||||
if _, err := store.Upload(ctx, tmpPath, key, storage.UploadOptions{ContentType: "application/x-yaml; charset=utf-8"}); err != nil {
|
||||
return fmt.Errorf("session init: upload remote session %q: %w", key, err)
|
||||
}
|
||||
_, err = fmt.Fprintf(out, "narratio session init: wrote s3://%s/%s\n", s3BucketName(base.Pipeline), key)
|
||||
return err
|
||||
}
|
||||
|
||||
func buildSessionYAML(campaign, sessionID, previousSessionID, date, title, audioS3Prefix, audioDir string) ([]byte, error) {
|
||||
if strings.TrimSpace(date) == "" && regexp.MustCompile(`^\d{4}-\d{2}-\d{2}$`).MatchString(strings.TrimSpace(sessionID)) {
|
||||
date = strings.TrimSpace(sessionID)
|
||||
}
|
||||
type audioS3 struct {
|
||||
Prefix string `yaml:"prefix"`
|
||||
}
|
||||
type inputs struct {
|
||||
AudioDir string `yaml:"audio_dir,omitempty"`
|
||||
AudioS3 *audioS3 `yaml:"audio_s3,omitempty"`
|
||||
}
|
||||
type sessionYAML struct {
|
||||
Campaign string `yaml:"campaign"`
|
||||
SessionID string `yaml:"session_id"`
|
||||
PreviousSessionID string `yaml:"previous_session_id,omitempty"`
|
||||
Date string `yaml:"date,omitempty"`
|
||||
Title string `yaml:"title,omitempty"`
|
||||
Inputs inputs `yaml:"inputs"`
|
||||
}
|
||||
in := inputs{AudioDir: strings.TrimSpace(audioDir)}
|
||||
if in.AudioDir == "" {
|
||||
prefix := strings.TrimSpace(audioS3Prefix)
|
||||
if prefix == "" {
|
||||
prefix = "audio/"
|
||||
}
|
||||
in.AudioS3 = &audioS3{Prefix: prefix}
|
||||
}
|
||||
data, err := yaml.Marshal(sessionYAML{
|
||||
Campaign: strings.TrimSpace(campaign),
|
||||
SessionID: strings.TrimSpace(sessionID),
|
||||
PreviousSessionID: strings.TrimSpace(previousSessionID),
|
||||
Date: strings.TrimSpace(date),
|
||||
Title: strings.TrimSpace(title),
|
||||
Inputs: in,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return data, nil
|
||||
}
|
||||
|
||||
type sessionInitInput struct {
|
||||
Campaign string
|
||||
CampaignPath string
|
||||
TemplateFile string
|
||||
SessionID string
|
||||
PreviousSessionID string
|
||||
Date string
|
||||
Title string
|
||||
AudioS3Prefix string
|
||||
AudioDir string
|
||||
}
|
||||
|
||||
func buildSessionInitYAML(in sessionInitInput) ([]byte, error) {
|
||||
if strings.TrimSpace(in.TemplateFile) == "" {
|
||||
return buildSessionYAML(in.Campaign, in.SessionID, in.PreviousSessionID, in.Date, in.Title, in.AudioS3Prefix, in.AudioDir)
|
||||
}
|
||||
templatePath := resolveSessionInitTemplatePath(in.CampaignPath, in.TemplateFile)
|
||||
templateBytes, err := os.ReadFile(templatePath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read session template %q: %w", templatePath, err)
|
||||
}
|
||||
rendered, err := renderSessionInitTemplate(string(templateBytes), in)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("render session template %q: %w", templatePath, err)
|
||||
}
|
||||
return []byte(rendered), nil
|
||||
}
|
||||
|
||||
func resolveSessionInitTemplatePath(campaignPath, templateFile string) string {
|
||||
templateFile = strings.TrimSpace(templateFile)
|
||||
if filepath.IsAbs(templateFile) {
|
||||
return filepath.Clean(templateFile)
|
||||
}
|
||||
return filepath.Clean(filepath.Join(filepath.Dir(campaignPath), templateFile))
|
||||
}
|
||||
|
||||
var sessionInitTemplatePattern = regexp.MustCompile(`\{\{\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*\}\}`)
|
||||
|
||||
func renderSessionInitTemplate(content string, in sessionInitInput) (string, error) {
|
||||
values := map[string]string{
|
||||
"session_id": strings.TrimSpace(in.SessionID),
|
||||
"previous_session_id": strings.TrimSpace(in.PreviousSessionID),
|
||||
"date": strings.TrimSpace(in.Date),
|
||||
"title": strings.TrimSpace(in.Title),
|
||||
"audio_s3_prefix": strings.TrimSpace(in.AudioS3Prefix),
|
||||
"audio_dir": strings.TrimSpace(in.AudioDir),
|
||||
}
|
||||
used := map[string]struct{}{}
|
||||
unknown := map[string]struct{}{}
|
||||
missing := map[string]struct{}{}
|
||||
rendered := sessionInitTemplatePattern.ReplaceAllStringFunc(content, func(match string) string {
|
||||
parts := sessionInitTemplatePattern.FindStringSubmatch(match)
|
||||
if len(parts) < 2 {
|
||||
return match
|
||||
}
|
||||
name := parts[1]
|
||||
value, ok := values[name]
|
||||
if !ok {
|
||||
unknown[name] = struct{}{}
|
||||
return match
|
||||
}
|
||||
used[name] = struct{}{}
|
||||
if value == "" {
|
||||
missing[name] = struct{}{}
|
||||
return match
|
||||
}
|
||||
return value
|
||||
})
|
||||
if len(unknown) > 0 {
|
||||
return "", fmt.Errorf("unsupported template variable(s): %s", sortedStringSet(unknown))
|
||||
}
|
||||
if len(missing) > 0 {
|
||||
return "", fmt.Errorf("missing required template variable value(s): %s", sortedStringSet(missing))
|
||||
}
|
||||
unused := map[string]struct{}{}
|
||||
for _, name := range []string{"previous_session_id", "date", "title", "audio_s3_prefix", "audio_dir"} {
|
||||
if values[name] == "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := used[name]; !ok {
|
||||
unused[name] = struct{}{}
|
||||
}
|
||||
}
|
||||
if len(unused) > 0 {
|
||||
return "", fmt.Errorf("unused template variable value(s): %s", sortedStringSet(unused))
|
||||
}
|
||||
return rendered, nil
|
||||
}
|
||||
|
||||
func sortedStringSet(set map[string]struct{}) string {
|
||||
items := make([]string, 0, len(set))
|
||||
for item := range set {
|
||||
items = append(items, item)
|
||||
}
|
||||
sort.Strings(items)
|
||||
return strings.Join(items, ", ")
|
||||
}
|
||||
84
internal/app/operator_session_validate.go
Normal file
84
internal/app/operator_session_validate.go
Normal file
@@ -0,0 +1,84 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
// SessionValidate performs a read-only session preflight.
|
||||
func SessionValidate(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("session validate", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var flags commonConfigFlags
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
if err := parseSessionAwareFlags("session validate", fs, args, &flags.sessionID); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(flags.sessionID) == "" {
|
||||
return fmt.Errorf("session validate: session_id is required")
|
||||
}
|
||||
|
||||
findings := []finding{}
|
||||
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||
if err != nil {
|
||||
findings = append(findings, errorFinding("config", err.Error()))
|
||||
return renderFindings(out, "", "", findings)
|
||||
}
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
findings = append(findings, errorFinding("config", err.Error()))
|
||||
} else {
|
||||
findings = append(findings, okFinding("config", "resolved pipeline, campaign, and session config"))
|
||||
}
|
||||
findings = append(findings, okFinding("session", fmt.Sprintf("session source: %s", sessionSourceSummary(cfg))))
|
||||
|
||||
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
findings = append(findings, validateStableInputFindings(cfg)...)
|
||||
findings = append(findings, validateLocalAudioFindings(cfg)...)
|
||||
|
||||
store, storeErr := objectStoreIfConfigured(ctx, cfg)
|
||||
if storeErr != nil {
|
||||
findings = append(findings, errorFinding("storage", storeErr.Error()))
|
||||
}
|
||||
if cfg.Session.Inputs.AudioS3 != nil {
|
||||
if storeErr != nil {
|
||||
findings = append(findings, errorFinding("audio", "remote audio cannot be checked because storage is unavailable"))
|
||||
} else {
|
||||
findings = append(findings, validateRemoteAudioFinding(ctx, cfg, store))
|
||||
}
|
||||
}
|
||||
|
||||
requirements := artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg))
|
||||
previous := inspectPreviousArtifactReadiness(ctx, cfg, store, requirements)
|
||||
if len(previous.Requirements) == 0 {
|
||||
findings = append(findings, okFinding("previous", "no previous-session artifacts required"))
|
||||
} else if previous.MissingID {
|
||||
findings = append(findings, errorFinding("previous", "previous_session_id is required by configured previous-session artifacts"))
|
||||
} else if previous.Err != nil {
|
||||
findings = append(findings, errorFinding("previous", previous.Err.Error()))
|
||||
} else {
|
||||
for _, req := range previous.Requirements {
|
||||
findings = append(findings, okFinding("previous", fmt.Sprintf("%s required=%t", req.Name, req.Required)))
|
||||
}
|
||||
}
|
||||
|
||||
locks := inspectEffectiveLocks(ctx, cfg, store)
|
||||
if locks.Err != nil {
|
||||
findings = append(findings, errorFinding("locks", locks.Err.Error()))
|
||||
} else if len(locks.Locks.All) == 0 {
|
||||
findings = append(findings, okFinding("locks", "no effective publish locks"))
|
||||
} else {
|
||||
for _, lock := range locks.Locks.All {
|
||||
findings = append(findings, warnFinding("locks", fmt.Sprintf("%s locked: %s", lock.Source, strings.TrimSpace(lock.Reason))))
|
||||
}
|
||||
}
|
||||
if paths.ManifestPath != "" {
|
||||
findings = append(findings, infoFinding("workspace", "manifest path: "+paths.ManifestPath))
|
||||
}
|
||||
return renderFindings(out, cfg.Session.Campaign, cfg.Session.SessionID, findings)
|
||||
}
|
||||
166
internal/app/operator_status.go
Normal file
166
internal/app/operator_status.go
Normal file
@@ -0,0 +1,166 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
// Status reports effective local/remote session state.
|
||||
func Status(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("status", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var flags commonConfigFlags
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
if err := parseSessionAwareFlags("status", fs, args, &flags.sessionID); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(flags.sessionID) == "" {
|
||||
return fmt.Errorf("status: session_id is required")
|
||||
}
|
||||
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||
if err != nil {
|
||||
return fmt.Errorf("status: %w", err)
|
||||
}
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
return fmt.Errorf("status: %w", err)
|
||||
}
|
||||
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
fmt.Fprintf(out, "Session: %s\n", cfg.Session.SessionID)
|
||||
fmt.Fprintf(out, "Campaign: %s\n", cfg.Session.Campaign)
|
||||
fmt.Fprintf(out, "Workspace: %s\n", paths.Root)
|
||||
fmt.Fprintf(out, "Session config: %s\n", sessionSourceSummary(cfg))
|
||||
writeStatusStableInputs(out, inspectStableInputs(cfg))
|
||||
writeStatusLocalAudio(out, inspectLocalAudioPresence(cfg))
|
||||
|
||||
if m, err := loadLocalManifest(ctx, paths.ManifestPath); err != nil {
|
||||
fmt.Fprintf(out, "Local manifest: error: %v\n", err)
|
||||
} else if m == nil {
|
||||
fmt.Fprintln(out, "Local manifest: missing")
|
||||
} else {
|
||||
fmt.Fprintf(out, "Local manifest: %s\n", paths.ManifestPath)
|
||||
writeStageStatuses(out, m)
|
||||
}
|
||||
|
||||
store, storeErr := objectStoreIfConfigured(ctx, cfg)
|
||||
if storeErr != nil {
|
||||
fmt.Fprintf(out, "Remote publish: unavailable: %v\n", storeErr)
|
||||
} else if store != nil {
|
||||
current := inspectRemoteCurrentState(ctx, cfg, store)
|
||||
if current.Err != nil {
|
||||
fmt.Fprintf(out, "Remote publish: missing or unavailable: %v\n", current.Err)
|
||||
} else {
|
||||
fmt.Fprintf(out, "Remote publish: current run %s\n", current.State.RunID)
|
||||
fmt.Fprintf(out, "Remote manifest: %s\n", current.State.CurrentManifestKey)
|
||||
}
|
||||
}
|
||||
writeStatusRemoteAudio(ctx, out, cfg, store, storeErr)
|
||||
writeStatusPreviousArtifacts(out, inspectPreviousArtifactReadiness(
|
||||
ctx,
|
||||
cfg,
|
||||
store,
|
||||
artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg)),
|
||||
))
|
||||
|
||||
lockChecks := inspectEffectiveLocks(ctx, cfg, store)
|
||||
locks := lockChecks.Locks
|
||||
lockErr := lockChecks.Err
|
||||
if catalog, catalogErr := buildHelperArtifactCatalog(cfg); catalogErr != nil {
|
||||
fmt.Fprintf(out, "Remote outputs: error: %v\n", catalogErr)
|
||||
} else if storeErr == nil {
|
||||
catalogLocks := locks
|
||||
if lockErr != nil {
|
||||
catalogLocks = &effectiveLocks{
|
||||
Static: staticPublishLocks(cfg),
|
||||
All: staticPublishLocks(cfg),
|
||||
}
|
||||
}
|
||||
publishedRemoteState := map[string]string{}
|
||||
if store != nil {
|
||||
publishedRemoteState = remotePublishedOutputAvailability(ctx, cfg, store, catalog)
|
||||
}
|
||||
fmt.Fprintln(out, "Remote outputs:")
|
||||
writeArtifactList(out, cfg, catalog, catalogLocks, publishedRemoteState)
|
||||
}
|
||||
if lockErr != nil {
|
||||
fmt.Fprintf(out, "Publish locks: error: %v\n", lockErr)
|
||||
} else {
|
||||
writeLocks(out, cfg, locks)
|
||||
}
|
||||
fmt.Fprintln(out, "Next actions:")
|
||||
fmt.Fprintf(out, "- narratio session validate %s\n", cfg.Session.SessionID)
|
||||
fmt.Fprintf(out, "- narratio session restore %s --dry-run\n", cfg.Session.SessionID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func writeStatusStableInputs(out io.Writer, checks []stableInputCheck) {
|
||||
if len(checks) == 0 {
|
||||
return
|
||||
}
|
||||
for _, check := range checks {
|
||||
if check.Err != nil {
|
||||
if strings.TrimSpace(check.Path) != "" {
|
||||
fmt.Fprintf(out, "Stable input %s: unavailable: %v\n", check.Name, check.Err)
|
||||
} else {
|
||||
fmt.Fprintf(out, "Stable input %s: unavailable: %s\n", check.Name, check.Err.Error())
|
||||
}
|
||||
continue
|
||||
}
|
||||
fmt.Fprintf(out, "Stable input %s: %s\n", check.Name, check.Path)
|
||||
}
|
||||
}
|
||||
|
||||
func writeStatusLocalAudio(out io.Writer, check localAudioCheck) {
|
||||
if !check.Checked {
|
||||
return
|
||||
}
|
||||
if check.Err != nil {
|
||||
fmt.Fprintf(out, "Local audio: unavailable: %v\n", check.Err)
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(out, "Local audio: %d file(s)\n", len(check.Paths))
|
||||
}
|
||||
|
||||
func writeStatusRemoteAudio(ctx context.Context, out io.Writer, cfg *config.Config, store storage.ObjectStore, storeErr error) {
|
||||
if cfg.Session.Inputs.AudioS3 == nil {
|
||||
return
|
||||
}
|
||||
if storeErr != nil {
|
||||
fmt.Fprintf(out, "Remote audio: unavailable: %v\n", storeErr)
|
||||
return
|
||||
}
|
||||
check := inspectRemoteAudioPresence(ctx, cfg, store)
|
||||
if check.Err != nil {
|
||||
fmt.Fprintf(out, "Remote audio: unavailable: %v\n", check.Err)
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(out, "Remote audio: %d .flac object(s)\n", len(check.Keys))
|
||||
}
|
||||
|
||||
func writeStatusPreviousArtifacts(out io.Writer, readiness previousArtifactReadiness) {
|
||||
if len(readiness.Requirements) == 0 {
|
||||
fmt.Fprintln(out, "Previous-session artifacts: not required")
|
||||
return
|
||||
}
|
||||
if readiness.MissingID {
|
||||
fmt.Fprintln(out, "Previous-session artifacts: unavailable: previous_session_id is required by configured previous-session artifacts")
|
||||
return
|
||||
}
|
||||
if readiness.Err != nil {
|
||||
fmt.Fprintf(out, "Previous-session artifacts: unavailable: %v\n", readiness.Err)
|
||||
return
|
||||
}
|
||||
names := make([]string, 0, len(readiness.Requirements))
|
||||
for _, req := range readiness.Requirements {
|
||||
names = append(names, fmt.Sprintf("%s(required=%t)", req.Name, req.Required))
|
||||
}
|
||||
sort.Strings(names)
|
||||
fmt.Fprintf(out, "Previous-session artifacts: ready: %s\n", strings.Join(names, ", "))
|
||||
}
|
||||
@@ -19,33 +19,18 @@ func Plan(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("plan", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
|
||||
var pipelinePath string
|
||||
var sessionPath string
|
||||
var sessionID string
|
||||
var flags commonConfigFlags
|
||||
var force bool
|
||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
||||
fs.StringVar(&sessionID, "session-id", "", "session identifier for session.yml templates")
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
||||
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("plan: invalid flags: %w", err)
|
||||
if err := parseSessionAwareFlags("plan", fs, args, &flags.sessionID); err != nil {
|
||||
return err
|
||||
}
|
||||
if fs.NArg() != 0 {
|
||||
return fmt.Errorf("plan: unexpected positional arguments")
|
||||
if flags.sessionID == "" {
|
||||
return fmt.Errorf("plan: session_id is required")
|
||||
}
|
||||
resolvedPipelinePath, err := resolvePipelineConfigPath(pipelinePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("plan: %w", err)
|
||||
}
|
||||
resolvedSessionPath, err := resolveSessionConfigPath(sessionPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("plan: %w", err)
|
||||
}
|
||||
|
||||
cfg, err := config.LoadWithSessionOptions(resolvedPipelinePath, resolvedSessionPath, config.SessionLoadOptions{
|
||||
SessionID: sessionID,
|
||||
})
|
||||
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||
if err != nil {
|
||||
return fmt.Errorf("plan: %w", err)
|
||||
}
|
||||
@@ -72,7 +57,7 @@ func Plan(ctx context.Context, args []string, out io.Writer) error {
|
||||
|
||||
runCount := 0
|
||||
skipCount := 0
|
||||
if _, err := fmt.Fprintf(out, "narratio plan: workdir prepared at %s\n", paths.Root); err != nil {
|
||||
if _, err := fmt.Fprintf(out, "narratio session plan: workdir prepared at %s\n", paths.Root); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, d := range decisions {
|
||||
|
||||
@@ -15,24 +15,24 @@ import (
|
||||
|
||||
func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
var out bytes.Buffer
|
||||
args := []string{"--config", pipelinePath, "--session", sessionPath}
|
||||
args := []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}
|
||||
|
||||
if err := Plan(context.Background(), args, &out); err != nil {
|
||||
t.Fatalf("first Plan() error = %v", err)
|
||||
}
|
||||
got := out.String()
|
||||
if !strings.Contains(got, "narratio plan: workdir prepared at") {
|
||||
if !strings.Contains(got, "narratio session plan: workdir prepared at") {
|
||||
t.Fatalf("first output = %q, want workdir prepared", got)
|
||||
}
|
||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
|
||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
|
||||
if !strings.Contains(got, name+": run") {
|
||||
t.Fatalf("first output = %q, missing stage %q", got, name)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(got, "totals: run=9 skip=0") {
|
||||
if !strings.Contains(got, "totals: run=10 skip=0") {
|
||||
t.Fatalf("first output = %q, want totals", got)
|
||||
}
|
||||
|
||||
@@ -55,14 +55,14 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
|
||||
if err := Plan(context.Background(), args, &out); err != nil {
|
||||
t.Fatalf("second Plan() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "narratio plan: workdir prepared at") {
|
||||
if !strings.Contains(out.String(), "narratio session plan: workdir prepared at") {
|
||||
t.Fatalf("second output = %q, want workdir prepared", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestPlanShowsRunAndSkipFromManifest(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
@@ -74,7 +74,7 @@ func TestPlanShowsRunAndSkipFromManifest(t *testing.T) {
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
if err := Plan(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath}, &out); err != nil {
|
||||
if err := Plan(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out); err != nil {
|
||||
t.Fatalf("Plan() error = %v", err)
|
||||
}
|
||||
got := out.String()
|
||||
@@ -84,8 +84,8 @@ func TestPlanShowsRunAndSkipFromManifest(t *testing.T) {
|
||||
if !strings.Contains(got, "trim: run") {
|
||||
t.Fatalf("output = %q, want trim run", got)
|
||||
}
|
||||
if !strings.Contains(got, "totals: run=7 skip=2") {
|
||||
t.Fatalf("output = %q, want totals run=7 skip=2", got)
|
||||
if !strings.Contains(got, "totals: run=8 skip=2") {
|
||||
t.Fatalf("output = %q, want totals run=8 skip=2", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -93,6 +93,7 @@ func TestPlanFailsWhenConfiguredSecretsDirMissing(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
configDir := t.TempDir()
|
||||
pipelinePath := filepath.Join(configDir, "pipeline.yml")
|
||||
campaignPath := writeAppTestCampaignConfig(t, configDir)
|
||||
sessionPath := filepath.Join(configDir, "session.yml")
|
||||
|
||||
pipelineYAML := `workspace:
|
||||
@@ -107,8 +108,6 @@ seriatim:
|
||||
binary: seriatim
|
||||
audita:
|
||||
binary: audita
|
||||
analyzer:
|
||||
timeout: 20m
|
||||
notification:
|
||||
timeout: 10s
|
||||
`
|
||||
@@ -119,6 +118,8 @@ inputs:
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
players_file: ./players.yml
|
||||
party_file: ./party.yml
|
||||
`
|
||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||
t.Fatalf("write pipeline.yml: %v", err)
|
||||
@@ -128,7 +129,7 @@ inputs:
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
err := Plan(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath}, &out)
|
||||
err := Plan(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
|
||||
@@ -4,7 +4,7 @@ import "testing"
|
||||
|
||||
func TestBuildFullPlanOrder(t *testing.T) {
|
||||
got := BuildFullPlan()
|
||||
want := []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"}
|
||||
want := []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("len(plan) = %d, want %d", len(got), len(want))
|
||||
}
|
||||
|
||||
@@ -12,18 +12,18 @@ import (
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m *manifest.Manifest, executed []string) error {
|
||||
func runPostPublishCleanup(ctx context.Context, env *Env, manifestPath string, m *manifest.Manifest, executed []string) error {
|
||||
if env == nil || env.Config == nil || env.Config.Pipeline == nil || m == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
spoolRequested := env.Config.Pipeline.Spool.DeleteAudioAfterArchive
|
||||
workRequested := env.Config.Pipeline.Workspace.CleanupAfterArchive
|
||||
spoolRequested := env.Config.Pipeline.Spool.DeleteAudioAfterPublish
|
||||
workRequested := env.Config.Pipeline.Workspace.CleanupAfterPublish
|
||||
if !spoolRequested && !workRequested {
|
||||
return nil
|
||||
}
|
||||
|
||||
sr := archiveStageRecordForCleanup(m, executed)
|
||||
sr := publishStageRecordForCleanup(m, executed)
|
||||
if sr == nil {
|
||||
return nil
|
||||
}
|
||||
@@ -33,7 +33,7 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
||||
sr.Metadata["spool_cleanup_requested"] = spoolRequested
|
||||
sr.Metadata["workdir_cleanup_requested"] = workRequested
|
||||
|
||||
eligible, reason := archiveCleanupEligible(env.Config, sr)
|
||||
eligible, reason := publishCleanupEligible(env.Config, sr)
|
||||
if !eligible {
|
||||
sr.Metadata["cleanup_skipped"] = true
|
||||
sr.Metadata["cleanup_skipped_reason"] = reason
|
||||
@@ -63,9 +63,9 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
||||
}
|
||||
|
||||
if spoolRequested {
|
||||
if err := removeRunScopedDir(strings.TrimSpace(env.Config.Pipeline.Spool.Root), spoolDir, "pipeline.spool.delete_audio_after_archive"); err != nil {
|
||||
if err := removeRunScopedDir(strings.TrimSpace(env.Config.Pipeline.Spool.Root), spoolDir, "pipeline.spool.delete_audio_after_publish"); err != nil {
|
||||
sr.Metadata["cleanup_failed"] = true
|
||||
sr.Metadata["cleanup_failed_policy"] = "pipeline.spool.delete_audio_after_archive"
|
||||
sr.Metadata["cleanup_failed_policy"] = "pipeline.spool.delete_audio_after_publish"
|
||||
sr.Metadata["cleanup_failed_path"] = spoolDir
|
||||
_ = env.ManifestStore.Save(ctx, manifestPath, m)
|
||||
return err
|
||||
@@ -82,9 +82,9 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
||||
return nil
|
||||
}
|
||||
|
||||
if err := removeRunScopedDir(strings.TrimSpace(env.Config.Pipeline.Workspace.Root), workDir, "pipeline.workspace.cleanup_after_archive"); err != nil {
|
||||
if err := removeRunScopedDir(strings.TrimSpace(env.Config.Pipeline.Workspace.Root), workDir, "pipeline.workspace.cleanup_after_publish"); err != nil {
|
||||
sr.Metadata["cleanup_failed"] = true
|
||||
sr.Metadata["cleanup_failed_policy"] = "pipeline.workspace.cleanup_after_archive"
|
||||
sr.Metadata["cleanup_failed_policy"] = "pipeline.workspace.cleanup_after_publish"
|
||||
sr.Metadata["cleanup_failed_path"] = workDir
|
||||
_ = env.ManifestStore.Save(ctx, manifestPath, m)
|
||||
return err
|
||||
@@ -96,112 +96,87 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
||||
return nil
|
||||
}
|
||||
|
||||
func archiveStageRecordForCleanup(m *manifest.Manifest, executed []string) *manifest.StageRecord {
|
||||
func publishStageRecordForCleanup(m *manifest.Manifest, executed []string) *manifest.StageRecord {
|
||||
if m == nil {
|
||||
return nil
|
||||
}
|
||||
archiveRan := false
|
||||
publishRan := false
|
||||
for _, name := range executed {
|
||||
if name == "archive" {
|
||||
archiveRan = true
|
||||
if name == "publish" {
|
||||
publishRan = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !archiveRan {
|
||||
if !publishRan {
|
||||
return nil
|
||||
}
|
||||
sr := m.Stages["archive"]
|
||||
sr := m.Stages["publish"]
|
||||
if sr == nil || sr.Status != manifest.StatusSucceeded {
|
||||
return nil
|
||||
}
|
||||
return sr
|
||||
}
|
||||
|
||||
func archiveCleanupEligible(cfg *config.Config, sr *manifest.StageRecord) (bool, string) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Archive == nil {
|
||||
return false, "archive configuration is missing"
|
||||
func publishCleanupEligible(cfg *config.Config, sr *manifest.StageRecord) (bool, string) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Publish == nil {
|
||||
return false, "publish configuration is missing"
|
||||
}
|
||||
enabled := true
|
||||
if cfg.Pipeline.Archive.Enabled != nil {
|
||||
enabled = *cfg.Pipeline.Archive.Enabled
|
||||
if cfg.Pipeline.Publish.Enabled != nil {
|
||||
enabled = *cfg.Pipeline.Publish.Enabled
|
||||
}
|
||||
if !enabled {
|
||||
return false, "archive.enabled is false"
|
||||
return false, "publish.enabled is false"
|
||||
}
|
||||
uploadRun := true
|
||||
if cfg.Pipeline.Archive.UploadRun != nil {
|
||||
uploadRun = *cfg.Pipeline.Archive.UploadRun
|
||||
if cfg.Pipeline.Publish.UploadRun != nil {
|
||||
uploadRun = *cfg.Pipeline.Publish.UploadRun
|
||||
}
|
||||
if !uploadRun {
|
||||
return false, "archive.upload_run is false"
|
||||
return false, "publish.upload_run is false"
|
||||
}
|
||||
if sr == nil || sr.Metadata == nil {
|
||||
return false, "archive metadata is missing"
|
||||
return false, "publish metadata is missing"
|
||||
}
|
||||
if skipped, _ := sr.Metadata["skipped"].(bool); skipped {
|
||||
return false, "archive stage was skipped"
|
||||
return false, "publish stage was skipped"
|
||||
}
|
||||
if uploaded, _ := sr.Metadata["uploaded"].(bool); !uploaded {
|
||||
return false, "archive did not upload run record"
|
||||
return false, "publish did not upload run record"
|
||||
}
|
||||
if pointer, _ := sr.Metadata["current_pointer_written"].(bool); !pointer {
|
||||
return false, "archive did not write current pointer"
|
||||
return false, "publish did not write current pointer"
|
||||
}
|
||||
if strings.TrimSpace(asString(sr.Metadata["current_run_id_key"])) == "" {
|
||||
return false, "archive current run pointer key is missing"
|
||||
return false, "publish current run pointer key is missing"
|
||||
}
|
||||
return true, ""
|
||||
}
|
||||
|
||||
type scopedDir struct {
|
||||
RootAbs string
|
||||
TargetAbs string
|
||||
Exists bool
|
||||
}
|
||||
|
||||
func removeRunScopedDir(root, target, policy string) error {
|
||||
cleanRoot := strings.TrimSpace(root)
|
||||
cleanTarget := strings.TrimSpace(target)
|
||||
if cleanRoot == "" {
|
||||
return fmt.Errorf("cleanup policy %s: root path is required", policy)
|
||||
}
|
||||
if cleanTarget == "" {
|
||||
return fmt.Errorf("cleanup policy %s: target path is required", policy)
|
||||
}
|
||||
|
||||
rootAbs, err := filepath.Abs(cleanRoot)
|
||||
dir, err := validateScopedDir(root, target, policy)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
||||
return err
|
||||
}
|
||||
targetAbs, err := filepath.Abs(cleanTarget)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cleanup policy %s: resolve target %q: %w", policy, cleanTarget, err)
|
||||
if !dir.Exists {
|
||||
return nil
|
||||
}
|
||||
|
||||
rel, err := filepath.Rel(rootAbs, targetAbs)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cleanup policy %s: relative path from %q to %q: %w", policy, rootAbs, targetAbs, err)
|
||||
}
|
||||
if rel == "." {
|
||||
return fmt.Errorf("cleanup policy %s: refusing to delete root directory %q", policy, rootAbs)
|
||||
}
|
||||
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
||||
return fmt.Errorf("cleanup policy %s: refusing to delete path outside root: root=%q target=%q", policy, rootAbs, targetAbs)
|
||||
}
|
||||
|
||||
info, err := os.Lstat(targetAbs)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("cleanup policy %s: stat target %q: %w", policy, targetAbs, err)
|
||||
}
|
||||
if info.Mode()&os.ModeSymlink != 0 {
|
||||
return fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, targetAbs)
|
||||
}
|
||||
if !info.IsDir() {
|
||||
return fmt.Errorf("cleanup policy %s: target %q is not a directory", policy, targetAbs)
|
||||
}
|
||||
if err := os.RemoveAll(targetAbs); err != nil {
|
||||
return fmt.Errorf("cleanup policy %s: remove %q: %w", policy, targetAbs, err)
|
||||
if err := os.RemoveAll(dir.TargetAbs); err != nil {
|
||||
return fmt.Errorf("cleanup policy %s: remove %q: %w", policy, dir.TargetAbs, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateScopedDir(root, target, policy string) (scopedDir, error) {
|
||||
return validateScopedTarget(root, target, policy, true)
|
||||
}
|
||||
|
||||
func asString(v any) string {
|
||||
s, _ := v.(string)
|
||||
return s
|
||||
@@ -16,15 +16,15 @@ import (
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||
)
|
||||
|
||||
type archiveSuccessStage struct {
|
||||
type publishSuccessStage struct {
|
||||
metadata map[string]any
|
||||
}
|
||||
|
||||
func (archiveSuccessStage) Name() string { return "archive" }
|
||||
func (archiveSuccessStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||
func (s archiveSuccessStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
||||
func (publishSuccessStage) Name() string { return "publish" }
|
||||
func (publishSuccessStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||
func (s publishSuccessStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
||||
md := map[string]any{
|
||||
"stage": "archive",
|
||||
"stage": "publish",
|
||||
"uploaded": true,
|
||||
"current_pointer_written": true,
|
||||
"current_run_id_key": "dnd/campaigns/sample-campaign/sessions/2026-05-03/current/run_id.txt",
|
||||
@@ -43,12 +43,12 @@ func (notifyFailStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest
|
||||
return nil, errors.New("notify failed")
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupDisabledKeepsLocalDirs(t *testing.T) {
|
||||
func TestPostPublishCleanupDisabledKeepsLocalDirs(t *testing.T) {
|
||||
cfg, seed := cleanupFixtureConfig(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = false
|
||||
cfg.Pipeline.Workspace.CleanupAfterArchive = false
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = false
|
||||
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
||||
|
||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -57,12 +57,12 @@ func TestPostArchiveCleanupDisabledKeepsLocalDirs(t *testing.T) {
|
||||
assertExists(t, seed.localSourceAudio)
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupSpoolOnly(t *testing.T) {
|
||||
func TestPostPublishCleanupSpoolOnly(t *testing.T) {
|
||||
cfg, seed := cleanupFixtureConfig(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterArchive = false
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
||||
|
||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -71,55 +71,57 @@ func TestPostArchiveCleanupSpoolOnly(t *testing.T) {
|
||||
assertExists(t, seed.localSourceAudio)
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupWorkdirOnly(t *testing.T) {
|
||||
func TestPostPublishCleanupWorkdirOnly(t *testing.T) {
|
||||
cfg, seed := cleanupFixtureConfig(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = false
|
||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = false
|
||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||
|
||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
|
||||
assertExists(t, cfg.Pipeline.Workspace.Root)
|
||||
assertExists(t, seed.otherRunDir)
|
||||
assertExists(t, seed.previousCachePath)
|
||||
assertMissing(t, seed.runWorkDir)
|
||||
assertExists(t, seed.spoolAudioDir)
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupBothPolicies(t *testing.T) {
|
||||
func TestPostPublishCleanupBothPolicies(t *testing.T) {
|
||||
cfg, seed := cleanupFixtureConfig(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||
|
||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
|
||||
assertMissing(t, seed.spoolAudioDir)
|
||||
assertMissing(t, seed.runWorkDir)
|
||||
assertExists(t, seed.otherRunDir)
|
||||
assertExists(t, seed.previousCachePath)
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupNotRunWhenArchiveFails(t *testing.T) {
|
||||
func TestPostPublishCleanupNotRunWhenPublishFails(t *testing.T) {
|
||||
cfg, seed := cleanupFixtureConfig(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||
|
||||
_, err := executeStages(context.Background(), cfg, []stage.Stage{failingStage{name: "archive", err: errors.New("archive failed")}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||
if err == nil || !strings.Contains(err.Error(), "stage \"archive\" failed") {
|
||||
t.Fatalf("executeStages() error = %v, want archive failure", err)
|
||||
_, err := executeStages(context.Background(), cfg, []stage.Stage{failingStage{name: "publish", err: errors.New("publish failed")}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||
if err == nil || !strings.Contains(err.Error(), "stage \"publish\" failed") {
|
||||
t.Fatalf("executeStages() error = %v, want publish failure", err)
|
||||
}
|
||||
|
||||
assertExists(t, seed.spoolAudioDir)
|
||||
assertExists(t, seed.runWorkDir)
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupNotRunWhenArchiveSkipped(t *testing.T) {
|
||||
func TestPostPublishCleanupNotRunWhenPublishSkipped(t *testing.T) {
|
||||
cfg, seed := cleanupFixtureConfig(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||
|
||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{metadata: map[string]any{"skipped": true}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{metadata: map[string]any{"skipped": true}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -127,12 +129,12 @@ func TestPostArchiveCleanupNotRunWhenArchiveSkipped(t *testing.T) {
|
||||
assertExists(t, seed.runWorkDir)
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupNotRunWhenCurrentPointerMissing(t *testing.T) {
|
||||
func TestPostPublishCleanupNotRunWhenCurrentPointerMissing(t *testing.T) {
|
||||
cfg, seed := cleanupFixtureConfig(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||
|
||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{metadata: map[string]any{"current_pointer_written": false}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{metadata: map[string]any{"current_pointer_written": false}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -140,13 +142,13 @@ func TestPostArchiveCleanupNotRunWhenCurrentPointerMissing(t *testing.T) {
|
||||
assertExists(t, seed.runWorkDir)
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupNotRunWhenArchiveUploadDisabled(t *testing.T) {
|
||||
func TestPostPublishCleanupNotRunWhenPublishUploadDisabled(t *testing.T) {
|
||||
cfg, seed := cleanupFixtureConfig(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
||||
cfg.Pipeline.Archive.UploadRun = boolPtr(false)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||
cfg.Pipeline.Publish.UploadRun = boolPtr(false)
|
||||
|
||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -154,12 +156,12 @@ func TestPostArchiveCleanupNotRunWhenArchiveUploadDisabled(t *testing.T) {
|
||||
assertExists(t, seed.runWorkDir)
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupWaitsUntilAllStagesSucceed(t *testing.T) {
|
||||
func TestPostPublishCleanupWaitsUntilAllStagesSucceed(t *testing.T) {
|
||||
cfg, seed := cleanupFixtureConfig(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||
|
||||
_, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}, notifyFailStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||
_, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}, notifyFailStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||
if err == nil || !strings.Contains(err.Error(), "stage \"notify\" failed") {
|
||||
t.Fatalf("executeStages() error = %v, want notify failure", err)
|
||||
}
|
||||
@@ -168,10 +170,10 @@ func TestPostArchiveCleanupWaitsUntilAllStagesSucceed(t *testing.T) {
|
||||
assertExists(t, seed.runWorkDir)
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupFailsOnUnsafePath(t *testing.T) {
|
||||
func TestPostPublishCleanupFailsOnUnsafePath(t *testing.T) {
|
||||
cfg, _ := cleanupFixtureConfig(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterArchive = false
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
||||
|
||||
manifestPath := manifestPathFor(cfg)
|
||||
store := &manifest.LocalStore{}
|
||||
@@ -184,27 +186,27 @@ func TestPostArchiveCleanupFailsOnUnsafePath(t *testing.T) {
|
||||
t.Fatalf("Save() error = %v", err)
|
||||
}
|
||||
|
||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||
if err == nil || !strings.Contains(err.Error(), "refusing to delete path outside root") {
|
||||
t.Fatalf("executeStages() error = %v, want safe-path failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
|
||||
cfg, seed, runID := archiveStageCleanupFixture(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
||||
cfg.Pipeline.Archive.PromoteArtifacts = []config.ArchivePromotionRule{
|
||||
{From: "artifacts/missing.md", To: "artifacts/missing.md", Required: boolPtr(true)},
|
||||
func TestPostPublishCleanupNotRunWhenOutputIsMissing(t *testing.T) {
|
||||
cfg, seed, runID := publishStageCleanupFixture(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||
cfg.Pipeline.Publish.Outputs = []config.PublishOutputRule{
|
||||
{Source: "narratio.transcript.base", Dest: "transcripts/base.json", Required: boolPtr(true)},
|
||||
}
|
||||
|
||||
archiveStageImpl, err := stage.Select("archive")
|
||||
publishStageImpl, err := stage.Select("publish")
|
||||
if err != nil {
|
||||
t.Fatalf("Select(archive) error = %v", err)
|
||||
t.Fatalf("Select(publish) error = %v", err)
|
||||
}
|
||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveStageImpl}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||
if err == nil || !strings.Contains(err.Error(), "required promotion source missing") {
|
||||
t.Fatalf("executeStages() error = %v, want promotion-missing failure", err)
|
||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishStageImpl}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||
if err == nil || !strings.Contains(err.Error(), "required output source unavailable") {
|
||||
t.Fatalf("executeStages() error = %v, want required output source unavailable failure", err)
|
||||
}
|
||||
|
||||
assertExists(t, seed.spoolAudioDir)
|
||||
@@ -213,17 +215,17 @@ func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
|
||||
assertExists(t, artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
||||
cfg, seed, _ := archiveStageCleanupFixture(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
||||
func TestPostPublishCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
||||
cfg, seed, _ := publishStageCleanupFixture(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||
failKey := seed.sessionPrefix + "current/manifest.json"
|
||||
|
||||
archiveStageImpl, err := stage.Select("archive")
|
||||
publishStageImpl, err := stage.Select("publish")
|
||||
if err != nil {
|
||||
t.Fatalf("Select(archive) error = %v", err)
|
||||
t.Fatalf("Select(publish) error = %v", err)
|
||||
}
|
||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveStageImpl}, RunOptions{
|
||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishStageImpl}, RunOptions{
|
||||
Env: &Env{ObjectStore: &failKeyStore{delegate: &storage.FakeBackend{}, failKey: failKey}},
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "current manifest") {
|
||||
@@ -234,17 +236,17 @@ func TestPostArchiveCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
||||
assertExists(t, seed.runWorkDir)
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupNotRunWhenCurrentPointerUploadFails(t *testing.T) {
|
||||
cfg, seed, _ := archiveStageCleanupFixture(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
||||
func TestPostPublishCleanupNotRunWhenCurrentPointerUploadFails(t *testing.T) {
|
||||
cfg, seed, _ := publishStageCleanupFixture(t)
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||
failKey := seed.sessionPrefix + "current/run_id.txt"
|
||||
|
||||
archiveStageImpl, err := stage.Select("archive")
|
||||
publishStageImpl, err := stage.Select("publish")
|
||||
if err != nil {
|
||||
t.Fatalf("Select(archive) error = %v", err)
|
||||
t.Fatalf("Select(publish) error = %v", err)
|
||||
}
|
||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveStageImpl}, RunOptions{
|
||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishStageImpl}, RunOptions{
|
||||
Env: &Env{ObjectStore: &failKeyStore{delegate: &storage.FakeBackend{}, failKey: failKey}},
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "current run pointer") {
|
||||
@@ -256,29 +258,37 @@ func TestPostArchiveCleanupNotRunWhenCurrentPointerUploadFails(t *testing.T) {
|
||||
}
|
||||
|
||||
type cleanupSeed struct {
|
||||
runWorkDir string
|
||||
otherRunDir string
|
||||
spoolAudioDir string
|
||||
localSourceAudio string
|
||||
sessionPrefix string
|
||||
runWorkDir string
|
||||
otherRunDir string
|
||||
spoolAudioDir string
|
||||
localSourceAudio string
|
||||
previousCachePath string
|
||||
sessionPrefix string
|
||||
}
|
||||
|
||||
func cleanupFixtureConfig(t *testing.T) (*config.Config, cleanupSeed) {
|
||||
t.Helper()
|
||||
|
||||
cfg := testConfig(t)
|
||||
cfg.Pipeline.Archive = &config.ArchiveConfig{Enabled: boolPtr(true), UploadRun: boolPtr(true)}
|
||||
cfg.Pipeline.Publish = &config.PublishConfig{Enabled: boolPtr(true), UploadRun: boolPtr(true)}
|
||||
cfg.Pipeline.Spool.Root = filepath.Join(t.TempDir(), "spool")
|
||||
|
||||
runID := "20260516T010203Z-1a2b3c4d"
|
||||
runWorkDir := artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||
otherRunDir := artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, "20260516T010204Z-5e6f7a8b")
|
||||
spoolAudioDir := artifacts.SessionSpoolAudioDir(cfg.Pipeline.Spool.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||
previousCachePath := artifacts.SessionPreviousArtifactPathForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
"session_recap.md",
|
||||
)
|
||||
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "manifest.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "logs", "stage.log"), "log\n")
|
||||
mustWriteFile(t, filepath.Join(otherRunDir, "logs", "stage.log"), "other\n")
|
||||
mustWriteFile(t, filepath.Join(spoolAudioDir, "speaker.flac"), "flac\n")
|
||||
mustWriteFile(t, previousCachePath, "# previous recap\n")
|
||||
|
||||
localSourceAudio := filepath.Join(filepath.Dir(cfg.SessionPath), "audio", "alice.flac")
|
||||
mustWriteFile(t, localSourceAudio, "source\n")
|
||||
@@ -301,15 +311,16 @@ func cleanupFixtureConfig(t *testing.T) (*config.Config, cleanupSeed) {
|
||||
}
|
||||
|
||||
return cfg, cleanupSeed{
|
||||
runWorkDir: runWorkDir,
|
||||
otherRunDir: otherRunDir,
|
||||
spoolAudioDir: spoolAudioDir,
|
||||
localSourceAudio: localSourceAudio,
|
||||
sessionPrefix: seed.S3SessionPrefix,
|
||||
runWorkDir: runWorkDir,
|
||||
otherRunDir: otherRunDir,
|
||||
spoolAudioDir: spoolAudioDir,
|
||||
localSourceAudio: localSourceAudio,
|
||||
previousCachePath: previousCachePath,
|
||||
sessionPrefix: seed.S3SessionPrefix,
|
||||
}
|
||||
}
|
||||
|
||||
func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, string) {
|
||||
func publishStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, string) {
|
||||
t.Helper()
|
||||
|
||||
cfg, seed := cleanupFixtureConfig(t)
|
||||
@@ -318,15 +329,22 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
||||
Bucket: "my-dnd-archive",
|
||||
RootPrefix: "dnd",
|
||||
}
|
||||
cfg.Pipeline.Archive = &config.ArchiveConfig{
|
||||
cfg.Pipeline.Publish = &config.PublishConfig{
|
||||
Enabled: boolPtr(true),
|
||||
UploadRun: boolPtr(true),
|
||||
PromoteArtifacts: []config.ArchivePromotionRule{
|
||||
{From: "transcripts/trimmed.json", To: "transcripts/trimmed.json", Required: boolPtr(true)},
|
||||
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md", Required: boolPtr(true)},
|
||||
Outputs: []config.PublishOutputRule{
|
||||
{Source: "narratio.transcript.final_trimmed", Dest: "transcripts/final.trimmed.json", Required: boolPtr(true)},
|
||||
{Source: "narratio.artifact.session_recap", Dest: "artifacts/session_recap.md", Required: boolPtr(true)},
|
||||
},
|
||||
}
|
||||
writeArchiveFixtureRunFiles(
|
||||
cfg.Pipeline.Scriptorium = &config.ScriptoriumConfig{
|
||||
Artifacts: map[string]config.ScriptoriumArtifactConfig{
|
||||
"session_recap": {
|
||||
OutputPath: "artifacts/session_recap.md",
|
||||
},
|
||||
},
|
||||
}
|
||||
writePublishFixtureRunFiles(
|
||||
t,
|
||||
seed.runWorkDir,
|
||||
artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID),
|
||||
@@ -337,7 +355,7 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze"} {
|
||||
seedManifest.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
||||
}
|
||||
seedManifest.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
@@ -349,18 +367,18 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
||||
return cfg, seed, runID
|
||||
}
|
||||
|
||||
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir, sessionRoot string) {
|
||||
func writePublishFixtureRunFiles(t *testing.T, runWorkDir, sessionRoot string) {
|
||||
t.Helper()
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "prepare", "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "trim", "outputs", "transcripts", "trimmed.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "trim", "outputs", "transcripts", "final.trimmed.json"), "{\"segments\":[]}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "analyze", "outputs", "artifacts", "session_recap.md"), "# recap\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "polish", "reports", "audita.report.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "merge", "config", "seriatim.generated.yml"), "key: value\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "logs", "audita.stderr.log"), "stderr\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "manifest.json"), "{}\n")
|
||||
|
||||
mustWriteFile(t, filepath.Join(sessionRoot, "transcripts", "trimmed.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(sessionRoot, "transcripts", "final.trimmed.json"), "{\"segments\":[]}\n")
|
||||
mustWriteFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# recap\n")
|
||||
}
|
||||
|
||||
157
internal/app/remote_locks.go
Normal file
157
internal/app/remote_locks.go
Normal file
@@ -0,0 +1,157 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
type effectiveLocks struct {
|
||||
Static []config.PublishLockRule
|
||||
Remote []config.PublishLockRule
|
||||
All []config.PublishLockRule
|
||||
Key string
|
||||
}
|
||||
|
||||
func remoteLocksKey(cfg *config.Config) (string, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
||||
return "", fmt.Errorf("resolved config is required")
|
||||
}
|
||||
if cfg.Pipeline.Storage.S3 == nil {
|
||||
return "", fmt.Errorf("pipeline.storage.s3 configuration is required")
|
||||
}
|
||||
sessionPrefix := artifacts.S3SessionPrefix(
|
||||
cfg.Pipeline.Storage.S3.RootPrefix,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
)
|
||||
return artifacts.S3SessionLocksKey(sessionPrefix), nil
|
||||
}
|
||||
|
||||
func loadRemoteLockStore(ctx context.Context, cfg *config.Config, store storage.ObjectStore) (*config.PublishLockStore, string, error) {
|
||||
key, err := remoteLocksKey(cfg)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
exists, err := store.Exists(ctx, key)
|
||||
if err != nil {
|
||||
return nil, key, fmt.Errorf("check remote locks %q: %w", key, err)
|
||||
}
|
||||
if !exists {
|
||||
return &config.PublishLockStore{}, key, nil
|
||||
}
|
||||
tmp, err := storage.DownloadObjectToTemp(ctx, store, key, "narratio-locks-*.yml")
|
||||
if err != nil {
|
||||
return nil, key, fmt.Errorf("download remote locks %q: %w", key, err)
|
||||
}
|
||||
defer func() { _ = os.Remove(tmp) }()
|
||||
data, err := os.ReadFile(tmp)
|
||||
if err != nil {
|
||||
return nil, key, fmt.Errorf("read remote locks %q: %w", key, err)
|
||||
}
|
||||
lockStore, err := config.LoadPublishLockStoreBytes("s3://"+s3BucketName(cfg.Pipeline)+"/"+key, data, cfg.Pipeline.Scriptorium)
|
||||
if err != nil {
|
||||
return nil, key, err
|
||||
}
|
||||
return lockStore, key, nil
|
||||
}
|
||||
|
||||
func loadEffectiveLocks(ctx context.Context, cfg *config.Config, store storage.ObjectStore) (*effectiveLocks, error) {
|
||||
staticLocks := staticPublishLocks(cfg)
|
||||
if store == nil {
|
||||
return &effectiveLocks{
|
||||
Static: staticLocks,
|
||||
All: append([]config.PublishLockRule(nil), staticLocks...),
|
||||
}, nil
|
||||
}
|
||||
lockStore, key, err := loadRemoteLockStore(ctx, cfg, store)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
remoteLocks := append([]config.PublishLockRule(nil), lockStore.Locks...)
|
||||
return &effectiveLocks{
|
||||
Static: staticLocks,
|
||||
Remote: remoteLocks,
|
||||
All: config.MergePublishLockRules(staticLocks, remoteLocks),
|
||||
Key: key,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func staticPublishLocks(cfg *config.Config) []config.PublishLockRule {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Publish == nil {
|
||||
return nil
|
||||
}
|
||||
return append([]config.PublishLockRule(nil), cfg.Pipeline.Publish.Locks...)
|
||||
}
|
||||
|
||||
func applyEffectiveLocks(cfg *config.Config, locks []config.PublishLockRule) {
|
||||
if cfg == nil || cfg.Pipeline == nil {
|
||||
return
|
||||
}
|
||||
if cfg.Pipeline.Publish == nil {
|
||||
cfg.Pipeline.Publish = &config.PublishConfig{}
|
||||
}
|
||||
cfg.Pipeline.Publish.Locks = append([]config.PublishLockRule(nil), locks...)
|
||||
}
|
||||
|
||||
func uploadRemoteLockStore(ctx context.Context, store storage.ObjectStore, key string, lockStore *config.PublishLockStore) error {
|
||||
data, err := config.MarshalPublishLockStore(lockStore)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp, err := os.CreateTemp("", "narratio-locks-upload-*.yml")
|
||||
if err != nil {
|
||||
return fmt.Errorf("create lock store temp file: %w", err)
|
||||
}
|
||||
tmpPath := tmp.Name()
|
||||
defer func() { _ = os.Remove(tmpPath) }()
|
||||
if _, err := tmp.Write(data); err != nil {
|
||||
_ = tmp.Close()
|
||||
return fmt.Errorf("write lock store temp file: %w", err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return fmt.Errorf("close lock store temp file: %w", err)
|
||||
}
|
||||
if _, err := store.Upload(ctx, tmpPath, key, storage.UploadOptions{ContentType: "application/x-yaml; charset=utf-8"}); err != nil {
|
||||
return fmt.Errorf("upload remote locks %q: %w", key, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func lockSourceSet(locks []config.PublishLockRule) map[string]config.PublishLockRule {
|
||||
out := make(map[string]config.PublishLockRule, len(locks))
|
||||
for _, lock := range locks {
|
||||
source := strings.TrimSpace(lock.Source)
|
||||
if source == "" {
|
||||
continue
|
||||
}
|
||||
lock.Source = source
|
||||
lock.Reason = strings.TrimSpace(lock.Reason)
|
||||
out[source] = lock
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func writeLocalFile(path string, data []byte, force bool) error {
|
||||
cleaned := filepath.Clean(strings.TrimSpace(path))
|
||||
if cleaned == "" || cleaned == "." {
|
||||
return fmt.Errorf("output path is required")
|
||||
}
|
||||
if !force {
|
||||
if _, err := os.Stat(cleaned); err == nil {
|
||||
return fmt.Errorf("output file %q already exists; pass --force to overwrite", cleaned)
|
||||
} else if err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("check output file %q: %w", cleaned, err)
|
||||
}
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(cleaned), 0o755); err != nil {
|
||||
return fmt.Errorf("create output directory: %w", err)
|
||||
}
|
||||
return os.WriteFile(cleaned, data, 0o644)
|
||||
}
|
||||
294
internal/app/remote_session_test.go
Normal file
294
internal/app/remote_session_test.go
Normal file
@@ -0,0 +1,294 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func TestExecuteRemoteSessionFallbackLoadsFromObjectStore(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
remoteKey := seedRemoteSessionConfig(t, fake, "2026-05-03", `session_id: 2026-05-03
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
`)
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{filepath.Join(t.TempDir(), "session.yml")})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if storeInitCalls != 1 {
|
||||
t.Fatalf("object store init calls = %d, want 1", storeInitCalls)
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "narratio session plan: workdir prepared") {
|
||||
t.Fatalf("stdout = %q, want plan output", stdout.String())
|
||||
}
|
||||
if _, ok := fake.Objects[remoteKey]; !ok {
|
||||
t.Fatalf("remote session key %q was not seeded", remoteKey)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRemoteSessionFallbackLoadsSecretsBeforeObjectStoreInit(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
accessKeyEnv := "NARRATIO_TEST_REMOTE_SESSION_KEY_ID"
|
||||
secretKeyEnv := "NARRATIO_TEST_REMOTE_SESSION_SECRET"
|
||||
restoreEnvAfterTest(t, accessKeyEnv, secretKeyEnv)
|
||||
secretsDir := t.TempDir()
|
||||
mustWriteTestFile(t, filepath.Join(secretsDir, accessKeyEnv), "remote-session-key-id\n")
|
||||
mustWriteTestFile(t, filepath.Join(secretsDir, secretKeyEnv), "remote-session-secret\n")
|
||||
addSecretsToPipelineConfig(t, pipelinePath, secretsDir, accessKeyEnv, secretKeyEnv)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
seedRemoteSessionConfig(t, fake, "2026-05-03", `session_id: 2026-05-03
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
`)
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
origSessionDefaults := append([]string(nil), config.DefaultSessionConfigSearchPaths...)
|
||||
config.DefaultSessionConfigSearchPaths = []string{filepath.Join(t.TempDir(), "session.yml")}
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
if os.Getenv(accessKeyEnv) != "remote-session-key-id" || os.Getenv(secretKeyEnv) != "remote-session-secret" {
|
||||
return nil, fmt.Errorf("secrets were not loaded before remote session object store init")
|
||||
}
|
||||
return fake, nil
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
config.DefaultSessionConfigSearchPaths = origSessionDefaults
|
||||
})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteExplicitLocalSessionPrecedenceSkipsRemote(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{filepath.Join(t.TempDir(), "session.yml")})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if storeInitCalls != 0 {
|
||||
t.Fatalf("object store init calls = %d, want 0", storeInitCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteLocalSessionDiscoveryPrecedenceSkipsRemote(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if storeInitCalls != 0 {
|
||||
t.Fatalf("object store init calls = %d, want 0", storeInitCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRemoteSessionMissingObjectFailsClearly(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
var storeInitCalls int
|
||||
missingSessionPath := filepath.Join(t.TempDir(), "session.yml")
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{missingSessionPath})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "remote session") || !strings.Contains(stderr.String(), "session.yml") || !strings.Contains(stderr.String(), "not found") {
|
||||
t.Fatalf("stderr = %q, want remote session not found context", stderr.String())
|
||||
}
|
||||
if !strings.Contains(stderr.String(), missingSessionPath) {
|
||||
t.Fatalf("stderr = %q, want local searched path", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRemoteSessionRequiresSessionID(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, &storage.FakeBackend{}, &storeInitCalls, []string{filepath.Join(t.TempDir(), "session.yml")})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "plan", "--config", pipelinePath, "--campaign-file", campaignPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "plan: session_id is required") {
|
||||
t.Fatalf("stderr = %q, want session_id guidance", stderr.String())
|
||||
}
|
||||
if storeInitCalls != 0 {
|
||||
t.Fatalf("object store init calls = %d, want 0", storeInitCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRemoteSessionStorageInitErrorFailsClearly(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
origSessionDefaults := append([]string(nil), config.DefaultSessionConfigSearchPaths...)
|
||||
config.DefaultSessionConfigSearchPaths = []string{filepath.Join(t.TempDir(), "session.yml")}
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
return nil, errors.New("storage unavailable")
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
config.DefaultSessionConfigSearchPaths = origSessionDefaults
|
||||
})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "storage unavailable") || !strings.Contains(stderr.String(), "remote session") {
|
||||
t.Fatalf("stderr = %q, want remote storage context", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRemoteSessionMalformedYAMLFailsStrictDecode(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
seedRemoteSessionConfig(t, fake, "2026-05-03", "session_id: 2026-05-03\nunknown: true\n")
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{filepath.Join(t.TempDir(), "session.yml")})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "strict decode failed") {
|
||||
t.Fatalf("stderr = %q, want strict decode context", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRemoteSessionTemplateFailsConcreteSessionCheck(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
seedRemoteSessionConfig(t, fake, "2026-05-03", `session_id: "{{ session_id }}"
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
`)
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{filepath.Join(t.TempDir(), "session.yml")})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "session.yml must be concrete") || !strings.Contains(stderr.String(), "run narratio session init") {
|
||||
t.Fatalf("stderr = %q, want concrete session guidance", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRemoteSessionMismatchFails(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
seedRemoteSessionConfig(t, fake, "2026-05-03", "session_id: 2026-05-04\ninputs:\n audio_s3:\n prefix: audio/\n")
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{filepath.Join(t.TempDir(), "session.yml")})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "session_id mismatch") {
|
||||
t.Fatalf("stderr = %q, want session_id mismatch", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func restoreAppConfigTestGlobals(t *testing.T, fake *storage.FakeBackend, storeInitCalls *int, sessionDefaults []string) {
|
||||
t.Helper()
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
origSessionDefaults := append([]string(nil), config.DefaultSessionConfigSearchPaths...)
|
||||
config.DefaultSessionConfigSearchPaths = append([]string(nil), sessionDefaults...)
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
if storeInitCalls != nil {
|
||||
(*storeInitCalls)++
|
||||
}
|
||||
return fake, nil
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
config.DefaultSessionConfigSearchPaths = origSessionDefaults
|
||||
})
|
||||
}
|
||||
|
||||
func seedRemoteSessionConfig(t *testing.T, fake *storage.FakeBackend, sessionID, content string) string {
|
||||
t.Helper()
|
||||
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", sessionID)
|
||||
remoteKey := artifacts.S3SessionConfigKey(sessionPrefix)
|
||||
fake.SeedObject(storage.FakeObject{
|
||||
Key: remoteKey,
|
||||
Data: []byte(content),
|
||||
ETag: "remote-session-etag",
|
||||
})
|
||||
return remoteKey
|
||||
}
|
||||
|
||||
func addSecretsToPipelineConfig(t *testing.T, pipelinePath, secretsDir, accessKeyEnv, secretKeyEnv string) {
|
||||
t.Helper()
|
||||
pipelineData, err := os.ReadFile(pipelinePath)
|
||||
if err != nil {
|
||||
t.Fatalf("read pipeline: %v", err)
|
||||
}
|
||||
pipelineYAML := strings.Replace(
|
||||
string(pipelineData),
|
||||
"storage:\n backend: s3\n s3:\n bucket: test-bucket\n",
|
||||
"storage:\n backend: s3\n s3:\n bucket: test-bucket\n access_key_id_env: "+accessKeyEnv+"\n secret_access_key_env: "+secretKeyEnv+"\nsecrets:\n env_dir: "+secretsDir+"\n",
|
||||
1,
|
||||
)
|
||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||
t.Fatalf("write pipeline: %v", err)
|
||||
}
|
||||
}
|
||||
138
internal/app/restore.go
Normal file
138
internal/app/restore.go
Normal file
@@ -0,0 +1,138 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/logging"
|
||||
)
|
||||
|
||||
var newObjectStoreFromConfigFn = storage.NewObjectStoreFromConfig
|
||||
var discoverRemoteCurrentStateFn = discoverRemoteCurrentState
|
||||
var buildRestorePlanFn = buildRestorePlan
|
||||
var executeRestorePlanFn = executeRestorePlan
|
||||
|
||||
// Restore validates restore CLI/config inputs and storage preflight for future restore phases.
|
||||
func Restore(ctx context.Context, args []string, out io.Writer) error {
|
||||
positionalSessionID, args := pullLeadingSessionID(args)
|
||||
fs := flag.NewFlagSet("restore", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
|
||||
var flags commonConfigFlags
|
||||
var dryRun bool
|
||||
var force bool
|
||||
var includeAudio bool
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
fs.BoolVar(&dryRun, "dry-run", false, "plan restore actions without writing local files")
|
||||
fs.BoolVar(&force, "force", false, "overwrite local conflicts with remote state")
|
||||
fs.BoolVar(&includeAudio, "include-audio", false, "include remote session-level audio objects")
|
||||
fs.Usage = func() {
|
||||
_, _ = fmt.Fprintln(out, "Usage: narratio session restore <session_id> [--config <path>] [--campaign <id>] [--campaign-file <path>] [--session <path>] [--previous-session-id <value>] [--dry-run] [--force] [--include-audio]")
|
||||
_, _ = fmt.Fprintln(out)
|
||||
_, _ = fmt.Fprintln(out, "Flags:")
|
||||
fs.PrintDefaults()
|
||||
}
|
||||
|
||||
if err := fs.Parse(args); err != nil {
|
||||
if errors.Is(err, flag.ErrHelp) {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("restore: invalid flags: %w", err)
|
||||
}
|
||||
if err := resolveParsedSessionID("restore", positionalSessionID, fs, &flags.sessionID); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(flags.sessionID) == "" {
|
||||
return fmt.Errorf("restore: session_id is required")
|
||||
}
|
||||
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: %w", err)
|
||||
}
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
return fmt.Errorf("restore: %w", err)
|
||||
}
|
||||
objectStore, err := newCommandObjectStore(ctx, cfg, logging.NewLogger(os.Stderr, slog.LevelInfo))
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: %w", err)
|
||||
}
|
||||
current, err := discoverRemoteCurrentStateFn(ctx, cfg, objectStore)
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: %w", err)
|
||||
}
|
||||
plan, err := buildRestorePlanFn(ctx, cfg, current, objectStore, RestorePlanOptions{
|
||||
IncludeAudio: includeAudio,
|
||||
Force: force,
|
||||
DryRun: dryRun,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: %w", err)
|
||||
}
|
||||
report, err := newRestoreReport(current, plan, RestorePlanOptions{
|
||||
IncludeAudio: includeAudio,
|
||||
Force: force,
|
||||
DryRun: dryRun,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: %w", err)
|
||||
}
|
||||
if dryRun {
|
||||
if err := writeRestoreDryRunSummary(out, report); err != nil {
|
||||
return fmt.Errorf("restore: write plan output: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
artifactStore := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
|
||||
if _, err := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID); err != nil {
|
||||
return fmt.Errorf("restore: prepare workdir: %w", err)
|
||||
}
|
||||
lock, err := artifactStore.AcquireSessionLockFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: acquire session lock: %w", err)
|
||||
}
|
||||
defer func() {
|
||||
_ = artifactStore.ReleaseSessionLock(lock)
|
||||
}()
|
||||
|
||||
if plan.ConflictCount > 0 && !force {
|
||||
report.setFailed(fmt.Errorf("conflict: %d conflicting path(s)", plan.ConflictCount))
|
||||
if _, reportErr := persistRestoreReport(artifactStore, cfg, report); reportErr != nil {
|
||||
return fmt.Errorf("restore: report failure: %w", reportErr)
|
||||
}
|
||||
return fmt.Errorf(
|
||||
"restore conflict: %d conflicting path(s); rerun with --force to overwrite (download=%d skip_same=%d conflicts=%d)",
|
||||
plan.ConflictCount,
|
||||
plan.DownloadCount,
|
||||
plan.SkipSameCount,
|
||||
plan.ConflictCount,
|
||||
)
|
||||
}
|
||||
|
||||
result, err := executeRestorePlanFn(ctx, cfg, current, plan, report, objectStore)
|
||||
if err != nil {
|
||||
report.setFailed(err)
|
||||
if _, reportErr := persistRestoreReport(artifactStore, cfg, report); reportErr != nil {
|
||||
return fmt.Errorf("restore: execute plan failed (%v) and report write failed (%v)", err, reportErr)
|
||||
}
|
||||
return fmt.Errorf("restore: execute plan: %w", err)
|
||||
}
|
||||
report.Execution.Downloaded = result.DownloadedCount
|
||||
report.setSucceeded()
|
||||
if _, err := persistRestoreReport(artifactStore, cfg, report); err != nil {
|
||||
return fmt.Errorf("restore: write report: %w", err)
|
||||
}
|
||||
if err := writeRestoreSuccessSummary(out, report); err != nil {
|
||||
return fmt.Errorf("restore: write summary: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
64
internal/app/restore_discovery.go
Normal file
64
internal/app/restore_discovery.go
Normal file
@@ -0,0 +1,64 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
// RemoteCurrentState captures discovered committed remote published current state for one session.
|
||||
type RemoteCurrentState struct {
|
||||
Bucket string
|
||||
SessionPrefix string
|
||||
CurrentRunIDKey string
|
||||
CurrentManifestKey string
|
||||
RunID string
|
||||
SessionID string
|
||||
Campaign string
|
||||
Manifest *manifest.Manifest
|
||||
}
|
||||
|
||||
func discoverRemoteCurrentState(ctx context.Context, cfg *config.Config, store storage.ObjectStore) (*RemoteCurrentState, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
||||
return nil, fmt.Errorf("resolved config with pipeline/session is required")
|
||||
}
|
||||
if store == nil {
|
||||
return nil, fmt.Errorf("remote object store is required")
|
||||
}
|
||||
|
||||
bucket := artifacts.ResolvePublishBucket(cfg, nil)
|
||||
if strings.TrimSpace(bucket) == "" {
|
||||
return nil, fmt.Errorf("publish bucket is required")
|
||||
}
|
||||
sessionPrefix, err := artifacts.ResolvePublishSessionPrefix(cfg, nil)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve publish session prefix: %w", err)
|
||||
}
|
||||
currentManifestKey, currentRunIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||
|
||||
requestedSession := strings.TrimSpace(cfg.Session.SessionID)
|
||||
requestedCampaign := strings.TrimSpace(cfg.Session.Campaign)
|
||||
current, err := artifacts.LoadCurrentState(ctx, store, sessionPrefix, artifacts.CurrentStateValidation{
|
||||
ExpectedSessionID: requestedSession,
|
||||
ExpectedCampaign: requestedCampaign,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("remote %w", err)
|
||||
}
|
||||
|
||||
return &RemoteCurrentState{
|
||||
Bucket: bucket,
|
||||
SessionPrefix: sessionPrefix,
|
||||
CurrentRunIDKey: currentRunIDKey,
|
||||
CurrentManifestKey: currentManifestKey,
|
||||
RunID: current.RunID,
|
||||
SessionID: strings.TrimSpace(current.Manifest.SessionID),
|
||||
Campaign: strings.TrimSpace(current.Manifest.Campaign),
|
||||
Manifest: current.Manifest,
|
||||
}, nil
|
||||
}
|
||||
239
internal/app/restore_discovery_test.go
Normal file
239
internal/app/restore_discovery_test.go
Normal file
@@ -0,0 +1,239 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func TestDiscoverRemoteCurrentStateSuccess(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
sessionPrefix, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
|
||||
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign)})
|
||||
|
||||
state, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err != nil {
|
||||
t.Fatalf("discoverRemoteCurrentState() error = %v", err)
|
||||
}
|
||||
if state.RunID != "20260519T010203Z-a1b2c3d4" {
|
||||
t.Fatalf("run id = %q, want 20260519T010203Z-a1b2c3d4", state.RunID)
|
||||
}
|
||||
if state.SessionPrefix != sessionPrefix {
|
||||
t.Fatalf("session prefix = %q, want %q", state.SessionPrefix, sessionPrefix)
|
||||
}
|
||||
if state.CurrentRunIDKey != runIDKey {
|
||||
t.Fatalf("current run id key = %q, want %q", state.CurrentRunIDKey, runIDKey)
|
||||
}
|
||||
if state.CurrentManifestKey != manifestKey {
|
||||
t.Fatalf("current manifest key = %q, want %q", state.CurrentManifestKey, manifestKey)
|
||||
}
|
||||
if state.Manifest == nil {
|
||||
t.Fatal("manifest is nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateMissingRunPointerFails(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err == nil || !strings.Contains(err.Error(), "remote current run pointer missing") {
|
||||
t.Fatalf("error = %v, want missing run pointer failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateEmptyRunPointerFails(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
_, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
|
||||
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte(" \n\t")})
|
||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign)})
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err == nil || !strings.Contains(err.Error(), "is empty") {
|
||||
t.Fatalf("error = %v, want empty run pointer failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateMissingManifestFails(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
_, _, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
|
||||
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err == nil || !strings.Contains(err.Error(), "remote current manifest missing") {
|
||||
t.Fatalf("error = %v, want missing manifest failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateInvalidManifestFails(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
_, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
|
||||
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: []byte("{invalid json")})
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err == nil || !strings.Contains(err.Error(), "remote current manifest decode failed") {
|
||||
t.Fatalf("error = %v, want manifest decode failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateSessionMismatchFails(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
_, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
|
||||
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "wrong-session", cfg.Session.Campaign)})
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err == nil || !strings.Contains(err.Error(), "does not match expected session_id") {
|
||||
t.Fatalf("error = %v, want session mismatch failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateCampaignMismatchFails(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
_, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
|
||||
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, "wrong-campaign")})
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err == nil || !strings.Contains(err.Error(), "does not match expected campaign") {
|
||||
t.Fatalf("error = %v, want campaign mismatch failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateEmptyCampaignFails(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
_, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
|
||||
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, "")})
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err == nil || !strings.Contains(err.Error(), "campaign is required") {
|
||||
t.Fatalf("error = %v, want empty campaign failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateUsesCurrentKeysUnderSessionPrefix(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
sessionPrefix, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
base := &storage.FakeBackend{}
|
||||
store := &captureObjectStore{delegate: base}
|
||||
|
||||
base.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||
base.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign)})
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err != nil {
|
||||
t.Fatalf("discoverRemoteCurrentState() error = %v", err)
|
||||
}
|
||||
|
||||
expectedRunKey := fmt.Sprintf("%scurrent/run_id.txt", sessionPrefix)
|
||||
expectedManifestKey := fmt.Sprintf("%scurrent/manifest.json", sessionPrefix)
|
||||
if !containsString(store.existsKeys, expectedRunKey) {
|
||||
t.Fatalf("exists keys = %#v, want run pointer key %q", store.existsKeys, expectedRunKey)
|
||||
}
|
||||
if !containsString(store.existsKeys, expectedManifestKey) {
|
||||
t.Fatalf("exists keys = %#v, want manifest key %q", store.existsKeys, expectedManifestKey)
|
||||
}
|
||||
if !containsString(store.downloadKeys, expectedRunKey) {
|
||||
t.Fatalf("download keys = %#v, want run pointer key %q", store.downloadKeys, expectedRunKey)
|
||||
}
|
||||
if !containsString(store.downloadKeys, expectedManifestKey) {
|
||||
t.Fatalf("download keys = %#v, want manifest key %q", store.downloadKeys, expectedManifestKey)
|
||||
}
|
||||
}
|
||||
|
||||
type captureObjectStore struct {
|
||||
delegate storage.ObjectStore
|
||||
existsKeys []string
|
||||
downloadKeys []string
|
||||
}
|
||||
|
||||
func (s *captureObjectStore) List(ctx context.Context, prefix string) ([]storage.ObjectInfo, error) {
|
||||
return s.delegate.List(ctx, prefix)
|
||||
}
|
||||
|
||||
func (s *captureObjectStore) Download(ctx context.Context, key, localPath string) error {
|
||||
s.downloadKeys = append(s.downloadKeys, key)
|
||||
return s.delegate.Download(ctx, key, localPath)
|
||||
}
|
||||
|
||||
func (s *captureObjectStore) Upload(ctx context.Context, localPath, key string, opts storage.UploadOptions) (storage.ObjectInfo, error) {
|
||||
return s.delegate.Upload(ctx, localPath, key, opts)
|
||||
}
|
||||
|
||||
func (s *captureObjectStore) Exists(ctx context.Context, key string) (bool, error) {
|
||||
s.existsKeys = append(s.existsKeys, key)
|
||||
return s.delegate.Exists(ctx, key)
|
||||
}
|
||||
|
||||
func restoreDiscoveryConfig() *config.Config {
|
||||
return &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Storage: config.StorageConfig{
|
||||
S3: &config.StorageS3Config{
|
||||
Bucket: "my-dnd-archive",
|
||||
RootPrefix: "dnd",
|
||||
},
|
||||
},
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func restoreDiscoveryKeys(cfg *config.Config) (sessionPrefix, manifestKey, runIDKey string) {
|
||||
sessionPrefix = artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
manifestKey, runIDKey = artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||
return sessionPrefix, manifestKey, runIDKey
|
||||
}
|
||||
|
||||
func restoreManifestJSON(t *testing.T, sessionID, campaign string) []byte {
|
||||
t.Helper()
|
||||
now := time.Date(2026, 5, 19, 23, 0, 0, 0, time.UTC).Format(time.RFC3339Nano)
|
||||
payload := map[string]any{
|
||||
"session_id": sessionID,
|
||||
"campaign": campaign,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
"stages": map[string]any{},
|
||||
}
|
||||
data, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal manifest payload: %v", err)
|
||||
}
|
||||
return append(data, '\n')
|
||||
}
|
||||
|
||||
func containsString(values []string, target string) bool {
|
||||
for _, value := range values {
|
||||
if value == target {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
215
internal/app/restore_execute.go
Normal file
215
internal/app/restore_execute.go
Normal file
@@ -0,0 +1,215 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/audio"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/fileops"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
// RestoreExecutionResult captures concrete file-install results for one restore execution.
|
||||
type RestoreExecutionResult struct {
|
||||
DownloadedCount int
|
||||
}
|
||||
|
||||
func executeRestorePlan(
|
||||
ctx context.Context,
|
||||
cfg *config.Config,
|
||||
current *RemoteCurrentState,
|
||||
plan *RestorePlan,
|
||||
report *RestoreReport,
|
||||
store storage.ObjectStore,
|
||||
) (*RestoreExecutionResult, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
||||
return nil, fmt.Errorf("resolved config with pipeline/session is required")
|
||||
}
|
||||
if current == nil {
|
||||
return nil, fmt.Errorf("remote current state is required")
|
||||
}
|
||||
if plan == nil {
|
||||
return nil, fmt.Errorf("restore plan is required")
|
||||
}
|
||||
if store == nil {
|
||||
return nil, fmt.Errorf("remote object store is required")
|
||||
}
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
manifestActions := make([]RestoreAction, 0, 1)
|
||||
actions := make([]RestoreAction, 0, len(plan.Actions))
|
||||
for _, action := range plan.Actions {
|
||||
if action.Kind != RestoreActionDownload {
|
||||
continue
|
||||
}
|
||||
if action.LocalRelativePath == config.PathManifestFile {
|
||||
manifestActions = append(manifestActions, action)
|
||||
continue
|
||||
}
|
||||
actions = append(actions, action)
|
||||
}
|
||||
if len(manifestActions) > 1 {
|
||||
return nil, fmt.Errorf("restore plan includes multiple manifest download actions")
|
||||
}
|
||||
if len(manifestActions) == 1 {
|
||||
actions = append(actions, manifestActions[0])
|
||||
}
|
||||
|
||||
result := &RestoreExecutionResult{}
|
||||
for _, action := range actions {
|
||||
if err := executeRestoreDownloadAction(ctx, cfg, sessionRoot, current, action, store); err != nil {
|
||||
if report != nil {
|
||||
report.markFailed(action, err)
|
||||
}
|
||||
return nil, fmt.Errorf("install %q from %q: %w", action.LocalRelativePath, action.RemoteKey, err)
|
||||
}
|
||||
if report != nil {
|
||||
report.markDownloaded(action)
|
||||
}
|
||||
result.DownloadedCount++
|
||||
}
|
||||
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func executeRestoreDownloadAction(
|
||||
ctx context.Context,
|
||||
cfg *config.Config,
|
||||
sessionRoot string,
|
||||
current *RemoteCurrentState,
|
||||
action RestoreAction,
|
||||
store storage.ObjectStore,
|
||||
) error {
|
||||
safeLocalPath, err := joinWithinSessionRoot(sessionRoot, action.LocalRelativePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("resolve safe local path: %w", err)
|
||||
}
|
||||
if strings.TrimSpace(action.LocalPath) != "" && filepath.Clean(action.LocalPath) != safeLocalPath {
|
||||
return fmt.Errorf("restore plan local path mismatch for %q", action.LocalRelativePath)
|
||||
}
|
||||
|
||||
if restoreActionIsAudio(action) {
|
||||
return executeRestoreAudioAction(ctx, cfg, safeLocalPath, action, store)
|
||||
}
|
||||
|
||||
tmpPath, err := downloadObjectToSiblingTemp(ctx, store, action.RemoteKey, safeLocalPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("download to temp file: %w", err)
|
||||
}
|
||||
removeTmp := true
|
||||
defer func() {
|
||||
if removeTmp {
|
||||
_ = os.Remove(tmpPath)
|
||||
}
|
||||
}()
|
||||
|
||||
if action.LocalRelativePath == config.PathManifestFile {
|
||||
if err := validateRestoredManifest(ctx, cfg, current, tmpPath); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if err := fileops.InstallDownloadedTempFile(tmpPath, safeLocalPath, 0o644); err != nil {
|
||||
return fmt.Errorf("install file atomically: %w", err)
|
||||
}
|
||||
removeTmp = false
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func executeRestoreAudioAction(
|
||||
ctx context.Context,
|
||||
cfg *config.Config,
|
||||
safeLocalPath string,
|
||||
action RestoreAction,
|
||||
store storage.ObjectStore,
|
||||
) error {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Storage.S3 == nil || cfg.Session == nil {
|
||||
return fmt.Errorf("resolved s3 config and session are required")
|
||||
}
|
||||
spoolDir := artifacts.SessionSpoolRestoreAudioDir(cfg.Pipeline.Spool.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
spoolPath := filepath.Join(spoolDir, filepath.Base(safeLocalPath))
|
||||
cacheEnabled := cfg.Pipeline.Cache.S3Audio == nil || *cfg.Pipeline.Cache.S3Audio
|
||||
_, err := audio.MaterializeS3Audio(ctx, audio.S3MaterializeRequest{
|
||||
Store: store,
|
||||
Object: storage.ObjectInfo{
|
||||
Key: action.RemoteKey,
|
||||
Size: action.Size,
|
||||
ETag: action.ETag,
|
||||
},
|
||||
Bucket: strings.TrimSpace(cfg.Pipeline.Storage.S3.Bucket),
|
||||
CacheRoot: strings.TrimSpace(cfg.Pipeline.Cache.Root),
|
||||
CacheEnabled: cacheEnabled,
|
||||
SpoolPath: spoolPath,
|
||||
DestPath: safeLocalPath,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("materialize audio: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func downloadObjectToSiblingTemp(ctx context.Context, store storage.ObjectStore, remoteKey, destPath string) (string, error) {
|
||||
if strings.TrimSpace(destPath) == "" {
|
||||
return "", fmt.Errorf("destination path is required")
|
||||
}
|
||||
dir := filepath.Dir(destPath)
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return "", fmt.Errorf("create destination directory: %w", err)
|
||||
}
|
||||
base := filepath.Base(destPath)
|
||||
tmp, err := os.CreateTemp(dir, "."+base+".restore-*.tmp")
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("create temp file: %w", err)
|
||||
}
|
||||
tmpPath := tmp.Name()
|
||||
if err := tmp.Close(); err != nil {
|
||||
_ = os.Remove(tmpPath)
|
||||
return "", fmt.Errorf("close temp file: %w", err)
|
||||
}
|
||||
|
||||
if err := store.Download(ctx, remoteKey, tmpPath); err != nil {
|
||||
_ = os.Remove(tmpPath)
|
||||
return "", err
|
||||
}
|
||||
|
||||
return tmpPath, nil
|
||||
}
|
||||
|
||||
func validateRestoredManifest(ctx context.Context, cfg *config.Config, current *RemoteCurrentState, path string) error {
|
||||
manifestStore := &manifest.LocalStore{}
|
||||
m, err := manifestStore.Load(ctx, path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("validate manifest decode: %w", err)
|
||||
}
|
||||
|
||||
requestedSession := strings.TrimSpace(cfg.Session.SessionID)
|
||||
requestedCampaign := strings.TrimSpace(cfg.Session.Campaign)
|
||||
manifestSession := strings.TrimSpace(m.SessionID)
|
||||
manifestCampaign := strings.TrimSpace(m.Campaign)
|
||||
if manifestSession != requestedSession {
|
||||
return fmt.Errorf("manifest session_id %q does not match requested session_id %q", manifestSession, requestedSession)
|
||||
}
|
||||
if manifestCampaign == "" {
|
||||
return fmt.Errorf("manifest campaign is required")
|
||||
}
|
||||
if manifestCampaign != requestedCampaign {
|
||||
return fmt.Errorf("manifest campaign %q does not match requested campaign %q", manifestCampaign, requestedCampaign)
|
||||
}
|
||||
if current != nil {
|
||||
if expected := strings.TrimSpace(current.SessionID); expected != "" && manifestSession != expected {
|
||||
return fmt.Errorf("manifest session_id %q does not match discovered session_id %q", manifestSession, expected)
|
||||
}
|
||||
if expected := strings.TrimSpace(current.Campaign); expected != "" && manifestCampaign != expected {
|
||||
return fmt.Errorf("manifest campaign %q does not match discovered campaign %q", manifestCampaign, expected)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
539
internal/app/restore_execution_test.go
Normal file
539
internal/app/restore_execution_test.go
Normal file
@@ -0,0 +1,539 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
func TestExecuteRestoreNonDryRunRestoresDurableFiles(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, manifestKey, runIDKey := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(fake, sessionPrefix+"transcripts/full.json", []byte(`{"segments":[1,2,3]}`))
|
||||
seedRestoreObject(fake, sessionPrefix+"artifacts/session_recap.md", []byte("# recap\n"))
|
||||
seedRestoreObject(fake, sessionPrefix+"audio/alice.flac", []byte("remote-audio"))
|
||||
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
|
||||
seedRestoreObject(fake, manifestKey, restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "restore", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if stderr.Len() != 0 {
|
||||
t.Fatalf("stderr = %q, want empty", stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "Restored session state for sample-campaign/2026-05-03") {
|
||||
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
||||
}
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "full.json"), `{"segments":[1,2,3]}`)
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# recap\n")
|
||||
reportPath := filepath.Join(sessionRoot, "reports", "restore-latest.json")
|
||||
report := mustReadRestoreReport(t, reportPath)
|
||||
if report.Status != "succeeded" {
|
||||
t.Fatalf("report status = %q, want succeeded", report.Status)
|
||||
}
|
||||
if report.Execution.Downloaded != 3 {
|
||||
t.Fatalf("report execution.downloaded = %d, want 3", report.Execution.Downloaded)
|
||||
}
|
||||
if len(report.Actions) == 0 {
|
||||
t.Fatal("report actions is empty")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(sessionRoot, "audio", "alice.flac")); !os.IsNotExist(err) {
|
||||
t.Fatalf("audio should not be restored by default; stat err=%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreIncludeAudioRestoresAudio(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(fake, sessionPrefix+"audio/alice.flac", []byte("remote-audio"))
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "restore", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--include-audio"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "audio", "alice.flac"), "remote-audio")
|
||||
report := mustReadRestoreReport(t, filepath.Join(sessionRoot, "reports", "restore-latest.json"))
|
||||
if !report.IncludeAudio {
|
||||
t.Fatalf("report include_audio = %v, want true", report.IncludeAudio)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreIncludeAudioUsesCacheAfterWorkspaceDeletion(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
audioKey := sessionPrefix + "audio/alice.flac"
|
||||
seedRestoreObject(fake, audioKey, []byte("remote-audio"))
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "restore", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--include-audio"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("first restore exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if got := fakeDownloadCount(fake, audioKey); got != 1 {
|
||||
t.Fatalf("audio downloads after first restore = %d, want 1", got)
|
||||
}
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "audio", "alice.flac"), "remote-audio")
|
||||
|
||||
if err := os.RemoveAll(sessionRoot); err != nil {
|
||||
t.Fatalf("remove session root: %v", err)
|
||||
}
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
code = Execute([]string{"session", "restore", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--include-audio"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("second restore exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if got := fakeDownloadCount(fake, audioKey); got != 1 {
|
||||
t.Fatalf("audio downloads after cached restore = %d, want still 1", got)
|
||||
}
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "audio", "alice.flac"), "remote-audio")
|
||||
}
|
||||
|
||||
func TestExecuteRestoreRestoresPreviousCacheWhenPresent(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
appendRestoreWorkflowPreviousInputConfig(t, pipelinePath, sessionPath)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, _, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestorePreviousCurrent(t, fake, cfg, "# previous recap\n")
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "restore", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
previousManifestBytes, err := os.ReadFile(filepath.Join(sessionRoot, "previous", "manifest.json"))
|
||||
if err != nil {
|
||||
t.Fatalf("read restored previous manifest: %v", err)
|
||||
}
|
||||
if !strings.Contains(string(previousManifestBytes), `"session_id":"2026-04-26"`) {
|
||||
t.Fatalf("restored previous manifest = %q, want previous session id", string(previousManifestBytes))
|
||||
}
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md"), "# previous recap\n")
|
||||
report := mustReadRestoreReport(t, filepath.Join(sessionRoot, "reports", "restore-latest.json"))
|
||||
if report.Execution.Downloaded != 3 {
|
||||
t.Fatalf("report execution.downloaded = %d, want 3", report.Execution.Downloaded)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreDryRunReportsPreviousCacheWithoutWriting(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
appendRestoreWorkflowPreviousInputConfig(t, pipelinePath, sessionPath)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, _, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestorePreviousCurrent(t, fake, cfg, "# previous recap\n")
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "restore", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--dry-run"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "previous/artifacts/session_recap.md") {
|
||||
t.Fatalf("stdout = %q, want planned previous-cache artifact", stdout.String())
|
||||
}
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if _, err := os.Stat(filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md")); !os.IsNotExist(err) {
|
||||
t.Fatalf("previous artifact should not be written during dry-run; stat err=%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreConflictWithoutForceDoesNotOverwrite(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(fake, sessionPrefix+"transcripts/full.json", []byte("remote-transcript"))
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustWriteTestFile(t, filepath.Join(sessionRoot, "transcripts", "full.json"), "local-transcript")
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "restore", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "conflicting path") {
|
||||
t.Fatalf("stderr = %q, want conflict failure", stderr.String())
|
||||
}
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "full.json"), "local-transcript")
|
||||
report := mustReadRestoreReport(t, filepath.Join(sessionRoot, "reports", "restore-latest.json"))
|
||||
if report.Status != "failed" {
|
||||
t.Fatalf("report status = %q, want failed", report.Status)
|
||||
}
|
||||
if report.Plan.Conflicts != 1 {
|
||||
t.Fatalf("report plan.conflicts = %d, want 1", report.Plan.Conflicts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreForceOverwritesDifferingFile(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(fake, sessionPrefix+"transcripts/full.json", []byte("remote-transcript"))
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustWriteTestFile(t, filepath.Join(sessionRoot, "transcripts", "full.json"), "local-transcript")
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "restore", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--force"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "full.json"), "remote-transcript")
|
||||
report := mustReadRestoreReport(t, filepath.Join(sessionRoot, "reports", "restore-latest.json"))
|
||||
if !report.Force {
|
||||
t.Fatalf("report force = %v, want true", report.Force)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreForceOverwritesDifferingPreviousCacheFile(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
appendRestoreWorkflowPreviousInputConfig(t, pipelinePath, sessionPath)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, _, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestorePreviousCurrent(t, fake, cfg, "# remote previous recap\n")
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustWriteTestFile(t, filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md"), "# local previous recap\n")
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "restore", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--force"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md"), "# remote previous recap\n")
|
||||
}
|
||||
|
||||
func TestExecuteRestoreLockConflictFailsAndWritesNothing(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(fake, sessionPrefix+"transcripts/full.json", []byte("remote-transcript"))
|
||||
|
||||
store := artifacts.NewLocalStore(workspaceRoot)
|
||||
lock, err := store.AcquireSessionLockFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
t.Fatalf("AcquireSessionLockFor() error = %v", err)
|
||||
}
|
||||
defer func() { _ = store.ReleaseSessionLock(lock) }()
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "restore", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "acquire session lock") {
|
||||
t.Fatalf("stderr = %q, want lock failure", stderr.String())
|
||||
}
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if _, err := os.Stat(filepath.Join(sessionRoot, "transcripts", "full.json")); !os.IsNotExist(err) {
|
||||
t.Fatalf("transcript should not be restored when lock acquisition fails; stat err=%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreInvalidManifestDoesNotCorruptExistingManifest(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
base := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, manifestKey, _ := seedRestoreCommittedState(t, base, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(base, sessionPrefix+"transcripts/full.json", []byte("remote-transcript"))
|
||||
|
||||
toggled := &stagedManifestDownloadStore{
|
||||
delegate: base,
|
||||
manifestKey: manifestKey,
|
||||
firstManifest: restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign),
|
||||
secondManifest: []byte("{invalid json"),
|
||||
manifestReads: 0,
|
||||
}
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
existing := manifest.New(cfg.Session.SessionID, nowUTC())
|
||||
existing.Campaign = cfg.Session.Campaign
|
||||
existingPath := filepath.Join(sessionRoot, "manifest.json")
|
||||
manifestStore := &manifest.LocalStore{}
|
||||
if err := manifestStore.Save(context.Background(), existingPath, existing); err != nil {
|
||||
t.Fatalf("save existing local manifest: %v", err)
|
||||
}
|
||||
existingData, err := os.ReadFile(existingPath)
|
||||
if err != nil {
|
||||
t.Fatalf("read existing local manifest: %v", err)
|
||||
}
|
||||
|
||||
restoreWithStoreAndRealPhases(t, toggled)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "restore", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--force"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "validate manifest decode") {
|
||||
t.Fatalf("stderr = %q, want manifest validation failure", stderr.String())
|
||||
}
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "full.json"), "remote-transcript")
|
||||
report := mustReadRestoreReport(t, filepath.Join(sessionRoot, "reports", "restore-latest.json"))
|
||||
if report.Status != "failed" {
|
||||
t.Fatalf("report status = %q, want failed", report.Status)
|
||||
}
|
||||
if strings.TrimSpace(report.Error) == "" {
|
||||
t.Fatal("report error is empty, want failure context")
|
||||
}
|
||||
afterData, err := os.ReadFile(existingPath)
|
||||
if err != nil {
|
||||
t.Fatalf("read local manifest after failure: %v", err)
|
||||
}
|
||||
if string(afterData) != string(existingData) {
|
||||
t.Fatalf("local manifest changed after failed restore; before=%q after=%q", string(existingData), string(afterData))
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestorePlanPathMismatchFails(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
seedRestoreObject(store, current.SessionPrefix+"transcripts/full.json", []byte("remote-transcript"))
|
||||
|
||||
plan := &RestorePlan{Actions: []RestoreAction{{
|
||||
Kind: RestoreActionDownload,
|
||||
RemoteKey: current.SessionPrefix + "transcripts/full.json",
|
||||
LocalRelativePath: "transcripts/full.json",
|
||||
LocalPath: "/tmp/escape.txt",
|
||||
}}}
|
||||
|
||||
report, err := newRestoreReport(current, plan, RestorePlanOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("newRestoreReport() error = %v", err)
|
||||
}
|
||||
_, err = executeRestorePlan(context.Background(), cfg, current, plan, report, store)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "local path mismatch") {
|
||||
t.Fatalf("error = %v, want local path mismatch", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustReadRestoreReport(t *testing.T, path string) *RestoreReport {
|
||||
t.Helper()
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile(%q): %v", path, err)
|
||||
}
|
||||
var report RestoreReport
|
||||
if err := json.Unmarshal(data, &report); err != nil {
|
||||
t.Fatalf("Unmarshal restore report %q: %v", path, err)
|
||||
}
|
||||
return &report
|
||||
}
|
||||
|
||||
func restoreWithStoreAndRealPhases(t *testing.T, objectStore storage.ObjectStore) {
|
||||
t.Helper()
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
origDiscoverFn := discoverRemoteCurrentStateFn
|
||||
origPlanFn := buildRestorePlanFn
|
||||
origExecuteFn := executeRestorePlanFn
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
discoverRemoteCurrentStateFn = origDiscoverFn
|
||||
buildRestorePlanFn = origPlanFn
|
||||
executeRestorePlanFn = origExecuteFn
|
||||
})
|
||||
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
return objectStore, nil
|
||||
}
|
||||
discoverRemoteCurrentStateFn = discoverRemoteCurrentState
|
||||
buildRestorePlanFn = buildRestorePlan
|
||||
executeRestorePlanFn = executeRestorePlan
|
||||
}
|
||||
|
||||
func seedRestoreCommittedState(t *testing.T, fake *storage.FakeBackend, pipelinePath, campaignPath, sessionPath string) (*config.Config, string, string, string) {
|
||||
t.Helper()
|
||||
cfg, err := config.LoadWithSessionOptions(pipelinePath, campaignPath, sessionPath, config.SessionLoadOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("LoadWithSessionOptions() error = %v", err)
|
||||
}
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
t.Fatalf("Validate() error = %v", err)
|
||||
}
|
||||
|
||||
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||
|
||||
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
|
||||
seedRestoreObject(fake, manifestKey, restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
|
||||
|
||||
return cfg, sessionPrefix, manifestKey, runIDKey
|
||||
}
|
||||
|
||||
func appendRestoreWorkflowPreviousInputConfig(t *testing.T, pipelinePath, sessionPath string) {
|
||||
t.Helper()
|
||||
appendRestoreWorkflowScriptoriumConfig(t, pipelinePath, `
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
previous_recap:
|
||||
source: narratio.previous_session.artifact.session_recap
|
||||
required: true
|
||||
`)
|
||||
appendRestoreWorkflowScriptoriumConfig(t, sessionPath, `
|
||||
previous_session_id: 2026-04-26
|
||||
`)
|
||||
}
|
||||
|
||||
func seedRestorePreviousCurrent(t *testing.T, fake *storage.FakeBackend, cfg *config.Config, artifactBody string) {
|
||||
t.Helper()
|
||||
seedRestorePreviousCurrentManifestOnly(t, fake, cfg)
|
||||
previousPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
||||
seedRestoreObject(fake, previousPrefix+"artifacts/session_recap.md", []byte(artifactBody))
|
||||
}
|
||||
|
||||
func seedRestorePreviousCurrentManifestOnly(t *testing.T, fake *storage.FakeBackend, cfg *config.Config) {
|
||||
t.Helper()
|
||||
previousPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
||||
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(previousPrefix)
|
||||
previousRunID := "20260426T010203Z-a1b2c3d4"
|
||||
seedRestoreObject(fake, runIDKey, []byte(previousRunID+"\n"))
|
||||
|
||||
m := manifest.New(cfg.Session.PreviousSessionID, nowUTC())
|
||||
m.Campaign = cfg.Session.Campaign
|
||||
m.RunID = previousRunID
|
||||
data, err := json.Marshal(m)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal previous restore manifest: %v", err)
|
||||
}
|
||||
seedRestoreObject(fake, manifestKey, append(data, '\n'))
|
||||
}
|
||||
|
||||
func mustReadEquals(t *testing.T, path, want string) {
|
||||
t.Helper()
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile(%q): %v", path, err)
|
||||
}
|
||||
if string(data) != want {
|
||||
t.Fatalf("file %q = %q, want %q", path, string(data), want)
|
||||
}
|
||||
}
|
||||
|
||||
func fakeDownloadCount(fake *storage.FakeBackend, key string) int {
|
||||
count := 0
|
||||
for _, call := range fake.Downloads {
|
||||
if call.Key == key {
|
||||
count++
|
||||
}
|
||||
}
|
||||
return count
|
||||
}
|
||||
|
||||
type stagedManifestDownloadStore struct {
|
||||
delegate *storage.FakeBackend
|
||||
manifestKey string
|
||||
firstManifest []byte
|
||||
secondManifest []byte
|
||||
manifestReads int
|
||||
}
|
||||
|
||||
func (s *stagedManifestDownloadStore) List(ctx context.Context, prefix string) ([]storage.ObjectInfo, error) {
|
||||
return s.delegate.List(ctx, prefix)
|
||||
}
|
||||
|
||||
func (s *stagedManifestDownloadStore) Download(ctx context.Context, key, localPath string) error {
|
||||
if strings.TrimSpace(key) == strings.TrimSpace(s.manifestKey) {
|
||||
s.manifestReads++
|
||||
payload := s.secondManifest
|
||||
if s.manifestReads <= 1 {
|
||||
payload = s.firstManifest
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(localPath), 0o755); err != nil {
|
||||
return fmt.Errorf("download staged manifest: create parent: %w", err)
|
||||
}
|
||||
if err := os.WriteFile(localPath, payload, 0o644); err != nil {
|
||||
return fmt.Errorf("download staged manifest: write local file: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return s.delegate.Download(ctx, key, localPath)
|
||||
}
|
||||
|
||||
func (s *stagedManifestDownloadStore) Upload(ctx context.Context, localPath, key string, opts storage.UploadOptions) (storage.ObjectInfo, error) {
|
||||
return s.delegate.Upload(ctx, localPath, key, opts)
|
||||
}
|
||||
|
||||
func (s *stagedManifestDownloadStore) Exists(ctx context.Context, key string) (bool, error) {
|
||||
return s.delegate.Exists(ctx, key)
|
||||
}
|
||||
421
internal/app/restore_plan.go
Normal file
421
internal/app/restore_plan.go
Normal file
@@ -0,0 +1,421 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/pathsafe"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/previouscache"
|
||||
)
|
||||
|
||||
// RestoreActionKind identifies one restore planner action.
|
||||
type RestoreActionKind string
|
||||
|
||||
const (
|
||||
RestoreActionDownload RestoreActionKind = "download"
|
||||
RestoreActionSkipSame RestoreActionKind = "skip_same"
|
||||
RestoreActionConflict RestoreActionKind = "conflict"
|
||||
)
|
||||
|
||||
// RestoreAction is one deterministic planner action.
|
||||
type RestoreAction struct {
|
||||
Kind RestoreActionKind
|
||||
RemoteKey string
|
||||
LocalRelativePath string
|
||||
LocalPath string
|
||||
Size int64
|
||||
ETag string
|
||||
ExistsLocal bool
|
||||
SameLocal bool
|
||||
Conflict bool
|
||||
Reason string
|
||||
}
|
||||
|
||||
// RestorePlan is the deterministic output of restore planning.
|
||||
type RestorePlan struct {
|
||||
Actions []RestoreAction
|
||||
DownloadCount int
|
||||
SkipSameCount int
|
||||
ConflictCount int
|
||||
}
|
||||
|
||||
// RestorePlanOptions control restore planning scope and classification.
|
||||
type RestorePlanOptions struct {
|
||||
IncludeAudio bool
|
||||
Force bool
|
||||
DryRun bool
|
||||
}
|
||||
|
||||
func buildRestorePlan(ctx context.Context, cfg *config.Config, current *RemoteCurrentState, store storage.ObjectStore, opts RestorePlanOptions) (*RestorePlan, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
||||
return nil, fmt.Errorf("resolved config with pipeline/session is required")
|
||||
}
|
||||
if current == nil {
|
||||
return nil, fmt.Errorf("remote current state is required")
|
||||
}
|
||||
if store == nil {
|
||||
return nil, fmt.Errorf("remote object store is required")
|
||||
}
|
||||
prefix := normalizeRemoteKey(current.SessionPrefix)
|
||||
if strings.TrimSpace(prefix) == "" {
|
||||
return nil, fmt.Errorf("remote session prefix is required")
|
||||
}
|
||||
if !strings.HasSuffix(prefix, "/") {
|
||||
prefix += "/"
|
||||
}
|
||||
|
||||
sessionPaths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
objects, err := store.List(ctx, prefix)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list remote session objects under %q: %w", prefix, err)
|
||||
}
|
||||
|
||||
candidates := make(map[string]storage.ObjectInfo, len(objects)+1)
|
||||
for _, obj := range objects {
|
||||
key := normalizeRemoteKey(obj.Key)
|
||||
if key == "" {
|
||||
continue
|
||||
}
|
||||
obj.Key = key
|
||||
candidates[key] = obj
|
||||
}
|
||||
if strings.TrimSpace(current.CurrentManifestKey) != "" {
|
||||
key := normalizeRemoteKey(current.CurrentManifestKey)
|
||||
if _, ok := candidates[key]; !ok {
|
||||
candidates[key] = storage.ObjectInfo{Key: key}
|
||||
}
|
||||
}
|
||||
|
||||
actions := make([]RestoreAction, 0, len(candidates))
|
||||
for key, obj := range candidates {
|
||||
rel, include, err := restoreLocalRelativePathForKey(prefix, normalizeRemoteKey(current.CurrentManifestKey), key, opts.IncludeAudio)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("map remote key %q: %w", key, err)
|
||||
}
|
||||
if !include {
|
||||
continue
|
||||
}
|
||||
localPath, err := joinWithinSessionRoot(sessionPaths.Root, rel)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("map remote key %q: %w", key, err)
|
||||
}
|
||||
|
||||
action, err := classifyRestoreAction(ctx, store, obj, rel, localPath, opts.Force)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("classify remote key %q: %w", key, err)
|
||||
}
|
||||
actions = append(actions, action)
|
||||
}
|
||||
|
||||
previousActions, err := buildPreviousCacheRestoreActions(ctx, cfg, sessionPaths, store, opts.Force)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
actions = append(actions, previousActions...)
|
||||
|
||||
sort.Slice(actions, func(i, j int) bool {
|
||||
if actions[i].LocalRelativePath == actions[j].LocalRelativePath {
|
||||
return actions[i].RemoteKey < actions[j].RemoteKey
|
||||
}
|
||||
return actions[i].LocalRelativePath < actions[j].LocalRelativePath
|
||||
})
|
||||
|
||||
plan := &RestorePlan{Actions: actions}
|
||||
for _, action := range actions {
|
||||
switch action.Kind {
|
||||
case RestoreActionDownload:
|
||||
plan.DownloadCount++
|
||||
case RestoreActionSkipSame:
|
||||
plan.SkipSameCount++
|
||||
case RestoreActionConflict:
|
||||
plan.ConflictCount++
|
||||
}
|
||||
}
|
||||
|
||||
_ = opts.DryRun
|
||||
return plan, nil
|
||||
}
|
||||
|
||||
func normalizeRemoteKey(v string) string {
|
||||
return strings.Trim(strings.ReplaceAll(strings.TrimSpace(v), "\\", "/"), "/")
|
||||
}
|
||||
|
||||
func restoreLocalRelativePathForKey(sessionPrefix, currentManifestKey, key string, includeAudio bool) (string, bool, error) {
|
||||
if key == "" {
|
||||
return "", false, nil
|
||||
}
|
||||
if key == currentManifestKey {
|
||||
return config.PathManifestFile, true, nil
|
||||
}
|
||||
if !strings.HasPrefix(key, sessionPrefix) {
|
||||
return "", false, fmt.Errorf("key is outside resolved session prefix %q", sessionPrefix)
|
||||
}
|
||||
|
||||
rel := strings.TrimPrefix(key, sessionPrefix)
|
||||
rel = strings.TrimSpace(rel)
|
||||
if rel == "" {
|
||||
return "", false, nil
|
||||
}
|
||||
|
||||
cleanRel := path.Clean(rel)
|
||||
if cleanRel == "." || cleanRel == "" {
|
||||
return "", false, nil
|
||||
}
|
||||
if cleanRel == ".." || strings.HasPrefix(cleanRel, "../") || strings.HasPrefix(cleanRel, "/") {
|
||||
return "", false, fmt.Errorf("key relative path %q escapes session scope", rel)
|
||||
}
|
||||
|
||||
if cleanRel == config.PathManifestFile {
|
||||
return config.PathManifestFile, true, nil
|
||||
}
|
||||
if strings.HasPrefix(cleanRel, config.S3CurrentSegment+"/") {
|
||||
return "", false, nil
|
||||
}
|
||||
if strings.HasPrefix(cleanRel, config.S3RunsSegment+"/") {
|
||||
return "", false, nil
|
||||
}
|
||||
|
||||
excludedRoots := []string{
|
||||
config.PathLogsDirSegment,
|
||||
config.PathReportsDirSegment,
|
||||
config.PathConfigDirSegment,
|
||||
config.PathInputsDirSegment,
|
||||
}
|
||||
for _, root := range excludedRoots {
|
||||
if cleanRel == root || strings.HasPrefix(cleanRel, root+"/") {
|
||||
return "", false, nil
|
||||
}
|
||||
}
|
||||
|
||||
if cleanRel == config.PathTranscriptsSegment || strings.HasPrefix(cleanRel, config.PathTranscriptsSegment+"/") {
|
||||
return cleanRel, true, nil
|
||||
}
|
||||
if cleanRel == config.PathArtifactsDirSegment || strings.HasPrefix(cleanRel, config.PathArtifactsDirSegment+"/") {
|
||||
return cleanRel, true, nil
|
||||
}
|
||||
if cleanRel == config.PathPreviousDirSegment || strings.HasPrefix(cleanRel, config.PathPreviousDirSegment+"/") {
|
||||
return "", false, nil
|
||||
}
|
||||
if includeAudio && (cleanRel == config.PathAudioDirSegment || strings.HasPrefix(cleanRel, config.PathAudioDirSegment+"/")) {
|
||||
return cleanRel, true, nil
|
||||
}
|
||||
|
||||
return "", false, nil
|
||||
}
|
||||
|
||||
func joinWithinSessionRoot(sessionRoot, relative string) (string, error) {
|
||||
if strings.TrimSpace(sessionRoot) == "" {
|
||||
return "", fmt.Errorf("session root is required")
|
||||
}
|
||||
joined, err := pathsafe.JoinSlashRelativeUnderRoot(sessionRoot, filepath.ToSlash(strings.TrimSpace(relative)))
|
||||
if err != nil {
|
||||
if errors.Is(err, pathsafe.ErrRelativePathRequired) {
|
||||
return "", fmt.Errorf("relative path is required")
|
||||
}
|
||||
if errors.Is(err, pathsafe.ErrRelativePathEscape) || errors.Is(err, pathsafe.ErrRelativePathAbsolute) {
|
||||
return "", fmt.Errorf("relative path escapes session root")
|
||||
}
|
||||
return "", fmt.Errorf("join relative path under session root: %w", err)
|
||||
}
|
||||
return joined, nil
|
||||
}
|
||||
|
||||
func buildPreviousCacheRestoreActions(
|
||||
ctx context.Context,
|
||||
cfg *config.Config,
|
||||
sessionPaths artifacts.SessionPaths,
|
||||
store storage.ObjectStore,
|
||||
force bool,
|
||||
) ([]RestoreAction, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Scriptorium == nil {
|
||||
return nil, nil
|
||||
}
|
||||
requirements := artifacts.CollectPreviousArtifactRequirements(cfg.Pipeline.Scriptorium.Artifacts)
|
||||
if len(requirements) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
plan, err := previouscache.BuildPlan(ctx, cfg, sessionPaths, requirements, store)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("plan previous-session cache restore: %w", err)
|
||||
}
|
||||
actions := make([]RestoreAction, 0, len(plan.Records))
|
||||
for _, record := range plan.Records {
|
||||
action, err := classifyRestoreAction(ctx, store, storage.ObjectInfo{Key: record.RemoteKey}, record.LocalRelativePath, record.LocalPath, force)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("classify previous-session cache object %q: %w", record.RemoteKey, err)
|
||||
}
|
||||
actions = append(actions, action)
|
||||
}
|
||||
return actions, nil
|
||||
}
|
||||
|
||||
func classifyRestoreAction(
|
||||
ctx context.Context,
|
||||
store storage.ObjectStore,
|
||||
object storage.ObjectInfo,
|
||||
localRelPath string,
|
||||
localPath string,
|
||||
force bool,
|
||||
) (RestoreAction, error) {
|
||||
action := RestoreAction{
|
||||
RemoteKey: normalizeRemoteKey(object.Key),
|
||||
LocalRelativePath: localRelPath,
|
||||
LocalPath: localPath,
|
||||
Size: object.Size,
|
||||
ETag: object.ETag,
|
||||
}
|
||||
|
||||
info, err := os.Stat(localPath)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
action.Kind = RestoreActionDownload
|
||||
action.Reason = "local file missing"
|
||||
return action, nil
|
||||
}
|
||||
return RestoreAction{}, fmt.Errorf("stat local file: %w", err)
|
||||
}
|
||||
|
||||
action.ExistsLocal = true
|
||||
if info.IsDir() {
|
||||
action.Kind = RestoreActionConflict
|
||||
action.Conflict = true
|
||||
action.Reason = "local path is a directory"
|
||||
return action, nil
|
||||
}
|
||||
|
||||
if restoreRelativePathIsAudio(localRelPath) {
|
||||
if object.Size > 0 {
|
||||
if info.Size() == object.Size {
|
||||
action.Kind = RestoreActionSkipSame
|
||||
action.SameLocal = true
|
||||
action.Reason = "local audio size matches remote content"
|
||||
return action, nil
|
||||
}
|
||||
if force {
|
||||
action.Kind = RestoreActionDownload
|
||||
action.Reason = "local audio differs (size mismatch); overwrite with --force"
|
||||
return action, nil
|
||||
}
|
||||
action.Kind = RestoreActionConflict
|
||||
action.Conflict = true
|
||||
action.Reason = "local audio differs (size mismatch)"
|
||||
return action, nil
|
||||
}
|
||||
if force {
|
||||
action.Kind = RestoreActionDownload
|
||||
action.Reason = "local audio exists; remote size unavailable; overwrite with --force"
|
||||
return action, nil
|
||||
}
|
||||
action.Kind = RestoreActionConflict
|
||||
action.Conflict = true
|
||||
action.Reason = "local audio exists; remote size unavailable"
|
||||
return action, nil
|
||||
}
|
||||
|
||||
if object.Size > 0 && info.Size() != object.Size {
|
||||
if force {
|
||||
action.Kind = RestoreActionDownload
|
||||
action.Reason = "local file differs (size mismatch); overwrite with --force"
|
||||
return action, nil
|
||||
}
|
||||
action.Kind = RestoreActionConflict
|
||||
action.Conflict = true
|
||||
action.Reason = "local file differs (size mismatch)"
|
||||
return action, nil
|
||||
}
|
||||
|
||||
localDigest, err := artifacts.SHA256File(localPath)
|
||||
if err != nil {
|
||||
return RestoreAction{}, fmt.Errorf("checksum local file: %w", err)
|
||||
}
|
||||
remotePath, err := storage.DownloadObjectToTemp(ctx, store, action.RemoteKey, "narratio-restore-plan-remote-*.tmp")
|
||||
if err != nil {
|
||||
return RestoreAction{}, fmt.Errorf("download remote object: %w", err)
|
||||
}
|
||||
defer func() { _ = os.Remove(remotePath) }()
|
||||
|
||||
remoteDigest, err := artifacts.SHA256File(remotePath)
|
||||
if err != nil {
|
||||
return RestoreAction{}, fmt.Errorf("checksum remote object: %w", err)
|
||||
}
|
||||
|
||||
if remoteDigest == localDigest {
|
||||
action.Kind = RestoreActionSkipSame
|
||||
action.SameLocal = true
|
||||
action.Reason = "local file matches remote content"
|
||||
return action, nil
|
||||
}
|
||||
|
||||
if force {
|
||||
action.Kind = RestoreActionDownload
|
||||
action.Reason = "local file differs; overwrite with --force"
|
||||
return action, nil
|
||||
}
|
||||
|
||||
action.Kind = RestoreActionConflict
|
||||
action.Conflict = true
|
||||
action.Reason = "local file differs"
|
||||
return action, nil
|
||||
}
|
||||
|
||||
func restoreActionIsAudio(action RestoreAction) bool {
|
||||
return restoreRelativePathIsAudio(action.LocalRelativePath)
|
||||
}
|
||||
|
||||
func restoreRelativePathIsAudio(rel string) bool {
|
||||
cleanRel := path.Clean(strings.TrimSpace(rel))
|
||||
return cleanRel == config.PathAudioDirSegment || strings.HasPrefix(cleanRel, config.PathAudioDirSegment+"/")
|
||||
}
|
||||
|
||||
func writeRestorePlan(out io.Writer, current *RemoteCurrentState, plan *RestorePlan, opts RestorePlanOptions) error {
|
||||
if out == nil {
|
||||
return fmt.Errorf("output writer is required")
|
||||
}
|
||||
if current == nil {
|
||||
return fmt.Errorf("remote current state is required")
|
||||
}
|
||||
if plan == nil {
|
||||
return fmt.Errorf("restore plan is required")
|
||||
}
|
||||
|
||||
if _, err := fmt.Fprintf(
|
||||
out,
|
||||
"restore plan: session %s/%s run=%s actions=%d download=%d skip_same=%d conflict=%d dry_run=%t force=%t include_audio=%t\n",
|
||||
current.Campaign,
|
||||
current.SessionID,
|
||||
current.RunID,
|
||||
len(plan.Actions),
|
||||
plan.DownloadCount,
|
||||
plan.SkipSameCount,
|
||||
plan.ConflictCount,
|
||||
opts.DryRun,
|
||||
opts.Force,
|
||||
opts.IncludeAudio,
|
||||
); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, action := range plan.Actions {
|
||||
if _, err := fmt.Fprintf(out, "%s %s <- %s", action.Kind, action.LocalRelativePath, action.RemoteKey); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(action.Reason) != "" {
|
||||
if _, err := fmt.Fprintf(out, " (%s)", action.Reason); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if _, err := fmt.Fprintln(out); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
354
internal/app/restore_plan_test.go
Normal file
354
internal/app/restore_plan_test.go
Normal file
@@ -0,0 +1,354 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func TestRestorePlanDefaultScope(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"transcripts/full.json", []byte(`{"segments":[1]}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("# recap\n"))
|
||||
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
|
||||
seedRestoreObject(store, current.SessionPrefix+"runs/20260519T010203Z-a1b2/manifest.json", []byte("{}"))
|
||||
seedRestoreObject(store, current.SessionPrefix+"logs/publish.log", []byte("log"))
|
||||
|
||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan() error = %v", err)
|
||||
}
|
||||
|
||||
got := actionRelPaths(plan.Actions)
|
||||
want := []string{"artifacts/session_recap.md", "manifest.json", "transcripts/full.json"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("action local paths = %#v, want %#v", got, want)
|
||||
}
|
||||
if plan.DownloadCount != 3 || plan.SkipSameCount != 0 || plan.ConflictCount != 0 {
|
||||
t.Fatalf("counts = download=%d skip_same=%d conflict=%d, want 3/0/0", plan.DownloadCount, plan.SkipSameCount, plan.ConflictCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanIncludeAudio(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
|
||||
|
||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{IncludeAudio: true})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan() error = %v", err)
|
||||
}
|
||||
got := actionRelPaths(plan.Actions)
|
||||
want := []string{"audio/alice.flac", "manifest.json"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("action local paths = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanExistingAudioUsesSizeWithoutRemoteChecksumDownload(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustWriteTestFile(t, filepath.Join(sessionRoot, "audio", "alice.flac"), "local")
|
||||
|
||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{IncludeAudio: true})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan() error = %v", err)
|
||||
}
|
||||
if len(store.Downloads) != 0 {
|
||||
t.Fatalf("downloads = %d, want no remote checksum download for audio", len(store.Downloads))
|
||||
}
|
||||
actionByRel := map[string]RestoreAction{}
|
||||
for _, action := range plan.Actions {
|
||||
actionByRel[action.LocalRelativePath] = action
|
||||
}
|
||||
audioAction := actionByRel["audio/alice.flac"]
|
||||
if audioAction.Kind != RestoreActionSkipSame {
|
||||
t.Fatalf("audio action kind = %q, want %q", audioAction.Kind, RestoreActionSkipSame)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanIncludesPreviousCacheByDefault(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
configureRestorePlanPreviousRequirement(cfg, true)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestorePreviousCurrent(t, store, cfg, "# previous recap\n")
|
||||
|
||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan() error = %v", err)
|
||||
}
|
||||
|
||||
got := actionRelPaths(plan.Actions)
|
||||
want := []string{"manifest.json", "previous/artifacts/session_recap.md", "previous/manifest.json"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("action local paths = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanIgnoresCurrentSessionArchivedPreviousCache(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"previous/manifest.json", []byte(`{"session_id":"2026-04-26"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"previous/artifacts/session_recap.md", []byte("# previous recap\n"))
|
||||
|
||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan() error = %v", err)
|
||||
}
|
||||
|
||||
got := actionRelPaths(plan.Actions)
|
||||
want := []string{"manifest.json"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("action local paths = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanMissingOptionalPreviousCacheSkipsArtifact(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
configureRestorePlanPreviousRequirement(cfg, false)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestorePreviousCurrentManifestOnly(t, store, cfg)
|
||||
|
||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan() error = %v", err)
|
||||
}
|
||||
|
||||
got := actionRelPaths(plan.Actions)
|
||||
want := []string{"manifest.json", "previous/manifest.json"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("action local paths = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanMissingRequiredPreviousCacheFails(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
configureRestorePlanPreviousRequirement(cfg, true)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestorePreviousCurrentManifestOnly(t, store, cfg)
|
||||
|
||||
_, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||
if err == nil || !strings.Contains(err.Error(), "required previous-session artifact") {
|
||||
t.Fatalf("buildRestorePlan() error = %v, want required previous artifact failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanPreviousCacheConflictRequiresForce(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
configureRestorePlanPreviousRequirement(cfg, true)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestorePreviousCurrent(t, store, cfg, "# remote previous recap\n")
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustWriteTestFile(t, filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md"), "# local previous recap\n")
|
||||
|
||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan() error = %v", err)
|
||||
}
|
||||
if plan.ConflictCount != 1 {
|
||||
t.Fatalf("ConflictCount = %d, want 1", plan.ConflictCount)
|
||||
}
|
||||
|
||||
plan, err = buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{Force: true})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan(force) error = %v", err)
|
||||
}
|
||||
if plan.ConflictCount != 0 {
|
||||
t.Fatalf("force ConflictCount = %d, want 0", plan.ConflictCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanClassifiesSameAndConflict(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"transcripts/full.json", []byte(`{"segments":[1]}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("remote-content\n"))
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustWriteTestFile(t, filepath.Join(sessionRoot, "transcripts", "full.json"), `{"segments":[1]}`)
|
||||
mustWriteTestFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "different\n")
|
||||
|
||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan() error = %v", err)
|
||||
}
|
||||
|
||||
if plan.SkipSameCount != 1 {
|
||||
t.Fatalf("SkipSameCount = %d, want 1", plan.SkipSameCount)
|
||||
}
|
||||
if plan.ConflictCount != 1 {
|
||||
t.Fatalf("ConflictCount = %d, want 1", plan.ConflictCount)
|
||||
}
|
||||
|
||||
actionByRel := map[string]RestoreAction{}
|
||||
for _, action := range plan.Actions {
|
||||
actionByRel[action.LocalRelativePath] = action
|
||||
}
|
||||
if actionByRel["transcripts/full.json"].Kind != RestoreActionSkipSame {
|
||||
t.Fatalf("transcripts/full.json kind = %q, want %q", actionByRel["transcripts/full.json"].Kind, RestoreActionSkipSame)
|
||||
}
|
||||
if actionByRel["artifacts/session_recap.md"].Kind != RestoreActionConflict {
|
||||
t.Fatalf("artifacts/session_recap.md kind = %q, want %q", actionByRel["artifacts/session_recap.md"].Kind, RestoreActionConflict)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanForceTurnsConflictsIntoDownloads(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("remote-content\n"))
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustWriteTestFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "different\n")
|
||||
|
||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{Force: true})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan() error = %v", err)
|
||||
}
|
||||
|
||||
actionByRel := map[string]RestoreAction{}
|
||||
for _, action := range plan.Actions {
|
||||
actionByRel[action.LocalRelativePath] = action
|
||||
}
|
||||
recap := actionByRel["artifacts/session_recap.md"]
|
||||
if recap.Kind != RestoreActionDownload {
|
||||
t.Fatalf("artifacts/session_recap.md kind = %q, want %q", recap.Kind, RestoreActionDownload)
|
||||
}
|
||||
if plan.ConflictCount != 0 {
|
||||
t.Fatalf("ConflictCount = %d, want 0", plan.ConflictCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanTraversalUnsafeKeyFails(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"artifacts/../../escape.txt", []byte("bad"))
|
||||
|
||||
_, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "escapes session scope") {
|
||||
t.Fatalf("error = %v, want traversal safety failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func seedRestoreObject(store *storage.FakeBackend, key string, data []byte) {
|
||||
store.SeedObject(storage.FakeObject{Key: key, Data: data})
|
||||
}
|
||||
|
||||
func actionRelPaths(actions []RestoreAction) []string {
|
||||
out := make([]string, 0, len(actions))
|
||||
for _, action := range actions {
|
||||
out = append(out, action.LocalRelativePath)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func restorePlanConfig(t *testing.T) *config.Config {
|
||||
t.Helper()
|
||||
workspaceRoot := t.TempDir()
|
||||
return &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Workspace: config.WorkspaceConfig{Root: workspaceRoot},
|
||||
Storage: config.StorageConfig{S3: &config.StorageS3Config{
|
||||
Bucket: "test-bucket",
|
||||
RootPrefix: "dnd",
|
||||
}},
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func configureRestorePlanPreviousRequirement(cfg *config.Config, required bool) {
|
||||
cfg.Session.PreviousSessionID = "2026-04-26"
|
||||
cfg.Pipeline.Scriptorium = &config.ScriptoriumConfig{
|
||||
Artifacts: map[string]config.ScriptoriumArtifactConfig{
|
||||
"session_recap": {
|
||||
Enabled: true,
|
||||
OutputPath: "artifacts/session_recap.md",
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"previous_recap": {
|
||||
Source: "narratio.previous_session.artifact.session_recap",
|
||||
Required: required,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func restorePlanCurrentState(t *testing.T, cfg *config.Config) *RemoteCurrentState {
|
||||
t.Helper()
|
||||
sessionPrefix := artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||
return &RemoteCurrentState{
|
||||
Bucket: "test-bucket",
|
||||
SessionPrefix: sessionPrefix,
|
||||
CurrentManifestKey: manifestKey,
|
||||
CurrentRunIDKey: runIDKey,
|
||||
RunID: "20260519T010203Z-a1b2c3d4",
|
||||
SessionID: cfg.Session.SessionID,
|
||||
Campaign: cfg.Session.Campaign,
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteRestorePlan(t *testing.T) {
|
||||
current := &RemoteCurrentState{Campaign: "sample-campaign", SessionID: "2026-05-03", RunID: "r-1"}
|
||||
plan := &RestorePlan{Actions: []RestoreAction{{Kind: RestoreActionDownload, LocalRelativePath: "manifest.json", RemoteKey: "k", Reason: "local file missing"}}, DownloadCount: 1}
|
||||
var out strings.Builder
|
||||
if err := writeRestorePlan(&out, current, plan, RestorePlanOptions{DryRun: true}); err != nil {
|
||||
t.Fatalf("writeRestorePlan() error = %v", err)
|
||||
}
|
||||
text := out.String()
|
||||
if !strings.Contains(text, "restore plan: session sample-campaign/2026-05-03 run=r-1") {
|
||||
t.Fatalf("output = %q, want plan summary", text)
|
||||
}
|
||||
if !strings.Contains(text, "download manifest.json <- k") {
|
||||
t.Fatalf("output = %q, want action line", text)
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user