Compare commits
35 Commits
f050b9dd54
...
v1.1.0
| Author | SHA1 | Date | |
|---|---|---|---|
| ffc07922c7 | |||
| f3310d4d16 | |||
| 88cee96d8d | |||
| 2fece10215 | |||
| 0658f2f642 | |||
| a51228c803 | |||
| 4491fb5ccd | |||
| 30b905765c | |||
| 03eac70881 | |||
| 0f7e6b979f | |||
| c366912586 | |||
| 9fe44cd00d | |||
| 094b0d2532 | |||
| 98649f4d81 | |||
| 8a559efd5b | |||
| 72deccb4e2 | |||
| 5620fc5bcf | |||
| be57e675e0 | |||
| 3971443831 | |||
| a6b0c33e9f | |||
| 96b886e711 | |||
| 7d584ee6cd | |||
| 572a112c31 | |||
| ea87c335d6 | |||
| 7169ff04df | |||
| ef1f650bc0 | |||
| 0d02cb9fa0 | |||
| 0299b128cf | |||
| d723384888 | |||
| 54228055c8 | |||
| 23ed716450 | |||
| ab59bab044 | |||
| 71395bb076 | |||
| 79737edf79 | |||
| df2c765b7f |
16
README.md
16
README.md
@@ -1,22 +1,22 @@
|
|||||||
# narratio
|
# narratio
|
||||||
|
|
||||||
Narratio is a Go orchestration application that turns D&D session audio into polished transcripts and generated session artifacts.
|
Narratio is a stage-driven Go orchestrator for turning D&D session audio into polished transcripts and generated artifacts.
|
||||||
|
|
||||||
It coordinates transcription, merge/polish/normalize/trim processing, artifact generation, archive publishing, and resumable run state in one operator workflow.
|
It runs a deterministic workflow across `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `render`, `analyze`, and `publish`, with manifest-driven continuation and restore support.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio run 2026-04-04
|
narratio run 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
This command requires discoverable `pipeline.yml` and `session.yml` files (or explicit `--config` and `--session` flags).
|
This requires resolvable `pipeline.yml`, `campaign.yml`, and concrete `session.yml` (or explicit config flags).
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
- [Configuration](docs/config.md)
|
|
||||||
- [CLI Reference](docs/cli.md)
|
- [CLI Reference](docs/cli.md)
|
||||||
- [Operations and Recovery](docs/operations.md)
|
- [Configuration](docs/config.md)
|
||||||
|
- [Operations](docs/operations.md)
|
||||||
- [Troubleshooting](docs/troubleshooting.md)
|
- [Troubleshooting](docs/troubleshooting.md)
|
||||||
- [Development Guide](docs/development.md)
|
|
||||||
- [Architecture Principles](docs/architecture.md)
|
|
||||||
- [Internal Component Contracts](docs/internal/README.md)
|
- [Internal Component Contracts](docs/internal/README.md)
|
||||||
- [Config Examples](examples/)
|
- [Development Guide](docs/policy/development.md)
|
||||||
|
- [Architecture Principles](docs/policy/architecture.md)
|
||||||
|
- [Maintained Examples](examples/)
|
||||||
|
|||||||
362
docs/cli.md
362
docs/cli.md
@@ -1,4 +1,4 @@
|
|||||||
# CLI
|
# CLI Reference
|
||||||
|
|
||||||
## Shortest Useful Command
|
## Shortest Useful Command
|
||||||
|
|
||||||
@@ -6,91 +6,83 @@
|
|||||||
narratio run 2026-04-04
|
narratio run 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
This command uses default system discovery for `pipeline.yml`, the pipeline default campaign ID, and local `session.yml`. If local session discovery misses and S3 storage is configured, the positional session ID loads remote `session.yml` from the canonical session prefix.
|
This runs the canonical full pipeline for session `2026-04-04`.
|
||||||
|
|
||||||
Default pipeline and session discovery checks system config locations only. Pass `--config`, `--campaign-file`, and `--session` to use files from the current working directory. Pass `--campaign <id>` to select a campaign from `pipeline.campaigns.root`.
|
|
||||||
|
|
||||||
Ordinary local and remote `session.yml` files must be concrete YAML. Templates belong to `narratio session init`, which renders a configured campaign template before writing the concrete file.
|
|
||||||
|
|
||||||
## Command Overview
|
## Command Overview
|
||||||
|
|
||||||
Top-level commands:
|
Top-level commands:
|
||||||
|
|
||||||
- `run <session_id>`: execute pipeline stages and persist manifest state.
|
- `run <session_id>`: run full stage order.
|
||||||
- `run-stage <stage> <session_id>`: execute exactly one stage.
|
- `run-stage <stage> <session_id>`: run one stage.
|
||||||
- `resume <session_id>`: continue from first non-succeeded stage unless forced.
|
- `analyze <session_id>`: force-run analyze.
|
||||||
- `analyze <session_id>`: force-rerun the analyze stage.
|
- `publish <session_id>`: force-run publish.
|
||||||
- `publish <session_id>`: force-rerun the archive stage.
|
- `clean <session_id>` or `clean --all`: remove local work/spool state.
|
||||||
- `clean <session_id>|--all`: remove local workspace/spool state.
|
- `session <subcommand>`: session helper commands.
|
||||||
- `session <subcommand>`: session-scoped helper commands.
|
|
||||||
|
|
||||||
Session subcommands:
|
Session subcommands:
|
||||||
|
|
||||||
- `session init <session_id>`: create local or remote `session.yml`.
|
- `session init <session_id>`
|
||||||
- `session validate <session_id>`: run read-only preflight checks.
|
- `session plan <session_id>`
|
||||||
- `session status <session_id>`: inspect local/remote session state.
|
- `session validate <session_id>`
|
||||||
- `session plan <session_id>`: validate config, prepare workspace layout, and print stage run/skip decisions.
|
- `session status <session_id>`
|
||||||
- `session restore <session_id>`: restore durable local state from committed remote archive state.
|
- `session restore <session_id>`
|
||||||
- `session artifacts <session_id>`: list effective artifact source IDs.
|
- `session artifacts <session_id>`
|
||||||
- `session locks <session_id>`: list archive promotion locks.
|
- `session locks <session_id>`
|
||||||
- `session locks add <session_id> <source>`: add or update a remote lock.
|
- `session locks add <session_id> <source>`
|
||||||
- `session locks remove <session_id> <source>`: remove a remote lock.
|
- `session locks remove <session_id> <source>`
|
||||||
|
|
||||||
Unknown commands print usage and exit non-zero.
|
## Common Config Flags
|
||||||
|
|
||||||
For config semantics, see [docs/config.md](./config.md). For operator lifecycle and recovery, see [docs/operations.md](./operations.md).
|
|
||||||
|
|
||||||
## Common Flags
|
|
||||||
|
|
||||||
Most session-aware commands accept:
|
Most session-aware commands accept:
|
||||||
|
|
||||||
- `--config <path>`: optional explicit `pipeline.yml` path.
|
- `--config <pipeline.yml>`
|
||||||
- `--campaign <id>`: optional campaign ID selector.
|
- `--campaign <id>`
|
||||||
- `--campaign-file <path>`: optional explicit `campaign.yml` path.
|
- `--campaign-file <campaign.yml>`
|
||||||
- `--session <path>`: optional explicit concrete `session.yml` path.
|
- `--session <session.yml>`
|
||||||
- `--previous-session-id <value>`: expected previous session identifier.
|
- `--session-id <session_id>`
|
||||||
|
- `--previous-session-id <session_id>`
|
||||||
|
|
||||||
The positional `<session_id>` is required even when `--session` is provided. It is used as the expected session identity and as the remote session lookup value when local session discovery misses.
|
Rules:
|
||||||
|
|
||||||
|
- `--campaign` and `--campaign-file` are mutually exclusive.
|
||||||
|
- `--session` is not used by `session init`.
|
||||||
|
- if both positional `<session_id>` and `--session-id` are provided, values must match.
|
||||||
|
- `clean --all` cannot be combined with campaign/session selectors.
|
||||||
|
|
||||||
|
## Session ID Input Rules
|
||||||
|
|
||||||
|
Session-aware commands accept one of these forms:
|
||||||
|
|
||||||
|
- positional session ID: `... <session_id>`
|
||||||
|
- compatibility flag: `... --session-id <session_id>`
|
||||||
|
|
||||||
|
When both are present, command parsing requires an exact match.
|
||||||
|
|
||||||
|
Commands with additional positionals keep their command-specific order:
|
||||||
|
|
||||||
|
- `run-stage <stage> <session_id>` or `run-stage <stage> --session-id <session_id>`
|
||||||
|
- `session locks add <session_id> <source>` or `session locks add --session-id <session_id> <source>`
|
||||||
|
- `session locks remove <session_id> <source>` or `session locks remove --session-id <session_id> <source>`
|
||||||
|
|
||||||
## Command Reference
|
## Command Reference
|
||||||
|
|
||||||
### `run`
|
### `run`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio run <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--force] [--artifacts <name[,name...]>]
|
narratio run <session_id> [--force] [--artifacts <name[,name...]>] [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Purpose:
|
Behavior:
|
||||||
- Execute configured stages in canonical order.
|
|
||||||
|
|
||||||
Success output:
|
- evaluates full stage order;
|
||||||
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
- skips already-succeeded stages unless `--force` is set;
|
||||||
|
- continues interrupted or partially completed sessions by running non-succeeded stages;
|
||||||
Common failure cases:
|
- writes session and run manifests.
|
||||||
- missing system default config/session paths when flags are omitted.
|
|
||||||
- missing selected campaign under `pipeline.campaigns.root`.
|
|
||||||
- missing local session plus missing/unavailable remote `session.yml`.
|
|
||||||
- templated `session.yml`; run `narratio session init` to generate concrete YAML.
|
|
||||||
- concrete session identity mismatch.
|
|
||||||
- unknown configured artifact key in `--artifacts`.
|
|
||||||
|
|
||||||
### `resume`
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio resume <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--force] [--artifacts <name[,name...]>]
|
|
||||||
```
|
|
||||||
|
|
||||||
Purpose:
|
|
||||||
- Continue from session-manifest stage status.
|
|
||||||
|
|
||||||
Success output:
|
|
||||||
- `narratio resume: session <session_id> has no remaining stages`
|
|
||||||
- or `narratio resume: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
|
||||||
|
|
||||||
### `run-stage`
|
### `run-stage`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio run-stage <stage> <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--force] [--artifacts <name[,name...]>]
|
narratio run-stage <stage> <session_id> [--force] [--artifacts <name[,name...]>] [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Valid stage names:
|
Valid stage names:
|
||||||
@@ -101,241 +93,189 @@ Valid stage names:
|
|||||||
- `polish`
|
- `polish`
|
||||||
- `normalize`
|
- `normalize`
|
||||||
- `trim`
|
- `trim`
|
||||||
|
- `render`
|
||||||
- `analyze`
|
- `analyze`
|
||||||
- `archive`
|
- `publish`
|
||||||
- `notify`
|
- `notify`
|
||||||
|
|
||||||
Success output:
|
Rules:
|
||||||
- `narratio run-stage: stage=<name> executed=<n> skipped=<n> force=<true|false>; manifest=<path>`
|
|
||||||
|
|
||||||
`--artifacts` is accepted only for `analyze` and `archive`.
|
- `--artifacts` is accepted only for `analyze` and `publish` stage targets.
|
||||||
|
|
||||||
### `analyze`
|
### `analyze`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio analyze <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--artifacts <name[,name...]>]
|
narratio analyze <session_id> [--artifacts <name[,name...]>] [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Purpose:
|
Equivalent to:
|
||||||
- Force-rerun the analyze stage.
|
|
||||||
- Shorter equivalent for `narratio run-stage analyze <session_id> --force`.
|
|
||||||
|
|
||||||
`analyze` is force-by-design and does not accept `--force`.
|
```bash
|
||||||
|
narratio run-stage analyze <session_id> --force [...common config flags]
|
||||||
|
```
|
||||||
|
|
||||||
### `publish`
|
### `publish`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio publish <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--artifacts <name[,name...]>]
|
narratio publish <session_id> [--artifacts <name[,name...]>] [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Purpose:
|
Equivalent to:
|
||||||
- Force-rerun the archive stage.
|
|
||||||
- Shorter equivalent for `narratio run-stage archive <session_id> --force`.
|
|
||||||
|
|
||||||
`publish` is force-by-design and does not accept `--force` or a stage positional argument.
|
```bash
|
||||||
|
narratio run-stage publish <session_id> --force [...common config flags]
|
||||||
|
```
|
||||||
|
|
||||||
### `clean`
|
### `clean`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio clean <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--dry-run] [--clear-cache]
|
narratio clean <session_id> [--dry-run] [--clear-cache] [...common config flags]
|
||||||
narratio clean --all [--config <pipeline.yml>] [--dry-run] [--clear-cache]
|
narratio clean --all [--dry-run] [--clear-cache] [--config <pipeline.yml>]
|
||||||
```
|
```
|
||||||
|
|
||||||
Session cleanup deletes:
|
Behavior:
|
||||||
- `{workspace.root}/work/{campaign}/{session_id}`
|
|
||||||
- `{spool.root}/{campaign}/{session_id}`
|
|
||||||
|
|
||||||
All-session cleanup deletes:
|
- session mode removes:
|
||||||
- `{workspace.root}/work`
|
- `{workspace.root}/work/{campaign}/{session_id}`
|
||||||
- the contents of `{spool.root}`, while preserving the spool root directory itself.
|
- `{spool.root}/{campaign}/{session_id}`
|
||||||
|
- `--all` removes:
|
||||||
Cache behavior:
|
- `{workspace.root}/work/*`
|
||||||
- cache is preserved by default.
|
- direct children under `{spool.root}`
|
||||||
- `--clear-cache` in session mode removes cached S3 audio files for the resolved session.
|
- cache remains unless `--clear-cache` is provided.
|
||||||
- `--all --clear-cache` removes the configured Narratio S3 audio cache namespace for the configured bucket/root prefix.
|
|
||||||
- `--clear-cache` does not delete arbitrary files under `pipeline.cache.root`.
|
|
||||||
|
|
||||||
### `session plan`
|
### `session plan`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session plan <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--force]
|
narratio session plan <session_id> [--force] [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Purpose:
|
Validates config, prepares local workdir layout, and prints run/skip decisions for each stage.
|
||||||
- Validate config, load secrets if configured, prepare workdir, and print stage run/skip decisions.
|
|
||||||
|
|
||||||
Success output includes:
|
|
||||||
- `narratio session plan: workdir prepared at <path>`
|
|
||||||
- one line per stage (`<stage>: run|skip`)
|
|
||||||
- `totals: run=<n> skip=<n>`
|
|
||||||
|
|
||||||
### `session status`
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio session status <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>]
|
|
||||||
```
|
|
||||||
|
|
||||||
Output includes:
|
|
||||||
- session ID, campaign, workspace, and session config source.
|
|
||||||
- local manifest state when present.
|
|
||||||
- remote current archive state when storage is configured.
|
|
||||||
- catalog-based promoted output availability for expected transcript and artifact sources.
|
|
||||||
- effective archive locks and conservative next actions.
|
|
||||||
|
|
||||||
### `session validate`
|
### `session validate`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session validate <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>]
|
narratio session validate <session_id> [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Checks include:
|
Read-only preflight checks for config validity, required inputs, audio mode, previous-session requirements, publish outputs, and effective locks.
|
||||||
- effective config and session source.
|
|
||||||
- stable input files.
|
|
||||||
- local or remote audio availability.
|
|
||||||
- previous-session requirements.
|
|
||||||
- archive promotions and effective locks.
|
|
||||||
|
|
||||||
Warnings do not fail the command. Any `ERROR` finding exits non-zero.
|
### `session status`
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session status <session_id> [...common config flags]
|
||||||
|
```
|
||||||
|
|
||||||
|
Prints local manifest state and, when storage is available, remote current-state and published-output status.
|
||||||
|
|
||||||
### `session init`
|
### `session init`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session init <session_id> --output ./session.yml
|
narratio session init <session_id> --output ./session.yml [options]
|
||||||
narratio session init <session_id> --remote
|
narratio session init <session_id> --remote [options]
|
||||||
narratio session init <session_id> --config <pipeline.yml> --campaign icewind --remote
|
|
||||||
narratio session init <session_id> --config <pipeline.yml> --campaign-file ./campaign.yml --remote
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Additional flags:
|
Required target selection:
|
||||||
|
|
||||||
- `--previous-session-id <value>`
|
- exactly one of:
|
||||||
- `--date <value>`
|
- `--output <path>`
|
||||||
- `--title <value>`
|
- `--remote`
|
||||||
- `--audio-s3-prefix <prefix>`: defaults to `audio/` when neither audio flag is provided.
|
|
||||||
- `--audio-dir <path>`: local audio directory; mutually exclusive with `--audio-s3-prefix`.
|
|
||||||
- `--force`: overwrite existing local or remote target.
|
|
||||||
|
|
||||||
Behavior:
|
Options:
|
||||||
- exactly one of `--output` or `--remote` is required.
|
|
||||||
- `--config`, `--campaign`, and `--campaign-file` are optional overrides; omitted campaign selection uses `pipeline.campaigns.default_campaign_id`.
|
- `--config <pipeline.yml>`
|
||||||
- `--campaign <id>` selects a campaign under `pipeline.campaigns.root`.
|
- `--campaign <id>` or `--campaign-file <campaign.yml>`
|
||||||
- `--campaign-file <path>` loads an explicit campaign file.
|
- `--previous-session-id <id>`
|
||||||
- if `campaign.yml` sets `session_template_file`, the template path is resolved relative to `campaign.yml` and rendered from init flags.
|
- `--date <YYYY-MM-DD>`
|
||||||
- if no session template is configured, a minimal concrete session file is generated directly.
|
- `--title <text>`
|
||||||
- template variables must be supplied by matching flags, and supplied template-related flags must be used by the template.
|
- `--audio-dir <path>`
|
||||||
- remote writes target `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`.
|
- `--audio-s3-prefix <prefix>`
|
||||||
- existing local or remote targets fail unless `--force` is passed.
|
- `--force`
|
||||||
- remote writes use existence checks, not compare-and-swap.
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- `--audio-dir` and `--audio-s3-prefix` are mutually exclusive.
|
||||||
|
- if campaign `session_template_file` is configured, `session init` renders it.
|
||||||
|
- generated session YAML must be concrete (no unresolved `{{ ... }}` placeholders).
|
||||||
|
|
||||||
### `session restore`
|
### `session restore`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session restore <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--dry-run] [--force] [--include-audio]
|
narratio session restore <session_id> [--dry-run] [--force] [--include-audio] [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Purpose:
|
Behavior:
|
||||||
- Restore durable session state from the committed remote archive current state.
|
|
||||||
- Default restore installs `manifest.json`, `transcripts/**`, and `artifacts/**` from the current session archive.
|
|
||||||
- When configured previous-session inputs require it, restore reconstructs `previous/**` from the previous session's committed current archive.
|
|
||||||
- `audio/**` is restored only with `--include-audio`.
|
|
||||||
|
|
||||||
Dry-run output may include planned previous-cache downloads. Existing differing files under `previous/**` follow the normal restore conflict policy and require `--force` to overwrite.
|
- discovers committed remote current state;
|
||||||
|
- plans local restores;
|
||||||
|
- writes `reports/restore-latest.json` on execution;
|
||||||
|
- blocks conflicting overwrites unless `--force` is set.
|
||||||
|
|
||||||
When `--include-audio` is set, S3 audio files are restored through the shared audio cache. Cache hits avoid re-downloading large audio objects.
|
Default restore scope:
|
||||||
|
|
||||||
|
- `manifest.json`
|
||||||
|
- `transcripts/**`
|
||||||
|
- `artifacts/**`
|
||||||
|
- `previous/**` when required by configured previous-session inputs
|
||||||
|
|
||||||
|
`audio/**` is included only with `--include-audio`.
|
||||||
|
|
||||||
### `session artifacts`
|
### `session artifacts`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session artifacts <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--remote]
|
narratio session artifacts <session_id> [--remote] [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Purpose:
|
Lists effective built-in and configured artifact sources, publish rules, lock state, and optional remote published-state availability.
|
||||||
- List built-in, configured, previous-session, promoted, and locked artifact sources.
|
|
||||||
|
|
||||||
`--remote` checks promoted top-level object availability through the storage adapter. Remote markers appear only in the `Promoted` section, which reports each configured archive promotion destination and includes `dest=<path>` when that destination differs from the source's canonical path.
|
|
||||||
|
|
||||||
### `session locks`
|
### `session locks`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session locks <session_id> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>]
|
narratio session locks <session_id> [...common config flags]
|
||||||
narratio session locks add <session_id> <source> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--reason <text>] [--force]
|
narratio session locks add <session_id> <source> [--reason <text>] [--force] [...common config flags]
|
||||||
narratio session locks remove <session_id> <source> [--config <pipeline.yml>] [--campaign <id>] [--campaign-file <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>]
|
narratio session locks remove <session_id> <source> [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Behavior:
|
Behavior:
|
||||||
- list mode prints effective locks from static `pipeline.archive.locks` and remote `{session_prefix}/locks.yml`.
|
|
||||||
- `locks add` writes only the remote lock store and fails if the source is already locked by pipeline config.
|
- list mode merges static `pipeline.publish.locks` with remote `{session_prefix}/locks.yml`;
|
||||||
- `locks remove` removes only remote locks and cannot remove static pipeline locks.
|
- add/remove mutate only remote locks;
|
||||||
- `locks add --force` is required to update an existing remote lock reason.
|
- static locks from pipeline config cannot be removed by CLI commands.
|
||||||
|
|
||||||
|
## `--artifacts` Selection Rules
|
||||||
|
|
||||||
|
- accepted on `run`, `run-stage`, `analyze`, and `publish`;
|
||||||
|
- names must exist in `pipeline.scriptorium.artifacts`;
|
||||||
|
- empty entries are invalid;
|
||||||
|
- repeated names are deduplicated.
|
||||||
|
|
||||||
|
Effects:
|
||||||
|
|
||||||
|
- filters analyze execution to selected configured artifacts;
|
||||||
|
- filters publish rules that source `narratio.artifact.<name>`;
|
||||||
|
- does not filter built-in transcript/bounds publish sources.
|
||||||
|
|
||||||
## Common Workflows
|
## Common Workflows
|
||||||
|
|
||||||
Default-discovery run:
|
Run full pipeline:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio run 2026-04-04
|
narratio run 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
Run only selected analyze artifacts:
|
Dry-run restore plan:
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio run 2026-04-04 --artifacts session_recap,player_handout
|
|
||||||
```
|
|
||||||
|
|
||||||
Resume with selected analyze artifacts:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio resume 2026-04-04 --artifacts player_handout
|
|
||||||
```
|
|
||||||
|
|
||||||
Force-rerun analyze with selected artifacts:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio analyze 2026-04-04 --artifacts player_handout
|
|
||||||
```
|
|
||||||
|
|
||||||
Force-rerun archive publishing:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio publish 2026-04-04
|
|
||||||
```
|
|
||||||
|
|
||||||
Preview restore actions without writes:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session restore 2026-04-04 --dry-run
|
narratio session restore 2026-04-04 --dry-run
|
||||||
```
|
```
|
||||||
|
|
||||||
Restore and then force analyze:
|
Generate a concrete session file from template/default structure:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session restore 2026-04-04
|
narratio session init 2026-04-04 --output ./session.yml --date 2026-04-04 --title "Session 12"
|
||||||
narratio analyze 2026-04-04
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Rehydrate canonical previous-session inputs after artifact-input changes:
|
Force publish only:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio run-stage prepare 2026-04-04 --force
|
narratio publish 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
Reset local state before testing restore:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio clean 2026-04-04 --dry-run
|
|
||||||
narratio clean 2026-04-04
|
|
||||||
narratio session restore 2026-04-04 --include-audio
|
|
||||||
```
|
|
||||||
|
|
||||||
Clean all local sessions while keeping cached S3 audio:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio clean --all
|
|
||||||
```
|
|
||||||
|
|
||||||
## `--artifacts` and `--force`
|
|
||||||
|
|
||||||
- `--artifacts` filters which configured artifacts are executable when analyze runs and which configured artifact promotions archive publishes.
|
|
||||||
- `--artifacts` does not imply `--force`.
|
|
||||||
- if analyze is already `succeeded` and `--force` is not set, runner-level skip still applies.
|
|
||||||
- `--artifacts` does not suppress built-in transcript or bounds promotions.
|
|
||||||
|
|||||||
552
docs/config.md
552
docs/config.md
@@ -1,149 +1,77 @@
|
|||||||
# Configuration
|
# Configuration Reference
|
||||||
|
|
||||||
## 1. Overview
|
## Purpose
|
||||||
|
|
||||||
Narratio loads three YAML files:
|
Narratio resolves three YAML documents:
|
||||||
|
|
||||||
- `pipeline.yml`: pipeline-level runtime configuration.
|
- `pipeline.yml`: pipeline/runtime settings
|
||||||
- `campaign.yml`: stable campaign identity and campaign-level input defaults.
|
- `campaign.yml`: campaign identity and stable input defaults
|
||||||
- `session.yml`: per-session metadata and input selection, loaded locally or from the configured S3 backend.
|
- `session.yml`: session identity, metadata, and audio source selection
|
||||||
|
|
||||||
These commands load and validate all three files before running:
|
## Discovery and Selection
|
||||||
|
|
||||||
- `narratio run`
|
|
||||||
- `narratio resume`
|
|
||||||
- `narratio run-stage`
|
|
||||||
- `narratio analyze`
|
|
||||||
- `narratio publish`
|
|
||||||
- `narratio session plan`
|
|
||||||
- `narratio session status`
|
|
||||||
- `narratio session validate`
|
|
||||||
- `narratio session restore`
|
|
||||||
- `narratio session artifacts`
|
|
||||||
- `narratio session locks`
|
|
||||||
- `narratio clean <session_id>`
|
|
||||||
|
|
||||||
Behavior:
|
|
||||||
|
|
||||||
- strict YAML decode is enabled (`KnownFields(true)`): unknown fields fail.
|
|
||||||
- ordinary local and remote `session.yml` files must be concrete YAML; template placeholders are rejected.
|
|
||||||
- defaults are applied for optional pipeline fields.
|
|
||||||
- campaign identity is selected by ID from the pipeline campaign registry unless `--campaign-file` is used.
|
|
||||||
- campaign-level stable input paths fill missing session input paths.
|
|
||||||
- session-level stable input paths override campaign-level input paths.
|
|
||||||
- campaign config may point `session init` to a session template.
|
|
||||||
- validation enforces required fields, value formats, and cross-field constraints.
|
|
||||||
|
|
||||||
## 2. Config file discovery
|
|
||||||
|
|
||||||
These commands use the same config discovery behavior:
|
|
||||||
|
|
||||||
- `narratio run`
|
|
||||||
- `narratio resume`
|
|
||||||
- `narratio run-stage`
|
|
||||||
- `narratio analyze`
|
|
||||||
- `narratio publish`
|
|
||||||
- `narratio session plan`
|
|
||||||
- `narratio session status`
|
|
||||||
- `narratio session validate`
|
|
||||||
- `narratio session restore`
|
|
||||||
- `narratio session artifacts`
|
|
||||||
- `narratio session locks`
|
|
||||||
- `narratio clean <session_id>`
|
|
||||||
|
|
||||||
Pipeline config lookup:
|
|
||||||
|
|
||||||
- if `--config <path>` is provided, that path is used.
|
|
||||||
- if omitted, Narratio searches in order:
|
|
||||||
1. `/usr/local/etc/narratio/pipeline.yml`
|
|
||||||
2. `/etc/narratio/pipeline.yml`
|
|
||||||
- first existing file wins.
|
|
||||||
|
|
||||||
Campaign config lookup:
|
|
||||||
|
|
||||||
- pipeline config is loaded first.
|
|
||||||
- if `--campaign-file <path>` is provided, that path is used.
|
|
||||||
- otherwise, if `--campaign <id>` is provided, Narratio loads:
|
|
||||||
- `{pipeline.campaigns.root}/{id}/campaign.yml`
|
|
||||||
- otherwise, Narratio uses `pipeline.campaigns.default_campaign_id` and loads:
|
|
||||||
- `{pipeline.campaigns.root}/{default_campaign_id}/campaign.yml`
|
|
||||||
- `--campaign` and `--campaign-file` are mutually exclusive.
|
|
||||||
- campaign IDs must be single path segments, not paths.
|
|
||||||
|
|
||||||
Session config lookup:
|
|
||||||
|
|
||||||
- if `--session <path>` is provided, that path is used.
|
|
||||||
- if `--session` is omitted, Narratio searches locally in order:
|
|
||||||
1. `/usr/local/etc/narratio/session.yml`
|
|
||||||
2. `/etc/narratio/session.yml`
|
|
||||||
- first existing local file wins.
|
|
||||||
- if no local session file is found, a positional `<session_id>` is present, storage is configured, and campaign identity is resolved, Narratio loads remote `session.yml` from:
|
|
||||||
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`
|
|
||||||
- local discovery always runs before remote fallback.
|
|
||||||
- local files in the current working directory are used only when passed explicitly, for example `--config ./pipeline.yml --campaign-file ./campaign.yml --session ./session.yml`.
|
|
||||||
|
|
||||||
## 3. Session templating
|
|
||||||
|
|
||||||
Template behavior for local and remote `session.yml` loaded by downstream commands:
|
|
||||||
|
|
||||||
- downstream commands do not render templates.
|
|
||||||
- local and remote `session.yml` must be concrete.
|
|
||||||
- any `{{ ... }}` placeholder in loaded `session.yml` fails with guidance to run `narratio session init`.
|
|
||||||
- if concrete `session_id` mismatches the positional `<session_id>`, load fails.
|
|
||||||
- if concrete `previous_session_id` mismatches `--previous-session-id`, load fails.
|
|
||||||
|
|
||||||
Template behavior for `narratio session init`:
|
|
||||||
|
|
||||||
- `campaign.yml` may set `session_template_file`.
|
|
||||||
- relative template paths resolve relative to `campaign.yml`.
|
|
||||||
- supported init template variables:
|
|
||||||
- `{{ session_id }}`
|
|
||||||
- `{{ previous_session_id }}`
|
|
||||||
- `{{ date }}`
|
|
||||||
- `{{ title }}`
|
|
||||||
- `{{ audio_s3_prefix }}`
|
|
||||||
- `{{ audio_dir }}`
|
|
||||||
- each template variable must be supplied by the matching `session init` flag.
|
|
||||||
- template-related flags such as `--date`, `--title`, `--audio-s3-prefix`, `--audio-dir`, and `--previous-session-id` fail if the configured template does not use them.
|
|
||||||
- rendered output is strict-decoded and validated before it is written locally or remotely.
|
|
||||||
- if `session_template_file` is omitted, `session init` generates the minimal concrete session YAML directly.
|
|
||||||
|
|
||||||
## 4. Minimal config set
|
|
||||||
|
|
||||||
### `pipeline.yml`
|
### `pipeline.yml`
|
||||||
|
|
||||||
|
When `--config` is omitted, search order is:
|
||||||
|
|
||||||
|
1. `/usr/local/etc/narratio/pipeline.yml`
|
||||||
|
2. `/etc/narratio/pipeline.yml`
|
||||||
|
|
||||||
|
### `campaign.yml`
|
||||||
|
|
||||||
|
Selection rules:
|
||||||
|
|
||||||
|
- if `--campaign-file` is set, use that path;
|
||||||
|
- else if `--campaign <id>` is set, use `{pipeline.campaigns.root}/{id}/campaign.yml`;
|
||||||
|
- else use `{pipeline.campaigns.root}/{pipeline.campaigns.default_campaign_id}/campaign.yml`.
|
||||||
|
|
||||||
|
### `session.yml`
|
||||||
|
|
||||||
|
When `--session` is omitted, local search order is:
|
||||||
|
|
||||||
|
1. `/usr/local/etc/narratio/session.yml`
|
||||||
|
2. `/etc/narratio/session.yml`
|
||||||
|
|
||||||
|
If local session discovery fails and a `session_id` is known, Narratio attempts remote session loading from:
|
||||||
|
|
||||||
|
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`
|
||||||
|
|
||||||
|
using configured object storage.
|
||||||
|
|
||||||
|
## Validation and Merge Rules
|
||||||
|
|
||||||
|
- YAML decode is strict (`KnownFields(true)`): unknown fields fail load.
|
||||||
|
- Session files must be concrete; unresolved `{{ ... }}` placeholders fail load.
|
||||||
|
- Pipeline defaults are applied before validation.
|
||||||
|
- Campaign and session identities must agree.
|
||||||
|
- Stable files (`speakers_file`, `autocorrect_file`, `glossary_file`) resolve from session overrides when provided, otherwise from campaign defaults.
|
||||||
|
- Exactly one audio mode must be configured in session input:
|
||||||
|
- local (`audio_dir` or `audio_files`), or
|
||||||
|
- S3 (`audio_s3.prefix`).
|
||||||
|
|
||||||
|
## Minimal Working Configuration
|
||||||
|
|
||||||
|
`pipeline.yml`
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
campaigns:
|
campaigns:
|
||||||
root: /usr/local/share/narratio/campaigns
|
root: /usr/local/share/narratio/campaigns
|
||||||
default_campaign_id: sample-campaign
|
default_campaign_id: sample-campaign
|
||||||
whisperx:
|
whisperx:
|
||||||
transcribe_url: "https://transcription.example.com/transcribe"
|
transcribe_url: https://transcription.example.com/transcribe
|
||||||
```
|
```
|
||||||
|
|
||||||
Why this is sufficient:
|
`campaign.yml`
|
||||||
|
|
||||||
- `whisperx.transcribe_url` is required.
|
|
||||||
- `campaigns.default_campaign_id` selects the default campaign when `--campaign` is omitted.
|
|
||||||
- `workspace.root` defaults to `/var/lib/narratio`.
|
|
||||||
- optional sections (`seriatim`, `audita`, `archive`, `scriptorium`, `trim`, `normalize`, etc.) receive defaults or stay inactive.
|
|
||||||
|
|
||||||
### `campaign.yml`
|
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
campaign_id: sample-campaign
|
campaign_id: sample-campaign
|
||||||
session_template_file: ./session.template.yml
|
|
||||||
inputs:
|
inputs:
|
||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
Why this is sufficient:
|
`session.yml` (local audio)
|
||||||
|
|
||||||
- `campaign_id` supplies the stable campaign identity.
|
|
||||||
- stable input files are required and resolve relative to `campaign.yml` when copied during `prepare`.
|
|
||||||
|
|
||||||
### `session.yml`
|
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
session_id: 2026-05-03
|
session_id: 2026-05-03
|
||||||
@@ -151,152 +79,59 @@ inputs:
|
|||||||
audio_dir: ./audio
|
audio_dir: ./audio
|
||||||
```
|
```
|
||||||
|
|
||||||
Why this is sufficient:
|
## Secrets Handling
|
||||||
|
|
||||||
- `session_id` is required.
|
- Do not place raw secrets in YAML.
|
||||||
- `campaign` can be omitted because it is supplied by `campaign.yml`.
|
- Use env var names in config (for example `pipeline.audita.llm_api_key_env`).
|
||||||
- stable input paths can be omitted because `campaign.yml` supplies defaults.
|
- Optionally load env files from `pipeline.secrets.env_dir`.
|
||||||
- local `audio_dir` resolves relative to `session.yml`.
|
- Commands that need storage/auth load filesystem secrets before constructing adapters.
|
||||||
|
|
||||||
Minimal local-file usage:
|
## Publish Configuration Summary
|
||||||
|
|
||||||
```bash
|
Publish rules live under `pipeline.publish`.
|
||||||
narratio run 2026-05-03 --config /path/to/pipeline.yml --campaign sample-campaign --session ./session.yml
|
|
||||||
narratio run 2026-05-03 --config /path/to/pipeline.yml --campaign-file ./campaign.yml --session ./session.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
Previous-session-enabled variant:
|
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
session_id: 2026-05-03
|
publish:
|
||||||
previous_session_id: 2026-04-26
|
|
||||||
inputs:
|
|
||||||
audio_dir: ./audio
|
|
||||||
```
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio run 2026-05-03 --config /path/to/pipeline.yml --campaign sample-campaign --session ./session.yml --previous-session-id 2026-04-26
|
|
||||||
```
|
|
||||||
|
|
||||||
## 5. Production-oriented config set
|
|
||||||
|
|
||||||
### `pipeline.yml`
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
workspace:
|
|
||||||
root: /var/lib/narratio/workspace
|
|
||||||
cleanup_after_archive: true
|
|
||||||
|
|
||||||
storage:
|
|
||||||
backend: s3
|
|
||||||
s3:
|
|
||||||
bucket: my-dnd-archive
|
|
||||||
root_prefix: dnd
|
|
||||||
region: us-east-1
|
|
||||||
access_key_id_env: OBJECT_STORAGE_KEY_ID
|
|
||||||
secret_access_key_env: OBJECT_STORAGE_KEY
|
|
||||||
|
|
||||||
campaigns:
|
|
||||||
root: /srv/narratio/campaigns
|
|
||||||
default_campaign_id: forsaken
|
|
||||||
|
|
||||||
spool:
|
|
||||||
root: /var/spool/narratio
|
|
||||||
delete_audio_after_archive: true
|
|
||||||
|
|
||||||
cache:
|
|
||||||
root: /var/cache/narratio
|
|
||||||
s3_audio: true
|
|
||||||
|
|
||||||
archive:
|
|
||||||
enabled: true
|
enabled: true
|
||||||
upload_run: true
|
upload_run: true
|
||||||
promote_artifacts:
|
outputs:
|
||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
required: true
|
required: true
|
||||||
|
- source: narratio.transcript.final_markdown
|
||||||
|
dest: transcripts/final.md
|
||||||
|
required: true
|
||||||
|
- source: narratio.transcript.final_trimmed_markdown
|
||||||
|
dest: transcripts/final.trimmed.md
|
||||||
|
required: true
|
||||||
- source: narratio.artifact.session_recap
|
- source: narratio.artifact.session_recap
|
||||||
dest: artifacts/session_recap.md
|
dest: artifacts/session_recap.md
|
||||||
required: true
|
required: true
|
||||||
locks:
|
locks:
|
||||||
- source: narratio.artifact.session_recap
|
- source: narratio.artifact.session_recap
|
||||||
reason: Final recap was manually edited.
|
reason: manual post-publish edits
|
||||||
|
|
||||||
whisperx:
|
|
||||||
transcribe_url: "https://transcription.example.com/transcribe"
|
|
||||||
|
|
||||||
scriptorium:
|
|
||||||
artifacts:
|
|
||||||
session_recap:
|
|
||||||
enabled: true
|
|
||||||
prompt_id: dnd.session_recap
|
|
||||||
output_path: artifacts/session_recap.md
|
|
||||||
inputs:
|
|
||||||
transcript:
|
|
||||||
source: narratio.transcript.final_trimmed
|
|
||||||
required: true
|
|
||||||
previous_recap:
|
|
||||||
source: narratio.previous_session.artifact.session_recap
|
|
||||||
required: false
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### `campaign.yml`
|
Rules:
|
||||||
|
|
||||||
```yaml
|
- `outputs[].source` is required.
|
||||||
campaign_id: forsaken
|
- `outputs[].dest` may be omitted when derivable from source.
|
||||||
inputs:
|
- `outputs[].required` defaults to `true`.
|
||||||
speakers_file: /srv/narratio/campaigns/forsaken/speakers.yml
|
- static locks (`pipeline.publish.locks`) merge with remote locks (`{session_prefix}/locks.yml`), with static locks taking precedence on duplicates.
|
||||||
autocorrect_file: /srv/narratio/campaigns/forsaken/autocorrect.yml
|
|
||||||
glossary_file: /srv/narratio/campaigns/forsaken/glossary.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
### Local `session.yml`
|
## Full Schema
|
||||||
|
|
||||||
```yaml
|
### Pipeline
|
||||||
session_id: 2026-05-03
|
|
||||||
previous_session_id: 2026-04-26
|
|
||||||
date: 2026-05-03
|
|
||||||
title: The Black Cabin
|
|
||||||
inputs:
|
|
||||||
audio_s3:
|
|
||||||
prefix: audio/
|
|
||||||
```
|
|
||||||
|
|
||||||
### S3-first session config
|
| Field | Type | Required | Default / Rule |
|
||||||
|
|
||||||
For S3-first operation, upload the same `session.yml` content to:
|
|
||||||
|
|
||||||
```text
|
|
||||||
{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
Then run with explicit or discovered pipeline/campaign config and no `--session`:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio run 2026-05-03 --config /usr/local/etc/narratio/pipeline.yml --campaign forsaken --previous-session-id 2026-04-26
|
|
||||||
```
|
|
||||||
|
|
||||||
Operational notes:
|
|
||||||
|
|
||||||
- archive promotion is explicit and source-based via `archive.promote_artifacts`.
|
|
||||||
- `source` is required; `dest` is optional and derived when omitted.
|
|
||||||
- `archive.locks` skips top-level promotion overwrites for static locked sources while preserving run-local uploads.
|
|
||||||
- operator-created mutable locks are stored at `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/locks.yml` and are merged with static locks.
|
|
||||||
- Narratio does not auto-promote all generated analyze artifacts.
|
|
||||||
- `restore` reads the same config/campaign/session inputs and restore scope is bounded by committed archive current state.
|
|
||||||
- `clean` removes workspace/spool state by default and preserves `pipeline.cache.root` unless `--clear-cache` is passed.
|
|
||||||
|
|
||||||
## 6. Full pipeline reference
|
|
||||||
|
|
||||||
| Path | Type | Required | Default |
|
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
|
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
|
||||||
| `pipeline.workspace.cleanup_after_archive` | bool | No | `false` |
|
| `pipeline.workspace.cleanup_after_publish` | bool | No | `false` |
|
||||||
| `pipeline.campaigns.root` | string | No | `/usr/local/share/narratio/campaigns` |
|
| `pipeline.campaigns.root` | string | No | `/usr/local/share/narratio/campaigns` |
|
||||||
| `pipeline.campaigns.default_campaign_id` | string | No | empty |
|
| `pipeline.campaigns.default_campaign_id` | string | No | empty |
|
||||||
| `pipeline.secrets.env_dir` | string | Conditional | none |
|
| `pipeline.secrets.env_dir` | string | No | empty |
|
||||||
| `pipeline.storage.backend` | string | No | empty |
|
| `pipeline.storage.backend` | string | No | empty |
|
||||||
| `pipeline.storage.s3.bucket` | string | Conditional | empty |
|
| `pipeline.storage.s3.bucket` | string | Conditional | required for S3 session-audio and for publish upload when backend is `s3` |
|
||||||
| `pipeline.storage.s3.root_prefix` | string | No | `dnd` |
|
| `pipeline.storage.s3.root_prefix` | string | No | `dnd` |
|
||||||
| `pipeline.storage.s3.region` | string | No | empty |
|
| `pipeline.storage.s3.region` | string | No | empty |
|
||||||
| `pipeline.storage.s3.endpoint` | string | No | empty |
|
| `pipeline.storage.s3.endpoint` | string | No | empty |
|
||||||
@@ -304,26 +139,26 @@ Operational notes:
|
|||||||
| `pipeline.storage.s3.access_key_id_env` | string | No | `OBJECT_STORAGE_KEY_ID` |
|
| `pipeline.storage.s3.access_key_id_env` | string | No | `OBJECT_STORAGE_KEY_ID` |
|
||||||
| `pipeline.storage.s3.secret_access_key_env` | string | No | `OBJECT_STORAGE_KEY` |
|
| `pipeline.storage.s3.secret_access_key_env` | string | No | `OBJECT_STORAGE_KEY` |
|
||||||
| `pipeline.spool.root` | string | No | `/var/spool/narratio` |
|
| `pipeline.spool.root` | string | No | `/var/spool/narratio` |
|
||||||
| `pipeline.spool.delete_audio_after_archive` | bool | No | `false` |
|
| `pipeline.spool.delete_audio_after_publish` | bool | No | `false` |
|
||||||
| `pipeline.cache.root` | string | No | `/var/cache/narratio` |
|
| `pipeline.cache.root` | string | No | `/var/cache/narratio` |
|
||||||
| `pipeline.cache.s3_audio` | bool | No | `true` |
|
| `pipeline.cache.s3_audio` | bool | No | `true` |
|
||||||
| `pipeline.archive.enabled` | bool | No | `true` |
|
| `pipeline.publish.enabled` | bool | No | `true` |
|
||||||
| `pipeline.archive.upload_run` | bool | No | `true` |
|
| `pipeline.publish.upload_run` | bool | No | `true` |
|
||||||
| `pipeline.archive.promote_artifacts[]` | list | No | final-trimmed transcript rule |
|
| `pipeline.publish.outputs[]` | list | No | defaults to final trimmed JSON plus final and final-trimmed Markdown outputs |
|
||||||
| `pipeline.archive.promote_artifacts[].source` | string | Yes (per rule) | none |
|
| `pipeline.publish.outputs[].source` | string | Yes (per rule) | must reference built-in or configured artifact source |
|
||||||
| `pipeline.archive.promote_artifacts[].dest` | string | No | derived from source |
|
| `pipeline.publish.outputs[].dest` | string | Conditional | derived if omitted and source supports derivation |
|
||||||
| `pipeline.archive.promote_artifacts[].required` | bool | No | `true` |
|
| `pipeline.publish.outputs[].required` | bool | No | `true` |
|
||||||
| `pipeline.archive.locks[]` | list | No | empty |
|
| `pipeline.publish.locks[]` | list | No | empty |
|
||||||
| `pipeline.archive.locks[].source` | string | Yes (per lock) | none |
|
| `pipeline.publish.locks[].source` | string | Yes (per lock) | must reference supported publish source |
|
||||||
| `pipeline.archive.locks[].reason` | string | No | empty |
|
| `pipeline.publish.locks[].reason` | string | No | empty |
|
||||||
| `pipeline.whisperx.transcribe_url` | string | Yes | none |
|
| `pipeline.whisperx.transcribe_url` | string | Yes | valid URL |
|
||||||
| `pipeline.whisperx.language` | string | No | `en` |
|
| `pipeline.whisperx.language` | string | No | `en` |
|
||||||
| `pipeline.whisperx.timeout` | duration string | No | `30m` |
|
| `pipeline.whisperx.timeout` | duration | No | `30m` |
|
||||||
| `pipeline.whisperx.retries` | int | No | `3` |
|
| `pipeline.whisperx.retries` | int | No | `3` |
|
||||||
| `pipeline.whisperx.retry_delay` | duration string | No | `2s` |
|
| `pipeline.whisperx.retry_delay` | duration | No | `2s` |
|
||||||
| `pipeline.whisperx.concurrency` | int | No | `2` |
|
| `pipeline.whisperx.concurrency` | int | No | `2` |
|
||||||
| `pipeline.seriatim.binary` | string | No | `seriatim` |
|
| `pipeline.seriatim.binary` | string | No | `seriatim` |
|
||||||
| `pipeline.seriatim.timeout` | duration string | No | `10m` |
|
| `pipeline.seriatim.timeout` | duration | No | `10m` |
|
||||||
| `pipeline.seriatim.output_schema` | string | No | `seriatim-intermediate` |
|
| `pipeline.seriatim.output_schema` | string | No | `seriatim-intermediate` |
|
||||||
| `pipeline.seriatim.coalesce_gap` | float | No | `3.0` |
|
| `pipeline.seriatim.coalesce_gap` | float | No | `3.0` |
|
||||||
| `pipeline.seriatim.report` | bool | No | `true` |
|
| `pipeline.seriatim.report` | bool | No | `true` |
|
||||||
@@ -332,7 +167,7 @@ Operational notes:
|
|||||||
| `pipeline.seriatim.env.backchannel_max_duration` | float | No | unset |
|
| `pipeline.seriatim.env.backchannel_max_duration` | float | No | unset |
|
||||||
| `pipeline.seriatim.env.filler_max_duration` | float | No | unset |
|
| `pipeline.seriatim.env.filler_max_duration` | float | No | unset |
|
||||||
| `pipeline.audita.binary` | string | No | `audita` |
|
| `pipeline.audita.binary` | string | No | `audita` |
|
||||||
| `pipeline.audita.timeout` | duration string | No | `3h` |
|
| `pipeline.audita.timeout` | duration | No | `3h` |
|
||||||
| `pipeline.audita.llm_api_key_env` | string | No | empty |
|
| `pipeline.audita.llm_api_key_env` | string | No | empty |
|
||||||
| `pipeline.audita.modules[]` | list[string] | No | empty |
|
| `pipeline.audita.modules[]` | list[string] | No | empty |
|
||||||
| `pipeline.audita.base_url` | string | No | empty |
|
| `pipeline.audita.base_url` | string | No | empty |
|
||||||
@@ -350,176 +185,91 @@ Operational notes:
|
|||||||
| `pipeline.normalize.output_schema` | string | No | `seriatim-intermediate` |
|
| `pipeline.normalize.output_schema` | string | No | `seriatim-intermediate` |
|
||||||
| `pipeline.normalize.report` | bool | No | `true` |
|
| `pipeline.normalize.report` | bool | No | `true` |
|
||||||
| `pipeline.trim.enabled` | bool | No | `false` |
|
| `pipeline.trim.enabled` | bool | No | `false` |
|
||||||
| `pipeline.trim.output_path` | string | Conditional | none |
|
| `pipeline.trim.output_path` | string | Conditional | required when trim enabled |
|
||||||
| `pipeline.trim.bounds.prompt_id` | string | Conditional | none |
|
| `pipeline.trim.bounds.prompt_id` | string | Conditional | required when trim enabled |
|
||||||
| `pipeline.trim.bounds.profile_id` | string | No | empty |
|
| `pipeline.trim.bounds.profile_id` | string | No | empty |
|
||||||
| `pipeline.trim.bounds.transcript_input_name` | string | Conditional | none |
|
| `pipeline.trim.bounds.transcript_input_name` | string | Conditional | required when trim enabled |
|
||||||
| `pipeline.trim.bounds.output_path` | string | Conditional | none |
|
| `pipeline.trim.bounds.output_path` | string | Conditional | required when trim enabled |
|
||||||
| `pipeline.trim.bounds.timeout` | duration string | No | `10m` |
|
| `pipeline.trim.bounds.timeout` | duration | No | `10m` |
|
||||||
| `pipeline.trim.bounds.render_debug` | bool | No | `false` |
|
| `pipeline.trim.bounds.render_debug` | bool | No | `false` |
|
||||||
| `pipeline.trim.bounds.render_output_path` | string | Conditional | none |
|
| `pipeline.trim.bounds.render_output_path` | string | Conditional | required when `render_debug` is true |
|
||||||
| `pipeline.trim.seriatim.report` | bool | No | `false` |
|
| `pipeline.trim.seriatim.report` | bool | No | `false` |
|
||||||
|
| `pipeline.render.enabled` | bool | No | `true` |
|
||||||
|
| `pipeline.render.format` | string | No | `markdown` (only supported value) |
|
||||||
|
| `pipeline.render.title` | string | No | empty (falls back to `session.title` when set) |
|
||||||
|
| `pipeline.render.include_timestamps` | bool | No | `true` |
|
||||||
|
| `pipeline.render.include_segment_ids` | bool | No | `false` |
|
||||||
|
| `pipeline.render.include_metadata` | bool | No | `false` |
|
||||||
| `pipeline.scriptorium.binary` | string | No | `scriptorium` |
|
| `pipeline.scriptorium.binary` | string | No | `scriptorium` |
|
||||||
| `pipeline.scriptorium.config_path` | string | No | empty |
|
| `pipeline.scriptorium.config_path` | string | No | empty |
|
||||||
| `pipeline.scriptorium.timeout` | duration string | No | `10m` |
|
| `pipeline.scriptorium.timeout` | duration | No | `10m` |
|
||||||
| `pipeline.scriptorium.render_debug` | bool | No | `false` |
|
| `pipeline.scriptorium.render_debug` | bool | No | `false` |
|
||||||
| `pipeline.scriptorium.artifacts` | map | No | empty |
|
| `pipeline.scriptorium.artifacts` | map | No | empty |
|
||||||
| `pipeline.scriptorium.artifacts.<name>.enabled` | bool | No | `false` |
|
|
||||||
| `pipeline.scriptorium.artifacts.<name>.depends_on[]` | list[string] | No | empty |
|
|
||||||
| `pipeline.scriptorium.artifacts.<name>.render_debug` | bool | No | unset |
|
|
||||||
| `pipeline.scriptorium.artifacts.<name>.prompt_id` | string | Conditional | none |
|
|
||||||
| `pipeline.scriptorium.artifacts.<name>.profile_id` | string | No | empty |
|
|
||||||
| `pipeline.scriptorium.artifacts.<name>.output_path` | string | Conditional | none |
|
|
||||||
| `pipeline.scriptorium.artifacts.<name>.timeout` | duration string | No | empty |
|
|
||||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` | string | Conditional | none |
|
|
||||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.artifact` | string | No | empty |
|
|
||||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.path` | string | No | empty |
|
|
||||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required` | bool | No | `false` |
|
|
||||||
| `pipeline.scriptorium.artifacts.<name>.vars.<key>` | map value | No | empty |
|
|
||||||
| `pipeline.notification.backend` | string | No | empty |
|
| `pipeline.notification.backend` | string | No | empty |
|
||||||
| `pipeline.notification.recipient` | string | No | empty |
|
| `pipeline.notification.recipient` | string | No | empty |
|
||||||
| `pipeline.notification.timeout` | duration string | No | empty |
|
| `pipeline.notification.timeout` | duration | No | `30s` |
|
||||||
|
|
||||||
Scriptorium artifact-key and dependency rules:
|
### Scriptorium Artifact Entries
|
||||||
|
|
||||||
- artifact keys must match `^[a-z][a-z0-9_]*$`.
|
For each `pipeline.scriptorium.artifacts.<name>`:
|
||||||
- enabled artifacts require `prompt_id` and `output_path`.
|
|
||||||
- `output_path` must be relative, traversal-safe, and under `artifacts/`.
|
|
||||||
- configured artifact input sources use `narratio.artifact.<name>`.
|
|
||||||
- if input source references `narratio.artifact.<name>`, artifact `<name>` must exist and must be listed in `depends_on`.
|
|
||||||
- every `depends_on` entry must be a configured artifact key.
|
|
||||||
- self-dependency is rejected.
|
|
||||||
- enabled dependency cycles are rejected.
|
|
||||||
- any artifact referenced by `depends_on` or `narratio.artifact.<name>` source must define `output_path` (even if not enabled).
|
|
||||||
|
|
||||||
Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values:
|
| Field | Type | Required | Rule |
|
||||||
|
|
||||||
- `narratio.previous_session.artifact.<configured_artifact_key>`
|
|
||||||
- `narratio.transcript.base`
|
|
||||||
- `narratio.transcript.polished`
|
|
||||||
- `narratio.transcript.final`
|
|
||||||
- `narratio.transcript.final_trimmed`
|
|
||||||
- `narratio.bounds.session`
|
|
||||||
- `narratio.artifact.<configured_artifact_key>`
|
|
||||||
|
|
||||||
`pipeline.archive.promote_artifacts[].source` values:
|
|
||||||
|
|
||||||
- `narratio.transcript.base`
|
|
||||||
- `narratio.transcript.polished`
|
|
||||||
- `narratio.transcript.final`
|
|
||||||
- `narratio.transcript.final_trimmed`
|
|
||||||
- `narratio.bounds.session`
|
|
||||||
- `narratio.artifact.<configured_artifact_key>`
|
|
||||||
|
|
||||||
`pipeline.archive.locks[].source` accepts the same source values as `pipeline.archive.promote_artifacts[].source`.
|
|
||||||
|
|
||||||
Archive promotion destination rules:
|
|
||||||
|
|
||||||
- `dest` must be a clean relative path (not absolute, no traversal).
|
|
||||||
- duplicate `dest` values are rejected.
|
|
||||||
- if `dest` is omitted:
|
|
||||||
- built-in sources derive their canonical destination path;
|
|
||||||
- configured sources derive from `pipeline.scriptorium.artifacts.<name>.output_path`;
|
|
||||||
- derivation failure is a config validation error.
|
|
||||||
|
|
||||||
Archive lock rules:
|
|
||||||
|
|
||||||
- locks are source-based and do not accept `dest`.
|
|
||||||
- duplicate lock sources are rejected.
|
|
||||||
- static `pipeline.archive.locks` win over remote mutable locks for the same source.
|
|
||||||
- locked promotions are recorded as intentional skips in archive metadata.
|
|
||||||
- locked required promotions do not fail archive by default.
|
|
||||||
- ordinary `--force` reruns do not override locks.
|
|
||||||
|
|
||||||
Remote mutable lock store:
|
|
||||||
|
|
||||||
- path: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/locks.yml`.
|
|
||||||
- strict YAML shape: top-level `locks`, each with `source` and optional `reason`.
|
|
||||||
- `narratio session locks add` and `narratio session locks remove` mutate only the remote lock store.
|
|
||||||
- writes use existence checks plus `--force` for updates; they are not compare-and-swap atomic.
|
|
||||||
|
|
||||||
Restore-related implications:
|
|
||||||
|
|
||||||
- restore remote identity requires archive S3 identity to resolve (`pipeline.storage.s3.bucket` and session prefix derivation inputs).
|
|
||||||
- restore scope considers committed current state and durable paths (`manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`, optional `audio/**`).
|
|
||||||
- S3 audio downloads use `pipeline.spool.root` for active downloads and `pipeline.cache.root` for reusable cached audio when `pipeline.cache.s3_audio` is true.
|
|
||||||
- `pipeline.cache.root` is durable local cache state. It is not workspace state and is preserved by default by `narratio clean`.
|
|
||||||
|
|
||||||
## 7. Full campaign reference
|
|
||||||
|
|
||||||
| Path | Type | Required | Default |
|
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| `campaign.campaign_id` | string | Yes | none |
|
| `enabled` | bool | No | `false` if omitted |
|
||||||
| `campaign.session_template_file` | string | No | none |
|
| `depends_on[]` | list[string] | No | must reference configured artifact keys; no self-reference; enabled graph must be acyclic |
|
||||||
| `campaign.inputs.speakers_file` | string | Yes | none |
|
| `render_debug` | bool | No | per-artifact override |
|
||||||
| `campaign.inputs.autocorrect_file` | string | Yes | none |
|
| `prompt_id` | string | Conditional | required when artifact is enabled |
|
||||||
| `campaign.inputs.glossary_file` | string | Yes | none |
|
| `profile_id` | string | No | empty |
|
||||||
|
| `output_path` | string | Conditional | required when enabled; also required when referenced by publish/output/input rules |
|
||||||
|
| `timeout` | duration | No | artifact override |
|
||||||
|
| `inputs` | map | No | input key names must be non-empty |
|
||||||
|
| `vars` | map | No | values must be string or bool |
|
||||||
|
|
||||||
Campaign input paths and `campaign.session_template_file` may be absolute or relative. Relative paths resolve from the directory containing `campaign.yml`.
|
For each artifact input `pipeline.scriptorium.artifacts.<name>.inputs.<input_name>`:
|
||||||
|
|
||||||
## 8. Full session reference
|
| Field | Type | Required | Rule |
|
||||||
|
|
||||||
| Path | Type | Required | Default |
|
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| `session.session_id` | string | Yes | none |
|
| `source` | string | Yes | built-in runtime source, `narratio.artifact.<name>`, or `narratio.previous_session.artifact.<name>` |
|
||||||
| `session.previous_session_id` | string | No | empty |
|
| `artifact` | string | No | optional passthrough adapter field |
|
||||||
| `session.campaign` | string | No | `campaign.campaign_id` |
|
| `path` | string | No | optional passthrough adapter field |
|
||||||
| `session.date` | string | No | empty |
|
| `required` | bool | No | optional input requirement |
|
||||||
| `session.title` | string | No | empty |
|
|
||||||
| `session.inputs.audio_dir` | string | Conditional | empty |
|
|
||||||
| `session.inputs.audio_files[]` | list[string] | Conditional | empty |
|
|
||||||
| `session.inputs.audio_s3.prefix` | string | Conditional | none |
|
|
||||||
| `session.inputs.speakers_file` | string | No | `campaign.inputs.speakers_file` |
|
|
||||||
| `session.inputs.autocorrect_file` | string | No | `campaign.inputs.autocorrect_file` |
|
|
||||||
| `session.inputs.glossary_file` | string | No | `campaign.inputs.glossary_file` |
|
|
||||||
|
|
||||||
Session input paths may be absolute or relative. Relative audio paths and session-level stable input overrides resolve from the directory containing `session.yml`. If both `campaign.yml` and `session.yml` specify campaign identity, the values must match.
|
### Campaign
|
||||||
|
|
||||||
Audio-source rule:
|
| Field | Type | Required | Notes |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `campaign_id` | string | Yes | canonical campaign identity |
|
||||||
|
| `session_template_file` | string | No | used by `session init` when set |
|
||||||
|
| `inputs.speakers_file` | string | Yes | stable input default |
|
||||||
|
| `inputs.autocorrect_file` | string | Yes | stable input default |
|
||||||
|
| `inputs.glossary_file` | string | Yes | stable input default |
|
||||||
|
|
||||||
- configure exactly one mode:
|
### Session
|
||||||
- `audio_dir`, or
|
|
||||||
- `audio_files` (at least one), or
|
|
||||||
- `audio_s3.prefix`
|
|
||||||
- `audio_s3` cannot be combined with local audio fields.
|
|
||||||
|
|
||||||
Previous-session rule:
|
| Field | Type | Required in session file | Notes |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `session_id` | string | Yes | must match CLI session target when provided |
|
||||||
|
| `previous_session_id` | string | No | must not equal `session_id` |
|
||||||
|
| `campaign` | string | No | filled from `campaign_id` during resolve if omitted |
|
||||||
|
| `date` | string | No | metadata |
|
||||||
|
| `title` | string | No | metadata |
|
||||||
|
| `inputs.speakers_file` | string | No | overrides campaign stable input |
|
||||||
|
| `inputs.autocorrect_file` | string | No | overrides campaign stable input |
|
||||||
|
| `inputs.glossary_file` | string | No | overrides campaign stable input |
|
||||||
|
| `inputs.audio_dir` | string | Conditional | local audio mode |
|
||||||
|
| `inputs.audio_files[]` | list[string] | Conditional | local audio mode |
|
||||||
|
| `inputs.audio_s3.prefix` | string | Conditional | S3 audio mode |
|
||||||
|
|
||||||
- if `session.previous_session_id` is set, it must not equal `session.session_id`.
|
Audio rules:
|
||||||
- canonical previous-session sources (`narratio.previous_session.artifact.<name>`) are hydrated during `prepare` from archive current state when required by enabled configured artifacts.
|
|
||||||
|
|
||||||
## 9. Secrets
|
- configure local mode (`audio_dir` or `audio_files`) or S3 mode (`audio_s3.prefix`), not both.
|
||||||
|
|
||||||
Narratio supports filesystem-based secret injection via `pipeline.secrets.env_dir`.
|
## Maintained Examples
|
||||||
|
|
||||||
Behavior:
|
|
||||||
|
|
||||||
- `env_dir` may be absolute or relative.
|
|
||||||
- relative `env_dir` resolves from current working directory.
|
|
||||||
- files with valid env-var names (`[A-Za-z_][A-Za-z0-9_]*`) are loaded.
|
|
||||||
- values are loaded from file contents with trailing newline trimming.
|
|
||||||
- existing process env vars are preserved.
|
|
||||||
- invalid names and subdirectories are skipped.
|
|
||||||
- missing/unreadable `env_dir` fails command execution.
|
|
||||||
|
|
||||||
Guidance:
|
|
||||||
|
|
||||||
- do not put secret values directly in YAML.
|
|
||||||
- configure env var names in config and provide values via env/secrets files.
|
|
||||||
|
|
||||||
## 10. Examples
|
|
||||||
|
|
||||||
Maintained examples:
|
|
||||||
|
|
||||||
- `examples/pipeline.minimal.yml`
|
- `examples/pipeline.minimal.yml`
|
||||||
- `examples/pipeline.production.yml`
|
- `examples/pipeline.production.yml`
|
||||||
- `examples/pipeline.full.annotated.yml`
|
- `examples/pipeline.full.annotated.yml`
|
||||||
- `examples/campaigns/sample-campaign/campaign.yml`
|
- `examples/campaigns/sample-campaign/campaign.yml`
|
||||||
- `examples/campaigns/sample-campaign/speakers.yml`
|
|
||||||
- `examples/campaigns/sample-campaign/autocorrect.yml`
|
|
||||||
- `examples/campaigns/sample-campaign/glossary.yml`
|
|
||||||
- `examples/campaigns/sample-campaign/session.template.yml`
|
|
||||||
- `examples/session.local-audio.yml`
|
- `examples/session.local-audio.yml`
|
||||||
- `examples/session.s3-audio.yml`
|
- `examples/session.s3-audio.yml`
|
||||||
|
- `examples/session.template.yml`
|
||||||
These examples are validated by `internal/config` tests.
|
|
||||||
|
|||||||
@@ -1,15 +1,19 @@
|
|||||||
# Integration Documentation Index
|
# Integrations Index
|
||||||
|
|
||||||
## Audience
|
## Audience
|
||||||
Developers and LLM coding agents changing Narratio's external integration contracts.
|
Developers and coding agents changing Narratio's external integration boundaries.
|
||||||
|
|
||||||
## Scope
|
## Scope
|
||||||
Implemented-only reference notes for the external systems Narratio currently integrates with.
|
`docs/integrations/` is the implementation-level reference for downstream tool adapter contracts.
|
||||||
|
|
||||||
## Integration Docs
|
These docs cover what Narratio expects from external tools and what each adapter guarantees back to stage code.
|
||||||
- `audita.md`: Audita adapter invocation and validation contract.
|
|
||||||
- `seriatim.md`: Seriatim normalize/merge/trim adapter contract.
|
|
||||||
- `scriptorium.md`: Scriptorium run/render adapter contract.
|
|
||||||
|
|
||||||
## Canonical Owner
|
## Integration Contracts
|
||||||
`docs/integrations/` is the canonical home for external integration reference notes per `docs/documentation/policy.md`.
|
- `audita.md`: transcript polishing adapter (`audita process`).
|
||||||
|
- `seriatim.md`: merge/normalize/trim/render adapter (`seriatim`).
|
||||||
|
- `scriptorium.md`: artifact run/render adapter (`scriptorium run|render`).
|
||||||
|
|
||||||
|
## Related Canonical Docs
|
||||||
|
- `docs/config.md`: operator-facing configuration reference.
|
||||||
|
- `docs/internal/adapters.md`: shared adapter boundary and runner wiring.
|
||||||
|
- `docs/internal/stage-*.md`: stage-specific integration usage.
|
||||||
|
|||||||
@@ -1,66 +1,60 @@
|
|||||||
# Integration: audita
|
# Integration: Audita
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define Narratio's adapter contract for transcript polishing via Audita CLI subprocess execution.
|
Define the Audita adapter contract used by the `polish` stage.
|
||||||
|
|
||||||
## Inputs and Outputs
|
## Adapter Boundary
|
||||||
Inputs (`audita.PolishRequest`):
|
Interface:
|
||||||
- base transcript path
|
- `audita.Runner`
|
||||||
- glossary path
|
- method: `Run(ctx, PolishRequest) (PolishResult, error)`
|
||||||
- output polished transcript path
|
|
||||||
- optional report path (required when report enabled)
|
|
||||||
- work dir
|
|
||||||
- generated config path
|
|
||||||
- stdout/stderr log paths
|
|
||||||
- optional module/model/base URL and concurrency knobs
|
|
||||||
|
|
||||||
Outputs (`audita.PolishResult`):
|
Primary implementation:
|
||||||
- polished transcript path
|
- `internal/adapters/audita/SubprocessRunner`
|
||||||
- optional report path
|
|
||||||
- generated config path
|
|
||||||
- stdout/stderr log paths
|
|
||||||
- exit code, duration, invoked binary
|
|
||||||
- adapter metadata
|
|
||||||
|
|
||||||
## Boundaries
|
Execution mode:
|
||||||
Owns:
|
- subprocess invocation of `audita process`
|
||||||
- Deterministic CLI argument construction for `audita process`
|
|
||||||
- Environment bridging for API credentials
|
|
||||||
- Invocation config emission
|
|
||||||
- Output validation for polished transcript and report
|
|
||||||
|
|
||||||
Does not own:
|
## Request Contract
|
||||||
- Upstream/downstream stage orchestration
|
`PolishRequest` carries:
|
||||||
- Credential sourcing policy beyond required env-var presence check
|
- required transcript/glossary/output/work-dir paths;
|
||||||
|
- optional report path (required when report mode is enabled);
|
||||||
|
- generated config and stdout/stderr log paths;
|
||||||
|
- optional module/model/base-url/config/output-schema/concurrency settings.
|
||||||
|
|
||||||
## Config Fields Used
|
## Result Contract
|
||||||
Via `pipeline.audita.*` mapped in app/stage wiring:
|
`PolishResult` returns:
|
||||||
- `binary`, `timeout`, `llm_api_key_env`, `modules`, `base_url`, `model`
|
- processed transcript path;
|
||||||
- `transcript_description`, `config_path`, `output_schema`, `work_dir_retention`
|
- optional report path;
|
||||||
- `total_llm_concurrency`, `proposal_llm_concurrency`, `validation_model`, `validation_llm_concurrency`, `report`
|
- work dir and generated-config/log paths;
|
||||||
|
- exit code, duration, binary provenance;
|
||||||
|
- adapter metadata map.
|
||||||
|
|
||||||
## External Adapters Used
|
## Validation and Failure Semantics
|
||||||
- Shared subprocess helper (`internal/adapters/subprocess`) to run CLI and capture logs.
|
Construction fails for invalid static config values, including:
|
||||||
|
- empty binary;
|
||||||
|
- non-positive timeout;
|
||||||
|
- invalid base URL;
|
||||||
|
- invalid output schema;
|
||||||
|
- invalid work-dir retention value;
|
||||||
|
- invalid concurrency values.
|
||||||
|
|
||||||
## State and Manifest Behavior
|
Run fails for:
|
||||||
- No direct manifest writes.
|
- missing required request paths;
|
||||||
- Stage-level metadata records adapter provenance and credential-present signal.
|
- missing required credential env var when configured (`llm_api_key_env`);
|
||||||
- Generated invocation YAML is written when `GeneratedConfigPath` is provided.
|
- subprocess execution failure;
|
||||||
|
- invalid processed transcript JSON (`segments` array required);
|
||||||
|
- invalid report JSON when reporting is enabled.
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
Failure results still include output/log/config/exit metadata for diagnostics.
|
||||||
- Adapter has no skip/resume logic. Stage/runner controls this.
|
|
||||||
|
|
||||||
## Failure Behavior
|
## Deterministic Behavior
|
||||||
- Constructor validation fails on invalid binary/timeout/schema/concurrency/URL values.
|
- CLI args are built from runner config + request in a fixed order.
|
||||||
- Run fails on missing required paths, missing required credential env var, subprocess errors, invalid polished JSON shape, or invalid report JSON.
|
- Generated invocation YAML (`audita.generated.v1`) is emitted when requested.
|
||||||
- Failures preserve stdout/stderr paths in returned result metadata.
|
- Manifest writes are stage-owned; adapter itself is stateless.
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
## Config Mapping
|
||||||
- `internal/adapters/audita/subprocess_test.go`
|
Config fields consumed through runner/stage wiring are under `pipeline.audita.*`.
|
||||||
- `internal/adapters/audita/fake_test.go`
|
|
||||||
- `internal/stage/polish_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
Maintained example with Audita config:
|
||||||
- Polished output must be valid JSON with top-level `segments` array.
|
- `examples/pipeline.full.annotated.yml`
|
||||||
- When report is enabled, report output must be valid JSON.
|
- `examples/pipeline.production.yml`
|
||||||
- If `llm_api_key_env` is configured, credential must be present in environment.
|
|
||||||
|
|||||||
@@ -1,64 +1,66 @@
|
|||||||
# Integration: scriptorium
|
# Integration: Scriptorium
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define Narratio's adapter contract for Scriptorium artifact generation and render-debug subprocess invocations.
|
Define the Scriptorium adapter contract used by `analyze` and trim-bounds generation in `trim`.
|
||||||
|
|
||||||
## Inputs and Outputs
|
## Adapter Boundary
|
||||||
Inputs:
|
Interface:
|
||||||
- `RunArtifactRequest`: binary, config path, prompt/profile IDs, input map, vars map, timeout, output path, logs/config paths, optional API env and working dir
|
- `scriptorium.Runner`
|
||||||
- `RenderArtifactRequest`: same core fields for render mode
|
- methods:
|
||||||
|
- `RunArtifact(ctx, RunArtifactRequest)`
|
||||||
|
- `RenderArtifact(ctx, RenderArtifactRequest)`
|
||||||
|
|
||||||
Outputs (`ArtifactResult`):
|
Primary implementation:
|
||||||
- output path
|
- `internal/adapters/scriptorium/SubprocessRunner`
|
||||||
- stdout/stderr log paths
|
|
||||||
- generated config path
|
|
||||||
- exit code and duration
|
|
||||||
- command mode (`run` or `render`)
|
|
||||||
- prompt/profile provenance
|
|
||||||
- validation failure signal
|
|
||||||
- adapter metadata
|
|
||||||
|
|
||||||
## Boundaries
|
Execution modes:
|
||||||
Owns:
|
- `scriptorium run`
|
||||||
- Deterministic CLI arg construction for `scriptorium run` and `scriptorium render`
|
- `scriptorium render`
|
||||||
- Common request validation
|
|
||||||
- Invocation config emission
|
|
||||||
- Output existence/non-empty checks
|
|
||||||
- Validation-failure mapping for run exit code 2
|
|
||||||
|
|
||||||
Does not own:
|
## Request Contract
|
||||||
- Artifact selection policy (`analyze` stage)
|
Both request types carry:
|
||||||
- Bounds semantic validation (`trim` stage)
|
- binary/config/prompt/profile IDs;
|
||||||
|
- input map and vars map;
|
||||||
|
- output path;
|
||||||
|
- timeout;
|
||||||
|
- generated config + stdout/stderr log paths;
|
||||||
|
- optional API-key env var name;
|
||||||
|
- optional working directory.
|
||||||
|
|
||||||
## Config Fields Used
|
## Result Contract
|
||||||
Via `pipeline.scriptorium.*` and stage-level artifact config:
|
`ArtifactResult` returns:
|
||||||
- `binary`, `config_path`, `timeout`, `render_debug`
|
- output/log/generated-config paths;
|
||||||
- artifact-level `prompt_id`, `profile_id`, `timeout`, `inputs`, `vars`, `output_path`
|
- exit code and duration;
|
||||||
|
- command mode (`run` or `render`);
|
||||||
|
- prompt/profile provenance;
|
||||||
|
- `ValidationFailed` marker;
|
||||||
|
- metadata map.
|
||||||
|
|
||||||
## External Adapters Used
|
## Validation and Failure Semantics
|
||||||
- Shared subprocess helper (`internal/adapters/subprocess`).
|
Request validation fails for:
|
||||||
|
- missing binary, prompt id, or output path;
|
||||||
|
- non-positive timeout;
|
||||||
|
- empty input/var names;
|
||||||
|
- empty input path values;
|
||||||
|
- missing required credential env var when `APIKeyEnv` is set.
|
||||||
|
|
||||||
## State and Manifest Behavior
|
Run behavior:
|
||||||
- No direct manifest writes.
|
- subprocess errors propagate with context;
|
||||||
- Stage metadata records adapter outputs and command mode.
|
- `run` exit code `2` is mapped to `ValidationFailed=true`;
|
||||||
- Generated invocation YAML is written when requested.
|
- successful subprocess still fails if output file is missing or empty.
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
Render behavior:
|
||||||
- Adapter has no skip/resume logic. Stage/runner controls execution.
|
- subprocess errors propagate;
|
||||||
|
- output file must exist and be non-empty.
|
||||||
|
|
||||||
## Failure Behavior
|
## Deterministic Behavior
|
||||||
- Request validation fails for missing binary/prompt/output, invalid timeout, invalid input/var names, or missing required API env var.
|
- input and var maps are sorted into deterministic `--input` and `--var` CLI args.
|
||||||
- Subprocess errors bubble with command context.
|
- generated invocation YAML (`scriptorium.generated.v1`) is emitted when requested.
|
||||||
- `run` exit code 2 is treated as `ValidationFailed=true` and surfaced as error by calling stage.
|
- adapter is stateless and does not own artifact-selection policy.
|
||||||
- Successful subprocess still fails if output file is missing/empty.
|
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
## Config Mapping
|
||||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
Config fields consumed through runner/stage wiring are under `pipeline.scriptorium.*` plus per-artifact settings under `pipeline.scriptorium.artifacts.*`.
|
||||||
- `internal/adapters/scriptorium/fake_test.go`
|
|
||||||
- `internal/stage/analyze_test.go`
|
|
||||||
- `internal/stage/trim_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
Maintained examples with Scriptorium config:
|
||||||
- Both modes require explicit timeout > 0.
|
- `examples/pipeline.full.annotated.yml`
|
||||||
- Input/var maps are sorted into deterministic CLI argument order.
|
- `examples/pipeline.production.yml`
|
||||||
- Run-mode validation failures are represented explicitly, not silently skipped.
|
|
||||||
|
|||||||
@@ -1,60 +1,61 @@
|
|||||||
# Integration: seriatim
|
# Integration: Seriatim
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define Narratio's adapter contract for merge, normalize, and trim subprocess invocations of Seriatim.
|
Define the Seriatim adapter contract used by `merge`, `normalize`, `trim`, and `render`.
|
||||||
|
|
||||||
## Inputs and Outputs
|
## Adapter Boundary
|
||||||
Inputs:
|
Interface:
|
||||||
- `MergeRequest`: raw/per-speaker normalized transcript inputs, base output path, optional report, speaker/autocorrect paths, logs/config
|
- `seriatim.Runner`
|
||||||
- `NormalizeRequest`: input transcript, output path, schema, optional report, timeout/log/config
|
- methods:
|
||||||
- `TrimRequest`: input transcript, output path, keep selector, timeout/log/config
|
- `Run(ctx, MergeRequest)`
|
||||||
|
- `Normalize(ctx, NormalizeRequest)`
|
||||||
|
- `Trim(ctx, TrimRequest)`
|
||||||
|
- `Render(ctx, RenderRequest)`
|
||||||
|
|
||||||
Outputs:
|
Primary implementation:
|
||||||
- `MergeResult`, `NormalizeResult`, `TrimResult` with output paths, logs/config paths, exit code, duration, binary provenance, and metadata.
|
- `internal/adapters/seriatim/SubprocessRunner`
|
||||||
|
|
||||||
## Boundaries
|
Execution modes:
|
||||||
Owns:
|
- `seriatim merge`
|
||||||
- Validated deterministic CLI invocation construction
|
- `seriatim normalize`
|
||||||
- Optional env tuning propagation for merge
|
- `seriatim trim`
|
||||||
- Invocation config file emission
|
- `seriatim render`
|
||||||
- JSON output validation
|
|
||||||
|
|
||||||
Does not own:
|
## Request/Result Contracts
|
||||||
- Transcript input selection/promotion logic (stage-owned)
|
- `MergeRequest`/`MergeResult`: multi-input merge to base transcript, optional report.
|
||||||
- Bounds computation (scriptorium/trim-stage-owned)
|
- `NormalizeRequest`/`NormalizeResult`: transcript normalization with explicit schema.
|
||||||
|
- `TrimRequest`/`TrimResult`: transcript trimming with required keep selector.
|
||||||
|
- `RenderRequest`/`RenderResult`: transcript-to-markdown rendering with explicit format and render booleans.
|
||||||
|
|
||||||
## Config Fields Used
|
Results include output/log/config paths, timing, exit code, and metadata.
|
||||||
Via `pipeline.seriatim.*` mapped in app/stage wiring:
|
|
||||||
- `binary`, `timeout`, `output_schema`, `coalesce_gap`, `report`
|
|
||||||
- `env.overlap_word_run_gap`
|
|
||||||
- `env.overlap_word_run_reorder_window`
|
|
||||||
- `env.backchannel_max_duration`
|
|
||||||
- `env.filler_max_duration`
|
|
||||||
|
|
||||||
## External Adapters Used
|
## Validation and Failure Semantics
|
||||||
- Shared subprocess helper (`internal/adapters/subprocess`).
|
Runner construction validates:
|
||||||
|
- binary presence;
|
||||||
|
- timeout > 0;
|
||||||
|
- supported output schema (`seriatim-minimal|seriatim-intermediate|seriatim-full`);
|
||||||
|
- non-negative coalesce gap.
|
||||||
|
|
||||||
## State and Manifest Behavior
|
Invocation fails on:
|
||||||
- No direct manifest writes.
|
- missing required request paths/inputs;
|
||||||
- Stage metadata consumes adapter result fields and preserves generated config/log references.
|
- invalid normalize schema override;
|
||||||
|
- unsupported render format;
|
||||||
|
- subprocess failure;
|
||||||
|
- invalid JSON outputs for merge/normalize/trim;
|
||||||
|
- missing `segments` array for normalize/trim transcript outputs;
|
||||||
|
- empty render output files.
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
When report paths are provided/enabled, report files must parse as JSON.
|
||||||
- Adapter has no skip/resume logic. Runner controls stage execution.
|
|
||||||
|
|
||||||
## Failure Behavior
|
## Deterministic Behavior
|
||||||
- Constructor fails for invalid binary/timeout/output-schema/coalesce-gap.
|
- argument ordering is deterministic per command construction.
|
||||||
- Merge fails on missing output path/inputs/report path (if enabled), subprocess errors, invalid merged output JSON, invalid report JSON.
|
- merge env overrides are explicit (`SERIATIM_*`) and only emitted when configured.
|
||||||
- Normalize fails on missing input/output, invalid schema, subprocess errors, invalid final output JSON shape, invalid report JSON.
|
- generated invocation YAML (`seriatim.generated.v1`) is emitted when requested.
|
||||||
- Trim fails on missing input/output/keep selector, subprocess errors, invalid final-trimmed output JSON shape.
|
- adapter does not write manifests or choose stage inputs.
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
## Config Mapping
|
||||||
- `internal/adapters/seriatim/subprocess_test.go`
|
Config fields consumed through runner/stage wiring are under `pipeline.seriatim.*` and `pipeline.render.*`.
|
||||||
- `internal/adapters/seriatim/fake_test.go`
|
|
||||||
- `internal/stage/merge_test.go`
|
|
||||||
- `internal/stage/normalize_test.go`
|
|
||||||
- `internal/stage/trim_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
Maintained examples with Seriatim config:
|
||||||
- Supported output schemas are limited to `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`.
|
- `examples/pipeline.full.annotated.yml`
|
||||||
- Final and final-trimmed outputs must include `segments` arrays.
|
- `examples/pipeline.production.yml`
|
||||||
- Merge/normalize/trim all route through deterministic subprocess invocation.
|
|
||||||
|
|||||||
@@ -1,29 +1,45 @@
|
|||||||
# Internal Documentation Index
|
# Internal Documentation Index
|
||||||
|
|
||||||
## Audience
|
## Audience
|
||||||
Developers and LLM coding agents changing Narratio internals.
|
Developers and coding agents changing Narratio internals.
|
||||||
|
|
||||||
## Scope
|
## Scope
|
||||||
Implementation-accurate contracts for workspace/state, manifests, stages, artifact resolution, adapter boundaries, and restore command behavior.
|
`docs/internal/` documents implemented internal contracts: stage boundaries, manifest/state behavior, artifact resolution, restore behavior, storage boundaries, and workspace invariants.
|
||||||
|
|
||||||
## Component Docs
|
User and operator behavior belongs in:
|
||||||
- `adapters.md`: external adapter map, runtime wiring, and boundary ownership.
|
- `docs/cli.md`
|
||||||
- `storage.md`: remote storage backend contracts and object-store invariants.
|
- `docs/config.md`
|
||||||
- `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics.
|
- `docs/operations.md`
|
||||||
- `artifacts.md`: built-in artifact registry, runtime artifact catalog, and source-resolution behavior.
|
- `docs/troubleshooting.md`
|
||||||
- `workspace.md`: local state model, manifests, run-local layout, promotion, and cleanup invariants.
|
|
||||||
- `command-restore.md`: restore command discovery/planning/execution/reporting contract.
|
|
||||||
- `stage-prepare.md`: input materialization and provenance capture.
|
|
||||||
- `stage-transcribe.md`: WhisperX transcript generation.
|
|
||||||
- `stage-merge.md`: Seriatim normalization + merge.
|
|
||||||
- `stage-polish.md`: Audita transcript polishing.
|
|
||||||
- `stage-normalize.md`: post-polish normalization.
|
|
||||||
- `stage-trim.md`: bounds-driven transcript trimming.
|
|
||||||
- `stage-analyze.md`: dependency-ordered Scriptorium artifact generation for selected configured artifacts.
|
|
||||||
- `stage-archive.md`: archive upload and current-pointer publish contract.
|
|
||||||
|
|
||||||
## External Integration Notes
|
## Pipeline Stage Set
|
||||||
- `../integrations/README.md`: canonical location for external integration contracts (`audita.md`, `seriatim.md`, `scriptorium.md`).
|
Canonical stage order from `internal/stage.All()`:
|
||||||
|
1. `prepare`
|
||||||
|
2. `transcribe`
|
||||||
|
3. `merge`
|
||||||
|
4. `polish`
|
||||||
|
5. `normalize`
|
||||||
|
6. `trim`
|
||||||
|
7. `render`
|
||||||
|
8. `analyze`
|
||||||
|
9. `publish`
|
||||||
|
10. `notify` (placeholder)
|
||||||
|
|
||||||
## Canonical Owner
|
`notify` is currently a placeholder stage with optional notifier call behavior; it has no persisted pipeline outputs.
|
||||||
`docs/internal/` is the canonical home for implemented internals per `docs/documentation/policy.md`.
|
|
||||||
|
## Internal Component Docs
|
||||||
|
- `adapters.md`: external adapter boundaries and default runtime wiring.
|
||||||
|
- `artifacts.md`: canonical source IDs, runtime catalog behavior, and resolution rules.
|
||||||
|
- `manifest.md`: session and run manifest contracts.
|
||||||
|
- `storage.md`: object-store interface and S3 implementation behavior.
|
||||||
|
- `workspace.md`: local session layout, run-local layout, and cleanup guardrails.
|
||||||
|
- `command-restore.md`: restore discovery, planning, execution, and reporting.
|
||||||
|
- `stage-prepare.md`
|
||||||
|
- `stage-transcribe.md`
|
||||||
|
- `stage-merge.md`
|
||||||
|
- `stage-polish.md`
|
||||||
|
- `stage-normalize.md`
|
||||||
|
- `stage-trim.md`
|
||||||
|
- `stage-render.md`
|
||||||
|
- `stage-analyze.md`
|
||||||
|
- `stage-publish.md`
|
||||||
|
|||||||
@@ -1,38 +1,12 @@
|
|||||||
# Internal: Adapters
|
# Internal: Adapters
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Describe the external adapter boundaries used by Narratio stages and app orchestration, including default runtime wiring.
|
Define external integration boundaries and default adapter wiring used by app/stage orchestration.
|
||||||
|
|
||||||
## Inputs and outputs
|
## Adapter Boundaries
|
||||||
Inputs:
|
Narratio stage logic depends on adapter interfaces, not transport-specific details.
|
||||||
- Stage requests passed through adapter interfaces (for example transcription, merge/normalize/trim, polish, artifact generation, object-store operations, notifications).
|
|
||||||
- Resolved config values used to construct default adapters.
|
|
||||||
|
|
||||||
Outputs:
|
Primary adapters:
|
||||||
- Adapter-specific result structs (paths, metadata, status/attempt info, duration/exit details).
|
|
||||||
- Adapter errors returned to stage/app orchestration.
|
|
||||||
|
|
||||||
## Boundaries
|
|
||||||
Owns:
|
|
||||||
- Transport/process/SDK details at system boundaries (`HTTP`, subprocess CLI invocation, AWS SDK calls).
|
|
||||||
- Request/response contracts in `internal/adapters/*` packages.
|
|
||||||
|
|
||||||
Does not own:
|
|
||||||
- Stage sequencing, skip/force/resume decisions.
|
|
||||||
- Manifest transition logic.
|
|
||||||
- Canonical workspace path policy.
|
|
||||||
|
|
||||||
## Config fields used
|
|
||||||
Default wiring and adapter calls consume:
|
|
||||||
- `pipeline.whisperx.*`
|
|
||||||
- `pipeline.seriatim.*`
|
|
||||||
- `pipeline.audita.*`
|
|
||||||
- `pipeline.scriptorium.*`
|
|
||||||
- `pipeline.storage.*` and `pipeline.archive.*` (object-store construction/gating)
|
|
||||||
- `pipeline.notification.*` (sender boundary exists; placeholder behavior today)
|
|
||||||
|
|
||||||
## External adapters used
|
|
||||||
Runtime env boundary fields (`internal/stage.Env`):
|
|
||||||
- `whisperx.Client`
|
- `whisperx.Client`
|
||||||
- `seriatim.Runner`
|
- `seriatim.Runner`
|
||||||
- `audita.Runner`
|
- `audita.Runner`
|
||||||
@@ -40,39 +14,36 @@ Runtime env boundary fields (`internal/stage.Env`):
|
|||||||
- `storage.ObjectStore`
|
- `storage.ObjectStore`
|
||||||
- `notify.Sender`
|
- `notify.Sender`
|
||||||
|
|
||||||
Current execution usage:
|
## Ownership
|
||||||
- Actively used by implemented stages: `WhisperX`, `Seriatim`, `Audita`, `Scriptorium`, `ObjectStore`, `Notifier`.
|
Adapters own:
|
||||||
- Present but not used by implemented stage set: legacy `storage.Backend`.
|
- HTTP/subprocess/SDK argument and transport details.
|
||||||
|
- Backend-specific request/response mapping.
|
||||||
|
|
||||||
Default construction in app runner:
|
Adapters do not own:
|
||||||
- Auto-constructed when not injected: WhisperX HTTP client, Seriatim subprocess runner, Audita subprocess runner, Scriptorium subprocess runner, object store (only when needed), and `notify.NoopSender`.
|
- stage ordering/skip/force logic;
|
||||||
- Object-store construction goes through app command orchestration so configured filesystem secrets are loaded before the storage adapter is initialized.
|
- manifest transitions;
|
||||||
- Callers can inject test/fake implementations through `app.RunOptions.Env`.
|
- canonical path policy.
|
||||||
|
|
||||||
## State and manifest behavior
|
## Default Wiring
|
||||||
- Adapters do not directly mutate session/run manifests.
|
`internal/app/runner.go` initializes default adapters when not injected:
|
||||||
- Stages and runner own manifest writes and stage status transitions.
|
- WhisperX HTTP client from pipeline config.
|
||||||
- Adapter outputs are persisted indirectly through stage result mapping (outputs/logs/generated configs/metadata).
|
- Seriatim subprocess runner.
|
||||||
|
- Audita subprocess runner.
|
||||||
|
- Scriptorium subprocess runner.
|
||||||
|
- Noop notifier (`notify.NoopSender`).
|
||||||
|
- Object store only when required by selected stages/config.
|
||||||
|
|
||||||
## Skip and resume behavior
|
Object-store construction goes through `newCommandObjectStore`, which loads configured filesystem secrets before adapter initialization.
|
||||||
- No adapter-level skip/resume semantics.
|
|
||||||
- Skip/resume/force behavior is decided by app runner using manifest stage state.
|
|
||||||
|
|
||||||
## Failure behavior
|
## Failure Semantics
|
||||||
- Adapter constructors validate config-derived values and fail early on invalid required inputs.
|
- Constructor errors fail stage execution setup early.
|
||||||
- Adapter run-time failures are returned to stage code with boundary context and are recorded as stage failures by runner logic.
|
- Runtime adapter errors propagate to stage code and then manifest failure handling.
|
||||||
- Subprocess adapters preserve stdout/stderr and generated-config paths to aid diagnosis.
|
- Subprocess adapters persist stage logs/generated configs through stage-managed paths.
|
||||||
|
|
||||||
## Tests to inspect before changing
|
## Test Surfaces
|
||||||
- `internal/adapters/whisperx/http_test.go`
|
- `internal/adapters/whisperx/http_test.go`
|
||||||
- `internal/adapters/seriatim/subprocess_test.go`
|
- `internal/adapters/seriatim/subprocess_test.go`
|
||||||
- `internal/adapters/audita/subprocess_test.go`
|
- `internal/adapters/audita/subprocess_test.go`
|
||||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||||
- `internal/adapters/storage/*_test.go`
|
- `internal/adapters/storage/*_test.go`
|
||||||
- `internal/adapters/notify/fake_test.go`
|
|
||||||
- `internal/app/runner_test.go`
|
- `internal/app/runner_test.go`
|
||||||
|
|
||||||
## Architectural invariants
|
|
||||||
- Stage code depends on adapter interfaces, not transport-specific implementation types.
|
|
||||||
- External SDK-specific types remain inside adapter implementations.
|
|
||||||
- Default app wiring must remain deterministic and overrideable via injected env dependencies.
|
|
||||||
|
|||||||
@@ -1,106 +1,115 @@
|
|||||||
# Internal: Artifacts
|
# Internal: Artifacts
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define Narratio artifact identity, catalog, and source-resolution behavior for:
|
Define canonical artifact IDs, runtime catalog behavior, source resolution rules, and shared current-state mechanics used by app and previous-cache code.
|
||||||
- built-in session artifacts;
|
|
||||||
- configured analyze artifacts;
|
|
||||||
- canonical previous-session artifact sources.
|
|
||||||
|
|
||||||
## Inputs and outputs
|
## Built-in Source IDs
|
||||||
Inputs:
|
|
||||||
- configured input sources (`pipeline.scriptorium.artifacts.*.inputs.*.source`);
|
|
||||||
- session paths and manifest inputs/outputs;
|
|
||||||
- runtime catalog state.
|
|
||||||
|
|
||||||
Outputs:
|
- `narratio.transcript.base` -> `transcripts/base.json` (`merge`)
|
||||||
- resolved artifact path + provenance (`ResolvedSessionArtifact`);
|
- `narratio.transcript.polished` -> `transcripts/polished.json` (`polish`)
|
||||||
- runtime catalog entries for built-ins and configured artifacts;
|
- `narratio.transcript.final` -> `transcripts/final.json` (`normalize`)
|
||||||
- requirement sets for canonical previous-session inputs.
|
- `narratio.transcript.final_trimmed` -> `transcripts/final.trimmed.json` (`trim`)
|
||||||
- canonical S3 session, run, current, session config, session locks, audio, and promoted artifact keys.
|
- `narratio.transcript.final_markdown` -> `transcripts/final.md` (`render`)
|
||||||
|
- `narratio.transcript.final_trimmed_markdown` -> `transcripts/final.trimmed.md` (`render`)
|
||||||
|
- `narratio.bounds.session` -> `artifacts/session_bounds.json` (`trim`)
|
||||||
|
|
||||||
## Boundaries
|
## Configured and Previous-Session Sources
|
||||||
Owns:
|
|
||||||
- built-in source registry and validation;
|
|
||||||
- configured artifact catalog identity (`narratio.artifact.<name>`);
|
|
||||||
- canonical previous-session source parsing and resolution;
|
|
||||||
- previous-session requirement collection (`CollectPreviousArtifactRequirements`).
|
|
||||||
|
|
||||||
Does not own:
|
- configured source ID format: `narratio.artifact.<artifact_key>`
|
||||||
- prepare-stage remote hydration;
|
- previous-session source ID format: `narratio.previous_session.artifact.<artifact_key>`
|
||||||
- stage success/skip transitions;
|
|
||||||
- archive upload orchestration.
|
|
||||||
|
|
||||||
## Built-in IDs
|
Both formats are validated by strict source-policy rules.
|
||||||
| Artifact ID | Canonical file | Producer stage | Output kind |
|
|
||||||
| --- | --- | --- | --- |
|
|
||||||
| `narratio.transcript.base` | `transcripts/base.json` | `merge` | `transcript_base` |
|
|
||||||
| `narratio.transcript.polished` | `transcripts/polished.json` | `polish` | `transcript_polished` |
|
|
||||||
| `narratio.transcript.final` | `transcripts/final.json` | `normalize` | `transcript_final` |
|
|
||||||
| `narratio.transcript.final_trimmed` | `transcripts/final.trimmed.json` | `trim` | `transcript_final_trimmed` |
|
|
||||||
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` |
|
|
||||||
|
|
||||||
## Source families
|
## Runtime Catalog
|
||||||
- built-in: `narratio.transcript.*`, `narratio.bounds.session`
|
|
||||||
- configured artifact: `narratio.artifact.<artifact_key>`
|
|
||||||
- canonical previous-session artifact: `narratio.previous_session.artifact.<artifact_key>`
|
|
||||||
|
|
||||||
## S3 key helpers
|
`ArtifactCatalog` tracks:
|
||||||
- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
|
||||||
- session config: `{session_prefix}/session.yml`
|
|
||||||
- session lock store: `{session_prefix}/locks.yml`
|
|
||||||
- run prefix: `{session_prefix}/runs/{run_id}/`
|
|
||||||
- audio prefix: `{session_prefix}/{session.inputs.audio_s3.prefix}`
|
|
||||||
- current manifest: `{session_prefix}/current/manifest.json`
|
|
||||||
- current run pointer: `{session_prefix}/current/run_id.txt`
|
|
||||||
|
|
||||||
## Runtime catalog model
|
- `planned`: source registered for run context;
|
||||||
Catalog entries track:
|
- `executable`: selected and enabled for analyze execution;
|
||||||
- `planned`: source is registered for this run;
|
- `available`: local file exists and validates;
|
||||||
- `executable`: configured artifact is selected for analyze execution;
|
- `provenance`: availability source.
|
||||||
- `available`: usable local file exists (generated this run or reused from disk).
|
|
||||||
|
Current provenance values:
|
||||||
|
|
||||||
Configured artifact provenance values include:
|
|
||||||
- `generated.current_analyze_run`
|
- `generated.current_analyze_run`
|
||||||
- `filesystem.disabled_artifact_output`
|
- `filesystem.disabled_artifact_output`
|
||||||
|
|
||||||
Previous-session canonical provenance values include:
|
|
||||||
- `manifest.inputs.previous_cache`
|
- `manifest.inputs.previous_cache`
|
||||||
- `current_session.previous_cache`
|
- `current_session.previous_cache`
|
||||||
|
|
||||||
## Resolution behavior
|
## Resolution Rules
|
||||||
- Built-ins resolve via manifest producer outputs first, then canonical fallback paths.
|
|
||||||
- Configured `narratio.artifact.<name>` sources resolve through catalog availability.
|
|
||||||
- Canonical previous-session sources resolve to current-session `previous/` cache candidates derived from configured artifact canonical output paths.
|
|
||||||
- Archive-relative configured artifact paths under `artifacts/` are cached without a redundant nested `artifacts/` segment.
|
|
||||||
- Previous-session canonical resolution prefers manifest-recorded input paths when present, then filesystem fallback under `previous/artifacts/**`.
|
|
||||||
|
|
||||||
## Previous-session requirement scanning
|
Built-ins:
|
||||||
`CollectPreviousArtifactRequirements`:
|
|
||||||
- scans enabled configured artifacts only;
|
|
||||||
- includes canonical previous-session sources only;
|
|
||||||
- deduplicates by artifact key;
|
|
||||||
- merges required/optional references (`required` wins);
|
|
||||||
- records deterministic sorted source locations for diagnostics.
|
|
||||||
|
|
||||||
## Validation behavior
|
1. manifest producer outputs (when present)
|
||||||
- transcript built-ins: JSON with top-level `segments` array;
|
2. canonical session-path fallback
|
||||||
|
|
||||||
|
Configured sources (`narratio.artifact.*`):
|
||||||
|
|
||||||
|
- resolve only through runtime catalog availability.
|
||||||
|
|
||||||
|
Previous-session sources (`narratio.previous_session.artifact.*`):
|
||||||
|
|
||||||
|
- resolve only from local `previous/` cache state;
|
||||||
|
- prefer manifest-backed previous-input paths;
|
||||||
|
- fallback to existing previous-cache filesystem paths.
|
||||||
|
|
||||||
|
Validation by content type:
|
||||||
|
|
||||||
|
- transcript JSON built-ins: JSON with top-level `segments` array;
|
||||||
|
- transcript Markdown built-ins: non-empty text file;
|
||||||
- bounds built-in: valid JSON;
|
- bounds built-in: valid JSON;
|
||||||
- configured and previous-session artifact files: non-empty text content.
|
- configured/previous-session artifact files: non-empty text file.
|
||||||
|
|
||||||
## Failure behavior
|
## Previous Requirement Collection
|
||||||
- unsupported source or malformed canonical previous source: validation/resolution error;
|
|
||||||
- known source unavailable: `ErrSessionArtifactNotFound`;
|
|
||||||
- configured/previous canonical source without catalog: error;
|
|
||||||
- resolved invalid file content: validation error.
|
|
||||||
|
|
||||||
## Tests to inspect before changing
|
`CollectPreviousArtifactRequirements`:
|
||||||
- `internal/artifacts/artifact_resolver_test.go`
|
|
||||||
- `internal/artifacts/catalog_test.go`
|
|
||||||
- `internal/artifacts/previous_requirements_test.go`
|
|
||||||
- `internal/stage/prepare_previous_test.go`
|
|
||||||
- `internal/stage/analyze_test.go`
|
|
||||||
|
|
||||||
## Architectural invariants
|
- scans enabled configured artifacts only;
|
||||||
- Built-in source IDs are static.
|
- extracts only canonical previous-session sources;
|
||||||
- Configured and previous-session source IDs are artifact-key based and validation-gated.
|
- deduplicates by artifact key;
|
||||||
- Resolution behavior remains deterministic and manifest-aware.
|
- merges required and optional references (required wins);
|
||||||
|
- returns deterministic ordering and source locations.
|
||||||
|
|
||||||
|
## Current-State Helpers
|
||||||
|
|
||||||
|
Artifacts package owns shared remote current-state loading mechanics used by restore, status/validate checks, and previous-cache planning.
|
||||||
|
|
||||||
|
Core helpers:
|
||||||
|
|
||||||
|
- `LoadCurrentRunPointer`
|
||||||
|
- `LoadCurrentManifest`
|
||||||
|
- `LoadCurrentState`
|
||||||
|
- `ValidateCurrentStateIdentity`
|
||||||
|
|
||||||
|
Typed missing-state errors:
|
||||||
|
|
||||||
|
- `CurrentRunPointerMissingError` (`ErrCurrentRunPointerMissing`)
|
||||||
|
- `CurrentManifestMissingError` (`ErrCurrentManifestMissing`)
|
||||||
|
|
||||||
|
Identity validation supports caller-provided expectations:
|
||||||
|
|
||||||
|
- expected campaign;
|
||||||
|
- expected session ID;
|
||||||
|
- expected run ID, or pointer/manifest run-ID consistency check.
|
||||||
|
|
||||||
|
Caller policy is intentionally outside artifacts helpers:
|
||||||
|
|
||||||
|
- some callers fail on missing current state;
|
||||||
|
- some callers downgrade missing state to status/findings;
|
||||||
|
- some callers skip optional behavior when state is missing.
|
||||||
|
|
||||||
|
## Key Path Helpers
|
||||||
|
|
||||||
|
`internal/artifacts/paths.go` and S3-key helpers define canonical helpers for:
|
||||||
|
|
||||||
|
- session/work/run paths;
|
||||||
|
- previous-cache paths;
|
||||||
|
- spool/cache paths;
|
||||||
|
- S3 session/run/current-state key layout.
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- source ID formats are stable contracts;
|
||||||
|
- artifact resolution is deterministic and manifest-aware;
|
||||||
|
- previous-session source resolution in `analyze` is local-only;
|
||||||
|
- remote current-state key construction remains centralized in artifacts helpers.
|
||||||
|
|||||||
@@ -1,106 +1,84 @@
|
|||||||
# Internal: Command Restore
|
# Internal: Command Restore
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define the implemented `narratio session restore` contract: committed remote-state discovery, deterministic plan classification, safe file install semantics, and restore reporting.
|
Define the implemented `narratio session restore` command contract:
|
||||||
|
|
||||||
## Inputs and outputs
|
- committed remote current-state discovery;
|
||||||
Inputs:
|
- deterministic restore planning;
|
||||||
- CLI syntax: `narratio session restore <session_id>`.
|
- safe local install semantics;
|
||||||
- CLI flags: `--config`, `--campaign`, `--campaign-file`, `--session`, `--previous-session-id`, `--dry-run`, `--force`, `--include-audio`.
|
- durable restore reporting.
|
||||||
- Resolved/validated `pipeline.yml` and `session.yml`.
|
|
||||||
- Configured remote object store.
|
|
||||||
- Remote committed current-state markers (`current/run_id.txt`, `current/manifest.json`).
|
|
||||||
|
|
||||||
Outputs:
|
## Discovery Contract
|
||||||
- Dry-run summary to stdout (plan + counts).
|
|
||||||
- Non-dry-run completion summary to stdout.
|
|
||||||
- Local durable session files restored under canonical session root.
|
|
||||||
- Non-dry-run restore report at `reports/restore-latest.json`.
|
|
||||||
|
|
||||||
## Boundaries
|
Restore resolves remote committed state from the session publish current pointers:
|
||||||
Owns:
|
|
||||||
- Restore command flag parsing and command wiring.
|
|
||||||
- Remote current-state discovery and identity validation.
|
|
||||||
- Restore plan construction and conflict classification.
|
|
||||||
- Restore execution for planned downloads.
|
|
||||||
- Restore report model and persistence.
|
|
||||||
|
|
||||||
Does not own:
|
- `current/run_id.txt` (required, non-empty);
|
||||||
- Stage execution orchestration (`run`, `resume`, `run-stage`).
|
- `current/manifest.json` (required, decodable).
|
||||||
- Archive publish behavior (owned by archive stage).
|
|
||||||
- Storage transport implementation details (owned by storage adapters).
|
|
||||||
|
|
||||||
## Config fields used
|
Current-state discovery uses shared artifacts-level mechanics and validates identity against the resolved request config:
|
||||||
- Config/session discovery and templating fields consumed by all commands.
|
|
||||||
- `pipeline.workspace.root` (local restore target root).
|
|
||||||
- `pipeline.storage.*` (remote backend + archive identity derivation).
|
|
||||||
- `pipeline.storage.s3.*` identity components used by archive prefix helpers.
|
|
||||||
- `pipeline.spool.root` for active audio downloads.
|
|
||||||
- `pipeline.cache.root` and `pipeline.cache.s3_audio` for reusable S3 audio cache.
|
|
||||||
- `session.session_id`
|
|
||||||
- `session.campaign`
|
|
||||||
|
|
||||||
## External adapters used
|
- campaign must match;
|
||||||
- `storage.ObjectStore` for `Exists`, `List`, `Download`.
|
- session ID must match.
|
||||||
- `artifacts.Store` (`LocalStore`) for layout and session lock management.
|
|
||||||
- `manifest.LocalStore` for manifest decode/validation and identity checks.
|
|
||||||
|
|
||||||
## State and manifest behavior
|
Restore treats any missing or invalid remote current state as a command error.
|
||||||
- Restore is not a pipeline run and does not create a run manifest.
|
|
||||||
- Restore uses committed remote current state only:
|
|
||||||
- `current/run_id.txt` must exist and be non-empty.
|
|
||||||
- `current/manifest.json` must decode and match requested session/campaign.
|
|
||||||
- Non-dry-run writes restore files to canonical session paths.
|
|
||||||
- With `--include-audio`, restore uses the shared S3 audio cache for `audio/**` objects. Cache hits avoid object downloads; cache misses download through spool, install the work file, and populate cache.
|
|
||||||
- Manifest install behavior:
|
|
||||||
- validated before replacement.
|
|
||||||
- installed last among download actions.
|
|
||||||
- existing local manifest is preserved if restored manifest validation/install fails.
|
|
||||||
- Non-dry-run report persists summary/action status metadata in `reports/restore-latest.json`.
|
|
||||||
|
|
||||||
Restore path scope:
|
## Planning Contract
|
||||||
- includes:
|
|
||||||
- `manifest.json`
|
|
||||||
- `transcripts/**`
|
|
||||||
- `artifacts/**`
|
|
||||||
- `previous/**`
|
|
||||||
- `audio/**` only when `--include-audio` is set
|
|
||||||
- excludes:
|
|
||||||
- `runs/**`
|
|
||||||
- `logs/**`
|
|
||||||
- `reports/**`
|
|
||||||
- `config/**`
|
|
||||||
- `inputs/**`
|
|
||||||
- remote `current/**` pointer files as local restore targets
|
|
||||||
|
|
||||||
## Skip and resume behavior
|
Restore planner action kinds:
|
||||||
- Restore does not participate in stage skip/resume decisions.
|
|
||||||
- Restore provides durable local state so subsequent stage commands can resume or rerun based on restored manifest state.
|
|
||||||
- Audio cache is outside the workspace and is reused across restore and prepare invocations.
|
|
||||||
- Dry-run is read-only and returns plan output only.
|
|
||||||
|
|
||||||
## Failure behavior
|
- `download`;
|
||||||
- Fails when storage backend is unavailable or archive identity cannot be resolved.
|
- `skip_same`;
|
||||||
- Fails when remote current pointer/manifest is missing or invalid.
|
- `conflict`.
|
||||||
- Fails when remote manifest identity mismatches requested campaign/session.
|
|
||||||
- Fails on local conflicts unless `--force` is set.
|
|
||||||
- Fails fast on session lock acquisition conflict for non-dry-run execution.
|
|
||||||
- On execution failure, previously installed files remain; no rollback is performed.
|
|
||||||
|
|
||||||
## Tests to inspect before changing
|
Planner behavior:
|
||||||
- `internal/app/restore_test.go`
|
|
||||||
- `internal/app/restore_discovery_test.go`
|
|
||||||
- `internal/app/restore_plan_test.go`
|
|
||||||
- `internal/app/restore_execution_test.go`
|
|
||||||
- `internal/app/restore_workflow_test.go`
|
|
||||||
- `internal/artifacts/archive_identity_test.go`
|
|
||||||
|
|
||||||
## Architectural invariants
|
- remote list scope is the resolved session prefix;
|
||||||
- Restore relies on centralized archive identity/key helpers (`internal/artifacts`) rather than ad hoc key building.
|
- remote-to-local mapping is traversal-safe;
|
||||||
- `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files.
|
- actions are sorted deterministically by local relative path.
|
||||||
- Local path mapping is traversal-safe and constrained to session root.
|
|
||||||
- Restore scope is deterministic and path-classified:
|
Restore scope from current remote state:
|
||||||
- include `manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`
|
|
||||||
- include `audio/**` only with `--include-audio`
|
- include `manifest.json`;
|
||||||
- exclude `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`
|
- include `transcripts/**`;
|
||||||
- Command remains standalone; no implicit `run --restore` behavior.
|
- include `artifacts/**`;
|
||||||
|
- include `audio/**` only with `--include-audio`.
|
||||||
|
|
||||||
|
Explicit exclusions from current remote state mapping:
|
||||||
|
|
||||||
|
- `current/**`;
|
||||||
|
- `runs/**`;
|
||||||
|
- `logs/**`;
|
||||||
|
- `reports/**`;
|
||||||
|
- `config/**`;
|
||||||
|
- `inputs/**`;
|
||||||
|
- `previous/**`.
|
||||||
|
|
||||||
|
Previous-cache files are planned separately through `previouscache.BuildPlan` when configured previous-session requirements exist.
|
||||||
|
|
||||||
|
## Execution Contract
|
||||||
|
|
||||||
|
Execution order and safety:
|
||||||
|
|
||||||
|
- non-manifest downloads happen before manifest install;
|
||||||
|
- `manifest.json` installs last;
|
||||||
|
- downloads use sibling temp files plus atomic rename;
|
||||||
|
- manifest replacement is validated before rename;
|
||||||
|
- failed installs do not roll back files already written in the same execution.
|
||||||
|
|
||||||
|
Audio restore path:
|
||||||
|
|
||||||
|
- uses `audio.MaterializeS3Audio`;
|
||||||
|
- integrates spool and S3 audio cache paths;
|
||||||
|
- supports cache-hit reuse without object redownload.
|
||||||
|
|
||||||
|
## Reporting Contract
|
||||||
|
|
||||||
|
- `--dry-run`: prints summary only; no local writes.
|
||||||
|
- non-dry-run: writes `reports/restore-latest.json`.
|
||||||
|
- report includes plan counts, per-action status, and execution failures.
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- restore uses committed remote current state as authority;
|
||||||
|
- `current/run_id.txt` is the remote publish commit marker;
|
||||||
|
- restore does not execute pipeline stages.
|
||||||
|
|||||||
@@ -1,81 +1,57 @@
|
|||||||
# Internal: Manifest
|
# Internal: Manifest
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Describe Narratio's durable execution state model for session-level and run-level manifests, including lifecycle transitions and persistence behavior.
|
Define durable session state (`manifest.json`) and invocation state (`runs/{run_id}/manifest.json`) contracts.
|
||||||
|
|
||||||
## Inputs and outputs
|
## Session Manifest
|
||||||
Inputs:
|
Path:
|
||||||
- Session identity and run identity from app orchestration.
|
- `{workspace.root}/work/{campaign}/{session_id}/manifest.json`
|
||||||
- Stage transition events and stage result payloads.
|
|
||||||
|
|
||||||
Outputs:
|
Primary model (`manifest.Manifest`):
|
||||||
- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`.
|
- identity (`session_id`, `campaign`, `run_id`)
|
||||||
- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`.
|
- local path metadata (`local_workdir`, `local_spool_dir`)
|
||||||
|
- remote identity metadata (`s3_bucket`, `s3_session_prefix`, `s3_run_prefix`)
|
||||||
|
- `inputs` records
|
||||||
|
- durable `artifacts` records
|
||||||
|
- per-stage `stages` map
|
||||||
|
|
||||||
## Boundaries
|
Stage status enum:
|
||||||
Owns:
|
- `pending`
|
||||||
- Manifest schemas (`Manifest`, `RunManifest`, stage records, error records, input/artifact records).
|
- `running`
|
||||||
- Stage status/action transition methods.
|
- `succeeded`
|
||||||
- Persistent store contract (`manifest.Store`) and local JSON store implementation.
|
- `failed`
|
||||||
|
- `skipped`
|
||||||
|
- `stale`
|
||||||
|
- `interrupted`
|
||||||
|
|
||||||
Does not own:
|
## Run Manifest
|
||||||
- Stage implementation details.
|
Path:
|
||||||
- Path construction policy outside manifest file persistence calls.
|
- `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`
|
||||||
- CLI command behavior.
|
|
||||||
|
|
||||||
## Config fields used
|
Run model (`manifest.RunManifest`):
|
||||||
Manifest package itself does not read config directly.
|
- invocation identity and `force` flag
|
||||||
|
- requested stages
|
||||||
|
- per-stage action (`run` or `skip`)
|
||||||
|
- per-stage status
|
||||||
|
- overall run status (`running`, `succeeded`, `failed`)
|
||||||
|
|
||||||
Manifest identity fields are populated by app/stage orchestration from:
|
## Persistence Semantics
|
||||||
- `session.session_id`
|
`manifest.LocalStore`:
|
||||||
- `session.campaign`
|
- validates loaded documents;
|
||||||
- `pipeline.workspace.root`
|
- normalizes missing maps/stage records;
|
||||||
- `pipeline.storage.s3.*` (when archive/S3 identity is set)
|
- writes atomically via temp file + rename;
|
||||||
|
- updates `updated_at` on save.
|
||||||
|
|
||||||
## External adapters used
|
## Execution Semantics
|
||||||
- No external service adapters.
|
Runner updates both manifests per stage transition:
|
||||||
- Uses local filesystem for persistence via `manifest.LocalStore`.
|
- mark running
|
||||||
|
- mark succeeded/failed/skipped
|
||||||
|
- persist logs/generated config refs and metadata
|
||||||
|
|
||||||
## State and manifest behavior
|
Session manifest is the authoritative stage-progress ledger across invocations.
|
||||||
Session manifest model:
|
Run manifest is invocation-scoped audit state.
|
||||||
- Tracks durable per-session stage state and provenance (`pending`, `running`, `succeeded`, `failed`, `skipped`, `stale`, `interrupted`).
|
|
||||||
- Stores resolved inputs, durable artifacts, stage logs/config refs, and stage metadata.
|
|
||||||
|
|
||||||
Run manifest model:
|
## Invariants
|
||||||
- Tracks one invocation (`run_id`) with requested stages and force mode.
|
- stage resume/skip decisions are session-manifest driven.
|
||||||
- Tracks per-stage action (`run` or `skip`) and per-stage status.
|
- force reruns stale downstream succeeded stages.
|
||||||
- Tracks overall run status (`running`, `succeeded`, `failed`).
|
- run manifest does not replace session manifest as progress authority.
|
||||||
|
|
||||||
Persistence behavior:
|
|
||||||
- Load validates required identity/timestamp fields and normalizes maps/records.
|
|
||||||
- Save updates `updated_at` and writes JSON atomically (temp file + rename).
|
|
||||||
- Session and run manifests are saved incrementally before/after stage transitions.
|
|
||||||
|
|
||||||
Relationship during execution:
|
|
||||||
- Runner updates both manifests for every stage transition.
|
|
||||||
- Session manifest is the durable pipeline-progress ledger.
|
|
||||||
- Run manifest is invocation history and audit record.
|
|
||||||
- Analyze stage outputs are persisted as `kind=scriptorium_artifact` with `source_id=narratio.artifact.<name>` for configured artifact identity.
|
|
||||||
|
|
||||||
## Skip and resume behavior
|
|
||||||
- Resume and skip decisions are based on session-manifest stage statuses.
|
|
||||||
- `--force` reruns selected stages and marks downstream succeeded stages as `stale` in session manifest.
|
|
||||||
- Run manifest records whether each stage was executed or skipped in that invocation.
|
|
||||||
|
|
||||||
## Failure behavior
|
|
||||||
- Stage failure marks both manifests failed for that stage and records error messages/timestamps.
|
|
||||||
- Save failures are returned immediately and fail the command.
|
|
||||||
- Invalid/malformed manifest files fail load with explicit validation/decode errors.
|
|
||||||
|
|
||||||
## Tests to inspect before changing
|
|
||||||
- `internal/manifest/manifest_test.go`
|
|
||||||
- `internal/manifest/run_manifest_test.go`
|
|
||||||
- `internal/manifest/store_test.go`
|
|
||||||
- `internal/app/runner_test.go`
|
|
||||||
- `internal/app/run_control_test.go`
|
|
||||||
- `internal/app/resume_run_stage_test.go`
|
|
||||||
|
|
||||||
## Architectural invariants
|
|
||||||
- Session manifest is authoritative for stage progression across invocations.
|
|
||||||
- Run manifest is invocation-scoped and never replaces session manifest as progress authority.
|
|
||||||
- Manifest writes are atomic and deterministic (JSON + newline, temp rename pattern).
|
|
||||||
|
|||||||
@@ -1,80 +1,39 @@
|
|||||||
# Stage: analyze
|
# Stage: analyze
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Execute selected configured Scriptorium artifacts in deterministic dependency order and promote successful outputs to canonical session artifact paths.
|
Execute selected configured Scriptorium artifacts in dependency order and materialize outputs.
|
||||||
|
|
||||||
## Inputs and outputs
|
## Inputs
|
||||||
Inputs:
|
- configured artifacts from `pipeline.scriptorium.artifacts`
|
||||||
- configured artifact definitions from `pipeline.scriptorium.artifacts`;
|
- optional selected artifact filter (`--artifacts`)
|
||||||
- selected artifact filter (`--artifacts`) when provided;
|
- built-in/configured/previous-session source references in artifact inputs
|
||||||
- resolved artifact sources from resolver/catalog.
|
|
||||||
|
|
||||||
Source types used by analyze:
|
Supported source families:
|
||||||
- built-ins: `narratio.transcript.*`, `narratio.bounds.session`;
|
- built-ins: `narratio.transcript.*`, `narratio.bounds.session`
|
||||||
- configured artifacts: `narratio.artifact.<artifact_key>`;
|
- configured artifacts: `narratio.artifact.<key>`
|
||||||
- canonical previous-session artifacts: `narratio.previous_session.artifact.<artifact_key>`.
|
- previous-session cache: `narratio.previous_session.artifact.<key>`
|
||||||
|
|
||||||
Outputs:
|
## Outputs
|
||||||
- promoted configured artifact files at each configured `output_path`;
|
- one materialized output per executed configured artifact (`output_path`)
|
||||||
- stage metadata (`generated_artifacts`, `reused_artifacts`, selected/order info).
|
- stage metadata describing selected/generated/reused artifacts
|
||||||
|
|
||||||
## Boundaries
|
## Key Behavior
|
||||||
Owns:
|
- skips with metadata when Scriptorium config is missing or no executable artifacts remain.
|
||||||
- runtime artifact catalog construction;
|
- builds runtime artifact catalog (built-ins + configured artifacts).
|
||||||
- selected-artifact planning and dependency ordering;
|
- marks non-executable configured artifacts as reusable when output files already exist.
|
||||||
- per-input resolution and required/optional handling;
|
- validates selected artifact dependency order (cycle-safe topo ordering).
|
||||||
- Scriptorium render/run invocation;
|
- resolves required/optional inputs per artifact source definition.
|
||||||
- run-local output generation and canonical promotion.
|
- resolves previous-session sources from local `previous/` cache only.
|
||||||
|
- runs optional render-debug, then artifact execution.
|
||||||
|
- validates non-empty output files and materializes canonical outputs.
|
||||||
|
|
||||||
Does not own:
|
## Failure Semantics
|
||||||
- prepare-time previous-session hydration;
|
- required missing configured/previous-session inputs fail.
|
||||||
- object-store access for previous-session sources;
|
- missing required previous-session source includes prepare rerun guidance.
|
||||||
- archive promotion policy.
|
- missing required `narratio.transcript.final_markdown` or `narratio.transcript.final_trimmed_markdown` inputs includes render rerun guidance.
|
||||||
|
- dependency cycles or unavailable required dependencies fail.
|
||||||
|
- adapter validation failures fail stage.
|
||||||
|
|
||||||
## Config fields used
|
## Invariants
|
||||||
- `session.session_id`
|
- `analyze` performs no remote storage calls for previous-session source resolution.
|
||||||
- `session.campaign`
|
- output provenance and metadata are deterministic per execution.
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `pipeline.scriptorium.binary`
|
|
||||||
- `pipeline.scriptorium.config_path`
|
|
||||||
- `pipeline.scriptorium.timeout`
|
|
||||||
- `pipeline.scriptorium.render_debug`
|
|
||||||
- `pipeline.scriptorium.artifacts.<name>.*`
|
|
||||||
|
|
||||||
## External adapters used
|
|
||||||
- Scriptorium adapter:
|
|
||||||
- optional `RenderArtifact` when render-debug is enabled;
|
|
||||||
- `RunArtifact` for artifact generation.
|
|
||||||
|
|
||||||
## State and manifest behavior
|
|
||||||
- If Scriptorium config is absent, or no artifacts are executable after filtering, analyze returns success metadata with `skipped=true`.
|
|
||||||
- Builds runtime catalog with built-ins and configured `narratio.artifact.<name>` entries.
|
|
||||||
- Non-executable configured artifacts may still be marked available from existing canonical output files.
|
|
||||||
- Resolves canonical previous-session sources from local prepared `previous/` cache:
|
|
||||||
- prefers manifest-backed previous input paths when present;
|
|
||||||
- may fall back to current-session `previous/` filesystem paths.
|
|
||||||
- Analyze does not call object storage for canonical previous-session source resolution.
|
|
||||||
- Required canonical previous-session input missing:
|
|
||||||
- fails with guidance to run `narratio run-stage --force prepare`.
|
|
||||||
- Optional missing sources are omitted from adapter input paths.
|
|
||||||
|
|
||||||
## Skip and resume behavior
|
|
||||||
- Runner-level skip applies when analyze is already `succeeded` and `--force` is not set.
|
|
||||||
- Analyze is stage-scoped for resume; no per-artifact manifest resume state.
|
|
||||||
- `--artifacts` filters executable artifacts but does not imply force rerun.
|
|
||||||
|
|
||||||
## Failure behavior
|
|
||||||
- Fails on dependency-order violations, missing required inputs, resolver validation failures, adapter errors, and missing/empty generated outputs.
|
|
||||||
- Required unavailable configured artifact source (`narratio.artifact.<name>`) fails before invocation.
|
|
||||||
- Required canonical previous-session source fails with prepare-rerun guidance.
|
|
||||||
|
|
||||||
## Tests to inspect before changing
|
|
||||||
- `internal/stage/analyze_test.go`
|
|
||||||
- `internal/artifacts/catalog_test.go`
|
|
||||||
- `internal/artifacts/artifact_resolver_test.go`
|
|
||||||
- `internal/app/restore_workflow_test.go`
|
|
||||||
|
|
||||||
## Architectural invariants
|
|
||||||
- Canonical previous-session behavior is local-cache only during analyze.
|
|
||||||
- Generated outputs are validated and promoted before stage success is recorded.
|
|
||||||
- Resolver/catalog decisions stay deterministic and validation-gated.
|
|
||||||
|
|||||||
@@ -1,86 +0,0 @@
|
|||||||
# Stage: archive
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
Publish durable run/session state to object storage, then atomically advance remote current state.
|
|
||||||
|
|
||||||
## Inputs and Outputs
|
|
||||||
Inputs:
|
|
||||||
- session manifest and prerequisite stage records
|
|
||||||
- run root contents under `runs/{run_id}/`
|
|
||||||
- promotion rules with artifact `source` IDs and archive `dest` paths (`archive.promote_artifacts`)
|
|
||||||
- effective source-based promotion locks from static config and remote session lock store
|
|
||||||
- session-level `previous/**` cache files when present
|
|
||||||
|
|
||||||
Outputs:
|
|
||||||
- uploaded run files under `{session_prefix}/runs/{run_id}/...`
|
|
||||||
- uploaded promoted artifacts under `{session_prefix}/...`
|
|
||||||
- uploaded session previous-cache files under `{session_prefix}/previous/...` when present
|
|
||||||
- `{session_prefix}/current/manifest.json`
|
|
||||||
- `{session_prefix}/current/run_id.txt` written last
|
|
||||||
|
|
||||||
## Boundaries
|
|
||||||
Owns:
|
|
||||||
- Archive enable/disable gate behavior
|
|
||||||
- Prerequisite stage success enforcement
|
|
||||||
- Run file collection and upload (excluding `audio/`)
|
|
||||||
- Promotion rule resolution and upload
|
|
||||||
- Promotion lock enforcement
|
|
||||||
- Session previous-cache file collection/upload
|
|
||||||
- Commit pointer publish order
|
|
||||||
|
|
||||||
Does not own:
|
|
||||||
- Stage execution before archive
|
|
||||||
- Post-archive local cleanup policy execution (handled by app cleanup logic)
|
|
||||||
|
|
||||||
## Config Fields Used
|
|
||||||
- `pipeline.archive.enabled`
|
|
||||||
- `pipeline.archive.upload_run`
|
|
||||||
- `pipeline.archive.promote_artifacts`
|
|
||||||
- `pipeline.archive.locks`
|
|
||||||
- `{session_prefix}/locks.yml` loaded by app orchestration before archive execution
|
|
||||||
- `pipeline.storage.s3.bucket`
|
|
||||||
- `pipeline.storage.s3.root_prefix`
|
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `session.campaign`
|
|
||||||
- `session.session_id`
|
|
||||||
|
|
||||||
## External Adapters Used
|
|
||||||
- Object storage backend (`env.ObjectStore`) for upload/list primitives.
|
|
||||||
|
|
||||||
## State and Manifest Behavior
|
|
||||||
- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
|
|
||||||
- Resolves bucket/prefix from manifest identity first, then config fallback.
|
|
||||||
- Uploads session `previous/**` files as durable session state when the local `previous/` directory exists.
|
|
||||||
- Skips top-level promotion uploads for effective locked sources; run-local uploads still publish.
|
|
||||||
- When selected configured artifact keys are supplied, skips promotion rules for unselected `narratio.artifact.<key>` sources; built-in transcript and bounds promotions still publish.
|
|
||||||
- Effective locks are the union of `pipeline.archive.locks` and remote `{session_prefix}/locks.yml`; static pipeline locks win on duplicate sources.
|
|
||||||
- Writes metadata including:
|
|
||||||
- upload counts/paths
|
|
||||||
- `previous_files_uploaded` and `previous_uploaded_paths`
|
|
||||||
- `skipped_unselected_promotions`
|
|
||||||
- `locked_promotion_count` and `locked_promotions`
|
|
||||||
- `current_manifest_key`
|
|
||||||
- `current_run_id_key`
|
|
||||||
- `current_pointer_written`
|
|
||||||
- On skipped archive path, returns metadata with `skipped=true` and pointer not written.
|
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
|
||||||
- Stage may self-skip (metadata skip) when archive disabled or run upload disabled.
|
|
||||||
- Runner-level skip also applies for previously succeeded stage unless forced.
|
|
||||||
|
|
||||||
## Failure Behavior
|
|
||||||
- Fails on missing prerequisite success, missing object store when required, missing run root, missing unlocked required promotion source, upload failures, or pointer write failures.
|
|
||||||
- Locked required promotions are intentional skips and do not fail archive.
|
|
||||||
- Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail.
|
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
|
||||||
- `internal/stage/archive_test.go`
|
|
||||||
- `internal/app/post_archive_cleanup_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
|
||||||
- Run upload excludes `audio/` subtree.
|
|
||||||
- Session `previous/**` is archiveable durable input/provenance state, not run-local output.
|
|
||||||
- Ordinary `--force` does not override archive locks.
|
|
||||||
- Malformed or unreadable remote lock store fails archive-capable execution before promotion.
|
|
||||||
- `current/manifest.json` uploads before `current/run_id.txt`.
|
|
||||||
- `current/run_id.txt` is the remote publish commit marker.
|
|
||||||
@@ -1,63 +1,25 @@
|
|||||||
# Stage: merge
|
# Stage: merge
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Normalize per-speaker raw transcripts and merge them into the base transcript via Seriatim.
|
Normalize raw transcript inputs and merge into base transcript via Seriatim.
|
||||||
|
|
||||||
## Inputs and Outputs
|
## Inputs
|
||||||
Inputs:
|
|
||||||
- `transcripts/raw/*.json`
|
- `transcripts/raw/*.json`
|
||||||
- `inputs/speakers.yml`
|
- `inputs/speakers.yml`
|
||||||
- `inputs/autocorrect.yml`
|
- `inputs/autocorrect.yml`
|
||||||
|
|
||||||
Outputs:
|
## Outputs
|
||||||
- `transcripts/base.json`
|
- `transcripts/base.json`
|
||||||
- optional `artifacts/seriatim.report.json` (when report enabled)
|
- optional `artifacts/seriatim.report.json`
|
||||||
|
|
||||||
## Boundaries
|
## Key Behavior
|
||||||
Owns:
|
- discovers and validates raw transcript inputs.
|
||||||
- Raw transcript discovery/validation
|
- normalizes each raw transcript (`seriatim.Normalize`) into run-local scratch output.
|
||||||
- Per-input normalize calls to Seriatim
|
- merges normalized inputs (`seriatim.Run`) into base transcript.
|
||||||
- Final merge call to Seriatim
|
- validates merged transcript and optional report JSON.
|
||||||
- Run-local log/config/report path wiring
|
- materializes canonical outputs and records stage logs/generated configs.
|
||||||
- Promotion of base/report outputs to canonical paths
|
|
||||||
|
|
||||||
Does not own:
|
## Invariants
|
||||||
- Transcript polishing or downstream artifact generation
|
- merge always consumes normalized forms of raw inputs.
|
||||||
|
- base transcript must validate before stage success.
|
||||||
## Config Fields Used
|
- report output is config-gated.
|
||||||
- `session.session_id`
|
|
||||||
- `session.campaign`
|
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `pipeline.seriatim.binary`
|
|
||||||
- `pipeline.seriatim.timeout`
|
|
||||||
- `pipeline.seriatim.output_schema`
|
|
||||||
- `pipeline.seriatim.coalesce_gap`
|
|
||||||
- `pipeline.seriatim.report`
|
|
||||||
- `pipeline.seriatim.env.*`
|
|
||||||
|
|
||||||
## External Adapters Used
|
|
||||||
- Seriatim adapter:
|
|
||||||
- `Normalize` for each raw input
|
|
||||||
- `Run` for final merge
|
|
||||||
|
|
||||||
## State and Manifest Behavior
|
|
||||||
- Reads transcript inputs from transcribe stage outputs in manifest when present; falls back to canonical raw directory.
|
|
||||||
- Writes run-local outputs/logs/config under `runs/{run_id}/merge/...` when enabled.
|
|
||||||
- Promotes canonical base transcript and optional report.
|
|
||||||
- Records normalized-input provenance and adapter metadata in stage metadata.
|
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
|
||||||
- Runner-level skip applies when already succeeded and not forced.
|
|
||||||
- Forced rerun of this or upstream stages can stale downstream succeeded stages via runner invalidation.
|
|
||||||
|
|
||||||
## Failure Behavior
|
|
||||||
- Fails on missing/invalid raw transcripts, missing speakers/autocorrect files, normalize failure, merge failure, invalid base output JSON, or invalid report JSON when enabled.
|
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
|
||||||
- `internal/stage/merge_test.go`
|
|
||||||
- `internal/adapters/seriatim/subprocess_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
|
||||||
- Merge consumes normalized forms of each raw transcript.
|
|
||||||
- Base transcript must validate before promotion.
|
|
||||||
- Report output is optional and gated by config.
|
|
||||||
|
|||||||
@@ -1,56 +1,22 @@
|
|||||||
# Stage: normalize
|
# Stage: normalize
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Normalize the polished transcript into the full final transcript and optionally emit a normalize report.
|
Normalize polished transcript into final transcript using Seriatim.
|
||||||
|
|
||||||
## Inputs and Outputs
|
## Inputs
|
||||||
Inputs:
|
|
||||||
- `transcripts/polished.json`
|
- `transcripts/polished.json`
|
||||||
|
|
||||||
Outputs:
|
## Outputs
|
||||||
- `transcripts/final.json` (or configured normalize output path)
|
- `transcripts/final.json` (or configured normalize output path)
|
||||||
- optional `artifacts/seriatim.normalize.report.json`
|
- optional `artifacts/seriatim.normalize.report.json`
|
||||||
|
|
||||||
## Boundaries
|
## Key Behavior
|
||||||
Owns:
|
- resolves polished transcript from manifest outputs/canonical fallback.
|
||||||
- Polished transcript discovery/validation
|
- applies `pipeline.normalize` config or default normalize config.
|
||||||
- Normalize request construction and invocation
|
- runs Seriatim normalize with configured timeout/binary.
|
||||||
- Optional normalize report wiring
|
- validates normalized transcript and optional report.
|
||||||
- Promotion of final transcript and optional report
|
- materializes canonical outputs and records logs/generated configs.
|
||||||
|
|
||||||
Does not own:
|
## Invariants
|
||||||
- Bounds detection or segment trimming
|
- final transcript must validate as processed transcript JSON (`segments` array).
|
||||||
|
- normalize defaults are applied when `pipeline.normalize` is unset.
|
||||||
## Config Fields Used
|
|
||||||
- `session.session_id`
|
|
||||||
- `session.campaign`
|
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `pipeline.normalize.output_path`
|
|
||||||
- `pipeline.normalize.output_schema`
|
|
||||||
- `pipeline.normalize.report`
|
|
||||||
- `pipeline.seriatim.binary`
|
|
||||||
- `pipeline.seriatim.timeout`
|
|
||||||
|
|
||||||
## External Adapters Used
|
|
||||||
- Seriatim adapter (`Normalize`).
|
|
||||||
|
|
||||||
## State and Manifest Behavior
|
|
||||||
- Reads polished transcript from polish outputs in manifest when present; falls back to canonical path.
|
|
||||||
- Uses run-local output/report/log/config paths when run layout is enabled.
|
|
||||||
- Promotes canonical final transcript and optional normalize report.
|
|
||||||
- Records adapter/result metadata including source path selection.
|
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
|
||||||
- Runner-level skip applies when already succeeded and not forced.
|
|
||||||
- Forced reruns can stale downstream succeeded stages.
|
|
||||||
|
|
||||||
## Failure Behavior
|
|
||||||
- Fails on missing/invalid polished transcript, adapter error, invalid final output, or invalid report output when report enabled.
|
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
|
||||||
- `internal/stage/normalize_test.go`
|
|
||||||
- `internal/adapters/seriatim/subprocess_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
|
||||||
- Final output must validate as transcript-compatible JSON (`segments` array required).
|
|
||||||
- Default normalize config is applied when `pipeline.normalize` is unset.
|
|
||||||
|
|||||||
@@ -1,69 +1,23 @@
|
|||||||
# Stage: polish
|
# Stage: polish
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Polish the base transcript with Audita and produce a polished transcript for downstream normalization/analyze.
|
Run Audita polishing on base transcript and produce polished transcript.
|
||||||
|
|
||||||
## Inputs and Outputs
|
## Inputs
|
||||||
Inputs:
|
|
||||||
- `transcripts/base.json`
|
- `transcripts/base.json`
|
||||||
- `inputs/glossary.yml`
|
- `inputs/glossary.yml`
|
||||||
|
|
||||||
Outputs:
|
## Outputs
|
||||||
- `transcripts/polished.json`
|
- `transcripts/polished.json`
|
||||||
- optional `artifacts/audita.report.json` (when report enabled)
|
- optional `artifacts/audita.report.json`
|
||||||
|
|
||||||
## Boundaries
|
## Key Behavior
|
||||||
Owns:
|
- resolves base transcript from merge outputs/canonical fallback.
|
||||||
- Base transcript discovery/validation
|
- invokes Audita with configured model/module/runtime options.
|
||||||
- Audita invocation request construction
|
- validates processed transcript structure (`segments` array required).
|
||||||
- Run-local logs/config/work-dir/report wiring
|
- validates optional report JSON.
|
||||||
- Promotion of polished transcript and optional report
|
- materializes canonical outputs; records logs/generated config and adapter metadata.
|
||||||
|
|
||||||
Does not own:
|
## Invariants
|
||||||
- Upstream merge normalization
|
- polished transcript schema validation is mandatory.
|
||||||
- Downstream normalize/trim/analyze logic
|
- report output is config-gated.
|
||||||
|
|
||||||
## Config Fields Used
|
|
||||||
- `session.session_id`
|
|
||||||
- `session.campaign`
|
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `pipeline.audita.binary`
|
|
||||||
- `pipeline.audita.timeout`
|
|
||||||
- `pipeline.audita.llm_api_key_env`
|
|
||||||
- `pipeline.audita.modules`
|
|
||||||
- `pipeline.audita.base_url`
|
|
||||||
- `pipeline.audita.model`
|
|
||||||
- `pipeline.audita.transcript_description`
|
|
||||||
- `pipeline.audita.config_path`
|
|
||||||
- `pipeline.audita.output_schema`
|
|
||||||
- `pipeline.audita.work_dir_retention`
|
|
||||||
- `pipeline.audita.total_llm_concurrency`
|
|
||||||
- `pipeline.audita.proposal_llm_concurrency`
|
|
||||||
- `pipeline.audita.validation_model`
|
|
||||||
- `pipeline.audita.validation_llm_concurrency`
|
|
||||||
- `pipeline.audita.report`
|
|
||||||
|
|
||||||
## External Adapters Used
|
|
||||||
- Audita adapter (`env.Audita.Run`).
|
|
||||||
|
|
||||||
## State and Manifest Behavior
|
|
||||||
- Reads base transcript from merge manifest outputs when available; falls back to canonical base path.
|
|
||||||
- Uses run-local output/report/log/config/scratch paths when run layout is enabled.
|
|
||||||
- Promotes canonical `transcripts/polished.json` and optional report.
|
|
||||||
- Records adapter invocation metadata, credential presence signal, and output provenance in stage metadata.
|
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
|
||||||
- Runner-level skip applies when already succeeded and not forced.
|
|
||||||
- Forced rerun can stale downstream succeeded stages via runner invalidation.
|
|
||||||
|
|
||||||
## Failure Behavior
|
|
||||||
- Fails on missing/invalid base transcript, missing glossary, adapter error, invalid polished output shape (`segments` array required), or invalid report JSON when enabled.
|
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
|
||||||
- `internal/stage/polish_test.go`
|
|
||||||
- `internal/adapters/audita/subprocess_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
|
||||||
- Polished transcript must contain a top-level `segments` array.
|
|
||||||
- Report behavior is strictly config-gated.
|
|
||||||
- Stage output canonicalization always ends at `transcripts/polished.json`.
|
|
||||||
|
|||||||
@@ -1,124 +1,42 @@
|
|||||||
# Stage: prepare
|
# Stage: prepare
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Materialize canonical current-session input state and provenance before downstream stages run.
|
Materialize canonical current-session inputs before processing stages.
|
||||||
|
|
||||||
Prepare owns:
|
## Inputs
|
||||||
- local input file materialization (`inputs/**`);
|
- resolved `campaign.yml`, `session.yml`, and pipeline config
|
||||||
- audio input materialization (`audio/**`);
|
- stable input files (`speakers`, `autocorrect`, `glossary`)
|
||||||
- previous-session cache hydration (`previous/**`) for canonical previous-session artifact sources.
|
|
||||||
|
|
||||||
## Inputs and outputs
|
|
||||||
Inputs:
|
|
||||||
- resolved config/campaign/session (`pipeline.yml`, `campaign.yml`, `session.yml`);
|
|
||||||
- remote session provenance when `session.yml` was loaded from S3;
|
|
||||||
- campaign or session input files (`speakers`, `autocorrect`, `glossary`);
|
|
||||||
- audio source:
|
- audio source:
|
||||||
- local: `session.inputs.audio_dir` or `session.inputs.audio_files`;
|
- local `audio_dir`/`audio_files`, or
|
||||||
- S3: `session.inputs.audio_s3.prefix`;
|
- S3 `audio_s3.prefix`
|
||||||
- configured enabled Scriptorium artifact inputs (for previous-session requirement scanning);
|
- enabled configured artifact input requirements for previous-session sources
|
||||||
- remote previous-session current archive state when previous hydration is required.
|
|
||||||
|
|
||||||
Outputs:
|
## Outputs
|
||||||
- `inputs/campaign.yml`;
|
- `inputs/campaign.yml`
|
||||||
- `inputs/session.yml`;
|
- `inputs/session.yml`
|
||||||
- `inputs/pipeline.resolved.yml`;
|
- `inputs/pipeline.resolved.yml`
|
||||||
- `inputs/speakers.yml`;
|
- `inputs/speakers.yml`
|
||||||
- `inputs/autocorrect.yml`;
|
- `inputs/autocorrect.yml`
|
||||||
- `inputs/glossary.yml`;
|
- `inputs/glossary.yml`
|
||||||
- `audio/*.flac` in canonical session `audio/`;
|
- `audio/*.flac`
|
||||||
- optional `previous/manifest.json`;
|
- optional `previous/manifest.json`
|
||||||
- optional `previous/artifacts/**`;
|
- optional `previous/artifacts/**`
|
||||||
- deterministic `manifest.Inputs` records with checksums and provenance metadata.
|
- deterministic `manifest.inputs` entries (checksums + provenance)
|
||||||
|
|
||||||
## Boundaries
|
## Key Behavior
|
||||||
Owns:
|
- validates required config/store state.
|
||||||
- input path resolution and materialization;
|
- enforces local audio vs S3 audio mutual exclusivity.
|
||||||
- S3 audio list/download/copy flow;
|
- materializes S3 audio through spool/cache-aware logic.
|
||||||
- previous-session artifact requirement collection from enabled configured artifacts;
|
- scans enabled configured artifact inputs for `narratio.previous_session.artifact.*` requirements.
|
||||||
- previous cache lifecycle when requirements exist (clear and rehydrate managed `previous/` state).
|
- when previous requirements exist:
|
||||||
|
|
||||||
Does not own:
|
|
||||||
- transcript or artifact generation;
|
|
||||||
- analyze-stage source resolution;
|
|
||||||
- archive commit behavior.
|
|
||||||
|
|
||||||
## Config fields used
|
|
||||||
- `session.session_id`
|
|
||||||
- `session.previous_session_id`
|
|
||||||
- `session.campaign`
|
|
||||||
- `session.inputs.speakers_file`
|
|
||||||
- `session.inputs.autocorrect_file`
|
|
||||||
- `session.inputs.glossary_file`
|
|
||||||
- `session.inputs.audio_dir`
|
|
||||||
- `session.inputs.audio_files`
|
|
||||||
- `session.inputs.audio_s3.prefix`
|
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `pipeline.spool.root`
|
|
||||||
- `pipeline.cache.root`
|
|
||||||
- `pipeline.cache.s3_audio`
|
|
||||||
- `pipeline.storage.s3.bucket`
|
|
||||||
- `pipeline.storage.s3.root_prefix`
|
|
||||||
- `pipeline.scriptorium.artifacts.<name>.enabled`
|
|
||||||
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source`
|
|
||||||
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required`
|
|
||||||
- `campaign.campaign_id`
|
|
||||||
- `campaign.inputs.speakers_file`
|
|
||||||
- `campaign.inputs.autocorrect_file`
|
|
||||||
- `campaign.inputs.glossary_file`
|
|
||||||
|
|
||||||
## External adapters used
|
|
||||||
- `storage.ObjectStore` for:
|
|
||||||
- S3 audio listing/downloads;
|
|
||||||
- previous-session current pointer/manifest/artifact object checks and downloads.
|
|
||||||
|
|
||||||
## State and manifest behavior
|
|
||||||
- Ensures workspace layout exists.
|
|
||||||
- Materializes canonical input files and audio files.
|
|
||||||
- For S3 audio, uses run-scoped spool for active downloads and durable cache for reusable audio files; cache hits copy directly to work audio without downloading the object again.
|
|
||||||
- Records `inputs/session.yml` provenance as local `session_config` or remote `session_config.s3`.
|
|
||||||
- Resolves campaign-provided stable input paths relative to `campaign.yml`.
|
|
||||||
- Resolves session-provided stable input overrides relative to `session.yml`.
|
|
||||||
- Scans enabled configured artifact inputs for canonical sources:
|
|
||||||
- `narratio.previous_session.artifact.<artifact_key>`
|
|
||||||
- If one or more canonical previous-session requirements exist:
|
|
||||||
- clears managed `previous/` state;
|
- clears managed `previous/` state;
|
||||||
- hydrates required/optional previous artifacts from the configured previous session’s committed archive current state;
|
- builds previous-cache remote plan;
|
||||||
- writes `previous/manifest.json` and hydrated `previous/artifacts/**`;
|
- downloads previous manifest/artifacts;
|
||||||
- stores archive-relative artifact paths such as `artifacts/session_recap.md` as `previous/artifacts/session_recap.md`, not `previous/artifacts/artifacts/session_recap.md`;
|
- records previous inputs in `manifest.inputs`.
|
||||||
- records hydrated previous inputs in `manifest.Inputs` with source `previous_session_archive.current`.
|
|
||||||
- If no canonical previous-session requirements exist, prepare does not manage `previous/`.
|
|
||||||
- `manifest.Inputs` is sorted deterministically by `(kind, path)`.
|
|
||||||
- S3 audio `manifest.Inputs` retain S3 provenance and include `cache_path`; `spool_path` is present only when the current prepare invocation downloaded the file.
|
|
||||||
|
|
||||||
## Required and optional previous-session behavior
|
Required previous-session inputs fail when unavailable; optional missing inputs are skipped.
|
||||||
- `previous_session_id` unset:
|
|
||||||
- if any referenced previous artifact is required: fail;
|
|
||||||
- if all referenced previous artifacts are optional: continue and omit them.
|
|
||||||
- Previous session archive current pointer or manifest missing:
|
|
||||||
- if any referenced previous artifact is required: fail;
|
|
||||||
- if all referenced previous artifacts are optional: continue and omit missing ones.
|
|
||||||
- Missing required previous artifact object: fail.
|
|
||||||
- Missing optional previous artifact object: omit.
|
|
||||||
- Downloaded previous artifacts must validate as non-empty files.
|
|
||||||
|
|
||||||
## Skip and resume behavior
|
## Invariants
|
||||||
- Runner-level skip remains authoritative:
|
- only `prepare` hydrates canonical `previous/` cache state.
|
||||||
- if `prepare` already succeeded and run is not forced, `prepare` does not run and no hydration/download occurs.
|
- managed previous artifacts are stored under `previous/artifacts/**` without duplicate `artifacts/artifacts/` nesting.
|
||||||
- If `prepare` runs (including with `--force`), it owns managed `previous/` state for canonical previous-session inputs.
|
- `manifest.inputs` ordering is deterministic (`kind`, `path`).
|
||||||
|
|
||||||
## Failure behavior
|
|
||||||
- Fails on missing required input files, invalid audio-source combinations, empty/duplicate audio inputs, missing object store for S3 modes, and remote access/download/validation errors.
|
|
||||||
- For required canonical previous-session inputs, analyze-time missing-input guidance is to rerun:
|
|
||||||
- `narratio run-stage --force prepare`
|
|
||||||
|
|
||||||
## Tests to inspect before changing
|
|
||||||
- `internal/stage/prepare_test.go`
|
|
||||||
- `internal/stage/prepare_previous_test.go`
|
|
||||||
- `internal/artifacts/previous_requirements_test.go`
|
|
||||||
- `internal/app/runner_test.go`
|
|
||||||
|
|
||||||
## Architectural invariants
|
|
||||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive.
|
|
||||||
- Storage keys are computed by callers using archive/path helpers; storage adapter receives explicit keys.
|
|
||||||
- `prepare` is the only stage that hydrates canonical previous-session cache state.
|
|
||||||
|
|||||||
44
docs/internal/stage-publish.md
Normal file
44
docs/internal/stage-publish.md
Normal file
@@ -0,0 +1,44 @@
|
|||||||
|
# Stage: publish
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Upload run/session outputs to object storage and atomically advance remote current state.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
- successful prerequisite stages: `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `render`, `analyze`
|
||||||
|
- run root `runs/{run_id}/**`
|
||||||
|
- publish output rules (`pipeline.publish.outputs`)
|
||||||
|
- effective publish locks (static + remote merged lock set)
|
||||||
|
- local `previous/**` files when present
|
||||||
|
|
||||||
|
## Outputs
|
||||||
|
- uploaded run files under remote `runs/{run_id}/...` (excluding `audio/**`)
|
||||||
|
- uploaded selected publish outputs under session prefix
|
||||||
|
- uploaded `previous/**` files under session prefix when present
|
||||||
|
- uploaded `current/manifest.json`
|
||||||
|
- uploaded `current/run_id.txt` written last
|
||||||
|
|
||||||
|
## Key Behavior
|
||||||
|
- stage can self-skip when publish disabled or run upload disabled.
|
||||||
|
- validates prerequisite stage success and object-store availability.
|
||||||
|
- collects deterministic run file list plus run `manifest.json`.
|
||||||
|
- resolves publish output sources through runtime artifact catalog and manifest-aware resolution.
|
||||||
|
- selected artifact filter applies to configured artifact sources only.
|
||||||
|
- locked outputs are skipped intentionally (including required ones).
|
||||||
|
- optional missing outputs are skipped; required missing unlocked outputs fail.
|
||||||
|
- writes remote current manifest before current run pointer.
|
||||||
|
|
||||||
|
## Metadata Signals
|
||||||
|
Includes counts/lists for:
|
||||||
|
- run uploads
|
||||||
|
- published output uploads
|
||||||
|
- previous uploads
|
||||||
|
- skipped optional outputs
|
||||||
|
- skipped unselected outputs
|
||||||
|
- locked outputs
|
||||||
|
- current-state key paths
|
||||||
|
- `current_pointer_written`
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
- `current/run_id.txt` is the remote commit marker and is written last.
|
||||||
|
- run upload excludes `audio/**`.
|
||||||
|
- publish locks are not overridden by `--force`.
|
||||||
29
docs/internal/stage-render.md
Normal file
29
docs/internal/stage-render.md
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
# Stage: render
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
Render Markdown transcript artifacts from normalized JSON transcripts via Seriatim.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
- `narratio.transcript.final` (`transcripts/final.json`)
|
||||||
|
- `narratio.transcript.final_trimmed` (`transcripts/final.trimmed.json`)
|
||||||
|
|
||||||
|
## Outputs
|
||||||
|
- `narratio.transcript.final_markdown` -> `transcripts/final.md`
|
||||||
|
- `narratio.transcript.final_trimmed_markdown` -> `transcripts/final.trimmed.md`
|
||||||
|
|
||||||
|
## Key Behavior
|
||||||
|
- uses `pipeline.render` settings (enabled/format/title/booleans).
|
||||||
|
- resolves inputs manifest-first, then canonical fallback.
|
||||||
|
- writes run-local outputs first, then materializes canonical session outputs.
|
||||||
|
- records input provenance, output paths, adapter metadata, logs, and generated config refs.
|
||||||
|
- skips with stage metadata when `pipeline.render.enabled=false`.
|
||||||
|
|
||||||
|
## Failure Semantics
|
||||||
|
- missing normalized input fails with normalize rerun guidance.
|
||||||
|
- missing trimmed input fails with trim rerun guidance.
|
||||||
|
- adapter/subprocess failure fails stage.
|
||||||
|
- empty render output files fail validation.
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
- only `format: markdown` is supported.
|
||||||
|
- render stage owns production of built-in Markdown transcript sources.
|
||||||
@@ -1,58 +1,22 @@
|
|||||||
# Stage: transcribe
|
# Stage: transcribe
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Generate per-speaker raw transcripts from prepared audio using WhisperX.
|
Generate raw per-speaker transcripts from prepared audio using WhisperX.
|
||||||
|
|
||||||
## Inputs and Outputs
|
## Inputs
|
||||||
Inputs:
|
- `audio/*.flac` from `prepare`
|
||||||
- `audio/*.flac` prepared by `prepare`
|
|
||||||
|
|
||||||
Outputs:
|
## Outputs
|
||||||
- `transcripts/raw/<speaker>.json` for each input audio file
|
- `transcripts/raw/<speaker>.json`
|
||||||
|
|
||||||
## Boundaries
|
## Key Behavior
|
||||||
Owns:
|
- discovers prepared audio from manifest inputs or canonical audio directory.
|
||||||
- Discovering prepared audio inputs
|
- derives speaker ID from `.flac` basename.
|
||||||
- Deriving speaker ids from audio basenames
|
- runs WhisperX with configured concurrency/retry settings.
|
||||||
- Parallel WhisperX invocation with bounded concurrency
|
- validates each output as JSON.
|
||||||
- Validating produced JSON and promoting run-local outputs
|
- writes run-local outputs then materializes canonical transcript outputs.
|
||||||
|
|
||||||
Does not own:
|
## Invariants
|
||||||
- Transcript merge/polish/normalize/trim/analyze
|
- speaker basenames must be unique.
|
||||||
|
- output path returned by adapter must match requested output path.
|
||||||
## Config Fields Used
|
- each successful output is validated before stage success.
|
||||||
- `session.session_id`
|
|
||||||
- `session.campaign`
|
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `pipeline.whisperx.transcribe_url`
|
|
||||||
- `pipeline.whisperx.language`
|
|
||||||
- `pipeline.whisperx.timeout`
|
|
||||||
- `pipeline.whisperx.retries`
|
|
||||||
- `pipeline.whisperx.retry_delay`
|
|
||||||
- `pipeline.whisperx.concurrency`
|
|
||||||
|
|
||||||
## External Adapters Used
|
|
||||||
- WhisperX adapter (`env.WhisperX.Transcribe`).
|
|
||||||
|
|
||||||
## State and Manifest Behavior
|
|
||||||
- Uses run-local output paths under `runs/{run_id}/transcribe/outputs/...` when run layout is enabled.
|
|
||||||
- Validates each generated transcript JSON before promotion.
|
|
||||||
- Promotes canonical outputs to `transcripts/raw/*.json`.
|
|
||||||
- Records per-file metadata (attempts/status/duration/output path) in stage metadata.
|
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
|
||||||
- Runner-level skip applies for previously succeeded stage unless forced.
|
|
||||||
- On forced upstream reruns, downstream succeeded stages can be marked `stale` by runner logic.
|
|
||||||
|
|
||||||
## Failure Behavior
|
|
||||||
- Fails if no prepared audio exists, duplicate speaker basenames are detected, adapter output path mismatches expected path, any output JSON is invalid, or one worker fails.
|
|
||||||
- Cancels in-flight workers after first terminal error.
|
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
|
||||||
- `internal/stage/transcribe_test.go`
|
|
||||||
- `internal/app/whisperx_wiring_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
|
||||||
- Speaker identity is derived from `.flac` basename and must be unique.
|
|
||||||
- Every successful speaker output must be valid JSON before promotion.
|
|
||||||
- Canonical raw transcript set is the only supported merge input surface.
|
|
||||||
|
|||||||
@@ -1,75 +1,28 @@
|
|||||||
# Stage: trim
|
# Stage: trim
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Optionally trim the final transcript to session bounds; always produce a durable final-trimmed transcript.
|
Produce a final-trimmed transcript; optionally generate bounds-driven trim.
|
||||||
|
|
||||||
## Inputs and Outputs
|
## Inputs
|
||||||
Inputs:
|
|
||||||
- `transcripts/final.json`
|
- `transcripts/final.json`
|
||||||
|
|
||||||
Outputs:
|
## Outputs
|
||||||
- `transcripts/final.trimmed.json` (or configured trim output path)
|
- `transcripts/final.trimmed.json` (or configured trim output path)
|
||||||
- when trim enabled: `artifacts/session_bounds.json`
|
- when trim enabled: `artifacts/session_bounds.json`
|
||||||
|
|
||||||
## Boundaries
|
## Key Behavior
|
||||||
Owns:
|
When `trim.enabled=false`:
|
||||||
- Trim-enabled switch behavior
|
- copies normalized transcript to trimmed output.
|
||||||
- Bounds generation via Scriptorium artifact run
|
|
||||||
- Bounds validation against final transcript
|
|
||||||
- Keep-selector derivation and Seriatim trim invocation
|
|
||||||
- Copy-through behavior when disabled or bounds indicate unchanged transcript
|
|
||||||
|
|
||||||
Does not own:
|
When `trim.enabled=true`:
|
||||||
- Upstream normalization
|
- runs Scriptorium bounds artifact generation;
|
||||||
- Downstream artifact analysis
|
- optionally runs render-debug output generation;
|
||||||
|
- validates bounds payload against transcript;
|
||||||
|
- derives keep selector;
|
||||||
|
- either copies unchanged transcript or runs Seriatim trim;
|
||||||
|
- validates trimmed transcript and materializes bounds output.
|
||||||
|
|
||||||
## Config Fields Used
|
## Invariants
|
||||||
- `session.session_id`
|
- normalized transcript is required input.
|
||||||
- `session.campaign`
|
- bounds output exists only in enabled trim path.
|
||||||
- `pipeline.workspace.root`
|
- render-debug output is diagnostic and not a declared stage output.
|
||||||
- `pipeline.trim.enabled`
|
|
||||||
- `pipeline.trim.output_path`
|
|
||||||
- `pipeline.trim.bounds.prompt_id`
|
|
||||||
- `pipeline.trim.bounds.profile_id`
|
|
||||||
- `pipeline.trim.bounds.timeout`
|
|
||||||
- `pipeline.trim.bounds.output_path`
|
|
||||||
- `pipeline.trim.bounds.transcript_input_name`
|
|
||||||
- `pipeline.trim.bounds.render_debug`
|
|
||||||
- `pipeline.trim.bounds.render_output_path`
|
|
||||||
- `pipeline.seriatim.binary`
|
|
||||||
- `pipeline.seriatim.timeout`
|
|
||||||
- `pipeline.scriptorium.binary`
|
|
||||||
- `pipeline.scriptorium.config_path`
|
|
||||||
- `pipeline.scriptorium.timeout`
|
|
||||||
|
|
||||||
## External Adapters Used
|
|
||||||
- Scriptorium adapter:
|
|
||||||
- optional `RenderArtifact` for bounds debug render
|
|
||||||
- `RunArtifact` for bounds output
|
|
||||||
- Seriatim adapter:
|
|
||||||
- `Trim` when bounds indicate trimming is required
|
|
||||||
|
|
||||||
## State and Manifest Behavior
|
|
||||||
- Reads final transcript from normalize manifest outputs when available; falls back to canonical path.
|
|
||||||
- Uses run-local outputs/logs/reports/config/scratch paths when run layout is enabled.
|
|
||||||
- Promotes canonical final-trimmed transcript; promotes session bounds when trim enabled.
|
|
||||||
- Records bounds diagnostics, trim action, keep selector, and adapter metadata.
|
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
|
||||||
- Runner-level skip applies when already succeeded and not forced.
|
|
||||||
- Forced reruns can stale downstream succeeded stages.
|
|
||||||
- When `trim.enabled=false`, stage still succeeds by copying final to final-trimmed output.
|
|
||||||
|
|
||||||
## Failure Behavior
|
|
||||||
- Fails on missing/invalid final transcript.
|
|
||||||
- With trim enabled, fails on missing adapters/config, bounds generation/validation errors, invalid bounds JSON, invalid range/segment ids, trim adapter failures, or invalid final-trimmed output.
|
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
|
||||||
- `internal/stage/trim_test.go`
|
|
||||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
|
||||||
- `internal/adapters/seriatim/subprocess_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
|
||||||
- Trim never falls back to polished transcript; final transcript is required input.
|
|
||||||
- `session_bounds` output exists only for enabled trim path.
|
|
||||||
- Render-debug artifacts are diagnostics and not declared stage outputs.
|
|
||||||
|
|||||||
@@ -1,75 +1,34 @@
|
|||||||
# Internal: Storage
|
# Internal: Storage
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Document Narratio's remote storage backend contracts and implementations under `internal/adapters/storage`.
|
Document remote object-store contracts and S3 implementation behavior.
|
||||||
|
|
||||||
## Inputs and outputs
|
## Primary Contract
|
||||||
Inputs:
|
`storage.ObjectStore` interface:
|
||||||
- Resolved storage config (`pipeline.storage.*`).
|
- `List(ctx, prefix)`
|
||||||
- Already-loaded environment variables for configured S3 credentials.
|
- `Download(ctx, key, localPath)`
|
||||||
- Bucket-relative object keys and local file paths from app/stage orchestration.
|
- `Upload(ctx, localPath, key, opts)`
|
||||||
|
- `Exists(ctx, key)`
|
||||||
|
|
||||||
Outputs:
|
Key invariant:
|
||||||
- Listed/downloaded/uploaded object metadata (`ObjectInfo`).
|
- callers pass full bucket-relative keys;
|
||||||
- Existence checks and storage-layer errors.
|
- storage implementations do not infer campaign/session/run prefixes.
|
||||||
|
|
||||||
## Boundaries
|
## Configuration
|
||||||
Owns:
|
`NewObjectStoreFromConfig` currently supports S3-backed stores from `pipeline.storage.*` config.
|
||||||
- Remote object-store interface and implementation details.
|
|
||||||
- S3 client wiring and API calls.
|
|
||||||
- Object key normalization and upload/download/list primitives.
|
|
||||||
|
|
||||||
Does not own:
|
S3 constructor behavior:
|
||||||
- Session/run prefix semantics.
|
- requires configured bucket;
|
||||||
- Archive commit order semantics.
|
- uses region/endpoint/path-style options when set;
|
||||||
- Manifest updates.
|
- resolves credentials from configured env var names (with defaults).
|
||||||
- Filesystem secret loading from `pipeline.secrets.env_dir`.
|
|
||||||
|
|
||||||
## Config fields used
|
## S3 Backend Behavior
|
||||||
- `pipeline.storage.backend`
|
- normalizes object keys.
|
||||||
- `pipeline.storage.s3.bucket`
|
- `List` paginates and returns normalized `ObjectInfo`.
|
||||||
- `pipeline.storage.s3.region`
|
- `Download` writes local files with parent directory creation.
|
||||||
- `pipeline.storage.s3.endpoint`
|
- `Upload` streams local file and returns remote metadata.
|
||||||
- `pipeline.storage.s3.force_path_style`
|
- `Exists` maps not-found responses to `false`.
|
||||||
- `pipeline.storage.s3.access_key_id_env`
|
|
||||||
- `pipeline.storage.s3.secret_access_key_env`
|
|
||||||
|
|
||||||
## External adapters used
|
## Invariants
|
||||||
Storage package contracts:
|
- storage layer is stateless regarding manifest/stage progression.
|
||||||
- `ObjectStore` (active remote object-store boundary): `List`, `Download`, `Upload`, `Exists`.
|
- publish ordering semantics are owned by stage/app code, not storage adapters.
|
||||||
- `Backend` (archive request boundary): currently implemented with `NoopBackend` only.
|
|
||||||
|
|
||||||
Implementations:
|
|
||||||
- `S3Backend`: AWS SDK-backed `ObjectStore` implementation.
|
|
||||||
- `FakeBackend`: deterministic test `ObjectStore` and archive backend.
|
|
||||||
- `NoopBackend`: deterministic no-op archive backend for compatibility wiring.
|
|
||||||
|
|
||||||
## State and manifest behavior
|
|
||||||
- Storage implementations are stateless with respect to manifest/session lifecycle.
|
|
||||||
- Caller supplies fully-qualified bucket-relative keys.
|
|
||||||
- Storage layer does not infer campaign/session/run/root-prefix semantics.
|
|
||||||
- Caller controls publish ordering; storage layer executes individual operations in the order invoked.
|
|
||||||
|
|
||||||
## Skip and resume behavior
|
|
||||||
- No storage-level skip/resume behavior.
|
|
||||||
- Skip/resume decisions are made by stage/app logic before storage calls occur.
|
|
||||||
|
|
||||||
## Failure behavior
|
|
||||||
- `NewObjectStoreFromConfig` fails when no remote backend is configured or required S3 config is missing.
|
|
||||||
- `S3Backend` constructor fails when required bucket is missing or AWS client setup fails.
|
|
||||||
- App command orchestration loads configured filesystem secrets before calling the object-store factory.
|
|
||||||
- CRUD operations return contextual errors (including not-found behavior via `Exists`).
|
|
||||||
- Key normalization is applied before operations (`\\` to `/`, leading slash trimmed).
|
|
||||||
- Remote session loading uses `List` to find the exact `session.yml` key and `Download` to materialize it to a local temp file.
|
|
||||||
|
|
||||||
## Tests to inspect before changing
|
|
||||||
- `internal/adapters/storage/factory_test.go`
|
|
||||||
- `internal/adapters/storage/s3_backend_test.go`
|
|
||||||
- `internal/adapters/storage/fake_test.go`
|
|
||||||
- `internal/adapters/storage/keys_test.go`
|
|
||||||
- `internal/adapters/storage/archive.go` + consumers in stage tests (`prepare`, `archive`)
|
|
||||||
|
|
||||||
## Architectural invariants
|
|
||||||
- Callers pass full bucket-relative keys.
|
|
||||||
- Storage backends must not prepend or infer narratio prefixes.
|
|
||||||
- Remote transport details remain isolated to storage adapter implementations.
|
|
||||||
|
|||||||
@@ -1,78 +1,57 @@
|
|||||||
# Workspace internals
|
# Internal: Workspace
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define the local durable and run-local workspace model used by stages, manifests, resume, and archive.
|
Define local session layout, run-local stage layout, and cleanup guardrails.
|
||||||
|
|
||||||
## Inputs and Outputs
|
## Canonical Session Layout
|
||||||
Inputs:
|
Session root:
|
||||||
- `pipeline.workspace.root`
|
- `{workspace.root}/work/{campaign}/{session_id}`
|
||||||
- `session.campaign`
|
|
||||||
- `session.session_id`
|
|
||||||
- generated `run_id`
|
|
||||||
|
|
||||||
Outputs:
|
Core directories/files:
|
||||||
- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`
|
- `inputs/`
|
||||||
- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`
|
- `audio/`
|
||||||
- Canonical durable session directories and run-local stage trees
|
- `transcripts/`
|
||||||
|
- `artifacts/`
|
||||||
|
- `reports/`
|
||||||
|
- `logs/`
|
||||||
|
- `config/`
|
||||||
|
- `current/`
|
||||||
|
- `runs/`
|
||||||
|
- `previous/`
|
||||||
|
- `manifest.json`
|
||||||
|
- `.lock`
|
||||||
|
|
||||||
## Boundaries
|
`previous/` reserved files:
|
||||||
Owns:
|
- `previous/manifest.json`
|
||||||
- Session-level path layout (`inputs/`, `audio/`, `transcripts/`, `artifacts/`, `reports/`, `logs/`, `config/`, `current/`, `runs/`, `previous/`)
|
- `previous/artifacts/**`
|
||||||
- `previous/manifest.json` and `previous/artifacts/**` are reserved for previous-session cache state materialized by `prepare` or `restore`
|
|
||||||
- Run-local stage sandbox layout under `runs/{run_id}/{stage}/`
|
|
||||||
- Session lock acquisition/release (`.lock`)
|
|
||||||
|
|
||||||
Does not own:
|
## Run-Local Stage Layout
|
||||||
- Stage business logic
|
When run context is available, stages use:
|
||||||
- Remote archive semantics (documented in `stage-archive.md`)
|
- `runs/{run_id}/{stage}/outputs/`
|
||||||
- CLI argument parsing
|
- `runs/{run_id}/{stage}/logs/`
|
||||||
|
- `runs/{run_id}/{stage}/reports/`
|
||||||
|
- `runs/{run_id}/{stage}/config/`
|
||||||
|
- `runs/{run_id}/{stage}/scratch/`
|
||||||
|
|
||||||
## Config Fields Used
|
Run-local outputs are materialized back into canonical session paths before stage success.
|
||||||
- `pipeline.workspace.root`
|
`previous/**` writes are never redirected to run-local output paths.
|
||||||
- `pipeline.workspace.cleanup_after_archive`
|
|
||||||
- `pipeline.spool.root`
|
|
||||||
- `pipeline.spool.delete_audio_after_archive`
|
|
||||||
- `pipeline.cache.root`
|
|
||||||
- `pipeline.cache.s3_audio`
|
|
||||||
- `session.campaign`
|
|
||||||
- `session.session_id`
|
|
||||||
|
|
||||||
## External Adapters Used
|
## Locking
|
||||||
None directly in this subsystem. Stages may use object storage adapters and then write local outputs into this layout.
|
`artifacts.LocalStore` enforces single-writer session lock via `.lock` file (`ErrLockConflict` on contention).
|
||||||
|
|
||||||
## State and Manifest Behavior
|
## Cleanup Semantics
|
||||||
- Session state is persisted in the session manifest (`manifest.Manifest`).
|
Automatic post-publish cleanup:
|
||||||
- Invocation history is persisted per run in run manifests under `runs/{run_id}/manifest.json`.
|
- only runs when publish actually executed and succeeded;
|
||||||
- During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and promoted to canonical session paths after stage success.
|
- requires `uploaded=true` and `current_pointer_written=true` metadata;
|
||||||
- `manifest.Artifacts` entries record `ProducerRunID` for durable outputs.
|
- respects `pipeline.spool.delete_audio_after_publish` and `pipeline.workspace.cleanup_after_publish`;
|
||||||
- For S3 audio sessions, `prepare` records work/cache paths, S3 provenance, and spool path when the invocation downloaded the object.
|
- refuses unsafe deletes (root delete, out-of-root delete, symlink paths).
|
||||||
- `previous/**` is reconstructed from configured previous-session requirements; restore uses the previous session's committed current archive rather than treating current-session archived `previous/**` as authoritative.
|
|
||||||
- Durable cache state under `pipeline.cache.root` is not workspace state and is preserved by default by `narratio clean`.
|
|
||||||
- `narratio clean <id>` removes the session work root and session spool root.
|
|
||||||
- `narratio clean --all` removes all local session work under `workspace.root/work` and spool children under `spool.root`.
|
|
||||||
- `narratio clean --clear-cache` is the explicit opt-in for deleting matching S3 audio cache entries.
|
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
Manual clean command:
|
||||||
- Skip/resume decisions are made in `internal/app` (`run_control.go`, `resume.go`) using stage status in the session manifest.
|
- `clean <session_id>` removes session work and spool subtree.
|
||||||
- `--force` reruns selected stages and marks downstream previously-succeeded stages as `stale`.
|
- `clean --all` removes all workspace work and spool children.
|
||||||
- Workspace layout is idempotent (`EnsureLayoutFor`) and reused across runs.
|
- durable cache is preserved unless `--clear-cache` is requested.
|
||||||
|
|
||||||
## Failure Behavior
|
## Invariants
|
||||||
- Failures preserve manifests and run-local files for inspection.
|
- campaign-aware session root is mandatory.
|
||||||
- Lock conflicts fail fast via `ErrLockConflict`.
|
- manifest-driven stage state is durable across runs.
|
||||||
- Cleanup can fail post-archive; failure is recorded in archive stage metadata and returned by the run.
|
- cleanup guardrails prevent destructive root/out-of-scope deletion.
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
|
||||||
- `internal/artifacts/local_test.go`
|
|
||||||
- `internal/stage/run_local_test.go`
|
|
||||||
- `internal/app/run_control_test.go`
|
|
||||||
- `internal/app/resume_run_stage_test.go`
|
|
||||||
- `internal/app/post_archive_cleanup_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
|
||||||
- Session root is campaign-aware: `{workspace.root}/work/{campaign}/{session_id}`.
|
|
||||||
- Run roots are always nested: `runs/{run_id}` under the session root.
|
|
||||||
- Run-local output promotion must end in canonical session paths.
|
|
||||||
- `previous/**` is session-durable state and must not be treated as run-local output scratch state.
|
|
||||||
- Automatic post-archive cleanup only targets run-scoped directories and must never delete configured root directories.
|
|
||||||
- Manual `clean` may delete session-scoped directories or the `workspace.root/work` directory, but it must preserve configured root directories and reject unsafe targets.
|
|
||||||
|
|||||||
@@ -1,285 +1,251 @@
|
|||||||
# Operations
|
# Operations Guide
|
||||||
|
|
||||||
This guide describes the implemented operator lifecycle for Narratio.
|
Operator workflow for running, recovering, and publishing Narratio sessions.
|
||||||
|
|
||||||
For field-level configuration, see [docs/config.md](./config.md). For full command/flag reference, see [docs/cli.md](./cli.md).
|
For command syntax, see [docs/cli.md](./cli.md). For field-level config, see [docs/config.md](./config.md).
|
||||||
|
|
||||||
## Normal workflow (S3-first path)
|
## Campaign and Session Selection
|
||||||
|
|
||||||
1. Create or upload `session.yml`, or pass a local `session.yml` explicitly.
|
Campaign selection priority:
|
||||||
2. Upload session `.flac` files to object storage under the configured session audio prefix.
|
|
||||||
3. Run Narratio:
|
- `--campaign-file`
|
||||||
|
- `--campaign`
|
||||||
|
- `pipeline.campaigns.default_campaign_id`
|
||||||
|
|
||||||
|
Session source priority:
|
||||||
|
|
||||||
|
- `--session`
|
||||||
|
- local default search paths
|
||||||
|
- remote session object (S3) when local session file is not found and storage is configured
|
||||||
|
|
||||||
|
## Session Initialization
|
||||||
|
|
||||||
|
Use `session init` to generate a concrete session file for local or remote use.
|
||||||
|
|
||||||
|
Local file:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio run 2026-04-04
|
narratio session init 2026-04-04 --output ./session.yml --date 2026-04-04 --title "Session 12"
|
||||||
```
|
```
|
||||||
|
|
||||||
4. Read success output:
|
Remote session object:
|
||||||
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
|
||||||
- use `narratio session status <session_id>` for inspection.
|
|
||||||
|
|
||||||
Notes:
|
|
||||||
- default pipeline/session discovery checks system config locations; campaign selection uses `pipeline.campaigns.default_campaign_id` unless `--campaign <id>` or `--campaign-file <path>` is passed.
|
|
||||||
- when local `session.yml` discovery misses, positional `<session_id>` loads remote `session.yml` from `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`.
|
|
||||||
- S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access.
|
|
||||||
|
|
||||||
Initialize a remote session skeleton:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session init 2026-04-04 --remote
|
narratio session init 2026-04-04 --remote --force
|
||||||
```
|
```
|
||||||
|
|
||||||
Remote init uses normal default config discovery and writes `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`. If `campaign.yml` sets `session_template_file`, init renders that template from the supplied flags and writes concrete YAML. Pass `--config`, `--campaign <id>`, or `--campaign-file <path>` when testing non-system config files. It fails if the object already exists unless `--force` is passed.
|
If `campaign.yml` sets `session_template_file`, `session init` renders it. Template variables must resolve to concrete values.
|
||||||
|
|
||||||
Validate before running:
|
## Standard Session Workflow
|
||||||
|
|
||||||
|
1. Select pipeline/campaign/session config.
|
||||||
|
2. Validate session readiness:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session validate 2026-04-04
|
narratio session validate 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
## Restore workflow
|
3. (Optional) inspect stage decisions:
|
||||||
|
|
||||||
Use restore when local durable session state is missing or stale and archive current state is authoritative.
|
|
||||||
|
|
||||||
Dry-run (no local writes):
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session restore 2026-04-04 --dry-run
|
narratio session plan 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
Execution:
|
4. Run the pipeline:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session restore 2026-04-04
|
narratio run 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
Post-restore analyze rerun pattern:
|
5. Check state:
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio analyze 2026-04-04
|
|
||||||
```
|
|
||||||
|
|
||||||
Restore source-of-truth:
|
|
||||||
- remote commit marker: `current/run_id.txt`
|
|
||||||
- remote current manifest: `current/manifest.json`
|
|
||||||
- configured previous-session requirements are reconstructed from the previous session's remote `current/` state, not from archived `previous/**` objects in the current session.
|
|
||||||
|
|
||||||
Restore default scope:
|
|
||||||
- includes `manifest.json`, `transcripts/**`, and `artifacts/**` from the current session archive.
|
|
||||||
- includes `previous/**` only when configured previous-session artifact inputs require it; restore hydrates those files the same way `prepare` would.
|
|
||||||
- includes `audio/**` only with `--include-audio`
|
|
||||||
- excludes `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`, and `current/**` (except remote `current/manifest.json` as source)
|
|
||||||
|
|
||||||
Reset local state before restore testing:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio clean 2026-04-04 --dry-run
|
|
||||||
narratio clean 2026-04-04
|
|
||||||
narratio session restore 2026-04-04 --include-audio
|
|
||||||
```
|
|
||||||
|
|
||||||
`clean` removes the local session work directory and session spool directory. It preserves the durable S3 audio cache by default, so repeated restore or forced prepare tests do not re-download large audio files.
|
|
||||||
|
|
||||||
## Local filesystem layout and state artifacts
|
|
||||||
|
|
||||||
Session root:
|
|
||||||
- `{workspace.root}/work/{campaign}/{session_id}/`
|
|
||||||
|
|
||||||
Primary state:
|
|
||||||
- `manifest.json`: session-level stage state.
|
|
||||||
- `runs/{run_id}/manifest.json`: invocation-level state.
|
|
||||||
- `.lock`: session lock while a modifying command is active.
|
|
||||||
- `inputs/campaign.yml`, `inputs/session.yml`, and `inputs/pipeline.resolved.yml`: materialized config inputs for the run.
|
|
||||||
|
|
||||||
Canonical session directories:
|
|
||||||
- `inputs/`
|
|
||||||
- `audio/`
|
|
||||||
- `transcripts/`
|
|
||||||
- `artifacts/`
|
|
||||||
- `previous/`
|
|
||||||
- `reports/`
|
|
||||||
- `logs/`
|
|
||||||
- `config/`
|
|
||||||
- `current/`
|
|
||||||
- `runs/`
|
|
||||||
|
|
||||||
Run-local stage directories:
|
|
||||||
- `runs/{run_id}/{stage}/` with stage-local `outputs/`, `logs/`, `reports/`, `config/`, `scratch/`.
|
|
||||||
|
|
||||||
Behavior:
|
|
||||||
- directory creation is idempotent.
|
|
||||||
- stage outputs are generally generated run-local first, then promoted to canonical paths on success.
|
|
||||||
- restore installs downloaded files to canonical session paths and does not recreate historical run sandboxes.
|
|
||||||
|
|
||||||
## Analyze artifact execution lifecycle
|
|
||||||
|
|
||||||
Analyze executes configured artifacts from `pipeline.scriptorium.artifacts`.
|
|
||||||
|
|
||||||
Execution model:
|
|
||||||
- executable set = enabled artifacts, filtered by `--artifacts` when provided.
|
|
||||||
- artifact-to-artifact dependencies are declared via `depends_on`.
|
|
||||||
- selected artifacts run in deterministic dependency order.
|
|
||||||
- after each successful artifact run, output is promoted to configured canonical `output_path`.
|
|
||||||
|
|
||||||
Configured artifact source reuse:
|
|
||||||
- a non-executable configured artifact can satisfy inputs if its configured output file already exists and is valid.
|
|
||||||
- reused configured artifact provenance is `filesystem.disabled_artifact_output`.
|
|
||||||
|
|
||||||
`--artifacts` behavior:
|
|
||||||
- accepted on `run`, `resume`, `run-stage analyze`, `run-stage archive`, `analyze`, and `publish`.
|
|
||||||
- filters analyze execution and configured artifact promotions.
|
|
||||||
- built-in transcript and bounds promotions are not filtered.
|
|
||||||
- does not imply force on `run`, `resume`, or `run-stage`; `narratio analyze` is force-by-design.
|
|
||||||
- `publish` is force-by-design and accepts `--artifacts` for configured artifact promotions.
|
|
||||||
|
|
||||||
Canonical previous-session input behavior:
|
|
||||||
- canonical sources use `narratio.previous_session.artifact.<artifact_key>`.
|
|
||||||
- these inputs are hydrated by `prepare` and by `restore`; `analyze` expects the local previous cache to already exist.
|
|
||||||
- if analyze fails due to missing canonical previous cache, rerun:
|
|
||||||
- `narratio run-stage prepare <id> --force`
|
|
||||||
- or `narratio session restore <id>` when remote archive current state is authoritative.
|
|
||||||
|
|
||||||
## Remote archive layout and publish contract
|
|
||||||
|
|
||||||
Preferred manual publish command:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio publish <id>
|
|
||||||
```
|
|
||||||
|
|
||||||
`publish` is equivalent to `narratio run-stage archive <id> --force`; use `run-stage` when you need the general single-stage command form.
|
|
||||||
|
|
||||||
When archive is enabled and run upload is enabled, archive publishes under:
|
|
||||||
|
|
||||||
- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
|
||||||
- run prefix: `{session_prefix}/runs/{run_id}/`
|
|
||||||
|
|
||||||
Archive uploads:
|
|
||||||
- run record files from run root (excluding `audio/`).
|
|
||||||
- promoted files from explicit `archive.promote_artifacts` rules.
|
|
||||||
- mutable session locks from helper commands live at `{session_prefix}/locks.yml`.
|
|
||||||
|
|
||||||
Publish order:
|
|
||||||
1. upload `current/manifest.json`
|
|
||||||
2. upload `current/run_id.txt` last
|
|
||||||
|
|
||||||
`current/run_id.txt` is the remote commit marker.
|
|
||||||
|
|
||||||
Archive promotion is explicit and source-based:
|
|
||||||
- Narratio does not auto-promote all generated analyze artifacts.
|
|
||||||
- each rule resolves `source` through the artifact resolver/catalog model, then uploads to `dest`.
|
|
||||||
- missing required promotion sources fail archive stage.
|
|
||||||
- missing optional promotion sources are skipped.
|
|
||||||
- invalid resolved artifacts fail archive stage.
|
|
||||||
- `archive.locks` skips top-level promotion overwrites for locked sources while run-local uploads still publish.
|
|
||||||
- remote locks from `{session_prefix}/locks.yml` are merged with static `archive.locks`; static locks win on duplicate sources.
|
|
||||||
- locked required promotions are treated as intentional successful skips and are recorded in archive metadata.
|
|
||||||
|
|
||||||
Lock helper behavior:
|
|
||||||
- `narratio session locks <id>` lists effective static and remote locks.
|
|
||||||
- `narratio session locks add <id> <source> --reason <text>` writes a remote lock.
|
|
||||||
- `narratio session locks add <id> <source> --force --reason <text>` updates an existing remote lock reason.
|
|
||||||
- `narratio session locks remove <id> <source>` removes only a remote lock.
|
|
||||||
- `locks remove` cannot remove static pipeline locks.
|
|
||||||
- remote lock writes check whether the lock store exists, but are not compare-and-swap atomic.
|
|
||||||
|
|
||||||
## Resume, retry, restore, and safe rerun behavior
|
|
||||||
|
|
||||||
Default skip:
|
|
||||||
- `run` and `run-stage` skip already-succeeded stages unless `--force` is set.
|
|
||||||
|
|
||||||
Resume:
|
|
||||||
- `resume` starts at first non-succeeded stage.
|
|
||||||
- `resume --force` runs full stage order.
|
|
||||||
|
|
||||||
Restore conflict policy:
|
|
||||||
- restore classifies local differences as conflicts.
|
|
||||||
- without `--force`, restore fails when conflicts exist.
|
|
||||||
- with `--force`, conflicting local files are overwritten by remote archive files.
|
|
||||||
|
|
||||||
Forced reruns:
|
|
||||||
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
|
|
||||||
- ordinary `--force` does not override archive locks.
|
|
||||||
|
|
||||||
Safe rerun pattern:
|
|
||||||
1. rerun the changed stage with `--force`.
|
|
||||||
2. run `resume` to rebuild downstream stages.
|
|
||||||
|
|
||||||
## Cleanup behavior
|
|
||||||
|
|
||||||
Automatic post-archive cleanup is considered only when archive stage executed and succeeded.
|
|
||||||
|
|
||||||
Automatic cleanup toggles:
|
|
||||||
- `pipeline.spool.delete_audio_after_archive=true` deletes run-scoped spool audio.
|
|
||||||
- `pipeline.workspace.cleanup_after_archive=true` deletes run-scoped local run directory.
|
|
||||||
|
|
||||||
Manual cleanup:
|
|
||||||
- `narratio clean <id>` deletes `{workspace.root}/work/{campaign}/{session_id}` and `{spool.root}/{campaign}/{session_id}`.
|
|
||||||
- `narratio clean --all` deletes all local session work under `{workspace.root}/work` and all spool children under `{spool.root}`.
|
|
||||||
- `--dry-run` prints targets without deleting.
|
|
||||||
- `--clear-cache` also removes matching S3 audio cache files. Without it, cache is preserved.
|
|
||||||
|
|
||||||
The S3 audio cache under `pipeline.cache.root` is durable input cache state, not workspace or spool state. Automatic cleanup and default manual cleanup do not delete it.
|
|
||||||
|
|
||||||
Cleanup eligibility gates:
|
|
||||||
- archive enabled
|
|
||||||
- archive run upload enabled
|
|
||||||
- run record upload completed
|
|
||||||
- current pointer write completed (`current/run_id.txt` written)
|
|
||||||
|
|
||||||
No cleanup for failed/incomplete/unarchived/archive-skipped runs.
|
|
||||||
|
|
||||||
## Failure and recovery playbooks
|
|
||||||
|
|
||||||
After run failure, Narratio keeps:
|
|
||||||
- session manifest
|
|
||||||
- run manifest
|
|
||||||
- run-local artifacts/logs/config/reports
|
|
||||||
|
|
||||||
Failed or incomplete runs remain local-only.
|
|
||||||
|
|
||||||
After restore failure:
|
|
||||||
- already-installed restore files remain in place.
|
|
||||||
- restore does not roll back prior successful installs.
|
|
||||||
- existing local manifest is preserved if restored manifest validation/install fails.
|
|
||||||
|
|
||||||
Recommended recovery:
|
|
||||||
|
|
||||||
1. inspect state:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session status 2026-04-04
|
narratio session status 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
This reports local manifest state, committed remote current state, expected remote transcript/artifact availability, and archive locks.
|
## Stage Execution and Continuation Behavior
|
||||||
|
|
||||||
2. for restore-specific checks, run:
|
Canonical stage order:
|
||||||
|
|
||||||
|
1. `prepare`
|
||||||
|
2. `transcribe`
|
||||||
|
3. `merge`
|
||||||
|
4. `polish`
|
||||||
|
5. `normalize`
|
||||||
|
6. `trim`
|
||||||
|
7. `render`
|
||||||
|
8. `analyze`
|
||||||
|
9. `publish`
|
||||||
|
10. `notify`
|
||||||
|
|
||||||
|
Execution rules:
|
||||||
|
|
||||||
|
- succeeded stages are skipped unless `--force` is set;
|
||||||
|
- `run` continues interrupted or partially completed sessions by running non-succeeded stages;
|
||||||
|
- force rerunning a succeeded upstream stage marks succeeded downstream stages as `stale`.
|
||||||
|
|
||||||
|
Single-stage execution:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage normalize 2026-04-04 --force
|
||||||
|
```
|
||||||
|
|
||||||
|
## Artifact Selection
|
||||||
|
|
||||||
|
`--artifacts` can be used on `run`, `run-stage`, `analyze`, and `publish`.
|
||||||
|
|
||||||
|
Selection behavior:
|
||||||
|
|
||||||
|
- validates names against `pipeline.scriptorium.artifacts`;
|
||||||
|
- filters analyze execution to selected configured artifacts;
|
||||||
|
- filters publish rules for `narratio.artifact.<name>` sources only;
|
||||||
|
- does not suppress built-in transcript or bounds publish sources.
|
||||||
|
|
||||||
|
## Publish Workflow
|
||||||
|
|
||||||
|
Run publish only:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio publish 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
Equivalent:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage publish 2026-04-04 --force
|
||||||
|
```
|
||||||
|
|
||||||
|
Publish commit model:
|
||||||
|
|
||||||
|
- uploads run files under `{session_prefix}/runs/{run_id}/`;
|
||||||
|
- uploads configured published outputs;
|
||||||
|
- uploads `previous/**` cache files when present;
|
||||||
|
- writes `current/manifest.json`;
|
||||||
|
- writes `current/run_id.txt` last.
|
||||||
|
|
||||||
|
`current/run_id.txt` is the remote current-state commit marker.
|
||||||
|
|
||||||
|
## Publish Locks
|
||||||
|
|
||||||
|
Lock sources:
|
||||||
|
|
||||||
|
- static locks in `pipeline.publish.locks`
|
||||||
|
- mutable remote locks in `{session_prefix}/locks.yml`
|
||||||
|
|
||||||
|
Effective lock rules:
|
||||||
|
|
||||||
|
- static and remote locks are merged;
|
||||||
|
- static locks win on source collisions;
|
||||||
|
- locked outputs are intentional skips;
|
||||||
|
- lock add/remove commands mutate only remote lock state.
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session locks 2026-04-04
|
||||||
|
narratio session locks add 2026-04-04 narratio.artifact.session_recap --reason "manual edits" --force
|
||||||
|
narratio session locks remove 2026-04-04 narratio.artifact.session_recap
|
||||||
|
```
|
||||||
|
|
||||||
|
## Restore Workflow
|
||||||
|
|
||||||
|
Use restore when local durable session state is missing or stale and remote committed current state is authoritative.
|
||||||
|
|
||||||
|
Dry run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session restore 2026-04-04 --dry-run
|
narratio session restore 2026-04-04 --dry-run
|
||||||
```
|
```
|
||||||
|
|
||||||
3. fix root cause (config/input/credentials/storage/service availability).
|
Apply:
|
||||||
4. continue with `resume`, or targeted `run-stage <stage> <id> --force` followed by `resume`.
|
|
||||||
|
|
||||||
## Restore report
|
```bash
|
||||||
|
narratio session restore 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
Non-dry-run restore writes a durable report at:
|
Default restore scope:
|
||||||
- `reports/restore-latest.json`
|
|
||||||
|
|
||||||
Report content includes:
|
- `manifest.json`
|
||||||
- identity (`campaign`, `session_id`, `run_id`)
|
- `transcripts/**`
|
||||||
- mode flags (`dry_run`, `force`, `include_audio`)
|
- `artifacts/**`
|
||||||
- plan counts and execution counts
|
- `previous/**` when needed by configured previous-session artifact inputs
|
||||||
- per-action status
|
|
||||||
|
|
||||||
Dry-run does not write restore report files.
|
Optional:
|
||||||
|
|
||||||
## Operational caveats
|
- `--include-audio` to include `audio/**`
|
||||||
|
- `--force` to overwrite local conflicts
|
||||||
|
|
||||||
- `session status <session_id>` uses normal config/session loading, including remote session fallback.
|
Restore writes an execution report at `reports/restore-latest.json`.
|
||||||
- `session status <session_id>` includes the same promoted remote output availability view as `session artifacts <session_id> --remote` when storage is configured.
|
|
||||||
- local and S3 audio input modes are mutually exclusive.
|
## Local State Layout
|
||||||
- archive publish requires upstream stages through `analyze` to be `succeeded`.
|
|
||||||
- required configured artifact promotions for unselected `--artifacts` keys are skipped intentionally; selected required promotions still fail if their files are missing.
|
Session root:
|
||||||
- restore requires configured remote object storage and committed remote current state.
|
|
||||||
|
- `{workspace.root}/work/{campaign}/{session_id}`
|
||||||
|
|
||||||
|
Durable session paths:
|
||||||
|
|
||||||
|
- `manifest.json`
|
||||||
|
- `inputs/**`
|
||||||
|
- `audio/**`
|
||||||
|
- `transcripts/**`
|
||||||
|
- `artifacts/**`
|
||||||
|
- `previous/**`
|
||||||
|
- `reports/**`
|
||||||
|
- `logs/**`
|
||||||
|
- `config/**`
|
||||||
|
- `runs/**`
|
||||||
|
|
||||||
|
Run-local layout:
|
||||||
|
|
||||||
|
- `runs/{run_id}/{stage}/outputs`
|
||||||
|
- `runs/{run_id}/{stage}/logs`
|
||||||
|
- `runs/{run_id}/{stage}/reports`
|
||||||
|
- `runs/{run_id}/{stage}/config`
|
||||||
|
- `runs/{run_id}/{stage}/scratch`
|
||||||
|
|
||||||
|
Spool layout (runtime/transient):
|
||||||
|
|
||||||
|
- `{spool.root}/{campaign}/{session_id}/{run_id}/...`
|
||||||
|
- restore audio spool under `{spool.root}/{campaign}/{session_id}/restore/audio`
|
||||||
|
|
||||||
|
Cache layout (durable S3 audio cache):
|
||||||
|
|
||||||
|
- `{cache.root}/s3/{bucket}/...`
|
||||||
|
|
||||||
|
## Cleanup
|
||||||
|
|
||||||
|
Session-scoped cleanup:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio clean 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
Global cleanup:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio clean --all
|
||||||
|
```
|
||||||
|
|
||||||
|
Dry-run and cache variants:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio clean 2026-04-04 --dry-run --clear-cache
|
||||||
|
narratio clean --all --dry-run --clear-cache
|
||||||
|
```
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- `clean` deletes work/spool session state;
|
||||||
|
- cache is preserved unless `--clear-cache` is set;
|
||||||
|
- automatic post-publish cleanup is gated by successful publish commit plus:
|
||||||
|
- `pipeline.spool.delete_audio_after_publish=true`
|
||||||
|
- `pipeline.workspace.cleanup_after_publish=true`
|
||||||
|
|
||||||
|
## Operational Caveats
|
||||||
|
|
||||||
|
- Local and S3 audio modes are mutually exclusive.
|
||||||
|
- Publish requires prerequisite stages through `render` and `analyze` to be succeeded.
|
||||||
|
- Markdown publish defaults require render outputs (`transcripts/final.md` and `transcripts/final.trimmed.md`).
|
||||||
|
- Restore requires configured object storage and committed remote current state.
|
||||||
|
- Storage-backed commands load filesystem secrets before object-store initialization.
|
||||||
|
|||||||
@@ -19,7 +19,7 @@ It coordinates specialized downstream systems rather than reimplementing their d
|
|||||||
- Audita handles transcript correction and polishing.
|
- Audita handles transcript correction and polishing.
|
||||||
- Scriptorium handles prompt execution and generated artifacts.
|
- Scriptorium handles prompt execution and generated artifacts.
|
||||||
|
|
||||||
Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and archive semantics.
|
Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and publish semantics.
|
||||||
|
|
||||||
Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine.
|
Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine.
|
||||||
|
|
||||||
@@ -80,7 +80,7 @@ It should record:
|
|||||||
- input and output refs;
|
- input and output refs;
|
||||||
- logs and generated config refs;
|
- logs and generated config refs;
|
||||||
- checksums or provenance where useful;
|
- checksums or provenance where useful;
|
||||||
- non-secret adapter and archive metadata.
|
- non-secret adapter and publish metadata.
|
||||||
|
|
||||||
Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress.
|
Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress.
|
||||||
|
|
||||||
@@ -117,19 +117,19 @@ Narratio should not become a secondary configuration system for downstream tools
|
|||||||
|
|
||||||
Local and remote paths are part of Narratio’s application contract.
|
Local and remote paths are part of Narratio’s application contract.
|
||||||
|
|
||||||
Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and archive paths. Stages should avoid reconstructing canonical paths through scattered string concatenation.
|
Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and publish/current paths. Stages should avoid reconstructing canonical paths through scattered string concatenation.
|
||||||
|
|
||||||
Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics.
|
Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics.
|
||||||
|
|
||||||
## Archive Invariants
|
## Publish Invariants
|
||||||
|
|
||||||
Archive behavior must preserve a clear commit boundary.
|
Publish behavior must preserve a clear commit boundary.
|
||||||
|
|
||||||
A remote run is current only after the archive stage has successfully uploaded the run record, required promoted outputs, `current/manifest.json`, and finally `current/run_id.txt`.
|
A remote run is current only after the publish stage has successfully uploaded the run record, required published outputs, `current/manifest.json`, and finally `current/run_id.txt`.
|
||||||
|
|
||||||
`current/run_id.txt` is the final remote commit marker and must be written last.
|
`current/run_id.txt` is the final remote commit marker and must be written last.
|
||||||
|
|
||||||
Failed, incomplete, skipped, or uncommitted archive attempts must not be presented as current remote state. Local cleanup is permitted only after successful archive commit and only when explicitly configured.
|
Failed, incomplete, skipped, or uncommitted publish attempts must not be presented as current remote state. Local cleanup is permitted only after successful publish commit and only when explicitly configured.
|
||||||
|
|
||||||
## Security and Privacy
|
## Security and Privacy
|
||||||
|
|
||||||
@@ -139,7 +139,7 @@ Rules:
|
|||||||
|
|
||||||
- Do not store raw secrets in pipeline or session YAML.
|
- Do not store raw secrets in pipeline or session YAML.
|
||||||
- Use environment variable names or secret-file references for secret handling.
|
- Use environment variable names or secret-file references for secret handling.
|
||||||
- Do not write raw secret values to manifests, logs, generated configs, or archive metadata.
|
- Do not write raw secret values to manifests, logs, generated configs, or publish metadata.
|
||||||
- Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive.
|
- Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive.
|
||||||
- Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason.
|
- Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason.
|
||||||
|
|
||||||
@@ -177,7 +177,7 @@ Tests should cover:
|
|||||||
- stage success, failure, skip, and resume behavior;
|
- stage success, failure, skip, and resume behavior;
|
||||||
- adapter command construction;
|
- adapter command construction;
|
||||||
- fake storage behavior;
|
- fake storage behavior;
|
||||||
- archive commit ordering;
|
- publish commit ordering;
|
||||||
- example config validity where practical.
|
- example config validity where practical.
|
||||||
|
|
||||||
Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs.
|
Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs.
|
||||||
@@ -6,7 +6,7 @@ Canonical contributor workflow and engineering conventions for implemented Narra
|
|||||||
## Repository layout
|
## Repository layout
|
||||||
|
|
||||||
- `cmd/narratio/`: CLI entrypoint.
|
- `cmd/narratio/`: CLI entrypoint.
|
||||||
- `internal/app/`: command handlers, plan/run/resume orchestration, cleanup gates, secrets loading.
|
- `internal/app/`: command handlers, run/stage orchestration, cleanup gates, secrets loading.
|
||||||
- `internal/config/`: strict YAML loading, defaults, and validation.
|
- `internal/config/`: strict YAML loading, defaults, and validation.
|
||||||
- `internal/stage/`: stage implementations and stage registry/order.
|
- `internal/stage/`: stage implementations and stage registry/order.
|
||||||
- `internal/adapters/`: external boundary adapters (WhisperX, Seriatim, Audita, Scriptorium, storage, notify).
|
- `internal/adapters/`: external boundary adapters (WhisperX, Seriatim, Audita, Scriptorium, storage, notify).
|
||||||
@@ -75,7 +75,7 @@ Remote-storage commands must obtain object storage through the app-level command
|
|||||||
|
|
||||||
1. Implement stage behavior in `internal/stage` with clear input/output boundaries.
|
1. Implement stage behavior in `internal/stage` with clear input/output boundaries.
|
||||||
2. Keep external transport/subprocess details in `internal/adapters`.
|
2. Keep external transport/subprocess details in `internal/adapters`.
|
||||||
3. Preserve manifest and promotion semantics expected by runner and archive logic.
|
3. Preserve manifest and publish-output semantics expected by runner and publish logic.
|
||||||
4. Add/update stage and adapter tests.
|
4. Add/update stage and adapter tests.
|
||||||
5. Update internal component contracts in `docs/internal/`.
|
5. Update internal component contracts in `docs/internal/`.
|
||||||
|
|
||||||
@@ -1,231 +0,0 @@
|
|||||||
# Roadmap: Campaign Registry
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
## Problem
|
|
||||||
|
|
||||||
Narratio currently treats campaign configuration as one selected
|
|
||||||
`campaign.yml` file:
|
|
||||||
|
|
||||||
- command flags use `--campaign <path>`;
|
|
||||||
- default discovery searches fixed system file locations;
|
|
||||||
- `campaign.yml` uses `campaign:` as the identity field.
|
|
||||||
|
|
||||||
That model works for a single campaign, but it is awkward for installations
|
|
||||||
that manage multiple campaigns. Operators need to pass file paths or maintain a
|
|
||||||
single global campaign config, while the newer session-oriented CLI already
|
|
||||||
uses concise positional session IDs and remote session lookup.
|
|
||||||
|
|
||||||
The campaign selection model should become ID-based and pipeline-owned.
|
|
||||||
Pipeline config should describe where campaigns live, commands should select a
|
|
||||||
campaign by ID, and each campaign directory should contain its stable campaign
|
|
||||||
materials.
|
|
||||||
|
|
||||||
## Target Model
|
|
||||||
|
|
||||||
`pipeline.yml` owns the campaign registry:
|
|
||||||
|
|
||||||
campaigns:
|
|
||||||
root: /usr/local/share/narratio/campaigns
|
|
||||||
default_campaign_id: dilfs
|
|
||||||
|
|
||||||
Campaign files live at the conventional path:
|
|
||||||
|
|
||||||
{campaigns.root}/{campaign_id}/campaign.yml
|
|
||||||
|
|
||||||
The first implementation should use only the conventional path. Recursive
|
|
||||||
discovery of every `campaign.yml` under `campaigns.root` is deferred to a
|
|
||||||
future stage.
|
|
||||||
|
|
||||||
Each campaign file uses `campaign_id` as the canonical identity field:
|
|
||||||
|
|
||||||
campaign_id: dilfs
|
|
||||||
session_template_file: ./session.template.yml
|
|
||||||
inputs:
|
|
||||||
speakers_file: ./speakers.yml
|
|
||||||
autocorrect_file: ./autocorrect.yml
|
|
||||||
glossary_file: ./glossary.yml
|
|
||||||
|
|
||||||
Campaign-relative files continue to resolve relative to the selected
|
|
||||||
`campaign.yml`, including stable input files and `session_template_file`.
|
|
||||||
|
|
||||||
The public CLI changes from path-based campaign selection to ID-based campaign
|
|
||||||
selection:
|
|
||||||
|
|
||||||
- `--campaign <id>` selects a campaign ID.
|
|
||||||
- `--campaign-file <path>` explicitly loads one campaign file for
|
|
||||||
development, tests, and unusual local workflows.
|
|
||||||
- `--campaign` and `--campaign-file` are mutually exclusive.
|
|
||||||
|
|
||||||
If neither `--campaign` nor `--campaign-file` is passed, Narratio uses
|
|
||||||
`pipeline.campaigns.default_campaign_id`. If no campaign can be selected,
|
|
||||||
commands fail clearly before session loading or stage execution.
|
|
||||||
|
|
||||||
Resolved campaign ID remains the campaign segment used for:
|
|
||||||
|
|
||||||
- workspace paths;
|
|
||||||
- spool paths;
|
|
||||||
- S3 session prefixes;
|
|
||||||
- remote `session.yml` lookup;
|
|
||||||
- archive locks and promoted output keys;
|
|
||||||
- session/campaign mismatch validation;
|
|
||||||
- status, plan, restore, and helper output.
|
|
||||||
|
|
||||||
## Compatibility Policy
|
|
||||||
|
|
||||||
This is a breaking public/config contract change.
|
|
||||||
|
|
||||||
After the cutover:
|
|
||||||
|
|
||||||
- `--campaign` no longer accepts a filesystem path;
|
|
||||||
- default fixed campaign file discovery is removed;
|
|
||||||
- `campaign:` is no longer accepted in `campaign.yml`;
|
|
||||||
- `campaign_id:` is required.
|
|
||||||
|
|
||||||
Keep `--campaign-file` as the only explicit file override. Do not retain hidden
|
|
||||||
aliases for the old `--campaign <path>` behavior.
|
|
||||||
|
|
||||||
## Implementation Stages
|
|
||||||
|
|
||||||
### Stage 1: Add Campaign Registry Selection
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Add the registry model and switch command loading to resolve campaigns through
|
|
||||||
pipeline config.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Add `pipeline.campaigns.root`.
|
|
||||||
- Add `pipeline.campaigns.default_campaign_id`.
|
|
||||||
- Add `campaign_id` to campaign config and make it the canonical identity.
|
|
||||||
- Resolve pipeline config first, then campaign selection.
|
|
||||||
- Use this selection order:
|
|
||||||
1. explicit `--campaign-file <path>`;
|
|
||||||
2. explicit `--campaign <id>`;
|
|
||||||
3. `pipeline.campaigns.default_campaign_id`;
|
|
||||||
4. fail clearly.
|
|
||||||
- For ID selection, load `{campaigns.root}/{campaign_id}/campaign.yml`.
|
|
||||||
- Validate that the loaded `campaign_id` matches the selected ID.
|
|
||||||
- Reject `--campaign` with `--campaign-file`.
|
|
||||||
- Preserve strict YAML decoding.
|
|
||||||
- Preserve campaign-relative stable input and session template resolution.
|
|
||||||
- Keep storage details behind the existing storage adapter and object-store
|
|
||||||
helper.
|
|
||||||
- Keep remote session lookup and archive key construction based on the
|
|
||||||
resolved campaign ID.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Commands can run with only a pipeline config and the pipeline default
|
|
||||||
campaign ID.
|
|
||||||
- Commands can select another campaign with `--campaign <id>`.
|
|
||||||
- Commands can load a specific file with `--campaign-file <path>`.
|
|
||||||
- Existing session loading, remote session fallback, prepare materialization,
|
|
||||||
restore, archive, locks, clean, analyze, and publish behavior continue to use
|
|
||||||
the same resolved campaign identity.
|
|
||||||
- No generic config registry framework is introduced.
|
|
||||||
|
|
||||||
### Stage 2: Remove Old Single-File Campaign Behavior
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Remove the old public campaign file model after registry selection is in
|
|
||||||
place.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Remove fixed default campaign config discovery from command loading.
|
|
||||||
- Remove `DefaultCampaignConfigSearchPaths` and related path-only resolution if
|
|
||||||
no current tests or helpers still need them.
|
|
||||||
- Remove support for `campaign:` from `campaign.yml`.
|
|
||||||
- Update validation errors to refer to `campaign_id`.
|
|
||||||
- Update examples to use campaign directories and `campaign_id`.
|
|
||||||
- Update current-behavior docs to document:
|
|
||||||
- `pipeline.campaigns.root`;
|
|
||||||
- `pipeline.campaigns.default_campaign_id`;
|
|
||||||
- `campaign_id`;
|
|
||||||
- `--campaign <id>`;
|
|
||||||
- `--campaign-file <path>`.
|
|
||||||
- Update troubleshooting examples that currently pass `--campaign <path>`.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- `campaign.yml` files with `campaign:` fail strict decoding.
|
|
||||||
- `--campaign /path/to/campaign.yml` is treated as a campaign ID and fails
|
|
||||||
unless that ID exists under `campaigns.root`.
|
|
||||||
- `--campaign-file /path/to/campaign.yml` is the supported file override.
|
|
||||||
- User-facing docs no longer describe fixed campaign config discovery.
|
|
||||||
|
|
||||||
## Test Guidance
|
|
||||||
|
|
||||||
Focused tests:
|
|
||||||
|
|
||||||
- `go test ./internal/config -v`
|
|
||||||
- `go test ./internal/app -v`
|
|
||||||
- `go test ./internal/stage -run Prepare -v`
|
|
||||||
|
|
||||||
Full validation:
|
|
||||||
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Config tests to add or update:
|
|
||||||
|
|
||||||
- strict decode accepts `pipeline.campaigns.root`;
|
|
||||||
- strict decode accepts `pipeline.campaigns.default_campaign_id`;
|
|
||||||
- strict decode accepts `campaign_id`;
|
|
||||||
- selected campaign ID mismatch fails;
|
|
||||||
- missing campaign root fails when ID selection is needed;
|
|
||||||
- missing default campaign ID fails when no explicit campaign selector is
|
|
||||||
passed;
|
|
||||||
- old `campaign:` fails after Stage 2.
|
|
||||||
|
|
||||||
App tests to add or update:
|
|
||||||
|
|
||||||
- `--campaign <id>` resolves `{campaigns.root}/{id}/campaign.yml`;
|
|
||||||
- omitted `--campaign` uses `pipeline.campaigns.default_campaign_id`;
|
|
||||||
- `--campaign-file` loads an explicit campaign file;
|
|
||||||
- `--campaign` plus `--campaign-file` fails;
|
|
||||||
- remote session fallback uses the resolved campaign ID;
|
|
||||||
- `session init`, `run`, `run-stage`, `resume`, `analyze`, `publish`, `clean`,
|
|
||||||
and `session` subcommands all use the same campaign selection path;
|
|
||||||
- path-based `--campaign` examples and tests are removed after Stage 2.
|
|
||||||
|
|
||||||
## Documentation Guidance
|
|
||||||
|
|
||||||
Update current-behavior docs only after implementation lands:
|
|
||||||
|
|
||||||
- `docs/config.md`
|
|
||||||
- `docs/cli.md`
|
|
||||||
- `docs/operations.md`
|
|
||||||
- `docs/troubleshooting.md`
|
|
||||||
- relevant files under `docs/internal/`
|
|
||||||
- `examples/`
|
|
||||||
|
|
||||||
Planned campaign registry behavior belongs only in this roadmap until the code,
|
|
||||||
tests, examples, and current-behavior docs are updated.
|
|
||||||
|
|
||||||
## Architecture Guardrails
|
|
||||||
|
|
||||||
- Keep Narratio explicit and stage-driven.
|
|
||||||
- Do not introduce a generic configuration registry or workflow framework.
|
|
||||||
- Keep YAML decoding strict.
|
|
||||||
- Keep defaults centralized and testable.
|
|
||||||
- Keep campaign-relative path resolution centralized.
|
|
||||||
- Use centralized S3 and workspace path helpers.
|
|
||||||
- Keep storage details behind `storage.ObjectStore`.
|
|
||||||
- Keep secret-backed object-store construction in `internal/app`.
|
|
||||||
- Preserve manifest-driven resume and restore behavior.
|
|
||||||
- Do not store raw secrets in campaign configs, manifests, logs, generated
|
|
||||||
configs, or archive metadata.
|
|
||||||
|
|
||||||
## Assumptions
|
|
||||||
|
|
||||||
- The canonical pipeline schema is grouped under `campaigns`.
|
|
||||||
- The canonical campaign identity field is `campaign_id`.
|
|
||||||
- `--campaign` means campaign ID.
|
|
||||||
- `--campaign-file` is retained as an explicit override.
|
|
||||||
- Recursive discovery is planned but not part of the first implementation.
|
|
||||||
- Existing production configs can be migrated from `campaign:` to
|
|
||||||
`campaign_id:` and from `--campaign <path>` to `--campaign <id>` or
|
|
||||||
`--campaign-file <path>`.
|
|
||||||
@@ -1,159 +0,0 @@
|
|||||||
# Roadmap: Legacy Config Cleanup
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
## Problem
|
|
||||||
|
|
||||||
Narratio's current pipeline config schema still accepts fields that predate the current storage, artifact, and previous-session models:
|
|
||||||
|
|
||||||
- `pipeline.storage.bucket`
|
|
||||||
- `pipeline.storage.prefix`
|
|
||||||
- `pipeline.analyzer.*`
|
|
||||||
- `previous_session_artifact`
|
|
||||||
|
|
||||||
These names make the config reference harder to trust because they suggest supported behavior that operators should no longer use. The modern interface is:
|
|
||||||
|
|
||||||
- `pipeline.storage.s3.*` for remote storage.
|
|
||||||
- Scriptorium configured artifacts under `pipeline.scriptorium.artifacts`.
|
|
||||||
- Canonical artifact source IDs such as `narratio.artifact.<configured_artifact_key>`.
|
|
||||||
- Canonical previous-session artifact sources such as `narratio.previous_session.artifact.<configured_artifact_key>`.
|
|
||||||
|
|
||||||
Strict YAML decoding should reject removed legacy fields once this cleanup lands.
|
|
||||||
|
|
||||||
## Current State
|
|
||||||
|
|
||||||
`pipeline.storage.bucket` and `pipeline.storage.prefix` were inert compatibility fields and have been removed:
|
|
||||||
|
|
||||||
- They are no longer present on `config.StorageConfig`.
|
|
||||||
- Strict decoding rejects them.
|
|
||||||
- Runtime S3 behavior uses `pipeline.storage.s3.bucket` and `pipeline.storage.s3.root_prefix`.
|
|
||||||
- No current code reads the top-level storage bucket or prefix fields.
|
|
||||||
|
|
||||||
`pipeline.analyzer.*` was legacy code surface and has been removed:
|
|
||||||
|
|
||||||
- `config.PipelineConfig` no longer includes analyzer config.
|
|
||||||
- Strict decoding rejects `pipeline.analyzer`.
|
|
||||||
- `stage.Env` no longer exposes an analyzer runner, and `internal/adapters/analyzer` has been deleted.
|
|
||||||
- Modern analyze execution is Scriptorium-backed; the analyzer adapter is not used by current stage execution.
|
|
||||||
|
|
||||||
`previous_session_artifact` was a live legacy behavior and has been removed:
|
|
||||||
|
|
||||||
- Config validation rejects it as an unsupported Scriptorium input source.
|
|
||||||
- The analyze stage no longer has path-based previous-artifact resolution through `inputs.<name>.path`.
|
|
||||||
- Tests cover canonical previous-session sources and the rejection of the legacy source.
|
|
||||||
- The canonical replacement is `narratio.previous_session.artifact.<configured_artifact_key>`, resolved through the previous-session cache/catalog model.
|
|
||||||
|
|
||||||
## Target Model
|
|
||||||
|
|
||||||
The pipeline config schema should expose only current behavior:
|
|
||||||
|
|
||||||
- Remote storage is configured only through `pipeline.storage.s3.*`.
|
|
||||||
- Generated artifacts are configured only through `pipeline.scriptorium.artifacts`.
|
|
||||||
- Scriptorium artifact inputs use canonical source IDs.
|
|
||||||
- Previous-session artifact inputs use `narratio.previous_session.artifact.<configured_artifact_key>`.
|
|
||||||
- Unknown legacy fields fail strict YAML decoding.
|
|
||||||
|
|
||||||
No compatibility aliases should remain unless a future migration requirement explicitly reintroduces them.
|
|
||||||
|
|
||||||
## Cleanup Order
|
|
||||||
|
|
||||||
### Stage 1: Remove Inert Storage Compatibility Fields
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Remove `pipeline.storage.bucket` and `pipeline.storage.prefix`.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Delete `StorageConfig.Bucket` and `StorageConfig.Prefix`.
|
|
||||||
- Keep `StorageConfig.Backend` and `StorageConfig.S3`.
|
|
||||||
- Confirm all runtime storage paths continue to use `storage.s3.bucket` and `storage.s3.root_prefix`.
|
|
||||||
- Update examples and docs to remove top-level storage `bucket` and `prefix`.
|
|
||||||
- Add or update strict-decode tests proving `pipeline.storage.bucket` and `pipeline.storage.prefix` are rejected.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Existing S3 workflows still pass with `pipeline.storage.s3.bucket`.
|
|
||||||
- Pipeline configs containing top-level `storage.bucket` or `storage.prefix` fail to load.
|
|
||||||
- No docs or examples present those fields as available.
|
|
||||||
|
|
||||||
### Stage 2: Remove Legacy Analyzer Schema and Adapter Surface
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Remove the unused analyzer configuration and adapter contract.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Delete `PipelineConfig.Analyzer`.
|
|
||||||
- Delete `AnalyzerConfig` and `ArtifactSettings`.
|
|
||||||
- Remove analyzer timeout validation.
|
|
||||||
- Remove `stage.Env.Analyzer`.
|
|
||||||
- Delete `internal/adapters/analyzer` if no remaining code imports it.
|
|
||||||
- Remove `pipeline.analyzer.*` from tests, examples, and docs.
|
|
||||||
- Add or update strict-decode tests proving `pipeline.analyzer` is rejected.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Analyze behavior remains fully Scriptorium-backed.
|
|
||||||
- No runtime code imports `internal/adapters/analyzer`.
|
|
||||||
- Pipeline configs containing `pipeline.analyzer` fail to load.
|
|
||||||
- Contributor and internal adapter docs no longer list the analyzer adapter.
|
|
||||||
|
|
||||||
### Stage 3: Remove Path-Based Previous Session Artifact Source
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Remove `previous_session_artifact` and require canonical previous-session artifact sources.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Remove `previous_session_artifact` from supported Scriptorium input sources.
|
|
||||||
- Remove analyze-stage special-case handling that resolves `inputs.<name>.path` for previous artifacts.
|
|
||||||
- Keep canonical handling for `narratio.previous_session.artifact.<configured_artifact_key>`.
|
|
||||||
- Rewrite tests that use `previous_session_artifact` to use canonical sources and prepared previous-cache fixtures.
|
|
||||||
- Add validation tests proving `previous_session_artifact` is rejected.
|
|
||||||
- Update docs to remove the legacy path-based source and document only canonical previous-session sources.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- `pipeline.scriptorium.artifacts.*.inputs.*.source: previous_session_artifact` fails validation.
|
|
||||||
- Canonical previous-session sources continue to work for required and optional inputs.
|
|
||||||
- Prepare/restore previous-cache behavior remains unchanged.
|
|
||||||
- No docs or examples mention `previous_session_artifact` as supported.
|
|
||||||
|
|
||||||
## Test Guidance
|
|
||||||
|
|
||||||
Run focused tests after each stage:
|
|
||||||
|
|
||||||
- `go test ./internal/config -v`
|
|
||||||
- `go test ./internal/stage -run Analyze -v`
|
|
||||||
- `go test ./internal/app -v`
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
For Stage 1, focus on config load/strict-decode and S3 workflow regression tests.
|
|
||||||
|
|
||||||
For Stage 2, focus on compile-time removal, config strict-decode tests, and full app/stage tests to catch stale adapter references.
|
|
||||||
|
|
||||||
For Stage 3, focus on Scriptorium config validation, analyze-stage input resolution, previous-cache behavior, and restore/analyze workflows.
|
|
||||||
|
|
||||||
## Documentation Updates
|
|
||||||
|
|
||||||
Update current-behavior docs only after the corresponding code removal lands:
|
|
||||||
|
|
||||||
- `docs/config.md`
|
|
||||||
- `docs/cli.md`, only if command behavior text references removed fields.
|
|
||||||
- `docs/operations.md`, only if operator workflow text references removed fields.
|
|
||||||
- `docs/internal/stage-analyze.md`
|
|
||||||
- `docs/internal/adapters.md`
|
|
||||||
- `examples/pipeline.full.annotated.yml`
|
|
||||||
- `examples/pipeline.production.yml`
|
|
||||||
|
|
||||||
Do not preserve removed fields in examples as compatibility notes. The goal is to make strict config behavior and documentation line up.
|
|
||||||
|
|
||||||
## Assumptions
|
|
||||||
|
|
||||||
- This is a hard cleanup; no backward-compatible aliases are retained.
|
|
||||||
- Current production configs can be migrated to `storage.s3.*`, Scriptorium artifacts, and canonical previous-session sources before this lands.
|
|
||||||
- Removing the unused analyzer adapter does not block any active stage behavior.
|
|
||||||
- The cleanup should be implemented in the listed order so inert schema removal is separated from behavior removal.
|
|
||||||
@@ -1,255 +0,0 @@
|
|||||||
# Roadmap: Session-Oriented CLI Cleanup
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
## Problem
|
|
||||||
|
|
||||||
Narratio's public CLI has accumulated too many top-level commands. Several
|
|
||||||
commands are session-scoped operator helpers, but they currently appear as
|
|
||||||
independent top-level verbs:
|
|
||||||
|
|
||||||
- `plan`
|
|
||||||
- `status`
|
|
||||||
- `restore`
|
|
||||||
- `artifacts list`
|
|
||||||
- `locks`
|
|
||||||
- `session validate`
|
|
||||||
- `session init`
|
|
||||||
|
|
||||||
This makes the command surface harder to learn because the CLI does not clearly
|
|
||||||
separate primary workflow actions from session inspection, initialization,
|
|
||||||
restore, and helper operations.
|
|
||||||
|
|
||||||
## Target Model
|
|
||||||
|
|
||||||
Keep primary workflow commands at top level:
|
|
||||||
|
|
||||||
- `run`
|
|
||||||
- `run-stage`
|
|
||||||
- `resume`
|
|
||||||
- `analyze`
|
|
||||||
- `publish`
|
|
||||||
- `clean`
|
|
||||||
- `session`
|
|
||||||
|
|
||||||
Keep `clean` top-level because it can operate on one session or all local
|
|
||||||
sessions and is a workspace maintenance command, not only a session helper.
|
|
||||||
|
|
||||||
Move session-scoped helper commands under `narratio session` and use positional
|
|
||||||
session identifiers:
|
|
||||||
|
|
||||||
- `narratio session init <session_id> [--remote|--output <path>] [--flags]`
|
|
||||||
- `narratio session validate <session_id> [--flags]`
|
|
||||||
- `narratio session status <session_id> [--flags]`
|
|
||||||
- `narratio session plan <session_id> [--flags]`
|
|
||||||
- `narratio session restore <session_id> [--flags]`
|
|
||||||
- `narratio session artifacts <session_id> [--remote] [--flags]`
|
|
||||||
- `narratio session locks <session_id> [--flags]`
|
|
||||||
- `narratio session locks add <session_id> <source> [--reason <text>] [--force] [--flags]`
|
|
||||||
- `narratio session locks remove <session_id> <source> [--flags]`
|
|
||||||
|
|
||||||
Update top-level workflow commands to use positional session identifiers:
|
|
||||||
|
|
||||||
- `narratio run <session_id> [--flags]`
|
|
||||||
- `narratio resume <session_id> [--flags]`
|
|
||||||
- `narratio analyze <session_id> [--flags]`
|
|
||||||
- `narratio publish <session_id> [--flags]`
|
|
||||||
- `narratio run-stage <stage> <session_id> [--flags]`
|
|
||||||
|
|
||||||
The positional session ID replaces `--session-id` as the primary public
|
|
||||||
interface. Existing `--config`, `--campaign`, `--session`, and
|
|
||||||
`--previous-session-id` flags remain available where they are meaningful.
|
|
||||||
|
|
||||||
## Command Mapping
|
|
||||||
|
|
||||||
| Current command | Target command |
|
|
||||||
| --- | --- |
|
|
||||||
| `narratio run --session-id <id>` | `narratio run <id>` |
|
|
||||||
| `narratio resume --session-id <id>` | `narratio resume <id>` |
|
|
||||||
| `narratio analyze --session-id <id>` | `narratio analyze <id>` |
|
|
||||||
| `narratio publish --session-id <id>` | `narratio publish <id>` |
|
|
||||||
| `narratio run-stage [flags] <stage> --session-id <id>` | `narratio run-stage <stage> <id> [flags]` |
|
|
||||||
| `narratio plan --session-id <id>` | `narratio session plan <id>` |
|
|
||||||
| `narratio status --session-id <id>` | `narratio session status <id>` |
|
|
||||||
| `narratio restore --session-id <id>` | `narratio session restore <id>` |
|
|
||||||
| `narratio artifacts list --session-id <id>` | `narratio session artifacts <id>` |
|
|
||||||
| `narratio locks --session-id <id>` | `narratio session locks <id>` |
|
|
||||||
| `narratio locks add --session-id <id> <source>` | `narratio session locks add <id> <source>` |
|
|
||||||
| `narratio locks remove --session-id <id> <source>` | `narratio session locks remove <id> <source>` |
|
|
||||||
| `narratio session validate --session-id <id>` | `narratio session validate <id>` |
|
|
||||||
| `narratio session init --session-id <id>` | `narratio session init <id>` |
|
|
||||||
| `narratio clean --session-id <id>` | `narratio clean <id>` |
|
|
||||||
| `narratio clean --all` | unchanged |
|
|
||||||
|
|
||||||
`clean` remains top-level, but its session-scoped form should also move from
|
|
||||||
`--session-id` to positional `<session_id>` for consistency.
|
|
||||||
|
|
||||||
## Compatibility Policy
|
|
||||||
|
|
||||||
This is a hard public CLI cleanup after the migration step lands.
|
|
||||||
|
|
||||||
During Step 1, old forms may remain as compatibility aliases to keep the
|
|
||||||
implementation reviewable. During Step 2, remove the old forms from command
|
|
||||||
dispatch, tests, docs, and examples:
|
|
||||||
|
|
||||||
- remove top-level `plan`;
|
|
||||||
- remove top-level `status`;
|
|
||||||
- remove top-level `restore`;
|
|
||||||
- remove top-level `artifacts`;
|
|
||||||
- remove top-level `locks`;
|
|
||||||
- remove `--session-id` from the public command syntax for session-aware
|
|
||||||
commands.
|
|
||||||
|
|
||||||
Do not keep long-term deprecated aliases unless a later roadmap explicitly
|
|
||||||
chooses a compatibility window.
|
|
||||||
|
|
||||||
`status --manifest` does not fit the session-oriented command shape. Remove it
|
|
||||||
from the public CLI in this cleanup. If direct manifest inspection is needed
|
|
||||||
later, add a separate diagnostic command in a future roadmap rather than keeping
|
|
||||||
it as a special case in `session status`.
|
|
||||||
|
|
||||||
## Implementation Step 1: Add New Session-Oriented Interface
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Add the target command forms while preserving current behavior internally.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Add positional session ID parsing helpers in `internal/app`.
|
|
||||||
- Keep the existing `loadCommandConfig` behavior and populate
|
|
||||||
`config.SessionLoadOptions.SessionID` from the positional ID.
|
|
||||||
- Add or update command wrappers:
|
|
||||||
- `Run(ctx, args, out)` parses `run <session_id>`.
|
|
||||||
- `Resume(ctx, args, out)` parses `resume <session_id>`.
|
|
||||||
- `Analyze(ctx, args, out)` parses `analyze <session_id>`.
|
|
||||||
- `Publish(ctx, args, out)` parses `publish <session_id>`.
|
|
||||||
- `RunStage(ctx, args, out)` parses `run-stage <stage> <session_id>`.
|
|
||||||
- `Clean(ctx, args, out)` parses `clean <session_id>` and keeps
|
|
||||||
`clean --all`.
|
|
||||||
- Extend `Session(ctx, args, out)` dispatch to support:
|
|
||||||
- `init <session_id>`
|
|
||||||
- `validate <session_id>`
|
|
||||||
- `status <session_id>`
|
|
||||||
- `plan <session_id>`
|
|
||||||
- `restore <session_id>`
|
|
||||||
- `artifacts <session_id>`
|
|
||||||
- `locks <session_id>`
|
|
||||||
- `locks add <session_id> <source>`
|
|
||||||
- `locks remove <session_id> <source>`
|
|
||||||
- Keep storage access through the existing app-level object-store helper.
|
|
||||||
- Keep AWS SDK details behind storage adapters.
|
|
||||||
- Keep the runner, stages, manifest behavior, archive behavior, restore
|
|
||||||
planning, lock semantics, and artifact catalog behavior unchanged.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- New forms execute the same code paths and produce equivalent results.
|
|
||||||
- Positional session ID mismatch with concrete local or remote `session.yml`
|
|
||||||
fails through existing session identity checks.
|
|
||||||
- Remote session fallback still uses the positional session ID as the lookup
|
|
||||||
value.
|
|
||||||
- Current command tests cover the new forms before old forms are removed.
|
|
||||||
|
|
||||||
## Implementation Step 2: Remove Old Public Forms
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Remove compatibility aliases and make the session-oriented interface the only
|
|
||||||
documented and supported public CLI.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Remove top-level dispatch for:
|
|
||||||
- `plan`
|
|
||||||
- `status`
|
|
||||||
- `restore`
|
|
||||||
- `artifacts`
|
|
||||||
- `locks`
|
|
||||||
- Remove `--session-id` flags from public session-aware commands.
|
|
||||||
- Keep `--previous-session-id` as an expected previous-session identity flag.
|
|
||||||
- Keep explicit `--session <path>` for loading a local concrete session file,
|
|
||||||
but still require the positional session ID for commands that operate on a
|
|
||||||
session.
|
|
||||||
- Remove `status --manifest`.
|
|
||||||
- Update usage text and invalid-command errors.
|
|
||||||
- Update `docs/cli.md` and `docs/operations.md` to use only the new forms.
|
|
||||||
- Update any roadmap docs that mention old helper command names.
|
|
||||||
- Update tests to expect old top-level helper commands and `--session-id` forms
|
|
||||||
to fail.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Top-level command list is exactly:
|
|
||||||
- `run`
|
|
||||||
- `run-stage`
|
|
||||||
- `resume`
|
|
||||||
- `analyze`
|
|
||||||
- `publish`
|
|
||||||
- `clean`
|
|
||||||
- `session`
|
|
||||||
- All session-oriented commands use `narratio session <subcommand>
|
|
||||||
<session_id> [--flags]`, except nested lock mutation forms, which use
|
|
||||||
`narratio session locks add|remove <session_id> <source> [--flags]`.
|
|
||||||
- `clean <session_id>` and `clean --all` remain top-level.
|
|
||||||
- Current-behavior docs and tests no longer advertise `--session-id`.
|
|
||||||
|
|
||||||
## Test Guidance
|
|
||||||
|
|
||||||
Focused tests:
|
|
||||||
|
|
||||||
- `go test ./internal/app -run TestExecute -v`
|
|
||||||
- `go test ./internal/app -run 'Session|Status|Restore|Clean|Locks|Artifacts|Plan|RunStage|Analyze|Publish' -v`
|
|
||||||
- `go test ./internal/config -v`
|
|
||||||
|
|
||||||
Full validation:
|
|
||||||
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Test cases to add or update:
|
|
||||||
|
|
||||||
- `run <session_id>` loads local and remote sessions through the existing
|
|
||||||
config path.
|
|
||||||
- `resume <session_id>`, `analyze <session_id>`, and `publish <session_id>`
|
|
||||||
preserve current behavior.
|
|
||||||
- `run-stage <stage> <session_id>` preserves current run-stage output and
|
|
||||||
force/artifact-selection behavior.
|
|
||||||
- `session plan <session_id>` replaces top-level `plan`.
|
|
||||||
- `session status <session_id>` replaces top-level session status.
|
|
||||||
- `session validate <session_id>` replaces `session validate --session-id`.
|
|
||||||
- `session init <session_id>` writes the same local or remote concrete
|
|
||||||
`session.yml`.
|
|
||||||
- `session restore <session_id>` preserves restore planning/execution.
|
|
||||||
- `session artifacts <session_id> --remote` preserves promoted-output
|
|
||||||
availability reporting.
|
|
||||||
- `session locks <session_id>`, `session locks add <session_id> <source>`, and
|
|
||||||
`session locks remove <session_id> <source>` preserve static/remote lock
|
|
||||||
semantics.
|
|
||||||
- `clean <session_id>` preserves session cleanup behavior, while `clean --all`
|
|
||||||
remains unchanged.
|
|
||||||
- Old top-level helper commands fail after Step 2.
|
|
||||||
- `--session-id` fails after Step 2.
|
|
||||||
- `status --manifest` fails after Step 2.
|
|
||||||
|
|
||||||
## Documentation Guidance
|
|
||||||
|
|
||||||
Update only after implementation lands:
|
|
||||||
|
|
||||||
- `docs/cli.md`
|
|
||||||
- `docs/operations.md`
|
|
||||||
- any internal docs that list command names or examples
|
|
||||||
|
|
||||||
Keep planned behavior only in this roadmap until the command refactor is
|
|
||||||
implemented.
|
|
||||||
|
|
||||||
## Architecture Guardrails
|
|
||||||
|
|
||||||
- Keep Narratio explicit and stage-driven.
|
|
||||||
- Do not introduce a generic workflow or command framework abstraction.
|
|
||||||
- Reuse existing app command helpers where practical.
|
|
||||||
- Keep config loading strict and centralized.
|
|
||||||
- Keep storage details behind `storage.ObjectStore`.
|
|
||||||
- Keep secret-backed object-store construction in `internal/app`.
|
|
||||||
- Preserve manifest-driven resume and restore behavior.
|
|
||||||
- Treat command renaming as a public CLI contract change, not a runtime stage
|
|
||||||
behavior change.
|
|
||||||
@@ -1,287 +0,0 @@
|
|||||||
# Roadmap: Publish Contract
|
|
||||||
|
|
||||||
Status: Planned
|
|
||||||
|
|
||||||
## Problem
|
|
||||||
|
|
||||||
Narratio currently uses several terms for one operator-facing concept:
|
|
||||||
|
|
||||||
- `archive` is the stage that uploads run state and commits remote current
|
|
||||||
state.
|
|
||||||
- `publish` is the convenience command that force-runs the archive stage.
|
|
||||||
- `promote`, `promoted`, and `promote_artifacts` describe configured top-level
|
|
||||||
remote output writes.
|
|
||||||
|
|
||||||
This mixed vocabulary makes the public contract harder to explain. Operators
|
|
||||||
should not need to distinguish "archive the run", "publish the run", and
|
|
||||||
"promote artifacts" when these are all part of the same publish action.
|
|
||||||
|
|
||||||
The public model should use:
|
|
||||||
|
|
||||||
- `publish` for the stage, command, config section, and action;
|
|
||||||
- `published` for an expected remote output that exists at its top-level
|
|
||||||
current destination;
|
|
||||||
- `publish rules` for the configured source-to-destination output rules;
|
|
||||||
- `locked` for sources whose top-level published destination must not be
|
|
||||||
overwritten;
|
|
||||||
- `run history` for immutable per-run records under `runs/<run_id>/`.
|
|
||||||
|
|
||||||
## Target Model
|
|
||||||
|
|
||||||
The public stage is `publish`.
|
|
||||||
|
|
||||||
The convenience command:
|
|
||||||
|
|
||||||
narratio publish <session_id>
|
|
||||||
|
|
||||||
is equivalent to:
|
|
||||||
|
|
||||||
narratio run-stage publish <session_id> --force
|
|
||||||
|
|
||||||
Pipeline configuration uses `publish`:
|
|
||||||
|
|
||||||
publish:
|
|
||||||
enabled: true
|
|
||||||
upload_run: true
|
|
||||||
outputs:
|
|
||||||
- source: narratio.transcript.final_trimmed
|
|
||||||
- source: narratio.artifact.session_recap
|
|
||||||
locks:
|
|
||||||
- source: narratio.artifact.session_recap
|
|
||||||
reason: Final recap was manually edited.
|
|
||||||
|
|
||||||
Publish output rules are source-based. Each rule writes one artifact source to
|
|
||||||
a top-level remote destination. If `dest` is omitted, Narratio derives the
|
|
||||||
destination from the artifact registry or configured artifact output path.
|
|
||||||
|
|
||||||
The mutable remote lock store remains:
|
|
||||||
|
|
||||||
{session_prefix}/locks.yml
|
|
||||||
|
|
||||||
Remote availability output uses `published`:
|
|
||||||
|
|
||||||
Published:
|
|
||||||
- narratio.transcript.final_trimmed remote=published
|
|
||||||
- narratio.artifact.session_recap locked remote=published
|
|
||||||
|
|
||||||
The remote key layout is otherwise unchanged:
|
|
||||||
|
|
||||||
- immutable run history stays under `{session_prefix}/runs/{run_id}/`;
|
|
||||||
- current state stays under `{session_prefix}/current/manifest.json`;
|
|
||||||
- the final commit marker stays `{session_prefix}/current/run_id.txt`;
|
|
||||||
- `current/run_id.txt` is still written last.
|
|
||||||
|
|
||||||
## Compatibility Policy
|
|
||||||
|
|
||||||
This is a hard cutover.
|
|
||||||
|
|
||||||
After implementation:
|
|
||||||
|
|
||||||
- `pipeline.archive` is rejected by strict YAML decoding.
|
|
||||||
- `pipeline.archive.promote_artifacts` is rejected.
|
|
||||||
- `pipeline.workspace.cleanup_after_archive` is rejected.
|
|
||||||
- `pipeline.spool.delete_audio_after_archive` is rejected.
|
|
||||||
- `narratio run-stage archive <session_id>` is an unknown stage.
|
|
||||||
- manifests that record an `archive` stage are not migrated.
|
|
||||||
- old archive/promotion metadata keys are not read as compatibility fallbacks.
|
|
||||||
|
|
||||||
Existing remote objects are not moved or renamed. Remote layout remains stable;
|
|
||||||
the rename changes configuration, stage names, status output, metadata, helper
|
|
||||||
names, tests, examples, and documentation.
|
|
||||||
|
|
||||||
## Implementation Stages
|
|
||||||
|
|
||||||
### Stage 1: Public Schema and Stage Cutover
|
|
||||||
|
|
||||||
Status: Planned
|
|
||||||
|
|
||||||
Switch the public config and stage contract to publish terminology.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Replace `pipeline.archive` with `pipeline.publish`.
|
|
||||||
- Replace `archive.promote_artifacts` with `publish.outputs`.
|
|
||||||
- Keep output rule fields:
|
|
||||||
- `source`
|
|
||||||
- `dest`
|
|
||||||
- `required`
|
|
||||||
- Replace `pipeline.archive.locks` with `pipeline.publish.locks`.
|
|
||||||
- Rename post-publish cleanup fields:
|
|
||||||
- `pipeline.workspace.cleanup_after_publish`
|
|
||||||
- `pipeline.spool.delete_audio_after_publish`
|
|
||||||
- Rename the registered stage from `archive` to `publish`.
|
|
||||||
- Update stage order so `publish` runs after `analyze` and before `notify`.
|
|
||||||
- Update top-level `narratio publish` to target stage `publish`.
|
|
||||||
- Keep `run-stage --artifacts <names> publish` support.
|
|
||||||
- Reject `run-stage --artifacts <names>` for stages other than `analyze` and
|
|
||||||
`publish`.
|
|
||||||
- Preserve the remote commit ordering and storage adapter boundaries.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- `narratio run-stage publish <session_id>` executes the publish stage.
|
|
||||||
- `narratio publish <session_id>` force-runs the publish stage.
|
|
||||||
- `narratio run-stage archive <session_id>` fails clearly as an unknown stage.
|
|
||||||
- Old archive config fields fail strict decoding.
|
|
||||||
- New publish config fields load, default, and validate.
|
|
||||||
|
|
||||||
### Stage 2: Runtime Terminology and Metadata Cutover
|
|
||||||
|
|
||||||
Status: Planned
|
|
||||||
|
|
||||||
Rename implementation concepts and runtime output to publish terminology.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Rename archive/promotion config and runtime types conceptually to
|
|
||||||
publish/output terms.
|
|
||||||
- Rename the remote key helper intent from promoted artifact to published
|
|
||||||
output while keeping generated keys unchanged.
|
|
||||||
- Change helper output:
|
|
||||||
- `Promoted:` becomes `Published:`
|
|
||||||
- `remote=promoted` becomes `remote=published`
|
|
||||||
- lock output uses `published` / `not-published`
|
|
||||||
- Rename publish-stage metadata, including:
|
|
||||||
- `promoted_paths` to `published_paths`
|
|
||||||
- `promoted_files_uploaded` to `published_files_uploaded`
|
|
||||||
- `skipped_optional_promotions` to `skipped_optional_outputs`
|
|
||||||
- `skipped_unselected_promotions` to `skipped_unselected_outputs`
|
|
||||||
- `locked_promotion_count` to `locked_output_count`
|
|
||||||
- `locked_promotions` to `locked_outputs`
|
|
||||||
- Update previous-cache and restore logic to use the `publish` stage and
|
|
||||||
`published_paths` metadata only.
|
|
||||||
- Keep run-local stage output materialization separate from remote publish
|
|
||||||
terminology. If local helper names are confusing, rename them to
|
|
||||||
materialization-oriented names rather than publish names.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Status and artifact helper output use `Published:` and `remote=published`.
|
|
||||||
- Publish metadata contains only publish/output terminology.
|
|
||||||
- Previous-cache and restore behavior works with publish metadata and does not
|
|
||||||
depend on old archive metadata.
|
|
||||||
- Storage adapters still receive explicit keys and no AWS SDK details leak into
|
|
||||||
app or stage logic.
|
|
||||||
|
|
||||||
### Stage 3: Documentation, Examples, and Final Cleanup
|
|
||||||
|
|
||||||
Status: Planned
|
|
||||||
|
|
||||||
Update implemented-behavior docs and remove stale public terminology after the
|
|
||||||
runtime cutover lands.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Update current-behavior docs:
|
|
||||||
- `docs/config.md`
|
|
||||||
- `docs/cli.md`
|
|
||||||
- `docs/operations.md`
|
|
||||||
- `docs/troubleshooting.md`
|
|
||||||
- `docs/architecture.md`
|
|
||||||
- relevant files under `docs/internal/`
|
|
||||||
- Rename `docs/internal/stage-archive.md` to
|
|
||||||
`docs/internal/stage-publish.md`.
|
|
||||||
- Update internal documentation links and references.
|
|
||||||
- Update examples to use:
|
|
||||||
- `publish.outputs`
|
|
||||||
- `publish.locks`
|
|
||||||
- `cleanup_after_publish`
|
|
||||||
- `delete_audio_after_publish`
|
|
||||||
- Update tests and final searches so old terminology remains only in this
|
|
||||||
roadmap as historical context.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Maintained examples load and validate.
|
|
||||||
- Current-behavior docs describe only implemented publish terminology.
|
|
||||||
- Internal docs describe run history, published outputs, locks, and current
|
|
||||||
commit ordering clearly.
|
|
||||||
- Old user-facing archive/promote wording is removed except where discussing
|
|
||||||
historical behavior in this roadmap.
|
|
||||||
|
|
||||||
## Test Guidance
|
|
||||||
|
|
||||||
Focused tests:
|
|
||||||
|
|
||||||
- `go test ./internal/config -v`
|
|
||||||
- `go test ./internal/app -v`
|
|
||||||
- `go test ./internal/stage -v`
|
|
||||||
- `go test ./internal/artifacts -v`
|
|
||||||
|
|
||||||
Full validation:
|
|
||||||
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Config tests to add or update:
|
|
||||||
|
|
||||||
- `publish.outputs` defaults and validates.
|
|
||||||
- `publish.outputs[].dest` derives from the artifact registry when omitted.
|
|
||||||
- `publish.locks` validates with the same source rules as publish outputs.
|
|
||||||
- old `archive` fails strict decode.
|
|
||||||
- old `promote_artifacts` fails strict decode.
|
|
||||||
- old cleanup fields fail strict decode.
|
|
||||||
|
|
||||||
App and stage tests to add or update:
|
|
||||||
|
|
||||||
- stage order uses `publish` before `notify`.
|
|
||||||
- `run-stage publish` succeeds.
|
|
||||||
- `run-stage archive` fails clearly.
|
|
||||||
- `narratio publish` force-runs the `publish` stage.
|
|
||||||
- `--artifacts` is accepted for `run-stage publish`.
|
|
||||||
- `--artifacts` error text names `analyze` and `publish`.
|
|
||||||
- status and artifact list output show `Published:` and `remote=published`.
|
|
||||||
- lock output says `published` or `not-published`.
|
|
||||||
- previous-cache and restore use `publish` stage metadata.
|
|
||||||
|
|
||||||
Final searches:
|
|
||||||
|
|
||||||
- Config/stage names:
|
|
||||||
- `pipeline.archive`
|
|
||||||
- `archive:`
|
|
||||||
- `promote_artifacts`
|
|
||||||
- `cleanup_after_archive`
|
|
||||||
- `delete_audio_after_archive`
|
|
||||||
- User-facing output:
|
|
||||||
- `Promoted:`
|
|
||||||
- `remote=promoted`
|
|
||||||
- `not-promoted`
|
|
||||||
- Runtime symbols and metadata:
|
|
||||||
- `ArchiveConfig`
|
|
||||||
- `ArchivePromotionRule`
|
|
||||||
- `S3PromotedArtifactKey`
|
|
||||||
- `promoted_paths`
|
|
||||||
- `promoted_files_uploaded`
|
|
||||||
- `locked_promotions`
|
|
||||||
|
|
||||||
Expected remaining matches should be limited to this roadmap and narrowly
|
|
||||||
justified historical references until the roadmap is fully retired.
|
|
||||||
|
|
||||||
## Architecture Guardrails
|
|
||||||
|
|
||||||
- Keep Narratio explicit and stage-driven.
|
|
||||||
- Do not introduce a generic workflow or DAG abstraction.
|
|
||||||
- Keep strict YAML decoding.
|
|
||||||
- Keep remote path construction centralized.
|
|
||||||
- Keep storage details behind `storage.ObjectStore`.
|
|
||||||
- Keep AWS SDK types inside storage adapters.
|
|
||||||
- Preserve manifest-driven resume and restore behavior.
|
|
||||||
- Preserve current-state commit ordering with `current/run_id.txt` written
|
|
||||||
last.
|
|
||||||
- Keep raw secrets out of configs, manifests, logs, generated configs, and
|
|
||||||
publish metadata.
|
|
||||||
- Keep planned behavior only in this roadmap until implementation lands.
|
|
||||||
|
|
||||||
## Assumptions
|
|
||||||
|
|
||||||
- This is a breaking public/config/stage contract change.
|
|
||||||
- No compatibility aliases are retained.
|
|
||||||
- No migration logic is needed for in-progress local manifests.
|
|
||||||
- No migration logic is needed for old remote manifests.
|
|
||||||
- Existing remote objects are not moved or renamed.
|
|
||||||
- `publish` means uploading run history, writing configured published outputs,
|
|
||||||
and committing current state.
|
|
||||||
- `run history` is the preferred term for immutable per-run records under
|
|
||||||
`runs/<run_id>/`.
|
|
||||||
- `archive` remains acceptable only as a generic English concept in historical
|
|
||||||
roadmap context, not as a public Narratio command, config field, stage name,
|
|
||||||
or metadata term after implementation.
|
|
||||||
@@ -1,210 +0,0 @@
|
|||||||
# Roadmap: Transcript Artifact Naming
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
## Problem
|
|
||||||
|
|
||||||
Narratio's built-in transcript artifact names and canonical paths currently mix
|
|
||||||
operator-facing artifact meaning with historical stage and tool terminology:
|
|
||||||
|
|
||||||
- `narratio.transcript.merged` maps to `transcripts/merged.json`.
|
|
||||||
- `narratio.transcript.polished` maps to `transcripts/processed.json`.
|
|
||||||
- `narratio.transcript.full` maps to `transcripts/normalized.json`.
|
|
||||||
- `narratio.transcript.trimmed` maps to `transcripts/trimmed.json`.
|
|
||||||
|
|
||||||
This makes the public artifact surface harder to reason about. Operators see
|
|
||||||
`full`, `normalized`, `processed`, `polished`, `merged`, and `trimmed` used in
|
|
||||||
different places for the same transcript lineage.
|
|
||||||
|
|
||||||
The transcript source IDs, canonical paths, and manifest output kinds should
|
|
||||||
use one vocabulary based on each transcript's role in the session artifact
|
|
||||||
model.
|
|
||||||
|
|
||||||
## Target Model
|
|
||||||
|
|
||||||
Built-in transcript artifacts should use these public source IDs, canonical
|
|
||||||
paths, and manifest output kinds:
|
|
||||||
|
|
||||||
| Source ID | Canonical path | Output kind | Meaning |
|
|
||||||
| --- | --- | --- | --- |
|
|
||||||
| `narratio.transcript.base` | `transcripts/base.json` | `transcript_base` | First unified transcript produced by merging per-speaker raw transcripts. |
|
|
||||||
| `narratio.transcript.polished` | `transcripts/polished.json` | `transcript_polished` | Audita-polished transcript. |
|
|
||||||
| `narratio.transcript.final` | `transcripts/final.json` | `transcript_final` | Full final transcript after normalization. |
|
|
||||||
| `narratio.transcript.final_trimmed` | `transcripts/final.trimmed.json` | `transcript_final_trimmed` | Trimmed version of the final transcript. |
|
|
||||||
|
|
||||||
Stage names remain process-oriented and unchanged:
|
|
||||||
|
|
||||||
- `merge`
|
|
||||||
- `polish`
|
|
||||||
- `normalize`
|
|
||||||
- `trim`
|
|
||||||
|
|
||||||
Downstream adapter contracts also remain process-oriented. The rename changes
|
|
||||||
Narratio's artifact model, canonical paths, config examples, archive promotion
|
|
||||||
sources, lock sources, status output, and documentation. It should not rename
|
|
||||||
the stages themselves or move external integration details into stage logic.
|
|
||||||
|
|
||||||
## Compatibility Policy
|
|
||||||
|
|
||||||
This is a hard cutover.
|
|
||||||
|
|
||||||
After implementation, these old source IDs should be rejected:
|
|
||||||
|
|
||||||
- `narratio.transcript.merged`
|
|
||||||
- `narratio.transcript.full`
|
|
||||||
- `narratio.transcript.trimmed`
|
|
||||||
|
|
||||||
These old canonical paths should not be compatibility fallbacks:
|
|
||||||
|
|
||||||
- `transcripts/merged.json`
|
|
||||||
- `transcripts/processed.json`
|
|
||||||
- `transcripts/normalized.json`
|
|
||||||
- `transcripts/trimmed.json`
|
|
||||||
|
|
||||||
Existing remote archives are not migrated automatically. Operators who want
|
|
||||||
new promoted keys for old sessions should republish those sessions after
|
|
||||||
updating configuration.
|
|
||||||
|
|
||||||
## Implementation Stages
|
|
||||||
|
|
||||||
### Stage 1: Centralize Transcript Artifact Naming
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Consolidate transcript artifact source IDs, canonical paths, and output kinds
|
|
||||||
in the artifact/path layer before changing runtime behavior.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Add or consolidate constants/helpers for built-in transcript source IDs.
|
|
||||||
- Add or consolidate constants/helpers for canonical transcript paths.
|
|
||||||
- Add or consolidate constants/helpers for transcript manifest output kinds.
|
|
||||||
- Keep source ID, path, and output-kind mappings in one registry or one
|
|
||||||
obviously shared artifact model.
|
|
||||||
- Update artifact registry tests to prove the target mapping.
|
|
||||||
- Avoid changing stage output behavior in this stage unless the implementation
|
|
||||||
is simpler and still reviewable.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- There is one clear source of truth for built-in transcript artifact names,
|
|
||||||
paths, and output kinds.
|
|
||||||
- Tests prove the new target mapping in the artifact layer.
|
|
||||||
- No generic workflow abstraction is introduced.
|
|
||||||
|
|
||||||
### Stage 2: Rename Runtime Outputs and Defaults
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Switch runtime behavior to the new transcript artifact model.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Update `merge` to write and record `transcripts/base.json` with
|
|
||||||
`transcript_base`.
|
|
||||||
- Update `polish` to write and record `transcripts/polished.json` with
|
|
||||||
`transcript_polished`.
|
|
||||||
- Update `normalize` to write and record `transcripts/final.json` with
|
|
||||||
`transcript_final`.
|
|
||||||
- Update `trim` to write and record `transcripts/final.trimmed.json` with
|
|
||||||
`transcript_final_trimmed`.
|
|
||||||
- Update normalize and trim defaults to:
|
|
||||||
- `pipeline.normalize.output_path: transcripts/final.json`
|
|
||||||
- `pipeline.trim.output_path: transcripts/final.trimmed.json`
|
|
||||||
- Update built-in artifact resolution, archive promotion destination
|
|
||||||
derivation, archive locks, status output, artifact catalog output,
|
|
||||||
previous-cache resolution, restore planning, and restore execution to use
|
|
||||||
the new registry values.
|
|
||||||
- Ensure old source IDs fail config validation.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- New runs produce the target canonical transcript files.
|
|
||||||
- Manifest outputs use the target output kinds.
|
|
||||||
- Archive promotion and lock validation accept new source IDs and reject old
|
|
||||||
source IDs.
|
|
||||||
- Status and artifact listing display new source IDs.
|
|
||||||
- Restore uses the new canonical paths and does not restore old transcript
|
|
||||||
paths as canonical outputs.
|
|
||||||
|
|
||||||
### Stage 3: Update Tests, Examples, and Current Documentation
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Update all implemented-behavior references after the runtime cutover lands.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Update examples to use `narratio.transcript.final_trimmed` and
|
|
||||||
`transcripts/final.trimmed.json` where trimmed final transcript is intended.
|
|
||||||
- Update examples that refer to full final transcripts to use
|
|
||||||
`narratio.transcript.final` and `transcripts/final.json`.
|
|
||||||
- Update `docs/config.md`, `docs/internal/artifacts.md`, stage docs,
|
|
||||||
CLI examples, operations examples, archive examples, lock examples, and
|
|
||||||
status/artifact-list examples.
|
|
||||||
- Add strict validation tests proving old source IDs are rejected.
|
|
||||||
- Mark roadmap stages implemented only after code, tests, examples, and
|
|
||||||
current-behavior docs agree.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Maintained examples load and validate.
|
|
||||||
- Current-behavior docs describe only implemented new names.
|
|
||||||
- Old names remain only in this roadmap as historical/planning context until
|
|
||||||
this roadmap is retired or archived.
|
|
||||||
|
|
||||||
## Test Guidance
|
|
||||||
|
|
||||||
Run focused tests while implementing:
|
|
||||||
|
|
||||||
- `go test ./internal/artifacts -v`
|
|
||||||
- `go test ./internal/config -v`
|
|
||||||
- `go test ./internal/stage -v`
|
|
||||||
- `go test ./internal/app -v`
|
|
||||||
|
|
||||||
Run full validation before finishing:
|
|
||||||
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Run final searches:
|
|
||||||
|
|
||||||
- Old source IDs:
|
|
||||||
- `narratio.transcript.merged`
|
|
||||||
- `narratio.transcript.full`
|
|
||||||
- `narratio.transcript.trimmed`
|
|
||||||
- Old paths:
|
|
||||||
- `transcripts/merged.json`
|
|
||||||
- `transcripts/processed.json`
|
|
||||||
- `transcripts/normalized.json`
|
|
||||||
- `transcripts/trimmed.json`
|
|
||||||
- Old output kinds:
|
|
||||||
- `transcript_merged`
|
|
||||||
- `transcript_processed`
|
|
||||||
- `transcript_normalized`
|
|
||||||
- `transcript_trimmed`
|
|
||||||
|
|
||||||
Expected remaining matches should be limited to this roadmap's
|
|
||||||
historical/planning references until the roadmap is fully completed.
|
|
||||||
|
|
||||||
## Architecture Guardrails
|
|
||||||
|
|
||||||
- Keep Narratio explicit and stage-driven; do not introduce a generic workflow
|
|
||||||
or DAG abstraction.
|
|
||||||
- Keep path and artifact naming in centralized helpers rather than scattered
|
|
||||||
string concatenation.
|
|
||||||
- Preserve manifest-driven resume behavior.
|
|
||||||
- Keep storage details behind storage adapters.
|
|
||||||
- Do not move Seriatim, Audita, or Scriptorium command details out of their
|
|
||||||
adapter boundaries.
|
|
||||||
- Keep current-behavior documentation in sync only after implementation lands;
|
|
||||||
planned behavior belongs in this roadmap until then.
|
|
||||||
|
|
||||||
## Assumptions
|
|
||||||
|
|
||||||
- The cutover is intentionally not backward-compatible.
|
|
||||||
- Existing remote archive objects are not renamed or migrated automatically.
|
|
||||||
- Stage names and downstream adapter request field names remain unchanged.
|
|
||||||
- The term `base` is preferred over `merged` for the first unified transcript.
|
|
||||||
- The term `final` is preferred over `full` or `normalized` for the full final
|
|
||||||
transcript.
|
|
||||||
- The trimmed final path is `transcripts/final.trimmed.json`.
|
|
||||||
@@ -1,379 +1,300 @@
|
|||||||
# Troubleshooting
|
# Troubleshooting
|
||||||
|
|
||||||
## Purpose
|
Operational diagnosis guide for common Narratio failures.
|
||||||
Canonical operator troubleshooting guide for recurring implemented Narratio failures.
|
|
||||||
|
|
||||||
## Config file discovery failure
|
## Config file not found
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
- `run`, `resume`, `run-stage`, `session plan`, or `session restore` fails with config/session not found.
|
|
||||||
|
|
||||||
Likely Cause:
|
- command fails to resolve `pipeline.yml`, `campaign.yml`, or `session.yml`.
|
||||||
- `pipeline.yml` or `session.yml` is missing from system discovery paths.
|
|
||||||
- the selected campaign ID does not exist under `pipeline.campaigns.root`.
|
Likely causes:
|
||||||
- a local working-directory config file was not passed explicitly.
|
|
||||||
|
- missing files in default search paths;
|
||||||
|
- wrong campaign selection;
|
||||||
|
- omitted explicit flags.
|
||||||
|
|
||||||
Diagnostics:
|
Diagnostics:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml
|
narratio session plan 2026-04-04
|
||||||
ls -l /usr/local/etc/narratio/session.yml /etc/narratio/session.yml
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe Fix:
|
Safe fix:
|
||||||
- pass explicit `--config`, `--campaign <id>`, `--campaign-file <path>`, and `--session` as appropriate.
|
|
||||||
- or place files in documented discovery paths and set `pipeline.campaigns.default_campaign_id`.
|
|
||||||
|
|
||||||
Links:
|
- pass explicit `--config`, `--campaign` or `--campaign-file`, and `--session`.
|
||||||
- [docs/config.md](./config.md)
|
|
||||||
- [docs/cli.md](./cli.md)
|
|
||||||
|
|
||||||
## Templated session file rejected
|
## Session template placeholders rejected
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
- load fails with a message that `session.yml must be concrete`.
|
|
||||||
|
|
||||||
Likely Cause:
|
- load error says session file must be concrete or contains `{{ ... }}` placeholders.
|
||||||
- a template authoring file such as `session.template.yml` was passed to `--session` or uploaded as remote `session.yml`.
|
|
||||||
- `session.yml` still contains `{{ ... }}` placeholders.
|
Likely cause:
|
||||||
|
|
||||||
|
- using template content as runtime session config.
|
||||||
|
|
||||||
Diagnostics:
|
Diagnostics:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session plan 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session ./session.yml
|
narratio session validate 2026-04-04 --session /path/session.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe Fix:
|
Safe fix:
|
||||||
- generate concrete YAML with `narratio session init`.
|
|
||||||
- pass the generated concrete `session.yml` to downstream commands or upload it through `session init --remote`.
|
|
||||||
|
|
||||||
Links:
|
- generate concrete session YAML with `narratio session init`.
|
||||||
- [docs/config.md](./config.md)
|
|
||||||
|
|
||||||
## Strict YAML decode or validation failure
|
## Strict decode or schema validation failure
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
- config load fails with unknown field or validation error.
|
|
||||||
|
|
||||||
Likely Cause:
|
- unknown field / invalid value error during config load.
|
||||||
- typo/stale field name.
|
|
||||||
- missing required fields or invalid constraints.
|
Likely cause:
|
||||||
|
|
||||||
|
- stale field name, typo, invalid enum, or invalid duration/path format.
|
||||||
|
|
||||||
Diagnostics:
|
Diagnostics:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session plan 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml
|
narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe Fix:
|
Safe fix:
|
||||||
- align fields/values to canonical config reference and examples.
|
|
||||||
|
|
||||||
Links:
|
- align config with [docs/config.md](./config.md) and maintained files under `examples/`.
|
||||||
- [docs/config.md](./config.md)
|
|
||||||
- [examples/](../examples/)
|
|
||||||
|
|
||||||
## `--artifacts` selection failure
|
## Audio mode conflict
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
- `run`/`resume`/`run-stage` fails with invalid or unknown artifact selection.
|
|
||||||
|
|
||||||
Likely Cause:
|
- validation fails on session audio configuration.
|
||||||
- `--artifacts` contains blank names or unknown artifact keys.
|
|
||||||
- `pipeline.scriptorium.artifacts` missing while using `--artifacts`.
|
Likely cause:
|
||||||
|
|
||||||
Diagnostics:
|
- configured both local and S3 session audio inputs.
|
||||||
|
|
||||||
```bash
|
Safe fix:
|
||||||
narratio run 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --artifacts player_handout
|
|
||||||
```
|
- use local mode (`audio_dir` or `audio_files`) or S3 mode (`audio_s3.prefix`), not both.
|
||||||
|
|
||||||
Safe Fix:
|
## `--artifacts` selection error
|
||||||
- use configured artifact keys only.
|
|
||||||
- ensure `pipeline.scriptorium.artifacts` is defined.
|
Symptom:
|
||||||
|
|
||||||
Links:
|
- unknown artifact key or invalid `--artifacts` usage.
|
||||||
- [docs/cli.md](./cli.md)
|
|
||||||
- [docs/config.md](./config.md)
|
Likely causes:
|
||||||
|
|
||||||
## `run-stage --artifacts` on unsupported stage
|
- key not defined in `pipeline.scriptorium.artifacts`;
|
||||||
|
- empty list entry (for example trailing comma);
|
||||||
Symptom:
|
- `run-stage` used with non-`analyze`/`publish` target.
|
||||||
- `run-stage` fails because `--artifacts` is only supported for `analyze` and `archive`.
|
|
||||||
|
Safe fix:
|
||||||
Likely Cause:
|
|
||||||
- `--artifacts` was used with a stage other than `analyze` or `archive`.
|
- provide only configured keys and use `--artifacts` with supported commands/stages.
|
||||||
|
|
||||||
Diagnostics:
|
## Previous-session artifact input missing
|
||||||
|
|
||||||
```bash
|
Symptom:
|
||||||
narratio run-stage polish 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --artifacts session_recap
|
|
||||||
```
|
- prepare/analyze fails due to missing required previous-session artifact cache input.
|
||||||
|
|
||||||
Safe Fix:
|
Likely causes:
|
||||||
- use `--artifacts` only with `run-stage analyze ...` or `run-stage archive ...`.
|
|
||||||
|
- missing `session.previous_session_id`;
|
||||||
Links:
|
- previous artifact not restored/published for source session.
|
||||||
- [docs/cli.md](./cli.md)
|
|
||||||
|
|
||||||
## Configured artifact dependency/input validation failure
|
|
||||||
|
|
||||||
Symptom:
|
|
||||||
- config validation fails for `depends_on`, `narratio.artifact.<name>` source, or artifact output path.
|
|
||||||
|
|
||||||
Likely Cause:
|
|
||||||
- `narratio.artifact.<name>` source missing matching `depends_on` key.
|
|
||||||
- dependency references unknown artifact key.
|
|
||||||
- dependency self-reference or enabled dependency cycle.
|
|
||||||
- artifact output path missing/invalid/outside `artifacts/` root.
|
|
||||||
|
|
||||||
Diagnostics:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio session plan 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
Safe Fix:
|
|
||||||
- ensure artifact-to-artifact inputs have explicit `depends_on` entries using artifact keys.
|
|
||||||
- ensure referenced artifacts exist and define valid `output_path` values.
|
|
||||||
- keep output paths relative and under `artifacts/`.
|
|
||||||
|
|
||||||
Links:
|
|
||||||
- [docs/config.md](./config.md)
|
|
||||||
- [docs/internal/stage-analyze.md](./internal/stage-analyze.md)
|
|
||||||
|
|
||||||
## Required configured artifact input unavailable at analyze time
|
|
||||||
|
|
||||||
Symptom:
|
|
||||||
- analyze fails because configured input source is unavailable.
|
|
||||||
|
|
||||||
Likely Cause:
|
|
||||||
- required upstream configured artifact was not selected/executed this run.
|
|
||||||
- non-executable dependency output file is missing or invalid on disk.
|
|
||||||
|
|
||||||
Diagnostics:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio session status 2026-04-04
|
|
||||||
narratio run-stage analyze 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --artifacts player_handout
|
|
||||||
```
|
|
||||||
|
|
||||||
Safe Fix:
|
|
||||||
- run analyze with needed artifacts selected.
|
|
||||||
- or ensure dependency output file exists at configured path and is valid.
|
|
||||||
|
|
||||||
Links:
|
|
||||||
- [docs/operations.md](./operations.md)
|
|
||||||
- [docs/config.md](./config.md)
|
|
||||||
|
|
||||||
## Manifest/status path failure
|
|
||||||
|
|
||||||
Symptom:
|
|
||||||
- `session status` fails because config/session state is missing, unreadable, or invalid.
|
|
||||||
|
|
||||||
Likely Cause:
|
|
||||||
- wrong session ID.
|
|
||||||
- wrong config/campaign/session file selected.
|
|
||||||
- manifest removed after cleanup.
|
|
||||||
|
|
||||||
Diagnostics:
|
Diagnostics:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
narratio session validate 2026-04-04
|
||||||
narratio session status 2026-04-04
|
narratio session status 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe Fix:
|
Safe fix:
|
||||||
- use the same session ID and config files that will be used for `run`, `resume`, or `run-stage`.
|
|
||||||
|
|
||||||
Links:
|
```bash
|
||||||
- [docs/cli.md](./cli.md)
|
narratio session restore 2026-04-04
|
||||||
- [docs/operations.md](./operations.md)
|
```
|
||||||
|
|
||||||
|
or rerun prepare after correcting session config:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage prepare 2026-04-04 --force
|
||||||
|
```
|
||||||
|
|
||||||
## Session lock conflict (`.lock`)
|
## Session lock conflict (`.lock`)
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
- `run`, `resume`, `run-stage`, or `session restore` fails with lock conflict for session workdir.
|
|
||||||
|
|
||||||
Likely Cause:
|
- command fails acquiring session lock.
|
||||||
- another Narratio process is running same session.
|
|
||||||
- stale lock from interrupted prior run.
|
Likely causes:
|
||||||
|
|
||||||
|
- another process is running for the same session;
|
||||||
|
- stale lock left by interrupted process.
|
||||||
|
|
||||||
Diagnostics:
|
Diagnostics:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ls -l {workspace.root}/work/{campaign}/{session_id}/.lock
|
ls -l {workspace.root}/work/{campaign}/{session_id}/.lock
|
||||||
cat {workspace.root}/work/{campaign}/{session_id}/.lock
|
|
||||||
ps aux | grep narratio
|
ps aux | grep narratio
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe Fix:
|
Safe fix:
|
||||||
- wait for active process to finish.
|
|
||||||
- if no process is active, remove only stale session `.lock` file.
|
|
||||||
|
|
||||||
Links:
|
- wait for active process completion;
|
||||||
- [docs/operations.md](./operations.md)
|
- remove stale lock only after confirming no live process owns it.
|
||||||
- [docs/internal/workspace.md](./internal/workspace.md)
|
|
||||||
|
|
||||||
## Restore remote current pointer or manifest missing
|
|
||||||
|
|
||||||
Symptom:
|
|
||||||
- `session restore` fails with remote current pointer or current manifest errors.
|
|
||||||
|
|
||||||
Likely Cause:
|
|
||||||
- `current/run_id.txt` was never published.
|
|
||||||
- `current/manifest.json` is missing for the session prefix.
|
|
||||||
- archive commit did not complete.
|
|
||||||
|
|
||||||
Diagnostics:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio session restore 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --dry-run
|
|
||||||
```
|
|
||||||
|
|
||||||
Safe Fix:
|
|
||||||
- verify archive stage succeeded for the target session.
|
|
||||||
- rerun/archive from a healthy source workspace so current pointers are published.
|
|
||||||
|
|
||||||
Links:
|
|
||||||
- [docs/operations.md](./operations.md)
|
|
||||||
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
|
|
||||||
|
|
||||||
## Restore manifest identity mismatch
|
|
||||||
|
|
||||||
Symptom:
|
|
||||||
- `session restore` fails because remote manifest session or campaign does not match requested values.
|
|
||||||
|
|
||||||
Likely Cause:
|
|
||||||
- wrong positional session ID or wrong session config selected.
|
|
||||||
- archive prefix points to a different campaign/session.
|
|
||||||
|
|
||||||
Diagnostics:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio session restore 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --dry-run
|
|
||||||
```
|
|
||||||
|
|
||||||
Safe Fix:
|
|
||||||
- use the correct session config and positional session ID.
|
|
||||||
- verify campaign/session identity in local config before restore.
|
|
||||||
|
|
||||||
Links:
|
|
||||||
- [docs/config.md](./config.md)
|
|
||||||
- [docs/operations.md](./operations.md)
|
|
||||||
|
|
||||||
## Restore conflict without `--force`
|
## Restore conflict without `--force`
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
- `session restore` fails with `restore conflict` and conflict counts.
|
|
||||||
|
|
||||||
Likely Cause:
|
- restore fails with conflict count.
|
||||||
- local durable file differs from remote file for one or more planned restore paths.
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- local durable files differ from remote restore sources.
|
||||||
|
|
||||||
Diagnostics:
|
Diagnostics:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session restore 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml --dry-run
|
narratio session restore 2026-04-04 --dry-run
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe Fix:
|
Safe fix:
|
||||||
- review planned conflicts.
|
|
||||||
- rerun with `--force` only when remote state should overwrite local state.
|
|
||||||
|
|
||||||
Links:
|
- review conflicts;
|
||||||
- [docs/cli.md](./cli.md)
|
- rerun with `--force` only when remote state should overwrite local.
|
||||||
- [docs/operations.md](./operations.md)
|
|
||||||
|
|
||||||
## Restore report expectations
|
## Restore current-state discovery failure
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
- operator expects restore report file but does not find one.
|
|
||||||
|
|
||||||
Likely Cause:
|
- restore cannot find current pointer or current manifest.
|
||||||
- restore was executed in `--dry-run` mode.
|
|
||||||
- restore failed before report persistence path (for example lock acquisition failure).
|
|
||||||
|
|
||||||
Diagnostics:
|
Likely causes:
|
||||||
|
|
||||||
```bash
|
- no committed publish current state;
|
||||||
ls -l {workspace.root}/work/{campaign}/{session_id}/reports/restore-latest.json
|
- storage credentials or connectivity failure.
|
||||||
```
|
|
||||||
|
|
||||||
Safe Fix:
|
|
||||||
- run non-dry-run restore for durable report output.
|
|
||||||
- resolve lock or early preflight failures and retry.
|
|
||||||
|
|
||||||
Links:
|
|
||||||
- [docs/operations.md](./operations.md)
|
|
||||||
|
|
||||||
## Secrets env-dir or credential-env failure
|
|
||||||
|
|
||||||
Symptom:
|
|
||||||
- startup fails loading secrets directory, or stage fails due to missing credential env vars.
|
|
||||||
|
|
||||||
Likely Cause:
|
|
||||||
- invalid `pipeline.secrets.env_dir` path/permissions.
|
|
||||||
- required credential env var unset/empty.
|
|
||||||
|
|
||||||
Diagnostics:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ls -la /path/to/secrets_dir
|
|
||||||
env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM'
|
|
||||||
```
|
|
||||||
|
|
||||||
Safe Fix:
|
|
||||||
- fix secrets directory and credential env vars.
|
|
||||||
- keep secret values out of YAML.
|
|
||||||
|
|
||||||
Links:
|
|
||||||
- [docs/config.md](./config.md)
|
|
||||||
|
|
||||||
## S3-audio prepare failure
|
|
||||||
|
|
||||||
Symptom:
|
|
||||||
- `prepare` fails in S3 mode (listing/downloading/no audio/backend error).
|
|
||||||
|
|
||||||
Likely Cause:
|
|
||||||
- wrong `session.inputs.audio_s3.prefix`.
|
|
||||||
- no `.flac` files at resolved prefix.
|
|
||||||
- invalid/missing object-store credentials or backend config.
|
|
||||||
- mixed local+S3 audio input config.
|
|
||||||
|
|
||||||
Diagnostics:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio run-stage prepare 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
Safe Fix:
|
|
||||||
- configure exactly one audio source mode.
|
|
||||||
- verify `.flac` files and storage access.
|
|
||||||
|
|
||||||
Links:
|
|
||||||
- [docs/config.md](./config.md)
|
|
||||||
- [docs/operations.md](./operations.md)
|
|
||||||
|
|
||||||
## Archive promotion/current-pointer failure
|
|
||||||
|
|
||||||
Symptom:
|
|
||||||
- archive fails on required promotion source missing or pointer write failure.
|
|
||||||
|
|
||||||
Likely Cause:
|
|
||||||
- required promoted file absent (including analyze outputs not generated for this run).
|
|
||||||
- storage upload failed before `current/run_id.txt` commit marker write.
|
|
||||||
|
|
||||||
Diagnostics:
|
Diagnostics:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session status 2026-04-04
|
narratio session status 2026-04-04
|
||||||
narratio run-stage archive 2026-04-04 --config /path/to/pipeline.yml --campaign-file /path/to/campaign.yml --session /path/to/session.yml
|
narratio session restore 2026-04-04 --dry-run
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe Fix:
|
Safe fix:
|
||||||
- rerun or resume upstream stages to generate required files.
|
|
||||||
- adjust promotion `source`/`dest` rules to match artifacts that must exist.
|
|
||||||
- retry after storage issue is resolved.
|
|
||||||
|
|
||||||
Links:
|
- resolve storage/auth issue;
|
||||||
- [docs/operations.md](./operations.md)
|
- republish from healthy local state if current pointer is missing.
|
||||||
|
|
||||||
|
## Publish output failure
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- publish fails on missing required source, upload error, or commit write.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- required source file not produced;
|
||||||
|
- lock/state expectations mismatch;
|
||||||
|
- remote storage failure.
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session artifacts 2026-04-04 --remote
|
||||||
|
narratio session status 2026-04-04
|
||||||
|
narratio run-stage publish 2026-04-04 --force
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- regenerate missing sources by rerunning prerequisite stages;
|
||||||
|
- correct publish source/destination rules;
|
||||||
|
- retry after storage failure is resolved.
|
||||||
|
|
||||||
|
## Render markdown source missing
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- analyze or publish fails because `narratio.transcript.final_markdown` or `narratio.transcript.final_trimmed_markdown` is unavailable.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- render stage was not executed after transcript changes;
|
||||||
|
- render stage failed before producing canonical markdown outputs.
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session status 2026-04-04
|
||||||
|
narratio run-stage render 2026-04-04 --force
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- rerun render and then retry downstream stage(s):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage render 2026-04-04 --force
|
||||||
|
narratio run-stage analyze 2026-04-04 --force
|
||||||
|
```
|
||||||
|
|
||||||
|
## Secrets or storage credential failure
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- object-store command fails at initialization/auth.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- invalid `pipeline.secrets.env_dir`;
|
||||||
|
- missing credential environment variables;
|
||||||
|
- invalid S3 endpoint/bucket settings.
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ls -la /path/to/secrets_dir
|
||||||
|
env | grep -E 'OBJECT_STORAGE|AWS|AUDITA|SCRIPTORIUM'
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- correct secret-file path and permissions;
|
||||||
|
- provide required env vars;
|
||||||
|
- keep secret values out of YAML.
|
||||||
|
|
||||||
|
## S3 audio prepare failure
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- prepare fails listing/downloading session S3 audio.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- incorrect `session.inputs.audio_s3.prefix`;
|
||||||
|
- no matching `.flac` objects;
|
||||||
|
- storage connectivity or permissions failure.
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage prepare 2026-04-04 --force
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- verify prefix contents and storage access;
|
||||||
|
- keep session audio mode consistent.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [docs/cli.md](./cli.md)
|
||||||
- [docs/config.md](./config.md)
|
- [docs/config.md](./config.md)
|
||||||
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
|
- [docs/operations.md](./operations.md)
|
||||||
|
- [docs/internal/stage-publish.md](./internal/stage-publish.md)
|
||||||
|
|||||||
@@ -4,18 +4,18 @@
|
|||||||
workspace:
|
workspace:
|
||||||
# Optional: defaults to /var/lib/narratio.
|
# Optional: defaults to /var/lib/narratio.
|
||||||
root: /var/lib/narratio/workspace
|
root: /var/lib/narratio/workspace
|
||||||
# Optional: remove run-scoped workdir after successful archive commit.
|
# Optional: remove run-scoped workdir after successful publish commit.
|
||||||
cleanup_after_archive: false
|
cleanup_after_publish: false
|
||||||
|
|
||||||
# Optional: local secret file loader (directory of ENV_VAR_NAME files).
|
# Optional: local secret file loader (directory of ENV_VAR_NAME files).
|
||||||
# secrets:
|
# secrets:
|
||||||
# env_dir: ./secrets
|
# env_dir: ./secrets
|
||||||
|
|
||||||
storage:
|
storage:
|
||||||
# Optional storage backend selector; use "s3" for archive + S3 audio workflows.
|
# Optional storage backend selector; use "s3" for publish + S3 audio workflows.
|
||||||
backend: s3
|
backend: s3
|
||||||
s3:
|
s3:
|
||||||
# Required when using S3 audio or S3 archive uploads.
|
# Required when using S3 audio or S3 publish uploads.
|
||||||
bucket: my-dnd-archive
|
bucket: my-dnd-archive
|
||||||
# Optional; defaults to "dnd".
|
# Optional; defaults to "dnd".
|
||||||
root_prefix: dnd
|
root_prefix: dnd
|
||||||
@@ -36,18 +36,24 @@ campaigns:
|
|||||||
spool:
|
spool:
|
||||||
# Optional; defaults to /var/spool/narratio.
|
# Optional; defaults to /var/spool/narratio.
|
||||||
root: /var/spool/narratio
|
root: /var/spool/narratio
|
||||||
# Optional cleanup of run-scoped spool audio after successful archive commit.
|
# Optional cleanup of run-scoped spool audio after successful publish commit.
|
||||||
delete_audio_after_archive: false
|
delete_audio_after_publish: false
|
||||||
|
|
||||||
archive:
|
publish:
|
||||||
# Optional booleans; defaults are true.
|
# Optional booleans; defaults are true.
|
||||||
enabled: true
|
enabled: true
|
||||||
upload_run: true
|
upload_run: true
|
||||||
# Optional promotion rules; sources use Narratio artifact source IDs.
|
# Optional publish output rules; sources use Narratio artifact source IDs.
|
||||||
promote_artifacts:
|
outputs:
|
||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
required: true
|
required: true
|
||||||
|
- source: narratio.transcript.final_markdown
|
||||||
|
dest: transcripts/final.md
|
||||||
|
required: true
|
||||||
|
- source: narratio.transcript.final_trimmed_markdown
|
||||||
|
dest: transcripts/final.trimmed.md
|
||||||
|
required: true
|
||||||
- source: narratio.artifact.session_recap
|
- source: narratio.artifact.session_recap
|
||||||
dest: artifacts/session_recap.md
|
dest: artifacts/session_recap.md
|
||||||
required: true
|
required: true
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
workspace:
|
workspace:
|
||||||
root: /var/lib/narratio/workspace
|
root: /var/lib/narratio/workspace
|
||||||
cleanup_after_archive: true
|
cleanup_after_publish: true
|
||||||
|
|
||||||
storage:
|
storage:
|
||||||
backend: s3
|
backend: s3
|
||||||
@@ -17,15 +17,21 @@ campaigns:
|
|||||||
|
|
||||||
spool:
|
spool:
|
||||||
root: /var/spool/narratio
|
root: /var/spool/narratio
|
||||||
delete_audio_after_archive: true
|
delete_audio_after_publish: true
|
||||||
|
|
||||||
archive:
|
publish:
|
||||||
enabled: true
|
enabled: true
|
||||||
upload_run: true
|
upload_run: true
|
||||||
promote_artifacts:
|
outputs:
|
||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
required: true
|
required: true
|
||||||
|
- source: narratio.transcript.final_markdown
|
||||||
|
dest: transcripts/final.md
|
||||||
|
required: true
|
||||||
|
- source: narratio.transcript.final_trimmed_markdown
|
||||||
|
dest: transcripts/final.trimmed.md
|
||||||
|
required: true
|
||||||
- source: narratio.artifact.session_recap
|
- source: narratio.artifact.session_recap
|
||||||
dest: artifacts/session_recap.md
|
dest: artifacts/session_recap.md
|
||||||
required: true
|
required: true
|
||||||
|
|||||||
@@ -68,6 +68,26 @@ func (n *NoopRunner) Normalize(ctx context.Context, req NormalizeRequest) (Norma
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Render returns the requested output path with placeholder metadata.
|
||||||
|
func (n *NoopRunner) Render(ctx context.Context, req RenderRequest) (RenderResult, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return RenderResult{}, err
|
||||||
|
}
|
||||||
|
if err := materializeRenderPlaceholders(req); err != nil {
|
||||||
|
return RenderResult{}, err
|
||||||
|
}
|
||||||
|
return RenderResult{
|
||||||
|
OutputRenderedPath: req.OutputRenderedPath,
|
||||||
|
StdoutLogPath: req.StdoutLogPath,
|
||||||
|
StderrLogPath: req.StderrLogPath,
|
||||||
|
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||||
|
InvokedBinary: "noop",
|
||||||
|
Format: req.Format,
|
||||||
|
Title: req.Title,
|
||||||
|
Metadata: map[string]any{"placeholder": true},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
// FakeRunner captures merge requests and returns deterministic responses.
|
// FakeRunner captures merge requests and returns deterministic responses.
|
||||||
type FakeRunner struct {
|
type FakeRunner struct {
|
||||||
Requests []MergeRequest
|
Requests []MergeRequest
|
||||||
@@ -79,6 +99,9 @@ type FakeRunner struct {
|
|||||||
TrimRequests []TrimRequest
|
TrimRequests []TrimRequest
|
||||||
TrimErr error
|
TrimErr error
|
||||||
TrimResult TrimResult
|
TrimResult TrimResult
|
||||||
|
RenderRequests []RenderRequest
|
||||||
|
RenderErr error
|
||||||
|
RenderResult RenderResult
|
||||||
}
|
}
|
||||||
|
|
||||||
// Run records request and returns configured response.
|
// Run records request and returns configured response.
|
||||||
@@ -195,6 +218,46 @@ func (f *FakeRunner) Normalize(ctx context.Context, req NormalizeRequest) (Norma
|
|||||||
return res, nil
|
return res, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Render records request and returns configured response.
|
||||||
|
func (f *FakeRunner) Render(ctx context.Context, req RenderRequest) (RenderResult, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return RenderResult{}, err
|
||||||
|
}
|
||||||
|
f.RenderRequests = append(f.RenderRequests, req)
|
||||||
|
if f.RenderErr != nil {
|
||||||
|
return RenderResult{}, f.RenderErr
|
||||||
|
}
|
||||||
|
if err := materializeRenderPlaceholders(req); err != nil {
|
||||||
|
return RenderResult{}, err
|
||||||
|
}
|
||||||
|
res := f.RenderResult
|
||||||
|
if res.OutputRenderedPath == "" {
|
||||||
|
res.OutputRenderedPath = req.OutputRenderedPath
|
||||||
|
}
|
||||||
|
if res.StdoutLogPath == "" {
|
||||||
|
res.StdoutLogPath = req.StdoutLogPath
|
||||||
|
}
|
||||||
|
if res.StderrLogPath == "" {
|
||||||
|
res.StderrLogPath = req.StderrLogPath
|
||||||
|
}
|
||||||
|
if res.GeneratedConfigPath == "" {
|
||||||
|
res.GeneratedConfigPath = req.GeneratedConfigPath
|
||||||
|
}
|
||||||
|
if res.InvokedBinary == "" {
|
||||||
|
res.InvokedBinary = "fake"
|
||||||
|
}
|
||||||
|
if res.Format == "" {
|
||||||
|
res.Format = req.Format
|
||||||
|
}
|
||||||
|
if res.Title == "" {
|
||||||
|
res.Title = req.Title
|
||||||
|
}
|
||||||
|
if res.Metadata == nil {
|
||||||
|
res.Metadata = map[string]any{"fake": true}
|
||||||
|
}
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
|
|
||||||
func materializePlaceholders(req MergeRequest) error {
|
func materializePlaceholders(req MergeRequest) error {
|
||||||
if req.OutputMergedTranscriptPath != "" {
|
if req.OutputMergedTranscriptPath != "" {
|
||||||
if err := subprocess.WriteFileAtomic(req.OutputMergedTranscriptPath, []byte(`{"schema":"seriatim.intermediate.v1","segments":[]}`), 0o644); err != nil {
|
if err := subprocess.WriteFileAtomic(req.OutputMergedTranscriptPath, []byte(`{"schema":"seriatim.intermediate.v1","segments":[]}`), 0o644); err != nil {
|
||||||
@@ -301,3 +364,39 @@ func materializeNormalizePlaceholders(req NormalizeRequest) error {
|
|||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func materializeRenderPlaceholders(req RenderRequest) error {
|
||||||
|
if req.OutputRenderedPath != "" {
|
||||||
|
if err := subprocess.WriteFileAtomic(req.OutputRenderedPath, []byte("# Transcript\n\nRendered markdown placeholder.\n"), 0o644); err != nil {
|
||||||
|
return fmt.Errorf("write rendered transcript %q: %w", req.OutputRenderedPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if req.GeneratedConfigPath != "" {
|
||||||
|
payload := map[string]any{
|
||||||
|
"schema": "seriatim.generated.v1",
|
||||||
|
"placeholder": true,
|
||||||
|
"command": "render",
|
||||||
|
"input_path": req.InputTranscriptPath,
|
||||||
|
"output_path": req.OutputRenderedPath,
|
||||||
|
"format": req.Format,
|
||||||
|
"title": req.Title,
|
||||||
|
"include_timestamps": req.IncludeTimestamps,
|
||||||
|
"include_segment_ids": req.IncludeSegmentIDs,
|
||||||
|
"include_metadata": req.IncludeMetadata,
|
||||||
|
}
|
||||||
|
if err := subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644); err != nil {
|
||||||
|
return fmt.Errorf("write generated config %q: %w", req.GeneratedConfigPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if req.StdoutLogPath != "" {
|
||||||
|
if err := subprocess.WriteFileAtomic(req.StdoutLogPath, []byte("seriatim noop/fake render stdout placeholder\n"), 0o644); err != nil {
|
||||||
|
return fmt.Errorf("write stdout log %q: %w", req.StdoutLogPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if req.StderrLogPath != "" {
|
||||||
|
if err := subprocess.WriteFileAtomic(req.StderrLogPath, []byte("seriatim noop/fake render stderr placeholder\n"), 0o644); err != nil {
|
||||||
|
return fmt.Errorf("write stderr log %q: %w", req.StderrLogPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -148,3 +148,58 @@ func TestFakeRunnerNormalizeError(t *testing.T) {
|
|||||||
t.Fatal("expected error, got nil")
|
t.Fatal("expected error, got nil")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFakeRunnerRenderCapturesRequestAndReturnsPath(t *testing.T) {
|
||||||
|
fake := &FakeRunner{}
|
||||||
|
dir := t.TempDir()
|
||||||
|
req := RenderRequest{
|
||||||
|
GeneratedConfigPath: filepath.Join(dir, "config", "seriatim.render.yml"),
|
||||||
|
InputTranscriptPath: filepath.Join(dir, "transcripts", "final.trimmed.json"),
|
||||||
|
OutputRenderedPath: filepath.Join(dir, "transcripts", "final.trimmed.md"),
|
||||||
|
Format: "markdown",
|
||||||
|
Title: "Session render",
|
||||||
|
IncludeTimestamps: true,
|
||||||
|
IncludeSegmentIDs: false,
|
||||||
|
IncludeMetadata: true,
|
||||||
|
StdoutLogPath: filepath.Join(dir, "logs", "seriatim.render.stdout.log"),
|
||||||
|
StderrLogPath: filepath.Join(dir, "logs", "seriatim.render.stderr.log"),
|
||||||
|
}
|
||||||
|
|
||||||
|
res, err := fake.Render(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Render() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(fake.RenderRequests) != 1 || fake.RenderRequests[0].GeneratedConfigPath == "" {
|
||||||
|
t.Fatalf("render requests = %#v, want captured request", fake.RenderRequests)
|
||||||
|
}
|
||||||
|
if res.OutputRenderedPath != req.OutputRenderedPath {
|
||||||
|
t.Fatalf("rendered path = %q, want %q", res.OutputRenderedPath, req.OutputRenderedPath)
|
||||||
|
}
|
||||||
|
if res.Format != req.Format {
|
||||||
|
t.Fatalf("format = %q, want %q", res.Format, req.Format)
|
||||||
|
}
|
||||||
|
if res.Title != req.Title {
|
||||||
|
t.Fatalf("title = %q, want %q", res.Title, req.Title)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfgData, err := os.ReadFile(req.GeneratedConfigPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read generated config: %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(cfgData), "command: render") {
|
||||||
|
t.Fatalf("generated config = %q, want render command marker", string(cfgData))
|
||||||
|
}
|
||||||
|
for _, path := range []string{req.StdoutLogPath, req.StderrLogPath, req.OutputRenderedPath} {
|
||||||
|
if _, err := os.Stat(path); err != nil {
|
||||||
|
t.Fatalf("expected file %q to exist: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFakeRunnerRenderError(t *testing.T) {
|
||||||
|
fake := &FakeRunner{RenderErr: errors.New("boom")}
|
||||||
|
_, err := fake.Render(context.Background(), RenderRequest{})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error, got nil")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
// Package seriatim declares the adapter contract for transcript merge/normalize/trim execution.
|
// Package seriatim declares the adapter contract for transcript merge/normalize/trim/render execution.
|
||||||
package seriatim
|
package seriatim
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -6,11 +6,12 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Runner is the adapter boundary for seriatim merge/normalize/trim invocations.
|
// Runner is the adapter boundary for seriatim merge/normalize/trim/render invocations.
|
||||||
type Runner interface {
|
type Runner interface {
|
||||||
Run(ctx context.Context, req MergeRequest) (MergeResult, error)
|
Run(ctx context.Context, req MergeRequest) (MergeResult, error)
|
||||||
Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error)
|
Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error)
|
||||||
Trim(ctx context.Context, req TrimRequest) (TrimResult, error)
|
Trim(ctx context.Context, req TrimRequest) (TrimResult, error)
|
||||||
|
Render(ctx context.Context, req RenderRequest) (RenderResult, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
// MergeRequest describes a seriatim merge invocation.
|
// MergeRequest describes a seriatim merge invocation.
|
||||||
@@ -90,3 +91,33 @@ type TrimResult struct {
|
|||||||
KeepSelector string
|
KeepSelector string
|
||||||
Metadata map[string]any
|
Metadata map[string]any
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RenderRequest describes a seriatim render invocation.
|
||||||
|
type RenderRequest struct {
|
||||||
|
Binary string
|
||||||
|
InputTranscriptPath string
|
||||||
|
OutputRenderedPath string
|
||||||
|
Format string
|
||||||
|
Title string
|
||||||
|
IncludeTimestamps bool
|
||||||
|
IncludeSegmentIDs bool
|
||||||
|
IncludeMetadata bool
|
||||||
|
StdoutLogPath string
|
||||||
|
StderrLogPath string
|
||||||
|
GeneratedConfigPath string
|
||||||
|
Timeout time.Duration
|
||||||
|
}
|
||||||
|
|
||||||
|
// RenderResult describes a render output.
|
||||||
|
type RenderResult struct {
|
||||||
|
OutputRenderedPath string
|
||||||
|
StdoutLogPath string
|
||||||
|
StderrLogPath string
|
||||||
|
GeneratedConfigPath string
|
||||||
|
ExitCode int
|
||||||
|
Duration time.Duration
|
||||||
|
InvokedBinary string
|
||||||
|
Format string
|
||||||
|
Title string
|
||||||
|
Metadata map[string]any
|
||||||
|
}
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import (
|
|||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/subprocess"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/subprocess"
|
||||||
)
|
)
|
||||||
@@ -384,6 +385,96 @@ func (r *SubprocessRunner) Normalize(ctx context.Context, req NormalizeRequest)
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Render executes Seriatim render with deterministic flags and validates non-empty text output.
|
||||||
|
func (r *SubprocessRunner) Render(ctx context.Context, req RenderRequest) (RenderResult, error) {
|
||||||
|
if r == nil {
|
||||||
|
return RenderResult{}, fmt.Errorf("seriatim subprocess runner is nil")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(req.InputTranscriptPath) == "" {
|
||||||
|
return RenderResult{}, fmt.Errorf("seriatim render input path is required")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(req.OutputRenderedPath) == "" {
|
||||||
|
return RenderResult{}, fmt.Errorf("seriatim render output path is required")
|
||||||
|
}
|
||||||
|
format := strings.TrimSpace(req.Format)
|
||||||
|
if format == "" {
|
||||||
|
format = "markdown"
|
||||||
|
}
|
||||||
|
if format != "markdown" {
|
||||||
|
return RenderResult{}, fmt.Errorf("seriatim render format %q is unsupported", req.Format)
|
||||||
|
}
|
||||||
|
|
||||||
|
binary := r.binary
|
||||||
|
if strings.TrimSpace(req.Binary) != "" {
|
||||||
|
binary = strings.TrimSpace(req.Binary)
|
||||||
|
}
|
||||||
|
|
||||||
|
timeout := r.timeout
|
||||||
|
if req.Timeout < 0 {
|
||||||
|
return RenderResult{}, fmt.Errorf("seriatim render timeout must be >= 0")
|
||||||
|
}
|
||||||
|
if req.Timeout > 0 {
|
||||||
|
timeout = req.Timeout
|
||||||
|
}
|
||||||
|
|
||||||
|
args := buildRenderArgs(req, format)
|
||||||
|
if req.GeneratedConfigPath != "" {
|
||||||
|
if err := writeRenderInvocationConfig(req, args, binary, timeout, format); err != nil {
|
||||||
|
return RenderResult{}, fmt.Errorf("write seriatim render invocation config %q: %w", req.GeneratedConfigPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
runRes, err := subprocess.Run(ctx, subprocess.RunRequest{
|
||||||
|
Executable: binary,
|
||||||
|
Args: args,
|
||||||
|
Timeout: timeout,
|
||||||
|
StdoutLogPath: req.StdoutLogPath,
|
||||||
|
StderrLogPath: req.StderrLogPath,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return RenderResult{
|
||||||
|
OutputRenderedPath: req.OutputRenderedPath,
|
||||||
|
StdoutLogPath: req.StdoutLogPath,
|
||||||
|
StderrLogPath: req.StderrLogPath,
|
||||||
|
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||||
|
ExitCode: runRes.ExitCode,
|
||||||
|
Duration: runRes.Duration,
|
||||||
|
InvokedBinary: binary,
|
||||||
|
Format: format,
|
||||||
|
Title: req.Title,
|
||||||
|
}, fmt.Errorf("run seriatim render (binary=%q): %w", binary, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := validateNonEmptyTextFile(req.OutputRenderedPath); err != nil {
|
||||||
|
return RenderResult{
|
||||||
|
OutputRenderedPath: req.OutputRenderedPath,
|
||||||
|
StdoutLogPath: req.StdoutLogPath,
|
||||||
|
StderrLogPath: req.StderrLogPath,
|
||||||
|
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||||
|
ExitCode: runRes.ExitCode,
|
||||||
|
Duration: runRes.Duration,
|
||||||
|
InvokedBinary: binary,
|
||||||
|
Format: format,
|
||||||
|
Title: req.Title,
|
||||||
|
}, fmt.Errorf("validate seriatim rendered output %q: %w", req.OutputRenderedPath, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return RenderResult{
|
||||||
|
OutputRenderedPath: req.OutputRenderedPath,
|
||||||
|
StdoutLogPath: req.StdoutLogPath,
|
||||||
|
StderrLogPath: req.StderrLogPath,
|
||||||
|
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||||
|
ExitCode: runRes.ExitCode,
|
||||||
|
Duration: runRes.Duration,
|
||||||
|
InvokedBinary: binary,
|
||||||
|
Format: format,
|
||||||
|
Title: req.Title,
|
||||||
|
Metadata: map[string]any{
|
||||||
|
"adapter": "seriatim_subprocess",
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (r *SubprocessRunner) buildMergeArgs(req MergeRequest) []string {
|
func (r *SubprocessRunner) buildMergeArgs(req MergeRequest) []string {
|
||||||
args := []string{"merge"}
|
args := []string{"merge"}
|
||||||
|
|
||||||
@@ -480,6 +571,22 @@ func buildNormalizeArgs(req NormalizeRequest, outputSchema string) []string {
|
|||||||
return args
|
return args
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func buildRenderArgs(req RenderRequest, format string) []string {
|
||||||
|
args := []string{
|
||||||
|
"render",
|
||||||
|
"--input-file", req.InputTranscriptPath,
|
||||||
|
"--output-file", req.OutputRenderedPath,
|
||||||
|
"--format", format,
|
||||||
|
"--include-timestamps", strconv.FormatBool(req.IncludeTimestamps),
|
||||||
|
"--include-segment-ids", strconv.FormatBool(req.IncludeSegmentIDs),
|
||||||
|
"--include-metadata", strconv.FormatBool(req.IncludeMetadata),
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(req.Title) != "" {
|
||||||
|
args = append(args, "--title", req.Title)
|
||||||
|
}
|
||||||
|
return args
|
||||||
|
}
|
||||||
|
|
||||||
func writeTrimInvocationConfig(req TrimRequest, args []string, binary string, timeout time.Duration) error {
|
func writeTrimInvocationConfig(req TrimRequest, args []string, binary string, timeout time.Duration) error {
|
||||||
payload := map[string]any{
|
payload := map[string]any{
|
||||||
"schema": "seriatim.generated.v1",
|
"schema": "seriatim.generated.v1",
|
||||||
@@ -509,6 +616,24 @@ func writeNormalizeInvocationConfig(req NormalizeRequest, args []string, binary
|
|||||||
return subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644)
|
return subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writeRenderInvocationConfig(req RenderRequest, args []string, binary string, timeout time.Duration, format string) error {
|
||||||
|
payload := map[string]any{
|
||||||
|
"schema": "seriatim.generated.v1",
|
||||||
|
"command": "render",
|
||||||
|
"binary": binary,
|
||||||
|
"args": args,
|
||||||
|
"timeout": timeout.String(),
|
||||||
|
"input_path": req.InputTranscriptPath,
|
||||||
|
"output_path": req.OutputRenderedPath,
|
||||||
|
"format": format,
|
||||||
|
"title": req.Title,
|
||||||
|
"include_timestamps": req.IncludeTimestamps,
|
||||||
|
"include_segment_ids": req.IncludeSegmentIDs,
|
||||||
|
"include_metadata": req.IncludeMetadata,
|
||||||
|
}
|
||||||
|
return subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644)
|
||||||
|
}
|
||||||
|
|
||||||
func validateJSONFile(path string) error {
|
func validateJSONFile(path string) error {
|
||||||
data, err := os.ReadFile(path)
|
data, err := os.ReadFile(path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -541,3 +666,20 @@ func validateJSONFileWithSegments(path string) error {
|
|||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validateNonEmptyTextFile(path string) error {
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("read file: %w", err)
|
||||||
|
}
|
||||||
|
if len(data) == 0 {
|
||||||
|
return fmt.Errorf("file is empty")
|
||||||
|
}
|
||||||
|
if !utf8.Valid(data) {
|
||||||
|
return fmt.Errorf("file is not valid utf-8 text")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(string(data)) == "" {
|
||||||
|
return fmt.Errorf("file has no non-whitespace content")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -569,6 +569,156 @@ func TestSubprocessRunnerNormalizeInvalidReportJSONFails(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerRenderSuccessInvocationAndProvenance(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("helper wrapper script uses /bin/sh")
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("SERIATIM_HELPER_MODE", "render_success")
|
||||||
|
recordPath := filepath.Join(t.TempDir(), "record.json")
|
||||||
|
t.Setenv("SERIATIM_HELPER_RECORD_PATH", recordPath)
|
||||||
|
|
||||||
|
wrapper := writeHelperWrapper(t)
|
||||||
|
runner := mustRunner(t, wrapper, false)
|
||||||
|
req := renderReqForTest(t)
|
||||||
|
|
||||||
|
res, err := runner.Render(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Render() error = %v", err)
|
||||||
|
}
|
||||||
|
if res.OutputRenderedPath != req.OutputRenderedPath {
|
||||||
|
t.Fatalf("OutputRenderedPath = %q, want %q", res.OutputRenderedPath, req.OutputRenderedPath)
|
||||||
|
}
|
||||||
|
if res.Format != req.Format {
|
||||||
|
t.Fatalf("Format = %q, want %q", res.Format, req.Format)
|
||||||
|
}
|
||||||
|
if res.Title != req.Title {
|
||||||
|
t.Fatalf("Title = %q, want %q", res.Title, req.Title)
|
||||||
|
}
|
||||||
|
if res.InvokedBinary != wrapper {
|
||||||
|
t.Fatalf("InvokedBinary = %q, want %q", res.InvokedBinary, wrapper)
|
||||||
|
}
|
||||||
|
if res.ExitCode != 0 {
|
||||||
|
t.Fatalf("ExitCode = %d, want 0", res.ExitCode)
|
||||||
|
}
|
||||||
|
if res.Duration <= 0 {
|
||||||
|
t.Fatalf("Duration = %s, want >0", res.Duration)
|
||||||
|
}
|
||||||
|
if res.Metadata == nil || res.Metadata["adapter"] != "seriatim_subprocess" {
|
||||||
|
t.Fatalf("Metadata = %#v, want adapter marker", res.Metadata)
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := os.Stat(req.OutputRenderedPath); err != nil {
|
||||||
|
t.Fatalf("rendered output missing: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(req.StdoutLogPath); err != nil {
|
||||||
|
t.Fatalf("stdout log missing: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(req.StderrLogPath); err != nil {
|
||||||
|
t.Fatalf("stderr log missing: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(req.GeneratedConfigPath); err != nil {
|
||||||
|
t.Fatalf("generated config missing: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := readHelperRecord(t, recordPath)
|
||||||
|
wantArgs := []string{
|
||||||
|
"render",
|
||||||
|
"--input-file", req.InputTranscriptPath,
|
||||||
|
"--output-file", req.OutputRenderedPath,
|
||||||
|
"--format", req.Format,
|
||||||
|
"--include-timestamps", "true",
|
||||||
|
"--include-segment-ids", "false",
|
||||||
|
"--include-metadata", "true",
|
||||||
|
"--title", req.Title,
|
||||||
|
}
|
||||||
|
if strings.Join(rec.Args, "\n") != strings.Join(wantArgs, "\n") {
|
||||||
|
t.Fatalf("args = %#v, want %#v", rec.Args, wantArgs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerRenderWithoutTitleOmitsTitleArg(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("helper wrapper script uses /bin/sh")
|
||||||
|
}
|
||||||
|
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("SERIATIM_HELPER_MODE", "render_success")
|
||||||
|
recordPath := filepath.Join(t.TempDir(), "record.json")
|
||||||
|
t.Setenv("SERIATIM_HELPER_RECORD_PATH", recordPath)
|
||||||
|
|
||||||
|
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||||
|
req := renderReqForTest(t)
|
||||||
|
req.Title = ""
|
||||||
|
if _, err := runner.Render(context.Background(), req); err != nil {
|
||||||
|
t.Fatalf("Render() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := readHelperRecord(t, recordPath)
|
||||||
|
for i := 0; i < len(rec.Args); i++ {
|
||||||
|
if rec.Args[i] == "--title" {
|
||||||
|
t.Fatalf("args = %#v, did not expect --title", rec.Args)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerRenderSubprocessFailure(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("helper wrapper script uses /bin/sh")
|
||||||
|
}
|
||||||
|
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("SERIATIM_HELPER_MODE", "fail")
|
||||||
|
t.Setenv("SERIATIM_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||||
|
|
||||||
|
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||||
|
req := renderReqForTest(t)
|
||||||
|
_, err := runner.Render(context.Background(), req)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Render() error = nil, want non-nil")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "run seriatim render") {
|
||||||
|
t.Fatalf("error = %q, want subprocess context", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerRenderMissingOutputFails(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("helper wrapper script uses /bin/sh")
|
||||||
|
}
|
||||||
|
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("SERIATIM_HELPER_MODE", "missing_output")
|
||||||
|
t.Setenv("SERIATIM_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||||
|
|
||||||
|
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||||
|
req := renderReqForTest(t)
|
||||||
|
_, err := runner.Render(context.Background(), req)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Render() error = nil, want non-nil")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "validate seriatim rendered output") {
|
||||||
|
t.Fatalf("error = %q, want output validation context", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerRenderEmptyOutputFails(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("helper wrapper script uses /bin/sh")
|
||||||
|
}
|
||||||
|
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("SERIATIM_HELPER_MODE", "render_empty_output")
|
||||||
|
t.Setenv("SERIATIM_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||||
|
|
||||||
|
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||||
|
req := renderReqForTest(t)
|
||||||
|
_, err := runner.Render(context.Background(), req)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Render() error = nil, want non-nil")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "file is empty") {
|
||||||
|
t.Fatalf("error = %q, want empty-file validation", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestSubprocessRunnerConstructorValidation(t *testing.T) {
|
func TestSubprocessRunnerConstructorValidation(t *testing.T) {
|
||||||
_, err := NewSubprocessRunnerFromConfigValues("", "10m", "seriatim-intermediate", nil, true, EnvConfig{})
|
_, err := NewSubprocessRunnerFromConfigValues("", "10m", "seriatim-intermediate", nil, true, EnvConfig{})
|
||||||
if err == nil {
|
if err == nil {
|
||||||
@@ -702,6 +852,14 @@ func TestSeriatimSubprocessHelper(t *testing.T) {
|
|||||||
case "normalize_report_missing":
|
case "normalize_report_missing":
|
||||||
writeSeriatimHelperFile(outputPath, `{"schema":"seriatim.intermediate.v1","segments":[]}`)
|
writeSeriatimHelperFile(outputPath, `{"schema":"seriatim.intermediate.v1","segments":[]}`)
|
||||||
os.Exit(0)
|
os.Exit(0)
|
||||||
|
case "render_success":
|
||||||
|
writeSeriatimHelperFile(outputPath, "# Rendered transcript\n\nHello.\n")
|
||||||
|
_, _ = os.Stdout.WriteString("seriatim helper render stdout\n")
|
||||||
|
_, _ = os.Stderr.WriteString("seriatim helper render stderr\n")
|
||||||
|
os.Exit(0)
|
||||||
|
case "render_empty_output":
|
||||||
|
writeSeriatimHelperFile(outputPath, "")
|
||||||
|
os.Exit(0)
|
||||||
default:
|
default:
|
||||||
_, _ = os.Stderr.WriteString(fmt.Sprintf("unknown helper mode %q\n", mode))
|
_, _ = os.Stderr.WriteString(fmt.Sprintf("unknown helper mode %q\n", mode))
|
||||||
os.Exit(2)
|
os.Exit(2)
|
||||||
@@ -777,6 +935,25 @@ func normalizeReqForTest(t *testing.T, withReport bool) NormalizeRequest {
|
|||||||
return req
|
return req
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func renderReqForTest(t *testing.T) RenderRequest {
|
||||||
|
t.Helper()
|
||||||
|
dir := t.TempDir()
|
||||||
|
input := filepath.Join(dir, "final.trimmed.json")
|
||||||
|
writeSeriatimFile(t, input, `{"schema":"seriatim.intermediate.v1","segments":[]}`)
|
||||||
|
return RenderRequest{
|
||||||
|
InputTranscriptPath: input,
|
||||||
|
OutputRenderedPath: filepath.Join(dir, "final.trimmed.md"),
|
||||||
|
Format: "markdown",
|
||||||
|
Title: "Session 42",
|
||||||
|
IncludeTimestamps: true,
|
||||||
|
IncludeSegmentIDs: false,
|
||||||
|
IncludeMetadata: true,
|
||||||
|
GeneratedConfigPath: filepath.Join(dir, "seriatim.render.generated.yml"),
|
||||||
|
StdoutLogPath: filepath.Join(dir, "seriatim.render.stdout.log"),
|
||||||
|
StderrLogPath: filepath.Join(dir, "seriatim.render.stderr.log"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func mustRunner(t *testing.T, binary string, report bool) *SubprocessRunner {
|
func mustRunner(t *testing.T, binary string, report bool) *SubprocessRunner {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
coalesce := 3.0
|
coalesce := 3.0
|
||||||
|
|||||||
@@ -1,31 +0,0 @@
|
|||||||
// Package storage declares archive/storage backend adapter boundaries.
|
|
||||||
package storage
|
|
||||||
|
|
||||||
import "context"
|
|
||||||
|
|
||||||
// TODO: implement remote storage/archive backends (S3/SFTP/etc.).
|
|
||||||
|
|
||||||
// Backend is the adapter boundary for archive/storage operations.
|
|
||||||
type Backend interface {
|
|
||||||
Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ArchiveItem describes one item to archive.
|
|
||||||
type ArchiveItem struct {
|
|
||||||
Kind string
|
|
||||||
LocalPath string
|
|
||||||
RemoteKey string
|
|
||||||
}
|
|
||||||
|
|
||||||
// ArchiveRequest describes one archive operation.
|
|
||||||
type ArchiveRequest struct {
|
|
||||||
SessionID string
|
|
||||||
ManifestPath string
|
|
||||||
Items []ArchiveItem
|
|
||||||
}
|
|
||||||
|
|
||||||
// ArchiveResult describes archive operation output.
|
|
||||||
type ArchiveResult struct {
|
|
||||||
Archived []ArchiveItem
|
|
||||||
Metadata map[string]any
|
|
||||||
}
|
|
||||||
@@ -10,23 +10,8 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// NoopBackend is a deterministic no-op archive/storage adapter.
|
// FakeBackend provides a deterministic in-memory object store for tests.
|
||||||
type NoopBackend struct{}
|
|
||||||
|
|
||||||
// Archive returns the requested items as archived with placeholder metadata.
|
|
||||||
func (n *NoopBackend) Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error) {
|
|
||||||
if err := ctx.Err(); err != nil {
|
|
||||||
return ArchiveResult{}, err
|
|
||||||
}
|
|
||||||
return ArchiveResult{Archived: append([]ArchiveItem(nil), req.Items...), Metadata: map[string]any{"placeholder": true}}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// FakeBackend captures archive requests and returns deterministic responses.
|
|
||||||
type FakeBackend struct {
|
type FakeBackend struct {
|
||||||
Requests []ArchiveRequest
|
|
||||||
Err error
|
|
||||||
Result ArchiveResult
|
|
||||||
|
|
||||||
Objects map[string]FakeObject
|
Objects map[string]FakeObject
|
||||||
Uploads []FakeUploadCall
|
Uploads []FakeUploadCall
|
||||||
Downloads []FakeDownloadCall
|
Downloads []FakeDownloadCall
|
||||||
@@ -50,25 +35,6 @@ type FakeDownloadCall struct {
|
|||||||
LocalPath string
|
LocalPath string
|
||||||
}
|
}
|
||||||
|
|
||||||
// Archive records request and returns configured response.
|
|
||||||
func (f *FakeBackend) Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error) {
|
|
||||||
if err := ctx.Err(); err != nil {
|
|
||||||
return ArchiveResult{}, err
|
|
||||||
}
|
|
||||||
f.Requests = append(f.Requests, req)
|
|
||||||
if f.Err != nil {
|
|
||||||
return ArchiveResult{}, f.Err
|
|
||||||
}
|
|
||||||
res := f.Result
|
|
||||||
if res.Archived == nil {
|
|
||||||
res.Archived = append([]ArchiveItem(nil), req.Items...)
|
|
||||||
}
|
|
||||||
if res.Metadata == nil {
|
|
||||||
res.Metadata = map[string]any{"fake": true}
|
|
||||||
}
|
|
||||||
return res, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// FakeObject is a deterministic fake object-store record.
|
// FakeObject is a deterministic fake object-store record.
|
||||||
type FakeObject struct {
|
type FakeObject struct {
|
||||||
Key string
|
Key string
|
||||||
|
|||||||
@@ -9,30 +9,6 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestFakeBackendCapturesRequestAndReturnsItems(t *testing.T) {
|
|
||||||
fake := &FakeBackend{}
|
|
||||||
req := ArchiveRequest{SessionID: "s1", Items: []ArchiveItem{{Kind: "artifact", LocalPath: "artifacts/log.md"}}}
|
|
||||||
|
|
||||||
res, err := fake.Archive(context.Background(), req)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Archive() error = %v", err)
|
|
||||||
}
|
|
||||||
if len(fake.Requests) != 1 || fake.Requests[0].SessionID != "s1" {
|
|
||||||
t.Fatalf("requests = %#v, want captured request", fake.Requests)
|
|
||||||
}
|
|
||||||
if len(res.Archived) != 1 {
|
|
||||||
t.Fatalf("archived len = %d, want 1", len(res.Archived))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFakeBackendError(t *testing.T) {
|
|
||||||
fake := &FakeBackend{Err: errors.New("boom")}
|
|
||||||
_, err := fake.Archive(context.Background(), ArchiveRequest{})
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("expected error, got nil")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFakeBackendListPrefixFiltering(t *testing.T) {
|
func TestFakeBackendListPrefixFiltering(t *testing.T) {
|
||||||
fake := &FakeBackend{}
|
fake := &FakeBackend{}
|
||||||
fake.SeedObject(FakeObject{Key: "dnd/campaigns/forsaken/audio/a.flac", Data: []byte("a")})
|
fake.SeedObject(FakeObject{Key: "dnd/campaigns/forsaken/audio/a.flac", Data: []byte("a")})
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ObjectStore is a remote object storage boundary used by future prepare/archive work.
|
// ObjectStore is a remote object storage boundary used by prepare, restore, and publish work.
|
||||||
//
|
//
|
||||||
// Key invariant:
|
// Key invariant:
|
||||||
// callers pass full bucket-relative object keys. Backend implementations do not
|
// callers pass full bucket-relative object keys. Backend implementations do not
|
||||||
|
|||||||
36
internal/adapters/storage/temp_download.go
Normal file
36
internal/adapters/storage/temp_download.go
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
package storage
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// DownloadObjectToTemp downloads an object into a temporary file and returns
|
||||||
|
// the cleaned local path.
|
||||||
|
func DownloadObjectToTemp(ctx context.Context, store ObjectStore, key, pattern string) (string, error) {
|
||||||
|
if store == nil {
|
||||||
|
return "", fmt.Errorf("object store is required")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(pattern) == "" {
|
||||||
|
return "", fmt.Errorf("temp file pattern is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
tmp, err := os.CreateTemp("", pattern)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("create temp file: %w", err)
|
||||||
|
}
|
||||||
|
path := tmp.Name()
|
||||||
|
if err := tmp.Close(); err != nil {
|
||||||
|
_ = os.Remove(path)
|
||||||
|
return "", fmt.Errorf("close temp file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := store.Download(ctx, key, path); err != nil {
|
||||||
|
_ = os.Remove(path)
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return filepath.Clean(path), nil
|
||||||
|
}
|
||||||
69
internal/adapters/storage/temp_download_test.go
Normal file
69
internal/adapters/storage/temp_download_test.go
Normal file
@@ -0,0 +1,69 @@
|
|||||||
|
package storage
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestDownloadObjectToTempSuccess(t *testing.T) {
|
||||||
|
store := &FakeBackend{}
|
||||||
|
store.SeedObject(FakeObject{Key: "sessions/a/current/run_id.txt", Data: []byte("run-123\n")})
|
||||||
|
|
||||||
|
path, err := DownloadObjectToTemp(context.Background(), store, "sessions/a/current/run_id.txt", "narratio-test-*.txt")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DownloadObjectToTemp() error = %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { _ = os.Remove(path) })
|
||||||
|
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile() error = %v", err)
|
||||||
|
}
|
||||||
|
if string(data) != "run-123\n" {
|
||||||
|
t.Fatalf("downloaded data = %q, want %q", string(data), "run-123\n")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDownloadObjectToTempFailedDownloadRemovesTempFile(t *testing.T) {
|
||||||
|
sentinel := errors.New("download failed")
|
||||||
|
store := &FakeBackend{DownloadErr: sentinel}
|
||||||
|
pattern := "narratio-test-fail-*.txt"
|
||||||
|
before, err := filepath.Glob(filepath.Join(os.TempDir(), "narratio-test-fail-*.txt"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Glob(before) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
path, err := DownloadObjectToTemp(context.Background(), store, "sessions/a/current/run_id.txt", pattern)
|
||||||
|
if !errors.Is(err, sentinel) {
|
||||||
|
t.Fatalf("DownloadObjectToTemp() error = %v, want %v", err, sentinel)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(path) != "" {
|
||||||
|
t.Fatalf("DownloadObjectToTemp() path = %q, want empty on failure", path)
|
||||||
|
}
|
||||||
|
after, err := filepath.Glob(filepath.Join(os.TempDir(), "narratio-test-fail-*.txt"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Glob(after) error = %v", err)
|
||||||
|
}
|
||||||
|
if len(after) != len(before) {
|
||||||
|
t.Fatalf("temp file count changed after failed download: before=%d after=%d", len(before), len(after))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDownloadObjectToTempCallerContextWrappingPreservesCause(t *testing.T) {
|
||||||
|
sentinel := errors.New("object missing")
|
||||||
|
store := &FakeBackend{DownloadErr: sentinel}
|
||||||
|
|
||||||
|
_, err := DownloadObjectToTemp(context.Background(), store, "sessions/a/current/run_id.txt", "narratio-test-*.txt")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("DownloadObjectToTemp() error = nil, want error")
|
||||||
|
}
|
||||||
|
err = fmt.Errorf("download run pointer failed: %w", err)
|
||||||
|
if !errors.Is(err, sentinel) {
|
||||||
|
t.Fatalf("wrapped error does not preserve sentinel cause: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -145,7 +145,7 @@ func TestHTTPClientDoesNotRetryOnNonRetryableStatus(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestHTTPClientInvalidJSONFailsAndDoesNotPromote(t *testing.T) {
|
func TestHTTPClientInvalidJSONFailsAndDoesNotInstallOutput(t *testing.T) {
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
_, _ = w.Write([]byte(`not-json`))
|
_, _ = w.Write([]byte(`not-json`))
|
||||||
}))
|
}))
|
||||||
|
|||||||
@@ -28,12 +28,12 @@ func TestExecuteRunStageArtifactsUnsupportedStageFails(t *testing.T) {
|
|||||||
if code == 0 {
|
if code == 0 {
|
||||||
t.Fatal("exit code = 0, want non-zero")
|
t.Fatal("exit code = 0, want non-zero")
|
||||||
}
|
}
|
||||||
if !strings.Contains(stderr.String(), `run-stage: --artifacts is only supported for stages "analyze" and "archive"`) {
|
if !strings.Contains(stderr.String(), `run-stage: --artifacts is only supported for stages "analyze" and "publish"`) {
|
||||||
t.Fatalf("stderr = %q, want stage-gating error", stderr.String())
|
t.Fatalf("stderr = %q, want stage-gating error", stderr.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteRunStageArchivePropagatesSelectedArtifacts(t *testing.T) {
|
func TestExecuteRunStagePublishPropagatesSelectedArtifacts(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
|
|
||||||
@@ -48,14 +48,14 @@ func TestExecuteRunStageArchivePropagatesSelectedArtifacts(t *testing.T) {
|
|||||||
capturedStages = append(capturedStages, s.Name())
|
capturedStages = append(capturedStages, s.Name())
|
||||||
}
|
}
|
||||||
capturedArtifacts = append([]string(nil), opts.SelectedArtifacts...)
|
capturedArtifacts = append([]string(nil), opts.SelectedArtifacts...)
|
||||||
return &RunSummary{ManifestPath: filepath.Join(workspaceRoot, "manifest.json"), Executed: []string{"archive"}}, nil
|
return &RunSummary{ManifestPath: filepath.Join(workspaceRoot, "manifest.json"), Executed: []string{"publish"}}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
var stdout bytes.Buffer
|
||||||
var stderr bytes.Buffer
|
var stderr bytes.Buffer
|
||||||
code := Execute(
|
code := Execute(
|
||||||
[]string{
|
[]string{
|
||||||
"run-stage", "archive", "2026-05-03",
|
"run-stage", "publish", "2026-05-03",
|
||||||
"--config", pipelinePath,
|
"--config", pipelinePath,
|
||||||
"--campaign-file", campaignPath,
|
"--campaign-file", campaignPath,
|
||||||
"--session", sessionPath,
|
"--session", sessionPath,
|
||||||
@@ -67,8 +67,8 @@ func TestExecuteRunStageArchivePropagatesSelectedArtifacts(t *testing.T) {
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
if len(capturedStages) != 1 || capturedStages[0] != "archive" {
|
if len(capturedStages) != 1 || capturedStages[0] != "publish" {
|
||||||
t.Fatalf("captured stages = %#v, want [archive]", capturedStages)
|
t.Fatalf("captured stages = %#v, want [publish]", capturedStages)
|
||||||
}
|
}
|
||||||
if strings.Join(capturedArtifacts, ",") != "session_recap" {
|
if strings.Join(capturedArtifacts, ",") != "session_recap" {
|
||||||
t.Fatalf("captured artifacts = %#v, want [session_recap]", capturedArtifacts)
|
t.Fatalf("captured artifacts = %#v, want [session_recap]", capturedArtifacts)
|
||||||
@@ -120,14 +120,14 @@ func TestRunStageArtifactsDoesNotImplyForce(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResumeArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
|
func TestRunArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||||
|
|
||||||
store := &manifest.LocalStore{}
|
store := &manifest.LocalStore{}
|
||||||
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
|
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
|
||||||
seed.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
seed.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||||
}
|
}
|
||||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||||
@@ -135,16 +135,16 @@ func TestResumeArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
err := Resume(
|
err := Run(
|
||||||
context.Background(),
|
context.Background(),
|
||||||
[]string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
[]string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
||||||
&out,
|
&out,
|
||||||
)
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "has no remaining stages") {
|
if !strings.Contains(out.String(), "executed=0 skipped=10") {
|
||||||
t.Fatalf("output = %q, want no remaining stages", out.String())
|
t.Fatalf("output = %q, want all stages skipped", out.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -281,7 +281,7 @@ func TestExecuteAnalyzeMissingConfigUsesRunStageLoadingPath(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecutePublishForceRunsArchive(t *testing.T) {
|
func TestExecutePublishForceRunsPublish(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
|
|
||||||
@@ -300,7 +300,7 @@ func TestExecutePublishForceRunsArchive(t *testing.T) {
|
|||||||
capturedArtifacts = append([]string(nil), opts.SelectedArtifacts...)
|
capturedArtifacts = append([]string(nil), opts.SelectedArtifacts...)
|
||||||
return &RunSummary{
|
return &RunSummary{
|
||||||
ManifestPath: filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json"),
|
ManifestPath: filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json"),
|
||||||
Executed: []string{"archive"},
|
Executed: []string{"publish"},
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -314,8 +314,8 @@ func TestExecutePublishForceRunsArchive(t *testing.T) {
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
if len(capturedStages) != 1 || capturedStages[0] != "archive" {
|
if len(capturedStages) != 1 || capturedStages[0] != "publish" {
|
||||||
t.Fatalf("captured stages = %#v, want [archive]", capturedStages)
|
t.Fatalf("captured stages = %#v, want [publish]", capturedStages)
|
||||||
}
|
}
|
||||||
if !capturedForce {
|
if !capturedForce {
|
||||||
t.Fatal("captured force = false, want true")
|
t.Fatal("captured force = false, want true")
|
||||||
|
|||||||
@@ -16,7 +16,6 @@ import (
|
|||||||
// Clean removes local workspace/spool state while preserving durable cache
|
// Clean removes local workspace/spool state while preserving durable cache
|
||||||
// state unless cache cleanup is explicitly requested.
|
// state unless cache cleanup is explicitly requested.
|
||||||
func Clean(ctx context.Context, args []string, out io.Writer) error {
|
func Clean(ctx context.Context, args []string, out io.Writer) error {
|
||||||
positionalSessionID, args := pullLeadingSessionID(args)
|
|
||||||
fs := flag.NewFlagSet("clean", flag.ContinueOnError)
|
fs := flag.NewFlagSet("clean", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
var flags commonConfigFlags
|
var flags commonConfigFlags
|
||||||
@@ -27,20 +26,8 @@ func Clean(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
fs.BoolVar(&all, "all", false, "clean all local session work/spool state")
|
fs.BoolVar(&all, "all", false, "clean all local session work/spool state")
|
||||||
fs.BoolVar(&dryRun, "dry-run", false, "print cleanup targets without deleting")
|
fs.BoolVar(&dryRun, "dry-run", false, "print cleanup targets without deleting")
|
||||||
fs.BoolVar(&clearCache, "clear-cache", false, "also clear durable S3 audio cache entries")
|
fs.BoolVar(&clearCache, "clear-cache", false, "also clear durable S3 audio cache entries")
|
||||||
if err := fs.Parse(args); err != nil {
|
if err := parseSessionAwareFlags("clean", fs, args, &flags.sessionID); err != nil {
|
||||||
return fmt.Errorf("clean: invalid flags: %w", err)
|
return err
|
||||||
}
|
|
||||||
if positionalSessionID == "" {
|
|
||||||
if err := applyParsedSessionIDArg("clean", fs, &flags.sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("clean: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("clean", positionalSessionID, &flags.sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if all {
|
if all {
|
||||||
return cleanAllLocal(flags, dryRun, clearCache, out)
|
return cleanAllLocal(flags, dryRun, clearCache, out)
|
||||||
@@ -182,26 +169,12 @@ func reportCleanRootChildren(out io.Writer, root, policy string, dryRun bool) er
|
|||||||
}
|
}
|
||||||
|
|
||||||
func cleanableRootChildren(root, policy string) (string, []string, error) {
|
func cleanableRootChildren(root, policy string) (string, []string, error) {
|
||||||
cleanRoot := strings.TrimSpace(root)
|
rootAbs, exists, err := validateCleanRoot(root, policy)
|
||||||
if cleanRoot == "" {
|
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
|
||||||
}
|
|
||||||
rootAbs, err := filepath.Abs(cleanRoot)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
return "", nil, err
|
||||||
}
|
}
|
||||||
info, err := os.Lstat(rootAbs)
|
if !exists {
|
||||||
if err != nil {
|
return rootAbs, nil, nil
|
||||||
if os.IsNotExist(err) {
|
|
||||||
return rootAbs, nil, nil
|
|
||||||
}
|
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: stat root %q: %w", policy, rootAbs, err)
|
|
||||||
}
|
|
||||||
if info.Mode()&os.ModeSymlink != 0 {
|
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: refusing to clean symlink root %q", policy, rootAbs)
|
|
||||||
}
|
|
||||||
if !info.IsDir() {
|
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: root %q is not a directory", policy, rootAbs)
|
|
||||||
}
|
}
|
||||||
entries, err := os.ReadDir(rootAbs)
|
entries, err := os.ReadDir(rootAbs)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -300,46 +273,7 @@ func reportCleanScopedFile(out io.Writer, root, target, policy string, dryRun bo
|
|||||||
}
|
}
|
||||||
|
|
||||||
func validateScopedFile(root, target, policy string) (scopedDir, error) {
|
func validateScopedFile(root, target, policy string) (scopedDir, error) {
|
||||||
cleanRoot := strings.TrimSpace(root)
|
return validateScopedTarget(root, target, policy, false)
|
||||||
cleanTarget := strings.TrimSpace(target)
|
|
||||||
if cleanRoot == "" {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
|
||||||
}
|
|
||||||
if cleanTarget == "" {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target path is required", policy)
|
|
||||||
}
|
|
||||||
rootAbs, err := filepath.Abs(cleanRoot)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
|
||||||
}
|
|
||||||
targetAbs, err := filepath.Abs(cleanTarget)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve target %q: %w", policy, cleanTarget, err)
|
|
||||||
}
|
|
||||||
rel, err := filepath.Rel(rootAbs, targetAbs)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: relative path from %q to %q: %w", policy, rootAbs, targetAbs, err)
|
|
||||||
}
|
|
||||||
if rel == "." {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete root directory %q", policy, rootAbs)
|
|
||||||
}
|
|
||||||
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete path outside root: root=%q target=%q", policy, rootAbs, targetAbs)
|
|
||||||
}
|
|
||||||
info, err := os.Lstat(targetAbs)
|
|
||||||
if err != nil {
|
|
||||||
if os.IsNotExist(err) {
|
|
||||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: false}, nil
|
|
||||||
}
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: stat target %q: %w", policy, targetAbs, err)
|
|
||||||
}
|
|
||||||
if info.Mode()&os.ModeSymlink != 0 {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, targetAbs)
|
|
||||||
}
|
|
||||||
if info.IsDir() {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is a directory", policy, targetAbs)
|
|
||||||
}
|
|
||||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: true}, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func cleanIsFlac(path string) bool {
|
func cleanIsFlac(path string) bool {
|
||||||
|
|||||||
82
internal/app/cleanup_targets.go
Normal file
82
internal/app/cleanup_targets.go
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
func validateScopedTarget(root, target, policy string, requireDir bool) (scopedDir, error) {
|
||||||
|
cleanRoot := strings.TrimSpace(root)
|
||||||
|
cleanTarget := strings.TrimSpace(target)
|
||||||
|
if cleanRoot == "" {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
||||||
|
}
|
||||||
|
if cleanTarget == "" {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: target path is required", policy)
|
||||||
|
}
|
||||||
|
|
||||||
|
rootAbs, err := filepath.Abs(cleanRoot)
|
||||||
|
if err != nil {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
||||||
|
}
|
||||||
|
targetAbs, err := filepath.Abs(cleanTarget)
|
||||||
|
if err != nil {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve target %q: %w", policy, cleanTarget, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
rel, err := filepath.Rel(rootAbs, targetAbs)
|
||||||
|
if err != nil {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: relative path from %q to %q: %w", policy, rootAbs, targetAbs, err)
|
||||||
|
}
|
||||||
|
if rel == "." {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete root directory %q", policy, rootAbs)
|
||||||
|
}
|
||||||
|
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete path outside root: root=%q target=%q", policy, rootAbs, targetAbs)
|
||||||
|
}
|
||||||
|
|
||||||
|
info, err := os.Lstat(targetAbs)
|
||||||
|
if err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: false}, nil
|
||||||
|
}
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: stat target %q: %w", policy, targetAbs, err)
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, targetAbs)
|
||||||
|
}
|
||||||
|
if requireDir && !info.IsDir() {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is not a directory", policy, targetAbs)
|
||||||
|
}
|
||||||
|
if !requireDir && info.IsDir() {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is a directory", policy, targetAbs)
|
||||||
|
}
|
||||||
|
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: true}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateCleanRoot(root, policy string) (string, bool, error) {
|
||||||
|
cleanRoot := strings.TrimSpace(root)
|
||||||
|
if cleanRoot == "" {
|
||||||
|
return "", false, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
||||||
|
}
|
||||||
|
rootAbs, err := filepath.Abs(cleanRoot)
|
||||||
|
if err != nil {
|
||||||
|
return "", false, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
||||||
|
}
|
||||||
|
info, err := os.Lstat(rootAbs)
|
||||||
|
if err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return rootAbs, false, nil
|
||||||
|
}
|
||||||
|
return "", false, fmt.Errorf("cleanup policy %s: stat root %q: %w", policy, rootAbs, err)
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 {
|
||||||
|
return "", false, fmt.Errorf("cleanup policy %s: refusing to clean symlink root %q", policy, rootAbs)
|
||||||
|
}
|
||||||
|
if !info.IsDir() {
|
||||||
|
return "", false, fmt.Errorf("cleanup policy %s: root %q is not a directory", policy, rootAbs)
|
||||||
|
}
|
||||||
|
return rootAbs, true, nil
|
||||||
|
}
|
||||||
103
internal/app/cleanup_targets_test.go
Normal file
103
internal/app/cleanup_targets_test.go
Normal file
@@ -0,0 +1,103 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestCleanValidateScopedDirAndFile(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
dirTarget := filepath.Join(root, "runs", "run-1")
|
||||||
|
fileTarget := filepath.Join(root, "cache", "a.flac")
|
||||||
|
if err := os.MkdirAll(dirTarget, 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(dirTarget) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.MkdirAll(filepath.Dir(fileTarget), 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(file parent) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(fileTarget, []byte("audio"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(fileTarget) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := validateScopedDir(root, dirTarget, "test.dir"); err != nil {
|
||||||
|
t.Fatalf("validateScopedDir() error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := validateScopedFile(root, fileTarget, "test.file"); err != nil {
|
||||||
|
t.Fatalf("validateScopedFile() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCleanValidateScopedTargetSafetyRules(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
outside := t.TempDir()
|
||||||
|
target := filepath.Join(root, "runs", "run-1")
|
||||||
|
if err := os.MkdirAll(target, 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(target) error = %v", err)
|
||||||
|
}
|
||||||
|
fileTarget := filepath.Join(root, "cache", "a.flac")
|
||||||
|
if err := os.MkdirAll(filepath.Dir(fileTarget), 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(file parent) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(fileTarget, []byte("audio"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(fileTarget) error = %v", err)
|
||||||
|
}
|
||||||
|
symlinkTarget := filepath.Join(root, "symlink")
|
||||||
|
if err := os.Symlink(target, symlinkTarget); err != nil {
|
||||||
|
t.Fatalf("Symlink() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := validateScopedDir(root, root, "test.root"); err == nil || !strings.Contains(err.Error(), "refusing to delete root directory") {
|
||||||
|
t.Fatalf("validateScopedDir(root) error = %v, want root deletion rejection", err)
|
||||||
|
}
|
||||||
|
if _, err := validateScopedDir(root, filepath.Join(outside, "x"), "test.outside"); err == nil || !strings.Contains(err.Error(), "outside root") {
|
||||||
|
t.Fatalf("validateScopedDir(outside) error = %v, want outside-root rejection", err)
|
||||||
|
}
|
||||||
|
if _, err := validateScopedDir(root, fileTarget, "test.file-as-dir"); err == nil || !strings.Contains(err.Error(), "is not a directory") {
|
||||||
|
t.Fatalf("validateScopedDir(file) error = %v, want not-a-directory rejection", err)
|
||||||
|
}
|
||||||
|
if _, err := validateScopedFile(root, target, "test.dir-as-file"); err == nil || !strings.Contains(err.Error(), "is a directory") {
|
||||||
|
t.Fatalf("validateScopedFile(dir) error = %v, want is-a-directory rejection", err)
|
||||||
|
}
|
||||||
|
if _, err := validateScopedDir(root, symlinkTarget, "test.symlink"); err == nil || !strings.Contains(err.Error(), "refusing to delete symlink path") {
|
||||||
|
t.Fatalf("validateScopedDir(symlink) error = %v, want symlink rejection", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCleanableRootChildrenRejectsSymlinkChild(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
realChild := filepath.Join(root, "runs")
|
||||||
|
if err := os.MkdirAll(realChild, 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(realChild) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.Symlink(realChild, filepath.Join(root, "link")); err != nil {
|
||||||
|
t.Fatalf("Symlink() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, _, err := cleanableRootChildren(root, "test.root.children")
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "refusing to delete symlink path") {
|
||||||
|
t.Fatalf("cleanableRootChildren() error = %v, want symlink rejection", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCleanValidateScopedTargetMissing(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
missingDir := filepath.Join(root, "runs", "missing")
|
||||||
|
got, err := validateScopedDir(root, missingDir, "test.missing")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("validateScopedDir(missing) error = %v", err)
|
||||||
|
}
|
||||||
|
if got.Exists {
|
||||||
|
t.Fatalf("validateScopedDir(missing).Exists = true, want false")
|
||||||
|
}
|
||||||
|
|
||||||
|
missingFile := filepath.Join(root, "cache", "missing.flac")
|
||||||
|
got, err = validateScopedFile(root, missingFile, "test.missing.file")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("validateScopedFile(missing) error = %v", err)
|
||||||
|
}
|
||||||
|
if got.Exists {
|
||||||
|
t.Fatalf("validateScopedFile(missing).Exists = true, want false")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -7,7 +7,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
var supportedCommands = []string{"run", "run-stage", "resume", "analyze", "publish", "clean", "session"}
|
var supportedCommands = []string{"run", "run-stage", "analyze", "publish", "clean", "session"}
|
||||||
|
|
||||||
// Execute dispatches CLI commands and returns a process exit code.
|
// Execute dispatches CLI commands and returns a process exit code.
|
||||||
func Execute(args []string, stdout, stderr io.Writer) int {
|
func Execute(args []string, stdout, stderr io.Writer) int {
|
||||||
@@ -24,8 +24,6 @@ func Execute(args []string, stdout, stderr io.Writer) int {
|
|||||||
switch cmd {
|
switch cmd {
|
||||||
case "run":
|
case "run":
|
||||||
err = Run(ctx, cmdArgs, stdout)
|
err = Run(ctx, cmdArgs, stdout)
|
||||||
case "resume":
|
|
||||||
err = Resume(ctx, cmdArgs, stdout)
|
|
||||||
case "run-stage":
|
case "run-stage":
|
||||||
err = RunStage(ctx, cmdArgs, stdout)
|
err = RunStage(ctx, cmdArgs, stdout)
|
||||||
case "analyze":
|
case "analyze":
|
||||||
|
|||||||
@@ -31,10 +31,9 @@ func TestExecuteValidCommands(t *testing.T) {
|
|||||||
args []string
|
args []string
|
||||||
wantOut string
|
wantOut string
|
||||||
}{
|
}{
|
||||||
{name: "run", args: []string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run: session 2026-05-03; executed=9 skipped=0; manifest="},
|
{name: "run", args: []string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run: session 2026-05-03; executed=10 skipped=0; manifest="},
|
||||||
{name: "session plan", args: []string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "prepare: skip\ntranscribe: skip\nmerge: skip\npolish: skip\nnormalize: skip\ntrim: skip\nanalyze: skip\narchive: skip\nnotify: skip"},
|
{name: "session plan", args: []string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "prepare: skip\ntranscribe: skip\nmerge: skip\npolish: skip\nnormalize: skip\ntrim: skip\nrender: skip\nanalyze: skip\npublish: skip\nnotify: skip"},
|
||||||
{name: "session status", args: []string{"session", "status", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "Session: 2026-05-03"},
|
{name: "session status", args: []string{"session", "status", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "Session: 2026-05-03"},
|
||||||
{name: "resume", args: []string{"resume", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio resume: session 2026-05-03 has no remaining stages"},
|
|
||||||
{name: "run-stage", args: []string{"run-stage", "polish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run-stage: stage=polish executed=0 skipped=1 force=false; manifest="},
|
{name: "run-stage", args: []string{"run-stage", "polish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run-stage: stage=polish executed=0 skipped=1 force=false; manifest="},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -66,7 +65,7 @@ func TestExecuteMissingRequiredFlags(t *testing.T) {
|
|||||||
{name: "run missing session", args: []string{"run"}, want: "run: session_id is required"},
|
{name: "run missing session", args: []string{"run"}, want: "run: session_id is required"},
|
||||||
{name: "plan old top-level removed", args: []string{"plan"}, want: `unknown command: "plan"`},
|
{name: "plan old top-level removed", args: []string{"plan"}, want: `unknown command: "plan"`},
|
||||||
{name: "status old top-level removed", args: []string{"status"}, want: `unknown command: "status"`},
|
{name: "status old top-level removed", args: []string{"status"}, want: `unknown command: "status"`},
|
||||||
{name: "resume missing session", args: []string{"resume"}, want: "resume: session_id is required"},
|
{name: "resume removed", args: []string{"resume"}, want: `unknown command: "resume"`},
|
||||||
{name: "run-stage missing name", args: []string{"run-stage", "--config", "a", "--session", "b"}, want: "run-stage: expected stage name and session_id"},
|
{name: "run-stage missing name", args: []string{"run-stage", "--config", "a", "--session", "b"}, want: "run-stage: expected stage name and session_id"},
|
||||||
{name: "run-stage missing session", args: []string{"run-stage", "polish"}, want: "run-stage: expected stage name and session_id"},
|
{name: "run-stage missing session", args: []string{"run-stage", "polish"}, want: "run-stage: expected stage name and session_id"},
|
||||||
{name: "run missing config uses defaults", args: []string{"run", "2026-05-03", "--session", "session.yml"}, want: "run: no pipeline config path provided and no default pipeline config found; searched:"},
|
{name: "run missing config uses defaults", args: []string{"run", "2026-05-03", "--session", "session.yml"}, want: "run: no pipeline config path provided and no default pipeline config found; searched:"},
|
||||||
@@ -107,6 +106,22 @@ func TestExecuteRunStageUnknownFails(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteRunStageArchiveAliasFails(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
|
||||||
|
code := Execute([]string{"run-stage", "archive", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||||
|
if code == 0 {
|
||||||
|
t.Fatal("exit code = 0, want non-zero")
|
||||||
|
}
|
||||||
|
if !strings.Contains(stderr.String(), `unknown stage "archive"`) {
|
||||||
|
t.Fatalf("stderr = %q, want unknown stage alias error", stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteRunStageNormalizeIsAccepted(t *testing.T) {
|
func TestExecuteRunStageNormalizeIsAccepted(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
|
||||||
@@ -316,7 +331,7 @@ func TestExecuteUsesDefaultPipelineConfigPathWhenConfigFlagOmitted(t *testing.T)
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "narratio run: session 2026-05-03; executed=9 skipped=0; manifest=") {
|
if !strings.Contains(stdout.String(), "narratio run: session 2026-05-03; executed=10 skipped=0; manifest=") {
|
||||||
t.Fatalf("stdout = %q, want successful run output", stdout.String())
|
t.Fatalf("stdout = %q, want successful run output", stdout.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -469,7 +484,7 @@ storage:
|
|||||||
backend: s3
|
backend: s3
|
||||||
s3:
|
s3:
|
||||||
bucket: test-bucket
|
bucket: test-bucket
|
||||||
archive:
|
publish:
|
||||||
enabled: true
|
enabled: true
|
||||||
upload_run: false
|
upload_run: false
|
||||||
whisperx:
|
whisperx:
|
||||||
|
|||||||
@@ -4,7 +4,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
@@ -59,7 +58,7 @@ func loadCommandConfig(ctx context.Context, pipelineFlag, campaignFlag, campaign
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, missingSessionConfigError(discoveredSession.Searched, err.Error())
|
return nil, missingSessionConfigError(discoveredSession.Searched, err.Error())
|
||||||
}
|
}
|
||||||
sessionTempPath, err := downloadRemoteSessionConfig(ctx, store, remoteKey)
|
sessionTempPath, err := storage.DownloadObjectToTemp(ctx, store, remoteKey, "narratio-session-*.yml")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, missingSessionConfigError(discoveredSession.Searched, fmt.Sprintf("remote session %q download failed: %v", remoteKey, err))
|
return nil, missingSessionConfigError(discoveredSession.Searched, fmt.Sprintf("remote session %q download failed: %v", remoteKey, err))
|
||||||
}
|
}
|
||||||
@@ -134,21 +133,6 @@ func findRemoteSessionConfig(ctx context.Context, store storage.ObjectStore, ses
|
|||||||
return storage.ObjectInfo{}, fmt.Errorf("remote session %q not found", remoteKey)
|
return storage.ObjectInfo{}, fmt.Errorf("remote session %q not found", remoteKey)
|
||||||
}
|
}
|
||||||
|
|
||||||
func downloadRemoteSessionConfig(ctx context.Context, store storage.ObjectStore, remoteKey string) (string, error) {
|
|
||||||
f, err := os.CreateTemp("", "narratio-session-*.yml")
|
|
||||||
if err != nil {
|
|
||||||
return "", fmt.Errorf("create temp file: %w", err)
|
|
||||||
}
|
|
||||||
path := f.Name()
|
|
||||||
if err := f.Close(); err != nil {
|
|
||||||
return "", fmt.Errorf("close temp file %q: %w", path, err)
|
|
||||||
}
|
|
||||||
if err := store.Download(ctx, remoteKey, path); err != nil {
|
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
return filepath.Clean(path), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func s3BucketName(cfg *config.PipelineConfig) string {
|
func s3BucketName(cfg *config.PipelineConfig) string {
|
||||||
if cfg == nil || cfg.Storage.S3 == nil {
|
if cfg == nil || cfg.Storage.S3 == nil {
|
||||||
return ""
|
return ""
|
||||||
|
|||||||
131
internal/app/operator_artifact_rendering.go
Normal file
131
internal/app/operator_artifact_rendering.go
Normal file
@@ -0,0 +1,131 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
func buildHelperArtifactCatalog(cfg *config.Config) (*artifacts.ArtifactCatalog, error) {
|
||||||
|
catalog := artifacts.NewArtifactCatalog()
|
||||||
|
if err := catalog.RegisterBuiltIns(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
configured := map[string]artifacts.ConfiguredArtifactDefinition{}
|
||||||
|
if cfg.Pipeline.Scriptorium != nil {
|
||||||
|
for key, item := range cfg.Pipeline.Scriptorium.Artifacts {
|
||||||
|
configured[key] = artifacts.ConfiguredArtifactDefinition{Enabled: item.Enabled, OutputPath: item.OutputPath}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := catalog.RegisterConfiguredArtifacts(configured, nil); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return catalog, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeArtifactList(out io.Writer, cfg *config.Config, catalog *artifacts.ArtifactCatalog, locks *effectiveLocks, publishedRemoteState map[string]string) {
|
||||||
|
lockSet := lockSourceSet(locks.All)
|
||||||
|
fmt.Fprintln(out, "Built-in:")
|
||||||
|
for _, transcript := range artifacts.RuntimeTranscriptArtifacts() {
|
||||||
|
writeArtifactLine(out, transcript.SourceID, lockSet)
|
||||||
|
}
|
||||||
|
writeArtifactLine(out, artifacts.ArtifactBoundsSession, lockSet)
|
||||||
|
fmt.Fprintln(out, "Configured:")
|
||||||
|
for _, entry := range catalog.ListConfigured() {
|
||||||
|
writeArtifactLine(out, entry.SourceID, lockSet)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Previous-session:")
|
||||||
|
for _, req := range artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg)) {
|
||||||
|
fmt.Fprintf(out, "- %s required=%t\n", artifactpolicy.PreviousSessionSourceID(req.Name), req.Required)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Published:")
|
||||||
|
for _, rule := range cfg.Pipeline.Publish.Outputs {
|
||||||
|
writePublishedOutputLine(out, rule, catalog, lockSet, publishedRemoteState)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeArtifactLine(out io.Writer, source string, lockSet map[string]config.PublishLockRule) {
|
||||||
|
parts := []string{source}
|
||||||
|
if _, ok := lockSet[source]; ok {
|
||||||
|
parts = append(parts, "locked")
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||||
|
}
|
||||||
|
|
||||||
|
func writePublishedOutputLine(out io.Writer, rule config.PublishOutputRule, catalog *artifacts.ArtifactCatalog, lockSet map[string]config.PublishLockRule, remoteState map[string]string) {
|
||||||
|
source := strings.TrimSpace(rule.Source)
|
||||||
|
parts := []string{source}
|
||||||
|
if _, ok := lockSet[source]; ok {
|
||||||
|
parts = append(parts, "locked")
|
||||||
|
}
|
||||||
|
dest, showDest, err := helperPublishedOutputDest(rule, catalog)
|
||||||
|
if err != nil {
|
||||||
|
parts = append(parts, "remote=error")
|
||||||
|
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if showDest {
|
||||||
|
parts = append(parts, "dest="+dest)
|
||||||
|
}
|
||||||
|
if state := remoteState[publishedOutputRemoteStateKey(source, dest)]; state != "" {
|
||||||
|
parts = append(parts, state)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||||
|
}
|
||||||
|
|
||||||
|
func remotePublishedOutputAvailability(ctx context.Context, cfg *config.Config, store storage.ObjectStore, catalog *artifacts.ArtifactCatalog) map[string]string {
|
||||||
|
out := map[string]string{}
|
||||||
|
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
for _, rule := range cfg.Pipeline.Publish.Outputs {
|
||||||
|
source := strings.TrimSpace(rule.Source)
|
||||||
|
dest, _, err := helperPublishedOutputDest(rule, catalog)
|
||||||
|
if err != nil {
|
||||||
|
out[publishedOutputRemoteStateKey(source, "")] = "remote=error"
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
key := artifacts.S3PublishedOutputKey(sessionPrefix, dest)
|
||||||
|
if exists, err := store.Exists(ctx, key); err == nil && exists {
|
||||||
|
out[publishedOutputRemoteStateKey(source, dest)] = "remote=published"
|
||||||
|
} else if err != nil {
|
||||||
|
out[publishedOutputRemoteStateKey(source, dest)] = "remote=error"
|
||||||
|
} else {
|
||||||
|
out[publishedOutputRemoteStateKey(source, dest)] = "remote=missing"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func helperPublishedOutputDest(rule config.PublishOutputRule, catalog *artifacts.ArtifactCatalog) (string, bool, error) {
|
||||||
|
source := strings.TrimSpace(rule.Source)
|
||||||
|
normalized, err := artifactpolicy.ResolvePublishedDestination(source, rule.Dest, helperConfiguredOutputPathMap(catalog))
|
||||||
|
if err != nil {
|
||||||
|
return "", false, err
|
||||||
|
}
|
||||||
|
entry, ok := catalog.Lookup(source)
|
||||||
|
showDest := !ok || strings.TrimSpace(entry.CanonicalRelPath) != normalized
|
||||||
|
return normalized, showDest, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func helperConfiguredOutputPathMap(catalog *artifacts.ArtifactCatalog) map[string]string {
|
||||||
|
out := map[string]string{}
|
||||||
|
if catalog == nil {
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
for _, entry := range catalog.ListConfigured() {
|
||||||
|
if strings.TrimSpace(entry.ConfiguredKey) == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out[entry.ConfiguredKey] = strings.TrimSpace(entry.CanonicalRelPath)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func publishedOutputRemoteStateKey(source, dest string) string {
|
||||||
|
return strings.TrimSpace(source) + "\x00" + strings.TrimSpace(dest)
|
||||||
|
}
|
||||||
39
internal/app/operator_artifacts_list.go
Normal file
39
internal/app/operator_artifacts_list.go
Normal file
@@ -0,0 +1,39 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ArtifactsList lists effective artifact sources.
|
||||||
|
func ArtifactsList(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("artifacts list", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
var remote bool
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
fs.BoolVar(&remote, "remote", false, "inspect remote publish availability")
|
||||||
|
if err := parseSessionAwareFlags("artifacts list", fs, args, &flags.sessionID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("artifacts list: session_id is required")
|
||||||
|
}
|
||||||
|
cfg, store, locks, _, err := loadHelperContext(ctx, flags, remote)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("artifacts list: %w", err)
|
||||||
|
}
|
||||||
|
catalog, err := buildHelperArtifactCatalog(cfg)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("artifacts list: %w", err)
|
||||||
|
}
|
||||||
|
publishedRemoteState := map[string]string{}
|
||||||
|
if remote && store != nil {
|
||||||
|
publishedRemoteState = remotePublishedOutputAvailability(ctx, cfg, store, catalog)
|
||||||
|
}
|
||||||
|
writeArtifactList(out, cfg, catalog, locks, publishedRemoteState)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
140
internal/app/operator_findings.go
Normal file
140
internal/app/operator_findings.go
Normal file
@@ -0,0 +1,140 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
|
)
|
||||||
|
|
||||||
|
type finding struct {
|
||||||
|
Severity string
|
||||||
|
Category string
|
||||||
|
Message string
|
||||||
|
}
|
||||||
|
|
||||||
|
type findingError struct {
|
||||||
|
count int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e findingError) Error() string {
|
||||||
|
return fmt.Sprintf("%d validation error(s)", e.count)
|
||||||
|
}
|
||||||
|
|
||||||
|
func renderFindings(out io.Writer, campaign, sessionID string, findings []finding) error {
|
||||||
|
if campaign != "" || sessionID != "" {
|
||||||
|
fmt.Fprintf(out, "Campaign: %s\n", campaign)
|
||||||
|
fmt.Fprintf(out, "Session: %s\n\n", sessionID)
|
||||||
|
}
|
||||||
|
errorsCount := 0
|
||||||
|
for _, f := range findings {
|
||||||
|
if f.Severity == "ERROR" {
|
||||||
|
errorsCount++
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "%-5s %-10s %s\n", f.Severity, f.Category, f.Message)
|
||||||
|
}
|
||||||
|
if errorsCount > 0 {
|
||||||
|
return findingError{count: errorsCount}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func okFinding(category, msg string) finding { return finding{"OK", category, msg} }
|
||||||
|
func infoFinding(category, msg string) finding { return finding{"INFO", category, msg} }
|
||||||
|
func warnFinding(category, msg string) finding { return finding{"WARN", category, msg} }
|
||||||
|
func errorFinding(category, msg string) finding { return finding{"ERROR", category, msg} }
|
||||||
|
|
||||||
|
func sessionSourceSummary(cfg *config.Config) string {
|
||||||
|
source := cfg.SessionSource.Source
|
||||||
|
if source == "" {
|
||||||
|
source = "session_config"
|
||||||
|
}
|
||||||
|
if cfg.SessionSource.S3Key != "" {
|
||||||
|
return source + " " + cfg.SessionSource.S3Key
|
||||||
|
}
|
||||||
|
return source + " " + cfg.SessionPath
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateStableInputFindings(cfg *config.Config) []finding {
|
||||||
|
checks := inspectStableInputs(cfg)
|
||||||
|
out := make([]finding, 0, len(checks))
|
||||||
|
for _, check := range checks {
|
||||||
|
if check.Err != nil {
|
||||||
|
msg := check.Name + ": " + check.Err.Error()
|
||||||
|
if strings.TrimSpace(check.Path) != "" {
|
||||||
|
msg = fmt.Sprintf("%s missing: %v", check.Name, check.Err)
|
||||||
|
}
|
||||||
|
out = append(out, errorFinding("inputs", msg))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, okFinding("inputs", check.Name+": "+check.Path))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveHelperConfigRelativePath(input config.ResolvedInputFile) (string, error) {
|
||||||
|
if strings.TrimSpace(input.ConfigPath) == "" {
|
||||||
|
return "", fmt.Errorf("source config path is required")
|
||||||
|
}
|
||||||
|
path := strings.TrimSpace(input.Path)
|
||||||
|
if path == "" {
|
||||||
|
return "", fmt.Errorf("path is required")
|
||||||
|
}
|
||||||
|
if filepath.IsAbs(path) {
|
||||||
|
return filepath.Clean(path), nil
|
||||||
|
}
|
||||||
|
return filepath.Clean(filepath.Join(filepath.Dir(input.ConfigPath), path)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateLocalAudioFindings(cfg *config.Config) []finding {
|
||||||
|
check := inspectLocalAudioPresence(cfg)
|
||||||
|
if !check.Checked {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if check.Err != nil {
|
||||||
|
return []finding{errorFinding("audio", check.Err.Error())}
|
||||||
|
}
|
||||||
|
return []finding{okFinding("audio", fmt.Sprintf("%d local audio file(s)", len(check.Paths)))}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateRemoteAudioFinding(ctx context.Context, cfg *config.Config, store storage.ObjectStore) finding {
|
||||||
|
check := inspectRemoteAudioPresence(ctx, cfg, store)
|
||||||
|
if check.Err != nil {
|
||||||
|
return errorFinding("audio", check.Err.Error())
|
||||||
|
}
|
||||||
|
return okFinding("audio", fmt.Sprintf("%d remote .flac object(s)", len(check.Keys)))
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadLocalManifest(ctx context.Context, path string) (*manifest.Manifest, error) {
|
||||||
|
if _, err := os.Stat(path); err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
store := &manifest.LocalStore{}
|
||||||
|
return store.Load(ctx, path)
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStageStatuses(out io.Writer, m *manifest.Manifest) {
|
||||||
|
if m == nil || len(m.Stages) == 0 {
|
||||||
|
fmt.Fprintln(out, "stages: no stages recorded")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "stages:")
|
||||||
|
names := make([]string, 0, len(m.Stages))
|
||||||
|
for name := range m.Stages {
|
||||||
|
names = append(names, name)
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
for _, name := range names {
|
||||||
|
fmt.Fprintf(out, "- %s: %s\n", name, m.Stages[name].Status)
|
||||||
|
}
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -423,7 +423,7 @@ inputs:
|
|||||||
}
|
}
|
||||||
|
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
audioKey := artifacts.S3PromotedArtifactKey(artifacts.S3AudioPrefix(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"), "audio/"), "alice.flac")
|
audioKey := artifacts.S3PublishedOutputKey(artifacts.S3AudioPrefix(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"), "audio/"), "alice.flac")
|
||||||
fake.SeedObject(storage.FakeObject{Key: audioKey, Data: []byte("audio")})
|
fake.SeedObject(storage.FakeObject{Key: audioKey, Data: []byte("audio")})
|
||||||
origStoreFn := newObjectStoreFromConfigFn
|
origStoreFn := newObjectStoreFromConfigFn
|
||||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||||
@@ -501,9 +501,9 @@ func TestExecuteLocksAddListAndRemoveUseRemoteLockStore(t *testing.T) {
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("locks remove exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("locks remove exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
store, err := config.LoadArchiveLockStoreBytes("locks.yml", fake.Objects[key].Data, nil)
|
store, err := config.LoadPublishLockStoreBytes("locks.yml", fake.Objects[key].Data, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("LoadArchiveLockStoreBytes() error = %v", err)
|
t.Fatalf("LoadPublishLockStoreBytes() error = %v", err)
|
||||||
}
|
}
|
||||||
if len(store.Locks) != 0 {
|
if len(store.Locks) != 0 {
|
||||||
t.Fatalf("locks after remove = %#v, want empty", store.Locks)
|
t.Fatalf("locks after remove = %#v, want empty", store.Locks)
|
||||||
@@ -593,10 +593,57 @@ func TestExecuteLocksRequireSessionID(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteLocksMutationRejectsSessionIDMismatch(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
args []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "add mismatch",
|
||||||
|
args: []string{
|
||||||
|
"session", "locks", "add", "2026-05-03", "narratio.transcript.final_trimmed",
|
||||||
|
"--session-id", "2026-05-04",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "remove mismatch",
|
||||||
|
args: []string{
|
||||||
|
"session", "locks", "remove", "2026-05-03", "narratio.transcript.final_trimmed",
|
||||||
|
"--session-id", "2026-05-04",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute(tt.args, &stdout, &stderr)
|
||||||
|
if code == 0 {
|
||||||
|
t.Fatal("exit code = 0, want non-zero")
|
||||||
|
}
|
||||||
|
if !strings.Contains(stderr.String(), "does not match expected session id") {
|
||||||
|
t.Fatalf("stderr = %q, want session-id mismatch guidance", stderr.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteLocksCannotModifyStaticLocks(t *testing.T) {
|
func TestExecuteLocksCannotModifyStaticLocks(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
addStaticArchiveLockToPipelineConfig(t, pipelinePath, "narratio.transcript.final_trimmed")
|
addStaticPublishLockToPipelineConfig(t, pipelinePath, "narratio.transcript.final_trimmed")
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
var storeInitCalls int
|
var storeInitCalls int
|
||||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
@@ -683,17 +730,17 @@ func addSessionTemplateToCampaign(t *testing.T, campaignPath, templateFile strin
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteArtifactsListRemoteReportsPromotedAvailability(t *testing.T) {
|
func TestExecuteArtifactsListRemoteReportsPublishedAvailability(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
addPublishOutputsToPipeline(t, pipelinePath, `
|
||||||
promote_artifacts:
|
outputs:
|
||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
required: true
|
required: true
|
||||||
`)
|
`)
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
trimmedKey := artifacts.S3PromotedArtifactKey(
|
trimmedKey := artifacts.S3PublishedOutputKey(
|
||||||
artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"),
|
artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"),
|
||||||
"transcripts/final.trimmed.json",
|
"transcripts/final.trimmed.json",
|
||||||
)
|
)
|
||||||
@@ -713,16 +760,16 @@ func TestExecuteArtifactsListRemoteReportsPromotedAvailability(t *testing.T) {
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "narratio.transcript.final_trimmed remote=promoted") {
|
if !strings.Contains(stdout.String(), "narratio.transcript.final_trimmed remote=published") {
|
||||||
t.Fatalf("stdout = %q, want promoted remote availability", stdout.String())
|
t.Fatalf("stdout = %q, want published remote availability", stdout.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteArtifactsListRemoteUsesPromotionDestinations(t *testing.T) {
|
func TestExecuteArtifactsListRemoteUsesPublishOutputDestinations(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
addPublishOutputsToPipeline(t, pipelinePath, `
|
||||||
promote_artifacts:
|
outputs:
|
||||||
- source: narratio.transcript.final
|
- source: narratio.transcript.final
|
||||||
dest: transcripts/full.json
|
dest: transcripts/full.json
|
||||||
required: true
|
required: true
|
||||||
@@ -732,8 +779,8 @@ func TestExecuteArtifactsListRemoteUsesPromotionDestinations(t *testing.T) {
|
|||||||
`)
|
`)
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03")
|
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03")
|
||||||
fake.SeedObject(storage.FakeObject{Key: artifacts.S3PromotedArtifactKey(sessionPrefix, "transcripts/full.json"), Data: []byte(`{"segments":[]}`)})
|
fake.SeedObject(storage.FakeObject{Key: artifacts.S3PublishedOutputKey(sessionPrefix, "transcripts/full.json"), Data: []byte(`{"segments":[]}`)})
|
||||||
fake.SeedObject(storage.FakeObject{Key: artifacts.S3PromotedArtifactKey(sessionPrefix, "transcripts/bounds.json"), Data: []byte(`{}`)})
|
fake.SeedObject(storage.FakeObject{Key: artifacts.S3PublishedOutputKey(sessionPrefix, "transcripts/bounds.json"), Data: []byte(`{}`)})
|
||||||
var storeInitCalls int
|
var storeInitCalls int
|
||||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
@@ -759,8 +806,8 @@ func TestExecuteArtifactsListRemoteUsesPromotionDestinations(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
for _, want := range []string{
|
for _, want := range []string{
|
||||||
"narratio.transcript.final dest=transcripts/full.json remote=promoted",
|
"narratio.transcript.final dest=transcripts/full.json remote=published",
|
||||||
"narratio.bounds.session dest=transcripts/bounds.json remote=promoted",
|
"narratio.bounds.session dest=transcripts/bounds.json remote=published",
|
||||||
} {
|
} {
|
||||||
if !strings.Contains(out, want) {
|
if !strings.Contains(out, want) {
|
||||||
t.Fatalf("stdout = %q, want %q", out, want)
|
t.Fatalf("stdout = %q, want %q", out, want)
|
||||||
@@ -771,8 +818,8 @@ func TestExecuteArtifactsListRemoteUsesPromotionDestinations(t *testing.T) {
|
|||||||
func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
addPublishOutputsToPipeline(t, pipelinePath, `
|
||||||
promote_artifacts:
|
outputs:
|
||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
required: true
|
required: true
|
||||||
@@ -782,9 +829,9 @@ func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
|||||||
`)
|
`)
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03")
|
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03")
|
||||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
trimmedKey := artifacts.S3PromotedArtifactKey(sessionPrefix, "transcripts/final.trimmed.json")
|
trimmedKey := artifacts.S3PublishedOutputKey(sessionPrefix, "transcripts/final.trimmed.json")
|
||||||
fullKey := artifacts.S3PromotedArtifactKey(sessionPrefix, "transcripts/full.json")
|
fullKey := artifacts.S3PublishedOutputKey(sessionPrefix, "transcripts/full.json")
|
||||||
lockKey := artifacts.S3SessionLocksKey(sessionPrefix)
|
lockKey := artifacts.S3SessionLocksKey(sessionPrefix)
|
||||||
fake.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
fake.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||||
fake.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "2026-05-03", "sample-campaign")})
|
fake.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "2026-05-03", "sample-campaign")})
|
||||||
@@ -811,10 +858,10 @@ func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
|||||||
"Built-in:",
|
"Built-in:",
|
||||||
"Configured:",
|
"Configured:",
|
||||||
"Previous-session:",
|
"Previous-session:",
|
||||||
"Promoted:",
|
"Published:",
|
||||||
"narratio.transcript.final_trimmed locked",
|
"narratio.transcript.final_trimmed locked",
|
||||||
"narratio.transcript.final_trimmed locked remote=promoted",
|
"narratio.transcript.final_trimmed locked remote=published",
|
||||||
"narratio.transcript.final dest=transcripts/full.json remote=promoted",
|
"narratio.transcript.final dest=transcripts/full.json remote=published",
|
||||||
} {
|
} {
|
||||||
if !strings.Contains(out, want) {
|
if !strings.Contains(out, want) {
|
||||||
t.Fatalf("stdout = %q, want %q", out, want)
|
t.Fatalf("stdout = %q, want %q", out, want)
|
||||||
@@ -828,8 +875,8 @@ func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
|||||||
func TestExecuteStatusReportsRemoteArtifactCatalogErrorsWithoutFailing(t *testing.T) {
|
func TestExecuteStatusReportsRemoteArtifactCatalogErrorsWithoutFailing(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
addPublishOutputsToPipeline(t, pipelinePath, `
|
||||||
promote_artifacts:
|
outputs:
|
||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
required: true
|
required: true
|
||||||
@@ -850,20 +897,99 @@ func TestExecuteStatusReportsRemoteArtifactCatalogErrorsWithoutFailing(t *testin
|
|||||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
out := stdout.String()
|
out := stdout.String()
|
||||||
if !strings.Contains(out, "Remote archive: missing or unavailable:") {
|
if !strings.Contains(out, "Remote publish: missing or unavailable:") {
|
||||||
t.Fatalf("stdout = %q, want remote archive unavailable state", out)
|
t.Fatalf("stdout = %q, want remote publish unavailable state", out)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out, "Remote outputs:") || !strings.Contains(out, "narratio.transcript.final_trimmed remote=error") {
|
if !strings.Contains(out, "Remote outputs:") || !strings.Contains(out, "narratio.transcript.final_trimmed remote=error") {
|
||||||
t.Fatalf("stdout = %q, want remote output error state", out)
|
t.Fatalf("stdout = %q, want remote output error state", out)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out, "Archive locks: error:") {
|
if !strings.Contains(out, "Publish locks: error:") {
|
||||||
t.Fatalf("stdout = %q, want archive locks error", out)
|
t.Fatalf("stdout = %q, want publish locks error", out)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteArchiveLoadsRemoteLocks(t *testing.T) {
|
func TestExecuteStatusReportsMissingRemoteCurrentStateWithoutFailing(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidArchiveConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{
|
||||||
|
"session", "status", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "Remote publish: missing or unavailable: remote current run pointer missing") {
|
||||||
|
t.Fatalf("stdout = %q, want missing remote current-state line", stdout.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteStatusReportsPreviousStateReadinessWithoutFailing(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
|
replaceInFileOrFatal(t, pipelinePath, "source: narratio.artifact.session_recap", "source: narratio.previous_session.artifact.session_recap")
|
||||||
|
replaceInFileOrFatal(t, sessionPath, "session_id: 2026-05-03\n", "session_id: 2026-05-03\nprevious_session_id: 2026-04-26\n")
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{
|
||||||
|
"session", "status", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "Previous-session artifacts: unavailable: remote current run pointer missing") {
|
||||||
|
t.Fatalf("stdout = %q, want previous readiness unavailable line", stdout.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteSessionValidateReportsPreviousStateFindingAndReturnsFindingError(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
|
replaceInFileOrFatal(t, pipelinePath, "source: narratio.artifact.session_recap", "source: narratio.previous_session.artifact.session_recap")
|
||||||
|
replaceInFileOrFatal(t, sessionPath, "session_id: 2026-05-03\n", "session_id: 2026-05-03\nprevious_session_id: 2026-04-26\n")
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{
|
||||||
|
"session", "validate", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code == 0 {
|
||||||
|
t.Fatal("exit code = 0, want non-zero")
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "ERROR previous") {
|
||||||
|
t.Fatalf("stdout = %q, want previous finding error", stdout.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "remote current run pointer missing") {
|
||||||
|
t.Fatalf("stdout = %q, want missing run pointer finding", stdout.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stderr.String(), "validation error(s)") {
|
||||||
|
t.Fatalf("stderr = %q, want finding error summary", stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecutePublishLoadsRemoteLocks(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidPublishRunConfigFiles(t, workspaceRoot)
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
lockKey := artifacts.S3SessionLocksKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"))
|
lockKey := artifacts.S3SessionLocksKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"))
|
||||||
fake.SeedObject(storage.FakeObject{Key: lockKey, Data: []byte("locks:\n - source: narratio.transcript.final_trimmed\n reason: remote review\n")})
|
fake.SeedObject(storage.FakeObject{Key: lockKey, Data: []byte("locks:\n - source: narratio.transcript.final_trimmed\n reason: remote review\n")})
|
||||||
@@ -871,40 +997,57 @@ func TestExecuteArchiveLoadsRemoteLocks(t *testing.T) {
|
|||||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze"} {
|
||||||
// The archive stage only checks the manifest statuses and source files.
|
// The publish stage only checks the manifest statuses and source files.
|
||||||
_ = stageName
|
_ = stageName
|
||||||
}
|
}
|
||||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.trimmed.json"), `{"segments":[]}`)
|
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.trimmed.json"), `{"segments":[]}`)
|
||||||
|
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.md"), "# final\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.trimmed.md"), "# final trimmed\n")
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
var stdout bytes.Buffer
|
||||||
var stderr bytes.Buffer
|
var stderr bytes.Buffer
|
||||||
code := Execute([]string{"run-stage", "archive", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--force"}, &stdout, &stderr)
|
code := Execute([]string{"run-stage", "publish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--force"}, &stdout, &stderr)
|
||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
promotedKey := artifacts.S3PromotedArtifactKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"), "transcripts/final.trimmed.json")
|
publishedKey := artifacts.S3PublishedOutputKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"), "transcripts/final.trimmed.json")
|
||||||
if _, ok := fake.Objects[promotedKey]; ok {
|
if _, ok := fake.Objects[publishedKey]; ok {
|
||||||
t.Fatalf("locked promoted key %q was uploaded", promotedKey)
|
t.Fatalf("locked published key %q was uploaded", publishedKey)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func addArchivePromotionsToPipeline(t *testing.T, pipelinePath, archiveYAML string) {
|
func addPublishOutputsToPipeline(t *testing.T, pipelinePath, publishYAML string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
data, err := os.ReadFile(pipelinePath)
|
data, err := os.ReadFile(pipelinePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("read pipeline: %v", err)
|
t.Fatalf("read pipeline: %v", err)
|
||||||
}
|
}
|
||||||
updated := strings.Replace(string(data), " upload_run: false\n", " upload_run: false\n"+archiveYAML, 1)
|
updated := strings.Replace(string(data), " upload_run: false\n", " upload_run: false\n"+publishYAML, 1)
|
||||||
if updated == string(data) {
|
if updated == string(data) {
|
||||||
t.Fatalf("pipeline %q did not contain archive upload_run marker", pipelinePath)
|
t.Fatalf("pipeline %q did not contain publish upload_run marker", pipelinePath)
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(pipelinePath, []byte(updated), 0o644); err != nil {
|
if err := os.WriteFile(pipelinePath, []byte(updated), 0o644); err != nil {
|
||||||
t.Fatalf("write pipeline: %v", err)
|
t.Fatalf("write pipeline: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeValidArchiveConfigFiles(t *testing.T, workspaceRoot string) (string, string, string) {
|
func replaceInFileOrFatal(t *testing.T, path, old, new string) {
|
||||||
|
t.Helper()
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read %s: %v", path, err)
|
||||||
|
}
|
||||||
|
updated := strings.Replace(string(data), old, new, 1)
|
||||||
|
if updated == string(data) {
|
||||||
|
t.Fatalf("%s did not contain %q", path, old)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(path, []byte(updated), 0o644); err != nil {
|
||||||
|
t.Fatalf("write %s: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeValidPublishRunConfigFiles(t *testing.T, workspaceRoot string) (string, string, string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
data, err := os.ReadFile(pipelinePath)
|
data, err := os.ReadFile(pipelinePath)
|
||||||
@@ -924,7 +1067,7 @@ func writeValidArchiveConfigFiles(t *testing.T, workspaceRoot string) (string, s
|
|||||||
m := manifest.New("2026-05-03", nowUTC())
|
m := manifest.New("2026-05-03", nowUTC())
|
||||||
m.Campaign = "sample-campaign"
|
m.Campaign = "sample-campaign"
|
||||||
m.RunID = "20260521T160000Z-test"
|
m.RunID = "20260521T160000Z-test"
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze"} {
|
||||||
m.MarkStageSucceeded(name, nowUTC(), nil)
|
m.MarkStageSucceeded(name, nowUTC(), nil)
|
||||||
}
|
}
|
||||||
path := artifacts.SessionManifestPathForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
path := artifacts.SessionManifestPathForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
@@ -941,7 +1084,7 @@ func writeValidArchiveConfigFiles(t *testing.T, workspaceRoot string) (string, s
|
|||||||
return pipelinePath, campaignPath, sessionPath
|
return pipelinePath, campaignPath, sessionPath
|
||||||
}
|
}
|
||||||
|
|
||||||
func addStaticArchiveLockToPipelineConfig(t *testing.T, pipelinePath, source string) {
|
func addStaticPublishLockToPipelineConfig(t *testing.T, pipelinePath, source string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
data, err := os.ReadFile(pipelinePath)
|
data, err := os.ReadFile(pipelinePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -949,12 +1092,12 @@ func addStaticArchiveLockToPipelineConfig(t *testing.T, pipelinePath, source str
|
|||||||
}
|
}
|
||||||
updated := strings.Replace(
|
updated := strings.Replace(
|
||||||
string(data),
|
string(data),
|
||||||
"archive:\n enabled: true\n upload_run: false\n",
|
"publish:\n enabled: true\n upload_run: false\n",
|
||||||
"archive:\n enabled: true\n upload_run: false\n locks:\n - source: "+source+"\n reason: static review\n",
|
"publish:\n enabled: true\n upload_run: false\n locks:\n - source: "+source+"\n reason: static review\n",
|
||||||
1,
|
1,
|
||||||
)
|
)
|
||||||
if updated == string(data) {
|
if updated == string(data) {
|
||||||
t.Fatalf("archive section not found in pipeline config")
|
t.Fatalf("publish section not found in pipeline config")
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(pipelinePath, []byte(updated), 0o644); err != nil {
|
if err := os.WriteFile(pipelinePath, []byte(updated), 0o644); err != nil {
|
||||||
t.Fatalf("write pipeline: %v", err)
|
t.Fatalf("write pipeline: %v", err)
|
||||||
|
|||||||
274
internal/app/operator_inspection.go
Normal file
274
internal/app/operator_inspection.go
Normal file
@@ -0,0 +1,274 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
type stableInputCheck struct {
|
||||||
|
Name string
|
||||||
|
Path string
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type localAudioCheck struct {
|
||||||
|
Checked bool
|
||||||
|
Paths []string
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type remoteAudioCheck struct {
|
||||||
|
Checked bool
|
||||||
|
Prefix string
|
||||||
|
Keys []string
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type previousArtifactReadiness struct {
|
||||||
|
Requirements []artifacts.PreviousArtifactRequirement
|
||||||
|
MissingID bool
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type remoteCurrentStateCheck struct {
|
||||||
|
State *RemoteCurrentState
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type effectiveLocksCheck struct {
|
||||||
|
Locks *effectiveLocks
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectStableInputs(cfg *config.Config) []stableInputCheck {
|
||||||
|
items := []struct {
|
||||||
|
name string
|
||||||
|
in config.ResolvedInputFile
|
||||||
|
}{
|
||||||
|
{name: "speakers", in: cfg.StableInputs.SpeakersFile},
|
||||||
|
{name: "autocorrect", in: cfg.StableInputs.AutocorrectFile},
|
||||||
|
{name: "glossary", in: cfg.StableInputs.GlossaryFile},
|
||||||
|
}
|
||||||
|
out := make([]stableInputCheck, 0, len(items))
|
||||||
|
for _, item := range items {
|
||||||
|
path, err := resolveHelperConfigRelativePath(item.in)
|
||||||
|
if err != nil {
|
||||||
|
out = append(out, stableInputCheck{Name: item.name, Err: err})
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(path); err != nil {
|
||||||
|
out = append(out, stableInputCheck{Name: item.name, Path: path, Err: err})
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, stableInputCheck{Name: item.name, Path: path})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectLocalAudioPresence(cfg *config.Config) localAudioCheck {
|
||||||
|
if cfg.Session.Inputs.AudioS3 != nil {
|
||||||
|
return localAudioCheck{}
|
||||||
|
}
|
||||||
|
|
||||||
|
sessionDir := filepath.Dir(cfg.SessionPath)
|
||||||
|
resolved, err := resolveLocalInspectionAudioPaths(sessionDir, cfg.Session.Inputs)
|
||||||
|
if err != nil {
|
||||||
|
return localAudioCheck{Checked: true, Err: err}
|
||||||
|
}
|
||||||
|
return localAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Paths: resolved,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectRemoteAudioPresence(ctx context.Context, cfg *config.Config, store storage.ObjectStore) remoteAudioCheck {
|
||||||
|
if cfg.Session.Inputs.AudioS3 == nil {
|
||||||
|
return remoteAudioCheck{}
|
||||||
|
}
|
||||||
|
if store == nil {
|
||||||
|
return remoteAudioCheck{Checked: true, Err: fmt.Errorf("storage backend is required for remote audio checks")}
|
||||||
|
}
|
||||||
|
|
||||||
|
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
audioPrefix := artifacts.S3AudioPrefix(sessionPrefix, cfg.Session.Inputs.AudioS3.Prefix)
|
||||||
|
objects, err := store.List(ctx, audioPrefix)
|
||||||
|
if err != nil {
|
||||||
|
return remoteAudioCheck{Checked: true, Prefix: audioPrefix, Err: err}
|
||||||
|
}
|
||||||
|
|
||||||
|
keys := make([]string, 0, len(objects))
|
||||||
|
seenBase := map[string]string{}
|
||||||
|
for _, obj := range objects {
|
||||||
|
key := strings.TrimSpace(obj.Key)
|
||||||
|
if key == "" || strings.HasSuffix(key, "/") || !isInspectionFlacPath(key) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
base := path.Base(key)
|
||||||
|
if prev, exists := seenBase[base]; exists && prev != key {
|
||||||
|
return remoteAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Prefix: audioPrefix,
|
||||||
|
Err: fmt.Errorf("duplicate s3 audio basename %q from %q and %q", base, prev, key),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
seenBase[base] = key
|
||||||
|
keys = append(keys, key)
|
||||||
|
}
|
||||||
|
sort.Strings(keys)
|
||||||
|
if len(keys) == 0 {
|
||||||
|
return remoteAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Prefix: audioPrefix,
|
||||||
|
Err: fmt.Errorf("no .flac files found under s3 audio prefix %q", audioPrefix),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return remoteAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Prefix: audioPrefix,
|
||||||
|
Keys: keys,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectPreviousArtifactReadiness(
|
||||||
|
ctx context.Context,
|
||||||
|
cfg *config.Config,
|
||||||
|
store storage.ObjectStore,
|
||||||
|
requirements []artifacts.PreviousArtifactRequirement,
|
||||||
|
) previousArtifactReadiness {
|
||||||
|
out := previousArtifactReadiness{
|
||||||
|
Requirements: append([]artifacts.PreviousArtifactRequirement(nil), requirements...),
|
||||||
|
}
|
||||||
|
if len(requirements) == 0 {
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(cfg.Session.PreviousSessionID) == "" {
|
||||||
|
out.MissingID = true
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
if store == nil {
|
||||||
|
out.Err = fmt.Errorf("previous-session artifacts cannot be checked because storage is unavailable")
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
prefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
||||||
|
if _, err := artifacts.LoadCurrentState(ctx, store, prefix, artifacts.CurrentStateValidation{
|
||||||
|
ExpectedSessionID: strings.TrimSpace(cfg.Session.PreviousSessionID),
|
||||||
|
ExpectedCampaign: strings.TrimSpace(cfg.Session.Campaign),
|
||||||
|
ValidateRunID: true,
|
||||||
|
}); err != nil {
|
||||||
|
out.Err = fmt.Errorf("remote %v", err)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectRemoteCurrentState(ctx context.Context, cfg *config.Config, store storage.ObjectStore) remoteCurrentStateCheck {
|
||||||
|
if store == nil {
|
||||||
|
return remoteCurrentStateCheck{}
|
||||||
|
}
|
||||||
|
current, err := discoverRemoteCurrentStateFn(ctx, cfg, store)
|
||||||
|
if err != nil {
|
||||||
|
return remoteCurrentStateCheck{Err: err}
|
||||||
|
}
|
||||||
|
return remoteCurrentStateCheck{State: current}
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectEffectiveLocks(ctx context.Context, cfg *config.Config, store storage.ObjectStore) effectiveLocksCheck {
|
||||||
|
locks, err := loadEffectiveLocks(ctx, cfg, store)
|
||||||
|
if err != nil {
|
||||||
|
return effectiveLocksCheck{Err: err}
|
||||||
|
}
|
||||||
|
return effectiveLocksCheck{Locks: locks}
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveLocalInspectionAudioPaths(sessionDir string, inputs config.SessionInputsConfig) ([]string, error) {
|
||||||
|
if len(inputs.AudioFiles) > 0 {
|
||||||
|
out := make([]string, 0, len(inputs.AudioFiles))
|
||||||
|
seenBase := map[string]string{}
|
||||||
|
for _, item := range inputs.AudioFiles {
|
||||||
|
resolved, err := resolveInspectionPath(sessionDir, item)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if !isInspectionFlacPath(resolved) {
|
||||||
|
return nil, fmt.Errorf("audio file %q must have .flac extension", resolved)
|
||||||
|
}
|
||||||
|
if err := requireInspectionFile(resolved, "audio file"); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
base := filepath.Base(resolved)
|
||||||
|
if prev, exists := seenBase[base]; exists && prev != resolved {
|
||||||
|
return nil, fmt.Errorf("duplicate audio basename %q from %q and %q", base, prev, resolved)
|
||||||
|
}
|
||||||
|
seenBase[base] = resolved
|
||||||
|
out = append(out, resolved)
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
audioDir, err := resolveInspectionPath(sessionDir, inputs.AudioDir)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
entries, err := os.ReadDir(audioDir)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read audio directory %q: %w", audioDir, err)
|
||||||
|
}
|
||||||
|
out := make([]string, 0, len(entries))
|
||||||
|
for _, entry := range entries {
|
||||||
|
if entry.IsDir() {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
full := filepath.Join(audioDir, entry.Name())
|
||||||
|
if !isInspectionFlacPath(full) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err := requireInspectionFile(full, "audio file"); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, full)
|
||||||
|
}
|
||||||
|
if len(out) == 0 {
|
||||||
|
return nil, fmt.Errorf("no .flac files found in audio directory %q", audioDir)
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveInspectionPath(baseDir, inputPath string) (string, error) {
|
||||||
|
pathValue := strings.TrimSpace(inputPath)
|
||||||
|
if pathValue == "" {
|
||||||
|
return "", fmt.Errorf("path is required")
|
||||||
|
}
|
||||||
|
if filepath.IsAbs(pathValue) {
|
||||||
|
return filepath.Clean(pathValue), nil
|
||||||
|
}
|
||||||
|
return filepath.Clean(filepath.Join(baseDir, pathValue)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireInspectionFile(path, label string) error {
|
||||||
|
info, err := os.Stat(path)
|
||||||
|
if err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return fmt.Errorf("%s %q does not exist", label, path)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("stat %s %q: %w", label, path, err)
|
||||||
|
}
|
||||||
|
if info.IsDir() {
|
||||||
|
return fmt.Errorf("%s %q is a directory", label, path)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func isInspectionFlacPath(path string) bool {
|
||||||
|
return strings.EqualFold(filepath.Ext(strings.TrimSpace(path)), ".flac")
|
||||||
|
}
|
||||||
172
internal/app/operator_locks.go
Normal file
172
internal/app/operator_locks.go
Normal file
@@ -0,0 +1,172 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Locks dispatches publish lock list and mutation helpers.
|
||||||
|
func Locks(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
|
||||||
|
switch args[0] {
|
||||||
|
case "add":
|
||||||
|
return LocksAdd(ctx, args[1:], out)
|
||||||
|
case "remove":
|
||||||
|
return LocksRemove(ctx, args[1:], out)
|
||||||
|
default:
|
||||||
|
return fmt.Errorf("locks: unknown subcommand %q", args[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return LocksList(ctx, args, out)
|
||||||
|
}
|
||||||
|
|
||||||
|
// LocksList lists effective publish locks.
|
||||||
|
func LocksList(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("locks", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
if err := parseSessionAwareFlags("locks", fs, args, &flags.sessionID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("locks: session_id is required")
|
||||||
|
}
|
||||||
|
cfg, _, locks, _, err := loadHelperContext(ctx, flags, true)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("locks: %w", err)
|
||||||
|
}
|
||||||
|
writeLocks(out, cfg, locks)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// LocksAdd adds or updates one remote lock.
|
||||||
|
func LocksAdd(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("locks add", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
var reason string
|
||||||
|
var force bool
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
fs.StringVar(&reason, "reason", "", "lock reason")
|
||||||
|
fs.BoolVar(&force, "force", false, "update existing remote lock")
|
||||||
|
source, err := parseSessionIDAndOnePositionalArg("locks add", "source id", fs, args, &flags.sessionID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("locks add: session_id is required")
|
||||||
|
}
|
||||||
|
cfg, store, locks, _, err := loadHelperContext(ctx, flags, true)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("locks add: %w", err)
|
||||||
|
}
|
||||||
|
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, "locks add"); err != nil {
|
||||||
|
return fmt.Errorf("locks add: %w", err)
|
||||||
|
}
|
||||||
|
if _, ok := lockSourceSet(locks.Static)[source]; ok {
|
||||||
|
return fmt.Errorf("locks add: source %q is locked by pipeline config and cannot be modified remotely", source)
|
||||||
|
}
|
||||||
|
remoteSet := lockSourceSet(locks.Remote)
|
||||||
|
if _, exists := remoteSet[source]; exists && !force {
|
||||||
|
return fmt.Errorf("locks add: remote lock for %q already exists; pass --force to update", source)
|
||||||
|
}
|
||||||
|
remoteSet[source] = config.PublishLockRule{Source: source, Reason: strings.TrimSpace(reason)}
|
||||||
|
remoteLocks := lockMapValues(remoteSet)
|
||||||
|
if _, err := config.ValidatePublishLockRules(remoteLocks, cfg.Pipeline.Scriptorium, "locks"); err != nil {
|
||||||
|
return fmt.Errorf("locks add: %w", err)
|
||||||
|
}
|
||||||
|
if err := uploadRemoteLockStore(ctx, store, locks.Key, &config.PublishLockStore{Locks: remoteLocks}); err != nil {
|
||||||
|
return fmt.Errorf("locks add: %w", err)
|
||||||
|
}
|
||||||
|
_, err = fmt.Fprintf(out, "narratio session locks add: locked %s\n", source)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// LocksRemove removes one remote lock.
|
||||||
|
func LocksRemove(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("locks remove", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
source, err := parseSessionIDAndOnePositionalArg("locks remove", "source id", fs, args, &flags.sessionID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("locks remove: session_id is required")
|
||||||
|
}
|
||||||
|
cfg, store, locks, _, err := loadHelperContext(ctx, flags, true)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("locks remove: %w", err)
|
||||||
|
}
|
||||||
|
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, "locks remove"); err != nil {
|
||||||
|
return fmt.Errorf("locks remove: %w", err)
|
||||||
|
}
|
||||||
|
remoteSet := lockSourceSet(locks.Remote)
|
||||||
|
if _, ok := remoteSet[source]; !ok {
|
||||||
|
if _, static := lockSourceSet(locks.Static)[source]; static {
|
||||||
|
return fmt.Errorf("locks remove: source %q is locked by pipeline config and cannot be unlocked remotely", source)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("locks remove: remote lock for %q does not exist", source)
|
||||||
|
}
|
||||||
|
delete(remoteSet, source)
|
||||||
|
remoteLocks := lockMapValues(remoteSet)
|
||||||
|
if err := uploadRemoteLockStore(ctx, store, locks.Key, &config.PublishLockStore{Locks: remoteLocks}); err != nil {
|
||||||
|
return fmt.Errorf("locks remove: %w", err)
|
||||||
|
}
|
||||||
|
_, err = fmt.Fprintf(out, "narratio session locks remove: unlocked %s\n", source)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeLocks(out io.Writer, cfg *config.Config, locks *effectiveLocks) {
|
||||||
|
if locks == nil || len(locks.All) == 0 {
|
||||||
|
fmt.Fprintln(out, "Publish locks: none")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Publish locks:")
|
||||||
|
published := map[string]config.PublishOutputRule{}
|
||||||
|
if cfg != nil && cfg.Pipeline != nil && cfg.Pipeline.Publish != nil {
|
||||||
|
for _, rule := range cfg.Pipeline.Publish.Outputs {
|
||||||
|
published[strings.TrimSpace(rule.Source)] = rule
|
||||||
|
}
|
||||||
|
}
|
||||||
|
staticSet := lockSourceSet(locks.Static)
|
||||||
|
for _, lock := range locks.All {
|
||||||
|
origin := "remote"
|
||||||
|
if _, ok := staticSet[lock.Source]; ok {
|
||||||
|
origin = "pipeline"
|
||||||
|
}
|
||||||
|
promo := "not-published"
|
||||||
|
if _, ok := published[lock.Source]; ok {
|
||||||
|
promo = "published"
|
||||||
|
}
|
||||||
|
reason := strings.TrimSpace(lock.Reason)
|
||||||
|
if reason == "" {
|
||||||
|
reason = "(no reason)"
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "- %s origin=%s %s reason=%s\n", lock.Source, origin, promo, reason)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func lockMapValues(in map[string]config.PublishLockRule) []config.PublishLockRule {
|
||||||
|
keys := make([]string, 0, len(in))
|
||||||
|
for key := range in {
|
||||||
|
keys = append(keys, key)
|
||||||
|
}
|
||||||
|
sort.Strings(keys)
|
||||||
|
out := make([]config.PublishLockRule, 0, len(keys))
|
||||||
|
for _, key := range keys {
|
||||||
|
item := in[key]
|
||||||
|
item.Source = key
|
||||||
|
item.Reason = strings.TrimSpace(item.Reason)
|
||||||
|
out = append(out, item)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
268
internal/app/operator_session_init.go
Normal file
268
internal/app/operator_session_init.go
Normal file
@@ -0,0 +1,268 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gopkg.in/yaml.v3"
|
||||||
|
)
|
||||||
|
|
||||||
|
// SessionInit creates a local or remote session.yml skeleton.
|
||||||
|
func SessionInit(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("session init", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var pipelinePath, campaignPath, campaignFilePath, sessionID, previousSessionID, date, title, output, audioS3Prefix, audioDir string
|
||||||
|
var remote, force bool
|
||||||
|
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||||
|
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
||||||
|
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
||||||
|
fs.StringVar(&sessionID, "session-id", "", "session identifier")
|
||||||
|
fs.StringVar(&previousSessionID, "previous-session-id", "", "previous session identifier")
|
||||||
|
fs.StringVar(&date, "date", "", "session date")
|
||||||
|
fs.StringVar(&title, "title", "", "session title")
|
||||||
|
fs.StringVar(&output, "output", "", "local output session.yml path")
|
||||||
|
fs.StringVar(&audioS3Prefix, "audio-s3-prefix", "", "session audio S3 prefix")
|
||||||
|
fs.StringVar(&audioDir, "audio-dir", "", "local audio directory")
|
||||||
|
fs.BoolVar(&remote, "remote", false, "write session.yml to S3 session prefix")
|
||||||
|
fs.BoolVar(&force, "force", false, "overwrite existing target")
|
||||||
|
if err := parseSessionAwareFlags("session init", fs, args, &sessionID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(sessionID) == "" {
|
||||||
|
return fmt.Errorf("session init: session_id is required")
|
||||||
|
}
|
||||||
|
if (strings.TrimSpace(output) == "") == !remote {
|
||||||
|
return fmt.Errorf("session init: specify exactly one target: --output <path> or --remote")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(audioDir) != "" && strings.TrimSpace(audioS3Prefix) != "" {
|
||||||
|
return fmt.Errorf("session init: --audio-dir and --audio-s3-prefix are mutually exclusive")
|
||||||
|
}
|
||||||
|
|
||||||
|
base, err := loadPipelineCampaignConfig(pipelinePath, campaignPath, campaignFilePath)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
input := sessionInitInput{
|
||||||
|
Campaign: config.CampaignID(base.Campaign),
|
||||||
|
CampaignPath: base.CampaignPath,
|
||||||
|
TemplateFile: base.Campaign.SessionTemplateFile,
|
||||||
|
SessionID: sessionID,
|
||||||
|
PreviousSessionID: previousSessionID,
|
||||||
|
Date: date,
|
||||||
|
Title: title,
|
||||||
|
AudioS3Prefix: audioS3Prefix,
|
||||||
|
AudioDir: audioDir,
|
||||||
|
}
|
||||||
|
data, err := buildSessionInitYAML(input)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
label := strings.TrimSpace(output)
|
||||||
|
if label == "" {
|
||||||
|
label = "remote session.yml"
|
||||||
|
}
|
||||||
|
sessionCfg, err := config.LoadSessionBytesWithOptions(label, data, config.SessionLoadOptions{
|
||||||
|
SessionID: sessionID,
|
||||||
|
PreviousSessionID: previousSessionID,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
cfg, err := config.Resolve(base.PipelinePath, base.Pipeline, base.CampaignPath, base.Campaign, label, sessionCfg, config.SessionSource{Source: "session_config", LocalPath: label})
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
if err := config.Validate(cfg); err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !remote {
|
||||||
|
if err := writeLocalFile(output, data, force); err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
_, err := fmt.Fprintf(out, "narratio session init: wrote %s\n", filepath.Clean(output))
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
store, err := newCommandObjectStore(ctx, cfg, nil)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
sessionPrefix := artifacts.S3SessionPrefix(base.Pipeline.Storage.S3.RootPrefix, config.CampaignID(base.Campaign), sessionID)
|
||||||
|
key := artifacts.S3SessionConfigKey(sessionPrefix)
|
||||||
|
exists, err := store.Exists(ctx, key)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: check remote session %q: %w", key, err)
|
||||||
|
}
|
||||||
|
if exists && !force {
|
||||||
|
return fmt.Errorf("session init: remote session %q already exists; pass --force to overwrite", key)
|
||||||
|
}
|
||||||
|
tmp, err := os.CreateTemp("", "narratio-session-init-*.yml")
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: create temp file: %w", err)
|
||||||
|
}
|
||||||
|
tmpPath := tmp.Name()
|
||||||
|
defer func() { _ = os.Remove(tmpPath) }()
|
||||||
|
if _, err := tmp.Write(data); err != nil {
|
||||||
|
_ = tmp.Close()
|
||||||
|
return fmt.Errorf("session init: write temp file: %w", err)
|
||||||
|
}
|
||||||
|
if err := tmp.Close(); err != nil {
|
||||||
|
return fmt.Errorf("session init: close temp file: %w", err)
|
||||||
|
}
|
||||||
|
if _, err := store.Upload(ctx, tmpPath, key, storage.UploadOptions{ContentType: "application/x-yaml; charset=utf-8"}); err != nil {
|
||||||
|
return fmt.Errorf("session init: upload remote session %q: %w", key, err)
|
||||||
|
}
|
||||||
|
_, err = fmt.Fprintf(out, "narratio session init: wrote s3://%s/%s\n", s3BucketName(base.Pipeline), key)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildSessionYAML(campaign, sessionID, previousSessionID, date, title, audioS3Prefix, audioDir string) ([]byte, error) {
|
||||||
|
if strings.TrimSpace(date) == "" && regexp.MustCompile(`^\d{4}-\d{2}-\d{2}$`).MatchString(strings.TrimSpace(sessionID)) {
|
||||||
|
date = strings.TrimSpace(sessionID)
|
||||||
|
}
|
||||||
|
type audioS3 struct {
|
||||||
|
Prefix string `yaml:"prefix"`
|
||||||
|
}
|
||||||
|
type inputs struct {
|
||||||
|
AudioDir string `yaml:"audio_dir,omitempty"`
|
||||||
|
AudioS3 *audioS3 `yaml:"audio_s3,omitempty"`
|
||||||
|
}
|
||||||
|
type sessionYAML struct {
|
||||||
|
Campaign string `yaml:"campaign"`
|
||||||
|
SessionID string `yaml:"session_id"`
|
||||||
|
PreviousSessionID string `yaml:"previous_session_id,omitempty"`
|
||||||
|
Date string `yaml:"date,omitempty"`
|
||||||
|
Title string `yaml:"title,omitempty"`
|
||||||
|
Inputs inputs `yaml:"inputs"`
|
||||||
|
}
|
||||||
|
in := inputs{AudioDir: strings.TrimSpace(audioDir)}
|
||||||
|
if in.AudioDir == "" {
|
||||||
|
prefix := strings.TrimSpace(audioS3Prefix)
|
||||||
|
if prefix == "" {
|
||||||
|
prefix = "audio/"
|
||||||
|
}
|
||||||
|
in.AudioS3 = &audioS3{Prefix: prefix}
|
||||||
|
}
|
||||||
|
data, err := yaml.Marshal(sessionYAML{
|
||||||
|
Campaign: strings.TrimSpace(campaign),
|
||||||
|
SessionID: strings.TrimSpace(sessionID),
|
||||||
|
PreviousSessionID: strings.TrimSpace(previousSessionID),
|
||||||
|
Date: strings.TrimSpace(date),
|
||||||
|
Title: strings.TrimSpace(title),
|
||||||
|
Inputs: in,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return data, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type sessionInitInput struct {
|
||||||
|
Campaign string
|
||||||
|
CampaignPath string
|
||||||
|
TemplateFile string
|
||||||
|
SessionID string
|
||||||
|
PreviousSessionID string
|
||||||
|
Date string
|
||||||
|
Title string
|
||||||
|
AudioS3Prefix string
|
||||||
|
AudioDir string
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildSessionInitYAML(in sessionInitInput) ([]byte, error) {
|
||||||
|
if strings.TrimSpace(in.TemplateFile) == "" {
|
||||||
|
return buildSessionYAML(in.Campaign, in.SessionID, in.PreviousSessionID, in.Date, in.Title, in.AudioS3Prefix, in.AudioDir)
|
||||||
|
}
|
||||||
|
templatePath := resolveSessionInitTemplatePath(in.CampaignPath, in.TemplateFile)
|
||||||
|
templateBytes, err := os.ReadFile(templatePath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read session template %q: %w", templatePath, err)
|
||||||
|
}
|
||||||
|
rendered, err := renderSessionInitTemplate(string(templateBytes), in)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("render session template %q: %w", templatePath, err)
|
||||||
|
}
|
||||||
|
return []byte(rendered), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveSessionInitTemplatePath(campaignPath, templateFile string) string {
|
||||||
|
templateFile = strings.TrimSpace(templateFile)
|
||||||
|
if filepath.IsAbs(templateFile) {
|
||||||
|
return filepath.Clean(templateFile)
|
||||||
|
}
|
||||||
|
return filepath.Clean(filepath.Join(filepath.Dir(campaignPath), templateFile))
|
||||||
|
}
|
||||||
|
|
||||||
|
var sessionInitTemplatePattern = regexp.MustCompile(`\{\{\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*\}\}`)
|
||||||
|
|
||||||
|
func renderSessionInitTemplate(content string, in sessionInitInput) (string, error) {
|
||||||
|
values := map[string]string{
|
||||||
|
"session_id": strings.TrimSpace(in.SessionID),
|
||||||
|
"previous_session_id": strings.TrimSpace(in.PreviousSessionID),
|
||||||
|
"date": strings.TrimSpace(in.Date),
|
||||||
|
"title": strings.TrimSpace(in.Title),
|
||||||
|
"audio_s3_prefix": strings.TrimSpace(in.AudioS3Prefix),
|
||||||
|
"audio_dir": strings.TrimSpace(in.AudioDir),
|
||||||
|
}
|
||||||
|
used := map[string]struct{}{}
|
||||||
|
unknown := map[string]struct{}{}
|
||||||
|
missing := map[string]struct{}{}
|
||||||
|
rendered := sessionInitTemplatePattern.ReplaceAllStringFunc(content, func(match string) string {
|
||||||
|
parts := sessionInitTemplatePattern.FindStringSubmatch(match)
|
||||||
|
if len(parts) < 2 {
|
||||||
|
return match
|
||||||
|
}
|
||||||
|
name := parts[1]
|
||||||
|
value, ok := values[name]
|
||||||
|
if !ok {
|
||||||
|
unknown[name] = struct{}{}
|
||||||
|
return match
|
||||||
|
}
|
||||||
|
used[name] = struct{}{}
|
||||||
|
if value == "" {
|
||||||
|
missing[name] = struct{}{}
|
||||||
|
return match
|
||||||
|
}
|
||||||
|
return value
|
||||||
|
})
|
||||||
|
if len(unknown) > 0 {
|
||||||
|
return "", fmt.Errorf("unsupported template variable(s): %s", sortedStringSet(unknown))
|
||||||
|
}
|
||||||
|
if len(missing) > 0 {
|
||||||
|
return "", fmt.Errorf("missing required template variable value(s): %s", sortedStringSet(missing))
|
||||||
|
}
|
||||||
|
unused := map[string]struct{}{}
|
||||||
|
for _, name := range []string{"previous_session_id", "date", "title", "audio_s3_prefix", "audio_dir"} {
|
||||||
|
if values[name] == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, ok := used[name]; !ok {
|
||||||
|
unused[name] = struct{}{}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(unused) > 0 {
|
||||||
|
return "", fmt.Errorf("unused template variable value(s): %s", sortedStringSet(unused))
|
||||||
|
}
|
||||||
|
return rendered, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func sortedStringSet(set map[string]struct{}) string {
|
||||||
|
items := make([]string, 0, len(set))
|
||||||
|
for item := range set {
|
||||||
|
items = append(items, item)
|
||||||
|
}
|
||||||
|
sort.Strings(items)
|
||||||
|
return strings.Join(items, ", ")
|
||||||
|
}
|
||||||
84
internal/app/operator_session_validate.go
Normal file
84
internal/app/operator_session_validate.go
Normal file
@@ -0,0 +1,84 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// SessionValidate performs a read-only session preflight.
|
||||||
|
func SessionValidate(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("session validate", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
if err := parseSessionAwareFlags("session validate", fs, args, &flags.sessionID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("session validate: session_id is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
findings := []finding{}
|
||||||
|
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||||
|
if err != nil {
|
||||||
|
findings = append(findings, errorFinding("config", err.Error()))
|
||||||
|
return renderFindings(out, "", "", findings)
|
||||||
|
}
|
||||||
|
if err := config.Validate(cfg); err != nil {
|
||||||
|
findings = append(findings, errorFinding("config", err.Error()))
|
||||||
|
} else {
|
||||||
|
findings = append(findings, okFinding("config", "resolved pipeline, campaign, and session config"))
|
||||||
|
}
|
||||||
|
findings = append(findings, okFinding("session", fmt.Sprintf("session source: %s", sessionSourceSummary(cfg))))
|
||||||
|
|
||||||
|
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
findings = append(findings, validateStableInputFindings(cfg)...)
|
||||||
|
findings = append(findings, validateLocalAudioFindings(cfg)...)
|
||||||
|
|
||||||
|
store, storeErr := objectStoreIfConfigured(ctx, cfg)
|
||||||
|
if storeErr != nil {
|
||||||
|
findings = append(findings, errorFinding("storage", storeErr.Error()))
|
||||||
|
}
|
||||||
|
if cfg.Session.Inputs.AudioS3 != nil {
|
||||||
|
if storeErr != nil {
|
||||||
|
findings = append(findings, errorFinding("audio", "remote audio cannot be checked because storage is unavailable"))
|
||||||
|
} else {
|
||||||
|
findings = append(findings, validateRemoteAudioFinding(ctx, cfg, store))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
requirements := artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg))
|
||||||
|
previous := inspectPreviousArtifactReadiness(ctx, cfg, store, requirements)
|
||||||
|
if len(previous.Requirements) == 0 {
|
||||||
|
findings = append(findings, okFinding("previous", "no previous-session artifacts required"))
|
||||||
|
} else if previous.MissingID {
|
||||||
|
findings = append(findings, errorFinding("previous", "previous_session_id is required by configured previous-session artifacts"))
|
||||||
|
} else if previous.Err != nil {
|
||||||
|
findings = append(findings, errorFinding("previous", previous.Err.Error()))
|
||||||
|
} else {
|
||||||
|
for _, req := range previous.Requirements {
|
||||||
|
findings = append(findings, okFinding("previous", fmt.Sprintf("%s required=%t", req.Name, req.Required)))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
locks := inspectEffectiveLocks(ctx, cfg, store)
|
||||||
|
if locks.Err != nil {
|
||||||
|
findings = append(findings, errorFinding("locks", locks.Err.Error()))
|
||||||
|
} else if len(locks.Locks.All) == 0 {
|
||||||
|
findings = append(findings, okFinding("locks", "no effective publish locks"))
|
||||||
|
} else {
|
||||||
|
for _, lock := range locks.Locks.All {
|
||||||
|
findings = append(findings, warnFinding("locks", fmt.Sprintf("%s locked: %s", lock.Source, strings.TrimSpace(lock.Reason))))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if paths.ManifestPath != "" {
|
||||||
|
findings = append(findings, infoFinding("workspace", "manifest path: "+paths.ManifestPath))
|
||||||
|
}
|
||||||
|
return renderFindings(out, cfg.Session.Campaign, cfg.Session.SessionID, findings)
|
||||||
|
}
|
||||||
166
internal/app/operator_status.go
Normal file
166
internal/app/operator_status.go
Normal file
@@ -0,0 +1,166 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Status reports effective local/remote session state.
|
||||||
|
func Status(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("status", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
if err := parseSessionAwareFlags("status", fs, args, &flags.sessionID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("status: session_id is required")
|
||||||
|
}
|
||||||
|
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("status: %w", err)
|
||||||
|
}
|
||||||
|
if err := config.Validate(cfg); err != nil {
|
||||||
|
return fmt.Errorf("status: %w", err)
|
||||||
|
}
|
||||||
|
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
fmt.Fprintf(out, "Session: %s\n", cfg.Session.SessionID)
|
||||||
|
fmt.Fprintf(out, "Campaign: %s\n", cfg.Session.Campaign)
|
||||||
|
fmt.Fprintf(out, "Workspace: %s\n", paths.Root)
|
||||||
|
fmt.Fprintf(out, "Session config: %s\n", sessionSourceSummary(cfg))
|
||||||
|
writeStatusStableInputs(out, inspectStableInputs(cfg))
|
||||||
|
writeStatusLocalAudio(out, inspectLocalAudioPresence(cfg))
|
||||||
|
|
||||||
|
if m, err := loadLocalManifest(ctx, paths.ManifestPath); err != nil {
|
||||||
|
fmt.Fprintf(out, "Local manifest: error: %v\n", err)
|
||||||
|
} else if m == nil {
|
||||||
|
fmt.Fprintln(out, "Local manifest: missing")
|
||||||
|
} else {
|
||||||
|
fmt.Fprintf(out, "Local manifest: %s\n", paths.ManifestPath)
|
||||||
|
writeStageStatuses(out, m)
|
||||||
|
}
|
||||||
|
|
||||||
|
store, storeErr := objectStoreIfConfigured(ctx, cfg)
|
||||||
|
if storeErr != nil {
|
||||||
|
fmt.Fprintf(out, "Remote publish: unavailable: %v\n", storeErr)
|
||||||
|
} else if store != nil {
|
||||||
|
current := inspectRemoteCurrentState(ctx, cfg, store)
|
||||||
|
if current.Err != nil {
|
||||||
|
fmt.Fprintf(out, "Remote publish: missing or unavailable: %v\n", current.Err)
|
||||||
|
} else {
|
||||||
|
fmt.Fprintf(out, "Remote publish: current run %s\n", current.State.RunID)
|
||||||
|
fmt.Fprintf(out, "Remote manifest: %s\n", current.State.CurrentManifestKey)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
writeStatusRemoteAudio(ctx, out, cfg, store, storeErr)
|
||||||
|
writeStatusPreviousArtifacts(out, inspectPreviousArtifactReadiness(
|
||||||
|
ctx,
|
||||||
|
cfg,
|
||||||
|
store,
|
||||||
|
artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg)),
|
||||||
|
))
|
||||||
|
|
||||||
|
lockChecks := inspectEffectiveLocks(ctx, cfg, store)
|
||||||
|
locks := lockChecks.Locks
|
||||||
|
lockErr := lockChecks.Err
|
||||||
|
if catalog, catalogErr := buildHelperArtifactCatalog(cfg); catalogErr != nil {
|
||||||
|
fmt.Fprintf(out, "Remote outputs: error: %v\n", catalogErr)
|
||||||
|
} else if storeErr == nil {
|
||||||
|
catalogLocks := locks
|
||||||
|
if lockErr != nil {
|
||||||
|
catalogLocks = &effectiveLocks{
|
||||||
|
Static: staticPublishLocks(cfg),
|
||||||
|
All: staticPublishLocks(cfg),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
publishedRemoteState := map[string]string{}
|
||||||
|
if store != nil {
|
||||||
|
publishedRemoteState = remotePublishedOutputAvailability(ctx, cfg, store, catalog)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Remote outputs:")
|
||||||
|
writeArtifactList(out, cfg, catalog, catalogLocks, publishedRemoteState)
|
||||||
|
}
|
||||||
|
if lockErr != nil {
|
||||||
|
fmt.Fprintf(out, "Publish locks: error: %v\n", lockErr)
|
||||||
|
} else {
|
||||||
|
writeLocks(out, cfg, locks)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Next actions:")
|
||||||
|
fmt.Fprintf(out, "- narratio session validate %s\n", cfg.Session.SessionID)
|
||||||
|
fmt.Fprintf(out, "- narratio session restore %s --dry-run\n", cfg.Session.SessionID)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStatusStableInputs(out io.Writer, checks []stableInputCheck) {
|
||||||
|
if len(checks) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for _, check := range checks {
|
||||||
|
if check.Err != nil {
|
||||||
|
if strings.TrimSpace(check.Path) != "" {
|
||||||
|
fmt.Fprintf(out, "Stable input %s: unavailable: %v\n", check.Name, check.Err)
|
||||||
|
} else {
|
||||||
|
fmt.Fprintf(out, "Stable input %s: unavailable: %s\n", check.Name, check.Err.Error())
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "Stable input %s: %s\n", check.Name, check.Path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStatusLocalAudio(out io.Writer, check localAudioCheck) {
|
||||||
|
if !check.Checked {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if check.Err != nil {
|
||||||
|
fmt.Fprintf(out, "Local audio: unavailable: %v\n", check.Err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "Local audio: %d file(s)\n", len(check.Paths))
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStatusRemoteAudio(ctx context.Context, out io.Writer, cfg *config.Config, store storage.ObjectStore, storeErr error) {
|
||||||
|
if cfg.Session.Inputs.AudioS3 == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if storeErr != nil {
|
||||||
|
fmt.Fprintf(out, "Remote audio: unavailable: %v\n", storeErr)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
check := inspectRemoteAudioPresence(ctx, cfg, store)
|
||||||
|
if check.Err != nil {
|
||||||
|
fmt.Fprintf(out, "Remote audio: unavailable: %v\n", check.Err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "Remote audio: %d .flac object(s)\n", len(check.Keys))
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStatusPreviousArtifacts(out io.Writer, readiness previousArtifactReadiness) {
|
||||||
|
if len(readiness.Requirements) == 0 {
|
||||||
|
fmt.Fprintln(out, "Previous-session artifacts: not required")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if readiness.MissingID {
|
||||||
|
fmt.Fprintln(out, "Previous-session artifacts: unavailable: previous_session_id is required by configured previous-session artifacts")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if readiness.Err != nil {
|
||||||
|
fmt.Fprintf(out, "Previous-session artifacts: unavailable: %v\n", readiness.Err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
names := make([]string, 0, len(readiness.Requirements))
|
||||||
|
for _, req := range readiness.Requirements {
|
||||||
|
names = append(names, fmt.Sprintf("%s(required=%t)", req.Name, req.Required))
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
fmt.Fprintf(out, "Previous-session artifacts: ready: %s\n", strings.Join(names, ", "))
|
||||||
|
}
|
||||||
@@ -7,7 +7,6 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"os"
|
"os"
|
||||||
"strings"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
@@ -17,46 +16,21 @@ import (
|
|||||||
|
|
||||||
// Plan validates configuration, prepares the local workdir, and prints stage order.
|
// Plan validates configuration, prepares the local workdir, and prints stage order.
|
||||||
func Plan(ctx context.Context, args []string, out io.Writer) error {
|
func Plan(ctx context.Context, args []string, out io.Writer) error {
|
||||||
positionalSessionID, args := pullLeadingSessionID(args)
|
|
||||||
fs := flag.NewFlagSet("plan", flag.ContinueOnError)
|
fs := flag.NewFlagSet("plan", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
|
|
||||||
var pipelinePath string
|
var flags commonConfigFlags
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var force bool
|
var force bool
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
if err := parseSessionAwareFlags("plan", fs, args, &flags.sessionID); err != nil {
|
||||||
return fmt.Errorf("plan: invalid flags: %w", err)
|
return err
|
||||||
}
|
}
|
||||||
if positionalSessionID == "" {
|
if flags.sessionID == "" {
|
||||||
if err := applyParsedSessionIDArg("plan", fs, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("plan: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("plan", positionalSessionID, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
|
||||||
return fmt.Errorf("plan: session_id is required")
|
return fmt.Errorf("plan: session_id is required")
|
||||||
}
|
}
|
||||||
cfg, err := loadCommandConfig(ctx, pipelinePath, campaignPath, campaignFilePath, sessionPath, config.SessionLoadOptions{
|
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||||
SessionID: sessionID,
|
|
||||||
PreviousSessionID: previousSessionID,
|
|
||||||
})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("plan: %w", err)
|
return fmt.Errorf("plan: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -27,12 +27,12 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
|
|||||||
if !strings.Contains(got, "narratio session plan: workdir prepared at") {
|
if !strings.Contains(got, "narratio session plan: workdir prepared at") {
|
||||||
t.Fatalf("first output = %q, want workdir prepared", got)
|
t.Fatalf("first output = %q, want workdir prepared", got)
|
||||||
}
|
}
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
|
||||||
if !strings.Contains(got, name+": run") {
|
if !strings.Contains(got, name+": run") {
|
||||||
t.Fatalf("first output = %q, missing stage %q", got, name)
|
t.Fatalf("first output = %q, missing stage %q", got, name)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !strings.Contains(got, "totals: run=9 skip=0") {
|
if !strings.Contains(got, "totals: run=10 skip=0") {
|
||||||
t.Fatalf("first output = %q, want totals", got)
|
t.Fatalf("first output = %q, want totals", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -84,8 +84,8 @@ func TestPlanShowsRunAndSkipFromManifest(t *testing.T) {
|
|||||||
if !strings.Contains(got, "trim: run") {
|
if !strings.Contains(got, "trim: run") {
|
||||||
t.Fatalf("output = %q, want trim run", got)
|
t.Fatalf("output = %q, want trim run", got)
|
||||||
}
|
}
|
||||||
if !strings.Contains(got, "totals: run=7 skip=2") {
|
if !strings.Contains(got, "totals: run=8 skip=2") {
|
||||||
t.Fatalf("output = %q, want totals run=7 skip=2", got)
|
t.Fatalf("output = %q, want totals run=8 skip=2", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ import "testing"
|
|||||||
|
|
||||||
func TestBuildFullPlanOrder(t *testing.T) {
|
func TestBuildFullPlanOrder(t *testing.T) {
|
||||||
got := BuildFullPlan()
|
got := BuildFullPlan()
|
||||||
want := []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"}
|
want := []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"}
|
||||||
if len(got) != len(want) {
|
if len(got) != len(want) {
|
||||||
t.Fatalf("len(plan) = %d, want %d", len(got), len(want))
|
t.Fatalf("len(plan) = %d, want %d", len(got), len(want))
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -12,18 +12,18 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m *manifest.Manifest, executed []string) error {
|
func runPostPublishCleanup(ctx context.Context, env *Env, manifestPath string, m *manifest.Manifest, executed []string) error {
|
||||||
if env == nil || env.Config == nil || env.Config.Pipeline == nil || m == nil {
|
if env == nil || env.Config == nil || env.Config.Pipeline == nil || m == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
spoolRequested := env.Config.Pipeline.Spool.DeleteAudioAfterArchive
|
spoolRequested := env.Config.Pipeline.Spool.DeleteAudioAfterPublish
|
||||||
workRequested := env.Config.Pipeline.Workspace.CleanupAfterArchive
|
workRequested := env.Config.Pipeline.Workspace.CleanupAfterPublish
|
||||||
if !spoolRequested && !workRequested {
|
if !spoolRequested && !workRequested {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
sr := archiveStageRecordForCleanup(m, executed)
|
sr := publishStageRecordForCleanup(m, executed)
|
||||||
if sr == nil {
|
if sr == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -33,7 +33,7 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
|||||||
sr.Metadata["spool_cleanup_requested"] = spoolRequested
|
sr.Metadata["spool_cleanup_requested"] = spoolRequested
|
||||||
sr.Metadata["workdir_cleanup_requested"] = workRequested
|
sr.Metadata["workdir_cleanup_requested"] = workRequested
|
||||||
|
|
||||||
eligible, reason := archiveCleanupEligible(env.Config, sr)
|
eligible, reason := publishCleanupEligible(env.Config, sr)
|
||||||
if !eligible {
|
if !eligible {
|
||||||
sr.Metadata["cleanup_skipped"] = true
|
sr.Metadata["cleanup_skipped"] = true
|
||||||
sr.Metadata["cleanup_skipped_reason"] = reason
|
sr.Metadata["cleanup_skipped_reason"] = reason
|
||||||
@@ -63,9 +63,9 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
|||||||
}
|
}
|
||||||
|
|
||||||
if spoolRequested {
|
if spoolRequested {
|
||||||
if err := removeRunScopedDir(strings.TrimSpace(env.Config.Pipeline.Spool.Root), spoolDir, "pipeline.spool.delete_audio_after_archive"); err != nil {
|
if err := removeRunScopedDir(strings.TrimSpace(env.Config.Pipeline.Spool.Root), spoolDir, "pipeline.spool.delete_audio_after_publish"); err != nil {
|
||||||
sr.Metadata["cleanup_failed"] = true
|
sr.Metadata["cleanup_failed"] = true
|
||||||
sr.Metadata["cleanup_failed_policy"] = "pipeline.spool.delete_audio_after_archive"
|
sr.Metadata["cleanup_failed_policy"] = "pipeline.spool.delete_audio_after_publish"
|
||||||
sr.Metadata["cleanup_failed_path"] = spoolDir
|
sr.Metadata["cleanup_failed_path"] = spoolDir
|
||||||
_ = env.ManifestStore.Save(ctx, manifestPath, m)
|
_ = env.ManifestStore.Save(ctx, manifestPath, m)
|
||||||
return err
|
return err
|
||||||
@@ -82,9 +82,9 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := removeRunScopedDir(strings.TrimSpace(env.Config.Pipeline.Workspace.Root), workDir, "pipeline.workspace.cleanup_after_archive"); err != nil {
|
if err := removeRunScopedDir(strings.TrimSpace(env.Config.Pipeline.Workspace.Root), workDir, "pipeline.workspace.cleanup_after_publish"); err != nil {
|
||||||
sr.Metadata["cleanup_failed"] = true
|
sr.Metadata["cleanup_failed"] = true
|
||||||
sr.Metadata["cleanup_failed_policy"] = "pipeline.workspace.cleanup_after_archive"
|
sr.Metadata["cleanup_failed_policy"] = "pipeline.workspace.cleanup_after_publish"
|
||||||
sr.Metadata["cleanup_failed_path"] = workDir
|
sr.Metadata["cleanup_failed_path"] = workDir
|
||||||
_ = env.ManifestStore.Save(ctx, manifestPath, m)
|
_ = env.ManifestStore.Save(ctx, manifestPath, m)
|
||||||
return err
|
return err
|
||||||
@@ -96,59 +96,59 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveStageRecordForCleanup(m *manifest.Manifest, executed []string) *manifest.StageRecord {
|
func publishStageRecordForCleanup(m *manifest.Manifest, executed []string) *manifest.StageRecord {
|
||||||
if m == nil {
|
if m == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
archiveRan := false
|
publishRan := false
|
||||||
for _, name := range executed {
|
for _, name := range executed {
|
||||||
if name == "archive" {
|
if name == "publish" {
|
||||||
archiveRan = true
|
publishRan = true
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !archiveRan {
|
if !publishRan {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
sr := m.Stages["archive"]
|
sr := m.Stages["publish"]
|
||||||
if sr == nil || sr.Status != manifest.StatusSucceeded {
|
if sr == nil || sr.Status != manifest.StatusSucceeded {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return sr
|
return sr
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveCleanupEligible(cfg *config.Config, sr *manifest.StageRecord) (bool, string) {
|
func publishCleanupEligible(cfg *config.Config, sr *manifest.StageRecord) (bool, string) {
|
||||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Archive == nil {
|
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Publish == nil {
|
||||||
return false, "archive configuration is missing"
|
return false, "publish configuration is missing"
|
||||||
}
|
}
|
||||||
enabled := true
|
enabled := true
|
||||||
if cfg.Pipeline.Archive.Enabled != nil {
|
if cfg.Pipeline.Publish.Enabled != nil {
|
||||||
enabled = *cfg.Pipeline.Archive.Enabled
|
enabled = *cfg.Pipeline.Publish.Enabled
|
||||||
}
|
}
|
||||||
if !enabled {
|
if !enabled {
|
||||||
return false, "archive.enabled is false"
|
return false, "publish.enabled is false"
|
||||||
}
|
}
|
||||||
uploadRun := true
|
uploadRun := true
|
||||||
if cfg.Pipeline.Archive.UploadRun != nil {
|
if cfg.Pipeline.Publish.UploadRun != nil {
|
||||||
uploadRun = *cfg.Pipeline.Archive.UploadRun
|
uploadRun = *cfg.Pipeline.Publish.UploadRun
|
||||||
}
|
}
|
||||||
if !uploadRun {
|
if !uploadRun {
|
||||||
return false, "archive.upload_run is false"
|
return false, "publish.upload_run is false"
|
||||||
}
|
}
|
||||||
if sr == nil || sr.Metadata == nil {
|
if sr == nil || sr.Metadata == nil {
|
||||||
return false, "archive metadata is missing"
|
return false, "publish metadata is missing"
|
||||||
}
|
}
|
||||||
if skipped, _ := sr.Metadata["skipped"].(bool); skipped {
|
if skipped, _ := sr.Metadata["skipped"].(bool); skipped {
|
||||||
return false, "archive stage was skipped"
|
return false, "publish stage was skipped"
|
||||||
}
|
}
|
||||||
if uploaded, _ := sr.Metadata["uploaded"].(bool); !uploaded {
|
if uploaded, _ := sr.Metadata["uploaded"].(bool); !uploaded {
|
||||||
return false, "archive did not upload run record"
|
return false, "publish did not upload run record"
|
||||||
}
|
}
|
||||||
if pointer, _ := sr.Metadata["current_pointer_written"].(bool); !pointer {
|
if pointer, _ := sr.Metadata["current_pointer_written"].(bool); !pointer {
|
||||||
return false, "archive did not write current pointer"
|
return false, "publish did not write current pointer"
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(asString(sr.Metadata["current_run_id_key"])) == "" {
|
if strings.TrimSpace(asString(sr.Metadata["current_run_id_key"])) == "" {
|
||||||
return false, "archive current run pointer key is missing"
|
return false, "publish current run pointer key is missing"
|
||||||
}
|
}
|
||||||
return true, ""
|
return true, ""
|
||||||
}
|
}
|
||||||
@@ -174,49 +174,7 @@ func removeRunScopedDir(root, target, policy string) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func validateScopedDir(root, target, policy string) (scopedDir, error) {
|
func validateScopedDir(root, target, policy string) (scopedDir, error) {
|
||||||
cleanRoot := strings.TrimSpace(root)
|
return validateScopedTarget(root, target, policy, true)
|
||||||
cleanTarget := strings.TrimSpace(target)
|
|
||||||
if cleanRoot == "" {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
|
||||||
}
|
|
||||||
if cleanTarget == "" {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target path is required", policy)
|
|
||||||
}
|
|
||||||
|
|
||||||
rootAbs, err := filepath.Abs(cleanRoot)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
|
||||||
}
|
|
||||||
targetAbs, err := filepath.Abs(cleanTarget)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve target %q: %w", policy, cleanTarget, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
rel, err := filepath.Rel(rootAbs, targetAbs)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: relative path from %q to %q: %w", policy, rootAbs, targetAbs, err)
|
|
||||||
}
|
|
||||||
if rel == "." {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete root directory %q", policy, rootAbs)
|
|
||||||
}
|
|
||||||
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete path outside root: root=%q target=%q", policy, rootAbs, targetAbs)
|
|
||||||
}
|
|
||||||
|
|
||||||
info, err := os.Lstat(targetAbs)
|
|
||||||
if err != nil {
|
|
||||||
if os.IsNotExist(err) {
|
|
||||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: false}, nil
|
|
||||||
}
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: stat target %q: %w", policy, targetAbs, err)
|
|
||||||
}
|
|
||||||
if info.Mode()&os.ModeSymlink != 0 {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, targetAbs)
|
|
||||||
}
|
|
||||||
if !info.IsDir() {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is not a directory", policy, targetAbs)
|
|
||||||
}
|
|
||||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: true}, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func asString(v any) string {
|
func asString(v any) string {
|
||||||
@@ -16,15 +16,15 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||||
)
|
)
|
||||||
|
|
||||||
type archiveSuccessStage struct {
|
type publishSuccessStage struct {
|
||||||
metadata map[string]any
|
metadata map[string]any
|
||||||
}
|
}
|
||||||
|
|
||||||
func (archiveSuccessStage) Name() string { return "archive" }
|
func (publishSuccessStage) Name() string { return "publish" }
|
||||||
func (archiveSuccessStage) Declares() stage.IODecl { return stage.IODecl{} }
|
func (publishSuccessStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||||
func (s archiveSuccessStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
func (s publishSuccessStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
||||||
md := map[string]any{
|
md := map[string]any{
|
||||||
"stage": "archive",
|
"stage": "publish",
|
||||||
"uploaded": true,
|
"uploaded": true,
|
||||||
"current_pointer_written": true,
|
"current_pointer_written": true,
|
||||||
"current_run_id_key": "dnd/campaigns/sample-campaign/sessions/2026-05-03/current/run_id.txt",
|
"current_run_id_key": "dnd/campaigns/sample-campaign/sessions/2026-05-03/current/run_id.txt",
|
||||||
@@ -43,12 +43,12 @@ func (notifyFailStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest
|
|||||||
return nil, errors.New("notify failed")
|
return nil, errors.New("notify failed")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupDisabledKeepsLocalDirs(t *testing.T) {
|
func TestPostPublishCleanupDisabledKeepsLocalDirs(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = false
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = false
|
||||||
cfg.Pipeline.Workspace.CleanupAfterArchive = false
|
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -57,12 +57,12 @@ func TestPostArchiveCleanupDisabledKeepsLocalDirs(t *testing.T) {
|
|||||||
assertExists(t, seed.localSourceAudio)
|
assertExists(t, seed.localSourceAudio)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupSpoolOnly(t *testing.T) {
|
func TestPostPublishCleanupSpoolOnly(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterArchive = false
|
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -71,12 +71,12 @@ func TestPostArchiveCleanupSpoolOnly(t *testing.T) {
|
|||||||
assertExists(t, seed.localSourceAudio)
|
assertExists(t, seed.localSourceAudio)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupWorkdirOnly(t *testing.T) {
|
func TestPostPublishCleanupWorkdirOnly(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = false
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = false
|
||||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -87,12 +87,12 @@ func TestPostArchiveCleanupWorkdirOnly(t *testing.T) {
|
|||||||
assertExists(t, seed.spoolAudioDir)
|
assertExists(t, seed.spoolAudioDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupBothPolicies(t *testing.T) {
|
func TestPostPublishCleanupBothPolicies(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -102,26 +102,26 @@ func TestPostArchiveCleanupBothPolicies(t *testing.T) {
|
|||||||
assertExists(t, seed.previousCachePath)
|
assertExists(t, seed.previousCachePath)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenArchiveFails(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenPublishFails(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
_, err := executeStages(context.Background(), cfg, []stage.Stage{failingStage{name: "archive", err: errors.New("archive failed")}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
_, err := executeStages(context.Background(), cfg, []stage.Stage{failingStage{name: "publish", err: errors.New("publish failed")}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||||
if err == nil || !strings.Contains(err.Error(), "stage \"archive\" failed") {
|
if err == nil || !strings.Contains(err.Error(), "stage \"publish\" failed") {
|
||||||
t.Fatalf("executeStages() error = %v, want archive failure", err)
|
t.Fatalf("executeStages() error = %v, want publish failure", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
assertExists(t, seed.spoolAudioDir)
|
assertExists(t, seed.spoolAudioDir)
|
||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenArchiveSkipped(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenPublishSkipped(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{metadata: map[string]any{"skipped": true}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{metadata: map[string]any{"skipped": true}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -129,12 +129,12 @@ func TestPostArchiveCleanupNotRunWhenArchiveSkipped(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenCurrentPointerMissing(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenCurrentPointerMissing(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{metadata: map[string]any{"current_pointer_written": false}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{metadata: map[string]any{"current_pointer_written": false}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -142,13 +142,13 @@ func TestPostArchiveCleanupNotRunWhenCurrentPointerMissing(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenArchiveUploadDisabled(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenPublishUploadDisabled(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
cfg.Pipeline.Archive.UploadRun = boolPtr(false)
|
cfg.Pipeline.Publish.UploadRun = boolPtr(false)
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -156,12 +156,12 @@ func TestPostArchiveCleanupNotRunWhenArchiveUploadDisabled(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupWaitsUntilAllStagesSucceed(t *testing.T) {
|
func TestPostPublishCleanupWaitsUntilAllStagesSucceed(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
_, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}, notifyFailStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
_, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}, notifyFailStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||||
if err == nil || !strings.Contains(err.Error(), "stage \"notify\" failed") {
|
if err == nil || !strings.Contains(err.Error(), "stage \"notify\" failed") {
|
||||||
t.Fatalf("executeStages() error = %v, want notify failure", err)
|
t.Fatalf("executeStages() error = %v, want notify failure", err)
|
||||||
}
|
}
|
||||||
@@ -170,10 +170,10 @@ func TestPostArchiveCleanupWaitsUntilAllStagesSucceed(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupFailsOnUnsafePath(t *testing.T) {
|
func TestPostPublishCleanupFailsOnUnsafePath(t *testing.T) {
|
||||||
cfg, _ := cleanupFixtureConfig(t)
|
cfg, _ := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterArchive = false
|
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
||||||
|
|
||||||
manifestPath := manifestPathFor(cfg)
|
manifestPath := manifestPathFor(cfg)
|
||||||
store := &manifest.LocalStore{}
|
store := &manifest.LocalStore{}
|
||||||
@@ -186,27 +186,27 @@ func TestPostArchiveCleanupFailsOnUnsafePath(t *testing.T) {
|
|||||||
t.Fatalf("Save() error = %v", err)
|
t.Fatalf("Save() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||||
if err == nil || !strings.Contains(err.Error(), "refusing to delete path outside root") {
|
if err == nil || !strings.Contains(err.Error(), "refusing to delete path outside root") {
|
||||||
t.Fatalf("executeStages() error = %v, want safe-path failure", err)
|
t.Fatalf("executeStages() error = %v, want safe-path failure", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenOutputIsMissing(t *testing.T) {
|
||||||
cfg, seed, runID := archiveStageCleanupFixture(t)
|
cfg, seed, runID := publishStageCleanupFixture(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
cfg.Pipeline.Archive.PromoteArtifacts = []config.ArchivePromotionRule{
|
cfg.Pipeline.Publish.Outputs = []config.PublishOutputRule{
|
||||||
{Source: "narratio.transcript.base", Dest: "transcripts/base.json", Required: boolPtr(true)},
|
{Source: "narratio.transcript.base", Dest: "transcripts/base.json", Required: boolPtr(true)},
|
||||||
}
|
}
|
||||||
|
|
||||||
archiveStageImpl, err := stage.Select("archive")
|
publishStageImpl, err := stage.Select("publish")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Select(archive) error = %v", err)
|
t.Fatalf("Select(publish) error = %v", err)
|
||||||
}
|
}
|
||||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveStageImpl}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishStageImpl}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||||
if err == nil || !strings.Contains(err.Error(), "required promotion source unavailable") {
|
if err == nil || !strings.Contains(err.Error(), "required output source unavailable") {
|
||||||
t.Fatalf("executeStages() error = %v, want promotion-missing failure", err)
|
t.Fatalf("executeStages() error = %v, want required output source unavailable failure", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
assertExists(t, seed.spoolAudioDir)
|
assertExists(t, seed.spoolAudioDir)
|
||||||
@@ -215,17 +215,17 @@ func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
|
|||||||
assertExists(t, artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
|
assertExists(t, artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
||||||
cfg, seed, _ := archiveStageCleanupFixture(t)
|
cfg, seed, _ := publishStageCleanupFixture(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
failKey := seed.sessionPrefix + "current/manifest.json"
|
failKey := seed.sessionPrefix + "current/manifest.json"
|
||||||
|
|
||||||
archiveStageImpl, err := stage.Select("archive")
|
publishStageImpl, err := stage.Select("publish")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Select(archive) error = %v", err)
|
t.Fatalf("Select(publish) error = %v", err)
|
||||||
}
|
}
|
||||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveStageImpl}, RunOptions{
|
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishStageImpl}, RunOptions{
|
||||||
Env: &Env{ObjectStore: &failKeyStore{delegate: &storage.FakeBackend{}, failKey: failKey}},
|
Env: &Env{ObjectStore: &failKeyStore{delegate: &storage.FakeBackend{}, failKey: failKey}},
|
||||||
})
|
})
|
||||||
if err == nil || !strings.Contains(err.Error(), "current manifest") {
|
if err == nil || !strings.Contains(err.Error(), "current manifest") {
|
||||||
@@ -236,17 +236,17 @@ func TestPostArchiveCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenCurrentPointerUploadFails(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenCurrentPointerUploadFails(t *testing.T) {
|
||||||
cfg, seed, _ := archiveStageCleanupFixture(t)
|
cfg, seed, _ := publishStageCleanupFixture(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
failKey := seed.sessionPrefix + "current/run_id.txt"
|
failKey := seed.sessionPrefix + "current/run_id.txt"
|
||||||
|
|
||||||
archiveStageImpl, err := stage.Select("archive")
|
publishStageImpl, err := stage.Select("publish")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Select(archive) error = %v", err)
|
t.Fatalf("Select(publish) error = %v", err)
|
||||||
}
|
}
|
||||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveStageImpl}, RunOptions{
|
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishStageImpl}, RunOptions{
|
||||||
Env: &Env{ObjectStore: &failKeyStore{delegate: &storage.FakeBackend{}, failKey: failKey}},
|
Env: &Env{ObjectStore: &failKeyStore{delegate: &storage.FakeBackend{}, failKey: failKey}},
|
||||||
})
|
})
|
||||||
if err == nil || !strings.Contains(err.Error(), "current run pointer") {
|
if err == nil || !strings.Contains(err.Error(), "current run pointer") {
|
||||||
@@ -270,7 +270,7 @@ func cleanupFixtureConfig(t *testing.T) (*config.Config, cleanupSeed) {
|
|||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
cfg := testConfig(t)
|
cfg := testConfig(t)
|
||||||
cfg.Pipeline.Archive = &config.ArchiveConfig{Enabled: boolPtr(true), UploadRun: boolPtr(true)}
|
cfg.Pipeline.Publish = &config.PublishConfig{Enabled: boolPtr(true), UploadRun: boolPtr(true)}
|
||||||
cfg.Pipeline.Spool.Root = filepath.Join(t.TempDir(), "spool")
|
cfg.Pipeline.Spool.Root = filepath.Join(t.TempDir(), "spool")
|
||||||
|
|
||||||
runID := "20260516T010203Z-1a2b3c4d"
|
runID := "20260516T010203Z-1a2b3c4d"
|
||||||
@@ -320,7 +320,7 @@ func cleanupFixtureConfig(t *testing.T) (*config.Config, cleanupSeed) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, string) {
|
func publishStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
@@ -329,10 +329,10 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
|||||||
Bucket: "my-dnd-archive",
|
Bucket: "my-dnd-archive",
|
||||||
RootPrefix: "dnd",
|
RootPrefix: "dnd",
|
||||||
}
|
}
|
||||||
cfg.Pipeline.Archive = &config.ArchiveConfig{
|
cfg.Pipeline.Publish = &config.PublishConfig{
|
||||||
Enabled: boolPtr(true),
|
Enabled: boolPtr(true),
|
||||||
UploadRun: boolPtr(true),
|
UploadRun: boolPtr(true),
|
||||||
PromoteArtifacts: []config.ArchivePromotionRule{
|
Outputs: []config.PublishOutputRule{
|
||||||
{Source: "narratio.transcript.final_trimmed", Dest: "transcripts/final.trimmed.json", Required: boolPtr(true)},
|
{Source: "narratio.transcript.final_trimmed", Dest: "transcripts/final.trimmed.json", Required: boolPtr(true)},
|
||||||
{Source: "narratio.artifact.session_recap", Dest: "artifacts/session_recap.md", Required: boolPtr(true)},
|
{Source: "narratio.artifact.session_recap", Dest: "artifacts/session_recap.md", Required: boolPtr(true)},
|
||||||
},
|
},
|
||||||
@@ -344,7 +344,7 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
|||||||
},
|
},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
writeArchiveFixtureRunFiles(
|
writePublishFixtureRunFiles(
|
||||||
t,
|
t,
|
||||||
seed.runWorkDir,
|
seed.runWorkDir,
|
||||||
artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID),
|
artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID),
|
||||||
@@ -355,7 +355,7 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Load() error = %v", err)
|
t.Fatalf("Load() error = %v", err)
|
||||||
}
|
}
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze"} {
|
||||||
seedManifest.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
seedManifest.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
||||||
}
|
}
|
||||||
seedManifest.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
|
seedManifest.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
@@ -367,7 +367,7 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
|||||||
return cfg, seed, runID
|
return cfg, seed, runID
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir, sessionRoot string) {
|
func writePublishFixtureRunFiles(t *testing.T, runWorkDir, sessionRoot string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
mustWriteFile(t, filepath.Join(runWorkDir, "prepare", "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
mustWriteFile(t, filepath.Join(runWorkDir, "prepare", "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
||||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
|
mustWriteFile(t, filepath.Join(runWorkDir, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
|
||||||
@@ -13,9 +13,9 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
type effectiveLocks struct {
|
type effectiveLocks struct {
|
||||||
Static []config.ArchiveLockRule
|
Static []config.PublishLockRule
|
||||||
Remote []config.ArchiveLockRule
|
Remote []config.PublishLockRule
|
||||||
All []config.ArchiveLockRule
|
All []config.PublishLockRule
|
||||||
Key string
|
Key string
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -34,7 +34,7 @@ func remoteLocksKey(cfg *config.Config) (string, error) {
|
|||||||
return artifacts.S3SessionLocksKey(sessionPrefix), nil
|
return artifacts.S3SessionLocksKey(sessionPrefix), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func loadRemoteLockStore(ctx context.Context, cfg *config.Config, store storage.ObjectStore) (*config.ArchiveLockStore, string, error) {
|
func loadRemoteLockStore(ctx context.Context, cfg *config.Config, store storage.ObjectStore) (*config.PublishLockStore, string, error) {
|
||||||
key, err := remoteLocksKey(cfg)
|
key, err := remoteLocksKey(cfg)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, "", err
|
return nil, "", err
|
||||||
@@ -44,9 +44,9 @@ func loadRemoteLockStore(ctx context.Context, cfg *config.Config, store storage.
|
|||||||
return nil, key, fmt.Errorf("check remote locks %q: %w", key, err)
|
return nil, key, fmt.Errorf("check remote locks %q: %w", key, err)
|
||||||
}
|
}
|
||||||
if !exists {
|
if !exists {
|
||||||
return &config.ArchiveLockStore{}, key, nil
|
return &config.PublishLockStore{}, key, nil
|
||||||
}
|
}
|
||||||
tmp, err := downloadObjectToTemp(ctx, store, key, "narratio-locks-*.yml")
|
tmp, err := storage.DownloadObjectToTemp(ctx, store, key, "narratio-locks-*.yml")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, key, fmt.Errorf("download remote locks %q: %w", key, err)
|
return nil, key, fmt.Errorf("download remote locks %q: %w", key, err)
|
||||||
}
|
}
|
||||||
@@ -55,7 +55,7 @@ func loadRemoteLockStore(ctx context.Context, cfg *config.Config, store storage.
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, key, fmt.Errorf("read remote locks %q: %w", key, err)
|
return nil, key, fmt.Errorf("read remote locks %q: %w", key, err)
|
||||||
}
|
}
|
||||||
lockStore, err := config.LoadArchiveLockStoreBytes("s3://"+s3BucketName(cfg.Pipeline)+"/"+key, data, cfg.Pipeline.Scriptorium)
|
lockStore, err := config.LoadPublishLockStoreBytes("s3://"+s3BucketName(cfg.Pipeline)+"/"+key, data, cfg.Pipeline.Scriptorium)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, key, err
|
return nil, key, err
|
||||||
}
|
}
|
||||||
@@ -63,45 +63,45 @@ func loadRemoteLockStore(ctx context.Context, cfg *config.Config, store storage.
|
|||||||
}
|
}
|
||||||
|
|
||||||
func loadEffectiveLocks(ctx context.Context, cfg *config.Config, store storage.ObjectStore) (*effectiveLocks, error) {
|
func loadEffectiveLocks(ctx context.Context, cfg *config.Config, store storage.ObjectStore) (*effectiveLocks, error) {
|
||||||
staticLocks := staticArchiveLocks(cfg)
|
staticLocks := staticPublishLocks(cfg)
|
||||||
if store == nil {
|
if store == nil {
|
||||||
return &effectiveLocks{
|
return &effectiveLocks{
|
||||||
Static: staticLocks,
|
Static: staticLocks,
|
||||||
All: append([]config.ArchiveLockRule(nil), staticLocks...),
|
All: append([]config.PublishLockRule(nil), staticLocks...),
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
lockStore, key, err := loadRemoteLockStore(ctx, cfg, store)
|
lockStore, key, err := loadRemoteLockStore(ctx, cfg, store)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
remoteLocks := append([]config.ArchiveLockRule(nil), lockStore.Locks...)
|
remoteLocks := append([]config.PublishLockRule(nil), lockStore.Locks...)
|
||||||
return &effectiveLocks{
|
return &effectiveLocks{
|
||||||
Static: staticLocks,
|
Static: staticLocks,
|
||||||
Remote: remoteLocks,
|
Remote: remoteLocks,
|
||||||
All: config.MergeArchiveLockRules(staticLocks, remoteLocks),
|
All: config.MergePublishLockRules(staticLocks, remoteLocks),
|
||||||
Key: key,
|
Key: key,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func staticArchiveLocks(cfg *config.Config) []config.ArchiveLockRule {
|
func staticPublishLocks(cfg *config.Config) []config.PublishLockRule {
|
||||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Archive == nil {
|
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Publish == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return append([]config.ArchiveLockRule(nil), cfg.Pipeline.Archive.Locks...)
|
return append([]config.PublishLockRule(nil), cfg.Pipeline.Publish.Locks...)
|
||||||
}
|
}
|
||||||
|
|
||||||
func applyEffectiveLocks(cfg *config.Config, locks []config.ArchiveLockRule) {
|
func applyEffectiveLocks(cfg *config.Config, locks []config.PublishLockRule) {
|
||||||
if cfg == nil || cfg.Pipeline == nil {
|
if cfg == nil || cfg.Pipeline == nil {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if cfg.Pipeline.Archive == nil {
|
if cfg.Pipeline.Publish == nil {
|
||||||
cfg.Pipeline.Archive = &config.ArchiveConfig{}
|
cfg.Pipeline.Publish = &config.PublishConfig{}
|
||||||
}
|
}
|
||||||
cfg.Pipeline.Archive.Locks = append([]config.ArchiveLockRule(nil), locks...)
|
cfg.Pipeline.Publish.Locks = append([]config.PublishLockRule(nil), locks...)
|
||||||
}
|
}
|
||||||
|
|
||||||
func uploadRemoteLockStore(ctx context.Context, store storage.ObjectStore, key string, lockStore *config.ArchiveLockStore) error {
|
func uploadRemoteLockStore(ctx context.Context, store storage.ObjectStore, key string, lockStore *config.PublishLockStore) error {
|
||||||
data, err := config.MarshalArchiveLockStore(lockStore)
|
data, err := config.MarshalPublishLockStore(lockStore)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -124,8 +124,8 @@ func uploadRemoteLockStore(ctx context.Context, store storage.ObjectStore, key s
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func lockSourceSet(locks []config.ArchiveLockRule) map[string]config.ArchiveLockRule {
|
func lockSourceSet(locks []config.PublishLockRule) map[string]config.PublishLockRule {
|
||||||
out := make(map[string]config.ArchiveLockRule, len(locks))
|
out := make(map[string]config.PublishLockRule, len(locks))
|
||||||
for _, lock := range locks {
|
for _, lock := range locks {
|
||||||
source := strings.TrimSpace(lock.Source)
|
source := strings.TrimSpace(lock.Source)
|
||||||
if source == "" {
|
if source == "" {
|
||||||
|
|||||||
@@ -27,23 +27,14 @@ func Restore(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
fs := flag.NewFlagSet("restore", flag.ContinueOnError)
|
fs := flag.NewFlagSet("restore", flag.ContinueOnError)
|
||||||
fs.SetOutput(out)
|
fs.SetOutput(out)
|
||||||
|
|
||||||
var pipelinePath string
|
var flags commonConfigFlags
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var dryRun bool
|
var dryRun bool
|
||||||
var force bool
|
var force bool
|
||||||
var includeAudio bool
|
var includeAudio bool
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.BoolVar(&dryRun, "dry-run", false, "plan restore actions without writing local files")
|
fs.BoolVar(&dryRun, "dry-run", false, "plan restore actions without writing local files")
|
||||||
fs.BoolVar(&force, "force", false, "overwrite local conflicts with remote state")
|
fs.BoolVar(&force, "force", false, "overwrite local conflicts with remote state")
|
||||||
fs.BoolVar(&includeAudio, "include-audio", false, "include archived session-level audio objects")
|
fs.BoolVar(&includeAudio, "include-audio", false, "include remote session-level audio objects")
|
||||||
fs.Usage = func() {
|
fs.Usage = func() {
|
||||||
_, _ = fmt.Fprintln(out, "Usage: narratio session restore <session_id> [--config <path>] [--campaign <id>] [--campaign-file <path>] [--session <path>] [--previous-session-id <value>] [--dry-run] [--force] [--include-audio]")
|
_, _ = fmt.Fprintln(out, "Usage: narratio session restore <session_id> [--config <path>] [--campaign <id>] [--campaign-file <path>] [--session <path>] [--previous-session-id <value>] [--dry-run] [--force] [--include-audio]")
|
||||||
_, _ = fmt.Fprintln(out)
|
_, _ = fmt.Fprintln(out)
|
||||||
@@ -57,25 +48,13 @@ func Restore(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
}
|
}
|
||||||
return fmt.Errorf("restore: invalid flags: %w", err)
|
return fmt.Errorf("restore: invalid flags: %w", err)
|
||||||
}
|
}
|
||||||
if positionalSessionID == "" {
|
if err := resolveParsedSessionID("restore", positionalSessionID, fs, &flags.sessionID); err != nil {
|
||||||
if err := applyParsedSessionIDArg("restore", fs, &sessionID); err != nil {
|
return err
|
||||||
return err
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("restore: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("restore", positionalSessionID, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
return fmt.Errorf("restore: session_id is required")
|
return fmt.Errorf("restore: session_id is required")
|
||||||
}
|
}
|
||||||
cfg, err := loadCommandConfig(ctx, pipelinePath, campaignPath, campaignFilePath, sessionPath, config.SessionLoadOptions{
|
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||||
SessionID: sessionID,
|
|
||||||
PreviousSessionID: previousSessionID,
|
|
||||||
})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("restore: %w", err)
|
return fmt.Errorf("restore: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ package app
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
@@ -12,7 +11,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
// RemoteCurrentState captures discovered committed remote archive state for one session.
|
// RemoteCurrentState captures discovered committed remote published current state for one session.
|
||||||
type RemoteCurrentState struct {
|
type RemoteCurrentState struct {
|
||||||
Bucket string
|
Bucket string
|
||||||
SessionPrefix string
|
SessionPrefix string
|
||||||
@@ -32,80 +31,24 @@ func discoverRemoteCurrentState(ctx context.Context, cfg *config.Config, store s
|
|||||||
return nil, fmt.Errorf("remote object store is required")
|
return nil, fmt.Errorf("remote object store is required")
|
||||||
}
|
}
|
||||||
|
|
||||||
bucket := artifacts.ResolveArchiveBucket(cfg, nil)
|
bucket := artifacts.ResolvePublishBucket(cfg, nil)
|
||||||
if strings.TrimSpace(bucket) == "" {
|
if strings.TrimSpace(bucket) == "" {
|
||||||
return nil, fmt.Errorf("archive bucket is required")
|
return nil, fmt.Errorf("publish bucket is required")
|
||||||
}
|
}
|
||||||
sessionPrefix, err := artifacts.ResolveArchiveSessionPrefix(cfg, nil)
|
sessionPrefix, err := artifacts.ResolvePublishSessionPrefix(cfg, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("resolve archive session prefix: %w", err)
|
return nil, fmt.Errorf("resolve publish session prefix: %w", err)
|
||||||
}
|
|
||||||
currentManifestKey, currentRunIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
|
||||||
|
|
||||||
exists, err := store.Exists(ctx, currentRunIDKey)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("check remote current run pointer %q: %w", currentRunIDKey, err)
|
|
||||||
}
|
|
||||||
if !exists {
|
|
||||||
return nil, fmt.Errorf("remote current run pointer missing: %q", currentRunIDKey)
|
|
||||||
}
|
|
||||||
|
|
||||||
runIDPath, err := downloadObjectToTemp(ctx, store, currentRunIDKey, "narratio-restore-current-run-id-*.txt")
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("download remote current run pointer %q: %w", currentRunIDKey, err)
|
|
||||||
}
|
|
||||||
defer func() { _ = os.Remove(runIDPath) }()
|
|
||||||
|
|
||||||
runIDData, err := os.ReadFile(runIDPath)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("read downloaded run pointer %q: %w", currentRunIDKey, err)
|
|
||||||
}
|
|
||||||
runID := strings.TrimSpace(string(runIDData))
|
|
||||||
if runID == "" {
|
|
||||||
return nil, fmt.Errorf("remote current run pointer %q is empty", currentRunIDKey)
|
|
||||||
}
|
|
||||||
|
|
||||||
exists, err = store.Exists(ctx, currentManifestKey)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("check remote current manifest %q: %w", currentManifestKey, err)
|
|
||||||
}
|
|
||||||
if !exists {
|
|
||||||
return nil, fmt.Errorf("remote current manifest missing: %q", currentManifestKey)
|
|
||||||
}
|
|
||||||
|
|
||||||
manifestPath, err := downloadObjectToTemp(ctx, store, currentManifestKey, "narratio-restore-current-manifest-*.json")
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("download remote current manifest %q: %w", currentManifestKey, err)
|
|
||||||
}
|
|
||||||
defer func() { _ = os.Remove(manifestPath) }()
|
|
||||||
|
|
||||||
manifestStore := &manifest.LocalStore{}
|
|
||||||
remoteManifest, err := manifestStore.Load(ctx, manifestPath)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("remote current manifest decode failed: %w", err)
|
|
||||||
}
|
}
|
||||||
|
currentManifestKey, currentRunIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
|
|
||||||
requestedSession := strings.TrimSpace(cfg.Session.SessionID)
|
requestedSession := strings.TrimSpace(cfg.Session.SessionID)
|
||||||
requestedCampaign := strings.TrimSpace(cfg.Session.Campaign)
|
requestedCampaign := strings.TrimSpace(cfg.Session.Campaign)
|
||||||
manifestSession := strings.TrimSpace(remoteManifest.SessionID)
|
current, err := artifacts.LoadCurrentState(ctx, store, sessionPrefix, artifacts.CurrentStateValidation{
|
||||||
manifestCampaign := strings.TrimSpace(remoteManifest.Campaign)
|
ExpectedSessionID: requestedSession,
|
||||||
|
ExpectedCampaign: requestedCampaign,
|
||||||
if manifestSession != requestedSession {
|
})
|
||||||
return nil, fmt.Errorf(
|
if err != nil {
|
||||||
"remote current manifest session_id %q does not match requested session_id %q",
|
return nil, fmt.Errorf("remote %w", err)
|
||||||
manifestSession,
|
|
||||||
requestedSession,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
if manifestCampaign == "" {
|
|
||||||
return nil, fmt.Errorf("remote current manifest campaign is required")
|
|
||||||
}
|
|
||||||
if manifestCampaign != requestedCampaign {
|
|
||||||
return nil, fmt.Errorf(
|
|
||||||
"remote current manifest campaign %q does not match requested campaign %q",
|
|
||||||
manifestCampaign,
|
|
||||||
requestedCampaign,
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return &RemoteCurrentState{
|
return &RemoteCurrentState{
|
||||||
@@ -113,27 +56,9 @@ func discoverRemoteCurrentState(ctx context.Context, cfg *config.Config, store s
|
|||||||
SessionPrefix: sessionPrefix,
|
SessionPrefix: sessionPrefix,
|
||||||
CurrentRunIDKey: currentRunIDKey,
|
CurrentRunIDKey: currentRunIDKey,
|
||||||
CurrentManifestKey: currentManifestKey,
|
CurrentManifestKey: currentManifestKey,
|
||||||
RunID: runID,
|
RunID: current.RunID,
|
||||||
SessionID: manifestSession,
|
SessionID: strings.TrimSpace(current.Manifest.SessionID),
|
||||||
Campaign: manifestCampaign,
|
Campaign: strings.TrimSpace(current.Manifest.Campaign),
|
||||||
Manifest: remoteManifest,
|
Manifest: current.Manifest,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func downloadObjectToTemp(ctx context.Context, store storage.ObjectStore, key, pattern string) (string, error) {
|
|
||||||
tmp, err := os.CreateTemp("", pattern)
|
|
||||||
if err != nil {
|
|
||||||
return "", fmt.Errorf("create temp file: %w", err)
|
|
||||||
}
|
|
||||||
path := tmp.Name()
|
|
||||||
if err := tmp.Close(); err != nil {
|
|
||||||
_ = os.Remove(path)
|
|
||||||
return "", fmt.Errorf("close temp file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := store.Download(ctx, key, path); err != nil {
|
|
||||||
_ = os.Remove(path)
|
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
return path, nil
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -102,7 +102,7 @@ func TestDiscoverRemoteCurrentStateSessionMismatchFails(t *testing.T) {
|
|||||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "wrong-session", cfg.Session.Campaign)})
|
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "wrong-session", cfg.Session.Campaign)})
|
||||||
|
|
||||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||||
if err == nil || !strings.Contains(err.Error(), "does not match requested session_id") {
|
if err == nil || !strings.Contains(err.Error(), "does not match expected session_id") {
|
||||||
t.Fatalf("error = %v, want session mismatch failure", err)
|
t.Fatalf("error = %v, want session mismatch failure", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -116,7 +116,7 @@ func TestDiscoverRemoteCurrentStateCampaignMismatchFails(t *testing.T) {
|
|||||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, "wrong-campaign")})
|
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, "wrong-campaign")})
|
||||||
|
|
||||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||||
if err == nil || !strings.Contains(err.Error(), "does not match requested campaign") {
|
if err == nil || !strings.Contains(err.Error(), "does not match expected campaign") {
|
||||||
t.Fatalf("error = %v, want campaign mismatch failure", err)
|
t.Fatalf("error = %v, want campaign mismatch failure", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -208,7 +208,7 @@ func restoreDiscoveryConfig() *config.Config {
|
|||||||
|
|
||||||
func restoreDiscoveryKeys(cfg *config.Config) (sessionPrefix, manifestKey, runIDKey string) {
|
func restoreDiscoveryKeys(cfg *config.Config) (sessionPrefix, manifestKey, runIDKey string) {
|
||||||
sessionPrefix = artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
sessionPrefix = artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
manifestKey, runIDKey = artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
manifestKey, runIDKey = artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
return sessionPrefix, manifestKey, runIDKey
|
return sessionPrefix, manifestKey, runIDKey
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/audio"
|
"gitea.maximumdirect.net/eric/narratio/internal/audio"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/fileops"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -114,10 +115,7 @@ func executeRestoreDownloadAction(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := os.Chmod(tmpPath, 0o644); err != nil {
|
if err := fileops.InstallDownloadedTempFile(tmpPath, safeLocalPath, 0o644); err != nil {
|
||||||
return fmt.Errorf("set file permissions: %w", err)
|
|
||||||
}
|
|
||||||
if err := os.Rename(tmpPath, safeLocalPath); err != nil {
|
|
||||||
return fmt.Errorf("install file atomically: %w", err)
|
return fmt.Errorf("install file atomically: %w", err)
|
||||||
}
|
}
|
||||||
removeTmp = false
|
removeTmp = false
|
||||||
|
|||||||
@@ -39,7 +39,7 @@ func TestExecuteRestoreNonDryRunRestoresDurableFiles(t *testing.T) {
|
|||||||
if stderr.Len() != 0 {
|
if stderr.Len() != 0 {
|
||||||
t.Fatalf("stderr = %q, want empty", stderr.String())
|
t.Fatalf("stderr = %q, want empty", stderr.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "Restored session archive for sample-campaign/2026-05-03") {
|
if !strings.Contains(stdout.String(), "Restored session state for sample-campaign/2026-05-03") {
|
||||||
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -427,7 +427,7 @@ func seedRestoreCommittedState(t *testing.T, fake *storage.FakeBackend, pipeline
|
|||||||
}
|
}
|
||||||
|
|
||||||
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
|
|
||||||
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
|
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
|
||||||
seedRestoreObject(fake, manifestKey, restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
|
seedRestoreObject(fake, manifestKey, restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
|
||||||
@@ -465,7 +465,7 @@ func seedRestorePreviousCurrent(t *testing.T, fake *storage.FakeBackend, cfg *co
|
|||||||
func seedRestorePreviousCurrentManifestOnly(t *testing.T, fake *storage.FakeBackend, cfg *config.Config) {
|
func seedRestorePreviousCurrentManifestOnly(t *testing.T, fake *storage.FakeBackend, cfg *config.Config) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
previousPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
previousPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
||||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(previousPrefix)
|
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(previousPrefix)
|
||||||
previousRunID := "20260426T010203Z-a1b2c3d4"
|
previousRunID := "20260426T010203Z-a1b2c3d4"
|
||||||
seedRestoreObject(fake, runIDKey, []byte(previousRunID+"\n"))
|
seedRestoreObject(fake, runIDKey, []byte(previousRunID+"\n"))
|
||||||
|
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package app
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
@@ -13,6 +14,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/pathsafe"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/previouscache"
|
"gitea.maximumdirect.net/eric/narratio/internal/previouscache"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -215,19 +217,17 @@ func joinWithinSessionRoot(sessionRoot, relative string) (string, error) {
|
|||||||
if strings.TrimSpace(sessionRoot) == "" {
|
if strings.TrimSpace(sessionRoot) == "" {
|
||||||
return "", fmt.Errorf("session root is required")
|
return "", fmt.Errorf("session root is required")
|
||||||
}
|
}
|
||||||
cleanRel := path.Clean(strings.TrimSpace(relative))
|
joined, err := pathsafe.JoinSlashRelativeUnderRoot(sessionRoot, filepath.ToSlash(strings.TrimSpace(relative)))
|
||||||
if cleanRel == "." || cleanRel == "" {
|
if err != nil {
|
||||||
return "", fmt.Errorf("relative path is required")
|
if errors.Is(err, pathsafe.ErrRelativePathRequired) {
|
||||||
|
return "", fmt.Errorf("relative path is required")
|
||||||
|
}
|
||||||
|
if errors.Is(err, pathsafe.ErrRelativePathEscape) || errors.Is(err, pathsafe.ErrRelativePathAbsolute) {
|
||||||
|
return "", fmt.Errorf("relative path escapes session root")
|
||||||
|
}
|
||||||
|
return "", fmt.Errorf("join relative path under session root: %w", err)
|
||||||
}
|
}
|
||||||
if cleanRel == ".." || strings.HasPrefix(cleanRel, "../") || strings.HasPrefix(cleanRel, "/") {
|
return joined, nil
|
||||||
return "", fmt.Errorf("relative path escapes session root")
|
|
||||||
}
|
|
||||||
abs := filepath.Clean(filepath.Join(sessionRoot, filepath.FromSlash(cleanRel)))
|
|
||||||
root := filepath.Clean(sessionRoot)
|
|
||||||
if abs != root && !strings.HasPrefix(abs, root+string(filepath.Separator)) {
|
|
||||||
return "", fmt.Errorf("resolved local path escapes session root")
|
|
||||||
}
|
|
||||||
return abs, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func buildPreviousCacheRestoreActions(
|
func buildPreviousCacheRestoreActions(
|
||||||
@@ -338,7 +338,7 @@ func classifyRestoreAction(
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return RestoreAction{}, fmt.Errorf("checksum local file: %w", err)
|
return RestoreAction{}, fmt.Errorf("checksum local file: %w", err)
|
||||||
}
|
}
|
||||||
remotePath, err := downloadObjectToTemp(ctx, store, action.RemoteKey, "narratio-restore-plan-remote-*.tmp")
|
remotePath, err := storage.DownloadObjectToTemp(ctx, store, action.RemoteKey, "narratio-restore-plan-remote-*.tmp")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return RestoreAction{}, fmt.Errorf("download remote object: %w", err)
|
return RestoreAction{}, fmt.Errorf("download remote object: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ func TestRestorePlanDefaultScope(t *testing.T) {
|
|||||||
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("# recap\n"))
|
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("# recap\n"))
|
||||||
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
|
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
|
||||||
seedRestoreObject(store, current.SessionPrefix+"runs/20260519T010203Z-a1b2/manifest.json", []byte("{}"))
|
seedRestoreObject(store, current.SessionPrefix+"runs/20260519T010203Z-a1b2/manifest.json", []byte("{}"))
|
||||||
seedRestoreObject(store, current.SessionPrefix+"logs/archive.log", []byte("log"))
|
seedRestoreObject(store, current.SessionPrefix+"logs/publish.log", []byte("log"))
|
||||||
|
|
||||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -325,7 +325,7 @@ func configureRestorePlanPreviousRequirement(cfg *config.Config, required bool)
|
|||||||
func restorePlanCurrentState(t *testing.T, cfg *config.Config) *RemoteCurrentState {
|
func restorePlanCurrentState(t *testing.T, cfg *config.Config) *RemoteCurrentState {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
sessionPrefix := artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
|
sessionPrefix := artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
return &RemoteCurrentState{
|
return &RemoteCurrentState{
|
||||||
Bucket: "test-bucket",
|
Bucket: "test-bucket",
|
||||||
SessionPrefix: sessionPrefix,
|
SessionPrefix: sessionPrefix,
|
||||||
|
|||||||
@@ -201,7 +201,7 @@ func writeRestoreSuccessSummary(out io.Writer, report *RestoreReport) error {
|
|||||||
if report == nil {
|
if report == nil {
|
||||||
return fmt.Errorf("restore report is required")
|
return fmt.Errorf("restore report is required")
|
||||||
}
|
}
|
||||||
if _, err := fmt.Fprintf(out, "Restored session archive for %s/%s\n", report.Campaign, report.SessionID); err != nil {
|
if _, err := fmt.Fprintf(out, "Restored session state for %s/%s\n", report.Campaign, report.SessionID); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if _, err := fmt.Fprintf(out, "Remote run: %s\n", report.RunID); err != nil {
|
if _, err := fmt.Fprintf(out, "Remote run: %s\n", report.RunID); err != nil {
|
||||||
|
|||||||
@@ -369,7 +369,7 @@ func TestExecuteRestoreNonDryRunForceExecutesPlan(t *testing.T) {
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "Restored session archive for sample-campaign/2026-05-03") {
|
if !strings.Contains(stdout.String(), "Restored session state for sample-campaign/2026-05-03") {
|
||||||
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
||||||
}
|
}
|
||||||
if stderr.Len() != 0 {
|
if stderr.Len() != 0 {
|
||||||
|
|||||||
@@ -1,128 +0,0 @@
|
|||||||
package app
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"flag"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"strings"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Resume continues execution from the first non-succeeded stage in the manifest.
|
|
||||||
func Resume(ctx context.Context, args []string, out io.Writer) error {
|
|
||||||
positionalSessionID, args := pullLeadingSessionID(args)
|
|
||||||
fs := flag.NewFlagSet("resume", flag.ContinueOnError)
|
|
||||||
fs.SetOutput(io.Discard)
|
|
||||||
|
|
||||||
var pipelinePath string
|
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var force bool
|
|
||||||
var selectedArtifacts artifactSelectionFlag
|
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.BoolVar(&force, "force", false, "force stage execution")
|
|
||||||
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute and publish (comma-separated or repeatable)")
|
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
|
||||||
return fmt.Errorf("resume: invalid flags: %w", err)
|
|
||||||
}
|
|
||||||
if positionalSessionID == "" {
|
|
||||||
if err := applyParsedSessionIDArg("resume", fs, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("resume: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("resume", positionalSessionID, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
|
||||||
return fmt.Errorf("resume: session_id is required")
|
|
||||||
}
|
|
||||||
cfg, err := loadCommandConfig(ctx, pipelinePath, campaignPath, campaignFilePath, sessionPath, config.SessionLoadOptions{
|
|
||||||
SessionID: sessionID,
|
|
||||||
PreviousSessionID: previousSessionID,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("resume: %w", err)
|
|
||||||
}
|
|
||||||
if err := config.Validate(cfg); err != nil {
|
|
||||||
return fmt.Errorf("resume: %w", err)
|
|
||||||
}
|
|
||||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("resume: invalid --artifacts: %w", err)
|
|
||||||
}
|
|
||||||
if err := validateSelectedArtifacts(cfg, normalizedArtifacts); err != nil {
|
|
||||||
return fmt.Errorf("resume: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
full := BuildFullPlan()
|
|
||||||
selected := full
|
|
||||||
if !force {
|
|
||||||
m, err := loadManifestIfPresent(ctx, cfg)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("resume: %w", err)
|
|
||||||
}
|
|
||||||
if m != nil {
|
|
||||||
start := firstNonSucceededIndex(full, m)
|
|
||||||
if start >= len(full) {
|
|
||||||
_, err := fmt.Fprintf(out, "narratio resume: session %s has no remaining stages\n", cfg.Session.SessionID)
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
selected = full[start:]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
summary, err := executeStagesFn(ctx, cfg, selected, RunOptions{
|
|
||||||
Force: force,
|
|
||||||
SelectedArtifacts: normalizedArtifacts,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("resume: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = fmt.Fprintf(
|
|
||||||
out,
|
|
||||||
"narratio resume: session %s; executed=%d skipped=%d; manifest=%s\n",
|
|
||||||
summary.SessionID,
|
|
||||||
len(summary.Executed),
|
|
||||||
len(summary.Skipped),
|
|
||||||
summary.ManifestPath,
|
|
||||||
)
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
func loadManifestIfPresent(ctx context.Context, cfg *config.Config) (*manifest.Manifest, error) {
|
|
||||||
path := artifacts.SessionManifestPathForCampaign(
|
|
||||||
cfg.Pipeline.Workspace.Root,
|
|
||||||
cfg.Session.Campaign,
|
|
||||||
cfg.Session.SessionID,
|
|
||||||
)
|
|
||||||
exists, err := fileExists(path)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("check manifest %q: %w", path, err)
|
|
||||||
}
|
|
||||||
if !exists {
|
|
||||||
return nil, nil
|
|
||||||
}
|
|
||||||
store := &manifest.LocalStore{}
|
|
||||||
m, err := store.Load(ctx, path)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("load manifest %q: %w", path, err)
|
|
||||||
}
|
|
||||||
return m, nil
|
|
||||||
}
|
|
||||||
@@ -5,55 +5,29 @@ import (
|
|||||||
"flag"
|
"flag"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"strings"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Run executes the pipeline plan and persists manifest state.
|
// Run executes the pipeline plan and persists manifest state.
|
||||||
func Run(ctx context.Context, args []string, out io.Writer) error {
|
func Run(ctx context.Context, args []string, out io.Writer) error {
|
||||||
positionalSessionID, args := pullLeadingSessionID(args)
|
|
||||||
fs := flag.NewFlagSet("run", flag.ContinueOnError)
|
fs := flag.NewFlagSet("run", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
|
|
||||||
var pipelinePath string
|
var flags commonConfigFlags
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var force bool
|
var force bool
|
||||||
var selectedArtifacts artifactSelectionFlag
|
var selectedArtifacts artifactSelectionFlag
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
||||||
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute and publish (comma-separated or repeatable)")
|
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute and publish (comma-separated or repeatable)")
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
if err := parseSessionAwareFlags("run", fs, args, &flags.sessionID); err != nil {
|
||||||
return fmt.Errorf("run: invalid flags: %w", err)
|
return err
|
||||||
}
|
}
|
||||||
if positionalSessionID == "" {
|
if flags.sessionID == "" {
|
||||||
if err := applyParsedSessionIDArg("run", fs, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("run: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("run", positionalSessionID, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
|
||||||
return fmt.Errorf("run: session_id is required")
|
return fmt.Errorf("run: session_id is required")
|
||||||
}
|
}
|
||||||
cfg, err := loadCommandConfig(ctx, pipelinePath, campaignPath, campaignFilePath, sessionPath, config.SessionLoadOptions{
|
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||||
SessionID: sessionID,
|
|
||||||
PreviousSessionID: previousSessionID,
|
|
||||||
})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("run: %w", err)
|
return fmt.Errorf("run: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,8 +1,12 @@
|
|||||||
package app
|
package app
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||||
)
|
)
|
||||||
@@ -43,13 +47,25 @@ func stageSucceeded(m *manifest.Manifest, name string) bool {
|
|||||||
return sr != nil && sr.Status == manifest.StatusSucceeded
|
return sr != nil && sr.Status == manifest.StatusSucceeded
|
||||||
}
|
}
|
||||||
|
|
||||||
func firstNonSucceededIndex(stages []stage.Stage, m *manifest.Manifest) int {
|
func loadManifestIfPresent(ctx context.Context, cfg *config.Config) (*manifest.Manifest, error) {
|
||||||
for i, s := range stages {
|
path := artifacts.SessionManifestPathForCampaign(
|
||||||
if !stageSucceeded(m, s.Name()) {
|
cfg.Pipeline.Workspace.Root,
|
||||||
return i
|
cfg.Session.Campaign,
|
||||||
}
|
cfg.Session.SessionID,
|
||||||
|
)
|
||||||
|
exists, err := fileExists(path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("check manifest %q: %w", path, err)
|
||||||
}
|
}
|
||||||
return len(stages)
|
if !exists {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
store := &manifest.LocalStore{}
|
||||||
|
m, err := store.Load(ctx, path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("load manifest %q: %w", path, err)
|
||||||
|
}
|
||||||
|
return m, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func canonicalStageNames() []string {
|
func canonicalStageNames() []string {
|
||||||
|
|||||||
@@ -8,18 +8,6 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestFirstNonSucceededIndex(t *testing.T) {
|
|
||||||
stages := BuildFullPlan()
|
|
||||||
m := manifest.New("2026-05-03", time.Now().UTC())
|
|
||||||
m.MarkStageSucceeded("prepare", time.Now().UTC(), nil)
|
|
||||||
m.MarkStageSucceeded("transcribe", time.Now().UTC(), nil)
|
|
||||||
|
|
||||||
got := firstNonSucceededIndex(stages, m)
|
|
||||||
if got != 2 {
|
|
||||||
t.Fatalf("firstNonSucceededIndex() = %d, want 2", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestDecideStageActions(t *testing.T) {
|
func TestDecideStageActions(t *testing.T) {
|
||||||
stages := BuildFullPlan()[:2]
|
stages := BuildFullPlan()[:2]
|
||||||
m := manifest.New("2026-05-03", time.Now().UTC())
|
m := manifest.New("2026-05-03", time.Now().UTC())
|
||||||
@@ -44,7 +32,7 @@ func TestDecideStageActions(t *testing.T) {
|
|||||||
|
|
||||||
func TestDownstreamStageNames(t *testing.T) {
|
func TestDownstreamStageNames(t *testing.T) {
|
||||||
got := downstreamStageNames("polish")
|
got := downstreamStageNames("polish")
|
||||||
want := []string{"normalize", "trim", "analyze", "archive", "notify"}
|
want := []string{"normalize", "trim", "render", "analyze", "publish", "notify"}
|
||||||
if !reflect.DeepEqual(got, want) {
|
if !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("downstreamStageNames(polish) = %#v, want %#v", got, want)
|
t.Fatalf("downstreamStageNames(polish) = %#v, want %#v", got, want)
|
||||||
}
|
}
|
||||||
@@ -64,12 +52,13 @@ func TestInvalidateDownstreamSucceededStages(t *testing.T) {
|
|||||||
m.MarkStageSucceeded("polish", now, nil)
|
m.MarkStageSucceeded("polish", now, nil)
|
||||||
m.MarkStageSucceeded("normalize", now, nil)
|
m.MarkStageSucceeded("normalize", now, nil)
|
||||||
m.MarkStageSucceeded("trim", now, nil)
|
m.MarkStageSucceeded("trim", now, nil)
|
||||||
|
m.MarkStageSucceeded("render", now, nil)
|
||||||
m.MarkStageFailed("analyze", now, "analysis failed")
|
m.MarkStageFailed("analyze", now, "analysis failed")
|
||||||
m.MarkStageSucceeded("archive", now, nil)
|
m.MarkStageSucceeded("publish", now, nil)
|
||||||
m.MarkStageSucceeded("notify", now, nil)
|
m.MarkStageSucceeded("notify", now, nil)
|
||||||
|
|
||||||
got := invalidateDownstreamSucceededStages(m, "polish", now.Add(1*time.Second))
|
got := invalidateDownstreamSucceededStages(m, "polish", now.Add(1*time.Second))
|
||||||
want := []string{"normalize", "trim", "archive", "notify"}
|
want := []string{"normalize", "trim", "render", "publish", "notify"}
|
||||||
if !reflect.DeepEqual(got, want) {
|
if !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("invalidateDownstreamSucceededStages() = %#v, want %#v", got, want)
|
t.Fatalf("invalidateDownstreamSucceededStages() = %#v, want %#v", got, want)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -23,19 +23,10 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
fs := flag.NewFlagSet("run-stage", flag.ContinueOnError)
|
fs := flag.NewFlagSet("run-stage", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
|
|
||||||
var pipelinePath string
|
var flags commonConfigFlags
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var force bool
|
var force bool
|
||||||
var selectedArtifacts artifactSelectionFlag
|
var selectedArtifacts artifactSelectionFlag
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
||||||
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute or publish (comma-separated or repeatable)")
|
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute or publish (comma-separated or repeatable)")
|
||||||
|
|
||||||
@@ -47,35 +38,40 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
case 2:
|
case 2:
|
||||||
stageName = strings.TrimSpace(fs.Arg(0))
|
stageName = strings.TrimSpace(fs.Arg(0))
|
||||||
positionalSessionID = strings.TrimSpace(fs.Arg(1))
|
positionalSessionID = strings.TrimSpace(fs.Arg(1))
|
||||||
|
case 1:
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("run-stage: expected stage name and session_id")
|
||||||
|
}
|
||||||
|
stageName = strings.TrimSpace(fs.Arg(0))
|
||||||
default:
|
default:
|
||||||
return fmt.Errorf("run-stage: expected stage name and session_id")
|
return fmt.Errorf("run-stage: expected stage name and session_id")
|
||||||
}
|
}
|
||||||
} else if fs.NArg() != 0 {
|
} else if fs.NArg() != 0 {
|
||||||
return fmt.Errorf("run-stage: unexpected positional arguments")
|
return fmt.Errorf("run-stage: unexpected positional arguments")
|
||||||
}
|
}
|
||||||
if err := applyPositionalSessionID("run-stage", positionalSessionID, &sessionID); err != nil {
|
if err := applyPositionalSessionID("run-stage", positionalSessionID, &flags.sessionID); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
return fmt.Errorf("run-stage: session_id is required")
|
return fmt.Errorf("run-stage: session_id is required")
|
||||||
}
|
}
|
||||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("run-stage: invalid --artifacts: %w", err)
|
return fmt.Errorf("run-stage: invalid --artifacts: %w", err)
|
||||||
}
|
}
|
||||||
if len(normalizedArtifacts) > 0 && stageName != "analyze" && stageName != "archive" {
|
if len(normalizedArtifacts) > 0 && stageName != "analyze" && stageName != "publish" {
|
||||||
return fmt.Errorf("run-stage: --artifacts is only supported for stages \"analyze\" and \"archive\"")
|
return fmt.Errorf("run-stage: --artifacts is only supported for stages \"analyze\" and \"publish\"")
|
||||||
}
|
}
|
||||||
|
|
||||||
summary, err := runSingleStageCommand(ctx, singleStageCommand{
|
summary, err := runSingleStageCommand(ctx, singleStageCommand{
|
||||||
CommandName: "run-stage",
|
CommandName: "run-stage",
|
||||||
StageName: stageName,
|
StageName: stageName,
|
||||||
PipelinePath: pipelinePath,
|
PipelinePath: flags.pipelinePath,
|
||||||
CampaignPath: campaignPath,
|
CampaignPath: flags.campaignPath,
|
||||||
CampaignFilePath: campaignFilePath,
|
CampaignFilePath: flags.campaignFilePath,
|
||||||
SessionPath: sessionPath,
|
SessionPath: flags.sessionPath,
|
||||||
SessionID: sessionID,
|
SessionID: flags.sessionID,
|
||||||
PreviousSessionID: previousSessionID,
|
PreviousSessionID: flags.previousSessionID,
|
||||||
Force: force,
|
Force: force,
|
||||||
SelectedArtifacts: normalizedArtifacts,
|
SelectedArtifacts: normalizedArtifacts,
|
||||||
})
|
})
|
||||||
@@ -97,40 +93,18 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
|
|
||||||
// Analyze force-runs the analyze stage.
|
// Analyze force-runs the analyze stage.
|
||||||
func Analyze(ctx context.Context, args []string, out io.Writer) error {
|
func Analyze(ctx context.Context, args []string, out io.Writer) error {
|
||||||
positionalSessionID, args := pullLeadingSessionID(args)
|
|
||||||
fs := flag.NewFlagSet("analyze", flag.ContinueOnError)
|
fs := flag.NewFlagSet("analyze", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
|
|
||||||
var pipelinePath string
|
var flags commonConfigFlags
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var selectedArtifacts artifactSelectionFlag
|
var selectedArtifacts artifactSelectionFlag
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute during analyze (comma-separated or repeatable)")
|
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute during analyze (comma-separated or repeatable)")
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
if err := parseSessionAwareFlags("analyze", fs, args, &flags.sessionID); err != nil {
|
||||||
return fmt.Errorf("analyze: invalid flags: %w", err)
|
return err
|
||||||
}
|
}
|
||||||
if positionalSessionID == "" {
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
if err := applyParsedSessionIDArg("analyze", fs, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("analyze: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("analyze", positionalSessionID, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
|
||||||
return fmt.Errorf("analyze: session_id is required")
|
return fmt.Errorf("analyze: session_id is required")
|
||||||
}
|
}
|
||||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
||||||
@@ -141,12 +115,12 @@ func Analyze(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
summary, err := runSingleStageCommand(ctx, singleStageCommand{
|
summary, err := runSingleStageCommand(ctx, singleStageCommand{
|
||||||
CommandName: "analyze",
|
CommandName: "analyze",
|
||||||
StageName: "analyze",
|
StageName: "analyze",
|
||||||
PipelinePath: pipelinePath,
|
PipelinePath: flags.pipelinePath,
|
||||||
CampaignPath: campaignPath,
|
CampaignPath: flags.campaignPath,
|
||||||
CampaignFilePath: campaignFilePath,
|
CampaignFilePath: flags.campaignFilePath,
|
||||||
SessionPath: sessionPath,
|
SessionPath: flags.sessionPath,
|
||||||
SessionID: sessionID,
|
SessionID: flags.sessionID,
|
||||||
PreviousSessionID: previousSessionID,
|
PreviousSessionID: flags.previousSessionID,
|
||||||
Force: true,
|
Force: true,
|
||||||
SelectedArtifacts: normalizedArtifacts,
|
SelectedArtifacts: normalizedArtifacts,
|
||||||
})
|
})
|
||||||
@@ -164,42 +138,20 @@ func Analyze(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
// Publish force-runs the archive stage.
|
// Publish force-runs the publish stage.
|
||||||
func Publish(ctx context.Context, args []string, out io.Writer) error {
|
func Publish(ctx context.Context, args []string, out io.Writer) error {
|
||||||
positionalSessionID, args := pullLeadingSessionID(args)
|
|
||||||
fs := flag.NewFlagSet("publish", flag.ContinueOnError)
|
fs := flag.NewFlagSet("publish", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
|
|
||||||
var pipelinePath string
|
var flags commonConfigFlags
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var selectedArtifacts artifactSelectionFlag
|
var selectedArtifacts artifactSelectionFlag
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to publish (comma-separated or repeatable)")
|
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to publish (comma-separated or repeatable)")
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
if err := parseSessionAwareFlags("publish", fs, args, &flags.sessionID); err != nil {
|
||||||
return fmt.Errorf("publish: invalid flags: %w", err)
|
return err
|
||||||
}
|
}
|
||||||
if positionalSessionID == "" {
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
if err := applyParsedSessionIDArg("publish", fs, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("publish: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("publish", positionalSessionID, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
|
||||||
return fmt.Errorf("publish: session_id is required")
|
return fmt.Errorf("publish: session_id is required")
|
||||||
}
|
}
|
||||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
||||||
@@ -209,13 +161,13 @@ func Publish(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
|
|
||||||
summary, err := runSingleStageCommand(ctx, singleStageCommand{
|
summary, err := runSingleStageCommand(ctx, singleStageCommand{
|
||||||
CommandName: "publish",
|
CommandName: "publish",
|
||||||
StageName: "archive",
|
StageName: "publish",
|
||||||
PipelinePath: pipelinePath,
|
PipelinePath: flags.pipelinePath,
|
||||||
CampaignPath: campaignPath,
|
CampaignPath: flags.campaignPath,
|
||||||
CampaignFilePath: campaignFilePath,
|
CampaignFilePath: flags.campaignFilePath,
|
||||||
SessionPath: sessionPath,
|
SessionPath: flags.sessionPath,
|
||||||
SessionID: sessionID,
|
SessionID: flags.sessionID,
|
||||||
PreviousSessionID: previousSessionID,
|
PreviousSessionID: flags.previousSessionID,
|
||||||
Force: true,
|
Force: true,
|
||||||
SelectedArtifacts: normalizedArtifacts,
|
SelectedArtifacts: normalizedArtifacts,
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
func TestRunContinuesAfterCompletedStages(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||||
@@ -32,12 +32,12 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
|||||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
err := Resume(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
err := Run(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "executed=7 skipped=0") {
|
if !strings.Contains(out.String(), "executed=8 skipped=2") {
|
||||||
t.Fatalf("output = %q, want executed=7 skipped=0", out.String())
|
t.Fatalf("output = %q, want executed=8 skipped=2", out.String())
|
||||||
}
|
}
|
||||||
|
|
||||||
loaded, err := store.Load(context.Background(), manifestPath)
|
loaded, err := store.Load(context.Background(), manifestPath)
|
||||||
@@ -49,14 +49,14 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResumeNoRemainingStages(t *testing.T) {
|
func TestRunNoRemainingStagesRecordsSkippedStages(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||||
|
|
||||||
store := &manifest.LocalStore{}
|
store := &manifest.LocalStore{}
|
||||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
|
||||||
m.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
m.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||||
}
|
}
|
||||||
if err := store.Save(context.Background(), manifestPath, m); err != nil {
|
if err := store.Save(context.Background(), manifestPath, m); err != nil {
|
||||||
@@ -64,20 +64,20 @@ func TestResumeNoRemainingStages(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
err := Resume(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
err := Run(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "has no remaining stages") {
|
if !strings.Contains(out.String(), "executed=0 skipped=10") {
|
||||||
t.Fatalf("output = %q, want no remaining stages", out.String())
|
t.Fatalf("output = %q, want executed=0 skipped=10", out.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResumeForceRerunsSucceeded(t *testing.T) {
|
func TestRunForceRerunsSucceeded(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
_, _ = w.Write([]byte(`{"source":"resume-force-test","segments":[{"speaker":"alice"}]}`))
|
_, _ = w.Write([]byte(`{"source":"run-force-test","segments":[{"speaker":"alice"}]}`))
|
||||||
}))
|
}))
|
||||||
defer srv.Close()
|
defer srv.Close()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||||
@@ -85,7 +85,7 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
|
|||||||
|
|
||||||
store := &manifest.LocalStore{}
|
store := &manifest.LocalStore{}
|
||||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
|
||||||
m.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
m.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||||
}
|
}
|
||||||
if err := store.Save(context.Background(), manifestPath, m); err != nil {
|
if err := store.Save(context.Background(), manifestPath, m); err != nil {
|
||||||
@@ -93,11 +93,11 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
err := Resume(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--force"}, &out)
|
err := Run(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--force"}, &out)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "executed=9 skipped=0") {
|
if !strings.Contains(out.String(), "executed=10 skipped=0") {
|
||||||
t.Fatalf("output = %q, want forced full rerun", out.String())
|
t.Fatalf("output = %q, want forced full rerun", out.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -166,7 +166,7 @@ func TestRunStageSkipAndForce(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunStageForceMarksDownstreamStaleAndResumeContinuesFromStale(t *testing.T) {
|
func TestRunStageForceMarksDownstreamStaleAndRunContinuesFromStale(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||||
@@ -176,7 +176,7 @@ func TestRunStageForceMarksDownstreamStaleAndResumeContinuesFromStale(t *testing
|
|||||||
|
|
||||||
store := &manifest.LocalStore{}
|
store := &manifest.LocalStore{}
|
||||||
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
|
||||||
seed.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
seed.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||||
}
|
}
|
||||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||||
@@ -196,19 +196,19 @@ func TestRunStageForceMarksDownstreamStaleAndResumeContinuesFromStale(t *testing
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("load manifest after force: %v", err)
|
t.Fatalf("load manifest after force: %v", err)
|
||||||
}
|
}
|
||||||
for _, name := range []string{"normalize", "trim", "analyze", "archive", "notify"} {
|
for _, name := range []string{"normalize", "trim", "render", "analyze", "publish", "notify"} {
|
||||||
if afterForce.Stages[name] == nil || afterForce.Stages[name].Status != manifest.StatusStale {
|
if afterForce.Stages[name] == nil || afterForce.Stages[name].Status != manifest.StatusStale {
|
||||||
t.Fatalf("stage %q = %#v, want stale", name, afterForce.Stages[name])
|
t.Fatalf("stage %q = %#v, want stale", name, afterForce.Stages[name])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
out.Reset()
|
out.Reset()
|
||||||
err = Resume(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
err = Run(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "executed=5 skipped=0") {
|
if !strings.Contains(out.String(), "executed=6 skipped=4") {
|
||||||
t.Fatalf("output = %q, want resume to execute normalize..notify", out.String())
|
t.Fatalf("output = %q, want run to execute stale downstream stages", out.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -266,3 +266,30 @@ func TestRunStageNormalizeExecutes(t *testing.T) {
|
|||||||
t.Fatalf("normalize stage = %#v, want succeeded", m.Stages["normalize"])
|
t.Fatalf("normalize stage = %#v, want succeeded", m.Stages["normalize"])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunStageRenderExecutes(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||||
|
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||||
|
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.json"), `{"segments":[{"id":1}]}`)
|
||||||
|
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.trimmed.json"), `{"segments":[{"id":2}]}`)
|
||||||
|
|
||||||
|
var out bytes.Buffer
|
||||||
|
err := RunStage(context.Background(), []string{"render", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("RunStage(render) error = %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(out.String(), "stage=render executed=1 skipped=0") {
|
||||||
|
t.Fatalf("output = %q, want stage=render executed", out.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
store := &manifest.LocalStore{}
|
||||||
|
m, err := store.Load(context.Background(), manifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("load manifest: %v", err)
|
||||||
|
}
|
||||||
|
if m.Stages["render"] == nil || m.Stages["render"].Status != manifest.StatusSucceeded {
|
||||||
|
t.Fatalf("render stage = %#v, want succeeded", m.Stages["render"])
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -11,8 +11,8 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/notify"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/notify"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/scriptorium"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/scriptorium"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/seriatim"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/seriatim"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/whisperx"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/whisperx"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/logging"
|
"gitea.maximumdirect.net/eric/narratio/internal/logging"
|
||||||
@@ -83,9 +83,6 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
|||||||
if env.Scriptorium == nil {
|
if env.Scriptorium == nil {
|
||||||
env.Scriptorium = scriptorium.NewSubprocessRunner()
|
env.Scriptorium = scriptorium.NewSubprocessRunner()
|
||||||
}
|
}
|
||||||
if env.Storage == nil {
|
|
||||||
env.Storage = &storage.NoopBackend{}
|
|
||||||
}
|
|
||||||
if env.ObjectStore == nil && needsObjectStoreForRun(env.Config, stages) {
|
if env.ObjectStore == nil && needsObjectStoreForRun(env.Config, stages) {
|
||||||
objectStore, err := newCommandObjectStore(ctx, env.Config, nil)
|
objectStore, err := newCommandObjectStore(ctx, env.Config, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -96,7 +93,7 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
|||||||
if needsRemoteLocksForRun(env.Config, stages) {
|
if needsRemoteLocksForRun(env.Config, stages) {
|
||||||
locks, err := loadEffectiveLocks(ctx, env.Config, env.ObjectStore)
|
locks, err := loadEffectiveLocks(ctx, env.Config, env.ObjectStore)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("load remote archive locks: %w", err)
|
return nil, fmt.Errorf("load remote publish locks: %w", err)
|
||||||
}
|
}
|
||||||
applyEffectiveLocks(env.Config, locks.All)
|
applyEffectiveLocks(env.Config, locks.All)
|
||||||
}
|
}
|
||||||
@@ -234,14 +231,14 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
|||||||
env.Logger.Info("stage succeeded", "stage", s.Name())
|
env.Logger.Info("stage succeeded", "stage", s.Name())
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := runPostArchiveCleanup(ctx, env, manifestPath, m, executed); err != nil {
|
if err := runPostPublishCleanup(ctx, env, manifestPath, m, executed); err != nil {
|
||||||
failedAt := nowUTC()
|
failedAt := nowUTC()
|
||||||
runManifest.MarkFailed(failedAt, err.Error())
|
runManifest.MarkFailed(failedAt, err.Error())
|
||||||
syncRunManifestIdentityFromSession(m, runManifest)
|
syncRunManifestIdentityFromSession(m, runManifest)
|
||||||
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
|
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
|
||||||
return nil, fmt.Errorf("post-archive cleanup failed (%v) and run-manifest save failed (%v)", err, saveErr)
|
return nil, fmt.Errorf("post-publish cleanup failed (%v) and run-manifest save failed (%v)", err, saveErr)
|
||||||
}
|
}
|
||||||
return nil, fmt.Errorf("post-archive cleanup: %w", err)
|
return nil, fmt.Errorf("post-publish cleanup: %w", err)
|
||||||
}
|
}
|
||||||
completedAt := nowUTC()
|
completedAt := nowUTC()
|
||||||
runManifest.MarkSucceeded(completedAt)
|
runManifest.MarkSucceeded(completedAt)
|
||||||
@@ -414,6 +411,8 @@ func mapResultOutputs(stageName string, result *stage.StageResult, runID string)
|
|||||||
if stageName == "analyze" {
|
if stageName == "analyze" {
|
||||||
sourceID = artifacts.ConfiguredArtifactSourceID(ref.Kind)
|
sourceID = artifacts.ConfiguredArtifactSourceID(ref.Kind)
|
||||||
kind = "scriptorium_artifact"
|
kind = "scriptorium_artifact"
|
||||||
|
} else {
|
||||||
|
sourceID = sourceIDForOutputKind(kind)
|
||||||
}
|
}
|
||||||
out = append(out, manifest.ArtifactRecord{
|
out = append(out, manifest.ArtifactRecord{
|
||||||
Kind: kind,
|
Kind: kind,
|
||||||
@@ -428,6 +427,22 @@ func mapResultOutputs(stageName string, result *stage.StageResult, runID string)
|
|||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func sourceIDForOutputKind(kind string) string {
|
||||||
|
trimmed := strings.TrimSpace(kind)
|
||||||
|
if trimmed == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
if trimmed == "session_bounds" {
|
||||||
|
return artifacts.ArtifactBoundsSession
|
||||||
|
}
|
||||||
|
for _, spec := range artifactmodel.RuntimeTranscriptArtifacts() {
|
||||||
|
if spec.OutputKind == trimmed {
|
||||||
|
return spec.SourceID
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
func applyStageResultToManifest(m *manifest.Manifest, stageName string, result *stage.StageResult) {
|
func applyStageResultToManifest(m *manifest.Manifest, stageName string, result *stage.StageResult) {
|
||||||
if m == nil || result == nil {
|
if m == nil || result == nil {
|
||||||
return
|
return
|
||||||
@@ -568,16 +583,16 @@ func needsObjectStoreForRun(cfg *config.Config, stages []stage.Stage) bool {
|
|||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !stageRequested("archive") {
|
if !stageRequested("publish") {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
if cfg.Pipeline.Archive == nil {
|
if cfg.Pipeline.Publish == nil {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
if cfg.Pipeline.Archive.Enabled != nil && !*cfg.Pipeline.Archive.Enabled {
|
if cfg.Pipeline.Publish.Enabled != nil && !*cfg.Pipeline.Publish.Enabled {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
if cfg.Pipeline.Archive.UploadRun != nil && !*cfg.Pipeline.Archive.UploadRun {
|
if cfg.Pipeline.Publish.UploadRun != nil && !*cfg.Pipeline.Publish.UploadRun {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
return true
|
return true
|
||||||
@@ -587,23 +602,23 @@ func needsRemoteLocksForRun(cfg *config.Config, stages []stage.Stage) bool {
|
|||||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
archiveRequested := false
|
publishRequested := false
|
||||||
for _, s := range stages {
|
for _, s := range stages {
|
||||||
if s != nil && s.Name() == "archive" {
|
if s != nil && s.Name() == "publish" {
|
||||||
archiveRequested = true
|
publishRequested = true
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !archiveRequested {
|
if !publishRequested {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
if cfg.Pipeline.Archive == nil {
|
if cfg.Pipeline.Publish == nil {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
if cfg.Pipeline.Archive.Enabled != nil && !*cfg.Pipeline.Archive.Enabled {
|
if cfg.Pipeline.Publish.Enabled != nil && !*cfg.Pipeline.Publish.Enabled {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
if cfg.Pipeline.Archive.UploadRun != nil && !*cfg.Pipeline.Archive.UploadRun {
|
if cfg.Pipeline.Publish.UploadRun != nil && !*cfg.Pipeline.Publish.UploadRun {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
return cfg.Pipeline.Storage.S3 != nil
|
return cfg.Pipeline.Storage.S3 != nil
|
||||||
|
|||||||
@@ -248,16 +248,16 @@ func TestNeedsObjectStoreForRunPrepareWithPreviousRequirements(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteStagesArchiveSkipsRequiredUnselectedConfiguredPromotion(t *testing.T) {
|
func TestExecuteStagesPublishSkipsRequiredUnselectedConfiguredOutput(t *testing.T) {
|
||||||
cfg := testConfig(t)
|
cfg := testConfig(t)
|
||||||
cfg.Pipeline.Storage.S3 = &config.StorageS3Config{
|
cfg.Pipeline.Storage.S3 = &config.StorageS3Config{
|
||||||
Bucket: "my-dnd-archive",
|
Bucket: "my-dnd-archive",
|
||||||
RootPrefix: "dnd",
|
RootPrefix: "dnd",
|
||||||
}
|
}
|
||||||
cfg.Pipeline.Archive = &config.ArchiveConfig{
|
cfg.Pipeline.Publish = &config.PublishConfig{
|
||||||
Enabled: boolPtr(true),
|
Enabled: boolPtr(true),
|
||||||
UploadRun: boolPtr(true),
|
UploadRun: boolPtr(true),
|
||||||
PromoteArtifacts: []config.ArchivePromotionRule{
|
Outputs: []config.PublishOutputRule{
|
||||||
{Source: "narratio.artifact.session_recap", Dest: "artifacts/session_recap.md", Required: boolPtr(true)},
|
{Source: "narratio.artifact.session_recap", Dest: "artifacts/session_recap.md", Required: boolPtr(true)},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -273,7 +273,7 @@ func TestExecuteStagesArchiveSkipsRequiredUnselectedConfiguredPromotion(t *testi
|
|||||||
manifestPath := manifestPathFor(cfg)
|
manifestPath := manifestPathFor(cfg)
|
||||||
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
||||||
seed.Campaign = cfg.Session.Campaign
|
seed.Campaign = cfg.Session.Campaign
|
||||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim"} {
|
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render"} {
|
||||||
seed.MarkStageSucceeded(stageName, time.Now().UTC(), nil)
|
seed.MarkStageSucceeded(stageName, time.Now().UTC(), nil)
|
||||||
}
|
}
|
||||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||||
@@ -283,9 +283,9 @@ func TestExecuteStagesArchiveSkipsRequiredUnselectedConfiguredPromotion(t *testi
|
|||||||
t.Fatalf("Save manifest error = %v", err)
|
t.Fatalf("Save manifest error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
archiveStageImpl, err := stage.Select("archive")
|
publishStageImpl, err := stage.Select("publish")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Select(archive) error = %v", err)
|
t.Fatalf("Select(publish) error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
summary, err := executeStages(
|
summary, err := executeStages(
|
||||||
@@ -293,7 +293,7 @@ func TestExecuteStagesArchiveSkipsRequiredUnselectedConfiguredPromotion(t *testi
|
|||||||
cfg,
|
cfg,
|
||||||
[]stage.Stage{
|
[]stage.Stage{
|
||||||
selectedAnalyzeArtifactStage{expected: []string{"player_handout"}},
|
selectedAnalyzeArtifactStage{expected: []string{"player_handout"}},
|
||||||
archiveStageImpl,
|
publishStageImpl,
|
||||||
},
|
},
|
||||||
RunOptions{
|
RunOptions{
|
||||||
SelectedArtifacts: []string{"player_handout"},
|
SelectedArtifacts: []string{"player_handout"},
|
||||||
@@ -303,25 +303,25 @@ func TestExecuteStagesArchiveSkipsRequiredUnselectedConfiguredPromotion(t *testi
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
if len(summary.Executed) != 2 || summary.Executed[0] != "analyze" || summary.Executed[1] != "archive" {
|
if len(summary.Executed) != 2 || summary.Executed[0] != "analyze" || summary.Executed[1] != "publish" {
|
||||||
t.Fatalf("executed = %#v, want analyze and archive", summary.Executed)
|
t.Fatalf("executed = %#v, want analyze and publish", summary.Executed)
|
||||||
}
|
}
|
||||||
|
|
||||||
loadedManifest, err := store.Load(context.Background(), summary.ManifestPath)
|
loadedManifest, err := store.Load(context.Background(), summary.ManifestPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Load manifest error = %v", err)
|
t.Fatalf("Load manifest error = %v", err)
|
||||||
}
|
}
|
||||||
meta := loadedManifest.Stages["archive"].Metadata
|
meta := loadedManifest.Stages["publish"].Metadata
|
||||||
skipped, ok := meta["skipped_unselected_promotions"].([]any)
|
skipped, ok := meta["skipped_unselected_outputs"].([]any)
|
||||||
if !ok || len(skipped) != 1 {
|
if !ok || len(skipped) != 1 {
|
||||||
t.Fatalf("skipped_unselected_promotions = %#v, want one item", meta["skipped_unselected_promotions"])
|
t.Fatalf("skipped_unselected_outputs = %#v, want one item", meta["skipped_unselected_outputs"])
|
||||||
}
|
}
|
||||||
item, ok := skipped[0].(map[string]any)
|
item, ok := skipped[0].(map[string]any)
|
||||||
if !ok {
|
if !ok {
|
||||||
t.Fatalf("skipped item = %#v, want object", skipped[0])
|
t.Fatalf("skipped item = %#v, want object", skipped[0])
|
||||||
}
|
}
|
||||||
if item["source"] != "narratio.artifact.session_recap" || item["dest"] != "artifacts/session_recap.md" || item["required"] != true {
|
if item["source"] != "narratio.artifact.session_recap" || item["dest"] != "artifacts/session_recap.md" || item["required"] != true {
|
||||||
t.Fatalf("skipped item = %#v, want required session_recap promotion", item)
|
t.Fatalf("skipped item = %#v, want required session_recap published output", item)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -332,8 +332,8 @@ func TestExecuteStagesPlaceholderSuccessUpdatesManifest(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
if len(summary.StageNames) != 9 || len(summary.Executed) != 9 || len(summary.Skipped) != 0 {
|
if len(summary.StageNames) != 10 || len(summary.Executed) != 10 || len(summary.Skipped) != 0 {
|
||||||
t.Fatalf("summary = %#v, want all 9 executed", summary)
|
t.Fatalf("summary = %#v, want all 10 executed", summary)
|
||||||
}
|
}
|
||||||
|
|
||||||
store := &manifest.LocalStore{}
|
store := &manifest.LocalStore{}
|
||||||
@@ -342,7 +342,7 @@ func TestExecuteStagesPlaceholderSuccessUpdatesManifest(t *testing.T) {
|
|||||||
t.Fatalf("Load manifest error = %v", err)
|
t.Fatalf("Load manifest error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
|
||||||
sr := m.Stages[name]
|
sr := m.Stages[name]
|
||||||
if sr == nil {
|
if sr == nil {
|
||||||
t.Fatalf("missing stage record %q", name)
|
t.Fatalf("missing stage record %q", name)
|
||||||
@@ -425,12 +425,21 @@ func TestExecuteStagesPlaceholderSuccessUpdatesManifest(t *testing.T) {
|
|||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if name == "archive" {
|
if name == "render" {
|
||||||
if sr.Metadata == nil || sr.Metadata["stage"] != "archive" {
|
if sr.Metadata == nil || sr.Metadata["stage"] != "render" {
|
||||||
t.Fatalf("archive metadata missing stage=archive: %#v", sr.Metadata)
|
t.Fatalf("render metadata missing stage=render: %#v", sr.Metadata)
|
||||||
|
}
|
||||||
|
if len(sr.Outputs) == 0 {
|
||||||
|
t.Fatalf("render outputs missing")
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if name == "publish" {
|
||||||
|
if sr.Metadata == nil || sr.Metadata["stage"] != "publish" {
|
||||||
|
t.Fatalf("publish metadata missing stage=publish: %#v", sr.Metadata)
|
||||||
}
|
}
|
||||||
if sr.Metadata["skipped"] != true {
|
if sr.Metadata["skipped"] != true {
|
||||||
t.Fatalf("archive metadata missing skipped=true for test config without archive section: %#v", sr.Metadata)
|
t.Fatalf("publish metadata missing skipped=true for test config without publish section: %#v", sr.Metadata)
|
||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -522,7 +531,7 @@ func TestExecuteStagesForceSuccessInvalidatesDownstreamSucceededStages(t *testin
|
|||||||
store := &manifest.LocalStore{}
|
store := &manifest.LocalStore{}
|
||||||
|
|
||||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "archive", "notify"} {
|
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "publish", "notify"} {
|
||||||
existing.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
existing.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
||||||
}
|
}
|
||||||
existing.MarkStageFailed("analyze", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), "previous analyze failure")
|
existing.MarkStageFailed("analyze", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), "previous analyze failure")
|
||||||
@@ -553,7 +562,7 @@ func TestExecuteStagesForceSuccessInvalidatesDownstreamSucceededStages(t *testin
|
|||||||
if loaded.Stages["polish"] == nil || loaded.Stages["polish"].Status != manifest.StatusSucceeded {
|
if loaded.Stages["polish"] == nil || loaded.Stages["polish"].Status != manifest.StatusSucceeded {
|
||||||
t.Fatalf("polish status = %#v, want succeeded", loaded.Stages["polish"])
|
t.Fatalf("polish status = %#v, want succeeded", loaded.Stages["polish"])
|
||||||
}
|
}
|
||||||
for _, stageName := range []string{"normalize", "trim", "archive", "notify"} {
|
for _, stageName := range []string{"normalize", "trim", "render", "publish", "notify"} {
|
||||||
if loaded.Stages[stageName] == nil || loaded.Stages[stageName].Status != manifest.StatusStale {
|
if loaded.Stages[stageName] == nil || loaded.Stages[stageName].Status != manifest.StatusStale {
|
||||||
t.Fatalf("%s status = %#v, want stale", stageName, loaded.Stages[stageName])
|
t.Fatalf("%s status = %#v, want stale", stageName, loaded.Stages[stageName])
|
||||||
}
|
}
|
||||||
@@ -729,7 +738,7 @@ func TestExecuteStagesRunManifestRecordsSkippedStage(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteStagesRunLocalArtifactsAndCanonicalPromotion(t *testing.T) {
|
func TestExecuteStagesRunLocalArtifactsAndCanonicalSync(t *testing.T) {
|
||||||
cfg := testConfig(t)
|
cfg := testConfig(t)
|
||||||
stages := []stage.Stage{
|
stages := []stage.Stage{
|
||||||
BuildFullPlan()[0], // prepare
|
BuildFullPlan()[0], // prepare
|
||||||
@@ -778,7 +787,7 @@ func TestExecuteStagesRunLocalArtifactsAndCanonicalPromotion(t *testing.T) {
|
|||||||
}
|
}
|
||||||
for _, p := range canonicalChecks {
|
for _, p := range canonicalChecks {
|
||||||
if _, statErr := os.Stat(p); statErr != nil {
|
if _, statErr := os.Stat(p); statErr != nil {
|
||||||
t.Fatalf("canonical promoted artifact missing at %q: %v", p, statErr)
|
t.Fatalf("canonical published artifact missing at %q: %v", p, statErr)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -864,7 +873,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
|||||||
{name: "merge", env: &Env{Seriatim: &seriatim.FakeRunner{Err: errors.New("merge fail")}}},
|
{name: "merge", env: &Env{Seriatim: &seriatim.FakeRunner{Err: errors.New("merge fail")}}},
|
||||||
{name: "polish", env: &Env{Audita: &audita.FakeRunner{Err: errors.New("polish fail")}}},
|
{name: "polish", env: &Env{Audita: &audita.FakeRunner{Err: errors.New("polish fail")}}},
|
||||||
{name: "analyze", env: &Env{Scriptorium: &scriptorium.FakeRunner{RunErr: errors.New("analyze fail")}}},
|
{name: "analyze", env: &Env{Scriptorium: &scriptorium.FakeRunner{RunErr: errors.New("analyze fail")}}},
|
||||||
{name: "archive", env: &Env{ObjectStore: &storage.FakeBackend{UploadErr: errors.New("archive fail")}}},
|
{name: "publish", env: &Env{ObjectStore: &storage.FakeBackend{UploadErr: errors.New("publish fail")}}},
|
||||||
{name: "notify", env: &Env{Notifier: ¬ify.FakeSender{Err: errors.New("notify fail")}}},
|
{name: "notify", env: &Env{Notifier: ¬ify.FakeSender{Err: errors.New("notify fail")}}},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -949,8 +958,8 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if tc.name == "archive" {
|
if tc.name == "publish" {
|
||||||
cfg.Pipeline.Archive = &config.ArchiveConfig{
|
cfg.Pipeline.Publish = &config.PublishConfig{
|
||||||
Enabled: boolPtr(true),
|
Enabled: boolPtr(true),
|
||||||
UploadRun: boolPtr(true),
|
UploadRun: boolPtr(true),
|
||||||
}
|
}
|
||||||
@@ -961,13 +970,13 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
|||||||
runID := "20260516T010203Z-0a1b2c3d"
|
runID := "20260516T010203Z-0a1b2c3d"
|
||||||
runWorkDir := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
runWorkDir := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||||
if err := os.MkdirAll(filepath.Join(runWorkDir, "inputs"), 0o755); err != nil {
|
if err := os.MkdirAll(filepath.Join(runWorkDir, "inputs"), 0o755); err != nil {
|
||||||
t.Fatalf("mkdir archive inputs dir: %v", err)
|
t.Fatalf("mkdir publish inputs dir: %v", err)
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(filepath.Join(runWorkDir, "inputs", "session.yml"), []byte("session_id: 2026-05-03\n"), 0o644); err != nil {
|
if err := os.WriteFile(filepath.Join(runWorkDir, "inputs", "session.yml"), []byte("session_id: 2026-05-03\n"), 0o644); err != nil {
|
||||||
t.Fatalf("write archive fixture session.yml: %v", err)
|
t.Fatalf("write publish fixture session.yml: %v", err)
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(filepath.Join(runWorkDir, "manifest.json"), []byte("{}\n"), 0o644); err != nil {
|
if err := os.WriteFile(filepath.Join(runWorkDir, "manifest.json"), []byte("{}\n"), 0o644); err != nil {
|
||||||
t.Fatalf("write archive fixture manifest.json: %v", err)
|
t.Fatalf("write publish fixture manifest.json: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
||||||
@@ -977,11 +986,11 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
|||||||
seed.S3Bucket = "my-dnd-archive"
|
seed.S3Bucket = "my-dnd-archive"
|
||||||
seed.S3SessionPrefix = "dnd/campaigns/" + cfg.Session.Campaign + "/sessions/" + cfg.Session.SessionID + "/"
|
seed.S3SessionPrefix = "dnd/campaigns/" + cfg.Session.Campaign + "/sessions/" + cfg.Session.SessionID + "/"
|
||||||
seed.S3RunPrefix = seed.S3SessionPrefix + "runs/" + runID + "/"
|
seed.S3RunPrefix = seed.S3SessionPrefix + "runs/" + runID + "/"
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze"} {
|
||||||
seed.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
seed.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
||||||
}
|
}
|
||||||
if err := tc.env.ManifestStore.Save(context.Background(), manifestPathFor(cfg), seed); err != nil {
|
if err := tc.env.ManifestStore.Save(context.Background(), manifestPathFor(cfg), seed); err != nil {
|
||||||
t.Fatalf("seed archive manifest: %v", err)
|
t.Fatalf("seed publish manifest: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -41,3 +41,55 @@ func applyParsedSessionIDArg(command string, fs *flag.FlagSet, sessionID *string
|
|||||||
return fmt.Errorf("%s: unexpected positional arguments", command)
|
return fmt.Errorf("%s: unexpected positional arguments", command)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func resolveParsedSessionID(command, positionalSessionID string, fs *flag.FlagSet, sessionID *string) error {
|
||||||
|
if strings.TrimSpace(positionalSessionID) == "" {
|
||||||
|
return applyParsedSessionIDArg(command, fs, sessionID)
|
||||||
|
}
|
||||||
|
if fs.NArg() != 0 {
|
||||||
|
return fmt.Errorf("%s: unexpected positional arguments", command)
|
||||||
|
}
|
||||||
|
return applyPositionalSessionID(command, positionalSessionID, sessionID)
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseSessionAwareFlags(command string, fs *flag.FlagSet, args []string, sessionID *string) error {
|
||||||
|
positionalSessionID, args := pullLeadingSessionID(args)
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return fmt.Errorf("%s: invalid flags: %w", command, err)
|
||||||
|
}
|
||||||
|
return resolveParsedSessionID(command, positionalSessionID, fs, sessionID)
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseSessionIDAndOnePositionalArg(command, argName string, fs *flag.FlagSet, args []string, sessionID *string) (string, error) {
|
||||||
|
var positionalSessionID string
|
||||||
|
value := ""
|
||||||
|
if len(args) >= 2 && !isCLIFlagToken(args[0]) && !isCLIFlagToken(args[1]) {
|
||||||
|
positionalSessionID = strings.TrimSpace(args[0])
|
||||||
|
value = strings.TrimSpace(args[1])
|
||||||
|
args = append([]string(nil), args[2:]...)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return "", fmt.Errorf("%s: invalid flags: %w", command, err)
|
||||||
|
}
|
||||||
|
if value == "" {
|
||||||
|
switch fs.NArg() {
|
||||||
|
case 2:
|
||||||
|
positionalSessionID = strings.TrimSpace(fs.Arg(0))
|
||||||
|
value = strings.TrimSpace(fs.Arg(1))
|
||||||
|
case 1:
|
||||||
|
if strings.TrimSpace(*sessionID) == "" {
|
||||||
|
return "", fmt.Errorf("%s: expected session_id and %s", command, argName)
|
||||||
|
}
|
||||||
|
value = strings.TrimSpace(fs.Arg(0))
|
||||||
|
default:
|
||||||
|
return "", fmt.Errorf("%s: expected session_id and %s", command, argName)
|
||||||
|
}
|
||||||
|
} else if fs.NArg() != 0 {
|
||||||
|
return "", fmt.Errorf("%s: unexpected positional arguments", command)
|
||||||
|
}
|
||||||
|
if err := applyPositionalSessionID(command, positionalSessionID, sessionID); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return value, nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -71,15 +71,36 @@ func TestExecutePositionalSessionIDMismatchFails(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteSessionIDFlagFails(t *testing.T) {
|
func TestExecuteSessionIDFlagMismatchFails(t *testing.T) {
|
||||||
var stdout bytes.Buffer
|
var stdout bytes.Buffer
|
||||||
var stderr bytes.Buffer
|
var stderr bytes.Buffer
|
||||||
code := Execute([]string{"session", "status", "2026-05-03", "--session-id", "2026-05-04"}, &stdout, &stderr)
|
code := Execute([]string{"session", "status", "2026-05-03", "--session-id", "2026-05-04"}, &stdout, &stderr)
|
||||||
if code == 0 {
|
if code == 0 {
|
||||||
t.Fatal("exit code = 0, want non-zero")
|
t.Fatal("exit code = 0, want non-zero")
|
||||||
}
|
}
|
||||||
if !strings.Contains(stderr.String(), "flag provided but not defined: -session-id") {
|
if !strings.Contains(stderr.String(), "does not match expected session id") {
|
||||||
t.Fatalf("stderr = %q, want invalid --session-id flag", stderr.String())
|
t.Fatalf("stderr = %q, want positional/flag mismatch", stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteSessionIDFlagAcceptedWithoutPositional(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{
|
||||||
|
"session", "status",
|
||||||
|
"--session-id", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "Session: 2026-05-03") {
|
||||||
|
t.Fatalf("stdout = %q, want status output", stdout.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -141,8 +162,8 @@ func TestExecuteWorkflowCommandsAcceptPositionalSessionID(t *testing.T) {
|
|||||||
wantForce bool
|
wantForce bool
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "resume",
|
name: "run",
|
||||||
args: []string{"resume", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath},
|
args: []string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath},
|
||||||
wantStage: "prepare",
|
wantStage: "prepare",
|
||||||
wantForce: false,
|
wantForce: false,
|
||||||
},
|
},
|
||||||
@@ -155,13 +176,13 @@ func TestExecuteWorkflowCommandsAcceptPositionalSessionID(t *testing.T) {
|
|||||||
{
|
{
|
||||||
name: "publish",
|
name: "publish",
|
||||||
args: []string{"publish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
args: []string{"publish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
||||||
wantStage: "archive",
|
wantStage: "publish",
|
||||||
wantForce: true,
|
wantForce: true,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "run-stage",
|
name: "run-stage",
|
||||||
args: []string{"run-stage", "archive", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
args: []string{"run-stage", "publish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
||||||
wantStage: "archive",
|
wantStage: "publish",
|
||||||
wantForce: false,
|
wantForce: false,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -212,7 +233,7 @@ func TestExecuteSessionSubcommandsAcceptPositionalSessionID(t *testing.T) {
|
|||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03")
|
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03")
|
||||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
fake.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
fake.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||||
fake.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "2026-05-03", "sample-campaign")})
|
fake.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "2026-05-03", "sample-campaign")})
|
||||||
var storeInitCalls int
|
var storeInitCalls int
|
||||||
@@ -246,7 +267,7 @@ func TestExecuteSessionSubcommandsAcceptPositionalSessionID(t *testing.T) {
|
|||||||
{
|
{
|
||||||
name: "locks",
|
name: "locks",
|
||||||
args: []string{"session", "locks", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath},
|
args: []string{"session", "locks", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath},
|
||||||
want: "Archive locks:",
|
want: "Publish locks:",
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -325,9 +346,9 @@ func TestExecuteSessionLocksMutationAcceptsPositionalSessionID(t *testing.T) {
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("locks remove exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("locks remove exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
store, err := config.LoadArchiveLockStoreBytes("locks.yml", fake.Objects[key].Data, nil)
|
store, err := config.LoadPublishLockStoreBytes("locks.yml", fake.Objects[key].Data, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("LoadArchiveLockStoreBytes() error = %v", err)
|
t.Fatalf("LoadPublishLockStoreBytes() error = %v", err)
|
||||||
}
|
}
|
||||||
if len(store.Locks) != 0 {
|
if len(store.Locks) != 0 {
|
||||||
t.Fatalf("locks after remove = %#v, want empty", store.Locks)
|
t.Fatalf("locks after remove = %#v, want empty", store.Locks)
|
||||||
|
|||||||
@@ -3,24 +3,30 @@ package artifactmodel
|
|||||||
import "strings"
|
import "strings"
|
||||||
|
|
||||||
const (
|
const (
|
||||||
SourceTranscriptBase = "narratio.transcript.base"
|
SourceTranscriptBase = "narratio.transcript.base"
|
||||||
SourceTranscriptPolished = "narratio.transcript.polished"
|
SourceTranscriptPolished = "narratio.transcript.polished"
|
||||||
SourceTranscriptFinal = "narratio.transcript.final"
|
SourceTranscriptFinal = "narratio.transcript.final"
|
||||||
SourceTranscriptFinalTrimmed = "narratio.transcript.final_trimmed"
|
SourceTranscriptFinalTrimmed = "narratio.transcript.final_trimmed"
|
||||||
|
SourceTranscriptFinalMarkdown = "narratio.transcript.final_markdown"
|
||||||
|
SourceTranscriptFinalTrimmedMarkdown = "narratio.transcript.final_trimmed_markdown"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
TranscriptPathBase = "transcripts/base.json"
|
TranscriptPathBase = "transcripts/base.json"
|
||||||
TranscriptPathPolished = "transcripts/polished.json"
|
TranscriptPathPolished = "transcripts/polished.json"
|
||||||
TranscriptPathFinal = "transcripts/final.json"
|
TranscriptPathFinal = "transcripts/final.json"
|
||||||
TranscriptPathFinalTrimmed = "transcripts/final.trimmed.json"
|
TranscriptPathFinalTrimmed = "transcripts/final.trimmed.json"
|
||||||
|
TranscriptPathFinalMarkdown = "transcripts/final.md"
|
||||||
|
TranscriptPathFinalTrimmedMarkdown = "transcripts/final.trimmed.md"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
TranscriptOutputKindBase = "transcript_base"
|
TranscriptOutputKindBase = "transcript_base"
|
||||||
TranscriptOutputKindPolished = "transcript_polished"
|
TranscriptOutputKindPolished = "transcript_polished"
|
||||||
TranscriptOutputKindFinal = "transcript_final"
|
TranscriptOutputKindFinal = "transcript_final"
|
||||||
TranscriptOutputKindFinalTrimmed = "transcript_final_trimmed"
|
TranscriptOutputKindFinalTrimmed = "transcript_final_trimmed"
|
||||||
|
TranscriptOutputKindFinalMarkdown = "transcript_final_markdown"
|
||||||
|
TranscriptOutputKindFinalTrimmedMarkdown = "transcript_final_trimmed_markdown"
|
||||||
)
|
)
|
||||||
|
|
||||||
// TranscriptArtifactSpec describes one built-in transcript artifact mapping.
|
// TranscriptArtifactSpec describes one built-in transcript artifact mapping.
|
||||||
@@ -56,6 +62,18 @@ var runtimeTranscriptArtifacts = []TranscriptArtifactSpec{
|
|||||||
ProducerStage: "trim",
|
ProducerStage: "trim",
|
||||||
OutputKind: TranscriptOutputKindFinalTrimmed,
|
OutputKind: TranscriptOutputKindFinalTrimmed,
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
SourceID: SourceTranscriptFinalMarkdown,
|
||||||
|
CanonicalRelPath: TranscriptPathFinalMarkdown,
|
||||||
|
ProducerStage: "render",
|
||||||
|
OutputKind: TranscriptOutputKindFinalMarkdown,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
SourceID: SourceTranscriptFinalTrimmedMarkdown,
|
||||||
|
CanonicalRelPath: TranscriptPathFinalTrimmedMarkdown,
|
||||||
|
ProducerStage: "render",
|
||||||
|
OutputKind: TranscriptOutputKindFinalTrimmedMarkdown,
|
||||||
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
// RuntimeTranscriptArtifacts returns transcript mappings in pipeline order.
|
// RuntimeTranscriptArtifacts returns transcript mappings in pipeline order.
|
||||||
|
|||||||
60
internal/artifactmodel/transcripts_test.go
Normal file
60
internal/artifactmodel/transcripts_test.go
Normal file
@@ -0,0 +1,60 @@
|
|||||||
|
package artifactmodel
|
||||||
|
|
||||||
|
import (
|
||||||
|
"reflect"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRuntimeTranscriptArtifactsIncludesMarkdownOutputs(t *testing.T) {
|
||||||
|
want := []TranscriptArtifactSpec{
|
||||||
|
{
|
||||||
|
SourceID: SourceTranscriptBase,
|
||||||
|
CanonicalRelPath: TranscriptPathBase,
|
||||||
|
ProducerStage: "merge",
|
||||||
|
OutputKind: TranscriptOutputKindBase,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
SourceID: SourceTranscriptPolished,
|
||||||
|
CanonicalRelPath: TranscriptPathPolished,
|
||||||
|
ProducerStage: "polish",
|
||||||
|
OutputKind: TranscriptOutputKindPolished,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
SourceID: SourceTranscriptFinal,
|
||||||
|
CanonicalRelPath: TranscriptPathFinal,
|
||||||
|
ProducerStage: "normalize",
|
||||||
|
OutputKind: TranscriptOutputKindFinal,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
SourceID: SourceTranscriptFinalTrimmed,
|
||||||
|
CanonicalRelPath: TranscriptPathFinalTrimmed,
|
||||||
|
ProducerStage: "trim",
|
||||||
|
OutputKind: TranscriptOutputKindFinalTrimmed,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
SourceID: SourceTranscriptFinalMarkdown,
|
||||||
|
CanonicalRelPath: TranscriptPathFinalMarkdown,
|
||||||
|
ProducerStage: "render",
|
||||||
|
OutputKind: TranscriptOutputKindFinalMarkdown,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
SourceID: SourceTranscriptFinalTrimmedMarkdown,
|
||||||
|
CanonicalRelPath: TranscriptPathFinalTrimmedMarkdown,
|
||||||
|
ProducerStage: "render",
|
||||||
|
OutputKind: TranscriptOutputKindFinalTrimmedMarkdown,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
got := RuntimeTranscriptArtifacts()
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("RuntimeTranscriptArtifacts() = %#v, want %#v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLookupRuntimeTranscriptArtifactFindsMarkdownOutputs(t *testing.T) {
|
||||||
|
for _, source := range []string{SourceTranscriptFinalMarkdown, SourceTranscriptFinalTrimmedMarkdown} {
|
||||||
|
if _, ok := LookupRuntimeTranscriptArtifact(source); !ok {
|
||||||
|
t.Fatalf("LookupRuntimeTranscriptArtifact(%q) ok=false, want true", source)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
237
internal/artifactpolicy/policy.go
Normal file
237
internal/artifactpolicy/policy.go
Normal file
@@ -0,0 +1,237 @@
|
|||||||
|
package artifactpolicy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/pathsafe"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
SourceBoundsSession = "narratio.bounds.session"
|
||||||
|
|
||||||
|
configuredSourcePrefix = "narratio.artifact."
|
||||||
|
previousConfiguredSrcPrefix = "narratio.previous_session.artifact."
|
||||||
|
)
|
||||||
|
|
||||||
|
var configuredSourceRE = regexp.MustCompile(`^narratio\.artifact\.([a-z][a-z0-9_]*)$`)
|
||||||
|
var previousSourceRE = regexp.MustCompile(`^narratio\.previous_session\.artifact\.([a-z][a-z0-9_]*)$`)
|
||||||
|
|
||||||
|
var (
|
||||||
|
ErrUnsupportedScriptoriumInputSource = errors.New("unsupported scriptorium input source")
|
||||||
|
ErrInvalidPreviousSessionSource = errors.New("invalid previous-session source format")
|
||||||
|
)
|
||||||
|
|
||||||
|
type SourceKind string
|
||||||
|
|
||||||
|
const (
|
||||||
|
SourceKindBuiltIn SourceKind = "built_in"
|
||||||
|
SourceKindConfiguredArtifact SourceKind = "configured_artifact"
|
||||||
|
SourceKindPreviousArtifact SourceKind = "previous_session_configured_artifact"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Source describes one normalized artifact source identifier.
|
||||||
|
type Source struct {
|
||||||
|
ID string
|
||||||
|
Kind SourceKind
|
||||||
|
ConfiguredKey string
|
||||||
|
}
|
||||||
|
|
||||||
|
// ScriptoriumInputSourceDescriptor describes one validated Scriptorium input source.
|
||||||
|
type ScriptoriumInputSourceDescriptor struct {
|
||||||
|
Source Source
|
||||||
|
PreviousSession *PreviousSessionSourceDescriptor
|
||||||
|
}
|
||||||
|
|
||||||
|
// PreviousSessionSourceDescriptor describes one canonical previous-session input source.
|
||||||
|
type PreviousSessionSourceDescriptor struct {
|
||||||
|
SourceID string
|
||||||
|
ConfiguredKey string
|
||||||
|
ConfiguredSourceID string
|
||||||
|
}
|
||||||
|
|
||||||
|
// UnknownConfiguredArtifactError reports a source that references an undefined configured artifact key.
|
||||||
|
type UnknownConfiguredArtifactError struct {
|
||||||
|
ConfiguredKey string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *UnknownConfiguredArtifactError) Error() string {
|
||||||
|
return fmt.Sprintf("references unknown artifact %q", e.ConfiguredKey)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ConfiguredSourceID converts a configured artifact key into source id form.
|
||||||
|
func ConfiguredSourceID(key string) string {
|
||||||
|
return configuredSourcePrefix + strings.TrimSpace(key)
|
||||||
|
}
|
||||||
|
|
||||||
|
// PreviousSessionSourceID converts a configured artifact key into previous-session source id form.
|
||||||
|
func PreviousSessionSourceID(key string) string {
|
||||||
|
return previousConfiguredSrcPrefix + strings.TrimSpace(key)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseConfiguredSource extracts configured key from narratio.artifact.<key>.
|
||||||
|
func ParseConfiguredSource(source string) (string, bool) {
|
||||||
|
matches := configuredSourceRE.FindStringSubmatch(strings.TrimSpace(source))
|
||||||
|
if len(matches) != 2 {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
return matches[1], true
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParsePreviousSessionSource extracts configured key from narratio.previous_session.artifact.<key>.
|
||||||
|
func ParsePreviousSessionSource(source string) (string, bool) {
|
||||||
|
matches := previousSourceRE.FindStringSubmatch(strings.TrimSpace(source))
|
||||||
|
if len(matches) != 2 {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
return matches[1], true
|
||||||
|
}
|
||||||
|
|
||||||
|
// ClassifySource classifies a source id as built-in, configured, or previous-session configured.
|
||||||
|
func ClassifySource(source string) (Source, error) {
|
||||||
|
trimmed := strings.TrimSpace(source)
|
||||||
|
if trimmed == "" {
|
||||||
|
return Source{}, fmt.Errorf("artifact source is required")
|
||||||
|
}
|
||||||
|
if _, ok := artifactmodel.LookupRuntimeTranscriptArtifact(trimmed); ok {
|
||||||
|
return Source{ID: trimmed, Kind: SourceKindBuiltIn}, nil
|
||||||
|
}
|
||||||
|
if trimmed == SourceBoundsSession {
|
||||||
|
return Source{ID: trimmed, Kind: SourceKindBuiltIn}, nil
|
||||||
|
}
|
||||||
|
if key, ok := ParseConfiguredSource(trimmed); ok {
|
||||||
|
return Source{ID: trimmed, Kind: SourceKindConfiguredArtifact, ConfiguredKey: key}, nil
|
||||||
|
}
|
||||||
|
if key, ok := ParsePreviousSessionSource(trimmed); ok {
|
||||||
|
return Source{ID: trimmed, Kind: SourceKindPreviousArtifact, ConfiguredKey: key}, nil
|
||||||
|
}
|
||||||
|
return Source{}, fmt.Errorf("unsupported artifact source %q", source)
|
||||||
|
}
|
||||||
|
|
||||||
|
// DescribeScriptoriumInputSource classifies one input source and returns descriptor
|
||||||
|
// metadata used by config validation, analyze input resolution, and previous-cache planning.
|
||||||
|
func DescribeScriptoriumInputSource(source string) (ScriptoriumInputSourceDescriptor, error) {
|
||||||
|
trimmed := strings.TrimSpace(source)
|
||||||
|
if trimmed == "" {
|
||||||
|
return ScriptoriumInputSourceDescriptor{}, ErrUnsupportedScriptoriumInputSource
|
||||||
|
}
|
||||||
|
if strings.HasPrefix(trimmed, "narratio.previous_session.artifact") {
|
||||||
|
descriptor, err := DescribePreviousSessionSource(trimmed)
|
||||||
|
if err != nil {
|
||||||
|
return ScriptoriumInputSourceDescriptor{}, err
|
||||||
|
}
|
||||||
|
return ScriptoriumInputSourceDescriptor{
|
||||||
|
Source: Source{
|
||||||
|
ID: descriptor.SourceID,
|
||||||
|
Kind: SourceKindPreviousArtifact,
|
||||||
|
ConfiguredKey: descriptor.ConfiguredKey,
|
||||||
|
},
|
||||||
|
PreviousSession: &descriptor,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
classified, err := ClassifySource(trimmed)
|
||||||
|
if err != nil {
|
||||||
|
return ScriptoriumInputSourceDescriptor{}, ErrUnsupportedScriptoriumInputSource
|
||||||
|
}
|
||||||
|
return ScriptoriumInputSourceDescriptor{Source: classified}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// DescribePreviousSessionSource validates a canonical previous-session source id
|
||||||
|
// and returns both previous and configured-source vocabulary descriptors.
|
||||||
|
func DescribePreviousSessionSource(source string) (PreviousSessionSourceDescriptor, error) {
|
||||||
|
configuredKey, ok := ParsePreviousSessionSource(source)
|
||||||
|
if !ok {
|
||||||
|
return PreviousSessionSourceDescriptor{}, ErrInvalidPreviousSessionSource
|
||||||
|
}
|
||||||
|
return PreviousSessionSourceDescriptor{
|
||||||
|
SourceID: PreviousSessionSourceID(configuredKey),
|
||||||
|
ConfiguredKey: configuredKey,
|
||||||
|
ConfiguredSourceID: ConfiguredSourceID(configuredKey),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// PreviousSessionSourceDescriptorForConfiguredKey derives a previous-session source descriptor
|
||||||
|
// from a configured artifact key.
|
||||||
|
func PreviousSessionSourceDescriptorForConfiguredKey(configuredKey string) (PreviousSessionSourceDescriptor, error) {
|
||||||
|
return DescribePreviousSessionSource(PreviousSessionSourceID(configuredKey))
|
||||||
|
}
|
||||||
|
|
||||||
|
// ValidateInputConfiguredReference checks that configured/previous-session sources
|
||||||
|
// reference configured artifacts known to the current Scriptorium config.
|
||||||
|
func ValidateInputConfiguredReference(
|
||||||
|
descriptor ScriptoriumInputSourceDescriptor,
|
||||||
|
configured map[string]struct{},
|
||||||
|
) error {
|
||||||
|
switch descriptor.Source.Kind {
|
||||||
|
case SourceKindConfiguredArtifact, SourceKindPreviousArtifact:
|
||||||
|
if _, ok := configured[descriptor.Source.ConfiguredKey]; !ok {
|
||||||
|
return &UnknownConfiguredArtifactError{ConfiguredKey: descriptor.Source.ConfiguredKey}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ValidatePublishSource validates that a source is publish-compatible and references a known configured artifact.
|
||||||
|
func ValidatePublishSource(source string, configured map[string]string) (Source, error) {
|
||||||
|
classified, err := ClassifySource(source)
|
||||||
|
if err != nil {
|
||||||
|
return Source{}, fmt.Errorf("must be a built-in source id or narratio.artifact.<name>")
|
||||||
|
}
|
||||||
|
if classified.Kind == SourceKindPreviousArtifact {
|
||||||
|
return Source{}, fmt.Errorf("must be a built-in source id or narratio.artifact.<name>")
|
||||||
|
}
|
||||||
|
if classified.Kind == SourceKindConfiguredArtifact {
|
||||||
|
if configured == nil {
|
||||||
|
return Source{}, fmt.Errorf("configured artifact %q is not defined in pipeline.scriptorium.artifacts", classified.ConfiguredKey)
|
||||||
|
}
|
||||||
|
if _, ok := configured[classified.ConfiguredKey]; !ok {
|
||||||
|
return Source{}, fmt.Errorf("configured artifact %q is not defined in pipeline.scriptorium.artifacts", classified.ConfiguredKey)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return classified, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// DeriveDefaultPublishedDestination returns the default publish destination for one source.
|
||||||
|
func DeriveDefaultPublishedDestination(source Source, configured map[string]string) (string, error) {
|
||||||
|
switch source.Kind {
|
||||||
|
case SourceKindBuiltIn:
|
||||||
|
if spec, ok := artifactmodel.LookupRuntimeTranscriptArtifact(source.ID); ok {
|
||||||
|
return pathsafe.NormalizeRelativeDestination(spec.CanonicalRelPath)
|
||||||
|
}
|
||||||
|
if source.ID == SourceBoundsSession {
|
||||||
|
return pathsafe.NormalizeRelativeDestination("artifacts/session_bounds.json")
|
||||||
|
}
|
||||||
|
return "", fmt.Errorf("unsupported built-in source %q", source.ID)
|
||||||
|
case SourceKindConfiguredArtifact:
|
||||||
|
if configured == nil {
|
||||||
|
return "", fmt.Errorf("configured artifact %q is not defined in pipeline.scriptorium.artifacts", source.ConfiguredKey)
|
||||||
|
}
|
||||||
|
outputPath, ok := configured[source.ConfiguredKey]
|
||||||
|
if !ok {
|
||||||
|
return "", fmt.Errorf("configured artifact %q is not defined in pipeline.scriptorium.artifacts", source.ConfiguredKey)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(outputPath) == "" {
|
||||||
|
return "", fmt.Errorf("pipeline.scriptorium.artifacts.%s.output_path is empty", source.ConfiguredKey)
|
||||||
|
}
|
||||||
|
return pathsafe.NormalizeRelativeDestination(outputPath)
|
||||||
|
default:
|
||||||
|
return "", fmt.Errorf("publish destination cannot be derived from source %q", source.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResolvePublishedDestination validates and normalizes an explicit destination,
|
||||||
|
// or derives one when omitted.
|
||||||
|
func ResolvePublishedDestination(sourceID, explicitDest string, configured map[string]string) (string, error) {
|
||||||
|
source, err := ValidatePublishSource(sourceID, configured)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(explicitDest) != "" {
|
||||||
|
return pathsafe.NormalizeRelativeDestination(explicitDest)
|
||||||
|
}
|
||||||
|
return DeriveDefaultPublishedDestination(source, configured)
|
||||||
|
}
|
||||||
200
internal/artifactpolicy/policy_test.go
Normal file
200
internal/artifactpolicy/policy_test.go
Normal file
@@ -0,0 +1,200 @@
|
|||||||
|
package artifactpolicy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestClassifySource(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
source string
|
||||||
|
wantKind SourceKind
|
||||||
|
wantKey string
|
||||||
|
wantErrLike string
|
||||||
|
}{
|
||||||
|
{name: "built in transcript", source: "narratio.transcript.final_trimmed", wantKind: SourceKindBuiltIn},
|
||||||
|
{name: "built in bounds", source: "narratio.bounds.session", wantKind: SourceKindBuiltIn},
|
||||||
|
{name: "configured artifact", source: "narratio.artifact.session_recap", wantKind: SourceKindConfiguredArtifact, wantKey: "session_recap"},
|
||||||
|
{name: "previous session configured", source: "narratio.previous_session.artifact.session_recap", wantKind: SourceKindPreviousArtifact, wantKey: "session_recap"},
|
||||||
|
{name: "unsupported", source: "narratio.unknown", wantErrLike: "unsupported artifact source"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got, err := ClassifySource(tt.source)
|
||||||
|
if tt.wantErrLike != "" {
|
||||||
|
if err == nil || !strings.Contains(err.Error(), tt.wantErrLike) {
|
||||||
|
t.Fatalf("ClassifySource() error = %v, want like %q", err, tt.wantErrLike)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ClassifySource() error = %v", err)
|
||||||
|
}
|
||||||
|
if got.Kind != tt.wantKind {
|
||||||
|
t.Fatalf("ClassifySource().Kind = %q, want %q", got.Kind, tt.wantKind)
|
||||||
|
}
|
||||||
|
if got.ConfiguredKey != tt.wantKey {
|
||||||
|
t.Fatalf("ClassifySource().ConfiguredKey = %q, want %q", got.ConfiguredKey, tt.wantKey)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatePublishSource(t *testing.T) {
|
||||||
|
configured := map[string]string{"session_recap": "artifacts/session_recap.md"}
|
||||||
|
if _, err := ValidatePublishSource("narratio.artifact.session_recap", configured); err != nil {
|
||||||
|
t.Fatalf("ValidatePublishSource(configured) error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := ValidatePublishSource("narratio.previous_session.artifact.session_recap", configured); err == nil {
|
||||||
|
t.Fatal("ValidatePublishSource(previous) error = nil, want error")
|
||||||
|
}
|
||||||
|
if _, err := ValidatePublishSource("narratio.artifact.missing", configured); err == nil {
|
||||||
|
t.Fatal("ValidatePublishSource(missing configured) error = nil, want error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePublishedDestination(t *testing.T) {
|
||||||
|
configured := map[string]string{"session_recap": "artifacts/session_recap.md"}
|
||||||
|
|
||||||
|
got, err := ResolvePublishedDestination("narratio.transcript.final_trimmed", "", configured)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePublishedDestination(built-in) error = %v", err)
|
||||||
|
}
|
||||||
|
if got != "transcripts/final.trimmed.json" {
|
||||||
|
t.Fatalf("built-in destination = %q, want transcripts/final.trimmed.json", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
got, err = ResolvePublishedDestination("narratio.transcript.final_markdown", "", configured)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePublishedDestination(markdown built-in) error = %v", err)
|
||||||
|
}
|
||||||
|
if got != "transcripts/final.md" {
|
||||||
|
t.Fatalf("markdown built-in destination = %q, want transcripts/final.md", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
got, err = ResolvePublishedDestination("narratio.artifact.session_recap", "", configured)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePublishedDestination(configured) error = %v", err)
|
||||||
|
}
|
||||||
|
if got != "artifacts/session_recap.md" {
|
||||||
|
t.Fatalf("configured destination = %q, want artifacts/session_recap.md", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
got, err = ResolvePublishedDestination("narratio.transcript.final_trimmed", "published/../published/final.json", configured)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolvePublishedDestination(explicit) error = %v", err)
|
||||||
|
}
|
||||||
|
if got != "published/final.json" {
|
||||||
|
t.Fatalf("explicit destination = %q, want published/final.json", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolvePublishedDestinationRejectsTraversal(t *testing.T) {
|
||||||
|
configured := map[string]string{"session_recap": "artifacts/session_recap.md"}
|
||||||
|
_, err := ResolvePublishedDestination("narratio.transcript.final_trimmed", "../escape.txt", configured)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("ResolvePublishedDestination() error = nil, want traversal rejection")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDescribeScriptoriumInputSource(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
source string
|
||||||
|
wantKind SourceKind
|
||||||
|
wantKey string
|
||||||
|
wantPrev bool
|
||||||
|
wantErr error
|
||||||
|
wantErrLike string
|
||||||
|
}{
|
||||||
|
{name: "built in", source: "narratio.transcript.final_trimmed", wantKind: SourceKindBuiltIn},
|
||||||
|
{name: "built in markdown", source: "narratio.transcript.final_markdown", wantKind: SourceKindBuiltIn},
|
||||||
|
{name: "configured", source: "narratio.artifact.session_recap", wantKind: SourceKindConfiguredArtifact, wantKey: "session_recap"},
|
||||||
|
{name: "previous", source: "narratio.previous_session.artifact.session_recap", wantKind: SourceKindPreviousArtifact, wantKey: "session_recap", wantPrev: true},
|
||||||
|
{name: "invalid previous", source: "narratio.previous_session.artifact.", wantErr: ErrInvalidPreviousSessionSource},
|
||||||
|
{name: "unsupported", source: "narratio.unknown", wantErr: ErrUnsupportedScriptoriumInputSource},
|
||||||
|
{name: "empty", source: " ", wantErr: ErrUnsupportedScriptoriumInputSource},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got, err := DescribeScriptoriumInputSource(tt.source)
|
||||||
|
if tt.wantErr != nil {
|
||||||
|
if !errors.Is(err, tt.wantErr) {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource() error = %v, want %v", err, tt.wantErr)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if tt.wantErrLike != "" {
|
||||||
|
if err == nil || !strings.Contains(err.Error(), tt.wantErrLike) {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource() error = %v, want like %q", err, tt.wantErrLike)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource() error = %v", err)
|
||||||
|
}
|
||||||
|
if got.Source.Kind != tt.wantKind {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource().Source.Kind = %q, want %q", got.Source.Kind, tt.wantKind)
|
||||||
|
}
|
||||||
|
if got.Source.ConfiguredKey != tt.wantKey {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource().Source.ConfiguredKey = %q, want %q", got.Source.ConfiguredKey, tt.wantKey)
|
||||||
|
}
|
||||||
|
if tt.wantPrev && got.PreviousSession == nil {
|
||||||
|
t.Fatal("DescribeScriptoriumInputSource().PreviousSession = nil, want descriptor")
|
||||||
|
}
|
||||||
|
if !tt.wantPrev && got.PreviousSession != nil {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource().PreviousSession = %#v, want nil", got.PreviousSession)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateInputConfiguredReference(t *testing.T) {
|
||||||
|
configured := map[string]struct{}{"session_recap": {}}
|
||||||
|
|
||||||
|
desc, err := DescribeScriptoriumInputSource("narratio.artifact.session_recap")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource(configured) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := ValidateInputConfiguredReference(desc, configured); err != nil {
|
||||||
|
t.Fatalf("ValidateInputConfiguredReference(configured) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
prevDesc, err := DescribeScriptoriumInputSource("narratio.previous_session.artifact.session_recap")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource(previous) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := ValidateInputConfiguredReference(prevDesc, configured); err != nil {
|
||||||
|
t.Fatalf("ValidateInputConfiguredReference(previous) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
missingDesc, err := DescribeScriptoriumInputSource("narratio.artifact.quest_log")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DescribeScriptoriumInputSource(missing configured) error = %v", err)
|
||||||
|
}
|
||||||
|
err = ValidateInputConfiguredReference(missingDesc, configured)
|
||||||
|
var unknown *UnknownConfiguredArtifactError
|
||||||
|
if !errors.As(err, &unknown) || unknown.ConfiguredKey != "quest_log" {
|
||||||
|
t.Fatalf("ValidateInputConfiguredReference(missing configured) error = %v, want UnknownConfiguredArtifactError(quest_log)", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPreviousSessionSourceDescriptorForConfiguredKey(t *testing.T) {
|
||||||
|
got, err := PreviousSessionSourceDescriptorForConfiguredKey("session_recap")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("PreviousSessionSourceDescriptorForConfiguredKey() error = %v", err)
|
||||||
|
}
|
||||||
|
if got.SourceID != "narratio.previous_session.artifact.session_recap" {
|
||||||
|
t.Fatalf("SourceID = %q, want narratio.previous_session.artifact.session_recap", got.SourceID)
|
||||||
|
}
|
||||||
|
if got.ConfiguredSourceID != "narratio.artifact.session_recap" {
|
||||||
|
t.Fatalf("ConfiguredSourceID = %q, want narratio.artifact.session_recap", got.ConfiguredSourceID)
|
||||||
|
}
|
||||||
|
if got.ConfiguredKey != "session_recap" {
|
||||||
|
t.Fatalf("ConfiguredKey = %q, want session_recap", got.ConfiguredKey)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -6,42 +6,46 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"regexp"
|
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
ArtifactTranscriptBase = artifactmodel.SourceTranscriptBase
|
ArtifactTranscriptBase = artifactmodel.SourceTranscriptBase
|
||||||
ArtifactTranscriptPolished = artifactmodel.SourceTranscriptPolished
|
ArtifactTranscriptPolished = artifactmodel.SourceTranscriptPolished
|
||||||
ArtifactTranscriptFinal = artifactmodel.SourceTranscriptFinal
|
ArtifactTranscriptFinal = artifactmodel.SourceTranscriptFinal
|
||||||
ArtifactTranscriptFinalTrimmed = artifactmodel.SourceTranscriptFinalTrimmed
|
ArtifactTranscriptFinalTrimmed = artifactmodel.SourceTranscriptFinalTrimmed
|
||||||
ArtifactBoundsSession = "narratio.bounds.session"
|
ArtifactTranscriptFinalMarkdown = artifactmodel.SourceTranscriptFinalMarkdown
|
||||||
|
ArtifactTranscriptFinalTrimmedMarkdown = artifactmodel.SourceTranscriptFinalTrimmedMarkdown
|
||||||
|
ArtifactBoundsSession = "narratio.bounds.session"
|
||||||
|
|
||||||
ArtifactProvenancePreviousCacheManifestInput = "manifest.inputs.previous_cache"
|
ArtifactProvenancePreviousCacheManifestInput = "manifest.inputs.previous_cache"
|
||||||
ArtifactProvenancePreviousCacheFilesystem = "current_session.previous_cache"
|
ArtifactProvenancePreviousCacheFilesystem = "current_session.previous_cache"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
TranscriptPathBase = artifactmodel.TranscriptPathBase
|
TranscriptPathBase = artifactmodel.TranscriptPathBase
|
||||||
TranscriptPathPolished = artifactmodel.TranscriptPathPolished
|
TranscriptPathPolished = artifactmodel.TranscriptPathPolished
|
||||||
TranscriptPathFinal = artifactmodel.TranscriptPathFinal
|
TranscriptPathFinal = artifactmodel.TranscriptPathFinal
|
||||||
TranscriptPathFinalTrimmed = artifactmodel.TranscriptPathFinalTrimmed
|
TranscriptPathFinalTrimmed = artifactmodel.TranscriptPathFinalTrimmed
|
||||||
|
TranscriptPathFinalMarkdown = artifactmodel.TranscriptPathFinalMarkdown
|
||||||
|
TranscriptPathFinalTrimmedMarkdown = artifactmodel.TranscriptPathFinalTrimmedMarkdown
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
TranscriptOutputKindBase = artifactmodel.TranscriptOutputKindBase
|
TranscriptOutputKindBase = artifactmodel.TranscriptOutputKindBase
|
||||||
TranscriptOutputKindPolished = artifactmodel.TranscriptOutputKindPolished
|
TranscriptOutputKindPolished = artifactmodel.TranscriptOutputKindPolished
|
||||||
TranscriptOutputKindFinal = artifactmodel.TranscriptOutputKindFinal
|
TranscriptOutputKindFinal = artifactmodel.TranscriptOutputKindFinal
|
||||||
TranscriptOutputKindFinalTrimmed = artifactmodel.TranscriptOutputKindFinalTrimmed
|
TranscriptOutputKindFinalTrimmed = artifactmodel.TranscriptOutputKindFinalTrimmed
|
||||||
|
TranscriptOutputKindFinalMarkdown = artifactmodel.TranscriptOutputKindFinalMarkdown
|
||||||
|
TranscriptOutputKindFinalTrimmedMarkdown = artifactmodel.TranscriptOutputKindFinalTrimmedMarkdown
|
||||||
)
|
)
|
||||||
|
|
||||||
// ErrSessionArtifactNotFound is returned when no readable artifact exists for a known ID.
|
// ErrSessionArtifactNotFound is returned when no readable artifact exists for a known ID.
|
||||||
var ErrSessionArtifactNotFound = errors.New("session artifact not found")
|
var ErrSessionArtifactNotFound = errors.New("session artifact not found")
|
||||||
var configuredArtifactSourceRE = regexp.MustCompile(`^narratio\.artifact\.([a-z][a-z0-9_]*)$`)
|
|
||||||
var previousSessionArtifactSourceRE = regexp.MustCompile(`^narratio\.previous_session\.artifact\.([a-z][a-z0-9_]*)$`)
|
|
||||||
|
|
||||||
type artifactContentKind string
|
type artifactContentKind string
|
||||||
|
|
||||||
@@ -69,7 +73,7 @@ func buildArtifactRegistry() map[string]artifactSpec {
|
|||||||
CanonicalRelPath: transcript.CanonicalRelPath,
|
CanonicalRelPath: transcript.CanonicalRelPath,
|
||||||
ProducerStage: transcript.ProducerStage,
|
ProducerStage: transcript.ProducerStage,
|
||||||
OutputKind: transcript.OutputKind,
|
OutputKind: transcript.OutputKind,
|
||||||
ContentKind: contentTranscriptJSON,
|
ContentKind: transcriptContentKind(transcript),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
registry[ArtifactBoundsSession] = artifactSpec{
|
registry[ArtifactBoundsSession] = artifactSpec{
|
||||||
@@ -82,6 +86,15 @@ func buildArtifactRegistry() map[string]artifactSpec {
|
|||||||
return registry
|
return registry
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func transcriptContentKind(transcript TranscriptArtifactSpec) artifactContentKind {
|
||||||
|
switch transcript.SourceID {
|
||||||
|
case ArtifactTranscriptFinalMarkdown, ArtifactTranscriptFinalTrimmedMarkdown:
|
||||||
|
return contentText
|
||||||
|
default:
|
||||||
|
return contentTranscriptJSON
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ResolvedSessionArtifact describes one session-level artifact lookup result.
|
// ResolvedSessionArtifact describes one session-level artifact lookup result.
|
||||||
type ResolvedSessionArtifact struct {
|
type ResolvedSessionArtifact struct {
|
||||||
ID string
|
ID string
|
||||||
@@ -107,43 +120,39 @@ func (e *SessionArtifactNotFoundError) Unwrap() error {
|
|||||||
|
|
||||||
// NormalizeSessionArtifactSource validates canonical artifact IDs.
|
// NormalizeSessionArtifactSource validates canonical artifact IDs.
|
||||||
func NormalizeSessionArtifactSource(source string) (string, error) {
|
func NormalizeSessionArtifactSource(source string) (string, error) {
|
||||||
normalized := strings.TrimSpace(source)
|
classified, err := artifactpolicy.ClassifySource(source)
|
||||||
if normalized == "" {
|
if err != nil {
|
||||||
return "", fmt.Errorf("artifact source is required")
|
|
||||||
}
|
|
||||||
if _, ok := artifactRegistry[normalized]; !ok {
|
|
||||||
return "", fmt.Errorf("unsupported artifact source %q", source)
|
return "", fmt.Errorf("unsupported artifact source %q", source)
|
||||||
}
|
}
|
||||||
return normalized, nil
|
if classified.Kind != artifactpolicy.SourceKindBuiltIn {
|
||||||
|
return "", fmt.Errorf("unsupported artifact source %q", source)
|
||||||
|
}
|
||||||
|
if _, ok := artifactRegistry[classified.ID]; !ok {
|
||||||
|
return "", fmt.Errorf("unsupported artifact source %q", source)
|
||||||
|
}
|
||||||
|
return classified.ID, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// IsConfiguredArtifactSource returns true when source is narratio.artifact.<name>.
|
// IsConfiguredArtifactSource returns true when source is narratio.artifact.<name>.
|
||||||
func IsConfiguredArtifactSource(source string) bool {
|
func IsConfiguredArtifactSource(source string) bool {
|
||||||
return configuredArtifactSourceRE.MatchString(strings.TrimSpace(source))
|
_, ok := artifactpolicy.ParseConfiguredSource(source)
|
||||||
|
return ok
|
||||||
}
|
}
|
||||||
|
|
||||||
// ConfiguredArtifactName extracts <name> from narratio.artifact.<name>.
|
// ConfiguredArtifactName extracts <name> from narratio.artifact.<name>.
|
||||||
func ConfiguredArtifactName(source string) (string, bool) {
|
func ConfiguredArtifactName(source string) (string, bool) {
|
||||||
matches := configuredArtifactSourceRE.FindStringSubmatch(strings.TrimSpace(source))
|
return artifactpolicy.ParseConfiguredSource(source)
|
||||||
if len(matches) != 2 {
|
|
||||||
return "", false
|
|
||||||
}
|
|
||||||
return matches[1], true
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// IsPreviousSessionArtifactSource returns true when source is narratio.previous_session.artifact.<name>.
|
// IsPreviousSessionArtifactSource returns true when source is narratio.previous_session.artifact.<name>.
|
||||||
func IsPreviousSessionArtifactSource(source string) bool {
|
func IsPreviousSessionArtifactSource(source string) bool {
|
||||||
_, ok := PreviousSessionArtifactName(source)
|
_, ok := artifactpolicy.ParsePreviousSessionSource(source)
|
||||||
return ok
|
return ok
|
||||||
}
|
}
|
||||||
|
|
||||||
// PreviousSessionArtifactName extracts <name> from narratio.previous_session.artifact.<name>.
|
// PreviousSessionArtifactName extracts <name> from narratio.previous_session.artifact.<name>.
|
||||||
func PreviousSessionArtifactName(source string) (string, bool) {
|
func PreviousSessionArtifactName(source string) (string, bool) {
|
||||||
matches := previousSessionArtifactSourceRE.FindStringSubmatch(strings.TrimSpace(source))
|
return artifactpolicy.ParsePreviousSessionSource(source)
|
||||||
if len(matches) != 2 {
|
|
||||||
return "", false
|
|
||||||
}
|
|
||||||
return matches[1], true
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ResolveSessionArtifact resolves a symbolic source to a readable local session artifact path.
|
// ResolveSessionArtifact resolves a symbolic source to a readable local session artifact path.
|
||||||
|
|||||||
@@ -211,6 +211,29 @@ func TestResolveSessionArtifactFallsBackToCanonicalPath(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestResolveSessionArtifactFallsBackToCanonicalMarkdownPath(t *testing.T) {
|
||||||
|
workspace := t.TempDir()
|
||||||
|
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||||
|
canonicalPath := filepath.Join(paths.TranscriptsDir, "final.md")
|
||||||
|
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll() error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(canonicalPath, []byte("# Final transcript\n"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
resolved, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptFinalMarkdown)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolveSessionArtifact() error = %v", err)
|
||||||
|
}
|
||||||
|
if resolved.Path != canonicalPath {
|
||||||
|
t.Fatalf("resolved path = %q, want %q", resolved.Path, canonicalPath)
|
||||||
|
}
|
||||||
|
if resolved.Provenance != "fallback.canonical_path" {
|
||||||
|
t.Fatalf("provenance = %q, want fallback.canonical_path", resolved.Provenance)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestResolveSessionArtifactMissingReturnsTypedError(t *testing.T) {
|
func TestResolveSessionArtifactMissingReturnsTypedError(t *testing.T) {
|
||||||
workspace := t.TempDir()
|
workspace := t.TempDir()
|
||||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||||
@@ -244,6 +267,26 @@ func TestResolveSessionArtifactValidatesTranscriptShape(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestResolveSessionArtifactRejectsEmptyMarkdownContent(t *testing.T) {
|
||||||
|
workspace := t.TempDir()
|
||||||
|
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||||
|
canonicalPath := filepath.Join(paths.TranscriptsDir, "final.trimmed.md")
|
||||||
|
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll() error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(canonicalPath, []byte{}, 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptFinalTrimmedMarkdown)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error, got nil")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "file is empty") {
|
||||||
|
t.Fatalf("error = %q, want empty file validation", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestResolveSessionArtifactWithCatalogBuiltInBehaviorUnchanged(t *testing.T) {
|
func TestResolveSessionArtifactWithCatalogBuiltInBehaviorUnchanged(t *testing.T) {
|
||||||
workspace := t.TempDir()
|
workspace := t.TempDir()
|
||||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||||
|
|||||||
@@ -4,6 +4,8 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
@@ -47,7 +49,7 @@ func NewArtifactCatalog() *ArtifactCatalog {
|
|||||||
|
|
||||||
// ConfiguredArtifactSourceID converts a configured artifact key into canonical source ID.
|
// ConfiguredArtifactSourceID converts a configured artifact key into canonical source ID.
|
||||||
func ConfiguredArtifactSourceID(key string) string {
|
func ConfiguredArtifactSourceID(key string) string {
|
||||||
return "narratio.artifact." + strings.TrimSpace(key)
|
return artifactpolicy.ConfiguredSourceID(key)
|
||||||
}
|
}
|
||||||
|
|
||||||
// RegisterBuiltIns registers built-in source definitions used by runtime artifact resolution.
|
// RegisterBuiltIns registers built-in source definitions used by runtime artifact resolution.
|
||||||
@@ -214,11 +216,10 @@ func (c *ArtifactCatalog) addEntry(entry CatalogEntry) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func runtimeBuiltInArtifactIDs() []string {
|
func runtimeBuiltInArtifactIDs() []string {
|
||||||
return []string{
|
ids := make([]string, 0, len(RuntimeTranscriptArtifacts())+1)
|
||||||
ArtifactTranscriptBase,
|
for _, transcript := range RuntimeTranscriptArtifacts() {
|
||||||
ArtifactTranscriptPolished,
|
ids = append(ids, transcript.SourceID)
|
||||||
ArtifactTranscriptFinal,
|
|
||||||
ArtifactTranscriptFinalTrimmed,
|
|
||||||
ArtifactBoundsSession,
|
|
||||||
}
|
}
|
||||||
|
ids = append(ids, ArtifactBoundsSession)
|
||||||
|
return ids
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -23,6 +23,26 @@ func TestArtifactCatalogRegisterBuiltInsAndLookup(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestArtifactCatalogRegisterBuiltInsIncludesMarkdownSources(t *testing.T) {
|
||||||
|
catalog := NewArtifactCatalog()
|
||||||
|
if err := catalog.RegisterBuiltIns(); err != nil {
|
||||||
|
t.Fatalf("RegisterBuiltIns() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, sourceID := range []string{
|
||||||
|
ArtifactTranscriptFinalMarkdown,
|
||||||
|
ArtifactTranscriptFinalTrimmedMarkdown,
|
||||||
|
} {
|
||||||
|
entry, ok := catalog.Lookup(sourceID)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("Lookup(%q) ok=false, want true", sourceID)
|
||||||
|
}
|
||||||
|
if !entry.Planned {
|
||||||
|
t.Fatalf("%s planned=false, want true", sourceID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestArtifactCatalogRegisterConfiguredArtifactsDefaultsToEnabled(t *testing.T) {
|
func TestArtifactCatalogRegisterConfiguredArtifactsDefaultsToEnabled(t *testing.T) {
|
||||||
catalog := NewArtifactCatalog()
|
catalog := NewArtifactCatalog()
|
||||||
if err := catalog.RegisterConfiguredArtifacts(map[string]ConfiguredArtifactDefinition{
|
if err := catalog.RegisterConfiguredArtifacts(map[string]ConfiguredArtifactDefinition{
|
||||||
|
|||||||
203
internal/artifacts/current_state.go
Normal file
203
internal/artifacts/current_state.go
Normal file
@@ -0,0 +1,203 @@
|
|||||||
|
package artifacts
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
ErrCurrentRunPointerMissing = errors.New("current run pointer missing")
|
||||||
|
ErrCurrentManifestMissing = errors.New("current manifest missing")
|
||||||
|
)
|
||||||
|
|
||||||
|
type CurrentRunPointerMissingError struct {
|
||||||
|
Key string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *CurrentRunPointerMissingError) Error() string {
|
||||||
|
return fmt.Sprintf("%s: %q", ErrCurrentRunPointerMissing, e.Key)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *CurrentRunPointerMissingError) Unwrap() error {
|
||||||
|
return ErrCurrentRunPointerMissing
|
||||||
|
}
|
||||||
|
|
||||||
|
type CurrentManifestMissingError struct {
|
||||||
|
Key string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *CurrentManifestMissingError) Error() string {
|
||||||
|
return fmt.Sprintf("%s: %q", ErrCurrentManifestMissing, e.Key)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *CurrentManifestMissingError) Unwrap() error {
|
||||||
|
return ErrCurrentManifestMissing
|
||||||
|
}
|
||||||
|
|
||||||
|
type CurrentState struct {
|
||||||
|
SessionPrefix string
|
||||||
|
CurrentRunIDKey string
|
||||||
|
CurrentManifestKey string
|
||||||
|
RunID string
|
||||||
|
Manifest *manifest.Manifest
|
||||||
|
}
|
||||||
|
|
||||||
|
type CurrentStateValidation struct {
|
||||||
|
ExpectedCampaign string
|
||||||
|
ExpectedSessionID string
|
||||||
|
ExpectedRunID string
|
||||||
|
ValidateRunID bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func LoadCurrentRunPointer(ctx context.Context, store storage.ObjectStore, currentRunIDKey string) (string, error) {
|
||||||
|
if store == nil {
|
||||||
|
return "", fmt.Errorf("object store is required")
|
||||||
|
}
|
||||||
|
key := strings.TrimSpace(currentRunIDKey)
|
||||||
|
if key == "" {
|
||||||
|
return "", fmt.Errorf("current run pointer key is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
exists, err := store.Exists(ctx, key)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("check current run pointer %q: %w", key, err)
|
||||||
|
}
|
||||||
|
if !exists {
|
||||||
|
return "", &CurrentRunPointerMissingError{Key: key}
|
||||||
|
}
|
||||||
|
|
||||||
|
localPath, err := storage.DownloadObjectToTemp(ctx, store, key, "narratio-current-run-id-*.txt")
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("download current run pointer %q: %w", key, err)
|
||||||
|
}
|
||||||
|
defer func() { _ = os.Remove(localPath) }()
|
||||||
|
|
||||||
|
data, err := os.ReadFile(localPath)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("read downloaded current run pointer %q: %w", key, err)
|
||||||
|
}
|
||||||
|
runID := strings.TrimSpace(string(data))
|
||||||
|
if runID == "" {
|
||||||
|
return "", fmt.Errorf("current run pointer %q is empty", key)
|
||||||
|
}
|
||||||
|
return runID, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func LoadCurrentManifest(ctx context.Context, store storage.ObjectStore, currentManifestKey string) (*manifest.Manifest, error) {
|
||||||
|
if store == nil {
|
||||||
|
return nil, fmt.Errorf("object store is required")
|
||||||
|
}
|
||||||
|
key := strings.TrimSpace(currentManifestKey)
|
||||||
|
if key == "" {
|
||||||
|
return nil, fmt.Errorf("current manifest key is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
exists, err := store.Exists(ctx, key)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("check current manifest %q: %w", key, err)
|
||||||
|
}
|
||||||
|
if !exists {
|
||||||
|
return nil, &CurrentManifestMissingError{Key: key}
|
||||||
|
}
|
||||||
|
|
||||||
|
localPath, err := storage.DownloadObjectToTemp(ctx, store, key, "narratio-current-manifest-*.json")
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("download current manifest %q: %w", key, err)
|
||||||
|
}
|
||||||
|
defer func() { _ = os.Remove(localPath) }()
|
||||||
|
|
||||||
|
manifestStore := &manifest.LocalStore{}
|
||||||
|
m, err := manifestStore.Load(ctx, localPath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("current manifest decode failed: %w", err)
|
||||||
|
}
|
||||||
|
return m, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func LoadCurrentState(
|
||||||
|
ctx context.Context,
|
||||||
|
store storage.ObjectStore,
|
||||||
|
sessionPrefix string,
|
||||||
|
validation CurrentStateValidation,
|
||||||
|
) (*CurrentState, error) {
|
||||||
|
prefix := strings.TrimSpace(sessionPrefix)
|
||||||
|
if prefix == "" {
|
||||||
|
return nil, fmt.Errorf("session prefix is required")
|
||||||
|
}
|
||||||
|
currentManifestKey, currentRunIDKey := ResolveCurrentStateKeys(prefix)
|
||||||
|
runID, err := LoadCurrentRunPointer(ctx, store, currentRunIDKey)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
m, err := LoadCurrentManifest(ctx, store, currentManifestKey)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
state := &CurrentState{
|
||||||
|
SessionPrefix: prefix,
|
||||||
|
CurrentRunIDKey: currentRunIDKey,
|
||||||
|
CurrentManifestKey: currentManifestKey,
|
||||||
|
RunID: runID,
|
||||||
|
Manifest: m,
|
||||||
|
}
|
||||||
|
if err := ValidateCurrentStateIdentity(state, validation); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return state, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func ValidateCurrentStateIdentity(state *CurrentState, validation CurrentStateValidation) error {
|
||||||
|
if state == nil || state.Manifest == nil {
|
||||||
|
return fmt.Errorf("current state with manifest is required")
|
||||||
|
}
|
||||||
|
expectedSessionID := strings.TrimSpace(validation.ExpectedSessionID)
|
||||||
|
expectedCampaign := strings.TrimSpace(validation.ExpectedCampaign)
|
||||||
|
expectedRunID := strings.TrimSpace(validation.ExpectedRunID)
|
||||||
|
manifestSessionID := strings.TrimSpace(state.Manifest.SessionID)
|
||||||
|
manifestCampaign := strings.TrimSpace(state.Manifest.Campaign)
|
||||||
|
manifestRunID := strings.TrimSpace(state.Manifest.RunID)
|
||||||
|
|
||||||
|
if expectedSessionID != "" && manifestSessionID != expectedSessionID {
|
||||||
|
return fmt.Errorf(
|
||||||
|
"current manifest session_id %q does not match expected session_id %q",
|
||||||
|
manifestSessionID,
|
||||||
|
expectedSessionID,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
if expectedCampaign != "" {
|
||||||
|
if manifestCampaign == "" {
|
||||||
|
return fmt.Errorf("current manifest campaign is required")
|
||||||
|
}
|
||||||
|
if manifestCampaign != expectedCampaign {
|
||||||
|
return fmt.Errorf(
|
||||||
|
"current manifest campaign %q does not match expected campaign %q",
|
||||||
|
manifestCampaign,
|
||||||
|
expectedCampaign,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if expectedRunID == "" && validation.ValidateRunID {
|
||||||
|
expectedRunID = strings.TrimSpace(state.RunID)
|
||||||
|
}
|
||||||
|
if expectedRunID != "" {
|
||||||
|
if manifestRunID == "" {
|
||||||
|
return fmt.Errorf("current manifest run_id is required")
|
||||||
|
}
|
||||||
|
if manifestRunID != expectedRunID {
|
||||||
|
return fmt.Errorf(
|
||||||
|
"current run pointer %q references run %q but current manifest run_id is %q",
|
||||||
|
state.CurrentRunIDKey,
|
||||||
|
expectedRunID,
|
||||||
|
manifestRunID,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user