Compare commits
65 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 083c01cfa0 | |||
| 2937696024 | |||
| b817a5b772 | |||
| 3022f20beb | |||
| ca1ded1821 | |||
| 3752f3ed28 | |||
| 870c2d69d5 | |||
| 135407ba7c | |||
| 228c348e42 | |||
| a813bd5a50 | |||
| d8f58dce31 | |||
| 7111edeca4 | |||
| 3aae4bbb12 | |||
| b29d8eeb50 | |||
| dffb432537 | |||
| 2dd38c7913 | |||
| bc2ade38d9 | |||
| 5be831eb13 | |||
| cae4d99a89 | |||
| e09dc0512d | |||
| ae82bc1ce0 | |||
| 01eb7aa1aa | |||
| 2ca700195c | |||
| 2b08c34539 | |||
| 79f1fc1e09 | |||
| 9c753270bd | |||
| b907cb01aa | |||
| 7824afd4a5 | |||
| 2a4e1e912c | |||
| dd03c09d75 | |||
| 5bc8e8683f | |||
| 648001a8fe | |||
| 6684774f52 | |||
| f3b63bd5e5 | |||
| 23d6470b0f | |||
| 128449040f | |||
| 02ab106ade | |||
| c128970f58 | |||
| d001baa660 | |||
| c5c35cd3b4 | |||
| 574b1cde6c | |||
| ebb21b9201 | |||
| 958f446387 | |||
| 86caf4b222 | |||
| e38ed8ba97 | |||
| 3e79cf4724 | |||
| 859ae1ae10 | |||
| c63ecbab32 | |||
| 8480b74283 | |||
| 087869f7fa | |||
| 2b2a314d65 | |||
| 08b0f4edc5 | |||
| 571a289296 | |||
| 9f80635b42 | |||
| 11a3e174b6 | |||
| 9c5e5d6dc1 | |||
| c4e87f58c7 | |||
| 37daab7857 | |||
| 2356688cb9 | |||
| 1054b64d9f | |||
| 01fb02426c | |||
| 7dc79e052f | |||
| cb525c0f72 | |||
| 622677d038 | |||
| 550288e008 |
465
README.md
465
README.md
@@ -1,461 +1,22 @@
|
||||
# narratio
|
||||
|
||||
`narratio` is a Go orchestration application for processing D&D session audio into transcripts and generated artifacts.
|
||||
Narratio is a Go orchestration application that turns D&D session audio into polished transcripts and generated session artifacts.
|
||||
|
||||
## Current Implementation
|
||||
|
||||
Implemented now:
|
||||
|
||||
- strict config loading/validation (`pipeline.yml` and `session.yml`)
|
||||
- local workspace/session layout, locking, and manifest persistence
|
||||
- resumable stage control (`run`, `plan`, `resume`, `run-stage`, `status`)
|
||||
- real `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` stages
|
||||
- real WhisperX, Seriatim, and Audita adapters
|
||||
- real Scriptorium subprocess adapter
|
||||
- optional Scriptorium render diagnostics (`render_debug`)
|
||||
|
||||
Not implemented yet:
|
||||
|
||||
- `notify` stage behavior
|
||||
- additional analyze artifacts beyond `session_recap`
|
||||
- generic DAG orchestration
|
||||
|
||||
## Config Files
|
||||
|
||||
Narratio expects two YAML files:
|
||||
|
||||
- `pipeline.yml`: pipeline/workspace settings
|
||||
- `session.yml`: per-session settings
|
||||
|
||||
Pipeline config lookup for CLI commands:
|
||||
|
||||
- if `--config <path>` is provided, Narratio uses that path
|
||||
- if `--config` is omitted, Narratio searches in this order:
|
||||
- `/usr/local/etc/narratio/pipeline.yml`
|
||||
- `/etc/narratio/pipeline.yml`
|
||||
|
||||
Session config lookup for CLI commands:
|
||||
|
||||
- if `--session <path>` is provided, Narratio uses that path
|
||||
- if `--session` is omitted, Narratio searches in this order:
|
||||
- `./session.yml`
|
||||
- `/usr/local/etc/narratio/session.yml`
|
||||
- `/etc/narratio/session.yml`
|
||||
|
||||
Session template support:
|
||||
|
||||
- Narratio renders `session.yml` templates before strict YAML decode.
|
||||
- `--session-id <value>` provides the `session_id` template variable.
|
||||
- Supported placeholder forms:
|
||||
- `{{session_id}}`
|
||||
- `{{ session_id }}`
|
||||
- unresolved template placeholders fail with a clear error.
|
||||
- strict YAML validation still runs after rendering.
|
||||
- concrete `session.yml` files without templates remain fully supported.
|
||||
|
||||
Optional secrets-from-files config:
|
||||
|
||||
- `pipeline.secrets.env_dir` may point to a directory of secret files
|
||||
- each top-level file with an env-var-style name is loaded as an environment variable:
|
||||
- file name = env var name
|
||||
- file contents = env var value (trailing newline/CRLF trimmed)
|
||||
- process environment wins: existing env vars are not overwritten
|
||||
- if configured, Narratio fails fast when `env_dir` is missing/unreadable
|
||||
- relative `env_dir` values resolve from Narratio’s current working directory
|
||||
|
||||
YAML decoding is strict (`KnownFields(true)`), so unknown fields fail fast.
|
||||
|
||||
Maintainer note: application defaults are centralized in [`internal/config/defaults.go`](internal/config/defaults.go).
|
||||
|
||||
## Storage And Archive Foundations
|
||||
|
||||
Narratio now includes configuration and path-model foundations for archive support, plus implemented prepare-stage S3 audio input.
|
||||
|
||||
Implemented foundations:
|
||||
|
||||
- `pipeline.storage.s3` config shape (`bucket`, `root_prefix`, `region`, `endpoint`, `force_path_style`, `access_key_id_env`, `secret_access_key_env`)
|
||||
- `pipeline.spool` config shape (`root`, `delete_audio_after_archive`)
|
||||
- `pipeline.archive` config shape (`enabled`, `upload_run`, `promote_artifacts`)
|
||||
- promotion-rule validation (`from`/`to` required, relative-only paths, traversal rejected)
|
||||
- `session.campaign` requirement for campaign-aware path construction
|
||||
- optional `session.inputs.audio_s3.prefix` modeling and prepare-stage S3 audio download
|
||||
- run ID generation and S3/local path helper foundations
|
||||
- manifest run/path identity fields
|
||||
|
||||
Current defaults:
|
||||
|
||||
- `pipeline.storage.s3.root_prefix`: `dnd`
|
||||
- `pipeline.storage.s3.access_key_id_env`: `OBJECT_STORAGE_KEY_ID`
|
||||
- `pipeline.storage.s3.secret_access_key_env`: `OBJECT_STORAGE_KEY`
|
||||
- `pipeline.workspace.cleanup_after_archive`: `false`
|
||||
- `pipeline.spool.root`: `/var/spool/narratio`
|
||||
- `pipeline.spool.delete_audio_after_archive`: `false`
|
||||
- `pipeline.archive.enabled`: `true`
|
||||
- `pipeline.archive.upload_run`: `true`
|
||||
- default `pipeline.archive.promote_artifacts`:
|
||||
- `transcripts/trimmed.json` -> `transcripts/trimmed.json` (`required: true`)
|
||||
- `artifacts/session_recap.md` -> `artifacts/session_recap.md` (`required: true`)
|
||||
|
||||
Current boundaries:
|
||||
|
||||
- local development audio (`audio_dir` / `audio_files`) still works
|
||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive
|
||||
- real S3-compatible backend now exists in the storage adapter package
|
||||
- storage backend tests use fake storage and do not require live S3
|
||||
- archive uploads successful run records under `runs/{run_id}/`
|
||||
- archive does not upload local audio by default
|
||||
- archive uploads promoted outputs to session-level keys using `archive.promote_artifacts`
|
||||
- archive uploads `current/manifest.json`
|
||||
- archive uploads `current/run_id.txt` last as the effective commit marker
|
||||
- required missing promotions fail archive
|
||||
- optional missing promotions are skipped and recorded
|
||||
- cleanup remains conservative and opt-in:
|
||||
- `pipeline.spool.delete_audio_after_archive: true` removes only the run-scoped spool audio directory after successful archive commit
|
||||
- `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir after successful archive commit
|
||||
- cleanup executes only after all selected stages for the command invocation succeed
|
||||
- cleanup does not run for failed, incomplete, skipped, or unarchived runs
|
||||
- local development `audio_dir`/`audio_files` source inputs are never deleted by spool cleanup
|
||||
- S3 credentials are resolved from configured env-var names when both are present; if either is missing, Narratio falls back to the AWS SDK default credential chain
|
||||
|
||||
S3 input details and current boundaries are documented in [docs/s3-audio-input.md](docs/s3-audio-input.md).
|
||||
|
||||
## Remote Storage Backend
|
||||
|
||||
Narratio includes an object-store backend layer for future prepare/archive work:
|
||||
|
||||
- `List(ctx, prefix)`
|
||||
- `Download(ctx, key, localPath)`
|
||||
- `Upload(ctx, localPath, key, opts)`
|
||||
- `Exists(ctx, key)`
|
||||
|
||||
Implemented backends:
|
||||
|
||||
- fake storage backend for deterministic tests
|
||||
- S3-compatible backend built from `pipeline.storage.s3`
|
||||
|
||||
Key invariant:
|
||||
|
||||
- callers pass full bucket-relative object keys
|
||||
- storage backends do not prepend `root_prefix` and do not infer session/campaign paths
|
||||
|
||||
Current boundary:
|
||||
|
||||
- `prepare` uses `List` + `Download` through the backend when `session.inputs.audio_s3` is configured
|
||||
- `archive` uses `Upload` through the backend for successful run-record uploads
|
||||
- `archive` also uses `Upload` for promotion writes and current pointers
|
||||
- no failed or incomplete runs are uploaded
|
||||
- local audio is not re-uploaded by default
|
||||
|
||||
Archive run-upload details and boundaries are documented in [docs/archive-storage.md](docs/archive-storage.md).
|
||||
|
||||
## Canonical Stage Order
|
||||
|
||||
1. `prepare`
|
||||
2. `transcribe`
|
||||
3. `merge`
|
||||
4. `polish`
|
||||
5. `normalize`
|
||||
6. `trim`
|
||||
7. `analyze`
|
||||
8. `archive`
|
||||
9. `notify`
|
||||
|
||||
## Transcript Tiers
|
||||
|
||||
- `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge
|
||||
- `transcripts/processed.json`: full raw Audita-polished transcript output
|
||||
- `transcripts/normalized.json`: Seriatim-normalized transcript from the normalize stage
|
||||
- `transcripts/trimmed.json`: gameplay-only normalized polished transcript from trim stage
|
||||
|
||||
## Seriatim Configuration
|
||||
|
||||
`pipeline.seriatim` configures the Seriatim subprocess adapter used by `merge`, `normalize`, and `trim`.
|
||||
|
||||
Minimal behavior:
|
||||
|
||||
- `pipeline.seriatim` may be omitted entirely.
|
||||
- when omitted, Narratio defaults to:
|
||||
- `binary: seriatim`
|
||||
- `timeout: 10m`
|
||||
- `output_schema: seriatim-intermediate`
|
||||
- `coalesce_gap: 3.0`
|
||||
- `report: true`
|
||||
|
||||
Optional overrides in `pipeline.seriatim` continue to work, including explicit binary paths and advanced `env` tuning values.
|
||||
|
||||
## Audita Configuration
|
||||
|
||||
`pipeline.audita` configures the real Audita subprocess adapter used by `polish`.
|
||||
|
||||
Minimal behavior:
|
||||
|
||||
- `pipeline.audita` may be omitted entirely.
|
||||
- when omitted, Narratio defaults to:
|
||||
- `binary: audita`
|
||||
- `timeout: 3h`
|
||||
- `report: true`
|
||||
|
||||
Optional:
|
||||
|
||||
- `llm_api_key_env` (when set, Narratio requires that env var and passes it to Audita as `AUDITA_LLM_API_KEY`)
|
||||
- `modules` override list (when empty/omitted, Narratio does not pass `--modules`)
|
||||
- `base_url` (when omitted, Narratio does not pass `--base-url`; Audita runtime defaults/config may apply)
|
||||
- `model` (when omitted, Narratio does not pass `--model`; Audita runtime defaults/config may apply)
|
||||
- `transcript_description`
|
||||
- `config_path`
|
||||
- `output_schema` (`bare-segments` or `audita-v1`)
|
||||
- `work_dir_retention` (`always`, `auto`, or `never`)
|
||||
- `total_llm_concurrency` (> 0 when provided)
|
||||
- `proposal_llm_concurrency` (> 0 when provided)
|
||||
- `validation_model`
|
||||
- `validation_llm_concurrency` (> 0 when provided)
|
||||
- `report` (defaults to `true`)
|
||||
|
||||
Narratio passes only configured optional Audita flags. Omitted optional values are left to Audita runtime defaults/config.
|
||||
|
||||
## Normalize Configuration
|
||||
|
||||
`pipeline.normalize` is optional. When omitted, Narratio defaults to:
|
||||
|
||||
- `output_path: transcripts/normalized.json`
|
||||
- `output_schema: seriatim-intermediate`
|
||||
- `report: true`
|
||||
|
||||
Allowed `normalize.output_schema` values:
|
||||
|
||||
- `seriatim-minimal`
|
||||
- `seriatim-intermediate`
|
||||
- `seriatim-full`
|
||||
|
||||
`normalize.output_path` is treated as session-workdir-relative when not absolute.
|
||||
|
||||
Normalize stage behavior summary:
|
||||
|
||||
- normalize runs after `polish` and before `trim`
|
||||
- normalize resolves `transcripts/processed.json`
|
||||
- normalize runs Seriatim `normalize` to produce `transcripts/normalized.json`
|
||||
- normalize diagnostics are written to:
|
||||
- `artifacts/seriatim.normalize.report.json` (when enabled)
|
||||
- `logs/seriatim.normalize.stdout.log`
|
||||
- `logs/seriatim.normalize.stderr.log`
|
||||
- `config/seriatim.normalize.generated.yml`
|
||||
|
||||
## Trim Configuration
|
||||
|
||||
`pipeline.trim` is optional. If omitted, no trim config is loaded. If `trim.enabled` is omitted, it defaults to `false`.
|
||||
|
||||
When `trim.enabled: true`:
|
||||
|
||||
- `trim.output_path` is required
|
||||
- `trim.bounds.prompt_id` is required
|
||||
- `trim.bounds.transcript_input_name` is required
|
||||
- `trim.bounds.output_path` is required
|
||||
- `trim.bounds.timeout` must be a valid Go duration when provided
|
||||
- `trim.bounds.render_debug: true` requires `trim.bounds.render_output_path`
|
||||
- `trim.bounds.profile_id` may be empty to use the prompt default profile
|
||||
|
||||
Trim paths are treated as session-workdir-relative when not absolute.
|
||||
|
||||
Example trim config:
|
||||
|
||||
```yaml
|
||||
trim:
|
||||
enabled: true
|
||||
output_path: "transcripts/trimmed.json"
|
||||
bounds:
|
||||
prompt_id: "dnd_session.bounds"
|
||||
profile_id: ""
|
||||
transcript_input_name: "transcript"
|
||||
output_path: "artifacts/session_bounds.json"
|
||||
timeout: "10m"
|
||||
render_debug: false
|
||||
render_output_path: "artifacts/session_bounds.render.json"
|
||||
seriatim:
|
||||
report: false
|
||||
```
|
||||
|
||||
Trim behavior summary:
|
||||
|
||||
- trim discovers and validates `transcripts/normalized.json`
|
||||
- trim uses Scriptorium bounds (`dnd_session.bounds` by example config) to produce `artifacts/session_bounds.json`
|
||||
- bounds IDs are validated against the same normalized transcript ID space that Seriatim trim will consume
|
||||
- trim converts bounds to Seriatim keep selector (for example `10-868`) and runs Seriatim trim
|
||||
- if trim is disabled, Narratio copies normalized transcript to trimmed transcript and records `trim_action=copy_disabled`
|
||||
|
||||
Trim outputs and diagnostics:
|
||||
|
||||
- `artifacts/session_bounds.json`
|
||||
- `transcripts/trimmed.json`
|
||||
- `logs/scriptorium.bounds.stdout.log`
|
||||
- `logs/scriptorium.bounds.stderr.log`
|
||||
- `config/scriptorium.bounds.generated.yml`
|
||||
- `logs/seriatim.trim.stdout.log`
|
||||
- `logs/seriatim.trim.stderr.log`
|
||||
- `config/seriatim.trim.generated.yml`
|
||||
- optional bounds render-debug outputs:
|
||||
- `artifacts/session_bounds.render.json`
|
||||
- `logs/scriptorium.bounds.render.stdout.log`
|
||||
- `logs/scriptorium.bounds.render.stderr.log`
|
||||
- `config/scriptorium.bounds.render.generated.yml`
|
||||
|
||||
Render-debug files are diagnostics and are not treated as canonical stage output artifact refs.
|
||||
|
||||
## Scriptorium Configuration
|
||||
|
||||
`pipeline.scriptorium` is optional. When present, Narratio validates and uses it for analyze-stage artifact generation.
|
||||
|
||||
Key points:
|
||||
|
||||
- `scriptorium.binary` defaults to `scriptorium` when section is present
|
||||
- `scriptorium.config_path` is optional
|
||||
- `scriptorium.timeout` defaults to `10m` when omitted
|
||||
- `scriptorium.render_debug` enables render diagnostics globally
|
||||
- artifacts are configured under `scriptorium.artifacts` (map shape supports multiple artifacts)
|
||||
- enabled artifacts require `prompt_id` and `output_path`
|
||||
- artifact `render_debug` may override global render setting
|
||||
- `vars` currently support boolean and string values
|
||||
|
||||
Example `session_recap` artifact definition:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
binary: "scriptorium"
|
||||
config_path: "/etc/scriptorium/config.yml"
|
||||
timeout: "10m"
|
||||
render_debug: false
|
||||
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: "dnd.session_recap"
|
||||
profile_id: "local-quality" # optional
|
||||
output_path: "artifacts/session_recap.md"
|
||||
timeout: "10m"
|
||||
# render_debug: true # optional per-artifact override
|
||||
|
||||
inputs:
|
||||
transcript:
|
||||
source: "trimmed_transcript"
|
||||
required: true
|
||||
|
||||
previous_recap:
|
||||
source: "previous_session_artifact"
|
||||
artifact: "session_recap"
|
||||
path: "" # optional; set when available
|
||||
required: false
|
||||
|
||||
vars:
|
||||
session_id: true
|
||||
session_date: true
|
||||
campaign_name: true
|
||||
previous_session_id: true
|
||||
output_kind: "session_recap"
|
||||
```
|
||||
|
||||
Prompt IDs and profile IDs are configuration values. They are not hardcoded in analyze-stage logic.
|
||||
|
||||
Do not put secrets in `pipeline.yml`. If API-key behavior is configured, use env var names only.
|
||||
|
||||
If `pipeline.secrets.env_dir` is configured, keep only references and secret files there; secret values are still not written to manifests, generated configs, or Narratio-managed logs.
|
||||
|
||||
## Scriptorium Runtime Behavior
|
||||
|
||||
Narratio integrates with Scriptorium through the public CLI subprocess contract:
|
||||
|
||||
- generation: `scriptorium run`
|
||||
- diagnostics/testing: `scriptorium render --format json` when `render_debug` is enabled
|
||||
|
||||
For the initial implementation, only `session_recap` generation is supported.
|
||||
|
||||
Analyze-stage session recap behavior:
|
||||
|
||||
- available transcript input sources for configured artifacts: `processed_transcript`, `normalized_transcript`, `trimmed_transcript`
|
||||
- session recap should use gameplay-only transcript input (`source: trimmed_transcript`)
|
||||
- Narratio resolves `trimmed_transcript` from trim manifest output (`transcript_trimmed`) or fallback `transcripts/trimmed.json`
|
||||
- Narratio resolves `normalized_transcript` from normalize manifest output (`transcript_normalized`) or fallback `transcripts/normalized.json`
|
||||
- missing trimmed transcript fails clearly and advises running trim stage first
|
||||
- `normalized_transcript` is the preferred full-transcript source for future table/meta-analysis artifacts
|
||||
- `processed_transcript` remains supported for advanced/debug use cases
|
||||
- optionally includes `previous_recap` when configured and resolvable
|
||||
- omits optional previous recap when unavailable
|
||||
- fails if required inputs are missing
|
||||
- validates output file exists and is non-empty
|
||||
|
||||
Expected session output paths:
|
||||
|
||||
- `artifacts/session_recap.md`
|
||||
- `logs/scriptorium.session_recap.stdout.log`
|
||||
- `logs/scriptorium.session_recap.stderr.log`
|
||||
- `config/scriptorium.session_recap.generated.yml`
|
||||
- `artifacts/session_recap.render.json` when render diagnostics are enabled
|
||||
|
||||
## Examples
|
||||
|
||||
Starter files:
|
||||
|
||||
- `examples/pipeline.minimal.yml`
|
||||
- `examples/pipeline.audita-overrides.yml`
|
||||
- `examples/session.minimal.yml`
|
||||
- `examples/session.template.yml`
|
||||
- `examples/speakers.yml`
|
||||
|
||||
## Commands
|
||||
|
||||
Run tests:
|
||||
It coordinates transcription, merge/polish/normalize/trim processing, artifact generation, archive publishing, and resumable run state in one operator workflow.
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Plan a run:
|
||||
This command requires discoverable `pipeline.yml` and `session.yml` files (or explicit `--config` and `--session` flags).
|
||||
|
||||
```bash
|
||||
go run ./cmd/narratio plan --session examples/session.minimal.yml
|
||||
```
|
||||
## Documentation
|
||||
|
||||
Use `--config <path>` to override default pipeline lookup when needed.
|
||||
|
||||
Run with a discoverable session template:
|
||||
|
||||
```bash
|
||||
go run ./cmd/narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Run full pipeline:
|
||||
|
||||
```bash
|
||||
go run ./cmd/narratio run --config examples/pipeline.minimal.yml --session examples/session.minimal.yml
|
||||
```
|
||||
|
||||
Run analyze only:
|
||||
|
||||
```bash
|
||||
go run ./cmd/narratio run-stage --config examples/pipeline.minimal.yml --session examples/session.minimal.yml analyze
|
||||
```
|
||||
|
||||
Resume with a template session ID:
|
||||
|
||||
```bash
|
||||
go run ./cmd/narratio resume --config examples/pipeline.minimal.yml --session examples/session.template.yml --session-id 2026-04-04
|
||||
```
|
||||
|
||||
## Operational Note
|
||||
|
||||
Checksum-based stale detection is not implemented yet.
|
||||
|
||||
If prepared inputs or prompt/runtime config change, rerun the appropriate upstream stages before relying on downstream artifacts.
|
||||
|
||||
Examples:
|
||||
|
||||
- glossary/autocorrect/speaker-context changes: rerun at least `merge`, `polish`, `normalize`, `trim`, and `analyze`
|
||||
- trim bounds prompt/profile/config changes: rerun at least `normalize`, `trim`, and `analyze`
|
||||
- session recap prompt/profile/input-source changes: rerun `analyze`
|
||||
|
||||
## Roadmap
|
||||
|
||||
Near-term roadmap:
|
||||
|
||||
- extend analyze to additional configured artifacts
|
||||
- support workflows where later artifacts consume earlier generated artifacts
|
||||
- keep orchestration explicit without a generic DAG engine
|
||||
- implement archive and notify backends
|
||||
- [Configuration](docs/config.md)
|
||||
- [CLI Reference](docs/cli.md)
|
||||
- [Operations and Recovery](docs/operations.md)
|
||||
- [Troubleshooting](docs/troubleshooting.md)
|
||||
- [Development Guide](docs/development.md)
|
||||
- [Architecture Principles](docs/architecture.md)
|
||||
- [Internal Component Contracts](docs/internal/README.md)
|
||||
- [Config Examples](examples/)
|
||||
|
||||
502
architecture.md
502
architecture.md
@@ -1,502 +0,0 @@
|
||||
# Narratio Architecture
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
`narratio` is a Go orchestrator for D&D session processing. It runs a stage-based local pipeline from audio input through transcript processing and artifact generation, with manifest-based skip/force/resume behavior.
|
||||
|
||||
Narratio integrates with Scriptorium through the **public CLI** (`scriptorium run` and `scriptorium render`) via synchronous subprocess execution.
|
||||
|
||||
## 2. Current Status
|
||||
|
||||
Implemented:
|
||||
|
||||
- strict `pipeline.yml` + `session.yml` loading with strict YAML field checking (`KnownFields(true)`)
|
||||
- local workspace/session layout, lock file handling, artifact path helpers, checksums, and atomic writes
|
||||
- manifest store and stage status transitions for resumable runs
|
||||
- real `prepare`, `transcribe`, `merge`, and `polish` stages
|
||||
- real WhisperX HTTP adapter
|
||||
- real Seriatim subprocess adapter
|
||||
- real Audita subprocess adapter
|
||||
- real Scriptorium subprocess adapter
|
||||
- real `normalize` stage producing `transcripts/normalized.json`
|
||||
- real `trim` stage producing `transcripts/trimmed.json`
|
||||
- real `analyze` stage for initial `session_recap` generation
|
||||
- optional Scriptorium render diagnostics (`render_debug`) before production run
|
||||
- storage/archive configuration and validation foundations for:
|
||||
- `pipeline.storage.s3`
|
||||
- `pipeline.spool`
|
||||
- `pipeline.archive` promotion rules
|
||||
- `session.inputs.audio_s3`
|
||||
- run identity and path-model foundations:
|
||||
- run ID generation (`YYYYMMDDTHHMMSSZ-xxxxxxxx`)
|
||||
- S3 session/run/current key builders
|
||||
- campaign/session/run local work/spool path helpers
|
||||
- manifest run/path identity fields (`campaign`, `run_id`, local and S3 prefixes)
|
||||
- remote storage backend layer:
|
||||
- narrow object-store interface (`List`, `Download`, `Upload`, `Exists`)
|
||||
- fake storage backend for deterministic tests (no network dependency)
|
||||
- S3-compatible backend using AWS SDK v2
|
||||
- config-based object-store construction helper
|
||||
|
||||
Still placeholder/future:
|
||||
|
||||
- `notify` stage behavior
|
||||
- additional Scriptorium artifact types beyond `session_recap`
|
||||
- artifact-to-artifact workflows beyond the initial single-artifact implementation
|
||||
- generic stale detection based on input/config checksums
|
||||
|
||||
## 3. Pipeline and Stage Boundaries
|
||||
|
||||
Canonical stage order:
|
||||
|
||||
1. `prepare`
|
||||
2. `transcribe`
|
||||
3. `merge`
|
||||
4. `polish`
|
||||
5. `normalize`
|
||||
6. `trim`
|
||||
7. `analyze`
|
||||
8. `archive`
|
||||
9. `notify`
|
||||
|
||||
Boundary rules:
|
||||
|
||||
- orchestration logic lives in `internal/app`
|
||||
- stage business logic lives in `internal/stage`
|
||||
- external-tool CLI construction lives in adapter packages
|
||||
- Scriptorium CLI details stay in `internal/adapters/scriptorium`
|
||||
|
||||
## 4. Scriptorium Integration Model
|
||||
|
||||
Integration mode:
|
||||
|
||||
- public CLI subprocesses only (no Scriptorium internal Go packages, no HTTP API)
|
||||
- production generation uses `scriptorium run`
|
||||
- diagnostics/testing render uses `scriptorium render --format json`
|
||||
|
||||
Run invocation shape used by adapter:
|
||||
|
||||
```bash
|
||||
scriptorium run --prompt <prompt_id> --input name=path --out <output_path>
|
||||
```
|
||||
|
||||
Optional flags passed when configured:
|
||||
|
||||
- `--config <path>`
|
||||
- `--profile <profile_id>`
|
||||
- repeated `--var name=value`
|
||||
- repeated `--input name=path`
|
||||
- `--timeout <duration>`
|
||||
- `--api-key-env <ENV_NAME>` when configured
|
||||
|
||||
Render invocation shape used by adapter:
|
||||
|
||||
```bash
|
||||
scriptorium render --prompt <prompt_id> --input name=path --format json --out <render_output_path>
|
||||
```
|
||||
|
||||
Adapter behavior:
|
||||
|
||||
- always passes `--out`
|
||||
- captures stdout/stderr separately
|
||||
- writes generated invocation metadata YAML (redacted, no secrets)
|
||||
- treats exit code `0` as success
|
||||
- treats exit code `1` as failure
|
||||
- treats exit code `2` as failure with `validation_failed=true` and preserves output metadata when available
|
||||
- validates successful output files exist and are non-empty
|
||||
- does not treat non-empty stderr as failure by itself
|
||||
|
||||
## 5. Configuration Contract
|
||||
|
||||
CLI pipeline config path resolution:
|
||||
|
||||
- when `--config <path>` is provided, that path is used
|
||||
- when `--config` is omitted, Narratio searches defaults in order:
|
||||
- `/usr/local/etc/narratio/pipeline.yml`
|
||||
- `/etc/narratio/pipeline.yml`
|
||||
- default values are centralized in `internal/config/defaults.go`
|
||||
|
||||
CLI session config path resolution:
|
||||
|
||||
- when `--session <path>` is provided, that path is used
|
||||
- when `--session` is omitted, Narratio searches defaults in order:
|
||||
- `./session.yml`
|
||||
- `/usr/local/etc/narratio/session.yml`
|
||||
- `/etc/narratio/session.yml`
|
||||
|
||||
Session template rendering:
|
||||
|
||||
- session templates are rendered before strict YAML decode
|
||||
- `--session-id <value>` provides the `session_id` template variable
|
||||
- supported placeholders:
|
||||
- `{{session_id}}`
|
||||
- `{{ session_id }}`
|
||||
- unresolved placeholders fail clearly
|
||||
- strict `KnownFields(true)` YAML validation still applies after rendering
|
||||
- if rendered `session.session_id` conflicts with `--session-id`, load fails clearly
|
||||
|
||||
Optional pipeline secrets directory:
|
||||
|
||||
- `pipeline.secrets.env_dir` enables loading environment variables from local files before command execution
|
||||
- file name = env var name; file contents = env var value (trailing newline/CRLF trimmed)
|
||||
- only env-var-style file names are considered; other entries are ignored
|
||||
- existing process environment values are preserved (not overwritten)
|
||||
- if configured, unreadable/missing `env_dir` fails command execution early
|
||||
- relative `env_dir` values are resolved from current working directory
|
||||
|
||||
Storage and archive foundations:
|
||||
|
||||
- `pipeline.storage.s3` is available for modeling S3 coordinates:
|
||||
- `bucket`
|
||||
- `root_prefix` (default `dnd`)
|
||||
- `region`
|
||||
- `endpoint`
|
||||
- `force_path_style` (default `false`)
|
||||
- `access_key_id_env` (default `OBJECT_STORAGE_KEY_ID`)
|
||||
- `secret_access_key_env` (default `OBJECT_STORAGE_KEY`)
|
||||
- `pipeline.spool.root` defaults to `/var/spool/narratio`
|
||||
- `pipeline.workspace.cleanup_after_archive` defaults to `false`
|
||||
- `pipeline.spool.delete_audio_after_archive` defaults to `false`
|
||||
- `pipeline.archive` is optional and defaults to:
|
||||
- `enabled: true`
|
||||
- `upload_run: true`
|
||||
- default `promote_artifacts`:
|
||||
- `transcripts/trimmed.json`
|
||||
- `artifacts/session_recap.md`
|
||||
- archive promotion rules enforce safe relative paths:
|
||||
- `from` and `to` are required
|
||||
- absolute paths are rejected
|
||||
- traversal segments such as `..` are rejected
|
||||
|
||||
Session input foundations:
|
||||
|
||||
- `session.campaign` is required
|
||||
- local audio remains supported through `session.inputs.audio_dir` or `session.inputs.audio_files`
|
||||
- optional S3 audio input shape is `session.inputs.audio_s3.prefix`
|
||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive
|
||||
- when `audio_s3` is configured, `prepare` lists and downloads `.flac` objects through the object-store backend
|
||||
|
||||
Cross-config validation scope:
|
||||
|
||||
- `pipeline.storage.s3.bucket` is required only when an S3-dependent feature is explicitly configured (for current foundations, that includes `session.inputs.audio_s3`, and archive upload intent when using `storage.backend: s3`)
|
||||
- no AWS credential values are stored in Narratio config; only env-var names are configured
|
||||
- when both configured credential env vars resolve to non-empty values, the S3 backend uses them as static credentials
|
||||
- when either configured credential value is missing, the S3 backend falls back to the AWS SDK default credential chain
|
||||
|
||||
Remote object-store backend scope:
|
||||
|
||||
- remote storage APIs are isolated to `internal/adapters/storage`
|
||||
- AWS SDK types remain contained within the S3 backend implementation package
|
||||
- S3 key/session path semantics remain outside the backend, with this invariant:
|
||||
- callers pass full bucket-relative object keys
|
||||
- backend methods do not prepend `root_prefix` or infer campaign/session/run paths
|
||||
- `prepare` now uses object-store `List` and `Download` for S3 audio input
|
||||
- `archive` now uses object-store `Upload` for successful run-record upload under the run prefix
|
||||
- `archive` now uses object-store `Upload` for promoted outputs and current pointers
|
||||
|
||||
Prepare S3 audio behavior (implemented):
|
||||
|
||||
- compute session prefix as `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
- resolve `session.inputs.audio_s3.prefix` under that session prefix
|
||||
- list objects under the computed audio prefix and filter `.flac` keys
|
||||
- fail clearly when no `.flac` objects are found
|
||||
- download selected objects to spool audio path:
|
||||
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- materialize audio files into workdir audio path:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- record S3 provenance in manifest input records (bucket/key/metadata/local paths/checksum)
|
||||
- no AWS SDK types are used in stage code; storage implementation details stay in storage adapter packages
|
||||
|
||||
Archive publishing behavior (implemented):
|
||||
|
||||
- `archive` verifies prerequisite stage success before upload:
|
||||
- `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`
|
||||
- only successful/completed runs are uploaded
|
||||
- uploaded run record destination is:
|
||||
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
|
||||
- uploaded existing local paths include:
|
||||
- `inputs/`, `transcripts/`, `artifacts/`, optional `reports/`, `config/`, `logs/`, and `manifest.json`
|
||||
- local `audio/` is intentionally excluded from upload by default
|
||||
- file upload order is deterministic (sorted relative paths)
|
||||
- `archive.enabled: false` and `archive.upload_run: false` skip upload cleanly
|
||||
- stage metadata records non-secret upload context:
|
||||
- run upload details, promoted output details, current manifest key, current pointer key
|
||||
- no secrets, transcript contents, prompt contents, or environment dumps
|
||||
- promotion rules:
|
||||
- `from` resolves from local workdir
|
||||
- `to` resolves under session-level S3 root
|
||||
- missing required source fails archive
|
||||
- missing optional source is skipped and recorded
|
||||
- default promoted outputs:
|
||||
- `transcripts/trimmed.json`
|
||||
- `artifacts/session_recap.md`
|
||||
- current pointers:
|
||||
- `current/manifest.json` uploaded after run upload and promotions
|
||||
- `current/run_id.txt` uploaded last with `{run_id}\n`
|
||||
- `current/run_id.txt` is the effective commit marker
|
||||
- if promotion or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`
|
||||
- failed/incomplete runs remain local and are not uploaded
|
||||
- post-archive local cleanup (implemented, opt-in):
|
||||
- cleanup runs only after archive succeeded and wrote `current/run_id.txt`
|
||||
- cleanup is executed after all selected stages in the command invocation succeed (for example, a later `notify` failure leaves local files intact)
|
||||
- `pipeline.spool.delete_audio_after_archive: true` removes only `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- `pipeline.workspace.cleanup_after_archive: true` removes only `{workspace.root}/work/{campaign}/{session_id}/{run_id}/`
|
||||
- cleanup does not run when archive is skipped/disabled/fails or when run upload is disabled
|
||||
- local development `audio_dir`/`audio_files` inputs are never removed by spool cleanup
|
||||
|
||||
`pipeline.scriptorium` is optional. Existing pipelines without Scriptorium continue to work.
|
||||
|
||||
`pipeline.trim` is optional. Existing pipelines without trim config continue to work.
|
||||
|
||||
`pipeline.normalize` is optional. Existing pipelines without normalize config continue to work.
|
||||
|
||||
`pipeline.audita` drives the real Audita subprocess adapter for the `polish` stage.
|
||||
|
||||
Audita defaulted fields:
|
||||
|
||||
- `binary` defaults to `audita`
|
||||
- `timeout` defaults to `3h`
|
||||
- `report` defaults to `true`
|
||||
|
||||
Audita optional fields:
|
||||
|
||||
- `llm_api_key_env` (enforced only when configured)
|
||||
- `modules` override list (when omitted/empty, Narratio does not pass `--modules`)
|
||||
- `base_url` (when omitted, Narratio does not pass `--base-url`)
|
||||
- `model` (when omitted, Narratio does not pass `--model`)
|
||||
- `transcript_description`
|
||||
- `config_path`
|
||||
- `output_schema` (`bare-segments` or `audita-v1`)
|
||||
- `work_dir_retention` (`always`, `auto`, `never`)
|
||||
- `total_llm_concurrency` (> 0 when provided)
|
||||
- `proposal_llm_concurrency` (> 0 when provided)
|
||||
- `validation_model`
|
||||
- `validation_llm_concurrency` (> 0 when provided)
|
||||
- `report` override
|
||||
|
||||
Narratio passes only configured optional Audita flags; omitted optional values defer to Audita runtime defaults/config.
|
||||
|
||||
Seriatim defaults:
|
||||
|
||||
- `pipeline.seriatim` may be omitted
|
||||
- `binary` defaults to `seriatim`
|
||||
- `timeout` defaults to `10m`
|
||||
- `output_schema` defaults to `seriatim-intermediate`
|
||||
- `coalesce_gap` defaults to `3.0`
|
||||
- `report` defaults to `true`
|
||||
|
||||
When `pipeline.normalize` is omitted, defaults are applied:
|
||||
|
||||
- `output_path: transcripts/normalized.json`
|
||||
- `output_schema: seriatim-intermediate`
|
||||
- `report: true`
|
||||
|
||||
When `pipeline.normalize` is present:
|
||||
|
||||
- `output_path` must be non-empty
|
||||
- `output_schema` must be one of `seriatim-minimal`, `seriatim-intermediate`, or `seriatim-full`
|
||||
- relative `output_path` values are session-workdir-relative paths
|
||||
- Seriatim binary settings still come from `pipeline.seriatim`
|
||||
|
||||
When `pipeline.trim` is present:
|
||||
|
||||
- `enabled` is optional and defaults to `false` when omitted
|
||||
- relative `output_path`, `bounds.output_path`, and `bounds.render_output_path` values are session-workdir-relative paths
|
||||
- do not store secrets in trim config values
|
||||
|
||||
When `pipeline.trim.enabled: true`:
|
||||
|
||||
- `output_path` is required and non-empty
|
||||
- `bounds.prompt_id` is required and non-empty
|
||||
- `bounds.transcript_input_name` is required and non-empty
|
||||
- `bounds.output_path` is required and non-empty
|
||||
- `bounds.timeout` must parse as a Go duration when provided
|
||||
- `bounds.render_debug: true` requires non-empty `bounds.render_output_path`
|
||||
- `bounds.profile_id` may be empty to use the prompt default profile
|
||||
- prompt IDs are config values, not hardcoded stage logic
|
||||
|
||||
When `pipeline.scriptorium` is present:
|
||||
|
||||
- `binary` defaults to `scriptorium` when omitted
|
||||
- `config_path` is optional; when provided it must be non-empty
|
||||
- `timeout` is optional; when provided it must parse as a Go duration
|
||||
- default `timeout` is `10m`
|
||||
- unknown YAML fields fail strict decode
|
||||
|
||||
Artifacts are configured as a map under `pipeline.scriptorium.artifacts` so multiple artifacts are possible in the config shape.
|
||||
|
||||
For each artifact definition:
|
||||
|
||||
- `enabled: true` requires non-empty `prompt_id`
|
||||
- `enabled: true` requires non-empty `output_path`
|
||||
- `timeout` must parse as Go duration when present
|
||||
- optional per-artifact `render_debug` may override global `scriptorium.render_debug`
|
||||
- `inputs` are named and each input requires non-empty `source`
|
||||
- inputs may be optional (`required: false`)
|
||||
- `vars` values currently support `string` and `bool`
|
||||
|
||||
Prompt IDs and profile IDs are configuration values, not hardcoded stage logic.
|
||||
|
||||
Trim config shape:
|
||||
|
||||
```yaml
|
||||
trim:
|
||||
enabled: true
|
||||
output_path: "transcripts/trimmed.json"
|
||||
bounds:
|
||||
prompt_id: "dnd_session.bounds"
|
||||
profile_id: ""
|
||||
transcript_input_name: "transcript"
|
||||
output_path: "artifacts/session_bounds.json"
|
||||
timeout: "10m"
|
||||
render_debug: false
|
||||
render_output_path: "artifacts/session_bounds.render.json"
|
||||
seriatim:
|
||||
report: false
|
||||
```
|
||||
|
||||
## 6. Transcript Tiers
|
||||
|
||||
Narratio currently produces and uses four transcript tiers:
|
||||
|
||||
- `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge
|
||||
- `transcripts/processed.json`: full raw Audita-polished transcript output (includes pre/post-game content)
|
||||
- `transcripts/normalized.json`: normalized transcript generated by Seriatim normalize
|
||||
- `transcripts/trimmed.json`: gameplay-only normalized polished transcript from trim stage
|
||||
|
||||
Trim reads `transcripts/normalized.json`, validates bounds IDs against that same transcript ID space, and writes `transcripts/trimmed.json`.
|
||||
|
||||
## 7. Normalize Stage (Current Implementation)
|
||||
|
||||
Normalize stage behavior:
|
||||
|
||||
- stage order position: after `polish` and before `trim`
|
||||
- discovers processed transcript from manifest polish outputs (`transcript_processed`) when present, else `work/<session_id>/transcripts/processed.json`
|
||||
- validates processed transcript JSON shape (`segments` array required)
|
||||
- runs Seriatim `normalize` to produce normalized transcript
|
||||
- validates normalized transcript JSON shape (`segments` array required)
|
||||
- validates normalize report JSON when enabled
|
||||
|
||||
Expected normalize outputs and diagnostics:
|
||||
|
||||
- `transcripts/normalized.json`
|
||||
- `artifacts/seriatim.normalize.report.json` (when normalize report is enabled)
|
||||
- `logs/seriatim.normalize.stdout.log`
|
||||
- `logs/seriatim.normalize.stderr.log`
|
||||
- `config/seriatim.normalize.generated.yml`
|
||||
|
||||
## 8. Trim Stage (Current Implementation)
|
||||
|
||||
Trim stage behavior:
|
||||
|
||||
- stage order position: after `normalize` and before `analyze`
|
||||
- discovers normalized transcript from manifest normalize outputs (`transcript_normalized`) when present, else `work/<session_id>/transcripts/normalized.json`
|
||||
- validates normalized transcript JSON shape (`segments` array required)
|
||||
- when `trim.enabled: false` (or trim config omitted), deterministically copies normalized transcript to `transcripts/trimmed.json` and records `trim_action=copy_disabled`
|
||||
- when `trim.enabled: true`:
|
||||
- runs Scriptorium bounds prompt using configured `trim.bounds.prompt_id`
|
||||
- writes bounds output to configured path (typically `artifacts/session_bounds.json`)
|
||||
- parses and validates bounds output against the same normalized transcript being trimmed
|
||||
- converts bounds range to Seriatim keep selector (for example `10-868`)
|
||||
- runs Seriatim `trim` to produce `transcripts/trimmed.json`
|
||||
- supports no-trim bounds actions (`none`/`copy`) by copying normalized transcript unchanged
|
||||
- validates trimmed transcript JSON shape (`segments` array required)
|
||||
|
||||
Expected trim outputs and diagnostics:
|
||||
|
||||
- `artifacts/session_bounds.json`
|
||||
- `transcripts/trimmed.json`
|
||||
- `logs/scriptorium.bounds.stdout.log`
|
||||
- `logs/scriptorium.bounds.stderr.log`
|
||||
- `config/scriptorium.bounds.generated.yml`
|
||||
- `logs/seriatim.trim.stdout.log`
|
||||
- `logs/seriatim.trim.stderr.log`
|
||||
- `config/seriatim.trim.generated.yml`
|
||||
- optional bounds render-debug outputs when enabled:
|
||||
- `artifacts/session_bounds.render.json`
|
||||
- `logs/scriptorium.bounds.render.stdout.log`
|
||||
- `logs/scriptorium.bounds.render.stderr.log`
|
||||
- `config/scriptorium.bounds.render.generated.yml`
|
||||
|
||||
Render-debug files are diagnostics. They are recorded in stage metadata/log/config refs and are not treated as canonical stage output artifact refs.
|
||||
|
||||
## 9. Analyze Stage (Current Implementation)
|
||||
|
||||
The current real analyze implementation supports only `scriptorium.artifacts.session_recap`.
|
||||
|
||||
Behavior:
|
||||
|
||||
- if `pipeline.scriptorium` is missing, analyze returns a skipped result with metadata
|
||||
- if no Scriptorium artifacts are enabled, analyze returns a skipped result with metadata
|
||||
- if enabled artifacts exist but `session_recap` is not enabled, analyze fails clearly
|
||||
- available transcript input sources for configured artifacts: `processed_transcript`, `normalized_transcript`, `trimmed_transcript`
|
||||
- `session_recap` should use `trimmed_transcript` input (`transcripts/trimmed.json`) for in-universe recap generation
|
||||
- `trimmed_transcript` input is resolved from manifest (`trim` output kind `transcript_trimmed`) when available, otherwise fallback path `work/<session_id>/transcripts/trimmed.json`
|
||||
- `normalized_transcript` input is resolved from manifest (`normalize` output kind `transcript_normalized`) when available, otherwise fallback path `work/<session_id>/transcripts/normalized.json`
|
||||
- `processed_transcript` input is resolved from manifest (`polish` output kind `transcript_processed`) when available, otherwise fallback path `work/<session_id>/transcripts/processed.json`
|
||||
- `normalized_transcript` is the preferred full-transcript source for future table/meta-analysis artifacts
|
||||
- `processed_transcript` remains available for advanced/debug use cases
|
||||
- transcript inputs are validated as JSON with top-level `segments` array
|
||||
- configured inputs are resolved by source
|
||||
- optional `previous_recap` is omitted when unavailable
|
||||
- required `previous_recap` fails before invocation when unavailable
|
||||
- vars are built from config + session metadata
|
||||
- `render_debug` controls pre-run `scriptorium render` diagnostics
|
||||
- render failure stops stage before production run
|
||||
- render output is validated as JSON
|
||||
- production call uses Scriptorium adapter `RunArtifact`
|
||||
- successful run output must exist and be non-empty
|
||||
- missing `trimmed_transcript` input for configured `trimmed_transcript` source fails clearly with guidance to run trim stage first
|
||||
- manifest records output refs, logs, generated config paths, and non-secret provenance metadata
|
||||
|
||||
## 10. Session Recap Paths
|
||||
|
||||
Current expected paths for `session_recap`:
|
||||
|
||||
- artifact output: `artifacts/session_recap.md`
|
||||
- run stdout log: `logs/scriptorium.session_recap.stdout.log`
|
||||
- run stderr log: `logs/scriptorium.session_recap.stderr.log`
|
||||
- run generated invocation/config: `config/scriptorium.session_recap.generated.yml`
|
||||
- render output (when enabled): `artifacts/session_recap.render.json`
|
||||
- render stdout log: `logs/scriptorium.session_recap.render.stdout.log`
|
||||
- render stderr log: `logs/scriptorium.session_recap.render.stderr.log`
|
||||
- render generated invocation/config: `config/scriptorium.session_recap.render.generated.yml`
|
||||
|
||||
## 11. Security and Privacy
|
||||
|
||||
- do not store secrets in pipeline YAML, generated invocation YAML, logs, or manifest metadata
|
||||
- if API-key integration is configured, pass env var names only (never raw key values)
|
||||
- with `pipeline.secrets.env_dir`, secret file values are loaded into process env only and are not persisted in manifest metadata or generated configs
|
||||
- avoid logging transcript content or rendered prompt content by default
|
||||
- treat generated artifacts and logs as potentially sensitive session material
|
||||
|
||||
## 12. Operational Caveat (Pre-Stale-Detection)
|
||||
|
||||
Checksum-based stale detection is not implemented yet.
|
||||
|
||||
If prepared inputs or prompt/runtime configuration change (for example glossary files, prompt IDs, profile IDs, or relevant pipeline settings), rerun the appropriate prior stages to refresh downstream artifacts.
|
||||
|
||||
Examples:
|
||||
|
||||
- glossary or autocorrect changes usually require rerunning at least `merge`, `polish`, `normalize`, `trim`, and `analyze`
|
||||
- trim prompt/profile changes require rerunning at least `normalize`, `trim`, and `analyze`
|
||||
- session recap prompt/profile/input-source changes require rerunning `analyze`
|
||||
|
||||
## 13. Roadmap
|
||||
|
||||
Planned next steps:
|
||||
|
||||
- extend analyze beyond `session_recap` to additional configured artifacts
|
||||
- support artifact inputs that consume prior generated artifacts
|
||||
- keep this composable without adding a generic DAG engine in the near term
|
||||
- implement real `archive` backend behavior
|
||||
- implement real `notify` backend behavior
|
||||
- add checksum-based stale detection and stale transitions
|
||||
|
||||
Architectural invariants remain:
|
||||
|
||||
- strict config decoding/validation
|
||||
- manifest-driven run control
|
||||
- clear stage/adapter separation
|
||||
- configuration-driven prompt/profile/input/vars/output mapping
|
||||
- Scriptorium integration through public CLI subprocess contract
|
||||
202
docs/architecture.md
Normal file
202
docs/architecture.md
Normal file
@@ -0,0 +1,202 @@
|
||||
# Narratio Architecture
|
||||
|
||||
## Purpose
|
||||
|
||||
`narratio` is a Go orchestration application for processing D&D session audio into polished transcripts and generated session artifacts.
|
||||
|
||||
This document defines the development principles for the project. It is inward-facing: its audience is developers and LLM coding agents. It should guide future changes, not serve as a complete implementation reference.
|
||||
|
||||
Implemented component details belong under `docs/internal/`.
|
||||
|
||||
## Project Shape
|
||||
|
||||
Narratio is a modular, stage-driven orchestrator.
|
||||
|
||||
It coordinates specialized downstream systems rather than reimplementing their domains:
|
||||
|
||||
- WhisperX handles transcription.
|
||||
- Seriatim handles deterministic transcript merge/normalization/trim behavior.
|
||||
- Audita handles transcript correction and polishing.
|
||||
- Scriptorium handles prompt execution and generated artifacts.
|
||||
|
||||
Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and archive semantics.
|
||||
|
||||
Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine.
|
||||
|
||||
## Core Principles
|
||||
|
||||
### Modular and composable
|
||||
|
||||
Code should be organized around clear responsibilities. Stages, adapters, config loading, manifest persistence, path construction, and storage behavior should remain separable and independently testable.
|
||||
|
||||
### Hexagonal boundaries
|
||||
|
||||
External systems should be isolated behind narrow adapters. Stage logic should depend on Narratio-level interfaces and data structures, not on external SDK types, subprocess argument construction, or transport-specific details.
|
||||
|
||||
### Standard library preference
|
||||
|
||||
Prefer the Go standard library. Add dependencies only when they provide substantial value, are necessary for an external integration, or are a widely used de facto standard.
|
||||
|
||||
Accepted examples include a YAML library for configuration and the AWS SDK for S3-compatible storage.
|
||||
|
||||
### Explicit orchestration
|
||||
|
||||
The pipeline should remain stage-driven and explicit. New behavior should be added through clear stage, adapter, config, or manifest contracts rather than implicit side effects or generic workflow abstraction.
|
||||
|
||||
## Stage Design
|
||||
|
||||
Each stage should have a clear scope of responsibility.
|
||||
|
||||
A stage should define:
|
||||
|
||||
- its purpose;
|
||||
- required input state;
|
||||
- produced output state;
|
||||
- config fields it consumes;
|
||||
- external adapters it uses;
|
||||
- manifest refs it reads or writes;
|
||||
- skip, force, and resume behavior;
|
||||
- failure behavior;
|
||||
- tests that protect its contract.
|
||||
|
||||
Stages should avoid reaching across boundaries. If shared behavior is needed, prefer a helper or service with a narrow interface over duplicating ad hoc logic between stages.
|
||||
|
||||
## Transactionality and Resume
|
||||
|
||||
A stage should behave transactionally.
|
||||
|
||||
A stage is complete only when its outputs have been written, validated, and recorded in the manifest. If a stage fails, Narratio should preserve enough local state for inspection, recovery, and resume.
|
||||
|
||||
A failed or incomplete run must not be treated as successful. Later stages should depend on manifest-recorded success, not merely on incidental files existing on disk.
|
||||
|
||||
## Manifest Model
|
||||
|
||||
The manifest is the durable local ledger for a run.
|
||||
|
||||
It should record:
|
||||
|
||||
- run identity;
|
||||
- stage status;
|
||||
- input and output refs;
|
||||
- logs and generated config refs;
|
||||
- checksums or provenance where useful;
|
||||
- non-secret adapter and archive metadata.
|
||||
|
||||
Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress.
|
||||
|
||||
## Adapter Boundaries
|
||||
|
||||
Adapters own external integration details.
|
||||
|
||||
Expected boundaries:
|
||||
|
||||
- WhisperX HTTP details stay in the WhisperX adapter.
|
||||
- Seriatim CLI construction stays in the Seriatim adapter.
|
||||
- Audita CLI construction stays in the Audita adapter.
|
||||
- Scriptorium CLI construction stays in the Scriptorium adapter.
|
||||
- Object-storage details stay behind the storage adapter interface.
|
||||
- AWS SDK types stay inside the S3 storage implementation.
|
||||
|
||||
Stage code should express intent in Narratio terms and call adapters through narrow contracts.
|
||||
|
||||
## Configuration Philosophy
|
||||
|
||||
Configuration should be strict, explicit, and operator-friendly.
|
||||
|
||||
Principles:
|
||||
|
||||
- YAML decoding should reject unknown fields.
|
||||
- Defaults should be centralized and testable.
|
||||
- Empty configured values should not silently override meaningful defaults.
|
||||
- Session templating should remain narrow and deterministic.
|
||||
- Template support should serve operator convenience, not become a general configuration language.
|
||||
|
||||
Narratio should not become a secondary configuration system for downstream tools. Seriatim, Audita, and Scriptorium should own their runtime defaults wherever practical. Narratio should pass required stage-contract paths and explicit operator overrides.
|
||||
|
||||
## Path and Storage Discipline
|
||||
|
||||
Local and remote paths are part of Narratio’s application contract.
|
||||
|
||||
Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and archive paths. Stages should avoid reconstructing canonical paths through scattered string concatenation.
|
||||
|
||||
Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics.
|
||||
|
||||
## Archive Invariants
|
||||
|
||||
Archive behavior must preserve a clear commit boundary.
|
||||
|
||||
A remote run is current only after the archive stage has successfully uploaded the run record, required promoted outputs, `current/manifest.json`, and finally `current/run_id.txt`.
|
||||
|
||||
`current/run_id.txt` is the final remote commit marker and must be written last.
|
||||
|
||||
Failed, incomplete, skipped, or uncommitted archive attempts must not be presented as current remote state. Local cleanup is permitted only after successful archive commit and only when explicitly configured.
|
||||
|
||||
## Security and Privacy
|
||||
|
||||
Narratio handles private campaign material.
|
||||
|
||||
Rules:
|
||||
|
||||
- Do not store raw secrets in pipeline or session YAML.
|
||||
- Use environment variable names or secret-file references for secret handling.
|
||||
- Do not write raw secret values to manifests, logs, generated configs, or archive metadata.
|
||||
- Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive.
|
||||
- Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
Diagnostics should be durable and discoverable, but distinct from canonical outputs.
|
||||
|
||||
Logs, reports, generated invocation/config files, and render-debug files support debugging. Transcript tiers and configured artifacts are pipeline products.
|
||||
|
||||
Manifest refs should preserve that distinction.
|
||||
|
||||
## Determinism
|
||||
|
||||
Where practical, Narratio should prefer deterministic behavior:
|
||||
|
||||
- stable local path layout;
|
||||
- stable remote key layout;
|
||||
- sorted upload order;
|
||||
- predictable generated config files;
|
||||
- repeatable command construction;
|
||||
- tests that do not depend on live external services.
|
||||
|
||||
Run IDs and timestamps may be intentionally variable, but surrounding behavior should remain testable.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
Core behavior should be testable without live external services.
|
||||
|
||||
Tests should cover:
|
||||
|
||||
- config loading, defaults, and validation;
|
||||
- CLI parsing and command construction;
|
||||
- path helpers;
|
||||
- manifest transitions;
|
||||
- stage success, failure, skip, and resume behavior;
|
||||
- adapter command construction;
|
||||
- fake storage behavior;
|
||||
- archive commit ordering;
|
||||
- example config validity where practical.
|
||||
|
||||
Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs.
|
||||
|
||||
## Documentation Expectations
|
||||
|
||||
Documentation must follow `docs/documentation/policy.md`.
|
||||
|
||||
Current behavior belongs in user-facing docs and `docs/internal/`. Future, planned, aspirational, experimental, or unimplemented work belongs only under `docs/roadmap/`.
|
||||
|
||||
`docs/architecture.md` should remain concise and principle-focused. It should not duplicate the full config reference, CLI reference, operations guide, or internal stage documentation.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
Narratio is not:
|
||||
|
||||
- a generic DAG or workflow engine;
|
||||
- a replacement configuration layer for Seriatim, Audita, or Scriptorium;
|
||||
- a storage backend abstraction beyond the needs of this pipeline;
|
||||
- a place to embed raw secrets;
|
||||
- a place for stage logic to depend directly on AWS SDK types or downstream tool internals;
|
||||
- a prompt-authoring system.
|
||||
531
docs/cli.md
Normal file
531
docs/cli.md
Normal file
@@ -0,0 +1,531 @@
|
||||
# CLI
|
||||
|
||||
## Shortest Useful Command
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
This command uses default system discovery for `pipeline.yml`, `campaign.yml`, and local `session.yml`. If local session discovery misses and S3 storage is configured, `--session-id` can load remote `session.yml` from the canonical session prefix.
|
||||
|
||||
Default discovery checks system config locations only. Pass `--config`, `--campaign`, and `--session` to use files from the current working directory.
|
||||
|
||||
## Command Overview
|
||||
|
||||
Implemented commands:
|
||||
|
||||
- `run`: execute pipeline stages and persist manifest state.
|
||||
- `plan`: validate config, prepare workspace layout, and print stage run/skip decisions.
|
||||
- `resume`: continue from first non-succeeded stage unless forced.
|
||||
- `status`: read an existing manifest or inspect local/remote state for a session.
|
||||
- `run-stage`: execute exactly one stage.
|
||||
- `analyze`: force-rerun the analyze stage.
|
||||
- `restore`: restore durable local session state from the committed remote archive state.
|
||||
- `session validate`: run read-only preflight checks for a session.
|
||||
- `session init`: create local or remote `session.yml`.
|
||||
- `artifacts list`: list effective artifact source IDs.
|
||||
- `locks`: list, add, and remove archive promotion locks.
|
||||
- `clean`: remove local workspace/spool state for one session or all local sessions.
|
||||
|
||||
Unknown commands print usage and exit non-zero.
|
||||
|
||||
For config semantics, see [docs/config.md](./config.md). For operator lifecycle and recovery, see [docs/operations.md](./operations.md).
|
||||
|
||||
## Complete Flag Reference
|
||||
|
||||
### `run`
|
||||
|
||||
- `--config <path>`: optional explicit `pipeline.yml` path.
|
||||
- `--campaign <path>`: optional explicit `campaign.yml` path.
|
||||
- `--session <path>`: optional explicit `session.yml` path.
|
||||
- `--session-id <value>`: session template variable value.
|
||||
- `--previous-session-id <value>`: previous-session template variable value.
|
||||
- `--force`: force stage execution.
|
||||
- `--artifacts <names>`: analyze artifact keys to execute (repeatable or comma-separated).
|
||||
|
||||
### `plan`
|
||||
|
||||
- `--config <path>`
|
||||
- `--campaign <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--previous-session-id <value>`
|
||||
- `--force`
|
||||
|
||||
### `resume`
|
||||
|
||||
- `--config <path>`
|
||||
- `--campaign <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--previous-session-id <value>`
|
||||
- `--force`
|
||||
- `--artifacts <names>`: analyze artifact keys to execute (repeatable or comma-separated).
|
||||
|
||||
### `run-stage`
|
||||
|
||||
- `--config <path>`
|
||||
- `--campaign <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--previous-session-id <value>`
|
||||
- `--force`
|
||||
- `--artifacts <names>`: analyze artifact keys to execute (repeatable or comma-separated).
|
||||
- positional `<stage>`: required stage name.
|
||||
|
||||
### `analyze`
|
||||
|
||||
- `--config <path>`
|
||||
- `--campaign <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--previous-session-id <value>`
|
||||
- `--artifacts <names>`: analyze artifact keys to execute (repeatable or comma-separated).
|
||||
|
||||
`analyze` is force-by-design and does not accept `--force`.
|
||||
|
||||
Valid stage names:
|
||||
|
||||
- `prepare`
|
||||
- `transcribe`
|
||||
- `merge`
|
||||
- `polish`
|
||||
- `normalize`
|
||||
- `trim`
|
||||
- `analyze`
|
||||
- `archive`
|
||||
- `notify`
|
||||
|
||||
### `restore`
|
||||
|
||||
- `--config <path>`
|
||||
- `--campaign <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--previous-session-id <value>`
|
||||
- `--dry-run`: plan restore actions without writing local files.
|
||||
- `--force`: overwrite local conflicting files with remote archive files.
|
||||
- `--include-audio`: include durable archived `audio/**` files in restore scope.
|
||||
|
||||
### `clean`
|
||||
|
||||
- `--session-id <value>`: required for session cleanup unless `--all` is set.
|
||||
- `--config <path>`
|
||||
- `--campaign <path>`
|
||||
- `--session <path>`
|
||||
- `--previous-session-id <value>`
|
||||
- `--all`: clean all local session work/spool state using pipeline config only.
|
||||
- `--dry-run`: print cleanup targets without deleting.
|
||||
- `--clear-cache`: also remove matching S3 audio cache entries.
|
||||
|
||||
### `status`
|
||||
|
||||
- `--manifest <path>`: inspect one manifest file.
|
||||
- `--config <path>`
|
||||
- `--campaign <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--previous-session-id <value>`
|
||||
|
||||
### `session validate`
|
||||
|
||||
- `--config <path>`
|
||||
- `--campaign <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--previous-session-id <value>`
|
||||
|
||||
### `session init`
|
||||
|
||||
- `--config <path>`: required.
|
||||
- `--campaign <path>`: required.
|
||||
- `--session-id <value>`: required.
|
||||
- `--output <path>`: local `session.yml` target; mutually exclusive with `--remote`.
|
||||
- `--remote`: write remote `session.yml` to the canonical session prefix; mutually exclusive with `--output`.
|
||||
- `--previous-session-id <value>`
|
||||
- `--date <value>`
|
||||
- `--title <value>`
|
||||
- `--audio-s3-prefix <prefix>`: defaults to `audio/` when neither audio flag is provided.
|
||||
- `--audio-dir <path>`: local audio directory; mutually exclusive with `--audio-s3-prefix`.
|
||||
- `--force`: overwrite existing local or remote target.
|
||||
|
||||
### `artifacts list`
|
||||
|
||||
- `--config <path>`
|
||||
- `--campaign <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--previous-session-id <value>`
|
||||
- `--remote`: check remote availability for configured archive promotion destinations.
|
||||
|
||||
### `locks`
|
||||
|
||||
- `--session-id <value>`: required for list, add, and remove.
|
||||
- `--config <path>`: optional explicit `pipeline.yml` path.
|
||||
- `--campaign <path>`: optional explicit `campaign.yml` path.
|
||||
- `--session <path>`: optional explicit `session.yml` path.
|
||||
- `--previous-session-id <value>`: optional session template value.
|
||||
- `add <source>`: add a remote lock for one artifact or transcript source.
|
||||
- `add --reason <text>`: record an optional remote lock reason.
|
||||
- `add --force`: update the reason for an existing remote lock.
|
||||
- `remove <source>`: remove one remote lock.
|
||||
|
||||
## Command Reference
|
||||
|
||||
### `run`
|
||||
|
||||
Purpose:
|
||||
- Execute configured stages in canonical order.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio run [--config <pipeline.yml>] [--campaign <campaign.yml>] [--session <session.yml>] [--session-id <id>] [--previous-session-id <id>] [--force] [--artifacts <name[,name...]>]
|
||||
```
|
||||
|
||||
Success output:
|
||||
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
||||
|
||||
Common failure cases:
|
||||
- missing system default config/campaign/session paths when flags omitted.
|
||||
- missing local session plus missing/unavailable remote `session.yml`.
|
||||
- invalid template/rendered session mismatch.
|
||||
- unknown/invalid `--artifacts` value.
|
||||
- `--artifacts` with unknown configured artifact key.
|
||||
|
||||
### `plan`
|
||||
|
||||
Purpose:
|
||||
- Validate config, load secrets (if configured), prepare workdir, and print stage run/skip decisions.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio plan [--config <pipeline.yml>] [--campaign <campaign.yml>] [--session <session.yml>] [--session-id <id>] [--previous-session-id <id>] [--force]
|
||||
```
|
||||
|
||||
Success output includes:
|
||||
- `narratio plan: workdir prepared at <path>`
|
||||
- one line per stage (`<stage>: run|skip`)
|
||||
- `totals: run=<n> skip=<n>`
|
||||
|
||||
Common failure cases:
|
||||
- same config/campaign/session discovery and validation failures as `run`.
|
||||
- remote session fallback failures when local session discovery misses.
|
||||
- secrets directory read failures when `pipeline.secrets.env_dir` is configured.
|
||||
|
||||
### `resume`
|
||||
|
||||
Purpose:
|
||||
- Continue from session-manifest stage status.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio resume [--config <pipeline.yml>] [--campaign <campaign.yml>] [--session <session.yml>] [--session-id <id>] [--previous-session-id <id>] [--force] [--artifacts <name[,name...]>]
|
||||
```
|
||||
|
||||
Success output:
|
||||
- `narratio resume: session <session_id> has no remaining stages`
|
||||
- or `narratio resume: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
||||
|
||||
Common failure cases:
|
||||
- same discovery/template/validation failures as `run`.
|
||||
- manifest load errors when existing manifest is unreadable.
|
||||
- invalid or unknown artifact selections.
|
||||
|
||||
### `status`
|
||||
|
||||
Purpose:
|
||||
- Inspect one manifest file, or inspect configured local/remote state for a session.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio status --manifest <manifest.json>
|
||||
narratio status [--config <pipeline.yml>] [--campaign <campaign.yml>] [--session <session.yml>] [--session-id <id>] [--previous-session-id <id>]
|
||||
```
|
||||
|
||||
Manifest output includes:
|
||||
- `session_id: <id>`
|
||||
- `updated_at: <timestamp>`
|
||||
- `stages:` entries (`- <stage>: <status>`)
|
||||
|
||||
Session output includes:
|
||||
- session ID, campaign, workspace, session config source.
|
||||
- local manifest state when present.
|
||||
- remote current archive state when storage is configured.
|
||||
- catalog-based remote output availability for expected transcript and artifact sources.
|
||||
- effective archive locks and conservative next actions.
|
||||
|
||||
Common failure cases:
|
||||
- missing `--manifest` when no config/session flags are provided.
|
||||
- unreadable or invalid manifest path.
|
||||
- invalid config or remote session fallback failure in session mode.
|
||||
|
||||
### `session validate`
|
||||
|
||||
Purpose:
|
||||
- Run read-only preflight checks for a session.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio session validate [--config <pipeline.yml>] [--campaign <campaign.yml>] [--session <session.yml>] [--session-id <id>] [--previous-session-id <id>]
|
||||
```
|
||||
|
||||
Checks include:
|
||||
- effective config and session source.
|
||||
- stable input files.
|
||||
- local or remote audio availability.
|
||||
- previous-session requirements.
|
||||
- archive promotions and effective locks.
|
||||
|
||||
Warnings do not fail the command. Any `ERROR` finding exits non-zero.
|
||||
|
||||
### `session init`
|
||||
|
||||
Purpose:
|
||||
- Create a strict-decoded session skeleton locally or in object storage.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio session init --config <pipeline.yml> --campaign <campaign.yml> --session-id <id> --output ./session.yml
|
||||
narratio session init --config <pipeline.yml> --campaign <campaign.yml> --session-id <id> --remote
|
||||
```
|
||||
|
||||
Behavior:
|
||||
- exactly one of `--output` or `--remote` is required.
|
||||
- remote writes target `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`.
|
||||
- existing local or remote targets fail unless `--force` is passed.
|
||||
- remote writes use existence checks, not compare-and-swap.
|
||||
|
||||
### `artifacts list`
|
||||
|
||||
Purpose:
|
||||
- List built-in, configured, previous-session, promoted, and locked artifact sources.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio artifacts list [--config <pipeline.yml>] [--campaign <campaign.yml>] [--session <session.yml>] [--session-id <id>] [--previous-session-id <id>] [--remote]
|
||||
```
|
||||
|
||||
`--remote` checks promoted top-level object availability through the storage adapter. Remote markers appear only in the `Promoted` section, which reports each configured archive promotion destination and includes `dest=<path>` when that destination differs from the source's canonical path.
|
||||
|
||||
### `locks`
|
||||
|
||||
Purpose:
|
||||
- Inspect and mutate source-based archive promotion locks for one session.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio locks --session-id <id>
|
||||
narratio locks add --session-id <id> [--reason <text>] [--force] <source>
|
||||
narratio locks remove --session-id <id> <source>
|
||||
```
|
||||
|
||||
Behavior:
|
||||
- `--session-id` is required for list, add, and remove.
|
||||
- optional `--config`, `--campaign`, and `--session` override default config discovery.
|
||||
- list mode prints effective locks from static `pipeline.archive.locks` and remote `{session_prefix}/locks.yml`.
|
||||
- `locks add` writes only the remote lock store and fails if the source is already locked by pipeline config.
|
||||
- `locks remove` removes only remote locks and cannot remove static pipeline locks.
|
||||
- `locks add --force` is required to update an existing remote lock reason.
|
||||
|
||||
Examples:
|
||||
|
||||
```bash
|
||||
narratio locks --session-id 2026-04-04
|
||||
narratio locks add --session-id 2026-04-04 --reason "manual transcript review" narratio.transcript.trimmed
|
||||
narratio locks remove --session-id 2026-04-04 narratio.transcript.trimmed
|
||||
```
|
||||
|
||||
### `run-stage`
|
||||
|
||||
Purpose:
|
||||
- Execute exactly one stage.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio run-stage [--config <pipeline.yml>] [--campaign <campaign.yml>] [--session <session.yml>] [--session-id <id>] [--previous-session-id <id>] [--force] [--artifacts <name[,name...]>] <stage>
|
||||
```
|
||||
|
||||
Success output:
|
||||
- `narratio run-stage: stage=<name> executed=<n> skipped=<n> force=<true|false>; manifest=<path>`
|
||||
|
||||
`--artifacts` behavior:
|
||||
- accepted only when `<stage>` is `analyze`.
|
||||
- names are normalized (trimmed, deduplicated, sorted).
|
||||
- unknown configured artifact keys fail.
|
||||
|
||||
Common failure cases:
|
||||
- missing stage positional arg.
|
||||
- unknown stage name.
|
||||
- using `--artifacts` with any non-`analyze` stage.
|
||||
|
||||
### `analyze`
|
||||
|
||||
Purpose:
|
||||
- Force-rerun the analyze stage.
|
||||
- Provide a shorter equivalent for `narratio run-stage --force analyze`.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio analyze [--config <pipeline.yml>] [--campaign <campaign.yml>] [--session <session.yml>] [--session-id <id>] [--previous-session-id <id>] [--artifacts <name[,name...]>]
|
||||
```
|
||||
|
||||
Success output:
|
||||
- `narratio analyze: executed=<n> skipped=<n> force=true; manifest=<path>`
|
||||
|
||||
Common failure cases:
|
||||
- positional arguments.
|
||||
- `--force`, because force is implicit.
|
||||
- unknown configured artifact keys.
|
||||
|
||||
### `restore`
|
||||
|
||||
Purpose:
|
||||
- Restore durable session state (`manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`, and optional `audio/**`) from the committed remote archive current state.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio restore [--config <pipeline.yml>] [--campaign <campaign.yml>] [--session <session.yml>] [--session-id <id>] [--previous-session-id <id>] [--dry-run] [--force] [--include-audio]
|
||||
```
|
||||
|
||||
Success output (dry-run):
|
||||
- `Restore plan for <campaign>/<session_id>`
|
||||
- `Remote run: <run_id>`
|
||||
- `Would download: <n>`
|
||||
- `Would skip unchanged: <n>`
|
||||
- `Conflicts: <n>`
|
||||
|
||||
Success output (non-dry-run):
|
||||
- `Restored session archive for <campaign>/<session_id>`
|
||||
- `Remote run: <run_id>`
|
||||
- `Downloaded: <n>`
|
||||
- `Skipped unchanged: <n>`
|
||||
- `Conflicts: <n>`
|
||||
|
||||
Common failure cases:
|
||||
- storage backend is not configured.
|
||||
- remote `current/run_id.txt` missing/empty.
|
||||
- remote `current/manifest.json` missing or invalid.
|
||||
- remote manifest session/campaign mismatch.
|
||||
- local conflicts without `--force`.
|
||||
- session lock conflict.
|
||||
|
||||
When `--include-audio` is set, S3 audio files are restored through the shared audio cache. Cache hits avoid re-downloading large audio objects.
|
||||
|
||||
### `clean`
|
||||
|
||||
Purpose:
|
||||
- Remove local Narratio work/spool state for testing, reruns, or recovery from corrupted local files.
|
||||
- Preserve durable S3 audio cache state unless `--clear-cache` is passed.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio clean --session-id <id> [--config <pipeline.yml>] [--campaign <campaign.yml>] [--session <session.yml>] [--previous-session-id <id>] [--dry-run] [--clear-cache]
|
||||
narratio clean --all [--config <pipeline.yml>] [--dry-run] [--clear-cache]
|
||||
```
|
||||
|
||||
Session cleanup deletes:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}`
|
||||
- `{spool.root}/{campaign}/{session_id}`
|
||||
|
||||
All-session cleanup deletes:
|
||||
- `{workspace.root}/work`
|
||||
- the contents of `{spool.root}`, while preserving the spool root directory itself.
|
||||
|
||||
Cache behavior:
|
||||
- cache is preserved by default.
|
||||
- `--clear-cache` in session mode removes cached S3 audio files for the resolved session.
|
||||
- `--all --clear-cache` removes the configured Narratio S3 audio cache namespace for the configured bucket/root prefix.
|
||||
- `--clear-cache` does not delete arbitrary files under `pipeline.cache.root`.
|
||||
|
||||
Common failure cases:
|
||||
- missing `--session-id` when `--all` is not set.
|
||||
- combining `--all` with `--campaign`, `--session`, `--session-id`, or `--previous-session-id`.
|
||||
- unsafe cleanup target, such as a symlink, a non-directory session target, a configured root directory, or a path outside the configured root.
|
||||
|
||||
## Common Workflows
|
||||
|
||||
Default-discovery run:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Run only selected analyze artifacts:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04 --artifacts session_recap,player_handout
|
||||
```
|
||||
|
||||
Resume with selected analyze artifacts:
|
||||
|
||||
```bash
|
||||
narratio resume --session-id 2026-04-04 --artifacts player_handout
|
||||
```
|
||||
|
||||
Force-rerun analyze with selected artifacts:
|
||||
|
||||
```bash
|
||||
narratio analyze --session-id 2026-04-04 --artifacts player_handout
|
||||
```
|
||||
|
||||
Preview restore actions without writes:
|
||||
|
||||
```bash
|
||||
narratio restore --session-id 2026-04-04 --dry-run
|
||||
```
|
||||
|
||||
Restore and then force analyze:
|
||||
|
||||
```bash
|
||||
narratio restore --session-id 2026-04-04
|
||||
narratio analyze --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Rehydrate canonical previous-session inputs after artifact-input changes:
|
||||
|
||||
```bash
|
||||
narratio run-stage --session-id 2026-04-04 --force prepare
|
||||
```
|
||||
|
||||
Reset local state before testing restore:
|
||||
|
||||
```bash
|
||||
narratio clean --session-id 2026-04-04 --dry-run
|
||||
narratio clean --session-id 2026-04-04
|
||||
narratio restore --session-id 2026-04-04 --include-audio
|
||||
```
|
||||
|
||||
Clean all local sessions while keeping cached S3 audio:
|
||||
|
||||
```bash
|
||||
narratio clean --all
|
||||
```
|
||||
|
||||
## Diagnostic / Recovery Commands
|
||||
|
||||
Inspect stage status:
|
||||
|
||||
```bash
|
||||
narratio status --manifest <manifest.json>
|
||||
```
|
||||
|
||||
Get manifest path from previous output:
|
||||
- `run`, `resume`, `run-stage`, and `analyze` print `manifest=<path>` on success.
|
||||
|
||||
## `--artifacts` and `--force`
|
||||
|
||||
- `--artifacts` filters which configured artifacts are executable when analyze runs.
|
||||
- `--artifacts` does not imply `--force`.
|
||||
- if analyze is already `succeeded` and `--force` is not set, runner-level skip still applies.
|
||||
487
docs/config.md
Normal file
487
docs/config.md
Normal file
@@ -0,0 +1,487 @@
|
||||
# Configuration
|
||||
|
||||
## 1. Overview
|
||||
|
||||
Narratio loads three YAML files:
|
||||
|
||||
- `pipeline.yml`: pipeline-level runtime configuration.
|
||||
- `campaign.yml`: stable campaign identity and campaign-level input defaults.
|
||||
- `session.yml`: per-session metadata and input selection, loaded locally or from the configured S3 backend.
|
||||
|
||||
These commands load and validate all three files before running:
|
||||
|
||||
- `narratio run`
|
||||
- `narratio plan`
|
||||
- `narratio resume`
|
||||
- `narratio run-stage`
|
||||
- `narratio restore`
|
||||
|
||||
Behavior:
|
||||
|
||||
- strict YAML decode is enabled (`KnownFields(true)`): unknown fields fail.
|
||||
- session templates render before session YAML decode.
|
||||
- remote `session.yml` uses the same strict decode and template behavior as local `session.yml`.
|
||||
- defaults are applied for optional pipeline fields.
|
||||
- campaign-level stable input paths fill missing session input paths.
|
||||
- session-level stable input paths override campaign-level input paths.
|
||||
- validation enforces required fields, value formats, and cross-field constraints.
|
||||
|
||||
## 2. Config file discovery
|
||||
|
||||
These commands use the same config discovery behavior:
|
||||
|
||||
- `narratio run`
|
||||
- `narratio plan`
|
||||
- `narratio resume`
|
||||
- `narratio run-stage`
|
||||
- `narratio restore`
|
||||
|
||||
Pipeline config lookup:
|
||||
|
||||
- if `--config <path>` is provided, that path is used.
|
||||
- if omitted, Narratio searches in order:
|
||||
1. `/usr/local/etc/narratio/pipeline.yml`
|
||||
2. `/etc/narratio/pipeline.yml`
|
||||
- first existing file wins.
|
||||
|
||||
Campaign config lookup:
|
||||
|
||||
- if `--campaign <path>` is provided, that path is used.
|
||||
- if omitted, Narratio searches in order:
|
||||
1. `/usr/local/etc/narratio/campaign.yml`
|
||||
2. `/etc/narratio/campaign.yml`
|
||||
- first existing file wins.
|
||||
|
||||
Session config lookup:
|
||||
|
||||
- if `--session <path>` is provided, that path is used.
|
||||
- if `--session` is omitted, Narratio searches locally in order:
|
||||
1. `/usr/local/etc/narratio/session.yml`
|
||||
2. `/etc/narratio/session.yml`
|
||||
- first existing local file wins.
|
||||
- if no local session file is found, `--session-id <value>` is present, storage is configured, and campaign identity is resolved, Narratio loads remote `session.yml` from:
|
||||
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`
|
||||
- local discovery always runs before remote fallback.
|
||||
- local files in the current working directory are used only when passed explicitly, for example `--config ./pipeline.yml --campaign ./campaign.yml --session ./session.yml`.
|
||||
|
||||
## 3. Session templating
|
||||
|
||||
Template behavior for local and remote `session.yml`:
|
||||
|
||||
- supported placeholders:
|
||||
- `{{session_id}}`
|
||||
- `{{ session_id }}`
|
||||
- `{{previous_session_id}}`
|
||||
- `{{ previous_session_id }}`
|
||||
- `--session-id <value>` supplies the placeholder value.
|
||||
- `--previous-session-id <value>` supplies the previous-session placeholder value.
|
||||
- unresolved placeholders fail load.
|
||||
- if rendered `session_id` mismatches `--session-id`, load fails.
|
||||
- if rendered `previous_session_id` mismatches `--previous-session-id`, load fails.
|
||||
|
||||
## 4. Minimal config set
|
||||
|
||||
### `pipeline.yml`
|
||||
|
||||
```yaml
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
```
|
||||
|
||||
Why this is sufficient:
|
||||
|
||||
- `whisperx.transcribe_url` is required.
|
||||
- `workspace.root` defaults to `/var/lib/narratio`.
|
||||
- optional sections (`seriatim`, `audita`, `archive`, `scriptorium`, `trim`, `normalize`, etc.) receive defaults or stay inactive.
|
||||
|
||||
### `campaign.yml`
|
||||
|
||||
```yaml
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
```
|
||||
|
||||
Why this is sufficient:
|
||||
|
||||
- `campaign` supplies the stable campaign identity.
|
||||
- stable input files are required and resolve relative to `campaign.yml` when copied during `prepare`.
|
||||
|
||||
### `session.yml`
|
||||
|
||||
```yaml
|
||||
session_id: "{{ session_id }}"
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
```
|
||||
|
||||
Why this is sufficient:
|
||||
|
||||
- `session_id` is required and can be rendered from `--session-id`.
|
||||
- `campaign` can be omitted because it is supplied by `campaign.yml`.
|
||||
- stable input paths can be omitted because `campaign.yml` supplies defaults.
|
||||
- local `audio_dir` resolves relative to `session.yml`.
|
||||
|
||||
Minimal local-file usage:
|
||||
|
||||
```bash
|
||||
narratio run --config /path/to/pipeline.yml --campaign ./campaign.yml --session ./session.yml --session-id 2026-05-03
|
||||
```
|
||||
|
||||
Previous-session-enabled variant:
|
||||
|
||||
```yaml
|
||||
session_id: "{{ session_id }}"
|
||||
previous_session_id: "{{ previous_session_id }}"
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
```
|
||||
|
||||
```bash
|
||||
narratio run --config /path/to/pipeline.yml --campaign ./campaign.yml --session ./session.yml --session-id 2026-05-03 --previous-session-id 2026-04-26
|
||||
```
|
||||
|
||||
## 5. Production-oriented config set
|
||||
|
||||
### `pipeline.yml`
|
||||
|
||||
```yaml
|
||||
workspace:
|
||||
root: /var/lib/narratio/workspace
|
||||
cleanup_after_archive: true
|
||||
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: my-dnd-archive
|
||||
root_prefix: dnd
|
||||
region: us-east-1
|
||||
access_key_id_env: OBJECT_STORAGE_KEY_ID
|
||||
secret_access_key_env: OBJECT_STORAGE_KEY
|
||||
|
||||
spool:
|
||||
root: /var/spool/narratio
|
||||
delete_audio_after_archive: true
|
||||
|
||||
cache:
|
||||
root: /var/cache/narratio
|
||||
s3_audio: true
|
||||
|
||||
archive:
|
||||
enabled: true
|
||||
upload_run: true
|
||||
promote_artifacts:
|
||||
- source: narratio.transcript.trimmed
|
||||
dest: transcripts/trimmed.json
|
||||
required: true
|
||||
- source: narratio.artifact.session_recap
|
||||
dest: artifacts/session_recap.md
|
||||
required: true
|
||||
locks:
|
||||
- source: narratio.artifact.session_recap
|
||||
reason: Final recap was manually edited.
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
|
||||
scriptorium:
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
previous_recap:
|
||||
source: narratio.previous_session.artifact.session_recap
|
||||
required: false
|
||||
```
|
||||
|
||||
### `campaign.yml`
|
||||
|
||||
```yaml
|
||||
campaign: forsaken
|
||||
inputs:
|
||||
speakers_file: /srv/narratio/campaigns/forsaken/speakers.yml
|
||||
autocorrect_file: /srv/narratio/campaigns/forsaken/autocorrect.yml
|
||||
glossary_file: /srv/narratio/campaigns/forsaken/glossary.yml
|
||||
```
|
||||
|
||||
### Local `session.yml`
|
||||
|
||||
```yaml
|
||||
session_id: "{{ session_id }}"
|
||||
previous_session_id: "{{ previous_session_id }}"
|
||||
date: 2026-05-03
|
||||
title: The Black Cabin
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
```
|
||||
|
||||
### S3-first session config
|
||||
|
||||
For S3-first operation, upload the same `session.yml` content to:
|
||||
|
||||
```text
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml
|
||||
```
|
||||
|
||||
Then run with explicit or discovered pipeline/campaign config and no `--session`:
|
||||
|
||||
```bash
|
||||
narratio run --config /usr/local/etc/narratio/pipeline.yml --campaign /usr/local/etc/narratio/campaign.yml --session-id 2026-05-03 --previous-session-id 2026-04-26
|
||||
```
|
||||
|
||||
Operational notes:
|
||||
|
||||
- archive promotion is explicit and source-based via `archive.promote_artifacts`.
|
||||
- `source` is required; `dest` is optional and derived when omitted.
|
||||
- `archive.locks` skips top-level promotion overwrites for static locked sources while preserving run-local uploads.
|
||||
- operator-created mutable locks are stored at `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/locks.yml` and are merged with static locks.
|
||||
- Narratio does not auto-promote all generated analyze artifacts.
|
||||
- `restore` reads the same config/campaign/session inputs and restore scope is bounded by committed archive current state.
|
||||
- `clean` removes workspace/spool state by default and preserves `pipeline.cache.root` unless `--clear-cache` is passed.
|
||||
|
||||
## 6. Full pipeline reference
|
||||
|
||||
| Path | Type | Required | Default |
|
||||
| --- | --- | --- | --- |
|
||||
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
|
||||
| `pipeline.workspace.cleanup_after_archive` | bool | No | `false` |
|
||||
| `pipeline.secrets.env_dir` | string | Conditional | none |
|
||||
| `pipeline.storage.backend` | string | No | empty |
|
||||
| `pipeline.storage.bucket` | string | No | empty |
|
||||
| `pipeline.storage.prefix` | string | No | empty |
|
||||
| `pipeline.storage.s3.bucket` | string | Conditional | empty |
|
||||
| `pipeline.storage.s3.root_prefix` | string | No | `dnd` |
|
||||
| `pipeline.storage.s3.region` | string | No | empty |
|
||||
| `pipeline.storage.s3.endpoint` | string | No | empty |
|
||||
| `pipeline.storage.s3.force_path_style` | bool | No | `false` |
|
||||
| `pipeline.storage.s3.access_key_id_env` | string | No | `OBJECT_STORAGE_KEY_ID` |
|
||||
| `pipeline.storage.s3.secret_access_key_env` | string | No | `OBJECT_STORAGE_KEY` |
|
||||
| `pipeline.spool.root` | string | No | `/var/spool/narratio` |
|
||||
| `pipeline.spool.delete_audio_after_archive` | bool | No | `false` |
|
||||
| `pipeline.cache.root` | string | No | `/var/cache/narratio` |
|
||||
| `pipeline.cache.s3_audio` | bool | No | `true` |
|
||||
| `pipeline.archive.enabled` | bool | No | `true` |
|
||||
| `pipeline.archive.upload_run` | bool | No | `true` |
|
||||
| `pipeline.archive.promote_artifacts[]` | list | No | trimmed transcript rule |
|
||||
| `pipeline.archive.promote_artifacts[].source` | string | Yes (per rule) | none |
|
||||
| `pipeline.archive.promote_artifacts[].dest` | string | No | derived from source |
|
||||
| `pipeline.archive.promote_artifacts[].required` | bool | No | `true` |
|
||||
| `pipeline.archive.locks[]` | list | No | empty |
|
||||
| `pipeline.archive.locks[].source` | string | Yes (per lock) | none |
|
||||
| `pipeline.archive.locks[].reason` | string | No | empty |
|
||||
| `pipeline.whisperx.transcribe_url` | string | Yes | none |
|
||||
| `pipeline.whisperx.language` | string | No | `en` |
|
||||
| `pipeline.whisperx.timeout` | duration string | No | `30m` |
|
||||
| `pipeline.whisperx.retries` | int | No | `3` |
|
||||
| `pipeline.whisperx.retry_delay` | duration string | No | `2s` |
|
||||
| `pipeline.whisperx.concurrency` | int | No | `2` |
|
||||
| `pipeline.seriatim.binary` | string | No | `seriatim` |
|
||||
| `pipeline.seriatim.timeout` | duration string | No | `10m` |
|
||||
| `pipeline.seriatim.output_schema` | string | No | `seriatim-intermediate` |
|
||||
| `pipeline.seriatim.coalesce_gap` | float | No | `3.0` |
|
||||
| `pipeline.seriatim.report` | bool | No | `true` |
|
||||
| `pipeline.seriatim.env.overlap_word_run_gap` | float | No | unset |
|
||||
| `pipeline.seriatim.env.overlap_word_run_reorder_window` | float | No | unset |
|
||||
| `pipeline.seriatim.env.backchannel_max_duration` | float | No | unset |
|
||||
| `pipeline.seriatim.env.filler_max_duration` | float | No | unset |
|
||||
| `pipeline.audita.binary` | string | No | `audita` |
|
||||
| `pipeline.audita.timeout` | duration string | No | `3h` |
|
||||
| `pipeline.audita.llm_api_key_env` | string | No | empty |
|
||||
| `pipeline.audita.modules[]` | list[string] | No | empty |
|
||||
| `pipeline.audita.base_url` | string | No | empty |
|
||||
| `pipeline.audita.model` | string | No | empty |
|
||||
| `pipeline.audita.total_llm_concurrency` | int | No | unset |
|
||||
| `pipeline.audita.proposal_llm_concurrency` | int | No | unset |
|
||||
| `pipeline.audita.validation_model` | string | No | empty |
|
||||
| `pipeline.audita.validation_llm_concurrency` | int | No | unset |
|
||||
| `pipeline.audita.transcript_description` | string | No | empty |
|
||||
| `pipeline.audita.config_path` | string | No | empty |
|
||||
| `pipeline.audita.output_schema` | string | No | empty |
|
||||
| `pipeline.audita.work_dir_retention` | string | No | empty |
|
||||
| `pipeline.audita.report` | bool | No | `true` |
|
||||
| `pipeline.normalize.output_path` | string | No | `transcripts/normalized.json` |
|
||||
| `pipeline.normalize.output_schema` | string | No | `seriatim-intermediate` |
|
||||
| `pipeline.normalize.report` | bool | No | `true` |
|
||||
| `pipeline.trim.enabled` | bool | No | `false` |
|
||||
| `pipeline.trim.output_path` | string | Conditional | none |
|
||||
| `pipeline.trim.bounds.prompt_id` | string | Conditional | none |
|
||||
| `pipeline.trim.bounds.profile_id` | string | No | empty |
|
||||
| `pipeline.trim.bounds.transcript_input_name` | string | Conditional | none |
|
||||
| `pipeline.trim.bounds.output_path` | string | Conditional | none |
|
||||
| `pipeline.trim.bounds.timeout` | duration string | No | `10m` |
|
||||
| `pipeline.trim.bounds.render_debug` | bool | No | `false` |
|
||||
| `pipeline.trim.bounds.render_output_path` | string | Conditional | none |
|
||||
| `pipeline.trim.seriatim.report` | bool | No | `false` |
|
||||
| `pipeline.scriptorium.binary` | string | No | `scriptorium` |
|
||||
| `pipeline.scriptorium.config_path` | string | No | empty |
|
||||
| `pipeline.scriptorium.timeout` | duration string | No | `10m` |
|
||||
| `pipeline.scriptorium.render_debug` | bool | No | `false` |
|
||||
| `pipeline.scriptorium.artifacts` | map | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.enabled` | bool | No | `false` |
|
||||
| `pipeline.scriptorium.artifacts.<name>.depends_on[]` | list[string] | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.render_debug` | bool | No | unset |
|
||||
| `pipeline.scriptorium.artifacts.<name>.prompt_id` | string | Conditional | none |
|
||||
| `pipeline.scriptorium.artifacts.<name>.profile_id` | string | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.output_path` | string | Conditional | none |
|
||||
| `pipeline.scriptorium.artifacts.<name>.timeout` | duration string | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` | string | Conditional | none |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.artifact` | string | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.path` | string | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required` | bool | No | `false` |
|
||||
| `pipeline.scriptorium.artifacts.<name>.vars.<key>` | map value | No | empty |
|
||||
| `pipeline.analyzer.binary_path` | string | No | empty |
|
||||
| `pipeline.analyzer.timeout` | duration string | No | empty |
|
||||
| `pipeline.analyzer.artifacts.output_dir` | string | No | empty |
|
||||
| `pipeline.analyzer.artifacts.types[]` | list[string] | No | empty |
|
||||
| `pipeline.notification.backend` | string | No | empty |
|
||||
| `pipeline.notification.recipient` | string | No | empty |
|
||||
| `pipeline.notification.timeout` | duration string | No | empty |
|
||||
|
||||
Scriptorium artifact-key and dependency rules:
|
||||
|
||||
- artifact keys must match `^[a-z][a-z0-9_]*$`.
|
||||
- enabled artifacts require `prompt_id` and `output_path`.
|
||||
- `output_path` must be relative, traversal-safe, and under `artifacts/`.
|
||||
- configured artifact input sources use `narratio.artifact.<name>`.
|
||||
- if input source references `narratio.artifact.<name>`, artifact `<name>` must exist and must be listed in `depends_on`.
|
||||
- every `depends_on` entry must be a configured artifact key.
|
||||
- self-dependency is rejected.
|
||||
- enabled dependency cycles are rejected.
|
||||
- any artifact referenced by `depends_on` or `narratio.artifact.<name>` source must define `output_path` (even if not enabled).
|
||||
|
||||
Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values:
|
||||
|
||||
- `narratio.previous_session.artifact.<configured_artifact_key>`
|
||||
- `narratio.transcript.merged`
|
||||
- `narratio.transcript.polished`
|
||||
- `narratio.transcript.full`
|
||||
- `narratio.transcript.trimmed`
|
||||
- `narratio.bounds.session`
|
||||
- `narratio.artifact.<configured_artifact_key>`
|
||||
- `previous_session_artifact` (legacy path-based source; uses `inputs.<key>.path`)
|
||||
|
||||
`pipeline.archive.promote_artifacts[].source` values:
|
||||
|
||||
- `narratio.transcript.merged`
|
||||
- `narratio.transcript.polished`
|
||||
- `narratio.transcript.full`
|
||||
- `narratio.transcript.trimmed`
|
||||
- `narratio.bounds.session`
|
||||
- `narratio.artifact.<configured_artifact_key>`
|
||||
|
||||
`pipeline.archive.locks[].source` accepts the same source values as `pipeline.archive.promote_artifacts[].source`.
|
||||
|
||||
Archive promotion destination rules:
|
||||
|
||||
- `dest` must be a clean relative path (not absolute, no traversal).
|
||||
- duplicate `dest` values are rejected.
|
||||
- if `dest` is omitted:
|
||||
- built-in sources derive their canonical destination path;
|
||||
- configured sources derive from `pipeline.scriptorium.artifacts.<name>.output_path`;
|
||||
- derivation failure is a config validation error.
|
||||
|
||||
Archive lock rules:
|
||||
|
||||
- locks are source-based and do not accept `dest`.
|
||||
- duplicate lock sources are rejected.
|
||||
- static `pipeline.archive.locks` win over remote mutable locks for the same source.
|
||||
- locked promotions are recorded as intentional skips in archive metadata.
|
||||
- locked required promotions do not fail archive by default.
|
||||
- ordinary `--force` reruns do not override locks.
|
||||
|
||||
Remote mutable lock store:
|
||||
|
||||
- path: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/locks.yml`.
|
||||
- strict YAML shape: top-level `locks`, each with `source` and optional `reason`.
|
||||
- `narratio locks add` and `narratio locks remove` mutate only the remote lock store.
|
||||
- writes use existence checks plus `--force` for updates; they are not compare-and-swap atomic.
|
||||
|
||||
Restore-related implications:
|
||||
|
||||
- restore remote identity requires archive S3 identity to resolve (`pipeline.storage.s3.bucket` and session prefix derivation inputs).
|
||||
- restore scope considers committed current state and durable paths (`manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`, optional `audio/**`).
|
||||
- S3 audio downloads use `pipeline.spool.root` for active downloads and `pipeline.cache.root` for reusable cached audio when `pipeline.cache.s3_audio` is true.
|
||||
- `pipeline.cache.root` is durable local cache state. It is not workspace state and is preserved by default by `narratio clean`.
|
||||
|
||||
## 7. Full campaign reference
|
||||
|
||||
| Path | Type | Required | Default |
|
||||
| --- | --- | --- | --- |
|
||||
| `campaign.campaign` | string | Yes | none |
|
||||
| `campaign.inputs.speakers_file` | string | Yes | none |
|
||||
| `campaign.inputs.autocorrect_file` | string | Yes | none |
|
||||
| `campaign.inputs.glossary_file` | string | Yes | none |
|
||||
|
||||
Campaign input paths may be absolute or relative. Relative paths resolve from the directory containing `campaign.yml`.
|
||||
|
||||
## 8. Full session reference
|
||||
|
||||
| Path | Type | Required | Default |
|
||||
| --- | --- | --- | --- |
|
||||
| `session.session_id` | string | Yes | none |
|
||||
| `session.previous_session_id` | string | No | empty |
|
||||
| `session.campaign` | string | No | `campaign.campaign` |
|
||||
| `session.date` | string | No | empty |
|
||||
| `session.title` | string | No | empty |
|
||||
| `session.inputs.audio_dir` | string | Conditional | empty |
|
||||
| `session.inputs.audio_files[]` | list[string] | Conditional | empty |
|
||||
| `session.inputs.audio_s3.prefix` | string | Conditional | none |
|
||||
| `session.inputs.speakers_file` | string | No | `campaign.inputs.speakers_file` |
|
||||
| `session.inputs.autocorrect_file` | string | No | `campaign.inputs.autocorrect_file` |
|
||||
| `session.inputs.glossary_file` | string | No | `campaign.inputs.glossary_file` |
|
||||
|
||||
Session input paths may be absolute or relative. Relative audio paths and session-level stable input overrides resolve from the directory containing `session.yml`. If both `campaign.yml` and `session.yml` specify campaign identity, the values must match.
|
||||
|
||||
Audio-source rule:
|
||||
|
||||
- configure exactly one mode:
|
||||
- `audio_dir`, or
|
||||
- `audio_files` (at least one), or
|
||||
- `audio_s3.prefix`
|
||||
- `audio_s3` cannot be combined with local audio fields.
|
||||
|
||||
Previous-session rule:
|
||||
|
||||
- if `session.previous_session_id` is set, it must not equal `session.session_id`.
|
||||
- canonical previous-session sources (`narratio.previous_session.artifact.<name>`) are hydrated during `prepare` from archive current state when required by enabled configured artifacts.
|
||||
|
||||
## 9. Secrets
|
||||
|
||||
Narratio supports filesystem-based secret injection via `pipeline.secrets.env_dir`.
|
||||
|
||||
Behavior:
|
||||
|
||||
- `env_dir` may be absolute or relative.
|
||||
- relative `env_dir` resolves from current working directory.
|
||||
- files with valid env-var names (`[A-Za-z_][A-Za-z0-9_]*`) are loaded.
|
||||
- values are loaded from file contents with trailing newline trimming.
|
||||
- existing process env vars are preserved.
|
||||
- invalid names and subdirectories are skipped.
|
||||
- missing/unreadable `env_dir` fails command execution.
|
||||
|
||||
Guidance:
|
||||
|
||||
- do not put secret values directly in YAML.
|
||||
- configure env var names in config and provide values via env/secrets files.
|
||||
|
||||
## 10. Examples
|
||||
|
||||
Maintained examples:
|
||||
|
||||
- `examples/pipeline.minimal.yml`
|
||||
- `examples/pipeline.production.yml`
|
||||
- `examples/pipeline.full.annotated.yml`
|
||||
- `examples/campaign.yml`
|
||||
- `examples/session.template.yml`
|
||||
- `examples/session.local-audio.yml`
|
||||
- `examples/session.s3-audio.yml`
|
||||
|
||||
These examples are validated by `internal/config` tests.
|
||||
94
docs/development.md
Normal file
94
docs/development.md
Normal file
@@ -0,0 +1,94 @@
|
||||
# Development Guide
|
||||
|
||||
## Purpose
|
||||
Canonical contributor workflow and engineering conventions for implemented Narratio behavior.
|
||||
|
||||
## Repository layout
|
||||
|
||||
- `cmd/narratio/`: CLI entrypoint.
|
||||
- `internal/app/`: command handlers, plan/run/resume orchestration, cleanup gates, secrets loading.
|
||||
- `internal/config/`: strict YAML loading, defaults, and validation.
|
||||
- `internal/stage/`: stage implementations and stage registry/order.
|
||||
- `internal/adapters/`: external boundary adapters (WhisperX, Seriatim, Audita, Scriptorium, storage, notify).
|
||||
- `internal/manifest/`: session/run manifest types and persistence.
|
||||
- `internal/artifacts/`: canonical local/remote path helpers and local artifact store.
|
||||
- `docs/`: canonical documentation set.
|
||||
- `examples/`: maintained config examples used by tests.
|
||||
|
||||
## Build and test commands
|
||||
|
||||
- Run focused CLI behavior checks:
|
||||
|
||||
```bash
|
||||
go test ./internal/app -run TestExecute -v
|
||||
```
|
||||
|
||||
- Run config example load/validate checks:
|
||||
|
||||
```bash
|
||||
go test ./internal/config -run TestExamplesLoadAndValidate -v
|
||||
```
|
||||
|
||||
- Run full test suite:
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Coding conventions
|
||||
|
||||
- Keep orchestration explicit and stage-driven; do not introduce generic workflow/DAG abstractions.
|
||||
- Keep external-system details inside adapter packages; stages should consume Narratio-level contracts only.
|
||||
- Use centralized path helpers from `internal/artifacts` rather than ad hoc path concatenation.
|
||||
- Preserve manifest-driven state transitions (`running`, `succeeded`, `failed`, `skipped`, `stale`) as the source of run progress.
|
||||
- Keep user/operator docs implementation-accurate; planned work belongs only under `docs/roadmap/`.
|
||||
|
||||
For design principles and invariants, see [docs/architecture.md](./architecture.md). For stage/adapter contracts, see [docs/internal/README.md](./internal/README.md).
|
||||
|
||||
## Dependency policy
|
||||
|
||||
- Prefer Go standard library where practical.
|
||||
- Add third-party dependencies only when they provide clear value for required behavior.
|
||||
- Keep dependency additions narrow to the boundary package that needs them.
|
||||
|
||||
## Change playbooks
|
||||
|
||||
### Add config fields
|
||||
|
||||
1. Add fields to config structs in `internal/config`.
|
||||
2. Set defaults in `internal/config/defaults.go` when appropriate.
|
||||
3. Add validation rules in `internal/config/validate.go`.
|
||||
4. Add or update load/validate tests in `internal/config/*_test.go`.
|
||||
5. Update canonical config docs and examples:
|
||||
- [docs/config.md](./config.md)
|
||||
- relevant files under `examples/`
|
||||
|
||||
### Add CLI flags or commands
|
||||
|
||||
1. Update command parsing and behavior in `internal/app`.
|
||||
2. Add or update command tests (`TestExecute` and command-specific tests).
|
||||
3. Update [docs/cli.md](./cli.md) and, if operator workflow changes, [docs/operations.md](./operations.md).
|
||||
|
||||
Remote-storage commands must obtain object storage through the app-level command object-store helper. Do not call `storage.NewObjectStoreFromConfig` directly from command handlers; the helper loads configured filesystem secrets before constructing the storage adapter.
|
||||
|
||||
### Add or modify stages/adapters
|
||||
|
||||
1. Implement stage behavior in `internal/stage` with clear input/output boundaries.
|
||||
2. Keep external transport/subprocess details in `internal/adapters`.
|
||||
3. Preserve manifest and promotion semantics expected by runner and archive logic.
|
||||
4. Add/update stage and adapter tests.
|
||||
5. Update internal component contracts in `docs/internal/`.
|
||||
|
||||
### Update examples
|
||||
|
||||
1. Keep canonical examples only in `examples/`.
|
||||
2. Ensure examples load and validate through runtime config paths.
|
||||
3. Update `internal/config/load_validate_test.go` as needed.
|
||||
4. Update links in `docs/config.md` if example filenames change.
|
||||
|
||||
### Update docs and roadmap
|
||||
|
||||
1. Keep implemented behavior in canonical docs (`README`, `docs/*.md`, `docs/internal/`).
|
||||
2. Keep planned/unimplemented behavior only in `docs/roadmap/`.
|
||||
3. After completing roadmap items, remove or mark them complete in `docs/roadmap/documentation.md`.
|
||||
4. Run a link/path sweep before finalizing changes.
|
||||
@@ -1,507 +0,0 @@
|
||||
# Workspace Architecture Implementation Plan (Audit)
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
**Complexity assessment:** **heavy**.
|
||||
|
||||
This is not a single path-helper refactor. The current codebase has a hybrid session/run model that works for current behavior, but diverges from `docs/development/workspace.md` in foundational places (workspace root shape, manifest responsibilities, stage output placement, and archive symmetry).
|
||||
|
||||
Highest-risk areas:
|
||||
|
||||
1. Splitting the current single manifest model into durable **session manifest** vs per-invocation **run manifest** without regressing skip/force/resume UX.
|
||||
2. Migrating path helpers and artifact-store interfaces from session-only roots (`work/{session}`) to campaign-aware roots (`work/{campaign}/{session}`) while preserving existing runs.
|
||||
3. Introducing run-local stage outputs + immediate promotion while keeping stage tests and archive behavior stable.
|
||||
4. Avoiding stale downstream skips after forced upstream reruns.
|
||||
|
||||
Surprising findings:
|
||||
|
||||
1. Code already has campaign/run-aware helpers (`SessionRunWorkDir`, `SessionSpoolAudioDir`) but core session helpers and manifest pathing remain campaign-unaware.
|
||||
2. Archive recently gained run/session fallback behavior for manifest/promotion sources, which confirms an existing hybrid-layout pressure point.
|
||||
3. Analyze input resolution is functional but ad hoc and stage-local; there is no centralized artifact registry/resolver.
|
||||
|
||||
---
|
||||
|
||||
## 2. Current-State Map
|
||||
|
||||
## 2.1 Workspace Path Construction
|
||||
|
||||
Primary path model:
|
||||
|
||||
- [`internal/artifacts/paths.go`](../../internal/artifacts/paths.go)
|
||||
- `SessionWorkDir(rootDir, sessionID)` => `{root}/work/{session_id}`
|
||||
- `buildSessionPaths(workspaceRoot, sessionID)` roots all canonical paths under `{root}/work/{session_id}`
|
||||
- `SessionRunWorkDir(rootDir, campaign, sessionID, runID)` exists, but is not the default session root helper.
|
||||
|
||||
Artifact store abstraction:
|
||||
|
||||
- [`internal/artifacts/store.go`](../../internal/artifacts/store.go)
|
||||
- `SessionPaths(sessionID string)` / `EnsureLayout(sessionID string)` are session-id-only (no campaign argument).
|
||||
- [`internal/artifacts/local.go`](../../internal/artifacts/local.go)
|
||||
- `EnsureLayout` creates session-level folders under `SessionWorkDir`.
|
||||
|
||||
Path normalization helper:
|
||||
|
||||
- [`internal/artifacts/resolve.go`](../../internal/artifacts/resolve.go)
|
||||
- `ResolveSessionLocalPathForRead` accepts absolute/workspace/session-relative values and probes filesystem.
|
||||
|
||||
Where campaign-aware/run-aware support exists:
|
||||
|
||||
- Local run path helpers: `SessionRunWorkDir`, `SessionSpoolAudioDir`.
|
||||
- S3 key helpers: [`internal/artifacts/s3_keys.go`](../../internal/artifacts/s3_keys.go) (`campaigns/sessions/runs/current`).
|
||||
|
||||
Where session-root assumptions remain `{workspace}/work/{session}`:
|
||||
|
||||
- Manifest path computation: [`internal/app/runner.go`](../../internal/app/runner.go) `manifestPathFor`.
|
||||
- Artifact store layout and most stage `paths := env.ArtifactStore.SessionPaths(sessionID)` calls.
|
||||
- Many tests hardcode `workspace/work/<session>/...` (examples below).
|
||||
|
||||
Manual/ad hoc path construction (not through a single resolver API):
|
||||
|
||||
- Common stage patterns: `filepath.Join(paths.TranscriptsDir, "...")`, `filepath.Join(paths.ArtifactsDir, "...")`, etc.
|
||||
- `prepare` run/work selection: [`internal/stage/prepare.go`](../../internal/stage/prepare.go) `pathsWorkDirForManifest`.
|
||||
- Analyze input fallback path resolution: `resolveInputPathForRead` in [`internal/stage/analyze.go`](../../internal/stage/analyze.go).
|
||||
|
||||
## 2.2 Manifest and Run Identity
|
||||
|
||||
Current manifest model:
|
||||
|
||||
- [`internal/manifest/manifest.go`](../../internal/manifest/manifest.go) `Manifest` includes both session and run fields:
|
||||
- `SessionID`, `Campaign`
|
||||
- `RunID`, `LocalWorkDir`, `LocalSpoolDir`
|
||||
- `S3Bucket`, `S3SessionPrefix`, `S3RunPrefix`
|
||||
- `Stages`, `Inputs`, stage outputs/logs/generated configs/metadata.
|
||||
|
||||
Current persistence:
|
||||
|
||||
- [`internal/manifest/store.go`](../../internal/manifest/store.go) `LocalStore` reads/writes one JSON manifest path.
|
||||
- Runner always loads/saves one manifest path via `manifestPathFor(cfg)` (session-root path under current layout).
|
||||
|
||||
Identity initialization:
|
||||
|
||||
- [`internal/app/runner.go`](../../internal/app/runner.go) `ensureManifestIdentity` populates run fields if absent.
|
||||
- `RunID` is generated once for an empty manifest and reused thereafter (hybrid semantics).
|
||||
|
||||
Interpretation today:
|
||||
|
||||
- Best described as a **hybrid session manifest** with run identity fields, not as distinct session + run manifests.
|
||||
|
||||
What this means for redesign:
|
||||
|
||||
- Session-vs-run split is not just file relocation; it requires new responsibilities and write flows.
|
||||
- A backward-compatible evolution path is possible by:
|
||||
- preserving current fields in session manifest for migration/read-compat,
|
||||
- adding explicit run-manifest type + path,
|
||||
- gradually moving invocation-specific details to run manifests.
|
||||
|
||||
## 2.3 Stage Output Paths (Current)
|
||||
|
||||
All implemented stages currently write canonical artifacts directly into session-level `paths.*` roots (under current session root), with logs/configs typically also session-level.
|
||||
|
||||
1. `prepare` ([`internal/stage/prepare.go`](../../internal/stage/prepare.go))
|
||||
- Inputs copied to `inputs/` (`session.yml`, `pipeline.resolved.yml`, `speakers.yml`, `autocorrect.yml`, `glossary.yml`)
|
||||
- Audio copied to `audio/`
|
||||
- Manifest input provenance recorded in `m.Inputs`
|
||||
- S3 audio uses run-scoped spool/work helpers when `RunID` is present.
|
||||
|
||||
2. `transcribe` ([`internal/stage/transcribe.go`](../../internal/stage/transcribe.go))
|
||||
- Outputs: `transcripts/raw/{speaker}.json`
|
||||
- Stage metadata only (no stage logs/config files produced here).
|
||||
|
||||
3. `merge` ([`internal/stage/merge.go`](../../internal/stage/merge.go))
|
||||
- Pre-normalize intermediates: `transcripts/raw/normalized/{basename}.normalized.json`
|
||||
- Merge output: `transcripts/merged.json`
|
||||
- Report: `artifacts/seriatim.report.json` (if enabled)
|
||||
- Logs/configs:
|
||||
- per-input normalize logs/configs in session-level `logs/` + `config/`
|
||||
- merge logs/config in session-level `logs/` + `config/`
|
||||
|
||||
4. `polish` ([`internal/stage/polish.go`](../../internal/stage/polish.go))
|
||||
- Output: `transcripts/processed.json`
|
||||
- Report: `artifacts/audita.report.json` (if enabled)
|
||||
- Work dir: `artifacts/audita-work`
|
||||
- Logs/config: `logs/audita.*`, `config/audita.generated.yml`
|
||||
|
||||
5. `normalize` ([`internal/stage/normalize.go`](../../internal/stage/normalize.go))
|
||||
- Output: `transcripts/normalized.json` (configurable)
|
||||
- Report: `artifacts/seriatim.normalize.report.json` (if enabled)
|
||||
- Logs/config: `logs/seriatim.normalize.*`, `config/seriatim.normalize.generated.yml`
|
||||
|
||||
6. `trim` ([`internal/stage/trim.go`](../../internal/stage/trim.go))
|
||||
- Output: `transcripts/trimmed.json` (configurable)
|
||||
- Bounds output path from config (default examples use `artifacts/session_bounds.json`)
|
||||
- Logs/configs in session-level `logs/` and `config/` for scriptorium + seriatim invocations
|
||||
|
||||
7. `analyze` ([`internal/stage/analyze.go`](../../internal/stage/analyze.go))
|
||||
- Current implemented artifact: `artifacts/session_recap.md`
|
||||
- Logs/config in session-level `logs/` + `config/`
|
||||
- Optional render diagnostics under session-level artifacts/logs/config.
|
||||
|
||||
8. `archive` ([`internal/stage/archive.go`](../../internal/stage/archive.go))
|
||||
- Reads from a "workDir" derived by `archiveWorkDir`:
|
||||
- prefers `m.LocalWorkDir` if it exists,
|
||||
- else tries run-scoped campaign/session/run path,
|
||||
- else falls back to legacy session path.
|
||||
- Uploads run record and promotions.
|
||||
- Current code now resolves manifest and promotion sources via run/session fallback.
|
||||
|
||||
9. `notify`
|
||||
- Placeholder only in [`internal/stage/placeholders.go`](../../internal/stage/placeholders.go); no durable outputs.
|
||||
|
||||
Durable-vs-diagnostic split today:
|
||||
|
||||
- Durable artifacts and diagnostics are mixed at session level.
|
||||
- No run-local stage directories exist yet.
|
||||
|
||||
## 2.4 Idempotency / Force / Resume / Sparse Runs
|
||||
|
||||
Run control implementation:
|
||||
|
||||
- [`internal/app/run_control.go`](../../internal/app/run_control.go)
|
||||
- skip rule: `!force && stageSucceeded(manifest, stage)`
|
||||
- stale detection TODO only; no invalidation logic.
|
||||
|
||||
Command behavior:
|
||||
|
||||
- `run`: full plan through `executeStages`.
|
||||
- `run-stage`: single selected stage through `executeStages`.
|
||||
- `resume`: starts at first non-succeeded stage unless `--force`.
|
||||
|
||||
Current assumptions:
|
||||
|
||||
- Command invocation mutates a single durable workspace + single manifest for the session path model.
|
||||
- There is no per-invocation run manifest write path.
|
||||
|
||||
Smallest safe UX-preserving invariant to keep during migration:
|
||||
|
||||
- Session manifest remains source-of-truth for skip decisions across invocations.
|
||||
|
||||
## 2.5 Archive and S3 Alignment
|
||||
|
||||
S3 semantics are relatively mature:
|
||||
|
||||
- [`internal/artifacts/s3_keys.go`](../../internal/artifacts/s3_keys.go)
|
||||
- session prefix + run prefix + `current/manifest.json` + `current/run_id.txt`.
|
||||
|
||||
Archive stage behavior:
|
||||
|
||||
- uploads run records under run prefix,
|
||||
- uploads promoted session outputs,
|
||||
- uploads current manifest then current run pointer,
|
||||
- current pointer is commit marker,
|
||||
- required promotions fail; optional promotions skipped.
|
||||
|
||||
Local-vs-remote mismatch still present:
|
||||
|
||||
- Local canonical session root currently defaults to `work/{session}` (artifact store),
|
||||
- while archive/run helpers expect campaign-aware run locations (`work/{campaign}/{session}/{run}`),
|
||||
- causing fallback logic and hybrid handling in `archive`.
|
||||
|
||||
## 2.6 Artifact Source Resolution in Analyze
|
||||
|
||||
Current implementation is stage-local and ad hoc:
|
||||
|
||||
- Transcript discoverers:
|
||||
- `discoverProcessedTranscript`
|
||||
- `discoverNormalizedTranscript`
|
||||
- `discoverTrimmedTranscript`
|
||||
- Input source switch in `resolveScriptoriumInput` supports:
|
||||
- `processed_transcript`
|
||||
- `normalized_transcript`
|
||||
- `trimmed_transcript`
|
||||
- `previous_session_artifact`
|
||||
|
||||
No first-class artifact registry exists yet. Aliases and canonical IDs are not modeled.
|
||||
|
||||
## 2.7 Stale/Invalidation
|
||||
|
||||
- `manifest.StageStatus` already defines `stale` (`internal/manifest/status.go`), but no stage uses it.
|
||||
- Skip logic ignores stale state and only checks `succeeded`.
|
||||
- Forced upstream rerun does not invalidate downstream stage success markers.
|
||||
|
||||
## 2.8 Test Coverage Relevant to Redesign
|
||||
|
||||
High-value existing coverage:
|
||||
|
||||
- Workspace helpers:
|
||||
- [`internal/artifacts/paths_model_test.go`](../../internal/artifacts/paths_model_test.go)
|
||||
- [`internal/artifacts/resolve_test.go`](../../internal/artifacts/resolve_test.go)
|
||||
- Runner semantics:
|
||||
- [`internal/app/runner_test.go`](../../internal/app/runner_test.go)
|
||||
- [`internal/app/resume_run_stage_test.go`](../../internal/app/resume_run_stage_test.go)
|
||||
- Stage path/output behavior:
|
||||
- `internal/stage/*_test.go` for prepare/transcribe/merge/polish/normalize/trim/analyze/archive
|
||||
- Archive behavior:
|
||||
- [`internal/stage/archive_test.go`](../../internal/stage/archive_test.go)
|
||||
- [`internal/app/post_archive_cleanup_test.go`](../../internal/app/post_archive_cleanup_test.go)
|
||||
|
||||
Tests likely to fail during workspace redesign:
|
||||
|
||||
- Any tests hardcoding `work/{session}` manifest and transcript paths (many in `internal/app/*test.go`, `internal/stage/*test.go`).
|
||||
- Archive tests assuming current hybrid fallback behavior.
|
||||
|
||||
---
|
||||
|
||||
## 3. Gap Analysis Against `docs/development/workspace.md`
|
||||
|
||||
| Intended concept | Current status | Notes |
|
||||
|---|---|---|
|
||||
| Session root at `work/{campaign}/{session}` | **Partial / mostly absent** | `SessionWorkDir` and artifact store still use `work/{session}`. Campaign-aware run helper exists separately. |
|
||||
| Distinct session manifest vs run manifest | **Absent** | One hybrid manifest model/file is used. |
|
||||
| Run-local stage dirs under `runs/{run_id}/{stage}` | **Absent** | Stages write canonical outputs/logs/config directly at session level. |
|
||||
| Immediate promotion run-local -> session canonical | **Absent** | No run-local staging area to promote from today. |
|
||||
| Session manifest skip source of truth | **Present** | Skip/resume use single manifest stage statuses. |
|
||||
| Sparse runs represented under `runs/{run_id}` | **Absent** | No run-manifest/per-run stage records yet. |
|
||||
| Artifact resolver with canonical IDs + aliases | **Absent** | Analyze resolves via stage-local source-name switch and fallback helpers. |
|
||||
| Downstream invalidation for forced upstream reruns | **Absent** | TODO only; no stale propagation or status clearing. |
|
||||
| Local semantics mirror archive semantics | **Partial** | Archive/S3 side models runs/current, local workspace core still session-layout-centric. |
|
||||
| Safe path helpers centralization | **Partial** | Good helper base exists, but many stage-level manual joins still encode conventions. |
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommended Implementation Sequence
|
||||
|
||||
## Step 1: Introduce campaign-aware session path model without behavior break
|
||||
|
||||
Purpose:
|
||||
|
||||
- Add first-class helpers for `work/{campaign}/{session}` and make them available everywhere.
|
||||
|
||||
Expected changes:
|
||||
|
||||
- `internal/artifacts`: add/extend path helpers and `SessionPaths` constructor variants that accept campaign.
|
||||
- `internal/app`: pass campaign into path-model entrypoints where available.
|
||||
|
||||
Behavior change:
|
||||
|
||||
- None initially (can keep legacy fallback reads).
|
||||
|
||||
Tests:
|
||||
|
||||
- Add campaign-aware path-model tests.
|
||||
- Keep legacy-path compatibility tests.
|
||||
|
||||
Risks:
|
||||
|
||||
- Wide compile-time touch due `Store` interface signatures.
|
||||
|
||||
Rollback:
|
||||
|
||||
- Keep legacy helper wrappers until full migration lands.
|
||||
|
||||
## Step 2: Split manifest responsibilities (session manifest + run manifest scaffolding)
|
||||
|
||||
Purpose:
|
||||
|
||||
- Preserve current UX while introducing explicit run execution records.
|
||||
|
||||
Expected changes:
|
||||
|
||||
- `internal/manifest`: add run manifest type/store helpers.
|
||||
- `internal/app/runner.go`: create/load session manifest and initialize per-invocation run manifest path.
|
||||
|
||||
Behavior change:
|
||||
|
||||
- Session manifest remains skip truth source.
|
||||
- Run manifest begins recording invocation metadata/stage actions.
|
||||
|
||||
Tests:
|
||||
|
||||
- New tests for both manifest files existing and being updated correctly.
|
||||
|
||||
Risks:
|
||||
|
||||
- Incorrect ordering of saves can regress crash consistency.
|
||||
|
||||
Rollback:
|
||||
|
||||
- Keep session-manifest-only decision logic until run manifest proves stable.
|
||||
|
||||
## Step 3: Move stage execution products to run-local directories with promotion
|
||||
|
||||
Purpose:
|
||||
|
||||
- Align with workspace architecture (`runs/{run_id}/{stage}/...`) while preserving canonical outputs.
|
||||
|
||||
Expected changes:
|
||||
|
||||
- `internal/stage`: each implemented stage writes outputs/logs/config/reports to run-local paths.
|
||||
- Introduce shared promotion helpers (atomic copy/rename + output validation + manifest provenance).
|
||||
|
||||
Behavior change:
|
||||
|
||||
- Canonical outputs remain session-level; run-local diagnostics now preserved per run.
|
||||
|
||||
Tests:
|
||||
|
||||
- Stage tests updated to assert run-local outputs + promoted canonical outputs.
|
||||
- New tests for immediate promotion and producer run metadata.
|
||||
|
||||
Risks:
|
||||
|
||||
- Highest regression risk (all implemented stages touched).
|
||||
|
||||
Rollback:
|
||||
|
||||
- Stage-by-stage migration flag or phased rollout by stage order.
|
||||
|
||||
## Step 4: Archive alignment pass
|
||||
|
||||
Purpose:
|
||||
|
||||
- Remove hybrid fallback complexity once local layout is canonical.
|
||||
|
||||
Expected changes:
|
||||
|
||||
- `internal/stage/archive.go`: resolve sources from canonical session outputs and run manifests deterministically.
|
||||
- Keep `current/manifest.json` + `current/run_id.txt` publication semantics.
|
||||
|
||||
Behavior change:
|
||||
|
||||
- Simpler source selection; fewer cross-layout heuristics.
|
||||
|
||||
Tests:
|
||||
|
||||
- Archive tests for run uploads, promotions, current pointers, and fallback removal/compat gates.
|
||||
|
||||
Risks:
|
||||
|
||||
- Breaking current mixed-layout compatibility too early.
|
||||
|
||||
Rollback:
|
||||
|
||||
- Retain fallback compatibility for one migration window.
|
||||
|
||||
## Step 5: Artifact registry/resolver (analyze first consumer)
|
||||
|
||||
Purpose:
|
||||
|
||||
- Replace ad hoc analyze source resolution with canonical artifact IDs and aliases.
|
||||
|
||||
Expected changes:
|
||||
|
||||
- New resolver package (for example `internal/artifacts/registry` or `internal/stage/artifactresolve`) with IDs:
|
||||
- `narratio.transcript.merged`
|
||||
- `narratio.transcript.polished`
|
||||
- `narratio.transcript.full`
|
||||
- `narratio.transcript.trimmed`
|
||||
- `narratio.bounds.session`
|
||||
- `narratio.artifact.session_recap`
|
||||
- Backward-compatible alias map for current source names.
|
||||
|
||||
Behavior change:
|
||||
|
||||
- Analyze input resolution becomes centralized and consistent.
|
||||
|
||||
Tests:
|
||||
|
||||
- Resolver unit tests for canonical IDs + aliases + missing-input error clarity.
|
||||
- Analyze tests updated to assert resolver usage.
|
||||
|
||||
Risks:
|
||||
|
||||
- Input resolution edge cases for `previous_session_artifact` and required/optional handling.
|
||||
|
||||
Rollback:
|
||||
|
||||
- Keep old resolver path behind a temporary compatibility function.
|
||||
|
||||
## Step 6: Minimal downstream invalidation after forced upstream reruns
|
||||
|
||||
Purpose:
|
||||
|
||||
- Prevent stale downstream skips before full checksum stale detection exists.
|
||||
|
||||
Expected changes:
|
||||
|
||||
- `internal/app/run_control.go` + manifest transition helpers.
|
||||
- On forced rerun success of stage `X`, clear/mark downstream success states.
|
||||
|
||||
Behavior change:
|
||||
|
||||
- Subsequent runs no longer skip stale downstream stages.
|
||||
|
||||
Tests:
|
||||
|
||||
- New run-control tests for force-induced downstream invalidation.
|
||||
- Resume tests with forced sparse runs.
|
||||
|
||||
Risks:
|
||||
|
||||
- Over-invalidating too broadly and degrading UX.
|
||||
|
||||
Rollback:
|
||||
|
||||
- Start with deterministic downstream stage list based on pipeline order only.
|
||||
|
||||
## Step 7: Legacy layout migration strategy
|
||||
|
||||
Purpose:
|
||||
|
||||
- Handle existing `work/{session}` data safely.
|
||||
|
||||
Expected changes:
|
||||
|
||||
- Startup detection/migration path in app layer.
|
||||
- Clear failure messages for ambiguous legacy states.
|
||||
|
||||
Behavior change:
|
||||
|
||||
- Explicit migration semantics instead of implicit fallback drift.
|
||||
|
||||
Tests:
|
||||
|
||||
- Migration detection tests for legacy-only, new-only, and ambiguous layouts.
|
||||
|
||||
Risks:
|
||||
|
||||
- Silent duplication if both layouts are partially populated.
|
||||
|
||||
Rollback:
|
||||
|
||||
- Prefer fail-fast ambiguity policy over auto-merge.
|
||||
|
||||
---
|
||||
|
||||
## 5. Minimal Viable v1.0 Scope
|
||||
|
||||
Must-have for v1.0:
|
||||
|
||||
1. Campaign-aware canonical session roots.
|
||||
2. Session manifest remains skip/resume authority.
|
||||
3. Introduce run manifests + run-local stage records.
|
||||
4. Immediate promotion from run-local outputs to canonical session outputs.
|
||||
5. Minimal downstream invalidation on forced upstream rerun.
|
||||
6. Archive/local semantic alignment with `current/*` behavior preserved.
|
||||
7. Backward-compatible analyze aliases (`processed_transcript`, `normalized_transcript`, `trimmed_transcript`).
|
||||
|
||||
Nice-to-have / defer if risky:
|
||||
|
||||
1. Full checksum-based stale detection graph.
|
||||
2. Broad manifest schema-version migration framework.
|
||||
3. Full artifact-registry rollout beyond analyze’s initial needs.
|
||||
4. Aggressive cleanup of all legacy fallback branches in one release.
|
||||
|
||||
---
|
||||
|
||||
## 6. Open Questions (with Recommendations)
|
||||
|
||||
1. **Should campaign be mandatory for all local path derivation immediately?**
|
||||
- Recommendation: yes for new layout writes; keep controlled read compatibility for legacy session-only roots during migration window.
|
||||
|
||||
2. **Session manifest location transition policy:** auto-migrate vs explicit migrate command?
|
||||
- Recommendation: if user base is small, prefer explicit fail-fast with actionable migration instructions to avoid silent split-brain state.
|
||||
|
||||
3. **Run manifest granularity:** per-stage detailed records vs summary + references?
|
||||
- Recommendation: start with summary + stage status/paths; avoid duplicating full session artifact state to keep write path simple.
|
||||
|
||||
4. **Invalidation marking:** use `stale` status now or clear `succeeded` markers?
|
||||
- Recommendation: if invasive to propagate new status semantics quickly, clear/overwrite downstream success states first; add formal `stale` usage in follow-up.
|
||||
|
||||
5. **Promotion timing:** promote every stage immediately vs delayed at end of run?
|
||||
- Recommendation: immediate per-stage promotion after validation (matches current idempotent skip expectations and simplifies resume behavior).
|
||||
|
||||
6. **Docs consistency order after implementation starts:**
|
||||
- Recommendation: update `README.md` and `architecture.md` in lockstep with each migration step. Current conflict is explicit:
|
||||
- code/README/architecture still primarily describe session roots at `work/{session}`
|
||||
- `docs/development/workspace.md` defines `work/{campaign}/{session}` plus run manifests.
|
||||
|
||||
@@ -1,753 +0,0 @@
|
||||
# Narratio Workspace, Run History, and Artifact Resolution Architecture
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
This document defines the intended v1.0 architecture for Narratio's local workspace layout, run history model, durable session outputs, manifest responsibilities, and artifact resolution contract.
|
||||
|
||||
Narratio is an idempotent session orchestrator. The command:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07
|
||||
```
|
||||
|
||||
means "bring the identified session to its desired completed state." It does **not** mean "always create an entirely new independent output tree and ignore prior session state."
|
||||
|
||||
This distinction drives the architecture:
|
||||
|
||||
* A **session** is the durable domain object and idempotency boundary.
|
||||
* A **run** is an execution attempt that may update the session's durable state.
|
||||
* Durable outputs live at the session level.
|
||||
* Run-specific outputs, logs, generated configs, scratch files, and diagnostics live under `runs/{run_id}/`.
|
||||
* Successful stage outputs are promoted from run-local locations into canonical session-level locations.
|
||||
* The session manifest records current durable state.
|
||||
* Run manifests record execution history and debugging/provenance details.
|
||||
|
||||
This model intentionally mirrors the S3 archive model: session-level current artifacts are distinct from run-record history.
|
||||
|
||||
## 2. Core Concepts
|
||||
|
||||
### 2.1 Session
|
||||
|
||||
A session is the stable unit of work identified by `campaign_id` and `session_id`.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
campaign_id = dilfs
|
||||
session_id = 2026-05-07
|
||||
```
|
||||
|
||||
The session directory represents the current durable local state for that session. Re-running Narratio for the same session should consult this state, skip already-completed stages by default, and produce no changes unless work is incomplete, stale, forced, or explicitly selected.
|
||||
|
||||
### 2.2 Run
|
||||
|
||||
A run is a particular execution attempt identified by a generated `run_id`, for example:
|
||||
|
||||
```text
|
||||
20260517T174748Z-abcd1234
|
||||
```
|
||||
|
||||
A run may execute all stages or only a sparse subset of stages. Sparse runs are expected and desirable when the user invokes `--force`, `run-stage`, or a stage-limited command.
|
||||
|
||||
Run directories are provenance/debug records. They should reflect what actually happened during that invocation, not a synthetic complete pipeline layout.
|
||||
|
||||
### 2.3 Durable Output
|
||||
|
||||
A durable output is a canonical session-level artifact intended for later stages, user consumption, archive promotion, or future idempotency decisions.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
transcripts/merged.json
|
||||
transcripts/processed.json
|
||||
transcripts/normalized.json
|
||||
transcripts/trimmed.json
|
||||
artifacts/session_recap.md
|
||||
```
|
||||
|
||||
Durable outputs live directly under the session directory, not under a particular run directory.
|
||||
|
||||
### 2.4 Run-Local Output
|
||||
|
||||
A run-local output is the file initially produced by a stage during a specific run. After validation, durable outputs are promoted from run-local paths to session-level canonical paths.
|
||||
|
||||
Run-local outputs, logs, generated configs, reports, and scratch files should remain under:
|
||||
|
||||
```text
|
||||
runs/{run_id}/{stage}/...
|
||||
```
|
||||
|
||||
## 3. Local Workspace Layout
|
||||
|
||||
The canonical local workspace layout is:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/
|
||||
manifest.json
|
||||
current/
|
||||
manifest.json
|
||||
run_id.txt
|
||||
inputs/
|
||||
transcripts/
|
||||
artifacts/
|
||||
reports/
|
||||
logs/
|
||||
config/
|
||||
runs/
|
||||
{run_id}/
|
||||
manifest.json
|
||||
prepare/
|
||||
transcribe/
|
||||
merge/
|
||||
polish/
|
||||
normalize/
|
||||
trim/
|
||||
analyze/
|
||||
archive/
|
||||
notify/
|
||||
```
|
||||
|
||||
Not every directory must exist at all times. Directories should be created idempotently when needed.
|
||||
|
||||
### 3.1 Session Root
|
||||
|
||||
The session root is:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/
|
||||
```
|
||||
|
||||
The session root is the stable local home for the session. It is the default base for resolving canonical artifact paths.
|
||||
|
||||
The only files that should live directly in the session root are core session-state files, primarily:
|
||||
|
||||
```text
|
||||
manifest.json
|
||||
```
|
||||
|
||||
Lock files may also be session-root scoped if the implementation uses file locks there, but transient locks should not be treated as durable artifacts.
|
||||
|
||||
### 3.2 Session-Level Canonical Directories
|
||||
|
||||
The following directories contain current durable session state:
|
||||
|
||||
```text
|
||||
inputs/
|
||||
transcripts/
|
||||
artifacts/
|
||||
reports/
|
||||
logs/
|
||||
config/
|
||||
current/
|
||||
```
|
||||
|
||||
Recommended meanings:
|
||||
|
||||
| Directory | Purpose |
|
||||
| -------------- | ----------------------------------------------------------------------------- |
|
||||
| `inputs/` | Materialized or copied input files used by the current durable session state. |
|
||||
| `transcripts/` | Canonical transcript tiers. |
|
||||
| `artifacts/` | User-facing and machine-readable generated artifacts. |
|
||||
| `reports/` | Canonical stage reports worth preserving at the session level. |
|
||||
| `logs/` | Optional session-level logs or promoted/latest logs. |
|
||||
| `config/` | Optional session-level generated config snapshots or promoted/latest configs. |
|
||||
| `current/` | Current published session pointers, mirroring the archive backend. |
|
||||
|
||||
Canonical durable outputs should use stable paths under these directories.
|
||||
|
||||
### 3.3 Run History Directory
|
||||
|
||||
Run history lives under:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/
|
||||
```
|
||||
|
||||
Each run directory records what happened during that invocation. A run may contain all stage directories or only a sparse subset.
|
||||
|
||||
Example full run:
|
||||
|
||||
```text
|
||||
runs/20260517T174748Z-abcd1234/
|
||||
manifest.json
|
||||
prepare/
|
||||
transcribe/
|
||||
merge/
|
||||
polish/
|
||||
normalize/
|
||||
trim/
|
||||
analyze/
|
||||
archive/
|
||||
notify/
|
||||
```
|
||||
|
||||
Example sparse forced analyze run:
|
||||
|
||||
```text
|
||||
runs/20260518T030000Z-efgh5678/
|
||||
manifest.json
|
||||
analyze/
|
||||
```
|
||||
|
||||
Example sparse polish-through-analyze rerun:
|
||||
|
||||
```text
|
||||
runs/20260518T041500Z-a1b2c3d4/
|
||||
manifest.json
|
||||
polish/
|
||||
normalize/
|
||||
trim/
|
||||
analyze/
|
||||
```
|
||||
|
||||
Run directories should not create stage folders for stages that were not selected, executed, skipped, or otherwise considered during that run unless there is a clear diagnostic reason to do so.
|
||||
|
||||
### 3.4 Stage Run-Local Directories
|
||||
|
||||
Each stage receives a run-local directory:
|
||||
|
||||
```text
|
||||
runs/{run_id}/{stage}/
|
||||
```
|
||||
|
||||
Within that stage directory, the stage may use subdirectories such as:
|
||||
|
||||
```text
|
||||
outputs/
|
||||
logs/
|
||||
reports/
|
||||
config/
|
||||
scratch/
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
runs/{run_id}/polish/
|
||||
outputs/transcripts/processed.json
|
||||
reports/audita.polish.report.json
|
||||
logs/stdout.log
|
||||
logs/stderr.log
|
||||
config/audita.polish.generated.yml
|
||||
scratch/
|
||||
```
|
||||
|
||||
The exact internal layout of a stage directory may vary by stage, but it should be deterministic, documented, and generated through centralized path helpers rather than ad hoc path joins.
|
||||
|
||||
## 4. Promotion Model
|
||||
|
||||
Narratio uses stage-level promotion with immediate promotion after successful validation.
|
||||
|
||||
The stage lifecycle is:
|
||||
|
||||
1. Resolve required inputs from the current session state and/or run-local context.
|
||||
2. Create the run-local stage directory.
|
||||
3. Execute the stage, writing outputs under `runs/{run_id}/{stage}/...`.
|
||||
4. Validate run-local outputs.
|
||||
5. Promote durable outputs into session-level canonical paths.
|
||||
6. Update the session manifest.
|
||||
7. Update the run manifest.
|
||||
|
||||
Promotion means an atomic or effectively atomic copy/rename from a run-local path to a session-level canonical path.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
runs/{run_id}/polish/outputs/transcripts/processed.json
|
||||
```
|
||||
|
||||
is promoted to:
|
||||
|
||||
```text
|
||||
transcripts/processed.json
|
||||
```
|
||||
|
||||
Promotion should be safe and deterministic:
|
||||
|
||||
* Validate before promotion.
|
||||
* Write promoted files atomically where possible.
|
||||
* Never leave partially written durable outputs.
|
||||
* Record the producing `run_id` in the session manifest.
|
||||
* Preserve run-local files for debugging unless retention policy deletes them.
|
||||
|
||||
## 5. Promotion Policy: Option A
|
||||
|
||||
Narratio uses immediate stage-level promotion.
|
||||
|
||||
If a selected stage succeeds, its durable outputs are promoted immediately, even if a later selected stage fails.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07 --force --stages polish,normalize,trim,analyze
|
||||
```
|
||||
|
||||
If `polish` succeeds and `normalize` fails:
|
||||
|
||||
* `transcripts/processed.json` may be updated from the new run.
|
||||
* `normalize`, `trim`, and `analyze` should not be marked succeeded for the new input state.
|
||||
* Downstream outputs may now be stale relative to the newly promoted polished transcript.
|
||||
|
||||
This policy is simpler, transparent, and consistent with stage-level resumability. It does require explicit stale/invalidation handling.
|
||||
|
||||
## 6. Stale and Invalidation Semantics
|
||||
|
||||
Full checksum-based stale detection may be implemented later. Before that exists, Narratio should still use a simple deterministic invalidation rule for forced or explicit upstream reruns.
|
||||
|
||||
When a stage is successfully re-executed and promoted, downstream stages should be marked stale unless they are also re-executed successfully in the same command invocation.
|
||||
|
||||
Example stage order:
|
||||
|
||||
```text
|
||||
prepare -> transcribe -> merge -> polish -> normalize -> trim -> analyze -> archive -> notify
|
||||
```
|
||||
|
||||
If `polish` is forced and promoted, then the following downstream stages should be invalidated unless rerun successfully:
|
||||
|
||||
```text
|
||||
normalize
|
||||
trim
|
||||
analyze
|
||||
archive
|
||||
notify
|
||||
```
|
||||
|
||||
A stale stage is not equivalent to a failed stage. It means its current durable outputs may no longer correspond to current upstream inputs or configuration.
|
||||
|
||||
Minimum manifest state model:
|
||||
|
||||
```text
|
||||
pending
|
||||
running
|
||||
succeeded
|
||||
failed
|
||||
skipped
|
||||
stale
|
||||
```
|
||||
|
||||
If adding a new `stale` state is too invasive for v1.0, the implementation should at least record stale metadata or clear downstream success markers in a way that prevents accidental idempotent skips based on obsolete outputs.
|
||||
|
||||
## 7. Manifest Responsibilities
|
||||
|
||||
Narratio should distinguish between session manifests and run manifests.
|
||||
|
||||
The same underlying Go types may be reused where practical, but the concepts should remain separate.
|
||||
|
||||
### 7.1 Session Manifest
|
||||
|
||||
Path:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
|
||||
```
|
||||
|
||||
The session manifest answers:
|
||||
|
||||
```text
|
||||
What is the current durable state of this session?
|
||||
```
|
||||
|
||||
It should record:
|
||||
|
||||
* campaign ID
|
||||
* session ID
|
||||
* current or latest run ID
|
||||
* current stage states
|
||||
* canonical durable output refs
|
||||
* artifact IDs and paths
|
||||
* producing run ID for each current stage output
|
||||
* relevant input/config checksums when available
|
||||
* stale/invalidated stage information
|
||||
* archive/current publication metadata
|
||||
|
||||
A session's durable state may be a composite of multiple runs.
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
transcripts/merged.json produced by run A
|
||||
transcripts/processed.json produced by run B
|
||||
transcripts/normalized.json produced by run B
|
||||
transcripts/trimmed.json produced by run B
|
||||
artifacts/session_recap.md produced by run C
|
||||
```
|
||||
|
||||
This is valid and expected.
|
||||
|
||||
### 7.2 Run Manifest
|
||||
|
||||
Path:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/manifest.json
|
||||
```
|
||||
|
||||
The run manifest answers:
|
||||
|
||||
```text
|
||||
What happened during this specific execution attempt?
|
||||
```
|
||||
|
||||
It should record:
|
||||
|
||||
* run ID
|
||||
* campaign ID
|
||||
* session ID
|
||||
* command mode and selected stages
|
||||
* force flags or stage selection flags
|
||||
* stages considered during this run
|
||||
* stages executed during this run
|
||||
* stages skipped during this run and reasons
|
||||
* run-local output paths
|
||||
* promoted output paths
|
||||
* logs
|
||||
* reports
|
||||
* generated configs
|
||||
* timings
|
||||
* errors
|
||||
* non-secret subprocess invocation metadata
|
||||
|
||||
Run manifests are primarily for debugging, auditability, and archive history.
|
||||
|
||||
## 8. Idempotency and Resume Behavior
|
||||
|
||||
The idempotency boundary is the session, not the run.
|
||||
|
||||
By default:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07
|
||||
```
|
||||
|
||||
should consult the session manifest and skip stages that are already succeeded and not stale.
|
||||
|
||||
If all stages are already complete, the command should execute zero stages and report that the session is already complete.
|
||||
|
||||
Forced execution creates a new run record but updates session-level durable state only for stages that actually succeed and promote outputs.
|
||||
|
||||
Examples:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07 --force
|
||||
```
|
||||
|
||||
Creates a new run and attempts to re-execute the selected/default stage set.
|
||||
|
||||
```bash
|
||||
narratio run-stage --session-id 2026-05-07 analyze --force
|
||||
```
|
||||
|
||||
Creates a sparse run that executes only `analyze`, then promotes updated analysis artifacts if successful.
|
||||
|
||||
```bash
|
||||
narratio resume --session-id 2026-05-07
|
||||
```
|
||||
|
||||
Uses the session manifest to determine what remains incomplete or stale. Resume does not need to resume the same `run_id` unless the implementation explicitly supports resuming an interrupted active run.
|
||||
|
||||
## 9. Artifact Resolution Contract
|
||||
|
||||
Narratio should provide a first-class artifact registry and resolver.
|
||||
|
||||
The resolver maps symbolic artifact source names to canonical session-level paths and manifest output kinds.
|
||||
|
||||
Stages and adapters should not hardcode path fragments when resolving cross-stage inputs. They should ask the artifact resolver for the current durable artifact by ID.
|
||||
|
||||
### 9.1 Canonical Artifact IDs
|
||||
|
||||
Preferred artifact IDs should be namespaced:
|
||||
|
||||
```text
|
||||
narratio.transcript.merged
|
||||
narratio.transcript.polished
|
||||
narratio.transcript.full
|
||||
narratio.transcript.trimmed
|
||||
narratio.bounds.session
|
||||
narratio.artifact.session_recap
|
||||
```
|
||||
|
||||
Recommended initial registry:
|
||||
|
||||
| Artifact ID | Canonical Path | Producer Stage | Output Kind | Meaning |
|
||||
| --------------------------------- | ------------------------------- | -------------- | ------------------------ | ------------------------------------- |
|
||||
| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` | Deterministic Seriatim merge. |
|
||||
| `narratio.transcript.polished` | `transcripts/processed.json` | `polish` | `transcript_processed` | Full Audita-polished transcript. |
|
||||
| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` | Preferred full normalized transcript. |
|
||||
| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` | Gameplay-only transcript. |
|
||||
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` | Trim bounds selected for the session. |
|
||||
| `narratio.artifact.session_recap` | `artifacts/session_recap.md` | `analyze` | `artifact_session_recap` | Generated session recap. |
|
||||
|
||||
### 9.2 Backward-Compatible Aliases
|
||||
|
||||
Existing source names should remain supported:
|
||||
|
||||
| Legacy Source | Preferred Artifact ID |
|
||||
| ----------------------- | ------------------------------ |
|
||||
| `processed_transcript` | `narratio.transcript.polished` |
|
||||
| `normalized_transcript` | `narratio.transcript.full` |
|
||||
| `trimmed_transcript` | `narratio.transcript.trimmed` |
|
||||
|
||||
These aliases may be supported silently for v1.0. Documentation should prefer namespaced IDs.
|
||||
|
||||
### 9.3 Resolver Behavior
|
||||
|
||||
Artifact resolution should follow this order:
|
||||
|
||||
1. Normalize aliases to canonical artifact IDs.
|
||||
2. Look for a current output reference in the session manifest.
|
||||
3. Fall back to the canonical session-level path.
|
||||
4. If the artifact is required, fail clearly if missing.
|
||||
5. If the artifact is optional and missing, omit it from the downstream invocation.
|
||||
6. Validate the artifact using the expected content validator.
|
||||
7. Return a resolved artifact record containing ID, path, producer stage, output kind, and provenance.
|
||||
|
||||
Example conceptual result:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "narratio.transcript.trimmed",
|
||||
"path": "/var/lib/narratio/work/dilfs/2026-05-07/transcripts/trimmed.json",
|
||||
"producer_stage": "trim",
|
||||
"producer_run_id": "20260517T174748Z-abcd1234",
|
||||
"output_kind": "transcript_trimmed",
|
||||
"content_type": "application/json"
|
||||
}
|
||||
```
|
||||
|
||||
### 9.4 Artifact Validation
|
||||
|
||||
Transcript artifacts must be valid JSON with a top-level `segments` array.
|
||||
|
||||
Markdown/text artifacts must exist and be non-empty when required.
|
||||
|
||||
Bounds artifacts must match the expected bounds schema and refer to segment IDs in the same transcript ID space used by the trim stage.
|
||||
|
||||
Validation should happen before a resolved artifact is passed to another stage or external subprocess.
|
||||
|
||||
## 10. Analyze Stage Implications
|
||||
|
||||
The analyze stage should consume artifacts through the artifact resolver.
|
||||
|
||||
Preferred Scriptorium config shape:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: "dnd.session_recap"
|
||||
output_path: "artifacts/session_recap.md"
|
||||
inputs:
|
||||
transcript:
|
||||
source: "narratio.transcript.trimmed"
|
||||
required: true
|
||||
```
|
||||
|
||||
Additional artifacts can choose different transcript tiers:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
table_summary:
|
||||
enabled: true
|
||||
prompt_id: "dnd.table_summary"
|
||||
output_path: "artifacts/table_summary.md"
|
||||
inputs:
|
||||
transcript:
|
||||
source: "narratio.transcript.full"
|
||||
required: true
|
||||
```
|
||||
|
||||
For v1.0, Narratio does not need a generic DAG engine. It may execute configured analyze artifacts in deterministic order and allow later artifacts to consume earlier artifacts only when that relationship is explicit and unambiguous.
|
||||
|
||||
Rules:
|
||||
|
||||
* Artifact inputs resolve from current session-level durable state.
|
||||
* Outputs are first written run-locally.
|
||||
* Successful analyze outputs are promoted to session-level `artifacts/` paths.
|
||||
* Manifest output refs record the producing run ID.
|
||||
* Optional inputs are omitted when unavailable.
|
||||
* Required missing inputs fail before invoking Scriptorium.
|
||||
|
||||
## 11. Archive Alignment
|
||||
|
||||
Local workspace semantics should mirror archive semantics.
|
||||
|
||||
Local session-level durable paths:
|
||||
|
||||
```text
|
||||
work/{campaign}/{session}/transcripts/trimmed.json
|
||||
work/{campaign}/{session}/artifacts/session_recap.md
|
||||
work/{campaign}/{session}/current/manifest.json
|
||||
work/{campaign}/{session}/current/run_id.txt
|
||||
work/{campaign}/{session}/runs/{run_id}/...
|
||||
```
|
||||
|
||||
should map naturally to remote archive paths:
|
||||
|
||||
```text
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/transcripts/trimmed.json
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/artifacts/session_recap.md
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/manifest.json
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/run_id.txt
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/runs/{run_id}/...
|
||||
```
|
||||
|
||||
The archive stage should publish run records and promoted current artifacts consistently with the local model.
|
||||
|
||||
`current/run_id.txt` remains the effective commit marker for the archived current session state.
|
||||
|
||||
## 12. Path Helper Requirements
|
||||
|
||||
All code should use centralized path helpers for workspace paths.
|
||||
|
||||
Stage code should not manually assemble durable cross-stage paths using raw string joins except through the path model.
|
||||
|
||||
Recommended helper surface:
|
||||
|
||||
```text
|
||||
SessionRoot(campaignID, sessionID)
|
||||
SessionManifestPath(campaignID, sessionID)
|
||||
SessionCurrentDir(campaignID, sessionID)
|
||||
SessionTranscriptsDir(campaignID, sessionID)
|
||||
SessionArtifactsDir(campaignID, sessionID)
|
||||
SessionReportsDir(campaignID, sessionID)
|
||||
SessionLogsDir(campaignID, sessionID)
|
||||
SessionConfigDir(campaignID, sessionID)
|
||||
RunsDir(campaignID, sessionID)
|
||||
RunRoot(campaignID, sessionID, runID)
|
||||
RunManifestPath(campaignID, sessionID, runID)
|
||||
RunStageDir(campaignID, sessionID, runID, stage)
|
||||
RunStageOutputsDir(campaignID, sessionID, runID, stage)
|
||||
RunStageLogsDir(campaignID, sessionID, runID, stage)
|
||||
RunStageReportsDir(campaignID, sessionID, runID, stage)
|
||||
RunStageConfigDir(campaignID, sessionID, runID, stage)
|
||||
CanonicalArtifactPath(campaignID, sessionID, artifactID)
|
||||
```
|
||||
|
||||
Path helpers should enforce safe relative paths for configured output paths:
|
||||
|
||||
* reject absolute paths unless explicitly allowed for a particular config field
|
||||
* reject `..` traversal
|
||||
* normalize separators
|
||||
* preserve deterministic output paths
|
||||
|
||||
## 13. Directory Creation Policy
|
||||
|
||||
Directory creation should be centralized and idempotent.
|
||||
|
||||
Recommended policy:
|
||||
|
||||
* `prepare` ensures the baseline session directory structure exists.
|
||||
* Every stage also calls shared layout helpers to ensure its required run-local directories exist before writing.
|
||||
* `run-stage` should not depend on a prior `prepare` invocation merely to create folders.
|
||||
* Missing directories should be created with appropriate permissions.
|
||||
* Directory creation should not imply stage success.
|
||||
|
||||
This provides consistent layout while keeping direct stage execution robust.
|
||||
|
||||
## 14. Cleanup and Retention
|
||||
|
||||
Cleanup must preserve the distinction between durable session state and run history.
|
||||
|
||||
Workspace cleanup after successful archive may remove selected local directories only according to explicit configuration.
|
||||
|
||||
Potential retention policies:
|
||||
|
||||
```text
|
||||
keep_all_runs
|
||||
keep_failed_runs
|
||||
keep_last_n_runs
|
||||
delete_run_after_success
|
||||
```
|
||||
|
||||
For v1.0, conservative retention is preferred:
|
||||
|
||||
* Do not delete durable session-level outputs unless explicitly requested.
|
||||
* Do not delete failed run directories by default.
|
||||
* If cleanup is enabled, remove only documented run-scoped or spool-scoped paths.
|
||||
* Local development audio inputs must never be deleted by workspace cleanup.
|
||||
|
||||
## 15. Migration From Existing Layout
|
||||
|
||||
Existing installations may currently use a simpler path such as:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{session_id}/manifest.json
|
||||
```
|
||||
|
||||
The v1.0 layout introduces campaign-aware session roots:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
|
||||
```
|
||||
|
||||
Migration options:
|
||||
|
||||
1. Best-effort automatic discovery of legacy session manifests.
|
||||
2. A one-time migration command.
|
||||
3. Clear release notes requiring users to move or regenerate workspace state.
|
||||
|
||||
For v1.0, it is acceptable to require explicit migration if the user base is small and the archive contains the authoritative durable outputs. However, the application should fail clearly when it detects an ambiguous legacy layout rather than silently creating duplicate state.
|
||||
|
||||
## 16. Documentation Updates Required
|
||||
|
||||
The following documentation should be updated to reflect this architecture:
|
||||
|
||||
* `README.md`
|
||||
* `docs/architecture.md`
|
||||
* a dedicated workspace/run-history document, such as this file
|
||||
* S3/archive documentation
|
||||
* analyze/artifact configuration documentation
|
||||
* example pipeline files
|
||||
|
||||
Documentation should consistently use the following terms:
|
||||
|
||||
| Term | Meaning |
|
||||
| ---------------- | ---------------------------------------------------------------------- |
|
||||
| Session | Durable domain object and idempotency boundary. |
|
||||
| Run | Execution attempt that may update session state. |
|
||||
| Durable output | Canonical current session-level output. |
|
||||
| Run-local output | Output produced inside a specific run directory before promotion. |
|
||||
| Promotion | Validated copy/rename from run-local output to durable session output. |
|
||||
| Session manifest | Current durable state of the session. |
|
||||
| Run manifest | Execution record for a particular run. |
|
||||
| Artifact ID | Symbolic source name resolved by the artifact registry. |
|
||||
|
||||
## 17. Architectural Invariants
|
||||
|
||||
The following invariants should hold after implementation:
|
||||
|
||||
1. `session_id` remains the idempotency boundary for normal operator commands.
|
||||
2. `run_id` identifies an execution attempt, not the primary durable workspace.
|
||||
3. Session-level canonical artifacts are the default inputs for downstream stages.
|
||||
4. Run-local outputs are promoted only after validation.
|
||||
5. A session's current durable state may be composed of outputs from multiple runs.
|
||||
6. Sparse run directories are valid and expected.
|
||||
7. The session manifest records current stage/artifact state and producer run IDs.
|
||||
8. The run manifest records what happened during one invocation.
|
||||
9. Artifact consumers resolve symbolic artifact IDs through a registry/resolver.
|
||||
10. Local workspace semantics mirror S3 archive semantics.
|
||||
11. Directory creation is centralized and idempotent.
|
||||
12. Stage code uses path helpers rather than ad hoc path construction.
|
||||
13. Forced upstream reruns invalidate downstream stage success unless downstream stages are rerun successfully.
|
||||
14. Cleanup never removes durable session outputs or local development inputs unless explicitly configured to do so.
|
||||
|
||||
## 18. Implementation Guidance
|
||||
|
||||
A practical implementation sequence is:
|
||||
|
||||
1. Add this architecture document.
|
||||
2. Add or revise path model helpers for session roots, run roots, stage directories, and canonical artifact paths.
|
||||
3. Introduce session manifest versus run manifest concepts.
|
||||
4. Route stage outputs through run-local directories.
|
||||
5. Add promotion helpers with validation and atomic writes.
|
||||
6. Update existing stages to promote durable outputs to session-level canonical paths.
|
||||
7. Add artifact registry and resolver.
|
||||
8. Update analyze to use artifact IDs and aliases.
|
||||
9. Add simple downstream stale invalidation for forced upstream reruns.
|
||||
10. Align archive/local path behavior and documentation.
|
||||
11. Update examples and README.
|
||||
12. Add tests for idempotency, sparse forced runs, promotion, manifest provenance, and artifact resolution.
|
||||
|
||||
This sequence intentionally avoids introducing a generic DAG engine. The v1.0 goal is a clear, deterministic, stage-oriented orchestrator with stable session-level outputs and inspectable run history.
|
||||
356
docs/documentation/policy.md
Normal file
356
docs/documentation/policy.md
Normal file
@@ -0,0 +1,356 @@
|
||||
# Go Project Documentation Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
Project documentation must help four audiences:
|
||||
|
||||
1. users who need to run the application;
|
||||
2. administrators/operators who need to configure and operate it;
|
||||
3. developers who need to understand and change it safely;
|
||||
4. LLM coding agents that need clear scope, boundaries, and invariants.
|
||||
|
||||
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
|
||||
|
||||
## Core Rules
|
||||
|
||||
### 1. Keep docs concise
|
||||
|
||||
Each document should cover a defined scope and only the essentials for that scope.
|
||||
|
||||
Avoid:
|
||||
- long background explanations;
|
||||
- repeated reference material;
|
||||
- implementation detail in user-facing docs;
|
||||
- aspirational language outside roadmap docs;
|
||||
- verbose examples where one minimal example is clearer.
|
||||
|
||||
### 2. Document only implemented behavior outside roadmap files
|
||||
|
||||
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
|
||||
|
||||
- `docs/roadmap/`
|
||||
|
||||
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
|
||||
|
||||
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
|
||||
|
||||
### 3. Use canonical homes
|
||||
|
||||
Each type of information should have one canonical location.
|
||||
|
||||
Canonical homes:
|
||||
|
||||
- project purpose and quickstart: `README.md`
|
||||
- development principles: `docs/architecture.md`
|
||||
- configuration reference: `docs/config.md`
|
||||
- CLI reference: `docs/cli.md`
|
||||
- operations and recovery: `docs/operations.md`
|
||||
- troubleshooting: `docs/troubleshooting.md`
|
||||
- implemented internals: `docs/internal/`
|
||||
- future work: `docs/roadmap/`
|
||||
- contributor workflow: `docs/development.md`
|
||||
- copyable examples: `examples/`
|
||||
|
||||
Other files should summarize briefly and link to the canonical source.
|
||||
|
||||
### 4. Keep examples real
|
||||
|
||||
Examples should be valid, maintained, and free of secrets.
|
||||
|
||||
Where practical:
|
||||
- example configs should load successfully;
|
||||
- example commands should match real CLI syntax;
|
||||
- important examples should be covered by tests.
|
||||
|
||||
## Documentation Profiles
|
||||
|
||||
All projects require:
|
||||
|
||||
- `README.md`
|
||||
- `docs/architecture.md`
|
||||
|
||||
Additional docs depend on the project.
|
||||
|
||||
### Small library
|
||||
|
||||
Recommended:
|
||||
- `docs/development.md`, if contributor conventions are non-obvious
|
||||
|
||||
### Simple CLI
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/development.md`
|
||||
|
||||
### Config-driven CLI
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`
|
||||
- `docs/config.md`
|
||||
|
||||
Recommended:
|
||||
- `examples/`
|
||||
- `docs/development.md`
|
||||
|
||||
### Stateful or operator-facing application
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`, if CLI-based
|
||||
- `docs/config.md`, if config-driven
|
||||
- `docs/operations.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/troubleshooting.md`
|
||||
- `examples/`
|
||||
- `docs/development.md`
|
||||
|
||||
### Modular, staged, service-oriented, or orchestration application
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`, if CLI-based
|
||||
- `docs/config.md`, if config-driven
|
||||
- `docs/operations.md`
|
||||
- `docs/internal/`
|
||||
- `docs/development.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/troubleshooting.md`
|
||||
- validated examples under `examples/`
|
||||
|
||||
## Required Documents
|
||||
|
||||
### README.md
|
||||
|
||||
**Audience:** users, administrators, operators
|
||||
|
||||
The README is the outward-facing project orientation page.
|
||||
|
||||
It should include, in order:
|
||||
|
||||
1. concise description;
|
||||
2. elevator pitch;
|
||||
3. shortest useful command or usage example;
|
||||
4. links to targeted docs.
|
||||
|
||||
The README should be short. It is not a manual.
|
||||
|
||||
The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.)
|
||||
|
||||
### docs/architecture.md
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
`docs/architecture.md` is required for every project.
|
||||
|
||||
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
|
||||
|
||||
It should include:
|
||||
|
||||
- project shape;
|
||||
- core design principles;
|
||||
- package and boundary philosophy;
|
||||
- state/persistence philosophy, if applicable;
|
||||
- external integration philosophy, if applicable;
|
||||
- error-handling and logging principles;
|
||||
- testing expectations;
|
||||
- documentation expectations;
|
||||
- architectural invariants;
|
||||
- explicit non-goals, if useful.
|
||||
|
||||
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
|
||||
|
||||
### docs/config.md
|
||||
|
||||
**Audience:** administrators, operators, advanced users
|
||||
|
||||
Required for applications with configuration files.
|
||||
|
||||
It should include, in order:
|
||||
|
||||
1. config file locations and discovery precedence;
|
||||
2. minimal working config;
|
||||
3. production-oriented config;
|
||||
4. full configuration reference;
|
||||
5. secrets handling, if applicable;
|
||||
6. links to maintained examples.
|
||||
|
||||
The full configuration reference should be canonical.
|
||||
|
||||
### docs/cli.md
|
||||
|
||||
**Audience:** users, administrators, operators
|
||||
|
||||
Required for CLI applications.
|
||||
|
||||
It should include, in order:
|
||||
|
||||
1. shortest useful command;
|
||||
2. command overview;
|
||||
3. complete flag reference;
|
||||
4. common workflows;
|
||||
5. diagnostic or recovery commands, if applicable.
|
||||
|
||||
Explain when commands are useful, not just their syntax.
|
||||
|
||||
### docs/operations.md
|
||||
|
||||
**Audience:** administrators, operators
|
||||
|
||||
Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures.
|
||||
|
||||
It should cover:
|
||||
|
||||
- normal workflow;
|
||||
- filesystem layout;
|
||||
- remote storage layout, if applicable;
|
||||
- logs and manifests;
|
||||
- resume/retry behavior;
|
||||
- cleanup behavior;
|
||||
- archive/backup behavior;
|
||||
- safe recovery procedures;
|
||||
- operational caveats.
|
||||
|
||||
### docs/troubleshooting.md
|
||||
|
||||
**Audience:** administrators, operators
|
||||
|
||||
Recommended once recurring failure modes exist.
|
||||
|
||||
Each entry should include:
|
||||
|
||||
- symptom;
|
||||
- likely cause;
|
||||
- diagnostic command or inspection step;
|
||||
- safe fix;
|
||||
- relevant links.
|
||||
|
||||
### docs/development.md
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
Required for projects maintained by humans and LLM coding agents.
|
||||
|
||||
It should include:
|
||||
|
||||
- repository layout;
|
||||
- build/test commands;
|
||||
- coding conventions;
|
||||
- dependency policy;
|
||||
- how to add config fields;
|
||||
- how to add CLI flags;
|
||||
- how to add stages/modules/adapters, if applicable;
|
||||
- how to update examples;
|
||||
- documentation update expectations.
|
||||
|
||||
### docs/internal/
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
Required for modular, staged, service-oriented, or orchestration projects.
|
||||
|
||||
This directory describes implemented internal components. It is not the roadmap.
|
||||
|
||||
Use one file per major component where useful.
|
||||
|
||||
Each component doc should include:
|
||||
|
||||
1. purpose;
|
||||
2. inputs and outputs;
|
||||
3. boundaries;
|
||||
4. config fields used;
|
||||
5. external adapters used;
|
||||
6. state or manifest behavior, if applicable;
|
||||
7. skip/resume behavior, if applicable;
|
||||
8. failure behavior;
|
||||
9. tests to inspect before changing;
|
||||
10. architectural invariants.
|
||||
|
||||
### docs/roadmap/
|
||||
|
||||
**Audience:** maintainers, developers, LLM coding agents
|
||||
|
||||
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
|
||||
|
||||
Roadmap docs should clearly distinguish:
|
||||
|
||||
- proposed work;
|
||||
- accepted plans;
|
||||
- deferred ideas;
|
||||
- rejected ideas;
|
||||
- implementation prompts or task breakdowns, if useful.
|
||||
|
||||
Roadmap docs should not be confused with current behavior.
|
||||
|
||||
### docs/integrations/
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
|
||||
|
||||
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses.
|
||||
|
||||
Use one file per integration where useful.
|
||||
|
||||
## Examples Directory
|
||||
|
||||
Projects with non-trivial configuration or workflows should include `examples/`.
|
||||
|
||||
Useful examples include:
|
||||
|
||||
- minimal working config;
|
||||
- production-oriented config;
|
||||
- full annotated config;
|
||||
- local development config;
|
||||
- remote/object-storage config;
|
||||
- minimal session/input file.
|
||||
|
||||
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
|
||||
|
||||
## Security and Privacy
|
||||
|
||||
Docs and examples must not include:
|
||||
|
||||
- real API keys;
|
||||
- tokens;
|
||||
- passwords;
|
||||
- private keys;
|
||||
- private environment dumps;
|
||||
- sensitive user data;
|
||||
- raw private transcripts;
|
||||
- private infrastructure details unless intentionally public.
|
||||
|
||||
Document secret-handling mechanisms, not actual secret values.
|
||||
|
||||
## Maintenance Rules
|
||||
|
||||
When docs change, verify the affected behavior.
|
||||
|
||||
Where practical:
|
||||
|
||||
- load example config files in tests;
|
||||
- test CLI examples or command parser behavior;
|
||||
- validate documented flags against real flags;
|
||||
- remove stale references;
|
||||
- update links after renames;
|
||||
- keep roadmap content out of non-roadmap docs.
|
||||
|
||||
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
|
||||
|
||||
Documentation is complete only when it matches the current code.
|
||||
|
||||
## Documentation Change Checklist
|
||||
|
||||
Before merging documentation changes, verify:
|
||||
|
||||
- README is concise and orientation-focused.
|
||||
- `docs/architecture.md` describes development principles.
|
||||
- Future work appears only under `docs/roadmap/`.
|
||||
- User-facing docs avoid unnecessary internals.
|
||||
- Developer-facing docs preserve boundaries and invariants.
|
||||
- Config examples match the schema.
|
||||
- CLI examples match real commands and flags.
|
||||
- Defaults appear in the canonical config reference.
|
||||
- No secrets or private data are included.
|
||||
- Links are accurate.
|
||||
15
docs/integrations/README.md
Normal file
15
docs/integrations/README.md
Normal file
@@ -0,0 +1,15 @@
|
||||
# Integration Documentation Index
|
||||
|
||||
## Audience
|
||||
Developers and LLM coding agents changing Narratio's external integration contracts.
|
||||
|
||||
## Scope
|
||||
Implemented-only reference notes for the external systems Narratio currently integrates with.
|
||||
|
||||
## Integration Docs
|
||||
- `audita.md`: Audita adapter invocation and validation contract.
|
||||
- `seriatim.md`: Seriatim normalize/merge/trim adapter contract.
|
||||
- `scriptorium.md`: Scriptorium run/render adapter contract.
|
||||
|
||||
## Canonical Owner
|
||||
`docs/integrations/` is the canonical home for external integration reference notes per `docs/documentation/policy.md`.
|
||||
@@ -1,96 +1,66 @@
|
||||
# Audita Subprocess Operations
|
||||
# Integration: audita
|
||||
|
||||
This document describes how parent processes should invoke `audita process` safely in production orchestration.
|
||||
## Purpose
|
||||
Define Narratio's adapter contract for transcript polishing via Audita CLI subprocess execution.
|
||||
|
||||
## Recommended command form
|
||||
## Inputs and Outputs
|
||||
Inputs (`audita.PolishRequest`):
|
||||
- merged transcript path
|
||||
- glossary path
|
||||
- output processed transcript path
|
||||
- optional report path (required when report enabled)
|
||||
- work dir
|
||||
- generated config path
|
||||
- stdout/stderr log paths
|
||||
- optional module/model/base URL and concurrency knobs
|
||||
|
||||
Use explicit file outputs for orchestrated runs:
|
||||
Outputs (`audita.PolishResult`):
|
||||
- processed transcript path
|
||||
- optional report path
|
||||
- generated config path
|
||||
- stdout/stderr log paths
|
||||
- exit code, duration, invoked binary
|
||||
- adapter metadata
|
||||
|
||||
```sh
|
||||
audita process <transcript.json> \
|
||||
--transcript-description "Brief context that may help resolve ambiguous terms." \
|
||||
--glossary <glossary.yaml> \
|
||||
--output <output-transcript.json> \
|
||||
--report-json <report.json>
|
||||
```
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Deterministic CLI argument construction for `audita process`
|
||||
- Environment bridging for API credentials
|
||||
- Invocation config emission
|
||||
- Output validation for processed transcript and report
|
||||
|
||||
Additional flags that may be situationally appropriate:
|
||||
- `--config <path>` to select an explicit versioned config file.
|
||||
- `--output-schema <bare-segments|audita-v1>` to select transcript output shape.
|
||||
- `--work-dir <dir>` to control diagnostics location.
|
||||
- `--work-dir-retention <always|auto|never>` to control retained run directories.
|
||||
- `--total-llm-concurrency`, `--proposal-llm-concurrency`, and `--validation-llm-concurrency` when orchestration needs to set explicit LLM throughput controls.
|
||||
- `--modules ...` only when intentionally overriding the default sequence.
|
||||
Does not own:
|
||||
- Upstream/downstream stage orchestration
|
||||
- Credential sourcing policy beyond required env-var presence check
|
||||
|
||||
For config-driven orchestration, validate config files in CI/preflight:
|
||||
## Config Fields Used
|
||||
Via `pipeline.audita.*` mapped in app/stage wiring:
|
||||
- `binary`, `timeout`, `llm_api_key_env`, `modules`, `base_url`, `model`
|
||||
- `transcript_description`, `config_path`, `output_schema`, `work_dir_retention`
|
||||
- `total_llm_concurrency`, `proposal_llm_concurrency`, `validation_model`, `validation_llm_concurrency`, `report`
|
||||
|
||||
```sh
|
||||
audita config validate --config <path>
|
||||
```
|
||||
## External Adapters Used
|
||||
- Shared subprocess helper (`internal/adapters/subprocess`) to run CLI and capture logs.
|
||||
|
||||
## Stdout behavior
|
||||
## State and Manifest Behavior
|
||||
- No direct manifest writes.
|
||||
- Stage-level metadata records adapter provenance and credential-present signal.
|
||||
- Generated invocation YAML is written when `GeneratedConfigPath` is provided.
|
||||
|
||||
- With `--output`: stdout is expected to be empty on success.
|
||||
- Without `--output`: stdout contains transcript JSON only on success.
|
||||
- Report JSON is never written to stdout.
|
||||
## Skip and Resume Behavior
|
||||
- Adapter has no skip/resume logic. Stage/runner controls this.
|
||||
|
||||
## Stderr behavior
|
||||
## Failure Behavior
|
||||
- Constructor validation fails on invalid binary/timeout/schema/concurrency/URL values.
|
||||
- Run fails on missing required paths, missing required credential env var, subprocess errors, invalid processed JSON shape, or invalid report JSON.
|
||||
- Failures preserve stdout/stderr paths in returned result metadata.
|
||||
|
||||
- Success path should be quiet or minimal human-readable logs.
|
||||
- Failure path writes concise human-readable errors.
|
||||
- When a diagnostics run directory exists, failure stderr includes its path.
|
||||
- Prompt/response diagnostic payloads are not streamed to stderr.
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/adapters/audita/subprocess_test.go`
|
||||
- `internal/adapters/audita/fake_test.go`
|
||||
- `internal/stage/polish_test.go`
|
||||
|
||||
## Output file behavior
|
||||
|
||||
- `--output` writes transcript JSON in the selected output schema to the provided path.
|
||||
- Output write failures return nonzero and surface actionable errors.
|
||||
- The command does not silently ignore output write errors.
|
||||
|
||||
## Report JSON behavior
|
||||
|
||||
- `--report-json` writes a machine-readable process report to the requested path.
|
||||
- Run-directory `report.json` is written independently under diagnostics.
|
||||
- Best-effort failure reports are emitted when possible without masking the primary failure.
|
||||
- Report write failures return nonzero with clear stderr messaging.
|
||||
- Report diagnostics metadata references run-directory artifacts including utilization diagnostics and correction ledger paths when available.
|
||||
|
||||
## Diagnostics directory behavior
|
||||
|
||||
- Each run creates (when possible) a per-run diagnostics directory.
|
||||
- Typical artifacts include transcript, normalization, chunking, invocation, effective config, LLM diagnostics, `utilization-diagnostics.json`, `correction-ledger.json`, `report.json`, and `error.log` on failure.
|
||||
- Failed runs retain diagnostics.
|
||||
- Under `auto` retention, successful runs with skipped/rejected corrections are retained; clean successful runs may be removed.
|
||||
|
||||
## Exit codes
|
||||
|
||||
- `0`: success.
|
||||
- Nonzero: failure (input/schema/config/module/LLM/runtime/output/report/diagnostics errors).
|
||||
|
||||
Treat any nonzero as a failed subprocess invocation.
|
||||
|
||||
## Timeout and cancellation
|
||||
|
||||
- Runtime operations propagate context cancellation and request timeouts through LLM/scheduler paths.
|
||||
- On cancellation or timeout, the process exits nonzero and should not hang.
|
||||
- If diagnostics were initialized before failure, failure artifacts remain available for debugging.
|
||||
|
||||
## Secret redaction expectations
|
||||
|
||||
API keys and configured secret values are redacted from:
|
||||
- reports (`--report-json` and run-dir `report.json`);
|
||||
- diagnostics artifacts (including effective config and LLM interaction artifacts);
|
||||
- surfaced adapter/runtime errors;
|
||||
- test fixtures and regression outputs.
|
||||
|
||||
Parent-process logs should still avoid printing raw environment variables.
|
||||
|
||||
## Parent-process pipe guidance
|
||||
|
||||
To avoid deadlocks in orchestrators:
|
||||
- always read both stdout and stderr concurrently when invoking as a subprocess;
|
||||
- prefer file outputs (`--output`, `--report-json`) for machine workflows;
|
||||
- treat stderr as human-readable diagnostics, not structured data;
|
||||
- parse structured results from output/report files.
|
||||
|
||||
For Go callers, prefer `exec.CommandContext` with explicit timeout/cancellation and buffered/streamed readers for both pipes.
|
||||
## Architectural Invariants
|
||||
- Processed output must be valid JSON with top-level `segments` array.
|
||||
- When report is enabled, report output must be valid JSON.
|
||||
- If `llm_api_key_env` is configured, credential must be present in environment.
|
||||
|
||||
@@ -1,339 +1,64 @@
|
||||
# Narratio -> Scriptorium CLI Integration
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
This document defines how Narratio should invoke Scriptorium through the **public CLI**.
|
||||
|
||||
This is a **subprocess integration contract**, not an internal Go API contract.
|
||||
|
||||
## 2. Assumptions
|
||||
|
||||
- `scriptorium` is installed and available on `PATH`.
|
||||
- Scriptorium is configured with `config.yml`.
|
||||
- `config.yml` provides `prompt_dir`, `profile_dir`, and `schema_dir` as needed.
|
||||
- Prompt and profile libraries are already deployed for the environment.
|
||||
- Narratio provides prepared artifact files (for example polished transcript, glossary, previous recap, campaign notes).
|
||||
- Initial integration is synchronous subprocess execution.
|
||||
- Narratio remains the orchestrator.
|
||||
|
||||
In normal operation, Narratio does not need to pass `--prompt-dir` and `--profile-dir` if they are supplied by Scriptorium config.
|
||||
|
||||
Narratio may pass `--config <PATH>` when it must use a non-default Scriptorium config file.
|
||||
|
||||
## 3. Core Commands Narratio May Call
|
||||
|
||||
Primary commands for subprocess integration:
|
||||
|
||||
- `scriptorium run`
|
||||
- `scriptorium render`
|
||||
|
||||
For production generation, use `scriptorium run`.
|
||||
|
||||
`scriptorium render` is for debugging, dry-runs, test assertions, and validating command construction without LLM execution.
|
||||
|
||||
Note: `scriptorium serve` and HTTP API exist, but they are not the initial integration path.
|
||||
|
||||
## 4. Command Selection Guidance
|
||||
|
||||
- Use `run` to generate an output artifact.
|
||||
- Use `render` to inspect the prepared prompt and effective settings without calling the LLM.
|
||||
- Use `render --format json` when Narratio/tests need structured prepare output.
|
||||
|
||||
## 5. Recommended `run` Invocation Shape
|
||||
|
||||
Production shape:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt <prompt_id> \
|
||||
--input transcript=<processed-transcript-path> \
|
||||
--out <output-artifact-path>
|
||||
```
|
||||
|
||||
Common optional additions:
|
||||
|
||||
- `--config <path>`: use a specific Scriptorium config file.
|
||||
- `--profile <profile_id>`: override prompt default profile.
|
||||
- `--var name=value` (repeatable): small metadata values.
|
||||
- `--input name=path` (repeatable): additional named artifacts.
|
||||
- `--timeout <duration>`: per-run timeout override.
|
||||
- Runtime model override flags (`--llm-base-url`, `--model`, etc.) only for exceptional/operator-directed cases.
|
||||
|
||||
## 6. Recommended `render` Invocation Shape
|
||||
|
||||
Human-readable debug shape:
|
||||
|
||||
```bash
|
||||
scriptorium render \
|
||||
--prompt <prompt_id> \
|
||||
--input transcript=<processed-transcript-path> \
|
||||
--format text
|
||||
```
|
||||
|
||||
Structured debug/test shape:
|
||||
|
||||
```bash
|
||||
scriptorium render \
|
||||
--prompt <prompt_id> \
|
||||
--input transcript=<processed-transcript-path> \
|
||||
--format json \
|
||||
--out <render-debug-path>
|
||||
```
|
||||
|
||||
`render` does **not** call the LLM, does **not** validate model output, and does **not** perform repair.
|
||||
|
||||
## 7. Inputs
|
||||
|
||||
- Pass inputs as repeated `--input name=path` flags.
|
||||
- `name` must match the Prompt Definition input name.
|
||||
- Prefer absolute paths, or paths relative to a working directory controlled by Narratio.
|
||||
- Pass Audita output as the primary transcript input.
|
||||
- Additional inputs may include glossary, previous recap, campaign notes, event logs, final state maps, or other prompt-specific artifacts.
|
||||
- Scriptorium reads input files directly; Narratio does not need to inline file content for CLI use.
|
||||
|
||||
## 8. Variables
|
||||
|
||||
Use repeated `--var name=value` for small metadata values.
|
||||
|
||||
Typical examples:
|
||||
|
||||
- `session_date`
|
||||
- `session_id`
|
||||
- `campaign_name`
|
||||
- `previous_session_id`
|
||||
- `output_kind`
|
||||
|
||||
Large content belongs in input files, not `--var` values.
|
||||
|
||||
## 9. Prompt IDs and Output Artifact Types
|
||||
|
||||
Narratio should treat prompt IDs as configuration, not hardcoded business logic.
|
||||
|
||||
Narratio config may map stage/output names to prompt IDs, for example:
|
||||
|
||||
- session recap prompt
|
||||
- structured event extraction prompt
|
||||
- glossary suggestion prompt
|
||||
- player-facing summary prompt
|
||||
|
||||
Prompt IDs used by Narratio should come from the deployed Scriptorium prompt library.
|
||||
|
||||
## 10. Profiles
|
||||
|
||||
- Prompts may declare `default_profile`.
|
||||
- Narratio may omit `--profile` to use prompt default profile.
|
||||
- Narratio may pass `--profile` to force profile selection.
|
||||
- This enables environment/profile selection like `local-fast`, `local-quality`, `frontier`, `batch`, or test profiles.
|
||||
- Profile names should generally be Narratio configuration values.
|
||||
|
||||
## 11. Runtime Overrides
|
||||
|
||||
Supported runtime override flags:
|
||||
|
||||
- `--llm-base-url`
|
||||
- `--model`
|
||||
- `--api-key-env`
|
||||
- `--temperature`
|
||||
- `--max-tokens`
|
||||
- `--top-p`
|
||||
- `--timeout`
|
||||
|
||||
Guidance:
|
||||
|
||||
- Keep normal model/runtime settings in Execution Profiles.
|
||||
- Use runtime overrides only for explicit per-run exceptions, tests, or operator overrides.
|
||||
- Never pass raw API keys on the command line.
|
||||
- `--api-key-env` names an environment variable; Narratio must ensure that variable is set in subprocess environment.
|
||||
|
||||
## 12. Config Behavior
|
||||
|
||||
- Default config path: `/etc/scriptorium/config.yml`.
|
||||
- `--config <PATH>` overrides default path.
|
||||
- Missing default config is allowed by Scriptorium.
|
||||
- If `--config` is provided explicitly, the file must exist and be valid.
|
||||
- CLI flags override `config.yml`.
|
||||
- `config.yml` overrides built-in application defaults.
|
||||
|
||||
Narratio can either:
|
||||
|
||||
- rely on system default config path, or
|
||||
- carry an explicit config path and pass `--config`.
|
||||
|
||||
## 13. Environment Handling
|
||||
|
||||
Subprocess environment recommendations:
|
||||
|
||||
- Pass through required API-key environment variables referenced by `api_key_env`.
|
||||
- Do not pass raw API keys as CLI arguments.
|
||||
- Avoid logging full environment dumps.
|
||||
- Capture stdout and stderr separately.
|
||||
- Use a controlled working directory.
|
||||
- Prefer absolute artifact paths.
|
||||
|
||||
## 14. Output Handling
|
||||
|
||||
For `scriptorium run`:
|
||||
|
||||
- Use `--out` when Narratio needs durable artifact files.
|
||||
- Without `--out`, artifact content is written to stdout.
|
||||
- Preferred orchestration pattern: always use `--out`, then treat the file as stage output artifact.
|
||||
- Capture stderr for diagnostics.
|
||||
|
||||
For `scriptorium render`:
|
||||
|
||||
- Use `--out` to store render diagnostics.
|
||||
- Use `--format json` when tests need to inspect selected profile, effective runtime settings, input hashes, prompt hash, and rendered messages.
|
||||
|
||||
## 15. Exit Status and Errors
|
||||
|
||||
Current CLI behavior (verified from implementation/tests):
|
||||
|
||||
- `0`: success.
|
||||
- `1`: runtime/parse/config/load/render/generation/IO error.
|
||||
- `2`: run completed but output validation failed (`ValidationFailed`).
|
||||
|
||||
Additional details:
|
||||
|
||||
- On `run`, output artifact write happens before exit code selection. If validation fails, artifact may still be written and exit code is `2`.
|
||||
- `stderr` carries both errors and normal run summary output; non-empty stderr alone does not imply failure.
|
||||
- `render` returns `0` on success and `1` on failures.
|
||||
|
||||
Narratio should treat non-zero exit codes as failed stage execution, but may record generated artifact paths if a run exited `2` and output file exists.
|
||||
|
||||
## 16. Recommended Narratio Integration Pattern
|
||||
|
||||
1. Build CLI args from Narratio stage configuration.
|
||||
2. Use subprocess context cancellation/timeout.
|
||||
3. Pass absolute input paths.
|
||||
4. Pass `--out` to a session-scoped artifact path.
|
||||
5. Add `--var` metadata values.
|
||||
6. Optionally add `--config`.
|
||||
7. Optionally add `--profile`.
|
||||
8. Ensure required API-key env vars are present.
|
||||
9. Run subprocess synchronously.
|
||||
10. Capture stdout/stderr separately.
|
||||
11. On success, store output artifact path and invocation metadata in stage artifacts.
|
||||
12. On failure, store exit code and stderr diagnostics in stage status.
|
||||
|
||||
## 17. Suggested Narratio Configuration Shape
|
||||
|
||||
Illustrative `pipeline.yml` shape:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
config_path: /etc/scriptorium/config.yml
|
||||
timeout: 10m
|
||||
render_debug: false
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
profile_id: local-quality # optional
|
||||
output_path: artifacts/session_recap.md
|
||||
timeout: 10m
|
||||
render_debug: false # optional artifact override
|
||||
inputs:
|
||||
transcript:
|
||||
source: trimmed_transcript
|
||||
required: true
|
||||
previous_recap:
|
||||
source: previous_session_artifact
|
||||
artifact: session_recap
|
||||
path: "" # optional
|
||||
required: false
|
||||
vars:
|
||||
session_id: true
|
||||
session_date: true
|
||||
campaign_name: true
|
||||
previous_session_id: true
|
||||
output_kind: session_recap
|
||||
```
|
||||
|
||||
The key idea: map Narratio artifact names to prompt ID, optional profile, expected inputs, vars, and output destination.
|
||||
|
||||
## 18. Testing Strategy for Narratio Integration
|
||||
|
||||
- Use `scriptorium render --format json` to verify command construction without LLM calls.
|
||||
- Use dedicated test prompt/profile libraries for integration tests.
|
||||
- Use small fixture transcripts.
|
||||
- Verify missing-input failure behavior.
|
||||
- Verify prompt `default_profile` behavior.
|
||||
- Verify explicit `--profile` override behavior.
|
||||
- Verify `--config` behavior (default and explicit).
|
||||
- Verify output file creation when `--out` is used.
|
||||
- Verify stderr capture on failures.
|
||||
- Avoid real API keys in tests.
|
||||
|
||||
## 19. Security and Privacy Notes
|
||||
|
||||
- Never pass raw API keys on command line.
|
||||
- Do not log full rendered prompts by default; transcripts may contain sensitive content.
|
||||
- Avoid logging prompt content unless explicit debug mode is enabled.
|
||||
- Treat generated artifacts as potentially sensitive.
|
||||
- Use session-scoped, access-controlled output paths.
|
||||
- `api_key_env` names should come from environment management, not embedded secrets.
|
||||
|
||||
## 20. Initial D&D Artifact Generation Examples
|
||||
|
||||
These are examples only. Use prompt IDs from the deployed prompt library.
|
||||
|
||||
Session recap:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.session_recap \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--input glossary=/work/session-42/glossary.yml \
|
||||
--out /work/session-42/artifacts/session_recap.md
|
||||
```
|
||||
|
||||
Structured events:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.structured_events \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--out /work/session-42/artifacts/structured_events.json
|
||||
```
|
||||
|
||||
Glossary suggestions:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.glossary_suggestions \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--input previous_recap=/work/session-41/artifacts/session_recap.md \
|
||||
--out /work/session-42/artifacts/glossary_suggestions.md
|
||||
```
|
||||
|
||||
Player-facing summary:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.player_summary \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--input structured_events=/work/session-42/artifacts/structured_events.json \
|
||||
--out /work/session-42/artifacts/player_summary.md
|
||||
```
|
||||
|
||||
## 21. Non-Goals
|
||||
|
||||
Initial Narratio integration should not:
|
||||
|
||||
- call Scriptorium internal Go packages
|
||||
- use HTTP API as the primary path
|
||||
- expect Scriptorium to read S3 refs directly
|
||||
- make Scriptorium responsible for Narratio stage state
|
||||
- make Scriptorium responsible for notification
|
||||
- require Scriptorium to understand D&D workflow semantics beyond prompt definitions
|
||||
|
||||
## 22. Future Extension Notes
|
||||
|
||||
Possible later extensions:
|
||||
|
||||
- HTTP API integration
|
||||
- S3 artifact references if Scriptorium adds S3 reader support
|
||||
- richer render diagnostics and policy controls
|
||||
- token budgeting/prompt-size checks
|
||||
- batch execution if Scriptorium later adds batch support
|
||||
# Integration: scriptorium
|
||||
|
||||
## Purpose
|
||||
Define Narratio's adapter contract for Scriptorium artifact generation and render-debug subprocess invocations.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `RunArtifactRequest`: binary, config path, prompt/profile IDs, input map, vars map, timeout, output path, logs/config paths, optional API env and working dir
|
||||
- `RenderArtifactRequest`: same core fields for render mode
|
||||
|
||||
Outputs (`ArtifactResult`):
|
||||
- output path
|
||||
- stdout/stderr log paths
|
||||
- generated config path
|
||||
- exit code and duration
|
||||
- command mode (`run` or `render`)
|
||||
- prompt/profile provenance
|
||||
- validation failure signal
|
||||
- adapter metadata
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Deterministic CLI arg construction for `scriptorium run` and `scriptorium render`
|
||||
- Common request validation
|
||||
- Invocation config emission
|
||||
- Output existence/non-empty checks
|
||||
- Validation-failure mapping for run exit code 2
|
||||
|
||||
Does not own:
|
||||
- Artifact selection policy (`analyze` stage)
|
||||
- Bounds semantic validation (`trim` stage)
|
||||
|
||||
## Config Fields Used
|
||||
Via `pipeline.scriptorium.*` and stage-level artifact config:
|
||||
- `binary`, `config_path`, `timeout`, `render_debug`
|
||||
- artifact-level `prompt_id`, `profile_id`, `timeout`, `inputs`, `vars`, `output_path`
|
||||
|
||||
## External Adapters Used
|
||||
- Shared subprocess helper (`internal/adapters/subprocess`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- No direct manifest writes.
|
||||
- Stage metadata records adapter outputs and command mode.
|
||||
- Generated invocation YAML is written when requested.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Adapter has no skip/resume logic. Stage/runner controls execution.
|
||||
|
||||
## Failure Behavior
|
||||
- Request validation fails for missing binary/prompt/output, invalid timeout, invalid input/var names, or missing required API env var.
|
||||
- Subprocess errors bubble with command context.
|
||||
- `run` exit code 2 is treated as `ValidationFailed=true` and surfaced as error by calling stage.
|
||||
- Successful subprocess still fails if output file is missing/empty.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
- `internal/adapters/scriptorium/fake_test.go`
|
||||
- `internal/stage/analyze_test.go`
|
||||
- `internal/stage/trim_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Both modes require explicit timeout > 0.
|
||||
- Input/var maps are sorted into deterministic CLI argument order.
|
||||
- Run-mode validation failures are represented explicitly, not silently skipped.
|
||||
|
||||
@@ -1,403 +1,60 @@
|
||||
# seriatim
|
||||
|
||||
`seriatim` merges per-speaker WhisperX-style JSON transcripts into a single JSON transcript that preserves speaker identity and chronological order.
|
||||
|
||||
The current implementation supports the `merge` command. It reads one or more input JSON files, optionally maps each input file to a canonical speaker using `speakers.yml`, sorts all segments by timestamp, detects and resolves overlaps when word-level timing is available, assigns consecutive numeric `id` values, and writes a merged JSON artifact.
|
||||
|
||||
## Usage
|
||||
|
||||
Run from source:
|
||||
|
||||
```sh
|
||||
go run ./cmd/seriatim merge \
|
||||
--input-file samples/raw/2026-04-19-Eric_Rakestraw.json \
|
||||
--input-file samples/raw/2026-04-19-Mike_Brown.json \
|
||||
--output-file merged.json
|
||||
```
|
||||
|
||||
Optional report output:
|
||||
|
||||
```sh
|
||||
go run ./cmd/seriatim merge \
|
||||
--input-file eric.json \
|
||||
--input-file mike.json \
|
||||
--output-file merged.json \
|
||||
--report-file report.json
|
||||
```
|
||||
|
||||
## CLI
|
||||
|
||||
```text
|
||||
seriatim merge [flags]
|
||||
```
|
||||
|
||||
Global flags:
|
||||
|
||||
| Flag | Description |
|
||||
| --- | --- |
|
||||
| `--help` | Show command help. |
|
||||
| `--version` | Show application version. Local builds default to `dev`; release builds inject the release version. |
|
||||
|
||||
`merge` flags:
|
||||
|
||||
| Flag | Required | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `--input-file` | Yes | none | Input transcript JSON file. Repeat once per speaker/input file. |
|
||||
| `--output-file` | Yes | none | Merged transcript JSON output path. |
|
||||
| `--report-file` | No | none | Optional report JSON output path. |
|
||||
| `--speakers` | No | none | Speaker map YAML file. When omitted, input file basenames are used as speaker labels. |
|
||||
| `--autocorrect` | No | none | Autocorrect rules YAML file. When omitted, the default `autocorrect` module leaves text unchanged. |
|
||||
| `--input-reader` | No | `json-files` | Input reader module. |
|
||||
| `--output-modules` | No | `json` | Comma-separated output modules. |
|
||||
| `--output-schema` | No | `seriatim-intermediate` | JSON output contract. Allowed values are `seriatim-minimal`, `seriatim-intermediate`, and `seriatim-full`. If omitted, the runtime default is used; consumers that depend on a specific shape should set this explicitly. |
|
||||
| `--preprocessing-modules` | No | `validate-raw,normalize-speakers,trim-text` | Comma-separated preprocessing modules, evaluated in order. |
|
||||
| `--postprocessing-modules` | No | `detect-overlaps,resolve-overlaps,backchannel,filler,resolve-danglers,coalesce,detect-overlaps,autocorrect,assign-ids,validate-output` | Comma-separated postprocessing modules, evaluated in order. |
|
||||
| `--coalesce-gap` | No | `3.0` | Maximum same-speaker gap in seconds for `coalesce`; also used as the `resolve-overlaps` context window. Must be a non-negative float. |
|
||||
|
||||
Environment variables:
|
||||
|
||||
| Environment Variable | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `SERIATIM_OUTPUT_SCHEMA` | `seriatim-intermediate` | Output schema used when `--output-schema` is not explicitly provided. Allowed values are `seriatim-minimal`, `seriatim-intermediate`, and `seriatim-full`. The CLI flag takes precedence. |
|
||||
| `SERIATIM_OVERLAP_WORD_RUN_GAP` | `1.0` | Maximum gap in seconds between adjacent timed words when `resolve-overlaps` builds word-run replacement segments. Must be a positive float. |
|
||||
| `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW` | `1.0` | Near-start window in seconds for ordering replacement word runs shortest-first. Must be a positive float. |
|
||||
| `SERIATIM_BACKCHANNEL_MAX_DURATION` | `2.0` | Maximum duration in seconds for `backchannel` classification. Must be a positive float. |
|
||||
| `SERIATIM_FILLER_MAX_DURATION` | `1.25` | Maximum duration in seconds for `filler` classification. Must be a positive float. |
|
||||
|
||||
## Input JSON Format
|
||||
|
||||
Each input file must be valid JSON with a top-level `segments` array. The current parser accepts the WhisperX segment subset needed for merging:
|
||||
|
||||
```json
|
||||
{
|
||||
"segments": [
|
||||
{
|
||||
"start": 1.25,
|
||||
"end": 3.5,
|
||||
"text": "Hello there.",
|
||||
"words": [
|
||||
{"word": "Hello", "start": 1.25, "end": 1.55, "score": 0.98},
|
||||
{"word": "there.", "start": 1.7, "end": 2.0}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Required segment fields:
|
||||
|
||||
- `start`: number, must be `>= 0`.
|
||||
- `end`: number, must be `>= start`.
|
||||
- `text`: string.
|
||||
|
||||
Optional word fields:
|
||||
|
||||
- `words`: array of word timing objects.
|
||||
- `words[].word`: string.
|
||||
- `words[].start`: optional number, must be `>= 0` when present.
|
||||
- `words[].end`: optional number, must be `>= start` when present with `start`.
|
||||
- `words[].score`: optional number.
|
||||
- `words[].speaker`: optional raw speaker label string.
|
||||
|
||||
Word-level timing is preserved internally for overlap resolution. If a word is missing `start` or `end`, seriatim keeps the word text, emits a warning in the optional report, and does not use that word as a timing anchor. Word timing is not emitted in the final JSON artifact.
|
||||
|
||||
## Speaker Map Format
|
||||
|
||||
`speakers.yml` maps input files to canonical speaker names using ordered substring rules:
|
||||
|
||||
This file is optional. If `--speakers` is omitted, `seriatim` uses each input file basename as the segment speaker label.
|
||||
|
||||
```yaml
|
||||
match:
|
||||
- speaker: "Eric Rakestraw"
|
||||
match:
|
||||
- "Eric_Rakestraw"
|
||||
- "Eric"
|
||||
|
||||
- speaker: "Mike Brown"
|
||||
match:
|
||||
- "Mike_Brown"
|
||||
- "mb"
|
||||
```
|
||||
|
||||
For each `--input-file`, `seriatim` takes the file basename and evaluates the rules in order. The first rule with a matching substring wins, and no later rules are evaluated.
|
||||
|
||||
For example, this input:
|
||||
|
||||
```text
|
||||
samples/raw/2026-04-19-Eric_Rakestraw.json
|
||||
```
|
||||
|
||||
matches this rule because the basename contains `Eric_Rakestraw`:
|
||||
|
||||
```yaml
|
||||
- speaker: "Eric Rakestraw"
|
||||
match:
|
||||
- "Eric_Rakestraw"
|
||||
```
|
||||
|
||||
Important details:
|
||||
|
||||
- Matching is against the input file basename, not the full path.
|
||||
- Matching is case-insensitive.
|
||||
- Rules are evaluated from first to last.
|
||||
- Each rule must have a non-empty `speaker`.
|
||||
- Each rule must have at least one non-empty `match` string.
|
||||
- Duplicate speaker names are invalid.
|
||||
- Every input file must match at least one rule or the command fails.
|
||||
|
||||
Deprecated old format:
|
||||
|
||||
```yaml
|
||||
inputs:
|
||||
eric.json:
|
||||
speaker: "Eric Rakestraw"
|
||||
```
|
||||
|
||||
The old `inputs:` direct mapping format is no longer supported.
|
||||
|
||||
## Output JSON Format
|
||||
|
||||
`--output-modules json` controls the writer. `--output-schema` controls the JSON contract that writer serializes.
|
||||
|
||||
The named schemas are stable public contracts. If a consumer depends on a specific shape, it should request that schema explicitly at runtime. The runtime default selection may change in a future release.
|
||||
|
||||
The `seriatim-intermediate` schema is the current default selection when neither `--output-schema` nor `SERIATIM_OUTPUT_SCHEMA` is set. It stays close to the minimal schema, but adds optional `categories` on each segment:
|
||||
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"application": "seriatim",
|
||||
"version": "dev",
|
||||
"output_schema": "seriatim-intermediate"
|
||||
},
|
||||
"segments": [
|
||||
{
|
||||
"id": 1,
|
||||
"start": 1.25,
|
||||
"end": 3.5,
|
||||
"speaker": "Eric Rakestraw",
|
||||
"text": "Hello there.",
|
||||
"categories": ["backchannel"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The `seriatim-full` schema uses the full seriatim envelope:
|
||||
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"application": "seriatim",
|
||||
"version": "dev",
|
||||
"input_reader": "json-files",
|
||||
"input_files": ["eric.json", "mike.json"],
|
||||
"preprocessing_modules": ["validate-raw", "normalize-speakers", "trim-text"],
|
||||
"postprocessing_modules": ["detect-overlaps", "resolve-overlaps", "backchannel", "filler", "resolve-danglers", "coalesce", "detect-overlaps", "autocorrect", "assign-ids", "validate-output"],
|
||||
"output_modules": ["json"]
|
||||
},
|
||||
"segments": [
|
||||
{
|
||||
"id": 1,
|
||||
"source": "eric.json",
|
||||
"source_segment_index": 0,
|
||||
"speaker": "Eric Rakestraw",
|
||||
"start": 1.25,
|
||||
"end": 3.5,
|
||||
"text": "Hello there.",
|
||||
"overlap_group_id": 1
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"source": "eric.json",
|
||||
"source_ref": "word-run:1:1:1",
|
||||
"derived_from": ["eric.json#0"],
|
||||
"speaker": "Eric Rakestraw",
|
||||
"start": 2.0,
|
||||
"end": 2.5,
|
||||
"text": "Resolved word run",
|
||||
"categories": ["backchannel"]
|
||||
}
|
||||
],
|
||||
"overlap_groups": [
|
||||
{
|
||||
"id": 1,
|
||||
"start": 1.25,
|
||||
"end": 4.0,
|
||||
"segments": ["eric.json#0", "mike.json#0"],
|
||||
"speakers": ["Eric Rakestraw", "Mike Brown"],
|
||||
"class": "unknown",
|
||||
"resolution": "unresolved"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The `seriatim-minimal` schema emits minimal metadata and compact ordered segments:
|
||||
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"application": "seriatim",
|
||||
"version": "dev",
|
||||
"output_schema": "seriatim-minimal"
|
||||
},
|
||||
"segments": [
|
||||
{
|
||||
"id": 1,
|
||||
"start": 1.25,
|
||||
"end": 3.5,
|
||||
"speaker": "Eric Rakestraw",
|
||||
"text": "Hello there."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Minimal output intentionally omits categories, overlap groups, source/provenance fields, and pipeline configuration metadata.
|
||||
|
||||
Intermediate output intentionally omits overlap groups and source/provenance fields, but keeps optional `categories` and minimal metadata.
|
||||
|
||||
Segments are sorted deterministically by:
|
||||
|
||||
```text
|
||||
(start, end, source, source_segment_index/source_ref, speaker)
|
||||
```
|
||||
|
||||
Final segment IDs are assigned after sorting and start at `1`.
|
||||
|
||||
The public Go output contract is available from:
|
||||
|
||||
```go
|
||||
import "gitea.maximumdirect.net/eric/seriatim/schema"
|
||||
```
|
||||
|
||||
The same package embeds machine-readable JSON Schemas in `schema/full-output.schema.json`, `schema/intermediate-output.schema.json`, and `schema/minimal-output.schema.json`. The default `validate-output` postprocessor validates the selected output shape and verifies final segment IDs are present, sequential, and start at `1`.
|
||||
|
||||
## Overlap Detection
|
||||
|
||||
The default postprocessing pipeline detects overlapping segment groups.
|
||||
|
||||
Overlap behavior:
|
||||
|
||||
- A strict timing overlap is required: `next.start < current_group_end`.
|
||||
- Segments that only touch at a boundary are not grouped.
|
||||
- Groups require at least two distinct speakers.
|
||||
- Transitive overlaps are grouped together.
|
||||
- Segments in detected groups receive `overlap_group_id`.
|
||||
- `overlap_groups[].segments` contains stable references in `source#source_segment_index` format.
|
||||
- `class` is currently `unknown`.
|
||||
- `resolution` is `unresolved` until `resolve-overlaps` replaces the group.
|
||||
|
||||
## Overlap Resolution
|
||||
|
||||
The default postprocessing pipeline runs `detect-overlaps`, then `resolve-overlaps`, then `backchannel`, then `filler`, then `resolve-danglers`, then `coalesce`, then a second `detect-overlaps` pass.
|
||||
|
||||
For each detected overlap group, `resolve-overlaps` uses preserved WhisperX word timing to build smaller word-run replacement segments:
|
||||
|
||||
- The resolution window expands the detected overlap group by `--coalesce-gap` seconds on both sides.
|
||||
- Nearby same-speaker context segments are included when they intersect the expanded window and their start or end is within `--coalesce-gap` of the original overlap boundary.
|
||||
- Once a segment is selected for replacement, all timed words from that segment participate in word-run construction; the window controls segment selection, not per-word clipping.
|
||||
- Context segments that are part of another detected overlap group are not pulled into the current group.
|
||||
- Untimed words are included in replacement text in original word order when nearby timed words create a replacement run.
|
||||
- Untimed words do not affect replacement segment start/end times or word-run gap splitting.
|
||||
- Words for the same speaker are merged into one run when the gap between adjacent words is no greater than `SERIATIM_OVERLAP_WORD_RUN_GAP`.
|
||||
- The default word-run gap is `1.0` seconds.
|
||||
- Set `SERIATIM_OVERLAP_WORD_RUN_GAP` to a positive number of seconds to override the default.
|
||||
- Near-start replacement word runs are reordered so shorter segments come first when adjacent starts are within `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW`.
|
||||
- The default word-run reorder window is `1.0` seconds.
|
||||
- Set `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW` to a positive number of seconds to override the default.
|
||||
- Replacement segment text is built by joining word text with single spaces.
|
||||
- Replacement segments include `source_ref` and `derived_from`.
|
||||
- Replacement segments omit `source_segment_index` because they are derived from one or more original segments.
|
||||
- Resolved overlap groups are removed before the second detection pass.
|
||||
- Replacement segments are left without `overlap_group_id` until the second detection pass annotates any remaining overlap.
|
||||
- If a speaker has no usable word timing in a group, that speaker's original segment is kept.
|
||||
- If no speakers in a group have usable word timing, the original group and annotations remain unchanged.
|
||||
|
||||
## Backchannels
|
||||
|
||||
The default pipeline runs `backchannel` before `coalesce`. It tags short acknowledgement segments with:
|
||||
|
||||
```json
|
||||
"categories": ["backchannel"]
|
||||
```
|
||||
|
||||
Backchannel matching is case-insensitive, ignores punctuation for matching and word-count purposes, trims surrounding whitespace, and requires a matching acknowledgement phrase, no more than three whitespace-delimited words, and duration no greater than `SERIATIM_BACKCHANNEL_MAX_DURATION` seconds. The default maximum duration is `2.0` seconds.
|
||||
|
||||
## Fillers
|
||||
|
||||
The default pipeline runs `filler` after `backchannel` and before `coalesce`. It tags short filler utterances with:
|
||||
|
||||
```json
|
||||
"categories": ["filler"]
|
||||
```
|
||||
|
||||
Filler matching is case-insensitive, ignores punctuation for matching and word-count purposes, trims surrounding whitespace, and requires only filler tokens such as `um`, `uh`, `er`, `erm`, `ah`, `eh`, `hmm`, `mm`, or repeated combinations of those tokens. Matching segments must contain no more than three whitespace-delimited words and have duration no greater than `SERIATIM_FILLER_MAX_DURATION` seconds. The default maximum duration is `1.25` seconds.
|
||||
|
||||
## Dangler Resolution
|
||||
|
||||
The default pipeline runs `resolve-danglers` before `coalesce` and before the second overlap detection pass. It repairs short derived fragments when they share provenance with a nearby segment:
|
||||
|
||||
- Dangling-end fragments have no more than two words and end in punctuation.
|
||||
- Dangling-start fragments have no more than two words.
|
||||
- Matching uses same-speaker segments with any shared `derived_from` value.
|
||||
- Merged segments use `source_ref` values such as `resolve-danglers:1`, keep the target segment's transcript position, and union `derived_from`.
|
||||
|
||||
## Coalescing
|
||||
|
||||
The default pipeline runs `coalesce` after `resolve-danglers` and before the second overlap detection pass. It merges adjacent same-speaker segments in the transcript's current order when `next.start - current.end <= --coalesce-gap`.
|
||||
|
||||
Coalesced segments use `source_ref` values such as `coalesce:1`, include `derived_from`, and omit `source_segment_index`.
|
||||
|
||||
Different-speaker backchannel and filler segments do not block coalescing of surrounding same-speaker segments. Same-speaker backchannel and filler segments are merged normally when they are within `--coalesce-gap`. When same-speaker segments are coalesced, any `backchannel` or `filler` category from the merged inputs is dropped from the coalesced segment.
|
||||
|
||||
## Autocorrect
|
||||
|
||||
Autocorrect is included in the default postprocessing pipeline. If `--autocorrect` is omitted, the module leaves transcript text unchanged and records a skip event in the optional report.
|
||||
|
||||
Enable corrections by passing `--autocorrect`:
|
||||
|
||||
```sh
|
||||
go run ./cmd/seriatim merge \
|
||||
--input-file input.json \
|
||||
--autocorrect autocorrect.yml \
|
||||
--output-file merged.json
|
||||
```
|
||||
|
||||
`autocorrect.yml` format:
|
||||
|
||||
```yaml
|
||||
autocorrect:
|
||||
- target: "Hrank"
|
||||
match:
|
||||
- "hrank"
|
||||
- "Frank"
|
||||
|
||||
- target: "Mike Brown"
|
||||
match:
|
||||
- "Mike Pat"
|
||||
```
|
||||
|
||||
Matching behavior:
|
||||
|
||||
- Matching is case-sensitive.
|
||||
- Matches apply only to whole tokens, not substrings inside larger words.
|
||||
- Punctuation and whitespace can surround a match.
|
||||
- Multi-word and hyphenated matches are supported.
|
||||
- Duplicate match strings are invalid, including duplicates across separate rules.
|
||||
|
||||
## Current Limitations
|
||||
|
||||
- Only JSON input is supported.
|
||||
- Overlap resolution depends on WhisperX word timing; groups without usable word timing remain unresolved.
|
||||
- Alternate output formats are not implemented yet.
|
||||
|
||||
## Release Builds
|
||||
|
||||
Local builds record version metadata as `dev`. Release builds should inject the release version with `ldflags`:
|
||||
|
||||
```sh
|
||||
go build -ldflags "-X gitea.maximumdirect.net/eric/seriatim/internal/buildinfo.Version=v1.0.0" ./cmd/seriatim
|
||||
```
|
||||
# Integration: seriatim
|
||||
|
||||
## Purpose
|
||||
Define Narratio's adapter contract for merge, normalize, and trim subprocess invocations of Seriatim.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `MergeRequest`: raw/normalized transcript inputs, output path, optional report, speaker/autocorrect paths, logs/config
|
||||
- `NormalizeRequest`: input transcript, output path, schema, optional report, timeout/log/config
|
||||
- `TrimRequest`: input transcript, output path, keep selector, timeout/log/config
|
||||
|
||||
Outputs:
|
||||
- `MergeResult`, `NormalizeResult`, `TrimResult` with output paths, logs/config paths, exit code, duration, binary provenance, and metadata.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Validated deterministic CLI invocation construction
|
||||
- Optional env tuning propagation for merge
|
||||
- Invocation config file emission
|
||||
- JSON output validation
|
||||
|
||||
Does not own:
|
||||
- Transcript input selection/promotion logic (stage-owned)
|
||||
- Bounds computation (scriptorium/trim-stage-owned)
|
||||
|
||||
## Config Fields Used
|
||||
Via `pipeline.seriatim.*` mapped in app/stage wiring:
|
||||
- `binary`, `timeout`, `output_schema`, `coalesce_gap`, `report`
|
||||
- `env.overlap_word_run_gap`
|
||||
- `env.overlap_word_run_reorder_window`
|
||||
- `env.backchannel_max_duration`
|
||||
- `env.filler_max_duration`
|
||||
|
||||
## External Adapters Used
|
||||
- Shared subprocess helper (`internal/adapters/subprocess`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- No direct manifest writes.
|
||||
- Stage metadata consumes adapter result fields and preserves generated config/log references.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Adapter has no skip/resume logic. Runner controls stage execution.
|
||||
|
||||
## Failure Behavior
|
||||
- Constructor fails for invalid binary/timeout/output-schema/coalesce-gap.
|
||||
- Merge fails on missing output path/inputs/report path (if enabled), subprocess errors, invalid merged output JSON, invalid report JSON.
|
||||
- Normalize fails on missing input/output, invalid schema, subprocess errors, invalid normalized output JSON shape, invalid report JSON.
|
||||
- Trim fails on missing input/output/keep selector, subprocess errors, invalid trimmed output JSON shape.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
- `internal/adapters/seriatim/fake_test.go`
|
||||
- `internal/stage/merge_test.go`
|
||||
- `internal/stage/normalize_test.go`
|
||||
- `internal/stage/trim_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Supported output schemas are limited to `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`.
|
||||
- Normalize/trim outputs must include `segments` arrays.
|
||||
- Merge/normalize/trim all route through deterministic subprocess invocation.
|
||||
|
||||
29
docs/internal/README.md
Normal file
29
docs/internal/README.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# Internal Documentation Index
|
||||
|
||||
## Audience
|
||||
Developers and LLM coding agents changing Narratio internals.
|
||||
|
||||
## Scope
|
||||
Implementation-accurate contracts for workspace/state, manifests, stages, artifact resolution, adapter boundaries, and restore command behavior.
|
||||
|
||||
## Component Docs
|
||||
- `adapters.md`: external adapter map, runtime wiring, and boundary ownership.
|
||||
- `storage.md`: remote storage backend contracts and object-store invariants.
|
||||
- `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics.
|
||||
- `artifacts.md`: built-in artifact registry, runtime artifact catalog, and source-resolution behavior.
|
||||
- `workspace.md`: local state model, manifests, run-local layout, promotion, and cleanup invariants.
|
||||
- `command-restore.md`: restore command discovery/planning/execution/reporting contract.
|
||||
- `stage-prepare.md`: input materialization and provenance capture.
|
||||
- `stage-transcribe.md`: WhisperX transcript generation.
|
||||
- `stage-merge.md`: Seriatim normalization + merge.
|
||||
- `stage-polish.md`: Audita transcript polishing.
|
||||
- `stage-normalize.md`: post-polish normalization.
|
||||
- `stage-trim.md`: bounds-driven transcript trimming.
|
||||
- `stage-analyze.md`: dependency-ordered Scriptorium artifact generation for selected configured artifacts.
|
||||
- `stage-archive.md`: archive upload and current-pointer publish contract.
|
||||
|
||||
## External Integration Notes
|
||||
- `../integrations/README.md`: canonical location for external integration contracts (`audita.md`, `seriatim.md`, `scriptorium.md`).
|
||||
|
||||
## Canonical Owner
|
||||
`docs/internal/` is the canonical home for implemented internals per `docs/documentation/policy.md`.
|
||||
80
docs/internal/adapters.md
Normal file
80
docs/internal/adapters.md
Normal file
@@ -0,0 +1,80 @@
|
||||
# Internal: Adapters
|
||||
|
||||
## Purpose
|
||||
Describe the external adapter boundaries used by Narratio stages and app orchestration, including default runtime wiring.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- Stage requests passed through adapter interfaces (for example transcription, merge/normalize/trim, polish, artifact generation, object-store operations, notifications).
|
||||
- Resolved config values used to construct default adapters.
|
||||
|
||||
Outputs:
|
||||
- Adapter-specific result structs (paths, metadata, status/attempt info, duration/exit details).
|
||||
- Adapter errors returned to stage/app orchestration.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Transport/process/SDK details at system boundaries (`HTTP`, subprocess CLI invocation, AWS SDK calls).
|
||||
- Request/response contracts in `internal/adapters/*` packages.
|
||||
|
||||
Does not own:
|
||||
- Stage sequencing, skip/force/resume decisions.
|
||||
- Manifest transition logic.
|
||||
- Canonical workspace path policy.
|
||||
|
||||
## Config fields used
|
||||
Default wiring and adapter calls consume:
|
||||
- `pipeline.whisperx.*`
|
||||
- `pipeline.seriatim.*`
|
||||
- `pipeline.audita.*`
|
||||
- `pipeline.scriptorium.*`
|
||||
- `pipeline.storage.*` and `pipeline.archive.*` (object-store construction/gating)
|
||||
- `pipeline.notification.*` (sender boundary exists; placeholder behavior today)
|
||||
|
||||
## External adapters used
|
||||
Runtime env boundary fields (`internal/stage.Env`):
|
||||
- `whisperx.Client`
|
||||
- `seriatim.Runner`
|
||||
- `audita.Runner`
|
||||
- `scriptorium.Runner`
|
||||
- `storage.ObjectStore`
|
||||
- `notify.Sender`
|
||||
- `analyzer.Runner`
|
||||
|
||||
Current execution usage:
|
||||
- Actively used by implemented stages: `WhisperX`, `Seriatim`, `Audita`, `Scriptorium`, `ObjectStore`, `Notifier`.
|
||||
- Present but not used by implemented stage set: `Analyzer`, legacy `storage.Backend`.
|
||||
|
||||
Default construction in app runner:
|
||||
- Auto-constructed when not injected: WhisperX HTTP client, Seriatim subprocess runner, Audita subprocess runner, Scriptorium subprocess runner, object store (only when needed), and `notify.NoopSender`.
|
||||
- Object-store construction goes through app command orchestration so configured filesystem secrets are loaded before the storage adapter is initialized.
|
||||
- Callers can inject test/fake implementations through `app.RunOptions.Env`.
|
||||
|
||||
## State and manifest behavior
|
||||
- Adapters do not directly mutate session/run manifests.
|
||||
- Stages and runner own manifest writes and stage status transitions.
|
||||
- Adapter outputs are persisted indirectly through stage result mapping (outputs/logs/generated configs/metadata).
|
||||
|
||||
## Skip and resume behavior
|
||||
- No adapter-level skip/resume semantics.
|
||||
- Skip/resume/force behavior is decided by app runner using manifest stage state.
|
||||
|
||||
## Failure behavior
|
||||
- Adapter constructors validate config-derived values and fail early on invalid required inputs.
|
||||
- Adapter run-time failures are returned to stage code with boundary context and are recorded as stage failures by runner logic.
|
||||
- Subprocess adapters preserve stdout/stderr and generated-config paths to aid diagnosis.
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/adapters/whisperx/http_test.go`
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
- `internal/adapters/audita/subprocess_test.go`
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
- `internal/adapters/storage/*_test.go`
|
||||
- `internal/adapters/notify/fake_test.go`
|
||||
- `internal/adapters/analyzer/fake_test.go`
|
||||
- `internal/app/runner_test.go`
|
||||
|
||||
## Architectural invariants
|
||||
- Stage code depends on adapter interfaces, not transport-specific implementation types.
|
||||
- External SDK-specific types remain inside adapter implementations.
|
||||
- Default app wiring must remain deterministic and overrideable via injected env dependencies.
|
||||
106
docs/internal/artifacts.md
Normal file
106
docs/internal/artifacts.md
Normal file
@@ -0,0 +1,106 @@
|
||||
# Internal: Artifacts
|
||||
|
||||
## Purpose
|
||||
Define Narratio artifact identity, catalog, and source-resolution behavior for:
|
||||
- built-in session artifacts;
|
||||
- configured analyze artifacts;
|
||||
- canonical previous-session artifact sources.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- configured input sources (`pipeline.scriptorium.artifacts.*.inputs.*.source`);
|
||||
- session paths and manifest inputs/outputs;
|
||||
- runtime catalog state.
|
||||
|
||||
Outputs:
|
||||
- resolved artifact path + provenance (`ResolvedSessionArtifact`);
|
||||
- runtime catalog entries for built-ins and configured artifacts;
|
||||
- requirement sets for canonical previous-session inputs.
|
||||
- canonical S3 session, run, current, session config, session locks, audio, and promoted artifact keys.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- built-in source registry and validation;
|
||||
- configured artifact catalog identity (`narratio.artifact.<name>`);
|
||||
- canonical previous-session source parsing and resolution;
|
||||
- previous-session requirement collection (`CollectPreviousArtifactRequirements`).
|
||||
|
||||
Does not own:
|
||||
- prepare-stage remote hydration;
|
||||
- stage success/skip transitions;
|
||||
- archive upload orchestration.
|
||||
|
||||
## Built-in IDs
|
||||
| Artifact ID | Canonical file | Producer stage | Output kind |
|
||||
| --- | --- | --- | --- |
|
||||
| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` |
|
||||
| `narratio.transcript.polished` | `transcripts/processed.json` | `polish` | `transcript_processed` |
|
||||
| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` |
|
||||
| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` |
|
||||
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` |
|
||||
|
||||
## Source families
|
||||
- built-in: `narratio.transcript.*`, `narratio.bounds.session`
|
||||
- configured artifact: `narratio.artifact.<artifact_key>`
|
||||
- canonical previous-session artifact: `narratio.previous_session.artifact.<artifact_key>`
|
||||
|
||||
## S3 key helpers
|
||||
- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
- session config: `{session_prefix}/session.yml`
|
||||
- session lock store: `{session_prefix}/locks.yml`
|
||||
- run prefix: `{session_prefix}/runs/{run_id}/`
|
||||
- audio prefix: `{session_prefix}/{session.inputs.audio_s3.prefix}`
|
||||
- current manifest: `{session_prefix}/current/manifest.json`
|
||||
- current run pointer: `{session_prefix}/current/run_id.txt`
|
||||
|
||||
## Runtime catalog model
|
||||
Catalog entries track:
|
||||
- `planned`: source is registered for this run;
|
||||
- `executable`: configured artifact is selected for analyze execution;
|
||||
- `available`: usable local file exists (generated this run or reused from disk).
|
||||
|
||||
Configured artifact provenance values include:
|
||||
- `generated.current_analyze_run`
|
||||
- `filesystem.disabled_artifact_output`
|
||||
|
||||
Previous-session canonical provenance values include:
|
||||
- `manifest.inputs.previous_cache`
|
||||
- `current_session.previous_cache`
|
||||
|
||||
## Resolution behavior
|
||||
- Built-ins resolve via manifest producer outputs first, then canonical fallback paths.
|
||||
- Configured `narratio.artifact.<name>` sources resolve through catalog availability.
|
||||
- Canonical previous-session sources resolve to current-session `previous/` cache candidates derived from configured artifact canonical output paths.
|
||||
- Archive-relative configured artifact paths under `artifacts/` are cached without a redundant nested `artifacts/` segment.
|
||||
- Previous-session canonical resolution prefers manifest-recorded input paths when present, then filesystem fallback under `previous/artifacts/**`.
|
||||
|
||||
## Previous-session requirement scanning
|
||||
`CollectPreviousArtifactRequirements`:
|
||||
- scans enabled configured artifacts only;
|
||||
- includes canonical previous-session sources only;
|
||||
- deduplicates by artifact key;
|
||||
- merges required/optional references (`required` wins);
|
||||
- records deterministic sorted source locations for diagnostics.
|
||||
|
||||
## Validation behavior
|
||||
- transcript built-ins: JSON with top-level `segments` array;
|
||||
- bounds built-in: valid JSON;
|
||||
- configured and previous-session artifact files: non-empty text content.
|
||||
|
||||
## Failure behavior
|
||||
- unsupported source or malformed canonical previous source: validation/resolution error;
|
||||
- known source unavailable: `ErrSessionArtifactNotFound`;
|
||||
- configured/previous canonical source without catalog: error;
|
||||
- resolved invalid file content: validation error.
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/artifacts/artifact_resolver_test.go`
|
||||
- `internal/artifacts/catalog_test.go`
|
||||
- `internal/artifacts/previous_requirements_test.go`
|
||||
- `internal/stage/prepare_previous_test.go`
|
||||
- `internal/stage/analyze_test.go`
|
||||
|
||||
## Architectural invariants
|
||||
- Built-in source IDs are static.
|
||||
- Configured and previous-session source IDs are artifact-key based and validation-gated.
|
||||
- Resolution behavior remains deterministic and manifest-aware.
|
||||
105
docs/internal/command-restore.md
Normal file
105
docs/internal/command-restore.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# Internal: Command Restore
|
||||
|
||||
## Purpose
|
||||
Define the implemented `narratio restore` contract: committed remote-state discovery, deterministic plan classification, safe file install semantics, and restore reporting.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- CLI flags: `--config`, `--session`, `--session-id`, `--previous-session-id`, `--dry-run`, `--force`, `--include-audio`.
|
||||
- Resolved/validated `pipeline.yml` and `session.yml`.
|
||||
- Configured remote object store.
|
||||
- Remote committed current-state markers (`current/run_id.txt`, `current/manifest.json`).
|
||||
|
||||
Outputs:
|
||||
- Dry-run summary to stdout (plan + counts).
|
||||
- Non-dry-run completion summary to stdout.
|
||||
- Local durable session files restored under canonical session root.
|
||||
- Non-dry-run restore report at `reports/restore-latest.json`.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Restore command flag parsing and command wiring.
|
||||
- Remote current-state discovery and identity validation.
|
||||
- Restore plan construction and conflict classification.
|
||||
- Restore execution for planned downloads.
|
||||
- Restore report model and persistence.
|
||||
|
||||
Does not own:
|
||||
- Stage execution orchestration (`run`, `resume`, `run-stage`).
|
||||
- Archive publish behavior (owned by archive stage).
|
||||
- Storage transport implementation details (owned by storage adapters).
|
||||
|
||||
## Config fields used
|
||||
- Config/session discovery and templating fields consumed by all commands.
|
||||
- `pipeline.workspace.root` (local restore target root).
|
||||
- `pipeline.storage.*` (remote backend + archive identity derivation).
|
||||
- `pipeline.storage.s3.*` identity components used by archive prefix helpers.
|
||||
- `pipeline.spool.root` for active audio downloads.
|
||||
- `pipeline.cache.root` and `pipeline.cache.s3_audio` for reusable S3 audio cache.
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
|
||||
## External adapters used
|
||||
- `storage.ObjectStore` for `Exists`, `List`, `Download`.
|
||||
- `artifacts.Store` (`LocalStore`) for layout and session lock management.
|
||||
- `manifest.LocalStore` for manifest decode/validation and identity checks.
|
||||
|
||||
## State and manifest behavior
|
||||
- Restore is not a pipeline run and does not create a run manifest.
|
||||
- Restore uses committed remote current state only:
|
||||
- `current/run_id.txt` must exist and be non-empty.
|
||||
- `current/manifest.json` must decode and match requested session/campaign.
|
||||
- Non-dry-run writes restore files to canonical session paths.
|
||||
- With `--include-audio`, restore uses the shared S3 audio cache for `audio/**` objects. Cache hits avoid object downloads; cache misses download through spool, install the work file, and populate cache.
|
||||
- Manifest install behavior:
|
||||
- validated before replacement.
|
||||
- installed last among download actions.
|
||||
- existing local manifest is preserved if restored manifest validation/install fails.
|
||||
- Non-dry-run report persists summary/action status metadata in `reports/restore-latest.json`.
|
||||
|
||||
Restore path scope:
|
||||
- includes:
|
||||
- `manifest.json`
|
||||
- `transcripts/**`
|
||||
- `artifacts/**`
|
||||
- `previous/**`
|
||||
- `audio/**` only when `--include-audio` is set
|
||||
- excludes:
|
||||
- `runs/**`
|
||||
- `logs/**`
|
||||
- `reports/**`
|
||||
- `config/**`
|
||||
- `inputs/**`
|
||||
- remote `current/**` pointer files as local restore targets
|
||||
|
||||
## Skip and resume behavior
|
||||
- Restore does not participate in stage skip/resume decisions.
|
||||
- Restore provides durable local state so subsequent stage commands can resume or rerun based on restored manifest state.
|
||||
- Audio cache is outside the workspace and is reused across restore and prepare invocations.
|
||||
- Dry-run is read-only and returns plan output only.
|
||||
|
||||
## Failure behavior
|
||||
- Fails when storage backend is unavailable or archive identity cannot be resolved.
|
||||
- Fails when remote current pointer/manifest is missing or invalid.
|
||||
- Fails when remote manifest identity mismatches requested campaign/session.
|
||||
- Fails on local conflicts unless `--force` is set.
|
||||
- Fails fast on session lock acquisition conflict for non-dry-run execution.
|
||||
- On execution failure, previously installed files remain; no rollback is performed.
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/app/restore_test.go`
|
||||
- `internal/app/restore_discovery_test.go`
|
||||
- `internal/app/restore_plan_test.go`
|
||||
- `internal/app/restore_execution_test.go`
|
||||
- `internal/app/restore_workflow_test.go`
|
||||
- `internal/artifacts/archive_identity_test.go`
|
||||
|
||||
## Architectural invariants
|
||||
- Restore relies on centralized archive identity/key helpers (`internal/artifacts`) rather than ad hoc key building.
|
||||
- `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files.
|
||||
- Local path mapping is traversal-safe and constrained to session root.
|
||||
- Restore scope is deterministic and path-classified:
|
||||
- include `manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`
|
||||
- include `audio/**` only with `--include-audio`
|
||||
- exclude `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`
|
||||
- Command remains standalone; no implicit `run --restore` behavior.
|
||||
81
docs/internal/manifest.md
Normal file
81
docs/internal/manifest.md
Normal file
@@ -0,0 +1,81 @@
|
||||
# Internal: Manifest
|
||||
|
||||
## Purpose
|
||||
Describe Narratio's durable execution state model for session-level and run-level manifests, including lifecycle transitions and persistence behavior.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- Session identity and run identity from app orchestration.
|
||||
- Stage transition events and stage result payloads.
|
||||
|
||||
Outputs:
|
||||
- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`.
|
||||
- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Manifest schemas (`Manifest`, `RunManifest`, stage records, error records, input/artifact records).
|
||||
- Stage status/action transition methods.
|
||||
- Persistent store contract (`manifest.Store`) and local JSON store implementation.
|
||||
|
||||
Does not own:
|
||||
- Stage implementation details.
|
||||
- Path construction policy outside manifest file persistence calls.
|
||||
- CLI command behavior.
|
||||
|
||||
## Config fields used
|
||||
Manifest package itself does not read config directly.
|
||||
|
||||
Manifest identity fields are populated by app/stage orchestration from:
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.storage.s3.*` (when archive/S3 identity is set)
|
||||
|
||||
## External adapters used
|
||||
- No external service adapters.
|
||||
- Uses local filesystem for persistence via `manifest.LocalStore`.
|
||||
|
||||
## State and manifest behavior
|
||||
Session manifest model:
|
||||
- Tracks durable per-session stage state and provenance (`pending`, `running`, `succeeded`, `failed`, `skipped`, `stale`, `interrupted`).
|
||||
- Stores resolved inputs, durable artifacts, stage logs/config refs, and stage metadata.
|
||||
|
||||
Run manifest model:
|
||||
- Tracks one invocation (`run_id`) with requested stages and force mode.
|
||||
- Tracks per-stage action (`run` or `skip`) and per-stage status.
|
||||
- Tracks overall run status (`running`, `succeeded`, `failed`).
|
||||
|
||||
Persistence behavior:
|
||||
- Load validates required identity/timestamp fields and normalizes maps/records.
|
||||
- Save updates `updated_at` and writes JSON atomically (temp file + rename).
|
||||
- Session and run manifests are saved incrementally before/after stage transitions.
|
||||
|
||||
Relationship during execution:
|
||||
- Runner updates both manifests for every stage transition.
|
||||
- Session manifest is the durable pipeline-progress ledger.
|
||||
- Run manifest is invocation history and audit record.
|
||||
- Analyze stage outputs are persisted as `kind=scriptorium_artifact` with `source_id=narratio.artifact.<name>` for configured artifact identity.
|
||||
|
||||
## Skip and resume behavior
|
||||
- Resume and skip decisions are based on session-manifest stage statuses.
|
||||
- `--force` reruns selected stages and marks downstream succeeded stages as `stale` in session manifest.
|
||||
- Run manifest records whether each stage was executed or skipped in that invocation.
|
||||
|
||||
## Failure behavior
|
||||
- Stage failure marks both manifests failed for that stage and records error messages/timestamps.
|
||||
- Save failures are returned immediately and fail the command.
|
||||
- Invalid/malformed manifest files fail load with explicit validation/decode errors.
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/manifest/manifest_test.go`
|
||||
- `internal/manifest/run_manifest_test.go`
|
||||
- `internal/manifest/store_test.go`
|
||||
- `internal/app/runner_test.go`
|
||||
- `internal/app/run_control_test.go`
|
||||
- `internal/app/resume_run_stage_test.go`
|
||||
|
||||
## Architectural invariants
|
||||
- Session manifest is authoritative for stage progression across invocations.
|
||||
- Run manifest is invocation-scoped and never replaces session manifest as progress authority.
|
||||
- Manifest writes are atomic and deterministic (JSON + newline, temp rename pattern).
|
||||
81
docs/internal/stage-analyze.md
Normal file
81
docs/internal/stage-analyze.md
Normal file
@@ -0,0 +1,81 @@
|
||||
# Stage: analyze
|
||||
|
||||
## Purpose
|
||||
Execute selected configured Scriptorium artifacts in deterministic dependency order and promote successful outputs to canonical session artifact paths.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- configured artifact definitions from `pipeline.scriptorium.artifacts`;
|
||||
- selected artifact filter (`--artifacts`) when provided;
|
||||
- resolved artifact sources from resolver/catalog.
|
||||
|
||||
Source types used by analyze:
|
||||
- built-ins: `narratio.transcript.*`, `narratio.bounds.session`;
|
||||
- configured artifacts: `narratio.artifact.<artifact_key>`;
|
||||
- canonical previous-session artifacts: `narratio.previous_session.artifact.<artifact_key>`;
|
||||
- legacy path-based previous-session source: `previous_session_artifact` (uses `inputs.*.path`).
|
||||
|
||||
Outputs:
|
||||
- promoted configured artifact files at each configured `output_path`;
|
||||
- stage metadata (`generated_artifacts`, `reused_artifacts`, selected/order info).
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- runtime artifact catalog construction;
|
||||
- selected-artifact planning and dependency ordering;
|
||||
- per-input resolution and required/optional handling;
|
||||
- Scriptorium render/run invocation;
|
||||
- run-local output generation and canonical promotion.
|
||||
|
||||
Does not own:
|
||||
- prepare-time previous-session hydration;
|
||||
- object-store access for previous-session sources;
|
||||
- archive promotion policy.
|
||||
|
||||
## Config fields used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.scriptorium.binary`
|
||||
- `pipeline.scriptorium.config_path`
|
||||
- `pipeline.scriptorium.timeout`
|
||||
- `pipeline.scriptorium.render_debug`
|
||||
- `pipeline.scriptorium.artifacts.<name>.*`
|
||||
|
||||
## External adapters used
|
||||
- Scriptorium adapter:
|
||||
- optional `RenderArtifact` when render-debug is enabled;
|
||||
- `RunArtifact` for artifact generation.
|
||||
|
||||
## State and manifest behavior
|
||||
- If Scriptorium config is absent, or no artifacts are executable after filtering, analyze returns success metadata with `skipped=true`.
|
||||
- Builds runtime catalog with built-ins and configured `narratio.artifact.<name>` entries.
|
||||
- Non-executable configured artifacts may still be marked available from existing canonical output files.
|
||||
- Resolves canonical previous-session sources from local prepared `previous/` cache:
|
||||
- prefers manifest-backed previous input paths when present;
|
||||
- may fall back to current-session `previous/` filesystem paths.
|
||||
- Analyze does not call object storage for canonical previous-session source resolution.
|
||||
- Required canonical previous-session input missing:
|
||||
- fails with guidance to run `narratio run-stage --force prepare`.
|
||||
- Optional missing sources are omitted from adapter input paths.
|
||||
|
||||
## Skip and resume behavior
|
||||
- Runner-level skip applies when analyze is already `succeeded` and `--force` is not set.
|
||||
- Analyze is stage-scoped for resume; no per-artifact manifest resume state.
|
||||
- `--artifacts` filters executable artifacts but does not imply force rerun.
|
||||
|
||||
## Failure behavior
|
||||
- Fails on dependency-order violations, missing required inputs, resolver validation failures, adapter errors, and missing/empty generated outputs.
|
||||
- Required unavailable configured artifact source (`narratio.artifact.<name>`) fails before invocation.
|
||||
- Required canonical previous-session source fails with prepare-rerun guidance.
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/stage/analyze_test.go`
|
||||
- `internal/artifacts/catalog_test.go`
|
||||
- `internal/artifacts/artifact_resolver_test.go`
|
||||
- `internal/app/restore_workflow_test.go`
|
||||
|
||||
## Architectural invariants
|
||||
- Canonical previous-session behavior is local-cache only during analyze.
|
||||
- Generated outputs are validated and promoted before stage success is recorded.
|
||||
- Resolver/catalog decisions stay deterministic and validation-gated.
|
||||
84
docs/internal/stage-archive.md
Normal file
84
docs/internal/stage-archive.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# Stage: archive
|
||||
|
||||
## Purpose
|
||||
Publish durable run/session state to object storage, then atomically advance remote current state.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- session manifest and prerequisite stage records
|
||||
- run root contents under `runs/{run_id}/`
|
||||
- promotion rules with artifact `source` IDs and archive `dest` paths (`archive.promote_artifacts`)
|
||||
- effective source-based promotion locks from static config and remote session lock store
|
||||
- session-level `previous/**` cache files when present
|
||||
|
||||
Outputs:
|
||||
- uploaded run files under `{session_prefix}/runs/{run_id}/...`
|
||||
- uploaded promoted artifacts under `{session_prefix}/...`
|
||||
- uploaded session previous-cache files under `{session_prefix}/previous/...` when present
|
||||
- `{session_prefix}/current/manifest.json`
|
||||
- `{session_prefix}/current/run_id.txt` written last
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Archive enable/disable gate behavior
|
||||
- Prerequisite stage success enforcement
|
||||
- Run file collection and upload (excluding `audio/`)
|
||||
- Promotion rule resolution and upload
|
||||
- Promotion lock enforcement
|
||||
- Session previous-cache file collection/upload
|
||||
- Commit pointer publish order
|
||||
|
||||
Does not own:
|
||||
- Stage execution before archive
|
||||
- Post-archive local cleanup policy execution (handled by app cleanup logic)
|
||||
|
||||
## Config Fields Used
|
||||
- `pipeline.archive.enabled`
|
||||
- `pipeline.archive.upload_run`
|
||||
- `pipeline.archive.promote_artifacts`
|
||||
- `pipeline.archive.locks`
|
||||
- `{session_prefix}/locks.yml` loaded by app orchestration before archive execution
|
||||
- `pipeline.storage.s3.bucket`
|
||||
- `pipeline.storage.s3.root_prefix`
|
||||
- `pipeline.workspace.root`
|
||||
- `session.campaign`
|
||||
- `session.session_id`
|
||||
|
||||
## External Adapters Used
|
||||
- Object storage backend (`env.ObjectStore`) for upload/list primitives.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
|
||||
- Resolves bucket/prefix from manifest identity first, then config fallback.
|
||||
- Uploads session `previous/**` files as durable session state when the local `previous/` directory exists.
|
||||
- Skips top-level promotion uploads for effective locked sources; run-local uploads still publish.
|
||||
- Effective locks are the union of `pipeline.archive.locks` and remote `{session_prefix}/locks.yml`; static pipeline locks win on duplicate sources.
|
||||
- Writes metadata including:
|
||||
- upload counts/paths
|
||||
- `previous_files_uploaded` and `previous_uploaded_paths`
|
||||
- `locked_promotion_count` and `locked_promotions`
|
||||
- `current_manifest_key`
|
||||
- `current_run_id_key`
|
||||
- `current_pointer_written`
|
||||
- On skipped archive path, returns metadata with `skipped=true` and pointer not written.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Stage may self-skip (metadata skip) when archive disabled or run upload disabled.
|
||||
- Runner-level skip also applies for previously succeeded stage unless forced.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing prerequisite success, missing object store when required, missing run root, missing unlocked required promotion source, upload failures, or pointer write failures.
|
||||
- Locked required promotions are intentional skips and do not fail archive.
|
||||
- Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/archive_test.go`
|
||||
- `internal/app/post_archive_cleanup_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Run upload excludes `audio/` subtree.
|
||||
- Session `previous/**` is archiveable durable input/provenance state, not run-local output.
|
||||
- Ordinary `--force` does not override archive locks.
|
||||
- Malformed or unreadable remote lock store fails archive-capable execution before promotion.
|
||||
- `current/manifest.json` uploads before `current/run_id.txt`.
|
||||
- `current/run_id.txt` is the remote publish commit marker.
|
||||
63
docs/internal/stage-merge.md
Normal file
63
docs/internal/stage-merge.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# Stage: merge
|
||||
|
||||
## Purpose
|
||||
Normalize per-speaker raw transcripts and merge them into one merged transcript via Seriatim.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `transcripts/raw/*.json`
|
||||
- `inputs/speakers.yml`
|
||||
- `inputs/autocorrect.yml`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/merged.json`
|
||||
- optional `artifacts/seriatim.report.json` (when report enabled)
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Raw transcript discovery/validation
|
||||
- Per-input normalize calls to Seriatim
|
||||
- Final merge call to Seriatim
|
||||
- Run-local log/config/report path wiring
|
||||
- Promotion of merged/report outputs to canonical paths
|
||||
|
||||
Does not own:
|
||||
- Transcript polishing or downstream artifact generation
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.seriatim.binary`
|
||||
- `pipeline.seriatim.timeout`
|
||||
- `pipeline.seriatim.output_schema`
|
||||
- `pipeline.seriatim.coalesce_gap`
|
||||
- `pipeline.seriatim.report`
|
||||
- `pipeline.seriatim.env.*`
|
||||
|
||||
## External Adapters Used
|
||||
- Seriatim adapter:
|
||||
- `Normalize` for each raw input
|
||||
- `Run` for final merge
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads transcript inputs from transcribe stage outputs in manifest when present; falls back to canonical raw directory.
|
||||
- Writes run-local outputs/logs/config under `runs/{run_id}/merge/...` when enabled.
|
||||
- Promotes canonical merged transcript and optional report.
|
||||
- Records normalized-input provenance and adapter metadata in stage metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced rerun of this or upstream stages can stale downstream succeeded stages via runner invalidation.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid raw transcripts, missing speakers/autocorrect files, normalize failure, merge failure, invalid merged output JSON, or invalid report JSON when enabled.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/merge_test.go`
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Merge consumes normalized forms of each raw transcript.
|
||||
- Merged transcript must validate before promotion.
|
||||
- Report output is optional and gated by config.
|
||||
56
docs/internal/stage-normalize.md
Normal file
56
docs/internal/stage-normalize.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# Stage: normalize
|
||||
|
||||
## Purpose
|
||||
Normalize the processed transcript into a deterministic intermediate schema for trim and optionally emit a normalize report.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `transcripts/processed.json`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/normalized.json` (or configured normalize output path)
|
||||
- optional `artifacts/seriatim.normalize.report.json`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Processed transcript discovery/validation
|
||||
- Normalize request construction and invocation
|
||||
- Optional normalize report wiring
|
||||
- Promotion of normalized transcript and optional report
|
||||
|
||||
Does not own:
|
||||
- Bounds detection or segment trimming
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.normalize.output_path`
|
||||
- `pipeline.normalize.output_schema`
|
||||
- `pipeline.normalize.report`
|
||||
- `pipeline.seriatim.binary`
|
||||
- `pipeline.seriatim.timeout`
|
||||
|
||||
## External Adapters Used
|
||||
- Seriatim adapter (`Normalize`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads processed transcript from polish outputs in manifest when present; falls back to canonical path.
|
||||
- Uses run-local output/report/log/config paths when run layout is enabled.
|
||||
- Promotes canonical normalized transcript and optional normalize report.
|
||||
- Records adapter/result metadata including source path selection.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced reruns can stale downstream succeeded stages.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid processed transcript, adapter error, invalid normalized output, or invalid report output when report enabled.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/normalize_test.go`
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Normalized output must validate as processed-transcript-compatible JSON (`segments` array required).
|
||||
- Default normalize config is applied when `pipeline.normalize` is unset.
|
||||
69
docs/internal/stage-polish.md
Normal file
69
docs/internal/stage-polish.md
Normal file
@@ -0,0 +1,69 @@
|
||||
# Stage: polish
|
||||
|
||||
## Purpose
|
||||
Polish merged transcript with Audita and produce a processed transcript for downstream normalization/analyze.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `transcripts/merged.json`
|
||||
- `inputs/glossary.yml`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/processed.json`
|
||||
- optional `artifacts/audita.report.json` (when report enabled)
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Merged transcript discovery/validation
|
||||
- Audita invocation request construction
|
||||
- Run-local logs/config/work-dir/report wiring
|
||||
- Promotion of processed transcript and optional report
|
||||
|
||||
Does not own:
|
||||
- Upstream merge normalization
|
||||
- Downstream normalize/trim/analyze logic
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.audita.binary`
|
||||
- `pipeline.audita.timeout`
|
||||
- `pipeline.audita.llm_api_key_env`
|
||||
- `pipeline.audita.modules`
|
||||
- `pipeline.audita.base_url`
|
||||
- `pipeline.audita.model`
|
||||
- `pipeline.audita.transcript_description`
|
||||
- `pipeline.audita.config_path`
|
||||
- `pipeline.audita.output_schema`
|
||||
- `pipeline.audita.work_dir_retention`
|
||||
- `pipeline.audita.total_llm_concurrency`
|
||||
- `pipeline.audita.proposal_llm_concurrency`
|
||||
- `pipeline.audita.validation_model`
|
||||
- `pipeline.audita.validation_llm_concurrency`
|
||||
- `pipeline.audita.report`
|
||||
|
||||
## External Adapters Used
|
||||
- Audita adapter (`env.Audita.Run`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads merged transcript from merge manifest outputs when available; falls back to canonical merged path.
|
||||
- Uses run-local output/report/log/config/scratch paths when run layout is enabled.
|
||||
- Promotes canonical `transcripts/processed.json` and optional report.
|
||||
- Records adapter invocation metadata, credential presence signal, and output provenance in stage metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced rerun can stale downstream succeeded stages via runner invalidation.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid merged transcript, missing glossary, adapter error, invalid processed output shape (`segments` array required), or invalid report JSON when enabled.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/polish_test.go`
|
||||
- `internal/adapters/audita/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Processed transcript must contain a top-level `segments` array.
|
||||
- Report behavior is strictly config-gated.
|
||||
- Stage output canonicalization always ends at `transcripts/processed.json`.
|
||||
124
docs/internal/stage-prepare.md
Normal file
124
docs/internal/stage-prepare.md
Normal file
@@ -0,0 +1,124 @@
|
||||
# Stage: prepare
|
||||
|
||||
## Purpose
|
||||
Materialize canonical current-session input state and provenance before downstream stages run.
|
||||
|
||||
Prepare owns:
|
||||
- local input file materialization (`inputs/**`);
|
||||
- audio input materialization (`audio/**`);
|
||||
- previous-session cache hydration (`previous/**`) for canonical previous-session artifact sources.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- resolved config/campaign/session (`pipeline.yml`, `campaign.yml`, `session.yml`);
|
||||
- remote session provenance when `session.yml` was loaded from S3;
|
||||
- campaign or session input files (`speakers`, `autocorrect`, `glossary`);
|
||||
- audio source:
|
||||
- local: `session.inputs.audio_dir` or `session.inputs.audio_files`;
|
||||
- S3: `session.inputs.audio_s3.prefix`;
|
||||
- configured enabled Scriptorium artifact inputs (for previous-session requirement scanning);
|
||||
- remote previous-session current archive state when previous hydration is required.
|
||||
|
||||
Outputs:
|
||||
- `inputs/campaign.yml`;
|
||||
- `inputs/session.yml`;
|
||||
- `inputs/pipeline.resolved.yml`;
|
||||
- `inputs/speakers.yml`;
|
||||
- `inputs/autocorrect.yml`;
|
||||
- `inputs/glossary.yml`;
|
||||
- `audio/*.flac` in canonical session `audio/`;
|
||||
- optional `previous/manifest.json`;
|
||||
- optional `previous/artifacts/**`;
|
||||
- deterministic `manifest.Inputs` records with checksums and provenance metadata.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- input path resolution and materialization;
|
||||
- S3 audio list/download/copy flow;
|
||||
- previous-session artifact requirement collection from enabled configured artifacts;
|
||||
- previous cache lifecycle when requirements exist (clear and rehydrate managed `previous/` state).
|
||||
|
||||
Does not own:
|
||||
- transcript or artifact generation;
|
||||
- analyze-stage source resolution;
|
||||
- archive commit behavior.
|
||||
|
||||
## Config fields used
|
||||
- `session.session_id`
|
||||
- `session.previous_session_id`
|
||||
- `session.campaign`
|
||||
- `session.inputs.speakers_file`
|
||||
- `session.inputs.autocorrect_file`
|
||||
- `session.inputs.glossary_file`
|
||||
- `session.inputs.audio_dir`
|
||||
- `session.inputs.audio_files`
|
||||
- `session.inputs.audio_s3.prefix`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.spool.root`
|
||||
- `pipeline.cache.root`
|
||||
- `pipeline.cache.s3_audio`
|
||||
- `pipeline.storage.s3.bucket`
|
||||
- `pipeline.storage.s3.root_prefix`
|
||||
- `pipeline.scriptorium.artifacts.<name>.enabled`
|
||||
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source`
|
||||
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required`
|
||||
- `campaign.campaign`
|
||||
- `campaign.inputs.speakers_file`
|
||||
- `campaign.inputs.autocorrect_file`
|
||||
- `campaign.inputs.glossary_file`
|
||||
|
||||
## External adapters used
|
||||
- `storage.ObjectStore` for:
|
||||
- S3 audio listing/downloads;
|
||||
- previous-session current pointer/manifest/artifact object checks and downloads.
|
||||
|
||||
## State and manifest behavior
|
||||
- Ensures workspace layout exists.
|
||||
- Materializes canonical input files and audio files.
|
||||
- For S3 audio, uses run-scoped spool for active downloads and durable cache for reusable audio files; cache hits copy directly to work audio without downloading the object again.
|
||||
- Records `inputs/session.yml` provenance as local `session_config` or remote `session_config.s3`.
|
||||
- Resolves campaign-provided stable input paths relative to `campaign.yml`.
|
||||
- Resolves session-provided stable input overrides relative to `session.yml`.
|
||||
- Scans enabled configured artifact inputs for canonical sources:
|
||||
- `narratio.previous_session.artifact.<artifact_key>`
|
||||
- If one or more canonical previous-session requirements exist:
|
||||
- clears managed `previous/` state;
|
||||
- hydrates required/optional previous artifacts from the configured previous session’s committed archive current state;
|
||||
- writes `previous/manifest.json` and hydrated `previous/artifacts/**`;
|
||||
- stores archive-relative artifact paths such as `artifacts/session_recap.md` as `previous/artifacts/session_recap.md`, not `previous/artifacts/artifacts/session_recap.md`;
|
||||
- records hydrated previous inputs in `manifest.Inputs` with source `previous_session_archive.current`.
|
||||
- If no canonical previous-session requirements exist, prepare does not manage `previous/`.
|
||||
- `manifest.Inputs` is sorted deterministically by `(kind, path)`.
|
||||
- S3 audio `manifest.Inputs` retain S3 provenance and include `cache_path`; `spool_path` is present only when the current prepare invocation downloaded the file.
|
||||
|
||||
## Required and optional previous-session behavior
|
||||
- `previous_session_id` unset:
|
||||
- if any referenced previous artifact is required: fail;
|
||||
- if all referenced previous artifacts are optional: continue and omit them.
|
||||
- Previous session archive current pointer or manifest missing:
|
||||
- if any referenced previous artifact is required: fail;
|
||||
- if all referenced previous artifacts are optional: continue and omit missing ones.
|
||||
- Missing required previous artifact object: fail.
|
||||
- Missing optional previous artifact object: omit.
|
||||
- Downloaded previous artifacts must validate as non-empty files.
|
||||
|
||||
## Skip and resume behavior
|
||||
- Runner-level skip remains authoritative:
|
||||
- if `prepare` already succeeded and run is not forced, `prepare` does not run and no hydration/download occurs.
|
||||
- If `prepare` runs (including with `--force`), it owns managed `previous/` state for canonical previous-session inputs.
|
||||
|
||||
## Failure behavior
|
||||
- Fails on missing required input files, invalid audio-source combinations, empty/duplicate audio inputs, missing object store for S3 modes, and remote access/download/validation errors.
|
||||
- For required canonical previous-session inputs, analyze-time missing-input guidance is to rerun:
|
||||
- `narratio run-stage --force prepare`
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/stage/prepare_test.go`
|
||||
- `internal/stage/prepare_previous_test.go`
|
||||
- `internal/artifacts/previous_requirements_test.go`
|
||||
- `internal/app/runner_test.go`
|
||||
|
||||
## Architectural invariants
|
||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive.
|
||||
- Storage keys are computed by callers using archive/path helpers; storage adapter receives explicit keys.
|
||||
- `prepare` is the only stage that hydrates canonical previous-session cache state.
|
||||
58
docs/internal/stage-transcribe.md
Normal file
58
docs/internal/stage-transcribe.md
Normal file
@@ -0,0 +1,58 @@
|
||||
# Stage: transcribe
|
||||
|
||||
## Purpose
|
||||
Generate per-speaker raw transcripts from prepared audio using WhisperX.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `audio/*.flac` prepared by `prepare`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/raw/<speaker>.json` for each input audio file
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Discovering prepared audio inputs
|
||||
- Deriving speaker ids from audio basenames
|
||||
- Parallel WhisperX invocation with bounded concurrency
|
||||
- Validating produced JSON and promoting run-local outputs
|
||||
|
||||
Does not own:
|
||||
- Transcript merge/polish/normalize/trim/analyze
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.whisperx.transcribe_url`
|
||||
- `pipeline.whisperx.language`
|
||||
- `pipeline.whisperx.timeout`
|
||||
- `pipeline.whisperx.retries`
|
||||
- `pipeline.whisperx.retry_delay`
|
||||
- `pipeline.whisperx.concurrency`
|
||||
|
||||
## External Adapters Used
|
||||
- WhisperX adapter (`env.WhisperX.Transcribe`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Uses run-local output paths under `runs/{run_id}/transcribe/outputs/...` when run layout is enabled.
|
||||
- Validates each generated transcript JSON before promotion.
|
||||
- Promotes canonical outputs to `transcripts/raw/*.json`.
|
||||
- Records per-file metadata (attempts/status/duration/output path) in stage metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies for previously succeeded stage unless forced.
|
||||
- On forced upstream reruns, downstream succeeded stages can be marked `stale` by runner logic.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails if no prepared audio exists, duplicate speaker basenames are detected, adapter output path mismatches expected path, any output JSON is invalid, or one worker fails.
|
||||
- Cancels in-flight workers after first terminal error.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/transcribe_test.go`
|
||||
- `internal/app/whisperx_wiring_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Speaker identity is derived from `.flac` basename and must be unique.
|
||||
- Every successful speaker output must be valid JSON before promotion.
|
||||
- Canonical raw transcript set is the only supported merge input surface.
|
||||
75
docs/internal/stage-trim.md
Normal file
75
docs/internal/stage-trim.md
Normal file
@@ -0,0 +1,75 @@
|
||||
# Stage: trim
|
||||
|
||||
## Purpose
|
||||
Optionally trim the normalized transcript to session bounds; always produce a durable trimmed transcript.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `transcripts/normalized.json`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/trimmed.json` (or configured trim output path)
|
||||
- when trim enabled: `artifacts/session_bounds.json`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Trim-enabled switch behavior
|
||||
- Bounds generation via Scriptorium artifact run
|
||||
- Bounds validation against normalized transcript
|
||||
- Keep-selector derivation and Seriatim trim invocation
|
||||
- Copy-through behavior when disabled or bounds indicate unchanged transcript
|
||||
|
||||
Does not own:
|
||||
- Upstream normalization
|
||||
- Downstream artifact analysis
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.trim.enabled`
|
||||
- `pipeline.trim.output_path`
|
||||
- `pipeline.trim.bounds.prompt_id`
|
||||
- `pipeline.trim.bounds.profile_id`
|
||||
- `pipeline.trim.bounds.timeout`
|
||||
- `pipeline.trim.bounds.output_path`
|
||||
- `pipeline.trim.bounds.transcript_input_name`
|
||||
- `pipeline.trim.bounds.render_debug`
|
||||
- `pipeline.trim.bounds.render_output_path`
|
||||
- `pipeline.seriatim.binary`
|
||||
- `pipeline.seriatim.timeout`
|
||||
- `pipeline.scriptorium.binary`
|
||||
- `pipeline.scriptorium.config_path`
|
||||
- `pipeline.scriptorium.timeout`
|
||||
|
||||
## External Adapters Used
|
||||
- Scriptorium adapter:
|
||||
- optional `RenderArtifact` for bounds debug render
|
||||
- `RunArtifact` for bounds output
|
||||
- Seriatim adapter:
|
||||
- `Trim` when bounds indicate trimming is required
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads normalized transcript from normalize manifest outputs when available; falls back to canonical path.
|
||||
- Uses run-local outputs/logs/reports/config/scratch paths when run layout is enabled.
|
||||
- Promotes canonical trimmed transcript; promotes session bounds when trim enabled.
|
||||
- Records bounds diagnostics, trim action, keep selector, and adapter metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced reruns can stale downstream succeeded stages.
|
||||
- When `trim.enabled=false`, stage still succeeds by copying normalized to trimmed output.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid normalized transcript.
|
||||
- With trim enabled, fails on missing adapters/config, bounds generation/validation errors, invalid bounds JSON, invalid range/segment ids, trim adapter failures, or invalid trimmed output.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/trim_test.go`
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Trim never falls back to processed transcript; normalized transcript is required input.
|
||||
- `session_bounds` output exists only for enabled trim path.
|
||||
- Render-debug artifacts are diagnostics and not declared stage outputs.
|
||||
75
docs/internal/storage.md
Normal file
75
docs/internal/storage.md
Normal file
@@ -0,0 +1,75 @@
|
||||
# Internal: Storage
|
||||
|
||||
## Purpose
|
||||
Document Narratio's remote storage backend contracts and implementations under `internal/adapters/storage`.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- Resolved storage config (`pipeline.storage.*`).
|
||||
- Already-loaded environment variables for configured S3 credentials.
|
||||
- Bucket-relative object keys and local file paths from app/stage orchestration.
|
||||
|
||||
Outputs:
|
||||
- Listed/downloaded/uploaded object metadata (`ObjectInfo`).
|
||||
- Existence checks and storage-layer errors.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Remote object-store interface and implementation details.
|
||||
- S3 client wiring and API calls.
|
||||
- Object key normalization and upload/download/list primitives.
|
||||
|
||||
Does not own:
|
||||
- Session/run prefix semantics.
|
||||
- Archive commit order semantics.
|
||||
- Manifest updates.
|
||||
- Filesystem secret loading from `pipeline.secrets.env_dir`.
|
||||
|
||||
## Config fields used
|
||||
- `pipeline.storage.backend`
|
||||
- `pipeline.storage.s3.bucket`
|
||||
- `pipeline.storage.s3.region`
|
||||
- `pipeline.storage.s3.endpoint`
|
||||
- `pipeline.storage.s3.force_path_style`
|
||||
- `pipeline.storage.s3.access_key_id_env`
|
||||
- `pipeline.storage.s3.secret_access_key_env`
|
||||
|
||||
## External adapters used
|
||||
Storage package contracts:
|
||||
- `ObjectStore` (active remote object-store boundary): `List`, `Download`, `Upload`, `Exists`.
|
||||
- `Backend` (archive request boundary): currently implemented with `NoopBackend` only.
|
||||
|
||||
Implementations:
|
||||
- `S3Backend`: AWS SDK-backed `ObjectStore` implementation.
|
||||
- `FakeBackend`: deterministic test `ObjectStore` and archive backend.
|
||||
- `NoopBackend`: deterministic no-op archive backend for compatibility wiring.
|
||||
|
||||
## State and manifest behavior
|
||||
- Storage implementations are stateless with respect to manifest/session lifecycle.
|
||||
- Caller supplies fully-qualified bucket-relative keys.
|
||||
- Storage layer does not infer campaign/session/run/root-prefix semantics.
|
||||
- Caller controls publish ordering; storage layer executes individual operations in the order invoked.
|
||||
|
||||
## Skip and resume behavior
|
||||
- No storage-level skip/resume behavior.
|
||||
- Skip/resume decisions are made by stage/app logic before storage calls occur.
|
||||
|
||||
## Failure behavior
|
||||
- `NewObjectStoreFromConfig` fails when no remote backend is configured or required S3 config is missing.
|
||||
- `S3Backend` constructor fails when required bucket is missing or AWS client setup fails.
|
||||
- App command orchestration loads configured filesystem secrets before calling the object-store factory.
|
||||
- CRUD operations return contextual errors (including not-found behavior via `Exists`).
|
||||
- Key normalization is applied before operations (`\\` to `/`, leading slash trimmed).
|
||||
- Remote session loading uses `List` to find the exact `session.yml` key and `Download` to materialize it to a local temp file.
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/adapters/storage/factory_test.go`
|
||||
- `internal/adapters/storage/s3_backend_test.go`
|
||||
- `internal/adapters/storage/fake_test.go`
|
||||
- `internal/adapters/storage/keys_test.go`
|
||||
- `internal/adapters/storage/archive.go` + consumers in stage tests (`prepare`, `archive`)
|
||||
|
||||
## Architectural invariants
|
||||
- Callers pass full bucket-relative keys.
|
||||
- Storage backends must not prepend or infer narratio prefixes.
|
||||
- Remote transport details remain isolated to storage adapter implementations.
|
||||
77
docs/internal/workspace.md
Normal file
77
docs/internal/workspace.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# Workspace internals
|
||||
|
||||
## Purpose
|
||||
Define the local durable and run-local workspace model used by stages, manifests, resume, and archive.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `pipeline.workspace.root`
|
||||
- `session.campaign`
|
||||
- `session.session_id`
|
||||
- generated `run_id`
|
||||
|
||||
Outputs:
|
||||
- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`
|
||||
- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`
|
||||
- Canonical durable session directories and run-local stage trees
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Session-level path layout (`inputs/`, `audio/`, `transcripts/`, `artifacts/`, `reports/`, `logs/`, `config/`, `current/`, `runs/`, `previous/`)
|
||||
- `previous/manifest.json` and `previous/artifacts/**` are reserved for prepared previous-session state
|
||||
- Run-local stage sandbox layout under `runs/{run_id}/{stage}/`
|
||||
- Session lock acquisition/release (`.lock`)
|
||||
|
||||
Does not own:
|
||||
- Stage business logic
|
||||
- Remote archive semantics (documented in `stage-archive.md`)
|
||||
- CLI argument parsing
|
||||
|
||||
## Config Fields Used
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.workspace.cleanup_after_archive`
|
||||
- `pipeline.spool.root`
|
||||
- `pipeline.spool.delete_audio_after_archive`
|
||||
- `pipeline.cache.root`
|
||||
- `pipeline.cache.s3_audio`
|
||||
- `session.campaign`
|
||||
- `session.session_id`
|
||||
|
||||
## External Adapters Used
|
||||
None directly in this subsystem. Stages may use object storage adapters and then write local outputs into this layout.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Session state is persisted in the session manifest (`manifest.Manifest`).
|
||||
- Invocation history is persisted per run in run manifests under `runs/{run_id}/manifest.json`.
|
||||
- During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and promoted to canonical session paths after stage success.
|
||||
- `manifest.Artifacts` entries record `ProducerRunID` for durable outputs.
|
||||
- For S3 audio sessions, `prepare` records work/cache paths, S3 provenance, and spool path when the invocation downloaded the object.
|
||||
- Durable cache state under `pipeline.cache.root` is not workspace state and is preserved by default by `narratio clean`.
|
||||
- `narratio clean --session-id <id>` removes the session work root and session spool root.
|
||||
- `narratio clean --all` removes all local session work under `workspace.root/work` and spool children under `spool.root`.
|
||||
- `narratio clean --clear-cache` is the explicit opt-in for deleting matching S3 audio cache entries.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Skip/resume decisions are made in `internal/app` (`run_control.go`, `resume.go`) using stage status in the session manifest.
|
||||
- `--force` reruns selected stages and marks downstream previously-succeeded stages as `stale`.
|
||||
- Workspace layout is idempotent (`EnsureLayoutFor`) and reused across runs.
|
||||
|
||||
## Failure Behavior
|
||||
- Failures preserve manifests and run-local files for inspection.
|
||||
- Lock conflicts fail fast via `ErrLockConflict`.
|
||||
- Cleanup can fail post-archive; failure is recorded in archive stage metadata and returned by the run.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/artifacts/local_test.go`
|
||||
- `internal/stage/run_local_test.go`
|
||||
- `internal/app/run_control_test.go`
|
||||
- `internal/app/resume_run_stage_test.go`
|
||||
- `internal/app/post_archive_cleanup_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Session root is campaign-aware: `{workspace.root}/work/{campaign}/{session_id}`.
|
||||
- Run roots are always nested: `runs/{run_id}` under the session root.
|
||||
- Run-local output promotion must end in canonical session paths.
|
||||
- `previous/**` is session-durable state and must not be treated as run-local output scratch state.
|
||||
- Automatic post-archive cleanup only targets run-scoped directories and must never delete configured root directories.
|
||||
- Manual `clean` may delete session-scoped directories or the `workspace.root/work` directory, but it must preserve configured root directories and reject unsafe targets.
|
||||
279
docs/operations.md
Normal file
279
docs/operations.md
Normal file
@@ -0,0 +1,279 @@
|
||||
# Operations
|
||||
|
||||
This guide describes the implemented operator lifecycle for Narratio.
|
||||
|
||||
For field-level configuration, see [docs/config.md](./config.md). For full command/flag reference, see [docs/cli.md](./cli.md).
|
||||
|
||||
## Normal workflow (S3-first path)
|
||||
|
||||
1. Create or upload `session.yml`, or pass a local `session.yml` explicitly.
|
||||
2. Upload session `.flac` files to object storage under the configured session audio prefix.
|
||||
3. Run Narratio:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
4. Read success output:
|
||||
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
||||
- use `manifest=<path>` with `status` for inspection.
|
||||
|
||||
Notes:
|
||||
- default config/campaign/session discovery checks system config locations unless `--config`, `--campaign`, and `--session` are passed.
|
||||
- when local `session.yml` discovery misses, `--session-id` loads remote `session.yml` from `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`.
|
||||
- S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access.
|
||||
|
||||
Initialize a remote session skeleton:
|
||||
|
||||
```bash
|
||||
narratio session init --config /etc/narratio/pipeline.yml --campaign /etc/narratio/campaign.yml --session-id 2026-04-04 --remote
|
||||
```
|
||||
|
||||
Remote init writes `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`. It fails if the object already exists unless `--force` is passed.
|
||||
|
||||
Validate before running:
|
||||
|
||||
```bash
|
||||
narratio session validate --session-id 2026-04-04
|
||||
```
|
||||
|
||||
## Restore workflow
|
||||
|
||||
Use restore when local durable session state is missing or stale and archive current state is authoritative.
|
||||
|
||||
Dry-run (no local writes):
|
||||
|
||||
```bash
|
||||
narratio restore --session-id 2026-04-04 --dry-run
|
||||
```
|
||||
|
||||
Execution:
|
||||
|
||||
```bash
|
||||
narratio restore --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Post-restore analyze rerun pattern:
|
||||
|
||||
```bash
|
||||
narratio analyze --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Restore source-of-truth:
|
||||
- remote commit marker: `current/run_id.txt`
|
||||
- remote current manifest: `current/manifest.json`
|
||||
|
||||
Restore default scope:
|
||||
- includes `manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`
|
||||
- includes `audio/**` only with `--include-audio`
|
||||
- excludes `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`, and `current/**` (except remote `current/manifest.json` as source)
|
||||
|
||||
Reset local state before restore testing:
|
||||
|
||||
```bash
|
||||
narratio clean --session-id 2026-04-04 --dry-run
|
||||
narratio clean --session-id 2026-04-04
|
||||
narratio restore --session-id 2026-04-04 --include-audio
|
||||
```
|
||||
|
||||
`clean` removes the local session work directory and session spool directory. It preserves the durable S3 audio cache by default, so repeated restore or forced prepare tests do not re-download large audio files.
|
||||
|
||||
## Local filesystem layout and state artifacts
|
||||
|
||||
Session root:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/`
|
||||
|
||||
Primary state:
|
||||
- `manifest.json`: session-level stage state.
|
||||
- `runs/{run_id}/manifest.json`: invocation-level state.
|
||||
- `.lock`: session lock while a modifying command is active.
|
||||
- `inputs/campaign.yml`, `inputs/session.yml`, and `inputs/pipeline.resolved.yml`: materialized config inputs for the run.
|
||||
|
||||
Canonical session directories:
|
||||
- `inputs/`
|
||||
- `audio/`
|
||||
- `transcripts/`
|
||||
- `artifacts/`
|
||||
- `previous/`
|
||||
- `reports/`
|
||||
- `logs/`
|
||||
- `config/`
|
||||
- `current/`
|
||||
- `runs/`
|
||||
|
||||
Run-local stage directories:
|
||||
- `runs/{run_id}/{stage}/` with stage-local `outputs/`, `logs/`, `reports/`, `config/`, `scratch/`.
|
||||
|
||||
Behavior:
|
||||
- directory creation is idempotent.
|
||||
- stage outputs are generally generated run-local first, then promoted to canonical paths on success.
|
||||
- restore installs downloaded files to canonical session paths and does not recreate historical run sandboxes.
|
||||
|
||||
## Analyze artifact execution lifecycle
|
||||
|
||||
Analyze executes configured artifacts from `pipeline.scriptorium.artifacts`.
|
||||
|
||||
Execution model:
|
||||
- executable set = enabled artifacts, filtered by `--artifacts` when provided.
|
||||
- artifact-to-artifact dependencies are declared via `depends_on`.
|
||||
- selected artifacts run in deterministic dependency order.
|
||||
- after each successful artifact run, output is promoted to configured canonical `output_path`.
|
||||
|
||||
Configured artifact source reuse:
|
||||
- a non-executable configured artifact can satisfy inputs if its configured output file already exists and is valid.
|
||||
- reused configured artifact provenance is `filesystem.disabled_artifact_output`.
|
||||
|
||||
`--artifacts` behavior:
|
||||
- accepted on `run`, `resume`, `run-stage analyze`, and `analyze`.
|
||||
- filters analyze execution only.
|
||||
- does not imply force on `run`, `resume`, or `run-stage`; `narratio analyze` is force-by-design.
|
||||
|
||||
Canonical previous-session input behavior:
|
||||
- canonical sources use `narratio.previous_session.artifact.<artifact_key>`.
|
||||
- these inputs are hydrated by `prepare`, not `analyze`.
|
||||
- if analyze fails due to missing canonical previous cache, rerun:
|
||||
- `narratio run-stage --session-id <id> --force prepare`
|
||||
|
||||
## Remote archive layout and publish contract
|
||||
|
||||
When archive is enabled and run upload is enabled, archive publishes under:
|
||||
|
||||
- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
- run prefix: `{session_prefix}/runs/{run_id}/`
|
||||
|
||||
Archive uploads:
|
||||
- run record files from run root (excluding `audio/`).
|
||||
- promoted files from explicit `archive.promote_artifacts` rules.
|
||||
- mutable session locks from helper commands live at `{session_prefix}/locks.yml`.
|
||||
|
||||
Publish order:
|
||||
1. upload `current/manifest.json`
|
||||
2. upload `current/run_id.txt` last
|
||||
|
||||
`current/run_id.txt` is the remote commit marker.
|
||||
|
||||
Archive promotion is explicit and source-based:
|
||||
- Narratio does not auto-promote all generated analyze artifacts.
|
||||
- each rule resolves `source` through the artifact resolver/catalog model, then uploads to `dest`.
|
||||
- missing required promotion sources fail archive stage.
|
||||
- missing optional promotion sources are skipped.
|
||||
- invalid resolved artifacts fail archive stage.
|
||||
- `archive.locks` skips top-level promotion overwrites for locked sources while run-local uploads still publish.
|
||||
- remote locks from `{session_prefix}/locks.yml` are merged with static `archive.locks`; static locks win on duplicate sources.
|
||||
- locked required promotions are treated as intentional successful skips and are recorded in archive metadata.
|
||||
|
||||
Lock helper behavior:
|
||||
- `narratio locks --session-id <id>` lists effective static and remote locks.
|
||||
- `narratio locks add --session-id <id> --reason <text> <source>` writes a remote lock.
|
||||
- `narratio locks add --session-id <id> --force --reason <text> <source>` updates an existing remote lock reason.
|
||||
- `narratio locks remove --session-id <id> <source>` removes only a remote lock.
|
||||
- `locks remove` cannot remove static pipeline locks.
|
||||
- remote lock writes check whether the lock store exists, but are not compare-and-swap atomic.
|
||||
|
||||
## Resume, retry, restore, and safe rerun behavior
|
||||
|
||||
Default skip:
|
||||
- `run` and `run-stage` skip already-succeeded stages unless `--force` is set.
|
||||
|
||||
Resume:
|
||||
- `resume` starts at first non-succeeded stage.
|
||||
- `resume --force` runs full stage order.
|
||||
|
||||
Restore conflict policy:
|
||||
- restore classifies local differences as conflicts.
|
||||
- without `--force`, restore fails when conflicts exist.
|
||||
- with `--force`, conflicting local files are overwritten by remote archive files.
|
||||
|
||||
Forced reruns:
|
||||
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
|
||||
- ordinary `--force` does not override archive locks.
|
||||
|
||||
Safe rerun pattern:
|
||||
1. rerun the changed stage with `--force`.
|
||||
2. run `resume` to rebuild downstream stages.
|
||||
|
||||
## Cleanup behavior
|
||||
|
||||
Automatic post-archive cleanup is considered only when archive stage executed and succeeded.
|
||||
|
||||
Automatic cleanup toggles:
|
||||
- `pipeline.spool.delete_audio_after_archive=true` deletes run-scoped spool audio.
|
||||
- `pipeline.workspace.cleanup_after_archive=true` deletes run-scoped local run directory.
|
||||
|
||||
Manual cleanup:
|
||||
- `narratio clean --session-id <id>` deletes `{workspace.root}/work/{campaign}/{session_id}` and `{spool.root}/{campaign}/{session_id}`.
|
||||
- `narratio clean --all` deletes all local session work under `{workspace.root}/work` and all spool children under `{spool.root}`.
|
||||
- `--dry-run` prints targets without deleting.
|
||||
- `--clear-cache` also removes matching S3 audio cache files. Without it, cache is preserved.
|
||||
|
||||
The S3 audio cache under `pipeline.cache.root` is durable input cache state, not workspace or spool state. Automatic cleanup and default manual cleanup do not delete it.
|
||||
|
||||
Cleanup eligibility gates:
|
||||
- archive enabled
|
||||
- archive run upload enabled
|
||||
- run record upload completed
|
||||
- current pointer write completed (`current/run_id.txt` written)
|
||||
|
||||
No cleanup for failed/incomplete/unarchived/archive-skipped runs.
|
||||
|
||||
## Failure and recovery playbooks
|
||||
|
||||
After run failure, Narratio keeps:
|
||||
- session manifest
|
||||
- run manifest
|
||||
- run-local artifacts/logs/config/reports
|
||||
|
||||
Failed or incomplete runs remain local-only.
|
||||
|
||||
After restore failure:
|
||||
- already-installed restore files remain in place.
|
||||
- restore does not roll back prior successful installs.
|
||||
- existing local manifest is preserved if restored manifest validation/install fails.
|
||||
|
||||
Recommended recovery:
|
||||
|
||||
1. inspect state:
|
||||
|
||||
```bash
|
||||
narratio status --session-id 2026-04-04
|
||||
```
|
||||
|
||||
This reports local manifest state, committed remote current state, expected remote transcript/artifact availability, and archive locks.
|
||||
|
||||
2. for one manifest file, run:
|
||||
|
||||
```bash
|
||||
narratio status --manifest <manifest-path>
|
||||
```
|
||||
|
||||
3. for restore-specific checks, run:
|
||||
|
||||
```bash
|
||||
narratio restore --session-id 2026-04-04 --dry-run
|
||||
```
|
||||
|
||||
4. fix root cause (config/input/credentials/storage/service availability).
|
||||
5. continue with `resume`, or targeted `run-stage --force` followed by `resume`.
|
||||
|
||||
## Restore report
|
||||
|
||||
Non-dry-run restore writes a durable report at:
|
||||
- `reports/restore-latest.json`
|
||||
|
||||
Report content includes:
|
||||
- identity (`campaign`, `session_id`, `run_id`)
|
||||
- mode flags (`dry_run`, `force`, `include_audio`)
|
||||
- plan counts and execution counts
|
||||
- per-action status
|
||||
|
||||
Dry-run does not write restore report files.
|
||||
|
||||
## Operational caveats
|
||||
|
||||
- `status` with no config/session flags still requires explicit `--manifest`.
|
||||
- `status --session-id <id>` uses normal config/session loading, including remote session fallback.
|
||||
- `status --session-id <id>` includes the same promoted remote output availability view as `artifacts list --remote` when storage is configured.
|
||||
- local and S3 audio input modes are mutually exclusive.
|
||||
- archive publish requires upstream stages through `analyze` to be `succeeded`.
|
||||
- required promotion rules can fail when selected analyze artifacts did not generate a required file path.
|
||||
- restore requires configured remote object storage and committed remote current state.
|
||||
52
docs/roadmap/operations.md
Normal file
52
docs/roadmap/operations.md
Normal file
@@ -0,0 +1,52 @@
|
||||
# Roadmap: Operator Helper Commands
|
||||
|
||||
## Status
|
||||
|
||||
Implemented.
|
||||
|
||||
The operator helper command set is no longer conceptual. Current behavior is documented in:
|
||||
|
||||
- `docs/cli.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/config.md`
|
||||
- `docs/internal/artifacts.md`
|
||||
- `docs/internal/stage-archive.md`
|
||||
|
||||
## Implemented Commands
|
||||
|
||||
- `narratio session validate`
|
||||
- `narratio status --manifest <path>`
|
||||
- `narratio status --session-id <id>`
|
||||
- `narratio session init --output <path>`
|
||||
- `narratio session init --remote`
|
||||
- `narratio artifacts list`
|
||||
- `narratio artifacts list --remote`
|
||||
- `narratio locks`
|
||||
- `narratio locks add <source>`
|
||||
- `narratio locks remove <source>`
|
||||
|
||||
## Implemented Decisions
|
||||
|
||||
- Helper output is text-only. No JSON schema exists yet.
|
||||
- `status` remains a top-level command.
|
||||
- `session validate`, `session init`, and `artifacts list` are nested helper commands.
|
||||
- `locks` is the single top-level command for listing, adding, and removing archive promotion locks.
|
||||
- Remote session initialization requires explicit `--remote`.
|
||||
- Local session initialization requires `--output`.
|
||||
- Remote artifact availability is opt-in with `artifacts list --remote`.
|
||||
- Mutable locks are source-based and stored at `{session_prefix}/locks.yml`.
|
||||
- The remote lock store uses strict YAML with top-level `locks`.
|
||||
- Static `pipeline.archive.locks` and remote locks are merged; static locks win on duplicate sources.
|
||||
- `locks remove` removes only remote locks.
|
||||
- Ordinary execution `--force` does not override locks.
|
||||
- Remote lock writes use existence checks and `--force` for updates; there is no compare-and-swap protection.
|
||||
|
||||
## Remaining Future Enhancements
|
||||
|
||||
These are intentionally not implemented:
|
||||
|
||||
- `--json` output for helper commands.
|
||||
- Optimistic concurrency or ETag compare-and-swap for remote lock mutations.
|
||||
- Rich remote artifact availability across historical run-local objects.
|
||||
- Session-lock acquisition for remote mutation helpers.
|
||||
- Broader campaign helper commands such as `campaign validate` or `campaign publish`.
|
||||
@@ -1,119 +0,0 @@
|
||||
# Archive Storage
|
||||
|
||||
This document describes implemented archive-stage publish behavior.
|
||||
|
||||
## S3 Paths
|
||||
|
||||
Session root:
|
||||
|
||||
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
|
||||
Run prefix:
|
||||
|
||||
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
|
||||
|
||||
## Scope
|
||||
|
||||
Implemented:
|
||||
|
||||
- archive uploads successful run records to remote object storage through the storage backend abstraction.
|
||||
- archive uploads configured promoted outputs to session-level keys.
|
||||
- archive uploads `current/manifest.json`.
|
||||
- archive uploads `current/run_id.txt` last as the effective commit marker.
|
||||
- optional post-archive local cleanup:
|
||||
- `pipeline.spool.delete_audio_after_archive: true` removes only the run-scoped spool audio directory
|
||||
- `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir
|
||||
- tests use fake storage and do not require live S3.
|
||||
|
||||
Future work:
|
||||
|
||||
- `notify` stage behavior
|
||||
- stale detection
|
||||
- optional future source-audio upload mode
|
||||
- additional artifact generation beyond current implemented set
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Archive verifies these stages succeeded before upload:
|
||||
|
||||
- `prepare`
|
||||
- `transcribe`
|
||||
- `merge`
|
||||
- `polish`
|
||||
- `normalize`
|
||||
- `trim`
|
||||
- `analyze`
|
||||
|
||||
If any prerequisite is missing or not succeeded, archive fails and does not upload.
|
||||
Failed or incomplete runs remain local only.
|
||||
|
||||
## Run Upload
|
||||
|
||||
Archive uploads existing files from the run workdir when present:
|
||||
|
||||
- `inputs/`
|
||||
- `transcripts/`
|
||||
- `artifacts/`
|
||||
- `reports/` (optional)
|
||||
- `config/`
|
||||
- `logs/`
|
||||
- `manifest.json`
|
||||
|
||||
Relative paths are preserved under `runs/{run_id}/`.
|
||||
|
||||
## Promotion Rules
|
||||
|
||||
Archive applies `archive.promote_artifacts` in config order.
|
||||
|
||||
Rule behavior:
|
||||
|
||||
- `from`: local workdir-relative source path
|
||||
- `to`: session-root-relative destination key
|
||||
- `required: true`: missing source fails archive
|
||||
- `required: false`: missing source is skipped and recorded
|
||||
|
||||
Default promoted outputs:
|
||||
|
||||
- `transcripts/trimmed.json`
|
||||
- `artifacts/session_recap.md`
|
||||
|
||||
## Current Pointers
|
||||
|
||||
Archive writes:
|
||||
|
||||
1. `current/manifest.json` (after run upload + promotions)
|
||||
2. `current/run_id.txt` last
|
||||
|
||||
`current/run_id.txt` contains exactly:
|
||||
|
||||
- `{run_id}` plus trailing newline
|
||||
|
||||
Writing `current/run_id.txt` last makes it the effective commit marker for published session state.
|
||||
|
||||
If any required run upload, promotion upload, or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`.
|
||||
Cleanup runs only after this commit-marker write has succeeded.
|
||||
|
||||
## Audio Upload Policy
|
||||
|
||||
Archive does not upload local `audio/` by default.
|
||||
Original audio is expected at the session-level audio prefix and is not duplicated under `runs/{run_id}/`.
|
||||
|
||||
## Config Controls
|
||||
|
||||
- `archive.enabled: false` skips archive cleanly.
|
||||
- `archive.upload_run: false` skips run upload cleanly.
|
||||
- both skip cases also skip post-archive local cleanup.
|
||||
|
||||
## Metadata
|
||||
|
||||
Archive stage metadata includes non-secret upload context (for example):
|
||||
|
||||
- `s3_bucket`
|
||||
- `s3_run_prefix`
|
||||
- run upload counts/paths
|
||||
- promoted upload counts/paths
|
||||
- skipped optional promotions
|
||||
- `current_manifest_key`
|
||||
- `current_run_id_key`
|
||||
- `current_pointer_written`
|
||||
- `audio_upload_skipped`
|
||||
@@ -1,84 +0,0 @@
|
||||
# S3 Audio Input
|
||||
|
||||
This document describes implemented S3 audio input behavior in `prepare`.
|
||||
|
||||
## Scope
|
||||
|
||||
Implemented:
|
||||
|
||||
- `prepare` can acquire source audio from S3 when `session.inputs.audio_s3.prefix` is configured.
|
||||
- object listing and download go through the storage backend abstraction.
|
||||
- tests use fake storage; no live S3 service is required for test runs.
|
||||
|
||||
Not implemented:
|
||||
|
||||
- uploads of failed runs
|
||||
|
||||
## Required Configuration
|
||||
|
||||
`pipeline.yml`:
|
||||
|
||||
- `storage.s3.bucket` must be set when S3 audio input is used.
|
||||
- `storage.s3.root_prefix` defaults to `dnd`.
|
||||
- `storage.s3.access_key_id_env` defaults to `OBJECT_STORAGE_KEY_ID`.
|
||||
- `storage.s3.secret_access_key_env` defaults to `OBJECT_STORAGE_KEY`.
|
||||
- `spool.root` defaults to `/var/spool/narratio`.
|
||||
|
||||
`session.yml`:
|
||||
|
||||
- configure `session.campaign` and `session.session_id`.
|
||||
- configure `session.inputs.audio_s3.prefix` for S3 audio input.
|
||||
- do not configure `inputs.audio_dir` or `inputs.audio_files` at the same time as `inputs.audio_s3`.
|
||||
|
||||
## Prefix Shape
|
||||
|
||||
Session S3 root:
|
||||
|
||||
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
|
||||
Audio prefix:
|
||||
|
||||
`{session_root}/{audio_s3.prefix}`
|
||||
|
||||
Example:
|
||||
|
||||
`dnd/campaigns/forsaken/sessions/2026-04-19/audio/`
|
||||
|
||||
Audio files must already exist in S3 before running Narratio.
|
||||
|
||||
## Prepare Behavior
|
||||
|
||||
When `inputs.audio_s3.prefix` is configured, `prepare`:
|
||||
|
||||
1. lists objects under the computed S3 audio prefix
|
||||
2. filters to `.flac` objects
|
||||
3. fails when no `.flac` objects are found
|
||||
4. downloads selected objects to spool audio:
|
||||
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
5. materializes audio into workdir audio:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
|
||||
6. records input provenance in the manifest (bucket, key, metadata, local paths, checksum)
|
||||
|
||||
Notes:
|
||||
|
||||
- `.flac` filtering is case-insensitive.
|
||||
- ETag is recorded as provider metadata only and is not treated as a checksum.
|
||||
|
||||
## Local Audio Development
|
||||
|
||||
Local audio workflows remain supported:
|
||||
|
||||
- `inputs.audio_dir`
|
||||
- `inputs.audio_files`
|
||||
|
||||
These options are mutually exclusive with `inputs.audio_s3`.
|
||||
|
||||
## Archive Boundary
|
||||
|
||||
Current archive behavior relevant to S3 audio input:
|
||||
|
||||
- successful runs are uploaded by archive under `runs/{run_id}/`
|
||||
- configured promotions are uploaded to session-level destinations
|
||||
- `current/manifest.json` and `current/run_id.txt` are published
|
||||
- local source audio is not re-uploaded by default
|
||||
- failed or incomplete runs are not uploaded
|
||||
380
docs/troubleshooting.md
Normal file
380
docs/troubleshooting.md
Normal file
@@ -0,0 +1,380 @@
|
||||
# Troubleshooting
|
||||
|
||||
## Purpose
|
||||
Canonical operator troubleshooting guide for recurring implemented Narratio failures.
|
||||
|
||||
## Config file discovery failure
|
||||
|
||||
Symptom:
|
||||
- `run`, `plan`, `resume`, `run-stage`, or `restore` fails with config/session not found.
|
||||
|
||||
Likely Cause:
|
||||
- `pipeline.yml`, `campaign.yml`, or `session.yml` is missing from system discovery paths.
|
||||
- a local working-directory config file was not passed explicitly.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml
|
||||
ls -l /usr/local/etc/narratio/campaign.yml /etc/narratio/campaign.yml
|
||||
ls -l /usr/local/etc/narratio/session.yml /etc/narratio/session.yml
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- pass explicit `--config`, `--campaign`, and `--session`.
|
||||
- or place files in documented discovery paths.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/cli.md](./cli.md)
|
||||
|
||||
## Session template rendering failure
|
||||
|
||||
Symptom:
|
||||
- load fails with unresolved placeholder or `session_id` mismatch.
|
||||
|
||||
Likely Cause:
|
||||
- templated `session.yml` used without `--session-id`.
|
||||
- rendered `session_id` differs from passed `--session-id`.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio plan --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session ./session.yml --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- pass `--session-id` when template placeholders are present.
|
||||
- ensure rendered `session_id` matches intended run session id.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## Strict YAML decode or validation failure
|
||||
|
||||
Symptom:
|
||||
- config load fails with unknown field or validation error.
|
||||
|
||||
Likely Cause:
|
||||
- typo/stale field name.
|
||||
- missing required fields or invalid constraints.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio plan --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- align fields/values to canonical config reference and examples.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [examples/](../examples/)
|
||||
|
||||
## `--artifacts` selection failure
|
||||
|
||||
Symptom:
|
||||
- `run`/`resume`/`run-stage` fails with invalid or unknown artifact selection.
|
||||
|
||||
Likely Cause:
|
||||
- `--artifacts` contains blank names or unknown artifact keys.
|
||||
- `pipeline.scriptorium.artifacts` missing while using `--artifacts`.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio run --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts player_handout
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- use configured artifact keys only.
|
||||
- ensure `pipeline.scriptorium.artifacts` is defined.
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## `run-stage --artifacts` on non-analyze stage
|
||||
|
||||
Symptom:
|
||||
- `run-stage` fails with `--artifacts is only supported for stage "analyze"`.
|
||||
|
||||
Likely Cause:
|
||||
- `--artifacts` was used with a non-`analyze` stage.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio run-stage --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts session_recap polish
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- use `--artifacts` only with `run-stage ... analyze`.
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
|
||||
## Configured artifact dependency/input validation failure
|
||||
|
||||
Symptom:
|
||||
- config validation fails for `depends_on`, `narratio.artifact.<name>` source, or artifact output path.
|
||||
|
||||
Likely Cause:
|
||||
- `narratio.artifact.<name>` source missing matching `depends_on` key.
|
||||
- dependency references unknown artifact key.
|
||||
- dependency self-reference or enabled dependency cycle.
|
||||
- artifact output path missing/invalid/outside `artifacts/` root.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio plan --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- ensure artifact-to-artifact inputs have explicit `depends_on` entries using artifact keys.
|
||||
- ensure referenced artifacts exist and define valid `output_path` values.
|
||||
- keep output paths relative and under `artifacts/`.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/internal/stage-analyze.md](./internal/stage-analyze.md)
|
||||
|
||||
## Required configured artifact input unavailable at analyze time
|
||||
|
||||
Symptom:
|
||||
- analyze fails because configured input source is unavailable.
|
||||
|
||||
Likely Cause:
|
||||
- required upstream configured artifact was not selected/executed this run.
|
||||
- non-executable dependency output file is missing or invalid on disk.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio status --manifest /path/to/manifest.json
|
||||
narratio run-stage --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts player_handout analyze
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- run analyze with needed artifacts selected.
|
||||
- or ensure dependency output file exists at configured path and is valid.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## Manifest/status path failure
|
||||
|
||||
Symptom:
|
||||
- `status` fails because manifest path is missing, unreadable, or invalid.
|
||||
|
||||
Likely Cause:
|
||||
- wrong manifest path.
|
||||
- manifest removed after cleanup.
|
||||
- `--manifest` omitted.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio status --manifest /path/to/manifest.json
|
||||
ls -l /path/to/manifest.json
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- use manifest path printed by `run`, `resume`, or `run-stage`.
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
|
||||
## Session lock conflict (`.lock`)
|
||||
|
||||
Symptom:
|
||||
- `run`, `resume`, `run-stage`, or `restore` fails with lock conflict for session workdir.
|
||||
|
||||
Likely Cause:
|
||||
- another Narratio process is running same session.
|
||||
- stale lock from interrupted prior run.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
ls -l {workspace.root}/work/{campaign}/{session_id}/.lock
|
||||
cat {workspace.root}/work/{campaign}/{session_id}/.lock
|
||||
ps aux | grep narratio
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- wait for active process to finish.
|
||||
- if no process is active, remove only stale session `.lock` file.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/internal/workspace.md](./internal/workspace.md)
|
||||
|
||||
## Restore remote current pointer or manifest missing
|
||||
|
||||
Symptom:
|
||||
- `restore` fails with remote current pointer or current manifest errors.
|
||||
|
||||
Likely Cause:
|
||||
- `current/run_id.txt` was never published.
|
||||
- `current/manifest.json` is missing for the session prefix.
|
||||
- archive commit did not complete.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio restore --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- verify archive stage succeeded for the target session.
|
||||
- rerun/archive from a healthy source workspace so current pointers are published.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
|
||||
|
||||
## Restore manifest identity mismatch
|
||||
|
||||
Symptom:
|
||||
- `restore` fails because remote manifest session or campaign does not match requested values.
|
||||
|
||||
Likely Cause:
|
||||
- wrong `--session-id` or wrong session config selected.
|
||||
- archive prefix points to a different campaign/session.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio restore --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- use the correct session config and `--session-id`.
|
||||
- verify campaign/session identity in local config before restore.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
|
||||
## Restore conflict without `--force`
|
||||
|
||||
Symptom:
|
||||
- `restore` fails with `restore conflict` and conflict counts.
|
||||
|
||||
Likely Cause:
|
||||
- local durable file differs from remote file for one or more planned restore paths.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio restore --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 --dry-run
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- review planned conflicts.
|
||||
- rerun with `--force` only when remote state should overwrite local state.
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
|
||||
## Restore report expectations
|
||||
|
||||
Symptom:
|
||||
- operator expects restore report file but does not find one.
|
||||
|
||||
Likely Cause:
|
||||
- restore was executed in `--dry-run` mode.
|
||||
- restore failed before report persistence path (for example lock acquisition failure).
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
ls -l {workspace.root}/work/{campaign}/{session_id}/reports/restore-latest.json
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- run non-dry-run restore for durable report output.
|
||||
- resolve lock or early preflight failures and retry.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
|
||||
## Secrets env-dir or credential-env failure
|
||||
|
||||
Symptom:
|
||||
- startup fails loading secrets directory, or stage fails due to missing credential env vars.
|
||||
|
||||
Likely Cause:
|
||||
- invalid `pipeline.secrets.env_dir` path/permissions.
|
||||
- required credential env var unset/empty.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
ls -la /path/to/secrets_dir
|
||||
env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM'
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- fix secrets directory and credential env vars.
|
||||
- keep secret values out of YAML.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## S3-audio prepare failure
|
||||
|
||||
Symptom:
|
||||
- `prepare` fails in S3 mode (listing/downloading/no audio/backend error).
|
||||
|
||||
Likely Cause:
|
||||
- wrong `session.inputs.audio_s3.prefix`.
|
||||
- no `.flac` files at resolved prefix.
|
||||
- invalid/missing object-store credentials or backend config.
|
||||
- mixed local+S3 audio input config.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio run-stage --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 prepare
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- configure exactly one audio source mode.
|
||||
- verify `.flac` files and storage access.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
|
||||
## Archive promotion/current-pointer failure
|
||||
|
||||
Symptom:
|
||||
- archive fails on required promotion source missing or pointer write failure.
|
||||
|
||||
Likely Cause:
|
||||
- required promoted file absent (including analyze outputs not generated for this run).
|
||||
- storage upload failed before `current/run_id.txt` commit marker write.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio status --manifest /path/to/manifest.json
|
||||
narratio run-stage --config /path/to/pipeline.yml --campaign /path/to/campaign.yml --session /path/to/session.yml --session-id 2026-04-04 archive
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- rerun or resume upstream stages to generate required files.
|
||||
- adjust promotion `source`/`dest` rules to match artifacts that must exist.
|
||||
- retry after storage issue is resolved.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
|
||||
5
examples/campaign.yml
Normal file
5
examples/campaign.yml
Normal file
@@ -0,0 +1,5 @@
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
@@ -1,26 +0,0 @@
|
||||
workspace:
|
||||
root: ./tmp/narratio-workspace
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
|
||||
seriatim:
|
||||
binary: "seriatim"
|
||||
timeout: "10m"
|
||||
output_schema: "seriatim-intermediate"
|
||||
coalesce_gap: 3.0
|
||||
|
||||
audita:
|
||||
binary: "audita"
|
||||
timeout: "3h"
|
||||
base_url: "https://openrouter.ai/api/v1"
|
||||
model: "openrouter/google/gemma-4-31b-it"
|
||||
llm_api_key_env: "AUDITA_LLM_API_KEY"
|
||||
modules: ["glossary", "homophones", "spoken_word", "grammar"]
|
||||
output_schema: "audita-v1"
|
||||
work_dir_retention: "auto"
|
||||
total_llm_concurrency: 2
|
||||
proposal_llm_concurrency: 1
|
||||
validation_model: "openrouter/google/gemma-4-31b-it"
|
||||
validation_llm_concurrency: 1
|
||||
report: true
|
||||
180
examples/pipeline.full.annotated.yml
Normal file
180
examples/pipeline.full.annotated.yml
Normal file
@@ -0,0 +1,180 @@
|
||||
# Full annotated pipeline example for implemented Narratio config fields.
|
||||
# Values are safe placeholders and must be adapted per environment.
|
||||
|
||||
workspace:
|
||||
# Optional: defaults to /var/lib/narratio.
|
||||
root: /var/lib/narratio/workspace
|
||||
# Optional: remove run-scoped workdir after successful archive commit.
|
||||
cleanup_after_archive: false
|
||||
|
||||
# Optional: local secret file loader (directory of ENV_VAR_NAME files).
|
||||
# secrets:
|
||||
# env_dir: ./secrets
|
||||
|
||||
storage:
|
||||
# Optional storage backend selector; use "s3" for archive + S3 audio workflows.
|
||||
backend: s3
|
||||
# Compatibility fields retained in schema.
|
||||
bucket: ""
|
||||
prefix: ""
|
||||
s3:
|
||||
# Required when using S3 audio or S3 archive uploads.
|
||||
bucket: my-dnd-archive
|
||||
# Optional; defaults to "dnd".
|
||||
root_prefix: dnd
|
||||
# Optional region/endpoint settings.
|
||||
region: us-east-1
|
||||
endpoint: ""
|
||||
force_path_style: false
|
||||
# Optional; defaults shown explicitly.
|
||||
access_key_id_env: OBJECT_STORAGE_KEY_ID
|
||||
secret_access_key_env: OBJECT_STORAGE_KEY
|
||||
|
||||
spool:
|
||||
# Optional; defaults to /var/spool/narratio.
|
||||
root: /var/spool/narratio
|
||||
# Optional cleanup of run-scoped spool audio after successful archive commit.
|
||||
delete_audio_after_archive: false
|
||||
|
||||
archive:
|
||||
# Optional booleans; defaults are true.
|
||||
enabled: true
|
||||
upload_run: true
|
||||
# Optional promotion rules; sources use Narratio artifact source IDs.
|
||||
promote_artifacts:
|
||||
- source: narratio.transcript.trimmed
|
||||
dest: transcripts/trimmed.json
|
||||
required: true
|
||||
- source: narratio.artifact.session_recap
|
||||
dest: artifacts/session_recap.md
|
||||
required: true
|
||||
- source: narratio.artifact.player_handout
|
||||
dest: artifacts/player_handout.md
|
||||
required: false
|
||||
|
||||
whisperx:
|
||||
# Required.
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
# Optional overrides; defaults shown explicitly.
|
||||
language: en
|
||||
timeout: 30m
|
||||
retries: 3
|
||||
retry_delay: 2s
|
||||
concurrency: 2
|
||||
|
||||
seriatim:
|
||||
# Optional overrides; defaults shown explicitly.
|
||||
binary: seriatim
|
||||
timeout: 10m
|
||||
output_schema: seriatim-intermediate
|
||||
coalesce_gap: 3.0
|
||||
report: true
|
||||
env:
|
||||
# Optional advanced tuning; set only when needed.
|
||||
overlap_word_run_gap: 1.0
|
||||
overlap_word_run_reorder_window: 1.0
|
||||
backchannel_max_duration: 2.0
|
||||
filler_max_duration: 1.25
|
||||
|
||||
audita:
|
||||
# Optional overrides; defaults shown explicitly where applicable.
|
||||
binary: audita
|
||||
timeout: 3h
|
||||
llm_api_key_env: AUDITA_LLM_API_KEY
|
||||
modules: [glossary, homophones, spoken_word, grammar]
|
||||
base_url: ""
|
||||
model: ""
|
||||
total_llm_concurrency: 2
|
||||
proposal_llm_concurrency: 1
|
||||
validation_model: ""
|
||||
validation_llm_concurrency: 1
|
||||
transcript_description: ""
|
||||
config_path: /usr/local/etc/audita/config.yml
|
||||
output_schema: audita-v1
|
||||
work_dir_retention: auto
|
||||
report: true
|
||||
|
||||
normalize:
|
||||
# Optional; defaults shown explicitly.
|
||||
output_path: transcripts/normalized.json
|
||||
output_schema: seriatim-intermediate
|
||||
report: true
|
||||
|
||||
trim:
|
||||
# Keep disabled unless bounds prompt integration is configured.
|
||||
enabled: false
|
||||
output_path: transcripts/trimmed.json
|
||||
bounds:
|
||||
prompt_id: dnd.session_bounds
|
||||
profile_id: local-fast
|
||||
transcript_input_name: transcript
|
||||
output_path: reports/session_bounds.json
|
||||
timeout: 10m
|
||||
render_debug: false
|
||||
render_output_path: reports/session_bounds.render.json
|
||||
seriatim:
|
||||
report: false
|
||||
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
config_path: /usr/local/etc/scriptorium/config.yml
|
||||
timeout: 10m
|
||||
render_debug: false
|
||||
artifacts:
|
||||
# Configured artifact keys map to source IDs narratio.artifact.<key>.
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
profile_id: local-fast
|
||||
output_path: artifacts/session_recap.md
|
||||
timeout: 10m
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
previous_recap:
|
||||
source: narratio.previous_session.artifact.session_recap
|
||||
required: false
|
||||
vars:
|
||||
session_id: true
|
||||
session_date: true
|
||||
campaign_name: true
|
||||
previous_session_id: true
|
||||
output_kind: session_recap
|
||||
|
||||
# Example dependent artifact:
|
||||
# - depends_on entries use artifact keys.
|
||||
# - narratio.artifact.<key> sources require matching depends_on membership.
|
||||
player_handout:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- session_recap
|
||||
prompt_id: dnd.player_handout
|
||||
profile_id: local-fast
|
||||
output_path: artifacts/player_handout.md
|
||||
timeout: 10m
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
vars:
|
||||
session_id: true
|
||||
campaign_name: true
|
||||
output_kind: player_handout
|
||||
|
||||
analyzer:
|
||||
# Optional adapter settings.
|
||||
binary_path: ""
|
||||
timeout: 2m
|
||||
artifacts:
|
||||
output_dir: ""
|
||||
types: []
|
||||
|
||||
notification:
|
||||
# Optional notification settings.
|
||||
backend: ""
|
||||
recipient: ""
|
||||
timeout: 30s
|
||||
@@ -1,55 +1,2 @@
|
||||
workspace:
|
||||
root: ./tmp/narratio-workspace
|
||||
cleanup_after_archive: false
|
||||
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: "my-dnd-archive"
|
||||
root_prefix: "dnd"
|
||||
region: "us-east-1"
|
||||
# Optional credential env-var names (defaulted when omitted):
|
||||
# access_key_id_env: "OBJECT_STORAGE_KEY_ID"
|
||||
# secret_access_key_env: "OBJECT_STORAGE_KEY"
|
||||
|
||||
spool:
|
||||
root: "/var/spool/narratio"
|
||||
delete_audio_after_archive: false
|
||||
|
||||
archive:
|
||||
enabled: true
|
||||
upload_run: true
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
|
||||
# Optional. When omitted entirely, Narratio defaults to seriatim binary + runtime defaults.
|
||||
seriatim: {}
|
||||
|
||||
# Optional runtime overrides. Model/provider can be owned by Audita runtime config.
|
||||
audita:
|
||||
config_path: "/usr/local/etc/audita/config.yml"
|
||||
llm_api_key_env: "AUDITA_LLM_API_KEY"
|
||||
|
||||
# Optional Scriptorium integration for analyze artifacts.
|
||||
scriptorium:
|
||||
config_path: "/usr/local/etc/scriptorium/config.yml"
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: "dnd.session_recap"
|
||||
output_path: "artifacts/session_recap.md"
|
||||
inputs:
|
||||
transcript:
|
||||
source: "trimmed_transcript"
|
||||
required: true
|
||||
previous_recap:
|
||||
source: "previous_session_artifact"
|
||||
artifact: "session_recap"
|
||||
required: false
|
||||
vars:
|
||||
session_id: true
|
||||
session_date: true
|
||||
campaign_name: true
|
||||
previous_session_id: true
|
||||
output_kind: "session_recap"
|
||||
|
||||
115
examples/pipeline.production.yml
Normal file
115
examples/pipeline.production.yml
Normal file
@@ -0,0 +1,115 @@
|
||||
workspace:
|
||||
root: /var/lib/narratio/workspace
|
||||
cleanup_after_archive: true
|
||||
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: my-dnd-archive
|
||||
root_prefix: dnd
|
||||
region: us-east-1
|
||||
access_key_id_env: OBJECT_STORAGE_KEY_ID
|
||||
secret_access_key_env: OBJECT_STORAGE_KEY
|
||||
|
||||
spool:
|
||||
root: /var/spool/narratio
|
||||
delete_audio_after_archive: true
|
||||
|
||||
archive:
|
||||
enabled: true
|
||||
upload_run: true
|
||||
promote_artifacts:
|
||||
- source: narratio.transcript.trimmed
|
||||
dest: transcripts/trimmed.json
|
||||
required: true
|
||||
- source: narratio.artifact.session_recap
|
||||
dest: artifacts/session_recap.md
|
||||
required: true
|
||||
- source: narratio.artifact.player_handout
|
||||
dest: artifacts/player_handout.md
|
||||
required: false
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
language: en
|
||||
timeout: 45m
|
||||
retries: 3
|
||||
retry_delay: 3s
|
||||
concurrency: 2
|
||||
|
||||
seriatim:
|
||||
binary: seriatim
|
||||
timeout: 10m
|
||||
output_schema: seriatim-intermediate
|
||||
coalesce_gap: 3.0
|
||||
report: true
|
||||
|
||||
audita:
|
||||
binary: audita
|
||||
timeout: 3h
|
||||
llm_api_key_env: AUDITA_LLM_API_KEY
|
||||
modules: [glossary, homophones, spoken_word, grammar]
|
||||
output_schema: audita-v1
|
||||
work_dir_retention: auto
|
||||
total_llm_concurrency: 2
|
||||
proposal_llm_concurrency: 1
|
||||
validation_llm_concurrency: 1
|
||||
report: true
|
||||
|
||||
normalize:
|
||||
output_path: transcripts/normalized.json
|
||||
output_schema: seriatim-intermediate
|
||||
report: true
|
||||
|
||||
trim:
|
||||
enabled: false
|
||||
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
config_path: /usr/local/etc/scriptorium/config.yml
|
||||
timeout: 10m
|
||||
render_debug: false
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
profile_id: local-fast
|
||||
output_path: artifacts/session_recap.md
|
||||
timeout: 10m
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
previous_recap:
|
||||
source: narratio.previous_session.artifact.session_recap
|
||||
required: false
|
||||
vars:
|
||||
session_id: true
|
||||
session_date: true
|
||||
campaign_name: true
|
||||
previous_session_id: true
|
||||
output_kind: session_recap
|
||||
player_handout:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- session_recap
|
||||
prompt_id: dnd.player_handout
|
||||
profile_id: local-fast
|
||||
output_path: artifacts/player_handout.md
|
||||
timeout: 10m
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
vars:
|
||||
session_id: true
|
||||
output_kind: player_handout
|
||||
|
||||
analyzer:
|
||||
timeout: 2m
|
||||
|
||||
notification:
|
||||
timeout: 30s
|
||||
5
examples/session.local-audio.yml
Normal file
5
examples/session.local-audio.yml
Normal file
@@ -0,0 +1,5 @@
|
||||
session_id: 2026-05-03
|
||||
date: 2026-05-03
|
||||
title: Sample Session
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
@@ -1,13 +0,0 @@
|
||||
session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
date: 2026-05-03
|
||||
title: Sample Session
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
# Optional S3 input alternative. Do not configure with audio_dir/audio_files.
|
||||
# Narratio prepare lists this prefix and downloads .flac files.
|
||||
# audio_s3:
|
||||
# prefix: "audio/"
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
6
examples/session.s3-audio.yml
Normal file
6
examples/session.s3-audio.yml
Normal file
@@ -0,0 +1,6 @@
|
||||
session_id: 2026-05-03
|
||||
date: 2026-05-03
|
||||
title: Sample Session
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
@@ -1,12 +1,3 @@
|
||||
session_id: "{{ session_id }}"
|
||||
campaign: sample-campaign
|
||||
date: ""
|
||||
title: ""
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
# Optional S3 input alternative. Do not configure with audio_dir/audio_files.
|
||||
# audio_s3:
|
||||
# prefix: "audio/{{ session_id }}/"
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
|
||||
2
go.mod
2
go.mod
@@ -4,6 +4,7 @@ go 1.25.0
|
||||
|
||||
require (
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.17
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.16
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0
|
||||
github.com/aws/smithy-go v1.25.1
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
@@ -12,7 +13,6 @@ require (
|
||||
require (
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.7 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.16 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.23 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 // indirect
|
||||
|
||||
@@ -27,8 +27,9 @@ type FakeBackend struct {
|
||||
Err error
|
||||
Result ArchiveResult
|
||||
|
||||
Objects map[string]FakeObject
|
||||
Uploads []FakeUploadCall
|
||||
Objects map[string]FakeObject
|
||||
Uploads []FakeUploadCall
|
||||
Downloads []FakeDownloadCall
|
||||
|
||||
ListErr error
|
||||
DownloadErr error
|
||||
@@ -43,6 +44,12 @@ type FakeUploadCall struct {
|
||||
Options UploadOptions
|
||||
}
|
||||
|
||||
// FakeDownloadCall captures one download invocation in call order.
|
||||
type FakeDownloadCall struct {
|
||||
Key string
|
||||
LocalPath string
|
||||
}
|
||||
|
||||
// Archive records request and returns configured response.
|
||||
func (f *FakeBackend) Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
@@ -130,6 +137,10 @@ func (f *FakeBackend) Download(ctx context.Context, key, localPath string) error
|
||||
if !ok {
|
||||
return fmt.Errorf("download object %q: %w", key, os.ErrNotExist)
|
||||
}
|
||||
f.Downloads = append(f.Downloads, FakeDownloadCall{
|
||||
Key: normalizeObjectKey(key),
|
||||
LocalPath: localPath,
|
||||
})
|
||||
|
||||
if err := os.MkdirAll(filepath.Dir(localPath), 0o755); err != nil {
|
||||
return fmt.Errorf("download object %q: create parent directory: %w", key, err)
|
||||
|
||||
@@ -56,7 +56,7 @@ func TestFakeBackendDownload(t *testing.T) {
|
||||
fake.SeedObject(FakeObject{Key: "audio/a.flac", Data: []byte("audio-a")})
|
||||
|
||||
dst := filepath.Join(t.TempDir(), "nested", "a.flac")
|
||||
if err := fake.Download(context.Background(), "audio/a.flac", dst); err != nil {
|
||||
if err := fake.Download(context.Background(), `audio\a.flac`, dst); err != nil {
|
||||
t.Fatalf("Download() error = %v", err)
|
||||
}
|
||||
data, err := os.ReadFile(dst)
|
||||
|
||||
66
internal/app/analyze_artifacts.go
Normal file
66
internal/app/analyze_artifacts.go
Normal file
@@ -0,0 +1,66 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
type artifactSelectionFlag struct {
|
||||
values []string
|
||||
}
|
||||
|
||||
func (f *artifactSelectionFlag) String() string {
|
||||
return strings.Join(f.values, ",")
|
||||
}
|
||||
|
||||
func (f *artifactSelectionFlag) Set(value string) error {
|
||||
f.values = append(f.values, value)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *artifactSelectionFlag) Normalize() ([]string, error) {
|
||||
if len(f.values) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
seen := map[string]struct{}{}
|
||||
out := make([]string, 0, len(f.values))
|
||||
for _, raw := range f.values {
|
||||
for _, part := range strings.Split(raw, ",") {
|
||||
name := strings.TrimSpace(part)
|
||||
if name == "" {
|
||||
return nil, fmt.Errorf("artifact names must be non-empty")
|
||||
}
|
||||
if _, ok := seen[name]; ok {
|
||||
continue
|
||||
}
|
||||
seen[name] = struct{}{}
|
||||
out = append(out, name)
|
||||
}
|
||||
}
|
||||
|
||||
sort.Strings(out)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func validateSelectedAnalyzeArtifacts(cfg *config.Config, selected []string) error {
|
||||
if len(selected) == 0 {
|
||||
return nil
|
||||
}
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Scriptorium == nil {
|
||||
return fmt.Errorf("--artifacts requires pipeline.scriptorium.artifacts to be configured")
|
||||
}
|
||||
configured := cfg.Pipeline.Scriptorium.Artifacts
|
||||
if len(configured) == 0 {
|
||||
return fmt.Errorf("--artifacts requires at least one configured artifact in pipeline.scriptorium.artifacts")
|
||||
}
|
||||
for _, name := range selected {
|
||||
if _, ok := configured[name]; !ok {
|
||||
return fmt.Errorf("--artifacts includes unknown artifact %q", name)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
286
internal/app/analyze_artifacts_commands_test.go
Normal file
286
internal/app/analyze_artifacts_commands_test.go
Normal file
@@ -0,0 +1,286 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||
)
|
||||
|
||||
func TestExecuteRunStageArtifactsNonAnalyzeFails(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{"run-stage", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--artifacts", "session_recap", "polish"},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `run-stage: --artifacts is only supported for stage "analyze"`) {
|
||||
t.Fatalf("stderr = %q, want stage-gating error", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteUnknownArtifactsFailValidation(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{"run", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--artifacts", "unknown_artifact"},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `run: --artifacts includes unknown artifact "unknown_artifact"`) {
|
||||
t.Fatalf("stderr = %q, want unknown-artifact validation error", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunStageArtifactsDoesNotImplyForce(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
seed.MarkStageSucceeded("analyze", time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||
t.Fatalf("save manifest: %v", err)
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
err := RunStage(
|
||||
context.Background(),
|
||||
[]string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--artifacts", "session_recap,session_recap", "analyze"},
|
||||
&out,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("RunStage() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "stage=analyze executed=0 skipped=1 force=false") {
|
||||
t.Fatalf("output = %q, want analyze skip without force", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestResumeArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
|
||||
seed.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||
t.Fatalf("save manifest: %v", err)
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
err := Resume(
|
||||
context.Background(),
|
||||
[]string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
||||
&out,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("Resume() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "has no remaining stages") {
|
||||
t.Fatalf("output = %q, want no remaining stages", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteAnalyzeForceRunsAnalyze(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
var capturedStages []string
|
||||
var capturedForce bool
|
||||
origExecuteStagesFn := executeStagesFn
|
||||
t.Cleanup(func() {
|
||||
executeStagesFn = origExecuteStagesFn
|
||||
})
|
||||
executeStagesFn = func(_ context.Context, _ *config.Config, stages []stage.Stage, opts RunOptions) (*RunSummary, error) {
|
||||
for _, s := range stages {
|
||||
capturedStages = append(capturedStages, s.Name())
|
||||
}
|
||||
capturedForce = opts.Force
|
||||
return &RunSummary{
|
||||
ManifestPath: filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json"),
|
||||
Executed: []string{"analyze"},
|
||||
}, nil
|
||||
}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{"analyze", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if len(capturedStages) != 1 || capturedStages[0] != "analyze" {
|
||||
t.Fatalf("captured stages = %#v, want [analyze]", capturedStages)
|
||||
}
|
||||
if !capturedForce {
|
||||
t.Fatal("captured force = false, want true")
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "narratio analyze: executed=1 skipped=0 force=true; manifest=") {
|
||||
t.Fatalf("stdout = %q, want analyze summary", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteAnalyzePropagatesSelectedArtifacts(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
var capturedArtifacts []string
|
||||
origExecuteStagesFn := executeStagesFn
|
||||
t.Cleanup(func() {
|
||||
executeStagesFn = origExecuteStagesFn
|
||||
})
|
||||
executeStagesFn = func(_ context.Context, _ *config.Config, _ []stage.Stage, opts RunOptions) (*RunSummary, error) {
|
||||
capturedArtifacts = append([]string(nil), opts.SelectedArtifacts...)
|
||||
return &RunSummary{ManifestPath: filepath.Join(workspaceRoot, "manifest.json"), Executed: []string{"analyze"}}, nil
|
||||
}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{
|
||||
"analyze",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--artifacts", "player_handout,session_recap",
|
||||
},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if strings.Join(capturedArtifacts, ",") != "player_handout,session_recap" {
|
||||
t.Fatalf("captured artifacts = %#v, want sorted selected artifacts", capturedArtifacts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteAnalyzeUnknownArtifactFailsValidation(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{"analyze", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--artifacts", "unknown_artifact"},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `analyze: --artifacts includes unknown artifact "unknown_artifact"`) {
|
||||
t.Fatalf("stderr = %q, want unknown-artifact validation error", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteAnalyzeRejectsPositionalArgsAndForceFlag(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
args []string
|
||||
want string
|
||||
}{
|
||||
{name: "positional", args: []string{"analyze", "extra"}, want: "analyze: unexpected positional arguments"},
|
||||
{name: "force flag", args: []string{"analyze", "--force"}, want: "analyze: invalid flags: flag provided but not defined: -force"},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(tc.args, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), tc.want) {
|
||||
t.Fatalf("stderr = %q, want %q", stderr.String(), tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteAnalyzeMissingConfigUsesRunStageLoadingPath(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"analyze"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "analyze: no pipeline config path provided and no default pipeline config found; searched:") {
|
||||
t.Fatalf("stderr = %q, want pipeline discovery error", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteUsageIncludesAnalyze(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(nil, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "analyze") {
|
||||
t.Fatalf("stderr = %q, want usage to include analyze", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func writeValidConfigFilesWithScriptoriumArtifacts(t *testing.T, workspaceRoot string) (string, string, string) {
|
||||
t.Helper()
|
||||
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
f, err := os.OpenFile(pipelinePath, os.O_APPEND|os.O_WRONLY, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("open pipeline config for append: %v", err)
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
extra := `
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
player_handout:
|
||||
enabled: true
|
||||
prompt_id: dnd.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
depends_on:
|
||||
- session_recap
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
`
|
||||
if _, err := f.WriteString(extra); err != nil {
|
||||
t.Fatalf("append scriptorium config: %v", err)
|
||||
}
|
||||
return pipelinePath, campaignPath, sessionPath
|
||||
}
|
||||
132
internal/app/analyze_artifacts_test.go
Normal file
132
internal/app/analyze_artifacts_test.go
Normal file
@@ -0,0 +1,132 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func TestArtifactSelectionFlagNormalize(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
inputs []string
|
||||
want []string
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "single value",
|
||||
inputs: []string{"session_recap"},
|
||||
want: []string{"session_recap"},
|
||||
},
|
||||
{
|
||||
name: "repeatable and comma separated values are deduped and sorted",
|
||||
inputs: []string{"session_recap,player_handout", "session_recap"},
|
||||
want: []string{"player_handout", "session_recap"},
|
||||
},
|
||||
{
|
||||
name: "empty token fails",
|
||||
inputs: []string{"session_recap,"},
|
||||
wantErr: "artifact names must be non-empty",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
var flag artifactSelectionFlag
|
||||
for _, in := range tt.inputs {
|
||||
if err := flag.Set(in); err != nil {
|
||||
t.Fatalf("Set(%q) error = %v", in, err)
|
||||
}
|
||||
}
|
||||
|
||||
got, err := flag.Normalize()
|
||||
if tt.wantErr != "" {
|
||||
if err == nil {
|
||||
t.Fatalf("Normalize() error = nil, want %q", tt.wantErr)
|
||||
}
|
||||
if err.Error() != tt.wantErr {
|
||||
t.Fatalf("Normalize() error = %q, want %q", err.Error(), tt.wantErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("Normalize() error = %v", err)
|
||||
}
|
||||
if len(got) != len(tt.want) {
|
||||
t.Fatalf("Normalize() len = %d, want %d; got=%v", len(got), len(tt.want), got)
|
||||
}
|
||||
for i := range got {
|
||||
if got[i] != tt.want[i] {
|
||||
t.Fatalf("Normalize()[%d] = %q, want %q", i, got[i], tt.want[i])
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateSelectedAnalyzeArtifacts(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
cfg *config.Config
|
||||
selected []string
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "empty selection is accepted",
|
||||
cfg: &config.Config{},
|
||||
selected: nil,
|
||||
},
|
||||
{
|
||||
name: "scriptorium required when selected artifacts present",
|
||||
cfg: &config.Config{Pipeline: &config.PipelineConfig{}},
|
||||
selected: []string{"session_recap"},
|
||||
wantErr: "--artifacts requires pipeline.scriptorium.artifacts to be configured",
|
||||
},
|
||||
{
|
||||
name: "unknown selected artifact fails",
|
||||
cfg: &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Scriptorium: &config.ScriptoriumConfig{
|
||||
Artifacts: map[string]config.ScriptoriumArtifactConfig{
|
||||
"session_recap": {Enabled: true, PromptID: "dnd.session_recap", OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
selected: []string{"player_handout"},
|
||||
wantErr: `--artifacts includes unknown artifact "player_handout"`,
|
||||
},
|
||||
{
|
||||
name: "known selected artifacts are accepted",
|
||||
cfg: &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Scriptorium: &config.ScriptoriumConfig{
|
||||
Artifacts: map[string]config.ScriptoriumArtifactConfig{
|
||||
"session_recap": {Enabled: true, PromptID: "dnd.session_recap", OutputPath: "artifacts/session_recap.md"},
|
||||
"player_handout": {Enabled: true, PromptID: "dnd.player_handout", OutputPath: "artifacts/player_handout.md"},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
selected: []string{"player_handout", "session_recap"},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := validateSelectedAnalyzeArtifacts(tt.cfg, tt.selected)
|
||||
if tt.wantErr != "" {
|
||||
if err == nil {
|
||||
t.Fatalf("error = nil, want %q", tt.wantErr)
|
||||
}
|
||||
if err.Error() != tt.wantErr {
|
||||
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("error = %v, want nil", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
49
internal/app/campaign_config_path.go
Normal file
49
internal/app/campaign_config_path.go
Normal file
@@ -0,0 +1,49 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func resolveCampaignConfigPath(flagValue string) (string, error) {
|
||||
return resolveCampaignConfigPathWithCandidates(flagValue, config.DefaultCampaignConfigSearchPaths)
|
||||
}
|
||||
|
||||
func resolveCampaignConfigPathWithCandidates(flagValue string, candidates []string) (string, error) {
|
||||
if explicit := strings.TrimSpace(flagValue); explicit != "" {
|
||||
return explicit, nil
|
||||
}
|
||||
|
||||
ordered := make([]string, 0, len(candidates))
|
||||
for _, raw := range candidates {
|
||||
path := strings.TrimSpace(raw)
|
||||
if path == "" {
|
||||
continue
|
||||
}
|
||||
ordered = append(ordered, path)
|
||||
info, err := os.Stat(path)
|
||||
if err == nil {
|
||||
if info.IsDir() {
|
||||
continue
|
||||
}
|
||||
return filepath.Clean(path), nil
|
||||
}
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
continue
|
||||
}
|
||||
return "", fmt.Errorf("check default campaign config %q: %w", path, err)
|
||||
}
|
||||
|
||||
if len(ordered) == 0 {
|
||||
return "", fmt.Errorf("no campaign config path provided and no default locations configured")
|
||||
}
|
||||
return "", fmt.Errorf(
|
||||
"no campaign config path provided and no default campaign config found; searched: %s; pass --campaign to use an explicit path",
|
||||
strings.Join(ordered, ", "),
|
||||
)
|
||||
}
|
||||
49
internal/app/campaign_config_path_test.go
Normal file
49
internal/app/campaign_config_path_test.go
Normal file
@@ -0,0 +1,49 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestResolveCampaignConfigPathExplicitWins(t *testing.T) {
|
||||
explicit := filepath.Join(t.TempDir(), "custom-campaign.yml")
|
||||
got, err := resolveCampaignConfigPathWithCandidates(explicit, []string{filepath.Join(t.TempDir(), "campaign.yml")})
|
||||
if err != nil {
|
||||
t.Fatalf("resolveCampaignConfigPathWithCandidates() error = %v", err)
|
||||
}
|
||||
if got != explicit {
|
||||
t.Fatalf("path = %q, want explicit path %q", got, explicit)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveCampaignConfigPathUsesFirstExistingDefault(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
missing := filepath.Join(dir, "missing.yml")
|
||||
found := filepath.Join(dir, "campaign.yml")
|
||||
if err := os.WriteFile(found, []byte("campaign: sample-campaign\n"), 0o644); err != nil {
|
||||
t.Fatalf("write campaign.yml: %v", err)
|
||||
}
|
||||
|
||||
got, err := resolveCampaignConfigPathWithCandidates("", []string{missing, found})
|
||||
if err != nil {
|
||||
t.Fatalf("resolveCampaignConfigPathWithCandidates() error = %v", err)
|
||||
}
|
||||
if got != filepath.Clean(found) {
|
||||
t.Fatalf("path = %q, want %q", got, filepath.Clean(found))
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveCampaignConfigPathErrorIncludesSearchedPaths(t *testing.T) {
|
||||
_, err := resolveCampaignConfigPathWithCandidates("", []string{"/usr/local/etc/narratio/campaign.yml", "/etc/narratio/campaign.yml"})
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "searched") {
|
||||
t.Fatalf("error = %q, want searched paths", err.Error())
|
||||
}
|
||||
if !strings.Contains(err.Error(), "pass --campaign") {
|
||||
t.Fatalf("error = %q, want explicit-campaign guidance", err.Error())
|
||||
}
|
||||
}
|
||||
336
internal/app/clean.go
Normal file
336
internal/app/clean.go
Normal file
@@ -0,0 +1,336 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
// Clean removes local workspace/spool state while preserving durable cache
|
||||
// state unless cache cleanup is explicitly requested.
|
||||
func Clean(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("clean", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var flags commonConfigFlags
|
||||
var all bool
|
||||
var dryRun bool
|
||||
var clearCache bool
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
fs.BoolVar(&all, "all", false, "clean all local session work/spool state")
|
||||
fs.BoolVar(&dryRun, "dry-run", false, "print cleanup targets without deleting")
|
||||
fs.BoolVar(&clearCache, "clear-cache", false, "also clear durable S3 audio cache entries")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("clean: invalid flags: %w", err)
|
||||
}
|
||||
if fs.NArg() != 0 {
|
||||
return fmt.Errorf("clean: unexpected positional arguments")
|
||||
}
|
||||
if all {
|
||||
return cleanAllLocal(flags, dryRun, clearCache, out)
|
||||
}
|
||||
return cleanSession(ctx, flags, dryRun, clearCache, out)
|
||||
}
|
||||
|
||||
func cleanSession(ctx context.Context, flags commonConfigFlags, dryRun, clearCache bool, out io.Writer) error {
|
||||
if strings.TrimSpace(flags.sessionID) == "" {
|
||||
return fmt.Errorf("clean: --session-id is required unless --all is set")
|
||||
}
|
||||
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.sessionPath, flags.sessionOptions())
|
||||
if err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
||||
return fmt.Errorf("clean: resolved pipeline and session config are required")
|
||||
}
|
||||
campaign := strings.TrimSpace(cfg.Session.Campaign)
|
||||
sessionID := strings.TrimSpace(cfg.Session.SessionID)
|
||||
if campaign == "" || sessionID == "" {
|
||||
return fmt.Errorf("clean: campaign and session_id are required")
|
||||
}
|
||||
|
||||
if dryRun {
|
||||
fmt.Fprintf(out, "Clean plan for %s/%s\n", campaign, sessionID)
|
||||
} else {
|
||||
fmt.Fprintf(out, "Cleaned %s/%s\n", campaign, sessionID)
|
||||
}
|
||||
|
||||
workDir := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, campaign, sessionID)
|
||||
spoolDir := artifacts.SessionSpoolDir(cfg.Pipeline.Spool.Root, campaign, sessionID)
|
||||
if err := reportCleanScopedDir(out, cfg.Pipeline.Workspace.Root, workDir, "clean.workspace.session", dryRun); err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
if err := reportCleanScopedDir(out, cfg.Pipeline.Spool.Root, spoolDir, "clean.spool.session", dryRun); err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
|
||||
if clearCache {
|
||||
if err := cleanSessionAudioCache(ctx, cfg, dryRun, out); err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
} else {
|
||||
fmt.Fprintln(out, "Cache: preserved")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func cleanAllLocal(flags commonConfigFlags, dryRun, clearCache bool, out io.Writer) error {
|
||||
if strings.TrimSpace(flags.campaignPath) != "" ||
|
||||
strings.TrimSpace(flags.sessionPath) != "" ||
|
||||
strings.TrimSpace(flags.sessionID) != "" ||
|
||||
strings.TrimSpace(flags.previousSessionID) != "" {
|
||||
return fmt.Errorf("clean: --all cannot be combined with --campaign, --session, --session-id, or --previous-session-id")
|
||||
}
|
||||
resolvedPipelinePath, err := resolvePipelineConfigPath(flags.pipelinePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
pipelineCfg, err := config.LoadPipeline(resolvedPipelinePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
|
||||
if dryRun {
|
||||
fmt.Fprintln(out, "Clean plan for all local sessions")
|
||||
} else {
|
||||
fmt.Fprintln(out, "Cleaned all local sessions")
|
||||
}
|
||||
|
||||
workRoot := filepath.Join(pipelineCfg.Workspace.Root, config.PathWorkDirSegment)
|
||||
if err := reportCleanScopedDir(out, pipelineCfg.Workspace.Root, workRoot, "clean.workspace.all", dryRun); err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
if err := reportCleanRootChildren(out, pipelineCfg.Spool.Root, "clean.spool.all", dryRun); err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
|
||||
if clearCache {
|
||||
if err := cleanAllAudioCache(pipelineCfg, dryRun, out); err != nil {
|
||||
return fmt.Errorf("clean: %w", err)
|
||||
}
|
||||
} else {
|
||||
fmt.Fprintln(out, "Cache: preserved")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func reportCleanScopedDir(out io.Writer, root, target, policy string, dryRun bool) error {
|
||||
dir, err := validateScopedDir(root, target, policy)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if dryRun {
|
||||
if dir.Exists {
|
||||
fmt.Fprintf(out, "Would delete: %s\n", dir.TargetAbs)
|
||||
} else {
|
||||
fmt.Fprintf(out, "Would skip missing: %s\n", dir.TargetAbs)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if !dir.Exists {
|
||||
fmt.Fprintf(out, "Missing: %s\n", dir.TargetAbs)
|
||||
return nil
|
||||
}
|
||||
if err := os.RemoveAll(dir.TargetAbs); err != nil {
|
||||
return fmt.Errorf("cleanup policy %s: remove %q: %w", policy, dir.TargetAbs, err)
|
||||
}
|
||||
fmt.Fprintf(out, "Deleted: %s\n", dir.TargetAbs)
|
||||
return nil
|
||||
}
|
||||
|
||||
func reportCleanRootChildren(out io.Writer, root, policy string, dryRun bool) error {
|
||||
rootAbs, entries, err := cleanableRootChildren(root, policy)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(entries) == 0 {
|
||||
if dryRun {
|
||||
fmt.Fprintf(out, "Would skip empty: %s\n", rootAbs)
|
||||
} else {
|
||||
fmt.Fprintf(out, "Empty: %s\n", rootAbs)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
for _, entry := range entries {
|
||||
if dryRun {
|
||||
fmt.Fprintf(out, "Would delete: %s\n", entry)
|
||||
continue
|
||||
}
|
||||
if err := os.RemoveAll(entry); err != nil {
|
||||
return fmt.Errorf("cleanup policy %s: remove %q: %w", policy, entry, err)
|
||||
}
|
||||
fmt.Fprintf(out, "Deleted: %s\n", entry)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func cleanableRootChildren(root, policy string) (string, []string, error) {
|
||||
cleanRoot := strings.TrimSpace(root)
|
||||
if cleanRoot == "" {
|
||||
return "", nil, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
||||
}
|
||||
rootAbs, err := filepath.Abs(cleanRoot)
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
||||
}
|
||||
info, err := os.Lstat(rootAbs)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return rootAbs, nil, nil
|
||||
}
|
||||
return "", nil, fmt.Errorf("cleanup policy %s: stat root %q: %w", policy, rootAbs, err)
|
||||
}
|
||||
if info.Mode()&os.ModeSymlink != 0 {
|
||||
return "", nil, fmt.Errorf("cleanup policy %s: refusing to clean symlink root %q", policy, rootAbs)
|
||||
}
|
||||
if !info.IsDir() {
|
||||
return "", nil, fmt.Errorf("cleanup policy %s: root %q is not a directory", policy, rootAbs)
|
||||
}
|
||||
entries, err := os.ReadDir(rootAbs)
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("cleanup policy %s: read root %q: %w", policy, rootAbs, err)
|
||||
}
|
||||
out := make([]string, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
path := filepath.Join(rootAbs, entry.Name())
|
||||
info, err := os.Lstat(path)
|
||||
if err != nil {
|
||||
return "", nil, fmt.Errorf("cleanup policy %s: stat child %q: %w", policy, path, err)
|
||||
}
|
||||
if info.Mode()&os.ModeSymlink != 0 {
|
||||
return "", nil, fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, path)
|
||||
}
|
||||
out = append(out, path)
|
||||
}
|
||||
return rootAbs, out, nil
|
||||
}
|
||||
|
||||
func cleanSessionAudioCache(ctx context.Context, cfg *config.Config, dryRun bool, out io.Writer) error {
|
||||
if cfg.Session.Inputs.AudioS3 == nil {
|
||||
fmt.Fprintln(out, "Cache: skipped (session does not use audio_s3)")
|
||||
return nil
|
||||
}
|
||||
if cfg.Pipeline.Storage.S3 == nil || strings.TrimSpace(cfg.Pipeline.Storage.S3.Bucket) == "" {
|
||||
return fmt.Errorf("clear cache requires pipeline.storage.s3.bucket")
|
||||
}
|
||||
store, err := newCommandObjectStore(ctx, cfg, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("initialize object store for cache cleanup: %w", err)
|
||||
}
|
||||
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
audioPrefix := artifacts.S3AudioPrefix(sessionPrefix, cfg.Session.Inputs.AudioS3.Prefix)
|
||||
objects, err := store.List(ctx, audioPrefix)
|
||||
if err != nil {
|
||||
return fmt.Errorf("list s3 audio objects under %q: %w", audioPrefix, err)
|
||||
}
|
||||
count := 0
|
||||
for _, obj := range objects {
|
||||
key := strings.TrimSpace(obj.Key)
|
||||
if key == "" || strings.HasSuffix(key, "/") || !cleanIsFlac(key) {
|
||||
continue
|
||||
}
|
||||
cachePath, err := artifacts.S3AudioCachePath(cfg.Pipeline.Cache.Root, cfg.Pipeline.Storage.S3.Bucket, key)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
deleted, err := reportCleanScopedFile(out, cfg.Pipeline.Cache.Root, cachePath, "clean.cache.session", dryRun)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if deleted {
|
||||
count++
|
||||
}
|
||||
}
|
||||
if count == 0 {
|
||||
fmt.Fprintf(out, "Cache: no cached S3 audio files found for %s\n", audioPrefix)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func cleanAllAudioCache(cfg *config.PipelineConfig, dryRun bool, out io.Writer) error {
|
||||
if cfg.Storage.S3 == nil || strings.TrimSpace(cfg.Storage.S3.Bucket) == "" {
|
||||
return fmt.Errorf("clear cache requires pipeline.storage.s3.bucket")
|
||||
}
|
||||
namespaceDir, err := artifacts.S3AudioCacheNamespaceDir(cfg.Cache.Root, cfg.Storage.S3.Bucket, cfg.Storage.S3.RootPrefix)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return reportCleanScopedDir(out, cfg.Cache.Root, namespaceDir, "clean.cache.all", dryRun)
|
||||
}
|
||||
|
||||
func reportCleanScopedFile(out io.Writer, root, target, policy string, dryRun bool) (bool, error) {
|
||||
file, err := validateScopedFile(root, target, policy)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if dryRun {
|
||||
if file.Exists {
|
||||
fmt.Fprintf(out, "Would delete cache file: %s\n", file.TargetAbs)
|
||||
return true, nil
|
||||
}
|
||||
fmt.Fprintf(out, "Would skip missing cache file: %s\n", file.TargetAbs)
|
||||
return false, nil
|
||||
}
|
||||
if !file.Exists {
|
||||
fmt.Fprintf(out, "Missing cache file: %s\n", file.TargetAbs)
|
||||
return false, nil
|
||||
}
|
||||
if err := os.Remove(file.TargetAbs); err != nil {
|
||||
return false, fmt.Errorf("cleanup policy %s: remove %q: %w", policy, file.TargetAbs, err)
|
||||
}
|
||||
fmt.Fprintf(out, "Deleted cache file: %s\n", file.TargetAbs)
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func validateScopedFile(root, target, policy string) (scopedDir, error) {
|
||||
cleanRoot := strings.TrimSpace(root)
|
||||
cleanTarget := strings.TrimSpace(target)
|
||||
if cleanRoot == "" {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
||||
}
|
||||
if cleanTarget == "" {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target path is required", policy)
|
||||
}
|
||||
rootAbs, err := filepath.Abs(cleanRoot)
|
||||
if err != nil {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
||||
}
|
||||
targetAbs, err := filepath.Abs(cleanTarget)
|
||||
if err != nil {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve target %q: %w", policy, cleanTarget, err)
|
||||
}
|
||||
rel, err := filepath.Rel(rootAbs, targetAbs)
|
||||
if err != nil {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: relative path from %q to %q: %w", policy, rootAbs, targetAbs, err)
|
||||
}
|
||||
if rel == "." {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete root directory %q", policy, rootAbs)
|
||||
}
|
||||
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete path outside root: root=%q target=%q", policy, rootAbs, targetAbs)
|
||||
}
|
||||
info, err := os.Lstat(targetAbs)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: false}, nil
|
||||
}
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: stat target %q: %w", policy, targetAbs, err)
|
||||
}
|
||||
if info.Mode()&os.ModeSymlink != 0 {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, targetAbs)
|
||||
}
|
||||
if info.IsDir() {
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is a directory", policy, targetAbs)
|
||||
}
|
||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: true}, nil
|
||||
}
|
||||
|
||||
func cleanIsFlac(path string) bool {
|
||||
return strings.EqualFold(filepath.Ext(path), ".flac")
|
||||
}
|
||||
255
internal/app/clean_test.go
Normal file
255
internal/app/clean_test.go
Normal file
@@ -0,0 +1,255 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
)
|
||||
|
||||
func TestExecuteCleanSessionDeletesWorkAndSpoolButPreservesCache(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
workDir := artifacts.SessionWorkDirForCampaign(workspaceRoot, "sample-campaign", "2026-05-03")
|
||||
spoolDir := artifacts.SessionSpoolDir(filepath.Join(workspaceRoot, "spool"), "sample-campaign", "2026-05-03")
|
||||
cachePath, err := artifacts.S3AudioCachePath(filepath.Join(workspaceRoot, "cache"), "test-bucket", "dnd/campaigns/sample-campaign/sessions/2026-05-03/audio/alice.flac")
|
||||
if err != nil {
|
||||
t.Fatalf("S3AudioCachePath() error = %v", err)
|
||||
}
|
||||
mustWriteTestFile(t, filepath.Join(workDir, "manifest.json"), "{}")
|
||||
mustWriteTestFile(t, filepath.Join(spoolDir, "run-1", "audio", "alice.flac"), "audio")
|
||||
mustWriteTestFile(t, cachePath, "cached-audio")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--session-id", "2026-05-03"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
cleanAssertMissing(t, workDir)
|
||||
cleanAssertMissing(t, spoolDir)
|
||||
cleanAssertExists(t, cachePath)
|
||||
if !strings.Contains(stdout.String(), "Cache: preserved") {
|
||||
t.Fatalf("stdout = %q, want cache preserved", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteCleanSessionDryRunDeletesNothing(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
workDir := artifacts.SessionWorkDirForCampaign(workspaceRoot, "sample-campaign", "2026-05-03")
|
||||
spoolDir := artifacts.SessionSpoolDir(filepath.Join(workspaceRoot, "spool"), "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workDir, "manifest.json"), "{}")
|
||||
mustWriteTestFile(t, filepath.Join(spoolDir, "run-1", "audio", "alice.flac"), "audio")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--session-id", "2026-05-03", "--dry-run"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
cleanAssertExists(t, workDir)
|
||||
cleanAssertExists(t, spoolDir)
|
||||
if !strings.Contains(stdout.String(), "Would delete:") {
|
||||
t.Fatalf("stdout = %q, want dry-run delete plan", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteCleanMissingSessionPathsSucceeds(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--session-id", "2026-05-03"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "Missing:") {
|
||||
t.Fatalf("stdout = %q, want missing path output", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteCleanSessionClearCacheRemovesOnlyS3AudioCache(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
if err := os.WriteFile(sessionPath, []byte(`session_id: 2026-05-03
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
`), 0o644); err != nil {
|
||||
t.Fatalf("write session: %v", err)
|
||||
}
|
||||
|
||||
audioKey := "dnd/campaigns/sample-campaign/sessions/2026-05-03/audio/alice.flac"
|
||||
fake := &storage.FakeBackend{}
|
||||
fake.SeedObject(storage.FakeObject{Key: audioKey, Data: []byte("audio")})
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||
|
||||
cacheRoot := filepath.Join(workspaceRoot, "cache")
|
||||
cachePath, err := artifacts.S3AudioCachePath(cacheRoot, "test-bucket", audioKey)
|
||||
if err != nil {
|
||||
t.Fatalf("S3AudioCachePath() error = %v", err)
|
||||
}
|
||||
otherCachePath, err := artifacts.S3AudioCachePath(cacheRoot, "test-bucket", "dnd/campaigns/other/sessions/2026-05-03/audio/bob.flac")
|
||||
if err != nil {
|
||||
t.Fatalf("S3AudioCachePath() error = %v", err)
|
||||
}
|
||||
mustWriteTestFile(t, cachePath, "cached-audio")
|
||||
mustWriteTestFile(t, otherCachePath, "other-audio")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--session-id", "2026-05-03", "--clear-cache"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
cleanAssertMissing(t, cachePath)
|
||||
cleanAssertExists(t, otherCachePath)
|
||||
if storeInitCalls != 1 {
|
||||
t.Fatalf("object store init calls = %d, want 1", storeInitCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteCleanLocalAudioClearCacheIsNoop(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--session-id", "2026-05-03", "--clear-cache"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "Cache: skipped (session does not use audio_s3)") {
|
||||
t.Fatalf("stdout = %q, want local audio cache no-op", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteCleanAllDeletesWorkAndSpoolContentsButPreservesCache(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, _, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
workRoot := filepath.Join(workspaceRoot, "work")
|
||||
spoolRoot := filepath.Join(workspaceRoot, "spool")
|
||||
cachePath := filepath.Join(workspaceRoot, "cache", "keep.txt")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "sample-campaign", "2026-05-03", "manifest.json"), "{}")
|
||||
mustWriteTestFile(t, filepath.Join(spoolRoot, "sample-campaign", "2026-05-03", "run-1", "audio", "alice.flac"), "audio")
|
||||
mustWriteTestFile(t, cachePath, "cache")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "--config", pipelinePath, "--all"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
cleanAssertMissing(t, workRoot)
|
||||
cleanAssertExists(t, spoolRoot)
|
||||
cleanAssertMissing(t, filepath.Join(spoolRoot, "sample-campaign"))
|
||||
cleanAssertExists(t, cachePath)
|
||||
}
|
||||
|
||||
func TestExecuteCleanAllClearCacheRemovesS3AudioNamespaceOnly(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, _, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
cacheRoot := filepath.Join(workspaceRoot, "cache")
|
||||
audioCachePath, err := artifacts.S3AudioCachePath(cacheRoot, "test-bucket", "dnd/campaigns/sample-campaign/sessions/2026-05-03/audio/alice.flac")
|
||||
if err != nil {
|
||||
t.Fatalf("S3AudioCachePath() error = %v", err)
|
||||
}
|
||||
otherCachePath, err := artifacts.S3AudioCachePath(cacheRoot, "test-bucket", "other-root/campaigns/sample-campaign/sessions/2026-05-03/audio/alice.flac")
|
||||
if err != nil {
|
||||
t.Fatalf("S3AudioCachePath() error = %v", err)
|
||||
}
|
||||
mustWriteTestFile(t, audioCachePath, "cached-audio")
|
||||
mustWriteTestFile(t, otherCachePath, "other-cache")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "--config", pipelinePath, "--all", "--clear-cache"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
cleanAssertMissing(t, audioCachePath)
|
||||
cleanAssertExists(t, otherCachePath)
|
||||
}
|
||||
|
||||
func TestExecuteCleanAllRejectsSessionScopedFlags(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean", "--config", pipelinePath, "--campaign", campaignPath, "--all"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "--all cannot be combined") {
|
||||
t.Fatalf("stderr = %q, want --all conflict", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanRequiresSessionID(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clean"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "--session-id is required unless --all is set") {
|
||||
t.Fatalf("stderr = %q, want missing session-id", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCleanRejectsUnsafeTargets(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
outside := t.TempDir()
|
||||
if err := reportCleanScopedDir(&bytes.Buffer{}, root, filepath.Join(outside, "target"), "test.outside", false); err == nil {
|
||||
t.Fatal("outside target error = nil, want error")
|
||||
}
|
||||
if err := reportCleanScopedDir(&bytes.Buffer{}, root, root, "test.root", false); err == nil {
|
||||
t.Fatal("root target error = nil, want error")
|
||||
}
|
||||
filePath := filepath.Join(root, "file.txt")
|
||||
mustWriteTestFile(t, filePath, "file")
|
||||
if err := reportCleanScopedDir(&bytes.Buffer{}, root, filePath, "test.file", false); err == nil {
|
||||
t.Fatal("file target error = nil, want error")
|
||||
}
|
||||
symlinkPath := filepath.Join(root, "link")
|
||||
if err := os.Symlink(filepath.Join(root, "missing"), symlinkPath); err != nil {
|
||||
t.Fatalf("Symlink() error = %v", err)
|
||||
}
|
||||
if err := reportCleanScopedDir(&bytes.Buffer{}, root, symlinkPath, "test.symlink", false); err == nil {
|
||||
t.Fatal("symlink target error = nil, want error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestClearIsNotCommandAlias(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"clear"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `unknown command: "clear"`) {
|
||||
t.Fatalf("stderr = %q, want unknown clear command", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func cleanAssertExists(t *testing.T, path string) {
|
||||
t.Helper()
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
t.Fatalf("expected %q to exist: %v", path, err)
|
||||
}
|
||||
}
|
||||
|
||||
func cleanAssertMissing(t *testing.T, path string) {
|
||||
t.Helper()
|
||||
if _, err := os.Stat(path); !os.IsNotExist(err) {
|
||||
t.Fatalf("expected %q to be missing, stat err=%v", path, err)
|
||||
}
|
||||
}
|
||||
@@ -7,7 +7,7 @@ import (
|
||||
"strings"
|
||||
)
|
||||
|
||||
var supportedCommands = []string{"run", "plan", "status", "resume", "run-stage"}
|
||||
var supportedCommands = []string{"run", "plan", "status", "resume", "run-stage", "analyze", "restore", "session", "artifacts", "locks", "clean"}
|
||||
|
||||
// Execute dispatches CLI commands and returns a process exit code.
|
||||
func Execute(args []string, stdout, stderr io.Writer) int {
|
||||
@@ -32,6 +32,18 @@ func Execute(args []string, stdout, stderr io.Writer) int {
|
||||
err = Resume(ctx, cmdArgs, stdout)
|
||||
case "run-stage":
|
||||
err = RunStage(ctx, cmdArgs, stdout)
|
||||
case "analyze":
|
||||
err = Analyze(ctx, cmdArgs, stdout)
|
||||
case "restore":
|
||||
err = Restore(ctx, cmdArgs, stdout)
|
||||
case "session":
|
||||
err = Session(ctx, cmdArgs, stdout)
|
||||
case "artifacts":
|
||||
err = Artifacts(ctx, cmdArgs, stdout)
|
||||
case "locks":
|
||||
err = Locks(ctx, cmdArgs, stdout)
|
||||
case "clean":
|
||||
err = Clean(ctx, cmdArgs, stdout)
|
||||
default:
|
||||
fmt.Fprintf(stderr, "unknown command: %q\n\n", cmd)
|
||||
printUsage(stderr)
|
||||
|
||||
@@ -24,7 +24,7 @@ func TestExecuteValidCommands(t *testing.T) {
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
manifestPath := writeManifestPathForExecute(t)
|
||||
|
||||
cases := []struct {
|
||||
@@ -32,11 +32,11 @@ func TestExecuteValidCommands(t *testing.T) {
|
||||
args []string
|
||||
wantOut string
|
||||
}{
|
||||
{name: "run", args: []string{"run", "--config", pipelinePath, "--session", sessionPath}, wantOut: "narratio run: session 2026-05-03; executed=9 skipped=0; manifest="},
|
||||
{name: "plan", args: []string{"plan", "--config", pipelinePath, "--session", sessionPath}, wantOut: "prepare: skip\ntranscribe: skip\nmerge: skip\npolish: skip\nnormalize: skip\ntrim: skip\nanalyze: skip\narchive: skip\nnotify: skip"},
|
||||
{name: "run", args: []string{"run", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, wantOut: "narratio run: session 2026-05-03; executed=9 skipped=0; manifest="},
|
||||
{name: "plan", args: []string{"plan", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, wantOut: "prepare: skip\ntranscribe: skip\nmerge: skip\npolish: skip\nnormalize: skip\ntrim: skip\nanalyze: skip\narchive: skip\nnotify: skip"},
|
||||
{name: "status", args: []string{"status", "--manifest", manifestPath}, wantOut: "session_id: 2026-05-03"},
|
||||
{name: "resume", args: []string{"resume", "--config", pipelinePath, "--session", sessionPath}, wantOut: "narratio resume: session 2026-05-03 has no remaining stages"},
|
||||
{name: "run-stage", args: []string{"run-stage", "--config", pipelinePath, "--session", sessionPath, "polish"}, wantOut: "narratio run-stage: stage=polish executed=0 skipped=1 force=false; manifest="},
|
||||
{name: "resume", args: []string{"resume", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, wantOut: "narratio resume: session 2026-05-03 has no remaining stages"},
|
||||
{name: "run-stage", args: []string{"run-stage", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "polish"}, wantOut: "narratio run-stage: stage=polish executed=0 skipped=1 force=false; manifest="},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
@@ -94,12 +94,12 @@ func TestExecuteMissingRequiredFlags(t *testing.T) {
|
||||
|
||||
func TestExecuteRunStageUnknownFails(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := Execute([]string{"run-stage", "--config", pipelinePath, "--session", sessionPath, "unknown"}, &stdout, &stderr)
|
||||
code := Execute([]string{"run-stage", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "unknown"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
@@ -110,14 +110,14 @@ func TestExecuteRunStageUnknownFails(t *testing.T) {
|
||||
|
||||
func TestExecuteRunStageNormalizeIsAccepted(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := Execute([]string{"run-stage", "--config", pipelinePath, "--session", sessionPath, "normalize"}, &stdout, &stderr)
|
||||
code := Execute([]string{"run-stage", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "normalize"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
@@ -136,19 +136,19 @@ func TestExecuteRunStageTranscribeUsesConfiguredWhisperXServer(t *testing.T) {
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := Execute([]string{"run-stage", "--config", pipelinePath, "--session", sessionPath, "prepare"}, &stdout, &stderr)
|
||||
code := Execute([]string{"run-stage", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "prepare"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("prepare exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
|
||||
code = Execute([]string{"run-stage", "--config", pipelinePath, "--session", sessionPath, "--force", "transcribe"}, &stdout, &stderr)
|
||||
code = Execute([]string{"run-stage", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--force", "transcribe"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("transcribe exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
@@ -156,7 +156,7 @@ func TestExecuteRunStageTranscribeUsesConfiguredWhisperXServer(t *testing.T) {
|
||||
t.Fatal("expected whisperx server to be called at least once")
|
||||
}
|
||||
|
||||
outPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "transcripts", "raw", "alice.json")
|
||||
outPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "transcripts", "raw", "alice.json")
|
||||
data, err := os.ReadFile(outPath)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile(%q): %v", outPath, err)
|
||||
@@ -188,6 +188,7 @@ func TestExecuteRunStagePolishLoadsCredentialFromSecretsDir(t *testing.T) {
|
||||
t.Setenv("GO_WANT_APP_AUDITA_HELPER", "1")
|
||||
|
||||
pipelinePath := filepath.Join(configDir, "pipeline.yml")
|
||||
campaignPath := writeAppTestCampaignConfig(t, configDir)
|
||||
sessionPath := filepath.Join(configDir, "session.yml")
|
||||
pipelineYAML := `workspace:
|
||||
root: ` + workspaceRoot + `
|
||||
@@ -233,13 +234,13 @@ inputs:
|
||||
_ = os.Chdir(originalWD)
|
||||
})
|
||||
|
||||
workRoot := filepath.Join(workspaceRoot, "work", sessionID)
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", sessionID)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"schema":"seriatim-intermediate","segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "[]\n")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"run-stage", "--config", pipelinePath, "--session", sessionPath, "--force", "polish"}, &stdout, &stderr)
|
||||
code := Execute([]string{"run-stage", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--force", "polish"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
@@ -252,6 +253,7 @@ func TestExecuteRunFailsWhenConfiguredSecretsDirMissing(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
configDir := t.TempDir()
|
||||
pipelinePath := filepath.Join(configDir, "pipeline.yml")
|
||||
campaignPath := writeAppTestCampaignConfig(t, configDir)
|
||||
sessionPath := filepath.Join(configDir, "session.yml")
|
||||
|
||||
pipelineYAML := `workspace:
|
||||
@@ -288,7 +290,7 @@ inputs:
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"run", "--config", pipelinePath, "--session", sessionPath}, &stdout, &stderr)
|
||||
code := Execute([]string{"run", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
@@ -305,11 +307,14 @@ func TestExecuteUsesDefaultPipelineConfigPathWhenConfigFlagOmitted(t *testing.T)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
originalDefaults := append([]string(nil), config.DefaultPipelineConfigSearchPaths...)
|
||||
originalCampaignDefaults := append([]string(nil), config.DefaultCampaignConfigSearchPaths...)
|
||||
config.DefaultPipelineConfigSearchPaths = []string{pipelinePath}
|
||||
config.DefaultCampaignConfigSearchPaths = []string{campaignPath}
|
||||
defer func() {
|
||||
config.DefaultPipelineConfigSearchPaths = originalDefaults
|
||||
config.DefaultCampaignConfigSearchPaths = originalCampaignDefaults
|
||||
}()
|
||||
|
||||
var stdout bytes.Buffer
|
||||
@@ -323,6 +328,33 @@ func TestExecuteUsesDefaultPipelineConfigPathWhenConfigFlagOmitted(t *testing.T)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteMissingCampaignConfigReportsSearchedPaths(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, _, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
missingCampaignPath := filepath.Join(t.TempDir(), "campaign.yml")
|
||||
originalCampaignDefaults := append([]string(nil), config.DefaultCampaignConfigSearchPaths...)
|
||||
config.DefaultCampaignConfigSearchPaths = []string{missingCampaignPath}
|
||||
defer func() {
|
||||
config.DefaultCampaignConfigSearchPaths = originalCampaignDefaults
|
||||
}()
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"run", "--config", pipelinePath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if stdout.Len() != 0 {
|
||||
t.Fatalf("stdout = %q, want empty", stdout.String())
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "no campaign config path provided and no default campaign config found; searched:") {
|
||||
t.Fatalf("stderr = %q, want campaign discovery failure", stderr.String())
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "pass --campaign") {
|
||||
t.Fatalf("stderr = %q, want explicit campaign guidance", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteInvalidCommand(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
@@ -359,11 +391,12 @@ func TestExecuteMissingCommand(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func writeValidConfigFiles(t *testing.T, workspaceRoot string, transcribeURL ...string) (string, string) {
|
||||
func writeValidConfigFiles(t *testing.T, workspaceRoot string, transcribeURL ...string) (string, string, string) {
|
||||
t.Helper()
|
||||
|
||||
dir := t.TempDir()
|
||||
pipelinePath := filepath.Join(dir, "pipeline.yml")
|
||||
campaignPath := filepath.Join(dir, "campaign.yml")
|
||||
sessionPath := filepath.Join(dir, "session.yml")
|
||||
url := "https://example.com/transcribe"
|
||||
if len(transcribeURL) > 0 && strings.TrimSpace(transcribeURL[0]) != "" {
|
||||
@@ -377,6 +410,10 @@ func writeValidConfigFiles(t *testing.T, workspaceRoot string, transcribeURL ...
|
||||
|
||||
pipelineYAML := `workspace:
|
||||
root: ` + workspaceRoot + `
|
||||
cache:
|
||||
root: ` + filepath.Join(workspaceRoot, "cache") + `
|
||||
spool:
|
||||
root: ` + filepath.Join(workspaceRoot, "spool") + `
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
@@ -407,9 +444,11 @@ notification:
|
||||
`
|
||||
|
||||
sessionYAML := `session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
`
|
||||
campaignYAML := `campaign: sample-campaign
|
||||
inputs:
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
@@ -418,6 +457,9 @@ inputs:
|
||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||
t.Fatalf("write pipeline config: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(campaignPath, []byte(campaignYAML), 0o644); err != nil {
|
||||
t.Fatalf("write campaign config: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(sessionPath, []byte(sessionYAML), 0o644); err != nil {
|
||||
t.Fatalf("write session config: %v", err)
|
||||
}
|
||||
@@ -427,7 +469,22 @@ inputs:
|
||||
mustWriteTestFile(t, filepath.Join(dir, "glossary.yml"), "[]\n")
|
||||
mustWriteTestFile(t, filepath.Join(dir, "audio", "alice.flac"), "audio-bytes")
|
||||
|
||||
return pipelinePath, sessionPath
|
||||
return pipelinePath, campaignPath, sessionPath
|
||||
}
|
||||
|
||||
func writeAppTestCampaignConfig(t *testing.T, dir string) string {
|
||||
t.Helper()
|
||||
campaignPath := filepath.Join(dir, "campaign.yml")
|
||||
campaignYAML := `campaign: sample-campaign
|
||||
inputs:
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`
|
||||
if err := os.WriteFile(campaignPath, []byte(campaignYAML), 0o644); err != nil {
|
||||
t.Fatalf("write campaign.yml: %v", err)
|
||||
}
|
||||
return campaignPath
|
||||
}
|
||||
|
||||
func writeManifestPathForExecute(t *testing.T) string {
|
||||
|
||||
133
internal/app/config_loader.go
Normal file
133
internal/app/config_loader.go
Normal file
@@ -0,0 +1,133 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func loadCommandConfig(ctx context.Context, pipelineFlag, campaignFlag, sessionFlag string, sessionOpts config.SessionLoadOptions) (*config.Config, error) {
|
||||
resolvedPipelinePath, err := resolvePipelineConfigPath(pipelineFlag)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resolvedCampaignPath, err := resolveCampaignConfigPath(campaignFlag)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if explicitSession := strings.TrimSpace(sessionFlag); explicitSession != "" {
|
||||
return config.LoadWithSessionOptions(resolvedPipelinePath, resolvedCampaignPath, explicitSession, sessionOpts)
|
||||
}
|
||||
|
||||
discoveredSession, err := discoverSessionConfigPathWithCandidates(config.DefaultSessionConfigSearchPaths)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if discoveredSession.Path != "" {
|
||||
return config.LoadWithSessionOptions(resolvedPipelinePath, resolvedCampaignPath, discoveredSession.Path, sessionOpts)
|
||||
}
|
||||
|
||||
pipelineCfg, err := config.LoadPipeline(resolvedPipelinePath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
campaignCfg, err := config.LoadCampaign(resolvedCampaignPath)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
sessionID := strings.TrimSpace(sessionOpts.SessionID)
|
||||
if sessionID == "" {
|
||||
return nil, missingSessionConfigError(discoveredSession.Searched, "remote session loading requires --session-id")
|
||||
}
|
||||
|
||||
sessionPrefix := artifacts.S3SessionPrefix(pipelineCfg.Storage.S3.RootPrefix, campaignCfg.Campaign, sessionID)
|
||||
remoteKey := artifacts.S3SessionConfigKey(sessionPrefix)
|
||||
partialCfg := &config.Config{
|
||||
Pipeline: pipelineCfg,
|
||||
Campaign: campaignCfg,
|
||||
PipelinePath: resolvedPipelinePath,
|
||||
CampaignPath: resolvedCampaignPath,
|
||||
}
|
||||
store, err := newCommandObjectStore(ctx, partialCfg, nil)
|
||||
if err != nil {
|
||||
return nil, missingSessionConfigError(discoveredSession.Searched, fmt.Sprintf("remote session %q unavailable: %v", remoteKey, err))
|
||||
}
|
||||
|
||||
sessionInfo, err := findRemoteSessionConfig(ctx, store, sessionPrefix, remoteKey)
|
||||
if err != nil {
|
||||
return nil, missingSessionConfigError(discoveredSession.Searched, err.Error())
|
||||
}
|
||||
sessionTempPath, err := downloadRemoteSessionConfig(ctx, store, remoteKey)
|
||||
if err != nil {
|
||||
return nil, missingSessionConfigError(discoveredSession.Searched, fmt.Sprintf("remote session %q download failed: %v", remoteKey, err))
|
||||
}
|
||||
sessionBytes, err := os.ReadFile(sessionTempPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read downloaded remote session %q: %w", sessionTempPath, err)
|
||||
}
|
||||
sessionCfg, err := config.LoadSessionBytesWithOptions("s3://"+s3BucketName(pipelineCfg)+"/"+remoteKey, sessionBytes, sessionOpts)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return config.Resolve(
|
||||
resolvedPipelinePath,
|
||||
pipelineCfg,
|
||||
resolvedCampaignPath,
|
||||
campaignCfg,
|
||||
sessionTempPath,
|
||||
sessionCfg,
|
||||
config.SessionSource{
|
||||
Source: "session_config.s3",
|
||||
LocalPath: sessionTempPath,
|
||||
S3Bucket: s3BucketName(pipelineCfg),
|
||||
S3Key: remoteKey,
|
||||
S3Size: sessionInfo.Size,
|
||||
S3ETag: sessionInfo.ETag,
|
||||
SpoolPath: sessionTempPath,
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
func findRemoteSessionConfig(ctx context.Context, store storage.ObjectStore, sessionPrefix, remoteKey string) (storage.ObjectInfo, error) {
|
||||
objects, err := store.List(ctx, sessionPrefix)
|
||||
if err != nil {
|
||||
return storage.ObjectInfo{}, fmt.Errorf("remote session %q list failed: %w", remoteKey, err)
|
||||
}
|
||||
for _, obj := range objects {
|
||||
if obj.Key == remoteKey {
|
||||
return obj, nil
|
||||
}
|
||||
}
|
||||
return storage.ObjectInfo{}, fmt.Errorf("remote session %q not found", remoteKey)
|
||||
}
|
||||
|
||||
func downloadRemoteSessionConfig(ctx context.Context, store storage.ObjectStore, remoteKey string) (string, error) {
|
||||
f, err := os.CreateTemp("", "narratio-session-*.yml")
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("create temp file: %w", err)
|
||||
}
|
||||
path := f.Name()
|
||||
if err := f.Close(); err != nil {
|
||||
return "", fmt.Errorf("close temp file %q: %w", path, err)
|
||||
}
|
||||
if err := store.Download(ctx, remoteKey, path); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return filepath.Clean(path), nil
|
||||
}
|
||||
|
||||
func s3BucketName(cfg *config.PipelineConfig) string {
|
||||
if cfg == nil || cfg.Storage.S3 == nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(cfg.Storage.S3.Bucket)
|
||||
}
|
||||
21
internal/app/object_store.go
Normal file
21
internal/app/object_store.go
Normal file
@@ -0,0 +1,21 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func newCommandObjectStore(ctx context.Context, cfg *config.Config, logger *slog.Logger) (storage.ObjectStore, error) {
|
||||
if _, err := loadSecretsFromConfig(cfg, logger); err != nil {
|
||||
return nil, fmt.Errorf("load secrets from files: %w", err)
|
||||
}
|
||||
store, err := newObjectStoreFromConfigFn(ctx, cfg)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("initialize object store backend: %w", err)
|
||||
}
|
||||
return store, nil
|
||||
}
|
||||
165
internal/app/object_store_test.go
Normal file
165
internal/app/object_store_test.go
Normal file
@@ -0,0 +1,165 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func TestNewCommandObjectStoreLoadsSecretsBeforeFactory(t *testing.T) {
|
||||
accessKeyEnv := "NARRATIO_TEST_COMMAND_STORE_KEY_ID"
|
||||
secretKeyEnv := "NARRATIO_TEST_COMMAND_STORE_SECRET"
|
||||
restoreEnvAfterTest(t, accessKeyEnv, secretKeyEnv)
|
||||
|
||||
secretsDir := t.TempDir()
|
||||
mustWriteSecretFile(t, filepath.Join(secretsDir, accessKeyEnv), "loaded-key-id\n")
|
||||
mustWriteSecretFile(t, filepath.Join(secretsDir, secretKeyEnv), "loaded-secret\n")
|
||||
|
||||
cfg := commandObjectStoreTestConfig(secretsDir)
|
||||
fake := &storage.FakeBackend{}
|
||||
called := false
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
called = true
|
||||
if got := os.Getenv(accessKeyEnv); got != "loaded-key-id" {
|
||||
return nil, errors.New("access key was not loaded before object store init")
|
||||
}
|
||||
if got := os.Getenv(secretKeyEnv); got != "loaded-secret" {
|
||||
return nil, errors.New("secret key was not loaded before object store init")
|
||||
}
|
||||
return fake, nil
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
})
|
||||
|
||||
store, err := newCommandObjectStore(context.Background(), cfg, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("newCommandObjectStore() error = %v", err)
|
||||
}
|
||||
if store != fake {
|
||||
t.Fatalf("store = %#v, want fake backend", store)
|
||||
}
|
||||
if !called {
|
||||
t.Fatal("object store factory was not called")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewCommandObjectStorePreservesExistingEnv(t *testing.T) {
|
||||
accessKeyEnv := "NARRATIO_TEST_COMMAND_STORE_EXISTING_KEY_ID"
|
||||
secretKeyEnv := "NARRATIO_TEST_COMMAND_STORE_EXISTING_SECRET"
|
||||
t.Setenv(accessKeyEnv, "existing-key-id")
|
||||
t.Setenv(secretKeyEnv, "existing-secret")
|
||||
|
||||
secretsDir := t.TempDir()
|
||||
mustWriteSecretFile(t, filepath.Join(secretsDir, accessKeyEnv), "file-key-id\n")
|
||||
mustWriteSecretFile(t, filepath.Join(secretsDir, secretKeyEnv), "file-secret\n")
|
||||
|
||||
cfg := commandObjectStoreTestConfig(secretsDir)
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
if got := os.Getenv(accessKeyEnv); got != "existing-key-id" {
|
||||
return nil, errors.New("existing access key was overwritten")
|
||||
}
|
||||
if got := os.Getenv(secretKeyEnv); got != "existing-secret" {
|
||||
return nil, errors.New("existing secret key was overwritten")
|
||||
}
|
||||
return &storage.FakeBackend{}, nil
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
})
|
||||
|
||||
if _, err := newCommandObjectStore(context.Background(), cfg, nil); err != nil {
|
||||
t.Fatalf("newCommandObjectStore() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewCommandObjectStoreSecretErrorStopsFactory(t *testing.T) {
|
||||
cfg := commandObjectStoreTestConfig(filepath.Join(t.TempDir(), "missing"))
|
||||
called := false
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
called = true
|
||||
return &storage.FakeBackend{}, nil
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
})
|
||||
|
||||
_, err := newCommandObjectStore(context.Background(), cfg, nil)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if called {
|
||||
t.Fatal("object store factory was called after secret load failure")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "load secrets from files") {
|
||||
t.Fatalf("error = %q, want secret loading context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewCommandObjectStoreFactoryErrorIsContextual(t *testing.T) {
|
||||
cfg := commandObjectStoreTestConfig("")
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
return nil, errors.New("factory boom")
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
})
|
||||
|
||||
_, err := newCommandObjectStore(context.Background(), cfg, nil)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "initialize object store backend") || !strings.Contains(err.Error(), "factory boom") {
|
||||
t.Fatalf("error = %q, want factory context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func commandObjectStoreTestConfig(secretsDir string) *config.Config {
|
||||
cfg := &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Storage: config.StorageConfig{
|
||||
Backend: "s3",
|
||||
S3: &config.StorageS3Config{
|
||||
Bucket: "test-bucket",
|
||||
AccessKeyIDEnv: "NARRATIO_TEST_COMMAND_STORE_KEY_ID",
|
||||
SecretKeyEnv: "NARRATIO_TEST_COMMAND_STORE_SECRET",
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
if strings.TrimSpace(secretsDir) != "" {
|
||||
cfg.Pipeline.Secrets = &config.SecretsConfig{EnvDir: secretsDir}
|
||||
}
|
||||
return cfg
|
||||
}
|
||||
|
||||
func restoreEnvAfterTest(t *testing.T, names ...string) {
|
||||
t.Helper()
|
||||
originals := make(map[string]string, len(names))
|
||||
present := make(map[string]bool, len(names))
|
||||
for _, name := range names {
|
||||
value, ok := os.LookupEnv(name)
|
||||
originals[name] = value
|
||||
present[name] = ok
|
||||
_ = os.Unsetenv(name)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
for _, name := range names {
|
||||
if present[name] {
|
||||
_ = os.Setenv(name, originals[name])
|
||||
} else {
|
||||
_ = os.Unsetenv(name)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
958
internal/app/operator_helpers.go
Normal file
958
internal/app/operator_helpers.go
Normal file
@@ -0,0 +1,958 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
type commonConfigFlags struct {
|
||||
pipelinePath string
|
||||
campaignPath string
|
||||
sessionPath string
|
||||
sessionID string
|
||||
previousSessionID string
|
||||
}
|
||||
|
||||
type finding struct {
|
||||
Severity string
|
||||
Category string
|
||||
Message string
|
||||
}
|
||||
|
||||
type findingError struct {
|
||||
count int
|
||||
}
|
||||
|
||||
func (e findingError) Error() string {
|
||||
return fmt.Sprintf("%d validation error(s)", e.count)
|
||||
}
|
||||
|
||||
func addCommonConfigFlags(fs *flag.FlagSet, flags *commonConfigFlags) {
|
||||
fs.StringVar(&flags.pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||
fs.StringVar(&flags.campaignPath, "campaign", "", "path to campaign.yml (optional; defaults searched)")
|
||||
fs.StringVar(&flags.sessionPath, "session", "", "path to session.yml")
|
||||
fs.StringVar(&flags.sessionID, "session-id", "", "session identifier for session.yml templates")
|
||||
fs.StringVar(&flags.previousSessionID, "previous-session-id", "", "previous session identifier for session.yml templates")
|
||||
}
|
||||
|
||||
func (f commonConfigFlags) sessionOptions() config.SessionLoadOptions {
|
||||
return config.SessionLoadOptions{
|
||||
SessionID: f.sessionID,
|
||||
PreviousSessionID: f.previousSessionID,
|
||||
}
|
||||
}
|
||||
|
||||
// Session dispatches session helper subcommands.
|
||||
func Session(ctx context.Context, args []string, out io.Writer) error {
|
||||
if len(args) == 0 {
|
||||
return fmt.Errorf("session: expected subcommand: validate|init")
|
||||
}
|
||||
switch args[0] {
|
||||
case "validate":
|
||||
return SessionValidate(ctx, args[1:], out)
|
||||
case "init":
|
||||
return SessionInit(ctx, args[1:], out)
|
||||
default:
|
||||
return fmt.Errorf("session: unknown subcommand %q", args[0])
|
||||
}
|
||||
}
|
||||
|
||||
// Artifacts dispatches artifact helper subcommands.
|
||||
func Artifacts(ctx context.Context, args []string, out io.Writer) error {
|
||||
if len(args) == 0 {
|
||||
return fmt.Errorf("artifacts: expected subcommand: list")
|
||||
}
|
||||
switch args[0] {
|
||||
case "list":
|
||||
return ArtifactsList(ctx, args[1:], out)
|
||||
default:
|
||||
return fmt.Errorf("artifacts: unknown subcommand %q", args[0])
|
||||
}
|
||||
}
|
||||
|
||||
// SessionValidate performs a read-only session preflight.
|
||||
func SessionValidate(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("session validate", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var flags commonConfigFlags
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("session validate: invalid flags: %w", err)
|
||||
}
|
||||
if fs.NArg() != 0 {
|
||||
return fmt.Errorf("session validate: unexpected positional arguments")
|
||||
}
|
||||
|
||||
findings := []finding{}
|
||||
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.sessionPath, flags.sessionOptions())
|
||||
if err != nil {
|
||||
findings = append(findings, errorFinding("config", err.Error()))
|
||||
return renderFindings(out, "", "", findings)
|
||||
}
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
findings = append(findings, errorFinding("config", err.Error()))
|
||||
} else {
|
||||
findings = append(findings, okFinding("config", "resolved pipeline, campaign, and session config"))
|
||||
}
|
||||
findings = append(findings, okFinding("session", fmt.Sprintf("session source: %s", sessionSourceSummary(cfg))))
|
||||
|
||||
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
findings = append(findings, validateStableInputFindings(cfg)...)
|
||||
findings = append(findings, validateLocalAudioFindings(cfg)...)
|
||||
|
||||
store, storeErr := objectStoreIfConfigured(ctx, cfg)
|
||||
if storeErr != nil {
|
||||
findings = append(findings, errorFinding("storage", storeErr.Error()))
|
||||
}
|
||||
if cfg.Session.Inputs.AudioS3 != nil {
|
||||
if storeErr != nil {
|
||||
findings = append(findings, errorFinding("audio", "remote audio cannot be checked because storage is unavailable"))
|
||||
} else {
|
||||
findings = append(findings, validateRemoteAudioFinding(ctx, cfg, store))
|
||||
}
|
||||
}
|
||||
|
||||
requirements := artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg))
|
||||
if len(requirements) == 0 {
|
||||
findings = append(findings, okFinding("previous", "no previous-session artifacts required"))
|
||||
} else if strings.TrimSpace(cfg.Session.PreviousSessionID) == "" {
|
||||
findings = append(findings, errorFinding("previous", "previous_session_id is required by configured previous-session artifacts"))
|
||||
} else if storeErr != nil {
|
||||
findings = append(findings, errorFinding("previous", "previous-session artifacts cannot be checked because storage is unavailable"))
|
||||
} else {
|
||||
findings = append(findings, validatePreviousArtifactFindings(ctx, cfg, store, requirements)...)
|
||||
}
|
||||
|
||||
locks, lockErr := loadEffectiveLocks(ctx, cfg, store)
|
||||
if lockErr != nil {
|
||||
findings = append(findings, errorFinding("locks", lockErr.Error()))
|
||||
} else if len(locks.All) == 0 {
|
||||
findings = append(findings, okFinding("locks", "no effective archive locks"))
|
||||
} else {
|
||||
for _, lock := range locks.All {
|
||||
findings = append(findings, warnFinding("locks", fmt.Sprintf("%s locked: %s", lock.Source, strings.TrimSpace(lock.Reason))))
|
||||
}
|
||||
}
|
||||
if paths.ManifestPath != "" {
|
||||
findings = append(findings, infoFinding("workspace", "manifest path: "+paths.ManifestPath))
|
||||
}
|
||||
return renderFindings(out, cfg.Session.Campaign, cfg.Session.SessionID, findings)
|
||||
}
|
||||
|
||||
// Status reports either a requested manifest or effective local/remote session state.
|
||||
func Status(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("status", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var manifestPath string
|
||||
var flags commonConfigFlags
|
||||
fs.StringVar(&manifestPath, "manifest", "", "path to manifest.json")
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("status: invalid flags: %w", err)
|
||||
}
|
||||
if fs.NArg() != 0 {
|
||||
return fmt.Errorf("status: unexpected positional arguments")
|
||||
}
|
||||
if strings.TrimSpace(manifestPath) != "" {
|
||||
return statusManifest(ctx, manifestPath, out)
|
||||
}
|
||||
if flags.pipelinePath == "" && flags.campaignPath == "" && flags.sessionPath == "" && flags.sessionID == "" && flags.previousSessionID == "" {
|
||||
return fmt.Errorf("status: --manifest is required")
|
||||
}
|
||||
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.sessionPath, flags.sessionOptions())
|
||||
if err != nil {
|
||||
return fmt.Errorf("status: %w", err)
|
||||
}
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
return fmt.Errorf("status: %w", err)
|
||||
}
|
||||
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
fmt.Fprintf(out, "Session: %s\n", cfg.Session.SessionID)
|
||||
fmt.Fprintf(out, "Campaign: %s\n", cfg.Session.Campaign)
|
||||
fmt.Fprintf(out, "Workspace: %s\n", paths.Root)
|
||||
fmt.Fprintf(out, "Session config: %s\n", sessionSourceSummary(cfg))
|
||||
|
||||
if m, err := loadLocalManifest(ctx, paths.ManifestPath); err != nil {
|
||||
fmt.Fprintf(out, "Local manifest: error: %v\n", err)
|
||||
} else if m == nil {
|
||||
fmt.Fprintln(out, "Local manifest: missing")
|
||||
} else {
|
||||
fmt.Fprintf(out, "Local manifest: %s\n", paths.ManifestPath)
|
||||
writeStageStatuses(out, m)
|
||||
}
|
||||
|
||||
store, storeErr := objectStoreIfConfigured(ctx, cfg)
|
||||
if storeErr != nil {
|
||||
fmt.Fprintf(out, "Remote archive: unavailable: %v\n", storeErr)
|
||||
} else if store != nil {
|
||||
current, err := discoverRemoteCurrentStateFn(ctx, cfg, store)
|
||||
if err != nil {
|
||||
fmt.Fprintf(out, "Remote archive: missing or unavailable: %v\n", err)
|
||||
} else {
|
||||
fmt.Fprintf(out, "Remote archive: current run %s\n", current.RunID)
|
||||
fmt.Fprintf(out, "Remote manifest: %s\n", current.CurrentManifestKey)
|
||||
}
|
||||
}
|
||||
|
||||
locks, err := loadEffectiveLocks(ctx, cfg, store)
|
||||
if catalog, catalogErr := buildHelperArtifactCatalog(cfg); catalogErr != nil {
|
||||
fmt.Fprintf(out, "Remote outputs: error: %v\n", catalogErr)
|
||||
} else if storeErr == nil {
|
||||
catalogLocks := locks
|
||||
if err != nil {
|
||||
catalogLocks = &effectiveLocks{
|
||||
Static: staticArchiveLocks(cfg),
|
||||
All: staticArchiveLocks(cfg),
|
||||
}
|
||||
}
|
||||
promotedRemoteState := map[string]string{}
|
||||
if store != nil {
|
||||
promotedRemoteState = remotePromotionAvailability(ctx, cfg, store, catalog)
|
||||
}
|
||||
fmt.Fprintln(out, "Remote outputs:")
|
||||
writeArtifactList(out, cfg, catalog, catalogLocks, promotedRemoteState)
|
||||
}
|
||||
if err != nil {
|
||||
fmt.Fprintf(out, "Archive locks: error: %v\n", err)
|
||||
} else {
|
||||
writeLocks(out, cfg, locks)
|
||||
}
|
||||
fmt.Fprintln(out, "Next actions:")
|
||||
fmt.Fprintf(out, "- narratio session validate --session-id %s\n", cfg.Session.SessionID)
|
||||
fmt.Fprintf(out, "- narratio restore --session-id %s --dry-run\n", cfg.Session.SessionID)
|
||||
return nil
|
||||
}
|
||||
|
||||
func statusManifest(ctx context.Context, manifestPath string, out io.Writer) error {
|
||||
store := &manifest.LocalStore{}
|
||||
m, err := store.Load(ctx, manifestPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("status: %w", err)
|
||||
}
|
||||
if _, err := fmt.Fprintf(out, "session_id: %s\n", m.SessionID); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := fmt.Fprintf(out, "updated_at: %s\n", m.UpdatedAt.UTC().Format("2006-01-02T15:04:05Z07:00")); err != nil {
|
||||
return err
|
||||
}
|
||||
writeStageStatuses(out, m)
|
||||
return nil
|
||||
}
|
||||
|
||||
// SessionInit creates a local or remote session.yml skeleton.
|
||||
func SessionInit(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("session init", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var pipelinePath, campaignPath, sessionID, previousSessionID, date, title, output, audioS3Prefix, audioDir string
|
||||
var remote, force bool
|
||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml")
|
||||
fs.StringVar(&campaignPath, "campaign", "", "path to campaign.yml")
|
||||
fs.StringVar(&sessionID, "session-id", "", "session identifier")
|
||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "previous session identifier")
|
||||
fs.StringVar(&date, "date", "", "session date")
|
||||
fs.StringVar(&title, "title", "", "session title")
|
||||
fs.StringVar(&output, "output", "", "local output session.yml path")
|
||||
fs.StringVar(&audioS3Prefix, "audio-s3-prefix", "", "session audio S3 prefix")
|
||||
fs.StringVar(&audioDir, "audio-dir", "", "local audio directory")
|
||||
fs.BoolVar(&remote, "remote", false, "write session.yml to S3 session prefix")
|
||||
fs.BoolVar(&force, "force", false, "overwrite existing target")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("session init: invalid flags: %w", err)
|
||||
}
|
||||
if fs.NArg() != 0 {
|
||||
return fmt.Errorf("session init: unexpected positional arguments")
|
||||
}
|
||||
if strings.TrimSpace(pipelinePath) == "" || strings.TrimSpace(campaignPath) == "" || strings.TrimSpace(sessionID) == "" {
|
||||
return fmt.Errorf("session init: --config, --campaign, and --session-id are required")
|
||||
}
|
||||
if (strings.TrimSpace(output) == "") == !remote {
|
||||
return fmt.Errorf("session init: specify exactly one target: --output <path> or --remote")
|
||||
}
|
||||
if strings.TrimSpace(audioDir) != "" && strings.TrimSpace(audioS3Prefix) != "" {
|
||||
return fmt.Errorf("session init: --audio-dir and --audio-s3-prefix are mutually exclusive")
|
||||
}
|
||||
|
||||
resolvedPipeline, err := resolvePipelineConfigPath(pipelinePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
resolvedCampaign, err := resolveCampaignConfigPath(campaignPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
pipelineCfg, err := config.LoadPipeline(resolvedPipeline)
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
campaignCfg, err := config.LoadCampaign(resolvedCampaign)
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
|
||||
data, err := buildSessionYAML(campaignCfg.Campaign, sessionID, previousSessionID, date, title, audioS3Prefix, audioDir)
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
label := strings.TrimSpace(output)
|
||||
if label == "" {
|
||||
label = "remote session.yml"
|
||||
}
|
||||
sessionCfg, err := config.LoadSessionBytesWithOptions(label, data, config.SessionLoadOptions{
|
||||
SessionID: sessionID,
|
||||
PreviousSessionID: previousSessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
cfg, err := config.Resolve(resolvedPipeline, pipelineCfg, resolvedCampaign, campaignCfg, label, sessionCfg, config.SessionSource{Source: "session_config", LocalPath: label})
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
|
||||
if !remote {
|
||||
if err := writeLocalFile(output, data, force); err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
_, err := fmt.Fprintf(out, "narratio session init: wrote %s\n", filepath.Clean(output))
|
||||
return err
|
||||
}
|
||||
|
||||
store, err := newCommandObjectStore(ctx, cfg, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: %w", err)
|
||||
}
|
||||
sessionPrefix := artifacts.S3SessionPrefix(pipelineCfg.Storage.S3.RootPrefix, campaignCfg.Campaign, sessionID)
|
||||
key := artifacts.S3SessionConfigKey(sessionPrefix)
|
||||
exists, err := store.Exists(ctx, key)
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: check remote session %q: %w", key, err)
|
||||
}
|
||||
if exists && !force {
|
||||
return fmt.Errorf("session init: remote session %q already exists; pass --force to overwrite", key)
|
||||
}
|
||||
tmp, err := os.CreateTemp("", "narratio-session-init-*.yml")
|
||||
if err != nil {
|
||||
return fmt.Errorf("session init: create temp file: %w", err)
|
||||
}
|
||||
tmpPath := tmp.Name()
|
||||
defer func() { _ = os.Remove(tmpPath) }()
|
||||
if _, err := tmp.Write(data); err != nil {
|
||||
_ = tmp.Close()
|
||||
return fmt.Errorf("session init: write temp file: %w", err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return fmt.Errorf("session init: close temp file: %w", err)
|
||||
}
|
||||
if _, err := store.Upload(ctx, tmpPath, key, storage.UploadOptions{ContentType: "application/x-yaml; charset=utf-8"}); err != nil {
|
||||
return fmt.Errorf("session init: upload remote session %q: %w", key, err)
|
||||
}
|
||||
_, err = fmt.Fprintf(out, "narratio session init: wrote s3://%s/%s\n", s3BucketName(pipelineCfg), key)
|
||||
return err
|
||||
}
|
||||
|
||||
// ArtifactsList lists effective artifact sources.
|
||||
func ArtifactsList(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("artifacts list", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var flags commonConfigFlags
|
||||
var remote bool
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
fs.BoolVar(&remote, "remote", false, "inspect remote archive availability")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("artifacts list: invalid flags: %w", err)
|
||||
}
|
||||
if fs.NArg() != 0 {
|
||||
return fmt.Errorf("artifacts list: unexpected positional arguments")
|
||||
}
|
||||
cfg, store, locks, _, err := loadHelperContext(ctx, flags, remote)
|
||||
if err != nil {
|
||||
return fmt.Errorf("artifacts list: %w", err)
|
||||
}
|
||||
catalog, err := buildHelperArtifactCatalog(cfg)
|
||||
if err != nil {
|
||||
return fmt.Errorf("artifacts list: %w", err)
|
||||
}
|
||||
promotedRemoteState := map[string]string{}
|
||||
if remote && store != nil {
|
||||
promotedRemoteState = remotePromotionAvailability(ctx, cfg, store, catalog)
|
||||
}
|
||||
writeArtifactList(out, cfg, catalog, locks, promotedRemoteState)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Locks dispatches archive lock list and mutation helpers.
|
||||
func Locks(ctx context.Context, args []string, out io.Writer) error {
|
||||
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
|
||||
switch args[0] {
|
||||
case "add":
|
||||
return LocksAdd(ctx, args[1:], out)
|
||||
case "remove":
|
||||
return LocksRemove(ctx, args[1:], out)
|
||||
default:
|
||||
return fmt.Errorf("locks: unknown subcommand %q", args[0])
|
||||
}
|
||||
}
|
||||
return LocksList(ctx, args, out)
|
||||
}
|
||||
|
||||
// LocksList lists effective archive locks.
|
||||
func LocksList(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("locks", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var flags commonConfigFlags
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("locks: invalid flags: %w", err)
|
||||
}
|
||||
if fs.NArg() != 0 {
|
||||
return fmt.Errorf("locks: unexpected positional arguments")
|
||||
}
|
||||
if strings.TrimSpace(flags.sessionID) == "" {
|
||||
return fmt.Errorf("locks: --session-id is required")
|
||||
}
|
||||
cfg, _, locks, _, err := loadHelperContext(ctx, flags, true)
|
||||
if err != nil {
|
||||
return fmt.Errorf("locks: %w", err)
|
||||
}
|
||||
writeLocks(out, cfg, locks)
|
||||
return nil
|
||||
}
|
||||
|
||||
// LocksAdd adds or updates one remote lock.
|
||||
func LocksAdd(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("locks add", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var flags commonConfigFlags
|
||||
var reason string
|
||||
var force bool
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
fs.StringVar(&reason, "reason", "", "lock reason")
|
||||
fs.BoolVar(&force, "force", false, "update existing remote lock")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("locks add: invalid flags: %w", err)
|
||||
}
|
||||
if fs.NArg() != 1 {
|
||||
return fmt.Errorf("locks add: expected exactly one source id")
|
||||
}
|
||||
if strings.TrimSpace(flags.sessionID) == "" {
|
||||
return fmt.Errorf("locks add: --session-id is required")
|
||||
}
|
||||
source := strings.TrimSpace(fs.Arg(0))
|
||||
cfg, store, locks, _, err := loadHelperContext(ctx, flags, true)
|
||||
if err != nil {
|
||||
return fmt.Errorf("locks add: %w", err)
|
||||
}
|
||||
if _, err := config.ValidateArchiveLockRules([]config.ArchiveLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, "locks add"); err != nil {
|
||||
return fmt.Errorf("locks add: %w", err)
|
||||
}
|
||||
if _, ok := lockSourceSet(locks.Static)[source]; ok {
|
||||
return fmt.Errorf("locks add: source %q is locked by pipeline config and cannot be modified remotely", source)
|
||||
}
|
||||
remoteSet := lockSourceSet(locks.Remote)
|
||||
if _, exists := remoteSet[source]; exists && !force {
|
||||
return fmt.Errorf("locks add: remote lock for %q already exists; pass --force to update", source)
|
||||
}
|
||||
remoteSet[source] = config.ArchiveLockRule{Source: source, Reason: strings.TrimSpace(reason)}
|
||||
remoteLocks := lockMapValues(remoteSet)
|
||||
if _, err := config.ValidateArchiveLockRules(remoteLocks, cfg.Pipeline.Scriptorium, "locks"); err != nil {
|
||||
return fmt.Errorf("locks add: %w", err)
|
||||
}
|
||||
if err := uploadRemoteLockStore(ctx, store, locks.Key, &config.ArchiveLockStore{Locks: remoteLocks}); err != nil {
|
||||
return fmt.Errorf("locks add: %w", err)
|
||||
}
|
||||
_, err = fmt.Fprintf(out, "narratio locks add: locked %s\n", source)
|
||||
return err
|
||||
}
|
||||
|
||||
// LocksRemove removes one remote lock.
|
||||
func LocksRemove(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("locks remove", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
var flags commonConfigFlags
|
||||
addCommonConfigFlags(fs, &flags)
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("locks remove: invalid flags: %w", err)
|
||||
}
|
||||
if fs.NArg() != 1 {
|
||||
return fmt.Errorf("locks remove: expected exactly one source id")
|
||||
}
|
||||
if strings.TrimSpace(flags.sessionID) == "" {
|
||||
return fmt.Errorf("locks remove: --session-id is required")
|
||||
}
|
||||
source := strings.TrimSpace(fs.Arg(0))
|
||||
cfg, store, locks, _, err := loadHelperContext(ctx, flags, true)
|
||||
if err != nil {
|
||||
return fmt.Errorf("locks remove: %w", err)
|
||||
}
|
||||
if _, err := config.ValidateArchiveLockRules([]config.ArchiveLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, "locks remove"); err != nil {
|
||||
return fmt.Errorf("locks remove: %w", err)
|
||||
}
|
||||
remoteSet := lockSourceSet(locks.Remote)
|
||||
if _, ok := remoteSet[source]; !ok {
|
||||
if _, static := lockSourceSet(locks.Static)[source]; static {
|
||||
return fmt.Errorf("locks remove: source %q is locked by pipeline config and cannot be unlocked remotely", source)
|
||||
}
|
||||
return fmt.Errorf("locks remove: remote lock for %q does not exist", source)
|
||||
}
|
||||
delete(remoteSet, source)
|
||||
remoteLocks := lockMapValues(remoteSet)
|
||||
if err := uploadRemoteLockStore(ctx, store, locks.Key, &config.ArchiveLockStore{Locks: remoteLocks}); err != nil {
|
||||
return fmt.Errorf("locks remove: %w", err)
|
||||
}
|
||||
_, err = fmt.Fprintf(out, "narratio locks remove: unlocked %s\n", source)
|
||||
return err
|
||||
}
|
||||
|
||||
func loadHelperContext(ctx context.Context, flags commonConfigFlags, needStore bool) (*config.Config, storage.ObjectStore, *effectiveLocks, *manifest.Manifest, error) {
|
||||
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.sessionPath, flags.sessionOptions())
|
||||
if err != nil {
|
||||
return nil, nil, nil, nil, err
|
||||
}
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
return nil, nil, nil, nil, err
|
||||
}
|
||||
var store storage.ObjectStore
|
||||
if needStore {
|
||||
store, err = newCommandObjectStore(ctx, cfg, nil)
|
||||
if err != nil {
|
||||
return nil, nil, nil, nil, err
|
||||
}
|
||||
} else {
|
||||
store, _ = objectStoreIfConfigured(ctx, cfg)
|
||||
}
|
||||
locks, err := loadEffectiveLocks(ctx, cfg, store)
|
||||
if err != nil {
|
||||
return nil, nil, nil, nil, err
|
||||
}
|
||||
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
m, err := loadLocalManifest(ctx, paths.ManifestPath)
|
||||
if err != nil {
|
||||
return nil, nil, nil, nil, err
|
||||
}
|
||||
return cfg, store, locks, m, nil
|
||||
}
|
||||
|
||||
func objectStoreIfConfigured(ctx context.Context, cfg *config.Config) (storage.ObjectStore, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Storage.S3 == nil || strings.TrimSpace(cfg.Pipeline.Storage.S3.Bucket) == "" {
|
||||
return nil, nil
|
||||
}
|
||||
store, err := newCommandObjectStore(ctx, cfg, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return store, nil
|
||||
}
|
||||
|
||||
func buildSessionYAML(campaign, sessionID, previousSessionID, date, title, audioS3Prefix, audioDir string) ([]byte, error) {
|
||||
if strings.TrimSpace(date) == "" && regexp.MustCompile(`^\d{4}-\d{2}-\d{2}$`).MatchString(strings.TrimSpace(sessionID)) {
|
||||
date = strings.TrimSpace(sessionID)
|
||||
}
|
||||
type audioS3 struct {
|
||||
Prefix string `yaml:"prefix"`
|
||||
}
|
||||
type inputs struct {
|
||||
AudioDir string `yaml:"audio_dir,omitempty"`
|
||||
AudioS3 *audioS3 `yaml:"audio_s3,omitempty"`
|
||||
}
|
||||
type sessionYAML struct {
|
||||
Campaign string `yaml:"campaign"`
|
||||
SessionID string `yaml:"session_id"`
|
||||
PreviousSessionID string `yaml:"previous_session_id,omitempty"`
|
||||
Date string `yaml:"date,omitempty"`
|
||||
Title string `yaml:"title,omitempty"`
|
||||
Inputs inputs `yaml:"inputs"`
|
||||
}
|
||||
in := inputs{AudioDir: strings.TrimSpace(audioDir)}
|
||||
if in.AudioDir == "" {
|
||||
prefix := strings.TrimSpace(audioS3Prefix)
|
||||
if prefix == "" {
|
||||
prefix = "audio/"
|
||||
}
|
||||
in.AudioS3 = &audioS3{Prefix: prefix}
|
||||
}
|
||||
data, err := yaml.Marshal(sessionYAML{
|
||||
Campaign: strings.TrimSpace(campaign),
|
||||
SessionID: strings.TrimSpace(sessionID),
|
||||
PreviousSessionID: strings.TrimSpace(previousSessionID),
|
||||
Date: strings.TrimSpace(date),
|
||||
Title: strings.TrimSpace(title),
|
||||
Inputs: in,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return data, nil
|
||||
}
|
||||
|
||||
func renderFindings(out io.Writer, campaign, sessionID string, findings []finding) error {
|
||||
if campaign != "" || sessionID != "" {
|
||||
fmt.Fprintf(out, "Campaign: %s\n", campaign)
|
||||
fmt.Fprintf(out, "Session: %s\n\n", sessionID)
|
||||
}
|
||||
errorsCount := 0
|
||||
for _, f := range findings {
|
||||
if f.Severity == "ERROR" {
|
||||
errorsCount++
|
||||
}
|
||||
fmt.Fprintf(out, "%-5s %-10s %s\n", f.Severity, f.Category, f.Message)
|
||||
}
|
||||
if errorsCount > 0 {
|
||||
return findingError{count: errorsCount}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func okFinding(category, msg string) finding { return finding{"OK", category, msg} }
|
||||
func infoFinding(category, msg string) finding { return finding{"INFO", category, msg} }
|
||||
func warnFinding(category, msg string) finding { return finding{"WARN", category, msg} }
|
||||
func errorFinding(category, msg string) finding { return finding{"ERROR", category, msg} }
|
||||
|
||||
func sessionSourceSummary(cfg *config.Config) string {
|
||||
source := cfg.SessionSource.Source
|
||||
if source == "" {
|
||||
source = "session_config"
|
||||
}
|
||||
if cfg.SessionSource.S3Key != "" {
|
||||
return source + " " + cfg.SessionSource.S3Key
|
||||
}
|
||||
return source + " " + cfg.SessionPath
|
||||
}
|
||||
|
||||
func validateStableInputFindings(cfg *config.Config) []finding {
|
||||
items := []struct {
|
||||
name string
|
||||
in config.ResolvedInputFile
|
||||
}{
|
||||
{"speakers", cfg.StableInputs.SpeakersFile},
|
||||
{"autocorrect", cfg.StableInputs.AutocorrectFile},
|
||||
{"glossary", cfg.StableInputs.GlossaryFile},
|
||||
}
|
||||
out := make([]finding, 0, len(items))
|
||||
for _, item := range items {
|
||||
path, err := resolveHelperConfigRelativePath(item.in)
|
||||
if err != nil {
|
||||
out = append(out, errorFinding("inputs", item.name+": "+err.Error()))
|
||||
continue
|
||||
}
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
out = append(out, errorFinding("inputs", fmt.Sprintf("%s missing: %v", item.name, err)))
|
||||
} else {
|
||||
out = append(out, okFinding("inputs", item.name+": "+path))
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func resolveHelperConfigRelativePath(input config.ResolvedInputFile) (string, error) {
|
||||
if strings.TrimSpace(input.ConfigPath) == "" {
|
||||
return "", fmt.Errorf("source config path is required")
|
||||
}
|
||||
path := strings.TrimSpace(input.Path)
|
||||
if path == "" {
|
||||
return "", fmt.Errorf("path is required")
|
||||
}
|
||||
if filepath.IsAbs(path) {
|
||||
return filepath.Clean(path), nil
|
||||
}
|
||||
return filepath.Clean(filepath.Join(filepath.Dir(input.ConfigPath), path)), nil
|
||||
}
|
||||
|
||||
func validateLocalAudioFindings(cfg *config.Config) []finding {
|
||||
if cfg.Session.Inputs.AudioS3 != nil {
|
||||
return nil
|
||||
}
|
||||
audioDir := strings.TrimSpace(cfg.Session.Inputs.AudioDir)
|
||||
if audioDir == "" && len(cfg.Session.Inputs.AudioFiles) == 0 {
|
||||
return []finding{errorFinding("audio", "audio_dir, audio_files, or audio_s3 is required")}
|
||||
}
|
||||
base := filepath.Dir(cfg.SessionPath)
|
||||
paths := []string{}
|
||||
if audioDir != "" {
|
||||
dir := audioDir
|
||||
if !filepath.IsAbs(dir) {
|
||||
dir = filepath.Join(base, dir)
|
||||
}
|
||||
matches, err := filepath.Glob(filepath.Join(dir, "*.flac"))
|
||||
if err != nil || len(matches) == 0 {
|
||||
return []finding{errorFinding("audio", "no .flac files found in "+dir)}
|
||||
}
|
||||
paths = append(paths, matches...)
|
||||
}
|
||||
for _, file := range cfg.Session.Inputs.AudioFiles {
|
||||
p := file
|
||||
if !filepath.IsAbs(p) {
|
||||
p = filepath.Join(base, p)
|
||||
}
|
||||
paths = append(paths, p)
|
||||
}
|
||||
for _, p := range paths {
|
||||
if _, err := os.Stat(p); err != nil {
|
||||
return []finding{errorFinding("audio", fmt.Sprintf("audio file missing: %v", err))}
|
||||
}
|
||||
}
|
||||
return []finding{okFinding("audio", fmt.Sprintf("%d local audio file(s)", len(paths)))}
|
||||
}
|
||||
|
||||
func validateRemoteAudioFinding(ctx context.Context, cfg *config.Config, store storage.ObjectStore) finding {
|
||||
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
audioPrefix := artifacts.S3AudioPrefix(sessionPrefix, cfg.Session.Inputs.AudioS3.Prefix)
|
||||
objects, err := store.List(ctx, audioPrefix)
|
||||
if err != nil {
|
||||
return errorFinding("audio", err.Error())
|
||||
}
|
||||
count := 0
|
||||
for _, obj := range objects {
|
||||
if strings.HasSuffix(strings.ToLower(obj.Key), ".flac") {
|
||||
count++
|
||||
}
|
||||
}
|
||||
if count == 0 {
|
||||
return errorFinding("audio", "no remote .flac objects found under "+audioPrefix)
|
||||
}
|
||||
return okFinding("audio", fmt.Sprintf("%d remote .flac object(s)", count))
|
||||
}
|
||||
|
||||
func validatePreviousArtifactFindings(ctx context.Context, cfg *config.Config, store storage.ObjectStore, requirements []artifacts.PreviousArtifactRequirement) []finding {
|
||||
out := []finding{}
|
||||
prefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(prefix)
|
||||
for _, key := range []string{runIDKey, manifestKey} {
|
||||
exists, err := store.Exists(ctx, key)
|
||||
if err != nil {
|
||||
out = append(out, errorFinding("previous", fmt.Sprintf("check %s: %v", key, err)))
|
||||
return out
|
||||
}
|
||||
if !exists {
|
||||
out = append(out, errorFinding("previous", "missing "+key))
|
||||
return out
|
||||
}
|
||||
}
|
||||
for _, req := range requirements {
|
||||
out = append(out, okFinding("previous", fmt.Sprintf("%s required=%t", req.Name, req.Required)))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func loadLocalManifest(ctx context.Context, path string) (*manifest.Manifest, error) {
|
||||
if _, err := os.Stat(path); err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
store := &manifest.LocalStore{}
|
||||
return store.Load(ctx, path)
|
||||
}
|
||||
|
||||
func writeStageStatuses(out io.Writer, m *manifest.Manifest) {
|
||||
if m == nil || len(m.Stages) == 0 {
|
||||
fmt.Fprintln(out, "stages: no stages recorded")
|
||||
return
|
||||
}
|
||||
fmt.Fprintln(out, "stages:")
|
||||
names := make([]string, 0, len(m.Stages))
|
||||
for name := range m.Stages {
|
||||
names = append(names, name)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, name := range names {
|
||||
fmt.Fprintf(out, "- %s: %s\n", name, m.Stages[name].Status)
|
||||
}
|
||||
}
|
||||
|
||||
func buildHelperArtifactCatalog(cfg *config.Config) (*artifacts.ArtifactCatalog, error) {
|
||||
catalog := artifacts.NewArtifactCatalog()
|
||||
if err := catalog.RegisterBuiltIns(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
configured := map[string]artifacts.ConfiguredArtifactDefinition{}
|
||||
if cfg.Pipeline.Scriptorium != nil {
|
||||
for key, item := range cfg.Pipeline.Scriptorium.Artifacts {
|
||||
configured[key] = artifacts.ConfiguredArtifactDefinition{Enabled: item.Enabled, OutputPath: item.OutputPath}
|
||||
}
|
||||
}
|
||||
if err := catalog.RegisterConfiguredArtifacts(configured, nil); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return catalog, nil
|
||||
}
|
||||
|
||||
func writeArtifactList(out io.Writer, cfg *config.Config, catalog *artifacts.ArtifactCatalog, locks *effectiveLocks, promotedRemoteState map[string]string) {
|
||||
lockSet := lockSourceSet(locks.All)
|
||||
fmt.Fprintln(out, "Built-in:")
|
||||
for _, id := range []string{
|
||||
artifacts.ArtifactTranscriptMerged,
|
||||
artifacts.ArtifactTranscriptPolished,
|
||||
artifacts.ArtifactTranscriptFull,
|
||||
artifacts.ArtifactTranscriptTrimmed,
|
||||
artifacts.ArtifactBoundsSession,
|
||||
} {
|
||||
writeArtifactLine(out, id, lockSet)
|
||||
}
|
||||
fmt.Fprintln(out, "Configured:")
|
||||
for _, entry := range catalog.ListConfigured() {
|
||||
writeArtifactLine(out, entry.SourceID, lockSet)
|
||||
}
|
||||
fmt.Fprintln(out, "Previous-session:")
|
||||
for _, req := range artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg)) {
|
||||
fmt.Fprintf(out, "- narratio.previous_session.artifact.%s required=%t\n", req.Name, req.Required)
|
||||
}
|
||||
fmt.Fprintln(out, "Promoted:")
|
||||
for _, rule := range cfg.Pipeline.Archive.PromoteArtifacts {
|
||||
writePromotedArtifactLine(out, rule, catalog, lockSet, promotedRemoteState)
|
||||
}
|
||||
}
|
||||
|
||||
func writeArtifactLine(out io.Writer, source string, lockSet map[string]config.ArchiveLockRule) {
|
||||
parts := []string{source}
|
||||
if _, ok := lockSet[source]; ok {
|
||||
parts = append(parts, "locked")
|
||||
}
|
||||
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||
}
|
||||
|
||||
func writePromotedArtifactLine(out io.Writer, rule config.ArchivePromotionRule, catalog *artifacts.ArtifactCatalog, lockSet map[string]config.ArchiveLockRule, remoteState map[string]string) {
|
||||
source := strings.TrimSpace(rule.Source)
|
||||
parts := []string{source}
|
||||
if _, ok := lockSet[source]; ok {
|
||||
parts = append(parts, "locked")
|
||||
}
|
||||
dest, showDest, err := helperPromotionDest(rule, catalog)
|
||||
if err != nil {
|
||||
parts = append(parts, "remote=error")
|
||||
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||
return
|
||||
}
|
||||
if showDest {
|
||||
parts = append(parts, "dest="+dest)
|
||||
}
|
||||
if state := remoteState[promotionRemoteStateKey(source, dest)]; state != "" {
|
||||
parts = append(parts, state)
|
||||
}
|
||||
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||
}
|
||||
|
||||
func remotePromotionAvailability(ctx context.Context, cfg *config.Config, store storage.ObjectStore, catalog *artifacts.ArtifactCatalog) map[string]string {
|
||||
out := map[string]string{}
|
||||
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
for _, rule := range cfg.Pipeline.Archive.PromoteArtifacts {
|
||||
source := strings.TrimSpace(rule.Source)
|
||||
dest, _, err := helperPromotionDest(rule, catalog)
|
||||
if err != nil {
|
||||
out[promotionRemoteStateKey(source, "")] = "remote=error"
|
||||
continue
|
||||
}
|
||||
key := artifacts.S3PromotedArtifactKey(sessionPrefix, dest)
|
||||
if exists, err := store.Exists(ctx, key); err == nil && exists {
|
||||
out[promotionRemoteStateKey(source, dest)] = "remote=promoted"
|
||||
} else if err != nil {
|
||||
out[promotionRemoteStateKey(source, dest)] = "remote=error"
|
||||
} else {
|
||||
out[promotionRemoteStateKey(source, dest)] = "remote=missing"
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func helperPromotionDest(rule config.ArchivePromotionRule, catalog *artifacts.ArtifactCatalog) (string, bool, error) {
|
||||
source := strings.TrimSpace(rule.Source)
|
||||
dest := strings.TrimSpace(rule.Dest)
|
||||
if dest == "" {
|
||||
entry, ok := catalog.Lookup(source)
|
||||
if !ok {
|
||||
return "", false, fmt.Errorf("destination omitted and source is unknown")
|
||||
}
|
||||
dest = strings.TrimSpace(entry.CanonicalRelPath)
|
||||
if dest == "" {
|
||||
return "", false, fmt.Errorf("destination omitted and no canonical destination is available")
|
||||
}
|
||||
}
|
||||
normalized, err := normalizeHelperArchiveRelativePath(dest)
|
||||
if err != nil {
|
||||
return "", false, err
|
||||
}
|
||||
entry, ok := catalog.Lookup(source)
|
||||
showDest := !ok || strings.TrimSpace(entry.CanonicalRelPath) != normalized
|
||||
return normalized, showDest, nil
|
||||
}
|
||||
|
||||
func normalizeHelperArchiveRelativePath(rel string) (string, error) {
|
||||
trimmed := strings.TrimSpace(rel)
|
||||
if trimmed == "" {
|
||||
return "", fmt.Errorf("relative path is required")
|
||||
}
|
||||
cleaned := filepath.ToSlash(filepath.Clean(filepath.FromSlash(trimmed)))
|
||||
if cleaned == "." || cleaned == "" {
|
||||
return "", fmt.Errorf("relative path is required")
|
||||
}
|
||||
if filepath.IsAbs(trimmed) || strings.HasPrefix(cleaned, "/") || cleaned == ".." || strings.HasPrefix(cleaned, "../") {
|
||||
return "", fmt.Errorf("path must be a clean relative path")
|
||||
}
|
||||
return cleaned, nil
|
||||
}
|
||||
|
||||
func promotionRemoteStateKey(source, dest string) string {
|
||||
return strings.TrimSpace(source) + "\x00" + strings.TrimSpace(dest)
|
||||
}
|
||||
|
||||
func writeLocks(out io.Writer, cfg *config.Config, locks *effectiveLocks) {
|
||||
if locks == nil || len(locks.All) == 0 {
|
||||
fmt.Fprintln(out, "Archive locks: none")
|
||||
return
|
||||
}
|
||||
fmt.Fprintln(out, "Archive locks:")
|
||||
promoted := map[string]config.ArchivePromotionRule{}
|
||||
if cfg != nil && cfg.Pipeline != nil && cfg.Pipeline.Archive != nil {
|
||||
for _, rule := range cfg.Pipeline.Archive.PromoteArtifacts {
|
||||
promoted[strings.TrimSpace(rule.Source)] = rule
|
||||
}
|
||||
}
|
||||
staticSet := lockSourceSet(locks.Static)
|
||||
for _, lock := range locks.All {
|
||||
origin := "remote"
|
||||
if _, ok := staticSet[lock.Source]; ok {
|
||||
origin = "pipeline"
|
||||
}
|
||||
promo := "not-promoted"
|
||||
if _, ok := promoted[lock.Source]; ok {
|
||||
promo = "promoted"
|
||||
}
|
||||
reason := strings.TrimSpace(lock.Reason)
|
||||
if reason == "" {
|
||||
reason = "(no reason)"
|
||||
}
|
||||
fmt.Fprintf(out, "- %s origin=%s %s reason=%s\n", lock.Source, origin, promo, reason)
|
||||
}
|
||||
}
|
||||
|
||||
func lockMapValues(in map[string]config.ArchiveLockRule) []config.ArchiveLockRule {
|
||||
keys := make([]string, 0, len(in))
|
||||
for key := range in {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
out := make([]config.ArchiveLockRule, 0, len(keys))
|
||||
for _, key := range keys {
|
||||
item := in[key]
|
||||
item.Source = key
|
||||
item.Reason = strings.TrimSpace(item.Reason)
|
||||
out = append(out, item)
|
||||
}
|
||||
return out
|
||||
}
|
||||
589
internal/app/operator_helpers_test.go
Normal file
589
internal/app/operator_helpers_test.go
Normal file
@@ -0,0 +1,589 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
func TestExecuteSessionInitRemoteWritesCanonicalSessionConfig(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{filepath.Join(t.TempDir(), "session.yml")})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{
|
||||
"session", "init",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session-id", "2026-06-07",
|
||||
"--title", "The Black Cabin",
|
||||
"--remote",
|
||||
}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
key := artifacts.S3SessionConfigKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-06-07"))
|
||||
obj, ok := fake.Objects[key]
|
||||
if !ok {
|
||||
t.Fatalf("remote session key %q not uploaded; objects=%v", key, fake.Objects)
|
||||
}
|
||||
if !strings.Contains(string(obj.Data), `session_id: "2026-06-07"`) || !strings.Contains(string(obj.Data), "prefix: audio/") {
|
||||
t.Fatalf("remote session data = %q", string(obj.Data))
|
||||
}
|
||||
if storeInitCalls != 1 {
|
||||
t.Fatalf("object store init calls = %d, want 1", storeInitCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteSessionValidateLoadsSecretsBeforeObjectStoreInit(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
accessKeyEnv := "NARRATIO_TEST_VALIDATE_OBJECT_KEY_ID"
|
||||
secretKeyEnv := "NARRATIO_TEST_VALIDATE_OBJECT_SECRET"
|
||||
restoreEnvAfterTest(t, accessKeyEnv, secretKeyEnv)
|
||||
secretsDir := t.TempDir()
|
||||
mustWriteTestFile(t, filepath.Join(secretsDir, accessKeyEnv), "test-key-id\n")
|
||||
mustWriteTestFile(t, filepath.Join(secretsDir, secretKeyEnv), "test-secret\n")
|
||||
addSecretsToPipelineConfig(t, pipelinePath, secretsDir, accessKeyEnv, secretKeyEnv)
|
||||
if err := os.WriteFile(sessionPath, []byte(`session_id: 2026-05-03
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
`), 0o644); err != nil {
|
||||
t.Fatalf("write session: %v", err)
|
||||
}
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
audioKey := artifacts.S3PromotedArtifactKey(artifacts.S3AudioPrefix(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"), "audio/"), "alice.flac")
|
||||
fake.SeedObject(storage.FakeObject{Key: audioKey, Data: []byte("audio")})
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
if os.Getenv(accessKeyEnv) != "test-key-id" || os.Getenv(secretKeyEnv) != "test-secret" {
|
||||
return nil, fmt.Errorf("secrets were not loaded before object store init")
|
||||
}
|
||||
return fake, nil
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"session", "validate", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "OK audio") {
|
||||
t.Fatalf("stdout = %q, want OK audio", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteLocksAddListAndRemoveUseRemoteLockStore(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{
|
||||
"locks", "add",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", "2026-05-03",
|
||||
"--reason", "manual edit",
|
||||
"narratio.transcript.trimmed",
|
||||
}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("locks add exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
key := artifacts.S3SessionLocksKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"))
|
||||
obj, ok := fake.Objects[key]
|
||||
if !ok {
|
||||
t.Fatalf("remote locks key %q not uploaded", key)
|
||||
}
|
||||
if !strings.Contains(string(obj.Data), "source: narratio.transcript.trimmed") || !strings.Contains(string(obj.Data), "reason: manual edit") {
|
||||
t.Fatalf("lock store data = %q", string(obj.Data))
|
||||
}
|
||||
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
code = Execute([]string{
|
||||
"locks",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", "2026-05-03",
|
||||
}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("locks list exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "- narratio.transcript.trimmed origin=remote") {
|
||||
t.Fatalf("stdout = %q, want remote lock", stdout.String())
|
||||
}
|
||||
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
code = Execute([]string{
|
||||
"locks", "remove",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", "2026-05-03",
|
||||
"narratio.transcript.trimmed",
|
||||
}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("locks remove exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
store, err := config.LoadArchiveLockStoreBytes("locks.yml", fake.Objects[key].Data, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadArchiveLockStoreBytes() error = %v", err)
|
||||
}
|
||||
if len(store.Locks) != 0 {
|
||||
t.Fatalf("locks after remove = %#v, want empty", store.Locks)
|
||||
}
|
||||
if storeInitCalls != 3 {
|
||||
t.Fatalf("object store init calls = %d, want 3", storeInitCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteLocksAddDuplicateRequiresForce(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{
|
||||
"locks", "add",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", "2026-05-03",
|
||||
"--reason", "first",
|
||||
"narratio.transcript.trimmed",
|
||||
}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("initial locks add exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
code = Execute([]string{
|
||||
"locks", "add",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", "2026-05-03",
|
||||
"--reason", "second",
|
||||
"narratio.transcript.trimmed",
|
||||
}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("duplicate locks add exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "pass --force to update") {
|
||||
t.Fatalf("stderr = %q, want force guidance", stderr.String())
|
||||
}
|
||||
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
code = Execute([]string{
|
||||
"locks", "add",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", "2026-05-03",
|
||||
"--reason", "second",
|
||||
"--force",
|
||||
"narratio.transcript.trimmed",
|
||||
}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("forced locks add exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
key := artifacts.S3SessionLocksKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"))
|
||||
if !strings.Contains(string(fake.Objects[key].Data), "reason: second") {
|
||||
t.Fatalf("lock store data = %q, want updated reason", string(fake.Objects[key].Data))
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteLocksRequireSessionID(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
args []string
|
||||
want string
|
||||
}{
|
||||
{"list", []string{"locks"}, "locks: --session-id is required"},
|
||||
{"add", []string{"locks", "add", "narratio.transcript.trimmed"}, "locks add: --session-id is required"},
|
||||
{"remove", []string{"locks", "remove", "narratio.transcript.trimmed"}, "locks remove: --session-id is required"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(tt.args, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), tt.want) {
|
||||
t.Fatalf("stderr = %q, want %q", stderr.String(), tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteLocksCannotModifyStaticLocks(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
addStaticArchiveLockToPipelineConfig(t, pipelinePath, "narratio.transcript.trimmed")
|
||||
fake := &storage.FakeBackend{}
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{
|
||||
"locks", "add",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", "2026-05-03",
|
||||
"narratio.transcript.trimmed",
|
||||
}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("locks add static lock exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "locked by pipeline config") {
|
||||
t.Fatalf("stderr = %q, want static lock error", stderr.String())
|
||||
}
|
||||
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
code = Execute([]string{
|
||||
"locks", "remove",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", "2026-05-03",
|
||||
"narratio.transcript.trimmed",
|
||||
}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("locks remove static lock exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "locked by pipeline config") {
|
||||
t.Fatalf("stderr = %q, want static lock error", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteTopLevelLockAndUnlockAreRemoved(t *testing.T) {
|
||||
tests := []string{"lock", "unlock"}
|
||||
for _, cmd := range tests {
|
||||
t.Run(cmd, func(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{cmd, "narratio.transcript.trimmed"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `unknown command: "`+cmd+`"`) {
|
||||
t.Fatalf("stderr = %q, want unknown command", stderr.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteArtifactsListRemoteReportsPromotedAvailability(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
||||
promote_artifacts:
|
||||
- source: narratio.transcript.trimmed
|
||||
dest: transcripts/trimmed.json
|
||||
required: true
|
||||
`)
|
||||
fake := &storage.FakeBackend{}
|
||||
trimmedKey := artifacts.S3PromotedArtifactKey(
|
||||
artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"),
|
||||
"transcripts/trimmed.json",
|
||||
)
|
||||
fake.SeedObject(storage.FakeObject{Key: trimmedKey, Data: []byte(`{"segments":[]}`)})
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{
|
||||
"artifacts", "list",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--remote",
|
||||
}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "narratio.transcript.trimmed remote=promoted") {
|
||||
t.Fatalf("stdout = %q, want promoted remote availability", stdout.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteArtifactsListRemoteUsesPromotionDestinations(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
||||
promote_artifacts:
|
||||
- source: narratio.transcript.full
|
||||
dest: transcripts/full.json
|
||||
required: true
|
||||
- source: narratio.bounds.session
|
||||
dest: transcripts/bounds.json
|
||||
required: true
|
||||
`)
|
||||
fake := &storage.FakeBackend{}
|
||||
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03")
|
||||
fake.SeedObject(storage.FakeObject{Key: artifacts.S3PromotedArtifactKey(sessionPrefix, "transcripts/full.json"), Data: []byte(`{"segments":[]}`)})
|
||||
fake.SeedObject(storage.FakeObject{Key: artifacts.S3PromotedArtifactKey(sessionPrefix, "transcripts/bounds.json"), Data: []byte(`{}`)})
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{
|
||||
"artifacts", "list",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--remote",
|
||||
}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
out := stdout.String()
|
||||
for _, unwanted := range []string{
|
||||
"narratio.transcript.full remote=missing",
|
||||
"narratio.bounds.session remote=missing",
|
||||
} {
|
||||
if strings.Contains(out, unwanted) {
|
||||
t.Fatalf("stdout = %q, did not want catalog remote marker %q", out, unwanted)
|
||||
}
|
||||
}
|
||||
for _, want := range []string{
|
||||
"narratio.transcript.full dest=transcripts/full.json remote=promoted",
|
||||
"narratio.bounds.session dest=transcripts/bounds.json remote=promoted",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Fatalf("stdout = %q, want %q", out, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
||||
promote_artifacts:
|
||||
- source: narratio.transcript.trimmed
|
||||
dest: transcripts/trimmed.json
|
||||
required: true
|
||||
- source: narratio.transcript.full
|
||||
dest: transcripts/full.json
|
||||
required: true
|
||||
`)
|
||||
fake := &storage.FakeBackend{}
|
||||
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03")
|
||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
||||
trimmedKey := artifacts.S3PromotedArtifactKey(sessionPrefix, "transcripts/trimmed.json")
|
||||
fullKey := artifacts.S3PromotedArtifactKey(sessionPrefix, "transcripts/full.json")
|
||||
lockKey := artifacts.S3SessionLocksKey(sessionPrefix)
|
||||
fake.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||
fake.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "2026-05-03", "sample-campaign")})
|
||||
fake.SeedObject(storage.FakeObject{Key: trimmedKey, Data: []byte(`{"segments":[]}`)})
|
||||
fake.SeedObject(storage.FakeObject{Key: fullKey, Data: []byte(`{"segments":[]}`)})
|
||||
fake.SeedObject(storage.FakeObject{Key: lockKey, Data: []byte("locks:\n - source: narratio.transcript.trimmed\n reason: remote review\n")})
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{
|
||||
"status",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", "2026-05-03",
|
||||
}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
out := stdout.String()
|
||||
for _, want := range []string{
|
||||
"Remote outputs:",
|
||||
"Built-in:",
|
||||
"Configured:",
|
||||
"Previous-session:",
|
||||
"Promoted:",
|
||||
"narratio.transcript.trimmed locked",
|
||||
"narratio.transcript.trimmed locked remote=promoted",
|
||||
"narratio.transcript.full dest=transcripts/full.json remote=promoted",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Fatalf("stdout = %q, want %q", out, want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(out, "narratio.transcript.merged remote=missing") {
|
||||
t.Fatalf("stdout = %q, did not want catalog remote marker", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStatusReportsRemoteArtifactCatalogErrorsWithoutFailing(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
||||
promote_artifacts:
|
||||
- source: narratio.transcript.trimmed
|
||||
dest: transcripts/trimmed.json
|
||||
required: true
|
||||
`)
|
||||
fake := &storage.FakeBackend{ExistsErr: fmt.Errorf("exists failed")}
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{
|
||||
"status",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", "2026-05-03",
|
||||
}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
out := stdout.String()
|
||||
if !strings.Contains(out, "Remote archive: missing or unavailable:") {
|
||||
t.Fatalf("stdout = %q, want remote archive unavailable state", out)
|
||||
}
|
||||
if !strings.Contains(out, "Remote outputs:") || !strings.Contains(out, "narratio.transcript.trimmed remote=error") {
|
||||
t.Fatalf("stdout = %q, want remote output error state", out)
|
||||
}
|
||||
if !strings.Contains(out, "Archive locks: error:") {
|
||||
t.Fatalf("stdout = %q, want archive locks error", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteArchiveLoadsRemoteLocks(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidArchiveConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
lockKey := artifacts.S3SessionLocksKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"))
|
||||
fake.SeedObject(storage.FakeObject{Key: lockKey, Data: []byte("locks:\n - source: narratio.transcript.trimmed\n reason: remote review\n")})
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
||||
// The archive stage only checks the manifest statuses and source files.
|
||||
_ = stageName
|
||||
}
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"run-stage", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--force", "archive"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
promotedKey := artifacts.S3PromotedArtifactKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"), "transcripts/trimmed.json")
|
||||
if _, ok := fake.Objects[promotedKey]; ok {
|
||||
t.Fatalf("locked promoted key %q was uploaded", promotedKey)
|
||||
}
|
||||
}
|
||||
|
||||
func addArchivePromotionsToPipeline(t *testing.T, pipelinePath, archiveYAML string) {
|
||||
t.Helper()
|
||||
data, err := os.ReadFile(pipelinePath)
|
||||
if err != nil {
|
||||
t.Fatalf("read pipeline: %v", err)
|
||||
}
|
||||
updated := strings.Replace(string(data), " upload_run: false\n", " upload_run: false\n"+archiveYAML, 1)
|
||||
if updated == string(data) {
|
||||
t.Fatalf("pipeline %q did not contain archive upload_run marker", pipelinePath)
|
||||
}
|
||||
if err := os.WriteFile(pipelinePath, []byte(updated), 0o644); err != nil {
|
||||
t.Fatalf("write pipeline: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func writeValidArchiveConfigFiles(t *testing.T, workspaceRoot string) (string, string, string) {
|
||||
t.Helper()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
data, err := os.ReadFile(pipelinePath)
|
||||
if err != nil {
|
||||
t.Fatalf("read pipeline: %v", err)
|
||||
}
|
||||
updated := strings.Replace(string(data), "upload_run: false", "upload_run: true", 1)
|
||||
if err := os.WriteFile(pipelinePath, []byte(updated), 0o644); err != nil {
|
||||
t.Fatalf("write pipeline: %v", err)
|
||||
}
|
||||
ctx := context.Background()
|
||||
cfg, err := config.LoadWithSessionOptions(pipelinePath, campaignPath, sessionPath, config.SessionLoadOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("LoadWithSessionOptions() error = %v", err)
|
||||
}
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", nowUTC())
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260521T160000Z-test"
|
||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
||||
m.MarkStageSucceeded(name, nowUTC(), nil)
|
||||
}
|
||||
path := artifacts.SessionManifestPathForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err := store.Save(ctx, path, m); err != nil {
|
||||
t.Fatalf("save manifest: %v", err)
|
||||
}
|
||||
runManifestPath := artifacts.SessionRunManifestPathForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, m.RunID)
|
||||
if err := os.MkdirAll(filepath.Dir(runManifestPath), 0o755); err != nil {
|
||||
t.Fatalf("mkdir run manifest: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(runManifestPath, []byte("{}\n"), 0o644); err != nil {
|
||||
t.Fatalf("write run manifest: %v", err)
|
||||
}
|
||||
return pipelinePath, campaignPath, sessionPath
|
||||
}
|
||||
|
||||
func addStaticArchiveLockToPipelineConfig(t *testing.T, pipelinePath, source string) {
|
||||
t.Helper()
|
||||
data, err := os.ReadFile(pipelinePath)
|
||||
if err != nil {
|
||||
t.Fatalf("read pipeline: %v", err)
|
||||
}
|
||||
updated := strings.Replace(
|
||||
string(data),
|
||||
"archive:\n enabled: true\n upload_run: false\n",
|
||||
"archive:\n enabled: true\n upload_run: false\n locks:\n - source: "+source+"\n reason: static review\n",
|
||||
1,
|
||||
)
|
||||
if updated == string(data) {
|
||||
t.Fatalf("archive section not found in pipeline config")
|
||||
}
|
||||
if err := os.WriteFile(pipelinePath, []byte(updated), 0o644); err != nil {
|
||||
t.Fatalf("write pipeline: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -20,12 +20,16 @@ func Plan(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs.SetOutput(io.Discard)
|
||||
|
||||
var pipelinePath string
|
||||
var campaignPath string
|
||||
var sessionPath string
|
||||
var sessionID string
|
||||
var previousSessionID string
|
||||
var force bool
|
||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||
fs.StringVar(&campaignPath, "campaign", "", "path to campaign.yml (optional; defaults searched)")
|
||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
||||
fs.StringVar(&sessionID, "session-id", "", "session identifier for session.yml templates")
|
||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "previous session identifier for session.yml templates")
|
||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
||||
|
||||
if err := fs.Parse(args); err != nil {
|
||||
@@ -34,17 +38,9 @@ func Plan(ctx context.Context, args []string, out io.Writer) error {
|
||||
if fs.NArg() != 0 {
|
||||
return fmt.Errorf("plan: unexpected positional arguments")
|
||||
}
|
||||
resolvedPipelinePath, err := resolvePipelineConfigPath(pipelinePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("plan: %w", err)
|
||||
}
|
||||
resolvedSessionPath, err := resolveSessionConfigPath(sessionPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("plan: %w", err)
|
||||
}
|
||||
|
||||
cfg, err := config.LoadWithSessionOptions(resolvedPipelinePath, resolvedSessionPath, config.SessionLoadOptions{
|
||||
SessionID: sessionID,
|
||||
cfg, err := loadCommandConfig(ctx, pipelinePath, campaignPath, sessionPath, config.SessionLoadOptions{
|
||||
SessionID: sessionID,
|
||||
PreviousSessionID: previousSessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("plan: %w", err)
|
||||
@@ -57,7 +53,7 @@ func Plan(ctx context.Context, args []string, out io.Writer) error {
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
|
||||
paths, err := store.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, err := store.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("plan: prepare workdir: %w", err)
|
||||
}
|
||||
|
||||
@@ -15,10 +15,10 @@ import (
|
||||
|
||||
func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
var out bytes.Buffer
|
||||
args := []string{"--config", pipelinePath, "--session", sessionPath}
|
||||
args := []string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}
|
||||
|
||||
if err := Plan(context.Background(), args, &out); err != nil {
|
||||
t.Fatalf("first Plan() error = %v", err)
|
||||
@@ -36,7 +36,7 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
|
||||
t.Fatalf("first output = %q, want totals", got)
|
||||
}
|
||||
|
||||
sessionWorkdir := artifacts.SessionWorkDir(workspaceRoot, "2026-05-03")
|
||||
sessionWorkdir := artifacts.SessionWorkDirForCampaign(workspaceRoot, "sample-campaign", "2026-05-03")
|
||||
expectedDirs := []string{
|
||||
sessionWorkdir,
|
||||
filepath.Join(sessionWorkdir, "inputs"),
|
||||
@@ -62,8 +62,8 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
|
||||
|
||||
func TestPlanShowsRunAndSkipFromManifest(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
@@ -74,7 +74,7 @@ func TestPlanShowsRunAndSkipFromManifest(t *testing.T) {
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
if err := Plan(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath}, &out); err != nil {
|
||||
if err := Plan(context.Background(), []string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, &out); err != nil {
|
||||
t.Fatalf("Plan() error = %v", err)
|
||||
}
|
||||
got := out.String()
|
||||
@@ -93,6 +93,7 @@ func TestPlanFailsWhenConfiguredSecretsDirMissing(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
configDir := t.TempDir()
|
||||
pipelinePath := filepath.Join(configDir, "pipeline.yml")
|
||||
campaignPath := writeAppTestCampaignConfig(t, configDir)
|
||||
sessionPath := filepath.Join(configDir, "session.yml")
|
||||
|
||||
pipelineYAML := `workspace:
|
||||
@@ -128,7 +129,7 @@ inputs:
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
err := Plan(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath}, &out)
|
||||
err := Plan(context.Background(), []string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, &out)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
|
||||
@@ -54,7 +54,7 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
||||
}
|
||||
workDir := strings.TrimSpace(m.LocalWorkDir)
|
||||
if workDir == "" {
|
||||
workDir = artifacts.SessionRunWorkDir(
|
||||
workDir = artifacts.SessionRunRootForCampaign(
|
||||
env.Config.Pipeline.Workspace.Root,
|
||||
strings.TrimSpace(env.Config.Session.Campaign),
|
||||
strings.TrimSpace(env.Config.Session.SessionID),
|
||||
@@ -153,53 +153,70 @@ func archiveCleanupEligible(cfg *config.Config, sr *manifest.StageRecord) (bool,
|
||||
return true, ""
|
||||
}
|
||||
|
||||
type scopedDir struct {
|
||||
RootAbs string
|
||||
TargetAbs string
|
||||
Exists bool
|
||||
}
|
||||
|
||||
func removeRunScopedDir(root, target, policy string) error {
|
||||
dir, err := validateScopedDir(root, target, policy)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !dir.Exists {
|
||||
return nil
|
||||
}
|
||||
if err := os.RemoveAll(dir.TargetAbs); err != nil {
|
||||
return fmt.Errorf("cleanup policy %s: remove %q: %w", policy, dir.TargetAbs, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateScopedDir(root, target, policy string) (scopedDir, error) {
|
||||
cleanRoot := strings.TrimSpace(root)
|
||||
cleanTarget := strings.TrimSpace(target)
|
||||
if cleanRoot == "" {
|
||||
return fmt.Errorf("cleanup policy %s: root path is required", policy)
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
||||
}
|
||||
if cleanTarget == "" {
|
||||
return fmt.Errorf("cleanup policy %s: target path is required", policy)
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target path is required", policy)
|
||||
}
|
||||
|
||||
rootAbs, err := filepath.Abs(cleanRoot)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
||||
}
|
||||
targetAbs, err := filepath.Abs(cleanTarget)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cleanup policy %s: resolve target %q: %w", policy, cleanTarget, err)
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve target %q: %w", policy, cleanTarget, err)
|
||||
}
|
||||
|
||||
rel, err := filepath.Rel(rootAbs, targetAbs)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cleanup policy %s: relative path from %q to %q: %w", policy, rootAbs, targetAbs, err)
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: relative path from %q to %q: %w", policy, rootAbs, targetAbs, err)
|
||||
}
|
||||
if rel == "." {
|
||||
return fmt.Errorf("cleanup policy %s: refusing to delete root directory %q", policy, rootAbs)
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete root directory %q", policy, rootAbs)
|
||||
}
|
||||
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
||||
return fmt.Errorf("cleanup policy %s: refusing to delete path outside root: root=%q target=%q", policy, rootAbs, targetAbs)
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete path outside root: root=%q target=%q", policy, rootAbs, targetAbs)
|
||||
}
|
||||
|
||||
info, err := os.Lstat(targetAbs)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return nil
|
||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: false}, nil
|
||||
}
|
||||
return fmt.Errorf("cleanup policy %s: stat target %q: %w", policy, targetAbs, err)
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: stat target %q: %w", policy, targetAbs, err)
|
||||
}
|
||||
if info.Mode()&os.ModeSymlink != 0 {
|
||||
return fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, targetAbs)
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, targetAbs)
|
||||
}
|
||||
if !info.IsDir() {
|
||||
return fmt.Errorf("cleanup policy %s: target %q is not a directory", policy, targetAbs)
|
||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is not a directory", policy, targetAbs)
|
||||
}
|
||||
if err := os.RemoveAll(targetAbs); err != nil {
|
||||
return fmt.Errorf("cleanup policy %s: remove %q: %w", policy, targetAbs, err)
|
||||
}
|
||||
return nil
|
||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: true}, nil
|
||||
}
|
||||
|
||||
func asString(v any) string {
|
||||
|
||||
@@ -82,6 +82,7 @@ func TestPostArchiveCleanupWorkdirOnly(t *testing.T) {
|
||||
|
||||
assertExists(t, cfg.Pipeline.Workspace.Root)
|
||||
assertExists(t, seed.otherRunDir)
|
||||
assertExists(t, seed.previousCachePath)
|
||||
assertMissing(t, seed.runWorkDir)
|
||||
assertExists(t, seed.spoolAudioDir)
|
||||
}
|
||||
@@ -98,6 +99,7 @@ func TestPostArchiveCleanupBothPolicies(t *testing.T) {
|
||||
assertMissing(t, seed.spoolAudioDir)
|
||||
assertMissing(t, seed.runWorkDir)
|
||||
assertExists(t, seed.otherRunDir)
|
||||
assertExists(t, seed.previousCachePath)
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupNotRunWhenArchiveFails(t *testing.T) {
|
||||
@@ -195,7 +197,7 @@ func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
|
||||
cfg.Pipeline.Spool.DeleteAudioAfterArchive = true
|
||||
cfg.Pipeline.Workspace.CleanupAfterArchive = true
|
||||
cfg.Pipeline.Archive.PromoteArtifacts = []config.ArchivePromotionRule{
|
||||
{From: "artifacts/missing.md", To: "artifacts/missing.md", Required: boolPtr(true)},
|
||||
{Source: "narratio.transcript.merged", Dest: "transcripts/merged.json", Required: boolPtr(true)},
|
||||
}
|
||||
|
||||
archiveStageImpl, err := stage.Select("archive")
|
||||
@@ -203,14 +205,14 @@ func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
|
||||
t.Fatalf("Select(archive) error = %v", err)
|
||||
}
|
||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveStageImpl}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||
if err == nil || !strings.Contains(err.Error(), "required promotion source missing") {
|
||||
if err == nil || !strings.Contains(err.Error(), "required promotion source unavailable") {
|
||||
t.Fatalf("executeStages() error = %v, want promotion-missing failure", err)
|
||||
}
|
||||
|
||||
assertExists(t, seed.spoolAudioDir)
|
||||
assertExists(t, seed.runWorkDir)
|
||||
assertExists(t, filepath.Join(seed.runWorkDir, "manifest.json"))
|
||||
assertExists(t, artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
|
||||
assertExists(t, artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
||||
@@ -256,11 +258,12 @@ func TestPostArchiveCleanupNotRunWhenCurrentPointerUploadFails(t *testing.T) {
|
||||
}
|
||||
|
||||
type cleanupSeed struct {
|
||||
runWorkDir string
|
||||
otherRunDir string
|
||||
spoolAudioDir string
|
||||
localSourceAudio string
|
||||
sessionPrefix string
|
||||
runWorkDir string
|
||||
otherRunDir string
|
||||
spoolAudioDir string
|
||||
localSourceAudio string
|
||||
previousCachePath string
|
||||
sessionPrefix string
|
||||
}
|
||||
|
||||
func cleanupFixtureConfig(t *testing.T) (*config.Config, cleanupSeed) {
|
||||
@@ -271,14 +274,21 @@ func cleanupFixtureConfig(t *testing.T) (*config.Config, cleanupSeed) {
|
||||
cfg.Pipeline.Spool.Root = filepath.Join(t.TempDir(), "spool")
|
||||
|
||||
runID := "20260516T010203Z-1a2b3c4d"
|
||||
runWorkDir := artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||
otherRunDir := artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, "20260516T010204Z-5e6f7a8b")
|
||||
runWorkDir := artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||
otherRunDir := artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, "20260516T010204Z-5e6f7a8b")
|
||||
spoolAudioDir := artifacts.SessionSpoolAudioDir(cfg.Pipeline.Spool.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||
previousCachePath := artifacts.SessionPreviousArtifactPathForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
"session_recap.md",
|
||||
)
|
||||
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "manifest.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "logs", "stage.log"), "log\n")
|
||||
mustWriteFile(t, filepath.Join(otherRunDir, "logs", "stage.log"), "other\n")
|
||||
mustWriteFile(t, filepath.Join(spoolAudioDir, "speaker.flac"), "flac\n")
|
||||
mustWriteFile(t, previousCachePath, "# previous recap\n")
|
||||
|
||||
localSourceAudio := filepath.Join(filepath.Dir(cfg.SessionPath), "audio", "alice.flac")
|
||||
mustWriteFile(t, localSourceAudio, "source\n")
|
||||
@@ -301,11 +311,12 @@ func cleanupFixtureConfig(t *testing.T) (*config.Config, cleanupSeed) {
|
||||
}
|
||||
|
||||
return cfg, cleanupSeed{
|
||||
runWorkDir: runWorkDir,
|
||||
otherRunDir: otherRunDir,
|
||||
spoolAudioDir: spoolAudioDir,
|
||||
localSourceAudio: localSourceAudio,
|
||||
sessionPrefix: seed.S3SessionPrefix,
|
||||
runWorkDir: runWorkDir,
|
||||
otherRunDir: otherRunDir,
|
||||
spoolAudioDir: spoolAudioDir,
|
||||
localSourceAudio: localSourceAudio,
|
||||
previousCachePath: previousCachePath,
|
||||
sessionPrefix: seed.S3SessionPrefix,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -322,11 +333,22 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
||||
Enabled: boolPtr(true),
|
||||
UploadRun: boolPtr(true),
|
||||
PromoteArtifacts: []config.ArchivePromotionRule{
|
||||
{From: "transcripts/trimmed.json", To: "transcripts/trimmed.json", Required: boolPtr(true)},
|
||||
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md", Required: boolPtr(true)},
|
||||
{Source: "narratio.transcript.trimmed", Dest: "transcripts/trimmed.json", Required: boolPtr(true)},
|
||||
{Source: "narratio.artifact.session_recap", Dest: "artifacts/session_recap.md", Required: boolPtr(true)},
|
||||
},
|
||||
}
|
||||
writeArchiveFixtureRunFiles(t, seed.runWorkDir)
|
||||
cfg.Pipeline.Scriptorium = &config.ScriptoriumConfig{
|
||||
Artifacts: map[string]config.ScriptoriumArtifactConfig{
|
||||
"session_recap": {
|
||||
OutputPath: "artifacts/session_recap.md",
|
||||
},
|
||||
},
|
||||
}
|
||||
writeArchiveFixtureRunFiles(
|
||||
t,
|
||||
seed.runWorkDir,
|
||||
artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID),
|
||||
)
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
seedManifest, err := store.Load(context.Background(), manifestPathFor(cfg))
|
||||
@@ -345,16 +367,19 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
||||
return cfg, seed, runID
|
||||
}
|
||||
|
||||
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir string) {
|
||||
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir, sessionRoot string) {
|
||||
t.Helper()
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcripts", "trimmed.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "artifacts", "session_recap.md"), "# recap\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "reports", "audita.report.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "config", "audita.generated.yml"), "key: value\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "prepare", "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "trim", "outputs", "transcripts", "trimmed.json"), "{\"segments\":[]}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "analyze", "outputs", "artifacts", "session_recap.md"), "# recap\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "polish", "reports", "audita.report.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "merge", "config", "seriatim.generated.yml"), "key: value\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "logs", "audita.stderr.log"), "stderr\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "manifest.json"), "{}\n")
|
||||
|
||||
mustWriteFile(t, filepath.Join(sessionRoot, "transcripts", "trimmed.json"), "{\"segments\":[]}\n")
|
||||
mustWriteFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# recap\n")
|
||||
}
|
||||
|
||||
type failKeyStore struct {
|
||||
|
||||
157
internal/app/remote_locks.go
Normal file
157
internal/app/remote_locks.go
Normal file
@@ -0,0 +1,157 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
type effectiveLocks struct {
|
||||
Static []config.ArchiveLockRule
|
||||
Remote []config.ArchiveLockRule
|
||||
All []config.ArchiveLockRule
|
||||
Key string
|
||||
}
|
||||
|
||||
func remoteLocksKey(cfg *config.Config) (string, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
||||
return "", fmt.Errorf("resolved config is required")
|
||||
}
|
||||
if cfg.Pipeline.Storage.S3 == nil {
|
||||
return "", fmt.Errorf("pipeline.storage.s3 configuration is required")
|
||||
}
|
||||
sessionPrefix := artifacts.S3SessionPrefix(
|
||||
cfg.Pipeline.Storage.S3.RootPrefix,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
)
|
||||
return artifacts.S3SessionLocksKey(sessionPrefix), nil
|
||||
}
|
||||
|
||||
func loadRemoteLockStore(ctx context.Context, cfg *config.Config, store storage.ObjectStore) (*config.ArchiveLockStore, string, error) {
|
||||
key, err := remoteLocksKey(cfg)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
exists, err := store.Exists(ctx, key)
|
||||
if err != nil {
|
||||
return nil, key, fmt.Errorf("check remote locks %q: %w", key, err)
|
||||
}
|
||||
if !exists {
|
||||
return &config.ArchiveLockStore{}, key, nil
|
||||
}
|
||||
tmp, err := downloadObjectToTemp(ctx, store, key, "narratio-locks-*.yml")
|
||||
if err != nil {
|
||||
return nil, key, fmt.Errorf("download remote locks %q: %w", key, err)
|
||||
}
|
||||
defer func() { _ = os.Remove(tmp) }()
|
||||
data, err := os.ReadFile(tmp)
|
||||
if err != nil {
|
||||
return nil, key, fmt.Errorf("read remote locks %q: %w", key, err)
|
||||
}
|
||||
lockStore, err := config.LoadArchiveLockStoreBytes("s3://"+s3BucketName(cfg.Pipeline)+"/"+key, data, cfg.Pipeline.Scriptorium)
|
||||
if err != nil {
|
||||
return nil, key, err
|
||||
}
|
||||
return lockStore, key, nil
|
||||
}
|
||||
|
||||
func loadEffectiveLocks(ctx context.Context, cfg *config.Config, store storage.ObjectStore) (*effectiveLocks, error) {
|
||||
staticLocks := staticArchiveLocks(cfg)
|
||||
if store == nil {
|
||||
return &effectiveLocks{
|
||||
Static: staticLocks,
|
||||
All: append([]config.ArchiveLockRule(nil), staticLocks...),
|
||||
}, nil
|
||||
}
|
||||
lockStore, key, err := loadRemoteLockStore(ctx, cfg, store)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
remoteLocks := append([]config.ArchiveLockRule(nil), lockStore.Locks...)
|
||||
return &effectiveLocks{
|
||||
Static: staticLocks,
|
||||
Remote: remoteLocks,
|
||||
All: config.MergeArchiveLockRules(staticLocks, remoteLocks),
|
||||
Key: key,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func staticArchiveLocks(cfg *config.Config) []config.ArchiveLockRule {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Archive == nil {
|
||||
return nil
|
||||
}
|
||||
return append([]config.ArchiveLockRule(nil), cfg.Pipeline.Archive.Locks...)
|
||||
}
|
||||
|
||||
func applyEffectiveLocks(cfg *config.Config, locks []config.ArchiveLockRule) {
|
||||
if cfg == nil || cfg.Pipeline == nil {
|
||||
return
|
||||
}
|
||||
if cfg.Pipeline.Archive == nil {
|
||||
cfg.Pipeline.Archive = &config.ArchiveConfig{}
|
||||
}
|
||||
cfg.Pipeline.Archive.Locks = append([]config.ArchiveLockRule(nil), locks...)
|
||||
}
|
||||
|
||||
func uploadRemoteLockStore(ctx context.Context, store storage.ObjectStore, key string, lockStore *config.ArchiveLockStore) error {
|
||||
data, err := config.MarshalArchiveLockStore(lockStore)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmp, err := os.CreateTemp("", "narratio-locks-upload-*.yml")
|
||||
if err != nil {
|
||||
return fmt.Errorf("create lock store temp file: %w", err)
|
||||
}
|
||||
tmpPath := tmp.Name()
|
||||
defer func() { _ = os.Remove(tmpPath) }()
|
||||
if _, err := tmp.Write(data); err != nil {
|
||||
_ = tmp.Close()
|
||||
return fmt.Errorf("write lock store temp file: %w", err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return fmt.Errorf("close lock store temp file: %w", err)
|
||||
}
|
||||
if _, err := store.Upload(ctx, tmpPath, key, storage.UploadOptions{ContentType: "application/x-yaml; charset=utf-8"}); err != nil {
|
||||
return fmt.Errorf("upload remote locks %q: %w", key, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func lockSourceSet(locks []config.ArchiveLockRule) map[string]config.ArchiveLockRule {
|
||||
out := make(map[string]config.ArchiveLockRule, len(locks))
|
||||
for _, lock := range locks {
|
||||
source := strings.TrimSpace(lock.Source)
|
||||
if source == "" {
|
||||
continue
|
||||
}
|
||||
lock.Source = source
|
||||
lock.Reason = strings.TrimSpace(lock.Reason)
|
||||
out[source] = lock
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func writeLocalFile(path string, data []byte, force bool) error {
|
||||
cleaned := filepath.Clean(strings.TrimSpace(path))
|
||||
if cleaned == "" || cleaned == "." {
|
||||
return fmt.Errorf("output path is required")
|
||||
}
|
||||
if !force {
|
||||
if _, err := os.Stat(cleaned); err == nil {
|
||||
return fmt.Errorf("output file %q already exists; pass --force to overwrite", cleaned)
|
||||
} else if err != nil && !os.IsNotExist(err) {
|
||||
return fmt.Errorf("check output file %q: %w", cleaned, err)
|
||||
}
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(cleaned), 0o755); err != nil {
|
||||
return fmt.Errorf("create output directory: %w", err)
|
||||
}
|
||||
return os.WriteFile(cleaned, data, 0o644)
|
||||
}
|
||||
252
internal/app/remote_session_test.go
Normal file
252
internal/app/remote_session_test.go
Normal file
@@ -0,0 +1,252 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func TestExecuteRemoteSessionFallbackLoadsFromObjectStore(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
remoteKey := seedRemoteSessionConfig(t, fake, "2026-05-03", `session_id: "{{ session_id }}"
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
`)
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{filepath.Join(t.TempDir(), "session.yml")})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"plan", "--config", pipelinePath, "--campaign", campaignPath, "--session-id", "2026-05-03"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if storeInitCalls != 1 {
|
||||
t.Fatalf("object store init calls = %d, want 1", storeInitCalls)
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "narratio plan: workdir prepared") {
|
||||
t.Fatalf("stdout = %q, want plan output", stdout.String())
|
||||
}
|
||||
if _, ok := fake.Objects[remoteKey]; !ok {
|
||||
t.Fatalf("remote session key %q was not seeded", remoteKey)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRemoteSessionFallbackLoadsSecretsBeforeObjectStoreInit(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
accessKeyEnv := "NARRATIO_TEST_REMOTE_SESSION_KEY_ID"
|
||||
secretKeyEnv := "NARRATIO_TEST_REMOTE_SESSION_SECRET"
|
||||
restoreEnvAfterTest(t, accessKeyEnv, secretKeyEnv)
|
||||
secretsDir := t.TempDir()
|
||||
mustWriteTestFile(t, filepath.Join(secretsDir, accessKeyEnv), "remote-session-key-id\n")
|
||||
mustWriteTestFile(t, filepath.Join(secretsDir, secretKeyEnv), "remote-session-secret\n")
|
||||
addSecretsToPipelineConfig(t, pipelinePath, secretsDir, accessKeyEnv, secretKeyEnv)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
seedRemoteSessionConfig(t, fake, "2026-05-03", `session_id: "{{ session_id }}"
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
`)
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
origSessionDefaults := append([]string(nil), config.DefaultSessionConfigSearchPaths...)
|
||||
config.DefaultSessionConfigSearchPaths = []string{filepath.Join(t.TempDir(), "session.yml")}
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
if os.Getenv(accessKeyEnv) != "remote-session-key-id" || os.Getenv(secretKeyEnv) != "remote-session-secret" {
|
||||
return nil, fmt.Errorf("secrets were not loaded before remote session object store init")
|
||||
}
|
||||
return fake, nil
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
config.DefaultSessionConfigSearchPaths = origSessionDefaults
|
||||
})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"plan", "--config", pipelinePath, "--campaign", campaignPath, "--session-id", "2026-05-03"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteExplicitLocalSessionPrecedenceSkipsRemote(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{filepath.Join(t.TempDir(), "session.yml")})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"plan", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--session-id", "2026-05-03"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if storeInitCalls != 0 {
|
||||
t.Fatalf("object store init calls = %d, want 0", storeInitCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteLocalSessionDiscoveryPrecedenceSkipsRemote(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"plan", "--config", pipelinePath, "--campaign", campaignPath, "--session-id", "2026-05-03"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if storeInitCalls != 0 {
|
||||
t.Fatalf("object store init calls = %d, want 0", storeInitCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRemoteSessionMissingObjectFailsClearly(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
var storeInitCalls int
|
||||
missingSessionPath := filepath.Join(t.TempDir(), "session.yml")
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{missingSessionPath})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"plan", "--config", pipelinePath, "--campaign", campaignPath, "--session-id", "2026-05-03"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "remote session") || !strings.Contains(stderr.String(), "session.yml") || !strings.Contains(stderr.String(), "not found") {
|
||||
t.Fatalf("stderr = %q, want remote session not found context", stderr.String())
|
||||
}
|
||||
if !strings.Contains(stderr.String(), missingSessionPath) {
|
||||
t.Fatalf("stderr = %q, want local searched path", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRemoteSessionRequiresSessionID(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, &storage.FakeBackend{}, &storeInitCalls, []string{filepath.Join(t.TempDir(), "session.yml")})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"plan", "--config", pipelinePath, "--campaign", campaignPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "remote session loading requires --session-id") {
|
||||
t.Fatalf("stderr = %q, want session-id guidance", stderr.String())
|
||||
}
|
||||
if storeInitCalls != 0 {
|
||||
t.Fatalf("object store init calls = %d, want 0", storeInitCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRemoteSessionStorageInitErrorFailsClearly(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
origSessionDefaults := append([]string(nil), config.DefaultSessionConfigSearchPaths...)
|
||||
config.DefaultSessionConfigSearchPaths = []string{filepath.Join(t.TempDir(), "session.yml")}
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
return nil, errors.New("storage unavailable")
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
config.DefaultSessionConfigSearchPaths = origSessionDefaults
|
||||
})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"plan", "--config", pipelinePath, "--campaign", campaignPath, "--session-id", "2026-05-03"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "storage unavailable") || !strings.Contains(stderr.String(), "remote session") {
|
||||
t.Fatalf("stderr = %q, want remote storage context", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRemoteSessionMalformedYAMLFailsStrictDecode(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, _ := writeValidConfigFiles(t, workspaceRoot)
|
||||
fake := &storage.FakeBackend{}
|
||||
seedRemoteSessionConfig(t, fake, "2026-05-03", "session_id: 2026-05-03\nunknown: true\n")
|
||||
var storeInitCalls int
|
||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{filepath.Join(t.TempDir(), "session.yml")})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"plan", "--config", pipelinePath, "--campaign", campaignPath, "--session-id", "2026-05-03"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "strict decode failed") {
|
||||
t.Fatalf("stderr = %q, want strict decode context", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func restoreAppConfigTestGlobals(t *testing.T, fake *storage.FakeBackend, storeInitCalls *int, sessionDefaults []string) {
|
||||
t.Helper()
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
origSessionDefaults := append([]string(nil), config.DefaultSessionConfigSearchPaths...)
|
||||
config.DefaultSessionConfigSearchPaths = append([]string(nil), sessionDefaults...)
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
if storeInitCalls != nil {
|
||||
(*storeInitCalls)++
|
||||
}
|
||||
return fake, nil
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
config.DefaultSessionConfigSearchPaths = origSessionDefaults
|
||||
})
|
||||
}
|
||||
|
||||
func seedRemoteSessionConfig(t *testing.T, fake *storage.FakeBackend, sessionID, content string) string {
|
||||
t.Helper()
|
||||
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", sessionID)
|
||||
remoteKey := artifacts.S3SessionConfigKey(sessionPrefix)
|
||||
fake.SeedObject(storage.FakeObject{
|
||||
Key: remoteKey,
|
||||
Data: []byte(content),
|
||||
ETag: "remote-session-etag",
|
||||
})
|
||||
return remoteKey
|
||||
}
|
||||
|
||||
func addSecretsToPipelineConfig(t *testing.T, pipelinePath, secretsDir, accessKeyEnv, secretKeyEnv string) {
|
||||
t.Helper()
|
||||
pipelineData, err := os.ReadFile(pipelinePath)
|
||||
if err != nil {
|
||||
t.Fatalf("read pipeline: %v", err)
|
||||
}
|
||||
pipelineYAML := strings.Replace(
|
||||
string(pipelineData),
|
||||
"storage:\n backend: s3\n s3:\n bucket: test-bucket\n",
|
||||
"storage:\n backend: s3\n s3:\n bucket: test-bucket\n access_key_id_env: "+accessKeyEnv+"\n secret_access_key_env: "+secretKeyEnv+"\nsecrets:\n env_dir: "+secretsDir+"\n",
|
||||
1,
|
||||
)
|
||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||
t.Fatalf("write pipeline: %v", err)
|
||||
}
|
||||
}
|
||||
144
internal/app/restore.go
Normal file
144
internal/app/restore.go
Normal file
@@ -0,0 +1,144 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"os"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/logging"
|
||||
)
|
||||
|
||||
var newObjectStoreFromConfigFn = storage.NewObjectStoreFromConfig
|
||||
var discoverRemoteCurrentStateFn = discoverRemoteCurrentState
|
||||
var buildRestorePlanFn = buildRestorePlan
|
||||
var executeRestorePlanFn = executeRestorePlan
|
||||
|
||||
// Restore validates restore CLI/config inputs and storage preflight for future restore phases.
|
||||
func Restore(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("restore", flag.ContinueOnError)
|
||||
fs.SetOutput(out)
|
||||
|
||||
var pipelinePath string
|
||||
var campaignPath string
|
||||
var sessionPath string
|
||||
var sessionID string
|
||||
var previousSessionID string
|
||||
var dryRun bool
|
||||
var force bool
|
||||
var includeAudio bool
|
||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||
fs.StringVar(&campaignPath, "campaign", "", "path to campaign.yml (optional; defaults searched)")
|
||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
||||
fs.StringVar(&sessionID, "session-id", "", "session identifier for session.yml templates")
|
||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "previous session identifier for session.yml templates")
|
||||
fs.BoolVar(&dryRun, "dry-run", false, "plan restore actions without writing local files")
|
||||
fs.BoolVar(&force, "force", false, "overwrite local conflicts with remote state")
|
||||
fs.BoolVar(&includeAudio, "include-audio", false, "include archived session-level audio objects")
|
||||
fs.Usage = func() {
|
||||
_, _ = fmt.Fprintln(out, "Usage: narratio restore [--config <path>] [--campaign <path>] [--session <path>] [--session-id <value>] [--previous-session-id <value>] [--dry-run] [--force] [--include-audio]")
|
||||
_, _ = fmt.Fprintln(out)
|
||||
_, _ = fmt.Fprintln(out, "Flags:")
|
||||
fs.PrintDefaults()
|
||||
}
|
||||
|
||||
if err := fs.Parse(args); err != nil {
|
||||
if errors.Is(err, flag.ErrHelp) {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("restore: invalid flags: %w", err)
|
||||
}
|
||||
if fs.NArg() != 0 {
|
||||
return fmt.Errorf("restore: unexpected positional arguments")
|
||||
}
|
||||
cfg, err := loadCommandConfig(ctx, pipelinePath, campaignPath, sessionPath, config.SessionLoadOptions{
|
||||
SessionID: sessionID,
|
||||
PreviousSessionID: previousSessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: %w", err)
|
||||
}
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
return fmt.Errorf("restore: %w", err)
|
||||
}
|
||||
objectStore, err := newCommandObjectStore(ctx, cfg, logging.NewLogger(os.Stderr, slog.LevelInfo))
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: %w", err)
|
||||
}
|
||||
current, err := discoverRemoteCurrentStateFn(ctx, cfg, objectStore)
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: %w", err)
|
||||
}
|
||||
plan, err := buildRestorePlanFn(ctx, cfg, current, objectStore, RestorePlanOptions{
|
||||
IncludeAudio: includeAudio,
|
||||
Force: force,
|
||||
DryRun: dryRun,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: %w", err)
|
||||
}
|
||||
report, err := newRestoreReport(current, plan, RestorePlanOptions{
|
||||
IncludeAudio: includeAudio,
|
||||
Force: force,
|
||||
DryRun: dryRun,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: %w", err)
|
||||
}
|
||||
if dryRun {
|
||||
if err := writeRestoreDryRunSummary(out, report); err != nil {
|
||||
return fmt.Errorf("restore: write plan output: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
artifactStore := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
|
||||
if _, err := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID); err != nil {
|
||||
return fmt.Errorf("restore: prepare workdir: %w", err)
|
||||
}
|
||||
lock, err := artifactStore.AcquireSessionLockFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("restore: acquire session lock: %w", err)
|
||||
}
|
||||
defer func() {
|
||||
_ = artifactStore.ReleaseSessionLock(lock)
|
||||
}()
|
||||
|
||||
if plan.ConflictCount > 0 && !force {
|
||||
report.setFailed(fmt.Errorf("conflict: %d conflicting path(s)", plan.ConflictCount))
|
||||
if _, reportErr := persistRestoreReport(artifactStore, cfg, report); reportErr != nil {
|
||||
return fmt.Errorf("restore: report failure: %w", reportErr)
|
||||
}
|
||||
return fmt.Errorf(
|
||||
"restore conflict: %d conflicting path(s); rerun with --force to overwrite (download=%d skip_same=%d conflicts=%d)",
|
||||
plan.ConflictCount,
|
||||
plan.DownloadCount,
|
||||
plan.SkipSameCount,
|
||||
plan.ConflictCount,
|
||||
)
|
||||
}
|
||||
|
||||
result, err := executeRestorePlanFn(ctx, cfg, current, plan, report, objectStore)
|
||||
if err != nil {
|
||||
report.setFailed(err)
|
||||
if _, reportErr := persistRestoreReport(artifactStore, cfg, report); reportErr != nil {
|
||||
return fmt.Errorf("restore: execute plan failed (%v) and report write failed (%v)", err, reportErr)
|
||||
}
|
||||
return fmt.Errorf("restore: execute plan: %w", err)
|
||||
}
|
||||
report.Execution.Downloaded = result.DownloadedCount
|
||||
report.setSucceeded()
|
||||
if _, err := persistRestoreReport(artifactStore, cfg, report); err != nil {
|
||||
return fmt.Errorf("restore: write report: %w", err)
|
||||
}
|
||||
if err := writeRestoreSuccessSummary(out, report); err != nil {
|
||||
return fmt.Errorf("restore: write summary: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
139
internal/app/restore_discovery.go
Normal file
139
internal/app/restore_discovery.go
Normal file
@@ -0,0 +1,139 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
// RemoteCurrentState captures discovered committed remote archive state for one session.
|
||||
type RemoteCurrentState struct {
|
||||
Bucket string
|
||||
SessionPrefix string
|
||||
CurrentRunIDKey string
|
||||
CurrentManifestKey string
|
||||
RunID string
|
||||
SessionID string
|
||||
Campaign string
|
||||
Manifest *manifest.Manifest
|
||||
}
|
||||
|
||||
func discoverRemoteCurrentState(ctx context.Context, cfg *config.Config, store storage.ObjectStore) (*RemoteCurrentState, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
||||
return nil, fmt.Errorf("resolved config with pipeline/session is required")
|
||||
}
|
||||
if store == nil {
|
||||
return nil, fmt.Errorf("remote object store is required")
|
||||
}
|
||||
|
||||
bucket := artifacts.ResolveArchiveBucket(cfg, nil)
|
||||
if strings.TrimSpace(bucket) == "" {
|
||||
return nil, fmt.Errorf("archive bucket is required")
|
||||
}
|
||||
sessionPrefix, err := artifacts.ResolveArchiveSessionPrefix(cfg, nil)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve archive session prefix: %w", err)
|
||||
}
|
||||
currentManifestKey, currentRunIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
||||
|
||||
exists, err := store.Exists(ctx, currentRunIDKey)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("check remote current run pointer %q: %w", currentRunIDKey, err)
|
||||
}
|
||||
if !exists {
|
||||
return nil, fmt.Errorf("remote current run pointer missing: %q", currentRunIDKey)
|
||||
}
|
||||
|
||||
runIDPath, err := downloadObjectToTemp(ctx, store, currentRunIDKey, "narratio-restore-current-run-id-*.txt")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("download remote current run pointer %q: %w", currentRunIDKey, err)
|
||||
}
|
||||
defer func() { _ = os.Remove(runIDPath) }()
|
||||
|
||||
runIDData, err := os.ReadFile(runIDPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read downloaded run pointer %q: %w", currentRunIDKey, err)
|
||||
}
|
||||
runID := strings.TrimSpace(string(runIDData))
|
||||
if runID == "" {
|
||||
return nil, fmt.Errorf("remote current run pointer %q is empty", currentRunIDKey)
|
||||
}
|
||||
|
||||
exists, err = store.Exists(ctx, currentManifestKey)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("check remote current manifest %q: %w", currentManifestKey, err)
|
||||
}
|
||||
if !exists {
|
||||
return nil, fmt.Errorf("remote current manifest missing: %q", currentManifestKey)
|
||||
}
|
||||
|
||||
manifestPath, err := downloadObjectToTemp(ctx, store, currentManifestKey, "narratio-restore-current-manifest-*.json")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("download remote current manifest %q: %w", currentManifestKey, err)
|
||||
}
|
||||
defer func() { _ = os.Remove(manifestPath) }()
|
||||
|
||||
manifestStore := &manifest.LocalStore{}
|
||||
remoteManifest, err := manifestStore.Load(ctx, manifestPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("remote current manifest decode failed: %w", err)
|
||||
}
|
||||
|
||||
requestedSession := strings.TrimSpace(cfg.Session.SessionID)
|
||||
requestedCampaign := strings.TrimSpace(cfg.Session.Campaign)
|
||||
manifestSession := strings.TrimSpace(remoteManifest.SessionID)
|
||||
manifestCampaign := strings.TrimSpace(remoteManifest.Campaign)
|
||||
|
||||
if manifestSession != requestedSession {
|
||||
return nil, fmt.Errorf(
|
||||
"remote current manifest session_id %q does not match requested session_id %q",
|
||||
manifestSession,
|
||||
requestedSession,
|
||||
)
|
||||
}
|
||||
if manifestCampaign == "" {
|
||||
return nil, fmt.Errorf("remote current manifest campaign is required")
|
||||
}
|
||||
if manifestCampaign != requestedCampaign {
|
||||
return nil, fmt.Errorf(
|
||||
"remote current manifest campaign %q does not match requested campaign %q",
|
||||
manifestCampaign,
|
||||
requestedCampaign,
|
||||
)
|
||||
}
|
||||
|
||||
return &RemoteCurrentState{
|
||||
Bucket: bucket,
|
||||
SessionPrefix: sessionPrefix,
|
||||
CurrentRunIDKey: currentRunIDKey,
|
||||
CurrentManifestKey: currentManifestKey,
|
||||
RunID: runID,
|
||||
SessionID: manifestSession,
|
||||
Campaign: manifestCampaign,
|
||||
Manifest: remoteManifest,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func downloadObjectToTemp(ctx context.Context, store storage.ObjectStore, key, pattern string) (string, error) {
|
||||
tmp, err := os.CreateTemp("", pattern)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("create temp file: %w", err)
|
||||
}
|
||||
path := tmp.Name()
|
||||
if err := tmp.Close(); err != nil {
|
||||
_ = os.Remove(path)
|
||||
return "", fmt.Errorf("close temp file: %w", err)
|
||||
}
|
||||
|
||||
if err := store.Download(ctx, key, path); err != nil {
|
||||
_ = os.Remove(path)
|
||||
return "", err
|
||||
}
|
||||
return path, nil
|
||||
}
|
||||
239
internal/app/restore_discovery_test.go
Normal file
239
internal/app/restore_discovery_test.go
Normal file
@@ -0,0 +1,239 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func TestDiscoverRemoteCurrentStateSuccess(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
sessionPrefix, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
|
||||
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign)})
|
||||
|
||||
state, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err != nil {
|
||||
t.Fatalf("discoverRemoteCurrentState() error = %v", err)
|
||||
}
|
||||
if state.RunID != "20260519T010203Z-a1b2c3d4" {
|
||||
t.Fatalf("run id = %q, want 20260519T010203Z-a1b2c3d4", state.RunID)
|
||||
}
|
||||
if state.SessionPrefix != sessionPrefix {
|
||||
t.Fatalf("session prefix = %q, want %q", state.SessionPrefix, sessionPrefix)
|
||||
}
|
||||
if state.CurrentRunIDKey != runIDKey {
|
||||
t.Fatalf("current run id key = %q, want %q", state.CurrentRunIDKey, runIDKey)
|
||||
}
|
||||
if state.CurrentManifestKey != manifestKey {
|
||||
t.Fatalf("current manifest key = %q, want %q", state.CurrentManifestKey, manifestKey)
|
||||
}
|
||||
if state.Manifest == nil {
|
||||
t.Fatal("manifest is nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateMissingRunPointerFails(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err == nil || !strings.Contains(err.Error(), "remote current run pointer missing") {
|
||||
t.Fatalf("error = %v, want missing run pointer failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateEmptyRunPointerFails(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
_, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
|
||||
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte(" \n\t")})
|
||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign)})
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err == nil || !strings.Contains(err.Error(), "is empty") {
|
||||
t.Fatalf("error = %v, want empty run pointer failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateMissingManifestFails(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
_, _, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
|
||||
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err == nil || !strings.Contains(err.Error(), "remote current manifest missing") {
|
||||
t.Fatalf("error = %v, want missing manifest failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateInvalidManifestFails(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
_, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
|
||||
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: []byte("{invalid json")})
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err == nil || !strings.Contains(err.Error(), "remote current manifest decode failed") {
|
||||
t.Fatalf("error = %v, want manifest decode failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateSessionMismatchFails(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
_, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
|
||||
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "wrong-session", cfg.Session.Campaign)})
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err == nil || !strings.Contains(err.Error(), "does not match requested session_id") {
|
||||
t.Fatalf("error = %v, want session mismatch failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateCampaignMismatchFails(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
_, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
|
||||
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, "wrong-campaign")})
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err == nil || !strings.Contains(err.Error(), "does not match requested campaign") {
|
||||
t.Fatalf("error = %v, want campaign mismatch failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateEmptyCampaignFails(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
store := &storage.FakeBackend{}
|
||||
_, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
|
||||
store.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, "")})
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err == nil || !strings.Contains(err.Error(), "campaign is required") {
|
||||
t.Fatalf("error = %v, want empty campaign failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiscoverRemoteCurrentStateUsesCurrentKeysUnderSessionPrefix(t *testing.T) {
|
||||
cfg := restoreDiscoveryConfig()
|
||||
sessionPrefix, manifestKey, runIDKey := restoreDiscoveryKeys(cfg)
|
||||
base := &storage.FakeBackend{}
|
||||
store := &captureObjectStore{delegate: base}
|
||||
|
||||
base.SeedObject(storage.FakeObject{Key: runIDKey, Data: []byte("20260519T010203Z-a1b2c3d4\n")})
|
||||
base.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign)})
|
||||
|
||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||
if err != nil {
|
||||
t.Fatalf("discoverRemoteCurrentState() error = %v", err)
|
||||
}
|
||||
|
||||
expectedRunKey := fmt.Sprintf("%scurrent/run_id.txt", sessionPrefix)
|
||||
expectedManifestKey := fmt.Sprintf("%scurrent/manifest.json", sessionPrefix)
|
||||
if !containsString(store.existsKeys, expectedRunKey) {
|
||||
t.Fatalf("exists keys = %#v, want run pointer key %q", store.existsKeys, expectedRunKey)
|
||||
}
|
||||
if !containsString(store.existsKeys, expectedManifestKey) {
|
||||
t.Fatalf("exists keys = %#v, want manifest key %q", store.existsKeys, expectedManifestKey)
|
||||
}
|
||||
if !containsString(store.downloadKeys, expectedRunKey) {
|
||||
t.Fatalf("download keys = %#v, want run pointer key %q", store.downloadKeys, expectedRunKey)
|
||||
}
|
||||
if !containsString(store.downloadKeys, expectedManifestKey) {
|
||||
t.Fatalf("download keys = %#v, want manifest key %q", store.downloadKeys, expectedManifestKey)
|
||||
}
|
||||
}
|
||||
|
||||
type captureObjectStore struct {
|
||||
delegate storage.ObjectStore
|
||||
existsKeys []string
|
||||
downloadKeys []string
|
||||
}
|
||||
|
||||
func (s *captureObjectStore) List(ctx context.Context, prefix string) ([]storage.ObjectInfo, error) {
|
||||
return s.delegate.List(ctx, prefix)
|
||||
}
|
||||
|
||||
func (s *captureObjectStore) Download(ctx context.Context, key, localPath string) error {
|
||||
s.downloadKeys = append(s.downloadKeys, key)
|
||||
return s.delegate.Download(ctx, key, localPath)
|
||||
}
|
||||
|
||||
func (s *captureObjectStore) Upload(ctx context.Context, localPath, key string, opts storage.UploadOptions) (storage.ObjectInfo, error) {
|
||||
return s.delegate.Upload(ctx, localPath, key, opts)
|
||||
}
|
||||
|
||||
func (s *captureObjectStore) Exists(ctx context.Context, key string) (bool, error) {
|
||||
s.existsKeys = append(s.existsKeys, key)
|
||||
return s.delegate.Exists(ctx, key)
|
||||
}
|
||||
|
||||
func restoreDiscoveryConfig() *config.Config {
|
||||
return &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Storage: config.StorageConfig{
|
||||
S3: &config.StorageS3Config{
|
||||
Bucket: "my-dnd-archive",
|
||||
RootPrefix: "dnd",
|
||||
},
|
||||
},
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func restoreDiscoveryKeys(cfg *config.Config) (sessionPrefix, manifestKey, runIDKey string) {
|
||||
sessionPrefix = artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
manifestKey, runIDKey = artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
||||
return sessionPrefix, manifestKey, runIDKey
|
||||
}
|
||||
|
||||
func restoreManifestJSON(t *testing.T, sessionID, campaign string) []byte {
|
||||
t.Helper()
|
||||
now := time.Date(2026, 5, 19, 23, 0, 0, 0, time.UTC).Format(time.RFC3339Nano)
|
||||
payload := map[string]any{
|
||||
"session_id": sessionID,
|
||||
"campaign": campaign,
|
||||
"created_at": now,
|
||||
"updated_at": now,
|
||||
"stages": map[string]any{},
|
||||
}
|
||||
data, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal manifest payload: %v", err)
|
||||
}
|
||||
return append(data, '\n')
|
||||
}
|
||||
|
||||
func containsString(values []string, target string) bool {
|
||||
for _, value := range values {
|
||||
if value == target {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
217
internal/app/restore_execute.go
Normal file
217
internal/app/restore_execute.go
Normal file
@@ -0,0 +1,217 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/audio"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
// RestoreExecutionResult captures concrete file-install results for one restore execution.
|
||||
type RestoreExecutionResult struct {
|
||||
DownloadedCount int
|
||||
}
|
||||
|
||||
func executeRestorePlan(
|
||||
ctx context.Context,
|
||||
cfg *config.Config,
|
||||
current *RemoteCurrentState,
|
||||
plan *RestorePlan,
|
||||
report *RestoreReport,
|
||||
store storage.ObjectStore,
|
||||
) (*RestoreExecutionResult, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
||||
return nil, fmt.Errorf("resolved config with pipeline/session is required")
|
||||
}
|
||||
if current == nil {
|
||||
return nil, fmt.Errorf("remote current state is required")
|
||||
}
|
||||
if plan == nil {
|
||||
return nil, fmt.Errorf("restore plan is required")
|
||||
}
|
||||
if store == nil {
|
||||
return nil, fmt.Errorf("remote object store is required")
|
||||
}
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
manifestActions := make([]RestoreAction, 0, 1)
|
||||
actions := make([]RestoreAction, 0, len(plan.Actions))
|
||||
for _, action := range plan.Actions {
|
||||
if action.Kind != RestoreActionDownload {
|
||||
continue
|
||||
}
|
||||
if action.LocalRelativePath == config.PathManifestFile {
|
||||
manifestActions = append(manifestActions, action)
|
||||
continue
|
||||
}
|
||||
actions = append(actions, action)
|
||||
}
|
||||
if len(manifestActions) > 1 {
|
||||
return nil, fmt.Errorf("restore plan includes multiple manifest download actions")
|
||||
}
|
||||
if len(manifestActions) == 1 {
|
||||
actions = append(actions, manifestActions[0])
|
||||
}
|
||||
|
||||
result := &RestoreExecutionResult{}
|
||||
for _, action := range actions {
|
||||
if err := executeRestoreDownloadAction(ctx, cfg, sessionRoot, current, action, store); err != nil {
|
||||
if report != nil {
|
||||
report.markFailed(action, err)
|
||||
}
|
||||
return nil, fmt.Errorf("install %q from %q: %w", action.LocalRelativePath, action.RemoteKey, err)
|
||||
}
|
||||
if report != nil {
|
||||
report.markDownloaded(action)
|
||||
}
|
||||
result.DownloadedCount++
|
||||
}
|
||||
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func executeRestoreDownloadAction(
|
||||
ctx context.Context,
|
||||
cfg *config.Config,
|
||||
sessionRoot string,
|
||||
current *RemoteCurrentState,
|
||||
action RestoreAction,
|
||||
store storage.ObjectStore,
|
||||
) error {
|
||||
safeLocalPath, err := joinWithinSessionRoot(sessionRoot, action.LocalRelativePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("resolve safe local path: %w", err)
|
||||
}
|
||||
if strings.TrimSpace(action.LocalPath) != "" && filepath.Clean(action.LocalPath) != safeLocalPath {
|
||||
return fmt.Errorf("restore plan local path mismatch for %q", action.LocalRelativePath)
|
||||
}
|
||||
|
||||
if restoreActionIsAudio(action) {
|
||||
return executeRestoreAudioAction(ctx, cfg, safeLocalPath, action, store)
|
||||
}
|
||||
|
||||
tmpPath, err := downloadObjectToSiblingTemp(ctx, store, action.RemoteKey, safeLocalPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("download to temp file: %w", err)
|
||||
}
|
||||
removeTmp := true
|
||||
defer func() {
|
||||
if removeTmp {
|
||||
_ = os.Remove(tmpPath)
|
||||
}
|
||||
}()
|
||||
|
||||
if action.LocalRelativePath == config.PathManifestFile {
|
||||
if err := validateRestoredManifest(ctx, cfg, current, tmpPath); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if err := os.Chmod(tmpPath, 0o644); err != nil {
|
||||
return fmt.Errorf("set file permissions: %w", err)
|
||||
}
|
||||
if err := os.Rename(tmpPath, safeLocalPath); err != nil {
|
||||
return fmt.Errorf("install file atomically: %w", err)
|
||||
}
|
||||
removeTmp = false
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func executeRestoreAudioAction(
|
||||
ctx context.Context,
|
||||
cfg *config.Config,
|
||||
safeLocalPath string,
|
||||
action RestoreAction,
|
||||
store storage.ObjectStore,
|
||||
) error {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Storage.S3 == nil || cfg.Session == nil {
|
||||
return fmt.Errorf("resolved s3 config and session are required")
|
||||
}
|
||||
spoolDir := artifacts.SessionSpoolRestoreAudioDir(cfg.Pipeline.Spool.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
spoolPath := filepath.Join(spoolDir, filepath.Base(safeLocalPath))
|
||||
cacheEnabled := cfg.Pipeline.Cache.S3Audio == nil || *cfg.Pipeline.Cache.S3Audio
|
||||
_, err := audio.MaterializeS3Audio(ctx, audio.S3MaterializeRequest{
|
||||
Store: store,
|
||||
Object: storage.ObjectInfo{
|
||||
Key: action.RemoteKey,
|
||||
Size: action.Size,
|
||||
ETag: action.ETag,
|
||||
},
|
||||
Bucket: strings.TrimSpace(cfg.Pipeline.Storage.S3.Bucket),
|
||||
CacheRoot: strings.TrimSpace(cfg.Pipeline.Cache.Root),
|
||||
CacheEnabled: cacheEnabled,
|
||||
SpoolPath: spoolPath,
|
||||
DestPath: safeLocalPath,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("materialize audio: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func downloadObjectToSiblingTemp(ctx context.Context, store storage.ObjectStore, remoteKey, destPath string) (string, error) {
|
||||
if strings.TrimSpace(destPath) == "" {
|
||||
return "", fmt.Errorf("destination path is required")
|
||||
}
|
||||
dir := filepath.Dir(destPath)
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return "", fmt.Errorf("create destination directory: %w", err)
|
||||
}
|
||||
base := filepath.Base(destPath)
|
||||
tmp, err := os.CreateTemp(dir, "."+base+".restore-*.tmp")
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("create temp file: %w", err)
|
||||
}
|
||||
tmpPath := tmp.Name()
|
||||
if err := tmp.Close(); err != nil {
|
||||
_ = os.Remove(tmpPath)
|
||||
return "", fmt.Errorf("close temp file: %w", err)
|
||||
}
|
||||
|
||||
if err := store.Download(ctx, remoteKey, tmpPath); err != nil {
|
||||
_ = os.Remove(tmpPath)
|
||||
return "", err
|
||||
}
|
||||
|
||||
return tmpPath, nil
|
||||
}
|
||||
|
||||
func validateRestoredManifest(ctx context.Context, cfg *config.Config, current *RemoteCurrentState, path string) error {
|
||||
manifestStore := &manifest.LocalStore{}
|
||||
m, err := manifestStore.Load(ctx, path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("validate manifest decode: %w", err)
|
||||
}
|
||||
|
||||
requestedSession := strings.TrimSpace(cfg.Session.SessionID)
|
||||
requestedCampaign := strings.TrimSpace(cfg.Session.Campaign)
|
||||
manifestSession := strings.TrimSpace(m.SessionID)
|
||||
manifestCampaign := strings.TrimSpace(m.Campaign)
|
||||
if manifestSession != requestedSession {
|
||||
return fmt.Errorf("manifest session_id %q does not match requested session_id %q", manifestSession, requestedSession)
|
||||
}
|
||||
if manifestCampaign == "" {
|
||||
return fmt.Errorf("manifest campaign is required")
|
||||
}
|
||||
if manifestCampaign != requestedCampaign {
|
||||
return fmt.Errorf("manifest campaign %q does not match requested campaign %q", manifestCampaign, requestedCampaign)
|
||||
}
|
||||
if current != nil {
|
||||
if expected := strings.TrimSpace(current.SessionID); expected != "" && manifestSession != expected {
|
||||
return fmt.Errorf("manifest session_id %q does not match discovered session_id %q", manifestSession, expected)
|
||||
}
|
||||
if expected := strings.TrimSpace(current.Campaign); expected != "" && manifestCampaign != expected {
|
||||
return fmt.Errorf("manifest campaign %q does not match discovered campaign %q", manifestCampaign, expected)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
461
internal/app/restore_execution_test.go
Normal file
461
internal/app/restore_execution_test.go
Normal file
@@ -0,0 +1,461 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
func TestExecuteRestoreNonDryRunRestoresDurableFiles(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, manifestKey, runIDKey := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(fake, sessionPrefix+"transcripts/full.json", []byte(`{"segments":[1,2,3]}`))
|
||||
seedRestoreObject(fake, sessionPrefix+"artifacts/session_recap.md", []byte("# recap\n"))
|
||||
seedRestoreObject(fake, sessionPrefix+"audio/alice.flac", []byte("remote-audio"))
|
||||
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
|
||||
seedRestoreObject(fake, manifestKey, restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"restore", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if stderr.Len() != 0 {
|
||||
t.Fatalf("stderr = %q, want empty", stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "Restored session archive for sample-campaign/2026-05-03") {
|
||||
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
||||
}
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "full.json"), `{"segments":[1,2,3]}`)
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# recap\n")
|
||||
reportPath := filepath.Join(sessionRoot, "reports", "restore-latest.json")
|
||||
report := mustReadRestoreReport(t, reportPath)
|
||||
if report.Status != "succeeded" {
|
||||
t.Fatalf("report status = %q, want succeeded", report.Status)
|
||||
}
|
||||
if report.Execution.Downloaded != 3 {
|
||||
t.Fatalf("report execution.downloaded = %d, want 3", report.Execution.Downloaded)
|
||||
}
|
||||
if len(report.Actions) == 0 {
|
||||
t.Fatal("report actions is empty")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(sessionRoot, "audio", "alice.flac")); !os.IsNotExist(err) {
|
||||
t.Fatalf("audio should not be restored by default; stat err=%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreIncludeAudioRestoresAudio(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(fake, sessionPrefix+"audio/alice.flac", []byte("remote-audio"))
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"restore", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--include-audio"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "audio", "alice.flac"), "remote-audio")
|
||||
report := mustReadRestoreReport(t, filepath.Join(sessionRoot, "reports", "restore-latest.json"))
|
||||
if !report.IncludeAudio {
|
||||
t.Fatalf("report include_audio = %v, want true", report.IncludeAudio)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreIncludeAudioUsesCacheAfterWorkspaceDeletion(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
audioKey := sessionPrefix + "audio/alice.flac"
|
||||
seedRestoreObject(fake, audioKey, []byte("remote-audio"))
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"restore", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--include-audio"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("first restore exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if got := fakeDownloadCount(fake, audioKey); got != 1 {
|
||||
t.Fatalf("audio downloads after first restore = %d, want 1", got)
|
||||
}
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "audio", "alice.flac"), "remote-audio")
|
||||
|
||||
if err := os.RemoveAll(sessionRoot); err != nil {
|
||||
t.Fatalf("remove session root: %v", err)
|
||||
}
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
code = Execute([]string{"restore", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--include-audio"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("second restore exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if got := fakeDownloadCount(fake, audioKey); got != 1 {
|
||||
t.Fatalf("audio downloads after cached restore = %d, want still 1", got)
|
||||
}
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "audio", "alice.flac"), "remote-audio")
|
||||
}
|
||||
|
||||
func TestExecuteRestoreRestoresPreviousCacheWhenPresent(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(fake, sessionPrefix+"previous/manifest.json", []byte(`{"session_id":"2026-04-26"}`))
|
||||
seedRestoreObject(fake, sessionPrefix+"previous/artifacts/session_recap.md", []byte("# previous recap\n"))
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"restore", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "previous", "manifest.json"), `{"session_id":"2026-04-26"}`)
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md"), "# previous recap\n")
|
||||
report := mustReadRestoreReport(t, filepath.Join(sessionRoot, "reports", "restore-latest.json"))
|
||||
if report.Execution.Downloaded != 3 {
|
||||
t.Fatalf("report execution.downloaded = %d, want 3", report.Execution.Downloaded)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreConflictWithoutForceDoesNotOverwrite(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(fake, sessionPrefix+"transcripts/full.json", []byte("remote-transcript"))
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustWriteTestFile(t, filepath.Join(sessionRoot, "transcripts", "full.json"), "local-transcript")
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"restore", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "conflicting path") {
|
||||
t.Fatalf("stderr = %q, want conflict failure", stderr.String())
|
||||
}
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "full.json"), "local-transcript")
|
||||
report := mustReadRestoreReport(t, filepath.Join(sessionRoot, "reports", "restore-latest.json"))
|
||||
if report.Status != "failed" {
|
||||
t.Fatalf("report status = %q, want failed", report.Status)
|
||||
}
|
||||
if report.Plan.Conflicts != 1 {
|
||||
t.Fatalf("report plan.conflicts = %d, want 1", report.Plan.Conflicts)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreForceOverwritesDifferingFile(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(fake, sessionPrefix+"transcripts/full.json", []byte("remote-transcript"))
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustWriteTestFile(t, filepath.Join(sessionRoot, "transcripts", "full.json"), "local-transcript")
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"restore", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--force"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "full.json"), "remote-transcript")
|
||||
report := mustReadRestoreReport(t, filepath.Join(sessionRoot, "reports", "restore-latest.json"))
|
||||
if !report.Force {
|
||||
t.Fatalf("report force = %v, want true", report.Force)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreForceOverwritesDifferingPreviousCacheFile(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(fake, sessionPrefix+"previous/artifacts/session_recap.md", []byte("# remote previous recap\n"))
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustWriteTestFile(t, filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md"), "# local previous recap\n")
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"restore", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--force"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md"), "# remote previous recap\n")
|
||||
}
|
||||
|
||||
func TestExecuteRestoreLockConflictFailsAndWritesNothing(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, _, _ := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(fake, sessionPrefix+"transcripts/full.json", []byte("remote-transcript"))
|
||||
|
||||
store := artifacts.NewLocalStore(workspaceRoot)
|
||||
lock, err := store.AcquireSessionLockFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
t.Fatalf("AcquireSessionLockFor() error = %v", err)
|
||||
}
|
||||
defer func() { _ = store.ReleaseSessionLock(lock) }()
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"restore", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "acquire session lock") {
|
||||
t.Fatalf("stderr = %q, want lock failure", stderr.String())
|
||||
}
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if _, err := os.Stat(filepath.Join(sessionRoot, "transcripts", "full.json")); !os.IsNotExist(err) {
|
||||
t.Fatalf("transcript should not be restored when lock acquisition fails; stat err=%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreInvalidManifestDoesNotCorruptExistingManifest(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
base := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, manifestKey, _ := seedRestoreCommittedState(t, base, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(base, sessionPrefix+"transcripts/full.json", []byte("remote-transcript"))
|
||||
|
||||
toggled := &stagedManifestDownloadStore{
|
||||
delegate: base,
|
||||
manifestKey: manifestKey,
|
||||
firstManifest: restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign),
|
||||
secondManifest: []byte("{invalid json"),
|
||||
manifestReads: 0,
|
||||
}
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
existing := manifest.New(cfg.Session.SessionID, nowUTC())
|
||||
existing.Campaign = cfg.Session.Campaign
|
||||
existingPath := filepath.Join(sessionRoot, "manifest.json")
|
||||
manifestStore := &manifest.LocalStore{}
|
||||
if err := manifestStore.Save(context.Background(), existingPath, existing); err != nil {
|
||||
t.Fatalf("save existing local manifest: %v", err)
|
||||
}
|
||||
existingData, err := os.ReadFile(existingPath)
|
||||
if err != nil {
|
||||
t.Fatalf("read existing local manifest: %v", err)
|
||||
}
|
||||
|
||||
restoreWithStoreAndRealPhases(t, toggled)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"restore", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--force"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "validate manifest decode") {
|
||||
t.Fatalf("stderr = %q, want manifest validation failure", stderr.String())
|
||||
}
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "full.json"), "remote-transcript")
|
||||
report := mustReadRestoreReport(t, filepath.Join(sessionRoot, "reports", "restore-latest.json"))
|
||||
if report.Status != "failed" {
|
||||
t.Fatalf("report status = %q, want failed", report.Status)
|
||||
}
|
||||
if strings.TrimSpace(report.Error) == "" {
|
||||
t.Fatal("report error is empty, want failure context")
|
||||
}
|
||||
afterData, err := os.ReadFile(existingPath)
|
||||
if err != nil {
|
||||
t.Fatalf("read local manifest after failure: %v", err)
|
||||
}
|
||||
if string(afterData) != string(existingData) {
|
||||
t.Fatalf("local manifest changed after failed restore; before=%q after=%q", string(existingData), string(afterData))
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestorePlanPathMismatchFails(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
seedRestoreObject(store, current.SessionPrefix+"transcripts/full.json", []byte("remote-transcript"))
|
||||
|
||||
plan := &RestorePlan{Actions: []RestoreAction{{
|
||||
Kind: RestoreActionDownload,
|
||||
RemoteKey: current.SessionPrefix + "transcripts/full.json",
|
||||
LocalRelativePath: "transcripts/full.json",
|
||||
LocalPath: "/tmp/escape.txt",
|
||||
}}}
|
||||
|
||||
report, err := newRestoreReport(current, plan, RestorePlanOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("newRestoreReport() error = %v", err)
|
||||
}
|
||||
_, err = executeRestorePlan(context.Background(), cfg, current, plan, report, store)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "local path mismatch") {
|
||||
t.Fatalf("error = %v, want local path mismatch", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustReadRestoreReport(t *testing.T, path string) *RestoreReport {
|
||||
t.Helper()
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile(%q): %v", path, err)
|
||||
}
|
||||
var report RestoreReport
|
||||
if err := json.Unmarshal(data, &report); err != nil {
|
||||
t.Fatalf("Unmarshal restore report %q: %v", path, err)
|
||||
}
|
||||
return &report
|
||||
}
|
||||
|
||||
func restoreWithStoreAndRealPhases(t *testing.T, objectStore storage.ObjectStore) {
|
||||
t.Helper()
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
origDiscoverFn := discoverRemoteCurrentStateFn
|
||||
origPlanFn := buildRestorePlanFn
|
||||
origExecuteFn := executeRestorePlanFn
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
discoverRemoteCurrentStateFn = origDiscoverFn
|
||||
buildRestorePlanFn = origPlanFn
|
||||
executeRestorePlanFn = origExecuteFn
|
||||
})
|
||||
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
return objectStore, nil
|
||||
}
|
||||
discoverRemoteCurrentStateFn = discoverRemoteCurrentState
|
||||
buildRestorePlanFn = buildRestorePlan
|
||||
executeRestorePlanFn = executeRestorePlan
|
||||
}
|
||||
|
||||
func seedRestoreCommittedState(t *testing.T, fake *storage.FakeBackend, pipelinePath, campaignPath, sessionPath string) (*config.Config, string, string, string) {
|
||||
t.Helper()
|
||||
cfg, err := config.LoadWithSessionOptions(pipelinePath, campaignPath, sessionPath, config.SessionLoadOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("LoadWithSessionOptions() error = %v", err)
|
||||
}
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
t.Fatalf("Validate() error = %v", err)
|
||||
}
|
||||
|
||||
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
||||
|
||||
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
|
||||
seedRestoreObject(fake, manifestKey, restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
|
||||
|
||||
return cfg, sessionPrefix, manifestKey, runIDKey
|
||||
}
|
||||
|
||||
func mustReadEquals(t *testing.T, path, want string) {
|
||||
t.Helper()
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile(%q): %v", path, err)
|
||||
}
|
||||
if string(data) != want {
|
||||
t.Fatalf("file %q = %q, want %q", path, string(data), want)
|
||||
}
|
||||
}
|
||||
|
||||
func fakeDownloadCount(fake *storage.FakeBackend, key string) int {
|
||||
count := 0
|
||||
for _, call := range fake.Downloads {
|
||||
if call.Key == key {
|
||||
count++
|
||||
}
|
||||
}
|
||||
return count
|
||||
}
|
||||
|
||||
type stagedManifestDownloadStore struct {
|
||||
delegate *storage.FakeBackend
|
||||
manifestKey string
|
||||
firstManifest []byte
|
||||
secondManifest []byte
|
||||
manifestReads int
|
||||
}
|
||||
|
||||
func (s *stagedManifestDownloadStore) List(ctx context.Context, prefix string) ([]storage.ObjectInfo, error) {
|
||||
return s.delegate.List(ctx, prefix)
|
||||
}
|
||||
|
||||
func (s *stagedManifestDownloadStore) Download(ctx context.Context, key, localPath string) error {
|
||||
if strings.TrimSpace(key) == strings.TrimSpace(s.manifestKey) {
|
||||
s.manifestReads++
|
||||
payload := s.secondManifest
|
||||
if s.manifestReads <= 1 {
|
||||
payload = s.firstManifest
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(localPath), 0o755); err != nil {
|
||||
return fmt.Errorf("download staged manifest: create parent: %w", err)
|
||||
}
|
||||
if err := os.WriteFile(localPath, payload, 0o644); err != nil {
|
||||
return fmt.Errorf("download staged manifest: write local file: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return s.delegate.Download(ctx, key, localPath)
|
||||
}
|
||||
|
||||
func (s *stagedManifestDownloadStore) Upload(ctx context.Context, localPath, key string, opts storage.UploadOptions) (storage.ObjectInfo, error) {
|
||||
return s.delegate.Upload(ctx, localPath, key, opts)
|
||||
}
|
||||
|
||||
func (s *stagedManifestDownloadStore) Exists(ctx context.Context, key string) (bool, error) {
|
||||
return s.delegate.Exists(ctx, key)
|
||||
}
|
||||
385
internal/app/restore_plan.go
Normal file
385
internal/app/restore_plan.go
Normal file
@@ -0,0 +1,385 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
// RestoreActionKind identifies one restore planner action.
|
||||
type RestoreActionKind string
|
||||
|
||||
const (
|
||||
RestoreActionDownload RestoreActionKind = "download"
|
||||
RestoreActionSkipSame RestoreActionKind = "skip_same"
|
||||
RestoreActionConflict RestoreActionKind = "conflict"
|
||||
)
|
||||
|
||||
// RestoreAction is one deterministic planner action.
|
||||
type RestoreAction struct {
|
||||
Kind RestoreActionKind
|
||||
RemoteKey string
|
||||
LocalRelativePath string
|
||||
LocalPath string
|
||||
Size int64
|
||||
ETag string
|
||||
ExistsLocal bool
|
||||
SameLocal bool
|
||||
Conflict bool
|
||||
Reason string
|
||||
}
|
||||
|
||||
// RestorePlan is the deterministic output of restore planning.
|
||||
type RestorePlan struct {
|
||||
Actions []RestoreAction
|
||||
DownloadCount int
|
||||
SkipSameCount int
|
||||
ConflictCount int
|
||||
}
|
||||
|
||||
// RestorePlanOptions control restore planning scope and classification.
|
||||
type RestorePlanOptions struct {
|
||||
IncludeAudio bool
|
||||
Force bool
|
||||
DryRun bool
|
||||
}
|
||||
|
||||
func buildRestorePlan(ctx context.Context, cfg *config.Config, current *RemoteCurrentState, store storage.ObjectStore, opts RestorePlanOptions) (*RestorePlan, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
||||
return nil, fmt.Errorf("resolved config with pipeline/session is required")
|
||||
}
|
||||
if current == nil {
|
||||
return nil, fmt.Errorf("remote current state is required")
|
||||
}
|
||||
if store == nil {
|
||||
return nil, fmt.Errorf("remote object store is required")
|
||||
}
|
||||
prefix := normalizeRemoteKey(current.SessionPrefix)
|
||||
if strings.TrimSpace(prefix) == "" {
|
||||
return nil, fmt.Errorf("remote session prefix is required")
|
||||
}
|
||||
if !strings.HasSuffix(prefix, "/") {
|
||||
prefix += "/"
|
||||
}
|
||||
|
||||
sessionPaths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
objects, err := store.List(ctx, prefix)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list remote session objects under %q: %w", prefix, err)
|
||||
}
|
||||
|
||||
candidates := make(map[string]storage.ObjectInfo, len(objects)+1)
|
||||
for _, obj := range objects {
|
||||
key := normalizeRemoteKey(obj.Key)
|
||||
if key == "" {
|
||||
continue
|
||||
}
|
||||
obj.Key = key
|
||||
candidates[key] = obj
|
||||
}
|
||||
if strings.TrimSpace(current.CurrentManifestKey) != "" {
|
||||
key := normalizeRemoteKey(current.CurrentManifestKey)
|
||||
if _, ok := candidates[key]; !ok {
|
||||
candidates[key] = storage.ObjectInfo{Key: key}
|
||||
}
|
||||
}
|
||||
|
||||
actions := make([]RestoreAction, 0, len(candidates))
|
||||
for key, obj := range candidates {
|
||||
rel, include, err := restoreLocalRelativePathForKey(prefix, normalizeRemoteKey(current.CurrentManifestKey), key, opts.IncludeAudio)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("map remote key %q: %w", key, err)
|
||||
}
|
||||
if !include {
|
||||
continue
|
||||
}
|
||||
localPath, err := joinWithinSessionRoot(sessionPaths.Root, rel)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("map remote key %q: %w", key, err)
|
||||
}
|
||||
|
||||
action, err := classifyRestoreAction(ctx, store, obj, rel, localPath, opts.Force)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("classify remote key %q: %w", key, err)
|
||||
}
|
||||
actions = append(actions, action)
|
||||
}
|
||||
|
||||
sort.Slice(actions, func(i, j int) bool {
|
||||
if actions[i].LocalRelativePath == actions[j].LocalRelativePath {
|
||||
return actions[i].RemoteKey < actions[j].RemoteKey
|
||||
}
|
||||
return actions[i].LocalRelativePath < actions[j].LocalRelativePath
|
||||
})
|
||||
|
||||
plan := &RestorePlan{Actions: actions}
|
||||
for _, action := range actions {
|
||||
switch action.Kind {
|
||||
case RestoreActionDownload:
|
||||
plan.DownloadCount++
|
||||
case RestoreActionSkipSame:
|
||||
plan.SkipSameCount++
|
||||
case RestoreActionConflict:
|
||||
plan.ConflictCount++
|
||||
}
|
||||
}
|
||||
|
||||
_ = opts.DryRun
|
||||
return plan, nil
|
||||
}
|
||||
|
||||
func normalizeRemoteKey(v string) string {
|
||||
return strings.Trim(strings.ReplaceAll(strings.TrimSpace(v), "\\", "/"), "/")
|
||||
}
|
||||
|
||||
func restoreLocalRelativePathForKey(sessionPrefix, currentManifestKey, key string, includeAudio bool) (string, bool, error) {
|
||||
if key == "" {
|
||||
return "", false, nil
|
||||
}
|
||||
if key == currentManifestKey {
|
||||
return config.PathManifestFile, true, nil
|
||||
}
|
||||
if !strings.HasPrefix(key, sessionPrefix) {
|
||||
return "", false, fmt.Errorf("key is outside resolved session prefix %q", sessionPrefix)
|
||||
}
|
||||
|
||||
rel := strings.TrimPrefix(key, sessionPrefix)
|
||||
rel = strings.TrimSpace(rel)
|
||||
if rel == "" {
|
||||
return "", false, nil
|
||||
}
|
||||
|
||||
cleanRel := path.Clean(rel)
|
||||
if cleanRel == "." || cleanRel == "" {
|
||||
return "", false, nil
|
||||
}
|
||||
if cleanRel == ".." || strings.HasPrefix(cleanRel, "../") || strings.HasPrefix(cleanRel, "/") {
|
||||
return "", false, fmt.Errorf("key relative path %q escapes session scope", rel)
|
||||
}
|
||||
|
||||
if cleanRel == config.PathManifestFile {
|
||||
return config.PathManifestFile, true, nil
|
||||
}
|
||||
if strings.HasPrefix(cleanRel, config.S3CurrentSegment+"/") {
|
||||
return "", false, nil
|
||||
}
|
||||
if strings.HasPrefix(cleanRel, config.S3RunsSegment+"/") {
|
||||
return "", false, nil
|
||||
}
|
||||
|
||||
excludedRoots := []string{
|
||||
config.PathLogsDirSegment,
|
||||
config.PathReportsDirSegment,
|
||||
config.PathConfigDirSegment,
|
||||
config.PathInputsDirSegment,
|
||||
}
|
||||
for _, root := range excludedRoots {
|
||||
if cleanRel == root || strings.HasPrefix(cleanRel, root+"/") {
|
||||
return "", false, nil
|
||||
}
|
||||
}
|
||||
|
||||
if cleanRel == config.PathTranscriptsSegment || strings.HasPrefix(cleanRel, config.PathTranscriptsSegment+"/") {
|
||||
return cleanRel, true, nil
|
||||
}
|
||||
if cleanRel == config.PathArtifactsDirSegment || strings.HasPrefix(cleanRel, config.PathArtifactsDirSegment+"/") {
|
||||
return cleanRel, true, nil
|
||||
}
|
||||
if cleanRel == config.PathPreviousDirSegment || strings.HasPrefix(cleanRel, config.PathPreviousDirSegment+"/") {
|
||||
return cleanRel, true, nil
|
||||
}
|
||||
if includeAudio && (cleanRel == config.PathAudioDirSegment || strings.HasPrefix(cleanRel, config.PathAudioDirSegment+"/")) {
|
||||
return cleanRel, true, nil
|
||||
}
|
||||
|
||||
return "", false, nil
|
||||
}
|
||||
|
||||
func joinWithinSessionRoot(sessionRoot, relative string) (string, error) {
|
||||
if strings.TrimSpace(sessionRoot) == "" {
|
||||
return "", fmt.Errorf("session root is required")
|
||||
}
|
||||
cleanRel := path.Clean(strings.TrimSpace(relative))
|
||||
if cleanRel == "." || cleanRel == "" {
|
||||
return "", fmt.Errorf("relative path is required")
|
||||
}
|
||||
if cleanRel == ".." || strings.HasPrefix(cleanRel, "../") || strings.HasPrefix(cleanRel, "/") {
|
||||
return "", fmt.Errorf("relative path escapes session root")
|
||||
}
|
||||
abs := filepath.Clean(filepath.Join(sessionRoot, filepath.FromSlash(cleanRel)))
|
||||
root := filepath.Clean(sessionRoot)
|
||||
if abs != root && !strings.HasPrefix(abs, root+string(filepath.Separator)) {
|
||||
return "", fmt.Errorf("resolved local path escapes session root")
|
||||
}
|
||||
return abs, nil
|
||||
}
|
||||
|
||||
func classifyRestoreAction(
|
||||
ctx context.Context,
|
||||
store storage.ObjectStore,
|
||||
object storage.ObjectInfo,
|
||||
localRelPath string,
|
||||
localPath string,
|
||||
force bool,
|
||||
) (RestoreAction, error) {
|
||||
action := RestoreAction{
|
||||
RemoteKey: normalizeRemoteKey(object.Key),
|
||||
LocalRelativePath: localRelPath,
|
||||
LocalPath: localPath,
|
||||
Size: object.Size,
|
||||
ETag: object.ETag,
|
||||
}
|
||||
|
||||
info, err := os.Stat(localPath)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
action.Kind = RestoreActionDownload
|
||||
action.Reason = "local file missing"
|
||||
return action, nil
|
||||
}
|
||||
return RestoreAction{}, fmt.Errorf("stat local file: %w", err)
|
||||
}
|
||||
|
||||
action.ExistsLocal = true
|
||||
if info.IsDir() {
|
||||
action.Kind = RestoreActionConflict
|
||||
action.Conflict = true
|
||||
action.Reason = "local path is a directory"
|
||||
return action, nil
|
||||
}
|
||||
|
||||
if restoreRelativePathIsAudio(localRelPath) {
|
||||
if object.Size > 0 {
|
||||
if info.Size() == object.Size {
|
||||
action.Kind = RestoreActionSkipSame
|
||||
action.SameLocal = true
|
||||
action.Reason = "local audio size matches remote content"
|
||||
return action, nil
|
||||
}
|
||||
if force {
|
||||
action.Kind = RestoreActionDownload
|
||||
action.Reason = "local audio differs (size mismatch); overwrite with --force"
|
||||
return action, nil
|
||||
}
|
||||
action.Kind = RestoreActionConflict
|
||||
action.Conflict = true
|
||||
action.Reason = "local audio differs (size mismatch)"
|
||||
return action, nil
|
||||
}
|
||||
if force {
|
||||
action.Kind = RestoreActionDownload
|
||||
action.Reason = "local audio exists; remote size unavailable; overwrite with --force"
|
||||
return action, nil
|
||||
}
|
||||
action.Kind = RestoreActionConflict
|
||||
action.Conflict = true
|
||||
action.Reason = "local audio exists; remote size unavailable"
|
||||
return action, nil
|
||||
}
|
||||
|
||||
if object.Size > 0 && info.Size() != object.Size {
|
||||
if force {
|
||||
action.Kind = RestoreActionDownload
|
||||
action.Reason = "local file differs (size mismatch); overwrite with --force"
|
||||
return action, nil
|
||||
}
|
||||
action.Kind = RestoreActionConflict
|
||||
action.Conflict = true
|
||||
action.Reason = "local file differs (size mismatch)"
|
||||
return action, nil
|
||||
}
|
||||
|
||||
localDigest, err := artifacts.SHA256File(localPath)
|
||||
if err != nil {
|
||||
return RestoreAction{}, fmt.Errorf("checksum local file: %w", err)
|
||||
}
|
||||
remotePath, err := downloadObjectToTemp(ctx, store, action.RemoteKey, "narratio-restore-plan-remote-*.tmp")
|
||||
if err != nil {
|
||||
return RestoreAction{}, fmt.Errorf("download remote object: %w", err)
|
||||
}
|
||||
defer func() { _ = os.Remove(remotePath) }()
|
||||
|
||||
remoteDigest, err := artifacts.SHA256File(remotePath)
|
||||
if err != nil {
|
||||
return RestoreAction{}, fmt.Errorf("checksum remote object: %w", err)
|
||||
}
|
||||
|
||||
if remoteDigest == localDigest {
|
||||
action.Kind = RestoreActionSkipSame
|
||||
action.SameLocal = true
|
||||
action.Reason = "local file matches remote content"
|
||||
return action, nil
|
||||
}
|
||||
|
||||
if force {
|
||||
action.Kind = RestoreActionDownload
|
||||
action.Reason = "local file differs; overwrite with --force"
|
||||
return action, nil
|
||||
}
|
||||
|
||||
action.Kind = RestoreActionConflict
|
||||
action.Conflict = true
|
||||
action.Reason = "local file differs"
|
||||
return action, nil
|
||||
}
|
||||
|
||||
func restoreActionIsAudio(action RestoreAction) bool {
|
||||
return restoreRelativePathIsAudio(action.LocalRelativePath)
|
||||
}
|
||||
|
||||
func restoreRelativePathIsAudio(rel string) bool {
|
||||
cleanRel := path.Clean(strings.TrimSpace(rel))
|
||||
return cleanRel == config.PathAudioDirSegment || strings.HasPrefix(cleanRel, config.PathAudioDirSegment+"/")
|
||||
}
|
||||
|
||||
func writeRestorePlan(out io.Writer, current *RemoteCurrentState, plan *RestorePlan, opts RestorePlanOptions) error {
|
||||
if out == nil {
|
||||
return fmt.Errorf("output writer is required")
|
||||
}
|
||||
if current == nil {
|
||||
return fmt.Errorf("remote current state is required")
|
||||
}
|
||||
if plan == nil {
|
||||
return fmt.Errorf("restore plan is required")
|
||||
}
|
||||
|
||||
if _, err := fmt.Fprintf(
|
||||
out,
|
||||
"restore plan: session %s/%s run=%s actions=%d download=%d skip_same=%d conflict=%d dry_run=%t force=%t include_audio=%t\n",
|
||||
current.Campaign,
|
||||
current.SessionID,
|
||||
current.RunID,
|
||||
len(plan.Actions),
|
||||
plan.DownloadCount,
|
||||
plan.SkipSameCount,
|
||||
plan.ConflictCount,
|
||||
opts.DryRun,
|
||||
opts.Force,
|
||||
opts.IncludeAudio,
|
||||
); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, action := range plan.Actions {
|
||||
if _, err := fmt.Fprintf(out, "%s %s <- %s", action.Kind, action.LocalRelativePath, action.RemoteKey); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(action.Reason) != "" {
|
||||
if _, err := fmt.Fprintf(out, " (%s)", action.Reason); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if _, err := fmt.Fprintln(out); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
247
internal/app/restore_plan_test.go
Normal file
247
internal/app/restore_plan_test.go
Normal file
@@ -0,0 +1,247 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func TestRestorePlanDefaultScope(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"transcripts/full.json", []byte(`{"segments":[1]}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("# recap\n"))
|
||||
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
|
||||
seedRestoreObject(store, current.SessionPrefix+"runs/20260519T010203Z-a1b2/manifest.json", []byte("{}"))
|
||||
seedRestoreObject(store, current.SessionPrefix+"logs/archive.log", []byte("log"))
|
||||
|
||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan() error = %v", err)
|
||||
}
|
||||
|
||||
got := actionRelPaths(plan.Actions)
|
||||
want := []string{"artifacts/session_recap.md", "manifest.json", "transcripts/full.json"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("action local paths = %#v, want %#v", got, want)
|
||||
}
|
||||
if plan.DownloadCount != 3 || plan.SkipSameCount != 0 || plan.ConflictCount != 0 {
|
||||
t.Fatalf("counts = download=%d skip_same=%d conflict=%d, want 3/0/0", plan.DownloadCount, plan.SkipSameCount, plan.ConflictCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanIncludeAudio(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
|
||||
|
||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{IncludeAudio: true})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan() error = %v", err)
|
||||
}
|
||||
got := actionRelPaths(plan.Actions)
|
||||
want := []string{"audio/alice.flac", "manifest.json"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("action local paths = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanExistingAudioUsesSizeWithoutRemoteChecksumDownload(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustWriteTestFile(t, filepath.Join(sessionRoot, "audio", "alice.flac"), "local")
|
||||
|
||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{IncludeAudio: true})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan() error = %v", err)
|
||||
}
|
||||
if len(store.Downloads) != 0 {
|
||||
t.Fatalf("downloads = %d, want no remote checksum download for audio", len(store.Downloads))
|
||||
}
|
||||
actionByRel := map[string]RestoreAction{}
|
||||
for _, action := range plan.Actions {
|
||||
actionByRel[action.LocalRelativePath] = action
|
||||
}
|
||||
audioAction := actionByRel["audio/alice.flac"]
|
||||
if audioAction.Kind != RestoreActionSkipSame {
|
||||
t.Fatalf("audio action kind = %q, want %q", audioAction.Kind, RestoreActionSkipSame)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanIncludesPreviousCacheByDefault(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"previous/manifest.json", []byte(`{"session_id":"2026-04-26"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"previous/artifacts/session_recap.md", []byte("# previous recap\n"))
|
||||
|
||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan() error = %v", err)
|
||||
}
|
||||
|
||||
got := actionRelPaths(plan.Actions)
|
||||
want := []string{"manifest.json", "previous/artifacts/session_recap.md", "previous/manifest.json"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("action local paths = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanClassifiesSameAndConflict(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"transcripts/full.json", []byte(`{"segments":[1]}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("remote-content\n"))
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustWriteTestFile(t, filepath.Join(sessionRoot, "transcripts", "full.json"), `{"segments":[1]}`)
|
||||
mustWriteTestFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "different\n")
|
||||
|
||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan() error = %v", err)
|
||||
}
|
||||
|
||||
if plan.SkipSameCount != 1 {
|
||||
t.Fatalf("SkipSameCount = %d, want 1", plan.SkipSameCount)
|
||||
}
|
||||
if plan.ConflictCount != 1 {
|
||||
t.Fatalf("ConflictCount = %d, want 1", plan.ConflictCount)
|
||||
}
|
||||
|
||||
actionByRel := map[string]RestoreAction{}
|
||||
for _, action := range plan.Actions {
|
||||
actionByRel[action.LocalRelativePath] = action
|
||||
}
|
||||
if actionByRel["transcripts/full.json"].Kind != RestoreActionSkipSame {
|
||||
t.Fatalf("transcripts/full.json kind = %q, want %q", actionByRel["transcripts/full.json"].Kind, RestoreActionSkipSame)
|
||||
}
|
||||
if actionByRel["artifacts/session_recap.md"].Kind != RestoreActionConflict {
|
||||
t.Fatalf("artifacts/session_recap.md kind = %q, want %q", actionByRel["artifacts/session_recap.md"].Kind, RestoreActionConflict)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanForceTurnsConflictsIntoDownloads(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("remote-content\n"))
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustWriteTestFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "different\n")
|
||||
|
||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{Force: true})
|
||||
if err != nil {
|
||||
t.Fatalf("buildRestorePlan() error = %v", err)
|
||||
}
|
||||
|
||||
actionByRel := map[string]RestoreAction{}
|
||||
for _, action := range plan.Actions {
|
||||
actionByRel[action.LocalRelativePath] = action
|
||||
}
|
||||
recap := actionByRel["artifacts/session_recap.md"]
|
||||
if recap.Kind != RestoreActionDownload {
|
||||
t.Fatalf("artifacts/session_recap.md kind = %q, want %q", recap.Kind, RestoreActionDownload)
|
||||
}
|
||||
if plan.ConflictCount != 0 {
|
||||
t.Fatalf("ConflictCount = %d, want 0", plan.ConflictCount)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestorePlanTraversalUnsafeKeyFails(t *testing.T) {
|
||||
cfg := restorePlanConfig(t)
|
||||
current := restorePlanCurrentState(t, cfg)
|
||||
store := &storage.FakeBackend{}
|
||||
|
||||
seedRestoreObject(store, current.CurrentManifestKey, []byte(`{"session_id":"2026-05-03"}`))
|
||||
seedRestoreObject(store, current.SessionPrefix+"artifacts/../../escape.txt", []byte("bad"))
|
||||
|
||||
_, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "escapes session scope") {
|
||||
t.Fatalf("error = %v, want traversal safety failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func seedRestoreObject(store *storage.FakeBackend, key string, data []byte) {
|
||||
store.SeedObject(storage.FakeObject{Key: key, Data: data})
|
||||
}
|
||||
|
||||
func actionRelPaths(actions []RestoreAction) []string {
|
||||
out := make([]string, 0, len(actions))
|
||||
for _, action := range actions {
|
||||
out = append(out, action.LocalRelativePath)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func restorePlanConfig(t *testing.T) *config.Config {
|
||||
t.Helper()
|
||||
workspaceRoot := t.TempDir()
|
||||
return &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Workspace: config.WorkspaceConfig{Root: workspaceRoot},
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func restorePlanCurrentState(t *testing.T, cfg *config.Config) *RemoteCurrentState {
|
||||
t.Helper()
|
||||
sessionPrefix := artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
||||
return &RemoteCurrentState{
|
||||
Bucket: "test-bucket",
|
||||
SessionPrefix: sessionPrefix,
|
||||
CurrentManifestKey: manifestKey,
|
||||
CurrentRunIDKey: runIDKey,
|
||||
RunID: "20260519T010203Z-a1b2c3d4",
|
||||
SessionID: cfg.Session.SessionID,
|
||||
Campaign: cfg.Session.Campaign,
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteRestorePlan(t *testing.T) {
|
||||
current := &RemoteCurrentState{Campaign: "sample-campaign", SessionID: "2026-05-03", RunID: "r-1"}
|
||||
plan := &RestorePlan{Actions: []RestoreAction{{Kind: RestoreActionDownload, LocalRelativePath: "manifest.json", RemoteKey: "k", Reason: "local file missing"}}, DownloadCount: 1}
|
||||
var out strings.Builder
|
||||
if err := writeRestorePlan(&out, current, plan, RestorePlanOptions{DryRun: true}); err != nil {
|
||||
t.Fatalf("writeRestorePlan() error = %v", err)
|
||||
}
|
||||
text := out.String()
|
||||
if !strings.Contains(text, "restore plan: session sample-campaign/2026-05-03 run=r-1") {
|
||||
t.Fatalf("output = %q, want plan summary", text)
|
||||
}
|
||||
if !strings.Contains(text, "download manifest.json <- k") {
|
||||
t.Fatalf("output = %q, want action line", text)
|
||||
}
|
||||
}
|
||||
243
internal/app/restore_report.go
Normal file
243
internal/app/restore_report.go
Normal file
@@ -0,0 +1,243 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
// RestoreReport is the durable restore diagnostic model.
|
||||
type RestoreReport struct {
|
||||
GeneratedAt string `json:"generated_at"`
|
||||
SessionID string `json:"session_id"`
|
||||
Campaign string `json:"campaign"`
|
||||
RunID string `json:"run_id"`
|
||||
DryRun bool `json:"dry_run"`
|
||||
Force bool `json:"force"`
|
||||
IncludeAudio bool `json:"include_audio"`
|
||||
Status string `json:"status"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Plan RestorePlanSummary `json:"plan"`
|
||||
Execution RestoreExecutionStats `json:"execution"`
|
||||
Actions []RestoreReportAction `json:"actions"`
|
||||
reportPathRel string
|
||||
}
|
||||
|
||||
type RestorePlanSummary struct {
|
||||
Actions int `json:"actions"`
|
||||
Download int `json:"download"`
|
||||
SkipSame int `json:"skip_same"`
|
||||
Conflicts int `json:"conflicts"`
|
||||
}
|
||||
|
||||
type RestoreExecutionStats struct {
|
||||
Downloaded int `json:"downloaded"`
|
||||
Failed int `json:"failed"`
|
||||
}
|
||||
|
||||
type RestoreReportAction struct {
|
||||
Kind string `json:"kind"`
|
||||
LocalRelativePath string `json:"local_relative_path"`
|
||||
RemoteKey string `json:"remote_key"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
Status string `json:"status"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
func newRestoreReport(current *RemoteCurrentState, plan *RestorePlan, opts RestorePlanOptions) (*RestoreReport, error) {
|
||||
if current == nil {
|
||||
return nil, fmt.Errorf("remote current state is required")
|
||||
}
|
||||
if plan == nil {
|
||||
return nil, fmt.Errorf("restore plan is required")
|
||||
}
|
||||
r := &RestoreReport{
|
||||
GeneratedAt: nowUTC().Format("2006-01-02T15:04:05.999999999Z07:00"),
|
||||
SessionID: current.SessionID,
|
||||
Campaign: current.Campaign,
|
||||
RunID: current.RunID,
|
||||
DryRun: opts.DryRun,
|
||||
Force: opts.Force,
|
||||
IncludeAudio: opts.IncludeAudio,
|
||||
Status: "planned",
|
||||
Plan: RestorePlanSummary{
|
||||
Actions: len(plan.Actions),
|
||||
Download: plan.DownloadCount,
|
||||
SkipSame: plan.SkipSameCount,
|
||||
Conflicts: plan.ConflictCount,
|
||||
},
|
||||
Actions: make([]RestoreReportAction, 0, len(plan.Actions)),
|
||||
reportPathRel: filepath.ToSlash(filepath.Join(config.PathReportsDirSegment, "restore-latest.json")),
|
||||
}
|
||||
for _, action := range plan.Actions {
|
||||
r.Actions = append(r.Actions, RestoreReportAction{
|
||||
Kind: string(action.Kind),
|
||||
LocalRelativePath: action.LocalRelativePath,
|
||||
RemoteKey: action.RemoteKey,
|
||||
Reason: action.Reason,
|
||||
Status: initialRestoreActionStatus(action.Kind),
|
||||
})
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
|
||||
func initialRestoreActionStatus(kind RestoreActionKind) string {
|
||||
switch kind {
|
||||
case RestoreActionDownload:
|
||||
return "planned_download"
|
||||
case RestoreActionSkipSame:
|
||||
return "skipped_same"
|
||||
case RestoreActionConflict:
|
||||
return "conflict"
|
||||
default:
|
||||
return "planned"
|
||||
}
|
||||
}
|
||||
|
||||
func (r *RestoreReport) markDownloaded(action RestoreAction) {
|
||||
if r == nil {
|
||||
return
|
||||
}
|
||||
if idx := r.findAction(action); idx >= 0 {
|
||||
r.Actions[idx].Status = "downloaded"
|
||||
r.Actions[idx].Error = ""
|
||||
}
|
||||
r.Execution.Downloaded++
|
||||
}
|
||||
|
||||
func (r *RestoreReport) markFailed(action RestoreAction, err error) {
|
||||
if r == nil {
|
||||
return
|
||||
}
|
||||
if idx := r.findAction(action); idx >= 0 {
|
||||
r.Actions[idx].Status = "failed"
|
||||
if err != nil {
|
||||
r.Actions[idx].Error = err.Error()
|
||||
}
|
||||
}
|
||||
r.Execution.Failed++
|
||||
}
|
||||
|
||||
func (r *RestoreReport) setFailed(err error) {
|
||||
if r == nil {
|
||||
return
|
||||
}
|
||||
r.Status = "failed"
|
||||
if err != nil {
|
||||
r.Error = err.Error()
|
||||
}
|
||||
}
|
||||
|
||||
func (r *RestoreReport) setSucceeded() {
|
||||
if r == nil {
|
||||
return
|
||||
}
|
||||
r.Status = "succeeded"
|
||||
r.Error = ""
|
||||
}
|
||||
|
||||
func (r *RestoreReport) findAction(action RestoreAction) int {
|
||||
if r == nil {
|
||||
return -1
|
||||
}
|
||||
for i := range r.Actions {
|
||||
if r.Actions[i].LocalRelativePath == action.LocalRelativePath && r.Actions[i].RemoteKey == action.RemoteKey {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
|
||||
func writeRestoreDryRunSummary(out io.Writer, report *RestoreReport) error {
|
||||
if out == nil {
|
||||
return fmt.Errorf("output writer is required")
|
||||
}
|
||||
if report == nil {
|
||||
return fmt.Errorf("restore report is required")
|
||||
}
|
||||
if _, err := fmt.Fprintf(out, "Restore plan for %s/%s\n", report.Campaign, report.SessionID); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := fmt.Fprintf(out, "Remote run: %s\n", report.RunID); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := fmt.Fprintf(out, "Would download: %d\n", report.Plan.Download); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := fmt.Fprintf(out, "Would skip unchanged: %d\n", report.Plan.SkipSame); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := fmt.Fprintf(out, "Conflicts: %d\n", report.Plan.Conflicts); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, action := range report.Actions {
|
||||
line := ""
|
||||
switch action.Status {
|
||||
case "planned_download":
|
||||
line = "Would download: " + action.LocalRelativePath
|
||||
case "skipped_same":
|
||||
line = "Would skip unchanged: " + action.LocalRelativePath
|
||||
case "conflict":
|
||||
line = "Conflict: " + action.LocalRelativePath
|
||||
default:
|
||||
line = strings.TrimSpace(action.Kind) + ": " + action.LocalRelativePath
|
||||
}
|
||||
if _, err := fmt.Fprintln(out, line); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func writeRestoreSuccessSummary(out io.Writer, report *RestoreReport) error {
|
||||
if out == nil {
|
||||
return fmt.Errorf("output writer is required")
|
||||
}
|
||||
if report == nil {
|
||||
return fmt.Errorf("restore report is required")
|
||||
}
|
||||
if _, err := fmt.Fprintf(out, "Restored session archive for %s/%s\n", report.Campaign, report.SessionID); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := fmt.Fprintf(out, "Remote run: %s\n", report.RunID); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := fmt.Fprintf(out, "Downloaded: %d\n", report.Execution.Downloaded); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := fmt.Fprintf(out, "Skipped unchanged: %d\n", report.Plan.SkipSame); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := fmt.Fprintf(out, "Conflicts: %d\n", report.Plan.Conflicts); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func persistRestoreReport(store artifacts.Store, cfg *config.Config, report *RestoreReport) (string, error) {
|
||||
if store == nil {
|
||||
return "", fmt.Errorf("artifact store is required")
|
||||
}
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
||||
return "", fmt.Errorf("resolved config with pipeline/session is required")
|
||||
}
|
||||
if report == nil {
|
||||
return "", fmt.Errorf("restore report is required")
|
||||
}
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
reportPath := filepath.Join(sessionRoot, filepath.FromSlash(report.reportPathRel))
|
||||
payload, err := json.MarshalIndent(report, "", " ")
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("marshal restore report: %w", err)
|
||||
}
|
||||
payload = append(payload, '\n')
|
||||
if err := store.WriteFileAtomic(reportPath, payload, 0o644); err != nil {
|
||||
return "", fmt.Errorf("write restore report %q: %w", reportPath, err)
|
||||
}
|
||||
return reportPath, nil
|
||||
}
|
||||
415
internal/app/restore_test.go
Normal file
415
internal/app/restore_test.go
Normal file
@@ -0,0 +1,415 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func TestExecuteRestoreHelp(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := Execute([]string{"restore", "--help"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0", code)
|
||||
}
|
||||
if stderr.Len() != 0 {
|
||||
t.Fatalf("stderr = %q, want empty", stderr.String())
|
||||
}
|
||||
out := stdout.String()
|
||||
if !strings.Contains(out, "Usage: narratio restore") {
|
||||
t.Fatalf("stdout = %q, want restore usage", out)
|
||||
}
|
||||
if !strings.Contains(out, "--include-audio") {
|
||||
t.Fatalf("stdout = %q, want --include-audio flag", out)
|
||||
}
|
||||
if !strings.Contains(out, "--campaign") {
|
||||
t.Fatalf("stdout = %q, want --campaign flag", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreRecognizedAndReturnsNYI(t *testing.T) {
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
origDiscoverFn := discoverRemoteCurrentStateFn
|
||||
origPlanFn := buildRestorePlanFn
|
||||
origExecuteFn := executeRestorePlanFn
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
discoverRemoteCurrentStateFn = origDiscoverFn
|
||||
buildRestorePlanFn = origPlanFn
|
||||
executeRestorePlanFn = origExecuteFn
|
||||
})
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
return &storage.FakeBackend{}, nil
|
||||
}
|
||||
discoverRemoteCurrentStateFn = func(context.Context, *config.Config, storage.ObjectStore) (*RemoteCurrentState, error) {
|
||||
return &RemoteCurrentState{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
RunID: "20260519T010203Z-a1b2c3d4",
|
||||
}, nil
|
||||
}
|
||||
buildRestorePlanFn = func(context.Context, *config.Config, *RemoteCurrentState, storage.ObjectStore, RestorePlanOptions) (*RestorePlan, error) {
|
||||
return &RestorePlan{
|
||||
Actions: []RestoreAction{
|
||||
{
|
||||
Kind: RestoreActionDownload,
|
||||
LocalRelativePath: "manifest.json",
|
||||
RemoteKey: "dnd/campaigns/sample-campaign/sessions/2026-05-03/current/manifest.json",
|
||||
Reason: "local file missing",
|
||||
},
|
||||
},
|
||||
DownloadCount: 1,
|
||||
}, nil
|
||||
}
|
||||
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{
|
||||
"restore",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", "2026-05-03",
|
||||
"--dry-run",
|
||||
"--force",
|
||||
"--include-audio",
|
||||
},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0 for --dry-run restore planning; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if stderr.Len() != 0 {
|
||||
t.Fatalf("stderr = %q, want empty", stderr.String())
|
||||
}
|
||||
outText := stdout.String()
|
||||
if !strings.Contains(outText, "Restore plan for sample-campaign/2026-05-03") {
|
||||
t.Fatalf("stdout = %q, want restore plan summary", outText)
|
||||
}
|
||||
if !strings.Contains(outText, "Would download: 1") {
|
||||
t.Fatalf("stdout = %q, want plan count output", outText)
|
||||
}
|
||||
if !strings.Contains(outText, "Would download: manifest.json") {
|
||||
t.Fatalf("stdout = %q, want action output", outText)
|
||||
}
|
||||
|
||||
manifestPath := artifacts.SessionManifestPathForCampaign(workspaceRoot, "sample-campaign", "2026-05-03")
|
||||
if _, err := os.Stat(manifestPath); !os.IsNotExist(err) {
|
||||
t.Fatalf("manifest should not be created during phase-4 restore planning; stat err=%v", err)
|
||||
}
|
||||
reportPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "reports", "restore-latest.json")
|
||||
if _, err := os.Stat(reportPath); !os.IsNotExist(err) {
|
||||
t.Fatalf("restore report should not be written during dry-run; stat err=%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreRejectsUnexpectedPositionalArguments(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"restore", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "extra"}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "restore: unexpected positional arguments") {
|
||||
t.Fatalf("stderr = %q, want positional-args failure", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreFailsWhenStorageBackendNotConfigured(t *testing.T) {
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
origDiscoverFn := discoverRemoteCurrentStateFn
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
discoverRemoteCurrentStateFn = origDiscoverFn
|
||||
})
|
||||
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeRestoreConfigWithoutStorage(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"restore", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "no remote object store backend is configured") {
|
||||
t.Fatalf("stderr = %q, want storage backend preflight failure", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreDiscoveryErrorSurfaced(t *testing.T) {
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
origDiscoverFn := discoverRemoteCurrentStateFn
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
discoverRemoteCurrentStateFn = origDiscoverFn
|
||||
})
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
return &storage.FakeBackend{}, nil
|
||||
}
|
||||
discoverRemoteCurrentStateFn = func(context.Context, *config.Config, storage.ObjectStore) (*RemoteCurrentState, error) {
|
||||
return nil, fmt.Errorf("remote current run pointer missing: %q", "dnd/campaigns/sample-campaign/sessions/2026-05-03/current/run_id.txt")
|
||||
}
|
||||
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"restore", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "remote current run pointer missing") {
|
||||
t.Fatalf("stderr = %q, want discovery error context", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreLoadsSecretsBeforeObjectStoreInit(t *testing.T) {
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
origDiscoverFn := discoverRemoteCurrentStateFn
|
||||
origPlanFn := buildRestorePlanFn
|
||||
origExecuteFn := executeRestorePlanFn
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
discoverRemoteCurrentStateFn = origDiscoverFn
|
||||
buildRestorePlanFn = origPlanFn
|
||||
executeRestorePlanFn = origExecuteFn
|
||||
})
|
||||
|
||||
const accessKeyEnv = "OBJECT_STORAGE_KEY_ID"
|
||||
const secretKeyEnv = "OBJECT_STORAGE_KEY"
|
||||
restoreEnv := func(name string) {
|
||||
value, exists := os.LookupEnv(name)
|
||||
_ = os.Unsetenv(name)
|
||||
t.Cleanup(func() {
|
||||
if exists {
|
||||
_ = os.Setenv(name, value)
|
||||
return
|
||||
}
|
||||
_ = os.Unsetenv(name)
|
||||
})
|
||||
}
|
||||
restoreEnv(accessKeyEnv)
|
||||
restoreEnv(secretKeyEnv)
|
||||
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
secretsDir := filepath.Join(t.TempDir(), "secrets")
|
||||
mustWriteTestFile(t, filepath.Join(secretsDir, accessKeyEnv), "test-access-key-id\n")
|
||||
mustWriteTestFile(t, filepath.Join(secretsDir, secretKeyEnv), "test-secret-key\n")
|
||||
f, err := os.OpenFile(pipelinePath, os.O_APPEND|os.O_WRONLY, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("open pipeline config for append: %v", err)
|
||||
}
|
||||
defer f.Close()
|
||||
if _, err := f.WriteString("\nsecrets:\n env_dir: " + secretsDir + "\n"); err != nil {
|
||||
t.Fatalf("append secrets config: %v", err)
|
||||
}
|
||||
|
||||
storeInitCalled := false
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
storeInitCalled = true
|
||||
gotID, okID := os.LookupEnv(accessKeyEnv)
|
||||
if !okID || gotID != "test-access-key-id" {
|
||||
return nil, fmt.Errorf("missing or unexpected %s: %q (set=%t)", accessKeyEnv, gotID, okID)
|
||||
}
|
||||
gotSecret, okSecret := os.LookupEnv(secretKeyEnv)
|
||||
if !okSecret || gotSecret != "test-secret-key" {
|
||||
return nil, fmt.Errorf("missing or unexpected %s: %q (set=%t)", secretKeyEnv, gotSecret, okSecret)
|
||||
}
|
||||
return &storage.FakeBackend{}, nil
|
||||
}
|
||||
discoverRemoteCurrentStateFn = func(context.Context, *config.Config, storage.ObjectStore) (*RemoteCurrentState, error) {
|
||||
return &RemoteCurrentState{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
RunID: "20260519T010203Z-a1b2c3d4",
|
||||
}, nil
|
||||
}
|
||||
buildRestorePlanFn = func(context.Context, *config.Config, *RemoteCurrentState, storage.ObjectStore, RestorePlanOptions) (*RestorePlan, error) {
|
||||
return &RestorePlan{
|
||||
Actions: []RestoreAction{
|
||||
{
|
||||
Kind: RestoreActionDownload,
|
||||
LocalRelativePath: "manifest.json",
|
||||
RemoteKey: "dnd/campaigns/sample-campaign/sessions/2026-05-03/current/manifest.json",
|
||||
Reason: "local file missing",
|
||||
},
|
||||
},
|
||||
DownloadCount: 1,
|
||||
}, nil
|
||||
}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{
|
||||
"restore",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", "2026-05-03",
|
||||
"--dry-run",
|
||||
},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if !storeInitCalled {
|
||||
t.Fatal("expected object store initialization to be called")
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreNonDryRunConflictFailsBeforeNYI(t *testing.T) {
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
origDiscoverFn := discoverRemoteCurrentStateFn
|
||||
origPlanFn := buildRestorePlanFn
|
||||
origExecuteFn := executeRestorePlanFn
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
discoverRemoteCurrentStateFn = origDiscoverFn
|
||||
buildRestorePlanFn = origPlanFn
|
||||
executeRestorePlanFn = origExecuteFn
|
||||
})
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
return &storage.FakeBackend{}, nil
|
||||
}
|
||||
discoverRemoteCurrentStateFn = func(context.Context, *config.Config, storage.ObjectStore) (*RemoteCurrentState, error) {
|
||||
return &RemoteCurrentState{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
RunID: "20260519T010203Z-a1b2c3d4",
|
||||
}, nil
|
||||
}
|
||||
buildRestorePlanFn = func(context.Context, *config.Config, *RemoteCurrentState, storage.ObjectStore, RestorePlanOptions) (*RestorePlan, error) {
|
||||
return &RestorePlan{
|
||||
Actions: []RestoreAction{
|
||||
{Kind: RestoreActionConflict, LocalRelativePath: "transcripts/full.json", RemoteKey: "k", Reason: "local file differs"},
|
||||
},
|
||||
ConflictCount: 1,
|
||||
}, nil
|
||||
}
|
||||
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"restore", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if stdout.Len() != 0 {
|
||||
t.Fatalf("stdout = %q, want empty on conflict failure", stdout.String())
|
||||
}
|
||||
if !strings.Contains(stderr.String(), "restore conflict: 1 conflicting path(s); rerun with --force to overwrite") {
|
||||
t.Fatalf("stderr = %q, want conflict failure", stderr.String())
|
||||
}
|
||||
if strings.Contains(stderr.String(), "phase 4: restore execution") {
|
||||
t.Fatalf("stderr = %q, should fail before phase-4 NYI boundary", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRestoreNonDryRunForceExecutesPlan(t *testing.T) {
|
||||
origStoreFn := newObjectStoreFromConfigFn
|
||||
origDiscoverFn := discoverRemoteCurrentStateFn
|
||||
origPlanFn := buildRestorePlanFn
|
||||
origExecuteFn := executeRestorePlanFn
|
||||
t.Cleanup(func() {
|
||||
newObjectStoreFromConfigFn = origStoreFn
|
||||
discoverRemoteCurrentStateFn = origDiscoverFn
|
||||
buildRestorePlanFn = origPlanFn
|
||||
executeRestorePlanFn = origExecuteFn
|
||||
})
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
return &storage.FakeBackend{}, nil
|
||||
}
|
||||
discoverRemoteCurrentStateFn = func(context.Context, *config.Config, storage.ObjectStore) (*RemoteCurrentState, error) {
|
||||
return &RemoteCurrentState{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
RunID: "20260519T010203Z-a1b2c3d4",
|
||||
}, nil
|
||||
}
|
||||
buildRestorePlanFn = func(context.Context, *config.Config, *RemoteCurrentState, storage.ObjectStore, RestorePlanOptions) (*RestorePlan, error) {
|
||||
return &RestorePlan{
|
||||
Actions: []RestoreAction{
|
||||
{Kind: RestoreActionDownload, LocalRelativePath: "transcripts/full.json", RemoteKey: "k", Reason: "local file differs; overwrite with --force"},
|
||||
},
|
||||
DownloadCount: 1,
|
||||
}, nil
|
||||
}
|
||||
executeRestorePlanFn = func(context.Context, *config.Config, *RemoteCurrentState, *RestorePlan, *RestoreReport, storage.ObjectStore) (*RestoreExecutionResult, error) {
|
||||
return &RestoreExecutionResult{DownloadedCount: 1}, nil
|
||||
}
|
||||
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute([]string{"restore", "--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--force"}, &stdout, &stderr)
|
||||
if code != 0 {
|
||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "Restored session archive for sample-campaign/2026-05-03") {
|
||||
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
||||
}
|
||||
if stderr.Len() != 0 {
|
||||
t.Fatalf("stderr = %q, want empty", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func writeRestoreConfigWithoutStorage(t *testing.T, workspaceRoot string) (string, string, string) {
|
||||
t.Helper()
|
||||
|
||||
dir := t.TempDir()
|
||||
pipelinePath := filepath.Join(dir, "pipeline.yml")
|
||||
campaignPath := writeAppTestCampaignConfig(t, dir)
|
||||
sessionPath := filepath.Join(dir, "session.yml")
|
||||
|
||||
pipelineYAML := `workspace:
|
||||
root: ` + workspaceRoot + `
|
||||
whisperx:
|
||||
transcribe_url: https://example.com/transcribe
|
||||
`
|
||||
sessionYAML := `session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`
|
||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||
t.Fatalf("write pipeline config: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(sessionPath, []byte(sessionYAML), 0o644); err != nil {
|
||||
t.Fatalf("write session config: %v", err)
|
||||
}
|
||||
mustWriteTestFile(t, filepath.Join(dir, "speakers.yml"), "alice: alice.flac\n")
|
||||
mustWriteTestFile(t, filepath.Join(dir, "autocorrect.yml"), "[]\n")
|
||||
mustWriteTestFile(t, filepath.Join(dir, "glossary.yml"), "[]\n")
|
||||
mustWriteTestFile(t, filepath.Join(dir, "audio", "alice.flac"), "audio-bytes")
|
||||
|
||||
return pipelinePath, campaignPath, sessionPath
|
||||
}
|
||||
304
internal/app/restore_workflow_test.go
Normal file
304
internal/app/restore_workflow_test.go
Normal file
@@ -0,0 +1,304 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/scriptorium"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||
)
|
||||
|
||||
func TestRestoreThenRunStageForceAnalyzeUsesRestoredDurableState(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, manifestKey, runIDKey := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
|
||||
seedRestoreObject(fake, manifestKey, restoreWorkflowManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
|
||||
seedRestoreObject(fake, sessionPrefix+"transcripts/full.json", []byte(`{"segments":[1,2,3]}`+"\n"))
|
||||
seedRestoreObject(fake, sessionPrefix+"artifacts/session_recap.md", []byte("# restored recap\n"))
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fake)
|
||||
|
||||
origExecuteStagesFn := executeStagesFn
|
||||
t.Cleanup(func() {
|
||||
executeStagesFn = origExecuteStagesFn
|
||||
})
|
||||
executeStagesFn = func(ctx context.Context, cfg *config.Config, stages []stage.Stage, opts RunOptions) (*RunSummary, error) {
|
||||
if opts.Env == nil {
|
||||
opts.Env = &Env{}
|
||||
}
|
||||
opts.Env.Scriptorium = &scriptorium.NoopRunner{}
|
||||
return executeStages(ctx, cfg, stages, opts)
|
||||
}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
restoreCode := Execute(
|
||||
[]string{
|
||||
"restore",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", cfg.Session.SessionID,
|
||||
},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if restoreCode != 0 {
|
||||
t.Fatalf("restore exit code = %d, want 0; stderr=%q", restoreCode, stderr.String())
|
||||
}
|
||||
if stderr.Len() != 0 {
|
||||
t.Fatalf("restore stderr = %q, want empty", stderr.String())
|
||||
}
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "full.json"), `{"segments":[1,2,3]}`+"\n")
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# restored recap\n")
|
||||
|
||||
manifestStore := &manifest.LocalStore{}
|
||||
sessionManifestPath := artifacts.SessionManifestPathForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
beforeAnalyze, err := manifestStore.Load(context.Background(), sessionManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("load restored session manifest: %v", err)
|
||||
}
|
||||
upstreamCompletedAt := map[string]time.Time{}
|
||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim"} {
|
||||
rec := beforeAnalyze.Stages[stageName]
|
||||
if rec == nil || rec.Status != manifest.StatusSucceeded || rec.CompletedAt == nil {
|
||||
t.Fatalf("restored manifest stage %q = %#v, want succeeded with completion timestamp", stageName, rec)
|
||||
}
|
||||
upstreamCompletedAt[stageName] = *rec.CompletedAt
|
||||
}
|
||||
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
runStageCode := Execute(
|
||||
[]string{
|
||||
"run-stage",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", cfg.Session.SessionID,
|
||||
"--force",
|
||||
"--artifacts", "player_handout",
|
||||
"analyze",
|
||||
},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if runStageCode != 0 {
|
||||
t.Fatalf("run-stage exit code = %d, want 0; stderr=%q", runStageCode, stderr.String())
|
||||
}
|
||||
if stderr.Len() != 0 {
|
||||
t.Fatalf("run-stage stderr = %q, want empty", stderr.String())
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "stage=analyze executed=1 skipped=0 force=true") {
|
||||
t.Fatalf("run-stage stdout = %q, want analyze execution summary", stdout.String())
|
||||
}
|
||||
|
||||
playerHandoutPath := filepath.Join(sessionRoot, "artifacts", "player_handout.md")
|
||||
if _, err := os.Stat(playerHandoutPath); err != nil {
|
||||
t.Fatalf("restored analyze output %q missing: %v", playerHandoutPath, err)
|
||||
}
|
||||
|
||||
afterAnalyze, err := manifestStore.Load(context.Background(), sessionManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("load session manifest after run-stage analyze: %v", err)
|
||||
}
|
||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim"} {
|
||||
rec := afterAnalyze.Stages[stageName]
|
||||
if rec == nil || rec.Status != manifest.StatusSucceeded || rec.CompletedAt == nil {
|
||||
t.Fatalf("post-analyze manifest stage %q = %#v, want succeeded with completion timestamp", stageName, rec)
|
||||
}
|
||||
if !rec.CompletedAt.Equal(upstreamCompletedAt[stageName]) {
|
||||
t.Fatalf(
|
||||
"stage %q completion changed: before=%s after=%s",
|
||||
stageName,
|
||||
upstreamCompletedAt[stageName].Format(time.RFC3339Nano),
|
||||
rec.CompletedAt.Format(time.RFC3339Nano),
|
||||
)
|
||||
}
|
||||
}
|
||||
analyzeRec := afterAnalyze.Stages["analyze"]
|
||||
if analyzeRec == nil || analyzeRec.Status != manifest.StatusSucceeded {
|
||||
t.Fatalf("post-analyze stage record = %#v, want succeeded", analyzeRec)
|
||||
}
|
||||
|
||||
runManifestPaths, err := filepath.Glob(filepath.Join(sessionRoot, "runs", "*", "manifest.json"))
|
||||
if err != nil {
|
||||
t.Fatalf("glob run manifests: %v", err)
|
||||
}
|
||||
if len(runManifestPaths) != 1 {
|
||||
t.Fatalf("run manifest count = %d, want 1; paths=%v", len(runManifestPaths), runManifestPaths)
|
||||
}
|
||||
runManifest, err := manifestStore.LoadRun(context.Background(), runManifestPaths[0])
|
||||
if err != nil {
|
||||
t.Fatalf("load run manifest %q: %v", runManifestPaths[0], err)
|
||||
}
|
||||
if len(runManifest.RequestedStages) != 1 || runManifest.RequestedStages[0] != "analyze" {
|
||||
t.Fatalf("run manifest requested_stages = %#v, want [analyze]", runManifest.RequestedStages)
|
||||
}
|
||||
if runManifest.Stages["analyze"] == nil || runManifest.Stages["analyze"].Status != manifest.StatusSucceeded {
|
||||
t.Fatalf("run manifest analyze stage = %#v, want succeeded", runManifest.Stages["analyze"])
|
||||
}
|
||||
if runManifest.Stages["prepare"] != nil {
|
||||
t.Fatalf("run manifest should not include upstream prepare stage, got %#v", runManifest.Stages["prepare"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestoreThenAnalyzeUsesRestoredPreviousCacheWithoutObjectStore(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
appendRestoreWorkflowScriptoriumConfig(t, pipelinePath, `
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
previous_recap:
|
||||
source: narratio.previous_session.artifact.session_recap
|
||||
required: true
|
||||
`)
|
||||
|
||||
fakeStore := &storage.FakeBackend{}
|
||||
cfg, sessionPrefix, manifestKey, runIDKey := seedRestoreCommittedState(t, fakeStore, pipelinePath, campaignPath, sessionPath)
|
||||
seedRestoreObject(fakeStore, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
|
||||
seedRestoreObject(fakeStore, manifestKey, restoreWorkflowManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
|
||||
seedRestoreObject(fakeStore, sessionPrefix+"transcripts/trimmed.json", []byte(`{"segments":[]}`+"\n"))
|
||||
seedRestoreObject(fakeStore, sessionPrefix+"previous/manifest.json", []byte(`{"session_id":"2026-04-26"}`))
|
||||
seedRestoreObject(fakeStore, sessionPrefix+"previous/artifacts/session_recap.md", []byte("# previous recap\n"))
|
||||
|
||||
restoreWithStoreAndRealPhases(t, fakeStore)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
restoreCode := Execute(
|
||||
[]string{
|
||||
"restore",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", cfg.Session.SessionID,
|
||||
},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if restoreCode != 0 {
|
||||
t.Fatalf("restore exit code = %d, want 0; stderr=%q", restoreCode, stderr.String())
|
||||
}
|
||||
if stderr.Len() != 0 {
|
||||
t.Fatalf("restore stderr = %q, want empty", stderr.String())
|
||||
}
|
||||
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "transcripts", "trimmed.json"), `{"segments":[]}`+"\n")
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "previous", "manifest.json"), `{"session_id":"2026-04-26"}`)
|
||||
mustReadEquals(t, filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md"), "# previous recap\n")
|
||||
|
||||
scriptoriumFake := &scriptorium.FakeRunner{}
|
||||
origExecuteStagesFn := executeStagesFn
|
||||
origObjectStoreFn := newObjectStoreFromConfigFn
|
||||
objectStoreConstructed := false
|
||||
t.Cleanup(func() {
|
||||
executeStagesFn = origExecuteStagesFn
|
||||
newObjectStoreFromConfigFn = origObjectStoreFn
|
||||
})
|
||||
executeStagesFn = func(ctx context.Context, cfg *config.Config, stages []stage.Stage, opts RunOptions) (*RunSummary, error) {
|
||||
if opts.Env == nil {
|
||||
opts.Env = &Env{}
|
||||
}
|
||||
opts.Env.Scriptorium = scriptoriumFake
|
||||
return executeStages(ctx, cfg, stages, opts)
|
||||
}
|
||||
newObjectStoreFromConfigFn = func(context.Context, *config.Config) (storage.ObjectStore, error) {
|
||||
objectStoreConstructed = true
|
||||
return nil, context.Canceled
|
||||
}
|
||||
|
||||
stdout.Reset()
|
||||
stderr.Reset()
|
||||
runStageCode := Execute(
|
||||
[]string{
|
||||
"run-stage",
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", cfg.Session.SessionID,
|
||||
"--force",
|
||||
"--artifacts", "session_recap",
|
||||
"analyze",
|
||||
},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if runStageCode != 0 {
|
||||
t.Fatalf("run-stage exit code = %d, want 0; stderr=%q", runStageCode, stderr.String())
|
||||
}
|
||||
if stderr.Len() != 0 {
|
||||
t.Fatalf("run-stage stderr = %q, want empty", stderr.String())
|
||||
}
|
||||
if objectStoreConstructed {
|
||||
t.Fatal("analyze run-stage should not construct object store for previous-session input resolution")
|
||||
}
|
||||
if len(scriptoriumFake.RunRequests) != 1 {
|
||||
t.Fatalf("scriptorium run requests = %d, want 1", len(scriptoriumFake.RunRequests))
|
||||
}
|
||||
req := scriptoriumFake.RunRequests[0]
|
||||
if got := req.InputPaths["transcript"]; got != filepath.Join(sessionRoot, "transcripts", "trimmed.json") {
|
||||
t.Fatalf("transcript input = %q, want trimmed transcript path", got)
|
||||
}
|
||||
if got := req.InputPaths["previous_recap"]; got != filepath.Join(sessionRoot, "previous", "artifacts", "session_recap.md") {
|
||||
t.Fatalf("previous_recap input = %q, want restored previous cache path", got)
|
||||
}
|
||||
}
|
||||
|
||||
func restoreWorkflowManifestJSON(t *testing.T, sessionID, campaign string) []byte {
|
||||
t.Helper()
|
||||
store := &manifest.LocalStore{}
|
||||
now := time.Date(2026, 5, 19, 23, 0, 0, 0, time.UTC)
|
||||
m := manifest.New(sessionID, now)
|
||||
m.Campaign = campaign
|
||||
m.RunID = "20260519T010203Z-a1b2c3d4"
|
||||
stages := []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim"}
|
||||
for i, stageName := range stages {
|
||||
m.MarkStageSucceeded(stageName, now.Add(time.Duration(i+1)*time.Minute), nil)
|
||||
}
|
||||
path := filepath.Join(t.TempDir(), "manifest.json")
|
||||
if err := store.Save(context.Background(), path, m); err != nil {
|
||||
t.Fatalf("save workflow manifest fixture: %v", err)
|
||||
}
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("read workflow manifest fixture: %v", err)
|
||||
}
|
||||
return data
|
||||
}
|
||||
|
||||
func appendRestoreWorkflowScriptoriumConfig(t *testing.T, pipelinePath, extra string) {
|
||||
t.Helper()
|
||||
f, err := os.OpenFile(pipelinePath, os.O_APPEND|os.O_WRONLY, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("open pipeline config for append: %v", err)
|
||||
}
|
||||
defer f.Close()
|
||||
if _, err := f.WriteString(extra); err != nil {
|
||||
t.Fatalf("append pipeline config: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
@@ -16,13 +17,19 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs.SetOutput(io.Discard)
|
||||
|
||||
var pipelinePath string
|
||||
var campaignPath string
|
||||
var sessionPath string
|
||||
var sessionID string
|
||||
var previousSessionID string
|
||||
var force bool
|
||||
var selectedArtifacts artifactSelectionFlag
|
||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||
fs.StringVar(&campaignPath, "campaign", "", "path to campaign.yml (optional; defaults searched)")
|
||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
||||
fs.StringVar(&sessionID, "session-id", "", "session identifier for session.yml templates")
|
||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "previous session identifier for session.yml templates")
|
||||
fs.BoolVar(&force, "force", false, "force stage execution")
|
||||
fs.Var(&selectedArtifacts, "artifacts", "artifact names to execute during analyze (comma-separated or repeatable)")
|
||||
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("resume: invalid flags: %w", err)
|
||||
@@ -30,17 +37,9 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
|
||||
if fs.NArg() != 0 {
|
||||
return fmt.Errorf("resume: unexpected positional arguments")
|
||||
}
|
||||
resolvedPipelinePath, err := resolvePipelineConfigPath(pipelinePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("resume: %w", err)
|
||||
}
|
||||
resolvedSessionPath, err := resolveSessionConfigPath(sessionPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("resume: %w", err)
|
||||
}
|
||||
|
||||
cfg, err := config.LoadWithSessionOptions(resolvedPipelinePath, resolvedSessionPath, config.SessionLoadOptions{
|
||||
SessionID: sessionID,
|
||||
cfg, err := loadCommandConfig(ctx, pipelinePath, campaignPath, sessionPath, config.SessionLoadOptions{
|
||||
SessionID: sessionID,
|
||||
PreviousSessionID: previousSessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("resume: %w", err)
|
||||
@@ -48,6 +47,13 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
return fmt.Errorf("resume: %w", err)
|
||||
}
|
||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
||||
if err != nil {
|
||||
return fmt.Errorf("resume: invalid --artifacts: %w", err)
|
||||
}
|
||||
if err := validateSelectedAnalyzeArtifacts(cfg, normalizedArtifacts); err != nil {
|
||||
return fmt.Errorf("resume: %w", err)
|
||||
}
|
||||
|
||||
full := BuildFullPlan()
|
||||
selected := full
|
||||
@@ -66,7 +72,10 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
|
||||
}
|
||||
}
|
||||
|
||||
summary, err := executeStages(ctx, cfg, selected, RunOptions{Force: force})
|
||||
summary, err := executeStagesFn(ctx, cfg, selected, RunOptions{
|
||||
Force: force,
|
||||
SelectedArtifacts: normalizedArtifacts,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("resume: %w", err)
|
||||
}
|
||||
@@ -83,7 +92,11 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
|
||||
}
|
||||
|
||||
func loadManifestIfPresent(ctx context.Context, cfg *config.Config) (*manifest.Manifest, error) {
|
||||
path := manifestPathFor(cfg)
|
||||
path := artifacts.SessionManifestPathForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
)
|
||||
exists, err := fileExists(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("check manifest %q: %w", path, err)
|
||||
|
||||
@@ -15,8 +15,8 @@ import (
|
||||
|
||||
func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
@@ -25,14 +25,14 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
||||
if err := store.Save(context.Background(), manifestPath, m); err != nil {
|
||||
t.Fatalf("save manifest: %v", err)
|
||||
}
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "raw", "alice.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "autocorrect.yml"), "[]\n")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||
|
||||
var out bytes.Buffer
|
||||
err := Resume(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath}, &out)
|
||||
err := Resume(context.Background(), []string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("Resume() error = %v", err)
|
||||
}
|
||||
@@ -51,8 +51,8 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
||||
|
||||
func TestResumeNoRemainingStages(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
@@ -64,7 +64,7 @@ func TestResumeNoRemainingStages(t *testing.T) {
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
err := Resume(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath}, &out)
|
||||
err := Resume(context.Background(), []string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("Resume() error = %v", err)
|
||||
}
|
||||
@@ -80,8 +80,8 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
|
||||
_, _ = w.Write([]byte(`{"source":"resume-force-test","segments":[{"speaker":"alice"}]}`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
@@ -93,7 +93,7 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
err := Resume(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath, "--force"}, &out)
|
||||
err := Resume(context.Background(), []string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--force"}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("Resume() error = %v", err)
|
||||
}
|
||||
@@ -104,14 +104,14 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
|
||||
|
||||
func TestRunStageExecutesOnlySelectedStage(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||
|
||||
var out bytes.Buffer
|
||||
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath, "polish"}, &out)
|
||||
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "polish"}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("RunStage() error = %v", err)
|
||||
}
|
||||
@@ -134,9 +134,9 @@ func TestRunStageExecutesOnlySelectedStage(t *testing.T) {
|
||||
|
||||
func TestRunStageSkipAndForce(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||
|
||||
@@ -148,7 +148,7 @@ func TestRunStageSkipAndForce(t *testing.T) {
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath, "polish"}, &out)
|
||||
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "polish"}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("RunStage() error = %v", err)
|
||||
}
|
||||
@@ -157,7 +157,7 @@ func TestRunStageSkipAndForce(t *testing.T) {
|
||||
}
|
||||
|
||||
out.Reset()
|
||||
err = RunStage(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath, "--force", "polish"}, &out)
|
||||
err = RunStage(context.Background(), []string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--force", "polish"}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("RunStage(force) error = %v", err)
|
||||
}
|
||||
@@ -166,15 +166,61 @@ func TestRunStageSkipAndForce(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunStageForceMarksDownstreamStaleAndResumeContinuesFromStale(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
|
||||
seed.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||
t.Fatalf("save manifest: %v", err)
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--force", "polish"}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("RunStage(force) error = %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "stage=polish executed=1 skipped=0 force=true") {
|
||||
t.Fatalf("output = %q, want forced polish rerun", out.String())
|
||||
}
|
||||
|
||||
afterForce, err := store.Load(context.Background(), manifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("load manifest after force: %v", err)
|
||||
}
|
||||
for _, name := range []string{"normalize", "trim", "analyze", "archive", "notify"} {
|
||||
if afterForce.Stages[name] == nil || afterForce.Stages[name].Status != manifest.StatusStale {
|
||||
t.Fatalf("stage %q = %#v, want stale", name, afterForce.Stages[name])
|
||||
}
|
||||
}
|
||||
|
||||
out.Reset()
|
||||
err = Resume(context.Background(), []string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("Resume() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "executed=5 skipped=0") {
|
||||
t.Fatalf("output = %q, want resume to execute normalize..notify", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunStageTrimExecutes(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
|
||||
var out bytes.Buffer
|
||||
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath, "trim"}, &out)
|
||||
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "trim"}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("RunStage(trim) error = %v", err)
|
||||
}
|
||||
@@ -197,13 +243,13 @@ func TestRunStageTrimExecutes(t *testing.T) {
|
||||
|
||||
func TestRunStageNormalizeExecutes(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "processed.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
|
||||
var out bytes.Buffer
|
||||
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath, "normalize"}, &out)
|
||||
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "normalize"}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("RunStage(normalize) error = %v", err)
|
||||
}
|
||||
|
||||
@@ -15,13 +15,19 @@ func Run(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs.SetOutput(io.Discard)
|
||||
|
||||
var pipelinePath string
|
||||
var campaignPath string
|
||||
var sessionPath string
|
||||
var sessionID string
|
||||
var previousSessionID string
|
||||
var force bool
|
||||
var selectedArtifacts artifactSelectionFlag
|
||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||
fs.StringVar(&campaignPath, "campaign", "", "path to campaign.yml (optional; defaults searched)")
|
||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
||||
fs.StringVar(&sessionID, "session-id", "", "session identifier for session.yml templates")
|
||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "previous session identifier for session.yml templates")
|
||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
||||
fs.Var(&selectedArtifacts, "artifacts", "artifact names to execute during analyze (comma-separated or repeatable)")
|
||||
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("run: invalid flags: %w", err)
|
||||
@@ -29,17 +35,9 @@ func Run(ctx context.Context, args []string, out io.Writer) error {
|
||||
if fs.NArg() != 0 {
|
||||
return fmt.Errorf("run: unexpected positional arguments")
|
||||
}
|
||||
resolvedPipelinePath, err := resolvePipelineConfigPath(pipelinePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("run: %w", err)
|
||||
}
|
||||
resolvedSessionPath, err := resolveSessionConfigPath(sessionPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("run: %w", err)
|
||||
}
|
||||
|
||||
cfg, err := config.LoadWithSessionOptions(resolvedPipelinePath, resolvedSessionPath, config.SessionLoadOptions{
|
||||
SessionID: sessionID,
|
||||
cfg, err := loadCommandConfig(ctx, pipelinePath, campaignPath, sessionPath, config.SessionLoadOptions{
|
||||
SessionID: sessionID,
|
||||
PreviousSessionID: previousSessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("run: %w", err)
|
||||
@@ -47,9 +45,19 @@ func Run(ctx context.Context, args []string, out io.Writer) error {
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
return fmt.Errorf("run: %w", err)
|
||||
}
|
||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
||||
if err != nil {
|
||||
return fmt.Errorf("run: invalid --artifacts: %w", err)
|
||||
}
|
||||
if err := validateSelectedAnalyzeArtifacts(cfg, normalizedArtifacts); err != nil {
|
||||
return fmt.Errorf("run: %w", err)
|
||||
}
|
||||
|
||||
stages := BuildFullPlan()
|
||||
summary, err := executeStages(ctx, cfg, stages, RunOptions{Force: force})
|
||||
summary, err := executeStagesFn(ctx, cfg, stages, RunOptions{
|
||||
Force: force,
|
||||
SelectedArtifacts: normalizedArtifacts,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("run: %w", err)
|
||||
}
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||
)
|
||||
@@ -49,3 +51,43 @@ func firstNonSucceededIndex(stages []stage.Stage, m *manifest.Manifest) int {
|
||||
}
|
||||
return len(stages)
|
||||
}
|
||||
|
||||
func canonicalStageNames() []string {
|
||||
all := stage.All()
|
||||
out := make([]string, 0, len(all))
|
||||
for _, s := range all {
|
||||
if s == nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, s.Name())
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func downstreamStageNames(stageName string) []string {
|
||||
names := canonicalStageNames()
|
||||
for i, name := range names {
|
||||
if name != stageName {
|
||||
continue
|
||||
}
|
||||
return append([]string(nil), names[i+1:]...)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func invalidateDownstreamSucceededStages(m *manifest.Manifest, upstreamStage string, at time.Time) []string {
|
||||
if m == nil || m.Stages == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
invalidated := make([]string, 0)
|
||||
for _, downstream := range downstreamStageNames(upstreamStage) {
|
||||
sr := m.Stages[downstream]
|
||||
if sr == nil || sr.Status != manifest.StatusSucceeded {
|
||||
continue
|
||||
}
|
||||
m.MarkStageStale(downstream, at, "upstream stage rerun with force")
|
||||
invalidated = append(invalidated, downstream)
|
||||
}
|
||||
return invalidated
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -40,3 +41,48 @@ func TestDecideStageActions(t *testing.T) {
|
||||
t.Fatalf("forced prepare action = %q, want %q", forced[0].Action, stageActionRun)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDownstreamStageNames(t *testing.T) {
|
||||
got := downstreamStageNames("polish")
|
||||
want := []string{"normalize", "trim", "analyze", "archive", "notify"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("downstreamStageNames(polish) = %#v, want %#v", got, want)
|
||||
}
|
||||
|
||||
missing := downstreamStageNames("unknown")
|
||||
if len(missing) != 0 {
|
||||
t.Fatalf("downstreamStageNames(unknown) = %#v, want empty", missing)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInvalidateDownstreamSucceededStages(t *testing.T) {
|
||||
now := time.Now().UTC()
|
||||
m := manifest.New("2026-05-03", now)
|
||||
m.MarkStageSucceeded("prepare", now, nil)
|
||||
m.MarkStageSucceeded("transcribe", now, nil)
|
||||
m.MarkStageSucceeded("merge", now, nil)
|
||||
m.MarkStageSucceeded("polish", now, nil)
|
||||
m.MarkStageSucceeded("normalize", now, nil)
|
||||
m.MarkStageSucceeded("trim", now, nil)
|
||||
m.MarkStageFailed("analyze", now, "analysis failed")
|
||||
m.MarkStageSucceeded("archive", now, nil)
|
||||
m.MarkStageSucceeded("notify", now, nil)
|
||||
|
||||
got := invalidateDownstreamSucceededStages(m, "polish", now.Add(1*time.Second))
|
||||
want := []string{"normalize", "trim", "archive", "notify"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("invalidateDownstreamSucceededStages() = %#v, want %#v", got, want)
|
||||
}
|
||||
|
||||
for _, stageName := range want {
|
||||
if m.Stages[stageName].Status != manifest.StatusStale {
|
||||
t.Fatalf("%s status = %q, want stale", stageName, m.Stages[stageName].Status)
|
||||
}
|
||||
}
|
||||
if m.Stages["analyze"].Status != manifest.StatusFailed {
|
||||
t.Fatalf("analyze status = %q, want failed", m.Stages["analyze"].Status)
|
||||
}
|
||||
if m.Stages["prepare"].Status != manifest.StatusSucceeded {
|
||||
t.Fatalf("prepare status = %q, want succeeded", m.Stages["prepare"].Status)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -15,13 +15,19 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs.SetOutput(io.Discard)
|
||||
|
||||
var pipelinePath string
|
||||
var campaignPath string
|
||||
var sessionPath string
|
||||
var sessionID string
|
||||
var previousSessionID string
|
||||
var force bool
|
||||
var selectedArtifacts artifactSelectionFlag
|
||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||
fs.StringVar(&campaignPath, "campaign", "", "path to campaign.yml (optional; defaults searched)")
|
||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
||||
fs.StringVar(&sessionID, "session-id", "", "session identifier for session.yml templates")
|
||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "previous session identifier for session.yml templates")
|
||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
||||
fs.Var(&selectedArtifacts, "artifacts", "artifact names to execute during analyze (comma-separated or repeatable)")
|
||||
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("run-stage: invalid flags: %w", err)
|
||||
@@ -29,40 +35,34 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
|
||||
if fs.NArg() != 1 {
|
||||
return fmt.Errorf("run-stage: expected exactly one stage name")
|
||||
}
|
||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
||||
if err != nil {
|
||||
return fmt.Errorf("run-stage: invalid --artifacts: %w", err)
|
||||
}
|
||||
stageName := fs.Arg(0)
|
||||
stages, err := BuildSingleStagePlan(stageName)
|
||||
if err != nil {
|
||||
return fmt.Errorf("run-stage: %w", err)
|
||||
if len(normalizedArtifacts) > 0 && stageName != "analyze" {
|
||||
return fmt.Errorf("run-stage: --artifacts is only supported for stage \"analyze\"")
|
||||
}
|
||||
|
||||
resolvedPipelinePath, err := resolvePipelineConfigPath(pipelinePath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("run-stage: %w", err)
|
||||
}
|
||||
resolvedSessionPath, err := resolveSessionConfigPath(sessionPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("run-stage: %w", err)
|
||||
}
|
||||
|
||||
cfg, err := config.LoadWithSessionOptions(resolvedPipelinePath, resolvedSessionPath, config.SessionLoadOptions{
|
||||
SessionID: sessionID,
|
||||
summary, err := runSingleStageCommand(ctx, singleStageCommand{
|
||||
CommandName: "run-stage",
|
||||
StageName: stageName,
|
||||
PipelinePath: pipelinePath,
|
||||
CampaignPath: campaignPath,
|
||||
SessionPath: sessionPath,
|
||||
SessionID: sessionID,
|
||||
PreviousSessionID: previousSessionID,
|
||||
Force: force,
|
||||
SelectedArtifacts: normalizedArtifacts,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("run-stage: %w", err)
|
||||
}
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
return fmt.Errorf("run-stage: %w", err)
|
||||
}
|
||||
|
||||
summary, err := executeStages(ctx, cfg, stages, RunOptions{Force: force})
|
||||
if err != nil {
|
||||
return fmt.Errorf("run-stage: %w", err)
|
||||
return err
|
||||
}
|
||||
|
||||
_, err = fmt.Fprintf(
|
||||
out,
|
||||
"narratio run-stage: stage=%s executed=%d skipped=%d force=%t; manifest=%s\n",
|
||||
stages[0].Name(),
|
||||
stageName,
|
||||
len(summary.Executed),
|
||||
len(summary.Skipped),
|
||||
force,
|
||||
@@ -70,3 +70,99 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
|
||||
)
|
||||
return err
|
||||
}
|
||||
|
||||
// Analyze force-runs the analyze stage.
|
||||
func Analyze(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("analyze", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
|
||||
var pipelinePath string
|
||||
var campaignPath string
|
||||
var sessionPath string
|
||||
var sessionID string
|
||||
var previousSessionID string
|
||||
var selectedArtifacts artifactSelectionFlag
|
||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||
fs.StringVar(&campaignPath, "campaign", "", "path to campaign.yml (optional; defaults searched)")
|
||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
||||
fs.StringVar(&sessionID, "session-id", "", "session identifier for session.yml templates")
|
||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "previous session identifier for session.yml templates")
|
||||
fs.Var(&selectedArtifacts, "artifacts", "artifact names to execute during analyze (comma-separated or repeatable)")
|
||||
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("analyze: invalid flags: %w", err)
|
||||
}
|
||||
if fs.NArg() != 0 {
|
||||
return fmt.Errorf("analyze: unexpected positional arguments")
|
||||
}
|
||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
||||
if err != nil {
|
||||
return fmt.Errorf("analyze: invalid --artifacts: %w", err)
|
||||
}
|
||||
|
||||
summary, err := runSingleStageCommand(ctx, singleStageCommand{
|
||||
CommandName: "analyze",
|
||||
StageName: "analyze",
|
||||
PipelinePath: pipelinePath,
|
||||
CampaignPath: campaignPath,
|
||||
SessionPath: sessionPath,
|
||||
SessionID: sessionID,
|
||||
PreviousSessionID: previousSessionID,
|
||||
Force: true,
|
||||
SelectedArtifacts: normalizedArtifacts,
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
_, err = fmt.Fprintf(
|
||||
out,
|
||||
"narratio analyze: executed=%d skipped=%d force=true; manifest=%s\n",
|
||||
len(summary.Executed),
|
||||
len(summary.Skipped),
|
||||
summary.ManifestPath,
|
||||
)
|
||||
return err
|
||||
}
|
||||
|
||||
type singleStageCommand struct {
|
||||
CommandName string
|
||||
StageName string
|
||||
PipelinePath string
|
||||
CampaignPath string
|
||||
SessionPath string
|
||||
SessionID string
|
||||
PreviousSessionID string
|
||||
Force bool
|
||||
SelectedArtifacts []string
|
||||
}
|
||||
|
||||
func runSingleStageCommand(ctx context.Context, req singleStageCommand) (*RunSummary, error) {
|
||||
stages, err := BuildSingleStagePlan(req.StageName)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", req.CommandName, err)
|
||||
}
|
||||
|
||||
cfg, err := loadCommandConfig(ctx, req.PipelinePath, req.CampaignPath, req.SessionPath, config.SessionLoadOptions{
|
||||
SessionID: req.SessionID,
|
||||
PreviousSessionID: req.PreviousSessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", req.CommandName, err)
|
||||
}
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", req.CommandName, err)
|
||||
}
|
||||
if err := validateSelectedAnalyzeArtifacts(cfg, req.SelectedArtifacts); err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", req.CommandName, err)
|
||||
}
|
||||
|
||||
summary, err := executeStagesFn(ctx, cfg, stages, RunOptions{
|
||||
Force: req.Force,
|
||||
SelectedArtifacts: req.SelectedArtifacts,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", req.CommandName, err)
|
||||
}
|
||||
return summary, nil
|
||||
}
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/audita"
|
||||
@@ -22,18 +21,23 @@ import (
|
||||
)
|
||||
|
||||
type RunOptions struct {
|
||||
Force bool
|
||||
Env *Env
|
||||
Force bool
|
||||
SelectedArtifacts []string
|
||||
Env *Env
|
||||
}
|
||||
|
||||
type RunSummary struct {
|
||||
SessionID string
|
||||
ManifestPath string
|
||||
StageNames []string
|
||||
Executed []string
|
||||
Skipped []string
|
||||
SessionID string
|
||||
RunID string
|
||||
ManifestPath string
|
||||
RunManifestPath string
|
||||
StageNames []string
|
||||
Executed []string
|
||||
Skipped []string
|
||||
}
|
||||
|
||||
var executeStagesFn = executeStages
|
||||
|
||||
func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage, opts RunOptions) (*RunSummary, error) {
|
||||
env := opts.Env
|
||||
if env == nil {
|
||||
@@ -42,6 +46,7 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
if env.Config == nil {
|
||||
env.Config = cfg
|
||||
}
|
||||
env.SelectedAnalyzeArtifacts = append([]string(nil), opts.SelectedArtifacts...)
|
||||
if env.ArtifactStore == nil {
|
||||
env.ArtifactStore = artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
|
||||
}
|
||||
@@ -82,23 +87,30 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
env.Storage = &storage.NoopBackend{}
|
||||
}
|
||||
if env.ObjectStore == nil && needsObjectStoreForRun(env.Config, stages) {
|
||||
objectStore, err := storage.NewObjectStoreFromConfig(ctx, env.Config)
|
||||
objectStore, err := newCommandObjectStore(ctx, env.Config, nil)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("initialize object store backend: %w", err)
|
||||
return nil, err
|
||||
}
|
||||
env.ObjectStore = objectStore
|
||||
}
|
||||
if needsRemoteLocksForRun(env.Config, stages) {
|
||||
locks, err := loadEffectiveLocks(ctx, env.Config, env.ObjectStore)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("load remote archive locks: %w", err)
|
||||
}
|
||||
applyEffectiveLocks(env.Config, locks.All)
|
||||
}
|
||||
if env.Notifier == nil {
|
||||
env.Notifier = ¬ify.NoopSender{}
|
||||
}
|
||||
|
||||
artifactStore := env.ArtifactStore
|
||||
paths, err := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, err := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("prepare workdir: %w", err)
|
||||
}
|
||||
|
||||
lock, err := artifactStore.AcquireSessionLock(cfg.Session.SessionID)
|
||||
lock, err := artifactStore.AcquireSessionLockFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("acquire session lock: %w", err)
|
||||
}
|
||||
@@ -111,7 +123,11 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
identityChanged, err := ensureManifestIdentity(cfg, m)
|
||||
runID, err := artifacts.NewRunID()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("generate run id: %w", err)
|
||||
}
|
||||
identityChanged, err := ensureManifestIdentity(cfg, m, runID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("initialize manifest identity: %w", err)
|
||||
}
|
||||
@@ -120,6 +136,29 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
return nil, fmt.Errorf("save manifest identity %q: %w", manifestPath, err)
|
||||
}
|
||||
}
|
||||
runManifestPath := artifacts.SessionRunManifestPathForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
runID,
|
||||
)
|
||||
runManifestStore := &manifest.LocalStore{}
|
||||
runManifest, err := runManifestStore.CreateRun(
|
||||
ctx,
|
||||
cfg.Session.SessionID,
|
||||
cfg.Session.Campaign,
|
||||
runID,
|
||||
opts.Force,
|
||||
requestedStageNames(stages),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create run manifest: %w", err)
|
||||
}
|
||||
runManifest.SessionManifestPath = manifestPath
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save initial run manifest %q: %w", runManifestPath, err)
|
||||
}
|
||||
|
||||
stageEnv := env
|
||||
|
||||
@@ -134,12 +173,23 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
|
||||
if d.Action == stageActionSkip {
|
||||
skipped = append(skipped, s.Name())
|
||||
skipAt := nowUTC()
|
||||
runManifest.SetStageAction(s.Name(), manifest.RunStageActionSkip, skipAt)
|
||||
runManifest.MarkStageSkipped(s.Name(), skipAt, "already_succeeded")
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save run manifest after skip %q: %w", s.Name(), err)
|
||||
}
|
||||
env.Logger.Info("skipping stage", "stage", s.Name(), "reason", "already_succeeded", "force", opts.Force)
|
||||
continue
|
||||
}
|
||||
executed = append(executed, s.Name())
|
||||
|
||||
now := nowUTC()
|
||||
runManifest.SetStageAction(s.Name(), manifest.RunStageActionRun, now)
|
||||
runManifest.MarkStageRunning(s.Name(), now)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save run manifest before stage %q: %w", s.Name(), err)
|
||||
}
|
||||
m.MarkStageRunning(s.Name(), now)
|
||||
env.Logger.Info("starting stage", "stage", s.Name())
|
||||
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
|
||||
@@ -149,35 +199,65 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
|
||||
result, err := s.Run(ctx, stageEnv, m)
|
||||
if err != nil {
|
||||
m.MarkStageFailed(s.Name(), nowUTC(), err.Error())
|
||||
failedAt := nowUTC()
|
||||
m.MarkStageFailed(s.Name(), failedAt, err.Error())
|
||||
if saveErr := env.ManifestStore.Save(ctx, manifestPath, m); saveErr != nil {
|
||||
return nil, fmt.Errorf("stage %q failed (%v) and manifest save failed (%v)", s.Name(), err, saveErr)
|
||||
}
|
||||
runManifest.MarkStageFailed(s.Name(), failedAt, err.Error())
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
|
||||
return nil, fmt.Errorf("stage %q failed (%v) and run-manifest save failed (%v)", s.Name(), err, saveErr)
|
||||
}
|
||||
env.Logger.Info("stage failed", "stage", s.Name(), "error", err)
|
||||
return nil, fmt.Errorf("stage %q failed: %w", s.Name(), err)
|
||||
}
|
||||
|
||||
outputs := mapResultOutputs(result)
|
||||
m.MarkStageSucceeded(s.Name(), nowUTC(), outputs)
|
||||
outputs := mapResultOutputs(s.Name(), result, runID)
|
||||
succeededAt := nowUTC()
|
||||
m.MarkStageSucceeded(s.Name(), succeededAt, outputs)
|
||||
applyStageResultToManifest(m, s.Name(), result)
|
||||
if opts.Force {
|
||||
invalidateDownstreamSucceededStages(m, s.Name(), succeededAt)
|
||||
}
|
||||
|
||||
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
|
||||
return nil, fmt.Errorf("save manifest after stage %q: %w", s.Name(), err)
|
||||
}
|
||||
runManifest.MarkStageSucceeded(s.Name(), succeededAt, outputs)
|
||||
applyStageResultToRunManifest(runManifest, s.Name(), result)
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save run manifest after stage %q: %w", s.Name(), err)
|
||||
}
|
||||
env.Logger.Debug("manifest saved", "stage", s.Name(), "transition", "succeeded", "path", manifestPath)
|
||||
env.Logger.Info("stage succeeded", "stage", s.Name())
|
||||
}
|
||||
|
||||
if err := runPostArchiveCleanup(ctx, env, manifestPath, m, executed); err != nil {
|
||||
failedAt := nowUTC()
|
||||
runManifest.MarkFailed(failedAt, err.Error())
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
|
||||
return nil, fmt.Errorf("post-archive cleanup failed (%v) and run-manifest save failed (%v)", err, saveErr)
|
||||
}
|
||||
return nil, fmt.Errorf("post-archive cleanup: %w", err)
|
||||
}
|
||||
completedAt := nowUTC()
|
||||
runManifest.MarkSucceeded(completedAt)
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save final run manifest %q: %w", runManifestPath, err)
|
||||
}
|
||||
|
||||
return &RunSummary{
|
||||
SessionID: cfg.Session.SessionID,
|
||||
ManifestPath: manifestPath,
|
||||
StageNames: runNames,
|
||||
Executed: executed,
|
||||
Skipped: skipped,
|
||||
SessionID: cfg.Session.SessionID,
|
||||
RunID: runID,
|
||||
ManifestPath: manifestPath,
|
||||
RunManifestPath: runManifestPath,
|
||||
StageNames: runNames,
|
||||
Executed: executed,
|
||||
Skipped: skipped,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -317,22 +397,31 @@ func fileExists(path string) (bool, error) {
|
||||
return false, err
|
||||
}
|
||||
|
||||
func mapResultOutputs(result *stage.StageResult) []manifest.ArtifactRecord {
|
||||
func mapResultOutputs(stageName string, result *stage.StageResult, runID string) []manifest.ArtifactRecord {
|
||||
if result == nil || len(result.Outputs) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
runID = strings.TrimSpace(runID)
|
||||
out := make([]manifest.ArtifactRecord, 0, len(result.Outputs))
|
||||
for _, ref := range result.Outputs {
|
||||
localPath := ref.AbsolutePath
|
||||
if localPath == "" {
|
||||
localPath = ref.RelativePath
|
||||
}
|
||||
kind := ref.Kind
|
||||
sourceID := ""
|
||||
if stageName == "analyze" {
|
||||
sourceID = artifacts.ConfiguredArtifactSourceID(ref.Kind)
|
||||
kind = "scriptorium_artifact"
|
||||
}
|
||||
out = append(out, manifest.ArtifactRecord{
|
||||
Kind: ref.Kind,
|
||||
LocalPath: localPath,
|
||||
RemoteKey: ref.RemoteKey,
|
||||
Checksum: ref.Checksum,
|
||||
Kind: kind,
|
||||
SourceID: sourceID,
|
||||
LocalPath: localPath,
|
||||
ProducerRunID: runID,
|
||||
RemoteKey: ref.RemoteKey,
|
||||
Checksum: ref.Checksum,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -358,7 +447,7 @@ func applyStageResultToManifest(m *manifest.Manifest, stageName string, result *
|
||||
}
|
||||
}
|
||||
|
||||
func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, error) {
|
||||
func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest, runID string) (bool, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil || m == nil {
|
||||
return false, nil
|
||||
}
|
||||
@@ -374,16 +463,13 @@ func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, err
|
||||
m.Campaign = campaign
|
||||
changed = true
|
||||
}
|
||||
if m.RunID == "" {
|
||||
runID, err := artifacts.NewRunID()
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
runID = strings.TrimSpace(runID)
|
||||
if runID != "" && m.RunID != runID {
|
||||
m.RunID = runID
|
||||
changed = true
|
||||
}
|
||||
if m.LocalWorkDir == "" && campaign != "" && sessionID != "" && m.RunID != "" {
|
||||
m.LocalWorkDir = artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, campaign, sessionID, m.RunID)
|
||||
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, campaign, sessionID, m.RunID)
|
||||
changed = true
|
||||
}
|
||||
if m.LocalSpoolDir == "" && campaign != "" && sessionID != "" && m.RunID != "" && strings.TrimSpace(cfg.Pipeline.Spool.Root) != "" {
|
||||
@@ -410,8 +496,54 @@ func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, err
|
||||
return changed, nil
|
||||
}
|
||||
|
||||
func requestedStageNames(stages []stage.Stage) []string {
|
||||
out := make([]string, 0, len(stages))
|
||||
for _, s := range stages {
|
||||
if s == nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, s.Name())
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func applyStageResultToRunManifest(m *manifest.RunManifest, stageName string, result *stage.StageResult) {
|
||||
if m == nil || result == nil {
|
||||
return
|
||||
}
|
||||
sr := m.Stages[stageName]
|
||||
if sr == nil {
|
||||
return
|
||||
}
|
||||
if len(result.Logs) > 0 {
|
||||
sr.Logs = append([]string(nil), result.Logs...)
|
||||
}
|
||||
if len(result.GeneratedConfigs) > 0 {
|
||||
sr.GeneratedConfigs = append([]string(nil), result.GeneratedConfigs...)
|
||||
}
|
||||
if len(result.Metadata) > 0 {
|
||||
sr.Metadata = result.Metadata
|
||||
}
|
||||
}
|
||||
|
||||
func syncRunManifestIdentityFromSession(session *manifest.Manifest, run *manifest.RunManifest) {
|
||||
if session == nil || run == nil {
|
||||
return
|
||||
}
|
||||
run.Campaign = session.Campaign
|
||||
run.LocalWorkDir = session.LocalWorkDir
|
||||
run.LocalSpoolDir = session.LocalSpoolDir
|
||||
run.S3Bucket = session.S3Bucket
|
||||
run.S3SessionPrefix = session.S3SessionPrefix
|
||||
run.S3RunPrefix = session.S3RunPrefix
|
||||
}
|
||||
|
||||
func manifestPathFor(cfg *config.Config) string {
|
||||
return filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "manifest.json")
|
||||
return artifacts.SessionManifestPathForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
)
|
||||
}
|
||||
|
||||
func needsObjectStoreForRun(cfg *config.Config, stages []stage.Stage) bool {
|
||||
@@ -430,6 +562,12 @@ func needsObjectStoreForRun(cfg *config.Config, stages []stage.Stage) bool {
|
||||
if cfg.Session.Inputs.AudioS3 != nil && stageRequested("prepare") {
|
||||
return true
|
||||
}
|
||||
if stageRequested("prepare") {
|
||||
requirements := artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg))
|
||||
if len(requirements) > 0 && strings.TrimSpace(cfg.Session.PreviousSessionID) != "" {
|
||||
return true
|
||||
}
|
||||
}
|
||||
if !stageRequested("archive") {
|
||||
return false
|
||||
}
|
||||
@@ -444,3 +582,36 @@ func needsObjectStoreForRun(cfg *config.Config, stages []stage.Stage) bool {
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func needsRemoteLocksForRun(cfg *config.Config, stages []stage.Stage) bool {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
||||
return false
|
||||
}
|
||||
archiveRequested := false
|
||||
for _, s := range stages {
|
||||
if s != nil && s.Name() == "archive" {
|
||||
archiveRequested = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !archiveRequested {
|
||||
return false
|
||||
}
|
||||
if cfg.Pipeline.Archive == nil {
|
||||
return false
|
||||
}
|
||||
if cfg.Pipeline.Archive.Enabled != nil && !*cfg.Pipeline.Archive.Enabled {
|
||||
return false
|
||||
}
|
||||
if cfg.Pipeline.Archive.UploadRun != nil && !*cfg.Pipeline.Archive.UploadRun {
|
||||
return false
|
||||
}
|
||||
return cfg.Pipeline.Storage.S3 != nil
|
||||
}
|
||||
|
||||
func configuredScriptoriumArtifacts(cfg *config.Config) map[string]config.ScriptoriumArtifactConfig {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Scriptorium == nil {
|
||||
return nil
|
||||
}
|
||||
return cfg.Pipeline.Scriptorium.Artifacts
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@ package app
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -44,6 +45,269 @@ func (s countingStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest
|
||||
return &stage.StageResult{Metadata: map[string]any{"counting": true}}, nil
|
||||
}
|
||||
|
||||
type captureSelectedArtifactsStage struct {
|
||||
name string
|
||||
captured *[]string
|
||||
}
|
||||
|
||||
func (s captureSelectedArtifactsStage) Name() string { return s.name }
|
||||
func (s captureSelectedArtifactsStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||
func (s captureSelectedArtifactsStage) Run(_ context.Context, env *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
||||
if s.captured != nil {
|
||||
*s.captured = append((*s.captured)[:0], env.SelectedAnalyzeArtifacts...)
|
||||
}
|
||||
return &stage.StageResult{Metadata: map[string]any{"captured": true}}, nil
|
||||
}
|
||||
|
||||
type analyzeOutputStage struct {
|
||||
output artifacts.Ref
|
||||
}
|
||||
|
||||
func (s analyzeOutputStage) Name() string { return "analyze" }
|
||||
func (s analyzeOutputStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||
func (s analyzeOutputStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
||||
return &stage.StageResult{
|
||||
Outputs: []artifacts.Ref{s.output},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type selectedAnalyzeArtifactStage struct {
|
||||
expected []string
|
||||
}
|
||||
|
||||
func (s selectedAnalyzeArtifactStage) Name() string { return "analyze" }
|
||||
func (s selectedAnalyzeArtifactStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||
func (s selectedAnalyzeArtifactStage) Run(_ context.Context, env *stage.Env, m *manifest.Manifest) (*stage.StageResult, error) {
|
||||
if len(env.SelectedAnalyzeArtifacts) != len(s.expected) {
|
||||
return nil, fmt.Errorf("selected artifacts len = %d, want %d", len(env.SelectedAnalyzeArtifacts), len(s.expected))
|
||||
}
|
||||
for i := range s.expected {
|
||||
if env.SelectedAnalyzeArtifacts[i] != s.expected[i] {
|
||||
return nil, fmt.Errorf("selected artifacts[%d] = %q, want %q", i, env.SelectedAnalyzeArtifacts[i], s.expected[i])
|
||||
}
|
||||
}
|
||||
|
||||
outputPath := filepath.Join(
|
||||
artifacts.SessionWorkDirForCampaign(env.Config.Pipeline.Workspace.Root, env.Config.Session.Campaign, m.SessionID),
|
||||
"artifacts",
|
||||
"player_handout.md",
|
||||
)
|
||||
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
|
||||
return nil, fmt.Errorf("mkdir artifact dir: %w", err)
|
||||
}
|
||||
if err := os.WriteFile(outputPath, []byte("player handout\n"), 0o644); err != nil {
|
||||
return nil, fmt.Errorf("write player handout: %w", err)
|
||||
}
|
||||
|
||||
return &stage.StageResult{
|
||||
Outputs: []artifacts.Ref{
|
||||
{
|
||||
Kind: "player_handout",
|
||||
Category: "artifacts",
|
||||
RelativePath: "artifacts/player_handout.md",
|
||||
AbsolutePath: outputPath,
|
||||
},
|
||||
},
|
||||
Metadata: map[string]any{
|
||||
"stage": "analyze",
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func TestExecuteStagesPropagatesSelectedArtifactsToEnv(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
|
||||
captured := []string{}
|
||||
stageToRun := captureSelectedArtifactsStage{name: "analyze", captured: &captured}
|
||||
_, err := executeStages(context.Background(), cfg, []stage.Stage{stageToRun}, RunOptions{
|
||||
SelectedArtifacts: []string{"player_handout", "session_recap"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
|
||||
if len(captured) != 2 {
|
||||
t.Fatalf("captured len = %d, want 2 (%v)", len(captured), captured)
|
||||
}
|
||||
if captured[0] != "player_handout" || captured[1] != "session_recap" {
|
||||
t.Fatalf("captured = %v, want [player_handout session_recap]", captured)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesAnalyzeOutputsPersistAsScriptoriumArtifacts(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
storeForPaths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
|
||||
sessionPaths := storeForPaths.SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
outputPath := filepath.Join(sessionPaths.ArtifactsDir, "session_recap.md")
|
||||
|
||||
stageToRun := analyzeOutputStage{
|
||||
output: artifacts.Ref{
|
||||
Kind: "session_recap",
|
||||
Category: "artifacts",
|
||||
RelativePath: "artifacts/session_recap.md",
|
||||
AbsolutePath: outputPath,
|
||||
},
|
||||
}
|
||||
summary, err := executeStages(context.Background(), cfg, []stage.Stage{stageToRun}, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
sessionManifest, err := store.Load(context.Background(), summary.ManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("load session manifest: %v", err)
|
||||
}
|
||||
sessionStage := sessionManifest.Stages["analyze"]
|
||||
if sessionStage == nil {
|
||||
t.Fatal("session manifest analyze stage missing")
|
||||
}
|
||||
if len(sessionStage.Outputs) != 1 {
|
||||
t.Fatalf("session analyze outputs len = %d, want 1", len(sessionStage.Outputs))
|
||||
}
|
||||
sessionOutput := sessionStage.Outputs[0]
|
||||
if sessionOutput.Kind != "scriptorium_artifact" {
|
||||
t.Fatalf("session output kind = %q, want scriptorium_artifact", sessionOutput.Kind)
|
||||
}
|
||||
if sessionOutput.SourceID != "narratio.artifact.session_recap" {
|
||||
t.Fatalf("session output source_id = %q, want narratio.artifact.session_recap", sessionOutput.SourceID)
|
||||
}
|
||||
if sessionOutput.LocalPath != outputPath {
|
||||
t.Fatalf("session output local_path = %q, want %q", sessionOutput.LocalPath, outputPath)
|
||||
}
|
||||
|
||||
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("load run manifest: %v", err)
|
||||
}
|
||||
runStage := runManifest.Stages["analyze"]
|
||||
if runStage == nil {
|
||||
t.Fatal("run manifest analyze stage missing")
|
||||
}
|
||||
if len(runStage.Outputs) != 1 {
|
||||
t.Fatalf("run analyze outputs len = %d, want 1", len(runStage.Outputs))
|
||||
}
|
||||
runOutput := runStage.Outputs[0]
|
||||
if runOutput.Kind != "scriptorium_artifact" {
|
||||
t.Fatalf("run output kind = %q, want scriptorium_artifact", runOutput.Kind)
|
||||
}
|
||||
if runOutput.SourceID != "narratio.artifact.session_recap" {
|
||||
t.Fatalf("run output source_id = %q, want narratio.artifact.session_recap", runOutput.SourceID)
|
||||
}
|
||||
if runOutput.LocalPath != outputPath {
|
||||
t.Fatalf("run output local_path = %q, want %q", runOutput.LocalPath, outputPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNeedsObjectStoreForRunPrepareWithPreviousRequirements(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
previousSessionID string
|
||||
want bool
|
||||
}{
|
||||
{
|
||||
name: "previous session configured",
|
||||
previousSessionID: "2026-05-10",
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "previous session missing",
|
||||
previousSessionID: "",
|
||||
want: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
cfg := &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Scriptorium: &config.ScriptoriumConfig{
|
||||
Artifacts: map[string]config.ScriptoriumArtifactConfig{
|
||||
"session_recap": {
|
||||
Enabled: true,
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"previous_recap": {
|
||||
Source: "narratio.previous_session.artifact.session_recap",
|
||||
Required: true,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
PreviousSessionID: tt.previousSessionID,
|
||||
},
|
||||
}
|
||||
|
||||
got := needsObjectStoreForRun(cfg, []stage.Stage{countingStage{name: "prepare", runs: new(int)}})
|
||||
if got != tt.want {
|
||||
t.Fatalf("needsObjectStoreForRun() = %v, want %v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesArchiveFailsWhenRequiredRecapPromotionMissingForSelectedArtifacts(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
cfg.Pipeline.Storage.S3 = &config.StorageS3Config{
|
||||
Bucket: "my-dnd-archive",
|
||||
RootPrefix: "dnd",
|
||||
}
|
||||
cfg.Pipeline.Archive = &config.ArchiveConfig{
|
||||
Enabled: boolPtr(true),
|
||||
UploadRun: boolPtr(true),
|
||||
PromoteArtifacts: []config.ArchivePromotionRule{
|
||||
{Source: "narratio.artifact.session_recap", Dest: "artifacts/session_recap.md", Required: boolPtr(true)},
|
||||
},
|
||||
}
|
||||
cfg.Pipeline.Scriptorium = &config.ScriptoriumConfig{
|
||||
Artifacts: map[string]config.ScriptoriumArtifactConfig{
|
||||
"session_recap": {
|
||||
OutputPath: "artifacts/session_recap.md",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
manifestPath := manifestPathFor(cfg)
|
||||
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
||||
seed.Campaign = cfg.Session.Campaign
|
||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim"} {
|
||||
seed.MarkStageSucceeded(stageName, time.Now().UTC(), nil)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||
t.Fatalf("Save manifest error = %v", err)
|
||||
}
|
||||
|
||||
archiveStageImpl, err := stage.Select("archive")
|
||||
if err != nil {
|
||||
t.Fatalf("Select(archive) error = %v", err)
|
||||
}
|
||||
|
||||
_, err = executeStages(
|
||||
context.Background(),
|
||||
cfg,
|
||||
[]stage.Stage{
|
||||
selectedAnalyzeArtifactStage{expected: []string{"player_handout"}},
|
||||
archiveStageImpl,
|
||||
},
|
||||
RunOptions{
|
||||
SelectedArtifacts: []string{"player_handout"},
|
||||
Env: &Env{ObjectStore: &storage.FakeBackend{}},
|
||||
},
|
||||
)
|
||||
if err == nil {
|
||||
t.Fatal("expected archive promotion failure, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "required promotion source unavailable") {
|
||||
t.Fatalf("error = %q, want required promotion source unavailable", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesPlaceholderSuccessUpdatesManifest(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
|
||||
@@ -235,6 +499,56 @@ func TestExecuteStagesForceRerunsSucceeded(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesForceSuccessInvalidatesDownstreamSucceededStages(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
manifestPath := manifestPathFor(cfg)
|
||||
store := &manifest.LocalStore{}
|
||||
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "archive", "notify"} {
|
||||
existing.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
||||
}
|
||||
existing.MarkStageFailed("analyze", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), "previous analyze failure")
|
||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
|
||||
t.Fatalf("Save manifest error = %v", err)
|
||||
}
|
||||
|
||||
runs := 0
|
||||
stageToRun := countingStage{name: "polish", runs: &runs}
|
||||
summary, err := executeStages(context.Background(), cfg, []stage.Stage{stageToRun}, RunOptions{Force: true})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if runs != 1 {
|
||||
t.Fatalf("runs = %d, want 1 with force", runs)
|
||||
}
|
||||
if len(summary.Executed) != 1 || summary.Executed[0] != "polish" || len(summary.Skipped) != 0 {
|
||||
t.Fatalf("summary = %#v, want executed polish", summary)
|
||||
}
|
||||
|
||||
loaded, err := store.Load(context.Background(), manifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load manifest error = %v", err)
|
||||
}
|
||||
if loaded.Stages["polish"] == nil || loaded.Stages["polish"].Status != manifest.StatusSucceeded {
|
||||
t.Fatalf("polish status = %#v, want succeeded", loaded.Stages["polish"])
|
||||
}
|
||||
for _, stageName := range []string{"normalize", "trim", "archive", "notify"} {
|
||||
if loaded.Stages[stageName] == nil || loaded.Stages[stageName].Status != manifest.StatusStale {
|
||||
t.Fatalf("%s status = %#v, want stale", stageName, loaded.Stages[stageName])
|
||||
}
|
||||
}
|
||||
if loaded.Stages["analyze"] == nil || loaded.Stages["analyze"].Status != manifest.StatusFailed {
|
||||
t.Fatalf("analyze status = %#v, want preserved failed", loaded.Stages["analyze"])
|
||||
}
|
||||
if loaded.Stages["transcribe"] == nil || loaded.Stages["transcribe"].Status != manifest.StatusSucceeded {
|
||||
t.Fatalf("transcribe status = %#v, want preserved succeeded", loaded.Stages["transcribe"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesFailureUpdatesManifest(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
|
||||
@@ -280,7 +594,14 @@ func TestExecuteStagesLoadsExistingManifest(t *testing.T) {
|
||||
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
existing.MarkStageSucceeded("prepare", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
||||
audioPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "audio", "alice.flac")
|
||||
audioPath := filepath.Join(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
"work",
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
"audio",
|
||||
"alice.flac",
|
||||
)
|
||||
if err := os.MkdirAll(filepath.Dir(audioPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
@@ -315,6 +636,208 @@ func TestExecuteStagesLoadsExistingManifest(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesCreatesRunManifestPerInvocation(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
run1, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[0]}, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("first executeStages() error = %v", err)
|
||||
}
|
||||
run2, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[0]}, RunOptions{Force: true})
|
||||
if err != nil {
|
||||
t.Fatalf("second executeStages() error = %v", err)
|
||||
}
|
||||
|
||||
if run1.RunID == "" || run2.RunID == "" {
|
||||
t.Fatalf("run ids must be set, got %q and %q", run1.RunID, run2.RunID)
|
||||
}
|
||||
if run1.RunID == run2.RunID {
|
||||
t.Fatalf("expected distinct run ids, got %q", run1.RunID)
|
||||
}
|
||||
if run1.RunManifestPath == "" || run2.RunManifestPath == "" {
|
||||
t.Fatalf("run manifest paths must be set, got %q and %q", run1.RunManifestPath, run2.RunManifestPath)
|
||||
}
|
||||
if run1.RunManifestPath == run2.RunManifestPath {
|
||||
t.Fatalf("expected distinct run manifest paths, got %q", run1.RunManifestPath)
|
||||
}
|
||||
for _, path := range []string{run1.RunManifestPath, run2.RunManifestPath} {
|
||||
if _, statErr := os.Stat(path); statErr != nil {
|
||||
t.Fatalf("run manifest missing at %q: %v", path, statErr)
|
||||
}
|
||||
}
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
sessionManifest, err := store.Load(context.Background(), run2.ManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load session manifest error = %v", err)
|
||||
}
|
||||
if sessionManifest.RunID != run2.RunID {
|
||||
t.Fatalf("session manifest run_id = %q, want latest run id %q", sessionManifest.RunID, run2.RunID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesRunManifestRecordsSkippedStage(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
store := &manifest.LocalStore{}
|
||||
manifestPath := manifestPathFor(cfg)
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
existing.MarkStageSucceeded("transcribe", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
|
||||
t.Fatalf("Save manifest error = %v", err)
|
||||
}
|
||||
|
||||
summary, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[1]}, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if len(summary.Skipped) != 1 || summary.Skipped[0] != "transcribe" {
|
||||
t.Fatalf("summary = %#v, want skipped transcribe", summary)
|
||||
}
|
||||
|
||||
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadRun() error = %v", err)
|
||||
}
|
||||
sr := runManifest.Stages["transcribe"]
|
||||
if sr == nil {
|
||||
t.Fatal("run manifest transcribe stage missing")
|
||||
}
|
||||
if sr.Action != manifest.RunStageActionSkip {
|
||||
t.Fatalf("action = %q, want %q", sr.Action, manifest.RunStageActionSkip)
|
||||
}
|
||||
if sr.Status != manifest.StatusSkipped {
|
||||
t.Fatalf("status = %q, want %q", sr.Status, manifest.StatusSkipped)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesRunLocalArtifactsAndCanonicalPromotion(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
stages := []stage.Stage{
|
||||
BuildFullPlan()[0], // prepare
|
||||
BuildFullPlan()[1], // transcribe
|
||||
BuildFullPlan()[2], // merge
|
||||
BuildFullPlan()[3], // polish
|
||||
BuildFullPlan()[4], // normalize
|
||||
BuildFullPlan()[5], // trim
|
||||
}
|
||||
|
||||
summary, err := executeStages(context.Background(), cfg, stages, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if summary.RunID == "" {
|
||||
t.Fatal("run id must be set")
|
||||
}
|
||||
|
||||
runRoot := artifacts.SessionRunRootForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
summary.RunID,
|
||||
)
|
||||
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
|
||||
runLocalChecks := []string{
|
||||
filepath.Join(runRoot, "transcribe", "outputs", "transcripts", "raw", "alice.json"),
|
||||
filepath.Join(runRoot, "merge", "logs", "seriatim.stdout.log"),
|
||||
filepath.Join(runRoot, "polish", "config", "audita.generated.yml"),
|
||||
filepath.Join(runRoot, "normalize", "logs", "seriatim.normalize.stdout.log"),
|
||||
filepath.Join(runRoot, "trim", "outputs", "transcripts", "trimmed.json"),
|
||||
}
|
||||
for _, p := range runLocalChecks {
|
||||
if _, statErr := os.Stat(p); statErr != nil {
|
||||
t.Fatalf("run-local artifact missing at %q: %v", p, statErr)
|
||||
}
|
||||
}
|
||||
|
||||
canonicalChecks := []string{
|
||||
filepath.Join(paths.TranscriptsRawDir, "alice.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "merged.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "processed.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "normalized.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "trimmed.json"),
|
||||
}
|
||||
for _, p := range canonicalChecks {
|
||||
if _, statErr := os.Stat(p); statErr != nil {
|
||||
t.Fatalf("canonical promoted artifact missing at %q: %v", p, statErr)
|
||||
}
|
||||
}
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
sessionManifest, err := store.Load(context.Background(), summary.ManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load manifest error = %v", err)
|
||||
}
|
||||
if got := sessionManifest.Stages["trim"]; got == nil || len(got.Outputs) == 0 {
|
||||
t.Fatalf("trim stage outputs missing in session manifest: %#v", got)
|
||||
}
|
||||
for _, out := range sessionManifest.Stages["trim"].Outputs {
|
||||
if strings.Contains(out.LocalPath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("session manifest output should be canonical, got run-local path %q", out.LocalPath)
|
||||
}
|
||||
if out.ProducerRunID != summary.RunID {
|
||||
t.Fatalf("producer_run_id = %q, want %q", out.ProducerRunID, summary.RunID)
|
||||
}
|
||||
}
|
||||
|
||||
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadRun() error = %v", err)
|
||||
}
|
||||
mergeStage := runManifest.Stages["merge"]
|
||||
if mergeStage == nil || len(mergeStage.Logs) == 0 {
|
||||
t.Fatalf("merge logs missing in run manifest: %#v", mergeStage)
|
||||
}
|
||||
for _, logPath := range mergeStage.Logs {
|
||||
if !strings.Contains(logPath, filepath.Join("runs", summary.RunID, "merge", "logs")) {
|
||||
t.Fatalf("run manifest merge log path = %q, want run-local merge logs path", logPath)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesSkippedStagePreservesExistingOutputsProvenance(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
store := &manifest.LocalStore{}
|
||||
manifestPath := manifestPathFor(cfg)
|
||||
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
existing.MarkStageSucceeded("transcribe", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), []manifest.ArtifactRecord{
|
||||
{
|
||||
Kind: "transcript_raw",
|
||||
LocalPath: "transcripts/raw/alice.json",
|
||||
ProducerRunID: "20260501T000000Z-deadbeef",
|
||||
},
|
||||
})
|
||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
|
||||
t.Fatalf("Save manifest error = %v", err)
|
||||
}
|
||||
|
||||
summary, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[1]}, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if len(summary.Skipped) != 1 || summary.Skipped[0] != "transcribe" {
|
||||
t.Fatalf("summary = %#v, want skipped transcribe", summary)
|
||||
}
|
||||
|
||||
loaded, err := store.Load(context.Background(), manifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load manifest error = %v", err)
|
||||
}
|
||||
got := loaded.Stages["transcribe"]
|
||||
if got == nil || len(got.Outputs) != 1 {
|
||||
t.Fatalf("transcribe outputs = %#v, want one preserved output", got)
|
||||
}
|
||||
if got.Outputs[0].ProducerRunID != "20260501T000000Z-deadbeef" {
|
||||
t.Fatalf("producer_run_id = %q, want preserved value", got.Outputs[0].ProducerRunID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
@@ -341,7 +864,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
tc.env.ArtifactStore = artifactStore
|
||||
tc.env.ManifestStore = &manifest.LocalStore{}
|
||||
if tc.name == "transcribe" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
@@ -356,7 +879,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
if tc.name == "merge" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
@@ -375,7 +898,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
if tc.name == "polish" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
@@ -387,7 +910,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
if tc.name == "analyze" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
@@ -403,7 +926,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
PromptID: "dnd.session_recap",
|
||||
OutputPath: "artifacts/session_recap.md",
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"transcript": {Source: "processed_transcript", Required: true},
|
||||
"transcript": {Source: "narratio.transcript.polished", Required: true},
|
||||
},
|
||||
},
|
||||
},
|
||||
@@ -471,10 +994,12 @@ func testConfig(t *testing.T) *config.Config {
|
||||
workspace := t.TempDir()
|
||||
cfgDir := t.TempDir()
|
||||
sessionPath := filepath.Join(cfgDir, "session.yml")
|
||||
campaignPath := filepath.Join(cfgDir, "campaign.yml")
|
||||
pipelinePath := filepath.Join(cfgDir, "pipeline.yml")
|
||||
|
||||
mustWriteFile(t, pipelinePath, "workspace:\n root: "+workspace+"\n")
|
||||
mustWriteFile(t, sessionPath, "session_id: 2026-05-03\ncampaign: sample-campaign\n")
|
||||
mustWriteFile(t, campaignPath, "campaign: sample-campaign\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n")
|
||||
mustWriteFile(t, sessionPath, "session_id: 2026-05-03\ncampaign: sample-campaign\ninputs:\n audio_dir: ./audio\n")
|
||||
mustWriteFile(t, filepath.Join(cfgDir, "speakers.yml"), "alice: alice.flac\n")
|
||||
mustWriteFile(t, filepath.Join(cfgDir, "autocorrect.yml"), "[]\n")
|
||||
mustWriteFile(t, filepath.Join(cfgDir, "glossary.yml"), "[]\n")
|
||||
@@ -482,8 +1007,27 @@ func testConfig(t *testing.T) *config.Config {
|
||||
|
||||
return &config.Config{
|
||||
Pipeline: &config.PipelineConfig{Workspace: config.WorkspaceConfig{Root: workspace}},
|
||||
Campaign: &config.CampaignConfig{Campaign: "sample-campaign"},
|
||||
PipelinePath: pipelinePath,
|
||||
CampaignPath: campaignPath,
|
||||
SessionPath: sessionPath,
|
||||
StableInputs: config.ResolvedStableInputs{
|
||||
SpeakersFile: config.ResolvedInputFile{
|
||||
Path: "./speakers.yml",
|
||||
ConfigPath: campaignPath,
|
||||
Source: "campaign_config",
|
||||
},
|
||||
AutocorrectFile: config.ResolvedInputFile{
|
||||
Path: "./autocorrect.yml",
|
||||
ConfigPath: campaignPath,
|
||||
Source: "campaign_config",
|
||||
},
|
||||
GlossaryFile: config.ResolvedInputFile{
|
||||
Path: "./glossary.yml",
|
||||
ConfigPath: campaignPath,
|
||||
Source: "campaign_config",
|
||||
},
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
@@ -500,6 +1044,7 @@ func testConfig(t *testing.T) *config.Config {
|
||||
func TestBuildDefaultRunnersWithOmittedToolSections(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
pipelinePath := filepath.Join(dir, "pipeline.yml")
|
||||
campaignPath := filepath.Join(dir, "campaign.yml")
|
||||
sessionPath := filepath.Join(dir, "session.yml")
|
||||
pipelineYAML := `workspace:
|
||||
root: ` + t.TempDir() + `
|
||||
@@ -509,6 +1054,12 @@ analyzer:
|
||||
timeout: 20m
|
||||
notification:
|
||||
timeout: 10s
|
||||
`
|
||||
campaignYAML := `campaign: sample-campaign
|
||||
inputs:
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`
|
||||
sessionYAML := `session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
@@ -519,6 +1070,7 @@ inputs:
|
||||
glossary_file: ./glossary.yml
|
||||
`
|
||||
mustWriteFile(t, pipelinePath, pipelineYAML)
|
||||
mustWriteFile(t, campaignPath, campaignYAML)
|
||||
mustWriteFile(t, sessionPath, sessionYAML)
|
||||
|
||||
cfg, err := config.Load(pipelinePath, sessionPath)
|
||||
|
||||
@@ -4,16 +4,18 @@ import (
|
||||
"bytes"
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func TestPlanUsesDiscoveredSessionTemplateWithSessionID(t *testing.T) {
|
||||
func TestPlanUsesDiscoveredSessionTemplateWithSessionIDs(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
sessionTemplate := `session_id: "{{ session_id }}"
|
||||
previous_session_id: "{{ previous_session_id }}"
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
@@ -25,18 +27,17 @@ inputs:
|
||||
t.Fatalf("write session template: %v", err)
|
||||
}
|
||||
|
||||
cwd := filepath.Dir(sessionPath)
|
||||
originalWD, err := os.Getwd()
|
||||
if err != nil {
|
||||
t.Fatalf("Getwd(): %v", err)
|
||||
}
|
||||
if err := os.Chdir(cwd); err != nil {
|
||||
t.Fatalf("Chdir(%q): %v", cwd, err)
|
||||
}
|
||||
t.Cleanup(func() { _ = os.Chdir(originalWD) })
|
||||
origSessionDefaults := append([]string(nil), config.DefaultSessionConfigSearchPaths...)
|
||||
config.DefaultSessionConfigSearchPaths = []string{sessionPath}
|
||||
t.Cleanup(func() { config.DefaultSessionConfigSearchPaths = origSessionDefaults })
|
||||
|
||||
var out bytes.Buffer
|
||||
if err := Plan(context.Background(), []string{"--config", pipelinePath, "--session-id", "2026-04-04"}, &out); err != nil {
|
||||
if err := Plan(context.Background(), []string{
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session-id", "2026-04-04",
|
||||
"--previous-session-id", "2026-03-28",
|
||||
}, &out); err != nil {
|
||||
t.Fatalf("Plan() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "narratio plan: workdir prepared") {
|
||||
@@ -46,10 +47,10 @@ inputs:
|
||||
|
||||
func TestPlanFailsWhenSessionIDMismatchesConcreteSession(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
var out bytes.Buffer
|
||||
err := Plan(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath, "--session-id", "2026-04-04"}, &out)
|
||||
err := Plan(context.Background(), []string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--session-id", "2026-04-04"}, &out)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
@@ -58,12 +59,45 @@ func TestPlanFailsWhenSessionIDMismatchesConcreteSession(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunStageAcceptsSessionIDFlagAndParsesStageName(t *testing.T) {
|
||||
func TestPlanFailsWhenPreviousSessionIDMismatchesConcreteSession(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
sessionYAML := `session_id: 2026-05-03
|
||||
previous_session_id: 2026-04-26
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`
|
||||
if err := os.WriteFile(sessionPath, []byte(sessionYAML), 0o644); err != nil {
|
||||
t.Fatalf("write session.yml: %v", err)
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath, "--session-id", "2026-05-03", "prepare"}, &out)
|
||||
err := Plan(context.Background(), []string{
|
||||
"--config", pipelinePath,
|
||||
"--campaign", campaignPath,
|
||||
"--session", sessionPath,
|
||||
"--session-id", "2026-05-03",
|
||||
"--previous-session-id", "2026-04-25",
|
||||
}, &out)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "previous_session_id mismatch") {
|
||||
t.Fatalf("error = %q, want mismatch context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunStageAcceptsSessionIDFlagAndParsesStageName(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
|
||||
var out bytes.Buffer
|
||||
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--campaign", campaignPath, "--session", sessionPath, "--session-id", "2026-05-03", "prepare"}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("RunStage() error = %v", err)
|
||||
}
|
||||
@@ -73,7 +107,7 @@ func TestRunStageAcceptsSessionIDFlagAndParsesStageName(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestResolveSessionConfigPathErrorIncludesSearchedPaths(t *testing.T) {
|
||||
_, err := resolveSessionConfigPathWithCandidates("", []string{"./session.yml", "/usr/local/etc/narratio/session.yml", "/etc/narratio/session.yml"})
|
||||
_, err := resolveSessionConfigPathWithCandidates("", []string{"/usr/local/etc/narratio/session.yml", "/etc/narratio/session.yml"})
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
|
||||
@@ -19,6 +19,22 @@ func resolveSessionConfigPathWithCandidates(flagValue string, candidates []strin
|
||||
return explicit, nil
|
||||
}
|
||||
|
||||
resolved, err := discoverSessionConfigPathWithCandidates(candidates)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if resolved.Path != "" {
|
||||
return resolved.Path, nil
|
||||
}
|
||||
return "", missingSessionConfigError(resolved.Searched, "")
|
||||
}
|
||||
|
||||
type sessionConfigDiscovery struct {
|
||||
Path string
|
||||
Searched []string
|
||||
}
|
||||
|
||||
func discoverSessionConfigPathWithCandidates(candidates []string) (sessionConfigDiscovery, error) {
|
||||
ordered := make([]string, 0, len(candidates))
|
||||
for _, raw := range candidates {
|
||||
path := strings.TrimSpace(raw)
|
||||
@@ -31,19 +47,32 @@ func resolveSessionConfigPathWithCandidates(flagValue string, candidates []strin
|
||||
if info.IsDir() {
|
||||
continue
|
||||
}
|
||||
return filepath.Clean(path), nil
|
||||
return sessionConfigDiscovery{Path: filepath.Clean(path), Searched: ordered}, nil
|
||||
}
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
continue
|
||||
}
|
||||
return "", fmt.Errorf("check default session config %q: %w", path, err)
|
||||
return sessionConfigDiscovery{}, fmt.Errorf("check default session config %q: %w", path, err)
|
||||
}
|
||||
|
||||
return sessionConfigDiscovery{Searched: ordered}, nil
|
||||
}
|
||||
|
||||
func missingSessionConfigError(searched []string, remoteDetail string) error {
|
||||
ordered := append([]string(nil), searched...)
|
||||
if len(ordered) == 0 {
|
||||
return "", fmt.Errorf("no session config path provided and no default locations configured")
|
||||
if strings.TrimSpace(remoteDetail) != "" {
|
||||
return fmt.Errorf("no session config path provided and no default locations configured; %s", remoteDetail)
|
||||
}
|
||||
return fmt.Errorf("no session config path provided and no default locations configured")
|
||||
}
|
||||
return "", fmt.Errorf(
|
||||
"no session config path provided and no default session config found; searched: %s; pass --session to use an explicit path",
|
||||
msg := fmt.Sprintf(
|
||||
"no session config path provided and no default session config found; searched: %s",
|
||||
strings.Join(ordered, ", "),
|
||||
)
|
||||
if strings.TrimSpace(remoteDetail) != "" {
|
||||
msg += "; " + strings.TrimSpace(remoteDetail)
|
||||
}
|
||||
msg += "; pass --session to use an explicit path"
|
||||
return fmt.Errorf("%s", msg)
|
||||
}
|
||||
|
||||
@@ -1,67 +0,0 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
// Status reads and prints stage statuses from an existing manifest.
|
||||
func Status(ctx context.Context, args []string, out io.Writer) error {
|
||||
fs := flag.NewFlagSet("status", flag.ContinueOnError)
|
||||
fs.SetOutput(io.Discard)
|
||||
|
||||
var manifestPath string
|
||||
fs.StringVar(&manifestPath, "manifest", "", "path to manifest.json")
|
||||
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("status: invalid flags: %w", err)
|
||||
}
|
||||
if fs.NArg() != 0 {
|
||||
return fmt.Errorf("status: unexpected positional arguments")
|
||||
}
|
||||
if manifestPath == "" {
|
||||
return fmt.Errorf("status: --manifest is required")
|
||||
}
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m, err := store.Load(ctx, manifestPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("status: %w", err)
|
||||
}
|
||||
|
||||
if _, err := fmt.Fprintf(out, "session_id: %s\n", m.SessionID); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := fmt.Fprintf(out, "updated_at: %s\n", m.UpdatedAt.UTC().Format("2006-01-02T15:04:05Z07:00")); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if len(m.Stages) == 0 {
|
||||
_, err := fmt.Fprintln(out, "stages: no stages recorded")
|
||||
return err
|
||||
}
|
||||
|
||||
if _, err := fmt.Fprintln(out, "stages:"); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
names := make([]string, 0, len(m.Stages))
|
||||
for name := range m.Stages {
|
||||
names = append(names, name)
|
||||
}
|
||||
sort.Strings(names)
|
||||
|
||||
for _, name := range names {
|
||||
status := m.Stages[name].Status
|
||||
if _, err := fmt.Fprintf(out, "- %s: %s\n", name, status); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -63,7 +63,7 @@ func TestExecuteStagesDefaultWiringUsesWhisperXHTTPClient(t *testing.T) {
|
||||
t.Fatal("audio file payload was empty")
|
||||
}
|
||||
|
||||
outPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "transcripts", "raw", "alice.json")
|
||||
outPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.Campaign, cfg.Session.SessionID, "transcripts", "raw", "alice.json")
|
||||
data, err := os.ReadFile(outPath)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile(%q) error = %v", outPath, err)
|
||||
|
||||
77
internal/artifacts/archive_identity.go
Normal file
77
internal/artifacts/archive_identity.go
Normal file
@@ -0,0 +1,77 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
// ResolveArchiveBucket resolves archive bucket identity with manifest-first precedence.
|
||||
func ResolveArchiveBucket(cfg *config.Config, m *manifest.Manifest) string {
|
||||
if m != nil && strings.TrimSpace(m.S3Bucket) != "" {
|
||||
return strings.TrimSpace(m.S3Bucket)
|
||||
}
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Storage.S3 == nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(cfg.Pipeline.Storage.S3.Bucket)
|
||||
}
|
||||
|
||||
// ResolveArchiveSessionPrefix resolves archive session prefix with manifest-first precedence.
|
||||
func ResolveArchiveSessionPrefix(cfg *config.Config, m *manifest.Manifest) (string, error) {
|
||||
if m != nil && strings.TrimSpace(m.S3SessionPrefix) != "" {
|
||||
return strings.TrimSpace(m.S3SessionPrefix), nil
|
||||
}
|
||||
if cfg == nil || cfg.Session == nil || cfg.Pipeline == nil {
|
||||
return "", fmt.Errorf("resolved config is required")
|
||||
}
|
||||
|
||||
sessionID := strings.TrimSpace(cfg.Session.SessionID)
|
||||
if sessionID == "" && m != nil {
|
||||
sessionID = strings.TrimSpace(m.SessionID)
|
||||
}
|
||||
campaign := strings.TrimSpace(cfg.Session.Campaign)
|
||||
if campaign == "" && m != nil {
|
||||
campaign = strings.TrimSpace(m.Campaign)
|
||||
}
|
||||
if cfg.Pipeline.Storage.S3 == nil {
|
||||
return "", fmt.Errorf("pipeline.storage.s3 configuration is required")
|
||||
}
|
||||
|
||||
sessionPrefix := S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, campaign, sessionID)
|
||||
if strings.TrimSpace(sessionPrefix) == "" {
|
||||
return "", fmt.Errorf("session prefix is required")
|
||||
}
|
||||
return sessionPrefix, nil
|
||||
}
|
||||
|
||||
// ResolveArchiveRunPrefix resolves archive run prefix with manifest-first precedence.
|
||||
func ResolveArchiveRunPrefix(cfg *config.Config, m *manifest.Manifest) (string, error) {
|
||||
if m != nil {
|
||||
runPrefix := strings.TrimSpace(m.S3RunPrefix)
|
||||
if runPrefix != "" {
|
||||
return runPrefix, nil
|
||||
}
|
||||
}
|
||||
|
||||
sessionPrefix, err := ResolveArchiveSessionPrefix(cfg, m)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
runID := ""
|
||||
if m != nil {
|
||||
runID = strings.TrimSpace(m.RunID)
|
||||
}
|
||||
if runID == "" {
|
||||
return "", fmt.Errorf("run id is required")
|
||||
}
|
||||
return S3RunPrefix(sessionPrefix, runID), nil
|
||||
}
|
||||
|
||||
// ResolveArchiveCurrentStateKeys returns current pointer keys for a session prefix.
|
||||
func ResolveArchiveCurrentStateKeys(sessionPrefix string) (manifestKey, runIDKey string) {
|
||||
return S3CurrentManifestKey(sessionPrefix), S3CurrentRunPointerKey(sessionPrefix)
|
||||
}
|
||||
130
internal/artifacts/archive_identity_test.go
Normal file
130
internal/artifacts/archive_identity_test.go
Normal file
@@ -0,0 +1,130 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
func TestResolveArchiveBucketPrefersManifestThenConfig(t *testing.T) {
|
||||
cfg := &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Storage: config.StorageConfig{
|
||||
S3: &config.StorageS3Config{Bucket: "cfg-bucket"},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
if got := ResolveArchiveBucket(cfg, &manifest.Manifest{S3Bucket: "manifest-bucket"}); got != "manifest-bucket" {
|
||||
t.Fatalf("bucket = %q, want manifest-bucket", got)
|
||||
}
|
||||
if got := ResolveArchiveBucket(cfg, &manifest.Manifest{}); got != "cfg-bucket" {
|
||||
t.Fatalf("bucket = %q, want cfg-bucket", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveArchiveSessionPrefixPrefersManifestThenConfig(t *testing.T) {
|
||||
cfg := &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Storage: config.StorageConfig{
|
||||
S3: &config.StorageS3Config{RootPrefix: "dnd"},
|
||||
},
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-04-19",
|
||||
Campaign: "forsaken",
|
||||
},
|
||||
}
|
||||
m := &manifest.Manifest{S3SessionPrefix: "manifest/session/prefix/"}
|
||||
|
||||
got, err := ResolveArchiveSessionPrefix(cfg, m)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveArchiveSessionPrefix() error = %v", err)
|
||||
}
|
||||
if got != "manifest/session/prefix/" {
|
||||
t.Fatalf("session prefix = %q, want manifest/session/prefix/", got)
|
||||
}
|
||||
|
||||
got, err = ResolveArchiveSessionPrefix(cfg, &manifest.Manifest{})
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveArchiveSessionPrefix() error = %v", err)
|
||||
}
|
||||
want := "dnd/campaigns/forsaken/sessions/2026-04-19/"
|
||||
if got != want {
|
||||
t.Fatalf("session prefix = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveArchiveRunPrefixPrefersManifestThenDerived(t *testing.T) {
|
||||
cfg := &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Storage: config.StorageConfig{
|
||||
S3: &config.StorageS3Config{RootPrefix: "dnd"},
|
||||
},
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-04-19",
|
||||
Campaign: "forsaken",
|
||||
},
|
||||
}
|
||||
|
||||
m := &manifest.Manifest{
|
||||
RunID: "20260516T010203Z-1a2b3c4d",
|
||||
S3RunPrefix: "manifest/run/prefix/",
|
||||
}
|
||||
got, err := ResolveArchiveRunPrefix(cfg, m)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveArchiveRunPrefix() error = %v", err)
|
||||
}
|
||||
if got != "manifest/run/prefix/" {
|
||||
t.Fatalf("run prefix = %q, want manifest/run/prefix/", got)
|
||||
}
|
||||
|
||||
m = &manifest.Manifest{
|
||||
RunID: "20260516T010203Z-1a2b3c4d",
|
||||
}
|
||||
got, err = ResolveArchiveRunPrefix(cfg, m)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveArchiveRunPrefix() error = %v", err)
|
||||
}
|
||||
want := "dnd/campaigns/forsaken/sessions/2026-04-19/runs/20260516T010203Z-1a2b3c4d/"
|
||||
if got != want {
|
||||
t.Fatalf("run prefix = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveArchiveIdentityErrorsAreDeterministic(t *testing.T) {
|
||||
cfgNoS3 := &config.Config{
|
||||
Pipeline: &config.PipelineConfig{},
|
||||
Session: &config.SessionConfig{SessionID: "2026-04-19", Campaign: "forsaken"},
|
||||
}
|
||||
_, err := ResolveArchiveSessionPrefix(cfgNoS3, &manifest.Manifest{})
|
||||
if err == nil || !strings.Contains(err.Error(), "pipeline.storage.s3 configuration is required") {
|
||||
t.Fatalf("error = %v, want missing storage.s3", err)
|
||||
}
|
||||
|
||||
cfg := &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Storage: config.StorageConfig{
|
||||
S3: &config.StorageS3Config{RootPrefix: "dnd"},
|
||||
},
|
||||
},
|
||||
Session: &config.SessionConfig{SessionID: "2026-04-19", Campaign: "forsaken"},
|
||||
}
|
||||
_, err = ResolveArchiveRunPrefix(cfg, &manifest.Manifest{})
|
||||
if err == nil || !strings.Contains(err.Error(), "run id is required") {
|
||||
t.Fatalf("error = %v, want missing run id", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveArchiveCurrentStateKeys(t *testing.T) {
|
||||
manifestKey, runIDKey := ResolveArchiveCurrentStateKeys("dnd/campaigns/forsaken/sessions/2026-04-19/")
|
||||
if manifestKey != "dnd/campaigns/forsaken/sessions/2026-04-19/current/manifest.json" {
|
||||
t.Fatalf("manifest key = %q", manifestKey)
|
||||
}
|
||||
if runIDKey != "dnd/campaigns/forsaken/sessions/2026-04-19/current/run_id.txt" {
|
||||
t.Fatalf("run id key = %q", runIDKey)
|
||||
}
|
||||
}
|
||||
445
internal/artifacts/artifact_resolver.go
Normal file
445
internal/artifacts/artifact_resolver.go
Normal file
@@ -0,0 +1,445 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
const (
|
||||
ArtifactTranscriptMerged = "narratio.transcript.merged"
|
||||
ArtifactTranscriptPolished = "narratio.transcript.polished"
|
||||
ArtifactTranscriptFull = "narratio.transcript.full"
|
||||
ArtifactTranscriptTrimmed = "narratio.transcript.trimmed"
|
||||
ArtifactBoundsSession = "narratio.bounds.session"
|
||||
|
||||
ArtifactProvenancePreviousCacheManifestInput = "manifest.inputs.previous_cache"
|
||||
ArtifactProvenancePreviousCacheFilesystem = "current_session.previous_cache"
|
||||
)
|
||||
|
||||
// ErrSessionArtifactNotFound is returned when no readable artifact exists for a known ID.
|
||||
var ErrSessionArtifactNotFound = errors.New("session artifact not found")
|
||||
var configuredArtifactSourceRE = regexp.MustCompile(`^narratio\.artifact\.[a-z][a-z0-9_]*$`)
|
||||
var previousSessionArtifactSourceRE = regexp.MustCompile(`^narratio\.previous_session\.artifact\.([a-z][a-z0-9_]*)$`)
|
||||
|
||||
type artifactContentKind string
|
||||
|
||||
const (
|
||||
contentTranscriptJSON artifactContentKind = "transcript_json"
|
||||
contentJSON artifactContentKind = "json"
|
||||
contentText artifactContentKind = "text"
|
||||
)
|
||||
|
||||
type artifactSpec struct {
|
||||
ID string
|
||||
CanonicalRelPath string
|
||||
ProducerStage string
|
||||
OutputKind string
|
||||
ContentKind artifactContentKind
|
||||
}
|
||||
|
||||
var artifactRegistry = map[string]artifactSpec{
|
||||
ArtifactTranscriptMerged: {
|
||||
ID: ArtifactTranscriptMerged,
|
||||
CanonicalRelPath: "transcripts/merged.json",
|
||||
ProducerStage: "merge",
|
||||
OutputKind: "transcript_merged",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactTranscriptPolished: {
|
||||
ID: ArtifactTranscriptPolished,
|
||||
CanonicalRelPath: "transcripts/processed.json",
|
||||
ProducerStage: "polish",
|
||||
OutputKind: "transcript_processed",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactTranscriptFull: {
|
||||
ID: ArtifactTranscriptFull,
|
||||
CanonicalRelPath: "transcripts/normalized.json",
|
||||
ProducerStage: "normalize",
|
||||
OutputKind: "transcript_normalized",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactTranscriptTrimmed: {
|
||||
ID: ArtifactTranscriptTrimmed,
|
||||
CanonicalRelPath: "transcripts/trimmed.json",
|
||||
ProducerStage: "trim",
|
||||
OutputKind: "transcript_trimmed",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactBoundsSession: {
|
||||
ID: ArtifactBoundsSession,
|
||||
CanonicalRelPath: "artifacts/session_bounds.json",
|
||||
ProducerStage: "trim",
|
||||
OutputKind: "session_bounds",
|
||||
ContentKind: contentJSON,
|
||||
},
|
||||
}
|
||||
|
||||
// ResolvedSessionArtifact describes one session-level artifact lookup result.
|
||||
type ResolvedSessionArtifact struct {
|
||||
ID string
|
||||
Path string
|
||||
ProducerStage string
|
||||
OutputKind string
|
||||
ProducerRunID string
|
||||
Provenance string
|
||||
}
|
||||
|
||||
// SessionArtifactNotFoundError includes context when a known artifact cannot be read.
|
||||
type SessionArtifactNotFoundError struct {
|
||||
ArtifactID string
|
||||
}
|
||||
|
||||
func (e *SessionArtifactNotFoundError) Error() string {
|
||||
return fmt.Sprintf("%s: %q", ErrSessionArtifactNotFound, e.ArtifactID)
|
||||
}
|
||||
|
||||
func (e *SessionArtifactNotFoundError) Unwrap() error {
|
||||
return ErrSessionArtifactNotFound
|
||||
}
|
||||
|
||||
// NormalizeSessionArtifactSource validates canonical artifact IDs.
|
||||
func NormalizeSessionArtifactSource(source string) (string, error) {
|
||||
normalized := strings.TrimSpace(source)
|
||||
if normalized == "" {
|
||||
return "", fmt.Errorf("artifact source is required")
|
||||
}
|
||||
if _, ok := artifactRegistry[normalized]; !ok {
|
||||
return "", fmt.Errorf("unsupported artifact source %q", source)
|
||||
}
|
||||
return normalized, nil
|
||||
}
|
||||
|
||||
// IsConfiguredArtifactSource returns true when source is narratio.artifact.<name>.
|
||||
func IsConfiguredArtifactSource(source string) bool {
|
||||
return configuredArtifactSourceRE.MatchString(strings.TrimSpace(source))
|
||||
}
|
||||
|
||||
// IsPreviousSessionArtifactSource returns true when source is narratio.previous_session.artifact.<name>.
|
||||
func IsPreviousSessionArtifactSource(source string) bool {
|
||||
_, ok := PreviousSessionArtifactName(source)
|
||||
return ok
|
||||
}
|
||||
|
||||
// PreviousSessionArtifactName extracts <name> from narratio.previous_session.artifact.<name>.
|
||||
func PreviousSessionArtifactName(source string) (string, bool) {
|
||||
matches := previousSessionArtifactSourceRE.FindStringSubmatch(strings.TrimSpace(source))
|
||||
if len(matches) != 2 {
|
||||
return "", false
|
||||
}
|
||||
return matches[1], true
|
||||
}
|
||||
|
||||
// ResolveSessionArtifact resolves a symbolic source to a readable local session artifact path.
|
||||
// Resolution order is manifest producer outputs first, then canonical session path fallback.
|
||||
func ResolveSessionArtifact(paths SessionPaths, m *manifest.Manifest, source string) (ResolvedSessionArtifact, error) {
|
||||
id, err := NormalizeSessionArtifactSource(source)
|
||||
if err != nil {
|
||||
return ResolvedSessionArtifact{}, err
|
||||
}
|
||||
spec := artifactRegistry[id]
|
||||
|
||||
for _, candidate := range manifestArtifactCandidates(paths, m, spec) {
|
||||
exists, isDir, statErr := pathExists(candidate.Path)
|
||||
if statErr != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("stat %q: %w", candidate.Path, statErr)
|
||||
}
|
||||
if !exists || isDir {
|
||||
continue
|
||||
}
|
||||
resolved := candidate
|
||||
resolved.ID = spec.ID
|
||||
resolved.ProducerStage = spec.ProducerStage
|
||||
resolved.OutputKind = spec.OutputKind
|
||||
if err := validateResolvedContent(resolved.Path, spec.ContentKind); err != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", resolved.ID, err)
|
||||
}
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
fallbackPath := filepath.Join(paths.Root, filepath.FromSlash(spec.CanonicalRelPath))
|
||||
exists, isDir, statErr := pathExists(fallbackPath)
|
||||
if statErr != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("stat %q: %w", fallbackPath, statErr)
|
||||
}
|
||||
if exists && !isDir {
|
||||
if err := validateResolvedContent(fallbackPath, spec.ContentKind); err != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", spec.ID, err)
|
||||
}
|
||||
return ResolvedSessionArtifact{
|
||||
ID: spec.ID,
|
||||
Path: filepath.Clean(fallbackPath),
|
||||
ProducerStage: spec.ProducerStage,
|
||||
OutputKind: spec.OutputKind,
|
||||
Provenance: "fallback.canonical_path",
|
||||
}, nil
|
||||
}
|
||||
|
||||
return ResolvedSessionArtifact{}, &SessionArtifactNotFoundError{ArtifactID: spec.ID}
|
||||
}
|
||||
|
||||
// ResolveSessionArtifactWithCatalog resolves built-in sources using existing rules and resolves
|
||||
// configured narratio.artifact.<name> sources through runtime catalog availability.
|
||||
func ResolveSessionArtifactWithCatalog(paths SessionPaths, m *manifest.Manifest, source string, catalog *ArtifactCatalog) (ResolvedSessionArtifact, error) {
|
||||
normalized := strings.TrimSpace(source)
|
||||
if IsPreviousSessionArtifactSource(normalized) {
|
||||
return ResolvePreviousSessionArtifactWithCatalog(paths, m, normalized, catalog)
|
||||
}
|
||||
if !IsConfiguredArtifactSource(normalized) {
|
||||
return ResolveSessionArtifact(paths, m, normalized)
|
||||
}
|
||||
if catalog == nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("configured artifact source %q requires runtime artifact catalog", source)
|
||||
}
|
||||
entry, ok := catalog.Lookup(normalized)
|
||||
if !ok {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("unsupported artifact source %q", source)
|
||||
}
|
||||
if !entry.Available {
|
||||
return ResolvedSessionArtifact{}, &SessionArtifactNotFoundError{ArtifactID: normalized}
|
||||
}
|
||||
if err := validateResolvedContent(entry.Path, contentText); err != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", normalized, err)
|
||||
}
|
||||
return ResolvedSessionArtifact{
|
||||
ID: normalized,
|
||||
Path: filepath.Clean(entry.Path),
|
||||
ProducerStage: entry.ProducerStage,
|
||||
OutputKind: entry.OutputKind,
|
||||
Provenance: entry.Provenance,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// ResolvePreviousSessionArtifactWithCatalog resolves one canonical previous-session source id
|
||||
// to the prepared current-session previous-cache path.
|
||||
func ResolvePreviousSessionArtifactWithCatalog(
|
||||
paths SessionPaths,
|
||||
m *manifest.Manifest,
|
||||
source string,
|
||||
catalog *ArtifactCatalog,
|
||||
) (ResolvedSessionArtifact, error) {
|
||||
artifactName, ok := PreviousSessionArtifactName(source)
|
||||
if !ok {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("unsupported previous-session artifact source %q", source)
|
||||
}
|
||||
if catalog == nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("previous-session artifact source %q requires runtime artifact catalog", source)
|
||||
}
|
||||
|
||||
configuredSourceID := ConfiguredArtifactSourceID(artifactName)
|
||||
entry, ok := catalog.Lookup(configuredSourceID)
|
||||
if !ok {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("unsupported previous-session artifact source %q", source)
|
||||
}
|
||||
candidates := previousSessionCacheCandidatePaths(paths, entry.CanonicalRelPath)
|
||||
if len(candidates) == 0 {
|
||||
return ResolvedSessionArtifact{}, &SessionArtifactNotFoundError{ArtifactID: source}
|
||||
}
|
||||
|
||||
manifestInputPaths := manifestInputPathSet(paths, m)
|
||||
fallback := ""
|
||||
for _, candidate := range candidates {
|
||||
exists, isDir, statErr := pathExists(candidate)
|
||||
if statErr != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("stat %q: %w", candidate, statErr)
|
||||
}
|
||||
if !exists || isDir {
|
||||
continue
|
||||
}
|
||||
if err := validateResolvedContent(candidate, contentText); err != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", source, err)
|
||||
}
|
||||
if _, ok := manifestInputPaths[candidate]; ok {
|
||||
return ResolvedSessionArtifact{
|
||||
ID: source,
|
||||
Path: candidate,
|
||||
ProducerStage: "prepare",
|
||||
OutputKind: "previous_session_artifact",
|
||||
Provenance: ArtifactProvenancePreviousCacheManifestInput,
|
||||
}, nil
|
||||
}
|
||||
if fallback == "" {
|
||||
fallback = candidate
|
||||
}
|
||||
}
|
||||
|
||||
if fallback != "" {
|
||||
return ResolvedSessionArtifact{
|
||||
ID: source,
|
||||
Path: fallback,
|
||||
ProducerStage: "prepare",
|
||||
OutputKind: "previous_session_artifact",
|
||||
Provenance: ArtifactProvenancePreviousCacheFilesystem,
|
||||
}, nil
|
||||
}
|
||||
|
||||
return ResolvedSessionArtifact{}, &SessionArtifactNotFoundError{ArtifactID: source}
|
||||
}
|
||||
|
||||
func manifestArtifactCandidates(paths SessionPaths, m *manifest.Manifest, spec artifactSpec) []ResolvedSessionArtifact {
|
||||
if m == nil || len(m.Stages) == 0 || spec.ProducerStage == "" || spec.OutputKind == "" {
|
||||
return nil
|
||||
}
|
||||
sr := m.Stages[spec.ProducerStage]
|
||||
if sr == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
candidates := make([]ResolvedSessionArtifact, 0, len(sr.Outputs))
|
||||
for _, out := range sr.Outputs {
|
||||
if strings.TrimSpace(out.Kind) != spec.OutputKind {
|
||||
continue
|
||||
}
|
||||
if strings.TrimSpace(out.LocalPath) == "" {
|
||||
continue
|
||||
}
|
||||
resolved := filepath.Clean(ResolveSessionLocalPathForRead(paths, out.LocalPath))
|
||||
if resolved == "" {
|
||||
continue
|
||||
}
|
||||
candidates = append(candidates, ResolvedSessionArtifact{
|
||||
Path: resolved,
|
||||
ProducerRunID: strings.TrimSpace(out.ProducerRunID),
|
||||
Provenance: "manifest." + spec.ProducerStage + ".outputs",
|
||||
})
|
||||
}
|
||||
return dedupeResolvedArtifacts(candidates)
|
||||
}
|
||||
|
||||
func dedupeResolvedArtifacts(values []ResolvedSessionArtifact) []ResolvedSessionArtifact {
|
||||
seen := map[string]struct{}{}
|
||||
out := make([]ResolvedSessionArtifact, 0, len(values))
|
||||
for _, value := range values {
|
||||
key := filepath.Clean(strings.TrimSpace(value.Path))
|
||||
if key == "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[key]; ok {
|
||||
continue
|
||||
}
|
||||
seen[key] = struct{}{}
|
||||
value.Path = key
|
||||
out = append(out, value)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func previousSessionCacheCandidatePaths(paths SessionPaths, canonicalRelPath string) []string {
|
||||
trimmed := strings.TrimSpace(canonicalRelPath)
|
||||
if trimmed == "" {
|
||||
return nil
|
||||
}
|
||||
normalized := filepath.ToSlash(filepath.Clean(filepath.FromSlash(trimmed)))
|
||||
if normalized == "." || normalized == "" || normalized == ".." || strings.HasPrefix(normalized, "../") || strings.HasPrefix(normalized, "/") {
|
||||
return nil
|
||||
}
|
||||
|
||||
relCandidates := []string{normalized}
|
||||
const artifactsPrefix = "artifacts/"
|
||||
if strings.HasPrefix(normalized, artifactsPrefix) && len(normalized) > len(artifactsPrefix) {
|
||||
relCandidates = append(relCandidates, strings.TrimPrefix(normalized, artifactsPrefix))
|
||||
}
|
||||
|
||||
out := make([]string, 0, len(relCandidates))
|
||||
seen := map[string]struct{}{}
|
||||
for _, rel := range relCandidates {
|
||||
abs := filepath.Clean(SessionPreviousArtifactPath(paths, rel))
|
||||
if _, ok := seen[abs]; ok {
|
||||
continue
|
||||
}
|
||||
seen[abs] = struct{}{}
|
||||
out = append(out, abs)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func manifestInputPathSet(paths SessionPaths, m *manifest.Manifest) map[string]struct{} {
|
||||
if m == nil || len(m.Inputs) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]struct{}, len(m.Inputs))
|
||||
for _, in := range m.Inputs {
|
||||
resolved := filepath.Clean(ResolveSessionLocalPathForRead(paths, in.Path))
|
||||
if strings.TrimSpace(resolved) == "" {
|
||||
continue
|
||||
}
|
||||
out[resolved] = struct{}{}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func pathExists(path string) (exists bool, isDir bool, err error) {
|
||||
info, err := os.Stat(path)
|
||||
if err == nil {
|
||||
return true, info.IsDir(), nil
|
||||
}
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return false, false, nil
|
||||
}
|
||||
return false, false, err
|
||||
}
|
||||
|
||||
func validateResolvedContent(path string, kind artifactContentKind) error {
|
||||
switch kind {
|
||||
case contentTranscriptJSON:
|
||||
return validateTranscriptSegmentsJSON(path)
|
||||
case contentJSON:
|
||||
return validateJSONContent(path)
|
||||
case contentText:
|
||||
return validateNonEmptyContent(path)
|
||||
default:
|
||||
return fmt.Errorf("unsupported content kind %q", kind)
|
||||
}
|
||||
}
|
||||
|
||||
func validateTranscriptSegmentsJSON(path string) error {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read file: %w", err)
|
||||
}
|
||||
var payload map[string]any
|
||||
if err := json.Unmarshal(data, &payload); err != nil {
|
||||
return fmt.Errorf("decode json: %w", err)
|
||||
}
|
||||
segments, ok := payload["segments"]
|
||||
if !ok {
|
||||
return fmt.Errorf("top-level segments is required")
|
||||
}
|
||||
if _, ok := segments.([]any); !ok {
|
||||
return fmt.Errorf("top-level segments must be an array")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateJSONContent(path string) error {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read file: %w", err)
|
||||
}
|
||||
var payload any
|
||||
if err := json.Unmarshal(data, &payload); err != nil {
|
||||
return fmt.Errorf("decode json: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateNonEmptyContent(path string) error {
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("stat file: %w", err)
|
||||
}
|
||||
if info.IsDir() {
|
||||
return fmt.Errorf("path is a directory")
|
||||
}
|
||||
if info.Size() <= 0 {
|
||||
return fmt.Errorf("file is empty")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
449
internal/artifacts/artifact_resolver_test.go
Normal file
449
internal/artifacts/artifact_resolver_test.go
Normal file
@@ -0,0 +1,449 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
func TestNormalizeSessionArtifactSource(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
source string
|
||||
wantID string
|
||||
wantErr string
|
||||
}{
|
||||
{name: "legacy alias processed unsupported", source: "processed_transcript", wantErr: "unsupported artifact source"},
|
||||
{name: "legacy alias normalized unsupported", source: "normalized_transcript", wantErr: "unsupported artifact source"},
|
||||
{name: "legacy alias trimmed unsupported", source: "trimmed_transcript", wantErr: "unsupported artifact source"},
|
||||
{name: "configured source unsupported in built-in normalization", source: "narratio.artifact.session_recap", wantErr: "unsupported artifact source"},
|
||||
{name: "canonical", source: ArtifactTranscriptTrimmed, wantID: ArtifactTranscriptTrimmed},
|
||||
{name: "unsupported", source: "narratio.unknown", wantErr: "unsupported artifact source"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got, err := NormalizeSessionArtifactSource(tt.source)
|
||||
if tt.wantErr != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("NormalizeSessionArtifactSource() error = %v, want contains %q", err, tt.wantErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("NormalizeSessionArtifactSource() error = %v", err)
|
||||
}
|
||||
if got != tt.wantID {
|
||||
t.Fatalf("NormalizeSessionArtifactSource() = %q, want %q", got, tt.wantID)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestPreviousSessionArtifactSourceHelpers(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
source string
|
||||
wantName string
|
||||
wantMatch bool
|
||||
}{
|
||||
{
|
||||
name: "valid",
|
||||
source: "narratio.previous_session.artifact.session_recap",
|
||||
wantName: "session_recap",
|
||||
wantMatch: true,
|
||||
},
|
||||
{
|
||||
name: "valid with surrounding whitespace",
|
||||
source: " narratio.previous_session.artifact.quest_log ",
|
||||
wantName: "quest_log",
|
||||
wantMatch: true,
|
||||
},
|
||||
{
|
||||
name: "missing name",
|
||||
source: "narratio.previous_session.artifact.",
|
||||
wantMatch: false,
|
||||
},
|
||||
{
|
||||
name: "invalid key characters",
|
||||
source: "narratio.previous_session.artifact.session-recap",
|
||||
wantMatch: false,
|
||||
},
|
||||
{
|
||||
name: "wrong prefix",
|
||||
source: "narratio.previous.artifact.session_recap",
|
||||
wantMatch: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
name, ok := PreviousSessionArtifactName(tt.source)
|
||||
if ok != tt.wantMatch {
|
||||
t.Fatalf("PreviousSessionArtifactName(%q) ok = %t, want %t", tt.source, ok, tt.wantMatch)
|
||||
}
|
||||
if name != tt.wantName {
|
||||
t.Fatalf("PreviousSessionArtifactName(%q) name = %q, want %q", tt.source, name, tt.wantName)
|
||||
}
|
||||
if got := IsPreviousSessionArtifactSource(tt.source); got != tt.wantMatch {
|
||||
t.Fatalf("IsPreviousSessionArtifactSource(%q) = %t, want %t", tt.source, got, tt.wantMatch)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactPrefersManifestOutput(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
if err := os.MkdirAll(paths.ArtifactsDir, 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
|
||||
if err := os.WriteFile(manifestPath, []byte(`{"segments":[]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
canonicalPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(canonicalPath, []byte(`{"segments":[{"id":123}]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
m := manifest.New("session", time.Now().UTC())
|
||||
m.MarkStageSucceeded("normalize", time.Now().UTC(), []manifest.ArtifactRecord{
|
||||
{Kind: "transcript_normalized", LocalPath: manifestPath, ProducerRunID: "run-123"},
|
||||
})
|
||||
|
||||
resolved, err := ResolveSessionArtifact(paths, m, ArtifactTranscriptFull)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveSessionArtifact() error = %v", err)
|
||||
}
|
||||
if resolved.Path != manifestPath {
|
||||
t.Fatalf("resolved path = %q, want %q", resolved.Path, manifestPath)
|
||||
}
|
||||
if resolved.Provenance != "manifest.normalize.outputs" {
|
||||
t.Fatalf("provenance = %q, want manifest.normalize.outputs", resolved.Provenance)
|
||||
}
|
||||
if resolved.ProducerRunID != "run-123" {
|
||||
t.Fatalf("producer run id = %q, want run-123", resolved.ProducerRunID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactFallsBackToCanonicalPath(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
canonicalPath := filepath.Join(paths.TranscriptsDir, "trimmed.json")
|
||||
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(canonicalPath, []byte(`{"segments":[]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
resolved, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptTrimmed)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveSessionArtifact() error = %v", err)
|
||||
}
|
||||
if resolved.Path != canonicalPath {
|
||||
t.Fatalf("resolved path = %q, want %q", resolved.Path, canonicalPath)
|
||||
}
|
||||
if resolved.Provenance != "fallback.canonical_path" {
|
||||
t.Fatalf("provenance = %q, want fallback.canonical_path", resolved.Provenance)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactMissingReturnsTypedError(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
|
||||
_, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptTrimmed)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !errors.Is(err, ErrSessionArtifactNotFound) {
|
||||
t.Fatalf("errors.Is(err, ErrSessionArtifactNotFound) = false; err=%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactValidatesTranscriptShape(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
canonicalPath := filepath.Join(paths.TranscriptsDir, "processed.json")
|
||||
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(canonicalPath, []byte(`{"not_segments":[]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
_, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptPolished)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "top-level segments is required") {
|
||||
t.Fatalf("error = %q, want segments validation error", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactWithCatalogBuiltInBehaviorUnchanged(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
canonicalPath := filepath.Join(paths.TranscriptsDir, "trimmed.json")
|
||||
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(canonicalPath, []byte(`{"segments":[]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
resolved, err := ResolveSessionArtifactWithCatalog(paths, nil, ArtifactTranscriptTrimmed, NewArtifactCatalog())
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveSessionArtifactWithCatalog() error = %v", err)
|
||||
}
|
||||
if resolved.Path != canonicalPath {
|
||||
t.Fatalf("resolved.Path = %q, want %q", resolved.Path, canonicalPath)
|
||||
}
|
||||
if resolved.Provenance != "fallback.canonical_path" {
|
||||
t.Fatalf("provenance = %q, want fallback.canonical_path", resolved.Provenance)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactWithCatalogConfiguredAvailableGenerated(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
outputPath := filepath.Join(paths.ArtifactsDir, "session_recap.md")
|
||||
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(outputPath, []byte("recap\n"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
sourceID := ConfiguredArtifactSourceID("session_recap")
|
||||
if err := catalog.MarkAvailableGenerated(sourceID, outputPath); err != nil {
|
||||
t.Fatalf("MarkAvailableGenerated() error = %v", err)
|
||||
}
|
||||
|
||||
resolved, err := ResolveSessionArtifactWithCatalog(paths, nil, sourceID, catalog)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveSessionArtifactWithCatalog() error = %v", err)
|
||||
}
|
||||
if resolved.Path != outputPath {
|
||||
t.Fatalf("resolved.Path = %q, want %q", resolved.Path, outputPath)
|
||||
}
|
||||
if resolved.Provenance != ArtifactProvenanceGeneratedCurrentAnalyzeRun {
|
||||
t.Fatalf("provenance = %q, want %q", resolved.Provenance, ArtifactProvenanceGeneratedCurrentAnalyzeRun)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactWithCatalogConfiguredAvailableFromDisk(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
outputPath := filepath.Join(paths.ArtifactsDir, "player_handout.md")
|
||||
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(outputPath, []byte("handout\n"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"player_handout": {Enabled: false, OutputPath: "artifacts/player_handout.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
sourceID := ConfiguredArtifactSourceID("player_handout")
|
||||
if err := catalog.MarkAvailableFromDisk(sourceID, outputPath); err != nil {
|
||||
t.Fatalf("MarkAvailableFromDisk() error = %v", err)
|
||||
}
|
||||
|
||||
resolved, err := ResolveSessionArtifactWithCatalog(paths, nil, sourceID, catalog)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveSessionArtifactWithCatalog() error = %v", err)
|
||||
}
|
||||
if resolved.Provenance != ArtifactProvenanceDisabledFromDisk {
|
||||
t.Fatalf("provenance = %q, want %q", resolved.Provenance, ArtifactProvenanceDisabledFromDisk)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactWithCatalogConfiguredPlannedButUnavailable(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
_, err := ResolveSessionArtifactWithCatalog(paths, nil, ConfiguredArtifactSourceID("session_recap"), catalog)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !errors.Is(err, ErrSessionArtifactNotFound) {
|
||||
t.Fatalf("errors.Is(err, ErrSessionArtifactNotFound)=false; err=%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactWithCatalogUnsupportedConfiguredSourceFails(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
catalog := NewArtifactCatalog()
|
||||
_, err := ResolveSessionArtifactWithCatalog(paths, nil, "narratio.artifact.unknown", catalog)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "unsupported artifact source") {
|
||||
t.Fatalf("error = %q, want unsupported artifact source", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePreviousSessionArtifactWithCatalogPrefersManifestInputRecord(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
|
||||
manifestBackedPath := SessionPreviousArtifactPath(paths, "artifacts/session_recap.md")
|
||||
fallbackPath := SessionPreviousArtifactPath(paths, "session_recap.md")
|
||||
if err := os.MkdirAll(filepath.Dir(manifestBackedPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(fallbackPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(manifestBackedPath, []byte("recap from manifest input\n"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(fallbackPath, []byte("recap fallback\n"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
|
||||
m := manifest.New("session", time.Now().UTC())
|
||||
m.Inputs = []manifest.InputRecord{
|
||||
{Kind: "previous_artifact", Path: manifestBackedPath},
|
||||
}
|
||||
|
||||
resolved, err := ResolvePreviousSessionArtifactWithCatalog(
|
||||
paths,
|
||||
m,
|
||||
"narratio.previous_session.artifact.session_recap",
|
||||
catalog,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePreviousSessionArtifactWithCatalog() error = %v", err)
|
||||
}
|
||||
if resolved.Path != manifestBackedPath {
|
||||
t.Fatalf("resolved path = %q, want %q", resolved.Path, manifestBackedPath)
|
||||
}
|
||||
if resolved.Provenance != ArtifactProvenancePreviousCacheManifestInput {
|
||||
t.Fatalf("provenance = %q, want %q", resolved.Provenance, ArtifactProvenancePreviousCacheManifestInput)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePreviousSessionArtifactWithCatalogFallsBackToPreparedCachePath(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
|
||||
fallbackPath := SessionPreviousArtifactPath(paths, "session_recap.md")
|
||||
if err := os.MkdirAll(filepath.Dir(fallbackPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(fallbackPath, []byte("recap fallback\n"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
|
||||
resolved, err := ResolvePreviousSessionArtifactWithCatalog(
|
||||
paths,
|
||||
nil,
|
||||
"narratio.previous_session.artifact.session_recap",
|
||||
catalog,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePreviousSessionArtifactWithCatalog() error = %v", err)
|
||||
}
|
||||
if resolved.Path != fallbackPath {
|
||||
t.Fatalf("resolved path = %q, want %q", resolved.Path, fallbackPath)
|
||||
}
|
||||
if resolved.Provenance != ArtifactProvenancePreviousCacheFilesystem {
|
||||
t.Fatalf("provenance = %q, want %q", resolved.Provenance, ArtifactProvenancePreviousCacheFilesystem)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPreviousSessionCacheCandidatePathsStripsArtifactsPrefix(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
|
||||
got := previousSessionCacheCandidatePaths(paths, "artifacts/session_recap.md")
|
||||
want := []string{filepath.Join(paths.PreviousArtifactsDir, "session_recap.md")}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("previousSessionCacheCandidatePaths() = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePreviousSessionArtifactWithCatalogMissingReturnsTypedError(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
|
||||
_, err := ResolvePreviousSessionArtifactWithCatalog(
|
||||
paths,
|
||||
nil,
|
||||
"narratio.previous_session.artifact.session_recap",
|
||||
catalog,
|
||||
)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !errors.Is(err, ErrSessionArtifactNotFound) {
|
||||
t.Fatalf("errors.Is(err, ErrSessionArtifactNotFound)=false; err=%v", err)
|
||||
}
|
||||
}
|
||||
224
internal/artifacts/catalog.go
Normal file
224
internal/artifacts/catalog.go
Normal file
@@ -0,0 +1,224 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const (
|
||||
ArtifactProvenanceGeneratedCurrentAnalyzeRun = "generated.current_analyze_run"
|
||||
ArtifactProvenanceDisabledFromDisk = "filesystem.disabled_artifact_output"
|
||||
)
|
||||
|
||||
// ConfiguredArtifactDefinition describes one configured analyze artifact.
|
||||
type ConfiguredArtifactDefinition struct {
|
||||
Enabled bool
|
||||
OutputPath string
|
||||
}
|
||||
|
||||
// CatalogEntry is one runtime catalog entry resolved by source ID.
|
||||
type CatalogEntry struct {
|
||||
SourceID string
|
||||
ConfiguredKey string
|
||||
CanonicalRelPath string
|
||||
ProducerStage string
|
||||
OutputKind string
|
||||
Planned bool
|
||||
Executable bool
|
||||
Available bool
|
||||
Path string
|
||||
Provenance string
|
||||
}
|
||||
|
||||
// ArtifactCatalog tracks built-in and configured artifact definitions and runtime state.
|
||||
type ArtifactCatalog struct {
|
||||
entries map[string]CatalogEntry
|
||||
configuredIndex map[string]string
|
||||
}
|
||||
|
||||
// NewArtifactCatalog returns an empty runtime artifact catalog.
|
||||
func NewArtifactCatalog() *ArtifactCatalog {
|
||||
return &ArtifactCatalog{
|
||||
entries: map[string]CatalogEntry{},
|
||||
configuredIndex: map[string]string{},
|
||||
}
|
||||
}
|
||||
|
||||
// ConfiguredArtifactSourceID converts a configured artifact key into canonical source ID.
|
||||
func ConfiguredArtifactSourceID(key string) string {
|
||||
return "narratio.artifact." + strings.TrimSpace(key)
|
||||
}
|
||||
|
||||
// RegisterBuiltIns registers built-in source definitions used by runtime artifact resolution.
|
||||
func (c *ArtifactCatalog) RegisterBuiltIns() error {
|
||||
for _, id := range runtimeBuiltInArtifactIDs() {
|
||||
spec, ok := artifactRegistry[id]
|
||||
if !ok {
|
||||
return fmt.Errorf("register built-ins: source %q not found in artifact registry", id)
|
||||
}
|
||||
if err := c.addEntry(CatalogEntry{
|
||||
SourceID: spec.ID,
|
||||
CanonicalRelPath: spec.CanonicalRelPath,
|
||||
ProducerStage: spec.ProducerStage,
|
||||
OutputKind: spec.OutputKind,
|
||||
Planned: true,
|
||||
}); err != nil {
|
||||
return fmt.Errorf("register built-ins: %w", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// RegisterConfiguredArtifacts registers configured artifacts and applies executable selection.
|
||||
func (c *ArtifactCatalog) RegisterConfiguredArtifacts(
|
||||
configured map[string]ConfiguredArtifactDefinition,
|
||||
selected []string,
|
||||
) error {
|
||||
keys := make([]string, 0, len(configured))
|
||||
for key := range configured {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
|
||||
selectedSet := map[string]struct{}{}
|
||||
for _, key := range selected {
|
||||
trimmed := strings.TrimSpace(key)
|
||||
if trimmed == "" {
|
||||
return fmt.Errorf("selected artifact keys must be non-empty")
|
||||
}
|
||||
selectedSet[trimmed] = struct{}{}
|
||||
}
|
||||
|
||||
for _, key := range keys {
|
||||
trimmed := strings.TrimSpace(key)
|
||||
if trimmed == "" {
|
||||
return fmt.Errorf("configured artifact keys must be non-empty")
|
||||
}
|
||||
def := configured[key]
|
||||
sourceID := ConfiguredArtifactSourceID(trimmed)
|
||||
if _, exists := c.configuredIndex[trimmed]; exists {
|
||||
return fmt.Errorf("duplicate configured artifact key %q", trimmed)
|
||||
}
|
||||
|
||||
executable := def.Enabled
|
||||
if len(selectedSet) > 0 {
|
||||
_, executable = selectedSet[trimmed]
|
||||
}
|
||||
|
||||
if err := c.addEntry(CatalogEntry{
|
||||
SourceID: sourceID,
|
||||
ConfiguredKey: trimmed,
|
||||
CanonicalRelPath: strings.TrimSpace(def.OutputPath),
|
||||
ProducerStage: "analyze",
|
||||
OutputKind: "scriptorium_artifact",
|
||||
Planned: true,
|
||||
Executable: executable,
|
||||
}); err != nil {
|
||||
return fmt.Errorf("register configured artifact %q: %w", trimmed, err)
|
||||
}
|
||||
c.configuredIndex[trimmed] = sourceID
|
||||
}
|
||||
|
||||
if len(selectedSet) > 0 {
|
||||
for key := range selectedSet {
|
||||
if _, ok := c.configuredIndex[key]; !ok {
|
||||
return fmt.Errorf("selected artifact %q is not configured", key)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Lookup returns one catalog entry by source ID.
|
||||
func (c *ArtifactCatalog) Lookup(sourceID string) (CatalogEntry, bool) {
|
||||
if c == nil {
|
||||
return CatalogEntry{}, false
|
||||
}
|
||||
entry, ok := c.entries[strings.TrimSpace(sourceID)]
|
||||
return entry, ok
|
||||
}
|
||||
|
||||
// SourceIDForConfiguredKey returns canonical source ID for one configured key.
|
||||
func (c *ArtifactCatalog) SourceIDForConfiguredKey(key string) (string, bool) {
|
||||
if c == nil {
|
||||
return "", false
|
||||
}
|
||||
sourceID, ok := c.configuredIndex[strings.TrimSpace(key)]
|
||||
return sourceID, ok
|
||||
}
|
||||
|
||||
// ListConfigured returns configured entries sorted by configured key.
|
||||
func (c *ArtifactCatalog) ListConfigured() []CatalogEntry {
|
||||
if c == nil || len(c.configuredIndex) == 0 {
|
||||
return nil
|
||||
}
|
||||
keys := make([]string, 0, len(c.configuredIndex))
|
||||
for key := range c.configuredIndex {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
out := make([]CatalogEntry, 0, len(keys))
|
||||
for _, key := range keys {
|
||||
sourceID := c.configuredIndex[key]
|
||||
out = append(out, c.entries[sourceID])
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// MarkAvailableGenerated marks one source as available in current analyze execution.
|
||||
func (c *ArtifactCatalog) MarkAvailableGenerated(sourceID, path string) error {
|
||||
return c.markAvailable(sourceID, path, ArtifactProvenanceGeneratedCurrentAnalyzeRun)
|
||||
}
|
||||
|
||||
// MarkAvailableFromDisk marks one source as available from disabled artifact on disk.
|
||||
func (c *ArtifactCatalog) MarkAvailableFromDisk(sourceID, path string) error {
|
||||
return c.markAvailable(sourceID, path, ArtifactProvenanceDisabledFromDisk)
|
||||
}
|
||||
|
||||
func (c *ArtifactCatalog) markAvailable(sourceID, path, provenance string) error {
|
||||
if c == nil {
|
||||
return fmt.Errorf("artifact catalog is nil")
|
||||
}
|
||||
normalizedID := strings.TrimSpace(sourceID)
|
||||
entry, ok := c.entries[normalizedID]
|
||||
if !ok {
|
||||
return fmt.Errorf("unknown artifact source %q", sourceID)
|
||||
}
|
||||
trimmedPath := strings.TrimSpace(path)
|
||||
if trimmedPath == "" {
|
||||
return fmt.Errorf("artifact path is required")
|
||||
}
|
||||
entry.Available = true
|
||||
entry.Path = trimmedPath
|
||||
entry.Provenance = provenance
|
||||
c.entries[normalizedID] = entry
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *ArtifactCatalog) addEntry(entry CatalogEntry) error {
|
||||
if c == nil {
|
||||
return fmt.Errorf("artifact catalog is nil")
|
||||
}
|
||||
sourceID := strings.TrimSpace(entry.SourceID)
|
||||
if sourceID == "" {
|
||||
return fmt.Errorf("source id is required")
|
||||
}
|
||||
if _, exists := c.entries[sourceID]; exists {
|
||||
return fmt.Errorf("source id %q is already registered", sourceID)
|
||||
}
|
||||
entry.SourceID = sourceID
|
||||
c.entries[sourceID] = entry
|
||||
return nil
|
||||
}
|
||||
|
||||
func runtimeBuiltInArtifactIDs() []string {
|
||||
return []string{
|
||||
ArtifactTranscriptMerged,
|
||||
ArtifactTranscriptPolished,
|
||||
ArtifactTranscriptFull,
|
||||
ArtifactTranscriptTrimmed,
|
||||
ArtifactBoundsSession,
|
||||
}
|
||||
}
|
||||
188
internal/artifacts/catalog_test.go
Normal file
188
internal/artifacts/catalog_test.go
Normal file
@@ -0,0 +1,188 @@
|
||||
package artifacts
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestArtifactCatalogRegisterBuiltInsAndLookup(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterBuiltIns(); err != nil {
|
||||
t.Fatalf("RegisterBuiltIns() error = %v", err)
|
||||
}
|
||||
|
||||
entry, ok := catalog.Lookup(ArtifactTranscriptFull)
|
||||
if !ok {
|
||||
t.Fatalf("Lookup(%q) ok = false, want true", ArtifactTranscriptFull)
|
||||
}
|
||||
if !entry.Planned {
|
||||
t.Fatalf("entry.Planned = false, want true")
|
||||
}
|
||||
if entry.Executable {
|
||||
t.Fatalf("entry.Executable = true, want false")
|
||||
}
|
||||
if entry.CanonicalRelPath != "transcripts/normalized.json" {
|
||||
t.Fatalf("entry.CanonicalRelPath = %q, want transcripts/normalized.json", entry.CanonicalRelPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCatalogRegisterConfiguredArtifactsDefaultsToEnabled(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(map[string]ConfiguredArtifactDefinition{
|
||||
"player_handout": {Enabled: false, OutputPath: "artifacts/player_handout.md"},
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
}, nil); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
|
||||
recapID, ok := catalog.SourceIDForConfiguredKey("session_recap")
|
||||
if !ok {
|
||||
t.Fatal("SourceIDForConfiguredKey(session_recap) ok = false, want true")
|
||||
}
|
||||
recap, ok := catalog.Lookup(recapID)
|
||||
if !ok {
|
||||
t.Fatalf("Lookup(%q) ok = false, want true", recapID)
|
||||
}
|
||||
if !recap.Executable {
|
||||
t.Fatalf("recap.Executable = false, want true")
|
||||
}
|
||||
|
||||
handoutID, ok := catalog.SourceIDForConfiguredKey("player_handout")
|
||||
if !ok {
|
||||
t.Fatal("SourceIDForConfiguredKey(player_handout) ok = false, want true")
|
||||
}
|
||||
handout, ok := catalog.Lookup(handoutID)
|
||||
if !ok {
|
||||
t.Fatalf("Lookup(%q) ok = false, want true", handoutID)
|
||||
}
|
||||
if handout.Executable {
|
||||
t.Fatalf("handout.Executable = true, want false")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCatalogRegisterConfiguredArtifactsSelectedSetOverridesEnabled(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
"player_handout": {Enabled: false, OutputPath: "artifacts/player_handout.md"},
|
||||
},
|
||||
[]string{"player_handout"},
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
|
||||
entries := catalog.ListConfigured()
|
||||
if len(entries) != 2 {
|
||||
t.Fatalf("ListConfigured() len = %d, want 2", len(entries))
|
||||
}
|
||||
if entries[0].ConfiguredKey != "player_handout" || entries[0].Executable != true {
|
||||
t.Fatalf("entries[0] = %+v, want player_handout executable", entries[0])
|
||||
}
|
||||
if entries[1].ConfiguredKey != "session_recap" || entries[1].Executable != false {
|
||||
t.Fatalf("entries[1] = %+v, want session_recap disabled by selection", entries[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCatalogRejectsSelectedUnknownArtifact(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
[]string{"unknown"},
|
||||
)
|
||||
if err == nil {
|
||||
t.Fatal("RegisterConfiguredArtifacts() error = nil, want non-nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCatalogRejectsConfiguredSourceConflictAcrossRegistrations(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() first call error = %v", err)
|
||||
}
|
||||
err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
)
|
||||
if err == nil {
|
||||
t.Fatal("RegisterConfiguredArtifacts() error = nil, want non-nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCatalogMarkAvailableGenerated(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
|
||||
sourceID, _ := catalog.SourceIDForConfiguredKey("session_recap")
|
||||
if err := catalog.MarkAvailableGenerated(sourceID, "/tmp/session_recap.md"); err != nil {
|
||||
t.Fatalf("MarkAvailableGenerated() error = %v", err)
|
||||
}
|
||||
entry, _ := catalog.Lookup(sourceID)
|
||||
if !entry.Available {
|
||||
t.Fatalf("entry.Available = false, want true")
|
||||
}
|
||||
if entry.Provenance != ArtifactProvenanceGeneratedCurrentAnalyzeRun {
|
||||
t.Fatalf("entry.Provenance = %q, want %q", entry.Provenance, ArtifactProvenanceGeneratedCurrentAnalyzeRun)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCatalogMarkAvailableFromDisk(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: false, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
|
||||
sourceID, _ := catalog.SourceIDForConfiguredKey("session_recap")
|
||||
if err := catalog.MarkAvailableFromDisk(sourceID, "/tmp/session_recap.md"); err != nil {
|
||||
t.Fatalf("MarkAvailableFromDisk() error = %v", err)
|
||||
}
|
||||
entry, _ := catalog.Lookup(sourceID)
|
||||
if !entry.Available {
|
||||
t.Fatalf("entry.Available = false, want true")
|
||||
}
|
||||
if entry.Provenance != ArtifactProvenanceDisabledFromDisk {
|
||||
t.Fatalf("entry.Provenance = %q, want %q", entry.Provenance, ArtifactProvenanceDisabledFromDisk)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCatalogLookupPlannedButUnavailable(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
|
||||
sourceID, _ := catalog.SourceIDForConfiguredKey("session_recap")
|
||||
entry, ok := catalog.Lookup(sourceID)
|
||||
if !ok {
|
||||
t.Fatalf("Lookup(%q) ok = false, want true", sourceID)
|
||||
}
|
||||
if !entry.Planned {
|
||||
t.Fatalf("entry.Planned = false, want true")
|
||||
}
|
||||
if entry.Available {
|
||||
t.Fatalf("entry.Available = true, want false")
|
||||
}
|
||||
}
|
||||
@@ -30,13 +30,13 @@ func NewLocalStore(workspaceRoot string) *LocalStore {
|
||||
return &LocalStore{WorkspaceRoot: workspaceRoot}
|
||||
}
|
||||
|
||||
// SessionPaths resolves canonical paths for a session workdir.
|
||||
func (s *LocalStore) SessionPaths(sessionID string) SessionPaths {
|
||||
return buildSessionPaths(s.WorkspaceRoot, sessionID)
|
||||
// SessionPathsFor resolves canonical campaign-aware paths for a session workdir.
|
||||
func (s *LocalStore) SessionPathsFor(campaign, sessionID string) SessionPaths {
|
||||
return buildSessionPaths(s.WorkspaceRoot, campaign, sessionID)
|
||||
}
|
||||
|
||||
// EnsureLayout creates and verifies the canonical session workdir directory layout.
|
||||
func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
// EnsureLayoutFor creates and verifies campaign-aware session layout.
|
||||
func (s *LocalStore) EnsureLayoutFor(campaign, sessionID string) (SessionPaths, error) {
|
||||
if strings.TrimSpace(s.WorkspaceRoot) == "" {
|
||||
return SessionPaths{}, fmt.Errorf("workspace root is required")
|
||||
}
|
||||
@@ -44,7 +44,22 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
return SessionPaths{}, fmt.Errorf("sessionID is required")
|
||||
}
|
||||
|
||||
paths := s.SessionPaths(sessionID)
|
||||
campaign = strings.TrimSpace(campaign)
|
||||
if campaign == "" {
|
||||
return SessionPaths{}, fmt.Errorf("campaign is required")
|
||||
}
|
||||
|
||||
return s.ensureLayout(s.SessionPathsFor(campaign, sessionID))
|
||||
}
|
||||
|
||||
func (s *LocalStore) ensureLayout(paths SessionPaths) (SessionPaths, error) {
|
||||
if strings.TrimSpace(s.WorkspaceRoot) == "" {
|
||||
return SessionPaths{}, fmt.Errorf("workspace root is required")
|
||||
}
|
||||
if strings.TrimSpace(paths.SessionID) == "" {
|
||||
return SessionPaths{}, fmt.Errorf("sessionID is required")
|
||||
}
|
||||
|
||||
dirs := []string{
|
||||
paths.Root,
|
||||
paths.InputsDir,
|
||||
@@ -53,8 +68,13 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
paths.TranscriptsRawDir,
|
||||
paths.TranscriptsTrimmedDir,
|
||||
paths.ArtifactsDir,
|
||||
paths.ReportsDir,
|
||||
paths.ConfigDir,
|
||||
paths.LogsDir,
|
||||
paths.CurrentDir,
|
||||
paths.RunsDir,
|
||||
paths.PreviousDir,
|
||||
paths.PreviousArtifactsDir,
|
||||
}
|
||||
|
||||
for _, dir := range dirs {
|
||||
@@ -66,13 +86,16 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
return paths, nil
|
||||
}
|
||||
|
||||
// CopyInput copies an input file into the session workdir under destRelativePath.
|
||||
func (s *LocalStore) CopyInput(sessionID, srcPath, destRelativePath string) (Ref, error) {
|
||||
paths, err := s.EnsureLayout(sessionID)
|
||||
// CopyInputFor copies an input file into the campaign-aware session workdir under destRelativePath.
|
||||
func (s *LocalStore) CopyInputFor(campaign, sessionID, srcPath, destRelativePath string) (Ref, error) {
|
||||
paths, err := s.EnsureLayoutFor(campaign, sessionID)
|
||||
if err != nil {
|
||||
return Ref{}, err
|
||||
}
|
||||
return s.copyInputWithPaths(paths, sessionID, srcPath, destRelativePath)
|
||||
}
|
||||
|
||||
func (s *LocalStore) copyInputWithPaths(paths SessionPaths, sessionID, srcPath, destRelativePath string) (Ref, error) {
|
||||
destAbs, err := resolveInRoot(paths.Root, destRelativePath)
|
||||
if err != nil {
|
||||
return Ref{}, fmt.Errorf("copy input: %w", err)
|
||||
@@ -173,13 +196,16 @@ func (s *LocalStore) Checksum(path string) (string, error) {
|
||||
return digest, nil
|
||||
}
|
||||
|
||||
// AcquireSessionLock acquires an exclusive lock file for a session workdir.
|
||||
func (s *LocalStore) AcquireSessionLock(sessionID string) (*LockHandle, error) {
|
||||
paths, err := s.EnsureLayout(sessionID)
|
||||
// AcquireSessionLockFor acquires an exclusive lock file for a campaign/session workdir.
|
||||
func (s *LocalStore) AcquireSessionLockFor(campaign, sessionID string) (*LockHandle, error) {
|
||||
paths, err := s.EnsureLayoutFor(campaign, sessionID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return s.acquireSessionLockForPaths(paths)
|
||||
}
|
||||
|
||||
func (s *LocalStore) acquireSessionLockForPaths(paths SessionPaths) (*LockHandle, error) {
|
||||
f, err := os.OpenFile(paths.LockPath, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o644)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrExist) {
|
||||
|
||||
@@ -10,9 +10,9 @@ import (
|
||||
|
||||
func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
paths, err := store.EnsureLayout("session-1")
|
||||
paths, err := store.EnsureLayoutFor("sample-campaign", "session-1")
|
||||
if err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
t.Fatalf("EnsureLayoutFor() error = %v", err)
|
||||
}
|
||||
|
||||
checkDirExists(t, paths.Root)
|
||||
@@ -22,17 +22,36 @@ func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
|
||||
checkDirExists(t, paths.TranscriptsRawDir)
|
||||
checkDirExists(t, paths.TranscriptsTrimmedDir)
|
||||
checkDirExists(t, paths.ArtifactsDir)
|
||||
checkDirExists(t, paths.ReportsDir)
|
||||
checkDirExists(t, paths.ConfigDir)
|
||||
checkDirExists(t, paths.LogsDir)
|
||||
checkDirExists(t, paths.CurrentDir)
|
||||
checkDirExists(t, paths.RunsDir)
|
||||
checkDirExists(t, paths.PreviousDir)
|
||||
checkDirExists(t, paths.PreviousArtifactsDir)
|
||||
|
||||
if filepath.Base(paths.ManifestPath) != "manifest.json" {
|
||||
t.Fatalf("ManifestPath = %q, want basename manifest.json", paths.ManifestPath)
|
||||
}
|
||||
if filepath.Base(paths.PreviousManifestPath) != "manifest.json" {
|
||||
t.Fatalf("PreviousManifestPath = %q, want basename manifest.json", paths.PreviousManifestPath)
|
||||
}
|
||||
if filepath.Base(paths.LockPath) != ".lock" {
|
||||
t.Fatalf("LockPath = %q, want basename .lock", paths.LockPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnsureLayoutForRequiresCampaign(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
_, err := store.EnsureLayoutFor("", "session-1")
|
||||
if err == nil {
|
||||
t.Fatal("expected campaign-required error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "campaign is required") {
|
||||
t.Fatalf("error = %v, want campaign-required error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestChecksumCalculation(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
path := filepath.Join(t.TempDir(), "sample.txt")
|
||||
@@ -53,9 +72,9 @@ func TestChecksumCalculation(t *testing.T) {
|
||||
func TestLockAcquireRelease(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
|
||||
lock, err := store.AcquireSessionLock("session-1")
|
||||
lock, err := store.AcquireSessionLockFor("sample-campaign", "session-1")
|
||||
if err != nil {
|
||||
t.Fatalf("AcquireSessionLock() error = %v", err)
|
||||
t.Fatalf("AcquireSessionLockFor() error = %v", err)
|
||||
}
|
||||
|
||||
exists, err := store.Exists(lock.path)
|
||||
@@ -81,15 +100,15 @@ func TestLockAcquireRelease(t *testing.T) {
|
||||
|
||||
func TestLockConflict(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
lock1, err := store.AcquireSessionLock("session-1")
|
||||
lock1, err := store.AcquireSessionLockFor("sample-campaign", "session-1")
|
||||
if err != nil {
|
||||
t.Fatalf("first AcquireSessionLock() error = %v", err)
|
||||
t.Fatalf("first AcquireSessionLockFor() error = %v", err)
|
||||
}
|
||||
defer func() {
|
||||
_ = store.ReleaseSessionLock(lock1)
|
||||
}()
|
||||
|
||||
_, err = store.AcquireSessionLock("session-1")
|
||||
_, err = store.AcquireSessionLockFor("sample-campaign", "session-1")
|
||||
if err == nil {
|
||||
t.Fatal("expected lock conflict error, got nil")
|
||||
}
|
||||
@@ -138,9 +157,9 @@ func TestCopyInput(t *testing.T) {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
ref, err := store.CopyInput("session-1", srcPath, "inputs/speakers.yml")
|
||||
ref, err := store.CopyInputFor("sample-campaign", "session-1", srcPath, "inputs/speakers.yml")
|
||||
if err != nil {
|
||||
t.Fatalf("CopyInput() error = %v", err)
|
||||
t.Fatalf("CopyInputFor() error = %v", err)
|
||||
}
|
||||
|
||||
if ref.Kind != "input" {
|
||||
|
||||
@@ -1,7 +1,10 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
@@ -9,6 +12,8 @@ import (
|
||||
// SessionPaths contains canonical local paths for one session work directory.
|
||||
type SessionPaths struct {
|
||||
WorkspaceRoot string
|
||||
CampaignID string
|
||||
SessionID string
|
||||
Root string
|
||||
InputsDir string
|
||||
AudioDir string
|
||||
@@ -16,20 +21,69 @@ type SessionPaths struct {
|
||||
TranscriptsRawDir string
|
||||
TranscriptsTrimmedDir string
|
||||
ArtifactsDir string
|
||||
ReportsDir string
|
||||
ConfigDir string
|
||||
LogsDir string
|
||||
CurrentDir string
|
||||
RunsDir string
|
||||
PreviousDir string
|
||||
PreviousManifestPath string
|
||||
PreviousArtifactsDir string
|
||||
ManifestPath string
|
||||
LockPath string
|
||||
}
|
||||
|
||||
// SessionWorkDir returns the work directory for one session.
|
||||
func SessionWorkDir(rootDir, sessionID string) string {
|
||||
return filepath.Join(rootDir, config.PathWorkDirSegment, sessionID)
|
||||
// SessionWorkDirForCampaign returns the canonical campaign-aware work directory for one session.
|
||||
func SessionWorkDirForCampaign(rootDir, campaign, sessionID string) string {
|
||||
return filepath.Join(rootDir, config.PathWorkDirSegment, campaign, sessionID)
|
||||
}
|
||||
|
||||
// SessionRunWorkDir returns the campaign/session/run scoped local work directory.
|
||||
func SessionRunWorkDir(rootDir, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(rootDir, config.PathWorkDirSegment, campaign, sessionID, runID)
|
||||
// SessionManifestPathForCampaign returns the canonical session manifest path.
|
||||
func SessionManifestPathForCampaign(rootDir, campaign, sessionID string) string {
|
||||
return filepath.Join(SessionWorkDirForCampaign(rootDir, campaign, sessionID), config.PathManifestFile)
|
||||
}
|
||||
|
||||
// SessionRunsDirForCampaign returns the canonical runs directory for one session.
|
||||
func SessionRunsDirForCampaign(rootDir, campaign, sessionID string) string {
|
||||
return filepath.Join(SessionWorkDirForCampaign(rootDir, campaign, sessionID), config.PathRunsDirSegment)
|
||||
}
|
||||
|
||||
// SessionPreviousDirForCampaign returns the canonical previous-session state directory.
|
||||
func SessionPreviousDirForCampaign(rootDir, campaign, sessionID string) string {
|
||||
return filepath.Join(SessionWorkDirForCampaign(rootDir, campaign, sessionID), config.PathPreviousDirSegment)
|
||||
}
|
||||
|
||||
// SessionPreviousManifestPathForCampaign returns the canonical previous-session manifest cache path.
|
||||
func SessionPreviousManifestPathForCampaign(rootDir, campaign, sessionID string) string {
|
||||
return filepath.Join(SessionPreviousDirForCampaign(rootDir, campaign, sessionID), config.PathManifestFile)
|
||||
}
|
||||
|
||||
// SessionPreviousArtifactsDirForCampaign returns the canonical previous-session artifact cache directory.
|
||||
func SessionPreviousArtifactsDirForCampaign(rootDir, campaign, sessionID string) string {
|
||||
return filepath.Join(SessionPreviousDirForCampaign(rootDir, campaign, sessionID), config.PathArtifactsDirSegment)
|
||||
}
|
||||
|
||||
// SessionPreviousArtifactPathForCampaign returns a path under previous/artifacts for one artifact.
|
||||
func SessionPreviousArtifactPathForCampaign(rootDir, campaign, sessionID, artifactRelativePath string) string {
|
||||
return filepath.Join(
|
||||
SessionPreviousArtifactsDirForCampaign(rootDir, campaign, sessionID),
|
||||
filepath.FromSlash(previousArtifactCacheRelativePath(artifactRelativePath)),
|
||||
)
|
||||
}
|
||||
|
||||
// SessionRunRootForCampaign returns the canonical run root under runs/{run_id}.
|
||||
func SessionRunRootForCampaign(rootDir, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(SessionRunsDirForCampaign(rootDir, campaign, sessionID), runID)
|
||||
}
|
||||
|
||||
// SessionRunManifestPathForCampaign returns the canonical run manifest path under runs/{run_id}/manifest.json.
|
||||
func SessionRunManifestPathForCampaign(rootDir, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(SessionRunRootForCampaign(rootDir, campaign, sessionID, runID), config.PathManifestFile)
|
||||
}
|
||||
|
||||
// SessionRunStageDirForCampaign returns the canonical stage directory under runs/{run_id}/{stage}.
|
||||
func SessionRunStageDirForCampaign(rootDir, campaign, sessionID, runID, stageName string) string {
|
||||
return filepath.Join(SessionRunRootForCampaign(rootDir, campaign, sessionID, runID), stageName)
|
||||
}
|
||||
|
||||
// SessionSpoolAudioDir returns the campaign/session/run scoped local spool audio path.
|
||||
@@ -37,10 +91,102 @@ func SessionSpoolAudioDir(spoolRoot, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(spoolRoot, campaign, sessionID, runID, config.PathAudioDirSegment)
|
||||
}
|
||||
|
||||
func buildSessionPaths(workspaceRoot, sessionID string) SessionPaths {
|
||||
root := SessionWorkDir(workspaceRoot, sessionID)
|
||||
// SessionSpoolDir returns the campaign/session scoped local spool root.
|
||||
func SessionSpoolDir(spoolRoot, campaign, sessionID string) string {
|
||||
return filepath.Join(spoolRoot, campaign, sessionID)
|
||||
}
|
||||
|
||||
// SessionSpoolRestoreAudioDir returns the local spool audio path for restore downloads.
|
||||
func SessionSpoolRestoreAudioDir(spoolRoot, campaign, sessionID string) string {
|
||||
return filepath.Join(SessionSpoolDir(spoolRoot, campaign, sessionID), "restore", config.PathAudioDirSegment)
|
||||
}
|
||||
|
||||
// S3AudioCachePath returns the durable local cache path for one S3 audio object.
|
||||
func S3AudioCachePath(cacheRoot, bucket, key string) (string, error) {
|
||||
root := filepath.Clean(strings.TrimSpace(cacheRoot))
|
||||
if root == "." || root == "" {
|
||||
return "", fmt.Errorf("cache root is required")
|
||||
}
|
||||
bucket = strings.Trim(strings.TrimSpace(bucket), "/")
|
||||
if bucket == "" || bucket == "." || bucket == ".." || strings.Contains(bucket, "/") || strings.Contains(bucket, `\`) {
|
||||
return "", fmt.Errorf("bucket is required and must be a single path segment")
|
||||
}
|
||||
rawKey := strings.ReplaceAll(strings.TrimSpace(key), `\`, "/")
|
||||
if strings.HasPrefix(rawKey, "/") {
|
||||
return "", fmt.Errorf("s3 key must not be absolute")
|
||||
}
|
||||
cleanKey := cleanCacheS3Key(rawKey)
|
||||
if cleanKey == "" {
|
||||
return "", fmt.Errorf("s3 key is required")
|
||||
}
|
||||
if cleanKey == ".." || strings.HasPrefix(cleanKey, "../") || strings.HasPrefix(cleanKey, "/") {
|
||||
return "", fmt.Errorf("s3 key must not escape cache root")
|
||||
}
|
||||
return filepath.Join(root, "s3", bucket, filepath.FromSlash(cleanKey)), nil
|
||||
}
|
||||
|
||||
// S3AudioCacheNamespaceDir returns the durable local cache namespace for all
|
||||
// Narratio S3 audio objects under one bucket/root prefix.
|
||||
func S3AudioCacheNamespaceDir(cacheRoot, bucket, rootPrefix string) (string, error) {
|
||||
sentinelKey := path.Join(cleanS3PathPart(rootPrefix), config.S3CampaignsSegment, ".narratio-cache-sentinel")
|
||||
sentinelPath, err := S3AudioCachePath(cacheRoot, bucket, sentinelKey)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return filepath.Dir(sentinelPath), nil
|
||||
}
|
||||
|
||||
func cleanCacheS3Key(key string) string {
|
||||
normalized := strings.ReplaceAll(strings.TrimSpace(key), `\`, "/")
|
||||
normalized = strings.Trim(normalized, "/")
|
||||
if normalized == "" {
|
||||
return ""
|
||||
}
|
||||
return path.Clean(normalized)
|
||||
}
|
||||
|
||||
// SessionPreviousDir returns the previous-session state directory for already-resolved session paths.
|
||||
func SessionPreviousDir(paths SessionPaths) string {
|
||||
return paths.PreviousDir
|
||||
}
|
||||
|
||||
// SessionPreviousManifestPath returns the previous-session manifest path for already-resolved session paths.
|
||||
func SessionPreviousManifestPath(paths SessionPaths) string {
|
||||
return paths.PreviousManifestPath
|
||||
}
|
||||
|
||||
// SessionPreviousArtifactsDir returns the previous-session artifact directory for already-resolved session paths.
|
||||
func SessionPreviousArtifactsDir(paths SessionPaths) string {
|
||||
return paths.PreviousArtifactsDir
|
||||
}
|
||||
|
||||
// SessionPreviousArtifactPath returns a path under previous/artifacts for already-resolved session paths.
|
||||
func SessionPreviousArtifactPath(paths SessionPaths, artifactRelativePath string) string {
|
||||
return filepath.Join(paths.PreviousArtifactsDir, filepath.FromSlash(previousArtifactCacheRelativePath(artifactRelativePath)))
|
||||
}
|
||||
|
||||
func previousArtifactCacheRelativePath(artifactRelativePath string) string {
|
||||
rel := filepath.ToSlash(filepath.Clean(filepath.FromSlash(strings.TrimSpace(artifactRelativePath))))
|
||||
if rel == "." {
|
||||
return ""
|
||||
}
|
||||
const artifactsPrefix = "artifacts/"
|
||||
if strings.HasPrefix(rel, artifactsPrefix) && len(rel) > len(artifactsPrefix) {
|
||||
return strings.TrimPrefix(rel, artifactsPrefix)
|
||||
}
|
||||
return rel
|
||||
}
|
||||
|
||||
func buildSessionPaths(workspaceRoot, campaign, sessionID string) SessionPaths {
|
||||
root := SessionWorkDirForCampaign(workspaceRoot, campaign, sessionID)
|
||||
return buildSessionPathsFromRoot(workspaceRoot, campaign, sessionID, root)
|
||||
}
|
||||
|
||||
func buildSessionPathsFromRoot(workspaceRoot, campaign, sessionID, root string) SessionPaths {
|
||||
return SessionPaths{
|
||||
WorkspaceRoot: workspaceRoot,
|
||||
CampaignID: campaign,
|
||||
SessionID: sessionID,
|
||||
Root: root,
|
||||
InputsDir: filepath.Join(root, config.PathInputsDirSegment),
|
||||
AudioDir: filepath.Join(root, config.PathAudioDirSegment),
|
||||
@@ -48,8 +194,14 @@ func buildSessionPaths(workspaceRoot, sessionID string) SessionPaths {
|
||||
TranscriptsRawDir: filepath.Join(root, filepath.FromSlash(config.PathTranscriptsRaw)),
|
||||
TranscriptsTrimmedDir: filepath.Join(root, filepath.FromSlash(config.PathTranscriptsTrimmed)),
|
||||
ArtifactsDir: filepath.Join(root, config.PathArtifactsDirSegment),
|
||||
ReportsDir: filepath.Join(root, config.PathReportsDirSegment),
|
||||
ConfigDir: filepath.Join(root, config.PathConfigDirSegment),
|
||||
LogsDir: filepath.Join(root, config.PathLogsDirSegment),
|
||||
CurrentDir: filepath.Join(root, config.PathCurrentDirSegment),
|
||||
RunsDir: filepath.Join(root, config.PathRunsDirSegment),
|
||||
PreviousDir: filepath.Join(root, config.PathPreviousDirSegment),
|
||||
PreviousManifestPath: filepath.Join(root, config.PathPreviousDirSegment, config.PathManifestFile),
|
||||
PreviousArtifactsDir: filepath.Join(root, config.PathPreviousDirSegment, config.PathArtifactsDirSegment),
|
||||
ManifestPath: filepath.Join(root, config.PathManifestFile),
|
||||
LockPath: filepath.Join(root, config.PathLockFile),
|
||||
}
|
||||
|
||||
@@ -5,12 +5,104 @@ import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSessionRunWorkDir(t *testing.T) {
|
||||
func TestSessionWorkDirForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
got := SessionRunWorkDir(root, "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4")
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4")
|
||||
got := SessionWorkDirForCampaign(root, "forsaken", "2026-04-19")
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19")
|
||||
if got != want {
|
||||
t.Fatalf("SessionRunWorkDir() = %q, want %q", got, want)
|
||||
t.Fatalf("SessionWorkDirForCampaign() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionManifestPathForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
got := SessionManifestPathForCampaign(root, "forsaken", "2026-04-19")
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "manifest.json")
|
||||
if got != want {
|
||||
t.Fatalf("SessionManifestPathForCampaign() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionRunRootAndStageDirForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
runID := "20260515T031522Z-a1b2c3d4"
|
||||
runRoot := SessionRunRootForCampaign(root, "forsaken", "2026-04-19", runID)
|
||||
wantRoot := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID)
|
||||
if runRoot != wantRoot {
|
||||
t.Fatalf("SessionRunRootForCampaign() = %q, want %q", runRoot, wantRoot)
|
||||
}
|
||||
|
||||
stageDir := SessionRunStageDirForCampaign(root, "forsaken", "2026-04-19", runID, "transcribe")
|
||||
wantStage := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID, "transcribe")
|
||||
if stageDir != wantStage {
|
||||
t.Fatalf("SessionRunStageDirForCampaign() = %q, want %q", stageDir, wantStage)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionRunManifestPathForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
runID := "20260515T031522Z-a1b2c3d4"
|
||||
got := SessionRunManifestPathForCampaign(root, "forsaken", "2026-04-19", runID)
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID, "manifest.json")
|
||||
if got != want {
|
||||
t.Fatalf("SessionRunManifestPathForCampaign() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionPreviousPathsForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
previousDir := SessionPreviousDirForCampaign(root, "forsaken", "2026-04-19")
|
||||
wantPreviousDir := filepath.Join(root, "work", "forsaken", "2026-04-19", "previous")
|
||||
if previousDir != wantPreviousDir {
|
||||
t.Fatalf("SessionPreviousDirForCampaign() = %q, want %q", previousDir, wantPreviousDir)
|
||||
}
|
||||
|
||||
manifestPath := SessionPreviousManifestPathForCampaign(root, "forsaken", "2026-04-19")
|
||||
wantManifestPath := filepath.Join(root, "work", "forsaken", "2026-04-19", "previous", "manifest.json")
|
||||
if manifestPath != wantManifestPath {
|
||||
t.Fatalf("SessionPreviousManifestPathForCampaign() = %q, want %q", manifestPath, wantManifestPath)
|
||||
}
|
||||
|
||||
artifactsDir := SessionPreviousArtifactsDirForCampaign(root, "forsaken", "2026-04-19")
|
||||
wantArtifactsDir := filepath.Join(root, "work", "forsaken", "2026-04-19", "previous", "artifacts")
|
||||
if artifactsDir != wantArtifactsDir {
|
||||
t.Fatalf("SessionPreviousArtifactsDirForCampaign() = %q, want %q", artifactsDir, wantArtifactsDir)
|
||||
}
|
||||
|
||||
artifactPath := SessionPreviousArtifactPathForCampaign(root, "forsaken", "2026-04-19", "session_recap.md")
|
||||
wantArtifactPath := filepath.Join(root, "work", "forsaken", "2026-04-19", "previous", "artifacts", "session_recap.md")
|
||||
if artifactPath != wantArtifactPath {
|
||||
t.Fatalf("SessionPreviousArtifactPathForCampaign() = %q, want %q", artifactPath, wantArtifactPath)
|
||||
}
|
||||
|
||||
archiveRelativeArtifactPath := SessionPreviousArtifactPathForCampaign(root, "forsaken", "2026-04-19", "artifacts/session_recap.md")
|
||||
if archiveRelativeArtifactPath != wantArtifactPath {
|
||||
t.Fatalf("SessionPreviousArtifactPathForCampaign(archive-relative) = %q, want %q", archiveRelativeArtifactPath, wantArtifactPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionPreviousPathsFromSessionPaths(t *testing.T) {
|
||||
paths := buildSessionPaths("/tmp/workspace", "forsaken", "2026-04-19")
|
||||
if got := SessionPreviousDir(paths); got != paths.PreviousDir {
|
||||
t.Fatalf("SessionPreviousDir() = %q, want %q", got, paths.PreviousDir)
|
||||
}
|
||||
if got := SessionPreviousManifestPath(paths); got != paths.PreviousManifestPath {
|
||||
t.Fatalf("SessionPreviousManifestPath() = %q, want %q", got, paths.PreviousManifestPath)
|
||||
}
|
||||
if got := SessionPreviousArtifactsDir(paths); got != paths.PreviousArtifactsDir {
|
||||
t.Fatalf("SessionPreviousArtifactsDir() = %q, want %q", got, paths.PreviousArtifactsDir)
|
||||
}
|
||||
|
||||
got := SessionPreviousArtifactPath(paths, "quest_log.json")
|
||||
want := filepath.Join(paths.PreviousArtifactsDir, "quest_log.json")
|
||||
if got != want {
|
||||
t.Fatalf("SessionPreviousArtifactPath() = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
got = SessionPreviousArtifactPath(paths, "artifacts/session_recap.md")
|
||||
want = filepath.Join(paths.PreviousArtifactsDir, "session_recap.md")
|
||||
if got != want {
|
||||
t.Fatalf("SessionPreviousArtifactPath(archive-relative) = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -22,3 +114,66 @@ func TestSessionSpoolAudioDir(t *testing.T) {
|
||||
t.Fatalf("SessionSpoolAudioDir() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionSpoolDir(t *testing.T) {
|
||||
root := "/var/spool/narratio"
|
||||
got := SessionSpoolDir(root, "forsaken", "2026-04-19")
|
||||
want := filepath.Join(root, "forsaken", "2026-04-19")
|
||||
if got != want {
|
||||
t.Fatalf("SessionSpoolDir() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionSpoolRestoreAudioDir(t *testing.T) {
|
||||
root := "/var/spool/narratio"
|
||||
got := SessionSpoolRestoreAudioDir(root, "forsaken", "2026-04-19")
|
||||
want := filepath.Join(root, "forsaken", "2026-04-19", "restore", "audio")
|
||||
if got != want {
|
||||
t.Fatalf("SessionSpoolRestoreAudioDir() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestS3AudioCachePath(t *testing.T) {
|
||||
got, err := S3AudioCachePath("/var/cache/narratio", "my-dnd-archive", "dnd/campaigns/forsaken/sessions/2026-04-19/audio/alice.flac")
|
||||
if err != nil {
|
||||
t.Fatalf("S3AudioCachePath() error = %v", err)
|
||||
}
|
||||
want := filepath.Join("/var/cache/narratio", "s3", "my-dnd-archive", "dnd", "campaigns", "forsaken", "sessions", "2026-04-19", "audio", "alice.flac")
|
||||
if got != want {
|
||||
t.Fatalf("S3AudioCachePath() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestS3AudioCacheNamespaceDir(t *testing.T) {
|
||||
got, err := S3AudioCacheNamespaceDir("/var/cache/narratio", "my-dnd-archive", "dnd")
|
||||
if err != nil {
|
||||
t.Fatalf("S3AudioCacheNamespaceDir() error = %v", err)
|
||||
}
|
||||
want := filepath.Join("/var/cache/narratio", "s3", "my-dnd-archive", "dnd", "campaigns")
|
||||
if got != want {
|
||||
t.Fatalf("S3AudioCacheNamespaceDir() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestS3AudioCachePathRejectsUnsafeInputs(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
root string
|
||||
bucket string
|
||||
key string
|
||||
}{
|
||||
{name: "empty root", root: "", bucket: "bucket", key: "audio/a.flac"},
|
||||
{name: "empty bucket", root: "/cache", bucket: "", key: "audio/a.flac"},
|
||||
{name: "bucket slash", root: "/cache", bucket: "bad/bucket", key: "audio/a.flac"},
|
||||
{name: "empty key", root: "/cache", bucket: "bucket", key: ""},
|
||||
{name: "escaping key", root: "/cache", bucket: "bucket", key: "../audio/a.flac"},
|
||||
{name: "absolute key", root: "/cache", bucket: "bucket", key: "/audio/a.flac"},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got, err := S3AudioCachePath(tt.root, tt.bucket, tt.key); err == nil {
|
||||
t.Fatalf("S3AudioCachePath() = %q, want error", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
112
internal/artifacts/previous_requirements.go
Normal file
112
internal/artifacts/previous_requirements.go
Normal file
@@ -0,0 +1,112 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
// PreviousArtifactRequirement describes one previous-session artifact dependency.
|
||||
type PreviousArtifactRequirement struct {
|
||||
Name string
|
||||
Required bool
|
||||
Sources []string
|
||||
}
|
||||
|
||||
// CollectPreviousArtifactRequirements scans enabled Scriptorium artifacts and returns
|
||||
// deduplicated previous-session artifact requirements in deterministic order.
|
||||
func CollectPreviousArtifactRequirements(
|
||||
artifactsCfg map[string]config.ScriptoriumArtifactConfig,
|
||||
) []PreviousArtifactRequirement {
|
||||
if len(artifactsCfg) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
artifactNames := sortedScriptoriumArtifactNames(artifactsCfg)
|
||||
byName := map[string]PreviousArtifactRequirement{}
|
||||
|
||||
for _, artifactName := range artifactNames {
|
||||
artifactCfg := artifactsCfg[artifactName]
|
||||
if !artifactCfg.Enabled {
|
||||
continue
|
||||
}
|
||||
|
||||
inputNames := sortedScriptoriumInputKeys(artifactCfg.Inputs)
|
||||
for _, inputName := range inputNames {
|
||||
inputCfg := artifactCfg.Inputs[inputName]
|
||||
previousName, ok := PreviousSessionArtifactName(inputCfg.Source)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
|
||||
location := fmt.Sprintf(
|
||||
"pipeline.scriptorium.artifacts.%s.inputs.%s.source",
|
||||
artifactName,
|
||||
inputName,
|
||||
)
|
||||
requirement := byName[previousName]
|
||||
requirement.Name = previousName
|
||||
requirement.Required = requirement.Required || inputCfg.Required
|
||||
requirement.Sources = append(requirement.Sources, location)
|
||||
byName[previousName] = requirement
|
||||
}
|
||||
}
|
||||
|
||||
if len(byName) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
requirements := make([]PreviousArtifactRequirement, 0, len(byName))
|
||||
for _, requirement := range byName {
|
||||
requirement.Sources = dedupeAndSortStrings(requirement.Sources)
|
||||
requirements = append(requirements, requirement)
|
||||
}
|
||||
sort.Slice(requirements, func(i, j int) bool {
|
||||
return requirements[i].Name < requirements[j].Name
|
||||
})
|
||||
return requirements
|
||||
}
|
||||
|
||||
func sortedScriptoriumArtifactNames(artifactsCfg map[string]config.ScriptoriumArtifactConfig) []string {
|
||||
names := make([]string, 0, len(artifactsCfg))
|
||||
for name := range artifactsCfg {
|
||||
names = append(names, name)
|
||||
}
|
||||
sort.Strings(names)
|
||||
return names
|
||||
}
|
||||
|
||||
func sortedScriptoriumInputKeys(inputs map[string]config.ScriptoriumInputConfig) []string {
|
||||
if len(inputs) == 0 {
|
||||
return nil
|
||||
}
|
||||
names := make([]string, 0, len(inputs))
|
||||
for name := range inputs {
|
||||
names = append(names, name)
|
||||
}
|
||||
sort.Strings(names)
|
||||
return names
|
||||
}
|
||||
|
||||
func dedupeAndSortStrings(values []string) []string {
|
||||
if len(values) == 0 {
|
||||
return nil
|
||||
}
|
||||
seen := map[string]struct{}{}
|
||||
out := make([]string, 0, len(values))
|
||||
for _, value := range values {
|
||||
trimmed := strings.TrimSpace(value)
|
||||
if trimmed == "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[trimmed]; ok {
|
||||
continue
|
||||
}
|
||||
seen[trimmed] = struct{}{}
|
||||
out = append(out, trimmed)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
190
internal/artifacts/previous_requirements_test.go
Normal file
190
internal/artifacts/previous_requirements_test.go
Normal file
@@ -0,0 +1,190 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func TestCollectPreviousArtifactRequirements(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
artifactsCfg map[string]config.ScriptoriumArtifactConfig
|
||||
want []PreviousArtifactRequirement
|
||||
}{
|
||||
{
|
||||
name: "no artifacts",
|
||||
artifactsCfg: nil,
|
||||
want: nil,
|
||||
},
|
||||
{
|
||||
name: "no previous inputs",
|
||||
artifactsCfg: map[string]config.ScriptoriumArtifactConfig{
|
||||
"session_recap": {
|
||||
Enabled: true,
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"transcript": {Source: "narratio.transcript.trimmed", Required: true},
|
||||
},
|
||||
},
|
||||
},
|
||||
want: nil,
|
||||
},
|
||||
{
|
||||
name: "one optional previous input",
|
||||
artifactsCfg: map[string]config.ScriptoriumArtifactConfig{
|
||||
"session_recap": {
|
||||
Enabled: true,
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"previous_recap": {Source: "narratio.previous_session.artifact.session_recap", Required: false},
|
||||
},
|
||||
},
|
||||
},
|
||||
want: []PreviousArtifactRequirement{
|
||||
{
|
||||
Name: "session_recap",
|
||||
Required: false,
|
||||
Sources: []string{"pipeline.scriptorium.artifacts.session_recap.inputs.previous_recap.source"},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "one required previous input",
|
||||
artifactsCfg: map[string]config.ScriptoriumArtifactConfig{
|
||||
"quest_log": {
|
||||
Enabled: true,
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"previous_quest_log": {Source: "narratio.previous_session.artifact.quest_log", Required: true},
|
||||
},
|
||||
},
|
||||
},
|
||||
want: []PreviousArtifactRequirement{
|
||||
{
|
||||
Name: "quest_log",
|
||||
Required: true,
|
||||
Sources: []string{"pipeline.scriptorium.artifacts.quest_log.inputs.previous_quest_log.source"},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "duplicate references are deduped",
|
||||
artifactsCfg: map[string]config.ScriptoriumArtifactConfig{
|
||||
"a": {
|
||||
Enabled: true,
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"x": {Source: "narratio.previous_session.artifact.session_recap", Required: false},
|
||||
},
|
||||
},
|
||||
"b": {
|
||||
Enabled: true,
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"y": {Source: "narratio.previous_session.artifact.session_recap", Required: false},
|
||||
},
|
||||
},
|
||||
},
|
||||
want: []PreviousArtifactRequirement{
|
||||
{
|
||||
Name: "session_recap",
|
||||
Required: false,
|
||||
Sources: []string{
|
||||
"pipeline.scriptorium.artifacts.a.inputs.x.source",
|
||||
"pipeline.scriptorium.artifacts.b.inputs.y.source",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "required plus optional reference becomes required",
|
||||
artifactsCfg: map[string]config.ScriptoriumArtifactConfig{
|
||||
"a": {
|
||||
Enabled: true,
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"x": {Source: "narratio.previous_session.artifact.session_recap", Required: false},
|
||||
},
|
||||
},
|
||||
"b": {
|
||||
Enabled: true,
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"y": {Source: "narratio.previous_session.artifact.session_recap", Required: true},
|
||||
},
|
||||
},
|
||||
},
|
||||
want: []PreviousArtifactRequirement{
|
||||
{
|
||||
Name: "session_recap",
|
||||
Required: true,
|
||||
Sources: []string{
|
||||
"pipeline.scriptorium.artifacts.a.inputs.x.source",
|
||||
"pipeline.scriptorium.artifacts.b.inputs.y.source",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "disabled artifact references are ignored",
|
||||
artifactsCfg: map[string]config.ScriptoriumArtifactConfig{
|
||||
"disabled_artifact": {
|
||||
Enabled: false,
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"x": {Source: "narratio.previous_session.artifact.session_recap", Required: true},
|
||||
},
|
||||
},
|
||||
"enabled_artifact": {
|
||||
Enabled: true,
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"y": {Source: "narratio.previous_session.artifact.quest_log", Required: false},
|
||||
},
|
||||
},
|
||||
},
|
||||
want: []PreviousArtifactRequirement{
|
||||
{
|
||||
Name: "quest_log",
|
||||
Required: false,
|
||||
Sources: []string{"pipeline.scriptorium.artifacts.enabled_artifact.inputs.y.source"},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "deterministic ordering",
|
||||
artifactsCfg: map[string]config.ScriptoriumArtifactConfig{
|
||||
"zz": {
|
||||
Enabled: true,
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"b_input": {Source: "narratio.previous_session.artifact.quest_log", Required: false},
|
||||
"a_input": {Source: "narratio.previous_session.artifact.session_recap", Required: false},
|
||||
},
|
||||
},
|
||||
"aa": {
|
||||
Enabled: true,
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"c_input": {Source: "narratio.previous_session.artifact.session_recap", Required: false},
|
||||
},
|
||||
},
|
||||
},
|
||||
want: []PreviousArtifactRequirement{
|
||||
{
|
||||
Name: "quest_log",
|
||||
Required: false,
|
||||
Sources: []string{"pipeline.scriptorium.artifacts.zz.inputs.b_input.source"},
|
||||
},
|
||||
{
|
||||
Name: "session_recap",
|
||||
Required: false,
|
||||
Sources: []string{
|
||||
"pipeline.scriptorium.artifacts.aa.inputs.c_input.source",
|
||||
"pipeline.scriptorium.artifacts.zz.inputs.a_input.source",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := CollectPreviousArtifactRequirements(tt.artifactsCfg)
|
||||
if !reflect.DeepEqual(got, tt.want) {
|
||||
t.Fatalf("CollectPreviousArtifactRequirements() = %#v, want %#v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -8,7 +8,7 @@ import (
|
||||
|
||||
func TestResolveSessionLocalPathForRead(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "s-1")
|
||||
paths := buildSessionPaths(workspace, "sample-campaign", "s-1")
|
||||
if err := os.MkdirAll(paths.TranscriptsRawDir, 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
@@ -41,7 +41,7 @@ func TestResolveSessionLocalPathForReadRelativeWorkspaceRootQualifiedPath(t *tes
|
||||
t.Fatalf("Rel() error = %v", err)
|
||||
}
|
||||
|
||||
paths := buildSessionPaths(workspaceRel, "s-1")
|
||||
paths := buildSessionPaths(workspaceRel, "sample-campaign", "s-1")
|
||||
target := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
@@ -50,7 +50,7 @@ func TestResolveSessionLocalPathForReadRelativeWorkspaceRootQualifiedPath(t *tes
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
manifestPath := filepath.Join(workspaceRel, "work", "s-1", "transcripts", "raw", "alice.json")
|
||||
manifestPath := filepath.Join(workspaceRel, "work", "sample-campaign", "s-1", "transcripts", "raw", "alice.json")
|
||||
got := ResolveSessionLocalPathForRead(paths, manifestPath)
|
||||
if got != filepath.Clean(manifestPath) {
|
||||
t.Fatalf("resolution = %q, want %q", got, filepath.Clean(manifestPath))
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user