Compare commits
26 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| c5c35cd3b4 | |||
| 574b1cde6c | |||
| ebb21b9201 | |||
| 958f446387 | |||
| 86caf4b222 | |||
| e38ed8ba97 | |||
| 3e79cf4724 | |||
| 859ae1ae10 | |||
| c63ecbab32 | |||
| 8480b74283 | |||
| 087869f7fa | |||
| 2b2a314d65 | |||
| 08b0f4edc5 | |||
| 571a289296 | |||
| 9f80635b42 | |||
| 11a3e174b6 | |||
| 9c5e5d6dc1 | |||
| c4e87f58c7 | |||
| 37daab7857 | |||
| 2356688cb9 | |||
| 1054b64d9f | |||
| 01fb02426c | |||
| 7dc79e052f | |||
| cb525c0f72 | |||
| 622677d038 | |||
| 550288e008 |
465
README.md
465
README.md
@@ -1,461 +1,22 @@
|
||||
# narratio
|
||||
|
||||
`narratio` is a Go orchestration application for processing D&D session audio into transcripts and generated artifacts.
|
||||
Narratio is a Go orchestration application that turns D&D session audio into polished transcripts and generated session artifacts.
|
||||
|
||||
## Current Implementation
|
||||
|
||||
Implemented now:
|
||||
|
||||
- strict config loading/validation (`pipeline.yml` and `session.yml`)
|
||||
- local workspace/session layout, locking, and manifest persistence
|
||||
- resumable stage control (`run`, `plan`, `resume`, `run-stage`, `status`)
|
||||
- real `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` stages
|
||||
- real WhisperX, Seriatim, and Audita adapters
|
||||
- real Scriptorium subprocess adapter
|
||||
- optional Scriptorium render diagnostics (`render_debug`)
|
||||
|
||||
Not implemented yet:
|
||||
|
||||
- `notify` stage behavior
|
||||
- additional analyze artifacts beyond `session_recap`
|
||||
- generic DAG orchestration
|
||||
|
||||
## Config Files
|
||||
|
||||
Narratio expects two YAML files:
|
||||
|
||||
- `pipeline.yml`: pipeline/workspace settings
|
||||
- `session.yml`: per-session settings
|
||||
|
||||
Pipeline config lookup for CLI commands:
|
||||
|
||||
- if `--config <path>` is provided, Narratio uses that path
|
||||
- if `--config` is omitted, Narratio searches in this order:
|
||||
- `/usr/local/etc/narratio/pipeline.yml`
|
||||
- `/etc/narratio/pipeline.yml`
|
||||
|
||||
Session config lookup for CLI commands:
|
||||
|
||||
- if `--session <path>` is provided, Narratio uses that path
|
||||
- if `--session` is omitted, Narratio searches in this order:
|
||||
- `./session.yml`
|
||||
- `/usr/local/etc/narratio/session.yml`
|
||||
- `/etc/narratio/session.yml`
|
||||
|
||||
Session template support:
|
||||
|
||||
- Narratio renders `session.yml` templates before strict YAML decode.
|
||||
- `--session-id <value>` provides the `session_id` template variable.
|
||||
- Supported placeholder forms:
|
||||
- `{{session_id}}`
|
||||
- `{{ session_id }}`
|
||||
- unresolved template placeholders fail with a clear error.
|
||||
- strict YAML validation still runs after rendering.
|
||||
- concrete `session.yml` files without templates remain fully supported.
|
||||
|
||||
Optional secrets-from-files config:
|
||||
|
||||
- `pipeline.secrets.env_dir` may point to a directory of secret files
|
||||
- each top-level file with an env-var-style name is loaded as an environment variable:
|
||||
- file name = env var name
|
||||
- file contents = env var value (trailing newline/CRLF trimmed)
|
||||
- process environment wins: existing env vars are not overwritten
|
||||
- if configured, Narratio fails fast when `env_dir` is missing/unreadable
|
||||
- relative `env_dir` values resolve from Narratio’s current working directory
|
||||
|
||||
YAML decoding is strict (`KnownFields(true)`), so unknown fields fail fast.
|
||||
|
||||
Maintainer note: application defaults are centralized in [`internal/config/defaults.go`](internal/config/defaults.go).
|
||||
|
||||
## Storage And Archive Foundations
|
||||
|
||||
Narratio now includes configuration and path-model foundations for archive support, plus implemented prepare-stage S3 audio input.
|
||||
|
||||
Implemented foundations:
|
||||
|
||||
- `pipeline.storage.s3` config shape (`bucket`, `root_prefix`, `region`, `endpoint`, `force_path_style`, `access_key_id_env`, `secret_access_key_env`)
|
||||
- `pipeline.spool` config shape (`root`, `delete_audio_after_archive`)
|
||||
- `pipeline.archive` config shape (`enabled`, `upload_run`, `promote_artifacts`)
|
||||
- promotion-rule validation (`from`/`to` required, relative-only paths, traversal rejected)
|
||||
- `session.campaign` requirement for campaign-aware path construction
|
||||
- optional `session.inputs.audio_s3.prefix` modeling and prepare-stage S3 audio download
|
||||
- run ID generation and S3/local path helper foundations
|
||||
- manifest run/path identity fields
|
||||
|
||||
Current defaults:
|
||||
|
||||
- `pipeline.storage.s3.root_prefix`: `dnd`
|
||||
- `pipeline.storage.s3.access_key_id_env`: `OBJECT_STORAGE_KEY_ID`
|
||||
- `pipeline.storage.s3.secret_access_key_env`: `OBJECT_STORAGE_KEY`
|
||||
- `pipeline.workspace.cleanup_after_archive`: `false`
|
||||
- `pipeline.spool.root`: `/var/spool/narratio`
|
||||
- `pipeline.spool.delete_audio_after_archive`: `false`
|
||||
- `pipeline.archive.enabled`: `true`
|
||||
- `pipeline.archive.upload_run`: `true`
|
||||
- default `pipeline.archive.promote_artifacts`:
|
||||
- `transcripts/trimmed.json` -> `transcripts/trimmed.json` (`required: true`)
|
||||
- `artifacts/session_recap.md` -> `artifacts/session_recap.md` (`required: true`)
|
||||
|
||||
Current boundaries:
|
||||
|
||||
- local development audio (`audio_dir` / `audio_files`) still works
|
||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive
|
||||
- real S3-compatible backend now exists in the storage adapter package
|
||||
- storage backend tests use fake storage and do not require live S3
|
||||
- archive uploads successful run records under `runs/{run_id}/`
|
||||
- archive does not upload local audio by default
|
||||
- archive uploads promoted outputs to session-level keys using `archive.promote_artifacts`
|
||||
- archive uploads `current/manifest.json`
|
||||
- archive uploads `current/run_id.txt` last as the effective commit marker
|
||||
- required missing promotions fail archive
|
||||
- optional missing promotions are skipped and recorded
|
||||
- cleanup remains conservative and opt-in:
|
||||
- `pipeline.spool.delete_audio_after_archive: true` removes only the run-scoped spool audio directory after successful archive commit
|
||||
- `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir after successful archive commit
|
||||
- cleanup executes only after all selected stages for the command invocation succeed
|
||||
- cleanup does not run for failed, incomplete, skipped, or unarchived runs
|
||||
- local development `audio_dir`/`audio_files` source inputs are never deleted by spool cleanup
|
||||
- S3 credentials are resolved from configured env-var names when both are present; if either is missing, Narratio falls back to the AWS SDK default credential chain
|
||||
|
||||
S3 input details and current boundaries are documented in [docs/s3-audio-input.md](docs/s3-audio-input.md).
|
||||
|
||||
## Remote Storage Backend
|
||||
|
||||
Narratio includes an object-store backend layer for future prepare/archive work:
|
||||
|
||||
- `List(ctx, prefix)`
|
||||
- `Download(ctx, key, localPath)`
|
||||
- `Upload(ctx, localPath, key, opts)`
|
||||
- `Exists(ctx, key)`
|
||||
|
||||
Implemented backends:
|
||||
|
||||
- fake storage backend for deterministic tests
|
||||
- S3-compatible backend built from `pipeline.storage.s3`
|
||||
|
||||
Key invariant:
|
||||
|
||||
- callers pass full bucket-relative object keys
|
||||
- storage backends do not prepend `root_prefix` and do not infer session/campaign paths
|
||||
|
||||
Current boundary:
|
||||
|
||||
- `prepare` uses `List` + `Download` through the backend when `session.inputs.audio_s3` is configured
|
||||
- `archive` uses `Upload` through the backend for successful run-record uploads
|
||||
- `archive` also uses `Upload` for promotion writes and current pointers
|
||||
- no failed or incomplete runs are uploaded
|
||||
- local audio is not re-uploaded by default
|
||||
|
||||
Archive run-upload details and boundaries are documented in [docs/archive-storage.md](docs/archive-storage.md).
|
||||
|
||||
## Canonical Stage Order
|
||||
|
||||
1. `prepare`
|
||||
2. `transcribe`
|
||||
3. `merge`
|
||||
4. `polish`
|
||||
5. `normalize`
|
||||
6. `trim`
|
||||
7. `analyze`
|
||||
8. `archive`
|
||||
9. `notify`
|
||||
|
||||
## Transcript Tiers
|
||||
|
||||
- `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge
|
||||
- `transcripts/processed.json`: full raw Audita-polished transcript output
|
||||
- `transcripts/normalized.json`: Seriatim-normalized transcript from the normalize stage
|
||||
- `transcripts/trimmed.json`: gameplay-only normalized polished transcript from trim stage
|
||||
|
||||
## Seriatim Configuration
|
||||
|
||||
`pipeline.seriatim` configures the Seriatim subprocess adapter used by `merge`, `normalize`, and `trim`.
|
||||
|
||||
Minimal behavior:
|
||||
|
||||
- `pipeline.seriatim` may be omitted entirely.
|
||||
- when omitted, Narratio defaults to:
|
||||
- `binary: seriatim`
|
||||
- `timeout: 10m`
|
||||
- `output_schema: seriatim-intermediate`
|
||||
- `coalesce_gap: 3.0`
|
||||
- `report: true`
|
||||
|
||||
Optional overrides in `pipeline.seriatim` continue to work, including explicit binary paths and advanced `env` tuning values.
|
||||
|
||||
## Audita Configuration
|
||||
|
||||
`pipeline.audita` configures the real Audita subprocess adapter used by `polish`.
|
||||
|
||||
Minimal behavior:
|
||||
|
||||
- `pipeline.audita` may be omitted entirely.
|
||||
- when omitted, Narratio defaults to:
|
||||
- `binary: audita`
|
||||
- `timeout: 3h`
|
||||
- `report: true`
|
||||
|
||||
Optional:
|
||||
|
||||
- `llm_api_key_env` (when set, Narratio requires that env var and passes it to Audita as `AUDITA_LLM_API_KEY`)
|
||||
- `modules` override list (when empty/omitted, Narratio does not pass `--modules`)
|
||||
- `base_url` (when omitted, Narratio does not pass `--base-url`; Audita runtime defaults/config may apply)
|
||||
- `model` (when omitted, Narratio does not pass `--model`; Audita runtime defaults/config may apply)
|
||||
- `transcript_description`
|
||||
- `config_path`
|
||||
- `output_schema` (`bare-segments` or `audita-v1`)
|
||||
- `work_dir_retention` (`always`, `auto`, or `never`)
|
||||
- `total_llm_concurrency` (> 0 when provided)
|
||||
- `proposal_llm_concurrency` (> 0 when provided)
|
||||
- `validation_model`
|
||||
- `validation_llm_concurrency` (> 0 when provided)
|
||||
- `report` (defaults to `true`)
|
||||
|
||||
Narratio passes only configured optional Audita flags. Omitted optional values are left to Audita runtime defaults/config.
|
||||
|
||||
## Normalize Configuration
|
||||
|
||||
`pipeline.normalize` is optional. When omitted, Narratio defaults to:
|
||||
|
||||
- `output_path: transcripts/normalized.json`
|
||||
- `output_schema: seriatim-intermediate`
|
||||
- `report: true`
|
||||
|
||||
Allowed `normalize.output_schema` values:
|
||||
|
||||
- `seriatim-minimal`
|
||||
- `seriatim-intermediate`
|
||||
- `seriatim-full`
|
||||
|
||||
`normalize.output_path` is treated as session-workdir-relative when not absolute.
|
||||
|
||||
Normalize stage behavior summary:
|
||||
|
||||
- normalize runs after `polish` and before `trim`
|
||||
- normalize resolves `transcripts/processed.json`
|
||||
- normalize runs Seriatim `normalize` to produce `transcripts/normalized.json`
|
||||
- normalize diagnostics are written to:
|
||||
- `artifacts/seriatim.normalize.report.json` (when enabled)
|
||||
- `logs/seriatim.normalize.stdout.log`
|
||||
- `logs/seriatim.normalize.stderr.log`
|
||||
- `config/seriatim.normalize.generated.yml`
|
||||
|
||||
## Trim Configuration
|
||||
|
||||
`pipeline.trim` is optional. If omitted, no trim config is loaded. If `trim.enabled` is omitted, it defaults to `false`.
|
||||
|
||||
When `trim.enabled: true`:
|
||||
|
||||
- `trim.output_path` is required
|
||||
- `trim.bounds.prompt_id` is required
|
||||
- `trim.bounds.transcript_input_name` is required
|
||||
- `trim.bounds.output_path` is required
|
||||
- `trim.bounds.timeout` must be a valid Go duration when provided
|
||||
- `trim.bounds.render_debug: true` requires `trim.bounds.render_output_path`
|
||||
- `trim.bounds.profile_id` may be empty to use the prompt default profile
|
||||
|
||||
Trim paths are treated as session-workdir-relative when not absolute.
|
||||
|
||||
Example trim config:
|
||||
|
||||
```yaml
|
||||
trim:
|
||||
enabled: true
|
||||
output_path: "transcripts/trimmed.json"
|
||||
bounds:
|
||||
prompt_id: "dnd_session.bounds"
|
||||
profile_id: ""
|
||||
transcript_input_name: "transcript"
|
||||
output_path: "artifacts/session_bounds.json"
|
||||
timeout: "10m"
|
||||
render_debug: false
|
||||
render_output_path: "artifacts/session_bounds.render.json"
|
||||
seriatim:
|
||||
report: false
|
||||
```
|
||||
|
||||
Trim behavior summary:
|
||||
|
||||
- trim discovers and validates `transcripts/normalized.json`
|
||||
- trim uses Scriptorium bounds (`dnd_session.bounds` by example config) to produce `artifacts/session_bounds.json`
|
||||
- bounds IDs are validated against the same normalized transcript ID space that Seriatim trim will consume
|
||||
- trim converts bounds to Seriatim keep selector (for example `10-868`) and runs Seriatim trim
|
||||
- if trim is disabled, Narratio copies normalized transcript to trimmed transcript and records `trim_action=copy_disabled`
|
||||
|
||||
Trim outputs and diagnostics:
|
||||
|
||||
- `artifacts/session_bounds.json`
|
||||
- `transcripts/trimmed.json`
|
||||
- `logs/scriptorium.bounds.stdout.log`
|
||||
- `logs/scriptorium.bounds.stderr.log`
|
||||
- `config/scriptorium.bounds.generated.yml`
|
||||
- `logs/seriatim.trim.stdout.log`
|
||||
- `logs/seriatim.trim.stderr.log`
|
||||
- `config/seriatim.trim.generated.yml`
|
||||
- optional bounds render-debug outputs:
|
||||
- `artifacts/session_bounds.render.json`
|
||||
- `logs/scriptorium.bounds.render.stdout.log`
|
||||
- `logs/scriptorium.bounds.render.stderr.log`
|
||||
- `config/scriptorium.bounds.render.generated.yml`
|
||||
|
||||
Render-debug files are diagnostics and are not treated as canonical stage output artifact refs.
|
||||
|
||||
## Scriptorium Configuration
|
||||
|
||||
`pipeline.scriptorium` is optional. When present, Narratio validates and uses it for analyze-stage artifact generation.
|
||||
|
||||
Key points:
|
||||
|
||||
- `scriptorium.binary` defaults to `scriptorium` when section is present
|
||||
- `scriptorium.config_path` is optional
|
||||
- `scriptorium.timeout` defaults to `10m` when omitted
|
||||
- `scriptorium.render_debug` enables render diagnostics globally
|
||||
- artifacts are configured under `scriptorium.artifacts` (map shape supports multiple artifacts)
|
||||
- enabled artifacts require `prompt_id` and `output_path`
|
||||
- artifact `render_debug` may override global render setting
|
||||
- `vars` currently support boolean and string values
|
||||
|
||||
Example `session_recap` artifact definition:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
binary: "scriptorium"
|
||||
config_path: "/etc/scriptorium/config.yml"
|
||||
timeout: "10m"
|
||||
render_debug: false
|
||||
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: "dnd.session_recap"
|
||||
profile_id: "local-quality" # optional
|
||||
output_path: "artifacts/session_recap.md"
|
||||
timeout: "10m"
|
||||
# render_debug: true # optional per-artifact override
|
||||
|
||||
inputs:
|
||||
transcript:
|
||||
source: "trimmed_transcript"
|
||||
required: true
|
||||
|
||||
previous_recap:
|
||||
source: "previous_session_artifact"
|
||||
artifact: "session_recap"
|
||||
path: "" # optional; set when available
|
||||
required: false
|
||||
|
||||
vars:
|
||||
session_id: true
|
||||
session_date: true
|
||||
campaign_name: true
|
||||
previous_session_id: true
|
||||
output_kind: "session_recap"
|
||||
```
|
||||
|
||||
Prompt IDs and profile IDs are configuration values. They are not hardcoded in analyze-stage logic.
|
||||
|
||||
Do not put secrets in `pipeline.yml`. If API-key behavior is configured, use env var names only.
|
||||
|
||||
If `pipeline.secrets.env_dir` is configured, keep only references and secret files there; secret values are still not written to manifests, generated configs, or Narratio-managed logs.
|
||||
|
||||
## Scriptorium Runtime Behavior
|
||||
|
||||
Narratio integrates with Scriptorium through the public CLI subprocess contract:
|
||||
|
||||
- generation: `scriptorium run`
|
||||
- diagnostics/testing: `scriptorium render --format json` when `render_debug` is enabled
|
||||
|
||||
For the initial implementation, only `session_recap` generation is supported.
|
||||
|
||||
Analyze-stage session recap behavior:
|
||||
|
||||
- available transcript input sources for configured artifacts: `processed_transcript`, `normalized_transcript`, `trimmed_transcript`
|
||||
- session recap should use gameplay-only transcript input (`source: trimmed_transcript`)
|
||||
- Narratio resolves `trimmed_transcript` from trim manifest output (`transcript_trimmed`) or fallback `transcripts/trimmed.json`
|
||||
- Narratio resolves `normalized_transcript` from normalize manifest output (`transcript_normalized`) or fallback `transcripts/normalized.json`
|
||||
- missing trimmed transcript fails clearly and advises running trim stage first
|
||||
- `normalized_transcript` is the preferred full-transcript source for future table/meta-analysis artifacts
|
||||
- `processed_transcript` remains supported for advanced/debug use cases
|
||||
- optionally includes `previous_recap` when configured and resolvable
|
||||
- omits optional previous recap when unavailable
|
||||
- fails if required inputs are missing
|
||||
- validates output file exists and is non-empty
|
||||
|
||||
Expected session output paths:
|
||||
|
||||
- `artifacts/session_recap.md`
|
||||
- `logs/scriptorium.session_recap.stdout.log`
|
||||
- `logs/scriptorium.session_recap.stderr.log`
|
||||
- `config/scriptorium.session_recap.generated.yml`
|
||||
- `artifacts/session_recap.render.json` when render diagnostics are enabled
|
||||
|
||||
## Examples
|
||||
|
||||
Starter files:
|
||||
|
||||
- `examples/pipeline.minimal.yml`
|
||||
- `examples/pipeline.audita-overrides.yml`
|
||||
- `examples/session.minimal.yml`
|
||||
- `examples/session.template.yml`
|
||||
- `examples/speakers.yml`
|
||||
|
||||
## Commands
|
||||
|
||||
Run tests:
|
||||
It coordinates transcription, merge/polish/normalize/trim processing, artifact generation, archive publishing, and resumable run state in one operator workflow.
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Plan a run:
|
||||
This command requires discoverable `pipeline.yml` and `session.yml` files (or explicit `--config` and `--session` flags).
|
||||
|
||||
```bash
|
||||
go run ./cmd/narratio plan --session examples/session.minimal.yml
|
||||
```
|
||||
## Documentation
|
||||
|
||||
Use `--config <path>` to override default pipeline lookup when needed.
|
||||
|
||||
Run with a discoverable session template:
|
||||
|
||||
```bash
|
||||
go run ./cmd/narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Run full pipeline:
|
||||
|
||||
```bash
|
||||
go run ./cmd/narratio run --config examples/pipeline.minimal.yml --session examples/session.minimal.yml
|
||||
```
|
||||
|
||||
Run analyze only:
|
||||
|
||||
```bash
|
||||
go run ./cmd/narratio run-stage --config examples/pipeline.minimal.yml --session examples/session.minimal.yml analyze
|
||||
```
|
||||
|
||||
Resume with a template session ID:
|
||||
|
||||
```bash
|
||||
go run ./cmd/narratio resume --config examples/pipeline.minimal.yml --session examples/session.template.yml --session-id 2026-04-04
|
||||
```
|
||||
|
||||
## Operational Note
|
||||
|
||||
Checksum-based stale detection is not implemented yet.
|
||||
|
||||
If prepared inputs or prompt/runtime config change, rerun the appropriate upstream stages before relying on downstream artifacts.
|
||||
|
||||
Examples:
|
||||
|
||||
- glossary/autocorrect/speaker-context changes: rerun at least `merge`, `polish`, `normalize`, `trim`, and `analyze`
|
||||
- trim bounds prompt/profile/config changes: rerun at least `normalize`, `trim`, and `analyze`
|
||||
- session recap prompt/profile/input-source changes: rerun `analyze`
|
||||
|
||||
## Roadmap
|
||||
|
||||
Near-term roadmap:
|
||||
|
||||
- extend analyze to additional configured artifacts
|
||||
- support workflows where later artifacts consume earlier generated artifacts
|
||||
- keep orchestration explicit without a generic DAG engine
|
||||
- implement archive and notify backends
|
||||
- [Configuration](docs/config.md)
|
||||
- [CLI Reference](docs/cli.md)
|
||||
- [Operations and Recovery](docs/operations.md)
|
||||
- [Troubleshooting](docs/troubleshooting.md)
|
||||
- [Development Guide](docs/development.md)
|
||||
- [Architecture Principles](docs/architecture.md)
|
||||
- [Internal Component Contracts](docs/internal/README.md)
|
||||
- [Config Examples](examples/)
|
||||
|
||||
502
architecture.md
502
architecture.md
@@ -1,502 +0,0 @@
|
||||
# Narratio Architecture
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
`narratio` is a Go orchestrator for D&D session processing. It runs a stage-based local pipeline from audio input through transcript processing and artifact generation, with manifest-based skip/force/resume behavior.
|
||||
|
||||
Narratio integrates with Scriptorium through the **public CLI** (`scriptorium run` and `scriptorium render`) via synchronous subprocess execution.
|
||||
|
||||
## 2. Current Status
|
||||
|
||||
Implemented:
|
||||
|
||||
- strict `pipeline.yml` + `session.yml` loading with strict YAML field checking (`KnownFields(true)`)
|
||||
- local workspace/session layout, lock file handling, artifact path helpers, checksums, and atomic writes
|
||||
- manifest store and stage status transitions for resumable runs
|
||||
- real `prepare`, `transcribe`, `merge`, and `polish` stages
|
||||
- real WhisperX HTTP adapter
|
||||
- real Seriatim subprocess adapter
|
||||
- real Audita subprocess adapter
|
||||
- real Scriptorium subprocess adapter
|
||||
- real `normalize` stage producing `transcripts/normalized.json`
|
||||
- real `trim` stage producing `transcripts/trimmed.json`
|
||||
- real `analyze` stage for initial `session_recap` generation
|
||||
- optional Scriptorium render diagnostics (`render_debug`) before production run
|
||||
- storage/archive configuration and validation foundations for:
|
||||
- `pipeline.storage.s3`
|
||||
- `pipeline.spool`
|
||||
- `pipeline.archive` promotion rules
|
||||
- `session.inputs.audio_s3`
|
||||
- run identity and path-model foundations:
|
||||
- run ID generation (`YYYYMMDDTHHMMSSZ-xxxxxxxx`)
|
||||
- S3 session/run/current key builders
|
||||
- campaign/session/run local work/spool path helpers
|
||||
- manifest run/path identity fields (`campaign`, `run_id`, local and S3 prefixes)
|
||||
- remote storage backend layer:
|
||||
- narrow object-store interface (`List`, `Download`, `Upload`, `Exists`)
|
||||
- fake storage backend for deterministic tests (no network dependency)
|
||||
- S3-compatible backend using AWS SDK v2
|
||||
- config-based object-store construction helper
|
||||
|
||||
Still placeholder/future:
|
||||
|
||||
- `notify` stage behavior
|
||||
- additional Scriptorium artifact types beyond `session_recap`
|
||||
- artifact-to-artifact workflows beyond the initial single-artifact implementation
|
||||
- generic stale detection based on input/config checksums
|
||||
|
||||
## 3. Pipeline and Stage Boundaries
|
||||
|
||||
Canonical stage order:
|
||||
|
||||
1. `prepare`
|
||||
2. `transcribe`
|
||||
3. `merge`
|
||||
4. `polish`
|
||||
5. `normalize`
|
||||
6. `trim`
|
||||
7. `analyze`
|
||||
8. `archive`
|
||||
9. `notify`
|
||||
|
||||
Boundary rules:
|
||||
|
||||
- orchestration logic lives in `internal/app`
|
||||
- stage business logic lives in `internal/stage`
|
||||
- external-tool CLI construction lives in adapter packages
|
||||
- Scriptorium CLI details stay in `internal/adapters/scriptorium`
|
||||
|
||||
## 4. Scriptorium Integration Model
|
||||
|
||||
Integration mode:
|
||||
|
||||
- public CLI subprocesses only (no Scriptorium internal Go packages, no HTTP API)
|
||||
- production generation uses `scriptorium run`
|
||||
- diagnostics/testing render uses `scriptorium render --format json`
|
||||
|
||||
Run invocation shape used by adapter:
|
||||
|
||||
```bash
|
||||
scriptorium run --prompt <prompt_id> --input name=path --out <output_path>
|
||||
```
|
||||
|
||||
Optional flags passed when configured:
|
||||
|
||||
- `--config <path>`
|
||||
- `--profile <profile_id>`
|
||||
- repeated `--var name=value`
|
||||
- repeated `--input name=path`
|
||||
- `--timeout <duration>`
|
||||
- `--api-key-env <ENV_NAME>` when configured
|
||||
|
||||
Render invocation shape used by adapter:
|
||||
|
||||
```bash
|
||||
scriptorium render --prompt <prompt_id> --input name=path --format json --out <render_output_path>
|
||||
```
|
||||
|
||||
Adapter behavior:
|
||||
|
||||
- always passes `--out`
|
||||
- captures stdout/stderr separately
|
||||
- writes generated invocation metadata YAML (redacted, no secrets)
|
||||
- treats exit code `0` as success
|
||||
- treats exit code `1` as failure
|
||||
- treats exit code `2` as failure with `validation_failed=true` and preserves output metadata when available
|
||||
- validates successful output files exist and are non-empty
|
||||
- does not treat non-empty stderr as failure by itself
|
||||
|
||||
## 5. Configuration Contract
|
||||
|
||||
CLI pipeline config path resolution:
|
||||
|
||||
- when `--config <path>` is provided, that path is used
|
||||
- when `--config` is omitted, Narratio searches defaults in order:
|
||||
- `/usr/local/etc/narratio/pipeline.yml`
|
||||
- `/etc/narratio/pipeline.yml`
|
||||
- default values are centralized in `internal/config/defaults.go`
|
||||
|
||||
CLI session config path resolution:
|
||||
|
||||
- when `--session <path>` is provided, that path is used
|
||||
- when `--session` is omitted, Narratio searches defaults in order:
|
||||
- `./session.yml`
|
||||
- `/usr/local/etc/narratio/session.yml`
|
||||
- `/etc/narratio/session.yml`
|
||||
|
||||
Session template rendering:
|
||||
|
||||
- session templates are rendered before strict YAML decode
|
||||
- `--session-id <value>` provides the `session_id` template variable
|
||||
- supported placeholders:
|
||||
- `{{session_id}}`
|
||||
- `{{ session_id }}`
|
||||
- unresolved placeholders fail clearly
|
||||
- strict `KnownFields(true)` YAML validation still applies after rendering
|
||||
- if rendered `session.session_id` conflicts with `--session-id`, load fails clearly
|
||||
|
||||
Optional pipeline secrets directory:
|
||||
|
||||
- `pipeline.secrets.env_dir` enables loading environment variables from local files before command execution
|
||||
- file name = env var name; file contents = env var value (trailing newline/CRLF trimmed)
|
||||
- only env-var-style file names are considered; other entries are ignored
|
||||
- existing process environment values are preserved (not overwritten)
|
||||
- if configured, unreadable/missing `env_dir` fails command execution early
|
||||
- relative `env_dir` values are resolved from current working directory
|
||||
|
||||
Storage and archive foundations:
|
||||
|
||||
- `pipeline.storage.s3` is available for modeling S3 coordinates:
|
||||
- `bucket`
|
||||
- `root_prefix` (default `dnd`)
|
||||
- `region`
|
||||
- `endpoint`
|
||||
- `force_path_style` (default `false`)
|
||||
- `access_key_id_env` (default `OBJECT_STORAGE_KEY_ID`)
|
||||
- `secret_access_key_env` (default `OBJECT_STORAGE_KEY`)
|
||||
- `pipeline.spool.root` defaults to `/var/spool/narratio`
|
||||
- `pipeline.workspace.cleanup_after_archive` defaults to `false`
|
||||
- `pipeline.spool.delete_audio_after_archive` defaults to `false`
|
||||
- `pipeline.archive` is optional and defaults to:
|
||||
- `enabled: true`
|
||||
- `upload_run: true`
|
||||
- default `promote_artifacts`:
|
||||
- `transcripts/trimmed.json`
|
||||
- `artifacts/session_recap.md`
|
||||
- archive promotion rules enforce safe relative paths:
|
||||
- `from` and `to` are required
|
||||
- absolute paths are rejected
|
||||
- traversal segments such as `..` are rejected
|
||||
|
||||
Session input foundations:
|
||||
|
||||
- `session.campaign` is required
|
||||
- local audio remains supported through `session.inputs.audio_dir` or `session.inputs.audio_files`
|
||||
- optional S3 audio input shape is `session.inputs.audio_s3.prefix`
|
||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive
|
||||
- when `audio_s3` is configured, `prepare` lists and downloads `.flac` objects through the object-store backend
|
||||
|
||||
Cross-config validation scope:
|
||||
|
||||
- `pipeline.storage.s3.bucket` is required only when an S3-dependent feature is explicitly configured (for current foundations, that includes `session.inputs.audio_s3`, and archive upload intent when using `storage.backend: s3`)
|
||||
- no AWS credential values are stored in Narratio config; only env-var names are configured
|
||||
- when both configured credential env vars resolve to non-empty values, the S3 backend uses them as static credentials
|
||||
- when either configured credential value is missing, the S3 backend falls back to the AWS SDK default credential chain
|
||||
|
||||
Remote object-store backend scope:
|
||||
|
||||
- remote storage APIs are isolated to `internal/adapters/storage`
|
||||
- AWS SDK types remain contained within the S3 backend implementation package
|
||||
- S3 key/session path semantics remain outside the backend, with this invariant:
|
||||
- callers pass full bucket-relative object keys
|
||||
- backend methods do not prepend `root_prefix` or infer campaign/session/run paths
|
||||
- `prepare` now uses object-store `List` and `Download` for S3 audio input
|
||||
- `archive` now uses object-store `Upload` for successful run-record upload under the run prefix
|
||||
- `archive` now uses object-store `Upload` for promoted outputs and current pointers
|
||||
|
||||
Prepare S3 audio behavior (implemented):
|
||||
|
||||
- compute session prefix as `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
- resolve `session.inputs.audio_s3.prefix` under that session prefix
|
||||
- list objects under the computed audio prefix and filter `.flac` keys
|
||||
- fail clearly when no `.flac` objects are found
|
||||
- download selected objects to spool audio path:
|
||||
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- materialize audio files into workdir audio path:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- record S3 provenance in manifest input records (bucket/key/metadata/local paths/checksum)
|
||||
- no AWS SDK types are used in stage code; storage implementation details stay in storage adapter packages
|
||||
|
||||
Archive publishing behavior (implemented):
|
||||
|
||||
- `archive` verifies prerequisite stage success before upload:
|
||||
- `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`
|
||||
- only successful/completed runs are uploaded
|
||||
- uploaded run record destination is:
|
||||
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
|
||||
- uploaded existing local paths include:
|
||||
- `inputs/`, `transcripts/`, `artifacts/`, optional `reports/`, `config/`, `logs/`, and `manifest.json`
|
||||
- local `audio/` is intentionally excluded from upload by default
|
||||
- file upload order is deterministic (sorted relative paths)
|
||||
- `archive.enabled: false` and `archive.upload_run: false` skip upload cleanly
|
||||
- stage metadata records non-secret upload context:
|
||||
- run upload details, promoted output details, current manifest key, current pointer key
|
||||
- no secrets, transcript contents, prompt contents, or environment dumps
|
||||
- promotion rules:
|
||||
- `from` resolves from local workdir
|
||||
- `to` resolves under session-level S3 root
|
||||
- missing required source fails archive
|
||||
- missing optional source is skipped and recorded
|
||||
- default promoted outputs:
|
||||
- `transcripts/trimmed.json`
|
||||
- `artifacts/session_recap.md`
|
||||
- current pointers:
|
||||
- `current/manifest.json` uploaded after run upload and promotions
|
||||
- `current/run_id.txt` uploaded last with `{run_id}\n`
|
||||
- `current/run_id.txt` is the effective commit marker
|
||||
- if promotion or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`
|
||||
- failed/incomplete runs remain local and are not uploaded
|
||||
- post-archive local cleanup (implemented, opt-in):
|
||||
- cleanup runs only after archive succeeded and wrote `current/run_id.txt`
|
||||
- cleanup is executed after all selected stages in the command invocation succeed (for example, a later `notify` failure leaves local files intact)
|
||||
- `pipeline.spool.delete_audio_after_archive: true` removes only `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- `pipeline.workspace.cleanup_after_archive: true` removes only `{workspace.root}/work/{campaign}/{session_id}/{run_id}/`
|
||||
- cleanup does not run when archive is skipped/disabled/fails or when run upload is disabled
|
||||
- local development `audio_dir`/`audio_files` inputs are never removed by spool cleanup
|
||||
|
||||
`pipeline.scriptorium` is optional. Existing pipelines without Scriptorium continue to work.
|
||||
|
||||
`pipeline.trim` is optional. Existing pipelines without trim config continue to work.
|
||||
|
||||
`pipeline.normalize` is optional. Existing pipelines without normalize config continue to work.
|
||||
|
||||
`pipeline.audita` drives the real Audita subprocess adapter for the `polish` stage.
|
||||
|
||||
Audita defaulted fields:
|
||||
|
||||
- `binary` defaults to `audita`
|
||||
- `timeout` defaults to `3h`
|
||||
- `report` defaults to `true`
|
||||
|
||||
Audita optional fields:
|
||||
|
||||
- `llm_api_key_env` (enforced only when configured)
|
||||
- `modules` override list (when omitted/empty, Narratio does not pass `--modules`)
|
||||
- `base_url` (when omitted, Narratio does not pass `--base-url`)
|
||||
- `model` (when omitted, Narratio does not pass `--model`)
|
||||
- `transcript_description`
|
||||
- `config_path`
|
||||
- `output_schema` (`bare-segments` or `audita-v1`)
|
||||
- `work_dir_retention` (`always`, `auto`, `never`)
|
||||
- `total_llm_concurrency` (> 0 when provided)
|
||||
- `proposal_llm_concurrency` (> 0 when provided)
|
||||
- `validation_model`
|
||||
- `validation_llm_concurrency` (> 0 when provided)
|
||||
- `report` override
|
||||
|
||||
Narratio passes only configured optional Audita flags; omitted optional values defer to Audita runtime defaults/config.
|
||||
|
||||
Seriatim defaults:
|
||||
|
||||
- `pipeline.seriatim` may be omitted
|
||||
- `binary` defaults to `seriatim`
|
||||
- `timeout` defaults to `10m`
|
||||
- `output_schema` defaults to `seriatim-intermediate`
|
||||
- `coalesce_gap` defaults to `3.0`
|
||||
- `report` defaults to `true`
|
||||
|
||||
When `pipeline.normalize` is omitted, defaults are applied:
|
||||
|
||||
- `output_path: transcripts/normalized.json`
|
||||
- `output_schema: seriatim-intermediate`
|
||||
- `report: true`
|
||||
|
||||
When `pipeline.normalize` is present:
|
||||
|
||||
- `output_path` must be non-empty
|
||||
- `output_schema` must be one of `seriatim-minimal`, `seriatim-intermediate`, or `seriatim-full`
|
||||
- relative `output_path` values are session-workdir-relative paths
|
||||
- Seriatim binary settings still come from `pipeline.seriatim`
|
||||
|
||||
When `pipeline.trim` is present:
|
||||
|
||||
- `enabled` is optional and defaults to `false` when omitted
|
||||
- relative `output_path`, `bounds.output_path`, and `bounds.render_output_path` values are session-workdir-relative paths
|
||||
- do not store secrets in trim config values
|
||||
|
||||
When `pipeline.trim.enabled: true`:
|
||||
|
||||
- `output_path` is required and non-empty
|
||||
- `bounds.prompt_id` is required and non-empty
|
||||
- `bounds.transcript_input_name` is required and non-empty
|
||||
- `bounds.output_path` is required and non-empty
|
||||
- `bounds.timeout` must parse as a Go duration when provided
|
||||
- `bounds.render_debug: true` requires non-empty `bounds.render_output_path`
|
||||
- `bounds.profile_id` may be empty to use the prompt default profile
|
||||
- prompt IDs are config values, not hardcoded stage logic
|
||||
|
||||
When `pipeline.scriptorium` is present:
|
||||
|
||||
- `binary` defaults to `scriptorium` when omitted
|
||||
- `config_path` is optional; when provided it must be non-empty
|
||||
- `timeout` is optional; when provided it must parse as a Go duration
|
||||
- default `timeout` is `10m`
|
||||
- unknown YAML fields fail strict decode
|
||||
|
||||
Artifacts are configured as a map under `pipeline.scriptorium.artifacts` so multiple artifacts are possible in the config shape.
|
||||
|
||||
For each artifact definition:
|
||||
|
||||
- `enabled: true` requires non-empty `prompt_id`
|
||||
- `enabled: true` requires non-empty `output_path`
|
||||
- `timeout` must parse as Go duration when present
|
||||
- optional per-artifact `render_debug` may override global `scriptorium.render_debug`
|
||||
- `inputs` are named and each input requires non-empty `source`
|
||||
- inputs may be optional (`required: false`)
|
||||
- `vars` values currently support `string` and `bool`
|
||||
|
||||
Prompt IDs and profile IDs are configuration values, not hardcoded stage logic.
|
||||
|
||||
Trim config shape:
|
||||
|
||||
```yaml
|
||||
trim:
|
||||
enabled: true
|
||||
output_path: "transcripts/trimmed.json"
|
||||
bounds:
|
||||
prompt_id: "dnd_session.bounds"
|
||||
profile_id: ""
|
||||
transcript_input_name: "transcript"
|
||||
output_path: "artifacts/session_bounds.json"
|
||||
timeout: "10m"
|
||||
render_debug: false
|
||||
render_output_path: "artifacts/session_bounds.render.json"
|
||||
seriatim:
|
||||
report: false
|
||||
```
|
||||
|
||||
## 6. Transcript Tiers
|
||||
|
||||
Narratio currently produces and uses four transcript tiers:
|
||||
|
||||
- `transcripts/merged.json`: canonical deterministic merged transcript from Seriatim merge
|
||||
- `transcripts/processed.json`: full raw Audita-polished transcript output (includes pre/post-game content)
|
||||
- `transcripts/normalized.json`: normalized transcript generated by Seriatim normalize
|
||||
- `transcripts/trimmed.json`: gameplay-only normalized polished transcript from trim stage
|
||||
|
||||
Trim reads `transcripts/normalized.json`, validates bounds IDs against that same transcript ID space, and writes `transcripts/trimmed.json`.
|
||||
|
||||
## 7. Normalize Stage (Current Implementation)
|
||||
|
||||
Normalize stage behavior:
|
||||
|
||||
- stage order position: after `polish` and before `trim`
|
||||
- discovers processed transcript from manifest polish outputs (`transcript_processed`) when present, else `work/<session_id>/transcripts/processed.json`
|
||||
- validates processed transcript JSON shape (`segments` array required)
|
||||
- runs Seriatim `normalize` to produce normalized transcript
|
||||
- validates normalized transcript JSON shape (`segments` array required)
|
||||
- validates normalize report JSON when enabled
|
||||
|
||||
Expected normalize outputs and diagnostics:
|
||||
|
||||
- `transcripts/normalized.json`
|
||||
- `artifacts/seriatim.normalize.report.json` (when normalize report is enabled)
|
||||
- `logs/seriatim.normalize.stdout.log`
|
||||
- `logs/seriatim.normalize.stderr.log`
|
||||
- `config/seriatim.normalize.generated.yml`
|
||||
|
||||
## 8. Trim Stage (Current Implementation)
|
||||
|
||||
Trim stage behavior:
|
||||
|
||||
- stage order position: after `normalize` and before `analyze`
|
||||
- discovers normalized transcript from manifest normalize outputs (`transcript_normalized`) when present, else `work/<session_id>/transcripts/normalized.json`
|
||||
- validates normalized transcript JSON shape (`segments` array required)
|
||||
- when `trim.enabled: false` (or trim config omitted), deterministically copies normalized transcript to `transcripts/trimmed.json` and records `trim_action=copy_disabled`
|
||||
- when `trim.enabled: true`:
|
||||
- runs Scriptorium bounds prompt using configured `trim.bounds.prompt_id`
|
||||
- writes bounds output to configured path (typically `artifacts/session_bounds.json`)
|
||||
- parses and validates bounds output against the same normalized transcript being trimmed
|
||||
- converts bounds range to Seriatim keep selector (for example `10-868`)
|
||||
- runs Seriatim `trim` to produce `transcripts/trimmed.json`
|
||||
- supports no-trim bounds actions (`none`/`copy`) by copying normalized transcript unchanged
|
||||
- validates trimmed transcript JSON shape (`segments` array required)
|
||||
|
||||
Expected trim outputs and diagnostics:
|
||||
|
||||
- `artifacts/session_bounds.json`
|
||||
- `transcripts/trimmed.json`
|
||||
- `logs/scriptorium.bounds.stdout.log`
|
||||
- `logs/scriptorium.bounds.stderr.log`
|
||||
- `config/scriptorium.bounds.generated.yml`
|
||||
- `logs/seriatim.trim.stdout.log`
|
||||
- `logs/seriatim.trim.stderr.log`
|
||||
- `config/seriatim.trim.generated.yml`
|
||||
- optional bounds render-debug outputs when enabled:
|
||||
- `artifacts/session_bounds.render.json`
|
||||
- `logs/scriptorium.bounds.render.stdout.log`
|
||||
- `logs/scriptorium.bounds.render.stderr.log`
|
||||
- `config/scriptorium.bounds.render.generated.yml`
|
||||
|
||||
Render-debug files are diagnostics. They are recorded in stage metadata/log/config refs and are not treated as canonical stage output artifact refs.
|
||||
|
||||
## 9. Analyze Stage (Current Implementation)
|
||||
|
||||
The current real analyze implementation supports only `scriptorium.artifacts.session_recap`.
|
||||
|
||||
Behavior:
|
||||
|
||||
- if `pipeline.scriptorium` is missing, analyze returns a skipped result with metadata
|
||||
- if no Scriptorium artifacts are enabled, analyze returns a skipped result with metadata
|
||||
- if enabled artifacts exist but `session_recap` is not enabled, analyze fails clearly
|
||||
- available transcript input sources for configured artifacts: `processed_transcript`, `normalized_transcript`, `trimmed_transcript`
|
||||
- `session_recap` should use `trimmed_transcript` input (`transcripts/trimmed.json`) for in-universe recap generation
|
||||
- `trimmed_transcript` input is resolved from manifest (`trim` output kind `transcript_trimmed`) when available, otherwise fallback path `work/<session_id>/transcripts/trimmed.json`
|
||||
- `normalized_transcript` input is resolved from manifest (`normalize` output kind `transcript_normalized`) when available, otherwise fallback path `work/<session_id>/transcripts/normalized.json`
|
||||
- `processed_transcript` input is resolved from manifest (`polish` output kind `transcript_processed`) when available, otherwise fallback path `work/<session_id>/transcripts/processed.json`
|
||||
- `normalized_transcript` is the preferred full-transcript source for future table/meta-analysis artifacts
|
||||
- `processed_transcript` remains available for advanced/debug use cases
|
||||
- transcript inputs are validated as JSON with top-level `segments` array
|
||||
- configured inputs are resolved by source
|
||||
- optional `previous_recap` is omitted when unavailable
|
||||
- required `previous_recap` fails before invocation when unavailable
|
||||
- vars are built from config + session metadata
|
||||
- `render_debug` controls pre-run `scriptorium render` diagnostics
|
||||
- render failure stops stage before production run
|
||||
- render output is validated as JSON
|
||||
- production call uses Scriptorium adapter `RunArtifact`
|
||||
- successful run output must exist and be non-empty
|
||||
- missing `trimmed_transcript` input for configured `trimmed_transcript` source fails clearly with guidance to run trim stage first
|
||||
- manifest records output refs, logs, generated config paths, and non-secret provenance metadata
|
||||
|
||||
## 10. Session Recap Paths
|
||||
|
||||
Current expected paths for `session_recap`:
|
||||
|
||||
- artifact output: `artifacts/session_recap.md`
|
||||
- run stdout log: `logs/scriptorium.session_recap.stdout.log`
|
||||
- run stderr log: `logs/scriptorium.session_recap.stderr.log`
|
||||
- run generated invocation/config: `config/scriptorium.session_recap.generated.yml`
|
||||
- render output (when enabled): `artifacts/session_recap.render.json`
|
||||
- render stdout log: `logs/scriptorium.session_recap.render.stdout.log`
|
||||
- render stderr log: `logs/scriptorium.session_recap.render.stderr.log`
|
||||
- render generated invocation/config: `config/scriptorium.session_recap.render.generated.yml`
|
||||
|
||||
## 11. Security and Privacy
|
||||
|
||||
- do not store secrets in pipeline YAML, generated invocation YAML, logs, or manifest metadata
|
||||
- if API-key integration is configured, pass env var names only (never raw key values)
|
||||
- with `pipeline.secrets.env_dir`, secret file values are loaded into process env only and are not persisted in manifest metadata or generated configs
|
||||
- avoid logging transcript content or rendered prompt content by default
|
||||
- treat generated artifacts and logs as potentially sensitive session material
|
||||
|
||||
## 12. Operational Caveat (Pre-Stale-Detection)
|
||||
|
||||
Checksum-based stale detection is not implemented yet.
|
||||
|
||||
If prepared inputs or prompt/runtime configuration change (for example glossary files, prompt IDs, profile IDs, or relevant pipeline settings), rerun the appropriate prior stages to refresh downstream artifacts.
|
||||
|
||||
Examples:
|
||||
|
||||
- glossary or autocorrect changes usually require rerunning at least `merge`, `polish`, `normalize`, `trim`, and `analyze`
|
||||
- trim prompt/profile changes require rerunning at least `normalize`, `trim`, and `analyze`
|
||||
- session recap prompt/profile/input-source changes require rerunning `analyze`
|
||||
|
||||
## 13. Roadmap
|
||||
|
||||
Planned next steps:
|
||||
|
||||
- extend analyze beyond `session_recap` to additional configured artifacts
|
||||
- support artifact inputs that consume prior generated artifacts
|
||||
- keep this composable without adding a generic DAG engine in the near term
|
||||
- implement real `archive` backend behavior
|
||||
- implement real `notify` backend behavior
|
||||
- add checksum-based stale detection and stale transitions
|
||||
|
||||
Architectural invariants remain:
|
||||
|
||||
- strict config decoding/validation
|
||||
- manifest-driven run control
|
||||
- clear stage/adapter separation
|
||||
- configuration-driven prompt/profile/input/vars/output mapping
|
||||
- Scriptorium integration through public CLI subprocess contract
|
||||
202
docs/architecture.md
Normal file
202
docs/architecture.md
Normal file
@@ -0,0 +1,202 @@
|
||||
# Narratio Architecture
|
||||
|
||||
## Purpose
|
||||
|
||||
`narratio` is a Go orchestration application for processing D&D session audio into polished transcripts and generated session artifacts.
|
||||
|
||||
This document defines the development principles for the project. It is inward-facing: its audience is developers and LLM coding agents. It should guide future changes, not serve as a complete implementation reference.
|
||||
|
||||
Implemented component details belong under `docs/internal/`.
|
||||
|
||||
## Project Shape
|
||||
|
||||
Narratio is a modular, stage-driven orchestrator.
|
||||
|
||||
It coordinates specialized downstream systems rather than reimplementing their domains:
|
||||
|
||||
- WhisperX handles transcription.
|
||||
- Seriatim handles deterministic transcript merge/normalization/trim behavior.
|
||||
- Audita handles transcript correction and polishing.
|
||||
- Scriptorium handles prompt execution and generated artifacts.
|
||||
|
||||
Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and archive semantics.
|
||||
|
||||
Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine.
|
||||
|
||||
## Core Principles
|
||||
|
||||
### Modular and composable
|
||||
|
||||
Code should be organized around clear responsibilities. Stages, adapters, config loading, manifest persistence, path construction, and storage behavior should remain separable and independently testable.
|
||||
|
||||
### Hexagonal boundaries
|
||||
|
||||
External systems should be isolated behind narrow adapters. Stage logic should depend on Narratio-level interfaces and data structures, not on external SDK types, subprocess argument construction, or transport-specific details.
|
||||
|
||||
### Standard library preference
|
||||
|
||||
Prefer the Go standard library. Add dependencies only when they provide substantial value, are necessary for an external integration, or are a widely used de facto standard.
|
||||
|
||||
Accepted examples include a YAML library for configuration and the AWS SDK for S3-compatible storage.
|
||||
|
||||
### Explicit orchestration
|
||||
|
||||
The pipeline should remain stage-driven and explicit. New behavior should be added through clear stage, adapter, config, or manifest contracts rather than implicit side effects or generic workflow abstraction.
|
||||
|
||||
## Stage Design
|
||||
|
||||
Each stage should have a clear scope of responsibility.
|
||||
|
||||
A stage should define:
|
||||
|
||||
- its purpose;
|
||||
- required input state;
|
||||
- produced output state;
|
||||
- config fields it consumes;
|
||||
- external adapters it uses;
|
||||
- manifest refs it reads or writes;
|
||||
- skip, force, and resume behavior;
|
||||
- failure behavior;
|
||||
- tests that protect its contract.
|
||||
|
||||
Stages should avoid reaching across boundaries. If shared behavior is needed, prefer a helper or service with a narrow interface over duplicating ad hoc logic between stages.
|
||||
|
||||
## Transactionality and Resume
|
||||
|
||||
A stage should behave transactionally.
|
||||
|
||||
A stage is complete only when its outputs have been written, validated, and recorded in the manifest. If a stage fails, Narratio should preserve enough local state for inspection, recovery, and resume.
|
||||
|
||||
A failed or incomplete run must not be treated as successful. Later stages should depend on manifest-recorded success, not merely on incidental files existing on disk.
|
||||
|
||||
## Manifest Model
|
||||
|
||||
The manifest is the durable local ledger for a run.
|
||||
|
||||
It should record:
|
||||
|
||||
- run identity;
|
||||
- stage status;
|
||||
- input and output refs;
|
||||
- logs and generated config refs;
|
||||
- checksums or provenance where useful;
|
||||
- non-secret adapter and archive metadata.
|
||||
|
||||
Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress.
|
||||
|
||||
## Adapter Boundaries
|
||||
|
||||
Adapters own external integration details.
|
||||
|
||||
Expected boundaries:
|
||||
|
||||
- WhisperX HTTP details stay in the WhisperX adapter.
|
||||
- Seriatim CLI construction stays in the Seriatim adapter.
|
||||
- Audita CLI construction stays in the Audita adapter.
|
||||
- Scriptorium CLI construction stays in the Scriptorium adapter.
|
||||
- Object-storage details stay behind the storage adapter interface.
|
||||
- AWS SDK types stay inside the S3 storage implementation.
|
||||
|
||||
Stage code should express intent in Narratio terms and call adapters through narrow contracts.
|
||||
|
||||
## Configuration Philosophy
|
||||
|
||||
Configuration should be strict, explicit, and operator-friendly.
|
||||
|
||||
Principles:
|
||||
|
||||
- YAML decoding should reject unknown fields.
|
||||
- Defaults should be centralized and testable.
|
||||
- Empty configured values should not silently override meaningful defaults.
|
||||
- Session templating should remain narrow and deterministic.
|
||||
- Template support should serve operator convenience, not become a general configuration language.
|
||||
|
||||
Narratio should not become a secondary configuration system for downstream tools. Seriatim, Audita, and Scriptorium should own their runtime defaults wherever practical. Narratio should pass required stage-contract paths and explicit operator overrides.
|
||||
|
||||
## Path and Storage Discipline
|
||||
|
||||
Local and remote paths are part of Narratio’s application contract.
|
||||
|
||||
Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and archive paths. Stages should avoid reconstructing canonical paths through scattered string concatenation.
|
||||
|
||||
Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics.
|
||||
|
||||
## Archive Invariants
|
||||
|
||||
Archive behavior must preserve a clear commit boundary.
|
||||
|
||||
A remote run is current only after the archive stage has successfully uploaded the run record, required promoted outputs, `current/manifest.json`, and finally `current/run_id.txt`.
|
||||
|
||||
`current/run_id.txt` is the final remote commit marker and must be written last.
|
||||
|
||||
Failed, incomplete, skipped, or uncommitted archive attempts must not be presented as current remote state. Local cleanup is permitted only after successful archive commit and only when explicitly configured.
|
||||
|
||||
## Security and Privacy
|
||||
|
||||
Narratio handles private campaign material.
|
||||
|
||||
Rules:
|
||||
|
||||
- Do not store raw secrets in pipeline or session YAML.
|
||||
- Use environment variable names or secret-file references for secret handling.
|
||||
- Do not write raw secret values to manifests, logs, generated configs, or archive metadata.
|
||||
- Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive.
|
||||
- Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
Diagnostics should be durable and discoverable, but distinct from canonical outputs.
|
||||
|
||||
Logs, reports, generated invocation/config files, and render-debug files support debugging. Transcript tiers and configured artifacts are pipeline products.
|
||||
|
||||
Manifest refs should preserve that distinction.
|
||||
|
||||
## Determinism
|
||||
|
||||
Where practical, Narratio should prefer deterministic behavior:
|
||||
|
||||
- stable local path layout;
|
||||
- stable remote key layout;
|
||||
- sorted upload order;
|
||||
- predictable generated config files;
|
||||
- repeatable command construction;
|
||||
- tests that do not depend on live external services.
|
||||
|
||||
Run IDs and timestamps may be intentionally variable, but surrounding behavior should remain testable.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
Core behavior should be testable without live external services.
|
||||
|
||||
Tests should cover:
|
||||
|
||||
- config loading, defaults, and validation;
|
||||
- CLI parsing and command construction;
|
||||
- path helpers;
|
||||
- manifest transitions;
|
||||
- stage success, failure, skip, and resume behavior;
|
||||
- adapter command construction;
|
||||
- fake storage behavior;
|
||||
- archive commit ordering;
|
||||
- example config validity where practical.
|
||||
|
||||
Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs.
|
||||
|
||||
## Documentation Expectations
|
||||
|
||||
Documentation must follow `docs/documentation/policy.md`.
|
||||
|
||||
Current behavior belongs in user-facing docs and `docs/internal/`. Future, planned, aspirational, experimental, or unimplemented work belongs only under `docs/roadmap/`.
|
||||
|
||||
`docs/architecture.md` should remain concise and principle-focused. It should not duplicate the full config reference, CLI reference, operations guide, or internal stage documentation.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
Narratio is not:
|
||||
|
||||
- a generic DAG or workflow engine;
|
||||
- a replacement configuration layer for Seriatim, Audita, or Scriptorium;
|
||||
- a storage backend abstraction beyond the needs of this pipeline;
|
||||
- a place to embed raw secrets;
|
||||
- a place for stage logic to depend directly on AWS SDK types or downstream tool internals;
|
||||
- a prompt-authoring system.
|
||||
222
docs/cli.md
Normal file
222
docs/cli.md
Normal file
@@ -0,0 +1,222 @@
|
||||
# CLI
|
||||
|
||||
## Shortest Useful Command
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
This command uses default config discovery for `pipeline.yml` and `session.yml`; both files must be discoverable unless you pass explicit `--config` and `--session` paths.
|
||||
|
||||
## Command Overview
|
||||
|
||||
Implemented commands:
|
||||
|
||||
- `run`: execute pipeline stages and persist manifest state.
|
||||
- `plan`: validate config, prepare workspace layout, and print stage run/skip decisions.
|
||||
- `resume`: continue from first non-succeeded stage unless forced.
|
||||
- `status`: read and print stage statuses from an existing manifest.
|
||||
- `run-stage`: execute exactly one stage.
|
||||
|
||||
Unknown commands print usage and exit non-zero.
|
||||
|
||||
For config semantics, see [docs/config.md](./config.md). For operator lifecycle and recovery, see [docs/operations.md](./operations.md).
|
||||
|
||||
## Complete Flag Reference
|
||||
|
||||
### `run`
|
||||
|
||||
- `--config <path>`: optional explicit `pipeline.yml` path.
|
||||
- `--session <path>`: optional explicit `session.yml` path.
|
||||
- `--session-id <value>`: session template variable value.
|
||||
- `--force`: force stage execution.
|
||||
- `--artifacts <names>`: analyze artifact keys to execute (repeatable or comma-separated).
|
||||
|
||||
### `plan`
|
||||
|
||||
- `--config <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--force`
|
||||
|
||||
### `resume`
|
||||
|
||||
- `--config <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--force`
|
||||
- `--artifacts <names>`: analyze artifact keys to execute (repeatable or comma-separated).
|
||||
|
||||
### `run-stage`
|
||||
|
||||
- `--config <path>`
|
||||
- `--session <path>`
|
||||
- `--session-id <value>`
|
||||
- `--force`
|
||||
- `--artifacts <names>`: analyze artifact keys to execute (repeatable or comma-separated).
|
||||
- positional `<stage>`: required stage name.
|
||||
|
||||
Valid stage names:
|
||||
|
||||
- `prepare`
|
||||
- `transcribe`
|
||||
- `merge`
|
||||
- `polish`
|
||||
- `normalize`
|
||||
- `trim`
|
||||
- `analyze`
|
||||
- `archive`
|
||||
- `notify`
|
||||
|
||||
### `status`
|
||||
|
||||
- `--manifest <path>`: required manifest path.
|
||||
|
||||
## Command Reference
|
||||
|
||||
### `run`
|
||||
|
||||
Purpose:
|
||||
- Execute configured stages in canonical order.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio run [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force] [--artifacts <name[,name...]>]
|
||||
```
|
||||
|
||||
Success output:
|
||||
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
||||
|
||||
Common failure cases:
|
||||
- missing default config/session paths when flags omitted.
|
||||
- invalid template/rendered session mismatch.
|
||||
- unknown/invalid `--artifacts` value.
|
||||
- `--artifacts` with unknown configured artifact key.
|
||||
|
||||
### `plan`
|
||||
|
||||
Purpose:
|
||||
- Validate config, load secrets (if configured), prepare workdir, and print stage run/skip decisions.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio plan [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force]
|
||||
```
|
||||
|
||||
Success output includes:
|
||||
- `narratio plan: workdir prepared at <path>`
|
||||
- one line per stage (`<stage>: run|skip`)
|
||||
- `totals: run=<n> skip=<n>`
|
||||
|
||||
Common failure cases:
|
||||
- same config/session discovery and validation failures as `run`.
|
||||
- secrets directory read failures when `pipeline.secrets.env_dir` is configured.
|
||||
|
||||
### `resume`
|
||||
|
||||
Purpose:
|
||||
- Continue from session-manifest stage status.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio resume [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force] [--artifacts <name[,name...]>]
|
||||
```
|
||||
|
||||
Success output:
|
||||
- `narratio resume: session <session_id> has no remaining stages`
|
||||
- or `narratio resume: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
||||
|
||||
Common failure cases:
|
||||
- same discovery/template/validation failures as `run`.
|
||||
- manifest load errors when existing manifest is unreadable.
|
||||
- invalid or unknown artifact selections.
|
||||
|
||||
### `status`
|
||||
|
||||
Purpose:
|
||||
- Inspect one manifest file without executing stages.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio status --manifest <manifest.json>
|
||||
```
|
||||
|
||||
Success output includes:
|
||||
- `session_id: <id>`
|
||||
- `updated_at: <timestamp>`
|
||||
- `stages:` entries (`- <stage>: <status>`)
|
||||
|
||||
Common failure cases:
|
||||
- missing `--manifest`.
|
||||
- unreadable or invalid manifest path.
|
||||
|
||||
### `run-stage`
|
||||
|
||||
Purpose:
|
||||
- Execute exactly one stage.
|
||||
|
||||
Syntax:
|
||||
|
||||
```bash
|
||||
narratio run-stage [--config <pipeline.yml>] [--session <session.yml>] [--session-id <id>] [--force] [--artifacts <name[,name...]>] <stage>
|
||||
```
|
||||
|
||||
Success output:
|
||||
- `narratio run-stage: stage=<name> executed=<n> skipped=<n> force=<true|false>; manifest=<path>`
|
||||
|
||||
`--artifacts` behavior:
|
||||
- accepted only when `<stage>` is `analyze`.
|
||||
- names are normalized (trimmed, deduplicated, sorted).
|
||||
- unknown configured artifact keys fail.
|
||||
|
||||
Common failure cases:
|
||||
- missing stage positional arg.
|
||||
- unknown stage name.
|
||||
- using `--artifacts` with any non-`analyze` stage.
|
||||
|
||||
## Common Workflows
|
||||
|
||||
Default-discovery run:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Run only selected analyze artifacts:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04 --artifacts session_recap,player_handout
|
||||
```
|
||||
|
||||
Resume with selected analyze artifacts:
|
||||
|
||||
```bash
|
||||
narratio resume --session-id 2026-04-04 --artifacts player_handout
|
||||
```
|
||||
|
||||
Run only analyze stage with selected artifacts:
|
||||
|
||||
```bash
|
||||
narratio run-stage --session-id 2026-04-04 --artifacts player_handout analyze
|
||||
```
|
||||
|
||||
## Diagnostic / Recovery Commands
|
||||
|
||||
Inspect stage status:
|
||||
|
||||
```bash
|
||||
narratio status --manifest <manifest.json>
|
||||
```
|
||||
|
||||
Get manifest path from previous output:
|
||||
- `run`, `resume`, and `run-stage` print `manifest=<path>` on success.
|
||||
|
||||
## `--artifacts` and `--force`
|
||||
|
||||
- `--artifacts` filters which configured artifacts are executable when analyze runs.
|
||||
- `--artifacts` does not imply `--force`.
|
||||
- If analyze is already `succeeded` and `--force` is not set, runner-level skip still applies.
|
||||
304
docs/config.md
Normal file
304
docs/config.md
Normal file
@@ -0,0 +1,304 @@
|
||||
# Configuration
|
||||
|
||||
## 1. Overview
|
||||
|
||||
Narratio loads two YAML files:
|
||||
|
||||
- `pipeline.yml`: pipeline-level runtime configuration.
|
||||
- `session.yml`: per-session metadata and input selection.
|
||||
|
||||
These commands load and validate both files before running:
|
||||
|
||||
- `narratio run`
|
||||
- `narratio plan`
|
||||
- `narratio resume`
|
||||
- `narratio run-stage`
|
||||
|
||||
Behavior:
|
||||
|
||||
- strict YAML decode is enabled (`KnownFields(true)`): unknown fields fail.
|
||||
- session templates render before session YAML decode.
|
||||
- defaults are applied for optional pipeline fields.
|
||||
- validation enforces required fields, value formats, and cross-field constraints.
|
||||
|
||||
## 2. Config file discovery
|
||||
|
||||
Pipeline config lookup for `run`, `plan`, `resume`, and `run-stage`:
|
||||
|
||||
- If `--config <path>` is provided, that path is used.
|
||||
- If omitted, Narratio searches in order:
|
||||
1. `/usr/local/etc/narratio/pipeline.yml`
|
||||
2. `/etc/narratio/pipeline.yml`
|
||||
- First existing file wins.
|
||||
|
||||
## 3. Session file discovery and templating
|
||||
|
||||
Session config lookup for `run`, `plan`, `resume`, and `run-stage`:
|
||||
|
||||
- If `--session <path>` is provided, that path is used.
|
||||
- If omitted, Narratio searches in order:
|
||||
1. `./session.yml`
|
||||
2. `/usr/local/etc/narratio/session.yml`
|
||||
3. `/etc/narratio/session.yml`
|
||||
- First existing file wins.
|
||||
|
||||
Template behavior:
|
||||
|
||||
- Supported placeholders:
|
||||
- `{{session_id}}`
|
||||
- `{{ session_id }}`
|
||||
- `--session-id <value>` supplies the placeholder value.
|
||||
- unresolved placeholders fail load.
|
||||
- if rendered `session_id` mismatches `--session-id`, load fails.
|
||||
|
||||
## 4. Minimal pipeline config
|
||||
|
||||
```yaml
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
```
|
||||
|
||||
Why this is sufficient:
|
||||
|
||||
- `whisperx.transcribe_url` is required.
|
||||
- `workspace.root` defaults to `/var/lib/narratio`.
|
||||
- optional sections (`seriatim`, `audita`, `archive`, `scriptorium`, `trim`, `normalize`, etc.) receive defaults or stay inactive.
|
||||
|
||||
## 5. Minimal session template
|
||||
|
||||
```yaml
|
||||
session_id: "{{ session_id }}"
|
||||
campaign: sample-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./examples/speakers.yml
|
||||
autocorrect_file: ./examples/autocorrect.yml
|
||||
glossary_file: ./examples/glossary.yml
|
||||
```
|
||||
|
||||
Usage:
|
||||
|
||||
```bash
|
||||
narratio run --config /path/to/pipeline.yml --session ./session.yml --session-id 2026-05-03
|
||||
```
|
||||
|
||||
## 6. Production-oriented config
|
||||
|
||||
```yaml
|
||||
workspace:
|
||||
root: /var/lib/narratio/workspace
|
||||
cleanup_after_archive: true
|
||||
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: my-dnd-archive
|
||||
root_prefix: dnd
|
||||
region: us-east-1
|
||||
access_key_id_env: OBJECT_STORAGE_KEY_ID
|
||||
secret_access_key_env: OBJECT_STORAGE_KEY
|
||||
|
||||
spool:
|
||||
root: /var/spool/narratio
|
||||
delete_audio_after_archive: true
|
||||
|
||||
archive:
|
||||
enabled: true
|
||||
upload_run: true
|
||||
promote_artifacts:
|
||||
- from: transcripts/trimmed.json
|
||||
to: transcripts/trimmed.json
|
||||
required: true
|
||||
- from: artifacts/session_recap.md
|
||||
to: artifacts/session_recap.md
|
||||
required: true
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
|
||||
scriptorium:
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
```
|
||||
|
||||
Operational notes:
|
||||
|
||||
- archive promotion is explicit and path-based via `archive.promote_artifacts`.
|
||||
- Narratio does not auto-promote all generated analyze artifacts.
|
||||
|
||||
## 7. Full pipeline reference
|
||||
|
||||
| Path | Type | Required | Default |
|
||||
| --- | --- | --- | --- |
|
||||
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
|
||||
| `pipeline.workspace.cleanup_after_archive` | bool | No | `false` |
|
||||
| `pipeline.secrets.env_dir` | string | Conditional | none |
|
||||
| `pipeline.storage.backend` | string | No | empty |
|
||||
| `pipeline.storage.bucket` | string | No | empty |
|
||||
| `pipeline.storage.prefix` | string | No | empty |
|
||||
| `pipeline.storage.s3.bucket` | string | Conditional | empty |
|
||||
| `pipeline.storage.s3.root_prefix` | string | No | `dnd` |
|
||||
| `pipeline.storage.s3.region` | string | No | empty |
|
||||
| `pipeline.storage.s3.endpoint` | string | No | empty |
|
||||
| `pipeline.storage.s3.force_path_style` | bool | No | `false` |
|
||||
| `pipeline.storage.s3.access_key_id_env` | string | No | `OBJECT_STORAGE_KEY_ID` |
|
||||
| `pipeline.storage.s3.secret_access_key_env` | string | No | `OBJECT_STORAGE_KEY` |
|
||||
| `pipeline.spool.root` | string | No | `/var/spool/narratio` |
|
||||
| `pipeline.spool.delete_audio_after_archive` | bool | No | `false` |
|
||||
| `pipeline.archive.enabled` | bool | No | `true` |
|
||||
| `pipeline.archive.upload_run` | bool | No | `true` |
|
||||
| `pipeline.archive.promote_artifacts[]` | list | No | trimmed + session_recap rules |
|
||||
| `pipeline.archive.promote_artifacts[].from` | string | Yes (per rule) | none |
|
||||
| `pipeline.archive.promote_artifacts[].to` | string | Yes (per rule) | none |
|
||||
| `pipeline.archive.promote_artifacts[].required` | bool | No | `true` |
|
||||
| `pipeline.whisperx.transcribe_url` | string | Yes | none |
|
||||
| `pipeline.whisperx.language` | string | No | `en` |
|
||||
| `pipeline.whisperx.timeout` | duration string | No | `30m` |
|
||||
| `pipeline.whisperx.retries` | int | No | `3` |
|
||||
| `pipeline.whisperx.retry_delay` | duration string | No | `2s` |
|
||||
| `pipeline.whisperx.concurrency` | int | No | `2` |
|
||||
| `pipeline.seriatim.binary` | string | No | `seriatim` |
|
||||
| `pipeline.seriatim.timeout` | duration string | No | `10m` |
|
||||
| `pipeline.seriatim.output_schema` | string | No | `seriatim-intermediate` |
|
||||
| `pipeline.seriatim.coalesce_gap` | float | No | `3.0` |
|
||||
| `pipeline.seriatim.report` | bool | No | `true` |
|
||||
| `pipeline.seriatim.env.overlap_word_run_gap` | float | No | unset |
|
||||
| `pipeline.seriatim.env.overlap_word_run_reorder_window` | float | No | unset |
|
||||
| `pipeline.seriatim.env.backchannel_max_duration` | float | No | unset |
|
||||
| `pipeline.seriatim.env.filler_max_duration` | float | No | unset |
|
||||
| `pipeline.audita.binary` | string | No | `audita` |
|
||||
| `pipeline.audita.timeout` | duration string | No | `3h` |
|
||||
| `pipeline.audita.llm_api_key_env` | string | No | empty |
|
||||
| `pipeline.audita.modules[]` | list[string] | No | empty |
|
||||
| `pipeline.audita.base_url` | string | No | empty |
|
||||
| `pipeline.audita.model` | string | No | empty |
|
||||
| `pipeline.audita.total_llm_concurrency` | int | No | unset |
|
||||
| `pipeline.audita.proposal_llm_concurrency` | int | No | unset |
|
||||
| `pipeline.audita.validation_model` | string | No | empty |
|
||||
| `pipeline.audita.validation_llm_concurrency` | int | No | unset |
|
||||
| `pipeline.audita.transcript_description` | string | No | empty |
|
||||
| `pipeline.audita.config_path` | string | No | empty |
|
||||
| `pipeline.audita.output_schema` | string | No | empty |
|
||||
| `pipeline.audita.work_dir_retention` | string | No | empty |
|
||||
| `pipeline.audita.report` | bool | No | `true` |
|
||||
| `pipeline.normalize.output_path` | string | No | `transcripts/normalized.json` |
|
||||
| `pipeline.normalize.output_schema` | string | No | `seriatim-intermediate` |
|
||||
| `pipeline.normalize.report` | bool | No | `true` |
|
||||
| `pipeline.trim.enabled` | bool | No | `false` |
|
||||
| `pipeline.trim.output_path` | string | Conditional | none |
|
||||
| `pipeline.trim.bounds.prompt_id` | string | Conditional | none |
|
||||
| `pipeline.trim.bounds.profile_id` | string | No | empty |
|
||||
| `pipeline.trim.bounds.transcript_input_name` | string | Conditional | none |
|
||||
| `pipeline.trim.bounds.output_path` | string | Conditional | none |
|
||||
| `pipeline.trim.bounds.timeout` | duration string | No | `10m` |
|
||||
| `pipeline.trim.bounds.render_debug` | bool | No | `false` |
|
||||
| `pipeline.trim.bounds.render_output_path` | string | Conditional | none |
|
||||
| `pipeline.trim.seriatim.report` | bool | No | `false` |
|
||||
| `pipeline.scriptorium.binary` | string | No | `scriptorium` |
|
||||
| `pipeline.scriptorium.config_path` | string | No | empty |
|
||||
| `pipeline.scriptorium.timeout` | duration string | No | `10m` |
|
||||
| `pipeline.scriptorium.render_debug` | bool | No | `false` |
|
||||
| `pipeline.scriptorium.artifacts` | map | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.enabled` | bool | No | `false` |
|
||||
| `pipeline.scriptorium.artifacts.<name>.depends_on[]` | list[string] | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.render_debug` | bool | No | unset |
|
||||
| `pipeline.scriptorium.artifacts.<name>.prompt_id` | string | Conditional | none |
|
||||
| `pipeline.scriptorium.artifacts.<name>.profile_id` | string | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.output_path` | string | Conditional | none |
|
||||
| `pipeline.scriptorium.artifacts.<name>.timeout` | duration string | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` | string | Conditional | none |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.artifact` | string | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.path` | string | No | empty |
|
||||
| `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required` | bool | No | `false` |
|
||||
| `pipeline.scriptorium.artifacts.<name>.vars.<key>` | map value | No | empty |
|
||||
| `pipeline.analyzer.binary_path` | string | No | empty |
|
||||
| `pipeline.analyzer.timeout` | duration string | No | empty |
|
||||
| `pipeline.analyzer.artifacts.output_dir` | string | No | empty |
|
||||
| `pipeline.analyzer.artifacts.types[]` | list[string] | No | empty |
|
||||
| `pipeline.notification.backend` | string | No | empty |
|
||||
| `pipeline.notification.recipient` | string | No | empty |
|
||||
| `pipeline.notification.timeout` | duration string | No | empty |
|
||||
|
||||
Scriptorium artifact-key and dependency rules:
|
||||
|
||||
- artifact keys must match `^[a-z][a-z0-9_]*$`.
|
||||
- enabled artifacts require `prompt_id` and `output_path`.
|
||||
- `output_path` must be relative, traversal-safe, and under `artifacts/`.
|
||||
- configured artifact input sources use `narratio.artifact.<name>`.
|
||||
- if input source references `narratio.artifact.<name>`, artifact `<name>` must exist and must be listed in `depends_on`.
|
||||
- every `depends_on` entry must be a configured artifact key.
|
||||
- self-dependency is rejected.
|
||||
- enabled dependency cycles are rejected.
|
||||
- any artifact referenced by `depends_on` or `narratio.artifact.<name>` source must define `output_path` (even if not enabled).
|
||||
|
||||
Allowed `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source` values:
|
||||
|
||||
- `previous_session_artifact`
|
||||
- `narratio.transcript.merged`
|
||||
- `narratio.transcript.polished`
|
||||
- `narratio.transcript.full`
|
||||
- `narratio.transcript.trimmed`
|
||||
- `narratio.bounds.session`
|
||||
- `narratio.artifact.<configured_artifact_key>`
|
||||
|
||||
## 8. Full session reference
|
||||
|
||||
| Path | Type | Required | Default |
|
||||
| --- | --- | --- | --- |
|
||||
| `session.session_id` | string | Yes | none |
|
||||
| `session.campaign` | string | Yes | none |
|
||||
| `session.date` | string | No | empty |
|
||||
| `session.title` | string | No | empty |
|
||||
| `session.inputs.audio_dir` | string | Conditional | empty |
|
||||
| `session.inputs.audio_files[]` | list[string] | Conditional | empty |
|
||||
| `session.inputs.audio_s3.prefix` | string | Conditional | none |
|
||||
| `session.inputs.speakers_file` | string | Yes | none |
|
||||
| `session.inputs.autocorrect_file` | string | Yes | none |
|
||||
| `session.inputs.glossary_file` | string | Yes | none |
|
||||
|
||||
Audio-source rule:
|
||||
|
||||
- configure exactly one mode:
|
||||
- `audio_dir`, or
|
||||
- `audio_files` (at least one), or
|
||||
- `audio_s3.prefix`
|
||||
- `audio_s3` cannot be combined with local audio fields.
|
||||
|
||||
## 9. Secrets
|
||||
|
||||
Narratio supports filesystem-based secret injection via `pipeline.secrets.env_dir`.
|
||||
|
||||
Behavior:
|
||||
|
||||
- `env_dir` may be absolute or relative.
|
||||
- relative `env_dir` resolves from current working directory.
|
||||
- files with valid env-var names (`[A-Za-z_][A-Za-z0-9_]*`) are loaded.
|
||||
- values are loaded from file contents with trailing newline trimming.
|
||||
- existing process env vars are preserved.
|
||||
- invalid names and subdirectories are skipped.
|
||||
- missing/unreadable `env_dir` fails command execution.
|
||||
|
||||
Guidance:
|
||||
|
||||
- do not put secret values directly in YAML.
|
||||
- configure env var names in config and provide values via env/secrets files.
|
||||
|
||||
## 10. Examples
|
||||
|
||||
Maintained examples:
|
||||
|
||||
- `examples/pipeline.minimal.yml`
|
||||
- `examples/pipeline.production.yml`
|
||||
- `examples/pipeline.full.annotated.yml`
|
||||
- `examples/session.template.yml`
|
||||
- `examples/session.local-audio.yml`
|
||||
- `examples/session.s3-audio.yml`
|
||||
|
||||
These examples are validated by `internal/config` tests.
|
||||
92
docs/development.md
Normal file
92
docs/development.md
Normal file
@@ -0,0 +1,92 @@
|
||||
# Development Guide
|
||||
|
||||
## Purpose
|
||||
Canonical contributor workflow and engineering conventions for implemented Narratio behavior.
|
||||
|
||||
## Repository layout
|
||||
|
||||
- `cmd/narratio/`: CLI entrypoint.
|
||||
- `internal/app/`: command handlers, plan/run/resume orchestration, cleanup gates, secrets loading.
|
||||
- `internal/config/`: strict YAML loading, defaults, and validation.
|
||||
- `internal/stage/`: stage implementations and stage registry/order.
|
||||
- `internal/adapters/`: external boundary adapters (WhisperX, Seriatim, Audita, Scriptorium, storage, notify).
|
||||
- `internal/manifest/`: session/run manifest types and persistence.
|
||||
- `internal/artifacts/`: canonical local/remote path helpers and local artifact store.
|
||||
- `docs/`: canonical documentation set.
|
||||
- `examples/`: maintained config examples used by tests.
|
||||
|
||||
## Build and test commands
|
||||
|
||||
- Run focused CLI behavior checks:
|
||||
|
||||
```bash
|
||||
go test ./internal/app -run TestExecute -v
|
||||
```
|
||||
|
||||
- Run config example load/validate checks:
|
||||
|
||||
```bash
|
||||
go test ./internal/config -run TestExamplesLoadAndValidate -v
|
||||
```
|
||||
|
||||
- Run full test suite:
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## Coding conventions
|
||||
|
||||
- Keep orchestration explicit and stage-driven; do not introduce generic workflow/DAG abstractions.
|
||||
- Keep external-system details inside adapter packages; stages should consume Narratio-level contracts only.
|
||||
- Use centralized path helpers from `internal/artifacts` rather than ad hoc path concatenation.
|
||||
- Preserve manifest-driven state transitions (`running`, `succeeded`, `failed`, `skipped`, `stale`) as the source of run progress.
|
||||
- Keep user/operator docs implementation-accurate; planned work belongs only under `docs/roadmap/`.
|
||||
|
||||
For design principles and invariants, see [docs/architecture.md](./architecture.md). For stage/adapter contracts, see [docs/internal/README.md](./internal/README.md).
|
||||
|
||||
## Dependency policy
|
||||
|
||||
- Prefer Go standard library where practical.
|
||||
- Add third-party dependencies only when they provide clear value for required behavior.
|
||||
- Keep dependency additions narrow to the boundary package that needs them.
|
||||
|
||||
## Change playbooks
|
||||
|
||||
### Add config fields
|
||||
|
||||
1. Add fields to config structs in `internal/config`.
|
||||
2. Set defaults in `internal/config/defaults.go` when appropriate.
|
||||
3. Add validation rules in `internal/config/validate.go`.
|
||||
4. Add or update load/validate tests in `internal/config/*_test.go`.
|
||||
5. Update canonical config docs and examples:
|
||||
- [docs/config.md](./config.md)
|
||||
- relevant files under `examples/`
|
||||
|
||||
### Add CLI flags or commands
|
||||
|
||||
1. Update command parsing and behavior in `internal/app`.
|
||||
2. Add or update command tests (`TestExecute` and command-specific tests).
|
||||
3. Update [docs/cli.md](./cli.md) and, if operator workflow changes, [docs/operations.md](./operations.md).
|
||||
|
||||
### Add or modify stages/adapters
|
||||
|
||||
1. Implement stage behavior in `internal/stage` with clear input/output boundaries.
|
||||
2. Keep external transport/subprocess details in `internal/adapters`.
|
||||
3. Preserve manifest and promotion semantics expected by runner and archive logic.
|
||||
4. Add/update stage and adapter tests.
|
||||
5. Update internal component contracts in `docs/internal/`.
|
||||
|
||||
### Update examples
|
||||
|
||||
1. Keep canonical examples only in `examples/`.
|
||||
2. Ensure examples load and validate through runtime config paths.
|
||||
3. Update `internal/config/load_validate_test.go` as needed.
|
||||
4. Update links in `docs/config.md` if example filenames change.
|
||||
|
||||
### Update docs and roadmap
|
||||
|
||||
1. Keep implemented behavior in canonical docs (`README`, `docs/*.md`, `docs/internal/`).
|
||||
2. Keep planned/unimplemented behavior only in `docs/roadmap/`.
|
||||
3. After completing roadmap items, remove or mark them complete in `docs/roadmap/documentation.md`.
|
||||
4. Run a link/path sweep before finalizing changes.
|
||||
@@ -1,507 +0,0 @@
|
||||
# Workspace Architecture Implementation Plan (Audit)
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
**Complexity assessment:** **heavy**.
|
||||
|
||||
This is not a single path-helper refactor. The current codebase has a hybrid session/run model that works for current behavior, but diverges from `docs/development/workspace.md` in foundational places (workspace root shape, manifest responsibilities, stage output placement, and archive symmetry).
|
||||
|
||||
Highest-risk areas:
|
||||
|
||||
1. Splitting the current single manifest model into durable **session manifest** vs per-invocation **run manifest** without regressing skip/force/resume UX.
|
||||
2. Migrating path helpers and artifact-store interfaces from session-only roots (`work/{session}`) to campaign-aware roots (`work/{campaign}/{session}`) while preserving existing runs.
|
||||
3. Introducing run-local stage outputs + immediate promotion while keeping stage tests and archive behavior stable.
|
||||
4. Avoiding stale downstream skips after forced upstream reruns.
|
||||
|
||||
Surprising findings:
|
||||
|
||||
1. Code already has campaign/run-aware helpers (`SessionRunWorkDir`, `SessionSpoolAudioDir`) but core session helpers and manifest pathing remain campaign-unaware.
|
||||
2. Archive recently gained run/session fallback behavior for manifest/promotion sources, which confirms an existing hybrid-layout pressure point.
|
||||
3. Analyze input resolution is functional but ad hoc and stage-local; there is no centralized artifact registry/resolver.
|
||||
|
||||
---
|
||||
|
||||
## 2. Current-State Map
|
||||
|
||||
## 2.1 Workspace Path Construction
|
||||
|
||||
Primary path model:
|
||||
|
||||
- [`internal/artifacts/paths.go`](../../internal/artifacts/paths.go)
|
||||
- `SessionWorkDir(rootDir, sessionID)` => `{root}/work/{session_id}`
|
||||
- `buildSessionPaths(workspaceRoot, sessionID)` roots all canonical paths under `{root}/work/{session_id}`
|
||||
- `SessionRunWorkDir(rootDir, campaign, sessionID, runID)` exists, but is not the default session root helper.
|
||||
|
||||
Artifact store abstraction:
|
||||
|
||||
- [`internal/artifacts/store.go`](../../internal/artifacts/store.go)
|
||||
- `SessionPaths(sessionID string)` / `EnsureLayout(sessionID string)` are session-id-only (no campaign argument).
|
||||
- [`internal/artifacts/local.go`](../../internal/artifacts/local.go)
|
||||
- `EnsureLayout` creates session-level folders under `SessionWorkDir`.
|
||||
|
||||
Path normalization helper:
|
||||
|
||||
- [`internal/artifacts/resolve.go`](../../internal/artifacts/resolve.go)
|
||||
- `ResolveSessionLocalPathForRead` accepts absolute/workspace/session-relative values and probes filesystem.
|
||||
|
||||
Where campaign-aware/run-aware support exists:
|
||||
|
||||
- Local run path helpers: `SessionRunWorkDir`, `SessionSpoolAudioDir`.
|
||||
- S3 key helpers: [`internal/artifacts/s3_keys.go`](../../internal/artifacts/s3_keys.go) (`campaigns/sessions/runs/current`).
|
||||
|
||||
Where session-root assumptions remain `{workspace}/work/{session}`:
|
||||
|
||||
- Manifest path computation: [`internal/app/runner.go`](../../internal/app/runner.go) `manifestPathFor`.
|
||||
- Artifact store layout and most stage `paths := env.ArtifactStore.SessionPaths(sessionID)` calls.
|
||||
- Many tests hardcode `workspace/work/<session>/...` (examples below).
|
||||
|
||||
Manual/ad hoc path construction (not through a single resolver API):
|
||||
|
||||
- Common stage patterns: `filepath.Join(paths.TranscriptsDir, "...")`, `filepath.Join(paths.ArtifactsDir, "...")`, etc.
|
||||
- `prepare` run/work selection: [`internal/stage/prepare.go`](../../internal/stage/prepare.go) `pathsWorkDirForManifest`.
|
||||
- Analyze input fallback path resolution: `resolveInputPathForRead` in [`internal/stage/analyze.go`](../../internal/stage/analyze.go).
|
||||
|
||||
## 2.2 Manifest and Run Identity
|
||||
|
||||
Current manifest model:
|
||||
|
||||
- [`internal/manifest/manifest.go`](../../internal/manifest/manifest.go) `Manifest` includes both session and run fields:
|
||||
- `SessionID`, `Campaign`
|
||||
- `RunID`, `LocalWorkDir`, `LocalSpoolDir`
|
||||
- `S3Bucket`, `S3SessionPrefix`, `S3RunPrefix`
|
||||
- `Stages`, `Inputs`, stage outputs/logs/generated configs/metadata.
|
||||
|
||||
Current persistence:
|
||||
|
||||
- [`internal/manifest/store.go`](../../internal/manifest/store.go) `LocalStore` reads/writes one JSON manifest path.
|
||||
- Runner always loads/saves one manifest path via `manifestPathFor(cfg)` (session-root path under current layout).
|
||||
|
||||
Identity initialization:
|
||||
|
||||
- [`internal/app/runner.go`](../../internal/app/runner.go) `ensureManifestIdentity` populates run fields if absent.
|
||||
- `RunID` is generated once for an empty manifest and reused thereafter (hybrid semantics).
|
||||
|
||||
Interpretation today:
|
||||
|
||||
- Best described as a **hybrid session manifest** with run identity fields, not as distinct session + run manifests.
|
||||
|
||||
What this means for redesign:
|
||||
|
||||
- Session-vs-run split is not just file relocation; it requires new responsibilities and write flows.
|
||||
- A backward-compatible evolution path is possible by:
|
||||
- preserving current fields in session manifest for migration/read-compat,
|
||||
- adding explicit run-manifest type + path,
|
||||
- gradually moving invocation-specific details to run manifests.
|
||||
|
||||
## 2.3 Stage Output Paths (Current)
|
||||
|
||||
All implemented stages currently write canonical artifacts directly into session-level `paths.*` roots (under current session root), with logs/configs typically also session-level.
|
||||
|
||||
1. `prepare` ([`internal/stage/prepare.go`](../../internal/stage/prepare.go))
|
||||
- Inputs copied to `inputs/` (`session.yml`, `pipeline.resolved.yml`, `speakers.yml`, `autocorrect.yml`, `glossary.yml`)
|
||||
- Audio copied to `audio/`
|
||||
- Manifest input provenance recorded in `m.Inputs`
|
||||
- S3 audio uses run-scoped spool/work helpers when `RunID` is present.
|
||||
|
||||
2. `transcribe` ([`internal/stage/transcribe.go`](../../internal/stage/transcribe.go))
|
||||
- Outputs: `transcripts/raw/{speaker}.json`
|
||||
- Stage metadata only (no stage logs/config files produced here).
|
||||
|
||||
3. `merge` ([`internal/stage/merge.go`](../../internal/stage/merge.go))
|
||||
- Pre-normalize intermediates: `transcripts/raw/normalized/{basename}.normalized.json`
|
||||
- Merge output: `transcripts/merged.json`
|
||||
- Report: `artifacts/seriatim.report.json` (if enabled)
|
||||
- Logs/configs:
|
||||
- per-input normalize logs/configs in session-level `logs/` + `config/`
|
||||
- merge logs/config in session-level `logs/` + `config/`
|
||||
|
||||
4. `polish` ([`internal/stage/polish.go`](../../internal/stage/polish.go))
|
||||
- Output: `transcripts/processed.json`
|
||||
- Report: `artifacts/audita.report.json` (if enabled)
|
||||
- Work dir: `artifacts/audita-work`
|
||||
- Logs/config: `logs/audita.*`, `config/audita.generated.yml`
|
||||
|
||||
5. `normalize` ([`internal/stage/normalize.go`](../../internal/stage/normalize.go))
|
||||
- Output: `transcripts/normalized.json` (configurable)
|
||||
- Report: `artifacts/seriatim.normalize.report.json` (if enabled)
|
||||
- Logs/config: `logs/seriatim.normalize.*`, `config/seriatim.normalize.generated.yml`
|
||||
|
||||
6. `trim` ([`internal/stage/trim.go`](../../internal/stage/trim.go))
|
||||
- Output: `transcripts/trimmed.json` (configurable)
|
||||
- Bounds output path from config (default examples use `artifacts/session_bounds.json`)
|
||||
- Logs/configs in session-level `logs/` and `config/` for scriptorium + seriatim invocations
|
||||
|
||||
7. `analyze` ([`internal/stage/analyze.go`](../../internal/stage/analyze.go))
|
||||
- Current implemented artifact: `artifacts/session_recap.md`
|
||||
- Logs/config in session-level `logs/` + `config/`
|
||||
- Optional render diagnostics under session-level artifacts/logs/config.
|
||||
|
||||
8. `archive` ([`internal/stage/archive.go`](../../internal/stage/archive.go))
|
||||
- Reads from a "workDir" derived by `archiveWorkDir`:
|
||||
- prefers `m.LocalWorkDir` if it exists,
|
||||
- else tries run-scoped campaign/session/run path,
|
||||
- else falls back to legacy session path.
|
||||
- Uploads run record and promotions.
|
||||
- Current code now resolves manifest and promotion sources via run/session fallback.
|
||||
|
||||
9. `notify`
|
||||
- Placeholder only in [`internal/stage/placeholders.go`](../../internal/stage/placeholders.go); no durable outputs.
|
||||
|
||||
Durable-vs-diagnostic split today:
|
||||
|
||||
- Durable artifacts and diagnostics are mixed at session level.
|
||||
- No run-local stage directories exist yet.
|
||||
|
||||
## 2.4 Idempotency / Force / Resume / Sparse Runs
|
||||
|
||||
Run control implementation:
|
||||
|
||||
- [`internal/app/run_control.go`](../../internal/app/run_control.go)
|
||||
- skip rule: `!force && stageSucceeded(manifest, stage)`
|
||||
- stale detection TODO only; no invalidation logic.
|
||||
|
||||
Command behavior:
|
||||
|
||||
- `run`: full plan through `executeStages`.
|
||||
- `run-stage`: single selected stage through `executeStages`.
|
||||
- `resume`: starts at first non-succeeded stage unless `--force`.
|
||||
|
||||
Current assumptions:
|
||||
|
||||
- Command invocation mutates a single durable workspace + single manifest for the session path model.
|
||||
- There is no per-invocation run manifest write path.
|
||||
|
||||
Smallest safe UX-preserving invariant to keep during migration:
|
||||
|
||||
- Session manifest remains source-of-truth for skip decisions across invocations.
|
||||
|
||||
## 2.5 Archive and S3 Alignment
|
||||
|
||||
S3 semantics are relatively mature:
|
||||
|
||||
- [`internal/artifacts/s3_keys.go`](../../internal/artifacts/s3_keys.go)
|
||||
- session prefix + run prefix + `current/manifest.json` + `current/run_id.txt`.
|
||||
|
||||
Archive stage behavior:
|
||||
|
||||
- uploads run records under run prefix,
|
||||
- uploads promoted session outputs,
|
||||
- uploads current manifest then current run pointer,
|
||||
- current pointer is commit marker,
|
||||
- required promotions fail; optional promotions skipped.
|
||||
|
||||
Local-vs-remote mismatch still present:
|
||||
|
||||
- Local canonical session root currently defaults to `work/{session}` (artifact store),
|
||||
- while archive/run helpers expect campaign-aware run locations (`work/{campaign}/{session}/{run}`),
|
||||
- causing fallback logic and hybrid handling in `archive`.
|
||||
|
||||
## 2.6 Artifact Source Resolution in Analyze
|
||||
|
||||
Current implementation is stage-local and ad hoc:
|
||||
|
||||
- Transcript discoverers:
|
||||
- `discoverProcessedTranscript`
|
||||
- `discoverNormalizedTranscript`
|
||||
- `discoverTrimmedTranscript`
|
||||
- Input source switch in `resolveScriptoriumInput` supports:
|
||||
- `processed_transcript`
|
||||
- `normalized_transcript`
|
||||
- `trimmed_transcript`
|
||||
- `previous_session_artifact`
|
||||
|
||||
No first-class artifact registry exists yet. Aliases and canonical IDs are not modeled.
|
||||
|
||||
## 2.7 Stale/Invalidation
|
||||
|
||||
- `manifest.StageStatus` already defines `stale` (`internal/manifest/status.go`), but no stage uses it.
|
||||
- Skip logic ignores stale state and only checks `succeeded`.
|
||||
- Forced upstream rerun does not invalidate downstream stage success markers.
|
||||
|
||||
## 2.8 Test Coverage Relevant to Redesign
|
||||
|
||||
High-value existing coverage:
|
||||
|
||||
- Workspace helpers:
|
||||
- [`internal/artifacts/paths_model_test.go`](../../internal/artifacts/paths_model_test.go)
|
||||
- [`internal/artifacts/resolve_test.go`](../../internal/artifacts/resolve_test.go)
|
||||
- Runner semantics:
|
||||
- [`internal/app/runner_test.go`](../../internal/app/runner_test.go)
|
||||
- [`internal/app/resume_run_stage_test.go`](../../internal/app/resume_run_stage_test.go)
|
||||
- Stage path/output behavior:
|
||||
- `internal/stage/*_test.go` for prepare/transcribe/merge/polish/normalize/trim/analyze/archive
|
||||
- Archive behavior:
|
||||
- [`internal/stage/archive_test.go`](../../internal/stage/archive_test.go)
|
||||
- [`internal/app/post_archive_cleanup_test.go`](../../internal/app/post_archive_cleanup_test.go)
|
||||
|
||||
Tests likely to fail during workspace redesign:
|
||||
|
||||
- Any tests hardcoding `work/{session}` manifest and transcript paths (many in `internal/app/*test.go`, `internal/stage/*test.go`).
|
||||
- Archive tests assuming current hybrid fallback behavior.
|
||||
|
||||
---
|
||||
|
||||
## 3. Gap Analysis Against `docs/development/workspace.md`
|
||||
|
||||
| Intended concept | Current status | Notes |
|
||||
|---|---|---|
|
||||
| Session root at `work/{campaign}/{session}` | **Partial / mostly absent** | `SessionWorkDir` and artifact store still use `work/{session}`. Campaign-aware run helper exists separately. |
|
||||
| Distinct session manifest vs run manifest | **Absent** | One hybrid manifest model/file is used. |
|
||||
| Run-local stage dirs under `runs/{run_id}/{stage}` | **Absent** | Stages write canonical outputs/logs/config directly at session level. |
|
||||
| Immediate promotion run-local -> session canonical | **Absent** | No run-local staging area to promote from today. |
|
||||
| Session manifest skip source of truth | **Present** | Skip/resume use single manifest stage statuses. |
|
||||
| Sparse runs represented under `runs/{run_id}` | **Absent** | No run-manifest/per-run stage records yet. |
|
||||
| Artifact resolver with canonical IDs + aliases | **Absent** | Analyze resolves via stage-local source-name switch and fallback helpers. |
|
||||
| Downstream invalidation for forced upstream reruns | **Absent** | TODO only; no stale propagation or status clearing. |
|
||||
| Local semantics mirror archive semantics | **Partial** | Archive/S3 side models runs/current, local workspace core still session-layout-centric. |
|
||||
| Safe path helpers centralization | **Partial** | Good helper base exists, but many stage-level manual joins still encode conventions. |
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommended Implementation Sequence
|
||||
|
||||
## Step 1: Introduce campaign-aware session path model without behavior break
|
||||
|
||||
Purpose:
|
||||
|
||||
- Add first-class helpers for `work/{campaign}/{session}` and make them available everywhere.
|
||||
|
||||
Expected changes:
|
||||
|
||||
- `internal/artifacts`: add/extend path helpers and `SessionPaths` constructor variants that accept campaign.
|
||||
- `internal/app`: pass campaign into path-model entrypoints where available.
|
||||
|
||||
Behavior change:
|
||||
|
||||
- None initially (can keep legacy fallback reads).
|
||||
|
||||
Tests:
|
||||
|
||||
- Add campaign-aware path-model tests.
|
||||
- Keep legacy-path compatibility tests.
|
||||
|
||||
Risks:
|
||||
|
||||
- Wide compile-time touch due `Store` interface signatures.
|
||||
|
||||
Rollback:
|
||||
|
||||
- Keep legacy helper wrappers until full migration lands.
|
||||
|
||||
## Step 2: Split manifest responsibilities (session manifest + run manifest scaffolding)
|
||||
|
||||
Purpose:
|
||||
|
||||
- Preserve current UX while introducing explicit run execution records.
|
||||
|
||||
Expected changes:
|
||||
|
||||
- `internal/manifest`: add run manifest type/store helpers.
|
||||
- `internal/app/runner.go`: create/load session manifest and initialize per-invocation run manifest path.
|
||||
|
||||
Behavior change:
|
||||
|
||||
- Session manifest remains skip truth source.
|
||||
- Run manifest begins recording invocation metadata/stage actions.
|
||||
|
||||
Tests:
|
||||
|
||||
- New tests for both manifest files existing and being updated correctly.
|
||||
|
||||
Risks:
|
||||
|
||||
- Incorrect ordering of saves can regress crash consistency.
|
||||
|
||||
Rollback:
|
||||
|
||||
- Keep session-manifest-only decision logic until run manifest proves stable.
|
||||
|
||||
## Step 3: Move stage execution products to run-local directories with promotion
|
||||
|
||||
Purpose:
|
||||
|
||||
- Align with workspace architecture (`runs/{run_id}/{stage}/...`) while preserving canonical outputs.
|
||||
|
||||
Expected changes:
|
||||
|
||||
- `internal/stage`: each implemented stage writes outputs/logs/config/reports to run-local paths.
|
||||
- Introduce shared promotion helpers (atomic copy/rename + output validation + manifest provenance).
|
||||
|
||||
Behavior change:
|
||||
|
||||
- Canonical outputs remain session-level; run-local diagnostics now preserved per run.
|
||||
|
||||
Tests:
|
||||
|
||||
- Stage tests updated to assert run-local outputs + promoted canonical outputs.
|
||||
- New tests for immediate promotion and producer run metadata.
|
||||
|
||||
Risks:
|
||||
|
||||
- Highest regression risk (all implemented stages touched).
|
||||
|
||||
Rollback:
|
||||
|
||||
- Stage-by-stage migration flag or phased rollout by stage order.
|
||||
|
||||
## Step 4: Archive alignment pass
|
||||
|
||||
Purpose:
|
||||
|
||||
- Remove hybrid fallback complexity once local layout is canonical.
|
||||
|
||||
Expected changes:
|
||||
|
||||
- `internal/stage/archive.go`: resolve sources from canonical session outputs and run manifests deterministically.
|
||||
- Keep `current/manifest.json` + `current/run_id.txt` publication semantics.
|
||||
|
||||
Behavior change:
|
||||
|
||||
- Simpler source selection; fewer cross-layout heuristics.
|
||||
|
||||
Tests:
|
||||
|
||||
- Archive tests for run uploads, promotions, current pointers, and fallback removal/compat gates.
|
||||
|
||||
Risks:
|
||||
|
||||
- Breaking current mixed-layout compatibility too early.
|
||||
|
||||
Rollback:
|
||||
|
||||
- Retain fallback compatibility for one migration window.
|
||||
|
||||
## Step 5: Artifact registry/resolver (analyze first consumer)
|
||||
|
||||
Purpose:
|
||||
|
||||
- Replace ad hoc analyze source resolution with canonical artifact IDs and aliases.
|
||||
|
||||
Expected changes:
|
||||
|
||||
- New resolver package (for example `internal/artifacts/registry` or `internal/stage/artifactresolve`) with IDs:
|
||||
- `narratio.transcript.merged`
|
||||
- `narratio.transcript.polished`
|
||||
- `narratio.transcript.full`
|
||||
- `narratio.transcript.trimmed`
|
||||
- `narratio.bounds.session`
|
||||
- `narratio.artifact.session_recap`
|
||||
- Backward-compatible alias map for current source names.
|
||||
|
||||
Behavior change:
|
||||
|
||||
- Analyze input resolution becomes centralized and consistent.
|
||||
|
||||
Tests:
|
||||
|
||||
- Resolver unit tests for canonical IDs + aliases + missing-input error clarity.
|
||||
- Analyze tests updated to assert resolver usage.
|
||||
|
||||
Risks:
|
||||
|
||||
- Input resolution edge cases for `previous_session_artifact` and required/optional handling.
|
||||
|
||||
Rollback:
|
||||
|
||||
- Keep old resolver path behind a temporary compatibility function.
|
||||
|
||||
## Step 6: Minimal downstream invalidation after forced upstream reruns
|
||||
|
||||
Purpose:
|
||||
|
||||
- Prevent stale downstream skips before full checksum stale detection exists.
|
||||
|
||||
Expected changes:
|
||||
|
||||
- `internal/app/run_control.go` + manifest transition helpers.
|
||||
- On forced rerun success of stage `X`, clear/mark downstream success states.
|
||||
|
||||
Behavior change:
|
||||
|
||||
- Subsequent runs no longer skip stale downstream stages.
|
||||
|
||||
Tests:
|
||||
|
||||
- New run-control tests for force-induced downstream invalidation.
|
||||
- Resume tests with forced sparse runs.
|
||||
|
||||
Risks:
|
||||
|
||||
- Over-invalidating too broadly and degrading UX.
|
||||
|
||||
Rollback:
|
||||
|
||||
- Start with deterministic downstream stage list based on pipeline order only.
|
||||
|
||||
## Step 7: Legacy layout migration strategy
|
||||
|
||||
Purpose:
|
||||
|
||||
- Handle existing `work/{session}` data safely.
|
||||
|
||||
Expected changes:
|
||||
|
||||
- Startup detection/migration path in app layer.
|
||||
- Clear failure messages for ambiguous legacy states.
|
||||
|
||||
Behavior change:
|
||||
|
||||
- Explicit migration semantics instead of implicit fallback drift.
|
||||
|
||||
Tests:
|
||||
|
||||
- Migration detection tests for legacy-only, new-only, and ambiguous layouts.
|
||||
|
||||
Risks:
|
||||
|
||||
- Silent duplication if both layouts are partially populated.
|
||||
|
||||
Rollback:
|
||||
|
||||
- Prefer fail-fast ambiguity policy over auto-merge.
|
||||
|
||||
---
|
||||
|
||||
## 5. Minimal Viable v1.0 Scope
|
||||
|
||||
Must-have for v1.0:
|
||||
|
||||
1. Campaign-aware canonical session roots.
|
||||
2. Session manifest remains skip/resume authority.
|
||||
3. Introduce run manifests + run-local stage records.
|
||||
4. Immediate promotion from run-local outputs to canonical session outputs.
|
||||
5. Minimal downstream invalidation on forced upstream rerun.
|
||||
6. Archive/local semantic alignment with `current/*` behavior preserved.
|
||||
7. Backward-compatible analyze aliases (`processed_transcript`, `normalized_transcript`, `trimmed_transcript`).
|
||||
|
||||
Nice-to-have / defer if risky:
|
||||
|
||||
1. Full checksum-based stale detection graph.
|
||||
2. Broad manifest schema-version migration framework.
|
||||
3. Full artifact-registry rollout beyond analyze’s initial needs.
|
||||
4. Aggressive cleanup of all legacy fallback branches in one release.
|
||||
|
||||
---
|
||||
|
||||
## 6. Open Questions (with Recommendations)
|
||||
|
||||
1. **Should campaign be mandatory for all local path derivation immediately?**
|
||||
- Recommendation: yes for new layout writes; keep controlled read compatibility for legacy session-only roots during migration window.
|
||||
|
||||
2. **Session manifest location transition policy:** auto-migrate vs explicit migrate command?
|
||||
- Recommendation: if user base is small, prefer explicit fail-fast with actionable migration instructions to avoid silent split-brain state.
|
||||
|
||||
3. **Run manifest granularity:** per-stage detailed records vs summary + references?
|
||||
- Recommendation: start with summary + stage status/paths; avoid duplicating full session artifact state to keep write path simple.
|
||||
|
||||
4. **Invalidation marking:** use `stale` status now or clear `succeeded` markers?
|
||||
- Recommendation: if invasive to propagate new status semantics quickly, clear/overwrite downstream success states first; add formal `stale` usage in follow-up.
|
||||
|
||||
5. **Promotion timing:** promote every stage immediately vs delayed at end of run?
|
||||
- Recommendation: immediate per-stage promotion after validation (matches current idempotent skip expectations and simplifies resume behavior).
|
||||
|
||||
6. **Docs consistency order after implementation starts:**
|
||||
- Recommendation: update `README.md` and `architecture.md` in lockstep with each migration step. Current conflict is explicit:
|
||||
- code/README/architecture still primarily describe session roots at `work/{session}`
|
||||
- `docs/development/workspace.md` defines `work/{campaign}/{session}` plus run manifests.
|
||||
|
||||
@@ -1,753 +0,0 @@
|
||||
# Narratio Workspace, Run History, and Artifact Resolution Architecture
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
This document defines the intended v1.0 architecture for Narratio's local workspace layout, run history model, durable session outputs, manifest responsibilities, and artifact resolution contract.
|
||||
|
||||
Narratio is an idempotent session orchestrator. The command:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07
|
||||
```
|
||||
|
||||
means "bring the identified session to its desired completed state." It does **not** mean "always create an entirely new independent output tree and ignore prior session state."
|
||||
|
||||
This distinction drives the architecture:
|
||||
|
||||
* A **session** is the durable domain object and idempotency boundary.
|
||||
* A **run** is an execution attempt that may update the session's durable state.
|
||||
* Durable outputs live at the session level.
|
||||
* Run-specific outputs, logs, generated configs, scratch files, and diagnostics live under `runs/{run_id}/`.
|
||||
* Successful stage outputs are promoted from run-local locations into canonical session-level locations.
|
||||
* The session manifest records current durable state.
|
||||
* Run manifests record execution history and debugging/provenance details.
|
||||
|
||||
This model intentionally mirrors the S3 archive model: session-level current artifacts are distinct from run-record history.
|
||||
|
||||
## 2. Core Concepts
|
||||
|
||||
### 2.1 Session
|
||||
|
||||
A session is the stable unit of work identified by `campaign_id` and `session_id`.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
campaign_id = dilfs
|
||||
session_id = 2026-05-07
|
||||
```
|
||||
|
||||
The session directory represents the current durable local state for that session. Re-running Narratio for the same session should consult this state, skip already-completed stages by default, and produce no changes unless work is incomplete, stale, forced, or explicitly selected.
|
||||
|
||||
### 2.2 Run
|
||||
|
||||
A run is a particular execution attempt identified by a generated `run_id`, for example:
|
||||
|
||||
```text
|
||||
20260517T174748Z-abcd1234
|
||||
```
|
||||
|
||||
A run may execute all stages or only a sparse subset of stages. Sparse runs are expected and desirable when the user invokes `--force`, `run-stage`, or a stage-limited command.
|
||||
|
||||
Run directories are provenance/debug records. They should reflect what actually happened during that invocation, not a synthetic complete pipeline layout.
|
||||
|
||||
### 2.3 Durable Output
|
||||
|
||||
A durable output is a canonical session-level artifact intended for later stages, user consumption, archive promotion, or future idempotency decisions.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
transcripts/merged.json
|
||||
transcripts/processed.json
|
||||
transcripts/normalized.json
|
||||
transcripts/trimmed.json
|
||||
artifacts/session_recap.md
|
||||
```
|
||||
|
||||
Durable outputs live directly under the session directory, not under a particular run directory.
|
||||
|
||||
### 2.4 Run-Local Output
|
||||
|
||||
A run-local output is the file initially produced by a stage during a specific run. After validation, durable outputs are promoted from run-local paths to session-level canonical paths.
|
||||
|
||||
Run-local outputs, logs, generated configs, reports, and scratch files should remain under:
|
||||
|
||||
```text
|
||||
runs/{run_id}/{stage}/...
|
||||
```
|
||||
|
||||
## 3. Local Workspace Layout
|
||||
|
||||
The canonical local workspace layout is:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/
|
||||
manifest.json
|
||||
current/
|
||||
manifest.json
|
||||
run_id.txt
|
||||
inputs/
|
||||
transcripts/
|
||||
artifacts/
|
||||
reports/
|
||||
logs/
|
||||
config/
|
||||
runs/
|
||||
{run_id}/
|
||||
manifest.json
|
||||
prepare/
|
||||
transcribe/
|
||||
merge/
|
||||
polish/
|
||||
normalize/
|
||||
trim/
|
||||
analyze/
|
||||
archive/
|
||||
notify/
|
||||
```
|
||||
|
||||
Not every directory must exist at all times. Directories should be created idempotently when needed.
|
||||
|
||||
### 3.1 Session Root
|
||||
|
||||
The session root is:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/
|
||||
```
|
||||
|
||||
The session root is the stable local home for the session. It is the default base for resolving canonical artifact paths.
|
||||
|
||||
The only files that should live directly in the session root are core session-state files, primarily:
|
||||
|
||||
```text
|
||||
manifest.json
|
||||
```
|
||||
|
||||
Lock files may also be session-root scoped if the implementation uses file locks there, but transient locks should not be treated as durable artifacts.
|
||||
|
||||
### 3.2 Session-Level Canonical Directories
|
||||
|
||||
The following directories contain current durable session state:
|
||||
|
||||
```text
|
||||
inputs/
|
||||
transcripts/
|
||||
artifacts/
|
||||
reports/
|
||||
logs/
|
||||
config/
|
||||
current/
|
||||
```
|
||||
|
||||
Recommended meanings:
|
||||
|
||||
| Directory | Purpose |
|
||||
| -------------- | ----------------------------------------------------------------------------- |
|
||||
| `inputs/` | Materialized or copied input files used by the current durable session state. |
|
||||
| `transcripts/` | Canonical transcript tiers. |
|
||||
| `artifacts/` | User-facing and machine-readable generated artifacts. |
|
||||
| `reports/` | Canonical stage reports worth preserving at the session level. |
|
||||
| `logs/` | Optional session-level logs or promoted/latest logs. |
|
||||
| `config/` | Optional session-level generated config snapshots or promoted/latest configs. |
|
||||
| `current/` | Current published session pointers, mirroring the archive backend. |
|
||||
|
||||
Canonical durable outputs should use stable paths under these directories.
|
||||
|
||||
### 3.3 Run History Directory
|
||||
|
||||
Run history lives under:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/
|
||||
```
|
||||
|
||||
Each run directory records what happened during that invocation. A run may contain all stage directories or only a sparse subset.
|
||||
|
||||
Example full run:
|
||||
|
||||
```text
|
||||
runs/20260517T174748Z-abcd1234/
|
||||
manifest.json
|
||||
prepare/
|
||||
transcribe/
|
||||
merge/
|
||||
polish/
|
||||
normalize/
|
||||
trim/
|
||||
analyze/
|
||||
archive/
|
||||
notify/
|
||||
```
|
||||
|
||||
Example sparse forced analyze run:
|
||||
|
||||
```text
|
||||
runs/20260518T030000Z-efgh5678/
|
||||
manifest.json
|
||||
analyze/
|
||||
```
|
||||
|
||||
Example sparse polish-through-analyze rerun:
|
||||
|
||||
```text
|
||||
runs/20260518T041500Z-a1b2c3d4/
|
||||
manifest.json
|
||||
polish/
|
||||
normalize/
|
||||
trim/
|
||||
analyze/
|
||||
```
|
||||
|
||||
Run directories should not create stage folders for stages that were not selected, executed, skipped, or otherwise considered during that run unless there is a clear diagnostic reason to do so.
|
||||
|
||||
### 3.4 Stage Run-Local Directories
|
||||
|
||||
Each stage receives a run-local directory:
|
||||
|
||||
```text
|
||||
runs/{run_id}/{stage}/
|
||||
```
|
||||
|
||||
Within that stage directory, the stage may use subdirectories such as:
|
||||
|
||||
```text
|
||||
outputs/
|
||||
logs/
|
||||
reports/
|
||||
config/
|
||||
scratch/
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
runs/{run_id}/polish/
|
||||
outputs/transcripts/processed.json
|
||||
reports/audita.polish.report.json
|
||||
logs/stdout.log
|
||||
logs/stderr.log
|
||||
config/audita.polish.generated.yml
|
||||
scratch/
|
||||
```
|
||||
|
||||
The exact internal layout of a stage directory may vary by stage, but it should be deterministic, documented, and generated through centralized path helpers rather than ad hoc path joins.
|
||||
|
||||
## 4. Promotion Model
|
||||
|
||||
Narratio uses stage-level promotion with immediate promotion after successful validation.
|
||||
|
||||
The stage lifecycle is:
|
||||
|
||||
1. Resolve required inputs from the current session state and/or run-local context.
|
||||
2. Create the run-local stage directory.
|
||||
3. Execute the stage, writing outputs under `runs/{run_id}/{stage}/...`.
|
||||
4. Validate run-local outputs.
|
||||
5. Promote durable outputs into session-level canonical paths.
|
||||
6. Update the session manifest.
|
||||
7. Update the run manifest.
|
||||
|
||||
Promotion means an atomic or effectively atomic copy/rename from a run-local path to a session-level canonical path.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
runs/{run_id}/polish/outputs/transcripts/processed.json
|
||||
```
|
||||
|
||||
is promoted to:
|
||||
|
||||
```text
|
||||
transcripts/processed.json
|
||||
```
|
||||
|
||||
Promotion should be safe and deterministic:
|
||||
|
||||
* Validate before promotion.
|
||||
* Write promoted files atomically where possible.
|
||||
* Never leave partially written durable outputs.
|
||||
* Record the producing `run_id` in the session manifest.
|
||||
* Preserve run-local files for debugging unless retention policy deletes them.
|
||||
|
||||
## 5. Promotion Policy: Option A
|
||||
|
||||
Narratio uses immediate stage-level promotion.
|
||||
|
||||
If a selected stage succeeds, its durable outputs are promoted immediately, even if a later selected stage fails.
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07 --force --stages polish,normalize,trim,analyze
|
||||
```
|
||||
|
||||
If `polish` succeeds and `normalize` fails:
|
||||
|
||||
* `transcripts/processed.json` may be updated from the new run.
|
||||
* `normalize`, `trim`, and `analyze` should not be marked succeeded for the new input state.
|
||||
* Downstream outputs may now be stale relative to the newly promoted polished transcript.
|
||||
|
||||
This policy is simpler, transparent, and consistent with stage-level resumability. It does require explicit stale/invalidation handling.
|
||||
|
||||
## 6. Stale and Invalidation Semantics
|
||||
|
||||
Full checksum-based stale detection may be implemented later. Before that exists, Narratio should still use a simple deterministic invalidation rule for forced or explicit upstream reruns.
|
||||
|
||||
When a stage is successfully re-executed and promoted, downstream stages should be marked stale unless they are also re-executed successfully in the same command invocation.
|
||||
|
||||
Example stage order:
|
||||
|
||||
```text
|
||||
prepare -> transcribe -> merge -> polish -> normalize -> trim -> analyze -> archive -> notify
|
||||
```
|
||||
|
||||
If `polish` is forced and promoted, then the following downstream stages should be invalidated unless rerun successfully:
|
||||
|
||||
```text
|
||||
normalize
|
||||
trim
|
||||
analyze
|
||||
archive
|
||||
notify
|
||||
```
|
||||
|
||||
A stale stage is not equivalent to a failed stage. It means its current durable outputs may no longer correspond to current upstream inputs or configuration.
|
||||
|
||||
Minimum manifest state model:
|
||||
|
||||
```text
|
||||
pending
|
||||
running
|
||||
succeeded
|
||||
failed
|
||||
skipped
|
||||
stale
|
||||
```
|
||||
|
||||
If adding a new `stale` state is too invasive for v1.0, the implementation should at least record stale metadata or clear downstream success markers in a way that prevents accidental idempotent skips based on obsolete outputs.
|
||||
|
||||
## 7. Manifest Responsibilities
|
||||
|
||||
Narratio should distinguish between session manifests and run manifests.
|
||||
|
||||
The same underlying Go types may be reused where practical, but the concepts should remain separate.
|
||||
|
||||
### 7.1 Session Manifest
|
||||
|
||||
Path:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
|
||||
```
|
||||
|
||||
The session manifest answers:
|
||||
|
||||
```text
|
||||
What is the current durable state of this session?
|
||||
```
|
||||
|
||||
It should record:
|
||||
|
||||
* campaign ID
|
||||
* session ID
|
||||
* current or latest run ID
|
||||
* current stage states
|
||||
* canonical durable output refs
|
||||
* artifact IDs and paths
|
||||
* producing run ID for each current stage output
|
||||
* relevant input/config checksums when available
|
||||
* stale/invalidated stage information
|
||||
* archive/current publication metadata
|
||||
|
||||
A session's durable state may be a composite of multiple runs.
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
transcripts/merged.json produced by run A
|
||||
transcripts/processed.json produced by run B
|
||||
transcripts/normalized.json produced by run B
|
||||
transcripts/trimmed.json produced by run B
|
||||
artifacts/session_recap.md produced by run C
|
||||
```
|
||||
|
||||
This is valid and expected.
|
||||
|
||||
### 7.2 Run Manifest
|
||||
|
||||
Path:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/runs/{run_id}/manifest.json
|
||||
```
|
||||
|
||||
The run manifest answers:
|
||||
|
||||
```text
|
||||
What happened during this specific execution attempt?
|
||||
```
|
||||
|
||||
It should record:
|
||||
|
||||
* run ID
|
||||
* campaign ID
|
||||
* session ID
|
||||
* command mode and selected stages
|
||||
* force flags or stage selection flags
|
||||
* stages considered during this run
|
||||
* stages executed during this run
|
||||
* stages skipped during this run and reasons
|
||||
* run-local output paths
|
||||
* promoted output paths
|
||||
* logs
|
||||
* reports
|
||||
* generated configs
|
||||
* timings
|
||||
* errors
|
||||
* non-secret subprocess invocation metadata
|
||||
|
||||
Run manifests are primarily for debugging, auditability, and archive history.
|
||||
|
||||
## 8. Idempotency and Resume Behavior
|
||||
|
||||
The idempotency boundary is the session, not the run.
|
||||
|
||||
By default:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07
|
||||
```
|
||||
|
||||
should consult the session manifest and skip stages that are already succeeded and not stale.
|
||||
|
||||
If all stages are already complete, the command should execute zero stages and report that the session is already complete.
|
||||
|
||||
Forced execution creates a new run record but updates session-level durable state only for stages that actually succeed and promote outputs.
|
||||
|
||||
Examples:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-05-07 --force
|
||||
```
|
||||
|
||||
Creates a new run and attempts to re-execute the selected/default stage set.
|
||||
|
||||
```bash
|
||||
narratio run-stage --session-id 2026-05-07 analyze --force
|
||||
```
|
||||
|
||||
Creates a sparse run that executes only `analyze`, then promotes updated analysis artifacts if successful.
|
||||
|
||||
```bash
|
||||
narratio resume --session-id 2026-05-07
|
||||
```
|
||||
|
||||
Uses the session manifest to determine what remains incomplete or stale. Resume does not need to resume the same `run_id` unless the implementation explicitly supports resuming an interrupted active run.
|
||||
|
||||
## 9. Artifact Resolution Contract
|
||||
|
||||
Narratio should provide a first-class artifact registry and resolver.
|
||||
|
||||
The resolver maps symbolic artifact source names to canonical session-level paths and manifest output kinds.
|
||||
|
||||
Stages and adapters should not hardcode path fragments when resolving cross-stage inputs. They should ask the artifact resolver for the current durable artifact by ID.
|
||||
|
||||
### 9.1 Canonical Artifact IDs
|
||||
|
||||
Preferred artifact IDs should be namespaced:
|
||||
|
||||
```text
|
||||
narratio.transcript.merged
|
||||
narratio.transcript.polished
|
||||
narratio.transcript.full
|
||||
narratio.transcript.trimmed
|
||||
narratio.bounds.session
|
||||
narratio.artifact.session_recap
|
||||
```
|
||||
|
||||
Recommended initial registry:
|
||||
|
||||
| Artifact ID | Canonical Path | Producer Stage | Output Kind | Meaning |
|
||||
| --------------------------------- | ------------------------------- | -------------- | ------------------------ | ------------------------------------- |
|
||||
| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` | Deterministic Seriatim merge. |
|
||||
| `narratio.transcript.polished` | `transcripts/processed.json` | `polish` | `transcript_processed` | Full Audita-polished transcript. |
|
||||
| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` | Preferred full normalized transcript. |
|
||||
| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` | Gameplay-only transcript. |
|
||||
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` | Trim bounds selected for the session. |
|
||||
| `narratio.artifact.session_recap` | `artifacts/session_recap.md` | `analyze` | `artifact_session_recap` | Generated session recap. |
|
||||
|
||||
### 9.2 Backward-Compatible Aliases
|
||||
|
||||
Existing source names should remain supported:
|
||||
|
||||
| Legacy Source | Preferred Artifact ID |
|
||||
| ----------------------- | ------------------------------ |
|
||||
| `processed_transcript` | `narratio.transcript.polished` |
|
||||
| `normalized_transcript` | `narratio.transcript.full` |
|
||||
| `trimmed_transcript` | `narratio.transcript.trimmed` |
|
||||
|
||||
These aliases may be supported silently for v1.0. Documentation should prefer namespaced IDs.
|
||||
|
||||
### 9.3 Resolver Behavior
|
||||
|
||||
Artifact resolution should follow this order:
|
||||
|
||||
1. Normalize aliases to canonical artifact IDs.
|
||||
2. Look for a current output reference in the session manifest.
|
||||
3. Fall back to the canonical session-level path.
|
||||
4. If the artifact is required, fail clearly if missing.
|
||||
5. If the artifact is optional and missing, omit it from the downstream invocation.
|
||||
6. Validate the artifact using the expected content validator.
|
||||
7. Return a resolved artifact record containing ID, path, producer stage, output kind, and provenance.
|
||||
|
||||
Example conceptual result:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "narratio.transcript.trimmed",
|
||||
"path": "/var/lib/narratio/work/dilfs/2026-05-07/transcripts/trimmed.json",
|
||||
"producer_stage": "trim",
|
||||
"producer_run_id": "20260517T174748Z-abcd1234",
|
||||
"output_kind": "transcript_trimmed",
|
||||
"content_type": "application/json"
|
||||
}
|
||||
```
|
||||
|
||||
### 9.4 Artifact Validation
|
||||
|
||||
Transcript artifacts must be valid JSON with a top-level `segments` array.
|
||||
|
||||
Markdown/text artifacts must exist and be non-empty when required.
|
||||
|
||||
Bounds artifacts must match the expected bounds schema and refer to segment IDs in the same transcript ID space used by the trim stage.
|
||||
|
||||
Validation should happen before a resolved artifact is passed to another stage or external subprocess.
|
||||
|
||||
## 10. Analyze Stage Implications
|
||||
|
||||
The analyze stage should consume artifacts through the artifact resolver.
|
||||
|
||||
Preferred Scriptorium config shape:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: "dnd.session_recap"
|
||||
output_path: "artifacts/session_recap.md"
|
||||
inputs:
|
||||
transcript:
|
||||
source: "narratio.transcript.trimmed"
|
||||
required: true
|
||||
```
|
||||
|
||||
Additional artifacts can choose different transcript tiers:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
table_summary:
|
||||
enabled: true
|
||||
prompt_id: "dnd.table_summary"
|
||||
output_path: "artifacts/table_summary.md"
|
||||
inputs:
|
||||
transcript:
|
||||
source: "narratio.transcript.full"
|
||||
required: true
|
||||
```
|
||||
|
||||
For v1.0, Narratio does not need a generic DAG engine. It may execute configured analyze artifacts in deterministic order and allow later artifacts to consume earlier artifacts only when that relationship is explicit and unambiguous.
|
||||
|
||||
Rules:
|
||||
|
||||
* Artifact inputs resolve from current session-level durable state.
|
||||
* Outputs are first written run-locally.
|
||||
* Successful analyze outputs are promoted to session-level `artifacts/` paths.
|
||||
* Manifest output refs record the producing run ID.
|
||||
* Optional inputs are omitted when unavailable.
|
||||
* Required missing inputs fail before invoking Scriptorium.
|
||||
|
||||
## 11. Archive Alignment
|
||||
|
||||
Local workspace semantics should mirror archive semantics.
|
||||
|
||||
Local session-level durable paths:
|
||||
|
||||
```text
|
||||
work/{campaign}/{session}/transcripts/trimmed.json
|
||||
work/{campaign}/{session}/artifacts/session_recap.md
|
||||
work/{campaign}/{session}/current/manifest.json
|
||||
work/{campaign}/{session}/current/run_id.txt
|
||||
work/{campaign}/{session}/runs/{run_id}/...
|
||||
```
|
||||
|
||||
should map naturally to remote archive paths:
|
||||
|
||||
```text
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/transcripts/trimmed.json
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/artifacts/session_recap.md
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/manifest.json
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/current/run_id.txt
|
||||
{root_prefix}/campaigns/{campaign}/sessions/{session}/runs/{run_id}/...
|
||||
```
|
||||
|
||||
The archive stage should publish run records and promoted current artifacts consistently with the local model.
|
||||
|
||||
`current/run_id.txt` remains the effective commit marker for the archived current session state.
|
||||
|
||||
## 12. Path Helper Requirements
|
||||
|
||||
All code should use centralized path helpers for workspace paths.
|
||||
|
||||
Stage code should not manually assemble durable cross-stage paths using raw string joins except through the path model.
|
||||
|
||||
Recommended helper surface:
|
||||
|
||||
```text
|
||||
SessionRoot(campaignID, sessionID)
|
||||
SessionManifestPath(campaignID, sessionID)
|
||||
SessionCurrentDir(campaignID, sessionID)
|
||||
SessionTranscriptsDir(campaignID, sessionID)
|
||||
SessionArtifactsDir(campaignID, sessionID)
|
||||
SessionReportsDir(campaignID, sessionID)
|
||||
SessionLogsDir(campaignID, sessionID)
|
||||
SessionConfigDir(campaignID, sessionID)
|
||||
RunsDir(campaignID, sessionID)
|
||||
RunRoot(campaignID, sessionID, runID)
|
||||
RunManifestPath(campaignID, sessionID, runID)
|
||||
RunStageDir(campaignID, sessionID, runID, stage)
|
||||
RunStageOutputsDir(campaignID, sessionID, runID, stage)
|
||||
RunStageLogsDir(campaignID, sessionID, runID, stage)
|
||||
RunStageReportsDir(campaignID, sessionID, runID, stage)
|
||||
RunStageConfigDir(campaignID, sessionID, runID, stage)
|
||||
CanonicalArtifactPath(campaignID, sessionID, artifactID)
|
||||
```
|
||||
|
||||
Path helpers should enforce safe relative paths for configured output paths:
|
||||
|
||||
* reject absolute paths unless explicitly allowed for a particular config field
|
||||
* reject `..` traversal
|
||||
* normalize separators
|
||||
* preserve deterministic output paths
|
||||
|
||||
## 13. Directory Creation Policy
|
||||
|
||||
Directory creation should be centralized and idempotent.
|
||||
|
||||
Recommended policy:
|
||||
|
||||
* `prepare` ensures the baseline session directory structure exists.
|
||||
* Every stage also calls shared layout helpers to ensure its required run-local directories exist before writing.
|
||||
* `run-stage` should not depend on a prior `prepare` invocation merely to create folders.
|
||||
* Missing directories should be created with appropriate permissions.
|
||||
* Directory creation should not imply stage success.
|
||||
|
||||
This provides consistent layout while keeping direct stage execution robust.
|
||||
|
||||
## 14. Cleanup and Retention
|
||||
|
||||
Cleanup must preserve the distinction between durable session state and run history.
|
||||
|
||||
Workspace cleanup after successful archive may remove selected local directories only according to explicit configuration.
|
||||
|
||||
Potential retention policies:
|
||||
|
||||
```text
|
||||
keep_all_runs
|
||||
keep_failed_runs
|
||||
keep_last_n_runs
|
||||
delete_run_after_success
|
||||
```
|
||||
|
||||
For v1.0, conservative retention is preferred:
|
||||
|
||||
* Do not delete durable session-level outputs unless explicitly requested.
|
||||
* Do not delete failed run directories by default.
|
||||
* If cleanup is enabled, remove only documented run-scoped or spool-scoped paths.
|
||||
* Local development audio inputs must never be deleted by workspace cleanup.
|
||||
|
||||
## 15. Migration From Existing Layout
|
||||
|
||||
Existing installations may currently use a simpler path such as:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{session_id}/manifest.json
|
||||
```
|
||||
|
||||
The v1.0 layout introduces campaign-aware session roots:
|
||||
|
||||
```text
|
||||
{workspace.root}/work/{campaign_id}/{session_id}/manifest.json
|
||||
```
|
||||
|
||||
Migration options:
|
||||
|
||||
1. Best-effort automatic discovery of legacy session manifests.
|
||||
2. A one-time migration command.
|
||||
3. Clear release notes requiring users to move or regenerate workspace state.
|
||||
|
||||
For v1.0, it is acceptable to require explicit migration if the user base is small and the archive contains the authoritative durable outputs. However, the application should fail clearly when it detects an ambiguous legacy layout rather than silently creating duplicate state.
|
||||
|
||||
## 16. Documentation Updates Required
|
||||
|
||||
The following documentation should be updated to reflect this architecture:
|
||||
|
||||
* `README.md`
|
||||
* `docs/architecture.md`
|
||||
* a dedicated workspace/run-history document, such as this file
|
||||
* S3/archive documentation
|
||||
* analyze/artifact configuration documentation
|
||||
* example pipeline files
|
||||
|
||||
Documentation should consistently use the following terms:
|
||||
|
||||
| Term | Meaning |
|
||||
| ---------------- | ---------------------------------------------------------------------- |
|
||||
| Session | Durable domain object and idempotency boundary. |
|
||||
| Run | Execution attempt that may update session state. |
|
||||
| Durable output | Canonical current session-level output. |
|
||||
| Run-local output | Output produced inside a specific run directory before promotion. |
|
||||
| Promotion | Validated copy/rename from run-local output to durable session output. |
|
||||
| Session manifest | Current durable state of the session. |
|
||||
| Run manifest | Execution record for a particular run. |
|
||||
| Artifact ID | Symbolic source name resolved by the artifact registry. |
|
||||
|
||||
## 17. Architectural Invariants
|
||||
|
||||
The following invariants should hold after implementation:
|
||||
|
||||
1. `session_id` remains the idempotency boundary for normal operator commands.
|
||||
2. `run_id` identifies an execution attempt, not the primary durable workspace.
|
||||
3. Session-level canonical artifacts are the default inputs for downstream stages.
|
||||
4. Run-local outputs are promoted only after validation.
|
||||
5. A session's current durable state may be composed of outputs from multiple runs.
|
||||
6. Sparse run directories are valid and expected.
|
||||
7. The session manifest records current stage/artifact state and producer run IDs.
|
||||
8. The run manifest records what happened during one invocation.
|
||||
9. Artifact consumers resolve symbolic artifact IDs through a registry/resolver.
|
||||
10. Local workspace semantics mirror S3 archive semantics.
|
||||
11. Directory creation is centralized and idempotent.
|
||||
12. Stage code uses path helpers rather than ad hoc path construction.
|
||||
13. Forced upstream reruns invalidate downstream stage success unless downstream stages are rerun successfully.
|
||||
14. Cleanup never removes durable session outputs or local development inputs unless explicitly configured to do so.
|
||||
|
||||
## 18. Implementation Guidance
|
||||
|
||||
A practical implementation sequence is:
|
||||
|
||||
1. Add this architecture document.
|
||||
2. Add or revise path model helpers for session roots, run roots, stage directories, and canonical artifact paths.
|
||||
3. Introduce session manifest versus run manifest concepts.
|
||||
4. Route stage outputs through run-local directories.
|
||||
5. Add promotion helpers with validation and atomic writes.
|
||||
6. Update existing stages to promote durable outputs to session-level canonical paths.
|
||||
7. Add artifact registry and resolver.
|
||||
8. Update analyze to use artifact IDs and aliases.
|
||||
9. Add simple downstream stale invalidation for forced upstream reruns.
|
||||
10. Align archive/local path behavior and documentation.
|
||||
11. Update examples and README.
|
||||
12. Add tests for idempotency, sparse forced runs, promotion, manifest provenance, and artifact resolution.
|
||||
|
||||
This sequence intentionally avoids introducing a generic DAG engine. The v1.0 goal is a clear, deterministic, stage-oriented orchestrator with stable session-level outputs and inspectable run history.
|
||||
356
docs/documentation/policy.md
Normal file
356
docs/documentation/policy.md
Normal file
@@ -0,0 +1,356 @@
|
||||
# Go Project Documentation Policy
|
||||
|
||||
## Purpose
|
||||
|
||||
Project documentation must help four audiences:
|
||||
|
||||
1. users who need to run the application;
|
||||
2. administrators/operators who need to configure and operate it;
|
||||
3. developers who need to understand and change it safely;
|
||||
4. LLM coding agents that need clear scope, boundaries, and invariants.
|
||||
|
||||
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
|
||||
|
||||
## Core Rules
|
||||
|
||||
### 1. Keep docs concise
|
||||
|
||||
Each document should cover a defined scope and only the essentials for that scope.
|
||||
|
||||
Avoid:
|
||||
- long background explanations;
|
||||
- repeated reference material;
|
||||
- implementation detail in user-facing docs;
|
||||
- aspirational language outside roadmap docs;
|
||||
- verbose examples where one minimal example is clearer.
|
||||
|
||||
### 2. Document only implemented behavior outside roadmap files
|
||||
|
||||
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
|
||||
|
||||
- `docs/roadmap/`
|
||||
|
||||
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
|
||||
|
||||
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
|
||||
|
||||
### 3. Use canonical homes
|
||||
|
||||
Each type of information should have one canonical location.
|
||||
|
||||
Canonical homes:
|
||||
|
||||
- project purpose and quickstart: `README.md`
|
||||
- development principles: `docs/architecture.md`
|
||||
- configuration reference: `docs/config.md`
|
||||
- CLI reference: `docs/cli.md`
|
||||
- operations and recovery: `docs/operations.md`
|
||||
- troubleshooting: `docs/troubleshooting.md`
|
||||
- implemented internals: `docs/internal/`
|
||||
- future work: `docs/roadmap/`
|
||||
- contributor workflow: `docs/development.md`
|
||||
- copyable examples: `examples/`
|
||||
|
||||
Other files should summarize briefly and link to the canonical source.
|
||||
|
||||
### 4. Keep examples real
|
||||
|
||||
Examples should be valid, maintained, and free of secrets.
|
||||
|
||||
Where practical:
|
||||
- example configs should load successfully;
|
||||
- example commands should match real CLI syntax;
|
||||
- important examples should be covered by tests.
|
||||
|
||||
## Documentation Profiles
|
||||
|
||||
All projects require:
|
||||
|
||||
- `README.md`
|
||||
- `docs/architecture.md`
|
||||
|
||||
Additional docs depend on the project.
|
||||
|
||||
### Small library
|
||||
|
||||
Recommended:
|
||||
- `docs/development.md`, if contributor conventions are non-obvious
|
||||
|
||||
### Simple CLI
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/development.md`
|
||||
|
||||
### Config-driven CLI
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`
|
||||
- `docs/config.md`
|
||||
|
||||
Recommended:
|
||||
- `examples/`
|
||||
- `docs/development.md`
|
||||
|
||||
### Stateful or operator-facing application
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`, if CLI-based
|
||||
- `docs/config.md`, if config-driven
|
||||
- `docs/operations.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/troubleshooting.md`
|
||||
- `examples/`
|
||||
- `docs/development.md`
|
||||
|
||||
### Modular, staged, service-oriented, or orchestration application
|
||||
|
||||
Required:
|
||||
- `docs/cli.md`, if CLI-based
|
||||
- `docs/config.md`, if config-driven
|
||||
- `docs/operations.md`
|
||||
- `docs/internal/`
|
||||
- `docs/development.md`
|
||||
|
||||
Recommended:
|
||||
- `docs/troubleshooting.md`
|
||||
- validated examples under `examples/`
|
||||
|
||||
## Required Documents
|
||||
|
||||
### README.md
|
||||
|
||||
**Audience:** users, administrators, operators
|
||||
|
||||
The README is the outward-facing project orientation page.
|
||||
|
||||
It should include, in order:
|
||||
|
||||
1. concise description;
|
||||
2. elevator pitch;
|
||||
3. shortest useful command or usage example;
|
||||
4. links to targeted docs.
|
||||
|
||||
The README should be short. It is not a manual.
|
||||
|
||||
The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.)
|
||||
|
||||
### docs/architecture.md
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
`docs/architecture.md` is required for every project.
|
||||
|
||||
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
|
||||
|
||||
It should include:
|
||||
|
||||
- project shape;
|
||||
- core design principles;
|
||||
- package and boundary philosophy;
|
||||
- state/persistence philosophy, if applicable;
|
||||
- external integration philosophy, if applicable;
|
||||
- error-handling and logging principles;
|
||||
- testing expectations;
|
||||
- documentation expectations;
|
||||
- architectural invariants;
|
||||
- explicit non-goals, if useful.
|
||||
|
||||
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
|
||||
|
||||
### docs/config.md
|
||||
|
||||
**Audience:** administrators, operators, advanced users
|
||||
|
||||
Required for applications with configuration files.
|
||||
|
||||
It should include, in order:
|
||||
|
||||
1. config file locations and discovery precedence;
|
||||
2. minimal working config;
|
||||
3. production-oriented config;
|
||||
4. full configuration reference;
|
||||
5. secrets handling, if applicable;
|
||||
6. links to maintained examples.
|
||||
|
||||
The full configuration reference should be canonical.
|
||||
|
||||
### docs/cli.md
|
||||
|
||||
**Audience:** users, administrators, operators
|
||||
|
||||
Required for CLI applications.
|
||||
|
||||
It should include, in order:
|
||||
|
||||
1. shortest useful command;
|
||||
2. command overview;
|
||||
3. complete flag reference;
|
||||
4. common workflows;
|
||||
5. diagnostic or recovery commands, if applicable.
|
||||
|
||||
Explain when commands are useful, not just their syntax.
|
||||
|
||||
### docs/operations.md
|
||||
|
||||
**Audience:** administrators, operators
|
||||
|
||||
Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures.
|
||||
|
||||
It should cover:
|
||||
|
||||
- normal workflow;
|
||||
- filesystem layout;
|
||||
- remote storage layout, if applicable;
|
||||
- logs and manifests;
|
||||
- resume/retry behavior;
|
||||
- cleanup behavior;
|
||||
- archive/backup behavior;
|
||||
- safe recovery procedures;
|
||||
- operational caveats.
|
||||
|
||||
### docs/troubleshooting.md
|
||||
|
||||
**Audience:** administrators, operators
|
||||
|
||||
Recommended once recurring failure modes exist.
|
||||
|
||||
Each entry should include:
|
||||
|
||||
- symptom;
|
||||
- likely cause;
|
||||
- diagnostic command or inspection step;
|
||||
- safe fix;
|
||||
- relevant links.
|
||||
|
||||
### docs/development.md
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
Required for projects maintained by humans and LLM coding agents.
|
||||
|
||||
It should include:
|
||||
|
||||
- repository layout;
|
||||
- build/test commands;
|
||||
- coding conventions;
|
||||
- dependency policy;
|
||||
- how to add config fields;
|
||||
- how to add CLI flags;
|
||||
- how to add stages/modules/adapters, if applicable;
|
||||
- how to update examples;
|
||||
- documentation update expectations.
|
||||
|
||||
### docs/internal/
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
Required for modular, staged, service-oriented, or orchestration projects.
|
||||
|
||||
This directory describes implemented internal components. It is not the roadmap.
|
||||
|
||||
Use one file per major component where useful.
|
||||
|
||||
Each component doc should include:
|
||||
|
||||
1. purpose;
|
||||
2. inputs and outputs;
|
||||
3. boundaries;
|
||||
4. config fields used;
|
||||
5. external adapters used;
|
||||
6. state or manifest behavior, if applicable;
|
||||
7. skip/resume behavior, if applicable;
|
||||
8. failure behavior;
|
||||
9. tests to inspect before changing;
|
||||
10. architectural invariants.
|
||||
|
||||
### docs/roadmap/
|
||||
|
||||
**Audience:** maintainers, developers, LLM coding agents
|
||||
|
||||
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
|
||||
|
||||
Roadmap docs should clearly distinguish:
|
||||
|
||||
- proposed work;
|
||||
- accepted plans;
|
||||
- deferred ideas;
|
||||
- rejected ideas;
|
||||
- implementation prompts or task breakdowns, if useful.
|
||||
|
||||
Roadmap docs should not be confused with current behavior.
|
||||
|
||||
### docs/integrations/
|
||||
|
||||
**Audience:** developers, LLM coding agents
|
||||
|
||||
Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
|
||||
|
||||
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses.
|
||||
|
||||
Use one file per integration where useful.
|
||||
|
||||
## Examples Directory
|
||||
|
||||
Projects with non-trivial configuration or workflows should include `examples/`.
|
||||
|
||||
Useful examples include:
|
||||
|
||||
- minimal working config;
|
||||
- production-oriented config;
|
||||
- full annotated config;
|
||||
- local development config;
|
||||
- remote/object-storage config;
|
||||
- minimal session/input file.
|
||||
|
||||
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
|
||||
|
||||
## Security and Privacy
|
||||
|
||||
Docs and examples must not include:
|
||||
|
||||
- real API keys;
|
||||
- tokens;
|
||||
- passwords;
|
||||
- private keys;
|
||||
- private environment dumps;
|
||||
- sensitive user data;
|
||||
- raw private transcripts;
|
||||
- private infrastructure details unless intentionally public.
|
||||
|
||||
Document secret-handling mechanisms, not actual secret values.
|
||||
|
||||
## Maintenance Rules
|
||||
|
||||
When docs change, verify the affected behavior.
|
||||
|
||||
Where practical:
|
||||
|
||||
- load example config files in tests;
|
||||
- test CLI examples or command parser behavior;
|
||||
- validate documented flags against real flags;
|
||||
- remove stale references;
|
||||
- update links after renames;
|
||||
- keep roadmap content out of non-roadmap docs.
|
||||
|
||||
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
|
||||
|
||||
Documentation is complete only when it matches the current code.
|
||||
|
||||
## Documentation Change Checklist
|
||||
|
||||
Before merging documentation changes, verify:
|
||||
|
||||
- README is concise and orientation-focused.
|
||||
- `docs/architecture.md` describes development principles.
|
||||
- Future work appears only under `docs/roadmap/`.
|
||||
- User-facing docs avoid unnecessary internals.
|
||||
- Developer-facing docs preserve boundaries and invariants.
|
||||
- Config examples match the schema.
|
||||
- CLI examples match real commands and flags.
|
||||
- Defaults appear in the canonical config reference.
|
||||
- No secrets or private data are included.
|
||||
- Links are accurate.
|
||||
15
docs/integrations/README.md
Normal file
15
docs/integrations/README.md
Normal file
@@ -0,0 +1,15 @@
|
||||
# Integration Documentation Index
|
||||
|
||||
## Audience
|
||||
Developers and LLM coding agents changing Narratio's external integration contracts.
|
||||
|
||||
## Scope
|
||||
Implemented-only reference notes for the external systems Narratio currently integrates with.
|
||||
|
||||
## Integration Docs
|
||||
- `audita.md`: Audita adapter invocation and validation contract.
|
||||
- `seriatim.md`: Seriatim normalize/merge/trim adapter contract.
|
||||
- `scriptorium.md`: Scriptorium run/render adapter contract.
|
||||
|
||||
## Canonical Owner
|
||||
`docs/integrations/` is the canonical home for external integration reference notes per `docs/documentation/policy.md`.
|
||||
@@ -1,96 +1,66 @@
|
||||
# Audita Subprocess Operations
|
||||
# Integration: audita
|
||||
|
||||
This document describes how parent processes should invoke `audita process` safely in production orchestration.
|
||||
## Purpose
|
||||
Define Narratio's adapter contract for transcript polishing via Audita CLI subprocess execution.
|
||||
|
||||
## Recommended command form
|
||||
## Inputs and Outputs
|
||||
Inputs (`audita.PolishRequest`):
|
||||
- merged transcript path
|
||||
- glossary path
|
||||
- output processed transcript path
|
||||
- optional report path (required when report enabled)
|
||||
- work dir
|
||||
- generated config path
|
||||
- stdout/stderr log paths
|
||||
- optional module/model/base URL and concurrency knobs
|
||||
|
||||
Use explicit file outputs for orchestrated runs:
|
||||
Outputs (`audita.PolishResult`):
|
||||
- processed transcript path
|
||||
- optional report path
|
||||
- generated config path
|
||||
- stdout/stderr log paths
|
||||
- exit code, duration, invoked binary
|
||||
- adapter metadata
|
||||
|
||||
```sh
|
||||
audita process <transcript.json> \
|
||||
--transcript-description "Brief context that may help resolve ambiguous terms." \
|
||||
--glossary <glossary.yaml> \
|
||||
--output <output-transcript.json> \
|
||||
--report-json <report.json>
|
||||
```
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Deterministic CLI argument construction for `audita process`
|
||||
- Environment bridging for API credentials
|
||||
- Invocation config emission
|
||||
- Output validation for processed transcript and report
|
||||
|
||||
Additional flags that may be situationally appropriate:
|
||||
- `--config <path>` to select an explicit versioned config file.
|
||||
- `--output-schema <bare-segments|audita-v1>` to select transcript output shape.
|
||||
- `--work-dir <dir>` to control diagnostics location.
|
||||
- `--work-dir-retention <always|auto|never>` to control retained run directories.
|
||||
- `--total-llm-concurrency`, `--proposal-llm-concurrency`, and `--validation-llm-concurrency` when orchestration needs to set explicit LLM throughput controls.
|
||||
- `--modules ...` only when intentionally overriding the default sequence.
|
||||
Does not own:
|
||||
- Upstream/downstream stage orchestration
|
||||
- Credential sourcing policy beyond required env-var presence check
|
||||
|
||||
For config-driven orchestration, validate config files in CI/preflight:
|
||||
## Config Fields Used
|
||||
Via `pipeline.audita.*` mapped in app/stage wiring:
|
||||
- `binary`, `timeout`, `llm_api_key_env`, `modules`, `base_url`, `model`
|
||||
- `transcript_description`, `config_path`, `output_schema`, `work_dir_retention`
|
||||
- `total_llm_concurrency`, `proposal_llm_concurrency`, `validation_model`, `validation_llm_concurrency`, `report`
|
||||
|
||||
```sh
|
||||
audita config validate --config <path>
|
||||
```
|
||||
## External Adapters Used
|
||||
- Shared subprocess helper (`internal/adapters/subprocess`) to run CLI and capture logs.
|
||||
|
||||
## Stdout behavior
|
||||
## State and Manifest Behavior
|
||||
- No direct manifest writes.
|
||||
- Stage-level metadata records adapter provenance and credential-present signal.
|
||||
- Generated invocation YAML is written when `GeneratedConfigPath` is provided.
|
||||
|
||||
- With `--output`: stdout is expected to be empty on success.
|
||||
- Without `--output`: stdout contains transcript JSON only on success.
|
||||
- Report JSON is never written to stdout.
|
||||
## Skip and Resume Behavior
|
||||
- Adapter has no skip/resume logic. Stage/runner controls this.
|
||||
|
||||
## Stderr behavior
|
||||
## Failure Behavior
|
||||
- Constructor validation fails on invalid binary/timeout/schema/concurrency/URL values.
|
||||
- Run fails on missing required paths, missing required credential env var, subprocess errors, invalid processed JSON shape, or invalid report JSON.
|
||||
- Failures preserve stdout/stderr paths in returned result metadata.
|
||||
|
||||
- Success path should be quiet or minimal human-readable logs.
|
||||
- Failure path writes concise human-readable errors.
|
||||
- When a diagnostics run directory exists, failure stderr includes its path.
|
||||
- Prompt/response diagnostic payloads are not streamed to stderr.
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/adapters/audita/subprocess_test.go`
|
||||
- `internal/adapters/audita/fake_test.go`
|
||||
- `internal/stage/polish_test.go`
|
||||
|
||||
## Output file behavior
|
||||
|
||||
- `--output` writes transcript JSON in the selected output schema to the provided path.
|
||||
- Output write failures return nonzero and surface actionable errors.
|
||||
- The command does not silently ignore output write errors.
|
||||
|
||||
## Report JSON behavior
|
||||
|
||||
- `--report-json` writes a machine-readable process report to the requested path.
|
||||
- Run-directory `report.json` is written independently under diagnostics.
|
||||
- Best-effort failure reports are emitted when possible without masking the primary failure.
|
||||
- Report write failures return nonzero with clear stderr messaging.
|
||||
- Report diagnostics metadata references run-directory artifacts including utilization diagnostics and correction ledger paths when available.
|
||||
|
||||
## Diagnostics directory behavior
|
||||
|
||||
- Each run creates (when possible) a per-run diagnostics directory.
|
||||
- Typical artifacts include transcript, normalization, chunking, invocation, effective config, LLM diagnostics, `utilization-diagnostics.json`, `correction-ledger.json`, `report.json`, and `error.log` on failure.
|
||||
- Failed runs retain diagnostics.
|
||||
- Under `auto` retention, successful runs with skipped/rejected corrections are retained; clean successful runs may be removed.
|
||||
|
||||
## Exit codes
|
||||
|
||||
- `0`: success.
|
||||
- Nonzero: failure (input/schema/config/module/LLM/runtime/output/report/diagnostics errors).
|
||||
|
||||
Treat any nonzero as a failed subprocess invocation.
|
||||
|
||||
## Timeout and cancellation
|
||||
|
||||
- Runtime operations propagate context cancellation and request timeouts through LLM/scheduler paths.
|
||||
- On cancellation or timeout, the process exits nonzero and should not hang.
|
||||
- If diagnostics were initialized before failure, failure artifacts remain available for debugging.
|
||||
|
||||
## Secret redaction expectations
|
||||
|
||||
API keys and configured secret values are redacted from:
|
||||
- reports (`--report-json` and run-dir `report.json`);
|
||||
- diagnostics artifacts (including effective config and LLM interaction artifacts);
|
||||
- surfaced adapter/runtime errors;
|
||||
- test fixtures and regression outputs.
|
||||
|
||||
Parent-process logs should still avoid printing raw environment variables.
|
||||
|
||||
## Parent-process pipe guidance
|
||||
|
||||
To avoid deadlocks in orchestrators:
|
||||
- always read both stdout and stderr concurrently when invoking as a subprocess;
|
||||
- prefer file outputs (`--output`, `--report-json`) for machine workflows;
|
||||
- treat stderr as human-readable diagnostics, not structured data;
|
||||
- parse structured results from output/report files.
|
||||
|
||||
For Go callers, prefer `exec.CommandContext` with explicit timeout/cancellation and buffered/streamed readers for both pipes.
|
||||
## Architectural Invariants
|
||||
- Processed output must be valid JSON with top-level `segments` array.
|
||||
- When report is enabled, report output must be valid JSON.
|
||||
- If `llm_api_key_env` is configured, credential must be present in environment.
|
||||
|
||||
@@ -1,339 +1,64 @@
|
||||
# Narratio -> Scriptorium CLI Integration
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
This document defines how Narratio should invoke Scriptorium through the **public CLI**.
|
||||
|
||||
This is a **subprocess integration contract**, not an internal Go API contract.
|
||||
|
||||
## 2. Assumptions
|
||||
|
||||
- `scriptorium` is installed and available on `PATH`.
|
||||
- Scriptorium is configured with `config.yml`.
|
||||
- `config.yml` provides `prompt_dir`, `profile_dir`, and `schema_dir` as needed.
|
||||
- Prompt and profile libraries are already deployed for the environment.
|
||||
- Narratio provides prepared artifact files (for example polished transcript, glossary, previous recap, campaign notes).
|
||||
- Initial integration is synchronous subprocess execution.
|
||||
- Narratio remains the orchestrator.
|
||||
|
||||
In normal operation, Narratio does not need to pass `--prompt-dir` and `--profile-dir` if they are supplied by Scriptorium config.
|
||||
|
||||
Narratio may pass `--config <PATH>` when it must use a non-default Scriptorium config file.
|
||||
|
||||
## 3. Core Commands Narratio May Call
|
||||
|
||||
Primary commands for subprocess integration:
|
||||
|
||||
- `scriptorium run`
|
||||
- `scriptorium render`
|
||||
|
||||
For production generation, use `scriptorium run`.
|
||||
|
||||
`scriptorium render` is for debugging, dry-runs, test assertions, and validating command construction without LLM execution.
|
||||
|
||||
Note: `scriptorium serve` and HTTP API exist, but they are not the initial integration path.
|
||||
|
||||
## 4. Command Selection Guidance
|
||||
|
||||
- Use `run` to generate an output artifact.
|
||||
- Use `render` to inspect the prepared prompt and effective settings without calling the LLM.
|
||||
- Use `render --format json` when Narratio/tests need structured prepare output.
|
||||
|
||||
## 5. Recommended `run` Invocation Shape
|
||||
|
||||
Production shape:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt <prompt_id> \
|
||||
--input transcript=<processed-transcript-path> \
|
||||
--out <output-artifact-path>
|
||||
```
|
||||
|
||||
Common optional additions:
|
||||
|
||||
- `--config <path>`: use a specific Scriptorium config file.
|
||||
- `--profile <profile_id>`: override prompt default profile.
|
||||
- `--var name=value` (repeatable): small metadata values.
|
||||
- `--input name=path` (repeatable): additional named artifacts.
|
||||
- `--timeout <duration>`: per-run timeout override.
|
||||
- Runtime model override flags (`--llm-base-url`, `--model`, etc.) only for exceptional/operator-directed cases.
|
||||
|
||||
## 6. Recommended `render` Invocation Shape
|
||||
|
||||
Human-readable debug shape:
|
||||
|
||||
```bash
|
||||
scriptorium render \
|
||||
--prompt <prompt_id> \
|
||||
--input transcript=<processed-transcript-path> \
|
||||
--format text
|
||||
```
|
||||
|
||||
Structured debug/test shape:
|
||||
|
||||
```bash
|
||||
scriptorium render \
|
||||
--prompt <prompt_id> \
|
||||
--input transcript=<processed-transcript-path> \
|
||||
--format json \
|
||||
--out <render-debug-path>
|
||||
```
|
||||
|
||||
`render` does **not** call the LLM, does **not** validate model output, and does **not** perform repair.
|
||||
|
||||
## 7. Inputs
|
||||
|
||||
- Pass inputs as repeated `--input name=path` flags.
|
||||
- `name` must match the Prompt Definition input name.
|
||||
- Prefer absolute paths, or paths relative to a working directory controlled by Narratio.
|
||||
- Pass Audita output as the primary transcript input.
|
||||
- Additional inputs may include glossary, previous recap, campaign notes, event logs, final state maps, or other prompt-specific artifacts.
|
||||
- Scriptorium reads input files directly; Narratio does not need to inline file content for CLI use.
|
||||
|
||||
## 8. Variables
|
||||
|
||||
Use repeated `--var name=value` for small metadata values.
|
||||
|
||||
Typical examples:
|
||||
|
||||
- `session_date`
|
||||
- `session_id`
|
||||
- `campaign_name`
|
||||
- `previous_session_id`
|
||||
- `output_kind`
|
||||
|
||||
Large content belongs in input files, not `--var` values.
|
||||
|
||||
## 9. Prompt IDs and Output Artifact Types
|
||||
|
||||
Narratio should treat prompt IDs as configuration, not hardcoded business logic.
|
||||
|
||||
Narratio config may map stage/output names to prompt IDs, for example:
|
||||
|
||||
- session recap prompt
|
||||
- structured event extraction prompt
|
||||
- glossary suggestion prompt
|
||||
- player-facing summary prompt
|
||||
|
||||
Prompt IDs used by Narratio should come from the deployed Scriptorium prompt library.
|
||||
|
||||
## 10. Profiles
|
||||
|
||||
- Prompts may declare `default_profile`.
|
||||
- Narratio may omit `--profile` to use prompt default profile.
|
||||
- Narratio may pass `--profile` to force profile selection.
|
||||
- This enables environment/profile selection like `local-fast`, `local-quality`, `frontier`, `batch`, or test profiles.
|
||||
- Profile names should generally be Narratio configuration values.
|
||||
|
||||
## 11. Runtime Overrides
|
||||
|
||||
Supported runtime override flags:
|
||||
|
||||
- `--llm-base-url`
|
||||
- `--model`
|
||||
- `--api-key-env`
|
||||
- `--temperature`
|
||||
- `--max-tokens`
|
||||
- `--top-p`
|
||||
- `--timeout`
|
||||
|
||||
Guidance:
|
||||
|
||||
- Keep normal model/runtime settings in Execution Profiles.
|
||||
- Use runtime overrides only for explicit per-run exceptions, tests, or operator overrides.
|
||||
- Never pass raw API keys on the command line.
|
||||
- `--api-key-env` names an environment variable; Narratio must ensure that variable is set in subprocess environment.
|
||||
|
||||
## 12. Config Behavior
|
||||
|
||||
- Default config path: `/etc/scriptorium/config.yml`.
|
||||
- `--config <PATH>` overrides default path.
|
||||
- Missing default config is allowed by Scriptorium.
|
||||
- If `--config` is provided explicitly, the file must exist and be valid.
|
||||
- CLI flags override `config.yml`.
|
||||
- `config.yml` overrides built-in application defaults.
|
||||
|
||||
Narratio can either:
|
||||
|
||||
- rely on system default config path, or
|
||||
- carry an explicit config path and pass `--config`.
|
||||
|
||||
## 13. Environment Handling
|
||||
|
||||
Subprocess environment recommendations:
|
||||
|
||||
- Pass through required API-key environment variables referenced by `api_key_env`.
|
||||
- Do not pass raw API keys as CLI arguments.
|
||||
- Avoid logging full environment dumps.
|
||||
- Capture stdout and stderr separately.
|
||||
- Use a controlled working directory.
|
||||
- Prefer absolute artifact paths.
|
||||
|
||||
## 14. Output Handling
|
||||
|
||||
For `scriptorium run`:
|
||||
|
||||
- Use `--out` when Narratio needs durable artifact files.
|
||||
- Without `--out`, artifact content is written to stdout.
|
||||
- Preferred orchestration pattern: always use `--out`, then treat the file as stage output artifact.
|
||||
- Capture stderr for diagnostics.
|
||||
|
||||
For `scriptorium render`:
|
||||
|
||||
- Use `--out` to store render diagnostics.
|
||||
- Use `--format json` when tests need to inspect selected profile, effective runtime settings, input hashes, prompt hash, and rendered messages.
|
||||
|
||||
## 15. Exit Status and Errors
|
||||
|
||||
Current CLI behavior (verified from implementation/tests):
|
||||
|
||||
- `0`: success.
|
||||
- `1`: runtime/parse/config/load/render/generation/IO error.
|
||||
- `2`: run completed but output validation failed (`ValidationFailed`).
|
||||
|
||||
Additional details:
|
||||
|
||||
- On `run`, output artifact write happens before exit code selection. If validation fails, artifact may still be written and exit code is `2`.
|
||||
- `stderr` carries both errors and normal run summary output; non-empty stderr alone does not imply failure.
|
||||
- `render` returns `0` on success and `1` on failures.
|
||||
|
||||
Narratio should treat non-zero exit codes as failed stage execution, but may record generated artifact paths if a run exited `2` and output file exists.
|
||||
|
||||
## 16. Recommended Narratio Integration Pattern
|
||||
|
||||
1. Build CLI args from Narratio stage configuration.
|
||||
2. Use subprocess context cancellation/timeout.
|
||||
3. Pass absolute input paths.
|
||||
4. Pass `--out` to a session-scoped artifact path.
|
||||
5. Add `--var` metadata values.
|
||||
6. Optionally add `--config`.
|
||||
7. Optionally add `--profile`.
|
||||
8. Ensure required API-key env vars are present.
|
||||
9. Run subprocess synchronously.
|
||||
10. Capture stdout/stderr separately.
|
||||
11. On success, store output artifact path and invocation metadata in stage artifacts.
|
||||
12. On failure, store exit code and stderr diagnostics in stage status.
|
||||
|
||||
## 17. Suggested Narratio Configuration Shape
|
||||
|
||||
Illustrative `pipeline.yml` shape:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
config_path: /etc/scriptorium/config.yml
|
||||
timeout: 10m
|
||||
render_debug: false
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
profile_id: local-quality # optional
|
||||
output_path: artifacts/session_recap.md
|
||||
timeout: 10m
|
||||
render_debug: false # optional artifact override
|
||||
inputs:
|
||||
transcript:
|
||||
source: trimmed_transcript
|
||||
required: true
|
||||
previous_recap:
|
||||
source: previous_session_artifact
|
||||
artifact: session_recap
|
||||
path: "" # optional
|
||||
required: false
|
||||
vars:
|
||||
session_id: true
|
||||
session_date: true
|
||||
campaign_name: true
|
||||
previous_session_id: true
|
||||
output_kind: session_recap
|
||||
```
|
||||
|
||||
The key idea: map Narratio artifact names to prompt ID, optional profile, expected inputs, vars, and output destination.
|
||||
|
||||
## 18. Testing Strategy for Narratio Integration
|
||||
|
||||
- Use `scriptorium render --format json` to verify command construction without LLM calls.
|
||||
- Use dedicated test prompt/profile libraries for integration tests.
|
||||
- Use small fixture transcripts.
|
||||
- Verify missing-input failure behavior.
|
||||
- Verify prompt `default_profile` behavior.
|
||||
- Verify explicit `--profile` override behavior.
|
||||
- Verify `--config` behavior (default and explicit).
|
||||
- Verify output file creation when `--out` is used.
|
||||
- Verify stderr capture on failures.
|
||||
- Avoid real API keys in tests.
|
||||
|
||||
## 19. Security and Privacy Notes
|
||||
|
||||
- Never pass raw API keys on command line.
|
||||
- Do not log full rendered prompts by default; transcripts may contain sensitive content.
|
||||
- Avoid logging prompt content unless explicit debug mode is enabled.
|
||||
- Treat generated artifacts as potentially sensitive.
|
||||
- Use session-scoped, access-controlled output paths.
|
||||
- `api_key_env` names should come from environment management, not embedded secrets.
|
||||
|
||||
## 20. Initial D&D Artifact Generation Examples
|
||||
|
||||
These are examples only. Use prompt IDs from the deployed prompt library.
|
||||
|
||||
Session recap:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.session_recap \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--input glossary=/work/session-42/glossary.yml \
|
||||
--out /work/session-42/artifacts/session_recap.md
|
||||
```
|
||||
|
||||
Structured events:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.structured_events \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--out /work/session-42/artifacts/structured_events.json
|
||||
```
|
||||
|
||||
Glossary suggestions:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.glossary_suggestions \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--input previous_recap=/work/session-41/artifacts/session_recap.md \
|
||||
--out /work/session-42/artifacts/glossary_suggestions.md
|
||||
```
|
||||
|
||||
Player-facing summary:
|
||||
|
||||
```bash
|
||||
scriptorium run \
|
||||
--prompt dnd.player_summary \
|
||||
--input transcript=/work/session-42/transcript.polished.md \
|
||||
--input structured_events=/work/session-42/artifacts/structured_events.json \
|
||||
--out /work/session-42/artifacts/player_summary.md
|
||||
```
|
||||
|
||||
## 21. Non-Goals
|
||||
|
||||
Initial Narratio integration should not:
|
||||
|
||||
- call Scriptorium internal Go packages
|
||||
- use HTTP API as the primary path
|
||||
- expect Scriptorium to read S3 refs directly
|
||||
- make Scriptorium responsible for Narratio stage state
|
||||
- make Scriptorium responsible for notification
|
||||
- require Scriptorium to understand D&D workflow semantics beyond prompt definitions
|
||||
|
||||
## 22. Future Extension Notes
|
||||
|
||||
Possible later extensions:
|
||||
|
||||
- HTTP API integration
|
||||
- S3 artifact references if Scriptorium adds S3 reader support
|
||||
- richer render diagnostics and policy controls
|
||||
- token budgeting/prompt-size checks
|
||||
- batch execution if Scriptorium later adds batch support
|
||||
# Integration: scriptorium
|
||||
|
||||
## Purpose
|
||||
Define Narratio's adapter contract for Scriptorium artifact generation and render-debug subprocess invocations.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `RunArtifactRequest`: binary, config path, prompt/profile IDs, input map, vars map, timeout, output path, logs/config paths, optional API env and working dir
|
||||
- `RenderArtifactRequest`: same core fields for render mode
|
||||
|
||||
Outputs (`ArtifactResult`):
|
||||
- output path
|
||||
- stdout/stderr log paths
|
||||
- generated config path
|
||||
- exit code and duration
|
||||
- command mode (`run` or `render`)
|
||||
- prompt/profile provenance
|
||||
- validation failure signal
|
||||
- adapter metadata
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Deterministic CLI arg construction for `scriptorium run` and `scriptorium render`
|
||||
- Common request validation
|
||||
- Invocation config emission
|
||||
- Output existence/non-empty checks
|
||||
- Validation-failure mapping for run exit code 2
|
||||
|
||||
Does not own:
|
||||
- Artifact selection policy (`analyze` stage)
|
||||
- Bounds semantic validation (`trim` stage)
|
||||
|
||||
## Config Fields Used
|
||||
Via `pipeline.scriptorium.*` and stage-level artifact config:
|
||||
- `binary`, `config_path`, `timeout`, `render_debug`
|
||||
- artifact-level `prompt_id`, `profile_id`, `timeout`, `inputs`, `vars`, `output_path`
|
||||
|
||||
## External Adapters Used
|
||||
- Shared subprocess helper (`internal/adapters/subprocess`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- No direct manifest writes.
|
||||
- Stage metadata records adapter outputs and command mode.
|
||||
- Generated invocation YAML is written when requested.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Adapter has no skip/resume logic. Stage/runner controls execution.
|
||||
|
||||
## Failure Behavior
|
||||
- Request validation fails for missing binary/prompt/output, invalid timeout, invalid input/var names, or missing required API env var.
|
||||
- Subprocess errors bubble with command context.
|
||||
- `run` exit code 2 is treated as `ValidationFailed=true` and surfaced as error by calling stage.
|
||||
- Successful subprocess still fails if output file is missing/empty.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
- `internal/adapters/scriptorium/fake_test.go`
|
||||
- `internal/stage/analyze_test.go`
|
||||
- `internal/stage/trim_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Both modes require explicit timeout > 0.
|
||||
- Input/var maps are sorted into deterministic CLI argument order.
|
||||
- Run-mode validation failures are represented explicitly, not silently skipped.
|
||||
|
||||
@@ -1,403 +1,60 @@
|
||||
# seriatim
|
||||
|
||||
`seriatim` merges per-speaker WhisperX-style JSON transcripts into a single JSON transcript that preserves speaker identity and chronological order.
|
||||
|
||||
The current implementation supports the `merge` command. It reads one or more input JSON files, optionally maps each input file to a canonical speaker using `speakers.yml`, sorts all segments by timestamp, detects and resolves overlaps when word-level timing is available, assigns consecutive numeric `id` values, and writes a merged JSON artifact.
|
||||
|
||||
## Usage
|
||||
|
||||
Run from source:
|
||||
|
||||
```sh
|
||||
go run ./cmd/seriatim merge \
|
||||
--input-file samples/raw/2026-04-19-Eric_Rakestraw.json \
|
||||
--input-file samples/raw/2026-04-19-Mike_Brown.json \
|
||||
--output-file merged.json
|
||||
```
|
||||
|
||||
Optional report output:
|
||||
|
||||
```sh
|
||||
go run ./cmd/seriatim merge \
|
||||
--input-file eric.json \
|
||||
--input-file mike.json \
|
||||
--output-file merged.json \
|
||||
--report-file report.json
|
||||
```
|
||||
|
||||
## CLI
|
||||
|
||||
```text
|
||||
seriatim merge [flags]
|
||||
```
|
||||
|
||||
Global flags:
|
||||
|
||||
| Flag | Description |
|
||||
| --- | --- |
|
||||
| `--help` | Show command help. |
|
||||
| `--version` | Show application version. Local builds default to `dev`; release builds inject the release version. |
|
||||
|
||||
`merge` flags:
|
||||
|
||||
| Flag | Required | Default | Description |
|
||||
| --- | --- | --- | --- |
|
||||
| `--input-file` | Yes | none | Input transcript JSON file. Repeat once per speaker/input file. |
|
||||
| `--output-file` | Yes | none | Merged transcript JSON output path. |
|
||||
| `--report-file` | No | none | Optional report JSON output path. |
|
||||
| `--speakers` | No | none | Speaker map YAML file. When omitted, input file basenames are used as speaker labels. |
|
||||
| `--autocorrect` | No | none | Autocorrect rules YAML file. When omitted, the default `autocorrect` module leaves text unchanged. |
|
||||
| `--input-reader` | No | `json-files` | Input reader module. |
|
||||
| `--output-modules` | No | `json` | Comma-separated output modules. |
|
||||
| `--output-schema` | No | `seriatim-intermediate` | JSON output contract. Allowed values are `seriatim-minimal`, `seriatim-intermediate`, and `seriatim-full`. If omitted, the runtime default is used; consumers that depend on a specific shape should set this explicitly. |
|
||||
| `--preprocessing-modules` | No | `validate-raw,normalize-speakers,trim-text` | Comma-separated preprocessing modules, evaluated in order. |
|
||||
| `--postprocessing-modules` | No | `detect-overlaps,resolve-overlaps,backchannel,filler,resolve-danglers,coalesce,detect-overlaps,autocorrect,assign-ids,validate-output` | Comma-separated postprocessing modules, evaluated in order. |
|
||||
| `--coalesce-gap` | No | `3.0` | Maximum same-speaker gap in seconds for `coalesce`; also used as the `resolve-overlaps` context window. Must be a non-negative float. |
|
||||
|
||||
Environment variables:
|
||||
|
||||
| Environment Variable | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `SERIATIM_OUTPUT_SCHEMA` | `seriatim-intermediate` | Output schema used when `--output-schema` is not explicitly provided. Allowed values are `seriatim-minimal`, `seriatim-intermediate`, and `seriatim-full`. The CLI flag takes precedence. |
|
||||
| `SERIATIM_OVERLAP_WORD_RUN_GAP` | `1.0` | Maximum gap in seconds between adjacent timed words when `resolve-overlaps` builds word-run replacement segments. Must be a positive float. |
|
||||
| `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW` | `1.0` | Near-start window in seconds for ordering replacement word runs shortest-first. Must be a positive float. |
|
||||
| `SERIATIM_BACKCHANNEL_MAX_DURATION` | `2.0` | Maximum duration in seconds for `backchannel` classification. Must be a positive float. |
|
||||
| `SERIATIM_FILLER_MAX_DURATION` | `1.25` | Maximum duration in seconds for `filler` classification. Must be a positive float. |
|
||||
|
||||
## Input JSON Format
|
||||
|
||||
Each input file must be valid JSON with a top-level `segments` array. The current parser accepts the WhisperX segment subset needed for merging:
|
||||
|
||||
```json
|
||||
{
|
||||
"segments": [
|
||||
{
|
||||
"start": 1.25,
|
||||
"end": 3.5,
|
||||
"text": "Hello there.",
|
||||
"words": [
|
||||
{"word": "Hello", "start": 1.25, "end": 1.55, "score": 0.98},
|
||||
{"word": "there.", "start": 1.7, "end": 2.0}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Required segment fields:
|
||||
|
||||
- `start`: number, must be `>= 0`.
|
||||
- `end`: number, must be `>= start`.
|
||||
- `text`: string.
|
||||
|
||||
Optional word fields:
|
||||
|
||||
- `words`: array of word timing objects.
|
||||
- `words[].word`: string.
|
||||
- `words[].start`: optional number, must be `>= 0` when present.
|
||||
- `words[].end`: optional number, must be `>= start` when present with `start`.
|
||||
- `words[].score`: optional number.
|
||||
- `words[].speaker`: optional raw speaker label string.
|
||||
|
||||
Word-level timing is preserved internally for overlap resolution. If a word is missing `start` or `end`, seriatim keeps the word text, emits a warning in the optional report, and does not use that word as a timing anchor. Word timing is not emitted in the final JSON artifact.
|
||||
|
||||
## Speaker Map Format
|
||||
|
||||
`speakers.yml` maps input files to canonical speaker names using ordered substring rules:
|
||||
|
||||
This file is optional. If `--speakers` is omitted, `seriatim` uses each input file basename as the segment speaker label.
|
||||
|
||||
```yaml
|
||||
match:
|
||||
- speaker: "Eric Rakestraw"
|
||||
match:
|
||||
- "Eric_Rakestraw"
|
||||
- "Eric"
|
||||
|
||||
- speaker: "Mike Brown"
|
||||
match:
|
||||
- "Mike_Brown"
|
||||
- "mb"
|
||||
```
|
||||
|
||||
For each `--input-file`, `seriatim` takes the file basename and evaluates the rules in order. The first rule with a matching substring wins, and no later rules are evaluated.
|
||||
|
||||
For example, this input:
|
||||
|
||||
```text
|
||||
samples/raw/2026-04-19-Eric_Rakestraw.json
|
||||
```
|
||||
|
||||
matches this rule because the basename contains `Eric_Rakestraw`:
|
||||
|
||||
```yaml
|
||||
- speaker: "Eric Rakestraw"
|
||||
match:
|
||||
- "Eric_Rakestraw"
|
||||
```
|
||||
|
||||
Important details:
|
||||
|
||||
- Matching is against the input file basename, not the full path.
|
||||
- Matching is case-insensitive.
|
||||
- Rules are evaluated from first to last.
|
||||
- Each rule must have a non-empty `speaker`.
|
||||
- Each rule must have at least one non-empty `match` string.
|
||||
- Duplicate speaker names are invalid.
|
||||
- Every input file must match at least one rule or the command fails.
|
||||
|
||||
Deprecated old format:
|
||||
|
||||
```yaml
|
||||
inputs:
|
||||
eric.json:
|
||||
speaker: "Eric Rakestraw"
|
||||
```
|
||||
|
||||
The old `inputs:` direct mapping format is no longer supported.
|
||||
|
||||
## Output JSON Format
|
||||
|
||||
`--output-modules json` controls the writer. `--output-schema` controls the JSON contract that writer serializes.
|
||||
|
||||
The named schemas are stable public contracts. If a consumer depends on a specific shape, it should request that schema explicitly at runtime. The runtime default selection may change in a future release.
|
||||
|
||||
The `seriatim-intermediate` schema is the current default selection when neither `--output-schema` nor `SERIATIM_OUTPUT_SCHEMA` is set. It stays close to the minimal schema, but adds optional `categories` on each segment:
|
||||
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"application": "seriatim",
|
||||
"version": "dev",
|
||||
"output_schema": "seriatim-intermediate"
|
||||
},
|
||||
"segments": [
|
||||
{
|
||||
"id": 1,
|
||||
"start": 1.25,
|
||||
"end": 3.5,
|
||||
"speaker": "Eric Rakestraw",
|
||||
"text": "Hello there.",
|
||||
"categories": ["backchannel"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The `seriatim-full` schema uses the full seriatim envelope:
|
||||
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"application": "seriatim",
|
||||
"version": "dev",
|
||||
"input_reader": "json-files",
|
||||
"input_files": ["eric.json", "mike.json"],
|
||||
"preprocessing_modules": ["validate-raw", "normalize-speakers", "trim-text"],
|
||||
"postprocessing_modules": ["detect-overlaps", "resolve-overlaps", "backchannel", "filler", "resolve-danglers", "coalesce", "detect-overlaps", "autocorrect", "assign-ids", "validate-output"],
|
||||
"output_modules": ["json"]
|
||||
},
|
||||
"segments": [
|
||||
{
|
||||
"id": 1,
|
||||
"source": "eric.json",
|
||||
"source_segment_index": 0,
|
||||
"speaker": "Eric Rakestraw",
|
||||
"start": 1.25,
|
||||
"end": 3.5,
|
||||
"text": "Hello there.",
|
||||
"overlap_group_id": 1
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"source": "eric.json",
|
||||
"source_ref": "word-run:1:1:1",
|
||||
"derived_from": ["eric.json#0"],
|
||||
"speaker": "Eric Rakestraw",
|
||||
"start": 2.0,
|
||||
"end": 2.5,
|
||||
"text": "Resolved word run",
|
||||
"categories": ["backchannel"]
|
||||
}
|
||||
],
|
||||
"overlap_groups": [
|
||||
{
|
||||
"id": 1,
|
||||
"start": 1.25,
|
||||
"end": 4.0,
|
||||
"segments": ["eric.json#0", "mike.json#0"],
|
||||
"speakers": ["Eric Rakestraw", "Mike Brown"],
|
||||
"class": "unknown",
|
||||
"resolution": "unresolved"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The `seriatim-minimal` schema emits minimal metadata and compact ordered segments:
|
||||
|
||||
```json
|
||||
{
|
||||
"metadata": {
|
||||
"application": "seriatim",
|
||||
"version": "dev",
|
||||
"output_schema": "seriatim-minimal"
|
||||
},
|
||||
"segments": [
|
||||
{
|
||||
"id": 1,
|
||||
"start": 1.25,
|
||||
"end": 3.5,
|
||||
"speaker": "Eric Rakestraw",
|
||||
"text": "Hello there."
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Minimal output intentionally omits categories, overlap groups, source/provenance fields, and pipeline configuration metadata.
|
||||
|
||||
Intermediate output intentionally omits overlap groups and source/provenance fields, but keeps optional `categories` and minimal metadata.
|
||||
|
||||
Segments are sorted deterministically by:
|
||||
|
||||
```text
|
||||
(start, end, source, source_segment_index/source_ref, speaker)
|
||||
```
|
||||
|
||||
Final segment IDs are assigned after sorting and start at `1`.
|
||||
|
||||
The public Go output contract is available from:
|
||||
|
||||
```go
|
||||
import "gitea.maximumdirect.net/eric/seriatim/schema"
|
||||
```
|
||||
|
||||
The same package embeds machine-readable JSON Schemas in `schema/full-output.schema.json`, `schema/intermediate-output.schema.json`, and `schema/minimal-output.schema.json`. The default `validate-output` postprocessor validates the selected output shape and verifies final segment IDs are present, sequential, and start at `1`.
|
||||
|
||||
## Overlap Detection
|
||||
|
||||
The default postprocessing pipeline detects overlapping segment groups.
|
||||
|
||||
Overlap behavior:
|
||||
|
||||
- A strict timing overlap is required: `next.start < current_group_end`.
|
||||
- Segments that only touch at a boundary are not grouped.
|
||||
- Groups require at least two distinct speakers.
|
||||
- Transitive overlaps are grouped together.
|
||||
- Segments in detected groups receive `overlap_group_id`.
|
||||
- `overlap_groups[].segments` contains stable references in `source#source_segment_index` format.
|
||||
- `class` is currently `unknown`.
|
||||
- `resolution` is `unresolved` until `resolve-overlaps` replaces the group.
|
||||
|
||||
## Overlap Resolution
|
||||
|
||||
The default postprocessing pipeline runs `detect-overlaps`, then `resolve-overlaps`, then `backchannel`, then `filler`, then `resolve-danglers`, then `coalesce`, then a second `detect-overlaps` pass.
|
||||
|
||||
For each detected overlap group, `resolve-overlaps` uses preserved WhisperX word timing to build smaller word-run replacement segments:
|
||||
|
||||
- The resolution window expands the detected overlap group by `--coalesce-gap` seconds on both sides.
|
||||
- Nearby same-speaker context segments are included when they intersect the expanded window and their start or end is within `--coalesce-gap` of the original overlap boundary.
|
||||
- Once a segment is selected for replacement, all timed words from that segment participate in word-run construction; the window controls segment selection, not per-word clipping.
|
||||
- Context segments that are part of another detected overlap group are not pulled into the current group.
|
||||
- Untimed words are included in replacement text in original word order when nearby timed words create a replacement run.
|
||||
- Untimed words do not affect replacement segment start/end times or word-run gap splitting.
|
||||
- Words for the same speaker are merged into one run when the gap between adjacent words is no greater than `SERIATIM_OVERLAP_WORD_RUN_GAP`.
|
||||
- The default word-run gap is `1.0` seconds.
|
||||
- Set `SERIATIM_OVERLAP_WORD_RUN_GAP` to a positive number of seconds to override the default.
|
||||
- Near-start replacement word runs are reordered so shorter segments come first when adjacent starts are within `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW`.
|
||||
- The default word-run reorder window is `1.0` seconds.
|
||||
- Set `SERIATIM_OVERLAP_WORD_RUN_REORDER_WINDOW` to a positive number of seconds to override the default.
|
||||
- Replacement segment text is built by joining word text with single spaces.
|
||||
- Replacement segments include `source_ref` and `derived_from`.
|
||||
- Replacement segments omit `source_segment_index` because they are derived from one or more original segments.
|
||||
- Resolved overlap groups are removed before the second detection pass.
|
||||
- Replacement segments are left without `overlap_group_id` until the second detection pass annotates any remaining overlap.
|
||||
- If a speaker has no usable word timing in a group, that speaker's original segment is kept.
|
||||
- If no speakers in a group have usable word timing, the original group and annotations remain unchanged.
|
||||
|
||||
## Backchannels
|
||||
|
||||
The default pipeline runs `backchannel` before `coalesce`. It tags short acknowledgement segments with:
|
||||
|
||||
```json
|
||||
"categories": ["backchannel"]
|
||||
```
|
||||
|
||||
Backchannel matching is case-insensitive, ignores punctuation for matching and word-count purposes, trims surrounding whitespace, and requires a matching acknowledgement phrase, no more than three whitespace-delimited words, and duration no greater than `SERIATIM_BACKCHANNEL_MAX_DURATION` seconds. The default maximum duration is `2.0` seconds.
|
||||
|
||||
## Fillers
|
||||
|
||||
The default pipeline runs `filler` after `backchannel` and before `coalesce`. It tags short filler utterances with:
|
||||
|
||||
```json
|
||||
"categories": ["filler"]
|
||||
```
|
||||
|
||||
Filler matching is case-insensitive, ignores punctuation for matching and word-count purposes, trims surrounding whitespace, and requires only filler tokens such as `um`, `uh`, `er`, `erm`, `ah`, `eh`, `hmm`, `mm`, or repeated combinations of those tokens. Matching segments must contain no more than three whitespace-delimited words and have duration no greater than `SERIATIM_FILLER_MAX_DURATION` seconds. The default maximum duration is `1.25` seconds.
|
||||
|
||||
## Dangler Resolution
|
||||
|
||||
The default pipeline runs `resolve-danglers` before `coalesce` and before the second overlap detection pass. It repairs short derived fragments when they share provenance with a nearby segment:
|
||||
|
||||
- Dangling-end fragments have no more than two words and end in punctuation.
|
||||
- Dangling-start fragments have no more than two words.
|
||||
- Matching uses same-speaker segments with any shared `derived_from` value.
|
||||
- Merged segments use `source_ref` values such as `resolve-danglers:1`, keep the target segment's transcript position, and union `derived_from`.
|
||||
|
||||
## Coalescing
|
||||
|
||||
The default pipeline runs `coalesce` after `resolve-danglers` and before the second overlap detection pass. It merges adjacent same-speaker segments in the transcript's current order when `next.start - current.end <= --coalesce-gap`.
|
||||
|
||||
Coalesced segments use `source_ref` values such as `coalesce:1`, include `derived_from`, and omit `source_segment_index`.
|
||||
|
||||
Different-speaker backchannel and filler segments do not block coalescing of surrounding same-speaker segments. Same-speaker backchannel and filler segments are merged normally when they are within `--coalesce-gap`. When same-speaker segments are coalesced, any `backchannel` or `filler` category from the merged inputs is dropped from the coalesced segment.
|
||||
|
||||
## Autocorrect
|
||||
|
||||
Autocorrect is included in the default postprocessing pipeline. If `--autocorrect` is omitted, the module leaves transcript text unchanged and records a skip event in the optional report.
|
||||
|
||||
Enable corrections by passing `--autocorrect`:
|
||||
|
||||
```sh
|
||||
go run ./cmd/seriatim merge \
|
||||
--input-file input.json \
|
||||
--autocorrect autocorrect.yml \
|
||||
--output-file merged.json
|
||||
```
|
||||
|
||||
`autocorrect.yml` format:
|
||||
|
||||
```yaml
|
||||
autocorrect:
|
||||
- target: "Hrank"
|
||||
match:
|
||||
- "hrank"
|
||||
- "Frank"
|
||||
|
||||
- target: "Mike Brown"
|
||||
match:
|
||||
- "Mike Pat"
|
||||
```
|
||||
|
||||
Matching behavior:
|
||||
|
||||
- Matching is case-sensitive.
|
||||
- Matches apply only to whole tokens, not substrings inside larger words.
|
||||
- Punctuation and whitespace can surround a match.
|
||||
- Multi-word and hyphenated matches are supported.
|
||||
- Duplicate match strings are invalid, including duplicates across separate rules.
|
||||
|
||||
## Current Limitations
|
||||
|
||||
- Only JSON input is supported.
|
||||
- Overlap resolution depends on WhisperX word timing; groups without usable word timing remain unresolved.
|
||||
- Alternate output formats are not implemented yet.
|
||||
|
||||
## Release Builds
|
||||
|
||||
Local builds record version metadata as `dev`. Release builds should inject the release version with `ldflags`:
|
||||
|
||||
```sh
|
||||
go build -ldflags "-X gitea.maximumdirect.net/eric/seriatim/internal/buildinfo.Version=v1.0.0" ./cmd/seriatim
|
||||
```
|
||||
# Integration: seriatim
|
||||
|
||||
## Purpose
|
||||
Define Narratio's adapter contract for merge, normalize, and trim subprocess invocations of Seriatim.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `MergeRequest`: raw/normalized transcript inputs, output path, optional report, speaker/autocorrect paths, logs/config
|
||||
- `NormalizeRequest`: input transcript, output path, schema, optional report, timeout/log/config
|
||||
- `TrimRequest`: input transcript, output path, keep selector, timeout/log/config
|
||||
|
||||
Outputs:
|
||||
- `MergeResult`, `NormalizeResult`, `TrimResult` with output paths, logs/config paths, exit code, duration, binary provenance, and metadata.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Validated deterministic CLI invocation construction
|
||||
- Optional env tuning propagation for merge
|
||||
- Invocation config file emission
|
||||
- JSON output validation
|
||||
|
||||
Does not own:
|
||||
- Transcript input selection/promotion logic (stage-owned)
|
||||
- Bounds computation (scriptorium/trim-stage-owned)
|
||||
|
||||
## Config Fields Used
|
||||
Via `pipeline.seriatim.*` mapped in app/stage wiring:
|
||||
- `binary`, `timeout`, `output_schema`, `coalesce_gap`, `report`
|
||||
- `env.overlap_word_run_gap`
|
||||
- `env.overlap_word_run_reorder_window`
|
||||
- `env.backchannel_max_duration`
|
||||
- `env.filler_max_duration`
|
||||
|
||||
## External Adapters Used
|
||||
- Shared subprocess helper (`internal/adapters/subprocess`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- No direct manifest writes.
|
||||
- Stage metadata consumes adapter result fields and preserves generated config/log references.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Adapter has no skip/resume logic. Runner controls stage execution.
|
||||
|
||||
## Failure Behavior
|
||||
- Constructor fails for invalid binary/timeout/output-schema/coalesce-gap.
|
||||
- Merge fails on missing output path/inputs/report path (if enabled), subprocess errors, invalid merged output JSON, invalid report JSON.
|
||||
- Normalize fails on missing input/output, invalid schema, subprocess errors, invalid normalized output JSON shape, invalid report JSON.
|
||||
- Trim fails on missing input/output/keep selector, subprocess errors, invalid trimmed output JSON shape.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
- `internal/adapters/seriatim/fake_test.go`
|
||||
- `internal/stage/merge_test.go`
|
||||
- `internal/stage/normalize_test.go`
|
||||
- `internal/stage/trim_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Supported output schemas are limited to `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`.
|
||||
- Normalize/trim outputs must include `segments` arrays.
|
||||
- Merge/normalize/trim all route through deterministic subprocess invocation.
|
||||
|
||||
28
docs/internal/README.md
Normal file
28
docs/internal/README.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# Internal Documentation Index
|
||||
|
||||
## Audience
|
||||
Developers and LLM coding agents changing Narratio internals.
|
||||
|
||||
## Scope
|
||||
Implementation-accurate contracts for workspace/state, manifests, stages, artifact resolution, and adapter boundaries.
|
||||
|
||||
## Component Docs
|
||||
- `adapters.md`: external adapter map, runtime wiring, and boundary ownership.
|
||||
- `storage.md`: remote storage backend contracts and object-store invariants.
|
||||
- `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics.
|
||||
- `artifacts.md`: built-in artifact registry, runtime artifact catalog, and source-resolution behavior.
|
||||
- `workspace.md`: local state model, manifests, run-local layout, promotion, and cleanup invariants.
|
||||
- `stage-prepare.md`: input materialization and provenance capture.
|
||||
- `stage-transcribe.md`: WhisperX transcript generation.
|
||||
- `stage-merge.md`: Seriatim normalization + merge.
|
||||
- `stage-polish.md`: Audita transcript polishing.
|
||||
- `stage-normalize.md`: post-polish normalization.
|
||||
- `stage-trim.md`: bounds-driven transcript trimming.
|
||||
- `stage-analyze.md`: dependency-ordered Scriptorium artifact generation for selected configured artifacts.
|
||||
- `stage-archive.md`: archive upload and current-pointer publish contract.
|
||||
|
||||
## External Integration Notes
|
||||
- `../integrations/README.md`: canonical location for external integration contracts (`audita.md`, `seriatim.md`, `scriptorium.md`).
|
||||
|
||||
## Canonical Owner
|
||||
`docs/internal/` is the canonical home for implemented internals per `docs/documentation/policy.md`.
|
||||
79
docs/internal/adapters.md
Normal file
79
docs/internal/adapters.md
Normal file
@@ -0,0 +1,79 @@
|
||||
# Internal: Adapters
|
||||
|
||||
## Purpose
|
||||
Describe the external adapter boundaries used by Narratio stages and app orchestration, including default runtime wiring.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- Stage requests passed through adapter interfaces (for example transcription, merge/normalize/trim, polish, artifact generation, object-store operations, notifications).
|
||||
- Resolved config values used to construct default adapters.
|
||||
|
||||
Outputs:
|
||||
- Adapter-specific result structs (paths, metadata, status/attempt info, duration/exit details).
|
||||
- Adapter errors returned to stage/app orchestration.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Transport/process/SDK details at system boundaries (`HTTP`, subprocess CLI invocation, AWS SDK calls).
|
||||
- Request/response contracts in `internal/adapters/*` packages.
|
||||
|
||||
Does not own:
|
||||
- Stage sequencing, skip/force/resume decisions.
|
||||
- Manifest transition logic.
|
||||
- Canonical workspace path policy.
|
||||
|
||||
## Config fields used
|
||||
Default wiring and adapter calls consume:
|
||||
- `pipeline.whisperx.*`
|
||||
- `pipeline.seriatim.*`
|
||||
- `pipeline.audita.*`
|
||||
- `pipeline.scriptorium.*`
|
||||
- `pipeline.storage.*` and `pipeline.archive.*` (object-store construction/gating)
|
||||
- `pipeline.notification.*` (sender boundary exists; placeholder behavior today)
|
||||
|
||||
## External adapters used
|
||||
Runtime env boundary fields (`internal/stage.Env`):
|
||||
- `whisperx.Client`
|
||||
- `seriatim.Runner`
|
||||
- `audita.Runner`
|
||||
- `scriptorium.Runner`
|
||||
- `storage.ObjectStore`
|
||||
- `notify.Sender`
|
||||
- `analyzer.Runner`
|
||||
|
||||
Current execution usage:
|
||||
- Actively used by implemented stages: `WhisperX`, `Seriatim`, `Audita`, `Scriptorium`, `ObjectStore`, `Notifier`.
|
||||
- Present but not used by implemented stage set: `Analyzer`, legacy `storage.Backend`.
|
||||
|
||||
Default construction in app runner:
|
||||
- Auto-constructed when not injected: WhisperX HTTP client, Seriatim subprocess runner, Audita subprocess runner, Scriptorium subprocess runner, object store (only when needed), and `notify.NoopSender`.
|
||||
- Callers can inject test/fake implementations through `app.RunOptions.Env`.
|
||||
|
||||
## State and manifest behavior
|
||||
- Adapters do not directly mutate session/run manifests.
|
||||
- Stages and runner own manifest writes and stage status transitions.
|
||||
- Adapter outputs are persisted indirectly through stage result mapping (outputs/logs/generated configs/metadata).
|
||||
|
||||
## Skip and resume behavior
|
||||
- No adapter-level skip/resume semantics.
|
||||
- Skip/resume/force behavior is decided by app runner using manifest stage state.
|
||||
|
||||
## Failure behavior
|
||||
- Adapter constructors validate config-derived values and fail early on invalid required inputs.
|
||||
- Adapter run-time failures are returned to stage code with boundary context and are recorded as stage failures by runner logic.
|
||||
- Subprocess adapters preserve stdout/stderr and generated-config paths to aid diagnosis.
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/adapters/whisperx/http_test.go`
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
- `internal/adapters/audita/subprocess_test.go`
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
- `internal/adapters/storage/*_test.go`
|
||||
- `internal/adapters/notify/fake_test.go`
|
||||
- `internal/adapters/analyzer/fake_test.go`
|
||||
- `internal/app/runner_test.go`
|
||||
|
||||
## Architectural invariants
|
||||
- Stage code depends on adapter interfaces, not transport-specific implementation types.
|
||||
- External SDK-specific types remain inside adapter implementations.
|
||||
- Default app wiring must remain deterministic and overrideable via injected env dependencies.
|
||||
87
docs/internal/artifacts.md
Normal file
87
docs/internal/artifacts.md
Normal file
@@ -0,0 +1,87 @@
|
||||
# Internal: Artifacts
|
||||
|
||||
## Purpose
|
||||
Define Narratio's artifact identity and resolution model for built-in transcript/bounds artifacts and runtime-configured analyze artifacts.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- artifact sources from config/runtime (`pipeline.scriptorium.artifacts.*.inputs.*.source`)
|
||||
- session paths and optional session manifest stage outputs
|
||||
- runtime artifact catalog state for configured artifact sources
|
||||
|
||||
Outputs:
|
||||
- resolved local artifact path and provenance (`ResolvedSessionArtifact`)
|
||||
- runtime catalog entries for planned/executable/available artifacts
|
||||
- validation errors for unsupported, missing, or invalid artifact sources
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- built-in artifact registry and content validation rules
|
||||
- runtime artifact catalog for configured artifact source IDs
|
||||
- source resolution behavior for built-in and configured artifact sources
|
||||
|
||||
Does not own:
|
||||
- artifact generation (stages produce files)
|
||||
- manifest transition policy
|
||||
- archive promotion behavior
|
||||
|
||||
## Config fields used
|
||||
- `pipeline.scriptorium.artifacts.<name>.enabled`
|
||||
- `pipeline.scriptorium.artifacts.<name>.output_path`
|
||||
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source`
|
||||
|
||||
## External adapters used
|
||||
- none
|
||||
|
||||
## State and manifest behavior
|
||||
Built-in registry entries:
|
||||
|
||||
| Artifact ID | Canonical file | Producer stage | Output kind |
|
||||
| --- | --- | --- | --- |
|
||||
| `narratio.transcript.merged` | `transcripts/merged.json` | `merge` | `transcript_merged` |
|
||||
| `narratio.transcript.polished` | `transcripts/processed.json` | `polish` | `transcript_processed` |
|
||||
| `narratio.transcript.full` | `transcripts/normalized.json` | `normalize` | `transcript_normalized` |
|
||||
| `narratio.transcript.trimmed` | `transcripts/trimmed.json` | `trim` | `transcript_trimmed` |
|
||||
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` |
|
||||
|
||||
Runtime catalog entries include built-ins and configured `narratio.artifact.<name>` sources.
|
||||
|
||||
Catalog states:
|
||||
- `planned`: source is registered and known for this run
|
||||
- `executable`: configured artifact is selected for analyze execution
|
||||
- `available`: artifact has a usable file path (generated this run or reused from disk)
|
||||
|
||||
Resolution behavior:
|
||||
- built-in sources resolve via manifest producer outputs first, then canonical fallback path
|
||||
- configured `narratio.artifact.<name>` sources resolve through runtime catalog availability
|
||||
- configured source lookup requires catalog context
|
||||
|
||||
Configured artifact provenance values:
|
||||
- `generated.current_analyze_run`
|
||||
- `filesystem.disabled_artifact_output`
|
||||
|
||||
Content validation:
|
||||
- transcript built-ins: JSON with top-level `segments` array
|
||||
- bounds built-in: valid JSON
|
||||
- configured artifacts: non-empty text file
|
||||
|
||||
## Skip and resume behavior
|
||||
- resolver and catalog have no direct skip/resume decisions
|
||||
- stage/runner skip-resume behavior consumes catalog/resolver results
|
||||
|
||||
## Failure behavior
|
||||
- unsupported source -> source validation error
|
||||
- known source unavailable -> `ErrSessionArtifactNotFound`
|
||||
- configured source without catalog -> resolution error
|
||||
- resolved file with invalid content -> validation error
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/artifacts/artifact_resolver_test.go`
|
||||
- `internal/artifacts/catalog_test.go`
|
||||
- `internal/stage/analyze_test.go`
|
||||
- `internal/config/scriptorium_test.go`
|
||||
|
||||
## Architectural invariants
|
||||
- built-in IDs are static and registry-backed
|
||||
- configured artifact IDs are runtime-derived (`narratio.artifact.<name>`) and catalog-backed
|
||||
- built-in/source resolution remains deterministic and validation-gated
|
||||
81
docs/internal/manifest.md
Normal file
81
docs/internal/manifest.md
Normal file
@@ -0,0 +1,81 @@
|
||||
# Internal: Manifest
|
||||
|
||||
## Purpose
|
||||
Describe Narratio's durable execution state model for session-level and run-level manifests, including lifecycle transitions and persistence behavior.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- Session identity and run identity from app orchestration.
|
||||
- Stage transition events and stage result payloads.
|
||||
|
||||
Outputs:
|
||||
- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`.
|
||||
- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Manifest schemas (`Manifest`, `RunManifest`, stage records, error records, input/artifact records).
|
||||
- Stage status/action transition methods.
|
||||
- Persistent store contract (`manifest.Store`) and local JSON store implementation.
|
||||
|
||||
Does not own:
|
||||
- Stage implementation details.
|
||||
- Path construction policy outside manifest file persistence calls.
|
||||
- CLI command behavior.
|
||||
|
||||
## Config fields used
|
||||
Manifest package itself does not read config directly.
|
||||
|
||||
Manifest identity fields are populated by app/stage orchestration from:
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.storage.s3.*` (when archive/S3 identity is set)
|
||||
|
||||
## External adapters used
|
||||
- No external service adapters.
|
||||
- Uses local filesystem for persistence via `manifest.LocalStore`.
|
||||
|
||||
## State and manifest behavior
|
||||
Session manifest model:
|
||||
- Tracks durable per-session stage state and provenance (`pending`, `running`, `succeeded`, `failed`, `skipped`, `stale`, `interrupted`).
|
||||
- Stores resolved inputs, durable artifacts, stage logs/config refs, and stage metadata.
|
||||
|
||||
Run manifest model:
|
||||
- Tracks one invocation (`run_id`) with requested stages and force mode.
|
||||
- Tracks per-stage action (`run` or `skip`) and per-stage status.
|
||||
- Tracks overall run status (`running`, `succeeded`, `failed`).
|
||||
|
||||
Persistence behavior:
|
||||
- Load validates required identity/timestamp fields and normalizes maps/records.
|
||||
- Save updates `updated_at` and writes JSON atomically (temp file + rename).
|
||||
- Session and run manifests are saved incrementally before/after stage transitions.
|
||||
|
||||
Relationship during execution:
|
||||
- Runner updates both manifests for every stage transition.
|
||||
- Session manifest is the durable pipeline-progress ledger.
|
||||
- Run manifest is invocation history and audit record.
|
||||
- Analyze stage outputs are persisted as `kind=scriptorium_artifact` with `source_id=narratio.artifact.<name>` for configured artifact identity.
|
||||
|
||||
## Skip and resume behavior
|
||||
- Resume and skip decisions are based on session-manifest stage statuses.
|
||||
- `--force` reruns selected stages and marks downstream succeeded stages as `stale` in session manifest.
|
||||
- Run manifest records whether each stage was executed or skipped in that invocation.
|
||||
|
||||
## Failure behavior
|
||||
- Stage failure marks both manifests failed for that stage and records error messages/timestamps.
|
||||
- Save failures are returned immediately and fail the command.
|
||||
- Invalid/malformed manifest files fail load with explicit validation/decode errors.
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/manifest/manifest_test.go`
|
||||
- `internal/manifest/run_manifest_test.go`
|
||||
- `internal/manifest/store_test.go`
|
||||
- `internal/app/runner_test.go`
|
||||
- `internal/app/run_control_test.go`
|
||||
- `internal/app/resume_run_stage_test.go`
|
||||
|
||||
## Architectural invariants
|
||||
- Session manifest is authoritative for stage progression across invocations.
|
||||
- Run manifest is invocation-scoped and never replaces session manifest as progress authority.
|
||||
- Manifest writes are atomic and deterministic (JSON + newline, temp rename pattern).
|
||||
84
docs/internal/stage-analyze.md
Normal file
84
docs/internal/stage-analyze.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# Stage: analyze
|
||||
|
||||
## Purpose
|
||||
Execute selected configured Scriptorium artifacts in deterministic dependency order and promote successful outputs to canonical session artifact paths.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- configured artifact definitions from `pipeline.scriptorium.artifacts`
|
||||
- selected artifact filter from runtime (`--artifacts`) when provided
|
||||
- resolved artifact input sources declared per artifact (`inputs.*.source`)
|
||||
- optional previous-session file inputs (`previous_session_artifact`)
|
||||
|
||||
Outputs:
|
||||
- one promoted output file per executed configured artifact at that artifact's configured `output_path`
|
||||
- stage metadata containing generated artifact entries and reused disabled-artifact entries
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- runtime artifact catalog construction for analyze execution
|
||||
- selected-artifact planning and dependency ordering
|
||||
- per-artifact input resolution, var resolution, timeout/render-debug resolution
|
||||
- Scriptorium run/render invocation for each selected artifact
|
||||
- run-local output generation and canonical promotion
|
||||
|
||||
Does not own:
|
||||
- transcript generation/processing stages
|
||||
- archive promotion policy
|
||||
- per-artifact resume semantics
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.scriptorium.binary`
|
||||
- `pipeline.scriptorium.config_path`
|
||||
- `pipeline.scriptorium.timeout`
|
||||
- `pipeline.scriptorium.render_debug`
|
||||
- `pipeline.scriptorium.artifacts.<name>.*`
|
||||
- `enabled`
|
||||
- `depends_on`
|
||||
- `prompt_id`
|
||||
- `profile_id`
|
||||
- `timeout`
|
||||
- `output_path`
|
||||
- `render_debug`
|
||||
- `inputs`
|
||||
- `vars`
|
||||
|
||||
## External Adapters Used
|
||||
- Scriptorium adapter:
|
||||
- optional `RenderArtifact` (render debug)
|
||||
- `RunArtifact` (artifact generation)
|
||||
|
||||
## State and Manifest Behavior
|
||||
- If `pipeline.scriptorium` is absent, stage returns success metadata with `skipped=true`.
|
||||
- If no artifacts are configured, stage returns success metadata with `skipped=true`.
|
||||
- If zero artifacts are executable after `enabled` + `--artifacts` filtering, stage returns success metadata with `skipped=true`.
|
||||
- Builds runtime catalog with built-ins and configured artifacts.
|
||||
- Non-executable configured artifacts are marked available only when their configured output file exists and is valid on disk.
|
||||
- Executes selected configured artifacts in topological order with deterministic tie-breaking.
|
||||
- For each generated artifact, records metadata fields including `name`, `source_id`, `output_kind`, `path`, `prompt_id`, `profile_id`, and `provenance`.
|
||||
- Reused disabled artifacts are recorded separately in `reused_artifacts` with provenance `filesystem.disabled_artifact_output`.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when analyze is already `succeeded` and `--force` is not set.
|
||||
- Analyze remains stage-scoped for resume/skip; there is no per-artifact resume state.
|
||||
- `--artifacts` filters which configured artifacts are executable when analyze runs; it does not imply `--force`.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on invalid dependency ordering, unavailable required configured inputs, invalid built-in input prerequisites, render/run adapter failures, validation-failed adapter results, or missing/empty outputs.
|
||||
- Required configured dependency missing from catalog availability fails clearly before invocation.
|
||||
- Optional missing inputs are omitted.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/analyze_test.go`
|
||||
- `internal/artifacts/catalog_test.go`
|
||||
- `internal/artifacts/artifact_resolver_test.go`
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Configured artifacts are identified by `narratio.artifact.<name>` source IDs.
|
||||
- Artifact-to-artifact references rely on explicit `depends_on` declarations validated in config.
|
||||
- Generated analyze outputs are treated uniformly as Scriptorium artifacts.
|
||||
- Successful outputs must exist and be non-empty before promotion.
|
||||
68
docs/internal/stage-archive.md
Normal file
68
docs/internal/stage-archive.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# Stage: archive
|
||||
|
||||
## Purpose
|
||||
Publish run records and promoted session artifacts to object storage, then atomically advance the remote current pointer.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- session manifest and prerequisite stage records
|
||||
- run root contents under `runs/{run_id}/`
|
||||
- promotion sources from session root (`archive.promote_artifacts`)
|
||||
|
||||
Outputs:
|
||||
- uploaded run files under `{session_prefix}/runs/{run_id}/...`
|
||||
- uploaded promoted artifacts under `{session_prefix}/...`
|
||||
- `{session_prefix}/current/manifest.json`
|
||||
- `{session_prefix}/current/run_id.txt` written last
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Archive enable/disable gate behavior
|
||||
- Prerequisite stage success enforcement
|
||||
- Run file collection and upload (excluding `audio/`)
|
||||
- Promotion rule resolution and upload
|
||||
- Commit pointer publish order
|
||||
|
||||
Does not own:
|
||||
- Stage execution before archive
|
||||
- Post-archive local cleanup policy execution (handled by app cleanup logic)
|
||||
|
||||
## Config Fields Used
|
||||
- `pipeline.archive.enabled`
|
||||
- `pipeline.archive.upload_run`
|
||||
- `pipeline.archive.promote_artifacts`
|
||||
- `pipeline.storage.s3.bucket`
|
||||
- `pipeline.storage.s3.root_prefix`
|
||||
- `pipeline.workspace.root`
|
||||
- `session.campaign`
|
||||
- `session.session_id`
|
||||
|
||||
## External Adapters Used
|
||||
- Object storage backend (`env.ObjectStore`) for upload/list primitives.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
|
||||
- Resolves bucket/prefix from manifest identity first, then config fallback.
|
||||
- Writes metadata including:
|
||||
- upload counts/paths
|
||||
- `current_manifest_key`
|
||||
- `current_run_id_key`
|
||||
- `current_pointer_written`
|
||||
- On skipped archive path, returns metadata with `skipped=true` and pointer not written.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Stage may self-skip (metadata skip) when archive disabled or run upload disabled.
|
||||
- Runner-level skip also applies for previously succeeded stage unless forced.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing prerequisite success, missing object store when required, missing run root, missing required promotion source, upload failures, or pointer write failures.
|
||||
- Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/archive_test.go`
|
||||
- `internal/app/post_archive_cleanup_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Run upload excludes `audio/` subtree.
|
||||
- `current/manifest.json` uploads before `current/run_id.txt`.
|
||||
- `current/run_id.txt` is the remote publish commit marker.
|
||||
63
docs/internal/stage-merge.md
Normal file
63
docs/internal/stage-merge.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# Stage: merge
|
||||
|
||||
## Purpose
|
||||
Normalize per-speaker raw transcripts and merge them into one merged transcript via Seriatim.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `transcripts/raw/*.json`
|
||||
- `inputs/speakers.yml`
|
||||
- `inputs/autocorrect.yml`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/merged.json`
|
||||
- optional `artifacts/seriatim.report.json` (when report enabled)
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Raw transcript discovery/validation
|
||||
- Per-input normalize calls to Seriatim
|
||||
- Final merge call to Seriatim
|
||||
- Run-local log/config/report path wiring
|
||||
- Promotion of merged/report outputs to canonical paths
|
||||
|
||||
Does not own:
|
||||
- Transcript polishing or downstream artifact generation
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.seriatim.binary`
|
||||
- `pipeline.seriatim.timeout`
|
||||
- `pipeline.seriatim.output_schema`
|
||||
- `pipeline.seriatim.coalesce_gap`
|
||||
- `pipeline.seriatim.report`
|
||||
- `pipeline.seriatim.env.*`
|
||||
|
||||
## External Adapters Used
|
||||
- Seriatim adapter:
|
||||
- `Normalize` for each raw input
|
||||
- `Run` for final merge
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads transcript inputs from transcribe stage outputs in manifest when present; falls back to canonical raw directory.
|
||||
- Writes run-local outputs/logs/config under `runs/{run_id}/merge/...` when enabled.
|
||||
- Promotes canonical merged transcript and optional report.
|
||||
- Records normalized-input provenance and adapter metadata in stage metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced rerun of this or upstream stages can stale downstream succeeded stages via runner invalidation.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid raw transcripts, missing speakers/autocorrect files, normalize failure, merge failure, invalid merged output JSON, or invalid report JSON when enabled.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/merge_test.go`
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Merge consumes normalized forms of each raw transcript.
|
||||
- Merged transcript must validate before promotion.
|
||||
- Report output is optional and gated by config.
|
||||
56
docs/internal/stage-normalize.md
Normal file
56
docs/internal/stage-normalize.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# Stage: normalize
|
||||
|
||||
## Purpose
|
||||
Normalize the processed transcript into a deterministic intermediate schema for trim and optionally emit a normalize report.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `transcripts/processed.json`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/normalized.json` (or configured normalize output path)
|
||||
- optional `artifacts/seriatim.normalize.report.json`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Processed transcript discovery/validation
|
||||
- Normalize request construction and invocation
|
||||
- Optional normalize report wiring
|
||||
- Promotion of normalized transcript and optional report
|
||||
|
||||
Does not own:
|
||||
- Bounds detection or segment trimming
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.normalize.output_path`
|
||||
- `pipeline.normalize.output_schema`
|
||||
- `pipeline.normalize.report`
|
||||
- `pipeline.seriatim.binary`
|
||||
- `pipeline.seriatim.timeout`
|
||||
|
||||
## External Adapters Used
|
||||
- Seriatim adapter (`Normalize`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads processed transcript from polish outputs in manifest when present; falls back to canonical path.
|
||||
- Uses run-local output/report/log/config paths when run layout is enabled.
|
||||
- Promotes canonical normalized transcript and optional normalize report.
|
||||
- Records adapter/result metadata including source path selection.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced reruns can stale downstream succeeded stages.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid processed transcript, adapter error, invalid normalized output, or invalid report output when report enabled.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/normalize_test.go`
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Normalized output must validate as processed-transcript-compatible JSON (`segments` array required).
|
||||
- Default normalize config is applied when `pipeline.normalize` is unset.
|
||||
69
docs/internal/stage-polish.md
Normal file
69
docs/internal/stage-polish.md
Normal file
@@ -0,0 +1,69 @@
|
||||
# Stage: polish
|
||||
|
||||
## Purpose
|
||||
Polish merged transcript with Audita and produce a processed transcript for downstream normalization/analyze.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `transcripts/merged.json`
|
||||
- `inputs/glossary.yml`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/processed.json`
|
||||
- optional `artifacts/audita.report.json` (when report enabled)
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Merged transcript discovery/validation
|
||||
- Audita invocation request construction
|
||||
- Run-local logs/config/work-dir/report wiring
|
||||
- Promotion of processed transcript and optional report
|
||||
|
||||
Does not own:
|
||||
- Upstream merge normalization
|
||||
- Downstream normalize/trim/analyze logic
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.audita.binary`
|
||||
- `pipeline.audita.timeout`
|
||||
- `pipeline.audita.llm_api_key_env`
|
||||
- `pipeline.audita.modules`
|
||||
- `pipeline.audita.base_url`
|
||||
- `pipeline.audita.model`
|
||||
- `pipeline.audita.transcript_description`
|
||||
- `pipeline.audita.config_path`
|
||||
- `pipeline.audita.output_schema`
|
||||
- `pipeline.audita.work_dir_retention`
|
||||
- `pipeline.audita.total_llm_concurrency`
|
||||
- `pipeline.audita.proposal_llm_concurrency`
|
||||
- `pipeline.audita.validation_model`
|
||||
- `pipeline.audita.validation_llm_concurrency`
|
||||
- `pipeline.audita.report`
|
||||
|
||||
## External Adapters Used
|
||||
- Audita adapter (`env.Audita.Run`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads merged transcript from merge manifest outputs when available; falls back to canonical merged path.
|
||||
- Uses run-local output/report/log/config/scratch paths when run layout is enabled.
|
||||
- Promotes canonical `transcripts/processed.json` and optional report.
|
||||
- Records adapter invocation metadata, credential presence signal, and output provenance in stage metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced rerun can stale downstream succeeded stages via runner invalidation.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid merged transcript, missing glossary, adapter error, invalid processed output shape (`segments` array required), or invalid report JSON when enabled.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/polish_test.go`
|
||||
- `internal/adapters/audita/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Processed transcript must contain a top-level `segments` array.
|
||||
- Report behavior is strictly config-gated.
|
||||
- Stage output canonicalization always ends at `transcripts/processed.json`.
|
||||
74
docs/internal/stage-prepare.md
Normal file
74
docs/internal/stage-prepare.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# Stage: prepare
|
||||
|
||||
## Purpose
|
||||
Materialize all required session inputs into canonical local workspace paths and record input provenance in the session manifest.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `session.yml` (resolved session config)
|
||||
- `pipeline.resolved.yml` (materialized from resolved pipeline config)
|
||||
- `speakers.yml`
|
||||
- `autocorrect.yml`
|
||||
- `glossary.yml`
|
||||
- audio source:
|
||||
- local (`session.inputs.audio_dir` or `session.inputs.audio_files`), or
|
||||
- S3 (`session.inputs.audio_s3.prefix`)
|
||||
|
||||
Outputs:
|
||||
- `inputs/session.yml`
|
||||
- `inputs/pipeline.resolved.yml`
|
||||
- `inputs/speakers.yml`
|
||||
- `inputs/autocorrect.yml`
|
||||
- `inputs/glossary.yml`
|
||||
- `audio/*.flac` in session workdir
|
||||
- `manifest.Inputs` records with checksums and source metadata
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Input path resolution and validation
|
||||
- Local copy/materialization of configs and audio files
|
||||
- S3 audio download to run-scoped spool, then copy into work audio dir
|
||||
|
||||
Does not own:
|
||||
- Transcript generation/processing
|
||||
- Archive publish behavior
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `session.inputs.speakers_file`
|
||||
- `session.inputs.autocorrect_file`
|
||||
- `session.inputs.glossary_file`
|
||||
- `session.inputs.audio_dir`
|
||||
- `session.inputs.audio_files`
|
||||
- `session.inputs.audio_s3.prefix`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.spool.root`
|
||||
- `pipeline.storage.s3.bucket`
|
||||
- `pipeline.storage.s3.root_prefix`
|
||||
|
||||
## External Adapters Used
|
||||
- Object storage backend (`env.ObjectStore`) for S3 audio list/download when `audio_s3` is configured.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Ensures workspace layout exists.
|
||||
- Writes resolved config and input files to canonical `inputs/` paths.
|
||||
- Records all prepared inputs into `manifest.Inputs` (sorted deterministically by kind/path).
|
||||
- For S3 audio, records `S3Bucket`, `S3Key`, `S3Size`, `S3ETag`, and `SpoolPath` in each audio input record.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when stage already `succeeded` and `--force` is not set.
|
||||
- Stage itself is deterministic/idempotent for unchanged inputs (`copyFileIfChanged`, `writeBytesIfChanged`).
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing required files, invalid audio source combinations, no discoverable `.flac` files, duplicate audio basenames, missing object store for S3 mode, or S3 list/download failures.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/prepare_test.go`
|
||||
- `internal/app/session_cli_test.go`
|
||||
- `internal/config/load_validate_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive.
|
||||
- Audio files must be `.flac`.
|
||||
- Canonical `inputs/*` and `audio/*` paths are the durable source for downstream stages.
|
||||
58
docs/internal/stage-transcribe.md
Normal file
58
docs/internal/stage-transcribe.md
Normal file
@@ -0,0 +1,58 @@
|
||||
# Stage: transcribe
|
||||
|
||||
## Purpose
|
||||
Generate per-speaker raw transcripts from prepared audio using WhisperX.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `audio/*.flac` prepared by `prepare`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/raw/<speaker>.json` for each input audio file
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Discovering prepared audio inputs
|
||||
- Deriving speaker ids from audio basenames
|
||||
- Parallel WhisperX invocation with bounded concurrency
|
||||
- Validating produced JSON and promoting run-local outputs
|
||||
|
||||
Does not own:
|
||||
- Transcript merge/polish/normalize/trim/analyze
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.whisperx.transcribe_url`
|
||||
- `pipeline.whisperx.language`
|
||||
- `pipeline.whisperx.timeout`
|
||||
- `pipeline.whisperx.retries`
|
||||
- `pipeline.whisperx.retry_delay`
|
||||
- `pipeline.whisperx.concurrency`
|
||||
|
||||
## External Adapters Used
|
||||
- WhisperX adapter (`env.WhisperX.Transcribe`).
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Uses run-local output paths under `runs/{run_id}/transcribe/outputs/...` when run layout is enabled.
|
||||
- Validates each generated transcript JSON before promotion.
|
||||
- Promotes canonical outputs to `transcripts/raw/*.json`.
|
||||
- Records per-file metadata (attempts/status/duration/output path) in stage metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies for previously succeeded stage unless forced.
|
||||
- On forced upstream reruns, downstream succeeded stages can be marked `stale` by runner logic.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails if no prepared audio exists, duplicate speaker basenames are detected, adapter output path mismatches expected path, any output JSON is invalid, or one worker fails.
|
||||
- Cancels in-flight workers after first terminal error.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/transcribe_test.go`
|
||||
- `internal/app/whisperx_wiring_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Speaker identity is derived from `.flac` basename and must be unique.
|
||||
- Every successful speaker output must be valid JSON before promotion.
|
||||
- Canonical raw transcript set is the only supported merge input surface.
|
||||
75
docs/internal/stage-trim.md
Normal file
75
docs/internal/stage-trim.md
Normal file
@@ -0,0 +1,75 @@
|
||||
# Stage: trim
|
||||
|
||||
## Purpose
|
||||
Optionally trim the normalized transcript to session bounds; always produce a durable trimmed transcript.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `transcripts/normalized.json`
|
||||
|
||||
Outputs:
|
||||
- `transcripts/trimmed.json` (or configured trim output path)
|
||||
- when trim enabled: `artifacts/session_bounds.json`
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Trim-enabled switch behavior
|
||||
- Bounds generation via Scriptorium artifact run
|
||||
- Bounds validation against normalized transcript
|
||||
- Keep-selector derivation and Seriatim trim invocation
|
||||
- Copy-through behavior when disabled or bounds indicate unchanged transcript
|
||||
|
||||
Does not own:
|
||||
- Upstream normalization
|
||||
- Downstream artifact analysis
|
||||
|
||||
## Config Fields Used
|
||||
- `session.session_id`
|
||||
- `session.campaign`
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.trim.enabled`
|
||||
- `pipeline.trim.output_path`
|
||||
- `pipeline.trim.bounds.prompt_id`
|
||||
- `pipeline.trim.bounds.profile_id`
|
||||
- `pipeline.trim.bounds.timeout`
|
||||
- `pipeline.trim.bounds.output_path`
|
||||
- `pipeline.trim.bounds.transcript_input_name`
|
||||
- `pipeline.trim.bounds.render_debug`
|
||||
- `pipeline.trim.bounds.render_output_path`
|
||||
- `pipeline.seriatim.binary`
|
||||
- `pipeline.seriatim.timeout`
|
||||
- `pipeline.scriptorium.binary`
|
||||
- `pipeline.scriptorium.config_path`
|
||||
- `pipeline.scriptorium.timeout`
|
||||
|
||||
## External Adapters Used
|
||||
- Scriptorium adapter:
|
||||
- optional `RenderArtifact` for bounds debug render
|
||||
- `RunArtifact` for bounds output
|
||||
- Seriatim adapter:
|
||||
- `Trim` when bounds indicate trimming is required
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Reads normalized transcript from normalize manifest outputs when available; falls back to canonical path.
|
||||
- Uses run-local outputs/logs/reports/config/scratch paths when run layout is enabled.
|
||||
- Promotes canonical trimmed transcript; promotes session bounds when trim enabled.
|
||||
- Records bounds diagnostics, trim action, keep selector, and adapter metadata.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Runner-level skip applies when already succeeded and not forced.
|
||||
- Forced reruns can stale downstream succeeded stages.
|
||||
- When `trim.enabled=false`, stage still succeeds by copying normalized to trimmed output.
|
||||
|
||||
## Failure Behavior
|
||||
- Fails on missing/invalid normalized transcript.
|
||||
- With trim enabled, fails on missing adapters/config, bounds generation/validation errors, invalid bounds JSON, invalid range/segment ids, trim adapter failures, or invalid trimmed output.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/stage/trim_test.go`
|
||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||
- `internal/adapters/seriatim/subprocess_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Trim never falls back to processed transcript; normalized transcript is required input.
|
||||
- `session_bounds` output exists only for enabled trim path.
|
||||
- Render-debug artifacts are diagnostics and not declared stage outputs.
|
||||
71
docs/internal/storage.md
Normal file
71
docs/internal/storage.md
Normal file
@@ -0,0 +1,71 @@
|
||||
# Internal: Storage
|
||||
|
||||
## Purpose
|
||||
Document Narratio's remote storage backend contracts and implementations under `internal/adapters/storage`.
|
||||
|
||||
## Inputs and outputs
|
||||
Inputs:
|
||||
- Resolved storage config (`pipeline.storage.*`).
|
||||
- Bucket-relative object keys and local file paths from stage/app orchestration.
|
||||
|
||||
Outputs:
|
||||
- Listed/downloaded/uploaded object metadata (`ObjectInfo`).
|
||||
- Existence checks and storage-layer errors.
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Remote object-store interface and implementation details.
|
||||
- S3 client wiring and API calls.
|
||||
- Object key normalization and upload/download/list primitives.
|
||||
|
||||
Does not own:
|
||||
- Session/run prefix semantics.
|
||||
- Archive commit order semantics.
|
||||
- Manifest updates.
|
||||
|
||||
## Config fields used
|
||||
- `pipeline.storage.backend`
|
||||
- `pipeline.storage.s3.bucket`
|
||||
- `pipeline.storage.s3.region`
|
||||
- `pipeline.storage.s3.endpoint`
|
||||
- `pipeline.storage.s3.force_path_style`
|
||||
- `pipeline.storage.s3.access_key_id_env`
|
||||
- `pipeline.storage.s3.secret_access_key_env`
|
||||
|
||||
## External adapters used
|
||||
Storage package contracts:
|
||||
- `ObjectStore` (active remote object-store boundary): `List`, `Download`, `Upload`, `Exists`.
|
||||
- `Backend` (archive request boundary): currently implemented with `NoopBackend` only.
|
||||
|
||||
Implementations:
|
||||
- `S3Backend`: AWS SDK-backed `ObjectStore` implementation.
|
||||
- `FakeBackend`: deterministic test `ObjectStore` and archive backend.
|
||||
- `NoopBackend`: deterministic no-op archive backend for compatibility wiring.
|
||||
|
||||
## State and manifest behavior
|
||||
- Storage implementations are stateless with respect to manifest/session lifecycle.
|
||||
- Caller supplies fully-qualified bucket-relative keys.
|
||||
- Storage layer does not infer campaign/session/run/root-prefix semantics.
|
||||
- Caller controls publish ordering; storage layer executes individual operations in the order invoked.
|
||||
|
||||
## Skip and resume behavior
|
||||
- No storage-level skip/resume behavior.
|
||||
- Skip/resume decisions are made by stage/app logic before storage calls occur.
|
||||
|
||||
## Failure behavior
|
||||
- `NewObjectStoreFromConfig` fails when no remote backend is configured or required S3 config is missing.
|
||||
- `S3Backend` constructor fails when required bucket is missing or AWS client setup fails.
|
||||
- CRUD operations return contextual errors (including not-found behavior via `Exists`).
|
||||
- Key normalization is applied before operations (`\\` to `/`, leading slash trimmed).
|
||||
|
||||
## Tests to inspect before changing
|
||||
- `internal/adapters/storage/factory_test.go`
|
||||
- `internal/adapters/storage/s3_backend_test.go`
|
||||
- `internal/adapters/storage/fake_test.go`
|
||||
- `internal/adapters/storage/keys_test.go`
|
||||
- `internal/adapters/storage/archive.go` + consumers in stage tests (`prepare`, `archive`)
|
||||
|
||||
## Architectural invariants
|
||||
- Callers pass full bucket-relative keys.
|
||||
- Storage backends must not prepend or infer narratio prefixes.
|
||||
- Remote transport details remain isolated to storage adapter implementations.
|
||||
68
docs/internal/workspace.md
Normal file
68
docs/internal/workspace.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# Workspace internals
|
||||
|
||||
## Purpose
|
||||
Define the local durable and run-local workspace model used by stages, manifests, resume, and archive.
|
||||
|
||||
## Inputs and Outputs
|
||||
Inputs:
|
||||
- `pipeline.workspace.root`
|
||||
- `session.campaign`
|
||||
- `session.session_id`
|
||||
- generated `run_id`
|
||||
|
||||
Outputs:
|
||||
- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`
|
||||
- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`
|
||||
- Canonical durable session directories and run-local stage trees
|
||||
|
||||
## Boundaries
|
||||
Owns:
|
||||
- Session-level path layout (`inputs/`, `audio/`, `transcripts/`, `artifacts/`, `reports/`, `logs/`, `config/`, `current/`, `runs/`)
|
||||
- Run-local stage sandbox layout under `runs/{run_id}/{stage}/`
|
||||
- Session lock acquisition/release (`.lock`)
|
||||
|
||||
Does not own:
|
||||
- Stage business logic
|
||||
- Remote archive semantics (documented in `stage-archive.md`)
|
||||
- CLI argument parsing
|
||||
|
||||
## Config Fields Used
|
||||
- `pipeline.workspace.root`
|
||||
- `pipeline.workspace.cleanup_after_archive`
|
||||
- `pipeline.spool.root`
|
||||
- `pipeline.spool.delete_audio_after_archive`
|
||||
- `session.campaign`
|
||||
- `session.session_id`
|
||||
|
||||
## External Adapters Used
|
||||
None directly in this subsystem. Stages may use object storage adapters and then write local outputs into this layout.
|
||||
|
||||
## State and Manifest Behavior
|
||||
- Session state is persisted in the session manifest (`manifest.Manifest`).
|
||||
- Invocation history is persisted per run in run manifests under `runs/{run_id}/manifest.json`.
|
||||
- During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and promoted to canonical session paths after stage success.
|
||||
- `manifest.Artifacts` entries record `ProducerRunID` for durable outputs.
|
||||
- For S3 audio sessions, `prepare` records spool/work paths and S3 provenance in `manifest.Inputs`.
|
||||
|
||||
## Skip and Resume Behavior
|
||||
- Skip/resume decisions are made in `internal/app` (`run_control.go`, `resume.go`) using stage status in the session manifest.
|
||||
- `--force` reruns selected stages and marks downstream previously-succeeded stages as `stale`.
|
||||
- Workspace layout is idempotent (`EnsureLayoutFor`) and reused across runs.
|
||||
|
||||
## Failure Behavior
|
||||
- Failures preserve manifests and run-local files for inspection.
|
||||
- Lock conflicts fail fast via `ErrLockConflict`.
|
||||
- Cleanup can fail post-archive; failure is recorded in archive stage metadata and returned by the run.
|
||||
|
||||
## Tests to Inspect Before Changing
|
||||
- `internal/artifacts/local_test.go`
|
||||
- `internal/stage/run_local_test.go`
|
||||
- `internal/app/run_control_test.go`
|
||||
- `internal/app/resume_run_stage_test.go`
|
||||
- `internal/app/post_archive_cleanup_test.go`
|
||||
|
||||
## Architectural Invariants
|
||||
- Session root is campaign-aware: `{workspace.root}/work/{campaign}/{session_id}`.
|
||||
- Run roots are always nested: `runs/{run_id}` under the session root.
|
||||
- Run-local output promotion must end in canonical session paths.
|
||||
- Cleanup only targets run-scoped directories and must never delete configured root directories.
|
||||
149
docs/operations.md
Normal file
149
docs/operations.md
Normal file
@@ -0,0 +1,149 @@
|
||||
# Operations
|
||||
|
||||
This guide describes the implemented operator lifecycle for Narratio.
|
||||
|
||||
For field-level configuration, see [docs/config.md](./config.md). For full command/flag reference, see [docs/cli.md](./cli.md).
|
||||
|
||||
## Normal workflow (S3-first path)
|
||||
|
||||
1. Upload session `.flac` files to object storage under the session audio prefix.
|
||||
2. Run Narratio:
|
||||
|
||||
```bash
|
||||
narratio run --session-id 2026-04-04
|
||||
```
|
||||
|
||||
3. Read success output:
|
||||
- `narratio run: session <session_id>; executed=<n> skipped=<n>; manifest=<path>`
|
||||
- use `manifest=<path>` with `status` for inspection.
|
||||
|
||||
Notes:
|
||||
- default config/session discovery applies unless `--config` and `--session` are passed.
|
||||
- S3 audio mode requires `session.inputs.audio_s3.prefix` and valid object-store access.
|
||||
|
||||
## Local filesystem layout and state artifacts
|
||||
|
||||
Session root:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/`
|
||||
|
||||
Primary state:
|
||||
- `manifest.json`: session-level stage state.
|
||||
- `runs/{run_id}/manifest.json`: invocation-level state.
|
||||
- `.lock`: session lock while a run is active.
|
||||
|
||||
Canonical session directories:
|
||||
- `inputs/`
|
||||
- `audio/`
|
||||
- `transcripts/`
|
||||
- `artifacts/`
|
||||
- `reports/`
|
||||
- `logs/`
|
||||
- `config/`
|
||||
- `current/`
|
||||
- `runs/`
|
||||
|
||||
Run-local stage directories:
|
||||
- `runs/{run_id}/{stage}/` with stage-local `outputs/`, `logs/`, `reports/`, `config/`, `scratch/`.
|
||||
|
||||
Behavior:
|
||||
- directory creation is idempotent.
|
||||
- stage outputs are generally generated run-local first, then promoted to canonical paths on success.
|
||||
|
||||
## Analyze artifact execution lifecycle
|
||||
|
||||
Analyze executes configured artifacts from `pipeline.scriptorium.artifacts`.
|
||||
|
||||
Execution model:
|
||||
- executable set = enabled artifacts, filtered by `--artifacts` when provided.
|
||||
- artifact-to-artifact dependencies are declared via `depends_on`.
|
||||
- selected artifacts run in deterministic dependency order.
|
||||
- after each successful artifact run, output is promoted to configured canonical `output_path`.
|
||||
|
||||
Configured artifact source reuse:
|
||||
- a non-executable configured artifact can satisfy inputs if its configured output file already exists and is valid.
|
||||
- reused configured artifact provenance is `filesystem.disabled_artifact_output`.
|
||||
|
||||
`--artifacts` behavior:
|
||||
- accepted on `run`, `resume`, and `run-stage analyze`.
|
||||
- filters analyze execution only; does not force stage rerun.
|
||||
|
||||
## Remote archive layout and publish contract
|
||||
|
||||
When archive is enabled and run upload is enabled, archive publishes under:
|
||||
|
||||
- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
- run prefix: `{session_prefix}/runs/{run_id}/`
|
||||
|
||||
Archive uploads:
|
||||
- run record files from run root (excluding `audio/`).
|
||||
- promoted files from explicit `archive.promote_artifacts` rules.
|
||||
|
||||
Publish order:
|
||||
1. upload `current/manifest.json`
|
||||
2. upload `current/run_id.txt` last
|
||||
|
||||
`current/run_id.txt` is the remote commit marker.
|
||||
|
||||
Archive promotion is explicit and path-based:
|
||||
- Narratio does not auto-promote all generated analyze artifacts.
|
||||
- missing required promotion sources fail archive stage.
|
||||
- missing optional promotion sources are skipped.
|
||||
|
||||
## Resume, retry, and safe rerun behavior
|
||||
|
||||
Default skip:
|
||||
- `run` and `run-stage` skip already-succeeded stages unless `--force` is set.
|
||||
|
||||
Resume:
|
||||
- `resume` starts at first non-succeeded stage.
|
||||
- `resume --force` runs full stage order.
|
||||
|
||||
Forced reruns:
|
||||
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
|
||||
|
||||
Safe rerun pattern:
|
||||
1. rerun the changed stage with `--force`.
|
||||
2. run `resume` to rebuild downstream stages.
|
||||
|
||||
## Cleanup behavior
|
||||
|
||||
Cleanup is considered only when archive stage executed and succeeded.
|
||||
|
||||
Cleanup toggles:
|
||||
- `pipeline.spool.delete_audio_after_archive=true` deletes run-scoped spool audio.
|
||||
- `pipeline.workspace.cleanup_after_archive=true` deletes run-scoped local run directory.
|
||||
|
||||
Cleanup eligibility gates:
|
||||
- archive enabled
|
||||
- archive run upload enabled
|
||||
- run record upload completed
|
||||
- current pointer write completed (`current/run_id.txt` written)
|
||||
|
||||
No cleanup for failed/incomplete/unarchived/archive-skipped runs.
|
||||
|
||||
## Failure and recovery playbooks
|
||||
|
||||
After failure, Narratio keeps:
|
||||
- session manifest
|
||||
- run manifest
|
||||
- run-local artifacts/logs/config/reports
|
||||
|
||||
Failed or incomplete runs remain local-only.
|
||||
|
||||
Recommended recovery:
|
||||
|
||||
1. inspect state:
|
||||
|
||||
```bash
|
||||
narratio status --manifest <manifest-path>
|
||||
```
|
||||
|
||||
2. fix root cause (config/input/credentials/service availability).
|
||||
3. continue with `resume`, or targeted `run-stage --force` followed by `resume`.
|
||||
|
||||
## Operational caveats
|
||||
|
||||
- `status` requires explicit `--manifest`; there is no session-id lookup command.
|
||||
- local and S3 audio input modes are mutually exclusive.
|
||||
- archive publish requires upstream stages through `analyze` to be `succeeded`.
|
||||
- required promotion rules can fail when selected analyze artifacts did not generate a required file path.
|
||||
762
docs/roadmap/runtime-artifacts.md
Normal file
762
docs/roadmap/runtime-artifacts.md
Normal file
@@ -0,0 +1,762 @@
|
||||
# Roadmap: Runtime-Defined Scriptorium Artifacts
|
||||
|
||||
## Status
|
||||
|
||||
Implementation roadmap for a pre-release hard cutover.
|
||||
|
||||
## Purpose
|
||||
|
||||
Narratio currently treats artifact generation as a narrow `analyze` stage that supports a hard-coded `session_recap` artifact. This roadmap describes how to generalize artifact generation so operators can define Scriptorium-backed output artifacts at runtime through `pipeline.yml`.
|
||||
|
||||
The goal is to keep Narratio as a fixed pipeline orchestrator while making the artifact generation step configurable, composable, deterministic, and easy to regenerate selectively.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
Operators should be able to define artifacts such as session recaps, player handouts, NPC summaries, quest logs, entity maps, or other campaign-specific outputs without changing Narratio code.
|
||||
|
||||
A configured artifact is declared under:
|
||||
|
||||
```text
|
||||
pipeline.scriptorium.artifacts.<name>
|
||||
```
|
||||
|
||||
Each configured artifact becomes a canonical runtime artifact source ID:
|
||||
|
||||
```text
|
||||
narratio.artifact.<name>
|
||||
```
|
||||
|
||||
For example:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd_session.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
```
|
||||
|
||||
This artifact is addressable by later artifacts as:
|
||||
|
||||
```text
|
||||
narratio.artifact.session_recap
|
||||
```
|
||||
|
||||
A dependent artifact can then consume it explicitly:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
player_handout:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- session_recap
|
||||
prompt_id: dnd_session.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
```
|
||||
|
||||
## Resolved Design Decisions
|
||||
|
||||
The following decisions are settled for the initial implementation:
|
||||
|
||||
1. Configured artifact outputs must live under Narratio's internal artifact output directory, initially `artifacts/`.
|
||||
2. The artifact output directory should be defined as an internal default in `internal/config/defaults.go`, but no public configuration knob should be exposed yet.
|
||||
3. Artifact `output_path` should remain explicit in the initial implementation to avoid guessing file extensions or output formats.
|
||||
4. A disabled artifact may still be referenced as an input if its declared output already exists on disk and passes basic validation.
|
||||
5. A disabled artifact is not executable during the current analyze run.
|
||||
6. Artifact-to-artifact references require an explicit `depends_on` entry. Narratio should fail fast if the dependency declaration is missing.
|
||||
7. The manifest remains stage-oriented: `analyze` succeeds or fails as a full stage.
|
||||
8. Analyze-stage metadata may record per-artifact output details for provenance and later resolution, but not for intra-stage resume semantics.
|
||||
9. `--artifacts` should be added as a CLI filter for selective artifact generation.
|
||||
10. `--artifacts` does not imply `--force`; it only changes which configured artifacts are treated as executable when `analyze` actually runs.
|
||||
11. Because Narratio is still pre-release, the hard-coded `session_recap` behavior should be removed immediately rather than deprecated gradually.
|
||||
|
||||
## Scope
|
||||
|
||||
This roadmap covers:
|
||||
|
||||
- introducing a runtime artifact catalog;
|
||||
- generalizing configured Scriptorium artifact execution;
|
||||
- supporting `narratio.artifact.<name>` source IDs;
|
||||
- adding explicit artifact dependencies;
|
||||
- supporting disabled-but-resolvable artifact inputs;
|
||||
- adding selective artifact execution via `--artifacts`;
|
||||
- recording generated artifacts in analyze-stage metadata and/or manifest outputs;
|
||||
- removing hard-coded `session_recap` behavior;
|
||||
- updating tests and documentation.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This feature should not turn Narratio into a general workflow engine.
|
||||
|
||||
The initial implementation should not add:
|
||||
|
||||
- arbitrary shell-command artifacts;
|
||||
- arbitrary user-defined stages;
|
||||
- loops or conditional branching;
|
||||
- automatic archive promotion of generated artifacts;
|
||||
- semantic knowledge of particular artifact types;
|
||||
- per-artifact resume semantics within a successful or failed analyze stage;
|
||||
- automatic dependency inference without `depends_on`.
|
||||
|
||||
Narratio should continue to orchestrate a fixed pipeline. The configurable part is the set of Scriptorium artifact invocations performed during the `analyze` stage.
|
||||
|
||||
## Current State
|
||||
|
||||
Narratio already has several relevant pieces in place:
|
||||
|
||||
- `pipeline.scriptorium.artifacts` is modeled as a map of artifact definitions.
|
||||
- The Scriptorium adapter already accepts generic run/render requests.
|
||||
- The artifact resolver already understands canonical artifact source IDs.
|
||||
- The `analyze` stage already resolves inputs, optionally runs render-debug, invokes Scriptorium, verifies output, and records metadata.
|
||||
|
||||
The main limitation is that `analyze` currently treats `session_recap` as the only executable artifact and rejects other enabled artifact definitions.
|
||||
|
||||
## Target Architecture
|
||||
|
||||
### Runtime Artifact Catalog
|
||||
|
||||
Introduce a per-run artifact catalog that tracks built-in artifacts and configured artifacts.
|
||||
|
||||
Conceptually:
|
||||
|
||||
```text
|
||||
ArtifactCatalog
|
||||
├── built-in artifacts
|
||||
│ ├── narratio.transcript.merged
|
||||
│ ├── narratio.transcript.polished
|
||||
│ ├── narratio.transcript.full
|
||||
│ ├── narratio.transcript.trimmed
|
||||
│ └── narratio.bounds.session
|
||||
│
|
||||
└── configured artifacts
|
||||
├── narratio.artifact.session_recap
|
||||
├── narratio.artifact.player_handout
|
||||
└── narratio.artifact.npc_summary
|
||||
```
|
||||
|
||||
The catalog should distinguish between three states:
|
||||
|
||||
```text
|
||||
planned valid configured or built-in artifact known to Narratio
|
||||
available artifact has been produced or otherwise resolved
|
||||
executable configured artifact selected for execution in this analyze run
|
||||
```
|
||||
|
||||
Configured artifacts can be planned without being executable. This distinction is important for disabled artifacts and for `--artifacts` filtering.
|
||||
|
||||
### Configured Artifact Source IDs
|
||||
|
||||
Configured artifact keys map directly to source IDs:
|
||||
|
||||
```text
|
||||
pipeline.scriptorium.artifacts.<name>
|
||||
→ narratio.artifact.<name>
|
||||
```
|
||||
|
||||
`session_recap` should no longer be a special built-in analyze artifact. Instead, it is just a conventional configured artifact key:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd_session.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
```
|
||||
|
||||
`narratio.artifact.session_recap` remains valid only because `session_recap` is configured.
|
||||
|
||||
### Artifact Output Directory
|
||||
|
||||
Add an internal default artifact output directory, initially:
|
||||
|
||||
```text
|
||||
artifacts
|
||||
```
|
||||
|
||||
This default should live in `internal/config/defaults.go` or the existing equivalent defaults location.
|
||||
|
||||
For the initial implementation:
|
||||
|
||||
- expose no public config knob for the artifact output directory;
|
||||
- require each configured artifact to provide an explicit `output_path`;
|
||||
- validate that each configured artifact `output_path` is run-relative;
|
||||
- validate that each configured artifact `output_path` is under the internal artifact output directory;
|
||||
- reject output paths that escape the run workspace or use path traversal.
|
||||
|
||||
This preserves future configurability without forcing Narratio to guess output extensions or formats now.
|
||||
|
||||
### Enabled, Disabled, and Selected Artifacts
|
||||
|
||||
Configured artifacts should have three distinct execution states:
|
||||
|
||||
```text
|
||||
enabled by config artifact has enabled: true
|
||||
selected for execution artifact remains executable after --artifacts filtering
|
||||
disabled for execution artifact is not executable, but may be resolvable from disk
|
||||
```
|
||||
|
||||
Without `--artifacts`, all configured artifacts with `enabled: true` are selected for execution.
|
||||
|
||||
With `--artifacts`, only the named artifacts are selected for execution. All other configured artifacts are treated as disabled for the current analyze invocation, regardless of their configured `enabled` value.
|
||||
|
||||
Disabled artifacts may still be resolved as inputs if their configured `output_path` exists on disk and passes validation.
|
||||
|
||||
### Disabled Artifact Resolution
|
||||
|
||||
If artifact `B` references artifact `A`, and `A` is disabled for execution, Narratio should attempt to resolve `A` from disk.
|
||||
|
||||
This should succeed only when:
|
||||
|
||||
1. `A` is defined in `pipeline.scriptorium.artifacts`;
|
||||
2. `A` has a valid `output_path`;
|
||||
3. the output path exists in the current run workspace;
|
||||
4. the output is non-empty, or otherwise passes any available artifact-specific validation.
|
||||
|
||||
The resolved provenance should make the source clear, for example:
|
||||
|
||||
```text
|
||||
filesystem.disabled_artifact_output
|
||||
```
|
||||
|
||||
If the file does not exist or fails validation, the dependent artifact should fail before invoking Scriptorium.
|
||||
|
||||
Example error wording:
|
||||
|
||||
```text
|
||||
artifact player_handout requires narratio.artifact.session_recap, but session_recap is disabled for execution and artifacts/session_recap.md does not exist
|
||||
```
|
||||
|
||||
### Explicit Dependencies
|
||||
|
||||
Artifact-to-artifact references require explicit `depends_on` entries.
|
||||
|
||||
If artifact `B` has an input source of `narratio.artifact.A`, then `B.depends_on` must include `A`.
|
||||
|
||||
This should fail:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
player_handout:
|
||||
enabled: true
|
||||
prompt_id: dnd_session.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
```
|
||||
|
||||
This should pass:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
artifacts:
|
||||
player_handout:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- session_recap
|
||||
prompt_id: dnd_session.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
```
|
||||
|
||||
`depends_on` values refer to configured artifact keys, not full source IDs.
|
||||
|
||||
Dependency validation should fail on:
|
||||
|
||||
- references to unknown artifact keys;
|
||||
- missing `depends_on` entries for artifact-to-artifact input references;
|
||||
- self-dependencies;
|
||||
- dependency cycles among executable artifacts.
|
||||
|
||||
Dependencies on disabled artifacts are permitted, but the disabled dependency must resolve from disk before the dependent artifact runs.
|
||||
|
||||
### Execution Order
|
||||
|
||||
The analyze stage should execute selected artifacts in dependency order.
|
||||
|
||||
Rules:
|
||||
|
||||
- selected artifacts are executable;
|
||||
- disabled artifacts are never executed;
|
||||
- selected artifacts may depend on other selected artifacts;
|
||||
- selected artifacts may depend on disabled artifacts if those disabled artifacts resolve from disk;
|
||||
- independent selected artifacts run in deterministic sorted-name order.
|
||||
|
||||
Use topological sorting over selected artifacts, while validating dependency references across the full configured artifact set.
|
||||
|
||||
### Input Resolution
|
||||
|
||||
Input resolution should use the artifact catalog and existing artifact resolver behavior.
|
||||
|
||||
For each configured artifact input:
|
||||
|
||||
- built-in sources resolve through existing resolver behavior;
|
||||
- `previous_session_artifact` preserves existing behavior;
|
||||
- `narratio.artifact.<name>` resolves through the runtime artifact catalog;
|
||||
- selected dependencies resolve after being produced earlier in the same analyze execution;
|
||||
- disabled dependencies resolve from their configured output path on disk;
|
||||
- optional missing inputs are omitted;
|
||||
- required missing inputs fail before Scriptorium is invoked.
|
||||
|
||||
### Analyze Stage Generalization
|
||||
|
||||
The `analyze` stage should become the generic Scriptorium artifact stage.
|
||||
|
||||
High-level flow:
|
||||
|
||||
1. Load configured Scriptorium artifacts.
|
||||
2. Apply the `--artifacts` filter, if present.
|
||||
3. If no artifacts are selected for execution, return success metadata with `skipped=true`.
|
||||
4. Build the runtime artifact catalog.
|
||||
5. Validate artifact names, output paths, source IDs, dependencies, selected artifacts, and required fields.
|
||||
6. Resolve any disabled dependencies that are required by selected artifacts.
|
||||
7. Sort selected artifacts by dependency order.
|
||||
8. For each selected artifact:
|
||||
- resolve configured inputs;
|
||||
- build the Scriptorium run request;
|
||||
- optionally run Scriptorium render-debug;
|
||||
- run Scriptorium;
|
||||
- fail on validation-failed result;
|
||||
- verify the output exists and is non-empty;
|
||||
- record artifact output metadata;
|
||||
- register `narratio.artifact.<name>` as available in the catalog.
|
||||
9. Return aggregate analyze-stage metadata containing all generated and reused artifacts relevant to the run.
|
||||
|
||||
The Scriptorium adapter should remain generic. It should not decide which artifacts run, how dependencies work, or how artifacts are registered.
|
||||
|
||||
### Manifest and Metadata
|
||||
|
||||
The manifest should remain stage-oriented.
|
||||
|
||||
This means:
|
||||
|
||||
- `analyze` succeeds or fails as a full stage;
|
||||
- if `analyze` has already succeeded and the user does not force it, the runner skips it as a full stage;
|
||||
- Narratio should not implement per-artifact resume in the first version.
|
||||
|
||||
However, analyze-stage metadata should still record artifact outputs for provenance and future resolution.
|
||||
|
||||
Recommended metadata shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"skipped": false,
|
||||
"artifacts": [
|
||||
{
|
||||
"name": "session_recap",
|
||||
"source_id": "narratio.artifact.session_recap",
|
||||
"output_kind": "scriptorium_artifact",
|
||||
"path": "artifacts/session_recap.md",
|
||||
"prompt_id": "dnd_session.session_recap",
|
||||
"profile_id": "local-gemma-31b",
|
||||
"provenance": "generated.current_analyze_run"
|
||||
},
|
||||
{
|
||||
"name": "player_handout",
|
||||
"source_id": "narratio.artifact.player_handout",
|
||||
"output_kind": "scriptorium_artifact",
|
||||
"path": "artifacts/player_handout.md",
|
||||
"prompt_id": "dnd_session.player_handout",
|
||||
"profile_id": "local-gemma-31b",
|
||||
"provenance": "generated.current_analyze_run"
|
||||
}
|
||||
],
|
||||
"reused_artifacts": [
|
||||
{
|
||||
"name": "session_recap",
|
||||
"source_id": "narratio.artifact.session_recap",
|
||||
"path": "artifacts/session_recap.md",
|
||||
"provenance": "filesystem.disabled_artifact_output"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The exact struct can differ from this example, but it should preserve:
|
||||
|
||||
- artifact name;
|
||||
- canonical source ID;
|
||||
- output path;
|
||||
- prompt/profile provenance for generated artifacts;
|
||||
- reused-vs-generated provenance.
|
||||
|
||||
### Resume and Force Behavior
|
||||
|
||||
Keep resume behavior stage-level.
|
||||
|
||||
Recommended semantics:
|
||||
|
||||
```text
|
||||
No --force, analyze already succeeded:
|
||||
runner skips analyze, regardless of --artifacts.
|
||||
|
||||
--force, no --artifacts:
|
||||
analyze regenerates all configured artifacts with enabled: true.
|
||||
|
||||
--force --artifacts player_handout:
|
||||
analyze treats only player_handout as executable.
|
||||
all other configured artifacts are disabled for execution.
|
||||
disabled dependencies may be reused from disk.
|
||||
|
||||
--artifacts player_handout on a not-yet-completed analyze stage:
|
||||
analyze runs only player_handout.
|
||||
disabled dependencies may be reused from disk.
|
||||
```
|
||||
|
||||
`--artifacts` should not imply `--force`. It is an execution filter, not a resume override.
|
||||
|
||||
### `--artifacts` CLI Flag
|
||||
|
||||
Add an `--artifacts` flag to commands that can execute or resume the analyze stage.
|
||||
|
||||
The flag should accept one or more configured artifact names. Internally, normalize values to a set of artifact keys.
|
||||
|
||||
Recommended behavior:
|
||||
|
||||
- validate all requested artifact names against `pipeline.scriptorium.artifacts`;
|
||||
- reject unknown artifact names before running stages;
|
||||
- treat requested artifacts as the only executable artifacts for the analyze stage;
|
||||
- treat all other configured artifacts as disabled for execution;
|
||||
- allow disabled artifacts to satisfy dependencies from disk as described above;
|
||||
- if `--artifacts` is used while executing a stage other than `analyze`, either reject it or ignore it with a clear validation error. Prefer rejection.
|
||||
|
||||
The exact CLI parsing style can follow Narratio's existing conventions. Both comma-separated and repeatable values are acceptable if the CLI package supports them cleanly, but the internal representation should be a set of artifact keys.
|
||||
|
||||
### Archive Behavior
|
||||
|
||||
Do not automatically archive every generated artifact.
|
||||
|
||||
Artifact generation and archive promotion should remain separate concerns. Operators should continue to use `archive.promote_artifacts` to decide which generated files should be promoted or uploaded.
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
archive:
|
||||
promote_artifacts:
|
||||
- from: artifacts/session_recap.md
|
||||
to: artifacts/session_recap.md
|
||||
required: true
|
||||
- from: artifacts/player_handout.md
|
||||
to: artifacts/player_handout.md
|
||||
required: false
|
||||
```
|
||||
|
||||
A later enhancement may add opt-in automatic promotion of configured artifacts, but explicit promotion should remain the default.
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Config Model and Defaults
|
||||
|
||||
Add or update the configured artifact model to include:
|
||||
|
||||
- `enabled`;
|
||||
- `depends_on`;
|
||||
- `prompt_id`;
|
||||
- `profile_id`;
|
||||
- `output_path`;
|
||||
- `timeout`;
|
||||
- `render_debug`;
|
||||
- `inputs`;
|
||||
- `vars`.
|
||||
|
||||
Add an internal default artifact output directory in `internal/config/defaults.go`, initially set to `artifacts`.
|
||||
|
||||
Validation rules:
|
||||
|
||||
- artifact names must match a conservative identifier pattern such as `^[a-z][a-z0-9_]*$`;
|
||||
- selected/executable artifacts require `prompt_id` and `output_path`;
|
||||
- configured artifacts that may be referenced while disabled require `output_path`;
|
||||
- configured artifact output paths must be run-relative;
|
||||
- configured artifact output paths must live under the internal artifact output directory;
|
||||
- configured artifact output paths must not escape the run workspace;
|
||||
- `narratio.artifact.<name>` input sources must refer to configured artifact keys;
|
||||
- any `narratio.artifact.<name>` input source must have a matching `depends_on` entry;
|
||||
- `depends_on` entries must refer to configured artifact keys;
|
||||
- dependencies must not contain self-references or executable cycles;
|
||||
- input names and var names must remain compatible with the Scriptorium adapter's validation rules;
|
||||
- unknown YAML fields must continue to fail strict decode.
|
||||
|
||||
Tests:
|
||||
|
||||
- valid single configured artifact;
|
||||
- valid multiple independent artifacts;
|
||||
- valid artifact-to-artifact dependency;
|
||||
- valid dependency on disabled artifact with output path;
|
||||
- invalid artifact name;
|
||||
- missing required fields;
|
||||
- output path outside `artifacts/`;
|
||||
- dependency on missing artifact;
|
||||
- missing `depends_on` for artifact input source;
|
||||
- self-dependency;
|
||||
- cycle detection;
|
||||
- typo in `narratio.artifact.<name>` source;
|
||||
- unknown YAML fields still fail strict decode.
|
||||
|
||||
### Phase 2: CLI Filtering
|
||||
|
||||
Add the `--artifacts` flag and carry the selected artifact set into the run execution options.
|
||||
|
||||
Implementation notes:
|
||||
|
||||
- parse values according to existing CLI conventions;
|
||||
- normalize to artifact key strings;
|
||||
- validate against configured artifact definitions after config load;
|
||||
- make the selected set available to the analyze stage;
|
||||
- reject use with commands or stages where analyze cannot run.
|
||||
|
||||
Tests:
|
||||
|
||||
- no `--artifacts` means all enabled artifacts are selected;
|
||||
- one requested artifact is selected;
|
||||
- multiple requested artifacts are selected;
|
||||
- unknown requested artifact fails;
|
||||
- `--artifacts` does not imply `--force`;
|
||||
- `--artifacts` with already-succeeded analyze stage is skipped unless forced;
|
||||
- `--artifacts` on unsupported stage command fails clearly.
|
||||
|
||||
### Phase 3: Runtime Artifact Catalog
|
||||
|
||||
Introduce an internal artifact catalog abstraction.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- register built-in artifact definitions;
|
||||
- register configured artifact definitions;
|
||||
- map configured artifact keys to `narratio.artifact.<name>` IDs;
|
||||
- track planned, available, and executable artifact states;
|
||||
- expose lookup by canonical source ID;
|
||||
- record generated provenance;
|
||||
- record disabled-from-disk provenance.
|
||||
|
||||
Keep the catalog narrow. It should not execute Scriptorium and should not understand prompt semantics.
|
||||
|
||||
Tests:
|
||||
|
||||
- built-in source lookup;
|
||||
- configured source registration;
|
||||
- duplicate/conflicting source handling;
|
||||
- planned but unavailable artifact lookup;
|
||||
- selected artifact state;
|
||||
- disabled artifact state;
|
||||
- registering an artifact as available after generation;
|
||||
- registering a disabled artifact as available from disk;
|
||||
- resolving a configured artifact from analyze metadata if that behavior is implemented.
|
||||
|
||||
### Phase 4: Resolver Integration
|
||||
|
||||
Update artifact resolution so configured artifact IDs are resolved through the runtime catalog.
|
||||
|
||||
Resolution behavior:
|
||||
|
||||
- built-in sources continue using existing resolver behavior;
|
||||
- configured artifact sources resolve from catalog availability/provenance;
|
||||
- selected configured artifacts become available after generation;
|
||||
- disabled configured artifacts may become available from disk;
|
||||
- missing optional configured artifact inputs are omitted;
|
||||
- missing required configured artifact inputs fail clearly.
|
||||
|
||||
Tests:
|
||||
|
||||
- configured artifact consumes a built-in transcript source;
|
||||
- configured artifact consumes another configured artifact produced earlier in the same analyze run;
|
||||
- configured artifact consumes a disabled artifact resolved from disk;
|
||||
- required disabled artifact missing on disk fails;
|
||||
- required configured artifact missing fails;
|
||||
- optional missing configured artifact is omitted;
|
||||
- reused artifact provenance is recorded distinctly from generated artifact provenance.
|
||||
|
||||
### Phase 5: Analyze Stage Generalization
|
||||
|
||||
Refactor `analyze` to execute selected configured artifacts.
|
||||
|
||||
Implementation notes:
|
||||
|
||||
- remove the hard-coded `session_recap` selection path;
|
||||
- remove the hard-coded rejection of non-`session_recap` artifacts;
|
||||
- preserve skip behavior when Scriptorium config is absent or no artifacts are selected;
|
||||
- build the runtime artifact catalog;
|
||||
- apply `--artifacts` filtering;
|
||||
- validate selected artifacts and their dependencies;
|
||||
- pre-resolve disabled dependencies from disk where required;
|
||||
- compute deterministic dependency order;
|
||||
- execute selected artifacts one at a time in dependency order;
|
||||
- keep render-debug behavior at global and artifact levels;
|
||||
- keep Scriptorium adapter invocation generic;
|
||||
- after each successful run, register the artifact as available in the catalog;
|
||||
- aggregate generated and reused artifact metadata.
|
||||
|
||||
Tests:
|
||||
|
||||
- no Scriptorium config skips;
|
||||
- empty artifact map skips;
|
||||
- no selected artifacts skips;
|
||||
- disabled artifacts do not run;
|
||||
- one selected artifact runs;
|
||||
- multiple independent artifacts run in deterministic order;
|
||||
- dependent selected artifact receives prior selected artifact as input;
|
||||
- dependent selected artifact receives disabled-from-disk artifact as input;
|
||||
- render-debug works for configured artifacts;
|
||||
- Scriptorium validation failure fails the stage;
|
||||
- missing required input fails the stage;
|
||||
- successful outputs are non-empty and recorded;
|
||||
- artifact filter executes only requested artifacts.
|
||||
|
||||
### Phase 6: Manifest and Stage Metadata
|
||||
|
||||
Update analyze-stage metadata and manifest output recording to support dynamic configured artifacts.
|
||||
|
||||
Recommended behavior:
|
||||
|
||||
- every generated configured artifact gets `source_id: narratio.artifact.<name>`;
|
||||
- every generated configured artifact gets a generic output kind such as `scriptorium_artifact`;
|
||||
- reused disabled artifacts are recorded separately from generated artifacts;
|
||||
- metadata is sufficient for debugging, provenance, and future resolver support;
|
||||
- metadata does not create per-artifact resume semantics.
|
||||
|
||||
Because this is a pre-release hard cutover, do not preserve a special legacy `session_recap` output kind unless a current internal test or archive path still requires it temporarily. Prefer updating tests and examples to treat `session_recap` as an ordinary configured artifact.
|
||||
|
||||
Tests:
|
||||
|
||||
- metadata records one generated configured artifact;
|
||||
- metadata records multiple generated configured artifacts;
|
||||
- metadata records reused disabled artifact provenance;
|
||||
- `session_recap` is recorded as a normal configured artifact;
|
||||
- manifest still treats `analyze` as a single succeeded or failed stage;
|
||||
- runner skip behavior remains stage-level.
|
||||
|
||||
### Phase 7: Archive and Promotion Review
|
||||
|
||||
Review archive behavior after dynamic artifacts are recorded.
|
||||
|
||||
Implementation notes:
|
||||
|
||||
- do not automatically promote every configured artifact;
|
||||
- keep `archive.promote_artifacts` explicit;
|
||||
- update default or example promotion rules to use configured `session_recap` output path;
|
||||
- ensure required promotion rules fail clearly when selected artifact generation did not produce a required file.
|
||||
|
||||
Tests:
|
||||
|
||||
- generated artifact can be promoted by explicit archive rule;
|
||||
- required archive promotion fails if selected artifact was not generated and no file exists;
|
||||
- optional archive promotion skips cleanly if file is absent;
|
||||
- hard cutover does not rely on hard-coded `session_recap` generation.
|
||||
|
||||
### Phase 8: Documentation and Examples
|
||||
|
||||
Status: complete.
|
||||
|
||||
Update documentation after the implementation is complete.
|
||||
|
||||
Recommended documentation changes:
|
||||
|
||||
- update `docs/config.md` with the generalized artifact configuration model;
|
||||
- update `docs/internal/artifacts.md` to describe the runtime artifact catalog;
|
||||
- update `docs/stages/analyze.md` to describe generic Scriptorium artifact generation;
|
||||
- update Scriptorium integration docs only if the adapter contract changes;
|
||||
- update full annotated pipeline examples;
|
||||
- add at least one example with multiple artifacts and one dependency;
|
||||
- document `--artifacts` behavior and its relationship to `--force`;
|
||||
- remove documentation stating that only `session_recap` is supported.
|
||||
|
||||
Documentation should make clear that:
|
||||
|
||||
- configured artifact source IDs use `narratio.artifact.<name>`;
|
||||
- `depends_on` uses artifact keys, not full source IDs;
|
||||
- artifact-to-artifact source references require explicit `depends_on`;
|
||||
- disabled artifacts can be reused from disk when required by selected artifacts;
|
||||
- `--artifacts` filters execution but does not imply `--force`;
|
||||
- archive promotion remains explicit;
|
||||
- per-artifact resume is not part of the initial implementation.
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
Because Narratio is pre-release, perform a hard cutover.
|
||||
|
||||
Required changes:
|
||||
|
||||
1. Remove the hard-coded `session_recap` analyze behavior.
|
||||
2. Require `session_recap` to be declared under `pipeline.scriptorium.artifacts.session_recap` if the operator wants a session recap.
|
||||
3. Treat `narratio.artifact.session_recap` as valid only when `session_recap` is a configured artifact key.
|
||||
4. Update config examples to show `session_recap` as a normal configured artifact.
|
||||
5. Update tests to stop assuming that `session_recap` is a built-in analyze artifact.
|
||||
6. Keep archive promotion explicit and path-based.
|
||||
|
||||
Example replacement config:
|
||||
|
||||
```yaml
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
config_path: /etc/scriptorium/config.yml
|
||||
timeout: 10m
|
||||
render_debug: false
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd_session.session_recap
|
||||
profile_id: local-gemma-31b
|
||||
output_path: artifacts/session_recap.md
|
||||
timeout: 20m
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
prior_recap:
|
||||
source: previous_session_artifact
|
||||
artifact: artifacts/session_recap.md
|
||||
required: false
|
||||
vars:
|
||||
artifact_title: Session Recap
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
The feature is complete when:
|
||||
|
||||
- operators can define more than one enabled Scriptorium artifact in `pipeline.yml`;
|
||||
- Narratio runs selected artifacts in deterministic dependency order;
|
||||
- configured artifacts are addressable as `narratio.artifact.<name>`;
|
||||
- one configured artifact can consume another configured artifact as an input;
|
||||
- artifact-to-artifact input references require explicit `depends_on`;
|
||||
- disabled artifacts can satisfy dependencies from existing on-disk outputs;
|
||||
- missing required disabled artifacts fail clearly;
|
||||
- optional missing inputs are omitted;
|
||||
- `--artifacts` can selectively execute valid configured artifact names;
|
||||
- `--artifacts` does not imply `--force`;
|
||||
- render-debug behavior works for all configured artifacts;
|
||||
- generated and reused artifacts are recorded in analyze-stage metadata;
|
||||
- `session_recap` is no longer hard-coded and works as a normal configured artifact;
|
||||
- archive promotion remains explicit;
|
||||
- tests cover config validation, dependency sorting, disabled artifact resolution, resolver behavior, CLI filtering, analyze execution, archive interactions, and metadata.
|
||||
|
||||
## Suggested Implementation Order
|
||||
|
||||
1. Config model, defaults, and validation.
|
||||
2. CLI parsing and propagation of `--artifacts` selection.
|
||||
3. Runtime artifact catalog.
|
||||
4. Resolver integration for configured artifacts.
|
||||
5. Analyze stage generalization.
|
||||
6. Stage metadata and manifest output recording.
|
||||
7. Archive behavior review.
|
||||
8. Documentation and examples.
|
||||
|
||||
This order keeps the most static pieces first, then moves into execution behavior once the configuration contract is explicit and well tested.
|
||||
@@ -1,119 +0,0 @@
|
||||
# Archive Storage
|
||||
|
||||
This document describes implemented archive-stage publish behavior.
|
||||
|
||||
## S3 Paths
|
||||
|
||||
Session root:
|
||||
|
||||
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
|
||||
Run prefix:
|
||||
|
||||
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
|
||||
|
||||
## Scope
|
||||
|
||||
Implemented:
|
||||
|
||||
- archive uploads successful run records to remote object storage through the storage backend abstraction.
|
||||
- archive uploads configured promoted outputs to session-level keys.
|
||||
- archive uploads `current/manifest.json`.
|
||||
- archive uploads `current/run_id.txt` last as the effective commit marker.
|
||||
- optional post-archive local cleanup:
|
||||
- `pipeline.spool.delete_audio_after_archive: true` removes only the run-scoped spool audio directory
|
||||
- `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir
|
||||
- tests use fake storage and do not require live S3.
|
||||
|
||||
Future work:
|
||||
|
||||
- `notify` stage behavior
|
||||
- stale detection
|
||||
- optional future source-audio upload mode
|
||||
- additional artifact generation beyond current implemented set
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Archive verifies these stages succeeded before upload:
|
||||
|
||||
- `prepare`
|
||||
- `transcribe`
|
||||
- `merge`
|
||||
- `polish`
|
||||
- `normalize`
|
||||
- `trim`
|
||||
- `analyze`
|
||||
|
||||
If any prerequisite is missing or not succeeded, archive fails and does not upload.
|
||||
Failed or incomplete runs remain local only.
|
||||
|
||||
## Run Upload
|
||||
|
||||
Archive uploads existing files from the run workdir when present:
|
||||
|
||||
- `inputs/`
|
||||
- `transcripts/`
|
||||
- `artifacts/`
|
||||
- `reports/` (optional)
|
||||
- `config/`
|
||||
- `logs/`
|
||||
- `manifest.json`
|
||||
|
||||
Relative paths are preserved under `runs/{run_id}/`.
|
||||
|
||||
## Promotion Rules
|
||||
|
||||
Archive applies `archive.promote_artifacts` in config order.
|
||||
|
||||
Rule behavior:
|
||||
|
||||
- `from`: local workdir-relative source path
|
||||
- `to`: session-root-relative destination key
|
||||
- `required: true`: missing source fails archive
|
||||
- `required: false`: missing source is skipped and recorded
|
||||
|
||||
Default promoted outputs:
|
||||
|
||||
- `transcripts/trimmed.json`
|
||||
- `artifacts/session_recap.md`
|
||||
|
||||
## Current Pointers
|
||||
|
||||
Archive writes:
|
||||
|
||||
1. `current/manifest.json` (after run upload + promotions)
|
||||
2. `current/run_id.txt` last
|
||||
|
||||
`current/run_id.txt` contains exactly:
|
||||
|
||||
- `{run_id}` plus trailing newline
|
||||
|
||||
Writing `current/run_id.txt` last makes it the effective commit marker for published session state.
|
||||
|
||||
If any required run upload, promotion upload, or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`.
|
||||
Cleanup runs only after this commit-marker write has succeeded.
|
||||
|
||||
## Audio Upload Policy
|
||||
|
||||
Archive does not upload local `audio/` by default.
|
||||
Original audio is expected at the session-level audio prefix and is not duplicated under `runs/{run_id}/`.
|
||||
|
||||
## Config Controls
|
||||
|
||||
- `archive.enabled: false` skips archive cleanly.
|
||||
- `archive.upload_run: false` skips run upload cleanly.
|
||||
- both skip cases also skip post-archive local cleanup.
|
||||
|
||||
## Metadata
|
||||
|
||||
Archive stage metadata includes non-secret upload context (for example):
|
||||
|
||||
- `s3_bucket`
|
||||
- `s3_run_prefix`
|
||||
- run upload counts/paths
|
||||
- promoted upload counts/paths
|
||||
- skipped optional promotions
|
||||
- `current_manifest_key`
|
||||
- `current_run_id_key`
|
||||
- `current_pointer_written`
|
||||
- `audio_upload_skipped`
|
||||
@@ -1,84 +0,0 @@
|
||||
# S3 Audio Input
|
||||
|
||||
This document describes implemented S3 audio input behavior in `prepare`.
|
||||
|
||||
## Scope
|
||||
|
||||
Implemented:
|
||||
|
||||
- `prepare` can acquire source audio from S3 when `session.inputs.audio_s3.prefix` is configured.
|
||||
- object listing and download go through the storage backend abstraction.
|
||||
- tests use fake storage; no live S3 service is required for test runs.
|
||||
|
||||
Not implemented:
|
||||
|
||||
- uploads of failed runs
|
||||
|
||||
## Required Configuration
|
||||
|
||||
`pipeline.yml`:
|
||||
|
||||
- `storage.s3.bucket` must be set when S3 audio input is used.
|
||||
- `storage.s3.root_prefix` defaults to `dnd`.
|
||||
- `storage.s3.access_key_id_env` defaults to `OBJECT_STORAGE_KEY_ID`.
|
||||
- `storage.s3.secret_access_key_env` defaults to `OBJECT_STORAGE_KEY`.
|
||||
- `spool.root` defaults to `/var/spool/narratio`.
|
||||
|
||||
`session.yml`:
|
||||
|
||||
- configure `session.campaign` and `session.session_id`.
|
||||
- configure `session.inputs.audio_s3.prefix` for S3 audio input.
|
||||
- do not configure `inputs.audio_dir` or `inputs.audio_files` at the same time as `inputs.audio_s3`.
|
||||
|
||||
## Prefix Shape
|
||||
|
||||
Session S3 root:
|
||||
|
||||
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
|
||||
Audio prefix:
|
||||
|
||||
`{session_root}/{audio_s3.prefix}`
|
||||
|
||||
Example:
|
||||
|
||||
`dnd/campaigns/forsaken/sessions/2026-04-19/audio/`
|
||||
|
||||
Audio files must already exist in S3 before running Narratio.
|
||||
|
||||
## Prepare Behavior
|
||||
|
||||
When `inputs.audio_s3.prefix` is configured, `prepare`:
|
||||
|
||||
1. lists objects under the computed S3 audio prefix
|
||||
2. filters to `.flac` objects
|
||||
3. fails when no `.flac` objects are found
|
||||
4. downloads selected objects to spool audio:
|
||||
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
5. materializes audio into workdir audio:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
|
||||
6. records input provenance in the manifest (bucket, key, metadata, local paths, checksum)
|
||||
|
||||
Notes:
|
||||
|
||||
- `.flac` filtering is case-insensitive.
|
||||
- ETag is recorded as provider metadata only and is not treated as a checksum.
|
||||
|
||||
## Local Audio Development
|
||||
|
||||
Local audio workflows remain supported:
|
||||
|
||||
- `inputs.audio_dir`
|
||||
- `inputs.audio_files`
|
||||
|
||||
These options are mutually exclusive with `inputs.audio_s3`.
|
||||
|
||||
## Archive Boundary
|
||||
|
||||
Current archive behavior relevant to S3 audio input:
|
||||
|
||||
- successful runs are uploaded by archive under `runs/{run_id}/`
|
||||
- configured promotions are uploaded to session-level destinations
|
||||
- `current/manifest.json` and `current/run_id.txt` are published
|
||||
- local source audio is not re-uploaded by default
|
||||
- failed or incomplete runs are not uploaded
|
||||
289
docs/troubleshooting.md
Normal file
289
docs/troubleshooting.md
Normal file
@@ -0,0 +1,289 @@
|
||||
# Troubleshooting
|
||||
|
||||
## Purpose
|
||||
Canonical operator troubleshooting guide for recurring implemented Narratio failures.
|
||||
|
||||
## Config file discovery failure
|
||||
|
||||
Symptom:
|
||||
- `run`, `plan`, `resume`, or `run-stage` fails with config/session not found.
|
||||
|
||||
Likely Cause:
|
||||
- `pipeline.yml` or `session.yml` is missing from discovery paths.
|
||||
- wrong working directory when relying on `./session.yml`.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
pwd
|
||||
ls -l ./session.yml
|
||||
ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- pass explicit `--config` and `--session`.
|
||||
- or place files in documented discovery paths.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/cli.md](./cli.md)
|
||||
|
||||
## Session template rendering failure
|
||||
|
||||
Symptom:
|
||||
- load fails with unresolved placeholder or `session_id` mismatch.
|
||||
|
||||
Likely Cause:
|
||||
- templated `session.yml` used without `--session-id`.
|
||||
- rendered `session_id` differs from passed `--session-id`.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio plan --session ./session.yml --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- pass `--session-id` when template placeholders are present.
|
||||
- ensure rendered `session_id` matches intended run session id.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## Strict YAML decode or validation failure
|
||||
|
||||
Symptom:
|
||||
- config load fails with unknown field or validation error.
|
||||
|
||||
Likely Cause:
|
||||
- typo/stale field name.
|
||||
- missing required fields or invalid constraints.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio plan --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- align fields/values to canonical config reference and examples.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [examples/](../examples/)
|
||||
|
||||
## `--artifacts` selection failure
|
||||
|
||||
Symptom:
|
||||
- `run`/`resume`/`run-stage` fails with invalid or unknown artifact selection.
|
||||
|
||||
Likely Cause:
|
||||
- `--artifacts` contains blank names or unknown artifact keys.
|
||||
- `pipeline.scriptorium.artifacts` missing while using `--artifacts`.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio run --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts player_handout
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- use configured artifact keys only.
|
||||
- ensure `pipeline.scriptorium.artifacts` is defined.
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## `run-stage --artifacts` on non-analyze stage
|
||||
|
||||
Symptom:
|
||||
- `run-stage` fails with `--artifacts is only supported for stage "analyze"`.
|
||||
|
||||
Likely Cause:
|
||||
- `--artifacts` was used with a non-`analyze` stage.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts session_recap polish
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- use `--artifacts` only with `run-stage ... analyze`.
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
|
||||
## Configured artifact dependency/input validation failure
|
||||
|
||||
Symptom:
|
||||
- config validation fails for `depends_on`, `narratio.artifact.<name>` source, or artifact output path.
|
||||
|
||||
Likely Cause:
|
||||
- `narratio.artifact.<name>` source missing matching `depends_on` key.
|
||||
- dependency references unknown artifact key.
|
||||
- dependency self-reference or enabled dependency cycle.
|
||||
- artifact output path missing/invalid/outside `artifacts/` root.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio plan --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- ensure artifact-to-artifact inputs have explicit `depends_on` entries using artifact keys.
|
||||
- ensure referenced artifacts exist and define valid `output_path` values.
|
||||
- keep output paths relative and under `artifacts/`.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/internal/stage-analyze.md](./internal/stage-analyze.md)
|
||||
|
||||
## Required configured artifact input unavailable at analyze time
|
||||
|
||||
Symptom:
|
||||
- analyze fails because configured input source is unavailable.
|
||||
|
||||
Likely Cause:
|
||||
- required upstream configured artifact was not selected/executed this run.
|
||||
- non-executable dependency output file is missing or invalid on disk.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio status --manifest /path/to/manifest.json
|
||||
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 --artifacts player_handout analyze
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- run analyze with needed artifacts selected.
|
||||
- or ensure dependency output file exists at configured path and is valid.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## Manifest/status path failure
|
||||
|
||||
Symptom:
|
||||
- `status` fails because manifest path is missing, unreadable, or invalid.
|
||||
|
||||
Likely Cause:
|
||||
- wrong manifest path.
|
||||
- manifest removed after cleanup.
|
||||
- `--manifest` omitted.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio status --manifest /path/to/manifest.json
|
||||
ls -l /path/to/manifest.json
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- use manifest path printed by `run`, `resume`, or `run-stage`.
|
||||
|
||||
Links:
|
||||
- [docs/cli.md](./cli.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
|
||||
## Session lock conflict (`.lock`)
|
||||
|
||||
Symptom:
|
||||
- run fails with lock conflict for session workdir.
|
||||
|
||||
Likely Cause:
|
||||
- another Narratio process is running same session.
|
||||
- stale lock from interrupted prior run.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
ls -l {workspace.root}/work/{campaign}/{session_id}/.lock
|
||||
cat {workspace.root}/work/{campaign}/{session_id}/.lock
|
||||
ps aux | grep narratio
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- wait for active run to finish.
|
||||
- if no process is active, remove only stale session `.lock` file.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/internal/workspace.md](./internal/workspace.md)
|
||||
|
||||
## Secrets env-dir or credential-env failure
|
||||
|
||||
Symptom:
|
||||
- startup fails loading secrets directory, or stage fails due to missing credential env vars.
|
||||
|
||||
Likely Cause:
|
||||
- invalid `pipeline.secrets.env_dir` path/permissions.
|
||||
- required credential env var unset/empty.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
ls -la /path/to/secrets_dir
|
||||
env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM'
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- fix secrets directory and credential env vars.
|
||||
- keep secret values out of YAML.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
|
||||
## S3-audio prepare failure
|
||||
|
||||
Symptom:
|
||||
- `prepare` fails in S3 mode (listing/downloading/no audio/backend error).
|
||||
|
||||
Likely Cause:
|
||||
- wrong `session.inputs.audio_s3.prefix`.
|
||||
- no `.flac` files at resolved prefix.
|
||||
- invalid/missing object-store credentials or backend config.
|
||||
- mixed local+S3 audio input config.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 prepare
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- configure exactly one audio source mode.
|
||||
- verify `.flac` files and storage access.
|
||||
|
||||
Links:
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/operations.md](./operations.md)
|
||||
|
||||
## Archive promotion/current-pointer failure
|
||||
|
||||
Symptom:
|
||||
- archive fails on required promotion source missing or pointer write failure.
|
||||
|
||||
Likely Cause:
|
||||
- required promoted file absent (including analyze outputs not generated for this run).
|
||||
- storage upload failed before `current/run_id.txt` commit marker write.
|
||||
|
||||
Diagnostics:
|
||||
|
||||
```bash
|
||||
narratio status --manifest /path/to/manifest.json
|
||||
narratio run-stage --config /path/to/pipeline.yml --session /path/to/session.yml --session-id 2026-04-04 archive
|
||||
```
|
||||
|
||||
Safe Fix:
|
||||
- rerun or resume upstream stages to generate required files.
|
||||
- adjust promotion rules to match files that must exist.
|
||||
- retry after storage issue is resolved.
|
||||
|
||||
Links:
|
||||
- [docs/operations.md](./operations.md)
|
||||
- [docs/config.md](./config.md)
|
||||
- [docs/internal/stage-archive.md](./internal/stage-archive.md)
|
||||
@@ -1,26 +0,0 @@
|
||||
workspace:
|
||||
root: ./tmp/narratio-workspace
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
|
||||
seriatim:
|
||||
binary: "seriatim"
|
||||
timeout: "10m"
|
||||
output_schema: "seriatim-intermediate"
|
||||
coalesce_gap: 3.0
|
||||
|
||||
audita:
|
||||
binary: "audita"
|
||||
timeout: "3h"
|
||||
base_url: "https://openrouter.ai/api/v1"
|
||||
model: "openrouter/google/gemma-4-31b-it"
|
||||
llm_api_key_env: "AUDITA_LLM_API_KEY"
|
||||
modules: ["glossary", "homophones", "spoken_word", "grammar"]
|
||||
output_schema: "audita-v1"
|
||||
work_dir_retention: "auto"
|
||||
total_llm_concurrency: 2
|
||||
proposal_llm_concurrency: 1
|
||||
validation_model: "openrouter/google/gemma-4-31b-it"
|
||||
validation_llm_concurrency: 1
|
||||
report: true
|
||||
182
examples/pipeline.full.annotated.yml
Normal file
182
examples/pipeline.full.annotated.yml
Normal file
@@ -0,0 +1,182 @@
|
||||
# Full annotated pipeline example for implemented Narratio config fields.
|
||||
# Values are safe placeholders and must be adapted per environment.
|
||||
|
||||
workspace:
|
||||
# Optional: defaults to /var/lib/narratio.
|
||||
root: /var/lib/narratio/workspace
|
||||
# Optional: remove run-scoped workdir after successful archive commit.
|
||||
cleanup_after_archive: false
|
||||
|
||||
# Optional: local secret file loader (directory of ENV_VAR_NAME files).
|
||||
# secrets:
|
||||
# env_dir: ./secrets
|
||||
|
||||
storage:
|
||||
# Optional storage backend selector; use "s3" for archive + S3 audio workflows.
|
||||
backend: s3
|
||||
# Compatibility fields retained in schema.
|
||||
bucket: ""
|
||||
prefix: ""
|
||||
s3:
|
||||
# Required when using S3 audio or S3 archive uploads.
|
||||
bucket: my-dnd-archive
|
||||
# Optional; defaults to "dnd".
|
||||
root_prefix: dnd
|
||||
# Optional region/endpoint settings.
|
||||
region: us-east-1
|
||||
endpoint: ""
|
||||
force_path_style: false
|
||||
# Optional; defaults shown explicitly.
|
||||
access_key_id_env: OBJECT_STORAGE_KEY_ID
|
||||
secret_access_key_env: OBJECT_STORAGE_KEY
|
||||
|
||||
spool:
|
||||
# Optional; defaults to /var/spool/narratio.
|
||||
root: /var/spool/narratio
|
||||
# Optional cleanup of run-scoped spool audio after successful archive commit.
|
||||
delete_audio_after_archive: false
|
||||
|
||||
archive:
|
||||
# Optional booleans; defaults are true.
|
||||
enabled: true
|
||||
upload_run: true
|
||||
# Optional promotion rules; required files fail archive if missing.
|
||||
promote_artifacts:
|
||||
- from: transcripts/trimmed.json
|
||||
to: transcripts/trimmed.json
|
||||
required: true
|
||||
- from: artifacts/session_recap.md
|
||||
to: artifacts/session_recap.md
|
||||
required: true
|
||||
- from: artifacts/player_handout.md
|
||||
to: artifacts/player_handout.md
|
||||
required: false
|
||||
|
||||
whisperx:
|
||||
# Required.
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
# Optional overrides; defaults shown explicitly.
|
||||
language: en
|
||||
timeout: 30m
|
||||
retries: 3
|
||||
retry_delay: 2s
|
||||
concurrency: 2
|
||||
|
||||
seriatim:
|
||||
# Optional overrides; defaults shown explicitly.
|
||||
binary: seriatim
|
||||
timeout: 10m
|
||||
output_schema: seriatim-intermediate
|
||||
coalesce_gap: 3.0
|
||||
report: true
|
||||
env:
|
||||
# Optional advanced tuning; set only when needed.
|
||||
overlap_word_run_gap: 1.0
|
||||
overlap_word_run_reorder_window: 1.0
|
||||
backchannel_max_duration: 2.0
|
||||
filler_max_duration: 1.25
|
||||
|
||||
audita:
|
||||
# Optional overrides; defaults shown explicitly where applicable.
|
||||
binary: audita
|
||||
timeout: 3h
|
||||
llm_api_key_env: AUDITA_LLM_API_KEY
|
||||
modules: [glossary, homophones, spoken_word, grammar]
|
||||
base_url: ""
|
||||
model: ""
|
||||
total_llm_concurrency: 2
|
||||
proposal_llm_concurrency: 1
|
||||
validation_model: ""
|
||||
validation_llm_concurrency: 1
|
||||
transcript_description: ""
|
||||
config_path: /usr/local/etc/audita/config.yml
|
||||
output_schema: audita-v1
|
||||
work_dir_retention: auto
|
||||
report: true
|
||||
|
||||
normalize:
|
||||
# Optional; defaults shown explicitly.
|
||||
output_path: transcripts/normalized.json
|
||||
output_schema: seriatim-intermediate
|
||||
report: true
|
||||
|
||||
trim:
|
||||
# Keep disabled unless bounds prompt integration is configured.
|
||||
enabled: false
|
||||
output_path: transcripts/trimmed.json
|
||||
bounds:
|
||||
prompt_id: dnd.session_bounds
|
||||
profile_id: local-fast
|
||||
transcript_input_name: transcript
|
||||
output_path: reports/session_bounds.json
|
||||
timeout: 10m
|
||||
render_debug: false
|
||||
render_output_path: reports/session_bounds.render.json
|
||||
seriatim:
|
||||
report: false
|
||||
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
config_path: /usr/local/etc/scriptorium/config.yml
|
||||
timeout: 10m
|
||||
render_debug: false
|
||||
artifacts:
|
||||
# Configured artifact keys map to source IDs narratio.artifact.<key>.
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
profile_id: local-fast
|
||||
output_path: artifacts/session_recap.md
|
||||
timeout: 10m
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
previous_recap:
|
||||
source: previous_session_artifact
|
||||
artifact: session_recap
|
||||
path: ""
|
||||
required: false
|
||||
vars:
|
||||
session_id: true
|
||||
session_date: true
|
||||
campaign_name: true
|
||||
previous_session_id: true
|
||||
output_kind: session_recap
|
||||
|
||||
# Example dependent artifact:
|
||||
# - depends_on entries use artifact keys.
|
||||
# - narratio.artifact.<key> sources require matching depends_on membership.
|
||||
player_handout:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- session_recap
|
||||
prompt_id: dnd.player_handout
|
||||
profile_id: local-fast
|
||||
output_path: artifacts/player_handout.md
|
||||
timeout: 10m
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
vars:
|
||||
session_id: true
|
||||
campaign_name: true
|
||||
output_kind: player_handout
|
||||
|
||||
analyzer:
|
||||
# Optional adapter settings.
|
||||
binary_path: ""
|
||||
timeout: 2m
|
||||
artifacts:
|
||||
output_dir: ""
|
||||
types: []
|
||||
|
||||
notification:
|
||||
# Optional notification settings.
|
||||
backend: ""
|
||||
recipient: ""
|
||||
timeout: 30s
|
||||
@@ -1,55 +1,2 @@
|
||||
workspace:
|
||||
root: ./tmp/narratio-workspace
|
||||
cleanup_after_archive: false
|
||||
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: "my-dnd-archive"
|
||||
root_prefix: "dnd"
|
||||
region: "us-east-1"
|
||||
# Optional credential env-var names (defaulted when omitted):
|
||||
# access_key_id_env: "OBJECT_STORAGE_KEY_ID"
|
||||
# secret_access_key_env: "OBJECT_STORAGE_KEY"
|
||||
|
||||
spool:
|
||||
root: "/var/spool/narratio"
|
||||
delete_audio_after_archive: false
|
||||
|
||||
archive:
|
||||
enabled: true
|
||||
upload_run: true
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
|
||||
# Optional. When omitted entirely, Narratio defaults to seriatim binary + runtime defaults.
|
||||
seriatim: {}
|
||||
|
||||
# Optional runtime overrides. Model/provider can be owned by Audita runtime config.
|
||||
audita:
|
||||
config_path: "/usr/local/etc/audita/config.yml"
|
||||
llm_api_key_env: "AUDITA_LLM_API_KEY"
|
||||
|
||||
# Optional Scriptorium integration for analyze artifacts.
|
||||
scriptorium:
|
||||
config_path: "/usr/local/etc/scriptorium/config.yml"
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: "dnd.session_recap"
|
||||
output_path: "artifacts/session_recap.md"
|
||||
inputs:
|
||||
transcript:
|
||||
source: "trimmed_transcript"
|
||||
required: true
|
||||
previous_recap:
|
||||
source: "previous_session_artifact"
|
||||
artifact: "session_recap"
|
||||
required: false
|
||||
vars:
|
||||
session_id: true
|
||||
session_date: true
|
||||
campaign_name: true
|
||||
previous_session_id: true
|
||||
output_kind: "session_recap"
|
||||
|
||||
116
examples/pipeline.production.yml
Normal file
116
examples/pipeline.production.yml
Normal file
@@ -0,0 +1,116 @@
|
||||
workspace:
|
||||
root: /var/lib/narratio/workspace
|
||||
cleanup_after_archive: true
|
||||
|
||||
storage:
|
||||
backend: s3
|
||||
s3:
|
||||
bucket: my-dnd-archive
|
||||
root_prefix: dnd
|
||||
region: us-east-1
|
||||
access_key_id_env: OBJECT_STORAGE_KEY_ID
|
||||
secret_access_key_env: OBJECT_STORAGE_KEY
|
||||
|
||||
spool:
|
||||
root: /var/spool/narratio
|
||||
delete_audio_after_archive: true
|
||||
|
||||
archive:
|
||||
enabled: true
|
||||
upload_run: true
|
||||
promote_artifacts:
|
||||
- from: transcripts/trimmed.json
|
||||
to: transcripts/trimmed.json
|
||||
required: true
|
||||
- from: artifacts/session_recap.md
|
||||
to: artifacts/session_recap.md
|
||||
required: true
|
||||
- from: artifacts/player_handout.md
|
||||
to: artifacts/player_handout.md
|
||||
required: false
|
||||
|
||||
whisperx:
|
||||
transcribe_url: "https://transcription.example.com/transcribe"
|
||||
language: en
|
||||
timeout: 45m
|
||||
retries: 3
|
||||
retry_delay: 3s
|
||||
concurrency: 2
|
||||
|
||||
seriatim:
|
||||
binary: seriatim
|
||||
timeout: 10m
|
||||
output_schema: seriatim-intermediate
|
||||
coalesce_gap: 3.0
|
||||
report: true
|
||||
|
||||
audita:
|
||||
binary: audita
|
||||
timeout: 3h
|
||||
llm_api_key_env: AUDITA_LLM_API_KEY
|
||||
modules: [glossary, homophones, spoken_word, grammar]
|
||||
output_schema: audita-v1
|
||||
work_dir_retention: auto
|
||||
total_llm_concurrency: 2
|
||||
proposal_llm_concurrency: 1
|
||||
validation_llm_concurrency: 1
|
||||
report: true
|
||||
|
||||
normalize:
|
||||
output_path: transcripts/normalized.json
|
||||
output_schema: seriatim-intermediate
|
||||
report: true
|
||||
|
||||
trim:
|
||||
enabled: false
|
||||
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
config_path: /usr/local/etc/scriptorium/config.yml
|
||||
timeout: 10m
|
||||
render_debug: false
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
profile_id: local-fast
|
||||
output_path: artifacts/session_recap.md
|
||||
timeout: 10m
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
previous_recap:
|
||||
source: previous_session_artifact
|
||||
artifact: session_recap
|
||||
required: false
|
||||
vars:
|
||||
session_id: true
|
||||
session_date: true
|
||||
campaign_name: true
|
||||
previous_session_id: true
|
||||
output_kind: session_recap
|
||||
player_handout:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- session_recap
|
||||
prompt_id: dnd.player_handout
|
||||
profile_id: local-fast
|
||||
output_path: artifacts/player_handout.md
|
||||
timeout: 10m
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
vars:
|
||||
session_id: true
|
||||
output_kind: player_handout
|
||||
|
||||
analyzer:
|
||||
timeout: 2m
|
||||
|
||||
notification:
|
||||
timeout: 30s
|
||||
9
examples/session.local-audio.yml
Normal file
9
examples/session.local-audio.yml
Normal file
@@ -0,0 +1,9 @@
|
||||
session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
date: 2026-05-03
|
||||
title: Sample Session
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./examples/speakers.yml
|
||||
autocorrect_file: ./examples/autocorrect.yml
|
||||
glossary_file: ./examples/glossary.yml
|
||||
@@ -1,13 +0,0 @@
|
||||
session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
date: 2026-05-03
|
||||
title: Sample Session
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
# Optional S3 input alternative. Do not configure with audio_dir/audio_files.
|
||||
# Narratio prepare lists this prefix and downloads .flac files.
|
||||
# audio_s3:
|
||||
# prefix: "audio/"
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
10
examples/session.s3-audio.yml
Normal file
10
examples/session.s3-audio.yml
Normal file
@@ -0,0 +1,10 @@
|
||||
session_id: 2026-05-03
|
||||
campaign: sample-campaign
|
||||
date: 2026-05-03
|
||||
title: Sample Session
|
||||
inputs:
|
||||
audio_s3:
|
||||
prefix: audio/
|
||||
speakers_file: ./examples/speakers.yml
|
||||
autocorrect_file: ./examples/autocorrect.yml
|
||||
glossary_file: ./examples/glossary.yml
|
||||
@@ -1,12 +1,7 @@
|
||||
session_id: "{{ session_id }}"
|
||||
campaign: sample-campaign
|
||||
date: ""
|
||||
title: ""
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
# Optional S3 input alternative. Do not configure with audio_dir/audio_files.
|
||||
# audio_s3:
|
||||
# prefix: "audio/{{ session_id }}/"
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
speakers_file: ./examples/speakers.yml
|
||||
autocorrect_file: ./examples/autocorrect.yml
|
||||
glossary_file: ./examples/glossary.yml
|
||||
|
||||
2
go.mod
2
go.mod
@@ -4,6 +4,7 @@ go 1.25.0
|
||||
|
||||
require (
|
||||
github.com/aws/aws-sdk-go-v2/config v1.32.17
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.16
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0
|
||||
github.com/aws/smithy-go v1.25.1
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
@@ -12,7 +13,6 @@ require (
|
||||
require (
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.7 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.16 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.23 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 // indirect
|
||||
|
||||
66
internal/app/analyze_artifacts.go
Normal file
66
internal/app/analyze_artifacts.go
Normal file
@@ -0,0 +1,66 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
type artifactSelectionFlag struct {
|
||||
values []string
|
||||
}
|
||||
|
||||
func (f *artifactSelectionFlag) String() string {
|
||||
return strings.Join(f.values, ",")
|
||||
}
|
||||
|
||||
func (f *artifactSelectionFlag) Set(value string) error {
|
||||
f.values = append(f.values, value)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *artifactSelectionFlag) Normalize() ([]string, error) {
|
||||
if len(f.values) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
seen := map[string]struct{}{}
|
||||
out := make([]string, 0, len(f.values))
|
||||
for _, raw := range f.values {
|
||||
for _, part := range strings.Split(raw, ",") {
|
||||
name := strings.TrimSpace(part)
|
||||
if name == "" {
|
||||
return nil, fmt.Errorf("artifact names must be non-empty")
|
||||
}
|
||||
if _, ok := seen[name]; ok {
|
||||
continue
|
||||
}
|
||||
seen[name] = struct{}{}
|
||||
out = append(out, name)
|
||||
}
|
||||
}
|
||||
|
||||
sort.Strings(out)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func validateSelectedAnalyzeArtifacts(cfg *config.Config, selected []string) error {
|
||||
if len(selected) == 0 {
|
||||
return nil
|
||||
}
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Scriptorium == nil {
|
||||
return fmt.Errorf("--artifacts requires pipeline.scriptorium.artifacts to be configured")
|
||||
}
|
||||
configured := cfg.Pipeline.Scriptorium.Artifacts
|
||||
if len(configured) == 0 {
|
||||
return fmt.Errorf("--artifacts requires at least one configured artifact in pipeline.scriptorium.artifacts")
|
||||
}
|
||||
for _, name := range selected {
|
||||
if _, ok := configured[name]; !ok {
|
||||
return fmt.Errorf("--artifacts includes unknown artifact %q", name)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
140
internal/app/analyze_artifacts_commands_test.go
Normal file
140
internal/app/analyze_artifacts_commands_test.go
Normal file
@@ -0,0 +1,140 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
func TestExecuteRunStageArtifactsNonAnalyzeFails(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{"run-stage", "--config", pipelinePath, "--session", sessionPath, "--artifacts", "session_recap", "polish"},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `run-stage: --artifacts is only supported for stage "analyze"`) {
|
||||
t.Fatalf("stderr = %q, want stage-gating error", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteUnknownArtifactsFailValidation(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := Execute(
|
||||
[]string{"run", "--config", pipelinePath, "--session", sessionPath, "--artifacts", "unknown_artifact"},
|
||||
&stdout,
|
||||
&stderr,
|
||||
)
|
||||
if code == 0 {
|
||||
t.Fatal("exit code = 0, want non-zero")
|
||||
}
|
||||
if !strings.Contains(stderr.String(), `run: --artifacts includes unknown artifact "unknown_artifact"`) {
|
||||
t.Fatalf("stderr = %q, want unknown-artifact validation error", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunStageArtifactsDoesNotImplyForce(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
seed.MarkStageSucceeded("analyze", time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||
t.Fatalf("save manifest: %v", err)
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
err := RunStage(
|
||||
context.Background(),
|
||||
[]string{"--config", pipelinePath, "--session", sessionPath, "--artifacts", "session_recap,session_recap", "analyze"},
|
||||
&out,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("RunStage() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "stage=analyze executed=0 skipped=1 force=false") {
|
||||
t.Fatalf("output = %q, want analyze skip without force", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestResumeArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
|
||||
seed.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||
t.Fatalf("save manifest: %v", err)
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
err := Resume(
|
||||
context.Background(),
|
||||
[]string{"--config", pipelinePath, "--session", sessionPath, "--artifacts", "session_recap"},
|
||||
&out,
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("Resume() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "has no remaining stages") {
|
||||
t.Fatalf("output = %q, want no remaining stages", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func writeValidConfigFilesWithScriptoriumArtifacts(t *testing.T, workspaceRoot string) (string, string) {
|
||||
t.Helper()
|
||||
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
f, err := os.OpenFile(pipelinePath, os.O_APPEND|os.O_WRONLY, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("open pipeline config for append: %v", err)
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
extra := `
|
||||
scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
player_handout:
|
||||
enabled: true
|
||||
prompt_id: dnd.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
depends_on:
|
||||
- session_recap
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
`
|
||||
if _, err := f.WriteString(extra); err != nil {
|
||||
t.Fatalf("append scriptorium config: %v", err)
|
||||
}
|
||||
return pipelinePath, sessionPath
|
||||
}
|
||||
132
internal/app/analyze_artifacts_test.go
Normal file
132
internal/app/analyze_artifacts_test.go
Normal file
@@ -0,0 +1,132 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
)
|
||||
|
||||
func TestArtifactSelectionFlagNormalize(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
inputs []string
|
||||
want []string
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "single value",
|
||||
inputs: []string{"session_recap"},
|
||||
want: []string{"session_recap"},
|
||||
},
|
||||
{
|
||||
name: "repeatable and comma separated values are deduped and sorted",
|
||||
inputs: []string{"session_recap,player_handout", "session_recap"},
|
||||
want: []string{"player_handout", "session_recap"},
|
||||
},
|
||||
{
|
||||
name: "empty token fails",
|
||||
inputs: []string{"session_recap,"},
|
||||
wantErr: "artifact names must be non-empty",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
var flag artifactSelectionFlag
|
||||
for _, in := range tt.inputs {
|
||||
if err := flag.Set(in); err != nil {
|
||||
t.Fatalf("Set(%q) error = %v", in, err)
|
||||
}
|
||||
}
|
||||
|
||||
got, err := flag.Normalize()
|
||||
if tt.wantErr != "" {
|
||||
if err == nil {
|
||||
t.Fatalf("Normalize() error = nil, want %q", tt.wantErr)
|
||||
}
|
||||
if err.Error() != tt.wantErr {
|
||||
t.Fatalf("Normalize() error = %q, want %q", err.Error(), tt.wantErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("Normalize() error = %v", err)
|
||||
}
|
||||
if len(got) != len(tt.want) {
|
||||
t.Fatalf("Normalize() len = %d, want %d; got=%v", len(got), len(tt.want), got)
|
||||
}
|
||||
for i := range got {
|
||||
if got[i] != tt.want[i] {
|
||||
t.Fatalf("Normalize()[%d] = %q, want %q", i, got[i], tt.want[i])
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateSelectedAnalyzeArtifacts(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
cfg *config.Config
|
||||
selected []string
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "empty selection is accepted",
|
||||
cfg: &config.Config{},
|
||||
selected: nil,
|
||||
},
|
||||
{
|
||||
name: "scriptorium required when selected artifacts present",
|
||||
cfg: &config.Config{Pipeline: &config.PipelineConfig{}},
|
||||
selected: []string{"session_recap"},
|
||||
wantErr: "--artifacts requires pipeline.scriptorium.artifacts to be configured",
|
||||
},
|
||||
{
|
||||
name: "unknown selected artifact fails",
|
||||
cfg: &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Scriptorium: &config.ScriptoriumConfig{
|
||||
Artifacts: map[string]config.ScriptoriumArtifactConfig{
|
||||
"session_recap": {Enabled: true, PromptID: "dnd.session_recap", OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
selected: []string{"player_handout"},
|
||||
wantErr: `--artifacts includes unknown artifact "player_handout"`,
|
||||
},
|
||||
{
|
||||
name: "known selected artifacts are accepted",
|
||||
cfg: &config.Config{
|
||||
Pipeline: &config.PipelineConfig{
|
||||
Scriptorium: &config.ScriptoriumConfig{
|
||||
Artifacts: map[string]config.ScriptoriumArtifactConfig{
|
||||
"session_recap": {Enabled: true, PromptID: "dnd.session_recap", OutputPath: "artifacts/session_recap.md"},
|
||||
"player_handout": {Enabled: true, PromptID: "dnd.player_handout", OutputPath: "artifacts/player_handout.md"},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
selected: []string{"player_handout", "session_recap"},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := validateSelectedAnalyzeArtifacts(tt.cfg, tt.selected)
|
||||
if tt.wantErr != "" {
|
||||
if err == nil {
|
||||
t.Fatalf("error = nil, want %q", tt.wantErr)
|
||||
}
|
||||
if err.Error() != tt.wantErr {
|
||||
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("error = %v, want nil", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -111,7 +111,7 @@ func TestExecuteRunStageUnknownFails(t *testing.T) {
|
||||
func TestExecuteRunStageNormalizeIsAccepted(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, "https://example.com/transcribe")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
@@ -156,7 +156,7 @@ func TestExecuteRunStageTranscribeUsesConfiguredWhisperXServer(t *testing.T) {
|
||||
t.Fatal("expected whisperx server to be called at least once")
|
||||
}
|
||||
|
||||
outPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "transcripts", "raw", "alice.json")
|
||||
outPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "transcripts", "raw", "alice.json")
|
||||
data, err := os.ReadFile(outPath)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile(%q): %v", outPath, err)
|
||||
@@ -233,7 +233,7 @@ inputs:
|
||||
_ = os.Chdir(originalWD)
|
||||
})
|
||||
|
||||
workRoot := filepath.Join(workspaceRoot, "work", sessionID)
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", sessionID)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"schema":"seriatim-intermediate","segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "[]\n")
|
||||
|
||||
|
||||
@@ -57,7 +57,7 @@ func Plan(ctx context.Context, args []string, out io.Writer) error {
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
|
||||
paths, err := store.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, err := store.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("plan: prepare workdir: %w", err)
|
||||
}
|
||||
|
||||
@@ -36,7 +36,7 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
|
||||
t.Fatalf("first output = %q, want totals", got)
|
||||
}
|
||||
|
||||
sessionWorkdir := artifacts.SessionWorkDir(workspaceRoot, "2026-05-03")
|
||||
sessionWorkdir := artifacts.SessionWorkDirForCampaign(workspaceRoot, "sample-campaign", "2026-05-03")
|
||||
expectedDirs := []string{
|
||||
sessionWorkdir,
|
||||
filepath.Join(sessionWorkdir, "inputs"),
|
||||
@@ -63,7 +63,7 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
|
||||
func TestPlanShowsRunAndSkipFromManifest(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
|
||||
@@ -54,7 +54,7 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
||||
}
|
||||
workDir := strings.TrimSpace(m.LocalWorkDir)
|
||||
if workDir == "" {
|
||||
workDir = artifacts.SessionRunWorkDir(
|
||||
workDir = artifacts.SessionRunRootForCampaign(
|
||||
env.Config.Pipeline.Workspace.Root,
|
||||
strings.TrimSpace(env.Config.Session.Campaign),
|
||||
strings.TrimSpace(env.Config.Session.SessionID),
|
||||
|
||||
@@ -210,7 +210,7 @@ func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
|
||||
assertExists(t, seed.spoolAudioDir)
|
||||
assertExists(t, seed.runWorkDir)
|
||||
assertExists(t, filepath.Join(seed.runWorkDir, "manifest.json"))
|
||||
assertExists(t, artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
|
||||
assertExists(t, artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
|
||||
}
|
||||
|
||||
func TestPostArchiveCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
||||
@@ -271,8 +271,8 @@ func cleanupFixtureConfig(t *testing.T) (*config.Config, cleanupSeed) {
|
||||
cfg.Pipeline.Spool.Root = filepath.Join(t.TempDir(), "spool")
|
||||
|
||||
runID := "20260516T010203Z-1a2b3c4d"
|
||||
runWorkDir := artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||
otherRunDir := artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, "20260516T010204Z-5e6f7a8b")
|
||||
runWorkDir := artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||
otherRunDir := artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, "20260516T010204Z-5e6f7a8b")
|
||||
spoolAudioDir := artifacts.SessionSpoolAudioDir(cfg.Pipeline.Spool.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID)
|
||||
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "manifest.json"), "{}\n")
|
||||
@@ -326,7 +326,11 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
||||
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md", Required: boolPtr(true)},
|
||||
},
|
||||
}
|
||||
writeArchiveFixtureRunFiles(t, seed.runWorkDir)
|
||||
writeArchiveFixtureRunFiles(
|
||||
t,
|
||||
seed.runWorkDir,
|
||||
artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID),
|
||||
)
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
seedManifest, err := store.Load(context.Background(), manifestPathFor(cfg))
|
||||
@@ -345,16 +349,19 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
||||
return cfg, seed, runID
|
||||
}
|
||||
|
||||
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir string) {
|
||||
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir, sessionRoot string) {
|
||||
t.Helper()
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcripts", "trimmed.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "artifacts", "session_recap.md"), "# recap\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "reports", "audita.report.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "config", "audita.generated.yml"), "key: value\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "prepare", "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "trim", "outputs", "transcripts", "trimmed.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "analyze", "outputs", "artifacts", "session_recap.md"), "# recap\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "polish", "reports", "audita.report.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "merge", "config", "seriatim.generated.yml"), "key: value\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "logs", "audita.stderr.log"), "stderr\n")
|
||||
mustWriteFile(t, filepath.Join(runWorkDir, "manifest.json"), "{}\n")
|
||||
|
||||
mustWriteFile(t, filepath.Join(sessionRoot, "transcripts", "trimmed.json"), "{}\n")
|
||||
mustWriteFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# recap\n")
|
||||
}
|
||||
|
||||
type failKeyStore struct {
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
@@ -19,10 +20,12 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
|
||||
var sessionPath string
|
||||
var sessionID string
|
||||
var force bool
|
||||
var selectedArtifacts artifactSelectionFlag
|
||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
||||
fs.StringVar(&sessionID, "session-id", "", "session identifier for session.yml templates")
|
||||
fs.BoolVar(&force, "force", false, "force stage execution")
|
||||
fs.Var(&selectedArtifacts, "artifacts", "artifact names to execute during analyze (comma-separated or repeatable)")
|
||||
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("resume: invalid flags: %w", err)
|
||||
@@ -48,6 +51,13 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
return fmt.Errorf("resume: %w", err)
|
||||
}
|
||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
||||
if err != nil {
|
||||
return fmt.Errorf("resume: invalid --artifacts: %w", err)
|
||||
}
|
||||
if err := validateSelectedAnalyzeArtifacts(cfg, normalizedArtifacts); err != nil {
|
||||
return fmt.Errorf("resume: %w", err)
|
||||
}
|
||||
|
||||
full := BuildFullPlan()
|
||||
selected := full
|
||||
@@ -66,7 +76,10 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
|
||||
}
|
||||
}
|
||||
|
||||
summary, err := executeStages(ctx, cfg, selected, RunOptions{Force: force})
|
||||
summary, err := executeStages(ctx, cfg, selected, RunOptions{
|
||||
Force: force,
|
||||
SelectedArtifacts: normalizedArtifacts,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("resume: %w", err)
|
||||
}
|
||||
@@ -83,7 +96,11 @@ func Resume(ctx context.Context, args []string, out io.Writer) error {
|
||||
}
|
||||
|
||||
func loadManifestIfPresent(ctx context.Context, cfg *config.Config) (*manifest.Manifest, error) {
|
||||
path := manifestPathFor(cfg)
|
||||
path := artifacts.SessionManifestPathForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
)
|
||||
exists, err := fileExists(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("check manifest %q: %w", path, err)
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
@@ -25,7 +25,7 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
||||
if err := store.Save(context.Background(), manifestPath, m); err != nil {
|
||||
t.Fatalf("save manifest: %v", err)
|
||||
}
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "raw", "alice.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "autocorrect.yml"), "[]\n")
|
||||
@@ -52,7 +52,7 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
||||
func TestResumeNoRemainingStages(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
@@ -81,7 +81,7 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
|
||||
}))
|
||||
defer srv.Close()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
@@ -105,8 +105,8 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
|
||||
func TestRunStageExecutesOnlySelectedStage(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||
|
||||
@@ -135,8 +135,8 @@ func TestRunStageExecutesOnlySelectedStage(t *testing.T) {
|
||||
func TestRunStageSkipAndForce(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||
|
||||
@@ -166,11 +166,57 @@ func TestRunStageSkipAndForce(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunStageForceMarksDownstreamStaleAndResumeContinuesFromStale(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "merged.json"), `{"segments":[]}`)
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "archive", "notify"} {
|
||||
seed.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||
t.Fatalf("save manifest: %v", err)
|
||||
}
|
||||
|
||||
var out bytes.Buffer
|
||||
err := RunStage(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath, "--force", "polish"}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("RunStage(force) error = %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "stage=polish executed=1 skipped=0 force=true") {
|
||||
t.Fatalf("output = %q, want forced polish rerun", out.String())
|
||||
}
|
||||
|
||||
afterForce, err := store.Load(context.Background(), manifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("load manifest after force: %v", err)
|
||||
}
|
||||
for _, name := range []string{"normalize", "trim", "analyze", "archive", "notify"} {
|
||||
if afterForce.Stages[name] == nil || afterForce.Stages[name].Status != manifest.StatusStale {
|
||||
t.Fatalf("stage %q = %#v, want stale", name, afterForce.Stages[name])
|
||||
}
|
||||
}
|
||||
|
||||
out.Reset()
|
||||
err = Resume(context.Background(), []string{"--config", pipelinePath, "--session", sessionPath}, &out)
|
||||
if err != nil {
|
||||
t.Fatalf("Resume() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(out.String(), "executed=5 skipped=0") {
|
||||
t.Fatalf("output = %q, want resume to execute normalize..notify", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunStageTrimExecutes(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "normalized.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
|
||||
var out bytes.Buffer
|
||||
@@ -198,8 +244,8 @@ func TestRunStageTrimExecutes(t *testing.T) {
|
||||
func TestRunStageNormalizeExecutes(t *testing.T) {
|
||||
workspaceRoot := t.TempDir()
|
||||
pipelinePath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "2026-05-03")
|
||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "processed.json"), `{"segments":[{"id":1},{"id":2}]}`)
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
@@ -18,10 +18,12 @@ func Run(ctx context.Context, args []string, out io.Writer) error {
|
||||
var sessionPath string
|
||||
var sessionID string
|
||||
var force bool
|
||||
var selectedArtifacts artifactSelectionFlag
|
||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
||||
fs.StringVar(&sessionID, "session-id", "", "session identifier for session.yml templates")
|
||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
||||
fs.Var(&selectedArtifacts, "artifacts", "artifact names to execute during analyze (comma-separated or repeatable)")
|
||||
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("run: invalid flags: %w", err)
|
||||
@@ -47,9 +49,19 @@ func Run(ctx context.Context, args []string, out io.Writer) error {
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
return fmt.Errorf("run: %w", err)
|
||||
}
|
||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
||||
if err != nil {
|
||||
return fmt.Errorf("run: invalid --artifacts: %w", err)
|
||||
}
|
||||
if err := validateSelectedAnalyzeArtifacts(cfg, normalizedArtifacts); err != nil {
|
||||
return fmt.Errorf("run: %w", err)
|
||||
}
|
||||
|
||||
stages := BuildFullPlan()
|
||||
summary, err := executeStages(ctx, cfg, stages, RunOptions{Force: force})
|
||||
summary, err := executeStages(ctx, cfg, stages, RunOptions{
|
||||
Force: force,
|
||||
SelectedArtifacts: normalizedArtifacts,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("run: %w", err)
|
||||
}
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||
)
|
||||
@@ -49,3 +51,43 @@ func firstNonSucceededIndex(stages []stage.Stage, m *manifest.Manifest) int {
|
||||
}
|
||||
return len(stages)
|
||||
}
|
||||
|
||||
func canonicalStageNames() []string {
|
||||
all := stage.All()
|
||||
out := make([]string, 0, len(all))
|
||||
for _, s := range all {
|
||||
if s == nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, s.Name())
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func downstreamStageNames(stageName string) []string {
|
||||
names := canonicalStageNames()
|
||||
for i, name := range names {
|
||||
if name != stageName {
|
||||
continue
|
||||
}
|
||||
return append([]string(nil), names[i+1:]...)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func invalidateDownstreamSucceededStages(m *manifest.Manifest, upstreamStage string, at time.Time) []string {
|
||||
if m == nil || m.Stages == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
invalidated := make([]string, 0)
|
||||
for _, downstream := range downstreamStageNames(upstreamStage) {
|
||||
sr := m.Stages[downstream]
|
||||
if sr == nil || sr.Status != manifest.StatusSucceeded {
|
||||
continue
|
||||
}
|
||||
m.MarkStageStale(downstream, at, "upstream stage rerun with force")
|
||||
invalidated = append(invalidated, downstream)
|
||||
}
|
||||
return invalidated
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package app
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -40,3 +41,48 @@ func TestDecideStageActions(t *testing.T) {
|
||||
t.Fatalf("forced prepare action = %q, want %q", forced[0].Action, stageActionRun)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDownstreamStageNames(t *testing.T) {
|
||||
got := downstreamStageNames("polish")
|
||||
want := []string{"normalize", "trim", "analyze", "archive", "notify"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("downstreamStageNames(polish) = %#v, want %#v", got, want)
|
||||
}
|
||||
|
||||
missing := downstreamStageNames("unknown")
|
||||
if len(missing) != 0 {
|
||||
t.Fatalf("downstreamStageNames(unknown) = %#v, want empty", missing)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInvalidateDownstreamSucceededStages(t *testing.T) {
|
||||
now := time.Now().UTC()
|
||||
m := manifest.New("2026-05-03", now)
|
||||
m.MarkStageSucceeded("prepare", now, nil)
|
||||
m.MarkStageSucceeded("transcribe", now, nil)
|
||||
m.MarkStageSucceeded("merge", now, nil)
|
||||
m.MarkStageSucceeded("polish", now, nil)
|
||||
m.MarkStageSucceeded("normalize", now, nil)
|
||||
m.MarkStageSucceeded("trim", now, nil)
|
||||
m.MarkStageFailed("analyze", now, "analysis failed")
|
||||
m.MarkStageSucceeded("archive", now, nil)
|
||||
m.MarkStageSucceeded("notify", now, nil)
|
||||
|
||||
got := invalidateDownstreamSucceededStages(m, "polish", now.Add(1*time.Second))
|
||||
want := []string{"normalize", "trim", "archive", "notify"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("invalidateDownstreamSucceededStages() = %#v, want %#v", got, want)
|
||||
}
|
||||
|
||||
for _, stageName := range want {
|
||||
if m.Stages[stageName].Status != manifest.StatusStale {
|
||||
t.Fatalf("%s status = %q, want stale", stageName, m.Stages[stageName].Status)
|
||||
}
|
||||
}
|
||||
if m.Stages["analyze"].Status != manifest.StatusFailed {
|
||||
t.Fatalf("analyze status = %q, want failed", m.Stages["analyze"].Status)
|
||||
}
|
||||
if m.Stages["prepare"].Status != manifest.StatusSucceeded {
|
||||
t.Fatalf("prepare status = %q, want succeeded", m.Stages["prepare"].Status)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -18,10 +18,12 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
|
||||
var sessionPath string
|
||||
var sessionID string
|
||||
var force bool
|
||||
var selectedArtifacts artifactSelectionFlag
|
||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
||||
fs.StringVar(&sessionID, "session-id", "", "session identifier for session.yml templates")
|
||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
||||
fs.Var(&selectedArtifacts, "artifacts", "artifact names to execute during analyze (comma-separated or repeatable)")
|
||||
|
||||
if err := fs.Parse(args); err != nil {
|
||||
return fmt.Errorf("run-stage: invalid flags: %w", err)
|
||||
@@ -30,6 +32,13 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
|
||||
return fmt.Errorf("run-stage: expected exactly one stage name")
|
||||
}
|
||||
stageName := fs.Arg(0)
|
||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
||||
if err != nil {
|
||||
return fmt.Errorf("run-stage: invalid --artifacts: %w", err)
|
||||
}
|
||||
if len(normalizedArtifacts) > 0 && stageName != "analyze" {
|
||||
return fmt.Errorf("run-stage: --artifacts is only supported for stage \"analyze\"")
|
||||
}
|
||||
stages, err := BuildSingleStagePlan(stageName)
|
||||
if err != nil {
|
||||
return fmt.Errorf("run-stage: %w", err)
|
||||
@@ -53,8 +62,14 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
|
||||
if err := config.Validate(cfg); err != nil {
|
||||
return fmt.Errorf("run-stage: %w", err)
|
||||
}
|
||||
if err := validateSelectedAnalyzeArtifacts(cfg, normalizedArtifacts); err != nil {
|
||||
return fmt.Errorf("run-stage: %w", err)
|
||||
}
|
||||
|
||||
summary, err := executeStages(ctx, cfg, stages, RunOptions{Force: force})
|
||||
summary, err := executeStages(ctx, cfg, stages, RunOptions{
|
||||
Force: force,
|
||||
SelectedArtifacts: normalizedArtifacts,
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("run-stage: %w", err)
|
||||
}
|
||||
|
||||
@@ -5,7 +5,6 @@ import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/audita"
|
||||
@@ -22,16 +21,19 @@ import (
|
||||
)
|
||||
|
||||
type RunOptions struct {
|
||||
Force bool
|
||||
Env *Env
|
||||
Force bool
|
||||
SelectedArtifacts []string
|
||||
Env *Env
|
||||
}
|
||||
|
||||
type RunSummary struct {
|
||||
SessionID string
|
||||
ManifestPath string
|
||||
StageNames []string
|
||||
Executed []string
|
||||
Skipped []string
|
||||
SessionID string
|
||||
RunID string
|
||||
ManifestPath string
|
||||
RunManifestPath string
|
||||
StageNames []string
|
||||
Executed []string
|
||||
Skipped []string
|
||||
}
|
||||
|
||||
func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage, opts RunOptions) (*RunSummary, error) {
|
||||
@@ -42,6 +44,7 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
if env.Config == nil {
|
||||
env.Config = cfg
|
||||
}
|
||||
env.SelectedAnalyzeArtifacts = append([]string(nil), opts.SelectedArtifacts...)
|
||||
if env.ArtifactStore == nil {
|
||||
env.ArtifactStore = artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
|
||||
}
|
||||
@@ -93,12 +96,12 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
}
|
||||
|
||||
artifactStore := env.ArtifactStore
|
||||
paths, err := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, err := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("prepare workdir: %w", err)
|
||||
}
|
||||
|
||||
lock, err := artifactStore.AcquireSessionLock(cfg.Session.SessionID)
|
||||
lock, err := artifactStore.AcquireSessionLockFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("acquire session lock: %w", err)
|
||||
}
|
||||
@@ -111,7 +114,11 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
identityChanged, err := ensureManifestIdentity(cfg, m)
|
||||
runID, err := artifacts.NewRunID()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("generate run id: %w", err)
|
||||
}
|
||||
identityChanged, err := ensureManifestIdentity(cfg, m, runID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("initialize manifest identity: %w", err)
|
||||
}
|
||||
@@ -120,6 +127,29 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
return nil, fmt.Errorf("save manifest identity %q: %w", manifestPath, err)
|
||||
}
|
||||
}
|
||||
runManifestPath := artifacts.SessionRunManifestPathForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
runID,
|
||||
)
|
||||
runManifestStore := &manifest.LocalStore{}
|
||||
runManifest, err := runManifestStore.CreateRun(
|
||||
ctx,
|
||||
cfg.Session.SessionID,
|
||||
cfg.Session.Campaign,
|
||||
runID,
|
||||
opts.Force,
|
||||
requestedStageNames(stages),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create run manifest: %w", err)
|
||||
}
|
||||
runManifest.SessionManifestPath = manifestPath
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save initial run manifest %q: %w", runManifestPath, err)
|
||||
}
|
||||
|
||||
stageEnv := env
|
||||
|
||||
@@ -134,12 +164,23 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
|
||||
if d.Action == stageActionSkip {
|
||||
skipped = append(skipped, s.Name())
|
||||
skipAt := nowUTC()
|
||||
runManifest.SetStageAction(s.Name(), manifest.RunStageActionSkip, skipAt)
|
||||
runManifest.MarkStageSkipped(s.Name(), skipAt, "already_succeeded")
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save run manifest after skip %q: %w", s.Name(), err)
|
||||
}
|
||||
env.Logger.Info("skipping stage", "stage", s.Name(), "reason", "already_succeeded", "force", opts.Force)
|
||||
continue
|
||||
}
|
||||
executed = append(executed, s.Name())
|
||||
|
||||
now := nowUTC()
|
||||
runManifest.SetStageAction(s.Name(), manifest.RunStageActionRun, now)
|
||||
runManifest.MarkStageRunning(s.Name(), now)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save run manifest before stage %q: %w", s.Name(), err)
|
||||
}
|
||||
m.MarkStageRunning(s.Name(), now)
|
||||
env.Logger.Info("starting stage", "stage", s.Name())
|
||||
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
|
||||
@@ -149,35 +190,65 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
||||
|
||||
result, err := s.Run(ctx, stageEnv, m)
|
||||
if err != nil {
|
||||
m.MarkStageFailed(s.Name(), nowUTC(), err.Error())
|
||||
failedAt := nowUTC()
|
||||
m.MarkStageFailed(s.Name(), failedAt, err.Error())
|
||||
if saveErr := env.ManifestStore.Save(ctx, manifestPath, m); saveErr != nil {
|
||||
return nil, fmt.Errorf("stage %q failed (%v) and manifest save failed (%v)", s.Name(), err, saveErr)
|
||||
}
|
||||
runManifest.MarkStageFailed(s.Name(), failedAt, err.Error())
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
|
||||
return nil, fmt.Errorf("stage %q failed (%v) and run-manifest save failed (%v)", s.Name(), err, saveErr)
|
||||
}
|
||||
env.Logger.Info("stage failed", "stage", s.Name(), "error", err)
|
||||
return nil, fmt.Errorf("stage %q failed: %w", s.Name(), err)
|
||||
}
|
||||
|
||||
outputs := mapResultOutputs(result)
|
||||
m.MarkStageSucceeded(s.Name(), nowUTC(), outputs)
|
||||
outputs := mapResultOutputs(s.Name(), result, runID)
|
||||
succeededAt := nowUTC()
|
||||
m.MarkStageSucceeded(s.Name(), succeededAt, outputs)
|
||||
applyStageResultToManifest(m, s.Name(), result)
|
||||
if opts.Force {
|
||||
invalidateDownstreamSucceededStages(m, s.Name(), succeededAt)
|
||||
}
|
||||
|
||||
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
|
||||
return nil, fmt.Errorf("save manifest after stage %q: %w", s.Name(), err)
|
||||
}
|
||||
runManifest.MarkStageSucceeded(s.Name(), succeededAt, outputs)
|
||||
applyStageResultToRunManifest(runManifest, s.Name(), result)
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save run manifest after stage %q: %w", s.Name(), err)
|
||||
}
|
||||
env.Logger.Debug("manifest saved", "stage", s.Name(), "transition", "succeeded", "path", manifestPath)
|
||||
env.Logger.Info("stage succeeded", "stage", s.Name())
|
||||
}
|
||||
|
||||
if err := runPostArchiveCleanup(ctx, env, manifestPath, m, executed); err != nil {
|
||||
failedAt := nowUTC()
|
||||
runManifest.MarkFailed(failedAt, err.Error())
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if saveErr := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); saveErr != nil {
|
||||
return nil, fmt.Errorf("post-archive cleanup failed (%v) and run-manifest save failed (%v)", err, saveErr)
|
||||
}
|
||||
return nil, fmt.Errorf("post-archive cleanup: %w", err)
|
||||
}
|
||||
completedAt := nowUTC()
|
||||
runManifest.MarkSucceeded(completedAt)
|
||||
syncRunManifestIdentityFromSession(m, runManifest)
|
||||
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||
return nil, fmt.Errorf("save final run manifest %q: %w", runManifestPath, err)
|
||||
}
|
||||
|
||||
return &RunSummary{
|
||||
SessionID: cfg.Session.SessionID,
|
||||
ManifestPath: manifestPath,
|
||||
StageNames: runNames,
|
||||
Executed: executed,
|
||||
Skipped: skipped,
|
||||
SessionID: cfg.Session.SessionID,
|
||||
RunID: runID,
|
||||
ManifestPath: manifestPath,
|
||||
RunManifestPath: runManifestPath,
|
||||
StageNames: runNames,
|
||||
Executed: executed,
|
||||
Skipped: skipped,
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -317,22 +388,31 @@ func fileExists(path string) (bool, error) {
|
||||
return false, err
|
||||
}
|
||||
|
||||
func mapResultOutputs(result *stage.StageResult) []manifest.ArtifactRecord {
|
||||
func mapResultOutputs(stageName string, result *stage.StageResult, runID string) []manifest.ArtifactRecord {
|
||||
if result == nil || len(result.Outputs) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
runID = strings.TrimSpace(runID)
|
||||
out := make([]manifest.ArtifactRecord, 0, len(result.Outputs))
|
||||
for _, ref := range result.Outputs {
|
||||
localPath := ref.AbsolutePath
|
||||
if localPath == "" {
|
||||
localPath = ref.RelativePath
|
||||
}
|
||||
kind := ref.Kind
|
||||
sourceID := ""
|
||||
if stageName == "analyze" {
|
||||
sourceID = artifacts.ConfiguredArtifactSourceID(ref.Kind)
|
||||
kind = "scriptorium_artifact"
|
||||
}
|
||||
out = append(out, manifest.ArtifactRecord{
|
||||
Kind: ref.Kind,
|
||||
LocalPath: localPath,
|
||||
RemoteKey: ref.RemoteKey,
|
||||
Checksum: ref.Checksum,
|
||||
Kind: kind,
|
||||
SourceID: sourceID,
|
||||
LocalPath: localPath,
|
||||
ProducerRunID: runID,
|
||||
RemoteKey: ref.RemoteKey,
|
||||
Checksum: ref.Checksum,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -358,7 +438,7 @@ func applyStageResultToManifest(m *manifest.Manifest, stageName string, result *
|
||||
}
|
||||
}
|
||||
|
||||
func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, error) {
|
||||
func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest, runID string) (bool, error) {
|
||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil || m == nil {
|
||||
return false, nil
|
||||
}
|
||||
@@ -374,16 +454,13 @@ func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, err
|
||||
m.Campaign = campaign
|
||||
changed = true
|
||||
}
|
||||
if m.RunID == "" {
|
||||
runID, err := artifacts.NewRunID()
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
runID = strings.TrimSpace(runID)
|
||||
if runID != "" && m.RunID != runID {
|
||||
m.RunID = runID
|
||||
changed = true
|
||||
}
|
||||
if m.LocalWorkDir == "" && campaign != "" && sessionID != "" && m.RunID != "" {
|
||||
m.LocalWorkDir = artifacts.SessionRunWorkDir(cfg.Pipeline.Workspace.Root, campaign, sessionID, m.RunID)
|
||||
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, campaign, sessionID, m.RunID)
|
||||
changed = true
|
||||
}
|
||||
if m.LocalSpoolDir == "" && campaign != "" && sessionID != "" && m.RunID != "" && strings.TrimSpace(cfg.Pipeline.Spool.Root) != "" {
|
||||
@@ -410,8 +487,54 @@ func ensureManifestIdentity(cfg *config.Config, m *manifest.Manifest) (bool, err
|
||||
return changed, nil
|
||||
}
|
||||
|
||||
func requestedStageNames(stages []stage.Stage) []string {
|
||||
out := make([]string, 0, len(stages))
|
||||
for _, s := range stages {
|
||||
if s == nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, s.Name())
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func applyStageResultToRunManifest(m *manifest.RunManifest, stageName string, result *stage.StageResult) {
|
||||
if m == nil || result == nil {
|
||||
return
|
||||
}
|
||||
sr := m.Stages[stageName]
|
||||
if sr == nil {
|
||||
return
|
||||
}
|
||||
if len(result.Logs) > 0 {
|
||||
sr.Logs = append([]string(nil), result.Logs...)
|
||||
}
|
||||
if len(result.GeneratedConfigs) > 0 {
|
||||
sr.GeneratedConfigs = append([]string(nil), result.GeneratedConfigs...)
|
||||
}
|
||||
if len(result.Metadata) > 0 {
|
||||
sr.Metadata = result.Metadata
|
||||
}
|
||||
}
|
||||
|
||||
func syncRunManifestIdentityFromSession(session *manifest.Manifest, run *manifest.RunManifest) {
|
||||
if session == nil || run == nil {
|
||||
return
|
||||
}
|
||||
run.Campaign = session.Campaign
|
||||
run.LocalWorkDir = session.LocalWorkDir
|
||||
run.LocalSpoolDir = session.LocalSpoolDir
|
||||
run.S3Bucket = session.S3Bucket
|
||||
run.S3SessionPrefix = session.S3SessionPrefix
|
||||
run.S3RunPrefix = session.S3RunPrefix
|
||||
}
|
||||
|
||||
func manifestPathFor(cfg *config.Config) string {
|
||||
return filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "manifest.json")
|
||||
return artifacts.SessionManifestPathForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
)
|
||||
}
|
||||
|
||||
func needsObjectStoreForRun(cfg *config.Config, stages []stage.Stage) bool {
|
||||
|
||||
@@ -3,6 +3,7 @@ package app
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -44,6 +45,213 @@ func (s countingStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest
|
||||
return &stage.StageResult{Metadata: map[string]any{"counting": true}}, nil
|
||||
}
|
||||
|
||||
type captureSelectedArtifactsStage struct {
|
||||
name string
|
||||
captured *[]string
|
||||
}
|
||||
|
||||
func (s captureSelectedArtifactsStage) Name() string { return s.name }
|
||||
func (s captureSelectedArtifactsStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||
func (s captureSelectedArtifactsStage) Run(_ context.Context, env *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
||||
if s.captured != nil {
|
||||
*s.captured = append((*s.captured)[:0], env.SelectedAnalyzeArtifacts...)
|
||||
}
|
||||
return &stage.StageResult{Metadata: map[string]any{"captured": true}}, nil
|
||||
}
|
||||
|
||||
type analyzeOutputStage struct {
|
||||
output artifacts.Ref
|
||||
}
|
||||
|
||||
func (s analyzeOutputStage) Name() string { return "analyze" }
|
||||
func (s analyzeOutputStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||
func (s analyzeOutputStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
||||
return &stage.StageResult{
|
||||
Outputs: []artifacts.Ref{s.output},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type selectedAnalyzeArtifactStage struct {
|
||||
expected []string
|
||||
}
|
||||
|
||||
func (s selectedAnalyzeArtifactStage) Name() string { return "analyze" }
|
||||
func (s selectedAnalyzeArtifactStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||
func (s selectedAnalyzeArtifactStage) Run(_ context.Context, env *stage.Env, m *manifest.Manifest) (*stage.StageResult, error) {
|
||||
if len(env.SelectedAnalyzeArtifacts) != len(s.expected) {
|
||||
return nil, fmt.Errorf("selected artifacts len = %d, want %d", len(env.SelectedAnalyzeArtifacts), len(s.expected))
|
||||
}
|
||||
for i := range s.expected {
|
||||
if env.SelectedAnalyzeArtifacts[i] != s.expected[i] {
|
||||
return nil, fmt.Errorf("selected artifacts[%d] = %q, want %q", i, env.SelectedAnalyzeArtifacts[i], s.expected[i])
|
||||
}
|
||||
}
|
||||
|
||||
outputPath := filepath.Join(
|
||||
artifacts.SessionWorkDirForCampaign(env.Config.Pipeline.Workspace.Root, env.Config.Session.Campaign, m.SessionID),
|
||||
"artifacts",
|
||||
"player_handout.md",
|
||||
)
|
||||
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
|
||||
return nil, fmt.Errorf("mkdir artifact dir: %w", err)
|
||||
}
|
||||
if err := os.WriteFile(outputPath, []byte("player handout\n"), 0o644); err != nil {
|
||||
return nil, fmt.Errorf("write player handout: %w", err)
|
||||
}
|
||||
|
||||
return &stage.StageResult{
|
||||
Outputs: []artifacts.Ref{
|
||||
{
|
||||
Kind: "player_handout",
|
||||
Category: "artifacts",
|
||||
RelativePath: "artifacts/player_handout.md",
|
||||
AbsolutePath: outputPath,
|
||||
},
|
||||
},
|
||||
Metadata: map[string]any{
|
||||
"stage": "analyze",
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func TestExecuteStagesPropagatesSelectedArtifactsToEnv(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
|
||||
captured := []string{}
|
||||
stageToRun := captureSelectedArtifactsStage{name: "analyze", captured: &captured}
|
||||
_, err := executeStages(context.Background(), cfg, []stage.Stage{stageToRun}, RunOptions{
|
||||
SelectedArtifacts: []string{"player_handout", "session_recap"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
|
||||
if len(captured) != 2 {
|
||||
t.Fatalf("captured len = %d, want 2 (%v)", len(captured), captured)
|
||||
}
|
||||
if captured[0] != "player_handout" || captured[1] != "session_recap" {
|
||||
t.Fatalf("captured = %v, want [player_handout session_recap]", captured)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesAnalyzeOutputsPersistAsScriptoriumArtifacts(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
storeForPaths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
|
||||
sessionPaths := storeForPaths.SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
outputPath := filepath.Join(sessionPaths.ArtifactsDir, "session_recap.md")
|
||||
|
||||
stageToRun := analyzeOutputStage{
|
||||
output: artifacts.Ref{
|
||||
Kind: "session_recap",
|
||||
Category: "artifacts",
|
||||
RelativePath: "artifacts/session_recap.md",
|
||||
AbsolutePath: outputPath,
|
||||
},
|
||||
}
|
||||
summary, err := executeStages(context.Background(), cfg, []stage.Stage{stageToRun}, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
sessionManifest, err := store.Load(context.Background(), summary.ManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("load session manifest: %v", err)
|
||||
}
|
||||
sessionStage := sessionManifest.Stages["analyze"]
|
||||
if sessionStage == nil {
|
||||
t.Fatal("session manifest analyze stage missing")
|
||||
}
|
||||
if len(sessionStage.Outputs) != 1 {
|
||||
t.Fatalf("session analyze outputs len = %d, want 1", len(sessionStage.Outputs))
|
||||
}
|
||||
sessionOutput := sessionStage.Outputs[0]
|
||||
if sessionOutput.Kind != "scriptorium_artifact" {
|
||||
t.Fatalf("session output kind = %q, want scriptorium_artifact", sessionOutput.Kind)
|
||||
}
|
||||
if sessionOutput.SourceID != "narratio.artifact.session_recap" {
|
||||
t.Fatalf("session output source_id = %q, want narratio.artifact.session_recap", sessionOutput.SourceID)
|
||||
}
|
||||
if sessionOutput.LocalPath != outputPath {
|
||||
t.Fatalf("session output local_path = %q, want %q", sessionOutput.LocalPath, outputPath)
|
||||
}
|
||||
|
||||
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("load run manifest: %v", err)
|
||||
}
|
||||
runStage := runManifest.Stages["analyze"]
|
||||
if runStage == nil {
|
||||
t.Fatal("run manifest analyze stage missing")
|
||||
}
|
||||
if len(runStage.Outputs) != 1 {
|
||||
t.Fatalf("run analyze outputs len = %d, want 1", len(runStage.Outputs))
|
||||
}
|
||||
runOutput := runStage.Outputs[0]
|
||||
if runOutput.Kind != "scriptorium_artifact" {
|
||||
t.Fatalf("run output kind = %q, want scriptorium_artifact", runOutput.Kind)
|
||||
}
|
||||
if runOutput.SourceID != "narratio.artifact.session_recap" {
|
||||
t.Fatalf("run output source_id = %q, want narratio.artifact.session_recap", runOutput.SourceID)
|
||||
}
|
||||
if runOutput.LocalPath != outputPath {
|
||||
t.Fatalf("run output local_path = %q, want %q", runOutput.LocalPath, outputPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesArchiveFailsWhenRequiredRecapPromotionMissingForSelectedArtifacts(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
cfg.Pipeline.Storage.S3 = &config.StorageS3Config{
|
||||
Bucket: "my-dnd-archive",
|
||||
RootPrefix: "dnd",
|
||||
}
|
||||
cfg.Pipeline.Archive = &config.ArchiveConfig{
|
||||
Enabled: boolPtr(true),
|
||||
UploadRun: boolPtr(true),
|
||||
PromoteArtifacts: []config.ArchivePromotionRule{
|
||||
{From: "artifacts/session_recap.md", To: "artifacts/session_recap.md", Required: boolPtr(true)},
|
||||
},
|
||||
}
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
manifestPath := manifestPathFor(cfg)
|
||||
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
||||
seed.Campaign = cfg.Session.Campaign
|
||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim"} {
|
||||
seed.MarkStageSucceeded(stageName, time.Now().UTC(), nil)
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||
t.Fatalf("Save manifest error = %v", err)
|
||||
}
|
||||
|
||||
archiveStageImpl, err := stage.Select("archive")
|
||||
if err != nil {
|
||||
t.Fatalf("Select(archive) error = %v", err)
|
||||
}
|
||||
|
||||
_, err = executeStages(
|
||||
context.Background(),
|
||||
cfg,
|
||||
[]stage.Stage{
|
||||
selectedAnalyzeArtifactStage{expected: []string{"player_handout"}},
|
||||
archiveStageImpl,
|
||||
},
|
||||
RunOptions{
|
||||
SelectedArtifacts: []string{"player_handout"},
|
||||
Env: &Env{ObjectStore: &storage.FakeBackend{}},
|
||||
},
|
||||
)
|
||||
if err == nil {
|
||||
t.Fatal("expected archive promotion failure, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "required promotion source missing") {
|
||||
t.Fatalf("error = %q, want required promotion source missing", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesPlaceholderSuccessUpdatesManifest(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
|
||||
@@ -235,6 +443,56 @@ func TestExecuteStagesForceRerunsSucceeded(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesForceSuccessInvalidatesDownstreamSucceededStages(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
manifestPath := manifestPathFor(cfg)
|
||||
store := &manifest.LocalStore{}
|
||||
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "archive", "notify"} {
|
||||
existing.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
||||
}
|
||||
existing.MarkStageFailed("analyze", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), "previous analyze failure")
|
||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
|
||||
t.Fatalf("Save manifest error = %v", err)
|
||||
}
|
||||
|
||||
runs := 0
|
||||
stageToRun := countingStage{name: "polish", runs: &runs}
|
||||
summary, err := executeStages(context.Background(), cfg, []stage.Stage{stageToRun}, RunOptions{Force: true})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if runs != 1 {
|
||||
t.Fatalf("runs = %d, want 1 with force", runs)
|
||||
}
|
||||
if len(summary.Executed) != 1 || summary.Executed[0] != "polish" || len(summary.Skipped) != 0 {
|
||||
t.Fatalf("summary = %#v, want executed polish", summary)
|
||||
}
|
||||
|
||||
loaded, err := store.Load(context.Background(), manifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load manifest error = %v", err)
|
||||
}
|
||||
if loaded.Stages["polish"] == nil || loaded.Stages["polish"].Status != manifest.StatusSucceeded {
|
||||
t.Fatalf("polish status = %#v, want succeeded", loaded.Stages["polish"])
|
||||
}
|
||||
for _, stageName := range []string{"normalize", "trim", "archive", "notify"} {
|
||||
if loaded.Stages[stageName] == nil || loaded.Stages[stageName].Status != manifest.StatusStale {
|
||||
t.Fatalf("%s status = %#v, want stale", stageName, loaded.Stages[stageName])
|
||||
}
|
||||
}
|
||||
if loaded.Stages["analyze"] == nil || loaded.Stages["analyze"].Status != manifest.StatusFailed {
|
||||
t.Fatalf("analyze status = %#v, want preserved failed", loaded.Stages["analyze"])
|
||||
}
|
||||
if loaded.Stages["transcribe"] == nil || loaded.Stages["transcribe"].Status != manifest.StatusSucceeded {
|
||||
t.Fatalf("transcribe status = %#v, want preserved succeeded", loaded.Stages["transcribe"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesFailureUpdatesManifest(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
|
||||
@@ -280,7 +538,14 @@ func TestExecuteStagesLoadsExistingManifest(t *testing.T) {
|
||||
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
existing.MarkStageSucceeded("prepare", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
||||
audioPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "audio", "alice.flac")
|
||||
audioPath := filepath.Join(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
"work",
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
"audio",
|
||||
"alice.flac",
|
||||
)
|
||||
if err := os.MkdirAll(filepath.Dir(audioPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
@@ -315,6 +580,208 @@ func TestExecuteStagesLoadsExistingManifest(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesCreatesRunManifestPerInvocation(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
run1, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[0]}, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("first executeStages() error = %v", err)
|
||||
}
|
||||
run2, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[0]}, RunOptions{Force: true})
|
||||
if err != nil {
|
||||
t.Fatalf("second executeStages() error = %v", err)
|
||||
}
|
||||
|
||||
if run1.RunID == "" || run2.RunID == "" {
|
||||
t.Fatalf("run ids must be set, got %q and %q", run1.RunID, run2.RunID)
|
||||
}
|
||||
if run1.RunID == run2.RunID {
|
||||
t.Fatalf("expected distinct run ids, got %q", run1.RunID)
|
||||
}
|
||||
if run1.RunManifestPath == "" || run2.RunManifestPath == "" {
|
||||
t.Fatalf("run manifest paths must be set, got %q and %q", run1.RunManifestPath, run2.RunManifestPath)
|
||||
}
|
||||
if run1.RunManifestPath == run2.RunManifestPath {
|
||||
t.Fatalf("expected distinct run manifest paths, got %q", run1.RunManifestPath)
|
||||
}
|
||||
for _, path := range []string{run1.RunManifestPath, run2.RunManifestPath} {
|
||||
if _, statErr := os.Stat(path); statErr != nil {
|
||||
t.Fatalf("run manifest missing at %q: %v", path, statErr)
|
||||
}
|
||||
}
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
sessionManifest, err := store.Load(context.Background(), run2.ManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load session manifest error = %v", err)
|
||||
}
|
||||
if sessionManifest.RunID != run2.RunID {
|
||||
t.Fatalf("session manifest run_id = %q, want latest run id %q", sessionManifest.RunID, run2.RunID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesRunManifestRecordsSkippedStage(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
store := &manifest.LocalStore{}
|
||||
manifestPath := manifestPathFor(cfg)
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
existing.MarkStageSucceeded("transcribe", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
|
||||
t.Fatalf("Save manifest error = %v", err)
|
||||
}
|
||||
|
||||
summary, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[1]}, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if len(summary.Skipped) != 1 || summary.Skipped[0] != "transcribe" {
|
||||
t.Fatalf("summary = %#v, want skipped transcribe", summary)
|
||||
}
|
||||
|
||||
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadRun() error = %v", err)
|
||||
}
|
||||
sr := runManifest.Stages["transcribe"]
|
||||
if sr == nil {
|
||||
t.Fatal("run manifest transcribe stage missing")
|
||||
}
|
||||
if sr.Action != manifest.RunStageActionSkip {
|
||||
t.Fatalf("action = %q, want %q", sr.Action, manifest.RunStageActionSkip)
|
||||
}
|
||||
if sr.Status != manifest.StatusSkipped {
|
||||
t.Fatalf("status = %q, want %q", sr.Status, manifest.StatusSkipped)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesRunLocalArtifactsAndCanonicalPromotion(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
stages := []stage.Stage{
|
||||
BuildFullPlan()[0], // prepare
|
||||
BuildFullPlan()[1], // transcribe
|
||||
BuildFullPlan()[2], // merge
|
||||
BuildFullPlan()[3], // polish
|
||||
BuildFullPlan()[4], // normalize
|
||||
BuildFullPlan()[5], // trim
|
||||
}
|
||||
|
||||
summary, err := executeStages(context.Background(), cfg, stages, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if summary.RunID == "" {
|
||||
t.Fatal("run id must be set")
|
||||
}
|
||||
|
||||
runRoot := artifacts.SessionRunRootForCampaign(
|
||||
cfg.Pipeline.Workspace.Root,
|
||||
cfg.Session.Campaign,
|
||||
cfg.Session.SessionID,
|
||||
summary.RunID,
|
||||
)
|
||||
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
|
||||
runLocalChecks := []string{
|
||||
filepath.Join(runRoot, "transcribe", "outputs", "transcripts", "raw", "alice.json"),
|
||||
filepath.Join(runRoot, "merge", "logs", "seriatim.stdout.log"),
|
||||
filepath.Join(runRoot, "polish", "config", "audita.generated.yml"),
|
||||
filepath.Join(runRoot, "normalize", "logs", "seriatim.normalize.stdout.log"),
|
||||
filepath.Join(runRoot, "trim", "outputs", "transcripts", "trimmed.json"),
|
||||
}
|
||||
for _, p := range runLocalChecks {
|
||||
if _, statErr := os.Stat(p); statErr != nil {
|
||||
t.Fatalf("run-local artifact missing at %q: %v", p, statErr)
|
||||
}
|
||||
}
|
||||
|
||||
canonicalChecks := []string{
|
||||
filepath.Join(paths.TranscriptsRawDir, "alice.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "merged.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "processed.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "normalized.json"),
|
||||
filepath.Join(paths.TranscriptsDir, "trimmed.json"),
|
||||
}
|
||||
for _, p := range canonicalChecks {
|
||||
if _, statErr := os.Stat(p); statErr != nil {
|
||||
t.Fatalf("canonical promoted artifact missing at %q: %v", p, statErr)
|
||||
}
|
||||
}
|
||||
|
||||
store := &manifest.LocalStore{}
|
||||
sessionManifest, err := store.Load(context.Background(), summary.ManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load manifest error = %v", err)
|
||||
}
|
||||
if got := sessionManifest.Stages["trim"]; got == nil || len(got.Outputs) == 0 {
|
||||
t.Fatalf("trim stage outputs missing in session manifest: %#v", got)
|
||||
}
|
||||
for _, out := range sessionManifest.Stages["trim"].Outputs {
|
||||
if strings.Contains(out.LocalPath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("session manifest output should be canonical, got run-local path %q", out.LocalPath)
|
||||
}
|
||||
if out.ProducerRunID != summary.RunID {
|
||||
t.Fatalf("producer_run_id = %q, want %q", out.ProducerRunID, summary.RunID)
|
||||
}
|
||||
}
|
||||
|
||||
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadRun() error = %v", err)
|
||||
}
|
||||
mergeStage := runManifest.Stages["merge"]
|
||||
if mergeStage == nil || len(mergeStage.Logs) == 0 {
|
||||
t.Fatalf("merge logs missing in run manifest: %#v", mergeStage)
|
||||
}
|
||||
for _, logPath := range mergeStage.Logs {
|
||||
if !strings.Contains(logPath, filepath.Join("runs", summary.RunID, "merge", "logs")) {
|
||||
t.Fatalf("run manifest merge log path = %q, want run-local merge logs path", logPath)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteStagesSkippedStagePreservesExistingOutputsProvenance(t *testing.T) {
|
||||
cfg := testConfig(t)
|
||||
store := &manifest.LocalStore{}
|
||||
manifestPath := manifestPathFor(cfg)
|
||||
|
||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||
existing.MarkStageSucceeded("transcribe", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), []manifest.ArtifactRecord{
|
||||
{
|
||||
Kind: "transcript_raw",
|
||||
LocalPath: "transcripts/raw/alice.json",
|
||||
ProducerRunID: "20260501T000000Z-deadbeef",
|
||||
},
|
||||
})
|
||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
|
||||
t.Fatalf("Save manifest error = %v", err)
|
||||
}
|
||||
|
||||
summary, err := executeStages(context.Background(), cfg, []stage.Stage{BuildFullPlan()[1]}, RunOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("executeStages() error = %v", err)
|
||||
}
|
||||
if len(summary.Skipped) != 1 || summary.Skipped[0] != "transcribe" {
|
||||
t.Fatalf("summary = %#v, want skipped transcribe", summary)
|
||||
}
|
||||
|
||||
loaded, err := store.Load(context.Background(), manifestPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load manifest error = %v", err)
|
||||
}
|
||||
got := loaded.Stages["transcribe"]
|
||||
if got == nil || len(got.Outputs) != 1 {
|
||||
t.Fatalf("transcribe outputs = %#v, want one preserved output", got)
|
||||
}
|
||||
if got.Outputs[0].ProducerRunID != "20260501T000000Z-deadbeef" {
|
||||
t.Fatalf("producer_run_id = %q, want preserved value", got.Outputs[0].ProducerRunID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
@@ -341,7 +808,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
tc.env.ArtifactStore = artifactStore
|
||||
tc.env.ManifestStore = &manifest.LocalStore{}
|
||||
if tc.name == "transcribe" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
@@ -356,7 +823,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
if tc.name == "merge" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
@@ -375,7 +842,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
if tc.name == "polish" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
@@ -387,7 +854,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
if tc.name == "analyze" {
|
||||
paths, ensureErr := artifactStore.EnsureLayout(cfg.Session.SessionID)
|
||||
paths, ensureErr := artifactStore.EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||
if ensureErr != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", ensureErr)
|
||||
}
|
||||
@@ -403,7 +870,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
||||
PromptID: "dnd.session_recap",
|
||||
OutputPath: "artifacts/session_recap.md",
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"transcript": {Source: "processed_transcript", Required: true},
|
||||
"transcript": {Source: "narratio.transcript.polished", Required: true},
|
||||
},
|
||||
},
|
||||
},
|
||||
|
||||
@@ -63,7 +63,7 @@ func TestExecuteStagesDefaultWiringUsesWhisperXHTTPClient(t *testing.T) {
|
||||
t.Fatal("audio file payload was empty")
|
||||
}
|
||||
|
||||
outPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.SessionID, "transcripts", "raw", "alice.json")
|
||||
outPath := filepath.Join(cfg.Pipeline.Workspace.Root, "work", cfg.Session.Campaign, cfg.Session.SessionID, "transcripts", "raw", "alice.json")
|
||||
data, err := os.ReadFile(outPath)
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile(%q) error = %v", outPath, err)
|
||||
|
||||
313
internal/artifacts/artifact_resolver.go
Normal file
313
internal/artifacts/artifact_resolver.go
Normal file
@@ -0,0 +1,313 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
const (
|
||||
ArtifactTranscriptMerged = "narratio.transcript.merged"
|
||||
ArtifactTranscriptPolished = "narratio.transcript.polished"
|
||||
ArtifactTranscriptFull = "narratio.transcript.full"
|
||||
ArtifactTranscriptTrimmed = "narratio.transcript.trimmed"
|
||||
ArtifactBoundsSession = "narratio.bounds.session"
|
||||
)
|
||||
|
||||
// ErrSessionArtifactNotFound is returned when no readable artifact exists for a known ID.
|
||||
var ErrSessionArtifactNotFound = errors.New("session artifact not found")
|
||||
var configuredArtifactSourceRE = regexp.MustCompile(`^narratio\.artifact\.[a-z][a-z0-9_]*$`)
|
||||
|
||||
type artifactContentKind string
|
||||
|
||||
const (
|
||||
contentTranscriptJSON artifactContentKind = "transcript_json"
|
||||
contentJSON artifactContentKind = "json"
|
||||
contentText artifactContentKind = "text"
|
||||
)
|
||||
|
||||
type artifactSpec struct {
|
||||
ID string
|
||||
CanonicalRelPath string
|
||||
ProducerStage string
|
||||
OutputKind string
|
||||
ContentKind artifactContentKind
|
||||
}
|
||||
|
||||
var artifactRegistry = map[string]artifactSpec{
|
||||
ArtifactTranscriptMerged: {
|
||||
ID: ArtifactTranscriptMerged,
|
||||
CanonicalRelPath: "transcripts/merged.json",
|
||||
ProducerStage: "merge",
|
||||
OutputKind: "transcript_merged",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactTranscriptPolished: {
|
||||
ID: ArtifactTranscriptPolished,
|
||||
CanonicalRelPath: "transcripts/processed.json",
|
||||
ProducerStage: "polish",
|
||||
OutputKind: "transcript_processed",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactTranscriptFull: {
|
||||
ID: ArtifactTranscriptFull,
|
||||
CanonicalRelPath: "transcripts/normalized.json",
|
||||
ProducerStage: "normalize",
|
||||
OutputKind: "transcript_normalized",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactTranscriptTrimmed: {
|
||||
ID: ArtifactTranscriptTrimmed,
|
||||
CanonicalRelPath: "transcripts/trimmed.json",
|
||||
ProducerStage: "trim",
|
||||
OutputKind: "transcript_trimmed",
|
||||
ContentKind: contentTranscriptJSON,
|
||||
},
|
||||
ArtifactBoundsSession: {
|
||||
ID: ArtifactBoundsSession,
|
||||
CanonicalRelPath: "artifacts/session_bounds.json",
|
||||
ProducerStage: "trim",
|
||||
OutputKind: "session_bounds",
|
||||
ContentKind: contentJSON,
|
||||
},
|
||||
}
|
||||
|
||||
// ResolvedSessionArtifact describes one session-level artifact lookup result.
|
||||
type ResolvedSessionArtifact struct {
|
||||
ID string
|
||||
Path string
|
||||
ProducerStage string
|
||||
OutputKind string
|
||||
ProducerRunID string
|
||||
Provenance string
|
||||
}
|
||||
|
||||
// SessionArtifactNotFoundError includes context when a known artifact cannot be read.
|
||||
type SessionArtifactNotFoundError struct {
|
||||
ArtifactID string
|
||||
}
|
||||
|
||||
func (e *SessionArtifactNotFoundError) Error() string {
|
||||
return fmt.Sprintf("%s: %q", ErrSessionArtifactNotFound, e.ArtifactID)
|
||||
}
|
||||
|
||||
func (e *SessionArtifactNotFoundError) Unwrap() error {
|
||||
return ErrSessionArtifactNotFound
|
||||
}
|
||||
|
||||
// NormalizeSessionArtifactSource validates canonical artifact IDs.
|
||||
func NormalizeSessionArtifactSource(source string) (string, error) {
|
||||
normalized := strings.TrimSpace(source)
|
||||
if normalized == "" {
|
||||
return "", fmt.Errorf("artifact source is required")
|
||||
}
|
||||
if _, ok := artifactRegistry[normalized]; !ok {
|
||||
return "", fmt.Errorf("unsupported artifact source %q", source)
|
||||
}
|
||||
return normalized, nil
|
||||
}
|
||||
|
||||
// IsConfiguredArtifactSource returns true when source is narratio.artifact.<name>.
|
||||
func IsConfiguredArtifactSource(source string) bool {
|
||||
return configuredArtifactSourceRE.MatchString(strings.TrimSpace(source))
|
||||
}
|
||||
|
||||
// ResolveSessionArtifact resolves a symbolic source to a readable local session artifact path.
|
||||
// Resolution order is manifest producer outputs first, then canonical session path fallback.
|
||||
func ResolveSessionArtifact(paths SessionPaths, m *manifest.Manifest, source string) (ResolvedSessionArtifact, error) {
|
||||
id, err := NormalizeSessionArtifactSource(source)
|
||||
if err != nil {
|
||||
return ResolvedSessionArtifact{}, err
|
||||
}
|
||||
spec := artifactRegistry[id]
|
||||
|
||||
for _, candidate := range manifestArtifactCandidates(paths, m, spec) {
|
||||
exists, isDir, statErr := pathExists(candidate.Path)
|
||||
if statErr != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("stat %q: %w", candidate.Path, statErr)
|
||||
}
|
||||
if !exists || isDir {
|
||||
continue
|
||||
}
|
||||
resolved := candidate
|
||||
resolved.ID = spec.ID
|
||||
resolved.ProducerStage = spec.ProducerStage
|
||||
resolved.OutputKind = spec.OutputKind
|
||||
if err := validateResolvedContent(resolved.Path, spec.ContentKind); err != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", resolved.ID, err)
|
||||
}
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
fallbackPath := filepath.Join(paths.Root, filepath.FromSlash(spec.CanonicalRelPath))
|
||||
exists, isDir, statErr := pathExists(fallbackPath)
|
||||
if statErr != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("stat %q: %w", fallbackPath, statErr)
|
||||
}
|
||||
if exists && !isDir {
|
||||
if err := validateResolvedContent(fallbackPath, spec.ContentKind); err != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", spec.ID, err)
|
||||
}
|
||||
return ResolvedSessionArtifact{
|
||||
ID: spec.ID,
|
||||
Path: filepath.Clean(fallbackPath),
|
||||
ProducerStage: spec.ProducerStage,
|
||||
OutputKind: spec.OutputKind,
|
||||
Provenance: "fallback.canonical_path",
|
||||
}, nil
|
||||
}
|
||||
|
||||
return ResolvedSessionArtifact{}, &SessionArtifactNotFoundError{ArtifactID: spec.ID}
|
||||
}
|
||||
|
||||
// ResolveSessionArtifactWithCatalog resolves built-in sources using existing rules and resolves
|
||||
// configured narratio.artifact.<name> sources through runtime catalog availability.
|
||||
func ResolveSessionArtifactWithCatalog(paths SessionPaths, m *manifest.Manifest, source string, catalog *ArtifactCatalog) (ResolvedSessionArtifact, error) {
|
||||
normalized := strings.TrimSpace(source)
|
||||
if !IsConfiguredArtifactSource(normalized) {
|
||||
return ResolveSessionArtifact(paths, m, normalized)
|
||||
}
|
||||
if catalog == nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("configured artifact source %q requires runtime artifact catalog", source)
|
||||
}
|
||||
entry, ok := catalog.Lookup(normalized)
|
||||
if !ok {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("unsupported artifact source %q", source)
|
||||
}
|
||||
if !entry.Available {
|
||||
return ResolvedSessionArtifact{}, &SessionArtifactNotFoundError{ArtifactID: normalized}
|
||||
}
|
||||
if err := validateResolvedContent(entry.Path, contentText); err != nil {
|
||||
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", normalized, err)
|
||||
}
|
||||
return ResolvedSessionArtifact{
|
||||
ID: normalized,
|
||||
Path: filepath.Clean(entry.Path),
|
||||
ProducerStage: entry.ProducerStage,
|
||||
OutputKind: entry.OutputKind,
|
||||
Provenance: entry.Provenance,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func manifestArtifactCandidates(paths SessionPaths, m *manifest.Manifest, spec artifactSpec) []ResolvedSessionArtifact {
|
||||
if m == nil || len(m.Stages) == 0 || spec.ProducerStage == "" || spec.OutputKind == "" {
|
||||
return nil
|
||||
}
|
||||
sr := m.Stages[spec.ProducerStage]
|
||||
if sr == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
candidates := make([]ResolvedSessionArtifact, 0, len(sr.Outputs))
|
||||
for _, out := range sr.Outputs {
|
||||
if strings.TrimSpace(out.Kind) != spec.OutputKind {
|
||||
continue
|
||||
}
|
||||
if strings.TrimSpace(out.LocalPath) == "" {
|
||||
continue
|
||||
}
|
||||
resolved := filepath.Clean(ResolveSessionLocalPathForRead(paths, out.LocalPath))
|
||||
if resolved == "" {
|
||||
continue
|
||||
}
|
||||
candidates = append(candidates, ResolvedSessionArtifact{
|
||||
Path: resolved,
|
||||
ProducerRunID: strings.TrimSpace(out.ProducerRunID),
|
||||
Provenance: "manifest." + spec.ProducerStage + ".outputs",
|
||||
})
|
||||
}
|
||||
return dedupeResolvedArtifacts(candidates)
|
||||
}
|
||||
|
||||
func dedupeResolvedArtifacts(values []ResolvedSessionArtifact) []ResolvedSessionArtifact {
|
||||
seen := map[string]struct{}{}
|
||||
out := make([]ResolvedSessionArtifact, 0, len(values))
|
||||
for _, value := range values {
|
||||
key := filepath.Clean(strings.TrimSpace(value.Path))
|
||||
if key == "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[key]; ok {
|
||||
continue
|
||||
}
|
||||
seen[key] = struct{}{}
|
||||
value.Path = key
|
||||
out = append(out, value)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func pathExists(path string) (exists bool, isDir bool, err error) {
|
||||
info, err := os.Stat(path)
|
||||
if err == nil {
|
||||
return true, info.IsDir(), nil
|
||||
}
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return false, false, nil
|
||||
}
|
||||
return false, false, err
|
||||
}
|
||||
|
||||
func validateResolvedContent(path string, kind artifactContentKind) error {
|
||||
switch kind {
|
||||
case contentTranscriptJSON:
|
||||
return validateTranscriptSegmentsJSON(path)
|
||||
case contentJSON:
|
||||
return validateJSONContent(path)
|
||||
case contentText:
|
||||
return validateNonEmptyContent(path)
|
||||
default:
|
||||
return fmt.Errorf("unsupported content kind %q", kind)
|
||||
}
|
||||
}
|
||||
|
||||
func validateTranscriptSegmentsJSON(path string) error {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read file: %w", err)
|
||||
}
|
||||
var payload map[string]any
|
||||
if err := json.Unmarshal(data, &payload); err != nil {
|
||||
return fmt.Errorf("decode json: %w", err)
|
||||
}
|
||||
segments, ok := payload["segments"]
|
||||
if !ok {
|
||||
return fmt.Errorf("top-level segments is required")
|
||||
}
|
||||
if _, ok := segments.([]any); !ok {
|
||||
return fmt.Errorf("top-level segments must be an array")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateJSONContent(path string) error {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read file: %w", err)
|
||||
}
|
||||
var payload any
|
||||
if err := json.Unmarshal(data, &payload); err != nil {
|
||||
return fmt.Errorf("decode json: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateNonEmptyContent(path string) error {
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("stat file: %w", err)
|
||||
}
|
||||
if info.IsDir() {
|
||||
return fmt.Errorf("path is a directory")
|
||||
}
|
||||
if info.Size() <= 0 {
|
||||
return fmt.Errorf("file is empty")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
268
internal/artifacts/artifact_resolver_test.go
Normal file
268
internal/artifacts/artifact_resolver_test.go
Normal file
@@ -0,0 +1,268 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
func TestNormalizeSessionArtifactSource(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
source string
|
||||
wantID string
|
||||
wantErr string
|
||||
}{
|
||||
{name: "legacy alias processed unsupported", source: "processed_transcript", wantErr: "unsupported artifact source"},
|
||||
{name: "legacy alias normalized unsupported", source: "normalized_transcript", wantErr: "unsupported artifact source"},
|
||||
{name: "legacy alias trimmed unsupported", source: "trimmed_transcript", wantErr: "unsupported artifact source"},
|
||||
{name: "configured source unsupported in built-in normalization", source: "narratio.artifact.session_recap", wantErr: "unsupported artifact source"},
|
||||
{name: "canonical", source: ArtifactTranscriptTrimmed, wantID: ArtifactTranscriptTrimmed},
|
||||
{name: "unsupported", source: "narratio.unknown", wantErr: "unsupported artifact source"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got, err := NormalizeSessionArtifactSource(tt.source)
|
||||
if tt.wantErr != "" {
|
||||
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
|
||||
t.Fatalf("NormalizeSessionArtifactSource() error = %v, want contains %q", err, tt.wantErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("NormalizeSessionArtifactSource() error = %v", err)
|
||||
}
|
||||
if got != tt.wantID {
|
||||
t.Fatalf("NormalizeSessionArtifactSource() = %q, want %q", got, tt.wantID)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactPrefersManifestOutput(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
if err := os.MkdirAll(paths.ArtifactsDir, 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
|
||||
if err := os.WriteFile(manifestPath, []byte(`{"segments":[]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
canonicalPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(canonicalPath, []byte(`{"segments":[{"id":123}]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
m := manifest.New("session", time.Now().UTC())
|
||||
m.MarkStageSucceeded("normalize", time.Now().UTC(), []manifest.ArtifactRecord{
|
||||
{Kind: "transcript_normalized", LocalPath: manifestPath, ProducerRunID: "run-123"},
|
||||
})
|
||||
|
||||
resolved, err := ResolveSessionArtifact(paths, m, ArtifactTranscriptFull)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveSessionArtifact() error = %v", err)
|
||||
}
|
||||
if resolved.Path != manifestPath {
|
||||
t.Fatalf("resolved path = %q, want %q", resolved.Path, manifestPath)
|
||||
}
|
||||
if resolved.Provenance != "manifest.normalize.outputs" {
|
||||
t.Fatalf("provenance = %q, want manifest.normalize.outputs", resolved.Provenance)
|
||||
}
|
||||
if resolved.ProducerRunID != "run-123" {
|
||||
t.Fatalf("producer run id = %q, want run-123", resolved.ProducerRunID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactFallsBackToCanonicalPath(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
canonicalPath := filepath.Join(paths.TranscriptsDir, "trimmed.json")
|
||||
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(canonicalPath, []byte(`{"segments":[]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
resolved, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptTrimmed)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveSessionArtifact() error = %v", err)
|
||||
}
|
||||
if resolved.Path != canonicalPath {
|
||||
t.Fatalf("resolved path = %q, want %q", resolved.Path, canonicalPath)
|
||||
}
|
||||
if resolved.Provenance != "fallback.canonical_path" {
|
||||
t.Fatalf("provenance = %q, want fallback.canonical_path", resolved.Provenance)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactMissingReturnsTypedError(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
|
||||
_, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptTrimmed)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !errors.Is(err, ErrSessionArtifactNotFound) {
|
||||
t.Fatalf("errors.Is(err, ErrSessionArtifactNotFound) = false; err=%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactValidatesTranscriptShape(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
canonicalPath := filepath.Join(paths.TranscriptsDir, "processed.json")
|
||||
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(canonicalPath, []byte(`{"not_segments":[]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
_, err := ResolveSessionArtifact(paths, nil, ArtifactTranscriptPolished)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "top-level segments is required") {
|
||||
t.Fatalf("error = %q, want segments validation error", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactWithCatalogBuiltInBehaviorUnchanged(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
canonicalPath := filepath.Join(paths.TranscriptsDir, "trimmed.json")
|
||||
if err := os.MkdirAll(filepath.Dir(canonicalPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(canonicalPath, []byte(`{"segments":[]}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
resolved, err := ResolveSessionArtifactWithCatalog(paths, nil, ArtifactTranscriptTrimmed, NewArtifactCatalog())
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveSessionArtifactWithCatalog() error = %v", err)
|
||||
}
|
||||
if resolved.Path != canonicalPath {
|
||||
t.Fatalf("resolved.Path = %q, want %q", resolved.Path, canonicalPath)
|
||||
}
|
||||
if resolved.Provenance != "fallback.canonical_path" {
|
||||
t.Fatalf("provenance = %q, want fallback.canonical_path", resolved.Provenance)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactWithCatalogConfiguredAvailableGenerated(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
outputPath := filepath.Join(paths.ArtifactsDir, "session_recap.md")
|
||||
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(outputPath, []byte("recap\n"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
sourceID := ConfiguredArtifactSourceID("session_recap")
|
||||
if err := catalog.MarkAvailableGenerated(sourceID, outputPath); err != nil {
|
||||
t.Fatalf("MarkAvailableGenerated() error = %v", err)
|
||||
}
|
||||
|
||||
resolved, err := ResolveSessionArtifactWithCatalog(paths, nil, sourceID, catalog)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveSessionArtifactWithCatalog() error = %v", err)
|
||||
}
|
||||
if resolved.Path != outputPath {
|
||||
t.Fatalf("resolved.Path = %q, want %q", resolved.Path, outputPath)
|
||||
}
|
||||
if resolved.Provenance != ArtifactProvenanceGeneratedCurrentAnalyzeRun {
|
||||
t.Fatalf("provenance = %q, want %q", resolved.Provenance, ArtifactProvenanceGeneratedCurrentAnalyzeRun)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactWithCatalogConfiguredAvailableFromDisk(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
outputPath := filepath.Join(paths.ArtifactsDir, "player_handout.md")
|
||||
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
if err := os.WriteFile(outputPath, []byte("handout\n"), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"player_handout": {Enabled: false, OutputPath: "artifacts/player_handout.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
sourceID := ConfiguredArtifactSourceID("player_handout")
|
||||
if err := catalog.MarkAvailableFromDisk(sourceID, outputPath); err != nil {
|
||||
t.Fatalf("MarkAvailableFromDisk() error = %v", err)
|
||||
}
|
||||
|
||||
resolved, err := ResolveSessionArtifactWithCatalog(paths, nil, sourceID, catalog)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveSessionArtifactWithCatalog() error = %v", err)
|
||||
}
|
||||
if resolved.Provenance != ArtifactProvenanceDisabledFromDisk {
|
||||
t.Fatalf("provenance = %q, want %q", resolved.Provenance, ArtifactProvenanceDisabledFromDisk)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactWithCatalogConfiguredPlannedButUnavailable(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
_, err := ResolveSessionArtifactWithCatalog(paths, nil, ConfiguredArtifactSourceID("session_recap"), catalog)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !errors.Is(err, ErrSessionArtifactNotFound) {
|
||||
t.Fatalf("errors.Is(err, ErrSessionArtifactNotFound)=false; err=%v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveSessionArtifactWithCatalogUnsupportedConfiguredSourceFails(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "campaign", "session")
|
||||
catalog := NewArtifactCatalog()
|
||||
_, err := ResolveSessionArtifactWithCatalog(paths, nil, "narratio.artifact.unknown", catalog)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "unsupported artifact source") {
|
||||
t.Fatalf("error = %q, want unsupported artifact source", err.Error())
|
||||
}
|
||||
}
|
||||
224
internal/artifacts/catalog.go
Normal file
224
internal/artifacts/catalog.go
Normal file
@@ -0,0 +1,224 @@
|
||||
package artifacts
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const (
|
||||
ArtifactProvenanceGeneratedCurrentAnalyzeRun = "generated.current_analyze_run"
|
||||
ArtifactProvenanceDisabledFromDisk = "filesystem.disabled_artifact_output"
|
||||
)
|
||||
|
||||
// ConfiguredArtifactDefinition describes one configured analyze artifact.
|
||||
type ConfiguredArtifactDefinition struct {
|
||||
Enabled bool
|
||||
OutputPath string
|
||||
}
|
||||
|
||||
// CatalogEntry is one runtime catalog entry resolved by source ID.
|
||||
type CatalogEntry struct {
|
||||
SourceID string
|
||||
ConfiguredKey string
|
||||
CanonicalRelPath string
|
||||
ProducerStage string
|
||||
OutputKind string
|
||||
Planned bool
|
||||
Executable bool
|
||||
Available bool
|
||||
Path string
|
||||
Provenance string
|
||||
}
|
||||
|
||||
// ArtifactCatalog tracks built-in and configured artifact definitions and runtime state.
|
||||
type ArtifactCatalog struct {
|
||||
entries map[string]CatalogEntry
|
||||
configuredIndex map[string]string
|
||||
}
|
||||
|
||||
// NewArtifactCatalog returns an empty runtime artifact catalog.
|
||||
func NewArtifactCatalog() *ArtifactCatalog {
|
||||
return &ArtifactCatalog{
|
||||
entries: map[string]CatalogEntry{},
|
||||
configuredIndex: map[string]string{},
|
||||
}
|
||||
}
|
||||
|
||||
// ConfiguredArtifactSourceID converts a configured artifact key into canonical source ID.
|
||||
func ConfiguredArtifactSourceID(key string) string {
|
||||
return "narratio.artifact." + strings.TrimSpace(key)
|
||||
}
|
||||
|
||||
// RegisterBuiltIns registers built-in source definitions used by runtime artifact resolution.
|
||||
func (c *ArtifactCatalog) RegisterBuiltIns() error {
|
||||
for _, id := range runtimeBuiltInArtifactIDs() {
|
||||
spec, ok := artifactRegistry[id]
|
||||
if !ok {
|
||||
return fmt.Errorf("register built-ins: source %q not found in artifact registry", id)
|
||||
}
|
||||
if err := c.addEntry(CatalogEntry{
|
||||
SourceID: spec.ID,
|
||||
CanonicalRelPath: spec.CanonicalRelPath,
|
||||
ProducerStage: spec.ProducerStage,
|
||||
OutputKind: spec.OutputKind,
|
||||
Planned: true,
|
||||
}); err != nil {
|
||||
return fmt.Errorf("register built-ins: %w", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// RegisterConfiguredArtifacts registers configured artifacts and applies executable selection.
|
||||
func (c *ArtifactCatalog) RegisterConfiguredArtifacts(
|
||||
configured map[string]ConfiguredArtifactDefinition,
|
||||
selected []string,
|
||||
) error {
|
||||
keys := make([]string, 0, len(configured))
|
||||
for key := range configured {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
|
||||
selectedSet := map[string]struct{}{}
|
||||
for _, key := range selected {
|
||||
trimmed := strings.TrimSpace(key)
|
||||
if trimmed == "" {
|
||||
return fmt.Errorf("selected artifact keys must be non-empty")
|
||||
}
|
||||
selectedSet[trimmed] = struct{}{}
|
||||
}
|
||||
|
||||
for _, key := range keys {
|
||||
trimmed := strings.TrimSpace(key)
|
||||
if trimmed == "" {
|
||||
return fmt.Errorf("configured artifact keys must be non-empty")
|
||||
}
|
||||
def := configured[key]
|
||||
sourceID := ConfiguredArtifactSourceID(trimmed)
|
||||
if _, exists := c.configuredIndex[trimmed]; exists {
|
||||
return fmt.Errorf("duplicate configured artifact key %q", trimmed)
|
||||
}
|
||||
|
||||
executable := def.Enabled
|
||||
if len(selectedSet) > 0 {
|
||||
_, executable = selectedSet[trimmed]
|
||||
}
|
||||
|
||||
if err := c.addEntry(CatalogEntry{
|
||||
SourceID: sourceID,
|
||||
ConfiguredKey: trimmed,
|
||||
CanonicalRelPath: strings.TrimSpace(def.OutputPath),
|
||||
ProducerStage: "analyze",
|
||||
OutputKind: "scriptorium_artifact",
|
||||
Planned: true,
|
||||
Executable: executable,
|
||||
}); err != nil {
|
||||
return fmt.Errorf("register configured artifact %q: %w", trimmed, err)
|
||||
}
|
||||
c.configuredIndex[trimmed] = sourceID
|
||||
}
|
||||
|
||||
if len(selectedSet) > 0 {
|
||||
for key := range selectedSet {
|
||||
if _, ok := c.configuredIndex[key]; !ok {
|
||||
return fmt.Errorf("selected artifact %q is not configured", key)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Lookup returns one catalog entry by source ID.
|
||||
func (c *ArtifactCatalog) Lookup(sourceID string) (CatalogEntry, bool) {
|
||||
if c == nil {
|
||||
return CatalogEntry{}, false
|
||||
}
|
||||
entry, ok := c.entries[strings.TrimSpace(sourceID)]
|
||||
return entry, ok
|
||||
}
|
||||
|
||||
// SourceIDForConfiguredKey returns canonical source ID for one configured key.
|
||||
func (c *ArtifactCatalog) SourceIDForConfiguredKey(key string) (string, bool) {
|
||||
if c == nil {
|
||||
return "", false
|
||||
}
|
||||
sourceID, ok := c.configuredIndex[strings.TrimSpace(key)]
|
||||
return sourceID, ok
|
||||
}
|
||||
|
||||
// ListConfigured returns configured entries sorted by configured key.
|
||||
func (c *ArtifactCatalog) ListConfigured() []CatalogEntry {
|
||||
if c == nil || len(c.configuredIndex) == 0 {
|
||||
return nil
|
||||
}
|
||||
keys := make([]string, 0, len(c.configuredIndex))
|
||||
for key := range c.configuredIndex {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
out := make([]CatalogEntry, 0, len(keys))
|
||||
for _, key := range keys {
|
||||
sourceID := c.configuredIndex[key]
|
||||
out = append(out, c.entries[sourceID])
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// MarkAvailableGenerated marks one source as available in current analyze execution.
|
||||
func (c *ArtifactCatalog) MarkAvailableGenerated(sourceID, path string) error {
|
||||
return c.markAvailable(sourceID, path, ArtifactProvenanceGeneratedCurrentAnalyzeRun)
|
||||
}
|
||||
|
||||
// MarkAvailableFromDisk marks one source as available from disabled artifact on disk.
|
||||
func (c *ArtifactCatalog) MarkAvailableFromDisk(sourceID, path string) error {
|
||||
return c.markAvailable(sourceID, path, ArtifactProvenanceDisabledFromDisk)
|
||||
}
|
||||
|
||||
func (c *ArtifactCatalog) markAvailable(sourceID, path, provenance string) error {
|
||||
if c == nil {
|
||||
return fmt.Errorf("artifact catalog is nil")
|
||||
}
|
||||
normalizedID := strings.TrimSpace(sourceID)
|
||||
entry, ok := c.entries[normalizedID]
|
||||
if !ok {
|
||||
return fmt.Errorf("unknown artifact source %q", sourceID)
|
||||
}
|
||||
trimmedPath := strings.TrimSpace(path)
|
||||
if trimmedPath == "" {
|
||||
return fmt.Errorf("artifact path is required")
|
||||
}
|
||||
entry.Available = true
|
||||
entry.Path = trimmedPath
|
||||
entry.Provenance = provenance
|
||||
c.entries[normalizedID] = entry
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *ArtifactCatalog) addEntry(entry CatalogEntry) error {
|
||||
if c == nil {
|
||||
return fmt.Errorf("artifact catalog is nil")
|
||||
}
|
||||
sourceID := strings.TrimSpace(entry.SourceID)
|
||||
if sourceID == "" {
|
||||
return fmt.Errorf("source id is required")
|
||||
}
|
||||
if _, exists := c.entries[sourceID]; exists {
|
||||
return fmt.Errorf("source id %q is already registered", sourceID)
|
||||
}
|
||||
entry.SourceID = sourceID
|
||||
c.entries[sourceID] = entry
|
||||
return nil
|
||||
}
|
||||
|
||||
func runtimeBuiltInArtifactIDs() []string {
|
||||
return []string{
|
||||
ArtifactTranscriptMerged,
|
||||
ArtifactTranscriptPolished,
|
||||
ArtifactTranscriptFull,
|
||||
ArtifactTranscriptTrimmed,
|
||||
ArtifactBoundsSession,
|
||||
}
|
||||
}
|
||||
188
internal/artifacts/catalog_test.go
Normal file
188
internal/artifacts/catalog_test.go
Normal file
@@ -0,0 +1,188 @@
|
||||
package artifacts
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestArtifactCatalogRegisterBuiltInsAndLookup(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterBuiltIns(); err != nil {
|
||||
t.Fatalf("RegisterBuiltIns() error = %v", err)
|
||||
}
|
||||
|
||||
entry, ok := catalog.Lookup(ArtifactTranscriptFull)
|
||||
if !ok {
|
||||
t.Fatalf("Lookup(%q) ok = false, want true", ArtifactTranscriptFull)
|
||||
}
|
||||
if !entry.Planned {
|
||||
t.Fatalf("entry.Planned = false, want true")
|
||||
}
|
||||
if entry.Executable {
|
||||
t.Fatalf("entry.Executable = true, want false")
|
||||
}
|
||||
if entry.CanonicalRelPath != "transcripts/normalized.json" {
|
||||
t.Fatalf("entry.CanonicalRelPath = %q, want transcripts/normalized.json", entry.CanonicalRelPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCatalogRegisterConfiguredArtifactsDefaultsToEnabled(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(map[string]ConfiguredArtifactDefinition{
|
||||
"player_handout": {Enabled: false, OutputPath: "artifacts/player_handout.md"},
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
}, nil); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
|
||||
recapID, ok := catalog.SourceIDForConfiguredKey("session_recap")
|
||||
if !ok {
|
||||
t.Fatal("SourceIDForConfiguredKey(session_recap) ok = false, want true")
|
||||
}
|
||||
recap, ok := catalog.Lookup(recapID)
|
||||
if !ok {
|
||||
t.Fatalf("Lookup(%q) ok = false, want true", recapID)
|
||||
}
|
||||
if !recap.Executable {
|
||||
t.Fatalf("recap.Executable = false, want true")
|
||||
}
|
||||
|
||||
handoutID, ok := catalog.SourceIDForConfiguredKey("player_handout")
|
||||
if !ok {
|
||||
t.Fatal("SourceIDForConfiguredKey(player_handout) ok = false, want true")
|
||||
}
|
||||
handout, ok := catalog.Lookup(handoutID)
|
||||
if !ok {
|
||||
t.Fatalf("Lookup(%q) ok = false, want true", handoutID)
|
||||
}
|
||||
if handout.Executable {
|
||||
t.Fatalf("handout.Executable = true, want false")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCatalogRegisterConfiguredArtifactsSelectedSetOverridesEnabled(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
"player_handout": {Enabled: false, OutputPath: "artifacts/player_handout.md"},
|
||||
},
|
||||
[]string{"player_handout"},
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
|
||||
entries := catalog.ListConfigured()
|
||||
if len(entries) != 2 {
|
||||
t.Fatalf("ListConfigured() len = %d, want 2", len(entries))
|
||||
}
|
||||
if entries[0].ConfiguredKey != "player_handout" || entries[0].Executable != true {
|
||||
t.Fatalf("entries[0] = %+v, want player_handout executable", entries[0])
|
||||
}
|
||||
if entries[1].ConfiguredKey != "session_recap" || entries[1].Executable != false {
|
||||
t.Fatalf("entries[1] = %+v, want session_recap disabled by selection", entries[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCatalogRejectsSelectedUnknownArtifact(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
[]string{"unknown"},
|
||||
)
|
||||
if err == nil {
|
||||
t.Fatal("RegisterConfiguredArtifacts() error = nil, want non-nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCatalogRejectsConfiguredSourceConflictAcrossRegistrations(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() first call error = %v", err)
|
||||
}
|
||||
err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
)
|
||||
if err == nil {
|
||||
t.Fatal("RegisterConfiguredArtifacts() error = nil, want non-nil")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCatalogMarkAvailableGenerated(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
|
||||
sourceID, _ := catalog.SourceIDForConfiguredKey("session_recap")
|
||||
if err := catalog.MarkAvailableGenerated(sourceID, "/tmp/session_recap.md"); err != nil {
|
||||
t.Fatalf("MarkAvailableGenerated() error = %v", err)
|
||||
}
|
||||
entry, _ := catalog.Lookup(sourceID)
|
||||
if !entry.Available {
|
||||
t.Fatalf("entry.Available = false, want true")
|
||||
}
|
||||
if entry.Provenance != ArtifactProvenanceGeneratedCurrentAnalyzeRun {
|
||||
t.Fatalf("entry.Provenance = %q, want %q", entry.Provenance, ArtifactProvenanceGeneratedCurrentAnalyzeRun)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCatalogMarkAvailableFromDisk(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: false, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
|
||||
sourceID, _ := catalog.SourceIDForConfiguredKey("session_recap")
|
||||
if err := catalog.MarkAvailableFromDisk(sourceID, "/tmp/session_recap.md"); err != nil {
|
||||
t.Fatalf("MarkAvailableFromDisk() error = %v", err)
|
||||
}
|
||||
entry, _ := catalog.Lookup(sourceID)
|
||||
if !entry.Available {
|
||||
t.Fatalf("entry.Available = false, want true")
|
||||
}
|
||||
if entry.Provenance != ArtifactProvenanceDisabledFromDisk {
|
||||
t.Fatalf("entry.Provenance = %q, want %q", entry.Provenance, ArtifactProvenanceDisabledFromDisk)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCatalogLookupPlannedButUnavailable(t *testing.T) {
|
||||
catalog := NewArtifactCatalog()
|
||||
if err := catalog.RegisterConfiguredArtifacts(
|
||||
map[string]ConfiguredArtifactDefinition{
|
||||
"session_recap": {Enabled: true, OutputPath: "artifacts/session_recap.md"},
|
||||
},
|
||||
nil,
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterConfiguredArtifacts() error = %v", err)
|
||||
}
|
||||
|
||||
sourceID, _ := catalog.SourceIDForConfiguredKey("session_recap")
|
||||
entry, ok := catalog.Lookup(sourceID)
|
||||
if !ok {
|
||||
t.Fatalf("Lookup(%q) ok = false, want true", sourceID)
|
||||
}
|
||||
if !entry.Planned {
|
||||
t.Fatalf("entry.Planned = false, want true")
|
||||
}
|
||||
if entry.Available {
|
||||
t.Fatalf("entry.Available = true, want false")
|
||||
}
|
||||
}
|
||||
@@ -30,13 +30,13 @@ func NewLocalStore(workspaceRoot string) *LocalStore {
|
||||
return &LocalStore{WorkspaceRoot: workspaceRoot}
|
||||
}
|
||||
|
||||
// SessionPaths resolves canonical paths for a session workdir.
|
||||
func (s *LocalStore) SessionPaths(sessionID string) SessionPaths {
|
||||
return buildSessionPaths(s.WorkspaceRoot, sessionID)
|
||||
// SessionPathsFor resolves canonical campaign-aware paths for a session workdir.
|
||||
func (s *LocalStore) SessionPathsFor(campaign, sessionID string) SessionPaths {
|
||||
return buildSessionPaths(s.WorkspaceRoot, campaign, sessionID)
|
||||
}
|
||||
|
||||
// EnsureLayout creates and verifies the canonical session workdir directory layout.
|
||||
func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
// EnsureLayoutFor creates and verifies campaign-aware session layout.
|
||||
func (s *LocalStore) EnsureLayoutFor(campaign, sessionID string) (SessionPaths, error) {
|
||||
if strings.TrimSpace(s.WorkspaceRoot) == "" {
|
||||
return SessionPaths{}, fmt.Errorf("workspace root is required")
|
||||
}
|
||||
@@ -44,7 +44,22 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
return SessionPaths{}, fmt.Errorf("sessionID is required")
|
||||
}
|
||||
|
||||
paths := s.SessionPaths(sessionID)
|
||||
campaign = strings.TrimSpace(campaign)
|
||||
if campaign == "" {
|
||||
return SessionPaths{}, fmt.Errorf("campaign is required")
|
||||
}
|
||||
|
||||
return s.ensureLayout(s.SessionPathsFor(campaign, sessionID))
|
||||
}
|
||||
|
||||
func (s *LocalStore) ensureLayout(paths SessionPaths) (SessionPaths, error) {
|
||||
if strings.TrimSpace(s.WorkspaceRoot) == "" {
|
||||
return SessionPaths{}, fmt.Errorf("workspace root is required")
|
||||
}
|
||||
if strings.TrimSpace(paths.SessionID) == "" {
|
||||
return SessionPaths{}, fmt.Errorf("sessionID is required")
|
||||
}
|
||||
|
||||
dirs := []string{
|
||||
paths.Root,
|
||||
paths.InputsDir,
|
||||
@@ -53,8 +68,11 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
paths.TranscriptsRawDir,
|
||||
paths.TranscriptsTrimmedDir,
|
||||
paths.ArtifactsDir,
|
||||
paths.ReportsDir,
|
||||
paths.ConfigDir,
|
||||
paths.LogsDir,
|
||||
paths.CurrentDir,
|
||||
paths.RunsDir,
|
||||
}
|
||||
|
||||
for _, dir := range dirs {
|
||||
@@ -66,13 +84,16 @@ func (s *LocalStore) EnsureLayout(sessionID string) (SessionPaths, error) {
|
||||
return paths, nil
|
||||
}
|
||||
|
||||
// CopyInput copies an input file into the session workdir under destRelativePath.
|
||||
func (s *LocalStore) CopyInput(sessionID, srcPath, destRelativePath string) (Ref, error) {
|
||||
paths, err := s.EnsureLayout(sessionID)
|
||||
// CopyInputFor copies an input file into the campaign-aware session workdir under destRelativePath.
|
||||
func (s *LocalStore) CopyInputFor(campaign, sessionID, srcPath, destRelativePath string) (Ref, error) {
|
||||
paths, err := s.EnsureLayoutFor(campaign, sessionID)
|
||||
if err != nil {
|
||||
return Ref{}, err
|
||||
}
|
||||
return s.copyInputWithPaths(paths, sessionID, srcPath, destRelativePath)
|
||||
}
|
||||
|
||||
func (s *LocalStore) copyInputWithPaths(paths SessionPaths, sessionID, srcPath, destRelativePath string) (Ref, error) {
|
||||
destAbs, err := resolveInRoot(paths.Root, destRelativePath)
|
||||
if err != nil {
|
||||
return Ref{}, fmt.Errorf("copy input: %w", err)
|
||||
@@ -173,13 +194,16 @@ func (s *LocalStore) Checksum(path string) (string, error) {
|
||||
return digest, nil
|
||||
}
|
||||
|
||||
// AcquireSessionLock acquires an exclusive lock file for a session workdir.
|
||||
func (s *LocalStore) AcquireSessionLock(sessionID string) (*LockHandle, error) {
|
||||
paths, err := s.EnsureLayout(sessionID)
|
||||
// AcquireSessionLockFor acquires an exclusive lock file for a campaign/session workdir.
|
||||
func (s *LocalStore) AcquireSessionLockFor(campaign, sessionID string) (*LockHandle, error) {
|
||||
paths, err := s.EnsureLayoutFor(campaign, sessionID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return s.acquireSessionLockForPaths(paths)
|
||||
}
|
||||
|
||||
func (s *LocalStore) acquireSessionLockForPaths(paths SessionPaths) (*LockHandle, error) {
|
||||
f, err := os.OpenFile(paths.LockPath, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o644)
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrExist) {
|
||||
|
||||
@@ -10,9 +10,9 @@ import (
|
||||
|
||||
func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
paths, err := store.EnsureLayout("session-1")
|
||||
paths, err := store.EnsureLayoutFor("sample-campaign", "session-1")
|
||||
if err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
t.Fatalf("EnsureLayoutFor() error = %v", err)
|
||||
}
|
||||
|
||||
checkDirExists(t, paths.Root)
|
||||
@@ -22,8 +22,11 @@ func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
|
||||
checkDirExists(t, paths.TranscriptsRawDir)
|
||||
checkDirExists(t, paths.TranscriptsTrimmedDir)
|
||||
checkDirExists(t, paths.ArtifactsDir)
|
||||
checkDirExists(t, paths.ReportsDir)
|
||||
checkDirExists(t, paths.ConfigDir)
|
||||
checkDirExists(t, paths.LogsDir)
|
||||
checkDirExists(t, paths.CurrentDir)
|
||||
checkDirExists(t, paths.RunsDir)
|
||||
|
||||
if filepath.Base(paths.ManifestPath) != "manifest.json" {
|
||||
t.Fatalf("ManifestPath = %q, want basename manifest.json", paths.ManifestPath)
|
||||
@@ -33,6 +36,17 @@ func TestEnsureLayoutCreatesExpectedDirectories(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnsureLayoutForRequiresCampaign(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
_, err := store.EnsureLayoutFor("", "session-1")
|
||||
if err == nil {
|
||||
t.Fatal("expected campaign-required error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "campaign is required") {
|
||||
t.Fatalf("error = %v, want campaign-required error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestChecksumCalculation(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
path := filepath.Join(t.TempDir(), "sample.txt")
|
||||
@@ -53,9 +67,9 @@ func TestChecksumCalculation(t *testing.T) {
|
||||
func TestLockAcquireRelease(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
|
||||
lock, err := store.AcquireSessionLock("session-1")
|
||||
lock, err := store.AcquireSessionLockFor("sample-campaign", "session-1")
|
||||
if err != nil {
|
||||
t.Fatalf("AcquireSessionLock() error = %v", err)
|
||||
t.Fatalf("AcquireSessionLockFor() error = %v", err)
|
||||
}
|
||||
|
||||
exists, err := store.Exists(lock.path)
|
||||
@@ -81,15 +95,15 @@ func TestLockAcquireRelease(t *testing.T) {
|
||||
|
||||
func TestLockConflict(t *testing.T) {
|
||||
store := NewLocalStore(t.TempDir())
|
||||
lock1, err := store.AcquireSessionLock("session-1")
|
||||
lock1, err := store.AcquireSessionLockFor("sample-campaign", "session-1")
|
||||
if err != nil {
|
||||
t.Fatalf("first AcquireSessionLock() error = %v", err)
|
||||
t.Fatalf("first AcquireSessionLockFor() error = %v", err)
|
||||
}
|
||||
defer func() {
|
||||
_ = store.ReleaseSessionLock(lock1)
|
||||
}()
|
||||
|
||||
_, err = store.AcquireSessionLock("session-1")
|
||||
_, err = store.AcquireSessionLockFor("sample-campaign", "session-1")
|
||||
if err == nil {
|
||||
t.Fatal("expected lock conflict error, got nil")
|
||||
}
|
||||
@@ -138,9 +152,9 @@ func TestCopyInput(t *testing.T) {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
ref, err := store.CopyInput("session-1", srcPath, "inputs/speakers.yml")
|
||||
ref, err := store.CopyInputFor("sample-campaign", "session-1", srcPath, "inputs/speakers.yml")
|
||||
if err != nil {
|
||||
t.Fatalf("CopyInput() error = %v", err)
|
||||
t.Fatalf("CopyInputFor() error = %v", err)
|
||||
}
|
||||
|
||||
if ref.Kind != "input" {
|
||||
|
||||
@@ -9,6 +9,8 @@ import (
|
||||
// SessionPaths contains canonical local paths for one session work directory.
|
||||
type SessionPaths struct {
|
||||
WorkspaceRoot string
|
||||
CampaignID string
|
||||
SessionID string
|
||||
Root string
|
||||
InputsDir string
|
||||
AudioDir string
|
||||
@@ -16,20 +18,43 @@ type SessionPaths struct {
|
||||
TranscriptsRawDir string
|
||||
TranscriptsTrimmedDir string
|
||||
ArtifactsDir string
|
||||
ReportsDir string
|
||||
ConfigDir string
|
||||
LogsDir string
|
||||
CurrentDir string
|
||||
RunsDir string
|
||||
ManifestPath string
|
||||
LockPath string
|
||||
}
|
||||
|
||||
// SessionWorkDir returns the work directory for one session.
|
||||
func SessionWorkDir(rootDir, sessionID string) string {
|
||||
return filepath.Join(rootDir, config.PathWorkDirSegment, sessionID)
|
||||
// SessionWorkDirForCampaign returns the canonical campaign-aware work directory for one session.
|
||||
func SessionWorkDirForCampaign(rootDir, campaign, sessionID string) string {
|
||||
return filepath.Join(rootDir, config.PathWorkDirSegment, campaign, sessionID)
|
||||
}
|
||||
|
||||
// SessionRunWorkDir returns the campaign/session/run scoped local work directory.
|
||||
func SessionRunWorkDir(rootDir, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(rootDir, config.PathWorkDirSegment, campaign, sessionID, runID)
|
||||
// SessionManifestPathForCampaign returns the canonical session manifest path.
|
||||
func SessionManifestPathForCampaign(rootDir, campaign, sessionID string) string {
|
||||
return filepath.Join(SessionWorkDirForCampaign(rootDir, campaign, sessionID), config.PathManifestFile)
|
||||
}
|
||||
|
||||
// SessionRunsDirForCampaign returns the canonical runs directory for one session.
|
||||
func SessionRunsDirForCampaign(rootDir, campaign, sessionID string) string {
|
||||
return filepath.Join(SessionWorkDirForCampaign(rootDir, campaign, sessionID), config.PathRunsDirSegment)
|
||||
}
|
||||
|
||||
// SessionRunRootForCampaign returns the canonical run root under runs/{run_id}.
|
||||
func SessionRunRootForCampaign(rootDir, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(SessionRunsDirForCampaign(rootDir, campaign, sessionID), runID)
|
||||
}
|
||||
|
||||
// SessionRunManifestPathForCampaign returns the canonical run manifest path under runs/{run_id}/manifest.json.
|
||||
func SessionRunManifestPathForCampaign(rootDir, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(SessionRunRootForCampaign(rootDir, campaign, sessionID, runID), config.PathManifestFile)
|
||||
}
|
||||
|
||||
// SessionRunStageDirForCampaign returns the canonical stage directory under runs/{run_id}/{stage}.
|
||||
func SessionRunStageDirForCampaign(rootDir, campaign, sessionID, runID, stageName string) string {
|
||||
return filepath.Join(SessionRunRootForCampaign(rootDir, campaign, sessionID, runID), stageName)
|
||||
}
|
||||
|
||||
// SessionSpoolAudioDir returns the campaign/session/run scoped local spool audio path.
|
||||
@@ -37,10 +62,16 @@ func SessionSpoolAudioDir(spoolRoot, campaign, sessionID, runID string) string {
|
||||
return filepath.Join(spoolRoot, campaign, sessionID, runID, config.PathAudioDirSegment)
|
||||
}
|
||||
|
||||
func buildSessionPaths(workspaceRoot, sessionID string) SessionPaths {
|
||||
root := SessionWorkDir(workspaceRoot, sessionID)
|
||||
func buildSessionPaths(workspaceRoot, campaign, sessionID string) SessionPaths {
|
||||
root := SessionWorkDirForCampaign(workspaceRoot, campaign, sessionID)
|
||||
return buildSessionPathsFromRoot(workspaceRoot, campaign, sessionID, root)
|
||||
}
|
||||
|
||||
func buildSessionPathsFromRoot(workspaceRoot, campaign, sessionID, root string) SessionPaths {
|
||||
return SessionPaths{
|
||||
WorkspaceRoot: workspaceRoot,
|
||||
CampaignID: campaign,
|
||||
SessionID: sessionID,
|
||||
Root: root,
|
||||
InputsDir: filepath.Join(root, config.PathInputsDirSegment),
|
||||
AudioDir: filepath.Join(root, config.PathAudioDirSegment),
|
||||
@@ -48,8 +79,11 @@ func buildSessionPaths(workspaceRoot, sessionID string) SessionPaths {
|
||||
TranscriptsRawDir: filepath.Join(root, filepath.FromSlash(config.PathTranscriptsRaw)),
|
||||
TranscriptsTrimmedDir: filepath.Join(root, filepath.FromSlash(config.PathTranscriptsTrimmed)),
|
||||
ArtifactsDir: filepath.Join(root, config.PathArtifactsDirSegment),
|
||||
ReportsDir: filepath.Join(root, config.PathReportsDirSegment),
|
||||
ConfigDir: filepath.Join(root, config.PathConfigDirSegment),
|
||||
LogsDir: filepath.Join(root, config.PathLogsDirSegment),
|
||||
CurrentDir: filepath.Join(root, config.PathCurrentDirSegment),
|
||||
RunsDir: filepath.Join(root, config.PathRunsDirSegment),
|
||||
ManifestPath: filepath.Join(root, config.PathManifestFile),
|
||||
LockPath: filepath.Join(root, config.PathLockFile),
|
||||
}
|
||||
|
||||
@@ -5,12 +5,47 @@ import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSessionRunWorkDir(t *testing.T) {
|
||||
func TestSessionWorkDirForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
got := SessionRunWorkDir(root, "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4")
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "20260515T031522Z-a1b2c3d4")
|
||||
got := SessionWorkDirForCampaign(root, "forsaken", "2026-04-19")
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19")
|
||||
if got != want {
|
||||
t.Fatalf("SessionRunWorkDir() = %q, want %q", got, want)
|
||||
t.Fatalf("SessionWorkDirForCampaign() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionManifestPathForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
got := SessionManifestPathForCampaign(root, "forsaken", "2026-04-19")
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "manifest.json")
|
||||
if got != want {
|
||||
t.Fatalf("SessionManifestPathForCampaign() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionRunRootAndStageDirForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
runID := "20260515T031522Z-a1b2c3d4"
|
||||
runRoot := SessionRunRootForCampaign(root, "forsaken", "2026-04-19", runID)
|
||||
wantRoot := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID)
|
||||
if runRoot != wantRoot {
|
||||
t.Fatalf("SessionRunRootForCampaign() = %q, want %q", runRoot, wantRoot)
|
||||
}
|
||||
|
||||
stageDir := SessionRunStageDirForCampaign(root, "forsaken", "2026-04-19", runID, "transcribe")
|
||||
wantStage := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID, "transcribe")
|
||||
if stageDir != wantStage {
|
||||
t.Fatalf("SessionRunStageDirForCampaign() = %q, want %q", stageDir, wantStage)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSessionRunManifestPathForCampaign(t *testing.T) {
|
||||
root := "/tmp/workspace"
|
||||
runID := "20260515T031522Z-a1b2c3d4"
|
||||
got := SessionRunManifestPathForCampaign(root, "forsaken", "2026-04-19", runID)
|
||||
want := filepath.Join(root, "work", "forsaken", "2026-04-19", "runs", runID, "manifest.json")
|
||||
if got != want {
|
||||
t.Fatalf("SessionRunManifestPathForCampaign() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ import (
|
||||
|
||||
func TestResolveSessionLocalPathForRead(t *testing.T) {
|
||||
workspace := t.TempDir()
|
||||
paths := buildSessionPaths(workspace, "s-1")
|
||||
paths := buildSessionPaths(workspace, "sample-campaign", "s-1")
|
||||
if err := os.MkdirAll(paths.TranscriptsRawDir, 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
}
|
||||
@@ -41,7 +41,7 @@ func TestResolveSessionLocalPathForReadRelativeWorkspaceRootQualifiedPath(t *tes
|
||||
t.Fatalf("Rel() error = %v", err)
|
||||
}
|
||||
|
||||
paths := buildSessionPaths(workspaceRel, "s-1")
|
||||
paths := buildSessionPaths(workspaceRel, "sample-campaign", "s-1")
|
||||
target := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil {
|
||||
t.Fatalf("MkdirAll() error = %v", err)
|
||||
@@ -50,7 +50,7 @@ func TestResolveSessionLocalPathForReadRelativeWorkspaceRootQualifiedPath(t *tes
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
manifestPath := filepath.Join(workspaceRel, "work", "s-1", "transcripts", "raw", "alice.json")
|
||||
manifestPath := filepath.Join(workspaceRel, "work", "sample-campaign", "s-1", "transcripts", "raw", "alice.json")
|
||||
got := ResolveSessionLocalPathForRead(paths, manifestPath)
|
||||
if got != filepath.Clean(manifestPath) {
|
||||
t.Fatalf("resolution = %q, want %q", got, filepath.Clean(manifestPath))
|
||||
|
||||
@@ -11,19 +11,19 @@ type S3Store struct {
|
||||
Prefix string
|
||||
}
|
||||
|
||||
// SessionPaths is not implemented for S3-backed storage.
|
||||
func (s *S3Store) SessionPaths(_ string) SessionPaths {
|
||||
// SessionPathsFor is not implemented for S3-backed storage.
|
||||
func (s *S3Store) SessionPathsFor(_, _ string) SessionPaths {
|
||||
return SessionPaths{}
|
||||
}
|
||||
|
||||
// EnsureLayout returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) EnsureLayout(_ string) (SessionPaths, error) {
|
||||
return SessionPaths{}, fmt.Errorf("artifacts s3 ensure layout: not yet implemented")
|
||||
// EnsureLayoutFor returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) EnsureLayoutFor(_, _ string) (SessionPaths, error) {
|
||||
return SessionPaths{}, fmt.Errorf("artifacts s3 ensure layout for campaign/session: not yet implemented")
|
||||
}
|
||||
|
||||
// CopyInput returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) CopyInput(_, _, _ string) (Ref, error) {
|
||||
return Ref{}, fmt.Errorf("artifacts s3 copy input: not yet implemented")
|
||||
// CopyInputFor returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) CopyInputFor(_, _, _, _ string) (Ref, error) {
|
||||
return Ref{}, fmt.Errorf("artifacts s3 copy input for campaign/session: not yet implemented")
|
||||
}
|
||||
|
||||
// Exists returns a not-yet-implemented error in the scaffold.
|
||||
@@ -46,9 +46,9 @@ func (s *S3Store) Checksum(_ string) (string, error) {
|
||||
return "", fmt.Errorf("artifacts s3 checksum: not yet implemented")
|
||||
}
|
||||
|
||||
// AcquireSessionLock returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) AcquireSessionLock(_ string) (*LockHandle, error) {
|
||||
return nil, fmt.Errorf("artifacts s3 acquire lock: not yet implemented")
|
||||
// AcquireSessionLockFor returns a not-yet-implemented error in the scaffold.
|
||||
func (s *S3Store) AcquireSessionLockFor(_, _ string) (*LockHandle, error) {
|
||||
return nil, fmt.Errorf("artifacts s3 acquire lock for campaign/session: not yet implemented")
|
||||
}
|
||||
|
||||
// ReleaseSessionLock returns a not-yet-implemented error in the scaffold.
|
||||
|
||||
@@ -15,13 +15,13 @@ type Ref struct {
|
||||
|
||||
// Store is the local artifact/workdir abstraction used by orchestration code.
|
||||
type Store interface {
|
||||
SessionPaths(sessionID string) SessionPaths
|
||||
EnsureLayout(sessionID string) (SessionPaths, error)
|
||||
CopyInput(sessionID, srcPath, destRelativePath string) (Ref, error)
|
||||
SessionPathsFor(campaign, sessionID string) SessionPaths
|
||||
EnsureLayoutFor(campaign, sessionID string) (SessionPaths, error)
|
||||
CopyInputFor(campaign, sessionID, srcPath, destRelativePath string) (Ref, error)
|
||||
Exists(path string) (bool, error)
|
||||
ExistsRef(ref Ref) (bool, error)
|
||||
WriteFileAtomic(path string, data []byte, perm os.FileMode) error
|
||||
Checksum(path string) (string, error)
|
||||
AcquireSessionLock(sessionID string) (*LockHandle, error)
|
||||
AcquireSessionLockFor(campaign, sessionID string) (*LockHandle, error)
|
||||
ReleaseSessionLock(lock *LockHandle) error
|
||||
}
|
||||
|
||||
@@ -176,6 +176,7 @@ type ScriptoriumConfig struct {
|
||||
// ScriptoriumArtifactConfig configures one named output artifact workflow.
|
||||
type ScriptoriumArtifactConfig struct {
|
||||
Enabled bool `yaml:"enabled"`
|
||||
DependsOn []string `yaml:"depends_on"`
|
||||
RenderDebug *bool `yaml:"render_debug"`
|
||||
PromptID string `yaml:"prompt_id"`
|
||||
ProfileID string `yaml:"profile_id"`
|
||||
|
||||
@@ -11,6 +11,7 @@ const (
|
||||
DefaultS3AccessKeyIDEnv = "OBJECT_STORAGE_KEY_ID"
|
||||
DefaultS3SecretAccessKeyEnv = "OBJECT_STORAGE_KEY"
|
||||
DefaultStorageS3RootPrefix = "dnd"
|
||||
DefaultWorkspaceRoot = "/var/lib/narratio"
|
||||
DefaultSpoolRoot = "/var/spool/narratio"
|
||||
|
||||
DefaultWhisperXLanguage = "en"
|
||||
@@ -29,8 +30,9 @@ const (
|
||||
DefaultAuditaTimeout = "3h"
|
||||
DefaultAuditaReport = true
|
||||
|
||||
DefaultScriptoriumBinary = "scriptorium"
|
||||
DefaultScriptoriumTimeout = "10m"
|
||||
DefaultScriptoriumBinary = "scriptorium"
|
||||
DefaultScriptoriumTimeout = "10m"
|
||||
DefaultScriptoriumArtifactOutputRoot = "artifacts"
|
||||
|
||||
DefaultTrimBoundsTimeout = "10m"
|
||||
DefaultTrimSeriatimReport = false
|
||||
@@ -49,8 +51,11 @@ const (
|
||||
PathTranscriptsRaw = "transcripts/raw"
|
||||
PathTranscriptsTrimmed = "transcripts/trimmed"
|
||||
PathArtifactsDirSegment = "artifacts"
|
||||
PathReportsDirSegment = "reports"
|
||||
PathConfigDirSegment = "config"
|
||||
PathLogsDirSegment = "logs"
|
||||
PathCurrentDirSegment = "current"
|
||||
PathRunsDirSegment = "runs"
|
||||
PathManifestFile = "manifest.json"
|
||||
PathLockFile = ".lock"
|
||||
PathTranscriptMerged = "transcripts/merged.json"
|
||||
|
||||
@@ -152,6 +152,7 @@ func applyPipelineDefaults(cfg *PipelineConfig) {
|
||||
if cfg == nil {
|
||||
return
|
||||
}
|
||||
applyWorkspaceDefaults(&cfg.Workspace)
|
||||
applyStorageDefaults(&cfg.Storage)
|
||||
applySpoolDefaults(&cfg.Spool)
|
||||
applyArchiveDefaults(&cfg.Archive)
|
||||
@@ -166,6 +167,15 @@ func applyPipelineDefaults(cfg *PipelineConfig) {
|
||||
applyScriptoriumDefaults(cfg.Scriptorium)
|
||||
}
|
||||
|
||||
func applyWorkspaceDefaults(cfg *WorkspaceConfig) {
|
||||
if cfg == nil {
|
||||
return
|
||||
}
|
||||
if cfg.Root == "" {
|
||||
cfg.Root = DefaultWorkspaceRoot
|
||||
}
|
||||
}
|
||||
|
||||
func applyStorageDefaults(cfg *StorageConfig) {
|
||||
if cfg == nil {
|
||||
return
|
||||
|
||||
@@ -15,6 +15,7 @@ func TestLoadAndValidate(t *testing.T) {
|
||||
wantLoadErr string
|
||||
wantValidate string
|
||||
checkDefault bool
|
||||
wantRoot string
|
||||
}{
|
||||
{
|
||||
name: "valid minimal config",
|
||||
@@ -39,6 +40,7 @@ inputs:
|
||||
glossary_file: ./glossary.yml
|
||||
`,
|
||||
checkDefault: true,
|
||||
wantRoot: "/tmp/narratio",
|
||||
},
|
||||
{
|
||||
name: "seriatim and audita sections can be omitted",
|
||||
@@ -59,6 +61,26 @@ inputs:
|
||||
glossary_file: ./glossary.yml
|
||||
`,
|
||||
checkDefault: true,
|
||||
wantRoot: "/tmp/narratio",
|
||||
},
|
||||
{
|
||||
name: "workspace root defaults when omitted",
|
||||
pipelineYAML: `whisperx:
|
||||
transcribe_url: https://transcription.ai.rakestrawhome.com/transcribe
|
||||
analyzer:
|
||||
timeout: 20m
|
||||
notification:
|
||||
timeout: 15s
|
||||
`,
|
||||
sessionYAML: `session_id: 2026-05-03
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./speakers.yml
|
||||
autocorrect_file: ./autocorrect.yml
|
||||
glossary_file: ./glossary.yml
|
||||
`,
|
||||
checkDefault: true,
|
||||
wantRoot: DefaultWorkspaceRoot,
|
||||
},
|
||||
{
|
||||
name: "unknown pipeline field fails",
|
||||
@@ -713,6 +735,9 @@ inputs:
|
||||
t.Fatalf("SessionPath = %q, want %q", cfg.SessionPath, sessionPath)
|
||||
}
|
||||
if tt.checkDefault {
|
||||
if tt.wantRoot != "" && cfg.Pipeline.Workspace.Root != tt.wantRoot {
|
||||
t.Fatalf("workspace.root = %q, want %q", cfg.Pipeline.Workspace.Root, tt.wantRoot)
|
||||
}
|
||||
if cfg.Pipeline.WhisperX.Language != "en" {
|
||||
t.Fatalf("whisperx.language = %q, want %q", cfg.Pipeline.WhisperX.Language, "en")
|
||||
}
|
||||
@@ -866,15 +891,59 @@ func TestValidateMissingAudioSource(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestExamplesLoadAndValidate(t *testing.T) {
|
||||
pipelinePath := filepath.Join("..", "..", "examples", "pipeline.minimal.yml")
|
||||
sessionPath := filepath.Join("..", "..", "examples", "session.minimal.yml")
|
||||
|
||||
cfg, err := Load(pipelinePath, sessionPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load(examples) error = %v", err)
|
||||
examplesDir := filepath.Join("..", "..", "examples")
|
||||
tests := []struct {
|
||||
name string
|
||||
pipelineFile string
|
||||
sessionFile string
|
||||
sessionOpts SessionLoadOptions
|
||||
}{
|
||||
{
|
||||
name: "minimal pipeline with local audio session",
|
||||
pipelineFile: "pipeline.minimal.yml",
|
||||
sessionFile: "session.local-audio.yml",
|
||||
},
|
||||
{
|
||||
name: "production pipeline with s3 audio session",
|
||||
pipelineFile: "pipeline.production.yml",
|
||||
sessionFile: "session.s3-audio.yml",
|
||||
},
|
||||
{
|
||||
name: "full annotated pipeline with local audio session",
|
||||
pipelineFile: "pipeline.full.annotated.yml",
|
||||
sessionFile: "session.local-audio.yml",
|
||||
},
|
||||
{
|
||||
name: "template session renders with session_id option",
|
||||
pipelineFile: "pipeline.minimal.yml",
|
||||
sessionFile: "session.template.yml",
|
||||
sessionOpts: SessionLoadOptions{
|
||||
SessionID: "2026-05-03",
|
||||
},
|
||||
},
|
||||
}
|
||||
if err := Validate(cfg); err != nil {
|
||||
t.Fatalf("Validate(examples) error = %v", err)
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
pipelinePath := filepath.Join(examplesDir, tt.pipelineFile)
|
||||
sessionPath := filepath.Join(examplesDir, tt.sessionFile)
|
||||
|
||||
var (
|
||||
cfg *Config
|
||||
err error
|
||||
)
|
||||
if strings.TrimSpace(tt.sessionOpts.SessionID) == "" {
|
||||
cfg, err = Load(pipelinePath, sessionPath)
|
||||
} else {
|
||||
cfg, err = LoadWithSessionOptions(pipelinePath, sessionPath, tt.sessionOpts)
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("load example config error = %v", err)
|
||||
}
|
||||
if err := Validate(cfg); err != nil {
|
||||
t.Fatalf("validate example config error = %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: processed_transcript
|
||||
source: narratio.transcript.polished
|
||||
required: true
|
||||
vars:
|
||||
session_id: true
|
||||
@@ -104,7 +104,7 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: processed_transcript
|
||||
source: narratio.transcript.polished
|
||||
required: true
|
||||
previous_recap:
|
||||
source: previous_session_artifact
|
||||
@@ -116,6 +116,37 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
|
||||
output_kind: session_recap
|
||||
`,
|
||||
},
|
||||
{
|
||||
name: "canonical artifact source is accepted",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
`,
|
||||
},
|
||||
{
|
||||
name: "unknown artifact source fails validation",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.unknown
|
||||
required: true
|
||||
`,
|
||||
wantValidateErr: `pipeline.scriptorium.artifacts.session_recap.inputs.transcript.source "narratio.unknown" is unsupported`,
|
||||
},
|
||||
{
|
||||
name: "artifact render_debug override is accepted",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
@@ -129,7 +160,7 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: processed_transcript
|
||||
source: narratio.transcript.polished
|
||||
required: true
|
||||
`,
|
||||
assert: func(t *testing.T, cfg *Config) {
|
||||
@@ -151,7 +182,7 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: processed_transcript
|
||||
source: narratio.transcript.polished
|
||||
required: true
|
||||
player_summary:
|
||||
enabled: true
|
||||
@@ -161,7 +192,7 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
|
||||
timeout: 3m
|
||||
inputs:
|
||||
transcript:
|
||||
source: processed_transcript
|
||||
source: narratio.transcript.polished
|
||||
required: true
|
||||
`,
|
||||
assert: func(t *testing.T, cfg *Config) {
|
||||
@@ -174,6 +205,195 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
|
||||
}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "valid artifact dependency is accepted",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.trimmed
|
||||
required: true
|
||||
player_handout:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- session_recap
|
||||
prompt_id: dnd.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
`,
|
||||
},
|
||||
{
|
||||
name: "valid dependency on disabled artifact with output path is accepted",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: false
|
||||
output_path: artifacts/session_recap.md
|
||||
player_handout:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- session_recap
|
||||
prompt_id: dnd.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
`,
|
||||
},
|
||||
{
|
||||
name: "invalid artifact name fails validation",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
SessionRecap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
`,
|
||||
wantValidateErr: "pipeline.scriptorium.artifacts keys must match ^[a-z][a-z0-9_]*$",
|
||||
},
|
||||
{
|
||||
name: "artifact output path outside artifacts root fails validation",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: transcripts/session_recap.md
|
||||
`,
|
||||
wantValidateErr: "pipeline.scriptorium.artifacts.session_recap.output_path must be under artifacts/",
|
||||
},
|
||||
{
|
||||
name: "missing depends_on for artifact source fails validation",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
inputs:
|
||||
transcript:
|
||||
source: narratio.transcript.polished
|
||||
required: true
|
||||
player_handout:
|
||||
enabled: true
|
||||
prompt_id: dnd.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
`,
|
||||
wantValidateErr: `pipeline.scriptorium.artifacts.player_handout.inputs.recap.source "narratio.artifact.session_recap" requires depends_on entry "session_recap"`,
|
||||
},
|
||||
{
|
||||
name: "dependency on unknown artifact fails validation",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
player_handout:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- session_recap
|
||||
prompt_id: dnd.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
`,
|
||||
wantValidateErr: `pipeline.scriptorium.artifacts.player_handout.depends_on[0] "session_recap" is not a configured artifact key`,
|
||||
},
|
||||
{
|
||||
name: "self dependency fails validation",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- session_recap
|
||||
prompt_id: dnd.session_recap
|
||||
output_path: artifacts/session_recap.md
|
||||
`,
|
||||
wantValidateErr: "pipeline.scriptorium.artifacts.session_recap.depends_on must not include itself",
|
||||
},
|
||||
{
|
||||
name: "enabled dependency cycle fails validation",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
artifact_a:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- artifact_b
|
||||
prompt_id: dnd.a
|
||||
output_path: artifacts/a.md
|
||||
inputs:
|
||||
b:
|
||||
source: narratio.artifact.artifact_b
|
||||
required: true
|
||||
artifact_b:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- artifact_a
|
||||
prompt_id: dnd.b
|
||||
output_path: artifacts/b.md
|
||||
inputs:
|
||||
a:
|
||||
source: narratio.artifact.artifact_a
|
||||
required: true
|
||||
`,
|
||||
wantValidateErr: "pipeline.scriptorium.artifacts enabled dependencies must not contain cycles",
|
||||
},
|
||||
{
|
||||
name: "artifact source typo fails validation",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
player_handout:
|
||||
enabled: true
|
||||
prompt_id: dnd.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session-recap
|
||||
required: true
|
||||
`,
|
||||
wantValidateErr: `pipeline.scriptorium.artifacts.player_handout.inputs.recap.source "narratio.artifact.session-recap" is unsupported`,
|
||||
},
|
||||
{
|
||||
name: "referenced disabled artifact missing output path fails validation",
|
||||
scriptoriumYAML: `scriptorium:
|
||||
binary: scriptorium
|
||||
artifacts:
|
||||
session_recap:
|
||||
enabled: false
|
||||
player_handout:
|
||||
enabled: true
|
||||
depends_on:
|
||||
- session_recap
|
||||
prompt_id: dnd.player_handout
|
||||
output_path: artifacts/player_handout.md
|
||||
inputs:
|
||||
recap:
|
||||
source: narratio.artifact.session_recap
|
||||
required: true
|
||||
`,
|
||||
wantValidateErr: "pipeline.scriptorium.artifacts.session_recap.output_path is required when artifact is referenced",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -227,6 +447,7 @@ audita:
|
||||
`
|
||||
|
||||
const testSessionBaseYAML = `session_id: 2026-05-03
|
||||
campaign: test-campaign
|
||||
inputs:
|
||||
audio_dir: ./audio
|
||||
speakers_file: ./speakers.yml
|
||||
|
||||
@@ -324,28 +324,78 @@ func validateScriptorium(cfg *ScriptoriumConfig) error {
|
||||
return err
|
||||
}
|
||||
|
||||
for artifactName, artifactCfg := range cfg.Artifacts {
|
||||
trimmedArtifactName := strings.TrimSpace(artifactName)
|
||||
if trimmedArtifactName == "" {
|
||||
return fmt.Errorf("pipeline.scriptorium.artifacts keys must be non-empty")
|
||||
configuredArtifacts := make(map[string]struct{}, len(cfg.Artifacts))
|
||||
referencedArtifacts := make(map[string]struct{})
|
||||
for artifactName := range cfg.Artifacts {
|
||||
if !scriptoriumArtifactKeyRE.MatchString(strings.TrimSpace(artifactName)) {
|
||||
return fmt.Errorf("pipeline.scriptorium.artifacts keys must match ^[a-z][a-z0-9_]*$")
|
||||
}
|
||||
configuredArtifacts[artifactName] = struct{}{}
|
||||
}
|
||||
|
||||
for artifactName, artifactCfg := range cfg.Artifacts {
|
||||
if artifactCfg.Enabled && strings.TrimSpace(artifactCfg.PromptID) == "" {
|
||||
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.prompt_id is required when enabled", artifactName)
|
||||
}
|
||||
if artifactCfg.Enabled && strings.TrimSpace(artifactCfg.OutputPath) == "" {
|
||||
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.output_path is required when enabled", artifactName)
|
||||
}
|
||||
if strings.TrimSpace(artifactCfg.OutputPath) != "" {
|
||||
pathField := "pipeline.scriptorium.artifacts." + artifactName + ".output_path"
|
||||
if err := validateRelativeSafePath(pathField, artifactCfg.OutputPath); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validatePathWithinRoot(pathField, artifactCfg.OutputPath, DefaultScriptoriumArtifactOutputRoot); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := validateDuration("pipeline.scriptorium.artifacts."+artifactName+".timeout", artifactCfg.Timeout); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
depSet := make(map[string]struct{}, len(artifactCfg.DependsOn))
|
||||
for i, depName := range artifactCfg.DependsOn {
|
||||
trimmedDep := strings.TrimSpace(depName)
|
||||
field := fmt.Sprintf("pipeline.scriptorium.artifacts.%s.depends_on[%d]", artifactName, i)
|
||||
if trimmedDep == "" {
|
||||
return fmt.Errorf("%s must be non-empty", field)
|
||||
}
|
||||
if _, ok := configuredArtifacts[trimmedDep]; !ok {
|
||||
return fmt.Errorf("%s %q is not a configured artifact key", field, depName)
|
||||
}
|
||||
if trimmedDep == artifactName {
|
||||
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.depends_on must not include itself", artifactName)
|
||||
}
|
||||
depSet[trimmedDep] = struct{}{}
|
||||
referencedArtifacts[trimmedDep] = struct{}{}
|
||||
}
|
||||
|
||||
for inputName, inputCfg := range artifactCfg.Inputs {
|
||||
trimmedInputName := strings.TrimSpace(inputName)
|
||||
if trimmedInputName == "" {
|
||||
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.inputs keys must be non-empty", artifactName)
|
||||
}
|
||||
if strings.TrimSpace(inputCfg.Source) == "" {
|
||||
source := strings.TrimSpace(inputCfg.Source)
|
||||
if source == "" {
|
||||
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.inputs.%s.source is required", artifactName, inputName)
|
||||
}
|
||||
|
||||
referencedArtifact, err := validateScriptoriumInputSource(artifactName, inputName, source, configuredArtifacts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if referencedArtifact != "" {
|
||||
if _, ok := depSet[referencedArtifact]; !ok {
|
||||
return fmt.Errorf(
|
||||
"pipeline.scriptorium.artifacts.%s.inputs.%s.source %q requires depends_on entry %q",
|
||||
artifactName,
|
||||
inputName,
|
||||
source,
|
||||
referencedArtifact,
|
||||
)
|
||||
}
|
||||
referencedArtifacts[referencedArtifact] = struct{}{}
|
||||
}
|
||||
}
|
||||
for varName, varValue := range artifactCfg.Vars {
|
||||
if strings.TrimSpace(varName) == "" {
|
||||
@@ -359,6 +409,17 @@ func validateScriptorium(cfg *ScriptoriumConfig) error {
|
||||
}
|
||||
}
|
||||
|
||||
for artifactName := range referencedArtifacts {
|
||||
artifactCfg := cfg.Artifacts[artifactName]
|
||||
if strings.TrimSpace(artifactCfg.OutputPath) == "" {
|
||||
return fmt.Errorf("pipeline.scriptorium.artifacts.%s.output_path is required when artifact is referenced", artifactName)
|
||||
}
|
||||
}
|
||||
|
||||
if err := validateEnabledArtifactDependencyCycles(cfg.Artifacts); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -437,6 +498,53 @@ func archiveUploadConfiguredForS3(pipeline *PipelineConfig) bool {
|
||||
|
||||
var windowsAbsPathRE = regexp.MustCompile(`^[A-Za-z]:[\\/].*`)
|
||||
var envVarNameRE = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`)
|
||||
var scriptoriumArtifactKeyRE = regexp.MustCompile(`^[a-z][a-z0-9_]*$`)
|
||||
var narratioArtifactSourceRE = regexp.MustCompile(`^narratio\.artifact\.([a-z][a-z0-9_]*)$`)
|
||||
|
||||
func validateScriptoriumInputSource(artifactName, inputName, source string, configuredArtifacts map[string]struct{}) (string, error) {
|
||||
if isStaticSupportedScriptoriumInputSource(source) {
|
||||
return "", nil
|
||||
}
|
||||
matches := narratioArtifactSourceRE.FindStringSubmatch(source)
|
||||
if len(matches) != 2 {
|
||||
return "", fmt.Errorf(
|
||||
"pipeline.scriptorium.artifacts.%s.inputs.%s.source %q is unsupported",
|
||||
artifactName,
|
||||
inputName,
|
||||
source,
|
||||
)
|
||||
}
|
||||
referenced := matches[1]
|
||||
if _, ok := configuredArtifacts[referenced]; !ok {
|
||||
return "", fmt.Errorf(
|
||||
"pipeline.scriptorium.artifacts.%s.inputs.%s.source %q references unknown artifact %q",
|
||||
artifactName,
|
||||
inputName,
|
||||
source,
|
||||
referenced,
|
||||
)
|
||||
}
|
||||
return referenced, nil
|
||||
}
|
||||
|
||||
func isStaticSupportedScriptoriumInputSource(source string) bool {
|
||||
switch strings.TrimSpace(source) {
|
||||
case "previous_session_artifact":
|
||||
return true
|
||||
case "narratio.transcript.merged":
|
||||
return true
|
||||
case "narratio.transcript.polished":
|
||||
return true
|
||||
case "narratio.transcript.full":
|
||||
return true
|
||||
case "narratio.transcript.trimmed":
|
||||
return true
|
||||
case "narratio.bounds.session":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
func validateEnvVarNameField(fieldName, value string) error {
|
||||
trimmed := strings.TrimSpace(value)
|
||||
@@ -467,6 +575,73 @@ func validateRelativeSafePath(fieldName, value string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func validatePathWithinRoot(fieldName, value, root string) error {
|
||||
normalizedValue := filepath.ToSlash(filepath.Clean(strings.TrimSpace(value)))
|
||||
normalizedRoot := filepath.ToSlash(filepath.Clean(strings.TrimSpace(root)))
|
||||
if normalizedValue == normalizedRoot {
|
||||
return nil
|
||||
}
|
||||
if strings.HasPrefix(normalizedValue, normalizedRoot+"/") {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("%s must be under %s/", fieldName, normalizedRoot)
|
||||
}
|
||||
|
||||
func validateEnabledArtifactDependencyCycles(artifacts map[string]ScriptoriumArtifactConfig) error {
|
||||
if len(artifacts) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
enabled := make(map[string]struct{}, len(artifacts))
|
||||
graph := make(map[string][]string, len(artifacts))
|
||||
for name, cfg := range artifacts {
|
||||
if !cfg.Enabled {
|
||||
continue
|
||||
}
|
||||
enabled[name] = struct{}{}
|
||||
}
|
||||
for name, cfg := range artifacts {
|
||||
if !cfg.Enabled {
|
||||
continue
|
||||
}
|
||||
for _, dep := range cfg.DependsOn {
|
||||
trimmedDep := strings.TrimSpace(dep)
|
||||
if _, ok := enabled[trimmedDep]; ok {
|
||||
graph[name] = append(graph[name], trimmedDep)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
visiting := make(map[string]bool, len(enabled))
|
||||
visited := make(map[string]bool, len(enabled))
|
||||
|
||||
var visit func(node string) error
|
||||
visit = func(node string) error {
|
||||
if visiting[node] {
|
||||
return fmt.Errorf("pipeline.scriptorium.artifacts enabled dependencies must not contain cycles")
|
||||
}
|
||||
if visited[node] {
|
||||
return nil
|
||||
}
|
||||
visiting[node] = true
|
||||
for _, dep := range graph[node] {
|
||||
if err := visit(dep); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
visiting[node] = false
|
||||
visited[node] = true
|
||||
return nil
|
||||
}
|
||||
|
||||
for node := range enabled {
|
||||
if err := visit(node); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateDuration(fieldName, value string) error {
|
||||
trimmed := strings.TrimSpace(value)
|
||||
if trimmed == "" {
|
||||
|
||||
@@ -28,9 +28,12 @@ type InputRecord struct {
|
||||
// ArtifactRecord captures one produced artifact and optional remote metadata.
|
||||
type ArtifactRecord struct {
|
||||
Kind string `json:"kind"`
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
LocalPath string `json:"local_path"`
|
||||
RemoteKey string `json:"remote_key,omitempty"`
|
||||
Checksum string `json:"checksum,omitempty"`
|
||||
// ProducerRunID identifies the run that produced this durable artifact.
|
||||
ProducerRunID string `json:"producer_run_id,omitempty"`
|
||||
RemoteKey string `json:"remote_key,omitempty"`
|
||||
Checksum string `json:"checksum,omitempty"`
|
||||
}
|
||||
|
||||
// StageRecord tracks lifecycle and provenance for one pipeline stage.
|
||||
@@ -120,6 +123,15 @@ func (m *Manifest) MarkStageSkipped(name string, at time.Time, reason string) {
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
// MarkStageStale marks a stage as stale so it is not skipped as idempotently complete.
|
||||
func (m *Manifest) MarkStageStale(name string, at time.Time, reason string) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Status = StatusStale
|
||||
s.Error = &ErrorRecord{Message: strings.TrimSpace(reason), Code: "stale", At: timePtr(at)}
|
||||
s.UpdatedAt = at
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *Manifest) ensureStage(name string, at time.Time) *StageRecord {
|
||||
if m.Stages == nil {
|
||||
m.Stages = map[string]*StageRecord{}
|
||||
|
||||
164
internal/manifest/run_manifest.go
Normal file
164
internal/manifest/run_manifest.go
Normal file
@@ -0,0 +1,164 @@
|
||||
package manifest
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
type RunManifestStatus string
|
||||
|
||||
const (
|
||||
RunManifestStatusRunning RunManifestStatus = "running"
|
||||
RunManifestStatusSucceeded RunManifestStatus = "succeeded"
|
||||
RunManifestStatusFailed RunManifestStatus = "failed"
|
||||
)
|
||||
|
||||
type RunStageAction string
|
||||
|
||||
const (
|
||||
RunStageActionRun RunStageAction = "run"
|
||||
RunStageActionSkip RunStageAction = "skip"
|
||||
)
|
||||
|
||||
// RunStageRecord tracks lifecycle and provenance for one stage within a single invocation.
|
||||
type RunStageRecord struct {
|
||||
Name string `json:"name"`
|
||||
Action RunStageAction `json:"action"`
|
||||
Status StageStatus `json:"status"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||
CompletedAt *time.Time `json:"completed_at,omitempty"`
|
||||
Outputs []ArtifactRecord `json:"outputs,omitempty"`
|
||||
Logs []string `json:"logs,omitempty"`
|
||||
GeneratedConfigs []string `json:"generated_configs,omitempty"`
|
||||
Error *ErrorRecord `json:"error,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
// RunManifest is the invocation-scoped execution record under runs/{run_id}/manifest.json.
|
||||
type RunManifest struct {
|
||||
SessionID string `json:"session_id"`
|
||||
Campaign string `json:"campaign,omitempty"`
|
||||
RunID string `json:"run_id"`
|
||||
Force bool `json:"force"`
|
||||
RequestedStages []string `json:"requested_stages,omitempty"`
|
||||
SessionManifestPath string `json:"session_manifest_path,omitempty"`
|
||||
LocalWorkDir string `json:"local_workdir,omitempty"`
|
||||
LocalSpoolDir string `json:"local_spool_dir,omitempty"`
|
||||
S3Bucket string `json:"s3_bucket,omitempty"`
|
||||
S3SessionPrefix string `json:"s3_session_prefix,omitempty"`
|
||||
S3RunPrefix string `json:"s3_run_prefix,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||
CompletedAt *time.Time `json:"completed_at,omitempty"`
|
||||
Status RunManifestStatus `json:"status"`
|
||||
LastError *ErrorRecord `json:"last_error,omitempty"`
|
||||
Stages map[string]*RunStageRecord `json:"stages"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
// NewRun constructs a new run manifest with deterministic timestamps.
|
||||
func NewRun(sessionID, campaign, runID string, force bool, requestedStages []string, now time.Time) *RunManifest {
|
||||
return &RunManifest{
|
||||
SessionID: strings.TrimSpace(sessionID),
|
||||
Campaign: strings.TrimSpace(campaign),
|
||||
RunID: strings.TrimSpace(runID),
|
||||
Force: force,
|
||||
RequestedStages: append([]string(nil), requestedStages...),
|
||||
CreatedAt: now,
|
||||
UpdatedAt: now,
|
||||
StartedAt: timePtr(now),
|
||||
Status: RunManifestStatusRunning,
|
||||
Stages: map[string]*RunStageRecord{},
|
||||
}
|
||||
}
|
||||
|
||||
func (m *RunManifest) SetStageAction(name string, action RunStageAction, at time.Time) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Action = action
|
||||
s.UpdatedAt = at
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkStageRunning(name string, at time.Time) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Status = StatusRunning
|
||||
s.StartedAt = timePtr(at)
|
||||
s.CompletedAt = nil
|
||||
s.Error = nil
|
||||
s.UpdatedAt = at
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkStageSucceeded(name string, at time.Time, outputs []ArtifactRecord) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Status = StatusSucceeded
|
||||
s.CompletedAt = timePtr(at)
|
||||
s.Error = nil
|
||||
s.Outputs = append([]ArtifactRecord(nil), outputs...)
|
||||
s.UpdatedAt = at
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkStageFailed(name string, at time.Time, message string) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Status = StatusFailed
|
||||
s.CompletedAt = timePtr(at)
|
||||
s.Error = &ErrorRecord{Message: strings.TrimSpace(message), At: timePtr(at)}
|
||||
s.UpdatedAt = at
|
||||
m.LastError = &ErrorRecord{Message: strings.TrimSpace(message), At: timePtr(at)}
|
||||
m.UpdatedAt = at
|
||||
m.Status = RunManifestStatusFailed
|
||||
m.CompletedAt = timePtr(at)
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkStageSkipped(name string, at time.Time, reason string) {
|
||||
s := m.ensureStage(name, at)
|
||||
s.Status = StatusSkipped
|
||||
s.CompletedAt = timePtr(at)
|
||||
s.Error = &ErrorRecord{Message: strings.TrimSpace(reason), Code: "skipped", At: timePtr(at)}
|
||||
s.UpdatedAt = at
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkSucceeded(at time.Time) {
|
||||
m.Status = RunManifestStatusSucceeded
|
||||
m.CompletedAt = timePtr(at)
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) MarkFailed(at time.Time, message string) {
|
||||
m.Status = RunManifestStatusFailed
|
||||
m.CompletedAt = timePtr(at)
|
||||
m.LastError = &ErrorRecord{Message: strings.TrimSpace(message), At: timePtr(at)}
|
||||
m.UpdatedAt = at
|
||||
}
|
||||
|
||||
func (m *RunManifest) ensureStage(name string, at time.Time) *RunStageRecord {
|
||||
if m.Stages == nil {
|
||||
m.Stages = map[string]*RunStageRecord{}
|
||||
}
|
||||
|
||||
stageName := strings.TrimSpace(name)
|
||||
s, ok := m.Stages[stageName]
|
||||
if !ok || s == nil {
|
||||
s = &RunStageRecord{
|
||||
Name: stageName,
|
||||
Action: RunStageActionRun,
|
||||
Status: StatusPending,
|
||||
CreatedAt: at,
|
||||
UpdatedAt: at,
|
||||
}
|
||||
m.Stages[stageName] = s
|
||||
}
|
||||
if s.Name == "" {
|
||||
s.Name = stageName
|
||||
}
|
||||
if s.CreatedAt.IsZero() {
|
||||
s.CreatedAt = at
|
||||
}
|
||||
|
||||
return s
|
||||
}
|
||||
50
internal/manifest/run_manifest_test.go
Normal file
50
internal/manifest/run_manifest_test.go
Normal file
@@ -0,0 +1,50 @@
|
||||
package manifest
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestRunManifestStageMarkHelpers(t *testing.T) {
|
||||
rm := NewRun("2026-05-03", "forsaken", "20260517T000000Z-abcdef12", false, []string{"prepare"}, time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||
if rm.Status != RunManifestStatusRunning {
|
||||
t.Fatalf("status = %q, want %q", rm.Status, RunManifestStatusRunning)
|
||||
}
|
||||
|
||||
runningAt := time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC)
|
||||
rm.SetStageAction("prepare", RunStageActionRun, runningAt)
|
||||
rm.MarkStageRunning("prepare", runningAt)
|
||||
rm.MarkStageSucceeded("prepare", runningAt.Add(30*time.Second), []ArtifactRecord{
|
||||
{Kind: "input", LocalPath: "inputs/session.yml"},
|
||||
})
|
||||
|
||||
stage := rm.Stages["prepare"]
|
||||
if stage == nil {
|
||||
t.Fatal("prepare stage missing")
|
||||
}
|
||||
if stage.Action != RunStageActionRun {
|
||||
t.Fatalf("action = %q, want %q", stage.Action, RunStageActionRun)
|
||||
}
|
||||
if stage.Status != StatusSucceeded {
|
||||
t.Fatalf("status = %q, want %q", stage.Status, StatusSucceeded)
|
||||
}
|
||||
|
||||
skippedAt := runningAt.Add(1 * time.Minute)
|
||||
rm.SetStageAction("notify", RunStageActionSkip, skippedAt)
|
||||
rm.MarkStageSkipped("notify", skippedAt, "already_succeeded")
|
||||
skipped := rm.Stages["notify"]
|
||||
if skipped == nil {
|
||||
t.Fatal("notify stage missing")
|
||||
}
|
||||
if skipped.Action != RunStageActionSkip {
|
||||
t.Fatalf("action = %q, want %q", skipped.Action, RunStageActionSkip)
|
||||
}
|
||||
if skipped.Status != StatusSkipped {
|
||||
t.Fatalf("status = %q, want %q", skipped.Status, StatusSkipped)
|
||||
}
|
||||
|
||||
rm.MarkSucceeded(skippedAt.Add(10 * time.Second))
|
||||
if rm.Status != RunManifestStatusSucceeded {
|
||||
t.Fatalf("status = %q, want %q", rm.Status, RunManifestStatusSucceeded)
|
||||
}
|
||||
}
|
||||
@@ -129,6 +129,89 @@ func (s *LocalStore) Save(ctx context.Context, path string, m *Manifest) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// CreateRun returns a new in-memory run manifest for one invocation.
|
||||
func (s *LocalStore) CreateRun(
|
||||
ctx context.Context,
|
||||
sessionID, campaign, runID string,
|
||||
force bool,
|
||||
requestedStages []string,
|
||||
) (*RunManifest, error) {
|
||||
if err := checkContext(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if strings.TrimSpace(sessionID) == "" {
|
||||
return nil, fmt.Errorf("create run manifest: session_id is required")
|
||||
}
|
||||
if strings.TrimSpace(runID) == "" {
|
||||
return nil, fmt.Errorf("create run manifest: run_id is required")
|
||||
}
|
||||
|
||||
now := time.Now().UTC()
|
||||
return NewRun(sessionID, campaign, runID, force, requestedStages, now), nil
|
||||
}
|
||||
|
||||
// LoadRun reads and validates a local JSON run manifest from path.
|
||||
func (s *LocalStore) LoadRun(ctx context.Context, path string) (*RunManifest, error) {
|
||||
if err := checkContext(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if strings.TrimSpace(path) == "" {
|
||||
return nil, fmt.Errorf("load run manifest: path is required")
|
||||
}
|
||||
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("load run manifest %q: %w", path, err)
|
||||
}
|
||||
|
||||
var m RunManifest
|
||||
if err := json.Unmarshal(data, &m); err != nil {
|
||||
return nil, fmt.Errorf("decode run manifest %q: %w", path, err)
|
||||
}
|
||||
|
||||
if err := validateLoadedRunManifest(&m); err != nil {
|
||||
return nil, fmt.Errorf("run manifest %q invalid: %w", path, err)
|
||||
}
|
||||
normalizeRunManifest(&m)
|
||||
|
||||
return &m, nil
|
||||
}
|
||||
|
||||
// SaveRun writes the run manifest to path atomically via temp file + rename.
|
||||
func (s *LocalStore) SaveRun(ctx context.Context, path string, m *RunManifest) error {
|
||||
if err := checkContext(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(path) == "" {
|
||||
return fmt.Errorf("save run manifest: path is required")
|
||||
}
|
||||
if m == nil {
|
||||
return fmt.Errorf("save run manifest: manifest is nil")
|
||||
}
|
||||
if strings.TrimSpace(m.SessionID) == "" {
|
||||
return fmt.Errorf("save run manifest: session_id is required")
|
||||
}
|
||||
if strings.TrimSpace(m.RunID) == "" {
|
||||
return fmt.Errorf("save run manifest: run_id is required")
|
||||
}
|
||||
if m.CreatedAt.IsZero() {
|
||||
return fmt.Errorf("save run manifest: created_at is required")
|
||||
}
|
||||
|
||||
m.UpdatedAt = time.Now().UTC()
|
||||
if m.Stages == nil {
|
||||
m.Stages = map[string]*RunStageRecord{}
|
||||
}
|
||||
|
||||
data, err := json.MarshalIndent(m, "", " ")
|
||||
if err != nil {
|
||||
return fmt.Errorf("save run manifest: marshal: %w", err)
|
||||
}
|
||||
data = append(data, '\n')
|
||||
|
||||
return writeJSONAtomically(ctx, path, ".run-manifest.json.tmp-*", data)
|
||||
}
|
||||
|
||||
func validateLoadedManifest(m *Manifest) error {
|
||||
if m == nil {
|
||||
return fmt.Errorf("manifest is nil")
|
||||
@@ -161,6 +244,88 @@ func normalizeManifest(m *Manifest) {
|
||||
}
|
||||
}
|
||||
|
||||
func validateLoadedRunManifest(m *RunManifest) error {
|
||||
if m == nil {
|
||||
return fmt.Errorf("manifest is nil")
|
||||
}
|
||||
if strings.TrimSpace(m.SessionID) == "" {
|
||||
return fmt.Errorf("session_id is required")
|
||||
}
|
||||
if strings.TrimSpace(m.RunID) == "" {
|
||||
return fmt.Errorf("run_id is required")
|
||||
}
|
||||
if m.CreatedAt.IsZero() {
|
||||
return fmt.Errorf("created_at is required")
|
||||
}
|
||||
if m.UpdatedAt.IsZero() {
|
||||
return fmt.Errorf("updated_at is required")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func normalizeRunManifest(m *RunManifest) {
|
||||
if m.Stages == nil {
|
||||
m.Stages = map[string]*RunStageRecord{}
|
||||
}
|
||||
for name, stage := range m.Stages {
|
||||
if stage == nil {
|
||||
stage = &RunStageRecord{
|
||||
Name: name,
|
||||
Action: RunStageActionRun,
|
||||
Status: StatusPending,
|
||||
CreatedAt: m.CreatedAt,
|
||||
UpdatedAt: m.UpdatedAt,
|
||||
}
|
||||
m.Stages[name] = stage
|
||||
}
|
||||
if stage.Name == "" {
|
||||
stage.Name = name
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func writeJSONAtomically(ctx context.Context, path, tempPattern string, data []byte) error {
|
||||
dir := filepath.Dir(path)
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return fmt.Errorf("create directory %q: %w", dir, err)
|
||||
}
|
||||
|
||||
tmp, err := os.CreateTemp(dir, tempPattern)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create temp file: %w", err)
|
||||
}
|
||||
tmpName := tmp.Name()
|
||||
removeTmp := true
|
||||
defer func() {
|
||||
if removeTmp {
|
||||
_ = os.Remove(tmpName)
|
||||
}
|
||||
}()
|
||||
|
||||
if _, err := tmp.Write(data); err != nil {
|
||||
_ = tmp.Close()
|
||||
return fmt.Errorf("write temp file: %w", err)
|
||||
}
|
||||
if err := tmp.Sync(); err != nil {
|
||||
_ = tmp.Close()
|
||||
return fmt.Errorf("sync temp file: %w", err)
|
||||
}
|
||||
if err := tmp.Close(); err != nil {
|
||||
return fmt.Errorf("close temp file: %w", err)
|
||||
}
|
||||
if err := checkContext(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := os.Rename(tmpName, path); err != nil {
|
||||
return fmt.Errorf("rename temp file: %w", err)
|
||||
}
|
||||
removeTmp = false
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func checkContext(ctx context.Context) error {
|
||||
if ctx == nil {
|
||||
return nil
|
||||
|
||||
@@ -150,3 +150,73 @@ func TestLoadRejectsInvalidManifest(t *testing.T) {
|
||||
t.Fatalf("error = %q, want session_id validation", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestLocalStoreCreateSaveLoadRunManifestRoundTrip(t *testing.T) {
|
||||
store := &LocalStore{}
|
||||
ctx := context.Background()
|
||||
|
||||
run, err := store.CreateRun(
|
||||
ctx,
|
||||
"2026-05-03",
|
||||
"forsaken",
|
||||
"20260517T000000Z-abcdef12",
|
||||
true,
|
||||
[]string{"prepare", "transcribe"},
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("CreateRun() error = %v", err)
|
||||
}
|
||||
run.SessionManifestPath = "/var/lib/narratio/work/forsaken/2026-05-03/manifest.json"
|
||||
run.MarkStageRunning("prepare", time.Date(2026, 5, 3, 12, 1, 0, 0, time.UTC))
|
||||
run.MarkStageSucceeded("prepare", time.Date(2026, 5, 3, 12, 2, 0, 0, time.UTC), []ArtifactRecord{
|
||||
{Kind: "input", LocalPath: "inputs/session.yml"},
|
||||
})
|
||||
run.MarkSucceeded(time.Date(2026, 5, 3, 12, 3, 0, 0, time.UTC))
|
||||
|
||||
path := filepath.Join(t.TempDir(), "run-manifest.json")
|
||||
if err := store.SaveRun(ctx, path, run); err != nil {
|
||||
t.Fatalf("SaveRun() error = %v", err)
|
||||
}
|
||||
|
||||
loaded, err := store.LoadRun(ctx, path)
|
||||
if err != nil {
|
||||
t.Fatalf("LoadRun() error = %v", err)
|
||||
}
|
||||
|
||||
if loaded.SessionID != "2026-05-03" {
|
||||
t.Fatalf("SessionID = %q, want %q", loaded.SessionID, "2026-05-03")
|
||||
}
|
||||
if loaded.Campaign != "forsaken" {
|
||||
t.Fatalf("Campaign = %q, want %q", loaded.Campaign, "forsaken")
|
||||
}
|
||||
if loaded.RunID != "20260517T000000Z-abcdef12" {
|
||||
t.Fatalf("RunID = %q, want %q", loaded.RunID, "20260517T000000Z-abcdef12")
|
||||
}
|
||||
if loaded.Status != RunManifestStatusSucceeded {
|
||||
t.Fatalf("Status = %q, want %q", loaded.Status, RunManifestStatusSucceeded)
|
||||
}
|
||||
if loaded.Stages["prepare"] == nil || loaded.Stages["prepare"].Status != StatusSucceeded {
|
||||
t.Fatalf("prepare stage = %#v, want succeeded", loaded.Stages["prepare"])
|
||||
}
|
||||
if loaded.Stages["prepare"].Action != RunStageActionRun {
|
||||
t.Fatalf("prepare action = %q, want %q", loaded.Stages["prepare"].Action, RunStageActionRun)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadRunRejectsInvalidManifest(t *testing.T) {
|
||||
store := &LocalStore{}
|
||||
ctx := context.Background()
|
||||
|
||||
path := filepath.Join(t.TempDir(), "run-manifest.json")
|
||||
if err := os.WriteFile(path, []byte(`{"session_id":"2026-05-03"}`), 0o644); err != nil {
|
||||
t.Fatalf("WriteFile() error = %v", err)
|
||||
}
|
||||
|
||||
_, err := store.LoadRun(ctx, path)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "run_id is required") {
|
||||
t.Fatalf("error = %q, want run_id validation", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@ package stage
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -27,12 +28,23 @@ func (analyzeStage) Declares() IODecl {
|
||||
{Kind: "transcript_normalized", Category: "transcripts", RelativePath: "transcripts/normalized.json"},
|
||||
{Kind: "transcript_trimmed", Category: "transcripts", RelativePath: "transcripts/trimmed.json"},
|
||||
},
|
||||
Outputs: []artifacts.Ref{
|
||||
{Kind: "session_recap", Category: "artifacts", RelativePath: "artifacts/session_recap.md"},
|
||||
},
|
||||
Outputs: nil,
|
||||
}
|
||||
}
|
||||
|
||||
type analyzeArtifactExecutionPlan struct {
|
||||
Name string
|
||||
Cfg config.ScriptoriumArtifactConfig
|
||||
}
|
||||
|
||||
type analyzeArtifactExecutionResult struct {
|
||||
Output artifacts.Ref
|
||||
Logs []string
|
||||
GeneratedConfigs []string
|
||||
Metadata map[string]any
|
||||
ReusedArtifacts []map[string]any
|
||||
}
|
||||
|
||||
func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*StageResult, error) {
|
||||
if env == nil || env.Config == nil {
|
||||
return nil, fmt.Errorf("analyze: stage environment config is required")
|
||||
@@ -58,96 +70,314 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
return nil, fmt.Errorf("analyze: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "analyze")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve run-stage layout: %w", err)
|
||||
}
|
||||
if env.Config.Pipeline.Scriptorium == nil {
|
||||
return &StageResult{
|
||||
Metadata: map[string]any{
|
||||
"stage": "analyze",
|
||||
"skipped": true,
|
||||
"reason": "pipeline.scriptorium is not configured",
|
||||
},
|
||||
}, nil
|
||||
return &StageResult{Metadata: map[string]any{
|
||||
"stage": "analyze",
|
||||
"skipped": true,
|
||||
"reason": "pipeline.scriptorium is not configured",
|
||||
}}, nil
|
||||
}
|
||||
|
||||
artifactName, artifactCfg, skipReason, err := selectAnalyzeArtifact(env.Config.Pipeline.Scriptorium)
|
||||
runtimeCatalog, err := buildAnalyzeRuntimeArtifactCatalog(paths, env.Config.Pipeline.Scriptorium, env.SelectedAnalyzeArtifacts)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: build runtime artifact catalog: %w", err)
|
||||
}
|
||||
|
||||
plans, skipReason, err := buildAnalyzeExecutionPlans(env.Config.Pipeline.Scriptorium, runtimeCatalog)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: %w", err)
|
||||
}
|
||||
if skipReason != "" {
|
||||
return &StageResult{
|
||||
Metadata: map[string]any{
|
||||
"stage": "analyze",
|
||||
"skipped": true,
|
||||
"reason": skipReason,
|
||||
},
|
||||
}, nil
|
||||
return &StageResult{Metadata: map[string]any{
|
||||
"stage": "analyze",
|
||||
"skipped": true,
|
||||
"reason": skipReason,
|
||||
}}, nil
|
||||
}
|
||||
|
||||
processedTranscriptPath, processedSource, err := discoverProcessedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve processed transcript: %w", err)
|
||||
}
|
||||
normalizedTranscriptPath, normalizedSource, err := discoverNormalizedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve normalized transcript: %w", err)
|
||||
}
|
||||
trimmedTranscriptPath, trimmedSource, err := discoverTrimmedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve trimmed transcript: %w", err)
|
||||
transcriptRefs := discoverAnalyzeTranscriptRefs(m, paths)
|
||||
sessionDir := filepath.Dir(strings.TrimSpace(env.Config.SessionPath))
|
||||
|
||||
outputs := make([]artifacts.Ref, 0, len(plans))
|
||||
logs := []string{}
|
||||
generatedConfigs := []string{}
|
||||
artifactMetadata := make([]map[string]any, 0, len(plans))
|
||||
reusedArtifacts := []map[string]any{}
|
||||
reusedSeen := map[string]struct{}{}
|
||||
|
||||
for _, plan := range plans {
|
||||
artifactResult, err := executeAnalyzeArtifact(
|
||||
ctx,
|
||||
env,
|
||||
m,
|
||||
paths,
|
||||
runLayout,
|
||||
sessionID,
|
||||
sessionDir,
|
||||
plan,
|
||||
transcriptRefs,
|
||||
runtimeCatalog,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
outputs = append(outputs, artifactResult.Output)
|
||||
logs = append(logs, artifactResult.Logs...)
|
||||
generatedConfigs = append(generatedConfigs, artifactResult.GeneratedConfigs...)
|
||||
artifactMetadata = append(artifactMetadata, artifactResult.Metadata)
|
||||
for _, reused := range artifactResult.ReusedArtifacts {
|
||||
sourceID, _ := reused["source_id"].(string)
|
||||
path, _ := reused["path"].(string)
|
||||
key := sourceID + "|" + path
|
||||
if _, exists := reusedSeen[key]; exists {
|
||||
continue
|
||||
}
|
||||
reusedSeen[key] = struct{}{}
|
||||
reusedArtifacts = append(reusedArtifacts, reused)
|
||||
}
|
||||
|
||||
sourceID, ok := runtimeCatalog.SourceIDForConfiguredKey(plan.Name)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("analyze: source id not found for artifact %q", plan.Name)
|
||||
}
|
||||
if err := runtimeCatalog.MarkAvailableGenerated(sourceID, artifactResult.Output.AbsolutePath); err != nil {
|
||||
return nil, fmt.Errorf("analyze: mark generated artifact %q available: %w", sourceID, err)
|
||||
}
|
||||
}
|
||||
|
||||
transcriptInputs := analyzeTranscriptInputs{
|
||||
ProcessedPath: processedTranscriptPath,
|
||||
ProcessedSource: processedSource,
|
||||
NormalizedPath: normalizedTranscriptPath,
|
||||
NormalizedSource: normalizedSource,
|
||||
TrimmedPath: trimmedTranscriptPath,
|
||||
TrimmedSource: trimmedSource,
|
||||
metadata := map[string]any{
|
||||
"stage": "analyze",
|
||||
"selected_artifacts": extractPlanNames(plans),
|
||||
"generated_artifacts": artifactMetadata,
|
||||
"reused_artifacts": reusedArtifacts,
|
||||
"artifact_count": len(artifactMetadata),
|
||||
"reused_artifact_count": len(reusedArtifacts),
|
||||
}
|
||||
if len(artifactMetadata) == 1 {
|
||||
for k, v := range artifactMetadata[0] {
|
||||
metadata[k] = v
|
||||
}
|
||||
}
|
||||
|
||||
return &StageResult{
|
||||
Outputs: outputs,
|
||||
Logs: dedupeAndSortPaths(logs),
|
||||
GeneratedConfigs: dedupeAndSortPaths(generatedConfigs),
|
||||
Metadata: metadata,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func buildAnalyzeExecutionPlans(
|
||||
scriptoriumCfg *config.ScriptoriumConfig,
|
||||
catalog *artifacts.ArtifactCatalog,
|
||||
) ([]analyzeArtifactExecutionPlan, string, error) {
|
||||
if scriptoriumCfg == nil {
|
||||
return nil, "pipeline.scriptorium is not configured", nil
|
||||
}
|
||||
if len(scriptoriumCfg.Artifacts) == 0 {
|
||||
return nil, "no scriptorium artifacts configured", nil
|
||||
}
|
||||
|
||||
entries := catalog.ListConfigured()
|
||||
selected := make([]string, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
if entry.Executable {
|
||||
selected = append(selected, entry.ConfiguredKey)
|
||||
}
|
||||
}
|
||||
if len(selected) == 0 {
|
||||
return nil, "no selected scriptorium artifacts to execute", nil
|
||||
}
|
||||
|
||||
ordered, err := orderSelectedScriptoriumArtifacts(scriptoriumCfg.Artifacts, selected, catalog)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
|
||||
plans := make([]analyzeArtifactExecutionPlan, 0, len(ordered))
|
||||
for _, name := range ordered {
|
||||
artifactCfg, ok := scriptoriumCfg.Artifacts[name]
|
||||
if !ok {
|
||||
return nil, "", fmt.Errorf("selected artifact %q is not configured", name)
|
||||
}
|
||||
plans = append(plans, analyzeArtifactExecutionPlan{Name: name, Cfg: artifactCfg})
|
||||
}
|
||||
return plans, "", nil
|
||||
}
|
||||
|
||||
func orderSelectedScriptoriumArtifacts(
|
||||
artifactsCfg map[string]config.ScriptoriumArtifactConfig,
|
||||
selected []string,
|
||||
catalog *artifacts.ArtifactCatalog,
|
||||
) ([]string, error) {
|
||||
selectedSet := map[string]struct{}{}
|
||||
for _, key := range selected {
|
||||
trimmed := strings.TrimSpace(key)
|
||||
if trimmed == "" {
|
||||
return nil, fmt.Errorf("selected artifact key must be non-empty")
|
||||
}
|
||||
selectedSet[trimmed] = struct{}{}
|
||||
}
|
||||
|
||||
for selectedKey := range selectedSet {
|
||||
cfg, ok := artifactsCfg[selectedKey]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("selected artifact %q is not configured", selectedKey)
|
||||
}
|
||||
for _, dep := range cfg.DependsOn {
|
||||
trimmedDep := strings.TrimSpace(dep)
|
||||
if trimmedDep == "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := selectedSet[trimmedDep]; ok {
|
||||
continue
|
||||
}
|
||||
sourceID, ok := catalog.SourceIDForConfiguredKey(trimmedDep)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("artifact %q depends on unknown configured artifact %q", selectedKey, trimmedDep)
|
||||
}
|
||||
entry, ok := catalog.Lookup(sourceID)
|
||||
if !ok || !entry.Available {
|
||||
return nil, fmt.Errorf("artifact %q depends on %q, but %q is unavailable", selectedKey, trimmedDep, sourceID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
indegree := map[string]int{}
|
||||
edges := map[string][]string{}
|
||||
for key := range selectedSet {
|
||||
indegree[key] = 0
|
||||
}
|
||||
for key := range selectedSet {
|
||||
cfg := artifactsCfg[key]
|
||||
for _, dep := range cfg.DependsOn {
|
||||
trimmedDep := strings.TrimSpace(dep)
|
||||
if _, ok := selectedSet[trimmedDep]; !ok {
|
||||
continue
|
||||
}
|
||||
edges[trimmedDep] = append(edges[trimmedDep], key)
|
||||
indegree[key]++
|
||||
}
|
||||
}
|
||||
|
||||
for key := range edges {
|
||||
sort.Strings(edges[key])
|
||||
}
|
||||
|
||||
ready := make([]string, 0, len(indegree))
|
||||
for key, degree := range indegree {
|
||||
if degree == 0 {
|
||||
ready = append(ready, key)
|
||||
}
|
||||
}
|
||||
sort.Strings(ready)
|
||||
|
||||
order := make([]string, 0, len(selectedSet))
|
||||
for len(ready) > 0 {
|
||||
node := ready[0]
|
||||
ready = ready[1:]
|
||||
order = append(order, node)
|
||||
for _, dep := range edges[node] {
|
||||
indegree[dep]--
|
||||
if indegree[dep] == 0 {
|
||||
ready = append(ready, dep)
|
||||
sort.Strings(ready)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if len(order) != len(selectedSet) {
|
||||
return nil, fmt.Errorf("selected scriptorium artifacts contain a dependency cycle")
|
||||
}
|
||||
return order, nil
|
||||
}
|
||||
|
||||
func executeAnalyzeArtifact(
|
||||
ctx context.Context,
|
||||
env *Env,
|
||||
m *manifest.Manifest,
|
||||
paths artifacts.SessionPaths,
|
||||
runLayout runStageLayout,
|
||||
sessionID string,
|
||||
sessionDir string,
|
||||
plan analyzeArtifactExecutionPlan,
|
||||
transcriptRefs analyzeTranscriptInputs,
|
||||
runtimeCatalog *artifacts.ArtifactCatalog,
|
||||
) (*analyzeArtifactExecutionResult, error) {
|
||||
artifactName := plan.Name
|
||||
artifactCfg := plan.Cfg
|
||||
|
||||
inputPaths := map[string]string{}
|
||||
omittedOptionalInputs := []string{}
|
||||
sessionDir := filepath.Dir(strings.TrimSpace(env.Config.SessionPath))
|
||||
reusedArtifacts := []map[string]any{}
|
||||
|
||||
inputNames := sortedScriptoriumInputNames(artifactCfg.Inputs)
|
||||
for _, inputName := range inputNames {
|
||||
inputCfg := artifactCfg.Inputs[inputName]
|
||||
resolvedPath, resolved, resolveErr := resolveScriptoriumInput(inputName, inputCfg, transcriptInputs, paths, sessionDir)
|
||||
resolvedPath, resolved, resolvedArtifact, resolveErr := resolveScriptoriumInput(inputName, inputCfg, m, paths, sessionDir, runtimeCatalog)
|
||||
if resolveErr != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve input %q: %w", inputName, resolveErr)
|
||||
return nil, fmt.Errorf("analyze: resolve input %q for artifact %q: %w", inputName, artifactName, resolveErr)
|
||||
}
|
||||
if !resolved {
|
||||
if inputCfg.Required {
|
||||
return nil, fmt.Errorf("analyze: required input %q could not be resolved", inputName)
|
||||
return nil, fmt.Errorf("analyze: required input %q for artifact %q could not be resolved", inputName, artifactName)
|
||||
}
|
||||
omittedOptionalInputs = append(omittedOptionalInputs, inputName)
|
||||
continue
|
||||
}
|
||||
inputPaths[inputName] = resolvedPath
|
||||
if resolvedArtifact != nil && resolvedArtifact.Provenance == artifacts.ArtifactProvenanceDisabledFromDisk {
|
||||
reusedArtifacts = append(reusedArtifacts, map[string]any{
|
||||
"name": configuredArtifactNameFromSourceID(resolvedArtifact.ID),
|
||||
"source_id": resolvedArtifact.ID,
|
||||
"path": resolvedArtifact.Path,
|
||||
"provenance": resolvedArtifact.Provenance,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
vars, err := buildScriptoriumVars(artifactCfg.Vars, env.Config.Session)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve vars: %w", err)
|
||||
return nil, fmt.Errorf("analyze: resolve vars for artifact %q: %w", artifactName, err)
|
||||
}
|
||||
|
||||
outputPath, err := resolveScriptoriumOutputPath(paths, artifactCfg.OutputPath)
|
||||
canonicalOutputPath, err := resolveScriptoriumOutputPath(paths, artifactCfg.OutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve output path: %w", err)
|
||||
return nil, fmt.Errorf("analyze: resolve output path for artifact %q: %w", artifactName, err)
|
||||
}
|
||||
outputPath, err := runLocalPathForCanonical(runLayout, paths, canonicalOutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve run-local output path for artifact %q: %w", artifactName, err)
|
||||
}
|
||||
|
||||
stdoutLogPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".stdout.log")
|
||||
stderrLogPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".stderr.log")
|
||||
generatedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium."+artifactName+".generated.yml")
|
||||
if runLayout.Enabled {
|
||||
stdoutLogPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".stdout.log")
|
||||
stderrLogPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".stderr.log")
|
||||
generatedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium."+artifactName+".generated.yml")
|
||||
}
|
||||
|
||||
timeout, err := resolveScriptoriumTimeout(env.Config.Pipeline.Scriptorium.Timeout, artifactCfg.Timeout)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: resolve timeout: %w", err)
|
||||
return nil, fmt.Errorf("analyze: resolve timeout for artifact %q: %w", artifactName, err)
|
||||
}
|
||||
|
||||
logPaths := []string{}
|
||||
generatedConfigs := []string{}
|
||||
meta := map[string]any{
|
||||
"stage": "analyze",
|
||||
"name": artifactName,
|
||||
"artifact_name": artifactName,
|
||||
"source_id": artifacts.ConfiguredArtifactSourceID(artifactName),
|
||||
"output_kind": "scriptorium_artifact",
|
||||
"prompt_id": artifactCfg.PromptID,
|
||||
"profile_id": artifactCfg.ProfileID,
|
||||
"binary": env.Config.Pipeline.Scriptorium.Binary,
|
||||
@@ -157,20 +387,28 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
"omitted_optional_inputs": omittedOptionalInputs,
|
||||
"vars": vars,
|
||||
"timeout": timeout.String(),
|
||||
"processed_transcript_path": processedTranscriptPath,
|
||||
"processed_transcript_source": processedSource,
|
||||
"normalized_transcript_path": normalizedTranscriptPath,
|
||||
"normalized_transcript_source": normalizedSource,
|
||||
"trimmed_transcript_path": trimmedTranscriptPath,
|
||||
"trimmed_transcript_source": trimmedSource,
|
||||
"processed_transcript_path": transcriptRefs.ProcessedPath,
|
||||
"processed_transcript_source": transcriptRefs.ProcessedSource,
|
||||
"normalized_transcript_path": transcriptRefs.NormalizedPath,
|
||||
"normalized_transcript_source": transcriptRefs.NormalizedSource,
|
||||
"trimmed_transcript_path": transcriptRefs.TrimmedPath,
|
||||
"trimmed_transcript_source": transcriptRefs.TrimmedSource,
|
||||
"render_debug_enabled": resolveRenderDebugEnabled(env.Config.Pipeline.Scriptorium.RenderDebug, artifactCfg.RenderDebug),
|
||||
}
|
||||
|
||||
if meta["render_debug_enabled"] == true {
|
||||
renderOutputPath := filepath.Join(paths.ArtifactsDir, artifactName+".render.json")
|
||||
if runLayout.Enabled {
|
||||
renderOutputPath = filepath.Join(runLayout.ReportsDir, artifactName+".render.json")
|
||||
}
|
||||
renderStdoutPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".render.stdout.log")
|
||||
renderStderrPath := filepath.Join(paths.LogsDir, "scriptorium."+artifactName+".render.stderr.log")
|
||||
renderGeneratedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium."+artifactName+".render.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
renderStdoutPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".render.stdout.log")
|
||||
renderStderrPath = filepath.Join(runLayout.LogsDir, "scriptorium."+artifactName+".render.stderr.log")
|
||||
renderGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium."+artifactName+".render.generated.yml")
|
||||
}
|
||||
|
||||
renderReq := scriptorium.RenderArtifactRequest{
|
||||
Binary: env.Config.Pipeline.Scriptorium.Binary,
|
||||
@@ -187,10 +425,10 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
}
|
||||
renderRes, renderErr := env.Scriptorium.RenderArtifact(ctx, renderReq)
|
||||
if renderErr != nil {
|
||||
return nil, fmt.Errorf("analyze: scriptorium render failed: %w", renderErr)
|
||||
return nil, fmt.Errorf("analyze: scriptorium render failed for artifact %q: %w", artifactName, renderErr)
|
||||
}
|
||||
if renderRes.ValidationFailed {
|
||||
return nil, fmt.Errorf("analyze: scriptorium render returned validation_failed=true")
|
||||
return nil, fmt.Errorf("analyze: scriptorium render returned validation_failed=true for artifact %q", artifactName)
|
||||
}
|
||||
finalRenderOutputPath := coalesceString(renderRes.OutputPath, renderReq.OutputPath)
|
||||
if err := requireNonEmptyFile(finalRenderOutputPath, artifactName+" render output"); err != nil {
|
||||
@@ -238,7 +476,8 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
if runErr != nil {
|
||||
if res.ValidationFailed {
|
||||
return nil, fmt.Errorf(
|
||||
"analyze: scriptorium validation failed (prompt_id=%q, output_path=%q, exit_code=%d, stdout_log=%q, stderr_log=%q): %w",
|
||||
"analyze: scriptorium validation failed (artifact=%q, prompt_id=%q, output_path=%q, exit_code=%d, stdout_log=%q, stderr_log=%q): %w",
|
||||
artifactName,
|
||||
req.PromptID,
|
||||
coalesceString(res.OutputPath, req.OutputPath),
|
||||
res.ExitCode,
|
||||
@@ -247,27 +486,30 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
runErr,
|
||||
)
|
||||
}
|
||||
return nil, fmt.Errorf("analyze: scriptorium run failed: %w", runErr)
|
||||
return nil, fmt.Errorf("analyze: scriptorium run failed for artifact %q: %w", artifactName, runErr)
|
||||
}
|
||||
if res.ValidationFailed {
|
||||
return nil, fmt.Errorf("analyze: scriptorium run returned validation_failed=true")
|
||||
return nil, fmt.Errorf("analyze: scriptorium run returned validation_failed=true for artifact %q", artifactName)
|
||||
}
|
||||
|
||||
finalOutputPath := coalesceString(res.OutputPath, req.OutputPath)
|
||||
if err := requireNonEmptyFile(finalOutputPath, artifactName+" output"); err != nil {
|
||||
return nil, fmt.Errorf("analyze: %w", err)
|
||||
}
|
||||
|
||||
artifactRef := artifacts.Ref{
|
||||
Kind: artifactName,
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalOutputPath,
|
||||
promotedArtifact, err := promoteRunLocalOutput(env.ArtifactStore, finalOutputPath, canonicalOutputPath, artifacts.Ref{
|
||||
Kind: artifactName,
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("analyze: promote artifact output for %q: %w", artifactName, err)
|
||||
}
|
||||
|
||||
logPaths = append(logPaths, stdoutLogPath, stderrLogPath)
|
||||
generatedConfigs = append(generatedConfigs, generatedConfigPath)
|
||||
meta["output_path"] = finalOutputPath
|
||||
meta["run_output_path"] = finalOutputPath
|
||||
meta["path"] = canonicalOutputPath
|
||||
meta["output_path"] = canonicalOutputPath
|
||||
meta["generated_config_path"] = generatedConfigPath
|
||||
meta["stdout_log_path"] = stdoutLogPath
|
||||
meta["stderr_log_path"] = stderrLogPath
|
||||
@@ -281,39 +523,38 @@ func (analyzeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
meta["adapter_generated_config"] = res.GeneratedConfigPath
|
||||
meta["adapter_stdout_log_path"] = res.StdoutLogPath
|
||||
meta["adapter_stderr_log_path"] = res.StderrLogPath
|
||||
meta["provenance"] = artifacts.ArtifactProvenanceGeneratedCurrentAnalyzeRun
|
||||
if res.Metadata != nil {
|
||||
meta["adapter_metadata"] = res.Metadata
|
||||
}
|
||||
|
||||
return &StageResult{
|
||||
Outputs: []artifacts.Ref{artifactRef},
|
||||
return &analyzeArtifactExecutionResult{
|
||||
Output: promotedArtifact,
|
||||
Logs: logPaths,
|
||||
GeneratedConfigs: generatedConfigs,
|
||||
Metadata: meta,
|
||||
ReusedArtifacts: reusedArtifacts,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func selectAnalyzeArtifact(cfg *config.ScriptoriumConfig) (string, config.ScriptoriumArtifactConfig, string, error) {
|
||||
if cfg == nil {
|
||||
return "", config.ScriptoriumArtifactConfig{}, "pipeline.scriptorium is not configured", nil
|
||||
func extractPlanNames(plans []analyzeArtifactExecutionPlan) []string {
|
||||
if len(plans) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, 0, len(plans))
|
||||
for _, plan := range plans {
|
||||
out = append(out, plan.Name)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
enabled := []string{}
|
||||
for name, artifact := range cfg.Artifacts {
|
||||
if artifact.Enabled {
|
||||
enabled = append(enabled, name)
|
||||
}
|
||||
func configuredArtifactNameFromSourceID(sourceID string) string {
|
||||
trimmed := strings.TrimSpace(sourceID)
|
||||
const prefix = "narratio.artifact."
|
||||
if !strings.HasPrefix(trimmed, prefix) {
|
||||
return ""
|
||||
}
|
||||
sort.Strings(enabled)
|
||||
if len(enabled) == 0 {
|
||||
return "", config.ScriptoriumArtifactConfig{}, "no enabled scriptorium artifacts configured", nil
|
||||
}
|
||||
|
||||
sessionRecapCfg, ok := cfg.Artifacts["session_recap"]
|
||||
if !ok || !sessionRecapCfg.Enabled {
|
||||
return "", config.ScriptoriumArtifactConfig{}, "", fmt.Errorf("only artifacts.session_recap is supported in this analyze implementation; enabled=%s", strings.Join(enabled, ","))
|
||||
}
|
||||
return "session_recap", sessionRecapCfg, "", nil
|
||||
return strings.TrimPrefix(trimmed, prefix)
|
||||
}
|
||||
|
||||
func discoverProcessedTranscript(m *manifest.Manifest, paths artifacts.SessionPaths) (string, string, error) {
|
||||
@@ -350,40 +591,6 @@ func discoverProcessedTranscript(m *manifest.Manifest, paths artifacts.SessionPa
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
func discoverTrimmedTranscript(m *manifest.Manifest, paths artifacts.SessionPaths) (string, string, error) {
|
||||
candidates := []string{}
|
||||
if m != nil && m.Stages != nil {
|
||||
if sr := m.Stages["trim"]; sr != nil {
|
||||
for _, out := range sr.Outputs {
|
||||
if out.Kind != "transcript_trimmed" {
|
||||
continue
|
||||
}
|
||||
p := strings.TrimSpace(out.LocalPath)
|
||||
if p == "" {
|
||||
continue
|
||||
}
|
||||
resolved := artifacts.ResolveSessionLocalPathForRead(paths, p)
|
||||
candidates = append(candidates, filepath.Clean(resolved))
|
||||
}
|
||||
}
|
||||
}
|
||||
deduped := dedupeAndSortPaths(candidates)
|
||||
for _, p := range deduped {
|
||||
if info, err := os.Stat(p); err == nil && !info.IsDir() {
|
||||
return p, "manifest.trim.outputs", nil
|
||||
}
|
||||
}
|
||||
|
||||
fallback := filepath.Join(paths.TranscriptsDir, "trimmed.json")
|
||||
if info, err := os.Stat(fallback); err == nil && !info.IsDir() {
|
||||
return filepath.Clean(fallback), "fallback.transcripts_dir", nil
|
||||
}
|
||||
if len(deduped) > 0 {
|
||||
return deduped[0], "manifest.trim.outputs", nil
|
||||
}
|
||||
return "", "", nil
|
||||
}
|
||||
|
||||
type analyzeTranscriptInputs struct {
|
||||
ProcessedPath string
|
||||
ProcessedSource string
|
||||
@@ -393,52 +600,125 @@ type analyzeTranscriptInputs struct {
|
||||
TrimmedSource string
|
||||
}
|
||||
|
||||
func discoverAnalyzeTranscriptRefs(m *manifest.Manifest, paths artifacts.SessionPaths) analyzeTranscriptInputs {
|
||||
processedPath, processedSource := discoverAnalyzeArtifactRef(m, paths, artifacts.ArtifactTranscriptPolished)
|
||||
normalizedPath, normalizedSource := discoverAnalyzeArtifactRef(m, paths, artifacts.ArtifactTranscriptFull)
|
||||
trimmedPath, trimmedSource := discoverAnalyzeArtifactRef(m, paths, artifacts.ArtifactTranscriptTrimmed)
|
||||
return analyzeTranscriptInputs{
|
||||
ProcessedPath: processedPath,
|
||||
ProcessedSource: processedSource,
|
||||
NormalizedPath: normalizedPath,
|
||||
NormalizedSource: normalizedSource,
|
||||
TrimmedPath: trimmedPath,
|
||||
TrimmedSource: trimmedSource,
|
||||
}
|
||||
}
|
||||
|
||||
func discoverAnalyzeArtifactRef(m *manifest.Manifest, paths artifacts.SessionPaths, source string) (string, string) {
|
||||
resolved, err := artifacts.ResolveSessionArtifact(paths, m, source)
|
||||
if err != nil {
|
||||
return "", ""
|
||||
}
|
||||
return resolved.Path, resolved.Provenance
|
||||
}
|
||||
|
||||
func resolveScriptoriumInput(
|
||||
inputName string,
|
||||
inputCfg config.ScriptoriumInputConfig,
|
||||
transcriptInputs analyzeTranscriptInputs,
|
||||
m *manifest.Manifest,
|
||||
paths artifacts.SessionPaths,
|
||||
sessionDir string,
|
||||
) (string, bool, error) {
|
||||
switch strings.TrimSpace(inputCfg.Source) {
|
||||
case "processed_transcript":
|
||||
if strings.TrimSpace(transcriptInputs.ProcessedPath) == "" {
|
||||
return "", false, nil
|
||||
}
|
||||
if err := validateProcessedTranscriptOutput(transcriptInputs.ProcessedPath); err != nil {
|
||||
return "", false, fmt.Errorf("processed transcript %q invalid: %w", transcriptInputs.ProcessedPath, err)
|
||||
}
|
||||
return transcriptInputs.ProcessedPath, true, nil
|
||||
case "normalized_transcript":
|
||||
if strings.TrimSpace(transcriptInputs.NormalizedPath) == "" {
|
||||
return "", false, fmt.Errorf("normalized transcript input is unavailable; run normalize stage first")
|
||||
}
|
||||
if err := validateProcessedTranscriptOutput(transcriptInputs.NormalizedPath); err != nil {
|
||||
return "", false, fmt.Errorf("normalized transcript %q invalid: %w", transcriptInputs.NormalizedPath, err)
|
||||
}
|
||||
return transcriptInputs.NormalizedPath, true, nil
|
||||
case "trimmed_transcript":
|
||||
if strings.TrimSpace(transcriptInputs.TrimmedPath) == "" {
|
||||
return "", false, fmt.Errorf("trimmed transcript input is unavailable; run trim stage first")
|
||||
}
|
||||
if err := validateProcessedTranscriptOutput(transcriptInputs.TrimmedPath); err != nil {
|
||||
return "", false, fmt.Errorf("trimmed transcript %q invalid: %w", transcriptInputs.TrimmedPath, err)
|
||||
}
|
||||
return transcriptInputs.TrimmedPath, true, nil
|
||||
runtimeCatalog *artifacts.ArtifactCatalog,
|
||||
) (string, bool, *artifacts.ResolvedSessionArtifact, error) {
|
||||
source := strings.TrimSpace(inputCfg.Source)
|
||||
switch source {
|
||||
case "previous_session_artifact":
|
||||
if strings.TrimSpace(inputCfg.Path) == "" {
|
||||
return "", false, nil
|
||||
return "", false, nil, nil
|
||||
}
|
||||
resolved := resolveInputPathForRead(paths, sessionDir, inputCfg.Path)
|
||||
if err := requireFile(resolved, "scriptorium input "+inputName); err != nil {
|
||||
return "", false, nil
|
||||
return "", false, nil, nil
|
||||
}
|
||||
return resolved, true, nil
|
||||
return resolved, true, nil, nil
|
||||
default:
|
||||
return "", false, fmt.Errorf("unsupported source %q", inputCfg.Source)
|
||||
resolved, err := artifacts.ResolveSessionArtifactWithCatalog(paths, m, source, runtimeCatalog)
|
||||
if err == nil {
|
||||
copy := resolved
|
||||
return resolved.Path, true, ©, nil
|
||||
}
|
||||
if errors.Is(err, artifacts.ErrSessionArtifactNotFound) {
|
||||
if artifacts.IsConfiguredArtifactSource(source) {
|
||||
if inputCfg.Required {
|
||||
return "", false, nil, fmt.Errorf("configured artifact source %q is unavailable", source)
|
||||
}
|
||||
return "", false, nil, nil
|
||||
}
|
||||
normalized, normalizeErr := artifacts.NormalizeSessionArtifactSource(source)
|
||||
if normalizeErr != nil {
|
||||
return "", false, nil, normalizeErr
|
||||
}
|
||||
switch normalized {
|
||||
case artifacts.ArtifactTranscriptPolished:
|
||||
return "", false, nil, nil
|
||||
case artifacts.ArtifactTranscriptFull:
|
||||
return "", false, nil, fmt.Errorf("normalized transcript input is unavailable; run normalize stage first")
|
||||
case artifacts.ArtifactTranscriptTrimmed:
|
||||
return "", false, nil, fmt.Errorf("trimmed transcript input is unavailable; run trim stage first")
|
||||
default:
|
||||
return "", false, nil, nil
|
||||
}
|
||||
}
|
||||
return "", false, nil, err
|
||||
}
|
||||
}
|
||||
|
||||
func buildAnalyzeRuntimeArtifactCatalog(
|
||||
paths artifacts.SessionPaths,
|
||||
scriptoriumCfg *config.ScriptoriumConfig,
|
||||
selectedArtifacts []string,
|
||||
) (*artifacts.ArtifactCatalog, error) {
|
||||
catalog := artifacts.NewArtifactCatalog()
|
||||
if err := catalog.RegisterBuiltIns(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if scriptoriumCfg == nil {
|
||||
return catalog, nil
|
||||
}
|
||||
|
||||
configured := map[string]artifacts.ConfiguredArtifactDefinition{}
|
||||
for key, artifactCfg := range scriptoriumCfg.Artifacts {
|
||||
configured[key] = artifacts.ConfiguredArtifactDefinition{
|
||||
Enabled: artifactCfg.Enabled,
|
||||
OutputPath: artifactCfg.OutputPath,
|
||||
}
|
||||
}
|
||||
if err := catalog.RegisterConfiguredArtifacts(configured, selectedArtifacts); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
for _, entry := range catalog.ListConfigured() {
|
||||
if entry.Executable {
|
||||
continue
|
||||
}
|
||||
if strings.TrimSpace(entry.CanonicalRelPath) == "" {
|
||||
continue
|
||||
}
|
||||
resolvedPath, err := resolveScriptoriumOutputPath(paths, entry.CanonicalRelPath)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if err := requireNonEmptyFile(resolvedPath, "configured artifact "+entry.SourceID); err != nil {
|
||||
continue
|
||||
}
|
||||
if err := catalog.MarkAvailableFromDisk(entry.SourceID, resolvedPath); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
return catalog, nil
|
||||
}
|
||||
|
||||
func resolveInputPathForRead(paths artifacts.SessionPaths, sessionDir, pathValue string) string {
|
||||
trimmed := strings.TrimSpace(pathValue)
|
||||
if trimmed == "" {
|
||||
|
||||
@@ -17,7 +17,7 @@ import (
|
||||
|
||||
func TestAnalyzeGeneratesSessionRecapFromTrimmedTranscript(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
result, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
@@ -70,7 +70,7 @@ func TestAnalyzeGeneratesSessionRecapFromTrimmedTranscript(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRenderDebugFalseDoesNotCallRenderArtifact(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = false
|
||||
@@ -86,7 +86,7 @@ func TestAnalyzeRenderDebugFalseDoesNotCallRenderArtifact(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRenderDebugArtifactOverrideFalseWinsOverGlobalTrue(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
@@ -109,7 +109,7 @@ func TestAnalyzeRenderDebugArtifactOverrideFalseWinsOverGlobalTrue(t *testing.T)
|
||||
|
||||
func TestAnalyzeRenderDebugTrueCallsRenderBeforeRun(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
@@ -130,7 +130,7 @@ func TestAnalyzeRenderDebugTrueCallsRenderBeforeRun(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRenderOutputPathIsRecorded(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
|
||||
@@ -154,7 +154,7 @@ func TestAnalyzeRenderOutputPathIsRecorded(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRenderFailurePreventsRun(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
fake.RenderErr = errors.New("render boom")
|
||||
@@ -176,7 +176,7 @@ func TestAnalyzeRenderFailurePreventsRun(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRenderInvalidJSONFailsClearly(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
runner := &orderedScriptoriumRunner{
|
||||
@@ -199,7 +199,7 @@ func TestAnalyzeRenderInvalidJSONFailsClearly(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRunStillSucceedsWhenRenderSucceeds(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
env.Config.Pipeline.Scriptorium.RenderDebug = true
|
||||
|
||||
@@ -220,7 +220,7 @@ func TestAnalyzeRunStillSucceedsWhenRenderSucceeds(t *testing.T) {
|
||||
|
||||
func TestAnalyzeOmitsOptionalPreviousRecapWhenUnavailable(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = config.ScriptoriumArtifactConfig{
|
||||
@@ -231,7 +231,7 @@ func TestAnalyzeOmitsOptionalPreviousRecapWhenUnavailable(t *testing.T) {
|
||||
Timeout: "2m",
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"transcript": {
|
||||
Source: "processed_transcript",
|
||||
Source: "narratio.transcript.polished",
|
||||
Required: true,
|
||||
},
|
||||
"previous_recap": {
|
||||
@@ -256,6 +256,32 @@ func TestAnalyzeOmitsOptionalPreviousRecapWhenUnavailable(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
result, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(fake.RunRequests) != 1 {
|
||||
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
|
||||
}
|
||||
if !strings.Contains(fake.RunRequests[0].OutputPath, filepath.Join("runs", m.RunID, "analyze", "outputs")) {
|
||||
t.Fatalf("run output path = %q, want run-local path", fake.RunRequests[0].OutputPath)
|
||||
}
|
||||
if len(result.Outputs) != 1 {
|
||||
t.Fatalf("outputs len = %d, want 1", len(result.Outputs))
|
||||
}
|
||||
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
|
||||
}
|
||||
}
|
||||
|
||||
type orderedScriptoriumRunner struct {
|
||||
Calls []string
|
||||
RenderErr error
|
||||
@@ -318,7 +344,7 @@ func (r *orderedScriptoriumRunner) RunArtifact(_ context.Context, req scriptoriu
|
||||
|
||||
func TestAnalyzeIncludesPreviousRecapWhenConfiguredAndAvailable(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
|
||||
|
||||
previousRecapPath := filepath.Join(filepath.Dir(env.Config.SessionPath), "previous", "session_recap.md")
|
||||
@@ -332,7 +358,7 @@ func TestAnalyzeIncludesPreviousRecapWhenConfiguredAndAvailable(t *testing.T) {
|
||||
Timeout: "2m",
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"transcript": {
|
||||
Source: "processed_transcript",
|
||||
Source: "narratio.transcript.polished",
|
||||
Required: true,
|
||||
},
|
||||
"previous_recap": {
|
||||
@@ -359,7 +385,7 @@ func TestAnalyzeIncludesPreviousRecapWhenConfiguredAndAvailable(t *testing.T) {
|
||||
|
||||
func TestAnalyzeFailsWhenRequiredPreviousRecapMissing(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = config.ScriptoriumArtifactConfig{
|
||||
@@ -368,7 +394,7 @@ func TestAnalyzeFailsWhenRequiredPreviousRecapMissing(t *testing.T) {
|
||||
OutputPath: "artifacts/session_recap.md",
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"transcript": {
|
||||
Source: "processed_transcript",
|
||||
Source: "narratio.transcript.polished",
|
||||
Required: true,
|
||||
},
|
||||
"previous_recap": {
|
||||
@@ -388,9 +414,312 @@ func TestAnalyzeFailsWhenRequiredPreviousRecapMissing(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeResolvesConfiguredArtifactInputFromDisabledArtifactOutput(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
playerHandoutPath := filepath.Join(paths.ArtifactsDir, "player_handout.md")
|
||||
writeAnalyzeFile(t, playerHandoutPath, "handout\n")
|
||||
|
||||
sessionRecap := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
sessionRecap.Inputs["recap"] = config.ScriptoriumInputConfig{
|
||||
Source: "narratio.artifact.player_handout",
|
||||
Required: true,
|
||||
}
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = sessionRecap
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
|
||||
Enabled: false,
|
||||
OutputPath: "artifacts/player_handout.md",
|
||||
}
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(fake.RunRequests) != 1 {
|
||||
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
|
||||
}
|
||||
if got := fake.RunRequests[0].InputPaths["recap"]; got != playerHandoutPath {
|
||||
t.Fatalf("recap input = %q, want %q", got, playerHandoutPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeMetadataIncludesGeneratedAndReusedArtifacts(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
playerHandoutPath := filepath.Join(paths.ArtifactsDir, "player_handout.md")
|
||||
writeAnalyzeFile(t, playerHandoutPath, "handout\n")
|
||||
|
||||
sessionRecap := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
sessionRecap.Inputs["recap"] = config.ScriptoriumInputConfig{
|
||||
Source: "narratio.artifact.player_handout",
|
||||
Required: true,
|
||||
}
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = sessionRecap
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
|
||||
Enabled: false,
|
||||
OutputPath: "artifacts/player_handout.md",
|
||||
}
|
||||
|
||||
result, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
|
||||
generated := mustArtifactEntryList(t, result.Metadata, "generated_artifacts")
|
||||
if len(generated) != 1 {
|
||||
t.Fatalf("generated_artifacts len = %d, want 1 (%#v)", len(generated), generated)
|
||||
}
|
||||
g0 := generated[0]
|
||||
if g0["name"] != "session_recap" {
|
||||
t.Fatalf("generated[0].name = %#v, want session_recap", g0["name"])
|
||||
}
|
||||
if g0["source_id"] != "narratio.artifact.session_recap" {
|
||||
t.Fatalf("generated[0].source_id = %#v, want narratio.artifact.session_recap", g0["source_id"])
|
||||
}
|
||||
if g0["output_kind"] != "scriptorium_artifact" {
|
||||
t.Fatalf("generated[0].output_kind = %#v, want scriptorium_artifact", g0["output_kind"])
|
||||
}
|
||||
if g0["path"] != filepath.Join(paths.ArtifactsDir, "session_recap.md") {
|
||||
t.Fatalf("generated[0].path = %#v, want session recap path", g0["path"])
|
||||
}
|
||||
if g0["prompt_id"] != "dnd.session_recap" {
|
||||
t.Fatalf("generated[0].prompt_id = %#v, want dnd.session_recap", g0["prompt_id"])
|
||||
}
|
||||
if g0["profile_id"] != "local-quality" {
|
||||
t.Fatalf("generated[0].profile_id = %#v, want local-quality", g0["profile_id"])
|
||||
}
|
||||
if g0["provenance"] != artifacts.ArtifactProvenanceGeneratedCurrentAnalyzeRun {
|
||||
t.Fatalf("generated[0].provenance = %#v, want %q", g0["provenance"], artifacts.ArtifactProvenanceGeneratedCurrentAnalyzeRun)
|
||||
}
|
||||
|
||||
reused := mustArtifactEntryList(t, result.Metadata, "reused_artifacts")
|
||||
if len(reused) != 1 {
|
||||
t.Fatalf("reused_artifacts len = %d, want 1 (%#v)", len(reused), reused)
|
||||
}
|
||||
r0 := reused[0]
|
||||
if r0["name"] != "player_handout" {
|
||||
t.Fatalf("reused[0].name = %#v, want player_handout", r0["name"])
|
||||
}
|
||||
if r0["source_id"] != "narratio.artifact.player_handout" {
|
||||
t.Fatalf("reused[0].source_id = %#v, want narratio.artifact.player_handout", r0["source_id"])
|
||||
}
|
||||
if r0["path"] != playerHandoutPath {
|
||||
t.Fatalf("reused[0].path = %#v, want %q", r0["path"], playerHandoutPath)
|
||||
}
|
||||
if r0["provenance"] != artifacts.ArtifactProvenanceDisabledFromDisk {
|
||||
t.Fatalf("reused[0].provenance = %#v, want %q", r0["provenance"], artifacts.ArtifactProvenanceDisabledFromDisk)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeRunsMultipleIndependentArtifactsInDeterministicOrder(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
|
||||
Enabled: true,
|
||||
PromptID: "dnd.player_handout",
|
||||
ProfileID: "local-quality",
|
||||
OutputPath: "artifacts/player_handout.md",
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"transcript": {
|
||||
Source: "narratio.transcript.trimmed",
|
||||
Required: true,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
result, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(fake.RunRequests) != 2 {
|
||||
t.Fatalf("run requests = %d, want 2", len(fake.RunRequests))
|
||||
}
|
||||
if fake.RunRequests[0].PromptID != "dnd.player_handout" {
|
||||
t.Fatalf("first prompt id = %q, want dnd.player_handout", fake.RunRequests[0].PromptID)
|
||||
}
|
||||
if fake.RunRequests[1].PromptID != "dnd.session_recap" {
|
||||
t.Fatalf("second prompt id = %q, want dnd.session_recap", fake.RunRequests[1].PromptID)
|
||||
}
|
||||
|
||||
selected, ok := result.Metadata["selected_artifacts"].([]string)
|
||||
if !ok {
|
||||
t.Fatalf("selected_artifacts = %#v, want []string", result.Metadata["selected_artifacts"])
|
||||
}
|
||||
if len(selected) != 2 || selected[0] != "player_handout" || selected[1] != "session_recap" {
|
||||
t.Fatalf("selected_artifacts = %#v, want [player_handout session_recap]", selected)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeRunsDependenciesBeforeDependents(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
|
||||
Enabled: true,
|
||||
DependsOn: []string{"session_recap"},
|
||||
PromptID: "dnd.player_handout",
|
||||
ProfileID: "local-quality",
|
||||
OutputPath: "artifacts/player_handout.md",
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"recap": {
|
||||
Source: "narratio.artifact.session_recap",
|
||||
Required: true,
|
||||
},
|
||||
"transcript": {
|
||||
Source: "narratio.transcript.trimmed",
|
||||
Required: true,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(fake.RunRequests) != 2 {
|
||||
t.Fatalf("run requests = %d, want 2", len(fake.RunRequests))
|
||||
}
|
||||
if fake.RunRequests[0].PromptID != "dnd.session_recap" {
|
||||
t.Fatalf("first prompt id = %q, want dnd.session_recap", fake.RunRequests[0].PromptID)
|
||||
}
|
||||
if fake.RunRequests[1].PromptID != "dnd.player_handout" {
|
||||
t.Fatalf("second prompt id = %q, want dnd.player_handout", fake.RunRequests[1].PromptID)
|
||||
}
|
||||
if got := fake.RunRequests[1].InputPaths["recap"]; got != filepath.Join(paths.ArtifactsDir, "session_recap.md") {
|
||||
t.Fatalf("dependent recap path = %q, want %q", got, filepath.Join(paths.ArtifactsDir, "session_recap.md"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeAppliesSelectedArtifactsFilter(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
|
||||
Enabled: true,
|
||||
PromptID: "dnd.player_handout",
|
||||
ProfileID: "local-quality",
|
||||
OutputPath: "artifacts/player_handout.md",
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"transcript": {
|
||||
Source: "narratio.transcript.trimmed",
|
||||
Required: true,
|
||||
},
|
||||
},
|
||||
}
|
||||
env.SelectedAnalyzeArtifacts = []string{"player_handout"}
|
||||
|
||||
result, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(fake.RunRequests) != 1 {
|
||||
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
|
||||
}
|
||||
if fake.RunRequests[0].PromptID != "dnd.player_handout" {
|
||||
t.Fatalf("prompt id = %q, want dnd.player_handout", fake.RunRequests[0].PromptID)
|
||||
}
|
||||
if len(result.Outputs) != 1 || result.Outputs[0].Kind != "player_handout" {
|
||||
t.Fatalf("outputs = %#v, want only player_handout", result.Outputs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeMetadataIncludesMultipleGeneratedArtifacts(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
|
||||
Enabled: true,
|
||||
PromptID: "dnd.player_handout",
|
||||
ProfileID: "local-quality",
|
||||
OutputPath: "artifacts/player_handout.md",
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"transcript": {
|
||||
Source: "narratio.transcript.trimmed",
|
||||
Required: true,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
result, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
generated := mustArtifactEntryList(t, result.Metadata, "generated_artifacts")
|
||||
if len(generated) != 2 {
|
||||
t.Fatalf("generated_artifacts len = %d, want 2 (%#v)", len(generated), generated)
|
||||
}
|
||||
for i, entry := range generated {
|
||||
for _, field := range []string{"name", "source_id", "output_kind", "path", "prompt_id", "profile_id", "provenance"} {
|
||||
if _, ok := entry[field]; !ok {
|
||||
t.Fatalf("generated[%d] missing field %q: %#v", i, field, entry)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeFailsWhenRequiredConfiguredArtifactMissing(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
sessionRecap := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
sessionRecap.Inputs["recap"] = config.ScriptoriumInputConfig{
|
||||
Source: "narratio.artifact.player_handout",
|
||||
Required: true,
|
||||
}
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = sessionRecap
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
|
||||
Enabled: false,
|
||||
OutputPath: "artifacts/player_handout.md",
|
||||
}
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err == nil {
|
||||
t.Fatal("expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), `configured artifact source "narratio.artifact.player_handout" is unavailable`) {
|
||||
t.Fatalf("error = %q, want configured artifact unavailable context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeOmitsOptionalMissingConfiguredArtifactInput(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
sessionRecap := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
sessionRecap.Inputs["recap"] = config.ScriptoriumInputConfig{
|
||||
Source: "narratio.artifact.player_handout",
|
||||
Required: false,
|
||||
}
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = sessionRecap
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["player_handout"] = config.ScriptoriumArtifactConfig{
|
||||
Enabled: false,
|
||||
OutputPath: "artifacts/player_handout.md",
|
||||
}
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(fake.RunRequests) != 1 {
|
||||
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
|
||||
}
|
||||
if _, exists := fake.RunRequests[0].InputPaths["recap"]; exists {
|
||||
t.Fatalf("optional recap input should be omitted when unavailable, got %q", fake.RunRequests[0].InputPaths["recap"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeFailsWhenOutputPathMissing(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
@@ -425,12 +754,12 @@ func TestAnalyzeFailsWhenTrimmedTranscriptMissing(t *testing.T) {
|
||||
|
||||
func TestAnalyzeSupportsProcessedTranscriptSourceWhenConfigured(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[]}`)
|
||||
|
||||
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
|
||||
Source: "processed_transcript",
|
||||
Source: "narratio.transcript.polished",
|
||||
Required: true,
|
||||
}
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
|
||||
@@ -447,15 +776,39 @@ func TestAnalyzeSupportsProcessedTranscriptSourceWhenConfigured(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeSupportsCanonicalTrimmedTranscriptSourceWhenConfigured(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
|
||||
Source: "narratio.transcript.trimmed",
|
||||
Required: true,
|
||||
}
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(fake.RunRequests) != 1 {
|
||||
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
|
||||
}
|
||||
if fake.RunRequests[0].InputPaths["transcript"] != filepath.Join(paths.TranscriptsDir, "trimmed.json") {
|
||||
t.Fatalf("transcript input = %q, want trimmed transcript path", fake.RunRequests[0].InputPaths["transcript"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeSupportsNormalizedTranscriptSourceWhenConfigured(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
normalizedPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
writeAnalyzeFile(t, normalizedPath, `{"segments":[{"id":1}]}`)
|
||||
|
||||
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
|
||||
Source: "normalized_transcript",
|
||||
Source: "narratio.transcript.full",
|
||||
Required: true,
|
||||
}
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
|
||||
@@ -472,9 +825,9 @@ func TestAnalyzeSupportsNormalizedTranscriptSourceWhenConfigured(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeSupportsNormalizedTranscriptSourceFromManifestOutput(t *testing.T) {
|
||||
func TestAnalyzeSupportsCanonicalNormalizedTranscriptSourceFromManifestOutput(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
fallbackPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
|
||||
writeAnalyzeFile(t, fallbackPath, `{"segments":[{"id":999}]}`)
|
||||
@@ -485,7 +838,37 @@ func TestAnalyzeSupportsNormalizedTranscriptSourceFromManifestOutput(t *testing.
|
||||
|
||||
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
|
||||
Source: "normalized_transcript",
|
||||
Source: "narratio.transcript.full",
|
||||
Required: true,
|
||||
}
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(fake.RunRequests) != 1 {
|
||||
t.Fatalf("run requests = %d, want 1", len(fake.RunRequests))
|
||||
}
|
||||
if fake.RunRequests[0].InputPaths["transcript"] != manifestPath {
|
||||
t.Fatalf("transcript input = %q, want manifest normalized transcript path", fake.RunRequests[0].InputPaths["transcript"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeSupportsNormalizedTranscriptSourceFromManifestOutput(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
fallbackPath := filepath.Join(paths.TranscriptsDir, "normalized.json")
|
||||
manifestPath := filepath.Join(paths.ArtifactsDir, "normalized.from-manifest.json")
|
||||
writeAnalyzeFile(t, fallbackPath, `{"segments":[{"id":999}]}`)
|
||||
writeAnalyzeFile(t, manifestPath, `{"segments":[{"id":10}]}`)
|
||||
m.MarkStageSucceeded("normalize", time.Now().UTC(), []manifest.ArtifactRecord{
|
||||
{Kind: "transcript_normalized", LocalPath: manifestPath},
|
||||
})
|
||||
|
||||
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
|
||||
Source: "narratio.transcript.full",
|
||||
Required: true,
|
||||
}
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
|
||||
@@ -506,7 +889,7 @@ func TestAnalyzeFailsWhenNormalizedTranscriptMissing(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
artifact := env.Config.Pipeline.Scriptorium.Artifacts["session_recap"]
|
||||
artifact.Inputs["transcript"] = config.ScriptoriumInputConfig{
|
||||
Source: "normalized_transcript",
|
||||
Source: "narratio.transcript.full",
|
||||
Required: true,
|
||||
}
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = artifact
|
||||
@@ -528,7 +911,7 @@ func TestAnalyzeFailsWhenNormalizedTranscriptMissing(t *testing.T) {
|
||||
|
||||
func TestAnalyzeFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{not-json`)
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
@@ -542,7 +925,7 @@ func TestAnalyzeFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
|
||||
|
||||
func TestAnalyzeFailsWhenProcessedTranscriptMissingSegmentsArray(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"not_segments":[]}`)
|
||||
|
||||
_, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
@@ -556,7 +939,7 @@ func TestAnalyzeFailsWhenProcessedTranscriptMissingSegmentsArray(t *testing.T) {
|
||||
|
||||
func TestAnalyzeRecordsRefsAndMetadata(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
result, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
@@ -585,7 +968,7 @@ func TestAnalyzeRecordsRefsAndMetadata(t *testing.T) {
|
||||
|
||||
func TestAnalyzeHandlesAdapterError(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
fake.RunErr = errors.New("adapter boom")
|
||||
|
||||
@@ -600,7 +983,7 @@ func TestAnalyzeHandlesAdapterError(t *testing.T) {
|
||||
|
||||
func TestAnalyzeHandlesValidationFailedResultAsError(t *testing.T) {
|
||||
env, m, fake := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
fake.RunResult = scriptorium.ArtifactResult{
|
||||
ValidationFailed: true,
|
||||
@@ -619,7 +1002,7 @@ func TestAnalyzeHandlesValidationFailedResultAsError(t *testing.T) {
|
||||
|
||||
func TestAnalyzeSkipsWhenNoEnabledScriptoriumArtifactsConfigured(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeAnalyzeFile(t, filepath.Join(paths.TranscriptsDir, "trimmed.json"), `{"segments":[]}`)
|
||||
|
||||
env.Config.Pipeline.Scriptorium.Artifacts["session_recap"] = config.ScriptoriumArtifactConfig{
|
||||
@@ -635,6 +1018,22 @@ func TestAnalyzeSkipsWhenNoEnabledScriptoriumArtifactsConfigured(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnalyzeSkipsWhenArtifactMapEmpty(t *testing.T) {
|
||||
env, m, _ := setupAnalyzeEnv(t)
|
||||
env.Config.Pipeline.Scriptorium.Artifacts = map[string]config.ScriptoriumArtifactConfig{}
|
||||
|
||||
result, err := (analyzeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if result.Metadata["skipped"] != true {
|
||||
t.Fatalf("metadata = %#v, want skipped=true", result.Metadata)
|
||||
}
|
||||
if result.Metadata["reason"] != "no scriptorium artifacts configured" {
|
||||
t.Fatalf("reason = %#v, want no scriptorium artifacts configured", result.Metadata["reason"])
|
||||
}
|
||||
}
|
||||
|
||||
func setupAnalyzeEnv(t *testing.T) (*Env, *manifest.Manifest, *scriptorium.FakeRunner) {
|
||||
t.Helper()
|
||||
workspace := t.TempDir()
|
||||
@@ -664,7 +1063,7 @@ func setupAnalyzeEnv(t *testing.T) (*Env, *manifest.Manifest, *scriptorium.FakeR
|
||||
Timeout: "2m",
|
||||
Inputs: map[string]config.ScriptoriumInputConfig{
|
||||
"transcript": {
|
||||
Source: "trimmed_transcript",
|
||||
Source: "narratio.transcript.trimmed",
|
||||
Required: true,
|
||||
},
|
||||
"previous_recap": {
|
||||
@@ -692,7 +1091,7 @@ func setupAnalyzeEnv(t *testing.T) (*Env, *manifest.Manifest, *scriptorium.FakeR
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(workspace)
|
||||
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
|
||||
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -722,3 +1121,27 @@ func writeAnalyzeFileNoTest(path, contents string) {
|
||||
_ = os.MkdirAll(filepath.Dir(path), 0o755)
|
||||
_ = os.WriteFile(path, []byte(contents), 0o644)
|
||||
}
|
||||
|
||||
func mustArtifactEntryList(t *testing.T, metadata map[string]any, key string) []map[string]any {
|
||||
t.Helper()
|
||||
raw, ok := metadata[key]
|
||||
if !ok {
|
||||
t.Fatalf("metadata missing key %q: %#v", key, metadata)
|
||||
}
|
||||
if typed, ok := raw.([]map[string]any); ok {
|
||||
return typed
|
||||
}
|
||||
asList, ok := raw.([]any)
|
||||
if !ok {
|
||||
t.Fatalf("metadata[%q] = %#v, want []map[string]any", key, raw)
|
||||
}
|
||||
out := make([]map[string]any, 0, len(asList))
|
||||
for _, item := range asList {
|
||||
m, ok := item.(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("metadata[%q] entry = %#v, want map[string]any", key, item)
|
||||
}
|
||||
out = append(out, m)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -33,15 +33,6 @@ var archivePrerequisiteStages = []string{
|
||||
"analyze",
|
||||
}
|
||||
|
||||
var archiveRunUploadDirs = []string{
|
||||
"inputs",
|
||||
"transcripts",
|
||||
"artifacts",
|
||||
"reports",
|
||||
"config",
|
||||
"logs",
|
||||
}
|
||||
|
||||
func (archiveStage) Name() string { return "archive" }
|
||||
|
||||
func (archiveStage) Declares() IODecl {
|
||||
@@ -87,16 +78,16 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
return nil, fmt.Errorf("archive: remote object store backend is required when archive run upload is enabled")
|
||||
}
|
||||
|
||||
workDir, err := archiveWorkDir(env, m)
|
||||
runRoot, err := resolveArchiveRunRoot(env, m)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve local workdir: %w", err)
|
||||
return nil, fmt.Errorf("archive: resolve run root: %w", err)
|
||||
}
|
||||
workDirInfo, err := os.Stat(workDir)
|
||||
runRootInfo, err := os.Stat(runRoot)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: local workdir %q: %w", workDir, err)
|
||||
return nil, fmt.Errorf("archive: run root %q: %w", runRoot, err)
|
||||
}
|
||||
if !workDirInfo.IsDir() {
|
||||
return nil, fmt.Errorf("archive: local workdir %q is not a directory", workDir)
|
||||
if !runRootInfo.IsDir() {
|
||||
return nil, fmt.Errorf("archive: run root %q is not a directory", runRoot)
|
||||
}
|
||||
|
||||
runPrefix, err := archiveRunPrefix(env, m)
|
||||
@@ -116,17 +107,20 @@ func (archiveStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
return nil, fmt.Errorf("archive: run id is required")
|
||||
}
|
||||
|
||||
manifestSource, err := resolveArchiveManifestSource(env, m, workDir)
|
||||
manifestSource, err := resolveArchiveRunManifestSource(runRoot)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve manifest source: %w", err)
|
||||
return nil, fmt.Errorf("archive: resolve run manifest source: %w", err)
|
||||
}
|
||||
|
||||
runFiles, err := collectArchiveRunFiles(workDir, manifestSource)
|
||||
runFiles, err := collectArchiveRunFiles(runRoot, manifestSource)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: collect run files: %w", err)
|
||||
}
|
||||
workDirCandidates := archiveSourceWorkDirs(env, m, workDir)
|
||||
promotions, err := resolveArchivePromotions(workDirCandidates, env.Config.Pipeline.Archive.PromoteArtifacts)
|
||||
sessionRoot, err := resolveArchiveSessionRoot(env, m)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve session root for promotions: %w", err)
|
||||
}
|
||||
promotions, err := resolveArchivePromotions(sessionRoot, env.Config.Pipeline.Archive.PromoteArtifacts)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("archive: resolve promotion rules: %w", err)
|
||||
}
|
||||
@@ -249,36 +243,53 @@ func validateArchivePrerequisites(m *manifest.Manifest) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func archiveWorkDir(env *Env, m *manifest.Manifest) (string, error) {
|
||||
workDir := strings.TrimSpace(m.LocalWorkDir)
|
||||
if workDir != "" {
|
||||
cleaned := filepath.Clean(workDir)
|
||||
if info, err := os.Stat(cleaned); err == nil && info.IsDir() {
|
||||
return cleaned, nil
|
||||
}
|
||||
}
|
||||
|
||||
func resolveArchiveRunRoot(env *Env, m *manifest.Manifest) (string, error) {
|
||||
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
||||
if sessionID == "" {
|
||||
if sessionID == "" && m != nil {
|
||||
sessionID = strings.TrimSpace(m.SessionID)
|
||||
}
|
||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||
if campaign == "" {
|
||||
if campaign == "" && m != nil {
|
||||
campaign = strings.TrimSpace(m.Campaign)
|
||||
}
|
||||
runID := strings.TrimSpace(m.RunID)
|
||||
runID := ""
|
||||
if m != nil {
|
||||
runID = strings.TrimSpace(m.RunID)
|
||||
}
|
||||
if sessionID == "" || campaign == "" {
|
||||
return "", fmt.Errorf("campaign and session id are required")
|
||||
}
|
||||
if runID == "" {
|
||||
return "", fmt.Errorf("run id is required")
|
||||
}
|
||||
if campaign == "" || sessionID == "" {
|
||||
return "", fmt.Errorf("campaign and session id are required")
|
||||
|
||||
canonical := filepath.Clean(artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID, runID))
|
||||
canonicalExists, err := directoryExists(canonical)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("check canonical run root %q: %w", canonical, err)
|
||||
}
|
||||
runScoped := artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, campaign, sessionID, runID)
|
||||
if info, err := os.Stat(runScoped); err == nil && info.IsDir() {
|
||||
return runScoped, nil
|
||||
if !canonicalExists {
|
||||
return "", fmt.Errorf("run root not found for campaign %q session %q run %q at canonical path %q", campaign, sessionID, runID, canonical)
|
||||
}
|
||||
legacy := artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, sessionID)
|
||||
return legacy, nil
|
||||
return canonical, nil
|
||||
}
|
||||
|
||||
func resolveArchiveSessionRoot(env *Env, m *manifest.Manifest) (string, error) {
|
||||
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
||||
if sessionID == "" && m != nil {
|
||||
sessionID = strings.TrimSpace(m.SessionID)
|
||||
}
|
||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||
if campaign == "" && m != nil {
|
||||
campaign = strings.TrimSpace(m.Campaign)
|
||||
}
|
||||
if sessionID == "" {
|
||||
return "", fmt.Errorf("session id is required")
|
||||
}
|
||||
if campaign == "" {
|
||||
return "", fmt.Errorf("campaign is required")
|
||||
}
|
||||
return filepath.Clean(artifacts.SessionWorkDirForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID)), nil
|
||||
}
|
||||
|
||||
func archiveRunPrefix(env *Env, m *manifest.Manifest) (string, error) {
|
||||
@@ -331,9 +342,10 @@ func archiveBucket(env *Env, m *manifest.Manifest) string {
|
||||
return strings.TrimSpace(env.Config.Pipeline.Storage.S3.Bucket)
|
||||
}
|
||||
|
||||
func resolveArchivePromotions(workDirs []string, rules []config.ArchivePromotionRule) ([]archivePromotion, error) {
|
||||
if len(workDirs) == 0 {
|
||||
return nil, fmt.Errorf("at least one workdir candidate is required")
|
||||
func resolveArchivePromotions(sessionRoot string, rules []config.ArchivePromotionRule) ([]archivePromotion, error) {
|
||||
sessionRoot = filepath.Clean(strings.TrimSpace(sessionRoot))
|
||||
if sessionRoot == "" {
|
||||
return nil, fmt.Errorf("session root is required")
|
||||
}
|
||||
out := make([]archivePromotion, 0, len(rules))
|
||||
for _, rule := range rules {
|
||||
@@ -341,28 +353,16 @@ func resolveArchivePromotions(workDirs []string, rules []config.ArchivePromotion
|
||||
to := strings.TrimSpace(rule.To)
|
||||
required := rule.Required == nil || *rule.Required
|
||||
|
||||
var (
|
||||
localPath string
|
||||
exists bool
|
||||
)
|
||||
for _, candidateRoot := range workDirs {
|
||||
resolvedPath, err := resolveWorkDirRelativePath(candidateRoot, from)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("promotion from %q: %w", from, err)
|
||||
}
|
||||
info, err := os.Stat(resolvedPath)
|
||||
if err == nil && !info.IsDir() {
|
||||
localPath = resolvedPath
|
||||
exists = true
|
||||
break
|
||||
}
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return nil, fmt.Errorf("promotion source %q: %w", from, err)
|
||||
}
|
||||
if localPath == "" {
|
||||
localPath = resolvedPath
|
||||
}
|
||||
resolvedPath, err := resolveWorkDirRelativePath(sessionRoot, from)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("promotion from %q: %w", from, err)
|
||||
}
|
||||
info, err := os.Stat(resolvedPath)
|
||||
exists := err == nil && !info.IsDir()
|
||||
if err != nil && !os.IsNotExist(err) {
|
||||
return nil, fmt.Errorf("promotion source %q: %w", from, err)
|
||||
}
|
||||
localPath := resolvedPath
|
||||
|
||||
out = append(out, archivePromotion{
|
||||
From: from,
|
||||
@@ -375,33 +375,6 @@ func resolveArchivePromotions(workDirs []string, rules []config.ArchivePromotion
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func archiveSourceWorkDirs(env *Env, m *manifest.Manifest, runWorkDir string) []string {
|
||||
candidates := make([]string, 0, 2)
|
||||
if strings.TrimSpace(runWorkDir) != "" {
|
||||
candidates = append(candidates, filepath.Clean(runWorkDir))
|
||||
}
|
||||
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
||||
if sessionID == "" && m != nil {
|
||||
sessionID = strings.TrimSpace(m.SessionID)
|
||||
}
|
||||
if sessionID != "" {
|
||||
sessionWorkDir := filepath.Clean(artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, sessionID))
|
||||
if sessionWorkDir != "" {
|
||||
candidates = append(candidates, sessionWorkDir)
|
||||
}
|
||||
}
|
||||
seen := make(map[string]struct{}, len(candidates))
|
||||
out := make([]string, 0, len(candidates))
|
||||
for _, c := range candidates {
|
||||
if _, ok := seen[c]; ok {
|
||||
continue
|
||||
}
|
||||
seen[c] = struct{}{}
|
||||
out = append(out, c)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func resolveWorkDirRelativePath(workDir, rel string) (string, error) {
|
||||
rel = filepath.Clean(filepath.FromSlash(strings.TrimSpace(rel)))
|
||||
if rel == "." || rel == "" {
|
||||
@@ -420,42 +393,40 @@ func resolveWorkDirRelativePath(workDir, rel string) (string, error) {
|
||||
return cleanedFull, nil
|
||||
}
|
||||
|
||||
func collectArchiveRunFiles(workDir, manifestPath string) ([]archiveUploadFile, error) {
|
||||
func collectArchiveRunFiles(runRoot, manifestPath string) ([]archiveUploadFile, error) {
|
||||
files := make([]archiveUploadFile, 0, 64)
|
||||
|
||||
for _, dirName := range archiveRunUploadDirs {
|
||||
fullDir := filepath.Join(workDir, dirName)
|
||||
info, err := os.Stat(fullDir)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
continue
|
||||
}
|
||||
return nil, fmt.Errorf("stat %q: %w", fullDir, err)
|
||||
err := filepath.WalkDir(runRoot, func(path string, d fs.DirEntry, walkErr error) error {
|
||||
if walkErr != nil {
|
||||
return walkErr
|
||||
}
|
||||
if !info.IsDir() {
|
||||
continue
|
||||
}
|
||||
|
||||
if err := filepath.WalkDir(fullDir, func(path string, d fs.DirEntry, walkErr error) error {
|
||||
if walkErr != nil {
|
||||
return walkErr
|
||||
}
|
||||
if d.IsDir() {
|
||||
if d.IsDir() {
|
||||
if path == runRoot {
|
||||
return nil
|
||||
}
|
||||
rel, err := filepath.Rel(workDir, path)
|
||||
relDir, err := filepath.Rel(runRoot, path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("relative path from %q to %q: %w", workDir, path, err)
|
||||
return fmt.Errorf("relative dir from %q to %q: %w", runRoot, path, err)
|
||||
}
|
||||
relDir = filepath.ToSlash(relDir)
|
||||
// Preserve existing behavior: audio is not uploaded in archive run record.
|
||||
if relDir == "audio" || strings.HasPrefix(relDir, "audio/") {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
rel = filepath.ToSlash(rel)
|
||||
files = append(files, archiveUploadFile{
|
||||
RelativePath: rel,
|
||||
LocalPath: path,
|
||||
})
|
||||
return nil
|
||||
}); err != nil {
|
||||
return nil, fmt.Errorf("walk %q: %w", fullDir, err)
|
||||
}
|
||||
rel, err := filepath.Rel(runRoot, path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("relative path from %q to %q: %w", runRoot, path, err)
|
||||
}
|
||||
rel = filepath.ToSlash(rel)
|
||||
files = append(files, archiveUploadFile{
|
||||
RelativePath: rel,
|
||||
LocalPath: path,
|
||||
})
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("walk %q: %w", runRoot, err)
|
||||
}
|
||||
|
||||
manifestInfo, err := os.Stat(manifestPath)
|
||||
@@ -472,6 +443,14 @@ func collectArchiveRunFiles(workDir, manifestPath string) ([]archiveUploadFile,
|
||||
RelativePath: "manifest.json",
|
||||
LocalPath: manifestPath,
|
||||
})
|
||||
seen := map[string]archiveUploadFile{}
|
||||
for _, file := range files {
|
||||
seen[file.RelativePath] = file
|
||||
}
|
||||
files = files[:0]
|
||||
for _, file := range seen {
|
||||
files = append(files, file)
|
||||
}
|
||||
|
||||
sort.Slice(files, func(i, j int) bool {
|
||||
return files[i].RelativePath < files[j].RelativePath
|
||||
@@ -479,39 +458,30 @@ func collectArchiveRunFiles(workDir, manifestPath string) ([]archiveUploadFile,
|
||||
return files, nil
|
||||
}
|
||||
|
||||
func resolveArchiveManifestSource(env *Env, m *manifest.Manifest, workDir string) (string, error) {
|
||||
candidates := make([]string, 0, 3)
|
||||
candidates = append(candidates, filepath.Join(workDir, "manifest.json"))
|
||||
func resolveArchiveRunManifestSource(runRoot string) (string, error) {
|
||||
path := filepath.Join(filepath.Clean(runRoot), "manifest.json")
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return "", fmt.Errorf("manifest.json not found in run root %q", runRoot)
|
||||
}
|
||||
return "", fmt.Errorf("stat %q: %w", path, err)
|
||||
}
|
||||
if info.IsDir() {
|
||||
return "", fmt.Errorf("manifest path %q is a directory", path)
|
||||
}
|
||||
return path, nil
|
||||
}
|
||||
|
||||
sessionID := strings.TrimSpace(env.Config.Session.SessionID)
|
||||
if sessionID == "" && m != nil {
|
||||
sessionID = strings.TrimSpace(m.SessionID)
|
||||
func directoryExists(path string) (bool, error) {
|
||||
info, err := os.Stat(path)
|
||||
if err == nil {
|
||||
return info.IsDir(), nil
|
||||
}
|
||||
if sessionID != "" {
|
||||
sessionManifest := filepath.Join(artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, sessionID), "manifest.json")
|
||||
candidates = append(candidates, sessionManifest)
|
||||
if os.IsNotExist(err) {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
seen := make(map[string]struct{}, len(candidates))
|
||||
for _, candidate := range candidates {
|
||||
clean := filepath.Clean(candidate)
|
||||
if _, ok := seen[clean]; ok {
|
||||
continue
|
||||
}
|
||||
seen[clean] = struct{}{}
|
||||
info, err := os.Stat(clean)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
continue
|
||||
}
|
||||
return "", fmt.Errorf("stat %q: %w", clean, err)
|
||||
}
|
||||
if info.IsDir() {
|
||||
continue
|
||||
}
|
||||
return clean, nil
|
||||
}
|
||||
return "", fmt.Errorf("manifest.json not found in run workdir or session workdir")
|
||||
return false, err
|
||||
}
|
||||
|
||||
func writeCurrentManifestSnapshot(m *manifest.Manifest, archiveMetadata map[string]any) (string, error) {
|
||||
|
||||
@@ -73,14 +73,15 @@ func TestArchiveUploadsRunRecordPromotionsAndCurrentPointer(t *testing.T) {
|
||||
runPrefix := m.S3RunPrefix
|
||||
sessionPrefix := m.S3SessionPrefix
|
||||
wantRunUploads := []string{
|
||||
"artifacts/session_recap.md",
|
||||
"config/audita.generated.yml",
|
||||
"inputs/session.yml",
|
||||
"analyze/outputs/artifacts/session_recap.md",
|
||||
"merge/config/seriatim.generated.yml",
|
||||
"prepare/inputs/session.yml",
|
||||
"prepare/outputs/audio/speaker.flac",
|
||||
"logs/audita.stderr.log",
|
||||
"manifest.json",
|
||||
"reports/audita.report.json",
|
||||
"transcripts/raw/speaker.json",
|
||||
"transcripts/trimmed.json",
|
||||
"polish/reports/audita.report.json",
|
||||
"transcribe/outputs/transcripts/raw/speaker.json",
|
||||
"trim/outputs/transcripts/trimmed.json",
|
||||
}
|
||||
for _, rel := range wantRunUploads {
|
||||
key := runPrefix + rel
|
||||
@@ -132,12 +133,11 @@ func TestArchiveUploadsRunRecordPromotionsAndCurrentPointer(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestArchiveUsesCustomPromotionRules(t *testing.T) {
|
||||
env, m, workDir := archiveFixture(t)
|
||||
env, m, _ := archiveFixture(t)
|
||||
env.Config.Pipeline.Archive.PromoteArtifacts = []config.ArchivePromotionRule{
|
||||
{From: "transcripts/trimmed.json", To: "published/trimmed.json", Required: boolPtr(true)},
|
||||
{From: "artifacts/session_recap.md", To: "published/recap.md", Required: boolPtr(true)},
|
||||
}
|
||||
writeStageTestFile(t, filepath.Join(workDir, "published", "ignored.txt"), "ignore\n")
|
||||
|
||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
@@ -183,26 +183,18 @@ func TestArchiveFailsWhenRequiredPromotionMissing(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestArchivePromotionFallsBackToSessionWorkDir(t *testing.T) {
|
||||
env, m, runWorkDir := archiveFixture(t)
|
||||
sessionWorkDir := artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, m.SessionID)
|
||||
sessionTrimmed := filepath.Join(sessionWorkDir, "transcripts", "trimmed.json")
|
||||
func TestArchiveFailsWhenCanonicalRunRootMissing(t *testing.T) {
|
||||
env, m, runRoot := archiveFixture(t)
|
||||
if err := os.RemoveAll(runRoot); err != nil {
|
||||
t.Fatalf("remove run root: %v", err)
|
||||
}
|
||||
|
||||
if err := os.Remove(filepath.Join(runWorkDir, "transcripts", "trimmed.json")); err != nil {
|
||||
t.Fatalf("remove run scoped trimmed transcript: %v", err)
|
||||
_, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err == nil {
|
||||
t.Fatal("expected missing run-root error, got nil")
|
||||
}
|
||||
writeStageTestFile(t, sessionTrimmed, "{}\n")
|
||||
|
||||
result, err := archiveStage{}.Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if result.Metadata["promoted_files_uploaded"] != 2 {
|
||||
t.Fatalf("metadata promoted_files_uploaded = %#v, want 2", result.Metadata["promoted_files_uploaded"])
|
||||
}
|
||||
fake := env.ObjectStore.(*storage.FakeBackend)
|
||||
if _, ok := fake.Objects[m.S3SessionPrefix+"transcripts/trimmed.json"]; !ok {
|
||||
t.Fatalf("missing promoted trimmed key from session fallback")
|
||||
if !strings.Contains(err.Error(), "run root not found") {
|
||||
t.Fatalf("error = %v, want missing run-root error", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -264,22 +256,27 @@ func archiveFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
|
||||
runID := "20260516T010203Z-1a2b3c4d"
|
||||
campaign := "forsaken"
|
||||
sessionID := "2026-04-19"
|
||||
workDir := filepath.Join(root, "work", campaign, sessionID, runID)
|
||||
sessionRoot := artifacts.SessionWorkDirForCampaign(root, campaign, sessionID)
|
||||
runRoot := artifacts.SessionRunRootForCampaign(root, campaign, sessionID, runID)
|
||||
|
||||
writeStageTestFile(t, filepath.Join(workDir, "inputs", "session.yml"), "session_id: 2026-04-19\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "transcripts", "trimmed.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "artifacts", "session_recap.md"), "# recap\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "reports", "audita.report.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "config", "audita.generated.yml"), "key: value\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "logs", "audita.stderr.log"), "stderr\n")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "audio", "speaker.flac"), "flac")
|
||||
writeStageTestFile(t, filepath.Join(workDir, "manifest.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(sessionRoot, "transcripts", "trimmed.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(sessionRoot, "artifacts", "session_recap.md"), "# recap\n")
|
||||
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "prepare", "inputs", "session.yml"), "session_id: 2026-04-19\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "prepare", "outputs", "audio", "speaker.flac"), "flac\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "trim", "outputs", "transcripts", "trimmed.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "analyze", "outputs", "artifacts", "session_recap.md"), "# recap\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "polish", "reports", "audita.report.json"), "{}\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "merge", "config", "seriatim.generated.yml"), "key: value\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "logs", "audita.stderr.log"), "stderr\n")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "audio", "speaker.flac"), "flac")
|
||||
writeStageTestFile(t, filepath.Join(runRoot, "manifest.json"), "{}\n")
|
||||
|
||||
m := manifest.New(sessionID, time.Date(2026, 5, 16, 1, 2, 3, 0, time.UTC))
|
||||
m.Campaign = campaign
|
||||
m.RunID = runID
|
||||
m.LocalWorkDir = workDir
|
||||
m.LocalWorkDir = runRoot
|
||||
m.S3Bucket = "my-dnd-archive"
|
||||
m.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", campaign, sessionID)
|
||||
m.S3RunPrefix = artifacts.S3RunPrefix(m.S3SessionPrefix, runID)
|
||||
@@ -313,7 +310,7 @@ func archiveFixture(t *testing.T) (*Env, *manifest.Manifest, string) {
|
||||
},
|
||||
ObjectStore: &storage.FakeBackend{},
|
||||
}
|
||||
return env, m, workDir
|
||||
return env, m, runRoot
|
||||
}
|
||||
|
||||
type promotionFailingStore struct {
|
||||
|
||||
@@ -57,7 +57,11 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
|
||||
return nil, fmt.Errorf("merge: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "merge")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("merge: resolve run-stage layout: %w", err)
|
||||
}
|
||||
|
||||
inputs, err := discoverRawTranscripts(m, paths)
|
||||
if err != nil {
|
||||
@@ -81,13 +85,29 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
|
||||
return nil, fmt.Errorf("merge: %w", err)
|
||||
}
|
||||
|
||||
mergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
|
||||
reportPath := filepath.Join(paths.ArtifactsDir, "seriatim.report.json")
|
||||
canonicalMergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
|
||||
mergedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalMergedPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("merge: resolve run-local merged transcript path: %w", err)
|
||||
}
|
||||
canonicalReportPath := filepath.Join(paths.ArtifactsDir, "seriatim.report.json")
|
||||
reportPath := canonicalReportPath
|
||||
if runLayout.Enabled {
|
||||
reportPath, err = runLocalPathForCanonical(runLayout, paths, canonicalReportPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("merge: resolve run-local report path: %w", err)
|
||||
}
|
||||
}
|
||||
stdoutPath := filepath.Join(paths.LogsDir, "seriatim.stdout.log")
|
||||
stderrPath := filepath.Join(paths.LogsDir, "seriatim.stderr.log")
|
||||
genCfgPath := filepath.Join(paths.ConfigDir, "seriatim.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
stdoutPath = filepath.Join(runLayout.LogsDir, "seriatim.stdout.log")
|
||||
stderrPath = filepath.Join(runLayout.LogsDir, "seriatim.stderr.log")
|
||||
genCfgPath = filepath.Join(runLayout.ConfigDir, "seriatim.generated.yml")
|
||||
}
|
||||
|
||||
normalizedInputs, normalizeLogs, normalizeConfigs, normalizeMeta, err := normalizeMergeInputs(ctx, env, inputs, paths)
|
||||
normalizedInputs, normalizeLogs, normalizeConfigs, normalizeMeta, err := normalizeMergeInputs(ctx, env, inputs, paths, runLayout)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -130,25 +150,35 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
|
||||
}
|
||||
}
|
||||
|
||||
outputs := []artifacts.Ref{{
|
||||
Kind: "transcript_merged",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalMergedPath,
|
||||
}}
|
||||
promotedMerged, err := promoteRunLocalOutput(env.ArtifactStore, finalMergedPath, canonicalMergedPath, artifacts.Ref{
|
||||
Kind: "transcript_merged",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("merge: promote merged transcript: %w", err)
|
||||
}
|
||||
outputs := []artifacts.Ref{promotedMerged}
|
||||
if reportEnabled {
|
||||
outputs = append(outputs, artifacts.Ref{
|
||||
Kind: "seriatim_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalReportPath,
|
||||
promotedReport, err := promoteRunLocalOutput(env.ArtifactStore, finalReportPath, canonicalReportPath, artifacts.Ref{
|
||||
Kind: "seriatim_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("merge: promote report: %w", err)
|
||||
}
|
||||
outputs = append(outputs, promotedReport)
|
||||
}
|
||||
|
||||
coalesceGap := any(nil)
|
||||
if env.Config.Pipeline.Seriatim.CoalesceGap != nil {
|
||||
coalesceGap = *env.Config.Pipeline.Seriatim.CoalesceGap
|
||||
}
|
||||
reportCanonicalPath := ""
|
||||
if reportEnabled {
|
||||
reportCanonicalPath = canonicalReportPath
|
||||
}
|
||||
|
||||
meta := map[string]any{
|
||||
"stage": "merge",
|
||||
@@ -160,8 +190,10 @@ func (mergeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Sta
|
||||
"output_schema": env.Config.Pipeline.Seriatim.OutputSchema,
|
||||
"coalesce_gap": coalesceGap,
|
||||
"report_enabled": reportEnabled,
|
||||
"output_path": finalMergedPath,
|
||||
"report_path": finalReportPath,
|
||||
"run_output_path": finalMergedPath,
|
||||
"output_path": canonicalMergedPath,
|
||||
"run_report_path": finalReportPath,
|
||||
"report_path": reportCanonicalPath,
|
||||
"timeout": env.Config.Pipeline.Seriatim.Timeout,
|
||||
"binary": env.Config.Pipeline.Seriatim.Binary,
|
||||
"adapter_duration_ms": res.Duration.Milliseconds(),
|
||||
@@ -201,12 +233,21 @@ type normalizeMergeInputMeta struct {
|
||||
AdapterOutputPath string `json:"adapter_output_path,omitempty"`
|
||||
}
|
||||
|
||||
func normalizeMergeInputs(ctx context.Context, env *Env, rawInputs []string, paths artifacts.SessionPaths) ([]string, []string, []string, []normalizeMergeInputMeta, error) {
|
||||
func normalizeMergeInputs(
|
||||
ctx context.Context,
|
||||
env *Env,
|
||||
rawInputs []string,
|
||||
paths artifacts.SessionPaths,
|
||||
runLayout runStageLayout,
|
||||
) ([]string, []string, []string, []normalizeMergeInputMeta, error) {
|
||||
normalizedInputs := make([]string, 0, len(rawInputs))
|
||||
logs := make([]string, 0, len(rawInputs)*2)
|
||||
configs := make([]string, 0, len(rawInputs))
|
||||
meta := make([]normalizeMergeInputMeta, 0, len(rawInputs))
|
||||
normalizedDir := filepath.Join(paths.TranscriptsRawDir, "normalized")
|
||||
if runLayout.Enabled {
|
||||
normalizedDir = filepath.Join(runLayout.ScratchDir, "normalized")
|
||||
}
|
||||
if err := os.MkdirAll(normalizedDir, 0o755); err != nil {
|
||||
return nil, nil, nil, nil, fmt.Errorf("merge: ensure normalized transcripts directory %q: %w", normalizedDir, err)
|
||||
}
|
||||
@@ -226,6 +267,11 @@ func normalizeMergeInputs(ctx context.Context, env *Env, rawInputs []string, pat
|
||||
stdoutPath := filepath.Join(paths.LogsDir, "seriatim.normalize."+base+".stdout.log")
|
||||
stderrPath := filepath.Join(paths.LogsDir, "seriatim.normalize."+base+".stderr.log")
|
||||
cfgPath := filepath.Join(paths.ConfigDir, "seriatim.normalize."+base+".generated.yml")
|
||||
if runLayout.Enabled {
|
||||
stdoutPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize."+base+".stdout.log")
|
||||
stderrPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize."+base+".stderr.log")
|
||||
cfgPath = filepath.Join(runLayout.ConfigDir, "seriatim.normalize."+base+".generated.yml")
|
||||
}
|
||||
|
||||
req := seriatim.NormalizeRequest{
|
||||
Binary: env.Config.Pipeline.Seriatim.Binary,
|
||||
|
||||
@@ -33,7 +33,7 @@ func (r *normalizeDirAssertingRunner) Normalize(ctx context.Context, req seriati
|
||||
|
||||
func TestMergeStageMergesRawTranscriptsAndRecordsMetadata(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
inA := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
inB := filepath.Join(paths.TranscriptsRawDir, "bob.json")
|
||||
@@ -122,7 +122,7 @@ func TestMergeStageMergesRawTranscriptsAndRecordsMetadata(t *testing.T) {
|
||||
|
||||
func TestMergeStageFailsWhenNoRawTranscripts(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
env.Seriatim = &seriatim.FakeRunner{}
|
||||
@@ -138,7 +138,7 @@ func TestMergeStageFailsWhenNoRawTranscripts(t *testing.T) {
|
||||
|
||||
func TestMergeStageFailsOnInvalidInputJSON(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsRawDir, "alice.json"), "not-json")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
@@ -155,7 +155,7 @@ func TestMergeStageFailsOnInvalidInputJSON(t *testing.T) {
|
||||
|
||||
func TestMergeStageFailsWhenAdapterFails(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsRawDir, "alice.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
@@ -172,7 +172,7 @@ func TestMergeStageFailsWhenAdapterFails(t *testing.T) {
|
||||
|
||||
func TestMergeStageFallsBackToRawDirectoryWhenTranscribeOutputsMissing(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsRawDir, "alice.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
@@ -193,7 +193,7 @@ func TestMergeStageFallsBackToRawDirectoryWhenTranscribeOutputsMissing(t *testin
|
||||
|
||||
func TestMergeStageResolvesWorkspaceQualifiedManifestOutputsWithoutDuplication(t *testing.T) {
|
||||
env, m := setupMergeEnvWithRelativeWorkspaceRoot(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
rawPath := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, rawPath, `{"segments":[]}`)
|
||||
@@ -224,7 +224,7 @@ func TestMergeStageResolvesWorkspaceQualifiedManifestOutputsWithoutDuplication(t
|
||||
|
||||
func TestMergeStageCreatesNormalizedRawDirectoryBeforeNormalize(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
rawPath := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, rawPath, `{"segments":[]}`)
|
||||
@@ -248,7 +248,7 @@ func TestMergeStageCreatesNormalizedRawDirectoryBeforeNormalize(t *testing.T) {
|
||||
|
||||
func TestMergeStageFailsWhenNormalizeAdapterFails(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
in := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, in, `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
@@ -266,7 +266,7 @@ func TestMergeStageFailsWhenNormalizeAdapterFails(t *testing.T) {
|
||||
|
||||
func TestMergeStageFailsWhenNormalizedOutputInvalid(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
in := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, in, `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
@@ -289,9 +289,42 @@ func TestMergeStageFailsWhenNormalizedOutputInvalid(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
in := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, in, `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "speakers.yml"), "match: []\n")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "autocorrect.yml"), "rules: []\n")
|
||||
|
||||
fake := &seriatim.FakeRunner{}
|
||||
env.Seriatim = fake
|
||||
result, err := (mergeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("merge.Run() error = %v", err)
|
||||
}
|
||||
if len(fake.Requests) != 1 {
|
||||
t.Fatalf("requests = %d, want 1", len(fake.Requests))
|
||||
}
|
||||
req := fake.Requests[0]
|
||||
if !strings.Contains(req.OutputMergedTranscriptPath, filepath.Join("runs", m.RunID, "merge", "outputs")) {
|
||||
t.Fatalf("run output path = %q, want run-local path", req.OutputMergedTranscriptPath)
|
||||
}
|
||||
if len(result.Outputs) == 0 {
|
||||
t.Fatalf("outputs = %#v, want promoted outputs", result.Outputs)
|
||||
}
|
||||
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeStageResolvesSessionRelativeManifestOutputs(t *testing.T) {
|
||||
env, m := setupMergeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
rawPath := filepath.Join(paths.TranscriptsRawDir, "alice.json")
|
||||
writeFile(t, rawPath, `{"segments":[]}`)
|
||||
@@ -373,7 +406,7 @@ func setupMergeEnvWithWorkspace(t *testing.T, workspace string) (*Env, *manifest
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(workspace)
|
||||
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
|
||||
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
return &Env{
|
||||
|
||||
@@ -54,7 +54,11 @@ func (normalizeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (
|
||||
return nil, fmt.Errorf("normalize: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "normalize")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve run-stage layout: %w", err)
|
||||
}
|
||||
processedPath, processedSource, err := discoverProcessedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve processed transcript: %w", err)
|
||||
@@ -67,18 +71,34 @@ func (normalizeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (
|
||||
}
|
||||
|
||||
normalizeCfg := normalizeConfigOrDefault(env.Config.Pipeline.Normalize)
|
||||
normalizedPath, err := resolveScriptoriumOutputPath(paths, normalizeCfg.OutputPath)
|
||||
canonicalNormalizedPath, err := resolveScriptoriumOutputPath(paths, normalizeCfg.OutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve normalized output path: %w", err)
|
||||
}
|
||||
normalizedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalNormalizedPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve run-local normalized output path: %w", err)
|
||||
}
|
||||
reportEnabled := normalizeCfg.Report != nil && *normalizeCfg.Report
|
||||
reportPath := ""
|
||||
canonicalReportPath := filepath.Join(paths.ArtifactsDir, "seriatim.normalize.report.json")
|
||||
if reportEnabled {
|
||||
reportPath = filepath.Join(paths.ArtifactsDir, "seriatim.normalize.report.json")
|
||||
reportPath = canonicalReportPath
|
||||
if runLayout.Enabled {
|
||||
reportPath, err = runLocalPathForCanonical(runLayout, paths, canonicalReportPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve run-local report path: %w", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
stdoutPath := filepath.Join(paths.LogsDir, "seriatim.normalize.stdout.log")
|
||||
stderrPath := filepath.Join(paths.LogsDir, "seriatim.normalize.stderr.log")
|
||||
generatedConfigPath := filepath.Join(paths.ConfigDir, "seriatim.normalize.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
stdoutPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize.stdout.log")
|
||||
stderrPath = filepath.Join(runLayout.LogsDir, "seriatim.normalize.stderr.log")
|
||||
generatedConfigPath = filepath.Join(runLayout.ConfigDir, "seriatim.normalize.generated.yml")
|
||||
}
|
||||
timeout, err := resolveTrimSeriatimTimeout(env.Config.Pipeline.Seriatim.Timeout)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: resolve seriatim timeout: %w", err)
|
||||
@@ -113,44 +133,56 @@ func (normalizeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (
|
||||
}
|
||||
}
|
||||
|
||||
outputs := []artifacts.Ref{{
|
||||
Kind: "transcript_normalized",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalNormalizedPath,
|
||||
}}
|
||||
promotedNormalized, err := promoteRunLocalOutput(env.ArtifactStore, finalNormalizedPath, canonicalNormalizedPath, artifacts.Ref{
|
||||
Kind: "transcript_normalized",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: promote normalized transcript: %w", err)
|
||||
}
|
||||
outputs := []artifacts.Ref{promotedNormalized}
|
||||
if reportEnabled {
|
||||
outputs = append(outputs, artifacts.Ref{
|
||||
Kind: "seriatim_normalize_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalReportPath,
|
||||
promotedReport, err := promoteRunLocalOutput(env.ArtifactStore, finalReportPath, canonicalReportPath, artifacts.Ref{
|
||||
Kind: "seriatim_normalize_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("normalize: promote report: %w", err)
|
||||
}
|
||||
outputs = append(outputs, promotedReport)
|
||||
}
|
||||
reportCanonicalPath := ""
|
||||
if reportEnabled {
|
||||
reportCanonicalPath = canonicalReportPath
|
||||
}
|
||||
|
||||
meta := map[string]any{
|
||||
"stage": "normalize",
|
||||
"processed_transcript_path": processedPath,
|
||||
"processed_transcript_source": processedSource,
|
||||
"normalized_transcript_path": finalNormalizedPath,
|
||||
"normalized_transcript_source": "stage.normalize.output",
|
||||
"output_schema": normalizeCfg.OutputSchema,
|
||||
"report_enabled": reportEnabled,
|
||||
"report_path": finalReportPath,
|
||||
"timeout": env.Config.Pipeline.Seriatim.Timeout,
|
||||
"binary": env.Config.Pipeline.Seriatim.Binary,
|
||||
"stdout_log_path": stdoutPath,
|
||||
"stderr_log_path": stderrPath,
|
||||
"generated_config_path": generatedConfigPath,
|
||||
"adapter_duration_ms": res.Duration.Milliseconds(),
|
||||
"adapter_exit_code": res.ExitCode,
|
||||
"adapter_invoked_binary": res.InvokedBinary,
|
||||
"adapter_output_schema": res.OutputSchema,
|
||||
"adapter_output_path": res.OutputNormalizedPath,
|
||||
"adapter_report_path": res.ReportPath,
|
||||
"adapter_generated_config": res.GeneratedConfigPath,
|
||||
"adapter_stdout_log_path": res.StdoutLogPath,
|
||||
"adapter_stderr_log_path": res.StderrLogPath,
|
||||
"stage": "normalize",
|
||||
"processed_transcript_path": processedPath,
|
||||
"processed_transcript_source": processedSource,
|
||||
"run_normalized_transcript_path": finalNormalizedPath,
|
||||
"normalized_transcript_path": canonicalNormalizedPath,
|
||||
"normalized_transcript_source": "stage.normalize.output",
|
||||
"output_schema": normalizeCfg.OutputSchema,
|
||||
"report_enabled": reportEnabled,
|
||||
"run_report_path": finalReportPath,
|
||||
"report_path": reportCanonicalPath,
|
||||
"timeout": env.Config.Pipeline.Seriatim.Timeout,
|
||||
"binary": env.Config.Pipeline.Seriatim.Binary,
|
||||
"stdout_log_path": stdoutPath,
|
||||
"stderr_log_path": stderrPath,
|
||||
"generated_config_path": generatedConfigPath,
|
||||
"adapter_duration_ms": res.Duration.Milliseconds(),
|
||||
"adapter_exit_code": res.ExitCode,
|
||||
"adapter_invoked_binary": res.InvokedBinary,
|
||||
"adapter_output_schema": res.OutputSchema,
|
||||
"adapter_output_path": res.OutputNormalizedPath,
|
||||
"adapter_report_path": res.ReportPath,
|
||||
"adapter_generated_config": res.GeneratedConfigPath,
|
||||
"adapter_stdout_log_path": res.StdoutLogPath,
|
||||
"adapter_stderr_log_path": res.StderrLogPath,
|
||||
}
|
||||
if res.Metadata != nil {
|
||||
meta["adapter_metadata"] = res.Metadata
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
|
||||
func TestNormalizeStageConsumesProcessedTranscriptFromManifest(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
manifestProcessed := filepath.Join(paths.ArtifactsDir, "processed.from-manifest.json")
|
||||
writeFile(t, manifestProcessed, `{"segments":[{"id":10}]}`)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":99}]}`)
|
||||
@@ -43,7 +43,7 @@ func TestNormalizeStageConsumesProcessedTranscriptFromManifest(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageFallsBackToProcessedTranscriptPath(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
fallback := filepath.Join(paths.TranscriptsDir, "processed.json")
|
||||
writeFile(t, fallback, `{"segments":[{"id":1}]}`)
|
||||
|
||||
@@ -72,7 +72,7 @@ func TestNormalizeStageFailsWhenProcessedTranscriptMissing(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
|
||||
env, m, _ := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), "not-json")
|
||||
|
||||
_, err := (normalizeStage{}).Run(context.Background(), env, m)
|
||||
@@ -86,7 +86,7 @@ func TestNormalizeStageFailsWhenProcessedTranscriptInvalidJSON(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageFailsWhenProcessedTranscriptMissingSegments(t *testing.T) {
|
||||
env, m, _ := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"schema":"audita.processed.v1"}`)
|
||||
|
||||
_, err := (normalizeStage{}).Run(context.Background(), env, m)
|
||||
@@ -100,7 +100,7 @@ func TestNormalizeStageFailsWhenProcessedTranscriptMissingSegments(t *testing.T)
|
||||
|
||||
func TestNormalizeStagePassesConfiguredOutputSchemaToAdapter(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
env.Config.Pipeline.Normalize.OutputSchema = "seriatim-full"
|
||||
|
||||
@@ -121,7 +121,7 @@ func TestNormalizeStagePassesConfiguredOutputSchemaToAdapter(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageRecordsNormalizedTranscriptOutputKind(t *testing.T) {
|
||||
env, m, _ := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
|
||||
result, err := (normalizeStage{}).Run(context.Background(), env, m)
|
||||
@@ -138,7 +138,7 @@ func TestNormalizeStageRecordsNormalizedTranscriptOutputKind(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageRecordsReportLogAndGeneratedConfigRefs(t *testing.T) {
|
||||
env, m, _ := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
report := true
|
||||
env.Config.Pipeline.Normalize.Report = &report
|
||||
@@ -164,7 +164,7 @@ func TestNormalizeStageRecordsReportLogAndGeneratedConfigRefs(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageFailsWhenAdapterReturnsError(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
ser.NormalizeErr = errors.New("normalize failed")
|
||||
|
||||
@@ -179,7 +179,7 @@ func TestNormalizeStageFailsWhenAdapterReturnsError(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageFailsWhenNormalizedOutputInvalid(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
badOutput := filepath.Join(paths.TranscriptsDir, "normalized.bad.json")
|
||||
writeFile(t, badOutput, "not-json")
|
||||
@@ -196,7 +196,7 @@ func TestNormalizeStageFailsWhenNormalizedOutputInvalid(t *testing.T) {
|
||||
|
||||
func TestNormalizeStageReportEnabledFailsWhenReportMissing(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
report := true
|
||||
env.Config.Pipeline.Normalize.Report = &report
|
||||
@@ -211,6 +211,33 @@ func TestNormalizeStageReportEnabledFailsWhenReportMissing(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
||||
env, m, ser := setupNormalizeEnv(t)
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "processed.json"), `{"segments":[{"id":1}]}`)
|
||||
|
||||
result, err := (normalizeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("normalize.Run() error = %v", err)
|
||||
}
|
||||
if len(ser.NormalizeRequests) != 1 {
|
||||
t.Fatalf("normalize requests = %d, want 1", len(ser.NormalizeRequests))
|
||||
}
|
||||
req := ser.NormalizeRequests[0]
|
||||
if !strings.Contains(req.OutputNormalizedPath, filepath.Join("runs", m.RunID, "normalize", "outputs")) {
|
||||
t.Fatalf("run output path = %q, want run-local path", req.OutputNormalizedPath)
|
||||
}
|
||||
if len(result.Outputs) == 0 {
|
||||
t.Fatalf("outputs = %#v, want promoted outputs", result.Outputs)
|
||||
}
|
||||
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
|
||||
}
|
||||
}
|
||||
|
||||
func setupNormalizeEnv(t *testing.T) (*Env, *manifest.Manifest, *seriatim.FakeRunner) {
|
||||
t.Helper()
|
||||
workspace := t.TempDir()
|
||||
@@ -243,7 +270,7 @@ func setupNormalizeEnv(t *testing.T) (*Env, *manifest.Manifest, *seriatim.FakeRu
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(workspace)
|
||||
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
|
||||
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
|
||||
|
||||
@@ -64,6 +64,7 @@ func TestStagesReturnExpectedMetadata(t *testing.T) {
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
Inputs: config.SessionInputsConfig{
|
||||
AudioDir: "./audio",
|
||||
SpeakersFile: "./speakers.yml",
|
||||
@@ -88,7 +89,7 @@ func TestStagesReturnExpectedMetadata(t *testing.T) {
|
||||
}
|
||||
m.RunID = "20260516T000000Z-abcdef12"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.LocalWorkDir = filepath.Join(root, "work", "sample-campaign", "2026-05-03", m.RunID)
|
||||
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(root, "sample-campaign", "2026-05-03", m.RunID)
|
||||
m.S3RunPrefix = "dnd/campaigns/sample-campaign/sessions/2026-05-03/runs/" + m.RunID + "/"
|
||||
m.S3Bucket = "my-dnd-archive"
|
||||
if err := os.MkdirAll(filepath.Join(m.LocalWorkDir, "inputs"), 0o755); err != nil {
|
||||
@@ -224,7 +225,7 @@ func TestPlaceholderAdapterErrorPropagation(t *testing.T) {
|
||||
|
||||
root := t.TempDir()
|
||||
store := artifacts.NewLocalStore(root)
|
||||
_, err := store.EnsureLayout("2026-05-03")
|
||||
_, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03")
|
||||
if err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
|
||||
@@ -55,7 +55,11 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
||||
return nil, fmt.Errorf("polish: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "polish")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: resolve run-stage layout: %w", err)
|
||||
}
|
||||
mergedPath, source, err := discoverMergedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: resolve merged transcript: %w", err)
|
||||
@@ -72,12 +76,29 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
||||
return nil, fmt.Errorf("polish: %w", err)
|
||||
}
|
||||
|
||||
processedPath := filepath.Join(paths.TranscriptsDir, "processed.json")
|
||||
reportPath := filepath.Join(paths.ArtifactsDir, "audita.report.json")
|
||||
canonicalProcessedPath := filepath.Join(paths.TranscriptsDir, "processed.json")
|
||||
processedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalProcessedPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: resolve run-local processed transcript path: %w", err)
|
||||
}
|
||||
canonicalReportPath := filepath.Join(paths.ArtifactsDir, "audita.report.json")
|
||||
reportPath := canonicalReportPath
|
||||
if runLayout.Enabled {
|
||||
reportPath, err = runLocalPathForCanonical(runLayout, paths, canonicalReportPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: resolve run-local report path: %w", err)
|
||||
}
|
||||
}
|
||||
workDir := filepath.Join(paths.ArtifactsDir, "audita-work")
|
||||
stdoutPath := filepath.Join(paths.LogsDir, "audita.stdout.log")
|
||||
stderrPath := filepath.Join(paths.LogsDir, "audita.stderr.log")
|
||||
generatedConfigPath := filepath.Join(paths.ConfigDir, "audita.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
workDir = filepath.Join(runLayout.ScratchDir, "audita-work")
|
||||
stdoutPath = filepath.Join(runLayout.LogsDir, "audita.stdout.log")
|
||||
stderrPath = filepath.Join(runLayout.LogsDir, "audita.stderr.log")
|
||||
generatedConfigPath = filepath.Join(runLayout.ConfigDir, "audita.generated.yml")
|
||||
}
|
||||
|
||||
reportEnabled := env.Config.Pipeline.Audita.Report != nil && *env.Config.Pipeline.Audita.Report
|
||||
req := audita.PolishRequest{
|
||||
@@ -128,19 +149,25 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
||||
}
|
||||
}
|
||||
|
||||
outputs := []artifacts.Ref{{
|
||||
Kind: "transcript_processed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalProcessedPath,
|
||||
}}
|
||||
promotedProcessed, err := promoteRunLocalOutput(env.ArtifactStore, finalProcessedPath, canonicalProcessedPath, artifacts.Ref{
|
||||
Kind: "transcript_processed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: promote processed transcript: %w", err)
|
||||
}
|
||||
outputs := []artifacts.Ref{promotedProcessed}
|
||||
if reportEnabled {
|
||||
outputs = append(outputs, artifacts.Ref{
|
||||
Kind: "audita_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalReportPath,
|
||||
promotedReport, err := promoteRunLocalOutput(env.ArtifactStore, finalReportPath, canonicalReportPath, artifacts.Ref{
|
||||
Kind: "audita_report",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("polish: promote report: %w", err)
|
||||
}
|
||||
outputs = append(outputs, promotedReport)
|
||||
}
|
||||
|
||||
var validationConcurrency any
|
||||
@@ -155,14 +182,20 @@ func (polishStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*St
|
||||
if env.Config.Pipeline.Audita.ProposalLLMConcurrency != nil {
|
||||
proposalLLMConcurrency = *env.Config.Pipeline.Audita.ProposalLLMConcurrency
|
||||
}
|
||||
reportCanonicalPath := ""
|
||||
if reportEnabled {
|
||||
reportCanonicalPath = canonicalReportPath
|
||||
}
|
||||
|
||||
meta := map[string]any{
|
||||
"stage": "polish",
|
||||
"merged_transcript_path": mergedPath,
|
||||
"merged_transcript_source": source,
|
||||
"glossary_path": glossaryPath,
|
||||
"output_path": finalProcessedPath,
|
||||
"report_path": finalReportPath,
|
||||
"run_output_path": finalProcessedPath,
|
||||
"output_path": canonicalProcessedPath,
|
||||
"run_report_path": finalReportPath,
|
||||
"report_path": reportCanonicalPath,
|
||||
"audita_work_dir": workDir,
|
||||
"report_enabled": reportEnabled,
|
||||
"modules": append([]string(nil), req.Modules...),
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
|
||||
func TestPolishStagePolishesMergedTranscriptAndRecordsMetadata(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
|
||||
mergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
|
||||
writeFile(t, mergedPath, `{"segments":[]}`)
|
||||
@@ -120,7 +120,7 @@ func TestPolishStagePolishesMergedTranscriptAndRecordsMetadata(t *testing.T) {
|
||||
|
||||
func TestPolishStageFallsBackToMergedTranscriptPath(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
mergedPath := filepath.Join(paths.TranscriptsDir, "merged.json")
|
||||
writeFile(t, mergedPath, `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
@@ -142,7 +142,7 @@ func TestPolishStageFallsBackToMergedTranscriptPath(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenMergedTranscriptMissing(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
env.Audita = &audita.FakeRunner{}
|
||||
|
||||
@@ -157,7 +157,7 @@ func TestPolishStageFailsWhenMergedTranscriptMissing(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenMergedTranscriptInvalidJSON(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), "not-json")
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
env.Audita = &audita.FakeRunner{}
|
||||
@@ -173,7 +173,7 @@ func TestPolishStageFailsWhenMergedTranscriptInvalidJSON(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenGlossaryMissing(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
|
||||
env.Audita = &audita.FakeRunner{}
|
||||
|
||||
@@ -188,7 +188,7 @@ func TestPolishStageFailsWhenGlossaryMissing(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenAdapterFails(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
env.Audita = &audita.FakeRunner{Err: errors.New("audita failed")}
|
||||
@@ -204,7 +204,7 @@ func TestPolishStageFailsWhenAdapterFails(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenProcessedOutputInvalid(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
badPath := filepath.Join(paths.TranscriptsDir, "processed.invalid.json")
|
||||
@@ -222,7 +222,7 @@ func TestPolishStageFailsWhenProcessedOutputInvalid(t *testing.T) {
|
||||
|
||||
func TestPolishStageFailsWhenReportInvalid(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
badReport := filepath.Join(paths.ArtifactsDir, "bad.report.json")
|
||||
@@ -238,6 +238,36 @@ func TestPolishStageFailsWhenReportInvalid(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestPolishStageUsesRunLocalPathsAndPromotesCanonical(t *testing.T) {
|
||||
env, m := setupPolishEnv(t)
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
writeFile(t, filepath.Join(paths.TranscriptsDir, "merged.json"), `{"segments":[]}`)
|
||||
writeFile(t, filepath.Join(paths.InputsDir, "glossary.yml"), "terms: []\n")
|
||||
|
||||
fake := &audita.FakeRunner{}
|
||||
env.Audita = fake
|
||||
result, err := (polishStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("polish.Run() error = %v", err)
|
||||
}
|
||||
if len(fake.Requests) != 1 {
|
||||
t.Fatalf("requests = %d, want 1", len(fake.Requests))
|
||||
}
|
||||
req := fake.Requests[0]
|
||||
if !strings.Contains(req.OutputProcessedPath, filepath.Join("runs", m.RunID, "polish", "outputs")) {
|
||||
t.Fatalf("run output path = %q, want run-local path", req.OutputProcessedPath)
|
||||
}
|
||||
if len(result.Outputs) == 0 {
|
||||
t.Fatalf("outputs = %#v, want promoted outputs", result.Outputs)
|
||||
}
|
||||
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
|
||||
}
|
||||
}
|
||||
|
||||
func setupPolishEnv(t *testing.T) (*Env, *manifest.Manifest) {
|
||||
t.Helper()
|
||||
workspace := t.TempDir()
|
||||
@@ -284,7 +314,7 @@ func setupPolishEnv(t *testing.T) (*Env, *manifest.Manifest) {
|
||||
}
|
||||
|
||||
store := artifacts.NewLocalStore(workspace)
|
||||
if _, err := store.EnsureLayout("2026-05-03"); err != nil {
|
||||
if _, err := store.EnsureLayoutFor("sample-campaign", "2026-05-03"); err != nil {
|
||||
t.Fatalf("EnsureLayout() error = %v", err)
|
||||
}
|
||||
return &Env{
|
||||
|
||||
@@ -54,7 +54,7 @@ func (prepareStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*S
|
||||
return nil, fmt.Errorf("prepare: session id is required")
|
||||
}
|
||||
|
||||
paths, err := env.ArtifactStore.EnsureLayout(sessionID)
|
||||
paths, err := ensureLayoutForEnv(env, sessionID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("prepare: ensure workdir layout: %w", err)
|
||||
}
|
||||
@@ -358,6 +358,10 @@ func pathsWorkDirForManifest(env *Env, m *manifest.Manifest, sessionID string) s
|
||||
if env == nil || env.Config == nil || env.Config.Pipeline == nil || env.Config.Session == nil {
|
||||
return ""
|
||||
}
|
||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||
if campaign == "" {
|
||||
return ""
|
||||
}
|
||||
if m != nil && strings.TrimSpace(m.LocalWorkDir) != "" {
|
||||
return strings.TrimSpace(m.LocalWorkDir)
|
||||
}
|
||||
@@ -366,9 +370,9 @@ func pathsWorkDirForManifest(env *Env, m *manifest.Manifest, sessionID string) s
|
||||
runID = strings.TrimSpace(m.RunID)
|
||||
}
|
||||
if runID != "" {
|
||||
return artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, env.Config.Session.Campaign, sessionID, runID)
|
||||
return artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID, runID)
|
||||
}
|
||||
return artifacts.SessionWorkDir(env.Config.Pipeline.Workspace.Root, sessionID)
|
||||
return artifacts.SessionWorkDirForCampaign(env.Config.Pipeline.Workspace.Root, campaign, sessionID)
|
||||
}
|
||||
|
||||
func resolvePath(baseDir, p string) (string, error) {
|
||||
|
||||
@@ -33,7 +33,7 @@ func TestPrepareStageExplicitAudioFiles(t *testing.T) {
|
||||
t.Fatalf("result metadata = %#v, want prepared=true", result)
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
for _, p := range []string{
|
||||
filepath.Join(paths.InputsDir, "session.yml"),
|
||||
filepath.Join(paths.InputsDir, "pipeline.resolved.yml"),
|
||||
@@ -76,7 +76,7 @@ func TestPrepareStageAudioDirEnumeration(t *testing.T) {
|
||||
t.Fatalf("audio_files_resolved = %#v, want 1", got)
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(m.SessionID)
|
||||
paths := sessionPathsForEnv(env, m.SessionID)
|
||||
if _, err := os.Stat(filepath.Join(paths.AudioDir, "a.flac")); err != nil {
|
||||
t.Fatalf("expected copied flac: %v", err)
|
||||
}
|
||||
@@ -160,7 +160,7 @@ func TestPrepareStageS3AudioDownloadAndMaterialization(t *testing.T) {
|
||||
env.Config.Pipeline.Spool = config.SpoolConfig{Root: filepath.Join(t.TempDir(), "spool")}
|
||||
env.Config.Pipeline.Storage.S3 = &config.StorageS3Config{Bucket: "my-dnd-archive", RootPrefix: "dnd"}
|
||||
m.RunID = "20260515T031522Z-a1b2c3d4"
|
||||
m.LocalWorkDir = artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
|
||||
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
|
||||
m.LocalSpoolDir = artifacts.SessionSpoolAudioDir(env.Config.Pipeline.Spool.Root, "forsaken", m.SessionID, m.RunID)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
@@ -256,7 +256,7 @@ func TestPrepareStageS3AudioFailures(t *testing.T) {
|
||||
env.Config.Pipeline.Spool = config.SpoolConfig{Root: filepath.Join(t.TempDir(), "spool")}
|
||||
env.Config.Pipeline.Storage.S3 = &config.StorageS3Config{Bucket: "my-dnd-archive", RootPrefix: "dnd"}
|
||||
m.RunID = "20260515T031522Z-a1b2c3d4"
|
||||
m.LocalWorkDir = artifacts.SessionRunWorkDir(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
|
||||
m.LocalWorkDir = artifacts.SessionRunRootForCampaign(env.Config.Pipeline.Workspace.Root, "forsaken", m.SessionID, m.RunID)
|
||||
m.LocalSpoolDir = artifacts.SessionSpoolAudioDir(env.Config.Pipeline.Spool.Root, "forsaken", m.SessionID, m.RunID)
|
||||
|
||||
fake := &storage.FakeBackend{}
|
||||
|
||||
136
internal/stage/run_local.go
Normal file
136
internal/stage/run_local.go
Normal file
@@ -0,0 +1,136 @@
|
||||
package stage
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||
)
|
||||
|
||||
type runStageLayout struct {
|
||||
Enabled bool
|
||||
Root string
|
||||
OutputsDir string
|
||||
LogsDir string
|
||||
ReportsDir string
|
||||
ConfigDir string
|
||||
ScratchDir string
|
||||
}
|
||||
|
||||
func resolveRunStageLayout(
|
||||
env *Env,
|
||||
m *manifest.Manifest,
|
||||
sessionPaths artifacts.SessionPaths,
|
||||
sessionID, stageName string,
|
||||
) (runStageLayout, error) {
|
||||
if env == nil || env.Config == nil || env.Config.Pipeline == nil {
|
||||
return runStageLayout{}, fmt.Errorf("stage environment pipeline config is required")
|
||||
}
|
||||
if strings.TrimSpace(sessionID) == "" {
|
||||
return runStageLayout{}, fmt.Errorf("session id is required")
|
||||
}
|
||||
stageName = strings.TrimSpace(stageName)
|
||||
if stageName == "" {
|
||||
return runStageLayout{}, fmt.Errorf("stage name is required")
|
||||
}
|
||||
|
||||
runID := ""
|
||||
if m != nil {
|
||||
runID = strings.TrimSpace(m.RunID)
|
||||
}
|
||||
campaign := strings.TrimSpace(env.Config.Session.Campaign)
|
||||
if campaign == "" && m != nil {
|
||||
campaign = strings.TrimSpace(m.Campaign)
|
||||
}
|
||||
|
||||
// Compatibility fallback for direct stage tests and older call paths
|
||||
// that execute a stage without a run id.
|
||||
if runID == "" || campaign == "" {
|
||||
return runStageLayout{}, nil
|
||||
}
|
||||
|
||||
root := artifacts.SessionRunStageDirForCampaign(
|
||||
env.Config.Pipeline.Workspace.Root,
|
||||
campaign,
|
||||
sessionID,
|
||||
runID,
|
||||
stageName,
|
||||
)
|
||||
layout := runStageLayout{
|
||||
Enabled: true,
|
||||
Root: root,
|
||||
OutputsDir: filepath.Join(root, "outputs"),
|
||||
LogsDir: filepath.Join(root, "logs"),
|
||||
ReportsDir: filepath.Join(root, "reports"),
|
||||
ConfigDir: filepath.Join(root, "config"),
|
||||
ScratchDir: filepath.Join(root, "scratch"),
|
||||
}
|
||||
for _, dir := range []string{
|
||||
layout.Root,
|
||||
layout.OutputsDir,
|
||||
layout.LogsDir,
|
||||
layout.ReportsDir,
|
||||
layout.ConfigDir,
|
||||
layout.ScratchDir,
|
||||
} {
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return runStageLayout{}, fmt.Errorf("create run-stage directory %q: %w", dir, err)
|
||||
}
|
||||
}
|
||||
return layout, nil
|
||||
}
|
||||
|
||||
func runLocalPathForCanonical(layout runStageLayout, sessionPaths artifacts.SessionPaths, canonicalPath string) (string, error) {
|
||||
if !layout.Enabled {
|
||||
return filepath.Clean(canonicalPath), nil
|
||||
}
|
||||
cleanCanonical := filepath.Clean(strings.TrimSpace(canonicalPath))
|
||||
if cleanCanonical == "" {
|
||||
return "", fmt.Errorf("canonical path is required")
|
||||
}
|
||||
rel, err := filepath.Rel(filepath.Clean(sessionPaths.Root), cleanCanonical)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("derive session-relative path for %q: %w", cleanCanonical, err)
|
||||
}
|
||||
rel = filepath.Clean(rel)
|
||||
if rel == "." || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
||||
return "", fmt.Errorf("canonical path %q is outside session root %q", cleanCanonical, sessionPaths.Root)
|
||||
}
|
||||
localPath := filepath.Join(layout.OutputsDir, rel)
|
||||
if err := os.MkdirAll(filepath.Dir(localPath), 0o755); err != nil {
|
||||
return "", fmt.Errorf("create run-local output parent for %q: %w", localPath, err)
|
||||
}
|
||||
return localPath, nil
|
||||
}
|
||||
|
||||
func promoteRunLocalOutput(
|
||||
store artifacts.Store,
|
||||
srcPath, canonicalPath string,
|
||||
ref artifacts.Ref,
|
||||
) (artifacts.Ref, error) {
|
||||
srcPath = filepath.Clean(strings.TrimSpace(srcPath))
|
||||
canonicalPath = filepath.Clean(strings.TrimSpace(canonicalPath))
|
||||
if srcPath == "" {
|
||||
return artifacts.Ref{}, fmt.Errorf("source path is required")
|
||||
}
|
||||
if canonicalPath == "" {
|
||||
return artifacts.Ref{}, fmt.Errorf("canonical destination path is required")
|
||||
}
|
||||
data, err := os.ReadFile(srcPath)
|
||||
if err != nil {
|
||||
return artifacts.Ref{}, fmt.Errorf("read run-local output %q: %w", srcPath, err)
|
||||
}
|
||||
if err := store.WriteFileAtomic(canonicalPath, data, 0o644); err != nil {
|
||||
return artifacts.Ref{}, fmt.Errorf("promote output to %q: %w", canonicalPath, err)
|
||||
}
|
||||
checksum, err := store.Checksum(canonicalPath)
|
||||
if err != nil {
|
||||
return artifacts.Ref{}, fmt.Errorf("checksum promoted output %q: %w", canonicalPath, err)
|
||||
}
|
||||
ref.AbsolutePath = canonicalPath
|
||||
ref.Checksum = checksum
|
||||
return ref, nil
|
||||
}
|
||||
35
internal/stage/run_local_test.go
Normal file
35
internal/stage/run_local_test.go
Normal file
@@ -0,0 +1,35 @@
|
||||
package stage
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
)
|
||||
|
||||
func TestRunLocalPathForCanonicalCreatesParentDirectories(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
sessionRoot := filepath.Join(root, "work", "dilfs", "2026-05-17")
|
||||
layout := runStageLayout{
|
||||
Enabled: true,
|
||||
OutputsDir: filepath.Join(sessionRoot, "runs", "run-1", "merge", "outputs"),
|
||||
}
|
||||
if err := os.MkdirAll(layout.OutputsDir, 0o755); err != nil {
|
||||
t.Fatalf("mkdir outputs dir: %v", err)
|
||||
}
|
||||
|
||||
canonical := filepath.Join(sessionRoot, "transcripts", "merged.json")
|
||||
got, err := runLocalPathForCanonical(layout, artifacts.SessionPaths{Root: sessionRoot}, canonical)
|
||||
if err != nil {
|
||||
t.Fatalf("runLocalPathForCanonical() error = %v", err)
|
||||
}
|
||||
|
||||
want := filepath.Join(layout.OutputsDir, "transcripts", "merged.json")
|
||||
if got != want {
|
||||
t.Fatalf("runLocalPathForCanonical() = %q, want %q", got, want)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Dir(got)); err != nil {
|
||||
t.Fatalf("expected run-local parent directory to exist: %v", err)
|
||||
}
|
||||
}
|
||||
23
internal/stage/session_paths.go
Normal file
23
internal/stage/session_paths.go
Normal file
@@ -0,0 +1,23 @@
|
||||
package stage
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||
)
|
||||
|
||||
func sessionPathsForEnv(env *Env, sessionID string) artifacts.SessionPaths {
|
||||
campaign := ""
|
||||
if env != nil && env.Config != nil && env.Config.Session != nil {
|
||||
campaign = strings.TrimSpace(env.Config.Session.Campaign)
|
||||
}
|
||||
return env.ArtifactStore.SessionPathsFor(campaign, sessionID)
|
||||
}
|
||||
|
||||
func ensureLayoutForEnv(env *Env, sessionID string) (artifacts.SessionPaths, error) {
|
||||
campaign := ""
|
||||
if env != nil && env.Config != nil && env.Config.Session != nil {
|
||||
campaign = strings.TrimSpace(env.Config.Session.Campaign)
|
||||
}
|
||||
return env.ArtifactStore.EnsureLayoutFor(campaign, sessionID)
|
||||
}
|
||||
@@ -18,10 +18,11 @@ import (
|
||||
|
||||
// Env is the shared dependency container visible to stages.
|
||||
type Env struct {
|
||||
Config *config.Config
|
||||
ArtifactStore artifacts.Store
|
||||
ManifestStore manifest.Store
|
||||
Logger *slog.Logger
|
||||
Config *config.Config
|
||||
SelectedAnalyzeArtifacts []string
|
||||
ArtifactStore artifacts.Store
|
||||
ManifestStore manifest.Store
|
||||
Logger *slog.Logger
|
||||
|
||||
WhisperX whisperx.Client
|
||||
Seriatim seriatim.Runner
|
||||
|
||||
@@ -55,7 +55,11 @@ func (transcribeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest)
|
||||
return nil, fmt.Errorf("transcribe: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "transcribe")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("transcribe: resolve run-stage layout: %w", err)
|
||||
}
|
||||
audioFiles, err := discoverPreparedAudio(m, paths.AudioDir)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("transcribe: resolve audio inputs: %w", err)
|
||||
@@ -88,10 +92,15 @@ func (transcribeStage) Run(ctx context.Context, env *Env, m *manifest.Manifest)
|
||||
return nil, fmt.Errorf("transcribe: duplicate speaker/audio basename %q from %q and %q", base, prev, audioPath)
|
||||
}
|
||||
seenSpeaker[base] = audioPath
|
||||
canonicalOut := filepath.Join(paths.TranscriptsRawDir, base+".json")
|
||||
runOut, err := runLocalPathForCanonical(runLayout, paths, canonicalOut)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("transcribe: resolve run-local output path for %q: %w", base, err)
|
||||
}
|
||||
jobs = append(jobs, job{
|
||||
speakerID: base,
|
||||
audioPath: audioPath,
|
||||
outPath: filepath.Join(paths.TranscriptsRawDir, base+".json"),
|
||||
outPath: runOut,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -197,12 +206,19 @@ dispatch:
|
||||
sort.Strings(speakers)
|
||||
|
||||
outputs := make([]artifacts.Ref, 0, len(speakers))
|
||||
runOutputPaths := make([]string, 0, len(speakers))
|
||||
outputPaths := make([]string, 0, len(speakers))
|
||||
orderedPerFile := make(map[string]any, len(speakers))
|
||||
for _, speaker := range speakers {
|
||||
ref := outputRef[speaker]
|
||||
outputs = append(outputs, ref)
|
||||
outputPaths = append(outputPaths, ref.AbsolutePath)
|
||||
runOutputPaths = append(runOutputPaths, ref.AbsolutePath)
|
||||
canonicalOut := filepath.Join(paths.TranscriptsRawDir, speaker+".json")
|
||||
promoted, err := promoteRunLocalOutput(env.ArtifactStore, ref.AbsolutePath, canonicalOut, ref)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("transcribe: promote %q output: %w", speaker, err)
|
||||
}
|
||||
outputs = append(outputs, promoted)
|
||||
outputPaths = append(outputPaths, canonicalOut)
|
||||
orderedPerFile[speaker] = perFile[speaker]
|
||||
}
|
||||
|
||||
@@ -221,6 +237,7 @@ dispatch:
|
||||
"retries": retries,
|
||||
"retry_delay": env.Config.Pipeline.WhisperX.RetryDelay,
|
||||
"timeout": env.Config.Pipeline.WhisperX.Timeout,
|
||||
"run_output_paths": runOutputPaths,
|
||||
"output_paths": outputPaths,
|
||||
"per_file": orderedPerFile,
|
||||
},
|
||||
|
||||
@@ -53,8 +53,8 @@ func TestTranscribeStageTranscribesPreparedAudio(t *testing.T) {
|
||||
}
|
||||
sort.Strings(gotPaths)
|
||||
wantPaths := []string{
|
||||
filepath.Join(env.ArtifactStore.SessionPaths(m.SessionID).TranscriptsRawDir, "alice.json"),
|
||||
filepath.Join(env.ArtifactStore.SessionPaths(m.SessionID).TranscriptsRawDir, "bob.json"),
|
||||
filepath.Join(sessionPathsForEnv(env, m.SessionID).TranscriptsRawDir, "alice.json"),
|
||||
filepath.Join(sessionPathsForEnv(env, m.SessionID).TranscriptsRawDir, "bob.json"),
|
||||
}
|
||||
sort.Strings(wantPaths)
|
||||
if strings.Join(gotPaths, "|") != strings.Join(wantPaths, "|") {
|
||||
@@ -191,6 +191,36 @@ func TestTranscribeStageInvalidJSONFails(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTranscribeStageUsesRunLocalOutputAndPromotesCanonical(t *testing.T) {
|
||||
env, m := setupTranscribeEnv(t, []string{"alice.flac"})
|
||||
env.Config.Session.Campaign = "sample-campaign"
|
||||
m.Campaign = "sample-campaign"
|
||||
m.RunID = "20260518T010203Z-abcdef12"
|
||||
fake := &whisperx.FakeClient{}
|
||||
env.WhisperX = fake
|
||||
|
||||
if _, err := (prepareStage{}).Run(context.Background(), env, m); err != nil {
|
||||
t.Fatalf("prepare.Run() error = %v", err)
|
||||
}
|
||||
result, err := (transcribeStage{}).Run(context.Background(), env, m)
|
||||
if err != nil {
|
||||
t.Fatalf("transcribe.Run() error = %v", err)
|
||||
}
|
||||
if len(fake.Requests) != 1 {
|
||||
t.Fatalf("requests = %d, want 1", len(fake.Requests))
|
||||
}
|
||||
runOut := fake.Requests[0].OutputRawTranscriptPath
|
||||
if !strings.Contains(runOut, filepath.Join("runs", m.RunID, "transcribe", "outputs")) {
|
||||
t.Fatalf("run-local output path = %q, want runs/{run_id}/transcribe/outputs path", runOut)
|
||||
}
|
||||
if len(result.Outputs) != 1 {
|
||||
t.Fatalf("outputs = %#v, want one output", result.Outputs)
|
||||
}
|
||||
if strings.Contains(result.Outputs[0].AbsolutePath, string(filepath.Separator)+"runs"+string(filepath.Separator)) {
|
||||
t.Fatalf("promoted output path = %q, want canonical session path", result.Outputs[0].AbsolutePath)
|
||||
}
|
||||
}
|
||||
|
||||
func setupTranscribeEnv(t *testing.T, audioFiles []string) (*Env, *manifest.Manifest) {
|
||||
t.Helper()
|
||||
|
||||
@@ -203,7 +233,7 @@ func setupTranscribeEnv(t *testing.T, audioFiles []string) (*Env, *manifest.Mani
|
||||
|
||||
sessionPath := filepath.Join(cfgDir, "session.yml")
|
||||
pipelinePath := filepath.Join(cfgDir, "pipeline.yml")
|
||||
writeFile(t, sessionPath, "session_id: 2026-05-03\n")
|
||||
writeFile(t, sessionPath, "session_id: 2026-05-03\ncampaign: sample-campaign\n")
|
||||
writeFile(t, pipelinePath, "workspace:\n root: "+workspace+"\n")
|
||||
writeFile(t, filepath.Join(cfgDir, "speakers.yml"), "alice: alice.flac\n")
|
||||
writeFile(t, filepath.Join(cfgDir, "autocorrect.yml"), "[]\n")
|
||||
@@ -227,6 +257,7 @@ func setupTranscribeEnv(t *testing.T, audioFiles []string) (*Env, *manifest.Mani
|
||||
},
|
||||
Session: &config.SessionConfig{
|
||||
SessionID: "2026-05-03",
|
||||
Campaign: "sample-campaign",
|
||||
Inputs: config.SessionInputsConfig{
|
||||
AudioDir: "./audio",
|
||||
SpeakersFile: "./speakers.yml",
|
||||
|
||||
@@ -54,7 +54,11 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
return nil, fmt.Errorf("trim: session id is required")
|
||||
}
|
||||
|
||||
paths := env.ArtifactStore.SessionPaths(sessionID)
|
||||
paths := sessionPathsForEnv(env, sessionID)
|
||||
runLayout, err := resolveRunStageLayout(env, m, paths, sessionID, "trim")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve run-stage layout: %w", err)
|
||||
}
|
||||
normalizedPath, normalizedSource, err := discoverNormalizedTranscript(m, paths)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve normalized transcript: %w", err)
|
||||
@@ -69,10 +73,14 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
trimCfg := env.Config.Pipeline.Trim
|
||||
enabled := trimCfg != nil && trimCfg.Enabled
|
||||
|
||||
trimmedPath, err := resolveTrimmedOutputPath(paths, trimCfg)
|
||||
canonicalTrimmedPath, err := resolveTrimmedOutputPath(paths, trimCfg)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve trimmed output path: %w", err)
|
||||
}
|
||||
trimmedPath, err := runLocalPathForCanonical(runLayout, paths, canonicalTrimmedPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve run-local trimmed output path: %w", err)
|
||||
}
|
||||
|
||||
logPaths := []string{}
|
||||
generatedConfigs := []string{}
|
||||
@@ -81,7 +89,8 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
"trim_enabled": enabled,
|
||||
"normalized_transcript_path": normalizedPath,
|
||||
"normalized_transcript_source": normalizedSource,
|
||||
"trimmed_output_path": trimmedPath,
|
||||
"run_trimmed_output_path": trimmedPath,
|
||||
"trimmed_output_path": canonicalTrimmedPath,
|
||||
}
|
||||
|
||||
if !enabled {
|
||||
@@ -91,14 +100,17 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
if err := validateProcessedTranscriptOutput(trimmedPath); err != nil {
|
||||
return nil, fmt.Errorf("trim: copied trimmed transcript %q invalid: %w", trimmedPath, err)
|
||||
}
|
||||
promotedTrimmed, err := promoteRunLocalOutput(env.ArtifactStore, trimmedPath, canonicalTrimmedPath, artifacts.Ref{
|
||||
Kind: "transcript_trimmed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: promote trimmed transcript: %w", err)
|
||||
}
|
||||
metadata["trim_action"] = "copy_disabled"
|
||||
return &StageResult{
|
||||
Outputs: []artifacts.Ref{{
|
||||
Kind: "transcript_trimmed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: trimmedPath,
|
||||
}},
|
||||
Outputs: []artifacts.Ref{promotedTrimmed},
|
||||
Metadata: metadata,
|
||||
}, nil
|
||||
}
|
||||
@@ -114,13 +126,22 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
}
|
||||
|
||||
boundsCfg := trimCfg.Bounds
|
||||
boundsOutputPath, err := resolveScriptoriumOutputPath(paths, boundsCfg.OutputPath)
|
||||
canonicalBoundsOutputPath, err := resolveScriptoriumOutputPath(paths, boundsCfg.OutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve bounds output path: %w", err)
|
||||
}
|
||||
boundsOutputPath, err := runLocalPathForCanonical(runLayout, paths, canonicalBoundsOutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve run-local bounds output path: %w", err)
|
||||
}
|
||||
boundsStdoutLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.stdout.log")
|
||||
boundsStderrLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.stderr.log")
|
||||
boundsGeneratedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium.bounds.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
boundsStdoutLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.stdout.log")
|
||||
boundsStderrLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.stderr.log")
|
||||
boundsGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium.bounds.generated.yml")
|
||||
}
|
||||
boundsTimeout, err := resolveScriptoriumTimeout(env.Config.Pipeline.Scriptorium.Timeout, boundsCfg.Timeout)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve bounds timeout: %w", err)
|
||||
@@ -133,7 +154,8 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
|
||||
metadata["bounds_prompt_id"] = boundsCfg.PromptID
|
||||
metadata["bounds_profile_id"] = boundsCfg.ProfileID
|
||||
metadata["bounds_output_path"] = boundsOutputPath
|
||||
metadata["run_bounds_output_path"] = boundsOutputPath
|
||||
metadata["bounds_output_path"] = canonicalBoundsOutputPath
|
||||
metadata["bounds_timeout"] = boundsTimeout.String()
|
||||
metadata["bounds_input_name"] = boundsCfg.TranscriptInputName
|
||||
metadata["bounds_input_path"] = normalizedPath
|
||||
@@ -141,13 +163,22 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
|
||||
renderOutputPath := ""
|
||||
if boundsCfg.RenderDebug {
|
||||
renderOutputPath, err = resolveScriptoriumOutputPath(paths, boundsCfg.RenderOutputPath)
|
||||
canonicalRenderOutputPath, err := resolveScriptoriumOutputPath(paths, boundsCfg.RenderOutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve bounds render output path: %w", err)
|
||||
}
|
||||
renderOutputPath, err = runLocalPathForCanonical(runLayout, paths, canonicalRenderOutputPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve run-local bounds render output path: %w", err)
|
||||
}
|
||||
renderStdoutLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.render.stdout.log")
|
||||
renderStderrLogPath := filepath.Join(paths.LogsDir, "scriptorium.bounds.render.stderr.log")
|
||||
renderGeneratedConfigPath := filepath.Join(paths.ConfigDir, "scriptorium.bounds.render.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
renderStdoutLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.render.stdout.log")
|
||||
renderStderrLogPath = filepath.Join(runLayout.LogsDir, "scriptorium.bounds.render.stderr.log")
|
||||
renderGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "scriptorium.bounds.render.generated.yml")
|
||||
}
|
||||
|
||||
renderReq := scriptorium.RenderArtifactRequest{
|
||||
Binary: env.Config.Pipeline.Scriptorium.Binary,
|
||||
@@ -254,7 +285,7 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
metadata["end_segment_id"] = boundsPayload.EndSegmentID
|
||||
metadata["warnings"] = boundsPayload.Warnings
|
||||
metadata["keep_selector"] = keepSelector
|
||||
metadata["bounds_output_path"] = finalBoundsOutputPath
|
||||
metadata["run_bounds_output_path"] = finalBoundsOutputPath
|
||||
metadata["bounds_stdout_log_path"] = boundsStdoutLogPath
|
||||
metadata["bounds_stderr_log_path"] = boundsStderrLogPath
|
||||
metadata["bounds_generated_config_path"] = boundsGeneratedConfigPath
|
||||
@@ -279,6 +310,11 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
trimStdoutLogPath := filepath.Join(paths.LogsDir, "seriatim.trim.stdout.log")
|
||||
trimStderrLogPath := filepath.Join(paths.LogsDir, "seriatim.trim.stderr.log")
|
||||
trimGeneratedConfigPath := filepath.Join(paths.ConfigDir, "seriatim.trim.generated.yml")
|
||||
if runLayout.Enabled {
|
||||
trimStdoutLogPath = filepath.Join(runLayout.LogsDir, "seriatim.trim.stdout.log")
|
||||
trimStderrLogPath = filepath.Join(runLayout.LogsDir, "seriatim.trim.stderr.log")
|
||||
trimGeneratedConfigPath = filepath.Join(runLayout.ConfigDir, "seriatim.trim.generated.yml")
|
||||
}
|
||||
trimTimeout, err := resolveTrimSeriatimTimeout(env.Config.Pipeline.Seriatim.Timeout)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: resolve seriatim timeout: %w", err)
|
||||
@@ -315,23 +351,25 @@ func (trimStage) Run(ctx context.Context, env *Env, m *manifest.Manifest) (*Stag
|
||||
return nil, fmt.Errorf("trim: trimmed transcript %q invalid: %w", trimmedPath, err)
|
||||
}
|
||||
|
||||
outputs := []artifacts.Ref{
|
||||
{
|
||||
Kind: "transcript_trimmed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: trimmedPath,
|
||||
},
|
||||
{
|
||||
Kind: "session_bounds",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
AbsolutePath: finalBoundsOutputPath,
|
||||
},
|
||||
promotedTrimmed, err := promoteRunLocalOutput(env.ArtifactStore, trimmedPath, canonicalTrimmedPath, artifacts.Ref{
|
||||
Kind: "transcript_trimmed",
|
||||
Category: "transcripts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: promote trimmed transcript: %w", err)
|
||||
}
|
||||
promotedBounds, err := promoteRunLocalOutput(env.ArtifactStore, finalBoundsOutputPath, canonicalBoundsOutputPath, artifacts.Ref{
|
||||
Kind: "session_bounds",
|
||||
Category: "artifacts",
|
||||
SessionID: sessionID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("trim: promote session bounds: %w", err)
|
||||
}
|
||||
|
||||
return &StageResult{
|
||||
Outputs: outputs,
|
||||
Outputs: []artifacts.Ref{promotedTrimmed, promotedBounds},
|
||||
Logs: logPaths,
|
||||
GeneratedConfigs: generatedConfigs,
|
||||
Metadata: metadata,
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user