Compare commits
64 Commits
71395bb076
...
v0.13.0
| Author | SHA1 | Date | |
|---|---|---|---|
| 74e2d21de5 | |||
| 7cb18a1a40 | |||
| b556fc2f4f | |||
| b99bd38eb4 | |||
| 701b6726d7 | |||
| 665039f4dc | |||
| ef8dae776e | |||
| d01775b68a | |||
| 0d6f2dd0ce | |||
| df40cbec6e | |||
| 0341e0c7c0 | |||
| 39af7d4f3c | |||
| bba582b4ca | |||
| 1f16a85330 | |||
| f9482639d4 | |||
| dce721cdbd | |||
| 98734644d6 | |||
| 951383226c | |||
| df58595d1e | |||
| c3c14e7468 | |||
| e7319ea016 | |||
| bd2d5e2496 | |||
| 115a44f629 | |||
| 18411dc5b5 | |||
| e23dc1ab6e | |||
| e1359ea227 | |||
| 7fdd99ec27 | |||
| a90231ce0c | |||
| ed879b8bb0 | |||
| 717451512a | |||
| 3ddb3a947b | |||
| c6632d5576 | |||
| ffc07922c7 | |||
| f3310d4d16 | |||
| 88cee96d8d | |||
| 2fece10215 | |||
| 0658f2f642 | |||
| a51228c803 | |||
| 4491fb5ccd | |||
| 30b905765c | |||
| 03eac70881 | |||
| 0f7e6b979f | |||
| c366912586 | |||
| 9fe44cd00d | |||
| 094b0d2532 | |||
| 98649f4d81 | |||
| 8a559efd5b | |||
| 72deccb4e2 | |||
| 5620fc5bcf | |||
| be57e675e0 | |||
| 3971443831 | |||
| a6b0c33e9f | |||
| 96b886e711 | |||
| 7d584ee6cd | |||
| 572a112c31 | |||
| ea87c335d6 | |||
| 7169ff04df | |||
| ef1f650bc0 | |||
| 0d02cb9fa0 | |||
| 0299b128cf | |||
| d723384888 | |||
| 54228055c8 | |||
| 23ed716450 | |||
| ab59bab044 |
44
README.md
44
README.md
@@ -1,22 +1,42 @@
|
|||||||
# narratio
|
# narratio
|
||||||
|
|
||||||
Narratio is a Go orchestration application that turns D&D session audio into polished transcripts and generated session artifacts.
|
Narratio is a stage-driven Go orchestrator for turning D&D session audio into
|
||||||
|
polished transcripts, validated Notarius extraction lanes, and generated
|
||||||
|
artifacts.
|
||||||
|
|
||||||
It coordinates transcription, merge/polish/normalize/trim processing, artifact generation, publish-stage uploads, and resumable run state in one operator workflow.
|
It runs a deterministic workflow with manifest-driven continuation, remote
|
||||||
|
publish, and restore support.
|
||||||
|
|
||||||
```bash
|
```sh
|
||||||
narratio run 2026-04-04
|
narratio run 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
This command requires discoverable `pipeline.yml` and `session.yml` files (or explicit `--config` and `--session` flags).
|
This requires resolvable `pipeline.yml`, `campaign.yml`, and concrete
|
||||||
|
`session.yml` files or their explicit command-line alternatives.
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
- [Configuration](docs/config.md)
|
- [CLI reference](docs/cli.md) — commands, arguments, flags, and invocation
|
||||||
- [CLI Reference](docs/cli.md)
|
behavior.
|
||||||
- [Operations and Recovery](docs/operations.md)
|
- [Configuration](docs/config.md) — discovery, fields, defaults, and
|
||||||
- [Troubleshooting](docs/troubleshooting.md)
|
validation.
|
||||||
- [Development Guide](docs/development.md)
|
- [Operations](docs/operations.md) — runtime workflow, state, publishing,
|
||||||
- [Architecture Principles](docs/architecture.md)
|
recovery, and cleanup.
|
||||||
- [Internal Component Contracts](docs/internal/README.md)
|
- [Troubleshooting](docs/troubleshooting.md) — symptom-driven diagnosis and
|
||||||
- [Config Examples](examples/)
|
safe remedies.
|
||||||
|
- [Integration contracts](docs/integrations/) — external tools, formats, and
|
||||||
|
compatibility expectations.
|
||||||
|
- [Maintained examples](examples/README.md) — complete copyable configuration
|
||||||
|
and input files.
|
||||||
|
|
||||||
|
## Maintainer Documentation
|
||||||
|
|
||||||
|
- [Development guide](docs/development.md) — first-read orientation and
|
||||||
|
task-specific reading routes.
|
||||||
|
- [Internal overview](docs/internal/overview.md) — implemented component map.
|
||||||
|
- [Architecture](docs/policy/architecture.md) — normative boundaries and
|
||||||
|
invariants.
|
||||||
|
- [Documentation policy](docs/policy/documentation.md) — canonical ownership
|
||||||
|
and maintenance rules.
|
||||||
|
- [Testing policy](docs/policy/testing.md) — test value, boundaries, and
|
||||||
|
sufficiency.
|
||||||
|
|||||||
@@ -1,202 +0,0 @@
|
|||||||
# Narratio Architecture
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
`narratio` is a Go orchestration application for processing D&D session audio into polished transcripts and generated session artifacts.
|
|
||||||
|
|
||||||
This document defines the development principles for the project. It is inward-facing: its audience is developers and LLM coding agents. It should guide future changes, not serve as a complete implementation reference.
|
|
||||||
|
|
||||||
Implemented component details belong under `docs/internal/`.
|
|
||||||
|
|
||||||
## Project Shape
|
|
||||||
|
|
||||||
Narratio is a modular, stage-driven orchestrator.
|
|
||||||
|
|
||||||
It coordinates specialized downstream systems rather than reimplementing their domains:
|
|
||||||
|
|
||||||
- WhisperX handles transcription.
|
|
||||||
- Seriatim handles deterministic transcript merge/normalization/trim behavior.
|
|
||||||
- Audita handles transcript correction and polishing.
|
|
||||||
- Scriptorium handles prompt execution and generated artifacts.
|
|
||||||
|
|
||||||
Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and publish semantics.
|
|
||||||
|
|
||||||
Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine.
|
|
||||||
|
|
||||||
## Core Principles
|
|
||||||
|
|
||||||
### Modular and composable
|
|
||||||
|
|
||||||
Code should be organized around clear responsibilities. Stages, adapters, config loading, manifest persistence, path construction, and storage behavior should remain separable and independently testable.
|
|
||||||
|
|
||||||
### Hexagonal boundaries
|
|
||||||
|
|
||||||
External systems should be isolated behind narrow adapters. Stage logic should depend on Narratio-level interfaces and data structures, not on external SDK types, subprocess argument construction, or transport-specific details.
|
|
||||||
|
|
||||||
### Standard library preference
|
|
||||||
|
|
||||||
Prefer the Go standard library. Add dependencies only when they provide substantial value, are necessary for an external integration, or are a widely used de facto standard.
|
|
||||||
|
|
||||||
Accepted examples include a YAML library for configuration and the AWS SDK for S3-compatible storage.
|
|
||||||
|
|
||||||
### Explicit orchestration
|
|
||||||
|
|
||||||
The pipeline should remain stage-driven and explicit. New behavior should be added through clear stage, adapter, config, or manifest contracts rather than implicit side effects or generic workflow abstraction.
|
|
||||||
|
|
||||||
## Stage Design
|
|
||||||
|
|
||||||
Each stage should have a clear scope of responsibility.
|
|
||||||
|
|
||||||
A stage should define:
|
|
||||||
|
|
||||||
- its purpose;
|
|
||||||
- required input state;
|
|
||||||
- produced output state;
|
|
||||||
- config fields it consumes;
|
|
||||||
- external adapters it uses;
|
|
||||||
- manifest refs it reads or writes;
|
|
||||||
- skip, force, and resume behavior;
|
|
||||||
- failure behavior;
|
|
||||||
- tests that protect its contract.
|
|
||||||
|
|
||||||
Stages should avoid reaching across boundaries. If shared behavior is needed, prefer a helper or service with a narrow interface over duplicating ad hoc logic between stages.
|
|
||||||
|
|
||||||
## Transactionality and Resume
|
|
||||||
|
|
||||||
A stage should behave transactionally.
|
|
||||||
|
|
||||||
A stage is complete only when its outputs have been written, validated, and recorded in the manifest. If a stage fails, Narratio should preserve enough local state for inspection, recovery, and resume.
|
|
||||||
|
|
||||||
A failed or incomplete run must not be treated as successful. Later stages should depend on manifest-recorded success, not merely on incidental files existing on disk.
|
|
||||||
|
|
||||||
## Manifest Model
|
|
||||||
|
|
||||||
The manifest is the durable local ledger for a run.
|
|
||||||
|
|
||||||
It should record:
|
|
||||||
|
|
||||||
- run identity;
|
|
||||||
- stage status;
|
|
||||||
- input and output refs;
|
|
||||||
- logs and generated config refs;
|
|
||||||
- checksums or provenance where useful;
|
|
||||||
- non-secret adapter and publish metadata.
|
|
||||||
|
|
||||||
Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress.
|
|
||||||
|
|
||||||
## Adapter Boundaries
|
|
||||||
|
|
||||||
Adapters own external integration details.
|
|
||||||
|
|
||||||
Expected boundaries:
|
|
||||||
|
|
||||||
- WhisperX HTTP details stay in the WhisperX adapter.
|
|
||||||
- Seriatim CLI construction stays in the Seriatim adapter.
|
|
||||||
- Audita CLI construction stays in the Audita adapter.
|
|
||||||
- Scriptorium CLI construction stays in the Scriptorium adapter.
|
|
||||||
- Object-storage details stay behind the storage adapter interface.
|
|
||||||
- AWS SDK types stay inside the S3 storage implementation.
|
|
||||||
|
|
||||||
Stage code should express intent in Narratio terms and call adapters through narrow contracts.
|
|
||||||
|
|
||||||
## Configuration Philosophy
|
|
||||||
|
|
||||||
Configuration should be strict, explicit, and operator-friendly.
|
|
||||||
|
|
||||||
Principles:
|
|
||||||
|
|
||||||
- YAML decoding should reject unknown fields.
|
|
||||||
- Defaults should be centralized and testable.
|
|
||||||
- Empty configured values should not silently override meaningful defaults.
|
|
||||||
- Session templating should remain narrow and deterministic.
|
|
||||||
- Template support should serve operator convenience, not become a general configuration language.
|
|
||||||
|
|
||||||
Narratio should not become a secondary configuration system for downstream tools. Seriatim, Audita, and Scriptorium should own their runtime defaults wherever practical. Narratio should pass required stage-contract paths and explicit operator overrides.
|
|
||||||
|
|
||||||
## Path and Storage Discipline
|
|
||||||
|
|
||||||
Local and remote paths are part of Narratio’s application contract.
|
|
||||||
|
|
||||||
Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and publish/current paths. Stages should avoid reconstructing canonical paths through scattered string concatenation.
|
|
||||||
|
|
||||||
Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics.
|
|
||||||
|
|
||||||
## Publish Invariants
|
|
||||||
|
|
||||||
Publish behavior must preserve a clear commit boundary.
|
|
||||||
|
|
||||||
A remote run is current only after the publish stage has successfully uploaded the run record, required published outputs, `current/manifest.json`, and finally `current/run_id.txt`.
|
|
||||||
|
|
||||||
`current/run_id.txt` is the final remote commit marker and must be written last.
|
|
||||||
|
|
||||||
Failed, incomplete, skipped, or uncommitted publish attempts must not be presented as current remote state. Local cleanup is permitted only after successful publish commit and only when explicitly configured.
|
|
||||||
|
|
||||||
## Security and Privacy
|
|
||||||
|
|
||||||
Narratio handles private campaign material.
|
|
||||||
|
|
||||||
Rules:
|
|
||||||
|
|
||||||
- Do not store raw secrets in pipeline or session YAML.
|
|
||||||
- Use environment variable names or secret-file references for secret handling.
|
|
||||||
- Do not write raw secret values to manifests, logs, generated configs, or publish metadata.
|
|
||||||
- Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive.
|
|
||||||
- Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason.
|
|
||||||
|
|
||||||
## Diagnostics
|
|
||||||
|
|
||||||
Diagnostics should be durable and discoverable, but distinct from canonical outputs.
|
|
||||||
|
|
||||||
Logs, reports, generated invocation/config files, and render-debug files support debugging. Transcript tiers and configured artifacts are pipeline products.
|
|
||||||
|
|
||||||
Manifest refs should preserve that distinction.
|
|
||||||
|
|
||||||
## Determinism
|
|
||||||
|
|
||||||
Where practical, Narratio should prefer deterministic behavior:
|
|
||||||
|
|
||||||
- stable local path layout;
|
|
||||||
- stable remote key layout;
|
|
||||||
- sorted upload order;
|
|
||||||
- predictable generated config files;
|
|
||||||
- repeatable command construction;
|
|
||||||
- tests that do not depend on live external services.
|
|
||||||
|
|
||||||
Run IDs and timestamps may be intentionally variable, but surrounding behavior should remain testable.
|
|
||||||
|
|
||||||
## Testing Expectations
|
|
||||||
|
|
||||||
Core behavior should be testable without live external services.
|
|
||||||
|
|
||||||
Tests should cover:
|
|
||||||
|
|
||||||
- config loading, defaults, and validation;
|
|
||||||
- CLI parsing and command construction;
|
|
||||||
- path helpers;
|
|
||||||
- manifest transitions;
|
|
||||||
- stage success, failure, skip, and resume behavior;
|
|
||||||
- adapter command construction;
|
|
||||||
- fake storage behavior;
|
|
||||||
- publish commit ordering;
|
|
||||||
- example config validity where practical.
|
|
||||||
|
|
||||||
Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs.
|
|
||||||
|
|
||||||
## Documentation Expectations
|
|
||||||
|
|
||||||
Documentation must follow `docs/documentation/policy.md`.
|
|
||||||
|
|
||||||
Current behavior belongs in user-facing docs and `docs/internal/`. Future, planned, aspirational, experimental, or unimplemented work belongs only under `docs/roadmap/`.
|
|
||||||
|
|
||||||
`docs/architecture.md` should remain concise and principle-focused. It should not duplicate the full config reference, CLI reference, operations guide, or internal stage documentation.
|
|
||||||
|
|
||||||
## Non-Goals
|
|
||||||
|
|
||||||
Narratio is not:
|
|
||||||
|
|
||||||
- a generic DAG or workflow engine;
|
|
||||||
- a replacement configuration layer for Seriatim, Audita, or Scriptorium;
|
|
||||||
- a storage backend abstraction beyond the needs of this pipeline;
|
|
||||||
- a place to embed raw secrets;
|
|
||||||
- a place for stage logic to depend directly on AWS SDK types or downstream tool internals;
|
|
||||||
- a prompt-authoring system.
|
|
||||||
227
docs/cli.md
227
docs/cli.md
@@ -1,4 +1,4 @@
|
|||||||
# CLI
|
# CLI Reference
|
||||||
|
|
||||||
## Shortest Useful Command
|
## Shortest Useful Command
|
||||||
|
|
||||||
@@ -6,18 +6,18 @@
|
|||||||
narratio run 2026-04-04
|
narratio run 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
This runs the full pipeline for the given session ID using default config discovery and campaign selection.
|
This runs the canonical full pipeline for session `2026-04-04`.
|
||||||
|
|
||||||
## Command Overview
|
## Command Overview
|
||||||
|
|
||||||
Top-level commands:
|
Top-level commands:
|
||||||
|
|
||||||
- `run <session_id>`: execute the pipeline.
|
- `run <session_id>`: run full stage order.
|
||||||
- `resume <session_id>`: continue from first non-succeeded stage.
|
- `run-stage <stage> <session_id>`: run one stage.
|
||||||
- `run-stage <stage> <session_id>`: execute exactly one stage.
|
- `analyze <session_id>`: force-run analyze.
|
||||||
- `analyze <session_id>`: force-rerun analyze stage.
|
- `publish <session_id>`: force-run publish.
|
||||||
- `publish <session_id>`: force-rerun publish stage.
|
- `clean <session_id>` or `clean --all`: remove local work/spool state.
|
||||||
- `clean <session_id>|--all`: remove local workspace/spool state.
|
- `session <subcommand>`: session helper commands.
|
||||||
- `session <subcommand>`: session-scoped helper commands.
|
|
||||||
|
|
||||||
Session subcommands:
|
Session subcommands:
|
||||||
|
|
||||||
@@ -31,39 +31,61 @@ Session subcommands:
|
|||||||
- `session locks add <session_id> <source>`
|
- `session locks add <session_id> <source>`
|
||||||
- `session locks remove <session_id> <source>`
|
- `session locks remove <session_id> <source>`
|
||||||
|
|
||||||
## Common Flags
|
## Common Config Flags
|
||||||
|
|
||||||
Most session-aware commands accept:
|
Most session-aware commands accept:
|
||||||
|
|
||||||
- `--config <pipeline.yml>`
|
- `--config <pipeline.yml>`
|
||||||
- `--campaign <id>`
|
- `--campaign <id>`
|
||||||
- `--campaign-file <campaign.yml>`
|
- `--campaign-file <campaign.yml>`
|
||||||
- `--session <session.yml>`
|
- `--session <session.yml>`
|
||||||
- `--previous-session-id <id>`
|
- `--session-id <session_id>`
|
||||||
|
- `--previous-session-id <session_id>`
|
||||||
|
|
||||||
`--campaign` and `--campaign-file` are mutually exclusive.
|
Rules:
|
||||||
|
|
||||||
|
- `--campaign` and `--campaign-file` are mutually exclusive.
|
||||||
|
- `--session` is not used by `session init`.
|
||||||
|
- if both positional `<session_id>` and `--session-id` are provided, values must match.
|
||||||
|
- `clean --all` cannot be combined with campaign/session selectors.
|
||||||
|
|
||||||
|
## Session ID Input Rules
|
||||||
|
|
||||||
|
Session-aware commands accept one of these forms:
|
||||||
|
|
||||||
|
- positional session ID: `... <session_id>`
|
||||||
|
- compatibility flag: `... --session-id <session_id>`
|
||||||
|
|
||||||
|
When both are present, command parsing requires an exact match.
|
||||||
|
|
||||||
|
Commands with additional positionals keep their command-specific order:
|
||||||
|
|
||||||
|
- `run-stage <stage> <session_id>` or `run-stage <stage> --session-id <session_id>`
|
||||||
|
- `session locks add <session_id> <source>` or `session locks add --session-id <session_id> <source>`
|
||||||
|
- `session locks remove <session_id> <source>` or `session locks remove --session-id <session_id> <source>`
|
||||||
|
|
||||||
## Command Reference
|
## Command Reference
|
||||||
|
|
||||||
### `run`
|
### `run`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio run <session_id> [--force] [--artifacts <name[,name...]>] [...common flags]
|
narratio run <session_id> [--force] [--artifacts <name[,name...]>] [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Runs stages in canonical order and writes manifest state.
|
Behavior:
|
||||||
|
|
||||||
### `resume`
|
- evaluates full stage order;
|
||||||
|
- runs `extract` between `trim` and `render`; an omitted or disabled Notarius
|
||||||
```bash
|
configuration records an explicit `notarius_disabled` self-skip;
|
||||||
narratio resume <session_id> [--force] [--artifacts <name[,name...]>] [...common flags]
|
- skips already-succeeded stages unless `--force` is set or a stage-specific
|
||||||
```
|
resume check finds its durable result obsolete;
|
||||||
|
- continues interrupted or partially completed sessions by running non-succeeded stages;
|
||||||
Starts at the first non-succeeded stage from the session manifest.
|
- writes session and run manifests.
|
||||||
|
|
||||||
### `run-stage`
|
### `run-stage`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio run-stage <stage> <session_id> [--force] [--artifacts <name[,name...]>] [...common flags]
|
narratio run-stage <stage> <session_id> [--force] [--artifacts <name[,name...]>] [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Valid stage names:
|
Valid stage names:
|
||||||
@@ -74,127 +96,168 @@ Valid stage names:
|
|||||||
- `polish`
|
- `polish`
|
||||||
- `normalize`
|
- `normalize`
|
||||||
- `trim`
|
- `trim`
|
||||||
|
- `extract`
|
||||||
|
- `render`
|
||||||
- `analyze`
|
- `analyze`
|
||||||
- `publish`
|
- `publish`
|
||||||
- `notify`
|
- `notify`
|
||||||
|
|
||||||
`--artifacts` is accepted only for `analyze` and `publish`.
|
Rules:
|
||||||
|
|
||||||
|
- `--artifacts` is accepted only for `analyze` and `publish` stage targets.
|
||||||
|
|
||||||
### `analyze`
|
### `analyze`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio analyze <session_id> [--artifacts <name[,name...]>] [...common flags]
|
narratio analyze <session_id> [--artifacts <name[,name...]>] [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Equivalent to `narratio run-stage analyze <session_id> --force`.
|
Equivalent to:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage analyze <session_id> --force [...common config flags]
|
||||||
|
```
|
||||||
|
|
||||||
### `publish`
|
### `publish`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio publish <session_id> [--artifacts <name[,name...]>] [...common flags]
|
narratio publish <session_id> [--artifacts <name[,name...]>] [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Equivalent to `narratio run-stage publish <session_id> --force`.
|
Equivalent to:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage publish <session_id> --force [...common config flags]
|
||||||
|
```
|
||||||
|
|
||||||
### `clean`
|
### `clean`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio clean <session_id> [--dry-run] [--clear-cache] [...common flags]
|
narratio clean <session_id> [--dry-run] [--clear-cache] [...common config flags]
|
||||||
narratio clean --all [--dry-run] [--clear-cache] [--config <pipeline.yml>]
|
narratio clean --all [--dry-run] [--clear-cache] [--config <pipeline.yml>]
|
||||||
```
|
```
|
||||||
|
|
||||||
- session mode deletes `{workspace.root}/work/{campaign}/{session_id}` and `{spool.root}/{campaign}/{session_id}`.
|
Behavior:
|
||||||
- `--all` deletes all session work and spool children.
|
|
||||||
- cache is preserved unless `--clear-cache` is passed.
|
- session mode removes the selected session's local work and spool state;
|
||||||
|
- `--all` removes all local session work and spool state;
|
||||||
|
- cache remains unless `--clear-cache` is provided.
|
||||||
|
|
||||||
|
See [Operations: Cleanup](./operations.md#cleanup) for deletion scope and
|
||||||
|
post-publish cleanup behavior.
|
||||||
|
|
||||||
### `session plan`
|
### `session plan`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session plan <session_id> [--force] [...common flags]
|
narratio session plan <session_id> [--force] [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Validates config and session inputs, prepares workdir layout, and prints stage run/skip decisions.
|
Validates config, prepares local workdir layout, and prints run/skip decisions for each stage.
|
||||||
|
|
||||||
### `session validate`
|
### `session validate`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session validate <session_id> [...common flags]
|
narratio session validate <session_id> [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Read-only preflight checks for config, inputs, audio availability, previous-session requirements, publish outputs, and effective locks.
|
Read-only preflight checks for config validity, required inputs, audio mode, previous-session requirements, publish outputs, and effective locks.
|
||||||
|
|
||||||
### `session status`
|
### `session status`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session status <session_id> [...common flags]
|
narratio session status <session_id> [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Shows local manifest state, remote current state (when storage is configured), published-output availability, and effective locks.
|
Prints local manifest state and, when storage is available, remote current-state and published-output status.
|
||||||
|
|
||||||
### `session init`
|
### `session init`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session init <session_id> --output ./session.yml
|
narratio session init <session_id> --output ./session.yml [options]
|
||||||
narratio session init <session_id> --remote
|
narratio session init <session_id> --remote [options]
|
||||||
narratio session init <session_id> --remote --force
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Flags:
|
Required target selection:
|
||||||
|
|
||||||
- `--output <path>` or `--remote` (exactly one is required)
|
- exactly one of:
|
||||||
|
- `--output <path>`
|
||||||
|
- `--remote`
|
||||||
|
|
||||||
|
Options:
|
||||||
|
|
||||||
|
- `--config <pipeline.yml>`
|
||||||
|
- `--campaign <id>` or `--campaign-file <campaign.yml>`
|
||||||
- `--previous-session-id <id>`
|
- `--previous-session-id <id>`
|
||||||
- `--date <date>`
|
- `--date <YYYY-MM-DD>`
|
||||||
- `--title <title>`
|
- `--title <text>`
|
||||||
- `--audio-dir <path>`
|
- `--audio-dir <path>`
|
||||||
- `--audio-s3-prefix <prefix>`
|
- `--audio-s3-prefix <prefix>`
|
||||||
- `--force`
|
- `--force`
|
||||||
- common config/campaign flags
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- `--audio-dir` and `--audio-s3-prefix` are mutually exclusive.
|
||||||
|
- if campaign `session_template_file` is configured, `session init` renders it.
|
||||||
|
- generated session YAML must be concrete (no unresolved `{{ ... }}` placeholders).
|
||||||
|
|
||||||
### `session restore`
|
### `session restore`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session restore <session_id> [--dry-run] [--force] [--include-audio] [...common flags]
|
narratio session restore <session_id> [--dry-run] [--force] [--include-audio] [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Restores durable local session files from committed remote current state.
|
Behavior:
|
||||||
|
|
||||||
Default restore scope:
|
- discovers committed remote current state;
|
||||||
|
- plans local restores;
|
||||||
|
- writes an execution report;
|
||||||
|
- blocks conflicting overwrites unless `--force` is set.
|
||||||
|
|
||||||
- `manifest.json`
|
See [Operations: Restore Workflow](./operations.md#restore-workflow) for the
|
||||||
- `transcripts/**`
|
default restore scope, report location, and conflict-handling workflow.
|
||||||
- `artifacts/**`
|
|
||||||
- `previous/**` when required by configured previous-session artifact inputs
|
|
||||||
|
|
||||||
`audio/**` is restored only when `--include-audio` is set.
|
|
||||||
|
|
||||||
### `session artifacts`
|
### `session artifacts`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session artifacts <session_id> [--remote] [...common flags]
|
narratio session artifacts <session_id> [--remote] [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Lists built-in sources, configured artifact sources, previous-session sources, publish output rules, and lock status. With `--remote`, includes remote published-state markers.
|
Lists effective built-in, configured Scriptorium, and configured extraction
|
||||||
|
sources; reports planned, available, unavailable, and published state without
|
||||||
|
reading payload bodies; and includes publish rules, lock state, and optional
|
||||||
|
remote published-state availability.
|
||||||
|
|
||||||
### `session locks`
|
### `session locks`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session locks <session_id> [...common flags]
|
narratio session locks <session_id> [...common config flags]
|
||||||
narratio session locks add <session_id> <source> [--reason <text>] [--force] [...common flags]
|
narratio session locks add <session_id> <source> [--reason <text>] [--force] [...common config flags]
|
||||||
narratio session locks remove <session_id> <source> [...common flags]
|
narratio session locks remove <session_id> <source> [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
- list mode prints effective locks from static `pipeline.publish.locks` and remote `{session_prefix}/locks.yml`.
|
Behavior:
|
||||||
- add/remove mutate only the remote lock store.
|
|
||||||
- static pipeline locks cannot be removed by lock commands.
|
|
||||||
|
|
||||||
## `--artifacts` Rules
|
- list mode reports the effective merge of static and remote locks;
|
||||||
|
- add/remove mutate only remote locks;
|
||||||
|
- static locks from pipeline config cannot be removed by CLI commands.
|
||||||
|
|
||||||
- accepted on `run`, `resume`, `run-stage`, `analyze`, and `publish`.
|
See [Operations: Publish Locks](./operations.md#publish-locks) for lock storage
|
||||||
- on `run-stage`, only valid for `analyze` and `publish`.
|
and precedence.
|
||||||
- filters configured analyze artifact execution.
|
|
||||||
- filters configured `pipeline.publish.outputs` entries for `narratio.artifact.<key>` sources.
|
## `--artifacts` Selection Rules
|
||||||
- does not suppress built-in transcript/bounds publish outputs.
|
|
||||||
- does not imply `--force` for `run`, `resume`, or `run-stage`.
|
- accepted on `run`, `run-stage`, `analyze`, and `publish`;
|
||||||
|
- names must exist in `pipeline.scriptorium.artifacts`;
|
||||||
|
- empty entries are invalid;
|
||||||
|
- repeated names are deduplicated.
|
||||||
|
|
||||||
|
Effects:
|
||||||
|
|
||||||
|
- filters analyze execution to selected configured artifacts;
|
||||||
|
- filters publish rules that source `narratio.artifact.<name>`;
|
||||||
|
- does not filter built-in transcript/bounds or explicitly configured
|
||||||
|
`narratio.extraction.<name>` publish sources; and
|
||||||
|
- does not select or filter Notarius lanes.
|
||||||
|
|
||||||
## Common Workflows
|
## Common Workflows
|
||||||
|
|
||||||
@@ -204,16 +267,16 @@ Run full pipeline:
|
|||||||
narratio run 2026-04-04
|
narratio run 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
Run only selected analyze artifacts:
|
Dry-run restore plan:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio run 2026-04-04 --artifacts session_recap,player_handout
|
narratio session restore 2026-04-04 --dry-run
|
||||||
```
|
```
|
||||||
|
|
||||||
Force analyze only:
|
Generate a concrete session file from template/default structure:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio analyze 2026-04-04 --artifacts player_handout
|
narratio session init 2026-04-04 --output ./session.yml --date 2026-04-04 --title "Session 12"
|
||||||
```
|
```
|
||||||
|
|
||||||
Force publish only:
|
Force publish only:
|
||||||
@@ -222,9 +285,17 @@ Force publish only:
|
|||||||
narratio publish 2026-04-04
|
narratio publish 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
Restore preview then apply:
|
## Output And Exit Behavior
|
||||||
|
|
||||||
```bash
|
- Successful commands write their result or summary to standard output and
|
||||||
narratio session restore 2026-04-04 --dry-run
|
exit with status `0`.
|
||||||
narratio session restore 2026-04-04
|
- Command failures and invalid invocations write an error to standard error and
|
||||||
```
|
exit with status `1`.
|
||||||
|
- An unknown top-level command also prints the top-level usage summary to
|
||||||
|
standard error.
|
||||||
|
- `session restore --help` prints its command-specific usage and exits with
|
||||||
|
status `0`.
|
||||||
|
|
||||||
|
Output is intended for operator inspection. Narratio does not currently offer
|
||||||
|
a machine-readable CLI output mode; durable machine-readable state is recorded
|
||||||
|
in manifests and reports described in [Operations](./operations.md).
|
||||||
|
|||||||
274
docs/config.md
274
docs/config.md
@@ -1,56 +1,56 @@
|
|||||||
# Configuration
|
# Configuration Reference
|
||||||
|
|
||||||
## Overview
|
## Purpose
|
||||||
Narratio loads three YAML files:
|
|
||||||
|
|
||||||
- `pipeline.yml`: pipeline-level runtime settings.
|
Narratio resolves three YAML documents:
|
||||||
- `campaign.yml`: stable campaign identity and campaign-level input defaults.
|
|
||||||
- `session.yml`: per-session metadata and input selection.
|
|
||||||
|
|
||||||
Commands that load and validate all three files include:
|
- `pipeline.yml`: pipeline/runtime settings
|
||||||
|
- `campaign.yml`: campaign identity and stable input defaults
|
||||||
|
- `session.yml`: session identity, metadata, and audio source selection
|
||||||
|
|
||||||
- `narratio run`
|
## Discovery and Selection
|
||||||
- `narratio resume`
|
|
||||||
- `narratio run-stage`
|
|
||||||
- `narratio analyze`
|
|
||||||
- `narratio publish`
|
|
||||||
- `narratio session plan`
|
|
||||||
- `narratio session status`
|
|
||||||
- `narratio session validate`
|
|
||||||
- `narratio session restore`
|
|
||||||
- `narratio session artifacts`
|
|
||||||
- `narratio session locks`
|
|
||||||
- `narratio clean <session_id>`
|
|
||||||
|
|
||||||
Validation behavior:
|
### `pipeline.yml`
|
||||||
|
|
||||||
- strict YAML decode is enabled (`KnownFields(true)`); unknown fields fail.
|
When `--config` is omitted, search order is:
|
||||||
- loaded `session.yml` files must be concrete YAML (no `{{ ... }}` placeholders).
|
|
||||||
- defaults are applied for optional pipeline fields.
|
|
||||||
- campaign/session identity mismatches fail load.
|
|
||||||
|
|
||||||
## File Discovery
|
|
||||||
Pipeline discovery order when `--config` is omitted:
|
|
||||||
|
|
||||||
1. `/usr/local/etc/narratio/pipeline.yml`
|
1. `/usr/local/etc/narratio/pipeline.yml`
|
||||||
2. `/etc/narratio/pipeline.yml`
|
2. `/etc/narratio/pipeline.yml`
|
||||||
|
|
||||||
Session discovery order when `--session` is omitted:
|
### `campaign.yml`
|
||||||
|
|
||||||
|
Selection rules:
|
||||||
|
|
||||||
|
- if `--campaign-file` is set, use that path;
|
||||||
|
- else if `--campaign <id>` is set, use `{pipeline.campaigns.root}/{id}/campaign.yml`;
|
||||||
|
- else use `{pipeline.campaigns.root}/{pipeline.campaigns.default_campaign_id}/campaign.yml`.
|
||||||
|
|
||||||
|
### `session.yml`
|
||||||
|
|
||||||
|
When `--session` is omitted, local search order is:
|
||||||
|
|
||||||
1. `/usr/local/etc/narratio/session.yml`
|
1. `/usr/local/etc/narratio/session.yml`
|
||||||
2. `/etc/narratio/session.yml`
|
2. `/etc/narratio/session.yml`
|
||||||
|
|
||||||
Campaign discovery when `--campaign-file` is omitted:
|
If local session discovery fails and a `session_id` is known, Narratio attempts remote session loading from:
|
||||||
|
|
||||||
- if `--campaign <id>` is set: `{pipeline.campaigns.root}/{id}/campaign.yml`
|
|
||||||
- otherwise: `{pipeline.campaigns.root}/{pipeline.campaigns.default_campaign_id}/campaign.yml`
|
|
||||||
|
|
||||||
Remote `session.yml` fallback:
|
|
||||||
|
|
||||||
- if local session discovery fails and storage is configured, Narratio can load:
|
|
||||||
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`
|
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml`
|
||||||
|
|
||||||
## Minimal Working Config
|
using configured object storage.
|
||||||
|
|
||||||
|
## Validation and Merge Rules
|
||||||
|
|
||||||
|
- YAML decode is strict (`KnownFields(true)`): unknown fields fail load.
|
||||||
|
- Session files must be concrete; unresolved `{{ ... }}` placeholders fail load.
|
||||||
|
- Pipeline defaults are applied before validation.
|
||||||
|
- Campaign and session identities must agree.
|
||||||
|
- Stable files (`speakers_file`, `autocorrect_file`, `glossary_file`, `players_file`, `party_file`) resolve from session overrides when provided, otherwise from campaign defaults.
|
||||||
|
- Exactly one audio mode must be configured in session input:
|
||||||
|
- local (`audio_dir` or `audio_files`), or
|
||||||
|
- S3 (`audio_s3.prefix`).
|
||||||
|
|
||||||
|
## Minimal Working Configuration
|
||||||
|
|
||||||
`pipeline.yml`
|
`pipeline.yml`
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
@@ -58,7 +58,7 @@ campaigns:
|
|||||||
root: /usr/local/share/narratio/campaigns
|
root: /usr/local/share/narratio/campaigns
|
||||||
default_campaign_id: sample-campaign
|
default_campaign_id: sample-campaign
|
||||||
whisperx:
|
whisperx:
|
||||||
transcribe_url: "https://transcription.example.com/transcribe"
|
transcribe_url: https://transcription.example.com/transcribe
|
||||||
```
|
```
|
||||||
|
|
||||||
`campaign.yml`
|
`campaign.yml`
|
||||||
@@ -69,9 +69,11 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
`session.yml`
|
`session.yml` (local audio)
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
session_id: 2026-05-03
|
session_id: 2026-05-03
|
||||||
@@ -79,8 +81,16 @@ inputs:
|
|||||||
audio_dir: ./audio
|
audio_dir: ./audio
|
||||||
```
|
```
|
||||||
|
|
||||||
## Publish Config
|
## Secrets Handling
|
||||||
Top-level publish settings live at `pipeline.publish`.
|
|
||||||
|
- Do not place raw secrets in YAML.
|
||||||
|
- Use env var names in config (for example `pipeline.audita.llm_api_key_env`).
|
||||||
|
- Optionally load env files from `pipeline.secrets.env_dir`.
|
||||||
|
- Commands that need storage/auth load filesystem secrets before constructing adapters.
|
||||||
|
|
||||||
|
## Publish Configuration Summary
|
||||||
|
|
||||||
|
Publish rules live under `pipeline.publish`.
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
publish:
|
publish:
|
||||||
@@ -90,31 +100,35 @@ publish:
|
|||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
required: true
|
required: true
|
||||||
|
- source: narratio.transcript.final_markdown
|
||||||
|
dest: transcripts/final.md
|
||||||
|
required: true
|
||||||
|
- source: narratio.transcript.final_trimmed_markdown
|
||||||
|
dest: transcripts/final.trimmed.md
|
||||||
|
required: true
|
||||||
- source: narratio.artifact.session_recap
|
- source: narratio.artifact.session_recap
|
||||||
dest: artifacts/session_recap.md
|
dest: artifacts/session_recap.md
|
||||||
required: true
|
required: true
|
||||||
locks:
|
locks:
|
||||||
- source: narratio.artifact.session_recap
|
- source: narratio.artifact.session_recap
|
||||||
reason: Final recap was manually edited.
|
reason: manual post-publish edits
|
||||||
```
|
```
|
||||||
|
|
||||||
Rules:
|
Rules:
|
||||||
|
|
||||||
- `outputs[].source` is required.
|
- `outputs[].source` is required.
|
||||||
- `outputs[].dest` is optional; when omitted, Narratio derives destination from the source.
|
- `outputs[].dest` may be omitted when derivable from source.
|
||||||
|
- extraction sources require an explicit `outputs[].dest` and publish only when
|
||||||
|
a rule names that source; the Notarius index and complete bundle are not
|
||||||
|
publish sources.
|
||||||
- `outputs[].required` defaults to `true`.
|
- `outputs[].required` defaults to `true`.
|
||||||
- static `publish.locks` and remote `{session_prefix}/locks.yml` are merged; static locks win on duplicates.
|
- static locks (`pipeline.publish.locks`) merge with remote locks (`{session_prefix}/locks.yml`), with static locks taking precedence on duplicates.
|
||||||
- locks prevent overwrite of top-level published destinations.
|
|
||||||
|
|
||||||
Supported publish source families:
|
## Full Schema
|
||||||
|
|
||||||
- built-ins: `narratio.transcript.base`, `narratio.transcript.polished`, `narratio.transcript.final`, `narratio.transcript.final_trimmed`, `narratio.bounds.session`
|
|
||||||
- configured artifacts: `narratio.artifact.<artifact_key>`
|
|
||||||
|
|
||||||
## Full Reference
|
|
||||||
|
|
||||||
### Pipeline
|
### Pipeline
|
||||||
| Path | Type | Required | Default |
|
|
||||||
|
| Field | Type | Required | Default / Rule |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
|
| `pipeline.workspace.root` | string | No | `/var/lib/narratio` |
|
||||||
| `pipeline.workspace.cleanup_after_publish` | bool | No | `false` |
|
| `pipeline.workspace.cleanup_after_publish` | bool | No | `false` |
|
||||||
@@ -122,7 +136,7 @@ Supported publish source families:
|
|||||||
| `pipeline.campaigns.default_campaign_id` | string | No | empty |
|
| `pipeline.campaigns.default_campaign_id` | string | No | empty |
|
||||||
| `pipeline.secrets.env_dir` | string | No | empty |
|
| `pipeline.secrets.env_dir` | string | No | empty |
|
||||||
| `pipeline.storage.backend` | string | No | empty |
|
| `pipeline.storage.backend` | string | No | empty |
|
||||||
| `pipeline.storage.s3.bucket` | string | Conditional | empty |
|
| `pipeline.storage.s3.bucket` | string | Conditional | required for S3 session-audio and for publish upload when backend is `s3` |
|
||||||
| `pipeline.storage.s3.root_prefix` | string | No | `dnd` |
|
| `pipeline.storage.s3.root_prefix` | string | No | `dnd` |
|
||||||
| `pipeline.storage.s3.region` | string | No | empty |
|
| `pipeline.storage.s3.region` | string | No | empty |
|
||||||
| `pipeline.storage.s3.endpoint` | string | No | empty |
|
| `pipeline.storage.s3.endpoint` | string | No | empty |
|
||||||
@@ -135,21 +149,21 @@ Supported publish source families:
|
|||||||
| `pipeline.cache.s3_audio` | bool | No | `true` |
|
| `pipeline.cache.s3_audio` | bool | No | `true` |
|
||||||
| `pipeline.publish.enabled` | bool | No | `true` |
|
| `pipeline.publish.enabled` | bool | No | `true` |
|
||||||
| `pipeline.publish.upload_run` | bool | No | `true` |
|
| `pipeline.publish.upload_run` | bool | No | `true` |
|
||||||
| `pipeline.publish.outputs[]` | list | No | one final-trimmed output rule |
|
| `pipeline.publish.outputs[]` | list | No | defaults to final trimmed JSON plus final and final-trimmed Markdown outputs |
|
||||||
| `pipeline.publish.outputs[].source` | string | Yes (per rule) | none |
|
| `pipeline.publish.outputs[].source` | string | Yes (per rule) | must reference built-in or configured artifact source |
|
||||||
| `pipeline.publish.outputs[].dest` | string | No | derived from source |
|
| `pipeline.publish.outputs[].dest` | string | Conditional | derived if omitted and source supports derivation |
|
||||||
| `pipeline.publish.outputs[].required` | bool | No | `true` |
|
| `pipeline.publish.outputs[].required` | bool | No | `true` |
|
||||||
| `pipeline.publish.locks[]` | list | No | empty |
|
| `pipeline.publish.locks[]` | list | No | empty |
|
||||||
| `pipeline.publish.locks[].source` | string | Yes (per lock) | none |
|
| `pipeline.publish.locks[].source` | string | Yes (per lock) | must reference supported publish source |
|
||||||
| `pipeline.publish.locks[].reason` | string | No | empty |
|
| `pipeline.publish.locks[].reason` | string | No | empty |
|
||||||
| `pipeline.whisperx.transcribe_url` | string | Yes | none |
|
| `pipeline.whisperx.transcribe_url` | string | Yes | valid URL |
|
||||||
| `pipeline.whisperx.language` | string | No | `en` |
|
| `pipeline.whisperx.language` | string | No | `en` |
|
||||||
| `pipeline.whisperx.timeout` | duration string | No | `30m` |
|
| `pipeline.whisperx.timeout` | duration | No | `30m` |
|
||||||
| `pipeline.whisperx.retries` | int | No | `3` |
|
| `pipeline.whisperx.retries` | int | No | `3` |
|
||||||
| `pipeline.whisperx.retry_delay` | duration string | No | `2s` |
|
| `pipeline.whisperx.retry_delay` | duration | No | `2s` |
|
||||||
| `pipeline.whisperx.concurrency` | int | No | `2` |
|
| `pipeline.whisperx.concurrency` | int | No | `2` |
|
||||||
| `pipeline.seriatim.binary` | string | No | `seriatim` |
|
| `pipeline.seriatim.binary` | string | No | `seriatim` |
|
||||||
| `pipeline.seriatim.timeout` | duration string | No | `10m` |
|
| `pipeline.seriatim.timeout` | duration | No | `10m` |
|
||||||
| `pipeline.seriatim.output_schema` | string | No | `seriatim-intermediate` |
|
| `pipeline.seriatim.output_schema` | string | No | `seriatim-intermediate` |
|
||||||
| `pipeline.seriatim.coalesce_gap` | float | No | `3.0` |
|
| `pipeline.seriatim.coalesce_gap` | float | No | `3.0` |
|
||||||
| `pipeline.seriatim.report` | bool | No | `true` |
|
| `pipeline.seriatim.report` | bool | No | `true` |
|
||||||
@@ -158,7 +172,7 @@ Supported publish source families:
|
|||||||
| `pipeline.seriatim.env.backchannel_max_duration` | float | No | unset |
|
| `pipeline.seriatim.env.backchannel_max_duration` | float | No | unset |
|
||||||
| `pipeline.seriatim.env.filler_max_duration` | float | No | unset |
|
| `pipeline.seriatim.env.filler_max_duration` | float | No | unset |
|
||||||
| `pipeline.audita.binary` | string | No | `audita` |
|
| `pipeline.audita.binary` | string | No | `audita` |
|
||||||
| `pipeline.audita.timeout` | duration string | No | `3h` |
|
| `pipeline.audita.timeout` | duration | No | `3h` |
|
||||||
| `pipeline.audita.llm_api_key_env` | string | No | empty |
|
| `pipeline.audita.llm_api_key_env` | string | No | empty |
|
||||||
| `pipeline.audita.modules[]` | list[string] | No | empty |
|
| `pipeline.audita.modules[]` | list[string] | No | empty |
|
||||||
| `pipeline.audita.base_url` | string | No | empty |
|
| `pipeline.audita.base_url` | string | No | empty |
|
||||||
@@ -175,53 +189,121 @@ Supported publish source families:
|
|||||||
| `pipeline.normalize.output_path` | string | No | `transcripts/final.json` |
|
| `pipeline.normalize.output_path` | string | No | `transcripts/final.json` |
|
||||||
| `pipeline.normalize.output_schema` | string | No | `seriatim-intermediate` |
|
| `pipeline.normalize.output_schema` | string | No | `seriatim-intermediate` |
|
||||||
| `pipeline.normalize.report` | bool | No | `true` |
|
| `pipeline.normalize.report` | bool | No | `true` |
|
||||||
| `pipeline.trim.enabled` | bool | No | `false` |
|
| `pipeline.trim.enabled` | bool | No | `true` |
|
||||||
| `pipeline.trim.output_path` | string | Conditional | none |
|
| `pipeline.trim.output_path` | string | No | `transcripts/final.trimmed.json` |
|
||||||
| `pipeline.trim.bounds.prompt_id` | string | Conditional | none |
|
| `pipeline.trim.bounds.prompt_id` | string | No | `dnd.session_bounds` |
|
||||||
| `pipeline.trim.bounds.profile_id` | string | No | empty |
|
| `pipeline.trim.bounds.profile_id` | string | No | empty |
|
||||||
| `pipeline.trim.bounds.transcript_input_name` | string | Conditional | none |
|
| `pipeline.trim.bounds.transcript_input_name` | string | No | `transcript` |
|
||||||
| `pipeline.trim.bounds.output_path` | string | Conditional | none |
|
| `pipeline.trim.bounds.output_path` | string | No | `artifacts/session_bounds.json` |
|
||||||
| `pipeline.trim.bounds.timeout` | duration string | No | `10m` |
|
| `pipeline.trim.bounds.timeout` | duration | No | `10m` |
|
||||||
| `pipeline.trim.bounds.render_debug` | bool | No | `false` |
|
| `pipeline.trim.bounds.render_debug` | bool | No | `false` |
|
||||||
| `pipeline.trim.bounds.render_output_path` | string | Conditional | none |
|
| `pipeline.trim.bounds.render_output_path` | string | Conditional | required when `render_debug` is true |
|
||||||
| `pipeline.trim.seriatim.report` | bool | No | `false` |
|
| `pipeline.trim.seriatim.report` | bool | No | `false` |
|
||||||
|
| `pipeline.notarius.enabled` | bool | No | `false` |
|
||||||
|
| `pipeline.notarius.binary` | string | No | `notarius` |
|
||||||
|
| `pipeline.notarius.config_path` | string | Conditional | required when enabled; relative paths resolve from the pipeline file directory |
|
||||||
|
| `pipeline.notarius.pipeline_id` | string | Conditional | required when enabled |
|
||||||
|
| `pipeline.notarius.timeout` | duration | No | `3h`; must be positive |
|
||||||
|
| `pipeline.notarius.working_directory` | string | No | directory containing resolved `config_path`; relative paths resolve from the pipeline file directory |
|
||||||
|
| `pipeline.notarius.outputs` | map | Conditional | at least one entry when enabled |
|
||||||
|
| `pipeline.render.enabled` | bool | No | `true` |
|
||||||
|
| `pipeline.render.format` | string | No | `markdown` (only supported value) |
|
||||||
|
| `pipeline.render.title` | string | No | empty (falls back to `session.title` when set) |
|
||||||
|
| `pipeline.render.include_timestamps` | bool | No | `true` |
|
||||||
|
| `pipeline.render.include_segment_ids` | bool | No | `true` |
|
||||||
|
| `pipeline.render.include_metadata` | bool | No | `false` |
|
||||||
| `pipeline.scriptorium.binary` | string | No | `scriptorium` |
|
| `pipeline.scriptorium.binary` | string | No | `scriptorium` |
|
||||||
| `pipeline.scriptorium.config_path` | string | No | empty |
|
| `pipeline.scriptorium.config_path` | string | No | empty |
|
||||||
| `pipeline.scriptorium.timeout` | duration string | No | `10m` |
|
| `pipeline.scriptorium.timeout` | duration | No | `10m` |
|
||||||
| `pipeline.scriptorium.render_debug` | bool | No | `false` |
|
| `pipeline.scriptorium.render_debug` | bool | No | `false` |
|
||||||
| `pipeline.scriptorium.artifacts` | map | No | empty |
|
| `pipeline.scriptorium.artifacts` | map | No | empty |
|
||||||
| `pipeline.notification.backend` | string | No | empty |
|
| `pipeline.notification.backend` | string | No | empty |
|
||||||
| `pipeline.notification.recipient` | string | No | empty |
|
| `pipeline.notification.recipient` | string | No | empty |
|
||||||
| `pipeline.notification.timeout` | duration string | No | `30s` |
|
| `pipeline.notification.timeout` | duration | No | empty |
|
||||||
|
|
||||||
|
### Notarius Output Entries
|
||||||
|
|
||||||
|
For each `pipeline.notarius.outputs.<name>`:
|
||||||
|
|
||||||
|
| Field | Type | Required | Rule |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `lane_id` | string | Yes | unique Notarius lane ID |
|
||||||
|
| `media_type` | string | Yes | exact accepted descriptor media type |
|
||||||
|
| `schema_id` | string | Yes | exact accepted descriptor schema ID |
|
||||||
|
| `schema_version` | string | Yes | exact accepted descriptor schema version |
|
||||||
|
| `module_key` | string | No | exact accepted module key when set |
|
||||||
|
|
||||||
|
Output names must match `^[a-z][a-z0-9_]*$` and become selectable sources named
|
||||||
|
`narratio.extraction.<name>`. Lane IDs must be unique. Every declared output is
|
||||||
|
required from a successful Notarius result; a missing, rejected, duplicate, or
|
||||||
|
contract-incompatible lane fails extraction. See the
|
||||||
|
[complete maintained example](../examples/pipeline.full.annotated.yml) for the
|
||||||
|
current ten-lane D&D mapping and the [Notarius contract](./integrations/notarius.md)
|
||||||
|
for compatibility ownership.
|
||||||
|
|
||||||
|
### Scriptorium Artifact Entries
|
||||||
|
|
||||||
|
For each `pipeline.scriptorium.artifacts.<name>`:
|
||||||
|
|
||||||
|
| Field | Type | Required | Rule |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `enabled` | bool | No | `false` if omitted |
|
||||||
|
| `depends_on[]` | list[string] | No | must reference configured artifact keys; no self-reference; enabled graph must be acyclic |
|
||||||
|
| `render_debug` | bool | No | per-artifact override |
|
||||||
|
| `prompt_id` | string | Conditional | required when artifact is enabled |
|
||||||
|
| `profile_id` | string | No | empty |
|
||||||
|
| `output_path` | string | Conditional | required when enabled; also required when referenced by publish/output/input rules |
|
||||||
|
| `timeout` | duration | No | artifact override |
|
||||||
|
| `inputs` | map | No | input key names must be non-empty |
|
||||||
|
| `vars` | map | No | values must be string or bool; `session_id` is reserved and overwritten by Narratio |
|
||||||
|
|
||||||
|
Narratio adds `session_id=narratio-session-<session_id>` to every Scriptorium request for sticky upstream LLM routing. If an artifact config sets `vars.session_id`, Narratio replaces that value before invoking Scriptorium. Use a different variable name if a prompt needs the raw Narratio session ID as content.
|
||||||
|
|
||||||
|
For each artifact input `pipeline.scriptorium.artifacts.<name>.inputs.<input_name>`:
|
||||||
|
|
||||||
|
| Field | Type | Required | Rule |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `source` | string | Yes | built-in runtime source, prepared input source, `narratio.extraction.<name>`, `narratio.artifact.<name>`, or `narratio.previous_session.artifact.<name>` |
|
||||||
|
| `artifact` | string | No | optional passthrough adapter field |
|
||||||
|
| `path` | string | No | optional passthrough adapter field |
|
||||||
|
| `required` | bool | No | optional input requirement |
|
||||||
|
|
||||||
### Campaign
|
### Campaign
|
||||||
| Path | Type | Required |
|
|
||||||
| --- | --- | --- |
|
| Field | Type | Required | Notes |
|
||||||
| `campaign_id` | string | Yes |
|
| --- | --- | --- | --- |
|
||||||
| `session_template_file` | string | No |
|
| `campaign_id` | string | Yes | canonical campaign identity |
|
||||||
| `inputs.speakers_file` | string | Yes |
|
| `session_template_file` | string | No | used by `session init` when set |
|
||||||
| `inputs.autocorrect_file` | string | Yes |
|
| `inputs.speakers_file` | string | Yes | stable input default |
|
||||||
| `inputs.glossary_file` | string | Yes |
|
| `inputs.autocorrect_file` | string | Yes | stable input default |
|
||||||
|
| `inputs.glossary_file` | string | Yes | stable input default |
|
||||||
|
| `inputs.players_file` | string | Yes | stable input default |
|
||||||
|
| `inputs.party_file` | string | Yes | stable input default |
|
||||||
|
|
||||||
### Session
|
### Session
|
||||||
| Path | Type | Required |
|
|
||||||
| --- | --- | --- |
|
| Field | Type | Required in session file | Notes |
|
||||||
| `session_id` | string | Yes |
|
| --- | --- | --- | --- |
|
||||||
| `previous_session_id` | string | No |
|
| `session_id` | string | Yes | must match CLI session target when provided |
|
||||||
| `campaign` | string | No |
|
| `previous_session_id` | string | No | must not equal `session_id` |
|
||||||
| `date` | string | No |
|
| `campaign` | string | No | filled from `campaign_id` during resolve if omitted |
|
||||||
| `title` | string | No |
|
| `date` | string | No | metadata |
|
||||||
| `inputs.speakers_file` | string | No |
|
| `title` | string | No | metadata |
|
||||||
| `inputs.autocorrect_file` | string | No |
|
| `inputs.speakers_file` | string | No | overrides campaign stable input |
|
||||||
| `inputs.glossary_file` | string | No |
|
| `inputs.autocorrect_file` | string | No | overrides campaign stable input |
|
||||||
| `inputs.audio_dir` | string | Conditional |
|
| `inputs.glossary_file` | string | No | overrides campaign stable input |
|
||||||
| `inputs.audio_files[]` | list[string] | Conditional |
|
| `inputs.players_file` | string | No | overrides campaign stable input |
|
||||||
| `inputs.audio_s3.prefix` | string | Conditional |
|
| `inputs.party_file` | string | No | overrides campaign stable input |
|
||||||
|
| `inputs.audio_dir` | string | Conditional | local audio mode |
|
||||||
|
| `inputs.audio_files[]` | list[string] | Conditional | local audio mode |
|
||||||
|
| `inputs.audio_s3.prefix` | string | Conditional | S3 audio mode |
|
||||||
|
|
||||||
|
Audio rules:
|
||||||
|
|
||||||
|
- configure local mode (`audio_dir` or `audio_files`) or S3 mode (`audio_s3.prefix`), not both.
|
||||||
|
|
||||||
## Maintained Examples
|
## Maintained Examples
|
||||||
- `examples/pipeline.minimal.yml`
|
|
||||||
- `examples/pipeline.production.yml`
|
See the [maintained examples index](../examples/README.md) for complete pipeline,
|
||||||
- `examples/pipeline.full.annotated.yml`
|
campaign, session, template, and input fixtures. Keep complete copyable files
|
||||||
- `examples/campaigns/sample-campaign/campaign.yml`
|
there rather than duplicating them in this reference.
|
||||||
- `examples/session.local-audio.yml`
|
|
||||||
- `examples/session.s3-audio.yml`
|
|
||||||
|
|||||||
@@ -1,94 +1,43 @@
|
|||||||
# Development Guide
|
# Development
|
||||||
|
|
||||||
## Purpose
|
This is the first-read landing page for people and LLM coding agents working on
|
||||||
Canonical contributor workflow and engineering conventions for implemented Narratio behavior.
|
Narratio. It provides a concise repository orientation and routes each kind of
|
||||||
|
change to its canonical documentation.
|
||||||
|
|
||||||
## Repository layout
|
Narratio is a stage-driven Go orchestrator for turning D&D session audio into
|
||||||
|
polished transcripts and generated artifacts. Start with the
|
||||||
|
[README](../README.md) for product context,
|
||||||
|
[Architecture](policy/architecture.md) for normative system boundaries, and the
|
||||||
|
[Internal Overview](internal/overview.md) for implemented component ownership.
|
||||||
|
|
||||||
- `cmd/narratio/`: CLI entrypoint.
|
## What To Read
|
||||||
- `internal/app/`: command handlers, plan/run/resume orchestration, cleanup gates, secrets loading.
|
|
||||||
- `internal/config/`: strict YAML loading, defaults, and validation.
|
|
||||||
- `internal/stage/`: stage implementations and stage registry/order.
|
|
||||||
- `internal/adapters/`: external boundary adapters (WhisperX, Seriatim, Audita, Scriptorium, storage, notify).
|
|
||||||
- `internal/manifest/`: session/run manifest types and persistence.
|
|
||||||
- `internal/artifacts/`: canonical local/remote path helpers and local artifact store.
|
|
||||||
- `docs/`: canonical documentation set.
|
|
||||||
- `examples/`: maintained config examples used by tests.
|
|
||||||
|
|
||||||
## Build and test commands
|
| When working on | Read | Why |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Finding the package or component that owns current behavior | [Internal Overview](internal/overview.md) | It is the implemented component inventory and routes to focused internal documents. |
|
||||||
|
| Application shape, boundaries, dependency direction, runtime invariants, safety properties, or dependencies | [Architecture](policy/architecture.md) | It defines the intended system shape, ownership, and non-goals. |
|
||||||
|
| Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical owners, audiences, current-behavior rules, and maintenance requirements. |
|
||||||
|
| Adding, changing, reviewing, rewriting, or deleting tests | [Testing Policy](policy/testing.md) | It defines risk-based sufficiency, durable test boundaries, test-double guidance, and test lifecycle decisions. |
|
||||||
|
| CLI composition or command behavior | [Internal Overview](internal/overview.md) and [CLI Reference](cli.md) | The overview routes to command ownership; the reference owns public syntax and invocation behavior. |
|
||||||
|
| Configuration loading, resolution, or user-visible configuration | [Internal Overview](internal/overview.md) and [Configuration](config.md) | The overview routes to implementation ownership; the reference owns fields, defaults, discovery, and validation. |
|
||||||
|
| Session workflow, status, restore, cleanup, or object storage | [Restore Internals](internal/command-restore.md), [Workspace Internals](internal/workspace.md), [Storage Internals](internal/storage.md), [Operations](operations.md), and [Troubleshooting](troubleshooting.md) | These separate implementation mechanics, operator procedures, and symptom-driven recovery. |
|
||||||
|
| Pipeline sequencing or the behavior of a stage | [Internal Overview](internal/overview.md) and its focused stage documents | The overview owns the implemented stage inventory and routes to each stage contract. |
|
||||||
|
| Adapters or external tool contracts | [Adapter Internals](internal/adapters.md) and [Integration Contracts](integrations/README.md) | The internal guide owns adapter composition and mechanics; integration documents own external formats and protocols. |
|
||||||
|
| Manifests, artifacts, workspace paths, or publish behavior | [Manifest Internals](internal/manifest.md), [Artifact Internals](internal/artifacts.md), [Workspace Internals](internal/workspace.md), [Publish Internals](internal/stage-publish.md), and [Operations](operations.md) | These separate implementation state and resolution from operator-visible layout and lifecycle. |
|
||||||
|
| Maintained configuration or input examples | [Configuration](config.md) and [Examples](../examples/README.md) | The reference owns field meanings; the examples directory owns complete copyable files. |
|
||||||
|
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
|
||||||
|
|
||||||
- Run focused CLI behavior checks:
|
For an existing subsystem, also inspect its focused tests and package-level
|
||||||
|
contracts before changing behavior.
|
||||||
|
|
||||||
```bash
|
## Validation
|
||||||
go test ./internal/app -run TestExecute -v
|
|
||||||
```
|
|
||||||
|
|
||||||
- Run config example load/validate checks:
|
Use focused package tests while iterating. Run the repository-wide checks when a
|
||||||
|
change affects shared contracts, application behavior, or maintained
|
||||||
|
documentation examples:
|
||||||
|
|
||||||
```bash
|
```sh
|
||||||
go test ./internal/config -run TestExamplesLoadAndValidate -v
|
|
||||||
```
|
|
||||||
|
|
||||||
- Run full test suite:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
go test ./...
|
go test ./...
|
||||||
|
go vet ./...
|
||||||
|
go build ./cmd/narratio
|
||||||
```
|
```
|
||||||
|
|
||||||
## Coding conventions
|
|
||||||
|
|
||||||
- Keep orchestration explicit and stage-driven; do not introduce generic workflow/DAG abstractions.
|
|
||||||
- Keep external-system details inside adapter packages; stages should consume Narratio-level contracts only.
|
|
||||||
- Use centralized path helpers from `internal/artifacts` rather than ad hoc path concatenation.
|
|
||||||
- Preserve manifest-driven state transitions (`running`, `succeeded`, `failed`, `skipped`, `stale`) as the source of run progress.
|
|
||||||
- Keep user/operator docs implementation-accurate; planned work belongs only under `docs/roadmap/`.
|
|
||||||
|
|
||||||
For design principles and invariants, see [docs/architecture.md](./architecture.md). For stage/adapter contracts, see [docs/internal/README.md](./internal/README.md).
|
|
||||||
|
|
||||||
## Dependency policy
|
|
||||||
|
|
||||||
- Prefer Go standard library where practical.
|
|
||||||
- Add third-party dependencies only when they provide clear value for required behavior.
|
|
||||||
- Keep dependency additions narrow to the boundary package that needs them.
|
|
||||||
|
|
||||||
## Change playbooks
|
|
||||||
|
|
||||||
### Add config fields
|
|
||||||
|
|
||||||
1. Add fields to config structs in `internal/config`.
|
|
||||||
2. Set defaults in `internal/config/defaults.go` when appropriate.
|
|
||||||
3. Add validation rules in `internal/config/validate.go`.
|
|
||||||
4. Add or update load/validate tests in `internal/config/*_test.go`.
|
|
||||||
5. Update canonical config docs and examples:
|
|
||||||
- [docs/config.md](./config.md)
|
|
||||||
- relevant files under `examples/`
|
|
||||||
|
|
||||||
### Add CLI flags or commands
|
|
||||||
|
|
||||||
1. Update command parsing and behavior in `internal/app`.
|
|
||||||
2. Add or update command tests (`TestExecute` and command-specific tests).
|
|
||||||
3. Update [docs/cli.md](./cli.md) and, if operator workflow changes, [docs/operations.md](./operations.md).
|
|
||||||
|
|
||||||
Remote-storage commands must obtain object storage through the app-level command object-store helper. Do not call `storage.NewObjectStoreFromConfig` directly from command handlers; the helper loads configured filesystem secrets before constructing the storage adapter.
|
|
||||||
|
|
||||||
### Add or modify stages/adapters
|
|
||||||
|
|
||||||
1. Implement stage behavior in `internal/stage` with clear input/output boundaries.
|
|
||||||
2. Keep external transport/subprocess details in `internal/adapters`.
|
|
||||||
3. Preserve manifest and publish-output semantics expected by runner and publish logic.
|
|
||||||
4. Add/update stage and adapter tests.
|
|
||||||
5. Update internal component contracts in `docs/internal/`.
|
|
||||||
|
|
||||||
### Update examples
|
|
||||||
|
|
||||||
1. Keep canonical examples only in `examples/`.
|
|
||||||
2. Ensure examples load and validate through runtime config paths.
|
|
||||||
3. Update `internal/config/load_validate_test.go` as needed.
|
|
||||||
4. Update links in `docs/config.md` if example filenames change.
|
|
||||||
|
|
||||||
### Update docs and roadmap
|
|
||||||
|
|
||||||
1. Keep implemented behavior in canonical docs (`README`, `docs/*.md`, `docs/internal/`).
|
|
||||||
2. Keep planned/unimplemented behavior only in `docs/roadmap/`.
|
|
||||||
3. After completing roadmap items, remove or mark them complete in `docs/roadmap/documentation.md`.
|
|
||||||
4. Run a link/path sweep before finalizing changes.
|
|
||||||
|
|||||||
@@ -1,356 +0,0 @@
|
|||||||
# Go Project Documentation Policy
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
Project documentation must help four audiences:
|
|
||||||
|
|
||||||
1. users who need to run the application;
|
|
||||||
2. administrators/operators who need to configure and operate it;
|
|
||||||
3. developers who need to understand and change it safely;
|
|
||||||
4. LLM coding agents that need clear scope, boundaries, and invariants.
|
|
||||||
|
|
||||||
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
|
|
||||||
|
|
||||||
## Core Rules
|
|
||||||
|
|
||||||
### 1. Keep docs concise
|
|
||||||
|
|
||||||
Each document should cover a defined scope and only the essentials for that scope.
|
|
||||||
|
|
||||||
Avoid:
|
|
||||||
- long background explanations;
|
|
||||||
- repeated reference material;
|
|
||||||
- implementation detail in user-facing docs;
|
|
||||||
- aspirational language outside roadmap docs;
|
|
||||||
- verbose examples where one minimal example is clearer.
|
|
||||||
|
|
||||||
### 2. Document only implemented behavior outside roadmap files
|
|
||||||
|
|
||||||
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
|
|
||||||
|
|
||||||
- `docs/roadmap/`
|
|
||||||
|
|
||||||
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
|
|
||||||
|
|
||||||
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
|
|
||||||
|
|
||||||
### 3. Use canonical homes
|
|
||||||
|
|
||||||
Each type of information should have one canonical location.
|
|
||||||
|
|
||||||
Canonical homes:
|
|
||||||
|
|
||||||
- project purpose and quickstart: `README.md`
|
|
||||||
- development principles: `docs/architecture.md`
|
|
||||||
- configuration reference: `docs/config.md`
|
|
||||||
- CLI reference: `docs/cli.md`
|
|
||||||
- operations and recovery: `docs/operations.md`
|
|
||||||
- troubleshooting: `docs/troubleshooting.md`
|
|
||||||
- implemented internals: `docs/internal/`
|
|
||||||
- future work: `docs/roadmap/`
|
|
||||||
- contributor workflow: `docs/development.md`
|
|
||||||
- copyable examples: `examples/`
|
|
||||||
|
|
||||||
Other files should summarize briefly and link to the canonical source.
|
|
||||||
|
|
||||||
### 4. Keep examples real
|
|
||||||
|
|
||||||
Examples should be valid, maintained, and free of secrets.
|
|
||||||
|
|
||||||
Where practical:
|
|
||||||
- example configs should load successfully;
|
|
||||||
- example commands should match real CLI syntax;
|
|
||||||
- important examples should be covered by tests.
|
|
||||||
|
|
||||||
## Documentation Profiles
|
|
||||||
|
|
||||||
All projects require:
|
|
||||||
|
|
||||||
- `README.md`
|
|
||||||
- `docs/architecture.md`
|
|
||||||
|
|
||||||
Additional docs depend on the project.
|
|
||||||
|
|
||||||
### Small library
|
|
||||||
|
|
||||||
Recommended:
|
|
||||||
- `docs/development.md`, if contributor conventions are non-obvious
|
|
||||||
|
|
||||||
### Simple CLI
|
|
||||||
|
|
||||||
Required:
|
|
||||||
- `docs/cli.md`
|
|
||||||
|
|
||||||
Recommended:
|
|
||||||
- `docs/development.md`
|
|
||||||
|
|
||||||
### Config-driven CLI
|
|
||||||
|
|
||||||
Required:
|
|
||||||
- `docs/cli.md`
|
|
||||||
- `docs/config.md`
|
|
||||||
|
|
||||||
Recommended:
|
|
||||||
- `examples/`
|
|
||||||
- `docs/development.md`
|
|
||||||
|
|
||||||
### Stateful or operator-facing application
|
|
||||||
|
|
||||||
Required:
|
|
||||||
- `docs/cli.md`, if CLI-based
|
|
||||||
- `docs/config.md`, if config-driven
|
|
||||||
- `docs/operations.md`
|
|
||||||
|
|
||||||
Recommended:
|
|
||||||
- `docs/troubleshooting.md`
|
|
||||||
- `examples/`
|
|
||||||
- `docs/development.md`
|
|
||||||
|
|
||||||
### Modular, staged, service-oriented, or orchestration application
|
|
||||||
|
|
||||||
Required:
|
|
||||||
- `docs/cli.md`, if CLI-based
|
|
||||||
- `docs/config.md`, if config-driven
|
|
||||||
- `docs/operations.md`
|
|
||||||
- `docs/internal/`
|
|
||||||
- `docs/development.md`
|
|
||||||
|
|
||||||
Recommended:
|
|
||||||
- `docs/troubleshooting.md`
|
|
||||||
- validated examples under `examples/`
|
|
||||||
|
|
||||||
## Required Documents
|
|
||||||
|
|
||||||
### README.md
|
|
||||||
|
|
||||||
**Audience:** users, administrators, operators
|
|
||||||
|
|
||||||
The README is the outward-facing project orientation page.
|
|
||||||
|
|
||||||
It should include, in order:
|
|
||||||
|
|
||||||
1. concise description;
|
|
||||||
2. elevator pitch;
|
|
||||||
3. shortest useful command or usage example;
|
|
||||||
4. links to targeted docs.
|
|
||||||
|
|
||||||
The README should be short. It is not a manual.
|
|
||||||
|
|
||||||
The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.)
|
|
||||||
|
|
||||||
### docs/architecture.md
|
|
||||||
|
|
||||||
**Audience:** developers, LLM coding agents
|
|
||||||
|
|
||||||
`docs/architecture.md` is required for every project.
|
|
||||||
|
|
||||||
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
|
|
||||||
|
|
||||||
It should include:
|
|
||||||
|
|
||||||
- project shape;
|
|
||||||
- core design principles;
|
|
||||||
- package and boundary philosophy;
|
|
||||||
- state/persistence philosophy, if applicable;
|
|
||||||
- external integration philosophy, if applicable;
|
|
||||||
- error-handling and logging principles;
|
|
||||||
- testing expectations;
|
|
||||||
- documentation expectations;
|
|
||||||
- architectural invariants;
|
|
||||||
- explicit non-goals, if useful.
|
|
||||||
|
|
||||||
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
|
|
||||||
|
|
||||||
### docs/config.md
|
|
||||||
|
|
||||||
**Audience:** administrators, operators, advanced users
|
|
||||||
|
|
||||||
Required for applications with configuration files.
|
|
||||||
|
|
||||||
It should include, in order:
|
|
||||||
|
|
||||||
1. config file locations and discovery precedence;
|
|
||||||
2. minimal working config;
|
|
||||||
3. production-oriented config;
|
|
||||||
4. full configuration reference;
|
|
||||||
5. secrets handling, if applicable;
|
|
||||||
6. links to maintained examples.
|
|
||||||
|
|
||||||
The full configuration reference should be canonical.
|
|
||||||
|
|
||||||
### docs/cli.md
|
|
||||||
|
|
||||||
**Audience:** users, administrators, operators
|
|
||||||
|
|
||||||
Required for CLI applications.
|
|
||||||
|
|
||||||
It should include, in order:
|
|
||||||
|
|
||||||
1. shortest useful command;
|
|
||||||
2. command overview;
|
|
||||||
3. complete flag reference;
|
|
||||||
4. common workflows;
|
|
||||||
5. diagnostic or recovery commands, if applicable.
|
|
||||||
|
|
||||||
Explain when commands are useful, not just their syntax.
|
|
||||||
|
|
||||||
### docs/operations.md
|
|
||||||
|
|
||||||
**Audience:** administrators, operators
|
|
||||||
|
|
||||||
Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures.
|
|
||||||
|
|
||||||
It should cover:
|
|
||||||
|
|
||||||
- normal workflow;
|
|
||||||
- filesystem layout;
|
|
||||||
- remote storage layout, if applicable;
|
|
||||||
- logs and manifests;
|
|
||||||
- resume/retry behavior;
|
|
||||||
- cleanup behavior;
|
|
||||||
- archive/backup behavior;
|
|
||||||
- safe recovery procedures;
|
|
||||||
- operational caveats.
|
|
||||||
|
|
||||||
### docs/troubleshooting.md
|
|
||||||
|
|
||||||
**Audience:** administrators, operators
|
|
||||||
|
|
||||||
Recommended once recurring failure modes exist.
|
|
||||||
|
|
||||||
Each entry should include:
|
|
||||||
|
|
||||||
- symptom;
|
|
||||||
- likely cause;
|
|
||||||
- diagnostic command or inspection step;
|
|
||||||
- safe fix;
|
|
||||||
- relevant links.
|
|
||||||
|
|
||||||
### docs/development.md
|
|
||||||
|
|
||||||
**Audience:** developers, LLM coding agents
|
|
||||||
|
|
||||||
Required for projects maintained by humans and LLM coding agents.
|
|
||||||
|
|
||||||
It should include:
|
|
||||||
|
|
||||||
- repository layout;
|
|
||||||
- build/test commands;
|
|
||||||
- coding conventions;
|
|
||||||
- dependency policy;
|
|
||||||
- how to add config fields;
|
|
||||||
- how to add CLI flags;
|
|
||||||
- how to add stages/modules/adapters, if applicable;
|
|
||||||
- how to update examples;
|
|
||||||
- documentation update expectations.
|
|
||||||
|
|
||||||
### docs/internal/
|
|
||||||
|
|
||||||
**Audience:** developers, LLM coding agents
|
|
||||||
|
|
||||||
Required for modular, staged, service-oriented, or orchestration projects.
|
|
||||||
|
|
||||||
This directory describes implemented internal components. It is not the roadmap.
|
|
||||||
|
|
||||||
Use one file per major component where useful.
|
|
||||||
|
|
||||||
Each component doc should include:
|
|
||||||
|
|
||||||
1. purpose;
|
|
||||||
2. inputs and outputs;
|
|
||||||
3. boundaries;
|
|
||||||
4. config fields used;
|
|
||||||
5. external adapters used;
|
|
||||||
6. state or manifest behavior, if applicable;
|
|
||||||
7. skip/resume behavior, if applicable;
|
|
||||||
8. failure behavior;
|
|
||||||
9. tests to inspect before changing;
|
|
||||||
10. architectural invariants.
|
|
||||||
|
|
||||||
### docs/roadmap/
|
|
||||||
|
|
||||||
**Audience:** maintainers, developers, LLM coding agents
|
|
||||||
|
|
||||||
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
|
|
||||||
|
|
||||||
Roadmap docs should clearly distinguish:
|
|
||||||
|
|
||||||
- proposed work;
|
|
||||||
- accepted plans;
|
|
||||||
- deferred ideas;
|
|
||||||
- rejected ideas;
|
|
||||||
- implementation prompts or task breakdowns, if useful.
|
|
||||||
|
|
||||||
Roadmap docs should not be confused with current behavior.
|
|
||||||
|
|
||||||
### docs/integrations/
|
|
||||||
|
|
||||||
**Audience:** developers, LLM coding agents
|
|
||||||
|
|
||||||
Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
|
|
||||||
|
|
||||||
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses.
|
|
||||||
|
|
||||||
Use one file per integration where useful.
|
|
||||||
|
|
||||||
## Examples Directory
|
|
||||||
|
|
||||||
Projects with non-trivial configuration or workflows should include `examples/`.
|
|
||||||
|
|
||||||
Useful examples include:
|
|
||||||
|
|
||||||
- minimal working config;
|
|
||||||
- production-oriented config;
|
|
||||||
- full annotated config;
|
|
||||||
- local development config;
|
|
||||||
- remote/object-storage config;
|
|
||||||
- minimal session/input file.
|
|
||||||
|
|
||||||
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
|
|
||||||
|
|
||||||
## Security and Privacy
|
|
||||||
|
|
||||||
Docs and examples must not include:
|
|
||||||
|
|
||||||
- real API keys;
|
|
||||||
- tokens;
|
|
||||||
- passwords;
|
|
||||||
- private keys;
|
|
||||||
- private environment dumps;
|
|
||||||
- sensitive user data;
|
|
||||||
- raw private transcripts;
|
|
||||||
- private infrastructure details unless intentionally public.
|
|
||||||
|
|
||||||
Document secret-handling mechanisms, not actual secret values.
|
|
||||||
|
|
||||||
## Maintenance Rules
|
|
||||||
|
|
||||||
When docs change, verify the affected behavior.
|
|
||||||
|
|
||||||
Where practical:
|
|
||||||
|
|
||||||
- load example config files in tests;
|
|
||||||
- test CLI examples or command parser behavior;
|
|
||||||
- validate documented flags against real flags;
|
|
||||||
- remove stale references;
|
|
||||||
- update links after renames;
|
|
||||||
- keep roadmap content out of non-roadmap docs.
|
|
||||||
|
|
||||||
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
|
|
||||||
|
|
||||||
Documentation is complete only when it matches the current code.
|
|
||||||
|
|
||||||
## Documentation Change Checklist
|
|
||||||
|
|
||||||
Before merging documentation changes, verify:
|
|
||||||
|
|
||||||
- README is concise and orientation-focused.
|
|
||||||
- `docs/architecture.md` describes development principles.
|
|
||||||
- Future work appears only under `docs/roadmap/`.
|
|
||||||
- User-facing docs avoid unnecessary internals.
|
|
||||||
- Developer-facing docs preserve boundaries and invariants.
|
|
||||||
- Config examples match the schema.
|
|
||||||
- CLI examples match real commands and flags.
|
|
||||||
- Defaults appear in the canonical config reference.
|
|
||||||
- No secrets or private data are included.
|
|
||||||
- Links are accurate.
|
|
||||||
@@ -1,15 +1,35 @@
|
|||||||
# Integration Documentation Index
|
# Integrations Index
|
||||||
|
|
||||||
## Audience
|
## Audience
|
||||||
Developers and LLM coding agents changing Narratio's external integration contracts.
|
|
||||||
|
Operators, developers, and coding agents who need to understand Narratio's
|
||||||
|
externally observable integration boundaries.
|
||||||
|
|
||||||
## Scope
|
## Scope
|
||||||
Implemented-only reference notes for the external systems Narratio currently integrates with.
|
|
||||||
|
|
||||||
## Integration Docs
|
`docs/integrations/` is the canonical reference for protocols, invocation and
|
||||||
- `audita.md`: Audita adapter invocation and validation contract.
|
data contracts, logical outputs, and compatibility behavior at external tool
|
||||||
- `seriatim.md`: Seriatim normalize/merge/trim adapter contract.
|
boundaries.
|
||||||
- `scriptorium.md`: Scriptorium run/render adapter contract.
|
|
||||||
|
|
||||||
## Canonical Owner
|
These documents describe what Narratio sends or invokes, what it accepts in
|
||||||
`docs/integrations/` is the canonical home for external integration reference notes per `docs/documentation/policy.md`.
|
return, and how failures are surfaced. Internal composition and stage mechanics
|
||||||
|
belong in [the adapter implementation guide](../internal/adapters.md) and the
|
||||||
|
focused stage documents.
|
||||||
|
|
||||||
|
## Integration Contracts
|
||||||
|
|
||||||
|
- [Audita](./audita.md): transcript polishing (`audita process`).
|
||||||
|
- [Notarius](./notarius.md): complete pipeline execution and safe JSON bundle
|
||||||
|
discovery (`notarius run`).
|
||||||
|
- [Seriatim](./seriatim.md): merge, normalize, trim, and render operations.
|
||||||
|
- [Scriptorium](./scriptorium.md): artifact generation and debug rendering
|
||||||
|
(`scriptorium run|render`).
|
||||||
|
- [WhisperX](./whisperx.md): speaker-audio transcription over HTTP.
|
||||||
|
|
||||||
|
## Related Canonical Docs
|
||||||
|
|
||||||
|
- [Configuration](../config.md): operator-facing configuration reference.
|
||||||
|
- [Adapter implementation](../internal/adapters.md): shared adapter boundary and
|
||||||
|
runner wiring.
|
||||||
|
- [Internal documentation](../internal/overview.md): stage-specific integration
|
||||||
|
usage and component ownership.
|
||||||
|
|||||||
@@ -1,66 +1,59 @@
|
|||||||
# Integration: audita
|
# Integration: Audita
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define Narratio's adapter contract for transcript polishing via Audita CLI subprocess execution.
|
Define the Audita adapter contract used by the `polish` stage.
|
||||||
|
|
||||||
## Inputs and Outputs
|
## External Boundary
|
||||||
Inputs (`audita.PolishRequest`):
|
|
||||||
- base transcript path
|
|
||||||
- glossary path
|
|
||||||
- output polished transcript path
|
|
||||||
- optional report path (required when report enabled)
|
|
||||||
- work dir
|
|
||||||
- generated config path
|
|
||||||
- stdout/stderr log paths
|
|
||||||
- optional module/model/base URL and concurrency knobs
|
|
||||||
|
|
||||||
Outputs (`audita.PolishResult`):
|
Narratio invokes `audita process` as a subprocess for each polish operation.
|
||||||
- polished transcript path
|
The configured timeout and parent cancellation bound the invocation. Internal
|
||||||
- optional report path
|
runner composition is documented in
|
||||||
- generated config path
|
[the adapter implementation guide](../internal/adapters.md).
|
||||||
- stdout/stderr log paths
|
|
||||||
- exit code, duration, invoked binary
|
|
||||||
- adapter metadata
|
|
||||||
|
|
||||||
## Boundaries
|
## Request Contract
|
||||||
Owns:
|
`PolishRequest` carries:
|
||||||
- Deterministic CLI argument construction for `audita process`
|
- required transcript/glossary/output/work-dir paths;
|
||||||
- Environment bridging for API credentials
|
- optional report path (required when report mode is enabled);
|
||||||
- Invocation config emission
|
- generated config and stdout/stderr log paths;
|
||||||
- Output validation for polished transcript and report
|
- optional module/model/base-url/config/output-schema/concurrency settings.
|
||||||
|
|
||||||
Does not own:
|
## Result Contract
|
||||||
- Upstream/downstream stage orchestration
|
`PolishResult` returns:
|
||||||
- Credential sourcing policy beyond required env-var presence check
|
- processed transcript path;
|
||||||
|
- optional report path;
|
||||||
|
- work dir and generated-config/log paths;
|
||||||
|
- exit code, duration, binary provenance;
|
||||||
|
- adapter metadata map.
|
||||||
|
|
||||||
## Config Fields Used
|
## Validation and Failure Semantics
|
||||||
Via `pipeline.audita.*` mapped in app/stage wiring:
|
Construction fails for invalid static config values, including:
|
||||||
- `binary`, `timeout`, `llm_api_key_env`, `modules`, `base_url`, `model`
|
- empty binary;
|
||||||
- `transcript_description`, `config_path`, `output_schema`, `work_dir_retention`
|
- non-positive timeout;
|
||||||
- `total_llm_concurrency`, `proposal_llm_concurrency`, `validation_model`, `validation_llm_concurrency`, `report`
|
- invalid base URL;
|
||||||
|
- invalid output schema;
|
||||||
|
- invalid work-dir retention value;
|
||||||
|
- invalid concurrency values.
|
||||||
|
|
||||||
## External Adapters Used
|
Run fails for:
|
||||||
- Shared subprocess helper (`internal/adapters/subprocess`) to run CLI and capture logs.
|
- missing required request paths;
|
||||||
|
- missing required credential env var when configured (`llm_api_key_env`);
|
||||||
|
- subprocess execution failure;
|
||||||
|
- invalid processed transcript JSON (`segments` array required);
|
||||||
|
- invalid report JSON when reporting is enabled.
|
||||||
|
|
||||||
## State and Manifest Behavior
|
Failure results still include output/log/config/exit metadata for diagnostics.
|
||||||
- No direct manifest writes.
|
|
||||||
- Stage-level metadata records adapter provenance and credential-present signal.
|
|
||||||
- Generated invocation YAML is written when `GeneratedConfigPath` is provided.
|
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
## Deterministic Behavior
|
||||||
- Adapter has no skip/resume logic. Stage/runner controls this.
|
- CLI args are built from runner config + request in a fixed order.
|
||||||
|
- Generated invocation YAML (`audita.generated.v1`) is emitted when requested.
|
||||||
|
- Manifest writes are stage-owned; adapter itself is stateless.
|
||||||
|
|
||||||
## Failure Behavior
|
## Configuration
|
||||||
- Constructor validation fails on invalid binary/timeout/schema/concurrency/URL values.
|
|
||||||
- Run fails on missing required paths, missing required credential env var, subprocess errors, invalid polished JSON shape, or invalid report JSON.
|
|
||||||
- Failures preserve stdout/stderr paths in returned result metadata.
|
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
Operator-selected values are defined under `pipeline.audita.*` in the
|
||||||
- `internal/adapters/audita/subprocess_test.go`
|
[configuration reference](../config.md#pipeline).
|
||||||
- `internal/adapters/audita/fake_test.go`
|
|
||||||
- `internal/stage/polish_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
Maintained example with Audita config:
|
||||||
- Polished output must be valid JSON with top-level `segments` array.
|
|
||||||
- When report is enabled, report output must be valid JSON.
|
- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml)
|
||||||
- If `llm_api_key_env` is configured, credential must be present in environment.
|
- [Production-shaped pipeline](../../examples/pipeline.production.yml)
|
||||||
|
|||||||
83
docs/integrations/notarius.md
Normal file
83
docs/integrations/notarius.md
Normal file
@@ -0,0 +1,83 @@
|
|||||||
|
# Notarius Integration Contract
|
||||||
|
|
||||||
|
## Boundary
|
||||||
|
|
||||||
|
Narratio uses Notarius as a subprocess to extract configured structured JSON
|
||||||
|
lanes from the final trimmed Seriatim transcript. Narratio owns invocation,
|
||||||
|
safe bundle discovery, lane selection, and its own artifact metadata. Notarius
|
||||||
|
owns pipeline definitions, lane schemas, the receipt, and bundle formats.
|
||||||
|
|
||||||
|
Canonical Notarius references:
|
||||||
|
|
||||||
|
- [Subprocess consumer contract](https://gitea.maximumdirect.net/eric/notarius/src/branch/main/docs/consumers/subprocess.md)
|
||||||
|
- [D&D pipeline and lane contracts](https://gitea.maximumdirect.net/eric/notarius/src/branch/main/docs/consumers/dnd-pipeline.md)
|
||||||
|
- [Run-result receipt](https://gitea.maximumdirect.net/eric/notarius/src/branch/main/docs/integrations/run-result.md)
|
||||||
|
- [JSON output bundle](https://gitea.maximumdirect.net/eric/notarius/src/branch/main/docs/integrations/json-output.md)
|
||||||
|
|
||||||
|
The [complete Narratio example](../../examples/pipeline.full.annotated.yml)
|
||||||
|
records the exact current constraints for all ten D&D lanes. Treat the linked
|
||||||
|
Notarius documents as canonical when changing those values; Narratio does not
|
||||||
|
duplicate the complete schemas.
|
||||||
|
|
||||||
|
## Invocation
|
||||||
|
|
||||||
|
When `pipeline.notarius.enabled` is true, Narratio resolves the executable,
|
||||||
|
configuration path, input path, output directory, and working directory to
|
||||||
|
absolute paths and invokes:
|
||||||
|
|
||||||
|
```text
|
||||||
|
notarius run <pipeline_id> --config <config_path> --input <trimmed_json> --output-dir <staging_dir> --json
|
||||||
|
```
|
||||||
|
|
||||||
|
Standard output is reserved for the JSON receipt. Standard error is captured
|
||||||
|
separately as diagnostic output. Narratio applies the configured timeout and
|
||||||
|
does not interpret stdout as a receipt unless the subprocess exits successfully.
|
||||||
|
It does not pass a Narratio session ID or run `notarius config validate`
|
||||||
|
automatically; the configured working directory and inherited environment
|
||||||
|
apply to the subprocess.
|
||||||
|
|
||||||
|
## Accepted Result
|
||||||
|
|
||||||
|
Narratio currently accepts receipt schema `notarius.run-result.v1`. The receipt
|
||||||
|
must identify the configured pipeline, and its `index_file` must be exactly
|
||||||
|
`index.json` beneath the reported bundle root. The production index must name
|
||||||
|
the management files exactly as `manifest.json`, `rejected.json`, and
|
||||||
|
`warnings.json`. All receipt, index, and lane paths must stay inside that
|
||||||
|
bundle; symlinks and non-regular lane payloads are rejected.
|
||||||
|
|
||||||
|
Supported receipt and index shapes tolerate unknown fields for forward
|
||||||
|
compatibility, while required identity, validation, count, manifest,
|
||||||
|
rejection, warning, and lane-list fields remain mandatory. Narratio applies
|
||||||
|
bounded reads to the receipt, index, rejection, and warning documents. Optional
|
||||||
|
chunk-map and evidence-context descriptors must carry their complete generic
|
||||||
|
contract metadata when present.
|
||||||
|
|
||||||
|
For every entry in `pipeline.notarius.outputs`, Narratio requires exactly one
|
||||||
|
index descriptor with the configured lane ID, media type, schema ID, schema
|
||||||
|
version, and, when configured, module key. Missing, duplicate, rejected, or
|
||||||
|
incompatible required lanes fail extraction even if Notarius exited zero.
|
||||||
|
Unconfigured lanes may remain in the preserved bundle but do not become
|
||||||
|
selectable Narratio sources.
|
||||||
|
|
||||||
|
Each accepted configured lane is registered as
|
||||||
|
`narratio.extraction.<output_key>`. The bundle index is retained for audit and
|
||||||
|
resume validation but is not selectable. Scriptorium and publish rules consume
|
||||||
|
only explicitly named lane sources; `--artifacts` never selects Notarius lanes.
|
||||||
|
|
||||||
|
## Failure And Compatibility Behavior
|
||||||
|
|
||||||
|
- Startup and nonzero-exit errors fail extraction and retain captured diagnostics.
|
||||||
|
- Invalid receipt JSON or an unsupported receipt schema fails before bundle use.
|
||||||
|
- Unsafe or incompatible index data and required-lane rejection fail before the
|
||||||
|
staged bundle is promoted to durable storage.
|
||||||
|
- Contract and external provenance metadata are preserved on lane artifact
|
||||||
|
records and through explicit publication.
|
||||||
|
|
||||||
|
Rejection and warning summaries retain structured stage, scope, lane, and
|
||||||
|
reason-code fields for diagnostics without exposing free-form external messages
|
||||||
|
or reading lane payload bodies.
|
||||||
|
|
||||||
|
Configuration fields and defaults are in [Configuration](../config.md).
|
||||||
|
Operator paths, rerun procedures, and bundle retention are in
|
||||||
|
[Operations](../operations.md). See [Troubleshooting](../troubleshooting.md)
|
||||||
|
for failure recovery.
|
||||||
@@ -1,64 +1,68 @@
|
|||||||
# Integration: scriptorium
|
# Integration: Scriptorium
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define Narratio's adapter contract for Scriptorium artifact generation and render-debug subprocess invocations.
|
Define the Scriptorium adapter contract used by `analyze` and trim-bounds generation in `trim`.
|
||||||
|
|
||||||
## Inputs and Outputs
|
## External Boundary
|
||||||
Inputs:
|
|
||||||
- `RunArtifactRequest`: binary, config path, prompt/profile IDs, input map, vars map, timeout, output path, logs/config paths, optional API env and working dir
|
|
||||||
- `RenderArtifactRequest`: same core fields for render mode
|
|
||||||
|
|
||||||
Outputs (`ArtifactResult`):
|
Narratio invokes Scriptorium as a subprocess in these modes:
|
||||||
- output path
|
|
||||||
- stdout/stderr log paths
|
|
||||||
- generated config path
|
|
||||||
- exit code and duration
|
|
||||||
- command mode (`run` or `render`)
|
|
||||||
- prompt/profile provenance
|
|
||||||
- validation failure signal
|
|
||||||
- adapter metadata
|
|
||||||
|
|
||||||
## Boundaries
|
- `scriptorium run`
|
||||||
Owns:
|
- `scriptorium render`
|
||||||
- Deterministic CLI arg construction for `scriptorium run` and `scriptorium render`
|
|
||||||
- Common request validation
|
|
||||||
- Invocation config emission
|
|
||||||
- Output existence/non-empty checks
|
|
||||||
- Validation-failure mapping for run exit code 2
|
|
||||||
|
|
||||||
Does not own:
|
The request timeout and parent cancellation bound each invocation. Internal
|
||||||
- Artifact selection policy (`analyze` stage)
|
runner composition is documented in
|
||||||
- Bounds semantic validation (`trim` stage)
|
[the adapter implementation guide](../internal/adapters.md).
|
||||||
|
|
||||||
## Config Fields Used
|
## Request Contract
|
||||||
Via `pipeline.scriptorium.*` and stage-level artifact config:
|
Both request types carry:
|
||||||
- `binary`, `config_path`, `timeout`, `render_debug`
|
- binary/config/prompt/profile IDs;
|
||||||
- artifact-level `prompt_id`, `profile_id`, `timeout`, `inputs`, `vars`, `output_path`
|
- input map and vars map;
|
||||||
|
- output path;
|
||||||
|
- timeout;
|
||||||
|
- generated config + stdout/stderr log paths;
|
||||||
|
- optional API-key env var name;
|
||||||
|
- optional working directory.
|
||||||
|
|
||||||
## External Adapters Used
|
## Result Contract
|
||||||
- Shared subprocess helper (`internal/adapters/subprocess`).
|
`ArtifactResult` returns:
|
||||||
|
- output/log/generated-config paths;
|
||||||
|
- exit code and duration;
|
||||||
|
- command mode (`run` or `render`);
|
||||||
|
- prompt/profile provenance;
|
||||||
|
- `ValidationFailed` marker;
|
||||||
|
- metadata map.
|
||||||
|
|
||||||
## State and Manifest Behavior
|
## Validation and Failure Semantics
|
||||||
- No direct manifest writes.
|
Request validation fails for:
|
||||||
- Stage metadata records adapter outputs and command mode.
|
- missing binary, prompt id, or output path;
|
||||||
- Generated invocation YAML is written when requested.
|
- non-positive timeout;
|
||||||
|
- empty input/var names;
|
||||||
|
- empty input path values;
|
||||||
|
- missing required credential env var when `APIKeyEnv` is set.
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
Run behavior:
|
||||||
- Adapter has no skip/resume logic. Stage/runner controls execution.
|
- subprocess errors propagate with context;
|
||||||
|
- `run` exit code `2` is mapped to `ValidationFailed=true`;
|
||||||
|
- successful subprocess still fails if output file is missing or empty.
|
||||||
|
|
||||||
## Failure Behavior
|
Render behavior:
|
||||||
- Request validation fails for missing binary/prompt/output, invalid timeout, invalid input/var names, or missing required API env var.
|
- subprocess errors propagate;
|
||||||
- Subprocess errors bubble with command context.
|
- output file must exist and be non-empty.
|
||||||
- `run` exit code 2 is treated as `ValidationFailed=true` and surfaced as error by calling stage.
|
|
||||||
- Successful subprocess still fails if output file is missing/empty.
|
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
## Deterministic Behavior
|
||||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
- input and var maps are sorted into deterministic `--input` and `--var` CLI args.
|
||||||
- `internal/adapters/scriptorium/fake_test.go`
|
- stage wiring adds `session_id=narratio-session-<session_id>` to every Scriptorium request for sticky upstream routing, overriding any configured `vars.session_id`.
|
||||||
- `internal/stage/analyze_test.go`
|
- generated invocation YAML (`scriptorium.generated.v1`) is emitted when requested.
|
||||||
- `internal/stage/trim_test.go`
|
- adapter is stateless and does not own artifact-selection policy.
|
||||||
|
|
||||||
## Architectural Invariants
|
## Configuration
|
||||||
- Both modes require explicit timeout > 0.
|
|
||||||
- Input/var maps are sorted into deterministic CLI argument order.
|
Operator-selected values are defined under `pipeline.scriptorium.*`, including
|
||||||
- Run-mode validation failures are represented explicitly, not silently skipped.
|
per-artifact settings under `pipeline.scriptorium.artifacts.*`, in the
|
||||||
|
[configuration reference](../config.md#pipeline).
|
||||||
|
|
||||||
|
Maintained examples with Scriptorium config:
|
||||||
|
|
||||||
|
- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml)
|
||||||
|
- [Production-shaped pipeline](../../examples/pipeline.production.yml)
|
||||||
|
|||||||
@@ -1,60 +1,60 @@
|
|||||||
# Integration: seriatim
|
# Integration: Seriatim
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define Narratio's adapter contract for merge, normalize, and trim subprocess invocations of Seriatim.
|
Define the Seriatim adapter contract used by `merge`, `normalize`, `trim`, and `render`.
|
||||||
|
|
||||||
## Inputs and Outputs
|
## External Boundary
|
||||||
Inputs:
|
|
||||||
- `MergeRequest`: raw/per-speaker normalized transcript inputs, base output path, optional report, speaker/autocorrect paths, logs/config
|
|
||||||
- `NormalizeRequest`: input transcript, output path, schema, optional report, timeout/log/config
|
|
||||||
- `TrimRequest`: input transcript, output path, keep selector, timeout/log/config
|
|
||||||
|
|
||||||
Outputs:
|
Narratio invokes Seriatim as a subprocess in these modes:
|
||||||
- `MergeResult`, `NormalizeResult`, `TrimResult` with output paths, logs/config paths, exit code, duration, binary provenance, and metadata.
|
|
||||||
|
|
||||||
## Boundaries
|
- `seriatim merge`
|
||||||
Owns:
|
- `seriatim normalize`
|
||||||
- Validated deterministic CLI invocation construction
|
- `seriatim trim`
|
||||||
- Optional env tuning propagation for merge
|
- `seriatim render`
|
||||||
- Invocation config file emission
|
|
||||||
- JSON output validation
|
|
||||||
|
|
||||||
Does not own:
|
The configured timeout and parent cancellation bound each invocation. Internal
|
||||||
- Transcript input selection/materialization logic (stage-owned)
|
runner composition is documented in
|
||||||
- Bounds computation (scriptorium/trim-stage-owned)
|
[the adapter implementation guide](../internal/adapters.md).
|
||||||
|
|
||||||
## Config Fields Used
|
## Request/Result Contracts
|
||||||
Via `pipeline.seriatim.*` mapped in app/stage wiring:
|
- `MergeRequest`/`MergeResult`: multi-input merge to base transcript, optional report.
|
||||||
- `binary`, `timeout`, `output_schema`, `coalesce_gap`, `report`
|
- `NormalizeRequest`/`NormalizeResult`: transcript normalization with explicit schema.
|
||||||
- `env.overlap_word_run_gap`
|
- `TrimRequest`/`TrimResult`: transcript trimming with required keep selector.
|
||||||
- `env.overlap_word_run_reorder_window`
|
- `RenderRequest`/`RenderResult`: transcript-to-markdown rendering with explicit format and render booleans.
|
||||||
- `env.backchannel_max_duration`
|
|
||||||
- `env.filler_max_duration`
|
|
||||||
|
|
||||||
## External Adapters Used
|
Results include output/log/config paths, timing, exit code, and metadata.
|
||||||
- Shared subprocess helper (`internal/adapters/subprocess`).
|
|
||||||
|
|
||||||
## State and Manifest Behavior
|
## Validation and Failure Semantics
|
||||||
- No direct manifest writes.
|
Runner construction validates:
|
||||||
- Stage metadata consumes adapter result fields and preserves generated config/log references.
|
- binary presence;
|
||||||
|
- timeout > 0;
|
||||||
|
- supported output schema (`seriatim-minimal|seriatim-intermediate|seriatim-full`);
|
||||||
|
- non-negative coalesce gap.
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
Invocation fails on:
|
||||||
- Adapter has no skip/resume logic. Runner controls stage execution.
|
- missing required request paths/inputs;
|
||||||
|
- invalid normalize schema override;
|
||||||
|
- unsupported render format;
|
||||||
|
- subprocess failure;
|
||||||
|
- invalid JSON outputs for merge/normalize/trim;
|
||||||
|
- missing `segments` array for normalize/trim transcript outputs;
|
||||||
|
- empty render output files.
|
||||||
|
|
||||||
## Failure Behavior
|
When report paths are provided/enabled, report files must parse as JSON.
|
||||||
- Constructor fails for invalid binary/timeout/output-schema/coalesce-gap.
|
|
||||||
- Merge fails on missing output path/inputs/report path (if enabled), subprocess errors, invalid merged output JSON, invalid report JSON.
|
|
||||||
- Normalize fails on missing input/output, invalid schema, subprocess errors, invalid final output JSON shape, invalid report JSON.
|
|
||||||
- Trim fails on missing input/output/keep selector, subprocess errors, invalid final-trimmed output JSON shape.
|
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
## Deterministic Behavior
|
||||||
- `internal/adapters/seriatim/subprocess_test.go`
|
- argument ordering is deterministic per command construction.
|
||||||
- `internal/adapters/seriatim/fake_test.go`
|
- merge env overrides are explicit (`SERIATIM_*`) and only emitted when configured.
|
||||||
- `internal/stage/merge_test.go`
|
- generated invocation YAML (`seriatim.generated.v1`) is emitted when requested.
|
||||||
- `internal/stage/normalize_test.go`
|
- adapter does not write manifests or choose stage inputs.
|
||||||
- `internal/stage/trim_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
## Configuration
|
||||||
- Supported output schemas are limited to `seriatim-minimal`, `seriatim-intermediate`, `seriatim-full`.
|
|
||||||
- Final and final-trimmed outputs must include `segments` arrays.
|
Operator-selected values are defined under `pipeline.seriatim.*` and
|
||||||
- Merge/normalize/trim all route through deterministic subprocess invocation.
|
`pipeline.render.*` in the
|
||||||
|
[configuration reference](../config.md#pipeline).
|
||||||
|
|
||||||
|
Maintained examples with Seriatim config:
|
||||||
|
|
||||||
|
- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml)
|
||||||
|
- [Production-shaped pipeline](../../examples/pipeline.production.yml)
|
||||||
|
|||||||
66
docs/integrations/whisperx.md
Normal file
66
docs/integrations/whisperx.md
Normal file
@@ -0,0 +1,66 @@
|
|||||||
|
# Integration: WhisperX
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
WhisperX transcribes each prepared speaker audio file for Narratio's
|
||||||
|
`transcribe` stage. Narratio uses an HTTP boundary and installs each successful
|
||||||
|
response as that speaker's raw transcript JSON.
|
||||||
|
|
||||||
|
## HTTP Boundary
|
||||||
|
|
||||||
|
Narratio sends an HTTP `POST` to the configured transcription URL using
|
||||||
|
`multipart/form-data` with:
|
||||||
|
|
||||||
|
- `file`: the audio file, retaining its base filename; and
|
||||||
|
- `language`: the configured language string.
|
||||||
|
|
||||||
|
The server must return a `2xx` response whose body is valid JSON. Narratio does
|
||||||
|
not currently require a more specific response schema at this boundary.
|
||||||
|
|
||||||
|
## Request And Result Contract
|
||||||
|
|
||||||
|
Each adapter request identifies a speaker, a readable audio file, and the
|
||||||
|
destination for the raw transcript. The HTTP request carries the audio and
|
||||||
|
language; the speaker identifier remains Narratio orchestration metadata.
|
||||||
|
|
||||||
|
On success, Narratio atomically writes the response body to the requested
|
||||||
|
destination. The adapter result reports that logical output together with the
|
||||||
|
attempt count, final HTTP status when available, elapsed duration, and adapter
|
||||||
|
identity metadata. A failed or invalid response is not installed as the
|
||||||
|
transcript output.
|
||||||
|
|
||||||
|
## Retry, Timeout, And Cancellation
|
||||||
|
|
||||||
|
- The configured timeout applies independently to each HTTP attempt.
|
||||||
|
- `retries` means additional attempts after the first.
|
||||||
|
- HTTP `429`, HTTP `5xx`, attempt timeouts, and network errors are retryable.
|
||||||
|
- Other HTTP `4xx` responses and explicit cancellation are not retryable.
|
||||||
|
- Narratio waits the configured retry delay between attempts and aborts that
|
||||||
|
wait when the parent context is canceled.
|
||||||
|
|
||||||
|
## Validation And Failure Semantics
|
||||||
|
|
||||||
|
Client construction rejects a missing or invalid absolute transcription URL,
|
||||||
|
a missing language, a non-positive timeout, negative retries, or a negative
|
||||||
|
retry delay. A request fails before transmission when its audio or output path
|
||||||
|
is missing.
|
||||||
|
|
||||||
|
Non-`2xx` status, transport failure, response-size overflow, invalid JSON, or
|
||||||
|
failure to install the output causes the transcription to fail. Errors include
|
||||||
|
attempt context, and the result retains attempts, final status when available,
|
||||||
|
and elapsed duration for diagnostics.
|
||||||
|
|
||||||
|
## Determinism And Concurrency
|
||||||
|
|
||||||
|
Each audio request has stable multipart field names, and successful bytes are
|
||||||
|
installed atomically. The transcribe stage may process speaker files in
|
||||||
|
parallel, bounded by the configured concurrency. It records results in stable
|
||||||
|
speaker order after all work completes; any speaker failure fails the stage.
|
||||||
|
|
||||||
|
## Related Canonical Docs
|
||||||
|
|
||||||
|
- [Configuration](../config.md#pipeline) defines the operator-selected
|
||||||
|
WhisperX URL, language, timeouts, retry policy, and concurrency.
|
||||||
|
- [Adapter implementation](../internal/adapters.md) describes internal wiring.
|
||||||
|
- [Transcribe stage](../internal/stage-transcribe.md) describes stage mechanics,
|
||||||
|
durable artifacts, and manifests.
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
# Internal Documentation Index
|
|
||||||
|
|
||||||
## Audience
|
|
||||||
Developers and LLM coding agents changing Narratio internals.
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
Implementation-accurate contracts for workspace/state, manifests, stages, artifact resolution, adapter boundaries, and restore command behavior.
|
|
||||||
|
|
||||||
## Component Docs
|
|
||||||
- `adapters.md`: external adapter map, runtime wiring, and boundary ownership.
|
|
||||||
- `storage.md`: remote storage backend contracts and object-store invariants.
|
|
||||||
- `manifest.md`: session/run manifest schemas, lifecycle transitions, and persistence semantics.
|
|
||||||
- `artifacts.md`: built-in artifact registry, runtime artifact catalog, and source-resolution behavior.
|
|
||||||
- `workspace.md`: local state model, manifests, run-local layout, materialization, and cleanup invariants.
|
|
||||||
- `command-restore.md`: restore command discovery/planning/execution/reporting contract.
|
|
||||||
- `stage-prepare.md`: input materialization and provenance capture.
|
|
||||||
- `stage-transcribe.md`: WhisperX transcript generation.
|
|
||||||
- `stage-merge.md`: Seriatim normalization + merge.
|
|
||||||
- `stage-polish.md`: Audita transcript polishing.
|
|
||||||
- `stage-normalize.md`: post-polish normalization.
|
|
||||||
- `stage-trim.md`: bounds-driven transcript trimming.
|
|
||||||
- `stage-analyze.md`: dependency-ordered Scriptorium artifact generation for selected configured artifacts.
|
|
||||||
- `stage-publish.md`: publish upload and current-pointer commit contract.
|
|
||||||
|
|
||||||
## External Integration Notes
|
|
||||||
- `../integrations/README.md`: canonical location for external integration contracts (`audita.md`, `seriatim.md`, `scriptorium.md`).
|
|
||||||
|
|
||||||
## Canonical Owner
|
|
||||||
`docs/internal/` is the canonical home for implemented internals per `docs/documentation/policy.md`.
|
|
||||||
@@ -1,78 +1,78 @@
|
|||||||
# Internal: Adapters
|
# Internal: Adapters
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Describe the external adapter boundaries used by Narratio stages and app orchestration, including default runtime wiring.
|
|
||||||
|
|
||||||
## Inputs and outputs
|
Explain the adapter interfaces and production composition used by application
|
||||||
Inputs:
|
and stage orchestration. Externally observable protocols and formats belong in
|
||||||
- Stage requests passed through adapter interfaces (for example transcription, merge/normalize/trim, polish, artifact generation, object-store operations, notifications).
|
the [integration contracts](../integrations/).
|
||||||
- Resolved config values used to construct default adapters.
|
|
||||||
|
|
||||||
Outputs:
|
## Adapter Boundaries
|
||||||
- Adapter-specific result structs (paths, metadata, status/attempt info, duration/exit details).
|
|
||||||
- Adapter errors returned to stage/app orchestration.
|
|
||||||
|
|
||||||
## Boundaries
|
Narratio stage logic depends on adapter interfaces, not transport-specific details.
|
||||||
Owns:
|
|
||||||
- Transport/process/SDK details at system boundaries (`HTTP`, subprocess CLI invocation, AWS SDK calls).
|
|
||||||
- Request/response contracts in `internal/adapters/*` packages.
|
|
||||||
|
|
||||||
Does not own:
|
Primary adapters:
|
||||||
- Stage sequencing, skip/force/resume decisions.
|
|
||||||
- Manifest transition logic.
|
|
||||||
- Canonical workspace path policy.
|
|
||||||
|
|
||||||
## Config fields used
|
|
||||||
Default wiring and adapter calls consume:
|
|
||||||
- `pipeline.whisperx.*`
|
|
||||||
- `pipeline.seriatim.*`
|
|
||||||
- `pipeline.audita.*`
|
|
||||||
- `pipeline.scriptorium.*`
|
|
||||||
- `pipeline.storage.*` and `pipeline.publish.*` (object-store construction/gating)
|
|
||||||
- `pipeline.notification.*` (sender boundary exists; placeholder behavior today)
|
|
||||||
|
|
||||||
## External adapters used
|
|
||||||
Runtime env boundary fields (`internal/stage.Env`):
|
|
||||||
- `whisperx.Client`
|
- `whisperx.Client`
|
||||||
- `seriatim.Runner`
|
- `seriatim.Runner`
|
||||||
- `audita.Runner`
|
- `audita.Runner`
|
||||||
- `scriptorium.Runner`
|
- `scriptorium.Runner`
|
||||||
|
- `notarius.Runner`
|
||||||
- `storage.ObjectStore`
|
- `storage.ObjectStore`
|
||||||
- `notify.Sender`
|
- `notify.Sender`
|
||||||
|
|
||||||
Current execution usage:
|
## Ownership
|
||||||
- Actively used by implemented stages: `WhisperX`, `Seriatim`, `Audita`, `Scriptorium`, `ObjectStore`, `Notifier`.
|
|
||||||
- Present but not used by implemented stage set: legacy `storage.Backend`.
|
|
||||||
|
|
||||||
Default construction in app runner:
|
Adapters own:
|
||||||
- Auto-constructed when not injected: WhisperX HTTP client, Seriatim subprocess runner, Audita subprocess runner, Scriptorium subprocess runner, object store (only when needed), and `notify.NoopSender`.
|
|
||||||
- Object-store construction goes through app command orchestration so configured filesystem secrets are loaded before the storage adapter is initialized.
|
|
||||||
- Callers can inject test/fake implementations through `app.RunOptions.Env`.
|
|
||||||
|
|
||||||
## State and manifest behavior
|
- HTTP/subprocess/SDK argument and transport details.
|
||||||
- Adapters do not directly mutate session/run manifests.
|
- Backend-specific request/response mapping.
|
||||||
- Stages and runner own manifest writes and stage status transitions.
|
|
||||||
- Adapter outputs are persisted indirectly through stage result mapping (outputs/logs/generated configs/metadata).
|
|
||||||
|
|
||||||
## Skip and resume behavior
|
Adapters do not own:
|
||||||
- No adapter-level skip/resume semantics.
|
|
||||||
- Skip/resume/force behavior is decided by app runner using manifest stage state.
|
|
||||||
|
|
||||||
## Failure behavior
|
- stage ordering/skip/force logic;
|
||||||
- Adapter constructors validate config-derived values and fail early on invalid required inputs.
|
- manifest transitions;
|
||||||
- Adapter run-time failures are returned to stage code with boundary context and are recorded as stage failures by runner logic.
|
- canonical path policy.
|
||||||
- Subprocess adapters preserve stdout/stderr and generated-config paths to aid diagnosis.
|
|
||||||
|
|
||||||
## Tests to inspect before changing
|
## Default Wiring
|
||||||
|
|
||||||
|
`internal/app/runner.go` initializes default adapters when not injected:
|
||||||
|
|
||||||
|
- WhisperX HTTP client from pipeline config.
|
||||||
|
- Seriatim subprocess runner.
|
||||||
|
- Audita subprocess runner.
|
||||||
|
- Scriptorium subprocess runner.
|
||||||
|
- Notarius subprocess runner when extraction is enabled.
|
||||||
|
- Noop notifier (`notify.NoopSender`).
|
||||||
|
- Object store only when required by selected stages/config.
|
||||||
|
|
||||||
|
Notarius is composed only when extraction is enabled; the extract stage owns
|
||||||
|
receipt, bundle, and configured-lane policy rather than the adapter.
|
||||||
|
|
||||||
|
Object-store construction goes through `newCommandObjectStore`, which loads
|
||||||
|
configured filesystem secrets before adapter initialization.
|
||||||
|
|
||||||
|
## Failure Semantics
|
||||||
|
|
||||||
|
- Constructor errors fail stage execution setup early.
|
||||||
|
- Runtime adapter errors propagate to stage code and then manifest failure handling.
|
||||||
|
- Subprocess adapters persist stage logs/generated configs through stage-managed paths.
|
||||||
|
|
||||||
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Composition: `internal/app/runner.go`, `internal/app/object_store.go`
|
||||||
|
- Shared subprocess mechanics: `internal/adapters/subprocess`
|
||||||
|
- Focused adapters: `internal/adapters/{whisperx,seriatim,audita,scriptorium,notarius,storage,notify}`
|
||||||
- `internal/adapters/whisperx/http_test.go`
|
- `internal/adapters/whisperx/http_test.go`
|
||||||
- `internal/adapters/seriatim/subprocess_test.go`
|
- `internal/adapters/seriatim/subprocess_test.go`
|
||||||
- `internal/adapters/audita/subprocess_test.go`
|
- `internal/adapters/audita/subprocess_test.go`
|
||||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||||
|
- `internal/adapters/notarius/subprocess_test.go`
|
||||||
- `internal/adapters/storage/*_test.go`
|
- `internal/adapters/storage/*_test.go`
|
||||||
- `internal/adapters/notify/fake_test.go`
|
|
||||||
- `internal/app/runner_test.go`
|
- `internal/app/runner_test.go`
|
||||||
|
|
||||||
## Architectural invariants
|
See the [WhisperX](../integrations/whisperx.md),
|
||||||
- Stage code depends on adapter interfaces, not transport-specific implementation types.
|
[Seriatim](../integrations/seriatim.md), [Audita](../integrations/audita.md),
|
||||||
- External SDK-specific types remain inside adapter implementations.
|
[Scriptorium](../integrations/scriptorium.md), and
|
||||||
- Default app wiring must remain deterministic and overrideable via injected env dependencies.
|
[Notarius](../integrations/notarius.md) contracts before changing an
|
||||||
|
externally visible boundary. Operator-selected values belong in
|
||||||
|
[Configuration](../config.md).
|
||||||
|
|||||||
@@ -1,106 +1,161 @@
|
|||||||
# Internal: Artifacts
|
# Internal: Artifacts
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define Narratio artifact identity, catalog, and source-resolution behavior for:
|
|
||||||
- built-in session artifacts;
|
|
||||||
- configured analyze artifacts;
|
|
||||||
- canonical previous-session artifact sources.
|
|
||||||
|
|
||||||
## Inputs and outputs
|
Explain the artifact registry, runtime catalog, resolver, previous-input
|
||||||
Inputs:
|
requirements, and shared remote current-state mechanics implemented by
|
||||||
- configured input sources (`pipeline.scriptorium.artifacts.*.inputs.*.source`);
|
`internal/artifacts`. Configuration fields that accept source IDs belong in
|
||||||
- session paths and manifest inputs/outputs;
|
[Configuration](../config.md); physical placement belongs in
|
||||||
- runtime catalog state.
|
[Operations](../operations.md).
|
||||||
|
|
||||||
Outputs:
|
## Built-in Source IDs
|
||||||
- resolved artifact path + provenance (`ResolvedSessionArtifact`);
|
|
||||||
- runtime catalog entries for built-ins and configured artifacts;
|
|
||||||
- requirement sets for canonical previous-session inputs;
|
|
||||||
- canonical S3 session, run, current, session config, session locks, audio, and published output keys.
|
|
||||||
|
|
||||||
## Boundaries
|
The internal registry recognizes these stable built-in source IDs:
|
||||||
Owns:
|
|
||||||
- built-in source registry and validation;
|
|
||||||
- configured artifact catalog identity (`narratio.artifact.<name>`);
|
|
||||||
- canonical previous-session source parsing and resolution;
|
|
||||||
- previous-session requirement collection (`CollectPreviousArtifactRequirements`).
|
|
||||||
|
|
||||||
Does not own:
|
- `narratio.transcript.base`
|
||||||
- prepare-stage remote hydration;
|
- `narratio.transcript.polished`
|
||||||
- stage success/skip transitions;
|
- `narratio.transcript.final`
|
||||||
- publish upload orchestration.
|
- `narratio.transcript.final_trimmed`
|
||||||
|
- `narratio.transcript.final_markdown`
|
||||||
|
- `narratio.transcript.final_trimmed_markdown`
|
||||||
|
- `narratio.bounds.session`
|
||||||
|
|
||||||
## Built-in IDs
|
Registry entries bind each ID to its producer, output kind, canonical fallback,
|
||||||
| Artifact ID | Canonical file | Producer stage | Output kind |
|
and content validator. The focused stage documents own their input/output flow;
|
||||||
| --- | --- | --- | --- |
|
[Configuration](../config.md) owns where operators may select these IDs.
|
||||||
| `narratio.transcript.base` | `transcripts/base.json` | `merge` | `transcript_base` |
|
|
||||||
| `narratio.transcript.polished` | `transcripts/polished.json` | `polish` | `transcript_polished` |
|
|
||||||
| `narratio.transcript.final` | `transcripts/final.json` | `normalize` | `transcript_final` |
|
|
||||||
| `narratio.transcript.final_trimmed` | `transcripts/final.trimmed.json` | `trim` | `transcript_final_trimmed` |
|
|
||||||
| `narratio.bounds.session` | `artifacts/session_bounds.json` | `trim` | `session_bounds` |
|
|
||||||
|
|
||||||
## Source families
|
## Configured, Extraction, And Previous-Session Sources
|
||||||
- built-in: `narratio.transcript.*`, `narratio.bounds.session`
|
|
||||||
- configured artifact: `narratio.artifact.<artifact_key>`
|
|
||||||
- canonical previous-session artifact: `narratio.previous_session.artifact.<artifact_key>`
|
|
||||||
|
|
||||||
## S3 key helpers
|
- configured source ID format: `narratio.artifact.<artifact_key>`
|
||||||
- session prefix: `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
- extraction source ID format: `narratio.extraction.<output_key>`
|
||||||
- session config: `{session_prefix}/session.yml`
|
- previous-session source ID format: `narratio.previous_session.artifact.<artifact_key>`
|
||||||
- session lock store: `{session_prefix}/locks.yml`
|
|
||||||
- run prefix: `{session_prefix}/runs/{run_id}/`
|
|
||||||
- audio prefix: `{session_prefix}/{session.inputs.audio_s3.prefix}`
|
|
||||||
- current manifest: `{session_prefix}/current/manifest.json`
|
|
||||||
- current run pointer: `{session_prefix}/current/run_id.txt`
|
|
||||||
|
|
||||||
## Runtime catalog model
|
All formats are validated by strict source-policy rules. Extraction sources are
|
||||||
Catalog entries track:
|
registered only from `pipeline.notarius.outputs`; the Notarius index has no
|
||||||
- `planned`: source is registered for this run;
|
selectable source ID.
|
||||||
- `executable`: configured artifact is selected for analyze execution;
|
|
||||||
- `available`: usable local file exists (generated this run or reused from disk).
|
## Runtime Catalog
|
||||||
|
|
||||||
|
`ArtifactCatalog` tracks:
|
||||||
|
|
||||||
|
- `planned`: source registered for run context;
|
||||||
|
- `executable`: selected and enabled for analyze execution;
|
||||||
|
- `available`: local file exists and validates;
|
||||||
|
- `provenance`: availability source.
|
||||||
|
|
||||||
|
Current provenance values:
|
||||||
|
|
||||||
Configured artifact provenance values include:
|
|
||||||
- `generated.current_analyze_run`
|
- `generated.current_analyze_run`
|
||||||
- `filesystem.disabled_artifact_output`
|
- `filesystem.disabled_artifact_output`
|
||||||
|
|
||||||
Previous-session canonical provenance values include:
|
|
||||||
- `manifest.inputs.previous_cache`
|
- `manifest.inputs.previous_cache`
|
||||||
- `current_session.previous_cache`
|
- `current_session.previous_cache`
|
||||||
|
|
||||||
## Resolution behavior
|
## Resolution Rules
|
||||||
- Built-ins resolve via manifest producer outputs first, then canonical fallback paths.
|
|
||||||
- Configured `narratio.artifact.<name>` sources resolve through catalog availability.
|
|
||||||
- Canonical previous-session sources resolve to current-session `previous/` cache candidates derived from configured artifact canonical output paths.
|
|
||||||
- Publish-relative configured artifact paths under `artifacts/` are cached without a redundant nested `artifacts/` segment.
|
|
||||||
- Previous-session canonical resolution prefers manifest-recorded input paths when present, then filesystem fallback under `previous/artifacts/**`.
|
|
||||||
|
|
||||||
## Previous-session requirement scanning
|
Built-ins:
|
||||||
`CollectPreviousArtifactRequirements`:
|
|
||||||
- scans enabled configured artifacts only;
|
|
||||||
- includes canonical previous-session sources only;
|
|
||||||
- deduplicates by artifact key;
|
|
||||||
- merges required/optional references (`required` wins);
|
|
||||||
- records deterministic sorted source locations for diagnostics.
|
|
||||||
|
|
||||||
## Validation behavior
|
1. manifest producer outputs (when present)
|
||||||
- transcript built-ins: JSON with top-level `segments` array;
|
2. canonical session-path fallback
|
||||||
|
|
||||||
|
Configured sources (`narratio.artifact.*`):
|
||||||
|
|
||||||
|
- resolve only through runtime catalog availability.
|
||||||
|
|
||||||
|
Extraction sources (`narratio.extraction.*`):
|
||||||
|
|
||||||
|
- use the shared registration and manifest hydration path in
|
||||||
|
`extraction_catalog.go`;
|
||||||
|
- require a current successful extract record with the exact configured source,
|
||||||
|
compatible contract and Notarius provenance, a confined regular durable
|
||||||
|
payload, and matching checksum; and
|
||||||
|
- are never inferred by scanning the Notarius bundle directory.
|
||||||
|
|
||||||
|
Previous-session sources (`narratio.previous_session.artifact.*`):
|
||||||
|
|
||||||
|
- resolve only from local `previous/` cache state;
|
||||||
|
- prefer manifest-backed previous-input paths;
|
||||||
|
- fallback to existing previous-cache filesystem paths.
|
||||||
|
|
||||||
|
Validation by content type:
|
||||||
|
|
||||||
|
- transcript JSON built-ins: JSON with top-level `segments` array;
|
||||||
|
- transcript Markdown built-ins: non-empty text file;
|
||||||
- bounds built-in: valid JSON;
|
- bounds built-in: valid JSON;
|
||||||
- configured and previous-session artifact files: non-empty text content.
|
- configured/previous-session artifact files: non-empty text file.
|
||||||
|
|
||||||
## Failure behavior
|
## Previous Requirement Collection
|
||||||
- unsupported source or malformed canonical previous source: validation/resolution error;
|
|
||||||
- known source unavailable: `ErrSessionArtifactNotFound`;
|
|
||||||
- configured/previous canonical source without catalog: error;
|
|
||||||
- resolved invalid file content: validation error.
|
|
||||||
|
|
||||||
## Tests to inspect before changing
|
`CollectPreviousArtifactRequirements`:
|
||||||
- `internal/artifacts/artifact_resolver_test.go`
|
|
||||||
- `internal/artifacts/catalog_test.go`
|
|
||||||
- `internal/artifacts/previous_requirements_test.go`
|
|
||||||
- `internal/stage/prepare_previous_test.go`
|
|
||||||
- `internal/stage/analyze_test.go`
|
|
||||||
|
|
||||||
## Architectural invariants
|
- scans enabled configured artifacts only;
|
||||||
- Built-in source IDs are static.
|
- extracts only canonical previous-session sources;
|
||||||
- Configured and previous-session source IDs are artifact-key based and validation-gated.
|
- deduplicates by artifact key;
|
||||||
- Resolution behavior remains deterministic and manifest-aware.
|
- merges required and optional references (required wins);
|
||||||
|
- returns deterministic ordering and source locations.
|
||||||
|
|
||||||
|
## Current-State Helpers
|
||||||
|
|
||||||
|
Artifacts package owns shared remote current-state loading mechanics used by
|
||||||
|
restore, status and validation checks, and previous-cache planning.
|
||||||
|
|
||||||
|
Core helpers:
|
||||||
|
|
||||||
|
- `LoadCurrentRunPointer`
|
||||||
|
- `LoadCurrentManifest`
|
||||||
|
- `LoadCurrentState`
|
||||||
|
- `ValidateCurrentStateIdentity`
|
||||||
|
|
||||||
|
Typed missing-state errors:
|
||||||
|
|
||||||
|
- `CurrentRunPointerMissingError` (`ErrCurrentRunPointerMissing`)
|
||||||
|
- `CurrentManifestMissingError` (`ErrCurrentManifestMissing`)
|
||||||
|
|
||||||
|
Identity validation supports caller-provided expectations:
|
||||||
|
|
||||||
|
- expected campaign;
|
||||||
|
- expected session ID;
|
||||||
|
- expected run ID, or pointer/manifest run-ID consistency check.
|
||||||
|
|
||||||
|
Caller policy is intentionally outside artifacts helpers:
|
||||||
|
|
||||||
|
- some callers fail on missing current state;
|
||||||
|
- some callers downgrade missing state to status/findings;
|
||||||
|
- some callers skip optional behavior when state is missing.
|
||||||
|
|
||||||
|
## Key Path Helpers
|
||||||
|
|
||||||
|
`internal/artifacts/paths.go` and S3-key helpers define canonical helpers for:
|
||||||
|
|
||||||
|
- session/work/run paths;
|
||||||
|
- previous-cache paths;
|
||||||
|
- spool/cache paths;
|
||||||
|
- S3 session/run/current-state key layout.
|
||||||
|
|
||||||
|
See [Workspace Internals](workspace.md) for how callers consume local helpers
|
||||||
|
and [Operations](../operations.md#local-state-layout) for the authoritative
|
||||||
|
physical layout.
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- source ID formats are stable contracts;
|
||||||
|
- artifact resolution is deterministic and manifest-aware;
|
||||||
|
- extraction sources are available only from a compatible successful manifest
|
||||||
|
record;
|
||||||
|
- previous-session source resolution in `analyze` is local-only;
|
||||||
|
- remote current-state key construction remains centralized in artifacts helpers.
|
||||||
|
|
||||||
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Registry and resolution: `internal/artifacts/artifact_resolver.go`,
|
||||||
|
`internal/artifacts/catalog.go`, `internal/artifacts/transcripts.go`,
|
||||||
|
`internal/artifacts/extraction_catalog.go`
|
||||||
|
- Current state: `internal/artifacts/current_state.go`
|
||||||
|
- Paths and keys: `internal/artifacts/paths.go`,
|
||||||
|
`internal/artifacts/s3_keys.go`
|
||||||
|
- Previous requirements: `internal/artifacts/previous_requirements.go`
|
||||||
|
- Tests: `internal/artifacts/artifact_resolver_test.go`,
|
||||||
|
`internal/artifacts/catalog_test.go`,
|
||||||
|
`internal/artifacts/extraction_catalog_test.go`,
|
||||||
|
`internal/artifacts/current_state_test.go`,
|
||||||
|
`internal/artifacts/paths_model_test.go`,
|
||||||
|
`internal/artifacts/previous_requirements_test.go`
|
||||||
|
|||||||
@@ -1,106 +1,81 @@
|
|||||||
# Internal: Command Restore
|
# Internal: Command Restore
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define the implemented `narratio session restore` contract: committed remote-state discovery, deterministic plan classification, safe file install semantics, and restore reporting.
|
|
||||||
|
|
||||||
## Inputs and outputs
|
Explain the implemented restore discovery, planning, installation, and
|
||||||
Inputs:
|
reporting flow in `internal/app`. User invocation belongs in
|
||||||
- CLI syntax: `narratio session restore <session_id>`.
|
[CLI](../cli.md#session-restore), and the operator recovery procedure and
|
||||||
- CLI flags: `--config`, `--campaign`, `--campaign-file`, `--session`, `--previous-session-id`, `--dry-run`, `--force`, `--include-audio`.
|
physical restore scope belong in
|
||||||
- Resolved/validated `pipeline.yml` and `session.yml`.
|
[Operations](../operations.md#restore-workflow).
|
||||||
- Configured remote object store.
|
|
||||||
- Remote committed current-state markers (`current/run_id.txt`, `current/manifest.json`).
|
|
||||||
|
|
||||||
Outputs:
|
Restore is split into explicit phases so remote authority, local conflict
|
||||||
- Dry-run summary to stdout (plan + counts).
|
policy, and filesystem mutation can be tested independently.
|
||||||
- Non-dry-run completion summary to stdout.
|
|
||||||
- Local durable session files restored under canonical session root.
|
|
||||||
- Non-dry-run restore report at `reports/restore-latest.json`.
|
|
||||||
|
|
||||||
## Boundaries
|
## Discovery Contract
|
||||||
Owns:
|
|
||||||
- Restore command flag parsing and command wiring.
|
|
||||||
- Remote current-state discovery and identity validation.
|
|
||||||
- Restore plan construction and conflict classification.
|
|
||||||
- Restore execution for planned downloads.
|
|
||||||
- Restore report model and persistence.
|
|
||||||
|
|
||||||
Does not own:
|
Discovery delegates current-state pointer and manifest loading to
|
||||||
- Stage execution orchestration (`run`, `resume`, `run-stage`).
|
`internal/artifacts`, then validates the result against the resolved request:
|
||||||
- Publish-stage behavior.
|
|
||||||
- Storage transport implementation details (owned by storage adapters).
|
|
||||||
|
|
||||||
## Config fields used
|
- campaign must match;
|
||||||
- Config/session discovery and templating fields consumed by all commands.
|
- session ID must match.
|
||||||
- `pipeline.workspace.root` (local restore target root).
|
|
||||||
- `pipeline.storage.*` (remote backend + publish identity derivation).
|
|
||||||
- `pipeline.storage.s3.*` identity components used by session-prefix helpers.
|
|
||||||
- `pipeline.spool.root` for active audio downloads.
|
|
||||||
- `pipeline.cache.root` and `pipeline.cache.s3_audio` for reusable S3 audio cache.
|
|
||||||
- `session.session_id`
|
|
||||||
- `session.campaign`
|
|
||||||
|
|
||||||
## External adapters used
|
Restore treats any missing or invalid remote current state as a command error.
|
||||||
- `storage.ObjectStore` for `Exists`, `List`, `Download`.
|
|
||||||
- `artifacts.Store` (`LocalStore`) for layout and session lock management.
|
|
||||||
- `manifest.LocalStore` for manifest decode/validation and identity checks.
|
|
||||||
|
|
||||||
## State and manifest behavior
|
## Planning Contract
|
||||||
- Restore is not a pipeline run and does not create a run manifest.
|
|
||||||
- Restore uses committed remote current state only:
|
|
||||||
- `current/run_id.txt` must exist and be non-empty.
|
|
||||||
- `current/manifest.json` must decode and match requested session/campaign.
|
|
||||||
- Non-dry-run writes restore files to canonical session paths.
|
|
||||||
- With `--include-audio`, restore uses the shared S3 audio cache for `audio/**` objects. Cache hits avoid object downloads; cache misses download through spool, install the work file, and populate cache.
|
|
||||||
- Manifest install behavior:
|
|
||||||
- validated before replacement.
|
|
||||||
- installed last among download actions.
|
|
||||||
- existing local manifest is preserved if restored manifest validation/install fails.
|
|
||||||
- Non-dry-run report persists summary/action status metadata in `reports/restore-latest.json`.
|
|
||||||
|
|
||||||
Restore path scope:
|
Restore planner action kinds:
|
||||||
- includes:
|
|
||||||
- `manifest.json`
|
|
||||||
- `transcripts/**`
|
|
||||||
- `artifacts/**`
|
|
||||||
- `previous/**`
|
|
||||||
- `audio/**` only when `--include-audio` is set
|
|
||||||
- excludes:
|
|
||||||
- `runs/**`
|
|
||||||
- `logs/**`
|
|
||||||
- `reports/**`
|
|
||||||
- `config/**`
|
|
||||||
- `inputs/**`
|
|
||||||
- remote `current/**` pointer files as local restore targets
|
|
||||||
|
|
||||||
## Skip and resume behavior
|
- `download`;
|
||||||
- Restore does not participate in stage skip/resume decisions.
|
- `skip_same`;
|
||||||
- Restore provides durable local state so subsequent stage commands can resume or rerun based on restored manifest state.
|
- `conflict`.
|
||||||
- Audio cache is outside the workspace and is reused across restore and prepare invocations.
|
|
||||||
- Dry-run is read-only and returns plan output only.
|
|
||||||
|
|
||||||
## Failure behavior
|
Planner behavior:
|
||||||
- Fails when storage backend is unavailable or publish identity cannot be resolved.
|
|
||||||
- Fails when remote current pointer/manifest is missing or invalid.
|
|
||||||
- Fails when remote manifest identity mismatches requested campaign/session.
|
|
||||||
- Fails on local conflicts unless `--force` is set.
|
|
||||||
- Fails fast on session lock acquisition conflict for non-dry-run execution.
|
|
||||||
- On execution failure, previously installed files remain; no rollback is performed.
|
|
||||||
|
|
||||||
## Tests to inspect before changing
|
- remote list scope is the resolved session prefix;
|
||||||
- `internal/app/restore_test.go`
|
- remote-to-local mapping is traversal-safe;
|
||||||
- `internal/app/restore_discovery_test.go`
|
- actions are sorted by local relative path and then remote key;
|
||||||
- `internal/app/restore_plan_test.go`
|
- force converts differing local targets from conflicts to downloads.
|
||||||
- `internal/app/restore_execution_test.go`
|
|
||||||
- `internal/app/restore_workflow_test.go`
|
|
||||||
- `internal/artifacts/archive_identity_test.go`
|
|
||||||
|
|
||||||
## Architectural invariants
|
Previous-cache files are planned separately through `previouscache.BuildPlan`
|
||||||
- Restore relies on centralized path/key helpers (`internal/artifacts`) rather than ad hoc key building.
|
when configured previous-session requirements exist.
|
||||||
- `current/run_id.txt` is the remote commit marker; restore must not infer committed state from incidental files.
|
|
||||||
- Local path mapping is traversal-safe and constrained to session root.
|
## Execution Contract
|
||||||
- Restore scope is deterministic and path-classified:
|
|
||||||
- include `manifest.json`, `transcripts/**`, `artifacts/**`, `previous/**`
|
Execution order and safety:
|
||||||
- include `audio/**` only with `--include-audio`
|
|
||||||
- exclude `runs/**`, `logs/**`, `reports/**`, `config/**`, `inputs/**`
|
- non-manifest downloads happen before manifest install;
|
||||||
- Command remains standalone; no implicit `run --restore` behavior.
|
- `manifest.json` installs last;
|
||||||
|
- downloads use sibling temp files plus atomic rename;
|
||||||
|
- manifest replacement is validated before rename;
|
||||||
|
- failed installs do not roll back files already written in the same execution.
|
||||||
|
|
||||||
|
Audio restore path:
|
||||||
|
|
||||||
|
- uses `audio.MaterializeS3Audio`;
|
||||||
|
- integrates spool and S3 audio cache paths;
|
||||||
|
- supports cache-hit reuse without object redownload.
|
||||||
|
|
||||||
|
## Reporting Contract
|
||||||
|
|
||||||
|
- dry-run mode prints a summary and performs no local writes;
|
||||||
|
- execution mode persists the canonical restore report described in
|
||||||
|
[Operations](../operations.md#restore-workflow);
|
||||||
|
- report includes plan counts, per-action status, and execution failures.
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- restore uses committed remote current state as authority;
|
||||||
|
- `current/run_id.txt` is the remote publish commit marker;
|
||||||
|
- restore does not execute pipeline stages.
|
||||||
|
|
||||||
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Discovery: `internal/app/restore_discovery.go`
|
||||||
|
- Planning: `internal/app/restore_plan.go`, `internal/previouscache`
|
||||||
|
- Execution: `internal/app/restore_execute.go`
|
||||||
|
- Reporting and command coordination: `internal/app/restore_report.go`,
|
||||||
|
`internal/app/restore.go`
|
||||||
|
- Tests: `internal/app/restore_discovery_test.go`,
|
||||||
|
`internal/app/restore_plan_test.go`,
|
||||||
|
`internal/app/restore_execution_test.go`,
|
||||||
|
`internal/app/restore_workflow_test.go`
|
||||||
|
|||||||
@@ -1,81 +1,100 @@
|
|||||||
# Internal: Manifest
|
# Internal: Manifest
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Describe Narratio's durable execution state model for session-level and run-level manifests, including lifecycle transitions and persistence behavior.
|
|
||||||
|
|
||||||
## Inputs and outputs
|
Explain the session-progress and invocation-audit models implemented by
|
||||||
Inputs:
|
`internal/manifest`. Physical manifest placement belongs in
|
||||||
- Session identity and run identity from app orchestration.
|
[Operations](../operations.md#local-state-layout).
|
||||||
- Stage transition events and stage result payloads.
|
|
||||||
|
|
||||||
Outputs:
|
## Session Manifest
|
||||||
- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`.
|
|
||||||
- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`.
|
|
||||||
|
|
||||||
## Boundaries
|
`manifest.Manifest` records:
|
||||||
Owns:
|
|
||||||
- Manifest schemas (`Manifest`, `RunManifest`, stage records, error records, input/artifact records).
|
|
||||||
- Stage status/action transition methods.
|
|
||||||
- Persistent store contract (`manifest.Store`) and local JSON store implementation.
|
|
||||||
|
|
||||||
Does not own:
|
- identity (`session_id`, `campaign`, `run_id`)
|
||||||
- Stage implementation details.
|
- local path metadata (`local_workdir`, `local_spool_dir`)
|
||||||
- Path construction policy outside manifest file persistence calls.
|
- remote identity metadata (`s3_bucket`, `s3_session_prefix`, `s3_run_prefix`)
|
||||||
- CLI command behavior.
|
- `inputs` records
|
||||||
|
- durable `artifacts` records
|
||||||
|
- per-stage `stages` map
|
||||||
|
|
||||||
## Config fields used
|
The model admits these stage states:
|
||||||
Manifest package itself does not read config directly.
|
|
||||||
|
|
||||||
Manifest identity fields are populated by app/stage orchestration from:
|
- `pending`
|
||||||
- `session.session_id`
|
- `running`
|
||||||
- `session.campaign`
|
- `succeeded`
|
||||||
- `pipeline.workspace.root`
|
- `failed`
|
||||||
- `pipeline.storage.s3.*` (when publish/S3 identity is set)
|
- `skipped`
|
||||||
|
- `stale`
|
||||||
|
- `interrupted`
|
||||||
|
|
||||||
## External adapters used
|
## Run Manifest
|
||||||
- No external service adapters.
|
|
||||||
- Uses local filesystem for persistence via `manifest.LocalStore`.
|
|
||||||
|
|
||||||
## State and manifest behavior
|
`manifest.RunManifest` is created for each invocation and records:
|
||||||
Session manifest model:
|
|
||||||
- Tracks durable per-session stage state and provenance (`pending`, `running`, `succeeded`, `failed`, `skipped`, `stale`, `interrupted`).
|
|
||||||
- Stores resolved inputs, durable artifacts, stage logs/config refs, and stage metadata.
|
|
||||||
|
|
||||||
Run manifest model:
|
- invocation identity and `force` flag
|
||||||
- Tracks one invocation (`run_id`) with requested stages and force mode.
|
- requested stages
|
||||||
- Tracks per-stage action (`run` or `skip`) and per-stage status.
|
- per-stage action (`run` or `skip`)
|
||||||
- Tracks overall run status (`running`, `succeeded`, `failed`).
|
- per-stage status
|
||||||
|
- overall run status (`running`, `succeeded`, `failed`)
|
||||||
|
|
||||||
Persistence behavior:
|
## Persistence Semantics
|
||||||
- Load validates required identity/timestamp fields and normalizes maps/records.
|
|
||||||
- Save updates `updated_at` and writes JSON atomically (temp file + rename).
|
|
||||||
- Session and run manifests are saved incrementally before/after stage transitions.
|
|
||||||
|
|
||||||
Relationship during execution:
|
`manifest.LocalStore`:
|
||||||
- Runner updates both manifests for every stage transition.
|
|
||||||
- Session manifest is the durable pipeline-progress ledger.
|
|
||||||
- Run manifest is invocation history and audit record.
|
|
||||||
- Analyze stage outputs are persisted as `kind=scriptorium_artifact` with `source_id=narratio.artifact.<name>` for configured artifact identity.
|
|
||||||
|
|
||||||
## Skip and resume behavior
|
- validates loaded documents;
|
||||||
- Resume and skip decisions are based on session-manifest stage statuses.
|
- normalizes missing maps/stage records;
|
||||||
- `--force` reruns selected stages and marks downstream succeeded stages as `stale` in session manifest.
|
- writes atomically via temp file + rename;
|
||||||
- Run manifest records whether each stage was executed or skipped in that invocation.
|
- updates `updated_at` on save.
|
||||||
|
|
||||||
## Failure behavior
|
## Execution Semantics
|
||||||
- Stage failure marks both manifests failed for that stage and records error messages/timestamps.
|
|
||||||
- Save failures are returned immediately and fail the command.
|
|
||||||
- Invalid/malformed manifest files fail load with explicit validation/decode errors.
|
|
||||||
|
|
||||||
## Tests to inspect before changing
|
The application runner marks an executing stage running and then succeeded or
|
||||||
- `internal/manifest/manifest_test.go`
|
failed in both manifests, persisting each transition. On success it records
|
||||||
- `internal/manifest/run_manifest_test.go`
|
outputs, logs, generated configuration references, and metadata. Artifact
|
||||||
- `internal/manifest/store_test.go`
|
records may include optional contract and external provenance objects; old
|
||||||
- `internal/app/runner_test.go`
|
manifests remain compatible when those fields are absent. A successful forced
|
||||||
- `internal/app/run_control_test.go`
|
rerun marks only succeeded downstream session-stage records stale.
|
||||||
- `internal/app/resume_run_stage_test.go`
|
|
||||||
|
|
||||||
## Architectural invariants
|
Starting an execution clears the current session-stage record's prior outputs,
|
||||||
- Session manifest is authoritative for stage progression across invocations.
|
logs, generated configuration references, and metadata. Failed and skipped
|
||||||
- Run manifest is invocation-scoped and never replaces session manifest as progress authority.
|
transitions enforce the same clearing rule directly, while success repopulates
|
||||||
- Manifest writes are atomic and deterministic (JSON + newline, temp rename pattern).
|
only fields returned by the new result. Marking a record stale does not clear
|
||||||
|
those details because resume validation and diagnosis may still require them
|
||||||
|
before execution begins. Invocation run manifests remain immutable audit
|
||||||
|
records of their own outcomes.
|
||||||
|
|
||||||
|
A stage may explicitly return a skipped disposition and stable reason. The
|
||||||
|
runner persists that outcome in both manifests, clears older outputs for the
|
||||||
|
session-stage record along with older logs, generated configuration references,
|
||||||
|
and metadata, then applies any bounded details from the current skip and
|
||||||
|
continues. This self-skip is distinct from deciding not to execute an
|
||||||
|
already-succeeded stage and is reconsidered on later runs. Skipped results
|
||||||
|
cannot contain outputs.
|
||||||
|
|
||||||
|
When an already-succeeded stage is skipped, the invocation run manifest records
|
||||||
|
the `skip` action and reason. The session manifest deliberately retains its
|
||||||
|
existing succeeded record because it remains the cross-invocation progress
|
||||||
|
authority. Stages with a resume validator, currently extraction, may reject an
|
||||||
|
otherwise eligible skip when the recorded durable result is obsolete; the
|
||||||
|
runner marks it stale and executes it.
|
||||||
|
|
||||||
|
Session manifest is the authoritative stage-progress ledger across invocations.
|
||||||
|
Run manifest is invocation-scoped audit state.
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- stage resume/skip decisions are session-manifest driven.
|
||||||
|
- running, failed, and self-skipped stages do not retain result payloads from
|
||||||
|
an earlier success.
|
||||||
|
- stale stages retain prior details until replacement execution starts.
|
||||||
|
- force reruns stale downstream succeeded stages.
|
||||||
|
- run manifest does not replace session manifest as progress authority.
|
||||||
|
|
||||||
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Models and transitions: `internal/manifest/manifest.go`,
|
||||||
|
`internal/manifest/run_manifest.go`
|
||||||
|
- Persistence and validation: `internal/manifest/store.go`
|
||||||
|
- Package tests: `internal/manifest/*_test.go`
|
||||||
|
- Assembled execution behavior: `internal/app/runner_test.go`,
|
||||||
|
`internal/app/run_stage_test.go`
|
||||||
|
|||||||
98
docs/internal/overview.md
Normal file
98
docs/internal/overview.md
Normal file
@@ -0,0 +1,98 @@
|
|||||||
|
# Internal Overview
|
||||||
|
|
||||||
|
This document is the implemented component map for Narratio. Normative system
|
||||||
|
boundaries and dependency direction belong in
|
||||||
|
[Architecture](../policy/architecture.md). User and operator contracts belong
|
||||||
|
in the [CLI](../cli.md), [Configuration](../config.md),
|
||||||
|
[Operations](../operations.md), and [Troubleshooting](../troubleshooting.md).
|
||||||
|
Externally observable tool and format contracts belong under
|
||||||
|
[Integrations](../integrations/).
|
||||||
|
|
||||||
|
## Execution Path
|
||||||
|
|
||||||
|
```text
|
||||||
|
cmd/narratio -> internal/app -> configuration and production composition
|
||||||
|
-> internal/stage -> adapters and external systems
|
||||||
|
-> manifests and artifact resolution -> durable local/remote output
|
||||||
|
```
|
||||||
|
|
||||||
|
The executable delegates process behavior to the application boundary. The
|
||||||
|
application resolves configuration, composes concrete collaborators, acquires
|
||||||
|
session safety controls, and runs commands. Pipeline commands execute the
|
||||||
|
canonical stage sequence through adapter interfaces, while manifests record
|
||||||
|
progress and artifact services resolve durable inputs and outputs.
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
| Area | Implemented owners | Responsibility |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Executable | `cmd/narratio` | Process entry, standard stream wiring, argument handoff, and exit status. |
|
||||||
|
| Application orchestration | `internal/app` | Command dispatch, configuration selection, secret-file environment loading, production composition, session locking, planning, execution, restore, cleanup gates, and user-facing reporting. |
|
||||||
|
| Configuration | `internal/config` | Strict YAML loading, discovery, defaults, normalization, session templating, and validation. |
|
||||||
|
| Pipeline stages | `internal/stage` | Canonical stage registry, shared stage contract, execution dependencies, and implemented stage behavior. |
|
||||||
|
| External boundaries | `internal/adapters`, `internal/audio` | WhisperX HTTP, downstream subprocesses, notification, object storage, and S3 audio materialization behind Narratio contracts. |
|
||||||
|
| Manifests | `internal/manifest` | Durable session progress, invocation audit state, stage transitions, validation, and atomic persistence. |
|
||||||
|
| Artifacts and paths | `internal/artifacts`, `internal/pathsafe` | Artifact identities and resolution, local and remote path/key models, current-state discovery, and confined relative destinations. |
|
||||||
|
| Previous-session cache | `internal/previouscache` | Deterministic planning and materialization requirements for configured previous-session inputs. |
|
||||||
|
| Artifact policy | `internal/artifactpolicy` | Source and destination policy, configured artifact identity validation, and publish destination safety. |
|
||||||
|
| Shared models and file operations | `internal/artifactmodel`, `internal/contracts`, `internal/fileops` | Transcript and artifact data contracts plus narrow atomic filesystem helpers. |
|
||||||
|
| Logging | `internal/logging` | Application logger construction and shared structured logging behavior. |
|
||||||
|
|
||||||
|
The application boundary composes concrete implementations. Stages depend on
|
||||||
|
Narratio-level contracts; external transport and SDK details remain in
|
||||||
|
adapters. The normative rules for these relationships remain in
|
||||||
|
[Architecture](../policy/architecture.md).
|
||||||
|
|
||||||
|
## Pipeline Stage Set
|
||||||
|
|
||||||
|
The implemented canonical order is:
|
||||||
|
|
||||||
|
1. [`prepare`](stage-prepare.md)
|
||||||
|
2. [`transcribe`](stage-transcribe.md)
|
||||||
|
3. [`merge`](stage-merge.md)
|
||||||
|
4. [`polish`](stage-polish.md)
|
||||||
|
5. [`normalize`](stage-normalize.md)
|
||||||
|
6. [`trim`](stage-trim.md)
|
||||||
|
7. [`extract`](stage-extract.md)
|
||||||
|
8. [`render`](stage-render.md)
|
||||||
|
9. [`analyze`](stage-analyze.md)
|
||||||
|
10. [`publish`](stage-publish.md)
|
||||||
|
11. `notify` (placeholder)
|
||||||
|
|
||||||
|
`notify` currently has optional notifier call behavior and no persisted pipeline
|
||||||
|
outputs; its default collaborator is a no-op sender. The focused stage
|
||||||
|
documents own implementation mechanics. The
|
||||||
|
[CLI](../cli.md) and [Operations](../operations.md) own user-visible invocation
|
||||||
|
and execution semantics.
|
||||||
|
|
||||||
|
## Focused Documentation
|
||||||
|
|
||||||
|
- [Adapter Internals](adapters.md): external adapter boundaries, composition,
|
||||||
|
failure behavior, and test surfaces.
|
||||||
|
- [Artifact Internals](artifacts.md): source identities, runtime catalog,
|
||||||
|
resolution, previous requirements, and current-state helpers.
|
||||||
|
- [Manifest Internals](manifest.md): session and run records, persistence, and
|
||||||
|
execution transitions.
|
||||||
|
- [Storage Internals](storage.md): object-store interface and S3 behavior.
|
||||||
|
- [Workspace Internals](workspace.md): local layout, locking, and cleanup
|
||||||
|
guardrails.
|
||||||
|
- [Restore Internals](command-restore.md): discovery, planning, execution, and
|
||||||
|
reporting.
|
||||||
|
- [`prepare`](stage-prepare.md)
|
||||||
|
- [`transcribe`](stage-transcribe.md)
|
||||||
|
- [`merge`](stage-merge.md)
|
||||||
|
- [`polish`](stage-polish.md)
|
||||||
|
- [`normalize`](stage-normalize.md)
|
||||||
|
- [`trim`](stage-trim.md)
|
||||||
|
- [`extract`](stage-extract.md)
|
||||||
|
- [`render`](stage-render.md)
|
||||||
|
- [`analyze`](stage-analyze.md)
|
||||||
|
- [`publish`](stage-publish.md)
|
||||||
|
|
||||||
|
Use this map to find an owner, then read its focused documentation and tests
|
||||||
|
before changing behavior.
|
||||||
|
|
||||||
|
The stage registry is implemented in `internal/stage/placeholders.go` and its
|
||||||
|
ordering is protected by `internal/app/planner_test.go`. Cross-invocation skip,
|
||||||
|
force, failure, and invalidation behavior is exercised in
|
||||||
|
`internal/app/runner_test.go` and `internal/app/run_stage_test.go`.
|
||||||
@@ -1,80 +1,60 @@
|
|||||||
# Stage: analyze
|
# Stage: analyze
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Execute selected configured Scriptorium artifacts in deterministic dependency order and materialize successful outputs to canonical session artifact paths.
|
|
||||||
|
|
||||||
## Inputs and outputs
|
Execute selected configured Scriptorium artifacts in dependency order and materialize outputs.
|
||||||
Inputs:
|
|
||||||
- configured artifact definitions from `pipeline.scriptorium.artifacts`;
|
|
||||||
- selected artifact filter (`--artifacts`) when provided;
|
|
||||||
- resolved artifact sources from resolver/catalog.
|
|
||||||
|
|
||||||
Source types used by analyze:
|
## Inputs
|
||||||
- built-ins: `narratio.transcript.*`, `narratio.bounds.session`;
|
|
||||||
- configured artifacts: `narratio.artifact.<artifact_key>`;
|
|
||||||
- canonical previous-session artifacts: `narratio.previous_session.artifact.<artifact_key>`.
|
|
||||||
|
|
||||||
Outputs:
|
- configured artifacts from `pipeline.scriptorium.artifacts`
|
||||||
- materialized configured artifact files at each configured `output_path`;
|
- optional selected artifact keys supplied through the stage environment
|
||||||
- stage metadata (`generated_artifacts`, `reused_artifacts`, selected/order info).
|
- built-in/configured/previous-session source references in artifact inputs
|
||||||
|
|
||||||
## Boundaries
|
Supported source families:
|
||||||
Owns:
|
- built-ins: `narratio.transcript.*`, `narratio.bounds.session`
|
||||||
- runtime artifact catalog construction;
|
- prepared stable inputs: `narratio.input.players`, `narratio.input.party`,
|
||||||
- selected-artifact planning and dependency ordering;
|
`narratio.input.glossary`
|
||||||
- per-input resolution and required/optional handling;
|
- configured artifacts: `narratio.artifact.<key>`
|
||||||
- Scriptorium render/run invocation;
|
- previous-session cache: `narratio.previous_session.artifact.<key>`
|
||||||
- run-local output generation and canonical materialization.
|
|
||||||
|
|
||||||
Does not own:
|
## Outputs
|
||||||
- prepare-time previous-session hydration;
|
|
||||||
- object-store access for previous-session sources;
|
|
||||||
- publish output rule behavior.
|
|
||||||
|
|
||||||
## Config fields used
|
- one materialized output per executed configured artifact (`output_path`)
|
||||||
- `session.session_id`
|
- stage metadata describing selected/generated/reused artifacts
|
||||||
- `session.campaign`
|
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `pipeline.scriptorium.binary`
|
|
||||||
- `pipeline.scriptorium.config_path`
|
|
||||||
- `pipeline.scriptorium.timeout`
|
|
||||||
- `pipeline.scriptorium.render_debug`
|
|
||||||
- `pipeline.scriptorium.artifacts.<name>.*`
|
|
||||||
|
|
||||||
## External adapters used
|
## Key Behavior
|
||||||
- Scriptorium adapter:
|
|
||||||
- optional `RenderArtifact` when render-debug is enabled;
|
|
||||||
- `RunArtifact` for artifact generation.
|
|
||||||
|
|
||||||
## State and manifest behavior
|
- skips with metadata when Scriptorium config is missing or no executable artifacts remain.
|
||||||
- If Scriptorium config is absent, or no artifacts are executable after filtering, analyze returns success metadata with `skipped=true`.
|
- builds runtime artifact catalog (built-ins + configured artifacts).
|
||||||
- Builds runtime catalog with built-ins and configured `narratio.artifact.<name>` entries.
|
- marks non-executable configured artifacts as reusable when output files already exist.
|
||||||
- Non-executable configured artifacts may still be marked available from existing canonical output files.
|
- validates selected artifact dependency order (cycle-safe topo ordering).
|
||||||
- Resolves canonical previous-session sources from local prepared `previous/` cache:
|
- resolves required/optional inputs per artifact source definition.
|
||||||
- prefers manifest-backed previous input paths when present;
|
- resolves prepared stable input sources from `inputs/*.yml` materialized by `prepare`.
|
||||||
- may fall back to current-session `previous/` filesystem paths.
|
- resolves previous-session sources from local `previous/` cache only.
|
||||||
- Analyze does not call object storage for canonical previous-session source resolution.
|
- runs optional render-debug, then artifact execution.
|
||||||
- Required canonical previous-session input missing:
|
- validates non-empty output files and materializes canonical outputs.
|
||||||
- fails with guidance to run `narratio run-stage --force prepare`.
|
|
||||||
- Optional missing sources are omitted from adapter input paths.
|
|
||||||
|
|
||||||
## Skip and resume behavior
|
## Failure Semantics
|
||||||
- Runner-level skip applies when analyze is already `succeeded` and `--force` is not set.
|
|
||||||
- Analyze is stage-scoped for resume; no per-artifact manifest resume state.
|
|
||||||
- `--artifacts` filters executable artifacts but does not imply force rerun.
|
|
||||||
|
|
||||||
## Failure behavior
|
- required missing configured/previous-session inputs fail.
|
||||||
- Fails on dependency-order violations, missing required inputs, resolver validation failures, adapter errors, and missing/empty generated outputs.
|
- missing required prepared stable input source includes prepare rerun guidance.
|
||||||
- Required unavailable configured artifact source (`narratio.artifact.<name>`) fails before invocation.
|
- missing required previous-session source includes prepare rerun guidance.
|
||||||
- Required canonical previous-session source fails with prepare-rerun guidance.
|
- missing required `narratio.transcript.final_markdown` or
|
||||||
|
`narratio.transcript.final_trimmed_markdown` inputs includes render rerun
|
||||||
|
guidance.
|
||||||
|
- dependency cycles or unavailable required dependencies fail.
|
||||||
|
- adapter validation failures fail stage.
|
||||||
|
|
||||||
## Tests to inspect before changing
|
## Invariants
|
||||||
- `internal/stage/analyze_test.go`
|
|
||||||
- `internal/artifacts/catalog_test.go`
|
|
||||||
- `internal/artifacts/artifact_resolver_test.go`
|
|
||||||
- `internal/app/restore_workflow_test.go`
|
|
||||||
|
|
||||||
## Architectural invariants
|
- `analyze` performs no remote storage calls for previous-session source resolution.
|
||||||
- Canonical previous-session behavior is local-cache only during analyze.
|
- output provenance and metadata are deterministic per execution.
|
||||||
- Generated outputs are validated and materialized before stage success is recorded.
|
|
||||||
- Resolver/catalog decisions stay deterministic and validation-gated.
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Configuration](../config.md#scriptorium-artifact-entries) owns artifact
|
||||||
|
fields and source-selection rules.
|
||||||
|
- [CLI](../cli.md) owns user-visible artifact selection.
|
||||||
|
- [Scriptorium](../integrations/scriptorium.md) owns the subprocess contract.
|
||||||
|
- Implementation and tests: `internal/stage/analyze.go`,
|
||||||
|
`internal/stage/analyze_test.go`
|
||||||
|
|||||||
84
docs/internal/stage-extract.md
Normal file
84
docs/internal/stage-extract.md
Normal file
@@ -0,0 +1,84 @@
|
|||||||
|
# Internal: Extract Stage
|
||||||
|
|
||||||
|
## Responsibility
|
||||||
|
|
||||||
|
`extract` runs after `trim` and before `render`. It converts the canonical
|
||||||
|
`narratio.transcript.final_trimmed` JSON into configured Notarius lane artifacts.
|
||||||
|
An omitted or disabled Notarius section makes the stage explicitly self-skip
|
||||||
|
with reason `notarius_disabled`, no outputs, and no Notarius runner.
|
||||||
|
|
||||||
|
The external protocol is documented in the
|
||||||
|
[Notarius integration contract](../integrations/notarius.md). Configuration
|
||||||
|
fields belong in [Configuration](../config.md), and physical paths and force
|
||||||
|
procedures belong in [Operations](../operations.md).
|
||||||
|
|
||||||
|
## Lifecycle
|
||||||
|
|
||||||
|
`internal/stage/extract.go`:
|
||||||
|
|
||||||
|
1. resolves the final trimmed transcript from the shared artifact catalog;
|
||||||
|
2. resolves and fingerprints the Notarius invocation contract;
|
||||||
|
3. creates a run-local staging directory and invokes the injected
|
||||||
|
`notarius.Runner`;
|
||||||
|
4. validates the successful receipt, confined index, configured required lane
|
||||||
|
descriptors, and regular payload files;
|
||||||
|
5. atomically promotes the complete bundle to its immutable durable location;
|
||||||
|
6. records one non-selectable `notarius_index` output and one selectable
|
||||||
|
`notarius_lane` output per configured lane; and
|
||||||
|
7. registers each lane as `narratio.extraction.<output_key>` for downstream
|
||||||
|
Scriptorium and publish resolution.
|
||||||
|
|
||||||
|
Lane records retain checksum, contract, producer run ID, and Notarius system,
|
||||||
|
run, pipeline, and lane provenance. Stage metadata retains the durable bundle
|
||||||
|
root, receipt, diagnostic paths, rejection/warning summaries, producing
|
||||||
|
Narratio run ID, and invocation fingerprint. Validation completes before
|
||||||
|
promotion, so a rejected result cannot expose a partial durable bundle.
|
||||||
|
|
||||||
|
Any executed extraction outcome that replaces a different effective outcome
|
||||||
|
marks succeeded downstream stages stale. Repeating the same disabled self-skip
|
||||||
|
with no outputs is stable and does not repeatedly invalidate downstream stages.
|
||||||
|
|
||||||
|
## Resume Validation
|
||||||
|
|
||||||
|
`internal/stage/extract_resume.go` permits a skip only when the existing stage
|
||||||
|
record succeeded and still matches the current invocation fingerprint. The
|
||||||
|
fingerprint covers the resolved executable and config paths, pipeline ID,
|
||||||
|
timeout, working directory, and sorted configured output contracts.
|
||||||
|
|
||||||
|
The validator then checks the producing run identity, canonical immutable
|
||||||
|
bundle root, path confinement and absence of symlink components, receipt
|
||||||
|
identity, exactly one canonical index, the exact configured source set,
|
||||||
|
contracts and provenance, regular-file status, and stored checksums. Missing or
|
||||||
|
obsolete results are non-resumable and run again; unsafe filesystem conditions
|
||||||
|
return an error rather than silently accepting or replacing data.
|
||||||
|
|
||||||
|
The fingerprint cannot observe files imported by Notarius configuration,
|
||||||
|
profile contents, prompt/module definitions, or other transitive inputs.
|
||||||
|
Operators must force extraction after changing any such input.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Adapter startup, timeout, nonzero exit, receipt decoding, path confinement,
|
||||||
|
index compatibility, required-lane rejection, payload inspection, checksum, or
|
||||||
|
promotion errors fail the stage through ordinary manifest transition handling.
|
||||||
|
Stdout receipt and stderr diagnostics remain separate. Downstream stages are
|
||||||
|
not given selectable extraction sources unless the complete configured result
|
||||||
|
has passed validation and promotion.
|
||||||
|
|
||||||
|
When a replacement attempt begins, the current session-stage record no longer
|
||||||
|
advertises payload from the previous success. A failed replacement therefore
|
||||||
|
has no current outputs, logs, generated configuration references, or metadata,
|
||||||
|
while the earlier invocation manifest and immutable promoted bundle remain
|
||||||
|
available for audit and recovery.
|
||||||
|
|
||||||
|
## Implementation And Focused Tests
|
||||||
|
|
||||||
|
- Stage execution, selection, and resume validation: `internal/stage/extract.go`,
|
||||||
|
`internal/stage/extract_resume.go`,
|
||||||
|
`internal/stage/extract_test.go`
|
||||||
|
- Subprocess boundary: `internal/adapters/notarius/subprocess.go`,
|
||||||
|
`internal/adapters/notarius/subprocess_test.go`
|
||||||
|
- Catalog hydration: `internal/artifacts/extraction_catalog.go`,
|
||||||
|
`internal/artifacts/extraction_catalog_test.go`
|
||||||
|
- Composition and downstream behavior: `internal/app/runner_test.go`,
|
||||||
|
`internal/stage/analyze_test.go`, `internal/stage/publish_test.go`
|
||||||
@@ -1,63 +1,37 @@
|
|||||||
# Stage: merge
|
# Stage: merge
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Normalize per-speaker raw transcripts and merge them into the base transcript via Seriatim.
|
|
||||||
|
|
||||||
## Inputs and Outputs
|
Normalize raw transcript inputs and merge into base transcript via Seriatim.
|
||||||
Inputs:
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
- `transcripts/raw/*.json`
|
- `transcripts/raw/*.json`
|
||||||
- `inputs/speakers.yml`
|
- `inputs/speakers.yml`
|
||||||
- `inputs/autocorrect.yml`
|
- `inputs/autocorrect.yml`
|
||||||
|
|
||||||
Outputs:
|
## Outputs
|
||||||
|
|
||||||
- `transcripts/base.json`
|
- `transcripts/base.json`
|
||||||
- optional `artifacts/seriatim.report.json` (when report enabled)
|
- optional `artifacts/seriatim.report.json`
|
||||||
|
|
||||||
## Boundaries
|
## Key Behavior
|
||||||
Owns:
|
|
||||||
- Raw transcript discovery/validation
|
|
||||||
- Per-input normalize calls to Seriatim
|
|
||||||
- Final merge call to Seriatim
|
|
||||||
- Run-local log/config/report path wiring
|
|
||||||
- Materialization of base/report outputs to canonical paths
|
|
||||||
|
|
||||||
Does not own:
|
- discovers and validates raw transcript inputs.
|
||||||
- Transcript polishing or downstream artifact generation
|
- normalizes each raw transcript (`seriatim.Normalize`) into run-local scratch output.
|
||||||
|
- merges normalized inputs (`seriatim.Run`) into base transcript.
|
||||||
|
- validates merged transcript and optional report JSON.
|
||||||
|
- materializes canonical outputs and records stage logs/generated configs.
|
||||||
|
|
||||||
## Config Fields Used
|
## Invariants
|
||||||
- `session.session_id`
|
|
||||||
- `session.campaign`
|
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `pipeline.seriatim.binary`
|
|
||||||
- `pipeline.seriatim.timeout`
|
|
||||||
- `pipeline.seriatim.output_schema`
|
|
||||||
- `pipeline.seriatim.coalesce_gap`
|
|
||||||
- `pipeline.seriatim.report`
|
|
||||||
- `pipeline.seriatim.env.*`
|
|
||||||
|
|
||||||
## External Adapters Used
|
- merge always consumes normalized forms of raw inputs.
|
||||||
- Seriatim adapter:
|
- base transcript must validate before stage success.
|
||||||
- `Normalize` for each raw input
|
- report output is config-gated.
|
||||||
- `Run` for final merge
|
|
||||||
|
|
||||||
## State and Manifest Behavior
|
## Related Contracts And Tests
|
||||||
- Reads transcript inputs from transcribe stage outputs in manifest when present; falls back to canonical raw directory.
|
|
||||||
- Writes run-local outputs/logs/config under `runs/{run_id}/merge/...` when enabled.
|
|
||||||
- Materializes canonical base transcript and optional report.
|
|
||||||
- Records normalized-input provenance and adapter metadata in stage metadata.
|
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
- [Seriatim](../integrations/seriatim.md) owns subprocess and output semantics.
|
||||||
- Runner-level skip applies when already succeeded and not forced.
|
- [Configuration](../config.md#pipeline) owns operator-selected Seriatim values.
|
||||||
- Forced rerun of this or upstream stages can stale downstream succeeded stages via runner invalidation.
|
- Implementation and tests: `internal/stage/merge.go`,
|
||||||
|
`internal/stage/merge_test.go`
|
||||||
## Failure Behavior
|
|
||||||
- Fails on missing/invalid raw transcripts, missing speakers/autocorrect files, normalize failure, merge failure, invalid base output JSON, or invalid report JSON when enabled.
|
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
|
||||||
- `internal/stage/merge_test.go`
|
|
||||||
- `internal/adapters/seriatim/subprocess_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
|
||||||
- Merge consumes normalized forms of each raw transcript.
|
|
||||||
- Base transcript must validate before materialization.
|
|
||||||
- Report output is optional and gated by config.
|
|
||||||
|
|||||||
@@ -1,56 +1,34 @@
|
|||||||
# Stage: normalize
|
# Stage: normalize
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Normalize the polished transcript into the full final transcript and optionally emit a normalize report.
|
|
||||||
|
|
||||||
## Inputs and Outputs
|
Normalize polished transcript into final transcript using Seriatim.
|
||||||
Inputs:
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
- `transcripts/polished.json`
|
- `transcripts/polished.json`
|
||||||
|
|
||||||
Outputs:
|
## Outputs
|
||||||
|
|
||||||
- `transcripts/final.json` (or configured normalize output path)
|
- `transcripts/final.json` (or configured normalize output path)
|
||||||
- optional `artifacts/seriatim.normalize.report.json`
|
- optional `artifacts/seriatim.normalize.report.json`
|
||||||
|
|
||||||
## Boundaries
|
## Key Behavior
|
||||||
Owns:
|
|
||||||
- Polished transcript discovery/validation
|
|
||||||
- Normalize request construction and invocation
|
|
||||||
- Optional normalize report wiring
|
|
||||||
- Promotion of final transcript and optional report
|
|
||||||
|
|
||||||
Does not own:
|
- resolves polished transcript from manifest outputs/canonical fallback.
|
||||||
- Bounds detection or segment trimming
|
- applies `pipeline.normalize` config or default normalize config.
|
||||||
|
- runs Seriatim normalize with configured timeout/binary.
|
||||||
|
- validates normalized transcript and optional report.
|
||||||
|
- materializes canonical outputs and records logs/generated configs.
|
||||||
|
|
||||||
## Config Fields Used
|
## Invariants
|
||||||
- `session.session_id`
|
|
||||||
- `session.campaign`
|
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `pipeline.normalize.output_path`
|
|
||||||
- `pipeline.normalize.output_schema`
|
|
||||||
- `pipeline.normalize.report`
|
|
||||||
- `pipeline.seriatim.binary`
|
|
||||||
- `pipeline.seriatim.timeout`
|
|
||||||
|
|
||||||
## External Adapters Used
|
- final transcript must validate as processed transcript JSON (`segments` array).
|
||||||
- Seriatim adapter (`Normalize`).
|
- normalize defaults are applied when `pipeline.normalize` is unset.
|
||||||
|
|
||||||
## State and Manifest Behavior
|
## Related Contracts And Tests
|
||||||
- Reads polished transcript from polish outputs in manifest when present; falls back to canonical path.
|
|
||||||
- Uses run-local output/report/log/config paths when run layout is enabled.
|
|
||||||
- Promotes canonical final transcript and optional normalize report.
|
|
||||||
- Records adapter/result metadata including source path selection.
|
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
- [Seriatim](../integrations/seriatim.md) owns subprocess and output semantics.
|
||||||
- Runner-level skip applies when already succeeded and not forced.
|
- [Configuration](../config.md#pipeline) owns normalize fields and defaults.
|
||||||
- Forced reruns can stale downstream succeeded stages.
|
- Implementation and tests: `internal/stage/normalize.go`,
|
||||||
|
`internal/stage/normalize_test.go`
|
||||||
## Failure Behavior
|
|
||||||
- Fails on missing/invalid polished transcript, adapter error, invalid final output, or invalid report output when report enabled.
|
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
|
||||||
- `internal/stage/normalize_test.go`
|
|
||||||
- `internal/adapters/seriatim/subprocess_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
|
||||||
- Final output must validate as transcript-compatible JSON (`segments` array required).
|
|
||||||
- Default normalize config is applied when `pipeline.normalize` is unset.
|
|
||||||
|
|||||||
@@ -1,69 +1,36 @@
|
|||||||
# Stage: polish
|
# Stage: polish
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Polish the base transcript with Audita and produce a polished transcript for downstream normalization/analyze.
|
|
||||||
|
|
||||||
## Inputs and Outputs
|
Run Audita polishing on base transcript and produce polished transcript.
|
||||||
Inputs:
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
- `transcripts/base.json`
|
- `transcripts/base.json`
|
||||||
- `inputs/glossary.yml`
|
- `inputs/glossary.yml`
|
||||||
|
|
||||||
Outputs:
|
## Outputs
|
||||||
|
|
||||||
- `transcripts/polished.json`
|
- `transcripts/polished.json`
|
||||||
- optional `artifacts/audita.report.json` (when report enabled)
|
- optional `artifacts/audita.report.json`
|
||||||
|
|
||||||
## Boundaries
|
## Key Behavior
|
||||||
Owns:
|
|
||||||
- Base transcript discovery/validation
|
|
||||||
- Audita invocation request construction
|
|
||||||
- Run-local logs/config/work-dir/report wiring
|
|
||||||
- Promotion of polished transcript and optional report
|
|
||||||
|
|
||||||
Does not own:
|
- resolves base transcript from merge outputs/canonical fallback.
|
||||||
- Upstream merge normalization
|
- invokes Audita with configured model/module/runtime options.
|
||||||
- Downstream normalize/trim/analyze logic
|
- validates processed transcript structure (`segments` array required).
|
||||||
|
- validates optional report JSON.
|
||||||
|
- materializes canonical outputs; records logs/generated config and adapter metadata.
|
||||||
|
|
||||||
## Config Fields Used
|
## Invariants
|
||||||
- `session.session_id`
|
|
||||||
- `session.campaign`
|
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `pipeline.audita.binary`
|
|
||||||
- `pipeline.audita.timeout`
|
|
||||||
- `pipeline.audita.llm_api_key_env`
|
|
||||||
- `pipeline.audita.modules`
|
|
||||||
- `pipeline.audita.base_url`
|
|
||||||
- `pipeline.audita.model`
|
|
||||||
- `pipeline.audita.transcript_description`
|
|
||||||
- `pipeline.audita.config_path`
|
|
||||||
- `pipeline.audita.output_schema`
|
|
||||||
- `pipeline.audita.work_dir_retention`
|
|
||||||
- `pipeline.audita.total_llm_concurrency`
|
|
||||||
- `pipeline.audita.proposal_llm_concurrency`
|
|
||||||
- `pipeline.audita.validation_model`
|
|
||||||
- `pipeline.audita.validation_llm_concurrency`
|
|
||||||
- `pipeline.audita.report`
|
|
||||||
|
|
||||||
## External Adapters Used
|
- polished transcript schema validation is mandatory.
|
||||||
- Audita adapter (`env.Audita.Run`).
|
- report output is config-gated.
|
||||||
|
|
||||||
## State and Manifest Behavior
|
## Related Contracts And Tests
|
||||||
- Reads base transcript from merge manifest outputs when available; falls back to canonical base path.
|
|
||||||
- Uses run-local output/report/log/config/scratch paths when run layout is enabled.
|
|
||||||
- Promotes canonical `transcripts/polished.json` and optional report.
|
|
||||||
- Records adapter invocation metadata, credential presence signal, and output provenance in stage metadata.
|
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
- [Audita](../integrations/audita.md) owns subprocess, validation, and failure
|
||||||
- Runner-level skip applies when already succeeded and not forced.
|
semantics.
|
||||||
- Forced rerun can stale downstream succeeded stages via runner invalidation.
|
- [Configuration](../config.md#pipeline) owns operator-selected Audita values.
|
||||||
|
- Implementation and tests: `internal/stage/polish.go`,
|
||||||
## Failure Behavior
|
`internal/stage/polish_test.go`
|
||||||
- Fails on missing/invalid base transcript, missing glossary, adapter error, invalid polished output shape (`segments` array required), or invalid report JSON when enabled.
|
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
|
||||||
- `internal/stage/polish_test.go`
|
|
||||||
- `internal/adapters/audita/subprocess_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
|
||||||
- Polished transcript must contain a top-level `segments` array.
|
|
||||||
- Report behavior is strictly config-gated.
|
|
||||||
- Stage output canonicalization always ends at `transcripts/polished.json`.
|
|
||||||
|
|||||||
@@ -1,124 +1,60 @@
|
|||||||
# Stage: prepare
|
# Stage: prepare
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Materialize canonical current-session input state and provenance before downstream stages run.
|
|
||||||
|
|
||||||
Prepare owns:
|
Materialize canonical current-session inputs before processing stages.
|
||||||
- local input file materialization (`inputs/**`);
|
|
||||||
- audio input materialization (`audio/**`);
|
|
||||||
- previous-session cache hydration (`previous/**`) for canonical previous-session artifact sources.
|
|
||||||
|
|
||||||
## Inputs and outputs
|
## Inputs
|
||||||
Inputs:
|
|
||||||
- resolved config/campaign/session (`pipeline.yml`, `campaign.yml`, `session.yml`);
|
|
||||||
- remote session provenance when `session.yml` was loaded from S3;
|
|
||||||
- campaign or session input files (`speakers`, `autocorrect`, `glossary`);
|
|
||||||
- audio source:
|
|
||||||
- local: `session.inputs.audio_dir` or `session.inputs.audio_files`;
|
|
||||||
- S3: `session.inputs.audio_s3.prefix`;
|
|
||||||
- configured enabled Scriptorium artifact inputs (for previous-session requirement scanning);
|
|
||||||
- remote previous-session current publish state when previous hydration is required.
|
|
||||||
|
|
||||||
Outputs:
|
- resolved campaign, session, and pipeline configuration
|
||||||
- `inputs/campaign.yml`;
|
- stable input files (`speakers`, `autocorrect`, `glossary`, `players`, `party`)
|
||||||
- `inputs/session.yml`;
|
- one resolved local or S3 audio source
|
||||||
- `inputs/pipeline.resolved.yml`;
|
- enabled configured artifact input requirements for previous-session sources
|
||||||
- `inputs/speakers.yml`;
|
|
||||||
- `inputs/autocorrect.yml`;
|
|
||||||
- `inputs/glossary.yml`;
|
|
||||||
- `audio/*.flac` in canonical session `audio/`;
|
|
||||||
- optional `previous/manifest.json`;
|
|
||||||
- optional `previous/artifacts/**`;
|
|
||||||
- deterministic `manifest.Inputs` records with checksums and provenance metadata.
|
|
||||||
|
|
||||||
## Boundaries
|
## Outputs
|
||||||
Owns:
|
|
||||||
- input path resolution and materialization;
|
|
||||||
- S3 audio list/download/copy flow;
|
|
||||||
- previous-session artifact requirement collection from enabled configured artifacts;
|
|
||||||
- previous cache lifecycle when requirements exist (clear and rehydrate managed `previous/` state).
|
|
||||||
|
|
||||||
Does not own:
|
- `inputs/campaign.yml`
|
||||||
- transcript or artifact generation;
|
- `inputs/session.yml`
|
||||||
- analyze-stage source resolution;
|
- `inputs/pipeline.resolved.yml`
|
||||||
- publish commit behavior.
|
- `inputs/speakers.yml`
|
||||||
|
- `inputs/autocorrect.yml`
|
||||||
|
- `inputs/glossary.yml`
|
||||||
|
- `inputs/players.yml`
|
||||||
|
- `inputs/party.yml`
|
||||||
|
- `audio/*.flac`
|
||||||
|
- optional `previous/manifest.json`
|
||||||
|
- optional `previous/artifacts/**`
|
||||||
|
- deterministic `manifest.inputs` entries (checksums + provenance)
|
||||||
|
|
||||||
## Config fields used
|
## Key Behavior
|
||||||
- `session.session_id`
|
|
||||||
- `session.previous_session_id`
|
|
||||||
- `session.campaign`
|
|
||||||
- `session.inputs.speakers_file`
|
|
||||||
- `session.inputs.autocorrect_file`
|
|
||||||
- `session.inputs.glossary_file`
|
|
||||||
- `session.inputs.audio_dir`
|
|
||||||
- `session.inputs.audio_files`
|
|
||||||
- `session.inputs.audio_s3.prefix`
|
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `pipeline.spool.root`
|
|
||||||
- `pipeline.cache.root`
|
|
||||||
- `pipeline.cache.s3_audio`
|
|
||||||
- `pipeline.storage.s3.bucket`
|
|
||||||
- `pipeline.storage.s3.root_prefix`
|
|
||||||
- `pipeline.scriptorium.artifacts.<name>.enabled`
|
|
||||||
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.source`
|
|
||||||
- `pipeline.scriptorium.artifacts.<name>.inputs.<key>.required`
|
|
||||||
- `campaign.campaign_id`
|
|
||||||
- `campaign.inputs.speakers_file`
|
|
||||||
- `campaign.inputs.autocorrect_file`
|
|
||||||
- `campaign.inputs.glossary_file`
|
|
||||||
|
|
||||||
## External adapters used
|
- validates required config/store state.
|
||||||
- `storage.ObjectStore` for:
|
- enforces local audio vs S3 audio mutual exclusivity.
|
||||||
- S3 audio listing/downloads;
|
- materializes S3 audio through spool/cache-aware logic.
|
||||||
- previous-session current pointer/manifest/artifact object checks and downloads.
|
- scans enabled configured artifact inputs for `narratio.previous_session.artifact.*` requirements.
|
||||||
|
- when previous requirements exist:
|
||||||
## State and manifest behavior
|
|
||||||
- Ensures workspace layout exists.
|
|
||||||
- Materializes canonical input files and audio files.
|
|
||||||
- For S3 audio, uses run-scoped spool for active downloads and durable cache for reusable audio files; cache hits copy directly to work audio without downloading the object again.
|
|
||||||
- Records `inputs/session.yml` provenance as local `session_config` or remote `session_config.s3`.
|
|
||||||
- Resolves campaign-provided stable input paths relative to `campaign.yml`.
|
|
||||||
- Resolves session-provided stable input overrides relative to `session.yml`.
|
|
||||||
- Scans enabled configured artifact inputs for canonical sources:
|
|
||||||
- `narratio.previous_session.artifact.<artifact_key>`
|
|
||||||
- If one or more canonical previous-session requirements exist:
|
|
||||||
- clears managed `previous/` state;
|
- clears managed `previous/` state;
|
||||||
- hydrates required/optional previous artifacts from the configured previous session’s committed publish current state;
|
- builds previous-cache remote plan;
|
||||||
- writes `previous/manifest.json` and hydrated `previous/artifacts/**`;
|
- downloads previous manifest/artifacts;
|
||||||
- stores publish-relative artifact paths such as `artifacts/session_recap.md` as `previous/artifacts/session_recap.md`, not `previous/artifacts/artifacts/session_recap.md`;
|
- records previous inputs in `manifest.inputs`.
|
||||||
- records hydrated previous inputs in `manifest.Inputs` with source `previous_session_publish.current`.
|
|
||||||
- If no canonical previous-session requirements exist, prepare does not manage `previous/`.
|
|
||||||
- `manifest.Inputs` is sorted deterministically by `(kind, path)`.
|
|
||||||
- S3 audio `manifest.Inputs` retain S3 provenance and include `cache_path`; `spool_path` is present only when the current prepare invocation downloaded the file.
|
|
||||||
|
|
||||||
## Required and optional previous-session behavior
|
Required previous-session inputs fail when unavailable; optional missing inputs are skipped.
|
||||||
- `previous_session_id` unset:
|
|
||||||
- if any referenced previous artifact is required: fail;
|
|
||||||
- if all referenced previous artifacts are optional: continue and omit them.
|
|
||||||
- Previous session publish current pointer or manifest missing:
|
|
||||||
- if any referenced previous artifact is required: fail;
|
|
||||||
- if all referenced previous artifacts are optional: continue and omit missing ones.
|
|
||||||
- Missing required previous artifact object: fail.
|
|
||||||
- Missing optional previous artifact object: omit.
|
|
||||||
- Downloaded previous artifacts must validate as non-empty files.
|
|
||||||
|
|
||||||
## Skip and resume behavior
|
## Invariants
|
||||||
- Runner-level skip remains authoritative:
|
|
||||||
- if `prepare` already succeeded and run is not forced, `prepare` does not run and no hydration/download occurs.
|
|
||||||
- If `prepare` runs (including with `--force`), it owns managed `previous/` state for canonical previous-session inputs.
|
|
||||||
|
|
||||||
## Failure behavior
|
- only `prepare` hydrates canonical `previous/` cache state.
|
||||||
- Fails on missing required input files, invalid audio-source combinations, empty/duplicate audio inputs, missing object store for S3 modes, and remote access/download/validation errors.
|
- managed previous artifacts are stored under `previous/artifacts/**` without
|
||||||
- For required canonical previous-session inputs, analyze-time missing-input guidance is to rerun:
|
duplicate `artifacts/artifacts/` nesting.
|
||||||
- `narratio run-stage --force prepare`
|
- `manifest.inputs` ordering is deterministic (`kind`, `path`).
|
||||||
|
|
||||||
## Tests to inspect before changing
|
## Related Contracts And Tests
|
||||||
- `internal/stage/prepare_test.go`
|
|
||||||
- `internal/stage/prepare_previous_test.go`
|
|
||||||
- `internal/artifacts/previous_requirements_test.go`
|
|
||||||
- `internal/app/runner_test.go`
|
|
||||||
|
|
||||||
## Architectural invariants
|
- [Configuration](../config.md) owns audio selection, stable input fields, and
|
||||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive.
|
previous-session settings.
|
||||||
- Storage keys are computed by callers using path helpers; storage adapter receives explicit keys.
|
- [Operations](../operations.md) owns physical input, audio, spool, cache, and
|
||||||
- `prepare` is the only stage that hydrates canonical previous-session cache state.
|
previous-state layout.
|
||||||
|
- [Storage Internals](storage.md) and [Artifact Internals](artifacts.md) explain
|
||||||
|
the internal collaborators.
|
||||||
|
- Implementation and tests: `internal/stage/prepare.go`,
|
||||||
|
`internal/stage/prepare_test.go`, `internal/audio/s3_audio_test.go`,
|
||||||
|
`internal/previouscache/*_test.go`
|
||||||
|
|||||||
@@ -1,88 +1,74 @@
|
|||||||
# Stage: publish
|
# Stage: publish
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Publish durable run/session state to object storage, then atomically advance remote current state.
|
|
||||||
|
|
||||||
## Inputs and Outputs
|
Upload run/session outputs to object storage and atomically advance remote current state.
|
||||||
Inputs:
|
|
||||||
- session manifest and prerequisite stage records
|
|
||||||
- run root contents under `runs/{run_id}/`
|
|
||||||
- publish output rules with artifact `source` IDs and publish `dest` paths (`pipeline.publish.outputs`)
|
|
||||||
- effective source-based publish locks from static config and remote session lock store
|
|
||||||
- session-level `previous/**` cache files when present
|
|
||||||
|
|
||||||
Outputs:
|
## Inputs
|
||||||
- uploaded run files under `{session_prefix}/runs/{run_id}/...`
|
|
||||||
- uploaded published outputs under `{session_prefix}/...`
|
|
||||||
- uploaded session previous-cache files under `{session_prefix}/previous/...` when present
|
|
||||||
- `{session_prefix}/current/manifest.json`
|
|
||||||
- `{session_prefix}/current/run_id.txt` written last
|
|
||||||
|
|
||||||
## Boundaries
|
- successful preceding stages from the [canonical stage set](overview.md#pipeline-stage-set)
|
||||||
Owns:
|
- invocation-scoped run files
|
||||||
- publish enable/disable gate behavior
|
- resolved publish output rules
|
||||||
- prerequisite stage success enforcement
|
- effective publish locks (static + remote merged lock set)
|
||||||
- run file collection and upload (excluding `audio/`)
|
- durable previous-session cache files when present
|
||||||
- publish output rule resolution and upload
|
|
||||||
- publish lock enforcement
|
|
||||||
- session previous-cache file collection/upload
|
|
||||||
- commit pointer publish order
|
|
||||||
|
|
||||||
Does not own:
|
## Outputs
|
||||||
- stage execution before publish
|
|
||||||
- post-publish local cleanup policy execution (handled by app cleanup logic)
|
|
||||||
|
|
||||||
## Config Fields Used
|
- uploaded invocation record and selected publish outputs;
|
||||||
- `pipeline.publish.enabled`
|
- uploaded durable previous-session cache files when present;
|
||||||
- `pipeline.publish.upload_run`
|
- updated remote current manifest; and
|
||||||
- `pipeline.publish.outputs`
|
- remote current-run commit marker, written last.
|
||||||
- `pipeline.publish.locks`
|
|
||||||
- `{session_prefix}/locks.yml` loaded by app orchestration before publish execution
|
|
||||||
- `pipeline.storage.s3.bucket`
|
|
||||||
- `pipeline.storage.s3.root_prefix`
|
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `session.campaign`
|
|
||||||
- `session.session_id`
|
|
||||||
|
|
||||||
## External Adapters Used
|
Exact remote placement and the operator workflow belong in
|
||||||
- Object storage backend (`env.ObjectStore`) for upload/list primitives.
|
[Operations](../operations.md#publish-workflow).
|
||||||
|
|
||||||
## State and Manifest Behavior
|
## Key Behavior
|
||||||
- Requires `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and `analyze` status `succeeded`.
|
|
||||||
- Resolves bucket/prefix from manifest identity first, then config fallback.
|
- stage can self-skip when publish disabled or run upload disabled.
|
||||||
- Uploads session `previous/**` files as durable session state when the local `previous/` directory exists.
|
- validates prerequisite stage success and object-store availability.
|
||||||
- Skips top-level published output uploads for effective locked sources; run-local materialized outputs remain unchanged.
|
- collects a deterministic run file list plus run `manifest.json`, excluding
|
||||||
- When selected configured artifact keys are supplied, skips publish rules for unselected `narratio.artifact.<key>` sources; built-in transcript and bounds outputs still publish.
|
`audio/**` and the run-local `extract/notarius-output/**` staging bundle.
|
||||||
- Effective locks are the union of `pipeline.publish.locks` and remote `{session_prefix}/locks.yml`; static pipeline locks win on duplicate sources.
|
- keeps run-local Notarius receipt and stderr diagnostics eligible for the run
|
||||||
- Writes metadata including:
|
archive.
|
||||||
- upload counts/paths
|
- resolves publish output sources through runtime artifact catalog and manifest-aware resolution.
|
||||||
- `previous_files_uploaded` and `previous_uploaded_paths`
|
- publishes extraction lanes only through explicit configured output rules;
|
||||||
- `published_files_uploaded` and `published_paths`
|
neither run-local nor durable Notarius bundles are scanned or uploaded wholesale.
|
||||||
- `skipped_optional_outputs`
|
- selected artifact filter applies to configured artifact sources only.
|
||||||
- `skipped_unselected_outputs`
|
- locked outputs are skipped intentionally (including required ones).
|
||||||
- `locked_output_count` and `locked_outputs`
|
- optional missing outputs are skipped; required missing unlocked outputs fail.
|
||||||
- `current_manifest_key`
|
- writes remote current manifest before current run pointer.
|
||||||
- `current_run_id_key`
|
|
||||||
|
## Metadata Signals
|
||||||
|
|
||||||
|
Includes counts/lists for:
|
||||||
|
- run uploads
|
||||||
|
- published output uploads
|
||||||
|
- previous uploads
|
||||||
|
- skipped optional outputs
|
||||||
|
- skipped unselected outputs
|
||||||
|
- locked outputs
|
||||||
|
- current-state key paths
|
||||||
- `current_pointer_written`
|
- `current_pointer_written`
|
||||||
- On skipped publish path, returns metadata with `skipped=true` and pointer not written.
|
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
## Invariants
|
||||||
- Stage may self-skip (metadata skip) when publish disabled or run upload disabled.
|
|
||||||
- Runner-level skip also applies for previously succeeded stage unless forced.
|
|
||||||
|
|
||||||
## Failure Behavior
|
- `current/run_id.txt` is the remote commit marker and is written last.
|
||||||
- Fails on missing prerequisite success, missing object store when required, missing run root, missing unlocked required output source, upload failures, or pointer write failures.
|
- run upload excludes `audio/**` and `extract/notarius-output/**`.
|
||||||
- Locked required outputs are intentional skips and do not fail publish.
|
- `extract/notarius.receipt.json` and `extract/notarius.stderr.log` remain
|
||||||
- Pointer semantics are fail-safe: `current/run_id.txt` is not written if prior required uploads fail.
|
eligible run-record diagnostics.
|
||||||
|
- publish locks are not overridden by `--force`.
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
The commit boundary and cleanup gate are normative architecture invariants; see
|
||||||
- `internal/stage/archive_test.go`
|
[Architecture](../policy/architecture.md#publish-commit-boundary).
|
||||||
- `internal/app/post_archive_cleanup_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
## Related Contracts And Tests
|
||||||
- Run upload excludes `audio/` subtree.
|
|
||||||
- Session `previous/**` is publishable durable input/provenance state, not run-local output.
|
- [Configuration](../config.md#publish-configuration-summary) owns output and
|
||||||
- Ordinary `--force` does not override publish locks.
|
static-lock fields.
|
||||||
- Malformed or unreadable remote lock store fails publish-capable execution before output uploads.
|
- [Operations](../operations.md#publish-locks) owns remote lock lifecycle and
|
||||||
- `current/manifest.json` uploads before `current/run_id.txt`.
|
physical remote state.
|
||||||
- `current/run_id.txt` is the remote publish commit marker.
|
- [Artifact Internals](artifacts.md) explains source resolution and current-state
|
||||||
|
helpers.
|
||||||
|
- Implementation and tests: `internal/stage/publish.go`,
|
||||||
|
`internal/stage/publish_test.go`, `internal/app/operator_helpers_test.go`,
|
||||||
|
`internal/app/post_publish_cleanup_test.go`
|
||||||
|
|||||||
42
docs/internal/stage-render.md
Normal file
42
docs/internal/stage-render.md
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
# Stage: render
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Render Markdown transcript artifacts from normalized JSON transcripts via Seriatim.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
- `narratio.transcript.final` (`transcripts/final.json`)
|
||||||
|
- `narratio.transcript.final_trimmed` (`transcripts/final.trimmed.json`)
|
||||||
|
|
||||||
|
## Outputs
|
||||||
|
|
||||||
|
- `narratio.transcript.final_markdown` -> `transcripts/final.md`
|
||||||
|
- `narratio.transcript.final_trimmed_markdown` -> `transcripts/final.trimmed.md`
|
||||||
|
|
||||||
|
## Key Behavior
|
||||||
|
|
||||||
|
- uses `pipeline.render` settings (enabled/format/title/booleans).
|
||||||
|
- resolves inputs manifest-first, then canonical fallback.
|
||||||
|
- writes run-local outputs first, then materializes canonical session outputs.
|
||||||
|
- records input provenance, output paths, adapter metadata, logs, and generated config refs.
|
||||||
|
- skips with stage metadata when `pipeline.render.enabled=false`.
|
||||||
|
|
||||||
|
## Failure Semantics
|
||||||
|
|
||||||
|
- missing normalized input fails with normalize rerun guidance.
|
||||||
|
- missing trimmed input fails with trim rerun guidance.
|
||||||
|
- adapter/subprocess failure fails stage.
|
||||||
|
- empty render output files fail validation.
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- only `format: markdown` is supported.
|
||||||
|
- render stage owns production of built-in Markdown transcript sources.
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Seriatim](../integrations/seriatim.md) owns render subprocess behavior.
|
||||||
|
- [Configuration](../config.md#pipeline) owns render fields and defaults.
|
||||||
|
- Implementation and tests: `internal/stage/render.go`,
|
||||||
|
`internal/stage/render_test.go`
|
||||||
@@ -1,58 +1,36 @@
|
|||||||
# Stage: transcribe
|
# Stage: transcribe
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Generate per-speaker raw transcripts from prepared audio using WhisperX.
|
|
||||||
|
|
||||||
## Inputs and Outputs
|
Generate raw per-speaker transcripts from prepared audio using WhisperX.
|
||||||
Inputs:
|
|
||||||
- `audio/*.flac` prepared by `prepare`
|
|
||||||
|
|
||||||
Outputs:
|
## Inputs
|
||||||
- `transcripts/raw/<speaker>.json` for each input audio file
|
|
||||||
|
|
||||||
## Boundaries
|
- `audio/*.flac` from `prepare`
|
||||||
Owns:
|
|
||||||
- Discovering prepared audio inputs
|
|
||||||
- Deriving speaker ids from audio basenames
|
|
||||||
- Parallel WhisperX invocation with bounded concurrency
|
|
||||||
- Validating produced JSON and materializing run-local outputs
|
|
||||||
|
|
||||||
Does not own:
|
## Outputs
|
||||||
- Transcript merge/polish/normalize/trim/analyze
|
|
||||||
|
|
||||||
## Config Fields Used
|
- `transcripts/raw/<speaker>.json`
|
||||||
- `session.session_id`
|
|
||||||
- `session.campaign`
|
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `pipeline.whisperx.transcribe_url`
|
|
||||||
- `pipeline.whisperx.language`
|
|
||||||
- `pipeline.whisperx.timeout`
|
|
||||||
- `pipeline.whisperx.retries`
|
|
||||||
- `pipeline.whisperx.retry_delay`
|
|
||||||
- `pipeline.whisperx.concurrency`
|
|
||||||
|
|
||||||
## External Adapters Used
|
## Key Behavior
|
||||||
- WhisperX adapter (`env.WhisperX.Transcribe`).
|
|
||||||
|
|
||||||
## State and Manifest Behavior
|
- discovers prepared audio from manifest inputs or canonical audio directory.
|
||||||
- Uses run-local output paths under `runs/{run_id}/transcribe/outputs/...` when run layout is enabled.
|
- derives speaker ID from `.flac` basename.
|
||||||
- Validates each generated transcript JSON before materialization.
|
- dispatches WhisperX requests through a bounded worker pool.
|
||||||
- Materializes canonical outputs to `transcripts/raw/*.json`.
|
- validates each output as JSON.
|
||||||
- Records per-file metadata (attempts/status/duration/output path) in stage metadata.
|
- writes run-local outputs then materializes canonical transcript outputs.
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
## Invariants
|
||||||
- Runner-level skip applies for previously succeeded stage unless forced.
|
|
||||||
- On forced upstream reruns, downstream succeeded stages can be marked `stale` by runner logic.
|
|
||||||
|
|
||||||
## Failure Behavior
|
- speaker basenames must be unique.
|
||||||
- Fails if no prepared audio exists, duplicate speaker basenames are detected, adapter output path mismatches expected path, any output JSON is invalid, or one worker fails.
|
- output path returned by adapter must match requested output path.
|
||||||
- Cancels in-flight workers after first terminal error.
|
- each successful output is validated before stage success.
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
## Related Contracts And Tests
|
||||||
- `internal/stage/transcribe_test.go`
|
|
||||||
- `internal/app/whisperx_wiring_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
- [WhisperX](../integrations/whisperx.md) owns HTTP, retry, timeout, and
|
||||||
- Speaker identity is derived from `.flac` basename and must be unique.
|
cancellation semantics.
|
||||||
- Every successful speaker output must be valid JSON before materialization.
|
- [Configuration](../config.md#pipeline) owns concurrency and other
|
||||||
- Canonical raw transcript set is the only supported merge input surface.
|
operator-selected values.
|
||||||
|
- Implementation and tests: `internal/stage/transcribe.go`,
|
||||||
|
`internal/stage/transcribe_test.go`
|
||||||
|
|||||||
@@ -1,75 +1,43 @@
|
|||||||
# Stage: trim
|
# Stage: trim
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Optionally trim the final transcript to session bounds; always produce a durable final-trimmed transcript.
|
|
||||||
|
|
||||||
## Inputs and Outputs
|
Produce a final-trimmed transcript. By default, the stage generates bounds and
|
||||||
Inputs:
|
applies a bounds-driven trim.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
- `transcripts/final.json`
|
- `transcripts/final.json`
|
||||||
|
|
||||||
Outputs:
|
## Outputs
|
||||||
|
|
||||||
- `transcripts/final.trimmed.json` (or configured trim output path)
|
- `transcripts/final.trimmed.json` (or configured trim output path)
|
||||||
- when trim enabled: `artifacts/session_bounds.json`
|
- when trim enabled: `artifacts/session_bounds.json`
|
||||||
|
|
||||||
## Boundaries
|
## Key Behavior
|
||||||
Owns:
|
|
||||||
- Trim-enabled switch behavior
|
|
||||||
- Bounds generation via Scriptorium artifact run
|
|
||||||
- Bounds validation against final transcript
|
|
||||||
- Keep-selector derivation and Seriatim trim invocation
|
|
||||||
- Copy-through behavior when disabled or bounds indicate unchanged transcript
|
|
||||||
|
|
||||||
Does not own:
|
When `trim.enabled=true`:
|
||||||
- Upstream normalization
|
- runs Scriptorium bounds artifact generation;
|
||||||
- Downstream artifact analysis
|
- optionally runs render-debug output generation;
|
||||||
|
- validates bounds payload against transcript;
|
||||||
|
- derives keep selector;
|
||||||
|
- either copies unchanged transcript or runs Seriatim trim;
|
||||||
|
- validates trimmed transcript and materializes bounds output.
|
||||||
|
|
||||||
## Config Fields Used
|
When `trim.enabled=false`:
|
||||||
- `session.session_id`
|
- copies normalized transcript to trimmed output.
|
||||||
- `session.campaign`
|
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `pipeline.trim.enabled`
|
|
||||||
- `pipeline.trim.output_path`
|
|
||||||
- `pipeline.trim.bounds.prompt_id`
|
|
||||||
- `pipeline.trim.bounds.profile_id`
|
|
||||||
- `pipeline.trim.bounds.timeout`
|
|
||||||
- `pipeline.trim.bounds.output_path`
|
|
||||||
- `pipeline.trim.bounds.transcript_input_name`
|
|
||||||
- `pipeline.trim.bounds.render_debug`
|
|
||||||
- `pipeline.trim.bounds.render_output_path`
|
|
||||||
- `pipeline.seriatim.binary`
|
|
||||||
- `pipeline.seriatim.timeout`
|
|
||||||
- `pipeline.scriptorium.binary`
|
|
||||||
- `pipeline.scriptorium.config_path`
|
|
||||||
- `pipeline.scriptorium.timeout`
|
|
||||||
|
|
||||||
## External Adapters Used
|
## Invariants
|
||||||
- Scriptorium adapter:
|
|
||||||
- optional `RenderArtifact` for bounds debug render
|
|
||||||
- `RunArtifact` for bounds output
|
|
||||||
- Seriatim adapter:
|
|
||||||
- `Trim` when bounds indicate trimming is required
|
|
||||||
|
|
||||||
## State and Manifest Behavior
|
- normalized transcript is required input.
|
||||||
- Reads final transcript from normalize manifest outputs when available; falls back to canonical path.
|
- bounds output exists only in enabled trim path.
|
||||||
- Uses run-local outputs/logs/reports/config/scratch paths when run layout is enabled.
|
- render-debug output is diagnostic and not a declared stage output.
|
||||||
- Materializes canonical final-trimmed transcript and session bounds when trim is enabled.
|
|
||||||
- Records bounds diagnostics, trim action, keep selector, and adapter metadata.
|
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
## Related Contracts And Tests
|
||||||
- Runner-level skip applies when already succeeded and not forced.
|
|
||||||
- Forced reruns can stale downstream succeeded stages.
|
|
||||||
- When `trim.enabled=false`, stage still succeeds by copying final to final-trimmed output.
|
|
||||||
|
|
||||||
## Failure Behavior
|
- [Scriptorium](../integrations/scriptorium.md) owns bounds generation and
|
||||||
- Fails on missing/invalid final transcript.
|
debug-render subprocess behavior.
|
||||||
- With trim enabled, fails on missing adapters/config, bounds generation/validation errors, invalid bounds JSON, invalid range/segment ids, trim adapter failures, or invalid final-trimmed output.
|
- [Seriatim](../integrations/seriatim.md) owns transcript trimming behavior.
|
||||||
|
- [Configuration](../config.md#pipeline) owns trim fields and defaults.
|
||||||
## Tests to Inspect Before Changing
|
- Implementation and tests: `internal/stage/trim.go`,
|
||||||
- `internal/stage/trim_test.go`
|
`internal/stage/trim_test.go`
|
||||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
|
||||||
- `internal/adapters/seriatim/subprocess_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
|
||||||
- Trim never falls back to polished transcript; final transcript is required input.
|
|
||||||
- `session_bounds` output exists only for enabled trim path.
|
|
||||||
- Render-debug artifacts are diagnostics and not declared stage outputs.
|
|
||||||
|
|||||||
@@ -1,75 +1,48 @@
|
|||||||
# Internal: Storage
|
# Internal: Storage
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Document Narratio's remote storage backend contracts and implementations under `internal/adapters/storage`.
|
|
||||||
|
|
||||||
## Inputs and outputs
|
Explain the object-store interface and S3 implementation used by Narratio.
|
||||||
Inputs:
|
Remote key layout and lifecycle belong in [Operations](../operations.md), while
|
||||||
- Resolved storage config (`pipeline.storage.*`).
|
operator-selected storage fields and credential mechanisms belong in
|
||||||
- Already-loaded environment variables for configured S3 credentials.
|
[Configuration](../config.md).
|
||||||
- Bucket-relative object keys and local file paths from app/stage orchestration.
|
|
||||||
|
|
||||||
Outputs:
|
## Primary Contract
|
||||||
- Listed/downloaded/uploaded object metadata (`ObjectInfo`).
|
|
||||||
- Existence checks and storage-layer errors.
|
|
||||||
|
|
||||||
## Boundaries
|
`storage.ObjectStore` interface:
|
||||||
Owns:
|
|
||||||
- Remote object-store interface and implementation details.
|
|
||||||
- S3 client wiring and API calls.
|
|
||||||
- Object key normalization and upload/download/list primitives.
|
|
||||||
|
|
||||||
Does not own:
|
- `List(ctx, prefix)`
|
||||||
- Session/run prefix semantics.
|
- `Download(ctx, key, localPath)`
|
||||||
- Archive commit order semantics.
|
- `Upload(ctx, localPath, key, opts)`
|
||||||
- Manifest updates.
|
- `Exists(ctx, key)`
|
||||||
- Filesystem secret loading from `pipeline.secrets.env_dir`.
|
|
||||||
|
|
||||||
## Config fields used
|
Key invariant:
|
||||||
- `pipeline.storage.backend`
|
- callers pass full bucket-relative keys;
|
||||||
- `pipeline.storage.s3.bucket`
|
- storage implementations do not infer campaign/session/run prefixes.
|
||||||
- `pipeline.storage.s3.region`
|
|
||||||
- `pipeline.storage.s3.endpoint`
|
|
||||||
- `pipeline.storage.s3.force_path_style`
|
|
||||||
- `pipeline.storage.s3.access_key_id_env`
|
|
||||||
- `pipeline.storage.s3.secret_access_key_env`
|
|
||||||
|
|
||||||
## External adapters used
|
## Composition
|
||||||
Storage package contracts:
|
|
||||||
- `ObjectStore` (active remote object-store boundary): `List`, `Download`, `Upload`, `Exists`.
|
|
||||||
- `Backend` (legacy compatibility boundary): currently implemented with `NoopBackend` only.
|
|
||||||
|
|
||||||
Implementations:
|
`NewObjectStoreFromConfig` constructs the S3-backed implementation from
|
||||||
- `S3Backend`: AWS SDK-backed `ObjectStore` implementation.
|
resolved configuration. The application loads configured filesystem secrets
|
||||||
- `FakeBackend`: deterministic test `ObjectStore` and compatibility backend.
|
before calling it. The storage adapter consumes already-resolved values; it does
|
||||||
- `NoopBackend`: deterministic no-op compatibility backend for wiring/tests.
|
not own discovery, defaults, or configuration validation.
|
||||||
|
|
||||||
## State and manifest behavior
|
## S3 Backend Behavior
|
||||||
- Storage implementations are stateless with respect to manifest/session lifecycle.
|
|
||||||
- Caller supplies fully-qualified bucket-relative keys.
|
|
||||||
- Storage layer does not infer campaign/session/run/root-prefix semantics.
|
|
||||||
- Caller controls publish ordering; storage layer executes individual operations in the order invoked.
|
|
||||||
|
|
||||||
## Skip and resume behavior
|
- normalizes object keys.
|
||||||
- No storage-level skip/resume behavior.
|
- `List` paginates and returns normalized `ObjectInfo`.
|
||||||
- Skip/resume decisions are made by stage/app logic before storage calls occur.
|
- `Download` writes local files with parent directory creation.
|
||||||
|
- `Upload` streams local file and returns remote metadata.
|
||||||
|
- `Exists` maps not-found responses to `false`.
|
||||||
|
|
||||||
## Failure behavior
|
## Invariants
|
||||||
- `NewObjectStoreFromConfig` fails when no remote backend is configured or required S3 config is missing.
|
|
||||||
- `S3Backend` constructor fails when required bucket is missing or AWS client setup fails.
|
|
||||||
- App command orchestration loads configured filesystem secrets before calling the object-store factory.
|
|
||||||
- CRUD operations return contextual errors (including not-found behavior via `Exists`).
|
|
||||||
- Key normalization is applied before operations (`\\` to `/`, leading slash trimmed).
|
|
||||||
- Remote session loading uses `List` to find the exact `session.yml` key and `Download` to materialize it to a local temp file.
|
|
||||||
|
|
||||||
## Tests to inspect before changing
|
- storage layer is stateless regarding manifest/stage progression.
|
||||||
- `internal/adapters/storage/factory_test.go`
|
- publish ordering semantics are owned by stage/app code, not storage adapters.
|
||||||
- `internal/adapters/storage/s3_backend_test.go`
|
|
||||||
- `internal/adapters/storage/fake_test.go`
|
|
||||||
- `internal/adapters/storage/keys_test.go`
|
|
||||||
- `internal/adapters/storage/archive.go` + consumers in stage tests (`prepare`, `publish`)
|
|
||||||
|
|
||||||
## Architectural invariants
|
## Implementation And Tests
|
||||||
- Callers pass full bucket-relative keys.
|
|
||||||
- Storage backends must not prepend or infer narratio prefixes.
|
- Contract and S3 adapter: `internal/adapters/storage`
|
||||||
- Remote transport details remain isolated to storage adapter implementations.
|
- Composition: `internal/app/object_store.go`
|
||||||
|
- Tests: `internal/adapters/storage/*_test.go`,
|
||||||
|
`internal/app/object_store_test.go`
|
||||||
|
|||||||
@@ -1,78 +1,74 @@
|
|||||||
# Workspace internals
|
# Internal: Workspace
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define the local durable and run-local workspace model used by stages, manifests, resume, and publish.
|
|
||||||
|
|
||||||
## Inputs and Outputs
|
Explain the helpers that construct local session and run paths, coordinate
|
||||||
Inputs:
|
single-writer access, and confine cleanup. The authoritative physical layout and
|
||||||
- `pipeline.workspace.root`
|
retention workflow belong in [Operations](../operations.md#local-state-layout).
|
||||||
- `session.campaign`
|
|
||||||
- `session.session_id`
|
|
||||||
- generated `run_id`
|
|
||||||
|
|
||||||
Outputs:
|
## Path Ownership
|
||||||
- Session manifest at `{workspace.root}/work/{campaign}/{session_id}/manifest.json`
|
|
||||||
- Run manifest at `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`
|
|
||||||
- Canonical durable session directories and run-local stage trees
|
|
||||||
|
|
||||||
## Boundaries
|
`internal/artifacts` owns canonical session, run, spool, cache, and
|
||||||
Owns:
|
previous-cache path construction. `SessionPathsFor` provides the session-scoped
|
||||||
- Session-level path layout (`inputs/`, `audio/`, `transcripts/`, `artifacts/`, `reports/`, `logs/`, `config/`, `current/`, `runs/`, `previous/`)
|
path model, and layout creation goes through `EnsureLayoutFor`. Callers should
|
||||||
- `previous/manifest.json` and `previous/artifacts/**` are reserved for previous-session cache state materialized by `prepare` or `restore`
|
consume those helpers instead of rebuilding relative paths.
|
||||||
- Run-local stage sandbox layout under `runs/{run_id}/{stage}/`
|
|
||||||
- Session lock acquisition/release (`.lock`)
|
|
||||||
|
|
||||||
Does not own:
|
`internal/pathsafe` and application cleanup helpers enforce confinement for
|
||||||
- Stage business logic
|
relative destinations and deletion targets.
|
||||||
- Remote publish semantics (documented in `stage-publish.md`)
|
|
||||||
- CLI argument parsing
|
|
||||||
|
|
||||||
## Config Fields Used
|
## Run-Local Stage Layout
|
||||||
- `pipeline.workspace.root`
|
|
||||||
- `pipeline.workspace.cleanup_after_publish`
|
|
||||||
- `pipeline.spool.root`
|
|
||||||
- `pipeline.spool.delete_audio_after_publish`
|
|
||||||
- `pipeline.cache.root`
|
|
||||||
- `pipeline.cache.s3_audio`
|
|
||||||
- `session.campaign`
|
|
||||||
- `session.session_id`
|
|
||||||
|
|
||||||
## External Adapters Used
|
`internal/stage/run_local.go` maps stage outputs and diagnostics into an
|
||||||
None directly in this subsystem. Stages may use object storage adapters and then write local outputs into this layout.
|
invocation-scoped layout. Successful outputs are validated and atomically
|
||||||
|
materialized into canonical session paths before stage success. Managed
|
||||||
|
previous-session cache paths remain session-durable and are never redirected
|
||||||
|
into run-local output space.
|
||||||
|
|
||||||
## State and Manifest Behavior
|
Extraction uses run-local receipt, stderr, and output-root helpers, then
|
||||||
- Session state is persisted in the session manifest (`manifest.Manifest`).
|
promotes the validated external bundle to the unique immutable Notarius bundle
|
||||||
- Invocation history is persisted per run in run manifests under `runs/{run_id}/manifest.json`.
|
path supplied by `internal/artifacts`. `internal/fileops.PromoteDirectory`
|
||||||
- During each run, stage outputs are often written run-local first (`runs/{run_id}/{stage}/outputs/...`) and then materialized to canonical session paths after stage success.
|
copies only regular files and directories to a same-filesystem temporary
|
||||||
- `manifest.Artifacts` entries record `ProducerRunID` for durable outputs.
|
sibling. Source traversal uses confined directory handles and identity checks
|
||||||
- For S3 audio sessions, `prepare` records work/cache paths, S3 provenance, and spool path when the invocation downloaded the object.
|
so replacing an inspected root, directory, or file is rejected rather than
|
||||||
- `previous/**` is reconstructed from configured previous-session requirements; restore uses the previous session's committed current publish state rather than treating current-session stored `previous/**` as authoritative.
|
followed. The completed tree is atomically renamed without replacing an
|
||||||
- Durable cache state under `pipeline.cache.root` is not workspace state and is preserved by default by `narratio clean`.
|
existing destination. Exact physical paths belong in
|
||||||
- `narratio clean <id>` removes the session work root and session spool root.
|
[Operations](../operations.md#extraction-workflow).
|
||||||
- `narratio clean --all` removes all local session work under `workspace.root/work` and spool children under `spool.root`.
|
|
||||||
- `narratio clean --clear-cache` is the explicit opt-in for deleting matching S3 audio cache entries.
|
|
||||||
|
|
||||||
## Skip and Resume Behavior
|
## Locking
|
||||||
- Skip/resume decisions are made in `internal/app` (`run_control.go`, `resume.go`) using stage status in the session manifest.
|
|
||||||
- `--force` reruns selected stages and marks downstream previously-succeeded stages as `stale`.
|
|
||||||
- Workspace layout is idempotent (`EnsureLayoutFor`) and reused across runs.
|
|
||||||
|
|
||||||
## Failure Behavior
|
`artifacts.LocalStore` enforces the single-writer session lock via `.lock`
|
||||||
- Failures preserve manifests and run-local files for inspection.
|
(`ErrLockConflict` on contention).
|
||||||
- Lock conflicts fail fast via `ErrLockConflict`.
|
|
||||||
- Cleanup can fail post-publish; failure is recorded in publish stage metadata and returned by the run.
|
|
||||||
|
|
||||||
## Tests to Inspect Before Changing
|
## Cleanup Semantics
|
||||||
- `internal/artifacts/local_test.go`
|
|
||||||
- `internal/stage/run_local_test.go`
|
|
||||||
- `internal/app/run_control_test.go`
|
|
||||||
- `internal/app/resume_run_stage_test.go`
|
|
||||||
- `internal/app/post_archive_cleanup_test.go`
|
|
||||||
|
|
||||||
## Architectural Invariants
|
Automatic post-publish cleanup:
|
||||||
- Session root is campaign-aware: `{workspace.root}/work/{campaign}/{session_id}`.
|
|
||||||
- Run roots are always nested: `runs/{run_id}` under the session root.
|
- only runs when publish actually executed and succeeded;
|
||||||
- Run-local output materialization must end in canonical session paths.
|
- requires `uploaded=true` and `current_pointer_written=true` metadata;
|
||||||
- `previous/**` is session-durable state and must not be treated as run-local output scratch state.
|
- consumes the resolved cleanup policy described in
|
||||||
- Automatic post-publish cleanup only targets run-scoped directories and must never delete configured root directories.
|
[Configuration](../config.md);
|
||||||
- Manual `clean` may delete session-scoped directories or the `workspace.root/work` directory, but it must preserve configured root directories and reject unsafe targets.
|
- refuses unsafe deletes (root delete, out-of-root delete, symlink paths).
|
||||||
|
|
||||||
|
Manual cleanup uses the same scoped-target checks. Invocation syntax and exact
|
||||||
|
deletion scope belong in [CLI](../cli.md#clean) and
|
||||||
|
[Operations](../operations.md#cleanup).
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- campaign-aware session root is mandatory.
|
||||||
|
- manifest-driven stage state is durable across runs.
|
||||||
|
- cleanup guardrails prevent destructive root/out-of-scope deletion.
|
||||||
|
|
||||||
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Path model and local store: `internal/artifacts/paths.go`,
|
||||||
|
`internal/artifacts/local.go`
|
||||||
|
- Run-local materialization: `internal/stage/run_local.go`
|
||||||
|
- Immutable bundle promotion: `internal/fileops/directory.go`
|
||||||
|
- Cleanup confinement: `internal/app/cleanup_targets.go`,
|
||||||
|
`internal/app/post_publish_cleanup.go`
|
||||||
|
- Tests: `internal/artifacts/paths_model_test.go`,
|
||||||
|
`internal/artifacts/local_test.go`, `internal/stage/run_local_test.go`,
|
||||||
|
`internal/fileops/directory_test.go`,
|
||||||
|
`internal/app/cleanup_targets_test.go`,
|
||||||
|
`internal/app/post_publish_cleanup_test.go`
|
||||||
|
|||||||
@@ -1,72 +1,224 @@
|
|||||||
# Operations
|
# Operations Guide
|
||||||
|
|
||||||
This guide covers the implemented operator lifecycle for Narratio.
|
Operator workflow for running, recovering, and publishing Narratio sessions.
|
||||||
|
|
||||||
For field-level settings, see [docs/config.md](./config.md). For syntax/flags, see [docs/cli.md](./cli.md).
|
For command syntax, see [docs/cli.md](./cli.md). For field-level config, see [docs/config.md](./config.md).
|
||||||
|
|
||||||
## Normal Workflow
|
## Campaign and Session Selection
|
||||||
|
|
||||||
1. Ensure `pipeline.yml`, `campaign.yml`, and `session.yml` are available.
|
Campaign selection priority:
|
||||||
2. Ensure session audio is available (local `audio_dir`/`audio_files` or S3 prefix).
|
|
||||||
3. Run:
|
- `--campaign-file`
|
||||||
|
- `--campaign`
|
||||||
|
- `pipeline.campaigns.default_campaign_id`
|
||||||
|
|
||||||
|
Session source priority:
|
||||||
|
|
||||||
|
- `--session`
|
||||||
|
- local default search paths
|
||||||
|
- remote session object (S3) when local session file is not found and storage is configured
|
||||||
|
|
||||||
|
## Session Initialization
|
||||||
|
|
||||||
|
Use `session init` to generate a concrete session file for local or remote use.
|
||||||
|
|
||||||
|
Local file:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session init 2026-04-04 --output ./session.yml --date 2026-04-04 --title "Session 12"
|
||||||
|
```
|
||||||
|
|
||||||
|
Remote session object:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session init 2026-04-04 --remote --force
|
||||||
|
```
|
||||||
|
|
||||||
|
If `campaign.yml` sets `session_template_file`, `session init` renders it. Template variables must resolve to concrete values.
|
||||||
|
|
||||||
|
Campaigns must provide stable input files for speakers, autocorrect, glossary, players, and party. Session files may override those paths for one session. The `prepare` stage materializes them under `inputs/`; configured Scriptorium artifacts can reference prepared `players`, `party`, and `glossary` files with `narratio.input.players`, `narratio.input.party`, and `narratio.input.glossary`.
|
||||||
|
|
||||||
|
## Standard Session Workflow
|
||||||
|
|
||||||
|
1. Select pipeline/campaign/session config.
|
||||||
|
2. Validate session readiness:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session validate 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
3. (Optional) inspect stage decisions:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session plan 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Run the pipeline:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio run 2026-04-04
|
narratio run 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
4. Inspect status:
|
5. Check state:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session status 2026-04-04
|
narratio session status 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Stage Execution and Continuation Behavior
|
||||||
|
|
||||||
|
Canonical stage order:
|
||||||
|
|
||||||
|
1. `prepare`
|
||||||
|
2. `transcribe`
|
||||||
|
3. `merge`
|
||||||
|
4. `polish`
|
||||||
|
5. `normalize`
|
||||||
|
6. `trim`
|
||||||
|
7. `extract`
|
||||||
|
8. `render`
|
||||||
|
9. `analyze`
|
||||||
|
10. `publish`
|
||||||
|
11. `notify`
|
||||||
|
|
||||||
|
Execution rules:
|
||||||
|
|
||||||
|
- succeeded stages are skipped unless `--force` is set;
|
||||||
|
- `run` continues interrupted or partially completed sessions by running non-succeeded stages;
|
||||||
|
- forcing an upstream stage marks succeeded downstream stages as `stale` before
|
||||||
|
the replacement runs; and
|
||||||
|
- an executed failure, changed self-skip, or success that replaces a different
|
||||||
|
effective upstream outcome also marks succeeded downstream stages stale. A
|
||||||
|
repeated self-skip with the same reason and no outputs is stable and does not
|
||||||
|
perpetually rerun downstream work.
|
||||||
|
|
||||||
|
Single-stage execution:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage normalize 2026-04-04 --force
|
||||||
|
```
|
||||||
|
|
||||||
|
## Artifact Selection
|
||||||
|
|
||||||
|
`--artifacts` can be used on `run`, `run-stage`, `analyze`, and `publish`.
|
||||||
|
|
||||||
|
Selection behavior:
|
||||||
|
|
||||||
|
- validates names against `pipeline.scriptorium.artifacts`;
|
||||||
|
- filters analyze execution to selected configured artifacts;
|
||||||
|
- filters publish rules for `narratio.artifact.<name>` sources only;
|
||||||
|
- does not suppress built-in transcript, bounds, or explicitly configured
|
||||||
|
`narratio.extraction.<name>` publish sources; and
|
||||||
|
- never partially selects Notarius lanes.
|
||||||
|
|
||||||
|
## Extraction Workflow
|
||||||
|
|
||||||
|
When Notarius is omitted or disabled, `extract` records an explicit skipped
|
||||||
|
outcome with reason `notarius_disabled` and no outputs. A later invocation
|
||||||
|
reconsiders the skipped stage, so enabling Notarius does not require force.
|
||||||
|
|
||||||
|
When Notarius extraction is enabled, the stage consumes the final trimmed JSON
|
||||||
|
and preserves the complete validated Notarius bundle at:
|
||||||
|
|
||||||
|
- `artifacts/notarius/{narratio_run_id}/`
|
||||||
|
|
||||||
|
The directory is immutable once promoted. Configured lanes become
|
||||||
|
`narratio.extraction.<name>` sources for Scriptorium and explicit publish rules;
|
||||||
|
the bundle and `index.json` are retained for audit and resume validation but
|
||||||
|
are not selectable or published implicitly.
|
||||||
|
|
||||||
|
Starting a replacement clears the previous extraction payload from the current
|
||||||
|
session-stage record. If that replacement fails or self-skips, the current
|
||||||
|
record does not fall back to the earlier outputs. The earlier run manifest and
|
||||||
|
immutable bundle remain available for inspection, but downstream resolution
|
||||||
|
requires a new current successful extraction record.
|
||||||
|
|
||||||
|
Atomic Notarius bundle promotion is supported on Linux, macOS, and Windows.
|
||||||
|
On other operating systems, extraction fails before copying the bundle into a
|
||||||
|
temporary promotion tree because Narratio has no verified atomic no-replace
|
||||||
|
directory primitive there. This is an extraction limitation, not a broader
|
||||||
|
platform-support guarantee for every Narratio workflow.
|
||||||
|
|
||||||
|
Run-local diagnostics are:
|
||||||
|
|
||||||
|
- `runs/{run_id}/extract/notarius.receipt.json`
|
||||||
|
- `runs/{run_id}/extract/notarius.stderr.log`
|
||||||
|
- `runs/{run_id}/extract/notarius-output/` before durable promotion
|
||||||
|
|
||||||
|
The run-record upload excludes the complete
|
||||||
|
`extract/notarius-output/**` subtree. The receipt and stderr files remain
|
||||||
|
eligible run-record diagnostics. The durable bundle is never scanned for
|
||||||
|
implicit publication; only lanes named by explicit `pipeline.publish.outputs`
|
||||||
|
rules are uploaded.
|
||||||
|
|
||||||
|
To intentionally replace the current extraction result, run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage extract 2026-04-04 --force
|
||||||
|
```
|
||||||
|
|
||||||
|
Narratio automatically reruns extraction when its recorded invocation contract
|
||||||
|
or durable output validation changes. It cannot fingerprint configuration
|
||||||
|
files, profiles, prompts, modules, or references loaded transitively by
|
||||||
|
Notarius. Force extraction after changing any of those inputs, even when the
|
||||||
|
top-level Narratio and Notarius config paths remain the same. A forced extract
|
||||||
|
marks successful downstream stages stale. Ordinary extraction failures or
|
||||||
|
outcome changes also stale affected downstream stages, while an identical
|
||||||
|
repeated `notarius_disabled` self-skip does not repeatedly invalidate them.
|
||||||
|
|
||||||
## Publish Workflow
|
## Publish Workflow
|
||||||
|
|
||||||
Publish is the stage that commits remote current state.
|
Run publish only:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio publish 2026-04-04
|
narratio publish 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
Equivalent command:
|
Equivalent:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio run-stage publish 2026-04-04 --force
|
narratio run-stage publish 2026-04-04 --force
|
||||||
```
|
```
|
||||||
|
|
||||||
Publish uploads:
|
Publish commit model:
|
||||||
|
|
||||||
- run history files under `{session_prefix}/runs/{run_id}/` (excluding `audio/`)
|
- uploads eligible run files under `{session_prefix}/runs/{run_id}/`, excluding
|
||||||
- configured published outputs from `pipeline.publish.outputs`
|
audio and the run-local Notarius staging bundle;
|
||||||
- `previous/**` cache files when present
|
- uploads configured published outputs, including only explicitly configured
|
||||||
- `current/manifest.json`
|
extraction lanes;
|
||||||
- `current/run_id.txt` last
|
- uploads `previous/**` cache files when present;
|
||||||
|
- writes `current/manifest.json`;
|
||||||
|
- writes `current/run_id.txt` last.
|
||||||
|
|
||||||
`current/run_id.txt` is the remote commit marker.
|
`current/run_id.txt` is the remote current-state commit marker.
|
||||||
|
|
||||||
## Published Outputs and Locks
|
## Publish Locks
|
||||||
|
|
||||||
Published output behavior:
|
Lock sources:
|
||||||
|
|
||||||
- outputs are source-based rules in `pipeline.publish.outputs`.
|
- static locks in `pipeline.publish.locks`
|
||||||
- required missing unlocked sources fail publish.
|
- mutable remote locks in `{session_prefix}/locks.yml`
|
||||||
- optional missing unlocked sources are skipped.
|
|
||||||
- selected artifacts (`--artifacts`) only filter configured `narratio.artifact.<key>` output rules.
|
|
||||||
- built-in transcript and bounds output rules are not filtered by `--artifacts`.
|
|
||||||
|
|
||||||
Lock behavior:
|
Effective lock rules:
|
||||||
|
|
||||||
- static locks: `pipeline.publish.locks`.
|
- static and remote locks are merged;
|
||||||
- mutable locks: `{session_prefix}/locks.yml`.
|
- static locks win on source collisions;
|
||||||
- effective lock set is static + mutable; static wins on duplicate sources.
|
- locked outputs are intentional skips;
|
||||||
- locked outputs are intentional skips and do not fail publish.
|
- lock add/remove commands mutate only remote lock state.
|
||||||
- lock commands mutate only remote mutable locks.
|
|
||||||
|
Examples:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session locks 2026-04-04
|
||||||
|
narratio session locks add 2026-04-04 narratio.artifact.session_recap --reason "manual edits" --force
|
||||||
|
narratio session locks remove 2026-04-04 narratio.artifact.session_recap
|
||||||
|
```
|
||||||
|
|
||||||
## Restore Workflow
|
## Restore Workflow
|
||||||
|
|
||||||
Use restore when local durable session state is missing/stale and committed remote current state is authoritative.
|
Use restore when local durable session state is missing or stale and remote committed current state is authoritative.
|
||||||
|
|
||||||
Preview:
|
Dry run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session restore 2026-04-04 --dry-run
|
narratio session restore 2026-04-04 --dry-run
|
||||||
@@ -83,21 +235,22 @@ Default restore scope:
|
|||||||
- `manifest.json`
|
- `manifest.json`
|
||||||
- `transcripts/**`
|
- `transcripts/**`
|
||||||
- `artifacts/**`
|
- `artifacts/**`
|
||||||
- `previous/**` when required by configured previous-session artifact inputs
|
- `previous/**` when needed by configured previous-session artifact inputs
|
||||||
|
|
||||||
Optional:
|
Optional:
|
||||||
|
|
||||||
- add `--include-audio` to restore `audio/**`.
|
- `--include-audio` to include `audio/**`
|
||||||
|
- `--force` to overwrite local conflicts
|
||||||
|
|
||||||
Restore reads committed current state only (`current/run_id.txt`, `current/manifest.json`).
|
Restore writes an execution report at `reports/restore-latest.json`.
|
||||||
|
|
||||||
## Workspace and State Layout
|
## Local State Layout
|
||||||
|
|
||||||
Session root:
|
Session root:
|
||||||
|
|
||||||
- `{workspace.root}/work/{campaign}/{session_id}/`
|
- `{workspace.root}/work/{campaign}/{session_id}`
|
||||||
|
|
||||||
Durable session state:
|
Durable session paths:
|
||||||
|
|
||||||
- `manifest.json`
|
- `manifest.json`
|
||||||
- `inputs/**`
|
- `inputs/**`
|
||||||
@@ -110,61 +263,60 @@ Durable session state:
|
|||||||
- `config/**`
|
- `config/**`
|
||||||
- `runs/**`
|
- `runs/**`
|
||||||
|
|
||||||
Run-local stage layout:
|
Validated Notarius bundles live below `artifacts/notarius/{run_id}/`; receipt,
|
||||||
|
stderr, and pre-promotion output remain in the producing run's `extract`
|
||||||
|
directory as described in [Extraction Workflow](#extraction-workflow).
|
||||||
|
|
||||||
- `runs/{run_id}/{stage}/outputs|logs|reports|config|scratch`
|
Run-local layout:
|
||||||
|
|
||||||
Stages typically write run-local outputs first, then materialize canonical session outputs on success.
|
- `runs/{run_id}/{stage}/outputs`
|
||||||
|
- `runs/{run_id}/{stage}/logs`
|
||||||
|
- `runs/{run_id}/{stage}/reports`
|
||||||
|
- `runs/{run_id}/{stage}/config`
|
||||||
|
- `runs/{run_id}/{stage}/scratch`
|
||||||
|
|
||||||
## Resume and Force Rules
|
Spool layout (runtime/transient):
|
||||||
|
|
||||||
- `run` and `run-stage` skip succeeded stages unless `--force` is set.
|
- `{spool.root}/{campaign}/{session_id}/{run_id}/...`
|
||||||
- `resume` starts at the first non-succeeded stage.
|
- restore audio spool under `{spool.root}/{campaign}/{session_id}/restore/audio`
|
||||||
- force-rerunning an upstream succeeded stage marks downstream succeeded stages as `stale`.
|
|
||||||
- `--force` does not bypass publish locks.
|
Cache layout (durable S3 audio cache):
|
||||||
|
|
||||||
|
- `{cache.root}/s3/{bucket}/...`
|
||||||
|
|
||||||
## Cleanup
|
## Cleanup
|
||||||
|
|
||||||
Automatic post-publish cleanup is considered only when publish executes successfully and commits current state.
|
Session-scoped cleanup:
|
||||||
|
|
||||||
Config toggles:
|
|
||||||
|
|
||||||
- `pipeline.spool.delete_audio_after_publish=true`
|
|
||||||
- `pipeline.workspace.cleanup_after_publish=true`
|
|
||||||
|
|
||||||
Manual cleanup:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio clean 2026-04-04
|
narratio clean 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
Global cleanup:
|
||||||
|
|
||||||
|
```bash
|
||||||
narratio clean --all
|
narratio clean --all
|
||||||
```
|
```
|
||||||
|
|
||||||
Cache is preserved by default. Use `--clear-cache` to remove matching S3 audio cache entries.
|
Dry-run and cache variants:
|
||||||
|
|
||||||
## Failure and Recovery
|
|
||||||
|
|
||||||
After stage failure, Narratio keeps manifests and run-local files for inspection.
|
|
||||||
|
|
||||||
Standard recovery flow:
|
|
||||||
|
|
||||||
1. inspect status:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session status 2026-04-04
|
narratio clean 2026-04-04 --dry-run --clear-cache
|
||||||
|
narratio clean --all --dry-run --clear-cache
|
||||||
```
|
```
|
||||||
|
|
||||||
2. if needed, inspect restore plan:
|
Rules:
|
||||||
|
|
||||||
```bash
|
- `clean` deletes work/spool session state;
|
||||||
narratio session restore 2026-04-04 --dry-run
|
- cache is preserved unless `--clear-cache` is set;
|
||||||
```
|
- automatic post-publish cleanup is gated by successful publish commit plus:
|
||||||
|
- `pipeline.spool.delete_audio_after_publish=true`
|
||||||
3. fix root cause.
|
- `pipeline.workspace.cleanup_after_publish=true`
|
||||||
4. continue with `resume`, or rerun a stage with `--force` then `resume`.
|
|
||||||
|
|
||||||
## Operational Caveats
|
## Operational Caveats
|
||||||
|
|
||||||
- local and S3 audio modes are mutually exclusive.
|
- Local and S3 audio modes are mutually exclusive.
|
||||||
- publish requires prerequisite stages through `analyze` to be `succeeded`.
|
- Publish requires prerequisite stages through `render` and `analyze` to be succeeded.
|
||||||
- restore requires configured object storage and committed current state.
|
- Markdown publish defaults require render outputs (`transcripts/final.md` and `transcripts/final.trimmed.md`).
|
||||||
- `session status` and `session artifacts --remote` both report remote published-output availability when storage is configured.
|
- Restore requires configured object storage and committed remote current state.
|
||||||
|
- Storage-backed commands load filesystem secrets before object-store initialization.
|
||||||
|
|||||||
241
docs/policy/architecture.md
Normal file
241
docs/policy/architecture.md
Normal file
@@ -0,0 +1,241 @@
|
|||||||
|
# Architecture
|
||||||
|
|
||||||
|
This document defines Narratio's intended high-level architecture and the
|
||||||
|
invariants that changes must preserve. Implemented component details belong in
|
||||||
|
the [Internal Overview](../internal/overview.md) and its linked documents.
|
||||||
|
Significant architectural decision history belongs under `docs/adr/` when such
|
||||||
|
records exist.
|
||||||
|
|
||||||
|
## System Shape
|
||||||
|
|
||||||
|
Narratio is a small Go application that turns D&D session audio into polished
|
||||||
|
transcripts and generated session artifacts. It is an explicit, stage-driven
|
||||||
|
orchestrator, not a general workflow engine.
|
||||||
|
|
||||||
|
Narratio coordinates specialized external systems rather than reimplementing
|
||||||
|
their domains:
|
||||||
|
|
||||||
|
- WhisperX performs transcription;
|
||||||
|
- Seriatim performs deterministic transcript processing and rendering;
|
||||||
|
- Audita performs transcript correction and polishing;
|
||||||
|
- Notarius extracts validated structured artifact bundles; and
|
||||||
|
- Scriptorium executes prompts and produces configured artifacts.
|
||||||
|
|
||||||
|
Narratio owns orchestration, configuration resolution, session and run state,
|
||||||
|
artifact and path modeling, manifest persistence, stage sequencing, resume,
|
||||||
|
restore, cleanup gates, and publish semantics. External contracts are defined
|
||||||
|
in the [integration documentation](../integrations/).
|
||||||
|
|
||||||
|
The pipeline has one canonical ordered stage set. Configuration may enable,
|
||||||
|
disable, or parameterize supported behavior, but it must not turn that sequence
|
||||||
|
into an arbitrary DAG or hide orchestration in generic workflow abstractions.
|
||||||
|
The implemented stage inventory belongs in the
|
||||||
|
[Internal Overview](../internal/overview.md).
|
||||||
|
|
||||||
|
Narratio is contract-first without being abstraction-heavy. Interfaces and
|
||||||
|
extension points should protect demonstrated boundaries. New abstraction is not
|
||||||
|
itself an architectural goal.
|
||||||
|
|
||||||
|
## Ownership And Dependency Direction
|
||||||
|
|
||||||
|
The application boundary owns command dispatch, configuration selection,
|
||||||
|
production composition, session locking, and top-level lifecycle. It may depend
|
||||||
|
on concrete implementations to assemble a run.
|
||||||
|
|
||||||
|
Stage orchestration expresses intent in Narratio-level data and interfaces.
|
||||||
|
Stages may depend on configuration, manifest, artifact, path, and adapter
|
||||||
|
contracts, but they must not depend on transport-specific request types,
|
||||||
|
subprocess argument construction, cloud SDK types, or downstream tool internals.
|
||||||
|
|
||||||
|
Adapters translate between Narratio contracts and external systems. They own
|
||||||
|
HTTP, subprocess, notification, and object-storage mechanics, including command
|
||||||
|
construction, transport behavior, provider response handling, and external
|
||||||
|
error adaptation. External dependency types must remain inside the adapter that
|
||||||
|
owns them unless that dependency is the adapter's explicit public contract.
|
||||||
|
WhisperX HTTP behavior, Seriatim, Audita, Notarius, and Scriptorium command
|
||||||
|
construction, notification transport, and object-storage SDK details remain
|
||||||
|
behind these boundaries.
|
||||||
|
|
||||||
|
State and path services must not infer stage policy. Storage implementations
|
||||||
|
receive explicit bucket-relative keys and do not infer campaign, session, run,
|
||||||
|
or root-prefix semantics. Manifest persistence records transitions but does not
|
||||||
|
choose orchestration policy. Artifact resolution identifies and validates
|
||||||
|
artifacts but does not execute producers.
|
||||||
|
|
||||||
|
Dependencies should remain narrow and point toward Narratio-owned contracts.
|
||||||
|
Prefer the Go standard library. Add an external dependency only when it provides
|
||||||
|
a clear correctness, security, interoperability, or complexity benefit, and
|
||||||
|
confine it to the boundary that needs it.
|
||||||
|
|
||||||
|
## Stage Boundaries
|
||||||
|
|
||||||
|
Each stage has one explicit responsibility and declares:
|
||||||
|
|
||||||
|
- required input state;
|
||||||
|
- produced output state;
|
||||||
|
- configuration it consumes;
|
||||||
|
- external adapters it uses;
|
||||||
|
- manifest references and metadata it reads or writes;
|
||||||
|
- skip, force, invalidation, and resume behavior; and
|
||||||
|
- failure behavior.
|
||||||
|
|
||||||
|
Stages write and validate run-local results before materializing canonical
|
||||||
|
outputs where that distinction applies. A stage is complete only after its
|
||||||
|
required outputs have been written, validated, and recorded in durable manifest
|
||||||
|
state. Later stages depend on recorded success and artifact resolution, not
|
||||||
|
merely on incidental files existing on disk.
|
||||||
|
|
||||||
|
A failed or interrupted stage must not be presented as successful. Failure
|
||||||
|
should preserve enough local state and diagnostics for inspection, recovery,
|
||||||
|
and resume. Forcing an upstream stage invalidates succeeded downstream work
|
||||||
|
according to the canonical stage order.
|
||||||
|
|
||||||
|
A stage may explicitly self-skip with a stable reason and no outputs. That
|
||||||
|
outcome is persisted, clears older outputs owned by the stage, and is
|
||||||
|
reconsidered on a later invocation. A stage may also validate whether an
|
||||||
|
otherwise successful recorded result is still resumable; an obsolete result
|
||||||
|
is staled and rerun, while an unsafe condition that prevents a sound decision
|
||||||
|
stops execution.
|
||||||
|
|
||||||
|
Shared behavior should live behind a narrow service or helper with one clear
|
||||||
|
owner. Stages must not reach across boundaries or reproduce adapter, manifest,
|
||||||
|
artifact, or path policy ad hoc.
|
||||||
|
|
||||||
|
## Manifest, Resume, And Restore
|
||||||
|
|
||||||
|
The session manifest is the durable ledger for progress across invocations. It
|
||||||
|
records session and run identity, stage state, input and output references,
|
||||||
|
diagnostic references, checksums or provenance where useful, and non-secret
|
||||||
|
adapter and publish metadata.
|
||||||
|
|
||||||
|
Resume and skip decisions are manifest-driven. Filesystem state may be
|
||||||
|
inspected and validated, but file presence alone does not replace recorded
|
||||||
|
stage state. Invocation-scoped run records provide an audit of one execution;
|
||||||
|
they do not replace the session manifest as progress authority.
|
||||||
|
|
||||||
|
Restore treats committed remote current state as its authority. It must plan
|
||||||
|
deterministically, confine remote-to-local paths, protect local conflicts, and
|
||||||
|
install the validated session manifest after other restored durable files. The
|
||||||
|
physical workflow and recovery procedures belong in
|
||||||
|
[Operations](../operations.md).
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Configuration is strict, explicit, centralized, and operator-oriented.
|
||||||
|
|
||||||
|
- YAML decoding rejects unknown fields.
|
||||||
|
- Defaults are centralized and testable.
|
||||||
|
- Empty configured values do not silently replace meaningful defaults.
|
||||||
|
- Validation rejects invalid composition before stage execution where
|
||||||
|
practical.
|
||||||
|
- Session templating remains narrow and deterministic rather than becoming a
|
||||||
|
general configuration language.
|
||||||
|
- Secret values are supplied indirectly and are not persisted in ordinary
|
||||||
|
configuration.
|
||||||
|
|
||||||
|
Narratio must not become a second configuration system for downstream tools.
|
||||||
|
External systems own their runtime defaults wherever practical; Narratio passes
|
||||||
|
the paths required by its stage contracts and explicit operator overrides. The
|
||||||
|
field-level contract and credential-supply mechanisms belong in
|
||||||
|
[Configuration](../config.md).
|
||||||
|
|
||||||
|
## Artifacts, Paths, And Storage
|
||||||
|
|
||||||
|
Artifact identities and local and remote paths are application contracts.
|
||||||
|
Canonical helpers own workspace, spool, cache, session, run, input, transcript,
|
||||||
|
artifact, log, report, configuration, and publish-current paths. Callers must
|
||||||
|
not reconstruct canonical paths through scattered string concatenation.
|
||||||
|
|
||||||
|
Artifact resolution is deterministic and manifest-aware. Producers materialize
|
||||||
|
canonical outputs before reporting success, and consumers resolve declared
|
||||||
|
artifact identities rather than infer files from unrelated directory contents.
|
||||||
|
External artifact bundles become current only through validated immutable
|
||||||
|
promotion and manifest records; directory presence alone never establishes
|
||||||
|
availability.
|
||||||
|
|
||||||
|
Writes, moves, replacements, and deletions must use narrow, explicit,
|
||||||
|
root-confined destinations. Symlinks, traversal, broad roots, and ambiguous
|
||||||
|
relative destinations must not expand the scope of an operation. Cleanup is
|
||||||
|
permitted only through explicit operator action or configured post-publish
|
||||||
|
gates, and it must preserve durable cache unless cache removal is explicitly
|
||||||
|
requested.
|
||||||
|
|
||||||
|
Physical layout, retention, and operational lifecycle belong in
|
||||||
|
[Operations](../operations.md). Logical external formats and durable integration
|
||||||
|
contracts belong under [Integrations](../integrations/).
|
||||||
|
|
||||||
|
## Publish Commit Boundary
|
||||||
|
|
||||||
|
Publish has one explicit remote commit boundary. A remote run becomes current
|
||||||
|
only after Narratio has successfully uploaded the run record, required published
|
||||||
|
outputs, `current/manifest.json`, and finally `current/run_id.txt`.
|
||||||
|
|
||||||
|
`current/run_id.txt` is the commit marker and must be written last. Failed,
|
||||||
|
incomplete, skipped, or uncommitted publish attempts must not be presented as
|
||||||
|
current remote state. Publish locks remain authoritative and are not bypassed by
|
||||||
|
a forced run.
|
||||||
|
|
||||||
|
Automatic local cleanup is permitted only after a successful publish commit,
|
||||||
|
only when explicitly configured, and only through the path-safety guardrails.
|
||||||
|
|
||||||
|
## Security, Privacy, And Diagnostics
|
||||||
|
|
||||||
|
Narratio handles private campaign material. Transcripts, prompts, generated
|
||||||
|
artifacts, reports, logs, manifests, and diagnostic files are potentially
|
||||||
|
sensitive.
|
||||||
|
|
||||||
|
Raw secrets must not be stored in pipeline, campaign, or session YAML or written
|
||||||
|
to manifests, logs, generated configuration, reports, publish metadata,
|
||||||
|
documentation, or examples. Secrets enter through configured environment
|
||||||
|
variable names or secret-file references. Diagnostics should avoid transcript
|
||||||
|
and prompt content unless a deliberate, bounded inspection mechanism requires
|
||||||
|
it.
|
||||||
|
|
||||||
|
Logs, reports, generated invocation files, generated configuration, and render
|
||||||
|
debug files are diagnostics, not canonical pipeline products. They should be
|
||||||
|
durable and discoverable where configured, and manifest references must preserve
|
||||||
|
the distinction between diagnostics and artifacts.
|
||||||
|
|
||||||
|
Documentation security rules belong in the
|
||||||
|
[Documentation Policy](documentation.md). Credential supply belongs in
|
||||||
|
[Configuration](../config.md), while permissions, sensitive runtime-artifact
|
||||||
|
handling, and recovery belong in [Operations](../operations.md).
|
||||||
|
|
||||||
|
## Determinism And Testability
|
||||||
|
|
||||||
|
Narratio prefers deterministic behavior where practical, including stable local
|
||||||
|
and remote layouts, sorted operation order, predictable generated
|
||||||
|
configuration, repeatable command construction, deterministic artifact
|
||||||
|
resolution, and reproducible planning.
|
||||||
|
|
||||||
|
Run IDs and timestamps may be intentionally variable, but surrounding behavior
|
||||||
|
must remain controllable in tests. Core behavior should be testable without live
|
||||||
|
external services; expensive, nondeterministic, destructive, or external
|
||||||
|
boundaries should be replaceable with focused test doubles. General testing
|
||||||
|
philosophy and sufficiency rules belong in the [Testing Policy](testing.md).
|
||||||
|
|
||||||
|
## Documentation And Decision Records
|
||||||
|
|
||||||
|
Documentation follows the [Documentation Policy](documentation.md). Current
|
||||||
|
behavior belongs in its canonical user, operator, integration, architecture, or
|
||||||
|
internal owner. Proposed behavior and implementation status belong under
|
||||||
|
`docs/roadmap/`.
|
||||||
|
|
||||||
|
Significant architectural decisions may be recorded under `docs/adr/` using the
|
||||||
|
format and lifecycle defined by the documentation policy. ADR acceptance does
|
||||||
|
not establish that a decision has been implemented.
|
||||||
|
|
||||||
|
## Architectural Non-Goals
|
||||||
|
|
||||||
|
Narratio does not aim to provide:
|
||||||
|
|
||||||
|
- a generic DAG or workflow engine;
|
||||||
|
- a replacement configuration layer for WhisperX, Seriatim, Audita,
|
||||||
|
Scriptorium, or other downstream tools;
|
||||||
|
- a storage abstraction broader than the needs of this pipeline;
|
||||||
|
- stage logic coupled directly to cloud SDKs, transports, subprocess details,
|
||||||
|
or downstream implementation internals;
|
||||||
|
- raw-secret persistence;
|
||||||
|
- implicit cross-stage behavior that bypasses manifest and artifact contracts;
|
||||||
|
or
|
||||||
|
- a prompt-authoring system.
|
||||||
148
docs/policy/documentation.md
Normal file
148
docs/policy/documentation.md
Normal file
@@ -0,0 +1,148 @@
|
|||||||
|
# Documentation Policy
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This policy assigns each documentation topic to one canonical owner. Its goal is
|
||||||
|
to keep Narratio documentation accurate, concise, discoverable, and resistant
|
||||||
|
to drift for users, operators, developers, integrators, and LLM coding agents.
|
||||||
|
|
||||||
|
## Core Rules
|
||||||
|
|
||||||
|
### One Canonical Owner
|
||||||
|
|
||||||
|
Each authoritative fact belongs in one document. A non-owning document may give
|
||||||
|
a short, stable summary for orientation, but it must link to the canonical owner
|
||||||
|
instead of repeating volatile details.
|
||||||
|
|
||||||
|
Volatile details include commands, flags, configuration fields and defaults,
|
||||||
|
stage or integration keys, schemas, file names, paths, status codes, retry
|
||||||
|
behavior, and runtime guarantees. If readers could reasonably treat a statement
|
||||||
|
as a contract, maintain it only in the owning document.
|
||||||
|
|
||||||
|
### Current And Future Behavior
|
||||||
|
|
||||||
|
Outside `docs/roadmap/`, documentation describes implemented behavior only.
|
||||||
|
Partial features may be described only to their implemented boundary.
|
||||||
|
|
||||||
|
ADRs are the narrow exception: an ADR may record an accepted architectural
|
||||||
|
decision before implementation, but acceptance must not be presented as proof
|
||||||
|
that the behavior exists. The roadmap owns implementation status and sequencing
|
||||||
|
until the decision is implemented. Current architecture, user, operator,
|
||||||
|
integration, and internal documentation are updated when the behavior lands.
|
||||||
|
|
||||||
|
### Audience And Detail
|
||||||
|
|
||||||
|
Write for the document's stated audience and include only the detail needed for
|
||||||
|
its owned topic. User and operator docs should not expose implementation detail.
|
||||||
|
Developer docs should link to user-facing and external contracts rather than
|
||||||
|
restate them.
|
||||||
|
|
||||||
|
### Examples
|
||||||
|
|
||||||
|
Complete copyable files belong in `examples/`. Documentation may use the
|
||||||
|
smallest illustrative snippet needed to explain its owned topic, but should link
|
||||||
|
to maintained examples instead of embedding a second complete copy.
|
||||||
|
|
||||||
|
Examples must be valid, secret-free, and tested where practical. Commands and
|
||||||
|
configuration used in documentation should match the application.
|
||||||
|
|
||||||
|
### Security And Privacy
|
||||||
|
|
||||||
|
Documentation and examples must not contain real credentials, private keys,
|
||||||
|
private environment dumps, sensitive source material, or private infrastructure
|
||||||
|
details unless intentionally public. Document secret-handling mechanisms, not
|
||||||
|
secret values.
|
||||||
|
|
||||||
|
## Canonical Ownership
|
||||||
|
|
||||||
|
| Topic | Canonical owner | Owned content | Content owned elsewhere |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| Product orientation and minimal end-to-end quickstart | `README.md` | What Narratio is, why it is useful, one shortest successful invocation, and links onward. | Complete command reference, configuration reference, operational procedures, implementation detail. |
|
||||||
|
| Contributor entry point | `docs/development.md` | Task-oriented reading guide, minimal contributor orientation, baseline validation commands, and links to canonical docs. | Package inventory, architecture rules, subsystem behavior, detailed change recipes. |
|
||||||
|
| Current application architecture | `docs/policy/architecture.md` | System shape, normative ownership, dependency direction, architectural boundaries, invariants, safety properties, and non-goals. | Concrete package inventory, implementation mechanics, contributor procedures, decision history, future work. |
|
||||||
|
| Documentation organization | `docs/policy/documentation.md` | Documentation ownership, audience boundaries, maintenance rules, and ADR/document lifecycle. | Application architecture or product behavior. |
|
||||||
|
| Testing policy | `docs/policy/testing.md` | Test philosophy, risk-based sufficiency, test boundaries, doubles, coverage guidance, regression-test policy, and criteria for adding, rewriting, or deleting tests. | Subsystem behavior, application contracts, subsystem-specific test inventories, and implementation plans. |
|
||||||
|
| CLI contract | `docs/cli.md` | Commands, arguments, flags, invocation semantics, output conventions, and exit behavior. | End-to-end operating procedures, configuration field definitions, runtime filesystem layout, stage implementation details. |
|
||||||
|
| Configuration contract | `docs/config.md` | Discovery and precedence, file schemas, fields, defaults, environment overrides, validation rules, and user-selectable stage or integration settings. | Complete example files, CLI syntax, runtime state lifecycle, implementation details. |
|
||||||
|
| Operations | `docs/operations.md` | Runtime workflows, physical filesystem and remote-state layout, output and diagnostic handling, resume, cleanup, permissions, recovery, and operational limits. | CLI flag syntax, configuration field definitions, logical artifact schemas, implementation mechanics. |
|
||||||
|
| Troubleshooting | `docs/troubleshooting.md` | Symptom-driven diagnosis, likely causes, safe inspection steps and remedies, and links to relevant contracts. | CLI syntax, configuration definitions, operational procedures, integration contracts, implementation mechanics. |
|
||||||
|
| Public HTTP contract, if introduced | `docs/api.md` | Routes, authentication, media types, request and response schemas, status codes, pagination, caching, idempotency, rate limits, and HTTP retry semantics. | Client walkthroughs, upstream or downstream integration internals, implementation detail. |
|
||||||
|
| Consumer guidance, if a public package or API is introduced | `docs/consumers/` | Task-oriented use of the public interface, minimal client examples, and consumer responsibilities. | HTTP wire semantics, external protocol contracts, internal implementation detail. |
|
||||||
|
| External and durable integration contracts | `docs/integrations/` | External file formats and protocols, upstream and downstream contracts, logical artifact paths and schemas, media types, and compatibility behavior. | Physical runtime placement and lifecycle, internal transformations, CLI syntax, configuration defaults. |
|
||||||
|
| Implemented component inventory | `docs/internal/overview.md` | Current packages and components, their implemented responsibilities, and links to focused internal docs. | Normative architecture, contributor reading policy, external contracts. |
|
||||||
|
| Internal component behavior | Other files under `docs/internal/` | Implementation flow, internal collaborators and state transitions, package-local guarantees and failures, and relevant tests. | Global architecture invariants, configuration definitions and defaults, external schemas, operator procedures. |
|
||||||
|
| Architectural decision history | `docs/adr/` | Significant decisions, context, alternatives, rationale, consequences, and supersession history. | Current behavior reference, implementation status, task sequencing. |
|
||||||
|
| Future work and implementation status | `docs/roadmap/` | Proposed, accepted, deferred, or rejected work; implementation status; sequencing; and task breakdowns. | Implemented behavior reference and architectural decision rationale. |
|
||||||
|
| Complete copyable artifacts | `examples/` | Maintained configuration, inputs, and other files intended to be copied or run. | Field-by-field reference, command reference, prose explanation. |
|
||||||
|
|
||||||
|
Documents that do not exist are required only when the corresponding interface
|
||||||
|
or responsibility exists. Do not create placeholder API, consumer, integration,
|
||||||
|
or operations documents for behavior the application does not have.
|
||||||
|
|
||||||
|
## Boundary Rules
|
||||||
|
|
||||||
|
### Orientation
|
||||||
|
|
||||||
|
The README owns product orientation. The developer guide routes contributors.
|
||||||
|
Architecture owns normative structure. Internal overview owns the current
|
||||||
|
concrete component map. These documents may link to one another but should not
|
||||||
|
maintain parallel package or behavior descriptions.
|
||||||
|
|
||||||
|
### Commands, Configuration, Operations, And Troubleshooting
|
||||||
|
|
||||||
|
CLI documentation answers how to invoke the application. Configuration
|
||||||
|
documentation answers what settings mean. Operations answers what happens to
|
||||||
|
runtime state and how to operate or recover the application. Troubleshooting
|
||||||
|
starts from observable symptoms and links readers to the owning command,
|
||||||
|
configuration, operational, or integration contract. When a workflow crosses
|
||||||
|
these topics, choose the document that owns the task and link to the other
|
||||||
|
contracts.
|
||||||
|
|
||||||
|
### Contracts And Implementation
|
||||||
|
|
||||||
|
Integration and API documents define externally observable shapes and
|
||||||
|
semantics. Internal documents explain how Narratio implements or consumes those
|
||||||
|
contracts. Internal docs may name a field, file, or protocol to identify a
|
||||||
|
dependency, but must link to its canonical contract for the definition.
|
||||||
|
|
||||||
|
### Security Topics
|
||||||
|
|
||||||
|
This policy owns what documentation and examples may contain. Architecture owns
|
||||||
|
application security invariants. Configuration owns credential-supply
|
||||||
|
mechanisms. Operations owns permissions and handling of sensitive runtime
|
||||||
|
artifacts. Troubleshooting owns safe diagnostic and remediation guidance.
|
||||||
|
Internal docs own implementation mechanisms only.
|
||||||
|
|
||||||
|
## Architecture Decision Records
|
||||||
|
|
||||||
|
Use sequentially numbered ADR filenames such as
|
||||||
|
`0001-record-architecture-decisions.md`. Follow the lightweight Nygard format:
|
||||||
|
|
||||||
|
1. title;
|
||||||
|
2. status;
|
||||||
|
3. date;
|
||||||
|
4. context;
|
||||||
|
5. decision;
|
||||||
|
6. alternatives considered;
|
||||||
|
7. consequences.
|
||||||
|
|
||||||
|
Treat the decision content of an accepted ADR as immutable. When a decision
|
||||||
|
changes, create a new ADR and update the earlier ADR's status to superseded.
|
||||||
|
Rejected architectural alternatives belong in the ADR; rejected product ideas
|
||||||
|
belong in the roadmap.
|
||||||
|
|
||||||
|
## Maintenance
|
||||||
|
|
||||||
|
When behavior changes, update its canonical owner in the same change. If
|
||||||
|
ownership moves, remove the old definition and replace it with a link where
|
||||||
|
navigation remains useful.
|
||||||
|
|
||||||
|
Before completing documentation work:
|
||||||
|
|
||||||
|
- verify affected behavior and examples;
|
||||||
|
- check commands, flags, fields, defaults, schemas, and paths against their
|
||||||
|
implementation;
|
||||||
|
- keep unimplemented behavior in the roadmap, subject to the ADR exception;
|
||||||
|
- remove stale references and validate links;
|
||||||
|
- confirm that non-owning documents summarize and link rather than redefine;
|
||||||
|
- confirm that no secrets or sensitive private data were added.
|
||||||
296
docs/policy/testing.md
Normal file
296
docs/policy/testing.md
Normal file
@@ -0,0 +1,296 @@
|
|||||||
|
# Testing Policy
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Our tests exist to make **incorrect changes expensive and correct changes cheap**.
|
||||||
|
|
||||||
|
We do not optimize for test count, line coverage, exhaustive isolation, or the fewest possible tests. We optimize for sufficient confidence in important behavior while imposing as little unnecessary friction as possible on future development.
|
||||||
|
|
||||||
|
## Every test has a cost
|
||||||
|
|
||||||
|
Testing is not an unqualified good. Every test imposes both an immediate cost and a continuing lifetime cost.
|
||||||
|
|
||||||
|
A test must be:
|
||||||
|
|
||||||
|
- written and reviewed;
|
||||||
|
- understood by future maintainers and coding agents;
|
||||||
|
- executed in local and CI workflows;
|
||||||
|
- diagnosed when it fails;
|
||||||
|
- updated when legitimate behavior changes;
|
||||||
|
- maintained as fixtures, APIs, and dependencies evolve; and
|
||||||
|
- removed or rewritten when it becomes redundant, brittle, misleading, or obsolete.
|
||||||
|
|
||||||
|
Tests also create cognitive and architectural friction. They can constrain refactoring, duplicate policy, slow feedback loops, add noise to failures, and cause harmless implementation changes to require unrelated edits across the suite.
|
||||||
|
|
||||||
|
A test is warranted only when the confidence it provides justifies these costs.
|
||||||
|
|
||||||
|
Apply this cost-benefit analysis at two levels:
|
||||||
|
|
||||||
|
1. **Per test:** What realistic defect does this test detect, how consequential would that defect be, and is that protection worth the test's lifetime cost?
|
||||||
|
2. **Across the suite:** Does this collection provide materially more confidence than a smaller, simpler suite would?
|
||||||
|
|
||||||
|
The preferred test suite is a **lean suite that provides sufficient confidence in the risks that matter, without redundant or low-value tests**. We seek sufficient confidence with the least unnecessary testing friction, not the fewest possible tests.
|
||||||
|
|
||||||
|
Some friction is intentional. Tests should make dangerous changes—such as breaking compatibility, corrupting data, violating security boundaries, or reintroducing subtle bugs—require deliberate review. They should not make ordinary internal changes needlessly expensive.
|
||||||
|
|
||||||
|
The cost of a test is not a reason to omit testing by default. Do not cite maintenance cost abstractly. When omitting a plausible test, be able to state why the protected failure is low-risk, already covered, obvious, reversible, or cheaper to detect elsewhere. For consequential, subtle, or difficult-to-observe behavior, the presumption should favor testing.
|
||||||
|
|
||||||
|
## Default testing style
|
||||||
|
|
||||||
|
Use a **classical/Detroit-style** approach:
|
||||||
|
|
||||||
|
- Test observable behavior, resulting state, contracts, and invariants.
|
||||||
|
- Use real internal collaborators when they are fast and deterministic.
|
||||||
|
- Use fakes, stubs, or mocks primarily at expensive, nondeterministic, destructive, or external boundaries.
|
||||||
|
- Prefer package-level behavioral tests over tests coupled to private helpers or internal call sequences.
|
||||||
|
- Treat exact collaborator interactions as testable behavior only when the interaction itself is a requirement.
|
||||||
|
|
||||||
|
Examples of appropriate seams include clocks, randomness, subprocesses, remote APIs, object storage, email, and paid LLM calls.
|
||||||
|
|
||||||
|
## Test execution requirements
|
||||||
|
|
||||||
|
Tests in the default suite must be deterministic, offline, and independent of real credentials. They must not invoke paid APIs or depend on mutable external services. Tests that require live infrastructure must be explicitly opt-in and clearly separated from the default suite.
|
||||||
|
|
||||||
|
Control clocks, randomness, environment variables, and other process-global or machine-specific state when they affect behavior. Tests should be safe to run repeatedly and alongside other tests without depending on execution order or state left by an earlier test.
|
||||||
|
|
||||||
|
## What deserves tests
|
||||||
|
|
||||||
|
Prioritize tests for:
|
||||||
|
|
||||||
|
1. Public and package-level contracts.
|
||||||
|
2. Domain rules and important invariants.
|
||||||
|
3. Boundary conditions and malformed input.
|
||||||
|
4. Failure handling, cancellation, retries, recovery, and partial success.
|
||||||
|
5. Serialization, schemas, compatibility, and round trips.
|
||||||
|
6. Previously observed or plausible regressions.
|
||||||
|
7. Representative integration and end-to-end workflows.
|
||||||
|
|
||||||
|
A package-level contract is behavior relied upon by another package or major collaborator, not every observable detail of a package implementation.
|
||||||
|
|
||||||
|
For behavior involving **data integrity, destructive operations, compatibility, security, concurrency, idempotency, or recovery**, presume that durable tests are required unless the behavior is already credibly protected at another layer.
|
||||||
|
|
||||||
|
Do not add tests merely because a function, branch, or line exists. Do not add a test when the same meaningful risk is already adequately protected elsewhere.
|
||||||
|
|
||||||
|
## Choose the right test boundary
|
||||||
|
|
||||||
|
Test through the narrowest stable boundary that expresses the behavior clearly.
|
||||||
|
|
||||||
|
This is often the package API, but it may instead be:
|
||||||
|
|
||||||
|
- a smaller pure function when dense domain logic is most clearly isolated there;
|
||||||
|
- a package-level operation when several internal collaborators jointly produce the behavior; or
|
||||||
|
- a larger integration boundary when correctness emerges from interaction with a real dependency.
|
||||||
|
|
||||||
|
Do not force all behavior through oversized end-to-end tests. Do not test every private helper merely because it exists. Choose the boundary that gives durable confidence with the least incidental coupling.
|
||||||
|
|
||||||
|
## Test behavior, not implementation
|
||||||
|
|
||||||
|
A test should protect a decision, contract, or invariant—not memorialize the current implementation.
|
||||||
|
|
||||||
|
Before adding or retaining a test, ask:
|
||||||
|
|
||||||
|
> What realistic defect would this test catch?
|
||||||
|
|
||||||
|
A test is suspect when its main purpose is to detect that someone:
|
||||||
|
|
||||||
|
- changed an internal constant;
|
||||||
|
- renamed or split a private helper;
|
||||||
|
- reordered equivalent internal operations;
|
||||||
|
- changed incidental formatting;
|
||||||
|
- replaced one correct algorithm with another; or
|
||||||
|
- refactored internal object structure without changing behavior.
|
||||||
|
|
||||||
|
Refactoring should normally require no test edits unless the refactored structure is itself part of the contract.
|
||||||
|
|
||||||
|
A test can be factually correct and still have negative value. Accurately describing current behavior is not enough; the protected behavior must be important enough to justify the future friction.
|
||||||
|
|
||||||
|
## Expected effects of different changes
|
||||||
|
|
||||||
|
Use the following expectations when evaluating test failures and test maintenance:
|
||||||
|
|
||||||
|
| Change | Expected effect on tests |
|
||||||
|
|---|---|
|
||||||
|
| Internal refactor that preserves behavior | Existing tests should normally remain unchanged and continue to pass. |
|
||||||
|
| Change to an internal default with no contractual significance | Behavioral tests should normally remain unchanged; tests should derive expectations from configuration or relationships rather than duplicate the old value. |
|
||||||
|
| Intentional change to public behavior, policy, schema, or compatibility guarantees | The relevant tests should be reviewed and changed deliberately. |
|
||||||
|
| Accidental violation of a contract or invariant | Tests should fail; fix the production code rather than rewriting the tests to accept the defect. |
|
||||||
|
|
||||||
|
A test failing is not the same as a test needing to be edited. Many tests may correctly fail because of one production defect. The maintenance smell is a correct internal change that requires unrelated expectation updates throughout the suite.
|
||||||
|
|
||||||
|
## Separate mechanism from policy
|
||||||
|
|
||||||
|
Configurable thresholds and defaults must not be duplicated throughout the test suite.
|
||||||
|
|
||||||
|
For example, do not encode an internal concurrency limit indirectly:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Production policy:
|
||||||
|
const maxConcurrency = 4
|
||||||
|
|
||||||
|
// Brittle test:
|
||||||
|
err := startProcesses(5)
|
||||||
|
require.Error(t, err)
|
||||||
|
```
|
||||||
|
|
||||||
|
Instead, test the mechanism relationally:
|
||||||
|
|
||||||
|
```go
|
||||||
|
const limit = 2
|
||||||
|
runner := NewRunner(limit)
|
||||||
|
|
||||||
|
require.NoError(t, runner.Start(limit))
|
||||||
|
require.ErrorIs(t, runner.Start(limit+1), ErrTooMuchConcurrency)
|
||||||
|
```
|
||||||
|
|
||||||
|
The test should prove:
|
||||||
|
|
||||||
|
- the configured limit is accepted; and
|
||||||
|
- one beyond the configured limit is rejected.
|
||||||
|
|
||||||
|
The production default should be tested exactly only when its literal value is itself a public, operational, safety, protocol, or compatibility requirement.
|
||||||
|
|
||||||
|
Apply the same rule to limits, timeouts, capacities, retry counts, and ranges: test relationships and behavior, not duplicated literals.
|
||||||
|
|
||||||
|
For concurrency limits, test both kinds of behavior when relevant:
|
||||||
|
|
||||||
|
1. **Configuration enforcement:** invalid or excessive requested values are handled correctly.
|
||||||
|
2. **Runtime enforcement:** observed peak concurrency never exceeds the configured limit.
|
||||||
|
|
||||||
|
Use a test-controlled limit and measure the behavior relative to that limit. Do not merely assert today's default value.
|
||||||
|
|
||||||
|
## Avoid semantic duplication across layers
|
||||||
|
|
||||||
|
Each behavior should have a clear test owner.
|
||||||
|
|
||||||
|
- Parser tests own parsing cases.
|
||||||
|
- Validator tests own validation rules.
|
||||||
|
- Domain tests own transformations and invariants.
|
||||||
|
- Adapter tests own external integration behavior.
|
||||||
|
- Orchestrator tests own coordination and failure propagation.
|
||||||
|
- CLI tests own argument and configuration mapping.
|
||||||
|
- End-to-end tests prove that representative assembled workflows work.
|
||||||
|
|
||||||
|
Higher-level tests should not repeat every lower-level case. A single intentional policy change should not require unrelated edits across many test files.
|
||||||
|
|
||||||
|
Tests that are individually reasonable may still be collectively redundant. Evaluate the marginal value of each additional test in light of the protection already provided by the rest of the suite.
|
||||||
|
|
||||||
|
## Use test doubles deliberately
|
||||||
|
|
||||||
|
Choose the least elaborate test double that provides the required control or observation.
|
||||||
|
|
||||||
|
As a default:
|
||||||
|
|
||||||
|
1. Prefer real collaborators when they are fast and deterministic.
|
||||||
|
2. Use small in-memory fakes when realistic stateful behavior is helpful.
|
||||||
|
3. Use stubs when a dependency only needs to provide controlled responses.
|
||||||
|
4. Use mocks when the interaction itself is contractual.
|
||||||
|
|
||||||
|
Mocks are appropriate when the contract includes facts such as:
|
||||||
|
|
||||||
|
- a notification is sent exactly once;
|
||||||
|
- a transaction is committed only after successful writes;
|
||||||
|
- cancellation reaches a subprocess;
|
||||||
|
- an expensive API is called no more than once; or
|
||||||
|
- a security audit event is emitted.
|
||||||
|
|
||||||
|
Do not use mocks merely to isolate every object or reproduce the implementation's call graph.
|
||||||
|
|
||||||
|
## Go-specific guidance
|
||||||
|
|
||||||
|
Use:
|
||||||
|
|
||||||
|
- table-driven tests for meaningful behavioral categories and boundaries;
|
||||||
|
- `t.TempDir()` for real filesystem behavior;
|
||||||
|
- `httptest.Server` for realistic HTTP interactions;
|
||||||
|
- fuzz tests for parsers, normalization, path handling, and broad input spaces;
|
||||||
|
- golden files only when the complete output is intentionally stable;
|
||||||
|
- integration tests where correctness depends on component interaction; and
|
||||||
|
- a small number of representative end-to-end tests.
|
||||||
|
|
||||||
|
Avoid exact error-string assertions unless the wording is itself contractual. Prefer `errors.Is`, `errors.As`, typed errors, or structured error fields.
|
||||||
|
|
||||||
|
At CLI boundaries, prefer exit classifications, structured output, and the smallest stable semantic fragment needed to identify the error. Do not snapshot complete diagnostic wording unless it is contractual.
|
||||||
|
|
||||||
|
Golden-file updates must require an explicit local flag. CI must not update golden files automatically, and reviewers must inspect the semantic diff before accepting an update.
|
||||||
|
|
||||||
|
Keep tests readable and direct. Test helpers and fixture frameworks must earn their own maintenance cost; do not build elaborate test infrastructure for small or isolated needs.
|
||||||
|
|
||||||
|
## Coverage
|
||||||
|
|
||||||
|
Coverage is a diagnostic, not a target.
|
||||||
|
|
||||||
|
Use it to find untested critical branches and unexpectedly weak packages. Do not write low-value tests solely to increase a percentage, and do not infer test quality from coverage alone.
|
||||||
|
|
||||||
|
Pure domain logic will often warrant higher coverage than CLI wiring or external adapters. Uneven coverage is acceptable when it reflects risk.
|
||||||
|
|
||||||
|
Increasing coverage is valuable only when the newly covered behavior protects a meaningful risk at an acceptable cost.
|
||||||
|
|
||||||
|
## Regression tests
|
||||||
|
|
||||||
|
A bug fix should normally include a regression test that fails before the fix and passes afterward.
|
||||||
|
|
||||||
|
Retain the test when the defect could realistically recur and its consequences justify the ongoing cost. Prefer the narrowest durable test of the violated contract or invariant; do not preserve accidental implementation details from the original bug.
|
||||||
|
|
||||||
|
Not every historical bug requires a permanent test. If the underlying design has made recurrence impossible, the test has become redundant, or a stronger invariant test now subsumes it, remove or consolidate it.
|
||||||
|
|
||||||
|
## Deleting or rewriting tests
|
||||||
|
|
||||||
|
Tests are maintained code, not permanent historical artifacts.
|
||||||
|
|
||||||
|
Delete or rewrite a test when its maintenance cost exceeds the confidence it provides.
|
||||||
|
|
||||||
|
Strong candidates include tests that:
|
||||||
|
|
||||||
|
- require updates after harmless internal changes;
|
||||||
|
- directly assert private constants without protecting a real contract;
|
||||||
|
- duplicate the same policy across several layers;
|
||||||
|
- verify mock choreography rather than outcomes;
|
||||||
|
- snapshot large amounts of incidental output;
|
||||||
|
- test trivial private helpers already exercised through stable package behavior;
|
||||||
|
- protect risks already covered more effectively elsewhere;
|
||||||
|
- are flaky, misleading, obsolete, or disproportionately expensive to diagnose; or
|
||||||
|
- no longer correspond to a plausible failure mode.
|
||||||
|
|
||||||
|
Several brittle tests may encode one genuine requirement. Replace them with one durable behavior-level or invariant test rather than preserving all of them.
|
||||||
|
|
||||||
|
Deleting a low-value test can improve the quality of the suite by reducing noise, maintenance burden, and friction around legitimate change.
|
||||||
|
|
||||||
|
## Reviewing a proposed test
|
||||||
|
|
||||||
|
Use the following questions when the value, boundary, or durability of a proposed test is not self-evident. Significant test additions should be reviewable against them, but written answers are not required for every routine test.
|
||||||
|
|
||||||
|
1. What realistic defect would it catch?
|
||||||
|
2. How likely is that defect?
|
||||||
|
3. How consequential would it be?
|
||||||
|
4. Is the behavior already protected elsewhere?
|
||||||
|
5. At which layer should this behavior be owned?
|
||||||
|
6. Does the test assert a durable contract or an incidental implementation detail?
|
||||||
|
7. Could the implementation be refactored without changing the behavior and without editing this test?
|
||||||
|
8. What should cause this test to fail?
|
||||||
|
9. What legitimate changes should not cause this test to fail?
|
||||||
|
10. What ongoing maintenance, execution, and diagnostic cost will the test impose?
|
||||||
|
11. Is there a smaller or more direct test that protects the same risk?
|
||||||
|
|
||||||
|
Do not add the test when its expected lifetime cost exceeds its expected protective value.
|
||||||
|
|
||||||
|
When deciding not to test plausible behavior, record or be able to explain why the risk is low, already protected, obvious, reversible, or cheaper to detect elsewhere.
|
||||||
|
|
||||||
|
## Definition of sufficient
|
||||||
|
|
||||||
|
A test suite is sufficient when:
|
||||||
|
|
||||||
|
- important contracts and invariants are protected;
|
||||||
|
- meaningful boundaries and failure modes are exercised;
|
||||||
|
- realistic and consequential regressions are credibly protected against silent recurrence;
|
||||||
|
- behavior involving data integrity, destructive operations, compatibility, security, concurrency, idempotency, and recovery is credibly protected;
|
||||||
|
- important external boundaries have realistic integration coverage;
|
||||||
|
- representative complete workflows are tested;
|
||||||
|
- failures provide useful signal rather than redundant noise;
|
||||||
|
- legitimate internal changes usually do not require test edits; and
|
||||||
|
- additional tests would mostly repeat existing protection or preserve inconsequential implementation details.
|
||||||
|
|
||||||
|
Sufficiency is a risk judgment, not a coverage percentage or test count. Reassess it as the application, its users, and the consequences of failure evolve.
|
||||||
|
|
||||||
|
The governing rule is:
|
||||||
|
|
||||||
|
> Test heavily where failure is consequential, subtle, or difficult to detect after the fact. Test lightly where failure is obvious, reversible, and inexpensive—and retain no test whose lifetime cost exceeds the confidence it provides.
|
||||||
@@ -1,231 +0,0 @@
|
|||||||
# Roadmap: Campaign Registry
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
## Problem
|
|
||||||
|
|
||||||
Narratio currently treats campaign configuration as one selected
|
|
||||||
`campaign.yml` file:
|
|
||||||
|
|
||||||
- command flags use `--campaign <path>`;
|
|
||||||
- default discovery searches fixed system file locations;
|
|
||||||
- `campaign.yml` uses `campaign:` as the identity field.
|
|
||||||
|
|
||||||
That model works for a single campaign, but it is awkward for installations
|
|
||||||
that manage multiple campaigns. Operators need to pass file paths or maintain a
|
|
||||||
single global campaign config, while the newer session-oriented CLI already
|
|
||||||
uses concise positional session IDs and remote session lookup.
|
|
||||||
|
|
||||||
The campaign selection model should become ID-based and pipeline-owned.
|
|
||||||
Pipeline config should describe where campaigns live, commands should select a
|
|
||||||
campaign by ID, and each campaign directory should contain its stable campaign
|
|
||||||
materials.
|
|
||||||
|
|
||||||
## Target Model
|
|
||||||
|
|
||||||
`pipeline.yml` owns the campaign registry:
|
|
||||||
|
|
||||||
campaigns:
|
|
||||||
root: /usr/local/share/narratio/campaigns
|
|
||||||
default_campaign_id: dilfs
|
|
||||||
|
|
||||||
Campaign files live at the conventional path:
|
|
||||||
|
|
||||||
{campaigns.root}/{campaign_id}/campaign.yml
|
|
||||||
|
|
||||||
The first implementation should use only the conventional path. Recursive
|
|
||||||
discovery of every `campaign.yml` under `campaigns.root` is deferred to a
|
|
||||||
future stage.
|
|
||||||
|
|
||||||
Each campaign file uses `campaign_id` as the canonical identity field:
|
|
||||||
|
|
||||||
campaign_id: dilfs
|
|
||||||
session_template_file: ./session.template.yml
|
|
||||||
inputs:
|
|
||||||
speakers_file: ./speakers.yml
|
|
||||||
autocorrect_file: ./autocorrect.yml
|
|
||||||
glossary_file: ./glossary.yml
|
|
||||||
|
|
||||||
Campaign-relative files continue to resolve relative to the selected
|
|
||||||
`campaign.yml`, including stable input files and `session_template_file`.
|
|
||||||
|
|
||||||
The public CLI changes from path-based campaign selection to ID-based campaign
|
|
||||||
selection:
|
|
||||||
|
|
||||||
- `--campaign <id>` selects a campaign ID.
|
|
||||||
- `--campaign-file <path>` explicitly loads one campaign file for
|
|
||||||
development, tests, and unusual local workflows.
|
|
||||||
- `--campaign` and `--campaign-file` are mutually exclusive.
|
|
||||||
|
|
||||||
If neither `--campaign` nor `--campaign-file` is passed, Narratio uses
|
|
||||||
`pipeline.campaigns.default_campaign_id`. If no campaign can be selected,
|
|
||||||
commands fail clearly before session loading or stage execution.
|
|
||||||
|
|
||||||
Resolved campaign ID remains the campaign segment used for:
|
|
||||||
|
|
||||||
- workspace paths;
|
|
||||||
- spool paths;
|
|
||||||
- S3 session prefixes;
|
|
||||||
- remote `session.yml` lookup;
|
|
||||||
- archive locks and promoted output keys;
|
|
||||||
- session/campaign mismatch validation;
|
|
||||||
- status, plan, restore, and helper output.
|
|
||||||
|
|
||||||
## Compatibility Policy
|
|
||||||
|
|
||||||
This is a breaking public/config contract change.
|
|
||||||
|
|
||||||
After the cutover:
|
|
||||||
|
|
||||||
- `--campaign` no longer accepts a filesystem path;
|
|
||||||
- default fixed campaign file discovery is removed;
|
|
||||||
- `campaign:` is no longer accepted in `campaign.yml`;
|
|
||||||
- `campaign_id:` is required.
|
|
||||||
|
|
||||||
Keep `--campaign-file` as the only explicit file override. Do not retain hidden
|
|
||||||
aliases for the old `--campaign <path>` behavior.
|
|
||||||
|
|
||||||
## Implementation Stages
|
|
||||||
|
|
||||||
### Stage 1: Add Campaign Registry Selection
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Add the registry model and switch command loading to resolve campaigns through
|
|
||||||
pipeline config.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Add `pipeline.campaigns.root`.
|
|
||||||
- Add `pipeline.campaigns.default_campaign_id`.
|
|
||||||
- Add `campaign_id` to campaign config and make it the canonical identity.
|
|
||||||
- Resolve pipeline config first, then campaign selection.
|
|
||||||
- Use this selection order:
|
|
||||||
1. explicit `--campaign-file <path>`;
|
|
||||||
2. explicit `--campaign <id>`;
|
|
||||||
3. `pipeline.campaigns.default_campaign_id`;
|
|
||||||
4. fail clearly.
|
|
||||||
- For ID selection, load `{campaigns.root}/{campaign_id}/campaign.yml`.
|
|
||||||
- Validate that the loaded `campaign_id` matches the selected ID.
|
|
||||||
- Reject `--campaign` with `--campaign-file`.
|
|
||||||
- Preserve strict YAML decoding.
|
|
||||||
- Preserve campaign-relative stable input and session template resolution.
|
|
||||||
- Keep storage details behind the existing storage adapter and object-store
|
|
||||||
helper.
|
|
||||||
- Keep remote session lookup and archive key construction based on the
|
|
||||||
resolved campaign ID.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Commands can run with only a pipeline config and the pipeline default
|
|
||||||
campaign ID.
|
|
||||||
- Commands can select another campaign with `--campaign <id>`.
|
|
||||||
- Commands can load a specific file with `--campaign-file <path>`.
|
|
||||||
- Existing session loading, remote session fallback, prepare materialization,
|
|
||||||
restore, archive, locks, clean, analyze, and publish behavior continue to use
|
|
||||||
the same resolved campaign identity.
|
|
||||||
- No generic config registry framework is introduced.
|
|
||||||
|
|
||||||
### Stage 2: Remove Old Single-File Campaign Behavior
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Remove the old public campaign file model after registry selection is in
|
|
||||||
place.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Remove fixed default campaign config discovery from command loading.
|
|
||||||
- Remove `DefaultCampaignConfigSearchPaths` and related path-only resolution if
|
|
||||||
no current tests or helpers still need them.
|
|
||||||
- Remove support for `campaign:` from `campaign.yml`.
|
|
||||||
- Update validation errors to refer to `campaign_id`.
|
|
||||||
- Update examples to use campaign directories and `campaign_id`.
|
|
||||||
- Update current-behavior docs to document:
|
|
||||||
- `pipeline.campaigns.root`;
|
|
||||||
- `pipeline.campaigns.default_campaign_id`;
|
|
||||||
- `campaign_id`;
|
|
||||||
- `--campaign <id>`;
|
|
||||||
- `--campaign-file <path>`.
|
|
||||||
- Update troubleshooting examples that currently pass `--campaign <path>`.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- `campaign.yml` files with `campaign:` fail strict decoding.
|
|
||||||
- `--campaign /path/to/campaign.yml` is treated as a campaign ID and fails
|
|
||||||
unless that ID exists under `campaigns.root`.
|
|
||||||
- `--campaign-file /path/to/campaign.yml` is the supported file override.
|
|
||||||
- User-facing docs no longer describe fixed campaign config discovery.
|
|
||||||
|
|
||||||
## Test Guidance
|
|
||||||
|
|
||||||
Focused tests:
|
|
||||||
|
|
||||||
- `go test ./internal/config -v`
|
|
||||||
- `go test ./internal/app -v`
|
|
||||||
- `go test ./internal/stage -run Prepare -v`
|
|
||||||
|
|
||||||
Full validation:
|
|
||||||
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Config tests to add or update:
|
|
||||||
|
|
||||||
- strict decode accepts `pipeline.campaigns.root`;
|
|
||||||
- strict decode accepts `pipeline.campaigns.default_campaign_id`;
|
|
||||||
- strict decode accepts `campaign_id`;
|
|
||||||
- selected campaign ID mismatch fails;
|
|
||||||
- missing campaign root fails when ID selection is needed;
|
|
||||||
- missing default campaign ID fails when no explicit campaign selector is
|
|
||||||
passed;
|
|
||||||
- old `campaign:` fails after Stage 2.
|
|
||||||
|
|
||||||
App tests to add or update:
|
|
||||||
|
|
||||||
- `--campaign <id>` resolves `{campaigns.root}/{id}/campaign.yml`;
|
|
||||||
- omitted `--campaign` uses `pipeline.campaigns.default_campaign_id`;
|
|
||||||
- `--campaign-file` loads an explicit campaign file;
|
|
||||||
- `--campaign` plus `--campaign-file` fails;
|
|
||||||
- remote session fallback uses the resolved campaign ID;
|
|
||||||
- `session init`, `run`, `run-stage`, `resume`, `analyze`, `publish`, `clean`,
|
|
||||||
and `session` subcommands all use the same campaign selection path;
|
|
||||||
- path-based `--campaign` examples and tests are removed after Stage 2.
|
|
||||||
|
|
||||||
## Documentation Guidance
|
|
||||||
|
|
||||||
Update current-behavior docs only after implementation lands:
|
|
||||||
|
|
||||||
- `docs/config.md`
|
|
||||||
- `docs/cli.md`
|
|
||||||
- `docs/operations.md`
|
|
||||||
- `docs/troubleshooting.md`
|
|
||||||
- relevant files under `docs/internal/`
|
|
||||||
- `examples/`
|
|
||||||
|
|
||||||
Planned campaign registry behavior belongs only in this roadmap until the code,
|
|
||||||
tests, examples, and current-behavior docs are updated.
|
|
||||||
|
|
||||||
## Architecture Guardrails
|
|
||||||
|
|
||||||
- Keep Narratio explicit and stage-driven.
|
|
||||||
- Do not introduce a generic configuration registry or workflow framework.
|
|
||||||
- Keep YAML decoding strict.
|
|
||||||
- Keep defaults centralized and testable.
|
|
||||||
- Keep campaign-relative path resolution centralized.
|
|
||||||
- Use centralized S3 and workspace path helpers.
|
|
||||||
- Keep storage details behind `storage.ObjectStore`.
|
|
||||||
- Keep secret-backed object-store construction in `internal/app`.
|
|
||||||
- Preserve manifest-driven resume and restore behavior.
|
|
||||||
- Do not store raw secrets in campaign configs, manifests, logs, generated
|
|
||||||
configs, or archive metadata.
|
|
||||||
|
|
||||||
## Assumptions
|
|
||||||
|
|
||||||
- The canonical pipeline schema is grouped under `campaigns`.
|
|
||||||
- The canonical campaign identity field is `campaign_id`.
|
|
||||||
- `--campaign` means campaign ID.
|
|
||||||
- `--campaign-file` is retained as an explicit override.
|
|
||||||
- Recursive discovery is planned but not part of the first implementation.
|
|
||||||
- Existing production configs can be migrated from `campaign:` to
|
|
||||||
`campaign_id:` and from `--campaign <path>` to `--campaign <id>` or
|
|
||||||
`--campaign-file <path>`.
|
|
||||||
@@ -1,159 +0,0 @@
|
|||||||
# Roadmap: Legacy Config Cleanup
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
## Problem
|
|
||||||
|
|
||||||
Narratio's current pipeline config schema still accepts fields that predate the current storage, artifact, and previous-session models:
|
|
||||||
|
|
||||||
- `pipeline.storage.bucket`
|
|
||||||
- `pipeline.storage.prefix`
|
|
||||||
- `pipeline.analyzer.*`
|
|
||||||
- `previous_session_artifact`
|
|
||||||
|
|
||||||
These names make the config reference harder to trust because they suggest supported behavior that operators should no longer use. The modern interface is:
|
|
||||||
|
|
||||||
- `pipeline.storage.s3.*` for remote storage.
|
|
||||||
- Scriptorium configured artifacts under `pipeline.scriptorium.artifacts`.
|
|
||||||
- Canonical artifact source IDs such as `narratio.artifact.<configured_artifact_key>`.
|
|
||||||
- Canonical previous-session artifact sources such as `narratio.previous_session.artifact.<configured_artifact_key>`.
|
|
||||||
|
|
||||||
Strict YAML decoding should reject removed legacy fields once this cleanup lands.
|
|
||||||
|
|
||||||
## Current State
|
|
||||||
|
|
||||||
`pipeline.storage.bucket` and `pipeline.storage.prefix` were inert compatibility fields and have been removed:
|
|
||||||
|
|
||||||
- They are no longer present on `config.StorageConfig`.
|
|
||||||
- Strict decoding rejects them.
|
|
||||||
- Runtime S3 behavior uses `pipeline.storage.s3.bucket` and `pipeline.storage.s3.root_prefix`.
|
|
||||||
- No current code reads the top-level storage bucket or prefix fields.
|
|
||||||
|
|
||||||
`pipeline.analyzer.*` was legacy code surface and has been removed:
|
|
||||||
|
|
||||||
- `config.PipelineConfig` no longer includes analyzer config.
|
|
||||||
- Strict decoding rejects `pipeline.analyzer`.
|
|
||||||
- `stage.Env` no longer exposes an analyzer runner, and `internal/adapters/analyzer` has been deleted.
|
|
||||||
- Modern analyze execution is Scriptorium-backed; the analyzer adapter is not used by current stage execution.
|
|
||||||
|
|
||||||
`previous_session_artifact` was a live legacy behavior and has been removed:
|
|
||||||
|
|
||||||
- Config validation rejects it as an unsupported Scriptorium input source.
|
|
||||||
- The analyze stage no longer has path-based previous-artifact resolution through `inputs.<name>.path`.
|
|
||||||
- Tests cover canonical previous-session sources and the rejection of the legacy source.
|
|
||||||
- The canonical replacement is `narratio.previous_session.artifact.<configured_artifact_key>`, resolved through the previous-session cache/catalog model.
|
|
||||||
|
|
||||||
## Target Model
|
|
||||||
|
|
||||||
The pipeline config schema should expose only current behavior:
|
|
||||||
|
|
||||||
- Remote storage is configured only through `pipeline.storage.s3.*`.
|
|
||||||
- Generated artifacts are configured only through `pipeline.scriptorium.artifacts`.
|
|
||||||
- Scriptorium artifact inputs use canonical source IDs.
|
|
||||||
- Previous-session artifact inputs use `narratio.previous_session.artifact.<configured_artifact_key>`.
|
|
||||||
- Unknown legacy fields fail strict YAML decoding.
|
|
||||||
|
|
||||||
No compatibility aliases should remain unless a future migration requirement explicitly reintroduces them.
|
|
||||||
|
|
||||||
## Cleanup Order
|
|
||||||
|
|
||||||
### Stage 1: Remove Inert Storage Compatibility Fields
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Remove `pipeline.storage.bucket` and `pipeline.storage.prefix`.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Delete `StorageConfig.Bucket` and `StorageConfig.Prefix`.
|
|
||||||
- Keep `StorageConfig.Backend` and `StorageConfig.S3`.
|
|
||||||
- Confirm all runtime storage paths continue to use `storage.s3.bucket` and `storage.s3.root_prefix`.
|
|
||||||
- Update examples and docs to remove top-level storage `bucket` and `prefix`.
|
|
||||||
- Add or update strict-decode tests proving `pipeline.storage.bucket` and `pipeline.storage.prefix` are rejected.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Existing S3 workflows still pass with `pipeline.storage.s3.bucket`.
|
|
||||||
- Pipeline configs containing top-level `storage.bucket` or `storage.prefix` fail to load.
|
|
||||||
- No docs or examples present those fields as available.
|
|
||||||
|
|
||||||
### Stage 2: Remove Legacy Analyzer Schema and Adapter Surface
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Remove the unused analyzer configuration and adapter contract.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Delete `PipelineConfig.Analyzer`.
|
|
||||||
- Delete `AnalyzerConfig` and `ArtifactSettings`.
|
|
||||||
- Remove analyzer timeout validation.
|
|
||||||
- Remove `stage.Env.Analyzer`.
|
|
||||||
- Delete `internal/adapters/analyzer` if no remaining code imports it.
|
|
||||||
- Remove `pipeline.analyzer.*` from tests, examples, and docs.
|
|
||||||
- Add or update strict-decode tests proving `pipeline.analyzer` is rejected.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Analyze behavior remains fully Scriptorium-backed.
|
|
||||||
- No runtime code imports `internal/adapters/analyzer`.
|
|
||||||
- Pipeline configs containing `pipeline.analyzer` fail to load.
|
|
||||||
- Contributor and internal adapter docs no longer list the analyzer adapter.
|
|
||||||
|
|
||||||
### Stage 3: Remove Path-Based Previous Session Artifact Source
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Remove `previous_session_artifact` and require canonical previous-session artifact sources.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Remove `previous_session_artifact` from supported Scriptorium input sources.
|
|
||||||
- Remove analyze-stage special-case handling that resolves `inputs.<name>.path` for previous artifacts.
|
|
||||||
- Keep canonical handling for `narratio.previous_session.artifact.<configured_artifact_key>`.
|
|
||||||
- Rewrite tests that use `previous_session_artifact` to use canonical sources and prepared previous-cache fixtures.
|
|
||||||
- Add validation tests proving `previous_session_artifact` is rejected.
|
|
||||||
- Update docs to remove the legacy path-based source and document only canonical previous-session sources.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- `pipeline.scriptorium.artifacts.*.inputs.*.source: previous_session_artifact` fails validation.
|
|
||||||
- Canonical previous-session sources continue to work for required and optional inputs.
|
|
||||||
- Prepare/restore previous-cache behavior remains unchanged.
|
|
||||||
- No docs or examples mention `previous_session_artifact` as supported.
|
|
||||||
|
|
||||||
## Test Guidance
|
|
||||||
|
|
||||||
Run focused tests after each stage:
|
|
||||||
|
|
||||||
- `go test ./internal/config -v`
|
|
||||||
- `go test ./internal/stage -run Analyze -v`
|
|
||||||
- `go test ./internal/app -v`
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
For Stage 1, focus on config load/strict-decode and S3 workflow regression tests.
|
|
||||||
|
|
||||||
For Stage 2, focus on compile-time removal, config strict-decode tests, and full app/stage tests to catch stale adapter references.
|
|
||||||
|
|
||||||
For Stage 3, focus on Scriptorium config validation, analyze-stage input resolution, previous-cache behavior, and restore/analyze workflows.
|
|
||||||
|
|
||||||
## Documentation Updates
|
|
||||||
|
|
||||||
Update current-behavior docs only after the corresponding code removal lands:
|
|
||||||
|
|
||||||
- `docs/config.md`
|
|
||||||
- `docs/cli.md`, only if command behavior text references removed fields.
|
|
||||||
- `docs/operations.md`, only if operator workflow text references removed fields.
|
|
||||||
- `docs/internal/stage-analyze.md`
|
|
||||||
- `docs/internal/adapters.md`
|
|
||||||
- `examples/pipeline.full.annotated.yml`
|
|
||||||
- `examples/pipeline.production.yml`
|
|
||||||
|
|
||||||
Do not preserve removed fields in examples as compatibility notes. The goal is to make strict config behavior and documentation line up.
|
|
||||||
|
|
||||||
## Assumptions
|
|
||||||
|
|
||||||
- This is a hard cleanup; no backward-compatible aliases are retained.
|
|
||||||
- Current production configs can be migrated to `storage.s3.*`, Scriptorium artifacts, and canonical previous-session sources before this lands.
|
|
||||||
- Removing the unused analyzer adapter does not block any active stage behavior.
|
|
||||||
- The cleanup should be implemented in the listed order so inert schema removal is separated from behavior removal.
|
|
||||||
@@ -1,255 +0,0 @@
|
|||||||
# Roadmap: Session-Oriented CLI Cleanup
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
## Problem
|
|
||||||
|
|
||||||
Narratio's public CLI has accumulated too many top-level commands. Several
|
|
||||||
commands are session-scoped operator helpers, but they currently appear as
|
|
||||||
independent top-level verbs:
|
|
||||||
|
|
||||||
- `plan`
|
|
||||||
- `status`
|
|
||||||
- `restore`
|
|
||||||
- `artifacts list`
|
|
||||||
- `locks`
|
|
||||||
- `session validate`
|
|
||||||
- `session init`
|
|
||||||
|
|
||||||
This makes the command surface harder to learn because the CLI does not clearly
|
|
||||||
separate primary workflow actions from session inspection, initialization,
|
|
||||||
restore, and helper operations.
|
|
||||||
|
|
||||||
## Target Model
|
|
||||||
|
|
||||||
Keep primary workflow commands at top level:
|
|
||||||
|
|
||||||
- `run`
|
|
||||||
- `run-stage`
|
|
||||||
- `resume`
|
|
||||||
- `analyze`
|
|
||||||
- `publish`
|
|
||||||
- `clean`
|
|
||||||
- `session`
|
|
||||||
|
|
||||||
Keep `clean` top-level because it can operate on one session or all local
|
|
||||||
sessions and is a workspace maintenance command, not only a session helper.
|
|
||||||
|
|
||||||
Move session-scoped helper commands under `narratio session` and use positional
|
|
||||||
session identifiers:
|
|
||||||
|
|
||||||
- `narratio session init <session_id> [--remote|--output <path>] [--flags]`
|
|
||||||
- `narratio session validate <session_id> [--flags]`
|
|
||||||
- `narratio session status <session_id> [--flags]`
|
|
||||||
- `narratio session plan <session_id> [--flags]`
|
|
||||||
- `narratio session restore <session_id> [--flags]`
|
|
||||||
- `narratio session artifacts <session_id> [--remote] [--flags]`
|
|
||||||
- `narratio session locks <session_id> [--flags]`
|
|
||||||
- `narratio session locks add <session_id> <source> [--reason <text>] [--force] [--flags]`
|
|
||||||
- `narratio session locks remove <session_id> <source> [--flags]`
|
|
||||||
|
|
||||||
Update top-level workflow commands to use positional session identifiers:
|
|
||||||
|
|
||||||
- `narratio run <session_id> [--flags]`
|
|
||||||
- `narratio resume <session_id> [--flags]`
|
|
||||||
- `narratio analyze <session_id> [--flags]`
|
|
||||||
- `narratio publish <session_id> [--flags]`
|
|
||||||
- `narratio run-stage <stage> <session_id> [--flags]`
|
|
||||||
|
|
||||||
The positional session ID replaces `--session-id` as the primary public
|
|
||||||
interface. Existing `--config`, `--campaign`, `--session`, and
|
|
||||||
`--previous-session-id` flags remain available where they are meaningful.
|
|
||||||
|
|
||||||
## Command Mapping
|
|
||||||
|
|
||||||
| Current command | Target command |
|
|
||||||
| --- | --- |
|
|
||||||
| `narratio run --session-id <id>` | `narratio run <id>` |
|
|
||||||
| `narratio resume --session-id <id>` | `narratio resume <id>` |
|
|
||||||
| `narratio analyze --session-id <id>` | `narratio analyze <id>` |
|
|
||||||
| `narratio publish --session-id <id>` | `narratio publish <id>` |
|
|
||||||
| `narratio run-stage [flags] <stage> --session-id <id>` | `narratio run-stage <stage> <id> [flags]` |
|
|
||||||
| `narratio plan --session-id <id>` | `narratio session plan <id>` |
|
|
||||||
| `narratio status --session-id <id>` | `narratio session status <id>` |
|
|
||||||
| `narratio restore --session-id <id>` | `narratio session restore <id>` |
|
|
||||||
| `narratio artifacts list --session-id <id>` | `narratio session artifacts <id>` |
|
|
||||||
| `narratio locks --session-id <id>` | `narratio session locks <id>` |
|
|
||||||
| `narratio locks add --session-id <id> <source>` | `narratio session locks add <id> <source>` |
|
|
||||||
| `narratio locks remove --session-id <id> <source>` | `narratio session locks remove <id> <source>` |
|
|
||||||
| `narratio session validate --session-id <id>` | `narratio session validate <id>` |
|
|
||||||
| `narratio session init --session-id <id>` | `narratio session init <id>` |
|
|
||||||
| `narratio clean --session-id <id>` | `narratio clean <id>` |
|
|
||||||
| `narratio clean --all` | unchanged |
|
|
||||||
|
|
||||||
`clean` remains top-level, but its session-scoped form should also move from
|
|
||||||
`--session-id` to positional `<session_id>` for consistency.
|
|
||||||
|
|
||||||
## Compatibility Policy
|
|
||||||
|
|
||||||
This is a hard public CLI cleanup after the migration step lands.
|
|
||||||
|
|
||||||
During Step 1, old forms may remain as compatibility aliases to keep the
|
|
||||||
implementation reviewable. During Step 2, remove the old forms from command
|
|
||||||
dispatch, tests, docs, and examples:
|
|
||||||
|
|
||||||
- remove top-level `plan`;
|
|
||||||
- remove top-level `status`;
|
|
||||||
- remove top-level `restore`;
|
|
||||||
- remove top-level `artifacts`;
|
|
||||||
- remove top-level `locks`;
|
|
||||||
- remove `--session-id` from the public command syntax for session-aware
|
|
||||||
commands.
|
|
||||||
|
|
||||||
Do not keep long-term deprecated aliases unless a later roadmap explicitly
|
|
||||||
chooses a compatibility window.
|
|
||||||
|
|
||||||
`status --manifest` does not fit the session-oriented command shape. Remove it
|
|
||||||
from the public CLI in this cleanup. If direct manifest inspection is needed
|
|
||||||
later, add a separate diagnostic command in a future roadmap rather than keeping
|
|
||||||
it as a special case in `session status`.
|
|
||||||
|
|
||||||
## Implementation Step 1: Add New Session-Oriented Interface
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Add the target command forms while preserving current behavior internally.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Add positional session ID parsing helpers in `internal/app`.
|
|
||||||
- Keep the existing `loadCommandConfig` behavior and populate
|
|
||||||
`config.SessionLoadOptions.SessionID` from the positional ID.
|
|
||||||
- Add or update command wrappers:
|
|
||||||
- `Run(ctx, args, out)` parses `run <session_id>`.
|
|
||||||
- `Resume(ctx, args, out)` parses `resume <session_id>`.
|
|
||||||
- `Analyze(ctx, args, out)` parses `analyze <session_id>`.
|
|
||||||
- `Publish(ctx, args, out)` parses `publish <session_id>`.
|
|
||||||
- `RunStage(ctx, args, out)` parses `run-stage <stage> <session_id>`.
|
|
||||||
- `Clean(ctx, args, out)` parses `clean <session_id>` and keeps
|
|
||||||
`clean --all`.
|
|
||||||
- Extend `Session(ctx, args, out)` dispatch to support:
|
|
||||||
- `init <session_id>`
|
|
||||||
- `validate <session_id>`
|
|
||||||
- `status <session_id>`
|
|
||||||
- `plan <session_id>`
|
|
||||||
- `restore <session_id>`
|
|
||||||
- `artifacts <session_id>`
|
|
||||||
- `locks <session_id>`
|
|
||||||
- `locks add <session_id> <source>`
|
|
||||||
- `locks remove <session_id> <source>`
|
|
||||||
- Keep storage access through the existing app-level object-store helper.
|
|
||||||
- Keep AWS SDK details behind storage adapters.
|
|
||||||
- Keep the runner, stages, manifest behavior, archive behavior, restore
|
|
||||||
planning, lock semantics, and artifact catalog behavior unchanged.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- New forms execute the same code paths and produce equivalent results.
|
|
||||||
- Positional session ID mismatch with concrete local or remote `session.yml`
|
|
||||||
fails through existing session identity checks.
|
|
||||||
- Remote session fallback still uses the positional session ID as the lookup
|
|
||||||
value.
|
|
||||||
- Current command tests cover the new forms before old forms are removed.
|
|
||||||
|
|
||||||
## Implementation Step 2: Remove Old Public Forms
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Remove compatibility aliases and make the session-oriented interface the only
|
|
||||||
documented and supported public CLI.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Remove top-level dispatch for:
|
|
||||||
- `plan`
|
|
||||||
- `status`
|
|
||||||
- `restore`
|
|
||||||
- `artifacts`
|
|
||||||
- `locks`
|
|
||||||
- Remove `--session-id` flags from public session-aware commands.
|
|
||||||
- Keep `--previous-session-id` as an expected previous-session identity flag.
|
|
||||||
- Keep explicit `--session <path>` for loading a local concrete session file,
|
|
||||||
but still require the positional session ID for commands that operate on a
|
|
||||||
session.
|
|
||||||
- Remove `status --manifest`.
|
|
||||||
- Update usage text and invalid-command errors.
|
|
||||||
- Update `docs/cli.md` and `docs/operations.md` to use only the new forms.
|
|
||||||
- Update any roadmap docs that mention old helper command names.
|
|
||||||
- Update tests to expect old top-level helper commands and `--session-id` forms
|
|
||||||
to fail.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Top-level command list is exactly:
|
|
||||||
- `run`
|
|
||||||
- `run-stage`
|
|
||||||
- `resume`
|
|
||||||
- `analyze`
|
|
||||||
- `publish`
|
|
||||||
- `clean`
|
|
||||||
- `session`
|
|
||||||
- All session-oriented commands use `narratio session <subcommand>
|
|
||||||
<session_id> [--flags]`, except nested lock mutation forms, which use
|
|
||||||
`narratio session locks add|remove <session_id> <source> [--flags]`.
|
|
||||||
- `clean <session_id>` and `clean --all` remain top-level.
|
|
||||||
- Current-behavior docs and tests no longer advertise `--session-id`.
|
|
||||||
|
|
||||||
## Test Guidance
|
|
||||||
|
|
||||||
Focused tests:
|
|
||||||
|
|
||||||
- `go test ./internal/app -run TestExecute -v`
|
|
||||||
- `go test ./internal/app -run 'Session|Status|Restore|Clean|Locks|Artifacts|Plan|RunStage|Analyze|Publish' -v`
|
|
||||||
- `go test ./internal/config -v`
|
|
||||||
|
|
||||||
Full validation:
|
|
||||||
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Test cases to add or update:
|
|
||||||
|
|
||||||
- `run <session_id>` loads local and remote sessions through the existing
|
|
||||||
config path.
|
|
||||||
- `resume <session_id>`, `analyze <session_id>`, and `publish <session_id>`
|
|
||||||
preserve current behavior.
|
|
||||||
- `run-stage <stage> <session_id>` preserves current run-stage output and
|
|
||||||
force/artifact-selection behavior.
|
|
||||||
- `session plan <session_id>` replaces top-level `plan`.
|
|
||||||
- `session status <session_id>` replaces top-level session status.
|
|
||||||
- `session validate <session_id>` replaces `session validate --session-id`.
|
|
||||||
- `session init <session_id>` writes the same local or remote concrete
|
|
||||||
`session.yml`.
|
|
||||||
- `session restore <session_id>` preserves restore planning/execution.
|
|
||||||
- `session artifacts <session_id> --remote` preserves promoted-output
|
|
||||||
availability reporting.
|
|
||||||
- `session locks <session_id>`, `session locks add <session_id> <source>`, and
|
|
||||||
`session locks remove <session_id> <source>` preserve static/remote lock
|
|
||||||
semantics.
|
|
||||||
- `clean <session_id>` preserves session cleanup behavior, while `clean --all`
|
|
||||||
remains unchanged.
|
|
||||||
- Old top-level helper commands fail after Step 2.
|
|
||||||
- `--session-id` fails after Step 2.
|
|
||||||
- `status --manifest` fails after Step 2.
|
|
||||||
|
|
||||||
## Documentation Guidance
|
|
||||||
|
|
||||||
Update only after implementation lands:
|
|
||||||
|
|
||||||
- `docs/cli.md`
|
|
||||||
- `docs/operations.md`
|
|
||||||
- any internal docs that list command names or examples
|
|
||||||
|
|
||||||
Keep planned behavior only in this roadmap until the command refactor is
|
|
||||||
implemented.
|
|
||||||
|
|
||||||
## Architecture Guardrails
|
|
||||||
|
|
||||||
- Keep Narratio explicit and stage-driven.
|
|
||||||
- Do not introduce a generic workflow or command framework abstraction.
|
|
||||||
- Reuse existing app command helpers where practical.
|
|
||||||
- Keep config loading strict and centralized.
|
|
||||||
- Keep storage details behind `storage.ObjectStore`.
|
|
||||||
- Keep secret-backed object-store construction in `internal/app`.
|
|
||||||
- Preserve manifest-driven resume and restore behavior.
|
|
||||||
- Treat command renaming as a public CLI contract change, not a runtime stage
|
|
||||||
behavior change.
|
|
||||||
@@ -1,287 +0,0 @@
|
|||||||
# Roadmap: Publish Contract
|
|
||||||
|
|
||||||
Status: Planned
|
|
||||||
|
|
||||||
## Problem
|
|
||||||
|
|
||||||
Narratio currently uses several terms for one operator-facing concept:
|
|
||||||
|
|
||||||
- `archive` is the stage that uploads run state and commits remote current
|
|
||||||
state.
|
|
||||||
- `publish` is the convenience command that force-runs the archive stage.
|
|
||||||
- `promote`, `promoted`, and `promote_artifacts` describe configured top-level
|
|
||||||
remote output writes.
|
|
||||||
|
|
||||||
This mixed vocabulary makes the public contract harder to explain. Operators
|
|
||||||
should not need to distinguish "archive the run", "publish the run", and
|
|
||||||
"promote artifacts" when these are all part of the same publish action.
|
|
||||||
|
|
||||||
The public model should use:
|
|
||||||
|
|
||||||
- `publish` for the stage, command, config section, and action;
|
|
||||||
- `published` for an expected remote output that exists at its top-level
|
|
||||||
current destination;
|
|
||||||
- `publish rules` for the configured source-to-destination output rules;
|
|
||||||
- `locked` for sources whose top-level published destination must not be
|
|
||||||
overwritten;
|
|
||||||
- `run history` for immutable per-run records under `runs/<run_id>/`.
|
|
||||||
|
|
||||||
## Target Model
|
|
||||||
|
|
||||||
The public stage is `publish`.
|
|
||||||
|
|
||||||
The convenience command:
|
|
||||||
|
|
||||||
narratio publish <session_id>
|
|
||||||
|
|
||||||
is equivalent to:
|
|
||||||
|
|
||||||
narratio run-stage publish <session_id> --force
|
|
||||||
|
|
||||||
Pipeline configuration uses `publish`:
|
|
||||||
|
|
||||||
publish:
|
|
||||||
enabled: true
|
|
||||||
upload_run: true
|
|
||||||
outputs:
|
|
||||||
- source: narratio.transcript.final_trimmed
|
|
||||||
- source: narratio.artifact.session_recap
|
|
||||||
locks:
|
|
||||||
- source: narratio.artifact.session_recap
|
|
||||||
reason: Final recap was manually edited.
|
|
||||||
|
|
||||||
Publish output rules are source-based. Each rule writes one artifact source to
|
|
||||||
a top-level remote destination. If `dest` is omitted, Narratio derives the
|
|
||||||
destination from the artifact registry or configured artifact output path.
|
|
||||||
|
|
||||||
The mutable remote lock store remains:
|
|
||||||
|
|
||||||
{session_prefix}/locks.yml
|
|
||||||
|
|
||||||
Remote availability output uses `published`:
|
|
||||||
|
|
||||||
Published:
|
|
||||||
- narratio.transcript.final_trimmed remote=published
|
|
||||||
- narratio.artifact.session_recap locked remote=published
|
|
||||||
|
|
||||||
The remote key layout is otherwise unchanged:
|
|
||||||
|
|
||||||
- immutable run history stays under `{session_prefix}/runs/{run_id}/`;
|
|
||||||
- current state stays under `{session_prefix}/current/manifest.json`;
|
|
||||||
- the final commit marker stays `{session_prefix}/current/run_id.txt`;
|
|
||||||
- `current/run_id.txt` is still written last.
|
|
||||||
|
|
||||||
## Compatibility Policy
|
|
||||||
|
|
||||||
This is a hard cutover.
|
|
||||||
|
|
||||||
After implementation:
|
|
||||||
|
|
||||||
- `pipeline.archive` is rejected by strict YAML decoding.
|
|
||||||
- `pipeline.archive.promote_artifacts` is rejected.
|
|
||||||
- `pipeline.workspace.cleanup_after_archive` is rejected.
|
|
||||||
- `pipeline.spool.delete_audio_after_archive` is rejected.
|
|
||||||
- `narratio run-stage archive <session_id>` is an unknown stage.
|
|
||||||
- manifests that record an `archive` stage are not migrated.
|
|
||||||
- old archive/promotion metadata keys are not read as compatibility fallbacks.
|
|
||||||
|
|
||||||
Existing remote objects are not moved or renamed. Remote layout remains stable;
|
|
||||||
the rename changes configuration, stage names, status output, metadata, helper
|
|
||||||
names, tests, examples, and documentation.
|
|
||||||
|
|
||||||
## Implementation Stages
|
|
||||||
|
|
||||||
### Stage 1: Public Schema and Stage Cutover
|
|
||||||
|
|
||||||
Status: Planned
|
|
||||||
|
|
||||||
Switch the public config and stage contract to publish terminology.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Replace `pipeline.archive` with `pipeline.publish`.
|
|
||||||
- Replace `archive.promote_artifacts` with `publish.outputs`.
|
|
||||||
- Keep output rule fields:
|
|
||||||
- `source`
|
|
||||||
- `dest`
|
|
||||||
- `required`
|
|
||||||
- Replace `pipeline.archive.locks` with `pipeline.publish.locks`.
|
|
||||||
- Rename post-publish cleanup fields:
|
|
||||||
- `pipeline.workspace.cleanup_after_publish`
|
|
||||||
- `pipeline.spool.delete_audio_after_publish`
|
|
||||||
- Rename the registered stage from `archive` to `publish`.
|
|
||||||
- Update stage order so `publish` runs after `analyze` and before `notify`.
|
|
||||||
- Update top-level `narratio publish` to target stage `publish`.
|
|
||||||
- Keep `run-stage --artifacts <names> publish` support.
|
|
||||||
- Reject `run-stage --artifacts <names>` for stages other than `analyze` and
|
|
||||||
`publish`.
|
|
||||||
- Preserve the remote commit ordering and storage adapter boundaries.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- `narratio run-stage publish <session_id>` executes the publish stage.
|
|
||||||
- `narratio publish <session_id>` force-runs the publish stage.
|
|
||||||
- `narratio run-stage archive <session_id>` fails clearly as an unknown stage.
|
|
||||||
- Old archive config fields fail strict decoding.
|
|
||||||
- New publish config fields load, default, and validate.
|
|
||||||
|
|
||||||
### Stage 2: Runtime Terminology and Metadata Cutover
|
|
||||||
|
|
||||||
Status: Planned
|
|
||||||
|
|
||||||
Rename implementation concepts and runtime output to publish terminology.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Rename archive/promotion config and runtime types conceptually to
|
|
||||||
publish/output terms.
|
|
||||||
- Rename the remote key helper intent from promoted artifact to published
|
|
||||||
output while keeping generated keys unchanged.
|
|
||||||
- Change helper output:
|
|
||||||
- `Promoted:` becomes `Published:`
|
|
||||||
- `remote=promoted` becomes `remote=published`
|
|
||||||
- lock output uses `published` / `not-published`
|
|
||||||
- Rename publish-stage metadata, including:
|
|
||||||
- `promoted_paths` to `published_paths`
|
|
||||||
- `promoted_files_uploaded` to `published_files_uploaded`
|
|
||||||
- `skipped_optional_promotions` to `skipped_optional_outputs`
|
|
||||||
- `skipped_unselected_promotions` to `skipped_unselected_outputs`
|
|
||||||
- `locked_promotion_count` to `locked_output_count`
|
|
||||||
- `locked_promotions` to `locked_outputs`
|
|
||||||
- Update previous-cache and restore logic to use the `publish` stage and
|
|
||||||
`published_paths` metadata only.
|
|
||||||
- Keep run-local stage output materialization separate from remote publish
|
|
||||||
terminology. If local helper names are confusing, rename them to
|
|
||||||
materialization-oriented names rather than publish names.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Status and artifact helper output use `Published:` and `remote=published`.
|
|
||||||
- Publish metadata contains only publish/output terminology.
|
|
||||||
- Previous-cache and restore behavior works with publish metadata and does not
|
|
||||||
depend on old archive metadata.
|
|
||||||
- Storage adapters still receive explicit keys and no AWS SDK details leak into
|
|
||||||
app or stage logic.
|
|
||||||
|
|
||||||
### Stage 3: Documentation, Examples, and Final Cleanup
|
|
||||||
|
|
||||||
Status: Planned
|
|
||||||
|
|
||||||
Update implemented-behavior docs and remove stale public terminology after the
|
|
||||||
runtime cutover lands.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Update current-behavior docs:
|
|
||||||
- `docs/config.md`
|
|
||||||
- `docs/cli.md`
|
|
||||||
- `docs/operations.md`
|
|
||||||
- `docs/troubleshooting.md`
|
|
||||||
- `docs/architecture.md`
|
|
||||||
- relevant files under `docs/internal/`
|
|
||||||
- Rename `docs/internal/stage-archive.md` to
|
|
||||||
`docs/internal/stage-publish.md`.
|
|
||||||
- Update internal documentation links and references.
|
|
||||||
- Update examples to use:
|
|
||||||
- `publish.outputs`
|
|
||||||
- `publish.locks`
|
|
||||||
- `cleanup_after_publish`
|
|
||||||
- `delete_audio_after_publish`
|
|
||||||
- Update tests and final searches so old terminology remains only in this
|
|
||||||
roadmap as historical context.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Maintained examples load and validate.
|
|
||||||
- Current-behavior docs describe only implemented publish terminology.
|
|
||||||
- Internal docs describe run history, published outputs, locks, and current
|
|
||||||
commit ordering clearly.
|
|
||||||
- Old user-facing archive/promote wording is removed except where discussing
|
|
||||||
historical behavior in this roadmap.
|
|
||||||
|
|
||||||
## Test Guidance
|
|
||||||
|
|
||||||
Focused tests:
|
|
||||||
|
|
||||||
- `go test ./internal/config -v`
|
|
||||||
- `go test ./internal/app -v`
|
|
||||||
- `go test ./internal/stage -v`
|
|
||||||
- `go test ./internal/artifacts -v`
|
|
||||||
|
|
||||||
Full validation:
|
|
||||||
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Config tests to add or update:
|
|
||||||
|
|
||||||
- `publish.outputs` defaults and validates.
|
|
||||||
- `publish.outputs[].dest` derives from the artifact registry when omitted.
|
|
||||||
- `publish.locks` validates with the same source rules as publish outputs.
|
|
||||||
- old `archive` fails strict decode.
|
|
||||||
- old `promote_artifacts` fails strict decode.
|
|
||||||
- old cleanup fields fail strict decode.
|
|
||||||
|
|
||||||
App and stage tests to add or update:
|
|
||||||
|
|
||||||
- stage order uses `publish` before `notify`.
|
|
||||||
- `run-stage publish` succeeds.
|
|
||||||
- `run-stage archive` fails clearly.
|
|
||||||
- `narratio publish` force-runs the `publish` stage.
|
|
||||||
- `--artifacts` is accepted for `run-stage publish`.
|
|
||||||
- `--artifacts` error text names `analyze` and `publish`.
|
|
||||||
- status and artifact list output show `Published:` and `remote=published`.
|
|
||||||
- lock output says `published` or `not-published`.
|
|
||||||
- previous-cache and restore use `publish` stage metadata.
|
|
||||||
|
|
||||||
Final searches:
|
|
||||||
|
|
||||||
- Config/stage names:
|
|
||||||
- `pipeline.archive`
|
|
||||||
- `archive:`
|
|
||||||
- `promote_artifacts`
|
|
||||||
- `cleanup_after_archive`
|
|
||||||
- `delete_audio_after_archive`
|
|
||||||
- User-facing output:
|
|
||||||
- `Promoted:`
|
|
||||||
- `remote=promoted`
|
|
||||||
- `not-promoted`
|
|
||||||
- Runtime symbols and metadata:
|
|
||||||
- `ArchiveConfig`
|
|
||||||
- `ArchivePromotionRule`
|
|
||||||
- `S3PromotedArtifactKey`
|
|
||||||
- `promoted_paths`
|
|
||||||
- `promoted_files_uploaded`
|
|
||||||
- `locked_promotions`
|
|
||||||
|
|
||||||
Expected remaining matches should be limited to this roadmap and narrowly
|
|
||||||
justified historical references until the roadmap is fully retired.
|
|
||||||
|
|
||||||
## Architecture Guardrails
|
|
||||||
|
|
||||||
- Keep Narratio explicit and stage-driven.
|
|
||||||
- Do not introduce a generic workflow or DAG abstraction.
|
|
||||||
- Keep strict YAML decoding.
|
|
||||||
- Keep remote path construction centralized.
|
|
||||||
- Keep storage details behind `storage.ObjectStore`.
|
|
||||||
- Keep AWS SDK types inside storage adapters.
|
|
||||||
- Preserve manifest-driven resume and restore behavior.
|
|
||||||
- Preserve current-state commit ordering with `current/run_id.txt` written
|
|
||||||
last.
|
|
||||||
- Keep raw secrets out of configs, manifests, logs, generated configs, and
|
|
||||||
publish metadata.
|
|
||||||
- Keep planned behavior only in this roadmap until implementation lands.
|
|
||||||
|
|
||||||
## Assumptions
|
|
||||||
|
|
||||||
- This is a breaking public/config/stage contract change.
|
|
||||||
- No compatibility aliases are retained.
|
|
||||||
- No migration logic is needed for in-progress local manifests.
|
|
||||||
- No migration logic is needed for old remote manifests.
|
|
||||||
- Existing remote objects are not moved or renamed.
|
|
||||||
- `publish` means uploading run history, writing configured published outputs,
|
|
||||||
and committing current state.
|
|
||||||
- `run history` is the preferred term for immutable per-run records under
|
|
||||||
`runs/<run_id>/`.
|
|
||||||
- `archive` remains acceptable only as a generic English concept in historical
|
|
||||||
roadmap context, not as a public Narratio command, config field, stage name,
|
|
||||||
or metadata term after implementation.
|
|
||||||
@@ -1,210 +0,0 @@
|
|||||||
# Roadmap: Transcript Artifact Naming
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
## Problem
|
|
||||||
|
|
||||||
Narratio's built-in transcript artifact names and canonical paths currently mix
|
|
||||||
operator-facing artifact meaning with historical stage and tool terminology:
|
|
||||||
|
|
||||||
- `narratio.transcript.merged` maps to `transcripts/merged.json`.
|
|
||||||
- `narratio.transcript.polished` maps to `transcripts/processed.json`.
|
|
||||||
- `narratio.transcript.full` maps to `transcripts/normalized.json`.
|
|
||||||
- `narratio.transcript.trimmed` maps to `transcripts/trimmed.json`.
|
|
||||||
|
|
||||||
This makes the public artifact surface harder to reason about. Operators see
|
|
||||||
`full`, `normalized`, `processed`, `polished`, `merged`, and `trimmed` used in
|
|
||||||
different places for the same transcript lineage.
|
|
||||||
|
|
||||||
The transcript source IDs, canonical paths, and manifest output kinds should
|
|
||||||
use one vocabulary based on each transcript's role in the session artifact
|
|
||||||
model.
|
|
||||||
|
|
||||||
## Target Model
|
|
||||||
|
|
||||||
Built-in transcript artifacts should use these public source IDs, canonical
|
|
||||||
paths, and manifest output kinds:
|
|
||||||
|
|
||||||
| Source ID | Canonical path | Output kind | Meaning |
|
|
||||||
| --- | --- | --- | --- |
|
|
||||||
| `narratio.transcript.base` | `transcripts/base.json` | `transcript_base` | First unified transcript produced by merging per-speaker raw transcripts. |
|
|
||||||
| `narratio.transcript.polished` | `transcripts/polished.json` | `transcript_polished` | Audita-polished transcript. |
|
|
||||||
| `narratio.transcript.final` | `transcripts/final.json` | `transcript_final` | Full final transcript after normalization. |
|
|
||||||
| `narratio.transcript.final_trimmed` | `transcripts/final.trimmed.json` | `transcript_final_trimmed` | Trimmed version of the final transcript. |
|
|
||||||
|
|
||||||
Stage names remain process-oriented and unchanged:
|
|
||||||
|
|
||||||
- `merge`
|
|
||||||
- `polish`
|
|
||||||
- `normalize`
|
|
||||||
- `trim`
|
|
||||||
|
|
||||||
Downstream adapter contracts also remain process-oriented. The rename changes
|
|
||||||
Narratio's artifact model, canonical paths, config examples, archive promotion
|
|
||||||
sources, lock sources, status output, and documentation. It should not rename
|
|
||||||
the stages themselves or move external integration details into stage logic.
|
|
||||||
|
|
||||||
## Compatibility Policy
|
|
||||||
|
|
||||||
This is a hard cutover.
|
|
||||||
|
|
||||||
After implementation, these old source IDs should be rejected:
|
|
||||||
|
|
||||||
- `narratio.transcript.merged`
|
|
||||||
- `narratio.transcript.full`
|
|
||||||
- `narratio.transcript.trimmed`
|
|
||||||
|
|
||||||
These old canonical paths should not be compatibility fallbacks:
|
|
||||||
|
|
||||||
- `transcripts/merged.json`
|
|
||||||
- `transcripts/processed.json`
|
|
||||||
- `transcripts/normalized.json`
|
|
||||||
- `transcripts/trimmed.json`
|
|
||||||
|
|
||||||
Existing remote archives are not migrated automatically. Operators who want
|
|
||||||
new promoted keys for old sessions should republish those sessions after
|
|
||||||
updating configuration.
|
|
||||||
|
|
||||||
## Implementation Stages
|
|
||||||
|
|
||||||
### Stage 1: Centralize Transcript Artifact Naming
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Consolidate transcript artifact source IDs, canonical paths, and output kinds
|
|
||||||
in the artifact/path layer before changing runtime behavior.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Add or consolidate constants/helpers for built-in transcript source IDs.
|
|
||||||
- Add or consolidate constants/helpers for canonical transcript paths.
|
|
||||||
- Add or consolidate constants/helpers for transcript manifest output kinds.
|
|
||||||
- Keep source ID, path, and output-kind mappings in one registry or one
|
|
||||||
obviously shared artifact model.
|
|
||||||
- Update artifact registry tests to prove the target mapping.
|
|
||||||
- Avoid changing stage output behavior in this stage unless the implementation
|
|
||||||
is simpler and still reviewable.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- There is one clear source of truth for built-in transcript artifact names,
|
|
||||||
paths, and output kinds.
|
|
||||||
- Tests prove the new target mapping in the artifact layer.
|
|
||||||
- No generic workflow abstraction is introduced.
|
|
||||||
|
|
||||||
### Stage 2: Rename Runtime Outputs and Defaults
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Switch runtime behavior to the new transcript artifact model.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Update `merge` to write and record `transcripts/base.json` with
|
|
||||||
`transcript_base`.
|
|
||||||
- Update `polish` to write and record `transcripts/polished.json` with
|
|
||||||
`transcript_polished`.
|
|
||||||
- Update `normalize` to write and record `transcripts/final.json` with
|
|
||||||
`transcript_final`.
|
|
||||||
- Update `trim` to write and record `transcripts/final.trimmed.json` with
|
|
||||||
`transcript_final_trimmed`.
|
|
||||||
- Update normalize and trim defaults to:
|
|
||||||
- `pipeline.normalize.output_path: transcripts/final.json`
|
|
||||||
- `pipeline.trim.output_path: transcripts/final.trimmed.json`
|
|
||||||
- Update built-in artifact resolution, archive promotion destination
|
|
||||||
derivation, archive locks, status output, artifact catalog output,
|
|
||||||
previous-cache resolution, restore planning, and restore execution to use
|
|
||||||
the new registry values.
|
|
||||||
- Ensure old source IDs fail config validation.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- New runs produce the target canonical transcript files.
|
|
||||||
- Manifest outputs use the target output kinds.
|
|
||||||
- Archive promotion and lock validation accept new source IDs and reject old
|
|
||||||
source IDs.
|
|
||||||
- Status and artifact listing display new source IDs.
|
|
||||||
- Restore uses the new canonical paths and does not restore old transcript
|
|
||||||
paths as canonical outputs.
|
|
||||||
|
|
||||||
### Stage 3: Update Tests, Examples, and Current Documentation
|
|
||||||
|
|
||||||
Status: Implemented
|
|
||||||
|
|
||||||
Update all implemented-behavior references after the runtime cutover lands.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Update examples to use `narratio.transcript.final_trimmed` and
|
|
||||||
`transcripts/final.trimmed.json` where trimmed final transcript is intended.
|
|
||||||
- Update examples that refer to full final transcripts to use
|
|
||||||
`narratio.transcript.final` and `transcripts/final.json`.
|
|
||||||
- Update `docs/config.md`, `docs/internal/artifacts.md`, stage docs,
|
|
||||||
CLI examples, operations examples, archive examples, lock examples, and
|
|
||||||
status/artifact-list examples.
|
|
||||||
- Add strict validation tests proving old source IDs are rejected.
|
|
||||||
- Mark roadmap stages implemented only after code, tests, examples, and
|
|
||||||
current-behavior docs agree.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Maintained examples load and validate.
|
|
||||||
- Current-behavior docs describe only implemented new names.
|
|
||||||
- Old names remain only in this roadmap as historical/planning context until
|
|
||||||
this roadmap is retired or archived.
|
|
||||||
|
|
||||||
## Test Guidance
|
|
||||||
|
|
||||||
Run focused tests while implementing:
|
|
||||||
|
|
||||||
- `go test ./internal/artifacts -v`
|
|
||||||
- `go test ./internal/config -v`
|
|
||||||
- `go test ./internal/stage -v`
|
|
||||||
- `go test ./internal/app -v`
|
|
||||||
|
|
||||||
Run full validation before finishing:
|
|
||||||
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Run final searches:
|
|
||||||
|
|
||||||
- Old source IDs:
|
|
||||||
- `narratio.transcript.merged`
|
|
||||||
- `narratio.transcript.full`
|
|
||||||
- `narratio.transcript.trimmed`
|
|
||||||
- Old paths:
|
|
||||||
- `transcripts/merged.json`
|
|
||||||
- `transcripts/processed.json`
|
|
||||||
- `transcripts/normalized.json`
|
|
||||||
- `transcripts/trimmed.json`
|
|
||||||
- Old output kinds:
|
|
||||||
- `transcript_merged`
|
|
||||||
- `transcript_processed`
|
|
||||||
- `transcript_normalized`
|
|
||||||
- `transcript_trimmed`
|
|
||||||
|
|
||||||
Expected remaining matches should be limited to this roadmap's
|
|
||||||
historical/planning references until the roadmap is fully completed.
|
|
||||||
|
|
||||||
## Architecture Guardrails
|
|
||||||
|
|
||||||
- Keep Narratio explicit and stage-driven; do not introduce a generic workflow
|
|
||||||
or DAG abstraction.
|
|
||||||
- Keep path and artifact naming in centralized helpers rather than scattered
|
|
||||||
string concatenation.
|
|
||||||
- Preserve manifest-driven resume behavior.
|
|
||||||
- Keep storage details behind storage adapters.
|
|
||||||
- Do not move Seriatim, Audita, or Scriptorium command details out of their
|
|
||||||
adapter boundaries.
|
|
||||||
- Keep current-behavior documentation in sync only after implementation lands;
|
|
||||||
planned behavior belongs in this roadmap until then.
|
|
||||||
|
|
||||||
## Assumptions
|
|
||||||
|
|
||||||
- The cutover is intentionally not backward-compatible.
|
|
||||||
- Existing remote archive objects are not renamed or migrated automatically.
|
|
||||||
- Stage names and downstream adapter request field names remain unchanged.
|
|
||||||
- The term `base` is preferred over `merged` for the first unified transcript.
|
|
||||||
- The term `final` is preferred over `full` or `normalized` for the full final
|
|
||||||
transcript.
|
|
||||||
- The trimmed final path is `transcripts/final.trimmed.json`.
|
|
||||||
@@ -1,35 +1,62 @@
|
|||||||
# Troubleshooting
|
# Troubleshooting
|
||||||
|
|
||||||
## Purpose
|
Operational diagnosis guide for common Narratio failures.
|
||||||
Canonical operator troubleshooting guide for recurring Narratio failures.
|
|
||||||
|
|
||||||
## Config discovery failure
|
## Config file not found
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
- command fails because `pipeline.yml`, `campaign.yml`, or `session.yml` was not found.
|
|
||||||
|
|
||||||
Likely cause:
|
- command fails to resolve `pipeline.yml`, `campaign.yml`, or `session.yml`.
|
||||||
- missing files in discovery paths.
|
|
||||||
- missing/incorrect campaign selection.
|
Likely causes:
|
||||||
- local file exists but was not passed explicitly.
|
|
||||||
|
- missing files in default search paths;
|
||||||
|
- wrong campaign selection;
|
||||||
|
- omitted explicit flags.
|
||||||
|
|
||||||
Diagnostics:
|
Diagnostics:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ls -l /usr/local/etc/narratio/pipeline.yml /etc/narratio/pipeline.yml
|
narratio session plan 2026-04-04
|
||||||
ls -l /usr/local/etc/narratio/session.yml /etc/narratio/session.yml
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe fix:
|
Safe fix:
|
||||||
|
|
||||||
- pass explicit `--config`, `--campaign` or `--campaign-file`, and `--session`.
|
- pass explicit `--config`, `--campaign` or `--campaign-file`, and `--session`.
|
||||||
|
|
||||||
## Templated session file rejected
|
Relevant reference: [Configuration discovery](./config.md#discovery-and-selection).
|
||||||
|
|
||||||
|
## Session template placeholders rejected
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
- load fails because `session.yml` must be concrete.
|
|
||||||
|
- load error says session file must be concrete or contains `{{ ... }}` placeholders.
|
||||||
|
|
||||||
Likely cause:
|
Likely cause:
|
||||||
- template placeholders (`{{ ... }}`) still present in loaded session config.
|
|
||||||
|
- using template content as runtime session config.
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session validate 2026-04-04 --session /path/session.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- generate concrete session YAML with `narratio session init`.
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Session Initialization](./operations.md#session-initialization).
|
||||||
|
|
||||||
|
## Strict decode or schema validation failure
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- unknown field / invalid value error during config load.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- stale field name, typo, invalid enum, or invalid duration/path format.
|
||||||
|
|
||||||
Diagnostics:
|
Diagnostics:
|
||||||
|
|
||||||
@@ -38,82 +65,241 @@ narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /pa
|
|||||||
```
|
```
|
||||||
|
|
||||||
Safe fix:
|
Safe fix:
|
||||||
- generate concrete session YAML via `narratio session init`.
|
|
||||||
|
|
||||||
## Strict decode or validation failure
|
- align config with [Configuration](./config.md) and the
|
||||||
|
[maintained examples](../examples/README.md).
|
||||||
|
|
||||||
|
Relevant reference: [Configuration](./config.md).
|
||||||
|
|
||||||
|
## Audio mode conflict
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
- unknown field or invalid value error during config load.
|
|
||||||
|
- validation fails on session audio configuration.
|
||||||
|
|
||||||
Likely cause:
|
Likely cause:
|
||||||
- typo, stale field name, or invalid value.
|
|
||||||
|
- configured both local and S3 session audio inputs.
|
||||||
|
|
||||||
Diagnostics:
|
Diagnostics:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
|
narratio session validate 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe fix:
|
Safe fix:
|
||||||
- align config with [docs/config.md](./config.md) and maintained examples.
|
|
||||||
|
|
||||||
## `--artifacts` selection failure
|
- use local mode (`audio_dir` or `audio_files`) or S3 mode (`audio_s3.prefix`), not both.
|
||||||
|
|
||||||
|
Relevant reference: [Session configuration](./config.md#session).
|
||||||
|
|
||||||
|
## `--artifacts` selection error
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
- command fails on unknown/invalid selected artifact key.
|
|
||||||
|
|
||||||
Likely cause:
|
- unknown artifact key or invalid `--artifacts` usage.
|
||||||
- artifact key not defined in `pipeline.scriptorium.artifacts`.
|
|
||||||
- empty token in `--artifacts` input.
|
|
||||||
|
|
||||||
Safe fix:
|
Likely causes:
|
||||||
- use only configured artifact keys.
|
|
||||||
|
|
||||||
## `run-stage --artifacts` unsupported stage
|
- key not defined in `pipeline.scriptorium.artifacts`;
|
||||||
|
- empty list entry (for example trailing comma);
|
||||||
Symptom:
|
- `run-stage` used with non-`analyze`/`publish` target.
|
||||||
- `run-stage` rejects `--artifacts` for the selected stage.
|
|
||||||
|
|
||||||
Likely cause:
|
|
||||||
- `--artifacts` used with a stage other than `analyze` or `publish`.
|
|
||||||
|
|
||||||
Safe fix:
|
|
||||||
- use `--artifacts` only with `run-stage analyze ...` or `run-stage publish ...`.
|
|
||||||
|
|
||||||
## Previous-session input unavailable
|
|
||||||
|
|
||||||
Symptom:
|
|
||||||
- analyze fails on required previous-session artifact input.
|
|
||||||
|
|
||||||
Likely cause:
|
|
||||||
- `previous/**` cache not hydrated for this session.
|
|
||||||
|
|
||||||
Diagnostics:
|
Diagnostics:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
narratio session artifacts 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- provide only configured keys and use `--artifacts` with supported commands/stages.
|
||||||
|
|
||||||
|
Relevant reference: [CLI artifact selection](./cli.md).
|
||||||
|
|
||||||
|
## Notarius executable missing
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- extraction fails while resolving or starting the Notarius executable.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- `pipeline.notarius.binary` is not installed, executable, or on `PATH`;
|
||||||
|
- a configured executable path is wrong.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- install a compatible Notarius release or correct the binary setting, then
|
||||||
|
rerun extraction.
|
||||||
|
|
||||||
|
Relevant references: [Notarius configuration](./config.md#notarius-output-entries)
|
||||||
|
and [Notarius integration](./integrations/notarius.md).
|
||||||
|
|
||||||
|
## Notarius exits nonzero
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- extraction reports a Notarius exit error instead of a receipt.
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
|
||||||
|
- inspect `runs/{run_id}/extract/notarius.stderr.log`; stdout is reserved for
|
||||||
|
the receipt and is not merged with diagnostics.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- correct the reported Notarius pipeline, input, provider, or configuration
|
||||||
|
failure and rerun extraction. Do not edit a staged output bundle into place.
|
||||||
|
|
||||||
|
After a failed replacement, an older immutable bundle may still exist even
|
||||||
|
though the current session manifest has no successful extraction payload. This
|
||||||
|
is expected audit state, not a signal to relink the old bundle manually.
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Extraction Workflow](./operations.md#extraction-workflow).
|
||||||
|
|
||||||
|
## Atomic Notarius promotion unsupported
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- extraction fails with `atomic no-replace directory promotion is unsupported`
|
||||||
|
before a durable bundle or temporary promotion tree is created.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Narratio is running on an operating system other than Linux, macOS, or
|
||||||
|
Windows, where the required atomic no-replace directory primitive has not
|
||||||
|
been implemented and verified.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- run extraction on Linux, macOS, or Windows. Do not replace the atomic commit
|
||||||
|
with a manual copy or move; the session manifest must never observe a partial
|
||||||
|
or overwritten bundle.
|
||||||
|
|
||||||
|
This is an extraction-specific platform boundary, not a support statement for
|
||||||
|
unrelated Narratio workflows. See
|
||||||
|
[Operations: Extraction Workflow](./operations.md#extraction-workflow).
|
||||||
|
|
||||||
|
## Notarius receipt or index incompatible
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- extraction rejects the receipt schema, pipeline identity, bundle/index path,
|
||||||
|
lane descriptor, or payload path even though Notarius exited successfully.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- Narratio and Notarius versions disagree on their consumer contract;
|
||||||
|
- the configured pipeline or lane constraints are stale;
|
||||||
|
- output paths escape the bundle or traverse symlinks.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- compare installed Notarius output with the canonical Notarius contracts,
|
||||||
|
including receipt `index_file: index.json` and index management names
|
||||||
|
`manifest.json`, `rejected.json`, and `warnings.json`; align
|
||||||
|
`pipeline.notarius` constraints and rerun. Do not bypass confinement or schema
|
||||||
|
checks.
|
||||||
|
|
||||||
|
Relevant reference: [Notarius integration](./integrations/notarius.md).
|
||||||
|
|
||||||
|
## Required Notarius lane rejected or missing
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- extraction fails because a configured lane is rejected, missing, duplicated,
|
||||||
|
or incompatible, including after a zero exit.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- inspect the Notarius diagnostic log and bundle rejection/warning information;
|
||||||
|
- correct the Notarius module or the exact declared lane contract;
|
||||||
|
- remove an output declaration only if downstream consumers genuinely no longer
|
||||||
|
require that source, then rerun extraction.
|
||||||
|
|
||||||
|
Every configured output is required. Narratio does not promote a partial result.
|
||||||
|
|
||||||
|
## Extraction resume invalidated
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- a previously successful extraction runs again during ordinary continuation.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- the executable/config path, pipeline ID, timeout, working directory, or
|
||||||
|
configured output contracts changed;
|
||||||
|
- the durable bundle, index, lane set, provenance, regular-file status, or
|
||||||
|
checksum no longer validates.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- allow the automatic rerun after verifying the current configuration. Treat
|
||||||
|
an unsafe path or symlink error as filesystem corruption or tampering and
|
||||||
|
investigate it rather than replacing files manually.
|
||||||
|
|
||||||
|
## Notarius transitive configuration changed
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- Notarius profiles, prompts, modules, imported files, or references changed,
|
||||||
|
but Narratio still considers the previous extraction resumable.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage extract 2026-04-04 --force
|
||||||
|
```
|
||||||
|
|
||||||
|
Narratio fingerprints its invocation contract, not the contents of transitive
|
||||||
|
Notarius inputs. Always force extraction after changing them; downstream
|
||||||
|
successful stages are then marked stale normally.
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Extraction Workflow](./operations.md#extraction-workflow).
|
||||||
|
|
||||||
|
## Previous-session artifact input missing
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- prepare/analyze fails due to missing required previous-session artifact cache input.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- missing `session.previous_session_id`;
|
||||||
|
- previous artifact not restored/published for source session.
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session validate 2026-04-04
|
||||||
narratio session status 2026-04-04
|
narratio session status 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe fix:
|
Safe fix:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session restore 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
or rerun prepare after correcting session config:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio run-stage prepare 2026-04-04 --force
|
narratio run-stage prepare 2026-04-04 --force
|
||||||
```
|
```
|
||||||
|
|
||||||
Or rehydrate from remote current state:
|
Relevant reference: [Operations: Restore Workflow](./operations.md#restore-workflow).
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio session restore 2026-04-04
|
|
||||||
```
|
|
||||||
|
|
||||||
## Session lock conflict (`.lock`)
|
## Session lock conflict (`.lock`)
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
- command fails with lock conflict.
|
|
||||||
|
|
||||||
Likely cause:
|
- command fails acquiring session lock.
|
||||||
- another process is running for the same session.
|
|
||||||
- stale lock file from interrupted command.
|
Likely causes:
|
||||||
|
|
||||||
|
- another process is running for the same session;
|
||||||
|
- stale lock left by interrupted process.
|
||||||
|
|
||||||
Diagnostics:
|
Diagnostics:
|
||||||
|
|
||||||
@@ -123,16 +309,21 @@ ps aux | grep narratio
|
|||||||
```
|
```
|
||||||
|
|
||||||
Safe fix:
|
Safe fix:
|
||||||
- wait for active process; remove stale lock only if no process is active.
|
|
||||||
|
|
||||||
## Restore current pointer/manifest missing
|
- wait for active process completion;
|
||||||
|
- remove stale lock only after confirming no live process owns it.
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Local State Layout](./operations.md#local-state-layout).
|
||||||
|
|
||||||
|
## Restore conflict without `--force`
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
- restore fails reading remote current state.
|
|
||||||
|
- restore fails with conflict count.
|
||||||
|
|
||||||
Likely cause:
|
Likely cause:
|
||||||
- publish commit did not complete.
|
|
||||||
- `current/run_id.txt` or `current/manifest.json` is missing.
|
- local durable files differ from remote restore sources.
|
||||||
|
|
||||||
Diagnostics:
|
Diagnostics:
|
||||||
|
|
||||||
@@ -141,83 +332,148 @@ narratio session restore 2026-04-04 --dry-run
|
|||||||
```
|
```
|
||||||
|
|
||||||
Safe fix:
|
Safe fix:
|
||||||
- republish from a healthy local session state.
|
|
||||||
|
|
||||||
## Restore conflict without `--force`
|
- review conflicts;
|
||||||
|
- rerun with `--force` only when remote state should overwrite local.
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Restore Workflow](./operations.md#restore-workflow).
|
||||||
|
|
||||||
|
## Restore current-state discovery failure
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
- restore reports conflict and exits.
|
|
||||||
|
|
||||||
Likely cause:
|
- restore cannot find current pointer or current manifest.
|
||||||
- local durable file differs from remote restore source.
|
|
||||||
|
|
||||||
Safe fix:
|
Likely causes:
|
||||||
- inspect with `--dry-run`.
|
|
||||||
- rerun with `--force` only when remote should overwrite local.
|
|
||||||
|
|
||||||
## Secrets or credentials failure
|
- no committed publish current state;
|
||||||
|
- storage credentials or connectivity failure.
|
||||||
Symptom:
|
|
||||||
- startup fails loading secrets dir, or storage/tool auth fails at runtime.
|
|
||||||
|
|
||||||
Likely cause:
|
|
||||||
- invalid `pipeline.secrets.env_dir`.
|
|
||||||
- missing credential env vars.
|
|
||||||
|
|
||||||
Diagnostics:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
ls -la /path/to/secrets_dir
|
|
||||||
env | grep -E 'AUDITA|OBJECT_STORAGE|AWS|SCRIPTORIUM'
|
|
||||||
```
|
|
||||||
|
|
||||||
Safe fix:
|
|
||||||
- fix path/permissions/env vars; keep secret values out of YAML.
|
|
||||||
|
|
||||||
## S3 audio prepare failure
|
|
||||||
|
|
||||||
Symptom:
|
|
||||||
- prepare fails in S3 mode (list/download/no files/backend error).
|
|
||||||
|
|
||||||
Likely cause:
|
|
||||||
- bad `session.inputs.audio_s3.prefix`.
|
|
||||||
- no `.flac` objects at prefix.
|
|
||||||
- bad storage credentials/config.
|
|
||||||
- mixed local+S3 audio config.
|
|
||||||
|
|
||||||
Diagnostics:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio run-stage prepare 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
|
|
||||||
```
|
|
||||||
|
|
||||||
Safe fix:
|
|
||||||
- configure exactly one audio mode and verify storage access.
|
|
||||||
|
|
||||||
## Publish output or current-pointer failure
|
|
||||||
|
|
||||||
Symptom:
|
|
||||||
- publish fails on required output source missing, upload error, or commit-marker write failure.
|
|
||||||
|
|
||||||
Likely cause:
|
|
||||||
- required source file not produced.
|
|
||||||
- storage upload failed before `current/run_id.txt` write.
|
|
||||||
|
|
||||||
Diagnostics:
|
Diagnostics:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio session status 2026-04-04
|
narratio session status 2026-04-04
|
||||||
narratio run-stage publish 2026-04-04 --config /path/pipeline.yml --campaign-file /path/campaign.yml --session /path/session.yml
|
narratio session restore 2026-04-04 --dry-run
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe fix:
|
Safe fix:
|
||||||
- rerun upstream stages to regenerate required outputs.
|
|
||||||
- adjust `pipeline.publish.outputs` source/dest rules.
|
|
||||||
- retry after storage issue is fixed.
|
|
||||||
|
|
||||||
## Helpful Links
|
- resolve storage/auth issue;
|
||||||
|
- republish from healthy local state if current pointer is missing.
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Publish Workflow](./operations.md#publish-workflow).
|
||||||
|
|
||||||
|
## Publish output failure
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- publish fails on missing required source, upload error, or commit write.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- required source file not produced;
|
||||||
|
- lock/state expectations mismatch;
|
||||||
|
- remote storage failure.
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session artifacts 2026-04-04 --remote
|
||||||
|
narratio session status 2026-04-04
|
||||||
|
narratio run-stage publish 2026-04-04 --force
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- regenerate missing sources by rerunning prerequisite stages;
|
||||||
|
- correct publish source/destination rules;
|
||||||
|
- retry after storage failure is resolved.
|
||||||
|
|
||||||
|
Relevant reference: [Publish configuration](./config.md#publish-configuration-summary).
|
||||||
|
|
||||||
|
## Render markdown source missing
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- analyze or publish fails because `narratio.transcript.final_markdown` or `narratio.transcript.final_trimmed_markdown` is unavailable.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- render stage was not executed after transcript changes;
|
||||||
|
- render stage failed before producing canonical markdown outputs.
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session status 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- rerun render and then retry downstream stage(s):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage render 2026-04-04 --force
|
||||||
|
narratio run-stage analyze 2026-04-04 --force
|
||||||
|
```
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Stage Execution](./operations.md#stage-execution-and-continuation-behavior).
|
||||||
|
|
||||||
|
## Secrets or storage credential failure
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- object-store command fails at initialization/auth.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- invalid `pipeline.secrets.env_dir`;
|
||||||
|
- missing credential environment variables;
|
||||||
|
- invalid S3 endpoint/bucket settings.
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ls -la /path/to/secrets_dir
|
||||||
|
env | grep -E 'OBJECT_STORAGE|AWS|AUDITA|SCRIPTORIUM'
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- correct secret-file path and permissions;
|
||||||
|
- provide required env vars;
|
||||||
|
- keep secret values out of YAML.
|
||||||
|
|
||||||
|
Relevant reference: [Secrets](./config.md#secrets-handling).
|
||||||
|
|
||||||
|
## S3 audio prepare failure
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- prepare fails listing/downloading session S3 audio.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- incorrect `session.inputs.audio_s3.prefix`;
|
||||||
|
- no matching `.flac` objects;
|
||||||
|
- storage connectivity or permissions failure.
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session validate 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- verify prefix contents and storage access;
|
||||||
|
- keep session audio mode consistent.
|
||||||
|
|
||||||
|
Relevant reference: [Operations](./operations.md).
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
- [docs/config.md](./config.md)
|
|
||||||
- [docs/cli.md](./cli.md)
|
- [docs/cli.md](./cli.md)
|
||||||
|
- [docs/config.md](./config.md)
|
||||||
- [docs/operations.md](./operations.md)
|
- [docs/operations.md](./operations.md)
|
||||||
- [docs/internal/stage-publish.md](./internal/stage-publish.md)
|
- [docs/internal/stage-publish.md](./internal/stage-publish.md)
|
||||||
|
|||||||
48
examples/README.md
Normal file
48
examples/README.md
Normal file
@@ -0,0 +1,48 @@
|
|||||||
|
# Maintained Examples
|
||||||
|
|
||||||
|
These files are safe, copyable starting points for Narratio configuration and
|
||||||
|
input structure. Replace placeholder identifiers, storage names, integration
|
||||||
|
URLs, and paths for the target environment. Field meanings and defaults belong
|
||||||
|
in the [configuration reference](../docs/config.md).
|
||||||
|
|
||||||
|
## Pipeline Configuration
|
||||||
|
|
||||||
|
- [Minimal pipeline](pipeline.minimal.yml): campaign discovery plus the required
|
||||||
|
WhisperX URL.
|
||||||
|
- [Production-shaped pipeline](pipeline.production.yml): S3 storage, publish,
|
||||||
|
external tools, and configured Scriptorium artifacts.
|
||||||
|
- [Full annotated pipeline](pipeline.full.annotated.yml): every implemented
|
||||||
|
pipeline section with explanatory comments.
|
||||||
|
- [Extraction subset pipeline](pipeline.extraction-subset.yml): a focused
|
||||||
|
Scriptorium artifact consuming only three declared Notarius lanes.
|
||||||
|
|
||||||
|
The existing `internal/config` example test loads and validates each pipeline
|
||||||
|
with the sample campaign and a compatible local- or S3-audio session.
|
||||||
|
|
||||||
|
## Campaign And Session Configuration
|
||||||
|
|
||||||
|
- [Sample campaign](campaigns/sample-campaign/campaign.yml), its
|
||||||
|
[session template](campaigns/sample-campaign/session.template.yml), and its
|
||||||
|
adjacent stable inputs provide a complete campaign directory shape.
|
||||||
|
- [Local-audio session](session.local-audio.yml) and
|
||||||
|
[S3-audio session](session.s3-audio.yml) are concrete session files.
|
||||||
|
- [Session template](session.template.yml) and the campaign-local equivalent
|
||||||
|
demonstrate the narrow placeholder syntax consumed by `session init`; they
|
||||||
|
are templates, not runtime session files.
|
||||||
|
|
||||||
|
## Input Fixtures
|
||||||
|
|
||||||
|
- [Speakers](speakers.yml), [autocorrect](autocorrect.yml), and
|
||||||
|
[glossary](glossary.yml) show the standalone input shapes.
|
||||||
|
- The sample campaign references its local
|
||||||
|
[speakers](campaigns/sample-campaign/speakers.yml),
|
||||||
|
[autocorrect](campaigns/sample-campaign/autocorrect.yml),
|
||||||
|
[glossary](campaigns/sample-campaign/glossary.yml),
|
||||||
|
[players](campaigns/sample-campaign/players.yml), and
|
||||||
|
[party](campaigns/sample-campaign/party.yml) fixtures.
|
||||||
|
- [Sample speaker audio](audio/sample-speaker.flac) is a text placeholder that
|
||||||
|
reserves the expected filename and directory shape. Replace it with a real
|
||||||
|
FLAC file before running transcription.
|
||||||
|
|
||||||
|
The examples contain environment-variable names but no credential values. They
|
||||||
|
use fictional campaign content and reserved example domains.
|
||||||
@@ -4,3 +4,5 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
|
|||||||
2
examples/campaigns/sample-campaign/party.yml
Normal file
2
examples/campaigns/sample-campaign/party.yml
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
- name: Example Hero
|
||||||
|
type: pc
|
||||||
2
examples/campaigns/sample-campaign/players.yml
Normal file
2
examples/campaigns/sample-campaign/players.yml
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
- name: Example Player
|
||||||
|
role: player
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
match:
|
match:
|
||||||
- speaker: "Eric Rakestraw"
|
- speaker: "Example Speaker"
|
||||||
match:
|
match:
|
||||||
- "Eric_Rakestraw"
|
- "Example_Speaker"
|
||||||
- "Eric"
|
- "Example"
|
||||||
|
|||||||
55
examples/pipeline.extraction-subset.yml
Normal file
55
examples/pipeline.extraction-subset.yml
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
# Purpose-specific extraction example: a Scriptorium session brief consumes
|
||||||
|
# only the three Notarius lanes it needs.
|
||||||
|
|
||||||
|
campaigns:
|
||||||
|
root: /usr/local/share/narratio/campaigns
|
||||||
|
default_campaign_id: sample-campaign
|
||||||
|
|
||||||
|
whisperx:
|
||||||
|
transcribe_url: "https://transcription.example.com/transcribe"
|
||||||
|
|
||||||
|
notarius:
|
||||||
|
enabled: true
|
||||||
|
binary: notarius
|
||||||
|
config_path: /usr/local/etc/notarius/config.yml
|
||||||
|
pipeline_id: dnd-session
|
||||||
|
timeout: 3h
|
||||||
|
outputs:
|
||||||
|
npc_registry:
|
||||||
|
lane_id: npc-registry
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.npc_registry
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/npc-registry
|
||||||
|
location_registry:
|
||||||
|
lane_id: location-registry
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.location_registry
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/location-registry
|
||||||
|
scene_descriptions:
|
||||||
|
lane_id: scene-descriptions
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.scene_descriptions
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/scene-descriptions
|
||||||
|
|
||||||
|
scriptorium:
|
||||||
|
binary: scriptorium
|
||||||
|
config_path: /usr/local/etc/scriptorium/config.yml
|
||||||
|
artifacts:
|
||||||
|
session_brief:
|
||||||
|
enabled: true
|
||||||
|
prompt_id: dnd.session_brief
|
||||||
|
output_path: artifacts/session_brief.md
|
||||||
|
inputs:
|
||||||
|
npcs:
|
||||||
|
source: narratio.extraction.npc_registry
|
||||||
|
required: true
|
||||||
|
locations:
|
||||||
|
source: narratio.extraction.location_registry
|
||||||
|
required: true
|
||||||
|
scenes:
|
||||||
|
source: narratio.extraction.scene_descriptions
|
||||||
|
required: true
|
||||||
|
|
||||||
@@ -48,12 +48,23 @@ publish:
|
|||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
required: true
|
required: true
|
||||||
|
- source: narratio.transcript.final_markdown
|
||||||
|
dest: transcripts/final.md
|
||||||
|
required: true
|
||||||
|
- source: narratio.transcript.final_trimmed_markdown
|
||||||
|
dest: transcripts/final.trimmed.md
|
||||||
|
required: true
|
||||||
- source: narratio.artifact.session_recap
|
- source: narratio.artifact.session_recap
|
||||||
dest: artifacts/session_recap.md
|
dest: artifacts/session_recap.md
|
||||||
required: true
|
required: true
|
||||||
- source: narratio.artifact.player_handout
|
- source: narratio.artifact.player_handout
|
||||||
dest: artifacts/player_handout.md
|
dest: artifacts/player_handout.md
|
||||||
required: false
|
required: false
|
||||||
|
# Extraction lanes publish only when named explicitly; the bundle and index
|
||||||
|
# are never implicit publish sources.
|
||||||
|
- source: narratio.extraction.npc_registry
|
||||||
|
dest: artifacts/extraction/npc-registry.json
|
||||||
|
required: true
|
||||||
|
|
||||||
whisperx:
|
whisperx:
|
||||||
# Required.
|
# Required.
|
||||||
@@ -104,20 +115,91 @@ normalize:
|
|||||||
report: true
|
report: true
|
||||||
|
|
||||||
trim:
|
trim:
|
||||||
# Keep disabled unless bounds prompt integration is configured.
|
# Optional; defaults shown explicitly.
|
||||||
enabled: false
|
enabled: true
|
||||||
output_path: transcripts/final.trimmed.json
|
output_path: transcripts/final.trimmed.json
|
||||||
bounds:
|
bounds:
|
||||||
prompt_id: dnd.session_bounds
|
prompt_id: dnd.session_bounds
|
||||||
profile_id: local-fast
|
profile_id: ""
|
||||||
transcript_input_name: transcript
|
transcript_input_name: transcript
|
||||||
output_path: reports/session_bounds.json
|
output_path: artifacts/session_bounds.json
|
||||||
timeout: 10m
|
timeout: 10m
|
||||||
render_debug: false
|
render_debug: false
|
||||||
render_output_path: reports/session_bounds.render.json
|
|
||||||
seriatim:
|
seriatim:
|
||||||
report: false
|
report: false
|
||||||
|
|
||||||
|
notarius:
|
||||||
|
# Optional structured extraction between trim and render.
|
||||||
|
enabled: true
|
||||||
|
binary: notarius
|
||||||
|
config_path: /usr/local/etc/notarius/config.yml
|
||||||
|
pipeline_id: dnd-session
|
||||||
|
timeout: 3h
|
||||||
|
working_directory: /usr/local/etc/notarius
|
||||||
|
# Each key creates source narratio.extraction.<key>. These constraints match
|
||||||
|
# the current Notarius D&D lane contracts; update them with Notarius.
|
||||||
|
outputs:
|
||||||
|
item_registry:
|
||||||
|
lane_id: item-registry
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.item_registry
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/item-registry
|
||||||
|
npc_registry:
|
||||||
|
lane_id: npc-registry
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.npc_registry
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/npc-registry
|
||||||
|
location_registry:
|
||||||
|
lane_id: location-registry
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.location_registry
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/location-registry
|
||||||
|
scene_descriptions:
|
||||||
|
lane_id: scene-descriptions
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.scene_descriptions
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/scene-descriptions
|
||||||
|
item_occurrences:
|
||||||
|
lane_id: item-occurrences
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.item_occurrences
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/item-occurrences
|
||||||
|
spells:
|
||||||
|
lane_id: spells
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.spells
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/spells
|
||||||
|
combat_turns:
|
||||||
|
lane_id: combat-turns
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.combat_turns
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/combat-turns
|
||||||
|
npc_occurrences:
|
||||||
|
lane_id: npc-occurrences
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.npc_occurrences
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/npc-occurrences
|
||||||
|
location_occurrences:
|
||||||
|
lane_id: location-occurrences
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.location_occurrences
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/location-occurrences
|
||||||
|
enemy_events:
|
||||||
|
lane_id: enemy-events
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.enemy_events
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/enemy-events
|
||||||
|
|
||||||
scriptorium:
|
scriptorium:
|
||||||
binary: scriptorium
|
binary: scriptorium
|
||||||
config_path: /usr/local/etc/scriptorium/config.yml
|
config_path: /usr/local/etc/scriptorium/config.yml
|
||||||
@@ -138,6 +220,15 @@ scriptorium:
|
|||||||
previous_recap:
|
previous_recap:
|
||||||
source: narratio.previous_session.artifact.session_recap
|
source: narratio.previous_session.artifact.session_recap
|
||||||
required: false
|
required: false
|
||||||
|
players:
|
||||||
|
source: narratio.input.players
|
||||||
|
required: true
|
||||||
|
party:
|
||||||
|
source: narratio.input.party
|
||||||
|
required: true
|
||||||
|
glossary:
|
||||||
|
source: narratio.input.glossary
|
||||||
|
required: false
|
||||||
vars:
|
vars:
|
||||||
session_id: true
|
session_id: true
|
||||||
session_date: true
|
session_date: true
|
||||||
|
|||||||
@@ -26,6 +26,12 @@ publish:
|
|||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
required: true
|
required: true
|
||||||
|
- source: narratio.transcript.final_markdown
|
||||||
|
dest: transcripts/final.md
|
||||||
|
required: true
|
||||||
|
- source: narratio.transcript.final_trimmed_markdown
|
||||||
|
dest: transcripts/final.trimmed.md
|
||||||
|
required: true
|
||||||
- source: narratio.artifact.session_recap
|
- source: narratio.artifact.session_recap
|
||||||
dest: artifacts/session_recap.md
|
dest: artifacts/session_recap.md
|
||||||
required: true
|
required: true
|
||||||
@@ -65,9 +71,6 @@ normalize:
|
|||||||
output_schema: seriatim-intermediate
|
output_schema: seriatim-intermediate
|
||||||
report: true
|
report: true
|
||||||
|
|
||||||
trim:
|
|
||||||
enabled: false
|
|
||||||
|
|
||||||
scriptorium:
|
scriptorium:
|
||||||
binary: scriptorium
|
binary: scriptorium
|
||||||
config_path: /usr/local/etc/scriptorium/config.yml
|
config_path: /usr/local/etc/scriptorium/config.yml
|
||||||
@@ -87,6 +90,15 @@ scriptorium:
|
|||||||
previous_recap:
|
previous_recap:
|
||||||
source: narratio.previous_session.artifact.session_recap
|
source: narratio.previous_session.artifact.session_recap
|
||||||
required: false
|
required: false
|
||||||
|
players:
|
||||||
|
source: narratio.input.players
|
||||||
|
required: true
|
||||||
|
party:
|
||||||
|
source: narratio.input.party
|
||||||
|
required: true
|
||||||
|
glossary:
|
||||||
|
source: narratio.input.glossary
|
||||||
|
required: false
|
||||||
vars:
|
vars:
|
||||||
session_id: true
|
session_id: true
|
||||||
session_date: true
|
session_date: true
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
match:
|
match:
|
||||||
- speaker: "Eric Rakestraw"
|
- speaker: "Example Speaker"
|
||||||
match:
|
match:
|
||||||
- "Eric_Rakestraw"
|
- "Example_Speaker"
|
||||||
- "Eric"
|
- "Example"
|
||||||
|
|||||||
1
go.mod
1
go.mod
@@ -7,6 +7,7 @@ require (
|
|||||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.16
|
github.com/aws/aws-sdk-go-v2/credentials v1.19.16
|
||||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0
|
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0
|
||||||
github.com/aws/smithy-go v1.25.1
|
github.com/aws/smithy-go v1.25.1
|
||||||
|
golang.org/x/sys v0.47.0
|
||||||
gopkg.in/yaml.v3 v3.0.1
|
gopkg.in/yaml.v3 v3.0.1
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
2
go.sum
2
go.sum
@@ -34,6 +34,8 @@ github.com/aws/aws-sdk-go-v2/service/sts v1.42.1 h1:F/M5Y9I3nwr2IEpshZgh1GeHpOIt
|
|||||||
github.com/aws/aws-sdk-go-v2/service/sts v1.42.1/go.mod h1:mTNxImtovCOEEuD65mKW7DCsL+2gjEH+RPEAexAzAio=
|
github.com/aws/aws-sdk-go-v2/service/sts v1.42.1/go.mod h1:mTNxImtovCOEEuD65mKW7DCsL+2gjEH+RPEAexAzAio=
|
||||||
github.com/aws/smithy-go v1.25.1 h1:J8ERsGSU7d+aCmdQur5Txg6bVoYelvQJgtZehD12GkI=
|
github.com/aws/smithy-go v1.25.1 h1:J8ERsGSU7d+aCmdQur5Txg6bVoYelvQJgtZehD12GkI=
|
||||||
github.com/aws/smithy-go v1.25.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
|
github.com/aws/smithy-go v1.25.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
|
||||||
|
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||||
|
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
|
|||||||
22
internal/adapters/notarius/fake.go
Normal file
22
internal/adapters/notarius/fake.go
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
package notarius
|
||||||
|
|
||||||
|
import "context"
|
||||||
|
|
||||||
|
// FakeRunner is a configurable in-memory runner for stage tests.
|
||||||
|
type FakeRunner struct {
|
||||||
|
Requests []RunRequest
|
||||||
|
Result RunResult
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run records the request and returns the configured result or error.
|
||||||
|
func (f *FakeRunner) Run(ctx context.Context, req RunRequest) (RunResult, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return RunResult{}, err
|
||||||
|
}
|
||||||
|
f.Requests = append(f.Requests, req)
|
||||||
|
if f.Err != nil {
|
||||||
|
return RunResult{}, f.Err
|
||||||
|
}
|
||||||
|
return f.Result, nil
|
||||||
|
}
|
||||||
108
internal/adapters/notarius/runner.go
Normal file
108
internal/adapters/notarius/runner.go
Normal file
@@ -0,0 +1,108 @@
|
|||||||
|
// Package notarius declares the adapter contract for Notarius CLI invocations.
|
||||||
|
package notarius
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const ReceiptSchemaVersion = "notarius.run-result.v1"
|
||||||
|
|
||||||
|
// Runner is the adapter boundary for a complete Notarius pipeline invocation.
|
||||||
|
type Runner interface {
|
||||||
|
Run(ctx context.Context, req RunRequest) (RunResult, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// RunRequest contains the resolved inputs and diagnostic destinations for one invocation.
|
||||||
|
type RunRequest struct {
|
||||||
|
Binary string
|
||||||
|
ConfigPath string
|
||||||
|
PipelineID string
|
||||||
|
InputPath string
|
||||||
|
OutputRoot string
|
||||||
|
WorkingDirectory string
|
||||||
|
ReceiptPath string
|
||||||
|
LogPath string
|
||||||
|
Timeout time.Duration
|
||||||
|
}
|
||||||
|
|
||||||
|
// Receipt is the transport-neutral successful run receipt.
|
||||||
|
type Receipt struct {
|
||||||
|
SchemaVersion string
|
||||||
|
RunID string
|
||||||
|
PipelineID string
|
||||||
|
OutputDirectory string
|
||||||
|
IndexFile string
|
||||||
|
NormalizedOutputCount int
|
||||||
|
RejectedOutputCount int
|
||||||
|
WarningCount int
|
||||||
|
ValidationStatus string
|
||||||
|
DebugDirectory string
|
||||||
|
}
|
||||||
|
|
||||||
|
// LaneDescriptor identifies one normalized lane payload discovered through the index.
|
||||||
|
type LaneDescriptor struct {
|
||||||
|
LaneID string
|
||||||
|
File string
|
||||||
|
Path string
|
||||||
|
MediaType string
|
||||||
|
ModuleKey string
|
||||||
|
SchemaID string
|
||||||
|
SchemaName string
|
||||||
|
SchemaVersion string
|
||||||
|
}
|
||||||
|
|
||||||
|
// PipelineDescriptor identifies a pipeline-wide artifact discovered through the index.
|
||||||
|
type PipelineDescriptor struct {
|
||||||
|
ArtifactKind string
|
||||||
|
File string
|
||||||
|
Path string
|
||||||
|
MediaType string
|
||||||
|
SchemaID string
|
||||||
|
SchemaName string
|
||||||
|
SchemaVersion string
|
||||||
|
}
|
||||||
|
|
||||||
|
// Index describes the validated bundle-management and artifact paths.
|
||||||
|
type Index struct {
|
||||||
|
Path string
|
||||||
|
ManifestFile string
|
||||||
|
ManifestPath string
|
||||||
|
RejectedFile string
|
||||||
|
RejectedPath string
|
||||||
|
WarningsFile string
|
||||||
|
WarningsPath string
|
||||||
|
Lanes []LaneDescriptor
|
||||||
|
ChunkMap *PipelineDescriptor
|
||||||
|
EvidenceContext *PipelineDescriptor
|
||||||
|
}
|
||||||
|
|
||||||
|
// RejectionSummary retains structured rejection identity without free-form messages.
|
||||||
|
type RejectionSummary struct {
|
||||||
|
Stage string
|
||||||
|
StepID string
|
||||||
|
LaneID string
|
||||||
|
ModuleKey string
|
||||||
|
ChunkID string
|
||||||
|
ValidatorName string
|
||||||
|
ReasonCode string
|
||||||
|
}
|
||||||
|
|
||||||
|
// WarningSummary retains structured warning identity without free-form messages.
|
||||||
|
type WarningSummary struct {
|
||||||
|
Scope string
|
||||||
|
ReasonCode string
|
||||||
|
}
|
||||||
|
|
||||||
|
// RunResult describes a successfully decoded and validated Notarius bundle.
|
||||||
|
type RunResult struct {
|
||||||
|
Receipt Receipt
|
||||||
|
Index Index
|
||||||
|
BundleRoot string
|
||||||
|
ReceiptPath string
|
||||||
|
LogPath string
|
||||||
|
ExitCode int
|
||||||
|
Duration time.Duration
|
||||||
|
Rejections []RejectionSummary
|
||||||
|
Warnings []WarningSummary
|
||||||
|
}
|
||||||
524
internal/adapters/notarius/subprocess.go
Normal file
524
internal/adapters/notarius/subprocess.go
Normal file
@@ -0,0 +1,524 @@
|
|||||||
|
package notarius
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/subprocess"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/pathsafe"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
maxReceiptBytes = 1 << 20
|
||||||
|
maxIndexBytes = 4 << 20
|
||||||
|
maxSummaryBytes = 4 << 20
|
||||||
|
canonicalIndexFile = "index.json"
|
||||||
|
canonicalManifestFile = "manifest.json"
|
||||||
|
canonicalRejectedFile = "rejected.json"
|
||||||
|
canonicalWarningsFile = "warnings.json"
|
||||||
|
)
|
||||||
|
|
||||||
|
type subprocessRun func(context.Context, subprocess.RunRequest) (subprocess.RunResult, error)
|
||||||
|
|
||||||
|
// SubprocessRunner invokes Notarius through its public CLI.
|
||||||
|
type SubprocessRunner struct {
|
||||||
|
run subprocessRun
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewSubprocessRunner constructs a production Notarius subprocess runner.
|
||||||
|
func NewSubprocessRunner() *SubprocessRunner {
|
||||||
|
return &SubprocessRunner{run: subprocess.Run}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run executes a complete Notarius pipeline and discovers its published bundle.
|
||||||
|
func (r *SubprocessRunner) Run(ctx context.Context, req RunRequest) (RunResult, error) {
|
||||||
|
if r == nil || r.run == nil {
|
||||||
|
return RunResult{}, fmt.Errorf("notarius subprocess runner is nil")
|
||||||
|
}
|
||||||
|
if err := validateRunRequest(req); err != nil {
|
||||||
|
return RunResult{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
args := []string{
|
||||||
|
"run", req.PipelineID,
|
||||||
|
"--config", req.ConfigPath,
|
||||||
|
"--input", req.InputPath,
|
||||||
|
"--output-dir", req.OutputRoot,
|
||||||
|
"--json",
|
||||||
|
}
|
||||||
|
processResult, err := r.run(ctx, subprocess.RunRequest{
|
||||||
|
Executable: req.Binary,
|
||||||
|
Args: args,
|
||||||
|
WorkingDir: req.WorkingDirectory,
|
||||||
|
Timeout: req.Timeout,
|
||||||
|
StdoutLogPath: req.ReceiptPath,
|
||||||
|
StderrLogPath: req.LogPath,
|
||||||
|
})
|
||||||
|
baseResult := RunResult{
|
||||||
|
ReceiptPath: req.ReceiptPath,
|
||||||
|
LogPath: req.LogPath,
|
||||||
|
ExitCode: processResult.ExitCode,
|
||||||
|
Duration: processResult.Duration,
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return baseResult, fmt.Errorf("run notarius pipeline %q: %w", req.PipelineID, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
receipt, err := loadReceipt(req.ReceiptPath, req.PipelineID)
|
||||||
|
if err != nil {
|
||||||
|
return baseResult, err
|
||||||
|
}
|
||||||
|
bundleRoot, err := validateBundleRoot(req.OutputRoot, receipt.OutputDirectory)
|
||||||
|
if err != nil {
|
||||||
|
return baseResult, err
|
||||||
|
}
|
||||||
|
indexPath, err := resolveRegularFile(bundleRoot, receipt.IndexFile)
|
||||||
|
if err != nil {
|
||||||
|
return baseResult, fmt.Errorf("resolve receipt index file: %w", err)
|
||||||
|
}
|
||||||
|
index, err := loadIndex(bundleRoot, indexPath)
|
||||||
|
if err != nil {
|
||||||
|
return baseResult, err
|
||||||
|
}
|
||||||
|
rejections, err := loadRejections(index.RejectedPath)
|
||||||
|
if err != nil {
|
||||||
|
return baseResult, err
|
||||||
|
}
|
||||||
|
warnings, err := loadWarnings(index.WarningsPath)
|
||||||
|
if err != nil {
|
||||||
|
return baseResult, err
|
||||||
|
}
|
||||||
|
|
||||||
|
baseResult.Receipt = receipt
|
||||||
|
baseResult.Index = index
|
||||||
|
baseResult.BundleRoot = bundleRoot
|
||||||
|
baseResult.Rejections = rejections
|
||||||
|
baseResult.Warnings = warnings
|
||||||
|
return baseResult, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateRunRequest(req RunRequest) error {
|
||||||
|
if strings.TrimSpace(req.Binary) == "" {
|
||||||
|
return fmt.Errorf("notarius binary is required")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(req.PipelineID) == "" {
|
||||||
|
return fmt.Errorf("notarius pipeline id is required")
|
||||||
|
}
|
||||||
|
if req.Timeout <= 0 {
|
||||||
|
return fmt.Errorf("notarius timeout must be positive")
|
||||||
|
}
|
||||||
|
for label, path := range map[string]string{
|
||||||
|
"config": req.ConfigPath,
|
||||||
|
"input": req.InputPath,
|
||||||
|
"output root": req.OutputRoot,
|
||||||
|
"working directory": req.WorkingDirectory,
|
||||||
|
"receipt": req.ReceiptPath,
|
||||||
|
"log": req.LogPath,
|
||||||
|
} {
|
||||||
|
if strings.TrimSpace(path) == "" {
|
||||||
|
return fmt.Errorf("notarius %s path is required", label)
|
||||||
|
}
|
||||||
|
if !filepath.IsAbs(path) {
|
||||||
|
return fmt.Errorf("notarius %s path must be absolute", label)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if filepath.Clean(req.ReceiptPath) == filepath.Clean(req.LogPath) {
|
||||||
|
return fmt.Errorf("notarius receipt and log paths must be different")
|
||||||
|
}
|
||||||
|
if err := requireRegularFile(req.ConfigPath); err != nil {
|
||||||
|
return fmt.Errorf("validate notarius config path: %w", err)
|
||||||
|
}
|
||||||
|
if err := requireRegularFile(req.InputPath); err != nil {
|
||||||
|
return fmt.Errorf("validate notarius input path: %w", err)
|
||||||
|
}
|
||||||
|
if err := requireDirectory(req.OutputRoot); err != nil {
|
||||||
|
return fmt.Errorf("validate notarius output root: %w", err)
|
||||||
|
}
|
||||||
|
if err := requireDirectory(req.WorkingDirectory); err != nil {
|
||||||
|
return fmt.Errorf("validate notarius working directory: %w", err)
|
||||||
|
}
|
||||||
|
if err := validateLogDestination(req.ReceiptPath); err != nil {
|
||||||
|
return fmt.Errorf("validate notarius receipt path: %w", err)
|
||||||
|
}
|
||||||
|
if err := validateLogDestination(req.LogPath); err != nil {
|
||||||
|
return fmt.Errorf("validate notarius log path: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type receiptDocument struct {
|
||||||
|
SchemaVersion string `json:"schema_version"`
|
||||||
|
RunID string `json:"run_id"`
|
||||||
|
PipelineID string `json:"pipeline_id"`
|
||||||
|
OutputDirectory string `json:"output_directory"`
|
||||||
|
IndexFile string `json:"index_file"`
|
||||||
|
NormalizedOutputCount *int `json:"normalized_output_count"`
|
||||||
|
RejectedOutputCount *int `json:"rejected_output_count"`
|
||||||
|
WarningCount *int `json:"warning_count"`
|
||||||
|
ValidationStatus string `json:"validation_status"`
|
||||||
|
DebugDirectory string `json:"debug_directory"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadReceipt(path, pipelineID string) (Receipt, error) {
|
||||||
|
var document receiptDocument
|
||||||
|
if err := decodeBoundedJSON(path, maxReceiptBytes, &document); err != nil {
|
||||||
|
return Receipt{}, fmt.Errorf("decode notarius receipt: %w", err)
|
||||||
|
}
|
||||||
|
if document.SchemaVersion != ReceiptSchemaVersion {
|
||||||
|
return Receipt{}, fmt.Errorf("unsupported notarius receipt schema version %q", document.SchemaVersion)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(document.RunID) == "" || strings.TrimSpace(document.PipelineID) == "" ||
|
||||||
|
strings.TrimSpace(document.OutputDirectory) == "" || strings.TrimSpace(document.ValidationStatus) == "" ||
|
||||||
|
document.NormalizedOutputCount == nil ||
|
||||||
|
document.RejectedOutputCount == nil || document.WarningCount == nil {
|
||||||
|
return Receipt{}, fmt.Errorf("notarius receipt is missing required fields")
|
||||||
|
}
|
||||||
|
if document.IndexFile != canonicalIndexFile {
|
||||||
|
return Receipt{}, fmt.Errorf("notarius receipt index_file %q is incompatible; want %q", document.IndexFile, canonicalIndexFile)
|
||||||
|
}
|
||||||
|
if document.PipelineID != pipelineID {
|
||||||
|
return Receipt{}, fmt.Errorf("notarius receipt pipeline id %q does not match requested pipeline %q", document.PipelineID, pipelineID)
|
||||||
|
}
|
||||||
|
if *document.NormalizedOutputCount < 0 || *document.RejectedOutputCount < 0 || *document.WarningCount < 0 {
|
||||||
|
return Receipt{}, fmt.Errorf("notarius receipt counts must be non-negative")
|
||||||
|
}
|
||||||
|
if !filepath.IsAbs(document.OutputDirectory) {
|
||||||
|
return Receipt{}, fmt.Errorf("notarius receipt output directory must be absolute")
|
||||||
|
}
|
||||||
|
if document.DebugDirectory != "" && !filepath.IsAbs(document.DebugDirectory) {
|
||||||
|
return Receipt{}, fmt.Errorf("notarius receipt debug directory must be absolute when present")
|
||||||
|
}
|
||||||
|
return Receipt{
|
||||||
|
SchemaVersion: document.SchemaVersion,
|
||||||
|
RunID: document.RunID,
|
||||||
|
PipelineID: document.PipelineID,
|
||||||
|
OutputDirectory: filepath.Clean(document.OutputDirectory),
|
||||||
|
IndexFile: document.IndexFile,
|
||||||
|
NormalizedOutputCount: *document.NormalizedOutputCount,
|
||||||
|
RejectedOutputCount: *document.RejectedOutputCount,
|
||||||
|
WarningCount: *document.WarningCount,
|
||||||
|
ValidationStatus: document.ValidationStatus,
|
||||||
|
DebugDirectory: document.DebugDirectory,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type indexDocument struct {
|
||||||
|
ManifestFile string `json:"manifest_file"`
|
||||||
|
OutputFiles *[]laneDocument `json:"output_files"`
|
||||||
|
RejectedFile string `json:"rejected_file"`
|
||||||
|
WarningsFile string `json:"warnings_file"`
|
||||||
|
ChunkMap *pipelineDocument `json:"chunk_map"`
|
||||||
|
EvidenceContext *pipelineDocument `json:"evidence_context"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type laneDocument struct {
|
||||||
|
LaneID string `json:"lane_id"`
|
||||||
|
File string `json:"file"`
|
||||||
|
MediaType string `json:"media_type"`
|
||||||
|
ModuleKey string `json:"module_key"`
|
||||||
|
SchemaID string `json:"schema_id"`
|
||||||
|
SchemaName string `json:"schema_name"`
|
||||||
|
SchemaVersion string `json:"schema_version"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type pipelineDocument struct {
|
||||||
|
ArtifactKind string `json:"artifact_kind"`
|
||||||
|
File string `json:"file"`
|
||||||
|
MediaType string `json:"media_type"`
|
||||||
|
SchemaID string `json:"schema_id"`
|
||||||
|
SchemaName string `json:"schema_name"`
|
||||||
|
SchemaVersion string `json:"schema_version"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadIndex(bundleRoot, indexPath string) (Index, error) {
|
||||||
|
var document indexDocument
|
||||||
|
if err := decodeBoundedJSON(indexPath, maxIndexBytes, &document); err != nil {
|
||||||
|
return Index{}, fmt.Errorf("decode notarius index: %w", err)
|
||||||
|
}
|
||||||
|
for _, field := range []struct {
|
||||||
|
name string
|
||||||
|
got string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "manifest_file", got: document.ManifestFile, want: canonicalManifestFile},
|
||||||
|
{name: "rejected_file", got: document.RejectedFile, want: canonicalRejectedFile},
|
||||||
|
{name: "warnings_file", got: document.WarningsFile, want: canonicalWarningsFile},
|
||||||
|
} {
|
||||||
|
if field.got != field.want {
|
||||||
|
return Index{}, fmt.Errorf("notarius index %s %q is incompatible; want %q", field.name, field.got, field.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if document.OutputFiles == nil {
|
||||||
|
return Index{}, fmt.Errorf("notarius index is missing required output_files")
|
||||||
|
}
|
||||||
|
|
||||||
|
index := Index{
|
||||||
|
Path: indexPath,
|
||||||
|
ManifestFile: document.ManifestFile,
|
||||||
|
RejectedFile: document.RejectedFile,
|
||||||
|
WarningsFile: document.WarningsFile,
|
||||||
|
}
|
||||||
|
var err error
|
||||||
|
if index.ManifestPath, err = resolveRegularFile(bundleRoot, index.ManifestFile); err != nil {
|
||||||
|
return Index{}, fmt.Errorf("resolve notarius manifest file: %w", err)
|
||||||
|
}
|
||||||
|
if index.RejectedPath, err = resolveRegularFile(bundleRoot, index.RejectedFile); err != nil {
|
||||||
|
return Index{}, fmt.Errorf("resolve notarius rejection file: %w", err)
|
||||||
|
}
|
||||||
|
if index.WarningsPath, err = resolveRegularFile(bundleRoot, index.WarningsFile); err != nil {
|
||||||
|
return Index{}, fmt.Errorf("resolve notarius warning file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
seenLanes := make(map[string]struct{}, len(*document.OutputFiles))
|
||||||
|
for _, lane := range *document.OutputFiles {
|
||||||
|
if strings.TrimSpace(lane.LaneID) == "" || strings.TrimSpace(lane.File) == "" {
|
||||||
|
return Index{}, fmt.Errorf("notarius lane descriptors require lane_id and file")
|
||||||
|
}
|
||||||
|
if _, exists := seenLanes[lane.LaneID]; exists {
|
||||||
|
return Index{}, fmt.Errorf("notarius index contains duplicate lane id %q", lane.LaneID)
|
||||||
|
}
|
||||||
|
seenLanes[lane.LaneID] = struct{}{}
|
||||||
|
path, err := resolveRegularFile(bundleRoot, lane.File)
|
||||||
|
if err != nil {
|
||||||
|
return Index{}, fmt.Errorf("resolve notarius lane %q file: %w", lane.LaneID, err)
|
||||||
|
}
|
||||||
|
index.Lanes = append(index.Lanes, LaneDescriptor{
|
||||||
|
LaneID: lane.LaneID, File: lane.File, Path: path, MediaType: lane.MediaType,
|
||||||
|
ModuleKey: lane.ModuleKey, SchemaID: lane.SchemaID, SchemaName: lane.SchemaName,
|
||||||
|
SchemaVersion: lane.SchemaVersion,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if document.ChunkMap != nil {
|
||||||
|
index.ChunkMap, err = resolvePipelineDescriptor(bundleRoot, "chunk_map", *document.ChunkMap)
|
||||||
|
if err != nil {
|
||||||
|
return Index{}, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if document.EvidenceContext != nil {
|
||||||
|
index.EvidenceContext, err = resolvePipelineDescriptor(bundleRoot, "evidence_context", *document.EvidenceContext)
|
||||||
|
if err != nil {
|
||||||
|
return Index{}, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return index, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolvePipelineDescriptor(bundleRoot, label string, document pipelineDocument) (*PipelineDescriptor, error) {
|
||||||
|
if strings.TrimSpace(document.ArtifactKind) == "" || strings.TrimSpace(document.File) == "" ||
|
||||||
|
strings.TrimSpace(document.MediaType) == "" || strings.TrimSpace(document.SchemaID) == "" ||
|
||||||
|
strings.TrimSpace(document.SchemaName) == "" || strings.TrimSpace(document.SchemaVersion) == "" {
|
||||||
|
return nil, fmt.Errorf("notarius %s descriptor is missing required fields", label)
|
||||||
|
}
|
||||||
|
path, err := resolveRegularFile(bundleRoot, document.File)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("resolve notarius %s file: %w", label, err)
|
||||||
|
}
|
||||||
|
return &PipelineDescriptor{
|
||||||
|
ArtifactKind: document.ArtifactKind, File: document.File, Path: path,
|
||||||
|
MediaType: document.MediaType, SchemaID: document.SchemaID,
|
||||||
|
SchemaName: document.SchemaName, SchemaVersion: document.SchemaVersion,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type rejectionDocument struct {
|
||||||
|
Rejected *[]struct {
|
||||||
|
Stage string `json:"stage"`
|
||||||
|
StepID string `json:"step_id"`
|
||||||
|
LaneID string `json:"lane_id"`
|
||||||
|
ModuleKey string `json:"module_key"`
|
||||||
|
ChunkID string `json:"chunk_id"`
|
||||||
|
ValidatorName string `json:"validator_name"`
|
||||||
|
ReasonCode string `json:"reason_code"`
|
||||||
|
Message string `json:"message"`
|
||||||
|
} `json:"rejected"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadRejections(path string) ([]RejectionSummary, error) {
|
||||||
|
var document rejectionDocument
|
||||||
|
if err := decodeBoundedJSON(path, maxSummaryBytes, &document); err != nil {
|
||||||
|
return nil, fmt.Errorf("decode notarius rejections: %w", err)
|
||||||
|
}
|
||||||
|
if document.Rejected == nil {
|
||||||
|
return nil, fmt.Errorf("notarius rejection document is missing rejected array")
|
||||||
|
}
|
||||||
|
summaries := make([]RejectionSummary, 0, len(*document.Rejected))
|
||||||
|
for _, item := range *document.Rejected {
|
||||||
|
if strings.TrimSpace(item.Stage) == "" || strings.TrimSpace(item.Message) == "" {
|
||||||
|
return nil, fmt.Errorf("notarius rejection entries require stage and message")
|
||||||
|
}
|
||||||
|
summaries = append(summaries, RejectionSummary{
|
||||||
|
Stage: item.Stage, StepID: item.StepID, LaneID: item.LaneID,
|
||||||
|
ModuleKey: item.ModuleKey, ChunkID: item.ChunkID,
|
||||||
|
ValidatorName: item.ValidatorName, ReasonCode: item.ReasonCode,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return summaries, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type warningDocument struct {
|
||||||
|
Warnings *[]struct {
|
||||||
|
Scope string `json:"scope"`
|
||||||
|
ReasonCode string `json:"reason_code"`
|
||||||
|
Message string `json:"message"`
|
||||||
|
} `json:"warnings"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadWarnings(path string) ([]WarningSummary, error) {
|
||||||
|
var document warningDocument
|
||||||
|
if err := decodeBoundedJSON(path, maxSummaryBytes, &document); err != nil {
|
||||||
|
return nil, fmt.Errorf("decode notarius warnings: %w", err)
|
||||||
|
}
|
||||||
|
if document.Warnings == nil {
|
||||||
|
return nil, fmt.Errorf("notarius warning document is missing warnings array")
|
||||||
|
}
|
||||||
|
summaries := make([]WarningSummary, 0, len(*document.Warnings))
|
||||||
|
for _, item := range *document.Warnings {
|
||||||
|
if strings.TrimSpace(item.ReasonCode) == "" || strings.TrimSpace(item.Message) == "" {
|
||||||
|
return nil, fmt.Errorf("notarius warning entries require reason_code and message")
|
||||||
|
}
|
||||||
|
summaries = append(summaries, WarningSummary{Scope: item.Scope, ReasonCode: item.ReasonCode})
|
||||||
|
}
|
||||||
|
return summaries, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeBoundedJSON(path string, limit int64, destination any) error {
|
||||||
|
inspected, err := os.Lstat(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if inspected.Mode()&os.ModeSymlink != 0 || !inspected.Mode().IsRegular() {
|
||||||
|
return fmt.Errorf("path %q must be a regular file without symlinks", path)
|
||||||
|
}
|
||||||
|
file, err := os.Open(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer func() { _ = file.Close() }()
|
||||||
|
opened, err := file.Stat()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !opened.Mode().IsRegular() || !os.SameFile(inspected, opened) {
|
||||||
|
return fmt.Errorf("file %q changed before it could be read", path)
|
||||||
|
}
|
||||||
|
reader := io.LimitReader(file, limit+1)
|
||||||
|
data, err := io.ReadAll(reader)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if int64(len(data)) > limit {
|
||||||
|
return fmt.Errorf("file %q exceeds %d-byte limit", path, limit)
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(data, destination); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateBundleRoot(outputRoot, bundleRoot string) (string, error) {
|
||||||
|
root := filepath.Clean(outputRoot)
|
||||||
|
bundle := filepath.Clean(bundleRoot)
|
||||||
|
relative, err := filepath.Rel(root, bundle)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("compare notarius output paths: %w", err)
|
||||||
|
}
|
||||||
|
if relative == "." || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
|
||||||
|
return "", fmt.Errorf("notarius output directory %q is not beneath output root %q", bundleRoot, outputRoot)
|
||||||
|
}
|
||||||
|
if err := requireDirectoryTree(root, relative); err != nil {
|
||||||
|
return "", fmt.Errorf("validate notarius output directory: %w", err)
|
||||||
|
}
|
||||||
|
return bundle, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveRegularFile(root, logicalPath string) (string, error) {
|
||||||
|
resolved, err := pathsafe.JoinSlashRelativeUnderRoot(root, logicalPath)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
relative, err := filepath.Rel(root, resolved)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
if err := requireRegularFileTree(root, relative); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return resolved, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireDirectoryTree(root, relative string) error {
|
||||||
|
if err := requireDirectory(root); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
current := root
|
||||||
|
for _, component := range strings.Split(relative, string(filepath.Separator)) {
|
||||||
|
current = filepath.Join(current, component)
|
||||||
|
if err := requireDirectory(current); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireRegularFileTree(root, relative string) error {
|
||||||
|
components := strings.Split(relative, string(filepath.Separator))
|
||||||
|
if len(components) == 0 {
|
||||||
|
return fmt.Errorf("regular file path is required")
|
||||||
|
}
|
||||||
|
if err := requireDirectory(root); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
current := root
|
||||||
|
for _, component := range components[:len(components)-1] {
|
||||||
|
current = filepath.Join(current, component)
|
||||||
|
if err := requireDirectory(current); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return requireRegularFile(filepath.Join(current, components[len(components)-1]))
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireDirectory(path string) error {
|
||||||
|
info, err := os.Lstat(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 || !info.IsDir() {
|
||||||
|
return fmt.Errorf("path %q must be a directory without symlinks", path)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireRegularFile(path string) error {
|
||||||
|
info, err := os.Lstat(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 || !info.Mode().IsRegular() {
|
||||||
|
return fmt.Errorf("path %q must be a regular file without symlinks", path)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateLogDestination(path string) error {
|
||||||
|
if err := requireDirectory(filepath.Dir(path)); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
info, err := os.Lstat(path)
|
||||||
|
if errors.Is(err, os.ErrNotExist) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 || !info.Mode().IsRegular() {
|
||||||
|
return fmt.Errorf("path %q must be absent or a regular file without symlinks", path)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
572
internal/adapters/notarius/subprocess_test.go
Normal file
572
internal/adapters/notarius/subprocess_test.go
Normal file
@@ -0,0 +1,572 @@
|
|||||||
|
package notarius
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
sharedsubprocess "gitea.maximumdirect.net/eric/narratio/internal/adapters/subprocess"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestSubprocessRunnerBuildsExactInvocationAndDiscoversBundle(t *testing.T) {
|
||||||
|
req := validRunRequest(t)
|
||||||
|
var captured sharedsubprocess.RunRequest
|
||||||
|
runner := &SubprocessRunner{run: func(_ context.Context, processReq sharedsubprocess.RunRequest) (sharedsubprocess.RunResult, error) {
|
||||||
|
captured = processReq
|
||||||
|
writeValidBundleAndReceipt(t, req, true)
|
||||||
|
return sharedsubprocess.RunResult{ExitCode: 0, Duration: 2 * time.Second}, nil
|
||||||
|
}}
|
||||||
|
|
||||||
|
result, err := runner.Run(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
wantArgs := []string{
|
||||||
|
"run", "dnd-session", "--config", req.ConfigPath, "--input", req.InputPath,
|
||||||
|
"--output-dir", req.OutputRoot, "--json",
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(captured.Args, wantArgs) {
|
||||||
|
t.Fatalf("subprocess args = %#v, want %#v", captured.Args, wantArgs)
|
||||||
|
}
|
||||||
|
if captured.Executable != req.Binary || captured.WorkingDir != req.WorkingDirectory || captured.Timeout != req.Timeout {
|
||||||
|
t.Fatalf("subprocess request = %#v", captured)
|
||||||
|
}
|
||||||
|
if captured.StdoutLogPath != req.ReceiptPath || captured.StderrLogPath != req.LogPath {
|
||||||
|
t.Fatalf("stream paths = stdout %q stderr %q", captured.StdoutLogPath, captured.StderrLogPath)
|
||||||
|
}
|
||||||
|
if captured.EnvOverrides != nil {
|
||||||
|
t.Fatalf("environment overrides = %#v, want inherited environment only", captured.EnvOverrides)
|
||||||
|
}
|
||||||
|
for _, arg := range captured.Args {
|
||||||
|
if arg == "--session-id" {
|
||||||
|
t.Fatal("subprocess args unexpectedly contain --session-id")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if result.Receipt.SchemaVersion != ReceiptSchemaVersion || result.Receipt.RunID != "notarius-run-1" {
|
||||||
|
t.Fatalf("receipt = %#v", result.Receipt)
|
||||||
|
}
|
||||||
|
if len(result.Index.Lanes) != 1 || result.Index.Lanes[0].LaneID != "npc-registry" {
|
||||||
|
t.Fatalf("lanes = %#v", result.Index.Lanes)
|
||||||
|
}
|
||||||
|
if result.Index.ChunkMap == nil || result.Index.ChunkMap.ArtifactKind != "chunk_map" {
|
||||||
|
t.Fatalf("chunk map = %#v", result.Index.ChunkMap)
|
||||||
|
}
|
||||||
|
if result.Index.EvidenceContext == nil || result.Index.EvidenceContext.ArtifactKind != "evidence_context" {
|
||||||
|
t.Fatalf("evidence context = %#v", result.Index.EvidenceContext)
|
||||||
|
}
|
||||||
|
if len(result.Rejections) != 1 || result.Rejections[0].LaneID != "spells" || result.Rejections[0].ReasonCode != "invalid_spell" {
|
||||||
|
t.Fatalf("rejections = %#v", result.Rejections)
|
||||||
|
}
|
||||||
|
if len(result.Warnings) != 1 || result.Warnings[0].Scope != "lane:npc-registry" || result.Warnings[0].ReasonCode != "normalized_name" {
|
||||||
|
t.Fatalf("warnings = %#v", result.Warnings)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerInheritsEnvironmentAndSeparatesStreams(t *testing.T) {
|
||||||
|
req := validRunRequest(t)
|
||||||
|
writeValidBundleAndReceipt(t, req, false)
|
||||||
|
receiptFixture := req.ReceiptPath + ".fixture"
|
||||||
|
data, err := os.ReadFile(req.ReceiptPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(receipt) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(receiptFixture, data, 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(receipt fixture) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.Remove(req.ReceiptPath); err != nil {
|
||||||
|
t.Fatalf("Remove(receipt) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
captureDir := filepath.Join(filepath.Dir(req.ReceiptPath), "capture")
|
||||||
|
if err := os.Mkdir(captureDir, 0o755); err != nil {
|
||||||
|
t.Fatalf("Mkdir(capture) error = %v", err)
|
||||||
|
}
|
||||||
|
script := writeShellScript(t, `#!/bin/sh
|
||||||
|
pwd > "$NOTARIUS_CAPTURE_DIR/working-directory"
|
||||||
|
printf '%s' "$NOTARIUS_INHERITED_VALUE" > "$NOTARIUS_CAPTURE_DIR/environment"
|
||||||
|
printf 'diagnostic stream\n' >&2
|
||||||
|
cat "$NOTARIUS_RECEIPT_FIXTURE"
|
||||||
|
`)
|
||||||
|
req.Binary = script
|
||||||
|
t.Setenv("NOTARIUS_CAPTURE_DIR", captureDir)
|
||||||
|
t.Setenv("NOTARIUS_INHERITED_VALUE", "inherited-value")
|
||||||
|
t.Setenv("NOTARIUS_RECEIPT_FIXTURE", receiptFixture)
|
||||||
|
|
||||||
|
if _, err := NewSubprocessRunner().Run(context.Background(), req); err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
assertTextFile(t, filepath.Join(captureDir, "working-directory"), req.WorkingDirectory+"\n")
|
||||||
|
assertTextFile(t, filepath.Join(captureDir, "environment"), "inherited-value")
|
||||||
|
assertTextFile(t, req.LogPath, "diagnostic stream\n")
|
||||||
|
receiptBytes, err := os.ReadFile(req.ReceiptPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(receipt) error = %v", err)
|
||||||
|
}
|
||||||
|
if strings.Contains(string(receiptBytes), "diagnostic stream") {
|
||||||
|
t.Fatal("receipt contains stderr output")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerReturnsProcessFailuresWithoutParsingStdout(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
scriptBody string
|
||||||
|
timeout time.Duration
|
||||||
|
cancel bool
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "nonzero", scriptBody: "printf '{malformed receipt'; printf 'failed\\n' >&2; exit 7\n", timeout: time.Second, want: "exit code 7"},
|
||||||
|
{name: "timeout", scriptBody: "sleep 5\n", timeout: 20 * time.Millisecond, want: "timed out"},
|
||||||
|
{name: "cancellation", scriptBody: "sleep 5\n", timeout: time.Second, cancel: true, want: "canceled"},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
req := validRunRequest(t)
|
||||||
|
req.Binary = writeShellScript(t, "#!/bin/sh\n"+test.scriptBody)
|
||||||
|
req.Timeout = test.timeout
|
||||||
|
ctx := context.Background()
|
||||||
|
if test.cancel {
|
||||||
|
cancelCtx, cancel := context.WithCancel(ctx)
|
||||||
|
ctx = cancelCtx
|
||||||
|
time.AfterFunc(20*time.Millisecond, cancel)
|
||||||
|
}
|
||||||
|
_, err := NewSubprocessRunner().Run(ctx, req)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||||
|
t.Fatalf("Run() error = %v, want fragment %q", err, test.want)
|
||||||
|
}
|
||||||
|
if strings.Contains(err.Error(), "decode notarius receipt") {
|
||||||
|
t.Fatalf("Run() parsed stdout after process failure: %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerReturnsSharedSubprocessErrorWithoutReadingReceipt(t *testing.T) {
|
||||||
|
req := validRunRequest(t)
|
||||||
|
if err := os.WriteFile(req.ReceiptPath, []byte("not json"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(receipt) error = %v", err)
|
||||||
|
}
|
||||||
|
wantErr := errors.New("process failed")
|
||||||
|
runner := &SubprocessRunner{run: func(context.Context, sharedsubprocess.RunRequest) (sharedsubprocess.RunResult, error) {
|
||||||
|
return sharedsubprocess.RunResult{ExitCode: 9}, wantErr
|
||||||
|
}}
|
||||||
|
_, err := runner.Run(context.Background(), req)
|
||||||
|
if !errors.Is(err, wantErr) {
|
||||||
|
t.Fatalf("Run() error = %v, want wrapped process error", err)
|
||||||
|
}
|
||||||
|
if strings.Contains(err.Error(), "decode") {
|
||||||
|
t.Fatalf("Run() parsed receipt after failure: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadReceiptValidation(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
valid := map[string]any{
|
||||||
|
"schema_version": ReceiptSchemaVersion, "run_id": "run-1", "pipeline_id": "pipeline-1",
|
||||||
|
"output_directory": filepath.Join(root, "outputs", "run-1"), "index_file": "index.json",
|
||||||
|
"normalized_output_count": 1, "rejected_output_count": 0, "warning_count": 0,
|
||||||
|
"validation_status": "approved", "future_field": true,
|
||||||
|
}
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
mutate func(map[string]any)
|
||||||
|
raw []byte
|
||||||
|
wantOK bool
|
||||||
|
wantError string
|
||||||
|
}{
|
||||||
|
{name: "unknown fields tolerated", wantOK: true},
|
||||||
|
{name: "malformed", raw: []byte("{")},
|
||||||
|
{name: "unsupported version", mutate: func(v map[string]any) { v["schema_version"] = "notarius.run-result.v2" }},
|
||||||
|
{name: "missing field", mutate: func(v map[string]any) { delete(v, "run_id") }},
|
||||||
|
{name: "pipeline mismatch", mutate: func(v map[string]any) { v["pipeline_id"] = "other" }},
|
||||||
|
{name: "relative output", mutate: func(v map[string]any) { v["output_directory"] = "run-1" }},
|
||||||
|
{name: "negative count", mutate: func(v map[string]any) { v["warning_count"] = -1 }},
|
||||||
|
{
|
||||||
|
name: "nested index", mutate: func(v map[string]any) { v["index_file"] = "nested/index.json" },
|
||||||
|
wantError: `index_file "nested/index.json"`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "cleanable index", mutate: func(v map[string]any) { v["index_file"] = "./index.json" },
|
||||||
|
wantError: `index_file "./index.json"`,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
path := filepath.Join(root, strings.ReplaceAll(test.name, " ", "-")+".json")
|
||||||
|
values := cloneMap(valid)
|
||||||
|
if test.mutate != nil {
|
||||||
|
test.mutate(values)
|
||||||
|
}
|
||||||
|
if test.raw != nil {
|
||||||
|
if err := os.WriteFile(path, test.raw, 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile() error = %v", err)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
writeJSONFile(t, path, values)
|
||||||
|
}
|
||||||
|
_, err := loadReceipt(path, "pipeline-1")
|
||||||
|
if test.wantOK && err != nil {
|
||||||
|
t.Fatalf("loadReceipt() error = %v", err)
|
||||||
|
}
|
||||||
|
if !test.wantOK && err == nil {
|
||||||
|
t.Fatal("loadReceipt() error = nil, want validation failure")
|
||||||
|
}
|
||||||
|
if test.wantError != "" && !strings.Contains(err.Error(), test.wantError) {
|
||||||
|
t.Fatalf("loadReceipt() error = %v, want fragment %q", err, test.wantError)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
oversized := filepath.Join(root, "oversized.json")
|
||||||
|
if err := os.WriteFile(oversized, []byte(strings.Repeat("x", maxReceiptBytes+1)), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(oversized) error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := loadReceipt(oversized, "pipeline-1"); err == nil || !strings.Contains(err.Error(), "exceeds") {
|
||||||
|
t.Fatalf("loadReceipt(oversized) error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateBundleRootRejectsEscapesAndSymlinks(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
outputRoot := filepath.Join(root, "output")
|
||||||
|
if err := os.Mkdir(outputRoot, 0o755); err != nil {
|
||||||
|
t.Fatalf("Mkdir(output root) error = %v", err)
|
||||||
|
}
|
||||||
|
validBundle := filepath.Join(outputRoot, "run-1")
|
||||||
|
if err := os.Mkdir(validBundle, 0o755); err != nil {
|
||||||
|
t.Fatalf("Mkdir(bundle) error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := validateBundleRoot(outputRoot, validBundle); err != nil {
|
||||||
|
t.Fatalf("validateBundleRoot(valid) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
outside := filepath.Join(root, "output-other")
|
||||||
|
if err := os.Mkdir(outside, 0o755); err != nil {
|
||||||
|
t.Fatalf("Mkdir(outside) error = %v", err)
|
||||||
|
}
|
||||||
|
for name, candidate := range map[string]string{"equal root": outputRoot, "escape": root, "prefix confusion": outside} {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
if _, err := validateBundleRoot(outputRoot, candidate); err == nil {
|
||||||
|
t.Fatalf("validateBundleRoot(%q) error = nil", candidate)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
symlink := filepath.Join(outputRoot, "linked")
|
||||||
|
if err := os.Symlink(outside, symlink); err != nil {
|
||||||
|
t.Skipf("Symlink() unavailable: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := validateBundleRoot(outputRoot, symlink); err == nil {
|
||||||
|
t.Fatal("validateBundleRoot(symlink) error = nil")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadIndexRejectsMalformedUnsafeAndUnsupportedDocuments(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
indexValue any
|
||||||
|
prepare func(*testing.T, string)
|
||||||
|
wantError string
|
||||||
|
}{
|
||||||
|
{name: "malformed", indexValue: json.RawMessage(`{"manifest_file":`)},
|
||||||
|
{name: "unsupported output shape", indexValue: map[string]any{"manifest_file": "manifest.json", "output_files": map[string]any{}, "rejected_file": "rejected.json", "warnings_file": "warnings.json"}},
|
||||||
|
{name: "missing management path", indexValue: map[string]any{"output_files": []any{}, "rejected_file": "rejected.json", "warnings_file": "warnings.json"}},
|
||||||
|
{name: "renamed manifest", indexValue: func() any {
|
||||||
|
value := validIndexValue([]any{})
|
||||||
|
value["manifest_file"] = "metadata.json"
|
||||||
|
return value
|
||||||
|
}(), wantError: `manifest_file "metadata.json"`},
|
||||||
|
{name: "cleanable manifest", indexValue: func() any {
|
||||||
|
value := validIndexValue([]any{})
|
||||||
|
value["manifest_file"] = "./manifest.json"
|
||||||
|
return value
|
||||||
|
}(), wantError: `manifest_file "./manifest.json"`},
|
||||||
|
{name: "renamed rejections", indexValue: func() any {
|
||||||
|
value := validIndexValue([]any{})
|
||||||
|
value["rejected_file"] = "rejections.json"
|
||||||
|
return value
|
||||||
|
}(), wantError: `rejected_file "rejections.json"`},
|
||||||
|
{name: "renamed warnings", indexValue: func() any {
|
||||||
|
value := validIndexValue([]any{})
|
||||||
|
value["warnings_file"] = "diagnostics/warnings.json"
|
||||||
|
return value
|
||||||
|
}(), wantError: `warnings_file "diagnostics/warnings.json"`},
|
||||||
|
{name: "duplicate lane", indexValue: validIndexValue([]any{
|
||||||
|
map[string]any{"lane_id": "npc", "file": "lanes/npc.json"},
|
||||||
|
map[string]any{"lane_id": "npc", "file": "lanes/npc.json"},
|
||||||
|
})},
|
||||||
|
{name: "absolute logical path", indexValue: validIndexValue([]any{map[string]any{"lane_id": "npc", "file": "/tmp/npc.json"}})},
|
||||||
|
{name: "lexical traversal", indexValue: validIndexValue([]any{map[string]any{"lane_id": "npc", "file": "../outside.json"}})},
|
||||||
|
{name: "root prefix confusion", indexValue: validIndexValue([]any{map[string]any{"lane_id": "npc", "file": "../bundle-other/npc.json"}})},
|
||||||
|
{name: "file symlink", indexValue: validIndexValue([]any{map[string]any{"lane_id": "npc", "file": "lanes/npc.json"}}), prepare: func(t *testing.T, bundle string) {
|
||||||
|
if err := os.Symlink(filepath.Join(bundle, "manifest.json"), filepath.Join(bundle, "lanes", "npc.json")); err != nil {
|
||||||
|
t.Skipf("Symlink() unavailable: %v", err)
|
||||||
|
}
|
||||||
|
}},
|
||||||
|
{name: "directory symlink", indexValue: validIndexValue([]any{map[string]any{"lane_id": "npc", "file": "linked/npc.json"}}), prepare: func(t *testing.T, bundle string) {
|
||||||
|
if err := os.Symlink(filepath.Join(bundle, "lanes"), filepath.Join(bundle, "linked")); err != nil {
|
||||||
|
t.Skipf("Symlink() unavailable: %v", err)
|
||||||
|
}
|
||||||
|
}},
|
||||||
|
{name: "missing management file", indexValue: validIndexValue([]any{}), prepare: func(t *testing.T, bundle string) {
|
||||||
|
if err := os.Remove(filepath.Join(bundle, "manifest.json")); err != nil {
|
||||||
|
t.Fatalf("Remove(manifest) error = %v", err)
|
||||||
|
}
|
||||||
|
}},
|
||||||
|
{name: "incomplete pipeline descriptor", indexValue: func() any {
|
||||||
|
value := validIndexValue([]any{})
|
||||||
|
value["chunk_map"] = map[string]any{"artifact_kind": "chunk_map", "file": "chunk-map.json"}
|
||||||
|
return value
|
||||||
|
}()},
|
||||||
|
{name: "pipeline descriptor escape", indexValue: func() any {
|
||||||
|
value := validIndexValue([]any{})
|
||||||
|
value["evidence_context"] = map[string]any{
|
||||||
|
"artifact_kind": "evidence_context", "file": "../evidence.json", "media_type": "application/json",
|
||||||
|
"schema_id": "evidence", "schema_name": "Evidence", "schema_version": "v1",
|
||||||
|
}
|
||||||
|
return value
|
||||||
|
}()},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
bundle := createBundleSkeleton(t)
|
||||||
|
indexPath := filepath.Join(bundle, "index.json")
|
||||||
|
if raw, ok := test.indexValue.(json.RawMessage); ok {
|
||||||
|
if err := os.WriteFile(indexPath, raw, 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(index) error = %v", err)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
writeJSONFile(t, indexPath, test.indexValue)
|
||||||
|
}
|
||||||
|
if test.prepare != nil {
|
||||||
|
test.prepare(t, bundle)
|
||||||
|
}
|
||||||
|
if _, err := loadIndex(bundle, indexPath); err == nil {
|
||||||
|
t.Fatal("loadIndex() error = nil, want failure")
|
||||||
|
} else if test.wantError != "" && !strings.Contains(err.Error(), test.wantError) {
|
||||||
|
t.Fatalf("loadIndex() error = %v, want fragment %q", err, test.wantError)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
bundle := createBundleSkeleton(t)
|
||||||
|
oversizedIndex := filepath.Join(bundle, "index.json")
|
||||||
|
if err := os.WriteFile(oversizedIndex, []byte(strings.Repeat("x", maxIndexBytes+1)), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(oversized index) error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := loadIndex(bundle, oversizedIndex); err == nil || !strings.Contains(err.Error(), "exceeds") {
|
||||||
|
t.Fatalf("loadIndex(oversized) error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadDiagnosticSummariesValidateBoundsAndTolerateUnknownFields(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
rejectedPath := filepath.Join(root, "rejected.json")
|
||||||
|
warningsPath := filepath.Join(root, "warnings.json")
|
||||||
|
writeJSONFile(t, rejectedPath, map[string]any{"rejected": []any{map[string]any{
|
||||||
|
"stage": "validate", "lane_id": "spells", "reason_code": "invalid", "message": "do not retain this", "future": true,
|
||||||
|
}}, "future": true})
|
||||||
|
writeJSONFile(t, warningsPath, map[string]any{"warnings": []any{map[string]any{
|
||||||
|
"scope": "lane:spells", "reason_code": "bounded", "message": "do not retain this", "future": true,
|
||||||
|
}}, "future": true})
|
||||||
|
rejections, err := loadRejections(rejectedPath)
|
||||||
|
if err != nil || len(rejections) != 1 || rejections[0].ReasonCode != "invalid" {
|
||||||
|
t.Fatalf("loadRejections() = %#v, %v", rejections, err)
|
||||||
|
}
|
||||||
|
warnings, err := loadWarnings(warningsPath)
|
||||||
|
if err != nil || len(warnings) != 1 || warnings[0].Scope != "lane:spells" {
|
||||||
|
t.Fatalf("loadWarnings() = %#v, %v", warnings, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, path := range map[string]string{"rejections": rejectedPath, "warnings": warningsPath} {
|
||||||
|
t.Run("malformed "+name, func(t *testing.T) {
|
||||||
|
if err := os.WriteFile(path, []byte("{"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile() error = %v", err)
|
||||||
|
}
|
||||||
|
var err error
|
||||||
|
if name == "rejections" {
|
||||||
|
_, err = loadRejections(path)
|
||||||
|
} else {
|
||||||
|
_, err = loadWarnings(path)
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("summary decoder error = nil")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
oversized := filepath.Join(root, "oversized.json")
|
||||||
|
if err := os.WriteFile(oversized, []byte(strings.Repeat("x", maxSummaryBytes+1)), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(oversized) error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := loadWarnings(oversized); err == nil || !strings.Contains(err.Error(), "exceeds") {
|
||||||
|
t.Fatalf("loadWarnings(oversized) error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := loadRejections(oversized); err == nil || !strings.Contains(err.Error(), "exceeds") {
|
||||||
|
t.Fatalf("loadRejections(oversized) error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFakeRunnerCapturesRequestsAndHonorsContextAndError(t *testing.T) {
|
||||||
|
req := RunRequest{PipelineID: "pipeline"}
|
||||||
|
want := RunResult{BundleRoot: "/bundle"}
|
||||||
|
fake := &FakeRunner{Result: want}
|
||||||
|
got, err := fake.Run(context.Background(), req)
|
||||||
|
if err != nil || !reflect.DeepEqual(got, want) || !reflect.DeepEqual(fake.Requests, []RunRequest{req}) {
|
||||||
|
t.Fatalf("Run() = %#v, %v; requests = %#v", got, err, fake.Requests)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantErr := errors.New("configured failure")
|
||||||
|
fake.Err = wantErr
|
||||||
|
if _, err := fake.Run(context.Background(), req); !errors.Is(err, wantErr) {
|
||||||
|
t.Fatalf("Run(configured error) = %v", err)
|
||||||
|
}
|
||||||
|
canceled, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
before := len(fake.Requests)
|
||||||
|
if _, err := fake.Run(canceled, req); !errors.Is(err, context.Canceled) || len(fake.Requests) != before {
|
||||||
|
t.Fatalf("Run(canceled) error = %v; requests = %d", err, len(fake.Requests))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validRunRequest(t *testing.T) RunRequest {
|
||||||
|
t.Helper()
|
||||||
|
root := t.TempDir()
|
||||||
|
configPath := filepath.Join(root, "notarius.yml")
|
||||||
|
inputPath := filepath.Join(root, "input.json")
|
||||||
|
outputRoot := filepath.Join(root, "outputs")
|
||||||
|
workingDirectory := filepath.Join(root, "work")
|
||||||
|
diagnostics := filepath.Join(root, "diagnostics")
|
||||||
|
for _, directory := range []string{outputRoot, workingDirectory, diagnostics} {
|
||||||
|
if err := os.Mkdir(directory, 0o755); err != nil {
|
||||||
|
t.Fatalf("Mkdir(%q) error = %v", directory, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(configPath, []byte("pipelines: {}\n"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(config) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(inputPath, []byte("{}\n"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(input) error = %v", err)
|
||||||
|
}
|
||||||
|
return RunRequest{
|
||||||
|
Binary: "notarius", ConfigPath: configPath, PipelineID: "dnd-session", InputPath: inputPath,
|
||||||
|
OutputRoot: outputRoot, WorkingDirectory: workingDirectory,
|
||||||
|
ReceiptPath: filepath.Join(diagnostics, "receipt.json"), LogPath: filepath.Join(diagnostics, "stderr.log"),
|
||||||
|
Timeout: time.Second,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeValidBundleAndReceipt(t *testing.T, req RunRequest, includeUnknown bool) {
|
||||||
|
t.Helper()
|
||||||
|
bundle := filepath.Join(req.OutputRoot, "notarius-run-1")
|
||||||
|
if err := os.MkdirAll(filepath.Join(bundle, "lanes"), 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(bundle) error = %v", err)
|
||||||
|
}
|
||||||
|
for path, data := range map[string]string{
|
||||||
|
"manifest.json": `{}`,
|
||||||
|
"lanes/npc.json": `{}`,
|
||||||
|
"chunk-map.json": `{}`,
|
||||||
|
"evidence-context.json": `{}`,
|
||||||
|
} {
|
||||||
|
if err := os.WriteFile(filepath.Join(bundle, filepath.FromSlash(path)), []byte(data), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(%q) error = %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
rejection := map[string]any{"stage": "validate", "lane_id": "spells", "reason_code": "invalid_spell", "message": strings.Repeat("external detail", 20)}
|
||||||
|
warning := map[string]any{"scope": "lane:npc-registry", "reason_code": "normalized_name", "message": strings.Repeat("external warning", 20)}
|
||||||
|
if includeUnknown {
|
||||||
|
rejection["future"] = true
|
||||||
|
warning["future"] = true
|
||||||
|
}
|
||||||
|
writeJSONFile(t, filepath.Join(bundle, "rejected.json"), map[string]any{"rejected": []any{rejection}, "future": true})
|
||||||
|
writeJSONFile(t, filepath.Join(bundle, "warnings.json"), map[string]any{"warnings": []any{warning}, "future": true})
|
||||||
|
index := validIndexValue([]any{map[string]any{
|
||||||
|
"lane_id": "npc-registry", "file": "lanes/npc.json", "media_type": "application/json",
|
||||||
|
"module_key": "dnd/npc-registry", "schema_id": "notarius.dnd.npc_registry",
|
||||||
|
"schema_name": "NPCRegistry", "schema_version": "v1", "future": true,
|
||||||
|
}})
|
||||||
|
index["chunk_map"] = map[string]any{
|
||||||
|
"artifact_kind": "chunk_map", "file": "chunk-map.json", "media_type": "application/json",
|
||||||
|
"schema_id": "notarius.chunk_map", "schema_name": "ChunkMap", "schema_version": "v1", "future": true,
|
||||||
|
}
|
||||||
|
index["evidence_context"] = map[string]any{
|
||||||
|
"artifact_kind": "evidence_context", "file": "evidence-context.json", "media_type": "application/json",
|
||||||
|
"schema_id": "notarius.evidence_context", "schema_name": "EvidenceContext", "schema_version": "v1", "future": true,
|
||||||
|
}
|
||||||
|
index["future"] = true
|
||||||
|
writeJSONFile(t, filepath.Join(bundle, "index.json"), index)
|
||||||
|
receipt := map[string]any{
|
||||||
|
"schema_version": ReceiptSchemaVersion, "run_id": "notarius-run-1", "pipeline_id": req.PipelineID,
|
||||||
|
"output_directory": bundle, "index_file": "index.json", "normalized_output_count": 1,
|
||||||
|
"rejected_output_count": 1, "warning_count": 1, "validation_status": "rejected",
|
||||||
|
}
|
||||||
|
if includeUnknown {
|
||||||
|
receipt["future"] = true
|
||||||
|
}
|
||||||
|
writeJSONFile(t, req.ReceiptPath, receipt)
|
||||||
|
}
|
||||||
|
|
||||||
|
func createBundleSkeleton(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
bundle := filepath.Join(t.TempDir(), "bundle")
|
||||||
|
if err := os.MkdirAll(filepath.Join(bundle, "lanes"), 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(bundle) error = %v", err)
|
||||||
|
}
|
||||||
|
for _, name := range []string{"manifest.json", "rejected.json", "warnings.json", "lanes/npc.json", "chunk-map.json"} {
|
||||||
|
if err := os.WriteFile(filepath.Join(bundle, filepath.FromSlash(name)), []byte("{}"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(%q) error = %v", name, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return bundle
|
||||||
|
}
|
||||||
|
|
||||||
|
func validIndexValue(lanes []any) map[string]any {
|
||||||
|
return map[string]any{
|
||||||
|
"manifest_file": "manifest.json", "output_files": lanes,
|
||||||
|
"rejected_file": "rejected.json", "warnings_file": "warnings.json",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeJSONFile(t *testing.T, path string, value any) {
|
||||||
|
t.Helper()
|
||||||
|
data, err := json.Marshal(value)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("json.Marshal() error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(path, data, 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(%q) error = %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeShellScript(t *testing.T, body string) string {
|
||||||
|
t.Helper()
|
||||||
|
path := filepath.Join(t.TempDir(), "notarius-helper")
|
||||||
|
if err := os.WriteFile(path, []byte(body), 0o755); err != nil {
|
||||||
|
t.Fatalf("WriteFile(script) error = %v", err)
|
||||||
|
}
|
||||||
|
return path
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertTextFile(t *testing.T, path, want string) {
|
||||||
|
t.Helper()
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(%q) error = %v", path, err)
|
||||||
|
}
|
||||||
|
if string(data) != want {
|
||||||
|
t.Fatalf("ReadFile(%q) = %q, want %q", path, string(data), want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneMap(source map[string]any) map[string]any {
|
||||||
|
result := make(map[string]any, len(source))
|
||||||
|
for key, value := range source {
|
||||||
|
result[key] = value
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
@@ -68,6 +68,26 @@ func (n *NoopRunner) Normalize(ctx context.Context, req NormalizeRequest) (Norma
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Render returns the requested output path with placeholder metadata.
|
||||||
|
func (n *NoopRunner) Render(ctx context.Context, req RenderRequest) (RenderResult, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return RenderResult{}, err
|
||||||
|
}
|
||||||
|
if err := materializeRenderPlaceholders(req); err != nil {
|
||||||
|
return RenderResult{}, err
|
||||||
|
}
|
||||||
|
return RenderResult{
|
||||||
|
OutputRenderedPath: req.OutputRenderedPath,
|
||||||
|
StdoutLogPath: req.StdoutLogPath,
|
||||||
|
StderrLogPath: req.StderrLogPath,
|
||||||
|
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||||
|
InvokedBinary: "noop",
|
||||||
|
Format: req.Format,
|
||||||
|
Title: req.Title,
|
||||||
|
Metadata: map[string]any{"placeholder": true},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
// FakeRunner captures merge requests and returns deterministic responses.
|
// FakeRunner captures merge requests and returns deterministic responses.
|
||||||
type FakeRunner struct {
|
type FakeRunner struct {
|
||||||
Requests []MergeRequest
|
Requests []MergeRequest
|
||||||
@@ -79,6 +99,9 @@ type FakeRunner struct {
|
|||||||
TrimRequests []TrimRequest
|
TrimRequests []TrimRequest
|
||||||
TrimErr error
|
TrimErr error
|
||||||
TrimResult TrimResult
|
TrimResult TrimResult
|
||||||
|
RenderRequests []RenderRequest
|
||||||
|
RenderErr error
|
||||||
|
RenderResult RenderResult
|
||||||
}
|
}
|
||||||
|
|
||||||
// Run records request and returns configured response.
|
// Run records request and returns configured response.
|
||||||
@@ -195,6 +218,46 @@ func (f *FakeRunner) Normalize(ctx context.Context, req NormalizeRequest) (Norma
|
|||||||
return res, nil
|
return res, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Render records request and returns configured response.
|
||||||
|
func (f *FakeRunner) Render(ctx context.Context, req RenderRequest) (RenderResult, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return RenderResult{}, err
|
||||||
|
}
|
||||||
|
f.RenderRequests = append(f.RenderRequests, req)
|
||||||
|
if f.RenderErr != nil {
|
||||||
|
return RenderResult{}, f.RenderErr
|
||||||
|
}
|
||||||
|
if err := materializeRenderPlaceholders(req); err != nil {
|
||||||
|
return RenderResult{}, err
|
||||||
|
}
|
||||||
|
res := f.RenderResult
|
||||||
|
if res.OutputRenderedPath == "" {
|
||||||
|
res.OutputRenderedPath = req.OutputRenderedPath
|
||||||
|
}
|
||||||
|
if res.StdoutLogPath == "" {
|
||||||
|
res.StdoutLogPath = req.StdoutLogPath
|
||||||
|
}
|
||||||
|
if res.StderrLogPath == "" {
|
||||||
|
res.StderrLogPath = req.StderrLogPath
|
||||||
|
}
|
||||||
|
if res.GeneratedConfigPath == "" {
|
||||||
|
res.GeneratedConfigPath = req.GeneratedConfigPath
|
||||||
|
}
|
||||||
|
if res.InvokedBinary == "" {
|
||||||
|
res.InvokedBinary = "fake"
|
||||||
|
}
|
||||||
|
if res.Format == "" {
|
||||||
|
res.Format = req.Format
|
||||||
|
}
|
||||||
|
if res.Title == "" {
|
||||||
|
res.Title = req.Title
|
||||||
|
}
|
||||||
|
if res.Metadata == nil {
|
||||||
|
res.Metadata = map[string]any{"fake": true}
|
||||||
|
}
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
|
|
||||||
func materializePlaceholders(req MergeRequest) error {
|
func materializePlaceholders(req MergeRequest) error {
|
||||||
if req.OutputMergedTranscriptPath != "" {
|
if req.OutputMergedTranscriptPath != "" {
|
||||||
if err := subprocess.WriteFileAtomic(req.OutputMergedTranscriptPath, []byte(`{"schema":"seriatim.intermediate.v1","segments":[]}`), 0o644); err != nil {
|
if err := subprocess.WriteFileAtomic(req.OutputMergedTranscriptPath, []byte(`{"schema":"seriatim.intermediate.v1","segments":[]}`), 0o644); err != nil {
|
||||||
@@ -301,3 +364,39 @@ func materializeNormalizePlaceholders(req NormalizeRequest) error {
|
|||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func materializeRenderPlaceholders(req RenderRequest) error {
|
||||||
|
if req.OutputRenderedPath != "" {
|
||||||
|
if err := subprocess.WriteFileAtomic(req.OutputRenderedPath, []byte("# Transcript\n\nRendered markdown placeholder.\n"), 0o644); err != nil {
|
||||||
|
return fmt.Errorf("write rendered transcript %q: %w", req.OutputRenderedPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if req.GeneratedConfigPath != "" {
|
||||||
|
payload := map[string]any{
|
||||||
|
"schema": "seriatim.generated.v1",
|
||||||
|
"placeholder": true,
|
||||||
|
"command": "render",
|
||||||
|
"input_path": req.InputTranscriptPath,
|
||||||
|
"output_path": req.OutputRenderedPath,
|
||||||
|
"format": req.Format,
|
||||||
|
"title": req.Title,
|
||||||
|
"include_timestamps": req.IncludeTimestamps,
|
||||||
|
"include_segment_ids": req.IncludeSegmentIDs,
|
||||||
|
"include_metadata": req.IncludeMetadata,
|
||||||
|
}
|
||||||
|
if err := subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644); err != nil {
|
||||||
|
return fmt.Errorf("write generated config %q: %w", req.GeneratedConfigPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if req.StdoutLogPath != "" {
|
||||||
|
if err := subprocess.WriteFileAtomic(req.StdoutLogPath, []byte("seriatim noop/fake render stdout placeholder\n"), 0o644); err != nil {
|
||||||
|
return fmt.Errorf("write stdout log %q: %w", req.StdoutLogPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if req.StderrLogPath != "" {
|
||||||
|
if err := subprocess.WriteFileAtomic(req.StderrLogPath, []byte("seriatim noop/fake render stderr placeholder\n"), 0o644); err != nil {
|
||||||
|
return fmt.Errorf("write stderr log %q: %w", req.StderrLogPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -148,3 +148,58 @@ func TestFakeRunnerNormalizeError(t *testing.T) {
|
|||||||
t.Fatal("expected error, got nil")
|
t.Fatal("expected error, got nil")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFakeRunnerRenderCapturesRequestAndReturnsPath(t *testing.T) {
|
||||||
|
fake := &FakeRunner{}
|
||||||
|
dir := t.TempDir()
|
||||||
|
req := RenderRequest{
|
||||||
|
GeneratedConfigPath: filepath.Join(dir, "config", "seriatim.render.yml"),
|
||||||
|
InputTranscriptPath: filepath.Join(dir, "transcripts", "final.trimmed.json"),
|
||||||
|
OutputRenderedPath: filepath.Join(dir, "transcripts", "final.trimmed.md"),
|
||||||
|
Format: "markdown",
|
||||||
|
Title: "Session render",
|
||||||
|
IncludeTimestamps: true,
|
||||||
|
IncludeSegmentIDs: false,
|
||||||
|
IncludeMetadata: true,
|
||||||
|
StdoutLogPath: filepath.Join(dir, "logs", "seriatim.render.stdout.log"),
|
||||||
|
StderrLogPath: filepath.Join(dir, "logs", "seriatim.render.stderr.log"),
|
||||||
|
}
|
||||||
|
|
||||||
|
res, err := fake.Render(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Render() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(fake.RenderRequests) != 1 || fake.RenderRequests[0].GeneratedConfigPath == "" {
|
||||||
|
t.Fatalf("render requests = %#v, want captured request", fake.RenderRequests)
|
||||||
|
}
|
||||||
|
if res.OutputRenderedPath != req.OutputRenderedPath {
|
||||||
|
t.Fatalf("rendered path = %q, want %q", res.OutputRenderedPath, req.OutputRenderedPath)
|
||||||
|
}
|
||||||
|
if res.Format != req.Format {
|
||||||
|
t.Fatalf("format = %q, want %q", res.Format, req.Format)
|
||||||
|
}
|
||||||
|
if res.Title != req.Title {
|
||||||
|
t.Fatalf("title = %q, want %q", res.Title, req.Title)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfgData, err := os.ReadFile(req.GeneratedConfigPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read generated config: %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(cfgData), "command: render") {
|
||||||
|
t.Fatalf("generated config = %q, want render command marker", string(cfgData))
|
||||||
|
}
|
||||||
|
for _, path := range []string{req.StdoutLogPath, req.StderrLogPath, req.OutputRenderedPath} {
|
||||||
|
if _, err := os.Stat(path); err != nil {
|
||||||
|
t.Fatalf("expected file %q to exist: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFakeRunnerRenderError(t *testing.T) {
|
||||||
|
fake := &FakeRunner{RenderErr: errors.New("boom")}
|
||||||
|
_, err := fake.Render(context.Background(), RenderRequest{})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error, got nil")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
// Package seriatim declares the adapter contract for transcript merge/normalize/trim execution.
|
// Package seriatim declares the adapter contract for transcript merge/normalize/trim/render execution.
|
||||||
package seriatim
|
package seriatim
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -6,11 +6,12 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Runner is the adapter boundary for seriatim merge/normalize/trim invocations.
|
// Runner is the adapter boundary for seriatim merge/normalize/trim/render invocations.
|
||||||
type Runner interface {
|
type Runner interface {
|
||||||
Run(ctx context.Context, req MergeRequest) (MergeResult, error)
|
Run(ctx context.Context, req MergeRequest) (MergeResult, error)
|
||||||
Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error)
|
Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error)
|
||||||
Trim(ctx context.Context, req TrimRequest) (TrimResult, error)
|
Trim(ctx context.Context, req TrimRequest) (TrimResult, error)
|
||||||
|
Render(ctx context.Context, req RenderRequest) (RenderResult, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
// MergeRequest describes a seriatim merge invocation.
|
// MergeRequest describes a seriatim merge invocation.
|
||||||
@@ -90,3 +91,33 @@ type TrimResult struct {
|
|||||||
KeepSelector string
|
KeepSelector string
|
||||||
Metadata map[string]any
|
Metadata map[string]any
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RenderRequest describes a seriatim render invocation.
|
||||||
|
type RenderRequest struct {
|
||||||
|
Binary string
|
||||||
|
InputTranscriptPath string
|
||||||
|
OutputRenderedPath string
|
||||||
|
Format string
|
||||||
|
Title string
|
||||||
|
IncludeTimestamps bool
|
||||||
|
IncludeSegmentIDs bool
|
||||||
|
IncludeMetadata bool
|
||||||
|
StdoutLogPath string
|
||||||
|
StderrLogPath string
|
||||||
|
GeneratedConfigPath string
|
||||||
|
Timeout time.Duration
|
||||||
|
}
|
||||||
|
|
||||||
|
// RenderResult describes a render output.
|
||||||
|
type RenderResult struct {
|
||||||
|
OutputRenderedPath string
|
||||||
|
StdoutLogPath string
|
||||||
|
StderrLogPath string
|
||||||
|
GeneratedConfigPath string
|
||||||
|
ExitCode int
|
||||||
|
Duration time.Duration
|
||||||
|
InvokedBinary string
|
||||||
|
Format string
|
||||||
|
Title string
|
||||||
|
Metadata map[string]any
|
||||||
|
}
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import (
|
|||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/subprocess"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/subprocess"
|
||||||
)
|
)
|
||||||
@@ -384,6 +385,96 @@ func (r *SubprocessRunner) Normalize(ctx context.Context, req NormalizeRequest)
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Render executes Seriatim render with deterministic flags and validates non-empty text output.
|
||||||
|
func (r *SubprocessRunner) Render(ctx context.Context, req RenderRequest) (RenderResult, error) {
|
||||||
|
if r == nil {
|
||||||
|
return RenderResult{}, fmt.Errorf("seriatim subprocess runner is nil")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(req.InputTranscriptPath) == "" {
|
||||||
|
return RenderResult{}, fmt.Errorf("seriatim render input path is required")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(req.OutputRenderedPath) == "" {
|
||||||
|
return RenderResult{}, fmt.Errorf("seriatim render output path is required")
|
||||||
|
}
|
||||||
|
format := strings.TrimSpace(req.Format)
|
||||||
|
if format == "" {
|
||||||
|
format = "markdown"
|
||||||
|
}
|
||||||
|
if format != "markdown" {
|
||||||
|
return RenderResult{}, fmt.Errorf("seriatim render format %q is unsupported", req.Format)
|
||||||
|
}
|
||||||
|
|
||||||
|
binary := r.binary
|
||||||
|
if strings.TrimSpace(req.Binary) != "" {
|
||||||
|
binary = strings.TrimSpace(req.Binary)
|
||||||
|
}
|
||||||
|
|
||||||
|
timeout := r.timeout
|
||||||
|
if req.Timeout < 0 {
|
||||||
|
return RenderResult{}, fmt.Errorf("seriatim render timeout must be >= 0")
|
||||||
|
}
|
||||||
|
if req.Timeout > 0 {
|
||||||
|
timeout = req.Timeout
|
||||||
|
}
|
||||||
|
|
||||||
|
args := buildRenderArgs(req, format)
|
||||||
|
if req.GeneratedConfigPath != "" {
|
||||||
|
if err := writeRenderInvocationConfig(req, args, binary, timeout, format); err != nil {
|
||||||
|
return RenderResult{}, fmt.Errorf("write seriatim render invocation config %q: %w", req.GeneratedConfigPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
runRes, err := subprocess.Run(ctx, subprocess.RunRequest{
|
||||||
|
Executable: binary,
|
||||||
|
Args: args,
|
||||||
|
Timeout: timeout,
|
||||||
|
StdoutLogPath: req.StdoutLogPath,
|
||||||
|
StderrLogPath: req.StderrLogPath,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return RenderResult{
|
||||||
|
OutputRenderedPath: req.OutputRenderedPath,
|
||||||
|
StdoutLogPath: req.StdoutLogPath,
|
||||||
|
StderrLogPath: req.StderrLogPath,
|
||||||
|
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||||
|
ExitCode: runRes.ExitCode,
|
||||||
|
Duration: runRes.Duration,
|
||||||
|
InvokedBinary: binary,
|
||||||
|
Format: format,
|
||||||
|
Title: req.Title,
|
||||||
|
}, fmt.Errorf("run seriatim render (binary=%q): %w", binary, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := validateNonEmptyTextFile(req.OutputRenderedPath); err != nil {
|
||||||
|
return RenderResult{
|
||||||
|
OutputRenderedPath: req.OutputRenderedPath,
|
||||||
|
StdoutLogPath: req.StdoutLogPath,
|
||||||
|
StderrLogPath: req.StderrLogPath,
|
||||||
|
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||||
|
ExitCode: runRes.ExitCode,
|
||||||
|
Duration: runRes.Duration,
|
||||||
|
InvokedBinary: binary,
|
||||||
|
Format: format,
|
||||||
|
Title: req.Title,
|
||||||
|
}, fmt.Errorf("validate seriatim rendered output %q: %w", req.OutputRenderedPath, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return RenderResult{
|
||||||
|
OutputRenderedPath: req.OutputRenderedPath,
|
||||||
|
StdoutLogPath: req.StdoutLogPath,
|
||||||
|
StderrLogPath: req.StderrLogPath,
|
||||||
|
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||||
|
ExitCode: runRes.ExitCode,
|
||||||
|
Duration: runRes.Duration,
|
||||||
|
InvokedBinary: binary,
|
||||||
|
Format: format,
|
||||||
|
Title: req.Title,
|
||||||
|
Metadata: map[string]any{
|
||||||
|
"adapter": "seriatim_subprocess",
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (r *SubprocessRunner) buildMergeArgs(req MergeRequest) []string {
|
func (r *SubprocessRunner) buildMergeArgs(req MergeRequest) []string {
|
||||||
args := []string{"merge"}
|
args := []string{"merge"}
|
||||||
|
|
||||||
@@ -480,6 +571,22 @@ func buildNormalizeArgs(req NormalizeRequest, outputSchema string) []string {
|
|||||||
return args
|
return args
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func buildRenderArgs(req RenderRequest, format string) []string {
|
||||||
|
args := []string{
|
||||||
|
"render",
|
||||||
|
"--input-file", req.InputTranscriptPath,
|
||||||
|
"--output-file", req.OutputRenderedPath,
|
||||||
|
"--format", format,
|
||||||
|
"--include-timestamps=" + strconv.FormatBool(req.IncludeTimestamps),
|
||||||
|
"--include-segment-ids=" + strconv.FormatBool(req.IncludeSegmentIDs),
|
||||||
|
"--include-metadata=" + strconv.FormatBool(req.IncludeMetadata),
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(req.Title) != "" {
|
||||||
|
args = append(args, "--title", req.Title)
|
||||||
|
}
|
||||||
|
return args
|
||||||
|
}
|
||||||
|
|
||||||
func writeTrimInvocationConfig(req TrimRequest, args []string, binary string, timeout time.Duration) error {
|
func writeTrimInvocationConfig(req TrimRequest, args []string, binary string, timeout time.Duration) error {
|
||||||
payload := map[string]any{
|
payload := map[string]any{
|
||||||
"schema": "seriatim.generated.v1",
|
"schema": "seriatim.generated.v1",
|
||||||
@@ -509,6 +616,24 @@ func writeNormalizeInvocationConfig(req NormalizeRequest, args []string, binary
|
|||||||
return subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644)
|
return subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writeRenderInvocationConfig(req RenderRequest, args []string, binary string, timeout time.Duration, format string) error {
|
||||||
|
payload := map[string]any{
|
||||||
|
"schema": "seriatim.generated.v1",
|
||||||
|
"command": "render",
|
||||||
|
"binary": binary,
|
||||||
|
"args": args,
|
||||||
|
"timeout": timeout.String(),
|
||||||
|
"input_path": req.InputTranscriptPath,
|
||||||
|
"output_path": req.OutputRenderedPath,
|
||||||
|
"format": format,
|
||||||
|
"title": req.Title,
|
||||||
|
"include_timestamps": req.IncludeTimestamps,
|
||||||
|
"include_segment_ids": req.IncludeSegmentIDs,
|
||||||
|
"include_metadata": req.IncludeMetadata,
|
||||||
|
}
|
||||||
|
return subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644)
|
||||||
|
}
|
||||||
|
|
||||||
func validateJSONFile(path string) error {
|
func validateJSONFile(path string) error {
|
||||||
data, err := os.ReadFile(path)
|
data, err := os.ReadFile(path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -541,3 +666,20 @@ func validateJSONFileWithSegments(path string) error {
|
|||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validateNonEmptyTextFile(path string) error {
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("read file: %w", err)
|
||||||
|
}
|
||||||
|
if len(data) == 0 {
|
||||||
|
return fmt.Errorf("file is empty")
|
||||||
|
}
|
||||||
|
if !utf8.Valid(data) {
|
||||||
|
return fmt.Errorf("file is not valid utf-8 text")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(string(data)) == "" {
|
||||||
|
return fmt.Errorf("file has no non-whitespace content")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -569,6 +569,156 @@ func TestSubprocessRunnerNormalizeInvalidReportJSONFails(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerRenderSuccessInvocationAndProvenance(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("helper wrapper script uses /bin/sh")
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("SERIATIM_HELPER_MODE", "render_success")
|
||||||
|
recordPath := filepath.Join(t.TempDir(), "record.json")
|
||||||
|
t.Setenv("SERIATIM_HELPER_RECORD_PATH", recordPath)
|
||||||
|
|
||||||
|
wrapper := writeHelperWrapper(t)
|
||||||
|
runner := mustRunner(t, wrapper, false)
|
||||||
|
req := renderReqForTest(t)
|
||||||
|
|
||||||
|
res, err := runner.Render(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Render() error = %v", err)
|
||||||
|
}
|
||||||
|
if res.OutputRenderedPath != req.OutputRenderedPath {
|
||||||
|
t.Fatalf("OutputRenderedPath = %q, want %q", res.OutputRenderedPath, req.OutputRenderedPath)
|
||||||
|
}
|
||||||
|
if res.Format != req.Format {
|
||||||
|
t.Fatalf("Format = %q, want %q", res.Format, req.Format)
|
||||||
|
}
|
||||||
|
if res.Title != req.Title {
|
||||||
|
t.Fatalf("Title = %q, want %q", res.Title, req.Title)
|
||||||
|
}
|
||||||
|
if res.InvokedBinary != wrapper {
|
||||||
|
t.Fatalf("InvokedBinary = %q, want %q", res.InvokedBinary, wrapper)
|
||||||
|
}
|
||||||
|
if res.ExitCode != 0 {
|
||||||
|
t.Fatalf("ExitCode = %d, want 0", res.ExitCode)
|
||||||
|
}
|
||||||
|
if res.Duration <= 0 {
|
||||||
|
t.Fatalf("Duration = %s, want >0", res.Duration)
|
||||||
|
}
|
||||||
|
if res.Metadata == nil || res.Metadata["adapter"] != "seriatim_subprocess" {
|
||||||
|
t.Fatalf("Metadata = %#v, want adapter marker", res.Metadata)
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := os.Stat(req.OutputRenderedPath); err != nil {
|
||||||
|
t.Fatalf("rendered output missing: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(req.StdoutLogPath); err != nil {
|
||||||
|
t.Fatalf("stdout log missing: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(req.StderrLogPath); err != nil {
|
||||||
|
t.Fatalf("stderr log missing: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(req.GeneratedConfigPath); err != nil {
|
||||||
|
t.Fatalf("generated config missing: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := readHelperRecord(t, recordPath)
|
||||||
|
wantArgs := []string{
|
||||||
|
"render",
|
||||||
|
"--input-file", req.InputTranscriptPath,
|
||||||
|
"--output-file", req.OutputRenderedPath,
|
||||||
|
"--format", req.Format,
|
||||||
|
"--include-timestamps=true",
|
||||||
|
"--include-segment-ids=true",
|
||||||
|
"--include-metadata=false",
|
||||||
|
"--title", req.Title,
|
||||||
|
}
|
||||||
|
if strings.Join(rec.Args, "\n") != strings.Join(wantArgs, "\n") {
|
||||||
|
t.Fatalf("args = %#v, want %#v", rec.Args, wantArgs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerRenderWithoutTitleOmitsTitleArg(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("helper wrapper script uses /bin/sh")
|
||||||
|
}
|
||||||
|
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("SERIATIM_HELPER_MODE", "render_success")
|
||||||
|
recordPath := filepath.Join(t.TempDir(), "record.json")
|
||||||
|
t.Setenv("SERIATIM_HELPER_RECORD_PATH", recordPath)
|
||||||
|
|
||||||
|
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||||
|
req := renderReqForTest(t)
|
||||||
|
req.Title = ""
|
||||||
|
if _, err := runner.Render(context.Background(), req); err != nil {
|
||||||
|
t.Fatalf("Render() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := readHelperRecord(t, recordPath)
|
||||||
|
for i := 0; i < len(rec.Args); i++ {
|
||||||
|
if rec.Args[i] == "--title" {
|
||||||
|
t.Fatalf("args = %#v, did not expect --title", rec.Args)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerRenderSubprocessFailure(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("helper wrapper script uses /bin/sh")
|
||||||
|
}
|
||||||
|
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("SERIATIM_HELPER_MODE", "fail")
|
||||||
|
t.Setenv("SERIATIM_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||||
|
|
||||||
|
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||||
|
req := renderReqForTest(t)
|
||||||
|
_, err := runner.Render(context.Background(), req)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Render() error = nil, want non-nil")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "run seriatim render") {
|
||||||
|
t.Fatalf("error = %q, want subprocess context", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerRenderMissingOutputFails(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("helper wrapper script uses /bin/sh")
|
||||||
|
}
|
||||||
|
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("SERIATIM_HELPER_MODE", "missing_output")
|
||||||
|
t.Setenv("SERIATIM_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||||
|
|
||||||
|
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||||
|
req := renderReqForTest(t)
|
||||||
|
_, err := runner.Render(context.Background(), req)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Render() error = nil, want non-nil")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "validate seriatim rendered output") {
|
||||||
|
t.Fatalf("error = %q, want output validation context", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerRenderEmptyOutputFails(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("helper wrapper script uses /bin/sh")
|
||||||
|
}
|
||||||
|
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("SERIATIM_HELPER_MODE", "render_empty_output")
|
||||||
|
t.Setenv("SERIATIM_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||||
|
|
||||||
|
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||||
|
req := renderReqForTest(t)
|
||||||
|
_, err := runner.Render(context.Background(), req)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Render() error = nil, want non-nil")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "file is empty") {
|
||||||
|
t.Fatalf("error = %q, want empty-file validation", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestSubprocessRunnerConstructorValidation(t *testing.T) {
|
func TestSubprocessRunnerConstructorValidation(t *testing.T) {
|
||||||
_, err := NewSubprocessRunnerFromConfigValues("", "10m", "seriatim-intermediate", nil, true, EnvConfig{})
|
_, err := NewSubprocessRunnerFromConfigValues("", "10m", "seriatim-intermediate", nil, true, EnvConfig{})
|
||||||
if err == nil {
|
if err == nil {
|
||||||
@@ -702,6 +852,14 @@ func TestSeriatimSubprocessHelper(t *testing.T) {
|
|||||||
case "normalize_report_missing":
|
case "normalize_report_missing":
|
||||||
writeSeriatimHelperFile(outputPath, `{"schema":"seriatim.intermediate.v1","segments":[]}`)
|
writeSeriatimHelperFile(outputPath, `{"schema":"seriatim.intermediate.v1","segments":[]}`)
|
||||||
os.Exit(0)
|
os.Exit(0)
|
||||||
|
case "render_success":
|
||||||
|
writeSeriatimHelperFile(outputPath, "# Rendered transcript\n\nHello.\n")
|
||||||
|
_, _ = os.Stdout.WriteString("seriatim helper render stdout\n")
|
||||||
|
_, _ = os.Stderr.WriteString("seriatim helper render stderr\n")
|
||||||
|
os.Exit(0)
|
||||||
|
case "render_empty_output":
|
||||||
|
writeSeriatimHelperFile(outputPath, "")
|
||||||
|
os.Exit(0)
|
||||||
default:
|
default:
|
||||||
_, _ = os.Stderr.WriteString(fmt.Sprintf("unknown helper mode %q\n", mode))
|
_, _ = os.Stderr.WriteString(fmt.Sprintf("unknown helper mode %q\n", mode))
|
||||||
os.Exit(2)
|
os.Exit(2)
|
||||||
@@ -777,6 +935,25 @@ func normalizeReqForTest(t *testing.T, withReport bool) NormalizeRequest {
|
|||||||
return req
|
return req
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func renderReqForTest(t *testing.T) RenderRequest {
|
||||||
|
t.Helper()
|
||||||
|
dir := t.TempDir()
|
||||||
|
input := filepath.Join(dir, "final.trimmed.json")
|
||||||
|
writeSeriatimFile(t, input, `{"schema":"seriatim.intermediate.v1","segments":[]}`)
|
||||||
|
return RenderRequest{
|
||||||
|
InputTranscriptPath: input,
|
||||||
|
OutputRenderedPath: filepath.Join(dir, "final.trimmed.md"),
|
||||||
|
Format: "markdown",
|
||||||
|
Title: "Session 42",
|
||||||
|
IncludeTimestamps: true,
|
||||||
|
IncludeSegmentIDs: true,
|
||||||
|
IncludeMetadata: false,
|
||||||
|
GeneratedConfigPath: filepath.Join(dir, "seriatim.render.generated.yml"),
|
||||||
|
StdoutLogPath: filepath.Join(dir, "seriatim.render.stdout.log"),
|
||||||
|
StderrLogPath: filepath.Join(dir, "seriatim.render.stderr.log"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func mustRunner(t *testing.T, binary string, report bool) *SubprocessRunner {
|
func mustRunner(t *testing.T, binary string, report bool) *SubprocessRunner {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
coalesce := 3.0
|
coalesce := 3.0
|
||||||
|
|||||||
@@ -1,31 +0,0 @@
|
|||||||
// Package storage declares archive/storage backend adapter boundaries.
|
|
||||||
package storage
|
|
||||||
|
|
||||||
import "context"
|
|
||||||
|
|
||||||
// TODO: implement remote storage/archive backends (S3/SFTP/etc.).
|
|
||||||
|
|
||||||
// Backend is the adapter boundary for archive/storage operations.
|
|
||||||
type Backend interface {
|
|
||||||
Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ArchiveItem describes one item to archive.
|
|
||||||
type ArchiveItem struct {
|
|
||||||
Kind string
|
|
||||||
LocalPath string
|
|
||||||
RemoteKey string
|
|
||||||
}
|
|
||||||
|
|
||||||
// ArchiveRequest describes one archive operation.
|
|
||||||
type ArchiveRequest struct {
|
|
||||||
SessionID string
|
|
||||||
ManifestPath string
|
|
||||||
Items []ArchiveItem
|
|
||||||
}
|
|
||||||
|
|
||||||
// ArchiveResult describes archive operation output.
|
|
||||||
type ArchiveResult struct {
|
|
||||||
Archived []ArchiveItem
|
|
||||||
Metadata map[string]any
|
|
||||||
}
|
|
||||||
@@ -10,23 +10,8 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// NoopBackend is a deterministic no-op archive/storage adapter.
|
// FakeBackend provides a deterministic in-memory object store for tests.
|
||||||
type NoopBackend struct{}
|
|
||||||
|
|
||||||
// Archive returns the requested items as archived with placeholder metadata.
|
|
||||||
func (n *NoopBackend) Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error) {
|
|
||||||
if err := ctx.Err(); err != nil {
|
|
||||||
return ArchiveResult{}, err
|
|
||||||
}
|
|
||||||
return ArchiveResult{Archived: append([]ArchiveItem(nil), req.Items...), Metadata: map[string]any{"placeholder": true}}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// FakeBackend captures archive requests and returns deterministic responses.
|
|
||||||
type FakeBackend struct {
|
type FakeBackend struct {
|
||||||
Requests []ArchiveRequest
|
|
||||||
Err error
|
|
||||||
Result ArchiveResult
|
|
||||||
|
|
||||||
Objects map[string]FakeObject
|
Objects map[string]FakeObject
|
||||||
Uploads []FakeUploadCall
|
Uploads []FakeUploadCall
|
||||||
Downloads []FakeDownloadCall
|
Downloads []FakeDownloadCall
|
||||||
@@ -50,25 +35,6 @@ type FakeDownloadCall struct {
|
|||||||
LocalPath string
|
LocalPath string
|
||||||
}
|
}
|
||||||
|
|
||||||
// Archive records request and returns configured response.
|
|
||||||
func (f *FakeBackend) Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error) {
|
|
||||||
if err := ctx.Err(); err != nil {
|
|
||||||
return ArchiveResult{}, err
|
|
||||||
}
|
|
||||||
f.Requests = append(f.Requests, req)
|
|
||||||
if f.Err != nil {
|
|
||||||
return ArchiveResult{}, f.Err
|
|
||||||
}
|
|
||||||
res := f.Result
|
|
||||||
if res.Archived == nil {
|
|
||||||
res.Archived = append([]ArchiveItem(nil), req.Items...)
|
|
||||||
}
|
|
||||||
if res.Metadata == nil {
|
|
||||||
res.Metadata = map[string]any{"fake": true}
|
|
||||||
}
|
|
||||||
return res, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// FakeObject is a deterministic fake object-store record.
|
// FakeObject is a deterministic fake object-store record.
|
||||||
type FakeObject struct {
|
type FakeObject struct {
|
||||||
Key string
|
Key string
|
||||||
|
|||||||
@@ -9,30 +9,6 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestFakeBackendCapturesRequestAndReturnsItems(t *testing.T) {
|
|
||||||
fake := &FakeBackend{}
|
|
||||||
req := ArchiveRequest{SessionID: "s1", Items: []ArchiveItem{{Kind: "artifact", LocalPath: "artifacts/log.md"}}}
|
|
||||||
|
|
||||||
res, err := fake.Archive(context.Background(), req)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Archive() error = %v", err)
|
|
||||||
}
|
|
||||||
if len(fake.Requests) != 1 || fake.Requests[0].SessionID != "s1" {
|
|
||||||
t.Fatalf("requests = %#v, want captured request", fake.Requests)
|
|
||||||
}
|
|
||||||
if len(res.Archived) != 1 {
|
|
||||||
t.Fatalf("archived len = %d, want 1", len(res.Archived))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFakeBackendError(t *testing.T) {
|
|
||||||
fake := &FakeBackend{Err: errors.New("boom")}
|
|
||||||
_, err := fake.Archive(context.Background(), ArchiveRequest{})
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("expected error, got nil")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFakeBackendListPrefixFiltering(t *testing.T) {
|
func TestFakeBackendListPrefixFiltering(t *testing.T) {
|
||||||
fake := &FakeBackend{}
|
fake := &FakeBackend{}
|
||||||
fake.SeedObject(FakeObject{Key: "dnd/campaigns/forsaken/audio/a.flac", Data: []byte("a")})
|
fake.SeedObject(FakeObject{Key: "dnd/campaigns/forsaken/audio/a.flac", Data: []byte("a")})
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ObjectStore is a remote object storage boundary used by future prepare/archive work.
|
// ObjectStore is a remote object storage boundary used by prepare, restore, and publish work.
|
||||||
//
|
//
|
||||||
// Key invariant:
|
// Key invariant:
|
||||||
// callers pass full bucket-relative object keys. Backend implementations do not
|
// callers pass full bucket-relative object keys. Backend implementations do not
|
||||||
|
|||||||
36
internal/adapters/storage/temp_download.go
Normal file
36
internal/adapters/storage/temp_download.go
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
package storage
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// DownloadObjectToTemp downloads an object into a temporary file and returns
|
||||||
|
// the cleaned local path.
|
||||||
|
func DownloadObjectToTemp(ctx context.Context, store ObjectStore, key, pattern string) (string, error) {
|
||||||
|
if store == nil {
|
||||||
|
return "", fmt.Errorf("object store is required")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(pattern) == "" {
|
||||||
|
return "", fmt.Errorf("temp file pattern is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
tmp, err := os.CreateTemp("", pattern)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("create temp file: %w", err)
|
||||||
|
}
|
||||||
|
path := tmp.Name()
|
||||||
|
if err := tmp.Close(); err != nil {
|
||||||
|
_ = os.Remove(path)
|
||||||
|
return "", fmt.Errorf("close temp file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := store.Download(ctx, key, path); err != nil {
|
||||||
|
_ = os.Remove(path)
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return filepath.Clean(path), nil
|
||||||
|
}
|
||||||
69
internal/adapters/storage/temp_download_test.go
Normal file
69
internal/adapters/storage/temp_download_test.go
Normal file
@@ -0,0 +1,69 @@
|
|||||||
|
package storage
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestDownloadObjectToTempSuccess(t *testing.T) {
|
||||||
|
store := &FakeBackend{}
|
||||||
|
store.SeedObject(FakeObject{Key: "sessions/a/current/run_id.txt", Data: []byte("run-123\n")})
|
||||||
|
|
||||||
|
path, err := DownloadObjectToTemp(context.Background(), store, "sessions/a/current/run_id.txt", "narratio-test-*.txt")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DownloadObjectToTemp() error = %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { _ = os.Remove(path) })
|
||||||
|
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile() error = %v", err)
|
||||||
|
}
|
||||||
|
if string(data) != "run-123\n" {
|
||||||
|
t.Fatalf("downloaded data = %q, want %q", string(data), "run-123\n")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDownloadObjectToTempFailedDownloadRemovesTempFile(t *testing.T) {
|
||||||
|
sentinel := errors.New("download failed")
|
||||||
|
store := &FakeBackend{DownloadErr: sentinel}
|
||||||
|
pattern := "narratio-test-fail-*.txt"
|
||||||
|
before, err := filepath.Glob(filepath.Join(os.TempDir(), "narratio-test-fail-*.txt"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Glob(before) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
path, err := DownloadObjectToTemp(context.Background(), store, "sessions/a/current/run_id.txt", pattern)
|
||||||
|
if !errors.Is(err, sentinel) {
|
||||||
|
t.Fatalf("DownloadObjectToTemp() error = %v, want %v", err, sentinel)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(path) != "" {
|
||||||
|
t.Fatalf("DownloadObjectToTemp() path = %q, want empty on failure", path)
|
||||||
|
}
|
||||||
|
after, err := filepath.Glob(filepath.Join(os.TempDir(), "narratio-test-fail-*.txt"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Glob(after) error = %v", err)
|
||||||
|
}
|
||||||
|
if len(after) != len(before) {
|
||||||
|
t.Fatalf("temp file count changed after failed download: before=%d after=%d", len(before), len(after))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDownloadObjectToTempCallerContextWrappingPreservesCause(t *testing.T) {
|
||||||
|
sentinel := errors.New("object missing")
|
||||||
|
store := &FakeBackend{DownloadErr: sentinel}
|
||||||
|
|
||||||
|
_, err := DownloadObjectToTemp(context.Background(), store, "sessions/a/current/run_id.txt", "narratio-test-*.txt")
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("DownloadObjectToTemp() error = nil, want error")
|
||||||
|
}
|
||||||
|
err = fmt.Errorf("download run pointer failed: %w", err)
|
||||||
|
if !errors.Is(err, sentinel) {
|
||||||
|
t.Fatalf("wrapped error does not preserve sentinel cause: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -145,7 +145,7 @@ func TestHTTPClientDoesNotRetryOnNonRetryableStatus(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestHTTPClientInvalidJSONFailsAndDoesNotPromote(t *testing.T) {
|
func TestHTTPClientInvalidJSONFailsAndDoesNotInstallOutput(t *testing.T) {
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
_, _ = w.Write([]byte(`not-json`))
|
_, _ = w.Write([]byte(`not-json`))
|
||||||
}))
|
}))
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ func TestExecuteRunStageArtifactsUnsupportedStageFails(t *testing.T) {
|
|||||||
var stdout bytes.Buffer
|
var stdout bytes.Buffer
|
||||||
var stderr bytes.Buffer
|
var stderr bytes.Buffer
|
||||||
code := Execute(
|
code := Execute(
|
||||||
[]string{"run-stage", "polish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
[]string{"run-stage", "extract", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
||||||
&stdout,
|
&stdout,
|
||||||
&stderr,
|
&stderr,
|
||||||
)
|
)
|
||||||
@@ -33,7 +33,7 @@ func TestExecuteRunStageArtifactsUnsupportedStageFails(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteRunStageArchivePropagatesSelectedArtifacts(t *testing.T) {
|
func TestExecuteRunStagePublishPropagatesSelectedArtifacts(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
|
|
||||||
@@ -120,31 +120,32 @@ func TestRunStageArtifactsDoesNotImplyForce(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResumeArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
|
func TestRunArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||||
|
|
||||||
store := &manifest.LocalStore{}
|
store := &manifest.LocalStore{}
|
||||||
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "publish", "notify"} {
|
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
|
||||||
seed.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
seed.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||||
}
|
}
|
||||||
|
seed.MarkStageSkipped("extract", time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), "notarius_disabled")
|
||||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||||
t.Fatalf("save manifest: %v", err)
|
t.Fatalf("save manifest: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
err := Resume(
|
err := Run(
|
||||||
context.Background(),
|
context.Background(),
|
||||||
[]string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
[]string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
||||||
&out,
|
&out,
|
||||||
)
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "has no remaining stages") {
|
if !strings.Contains(out.String(), "executed=1 skipped=11") {
|
||||||
t.Fatalf("output = %q, want no remaining stages", out.String())
|
t.Fatalf("output = %q, want all stages skipped", out.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -281,7 +282,7 @@ func TestExecuteAnalyzeMissingConfigUsesRunStageLoadingPath(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecutePublishForceRunsArchive(t *testing.T) {
|
func TestExecutePublishForceRunsPublish(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
|
|
||||||
|
|||||||
@@ -16,7 +16,6 @@ import (
|
|||||||
// Clean removes local workspace/spool state while preserving durable cache
|
// Clean removes local workspace/spool state while preserving durable cache
|
||||||
// state unless cache cleanup is explicitly requested.
|
// state unless cache cleanup is explicitly requested.
|
||||||
func Clean(ctx context.Context, args []string, out io.Writer) error {
|
func Clean(ctx context.Context, args []string, out io.Writer) error {
|
||||||
positionalSessionID, args := pullLeadingSessionID(args)
|
|
||||||
fs := flag.NewFlagSet("clean", flag.ContinueOnError)
|
fs := flag.NewFlagSet("clean", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
var flags commonConfigFlags
|
var flags commonConfigFlags
|
||||||
@@ -27,21 +26,9 @@ func Clean(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
fs.BoolVar(&all, "all", false, "clean all local session work/spool state")
|
fs.BoolVar(&all, "all", false, "clean all local session work/spool state")
|
||||||
fs.BoolVar(&dryRun, "dry-run", false, "print cleanup targets without deleting")
|
fs.BoolVar(&dryRun, "dry-run", false, "print cleanup targets without deleting")
|
||||||
fs.BoolVar(&clearCache, "clear-cache", false, "also clear durable S3 audio cache entries")
|
fs.BoolVar(&clearCache, "clear-cache", false, "also clear durable S3 audio cache entries")
|
||||||
if err := fs.Parse(args); err != nil {
|
if err := parseSessionAwareFlags("clean", fs, args, &flags.sessionID); err != nil {
|
||||||
return fmt.Errorf("clean: invalid flags: %w", err)
|
|
||||||
}
|
|
||||||
if positionalSessionID == "" {
|
|
||||||
if err := applyParsedSessionIDArg("clean", fs, &flags.sessionID); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
} else {
|
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("clean: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("clean", positionalSessionID, &flags.sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if all {
|
if all {
|
||||||
return cleanAllLocal(flags, dryRun, clearCache, out)
|
return cleanAllLocal(flags, dryRun, clearCache, out)
|
||||||
}
|
}
|
||||||
@@ -182,27 +169,13 @@ func reportCleanRootChildren(out io.Writer, root, policy string, dryRun bool) er
|
|||||||
}
|
}
|
||||||
|
|
||||||
func cleanableRootChildren(root, policy string) (string, []string, error) {
|
func cleanableRootChildren(root, policy string) (string, []string, error) {
|
||||||
cleanRoot := strings.TrimSpace(root)
|
rootAbs, exists, err := validateCleanRoot(root, policy)
|
||||||
if cleanRoot == "" {
|
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
|
||||||
}
|
|
||||||
rootAbs, err := filepath.Abs(cleanRoot)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
return "", nil, err
|
||||||
}
|
}
|
||||||
info, err := os.Lstat(rootAbs)
|
if !exists {
|
||||||
if err != nil {
|
|
||||||
if os.IsNotExist(err) {
|
|
||||||
return rootAbs, nil, nil
|
return rootAbs, nil, nil
|
||||||
}
|
}
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: stat root %q: %w", policy, rootAbs, err)
|
|
||||||
}
|
|
||||||
if info.Mode()&os.ModeSymlink != 0 {
|
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: refusing to clean symlink root %q", policy, rootAbs)
|
|
||||||
}
|
|
||||||
if !info.IsDir() {
|
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: root %q is not a directory", policy, rootAbs)
|
|
||||||
}
|
|
||||||
entries, err := os.ReadDir(rootAbs)
|
entries, err := os.ReadDir(rootAbs)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", nil, fmt.Errorf("cleanup policy %s: read root %q: %w", policy, rootAbs, err)
|
return "", nil, fmt.Errorf("cleanup policy %s: read root %q: %w", policy, rootAbs, err)
|
||||||
@@ -300,46 +273,7 @@ func reportCleanScopedFile(out io.Writer, root, target, policy string, dryRun bo
|
|||||||
}
|
}
|
||||||
|
|
||||||
func validateScopedFile(root, target, policy string) (scopedDir, error) {
|
func validateScopedFile(root, target, policy string) (scopedDir, error) {
|
||||||
cleanRoot := strings.TrimSpace(root)
|
return validateScopedTarget(root, target, policy, false)
|
||||||
cleanTarget := strings.TrimSpace(target)
|
|
||||||
if cleanRoot == "" {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
|
||||||
}
|
|
||||||
if cleanTarget == "" {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target path is required", policy)
|
|
||||||
}
|
|
||||||
rootAbs, err := filepath.Abs(cleanRoot)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
|
||||||
}
|
|
||||||
targetAbs, err := filepath.Abs(cleanTarget)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve target %q: %w", policy, cleanTarget, err)
|
|
||||||
}
|
|
||||||
rel, err := filepath.Rel(rootAbs, targetAbs)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: relative path from %q to %q: %w", policy, rootAbs, targetAbs, err)
|
|
||||||
}
|
|
||||||
if rel == "." {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete root directory %q", policy, rootAbs)
|
|
||||||
}
|
|
||||||
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete path outside root: root=%q target=%q", policy, rootAbs, targetAbs)
|
|
||||||
}
|
|
||||||
info, err := os.Lstat(targetAbs)
|
|
||||||
if err != nil {
|
|
||||||
if os.IsNotExist(err) {
|
|
||||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: false}, nil
|
|
||||||
}
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: stat target %q: %w", policy, targetAbs, err)
|
|
||||||
}
|
|
||||||
if info.Mode()&os.ModeSymlink != 0 {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, targetAbs)
|
|
||||||
}
|
|
||||||
if info.IsDir() {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is a directory", policy, targetAbs)
|
|
||||||
}
|
|
||||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: true}, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func cleanIsFlac(path string) bool {
|
func cleanIsFlac(path string) bool {
|
||||||
|
|||||||
82
internal/app/cleanup_targets.go
Normal file
82
internal/app/cleanup_targets.go
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
func validateScopedTarget(root, target, policy string, requireDir bool) (scopedDir, error) {
|
||||||
|
cleanRoot := strings.TrimSpace(root)
|
||||||
|
cleanTarget := strings.TrimSpace(target)
|
||||||
|
if cleanRoot == "" {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
||||||
|
}
|
||||||
|
if cleanTarget == "" {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: target path is required", policy)
|
||||||
|
}
|
||||||
|
|
||||||
|
rootAbs, err := filepath.Abs(cleanRoot)
|
||||||
|
if err != nil {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
||||||
|
}
|
||||||
|
targetAbs, err := filepath.Abs(cleanTarget)
|
||||||
|
if err != nil {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve target %q: %w", policy, cleanTarget, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
rel, err := filepath.Rel(rootAbs, targetAbs)
|
||||||
|
if err != nil {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: relative path from %q to %q: %w", policy, rootAbs, targetAbs, err)
|
||||||
|
}
|
||||||
|
if rel == "." {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete root directory %q", policy, rootAbs)
|
||||||
|
}
|
||||||
|
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete path outside root: root=%q target=%q", policy, rootAbs, targetAbs)
|
||||||
|
}
|
||||||
|
|
||||||
|
info, err := os.Lstat(targetAbs)
|
||||||
|
if err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: false}, nil
|
||||||
|
}
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: stat target %q: %w", policy, targetAbs, err)
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, targetAbs)
|
||||||
|
}
|
||||||
|
if requireDir && !info.IsDir() {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is not a directory", policy, targetAbs)
|
||||||
|
}
|
||||||
|
if !requireDir && info.IsDir() {
|
||||||
|
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is a directory", policy, targetAbs)
|
||||||
|
}
|
||||||
|
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: true}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateCleanRoot(root, policy string) (string, bool, error) {
|
||||||
|
cleanRoot := strings.TrimSpace(root)
|
||||||
|
if cleanRoot == "" {
|
||||||
|
return "", false, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
||||||
|
}
|
||||||
|
rootAbs, err := filepath.Abs(cleanRoot)
|
||||||
|
if err != nil {
|
||||||
|
return "", false, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
||||||
|
}
|
||||||
|
info, err := os.Lstat(rootAbs)
|
||||||
|
if err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return rootAbs, false, nil
|
||||||
|
}
|
||||||
|
return "", false, fmt.Errorf("cleanup policy %s: stat root %q: %w", policy, rootAbs, err)
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 {
|
||||||
|
return "", false, fmt.Errorf("cleanup policy %s: refusing to clean symlink root %q", policy, rootAbs)
|
||||||
|
}
|
||||||
|
if !info.IsDir() {
|
||||||
|
return "", false, fmt.Errorf("cleanup policy %s: root %q is not a directory", policy, rootAbs)
|
||||||
|
}
|
||||||
|
return rootAbs, true, nil
|
||||||
|
}
|
||||||
103
internal/app/cleanup_targets_test.go
Normal file
103
internal/app/cleanup_targets_test.go
Normal file
@@ -0,0 +1,103 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestCleanValidateScopedDirAndFile(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
dirTarget := filepath.Join(root, "runs", "run-1")
|
||||||
|
fileTarget := filepath.Join(root, "cache", "a.flac")
|
||||||
|
if err := os.MkdirAll(dirTarget, 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(dirTarget) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.MkdirAll(filepath.Dir(fileTarget), 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(file parent) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(fileTarget, []byte("audio"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(fileTarget) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := validateScopedDir(root, dirTarget, "test.dir"); err != nil {
|
||||||
|
t.Fatalf("validateScopedDir() error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := validateScopedFile(root, fileTarget, "test.file"); err != nil {
|
||||||
|
t.Fatalf("validateScopedFile() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCleanValidateScopedTargetSafetyRules(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
outside := t.TempDir()
|
||||||
|
target := filepath.Join(root, "runs", "run-1")
|
||||||
|
if err := os.MkdirAll(target, 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(target) error = %v", err)
|
||||||
|
}
|
||||||
|
fileTarget := filepath.Join(root, "cache", "a.flac")
|
||||||
|
if err := os.MkdirAll(filepath.Dir(fileTarget), 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(file parent) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(fileTarget, []byte("audio"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(fileTarget) error = %v", err)
|
||||||
|
}
|
||||||
|
symlinkTarget := filepath.Join(root, "symlink")
|
||||||
|
if err := os.Symlink(target, symlinkTarget); err != nil {
|
||||||
|
t.Fatalf("Symlink() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := validateScopedDir(root, root, "test.root"); err == nil || !strings.Contains(err.Error(), "refusing to delete root directory") {
|
||||||
|
t.Fatalf("validateScopedDir(root) error = %v, want root deletion rejection", err)
|
||||||
|
}
|
||||||
|
if _, err := validateScopedDir(root, filepath.Join(outside, "x"), "test.outside"); err == nil || !strings.Contains(err.Error(), "outside root") {
|
||||||
|
t.Fatalf("validateScopedDir(outside) error = %v, want outside-root rejection", err)
|
||||||
|
}
|
||||||
|
if _, err := validateScopedDir(root, fileTarget, "test.file-as-dir"); err == nil || !strings.Contains(err.Error(), "is not a directory") {
|
||||||
|
t.Fatalf("validateScopedDir(file) error = %v, want not-a-directory rejection", err)
|
||||||
|
}
|
||||||
|
if _, err := validateScopedFile(root, target, "test.dir-as-file"); err == nil || !strings.Contains(err.Error(), "is a directory") {
|
||||||
|
t.Fatalf("validateScopedFile(dir) error = %v, want is-a-directory rejection", err)
|
||||||
|
}
|
||||||
|
if _, err := validateScopedDir(root, symlinkTarget, "test.symlink"); err == nil || !strings.Contains(err.Error(), "refusing to delete symlink path") {
|
||||||
|
t.Fatalf("validateScopedDir(symlink) error = %v, want symlink rejection", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCleanableRootChildrenRejectsSymlinkChild(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
realChild := filepath.Join(root, "runs")
|
||||||
|
if err := os.MkdirAll(realChild, 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(realChild) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.Symlink(realChild, filepath.Join(root, "link")); err != nil {
|
||||||
|
t.Fatalf("Symlink() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, _, err := cleanableRootChildren(root, "test.root.children")
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "refusing to delete symlink path") {
|
||||||
|
t.Fatalf("cleanableRootChildren() error = %v, want symlink rejection", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCleanValidateScopedTargetMissing(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
missingDir := filepath.Join(root, "runs", "missing")
|
||||||
|
got, err := validateScopedDir(root, missingDir, "test.missing")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("validateScopedDir(missing) error = %v", err)
|
||||||
|
}
|
||||||
|
if got.Exists {
|
||||||
|
t.Fatalf("validateScopedDir(missing).Exists = true, want false")
|
||||||
|
}
|
||||||
|
|
||||||
|
missingFile := filepath.Join(root, "cache", "missing.flac")
|
||||||
|
got, err = validateScopedFile(root, missingFile, "test.missing.file")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("validateScopedFile(missing) error = %v", err)
|
||||||
|
}
|
||||||
|
if got.Exists {
|
||||||
|
t.Fatalf("validateScopedFile(missing).Exists = true, want false")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -7,7 +7,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
var supportedCommands = []string{"run", "run-stage", "resume", "analyze", "publish", "clean", "session"}
|
var supportedCommands = []string{"run", "run-stage", "analyze", "publish", "clean", "session"}
|
||||||
|
|
||||||
// Execute dispatches CLI commands and returns a process exit code.
|
// Execute dispatches CLI commands and returns a process exit code.
|
||||||
func Execute(args []string, stdout, stderr io.Writer) int {
|
func Execute(args []string, stdout, stderr io.Writer) int {
|
||||||
@@ -24,8 +24,6 @@ func Execute(args []string, stdout, stderr io.Writer) int {
|
|||||||
switch cmd {
|
switch cmd {
|
||||||
case "run":
|
case "run":
|
||||||
err = Run(ctx, cmdArgs, stdout)
|
err = Run(ctx, cmdArgs, stdout)
|
||||||
case "resume":
|
|
||||||
err = Resume(ctx, cmdArgs, stdout)
|
|
||||||
case "run-stage":
|
case "run-stage":
|
||||||
err = RunStage(ctx, cmdArgs, stdout)
|
err = RunStage(ctx, cmdArgs, stdout)
|
||||||
case "analyze":
|
case "analyze":
|
||||||
@@ -53,8 +51,3 @@ func Execute(args []string, stdout, stderr io.Writer) int {
|
|||||||
func printUsage(w io.Writer) {
|
func printUsage(w io.Writer) {
|
||||||
fmt.Fprintf(w, "Usage: narratio <%s>\n", strings.Join(supportedCommands, "|"))
|
fmt.Fprintf(w, "Usage: narratio <%s>\n", strings.Join(supportedCommands, "|"))
|
||||||
}
|
}
|
||||||
|
|
||||||
func placeholder(out io.Writer, command string) error {
|
|
||||||
_, err := fmt.Fprintf(out, "narratio %s: not yet implemented\n", command)
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -31,10 +31,9 @@ func TestExecuteValidCommands(t *testing.T) {
|
|||||||
args []string
|
args []string
|
||||||
wantOut string
|
wantOut string
|
||||||
}{
|
}{
|
||||||
{name: "run", args: []string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run: session 2026-05-03; executed=9 skipped=0; manifest="},
|
{name: "run", args: []string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run: session 2026-05-03; executed=11 skipped=1; manifest="},
|
||||||
{name: "session plan", args: []string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "prepare: skip\ntranscribe: skip\nmerge: skip\npolish: skip\nnormalize: skip\ntrim: skip\nanalyze: skip\npublish: skip\nnotify: skip"},
|
{name: "session plan", args: []string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "prepare: skip\ntranscribe: skip\nmerge: skip\npolish: skip\nnormalize: skip\ntrim: skip\nextract: run\nrender: skip\nanalyze: skip\npublish: skip\nnotify: skip"},
|
||||||
{name: "session status", args: []string{"session", "status", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "Session: 2026-05-03"},
|
{name: "session status", args: []string{"session", "status", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "Session: 2026-05-03"},
|
||||||
{name: "resume", args: []string{"resume", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio resume: session 2026-05-03 has no remaining stages"},
|
|
||||||
{name: "run-stage", args: []string{"run-stage", "polish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run-stage: stage=polish executed=0 skipped=1 force=false; manifest="},
|
{name: "run-stage", args: []string{"run-stage", "polish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run-stage: stage=polish executed=0 skipped=1 force=false; manifest="},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -66,7 +65,7 @@ func TestExecuteMissingRequiredFlags(t *testing.T) {
|
|||||||
{name: "run missing session", args: []string{"run"}, want: "run: session_id is required"},
|
{name: "run missing session", args: []string{"run"}, want: "run: session_id is required"},
|
||||||
{name: "plan old top-level removed", args: []string{"plan"}, want: `unknown command: "plan"`},
|
{name: "plan old top-level removed", args: []string{"plan"}, want: `unknown command: "plan"`},
|
||||||
{name: "status old top-level removed", args: []string{"status"}, want: `unknown command: "status"`},
|
{name: "status old top-level removed", args: []string{"status"}, want: `unknown command: "status"`},
|
||||||
{name: "resume missing session", args: []string{"resume"}, want: "resume: session_id is required"},
|
{name: "resume removed", args: []string{"resume"}, want: `unknown command: "resume"`},
|
||||||
{name: "run-stage missing name", args: []string{"run-stage", "--config", "a", "--session", "b"}, want: "run-stage: expected stage name and session_id"},
|
{name: "run-stage missing name", args: []string{"run-stage", "--config", "a", "--session", "b"}, want: "run-stage: expected stage name and session_id"},
|
||||||
{name: "run-stage missing session", args: []string{"run-stage", "polish"}, want: "run-stage: expected stage name and session_id"},
|
{name: "run-stage missing session", args: []string{"run-stage", "polish"}, want: "run-stage: expected stage name and session_id"},
|
||||||
{name: "run missing config uses defaults", args: []string{"run", "2026-05-03", "--session", "session.yml"}, want: "run: no pipeline config path provided and no default pipeline config found; searched:"},
|
{name: "run missing config uses defaults", args: []string{"run", "2026-05-03", "--session", "session.yml"}, want: "run: no pipeline config path provided and no default pipeline config found; searched:"},
|
||||||
@@ -119,7 +118,7 @@ func TestExecuteRunStageArchiveAliasFails(t *testing.T) {
|
|||||||
t.Fatal("exit code = 0, want non-zero")
|
t.Fatal("exit code = 0, want non-zero")
|
||||||
}
|
}
|
||||||
if !strings.Contains(stderr.String(), `unknown stage "archive"`) {
|
if !strings.Contains(stderr.String(), `unknown stage "archive"`) {
|
||||||
t.Fatalf("stderr = %q, want unknown archive stage error", stderr.String())
|
t.Fatalf("stderr = %q, want unknown stage alias error", stderr.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -228,6 +227,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`
|
`
|
||||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||||
t.Fatalf("write pipeline.yml: %v", err)
|
t.Fatalf("write pipeline.yml: %v", err)
|
||||||
@@ -291,6 +292,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`
|
`
|
||||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||||
t.Fatalf("write pipeline.yml: %v", err)
|
t.Fatalf("write pipeline.yml: %v", err)
|
||||||
@@ -332,7 +335,7 @@ func TestExecuteUsesDefaultPipelineConfigPathWhenConfigFlagOmitted(t *testing.T)
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "narratio run: session 2026-05-03; executed=9 skipped=0; manifest=") {
|
if !strings.Contains(stdout.String(), "narratio run: session 2026-05-03; executed=11 skipped=1; manifest=") {
|
||||||
t.Fatalf("stdout = %q, want successful run output", stdout.String())
|
t.Fatalf("stdout = %q, want successful run output", stdout.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -386,10 +389,14 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`)
|
`)
|
||||||
mustWriteTestFile(t, filepath.Join(otherDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
mustWriteTestFile(t, filepath.Join(otherDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(otherDir, "autocorrect.yml"), "[]\n")
|
mustWriteTestFile(t, filepath.Join(otherDir, "autocorrect.yml"), "[]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(otherDir, "glossary.yml"), "[]\n")
|
mustWriteTestFile(t, filepath.Join(otherDir, "glossary.yml"), "[]\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(otherDir, "players.yml"), "[]\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(otherDir, "party.yml"), "[]\n")
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
var stdout bytes.Buffer
|
||||||
var stderr bytes.Buffer
|
var stderr bytes.Buffer
|
||||||
@@ -467,10 +474,13 @@ func writeValidConfigFiles(t *testing.T, workspaceRoot string, transcribeURL ...
|
|||||||
url = transcribeURL[0]
|
url = transcribeURL[0]
|
||||||
}
|
}
|
||||||
seriatimBinary := writeSeriatimAppTestWrapper(t)
|
seriatimBinary := writeSeriatimAppTestWrapper(t)
|
||||||
|
scriptoriumBinary := writeScriptoriumAppTestWrapper(t)
|
||||||
auditaBinary := writeAuditaAppTestWrapper(t)
|
auditaBinary := writeAuditaAppTestWrapper(t)
|
||||||
t.Setenv("GO_WANT_APP_SERIATIM_HELPER", "1")
|
t.Setenv("GO_WANT_APP_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("GO_WANT_APP_SCRIPTORIUM_HELPER", "1")
|
||||||
t.Setenv("GO_WANT_APP_AUDITA_HELPER", "1")
|
t.Setenv("GO_WANT_APP_AUDITA_HELPER", "1")
|
||||||
t.Setenv("AUDITA_LLM_API_KEY", "test-audita-key")
|
t.Setenv("AUDITA_LLM_API_KEY", "test-audita-key")
|
||||||
|
t.Setenv("PATH", filepath.Dir(scriptoriumBinary)+string(os.PathListSeparator)+os.Getenv("PATH"))
|
||||||
|
|
||||||
pipelineYAML := `workspace:
|
pipelineYAML := `workspace:
|
||||||
root: ` + workspaceRoot + `
|
root: ` + workspaceRoot + `
|
||||||
@@ -515,6 +525,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`
|
`
|
||||||
|
|
||||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||||
@@ -533,6 +545,8 @@ inputs:
|
|||||||
mustWriteTestFile(t, filepath.Join(campaignDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
mustWriteTestFile(t, filepath.Join(campaignDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(campaignDir, "autocorrect.yml"), "[]\n")
|
mustWriteTestFile(t, filepath.Join(campaignDir, "autocorrect.yml"), "[]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(campaignDir, "glossary.yml"), "[]\n")
|
mustWriteTestFile(t, filepath.Join(campaignDir, "glossary.yml"), "[]\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(campaignDir, "players.yml"), "[]\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(campaignDir, "party.yml"), "[]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(dir, "audio", "alice.flac"), "audio-bytes")
|
mustWriteTestFile(t, filepath.Join(dir, "audio", "alice.flac"), "audio-bytes")
|
||||||
|
|
||||||
return pipelinePath, campaignPath, sessionPath
|
return pipelinePath, campaignPath, sessionPath
|
||||||
@@ -546,6 +560,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`
|
`
|
||||||
if err := os.WriteFile(campaignPath, []byte(campaignYAML), 0o644); err != nil {
|
if err := os.WriteFile(campaignPath, []byte(campaignYAML), 0o644); err != nil {
|
||||||
t.Fatalf("write campaign.yml: %v", err)
|
t.Fatalf("write campaign.yml: %v", err)
|
||||||
@@ -592,6 +608,60 @@ func writeSeriatimAppTestWrapper(t *testing.T) string {
|
|||||||
return path
|
return path
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writeScriptoriumAppTestWrapper(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
exe, err := os.Executable()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("os.Executable() error = %v", err)
|
||||||
|
}
|
||||||
|
path := filepath.Join(t.TempDir(), "scriptorium")
|
||||||
|
content := "#!/bin/sh\nexec \"" + exe + "\" -test.run=TestScriptoriumAppHelper -- \"$@\"\n"
|
||||||
|
if err := os.WriteFile(path, []byte(content), 0o755); err != nil {
|
||||||
|
t.Fatalf("WriteFile(%q): %v", path, err)
|
||||||
|
}
|
||||||
|
return path
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestScriptoriumAppHelper(t *testing.T) {
|
||||||
|
if os.Getenv("GO_WANT_APP_SCRIPTORIUM_HELPER") != "1" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
args := os.Args
|
||||||
|
start := -1
|
||||||
|
for i := range args {
|
||||||
|
if args[i] == "--" {
|
||||||
|
start = i + 1
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if start < 0 || start >= len(args) {
|
||||||
|
_, _ = os.Stderr.WriteString("missing -- args separator\n")
|
||||||
|
os.Exit(2)
|
||||||
|
}
|
||||||
|
runArgs := args[start:]
|
||||||
|
|
||||||
|
outputPath := appSeriatimFlagValue(runArgs, "--out")
|
||||||
|
if strings.TrimSpace(outputPath) == "" {
|
||||||
|
outputPath = appSeriatimFlagValue(runArgs, "--output")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(outputPath) == "" {
|
||||||
|
_, _ = os.Stderr.WriteString("missing output flag\n")
|
||||||
|
os.Exit(2)
|
||||||
|
}
|
||||||
|
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
|
||||||
|
_, _ = os.Stderr.WriteString(fmt.Sprintf("mkdir output dir: %v\n", err))
|
||||||
|
os.Exit(2)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(outputPath, []byte(`{"trim_action":"copy","warnings":[]}`), 0o644); err != nil {
|
||||||
|
_, _ = os.Stderr.WriteString(fmt.Sprintf("write output: %v\n", err))
|
||||||
|
os.Exit(2)
|
||||||
|
}
|
||||||
|
_, _ = os.Stdout.WriteString("scriptorium helper stdout\n")
|
||||||
|
_, _ = os.Stderr.WriteString("scriptorium helper stderr\n")
|
||||||
|
os.Exit(0)
|
||||||
|
}
|
||||||
|
|
||||||
func TestSeriatimAppHelper(t *testing.T) {
|
func TestSeriatimAppHelper(t *testing.T) {
|
||||||
if os.Getenv("GO_WANT_APP_SERIATIM_HELPER") != "1" {
|
if os.Getenv("GO_WANT_APP_SERIATIM_HELPER") != "1" {
|
||||||
return
|
return
|
||||||
|
|||||||
@@ -4,7 +4,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
@@ -59,7 +58,7 @@ func loadCommandConfig(ctx context.Context, pipelineFlag, campaignFlag, campaign
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, missingSessionConfigError(discoveredSession.Searched, err.Error())
|
return nil, missingSessionConfigError(discoveredSession.Searched, err.Error())
|
||||||
}
|
}
|
||||||
sessionTempPath, err := downloadRemoteSessionConfig(ctx, store, remoteKey)
|
sessionTempPath, err := storage.DownloadObjectToTemp(ctx, store, remoteKey, "narratio-session-*.yml")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, missingSessionConfigError(discoveredSession.Searched, fmt.Sprintf("remote session %q download failed: %v", remoteKey, err))
|
return nil, missingSessionConfigError(discoveredSession.Searched, fmt.Sprintf("remote session %q download failed: %v", remoteKey, err))
|
||||||
}
|
}
|
||||||
@@ -134,21 +133,6 @@ func findRemoteSessionConfig(ctx context.Context, store storage.ObjectStore, ses
|
|||||||
return storage.ObjectInfo{}, fmt.Errorf("remote session %q not found", remoteKey)
|
return storage.ObjectInfo{}, fmt.Errorf("remote session %q not found", remoteKey)
|
||||||
}
|
}
|
||||||
|
|
||||||
func downloadRemoteSessionConfig(ctx context.Context, store storage.ObjectStore, remoteKey string) (string, error) {
|
|
||||||
f, err := os.CreateTemp("", "narratio-session-*.yml")
|
|
||||||
if err != nil {
|
|
||||||
return "", fmt.Errorf("create temp file: %w", err)
|
|
||||||
}
|
|
||||||
path := f.Name()
|
|
||||||
if err := f.Close(); err != nil {
|
|
||||||
return "", fmt.Errorf("close temp file %q: %w", path, err)
|
|
||||||
}
|
|
||||||
if err := store.Download(ctx, remoteKey, path); err != nil {
|
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
return filepath.Clean(path), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func s3BucketName(cfg *config.PipelineConfig) string {
|
func s3BucketName(cfg *config.PipelineConfig) string {
|
||||||
if cfg == nil || cfg.Storage.S3 == nil {
|
if cfg == nil || cfg.Storage.S3 == nil {
|
||||||
return ""
|
return ""
|
||||||
|
|||||||
416
internal/app/extract_lifecycle_test.go
Normal file
416
internal/app/extract_lifecycle_test.go
Normal file
@@ -0,0 +1,416 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/notarius"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||||
|
)
|
||||||
|
|
||||||
|
type materializingNotariusRunner struct {
|
||||||
|
cfg *config.NotariusConfig
|
||||||
|
requests []notarius.RunRequest
|
||||||
|
failuresRemaining int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *materializingNotariusRunner) Run(_ context.Context, req notarius.RunRequest) (notarius.RunResult, error) {
|
||||||
|
r.requests = append(r.requests, req)
|
||||||
|
if r.failuresRemaining > 0 {
|
||||||
|
r.failuresRemaining--
|
||||||
|
return notarius.RunResult{}, errors.New("notarius execution failed")
|
||||||
|
}
|
||||||
|
externalRunID := fmt.Sprintf("notarius-run-%d", len(r.requests))
|
||||||
|
bundle := filepath.Join(req.OutputRoot, externalRunID)
|
||||||
|
lanesDir := filepath.Join(bundle, "lanes")
|
||||||
|
if err := os.MkdirAll(lanesDir, 0o755); err != nil {
|
||||||
|
return notarius.RunResult{}, err
|
||||||
|
}
|
||||||
|
for path, content := range map[string]string{
|
||||||
|
filepath.Join(bundle, "index.json"): `{"manifest_file":"manifest.json"}`,
|
||||||
|
filepath.Join(bundle, "manifest.json"): `{}`,
|
||||||
|
filepath.Join(bundle, "rejected.json"): `{"rejected":[]}`,
|
||||||
|
filepath.Join(bundle, "warnings.json"): `{"warnings":[]}`,
|
||||||
|
filepath.Join(lanesDir, "npcs.json"): `{"npcs":[]}`,
|
||||||
|
} {
|
||||||
|
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||||
|
return notarius.RunResult{}, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
output := r.cfg.Outputs["npc_registry"]
|
||||||
|
return notarius.RunResult{
|
||||||
|
Receipt: notarius.Receipt{
|
||||||
|
SchemaVersion: notarius.ReceiptSchemaVersion, RunID: externalRunID,
|
||||||
|
PipelineID: req.PipelineID, OutputDirectory: bundle, IndexFile: "index.json",
|
||||||
|
NormalizedOutputCount: 1, ValidationStatus: "valid",
|
||||||
|
},
|
||||||
|
BundleRoot: bundle,
|
||||||
|
Index: notarius.Index{
|
||||||
|
Path: filepath.Join(bundle, "index.json"), RejectedPath: filepath.Join(bundle, "rejected.json"),
|
||||||
|
WarningsPath: filepath.Join(bundle, "warnings.json"),
|
||||||
|
Lanes: []notarius.LaneDescriptor{{
|
||||||
|
LaneID: output.LaneID, File: "lanes/npcs.json", Path: filepath.Join(lanesDir, "npcs.json"),
|
||||||
|
MediaType: output.MediaType, SchemaID: output.SchemaID,
|
||||||
|
SchemaVersion: output.SchemaVersion, ModuleKey: output.ModuleKey,
|
||||||
|
}},
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleDisabledThenEnabled(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, false)
|
||||||
|
plan, err := BuildSingleStagePlan("extract")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("BuildSingleStagePlan(extract) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
first, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("disabled executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(first.Executed) != 1 || len(first.Skipped) != 1 || first.Skipped[0] != "extract" || len(runner.requests) != 0 {
|
||||||
|
t.Fatalf("disabled summary = %#v requests=%d", first, len(runner.requests))
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg.Pipeline.Notarius.Enabled = true
|
||||||
|
second, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("enabled executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(second.Executed) != 1 || len(second.Skipped) != 0 || len(runner.requests) != 1 {
|
||||||
|
t.Fatalf("enabled summary = %#v requests=%d", second, len(runner.requests))
|
||||||
|
}
|
||||||
|
loaded, err := (&manifest.LocalStore{}).Load(context.Background(), second.ManifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() error = %v", err)
|
||||||
|
}
|
||||||
|
if loaded.Stages["extract"] == nil || loaded.Stages["extract"].Status != manifest.StatusSucceeded || len(loaded.Stages["extract"].Outputs) != 2 {
|
||||||
|
t.Fatalf("extract record = %#v, want succeeded manifest-ready outputs", loaded.Stages["extract"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleChangedOutcomeRerunsSucceededDownstream(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, false)
|
||||||
|
analyzeRuns := 0
|
||||||
|
plan := extractionLifecyclePlan(t, &analyzeRuns)
|
||||||
|
|
||||||
|
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env}); err != nil {
|
||||||
|
t.Fatalf("disabled executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if analyzeRuns != 1 || len(runner.requests) != 0 {
|
||||||
|
t.Fatalf("disabled run analyze=%d Notarius=%d, want 1 and 0", analyzeRuns, len(runner.requests))
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg.Pipeline.Notarius.Enabled = true
|
||||||
|
summary, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("enabled executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if analyzeRuns != 2 || len(runner.requests) != 1 {
|
||||||
|
t.Fatalf("enabled run analyze=%d Notarius=%d, want 2 and 1", analyzeRuns, len(runner.requests))
|
||||||
|
}
|
||||||
|
if len(summary.Executed) != 2 || len(summary.Skipped) != 0 {
|
||||||
|
t.Fatalf("enabled summary = %#v, want extract and analyze executed", summary)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleFailureInvalidatesAndOrdinaryRetryRerunsDownstream(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, true)
|
||||||
|
runner.failuresRemaining = 1
|
||||||
|
markLifecycleStageSucceeded(t, cfg, "analyze")
|
||||||
|
analyzeRuns := 0
|
||||||
|
plan := extractionLifecyclePlan(t, &analyzeRuns)
|
||||||
|
|
||||||
|
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env}); err == nil || !strings.Contains(err.Error(), "notarius execution failed") {
|
||||||
|
t.Fatalf("failed executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
failed := loadLifecycleManifest(t, cfg)
|
||||||
|
if failed.Stages["extract"].Status != manifest.StatusFailed || failed.Stages["analyze"].Status != manifest.StatusStale {
|
||||||
|
t.Fatalf("failed lifecycle extract=%#v analyze=%#v", failed.Stages["extract"], failed.Stages["analyze"])
|
||||||
|
}
|
||||||
|
if failed.Stages["analyze"].Error == nil || failed.Stages["analyze"].Error.Message != staleReasonFailure {
|
||||||
|
t.Fatalf("analyze stale reason = %#v, want %q", failed.Stages["analyze"].Error, staleReasonFailure)
|
||||||
|
}
|
||||||
|
|
||||||
|
summary, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("retry executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if analyzeRuns != 1 || len(runner.requests) != 2 || len(summary.Executed) != 2 {
|
||||||
|
t.Fatalf("retry analyze=%d Notarius=%d summary=%#v", analyzeRuns, len(runner.requests), summary)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleForcedSelfSkipInvalidatesDownstream(t *testing.T) {
|
||||||
|
cfg, env, _ := extractionLifecycleFixture(t, false)
|
||||||
|
markLifecycleStageSucceeded(t, cfg, "analyze")
|
||||||
|
plan, _ := BuildSingleStagePlan("extract")
|
||||||
|
|
||||||
|
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env, Force: true}); err != nil {
|
||||||
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
loaded := loadLifecycleManifest(t, cfg)
|
||||||
|
if loaded.Stages["extract"].Status != manifest.StatusSkipped || loaded.Stages["analyze"].Status != manifest.StatusStale {
|
||||||
|
t.Fatalf("forced self-skip extract=%#v analyze=%#v", loaded.Stages["extract"], loaded.Stages["analyze"])
|
||||||
|
}
|
||||||
|
if loaded.Stages["analyze"].Error == nil || loaded.Stages["analyze"].Error.Message != staleReasonForcedReplacement {
|
||||||
|
t.Fatalf("analyze stale reason = %#v, want %q", loaded.Stages["analyze"].Error, staleReasonForcedReplacement)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleForcedFailureInvalidatesDownstream(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, true)
|
||||||
|
plan, _ := BuildSingleStagePlan("extract")
|
||||||
|
|
||||||
|
first, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("initial executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
succeeded := loadLifecycleManifest(t, cfg).Stages["extract"]
|
||||||
|
if succeeded == nil || succeeded.Status != manifest.StatusSucceeded || len(succeeded.Outputs) == 0 || len(succeeded.Logs) == 0 || len(succeeded.Metadata) == 0 {
|
||||||
|
t.Fatalf("initial extraction result = %#v, want succeeded result details", succeeded)
|
||||||
|
}
|
||||||
|
|
||||||
|
markLifecycleStageSucceeded(t, cfg, "analyze")
|
||||||
|
runner.failuresRemaining = 1
|
||||||
|
|
||||||
|
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env, Force: true}); err == nil {
|
||||||
|
t.Fatal("executeStages() error = nil, want forced extraction failure")
|
||||||
|
}
|
||||||
|
loaded := loadLifecycleManifest(t, cfg)
|
||||||
|
if loaded.Stages["extract"].Status != manifest.StatusFailed || loaded.Stages["analyze"].Status != manifest.StatusStale {
|
||||||
|
t.Fatalf("forced failure extract=%#v analyze=%#v", loaded.Stages["extract"], loaded.Stages["analyze"])
|
||||||
|
}
|
||||||
|
if loaded.Stages["analyze"].Error == nil || loaded.Stages["analyze"].Error.Message != staleReasonForcedReplacement {
|
||||||
|
t.Fatalf("analyze stale reason = %#v, want %q", loaded.Stages["analyze"].Error, staleReasonForcedReplacement)
|
||||||
|
}
|
||||||
|
failed := loaded.Stages["extract"]
|
||||||
|
if len(failed.Outputs) != 0 || len(failed.Logs) != 0 || len(failed.GeneratedConfigs) != 0 || len(failed.Metadata) != 0 {
|
||||||
|
t.Fatalf("failed replacement inherited extraction result details: %#v", failed)
|
||||||
|
}
|
||||||
|
historical, err := (&manifest.LocalStore{}).LoadRun(context.Background(), first.RunManifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("LoadRun(initial) error = %v", err)
|
||||||
|
}
|
||||||
|
historicalExtract := historical.Stages["extract"]
|
||||||
|
if historicalExtract == nil || historicalExtract.Status != manifest.StatusSucceeded || len(historicalExtract.Outputs) == 0 || len(historicalExtract.Logs) == 0 || len(historicalExtract.Metadata) == 0 {
|
||||||
|
t.Fatalf("historical extraction result = %#v, want preserved succeeded details", historicalExtract)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(succeeded.Outputs[0].LocalPath); err != nil {
|
||||||
|
t.Fatalf("durable extraction output was not preserved: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleRepeatedSelfSkipPreservesSucceededDownstream(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, false)
|
||||||
|
analyzeRuns := 0
|
||||||
|
plan := extractionLifecyclePlan(t, &analyzeRuns)
|
||||||
|
|
||||||
|
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env}); err != nil {
|
||||||
|
t.Fatalf("first executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
second, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("second executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if analyzeRuns != 1 || len(runner.requests) != 0 {
|
||||||
|
t.Fatalf("repeated disabled run analyze=%d Notarius=%d, want 1 and 0", analyzeRuns, len(runner.requests))
|
||||||
|
}
|
||||||
|
if len(second.Executed) != 1 || len(second.Skipped) != 2 {
|
||||||
|
t.Fatalf("second summary = %#v, want executed self-skip and skipped analyze", second)
|
||||||
|
}
|
||||||
|
loaded := loadLifecycleManifest(t, cfg)
|
||||||
|
if loaded.Stages["analyze"].Status != manifest.StatusSucceeded {
|
||||||
|
t.Fatalf("analyze = %#v, want succeeded", loaded.Stages["analyze"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleSkipsCurrentResumableResult(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, true)
|
||||||
|
analyzeRuns := 0
|
||||||
|
plan := extractionLifecyclePlan(t, &analyzeRuns)
|
||||||
|
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env}); err != nil {
|
||||||
|
t.Fatalf("first executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
resumed, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("resume executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(resumed.Executed) != 0 || len(resumed.Skipped) != 2 || len(runner.requests) != 1 || analyzeRuns != 1 {
|
||||||
|
t.Fatalf("resume summary = %#v requests=%d analyze=%d", resumed, len(runner.requests), analyzeRuns)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleResumesAndRerunsObsoleteResults(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
mutate func(*testing.T, *config.Config, *manifest.Manifest)
|
||||||
|
}{
|
||||||
|
{name: "configuration changed", mutate: func(_ *testing.T, cfg *config.Config, _ *manifest.Manifest) {
|
||||||
|
output := cfg.Pipeline.Notarius.Outputs["npc_registry"]
|
||||||
|
output.SchemaVersion = "v2"
|
||||||
|
cfg.Pipeline.Notarius.Outputs["npc_registry"] = output
|
||||||
|
}},
|
||||||
|
{name: "payload missing", mutate: func(t *testing.T, _ *config.Config, m *manifest.Manifest) {
|
||||||
|
if err := os.Remove(m.Stages["extract"].Outputs[0].LocalPath); err != nil {
|
||||||
|
t.Fatalf("Remove() error = %v", err)
|
||||||
|
}
|
||||||
|
}},
|
||||||
|
{name: "payload tampered", mutate: func(t *testing.T, _ *config.Config, m *manifest.Manifest) {
|
||||||
|
if err := os.WriteFile(m.Stages["extract"].Outputs[0].LocalPath, []byte(`{"npcs":["tampered"]}`), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile() error = %v", err)
|
||||||
|
}
|
||||||
|
}},
|
||||||
|
{name: "record incompatible", mutate: func(_ *testing.T, _ *config.Config, m *manifest.Manifest) {
|
||||||
|
m.Stages["extract"].Outputs[0].Contract.SchemaID = "incompatible"
|
||||||
|
}},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, true)
|
||||||
|
plan, _ := BuildSingleStagePlan("extract")
|
||||||
|
first, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("first executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
persisted, err := (&manifest.LocalStore{}).Load(context.Background(), first.ManifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
test.mutate(t, cfg, persisted)
|
||||||
|
if err := (&manifest.LocalStore{}).Save(context.Background(), first.ManifestPath, persisted); err != nil {
|
||||||
|
t.Fatalf("Save(mutated) error = %v", err)
|
||||||
|
}
|
||||||
|
rerun, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("rerun executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(rerun.Executed) != 1 || len(rerun.Skipped) != 0 || len(runner.requests) != 2 {
|
||||||
|
t.Fatalf("rerun summary = %#v requests=%d", rerun, len(runner.requests))
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleUnsafeResumeErrorPreservesSuccess(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, true)
|
||||||
|
analyzeRuns := 0
|
||||||
|
plan := extractionLifecyclePlan(t, &analyzeRuns)
|
||||||
|
first, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("first executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
store := &manifest.LocalStore{}
|
||||||
|
persisted, err := store.Load(context.Background(), first.ManifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() error = %v", err)
|
||||||
|
}
|
||||||
|
persisted.Stages["extract"].Outputs[0].LocalPath = filepath.Join(cfg.Pipeline.Workspace.Root, "outside.json")
|
||||||
|
if err := store.Save(context.Background(), first.ManifestPath, persisted); err != nil {
|
||||||
|
t.Fatalf("Save() error = %v", err)
|
||||||
|
}
|
||||||
|
before, _ := json.Marshal(map[string]*manifest.StageRecord{
|
||||||
|
"extract": persisted.Stages["extract"],
|
||||||
|
"analyze": persisted.Stages["analyze"],
|
||||||
|
})
|
||||||
|
|
||||||
|
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env}); err == nil || !strings.Contains(err.Error(), "unsafe") {
|
||||||
|
t.Fatalf("executeStages() error = %v, want unsafe resume failure", err)
|
||||||
|
}
|
||||||
|
afterManifest, err := store.Load(context.Background(), first.ManifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load(after) error = %v", err)
|
||||||
|
}
|
||||||
|
after, _ := json.Marshal(map[string]*manifest.StageRecord{
|
||||||
|
"extract": afterManifest.Stages["extract"],
|
||||||
|
"analyze": afterManifest.Stages["analyze"],
|
||||||
|
})
|
||||||
|
if string(before) != string(after) || len(runner.requests) != 1 || analyzeRuns != 1 {
|
||||||
|
t.Fatalf("successful records changed: before=%s after=%s requests=%d analyze=%d", before, after, len(runner.requests), analyzeRuns)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func extractionLifecyclePlan(t *testing.T, analyzeRuns *int) []stage.Stage {
|
||||||
|
t.Helper()
|
||||||
|
plan, err := BuildSingleStagePlan("extract")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("BuildSingleStagePlan(extract) error = %v", err)
|
||||||
|
}
|
||||||
|
return append(plan, countingStage{name: "analyze", runs: analyzeRuns})
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadLifecycleManifest(t *testing.T, cfg *config.Config) *manifest.Manifest {
|
||||||
|
t.Helper()
|
||||||
|
loaded, err := (&manifest.LocalStore{}).Load(context.Background(), manifestPathFor(cfg))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() error = %v", err)
|
||||||
|
}
|
||||||
|
return loaded
|
||||||
|
}
|
||||||
|
|
||||||
|
func markLifecycleStageSucceeded(t *testing.T, cfg *config.Config, name string) {
|
||||||
|
t.Helper()
|
||||||
|
loaded := loadLifecycleManifest(t, cfg)
|
||||||
|
loaded.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
||||||
|
if err := (&manifest.LocalStore{}).Save(context.Background(), manifestPathFor(cfg), loaded); err != nil {
|
||||||
|
t.Fatalf("Save() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func extractionLifecycleFixture(t *testing.T, enabled bool) (*config.Config, *stage.Env, *materializingNotariusRunner) {
|
||||||
|
t.Helper()
|
||||||
|
cfg := testConfig(t)
|
||||||
|
root := cfg.Pipeline.Workspace.Root
|
||||||
|
binary := filepath.Join(root, "notarius")
|
||||||
|
configPath := filepath.Join(root, "notarius.yml")
|
||||||
|
workingDirectory := filepath.Join(root, "notarius-work")
|
||||||
|
if err := os.WriteFile(binary, []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil {
|
||||||
|
t.Fatalf("WriteFile(binary) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(configPath, []byte("pipelines: {}\n"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(config) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.Mkdir(workingDirectory, 0o755); err != nil {
|
||||||
|
t.Fatalf("Mkdir(working directory) error = %v", err)
|
||||||
|
}
|
||||||
|
cfg.Pipeline.Notarius = &config.NotariusConfig{
|
||||||
|
Enabled: enabled, Binary: binary, ConfigPath: configPath, PipelineID: "dnd-session",
|
||||||
|
Timeout: "45m", WorkingDirectory: workingDirectory,
|
||||||
|
Outputs: map[string]config.NotariusOutputConfig{
|
||||||
|
"npc_registry": {
|
||||||
|
LaneID: "npc-registry", MediaType: "application/json", SchemaID: "notarius.dnd.npc_registry",
|
||||||
|
SchemaVersion: "v1", ModuleKey: "dnd/npc-registry",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
paths, err := artifacts.NewLocalStore(root).EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("EnsureLayoutFor() error = %v", err)
|
||||||
|
}
|
||||||
|
inputPath := filepath.Join(paths.ArtifactsDir, "trimmed.from-manifest.json")
|
||||||
|
if err := os.WriteFile(inputPath, []byte(`{"segments":[]}`), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(input) error = %v", err)
|
||||||
|
}
|
||||||
|
m := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
||||||
|
m.Campaign = cfg.Session.Campaign
|
||||||
|
m.MarkStageSucceeded("trim", time.Now().UTC(), []manifest.ArtifactRecord{{
|
||||||
|
Kind: artifactmodel.TranscriptOutputKindFinalTrimmed, SourceID: artifactmodel.SourceTranscriptFinalTrimmed,
|
||||||
|
LocalPath: inputPath,
|
||||||
|
}})
|
||||||
|
if err := (&manifest.LocalStore{}).Save(context.Background(), paths.ManifestPath, m); err != nil {
|
||||||
|
t.Fatalf("Save(seed) error = %v", err)
|
||||||
|
}
|
||||||
|
runner := &materializingNotariusRunner{cfg: cfg.Pipeline.Notarius}
|
||||||
|
return cfg, &stage.Env{Notarius: runner}, runner
|
||||||
|
}
|
||||||
177
internal/app/operator_artifact_rendering.go
Normal file
177
internal/app/operator_artifact_rendering.go
Normal file
@@ -0,0 +1,177 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
|
)
|
||||||
|
|
||||||
|
func buildHelperArtifactCatalog(cfg *config.Config, m *manifest.Manifest) (*artifacts.ArtifactCatalog, error) {
|
||||||
|
catalog := artifacts.NewArtifactCatalog()
|
||||||
|
if err := catalog.RegisterBuiltIns(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
configured := map[string]artifacts.ConfiguredArtifactDefinition{}
|
||||||
|
if cfg.Pipeline.Scriptorium != nil {
|
||||||
|
for key, item := range cfg.Pipeline.Scriptorium.Artifacts {
|
||||||
|
configured[key] = artifacts.ConfiguredArtifactDefinition{Enabled: item.Enabled, OutputPath: item.OutputPath}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := catalog.RegisterConfiguredArtifacts(configured, nil); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
extractionDefinitions := artifacts.ExtractionDefinitionsFromConfig(cfg.Pipeline.Notarius)
|
||||||
|
if err := catalog.RegisterExtractionArtifacts(extractionDefinitions); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
if cfg.Pipeline.Notarius != nil && cfg.Pipeline.Notarius.Enabled {
|
||||||
|
catalog.HydrateExtractionArtifacts(paths, m, extractionDefinitions)
|
||||||
|
}
|
||||||
|
return catalog, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeArtifactList(out io.Writer, cfg *config.Config, catalog *artifacts.ArtifactCatalog, locks *effectiveLocks, publishedRemoteState map[string]string) {
|
||||||
|
lockSet := lockSourceSet(locks.All)
|
||||||
|
fmt.Fprintln(out, "Built-in:")
|
||||||
|
for _, transcript := range artifacts.RuntimeTranscriptArtifacts() {
|
||||||
|
writeArtifactLine(out, transcript.SourceID, lockSet)
|
||||||
|
}
|
||||||
|
writeArtifactLine(out, artifacts.ArtifactBoundsSession, lockSet)
|
||||||
|
fmt.Fprintln(out, "Configured:")
|
||||||
|
for _, entry := range catalog.ListConfigured() {
|
||||||
|
writeArtifactLine(out, entry.SourceID, lockSet)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Extraction:")
|
||||||
|
for _, entry := range catalog.ListExtraction() {
|
||||||
|
state := "unavailable"
|
||||||
|
if entry.Available {
|
||||||
|
state = "available"
|
||||||
|
}
|
||||||
|
writeExtractionArtifactLine(out, entry.SourceID, state, entry.Provenance, lockSet)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Previous-session:")
|
||||||
|
for _, req := range artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg)) {
|
||||||
|
fmt.Fprintf(out, "- %s required=%t\n", artifactpolicy.PreviousSessionSourceID(req.Name), req.Required)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Published:")
|
||||||
|
for _, rule := range cfg.Pipeline.Publish.Outputs {
|
||||||
|
writePublishedOutputLine(out, rule, catalog, lockSet, publishedRemoteState)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeExtractionArtifactLine(out io.Writer, source, state, provenance string, lockSet map[string]config.PublishLockRule) {
|
||||||
|
parts := []string{source, "planned", state}
|
||||||
|
if strings.TrimSpace(provenance) != "" {
|
||||||
|
parts = append(parts, "provenance="+strings.TrimSpace(provenance))
|
||||||
|
}
|
||||||
|
if _, ok := lockSet[source]; ok {
|
||||||
|
parts = append(parts, "locked")
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeArtifactLine(out io.Writer, source string, lockSet map[string]config.PublishLockRule) {
|
||||||
|
parts := []string{source}
|
||||||
|
if _, ok := lockSet[source]; ok {
|
||||||
|
parts = append(parts, "locked")
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||||
|
}
|
||||||
|
|
||||||
|
func writePublishedOutputLine(out io.Writer, rule config.PublishOutputRule, catalog *artifacts.ArtifactCatalog, lockSet map[string]config.PublishLockRule, remoteState map[string]string) {
|
||||||
|
source := strings.TrimSpace(rule.Source)
|
||||||
|
parts := []string{source}
|
||||||
|
if _, ok := lockSet[source]; ok {
|
||||||
|
parts = append(parts, "locked")
|
||||||
|
}
|
||||||
|
dest, showDest, err := helperPublishedOutputDest(rule, catalog)
|
||||||
|
if err != nil {
|
||||||
|
parts = append(parts, "remote=error")
|
||||||
|
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if showDest {
|
||||||
|
parts = append(parts, "dest="+dest)
|
||||||
|
}
|
||||||
|
if state := remoteState[publishedOutputRemoteStateKey(source, dest)]; state != "" {
|
||||||
|
parts = append(parts, state)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||||
|
}
|
||||||
|
|
||||||
|
func remotePublishedOutputAvailability(ctx context.Context, cfg *config.Config, store storage.ObjectStore, catalog *artifacts.ArtifactCatalog) map[string]string {
|
||||||
|
out := map[string]string{}
|
||||||
|
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
for _, rule := range cfg.Pipeline.Publish.Outputs {
|
||||||
|
source := strings.TrimSpace(rule.Source)
|
||||||
|
dest, _, err := helperPublishedOutputDest(rule, catalog)
|
||||||
|
if err != nil {
|
||||||
|
out[publishedOutputRemoteStateKey(source, "")] = "remote=error"
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
key := artifacts.S3PublishedOutputKey(sessionPrefix, dest)
|
||||||
|
if exists, err := store.Exists(ctx, key); err == nil && exists {
|
||||||
|
out[publishedOutputRemoteStateKey(source, dest)] = "remote=published"
|
||||||
|
} else if err != nil {
|
||||||
|
out[publishedOutputRemoteStateKey(source, dest)] = "remote=error"
|
||||||
|
} else {
|
||||||
|
out[publishedOutputRemoteStateKey(source, dest)] = "remote=missing"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func helperPublishedOutputDest(rule config.PublishOutputRule, catalog *artifacts.ArtifactCatalog) (string, bool, error) {
|
||||||
|
source := strings.TrimSpace(rule.Source)
|
||||||
|
normalized, err := artifactpolicy.ResolvePublishedDestinationWithExtractions(
|
||||||
|
source,
|
||||||
|
rule.Dest,
|
||||||
|
helperConfiguredOutputPathMap(catalog),
|
||||||
|
helperExtractionOutputSet(catalog),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
return "", false, err
|
||||||
|
}
|
||||||
|
entry, ok := catalog.Lookup(source)
|
||||||
|
showDest := !ok || strings.TrimSpace(entry.CanonicalRelPath) != normalized
|
||||||
|
return normalized, showDest, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func helperExtractionOutputSet(catalog *artifacts.ArtifactCatalog) map[string]struct{} {
|
||||||
|
out := map[string]struct{}{}
|
||||||
|
if catalog == nil {
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
for _, entry := range catalog.ListExtraction() {
|
||||||
|
if strings.TrimSpace(entry.ExtractionKey) != "" {
|
||||||
|
out[entry.ExtractionKey] = struct{}{}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func helperConfiguredOutputPathMap(catalog *artifacts.ArtifactCatalog) map[string]string {
|
||||||
|
out := map[string]string{}
|
||||||
|
if catalog == nil {
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
for _, entry := range catalog.ListConfigured() {
|
||||||
|
if strings.TrimSpace(entry.ConfiguredKey) == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out[entry.ConfiguredKey] = strings.TrimSpace(entry.CanonicalRelPath)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func publishedOutputRemoteStateKey(source, dest string) string {
|
||||||
|
return strings.TrimSpace(source) + "\x00" + strings.TrimSpace(dest)
|
||||||
|
}
|
||||||
39
internal/app/operator_artifacts_list.go
Normal file
39
internal/app/operator_artifacts_list.go
Normal file
@@ -0,0 +1,39 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ArtifactsList lists effective artifact sources.
|
||||||
|
func ArtifactsList(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("artifacts list", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
var remote bool
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
fs.BoolVar(&remote, "remote", false, "inspect remote publish availability")
|
||||||
|
if err := parseSessionAwareFlags("artifacts list", fs, args, &flags.sessionID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("artifacts list: session_id is required")
|
||||||
|
}
|
||||||
|
cfg, store, locks, m, err := loadHelperContext(ctx, flags, remote)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("artifacts list: %w", err)
|
||||||
|
}
|
||||||
|
catalog, err := buildHelperArtifactCatalog(cfg, m)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("artifacts list: %w", err)
|
||||||
|
}
|
||||||
|
publishedRemoteState := map[string]string{}
|
||||||
|
if remote && store != nil {
|
||||||
|
publishedRemoteState = remotePublishedOutputAvailability(ctx, cfg, store, catalog)
|
||||||
|
}
|
||||||
|
writeArtifactList(out, cfg, catalog, locks, publishedRemoteState)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
140
internal/app/operator_findings.go
Normal file
140
internal/app/operator_findings.go
Normal file
@@ -0,0 +1,140 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
|
)
|
||||||
|
|
||||||
|
type finding struct {
|
||||||
|
Severity string
|
||||||
|
Category string
|
||||||
|
Message string
|
||||||
|
}
|
||||||
|
|
||||||
|
type findingError struct {
|
||||||
|
count int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e findingError) Error() string {
|
||||||
|
return fmt.Sprintf("%d validation error(s)", e.count)
|
||||||
|
}
|
||||||
|
|
||||||
|
func renderFindings(out io.Writer, campaign, sessionID string, findings []finding) error {
|
||||||
|
if campaign != "" || sessionID != "" {
|
||||||
|
fmt.Fprintf(out, "Campaign: %s\n", campaign)
|
||||||
|
fmt.Fprintf(out, "Session: %s\n\n", sessionID)
|
||||||
|
}
|
||||||
|
errorsCount := 0
|
||||||
|
for _, f := range findings {
|
||||||
|
if f.Severity == "ERROR" {
|
||||||
|
errorsCount++
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "%-5s %-10s %s\n", f.Severity, f.Category, f.Message)
|
||||||
|
}
|
||||||
|
if errorsCount > 0 {
|
||||||
|
return findingError{count: errorsCount}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func okFinding(category, msg string) finding { return finding{"OK", category, msg} }
|
||||||
|
func infoFinding(category, msg string) finding { return finding{"INFO", category, msg} }
|
||||||
|
func warnFinding(category, msg string) finding { return finding{"WARN", category, msg} }
|
||||||
|
func errorFinding(category, msg string) finding { return finding{"ERROR", category, msg} }
|
||||||
|
|
||||||
|
func sessionSourceSummary(cfg *config.Config) string {
|
||||||
|
source := cfg.SessionSource.Source
|
||||||
|
if source == "" {
|
||||||
|
source = "session_config"
|
||||||
|
}
|
||||||
|
if cfg.SessionSource.S3Key != "" {
|
||||||
|
return source + " " + cfg.SessionSource.S3Key
|
||||||
|
}
|
||||||
|
return source + " " + cfg.SessionPath
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateStableInputFindings(cfg *config.Config) []finding {
|
||||||
|
checks := inspectStableInputs(cfg)
|
||||||
|
out := make([]finding, 0, len(checks))
|
||||||
|
for _, check := range checks {
|
||||||
|
if check.Err != nil {
|
||||||
|
msg := check.Name + ": " + check.Err.Error()
|
||||||
|
if strings.TrimSpace(check.Path) != "" {
|
||||||
|
msg = fmt.Sprintf("%s missing: %v", check.Name, check.Err)
|
||||||
|
}
|
||||||
|
out = append(out, errorFinding("inputs", msg))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, okFinding("inputs", check.Name+": "+check.Path))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveHelperConfigRelativePath(input config.ResolvedInputFile) (string, error) {
|
||||||
|
if strings.TrimSpace(input.ConfigPath) == "" {
|
||||||
|
return "", fmt.Errorf("source config path is required")
|
||||||
|
}
|
||||||
|
path := strings.TrimSpace(input.Path)
|
||||||
|
if path == "" {
|
||||||
|
return "", fmt.Errorf("path is required")
|
||||||
|
}
|
||||||
|
if filepath.IsAbs(path) {
|
||||||
|
return filepath.Clean(path), nil
|
||||||
|
}
|
||||||
|
return filepath.Clean(filepath.Join(filepath.Dir(input.ConfigPath), path)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateLocalAudioFindings(cfg *config.Config) []finding {
|
||||||
|
check := inspectLocalAudioPresence(cfg)
|
||||||
|
if !check.Checked {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if check.Err != nil {
|
||||||
|
return []finding{errorFinding("audio", check.Err.Error())}
|
||||||
|
}
|
||||||
|
return []finding{okFinding("audio", fmt.Sprintf("%d local audio file(s)", len(check.Paths)))}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateRemoteAudioFinding(ctx context.Context, cfg *config.Config, store storage.ObjectStore) finding {
|
||||||
|
check := inspectRemoteAudioPresence(ctx, cfg, store)
|
||||||
|
if check.Err != nil {
|
||||||
|
return errorFinding("audio", check.Err.Error())
|
||||||
|
}
|
||||||
|
return okFinding("audio", fmt.Sprintf("%d remote .flac object(s)", len(check.Keys)))
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadLocalManifest(ctx context.Context, path string) (*manifest.Manifest, error) {
|
||||||
|
if _, err := os.Stat(path); err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
store := &manifest.LocalStore{}
|
||||||
|
return store.Load(ctx, path)
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStageStatuses(out io.Writer, m *manifest.Manifest) {
|
||||||
|
if m == nil || len(m.Stages) == 0 {
|
||||||
|
fmt.Fprintln(out, "stages: no stages recorded")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "stages:")
|
||||||
|
names := make([]string, 0, len(m.Stages))
|
||||||
|
for name := range m.Stages {
|
||||||
|
names = append(names, name)
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
for _, name := range names {
|
||||||
|
fmt.Fprintf(out, "- %s: %s\n", name, m.Stages[name].Status)
|
||||||
|
}
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -8,8 +8,10 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
@@ -109,12 +111,16 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`), 0o644); err != nil {
|
`), 0o644); err != nil {
|
||||||
t.Fatalf("write explicit campaign: %v", err)
|
t.Fatalf("write explicit campaign: %v", err)
|
||||||
}
|
}
|
||||||
mustWriteTestFile(t, filepath.Join(explicitDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
mustWriteTestFile(t, filepath.Join(explicitDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(explicitDir, "autocorrect.yml"), "[]\n")
|
mustWriteTestFile(t, filepath.Join(explicitDir, "autocorrect.yml"), "[]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(explicitDir, "glossary.yml"), "[]\n")
|
mustWriteTestFile(t, filepath.Join(explicitDir, "glossary.yml"), "[]\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(explicitDir, "players.yml"), "[]\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(explicitDir, "party.yml"), "[]\n")
|
||||||
|
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
var storeInitCalls int
|
var storeInitCalls int
|
||||||
@@ -445,6 +451,9 @@ inputs:
|
|||||||
if !strings.Contains(stdout.String(), "OK audio") {
|
if !strings.Contains(stdout.String(), "OK audio") {
|
||||||
t.Fatalf("stdout = %q, want OK audio", stdout.String())
|
t.Fatalf("stdout = %q, want OK audio", stdout.String())
|
||||||
}
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "OK inputs players:") || !strings.Contains(stdout.String(), "OK inputs party:") {
|
||||||
|
t.Fatalf("stdout = %q, want players and party input findings", stdout.String())
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteLocksAddListAndRemoveUseRemoteLockStore(t *testing.T) {
|
func TestExecuteLocksAddListAndRemoveUseRemoteLockStore(t *testing.T) {
|
||||||
@@ -501,7 +510,7 @@ func TestExecuteLocksAddListAndRemoveUseRemoteLockStore(t *testing.T) {
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("locks remove exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("locks remove exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
store, err := config.LoadPublishLockStoreBytes("locks.yml", fake.Objects[key].Data, nil)
|
store, err := config.LoadPublishLockStoreBytes("locks.yml", fake.Objects[key].Data, nil, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("LoadPublishLockStoreBytes() error = %v", err)
|
t.Fatalf("LoadPublishLockStoreBytes() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -593,10 +602,57 @@ func TestExecuteLocksRequireSessionID(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteLocksMutationRejectsSessionIDMismatch(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
args []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "add mismatch",
|
||||||
|
args: []string{
|
||||||
|
"session", "locks", "add", "2026-05-03", "narratio.transcript.final_trimmed",
|
||||||
|
"--session-id", "2026-05-04",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "remove mismatch",
|
||||||
|
args: []string{
|
||||||
|
"session", "locks", "remove", "2026-05-03", "narratio.transcript.final_trimmed",
|
||||||
|
"--session-id", "2026-05-04",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute(tt.args, &stdout, &stderr)
|
||||||
|
if code == 0 {
|
||||||
|
t.Fatal("exit code = 0, want non-zero")
|
||||||
|
}
|
||||||
|
if !strings.Contains(stderr.String(), "does not match expected session id") {
|
||||||
|
t.Fatalf("stderr = %q, want session-id mismatch guidance", stderr.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteLocksCannotModifyStaticLocks(t *testing.T) {
|
func TestExecuteLocksCannotModifyStaticLocks(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
addStaticArchiveLockToPipelineConfig(t, pipelinePath, "narratio.transcript.final_trimmed")
|
addStaticPublishLockToPipelineConfig(t, pipelinePath, "narratio.transcript.final_trimmed")
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
var storeInitCalls int
|
var storeInitCalls int
|
||||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
@@ -683,10 +739,10 @@ func addSessionTemplateToCampaign(t *testing.T, campaignPath, templateFile strin
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteArtifactsListRemoteReportsPromotedAvailability(t *testing.T) {
|
func TestExecuteArtifactsListRemoteReportsPublishedAvailability(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
addPublishOutputsToPipeline(t, pipelinePath, `
|
||||||
outputs:
|
outputs:
|
||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
@@ -714,14 +770,79 @@ func TestExecuteArtifactsListRemoteReportsPromotedAvailability(t *testing.T) {
|
|||||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "narratio.transcript.final_trimmed remote=published") {
|
if !strings.Contains(stdout.String(), "narratio.transcript.final_trimmed remote=published") {
|
||||||
t.Fatalf("stdout = %q, want promoted remote availability", stdout.String())
|
t.Fatalf("stdout = %q, want published remote availability", stdout.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteArtifactsListRemoteUsesPromotionDestinations(t *testing.T) {
|
func TestExecuteArtifactsListReportsExtractionLifecycleWithoutPayload(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
addExtractionOutputToPipeline(t, pipelinePath)
|
||||||
|
addPublishOutputsToPipeline(t, pipelinePath, `
|
||||||
|
outputs:
|
||||||
|
- source: narratio.extraction.encounters
|
||||||
|
dest: artifacts/encounters.json
|
||||||
|
required: true
|
||||||
|
`)
|
||||||
|
lanePath := writeOperatorExtractionManifest(t, workspaceRoot)
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
publishedKey := artifacts.S3PublishedOutputKey(
|
||||||
|
artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"),
|
||||||
|
"artifacts/encounters.json",
|
||||||
|
)
|
||||||
|
fake.SeedObject(storage.FakeObject{Key: publishedKey, Data: []byte(`{"secret":"DO_NOT_PRINT"}`)})
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{
|
||||||
|
"session", "artifacts", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
"--remote",
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
out := stdout.String()
|
||||||
|
for _, want := range []string{
|
||||||
|
"Extraction:",
|
||||||
|
"narratio.extraction.encounters planned available provenance=manifest.current_extract_run",
|
||||||
|
"narratio.extraction.encounters dest=artifacts/encounters.json remote=published",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(out, want) {
|
||||||
|
t.Fatalf("stdout = %q, want %q", out, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if strings.Contains(out, "DO_NOT_PRINT") {
|
||||||
|
t.Fatalf("operator output exposed extraction payload: %q", out)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := os.Remove(lanePath); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
stdout.Reset()
|
||||||
|
stderr.Reset()
|
||||||
|
code = Execute([]string{
|
||||||
|
"session", "artifacts", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("unavailable exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "narratio.extraction.encounters planned unavailable") {
|
||||||
|
t.Fatalf("stdout = %q, want unavailable extraction state", stdout.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteArtifactsListRemoteUsesPublishOutputDestinations(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
addPublishOutputsToPipeline(t, pipelinePath, `
|
||||||
outputs:
|
outputs:
|
||||||
- source: narratio.transcript.final
|
- source: narratio.transcript.final
|
||||||
dest: transcripts/full.json
|
dest: transcripts/full.json
|
||||||
@@ -771,7 +892,7 @@ func TestExecuteArtifactsListRemoteUsesPromotionDestinations(t *testing.T) {
|
|||||||
func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
addPublishOutputsToPipeline(t, pipelinePath, `
|
||||||
outputs:
|
outputs:
|
||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
@@ -782,7 +903,7 @@ func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
|||||||
`)
|
`)
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03")
|
sessionPrefix := artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03")
|
||||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
trimmedKey := artifacts.S3PublishedOutputKey(sessionPrefix, "transcripts/final.trimmed.json")
|
trimmedKey := artifacts.S3PublishedOutputKey(sessionPrefix, "transcripts/final.trimmed.json")
|
||||||
fullKey := artifacts.S3PublishedOutputKey(sessionPrefix, "transcripts/full.json")
|
fullKey := artifacts.S3PublishedOutputKey(sessionPrefix, "transcripts/full.json")
|
||||||
lockKey := artifacts.S3SessionLocksKey(sessionPrefix)
|
lockKey := artifacts.S3SessionLocksKey(sessionPrefix)
|
||||||
@@ -815,6 +936,8 @@ func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
|||||||
"narratio.transcript.final_trimmed locked",
|
"narratio.transcript.final_trimmed locked",
|
||||||
"narratio.transcript.final_trimmed locked remote=published",
|
"narratio.transcript.final_trimmed locked remote=published",
|
||||||
"narratio.transcript.final dest=transcripts/full.json remote=published",
|
"narratio.transcript.final dest=transcripts/full.json remote=published",
|
||||||
|
"Stable input players:",
|
||||||
|
"Stable input party:",
|
||||||
} {
|
} {
|
||||||
if !strings.Contains(out, want) {
|
if !strings.Contains(out, want) {
|
||||||
t.Fatalf("stdout = %q, want %q", out, want)
|
t.Fatalf("stdout = %q, want %q", out, want)
|
||||||
@@ -828,7 +951,7 @@ func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
|||||||
func TestExecuteStatusReportsRemoteArtifactCatalogErrorsWithoutFailing(t *testing.T) {
|
func TestExecuteStatusReportsRemoteArtifactCatalogErrorsWithoutFailing(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
addArchivePromotionsToPipeline(t, pipelinePath, `
|
addPublishOutputsToPipeline(t, pipelinePath, `
|
||||||
outputs:
|
outputs:
|
||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
@@ -851,19 +974,98 @@ func TestExecuteStatusReportsRemoteArtifactCatalogErrorsWithoutFailing(t *testin
|
|||||||
}
|
}
|
||||||
out := stdout.String()
|
out := stdout.String()
|
||||||
if !strings.Contains(out, "Remote publish: missing or unavailable:") {
|
if !strings.Contains(out, "Remote publish: missing or unavailable:") {
|
||||||
t.Fatalf("stdout = %q, want remote archive unavailable state", out)
|
t.Fatalf("stdout = %q, want remote publish unavailable state", out)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out, "Remote outputs:") || !strings.Contains(out, "narratio.transcript.final_trimmed remote=error") {
|
if !strings.Contains(out, "Remote outputs:") || !strings.Contains(out, "narratio.transcript.final_trimmed remote=error") {
|
||||||
t.Fatalf("stdout = %q, want remote output error state", out)
|
t.Fatalf("stdout = %q, want remote output error state", out)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out, "Publish locks: error:") {
|
if !strings.Contains(out, "Publish locks: error:") {
|
||||||
t.Fatalf("stdout = %q, want archive locks error", out)
|
t.Fatalf("stdout = %q, want publish locks error", out)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteArchiveLoadsRemoteLocks(t *testing.T) {
|
func TestExecuteStatusReportsMissingRemoteCurrentStateWithoutFailing(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidArchiveConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{
|
||||||
|
"session", "status", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "Remote publish: missing or unavailable: remote current run pointer missing") {
|
||||||
|
t.Fatalf("stdout = %q, want missing remote current-state line", stdout.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteStatusReportsPreviousStateReadinessWithoutFailing(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
|
replaceInFileOrFatal(t, pipelinePath, "source: narratio.artifact.session_recap", "source: narratio.previous_session.artifact.session_recap")
|
||||||
|
replaceInFileOrFatal(t, sessionPath, "session_id: 2026-05-03\n", "session_id: 2026-05-03\nprevious_session_id: 2026-04-26\n")
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{
|
||||||
|
"session", "status", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "Previous-session artifacts: unavailable: remote current run pointer missing") {
|
||||||
|
t.Fatalf("stdout = %q, want previous readiness unavailable line", stdout.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteSessionValidateReportsPreviousStateFindingAndReturnsFindingError(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
|
replaceInFileOrFatal(t, pipelinePath, "source: narratio.artifact.session_recap", "source: narratio.previous_session.artifact.session_recap")
|
||||||
|
replaceInFileOrFatal(t, sessionPath, "session_id: 2026-05-03\n", "session_id: 2026-05-03\nprevious_session_id: 2026-04-26\n")
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{
|
||||||
|
"session", "validate", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code == 0 {
|
||||||
|
t.Fatal("exit code = 0, want non-zero")
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "ERROR previous") {
|
||||||
|
t.Fatalf("stdout = %q, want previous finding error", stdout.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "remote current run pointer missing") {
|
||||||
|
t.Fatalf("stdout = %q, want missing run pointer finding", stdout.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stderr.String(), "validation error(s)") {
|
||||||
|
t.Fatalf("stderr = %q, want finding error summary", stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecutePublishLoadsRemoteLocks(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidPublishRunConfigFiles(t, workspaceRoot)
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
lockKey := artifacts.S3SessionLocksKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"))
|
lockKey := artifacts.S3SessionLocksKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"))
|
||||||
fake.SeedObject(storage.FakeObject{Key: lockKey, Data: []byte("locks:\n - source: narratio.transcript.final_trimmed\n reason: remote review\n")})
|
fake.SeedObject(storage.FakeObject{Key: lockKey, Data: []byte("locks:\n - source: narratio.transcript.final_trimmed\n reason: remote review\n")})
|
||||||
@@ -871,11 +1073,13 @@ func TestExecuteArchiveLoadsRemoteLocks(t *testing.T) {
|
|||||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze"} {
|
||||||
// The archive stage only checks the manifest statuses and source files.
|
// The publish stage only checks the manifest statuses and source files.
|
||||||
_ = stageName
|
_ = stageName
|
||||||
}
|
}
|
||||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.trimmed.json"), `{"segments":[]}`)
|
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.trimmed.json"), `{"segments":[]}`)
|
||||||
|
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.md"), "# final\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.trimmed.md"), "# final trimmed\n")
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
var stdout bytes.Buffer
|
||||||
var stderr bytes.Buffer
|
var stderr bytes.Buffer
|
||||||
@@ -883,28 +1087,109 @@ func TestExecuteArchiveLoadsRemoteLocks(t *testing.T) {
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
promotedKey := artifacts.S3PublishedOutputKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"), "transcripts/final.trimmed.json")
|
publishedKey := artifacts.S3PublishedOutputKey(artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"), "transcripts/final.trimmed.json")
|
||||||
if _, ok := fake.Objects[promotedKey]; ok {
|
if _, ok := fake.Objects[publishedKey]; ok {
|
||||||
t.Fatalf("locked promoted key %q was uploaded", promotedKey)
|
t.Fatalf("locked published key %q was uploaded", publishedKey)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func addArchivePromotionsToPipeline(t *testing.T, pipelinePath, archiveYAML string) {
|
func addPublishOutputsToPipeline(t *testing.T, pipelinePath, publishYAML string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
data, err := os.ReadFile(pipelinePath)
|
data, err := os.ReadFile(pipelinePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("read pipeline: %v", err)
|
t.Fatalf("read pipeline: %v", err)
|
||||||
}
|
}
|
||||||
updated := strings.Replace(string(data), " upload_run: false\n", " upload_run: false\n"+archiveYAML, 1)
|
updated := strings.Replace(string(data), " upload_run: false\n", " upload_run: false\n"+publishYAML, 1)
|
||||||
if updated == string(data) {
|
if updated == string(data) {
|
||||||
t.Fatalf("pipeline %q did not contain archive upload_run marker", pipelinePath)
|
t.Fatalf("pipeline %q did not contain publish upload_run marker", pipelinePath)
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(pipelinePath, []byte(updated), 0o644); err != nil {
|
if err := os.WriteFile(pipelinePath, []byte(updated), 0o644); err != nil {
|
||||||
t.Fatalf("write pipeline: %v", err)
|
t.Fatalf("write pipeline: %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeValidArchiveConfigFiles(t *testing.T, workspaceRoot string) (string, string, string) {
|
func addExtractionOutputToPipeline(t *testing.T, pipelinePath string) {
|
||||||
|
t.Helper()
|
||||||
|
data, err := os.ReadFile(pipelinePath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
data = append(data, []byte(`notarius:
|
||||||
|
enabled: true
|
||||||
|
config_path: notarius.yml
|
||||||
|
pipeline_id: campaign.extract
|
||||||
|
outputs:
|
||||||
|
encounters:
|
||||||
|
lane_id: encounters
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: encounters
|
||||||
|
schema_version: "1"
|
||||||
|
module_key: encounters
|
||||||
|
`)...)
|
||||||
|
if err := os.WriteFile(pipelinePath, data, 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(filepath.Dir(pipelinePath), "notarius.yml"), []byte("{}\n"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeOperatorExtractionManifest(t *testing.T, workspaceRoot string) string {
|
||||||
|
t.Helper()
|
||||||
|
paths := artifacts.NewLocalStore(workspaceRoot).SessionPathsFor("sample-campaign", "2026-05-03")
|
||||||
|
bundleRoot := filepath.Join(paths.ArtifactsDir, "notarius", "extract-run-1")
|
||||||
|
lanePath := filepath.Join(bundleRoot, "lanes", "encounters.json")
|
||||||
|
indexPath := filepath.Join(bundleRoot, "index.json")
|
||||||
|
mustWriteTestFile(t, lanePath, `{"secret":"DO_NOT_PRINT"}`)
|
||||||
|
mustWriteTestFile(t, indexPath, `{"lanes":[]}`)
|
||||||
|
laneChecksum, err := artifacts.SHA256File(lanePath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
indexChecksum, err := artifacts.SHA256File(indexPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
m := manifest.New("2026-05-03", time.Now().UTC())
|
||||||
|
m.Campaign = "sample-campaign"
|
||||||
|
m.Stages["extract"] = &manifest.StageRecord{
|
||||||
|
Name: "extract", Status: manifest.StatusSucceeded,
|
||||||
|
Metadata: map[string]any{
|
||||||
|
"narratio_run_id": "extract-run-1", "bundle_root": bundleRoot,
|
||||||
|
"receipt": map[string]any{"run_id": "notarius-run-1", "pipeline_id": "campaign.extract"},
|
||||||
|
},
|
||||||
|
Outputs: []manifest.ArtifactRecord{
|
||||||
|
{
|
||||||
|
Kind: "notarius_lane", SourceID: artifacts.ExtractionArtifactSourceID("encounters"), LocalPath: lanePath,
|
||||||
|
ProducerRunID: "extract-run-1", Checksum: laneChecksum,
|
||||||
|
Contract: &artifactmodel.ContractMetadata{MediaType: "application/json", SchemaID: "encounters", SchemaVersion: "1", ModuleKey: "encounters"},
|
||||||
|
ExternalProvenance: &artifactmodel.ExternalProvenance{System: "notarius", RunID: "notarius-run-1", PipelineID: "campaign.extract", ArtifactID: "encounters"},
|
||||||
|
},
|
||||||
|
{Kind: "notarius_index", LocalPath: indexPath, ProducerRunID: "extract-run-1", Checksum: indexChecksum},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
if err := (&manifest.LocalStore{}).Save(context.Background(), paths.ManifestPath, m); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return lanePath
|
||||||
|
}
|
||||||
|
|
||||||
|
func replaceInFileOrFatal(t *testing.T, path, old, new string) {
|
||||||
|
t.Helper()
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read %s: %v", path, err)
|
||||||
|
}
|
||||||
|
updated := strings.Replace(string(data), old, new, 1)
|
||||||
|
if updated == string(data) {
|
||||||
|
t.Fatalf("%s did not contain %q", path, old)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(path, []byte(updated), 0o644); err != nil {
|
||||||
|
t.Fatalf("write %s: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeValidPublishRunConfigFiles(t *testing.T, workspaceRoot string) (string, string, string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
data, err := os.ReadFile(pipelinePath)
|
data, err := os.ReadFile(pipelinePath)
|
||||||
@@ -924,7 +1209,7 @@ func writeValidArchiveConfigFiles(t *testing.T, workspaceRoot string) (string, s
|
|||||||
m := manifest.New("2026-05-03", nowUTC())
|
m := manifest.New("2026-05-03", nowUTC())
|
||||||
m.Campaign = "sample-campaign"
|
m.Campaign = "sample-campaign"
|
||||||
m.RunID = "20260521T160000Z-test"
|
m.RunID = "20260521T160000Z-test"
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze"} {
|
||||||
m.MarkStageSucceeded(name, nowUTC(), nil)
|
m.MarkStageSucceeded(name, nowUTC(), nil)
|
||||||
}
|
}
|
||||||
path := artifacts.SessionManifestPathForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
path := artifacts.SessionManifestPathForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
@@ -941,7 +1226,7 @@ func writeValidArchiveConfigFiles(t *testing.T, workspaceRoot string) (string, s
|
|||||||
return pipelinePath, campaignPath, sessionPath
|
return pipelinePath, campaignPath, sessionPath
|
||||||
}
|
}
|
||||||
|
|
||||||
func addStaticArchiveLockToPipelineConfig(t *testing.T, pipelinePath, source string) {
|
func addStaticPublishLockToPipelineConfig(t *testing.T, pipelinePath, source string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
data, err := os.ReadFile(pipelinePath)
|
data, err := os.ReadFile(pipelinePath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -954,7 +1239,7 @@ func addStaticArchiveLockToPipelineConfig(t *testing.T, pipelinePath, source str
|
|||||||
1,
|
1,
|
||||||
)
|
)
|
||||||
if updated == string(data) {
|
if updated == string(data) {
|
||||||
t.Fatalf("archive section not found in pipeline config")
|
t.Fatalf("publish section not found in pipeline config")
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(pipelinePath, []byte(updated), 0o644); err != nil {
|
if err := os.WriteFile(pipelinePath, []byte(updated), 0o644); err != nil {
|
||||||
t.Fatalf("write pipeline: %v", err)
|
t.Fatalf("write pipeline: %v", err)
|
||||||
|
|||||||
276
internal/app/operator_inspection.go
Normal file
276
internal/app/operator_inspection.go
Normal file
@@ -0,0 +1,276 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
type stableInputCheck struct {
|
||||||
|
Name string
|
||||||
|
Path string
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type localAudioCheck struct {
|
||||||
|
Checked bool
|
||||||
|
Paths []string
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type remoteAudioCheck struct {
|
||||||
|
Checked bool
|
||||||
|
Prefix string
|
||||||
|
Keys []string
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type previousArtifactReadiness struct {
|
||||||
|
Requirements []artifacts.PreviousArtifactRequirement
|
||||||
|
MissingID bool
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type remoteCurrentStateCheck struct {
|
||||||
|
State *RemoteCurrentState
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type effectiveLocksCheck struct {
|
||||||
|
Locks *effectiveLocks
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectStableInputs(cfg *config.Config) []stableInputCheck {
|
||||||
|
items := []struct {
|
||||||
|
name string
|
||||||
|
in config.ResolvedInputFile
|
||||||
|
}{
|
||||||
|
{name: "speakers", in: cfg.StableInputs.SpeakersFile},
|
||||||
|
{name: "autocorrect", in: cfg.StableInputs.AutocorrectFile},
|
||||||
|
{name: "glossary", in: cfg.StableInputs.GlossaryFile},
|
||||||
|
{name: "players", in: cfg.StableInputs.PlayersFile},
|
||||||
|
{name: "party", in: cfg.StableInputs.PartyFile},
|
||||||
|
}
|
||||||
|
out := make([]stableInputCheck, 0, len(items))
|
||||||
|
for _, item := range items {
|
||||||
|
path, err := resolveHelperConfigRelativePath(item.in)
|
||||||
|
if err != nil {
|
||||||
|
out = append(out, stableInputCheck{Name: item.name, Err: err})
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(path); err != nil {
|
||||||
|
out = append(out, stableInputCheck{Name: item.name, Path: path, Err: err})
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, stableInputCheck{Name: item.name, Path: path})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectLocalAudioPresence(cfg *config.Config) localAudioCheck {
|
||||||
|
if cfg.Session.Inputs.AudioS3 != nil {
|
||||||
|
return localAudioCheck{}
|
||||||
|
}
|
||||||
|
|
||||||
|
sessionDir := filepath.Dir(cfg.SessionPath)
|
||||||
|
resolved, err := resolveLocalInspectionAudioPaths(sessionDir, cfg.Session.Inputs)
|
||||||
|
if err != nil {
|
||||||
|
return localAudioCheck{Checked: true, Err: err}
|
||||||
|
}
|
||||||
|
return localAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Paths: resolved,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectRemoteAudioPresence(ctx context.Context, cfg *config.Config, store storage.ObjectStore) remoteAudioCheck {
|
||||||
|
if cfg.Session.Inputs.AudioS3 == nil {
|
||||||
|
return remoteAudioCheck{}
|
||||||
|
}
|
||||||
|
if store == nil {
|
||||||
|
return remoteAudioCheck{Checked: true, Err: fmt.Errorf("storage backend is required for remote audio checks")}
|
||||||
|
}
|
||||||
|
|
||||||
|
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
audioPrefix := artifacts.S3AudioPrefix(sessionPrefix, cfg.Session.Inputs.AudioS3.Prefix)
|
||||||
|
objects, err := store.List(ctx, audioPrefix)
|
||||||
|
if err != nil {
|
||||||
|
return remoteAudioCheck{Checked: true, Prefix: audioPrefix, Err: err}
|
||||||
|
}
|
||||||
|
|
||||||
|
keys := make([]string, 0, len(objects))
|
||||||
|
seenBase := map[string]string{}
|
||||||
|
for _, obj := range objects {
|
||||||
|
key := strings.TrimSpace(obj.Key)
|
||||||
|
if key == "" || strings.HasSuffix(key, "/") || !isInspectionFlacPath(key) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
base := path.Base(key)
|
||||||
|
if prev, exists := seenBase[base]; exists && prev != key {
|
||||||
|
return remoteAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Prefix: audioPrefix,
|
||||||
|
Err: fmt.Errorf("duplicate s3 audio basename %q from %q and %q", base, prev, key),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
seenBase[base] = key
|
||||||
|
keys = append(keys, key)
|
||||||
|
}
|
||||||
|
sort.Strings(keys)
|
||||||
|
if len(keys) == 0 {
|
||||||
|
return remoteAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Prefix: audioPrefix,
|
||||||
|
Err: fmt.Errorf("no .flac files found under s3 audio prefix %q", audioPrefix),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return remoteAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Prefix: audioPrefix,
|
||||||
|
Keys: keys,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectPreviousArtifactReadiness(
|
||||||
|
ctx context.Context,
|
||||||
|
cfg *config.Config,
|
||||||
|
store storage.ObjectStore,
|
||||||
|
requirements []artifacts.PreviousArtifactRequirement,
|
||||||
|
) previousArtifactReadiness {
|
||||||
|
out := previousArtifactReadiness{
|
||||||
|
Requirements: append([]artifacts.PreviousArtifactRequirement(nil), requirements...),
|
||||||
|
}
|
||||||
|
if len(requirements) == 0 {
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(cfg.Session.PreviousSessionID) == "" {
|
||||||
|
out.MissingID = true
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
if store == nil {
|
||||||
|
out.Err = fmt.Errorf("previous-session artifacts cannot be checked because storage is unavailable")
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
prefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
||||||
|
if _, err := artifacts.LoadCurrentState(ctx, store, prefix, artifacts.CurrentStateValidation{
|
||||||
|
ExpectedSessionID: strings.TrimSpace(cfg.Session.PreviousSessionID),
|
||||||
|
ExpectedCampaign: strings.TrimSpace(cfg.Session.Campaign),
|
||||||
|
ValidateRunID: true,
|
||||||
|
}); err != nil {
|
||||||
|
out.Err = fmt.Errorf("remote %v", err)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectRemoteCurrentState(ctx context.Context, cfg *config.Config, store storage.ObjectStore) remoteCurrentStateCheck {
|
||||||
|
if store == nil {
|
||||||
|
return remoteCurrentStateCheck{}
|
||||||
|
}
|
||||||
|
current, err := discoverRemoteCurrentStateFn(ctx, cfg, store)
|
||||||
|
if err != nil {
|
||||||
|
return remoteCurrentStateCheck{Err: err}
|
||||||
|
}
|
||||||
|
return remoteCurrentStateCheck{State: current}
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectEffectiveLocks(ctx context.Context, cfg *config.Config, store storage.ObjectStore) effectiveLocksCheck {
|
||||||
|
locks, err := loadEffectiveLocks(ctx, cfg, store)
|
||||||
|
if err != nil {
|
||||||
|
return effectiveLocksCheck{Err: err}
|
||||||
|
}
|
||||||
|
return effectiveLocksCheck{Locks: locks}
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveLocalInspectionAudioPaths(sessionDir string, inputs config.SessionInputsConfig) ([]string, error) {
|
||||||
|
if len(inputs.AudioFiles) > 0 {
|
||||||
|
out := make([]string, 0, len(inputs.AudioFiles))
|
||||||
|
seenBase := map[string]string{}
|
||||||
|
for _, item := range inputs.AudioFiles {
|
||||||
|
resolved, err := resolveInspectionPath(sessionDir, item)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if !isInspectionFlacPath(resolved) {
|
||||||
|
return nil, fmt.Errorf("audio file %q must have .flac extension", resolved)
|
||||||
|
}
|
||||||
|
if err := requireInspectionFile(resolved, "audio file"); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
base := filepath.Base(resolved)
|
||||||
|
if prev, exists := seenBase[base]; exists && prev != resolved {
|
||||||
|
return nil, fmt.Errorf("duplicate audio basename %q from %q and %q", base, prev, resolved)
|
||||||
|
}
|
||||||
|
seenBase[base] = resolved
|
||||||
|
out = append(out, resolved)
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
audioDir, err := resolveInspectionPath(sessionDir, inputs.AudioDir)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
entries, err := os.ReadDir(audioDir)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read audio directory %q: %w", audioDir, err)
|
||||||
|
}
|
||||||
|
out := make([]string, 0, len(entries))
|
||||||
|
for _, entry := range entries {
|
||||||
|
if entry.IsDir() {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
full := filepath.Join(audioDir, entry.Name())
|
||||||
|
if !isInspectionFlacPath(full) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err := requireInspectionFile(full, "audio file"); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, full)
|
||||||
|
}
|
||||||
|
if len(out) == 0 {
|
||||||
|
return nil, fmt.Errorf("no .flac files found in audio directory %q", audioDir)
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveInspectionPath(baseDir, inputPath string) (string, error) {
|
||||||
|
pathValue := strings.TrimSpace(inputPath)
|
||||||
|
if pathValue == "" {
|
||||||
|
return "", fmt.Errorf("path is required")
|
||||||
|
}
|
||||||
|
if filepath.IsAbs(pathValue) {
|
||||||
|
return filepath.Clean(pathValue), nil
|
||||||
|
}
|
||||||
|
return filepath.Clean(filepath.Join(baseDir, pathValue)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireInspectionFile(path, label string) error {
|
||||||
|
info, err := os.Stat(path)
|
||||||
|
if err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return fmt.Errorf("%s %q does not exist", label, path)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("stat %s %q: %w", label, path, err)
|
||||||
|
}
|
||||||
|
if info.IsDir() {
|
||||||
|
return fmt.Errorf("%s %q is a directory", label, path)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func isInspectionFlacPath(path string) bool {
|
||||||
|
return strings.EqualFold(filepath.Ext(strings.TrimSpace(path)), ".flac")
|
||||||
|
}
|
||||||
172
internal/app/operator_locks.go
Normal file
172
internal/app/operator_locks.go
Normal file
@@ -0,0 +1,172 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Locks dispatches publish lock list and mutation helpers.
|
||||||
|
func Locks(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
if len(args) > 0 && !strings.HasPrefix(args[0], "-") {
|
||||||
|
switch args[0] {
|
||||||
|
case "add":
|
||||||
|
return LocksAdd(ctx, args[1:], out)
|
||||||
|
case "remove":
|
||||||
|
return LocksRemove(ctx, args[1:], out)
|
||||||
|
default:
|
||||||
|
return fmt.Errorf("locks: unknown subcommand %q", args[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return LocksList(ctx, args, out)
|
||||||
|
}
|
||||||
|
|
||||||
|
// LocksList lists effective publish locks.
|
||||||
|
func LocksList(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("locks", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
if err := parseSessionAwareFlags("locks", fs, args, &flags.sessionID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("locks: session_id is required")
|
||||||
|
}
|
||||||
|
cfg, _, locks, _, err := loadHelperContext(ctx, flags, true)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("locks: %w", err)
|
||||||
|
}
|
||||||
|
writeLocks(out, cfg, locks)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// LocksAdd adds or updates one remote lock.
|
||||||
|
func LocksAdd(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("locks add", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
var reason string
|
||||||
|
var force bool
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
fs.StringVar(&reason, "reason", "", "lock reason")
|
||||||
|
fs.BoolVar(&force, "force", false, "update existing remote lock")
|
||||||
|
source, err := parseSessionIDAndOnePositionalArg("locks add", "source id", fs, args, &flags.sessionID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("locks add: session_id is required")
|
||||||
|
}
|
||||||
|
cfg, store, locks, _, err := loadHelperContext(ctx, flags, true)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("locks add: %w", err)
|
||||||
|
}
|
||||||
|
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, cfg.Pipeline.Notarius, "locks add"); err != nil {
|
||||||
|
return fmt.Errorf("locks add: %w", err)
|
||||||
|
}
|
||||||
|
if _, ok := lockSourceSet(locks.Static)[source]; ok {
|
||||||
|
return fmt.Errorf("locks add: source %q is locked by pipeline config and cannot be modified remotely", source)
|
||||||
|
}
|
||||||
|
remoteSet := lockSourceSet(locks.Remote)
|
||||||
|
if _, exists := remoteSet[source]; exists && !force {
|
||||||
|
return fmt.Errorf("locks add: remote lock for %q already exists; pass --force to update", source)
|
||||||
|
}
|
||||||
|
remoteSet[source] = config.PublishLockRule{Source: source, Reason: strings.TrimSpace(reason)}
|
||||||
|
remoteLocks := lockMapValues(remoteSet)
|
||||||
|
if _, err := config.ValidatePublishLockRules(remoteLocks, cfg.Pipeline.Scriptorium, cfg.Pipeline.Notarius, "locks"); err != nil {
|
||||||
|
return fmt.Errorf("locks add: %w", err)
|
||||||
|
}
|
||||||
|
if err := uploadRemoteLockStore(ctx, store, locks.Key, &config.PublishLockStore{Locks: remoteLocks}); err != nil {
|
||||||
|
return fmt.Errorf("locks add: %w", err)
|
||||||
|
}
|
||||||
|
_, err = fmt.Fprintf(out, "narratio session locks add: locked %s\n", source)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// LocksRemove removes one remote lock.
|
||||||
|
func LocksRemove(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("locks remove", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
source, err := parseSessionIDAndOnePositionalArg("locks remove", "source id", fs, args, &flags.sessionID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("locks remove: session_id is required")
|
||||||
|
}
|
||||||
|
cfg, store, locks, _, err := loadHelperContext(ctx, flags, true)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("locks remove: %w", err)
|
||||||
|
}
|
||||||
|
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, cfg.Pipeline.Notarius, "locks remove"); err != nil {
|
||||||
|
return fmt.Errorf("locks remove: %w", err)
|
||||||
|
}
|
||||||
|
remoteSet := lockSourceSet(locks.Remote)
|
||||||
|
if _, ok := remoteSet[source]; !ok {
|
||||||
|
if _, static := lockSourceSet(locks.Static)[source]; static {
|
||||||
|
return fmt.Errorf("locks remove: source %q is locked by pipeline config and cannot be unlocked remotely", source)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("locks remove: remote lock for %q does not exist", source)
|
||||||
|
}
|
||||||
|
delete(remoteSet, source)
|
||||||
|
remoteLocks := lockMapValues(remoteSet)
|
||||||
|
if err := uploadRemoteLockStore(ctx, store, locks.Key, &config.PublishLockStore{Locks: remoteLocks}); err != nil {
|
||||||
|
return fmt.Errorf("locks remove: %w", err)
|
||||||
|
}
|
||||||
|
_, err = fmt.Fprintf(out, "narratio session locks remove: unlocked %s\n", source)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeLocks(out io.Writer, cfg *config.Config, locks *effectiveLocks) {
|
||||||
|
if locks == nil || len(locks.All) == 0 {
|
||||||
|
fmt.Fprintln(out, "Publish locks: none")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Publish locks:")
|
||||||
|
published := map[string]config.PublishOutputRule{}
|
||||||
|
if cfg != nil && cfg.Pipeline != nil && cfg.Pipeline.Publish != nil {
|
||||||
|
for _, rule := range cfg.Pipeline.Publish.Outputs {
|
||||||
|
published[strings.TrimSpace(rule.Source)] = rule
|
||||||
|
}
|
||||||
|
}
|
||||||
|
staticSet := lockSourceSet(locks.Static)
|
||||||
|
for _, lock := range locks.All {
|
||||||
|
origin := "remote"
|
||||||
|
if _, ok := staticSet[lock.Source]; ok {
|
||||||
|
origin = "pipeline"
|
||||||
|
}
|
||||||
|
promo := "not-published"
|
||||||
|
if _, ok := published[lock.Source]; ok {
|
||||||
|
promo = "published"
|
||||||
|
}
|
||||||
|
reason := strings.TrimSpace(lock.Reason)
|
||||||
|
if reason == "" {
|
||||||
|
reason = "(no reason)"
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "- %s origin=%s %s reason=%s\n", lock.Source, origin, promo, reason)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func lockMapValues(in map[string]config.PublishLockRule) []config.PublishLockRule {
|
||||||
|
keys := make([]string, 0, len(in))
|
||||||
|
for key := range in {
|
||||||
|
keys = append(keys, key)
|
||||||
|
}
|
||||||
|
sort.Strings(keys)
|
||||||
|
out := make([]config.PublishLockRule, 0, len(keys))
|
||||||
|
for _, key := range keys {
|
||||||
|
item := in[key]
|
||||||
|
item.Source = key
|
||||||
|
item.Reason = strings.TrimSpace(item.Reason)
|
||||||
|
out = append(out, item)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
268
internal/app/operator_session_init.go
Normal file
268
internal/app/operator_session_init.go
Normal file
@@ -0,0 +1,268 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gopkg.in/yaml.v3"
|
||||||
|
)
|
||||||
|
|
||||||
|
// SessionInit creates a local or remote session.yml skeleton.
|
||||||
|
func SessionInit(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("session init", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var pipelinePath, campaignPath, campaignFilePath, sessionID, previousSessionID, date, title, output, audioS3Prefix, audioDir string
|
||||||
|
var remote, force bool
|
||||||
|
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
||||||
|
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
||||||
|
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
||||||
|
fs.StringVar(&sessionID, "session-id", "", "session identifier")
|
||||||
|
fs.StringVar(&previousSessionID, "previous-session-id", "", "previous session identifier")
|
||||||
|
fs.StringVar(&date, "date", "", "session date")
|
||||||
|
fs.StringVar(&title, "title", "", "session title")
|
||||||
|
fs.StringVar(&output, "output", "", "local output session.yml path")
|
||||||
|
fs.StringVar(&audioS3Prefix, "audio-s3-prefix", "", "session audio S3 prefix")
|
||||||
|
fs.StringVar(&audioDir, "audio-dir", "", "local audio directory")
|
||||||
|
fs.BoolVar(&remote, "remote", false, "write session.yml to S3 session prefix")
|
||||||
|
fs.BoolVar(&force, "force", false, "overwrite existing target")
|
||||||
|
if err := parseSessionAwareFlags("session init", fs, args, &sessionID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(sessionID) == "" {
|
||||||
|
return fmt.Errorf("session init: session_id is required")
|
||||||
|
}
|
||||||
|
if (strings.TrimSpace(output) == "") == !remote {
|
||||||
|
return fmt.Errorf("session init: specify exactly one target: --output <path> or --remote")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(audioDir) != "" && strings.TrimSpace(audioS3Prefix) != "" {
|
||||||
|
return fmt.Errorf("session init: --audio-dir and --audio-s3-prefix are mutually exclusive")
|
||||||
|
}
|
||||||
|
|
||||||
|
base, err := loadPipelineCampaignConfig(pipelinePath, campaignPath, campaignFilePath)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
input := sessionInitInput{
|
||||||
|
Campaign: config.CampaignID(base.Campaign),
|
||||||
|
CampaignPath: base.CampaignPath,
|
||||||
|
TemplateFile: base.Campaign.SessionTemplateFile,
|
||||||
|
SessionID: sessionID,
|
||||||
|
PreviousSessionID: previousSessionID,
|
||||||
|
Date: date,
|
||||||
|
Title: title,
|
||||||
|
AudioS3Prefix: audioS3Prefix,
|
||||||
|
AudioDir: audioDir,
|
||||||
|
}
|
||||||
|
data, err := buildSessionInitYAML(input)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
label := strings.TrimSpace(output)
|
||||||
|
if label == "" {
|
||||||
|
label = "remote session.yml"
|
||||||
|
}
|
||||||
|
sessionCfg, err := config.LoadSessionBytesWithOptions(label, data, config.SessionLoadOptions{
|
||||||
|
SessionID: sessionID,
|
||||||
|
PreviousSessionID: previousSessionID,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
cfg, err := config.Resolve(base.PipelinePath, base.Pipeline, base.CampaignPath, base.Campaign, label, sessionCfg, config.SessionSource{Source: "session_config", LocalPath: label})
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
if err := config.Validate(cfg); err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !remote {
|
||||||
|
if err := writeLocalFile(output, data, force); err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
_, err := fmt.Fprintf(out, "narratio session init: wrote %s\n", filepath.Clean(output))
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
store, err := newCommandObjectStore(ctx, cfg, nil)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: %w", err)
|
||||||
|
}
|
||||||
|
sessionPrefix := artifacts.S3SessionPrefix(base.Pipeline.Storage.S3.RootPrefix, config.CampaignID(base.Campaign), sessionID)
|
||||||
|
key := artifacts.S3SessionConfigKey(sessionPrefix)
|
||||||
|
exists, err := store.Exists(ctx, key)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: check remote session %q: %w", key, err)
|
||||||
|
}
|
||||||
|
if exists && !force {
|
||||||
|
return fmt.Errorf("session init: remote session %q already exists; pass --force to overwrite", key)
|
||||||
|
}
|
||||||
|
tmp, err := os.CreateTemp("", "narratio-session-init-*.yml")
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("session init: create temp file: %w", err)
|
||||||
|
}
|
||||||
|
tmpPath := tmp.Name()
|
||||||
|
defer func() { _ = os.Remove(tmpPath) }()
|
||||||
|
if _, err := tmp.Write(data); err != nil {
|
||||||
|
_ = tmp.Close()
|
||||||
|
return fmt.Errorf("session init: write temp file: %w", err)
|
||||||
|
}
|
||||||
|
if err := tmp.Close(); err != nil {
|
||||||
|
return fmt.Errorf("session init: close temp file: %w", err)
|
||||||
|
}
|
||||||
|
if _, err := store.Upload(ctx, tmpPath, key, storage.UploadOptions{ContentType: "application/x-yaml; charset=utf-8"}); err != nil {
|
||||||
|
return fmt.Errorf("session init: upload remote session %q: %w", key, err)
|
||||||
|
}
|
||||||
|
_, err = fmt.Fprintf(out, "narratio session init: wrote s3://%s/%s\n", s3BucketName(base.Pipeline), key)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildSessionYAML(campaign, sessionID, previousSessionID, date, title, audioS3Prefix, audioDir string) ([]byte, error) {
|
||||||
|
if strings.TrimSpace(date) == "" && regexp.MustCompile(`^\d{4}-\d{2}-\d{2}$`).MatchString(strings.TrimSpace(sessionID)) {
|
||||||
|
date = strings.TrimSpace(sessionID)
|
||||||
|
}
|
||||||
|
type audioS3 struct {
|
||||||
|
Prefix string `yaml:"prefix"`
|
||||||
|
}
|
||||||
|
type inputs struct {
|
||||||
|
AudioDir string `yaml:"audio_dir,omitempty"`
|
||||||
|
AudioS3 *audioS3 `yaml:"audio_s3,omitempty"`
|
||||||
|
}
|
||||||
|
type sessionYAML struct {
|
||||||
|
Campaign string `yaml:"campaign"`
|
||||||
|
SessionID string `yaml:"session_id"`
|
||||||
|
PreviousSessionID string `yaml:"previous_session_id,omitempty"`
|
||||||
|
Date string `yaml:"date,omitempty"`
|
||||||
|
Title string `yaml:"title,omitempty"`
|
||||||
|
Inputs inputs `yaml:"inputs"`
|
||||||
|
}
|
||||||
|
in := inputs{AudioDir: strings.TrimSpace(audioDir)}
|
||||||
|
if in.AudioDir == "" {
|
||||||
|
prefix := strings.TrimSpace(audioS3Prefix)
|
||||||
|
if prefix == "" {
|
||||||
|
prefix = "audio/"
|
||||||
|
}
|
||||||
|
in.AudioS3 = &audioS3{Prefix: prefix}
|
||||||
|
}
|
||||||
|
data, err := yaml.Marshal(sessionYAML{
|
||||||
|
Campaign: strings.TrimSpace(campaign),
|
||||||
|
SessionID: strings.TrimSpace(sessionID),
|
||||||
|
PreviousSessionID: strings.TrimSpace(previousSessionID),
|
||||||
|
Date: strings.TrimSpace(date),
|
||||||
|
Title: strings.TrimSpace(title),
|
||||||
|
Inputs: in,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return data, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type sessionInitInput struct {
|
||||||
|
Campaign string
|
||||||
|
CampaignPath string
|
||||||
|
TemplateFile string
|
||||||
|
SessionID string
|
||||||
|
PreviousSessionID string
|
||||||
|
Date string
|
||||||
|
Title string
|
||||||
|
AudioS3Prefix string
|
||||||
|
AudioDir string
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildSessionInitYAML(in sessionInitInput) ([]byte, error) {
|
||||||
|
if strings.TrimSpace(in.TemplateFile) == "" {
|
||||||
|
return buildSessionYAML(in.Campaign, in.SessionID, in.PreviousSessionID, in.Date, in.Title, in.AudioS3Prefix, in.AudioDir)
|
||||||
|
}
|
||||||
|
templatePath := resolveSessionInitTemplatePath(in.CampaignPath, in.TemplateFile)
|
||||||
|
templateBytes, err := os.ReadFile(templatePath)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read session template %q: %w", templatePath, err)
|
||||||
|
}
|
||||||
|
rendered, err := renderSessionInitTemplate(string(templateBytes), in)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("render session template %q: %w", templatePath, err)
|
||||||
|
}
|
||||||
|
return []byte(rendered), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveSessionInitTemplatePath(campaignPath, templateFile string) string {
|
||||||
|
templateFile = strings.TrimSpace(templateFile)
|
||||||
|
if filepath.IsAbs(templateFile) {
|
||||||
|
return filepath.Clean(templateFile)
|
||||||
|
}
|
||||||
|
return filepath.Clean(filepath.Join(filepath.Dir(campaignPath), templateFile))
|
||||||
|
}
|
||||||
|
|
||||||
|
var sessionInitTemplatePattern = regexp.MustCompile(`\{\{\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*\}\}`)
|
||||||
|
|
||||||
|
func renderSessionInitTemplate(content string, in sessionInitInput) (string, error) {
|
||||||
|
values := map[string]string{
|
||||||
|
"session_id": strings.TrimSpace(in.SessionID),
|
||||||
|
"previous_session_id": strings.TrimSpace(in.PreviousSessionID),
|
||||||
|
"date": strings.TrimSpace(in.Date),
|
||||||
|
"title": strings.TrimSpace(in.Title),
|
||||||
|
"audio_s3_prefix": strings.TrimSpace(in.AudioS3Prefix),
|
||||||
|
"audio_dir": strings.TrimSpace(in.AudioDir),
|
||||||
|
}
|
||||||
|
used := map[string]struct{}{}
|
||||||
|
unknown := map[string]struct{}{}
|
||||||
|
missing := map[string]struct{}{}
|
||||||
|
rendered := sessionInitTemplatePattern.ReplaceAllStringFunc(content, func(match string) string {
|
||||||
|
parts := sessionInitTemplatePattern.FindStringSubmatch(match)
|
||||||
|
if len(parts) < 2 {
|
||||||
|
return match
|
||||||
|
}
|
||||||
|
name := parts[1]
|
||||||
|
value, ok := values[name]
|
||||||
|
if !ok {
|
||||||
|
unknown[name] = struct{}{}
|
||||||
|
return match
|
||||||
|
}
|
||||||
|
used[name] = struct{}{}
|
||||||
|
if value == "" {
|
||||||
|
missing[name] = struct{}{}
|
||||||
|
return match
|
||||||
|
}
|
||||||
|
return value
|
||||||
|
})
|
||||||
|
if len(unknown) > 0 {
|
||||||
|
return "", fmt.Errorf("unsupported template variable(s): %s", sortedStringSet(unknown))
|
||||||
|
}
|
||||||
|
if len(missing) > 0 {
|
||||||
|
return "", fmt.Errorf("missing required template variable value(s): %s", sortedStringSet(missing))
|
||||||
|
}
|
||||||
|
unused := map[string]struct{}{}
|
||||||
|
for _, name := range []string{"previous_session_id", "date", "title", "audio_s3_prefix", "audio_dir"} {
|
||||||
|
if values[name] == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, ok := used[name]; !ok {
|
||||||
|
unused[name] = struct{}{}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(unused) > 0 {
|
||||||
|
return "", fmt.Errorf("unused template variable value(s): %s", sortedStringSet(unused))
|
||||||
|
}
|
||||||
|
return rendered, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func sortedStringSet(set map[string]struct{}) string {
|
||||||
|
items := make([]string, 0, len(set))
|
||||||
|
for item := range set {
|
||||||
|
items = append(items, item)
|
||||||
|
}
|
||||||
|
sort.Strings(items)
|
||||||
|
return strings.Join(items, ", ")
|
||||||
|
}
|
||||||
84
internal/app/operator_session_validate.go
Normal file
84
internal/app/operator_session_validate.go
Normal file
@@ -0,0 +1,84 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// SessionValidate performs a read-only session preflight.
|
||||||
|
func SessionValidate(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("session validate", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
if err := parseSessionAwareFlags("session validate", fs, args, &flags.sessionID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("session validate: session_id is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
findings := []finding{}
|
||||||
|
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||||
|
if err != nil {
|
||||||
|
findings = append(findings, errorFinding("config", err.Error()))
|
||||||
|
return renderFindings(out, "", "", findings)
|
||||||
|
}
|
||||||
|
if err := config.Validate(cfg); err != nil {
|
||||||
|
findings = append(findings, errorFinding("config", err.Error()))
|
||||||
|
} else {
|
||||||
|
findings = append(findings, okFinding("config", "resolved pipeline, campaign, and session config"))
|
||||||
|
}
|
||||||
|
findings = append(findings, okFinding("session", fmt.Sprintf("session source: %s", sessionSourceSummary(cfg))))
|
||||||
|
|
||||||
|
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
findings = append(findings, validateStableInputFindings(cfg)...)
|
||||||
|
findings = append(findings, validateLocalAudioFindings(cfg)...)
|
||||||
|
|
||||||
|
store, storeErr := objectStoreIfConfigured(ctx, cfg)
|
||||||
|
if storeErr != nil {
|
||||||
|
findings = append(findings, errorFinding("storage", storeErr.Error()))
|
||||||
|
}
|
||||||
|
if cfg.Session.Inputs.AudioS3 != nil {
|
||||||
|
if storeErr != nil {
|
||||||
|
findings = append(findings, errorFinding("audio", "remote audio cannot be checked because storage is unavailable"))
|
||||||
|
} else {
|
||||||
|
findings = append(findings, validateRemoteAudioFinding(ctx, cfg, store))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
requirements := artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg))
|
||||||
|
previous := inspectPreviousArtifactReadiness(ctx, cfg, store, requirements)
|
||||||
|
if len(previous.Requirements) == 0 {
|
||||||
|
findings = append(findings, okFinding("previous", "no previous-session artifacts required"))
|
||||||
|
} else if previous.MissingID {
|
||||||
|
findings = append(findings, errorFinding("previous", "previous_session_id is required by configured previous-session artifacts"))
|
||||||
|
} else if previous.Err != nil {
|
||||||
|
findings = append(findings, errorFinding("previous", previous.Err.Error()))
|
||||||
|
} else {
|
||||||
|
for _, req := range previous.Requirements {
|
||||||
|
findings = append(findings, okFinding("previous", fmt.Sprintf("%s required=%t", req.Name, req.Required)))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
locks := inspectEffectiveLocks(ctx, cfg, store)
|
||||||
|
if locks.Err != nil {
|
||||||
|
findings = append(findings, errorFinding("locks", locks.Err.Error()))
|
||||||
|
} else if len(locks.Locks.All) == 0 {
|
||||||
|
findings = append(findings, okFinding("locks", "no effective publish locks"))
|
||||||
|
} else {
|
||||||
|
for _, lock := range locks.Locks.All {
|
||||||
|
findings = append(findings, warnFinding("locks", fmt.Sprintf("%s locked: %s", lock.Source, strings.TrimSpace(lock.Reason))))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if paths.ManifestPath != "" {
|
||||||
|
findings = append(findings, infoFinding("workspace", "manifest path: "+paths.ManifestPath))
|
||||||
|
}
|
||||||
|
return renderFindings(out, cfg.Session.Campaign, cfg.Session.SessionID, findings)
|
||||||
|
}
|
||||||
169
internal/app/operator_status.go
Normal file
169
internal/app/operator_status.go
Normal file
@@ -0,0 +1,169 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Status reports effective local/remote session state.
|
||||||
|
func Status(ctx context.Context, args []string, out io.Writer) error {
|
||||||
|
fs := flag.NewFlagSet("status", flag.ContinueOnError)
|
||||||
|
fs.SetOutput(io.Discard)
|
||||||
|
var flags commonConfigFlags
|
||||||
|
addCommonConfigFlags(fs, &flags)
|
||||||
|
if err := parseSessionAwareFlags("status", fs, args, &flags.sessionID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
|
return fmt.Errorf("status: session_id is required")
|
||||||
|
}
|
||||||
|
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("status: %w", err)
|
||||||
|
}
|
||||||
|
if err := config.Validate(cfg); err != nil {
|
||||||
|
return fmt.Errorf("status: %w", err)
|
||||||
|
}
|
||||||
|
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
fmt.Fprintf(out, "Session: %s\n", cfg.Session.SessionID)
|
||||||
|
fmt.Fprintf(out, "Campaign: %s\n", cfg.Session.Campaign)
|
||||||
|
fmt.Fprintf(out, "Workspace: %s\n", paths.Root)
|
||||||
|
fmt.Fprintf(out, "Session config: %s\n", sessionSourceSummary(cfg))
|
||||||
|
writeStatusStableInputs(out, inspectStableInputs(cfg))
|
||||||
|
writeStatusLocalAudio(out, inspectLocalAudioPresence(cfg))
|
||||||
|
|
||||||
|
var localManifest *manifest.Manifest
|
||||||
|
if m, err := loadLocalManifest(ctx, paths.ManifestPath); err != nil {
|
||||||
|
fmt.Fprintf(out, "Local manifest: error: %v\n", err)
|
||||||
|
} else if m == nil {
|
||||||
|
fmt.Fprintln(out, "Local manifest: missing")
|
||||||
|
} else {
|
||||||
|
localManifest = m
|
||||||
|
fmt.Fprintf(out, "Local manifest: %s\n", paths.ManifestPath)
|
||||||
|
writeStageStatuses(out, m)
|
||||||
|
}
|
||||||
|
|
||||||
|
store, storeErr := objectStoreIfConfigured(ctx, cfg)
|
||||||
|
if storeErr != nil {
|
||||||
|
fmt.Fprintf(out, "Remote publish: unavailable: %v\n", storeErr)
|
||||||
|
} else if store != nil {
|
||||||
|
current := inspectRemoteCurrentState(ctx, cfg, store)
|
||||||
|
if current.Err != nil {
|
||||||
|
fmt.Fprintf(out, "Remote publish: missing or unavailable: %v\n", current.Err)
|
||||||
|
} else {
|
||||||
|
fmt.Fprintf(out, "Remote publish: current run %s\n", current.State.RunID)
|
||||||
|
fmt.Fprintf(out, "Remote manifest: %s\n", current.State.CurrentManifestKey)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
writeStatusRemoteAudio(ctx, out, cfg, store, storeErr)
|
||||||
|
writeStatusPreviousArtifacts(out, inspectPreviousArtifactReadiness(
|
||||||
|
ctx,
|
||||||
|
cfg,
|
||||||
|
store,
|
||||||
|
artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg)),
|
||||||
|
))
|
||||||
|
|
||||||
|
lockChecks := inspectEffectiveLocks(ctx, cfg, store)
|
||||||
|
locks := lockChecks.Locks
|
||||||
|
lockErr := lockChecks.Err
|
||||||
|
if catalog, catalogErr := buildHelperArtifactCatalog(cfg, localManifest); catalogErr != nil {
|
||||||
|
fmt.Fprintf(out, "Remote outputs: error: %v\n", catalogErr)
|
||||||
|
} else if storeErr == nil {
|
||||||
|
catalogLocks := locks
|
||||||
|
if lockErr != nil {
|
||||||
|
catalogLocks = &effectiveLocks{
|
||||||
|
Static: staticPublishLocks(cfg),
|
||||||
|
All: staticPublishLocks(cfg),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
publishedRemoteState := map[string]string{}
|
||||||
|
if store != nil {
|
||||||
|
publishedRemoteState = remotePublishedOutputAvailability(ctx, cfg, store, catalog)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Remote outputs:")
|
||||||
|
writeArtifactList(out, cfg, catalog, catalogLocks, publishedRemoteState)
|
||||||
|
}
|
||||||
|
if lockErr != nil {
|
||||||
|
fmt.Fprintf(out, "Publish locks: error: %v\n", lockErr)
|
||||||
|
} else {
|
||||||
|
writeLocks(out, cfg, locks)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(out, "Next actions:")
|
||||||
|
fmt.Fprintf(out, "- narratio session validate %s\n", cfg.Session.SessionID)
|
||||||
|
fmt.Fprintf(out, "- narratio session restore %s --dry-run\n", cfg.Session.SessionID)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStatusStableInputs(out io.Writer, checks []stableInputCheck) {
|
||||||
|
if len(checks) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for _, check := range checks {
|
||||||
|
if check.Err != nil {
|
||||||
|
if strings.TrimSpace(check.Path) != "" {
|
||||||
|
fmt.Fprintf(out, "Stable input %s: unavailable: %v\n", check.Name, check.Err)
|
||||||
|
} else {
|
||||||
|
fmt.Fprintf(out, "Stable input %s: unavailable: %s\n", check.Name, check.Err.Error())
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "Stable input %s: %s\n", check.Name, check.Path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStatusLocalAudio(out io.Writer, check localAudioCheck) {
|
||||||
|
if !check.Checked {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if check.Err != nil {
|
||||||
|
fmt.Fprintf(out, "Local audio: unavailable: %v\n", check.Err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "Local audio: %d file(s)\n", len(check.Paths))
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStatusRemoteAudio(ctx context.Context, out io.Writer, cfg *config.Config, store storage.ObjectStore, storeErr error) {
|
||||||
|
if cfg.Session.Inputs.AudioS3 == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if storeErr != nil {
|
||||||
|
fmt.Fprintf(out, "Remote audio: unavailable: %v\n", storeErr)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
check := inspectRemoteAudioPresence(ctx, cfg, store)
|
||||||
|
if check.Err != nil {
|
||||||
|
fmt.Fprintf(out, "Remote audio: unavailable: %v\n", check.Err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "Remote audio: %d .flac object(s)\n", len(check.Keys))
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStatusPreviousArtifacts(out io.Writer, readiness previousArtifactReadiness) {
|
||||||
|
if len(readiness.Requirements) == 0 {
|
||||||
|
fmt.Fprintln(out, "Previous-session artifacts: not required")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if readiness.MissingID {
|
||||||
|
fmt.Fprintln(out, "Previous-session artifacts: unavailable: previous_session_id is required by configured previous-session artifacts")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if readiness.Err != nil {
|
||||||
|
fmt.Fprintf(out, "Previous-session artifacts: unavailable: %v\n", readiness.Err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
names := make([]string, 0, len(readiness.Requirements))
|
||||||
|
for _, req := range readiness.Requirements {
|
||||||
|
names = append(names, fmt.Sprintf("%s(required=%t)", req.Name, req.Required))
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
fmt.Fprintf(out, "Previous-session artifacts: ready: %s\n", strings.Join(names, ", "))
|
||||||
|
}
|
||||||
@@ -7,7 +7,6 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
"os"
|
"os"
|
||||||
"strings"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
@@ -17,46 +16,21 @@ import (
|
|||||||
|
|
||||||
// Plan validates configuration, prepares the local workdir, and prints stage order.
|
// Plan validates configuration, prepares the local workdir, and prints stage order.
|
||||||
func Plan(ctx context.Context, args []string, out io.Writer) error {
|
func Plan(ctx context.Context, args []string, out io.Writer) error {
|
||||||
positionalSessionID, args := pullLeadingSessionID(args)
|
|
||||||
fs := flag.NewFlagSet("plan", flag.ContinueOnError)
|
fs := flag.NewFlagSet("plan", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
|
|
||||||
var pipelinePath string
|
var flags commonConfigFlags
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var force bool
|
var force bool
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
fs.BoolVar(&force, "force", false, "show all stages as scheduled to rerun")
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
if err := parseSessionAwareFlags("plan", fs, args, &flags.sessionID); err != nil {
|
||||||
return fmt.Errorf("plan: invalid flags: %w", err)
|
|
||||||
}
|
|
||||||
if positionalSessionID == "" {
|
|
||||||
if err := applyParsedSessionIDArg("plan", fs, &sessionID); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
} else {
|
if flags.sessionID == "" {
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("plan: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("plan", positionalSessionID, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
|
||||||
return fmt.Errorf("plan: session_id is required")
|
return fmt.Errorf("plan: session_id is required")
|
||||||
}
|
}
|
||||||
cfg, err := loadCommandConfig(ctx, pipelinePath, campaignPath, campaignFilePath, sessionPath, config.SessionLoadOptions{
|
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||||
SessionID: sessionID,
|
|
||||||
PreviousSessionID: previousSessionID,
|
|
||||||
})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("plan: %w", err)
|
return fmt.Errorf("plan: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -27,12 +27,12 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
|
|||||||
if !strings.Contains(got, "narratio session plan: workdir prepared at") {
|
if !strings.Contains(got, "narratio session plan: workdir prepared at") {
|
||||||
t.Fatalf("first output = %q, want workdir prepared", got)
|
t.Fatalf("first output = %q, want workdir prepared", got)
|
||||||
}
|
}
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "publish", "notify"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze", "publish", "notify"} {
|
||||||
if !strings.Contains(got, name+": run") {
|
if !strings.Contains(got, name+": run") {
|
||||||
t.Fatalf("first output = %q, missing stage %q", got, name)
|
t.Fatalf("first output = %q, missing stage %q", got, name)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !strings.Contains(got, "totals: run=9 skip=0") {
|
if !strings.Contains(got, "totals: run=11 skip=0") {
|
||||||
t.Fatalf("first output = %q, want totals", got)
|
t.Fatalf("first output = %q, want totals", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -84,8 +84,8 @@ func TestPlanShowsRunAndSkipFromManifest(t *testing.T) {
|
|||||||
if !strings.Contains(got, "trim: run") {
|
if !strings.Contains(got, "trim: run") {
|
||||||
t.Fatalf("output = %q, want trim run", got)
|
t.Fatalf("output = %q, want trim run", got)
|
||||||
}
|
}
|
||||||
if !strings.Contains(got, "totals: run=7 skip=2") {
|
if !strings.Contains(got, "totals: run=9 skip=2") {
|
||||||
t.Fatalf("output = %q, want totals run=7 skip=2", got)
|
t.Fatalf("output = %q, want totals run=9 skip=2", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -118,6 +118,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`
|
`
|
||||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||||
t.Fatalf("write pipeline.yml: %v", err)
|
t.Fatalf("write pipeline.yml: %v", err)
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ import "testing"
|
|||||||
|
|
||||||
func TestBuildFullPlanOrder(t *testing.T) {
|
func TestBuildFullPlanOrder(t *testing.T) {
|
||||||
got := BuildFullPlan()
|
got := BuildFullPlan()
|
||||||
want := []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "publish", "notify"}
|
want := []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze", "publish", "notify"}
|
||||||
if len(got) != len(want) {
|
if len(got) != len(want) {
|
||||||
t.Fatalf("len(plan) = %d, want %d", len(got), len(want))
|
t.Fatalf("len(plan) = %d, want %d", len(got), len(want))
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -12,7 +12,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m *manifest.Manifest, executed []string) error {
|
func runPostPublishCleanup(ctx context.Context, env *Env, manifestPath string, m *manifest.Manifest, executed []string) error {
|
||||||
if env == nil || env.Config == nil || env.Config.Pipeline == nil || m == nil {
|
if env == nil || env.Config == nil || env.Config.Pipeline == nil || m == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -23,7 +23,7 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
sr := archiveStageRecordForCleanup(m, executed)
|
sr := publishStageRecordForCleanup(m, executed)
|
||||||
if sr == nil {
|
if sr == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -33,7 +33,7 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
|||||||
sr.Metadata["spool_cleanup_requested"] = spoolRequested
|
sr.Metadata["spool_cleanup_requested"] = spoolRequested
|
||||||
sr.Metadata["workdir_cleanup_requested"] = workRequested
|
sr.Metadata["workdir_cleanup_requested"] = workRequested
|
||||||
|
|
||||||
eligible, reason := archiveCleanupEligible(env.Config, sr)
|
eligible, reason := publishCleanupEligible(env.Config, sr)
|
||||||
if !eligible {
|
if !eligible {
|
||||||
sr.Metadata["cleanup_skipped"] = true
|
sr.Metadata["cleanup_skipped"] = true
|
||||||
sr.Metadata["cleanup_skipped_reason"] = reason
|
sr.Metadata["cleanup_skipped_reason"] = reason
|
||||||
@@ -96,7 +96,7 @@ func runPostArchiveCleanup(ctx context.Context, env *Env, manifestPath string, m
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveStageRecordForCleanup(m *manifest.Manifest, executed []string) *manifest.StageRecord {
|
func publishStageRecordForCleanup(m *manifest.Manifest, executed []string) *manifest.StageRecord {
|
||||||
if m == nil {
|
if m == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -117,7 +117,7 @@ func archiveStageRecordForCleanup(m *manifest.Manifest, executed []string) *mani
|
|||||||
return sr
|
return sr
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveCleanupEligible(cfg *config.Config, sr *manifest.StageRecord) (bool, string) {
|
func publishCleanupEligible(cfg *config.Config, sr *manifest.StageRecord) (bool, string) {
|
||||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Publish == nil {
|
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Publish == nil {
|
||||||
return false, "publish configuration is missing"
|
return false, "publish configuration is missing"
|
||||||
}
|
}
|
||||||
@@ -174,49 +174,7 @@ func removeRunScopedDir(root, target, policy string) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func validateScopedDir(root, target, policy string) (scopedDir, error) {
|
func validateScopedDir(root, target, policy string) (scopedDir, error) {
|
||||||
cleanRoot := strings.TrimSpace(root)
|
return validateScopedTarget(root, target, policy, true)
|
||||||
cleanTarget := strings.TrimSpace(target)
|
|
||||||
if cleanRoot == "" {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: root path is required", policy)
|
|
||||||
}
|
|
||||||
if cleanTarget == "" {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target path is required", policy)
|
|
||||||
}
|
|
||||||
|
|
||||||
rootAbs, err := filepath.Abs(cleanRoot)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve root %q: %w", policy, cleanRoot, err)
|
|
||||||
}
|
|
||||||
targetAbs, err := filepath.Abs(cleanTarget)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: resolve target %q: %w", policy, cleanTarget, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
rel, err := filepath.Rel(rootAbs, targetAbs)
|
|
||||||
if err != nil {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: relative path from %q to %q: %w", policy, rootAbs, targetAbs, err)
|
|
||||||
}
|
|
||||||
if rel == "." {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete root directory %q", policy, rootAbs)
|
|
||||||
}
|
|
||||||
if rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete path outside root: root=%q target=%q", policy, rootAbs, targetAbs)
|
|
||||||
}
|
|
||||||
|
|
||||||
info, err := os.Lstat(targetAbs)
|
|
||||||
if err != nil {
|
|
||||||
if os.IsNotExist(err) {
|
|
||||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: false}, nil
|
|
||||||
}
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: stat target %q: %w", policy, targetAbs, err)
|
|
||||||
}
|
|
||||||
if info.Mode()&os.ModeSymlink != 0 {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: refusing to delete symlink path %q", policy, targetAbs)
|
|
||||||
}
|
|
||||||
if !info.IsDir() {
|
|
||||||
return scopedDir{}, fmt.Errorf("cleanup policy %s: target %q is not a directory", policy, targetAbs)
|
|
||||||
}
|
|
||||||
return scopedDir{RootAbs: rootAbs, TargetAbs: targetAbs, Exists: true}, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func asString(v any) string {
|
func asString(v any) string {
|
||||||
@@ -16,13 +16,13 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||||
)
|
)
|
||||||
|
|
||||||
type archiveSuccessStage struct {
|
type publishSuccessStage struct {
|
||||||
metadata map[string]any
|
metadata map[string]any
|
||||||
}
|
}
|
||||||
|
|
||||||
func (archiveSuccessStage) Name() string { return "publish" }
|
func (publishSuccessStage) Name() string { return "publish" }
|
||||||
func (archiveSuccessStage) Declares() stage.IODecl { return stage.IODecl{} }
|
func (publishSuccessStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||||
func (s archiveSuccessStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
func (s publishSuccessStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
||||||
md := map[string]any{
|
md := map[string]any{
|
||||||
"stage": "publish",
|
"stage": "publish",
|
||||||
"uploaded": true,
|
"uploaded": true,
|
||||||
@@ -43,12 +43,12 @@ func (notifyFailStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest
|
|||||||
return nil, errors.New("notify failed")
|
return nil, errors.New("notify failed")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupDisabledKeepsLocalDirs(t *testing.T) {
|
func TestPostPublishCleanupDisabledKeepsLocalDirs(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = false
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = false
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -57,12 +57,12 @@ func TestPostArchiveCleanupDisabledKeepsLocalDirs(t *testing.T) {
|
|||||||
assertExists(t, seed.localSourceAudio)
|
assertExists(t, seed.localSourceAudio)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupSpoolOnly(t *testing.T) {
|
func TestPostPublishCleanupSpoolOnly(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -71,12 +71,12 @@ func TestPostArchiveCleanupSpoolOnly(t *testing.T) {
|
|||||||
assertExists(t, seed.localSourceAudio)
|
assertExists(t, seed.localSourceAudio)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupWorkdirOnly(t *testing.T) {
|
func TestPostPublishCleanupWorkdirOnly(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = false
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = false
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -87,12 +87,12 @@ func TestPostArchiveCleanupWorkdirOnly(t *testing.T) {
|
|||||||
assertExists(t, seed.spoolAudioDir)
|
assertExists(t, seed.spoolAudioDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupBothPolicies(t *testing.T) {
|
func TestPostPublishCleanupBothPolicies(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -102,12 +102,12 @@ func TestPostArchiveCleanupBothPolicies(t *testing.T) {
|
|||||||
assertExists(t, seed.previousCachePath)
|
assertExists(t, seed.previousCachePath)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenArchiveFails(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenPublishFails(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
_, err := executeStages(context.Background(), cfg, []stage.Stage{failingStage{name: "publish", err: errors.New("archive failed")}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
_, err := executeStages(context.Background(), cfg, []stage.Stage{failingStage{name: "publish", err: errors.New("publish failed")}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||||
if err == nil || !strings.Contains(err.Error(), "stage \"publish\" failed") {
|
if err == nil || !strings.Contains(err.Error(), "stage \"publish\" failed") {
|
||||||
t.Fatalf("executeStages() error = %v, want publish failure", err)
|
t.Fatalf("executeStages() error = %v, want publish failure", err)
|
||||||
}
|
}
|
||||||
@@ -116,12 +116,12 @@ func TestPostArchiveCleanupNotRunWhenArchiveFails(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenArchiveSkipped(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenPublishSkipped(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{metadata: map[string]any{"skipped": true}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{metadata: map[string]any{"skipped": true}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -129,12 +129,12 @@ func TestPostArchiveCleanupNotRunWhenArchiveSkipped(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenCurrentPointerMissing(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenCurrentPointerMissing(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{metadata: map[string]any{"current_pointer_written": false}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{metadata: map[string]any{"current_pointer_written": false}}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -142,13 +142,13 @@ func TestPostArchiveCleanupNotRunWhenCurrentPointerMissing(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenArchiveUploadDisabled(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenPublishUploadDisabled(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
cfg.Pipeline.Publish.UploadRun = boolPtr(false)
|
cfg.Pipeline.Publish.UploadRun = boolPtr(false)
|
||||||
|
|
||||||
if _, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}}); err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -156,12 +156,12 @@ func TestPostArchiveCleanupNotRunWhenArchiveUploadDisabled(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupWaitsUntilAllStagesSucceed(t *testing.T) {
|
func TestPostPublishCleanupWaitsUntilAllStagesSucceed(t *testing.T) {
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
|
|
||||||
_, err := executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}, notifyFailStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
_, err := executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}, notifyFailStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||||
if err == nil || !strings.Contains(err.Error(), "stage \"notify\" failed") {
|
if err == nil || !strings.Contains(err.Error(), "stage \"notify\" failed") {
|
||||||
t.Fatalf("executeStages() error = %v, want notify failure", err)
|
t.Fatalf("executeStages() error = %v, want notify failure", err)
|
||||||
}
|
}
|
||||||
@@ -170,7 +170,7 @@ func TestPostArchiveCleanupWaitsUntilAllStagesSucceed(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupFailsOnUnsafePath(t *testing.T) {
|
func TestPostPublishCleanupFailsOnUnsafePath(t *testing.T) {
|
||||||
cfg, _ := cleanupFixtureConfig(t)
|
cfg, _ := cleanupFixtureConfig(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
cfg.Pipeline.Workspace.CleanupAfterPublish = false
|
||||||
@@ -186,25 +186,25 @@ func TestPostArchiveCleanupFailsOnUnsafePath(t *testing.T) {
|
|||||||
t.Fatalf("Save() error = %v", err)
|
t.Fatalf("Save() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishSuccessStage{}}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||||
if err == nil || !strings.Contains(err.Error(), "refusing to delete path outside root") {
|
if err == nil || !strings.Contains(err.Error(), "refusing to delete path outside root") {
|
||||||
t.Fatalf("executeStages() error = %v, want safe-path failure", err)
|
t.Fatalf("executeStages() error = %v, want safe-path failure", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenOutputIsMissing(t *testing.T) {
|
||||||
cfg, seed, runID := archiveStageCleanupFixture(t)
|
cfg, seed, runID := publishStageCleanupFixture(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
cfg.Pipeline.Publish.Outputs = []config.PublishOutputRule{
|
cfg.Pipeline.Publish.Outputs = []config.PublishOutputRule{
|
||||||
{Source: "narratio.transcript.base", Dest: "transcripts/base.json", Required: boolPtr(true)},
|
{Source: "narratio.transcript.base", Dest: "transcripts/base.json", Required: boolPtr(true)},
|
||||||
}
|
}
|
||||||
|
|
||||||
archiveStageImpl, err := stage.Select("publish")
|
publishStageImpl, err := stage.Select("publish")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Select(publish) error = %v", err)
|
t.Fatalf("Select(publish) error = %v", err)
|
||||||
}
|
}
|
||||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveStageImpl}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishStageImpl}, RunOptions{Env: &Env{ObjectStore: &storage.FakeBackend{}}})
|
||||||
if err == nil || !strings.Contains(err.Error(), "required output source unavailable") {
|
if err == nil || !strings.Contains(err.Error(), "required output source unavailable") {
|
||||||
t.Fatalf("executeStages() error = %v, want required output source unavailable failure", err)
|
t.Fatalf("executeStages() error = %v, want required output source unavailable failure", err)
|
||||||
}
|
}
|
||||||
@@ -215,17 +215,17 @@ func TestPostArchiveCleanupNotRunWhenPromotionIsMissing(t *testing.T) {
|
|||||||
assertExists(t, artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
|
assertExists(t, artifacts.SessionRunRootForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID, runID))
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
||||||
cfg, seed, _ := archiveStageCleanupFixture(t)
|
cfg, seed, _ := publishStageCleanupFixture(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
failKey := seed.sessionPrefix + "current/manifest.json"
|
failKey := seed.sessionPrefix + "current/manifest.json"
|
||||||
|
|
||||||
archiveStageImpl, err := stage.Select("publish")
|
publishStageImpl, err := stage.Select("publish")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Select(publish) error = %v", err)
|
t.Fatalf("Select(publish) error = %v", err)
|
||||||
}
|
}
|
||||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveStageImpl}, RunOptions{
|
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishStageImpl}, RunOptions{
|
||||||
Env: &Env{ObjectStore: &failKeyStore{delegate: &storage.FakeBackend{}, failKey: failKey}},
|
Env: &Env{ObjectStore: &failKeyStore{delegate: &storage.FakeBackend{}, failKey: failKey}},
|
||||||
})
|
})
|
||||||
if err == nil || !strings.Contains(err.Error(), "current manifest") {
|
if err == nil || !strings.Contains(err.Error(), "current manifest") {
|
||||||
@@ -236,17 +236,17 @@ func TestPostArchiveCleanupNotRunWhenCurrentManifestUploadFails(t *testing.T) {
|
|||||||
assertExists(t, seed.runWorkDir)
|
assertExists(t, seed.runWorkDir)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestPostArchiveCleanupNotRunWhenCurrentPointerUploadFails(t *testing.T) {
|
func TestPostPublishCleanupNotRunWhenCurrentPointerUploadFails(t *testing.T) {
|
||||||
cfg, seed, _ := archiveStageCleanupFixture(t)
|
cfg, seed, _ := publishStageCleanupFixture(t)
|
||||||
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
cfg.Pipeline.Spool.DeleteAudioAfterPublish = true
|
||||||
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
cfg.Pipeline.Workspace.CleanupAfterPublish = true
|
||||||
failKey := seed.sessionPrefix + "current/run_id.txt"
|
failKey := seed.sessionPrefix + "current/run_id.txt"
|
||||||
|
|
||||||
archiveStageImpl, err := stage.Select("publish")
|
publishStageImpl, err := stage.Select("publish")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Select(publish) error = %v", err)
|
t.Fatalf("Select(publish) error = %v", err)
|
||||||
}
|
}
|
||||||
_, err = executeStages(context.Background(), cfg, []stage.Stage{archiveStageImpl}, RunOptions{
|
_, err = executeStages(context.Background(), cfg, []stage.Stage{publishStageImpl}, RunOptions{
|
||||||
Env: &Env{ObjectStore: &failKeyStore{delegate: &storage.FakeBackend{}, failKey: failKey}},
|
Env: &Env{ObjectStore: &failKeyStore{delegate: &storage.FakeBackend{}, failKey: failKey}},
|
||||||
})
|
})
|
||||||
if err == nil || !strings.Contains(err.Error(), "current run pointer") {
|
if err == nil || !strings.Contains(err.Error(), "current run pointer") {
|
||||||
@@ -320,7 +320,7 @@ func cleanupFixtureConfig(t *testing.T) (*config.Config, cleanupSeed) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, string) {
|
func publishStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
cfg, seed := cleanupFixtureConfig(t)
|
cfg, seed := cleanupFixtureConfig(t)
|
||||||
@@ -344,7 +344,7 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
|||||||
},
|
},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
writeArchiveFixtureRunFiles(
|
writePublishFixtureRunFiles(
|
||||||
t,
|
t,
|
||||||
seed.runWorkDir,
|
seed.runWorkDir,
|
||||||
artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID),
|
artifacts.SessionWorkDirForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID),
|
||||||
@@ -355,7 +355,7 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Load() error = %v", err)
|
t.Fatalf("Load() error = %v", err)
|
||||||
}
|
}
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze"} {
|
||||||
seedManifest.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
seedManifest.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
||||||
}
|
}
|
||||||
seedManifest.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
|
seedManifest.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
@@ -367,7 +367,7 @@ func archiveStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
|||||||
return cfg, seed, runID
|
return cfg, seed, runID
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeArchiveFixtureRunFiles(t *testing.T, runWorkDir, sessionRoot string) {
|
func writePublishFixtureRunFiles(t *testing.T, runWorkDir, sessionRoot string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
mustWriteFile(t, filepath.Join(runWorkDir, "prepare", "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
mustWriteFile(t, filepath.Join(runWorkDir, "prepare", "inputs", "session.yml"), "session_id: 2026-05-03\n")
|
||||||
mustWriteFile(t, filepath.Join(runWorkDir, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
|
mustWriteFile(t, filepath.Join(runWorkDir, "transcribe", "outputs", "transcripts", "raw", "speaker.json"), "{}\n")
|
||||||
@@ -46,7 +46,7 @@ func loadRemoteLockStore(ctx context.Context, cfg *config.Config, store storage.
|
|||||||
if !exists {
|
if !exists {
|
||||||
return &config.PublishLockStore{}, key, nil
|
return &config.PublishLockStore{}, key, nil
|
||||||
}
|
}
|
||||||
tmp, err := downloadObjectToTemp(ctx, store, key, "narratio-locks-*.yml")
|
tmp, err := storage.DownloadObjectToTemp(ctx, store, key, "narratio-locks-*.yml")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, key, fmt.Errorf("download remote locks %q: %w", key, err)
|
return nil, key, fmt.Errorf("download remote locks %q: %w", key, err)
|
||||||
}
|
}
|
||||||
@@ -55,7 +55,7 @@ func loadRemoteLockStore(ctx context.Context, cfg *config.Config, store storage.
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, key, fmt.Errorf("read remote locks %q: %w", key, err)
|
return nil, key, fmt.Errorf("read remote locks %q: %w", key, err)
|
||||||
}
|
}
|
||||||
lockStore, err := config.LoadPublishLockStoreBytes("s3://"+s3BucketName(cfg.Pipeline)+"/"+key, data, cfg.Pipeline.Scriptorium)
|
lockStore, err := config.LoadPublishLockStoreBytes("s3://"+s3BucketName(cfg.Pipeline)+"/"+key, data, cfg.Pipeline.Scriptorium, cfg.Pipeline.Notarius)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, key, err
|
return nil, key, err
|
||||||
}
|
}
|
||||||
@@ -63,7 +63,7 @@ func loadRemoteLockStore(ctx context.Context, cfg *config.Config, store storage.
|
|||||||
}
|
}
|
||||||
|
|
||||||
func loadEffectiveLocks(ctx context.Context, cfg *config.Config, store storage.ObjectStore) (*effectiveLocks, error) {
|
func loadEffectiveLocks(ctx context.Context, cfg *config.Config, store storage.ObjectStore) (*effectiveLocks, error) {
|
||||||
staticLocks := staticArchiveLocks(cfg)
|
staticLocks := staticPublishLocks(cfg)
|
||||||
if store == nil {
|
if store == nil {
|
||||||
return &effectiveLocks{
|
return &effectiveLocks{
|
||||||
Static: staticLocks,
|
Static: staticLocks,
|
||||||
@@ -83,7 +83,7 @@ func loadEffectiveLocks(ctx context.Context, cfg *config.Config, store storage.O
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func staticArchiveLocks(cfg *config.Config) []config.PublishLockRule {
|
func staticPublishLocks(cfg *config.Config) []config.PublishLockRule {
|
||||||
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Publish == nil {
|
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Publish == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -27,23 +27,14 @@ func Restore(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
fs := flag.NewFlagSet("restore", flag.ContinueOnError)
|
fs := flag.NewFlagSet("restore", flag.ContinueOnError)
|
||||||
fs.SetOutput(out)
|
fs.SetOutput(out)
|
||||||
|
|
||||||
var pipelinePath string
|
var flags commonConfigFlags
|
||||||
var campaignPath string
|
|
||||||
var campaignFilePath string
|
|
||||||
var sessionPath string
|
|
||||||
var sessionID string
|
|
||||||
var previousSessionID string
|
|
||||||
var dryRun bool
|
var dryRun bool
|
||||||
var force bool
|
var force bool
|
||||||
var includeAudio bool
|
var includeAudio bool
|
||||||
fs.StringVar(&pipelinePath, "config", "", "path to pipeline.yml (optional; defaults searched)")
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.StringVar(&campaignPath, "campaign", "", "campaign ID")
|
|
||||||
fs.StringVar(&campaignFilePath, "campaign-file", "", "path to campaign.yml")
|
|
||||||
fs.StringVar(&sessionPath, "session", "", "path to session.yml")
|
|
||||||
fs.StringVar(&previousSessionID, "previous-session-id", "", "expected previous session identifier")
|
|
||||||
fs.BoolVar(&dryRun, "dry-run", false, "plan restore actions without writing local files")
|
fs.BoolVar(&dryRun, "dry-run", false, "plan restore actions without writing local files")
|
||||||
fs.BoolVar(&force, "force", false, "overwrite local conflicts with remote state")
|
fs.BoolVar(&force, "force", false, "overwrite local conflicts with remote state")
|
||||||
fs.BoolVar(&includeAudio, "include-audio", false, "include archived session-level audio objects")
|
fs.BoolVar(&includeAudio, "include-audio", false, "include remote session-level audio objects")
|
||||||
fs.Usage = func() {
|
fs.Usage = func() {
|
||||||
_, _ = fmt.Fprintln(out, "Usage: narratio session restore <session_id> [--config <path>] [--campaign <id>] [--campaign-file <path>] [--session <path>] [--previous-session-id <value>] [--dry-run] [--force] [--include-audio]")
|
_, _ = fmt.Fprintln(out, "Usage: narratio session restore <session_id> [--config <path>] [--campaign <id>] [--campaign-file <path>] [--session <path>] [--previous-session-id <value>] [--dry-run] [--force] [--include-audio]")
|
||||||
_, _ = fmt.Fprintln(out)
|
_, _ = fmt.Fprintln(out)
|
||||||
@@ -57,25 +48,13 @@ func Restore(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
}
|
}
|
||||||
return fmt.Errorf("restore: invalid flags: %w", err)
|
return fmt.Errorf("restore: invalid flags: %w", err)
|
||||||
}
|
}
|
||||||
if positionalSessionID == "" {
|
if err := resolveParsedSessionID("restore", positionalSessionID, fs, &flags.sessionID); err != nil {
|
||||||
if err := applyParsedSessionIDArg("restore", fs, &sessionID); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
} else {
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("restore: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("restore", positionalSessionID, &sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(sessionID) == "" {
|
|
||||||
return fmt.Errorf("restore: session_id is required")
|
return fmt.Errorf("restore: session_id is required")
|
||||||
}
|
}
|
||||||
cfg, err := loadCommandConfig(ctx, pipelinePath, campaignPath, campaignFilePath, sessionPath, config.SessionLoadOptions{
|
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
||||||
SessionID: sessionID,
|
|
||||||
PreviousSessionID: previousSessionID,
|
|
||||||
})
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("restore: %w", err)
|
return fmt.Errorf("restore: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ package app
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
@@ -12,7 +11,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
// RemoteCurrentState captures discovered committed remote archive state for one session.
|
// RemoteCurrentState captures discovered committed remote published current state for one session.
|
||||||
type RemoteCurrentState struct {
|
type RemoteCurrentState struct {
|
||||||
Bucket string
|
Bucket string
|
||||||
SessionPrefix string
|
SessionPrefix string
|
||||||
@@ -32,80 +31,24 @@ func discoverRemoteCurrentState(ctx context.Context, cfg *config.Config, store s
|
|||||||
return nil, fmt.Errorf("remote object store is required")
|
return nil, fmt.Errorf("remote object store is required")
|
||||||
}
|
}
|
||||||
|
|
||||||
bucket := artifacts.ResolveArchiveBucket(cfg, nil)
|
bucket := artifacts.ResolvePublishBucket(cfg, nil)
|
||||||
if strings.TrimSpace(bucket) == "" {
|
if strings.TrimSpace(bucket) == "" {
|
||||||
return nil, fmt.Errorf("archive bucket is required")
|
return nil, fmt.Errorf("publish bucket is required")
|
||||||
}
|
}
|
||||||
sessionPrefix, err := artifacts.ResolveArchiveSessionPrefix(cfg, nil)
|
sessionPrefix, err := artifacts.ResolvePublishSessionPrefix(cfg, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("resolve archive session prefix: %w", err)
|
return nil, fmt.Errorf("resolve publish session prefix: %w", err)
|
||||||
}
|
|
||||||
currentManifestKey, currentRunIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
|
||||||
|
|
||||||
exists, err := store.Exists(ctx, currentRunIDKey)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("check remote current run pointer %q: %w", currentRunIDKey, err)
|
|
||||||
}
|
|
||||||
if !exists {
|
|
||||||
return nil, fmt.Errorf("remote current run pointer missing: %q", currentRunIDKey)
|
|
||||||
}
|
|
||||||
|
|
||||||
runIDPath, err := downloadObjectToTemp(ctx, store, currentRunIDKey, "narratio-restore-current-run-id-*.txt")
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("download remote current run pointer %q: %w", currentRunIDKey, err)
|
|
||||||
}
|
|
||||||
defer func() { _ = os.Remove(runIDPath) }()
|
|
||||||
|
|
||||||
runIDData, err := os.ReadFile(runIDPath)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("read downloaded run pointer %q: %w", currentRunIDKey, err)
|
|
||||||
}
|
|
||||||
runID := strings.TrimSpace(string(runIDData))
|
|
||||||
if runID == "" {
|
|
||||||
return nil, fmt.Errorf("remote current run pointer %q is empty", currentRunIDKey)
|
|
||||||
}
|
|
||||||
|
|
||||||
exists, err = store.Exists(ctx, currentManifestKey)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("check remote current manifest %q: %w", currentManifestKey, err)
|
|
||||||
}
|
|
||||||
if !exists {
|
|
||||||
return nil, fmt.Errorf("remote current manifest missing: %q", currentManifestKey)
|
|
||||||
}
|
|
||||||
|
|
||||||
manifestPath, err := downloadObjectToTemp(ctx, store, currentManifestKey, "narratio-restore-current-manifest-*.json")
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("download remote current manifest %q: %w", currentManifestKey, err)
|
|
||||||
}
|
|
||||||
defer func() { _ = os.Remove(manifestPath) }()
|
|
||||||
|
|
||||||
manifestStore := &manifest.LocalStore{}
|
|
||||||
remoteManifest, err := manifestStore.Load(ctx, manifestPath)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("remote current manifest decode failed: %w", err)
|
|
||||||
}
|
}
|
||||||
|
currentManifestKey, currentRunIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
|
|
||||||
requestedSession := strings.TrimSpace(cfg.Session.SessionID)
|
requestedSession := strings.TrimSpace(cfg.Session.SessionID)
|
||||||
requestedCampaign := strings.TrimSpace(cfg.Session.Campaign)
|
requestedCampaign := strings.TrimSpace(cfg.Session.Campaign)
|
||||||
manifestSession := strings.TrimSpace(remoteManifest.SessionID)
|
current, err := artifacts.LoadCurrentState(ctx, store, sessionPrefix, artifacts.CurrentStateValidation{
|
||||||
manifestCampaign := strings.TrimSpace(remoteManifest.Campaign)
|
ExpectedSessionID: requestedSession,
|
||||||
|
ExpectedCampaign: requestedCampaign,
|
||||||
if manifestSession != requestedSession {
|
})
|
||||||
return nil, fmt.Errorf(
|
if err != nil {
|
||||||
"remote current manifest session_id %q does not match requested session_id %q",
|
return nil, fmt.Errorf("remote %w", err)
|
||||||
manifestSession,
|
|
||||||
requestedSession,
|
|
||||||
)
|
|
||||||
}
|
|
||||||
if manifestCampaign == "" {
|
|
||||||
return nil, fmt.Errorf("remote current manifest campaign is required")
|
|
||||||
}
|
|
||||||
if manifestCampaign != requestedCampaign {
|
|
||||||
return nil, fmt.Errorf(
|
|
||||||
"remote current manifest campaign %q does not match requested campaign %q",
|
|
||||||
manifestCampaign,
|
|
||||||
requestedCampaign,
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return &RemoteCurrentState{
|
return &RemoteCurrentState{
|
||||||
@@ -113,27 +56,9 @@ func discoverRemoteCurrentState(ctx context.Context, cfg *config.Config, store s
|
|||||||
SessionPrefix: sessionPrefix,
|
SessionPrefix: sessionPrefix,
|
||||||
CurrentRunIDKey: currentRunIDKey,
|
CurrentRunIDKey: currentRunIDKey,
|
||||||
CurrentManifestKey: currentManifestKey,
|
CurrentManifestKey: currentManifestKey,
|
||||||
RunID: runID,
|
RunID: current.RunID,
|
||||||
SessionID: manifestSession,
|
SessionID: strings.TrimSpace(current.Manifest.SessionID),
|
||||||
Campaign: manifestCampaign,
|
Campaign: strings.TrimSpace(current.Manifest.Campaign),
|
||||||
Manifest: remoteManifest,
|
Manifest: current.Manifest,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func downloadObjectToTemp(ctx context.Context, store storage.ObjectStore, key, pattern string) (string, error) {
|
|
||||||
tmp, err := os.CreateTemp("", pattern)
|
|
||||||
if err != nil {
|
|
||||||
return "", fmt.Errorf("create temp file: %w", err)
|
|
||||||
}
|
|
||||||
path := tmp.Name()
|
|
||||||
if err := tmp.Close(); err != nil {
|
|
||||||
_ = os.Remove(path)
|
|
||||||
return "", fmt.Errorf("close temp file: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if err := store.Download(ctx, key, path); err != nil {
|
|
||||||
_ = os.Remove(path)
|
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
return path, nil
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -102,7 +102,7 @@ func TestDiscoverRemoteCurrentStateSessionMismatchFails(t *testing.T) {
|
|||||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "wrong-session", cfg.Session.Campaign)})
|
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, "wrong-session", cfg.Session.Campaign)})
|
||||||
|
|
||||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||||
if err == nil || !strings.Contains(err.Error(), "does not match requested session_id") {
|
if err == nil || !strings.Contains(err.Error(), "does not match expected session_id") {
|
||||||
t.Fatalf("error = %v, want session mismatch failure", err)
|
t.Fatalf("error = %v, want session mismatch failure", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -116,7 +116,7 @@ func TestDiscoverRemoteCurrentStateCampaignMismatchFails(t *testing.T) {
|
|||||||
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, "wrong-campaign")})
|
store.SeedObject(storage.FakeObject{Key: manifestKey, Data: restoreManifestJSON(t, cfg.Session.SessionID, "wrong-campaign")})
|
||||||
|
|
||||||
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
_, err := discoverRemoteCurrentState(context.Background(), cfg, store)
|
||||||
if err == nil || !strings.Contains(err.Error(), "does not match requested campaign") {
|
if err == nil || !strings.Contains(err.Error(), "does not match expected campaign") {
|
||||||
t.Fatalf("error = %v, want campaign mismatch failure", err)
|
t.Fatalf("error = %v, want campaign mismatch failure", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -208,7 +208,7 @@ func restoreDiscoveryConfig() *config.Config {
|
|||||||
|
|
||||||
func restoreDiscoveryKeys(cfg *config.Config) (sessionPrefix, manifestKey, runIDKey string) {
|
func restoreDiscoveryKeys(cfg *config.Config) (sessionPrefix, manifestKey, runIDKey string) {
|
||||||
sessionPrefix = artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
sessionPrefix = artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
manifestKey, runIDKey = artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
manifestKey, runIDKey = artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
return sessionPrefix, manifestKey, runIDKey
|
return sessionPrefix, manifestKey, runIDKey
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/audio"
|
"gitea.maximumdirect.net/eric/narratio/internal/audio"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/fileops"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -114,10 +115,7 @@ func executeRestoreDownloadAction(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := os.Chmod(tmpPath, 0o644); err != nil {
|
if err := fileops.InstallDownloadedTempFile(tmpPath, safeLocalPath, 0o644); err != nil {
|
||||||
return fmt.Errorf("set file permissions: %w", err)
|
|
||||||
}
|
|
||||||
if err := os.Rename(tmpPath, safeLocalPath); err != nil {
|
|
||||||
return fmt.Errorf("install file atomically: %w", err)
|
return fmt.Errorf("install file atomically: %w", err)
|
||||||
}
|
}
|
||||||
removeTmp = false
|
removeTmp = false
|
||||||
|
|||||||
@@ -9,8 +9,10 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
@@ -39,7 +41,7 @@ func TestExecuteRestoreNonDryRunRestoresDurableFiles(t *testing.T) {
|
|||||||
if stderr.Len() != 0 {
|
if stderr.Len() != 0 {
|
||||||
t.Fatalf("stderr = %q, want empty", stderr.String())
|
t.Fatalf("stderr = %q, want empty", stderr.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "Restored session archive for sample-campaign/2026-05-03") {
|
if !strings.Contains(stdout.String(), "Restored session state for sample-campaign/2026-05-03") {
|
||||||
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -62,6 +64,55 @@ func TestExecuteRestoreNonDryRunRestoresDurableFiles(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteRestoreRoundTripsPublishedExtractionAndManifestMetadata(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
cfg, sessionPrefix, manifestKey, runIDKey := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||||
|
seedRestoreObject(fake, sessionPrefix+"artifacts/encounters.json", []byte(`{"encounters":[]}`))
|
||||||
|
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
|
||||||
|
|
||||||
|
remoteManifest := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
||||||
|
remoteManifest.Campaign = cfg.Session.Campaign
|
||||||
|
remoteManifest.RunID = "20260519T010203Z-a1b2c3d4"
|
||||||
|
remoteManifest.Stages["extract"] = &manifest.StageRecord{
|
||||||
|
Name: "extract", Status: manifest.StatusSucceeded,
|
||||||
|
Outputs: []manifest.ArtifactRecord{{
|
||||||
|
Kind: "notarius_lane", SourceID: artifacts.ExtractionArtifactSourceID("encounters"),
|
||||||
|
LocalPath: "/prior/workspace/artifacts/notarius/extract-run-1/lanes/encounters.json",
|
||||||
|
Contract: &artifactmodel.ContractMetadata{
|
||||||
|
MediaType: "application/json", SchemaID: "encounters", SchemaVersion: "1",
|
||||||
|
},
|
||||||
|
ExternalProvenance: &artifactmodel.ExternalProvenance{
|
||||||
|
System: "notarius", RunID: "notarius-run-1", PipelineID: "campaign.extract", ArtifactID: "encounters",
|
||||||
|
},
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
manifestBody, err := json.Marshal(remoteManifest)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
seedRestoreObject(fake, manifestKey, manifestBody)
|
||||||
|
restoreWithStoreAndRealPhases(t, fake)
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{"session", "restore", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
mustReadEquals(t, filepath.Join(sessionRoot, "artifacts", "encounters.json"), `{"encounters":[]}`)
|
||||||
|
restored, err := (&manifest.LocalStore{}).Load(context.Background(), filepath.Join(sessionRoot, "manifest.json"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("load restored manifest: %v", err)
|
||||||
|
}
|
||||||
|
lane := restored.Stages["extract"].Outputs[0]
|
||||||
|
if lane.Contract == nil || lane.Contract.SchemaID != "encounters" || lane.ExternalProvenance == nil || lane.ExternalProvenance.RunID != "notarius-run-1" {
|
||||||
|
t.Fatalf("restored extraction metadata = %#v", lane)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteRestoreIncludeAudioRestoresAudio(t *testing.T) {
|
func TestExecuteRestoreIncludeAudioRestoresAudio(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
@@ -427,7 +478,7 @@ func seedRestoreCommittedState(t *testing.T, fake *storage.FakeBackend, pipeline
|
|||||||
}
|
}
|
||||||
|
|
||||||
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
|
|
||||||
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
|
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
|
||||||
seedRestoreObject(fake, manifestKey, restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
|
seedRestoreObject(fake, manifestKey, restoreManifestJSON(t, cfg.Session.SessionID, cfg.Session.Campaign))
|
||||||
@@ -465,7 +516,7 @@ func seedRestorePreviousCurrent(t *testing.T, fake *storage.FakeBackend, cfg *co
|
|||||||
func seedRestorePreviousCurrentManifestOnly(t *testing.T, fake *storage.FakeBackend, cfg *config.Config) {
|
func seedRestorePreviousCurrentManifestOnly(t *testing.T, fake *storage.FakeBackend, cfg *config.Config) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
previousPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
previousPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
||||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(previousPrefix)
|
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(previousPrefix)
|
||||||
previousRunID := "20260426T010203Z-a1b2c3d4"
|
previousRunID := "20260426T010203Z-a1b2c3d4"
|
||||||
seedRestoreObject(fake, runIDKey, []byte(previousRunID+"\n"))
|
seedRestoreObject(fake, runIDKey, []byte(previousRunID+"\n"))
|
||||||
|
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package app
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
@@ -13,6 +14,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/pathsafe"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/previouscache"
|
"gitea.maximumdirect.net/eric/narratio/internal/previouscache"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -215,19 +217,17 @@ func joinWithinSessionRoot(sessionRoot, relative string) (string, error) {
|
|||||||
if strings.TrimSpace(sessionRoot) == "" {
|
if strings.TrimSpace(sessionRoot) == "" {
|
||||||
return "", fmt.Errorf("session root is required")
|
return "", fmt.Errorf("session root is required")
|
||||||
}
|
}
|
||||||
cleanRel := path.Clean(strings.TrimSpace(relative))
|
joined, err := pathsafe.JoinSlashRelativeUnderRoot(sessionRoot, filepath.ToSlash(strings.TrimSpace(relative)))
|
||||||
if cleanRel == "." || cleanRel == "" {
|
if err != nil {
|
||||||
|
if errors.Is(err, pathsafe.ErrRelativePathRequired) {
|
||||||
return "", fmt.Errorf("relative path is required")
|
return "", fmt.Errorf("relative path is required")
|
||||||
}
|
}
|
||||||
if cleanRel == ".." || strings.HasPrefix(cleanRel, "../") || strings.HasPrefix(cleanRel, "/") {
|
if errors.Is(err, pathsafe.ErrRelativePathEscape) || errors.Is(err, pathsafe.ErrRelativePathAbsolute) {
|
||||||
return "", fmt.Errorf("relative path escapes session root")
|
return "", fmt.Errorf("relative path escapes session root")
|
||||||
}
|
}
|
||||||
abs := filepath.Clean(filepath.Join(sessionRoot, filepath.FromSlash(cleanRel)))
|
return "", fmt.Errorf("join relative path under session root: %w", err)
|
||||||
root := filepath.Clean(sessionRoot)
|
|
||||||
if abs != root && !strings.HasPrefix(abs, root+string(filepath.Separator)) {
|
|
||||||
return "", fmt.Errorf("resolved local path escapes session root")
|
|
||||||
}
|
}
|
||||||
return abs, nil
|
return joined, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func buildPreviousCacheRestoreActions(
|
func buildPreviousCacheRestoreActions(
|
||||||
@@ -338,7 +338,7 @@ func classifyRestoreAction(
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return RestoreAction{}, fmt.Errorf("checksum local file: %w", err)
|
return RestoreAction{}, fmt.Errorf("checksum local file: %w", err)
|
||||||
}
|
}
|
||||||
remotePath, err := downloadObjectToTemp(ctx, store, action.RemoteKey, "narratio-restore-plan-remote-*.tmp")
|
remotePath, err := storage.DownloadObjectToTemp(ctx, store, action.RemoteKey, "narratio-restore-plan-remote-*.tmp")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return RestoreAction{}, fmt.Errorf("download remote object: %w", err)
|
return RestoreAction{}, fmt.Errorf("download remote object: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ func TestRestorePlanDefaultScope(t *testing.T) {
|
|||||||
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("# recap\n"))
|
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("# recap\n"))
|
||||||
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
|
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
|
||||||
seedRestoreObject(store, current.SessionPrefix+"runs/20260519T010203Z-a1b2/manifest.json", []byte("{}"))
|
seedRestoreObject(store, current.SessionPrefix+"runs/20260519T010203Z-a1b2/manifest.json", []byte("{}"))
|
||||||
seedRestoreObject(store, current.SessionPrefix+"logs/archive.log", []byte("log"))
|
seedRestoreObject(store, current.SessionPrefix+"logs/publish.log", []byte("log"))
|
||||||
|
|
||||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -325,7 +325,7 @@ func configureRestorePlanPreviousRequirement(cfg *config.Config, required bool)
|
|||||||
func restorePlanCurrentState(t *testing.T, cfg *config.Config) *RemoteCurrentState {
|
func restorePlanCurrentState(t *testing.T, cfg *config.Config) *RemoteCurrentState {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
sessionPrefix := artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
|
sessionPrefix := artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
manifestKey, runIDKey := artifacts.ResolveArchiveCurrentStateKeys(sessionPrefix)
|
manifestKey, runIDKey := artifacts.ResolveCurrentStateKeys(sessionPrefix)
|
||||||
return &RemoteCurrentState{
|
return &RemoteCurrentState{
|
||||||
Bucket: "test-bucket",
|
Bucket: "test-bucket",
|
||||||
SessionPrefix: sessionPrefix,
|
SessionPrefix: sessionPrefix,
|
||||||
|
|||||||
@@ -201,7 +201,7 @@ func writeRestoreSuccessSummary(out io.Writer, report *RestoreReport) error {
|
|||||||
if report == nil {
|
if report == nil {
|
||||||
return fmt.Errorf("restore report is required")
|
return fmt.Errorf("restore report is required")
|
||||||
}
|
}
|
||||||
if _, err := fmt.Fprintf(out, "Restored session archive for %s/%s\n", report.Campaign, report.SessionID); err != nil {
|
if _, err := fmt.Fprintf(out, "Restored session state for %s/%s\n", report.Campaign, report.SessionID); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if _, err := fmt.Fprintf(out, "Remote run: %s\n", report.RunID); err != nil {
|
if _, err := fmt.Fprintf(out, "Remote run: %s\n", report.RunID); err != nil {
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user