Compare commits
48 Commits
5620fc5bcf
...
v0.13.0
| Author | SHA1 | Date | |
|---|---|---|---|
| 74e2d21de5 | |||
| 7cb18a1a40 | |||
| b556fc2f4f | |||
| b99bd38eb4 | |||
| 701b6726d7 | |||
| 665039f4dc | |||
| ef8dae776e | |||
| d01775b68a | |||
| 0d6f2dd0ce | |||
| df40cbec6e | |||
| 0341e0c7c0 | |||
| 39af7d4f3c | |||
| bba582b4ca | |||
| 1f16a85330 | |||
| f9482639d4 | |||
| dce721cdbd | |||
| 98734644d6 | |||
| 951383226c | |||
| df58595d1e | |||
| c3c14e7468 | |||
| e7319ea016 | |||
| bd2d5e2496 | |||
| 115a44f629 | |||
| 18411dc5b5 | |||
| e23dc1ab6e | |||
| e1359ea227 | |||
| 7fdd99ec27 | |||
| a90231ce0c | |||
| ed879b8bb0 | |||
| 717451512a | |||
| 3ddb3a947b | |||
| c6632d5576 | |||
| ffc07922c7 | |||
| f3310d4d16 | |||
| 88cee96d8d | |||
| 2fece10215 | |||
| 0658f2f642 | |||
| a51228c803 | |||
| 4491fb5ccd | |||
| 30b905765c | |||
| 03eac70881 | |||
| 0f7e6b979f | |||
| c366912586 | |||
| 9fe44cd00d | |||
| 094b0d2532 | |||
| 98649f4d81 | |||
| 8a559efd5b | |||
| 72deccb4e2 |
44
README.md
44
README.md
@@ -1,22 +1,42 @@
|
|||||||
# narratio
|
# narratio
|
||||||
|
|
||||||
Narratio is a stage-driven Go orchestrator for turning D&D session audio into polished transcripts and generated artifacts.
|
Narratio is a stage-driven Go orchestrator for turning D&D session audio into
|
||||||
|
polished transcripts, validated Notarius extraction lanes, and generated
|
||||||
|
artifacts.
|
||||||
|
|
||||||
It runs a deterministic workflow across `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`, and `publish`, with manifest-driven resume and restore support.
|
It runs a deterministic workflow with manifest-driven continuation, remote
|
||||||
|
publish, and restore support.
|
||||||
|
|
||||||
```bash
|
```sh
|
||||||
narratio run 2026-04-04
|
narratio run 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
This requires resolvable `pipeline.yml`, `campaign.yml`, and concrete `session.yml` (or explicit config flags).
|
This requires resolvable `pipeline.yml`, `campaign.yml`, and concrete
|
||||||
|
`session.yml` files or their explicit command-line alternatives.
|
||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
- [CLI Reference](docs/cli.md)
|
- [CLI reference](docs/cli.md) — commands, arguments, flags, and invocation
|
||||||
- [Configuration](docs/config.md)
|
behavior.
|
||||||
- [Operations](docs/operations.md)
|
- [Configuration](docs/config.md) — discovery, fields, defaults, and
|
||||||
- [Troubleshooting](docs/troubleshooting.md)
|
validation.
|
||||||
- [Internal Component Contracts](docs/internal/README.md)
|
- [Operations](docs/operations.md) — runtime workflow, state, publishing,
|
||||||
- [Development Guide](docs/policy/development.md)
|
recovery, and cleanup.
|
||||||
- [Architecture Principles](docs/policy/architecture.md)
|
- [Troubleshooting](docs/troubleshooting.md) — symptom-driven diagnosis and
|
||||||
- [Maintained Examples](examples/)
|
safe remedies.
|
||||||
|
- [Integration contracts](docs/integrations/) — external tools, formats, and
|
||||||
|
compatibility expectations.
|
||||||
|
- [Maintained examples](examples/README.md) — complete copyable configuration
|
||||||
|
and input files.
|
||||||
|
|
||||||
|
## Maintainer Documentation
|
||||||
|
|
||||||
|
- [Development guide](docs/development.md) — first-read orientation and
|
||||||
|
task-specific reading routes.
|
||||||
|
- [Internal overview](docs/internal/overview.md) — implemented component map.
|
||||||
|
- [Architecture](docs/policy/architecture.md) — normative boundaries and
|
||||||
|
invariants.
|
||||||
|
- [Documentation policy](docs/policy/documentation.md) — canonical ownership
|
||||||
|
and maintenance rules.
|
||||||
|
- [Testing policy](docs/policy/testing.md) — test value, boundaries, and
|
||||||
|
sufficiency.
|
||||||
|
|||||||
74
docs/cli.md
74
docs/cli.md
@@ -13,7 +13,6 @@ This runs the canonical full pipeline for session `2026-04-04`.
|
|||||||
Top-level commands:
|
Top-level commands:
|
||||||
|
|
||||||
- `run <session_id>`: run full stage order.
|
- `run <session_id>`: run full stage order.
|
||||||
- `resume <session_id>`: continue from first non-succeeded stage.
|
|
||||||
- `run-stage <stage> <session_id>`: run one stage.
|
- `run-stage <stage> <session_id>`: run one stage.
|
||||||
- `analyze <session_id>`: force-run analyze.
|
- `analyze <session_id>`: force-run analyze.
|
||||||
- `publish <session_id>`: force-run publish.
|
- `publish <session_id>`: force-run publish.
|
||||||
@@ -76,20 +75,13 @@ narratio run <session_id> [--force] [--artifacts <name[,name...]>] [...common co
|
|||||||
Behavior:
|
Behavior:
|
||||||
|
|
||||||
- evaluates full stage order;
|
- evaluates full stage order;
|
||||||
- skips already-succeeded stages unless `--force` is set;
|
- runs `extract` between `trim` and `render`; an omitted or disabled Notarius
|
||||||
|
configuration records an explicit `notarius_disabled` self-skip;
|
||||||
|
- skips already-succeeded stages unless `--force` is set or a stage-specific
|
||||||
|
resume check finds its durable result obsolete;
|
||||||
|
- continues interrupted or partially completed sessions by running non-succeeded stages;
|
||||||
- writes session and run manifests.
|
- writes session and run manifests.
|
||||||
|
|
||||||
### `resume`
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio resume <session_id> [--force] [--artifacts <name[,name...]>] [...common config flags]
|
|
||||||
```
|
|
||||||
|
|
||||||
Behavior:
|
|
||||||
|
|
||||||
- when not forced, starts at first non-succeeded stage in manifest order;
|
|
||||||
- with `--force`, reevaluates the selected stage list as runnable.
|
|
||||||
|
|
||||||
### `run-stage`
|
### `run-stage`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -104,6 +96,8 @@ Valid stage names:
|
|||||||
- `polish`
|
- `polish`
|
||||||
- `normalize`
|
- `normalize`
|
||||||
- `trim`
|
- `trim`
|
||||||
|
- `extract`
|
||||||
|
- `render`
|
||||||
- `analyze`
|
- `analyze`
|
||||||
- `publish`
|
- `publish`
|
||||||
- `notify`
|
- `notify`
|
||||||
@@ -145,14 +139,13 @@ narratio clean --all [--dry-run] [--clear-cache] [--config <pipeline.yml>]
|
|||||||
|
|
||||||
Behavior:
|
Behavior:
|
||||||
|
|
||||||
- session mode removes:
|
- session mode removes the selected session's local work and spool state;
|
||||||
- `{workspace.root}/work/{campaign}/{session_id}`
|
- `--all` removes all local session work and spool state;
|
||||||
- `{spool.root}/{campaign}/{session_id}`
|
|
||||||
- `--all` removes:
|
|
||||||
- `{workspace.root}/work/*`
|
|
||||||
- direct children under `{spool.root}`
|
|
||||||
- cache remains unless `--clear-cache` is provided.
|
- cache remains unless `--clear-cache` is provided.
|
||||||
|
|
||||||
|
See [Operations: Cleanup](./operations.md#cleanup) for deletion scope and
|
||||||
|
post-publish cleanup behavior.
|
||||||
|
|
||||||
### `session plan`
|
### `session plan`
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -217,17 +210,11 @@ Behavior:
|
|||||||
|
|
||||||
- discovers committed remote current state;
|
- discovers committed remote current state;
|
||||||
- plans local restores;
|
- plans local restores;
|
||||||
- writes `reports/restore-latest.json` on execution;
|
- writes an execution report;
|
||||||
- blocks conflicting overwrites unless `--force` is set.
|
- blocks conflicting overwrites unless `--force` is set.
|
||||||
|
|
||||||
Default restore scope:
|
See [Operations: Restore Workflow](./operations.md#restore-workflow) for the
|
||||||
|
default restore scope, report location, and conflict-handling workflow.
|
||||||
- `manifest.json`
|
|
||||||
- `transcripts/**`
|
|
||||||
- `artifacts/**`
|
|
||||||
- `previous/**` when required by configured previous-session inputs
|
|
||||||
|
|
||||||
`audio/**` is included only with `--include-audio`.
|
|
||||||
|
|
||||||
### `session artifacts`
|
### `session artifacts`
|
||||||
|
|
||||||
@@ -235,7 +222,10 @@ Default restore scope:
|
|||||||
narratio session artifacts <session_id> [--remote] [...common config flags]
|
narratio session artifacts <session_id> [--remote] [...common config flags]
|
||||||
```
|
```
|
||||||
|
|
||||||
Lists effective built-in and configured artifact sources, publish rules, lock state, and optional remote published-state availability.
|
Lists effective built-in, configured Scriptorium, and configured extraction
|
||||||
|
sources; reports planned, available, unavailable, and published state without
|
||||||
|
reading payload bodies; and includes publish rules, lock state, and optional
|
||||||
|
remote published-state availability.
|
||||||
|
|
||||||
### `session locks`
|
### `session locks`
|
||||||
|
|
||||||
@@ -247,13 +237,16 @@ narratio session locks remove <session_id> <source> [...common config flags]
|
|||||||
|
|
||||||
Behavior:
|
Behavior:
|
||||||
|
|
||||||
- list mode merges static `pipeline.publish.locks` with remote `{session_prefix}/locks.yml`;
|
- list mode reports the effective merge of static and remote locks;
|
||||||
- add/remove mutate only remote locks;
|
- add/remove mutate only remote locks;
|
||||||
- static locks from pipeline config cannot be removed by CLI commands.
|
- static locks from pipeline config cannot be removed by CLI commands.
|
||||||
|
|
||||||
|
See [Operations: Publish Locks](./operations.md#publish-locks) for lock storage
|
||||||
|
and precedence.
|
||||||
|
|
||||||
## `--artifacts` Selection Rules
|
## `--artifacts` Selection Rules
|
||||||
|
|
||||||
- accepted on `run`, `resume`, `run-stage`, `analyze`, and `publish`;
|
- accepted on `run`, `run-stage`, `analyze`, and `publish`;
|
||||||
- names must exist in `pipeline.scriptorium.artifacts`;
|
- names must exist in `pipeline.scriptorium.artifacts`;
|
||||||
- empty entries are invalid;
|
- empty entries are invalid;
|
||||||
- repeated names are deduplicated.
|
- repeated names are deduplicated.
|
||||||
@@ -262,7 +255,9 @@ Effects:
|
|||||||
|
|
||||||
- filters analyze execution to selected configured artifacts;
|
- filters analyze execution to selected configured artifacts;
|
||||||
- filters publish rules that source `narratio.artifact.<name>`;
|
- filters publish rules that source `narratio.artifact.<name>`;
|
||||||
- does not filter built-in transcript/bounds publish sources.
|
- does not filter built-in transcript/bounds or explicitly configured
|
||||||
|
`narratio.extraction.<name>` publish sources; and
|
||||||
|
- does not select or filter Notarius lanes.
|
||||||
|
|
||||||
## Common Workflows
|
## Common Workflows
|
||||||
|
|
||||||
@@ -289,3 +284,18 @@ Force publish only:
|
|||||||
```bash
|
```bash
|
||||||
narratio publish 2026-04-04
|
narratio publish 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Output And Exit Behavior
|
||||||
|
|
||||||
|
- Successful commands write their result or summary to standard output and
|
||||||
|
exit with status `0`.
|
||||||
|
- Command failures and invalid invocations write an error to standard error and
|
||||||
|
exit with status `1`.
|
||||||
|
- An unknown top-level command also prints the top-level usage summary to
|
||||||
|
standard error.
|
||||||
|
- `session restore --help` prints its command-specific usage and exits with
|
||||||
|
status `0`.
|
||||||
|
|
||||||
|
Output is intended for operator inspection. Narratio does not currently offer
|
||||||
|
a machine-readable CLI output mode; durable machine-readable state is recorded
|
||||||
|
in manifests and reports described in [Operations](./operations.md).
|
||||||
|
|||||||
@@ -44,7 +44,7 @@ using configured object storage.
|
|||||||
- Session files must be concrete; unresolved `{{ ... }}` placeholders fail load.
|
- Session files must be concrete; unresolved `{{ ... }}` placeholders fail load.
|
||||||
- Pipeline defaults are applied before validation.
|
- Pipeline defaults are applied before validation.
|
||||||
- Campaign and session identities must agree.
|
- Campaign and session identities must agree.
|
||||||
- Stable files (`speakers_file`, `autocorrect_file`, `glossary_file`) resolve from session overrides when provided, otherwise from campaign defaults.
|
- Stable files (`speakers_file`, `autocorrect_file`, `glossary_file`, `players_file`, `party_file`) resolve from session overrides when provided, otherwise from campaign defaults.
|
||||||
- Exactly one audio mode must be configured in session input:
|
- Exactly one audio mode must be configured in session input:
|
||||||
- local (`audio_dir` or `audio_files`), or
|
- local (`audio_dir` or `audio_files`), or
|
||||||
- S3 (`audio_s3.prefix`).
|
- S3 (`audio_s3.prefix`).
|
||||||
@@ -69,6 +69,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
```
|
```
|
||||||
|
|
||||||
`session.yml` (local audio)
|
`session.yml` (local audio)
|
||||||
@@ -98,6 +100,12 @@ publish:
|
|||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
required: true
|
required: true
|
||||||
|
- source: narratio.transcript.final_markdown
|
||||||
|
dest: transcripts/final.md
|
||||||
|
required: true
|
||||||
|
- source: narratio.transcript.final_trimmed_markdown
|
||||||
|
dest: transcripts/final.trimmed.md
|
||||||
|
required: true
|
||||||
- source: narratio.artifact.session_recap
|
- source: narratio.artifact.session_recap
|
||||||
dest: artifacts/session_recap.md
|
dest: artifacts/session_recap.md
|
||||||
required: true
|
required: true
|
||||||
@@ -110,6 +118,9 @@ Rules:
|
|||||||
|
|
||||||
- `outputs[].source` is required.
|
- `outputs[].source` is required.
|
||||||
- `outputs[].dest` may be omitted when derivable from source.
|
- `outputs[].dest` may be omitted when derivable from source.
|
||||||
|
- extraction sources require an explicit `outputs[].dest` and publish only when
|
||||||
|
a rule names that source; the Notarius index and complete bundle are not
|
||||||
|
publish sources.
|
||||||
- `outputs[].required` defaults to `true`.
|
- `outputs[].required` defaults to `true`.
|
||||||
- static locks (`pipeline.publish.locks`) merge with remote locks (`{session_prefix}/locks.yml`), with static locks taking precedence on duplicates.
|
- static locks (`pipeline.publish.locks`) merge with remote locks (`{session_prefix}/locks.yml`), with static locks taking precedence on duplicates.
|
||||||
|
|
||||||
@@ -138,7 +149,7 @@ Rules:
|
|||||||
| `pipeline.cache.s3_audio` | bool | No | `true` |
|
| `pipeline.cache.s3_audio` | bool | No | `true` |
|
||||||
| `pipeline.publish.enabled` | bool | No | `true` |
|
| `pipeline.publish.enabled` | bool | No | `true` |
|
||||||
| `pipeline.publish.upload_run` | bool | No | `true` |
|
| `pipeline.publish.upload_run` | bool | No | `true` |
|
||||||
| `pipeline.publish.outputs[]` | list | No | defaults to final trimmed transcript output |
|
| `pipeline.publish.outputs[]` | list | No | defaults to final trimmed JSON plus final and final-trimmed Markdown outputs |
|
||||||
| `pipeline.publish.outputs[].source` | string | Yes (per rule) | must reference built-in or configured artifact source |
|
| `pipeline.publish.outputs[].source` | string | Yes (per rule) | must reference built-in or configured artifact source |
|
||||||
| `pipeline.publish.outputs[].dest` | string | Conditional | derived if omitted and source supports derivation |
|
| `pipeline.publish.outputs[].dest` | string | Conditional | derived if omitted and source supports derivation |
|
||||||
| `pipeline.publish.outputs[].required` | bool | No | `true` |
|
| `pipeline.publish.outputs[].required` | bool | No | `true` |
|
||||||
@@ -178,16 +189,29 @@ Rules:
|
|||||||
| `pipeline.normalize.output_path` | string | No | `transcripts/final.json` |
|
| `pipeline.normalize.output_path` | string | No | `transcripts/final.json` |
|
||||||
| `pipeline.normalize.output_schema` | string | No | `seriatim-intermediate` |
|
| `pipeline.normalize.output_schema` | string | No | `seriatim-intermediate` |
|
||||||
| `pipeline.normalize.report` | bool | No | `true` |
|
| `pipeline.normalize.report` | bool | No | `true` |
|
||||||
| `pipeline.trim.enabled` | bool | No | `false` |
|
| `pipeline.trim.enabled` | bool | No | `true` |
|
||||||
| `pipeline.trim.output_path` | string | Conditional | required when trim enabled |
|
| `pipeline.trim.output_path` | string | No | `transcripts/final.trimmed.json` |
|
||||||
| `pipeline.trim.bounds.prompt_id` | string | Conditional | required when trim enabled |
|
| `pipeline.trim.bounds.prompt_id` | string | No | `dnd.session_bounds` |
|
||||||
| `pipeline.trim.bounds.profile_id` | string | No | empty |
|
| `pipeline.trim.bounds.profile_id` | string | No | empty |
|
||||||
| `pipeline.trim.bounds.transcript_input_name` | string | Conditional | required when trim enabled |
|
| `pipeline.trim.bounds.transcript_input_name` | string | No | `transcript` |
|
||||||
| `pipeline.trim.bounds.output_path` | string | Conditional | required when trim enabled |
|
| `pipeline.trim.bounds.output_path` | string | No | `artifacts/session_bounds.json` |
|
||||||
| `pipeline.trim.bounds.timeout` | duration | No | `10m` |
|
| `pipeline.trim.bounds.timeout` | duration | No | `10m` |
|
||||||
| `pipeline.trim.bounds.render_debug` | bool | No | `false` |
|
| `pipeline.trim.bounds.render_debug` | bool | No | `false` |
|
||||||
| `pipeline.trim.bounds.render_output_path` | string | Conditional | required when `render_debug` is true |
|
| `pipeline.trim.bounds.render_output_path` | string | Conditional | required when `render_debug` is true |
|
||||||
| `pipeline.trim.seriatim.report` | bool | No | `false` |
|
| `pipeline.trim.seriatim.report` | bool | No | `false` |
|
||||||
|
| `pipeline.notarius.enabled` | bool | No | `false` |
|
||||||
|
| `pipeline.notarius.binary` | string | No | `notarius` |
|
||||||
|
| `pipeline.notarius.config_path` | string | Conditional | required when enabled; relative paths resolve from the pipeline file directory |
|
||||||
|
| `pipeline.notarius.pipeline_id` | string | Conditional | required when enabled |
|
||||||
|
| `pipeline.notarius.timeout` | duration | No | `3h`; must be positive |
|
||||||
|
| `pipeline.notarius.working_directory` | string | No | directory containing resolved `config_path`; relative paths resolve from the pipeline file directory |
|
||||||
|
| `pipeline.notarius.outputs` | map | Conditional | at least one entry when enabled |
|
||||||
|
| `pipeline.render.enabled` | bool | No | `true` |
|
||||||
|
| `pipeline.render.format` | string | No | `markdown` (only supported value) |
|
||||||
|
| `pipeline.render.title` | string | No | empty (falls back to `session.title` when set) |
|
||||||
|
| `pipeline.render.include_timestamps` | bool | No | `true` |
|
||||||
|
| `pipeline.render.include_segment_ids` | bool | No | `true` |
|
||||||
|
| `pipeline.render.include_metadata` | bool | No | `false` |
|
||||||
| `pipeline.scriptorium.binary` | string | No | `scriptorium` |
|
| `pipeline.scriptorium.binary` | string | No | `scriptorium` |
|
||||||
| `pipeline.scriptorium.config_path` | string | No | empty |
|
| `pipeline.scriptorium.config_path` | string | No | empty |
|
||||||
| `pipeline.scriptorium.timeout` | duration | No | `10m` |
|
| `pipeline.scriptorium.timeout` | duration | No | `10m` |
|
||||||
@@ -195,7 +219,27 @@ Rules:
|
|||||||
| `pipeline.scriptorium.artifacts` | map | No | empty |
|
| `pipeline.scriptorium.artifacts` | map | No | empty |
|
||||||
| `pipeline.notification.backend` | string | No | empty |
|
| `pipeline.notification.backend` | string | No | empty |
|
||||||
| `pipeline.notification.recipient` | string | No | empty |
|
| `pipeline.notification.recipient` | string | No | empty |
|
||||||
| `pipeline.notification.timeout` | duration | No | `30s` |
|
| `pipeline.notification.timeout` | duration | No | empty |
|
||||||
|
|
||||||
|
### Notarius Output Entries
|
||||||
|
|
||||||
|
For each `pipeline.notarius.outputs.<name>`:
|
||||||
|
|
||||||
|
| Field | Type | Required | Rule |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `lane_id` | string | Yes | unique Notarius lane ID |
|
||||||
|
| `media_type` | string | Yes | exact accepted descriptor media type |
|
||||||
|
| `schema_id` | string | Yes | exact accepted descriptor schema ID |
|
||||||
|
| `schema_version` | string | Yes | exact accepted descriptor schema version |
|
||||||
|
| `module_key` | string | No | exact accepted module key when set |
|
||||||
|
|
||||||
|
Output names must match `^[a-z][a-z0-9_]*$` and become selectable sources named
|
||||||
|
`narratio.extraction.<name>`. Lane IDs must be unique. Every declared output is
|
||||||
|
required from a successful Notarius result; a missing, rejected, duplicate, or
|
||||||
|
contract-incompatible lane fails extraction. See the
|
||||||
|
[complete maintained example](../examples/pipeline.full.annotated.yml) for the
|
||||||
|
current ten-lane D&D mapping and the [Notarius contract](./integrations/notarius.md)
|
||||||
|
for compatibility ownership.
|
||||||
|
|
||||||
### Scriptorium Artifact Entries
|
### Scriptorium Artifact Entries
|
||||||
|
|
||||||
@@ -211,13 +255,15 @@ For each `pipeline.scriptorium.artifacts.<name>`:
|
|||||||
| `output_path` | string | Conditional | required when enabled; also required when referenced by publish/output/input rules |
|
| `output_path` | string | Conditional | required when enabled; also required when referenced by publish/output/input rules |
|
||||||
| `timeout` | duration | No | artifact override |
|
| `timeout` | duration | No | artifact override |
|
||||||
| `inputs` | map | No | input key names must be non-empty |
|
| `inputs` | map | No | input key names must be non-empty |
|
||||||
| `vars` | map | No | values must be string or bool |
|
| `vars` | map | No | values must be string or bool; `session_id` is reserved and overwritten by Narratio |
|
||||||
|
|
||||||
|
Narratio adds `session_id=narratio-session-<session_id>` to every Scriptorium request for sticky upstream LLM routing. If an artifact config sets `vars.session_id`, Narratio replaces that value before invoking Scriptorium. Use a different variable name if a prompt needs the raw Narratio session ID as content.
|
||||||
|
|
||||||
For each artifact input `pipeline.scriptorium.artifacts.<name>.inputs.<input_name>`:
|
For each artifact input `pipeline.scriptorium.artifacts.<name>.inputs.<input_name>`:
|
||||||
|
|
||||||
| Field | Type | Required | Rule |
|
| Field | Type | Required | Rule |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| `source` | string | Yes | built-in runtime source, `narratio.artifact.<name>`, or `narratio.previous_session.artifact.<name>` |
|
| `source` | string | Yes | built-in runtime source, prepared input source, `narratio.extraction.<name>`, `narratio.artifact.<name>`, or `narratio.previous_session.artifact.<name>` |
|
||||||
| `artifact` | string | No | optional passthrough adapter field |
|
| `artifact` | string | No | optional passthrough adapter field |
|
||||||
| `path` | string | No | optional passthrough adapter field |
|
| `path` | string | No | optional passthrough adapter field |
|
||||||
| `required` | bool | No | optional input requirement |
|
| `required` | bool | No | optional input requirement |
|
||||||
@@ -231,6 +277,8 @@ For each artifact input `pipeline.scriptorium.artifacts.<name>.inputs.<input_nam
|
|||||||
| `inputs.speakers_file` | string | Yes | stable input default |
|
| `inputs.speakers_file` | string | Yes | stable input default |
|
||||||
| `inputs.autocorrect_file` | string | Yes | stable input default |
|
| `inputs.autocorrect_file` | string | Yes | stable input default |
|
||||||
| `inputs.glossary_file` | string | Yes | stable input default |
|
| `inputs.glossary_file` | string | Yes | stable input default |
|
||||||
|
| `inputs.players_file` | string | Yes | stable input default |
|
||||||
|
| `inputs.party_file` | string | Yes | stable input default |
|
||||||
|
|
||||||
### Session
|
### Session
|
||||||
|
|
||||||
@@ -244,6 +292,8 @@ For each artifact input `pipeline.scriptorium.artifacts.<name>.inputs.<input_nam
|
|||||||
| `inputs.speakers_file` | string | No | overrides campaign stable input |
|
| `inputs.speakers_file` | string | No | overrides campaign stable input |
|
||||||
| `inputs.autocorrect_file` | string | No | overrides campaign stable input |
|
| `inputs.autocorrect_file` | string | No | overrides campaign stable input |
|
||||||
| `inputs.glossary_file` | string | No | overrides campaign stable input |
|
| `inputs.glossary_file` | string | No | overrides campaign stable input |
|
||||||
|
| `inputs.players_file` | string | No | overrides campaign stable input |
|
||||||
|
| `inputs.party_file` | string | No | overrides campaign stable input |
|
||||||
| `inputs.audio_dir` | string | Conditional | local audio mode |
|
| `inputs.audio_dir` | string | Conditional | local audio mode |
|
||||||
| `inputs.audio_files[]` | list[string] | Conditional | local audio mode |
|
| `inputs.audio_files[]` | list[string] | Conditional | local audio mode |
|
||||||
| `inputs.audio_s3.prefix` | string | Conditional | S3 audio mode |
|
| `inputs.audio_s3.prefix` | string | Conditional | S3 audio mode |
|
||||||
@@ -254,10 +304,6 @@ Audio rules:
|
|||||||
|
|
||||||
## Maintained Examples
|
## Maintained Examples
|
||||||
|
|
||||||
- `examples/pipeline.minimal.yml`
|
See the [maintained examples index](../examples/README.md) for complete pipeline,
|
||||||
- `examples/pipeline.production.yml`
|
campaign, session, template, and input fixtures. Keep complete copyable files
|
||||||
- `examples/pipeline.full.annotated.yml`
|
there rather than duplicating them in this reference.
|
||||||
- `examples/campaigns/sample-campaign/campaign.yml`
|
|
||||||
- `examples/session.local-audio.yml`
|
|
||||||
- `examples/session.s3-audio.yml`
|
|
||||||
- `examples/session.template.yml`
|
|
||||||
|
|||||||
43
docs/development.md
Normal file
43
docs/development.md
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
# Development
|
||||||
|
|
||||||
|
This is the first-read landing page for people and LLM coding agents working on
|
||||||
|
Narratio. It provides a concise repository orientation and routes each kind of
|
||||||
|
change to its canonical documentation.
|
||||||
|
|
||||||
|
Narratio is a stage-driven Go orchestrator for turning D&D session audio into
|
||||||
|
polished transcripts and generated artifacts. Start with the
|
||||||
|
[README](../README.md) for product context,
|
||||||
|
[Architecture](policy/architecture.md) for normative system boundaries, and the
|
||||||
|
[Internal Overview](internal/overview.md) for implemented component ownership.
|
||||||
|
|
||||||
|
## What To Read
|
||||||
|
|
||||||
|
| When working on | Read | Why |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Finding the package or component that owns current behavior | [Internal Overview](internal/overview.md) | It is the implemented component inventory and routes to focused internal documents. |
|
||||||
|
| Application shape, boundaries, dependency direction, runtime invariants, safety properties, or dependencies | [Architecture](policy/architecture.md) | It defines the intended system shape, ownership, and non-goals. |
|
||||||
|
| Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical owners, audiences, current-behavior rules, and maintenance requirements. |
|
||||||
|
| Adding, changing, reviewing, rewriting, or deleting tests | [Testing Policy](policy/testing.md) | It defines risk-based sufficiency, durable test boundaries, test-double guidance, and test lifecycle decisions. |
|
||||||
|
| CLI composition or command behavior | [Internal Overview](internal/overview.md) and [CLI Reference](cli.md) | The overview routes to command ownership; the reference owns public syntax and invocation behavior. |
|
||||||
|
| Configuration loading, resolution, or user-visible configuration | [Internal Overview](internal/overview.md) and [Configuration](config.md) | The overview routes to implementation ownership; the reference owns fields, defaults, discovery, and validation. |
|
||||||
|
| Session workflow, status, restore, cleanup, or object storage | [Restore Internals](internal/command-restore.md), [Workspace Internals](internal/workspace.md), [Storage Internals](internal/storage.md), [Operations](operations.md), and [Troubleshooting](troubleshooting.md) | These separate implementation mechanics, operator procedures, and symptom-driven recovery. |
|
||||||
|
| Pipeline sequencing or the behavior of a stage | [Internal Overview](internal/overview.md) and its focused stage documents | The overview owns the implemented stage inventory and routes to each stage contract. |
|
||||||
|
| Adapters or external tool contracts | [Adapter Internals](internal/adapters.md) and [Integration Contracts](integrations/README.md) | The internal guide owns adapter composition and mechanics; integration documents own external formats and protocols. |
|
||||||
|
| Manifests, artifacts, workspace paths, or publish behavior | [Manifest Internals](internal/manifest.md), [Artifact Internals](internal/artifacts.md), [Workspace Internals](internal/workspace.md), [Publish Internals](internal/stage-publish.md), and [Operations](operations.md) | These separate implementation state and resolution from operator-visible layout and lifecycle. |
|
||||||
|
| Maintained configuration or input examples | [Configuration](config.md) and [Examples](../examples/README.md) | The reference owns field meanings; the examples directory owns complete copyable files. |
|
||||||
|
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
|
||||||
|
|
||||||
|
For an existing subsystem, also inspect its focused tests and package-level
|
||||||
|
contracts before changing behavior.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
Use focused package tests while iterating. Run the repository-wide checks when a
|
||||||
|
change affects shared contracts, application behavior, or maintained
|
||||||
|
documentation examples:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go test ./...
|
||||||
|
go vet ./...
|
||||||
|
go build ./cmd/narratio
|
||||||
|
```
|
||||||
@@ -1,19 +1,35 @@
|
|||||||
# Integrations Index
|
# Integrations Index
|
||||||
|
|
||||||
## Audience
|
## Audience
|
||||||
Developers and coding agents changing Narratio's external integration boundaries.
|
|
||||||
|
Operators, developers, and coding agents who need to understand Narratio's
|
||||||
|
externally observable integration boundaries.
|
||||||
|
|
||||||
## Scope
|
## Scope
|
||||||
`docs/integrations/` is the implementation-level reference for downstream tool adapter contracts.
|
|
||||||
|
|
||||||
These docs cover what Narratio expects from external tools and what each adapter guarantees back to stage code.
|
`docs/integrations/` is the canonical reference for protocols, invocation and
|
||||||
|
data contracts, logical outputs, and compatibility behavior at external tool
|
||||||
|
boundaries.
|
||||||
|
|
||||||
|
These documents describe what Narratio sends or invokes, what it accepts in
|
||||||
|
return, and how failures are surfaced. Internal composition and stage mechanics
|
||||||
|
belong in [the adapter implementation guide](../internal/adapters.md) and the
|
||||||
|
focused stage documents.
|
||||||
|
|
||||||
## Integration Contracts
|
## Integration Contracts
|
||||||
- `audita.md`: transcript polishing adapter (`audita process`).
|
|
||||||
- `seriatim.md`: merge/normalize/trim adapter (`seriatim`).
|
- [Audita](./audita.md): transcript polishing (`audita process`).
|
||||||
- `scriptorium.md`: artifact run/render adapter (`scriptorium run|render`).
|
- [Notarius](./notarius.md): complete pipeline execution and safe JSON bundle
|
||||||
|
discovery (`notarius run`).
|
||||||
|
- [Seriatim](./seriatim.md): merge, normalize, trim, and render operations.
|
||||||
|
- [Scriptorium](./scriptorium.md): artifact generation and debug rendering
|
||||||
|
(`scriptorium run|render`).
|
||||||
|
- [WhisperX](./whisperx.md): speaker-audio transcription over HTTP.
|
||||||
|
|
||||||
## Related Canonical Docs
|
## Related Canonical Docs
|
||||||
- `docs/config.md`: operator-facing configuration reference.
|
|
||||||
- `docs/internal/adapters.md`: shared adapter boundary and runner wiring.
|
- [Configuration](../config.md): operator-facing configuration reference.
|
||||||
- `docs/internal/stage-*.md`: stage-specific integration usage.
|
- [Adapter implementation](../internal/adapters.md): shared adapter boundary and
|
||||||
|
runner wiring.
|
||||||
|
- [Internal documentation](../internal/overview.md): stage-specific integration
|
||||||
|
usage and component ownership.
|
||||||
|
|||||||
@@ -3,16 +3,12 @@
|
|||||||
## Purpose
|
## Purpose
|
||||||
Define the Audita adapter contract used by the `polish` stage.
|
Define the Audita adapter contract used by the `polish` stage.
|
||||||
|
|
||||||
## Adapter Boundary
|
## External Boundary
|
||||||
Interface:
|
|
||||||
- `audita.Runner`
|
|
||||||
- method: `Run(ctx, PolishRequest) (PolishResult, error)`
|
|
||||||
|
|
||||||
Primary implementation:
|
Narratio invokes `audita process` as a subprocess for each polish operation.
|
||||||
- `internal/adapters/audita/SubprocessRunner`
|
The configured timeout and parent cancellation bound the invocation. Internal
|
||||||
|
runner composition is documented in
|
||||||
Execution mode:
|
[the adapter implementation guide](../internal/adapters.md).
|
||||||
- subprocess invocation of `audita process`
|
|
||||||
|
|
||||||
## Request Contract
|
## Request Contract
|
||||||
`PolishRequest` carries:
|
`PolishRequest` carries:
|
||||||
@@ -52,9 +48,12 @@ Failure results still include output/log/config/exit metadata for diagnostics.
|
|||||||
- Generated invocation YAML (`audita.generated.v1`) is emitted when requested.
|
- Generated invocation YAML (`audita.generated.v1`) is emitted when requested.
|
||||||
- Manifest writes are stage-owned; adapter itself is stateless.
|
- Manifest writes are stage-owned; adapter itself is stateless.
|
||||||
|
|
||||||
## Config Mapping
|
## Configuration
|
||||||
Config fields consumed through runner/stage wiring are under `pipeline.audita.*`.
|
|
||||||
|
Operator-selected values are defined under `pipeline.audita.*` in the
|
||||||
|
[configuration reference](../config.md#pipeline).
|
||||||
|
|
||||||
Maintained example with Audita config:
|
Maintained example with Audita config:
|
||||||
- `examples/pipeline.full.annotated.yml`
|
|
||||||
- `examples/pipeline.production.yml`
|
- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml)
|
||||||
|
- [Production-shaped pipeline](../../examples/pipeline.production.yml)
|
||||||
|
|||||||
83
docs/integrations/notarius.md
Normal file
83
docs/integrations/notarius.md
Normal file
@@ -0,0 +1,83 @@
|
|||||||
|
# Notarius Integration Contract
|
||||||
|
|
||||||
|
## Boundary
|
||||||
|
|
||||||
|
Narratio uses Notarius as a subprocess to extract configured structured JSON
|
||||||
|
lanes from the final trimmed Seriatim transcript. Narratio owns invocation,
|
||||||
|
safe bundle discovery, lane selection, and its own artifact metadata. Notarius
|
||||||
|
owns pipeline definitions, lane schemas, the receipt, and bundle formats.
|
||||||
|
|
||||||
|
Canonical Notarius references:
|
||||||
|
|
||||||
|
- [Subprocess consumer contract](https://gitea.maximumdirect.net/eric/notarius/src/branch/main/docs/consumers/subprocess.md)
|
||||||
|
- [D&D pipeline and lane contracts](https://gitea.maximumdirect.net/eric/notarius/src/branch/main/docs/consumers/dnd-pipeline.md)
|
||||||
|
- [Run-result receipt](https://gitea.maximumdirect.net/eric/notarius/src/branch/main/docs/integrations/run-result.md)
|
||||||
|
- [JSON output bundle](https://gitea.maximumdirect.net/eric/notarius/src/branch/main/docs/integrations/json-output.md)
|
||||||
|
|
||||||
|
The [complete Narratio example](../../examples/pipeline.full.annotated.yml)
|
||||||
|
records the exact current constraints for all ten D&D lanes. Treat the linked
|
||||||
|
Notarius documents as canonical when changing those values; Narratio does not
|
||||||
|
duplicate the complete schemas.
|
||||||
|
|
||||||
|
## Invocation
|
||||||
|
|
||||||
|
When `pipeline.notarius.enabled` is true, Narratio resolves the executable,
|
||||||
|
configuration path, input path, output directory, and working directory to
|
||||||
|
absolute paths and invokes:
|
||||||
|
|
||||||
|
```text
|
||||||
|
notarius run <pipeline_id> --config <config_path> --input <trimmed_json> --output-dir <staging_dir> --json
|
||||||
|
```
|
||||||
|
|
||||||
|
Standard output is reserved for the JSON receipt. Standard error is captured
|
||||||
|
separately as diagnostic output. Narratio applies the configured timeout and
|
||||||
|
does not interpret stdout as a receipt unless the subprocess exits successfully.
|
||||||
|
It does not pass a Narratio session ID or run `notarius config validate`
|
||||||
|
automatically; the configured working directory and inherited environment
|
||||||
|
apply to the subprocess.
|
||||||
|
|
||||||
|
## Accepted Result
|
||||||
|
|
||||||
|
Narratio currently accepts receipt schema `notarius.run-result.v1`. The receipt
|
||||||
|
must identify the configured pipeline, and its `index_file` must be exactly
|
||||||
|
`index.json` beneath the reported bundle root. The production index must name
|
||||||
|
the management files exactly as `manifest.json`, `rejected.json`, and
|
||||||
|
`warnings.json`. All receipt, index, and lane paths must stay inside that
|
||||||
|
bundle; symlinks and non-regular lane payloads are rejected.
|
||||||
|
|
||||||
|
Supported receipt and index shapes tolerate unknown fields for forward
|
||||||
|
compatibility, while required identity, validation, count, manifest,
|
||||||
|
rejection, warning, and lane-list fields remain mandatory. Narratio applies
|
||||||
|
bounded reads to the receipt, index, rejection, and warning documents. Optional
|
||||||
|
chunk-map and evidence-context descriptors must carry their complete generic
|
||||||
|
contract metadata when present.
|
||||||
|
|
||||||
|
For every entry in `pipeline.notarius.outputs`, Narratio requires exactly one
|
||||||
|
index descriptor with the configured lane ID, media type, schema ID, schema
|
||||||
|
version, and, when configured, module key. Missing, duplicate, rejected, or
|
||||||
|
incompatible required lanes fail extraction even if Notarius exited zero.
|
||||||
|
Unconfigured lanes may remain in the preserved bundle but do not become
|
||||||
|
selectable Narratio sources.
|
||||||
|
|
||||||
|
Each accepted configured lane is registered as
|
||||||
|
`narratio.extraction.<output_key>`. The bundle index is retained for audit and
|
||||||
|
resume validation but is not selectable. Scriptorium and publish rules consume
|
||||||
|
only explicitly named lane sources; `--artifacts` never selects Notarius lanes.
|
||||||
|
|
||||||
|
## Failure And Compatibility Behavior
|
||||||
|
|
||||||
|
- Startup and nonzero-exit errors fail extraction and retain captured diagnostics.
|
||||||
|
- Invalid receipt JSON or an unsupported receipt schema fails before bundle use.
|
||||||
|
- Unsafe or incompatible index data and required-lane rejection fail before the
|
||||||
|
staged bundle is promoted to durable storage.
|
||||||
|
- Contract and external provenance metadata are preserved on lane artifact
|
||||||
|
records and through explicit publication.
|
||||||
|
|
||||||
|
Rejection and warning summaries retain structured stage, scope, lane, and
|
||||||
|
reason-code fields for diagnostics without exposing free-form external messages
|
||||||
|
or reading lane payload bodies.
|
||||||
|
|
||||||
|
Configuration fields and defaults are in [Configuration](../config.md).
|
||||||
|
Operator paths, rerun procedures, and bundle retention are in
|
||||||
|
[Operations](../operations.md). See [Troubleshooting](../troubleshooting.md)
|
||||||
|
for failure recovery.
|
||||||
@@ -3,20 +3,17 @@
|
|||||||
## Purpose
|
## Purpose
|
||||||
Define the Scriptorium adapter contract used by `analyze` and trim-bounds generation in `trim`.
|
Define the Scriptorium adapter contract used by `analyze` and trim-bounds generation in `trim`.
|
||||||
|
|
||||||
## Adapter Boundary
|
## External Boundary
|
||||||
Interface:
|
|
||||||
- `scriptorium.Runner`
|
|
||||||
- methods:
|
|
||||||
- `RunArtifact(ctx, RunArtifactRequest)`
|
|
||||||
- `RenderArtifact(ctx, RenderArtifactRequest)`
|
|
||||||
|
|
||||||
Primary implementation:
|
Narratio invokes Scriptorium as a subprocess in these modes:
|
||||||
- `internal/adapters/scriptorium/SubprocessRunner`
|
|
||||||
|
|
||||||
Execution modes:
|
|
||||||
- `scriptorium run`
|
- `scriptorium run`
|
||||||
- `scriptorium render`
|
- `scriptorium render`
|
||||||
|
|
||||||
|
The request timeout and parent cancellation bound each invocation. Internal
|
||||||
|
runner composition is documented in
|
||||||
|
[the adapter implementation guide](../internal/adapters.md).
|
||||||
|
|
||||||
## Request Contract
|
## Request Contract
|
||||||
Both request types carry:
|
Both request types carry:
|
||||||
- binary/config/prompt/profile IDs;
|
- binary/config/prompt/profile IDs;
|
||||||
@@ -55,12 +52,17 @@ Render behavior:
|
|||||||
|
|
||||||
## Deterministic Behavior
|
## Deterministic Behavior
|
||||||
- input and var maps are sorted into deterministic `--input` and `--var` CLI args.
|
- input and var maps are sorted into deterministic `--input` and `--var` CLI args.
|
||||||
|
- stage wiring adds `session_id=narratio-session-<session_id>` to every Scriptorium request for sticky upstream routing, overriding any configured `vars.session_id`.
|
||||||
- generated invocation YAML (`scriptorium.generated.v1`) is emitted when requested.
|
- generated invocation YAML (`scriptorium.generated.v1`) is emitted when requested.
|
||||||
- adapter is stateless and does not own artifact-selection policy.
|
- adapter is stateless and does not own artifact-selection policy.
|
||||||
|
|
||||||
## Config Mapping
|
## Configuration
|
||||||
Config fields consumed through runner/stage wiring are under `pipeline.scriptorium.*` plus per-artifact settings under `pipeline.scriptorium.artifacts.*`.
|
|
||||||
|
Operator-selected values are defined under `pipeline.scriptorium.*`, including
|
||||||
|
per-artifact settings under `pipeline.scriptorium.artifacts.*`, in the
|
||||||
|
[configuration reference](../config.md#pipeline).
|
||||||
|
|
||||||
Maintained examples with Scriptorium config:
|
Maintained examples with Scriptorium config:
|
||||||
- `examples/pipeline.full.annotated.yml`
|
|
||||||
- `examples/pipeline.production.yml`
|
- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml)
|
||||||
|
- [Production-shaped pipeline](../../examples/pipeline.production.yml)
|
||||||
|
|||||||
@@ -1,28 +1,26 @@
|
|||||||
# Integration: Seriatim
|
# Integration: Seriatim
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define the Seriatim adapter contract used by `merge`, `normalize`, and `trim`.
|
Define the Seriatim adapter contract used by `merge`, `normalize`, `trim`, and `render`.
|
||||||
|
|
||||||
## Adapter Boundary
|
## External Boundary
|
||||||
Interface:
|
|
||||||
- `seriatim.Runner`
|
|
||||||
- methods:
|
|
||||||
- `Run(ctx, MergeRequest)`
|
|
||||||
- `Normalize(ctx, NormalizeRequest)`
|
|
||||||
- `Trim(ctx, TrimRequest)`
|
|
||||||
|
|
||||||
Primary implementation:
|
Narratio invokes Seriatim as a subprocess in these modes:
|
||||||
- `internal/adapters/seriatim/SubprocessRunner`
|
|
||||||
|
|
||||||
Execution modes:
|
|
||||||
- `seriatim merge`
|
- `seriatim merge`
|
||||||
- `seriatim normalize`
|
- `seriatim normalize`
|
||||||
- `seriatim trim`
|
- `seriatim trim`
|
||||||
|
- `seriatim render`
|
||||||
|
|
||||||
|
The configured timeout and parent cancellation bound each invocation. Internal
|
||||||
|
runner composition is documented in
|
||||||
|
[the adapter implementation guide](../internal/adapters.md).
|
||||||
|
|
||||||
## Request/Result Contracts
|
## Request/Result Contracts
|
||||||
- `MergeRequest`/`MergeResult`: multi-input merge to base transcript, optional report.
|
- `MergeRequest`/`MergeResult`: multi-input merge to base transcript, optional report.
|
||||||
- `NormalizeRequest`/`NormalizeResult`: transcript normalization with explicit schema.
|
- `NormalizeRequest`/`NormalizeResult`: transcript normalization with explicit schema.
|
||||||
- `TrimRequest`/`TrimResult`: transcript trimming with required keep selector.
|
- `TrimRequest`/`TrimResult`: transcript trimming with required keep selector.
|
||||||
|
- `RenderRequest`/`RenderResult`: transcript-to-markdown rendering with explicit format and render booleans.
|
||||||
|
|
||||||
Results include output/log/config paths, timing, exit code, and metadata.
|
Results include output/log/config paths, timing, exit code, and metadata.
|
||||||
|
|
||||||
@@ -36,9 +34,11 @@ Runner construction validates:
|
|||||||
Invocation fails on:
|
Invocation fails on:
|
||||||
- missing required request paths/inputs;
|
- missing required request paths/inputs;
|
||||||
- invalid normalize schema override;
|
- invalid normalize schema override;
|
||||||
|
- unsupported render format;
|
||||||
- subprocess failure;
|
- subprocess failure;
|
||||||
- invalid JSON outputs;
|
- invalid JSON outputs for merge/normalize/trim;
|
||||||
- missing `segments` array for normalize/trim transcript outputs.
|
- missing `segments` array for normalize/trim transcript outputs;
|
||||||
|
- empty render output files.
|
||||||
|
|
||||||
When report paths are provided/enabled, report files must parse as JSON.
|
When report paths are provided/enabled, report files must parse as JSON.
|
||||||
|
|
||||||
@@ -48,9 +48,13 @@ When report paths are provided/enabled, report files must parse as JSON.
|
|||||||
- generated invocation YAML (`seriatim.generated.v1`) is emitted when requested.
|
- generated invocation YAML (`seriatim.generated.v1`) is emitted when requested.
|
||||||
- adapter does not write manifests or choose stage inputs.
|
- adapter does not write manifests or choose stage inputs.
|
||||||
|
|
||||||
## Config Mapping
|
## Configuration
|
||||||
Config fields consumed through runner/stage wiring are under `pipeline.seriatim.*`.
|
|
||||||
|
Operator-selected values are defined under `pipeline.seriatim.*` and
|
||||||
|
`pipeline.render.*` in the
|
||||||
|
[configuration reference](../config.md#pipeline).
|
||||||
|
|
||||||
Maintained examples with Seriatim config:
|
Maintained examples with Seriatim config:
|
||||||
- `examples/pipeline.full.annotated.yml`
|
|
||||||
- `examples/pipeline.production.yml`
|
- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml)
|
||||||
|
- [Production-shaped pipeline](../../examples/pipeline.production.yml)
|
||||||
|
|||||||
66
docs/integrations/whisperx.md
Normal file
66
docs/integrations/whisperx.md
Normal file
@@ -0,0 +1,66 @@
|
|||||||
|
# Integration: WhisperX
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
WhisperX transcribes each prepared speaker audio file for Narratio's
|
||||||
|
`transcribe` stage. Narratio uses an HTTP boundary and installs each successful
|
||||||
|
response as that speaker's raw transcript JSON.
|
||||||
|
|
||||||
|
## HTTP Boundary
|
||||||
|
|
||||||
|
Narratio sends an HTTP `POST` to the configured transcription URL using
|
||||||
|
`multipart/form-data` with:
|
||||||
|
|
||||||
|
- `file`: the audio file, retaining its base filename; and
|
||||||
|
- `language`: the configured language string.
|
||||||
|
|
||||||
|
The server must return a `2xx` response whose body is valid JSON. Narratio does
|
||||||
|
not currently require a more specific response schema at this boundary.
|
||||||
|
|
||||||
|
## Request And Result Contract
|
||||||
|
|
||||||
|
Each adapter request identifies a speaker, a readable audio file, and the
|
||||||
|
destination for the raw transcript. The HTTP request carries the audio and
|
||||||
|
language; the speaker identifier remains Narratio orchestration metadata.
|
||||||
|
|
||||||
|
On success, Narratio atomically writes the response body to the requested
|
||||||
|
destination. The adapter result reports that logical output together with the
|
||||||
|
attempt count, final HTTP status when available, elapsed duration, and adapter
|
||||||
|
identity metadata. A failed or invalid response is not installed as the
|
||||||
|
transcript output.
|
||||||
|
|
||||||
|
## Retry, Timeout, And Cancellation
|
||||||
|
|
||||||
|
- The configured timeout applies independently to each HTTP attempt.
|
||||||
|
- `retries` means additional attempts after the first.
|
||||||
|
- HTTP `429`, HTTP `5xx`, attempt timeouts, and network errors are retryable.
|
||||||
|
- Other HTTP `4xx` responses and explicit cancellation are not retryable.
|
||||||
|
- Narratio waits the configured retry delay between attempts and aborts that
|
||||||
|
wait when the parent context is canceled.
|
||||||
|
|
||||||
|
## Validation And Failure Semantics
|
||||||
|
|
||||||
|
Client construction rejects a missing or invalid absolute transcription URL,
|
||||||
|
a missing language, a non-positive timeout, negative retries, or a negative
|
||||||
|
retry delay. A request fails before transmission when its audio or output path
|
||||||
|
is missing.
|
||||||
|
|
||||||
|
Non-`2xx` status, transport failure, response-size overflow, invalid JSON, or
|
||||||
|
failure to install the output causes the transcription to fail. Errors include
|
||||||
|
attempt context, and the result retains attempts, final status when available,
|
||||||
|
and elapsed duration for diagnostics.
|
||||||
|
|
||||||
|
## Determinism And Concurrency
|
||||||
|
|
||||||
|
Each audio request has stable multipart field names, and successful bytes are
|
||||||
|
installed atomically. The transcribe stage may process speaker files in
|
||||||
|
parallel, bounded by the configured concurrency. It records results in stable
|
||||||
|
speaker order after all work completes; any speaker failure fails the stage.
|
||||||
|
|
||||||
|
## Related Canonical Docs
|
||||||
|
|
||||||
|
- [Configuration](../config.md#pipeline) defines the operator-selected
|
||||||
|
WhisperX URL, language, timeouts, retry policy, and concurrency.
|
||||||
|
- [Adapter implementation](../internal/adapters.md) describes internal wiring.
|
||||||
|
- [Transcribe stage](../internal/stage-transcribe.md) describes stage mechanics,
|
||||||
|
durable artifacts, and manifests.
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
# Internal Documentation Index
|
|
||||||
|
|
||||||
## Audience
|
|
||||||
Developers and coding agents changing Narratio internals.
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
`docs/internal/` documents implemented internal contracts: stage boundaries, manifest/state behavior, artifact resolution, restore behavior, storage boundaries, and workspace invariants.
|
|
||||||
|
|
||||||
User and operator behavior belongs in:
|
|
||||||
- `docs/cli.md`
|
|
||||||
- `docs/config.md`
|
|
||||||
- `docs/operations.md`
|
|
||||||
- `docs/troubleshooting.md`
|
|
||||||
|
|
||||||
## Pipeline Stage Set
|
|
||||||
Canonical stage order from `internal/stage.All()`:
|
|
||||||
1. `prepare`
|
|
||||||
2. `transcribe`
|
|
||||||
3. `merge`
|
|
||||||
4. `polish`
|
|
||||||
5. `normalize`
|
|
||||||
6. `trim`
|
|
||||||
7. `analyze`
|
|
||||||
8. `publish`
|
|
||||||
9. `notify` (placeholder)
|
|
||||||
|
|
||||||
`notify` is currently a placeholder stage with optional notifier call behavior; it has no persisted pipeline outputs.
|
|
||||||
|
|
||||||
## Internal Component Docs
|
|
||||||
- `adapters.md`: external adapter boundaries and default runtime wiring.
|
|
||||||
- `artifacts.md`: canonical source IDs, runtime catalog behavior, and resolution rules.
|
|
||||||
- `manifest.md`: session and run manifest contracts.
|
|
||||||
- `storage.md`: object-store interface and S3 implementation behavior.
|
|
||||||
- `workspace.md`: local session layout, run-local layout, and cleanup guardrails.
|
|
||||||
- `command-restore.md`: restore discovery, planning, execution, and reporting.
|
|
||||||
- `stage-prepare.md`
|
|
||||||
- `stage-transcribe.md`
|
|
||||||
- `stage-merge.md`
|
|
||||||
- `stage-polish.md`
|
|
||||||
- `stage-normalize.md`
|
|
||||||
- `stage-trim.md`
|
|
||||||
- `stage-analyze.md`
|
|
||||||
- `stage-publish.md`
|
|
||||||
@@ -1,52 +1,78 @@
|
|||||||
# Internal: Adapters
|
# Internal: Adapters
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define external integration boundaries and default adapter wiring used by app/stage orchestration.
|
|
||||||
|
Explain the adapter interfaces and production composition used by application
|
||||||
|
and stage orchestration. Externally observable protocols and formats belong in
|
||||||
|
the [integration contracts](../integrations/).
|
||||||
|
|
||||||
## Adapter Boundaries
|
## Adapter Boundaries
|
||||||
|
|
||||||
Narratio stage logic depends on adapter interfaces, not transport-specific details.
|
Narratio stage logic depends on adapter interfaces, not transport-specific details.
|
||||||
|
|
||||||
Primary adapters:
|
Primary adapters:
|
||||||
|
|
||||||
- `whisperx.Client`
|
- `whisperx.Client`
|
||||||
- `seriatim.Runner`
|
- `seriatim.Runner`
|
||||||
- `audita.Runner`
|
- `audita.Runner`
|
||||||
- `scriptorium.Runner`
|
- `scriptorium.Runner`
|
||||||
|
- `notarius.Runner`
|
||||||
- `storage.ObjectStore`
|
- `storage.ObjectStore`
|
||||||
- `notify.Sender`
|
- `notify.Sender`
|
||||||
|
|
||||||
Legacy compatibility boundary:
|
|
||||||
- `storage.Backend` remains in the storage adapter package and defaults to `NoopBackend`; current pipeline stages use `storage.ObjectStore`.
|
|
||||||
|
|
||||||
## Ownership
|
## Ownership
|
||||||
|
|
||||||
Adapters own:
|
Adapters own:
|
||||||
|
|
||||||
- HTTP/subprocess/SDK argument and transport details.
|
- HTTP/subprocess/SDK argument and transport details.
|
||||||
- Backend-specific request/response mapping.
|
- Backend-specific request/response mapping.
|
||||||
|
|
||||||
Adapters do not own:
|
Adapters do not own:
|
||||||
|
|
||||||
- stage ordering/skip/force logic;
|
- stage ordering/skip/force logic;
|
||||||
- manifest transitions;
|
- manifest transitions;
|
||||||
- canonical path policy.
|
- canonical path policy.
|
||||||
|
|
||||||
## Default Wiring
|
## Default Wiring
|
||||||
|
|
||||||
`internal/app/runner.go` initializes default adapters when not injected:
|
`internal/app/runner.go` initializes default adapters when not injected:
|
||||||
|
|
||||||
- WhisperX HTTP client from pipeline config.
|
- WhisperX HTTP client from pipeline config.
|
||||||
- Seriatim subprocess runner.
|
- Seriatim subprocess runner.
|
||||||
- Audita subprocess runner.
|
- Audita subprocess runner.
|
||||||
- Scriptorium subprocess runner.
|
- Scriptorium subprocess runner.
|
||||||
|
- Notarius subprocess runner when extraction is enabled.
|
||||||
- Noop notifier (`notify.NoopSender`).
|
- Noop notifier (`notify.NoopSender`).
|
||||||
- Object store only when required by selected stages/config.
|
- Object store only when required by selected stages/config.
|
||||||
|
|
||||||
Object-store construction goes through `newCommandObjectStore`, which loads configured filesystem secrets before adapter initialization.
|
Notarius is composed only when extraction is enabled; the extract stage owns
|
||||||
|
receipt, bundle, and configured-lane policy rather than the adapter.
|
||||||
|
|
||||||
|
Object-store construction goes through `newCommandObjectStore`, which loads
|
||||||
|
configured filesystem secrets before adapter initialization.
|
||||||
|
|
||||||
## Failure Semantics
|
## Failure Semantics
|
||||||
|
|
||||||
- Constructor errors fail stage execution setup early.
|
- Constructor errors fail stage execution setup early.
|
||||||
- Runtime adapter errors propagate to stage code and then manifest failure handling.
|
- Runtime adapter errors propagate to stage code and then manifest failure handling.
|
||||||
- Subprocess adapters persist stage logs/generated configs through stage-managed paths.
|
- Subprocess adapters persist stage logs/generated configs through stage-managed paths.
|
||||||
|
|
||||||
## Test Surfaces
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Composition: `internal/app/runner.go`, `internal/app/object_store.go`
|
||||||
|
- Shared subprocess mechanics: `internal/adapters/subprocess`
|
||||||
|
- Focused adapters: `internal/adapters/{whisperx,seriatim,audita,scriptorium,notarius,storage,notify}`
|
||||||
- `internal/adapters/whisperx/http_test.go`
|
- `internal/adapters/whisperx/http_test.go`
|
||||||
- `internal/adapters/seriatim/subprocess_test.go`
|
- `internal/adapters/seriatim/subprocess_test.go`
|
||||||
- `internal/adapters/audita/subprocess_test.go`
|
- `internal/adapters/audita/subprocess_test.go`
|
||||||
- `internal/adapters/scriptorium/subprocess_test.go`
|
- `internal/adapters/scriptorium/subprocess_test.go`
|
||||||
|
- `internal/adapters/notarius/subprocess_test.go`
|
||||||
- `internal/adapters/storage/*_test.go`
|
- `internal/adapters/storage/*_test.go`
|
||||||
- `internal/app/runner_test.go`
|
- `internal/app/runner_test.go`
|
||||||
|
|
||||||
|
See the [WhisperX](../integrations/whisperx.md),
|
||||||
|
[Seriatim](../integrations/seriatim.md), [Audita](../integrations/audita.md),
|
||||||
|
[Scriptorium](../integrations/scriptorium.md), and
|
||||||
|
[Notarius](../integrations/notarius.md) contracts before changing an
|
||||||
|
externally visible boundary. Operator-selected values belong in
|
||||||
|
[Configuration](../config.md).
|
||||||
|
|||||||
@@ -1,22 +1,38 @@
|
|||||||
# Internal: Artifacts
|
# Internal: Artifacts
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define canonical artifact IDs, runtime catalog behavior, source resolution rules, and shared current-state mechanics used by app and previous-cache code.
|
|
||||||
|
Explain the artifact registry, runtime catalog, resolver, previous-input
|
||||||
|
requirements, and shared remote current-state mechanics implemented by
|
||||||
|
`internal/artifacts`. Configuration fields that accept source IDs belong in
|
||||||
|
[Configuration](../config.md); physical placement belongs in
|
||||||
|
[Operations](../operations.md).
|
||||||
|
|
||||||
## Built-in Source IDs
|
## Built-in Source IDs
|
||||||
|
|
||||||
- `narratio.transcript.base` -> `transcripts/base.json` (`merge`)
|
The internal registry recognizes these stable built-in source IDs:
|
||||||
- `narratio.transcript.polished` -> `transcripts/polished.json` (`polish`)
|
|
||||||
- `narratio.transcript.final` -> `transcripts/final.json` (`normalize`)
|
|
||||||
- `narratio.transcript.final_trimmed` -> `transcripts/final.trimmed.json` (`trim`)
|
|
||||||
- `narratio.bounds.session` -> `artifacts/session_bounds.json` (`trim`)
|
|
||||||
|
|
||||||
## Configured and Previous-Session Sources
|
- `narratio.transcript.base`
|
||||||
|
- `narratio.transcript.polished`
|
||||||
|
- `narratio.transcript.final`
|
||||||
|
- `narratio.transcript.final_trimmed`
|
||||||
|
- `narratio.transcript.final_markdown`
|
||||||
|
- `narratio.transcript.final_trimmed_markdown`
|
||||||
|
- `narratio.bounds.session`
|
||||||
|
|
||||||
|
Registry entries bind each ID to its producer, output kind, canonical fallback,
|
||||||
|
and content validator. The focused stage documents own their input/output flow;
|
||||||
|
[Configuration](../config.md) owns where operators may select these IDs.
|
||||||
|
|
||||||
|
## Configured, Extraction, And Previous-Session Sources
|
||||||
|
|
||||||
- configured source ID format: `narratio.artifact.<artifact_key>`
|
- configured source ID format: `narratio.artifact.<artifact_key>`
|
||||||
|
- extraction source ID format: `narratio.extraction.<output_key>`
|
||||||
- previous-session source ID format: `narratio.previous_session.artifact.<artifact_key>`
|
- previous-session source ID format: `narratio.previous_session.artifact.<artifact_key>`
|
||||||
|
|
||||||
Both formats are validated by strict source-policy rules.
|
All formats are validated by strict source-policy rules. Extraction sources are
|
||||||
|
registered only from `pipeline.notarius.outputs`; the Notarius index has no
|
||||||
|
selectable source ID.
|
||||||
|
|
||||||
## Runtime Catalog
|
## Runtime Catalog
|
||||||
|
|
||||||
@@ -45,6 +61,15 @@ Configured sources (`narratio.artifact.*`):
|
|||||||
|
|
||||||
- resolve only through runtime catalog availability.
|
- resolve only through runtime catalog availability.
|
||||||
|
|
||||||
|
Extraction sources (`narratio.extraction.*`):
|
||||||
|
|
||||||
|
- use the shared registration and manifest hydration path in
|
||||||
|
`extraction_catalog.go`;
|
||||||
|
- require a current successful extract record with the exact configured source,
|
||||||
|
compatible contract and Notarius provenance, a confined regular durable
|
||||||
|
payload, and matching checksum; and
|
||||||
|
- are never inferred by scanning the Notarius bundle directory.
|
||||||
|
|
||||||
Previous-session sources (`narratio.previous_session.artifact.*`):
|
Previous-session sources (`narratio.previous_session.artifact.*`):
|
||||||
|
|
||||||
- resolve only from local `previous/` cache state;
|
- resolve only from local `previous/` cache state;
|
||||||
@@ -53,7 +78,8 @@ Previous-session sources (`narratio.previous_session.artifact.*`):
|
|||||||
|
|
||||||
Validation by content type:
|
Validation by content type:
|
||||||
|
|
||||||
- transcript built-ins: JSON with top-level `segments` array;
|
- transcript JSON built-ins: JSON with top-level `segments` array;
|
||||||
|
- transcript Markdown built-ins: non-empty text file;
|
||||||
- bounds built-in: valid JSON;
|
- bounds built-in: valid JSON;
|
||||||
- configured/previous-session artifact files: non-empty text file.
|
- configured/previous-session artifact files: non-empty text file.
|
||||||
|
|
||||||
@@ -69,7 +95,8 @@ Validation by content type:
|
|||||||
|
|
||||||
## Current-State Helpers
|
## Current-State Helpers
|
||||||
|
|
||||||
Artifacts package owns shared remote current-state loading mechanics used by restore, status/validate checks, and previous-cache planning.
|
Artifacts package owns shared remote current-state loading mechanics used by
|
||||||
|
restore, status and validation checks, and previous-cache planning.
|
||||||
|
|
||||||
Core helpers:
|
Core helpers:
|
||||||
|
|
||||||
@@ -104,9 +131,31 @@ Caller policy is intentionally outside artifacts helpers:
|
|||||||
- spool/cache paths;
|
- spool/cache paths;
|
||||||
- S3 session/run/current-state key layout.
|
- S3 session/run/current-state key layout.
|
||||||
|
|
||||||
|
See [Workspace Internals](workspace.md) for how callers consume local helpers
|
||||||
|
and [Operations](../operations.md#local-state-layout) for the authoritative
|
||||||
|
physical layout.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- source ID formats are stable contracts;
|
- source ID formats are stable contracts;
|
||||||
- artifact resolution is deterministic and manifest-aware;
|
- artifact resolution is deterministic and manifest-aware;
|
||||||
|
- extraction sources are available only from a compatible successful manifest
|
||||||
|
record;
|
||||||
- previous-session source resolution in `analyze` is local-only;
|
- previous-session source resolution in `analyze` is local-only;
|
||||||
- remote current-state key construction remains centralized in artifacts helpers.
|
- remote current-state key construction remains centralized in artifacts helpers.
|
||||||
|
|
||||||
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Registry and resolution: `internal/artifacts/artifact_resolver.go`,
|
||||||
|
`internal/artifacts/catalog.go`, `internal/artifacts/transcripts.go`,
|
||||||
|
`internal/artifacts/extraction_catalog.go`
|
||||||
|
- Current state: `internal/artifacts/current_state.go`
|
||||||
|
- Paths and keys: `internal/artifacts/paths.go`,
|
||||||
|
`internal/artifacts/s3_keys.go`
|
||||||
|
- Previous requirements: `internal/artifacts/previous_requirements.go`
|
||||||
|
- Tests: `internal/artifacts/artifact_resolver_test.go`,
|
||||||
|
`internal/artifacts/catalog_test.go`,
|
||||||
|
`internal/artifacts/extraction_catalog_test.go`,
|
||||||
|
`internal/artifacts/current_state_test.go`,
|
||||||
|
`internal/artifacts/paths_model_test.go`,
|
||||||
|
`internal/artifacts/previous_requirements_test.go`
|
||||||
|
|||||||
@@ -1,21 +1,20 @@
|
|||||||
# Internal: Command Restore
|
# Internal: Command Restore
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define the implemented `narratio session restore` command contract:
|
|
||||||
|
|
||||||
- committed remote current-state discovery;
|
Explain the implemented restore discovery, planning, installation, and
|
||||||
- deterministic restore planning;
|
reporting flow in `internal/app`. User invocation belongs in
|
||||||
- safe local install semantics;
|
[CLI](../cli.md#session-restore), and the operator recovery procedure and
|
||||||
- durable restore reporting.
|
physical restore scope belong in
|
||||||
|
[Operations](../operations.md#restore-workflow).
|
||||||
|
|
||||||
|
Restore is split into explicit phases so remote authority, local conflict
|
||||||
|
policy, and filesystem mutation can be tested independently.
|
||||||
|
|
||||||
## Discovery Contract
|
## Discovery Contract
|
||||||
|
|
||||||
Restore resolves remote committed state from the session publish current pointers:
|
Discovery delegates current-state pointer and manifest loading to
|
||||||
|
`internal/artifacts`, then validates the result against the resolved request:
|
||||||
- `current/run_id.txt` (required, non-empty);
|
|
||||||
- `current/manifest.json` (required, decodable).
|
|
||||||
|
|
||||||
Current-state discovery uses shared artifacts-level mechanics and validates identity against the resolved request config:
|
|
||||||
|
|
||||||
- campaign must match;
|
- campaign must match;
|
||||||
- session ID must match.
|
- session ID must match.
|
||||||
@@ -34,26 +33,11 @@ Planner behavior:
|
|||||||
|
|
||||||
- remote list scope is the resolved session prefix;
|
- remote list scope is the resolved session prefix;
|
||||||
- remote-to-local mapping is traversal-safe;
|
- remote-to-local mapping is traversal-safe;
|
||||||
- actions are sorted deterministically by local relative path.
|
- actions are sorted by local relative path and then remote key;
|
||||||
|
- force converts differing local targets from conflicts to downloads.
|
||||||
|
|
||||||
Restore scope from current remote state:
|
Previous-cache files are planned separately through `previouscache.BuildPlan`
|
||||||
|
when configured previous-session requirements exist.
|
||||||
- include `manifest.json`;
|
|
||||||
- include `transcripts/**`;
|
|
||||||
- include `artifacts/**`;
|
|
||||||
- include `audio/**` only with `--include-audio`.
|
|
||||||
|
|
||||||
Explicit exclusions from current remote state mapping:
|
|
||||||
|
|
||||||
- `current/**`;
|
|
||||||
- `runs/**`;
|
|
||||||
- `logs/**`;
|
|
||||||
- `reports/**`;
|
|
||||||
- `config/**`;
|
|
||||||
- `inputs/**`;
|
|
||||||
- `previous/**`.
|
|
||||||
|
|
||||||
Previous-cache files are planned separately through `previouscache.BuildPlan` when configured previous-session requirements exist.
|
|
||||||
|
|
||||||
## Execution Contract
|
## Execution Contract
|
||||||
|
|
||||||
@@ -73,8 +57,9 @@ Audio restore path:
|
|||||||
|
|
||||||
## Reporting Contract
|
## Reporting Contract
|
||||||
|
|
||||||
- `--dry-run`: prints summary only; no local writes.
|
- dry-run mode prints a summary and performs no local writes;
|
||||||
- non-dry-run: writes `reports/restore-latest.json`.
|
- execution mode persists the canonical restore report described in
|
||||||
|
[Operations](../operations.md#restore-workflow);
|
||||||
- report includes plan counts, per-action status, and execution failures.
|
- report includes plan counts, per-action status, and execution failures.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
@@ -82,3 +67,15 @@ Audio restore path:
|
|||||||
- restore uses committed remote current state as authority;
|
- restore uses committed remote current state as authority;
|
||||||
- `current/run_id.txt` is the remote publish commit marker;
|
- `current/run_id.txt` is the remote publish commit marker;
|
||||||
- restore does not execute pipeline stages.
|
- restore does not execute pipeline stages.
|
||||||
|
|
||||||
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Discovery: `internal/app/restore_discovery.go`
|
||||||
|
- Planning: `internal/app/restore_plan.go`, `internal/previouscache`
|
||||||
|
- Execution: `internal/app/restore_execute.go`
|
||||||
|
- Reporting and command coordination: `internal/app/restore_report.go`,
|
||||||
|
`internal/app/restore.go`
|
||||||
|
- Tests: `internal/app/restore_discovery_test.go`,
|
||||||
|
`internal/app/restore_plan_test.go`,
|
||||||
|
`internal/app/restore_execution_test.go`,
|
||||||
|
`internal/app/restore_workflow_test.go`
|
||||||
|
|||||||
@@ -1,13 +1,15 @@
|
|||||||
# Internal: Manifest
|
# Internal: Manifest
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define durable session state (`manifest.json`) and invocation state (`runs/{run_id}/manifest.json`) contracts.
|
|
||||||
|
Explain the session-progress and invocation-audit models implemented by
|
||||||
|
`internal/manifest`. Physical manifest placement belongs in
|
||||||
|
[Operations](../operations.md#local-state-layout).
|
||||||
|
|
||||||
## Session Manifest
|
## Session Manifest
|
||||||
Path:
|
|
||||||
- `{workspace.root}/work/{campaign}/{session_id}/manifest.json`
|
|
||||||
|
|
||||||
Primary model (`manifest.Manifest`):
|
`manifest.Manifest` records:
|
||||||
|
|
||||||
- identity (`session_id`, `campaign`, `run_id`)
|
- identity (`session_id`, `campaign`, `run_id`)
|
||||||
- local path metadata (`local_workdir`, `local_spool_dir`)
|
- local path metadata (`local_workdir`, `local_spool_dir`)
|
||||||
- remote identity metadata (`s3_bucket`, `s3_session_prefix`, `s3_run_prefix`)
|
- remote identity metadata (`s3_bucket`, `s3_session_prefix`, `s3_run_prefix`)
|
||||||
@@ -15,7 +17,8 @@ Primary model (`manifest.Manifest`):
|
|||||||
- durable `artifacts` records
|
- durable `artifacts` records
|
||||||
- per-stage `stages` map
|
- per-stage `stages` map
|
||||||
|
|
||||||
Stage status enum:
|
The model admits these stage states:
|
||||||
|
|
||||||
- `pending`
|
- `pending`
|
||||||
- `running`
|
- `running`
|
||||||
- `succeeded`
|
- `succeeded`
|
||||||
@@ -25,10 +28,9 @@ Stage status enum:
|
|||||||
- `interrupted`
|
- `interrupted`
|
||||||
|
|
||||||
## Run Manifest
|
## Run Manifest
|
||||||
Path:
|
|
||||||
- `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`
|
|
||||||
|
|
||||||
Run model (`manifest.RunManifest`):
|
`manifest.RunManifest` is created for each invocation and records:
|
||||||
|
|
||||||
- invocation identity and `force` flag
|
- invocation identity and `force` flag
|
||||||
- requested stages
|
- requested stages
|
||||||
- per-stage action (`run` or `skip`)
|
- per-stage action (`run` or `skip`)
|
||||||
@@ -36,22 +38,63 @@ Run model (`manifest.RunManifest`):
|
|||||||
- overall run status (`running`, `succeeded`, `failed`)
|
- overall run status (`running`, `succeeded`, `failed`)
|
||||||
|
|
||||||
## Persistence Semantics
|
## Persistence Semantics
|
||||||
|
|
||||||
`manifest.LocalStore`:
|
`manifest.LocalStore`:
|
||||||
|
|
||||||
- validates loaded documents;
|
- validates loaded documents;
|
||||||
- normalizes missing maps/stage records;
|
- normalizes missing maps/stage records;
|
||||||
- writes atomically via temp file + rename;
|
- writes atomically via temp file + rename;
|
||||||
- updates `updated_at` on save.
|
- updates `updated_at` on save.
|
||||||
|
|
||||||
## Execution Semantics
|
## Execution Semantics
|
||||||
Runner updates both manifests per stage transition:
|
|
||||||
- mark running
|
The application runner marks an executing stage running and then succeeded or
|
||||||
- mark succeeded/failed/skipped
|
failed in both manifests, persisting each transition. On success it records
|
||||||
- persist logs/generated config refs and metadata
|
outputs, logs, generated configuration references, and metadata. Artifact
|
||||||
|
records may include optional contract and external provenance objects; old
|
||||||
|
manifests remain compatible when those fields are absent. A successful forced
|
||||||
|
rerun marks only succeeded downstream session-stage records stale.
|
||||||
|
|
||||||
|
Starting an execution clears the current session-stage record's prior outputs,
|
||||||
|
logs, generated configuration references, and metadata. Failed and skipped
|
||||||
|
transitions enforce the same clearing rule directly, while success repopulates
|
||||||
|
only fields returned by the new result. Marking a record stale does not clear
|
||||||
|
those details because resume validation and diagnosis may still require them
|
||||||
|
before execution begins. Invocation run manifests remain immutable audit
|
||||||
|
records of their own outcomes.
|
||||||
|
|
||||||
|
A stage may explicitly return a skipped disposition and stable reason. The
|
||||||
|
runner persists that outcome in both manifests, clears older outputs for the
|
||||||
|
session-stage record along with older logs, generated configuration references,
|
||||||
|
and metadata, then applies any bounded details from the current skip and
|
||||||
|
continues. This self-skip is distinct from deciding not to execute an
|
||||||
|
already-succeeded stage and is reconsidered on later runs. Skipped results
|
||||||
|
cannot contain outputs.
|
||||||
|
|
||||||
|
When an already-succeeded stage is skipped, the invocation run manifest records
|
||||||
|
the `skip` action and reason. The session manifest deliberately retains its
|
||||||
|
existing succeeded record because it remains the cross-invocation progress
|
||||||
|
authority. Stages with a resume validator, currently extraction, may reject an
|
||||||
|
otherwise eligible skip when the recorded durable result is obsolete; the
|
||||||
|
runner marks it stale and executes it.
|
||||||
|
|
||||||
Session manifest is the authoritative stage-progress ledger across invocations.
|
Session manifest is the authoritative stage-progress ledger across invocations.
|
||||||
Run manifest is invocation-scoped audit state.
|
Run manifest is invocation-scoped audit state.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- stage resume/skip decisions are session-manifest driven.
|
- stage resume/skip decisions are session-manifest driven.
|
||||||
|
- running, failed, and self-skipped stages do not retain result payloads from
|
||||||
|
an earlier success.
|
||||||
|
- stale stages retain prior details until replacement execution starts.
|
||||||
- force reruns stale downstream succeeded stages.
|
- force reruns stale downstream succeeded stages.
|
||||||
- run manifest does not replace session manifest as progress authority.
|
- run manifest does not replace session manifest as progress authority.
|
||||||
|
|
||||||
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Models and transitions: `internal/manifest/manifest.go`,
|
||||||
|
`internal/manifest/run_manifest.go`
|
||||||
|
- Persistence and validation: `internal/manifest/store.go`
|
||||||
|
- Package tests: `internal/manifest/*_test.go`
|
||||||
|
- Assembled execution behavior: `internal/app/runner_test.go`,
|
||||||
|
`internal/app/run_stage_test.go`
|
||||||
|
|||||||
98
docs/internal/overview.md
Normal file
98
docs/internal/overview.md
Normal file
@@ -0,0 +1,98 @@
|
|||||||
|
# Internal Overview
|
||||||
|
|
||||||
|
This document is the implemented component map for Narratio. Normative system
|
||||||
|
boundaries and dependency direction belong in
|
||||||
|
[Architecture](../policy/architecture.md). User and operator contracts belong
|
||||||
|
in the [CLI](../cli.md), [Configuration](../config.md),
|
||||||
|
[Operations](../operations.md), and [Troubleshooting](../troubleshooting.md).
|
||||||
|
Externally observable tool and format contracts belong under
|
||||||
|
[Integrations](../integrations/).
|
||||||
|
|
||||||
|
## Execution Path
|
||||||
|
|
||||||
|
```text
|
||||||
|
cmd/narratio -> internal/app -> configuration and production composition
|
||||||
|
-> internal/stage -> adapters and external systems
|
||||||
|
-> manifests and artifact resolution -> durable local/remote output
|
||||||
|
```
|
||||||
|
|
||||||
|
The executable delegates process behavior to the application boundary. The
|
||||||
|
application resolves configuration, composes concrete collaborators, acquires
|
||||||
|
session safety controls, and runs commands. Pipeline commands execute the
|
||||||
|
canonical stage sequence through adapter interfaces, while manifests record
|
||||||
|
progress and artifact services resolve durable inputs and outputs.
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
| Area | Implemented owners | Responsibility |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Executable | `cmd/narratio` | Process entry, standard stream wiring, argument handoff, and exit status. |
|
||||||
|
| Application orchestration | `internal/app` | Command dispatch, configuration selection, secret-file environment loading, production composition, session locking, planning, execution, restore, cleanup gates, and user-facing reporting. |
|
||||||
|
| Configuration | `internal/config` | Strict YAML loading, discovery, defaults, normalization, session templating, and validation. |
|
||||||
|
| Pipeline stages | `internal/stage` | Canonical stage registry, shared stage contract, execution dependencies, and implemented stage behavior. |
|
||||||
|
| External boundaries | `internal/adapters`, `internal/audio` | WhisperX HTTP, downstream subprocesses, notification, object storage, and S3 audio materialization behind Narratio contracts. |
|
||||||
|
| Manifests | `internal/manifest` | Durable session progress, invocation audit state, stage transitions, validation, and atomic persistence. |
|
||||||
|
| Artifacts and paths | `internal/artifacts`, `internal/pathsafe` | Artifact identities and resolution, local and remote path/key models, current-state discovery, and confined relative destinations. |
|
||||||
|
| Previous-session cache | `internal/previouscache` | Deterministic planning and materialization requirements for configured previous-session inputs. |
|
||||||
|
| Artifact policy | `internal/artifactpolicy` | Source and destination policy, configured artifact identity validation, and publish destination safety. |
|
||||||
|
| Shared models and file operations | `internal/artifactmodel`, `internal/contracts`, `internal/fileops` | Transcript and artifact data contracts plus narrow atomic filesystem helpers. |
|
||||||
|
| Logging | `internal/logging` | Application logger construction and shared structured logging behavior. |
|
||||||
|
|
||||||
|
The application boundary composes concrete implementations. Stages depend on
|
||||||
|
Narratio-level contracts; external transport and SDK details remain in
|
||||||
|
adapters. The normative rules for these relationships remain in
|
||||||
|
[Architecture](../policy/architecture.md).
|
||||||
|
|
||||||
|
## Pipeline Stage Set
|
||||||
|
|
||||||
|
The implemented canonical order is:
|
||||||
|
|
||||||
|
1. [`prepare`](stage-prepare.md)
|
||||||
|
2. [`transcribe`](stage-transcribe.md)
|
||||||
|
3. [`merge`](stage-merge.md)
|
||||||
|
4. [`polish`](stage-polish.md)
|
||||||
|
5. [`normalize`](stage-normalize.md)
|
||||||
|
6. [`trim`](stage-trim.md)
|
||||||
|
7. [`extract`](stage-extract.md)
|
||||||
|
8. [`render`](stage-render.md)
|
||||||
|
9. [`analyze`](stage-analyze.md)
|
||||||
|
10. [`publish`](stage-publish.md)
|
||||||
|
11. `notify` (placeholder)
|
||||||
|
|
||||||
|
`notify` currently has optional notifier call behavior and no persisted pipeline
|
||||||
|
outputs; its default collaborator is a no-op sender. The focused stage
|
||||||
|
documents own implementation mechanics. The
|
||||||
|
[CLI](../cli.md) and [Operations](../operations.md) own user-visible invocation
|
||||||
|
and execution semantics.
|
||||||
|
|
||||||
|
## Focused Documentation
|
||||||
|
|
||||||
|
- [Adapter Internals](adapters.md): external adapter boundaries, composition,
|
||||||
|
failure behavior, and test surfaces.
|
||||||
|
- [Artifact Internals](artifacts.md): source identities, runtime catalog,
|
||||||
|
resolution, previous requirements, and current-state helpers.
|
||||||
|
- [Manifest Internals](manifest.md): session and run records, persistence, and
|
||||||
|
execution transitions.
|
||||||
|
- [Storage Internals](storage.md): object-store interface and S3 behavior.
|
||||||
|
- [Workspace Internals](workspace.md): local layout, locking, and cleanup
|
||||||
|
guardrails.
|
||||||
|
- [Restore Internals](command-restore.md): discovery, planning, execution, and
|
||||||
|
reporting.
|
||||||
|
- [`prepare`](stage-prepare.md)
|
||||||
|
- [`transcribe`](stage-transcribe.md)
|
||||||
|
- [`merge`](stage-merge.md)
|
||||||
|
- [`polish`](stage-polish.md)
|
||||||
|
- [`normalize`](stage-normalize.md)
|
||||||
|
- [`trim`](stage-trim.md)
|
||||||
|
- [`extract`](stage-extract.md)
|
||||||
|
- [`render`](stage-render.md)
|
||||||
|
- [`analyze`](stage-analyze.md)
|
||||||
|
- [`publish`](stage-publish.md)
|
||||||
|
|
||||||
|
Use this map to find an owner, then read its focused documentation and tests
|
||||||
|
before changing behavior.
|
||||||
|
|
||||||
|
The stage registry is implemented in `internal/stage/placeholders.go` and its
|
||||||
|
ordering is protected by `internal/app/planner_test.go`. Cross-invocation skip,
|
||||||
|
force, failure, and invalidation behavior is exercised in
|
||||||
|
`internal/app/runner_test.go` and `internal/app/run_stage_test.go`.
|
||||||
@@ -1,38 +1,60 @@
|
|||||||
# Stage: analyze
|
# Stage: analyze
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Execute selected configured Scriptorium artifacts in dependency order and materialize outputs.
|
Execute selected configured Scriptorium artifacts in dependency order and materialize outputs.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
|
|
||||||
- configured artifacts from `pipeline.scriptorium.artifacts`
|
- configured artifacts from `pipeline.scriptorium.artifacts`
|
||||||
- optional selected artifact filter (`--artifacts`)
|
- optional selected artifact keys supplied through the stage environment
|
||||||
- built-in/configured/previous-session source references in artifact inputs
|
- built-in/configured/previous-session source references in artifact inputs
|
||||||
|
|
||||||
Supported source families:
|
Supported source families:
|
||||||
- built-ins: `narratio.transcript.*`, `narratio.bounds.session`
|
- built-ins: `narratio.transcript.*`, `narratio.bounds.session`
|
||||||
|
- prepared stable inputs: `narratio.input.players`, `narratio.input.party`,
|
||||||
|
`narratio.input.glossary`
|
||||||
- configured artifacts: `narratio.artifact.<key>`
|
- configured artifacts: `narratio.artifact.<key>`
|
||||||
- previous-session cache: `narratio.previous_session.artifact.<key>`
|
- previous-session cache: `narratio.previous_session.artifact.<key>`
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
- one materialized output per executed configured artifact (`output_path`)
|
- one materialized output per executed configured artifact (`output_path`)
|
||||||
- stage metadata describing selected/generated/reused artifacts
|
- stage metadata describing selected/generated/reused artifacts
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
- skips with metadata when Scriptorium config is missing or no executable artifacts remain.
|
- skips with metadata when Scriptorium config is missing or no executable artifacts remain.
|
||||||
- builds runtime artifact catalog (built-ins + configured artifacts).
|
- builds runtime artifact catalog (built-ins + configured artifacts).
|
||||||
- marks non-executable configured artifacts as reusable when output files already exist.
|
- marks non-executable configured artifacts as reusable when output files already exist.
|
||||||
- validates selected artifact dependency order (cycle-safe topo ordering).
|
- validates selected artifact dependency order (cycle-safe topo ordering).
|
||||||
- resolves required/optional inputs per artifact source definition.
|
- resolves required/optional inputs per artifact source definition.
|
||||||
|
- resolves prepared stable input sources from `inputs/*.yml` materialized by `prepare`.
|
||||||
- resolves previous-session sources from local `previous/` cache only.
|
- resolves previous-session sources from local `previous/` cache only.
|
||||||
- runs optional render-debug, then artifact execution.
|
- runs optional render-debug, then artifact execution.
|
||||||
- validates non-empty output files and materializes canonical outputs.
|
- validates non-empty output files and materializes canonical outputs.
|
||||||
|
|
||||||
## Failure Semantics
|
## Failure Semantics
|
||||||
|
|
||||||
- required missing configured/previous-session inputs fail.
|
- required missing configured/previous-session inputs fail.
|
||||||
|
- missing required prepared stable input source includes prepare rerun guidance.
|
||||||
- missing required previous-session source includes prepare rerun guidance.
|
- missing required previous-session source includes prepare rerun guidance.
|
||||||
|
- missing required `narratio.transcript.final_markdown` or
|
||||||
|
`narratio.transcript.final_trimmed_markdown` inputs includes render rerun
|
||||||
|
guidance.
|
||||||
- dependency cycles or unavailable required dependencies fail.
|
- dependency cycles or unavailable required dependencies fail.
|
||||||
- adapter validation failures fail stage.
|
- adapter validation failures fail stage.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- `analyze` performs no remote storage calls for previous-session source resolution.
|
- `analyze` performs no remote storage calls for previous-session source resolution.
|
||||||
- output provenance and metadata are deterministic per execution.
|
- output provenance and metadata are deterministic per execution.
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Configuration](../config.md#scriptorium-artifact-entries) owns artifact
|
||||||
|
fields and source-selection rules.
|
||||||
|
- [CLI](../cli.md) owns user-visible artifact selection.
|
||||||
|
- [Scriptorium](../integrations/scriptorium.md) owns the subprocess contract.
|
||||||
|
- Implementation and tests: `internal/stage/analyze.go`,
|
||||||
|
`internal/stage/analyze_test.go`
|
||||||
|
|||||||
84
docs/internal/stage-extract.md
Normal file
84
docs/internal/stage-extract.md
Normal file
@@ -0,0 +1,84 @@
|
|||||||
|
# Internal: Extract Stage
|
||||||
|
|
||||||
|
## Responsibility
|
||||||
|
|
||||||
|
`extract` runs after `trim` and before `render`. It converts the canonical
|
||||||
|
`narratio.transcript.final_trimmed` JSON into configured Notarius lane artifacts.
|
||||||
|
An omitted or disabled Notarius section makes the stage explicitly self-skip
|
||||||
|
with reason `notarius_disabled`, no outputs, and no Notarius runner.
|
||||||
|
|
||||||
|
The external protocol is documented in the
|
||||||
|
[Notarius integration contract](../integrations/notarius.md). Configuration
|
||||||
|
fields belong in [Configuration](../config.md), and physical paths and force
|
||||||
|
procedures belong in [Operations](../operations.md).
|
||||||
|
|
||||||
|
## Lifecycle
|
||||||
|
|
||||||
|
`internal/stage/extract.go`:
|
||||||
|
|
||||||
|
1. resolves the final trimmed transcript from the shared artifact catalog;
|
||||||
|
2. resolves and fingerprints the Notarius invocation contract;
|
||||||
|
3. creates a run-local staging directory and invokes the injected
|
||||||
|
`notarius.Runner`;
|
||||||
|
4. validates the successful receipt, confined index, configured required lane
|
||||||
|
descriptors, and regular payload files;
|
||||||
|
5. atomically promotes the complete bundle to its immutable durable location;
|
||||||
|
6. records one non-selectable `notarius_index` output and one selectable
|
||||||
|
`notarius_lane` output per configured lane; and
|
||||||
|
7. registers each lane as `narratio.extraction.<output_key>` for downstream
|
||||||
|
Scriptorium and publish resolution.
|
||||||
|
|
||||||
|
Lane records retain checksum, contract, producer run ID, and Notarius system,
|
||||||
|
run, pipeline, and lane provenance. Stage metadata retains the durable bundle
|
||||||
|
root, receipt, diagnostic paths, rejection/warning summaries, producing
|
||||||
|
Narratio run ID, and invocation fingerprint. Validation completes before
|
||||||
|
promotion, so a rejected result cannot expose a partial durable bundle.
|
||||||
|
|
||||||
|
Any executed extraction outcome that replaces a different effective outcome
|
||||||
|
marks succeeded downstream stages stale. Repeating the same disabled self-skip
|
||||||
|
with no outputs is stable and does not repeatedly invalidate downstream stages.
|
||||||
|
|
||||||
|
## Resume Validation
|
||||||
|
|
||||||
|
`internal/stage/extract_resume.go` permits a skip only when the existing stage
|
||||||
|
record succeeded and still matches the current invocation fingerprint. The
|
||||||
|
fingerprint covers the resolved executable and config paths, pipeline ID,
|
||||||
|
timeout, working directory, and sorted configured output contracts.
|
||||||
|
|
||||||
|
The validator then checks the producing run identity, canonical immutable
|
||||||
|
bundle root, path confinement and absence of symlink components, receipt
|
||||||
|
identity, exactly one canonical index, the exact configured source set,
|
||||||
|
contracts and provenance, regular-file status, and stored checksums. Missing or
|
||||||
|
obsolete results are non-resumable and run again; unsafe filesystem conditions
|
||||||
|
return an error rather than silently accepting or replacing data.
|
||||||
|
|
||||||
|
The fingerprint cannot observe files imported by Notarius configuration,
|
||||||
|
profile contents, prompt/module definitions, or other transitive inputs.
|
||||||
|
Operators must force extraction after changing any such input.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Adapter startup, timeout, nonzero exit, receipt decoding, path confinement,
|
||||||
|
index compatibility, required-lane rejection, payload inspection, checksum, or
|
||||||
|
promotion errors fail the stage through ordinary manifest transition handling.
|
||||||
|
Stdout receipt and stderr diagnostics remain separate. Downstream stages are
|
||||||
|
not given selectable extraction sources unless the complete configured result
|
||||||
|
has passed validation and promotion.
|
||||||
|
|
||||||
|
When a replacement attempt begins, the current session-stage record no longer
|
||||||
|
advertises payload from the previous success. A failed replacement therefore
|
||||||
|
has no current outputs, logs, generated configuration references, or metadata,
|
||||||
|
while the earlier invocation manifest and immutable promoted bundle remain
|
||||||
|
available for audit and recovery.
|
||||||
|
|
||||||
|
## Implementation And Focused Tests
|
||||||
|
|
||||||
|
- Stage execution, selection, and resume validation: `internal/stage/extract.go`,
|
||||||
|
`internal/stage/extract_resume.go`,
|
||||||
|
`internal/stage/extract_test.go`
|
||||||
|
- Subprocess boundary: `internal/adapters/notarius/subprocess.go`,
|
||||||
|
`internal/adapters/notarius/subprocess_test.go`
|
||||||
|
- Catalog hydration: `internal/artifacts/extraction_catalog.go`,
|
||||||
|
`internal/artifacts/extraction_catalog_test.go`
|
||||||
|
- Composition and downstream behavior: `internal/app/runner_test.go`,
|
||||||
|
`internal/stage/analyze_test.go`, `internal/stage/publish_test.go`
|
||||||
@@ -1,18 +1,22 @@
|
|||||||
# Stage: merge
|
# Stage: merge
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Normalize raw transcript inputs and merge into base transcript via Seriatim.
|
Normalize raw transcript inputs and merge into base transcript via Seriatim.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
|
|
||||||
- `transcripts/raw/*.json`
|
- `transcripts/raw/*.json`
|
||||||
- `inputs/speakers.yml`
|
- `inputs/speakers.yml`
|
||||||
- `inputs/autocorrect.yml`
|
- `inputs/autocorrect.yml`
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
- `transcripts/base.json`
|
- `transcripts/base.json`
|
||||||
- optional `artifacts/seriatim.report.json`
|
- optional `artifacts/seriatim.report.json`
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
- discovers and validates raw transcript inputs.
|
- discovers and validates raw transcript inputs.
|
||||||
- normalizes each raw transcript (`seriatim.Normalize`) into run-local scratch output.
|
- normalizes each raw transcript (`seriatim.Normalize`) into run-local scratch output.
|
||||||
- merges normalized inputs (`seriatim.Run`) into base transcript.
|
- merges normalized inputs (`seriatim.Run`) into base transcript.
|
||||||
@@ -20,6 +24,14 @@ Normalize raw transcript inputs and merge into base transcript via Seriatim.
|
|||||||
- materializes canonical outputs and records stage logs/generated configs.
|
- materializes canonical outputs and records stage logs/generated configs.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- merge always consumes normalized forms of raw inputs.
|
- merge always consumes normalized forms of raw inputs.
|
||||||
- base transcript must validate before stage success.
|
- base transcript must validate before stage success.
|
||||||
- report output is config-gated.
|
- report output is config-gated.
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Seriatim](../integrations/seriatim.md) owns subprocess and output semantics.
|
||||||
|
- [Configuration](../config.md#pipeline) owns operator-selected Seriatim values.
|
||||||
|
- Implementation and tests: `internal/stage/merge.go`,
|
||||||
|
`internal/stage/merge_test.go`
|
||||||
|
|||||||
@@ -1,16 +1,20 @@
|
|||||||
# Stage: normalize
|
# Stage: normalize
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Normalize polished transcript into final transcript using Seriatim.
|
Normalize polished transcript into final transcript using Seriatim.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
|
|
||||||
- `transcripts/polished.json`
|
- `transcripts/polished.json`
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
- `transcripts/final.json` (or configured normalize output path)
|
- `transcripts/final.json` (or configured normalize output path)
|
||||||
- optional `artifacts/seriatim.normalize.report.json`
|
- optional `artifacts/seriatim.normalize.report.json`
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
- resolves polished transcript from manifest outputs/canonical fallback.
|
- resolves polished transcript from manifest outputs/canonical fallback.
|
||||||
- applies `pipeline.normalize` config or default normalize config.
|
- applies `pipeline.normalize` config or default normalize config.
|
||||||
- runs Seriatim normalize with configured timeout/binary.
|
- runs Seriatim normalize with configured timeout/binary.
|
||||||
@@ -18,5 +22,13 @@ Normalize polished transcript into final transcript using Seriatim.
|
|||||||
- materializes canonical outputs and records logs/generated configs.
|
- materializes canonical outputs and records logs/generated configs.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- final transcript must validate as processed transcript JSON (`segments` array).
|
- final transcript must validate as processed transcript JSON (`segments` array).
|
||||||
- normalize defaults are applied when `pipeline.normalize` is unset.
|
- normalize defaults are applied when `pipeline.normalize` is unset.
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Seriatim](../integrations/seriatim.md) owns subprocess and output semantics.
|
||||||
|
- [Configuration](../config.md#pipeline) owns normalize fields and defaults.
|
||||||
|
- Implementation and tests: `internal/stage/normalize.go`,
|
||||||
|
`internal/stage/normalize_test.go`
|
||||||
|
|||||||
@@ -1,17 +1,21 @@
|
|||||||
# Stage: polish
|
# Stage: polish
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Run Audita polishing on base transcript and produce polished transcript.
|
Run Audita polishing on base transcript and produce polished transcript.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
|
|
||||||
- `transcripts/base.json`
|
- `transcripts/base.json`
|
||||||
- `inputs/glossary.yml`
|
- `inputs/glossary.yml`
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
- `transcripts/polished.json`
|
- `transcripts/polished.json`
|
||||||
- optional `artifacts/audita.report.json`
|
- optional `artifacts/audita.report.json`
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
- resolves base transcript from merge outputs/canonical fallback.
|
- resolves base transcript from merge outputs/canonical fallback.
|
||||||
- invokes Audita with configured model/module/runtime options.
|
- invokes Audita with configured model/module/runtime options.
|
||||||
- validates processed transcript structure (`segments` array required).
|
- validates processed transcript structure (`segments` array required).
|
||||||
@@ -19,5 +23,14 @@ Run Audita polishing on base transcript and produce polished transcript.
|
|||||||
- materializes canonical outputs; records logs/generated config and adapter metadata.
|
- materializes canonical outputs; records logs/generated config and adapter metadata.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- polished transcript schema validation is mandatory.
|
- polished transcript schema validation is mandatory.
|
||||||
- report output is config-gated.
|
- report output is config-gated.
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Audita](../integrations/audita.md) owns subprocess, validation, and failure
|
||||||
|
semantics.
|
||||||
|
- [Configuration](../config.md#pipeline) owns operator-selected Audita values.
|
||||||
|
- Implementation and tests: `internal/stage/polish.go`,
|
||||||
|
`internal/stage/polish_test.go`
|
||||||
|
|||||||
@@ -1,29 +1,33 @@
|
|||||||
# Stage: prepare
|
# Stage: prepare
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Materialize canonical current-session inputs before processing stages.
|
Materialize canonical current-session inputs before processing stages.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
- resolved `campaign.yml`, `session.yml`, and pipeline config
|
|
||||||
- stable input files (`speakers`, `autocorrect`, `glossary`)
|
- resolved campaign, session, and pipeline configuration
|
||||||
- audio source:
|
- stable input files (`speakers`, `autocorrect`, `glossary`, `players`, `party`)
|
||||||
- local `audio_dir`/`audio_files`, or
|
- one resolved local or S3 audio source
|
||||||
- S3 `audio_s3.prefix`
|
|
||||||
- enabled configured artifact input requirements for previous-session sources
|
- enabled configured artifact input requirements for previous-session sources
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
- `inputs/campaign.yml`
|
- `inputs/campaign.yml`
|
||||||
- `inputs/session.yml`
|
- `inputs/session.yml`
|
||||||
- `inputs/pipeline.resolved.yml`
|
- `inputs/pipeline.resolved.yml`
|
||||||
- `inputs/speakers.yml`
|
- `inputs/speakers.yml`
|
||||||
- `inputs/autocorrect.yml`
|
- `inputs/autocorrect.yml`
|
||||||
- `inputs/glossary.yml`
|
- `inputs/glossary.yml`
|
||||||
|
- `inputs/players.yml`
|
||||||
|
- `inputs/party.yml`
|
||||||
- `audio/*.flac`
|
- `audio/*.flac`
|
||||||
- optional `previous/manifest.json`
|
- optional `previous/manifest.json`
|
||||||
- optional `previous/artifacts/**`
|
- optional `previous/artifacts/**`
|
||||||
- deterministic `manifest.inputs` entries (checksums + provenance)
|
- deterministic `manifest.inputs` entries (checksums + provenance)
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
- validates required config/store state.
|
- validates required config/store state.
|
||||||
- enforces local audio vs S3 audio mutual exclusivity.
|
- enforces local audio vs S3 audio mutual exclusivity.
|
||||||
- materializes S3 audio through spool/cache-aware logic.
|
- materializes S3 audio through spool/cache-aware logic.
|
||||||
@@ -37,6 +41,20 @@ Materialize canonical current-session inputs before processing stages.
|
|||||||
Required previous-session inputs fail when unavailable; optional missing inputs are skipped.
|
Required previous-session inputs fail when unavailable; optional missing inputs are skipped.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- only `prepare` hydrates canonical `previous/` cache state.
|
- only `prepare` hydrates canonical `previous/` cache state.
|
||||||
- managed previous artifacts are stored under `previous/artifacts/**` without duplicate `artifacts/artifacts/` nesting.
|
- managed previous artifacts are stored under `previous/artifacts/**` without
|
||||||
|
duplicate `artifacts/artifacts/` nesting.
|
||||||
- `manifest.inputs` ordering is deterministic (`kind`, `path`).
|
- `manifest.inputs` ordering is deterministic (`kind`, `path`).
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Configuration](../config.md) owns audio selection, stable input fields, and
|
||||||
|
previous-session settings.
|
||||||
|
- [Operations](../operations.md) owns physical input, audio, spool, cache, and
|
||||||
|
previous-state layout.
|
||||||
|
- [Storage Internals](storage.md) and [Artifact Internals](artifacts.md) explain
|
||||||
|
the internal collaborators.
|
||||||
|
- Implementation and tests: `internal/stage/prepare.go`,
|
||||||
|
`internal/stage/prepare_test.go`, `internal/audio/s3_audio_test.go`,
|
||||||
|
`internal/previouscache/*_test.go`
|
||||||
|
|||||||
@@ -1,33 +1,45 @@
|
|||||||
# Stage: publish
|
# Stage: publish
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Upload run/session outputs to object storage and atomically advance remote current state.
|
Upload run/session outputs to object storage and atomically advance remote current state.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
- successful prerequisite stages: `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`
|
|
||||||
- run root `runs/{run_id}/**`
|
- successful preceding stages from the [canonical stage set](overview.md#pipeline-stage-set)
|
||||||
- publish output rules (`pipeline.publish.outputs`)
|
- invocation-scoped run files
|
||||||
|
- resolved publish output rules
|
||||||
- effective publish locks (static + remote merged lock set)
|
- effective publish locks (static + remote merged lock set)
|
||||||
- local `previous/**` files when present
|
- durable previous-session cache files when present
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
- uploaded run files under remote `runs/{run_id}/...` (excluding `audio/**`)
|
|
||||||
- uploaded selected publish outputs under session prefix
|
- uploaded invocation record and selected publish outputs;
|
||||||
- uploaded `previous/**` files under session prefix when present
|
- uploaded durable previous-session cache files when present;
|
||||||
- uploaded `current/manifest.json`
|
- updated remote current manifest; and
|
||||||
- uploaded `current/run_id.txt` written last
|
- remote current-run commit marker, written last.
|
||||||
|
|
||||||
|
Exact remote placement and the operator workflow belong in
|
||||||
|
[Operations](../operations.md#publish-workflow).
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
- stage can self-skip when publish disabled or run upload disabled.
|
- stage can self-skip when publish disabled or run upload disabled.
|
||||||
- validates prerequisite stage success and object-store availability.
|
- validates prerequisite stage success and object-store availability.
|
||||||
- collects deterministic run file list plus run `manifest.json`.
|
- collects a deterministic run file list plus run `manifest.json`, excluding
|
||||||
|
`audio/**` and the run-local `extract/notarius-output/**` staging bundle.
|
||||||
|
- keeps run-local Notarius receipt and stderr diagnostics eligible for the run
|
||||||
|
archive.
|
||||||
- resolves publish output sources through runtime artifact catalog and manifest-aware resolution.
|
- resolves publish output sources through runtime artifact catalog and manifest-aware resolution.
|
||||||
|
- publishes extraction lanes only through explicit configured output rules;
|
||||||
|
neither run-local nor durable Notarius bundles are scanned or uploaded wholesale.
|
||||||
- selected artifact filter applies to configured artifact sources only.
|
- selected artifact filter applies to configured artifact sources only.
|
||||||
- locked outputs are skipped intentionally (including required ones).
|
- locked outputs are skipped intentionally (including required ones).
|
||||||
- optional missing outputs are skipped; required missing unlocked outputs fail.
|
- optional missing outputs are skipped; required missing unlocked outputs fail.
|
||||||
- writes remote current manifest before current run pointer.
|
- writes remote current manifest before current run pointer.
|
||||||
|
|
||||||
## Metadata Signals
|
## Metadata Signals
|
||||||
|
|
||||||
Includes counts/lists for:
|
Includes counts/lists for:
|
||||||
- run uploads
|
- run uploads
|
||||||
- published output uploads
|
- published output uploads
|
||||||
@@ -39,6 +51,24 @@ Includes counts/lists for:
|
|||||||
- `current_pointer_written`
|
- `current_pointer_written`
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- `current/run_id.txt` is the remote commit marker and is written last.
|
- `current/run_id.txt` is the remote commit marker and is written last.
|
||||||
- run upload excludes `audio/**`.
|
- run upload excludes `audio/**` and `extract/notarius-output/**`.
|
||||||
|
- `extract/notarius.receipt.json` and `extract/notarius.stderr.log` remain
|
||||||
|
eligible run-record diagnostics.
|
||||||
- publish locks are not overridden by `--force`.
|
- publish locks are not overridden by `--force`.
|
||||||
|
|
||||||
|
The commit boundary and cleanup gate are normative architecture invariants; see
|
||||||
|
[Architecture](../policy/architecture.md#publish-commit-boundary).
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Configuration](../config.md#publish-configuration-summary) owns output and
|
||||||
|
static-lock fields.
|
||||||
|
- [Operations](../operations.md#publish-locks) owns remote lock lifecycle and
|
||||||
|
physical remote state.
|
||||||
|
- [Artifact Internals](artifacts.md) explains source resolution and current-state
|
||||||
|
helpers.
|
||||||
|
- Implementation and tests: `internal/stage/publish.go`,
|
||||||
|
`internal/stage/publish_test.go`, `internal/app/operator_helpers_test.go`,
|
||||||
|
`internal/app/post_publish_cleanup_test.go`
|
||||||
|
|||||||
42
docs/internal/stage-render.md
Normal file
42
docs/internal/stage-render.md
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
# Stage: render
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Render Markdown transcript artifacts from normalized JSON transcripts via Seriatim.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
- `narratio.transcript.final` (`transcripts/final.json`)
|
||||||
|
- `narratio.transcript.final_trimmed` (`transcripts/final.trimmed.json`)
|
||||||
|
|
||||||
|
## Outputs
|
||||||
|
|
||||||
|
- `narratio.transcript.final_markdown` -> `transcripts/final.md`
|
||||||
|
- `narratio.transcript.final_trimmed_markdown` -> `transcripts/final.trimmed.md`
|
||||||
|
|
||||||
|
## Key Behavior
|
||||||
|
|
||||||
|
- uses `pipeline.render` settings (enabled/format/title/booleans).
|
||||||
|
- resolves inputs manifest-first, then canonical fallback.
|
||||||
|
- writes run-local outputs first, then materializes canonical session outputs.
|
||||||
|
- records input provenance, output paths, adapter metadata, logs, and generated config refs.
|
||||||
|
- skips with stage metadata when `pipeline.render.enabled=false`.
|
||||||
|
|
||||||
|
## Failure Semantics
|
||||||
|
|
||||||
|
- missing normalized input fails with normalize rerun guidance.
|
||||||
|
- missing trimmed input fails with trim rerun guidance.
|
||||||
|
- adapter/subprocess failure fails stage.
|
||||||
|
- empty render output files fail validation.
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- only `format: markdown` is supported.
|
||||||
|
- render stage owns production of built-in Markdown transcript sources.
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Seriatim](../integrations/seriatim.md) owns render subprocess behavior.
|
||||||
|
- [Configuration](../config.md#pipeline) owns render fields and defaults.
|
||||||
|
- Implementation and tests: `internal/stage/render.go`,
|
||||||
|
`internal/stage/render_test.go`
|
||||||
@@ -1,22 +1,36 @@
|
|||||||
# Stage: transcribe
|
# Stage: transcribe
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Generate raw per-speaker transcripts from prepared audio using WhisperX.
|
Generate raw per-speaker transcripts from prepared audio using WhisperX.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
|
|
||||||
- `audio/*.flac` from `prepare`
|
- `audio/*.flac` from `prepare`
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
- `transcripts/raw/<speaker>.json`
|
- `transcripts/raw/<speaker>.json`
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
|
|
||||||
- discovers prepared audio from manifest inputs or canonical audio directory.
|
- discovers prepared audio from manifest inputs or canonical audio directory.
|
||||||
- derives speaker ID from `.flac` basename.
|
- derives speaker ID from `.flac` basename.
|
||||||
- runs WhisperX with configured concurrency/retry settings.
|
- dispatches WhisperX requests through a bounded worker pool.
|
||||||
- validates each output as JSON.
|
- validates each output as JSON.
|
||||||
- writes run-local outputs then materializes canonical transcript outputs.
|
- writes run-local outputs then materializes canonical transcript outputs.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- speaker basenames must be unique.
|
- speaker basenames must be unique.
|
||||||
- output path returned by adapter must match requested output path.
|
- output path returned by adapter must match requested output path.
|
||||||
- each successful output is validated before stage success.
|
- each successful output is validated before stage success.
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [WhisperX](../integrations/whisperx.md) owns HTTP, retry, timeout, and
|
||||||
|
cancellation semantics.
|
||||||
|
- [Configuration](../config.md#pipeline) owns concurrency and other
|
||||||
|
operator-selected values.
|
||||||
|
- Implementation and tests: `internal/stage/transcribe.go`,
|
||||||
|
`internal/stage/transcribe_test.go`
|
||||||
|
|||||||
@@ -1,18 +1,20 @@
|
|||||||
# Stage: trim
|
# Stage: trim
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Produce a final-trimmed transcript; optionally generate bounds-driven trim.
|
|
||||||
|
Produce a final-trimmed transcript. By default, the stage generates bounds and
|
||||||
|
applies a bounds-driven trim.
|
||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
|
|
||||||
- `transcripts/final.json`
|
- `transcripts/final.json`
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
|
|
||||||
- `transcripts/final.trimmed.json` (or configured trim output path)
|
- `transcripts/final.trimmed.json` (or configured trim output path)
|
||||||
- when trim enabled: `artifacts/session_bounds.json`
|
- when trim enabled: `artifacts/session_bounds.json`
|
||||||
|
|
||||||
## Key Behavior
|
## Key Behavior
|
||||||
When `trim.enabled=false`:
|
|
||||||
- copies normalized transcript to trimmed output.
|
|
||||||
|
|
||||||
When `trim.enabled=true`:
|
When `trim.enabled=true`:
|
||||||
- runs Scriptorium bounds artifact generation;
|
- runs Scriptorium bounds artifact generation;
|
||||||
@@ -22,7 +24,20 @@ When `trim.enabled=true`:
|
|||||||
- either copies unchanged transcript or runs Seriatim trim;
|
- either copies unchanged transcript or runs Seriatim trim;
|
||||||
- validates trimmed transcript and materializes bounds output.
|
- validates trimmed transcript and materializes bounds output.
|
||||||
|
|
||||||
|
When `trim.enabled=false`:
|
||||||
|
- copies normalized transcript to trimmed output.
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- normalized transcript is required input.
|
- normalized transcript is required input.
|
||||||
- bounds output exists only in enabled trim path.
|
- bounds output exists only in enabled trim path.
|
||||||
- render-debug output is diagnostic and not a declared stage output.
|
- render-debug output is diagnostic and not a declared stage output.
|
||||||
|
|
||||||
|
## Related Contracts And Tests
|
||||||
|
|
||||||
|
- [Scriptorium](../integrations/scriptorium.md) owns bounds generation and
|
||||||
|
debug-render subprocess behavior.
|
||||||
|
- [Seriatim](../integrations/seriatim.md) owns transcript trimming behavior.
|
||||||
|
- [Configuration](../config.md#pipeline) owns trim fields and defaults.
|
||||||
|
- Implementation and tests: `internal/stage/trim.go`,
|
||||||
|
`internal/stage/trim_test.go`
|
||||||
|
|||||||
@@ -1,10 +1,16 @@
|
|||||||
# Internal: Storage
|
# Internal: Storage
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Document remote object-store contracts and S3 implementation behavior.
|
|
||||||
|
Explain the object-store interface and S3 implementation used by Narratio.
|
||||||
|
Remote key layout and lifecycle belong in [Operations](../operations.md), while
|
||||||
|
operator-selected storage fields and credential mechanisms belong in
|
||||||
|
[Configuration](../config.md).
|
||||||
|
|
||||||
## Primary Contract
|
## Primary Contract
|
||||||
|
|
||||||
`storage.ObjectStore` interface:
|
`storage.ObjectStore` interface:
|
||||||
|
|
||||||
- `List(ctx, prefix)`
|
- `List(ctx, prefix)`
|
||||||
- `Download(ctx, key, localPath)`
|
- `Download(ctx, key, localPath)`
|
||||||
- `Upload(ctx, localPath, key, opts)`
|
- `Upload(ctx, localPath, key, opts)`
|
||||||
@@ -14,24 +20,29 @@ Key invariant:
|
|||||||
- callers pass full bucket-relative keys;
|
- callers pass full bucket-relative keys;
|
||||||
- storage implementations do not infer campaign/session/run prefixes.
|
- storage implementations do not infer campaign/session/run prefixes.
|
||||||
|
|
||||||
## Configuration
|
## Composition
|
||||||
`NewObjectStoreFromConfig` currently supports S3-backed stores from `pipeline.storage.*` config.
|
|
||||||
|
|
||||||
S3 constructor behavior:
|
`NewObjectStoreFromConfig` constructs the S3-backed implementation from
|
||||||
- requires configured bucket;
|
resolved configuration. The application loads configured filesystem secrets
|
||||||
- uses region/endpoint/path-style options when set;
|
before calling it. The storage adapter consumes already-resolved values; it does
|
||||||
- resolves credentials from configured env var names (with defaults).
|
not own discovery, defaults, or configuration validation.
|
||||||
|
|
||||||
## S3 Backend Behavior
|
## S3 Backend Behavior
|
||||||
|
|
||||||
- normalizes object keys.
|
- normalizes object keys.
|
||||||
- `List` paginates and returns normalized `ObjectInfo`.
|
- `List` paginates and returns normalized `ObjectInfo`.
|
||||||
- `Download` writes local files with parent directory creation.
|
- `Download` writes local files with parent directory creation.
|
||||||
- `Upload` streams local file and returns remote metadata.
|
- `Upload` streams local file and returns remote metadata.
|
||||||
- `Exists` maps not-found responses to `false`.
|
- `Exists` maps not-found responses to `false`.
|
||||||
|
|
||||||
## Legacy Compatibility Interface
|
|
||||||
`storage.Backend` (with `ArchiveRequest`) remains as compatibility surface with `NoopBackend`; it is not used by current stage execution.
|
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- storage layer is stateless regarding manifest/stage progression.
|
- storage layer is stateless regarding manifest/stage progression.
|
||||||
- publish ordering semantics are owned by stage/app code, not storage adapters.
|
- publish ordering semantics are owned by stage/app code, not storage adapters.
|
||||||
|
|
||||||
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Contract and S3 adapter: `internal/adapters/storage`
|
||||||
|
- Composition: `internal/app/object_store.go`
|
||||||
|
- Tests: `internal/adapters/storage/*_test.go`,
|
||||||
|
`internal/app/object_store_test.go`
|
||||||
|
|||||||
@@ -1,57 +1,74 @@
|
|||||||
# Internal: Workspace
|
# Internal: Workspace
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
Define local session layout, run-local stage layout, and cleanup guardrails.
|
|
||||||
|
|
||||||
## Canonical Session Layout
|
Explain the helpers that construct local session and run paths, coordinate
|
||||||
Session root:
|
single-writer access, and confine cleanup. The authoritative physical layout and
|
||||||
- `{workspace.root}/work/{campaign}/{session_id}`
|
retention workflow belong in [Operations](../operations.md#local-state-layout).
|
||||||
|
|
||||||
Core directories/files:
|
## Path Ownership
|
||||||
- `inputs/`
|
|
||||||
- `audio/`
|
|
||||||
- `transcripts/`
|
|
||||||
- `artifacts/`
|
|
||||||
- `reports/`
|
|
||||||
- `logs/`
|
|
||||||
- `config/`
|
|
||||||
- `current/`
|
|
||||||
- `runs/`
|
|
||||||
- `previous/`
|
|
||||||
- `manifest.json`
|
|
||||||
- `.lock`
|
|
||||||
|
|
||||||
`previous/` reserved files:
|
`internal/artifacts` owns canonical session, run, spool, cache, and
|
||||||
- `previous/manifest.json`
|
previous-cache path construction. `SessionPathsFor` provides the session-scoped
|
||||||
- `previous/artifacts/**`
|
path model, and layout creation goes through `EnsureLayoutFor`. Callers should
|
||||||
|
consume those helpers instead of rebuilding relative paths.
|
||||||
|
|
||||||
|
`internal/pathsafe` and application cleanup helpers enforce confinement for
|
||||||
|
relative destinations and deletion targets.
|
||||||
|
|
||||||
## Run-Local Stage Layout
|
## Run-Local Stage Layout
|
||||||
When run context is available, stages use:
|
|
||||||
- `runs/{run_id}/{stage}/outputs/`
|
|
||||||
- `runs/{run_id}/{stage}/logs/`
|
|
||||||
- `runs/{run_id}/{stage}/reports/`
|
|
||||||
- `runs/{run_id}/{stage}/config/`
|
|
||||||
- `runs/{run_id}/{stage}/scratch/`
|
|
||||||
|
|
||||||
Run-local outputs are materialized back into canonical session paths before stage success.
|
`internal/stage/run_local.go` maps stage outputs and diagnostics into an
|
||||||
`previous/**` writes are never redirected to run-local output paths.
|
invocation-scoped layout. Successful outputs are validated and atomically
|
||||||
|
materialized into canonical session paths before stage success. Managed
|
||||||
|
previous-session cache paths remain session-durable and are never redirected
|
||||||
|
into run-local output space.
|
||||||
|
|
||||||
|
Extraction uses run-local receipt, stderr, and output-root helpers, then
|
||||||
|
promotes the validated external bundle to the unique immutable Notarius bundle
|
||||||
|
path supplied by `internal/artifacts`. `internal/fileops.PromoteDirectory`
|
||||||
|
copies only regular files and directories to a same-filesystem temporary
|
||||||
|
sibling. Source traversal uses confined directory handles and identity checks
|
||||||
|
so replacing an inspected root, directory, or file is rejected rather than
|
||||||
|
followed. The completed tree is atomically renamed without replacing an
|
||||||
|
existing destination. Exact physical paths belong in
|
||||||
|
[Operations](../operations.md#extraction-workflow).
|
||||||
|
|
||||||
## Locking
|
## Locking
|
||||||
`artifacts.LocalStore` enforces single-writer session lock via `.lock` file (`ErrLockConflict` on contention).
|
|
||||||
|
`artifacts.LocalStore` enforces the single-writer session lock via `.lock`
|
||||||
|
(`ErrLockConflict` on contention).
|
||||||
|
|
||||||
## Cleanup Semantics
|
## Cleanup Semantics
|
||||||
Automatic post-publish cleanup (`runPostArchiveCleanup`):
|
|
||||||
|
Automatic post-publish cleanup:
|
||||||
|
|
||||||
- only runs when publish actually executed and succeeded;
|
- only runs when publish actually executed and succeeded;
|
||||||
- requires `uploaded=true` and `current_pointer_written=true` metadata;
|
- requires `uploaded=true` and `current_pointer_written=true` metadata;
|
||||||
- respects `pipeline.spool.delete_audio_after_publish` and `pipeline.workspace.cleanup_after_publish`;
|
- consumes the resolved cleanup policy described in
|
||||||
|
[Configuration](../config.md);
|
||||||
- refuses unsafe deletes (root delete, out-of-root delete, symlink paths).
|
- refuses unsafe deletes (root delete, out-of-root delete, symlink paths).
|
||||||
|
|
||||||
Manual clean command:
|
Manual cleanup uses the same scoped-target checks. Invocation syntax and exact
|
||||||
- `clean <session_id>` removes session work and spool subtree.
|
deletion scope belong in [CLI](../cli.md#clean) and
|
||||||
- `clean --all` removes all workspace work and spool children.
|
[Operations](../operations.md#cleanup).
|
||||||
- durable cache is preserved unless `--clear-cache` is requested.
|
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- campaign-aware session root is mandatory.
|
- campaign-aware session root is mandatory.
|
||||||
- manifest-driven stage state is durable across runs.
|
- manifest-driven stage state is durable across runs.
|
||||||
- cleanup guardrails prevent destructive root/out-of-scope deletion.
|
- cleanup guardrails prevent destructive root/out-of-scope deletion.
|
||||||
|
|
||||||
|
## Implementation And Tests
|
||||||
|
|
||||||
|
- Path model and local store: `internal/artifacts/paths.go`,
|
||||||
|
`internal/artifacts/local.go`
|
||||||
|
- Run-local materialization: `internal/stage/run_local.go`
|
||||||
|
- Immutable bundle promotion: `internal/fileops/directory.go`
|
||||||
|
- Cleanup confinement: `internal/app/cleanup_targets.go`,
|
||||||
|
`internal/app/post_publish_cleanup.go`
|
||||||
|
- Tests: `internal/artifacts/paths_model_test.go`,
|
||||||
|
`internal/artifacts/local_test.go`, `internal/stage/run_local_test.go`,
|
||||||
|
`internal/fileops/directory_test.go`,
|
||||||
|
`internal/app/cleanup_targets_test.go`,
|
||||||
|
`internal/app/post_publish_cleanup_test.go`
|
||||||
|
|||||||
@@ -4,33 +4,6 @@ Operator workflow for running, recovering, and publishing Narratio sessions.
|
|||||||
|
|
||||||
For command syntax, see [docs/cli.md](./cli.md). For field-level config, see [docs/config.md](./config.md).
|
For command syntax, see [docs/cli.md](./cli.md). For field-level config, see [docs/config.md](./config.md).
|
||||||
|
|
||||||
## Standard Session Workflow
|
|
||||||
|
|
||||||
1. Select pipeline/campaign/session config.
|
|
||||||
2. Validate session readiness:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio session validate 2026-04-04
|
|
||||||
```
|
|
||||||
|
|
||||||
3. (Optional) inspect stage decisions:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio session plan 2026-04-04
|
|
||||||
```
|
|
||||||
|
|
||||||
4. Run the pipeline:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio run 2026-04-04
|
|
||||||
```
|
|
||||||
|
|
||||||
5. Check state:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
narratio session status 2026-04-04
|
|
||||||
```
|
|
||||||
|
|
||||||
## Campaign and Session Selection
|
## Campaign and Session Selection
|
||||||
|
|
||||||
Campaign selection priority:
|
Campaign selection priority:
|
||||||
@@ -63,7 +36,36 @@ narratio session init 2026-04-04 --remote --force
|
|||||||
|
|
||||||
If `campaign.yml` sets `session_template_file`, `session init` renders it. Template variables must resolve to concrete values.
|
If `campaign.yml` sets `session_template_file`, `session init` renders it. Template variables must resolve to concrete values.
|
||||||
|
|
||||||
## Stage Execution and Resume Behavior
|
Campaigns must provide stable input files for speakers, autocorrect, glossary, players, and party. Session files may override those paths for one session. The `prepare` stage materializes them under `inputs/`; configured Scriptorium artifacts can reference prepared `players`, `party`, and `glossary` files with `narratio.input.players`, `narratio.input.party`, and `narratio.input.glossary`.
|
||||||
|
|
||||||
|
## Standard Session Workflow
|
||||||
|
|
||||||
|
1. Select pipeline/campaign/session config.
|
||||||
|
2. Validate session readiness:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session validate 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
3. (Optional) inspect stage decisions:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session plan 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
4. Run the pipeline:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
5. Check state:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session status 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
## Stage Execution and Continuation Behavior
|
||||||
|
|
||||||
Canonical stage order:
|
Canonical stage order:
|
||||||
|
|
||||||
@@ -73,15 +75,22 @@ Canonical stage order:
|
|||||||
4. `polish`
|
4. `polish`
|
||||||
5. `normalize`
|
5. `normalize`
|
||||||
6. `trim`
|
6. `trim`
|
||||||
7. `analyze`
|
7. `extract`
|
||||||
8. `publish`
|
8. `render`
|
||||||
9. `notify`
|
9. `analyze`
|
||||||
|
10. `publish`
|
||||||
|
11. `notify`
|
||||||
|
|
||||||
Execution rules:
|
Execution rules:
|
||||||
|
|
||||||
- succeeded stages are skipped unless `--force` is set;
|
- succeeded stages are skipped unless `--force` is set;
|
||||||
- `resume` starts at first non-succeeded stage;
|
- `run` continues interrupted or partially completed sessions by running non-succeeded stages;
|
||||||
- force rerunning a succeeded upstream stage marks succeeded downstream stages as `stale`.
|
- forcing an upstream stage marks succeeded downstream stages as `stale` before
|
||||||
|
the replacement runs; and
|
||||||
|
- an executed failure, changed self-skip, or success that replaces a different
|
||||||
|
effective upstream outcome also marks succeeded downstream stages stale. A
|
||||||
|
repeated self-skip with the same reason and no outputs is stable and does not
|
||||||
|
perpetually rerun downstream work.
|
||||||
|
|
||||||
Single-stage execution:
|
Single-stage execution:
|
||||||
|
|
||||||
@@ -91,14 +100,71 @@ narratio run-stage normalize 2026-04-04 --force
|
|||||||
|
|
||||||
## Artifact Selection
|
## Artifact Selection
|
||||||
|
|
||||||
`--artifacts` can be used on `run`, `resume`, `run-stage`, `analyze`, and `publish`.
|
`--artifacts` can be used on `run`, `run-stage`, `analyze`, and `publish`.
|
||||||
|
|
||||||
Selection behavior:
|
Selection behavior:
|
||||||
|
|
||||||
- validates names against `pipeline.scriptorium.artifacts`;
|
- validates names against `pipeline.scriptorium.artifacts`;
|
||||||
- filters analyze execution to selected configured artifacts;
|
- filters analyze execution to selected configured artifacts;
|
||||||
- filters publish rules for `narratio.artifact.<name>` sources only;
|
- filters publish rules for `narratio.artifact.<name>` sources only;
|
||||||
- does not suppress built-in transcript or bounds publish sources.
|
- does not suppress built-in transcript, bounds, or explicitly configured
|
||||||
|
`narratio.extraction.<name>` publish sources; and
|
||||||
|
- never partially selects Notarius lanes.
|
||||||
|
|
||||||
|
## Extraction Workflow
|
||||||
|
|
||||||
|
When Notarius is omitted or disabled, `extract` records an explicit skipped
|
||||||
|
outcome with reason `notarius_disabled` and no outputs. A later invocation
|
||||||
|
reconsiders the skipped stage, so enabling Notarius does not require force.
|
||||||
|
|
||||||
|
When Notarius extraction is enabled, the stage consumes the final trimmed JSON
|
||||||
|
and preserves the complete validated Notarius bundle at:
|
||||||
|
|
||||||
|
- `artifacts/notarius/{narratio_run_id}/`
|
||||||
|
|
||||||
|
The directory is immutable once promoted. Configured lanes become
|
||||||
|
`narratio.extraction.<name>` sources for Scriptorium and explicit publish rules;
|
||||||
|
the bundle and `index.json` are retained for audit and resume validation but
|
||||||
|
are not selectable or published implicitly.
|
||||||
|
|
||||||
|
Starting a replacement clears the previous extraction payload from the current
|
||||||
|
session-stage record. If that replacement fails or self-skips, the current
|
||||||
|
record does not fall back to the earlier outputs. The earlier run manifest and
|
||||||
|
immutable bundle remain available for inspection, but downstream resolution
|
||||||
|
requires a new current successful extraction record.
|
||||||
|
|
||||||
|
Atomic Notarius bundle promotion is supported on Linux, macOS, and Windows.
|
||||||
|
On other operating systems, extraction fails before copying the bundle into a
|
||||||
|
temporary promotion tree because Narratio has no verified atomic no-replace
|
||||||
|
directory primitive there. This is an extraction limitation, not a broader
|
||||||
|
platform-support guarantee for every Narratio workflow.
|
||||||
|
|
||||||
|
Run-local diagnostics are:
|
||||||
|
|
||||||
|
- `runs/{run_id}/extract/notarius.receipt.json`
|
||||||
|
- `runs/{run_id}/extract/notarius.stderr.log`
|
||||||
|
- `runs/{run_id}/extract/notarius-output/` before durable promotion
|
||||||
|
|
||||||
|
The run-record upload excludes the complete
|
||||||
|
`extract/notarius-output/**` subtree. The receipt and stderr files remain
|
||||||
|
eligible run-record diagnostics. The durable bundle is never scanned for
|
||||||
|
implicit publication; only lanes named by explicit `pipeline.publish.outputs`
|
||||||
|
rules are uploaded.
|
||||||
|
|
||||||
|
To intentionally replace the current extraction result, run:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage extract 2026-04-04 --force
|
||||||
|
```
|
||||||
|
|
||||||
|
Narratio automatically reruns extraction when its recorded invocation contract
|
||||||
|
or durable output validation changes. It cannot fingerprint configuration
|
||||||
|
files, profiles, prompts, modules, or references loaded transitively by
|
||||||
|
Notarius. Force extraction after changing any of those inputs, even when the
|
||||||
|
top-level Narratio and Notarius config paths remain the same. A forced extract
|
||||||
|
marks successful downstream stages stale. Ordinary extraction failures or
|
||||||
|
outcome changes also stale affected downstream stages, while an identical
|
||||||
|
repeated `notarius_disabled` self-skip does not repeatedly invalidate them.
|
||||||
|
|
||||||
## Publish Workflow
|
## Publish Workflow
|
||||||
|
|
||||||
@@ -116,8 +182,10 @@ narratio run-stage publish 2026-04-04 --force
|
|||||||
|
|
||||||
Publish commit model:
|
Publish commit model:
|
||||||
|
|
||||||
- uploads run files under `{session_prefix}/runs/{run_id}/`;
|
- uploads eligible run files under `{session_prefix}/runs/{run_id}/`, excluding
|
||||||
- uploads configured published outputs;
|
audio and the run-local Notarius staging bundle;
|
||||||
|
- uploads configured published outputs, including only explicitly configured
|
||||||
|
extraction lanes;
|
||||||
- uploads `previous/**` cache files when present;
|
- uploads `previous/**` cache files when present;
|
||||||
- writes `current/manifest.json`;
|
- writes `current/manifest.json`;
|
||||||
- writes `current/run_id.txt` last.
|
- writes `current/run_id.txt` last.
|
||||||
@@ -195,6 +263,10 @@ Durable session paths:
|
|||||||
- `config/**`
|
- `config/**`
|
||||||
- `runs/**`
|
- `runs/**`
|
||||||
|
|
||||||
|
Validated Notarius bundles live below `artifacts/notarius/{run_id}/`; receipt,
|
||||||
|
stderr, and pre-promotion output remain in the producing run's `extract`
|
||||||
|
directory as described in [Extraction Workflow](#extraction-workflow).
|
||||||
|
|
||||||
Run-local layout:
|
Run-local layout:
|
||||||
|
|
||||||
- `runs/{run_id}/{stage}/outputs`
|
- `runs/{run_id}/{stage}/outputs`
|
||||||
@@ -244,6 +316,7 @@ Rules:
|
|||||||
## Operational Caveats
|
## Operational Caveats
|
||||||
|
|
||||||
- Local and S3 audio modes are mutually exclusive.
|
- Local and S3 audio modes are mutually exclusive.
|
||||||
- Publish requires prerequisite stages through analyze to be succeeded.
|
- Publish requires prerequisite stages through `render` and `analyze` to be succeeded.
|
||||||
|
- Markdown publish defaults require render outputs (`transcripts/final.md` and `transcripts/final.trimmed.md`).
|
||||||
- Restore requires configured object storage and committed remote current state.
|
- Restore requires configured object storage and committed remote current state.
|
||||||
- Storage-backed commands load filesystem secrets before object-store initialization.
|
- Storage-backed commands load filesystem secrets before object-store initialization.
|
||||||
|
|||||||
@@ -1,202 +1,241 @@
|
|||||||
# Narratio Architecture
|
# Architecture
|
||||||
|
|
||||||
## Purpose
|
This document defines Narratio's intended high-level architecture and the
|
||||||
|
invariants that changes must preserve. Implemented component details belong in
|
||||||
|
the [Internal Overview](../internal/overview.md) and its linked documents.
|
||||||
|
Significant architectural decision history belongs under `docs/adr/` when such
|
||||||
|
records exist.
|
||||||
|
|
||||||
`narratio` is a Go orchestration application for processing D&D session audio into polished transcripts and generated session artifacts.
|
## System Shape
|
||||||
|
|
||||||
This document defines the development principles for the project. It is inward-facing: its audience is developers and LLM coding agents. It should guide future changes, not serve as a complete implementation reference.
|
Narratio is a small Go application that turns D&D session audio into polished
|
||||||
|
transcripts and generated session artifacts. It is an explicit, stage-driven
|
||||||
|
orchestrator, not a general workflow engine.
|
||||||
|
|
||||||
Implemented component details belong under `docs/internal/`.
|
Narratio coordinates specialized external systems rather than reimplementing
|
||||||
|
their domains:
|
||||||
|
|
||||||
## Project Shape
|
- WhisperX performs transcription;
|
||||||
|
- Seriatim performs deterministic transcript processing and rendering;
|
||||||
|
- Audita performs transcript correction and polishing;
|
||||||
|
- Notarius extracts validated structured artifact bundles; and
|
||||||
|
- Scriptorium executes prompts and produces configured artifacts.
|
||||||
|
|
||||||
Narratio is a modular, stage-driven orchestrator.
|
Narratio owns orchestration, configuration resolution, session and run state,
|
||||||
|
artifact and path modeling, manifest persistence, stage sequencing, resume,
|
||||||
|
restore, cleanup gates, and publish semantics. External contracts are defined
|
||||||
|
in the [integration documentation](../integrations/).
|
||||||
|
|
||||||
It coordinates specialized downstream systems rather than reimplementing their domains:
|
The pipeline has one canonical ordered stage set. Configuration may enable,
|
||||||
|
disable, or parameterize supported behavior, but it must not turn that sequence
|
||||||
|
into an arbitrary DAG or hide orchestration in generic workflow abstractions.
|
||||||
|
The implemented stage inventory belongs in the
|
||||||
|
[Internal Overview](../internal/overview.md).
|
||||||
|
|
||||||
- WhisperX handles transcription.
|
Narratio is contract-first without being abstraction-heavy. Interfaces and
|
||||||
- Seriatim handles deterministic transcript merge/normalization/trim behavior.
|
extension points should protect demonstrated boundaries. New abstraction is not
|
||||||
- Audita handles transcript correction and polishing.
|
itself an architectural goal.
|
||||||
- Scriptorium handles prompt execution and generated artifacts.
|
|
||||||
|
|
||||||
Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and publish semantics.
|
## Ownership And Dependency Direction
|
||||||
|
|
||||||
Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine.
|
The application boundary owns command dispatch, configuration selection,
|
||||||
|
production composition, session locking, and top-level lifecycle. It may depend
|
||||||
|
on concrete implementations to assemble a run.
|
||||||
|
|
||||||
## Core Principles
|
Stage orchestration expresses intent in Narratio-level data and interfaces.
|
||||||
|
Stages may depend on configuration, manifest, artifact, path, and adapter
|
||||||
|
contracts, but they must not depend on transport-specific request types,
|
||||||
|
subprocess argument construction, cloud SDK types, or downstream tool internals.
|
||||||
|
|
||||||
### Modular and composable
|
Adapters translate between Narratio contracts and external systems. They own
|
||||||
|
HTTP, subprocess, notification, and object-storage mechanics, including command
|
||||||
|
construction, transport behavior, provider response handling, and external
|
||||||
|
error adaptation. External dependency types must remain inside the adapter that
|
||||||
|
owns them unless that dependency is the adapter's explicit public contract.
|
||||||
|
WhisperX HTTP behavior, Seriatim, Audita, Notarius, and Scriptorium command
|
||||||
|
construction, notification transport, and object-storage SDK details remain
|
||||||
|
behind these boundaries.
|
||||||
|
|
||||||
Code should be organized around clear responsibilities. Stages, adapters, config loading, manifest persistence, path construction, and storage behavior should remain separable and independently testable.
|
State and path services must not infer stage policy. Storage implementations
|
||||||
|
receive explicit bucket-relative keys and do not infer campaign, session, run,
|
||||||
|
or root-prefix semantics. Manifest persistence records transitions but does not
|
||||||
|
choose orchestration policy. Artifact resolution identifies and validates
|
||||||
|
artifacts but does not execute producers.
|
||||||
|
|
||||||
### Hexagonal boundaries
|
Dependencies should remain narrow and point toward Narratio-owned contracts.
|
||||||
|
Prefer the Go standard library. Add an external dependency only when it provides
|
||||||
|
a clear correctness, security, interoperability, or complexity benefit, and
|
||||||
|
confine it to the boundary that needs it.
|
||||||
|
|
||||||
External systems should be isolated behind narrow adapters. Stage logic should depend on Narratio-level interfaces and data structures, not on external SDK types, subprocess argument construction, or transport-specific details.
|
## Stage Boundaries
|
||||||
|
|
||||||
### Standard library preference
|
Each stage has one explicit responsibility and declares:
|
||||||
|
|
||||||
Prefer the Go standard library. Add dependencies only when they provide substantial value, are necessary for an external integration, or are a widely used de facto standard.
|
|
||||||
|
|
||||||
Accepted examples include a YAML library for configuration and the AWS SDK for S3-compatible storage.
|
|
||||||
|
|
||||||
### Explicit orchestration
|
|
||||||
|
|
||||||
The pipeline should remain stage-driven and explicit. New behavior should be added through clear stage, adapter, config, or manifest contracts rather than implicit side effects or generic workflow abstraction.
|
|
||||||
|
|
||||||
## Stage Design
|
|
||||||
|
|
||||||
Each stage should have a clear scope of responsibility.
|
|
||||||
|
|
||||||
A stage should define:
|
|
||||||
|
|
||||||
- its purpose;
|
|
||||||
- required input state;
|
- required input state;
|
||||||
- produced output state;
|
- produced output state;
|
||||||
- config fields it consumes;
|
- configuration it consumes;
|
||||||
- external adapters it uses;
|
- external adapters it uses;
|
||||||
- manifest refs it reads or writes;
|
- manifest references and metadata it reads or writes;
|
||||||
- skip, force, and resume behavior;
|
- skip, force, invalidation, and resume behavior; and
|
||||||
- failure behavior;
|
- failure behavior.
|
||||||
- tests that protect its contract.
|
|
||||||
|
|
||||||
Stages should avoid reaching across boundaries. If shared behavior is needed, prefer a helper or service with a narrow interface over duplicating ad hoc logic between stages.
|
Stages write and validate run-local results before materializing canonical
|
||||||
|
outputs where that distinction applies. A stage is complete only after its
|
||||||
|
required outputs have been written, validated, and recorded in durable manifest
|
||||||
|
state. Later stages depend on recorded success and artifact resolution, not
|
||||||
|
merely on incidental files existing on disk.
|
||||||
|
|
||||||
## Transactionality and Resume
|
A failed or interrupted stage must not be presented as successful. Failure
|
||||||
|
should preserve enough local state and diagnostics for inspection, recovery,
|
||||||
|
and resume. Forcing an upstream stage invalidates succeeded downstream work
|
||||||
|
according to the canonical stage order.
|
||||||
|
|
||||||
A stage should behave transactionally.
|
A stage may explicitly self-skip with a stable reason and no outputs. That
|
||||||
|
outcome is persisted, clears older outputs owned by the stage, and is
|
||||||
|
reconsidered on a later invocation. A stage may also validate whether an
|
||||||
|
otherwise successful recorded result is still resumable; an obsolete result
|
||||||
|
is staled and rerun, while an unsafe condition that prevents a sound decision
|
||||||
|
stops execution.
|
||||||
|
|
||||||
A stage is complete only when its outputs have been written, validated, and recorded in the manifest. If a stage fails, Narratio should preserve enough local state for inspection, recovery, and resume.
|
Shared behavior should live behind a narrow service or helper with one clear
|
||||||
|
owner. Stages must not reach across boundaries or reproduce adapter, manifest,
|
||||||
|
artifact, or path policy ad hoc.
|
||||||
|
|
||||||
A failed or incomplete run must not be treated as successful. Later stages should depend on manifest-recorded success, not merely on incidental files existing on disk.
|
## Manifest, Resume, And Restore
|
||||||
|
|
||||||
## Manifest Model
|
The session manifest is the durable ledger for progress across invocations. It
|
||||||
|
records session and run identity, stage state, input and output references,
|
||||||
|
diagnostic references, checksums or provenance where useful, and non-secret
|
||||||
|
adapter and publish metadata.
|
||||||
|
|
||||||
The manifest is the durable local ledger for a run.
|
Resume and skip decisions are manifest-driven. Filesystem state may be
|
||||||
|
inspected and validated, but file presence alone does not replace recorded
|
||||||
|
stage state. Invocation-scoped run records provide an audit of one execution;
|
||||||
|
they do not replace the session manifest as progress authority.
|
||||||
|
|
||||||
It should record:
|
Restore treats committed remote current state as its authority. It must plan
|
||||||
|
deterministically, confine remote-to-local paths, protect local conflicts, and
|
||||||
|
install the validated session manifest after other restored durable files. The
|
||||||
|
physical workflow and recovery procedures belong in
|
||||||
|
[Operations](../operations.md).
|
||||||
|
|
||||||
- run identity;
|
## Configuration
|
||||||
- stage status;
|
|
||||||
- input and output refs;
|
|
||||||
- logs and generated config refs;
|
|
||||||
- checksums or provenance where useful;
|
|
||||||
- non-secret adapter and publish metadata.
|
|
||||||
|
|
||||||
Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress.
|
Configuration is strict, explicit, centralized, and operator-oriented.
|
||||||
|
|
||||||
## Adapter Boundaries
|
- YAML decoding rejects unknown fields.
|
||||||
|
- Defaults are centralized and testable.
|
||||||
|
- Empty configured values do not silently replace meaningful defaults.
|
||||||
|
- Validation rejects invalid composition before stage execution where
|
||||||
|
practical.
|
||||||
|
- Session templating remains narrow and deterministic rather than becoming a
|
||||||
|
general configuration language.
|
||||||
|
- Secret values are supplied indirectly and are not persisted in ordinary
|
||||||
|
configuration.
|
||||||
|
|
||||||
Adapters own external integration details.
|
Narratio must not become a second configuration system for downstream tools.
|
||||||
|
External systems own their runtime defaults wherever practical; Narratio passes
|
||||||
|
the paths required by its stage contracts and explicit operator overrides. The
|
||||||
|
field-level contract and credential-supply mechanisms belong in
|
||||||
|
[Configuration](../config.md).
|
||||||
|
|
||||||
Expected boundaries:
|
## Artifacts, Paths, And Storage
|
||||||
|
|
||||||
- WhisperX HTTP details stay in the WhisperX adapter.
|
Artifact identities and local and remote paths are application contracts.
|
||||||
- Seriatim CLI construction stays in the Seriatim adapter.
|
Canonical helpers own workspace, spool, cache, session, run, input, transcript,
|
||||||
- Audita CLI construction stays in the Audita adapter.
|
artifact, log, report, configuration, and publish-current paths. Callers must
|
||||||
- Scriptorium CLI construction stays in the Scriptorium adapter.
|
not reconstruct canonical paths through scattered string concatenation.
|
||||||
- Object-storage details stay behind the storage adapter interface.
|
|
||||||
- AWS SDK types stay inside the S3 storage implementation.
|
|
||||||
|
|
||||||
Stage code should express intent in Narratio terms and call adapters through narrow contracts.
|
Artifact resolution is deterministic and manifest-aware. Producers materialize
|
||||||
|
canonical outputs before reporting success, and consumers resolve declared
|
||||||
|
artifact identities rather than infer files from unrelated directory contents.
|
||||||
|
External artifact bundles become current only through validated immutable
|
||||||
|
promotion and manifest records; directory presence alone never establishes
|
||||||
|
availability.
|
||||||
|
|
||||||
## Configuration Philosophy
|
Writes, moves, replacements, and deletions must use narrow, explicit,
|
||||||
|
root-confined destinations. Symlinks, traversal, broad roots, and ambiguous
|
||||||
|
relative destinations must not expand the scope of an operation. Cleanup is
|
||||||
|
permitted only through explicit operator action or configured post-publish
|
||||||
|
gates, and it must preserve durable cache unless cache removal is explicitly
|
||||||
|
requested.
|
||||||
|
|
||||||
Configuration should be strict, explicit, and operator-friendly.
|
Physical layout, retention, and operational lifecycle belong in
|
||||||
|
[Operations](../operations.md). Logical external formats and durable integration
|
||||||
|
contracts belong under [Integrations](../integrations/).
|
||||||
|
|
||||||
Principles:
|
## Publish Commit Boundary
|
||||||
|
|
||||||
- YAML decoding should reject unknown fields.
|
Publish has one explicit remote commit boundary. A remote run becomes current
|
||||||
- Defaults should be centralized and testable.
|
only after Narratio has successfully uploaded the run record, required published
|
||||||
- Empty configured values should not silently override meaningful defaults.
|
outputs, `current/manifest.json`, and finally `current/run_id.txt`.
|
||||||
- Session templating should remain narrow and deterministic.
|
|
||||||
- Template support should serve operator convenience, not become a general configuration language.
|
|
||||||
|
|
||||||
Narratio should not become a secondary configuration system for downstream tools. Seriatim, Audita, and Scriptorium should own their runtime defaults wherever practical. Narratio should pass required stage-contract paths and explicit operator overrides.
|
`current/run_id.txt` is the commit marker and must be written last. Failed,
|
||||||
|
incomplete, skipped, or uncommitted publish attempts must not be presented as
|
||||||
|
current remote state. Publish locks remain authoritative and are not bypassed by
|
||||||
|
a forced run.
|
||||||
|
|
||||||
## Path and Storage Discipline
|
Automatic local cleanup is permitted only after a successful publish commit,
|
||||||
|
only when explicitly configured, and only through the path-safety guardrails.
|
||||||
|
|
||||||
Local and remote paths are part of Narratio’s application contract.
|
## Security, Privacy, And Diagnostics
|
||||||
|
|
||||||
Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and publish/current paths. Stages should avoid reconstructing canonical paths through scattered string concatenation.
|
Narratio handles private campaign material. Transcripts, prompts, generated
|
||||||
|
artifacts, reports, logs, manifests, and diagnostic files are potentially
|
||||||
|
sensitive.
|
||||||
|
|
||||||
Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics.
|
Raw secrets must not be stored in pipeline, campaign, or session YAML or written
|
||||||
|
to manifests, logs, generated configuration, reports, publish metadata,
|
||||||
|
documentation, or examples. Secrets enter through configured environment
|
||||||
|
variable names or secret-file references. Diagnostics should avoid transcript
|
||||||
|
and prompt content unless a deliberate, bounded inspection mechanism requires
|
||||||
|
it.
|
||||||
|
|
||||||
## Publish Invariants
|
Logs, reports, generated invocation files, generated configuration, and render
|
||||||
|
debug files are diagnostics, not canonical pipeline products. They should be
|
||||||
|
durable and discoverable where configured, and manifest references must preserve
|
||||||
|
the distinction between diagnostics and artifacts.
|
||||||
|
|
||||||
Publish behavior must preserve a clear commit boundary.
|
Documentation security rules belong in the
|
||||||
|
[Documentation Policy](documentation.md). Credential supply belongs in
|
||||||
|
[Configuration](../config.md), while permissions, sensitive runtime-artifact
|
||||||
|
handling, and recovery belong in [Operations](../operations.md).
|
||||||
|
|
||||||
A remote run is current only after the publish stage has successfully uploaded the run record, required published outputs, `current/manifest.json`, and finally `current/run_id.txt`.
|
## Determinism And Testability
|
||||||
|
|
||||||
`current/run_id.txt` is the final remote commit marker and must be written last.
|
Narratio prefers deterministic behavior where practical, including stable local
|
||||||
|
and remote layouts, sorted operation order, predictable generated
|
||||||
|
configuration, repeatable command construction, deterministic artifact
|
||||||
|
resolution, and reproducible planning.
|
||||||
|
|
||||||
Failed, incomplete, skipped, or uncommitted publish attempts must not be presented as current remote state. Local cleanup is permitted only after successful publish commit and only when explicitly configured.
|
Run IDs and timestamps may be intentionally variable, but surrounding behavior
|
||||||
|
must remain controllable in tests. Core behavior should be testable without live
|
||||||
|
external services; expensive, nondeterministic, destructive, or external
|
||||||
|
boundaries should be replaceable with focused test doubles. General testing
|
||||||
|
philosophy and sufficiency rules belong in the [Testing Policy](testing.md).
|
||||||
|
|
||||||
## Security and Privacy
|
## Documentation And Decision Records
|
||||||
|
|
||||||
Narratio handles private campaign material.
|
Documentation follows the [Documentation Policy](documentation.md). Current
|
||||||
|
behavior belongs in its canonical user, operator, integration, architecture, or
|
||||||
|
internal owner. Proposed behavior and implementation status belong under
|
||||||
|
`docs/roadmap/`.
|
||||||
|
|
||||||
Rules:
|
Significant architectural decisions may be recorded under `docs/adr/` using the
|
||||||
|
format and lifecycle defined by the documentation policy. ADR acceptance does
|
||||||
|
not establish that a decision has been implemented.
|
||||||
|
|
||||||
- Do not store raw secrets in pipeline or session YAML.
|
## Architectural Non-Goals
|
||||||
- Use environment variable names or secret-file references for secret handling.
|
|
||||||
- Do not write raw secret values to manifests, logs, generated configs, or publish metadata.
|
|
||||||
- Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive.
|
|
||||||
- Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason.
|
|
||||||
|
|
||||||
## Diagnostics
|
Narratio does not aim to provide:
|
||||||
|
|
||||||
Diagnostics should be durable and discoverable, but distinct from canonical outputs.
|
|
||||||
|
|
||||||
Logs, reports, generated invocation/config files, and render-debug files support debugging. Transcript tiers and configured artifacts are pipeline products.
|
|
||||||
|
|
||||||
Manifest refs should preserve that distinction.
|
|
||||||
|
|
||||||
## Determinism
|
|
||||||
|
|
||||||
Where practical, Narratio should prefer deterministic behavior:
|
|
||||||
|
|
||||||
- stable local path layout;
|
|
||||||
- stable remote key layout;
|
|
||||||
- sorted upload order;
|
|
||||||
- predictable generated config files;
|
|
||||||
- repeatable command construction;
|
|
||||||
- tests that do not depend on live external services.
|
|
||||||
|
|
||||||
Run IDs and timestamps may be intentionally variable, but surrounding behavior should remain testable.
|
|
||||||
|
|
||||||
## Testing Expectations
|
|
||||||
|
|
||||||
Core behavior should be testable without live external services.
|
|
||||||
|
|
||||||
Tests should cover:
|
|
||||||
|
|
||||||
- config loading, defaults, and validation;
|
|
||||||
- CLI parsing and command construction;
|
|
||||||
- path helpers;
|
|
||||||
- manifest transitions;
|
|
||||||
- stage success, failure, skip, and resume behavior;
|
|
||||||
- adapter command construction;
|
|
||||||
- fake storage behavior;
|
|
||||||
- publish commit ordering;
|
|
||||||
- example config validity where practical.
|
|
||||||
|
|
||||||
Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs.
|
|
||||||
|
|
||||||
## Documentation Expectations
|
|
||||||
|
|
||||||
Documentation must follow `docs/documentation/policy.md`.
|
|
||||||
|
|
||||||
Current behavior belongs in user-facing docs and `docs/internal/`. Future, planned, aspirational, experimental, or unimplemented work belongs only under `docs/roadmap/`.
|
|
||||||
|
|
||||||
`docs/architecture.md` should remain concise and principle-focused. It should not duplicate the full config reference, CLI reference, operations guide, or internal stage documentation.
|
|
||||||
|
|
||||||
## Non-Goals
|
|
||||||
|
|
||||||
Narratio is not:
|
|
||||||
|
|
||||||
- a generic DAG or workflow engine;
|
- a generic DAG or workflow engine;
|
||||||
- a replacement configuration layer for Seriatim, Audita, or Scriptorium;
|
- a replacement configuration layer for WhisperX, Seriatim, Audita,
|
||||||
- a storage backend abstraction beyond the needs of this pipeline;
|
Scriptorium, or other downstream tools;
|
||||||
- a place to embed raw secrets;
|
- a storage abstraction broader than the needs of this pipeline;
|
||||||
- a place for stage logic to depend directly on AWS SDK types or downstream tool internals;
|
- stage logic coupled directly to cloud SDKs, transports, subprocess details,
|
||||||
|
or downstream implementation internals;
|
||||||
|
- raw-secret persistence;
|
||||||
|
- implicit cross-stage behavior that bypasses manifest and artifact contracts;
|
||||||
|
or
|
||||||
- a prompt-authoring system.
|
- a prompt-authoring system.
|
||||||
|
|||||||
@@ -1,94 +0,0 @@
|
|||||||
# Development Guide
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
Canonical contributor workflow and engineering conventions for implemented Narratio behavior.
|
|
||||||
|
|
||||||
## Repository layout
|
|
||||||
|
|
||||||
- `cmd/narratio/`: CLI entrypoint.
|
|
||||||
- `internal/app/`: command handlers, plan/run/resume orchestration, cleanup gates, secrets loading.
|
|
||||||
- `internal/config/`: strict YAML loading, defaults, and validation.
|
|
||||||
- `internal/stage/`: stage implementations and stage registry/order.
|
|
||||||
- `internal/adapters/`: external boundary adapters (WhisperX, Seriatim, Audita, Scriptorium, storage, notify).
|
|
||||||
- `internal/manifest/`: session/run manifest types and persistence.
|
|
||||||
- `internal/artifacts/`: canonical local/remote path helpers and local artifact store.
|
|
||||||
- `docs/`: canonical documentation set.
|
|
||||||
- `examples/`: maintained config examples used by tests.
|
|
||||||
|
|
||||||
## Build and test commands
|
|
||||||
|
|
||||||
- Run focused CLI behavior checks:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
go test ./internal/app -run TestExecute -v
|
|
||||||
```
|
|
||||||
|
|
||||||
- Run config example load/validate checks:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
go test ./internal/config -run TestExamplesLoadAndValidate -v
|
|
||||||
```
|
|
||||||
|
|
||||||
- Run full test suite:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
go test ./...
|
|
||||||
```
|
|
||||||
|
|
||||||
## Coding conventions
|
|
||||||
|
|
||||||
- Keep orchestration explicit and stage-driven; do not introduce generic workflow/DAG abstractions.
|
|
||||||
- Keep external-system details inside adapter packages; stages should consume Narratio-level contracts only.
|
|
||||||
- Use centralized path helpers from `internal/artifacts` rather than ad hoc path concatenation.
|
|
||||||
- Preserve manifest-driven state transitions (`running`, `succeeded`, `failed`, `skipped`, `stale`) as the source of run progress.
|
|
||||||
- Keep user/operator docs implementation-accurate; planned work belongs only under `docs/roadmap/`.
|
|
||||||
|
|
||||||
For design principles and invariants, see [docs/architecture.md](./architecture.md). For stage/adapter contracts, see [docs/internal/README.md](./internal/README.md).
|
|
||||||
|
|
||||||
## Dependency policy
|
|
||||||
|
|
||||||
- Prefer Go standard library where practical.
|
|
||||||
- Add third-party dependencies only when they provide clear value for required behavior.
|
|
||||||
- Keep dependency additions narrow to the boundary package that needs them.
|
|
||||||
|
|
||||||
## Change playbooks
|
|
||||||
|
|
||||||
### Add config fields
|
|
||||||
|
|
||||||
1. Add fields to config structs in `internal/config`.
|
|
||||||
2. Set defaults in `internal/config/defaults.go` when appropriate.
|
|
||||||
3. Add validation rules in `internal/config/validate.go`.
|
|
||||||
4. Add or update load/validate tests in `internal/config/*_test.go`.
|
|
||||||
5. Update canonical config docs and examples:
|
|
||||||
- [docs/config.md](./config.md)
|
|
||||||
- relevant files under `examples/`
|
|
||||||
|
|
||||||
### Add CLI flags or commands
|
|
||||||
|
|
||||||
1. Update command parsing and behavior in `internal/app`.
|
|
||||||
2. Add or update command tests (`TestExecute` and command-specific tests).
|
|
||||||
3. Update [docs/cli.md](./cli.md) and, if operator workflow changes, [docs/operations.md](./operations.md).
|
|
||||||
|
|
||||||
Remote-storage commands must obtain object storage through the app-level command object-store helper. Do not call `storage.NewObjectStoreFromConfig` directly from command handlers; the helper loads configured filesystem secrets before constructing the storage adapter.
|
|
||||||
|
|
||||||
### Add or modify stages/adapters
|
|
||||||
|
|
||||||
1. Implement stage behavior in `internal/stage` with clear input/output boundaries.
|
|
||||||
2. Keep external transport/subprocess details in `internal/adapters`.
|
|
||||||
3. Preserve manifest and publish-output semantics expected by runner and publish logic.
|
|
||||||
4. Add/update stage and adapter tests.
|
|
||||||
5. Update internal component contracts in `docs/internal/`.
|
|
||||||
|
|
||||||
### Update examples
|
|
||||||
|
|
||||||
1. Keep canonical examples only in `examples/`.
|
|
||||||
2. Ensure examples load and validate through runtime config paths.
|
|
||||||
3. Update `internal/config/load_validate_test.go` as needed.
|
|
||||||
4. Update links in `docs/config.md` if example filenames change.
|
|
||||||
|
|
||||||
### Update docs and roadmap
|
|
||||||
|
|
||||||
1. Keep implemented behavior in canonical docs (`README`, `docs/*.md`, `docs/internal/`).
|
|
||||||
2. Keep planned/unimplemented behavior only in `docs/roadmap/`.
|
|
||||||
3. After completing roadmap items, remove or mark them complete in `docs/roadmap/documentation.md`.
|
|
||||||
4. Run a link/path sweep before finalizing changes.
|
|
||||||
@@ -1,356 +1,148 @@
|
|||||||
# Go Project Documentation Policy
|
# Documentation Policy
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Project documentation must help four audiences:
|
This policy assigns each documentation topic to one canonical owner. Its goal is
|
||||||
|
to keep Narratio documentation accurate, concise, discoverable, and resistant
|
||||||
1. users who need to run the application;
|
to drift for users, operators, developers, integrators, and LLM coding agents.
|
||||||
2. administrators/operators who need to configure and operate it;
|
|
||||||
3. developers who need to understand and change it safely;
|
|
||||||
4. LLM coding agents that need clear scope, boundaries, and invariants.
|
|
||||||
|
|
||||||
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
|
|
||||||
|
|
||||||
## Core Rules
|
## Core Rules
|
||||||
|
|
||||||
### 1. Keep docs concise
|
### One Canonical Owner
|
||||||
|
|
||||||
|
Each authoritative fact belongs in one document. A non-owning document may give
|
||||||
|
a short, stable summary for orientation, but it must link to the canonical owner
|
||||||
|
instead of repeating volatile details.
|
||||||
|
|
||||||
|
Volatile details include commands, flags, configuration fields and defaults,
|
||||||
|
stage or integration keys, schemas, file names, paths, status codes, retry
|
||||||
|
behavior, and runtime guarantees. If readers could reasonably treat a statement
|
||||||
|
as a contract, maintain it only in the owning document.
|
||||||
|
|
||||||
Each document should cover a defined scope and only the essentials for that scope.
|
### Current And Future Behavior
|
||||||
|
|
||||||
|
Outside `docs/roadmap/`, documentation describes implemented behavior only.
|
||||||
|
Partial features may be described only to their implemented boundary.
|
||||||
|
|
||||||
|
ADRs are the narrow exception: an ADR may record an accepted architectural
|
||||||
|
decision before implementation, but acceptance must not be presented as proof
|
||||||
|
that the behavior exists. The roadmap owns implementation status and sequencing
|
||||||
|
until the decision is implemented. Current architecture, user, operator,
|
||||||
|
integration, and internal documentation are updated when the behavior lands.
|
||||||
|
|
||||||
|
### Audience And Detail
|
||||||
|
|
||||||
|
Write for the document's stated audience and include only the detail needed for
|
||||||
|
its owned topic. User and operator docs should not expose implementation detail.
|
||||||
|
Developer docs should link to user-facing and external contracts rather than
|
||||||
|
restate them.
|
||||||
|
|
||||||
|
### Examples
|
||||||
|
|
||||||
Avoid:
|
Complete copyable files belong in `examples/`. Documentation may use the
|
||||||
- long background explanations;
|
smallest illustrative snippet needed to explain its owned topic, but should link
|
||||||
- repeated reference material;
|
to maintained examples instead of embedding a second complete copy.
|
||||||
- implementation detail in user-facing docs;
|
|
||||||
- aspirational language outside roadmap docs;
|
Examples must be valid, secret-free, and tested where practical. Commands and
|
||||||
- verbose examples where one minimal example is clearer.
|
configuration used in documentation should match the application.
|
||||||
|
|
||||||
### 2. Document only implemented behavior outside roadmap files
|
### Security And Privacy
|
||||||
|
|
||||||
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
|
Documentation and examples must not contain real credentials, private keys,
|
||||||
|
private environment dumps, sensitive source material, or private infrastructure
|
||||||
- `docs/roadmap/`
|
details unless intentionally public. Document secret-handling mechanisms, not
|
||||||
|
secret values.
|
||||||
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
|
|
||||||
|
## Canonical Ownership
|
||||||
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
|
|
||||||
|
| Topic | Canonical owner | Owned content | Content owned elsewhere |
|
||||||
### 3. Use canonical homes
|
| --- | --- | --- | --- |
|
||||||
|
| Product orientation and minimal end-to-end quickstart | `README.md` | What Narratio is, why it is useful, one shortest successful invocation, and links onward. | Complete command reference, configuration reference, operational procedures, implementation detail. |
|
||||||
Each type of information should have one canonical location.
|
| Contributor entry point | `docs/development.md` | Task-oriented reading guide, minimal contributor orientation, baseline validation commands, and links to canonical docs. | Package inventory, architecture rules, subsystem behavior, detailed change recipes. |
|
||||||
|
| Current application architecture | `docs/policy/architecture.md` | System shape, normative ownership, dependency direction, architectural boundaries, invariants, safety properties, and non-goals. | Concrete package inventory, implementation mechanics, contributor procedures, decision history, future work. |
|
||||||
Canonical homes:
|
| Documentation organization | `docs/policy/documentation.md` | Documentation ownership, audience boundaries, maintenance rules, and ADR/document lifecycle. | Application architecture or product behavior. |
|
||||||
|
| Testing policy | `docs/policy/testing.md` | Test philosophy, risk-based sufficiency, test boundaries, doubles, coverage guidance, regression-test policy, and criteria for adding, rewriting, or deleting tests. | Subsystem behavior, application contracts, subsystem-specific test inventories, and implementation plans. |
|
||||||
- project purpose and quickstart: `README.md`
|
| CLI contract | `docs/cli.md` | Commands, arguments, flags, invocation semantics, output conventions, and exit behavior. | End-to-end operating procedures, configuration field definitions, runtime filesystem layout, stage implementation details. |
|
||||||
- development principles: `docs/architecture.md`
|
| Configuration contract | `docs/config.md` | Discovery and precedence, file schemas, fields, defaults, environment overrides, validation rules, and user-selectable stage or integration settings. | Complete example files, CLI syntax, runtime state lifecycle, implementation details. |
|
||||||
- configuration reference: `docs/config.md`
|
| Operations | `docs/operations.md` | Runtime workflows, physical filesystem and remote-state layout, output and diagnostic handling, resume, cleanup, permissions, recovery, and operational limits. | CLI flag syntax, configuration field definitions, logical artifact schemas, implementation mechanics. |
|
||||||
- CLI reference: `docs/cli.md`
|
| Troubleshooting | `docs/troubleshooting.md` | Symptom-driven diagnosis, likely causes, safe inspection steps and remedies, and links to relevant contracts. | CLI syntax, configuration definitions, operational procedures, integration contracts, implementation mechanics. |
|
||||||
- operations and recovery: `docs/operations.md`
|
| Public HTTP contract, if introduced | `docs/api.md` | Routes, authentication, media types, request and response schemas, status codes, pagination, caching, idempotency, rate limits, and HTTP retry semantics. | Client walkthroughs, upstream or downstream integration internals, implementation detail. |
|
||||||
- troubleshooting: `docs/troubleshooting.md`
|
| Consumer guidance, if a public package or API is introduced | `docs/consumers/` | Task-oriented use of the public interface, minimal client examples, and consumer responsibilities. | HTTP wire semantics, external protocol contracts, internal implementation detail. |
|
||||||
- implemented internals: `docs/internal/`
|
| External and durable integration contracts | `docs/integrations/` | External file formats and protocols, upstream and downstream contracts, logical artifact paths and schemas, media types, and compatibility behavior. | Physical runtime placement and lifecycle, internal transformations, CLI syntax, configuration defaults. |
|
||||||
- future work: `docs/roadmap/`
|
| Implemented component inventory | `docs/internal/overview.md` | Current packages and components, their implemented responsibilities, and links to focused internal docs. | Normative architecture, contributor reading policy, external contracts. |
|
||||||
- contributor workflow: `docs/development.md`
|
| Internal component behavior | Other files under `docs/internal/` | Implementation flow, internal collaborators and state transitions, package-local guarantees and failures, and relevant tests. | Global architecture invariants, configuration definitions and defaults, external schemas, operator procedures. |
|
||||||
- copyable examples: `examples/`
|
| Architectural decision history | `docs/adr/` | Significant decisions, context, alternatives, rationale, consequences, and supersession history. | Current behavior reference, implementation status, task sequencing. |
|
||||||
|
| Future work and implementation status | `docs/roadmap/` | Proposed, accepted, deferred, or rejected work; implementation status; sequencing; and task breakdowns. | Implemented behavior reference and architectural decision rationale. |
|
||||||
Other files should summarize briefly and link to the canonical source.
|
| Complete copyable artifacts | `examples/` | Maintained configuration, inputs, and other files intended to be copied or run. | Field-by-field reference, command reference, prose explanation. |
|
||||||
|
|
||||||
### 4. Keep examples real
|
Documents that do not exist are required only when the corresponding interface
|
||||||
|
or responsibility exists. Do not create placeholder API, consumer, integration,
|
||||||
Examples should be valid, maintained, and free of secrets.
|
or operations documents for behavior the application does not have.
|
||||||
|
|
||||||
Where practical:
|
## Boundary Rules
|
||||||
- example configs should load successfully;
|
|
||||||
- example commands should match real CLI syntax;
|
### Orientation
|
||||||
- important examples should be covered by tests.
|
|
||||||
|
The README owns product orientation. The developer guide routes contributors.
|
||||||
## Documentation Profiles
|
Architecture owns normative structure. Internal overview owns the current
|
||||||
|
concrete component map. These documents may link to one another but should not
|
||||||
All projects require:
|
maintain parallel package or behavior descriptions.
|
||||||
|
|
||||||
- `README.md`
|
### Commands, Configuration, Operations, And Troubleshooting
|
||||||
- `docs/architecture.md`
|
|
||||||
|
CLI documentation answers how to invoke the application. Configuration
|
||||||
Additional docs depend on the project.
|
documentation answers what settings mean. Operations answers what happens to
|
||||||
|
runtime state and how to operate or recover the application. Troubleshooting
|
||||||
### Small library
|
starts from observable symptoms and links readers to the owning command,
|
||||||
|
configuration, operational, or integration contract. When a workflow crosses
|
||||||
Recommended:
|
these topics, choose the document that owns the task and link to the other
|
||||||
- `docs/development.md`, if contributor conventions are non-obvious
|
contracts.
|
||||||
|
|
||||||
### Simple CLI
|
### Contracts And Implementation
|
||||||
|
|
||||||
Required:
|
Integration and API documents define externally observable shapes and
|
||||||
- `docs/cli.md`
|
semantics. Internal documents explain how Narratio implements or consumes those
|
||||||
|
contracts. Internal docs may name a field, file, or protocol to identify a
|
||||||
Recommended:
|
dependency, but must link to its canonical contract for the definition.
|
||||||
- `docs/development.md`
|
|
||||||
|
### Security Topics
|
||||||
### Config-driven CLI
|
|
||||||
|
This policy owns what documentation and examples may contain. Architecture owns
|
||||||
Required:
|
application security invariants. Configuration owns credential-supply
|
||||||
- `docs/cli.md`
|
mechanisms. Operations owns permissions and handling of sensitive runtime
|
||||||
- `docs/config.md`
|
artifacts. Troubleshooting owns safe diagnostic and remediation guidance.
|
||||||
|
Internal docs own implementation mechanisms only.
|
||||||
Recommended:
|
|
||||||
- `examples/`
|
## Architecture Decision Records
|
||||||
- `docs/development.md`
|
|
||||||
|
Use sequentially numbered ADR filenames such as
|
||||||
### Stateful or operator-facing application
|
`0001-record-architecture-decisions.md`. Follow the lightweight Nygard format:
|
||||||
|
|
||||||
Required:
|
1. title;
|
||||||
- `docs/cli.md`, if CLI-based
|
2. status;
|
||||||
- `docs/config.md`, if config-driven
|
3. date;
|
||||||
- `docs/operations.md`
|
4. context;
|
||||||
|
5. decision;
|
||||||
Recommended:
|
6. alternatives considered;
|
||||||
- `docs/troubleshooting.md`
|
7. consequences.
|
||||||
- `examples/`
|
|
||||||
- `docs/development.md`
|
Treat the decision content of an accepted ADR as immutable. When a decision
|
||||||
|
changes, create a new ADR and update the earlier ADR's status to superseded.
|
||||||
### Modular, staged, service-oriented, or orchestration application
|
Rejected architectural alternatives belong in the ADR; rejected product ideas
|
||||||
|
belong in the roadmap.
|
||||||
Required:
|
|
||||||
- `docs/cli.md`, if CLI-based
|
## Maintenance
|
||||||
- `docs/config.md`, if config-driven
|
|
||||||
- `docs/operations.md`
|
When behavior changes, update its canonical owner in the same change. If
|
||||||
- `docs/internal/`
|
ownership moves, remove the old definition and replace it with a link where
|
||||||
- `docs/development.md`
|
navigation remains useful.
|
||||||
|
|
||||||
Recommended:
|
Before completing documentation work:
|
||||||
- `docs/troubleshooting.md`
|
|
||||||
- validated examples under `examples/`
|
- verify affected behavior and examples;
|
||||||
|
- check commands, flags, fields, defaults, schemas, and paths against their
|
||||||
## Required Documents
|
implementation;
|
||||||
|
- keep unimplemented behavior in the roadmap, subject to the ADR exception;
|
||||||
### README.md
|
- remove stale references and validate links;
|
||||||
|
- confirm that non-owning documents summarize and link rather than redefine;
|
||||||
**Audience:** users, administrators, operators
|
- confirm that no secrets or sensitive private data were added.
|
||||||
|
|
||||||
The README is the outward-facing project orientation page.
|
|
||||||
|
|
||||||
It should include, in order:
|
|
||||||
|
|
||||||
1. concise description;
|
|
||||||
2. elevator pitch;
|
|
||||||
3. shortest useful command or usage example;
|
|
||||||
4. links to targeted docs.
|
|
||||||
|
|
||||||
The README should be short. It is not a manual.
|
|
||||||
|
|
||||||
The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.)
|
|
||||||
|
|
||||||
### docs/architecture.md
|
|
||||||
|
|
||||||
**Audience:** developers, LLM coding agents
|
|
||||||
|
|
||||||
`docs/architecture.md` is required for every project.
|
|
||||||
|
|
||||||
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
|
|
||||||
|
|
||||||
It should include:
|
|
||||||
|
|
||||||
- project shape;
|
|
||||||
- core design principles;
|
|
||||||
- package and boundary philosophy;
|
|
||||||
- state/persistence philosophy, if applicable;
|
|
||||||
- external integration philosophy, if applicable;
|
|
||||||
- error-handling and logging principles;
|
|
||||||
- testing expectations;
|
|
||||||
- documentation expectations;
|
|
||||||
- architectural invariants;
|
|
||||||
- explicit non-goals, if useful.
|
|
||||||
|
|
||||||
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
|
|
||||||
|
|
||||||
### docs/config.md
|
|
||||||
|
|
||||||
**Audience:** administrators, operators, advanced users
|
|
||||||
|
|
||||||
Required for applications with configuration files.
|
|
||||||
|
|
||||||
It should include, in order:
|
|
||||||
|
|
||||||
1. config file locations and discovery precedence;
|
|
||||||
2. minimal working config;
|
|
||||||
3. production-oriented config;
|
|
||||||
4. full configuration reference;
|
|
||||||
5. secrets handling, if applicable;
|
|
||||||
6. links to maintained examples.
|
|
||||||
|
|
||||||
The full configuration reference should be canonical.
|
|
||||||
|
|
||||||
### docs/cli.md
|
|
||||||
|
|
||||||
**Audience:** users, administrators, operators
|
|
||||||
|
|
||||||
Required for CLI applications.
|
|
||||||
|
|
||||||
It should include, in order:
|
|
||||||
|
|
||||||
1. shortest useful command;
|
|
||||||
2. command overview;
|
|
||||||
3. complete flag reference;
|
|
||||||
4. common workflows;
|
|
||||||
5. diagnostic or recovery commands, if applicable.
|
|
||||||
|
|
||||||
Explain when commands are useful, not just their syntax.
|
|
||||||
|
|
||||||
### docs/operations.md
|
|
||||||
|
|
||||||
**Audience:** administrators, operators
|
|
||||||
|
|
||||||
Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures.
|
|
||||||
|
|
||||||
It should cover:
|
|
||||||
|
|
||||||
- normal workflow;
|
|
||||||
- filesystem layout;
|
|
||||||
- remote storage layout, if applicable;
|
|
||||||
- logs and manifests;
|
|
||||||
- resume/retry behavior;
|
|
||||||
- cleanup behavior;
|
|
||||||
- archive/backup behavior;
|
|
||||||
- safe recovery procedures;
|
|
||||||
- operational caveats.
|
|
||||||
|
|
||||||
### docs/troubleshooting.md
|
|
||||||
|
|
||||||
**Audience:** administrators, operators
|
|
||||||
|
|
||||||
Recommended once recurring failure modes exist.
|
|
||||||
|
|
||||||
Each entry should include:
|
|
||||||
|
|
||||||
- symptom;
|
|
||||||
- likely cause;
|
|
||||||
- diagnostic command or inspection step;
|
|
||||||
- safe fix;
|
|
||||||
- relevant links.
|
|
||||||
|
|
||||||
### docs/development.md
|
|
||||||
|
|
||||||
**Audience:** developers, LLM coding agents
|
|
||||||
|
|
||||||
Required for projects maintained by humans and LLM coding agents.
|
|
||||||
|
|
||||||
It should include:
|
|
||||||
|
|
||||||
- repository layout;
|
|
||||||
- build/test commands;
|
|
||||||
- coding conventions;
|
|
||||||
- dependency policy;
|
|
||||||
- how to add config fields;
|
|
||||||
- how to add CLI flags;
|
|
||||||
- how to add stages/modules/adapters, if applicable;
|
|
||||||
- how to update examples;
|
|
||||||
- documentation update expectations.
|
|
||||||
|
|
||||||
### docs/internal/
|
|
||||||
|
|
||||||
**Audience:** developers, LLM coding agents
|
|
||||||
|
|
||||||
Required for modular, staged, service-oriented, or orchestration projects.
|
|
||||||
|
|
||||||
This directory describes implemented internal components. It is not the roadmap.
|
|
||||||
|
|
||||||
Use one file per major component where useful.
|
|
||||||
|
|
||||||
Each component doc should include:
|
|
||||||
|
|
||||||
1. purpose;
|
|
||||||
2. inputs and outputs;
|
|
||||||
3. boundaries;
|
|
||||||
4. config fields used;
|
|
||||||
5. external adapters used;
|
|
||||||
6. state or manifest behavior, if applicable;
|
|
||||||
7. skip/resume behavior, if applicable;
|
|
||||||
8. failure behavior;
|
|
||||||
9. tests to inspect before changing;
|
|
||||||
10. architectural invariants.
|
|
||||||
|
|
||||||
### docs/roadmap/
|
|
||||||
|
|
||||||
**Audience:** maintainers, developers, LLM coding agents
|
|
||||||
|
|
||||||
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
|
|
||||||
|
|
||||||
Roadmap docs should clearly distinguish:
|
|
||||||
|
|
||||||
- proposed work;
|
|
||||||
- accepted plans;
|
|
||||||
- deferred ideas;
|
|
||||||
- rejected ideas;
|
|
||||||
- implementation prompts or task breakdowns, if useful.
|
|
||||||
|
|
||||||
Roadmap docs should not be confused with current behavior.
|
|
||||||
|
|
||||||
### docs/integrations/
|
|
||||||
|
|
||||||
**Audience:** developers, LLM coding agents
|
|
||||||
|
|
||||||
Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
|
|
||||||
|
|
||||||
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses.
|
|
||||||
|
|
||||||
Use one file per integration where useful.
|
|
||||||
|
|
||||||
## Examples Directory
|
|
||||||
|
|
||||||
Projects with non-trivial configuration or workflows should include `examples/`.
|
|
||||||
|
|
||||||
Useful examples include:
|
|
||||||
|
|
||||||
- minimal working config;
|
|
||||||
- production-oriented config;
|
|
||||||
- full annotated config;
|
|
||||||
- local development config;
|
|
||||||
- remote/object-storage config;
|
|
||||||
- minimal session/input file.
|
|
||||||
|
|
||||||
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
|
|
||||||
|
|
||||||
## Security and Privacy
|
|
||||||
|
|
||||||
Docs and examples must not include:
|
|
||||||
|
|
||||||
- real API keys;
|
|
||||||
- tokens;
|
|
||||||
- passwords;
|
|
||||||
- private keys;
|
|
||||||
- private environment dumps;
|
|
||||||
- sensitive user data;
|
|
||||||
- raw private transcripts;
|
|
||||||
- private infrastructure details unless intentionally public.
|
|
||||||
|
|
||||||
Document secret-handling mechanisms, not actual secret values.
|
|
||||||
|
|
||||||
## Maintenance Rules
|
|
||||||
|
|
||||||
When docs change, verify the affected behavior.
|
|
||||||
|
|
||||||
Where practical:
|
|
||||||
|
|
||||||
- load example config files in tests;
|
|
||||||
- test CLI examples or command parser behavior;
|
|
||||||
- validate documented flags against real flags;
|
|
||||||
- remove stale references;
|
|
||||||
- update links after renames;
|
|
||||||
- keep roadmap content out of non-roadmap docs.
|
|
||||||
|
|
||||||
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
|
|
||||||
|
|
||||||
Documentation is complete only when it matches the current code.
|
|
||||||
|
|
||||||
## Documentation Change Checklist
|
|
||||||
|
|
||||||
Before merging documentation changes, verify:
|
|
||||||
|
|
||||||
- README is concise and orientation-focused.
|
|
||||||
- `docs/architecture.md` describes development principles.
|
|
||||||
- Future work appears only under `docs/roadmap/`.
|
|
||||||
- User-facing docs avoid unnecessary internals.
|
|
||||||
- Developer-facing docs preserve boundaries and invariants.
|
|
||||||
- Config examples match the schema.
|
|
||||||
- CLI examples match real commands and flags.
|
|
||||||
- Defaults appear in the canonical config reference.
|
|
||||||
- No secrets or private data are included.
|
|
||||||
- Links are accurate.
|
|
||||||
|
|||||||
296
docs/policy/testing.md
Normal file
296
docs/policy/testing.md
Normal file
@@ -0,0 +1,296 @@
|
|||||||
|
# Testing Policy
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Our tests exist to make **incorrect changes expensive and correct changes cheap**.
|
||||||
|
|
||||||
|
We do not optimize for test count, line coverage, exhaustive isolation, or the fewest possible tests. We optimize for sufficient confidence in important behavior while imposing as little unnecessary friction as possible on future development.
|
||||||
|
|
||||||
|
## Every test has a cost
|
||||||
|
|
||||||
|
Testing is not an unqualified good. Every test imposes both an immediate cost and a continuing lifetime cost.
|
||||||
|
|
||||||
|
A test must be:
|
||||||
|
|
||||||
|
- written and reviewed;
|
||||||
|
- understood by future maintainers and coding agents;
|
||||||
|
- executed in local and CI workflows;
|
||||||
|
- diagnosed when it fails;
|
||||||
|
- updated when legitimate behavior changes;
|
||||||
|
- maintained as fixtures, APIs, and dependencies evolve; and
|
||||||
|
- removed or rewritten when it becomes redundant, brittle, misleading, or obsolete.
|
||||||
|
|
||||||
|
Tests also create cognitive and architectural friction. They can constrain refactoring, duplicate policy, slow feedback loops, add noise to failures, and cause harmless implementation changes to require unrelated edits across the suite.
|
||||||
|
|
||||||
|
A test is warranted only when the confidence it provides justifies these costs.
|
||||||
|
|
||||||
|
Apply this cost-benefit analysis at two levels:
|
||||||
|
|
||||||
|
1. **Per test:** What realistic defect does this test detect, how consequential would that defect be, and is that protection worth the test's lifetime cost?
|
||||||
|
2. **Across the suite:** Does this collection provide materially more confidence than a smaller, simpler suite would?
|
||||||
|
|
||||||
|
The preferred test suite is a **lean suite that provides sufficient confidence in the risks that matter, without redundant or low-value tests**. We seek sufficient confidence with the least unnecessary testing friction, not the fewest possible tests.
|
||||||
|
|
||||||
|
Some friction is intentional. Tests should make dangerous changes—such as breaking compatibility, corrupting data, violating security boundaries, or reintroducing subtle bugs—require deliberate review. They should not make ordinary internal changes needlessly expensive.
|
||||||
|
|
||||||
|
The cost of a test is not a reason to omit testing by default. Do not cite maintenance cost abstractly. When omitting a plausible test, be able to state why the protected failure is low-risk, already covered, obvious, reversible, or cheaper to detect elsewhere. For consequential, subtle, or difficult-to-observe behavior, the presumption should favor testing.
|
||||||
|
|
||||||
|
## Default testing style
|
||||||
|
|
||||||
|
Use a **classical/Detroit-style** approach:
|
||||||
|
|
||||||
|
- Test observable behavior, resulting state, contracts, and invariants.
|
||||||
|
- Use real internal collaborators when they are fast and deterministic.
|
||||||
|
- Use fakes, stubs, or mocks primarily at expensive, nondeterministic, destructive, or external boundaries.
|
||||||
|
- Prefer package-level behavioral tests over tests coupled to private helpers or internal call sequences.
|
||||||
|
- Treat exact collaborator interactions as testable behavior only when the interaction itself is a requirement.
|
||||||
|
|
||||||
|
Examples of appropriate seams include clocks, randomness, subprocesses, remote APIs, object storage, email, and paid LLM calls.
|
||||||
|
|
||||||
|
## Test execution requirements
|
||||||
|
|
||||||
|
Tests in the default suite must be deterministic, offline, and independent of real credentials. They must not invoke paid APIs or depend on mutable external services. Tests that require live infrastructure must be explicitly opt-in and clearly separated from the default suite.
|
||||||
|
|
||||||
|
Control clocks, randomness, environment variables, and other process-global or machine-specific state when they affect behavior. Tests should be safe to run repeatedly and alongside other tests without depending on execution order or state left by an earlier test.
|
||||||
|
|
||||||
|
## What deserves tests
|
||||||
|
|
||||||
|
Prioritize tests for:
|
||||||
|
|
||||||
|
1. Public and package-level contracts.
|
||||||
|
2. Domain rules and important invariants.
|
||||||
|
3. Boundary conditions and malformed input.
|
||||||
|
4. Failure handling, cancellation, retries, recovery, and partial success.
|
||||||
|
5. Serialization, schemas, compatibility, and round trips.
|
||||||
|
6. Previously observed or plausible regressions.
|
||||||
|
7. Representative integration and end-to-end workflows.
|
||||||
|
|
||||||
|
A package-level contract is behavior relied upon by another package or major collaborator, not every observable detail of a package implementation.
|
||||||
|
|
||||||
|
For behavior involving **data integrity, destructive operations, compatibility, security, concurrency, idempotency, or recovery**, presume that durable tests are required unless the behavior is already credibly protected at another layer.
|
||||||
|
|
||||||
|
Do not add tests merely because a function, branch, or line exists. Do not add a test when the same meaningful risk is already adequately protected elsewhere.
|
||||||
|
|
||||||
|
## Choose the right test boundary
|
||||||
|
|
||||||
|
Test through the narrowest stable boundary that expresses the behavior clearly.
|
||||||
|
|
||||||
|
This is often the package API, but it may instead be:
|
||||||
|
|
||||||
|
- a smaller pure function when dense domain logic is most clearly isolated there;
|
||||||
|
- a package-level operation when several internal collaborators jointly produce the behavior; or
|
||||||
|
- a larger integration boundary when correctness emerges from interaction with a real dependency.
|
||||||
|
|
||||||
|
Do not force all behavior through oversized end-to-end tests. Do not test every private helper merely because it exists. Choose the boundary that gives durable confidence with the least incidental coupling.
|
||||||
|
|
||||||
|
## Test behavior, not implementation
|
||||||
|
|
||||||
|
A test should protect a decision, contract, or invariant—not memorialize the current implementation.
|
||||||
|
|
||||||
|
Before adding or retaining a test, ask:
|
||||||
|
|
||||||
|
> What realistic defect would this test catch?
|
||||||
|
|
||||||
|
A test is suspect when its main purpose is to detect that someone:
|
||||||
|
|
||||||
|
- changed an internal constant;
|
||||||
|
- renamed or split a private helper;
|
||||||
|
- reordered equivalent internal operations;
|
||||||
|
- changed incidental formatting;
|
||||||
|
- replaced one correct algorithm with another; or
|
||||||
|
- refactored internal object structure without changing behavior.
|
||||||
|
|
||||||
|
Refactoring should normally require no test edits unless the refactored structure is itself part of the contract.
|
||||||
|
|
||||||
|
A test can be factually correct and still have negative value. Accurately describing current behavior is not enough; the protected behavior must be important enough to justify the future friction.
|
||||||
|
|
||||||
|
## Expected effects of different changes
|
||||||
|
|
||||||
|
Use the following expectations when evaluating test failures and test maintenance:
|
||||||
|
|
||||||
|
| Change | Expected effect on tests |
|
||||||
|
|---|---|
|
||||||
|
| Internal refactor that preserves behavior | Existing tests should normally remain unchanged and continue to pass. |
|
||||||
|
| Change to an internal default with no contractual significance | Behavioral tests should normally remain unchanged; tests should derive expectations from configuration or relationships rather than duplicate the old value. |
|
||||||
|
| Intentional change to public behavior, policy, schema, or compatibility guarantees | The relevant tests should be reviewed and changed deliberately. |
|
||||||
|
| Accidental violation of a contract or invariant | Tests should fail; fix the production code rather than rewriting the tests to accept the defect. |
|
||||||
|
|
||||||
|
A test failing is not the same as a test needing to be edited. Many tests may correctly fail because of one production defect. The maintenance smell is a correct internal change that requires unrelated expectation updates throughout the suite.
|
||||||
|
|
||||||
|
## Separate mechanism from policy
|
||||||
|
|
||||||
|
Configurable thresholds and defaults must not be duplicated throughout the test suite.
|
||||||
|
|
||||||
|
For example, do not encode an internal concurrency limit indirectly:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Production policy:
|
||||||
|
const maxConcurrency = 4
|
||||||
|
|
||||||
|
// Brittle test:
|
||||||
|
err := startProcesses(5)
|
||||||
|
require.Error(t, err)
|
||||||
|
```
|
||||||
|
|
||||||
|
Instead, test the mechanism relationally:
|
||||||
|
|
||||||
|
```go
|
||||||
|
const limit = 2
|
||||||
|
runner := NewRunner(limit)
|
||||||
|
|
||||||
|
require.NoError(t, runner.Start(limit))
|
||||||
|
require.ErrorIs(t, runner.Start(limit+1), ErrTooMuchConcurrency)
|
||||||
|
```
|
||||||
|
|
||||||
|
The test should prove:
|
||||||
|
|
||||||
|
- the configured limit is accepted; and
|
||||||
|
- one beyond the configured limit is rejected.
|
||||||
|
|
||||||
|
The production default should be tested exactly only when its literal value is itself a public, operational, safety, protocol, or compatibility requirement.
|
||||||
|
|
||||||
|
Apply the same rule to limits, timeouts, capacities, retry counts, and ranges: test relationships and behavior, not duplicated literals.
|
||||||
|
|
||||||
|
For concurrency limits, test both kinds of behavior when relevant:
|
||||||
|
|
||||||
|
1. **Configuration enforcement:** invalid or excessive requested values are handled correctly.
|
||||||
|
2. **Runtime enforcement:** observed peak concurrency never exceeds the configured limit.
|
||||||
|
|
||||||
|
Use a test-controlled limit and measure the behavior relative to that limit. Do not merely assert today's default value.
|
||||||
|
|
||||||
|
## Avoid semantic duplication across layers
|
||||||
|
|
||||||
|
Each behavior should have a clear test owner.
|
||||||
|
|
||||||
|
- Parser tests own parsing cases.
|
||||||
|
- Validator tests own validation rules.
|
||||||
|
- Domain tests own transformations and invariants.
|
||||||
|
- Adapter tests own external integration behavior.
|
||||||
|
- Orchestrator tests own coordination and failure propagation.
|
||||||
|
- CLI tests own argument and configuration mapping.
|
||||||
|
- End-to-end tests prove that representative assembled workflows work.
|
||||||
|
|
||||||
|
Higher-level tests should not repeat every lower-level case. A single intentional policy change should not require unrelated edits across many test files.
|
||||||
|
|
||||||
|
Tests that are individually reasonable may still be collectively redundant. Evaluate the marginal value of each additional test in light of the protection already provided by the rest of the suite.
|
||||||
|
|
||||||
|
## Use test doubles deliberately
|
||||||
|
|
||||||
|
Choose the least elaborate test double that provides the required control or observation.
|
||||||
|
|
||||||
|
As a default:
|
||||||
|
|
||||||
|
1. Prefer real collaborators when they are fast and deterministic.
|
||||||
|
2. Use small in-memory fakes when realistic stateful behavior is helpful.
|
||||||
|
3. Use stubs when a dependency only needs to provide controlled responses.
|
||||||
|
4. Use mocks when the interaction itself is contractual.
|
||||||
|
|
||||||
|
Mocks are appropriate when the contract includes facts such as:
|
||||||
|
|
||||||
|
- a notification is sent exactly once;
|
||||||
|
- a transaction is committed only after successful writes;
|
||||||
|
- cancellation reaches a subprocess;
|
||||||
|
- an expensive API is called no more than once; or
|
||||||
|
- a security audit event is emitted.
|
||||||
|
|
||||||
|
Do not use mocks merely to isolate every object or reproduce the implementation's call graph.
|
||||||
|
|
||||||
|
## Go-specific guidance
|
||||||
|
|
||||||
|
Use:
|
||||||
|
|
||||||
|
- table-driven tests for meaningful behavioral categories and boundaries;
|
||||||
|
- `t.TempDir()` for real filesystem behavior;
|
||||||
|
- `httptest.Server` for realistic HTTP interactions;
|
||||||
|
- fuzz tests for parsers, normalization, path handling, and broad input spaces;
|
||||||
|
- golden files only when the complete output is intentionally stable;
|
||||||
|
- integration tests where correctness depends on component interaction; and
|
||||||
|
- a small number of representative end-to-end tests.
|
||||||
|
|
||||||
|
Avoid exact error-string assertions unless the wording is itself contractual. Prefer `errors.Is`, `errors.As`, typed errors, or structured error fields.
|
||||||
|
|
||||||
|
At CLI boundaries, prefer exit classifications, structured output, and the smallest stable semantic fragment needed to identify the error. Do not snapshot complete diagnostic wording unless it is contractual.
|
||||||
|
|
||||||
|
Golden-file updates must require an explicit local flag. CI must not update golden files automatically, and reviewers must inspect the semantic diff before accepting an update.
|
||||||
|
|
||||||
|
Keep tests readable and direct. Test helpers and fixture frameworks must earn their own maintenance cost; do not build elaborate test infrastructure for small or isolated needs.
|
||||||
|
|
||||||
|
## Coverage
|
||||||
|
|
||||||
|
Coverage is a diagnostic, not a target.
|
||||||
|
|
||||||
|
Use it to find untested critical branches and unexpectedly weak packages. Do not write low-value tests solely to increase a percentage, and do not infer test quality from coverage alone.
|
||||||
|
|
||||||
|
Pure domain logic will often warrant higher coverage than CLI wiring or external adapters. Uneven coverage is acceptable when it reflects risk.
|
||||||
|
|
||||||
|
Increasing coverage is valuable only when the newly covered behavior protects a meaningful risk at an acceptable cost.
|
||||||
|
|
||||||
|
## Regression tests
|
||||||
|
|
||||||
|
A bug fix should normally include a regression test that fails before the fix and passes afterward.
|
||||||
|
|
||||||
|
Retain the test when the defect could realistically recur and its consequences justify the ongoing cost. Prefer the narrowest durable test of the violated contract or invariant; do not preserve accidental implementation details from the original bug.
|
||||||
|
|
||||||
|
Not every historical bug requires a permanent test. If the underlying design has made recurrence impossible, the test has become redundant, or a stronger invariant test now subsumes it, remove or consolidate it.
|
||||||
|
|
||||||
|
## Deleting or rewriting tests
|
||||||
|
|
||||||
|
Tests are maintained code, not permanent historical artifacts.
|
||||||
|
|
||||||
|
Delete or rewrite a test when its maintenance cost exceeds the confidence it provides.
|
||||||
|
|
||||||
|
Strong candidates include tests that:
|
||||||
|
|
||||||
|
- require updates after harmless internal changes;
|
||||||
|
- directly assert private constants without protecting a real contract;
|
||||||
|
- duplicate the same policy across several layers;
|
||||||
|
- verify mock choreography rather than outcomes;
|
||||||
|
- snapshot large amounts of incidental output;
|
||||||
|
- test trivial private helpers already exercised through stable package behavior;
|
||||||
|
- protect risks already covered more effectively elsewhere;
|
||||||
|
- are flaky, misleading, obsolete, or disproportionately expensive to diagnose; or
|
||||||
|
- no longer correspond to a plausible failure mode.
|
||||||
|
|
||||||
|
Several brittle tests may encode one genuine requirement. Replace them with one durable behavior-level or invariant test rather than preserving all of them.
|
||||||
|
|
||||||
|
Deleting a low-value test can improve the quality of the suite by reducing noise, maintenance burden, and friction around legitimate change.
|
||||||
|
|
||||||
|
## Reviewing a proposed test
|
||||||
|
|
||||||
|
Use the following questions when the value, boundary, or durability of a proposed test is not self-evident. Significant test additions should be reviewable against them, but written answers are not required for every routine test.
|
||||||
|
|
||||||
|
1. What realistic defect would it catch?
|
||||||
|
2. How likely is that defect?
|
||||||
|
3. How consequential would it be?
|
||||||
|
4. Is the behavior already protected elsewhere?
|
||||||
|
5. At which layer should this behavior be owned?
|
||||||
|
6. Does the test assert a durable contract or an incidental implementation detail?
|
||||||
|
7. Could the implementation be refactored without changing the behavior and without editing this test?
|
||||||
|
8. What should cause this test to fail?
|
||||||
|
9. What legitimate changes should not cause this test to fail?
|
||||||
|
10. What ongoing maintenance, execution, and diagnostic cost will the test impose?
|
||||||
|
11. Is there a smaller or more direct test that protects the same risk?
|
||||||
|
|
||||||
|
Do not add the test when its expected lifetime cost exceeds its expected protective value.
|
||||||
|
|
||||||
|
When deciding not to test plausible behavior, record or be able to explain why the risk is low, already protected, obvious, reversible, or cheaper to detect elsewhere.
|
||||||
|
|
||||||
|
## Definition of sufficient
|
||||||
|
|
||||||
|
A test suite is sufficient when:
|
||||||
|
|
||||||
|
- important contracts and invariants are protected;
|
||||||
|
- meaningful boundaries and failure modes are exercised;
|
||||||
|
- realistic and consequential regressions are credibly protected against silent recurrence;
|
||||||
|
- behavior involving data integrity, destructive operations, compatibility, security, concurrency, idempotency, and recovery is credibly protected;
|
||||||
|
- important external boundaries have realistic integration coverage;
|
||||||
|
- representative complete workflows are tested;
|
||||||
|
- failures provide useful signal rather than redundant noise;
|
||||||
|
- legitimate internal changes usually do not require test edits; and
|
||||||
|
- additional tests would mostly repeat existing protection or preserve inconsequential implementation details.
|
||||||
|
|
||||||
|
Sufficiency is a risk judgment, not a coverage percentage or test count. Reassess it as the application, its users, and the consequences of failure evolve.
|
||||||
|
|
||||||
|
The governing rule is:
|
||||||
|
|
||||||
|
> Test heavily where failure is consequential, subtle, or difficult to detect after the fact. Test lightly where failure is obvious, reversible, and inexpensive—and retain no test whose lifetime cost exceeds the confidence it provides.
|
||||||
@@ -1,556 +0,0 @@
|
|||||||
# Roadmap: Code Quality and Deduplication Audit
|
|
||||||
|
|
||||||
Status: Draft audit report
|
|
||||||
|
|
||||||
This report is a pre-1.0 implementation audit focused on high-confidence opportunities to simplify, centralize, or clarify Narratio before release. It is intentionally report-only: no refactors are included here.
|
|
||||||
|
|
||||||
The requested `docs/architecture.md` and `docs/development.md` paths do not exist in the current tree. This audit used the current policy documents at `docs/policy/architecture.md` and `docs/policy/development.md`, plus the current user, operator, and internal docs.
|
|
||||||
|
|
||||||
## 1. Executive Summary
|
|
||||||
|
|
||||||
Overall code quality is solid. The codebase has strong package boundaries in the important places: storage adapters expose a narrow object-store interface, AWS SDK types do not leak into app or stage logic, config loading is strict, and pipeline execution remains explicit and stage-driven. Recent pre-1.0 work has also produced useful central points for campaign/session config loading, secret-backed object-store creation, S3 audio caching, transcript artifact naming, local session paths, and S3 key construction.
|
|
||||||
|
|
||||||
The main release risk is not a large architectural flaw. It is policy drift from rapid feature growth. Several public-interface decisions now appear in more than one implementation path: artifact source interpretation, publish-output destination derivation, remote current-state inspection, cleanup safety checks, and session-oriented command parsing. Most of these are correct today, but a future bug fix would likely have to be made in multiple files.
|
|
||||||
|
|
||||||
Top three refactoring targets before 1.0:
|
|
||||||
|
|
||||||
1. Centralize artifact source and publish-output resolution across config validation, publish execution, status/artifacts output, restore, previous-cache hydration, and analyze input resolution.
|
|
||||||
2. Consolidate shared session-command flag parsing and config-loading context for run/resume/run-stage/analyze/publish/restore/clean/session helpers without introducing a generic command framework.
|
|
||||||
3. Finish the publish terminology cleanup internally so public `publish` behavior is not implemented through `archive`-named files, helpers, errors, and tests.
|
|
||||||
|
|
||||||
The codebase appears ready for a limited cleanup pass. No major architecture rewrite is warranted before 1.0.
|
|
||||||
|
|
||||||
## 2. High-Confidence Deduplication Opportunities
|
|
||||||
|
|
||||||
### Artifact Source and Publish Destination Policy Is Split Across Packages
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/config/validate.go`
|
|
||||||
- `internal/artifacts/artifact_resolver.go`
|
|
||||||
- `internal/artifacts/catalog.go`
|
|
||||||
- `internal/stage/archive.go`
|
|
||||||
- `internal/app/operator_helpers.go`
|
|
||||||
- `internal/previouscache/previouscache.go`
|
|
||||||
- `internal/stage/analyze.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
- Config validation accepts and derives destinations for `pipeline.publish.outputs[]` in `publishSourceKnown` and `derivePublishOutputDest`.
|
|
||||||
- Publish execution derives destinations again in `resolvePublishOutputDest`.
|
|
||||||
- Status and `artifacts list` derive destination display and remote checks in `helperPublishedOutputDest`.
|
|
||||||
- Previous-cache hydration reconstructs candidate artifact locations from manifest outputs, published paths, and configured Scriptorium paths in `artifactRelativePathCandidates`.
|
|
||||||
- Analyze resolves previous-session, built-in, and configured artifact sources separately in `resolveScriptoriumInput`.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
Artifact source IDs now define the public contract for analyze inputs, previous-session inputs, publish outputs, locks, status, artifacts listing, restore, and validation. When source interpretation is spread across these packages, it is easy for one path to accept, reject, or resolve a source differently from another.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Create one small artifact-source policy layer, likely in `internal/artifacts` or a dependency-light sibling of `internal/artifactmodel`, that can:
|
|
||||||
|
|
||||||
- classify source IDs as built-in, configured artifact, or previous-session configured artifact;
|
|
||||||
- validate a source against the current Scriptorium config;
|
|
||||||
- derive the default published destination for a source;
|
|
||||||
- normalize relative artifact destinations;
|
|
||||||
- return consistent display metadata for status and artifacts output.
|
|
||||||
|
|
||||||
Then update config validation, publish execution, helper commands, previous-cache planning, and analyze input resolution to call that policy instead of deriving partial answers locally.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- `internal/artifacts`: source classification, configured artifact validation, default destination derivation, relative destination normalization.
|
|
||||||
- `internal/config`: publish outputs and locks validate through the shared policy.
|
|
||||||
- `internal/stage`: publish output resolution preserves locked, optional, required, and selected-artifact behavior.
|
|
||||||
- `internal/app`: `artifacts list`, `status`, and locks use the same source rules as publish.
|
|
||||||
- `internal/previouscache`: previous-session source resolution still checks manifest outputs, published paths, and configured output paths in the intended order.
|
|
||||||
|
|
||||||
Risk level: Medium. The behavior is public, but a table-driven shared policy should reduce risk if introduced behind existing tests.
|
|
||||||
|
|
||||||
### Publish Terminology Cleanup Is Incomplete Internally
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/stage/archive.go`
|
|
||||||
- `internal/stage/archive_test.go`
|
|
||||||
- `internal/artifacts/archive_identity.go`
|
|
||||||
- `internal/app/post_archive_cleanup.go`
|
|
||||||
- `internal/app/remote_locks.go`
|
|
||||||
- `internal/app/operator_helpers.go`
|
|
||||||
- tests under `internal/app` and `internal/config`
|
|
||||||
- `internal/adapters/storage/archive.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
The public contract now uses `publish`, `published`, and `publish outputs`, but several internal names still use `archive`, `promotion`, or `promoted`. Examples include `archiveStage`, `ResolveArchiveSessionPrefix`, `ResolveArchiveCurrentStateKeys`, `runPostArchiveCleanup`, `staticArchiveLocks`, `normalizeArchiveRelativePath`, and test names such as `TestArchiveUploadsRunRecordPromotionsAndCurrentPointer`.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
This is mostly clarity risk, not current behavior risk. However, public docs and config now use publish terminology, while implementation and tests still use old names. This makes code review harder and increases the chance that future work reintroduces old config or command language.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Do a mechanical naming cleanup after artifact-source policy is centralized:
|
|
||||||
|
|
||||||
- rename `internal/stage/archive.go` to a publish-oriented file and rename `archiveStage` to `publishStage`;
|
|
||||||
- rename archive identity helpers to publish/current-state helpers while keeping S3 layout unchanged;
|
|
||||||
- rename post-archive cleanup helpers and tests to post-publish cleanup;
|
|
||||||
- update old comments and test failure messages that still say archive/promote when they mean publish/published;
|
|
||||||
- leave the immutable run-history path `runs/{run_id}` unchanged.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Existing `internal/stage`, `internal/app`, and `internal/artifacts` tests.
|
|
||||||
- A final term sweep for old terminology, allowing only historical roadmap references and adapter names that are intentionally retained.
|
|
||||||
|
|
||||||
Risk level: Low to Medium. Mostly mechanical, but broad enough to create churn.
|
|
||||||
|
|
||||||
### Session-Oriented CLI Parsing Is Repeated
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/run.go`
|
|
||||||
- `internal/app/resume.go`
|
|
||||||
- `internal/app/run_stage.go`
|
|
||||||
- `internal/app/restore.go`
|
|
||||||
- `internal/app/clean.go`
|
|
||||||
- `internal/app/operator_helpers.go`
|
|
||||||
- `internal/app/session_args.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
Many commands repeat the same flag setup and session ID handling:
|
|
||||||
|
|
||||||
- `--config`, `--campaign`, `--campaign-file`, `--session`, and `--previous-session-id`;
|
|
||||||
- positional session ID extraction;
|
|
||||||
- `--session-id` compatibility through `applyParsedSessionIDArg`;
|
|
||||||
- selected artifact parsing and validation for run/resume/analyze/publish/run-stage;
|
|
||||||
- load through `loadCommandConfig` followed by `config.Validate`.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
The command set has recently moved toward `narratio session <subcommand> <session_id>` and shorter top-level convenience commands. Repeated parser setup makes it easy for one command to miss a new flag, use a stale help string, or apply session ID precedence differently.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Keep command functions explicit, but add a small internal parser helper for common session-aware commands. Avoid a generic CLI framework. A good target is a helper that returns:
|
|
||||||
|
|
||||||
- common config flags;
|
|
||||||
- resolved positional/flag session ID;
|
|
||||||
- previous session override;
|
|
||||||
- optional selected configured artifacts;
|
|
||||||
- normalized command-specific positional validation.
|
|
||||||
|
|
||||||
`run-stage` can remain special because it has both stage and session positional arguments, but it should reuse the same common flag registration and selected-artifact parsing.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Existing app command tests for run, resume, run-stage, analyze, publish, restore, clean, and session subcommands.
|
|
||||||
- Focused tests for positional session ID vs `--session-id` mismatch, missing session ID, and unsupported `--artifacts` by command/stage.
|
|
||||||
|
|
||||||
Risk level: Medium. Refactor is local to app parsing but touches many public commands.
|
|
||||||
|
|
||||||
### Remote Current-State Discovery Is Reimplemented in Several Forms
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/restore_discovery.go`
|
|
||||||
- `internal/previouscache/previouscache.go`
|
|
||||||
- `internal/app/operator_helpers.go`
|
|
||||||
- `internal/stage/prepare_previous.go`
|
|
||||||
- `internal/app/remote_locks.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
Several paths check or download remote current state:
|
|
||||||
|
|
||||||
- restore discovers current run ID and current manifest, validates campaign/session identity, and decodes the manifest;
|
|
||||||
- previous-cache planning repeats current run pointer and manifest checks for the previous session;
|
|
||||||
- session validation checks previous current state with `Exists` calls;
|
|
||||||
- remote lock loading separately checks and downloads `locks.yml`;
|
|
||||||
- remote session fallback lists and downloads `session.yml`.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
These workflows are similar but not identical. Some need missing remote state to be an error, while status treats it as state. Still, the low-level sequence of key construction, `Exists`, temp download, decode, and campaign/session/run validation appears multiple times.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Extract narrow app-level or artifact-level helpers for remote session state objects, not a generic storage workflow engine. Candidate helpers:
|
|
||||||
|
|
||||||
- download object to temp safely;
|
|
||||||
- load current run pointer and manifest for a supplied session prefix;
|
|
||||||
- validate downloaded current manifest identity;
|
|
||||||
- represent missing current state as a typed error so status can downgrade it while restore/prepare fail.
|
|
||||||
|
|
||||||
Keep `storage.ObjectStore` as the boundary and keep S3 key construction in `internal/artifacts`.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- `internal/app`: restore current-state discovery, status missing-state behavior, session validate previous-state behavior.
|
|
||||||
- `internal/previouscache`: required vs optional previous artifact behavior with missing current pointers/manifests.
|
|
||||||
- `internal/app`: malformed remote lock/session data still fails closed where publish-capable execution requires it.
|
|
||||||
|
|
||||||
Risk level: Medium. The missing-state policy differs by caller, so the refactor should centralize mechanics and typed outcomes, not final command decisions.
|
|
||||||
|
|
||||||
### Safe Local Deletion Policy Is Duplicated
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/clean.go`
|
|
||||||
- `internal/app/post_archive_cleanup.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
Both `clean` and post-publish cleanup implement scoped deletion checks:
|
|
||||||
|
|
||||||
- reject empty roots/targets;
|
|
||||||
- resolve absolute paths;
|
|
||||||
- refuse root deletion;
|
|
||||||
- refuse deletion outside the configured root;
|
|
||||||
- refuse symlink deletion;
|
|
||||||
- handle missing targets as successful no-ops.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
Deletion policy is high-risk code. Even if the current implementations agree, future fixes should not need to be made twice.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Extract a small app-level cleanup safety helper, for example `cleanup_target.go`, with functions for:
|
|
||||||
|
|
||||||
- validating a scoped directory target;
|
|
||||||
- validating a scoped file target;
|
|
||||||
- validating removable children under a root.
|
|
||||||
|
|
||||||
Keep command-specific reporting in `clean.go` and manifest metadata handling in post-publish cleanup.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Move the existing focused unsafe-path tests to the shared helper.
|
|
||||||
- Preserve `clean` dry-run tests and post-publish cleanup eligibility tests.
|
|
||||||
|
|
||||||
Risk level: Low. This is a contained refactor with clear behavior preservation.
|
|
||||||
|
|
||||||
### Temp Object Download Helper Is Duplicated
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/restore_discovery.go`
|
|
||||||
- `internal/previouscache/previouscache.go`
|
|
||||||
- `internal/app/remote_locks.go`
|
|
||||||
- `internal/app/config_loader.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
Multiple call sites create a temp file, close it, download an object into it, and delete it on error or defer deletion. The app package has one `downloadObjectToTemp`, while `internal/previouscache` has another copy.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
Temp-download behavior affects cleanup, error wording, and future hardening. It is not worth abstracting all storage use, but this small operation is repeated enough to centralize.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Add a narrow helper close to the storage boundary. Options:
|
|
||||||
|
|
||||||
- `internal/adapters/storage` helper only if it does not learn Narratio session semantics;
|
|
||||||
- `internal/storageutil` if a small internal utility package is acceptable;
|
|
||||||
- app-level helper plus a previouscache dependency inversion if the team wants to avoid a new package.
|
|
||||||
|
|
||||||
The helper should not hide `ObjectStore`; it should only implement safe temp download mechanics.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- temp file cleanup on failed download;
|
|
||||||
- successful download returns a cleaned temp path;
|
|
||||||
- callers preserve their current contextual error messages.
|
|
||||||
|
|
||||||
Risk level: Low.
|
|
||||||
|
|
||||||
## 3. Medium-Confidence Opportunities
|
|
||||||
|
|
||||||
### Operator Helper Implementation Is Too Broad for One File
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/operator_helpers.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
This 1,100+ line file owns session validation, status, session init, artifacts listing, locks list/add/remove, lock-store mutation, artifact catalog rendering, remote output availability, finding formatting, local input validation, and template rendering.
|
|
||||||
|
|
||||||
Why it matters:
|
|
||||||
|
|
||||||
The code is not inherently wrong, and keeping helper commands in `internal/app` fits the architecture. The issue is discoverability and local coupling. Small changes to one helper command require navigating unrelated helper behavior.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Split by command or responsibility:
|
|
||||||
|
|
||||||
- `session_init.go`
|
|
||||||
- `session_validate.go`
|
|
||||||
- `status.go`
|
|
||||||
- `artifacts_list.go`
|
|
||||||
- `locks.go`
|
|
||||||
- `helper_findings.go`
|
|
||||||
- `helper_artifacts.go`
|
|
||||||
|
|
||||||
Do this only after higher-value policy centralization so the file split does not preserve duplicated logic under new names.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Existing `internal/app/operator_helpers_test.go` can be split later, but a file split alone should not require behavior changes.
|
|
||||||
|
|
||||||
Risk level: Low.
|
|
||||||
|
|
||||||
### Restore Planning Contains Its Own Remote-to-Local Path Policy
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/restore_plan.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
Restore maps remote session keys back to local session paths in `restoreLocalRelativePathForKey`, with explicit include/exclude rules for `current/`, `runs/`, `logs/`, `reports/`, `config/`, `inputs/`, `transcripts/`, `artifacts/`, `previous/`, and optional `audio/`.
|
|
||||||
|
|
||||||
Why it may be intentional:
|
|
||||||
|
|
||||||
Restore is the only command that should translate an entire remote session prefix into a local session subset. It has command-specific conflict and `--include-audio` semantics.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Do not generalize this immediately. If it changes again, move only the remote-key-to-local-restore-scope classifier into a small helper with table-driven tests. Leave restore action classification local to restore.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Restore scope tests for every included/excluded root.
|
|
||||||
- Audio-specific conflict behavior remains separate.
|
|
||||||
|
|
||||||
Risk level: Low.
|
|
||||||
|
|
||||||
### Manifest Output Scanning Is Repeated but Mostly Stage-Specific
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/artifacts/artifact_resolver.go`
|
|
||||||
- `internal/previouscache/previouscache.go`
|
|
||||||
- `internal/app/runner.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
Several call sites inspect manifest stage outputs or metadata to find artifact paths, published paths, run roots, or configured artifact outputs.
|
|
||||||
|
|
||||||
Why it may be intentional:
|
|
||||||
|
|
||||||
Manifest state has different meanings depending on caller: runtime artifact resolution, previous-cache reconstruction, and run summary construction are not the same policy.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Avoid a broad manifest-query abstraction before 1.0. Consider adding only narrow helpers for stable metadata reads, such as reading `published_paths` from the publish stage, if the previous-cache and restore paths continue to grow.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Existing manifest resolver tests plus previous-cache tests.
|
|
||||||
|
|
||||||
Risk level: Low.
|
|
||||||
|
|
||||||
### Command Output Formatting Could Be More Consistent
|
|
||||||
|
|
||||||
Affected files/packages:
|
|
||||||
|
|
||||||
- `internal/app/operator_helpers.go`
|
|
||||||
- `internal/app/restore_report.go`
|
|
||||||
- `internal/app/restore_plan.go`
|
|
||||||
- `internal/app/clean.go`
|
|
||||||
- `internal/app/plan.go`
|
|
||||||
|
|
||||||
Duplicated or near-duplicated behavior:
|
|
||||||
|
|
||||||
Status, session validate, artifacts list, locks, clean dry-run, restore dry-run, and plan all render text directly with `fmt.Fprintf`.
|
|
||||||
|
|
||||||
Why it may be intentional:
|
|
||||||
|
|
||||||
The output remains text-only and command-specific. A generic renderer would add complexity without much value.
|
|
||||||
|
|
||||||
Recommended refactor:
|
|
||||||
|
|
||||||
Postpone unless user-facing inconsistencies become painful. A small findings renderer already exists for validation-style output; that is enough for now.
|
|
||||||
|
|
||||||
Suggested tests:
|
|
||||||
|
|
||||||
- Snapshot-style output tests only for stable operator-facing lines that support workflows.
|
|
||||||
|
|
||||||
Risk level: Low.
|
|
||||||
|
|
||||||
## 4. Boundary and Responsibility Concerns
|
|
||||||
|
|
||||||
The major boundaries are healthy:
|
|
||||||
|
|
||||||
- `internal/adapters/storage` owns external storage implementation details.
|
|
||||||
- App code creates object stores through `newCommandObjectStore`, which loads filesystem secrets first.
|
|
||||||
- Stage code depends on `storage.ObjectStore`, not AWS SDK types.
|
|
||||||
- `internal/audio` correctly centralizes S3 audio cache materialization without making the storage adapter aware of cache policy.
|
|
||||||
- `internal/artifacts` owns most local paths and S3 keys.
|
|
||||||
|
|
||||||
Concerns to address:
|
|
||||||
|
|
||||||
- Artifact source policy is split between `internal/config`, `internal/artifacts`, `internal/stage`, `internal/app`, and `internal/previouscache`. This is the clearest boundary drift because source IDs are a shared public contract.
|
|
||||||
- `internal/config` currently derives default publish destinations. Validation should be able to call source policy, but the canonical mapping itself should live outside config.
|
|
||||||
- `internal/app/operator_helpers.go` owns artifact catalog rendering and remote published-output state. That is acceptable for formatting, but destination derivation and source classification should move out.
|
|
||||||
- `internal/stage/archive.go` implements the public `publish` stage. This does not violate boundaries, but it creates conceptual drift.
|
|
||||||
|
|
||||||
Recommended home for shared logic:
|
|
||||||
|
|
||||||
- Source classification and destination derivation: `internal/artifacts` or `internal/artifactmodel` plus a small adapter from Scriptorium config.
|
|
||||||
- Remote key construction: continue using `internal/artifacts`.
|
|
||||||
- Object-store initialization: keep in `internal/app`.
|
|
||||||
- Command parsing: keep in `internal/app`.
|
|
||||||
- Stage-specific execution policy: keep in `internal/stage`.
|
|
||||||
|
|
||||||
## 5. Path and Remote Key Construction Review
|
|
||||||
|
|
||||||
Local path construction is mostly centralized:
|
|
||||||
|
|
||||||
- `internal/artifacts/paths.go` owns session work roots, run roots, spool paths, previous-cache paths, and audio cache paths.
|
|
||||||
- Stage code often gets `artifacts.SessionPaths` and joins stage-local files from those roots, which is appropriate.
|
|
||||||
- The previous-cache redundant nested artifact path has already been addressed by `previousArtifactCacheRelativePath`.
|
|
||||||
|
|
||||||
Remote key construction is mostly centralized:
|
|
||||||
|
|
||||||
- `internal/artifacts/s3_keys.go` owns session prefixes, run prefixes, audio prefixes, `session.yml`, `locks.yml`, current manifest/run pointer keys, published output keys, and run-relative keys.
|
|
||||||
- App and stage code call these helpers rather than scattering full S3 key string concatenation.
|
|
||||||
|
|
||||||
Areas needing cleanup:
|
|
||||||
|
|
||||||
- `ResolveArchiveBucket`, `ResolveArchiveSessionPrefix`, `ResolveArchiveRunPrefix`, and `ResolveArchiveCurrentStateKeys` should be renamed to publish/current-state terminology.
|
|
||||||
- `normalizeArchiveRelativePath` exists in both `internal/stage/archive.go` and `internal/previouscache/previouscache.go`; `normalizeHelperArchiveRelativePath` exists in `internal/app/operator_helpers.go`. These should converge into one helper for clean relative artifact destination paths.
|
|
||||||
- `restore_plan.go` owns `normalizeRemoteKey` and remote key scope mapping. That may remain restore-specific, but it should be watched because it overlaps with S3 key normalization helpers.
|
|
||||||
- `downloadObjectToTemp` exists in more than one package and can be centralized.
|
|
||||||
|
|
||||||
## 6. Artifact/Catalog/Source Resolution Review
|
|
||||||
|
|
||||||
Artifact source handling has a strong foundation:
|
|
||||||
|
|
||||||
- Transcript source IDs and paths are centralized in `internal/artifactmodel/transcripts.go`.
|
|
||||||
- Runtime artifact registry and resolver live in `internal/artifacts/artifact_resolver.go`.
|
|
||||||
- Configured artifact source IDs are consistently formed by `artifacts.ConfiguredArtifactSourceID`.
|
|
||||||
- Previous-session source IDs are recognized by `artifacts.PreviousSessionArtifactName`.
|
|
||||||
- The runtime catalog supports built-ins, configured artifacts, selected artifact execution, and availability.
|
|
||||||
|
|
||||||
The remaining issue is that consumers still build their own partial views of this model:
|
|
||||||
|
|
||||||
- config validation validates and derives publish output destinations;
|
|
||||||
- publish execution resolves included outputs, skipped optional outputs, skipped unselected outputs, and locked outputs;
|
|
||||||
- status/artifacts list derives display destinations and remote published state;
|
|
||||||
- previous-cache planning reconstructs candidate remote paths from previous manifests and publish metadata;
|
|
||||||
- analyze input resolution has its own missing-source messages and previous-session behavior.
|
|
||||||
|
|
||||||
Recommendation:
|
|
||||||
|
|
||||||
Make artifact/source resolution the next cleanup target. The goal is not to create one all-purpose resolver. The goal is to centralize the public source vocabulary and destination derivation so each caller can keep its own policy for missing/required/locked behavior.
|
|
||||||
|
|
||||||
## 7. Config and Command-Loading Review
|
|
||||||
|
|
||||||
Config loading is generally consistent:
|
|
||||||
|
|
||||||
- `loadCommandConfig` is the main command path for pipeline, campaign, session, local discovery, and remote session fallback.
|
|
||||||
- `loadPipelineCampaignConfig` covers commands that create session config and therefore cannot load an existing session.
|
|
||||||
- `newCommandObjectStore` correctly centralizes secret-backed object-store creation.
|
|
||||||
- `config.LoadSessionBytesWithOptions` now rejects session templates outside `session init`, preserving strict concrete session loading.
|
|
||||||
|
|
||||||
Intentional differences:
|
|
||||||
|
|
||||||
- `session init` loads only pipeline and campaign because it creates `session.yml`.
|
|
||||||
- `clean --all` loads only pipeline because it is not session-specific.
|
|
||||||
- `status --manifest` remains a compatibility/local-manifest mode.
|
|
||||||
|
|
||||||
Likely accidental drift to clean up:
|
|
||||||
|
|
||||||
- Common flags and help strings are repeated across commands.
|
|
||||||
- Some helper-command messages still say archive where they now mean publish.
|
|
||||||
- `restore` uses `fs.SetOutput(out)` while most other command parsers discard flag package output and wrap errors themselves. This may be intentional for `--help`, but it is a difference worth documenting or standardizing.
|
|
||||||
- App tests and helper names still contain old archive/promotion terminology, making it harder to see which public contract is current.
|
|
||||||
|
|
||||||
## 8. Refactors to Avoid Before 1.0
|
|
||||||
|
|
||||||
Avoid these before release:
|
|
||||||
|
|
||||||
- A generic workflow engine or DAG abstraction. The explicit stage list is a core design choice and is working.
|
|
||||||
- A broad manifest query framework. Add narrow helpers only where repeated policy is clear.
|
|
||||||
- Moving secret loading into storage adapters. Secret loading is app orchestration policy and should stay out of adapters.
|
|
||||||
- Making storage adapters infer campaign/session/root-prefix semantics. They should continue to receive concrete keys.
|
|
||||||
- Replacing command functions with a generic CLI framework. Small shared flag parsers are enough.
|
|
||||||
- Generalizing all file copy/download behavior. S3 audio cache materialization is intentionally special; ordinary restore/download logic has different semantics.
|
|
||||||
- Adding compatibility aliases for old archive/promote or old transcript names during cleanup. The repo has intentionally made hard cutovers.
|
|
||||||
|
|
||||||
## 9. Recommended Implementation Sequence
|
|
||||||
|
|
||||||
1. Centralize relative artifact destination normalization and temp object download helpers.
|
|
||||||
- Scope: low-risk shared helpers for repeated mechanics.
|
|
||||||
- Tests: `internal/artifacts` or helper-package tests, plus existing app/stage tests.
|
|
||||||
|
|
||||||
2. Centralize artifact source and publish-output policy.
|
|
||||||
- Scope: source classification, source validation, default published destination derivation, destination normalization.
|
|
||||||
- Tests: `internal/artifacts`, `internal/config`, `internal/stage -run Publish`, `internal/app -run 'Artifacts|Status|Locks'`, `internal/previouscache`.
|
|
||||||
|
|
||||||
3. Finish publish terminology cleanup.
|
|
||||||
- Scope: rename archive-named files/helpers/tests/comments where they now mean publish; keep S3 layout stable.
|
|
||||||
- Tests: `go test ./internal/stage -v`, `go test ./internal/app -v`, `go test ./internal/artifacts -v`.
|
|
||||||
|
|
||||||
4. Consolidate session-aware command parsing.
|
|
||||||
- Scope: common config/session/artifact flag registration and session ID resolution; no public CLI behavior change.
|
|
||||||
- Tests: app command tests for run, resume, run-stage, analyze, publish, restore, clean, session helpers.
|
|
||||||
|
|
||||||
5. Extract remote current-state mechanics.
|
|
||||||
- Scope: shared helpers for current run pointer/manifest load and identity validation, with typed missing-state errors.
|
|
||||||
- Tests: restore discovery, previous-cache, status, session validate.
|
|
||||||
|
|
||||||
6. Split operator helper implementation by responsibility.
|
|
||||||
- Scope: file organization and small formatting/helper extraction only after policy deduplication.
|
|
||||||
- Tests: existing `internal/app` tests.
|
|
||||||
|
|
||||||
7. Sweep dead transitional terminology and stale tests.
|
|
||||||
- Scope: comments, test names, old strings, internal docs that still say archive/promote where publish is now canonical.
|
|
||||||
- Tests: final `rg` sweeps plus full test run.
|
|
||||||
|
|
||||||
## 10. Test Strategy
|
|
||||||
|
|
||||||
Focused package checks for cleanup work:
|
|
||||||
|
|
||||||
- `go test ./internal/artifacts -v`
|
|
||||||
- `go test ./internal/config -v`
|
|
||||||
- `go test ./internal/stage -run 'Analyze|Publish|Prepare|Restore' -v`
|
|
||||||
- `go test ./internal/app -run 'Run|RunStage|Analyze|Publish|Restore|Clean|Status|Artifacts|Locks|Session' -v`
|
|
||||||
- `go test ./internal/previouscache -v`
|
|
||||||
- `go test ./internal/adapters/storage -v`
|
|
||||||
- `go test ./internal/manifest -v`
|
|
||||||
|
|
||||||
Tests to add or strengthen during follow-up refactors:
|
|
||||||
|
|
||||||
- one table of valid/invalid artifact source IDs used by config validation, publish, locks, status, and analyze;
|
|
||||||
- one table of default published destination derivation for built-in and configured artifacts;
|
|
||||||
- relative destination normalization and path traversal rejection;
|
|
||||||
- shared remote current-state load outcomes: missing pointer, missing manifest, malformed manifest, campaign mismatch, session mismatch, run ID mismatch;
|
|
||||||
- shared cleanup safety helper behavior for files, directories, roots, symlinks, and outside-root paths;
|
|
||||||
- common session command parsing behavior for positional session IDs, `--session-id`, mismatch errors, and unsupported artifacts flags.
|
|
||||||
|
|
||||||
Full validation after each cleanup commit:
|
|
||||||
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Useful final searches:
|
|
||||||
|
|
||||||
- `rg -n "archive|promote|promoted|promotion" internal docs examples cmd`
|
|
||||||
- `rg -n "ResolveArchive|archiveStage|post_archive|staticArchive|normalizeArchive" internal`
|
|
||||||
- `rg -n "narratio.transcript.merged|narratio.transcript.full|narratio.transcript.trimmed" internal docs examples`
|
|
||||||
- `rg -n "previous_session_artifact|promote_artifacts|pipeline.archive" internal docs examples`
|
|
||||||
|
|
||||||
## 11. Appendix: Findings Not Worth Acting On
|
|
||||||
|
|
||||||
- Stage-local path joins for files inside a stage run directory are acceptable. They are local implementation details, not shared path policy.
|
|
||||||
- Direct `fmt.Fprintf` output in simple commands is acceptable. A generic renderer would likely obscure behavior.
|
|
||||||
- Restore's remote-session-prefix filtering is command-specific enough to stay local unless restore scope changes again.
|
|
||||||
- `session init` template rendering should remain separate from ordinary session loading. That separation is now a useful safety boundary.
|
|
||||||
- S3 audio cache materialization is already centralized in `internal/audio`; do not fold it into a generic downloader.
|
|
||||||
- Manifest-driven resume behavior should not be abstracted broadly. The explicit runner behavior is easier to audit.
|
|
||||||
@@ -1,294 +0,0 @@
|
|||||||
# Roadmap: Pre-1.0 Code Cleanup
|
|
||||||
|
|
||||||
Status: Implemented (Stages 1-6 complete)
|
|
||||||
|
|
||||||
This roadmap turns the findings in `docs/roadmap/audit.md` into staged cleanup work for the 1.0 release and records completion status for each selected stage.
|
|
||||||
|
|
||||||
The cleanup work must follow the policy documents under `docs/policy/`, especially these invariants:
|
|
||||||
|
|
||||||
- keep Narratio explicit and stage-driven;
|
|
||||||
- do not introduce a generic workflow engine, DAG abstraction, or generic CLI framework;
|
|
||||||
- keep external-system details behind adapters;
|
|
||||||
- do not move campaign/session/root-prefix semantics into storage adapters;
|
|
||||||
- keep AWS SDK types out of app and stage logic;
|
|
||||||
- keep path and remote key construction centralized;
|
|
||||||
- preserve manifest-driven run state;
|
|
||||||
- keep public CLI/config behavior stable unless a stage explicitly says it is an internal naming cleanup.
|
|
||||||
|
|
||||||
## Non-Goals
|
|
||||||
|
|
||||||
- Do not change public command syntax, config schema, S3 key layout, manifest schema, or artifact source IDs as part of this cleanup.
|
|
||||||
- Do not add compatibility aliases or migration logic.
|
|
||||||
- Do not rewrite stage execution, manifest state transitions, or adapter contracts.
|
|
||||||
- Do not generalize text output into a generic reporting framework.
|
|
||||||
- Do not fold S3 audio cache behavior into a generic downloader.
|
|
||||||
- Do not move secret loading into storage adapters.
|
|
||||||
|
|
||||||
## Stage 1: Shared Low-Risk Mechanics
|
|
||||||
|
|
||||||
Goal: remove duplicated mechanics that are easy to test and should not affect public behavior.
|
|
||||||
|
|
||||||
Implementation decisions:
|
|
||||||
|
|
||||||
- Add one shared helper for safe relative artifact destination normalization.
|
|
||||||
- It must reject empty paths, absolute paths, `.`, `..`, and traversal outside the artifact/session scope.
|
|
||||||
- It must normalize separators to slash-form for artifact and S3 destination logic.
|
|
||||||
- It must be dependency-light enough to be called from config validation, app helpers, publish execution, and previous-cache planning.
|
|
||||||
- Add one shared object-store temp download helper.
|
|
||||||
- It must take `context.Context`, `storage.ObjectStore`, a key, and a temp-file pattern.
|
|
||||||
- It must create and close the temp file before download, remove the temp file on failed download, and return a cleaned local path on success.
|
|
||||||
- It must not infer bucket, campaign, session, run, or root-prefix semantics.
|
|
||||||
- Extract shared cleanup target validation for local deletion.
|
|
||||||
- Cover scoped directory deletion, scoped file deletion, and removable children under a root.
|
|
||||||
- Preserve existing safety rules: reject empty roots/targets, root deletion, outside-root paths, symlinks, and wrong target types.
|
|
||||||
- Keep command-specific output in `clean` and manifest metadata handling in post-publish cleanup.
|
|
||||||
|
|
||||||
Expected callers:
|
|
||||||
|
|
||||||
- replace duplicate relative destination normalization in publish execution, helper command rendering, and previous-cache planning;
|
|
||||||
- replace duplicate temp download helpers in app and previous-cache code;
|
|
||||||
- replace duplicate scoped deletion validation in clean and post-publish cleanup.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- Add focused tests for destination normalization and path traversal rejection.
|
|
||||||
- Add temp download tests for success, failed download cleanup, and preserved contextual caller errors.
|
|
||||||
- Add shared cleanup validation tests for directories, files, symlinks, missing targets, root deletion, and outside-root targets.
|
|
||||||
- Run:
|
|
||||||
- `go test ./internal/artifacts -v`
|
|
||||||
- `go test ./internal/adapters/storage -v`
|
|
||||||
- `go test ./internal/app -run 'Clean|Post' -v`
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- duplicated low-level mechanics are removed;
|
|
||||||
- public behavior and output are unchanged;
|
|
||||||
- no stage, command, or config semantics move into storage adapters.
|
|
||||||
|
|
||||||
## Stage 2: Artifact Source and Published Output Policy
|
|
||||||
|
|
||||||
Goal: make artifact source IDs and published-output destination derivation a single shared policy.
|
|
||||||
|
|
||||||
Implementation decisions:
|
|
||||||
|
|
||||||
- Introduce `internal/artifactpolicy` as the shared source policy package.
|
|
||||||
- This package is the long-term home because it avoids config/artifacts import cycles.
|
|
||||||
- It may depend on dependency-light model packages, but it must not depend on app, stage, manifest stores, storage adapters, or downstream adapters.
|
|
||||||
- Centralize these behaviors in `internal/artifactpolicy`:
|
|
||||||
- classify source IDs as built-in, configured artifact, or previous-session configured artifact;
|
|
||||||
- parse configured artifact keys from `narratio.artifact.<key>`;
|
|
||||||
- parse previous-session artifact keys from `narratio.previous_session.artifact.<key>`;
|
|
||||||
- validate configured artifact sources against `pipeline.scriptorium.artifacts`;
|
|
||||||
- validate publish lock/output sources;
|
|
||||||
- derive default published destinations for built-in and configured artifact sources;
|
|
||||||
- normalize safe relative published-output destinations.
|
|
||||||
- Update callers to consume the shared policy:
|
|
||||||
- config validation for `publish.outputs` and `publish.locks`;
|
|
||||||
- publish-stage output resolution;
|
|
||||||
- status and `artifacts list` rendering;
|
|
||||||
- locks list/add/remove validation;
|
|
||||||
- analyze input source handling;
|
|
||||||
- previous-cache candidate planning.
|
|
||||||
- Preserve caller-specific policy at call sites.
|
|
||||||
- Required vs optional behavior remains in publish, analyze, restore, and previous-cache callers.
|
|
||||||
- Locked output behavior remains in publish.
|
|
||||||
- Text formatting remains in app commands.
|
|
||||||
- Manifest path scanning remains in artifact/previous-cache logic unless directly tied to source policy.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- Add `internal/artifactpolicy` table tests for source classification, configured artifact validation, previous-session parsing, default destination derivation, and destination normalization.
|
|
||||||
- Update `internal/config` tests so publish outputs and locks validate through the shared policy.
|
|
||||||
- Update `internal/stage` publish tests for selected, unselected, optional, required, and locked output behavior.
|
|
||||||
- Update `internal/app` tests for status, artifacts list, and locks.
|
|
||||||
- Update `internal/previouscache` tests for previous-session candidate ordering.
|
|
||||||
- Run:
|
|
||||||
- `go test ./internal/artifacts -v`
|
|
||||||
- `go test ./internal/config -v`
|
|
||||||
- `go test ./internal/stage -run 'Analyze|Publish' -v`
|
|
||||||
- `go test ./internal/app -run 'Artifacts|Status|Locks' -v`
|
|
||||||
- `go test ./internal/previouscache -v`
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- artifact source vocabulary and destination derivation are no longer reimplemented in config, app, stage, and previous-cache packages;
|
|
||||||
- every caller still owns its own missing/required/optional/locked decision;
|
|
||||||
- public behavior is unchanged.
|
|
||||||
|
|
||||||
## Stage 3: Publish Terminology Cleanup
|
|
||||||
|
|
||||||
Goal: align internal implementation names with the public publish contract.
|
|
||||||
|
|
||||||
Implementation decisions:
|
|
||||||
|
|
||||||
- Rename archive-named internal files, types, helpers, comments, and tests that now implement publish behavior.
|
|
||||||
- Replace names such as:
|
|
||||||
- `archiveStage` with `publishStage`;
|
|
||||||
- `ResolveArchiveSessionPrefix` with publish/current-state terminology;
|
|
||||||
- `ResolveArchiveRunPrefix` with publish/run-history terminology;
|
|
||||||
- `ResolveArchiveCurrentStateKeys` with current-state terminology;
|
|
||||||
- `runPostArchiveCleanup` with post-publish cleanup terminology;
|
|
||||||
- `staticArchiveLocks` with publish lock terminology.
|
|
||||||
- Keep the S3 layout stable:
|
|
||||||
- `{session_prefix}/runs/{run_id}/`;
|
|
||||||
- `{session_prefix}/current/manifest.json`;
|
|
||||||
- `{session_prefix}/current/run_id.txt`;
|
|
||||||
- `{session_prefix}/locks.yml`.
|
|
||||||
- Keep the public stage name `publish`.
|
|
||||||
- Keep old archive/promote references only where they are historical roadmap context or intentionally describe immutable run history.
|
|
||||||
|
|
||||||
Tests and checks:
|
|
||||||
|
|
||||||
- Run:
|
|
||||||
- `go test ./internal/stage -v`
|
|
||||||
- `go test ./internal/app -v`
|
|
||||||
- `go test ./internal/artifacts -v`
|
|
||||||
- `go test ./...`
|
|
||||||
- Run stale-term sweeps:
|
|
||||||
- `rg -n "archive|promote|promoted|promotion" internal docs examples cmd`
|
|
||||||
- `rg -n "ResolveArchive|archiveStage|post_archive|staticArchive|normalizeArchive" internal`
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- public publish behavior is no longer implemented through archive/promote names;
|
|
||||||
- remaining old terms are intentionally historical, test-fixture bucket names, or roadmap-only context;
|
|
||||||
- no config, CLI, manifest, or S3 layout changes are introduced.
|
|
||||||
|
|
||||||
## Stage 4: Session Command Parsing Consolidation
|
|
||||||
|
|
||||||
Goal: reduce command-loading drift while keeping command handlers explicit.
|
|
||||||
|
|
||||||
Implementation decisions:
|
|
||||||
|
|
||||||
- Add a small app-level parser helper for common session-aware commands.
|
|
||||||
- Centralize:
|
|
||||||
- common config flags: `--config`, `--campaign`, `--campaign-file`, `--session`;
|
|
||||||
- positional session ID handling;
|
|
||||||
- `--session-id` compatibility;
|
|
||||||
- `--previous-session-id`;
|
|
||||||
- optional selected-artifact parsing for commands that support it.
|
|
||||||
- Keep command handlers explicit and readable.
|
|
||||||
- Do not introduce a generic CLI framework.
|
|
||||||
- Treat these as intentional special cases:
|
|
||||||
- `session init` loads pipeline and campaign but not session;
|
|
||||||
- `clean --all` loads pipeline only;
|
|
||||||
- `status --manifest` remains local-manifest mode;
|
|
||||||
- `run-stage` keeps its stage-name positional handling but reuses common flag parsing where practical.
|
|
||||||
- Standardize flag help text where commands use the same semantics.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- Update app command tests for:
|
|
||||||
- positional session ID;
|
|
||||||
- `--session-id`;
|
|
||||||
- positional/flag mismatch;
|
|
||||||
- missing session ID;
|
|
||||||
- `--previous-session-id`;
|
|
||||||
- unsupported `--artifacts` by command/stage;
|
|
||||||
- unchanged behavior for `session init`, `clean --all`, and `status --manifest`.
|
|
||||||
- Run:
|
|
||||||
- `go test ./internal/app -run 'Run|RunStage|Analyze|Publish|Restore|Clean|Session' -v`
|
|
||||||
- `go test ./internal/app -v`
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- shared session flag/session ID behavior has one implementation;
|
|
||||||
- command handlers remain command-specific;
|
|
||||||
- public command syntax and output stay unchanged.
|
|
||||||
|
|
||||||
## Stage 5: Remote Current-State Mechanics
|
|
||||||
|
|
||||||
Goal: centralize remote current-state loading mechanics without hiding caller policy.
|
|
||||||
|
|
||||||
Implementation decisions:
|
|
||||||
|
|
||||||
- Extract narrow helpers for remote current state.
|
|
||||||
- Load current run pointer through `storage.ObjectStore`.
|
|
||||||
- Load and decode current manifest through `storage.ObjectStore`.
|
|
||||||
- Validate campaign, session, and run identity when requested by the caller.
|
|
||||||
- Return typed missing-state errors.
|
|
||||||
- Preserve caller policy:
|
|
||||||
- restore treats missing or invalid current state as an error;
|
|
||||||
- previous-cache hydration fails for required previous artifacts and skips optional missing artifacts;
|
|
||||||
- status reports missing remote state as state, not command failure;
|
|
||||||
- session validate emits findings and fails only for error findings.
|
|
||||||
- Keep all remote key construction in `internal/artifacts`.
|
|
||||||
- Keep object-store initialization in `internal/app`.
|
|
||||||
- Do not add storage adapter knowledge of campaigns, sessions, runs, root prefixes, current state, or manifests.
|
|
||||||
|
|
||||||
Tests:
|
|
||||||
|
|
||||||
- Add helper tests for:
|
|
||||||
- missing current run pointer;
|
|
||||||
- missing current manifest;
|
|
||||||
- empty run pointer;
|
|
||||||
- malformed manifest;
|
|
||||||
- campaign mismatch;
|
|
||||||
- session mismatch;
|
|
||||||
- run ID mismatch.
|
|
||||||
- Update restore, previous-cache, status, and session validate tests to prove their caller-specific behavior is unchanged.
|
|
||||||
- Run:
|
|
||||||
- `go test ./internal/app -run 'Restore|Status|SessionValidate' -v`
|
|
||||||
- `go test ./internal/previouscache -v`
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- low-level remote current-state mechanics are shared;
|
|
||||||
- missing-state behavior remains caller-specific;
|
|
||||||
- storage adapter boundaries remain unchanged.
|
|
||||||
|
|
||||||
## Stage 6: Operator Helper File Split and Final Sweep
|
|
||||||
|
|
||||||
Goal: improve maintainability after shared policy and mechanics are already centralized.
|
|
||||||
|
|
||||||
Implementation decisions:
|
|
||||||
|
|
||||||
- Split the large operator helper implementation by command or responsibility.
|
|
||||||
- Suggested file grouping:
|
|
||||||
- session init;
|
|
||||||
- session validate;
|
|
||||||
- status;
|
|
||||||
- artifacts list;
|
|
||||||
- locks;
|
|
||||||
- helper findings;
|
|
||||||
- helper artifact rendering.
|
|
||||||
- Do not change command syntax, text output, config loading, remote loading, lock behavior, or artifact catalog behavior during the split.
|
|
||||||
- Keep output formatting text-only and command-specific unless a concrete inconsistency remains after the split.
|
|
||||||
- Update roadmap status notes after each completed stage.
|
|
||||||
|
|
||||||
Tests and checks:
|
|
||||||
|
|
||||||
- Run:
|
|
||||||
- `go test ./internal/app -v`
|
|
||||||
- `go test ./...`
|
|
||||||
- Final searches:
|
|
||||||
- `rg -n "archive|promote|promoted|promotion" internal docs examples cmd`
|
|
||||||
- `rg -n "narratio.transcript.merged|narratio.transcript.full|narratio.transcript.trimmed" internal docs examples`
|
|
||||||
- `rg -n "previous_session_artifact|promote_artifacts|pipeline.archive" internal docs examples`
|
|
||||||
|
|
||||||
Completion criteria:
|
|
||||||
|
|
||||||
- operator helper code is easier to navigate;
|
|
||||||
- stale implementation terminology is removed or intentionally documented;
|
|
||||||
- no behavior changes are introduced by file organization.
|
|
||||||
|
|
||||||
## Overall Validation
|
|
||||||
|
|
||||||
After each implementation stage:
|
|
||||||
|
|
||||||
- run the focused tests listed for that stage;
|
|
||||||
- run `go test ./...`;
|
|
||||||
- run `git status --short`;
|
|
||||||
- update this roadmap to mark the completed stage implemented only after code, tests, and documentation are aligned.
|
|
||||||
|
|
||||||
## Assumptions
|
|
||||||
|
|
||||||
- This roadmap is a cleanup plan, not a feature plan.
|
|
||||||
- Stages may be implemented as separate prompts/commits.
|
|
||||||
- `internal/artifactpolicy` is the chosen home for shared source policy.
|
|
||||||
- Shared object-store temp download helpers must not learn Narratio session semantics.
|
|
||||||
- Public behavior must remain stable unless a stage explicitly says it is internal terminology cleanup.
|
|
||||||
@@ -1,105 +0,0 @@
|
|||||||
# Documentation Pass: Stage 1 Audit
|
|
||||||
|
|
||||||
Status: Completed (2026-05-23)
|
|
||||||
|
|
||||||
## Scope Reviewed
|
|
||||||
|
|
||||||
- All non-policy documentation files under `docs/`
|
|
||||||
- `README.md`
|
|
||||||
- Documentation references to maintained `examples/` files
|
|
||||||
- Documentation-related expectations in tests under `internal/**`
|
|
||||||
|
|
||||||
## File Inventory and Canonical Scope
|
|
||||||
|
|
||||||
| File | Intended audience | Canonical scope (per policy) | Primary source-of-truth anchors |
|
|
||||||
| --- | --- | --- | --- |
|
|
||||||
| `README.md` | Users, operators | Project orientation and links | `cmd/narratio`, `internal/app/commands.go`, docs index files |
|
|
||||||
| `docs/cli.md` | Users, operators | CLI syntax, flags, command workflows | `internal/app/*.go`, `internal/app/*_test.go` |
|
|
||||||
| `docs/config.md` | Operators, advanced users | Config discovery, schema, defaults, examples | `internal/config/*.go`, `internal/config/*_test.go`, `examples/*` |
|
|
||||||
| `docs/operations.md` | Operators | Run/resume/publish/restore/cleanup workflows | `internal/app/runner.go`, `internal/app/restore*.go`, `internal/stage/archive.go`, `internal/artifacts/*.go` |
|
|
||||||
| `docs/troubleshooting.md` | Operators | Failure diagnosis and safe fixes | `internal/app`, `internal/stage`, related tests |
|
|
||||||
| `docs/internal/README.md` | Developers, LLM coding agents | Internal docs index and scope boundaries | `docs/internal/*.md`, policy docs |
|
|
||||||
| `docs/internal/adapters.md` | Developers, LLM coding agents | Adapter boundaries and ownership | `internal/adapters/*`, `internal/stage/*` |
|
|
||||||
| `docs/internal/artifacts.md` | Developers, LLM coding agents | Artifact catalog and source resolution contracts | `internal/artifacts/*`, `internal/stage/analyze.go`, `internal/stage/prepare_previous.go` |
|
|
||||||
| `docs/internal/command-restore.md` | Developers, LLM coding agents | Restore command architecture and contracts | `internal/app/restore*.go`, `internal/app/restore*_test.go` |
|
|
||||||
| `docs/internal/manifest.md` | Developers, LLM coding agents | Session/run manifest contracts and transitions | `internal/manifest/*`, `internal/app/runner.go`, `internal/stage/*` |
|
|
||||||
| `docs/internal/stage-prepare.md` | Developers, LLM coding agents | Prepare stage IO and invariants | `internal/stage/prepare.go`, `internal/stage/prepare*_test.go` |
|
|
||||||
| `docs/internal/stage-transcribe.md` | Developers, LLM coding agents | Transcribe stage IO and invariants | `internal/stage/transcribe.go`, `internal/stage/transcribe_test.go` |
|
|
||||||
| `docs/internal/stage-merge.md` | Developers, LLM coding agents | Merge stage IO and invariants | `internal/stage/merge.go`, `internal/stage/merge_test.go` |
|
|
||||||
| `docs/internal/stage-polish.md` | Developers, LLM coding agents | Polish stage IO and invariants | `internal/stage/polish.go`, `internal/stage/polish_test.go` |
|
|
||||||
| `docs/internal/stage-normalize.md` | Developers, LLM coding agents | Normalize stage IO and invariants | `internal/stage/normalize.go`, `internal/stage/normalize_test.go` |
|
|
||||||
| `docs/internal/stage-trim.md` | Developers, LLM coding agents | Trim stage IO and invariants | `internal/stage/trim.go`, `internal/stage/trim_test.go` |
|
|
||||||
| `docs/internal/stage-analyze.md` | Developers, LLM coding agents | Analyze stage artifact execution and selection | `internal/stage/analyze.go`, `internal/stage/analyze_test.go` |
|
|
||||||
| `docs/internal/stage-publish.md` | Developers, LLM coding agents | Publish-stage commit/upload invariants | `internal/stage/archive.go`, `internal/stage/archive_test.go` |
|
|
||||||
| `docs/internal/storage.md` | Developers, LLM coding agents | Storage adapter contracts and semantics | `internal/adapters/storage/*`, `internal/app/object_store.go` |
|
|
||||||
| `docs/internal/workspace.md` | Developers, LLM coding agents | Local workspace/session/run path model | `internal/artifacts/*`, `internal/app/runner.go`, `internal/stage/run_local.go` |
|
|
||||||
| `docs/integrations/README.md` | Developers, LLM coding agents | Integration docs index | `docs/integrations/*.md` |
|
|
||||||
| `docs/integrations/audita.md` | Developers, integration maintainers | Audita adapter contract | `internal/adapters/audita/*`, `internal/stage/polish.go` |
|
|
||||||
| `docs/integrations/seriatim.md` | Developers, integration maintainers | Seriatim adapter contract | `internal/adapters/seriatim/*`, `internal/stage/merge.go`, `internal/stage/normalize.go`, `internal/stage/trim.go` |
|
|
||||||
| `docs/integrations/scriptorium.md` | Developers, integration maintainers | Scriptorium adapter contract | `internal/adapters/scriptorium/*`, `internal/stage/analyze.go`, `internal/stage/trim.go` |
|
|
||||||
| `docs/roadmap/documentation.md` | Developers, maintainers | Planning and implementation sequencing for documentation pass | N/A (planning artifact) |
|
|
||||||
| `docs/roadmap/documentation-stage1-audit.md` | Developers, maintainers | Stage-1 inventory and source-of-truth audit record | N/A (planning artifact) |
|
|
||||||
|
|
||||||
## Source-of-Truth Mapping Summary
|
|
||||||
|
|
||||||
- CLI behaviors and command names are grounded in `internal/app/commands.go` and command handlers in `internal/app/*.go`.
|
|
||||||
- Stage order and canonical stage names are grounded in `internal/stage/placeholders.go` (`prepare` -> `transcribe` -> `merge` -> `polish` -> `normalize` -> `trim` -> `analyze` -> `publish` -> `notify`).
|
|
||||||
- Publish behavior and current-pointer commit semantics are grounded in `internal/stage/archive.go`.
|
|
||||||
- Config schema/defaults/validation are grounded in `internal/config/*`.
|
|
||||||
- Local/remote paths, publish keys, and workspace layout are grounded in `internal/artifacts/*`.
|
|
||||||
- Restore behavior and report contracts are grounded in `internal/app/restore*.go`.
|
|
||||||
- Maintained examples and schema compatibility are grounded in `examples/*` plus `internal/config/load_validate_test.go` (`TestExamplesLoadAndValidate`).
|
|
||||||
|
|
||||||
## Findings
|
|
||||||
|
|
||||||
### Broken or stale references
|
|
||||||
|
|
||||||
1. `README.md` linked to non-existent files:
|
|
||||||
- `docs/development.md`
|
|
||||||
- `docs/architecture.md`
|
|
||||||
2. `docs/internal/README.md` and `docs/integrations/README.md` linked to non-existent path:
|
|
||||||
- `docs/documentation/policy.md`
|
|
||||||
|
|
||||||
Stage-1 fix applied:
|
|
||||||
- Updated those links to existing policy docs under `docs/policy/`.
|
|
||||||
|
|
||||||
### Stale terminology sweep
|
|
||||||
|
|
||||||
Sweep terms used: `archive`, `promote`, `promoted`, `promote_artifacts`, `run-stage archive`.
|
|
||||||
|
|
||||||
Findings:
|
|
||||||
- User-facing docs in scope did not show obvious stale command examples requiring immediate correction.
|
|
||||||
- Internal code and tests still contain historical `archive` identifiers while user-facing command/stage naming is `publish` (for example, `internal/stage/archive.go` type names). This is acceptable for now but should be normalized deliberately, not incidentally.
|
|
||||||
|
|
||||||
Stage-1 fix applied:
|
|
||||||
- Updated clearly stale publish-related wording in test expectation messages/comments:
|
|
||||||
- `internal/app/commands_test.go`
|
|
||||||
- `internal/app/operator_helpers_test.go`
|
|
||||||
|
|
||||||
### Example path validation
|
|
||||||
|
|
||||||
- All `examples/...` paths referenced from non-policy docs resolve to existing files.
|
|
||||||
- `internal/config/load_validate_test.go` includes `TestExamplesLoadAndValidate` and points to current example files.
|
|
||||||
|
|
||||||
### Roadmap leakage into current-behavior docs
|
|
||||||
|
|
||||||
- No obvious roadmap-only behavior leakage found in non-roadmap docs during this sweep.
|
|
||||||
|
|
||||||
### Duplicate content and scope drift
|
|
||||||
|
|
||||||
- No severe duplication requiring immediate rewrite in this stage.
|
|
||||||
- Existing docs still need full content rewrite for 1.0 readiness in later stages (user/operator first, then internal/integrations), as planned.
|
|
||||||
|
|
||||||
### Canonical-home inconsistency to resolve in rewrite stages
|
|
||||||
|
|
||||||
- Policy canonical-home language names `docs/architecture.md` and `docs/development.md`, while current repository stores those policy documents under `docs/policy/`.
|
|
||||||
- Stage 1 preserves repository behavior by fixing broken links to existing files. Later rewrite stages should converge canonical-home paths and references consistently across docs.
|
|
||||||
|
|
||||||
## Stage-1 Completion Check
|
|
||||||
|
|
||||||
Completed for this stage:
|
|
||||||
- Full non-policy file inventory with audience and scope mapping.
|
|
||||||
- Source-of-truth crosswalk to code/tests.
|
|
||||||
- Stale-term, link, and example-path sweeps.
|
|
||||||
- Documentation-related stale test wording corrections.
|
|
||||||
- Minimal fixes only; broad rewrites intentionally deferred.
|
|
||||||
@@ -1,237 +0,0 @@
|
|||||||
# Roadmap: 1.0 Documentation Pass
|
|
||||||
|
|
||||||
Status: Completed (2026-05-23)
|
|
||||||
|
|
||||||
## Goal
|
|
||||||
|
|
||||||
Prepare Narratio documentation for a 1.0 release by reviewing and rewriting
|
|
||||||
every non-policy document under `docs/` against the implemented codebase.
|
|
||||||
|
|
||||||
The finished documentation set should be accurate, concise, complete for its
|
|
||||||
audience, and compliant with:
|
|
||||||
|
|
||||||
- `docs/policy/documentation.md`
|
|
||||||
- `docs/policy/architecture.md`
|
|
||||||
- `docs/policy/development.md`
|
|
||||||
|
|
||||||
Do not modify files under `docs/policy/` during this pass.
|
|
||||||
|
|
||||||
Current behavior belongs in canonical docs. Future, planned, aspirational, or
|
|
||||||
unimplemented behavior belongs only under `docs/roadmap/`.
|
|
||||||
|
|
||||||
## Scope
|
|
||||||
|
|
||||||
In scope:
|
|
||||||
|
|
||||||
- `docs/*.md`
|
|
||||||
- `docs/internal/*.md`
|
|
||||||
- `docs/integrations/*.md`
|
|
||||||
- `docs/roadmap/*.md`
|
|
||||||
- documentation references to files under `examples/`
|
|
||||||
- test expectation fixes when the documentation review exposes stale or
|
|
||||||
incorrect doc/example/path expectations
|
|
||||||
|
|
||||||
Out of scope:
|
|
||||||
|
|
||||||
- product/runtime code changes
|
|
||||||
- feature implementation
|
|
||||||
- edits under `docs/policy/`
|
|
||||||
- adding roadmap behavior to current-behavior docs before that behavior is
|
|
||||||
implemented
|
|
||||||
|
|
||||||
## Implementation Stages
|
|
||||||
|
|
||||||
### Stage 1: Inventory and Source-of-Truth Audit
|
|
||||||
|
|
||||||
Status: Completed (2026-05-23)
|
|
||||||
|
|
||||||
Create a file-by-file inventory of all non-policy docs before rewriting.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- List every non-policy documentation file and assign its intended audience.
|
|
||||||
- Identify each document's canonical scope using `docs/policy/documentation.md`.
|
|
||||||
- Compare docs against the current code and tests, especially:
|
|
||||||
- `internal/app`
|
|
||||||
- `internal/config`
|
|
||||||
- `internal/stage`
|
|
||||||
- `internal/artifacts`
|
|
||||||
- `examples`
|
|
||||||
- relevant tests under `internal/**`
|
|
||||||
- Record stale terminology, broken links, stale example paths, duplicate
|
|
||||||
content, and roadmap-only behavior that leaked into current-behavior docs.
|
|
||||||
- Record stale test expectations related to docs, examples, paths, or command
|
|
||||||
text.
|
|
||||||
- Do not rewrite content in this stage except obvious broken links or test
|
|
||||||
corrections needed to make documentation validation meaningful.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- The rewrite has a concrete file inventory and source-of-truth map.
|
|
||||||
- The team knows which docs are canonical and which should link elsewhere.
|
|
||||||
- Known stale terms and broken references are identified before broad edits.
|
|
||||||
|
|
||||||
### Stage 2: User and Operator Docs
|
|
||||||
|
|
||||||
Status: Completed (2026-05-23)
|
|
||||||
|
|
||||||
Rewrite the user-facing and operator-facing docs first.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Rewrite these docs as fresh, concise current-behavior references:
|
|
||||||
- `README.md`, if present
|
|
||||||
- `docs/cli.md`
|
|
||||||
- `docs/config.md`
|
|
||||||
- `docs/operations.md`
|
|
||||||
- `docs/troubleshooting.md`
|
|
||||||
- Verify every command, flag, config field, discovery rule, path, and workflow
|
|
||||||
against implemented behavior.
|
|
||||||
- Cover implemented 1.0 behavior, including:
|
|
||||||
- campaign registry selection;
|
|
||||||
- concrete session loading and template-driven `session init`;
|
|
||||||
- session-oriented helper commands;
|
|
||||||
- clean, restore, analyze, and publish workflows;
|
|
||||||
- artifact selection behavior;
|
|
||||||
- locks and published output behavior;
|
|
||||||
- workspace, spool, and cache behavior;
|
|
||||||
- secrets loading and S3-backed operation.
|
|
||||||
- Keep examples short and link to maintained files under `examples/` instead
|
|
||||||
of duplicating large config blocks.
|
|
||||||
- Fix tests only when they assert stale doc paths, example paths, command
|
|
||||||
names, or current-behavior text.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- User/operator docs are task-oriented and match actual CLI/config behavior.
|
|
||||||
- Current-behavior docs do not depend on roadmaps for normal usage.
|
|
||||||
- No current-behavior doc describes unimplemented roadmap items.
|
|
||||||
|
|
||||||
### Stage 3: Internal Developer Docs
|
|
||||||
|
|
||||||
Status: Completed (2026-05-23)
|
|
||||||
|
|
||||||
Rewrite implemented internal component docs after public docs stabilize.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Rewrite:
|
|
||||||
- `docs/internal/README.md`
|
|
||||||
- `docs/internal/adapters.md`
|
|
||||||
- `docs/internal/artifacts.md`
|
|
||||||
- `docs/internal/command-restore.md`
|
|
||||||
- `docs/internal/manifest.md`
|
|
||||||
- `docs/internal/stage-*.md`
|
|
||||||
- `docs/internal/storage.md`
|
|
||||||
- `docs/internal/workspace.md`
|
|
||||||
- Verify stage docs against current stage names, stage ordering, manifest
|
|
||||||
records, declared inputs/outputs, adapters, path helpers, storage behavior,
|
|
||||||
publish/current-state behavior, restore behavior, cache behavior, and
|
|
||||||
workspace cleanup.
|
|
||||||
- Keep implementation details in `docs/internal/`, not in user-facing docs.
|
|
||||||
- Avoid turning internal docs into duplicate config or CLI references; link to
|
|
||||||
canonical docs when needed.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Internal docs are accurate enough for developers and LLM coding agents to
|
|
||||||
change the system safely.
|
|
||||||
- Stage and adapter boundaries match `docs/policy/architecture.md`.
|
|
||||||
- Manifest, path, storage, and publish invariants are explicit and current.
|
|
||||||
|
|
||||||
### Stage 4: Integrations and Examples
|
|
||||||
|
|
||||||
Status: Completed (2026-05-23)
|
|
||||||
|
|
||||||
Review integration docs and maintained examples after core docs are rewritten.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Rewrite:
|
|
||||||
- `docs/integrations/README.md`
|
|
||||||
- `docs/integrations/audita.md`
|
|
||||||
- `docs/integrations/scriptorium.md`
|
|
||||||
- `docs/integrations/seriatim.md`
|
|
||||||
- Verify integration docs against current adapter contracts and expected
|
|
||||||
downstream tool behavior.
|
|
||||||
- Confirm every referenced example file exists.
|
|
||||||
- Confirm examples match current schema and command usage.
|
|
||||||
- Run or rely on example validation tests.
|
|
||||||
- Fix tests when they reference moved, renamed, or intentionally retired
|
|
||||||
examples.
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- Integration docs describe only implemented adapter expectations.
|
|
||||||
- Maintained examples are valid, secret-free, and linked from canonical docs.
|
|
||||||
- Example validation tests reflect the documented example set.
|
|
||||||
|
|
||||||
### Stage 5: Roadmap Cleanup and Final Sweep
|
|
||||||
|
|
||||||
Status: Completed (2026-05-23)
|
|
||||||
|
|
||||||
Clean up roadmap state and run final documentation validation.
|
|
||||||
|
|
||||||
Implementation requirements:
|
|
||||||
|
|
||||||
- Review `docs/roadmap/**` for implemented items that should be marked
|
|
||||||
implemented, retired, or left planned.
|
|
||||||
- Keep historical and planned behavior in roadmaps only.
|
|
||||||
- Run final link/path/term sweeps.
|
|
||||||
- Run validation commands:
|
|
||||||
- `go test ./internal/config -run TestExamplesLoadAndValidate -v`
|
|
||||||
- `go test ./internal/app -run TestExecute -v`
|
|
||||||
- `go test ./...`
|
|
||||||
|
|
||||||
Acceptance criteria:
|
|
||||||
|
|
||||||
- All non-policy docs are current for the 1.0 release.
|
|
||||||
- Roadmaps do not serve as required user/operator documentation.
|
|
||||||
- Tests pass after allowed documentation-related test expectation fixes.
|
|
||||||
|
|
||||||
## Required Checks
|
|
||||||
|
|
||||||
Run searches for stale terminology and references during the pass.
|
|
||||||
|
|
||||||
Stale terminology:
|
|
||||||
|
|
||||||
- `archive`
|
|
||||||
- `promote`
|
|
||||||
- `promoted`
|
|
||||||
- `promote_artifacts`
|
|
||||||
- legacy campaign path/discovery language
|
|
||||||
- old transcript names and paths
|
|
||||||
- removed CLI commands or aliases
|
|
||||||
|
|
||||||
Broken or stale references:
|
|
||||||
|
|
||||||
- missing local doc links;
|
|
||||||
- stale `examples/` paths;
|
|
||||||
- stale internal doc filenames;
|
|
||||||
- references to `docs/policy/**` as editable targets;
|
|
||||||
- command examples that no longer match the CLI.
|
|
||||||
|
|
||||||
Policy checks:
|
|
||||||
|
|
||||||
- Current-behavior docs mention only implemented behavior.
|
|
||||||
- Planned behavior appears only under `docs/roadmap/`.
|
|
||||||
- Docs do not expose raw secrets or recommend storing secrets in config.
|
|
||||||
- Docs use canonical homes:
|
|
||||||
- `docs/config.md` for config schema;
|
|
||||||
- `docs/cli.md` for command syntax;
|
|
||||||
- `docs/operations.md` for operator workflows;
|
|
||||||
- `docs/troubleshooting.md` for failure diagnosis;
|
|
||||||
- `docs/internal/` for implementation contracts;
|
|
||||||
- `docs/integrations/` for downstream tool integration notes;
|
|
||||||
- `docs/roadmap/` for future work.
|
|
||||||
|
|
||||||
## Assumptions
|
|
||||||
|
|
||||||
- `docs/policy/**` is read-only for this documentation pass.
|
|
||||||
- This pass is for 1.0 release readiness, not feature implementation.
|
|
||||||
- Product and runtime code changes are out of scope.
|
|
||||||
- Test fixes are in scope when they correct stale documentation, example, path,
|
|
||||||
command, or current-behavior expectations uncovered during the review.
|
|
||||||
- Roadmap files may remain as planning and historical records.
|
|
||||||
- Current-behavior docs must be sufficient for normal use without requiring
|
|
||||||
readers to consult roadmaps.
|
|
||||||
@@ -24,6 +24,8 @@ Safe fix:
|
|||||||
|
|
||||||
- pass explicit `--config`, `--campaign` or `--campaign-file`, and `--session`.
|
- pass explicit `--config`, `--campaign` or `--campaign-file`, and `--session`.
|
||||||
|
|
||||||
|
Relevant reference: [Configuration discovery](./config.md#discovery-and-selection).
|
||||||
|
|
||||||
## Session template placeholders rejected
|
## Session template placeholders rejected
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
@@ -44,6 +46,8 @@ Safe fix:
|
|||||||
|
|
||||||
- generate concrete session YAML with `narratio session init`.
|
- generate concrete session YAML with `narratio session init`.
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Session Initialization](./operations.md#session-initialization).
|
||||||
|
|
||||||
## Strict decode or schema validation failure
|
## Strict decode or schema validation failure
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
@@ -62,7 +66,10 @@ narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /pa
|
|||||||
|
|
||||||
Safe fix:
|
Safe fix:
|
||||||
|
|
||||||
- align config with [docs/config.md](./config.md) and maintained files under `examples/`.
|
- align config with [Configuration](./config.md) and the
|
||||||
|
[maintained examples](../examples/README.md).
|
||||||
|
|
||||||
|
Relevant reference: [Configuration](./config.md).
|
||||||
|
|
||||||
## Audio mode conflict
|
## Audio mode conflict
|
||||||
|
|
||||||
@@ -74,10 +81,18 @@ Likely cause:
|
|||||||
|
|
||||||
- configured both local and S3 session audio inputs.
|
- configured both local and S3 session audio inputs.
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session validate 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
Safe fix:
|
Safe fix:
|
||||||
|
|
||||||
- use local mode (`audio_dir` or `audio_files`) or S3 mode (`audio_s3.prefix`), not both.
|
- use local mode (`audio_dir` or `audio_files`) or S3 mode (`audio_s3.prefix`), not both.
|
||||||
|
|
||||||
|
Relevant reference: [Session configuration](./config.md#session).
|
||||||
|
|
||||||
## `--artifacts` selection error
|
## `--artifacts` selection error
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
@@ -90,10 +105,159 @@ Likely causes:
|
|||||||
- empty list entry (for example trailing comma);
|
- empty list entry (for example trailing comma);
|
||||||
- `run-stage` used with non-`analyze`/`publish` target.
|
- `run-stage` used with non-`analyze`/`publish` target.
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session artifacts 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
Safe fix:
|
Safe fix:
|
||||||
|
|
||||||
- provide only configured keys and use `--artifacts` with supported commands/stages.
|
- provide only configured keys and use `--artifacts` with supported commands/stages.
|
||||||
|
|
||||||
|
Relevant reference: [CLI artifact selection](./cli.md).
|
||||||
|
|
||||||
|
## Notarius executable missing
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- extraction fails while resolving or starting the Notarius executable.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- `pipeline.notarius.binary` is not installed, executable, or on `PATH`;
|
||||||
|
- a configured executable path is wrong.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- install a compatible Notarius release or correct the binary setting, then
|
||||||
|
rerun extraction.
|
||||||
|
|
||||||
|
Relevant references: [Notarius configuration](./config.md#notarius-output-entries)
|
||||||
|
and [Notarius integration](./integrations/notarius.md).
|
||||||
|
|
||||||
|
## Notarius exits nonzero
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- extraction reports a Notarius exit error instead of a receipt.
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
|
||||||
|
- inspect `runs/{run_id}/extract/notarius.stderr.log`; stdout is reserved for
|
||||||
|
the receipt and is not merged with diagnostics.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- correct the reported Notarius pipeline, input, provider, or configuration
|
||||||
|
failure and rerun extraction. Do not edit a staged output bundle into place.
|
||||||
|
|
||||||
|
After a failed replacement, an older immutable bundle may still exist even
|
||||||
|
though the current session manifest has no successful extraction payload. This
|
||||||
|
is expected audit state, not a signal to relink the old bundle manually.
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Extraction Workflow](./operations.md#extraction-workflow).
|
||||||
|
|
||||||
|
## Atomic Notarius promotion unsupported
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- extraction fails with `atomic no-replace directory promotion is unsupported`
|
||||||
|
before a durable bundle or temporary promotion tree is created.
|
||||||
|
|
||||||
|
Likely cause:
|
||||||
|
|
||||||
|
- Narratio is running on an operating system other than Linux, macOS, or
|
||||||
|
Windows, where the required atomic no-replace directory primitive has not
|
||||||
|
been implemented and verified.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- run extraction on Linux, macOS, or Windows. Do not replace the atomic commit
|
||||||
|
with a manual copy or move; the session manifest must never observe a partial
|
||||||
|
or overwritten bundle.
|
||||||
|
|
||||||
|
This is an extraction-specific platform boundary, not a support statement for
|
||||||
|
unrelated Narratio workflows. See
|
||||||
|
[Operations: Extraction Workflow](./operations.md#extraction-workflow).
|
||||||
|
|
||||||
|
## Notarius receipt or index incompatible
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- extraction rejects the receipt schema, pipeline identity, bundle/index path,
|
||||||
|
lane descriptor, or payload path even though Notarius exited successfully.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- Narratio and Notarius versions disagree on their consumer contract;
|
||||||
|
- the configured pipeline or lane constraints are stale;
|
||||||
|
- output paths escape the bundle or traverse symlinks.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- compare installed Notarius output with the canonical Notarius contracts,
|
||||||
|
including receipt `index_file: index.json` and index management names
|
||||||
|
`manifest.json`, `rejected.json`, and `warnings.json`; align
|
||||||
|
`pipeline.notarius` constraints and rerun. Do not bypass confinement or schema
|
||||||
|
checks.
|
||||||
|
|
||||||
|
Relevant reference: [Notarius integration](./integrations/notarius.md).
|
||||||
|
|
||||||
|
## Required Notarius lane rejected or missing
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- extraction fails because a configured lane is rejected, missing, duplicated,
|
||||||
|
or incompatible, including after a zero exit.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- inspect the Notarius diagnostic log and bundle rejection/warning information;
|
||||||
|
- correct the Notarius module or the exact declared lane contract;
|
||||||
|
- remove an output declaration only if downstream consumers genuinely no longer
|
||||||
|
require that source, then rerun extraction.
|
||||||
|
|
||||||
|
Every configured output is required. Narratio does not promote a partial result.
|
||||||
|
|
||||||
|
## Extraction resume invalidated
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- a previously successful extraction runs again during ordinary continuation.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- the executable/config path, pipeline ID, timeout, working directory, or
|
||||||
|
configured output contracts changed;
|
||||||
|
- the durable bundle, index, lane set, provenance, regular-file status, or
|
||||||
|
checksum no longer validates.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- allow the automatic rerun after verifying the current configuration. Treat
|
||||||
|
an unsafe path or symlink error as filesystem corruption or tampering and
|
||||||
|
investigate it rather than replacing files manually.
|
||||||
|
|
||||||
|
## Notarius transitive configuration changed
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- Notarius profiles, prompts, modules, imported files, or references changed,
|
||||||
|
but Narratio still considers the previous extraction resumable.
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage extract 2026-04-04 --force
|
||||||
|
```
|
||||||
|
|
||||||
|
Narratio fingerprints its invocation contract, not the contents of transitive
|
||||||
|
Notarius inputs. Always force extraction after changing them; downstream
|
||||||
|
successful stages are then marked stale normally.
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Extraction Workflow](./operations.md#extraction-workflow).
|
||||||
|
|
||||||
## Previous-session artifact input missing
|
## Previous-session artifact input missing
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
@@ -124,6 +288,8 @@ or rerun prepare after correcting session config:
|
|||||||
narratio run-stage prepare 2026-04-04 --force
|
narratio run-stage prepare 2026-04-04 --force
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Restore Workflow](./operations.md#restore-workflow).
|
||||||
|
|
||||||
## Session lock conflict (`.lock`)
|
## Session lock conflict (`.lock`)
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
@@ -147,6 +313,8 @@ Safe fix:
|
|||||||
- wait for active process completion;
|
- wait for active process completion;
|
||||||
- remove stale lock only after confirming no live process owns it.
|
- remove stale lock only after confirming no live process owns it.
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Local State Layout](./operations.md#local-state-layout).
|
||||||
|
|
||||||
## Restore conflict without `--force`
|
## Restore conflict without `--force`
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
@@ -168,6 +336,8 @@ Safe fix:
|
|||||||
- review conflicts;
|
- review conflicts;
|
||||||
- rerun with `--force` only when remote state should overwrite local.
|
- rerun with `--force` only when remote state should overwrite local.
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Restore Workflow](./operations.md#restore-workflow).
|
||||||
|
|
||||||
## Restore current-state discovery failure
|
## Restore current-state discovery failure
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
@@ -191,6 +361,8 @@ Safe fix:
|
|||||||
- resolve storage/auth issue;
|
- resolve storage/auth issue;
|
||||||
- republish from healthy local state if current pointer is missing.
|
- republish from healthy local state if current pointer is missing.
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Publish Workflow](./operations.md#publish-workflow).
|
||||||
|
|
||||||
## Publish output failure
|
## Publish output failure
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
@@ -217,6 +389,36 @@ Safe fix:
|
|||||||
- correct publish source/destination rules;
|
- correct publish source/destination rules;
|
||||||
- retry after storage failure is resolved.
|
- retry after storage failure is resolved.
|
||||||
|
|
||||||
|
Relevant reference: [Publish configuration](./config.md#publish-configuration-summary).
|
||||||
|
|
||||||
|
## Render markdown source missing
|
||||||
|
|
||||||
|
Symptom:
|
||||||
|
|
||||||
|
- analyze or publish fails because `narratio.transcript.final_markdown` or `narratio.transcript.final_trimmed_markdown` is unavailable.
|
||||||
|
|
||||||
|
Likely causes:
|
||||||
|
|
||||||
|
- render stage was not executed after transcript changes;
|
||||||
|
- render stage failed before producing canonical markdown outputs.
|
||||||
|
|
||||||
|
Diagnostics:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio session status 2026-04-04
|
||||||
|
```
|
||||||
|
|
||||||
|
Safe fix:
|
||||||
|
|
||||||
|
- rerun render and then retry downstream stage(s):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
narratio run-stage render 2026-04-04 --force
|
||||||
|
narratio run-stage analyze 2026-04-04 --force
|
||||||
|
```
|
||||||
|
|
||||||
|
Relevant reference: [Operations: Stage Execution](./operations.md#stage-execution-and-continuation-behavior).
|
||||||
|
|
||||||
## Secrets or storage credential failure
|
## Secrets or storage credential failure
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
@@ -242,6 +444,8 @@ Safe fix:
|
|||||||
- provide required env vars;
|
- provide required env vars;
|
||||||
- keep secret values out of YAML.
|
- keep secret values out of YAML.
|
||||||
|
|
||||||
|
Relevant reference: [Secrets](./config.md#secrets-handling).
|
||||||
|
|
||||||
## S3 audio prepare failure
|
## S3 audio prepare failure
|
||||||
|
|
||||||
Symptom:
|
Symptom:
|
||||||
@@ -257,7 +461,7 @@ Likely causes:
|
|||||||
Diagnostics:
|
Diagnostics:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
narratio run-stage prepare 2026-04-04 --force
|
narratio session validate 2026-04-04
|
||||||
```
|
```
|
||||||
|
|
||||||
Safe fix:
|
Safe fix:
|
||||||
@@ -265,6 +469,8 @@ Safe fix:
|
|||||||
- verify prefix contents and storage access;
|
- verify prefix contents and storage access;
|
||||||
- keep session audio mode consistent.
|
- keep session audio mode consistent.
|
||||||
|
|
||||||
|
Relevant reference: [Operations](./operations.md).
|
||||||
|
|
||||||
## References
|
## References
|
||||||
|
|
||||||
- [docs/cli.md](./cli.md)
|
- [docs/cli.md](./cli.md)
|
||||||
|
|||||||
48
examples/README.md
Normal file
48
examples/README.md
Normal file
@@ -0,0 +1,48 @@
|
|||||||
|
# Maintained Examples
|
||||||
|
|
||||||
|
These files are safe, copyable starting points for Narratio configuration and
|
||||||
|
input structure. Replace placeholder identifiers, storage names, integration
|
||||||
|
URLs, and paths for the target environment. Field meanings and defaults belong
|
||||||
|
in the [configuration reference](../docs/config.md).
|
||||||
|
|
||||||
|
## Pipeline Configuration
|
||||||
|
|
||||||
|
- [Minimal pipeline](pipeline.minimal.yml): campaign discovery plus the required
|
||||||
|
WhisperX URL.
|
||||||
|
- [Production-shaped pipeline](pipeline.production.yml): S3 storage, publish,
|
||||||
|
external tools, and configured Scriptorium artifacts.
|
||||||
|
- [Full annotated pipeline](pipeline.full.annotated.yml): every implemented
|
||||||
|
pipeline section with explanatory comments.
|
||||||
|
- [Extraction subset pipeline](pipeline.extraction-subset.yml): a focused
|
||||||
|
Scriptorium artifact consuming only three declared Notarius lanes.
|
||||||
|
|
||||||
|
The existing `internal/config` example test loads and validates each pipeline
|
||||||
|
with the sample campaign and a compatible local- or S3-audio session.
|
||||||
|
|
||||||
|
## Campaign And Session Configuration
|
||||||
|
|
||||||
|
- [Sample campaign](campaigns/sample-campaign/campaign.yml), its
|
||||||
|
[session template](campaigns/sample-campaign/session.template.yml), and its
|
||||||
|
adjacent stable inputs provide a complete campaign directory shape.
|
||||||
|
- [Local-audio session](session.local-audio.yml) and
|
||||||
|
[S3-audio session](session.s3-audio.yml) are concrete session files.
|
||||||
|
- [Session template](session.template.yml) and the campaign-local equivalent
|
||||||
|
demonstrate the narrow placeholder syntax consumed by `session init`; they
|
||||||
|
are templates, not runtime session files.
|
||||||
|
|
||||||
|
## Input Fixtures
|
||||||
|
|
||||||
|
- [Speakers](speakers.yml), [autocorrect](autocorrect.yml), and
|
||||||
|
[glossary](glossary.yml) show the standalone input shapes.
|
||||||
|
- The sample campaign references its local
|
||||||
|
[speakers](campaigns/sample-campaign/speakers.yml),
|
||||||
|
[autocorrect](campaigns/sample-campaign/autocorrect.yml),
|
||||||
|
[glossary](campaigns/sample-campaign/glossary.yml),
|
||||||
|
[players](campaigns/sample-campaign/players.yml), and
|
||||||
|
[party](campaigns/sample-campaign/party.yml) fixtures.
|
||||||
|
- [Sample speaker audio](audio/sample-speaker.flac) is a text placeholder that
|
||||||
|
reserves the expected filename and directory shape. Replace it with a real
|
||||||
|
FLAC file before running transcription.
|
||||||
|
|
||||||
|
The examples contain environment-variable names but no credential values. They
|
||||||
|
use fictional campaign content and reserved example domains.
|
||||||
@@ -4,3 +4,5 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
|
|||||||
2
examples/campaigns/sample-campaign/party.yml
Normal file
2
examples/campaigns/sample-campaign/party.yml
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
- name: Example Hero
|
||||||
|
type: pc
|
||||||
2
examples/campaigns/sample-campaign/players.yml
Normal file
2
examples/campaigns/sample-campaign/players.yml
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
- name: Example Player
|
||||||
|
role: player
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
match:
|
match:
|
||||||
- speaker: "Eric Rakestraw"
|
- speaker: "Example Speaker"
|
||||||
match:
|
match:
|
||||||
- "Eric_Rakestraw"
|
- "Example_Speaker"
|
||||||
- "Eric"
|
- "Example"
|
||||||
|
|||||||
55
examples/pipeline.extraction-subset.yml
Normal file
55
examples/pipeline.extraction-subset.yml
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
# Purpose-specific extraction example: a Scriptorium session brief consumes
|
||||||
|
# only the three Notarius lanes it needs.
|
||||||
|
|
||||||
|
campaigns:
|
||||||
|
root: /usr/local/share/narratio/campaigns
|
||||||
|
default_campaign_id: sample-campaign
|
||||||
|
|
||||||
|
whisperx:
|
||||||
|
transcribe_url: "https://transcription.example.com/transcribe"
|
||||||
|
|
||||||
|
notarius:
|
||||||
|
enabled: true
|
||||||
|
binary: notarius
|
||||||
|
config_path: /usr/local/etc/notarius/config.yml
|
||||||
|
pipeline_id: dnd-session
|
||||||
|
timeout: 3h
|
||||||
|
outputs:
|
||||||
|
npc_registry:
|
||||||
|
lane_id: npc-registry
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.npc_registry
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/npc-registry
|
||||||
|
location_registry:
|
||||||
|
lane_id: location-registry
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.location_registry
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/location-registry
|
||||||
|
scene_descriptions:
|
||||||
|
lane_id: scene-descriptions
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.scene_descriptions
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/scene-descriptions
|
||||||
|
|
||||||
|
scriptorium:
|
||||||
|
binary: scriptorium
|
||||||
|
config_path: /usr/local/etc/scriptorium/config.yml
|
||||||
|
artifacts:
|
||||||
|
session_brief:
|
||||||
|
enabled: true
|
||||||
|
prompt_id: dnd.session_brief
|
||||||
|
output_path: artifacts/session_brief.md
|
||||||
|
inputs:
|
||||||
|
npcs:
|
||||||
|
source: narratio.extraction.npc_registry
|
||||||
|
required: true
|
||||||
|
locations:
|
||||||
|
source: narratio.extraction.location_registry
|
||||||
|
required: true
|
||||||
|
scenes:
|
||||||
|
source: narratio.extraction.scene_descriptions
|
||||||
|
required: true
|
||||||
|
|
||||||
@@ -48,12 +48,23 @@ publish:
|
|||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
required: true
|
required: true
|
||||||
|
- source: narratio.transcript.final_markdown
|
||||||
|
dest: transcripts/final.md
|
||||||
|
required: true
|
||||||
|
- source: narratio.transcript.final_trimmed_markdown
|
||||||
|
dest: transcripts/final.trimmed.md
|
||||||
|
required: true
|
||||||
- source: narratio.artifact.session_recap
|
- source: narratio.artifact.session_recap
|
||||||
dest: artifacts/session_recap.md
|
dest: artifacts/session_recap.md
|
||||||
required: true
|
required: true
|
||||||
- source: narratio.artifact.player_handout
|
- source: narratio.artifact.player_handout
|
||||||
dest: artifacts/player_handout.md
|
dest: artifacts/player_handout.md
|
||||||
required: false
|
required: false
|
||||||
|
# Extraction lanes publish only when named explicitly; the bundle and index
|
||||||
|
# are never implicit publish sources.
|
||||||
|
- source: narratio.extraction.npc_registry
|
||||||
|
dest: artifacts/extraction/npc-registry.json
|
||||||
|
required: true
|
||||||
|
|
||||||
whisperx:
|
whisperx:
|
||||||
# Required.
|
# Required.
|
||||||
@@ -104,20 +115,91 @@ normalize:
|
|||||||
report: true
|
report: true
|
||||||
|
|
||||||
trim:
|
trim:
|
||||||
# Keep disabled unless bounds prompt integration is configured.
|
# Optional; defaults shown explicitly.
|
||||||
enabled: false
|
enabled: true
|
||||||
output_path: transcripts/final.trimmed.json
|
output_path: transcripts/final.trimmed.json
|
||||||
bounds:
|
bounds:
|
||||||
prompt_id: dnd.session_bounds
|
prompt_id: dnd.session_bounds
|
||||||
profile_id: local-fast
|
profile_id: ""
|
||||||
transcript_input_name: transcript
|
transcript_input_name: transcript
|
||||||
output_path: reports/session_bounds.json
|
output_path: artifacts/session_bounds.json
|
||||||
timeout: 10m
|
timeout: 10m
|
||||||
render_debug: false
|
render_debug: false
|
||||||
render_output_path: reports/session_bounds.render.json
|
|
||||||
seriatim:
|
seriatim:
|
||||||
report: false
|
report: false
|
||||||
|
|
||||||
|
notarius:
|
||||||
|
# Optional structured extraction between trim and render.
|
||||||
|
enabled: true
|
||||||
|
binary: notarius
|
||||||
|
config_path: /usr/local/etc/notarius/config.yml
|
||||||
|
pipeline_id: dnd-session
|
||||||
|
timeout: 3h
|
||||||
|
working_directory: /usr/local/etc/notarius
|
||||||
|
# Each key creates source narratio.extraction.<key>. These constraints match
|
||||||
|
# the current Notarius D&D lane contracts; update them with Notarius.
|
||||||
|
outputs:
|
||||||
|
item_registry:
|
||||||
|
lane_id: item-registry
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.item_registry
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/item-registry
|
||||||
|
npc_registry:
|
||||||
|
lane_id: npc-registry
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.npc_registry
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/npc-registry
|
||||||
|
location_registry:
|
||||||
|
lane_id: location-registry
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.location_registry
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/location-registry
|
||||||
|
scene_descriptions:
|
||||||
|
lane_id: scene-descriptions
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.scene_descriptions
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/scene-descriptions
|
||||||
|
item_occurrences:
|
||||||
|
lane_id: item-occurrences
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.item_occurrences
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/item-occurrences
|
||||||
|
spells:
|
||||||
|
lane_id: spells
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.spells
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/spells
|
||||||
|
combat_turns:
|
||||||
|
lane_id: combat-turns
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.combat_turns
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/combat-turns
|
||||||
|
npc_occurrences:
|
||||||
|
lane_id: npc-occurrences
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.npc_occurrences
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/npc-occurrences
|
||||||
|
location_occurrences:
|
||||||
|
lane_id: location-occurrences
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.location_occurrences
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/location-occurrences
|
||||||
|
enemy_events:
|
||||||
|
lane_id: enemy-events
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: notarius.dnd.enemy_events
|
||||||
|
schema_version: v1
|
||||||
|
module_key: dnd/enemy-events
|
||||||
|
|
||||||
scriptorium:
|
scriptorium:
|
||||||
binary: scriptorium
|
binary: scriptorium
|
||||||
config_path: /usr/local/etc/scriptorium/config.yml
|
config_path: /usr/local/etc/scriptorium/config.yml
|
||||||
@@ -138,6 +220,15 @@ scriptorium:
|
|||||||
previous_recap:
|
previous_recap:
|
||||||
source: narratio.previous_session.artifact.session_recap
|
source: narratio.previous_session.artifact.session_recap
|
||||||
required: false
|
required: false
|
||||||
|
players:
|
||||||
|
source: narratio.input.players
|
||||||
|
required: true
|
||||||
|
party:
|
||||||
|
source: narratio.input.party
|
||||||
|
required: true
|
||||||
|
glossary:
|
||||||
|
source: narratio.input.glossary
|
||||||
|
required: false
|
||||||
vars:
|
vars:
|
||||||
session_id: true
|
session_id: true
|
||||||
session_date: true
|
session_date: true
|
||||||
|
|||||||
@@ -26,6 +26,12 @@ publish:
|
|||||||
- source: narratio.transcript.final_trimmed
|
- source: narratio.transcript.final_trimmed
|
||||||
dest: transcripts/final.trimmed.json
|
dest: transcripts/final.trimmed.json
|
||||||
required: true
|
required: true
|
||||||
|
- source: narratio.transcript.final_markdown
|
||||||
|
dest: transcripts/final.md
|
||||||
|
required: true
|
||||||
|
- source: narratio.transcript.final_trimmed_markdown
|
||||||
|
dest: transcripts/final.trimmed.md
|
||||||
|
required: true
|
||||||
- source: narratio.artifact.session_recap
|
- source: narratio.artifact.session_recap
|
||||||
dest: artifacts/session_recap.md
|
dest: artifacts/session_recap.md
|
||||||
required: true
|
required: true
|
||||||
@@ -65,9 +71,6 @@ normalize:
|
|||||||
output_schema: seriatim-intermediate
|
output_schema: seriatim-intermediate
|
||||||
report: true
|
report: true
|
||||||
|
|
||||||
trim:
|
|
||||||
enabled: false
|
|
||||||
|
|
||||||
scriptorium:
|
scriptorium:
|
||||||
binary: scriptorium
|
binary: scriptorium
|
||||||
config_path: /usr/local/etc/scriptorium/config.yml
|
config_path: /usr/local/etc/scriptorium/config.yml
|
||||||
@@ -87,6 +90,15 @@ scriptorium:
|
|||||||
previous_recap:
|
previous_recap:
|
||||||
source: narratio.previous_session.artifact.session_recap
|
source: narratio.previous_session.artifact.session_recap
|
||||||
required: false
|
required: false
|
||||||
|
players:
|
||||||
|
source: narratio.input.players
|
||||||
|
required: true
|
||||||
|
party:
|
||||||
|
source: narratio.input.party
|
||||||
|
required: true
|
||||||
|
glossary:
|
||||||
|
source: narratio.input.glossary
|
||||||
|
required: false
|
||||||
vars:
|
vars:
|
||||||
session_id: true
|
session_id: true
|
||||||
session_date: true
|
session_date: true
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
match:
|
match:
|
||||||
- speaker: "Eric Rakestraw"
|
- speaker: "Example Speaker"
|
||||||
match:
|
match:
|
||||||
- "Eric_Rakestraw"
|
- "Example_Speaker"
|
||||||
- "Eric"
|
- "Example"
|
||||||
|
|||||||
1
go.mod
1
go.mod
@@ -7,6 +7,7 @@ require (
|
|||||||
github.com/aws/aws-sdk-go-v2/credentials v1.19.16
|
github.com/aws/aws-sdk-go-v2/credentials v1.19.16
|
||||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0
|
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0
|
||||||
github.com/aws/smithy-go v1.25.1
|
github.com/aws/smithy-go v1.25.1
|
||||||
|
golang.org/x/sys v0.47.0
|
||||||
gopkg.in/yaml.v3 v3.0.1
|
gopkg.in/yaml.v3 v3.0.1
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
2
go.sum
2
go.sum
@@ -34,6 +34,8 @@ github.com/aws/aws-sdk-go-v2/service/sts v1.42.1 h1:F/M5Y9I3nwr2IEpshZgh1GeHpOIt
|
|||||||
github.com/aws/aws-sdk-go-v2/service/sts v1.42.1/go.mod h1:mTNxImtovCOEEuD65mKW7DCsL+2gjEH+RPEAexAzAio=
|
github.com/aws/aws-sdk-go-v2/service/sts v1.42.1/go.mod h1:mTNxImtovCOEEuD65mKW7DCsL+2gjEH+RPEAexAzAio=
|
||||||
github.com/aws/smithy-go v1.25.1 h1:J8ERsGSU7d+aCmdQur5Txg6bVoYelvQJgtZehD12GkI=
|
github.com/aws/smithy-go v1.25.1 h1:J8ERsGSU7d+aCmdQur5Txg6bVoYelvQJgtZehD12GkI=
|
||||||
github.com/aws/smithy-go v1.25.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
|
github.com/aws/smithy-go v1.25.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
|
||||||
|
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||||
|
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
|
|||||||
22
internal/adapters/notarius/fake.go
Normal file
22
internal/adapters/notarius/fake.go
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
package notarius
|
||||||
|
|
||||||
|
import "context"
|
||||||
|
|
||||||
|
// FakeRunner is a configurable in-memory runner for stage tests.
|
||||||
|
type FakeRunner struct {
|
||||||
|
Requests []RunRequest
|
||||||
|
Result RunResult
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run records the request and returns the configured result or error.
|
||||||
|
func (f *FakeRunner) Run(ctx context.Context, req RunRequest) (RunResult, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return RunResult{}, err
|
||||||
|
}
|
||||||
|
f.Requests = append(f.Requests, req)
|
||||||
|
if f.Err != nil {
|
||||||
|
return RunResult{}, f.Err
|
||||||
|
}
|
||||||
|
return f.Result, nil
|
||||||
|
}
|
||||||
108
internal/adapters/notarius/runner.go
Normal file
108
internal/adapters/notarius/runner.go
Normal file
@@ -0,0 +1,108 @@
|
|||||||
|
// Package notarius declares the adapter contract for Notarius CLI invocations.
|
||||||
|
package notarius
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const ReceiptSchemaVersion = "notarius.run-result.v1"
|
||||||
|
|
||||||
|
// Runner is the adapter boundary for a complete Notarius pipeline invocation.
|
||||||
|
type Runner interface {
|
||||||
|
Run(ctx context.Context, req RunRequest) (RunResult, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// RunRequest contains the resolved inputs and diagnostic destinations for one invocation.
|
||||||
|
type RunRequest struct {
|
||||||
|
Binary string
|
||||||
|
ConfigPath string
|
||||||
|
PipelineID string
|
||||||
|
InputPath string
|
||||||
|
OutputRoot string
|
||||||
|
WorkingDirectory string
|
||||||
|
ReceiptPath string
|
||||||
|
LogPath string
|
||||||
|
Timeout time.Duration
|
||||||
|
}
|
||||||
|
|
||||||
|
// Receipt is the transport-neutral successful run receipt.
|
||||||
|
type Receipt struct {
|
||||||
|
SchemaVersion string
|
||||||
|
RunID string
|
||||||
|
PipelineID string
|
||||||
|
OutputDirectory string
|
||||||
|
IndexFile string
|
||||||
|
NormalizedOutputCount int
|
||||||
|
RejectedOutputCount int
|
||||||
|
WarningCount int
|
||||||
|
ValidationStatus string
|
||||||
|
DebugDirectory string
|
||||||
|
}
|
||||||
|
|
||||||
|
// LaneDescriptor identifies one normalized lane payload discovered through the index.
|
||||||
|
type LaneDescriptor struct {
|
||||||
|
LaneID string
|
||||||
|
File string
|
||||||
|
Path string
|
||||||
|
MediaType string
|
||||||
|
ModuleKey string
|
||||||
|
SchemaID string
|
||||||
|
SchemaName string
|
||||||
|
SchemaVersion string
|
||||||
|
}
|
||||||
|
|
||||||
|
// PipelineDescriptor identifies a pipeline-wide artifact discovered through the index.
|
||||||
|
type PipelineDescriptor struct {
|
||||||
|
ArtifactKind string
|
||||||
|
File string
|
||||||
|
Path string
|
||||||
|
MediaType string
|
||||||
|
SchemaID string
|
||||||
|
SchemaName string
|
||||||
|
SchemaVersion string
|
||||||
|
}
|
||||||
|
|
||||||
|
// Index describes the validated bundle-management and artifact paths.
|
||||||
|
type Index struct {
|
||||||
|
Path string
|
||||||
|
ManifestFile string
|
||||||
|
ManifestPath string
|
||||||
|
RejectedFile string
|
||||||
|
RejectedPath string
|
||||||
|
WarningsFile string
|
||||||
|
WarningsPath string
|
||||||
|
Lanes []LaneDescriptor
|
||||||
|
ChunkMap *PipelineDescriptor
|
||||||
|
EvidenceContext *PipelineDescriptor
|
||||||
|
}
|
||||||
|
|
||||||
|
// RejectionSummary retains structured rejection identity without free-form messages.
|
||||||
|
type RejectionSummary struct {
|
||||||
|
Stage string
|
||||||
|
StepID string
|
||||||
|
LaneID string
|
||||||
|
ModuleKey string
|
||||||
|
ChunkID string
|
||||||
|
ValidatorName string
|
||||||
|
ReasonCode string
|
||||||
|
}
|
||||||
|
|
||||||
|
// WarningSummary retains structured warning identity without free-form messages.
|
||||||
|
type WarningSummary struct {
|
||||||
|
Scope string
|
||||||
|
ReasonCode string
|
||||||
|
}
|
||||||
|
|
||||||
|
// RunResult describes a successfully decoded and validated Notarius bundle.
|
||||||
|
type RunResult struct {
|
||||||
|
Receipt Receipt
|
||||||
|
Index Index
|
||||||
|
BundleRoot string
|
||||||
|
ReceiptPath string
|
||||||
|
LogPath string
|
||||||
|
ExitCode int
|
||||||
|
Duration time.Duration
|
||||||
|
Rejections []RejectionSummary
|
||||||
|
Warnings []WarningSummary
|
||||||
|
}
|
||||||
524
internal/adapters/notarius/subprocess.go
Normal file
524
internal/adapters/notarius/subprocess.go
Normal file
@@ -0,0 +1,524 @@
|
|||||||
|
package notarius
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/subprocess"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/pathsafe"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
maxReceiptBytes = 1 << 20
|
||||||
|
maxIndexBytes = 4 << 20
|
||||||
|
maxSummaryBytes = 4 << 20
|
||||||
|
canonicalIndexFile = "index.json"
|
||||||
|
canonicalManifestFile = "manifest.json"
|
||||||
|
canonicalRejectedFile = "rejected.json"
|
||||||
|
canonicalWarningsFile = "warnings.json"
|
||||||
|
)
|
||||||
|
|
||||||
|
type subprocessRun func(context.Context, subprocess.RunRequest) (subprocess.RunResult, error)
|
||||||
|
|
||||||
|
// SubprocessRunner invokes Notarius through its public CLI.
|
||||||
|
type SubprocessRunner struct {
|
||||||
|
run subprocessRun
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewSubprocessRunner constructs a production Notarius subprocess runner.
|
||||||
|
func NewSubprocessRunner() *SubprocessRunner {
|
||||||
|
return &SubprocessRunner{run: subprocess.Run}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Run executes a complete Notarius pipeline and discovers its published bundle.
|
||||||
|
func (r *SubprocessRunner) Run(ctx context.Context, req RunRequest) (RunResult, error) {
|
||||||
|
if r == nil || r.run == nil {
|
||||||
|
return RunResult{}, fmt.Errorf("notarius subprocess runner is nil")
|
||||||
|
}
|
||||||
|
if err := validateRunRequest(req); err != nil {
|
||||||
|
return RunResult{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
args := []string{
|
||||||
|
"run", req.PipelineID,
|
||||||
|
"--config", req.ConfigPath,
|
||||||
|
"--input", req.InputPath,
|
||||||
|
"--output-dir", req.OutputRoot,
|
||||||
|
"--json",
|
||||||
|
}
|
||||||
|
processResult, err := r.run(ctx, subprocess.RunRequest{
|
||||||
|
Executable: req.Binary,
|
||||||
|
Args: args,
|
||||||
|
WorkingDir: req.WorkingDirectory,
|
||||||
|
Timeout: req.Timeout,
|
||||||
|
StdoutLogPath: req.ReceiptPath,
|
||||||
|
StderrLogPath: req.LogPath,
|
||||||
|
})
|
||||||
|
baseResult := RunResult{
|
||||||
|
ReceiptPath: req.ReceiptPath,
|
||||||
|
LogPath: req.LogPath,
|
||||||
|
ExitCode: processResult.ExitCode,
|
||||||
|
Duration: processResult.Duration,
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return baseResult, fmt.Errorf("run notarius pipeline %q: %w", req.PipelineID, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
receipt, err := loadReceipt(req.ReceiptPath, req.PipelineID)
|
||||||
|
if err != nil {
|
||||||
|
return baseResult, err
|
||||||
|
}
|
||||||
|
bundleRoot, err := validateBundleRoot(req.OutputRoot, receipt.OutputDirectory)
|
||||||
|
if err != nil {
|
||||||
|
return baseResult, err
|
||||||
|
}
|
||||||
|
indexPath, err := resolveRegularFile(bundleRoot, receipt.IndexFile)
|
||||||
|
if err != nil {
|
||||||
|
return baseResult, fmt.Errorf("resolve receipt index file: %w", err)
|
||||||
|
}
|
||||||
|
index, err := loadIndex(bundleRoot, indexPath)
|
||||||
|
if err != nil {
|
||||||
|
return baseResult, err
|
||||||
|
}
|
||||||
|
rejections, err := loadRejections(index.RejectedPath)
|
||||||
|
if err != nil {
|
||||||
|
return baseResult, err
|
||||||
|
}
|
||||||
|
warnings, err := loadWarnings(index.WarningsPath)
|
||||||
|
if err != nil {
|
||||||
|
return baseResult, err
|
||||||
|
}
|
||||||
|
|
||||||
|
baseResult.Receipt = receipt
|
||||||
|
baseResult.Index = index
|
||||||
|
baseResult.BundleRoot = bundleRoot
|
||||||
|
baseResult.Rejections = rejections
|
||||||
|
baseResult.Warnings = warnings
|
||||||
|
return baseResult, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateRunRequest(req RunRequest) error {
|
||||||
|
if strings.TrimSpace(req.Binary) == "" {
|
||||||
|
return fmt.Errorf("notarius binary is required")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(req.PipelineID) == "" {
|
||||||
|
return fmt.Errorf("notarius pipeline id is required")
|
||||||
|
}
|
||||||
|
if req.Timeout <= 0 {
|
||||||
|
return fmt.Errorf("notarius timeout must be positive")
|
||||||
|
}
|
||||||
|
for label, path := range map[string]string{
|
||||||
|
"config": req.ConfigPath,
|
||||||
|
"input": req.InputPath,
|
||||||
|
"output root": req.OutputRoot,
|
||||||
|
"working directory": req.WorkingDirectory,
|
||||||
|
"receipt": req.ReceiptPath,
|
||||||
|
"log": req.LogPath,
|
||||||
|
} {
|
||||||
|
if strings.TrimSpace(path) == "" {
|
||||||
|
return fmt.Errorf("notarius %s path is required", label)
|
||||||
|
}
|
||||||
|
if !filepath.IsAbs(path) {
|
||||||
|
return fmt.Errorf("notarius %s path must be absolute", label)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if filepath.Clean(req.ReceiptPath) == filepath.Clean(req.LogPath) {
|
||||||
|
return fmt.Errorf("notarius receipt and log paths must be different")
|
||||||
|
}
|
||||||
|
if err := requireRegularFile(req.ConfigPath); err != nil {
|
||||||
|
return fmt.Errorf("validate notarius config path: %w", err)
|
||||||
|
}
|
||||||
|
if err := requireRegularFile(req.InputPath); err != nil {
|
||||||
|
return fmt.Errorf("validate notarius input path: %w", err)
|
||||||
|
}
|
||||||
|
if err := requireDirectory(req.OutputRoot); err != nil {
|
||||||
|
return fmt.Errorf("validate notarius output root: %w", err)
|
||||||
|
}
|
||||||
|
if err := requireDirectory(req.WorkingDirectory); err != nil {
|
||||||
|
return fmt.Errorf("validate notarius working directory: %w", err)
|
||||||
|
}
|
||||||
|
if err := validateLogDestination(req.ReceiptPath); err != nil {
|
||||||
|
return fmt.Errorf("validate notarius receipt path: %w", err)
|
||||||
|
}
|
||||||
|
if err := validateLogDestination(req.LogPath); err != nil {
|
||||||
|
return fmt.Errorf("validate notarius log path: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type receiptDocument struct {
|
||||||
|
SchemaVersion string `json:"schema_version"`
|
||||||
|
RunID string `json:"run_id"`
|
||||||
|
PipelineID string `json:"pipeline_id"`
|
||||||
|
OutputDirectory string `json:"output_directory"`
|
||||||
|
IndexFile string `json:"index_file"`
|
||||||
|
NormalizedOutputCount *int `json:"normalized_output_count"`
|
||||||
|
RejectedOutputCount *int `json:"rejected_output_count"`
|
||||||
|
WarningCount *int `json:"warning_count"`
|
||||||
|
ValidationStatus string `json:"validation_status"`
|
||||||
|
DebugDirectory string `json:"debug_directory"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadReceipt(path, pipelineID string) (Receipt, error) {
|
||||||
|
var document receiptDocument
|
||||||
|
if err := decodeBoundedJSON(path, maxReceiptBytes, &document); err != nil {
|
||||||
|
return Receipt{}, fmt.Errorf("decode notarius receipt: %w", err)
|
||||||
|
}
|
||||||
|
if document.SchemaVersion != ReceiptSchemaVersion {
|
||||||
|
return Receipt{}, fmt.Errorf("unsupported notarius receipt schema version %q", document.SchemaVersion)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(document.RunID) == "" || strings.TrimSpace(document.PipelineID) == "" ||
|
||||||
|
strings.TrimSpace(document.OutputDirectory) == "" || strings.TrimSpace(document.ValidationStatus) == "" ||
|
||||||
|
document.NormalizedOutputCount == nil ||
|
||||||
|
document.RejectedOutputCount == nil || document.WarningCount == nil {
|
||||||
|
return Receipt{}, fmt.Errorf("notarius receipt is missing required fields")
|
||||||
|
}
|
||||||
|
if document.IndexFile != canonicalIndexFile {
|
||||||
|
return Receipt{}, fmt.Errorf("notarius receipt index_file %q is incompatible; want %q", document.IndexFile, canonicalIndexFile)
|
||||||
|
}
|
||||||
|
if document.PipelineID != pipelineID {
|
||||||
|
return Receipt{}, fmt.Errorf("notarius receipt pipeline id %q does not match requested pipeline %q", document.PipelineID, pipelineID)
|
||||||
|
}
|
||||||
|
if *document.NormalizedOutputCount < 0 || *document.RejectedOutputCount < 0 || *document.WarningCount < 0 {
|
||||||
|
return Receipt{}, fmt.Errorf("notarius receipt counts must be non-negative")
|
||||||
|
}
|
||||||
|
if !filepath.IsAbs(document.OutputDirectory) {
|
||||||
|
return Receipt{}, fmt.Errorf("notarius receipt output directory must be absolute")
|
||||||
|
}
|
||||||
|
if document.DebugDirectory != "" && !filepath.IsAbs(document.DebugDirectory) {
|
||||||
|
return Receipt{}, fmt.Errorf("notarius receipt debug directory must be absolute when present")
|
||||||
|
}
|
||||||
|
return Receipt{
|
||||||
|
SchemaVersion: document.SchemaVersion,
|
||||||
|
RunID: document.RunID,
|
||||||
|
PipelineID: document.PipelineID,
|
||||||
|
OutputDirectory: filepath.Clean(document.OutputDirectory),
|
||||||
|
IndexFile: document.IndexFile,
|
||||||
|
NormalizedOutputCount: *document.NormalizedOutputCount,
|
||||||
|
RejectedOutputCount: *document.RejectedOutputCount,
|
||||||
|
WarningCount: *document.WarningCount,
|
||||||
|
ValidationStatus: document.ValidationStatus,
|
||||||
|
DebugDirectory: document.DebugDirectory,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type indexDocument struct {
|
||||||
|
ManifestFile string `json:"manifest_file"`
|
||||||
|
OutputFiles *[]laneDocument `json:"output_files"`
|
||||||
|
RejectedFile string `json:"rejected_file"`
|
||||||
|
WarningsFile string `json:"warnings_file"`
|
||||||
|
ChunkMap *pipelineDocument `json:"chunk_map"`
|
||||||
|
EvidenceContext *pipelineDocument `json:"evidence_context"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type laneDocument struct {
|
||||||
|
LaneID string `json:"lane_id"`
|
||||||
|
File string `json:"file"`
|
||||||
|
MediaType string `json:"media_type"`
|
||||||
|
ModuleKey string `json:"module_key"`
|
||||||
|
SchemaID string `json:"schema_id"`
|
||||||
|
SchemaName string `json:"schema_name"`
|
||||||
|
SchemaVersion string `json:"schema_version"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type pipelineDocument struct {
|
||||||
|
ArtifactKind string `json:"artifact_kind"`
|
||||||
|
File string `json:"file"`
|
||||||
|
MediaType string `json:"media_type"`
|
||||||
|
SchemaID string `json:"schema_id"`
|
||||||
|
SchemaName string `json:"schema_name"`
|
||||||
|
SchemaVersion string `json:"schema_version"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadIndex(bundleRoot, indexPath string) (Index, error) {
|
||||||
|
var document indexDocument
|
||||||
|
if err := decodeBoundedJSON(indexPath, maxIndexBytes, &document); err != nil {
|
||||||
|
return Index{}, fmt.Errorf("decode notarius index: %w", err)
|
||||||
|
}
|
||||||
|
for _, field := range []struct {
|
||||||
|
name string
|
||||||
|
got string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "manifest_file", got: document.ManifestFile, want: canonicalManifestFile},
|
||||||
|
{name: "rejected_file", got: document.RejectedFile, want: canonicalRejectedFile},
|
||||||
|
{name: "warnings_file", got: document.WarningsFile, want: canonicalWarningsFile},
|
||||||
|
} {
|
||||||
|
if field.got != field.want {
|
||||||
|
return Index{}, fmt.Errorf("notarius index %s %q is incompatible; want %q", field.name, field.got, field.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if document.OutputFiles == nil {
|
||||||
|
return Index{}, fmt.Errorf("notarius index is missing required output_files")
|
||||||
|
}
|
||||||
|
|
||||||
|
index := Index{
|
||||||
|
Path: indexPath,
|
||||||
|
ManifestFile: document.ManifestFile,
|
||||||
|
RejectedFile: document.RejectedFile,
|
||||||
|
WarningsFile: document.WarningsFile,
|
||||||
|
}
|
||||||
|
var err error
|
||||||
|
if index.ManifestPath, err = resolveRegularFile(bundleRoot, index.ManifestFile); err != nil {
|
||||||
|
return Index{}, fmt.Errorf("resolve notarius manifest file: %w", err)
|
||||||
|
}
|
||||||
|
if index.RejectedPath, err = resolveRegularFile(bundleRoot, index.RejectedFile); err != nil {
|
||||||
|
return Index{}, fmt.Errorf("resolve notarius rejection file: %w", err)
|
||||||
|
}
|
||||||
|
if index.WarningsPath, err = resolveRegularFile(bundleRoot, index.WarningsFile); err != nil {
|
||||||
|
return Index{}, fmt.Errorf("resolve notarius warning file: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
seenLanes := make(map[string]struct{}, len(*document.OutputFiles))
|
||||||
|
for _, lane := range *document.OutputFiles {
|
||||||
|
if strings.TrimSpace(lane.LaneID) == "" || strings.TrimSpace(lane.File) == "" {
|
||||||
|
return Index{}, fmt.Errorf("notarius lane descriptors require lane_id and file")
|
||||||
|
}
|
||||||
|
if _, exists := seenLanes[lane.LaneID]; exists {
|
||||||
|
return Index{}, fmt.Errorf("notarius index contains duplicate lane id %q", lane.LaneID)
|
||||||
|
}
|
||||||
|
seenLanes[lane.LaneID] = struct{}{}
|
||||||
|
path, err := resolveRegularFile(bundleRoot, lane.File)
|
||||||
|
if err != nil {
|
||||||
|
return Index{}, fmt.Errorf("resolve notarius lane %q file: %w", lane.LaneID, err)
|
||||||
|
}
|
||||||
|
index.Lanes = append(index.Lanes, LaneDescriptor{
|
||||||
|
LaneID: lane.LaneID, File: lane.File, Path: path, MediaType: lane.MediaType,
|
||||||
|
ModuleKey: lane.ModuleKey, SchemaID: lane.SchemaID, SchemaName: lane.SchemaName,
|
||||||
|
SchemaVersion: lane.SchemaVersion,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if document.ChunkMap != nil {
|
||||||
|
index.ChunkMap, err = resolvePipelineDescriptor(bundleRoot, "chunk_map", *document.ChunkMap)
|
||||||
|
if err != nil {
|
||||||
|
return Index{}, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if document.EvidenceContext != nil {
|
||||||
|
index.EvidenceContext, err = resolvePipelineDescriptor(bundleRoot, "evidence_context", *document.EvidenceContext)
|
||||||
|
if err != nil {
|
||||||
|
return Index{}, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return index, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolvePipelineDescriptor(bundleRoot, label string, document pipelineDocument) (*PipelineDescriptor, error) {
|
||||||
|
if strings.TrimSpace(document.ArtifactKind) == "" || strings.TrimSpace(document.File) == "" ||
|
||||||
|
strings.TrimSpace(document.MediaType) == "" || strings.TrimSpace(document.SchemaID) == "" ||
|
||||||
|
strings.TrimSpace(document.SchemaName) == "" || strings.TrimSpace(document.SchemaVersion) == "" {
|
||||||
|
return nil, fmt.Errorf("notarius %s descriptor is missing required fields", label)
|
||||||
|
}
|
||||||
|
path, err := resolveRegularFile(bundleRoot, document.File)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("resolve notarius %s file: %w", label, err)
|
||||||
|
}
|
||||||
|
return &PipelineDescriptor{
|
||||||
|
ArtifactKind: document.ArtifactKind, File: document.File, Path: path,
|
||||||
|
MediaType: document.MediaType, SchemaID: document.SchemaID,
|
||||||
|
SchemaName: document.SchemaName, SchemaVersion: document.SchemaVersion,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type rejectionDocument struct {
|
||||||
|
Rejected *[]struct {
|
||||||
|
Stage string `json:"stage"`
|
||||||
|
StepID string `json:"step_id"`
|
||||||
|
LaneID string `json:"lane_id"`
|
||||||
|
ModuleKey string `json:"module_key"`
|
||||||
|
ChunkID string `json:"chunk_id"`
|
||||||
|
ValidatorName string `json:"validator_name"`
|
||||||
|
ReasonCode string `json:"reason_code"`
|
||||||
|
Message string `json:"message"`
|
||||||
|
} `json:"rejected"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadRejections(path string) ([]RejectionSummary, error) {
|
||||||
|
var document rejectionDocument
|
||||||
|
if err := decodeBoundedJSON(path, maxSummaryBytes, &document); err != nil {
|
||||||
|
return nil, fmt.Errorf("decode notarius rejections: %w", err)
|
||||||
|
}
|
||||||
|
if document.Rejected == nil {
|
||||||
|
return nil, fmt.Errorf("notarius rejection document is missing rejected array")
|
||||||
|
}
|
||||||
|
summaries := make([]RejectionSummary, 0, len(*document.Rejected))
|
||||||
|
for _, item := range *document.Rejected {
|
||||||
|
if strings.TrimSpace(item.Stage) == "" || strings.TrimSpace(item.Message) == "" {
|
||||||
|
return nil, fmt.Errorf("notarius rejection entries require stage and message")
|
||||||
|
}
|
||||||
|
summaries = append(summaries, RejectionSummary{
|
||||||
|
Stage: item.Stage, StepID: item.StepID, LaneID: item.LaneID,
|
||||||
|
ModuleKey: item.ModuleKey, ChunkID: item.ChunkID,
|
||||||
|
ValidatorName: item.ValidatorName, ReasonCode: item.ReasonCode,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return summaries, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type warningDocument struct {
|
||||||
|
Warnings *[]struct {
|
||||||
|
Scope string `json:"scope"`
|
||||||
|
ReasonCode string `json:"reason_code"`
|
||||||
|
Message string `json:"message"`
|
||||||
|
} `json:"warnings"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadWarnings(path string) ([]WarningSummary, error) {
|
||||||
|
var document warningDocument
|
||||||
|
if err := decodeBoundedJSON(path, maxSummaryBytes, &document); err != nil {
|
||||||
|
return nil, fmt.Errorf("decode notarius warnings: %w", err)
|
||||||
|
}
|
||||||
|
if document.Warnings == nil {
|
||||||
|
return nil, fmt.Errorf("notarius warning document is missing warnings array")
|
||||||
|
}
|
||||||
|
summaries := make([]WarningSummary, 0, len(*document.Warnings))
|
||||||
|
for _, item := range *document.Warnings {
|
||||||
|
if strings.TrimSpace(item.ReasonCode) == "" || strings.TrimSpace(item.Message) == "" {
|
||||||
|
return nil, fmt.Errorf("notarius warning entries require reason_code and message")
|
||||||
|
}
|
||||||
|
summaries = append(summaries, WarningSummary{Scope: item.Scope, ReasonCode: item.ReasonCode})
|
||||||
|
}
|
||||||
|
return summaries, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeBoundedJSON(path string, limit int64, destination any) error {
|
||||||
|
inspected, err := os.Lstat(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if inspected.Mode()&os.ModeSymlink != 0 || !inspected.Mode().IsRegular() {
|
||||||
|
return fmt.Errorf("path %q must be a regular file without symlinks", path)
|
||||||
|
}
|
||||||
|
file, err := os.Open(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer func() { _ = file.Close() }()
|
||||||
|
opened, err := file.Stat()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !opened.Mode().IsRegular() || !os.SameFile(inspected, opened) {
|
||||||
|
return fmt.Errorf("file %q changed before it could be read", path)
|
||||||
|
}
|
||||||
|
reader := io.LimitReader(file, limit+1)
|
||||||
|
data, err := io.ReadAll(reader)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if int64(len(data)) > limit {
|
||||||
|
return fmt.Errorf("file %q exceeds %d-byte limit", path, limit)
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(data, destination); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateBundleRoot(outputRoot, bundleRoot string) (string, error) {
|
||||||
|
root := filepath.Clean(outputRoot)
|
||||||
|
bundle := filepath.Clean(bundleRoot)
|
||||||
|
relative, err := filepath.Rel(root, bundle)
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("compare notarius output paths: %w", err)
|
||||||
|
}
|
||||||
|
if relative == "." || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
|
||||||
|
return "", fmt.Errorf("notarius output directory %q is not beneath output root %q", bundleRoot, outputRoot)
|
||||||
|
}
|
||||||
|
if err := requireDirectoryTree(root, relative); err != nil {
|
||||||
|
return "", fmt.Errorf("validate notarius output directory: %w", err)
|
||||||
|
}
|
||||||
|
return bundle, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveRegularFile(root, logicalPath string) (string, error) {
|
||||||
|
resolved, err := pathsafe.JoinSlashRelativeUnderRoot(root, logicalPath)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
relative, err := filepath.Rel(root, resolved)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
if err := requireRegularFileTree(root, relative); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return resolved, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireDirectoryTree(root, relative string) error {
|
||||||
|
if err := requireDirectory(root); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
current := root
|
||||||
|
for _, component := range strings.Split(relative, string(filepath.Separator)) {
|
||||||
|
current = filepath.Join(current, component)
|
||||||
|
if err := requireDirectory(current); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireRegularFileTree(root, relative string) error {
|
||||||
|
components := strings.Split(relative, string(filepath.Separator))
|
||||||
|
if len(components) == 0 {
|
||||||
|
return fmt.Errorf("regular file path is required")
|
||||||
|
}
|
||||||
|
if err := requireDirectory(root); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
current := root
|
||||||
|
for _, component := range components[:len(components)-1] {
|
||||||
|
current = filepath.Join(current, component)
|
||||||
|
if err := requireDirectory(current); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return requireRegularFile(filepath.Join(current, components[len(components)-1]))
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireDirectory(path string) error {
|
||||||
|
info, err := os.Lstat(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 || !info.IsDir() {
|
||||||
|
return fmt.Errorf("path %q must be a directory without symlinks", path)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireRegularFile(path string) error {
|
||||||
|
info, err := os.Lstat(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 || !info.Mode().IsRegular() {
|
||||||
|
return fmt.Errorf("path %q must be a regular file without symlinks", path)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateLogDestination(path string) error {
|
||||||
|
if err := requireDirectory(filepath.Dir(path)); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
info, err := os.Lstat(path)
|
||||||
|
if errors.Is(err, os.ErrNotExist) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if info.Mode()&os.ModeSymlink != 0 || !info.Mode().IsRegular() {
|
||||||
|
return fmt.Errorf("path %q must be absent or a regular file without symlinks", path)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
572
internal/adapters/notarius/subprocess_test.go
Normal file
572
internal/adapters/notarius/subprocess_test.go
Normal file
@@ -0,0 +1,572 @@
|
|||||||
|
package notarius
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
sharedsubprocess "gitea.maximumdirect.net/eric/narratio/internal/adapters/subprocess"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestSubprocessRunnerBuildsExactInvocationAndDiscoversBundle(t *testing.T) {
|
||||||
|
req := validRunRequest(t)
|
||||||
|
var captured sharedsubprocess.RunRequest
|
||||||
|
runner := &SubprocessRunner{run: func(_ context.Context, processReq sharedsubprocess.RunRequest) (sharedsubprocess.RunResult, error) {
|
||||||
|
captured = processReq
|
||||||
|
writeValidBundleAndReceipt(t, req, true)
|
||||||
|
return sharedsubprocess.RunResult{ExitCode: 0, Duration: 2 * time.Second}, nil
|
||||||
|
}}
|
||||||
|
|
||||||
|
result, err := runner.Run(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
wantArgs := []string{
|
||||||
|
"run", "dnd-session", "--config", req.ConfigPath, "--input", req.InputPath,
|
||||||
|
"--output-dir", req.OutputRoot, "--json",
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(captured.Args, wantArgs) {
|
||||||
|
t.Fatalf("subprocess args = %#v, want %#v", captured.Args, wantArgs)
|
||||||
|
}
|
||||||
|
if captured.Executable != req.Binary || captured.WorkingDir != req.WorkingDirectory || captured.Timeout != req.Timeout {
|
||||||
|
t.Fatalf("subprocess request = %#v", captured)
|
||||||
|
}
|
||||||
|
if captured.StdoutLogPath != req.ReceiptPath || captured.StderrLogPath != req.LogPath {
|
||||||
|
t.Fatalf("stream paths = stdout %q stderr %q", captured.StdoutLogPath, captured.StderrLogPath)
|
||||||
|
}
|
||||||
|
if captured.EnvOverrides != nil {
|
||||||
|
t.Fatalf("environment overrides = %#v, want inherited environment only", captured.EnvOverrides)
|
||||||
|
}
|
||||||
|
for _, arg := range captured.Args {
|
||||||
|
if arg == "--session-id" {
|
||||||
|
t.Fatal("subprocess args unexpectedly contain --session-id")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if result.Receipt.SchemaVersion != ReceiptSchemaVersion || result.Receipt.RunID != "notarius-run-1" {
|
||||||
|
t.Fatalf("receipt = %#v", result.Receipt)
|
||||||
|
}
|
||||||
|
if len(result.Index.Lanes) != 1 || result.Index.Lanes[0].LaneID != "npc-registry" {
|
||||||
|
t.Fatalf("lanes = %#v", result.Index.Lanes)
|
||||||
|
}
|
||||||
|
if result.Index.ChunkMap == nil || result.Index.ChunkMap.ArtifactKind != "chunk_map" {
|
||||||
|
t.Fatalf("chunk map = %#v", result.Index.ChunkMap)
|
||||||
|
}
|
||||||
|
if result.Index.EvidenceContext == nil || result.Index.EvidenceContext.ArtifactKind != "evidence_context" {
|
||||||
|
t.Fatalf("evidence context = %#v", result.Index.EvidenceContext)
|
||||||
|
}
|
||||||
|
if len(result.Rejections) != 1 || result.Rejections[0].LaneID != "spells" || result.Rejections[0].ReasonCode != "invalid_spell" {
|
||||||
|
t.Fatalf("rejections = %#v", result.Rejections)
|
||||||
|
}
|
||||||
|
if len(result.Warnings) != 1 || result.Warnings[0].Scope != "lane:npc-registry" || result.Warnings[0].ReasonCode != "normalized_name" {
|
||||||
|
t.Fatalf("warnings = %#v", result.Warnings)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerInheritsEnvironmentAndSeparatesStreams(t *testing.T) {
|
||||||
|
req := validRunRequest(t)
|
||||||
|
writeValidBundleAndReceipt(t, req, false)
|
||||||
|
receiptFixture := req.ReceiptPath + ".fixture"
|
||||||
|
data, err := os.ReadFile(req.ReceiptPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(receipt) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(receiptFixture, data, 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(receipt fixture) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.Remove(req.ReceiptPath); err != nil {
|
||||||
|
t.Fatalf("Remove(receipt) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
captureDir := filepath.Join(filepath.Dir(req.ReceiptPath), "capture")
|
||||||
|
if err := os.Mkdir(captureDir, 0o755); err != nil {
|
||||||
|
t.Fatalf("Mkdir(capture) error = %v", err)
|
||||||
|
}
|
||||||
|
script := writeShellScript(t, `#!/bin/sh
|
||||||
|
pwd > "$NOTARIUS_CAPTURE_DIR/working-directory"
|
||||||
|
printf '%s' "$NOTARIUS_INHERITED_VALUE" > "$NOTARIUS_CAPTURE_DIR/environment"
|
||||||
|
printf 'diagnostic stream\n' >&2
|
||||||
|
cat "$NOTARIUS_RECEIPT_FIXTURE"
|
||||||
|
`)
|
||||||
|
req.Binary = script
|
||||||
|
t.Setenv("NOTARIUS_CAPTURE_DIR", captureDir)
|
||||||
|
t.Setenv("NOTARIUS_INHERITED_VALUE", "inherited-value")
|
||||||
|
t.Setenv("NOTARIUS_RECEIPT_FIXTURE", receiptFixture)
|
||||||
|
|
||||||
|
if _, err := NewSubprocessRunner().Run(context.Background(), req); err != nil {
|
||||||
|
t.Fatalf("Run() error = %v", err)
|
||||||
|
}
|
||||||
|
assertTextFile(t, filepath.Join(captureDir, "working-directory"), req.WorkingDirectory+"\n")
|
||||||
|
assertTextFile(t, filepath.Join(captureDir, "environment"), "inherited-value")
|
||||||
|
assertTextFile(t, req.LogPath, "diagnostic stream\n")
|
||||||
|
receiptBytes, err := os.ReadFile(req.ReceiptPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(receipt) error = %v", err)
|
||||||
|
}
|
||||||
|
if strings.Contains(string(receiptBytes), "diagnostic stream") {
|
||||||
|
t.Fatal("receipt contains stderr output")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerReturnsProcessFailuresWithoutParsingStdout(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
scriptBody string
|
||||||
|
timeout time.Duration
|
||||||
|
cancel bool
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "nonzero", scriptBody: "printf '{malformed receipt'; printf 'failed\\n' >&2; exit 7\n", timeout: time.Second, want: "exit code 7"},
|
||||||
|
{name: "timeout", scriptBody: "sleep 5\n", timeout: 20 * time.Millisecond, want: "timed out"},
|
||||||
|
{name: "cancellation", scriptBody: "sleep 5\n", timeout: time.Second, cancel: true, want: "canceled"},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
req := validRunRequest(t)
|
||||||
|
req.Binary = writeShellScript(t, "#!/bin/sh\n"+test.scriptBody)
|
||||||
|
req.Timeout = test.timeout
|
||||||
|
ctx := context.Background()
|
||||||
|
if test.cancel {
|
||||||
|
cancelCtx, cancel := context.WithCancel(ctx)
|
||||||
|
ctx = cancelCtx
|
||||||
|
time.AfterFunc(20*time.Millisecond, cancel)
|
||||||
|
}
|
||||||
|
_, err := NewSubprocessRunner().Run(ctx, req)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||||
|
t.Fatalf("Run() error = %v, want fragment %q", err, test.want)
|
||||||
|
}
|
||||||
|
if strings.Contains(err.Error(), "decode notarius receipt") {
|
||||||
|
t.Fatalf("Run() parsed stdout after process failure: %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerReturnsSharedSubprocessErrorWithoutReadingReceipt(t *testing.T) {
|
||||||
|
req := validRunRequest(t)
|
||||||
|
if err := os.WriteFile(req.ReceiptPath, []byte("not json"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(receipt) error = %v", err)
|
||||||
|
}
|
||||||
|
wantErr := errors.New("process failed")
|
||||||
|
runner := &SubprocessRunner{run: func(context.Context, sharedsubprocess.RunRequest) (sharedsubprocess.RunResult, error) {
|
||||||
|
return sharedsubprocess.RunResult{ExitCode: 9}, wantErr
|
||||||
|
}}
|
||||||
|
_, err := runner.Run(context.Background(), req)
|
||||||
|
if !errors.Is(err, wantErr) {
|
||||||
|
t.Fatalf("Run() error = %v, want wrapped process error", err)
|
||||||
|
}
|
||||||
|
if strings.Contains(err.Error(), "decode") {
|
||||||
|
t.Fatalf("Run() parsed receipt after failure: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadReceiptValidation(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
valid := map[string]any{
|
||||||
|
"schema_version": ReceiptSchemaVersion, "run_id": "run-1", "pipeline_id": "pipeline-1",
|
||||||
|
"output_directory": filepath.Join(root, "outputs", "run-1"), "index_file": "index.json",
|
||||||
|
"normalized_output_count": 1, "rejected_output_count": 0, "warning_count": 0,
|
||||||
|
"validation_status": "approved", "future_field": true,
|
||||||
|
}
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
mutate func(map[string]any)
|
||||||
|
raw []byte
|
||||||
|
wantOK bool
|
||||||
|
wantError string
|
||||||
|
}{
|
||||||
|
{name: "unknown fields tolerated", wantOK: true},
|
||||||
|
{name: "malformed", raw: []byte("{")},
|
||||||
|
{name: "unsupported version", mutate: func(v map[string]any) { v["schema_version"] = "notarius.run-result.v2" }},
|
||||||
|
{name: "missing field", mutate: func(v map[string]any) { delete(v, "run_id") }},
|
||||||
|
{name: "pipeline mismatch", mutate: func(v map[string]any) { v["pipeline_id"] = "other" }},
|
||||||
|
{name: "relative output", mutate: func(v map[string]any) { v["output_directory"] = "run-1" }},
|
||||||
|
{name: "negative count", mutate: func(v map[string]any) { v["warning_count"] = -1 }},
|
||||||
|
{
|
||||||
|
name: "nested index", mutate: func(v map[string]any) { v["index_file"] = "nested/index.json" },
|
||||||
|
wantError: `index_file "nested/index.json"`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "cleanable index", mutate: func(v map[string]any) { v["index_file"] = "./index.json" },
|
||||||
|
wantError: `index_file "./index.json"`,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
path := filepath.Join(root, strings.ReplaceAll(test.name, " ", "-")+".json")
|
||||||
|
values := cloneMap(valid)
|
||||||
|
if test.mutate != nil {
|
||||||
|
test.mutate(values)
|
||||||
|
}
|
||||||
|
if test.raw != nil {
|
||||||
|
if err := os.WriteFile(path, test.raw, 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile() error = %v", err)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
writeJSONFile(t, path, values)
|
||||||
|
}
|
||||||
|
_, err := loadReceipt(path, "pipeline-1")
|
||||||
|
if test.wantOK && err != nil {
|
||||||
|
t.Fatalf("loadReceipt() error = %v", err)
|
||||||
|
}
|
||||||
|
if !test.wantOK && err == nil {
|
||||||
|
t.Fatal("loadReceipt() error = nil, want validation failure")
|
||||||
|
}
|
||||||
|
if test.wantError != "" && !strings.Contains(err.Error(), test.wantError) {
|
||||||
|
t.Fatalf("loadReceipt() error = %v, want fragment %q", err, test.wantError)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
oversized := filepath.Join(root, "oversized.json")
|
||||||
|
if err := os.WriteFile(oversized, []byte(strings.Repeat("x", maxReceiptBytes+1)), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(oversized) error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := loadReceipt(oversized, "pipeline-1"); err == nil || !strings.Contains(err.Error(), "exceeds") {
|
||||||
|
t.Fatalf("loadReceipt(oversized) error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateBundleRootRejectsEscapesAndSymlinks(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
outputRoot := filepath.Join(root, "output")
|
||||||
|
if err := os.Mkdir(outputRoot, 0o755); err != nil {
|
||||||
|
t.Fatalf("Mkdir(output root) error = %v", err)
|
||||||
|
}
|
||||||
|
validBundle := filepath.Join(outputRoot, "run-1")
|
||||||
|
if err := os.Mkdir(validBundle, 0o755); err != nil {
|
||||||
|
t.Fatalf("Mkdir(bundle) error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := validateBundleRoot(outputRoot, validBundle); err != nil {
|
||||||
|
t.Fatalf("validateBundleRoot(valid) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
outside := filepath.Join(root, "output-other")
|
||||||
|
if err := os.Mkdir(outside, 0o755); err != nil {
|
||||||
|
t.Fatalf("Mkdir(outside) error = %v", err)
|
||||||
|
}
|
||||||
|
for name, candidate := range map[string]string{"equal root": outputRoot, "escape": root, "prefix confusion": outside} {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
if _, err := validateBundleRoot(outputRoot, candidate); err == nil {
|
||||||
|
t.Fatalf("validateBundleRoot(%q) error = nil", candidate)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
symlink := filepath.Join(outputRoot, "linked")
|
||||||
|
if err := os.Symlink(outside, symlink); err != nil {
|
||||||
|
t.Skipf("Symlink() unavailable: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := validateBundleRoot(outputRoot, symlink); err == nil {
|
||||||
|
t.Fatal("validateBundleRoot(symlink) error = nil")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadIndexRejectsMalformedUnsafeAndUnsupportedDocuments(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
indexValue any
|
||||||
|
prepare func(*testing.T, string)
|
||||||
|
wantError string
|
||||||
|
}{
|
||||||
|
{name: "malformed", indexValue: json.RawMessage(`{"manifest_file":`)},
|
||||||
|
{name: "unsupported output shape", indexValue: map[string]any{"manifest_file": "manifest.json", "output_files": map[string]any{}, "rejected_file": "rejected.json", "warnings_file": "warnings.json"}},
|
||||||
|
{name: "missing management path", indexValue: map[string]any{"output_files": []any{}, "rejected_file": "rejected.json", "warnings_file": "warnings.json"}},
|
||||||
|
{name: "renamed manifest", indexValue: func() any {
|
||||||
|
value := validIndexValue([]any{})
|
||||||
|
value["manifest_file"] = "metadata.json"
|
||||||
|
return value
|
||||||
|
}(), wantError: `manifest_file "metadata.json"`},
|
||||||
|
{name: "cleanable manifest", indexValue: func() any {
|
||||||
|
value := validIndexValue([]any{})
|
||||||
|
value["manifest_file"] = "./manifest.json"
|
||||||
|
return value
|
||||||
|
}(), wantError: `manifest_file "./manifest.json"`},
|
||||||
|
{name: "renamed rejections", indexValue: func() any {
|
||||||
|
value := validIndexValue([]any{})
|
||||||
|
value["rejected_file"] = "rejections.json"
|
||||||
|
return value
|
||||||
|
}(), wantError: `rejected_file "rejections.json"`},
|
||||||
|
{name: "renamed warnings", indexValue: func() any {
|
||||||
|
value := validIndexValue([]any{})
|
||||||
|
value["warnings_file"] = "diagnostics/warnings.json"
|
||||||
|
return value
|
||||||
|
}(), wantError: `warnings_file "diagnostics/warnings.json"`},
|
||||||
|
{name: "duplicate lane", indexValue: validIndexValue([]any{
|
||||||
|
map[string]any{"lane_id": "npc", "file": "lanes/npc.json"},
|
||||||
|
map[string]any{"lane_id": "npc", "file": "lanes/npc.json"},
|
||||||
|
})},
|
||||||
|
{name: "absolute logical path", indexValue: validIndexValue([]any{map[string]any{"lane_id": "npc", "file": "/tmp/npc.json"}})},
|
||||||
|
{name: "lexical traversal", indexValue: validIndexValue([]any{map[string]any{"lane_id": "npc", "file": "../outside.json"}})},
|
||||||
|
{name: "root prefix confusion", indexValue: validIndexValue([]any{map[string]any{"lane_id": "npc", "file": "../bundle-other/npc.json"}})},
|
||||||
|
{name: "file symlink", indexValue: validIndexValue([]any{map[string]any{"lane_id": "npc", "file": "lanes/npc.json"}}), prepare: func(t *testing.T, bundle string) {
|
||||||
|
if err := os.Symlink(filepath.Join(bundle, "manifest.json"), filepath.Join(bundle, "lanes", "npc.json")); err != nil {
|
||||||
|
t.Skipf("Symlink() unavailable: %v", err)
|
||||||
|
}
|
||||||
|
}},
|
||||||
|
{name: "directory symlink", indexValue: validIndexValue([]any{map[string]any{"lane_id": "npc", "file": "linked/npc.json"}}), prepare: func(t *testing.T, bundle string) {
|
||||||
|
if err := os.Symlink(filepath.Join(bundle, "lanes"), filepath.Join(bundle, "linked")); err != nil {
|
||||||
|
t.Skipf("Symlink() unavailable: %v", err)
|
||||||
|
}
|
||||||
|
}},
|
||||||
|
{name: "missing management file", indexValue: validIndexValue([]any{}), prepare: func(t *testing.T, bundle string) {
|
||||||
|
if err := os.Remove(filepath.Join(bundle, "manifest.json")); err != nil {
|
||||||
|
t.Fatalf("Remove(manifest) error = %v", err)
|
||||||
|
}
|
||||||
|
}},
|
||||||
|
{name: "incomplete pipeline descriptor", indexValue: func() any {
|
||||||
|
value := validIndexValue([]any{})
|
||||||
|
value["chunk_map"] = map[string]any{"artifact_kind": "chunk_map", "file": "chunk-map.json"}
|
||||||
|
return value
|
||||||
|
}()},
|
||||||
|
{name: "pipeline descriptor escape", indexValue: func() any {
|
||||||
|
value := validIndexValue([]any{})
|
||||||
|
value["evidence_context"] = map[string]any{
|
||||||
|
"artifact_kind": "evidence_context", "file": "../evidence.json", "media_type": "application/json",
|
||||||
|
"schema_id": "evidence", "schema_name": "Evidence", "schema_version": "v1",
|
||||||
|
}
|
||||||
|
return value
|
||||||
|
}()},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
bundle := createBundleSkeleton(t)
|
||||||
|
indexPath := filepath.Join(bundle, "index.json")
|
||||||
|
if raw, ok := test.indexValue.(json.RawMessage); ok {
|
||||||
|
if err := os.WriteFile(indexPath, raw, 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(index) error = %v", err)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
writeJSONFile(t, indexPath, test.indexValue)
|
||||||
|
}
|
||||||
|
if test.prepare != nil {
|
||||||
|
test.prepare(t, bundle)
|
||||||
|
}
|
||||||
|
if _, err := loadIndex(bundle, indexPath); err == nil {
|
||||||
|
t.Fatal("loadIndex() error = nil, want failure")
|
||||||
|
} else if test.wantError != "" && !strings.Contains(err.Error(), test.wantError) {
|
||||||
|
t.Fatalf("loadIndex() error = %v, want fragment %q", err, test.wantError)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
bundle := createBundleSkeleton(t)
|
||||||
|
oversizedIndex := filepath.Join(bundle, "index.json")
|
||||||
|
if err := os.WriteFile(oversizedIndex, []byte(strings.Repeat("x", maxIndexBytes+1)), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(oversized index) error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := loadIndex(bundle, oversizedIndex); err == nil || !strings.Contains(err.Error(), "exceeds") {
|
||||||
|
t.Fatalf("loadIndex(oversized) error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadDiagnosticSummariesValidateBoundsAndTolerateUnknownFields(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
rejectedPath := filepath.Join(root, "rejected.json")
|
||||||
|
warningsPath := filepath.Join(root, "warnings.json")
|
||||||
|
writeJSONFile(t, rejectedPath, map[string]any{"rejected": []any{map[string]any{
|
||||||
|
"stage": "validate", "lane_id": "spells", "reason_code": "invalid", "message": "do not retain this", "future": true,
|
||||||
|
}}, "future": true})
|
||||||
|
writeJSONFile(t, warningsPath, map[string]any{"warnings": []any{map[string]any{
|
||||||
|
"scope": "lane:spells", "reason_code": "bounded", "message": "do not retain this", "future": true,
|
||||||
|
}}, "future": true})
|
||||||
|
rejections, err := loadRejections(rejectedPath)
|
||||||
|
if err != nil || len(rejections) != 1 || rejections[0].ReasonCode != "invalid" {
|
||||||
|
t.Fatalf("loadRejections() = %#v, %v", rejections, err)
|
||||||
|
}
|
||||||
|
warnings, err := loadWarnings(warningsPath)
|
||||||
|
if err != nil || len(warnings) != 1 || warnings[0].Scope != "lane:spells" {
|
||||||
|
t.Fatalf("loadWarnings() = %#v, %v", warnings, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, path := range map[string]string{"rejections": rejectedPath, "warnings": warningsPath} {
|
||||||
|
t.Run("malformed "+name, func(t *testing.T) {
|
||||||
|
if err := os.WriteFile(path, []byte("{"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile() error = %v", err)
|
||||||
|
}
|
||||||
|
var err error
|
||||||
|
if name == "rejections" {
|
||||||
|
_, err = loadRejections(path)
|
||||||
|
} else {
|
||||||
|
_, err = loadWarnings(path)
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("summary decoder error = nil")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
oversized := filepath.Join(root, "oversized.json")
|
||||||
|
if err := os.WriteFile(oversized, []byte(strings.Repeat("x", maxSummaryBytes+1)), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(oversized) error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := loadWarnings(oversized); err == nil || !strings.Contains(err.Error(), "exceeds") {
|
||||||
|
t.Fatalf("loadWarnings(oversized) error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := loadRejections(oversized); err == nil || !strings.Contains(err.Error(), "exceeds") {
|
||||||
|
t.Fatalf("loadRejections(oversized) error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFakeRunnerCapturesRequestsAndHonorsContextAndError(t *testing.T) {
|
||||||
|
req := RunRequest{PipelineID: "pipeline"}
|
||||||
|
want := RunResult{BundleRoot: "/bundle"}
|
||||||
|
fake := &FakeRunner{Result: want}
|
||||||
|
got, err := fake.Run(context.Background(), req)
|
||||||
|
if err != nil || !reflect.DeepEqual(got, want) || !reflect.DeepEqual(fake.Requests, []RunRequest{req}) {
|
||||||
|
t.Fatalf("Run() = %#v, %v; requests = %#v", got, err, fake.Requests)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantErr := errors.New("configured failure")
|
||||||
|
fake.Err = wantErr
|
||||||
|
if _, err := fake.Run(context.Background(), req); !errors.Is(err, wantErr) {
|
||||||
|
t.Fatalf("Run(configured error) = %v", err)
|
||||||
|
}
|
||||||
|
canceled, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
before := len(fake.Requests)
|
||||||
|
if _, err := fake.Run(canceled, req); !errors.Is(err, context.Canceled) || len(fake.Requests) != before {
|
||||||
|
t.Fatalf("Run(canceled) error = %v; requests = %d", err, len(fake.Requests))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validRunRequest(t *testing.T) RunRequest {
|
||||||
|
t.Helper()
|
||||||
|
root := t.TempDir()
|
||||||
|
configPath := filepath.Join(root, "notarius.yml")
|
||||||
|
inputPath := filepath.Join(root, "input.json")
|
||||||
|
outputRoot := filepath.Join(root, "outputs")
|
||||||
|
workingDirectory := filepath.Join(root, "work")
|
||||||
|
diagnostics := filepath.Join(root, "diagnostics")
|
||||||
|
for _, directory := range []string{outputRoot, workingDirectory, diagnostics} {
|
||||||
|
if err := os.Mkdir(directory, 0o755); err != nil {
|
||||||
|
t.Fatalf("Mkdir(%q) error = %v", directory, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(configPath, []byte("pipelines: {}\n"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(config) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(inputPath, []byte("{}\n"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(input) error = %v", err)
|
||||||
|
}
|
||||||
|
return RunRequest{
|
||||||
|
Binary: "notarius", ConfigPath: configPath, PipelineID: "dnd-session", InputPath: inputPath,
|
||||||
|
OutputRoot: outputRoot, WorkingDirectory: workingDirectory,
|
||||||
|
ReceiptPath: filepath.Join(diagnostics, "receipt.json"), LogPath: filepath.Join(diagnostics, "stderr.log"),
|
||||||
|
Timeout: time.Second,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeValidBundleAndReceipt(t *testing.T, req RunRequest, includeUnknown bool) {
|
||||||
|
t.Helper()
|
||||||
|
bundle := filepath.Join(req.OutputRoot, "notarius-run-1")
|
||||||
|
if err := os.MkdirAll(filepath.Join(bundle, "lanes"), 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(bundle) error = %v", err)
|
||||||
|
}
|
||||||
|
for path, data := range map[string]string{
|
||||||
|
"manifest.json": `{}`,
|
||||||
|
"lanes/npc.json": `{}`,
|
||||||
|
"chunk-map.json": `{}`,
|
||||||
|
"evidence-context.json": `{}`,
|
||||||
|
} {
|
||||||
|
if err := os.WriteFile(filepath.Join(bundle, filepath.FromSlash(path)), []byte(data), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(%q) error = %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
rejection := map[string]any{"stage": "validate", "lane_id": "spells", "reason_code": "invalid_spell", "message": strings.Repeat("external detail", 20)}
|
||||||
|
warning := map[string]any{"scope": "lane:npc-registry", "reason_code": "normalized_name", "message": strings.Repeat("external warning", 20)}
|
||||||
|
if includeUnknown {
|
||||||
|
rejection["future"] = true
|
||||||
|
warning["future"] = true
|
||||||
|
}
|
||||||
|
writeJSONFile(t, filepath.Join(bundle, "rejected.json"), map[string]any{"rejected": []any{rejection}, "future": true})
|
||||||
|
writeJSONFile(t, filepath.Join(bundle, "warnings.json"), map[string]any{"warnings": []any{warning}, "future": true})
|
||||||
|
index := validIndexValue([]any{map[string]any{
|
||||||
|
"lane_id": "npc-registry", "file": "lanes/npc.json", "media_type": "application/json",
|
||||||
|
"module_key": "dnd/npc-registry", "schema_id": "notarius.dnd.npc_registry",
|
||||||
|
"schema_name": "NPCRegistry", "schema_version": "v1", "future": true,
|
||||||
|
}})
|
||||||
|
index["chunk_map"] = map[string]any{
|
||||||
|
"artifact_kind": "chunk_map", "file": "chunk-map.json", "media_type": "application/json",
|
||||||
|
"schema_id": "notarius.chunk_map", "schema_name": "ChunkMap", "schema_version": "v1", "future": true,
|
||||||
|
}
|
||||||
|
index["evidence_context"] = map[string]any{
|
||||||
|
"artifact_kind": "evidence_context", "file": "evidence-context.json", "media_type": "application/json",
|
||||||
|
"schema_id": "notarius.evidence_context", "schema_name": "EvidenceContext", "schema_version": "v1", "future": true,
|
||||||
|
}
|
||||||
|
index["future"] = true
|
||||||
|
writeJSONFile(t, filepath.Join(bundle, "index.json"), index)
|
||||||
|
receipt := map[string]any{
|
||||||
|
"schema_version": ReceiptSchemaVersion, "run_id": "notarius-run-1", "pipeline_id": req.PipelineID,
|
||||||
|
"output_directory": bundle, "index_file": "index.json", "normalized_output_count": 1,
|
||||||
|
"rejected_output_count": 1, "warning_count": 1, "validation_status": "rejected",
|
||||||
|
}
|
||||||
|
if includeUnknown {
|
||||||
|
receipt["future"] = true
|
||||||
|
}
|
||||||
|
writeJSONFile(t, req.ReceiptPath, receipt)
|
||||||
|
}
|
||||||
|
|
||||||
|
func createBundleSkeleton(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
bundle := filepath.Join(t.TempDir(), "bundle")
|
||||||
|
if err := os.MkdirAll(filepath.Join(bundle, "lanes"), 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll(bundle) error = %v", err)
|
||||||
|
}
|
||||||
|
for _, name := range []string{"manifest.json", "rejected.json", "warnings.json", "lanes/npc.json", "chunk-map.json"} {
|
||||||
|
if err := os.WriteFile(filepath.Join(bundle, filepath.FromSlash(name)), []byte("{}"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(%q) error = %v", name, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return bundle
|
||||||
|
}
|
||||||
|
|
||||||
|
func validIndexValue(lanes []any) map[string]any {
|
||||||
|
return map[string]any{
|
||||||
|
"manifest_file": "manifest.json", "output_files": lanes,
|
||||||
|
"rejected_file": "rejected.json", "warnings_file": "warnings.json",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeJSONFile(t *testing.T, path string, value any) {
|
||||||
|
t.Helper()
|
||||||
|
data, err := json.Marshal(value)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("json.Marshal() error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(path, data, 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(%q) error = %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeShellScript(t *testing.T, body string) string {
|
||||||
|
t.Helper()
|
||||||
|
path := filepath.Join(t.TempDir(), "notarius-helper")
|
||||||
|
if err := os.WriteFile(path, []byte(body), 0o755); err != nil {
|
||||||
|
t.Fatalf("WriteFile(script) error = %v", err)
|
||||||
|
}
|
||||||
|
return path
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertTextFile(t *testing.T, path, want string) {
|
||||||
|
t.Helper()
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(%q) error = %v", path, err)
|
||||||
|
}
|
||||||
|
if string(data) != want {
|
||||||
|
t.Fatalf("ReadFile(%q) = %q, want %q", path, string(data), want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneMap(source map[string]any) map[string]any {
|
||||||
|
result := make(map[string]any, len(source))
|
||||||
|
for key, value := range source {
|
||||||
|
result[key] = value
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
@@ -68,6 +68,26 @@ func (n *NoopRunner) Normalize(ctx context.Context, req NormalizeRequest) (Norma
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Render returns the requested output path with placeholder metadata.
|
||||||
|
func (n *NoopRunner) Render(ctx context.Context, req RenderRequest) (RenderResult, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return RenderResult{}, err
|
||||||
|
}
|
||||||
|
if err := materializeRenderPlaceholders(req); err != nil {
|
||||||
|
return RenderResult{}, err
|
||||||
|
}
|
||||||
|
return RenderResult{
|
||||||
|
OutputRenderedPath: req.OutputRenderedPath,
|
||||||
|
StdoutLogPath: req.StdoutLogPath,
|
||||||
|
StderrLogPath: req.StderrLogPath,
|
||||||
|
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||||
|
InvokedBinary: "noop",
|
||||||
|
Format: req.Format,
|
||||||
|
Title: req.Title,
|
||||||
|
Metadata: map[string]any{"placeholder": true},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
// FakeRunner captures merge requests and returns deterministic responses.
|
// FakeRunner captures merge requests and returns deterministic responses.
|
||||||
type FakeRunner struct {
|
type FakeRunner struct {
|
||||||
Requests []MergeRequest
|
Requests []MergeRequest
|
||||||
@@ -79,6 +99,9 @@ type FakeRunner struct {
|
|||||||
TrimRequests []TrimRequest
|
TrimRequests []TrimRequest
|
||||||
TrimErr error
|
TrimErr error
|
||||||
TrimResult TrimResult
|
TrimResult TrimResult
|
||||||
|
RenderRequests []RenderRequest
|
||||||
|
RenderErr error
|
||||||
|
RenderResult RenderResult
|
||||||
}
|
}
|
||||||
|
|
||||||
// Run records request and returns configured response.
|
// Run records request and returns configured response.
|
||||||
@@ -195,6 +218,46 @@ func (f *FakeRunner) Normalize(ctx context.Context, req NormalizeRequest) (Norma
|
|||||||
return res, nil
|
return res, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Render records request and returns configured response.
|
||||||
|
func (f *FakeRunner) Render(ctx context.Context, req RenderRequest) (RenderResult, error) {
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return RenderResult{}, err
|
||||||
|
}
|
||||||
|
f.RenderRequests = append(f.RenderRequests, req)
|
||||||
|
if f.RenderErr != nil {
|
||||||
|
return RenderResult{}, f.RenderErr
|
||||||
|
}
|
||||||
|
if err := materializeRenderPlaceholders(req); err != nil {
|
||||||
|
return RenderResult{}, err
|
||||||
|
}
|
||||||
|
res := f.RenderResult
|
||||||
|
if res.OutputRenderedPath == "" {
|
||||||
|
res.OutputRenderedPath = req.OutputRenderedPath
|
||||||
|
}
|
||||||
|
if res.StdoutLogPath == "" {
|
||||||
|
res.StdoutLogPath = req.StdoutLogPath
|
||||||
|
}
|
||||||
|
if res.StderrLogPath == "" {
|
||||||
|
res.StderrLogPath = req.StderrLogPath
|
||||||
|
}
|
||||||
|
if res.GeneratedConfigPath == "" {
|
||||||
|
res.GeneratedConfigPath = req.GeneratedConfigPath
|
||||||
|
}
|
||||||
|
if res.InvokedBinary == "" {
|
||||||
|
res.InvokedBinary = "fake"
|
||||||
|
}
|
||||||
|
if res.Format == "" {
|
||||||
|
res.Format = req.Format
|
||||||
|
}
|
||||||
|
if res.Title == "" {
|
||||||
|
res.Title = req.Title
|
||||||
|
}
|
||||||
|
if res.Metadata == nil {
|
||||||
|
res.Metadata = map[string]any{"fake": true}
|
||||||
|
}
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
|
|
||||||
func materializePlaceholders(req MergeRequest) error {
|
func materializePlaceholders(req MergeRequest) error {
|
||||||
if req.OutputMergedTranscriptPath != "" {
|
if req.OutputMergedTranscriptPath != "" {
|
||||||
if err := subprocess.WriteFileAtomic(req.OutputMergedTranscriptPath, []byte(`{"schema":"seriatim.intermediate.v1","segments":[]}`), 0o644); err != nil {
|
if err := subprocess.WriteFileAtomic(req.OutputMergedTranscriptPath, []byte(`{"schema":"seriatim.intermediate.v1","segments":[]}`), 0o644); err != nil {
|
||||||
@@ -301,3 +364,39 @@ func materializeNormalizePlaceholders(req NormalizeRequest) error {
|
|||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func materializeRenderPlaceholders(req RenderRequest) error {
|
||||||
|
if req.OutputRenderedPath != "" {
|
||||||
|
if err := subprocess.WriteFileAtomic(req.OutputRenderedPath, []byte("# Transcript\n\nRendered markdown placeholder.\n"), 0o644); err != nil {
|
||||||
|
return fmt.Errorf("write rendered transcript %q: %w", req.OutputRenderedPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if req.GeneratedConfigPath != "" {
|
||||||
|
payload := map[string]any{
|
||||||
|
"schema": "seriatim.generated.v1",
|
||||||
|
"placeholder": true,
|
||||||
|
"command": "render",
|
||||||
|
"input_path": req.InputTranscriptPath,
|
||||||
|
"output_path": req.OutputRenderedPath,
|
||||||
|
"format": req.Format,
|
||||||
|
"title": req.Title,
|
||||||
|
"include_timestamps": req.IncludeTimestamps,
|
||||||
|
"include_segment_ids": req.IncludeSegmentIDs,
|
||||||
|
"include_metadata": req.IncludeMetadata,
|
||||||
|
}
|
||||||
|
if err := subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644); err != nil {
|
||||||
|
return fmt.Errorf("write generated config %q: %w", req.GeneratedConfigPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if req.StdoutLogPath != "" {
|
||||||
|
if err := subprocess.WriteFileAtomic(req.StdoutLogPath, []byte("seriatim noop/fake render stdout placeholder\n"), 0o644); err != nil {
|
||||||
|
return fmt.Errorf("write stdout log %q: %w", req.StdoutLogPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if req.StderrLogPath != "" {
|
||||||
|
if err := subprocess.WriteFileAtomic(req.StderrLogPath, []byte("seriatim noop/fake render stderr placeholder\n"), 0o644); err != nil {
|
||||||
|
return fmt.Errorf("write stderr log %q: %w", req.StderrLogPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -148,3 +148,58 @@ func TestFakeRunnerNormalizeError(t *testing.T) {
|
|||||||
t.Fatal("expected error, got nil")
|
t.Fatal("expected error, got nil")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestFakeRunnerRenderCapturesRequestAndReturnsPath(t *testing.T) {
|
||||||
|
fake := &FakeRunner{}
|
||||||
|
dir := t.TempDir()
|
||||||
|
req := RenderRequest{
|
||||||
|
GeneratedConfigPath: filepath.Join(dir, "config", "seriatim.render.yml"),
|
||||||
|
InputTranscriptPath: filepath.Join(dir, "transcripts", "final.trimmed.json"),
|
||||||
|
OutputRenderedPath: filepath.Join(dir, "transcripts", "final.trimmed.md"),
|
||||||
|
Format: "markdown",
|
||||||
|
Title: "Session render",
|
||||||
|
IncludeTimestamps: true,
|
||||||
|
IncludeSegmentIDs: false,
|
||||||
|
IncludeMetadata: true,
|
||||||
|
StdoutLogPath: filepath.Join(dir, "logs", "seriatim.render.stdout.log"),
|
||||||
|
StderrLogPath: filepath.Join(dir, "logs", "seriatim.render.stderr.log"),
|
||||||
|
}
|
||||||
|
|
||||||
|
res, err := fake.Render(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Render() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(fake.RenderRequests) != 1 || fake.RenderRequests[0].GeneratedConfigPath == "" {
|
||||||
|
t.Fatalf("render requests = %#v, want captured request", fake.RenderRequests)
|
||||||
|
}
|
||||||
|
if res.OutputRenderedPath != req.OutputRenderedPath {
|
||||||
|
t.Fatalf("rendered path = %q, want %q", res.OutputRenderedPath, req.OutputRenderedPath)
|
||||||
|
}
|
||||||
|
if res.Format != req.Format {
|
||||||
|
t.Fatalf("format = %q, want %q", res.Format, req.Format)
|
||||||
|
}
|
||||||
|
if res.Title != req.Title {
|
||||||
|
t.Fatalf("title = %q, want %q", res.Title, req.Title)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfgData, err := os.ReadFile(req.GeneratedConfigPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read generated config: %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(cfgData), "command: render") {
|
||||||
|
t.Fatalf("generated config = %q, want render command marker", string(cfgData))
|
||||||
|
}
|
||||||
|
for _, path := range []string{req.StdoutLogPath, req.StderrLogPath, req.OutputRenderedPath} {
|
||||||
|
if _, err := os.Stat(path); err != nil {
|
||||||
|
t.Fatalf("expected file %q to exist: %v", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFakeRunnerRenderError(t *testing.T) {
|
||||||
|
fake := &FakeRunner{RenderErr: errors.New("boom")}
|
||||||
|
_, err := fake.Render(context.Background(), RenderRequest{})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected error, got nil")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
// Package seriatim declares the adapter contract for transcript merge/normalize/trim execution.
|
// Package seriatim declares the adapter contract for transcript merge/normalize/trim/render execution.
|
||||||
package seriatim
|
package seriatim
|
||||||
|
|
||||||
import (
|
import (
|
||||||
@@ -6,11 +6,12 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Runner is the adapter boundary for seriatim merge/normalize/trim invocations.
|
// Runner is the adapter boundary for seriatim merge/normalize/trim/render invocations.
|
||||||
type Runner interface {
|
type Runner interface {
|
||||||
Run(ctx context.Context, req MergeRequest) (MergeResult, error)
|
Run(ctx context.Context, req MergeRequest) (MergeResult, error)
|
||||||
Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error)
|
Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error)
|
||||||
Trim(ctx context.Context, req TrimRequest) (TrimResult, error)
|
Trim(ctx context.Context, req TrimRequest) (TrimResult, error)
|
||||||
|
Render(ctx context.Context, req RenderRequest) (RenderResult, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
// MergeRequest describes a seriatim merge invocation.
|
// MergeRequest describes a seriatim merge invocation.
|
||||||
@@ -90,3 +91,33 @@ type TrimResult struct {
|
|||||||
KeepSelector string
|
KeepSelector string
|
||||||
Metadata map[string]any
|
Metadata map[string]any
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RenderRequest describes a seriatim render invocation.
|
||||||
|
type RenderRequest struct {
|
||||||
|
Binary string
|
||||||
|
InputTranscriptPath string
|
||||||
|
OutputRenderedPath string
|
||||||
|
Format string
|
||||||
|
Title string
|
||||||
|
IncludeTimestamps bool
|
||||||
|
IncludeSegmentIDs bool
|
||||||
|
IncludeMetadata bool
|
||||||
|
StdoutLogPath string
|
||||||
|
StderrLogPath string
|
||||||
|
GeneratedConfigPath string
|
||||||
|
Timeout time.Duration
|
||||||
|
}
|
||||||
|
|
||||||
|
// RenderResult describes a render output.
|
||||||
|
type RenderResult struct {
|
||||||
|
OutputRenderedPath string
|
||||||
|
StdoutLogPath string
|
||||||
|
StderrLogPath string
|
||||||
|
GeneratedConfigPath string
|
||||||
|
ExitCode int
|
||||||
|
Duration time.Duration
|
||||||
|
InvokedBinary string
|
||||||
|
Format string
|
||||||
|
Title string
|
||||||
|
Metadata map[string]any
|
||||||
|
}
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import (
|
|||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/subprocess"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/subprocess"
|
||||||
)
|
)
|
||||||
@@ -384,6 +385,96 @@ func (r *SubprocessRunner) Normalize(ctx context.Context, req NormalizeRequest)
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Render executes Seriatim render with deterministic flags and validates non-empty text output.
|
||||||
|
func (r *SubprocessRunner) Render(ctx context.Context, req RenderRequest) (RenderResult, error) {
|
||||||
|
if r == nil {
|
||||||
|
return RenderResult{}, fmt.Errorf("seriatim subprocess runner is nil")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(req.InputTranscriptPath) == "" {
|
||||||
|
return RenderResult{}, fmt.Errorf("seriatim render input path is required")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(req.OutputRenderedPath) == "" {
|
||||||
|
return RenderResult{}, fmt.Errorf("seriatim render output path is required")
|
||||||
|
}
|
||||||
|
format := strings.TrimSpace(req.Format)
|
||||||
|
if format == "" {
|
||||||
|
format = "markdown"
|
||||||
|
}
|
||||||
|
if format != "markdown" {
|
||||||
|
return RenderResult{}, fmt.Errorf("seriatim render format %q is unsupported", req.Format)
|
||||||
|
}
|
||||||
|
|
||||||
|
binary := r.binary
|
||||||
|
if strings.TrimSpace(req.Binary) != "" {
|
||||||
|
binary = strings.TrimSpace(req.Binary)
|
||||||
|
}
|
||||||
|
|
||||||
|
timeout := r.timeout
|
||||||
|
if req.Timeout < 0 {
|
||||||
|
return RenderResult{}, fmt.Errorf("seriatim render timeout must be >= 0")
|
||||||
|
}
|
||||||
|
if req.Timeout > 0 {
|
||||||
|
timeout = req.Timeout
|
||||||
|
}
|
||||||
|
|
||||||
|
args := buildRenderArgs(req, format)
|
||||||
|
if req.GeneratedConfigPath != "" {
|
||||||
|
if err := writeRenderInvocationConfig(req, args, binary, timeout, format); err != nil {
|
||||||
|
return RenderResult{}, fmt.Errorf("write seriatim render invocation config %q: %w", req.GeneratedConfigPath, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
runRes, err := subprocess.Run(ctx, subprocess.RunRequest{
|
||||||
|
Executable: binary,
|
||||||
|
Args: args,
|
||||||
|
Timeout: timeout,
|
||||||
|
StdoutLogPath: req.StdoutLogPath,
|
||||||
|
StderrLogPath: req.StderrLogPath,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return RenderResult{
|
||||||
|
OutputRenderedPath: req.OutputRenderedPath,
|
||||||
|
StdoutLogPath: req.StdoutLogPath,
|
||||||
|
StderrLogPath: req.StderrLogPath,
|
||||||
|
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||||
|
ExitCode: runRes.ExitCode,
|
||||||
|
Duration: runRes.Duration,
|
||||||
|
InvokedBinary: binary,
|
||||||
|
Format: format,
|
||||||
|
Title: req.Title,
|
||||||
|
}, fmt.Errorf("run seriatim render (binary=%q): %w", binary, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := validateNonEmptyTextFile(req.OutputRenderedPath); err != nil {
|
||||||
|
return RenderResult{
|
||||||
|
OutputRenderedPath: req.OutputRenderedPath,
|
||||||
|
StdoutLogPath: req.StdoutLogPath,
|
||||||
|
StderrLogPath: req.StderrLogPath,
|
||||||
|
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||||
|
ExitCode: runRes.ExitCode,
|
||||||
|
Duration: runRes.Duration,
|
||||||
|
InvokedBinary: binary,
|
||||||
|
Format: format,
|
||||||
|
Title: req.Title,
|
||||||
|
}, fmt.Errorf("validate seriatim rendered output %q: %w", req.OutputRenderedPath, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return RenderResult{
|
||||||
|
OutputRenderedPath: req.OutputRenderedPath,
|
||||||
|
StdoutLogPath: req.StdoutLogPath,
|
||||||
|
StderrLogPath: req.StderrLogPath,
|
||||||
|
GeneratedConfigPath: req.GeneratedConfigPath,
|
||||||
|
ExitCode: runRes.ExitCode,
|
||||||
|
Duration: runRes.Duration,
|
||||||
|
InvokedBinary: binary,
|
||||||
|
Format: format,
|
||||||
|
Title: req.Title,
|
||||||
|
Metadata: map[string]any{
|
||||||
|
"adapter": "seriatim_subprocess",
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (r *SubprocessRunner) buildMergeArgs(req MergeRequest) []string {
|
func (r *SubprocessRunner) buildMergeArgs(req MergeRequest) []string {
|
||||||
args := []string{"merge"}
|
args := []string{"merge"}
|
||||||
|
|
||||||
@@ -480,6 +571,22 @@ func buildNormalizeArgs(req NormalizeRequest, outputSchema string) []string {
|
|||||||
return args
|
return args
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func buildRenderArgs(req RenderRequest, format string) []string {
|
||||||
|
args := []string{
|
||||||
|
"render",
|
||||||
|
"--input-file", req.InputTranscriptPath,
|
||||||
|
"--output-file", req.OutputRenderedPath,
|
||||||
|
"--format", format,
|
||||||
|
"--include-timestamps=" + strconv.FormatBool(req.IncludeTimestamps),
|
||||||
|
"--include-segment-ids=" + strconv.FormatBool(req.IncludeSegmentIDs),
|
||||||
|
"--include-metadata=" + strconv.FormatBool(req.IncludeMetadata),
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(req.Title) != "" {
|
||||||
|
args = append(args, "--title", req.Title)
|
||||||
|
}
|
||||||
|
return args
|
||||||
|
}
|
||||||
|
|
||||||
func writeTrimInvocationConfig(req TrimRequest, args []string, binary string, timeout time.Duration) error {
|
func writeTrimInvocationConfig(req TrimRequest, args []string, binary string, timeout time.Duration) error {
|
||||||
payload := map[string]any{
|
payload := map[string]any{
|
||||||
"schema": "seriatim.generated.v1",
|
"schema": "seriatim.generated.v1",
|
||||||
@@ -509,6 +616,24 @@ func writeNormalizeInvocationConfig(req NormalizeRequest, args []string, binary
|
|||||||
return subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644)
|
return subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writeRenderInvocationConfig(req RenderRequest, args []string, binary string, timeout time.Duration, format string) error {
|
||||||
|
payload := map[string]any{
|
||||||
|
"schema": "seriatim.generated.v1",
|
||||||
|
"command": "render",
|
||||||
|
"binary": binary,
|
||||||
|
"args": args,
|
||||||
|
"timeout": timeout.String(),
|
||||||
|
"input_path": req.InputTranscriptPath,
|
||||||
|
"output_path": req.OutputRenderedPath,
|
||||||
|
"format": format,
|
||||||
|
"title": req.Title,
|
||||||
|
"include_timestamps": req.IncludeTimestamps,
|
||||||
|
"include_segment_ids": req.IncludeSegmentIDs,
|
||||||
|
"include_metadata": req.IncludeMetadata,
|
||||||
|
}
|
||||||
|
return subprocess.WriteYAMLAtomic(req.GeneratedConfigPath, payload, 0o644)
|
||||||
|
}
|
||||||
|
|
||||||
func validateJSONFile(path string) error {
|
func validateJSONFile(path string) error {
|
||||||
data, err := os.ReadFile(path)
|
data, err := os.ReadFile(path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -541,3 +666,20 @@ func validateJSONFileWithSegments(path string) error {
|
|||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validateNonEmptyTextFile(path string) error {
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("read file: %w", err)
|
||||||
|
}
|
||||||
|
if len(data) == 0 {
|
||||||
|
return fmt.Errorf("file is empty")
|
||||||
|
}
|
||||||
|
if !utf8.Valid(data) {
|
||||||
|
return fmt.Errorf("file is not valid utf-8 text")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(string(data)) == "" {
|
||||||
|
return fmt.Errorf("file has no non-whitespace content")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -569,6 +569,156 @@ func TestSubprocessRunnerNormalizeInvalidReportJSONFails(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerRenderSuccessInvocationAndProvenance(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("helper wrapper script uses /bin/sh")
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("SERIATIM_HELPER_MODE", "render_success")
|
||||||
|
recordPath := filepath.Join(t.TempDir(), "record.json")
|
||||||
|
t.Setenv("SERIATIM_HELPER_RECORD_PATH", recordPath)
|
||||||
|
|
||||||
|
wrapper := writeHelperWrapper(t)
|
||||||
|
runner := mustRunner(t, wrapper, false)
|
||||||
|
req := renderReqForTest(t)
|
||||||
|
|
||||||
|
res, err := runner.Render(context.Background(), req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Render() error = %v", err)
|
||||||
|
}
|
||||||
|
if res.OutputRenderedPath != req.OutputRenderedPath {
|
||||||
|
t.Fatalf("OutputRenderedPath = %q, want %q", res.OutputRenderedPath, req.OutputRenderedPath)
|
||||||
|
}
|
||||||
|
if res.Format != req.Format {
|
||||||
|
t.Fatalf("Format = %q, want %q", res.Format, req.Format)
|
||||||
|
}
|
||||||
|
if res.Title != req.Title {
|
||||||
|
t.Fatalf("Title = %q, want %q", res.Title, req.Title)
|
||||||
|
}
|
||||||
|
if res.InvokedBinary != wrapper {
|
||||||
|
t.Fatalf("InvokedBinary = %q, want %q", res.InvokedBinary, wrapper)
|
||||||
|
}
|
||||||
|
if res.ExitCode != 0 {
|
||||||
|
t.Fatalf("ExitCode = %d, want 0", res.ExitCode)
|
||||||
|
}
|
||||||
|
if res.Duration <= 0 {
|
||||||
|
t.Fatalf("Duration = %s, want >0", res.Duration)
|
||||||
|
}
|
||||||
|
if res.Metadata == nil || res.Metadata["adapter"] != "seriatim_subprocess" {
|
||||||
|
t.Fatalf("Metadata = %#v, want adapter marker", res.Metadata)
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := os.Stat(req.OutputRenderedPath); err != nil {
|
||||||
|
t.Fatalf("rendered output missing: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(req.StdoutLogPath); err != nil {
|
||||||
|
t.Fatalf("stdout log missing: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(req.StderrLogPath); err != nil {
|
||||||
|
t.Fatalf("stderr log missing: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(req.GeneratedConfigPath); err != nil {
|
||||||
|
t.Fatalf("generated config missing: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := readHelperRecord(t, recordPath)
|
||||||
|
wantArgs := []string{
|
||||||
|
"render",
|
||||||
|
"--input-file", req.InputTranscriptPath,
|
||||||
|
"--output-file", req.OutputRenderedPath,
|
||||||
|
"--format", req.Format,
|
||||||
|
"--include-timestamps=true",
|
||||||
|
"--include-segment-ids=true",
|
||||||
|
"--include-metadata=false",
|
||||||
|
"--title", req.Title,
|
||||||
|
}
|
||||||
|
if strings.Join(rec.Args, "\n") != strings.Join(wantArgs, "\n") {
|
||||||
|
t.Fatalf("args = %#v, want %#v", rec.Args, wantArgs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerRenderWithoutTitleOmitsTitleArg(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("helper wrapper script uses /bin/sh")
|
||||||
|
}
|
||||||
|
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("SERIATIM_HELPER_MODE", "render_success")
|
||||||
|
recordPath := filepath.Join(t.TempDir(), "record.json")
|
||||||
|
t.Setenv("SERIATIM_HELPER_RECORD_PATH", recordPath)
|
||||||
|
|
||||||
|
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||||
|
req := renderReqForTest(t)
|
||||||
|
req.Title = ""
|
||||||
|
if _, err := runner.Render(context.Background(), req); err != nil {
|
||||||
|
t.Fatalf("Render() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
rec := readHelperRecord(t, recordPath)
|
||||||
|
for i := 0; i < len(rec.Args); i++ {
|
||||||
|
if rec.Args[i] == "--title" {
|
||||||
|
t.Fatalf("args = %#v, did not expect --title", rec.Args)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerRenderSubprocessFailure(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("helper wrapper script uses /bin/sh")
|
||||||
|
}
|
||||||
|
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("SERIATIM_HELPER_MODE", "fail")
|
||||||
|
t.Setenv("SERIATIM_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||||
|
|
||||||
|
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||||
|
req := renderReqForTest(t)
|
||||||
|
_, err := runner.Render(context.Background(), req)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Render() error = nil, want non-nil")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "run seriatim render") {
|
||||||
|
t.Fatalf("error = %q, want subprocess context", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerRenderMissingOutputFails(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("helper wrapper script uses /bin/sh")
|
||||||
|
}
|
||||||
|
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("SERIATIM_HELPER_MODE", "missing_output")
|
||||||
|
t.Setenv("SERIATIM_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||||
|
|
||||||
|
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||||
|
req := renderReqForTest(t)
|
||||||
|
_, err := runner.Render(context.Background(), req)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Render() error = nil, want non-nil")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "validate seriatim rendered output") {
|
||||||
|
t.Fatalf("error = %q, want output validation context", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSubprocessRunnerRenderEmptyOutputFails(t *testing.T) {
|
||||||
|
if runtime.GOOS == "windows" {
|
||||||
|
t.Skip("helper wrapper script uses /bin/sh")
|
||||||
|
}
|
||||||
|
t.Setenv("GO_WANT_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("SERIATIM_HELPER_MODE", "render_empty_output")
|
||||||
|
t.Setenv("SERIATIM_HELPER_RECORD_PATH", filepath.Join(t.TempDir(), "record.json"))
|
||||||
|
|
||||||
|
runner := mustRunner(t, writeHelperWrapper(t), false)
|
||||||
|
req := renderReqForTest(t)
|
||||||
|
_, err := runner.Render(context.Background(), req)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Render() error = nil, want non-nil")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "file is empty") {
|
||||||
|
t.Fatalf("error = %q, want empty-file validation", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestSubprocessRunnerConstructorValidation(t *testing.T) {
|
func TestSubprocessRunnerConstructorValidation(t *testing.T) {
|
||||||
_, err := NewSubprocessRunnerFromConfigValues("", "10m", "seriatim-intermediate", nil, true, EnvConfig{})
|
_, err := NewSubprocessRunnerFromConfigValues("", "10m", "seriatim-intermediate", nil, true, EnvConfig{})
|
||||||
if err == nil {
|
if err == nil {
|
||||||
@@ -702,6 +852,14 @@ func TestSeriatimSubprocessHelper(t *testing.T) {
|
|||||||
case "normalize_report_missing":
|
case "normalize_report_missing":
|
||||||
writeSeriatimHelperFile(outputPath, `{"schema":"seriatim.intermediate.v1","segments":[]}`)
|
writeSeriatimHelperFile(outputPath, `{"schema":"seriatim.intermediate.v1","segments":[]}`)
|
||||||
os.Exit(0)
|
os.Exit(0)
|
||||||
|
case "render_success":
|
||||||
|
writeSeriatimHelperFile(outputPath, "# Rendered transcript\n\nHello.\n")
|
||||||
|
_, _ = os.Stdout.WriteString("seriatim helper render stdout\n")
|
||||||
|
_, _ = os.Stderr.WriteString("seriatim helper render stderr\n")
|
||||||
|
os.Exit(0)
|
||||||
|
case "render_empty_output":
|
||||||
|
writeSeriatimHelperFile(outputPath, "")
|
||||||
|
os.Exit(0)
|
||||||
default:
|
default:
|
||||||
_, _ = os.Stderr.WriteString(fmt.Sprintf("unknown helper mode %q\n", mode))
|
_, _ = os.Stderr.WriteString(fmt.Sprintf("unknown helper mode %q\n", mode))
|
||||||
os.Exit(2)
|
os.Exit(2)
|
||||||
@@ -777,6 +935,25 @@ func normalizeReqForTest(t *testing.T, withReport bool) NormalizeRequest {
|
|||||||
return req
|
return req
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func renderReqForTest(t *testing.T) RenderRequest {
|
||||||
|
t.Helper()
|
||||||
|
dir := t.TempDir()
|
||||||
|
input := filepath.Join(dir, "final.trimmed.json")
|
||||||
|
writeSeriatimFile(t, input, `{"schema":"seriatim.intermediate.v1","segments":[]}`)
|
||||||
|
return RenderRequest{
|
||||||
|
InputTranscriptPath: input,
|
||||||
|
OutputRenderedPath: filepath.Join(dir, "final.trimmed.md"),
|
||||||
|
Format: "markdown",
|
||||||
|
Title: "Session 42",
|
||||||
|
IncludeTimestamps: true,
|
||||||
|
IncludeSegmentIDs: true,
|
||||||
|
IncludeMetadata: false,
|
||||||
|
GeneratedConfigPath: filepath.Join(dir, "seriatim.render.generated.yml"),
|
||||||
|
StdoutLogPath: filepath.Join(dir, "seriatim.render.stdout.log"),
|
||||||
|
StderrLogPath: filepath.Join(dir, "seriatim.render.stderr.log"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func mustRunner(t *testing.T, binary string, report bool) *SubprocessRunner {
|
func mustRunner(t *testing.T, binary string, report bool) *SubprocessRunner {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
coalesce := 3.0
|
coalesce := 3.0
|
||||||
|
|||||||
@@ -1,31 +0,0 @@
|
|||||||
// Package storage declares archive/storage backend adapter boundaries.
|
|
||||||
package storage
|
|
||||||
|
|
||||||
import "context"
|
|
||||||
|
|
||||||
// TODO: implement remote storage/archive backends (S3/SFTP/etc.).
|
|
||||||
|
|
||||||
// Backend is the adapter boundary for archive/storage operations.
|
|
||||||
type Backend interface {
|
|
||||||
Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
// ArchiveItem describes one item to archive.
|
|
||||||
type ArchiveItem struct {
|
|
||||||
Kind string
|
|
||||||
LocalPath string
|
|
||||||
RemoteKey string
|
|
||||||
}
|
|
||||||
|
|
||||||
// ArchiveRequest describes one archive operation.
|
|
||||||
type ArchiveRequest struct {
|
|
||||||
SessionID string
|
|
||||||
ManifestPath string
|
|
||||||
Items []ArchiveItem
|
|
||||||
}
|
|
||||||
|
|
||||||
// ArchiveResult describes archive operation output.
|
|
||||||
type ArchiveResult struct {
|
|
||||||
Archived []ArchiveItem
|
|
||||||
Metadata map[string]any
|
|
||||||
}
|
|
||||||
@@ -10,23 +10,8 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// NoopBackend is a deterministic no-op archive/storage adapter.
|
// FakeBackend provides a deterministic in-memory object store for tests.
|
||||||
type NoopBackend struct{}
|
|
||||||
|
|
||||||
// Archive returns the requested items as archived with placeholder metadata.
|
|
||||||
func (n *NoopBackend) Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error) {
|
|
||||||
if err := ctx.Err(); err != nil {
|
|
||||||
return ArchiveResult{}, err
|
|
||||||
}
|
|
||||||
return ArchiveResult{Archived: append([]ArchiveItem(nil), req.Items...), Metadata: map[string]any{"placeholder": true}}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// FakeBackend captures archive requests and returns deterministic responses.
|
|
||||||
type FakeBackend struct {
|
type FakeBackend struct {
|
||||||
Requests []ArchiveRequest
|
|
||||||
Err error
|
|
||||||
Result ArchiveResult
|
|
||||||
|
|
||||||
Objects map[string]FakeObject
|
Objects map[string]FakeObject
|
||||||
Uploads []FakeUploadCall
|
Uploads []FakeUploadCall
|
||||||
Downloads []FakeDownloadCall
|
Downloads []FakeDownloadCall
|
||||||
@@ -50,25 +35,6 @@ type FakeDownloadCall struct {
|
|||||||
LocalPath string
|
LocalPath string
|
||||||
}
|
}
|
||||||
|
|
||||||
// Archive records request and returns configured response.
|
|
||||||
func (f *FakeBackend) Archive(ctx context.Context, req ArchiveRequest) (ArchiveResult, error) {
|
|
||||||
if err := ctx.Err(); err != nil {
|
|
||||||
return ArchiveResult{}, err
|
|
||||||
}
|
|
||||||
f.Requests = append(f.Requests, req)
|
|
||||||
if f.Err != nil {
|
|
||||||
return ArchiveResult{}, f.Err
|
|
||||||
}
|
|
||||||
res := f.Result
|
|
||||||
if res.Archived == nil {
|
|
||||||
res.Archived = append([]ArchiveItem(nil), req.Items...)
|
|
||||||
}
|
|
||||||
if res.Metadata == nil {
|
|
||||||
res.Metadata = map[string]any{"fake": true}
|
|
||||||
}
|
|
||||||
return res, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// FakeObject is a deterministic fake object-store record.
|
// FakeObject is a deterministic fake object-store record.
|
||||||
type FakeObject struct {
|
type FakeObject struct {
|
||||||
Key string
|
Key string
|
||||||
|
|||||||
@@ -9,30 +9,6 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestFakeBackendCapturesRequestAndReturnsItems(t *testing.T) {
|
|
||||||
fake := &FakeBackend{}
|
|
||||||
req := ArchiveRequest{SessionID: "s1", Items: []ArchiveItem{{Kind: "artifact", LocalPath: "artifacts/log.md"}}}
|
|
||||||
|
|
||||||
res, err := fake.Archive(context.Background(), req)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Archive() error = %v", err)
|
|
||||||
}
|
|
||||||
if len(fake.Requests) != 1 || fake.Requests[0].SessionID != "s1" {
|
|
||||||
t.Fatalf("requests = %#v, want captured request", fake.Requests)
|
|
||||||
}
|
|
||||||
if len(res.Archived) != 1 {
|
|
||||||
t.Fatalf("archived len = %d, want 1", len(res.Archived))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFakeBackendError(t *testing.T) {
|
|
||||||
fake := &FakeBackend{Err: errors.New("boom")}
|
|
||||||
_, err := fake.Archive(context.Background(), ArchiveRequest{})
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("expected error, got nil")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestFakeBackendListPrefixFiltering(t *testing.T) {
|
func TestFakeBackendListPrefixFiltering(t *testing.T) {
|
||||||
fake := &FakeBackend{}
|
fake := &FakeBackend{}
|
||||||
fake.SeedObject(FakeObject{Key: "dnd/campaigns/forsaken/audio/a.flac", Data: []byte("a")})
|
fake.SeedObject(FakeObject{Key: "dnd/campaigns/forsaken/audio/a.flac", Data: []byte("a")})
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ObjectStore is a remote object storage boundary used by future prepare/archive work.
|
// ObjectStore is a remote object storage boundary used by prepare, restore, and publish work.
|
||||||
//
|
//
|
||||||
// Key invariant:
|
// Key invariant:
|
||||||
// callers pass full bucket-relative object keys. Backend implementations do not
|
// callers pass full bucket-relative object keys. Backend implementations do not
|
||||||
|
|||||||
@@ -145,7 +145,7 @@ func TestHTTPClientDoesNotRetryOnNonRetryableStatus(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestHTTPClientInvalidJSONFailsAndDoesNotPromote(t *testing.T) {
|
func TestHTTPClientInvalidJSONFailsAndDoesNotInstallOutput(t *testing.T) {
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
_, _ = w.Write([]byte(`not-json`))
|
_, _ = w.Write([]byte(`not-json`))
|
||||||
}))
|
}))
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ func TestExecuteRunStageArtifactsUnsupportedStageFails(t *testing.T) {
|
|||||||
var stdout bytes.Buffer
|
var stdout bytes.Buffer
|
||||||
var stderr bytes.Buffer
|
var stderr bytes.Buffer
|
||||||
code := Execute(
|
code := Execute(
|
||||||
[]string{"run-stage", "polish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
[]string{"run-stage", "extract", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
||||||
&stdout,
|
&stdout,
|
||||||
&stderr,
|
&stderr,
|
||||||
)
|
)
|
||||||
@@ -120,31 +120,32 @@ func TestRunStageArtifactsDoesNotImplyForce(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResumeArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
|
func TestRunArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||||
|
|
||||||
store := &manifest.LocalStore{}
|
store := &manifest.LocalStore{}
|
||||||
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "publish", "notify"} {
|
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
|
||||||
seed.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
seed.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||||
}
|
}
|
||||||
|
seed.MarkStageSkipped("extract", time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), "notarius_disabled")
|
||||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||||
t.Fatalf("save manifest: %v", err)
|
t.Fatalf("save manifest: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
err := Resume(
|
err := Run(
|
||||||
context.Background(),
|
context.Background(),
|
||||||
[]string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
[]string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
|
||||||
&out,
|
&out,
|
||||||
)
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "has no remaining stages") {
|
if !strings.Contains(out.String(), "executed=1 skipped=11") {
|
||||||
t.Fatalf("output = %q, want no remaining stages", out.String())
|
t.Fatalf("output = %q, want all stages skipped", out.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
var supportedCommands = []string{"run", "run-stage", "resume", "analyze", "publish", "clean", "session"}
|
var supportedCommands = []string{"run", "run-stage", "analyze", "publish", "clean", "session"}
|
||||||
|
|
||||||
// Execute dispatches CLI commands and returns a process exit code.
|
// Execute dispatches CLI commands and returns a process exit code.
|
||||||
func Execute(args []string, stdout, stderr io.Writer) int {
|
func Execute(args []string, stdout, stderr io.Writer) int {
|
||||||
@@ -24,8 +24,6 @@ func Execute(args []string, stdout, stderr io.Writer) int {
|
|||||||
switch cmd {
|
switch cmd {
|
||||||
case "run":
|
case "run":
|
||||||
err = Run(ctx, cmdArgs, stdout)
|
err = Run(ctx, cmdArgs, stdout)
|
||||||
case "resume":
|
|
||||||
err = Resume(ctx, cmdArgs, stdout)
|
|
||||||
case "run-stage":
|
case "run-stage":
|
||||||
err = RunStage(ctx, cmdArgs, stdout)
|
err = RunStage(ctx, cmdArgs, stdout)
|
||||||
case "analyze":
|
case "analyze":
|
||||||
@@ -53,8 +51,3 @@ func Execute(args []string, stdout, stderr io.Writer) int {
|
|||||||
func printUsage(w io.Writer) {
|
func printUsage(w io.Writer) {
|
||||||
fmt.Fprintf(w, "Usage: narratio <%s>\n", strings.Join(supportedCommands, "|"))
|
fmt.Fprintf(w, "Usage: narratio <%s>\n", strings.Join(supportedCommands, "|"))
|
||||||
}
|
}
|
||||||
|
|
||||||
func placeholder(out io.Writer, command string) error {
|
|
||||||
_, err := fmt.Fprintf(out, "narratio %s: not yet implemented\n", command)
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -31,10 +31,9 @@ func TestExecuteValidCommands(t *testing.T) {
|
|||||||
args []string
|
args []string
|
||||||
wantOut string
|
wantOut string
|
||||||
}{
|
}{
|
||||||
{name: "run", args: []string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run: session 2026-05-03; executed=9 skipped=0; manifest="},
|
{name: "run", args: []string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run: session 2026-05-03; executed=11 skipped=1; manifest="},
|
||||||
{name: "session plan", args: []string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "prepare: skip\ntranscribe: skip\nmerge: skip\npolish: skip\nnormalize: skip\ntrim: skip\nanalyze: skip\npublish: skip\nnotify: skip"},
|
{name: "session plan", args: []string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "prepare: skip\ntranscribe: skip\nmerge: skip\npolish: skip\nnormalize: skip\ntrim: skip\nextract: run\nrender: skip\nanalyze: skip\npublish: skip\nnotify: skip"},
|
||||||
{name: "session status", args: []string{"session", "status", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "Session: 2026-05-03"},
|
{name: "session status", args: []string{"session", "status", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "Session: 2026-05-03"},
|
||||||
{name: "resume", args: []string{"resume", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio resume: session 2026-05-03 has no remaining stages"},
|
|
||||||
{name: "run-stage", args: []string{"run-stage", "polish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run-stage: stage=polish executed=0 skipped=1 force=false; manifest="},
|
{name: "run-stage", args: []string{"run-stage", "polish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run-stage: stage=polish executed=0 skipped=1 force=false; manifest="},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -66,7 +65,7 @@ func TestExecuteMissingRequiredFlags(t *testing.T) {
|
|||||||
{name: "run missing session", args: []string{"run"}, want: "run: session_id is required"},
|
{name: "run missing session", args: []string{"run"}, want: "run: session_id is required"},
|
||||||
{name: "plan old top-level removed", args: []string{"plan"}, want: `unknown command: "plan"`},
|
{name: "plan old top-level removed", args: []string{"plan"}, want: `unknown command: "plan"`},
|
||||||
{name: "status old top-level removed", args: []string{"status"}, want: `unknown command: "status"`},
|
{name: "status old top-level removed", args: []string{"status"}, want: `unknown command: "status"`},
|
||||||
{name: "resume missing session", args: []string{"resume"}, want: "resume: session_id is required"},
|
{name: "resume removed", args: []string{"resume"}, want: `unknown command: "resume"`},
|
||||||
{name: "run-stage missing name", args: []string{"run-stage", "--config", "a", "--session", "b"}, want: "run-stage: expected stage name and session_id"},
|
{name: "run-stage missing name", args: []string{"run-stage", "--config", "a", "--session", "b"}, want: "run-stage: expected stage name and session_id"},
|
||||||
{name: "run-stage missing session", args: []string{"run-stage", "polish"}, want: "run-stage: expected stage name and session_id"},
|
{name: "run-stage missing session", args: []string{"run-stage", "polish"}, want: "run-stage: expected stage name and session_id"},
|
||||||
{name: "run missing config uses defaults", args: []string{"run", "2026-05-03", "--session", "session.yml"}, want: "run: no pipeline config path provided and no default pipeline config found; searched:"},
|
{name: "run missing config uses defaults", args: []string{"run", "2026-05-03", "--session", "session.yml"}, want: "run: no pipeline config path provided and no default pipeline config found; searched:"},
|
||||||
@@ -228,6 +227,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`
|
`
|
||||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||||
t.Fatalf("write pipeline.yml: %v", err)
|
t.Fatalf("write pipeline.yml: %v", err)
|
||||||
@@ -291,6 +292,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`
|
`
|
||||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||||
t.Fatalf("write pipeline.yml: %v", err)
|
t.Fatalf("write pipeline.yml: %v", err)
|
||||||
@@ -332,7 +335,7 @@ func TestExecuteUsesDefaultPipelineConfigPathWhenConfigFlagOmitted(t *testing.T)
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "narratio run: session 2026-05-03; executed=9 skipped=0; manifest=") {
|
if !strings.Contains(stdout.String(), "narratio run: session 2026-05-03; executed=11 skipped=1; manifest=") {
|
||||||
t.Fatalf("stdout = %q, want successful run output", stdout.String())
|
t.Fatalf("stdout = %q, want successful run output", stdout.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -386,10 +389,14 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`)
|
`)
|
||||||
mustWriteTestFile(t, filepath.Join(otherDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
mustWriteTestFile(t, filepath.Join(otherDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(otherDir, "autocorrect.yml"), "[]\n")
|
mustWriteTestFile(t, filepath.Join(otherDir, "autocorrect.yml"), "[]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(otherDir, "glossary.yml"), "[]\n")
|
mustWriteTestFile(t, filepath.Join(otherDir, "glossary.yml"), "[]\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(otherDir, "players.yml"), "[]\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(otherDir, "party.yml"), "[]\n")
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
var stdout bytes.Buffer
|
||||||
var stderr bytes.Buffer
|
var stderr bytes.Buffer
|
||||||
@@ -467,10 +474,13 @@ func writeValidConfigFiles(t *testing.T, workspaceRoot string, transcribeURL ...
|
|||||||
url = transcribeURL[0]
|
url = transcribeURL[0]
|
||||||
}
|
}
|
||||||
seriatimBinary := writeSeriatimAppTestWrapper(t)
|
seriatimBinary := writeSeriatimAppTestWrapper(t)
|
||||||
|
scriptoriumBinary := writeScriptoriumAppTestWrapper(t)
|
||||||
auditaBinary := writeAuditaAppTestWrapper(t)
|
auditaBinary := writeAuditaAppTestWrapper(t)
|
||||||
t.Setenv("GO_WANT_APP_SERIATIM_HELPER", "1")
|
t.Setenv("GO_WANT_APP_SERIATIM_HELPER", "1")
|
||||||
|
t.Setenv("GO_WANT_APP_SCRIPTORIUM_HELPER", "1")
|
||||||
t.Setenv("GO_WANT_APP_AUDITA_HELPER", "1")
|
t.Setenv("GO_WANT_APP_AUDITA_HELPER", "1")
|
||||||
t.Setenv("AUDITA_LLM_API_KEY", "test-audita-key")
|
t.Setenv("AUDITA_LLM_API_KEY", "test-audita-key")
|
||||||
|
t.Setenv("PATH", filepath.Dir(scriptoriumBinary)+string(os.PathListSeparator)+os.Getenv("PATH"))
|
||||||
|
|
||||||
pipelineYAML := `workspace:
|
pipelineYAML := `workspace:
|
||||||
root: ` + workspaceRoot + `
|
root: ` + workspaceRoot + `
|
||||||
@@ -515,6 +525,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`
|
`
|
||||||
|
|
||||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||||
@@ -533,6 +545,8 @@ inputs:
|
|||||||
mustWriteTestFile(t, filepath.Join(campaignDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
mustWriteTestFile(t, filepath.Join(campaignDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(campaignDir, "autocorrect.yml"), "[]\n")
|
mustWriteTestFile(t, filepath.Join(campaignDir, "autocorrect.yml"), "[]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(campaignDir, "glossary.yml"), "[]\n")
|
mustWriteTestFile(t, filepath.Join(campaignDir, "glossary.yml"), "[]\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(campaignDir, "players.yml"), "[]\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(campaignDir, "party.yml"), "[]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(dir, "audio", "alice.flac"), "audio-bytes")
|
mustWriteTestFile(t, filepath.Join(dir, "audio", "alice.flac"), "audio-bytes")
|
||||||
|
|
||||||
return pipelinePath, campaignPath, sessionPath
|
return pipelinePath, campaignPath, sessionPath
|
||||||
@@ -546,6 +560,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`
|
`
|
||||||
if err := os.WriteFile(campaignPath, []byte(campaignYAML), 0o644); err != nil {
|
if err := os.WriteFile(campaignPath, []byte(campaignYAML), 0o644); err != nil {
|
||||||
t.Fatalf("write campaign.yml: %v", err)
|
t.Fatalf("write campaign.yml: %v", err)
|
||||||
@@ -592,6 +608,60 @@ func writeSeriatimAppTestWrapper(t *testing.T) string {
|
|||||||
return path
|
return path
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writeScriptoriumAppTestWrapper(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
exe, err := os.Executable()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("os.Executable() error = %v", err)
|
||||||
|
}
|
||||||
|
path := filepath.Join(t.TempDir(), "scriptorium")
|
||||||
|
content := "#!/bin/sh\nexec \"" + exe + "\" -test.run=TestScriptoriumAppHelper -- \"$@\"\n"
|
||||||
|
if err := os.WriteFile(path, []byte(content), 0o755); err != nil {
|
||||||
|
t.Fatalf("WriteFile(%q): %v", path, err)
|
||||||
|
}
|
||||||
|
return path
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestScriptoriumAppHelper(t *testing.T) {
|
||||||
|
if os.Getenv("GO_WANT_APP_SCRIPTORIUM_HELPER") != "1" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
args := os.Args
|
||||||
|
start := -1
|
||||||
|
for i := range args {
|
||||||
|
if args[i] == "--" {
|
||||||
|
start = i + 1
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if start < 0 || start >= len(args) {
|
||||||
|
_, _ = os.Stderr.WriteString("missing -- args separator\n")
|
||||||
|
os.Exit(2)
|
||||||
|
}
|
||||||
|
runArgs := args[start:]
|
||||||
|
|
||||||
|
outputPath := appSeriatimFlagValue(runArgs, "--out")
|
||||||
|
if strings.TrimSpace(outputPath) == "" {
|
||||||
|
outputPath = appSeriatimFlagValue(runArgs, "--output")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(outputPath) == "" {
|
||||||
|
_, _ = os.Stderr.WriteString("missing output flag\n")
|
||||||
|
os.Exit(2)
|
||||||
|
}
|
||||||
|
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
|
||||||
|
_, _ = os.Stderr.WriteString(fmt.Sprintf("mkdir output dir: %v\n", err))
|
||||||
|
os.Exit(2)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(outputPath, []byte(`{"trim_action":"copy","warnings":[]}`), 0o644); err != nil {
|
||||||
|
_, _ = os.Stderr.WriteString(fmt.Sprintf("write output: %v\n", err))
|
||||||
|
os.Exit(2)
|
||||||
|
}
|
||||||
|
_, _ = os.Stdout.WriteString("scriptorium helper stdout\n")
|
||||||
|
_, _ = os.Stderr.WriteString("scriptorium helper stderr\n")
|
||||||
|
os.Exit(0)
|
||||||
|
}
|
||||||
|
|
||||||
func TestSeriatimAppHelper(t *testing.T) {
|
func TestSeriatimAppHelper(t *testing.T) {
|
||||||
if os.Getenv("GO_WANT_APP_SERIATIM_HELPER") != "1" {
|
if os.Getenv("GO_WANT_APP_SERIATIM_HELPER") != "1" {
|
||||||
return
|
return
|
||||||
|
|||||||
@@ -4,7 +4,6 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
@@ -59,7 +58,7 @@ func loadCommandConfig(ctx context.Context, pipelineFlag, campaignFlag, campaign
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, missingSessionConfigError(discoveredSession.Searched, err.Error())
|
return nil, missingSessionConfigError(discoveredSession.Searched, err.Error())
|
||||||
}
|
}
|
||||||
sessionTempPath, err := downloadRemoteSessionConfig(ctx, store, remoteKey)
|
sessionTempPath, err := storage.DownloadObjectToTemp(ctx, store, remoteKey, "narratio-session-*.yml")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, missingSessionConfigError(discoveredSession.Searched, fmt.Sprintf("remote session %q download failed: %v", remoteKey, err))
|
return nil, missingSessionConfigError(discoveredSession.Searched, fmt.Sprintf("remote session %q download failed: %v", remoteKey, err))
|
||||||
}
|
}
|
||||||
@@ -134,21 +133,6 @@ func findRemoteSessionConfig(ctx context.Context, store storage.ObjectStore, ses
|
|||||||
return storage.ObjectInfo{}, fmt.Errorf("remote session %q not found", remoteKey)
|
return storage.ObjectInfo{}, fmt.Errorf("remote session %q not found", remoteKey)
|
||||||
}
|
}
|
||||||
|
|
||||||
func downloadRemoteSessionConfig(ctx context.Context, store storage.ObjectStore, remoteKey string) (string, error) {
|
|
||||||
f, err := os.CreateTemp("", "narratio-session-*.yml")
|
|
||||||
if err != nil {
|
|
||||||
return "", fmt.Errorf("create temp file: %w", err)
|
|
||||||
}
|
|
||||||
path := f.Name()
|
|
||||||
if err := f.Close(); err != nil {
|
|
||||||
return "", fmt.Errorf("close temp file %q: %w", path, err)
|
|
||||||
}
|
|
||||||
if err := store.Download(ctx, remoteKey, path); err != nil {
|
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
return filepath.Clean(path), nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func s3BucketName(cfg *config.PipelineConfig) string {
|
func s3BucketName(cfg *config.PipelineConfig) string {
|
||||||
if cfg == nil || cfg.Storage.S3 == nil {
|
if cfg == nil || cfg.Storage.S3 == nil {
|
||||||
return ""
|
return ""
|
||||||
|
|||||||
416
internal/app/extract_lifecycle_test.go
Normal file
416
internal/app/extract_lifecycle_test.go
Normal file
@@ -0,0 +1,416 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/notarius"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||||
|
)
|
||||||
|
|
||||||
|
type materializingNotariusRunner struct {
|
||||||
|
cfg *config.NotariusConfig
|
||||||
|
requests []notarius.RunRequest
|
||||||
|
failuresRemaining int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *materializingNotariusRunner) Run(_ context.Context, req notarius.RunRequest) (notarius.RunResult, error) {
|
||||||
|
r.requests = append(r.requests, req)
|
||||||
|
if r.failuresRemaining > 0 {
|
||||||
|
r.failuresRemaining--
|
||||||
|
return notarius.RunResult{}, errors.New("notarius execution failed")
|
||||||
|
}
|
||||||
|
externalRunID := fmt.Sprintf("notarius-run-%d", len(r.requests))
|
||||||
|
bundle := filepath.Join(req.OutputRoot, externalRunID)
|
||||||
|
lanesDir := filepath.Join(bundle, "lanes")
|
||||||
|
if err := os.MkdirAll(lanesDir, 0o755); err != nil {
|
||||||
|
return notarius.RunResult{}, err
|
||||||
|
}
|
||||||
|
for path, content := range map[string]string{
|
||||||
|
filepath.Join(bundle, "index.json"): `{"manifest_file":"manifest.json"}`,
|
||||||
|
filepath.Join(bundle, "manifest.json"): `{}`,
|
||||||
|
filepath.Join(bundle, "rejected.json"): `{"rejected":[]}`,
|
||||||
|
filepath.Join(bundle, "warnings.json"): `{"warnings":[]}`,
|
||||||
|
filepath.Join(lanesDir, "npcs.json"): `{"npcs":[]}`,
|
||||||
|
} {
|
||||||
|
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||||
|
return notarius.RunResult{}, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
output := r.cfg.Outputs["npc_registry"]
|
||||||
|
return notarius.RunResult{
|
||||||
|
Receipt: notarius.Receipt{
|
||||||
|
SchemaVersion: notarius.ReceiptSchemaVersion, RunID: externalRunID,
|
||||||
|
PipelineID: req.PipelineID, OutputDirectory: bundle, IndexFile: "index.json",
|
||||||
|
NormalizedOutputCount: 1, ValidationStatus: "valid",
|
||||||
|
},
|
||||||
|
BundleRoot: bundle,
|
||||||
|
Index: notarius.Index{
|
||||||
|
Path: filepath.Join(bundle, "index.json"), RejectedPath: filepath.Join(bundle, "rejected.json"),
|
||||||
|
WarningsPath: filepath.Join(bundle, "warnings.json"),
|
||||||
|
Lanes: []notarius.LaneDescriptor{{
|
||||||
|
LaneID: output.LaneID, File: "lanes/npcs.json", Path: filepath.Join(lanesDir, "npcs.json"),
|
||||||
|
MediaType: output.MediaType, SchemaID: output.SchemaID,
|
||||||
|
SchemaVersion: output.SchemaVersion, ModuleKey: output.ModuleKey,
|
||||||
|
}},
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleDisabledThenEnabled(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, false)
|
||||||
|
plan, err := BuildSingleStagePlan("extract")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("BuildSingleStagePlan(extract) error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
first, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("disabled executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(first.Executed) != 1 || len(first.Skipped) != 1 || first.Skipped[0] != "extract" || len(runner.requests) != 0 {
|
||||||
|
t.Fatalf("disabled summary = %#v requests=%d", first, len(runner.requests))
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg.Pipeline.Notarius.Enabled = true
|
||||||
|
second, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("enabled executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(second.Executed) != 1 || len(second.Skipped) != 0 || len(runner.requests) != 1 {
|
||||||
|
t.Fatalf("enabled summary = %#v requests=%d", second, len(runner.requests))
|
||||||
|
}
|
||||||
|
loaded, err := (&manifest.LocalStore{}).Load(context.Background(), second.ManifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() error = %v", err)
|
||||||
|
}
|
||||||
|
if loaded.Stages["extract"] == nil || loaded.Stages["extract"].Status != manifest.StatusSucceeded || len(loaded.Stages["extract"].Outputs) != 2 {
|
||||||
|
t.Fatalf("extract record = %#v, want succeeded manifest-ready outputs", loaded.Stages["extract"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleChangedOutcomeRerunsSucceededDownstream(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, false)
|
||||||
|
analyzeRuns := 0
|
||||||
|
plan := extractionLifecyclePlan(t, &analyzeRuns)
|
||||||
|
|
||||||
|
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env}); err != nil {
|
||||||
|
t.Fatalf("disabled executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if analyzeRuns != 1 || len(runner.requests) != 0 {
|
||||||
|
t.Fatalf("disabled run analyze=%d Notarius=%d, want 1 and 0", analyzeRuns, len(runner.requests))
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg.Pipeline.Notarius.Enabled = true
|
||||||
|
summary, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("enabled executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if analyzeRuns != 2 || len(runner.requests) != 1 {
|
||||||
|
t.Fatalf("enabled run analyze=%d Notarius=%d, want 2 and 1", analyzeRuns, len(runner.requests))
|
||||||
|
}
|
||||||
|
if len(summary.Executed) != 2 || len(summary.Skipped) != 0 {
|
||||||
|
t.Fatalf("enabled summary = %#v, want extract and analyze executed", summary)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleFailureInvalidatesAndOrdinaryRetryRerunsDownstream(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, true)
|
||||||
|
runner.failuresRemaining = 1
|
||||||
|
markLifecycleStageSucceeded(t, cfg, "analyze")
|
||||||
|
analyzeRuns := 0
|
||||||
|
plan := extractionLifecyclePlan(t, &analyzeRuns)
|
||||||
|
|
||||||
|
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env}); err == nil || !strings.Contains(err.Error(), "notarius execution failed") {
|
||||||
|
t.Fatalf("failed executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
failed := loadLifecycleManifest(t, cfg)
|
||||||
|
if failed.Stages["extract"].Status != manifest.StatusFailed || failed.Stages["analyze"].Status != manifest.StatusStale {
|
||||||
|
t.Fatalf("failed lifecycle extract=%#v analyze=%#v", failed.Stages["extract"], failed.Stages["analyze"])
|
||||||
|
}
|
||||||
|
if failed.Stages["analyze"].Error == nil || failed.Stages["analyze"].Error.Message != staleReasonFailure {
|
||||||
|
t.Fatalf("analyze stale reason = %#v, want %q", failed.Stages["analyze"].Error, staleReasonFailure)
|
||||||
|
}
|
||||||
|
|
||||||
|
summary, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("retry executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if analyzeRuns != 1 || len(runner.requests) != 2 || len(summary.Executed) != 2 {
|
||||||
|
t.Fatalf("retry analyze=%d Notarius=%d summary=%#v", analyzeRuns, len(runner.requests), summary)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleForcedSelfSkipInvalidatesDownstream(t *testing.T) {
|
||||||
|
cfg, env, _ := extractionLifecycleFixture(t, false)
|
||||||
|
markLifecycleStageSucceeded(t, cfg, "analyze")
|
||||||
|
plan, _ := BuildSingleStagePlan("extract")
|
||||||
|
|
||||||
|
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env, Force: true}); err != nil {
|
||||||
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
loaded := loadLifecycleManifest(t, cfg)
|
||||||
|
if loaded.Stages["extract"].Status != manifest.StatusSkipped || loaded.Stages["analyze"].Status != manifest.StatusStale {
|
||||||
|
t.Fatalf("forced self-skip extract=%#v analyze=%#v", loaded.Stages["extract"], loaded.Stages["analyze"])
|
||||||
|
}
|
||||||
|
if loaded.Stages["analyze"].Error == nil || loaded.Stages["analyze"].Error.Message != staleReasonForcedReplacement {
|
||||||
|
t.Fatalf("analyze stale reason = %#v, want %q", loaded.Stages["analyze"].Error, staleReasonForcedReplacement)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleForcedFailureInvalidatesDownstream(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, true)
|
||||||
|
plan, _ := BuildSingleStagePlan("extract")
|
||||||
|
|
||||||
|
first, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("initial executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
succeeded := loadLifecycleManifest(t, cfg).Stages["extract"]
|
||||||
|
if succeeded == nil || succeeded.Status != manifest.StatusSucceeded || len(succeeded.Outputs) == 0 || len(succeeded.Logs) == 0 || len(succeeded.Metadata) == 0 {
|
||||||
|
t.Fatalf("initial extraction result = %#v, want succeeded result details", succeeded)
|
||||||
|
}
|
||||||
|
|
||||||
|
markLifecycleStageSucceeded(t, cfg, "analyze")
|
||||||
|
runner.failuresRemaining = 1
|
||||||
|
|
||||||
|
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env, Force: true}); err == nil {
|
||||||
|
t.Fatal("executeStages() error = nil, want forced extraction failure")
|
||||||
|
}
|
||||||
|
loaded := loadLifecycleManifest(t, cfg)
|
||||||
|
if loaded.Stages["extract"].Status != manifest.StatusFailed || loaded.Stages["analyze"].Status != manifest.StatusStale {
|
||||||
|
t.Fatalf("forced failure extract=%#v analyze=%#v", loaded.Stages["extract"], loaded.Stages["analyze"])
|
||||||
|
}
|
||||||
|
if loaded.Stages["analyze"].Error == nil || loaded.Stages["analyze"].Error.Message != staleReasonForcedReplacement {
|
||||||
|
t.Fatalf("analyze stale reason = %#v, want %q", loaded.Stages["analyze"].Error, staleReasonForcedReplacement)
|
||||||
|
}
|
||||||
|
failed := loaded.Stages["extract"]
|
||||||
|
if len(failed.Outputs) != 0 || len(failed.Logs) != 0 || len(failed.GeneratedConfigs) != 0 || len(failed.Metadata) != 0 {
|
||||||
|
t.Fatalf("failed replacement inherited extraction result details: %#v", failed)
|
||||||
|
}
|
||||||
|
historical, err := (&manifest.LocalStore{}).LoadRun(context.Background(), first.RunManifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("LoadRun(initial) error = %v", err)
|
||||||
|
}
|
||||||
|
historicalExtract := historical.Stages["extract"]
|
||||||
|
if historicalExtract == nil || historicalExtract.Status != manifest.StatusSucceeded || len(historicalExtract.Outputs) == 0 || len(historicalExtract.Logs) == 0 || len(historicalExtract.Metadata) == 0 {
|
||||||
|
t.Fatalf("historical extraction result = %#v, want preserved succeeded details", historicalExtract)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(succeeded.Outputs[0].LocalPath); err != nil {
|
||||||
|
t.Fatalf("durable extraction output was not preserved: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleRepeatedSelfSkipPreservesSucceededDownstream(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, false)
|
||||||
|
analyzeRuns := 0
|
||||||
|
plan := extractionLifecyclePlan(t, &analyzeRuns)
|
||||||
|
|
||||||
|
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env}); err != nil {
|
||||||
|
t.Fatalf("first executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
second, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("second executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if analyzeRuns != 1 || len(runner.requests) != 0 {
|
||||||
|
t.Fatalf("repeated disabled run analyze=%d Notarius=%d, want 1 and 0", analyzeRuns, len(runner.requests))
|
||||||
|
}
|
||||||
|
if len(second.Executed) != 1 || len(second.Skipped) != 2 {
|
||||||
|
t.Fatalf("second summary = %#v, want executed self-skip and skipped analyze", second)
|
||||||
|
}
|
||||||
|
loaded := loadLifecycleManifest(t, cfg)
|
||||||
|
if loaded.Stages["analyze"].Status != manifest.StatusSucceeded {
|
||||||
|
t.Fatalf("analyze = %#v, want succeeded", loaded.Stages["analyze"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleSkipsCurrentResumableResult(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, true)
|
||||||
|
analyzeRuns := 0
|
||||||
|
plan := extractionLifecyclePlan(t, &analyzeRuns)
|
||||||
|
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env}); err != nil {
|
||||||
|
t.Fatalf("first executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
resumed, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("resume executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(resumed.Executed) != 0 || len(resumed.Skipped) != 2 || len(runner.requests) != 1 || analyzeRuns != 1 {
|
||||||
|
t.Fatalf("resume summary = %#v requests=%d analyze=%d", resumed, len(runner.requests), analyzeRuns)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleResumesAndRerunsObsoleteResults(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
mutate func(*testing.T, *config.Config, *manifest.Manifest)
|
||||||
|
}{
|
||||||
|
{name: "configuration changed", mutate: func(_ *testing.T, cfg *config.Config, _ *manifest.Manifest) {
|
||||||
|
output := cfg.Pipeline.Notarius.Outputs["npc_registry"]
|
||||||
|
output.SchemaVersion = "v2"
|
||||||
|
cfg.Pipeline.Notarius.Outputs["npc_registry"] = output
|
||||||
|
}},
|
||||||
|
{name: "payload missing", mutate: func(t *testing.T, _ *config.Config, m *manifest.Manifest) {
|
||||||
|
if err := os.Remove(m.Stages["extract"].Outputs[0].LocalPath); err != nil {
|
||||||
|
t.Fatalf("Remove() error = %v", err)
|
||||||
|
}
|
||||||
|
}},
|
||||||
|
{name: "payload tampered", mutate: func(t *testing.T, _ *config.Config, m *manifest.Manifest) {
|
||||||
|
if err := os.WriteFile(m.Stages["extract"].Outputs[0].LocalPath, []byte(`{"npcs":["tampered"]}`), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile() error = %v", err)
|
||||||
|
}
|
||||||
|
}},
|
||||||
|
{name: "record incompatible", mutate: func(_ *testing.T, _ *config.Config, m *manifest.Manifest) {
|
||||||
|
m.Stages["extract"].Outputs[0].Contract.SchemaID = "incompatible"
|
||||||
|
}},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, true)
|
||||||
|
plan, _ := BuildSingleStagePlan("extract")
|
||||||
|
first, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("first executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
persisted, err := (&manifest.LocalStore{}).Load(context.Background(), first.ManifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
test.mutate(t, cfg, persisted)
|
||||||
|
if err := (&manifest.LocalStore{}).Save(context.Background(), first.ManifestPath, persisted); err != nil {
|
||||||
|
t.Fatalf("Save(mutated) error = %v", err)
|
||||||
|
}
|
||||||
|
rerun, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("rerun executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(rerun.Executed) != 1 || len(rerun.Skipped) != 0 || len(runner.requests) != 2 {
|
||||||
|
t.Fatalf("rerun summary = %#v requests=%d", rerun, len(runner.requests))
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractLifecycleUnsafeResumeErrorPreservesSuccess(t *testing.T) {
|
||||||
|
cfg, env, runner := extractionLifecycleFixture(t, true)
|
||||||
|
analyzeRuns := 0
|
||||||
|
plan := extractionLifecyclePlan(t, &analyzeRuns)
|
||||||
|
first, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("first executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
store := &manifest.LocalStore{}
|
||||||
|
persisted, err := store.Load(context.Background(), first.ManifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() error = %v", err)
|
||||||
|
}
|
||||||
|
persisted.Stages["extract"].Outputs[0].LocalPath = filepath.Join(cfg.Pipeline.Workspace.Root, "outside.json")
|
||||||
|
if err := store.Save(context.Background(), first.ManifestPath, persisted); err != nil {
|
||||||
|
t.Fatalf("Save() error = %v", err)
|
||||||
|
}
|
||||||
|
before, _ := json.Marshal(map[string]*manifest.StageRecord{
|
||||||
|
"extract": persisted.Stages["extract"],
|
||||||
|
"analyze": persisted.Stages["analyze"],
|
||||||
|
})
|
||||||
|
|
||||||
|
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env}); err == nil || !strings.Contains(err.Error(), "unsafe") {
|
||||||
|
t.Fatalf("executeStages() error = %v, want unsafe resume failure", err)
|
||||||
|
}
|
||||||
|
afterManifest, err := store.Load(context.Background(), first.ManifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load(after) error = %v", err)
|
||||||
|
}
|
||||||
|
after, _ := json.Marshal(map[string]*manifest.StageRecord{
|
||||||
|
"extract": afterManifest.Stages["extract"],
|
||||||
|
"analyze": afterManifest.Stages["analyze"],
|
||||||
|
})
|
||||||
|
if string(before) != string(after) || len(runner.requests) != 1 || analyzeRuns != 1 {
|
||||||
|
t.Fatalf("successful records changed: before=%s after=%s requests=%d analyze=%d", before, after, len(runner.requests), analyzeRuns)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func extractionLifecyclePlan(t *testing.T, analyzeRuns *int) []stage.Stage {
|
||||||
|
t.Helper()
|
||||||
|
plan, err := BuildSingleStagePlan("extract")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("BuildSingleStagePlan(extract) error = %v", err)
|
||||||
|
}
|
||||||
|
return append(plan, countingStage{name: "analyze", runs: analyzeRuns})
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadLifecycleManifest(t *testing.T, cfg *config.Config) *manifest.Manifest {
|
||||||
|
t.Helper()
|
||||||
|
loaded, err := (&manifest.LocalStore{}).Load(context.Background(), manifestPathFor(cfg))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() error = %v", err)
|
||||||
|
}
|
||||||
|
return loaded
|
||||||
|
}
|
||||||
|
|
||||||
|
func markLifecycleStageSucceeded(t *testing.T, cfg *config.Config, name string) {
|
||||||
|
t.Helper()
|
||||||
|
loaded := loadLifecycleManifest(t, cfg)
|
||||||
|
loaded.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
||||||
|
if err := (&manifest.LocalStore{}).Save(context.Background(), manifestPathFor(cfg), loaded); err != nil {
|
||||||
|
t.Fatalf("Save() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func extractionLifecycleFixture(t *testing.T, enabled bool) (*config.Config, *stage.Env, *materializingNotariusRunner) {
|
||||||
|
t.Helper()
|
||||||
|
cfg := testConfig(t)
|
||||||
|
root := cfg.Pipeline.Workspace.Root
|
||||||
|
binary := filepath.Join(root, "notarius")
|
||||||
|
configPath := filepath.Join(root, "notarius.yml")
|
||||||
|
workingDirectory := filepath.Join(root, "notarius-work")
|
||||||
|
if err := os.WriteFile(binary, []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil {
|
||||||
|
t.Fatalf("WriteFile(binary) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(configPath, []byte("pipelines: {}\n"), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(config) error = %v", err)
|
||||||
|
}
|
||||||
|
if err := os.Mkdir(workingDirectory, 0o755); err != nil {
|
||||||
|
t.Fatalf("Mkdir(working directory) error = %v", err)
|
||||||
|
}
|
||||||
|
cfg.Pipeline.Notarius = &config.NotariusConfig{
|
||||||
|
Enabled: enabled, Binary: binary, ConfigPath: configPath, PipelineID: "dnd-session",
|
||||||
|
Timeout: "45m", WorkingDirectory: workingDirectory,
|
||||||
|
Outputs: map[string]config.NotariusOutputConfig{
|
||||||
|
"npc_registry": {
|
||||||
|
LaneID: "npc-registry", MediaType: "application/json", SchemaID: "notarius.dnd.npc_registry",
|
||||||
|
SchemaVersion: "v1", ModuleKey: "dnd/npc-registry",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
paths, err := artifacts.NewLocalStore(root).EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("EnsureLayoutFor() error = %v", err)
|
||||||
|
}
|
||||||
|
inputPath := filepath.Join(paths.ArtifactsDir, "trimmed.from-manifest.json")
|
||||||
|
if err := os.WriteFile(inputPath, []byte(`{"segments":[]}`), 0o644); err != nil {
|
||||||
|
t.Fatalf("WriteFile(input) error = %v", err)
|
||||||
|
}
|
||||||
|
m := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
||||||
|
m.Campaign = cfg.Session.Campaign
|
||||||
|
m.MarkStageSucceeded("trim", time.Now().UTC(), []manifest.ArtifactRecord{{
|
||||||
|
Kind: artifactmodel.TranscriptOutputKindFinalTrimmed, SourceID: artifactmodel.SourceTranscriptFinalTrimmed,
|
||||||
|
LocalPath: inputPath,
|
||||||
|
}})
|
||||||
|
if err := (&manifest.LocalStore{}).Save(context.Background(), paths.ManifestPath, m); err != nil {
|
||||||
|
t.Fatalf("Save(seed) error = %v", err)
|
||||||
|
}
|
||||||
|
runner := &materializingNotariusRunner{cfg: cfg.Pipeline.Notarius}
|
||||||
|
return cfg, &stage.Env{Notarius: runner}, runner
|
||||||
|
}
|
||||||
@@ -10,9 +10,10 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
func buildHelperArtifactCatalog(cfg *config.Config) (*artifacts.ArtifactCatalog, error) {
|
func buildHelperArtifactCatalog(cfg *config.Config, m *manifest.Manifest) (*artifacts.ArtifactCatalog, error) {
|
||||||
catalog := artifacts.NewArtifactCatalog()
|
catalog := artifacts.NewArtifactCatalog()
|
||||||
if err := catalog.RegisterBuiltIns(); err != nil {
|
if err := catalog.RegisterBuiltIns(); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@@ -26,25 +27,36 @@ func buildHelperArtifactCatalog(cfg *config.Config) (*artifacts.ArtifactCatalog,
|
|||||||
if err := catalog.RegisterConfiguredArtifacts(configured, nil); err != nil {
|
if err := catalog.RegisterConfiguredArtifacts(configured, nil); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
extractionDefinitions := artifacts.ExtractionDefinitionsFromConfig(cfg.Pipeline.Notarius)
|
||||||
|
if err := catalog.RegisterExtractionArtifacts(extractionDefinitions); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
if cfg.Pipeline.Notarius != nil && cfg.Pipeline.Notarius.Enabled {
|
||||||
|
catalog.HydrateExtractionArtifacts(paths, m, extractionDefinitions)
|
||||||
|
}
|
||||||
return catalog, nil
|
return catalog, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeArtifactList(out io.Writer, cfg *config.Config, catalog *artifacts.ArtifactCatalog, locks *effectiveLocks, publishedRemoteState map[string]string) {
|
func writeArtifactList(out io.Writer, cfg *config.Config, catalog *artifacts.ArtifactCatalog, locks *effectiveLocks, publishedRemoteState map[string]string) {
|
||||||
lockSet := lockSourceSet(locks.All)
|
lockSet := lockSourceSet(locks.All)
|
||||||
fmt.Fprintln(out, "Built-in:")
|
fmt.Fprintln(out, "Built-in:")
|
||||||
for _, id := range []string{
|
for _, transcript := range artifacts.RuntimeTranscriptArtifacts() {
|
||||||
artifacts.ArtifactTranscriptBase,
|
writeArtifactLine(out, transcript.SourceID, lockSet)
|
||||||
artifacts.ArtifactTranscriptPolished,
|
|
||||||
artifacts.ArtifactTranscriptFinal,
|
|
||||||
artifacts.ArtifactTranscriptFinalTrimmed,
|
|
||||||
artifacts.ArtifactBoundsSession,
|
|
||||||
} {
|
|
||||||
writeArtifactLine(out, id, lockSet)
|
|
||||||
}
|
}
|
||||||
|
writeArtifactLine(out, artifacts.ArtifactBoundsSession, lockSet)
|
||||||
fmt.Fprintln(out, "Configured:")
|
fmt.Fprintln(out, "Configured:")
|
||||||
for _, entry := range catalog.ListConfigured() {
|
for _, entry := range catalog.ListConfigured() {
|
||||||
writeArtifactLine(out, entry.SourceID, lockSet)
|
writeArtifactLine(out, entry.SourceID, lockSet)
|
||||||
}
|
}
|
||||||
|
fmt.Fprintln(out, "Extraction:")
|
||||||
|
for _, entry := range catalog.ListExtraction() {
|
||||||
|
state := "unavailable"
|
||||||
|
if entry.Available {
|
||||||
|
state = "available"
|
||||||
|
}
|
||||||
|
writeExtractionArtifactLine(out, entry.SourceID, state, entry.Provenance, lockSet)
|
||||||
|
}
|
||||||
fmt.Fprintln(out, "Previous-session:")
|
fmt.Fprintln(out, "Previous-session:")
|
||||||
for _, req := range artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg)) {
|
for _, req := range artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg)) {
|
||||||
fmt.Fprintf(out, "- %s required=%t\n", artifactpolicy.PreviousSessionSourceID(req.Name), req.Required)
|
fmt.Fprintf(out, "- %s required=%t\n", artifactpolicy.PreviousSessionSourceID(req.Name), req.Required)
|
||||||
@@ -55,6 +67,17 @@ func writeArtifactList(out io.Writer, cfg *config.Config, catalog *artifacts.Art
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writeExtractionArtifactLine(out io.Writer, source, state, provenance string, lockSet map[string]config.PublishLockRule) {
|
||||||
|
parts := []string{source, "planned", state}
|
||||||
|
if strings.TrimSpace(provenance) != "" {
|
||||||
|
parts = append(parts, "provenance="+strings.TrimSpace(provenance))
|
||||||
|
}
|
||||||
|
if _, ok := lockSet[source]; ok {
|
||||||
|
parts = append(parts, "locked")
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
|
||||||
|
}
|
||||||
|
|
||||||
func writeArtifactLine(out io.Writer, source string, lockSet map[string]config.PublishLockRule) {
|
func writeArtifactLine(out io.Writer, source string, lockSet map[string]config.PublishLockRule) {
|
||||||
parts := []string{source}
|
parts := []string{source}
|
||||||
if _, ok := lockSet[source]; ok {
|
if _, ok := lockSet[source]; ok {
|
||||||
@@ -108,7 +131,12 @@ func remotePublishedOutputAvailability(ctx context.Context, cfg *config.Config,
|
|||||||
|
|
||||||
func helperPublishedOutputDest(rule config.PublishOutputRule, catalog *artifacts.ArtifactCatalog) (string, bool, error) {
|
func helperPublishedOutputDest(rule config.PublishOutputRule, catalog *artifacts.ArtifactCatalog) (string, bool, error) {
|
||||||
source := strings.TrimSpace(rule.Source)
|
source := strings.TrimSpace(rule.Source)
|
||||||
normalized, err := artifactpolicy.ResolvePublishedDestination(source, rule.Dest, helperConfiguredOutputPathMap(catalog))
|
normalized, err := artifactpolicy.ResolvePublishedDestinationWithExtractions(
|
||||||
|
source,
|
||||||
|
rule.Dest,
|
||||||
|
helperConfiguredOutputPathMap(catalog),
|
||||||
|
helperExtractionOutputSet(catalog),
|
||||||
|
)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", false, err
|
return "", false, err
|
||||||
}
|
}
|
||||||
@@ -117,6 +145,19 @@ func helperPublishedOutputDest(rule config.PublishOutputRule, catalog *artifacts
|
|||||||
return normalized, showDest, nil
|
return normalized, showDest, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func helperExtractionOutputSet(catalog *artifacts.ArtifactCatalog) map[string]struct{} {
|
||||||
|
out := map[string]struct{}{}
|
||||||
|
if catalog == nil {
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
for _, entry := range catalog.ListExtraction() {
|
||||||
|
if strings.TrimSpace(entry.ExtractionKey) != "" {
|
||||||
|
out[entry.ExtractionKey] = struct{}{}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
func helperConfiguredOutputPathMap(catalog *artifacts.ArtifactCatalog) map[string]string {
|
func helperConfiguredOutputPathMap(catalog *artifacts.ArtifactCatalog) map[string]string {
|
||||||
out := map[string]string{}
|
out := map[string]string{}
|
||||||
if catalog == nil {
|
if catalog == nil {
|
||||||
|
|||||||
@@ -22,11 +22,11 @@ func ArtifactsList(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
if strings.TrimSpace(flags.sessionID) == "" {
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
return fmt.Errorf("artifacts list: session_id is required")
|
return fmt.Errorf("artifacts list: session_id is required")
|
||||||
}
|
}
|
||||||
cfg, store, locks, _, err := loadHelperContext(ctx, flags, remote)
|
cfg, store, locks, m, err := loadHelperContext(ctx, flags, remote)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("artifacts list: %w", err)
|
return fmt.Errorf("artifacts list: %w", err)
|
||||||
}
|
}
|
||||||
catalog, err := buildHelperArtifactCatalog(cfg)
|
catalog, err := buildHelperArtifactCatalog(cfg, m)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("artifacts list: %w", err)
|
return fmt.Errorf("artifacts list: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -10,7 +10,6 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
@@ -64,26 +63,18 @@ func sessionSourceSummary(cfg *config.Config) string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func validateStableInputFindings(cfg *config.Config) []finding {
|
func validateStableInputFindings(cfg *config.Config) []finding {
|
||||||
items := []struct {
|
checks := inspectStableInputs(cfg)
|
||||||
name string
|
out := make([]finding, 0, len(checks))
|
||||||
in config.ResolvedInputFile
|
for _, check := range checks {
|
||||||
}{
|
if check.Err != nil {
|
||||||
{"speakers", cfg.StableInputs.SpeakersFile},
|
msg := check.Name + ": " + check.Err.Error()
|
||||||
{"autocorrect", cfg.StableInputs.AutocorrectFile},
|
if strings.TrimSpace(check.Path) != "" {
|
||||||
{"glossary", cfg.StableInputs.GlossaryFile},
|
msg = fmt.Sprintf("%s missing: %v", check.Name, check.Err)
|
||||||
}
|
}
|
||||||
out := make([]finding, 0, len(items))
|
out = append(out, errorFinding("inputs", msg))
|
||||||
for _, item := range items {
|
|
||||||
path, err := resolveHelperConfigRelativePath(item.in)
|
|
||||||
if err != nil {
|
|
||||||
out = append(out, errorFinding("inputs", item.name+": "+err.Error()))
|
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if _, err := os.Stat(path); err != nil {
|
out = append(out, okFinding("inputs", check.Name+": "+check.Path))
|
||||||
out = append(out, errorFinding("inputs", fmt.Sprintf("%s missing: %v", item.name, err)))
|
|
||||||
} else {
|
|
||||||
out = append(out, okFinding("inputs", item.name+": "+path))
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
@@ -103,76 +94,22 @@ func resolveHelperConfigRelativePath(input config.ResolvedInputFile) (string, er
|
|||||||
}
|
}
|
||||||
|
|
||||||
func validateLocalAudioFindings(cfg *config.Config) []finding {
|
func validateLocalAudioFindings(cfg *config.Config) []finding {
|
||||||
if cfg.Session.Inputs.AudioS3 != nil {
|
check := inspectLocalAudioPresence(cfg)
|
||||||
|
if !check.Checked {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
audioDir := strings.TrimSpace(cfg.Session.Inputs.AudioDir)
|
if check.Err != nil {
|
||||||
if audioDir == "" && len(cfg.Session.Inputs.AudioFiles) == 0 {
|
return []finding{errorFinding("audio", check.Err.Error())}
|
||||||
return []finding{errorFinding("audio", "audio_dir, audio_files, or audio_s3 is required")}
|
|
||||||
}
|
}
|
||||||
base := filepath.Dir(cfg.SessionPath)
|
return []finding{okFinding("audio", fmt.Sprintf("%d local audio file(s)", len(check.Paths)))}
|
||||||
paths := []string{}
|
|
||||||
if audioDir != "" {
|
|
||||||
dir := audioDir
|
|
||||||
if !filepath.IsAbs(dir) {
|
|
||||||
dir = filepath.Join(base, dir)
|
|
||||||
}
|
|
||||||
matches, err := filepath.Glob(filepath.Join(dir, "*.flac"))
|
|
||||||
if err != nil || len(matches) == 0 {
|
|
||||||
return []finding{errorFinding("audio", "no .flac files found in "+dir)}
|
|
||||||
}
|
|
||||||
paths = append(paths, matches...)
|
|
||||||
}
|
|
||||||
for _, file := range cfg.Session.Inputs.AudioFiles {
|
|
||||||
p := file
|
|
||||||
if !filepath.IsAbs(p) {
|
|
||||||
p = filepath.Join(base, p)
|
|
||||||
}
|
|
||||||
paths = append(paths, p)
|
|
||||||
}
|
|
||||||
for _, p := range paths {
|
|
||||||
if _, err := os.Stat(p); err != nil {
|
|
||||||
return []finding{errorFinding("audio", fmt.Sprintf("audio file missing: %v", err))}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return []finding{okFinding("audio", fmt.Sprintf("%d local audio file(s)", len(paths)))}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func validateRemoteAudioFinding(ctx context.Context, cfg *config.Config, store storage.ObjectStore) finding {
|
func validateRemoteAudioFinding(ctx context.Context, cfg *config.Config, store storage.ObjectStore) finding {
|
||||||
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
check := inspectRemoteAudioPresence(ctx, cfg, store)
|
||||||
audioPrefix := artifacts.S3AudioPrefix(sessionPrefix, cfg.Session.Inputs.AudioS3.Prefix)
|
if check.Err != nil {
|
||||||
objects, err := store.List(ctx, audioPrefix)
|
return errorFinding("audio", check.Err.Error())
|
||||||
if err != nil {
|
|
||||||
return errorFinding("audio", err.Error())
|
|
||||||
}
|
}
|
||||||
count := 0
|
return okFinding("audio", fmt.Sprintf("%d remote .flac object(s)", len(check.Keys)))
|
||||||
for _, obj := range objects {
|
|
||||||
if strings.HasSuffix(strings.ToLower(obj.Key), ".flac") {
|
|
||||||
count++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if count == 0 {
|
|
||||||
return errorFinding("audio", "no remote .flac objects found under "+audioPrefix)
|
|
||||||
}
|
|
||||||
return okFinding("audio", fmt.Sprintf("%d remote .flac object(s)", count))
|
|
||||||
}
|
|
||||||
|
|
||||||
func validatePreviousArtifactFindings(ctx context.Context, cfg *config.Config, store storage.ObjectStore, requirements []artifacts.PreviousArtifactRequirement) []finding {
|
|
||||||
out := []finding{}
|
|
||||||
prefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
|
||||||
_, err := artifacts.LoadCurrentState(ctx, store, prefix, artifacts.CurrentStateValidation{
|
|
||||||
ExpectedSessionID: strings.TrimSpace(cfg.Session.PreviousSessionID),
|
|
||||||
ExpectedCampaign: strings.TrimSpace(cfg.Session.Campaign),
|
|
||||||
ValidateRunID: true,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
out = append(out, errorFinding("previous", fmt.Sprintf("remote %v", err)))
|
|
||||||
return out
|
|
||||||
}
|
|
||||||
for _, req := range requirements {
|
|
||||||
out = append(out, okFinding("previous", fmt.Sprintf("%s required=%t", req.Name, req.Required)))
|
|
||||||
}
|
|
||||||
return out
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func loadLocalManifest(ctx context.Context, path string) (*manifest.Manifest, error) {
|
func loadLocalManifest(ctx context.Context, path string) (*manifest.Manifest, error) {
|
||||||
|
|||||||
@@ -8,8 +8,10 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
@@ -109,12 +111,16 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`), 0o644); err != nil {
|
`), 0o644); err != nil {
|
||||||
t.Fatalf("write explicit campaign: %v", err)
|
t.Fatalf("write explicit campaign: %v", err)
|
||||||
}
|
}
|
||||||
mustWriteTestFile(t, filepath.Join(explicitDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
mustWriteTestFile(t, filepath.Join(explicitDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(explicitDir, "autocorrect.yml"), "[]\n")
|
mustWriteTestFile(t, filepath.Join(explicitDir, "autocorrect.yml"), "[]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(explicitDir, "glossary.yml"), "[]\n")
|
mustWriteTestFile(t, filepath.Join(explicitDir, "glossary.yml"), "[]\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(explicitDir, "players.yml"), "[]\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(explicitDir, "party.yml"), "[]\n")
|
||||||
|
|
||||||
fake := &storage.FakeBackend{}
|
fake := &storage.FakeBackend{}
|
||||||
var storeInitCalls int
|
var storeInitCalls int
|
||||||
@@ -445,6 +451,9 @@ inputs:
|
|||||||
if !strings.Contains(stdout.String(), "OK audio") {
|
if !strings.Contains(stdout.String(), "OK audio") {
|
||||||
t.Fatalf("stdout = %q, want OK audio", stdout.String())
|
t.Fatalf("stdout = %q, want OK audio", stdout.String())
|
||||||
}
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "OK inputs players:") || !strings.Contains(stdout.String(), "OK inputs party:") {
|
||||||
|
t.Fatalf("stdout = %q, want players and party input findings", stdout.String())
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestExecuteLocksAddListAndRemoveUseRemoteLockStore(t *testing.T) {
|
func TestExecuteLocksAddListAndRemoveUseRemoteLockStore(t *testing.T) {
|
||||||
@@ -501,7 +510,7 @@ func TestExecuteLocksAddListAndRemoveUseRemoteLockStore(t *testing.T) {
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("locks remove exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("locks remove exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
store, err := config.LoadPublishLockStoreBytes("locks.yml", fake.Objects[key].Data, nil)
|
store, err := config.LoadPublishLockStoreBytes("locks.yml", fake.Objects[key].Data, nil, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("LoadPublishLockStoreBytes() error = %v", err)
|
t.Fatalf("LoadPublishLockStoreBytes() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -593,6 +602,53 @@ func TestExecuteLocksRequireSessionID(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteLocksMutationRejectsSessionIDMismatch(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
args []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "add mismatch",
|
||||||
|
args: []string{
|
||||||
|
"session", "locks", "add", "2026-05-03", "narratio.transcript.final_trimmed",
|
||||||
|
"--session-id", "2026-05-04",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "remove mismatch",
|
||||||
|
args: []string{
|
||||||
|
"session", "locks", "remove", "2026-05-03", "narratio.transcript.final_trimmed",
|
||||||
|
"--session-id", "2026-05-04",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute(tt.args, &stdout, &stderr)
|
||||||
|
if code == 0 {
|
||||||
|
t.Fatal("exit code = 0, want non-zero")
|
||||||
|
}
|
||||||
|
if !strings.Contains(stderr.String(), "does not match expected session id") {
|
||||||
|
t.Fatalf("stderr = %q, want session-id mismatch guidance", stderr.String())
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteLocksCannotModifyStaticLocks(t *testing.T) {
|
func TestExecuteLocksCannotModifyStaticLocks(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
@@ -718,6 +774,71 @@ func TestExecuteArtifactsListRemoteReportsPublishedAvailability(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteArtifactsListReportsExtractionLifecycleWithoutPayload(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
addExtractionOutputToPipeline(t, pipelinePath)
|
||||||
|
addPublishOutputsToPipeline(t, pipelinePath, `
|
||||||
|
outputs:
|
||||||
|
- source: narratio.extraction.encounters
|
||||||
|
dest: artifacts/encounters.json
|
||||||
|
required: true
|
||||||
|
`)
|
||||||
|
lanePath := writeOperatorExtractionManifest(t, workspaceRoot)
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
publishedKey := artifacts.S3PublishedOutputKey(
|
||||||
|
artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"),
|
||||||
|
"artifacts/encounters.json",
|
||||||
|
)
|
||||||
|
fake.SeedObject(storage.FakeObject{Key: publishedKey, Data: []byte(`{"secret":"DO_NOT_PRINT"}`)})
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{
|
||||||
|
"session", "artifacts", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
"--remote",
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
out := stdout.String()
|
||||||
|
for _, want := range []string{
|
||||||
|
"Extraction:",
|
||||||
|
"narratio.extraction.encounters planned available provenance=manifest.current_extract_run",
|
||||||
|
"narratio.extraction.encounters dest=artifacts/encounters.json remote=published",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(out, want) {
|
||||||
|
t.Fatalf("stdout = %q, want %q", out, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if strings.Contains(out, "DO_NOT_PRINT") {
|
||||||
|
t.Fatalf("operator output exposed extraction payload: %q", out)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := os.Remove(lanePath); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
stdout.Reset()
|
||||||
|
stderr.Reset()
|
||||||
|
code = Execute([]string{
|
||||||
|
"session", "artifacts", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("unavailable exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "narratio.extraction.encounters planned unavailable") {
|
||||||
|
t.Fatalf("stdout = %q, want unavailable extraction state", stdout.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteArtifactsListRemoteUsesPublishOutputDestinations(t *testing.T) {
|
func TestExecuteArtifactsListRemoteUsesPublishOutputDestinations(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
@@ -815,6 +936,8 @@ func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
|
|||||||
"narratio.transcript.final_trimmed locked",
|
"narratio.transcript.final_trimmed locked",
|
||||||
"narratio.transcript.final_trimmed locked remote=published",
|
"narratio.transcript.final_trimmed locked remote=published",
|
||||||
"narratio.transcript.final dest=transcripts/full.json remote=published",
|
"narratio.transcript.final dest=transcripts/full.json remote=published",
|
||||||
|
"Stable input players:",
|
||||||
|
"Stable input party:",
|
||||||
} {
|
} {
|
||||||
if !strings.Contains(out, want) {
|
if !strings.Contains(out, want) {
|
||||||
t.Fatalf("stdout = %q, want %q", out, want)
|
t.Fatalf("stdout = %q, want %q", out, want)
|
||||||
@@ -884,6 +1007,31 @@ func TestExecuteStatusReportsMissingRemoteCurrentStateWithoutFailing(t *testing.
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteStatusReportsPreviousStateReadinessWithoutFailing(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
|
replaceInFileOrFatal(t, pipelinePath, "source: narratio.artifact.session_recap", "source: narratio.previous_session.artifact.session_recap")
|
||||||
|
replaceInFileOrFatal(t, sessionPath, "session_id: 2026-05-03\n", "session_id: 2026-05-03\nprevious_session_id: 2026-04-26\n")
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
var storeInitCalls int
|
||||||
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{
|
||||||
|
"session", "status", "2026-05-03",
|
||||||
|
"--config", pipelinePath,
|
||||||
|
"--campaign-file", campaignPath,
|
||||||
|
"--session", sessionPath,
|
||||||
|
}, &stdout, &stderr)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(stdout.String(), "Previous-session artifacts: unavailable: remote current run pointer missing") {
|
||||||
|
t.Fatalf("stdout = %q, want previous readiness unavailable line", stdout.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteSessionValidateReportsPreviousStateFindingAndReturnsFindingError(t *testing.T) {
|
func TestExecuteSessionValidateReportsPreviousStateFindingAndReturnsFindingError(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFilesWithScriptoriumArtifacts(t, workspaceRoot)
|
||||||
@@ -925,11 +1073,13 @@ func TestExecutePublishLoadsRemoteLocks(t *testing.T) {
|
|||||||
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
|
||||||
|
|
||||||
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze"} {
|
||||||
// The publish stage only checks the manifest statuses and source files.
|
// The publish stage only checks the manifest statuses and source files.
|
||||||
_ = stageName
|
_ = stageName
|
||||||
}
|
}
|
||||||
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.trimmed.json"), `{"segments":[]}`)
|
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.trimmed.json"), `{"segments":[]}`)
|
||||||
|
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.md"), "# final\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.trimmed.md"), "# final trimmed\n")
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
var stdout bytes.Buffer
|
||||||
var stderr bytes.Buffer
|
var stderr bytes.Buffer
|
||||||
@@ -958,6 +1108,72 @@ func addPublishOutputsToPipeline(t *testing.T, pipelinePath, publishYAML string)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func addExtractionOutputToPipeline(t *testing.T, pipelinePath string) {
|
||||||
|
t.Helper()
|
||||||
|
data, err := os.ReadFile(pipelinePath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
data = append(data, []byte(`notarius:
|
||||||
|
enabled: true
|
||||||
|
config_path: notarius.yml
|
||||||
|
pipeline_id: campaign.extract
|
||||||
|
outputs:
|
||||||
|
encounters:
|
||||||
|
lane_id: encounters
|
||||||
|
media_type: application/json
|
||||||
|
schema_id: encounters
|
||||||
|
schema_version: "1"
|
||||||
|
module_key: encounters
|
||||||
|
`)...)
|
||||||
|
if err := os.WriteFile(pipelinePath, data, 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(filepath.Dir(pipelinePath), "notarius.yml"), []byte("{}\n"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeOperatorExtractionManifest(t *testing.T, workspaceRoot string) string {
|
||||||
|
t.Helper()
|
||||||
|
paths := artifacts.NewLocalStore(workspaceRoot).SessionPathsFor("sample-campaign", "2026-05-03")
|
||||||
|
bundleRoot := filepath.Join(paths.ArtifactsDir, "notarius", "extract-run-1")
|
||||||
|
lanePath := filepath.Join(bundleRoot, "lanes", "encounters.json")
|
||||||
|
indexPath := filepath.Join(bundleRoot, "index.json")
|
||||||
|
mustWriteTestFile(t, lanePath, `{"secret":"DO_NOT_PRINT"}`)
|
||||||
|
mustWriteTestFile(t, indexPath, `{"lanes":[]}`)
|
||||||
|
laneChecksum, err := artifacts.SHA256File(lanePath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
indexChecksum, err := artifacts.SHA256File(indexPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
m := manifest.New("2026-05-03", time.Now().UTC())
|
||||||
|
m.Campaign = "sample-campaign"
|
||||||
|
m.Stages["extract"] = &manifest.StageRecord{
|
||||||
|
Name: "extract", Status: manifest.StatusSucceeded,
|
||||||
|
Metadata: map[string]any{
|
||||||
|
"narratio_run_id": "extract-run-1", "bundle_root": bundleRoot,
|
||||||
|
"receipt": map[string]any{"run_id": "notarius-run-1", "pipeline_id": "campaign.extract"},
|
||||||
|
},
|
||||||
|
Outputs: []manifest.ArtifactRecord{
|
||||||
|
{
|
||||||
|
Kind: "notarius_lane", SourceID: artifacts.ExtractionArtifactSourceID("encounters"), LocalPath: lanePath,
|
||||||
|
ProducerRunID: "extract-run-1", Checksum: laneChecksum,
|
||||||
|
Contract: &artifactmodel.ContractMetadata{MediaType: "application/json", SchemaID: "encounters", SchemaVersion: "1", ModuleKey: "encounters"},
|
||||||
|
ExternalProvenance: &artifactmodel.ExternalProvenance{System: "notarius", RunID: "notarius-run-1", PipelineID: "campaign.extract", ArtifactID: "encounters"},
|
||||||
|
},
|
||||||
|
{Kind: "notarius_index", LocalPath: indexPath, ProducerRunID: "extract-run-1", Checksum: indexChecksum},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
if err := (&manifest.LocalStore{}).Save(context.Background(), paths.ManifestPath, m); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return lanePath
|
||||||
|
}
|
||||||
|
|
||||||
func replaceInFileOrFatal(t *testing.T, path, old, new string) {
|
func replaceInFileOrFatal(t *testing.T, path, old, new string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
data, err := os.ReadFile(path)
|
data, err := os.ReadFile(path)
|
||||||
@@ -993,7 +1209,7 @@ func writeValidPublishRunConfigFiles(t *testing.T, workspaceRoot string) (string
|
|||||||
m := manifest.New("2026-05-03", nowUTC())
|
m := manifest.New("2026-05-03", nowUTC())
|
||||||
m.Campaign = "sample-campaign"
|
m.Campaign = "sample-campaign"
|
||||||
m.RunID = "20260521T160000Z-test"
|
m.RunID = "20260521T160000Z-test"
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze"} {
|
||||||
m.MarkStageSucceeded(name, nowUTC(), nil)
|
m.MarkStageSucceeded(name, nowUTC(), nil)
|
||||||
}
|
}
|
||||||
path := artifacts.SessionManifestPathForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
path := artifacts.SessionManifestPathForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
|||||||
276
internal/app/operator_inspection.go
Normal file
276
internal/app/operator_inspection.go
Normal file
@@ -0,0 +1,276 @@
|
|||||||
|
package app
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
type stableInputCheck struct {
|
||||||
|
Name string
|
||||||
|
Path string
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type localAudioCheck struct {
|
||||||
|
Checked bool
|
||||||
|
Paths []string
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type remoteAudioCheck struct {
|
||||||
|
Checked bool
|
||||||
|
Prefix string
|
||||||
|
Keys []string
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type previousArtifactReadiness struct {
|
||||||
|
Requirements []artifacts.PreviousArtifactRequirement
|
||||||
|
MissingID bool
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type remoteCurrentStateCheck struct {
|
||||||
|
State *RemoteCurrentState
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type effectiveLocksCheck struct {
|
||||||
|
Locks *effectiveLocks
|
||||||
|
Err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectStableInputs(cfg *config.Config) []stableInputCheck {
|
||||||
|
items := []struct {
|
||||||
|
name string
|
||||||
|
in config.ResolvedInputFile
|
||||||
|
}{
|
||||||
|
{name: "speakers", in: cfg.StableInputs.SpeakersFile},
|
||||||
|
{name: "autocorrect", in: cfg.StableInputs.AutocorrectFile},
|
||||||
|
{name: "glossary", in: cfg.StableInputs.GlossaryFile},
|
||||||
|
{name: "players", in: cfg.StableInputs.PlayersFile},
|
||||||
|
{name: "party", in: cfg.StableInputs.PartyFile},
|
||||||
|
}
|
||||||
|
out := make([]stableInputCheck, 0, len(items))
|
||||||
|
for _, item := range items {
|
||||||
|
path, err := resolveHelperConfigRelativePath(item.in)
|
||||||
|
if err != nil {
|
||||||
|
out = append(out, stableInputCheck{Name: item.name, Err: err})
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(path); err != nil {
|
||||||
|
out = append(out, stableInputCheck{Name: item.name, Path: path, Err: err})
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, stableInputCheck{Name: item.name, Path: path})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectLocalAudioPresence(cfg *config.Config) localAudioCheck {
|
||||||
|
if cfg.Session.Inputs.AudioS3 != nil {
|
||||||
|
return localAudioCheck{}
|
||||||
|
}
|
||||||
|
|
||||||
|
sessionDir := filepath.Dir(cfg.SessionPath)
|
||||||
|
resolved, err := resolveLocalInspectionAudioPaths(sessionDir, cfg.Session.Inputs)
|
||||||
|
if err != nil {
|
||||||
|
return localAudioCheck{Checked: true, Err: err}
|
||||||
|
}
|
||||||
|
return localAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Paths: resolved,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectRemoteAudioPresence(ctx context.Context, cfg *config.Config, store storage.ObjectStore) remoteAudioCheck {
|
||||||
|
if cfg.Session.Inputs.AudioS3 == nil {
|
||||||
|
return remoteAudioCheck{}
|
||||||
|
}
|
||||||
|
if store == nil {
|
||||||
|
return remoteAudioCheck{Checked: true, Err: fmt.Errorf("storage backend is required for remote audio checks")}
|
||||||
|
}
|
||||||
|
|
||||||
|
sessionPrefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
audioPrefix := artifacts.S3AudioPrefix(sessionPrefix, cfg.Session.Inputs.AudioS3.Prefix)
|
||||||
|
objects, err := store.List(ctx, audioPrefix)
|
||||||
|
if err != nil {
|
||||||
|
return remoteAudioCheck{Checked: true, Prefix: audioPrefix, Err: err}
|
||||||
|
}
|
||||||
|
|
||||||
|
keys := make([]string, 0, len(objects))
|
||||||
|
seenBase := map[string]string{}
|
||||||
|
for _, obj := range objects {
|
||||||
|
key := strings.TrimSpace(obj.Key)
|
||||||
|
if key == "" || strings.HasSuffix(key, "/") || !isInspectionFlacPath(key) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
base := path.Base(key)
|
||||||
|
if prev, exists := seenBase[base]; exists && prev != key {
|
||||||
|
return remoteAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Prefix: audioPrefix,
|
||||||
|
Err: fmt.Errorf("duplicate s3 audio basename %q from %q and %q", base, prev, key),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
seenBase[base] = key
|
||||||
|
keys = append(keys, key)
|
||||||
|
}
|
||||||
|
sort.Strings(keys)
|
||||||
|
if len(keys) == 0 {
|
||||||
|
return remoteAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Prefix: audioPrefix,
|
||||||
|
Err: fmt.Errorf("no .flac files found under s3 audio prefix %q", audioPrefix),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return remoteAudioCheck{
|
||||||
|
Checked: true,
|
||||||
|
Prefix: audioPrefix,
|
||||||
|
Keys: keys,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectPreviousArtifactReadiness(
|
||||||
|
ctx context.Context,
|
||||||
|
cfg *config.Config,
|
||||||
|
store storage.ObjectStore,
|
||||||
|
requirements []artifacts.PreviousArtifactRequirement,
|
||||||
|
) previousArtifactReadiness {
|
||||||
|
out := previousArtifactReadiness{
|
||||||
|
Requirements: append([]artifacts.PreviousArtifactRequirement(nil), requirements...),
|
||||||
|
}
|
||||||
|
if len(requirements) == 0 {
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(cfg.Session.PreviousSessionID) == "" {
|
||||||
|
out.MissingID = true
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
if store == nil {
|
||||||
|
out.Err = fmt.Errorf("previous-session artifacts cannot be checked because storage is unavailable")
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
prefix := artifacts.S3SessionPrefix(cfg.Pipeline.Storage.S3.RootPrefix, cfg.Session.Campaign, cfg.Session.PreviousSessionID)
|
||||||
|
if _, err := artifacts.LoadCurrentState(ctx, store, prefix, artifacts.CurrentStateValidation{
|
||||||
|
ExpectedSessionID: strings.TrimSpace(cfg.Session.PreviousSessionID),
|
||||||
|
ExpectedCampaign: strings.TrimSpace(cfg.Session.Campaign),
|
||||||
|
ValidateRunID: true,
|
||||||
|
}); err != nil {
|
||||||
|
out.Err = fmt.Errorf("remote %v", err)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectRemoteCurrentState(ctx context.Context, cfg *config.Config, store storage.ObjectStore) remoteCurrentStateCheck {
|
||||||
|
if store == nil {
|
||||||
|
return remoteCurrentStateCheck{}
|
||||||
|
}
|
||||||
|
current, err := discoverRemoteCurrentStateFn(ctx, cfg, store)
|
||||||
|
if err != nil {
|
||||||
|
return remoteCurrentStateCheck{Err: err}
|
||||||
|
}
|
||||||
|
return remoteCurrentStateCheck{State: current}
|
||||||
|
}
|
||||||
|
|
||||||
|
func inspectEffectiveLocks(ctx context.Context, cfg *config.Config, store storage.ObjectStore) effectiveLocksCheck {
|
||||||
|
locks, err := loadEffectiveLocks(ctx, cfg, store)
|
||||||
|
if err != nil {
|
||||||
|
return effectiveLocksCheck{Err: err}
|
||||||
|
}
|
||||||
|
return effectiveLocksCheck{Locks: locks}
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveLocalInspectionAudioPaths(sessionDir string, inputs config.SessionInputsConfig) ([]string, error) {
|
||||||
|
if len(inputs.AudioFiles) > 0 {
|
||||||
|
out := make([]string, 0, len(inputs.AudioFiles))
|
||||||
|
seenBase := map[string]string{}
|
||||||
|
for _, item := range inputs.AudioFiles {
|
||||||
|
resolved, err := resolveInspectionPath(sessionDir, item)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if !isInspectionFlacPath(resolved) {
|
||||||
|
return nil, fmt.Errorf("audio file %q must have .flac extension", resolved)
|
||||||
|
}
|
||||||
|
if err := requireInspectionFile(resolved, "audio file"); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
base := filepath.Base(resolved)
|
||||||
|
if prev, exists := seenBase[base]; exists && prev != resolved {
|
||||||
|
return nil, fmt.Errorf("duplicate audio basename %q from %q and %q", base, prev, resolved)
|
||||||
|
}
|
||||||
|
seenBase[base] = resolved
|
||||||
|
out = append(out, resolved)
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
audioDir, err := resolveInspectionPath(sessionDir, inputs.AudioDir)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
entries, err := os.ReadDir(audioDir)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("read audio directory %q: %w", audioDir, err)
|
||||||
|
}
|
||||||
|
out := make([]string, 0, len(entries))
|
||||||
|
for _, entry := range entries {
|
||||||
|
if entry.IsDir() {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
full := filepath.Join(audioDir, entry.Name())
|
||||||
|
if !isInspectionFlacPath(full) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err := requireInspectionFile(full, "audio file"); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, full)
|
||||||
|
}
|
||||||
|
if len(out) == 0 {
|
||||||
|
return nil, fmt.Errorf("no .flac files found in audio directory %q", audioDir)
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveInspectionPath(baseDir, inputPath string) (string, error) {
|
||||||
|
pathValue := strings.TrimSpace(inputPath)
|
||||||
|
if pathValue == "" {
|
||||||
|
return "", fmt.Errorf("path is required")
|
||||||
|
}
|
||||||
|
if filepath.IsAbs(pathValue) {
|
||||||
|
return filepath.Clean(pathValue), nil
|
||||||
|
}
|
||||||
|
return filepath.Clean(filepath.Join(baseDir, pathValue)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func requireInspectionFile(path, label string) error {
|
||||||
|
info, err := os.Stat(path)
|
||||||
|
if err != nil {
|
||||||
|
if os.IsNotExist(err) {
|
||||||
|
return fmt.Errorf("%s %q does not exist", label, path)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("stat %s %q: %w", label, path, err)
|
||||||
|
}
|
||||||
|
if info.IsDir() {
|
||||||
|
return fmt.Errorf("%s %q is a directory", label, path)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func isInspectionFlacPath(path string) bool {
|
||||||
|
return strings.EqualFold(filepath.Ext(strings.TrimSpace(path)), ".flac")
|
||||||
|
}
|
||||||
@@ -48,13 +48,6 @@ func LocksList(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
|
|
||||||
// LocksAdd adds or updates one remote lock.
|
// LocksAdd adds or updates one remote lock.
|
||||||
func LocksAdd(ctx context.Context, args []string, out io.Writer) error {
|
func LocksAdd(ctx context.Context, args []string, out io.Writer) error {
|
||||||
var positionalSessionID string
|
|
||||||
var source string
|
|
||||||
if len(args) >= 2 && !isCLIFlagToken(args[0]) && !isCLIFlagToken(args[1]) {
|
|
||||||
positionalSessionID = strings.TrimSpace(args[0])
|
|
||||||
source = strings.TrimSpace(args[1])
|
|
||||||
args = append([]string(nil), args[2:]...)
|
|
||||||
}
|
|
||||||
fs := flag.NewFlagSet("locks add", flag.ContinueOnError)
|
fs := flag.NewFlagSet("locks add", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
var flags commonConfigFlags
|
var flags commonConfigFlags
|
||||||
@@ -63,26 +56,8 @@ func LocksAdd(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
addCommonConfigFlags(fs, &flags)
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.StringVar(&reason, "reason", "", "lock reason")
|
fs.StringVar(&reason, "reason", "", "lock reason")
|
||||||
fs.BoolVar(&force, "force", false, "update existing remote lock")
|
fs.BoolVar(&force, "force", false, "update existing remote lock")
|
||||||
if err := fs.Parse(args); err != nil {
|
source, err := parseSessionIDAndOnePositionalArg("locks add", "source id", fs, args, &flags.sessionID)
|
||||||
return fmt.Errorf("locks add: invalid flags: %w", err)
|
if err != nil {
|
||||||
}
|
|
||||||
if source == "" {
|
|
||||||
switch fs.NArg() {
|
|
||||||
case 2:
|
|
||||||
positionalSessionID = strings.TrimSpace(fs.Arg(0))
|
|
||||||
source = strings.TrimSpace(fs.Arg(1))
|
|
||||||
case 1:
|
|
||||||
if strings.TrimSpace(flags.sessionID) == "" {
|
|
||||||
return fmt.Errorf("locks add: expected session_id and source id")
|
|
||||||
}
|
|
||||||
source = strings.TrimSpace(fs.Arg(0))
|
|
||||||
default:
|
|
||||||
return fmt.Errorf("locks add: expected session_id and source id")
|
|
||||||
}
|
|
||||||
} else if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("locks add: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("locks add", positionalSessionID, &flags.sessionID); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(flags.sessionID) == "" {
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
@@ -92,7 +67,7 @@ func LocksAdd(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("locks add: %w", err)
|
return fmt.Errorf("locks add: %w", err)
|
||||||
}
|
}
|
||||||
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, "locks add"); err != nil {
|
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, cfg.Pipeline.Notarius, "locks add"); err != nil {
|
||||||
return fmt.Errorf("locks add: %w", err)
|
return fmt.Errorf("locks add: %w", err)
|
||||||
}
|
}
|
||||||
if _, ok := lockSourceSet(locks.Static)[source]; ok {
|
if _, ok := lockSourceSet(locks.Static)[source]; ok {
|
||||||
@@ -104,7 +79,7 @@ func LocksAdd(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
}
|
}
|
||||||
remoteSet[source] = config.PublishLockRule{Source: source, Reason: strings.TrimSpace(reason)}
|
remoteSet[source] = config.PublishLockRule{Source: source, Reason: strings.TrimSpace(reason)}
|
||||||
remoteLocks := lockMapValues(remoteSet)
|
remoteLocks := lockMapValues(remoteSet)
|
||||||
if _, err := config.ValidatePublishLockRules(remoteLocks, cfg.Pipeline.Scriptorium, "locks"); err != nil {
|
if _, err := config.ValidatePublishLockRules(remoteLocks, cfg.Pipeline.Scriptorium, cfg.Pipeline.Notarius, "locks"); err != nil {
|
||||||
return fmt.Errorf("locks add: %w", err)
|
return fmt.Errorf("locks add: %w", err)
|
||||||
}
|
}
|
||||||
if err := uploadRemoteLockStore(ctx, store, locks.Key, &config.PublishLockStore{Locks: remoteLocks}); err != nil {
|
if err := uploadRemoteLockStore(ctx, store, locks.Key, &config.PublishLockStore{Locks: remoteLocks}); err != nil {
|
||||||
@@ -116,37 +91,12 @@ func LocksAdd(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
|
|
||||||
// LocksRemove removes one remote lock.
|
// LocksRemove removes one remote lock.
|
||||||
func LocksRemove(ctx context.Context, args []string, out io.Writer) error {
|
func LocksRemove(ctx context.Context, args []string, out io.Writer) error {
|
||||||
var positionalSessionID string
|
|
||||||
var source string
|
|
||||||
if len(args) >= 2 && !isCLIFlagToken(args[0]) && !isCLIFlagToken(args[1]) {
|
|
||||||
positionalSessionID = strings.TrimSpace(args[0])
|
|
||||||
source = strings.TrimSpace(args[1])
|
|
||||||
args = append([]string(nil), args[2:]...)
|
|
||||||
}
|
|
||||||
fs := flag.NewFlagSet("locks remove", flag.ContinueOnError)
|
fs := flag.NewFlagSet("locks remove", flag.ContinueOnError)
|
||||||
fs.SetOutput(io.Discard)
|
fs.SetOutput(io.Discard)
|
||||||
var flags commonConfigFlags
|
var flags commonConfigFlags
|
||||||
addCommonConfigFlags(fs, &flags)
|
addCommonConfigFlags(fs, &flags)
|
||||||
if err := fs.Parse(args); err != nil {
|
source, err := parseSessionIDAndOnePositionalArg("locks remove", "source id", fs, args, &flags.sessionID)
|
||||||
return fmt.Errorf("locks remove: invalid flags: %w", err)
|
if err != nil {
|
||||||
}
|
|
||||||
if source == "" {
|
|
||||||
switch fs.NArg() {
|
|
||||||
case 2:
|
|
||||||
positionalSessionID = strings.TrimSpace(fs.Arg(0))
|
|
||||||
source = strings.TrimSpace(fs.Arg(1))
|
|
||||||
case 1:
|
|
||||||
if strings.TrimSpace(flags.sessionID) == "" {
|
|
||||||
return fmt.Errorf("locks remove: expected session_id and source id")
|
|
||||||
}
|
|
||||||
source = strings.TrimSpace(fs.Arg(0))
|
|
||||||
default:
|
|
||||||
return fmt.Errorf("locks remove: expected session_id and source id")
|
|
||||||
}
|
|
||||||
} else if fs.NArg() != 0 {
|
|
||||||
return fmt.Errorf("locks remove: unexpected positional arguments")
|
|
||||||
}
|
|
||||||
if err := applyPositionalSessionID("locks remove", positionalSessionID, &flags.sessionID); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(flags.sessionID) == "" {
|
if strings.TrimSpace(flags.sessionID) == "" {
|
||||||
@@ -156,7 +106,7 @@ func LocksRemove(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("locks remove: %w", err)
|
return fmt.Errorf("locks remove: %w", err)
|
||||||
}
|
}
|
||||||
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, "locks remove"); err != nil {
|
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, cfg.Pipeline.Notarius, "locks remove"); err != nil {
|
||||||
return fmt.Errorf("locks remove: %w", err)
|
return fmt.Errorf("locks remove: %w", err)
|
||||||
}
|
}
|
||||||
remoteSet := lockSourceSet(locks.Remote)
|
remoteSet := lockSourceSet(locks.Remote)
|
||||||
|
|||||||
@@ -54,23 +54,26 @@ func SessionValidate(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
requirements := artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg))
|
requirements := artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg))
|
||||||
if len(requirements) == 0 {
|
previous := inspectPreviousArtifactReadiness(ctx, cfg, store, requirements)
|
||||||
|
if len(previous.Requirements) == 0 {
|
||||||
findings = append(findings, okFinding("previous", "no previous-session artifacts required"))
|
findings = append(findings, okFinding("previous", "no previous-session artifacts required"))
|
||||||
} else if strings.TrimSpace(cfg.Session.PreviousSessionID) == "" {
|
} else if previous.MissingID {
|
||||||
findings = append(findings, errorFinding("previous", "previous_session_id is required by configured previous-session artifacts"))
|
findings = append(findings, errorFinding("previous", "previous_session_id is required by configured previous-session artifacts"))
|
||||||
} else if storeErr != nil {
|
} else if previous.Err != nil {
|
||||||
findings = append(findings, errorFinding("previous", "previous-session artifacts cannot be checked because storage is unavailable"))
|
findings = append(findings, errorFinding("previous", previous.Err.Error()))
|
||||||
} else {
|
} else {
|
||||||
findings = append(findings, validatePreviousArtifactFindings(ctx, cfg, store, requirements)...)
|
for _, req := range previous.Requirements {
|
||||||
|
findings = append(findings, okFinding("previous", fmt.Sprintf("%s required=%t", req.Name, req.Required)))
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
locks, lockErr := loadEffectiveLocks(ctx, cfg, store)
|
locks := inspectEffectiveLocks(ctx, cfg, store)
|
||||||
if lockErr != nil {
|
if locks.Err != nil {
|
||||||
findings = append(findings, errorFinding("locks", lockErr.Error()))
|
findings = append(findings, errorFinding("locks", locks.Err.Error()))
|
||||||
} else if len(locks.All) == 0 {
|
} else if len(locks.Locks.All) == 0 {
|
||||||
findings = append(findings, okFinding("locks", "no effective publish locks"))
|
findings = append(findings, okFinding("locks", "no effective publish locks"))
|
||||||
} else {
|
} else {
|
||||||
for _, lock := range locks.All {
|
for _, lock := range locks.Locks.All {
|
||||||
findings = append(findings, warnFinding("locks", fmt.Sprintf("%s locked: %s", lock.Source, strings.TrimSpace(lock.Reason))))
|
findings = append(findings, warnFinding("locks", fmt.Sprintf("%s locked: %s", lock.Source, strings.TrimSpace(lock.Reason))))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,10 +5,13 @@ import (
|
|||||||
"flag"
|
"flag"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Status reports effective local/remote session state.
|
// Status reports effective local/remote session state.
|
||||||
@@ -35,12 +38,16 @@ func Status(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
fmt.Fprintf(out, "Campaign: %s\n", cfg.Session.Campaign)
|
fmt.Fprintf(out, "Campaign: %s\n", cfg.Session.Campaign)
|
||||||
fmt.Fprintf(out, "Workspace: %s\n", paths.Root)
|
fmt.Fprintf(out, "Workspace: %s\n", paths.Root)
|
||||||
fmt.Fprintf(out, "Session config: %s\n", sessionSourceSummary(cfg))
|
fmt.Fprintf(out, "Session config: %s\n", sessionSourceSummary(cfg))
|
||||||
|
writeStatusStableInputs(out, inspectStableInputs(cfg))
|
||||||
|
writeStatusLocalAudio(out, inspectLocalAudioPresence(cfg))
|
||||||
|
|
||||||
|
var localManifest *manifest.Manifest
|
||||||
if m, err := loadLocalManifest(ctx, paths.ManifestPath); err != nil {
|
if m, err := loadLocalManifest(ctx, paths.ManifestPath); err != nil {
|
||||||
fmt.Fprintf(out, "Local manifest: error: %v\n", err)
|
fmt.Fprintf(out, "Local manifest: error: %v\n", err)
|
||||||
} else if m == nil {
|
} else if m == nil {
|
||||||
fmt.Fprintln(out, "Local manifest: missing")
|
fmt.Fprintln(out, "Local manifest: missing")
|
||||||
} else {
|
} else {
|
||||||
|
localManifest = m
|
||||||
fmt.Fprintf(out, "Local manifest: %s\n", paths.ManifestPath)
|
fmt.Fprintf(out, "Local manifest: %s\n", paths.ManifestPath)
|
||||||
writeStageStatuses(out, m)
|
writeStageStatuses(out, m)
|
||||||
}
|
}
|
||||||
@@ -49,21 +56,30 @@ func Status(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
if storeErr != nil {
|
if storeErr != nil {
|
||||||
fmt.Fprintf(out, "Remote publish: unavailable: %v\n", storeErr)
|
fmt.Fprintf(out, "Remote publish: unavailable: %v\n", storeErr)
|
||||||
} else if store != nil {
|
} else if store != nil {
|
||||||
current, err := discoverRemoteCurrentStateFn(ctx, cfg, store)
|
current := inspectRemoteCurrentState(ctx, cfg, store)
|
||||||
if err != nil {
|
if current.Err != nil {
|
||||||
fmt.Fprintf(out, "Remote publish: missing or unavailable: %v\n", err)
|
fmt.Fprintf(out, "Remote publish: missing or unavailable: %v\n", current.Err)
|
||||||
} else {
|
} else {
|
||||||
fmt.Fprintf(out, "Remote publish: current run %s\n", current.RunID)
|
fmt.Fprintf(out, "Remote publish: current run %s\n", current.State.RunID)
|
||||||
fmt.Fprintf(out, "Remote manifest: %s\n", current.CurrentManifestKey)
|
fmt.Fprintf(out, "Remote manifest: %s\n", current.State.CurrentManifestKey)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
writeStatusRemoteAudio(ctx, out, cfg, store, storeErr)
|
||||||
|
writeStatusPreviousArtifacts(out, inspectPreviousArtifactReadiness(
|
||||||
|
ctx,
|
||||||
|
cfg,
|
||||||
|
store,
|
||||||
|
artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg)),
|
||||||
|
))
|
||||||
|
|
||||||
locks, err := loadEffectiveLocks(ctx, cfg, store)
|
lockChecks := inspectEffectiveLocks(ctx, cfg, store)
|
||||||
if catalog, catalogErr := buildHelperArtifactCatalog(cfg); catalogErr != nil {
|
locks := lockChecks.Locks
|
||||||
|
lockErr := lockChecks.Err
|
||||||
|
if catalog, catalogErr := buildHelperArtifactCatalog(cfg, localManifest); catalogErr != nil {
|
||||||
fmt.Fprintf(out, "Remote outputs: error: %v\n", catalogErr)
|
fmt.Fprintf(out, "Remote outputs: error: %v\n", catalogErr)
|
||||||
} else if storeErr == nil {
|
} else if storeErr == nil {
|
||||||
catalogLocks := locks
|
catalogLocks := locks
|
||||||
if err != nil {
|
if lockErr != nil {
|
||||||
catalogLocks = &effectiveLocks{
|
catalogLocks = &effectiveLocks{
|
||||||
Static: staticPublishLocks(cfg),
|
Static: staticPublishLocks(cfg),
|
||||||
All: staticPublishLocks(cfg),
|
All: staticPublishLocks(cfg),
|
||||||
@@ -76,8 +92,8 @@ func Status(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
fmt.Fprintln(out, "Remote outputs:")
|
fmt.Fprintln(out, "Remote outputs:")
|
||||||
writeArtifactList(out, cfg, catalog, catalogLocks, publishedRemoteState)
|
writeArtifactList(out, cfg, catalog, catalogLocks, publishedRemoteState)
|
||||||
}
|
}
|
||||||
if err != nil {
|
if lockErr != nil {
|
||||||
fmt.Fprintf(out, "Publish locks: error: %v\n", err)
|
fmt.Fprintf(out, "Publish locks: error: %v\n", lockErr)
|
||||||
} else {
|
} else {
|
||||||
writeLocks(out, cfg, locks)
|
writeLocks(out, cfg, locks)
|
||||||
}
|
}
|
||||||
@@ -86,3 +102,68 @@ func Status(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
fmt.Fprintf(out, "- narratio session restore %s --dry-run\n", cfg.Session.SessionID)
|
fmt.Fprintf(out, "- narratio session restore %s --dry-run\n", cfg.Session.SessionID)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writeStatusStableInputs(out io.Writer, checks []stableInputCheck) {
|
||||||
|
if len(checks) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for _, check := range checks {
|
||||||
|
if check.Err != nil {
|
||||||
|
if strings.TrimSpace(check.Path) != "" {
|
||||||
|
fmt.Fprintf(out, "Stable input %s: unavailable: %v\n", check.Name, check.Err)
|
||||||
|
} else {
|
||||||
|
fmt.Fprintf(out, "Stable input %s: unavailable: %s\n", check.Name, check.Err.Error())
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "Stable input %s: %s\n", check.Name, check.Path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStatusLocalAudio(out io.Writer, check localAudioCheck) {
|
||||||
|
if !check.Checked {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if check.Err != nil {
|
||||||
|
fmt.Fprintf(out, "Local audio: unavailable: %v\n", check.Err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "Local audio: %d file(s)\n", len(check.Paths))
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStatusRemoteAudio(ctx context.Context, out io.Writer, cfg *config.Config, store storage.ObjectStore, storeErr error) {
|
||||||
|
if cfg.Session.Inputs.AudioS3 == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if storeErr != nil {
|
||||||
|
fmt.Fprintf(out, "Remote audio: unavailable: %v\n", storeErr)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
check := inspectRemoteAudioPresence(ctx, cfg, store)
|
||||||
|
if check.Err != nil {
|
||||||
|
fmt.Fprintf(out, "Remote audio: unavailable: %v\n", check.Err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fmt.Fprintf(out, "Remote audio: %d .flac object(s)\n", len(check.Keys))
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeStatusPreviousArtifacts(out io.Writer, readiness previousArtifactReadiness) {
|
||||||
|
if len(readiness.Requirements) == 0 {
|
||||||
|
fmt.Fprintln(out, "Previous-session artifacts: not required")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if readiness.MissingID {
|
||||||
|
fmt.Fprintln(out, "Previous-session artifacts: unavailable: previous_session_id is required by configured previous-session artifacts")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if readiness.Err != nil {
|
||||||
|
fmt.Fprintf(out, "Previous-session artifacts: unavailable: %v\n", readiness.Err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
names := make([]string, 0, len(readiness.Requirements))
|
||||||
|
for _, req := range readiness.Requirements {
|
||||||
|
names = append(names, fmt.Sprintf("%s(required=%t)", req.Name, req.Required))
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
fmt.Fprintf(out, "Previous-session artifacts: ready: %s\n", strings.Join(names, ", "))
|
||||||
|
}
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ func Plan(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
var flags commonConfigFlags
|
var flags commonConfigFlags
|
||||||
var force bool
|
var force bool
|
||||||
addCommonConfigFlags(fs, &flags)
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
fs.BoolVar(&force, "force", false, "show all stages as scheduled to rerun")
|
||||||
|
|
||||||
if err := parseSessionAwareFlags("plan", fs, args, &flags.sessionID); err != nil {
|
if err := parseSessionAwareFlags("plan", fs, args, &flags.sessionID); err != nil {
|
||||||
return err
|
return err
|
||||||
|
|||||||
@@ -27,12 +27,12 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
|
|||||||
if !strings.Contains(got, "narratio session plan: workdir prepared at") {
|
if !strings.Contains(got, "narratio session plan: workdir prepared at") {
|
||||||
t.Fatalf("first output = %q, want workdir prepared", got)
|
t.Fatalf("first output = %q, want workdir prepared", got)
|
||||||
}
|
}
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "publish", "notify"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze", "publish", "notify"} {
|
||||||
if !strings.Contains(got, name+": run") {
|
if !strings.Contains(got, name+": run") {
|
||||||
t.Fatalf("first output = %q, missing stage %q", got, name)
|
t.Fatalf("first output = %q, missing stage %q", got, name)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !strings.Contains(got, "totals: run=9 skip=0") {
|
if !strings.Contains(got, "totals: run=11 skip=0") {
|
||||||
t.Fatalf("first output = %q, want totals", got)
|
t.Fatalf("first output = %q, want totals", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -84,8 +84,8 @@ func TestPlanShowsRunAndSkipFromManifest(t *testing.T) {
|
|||||||
if !strings.Contains(got, "trim: run") {
|
if !strings.Contains(got, "trim: run") {
|
||||||
t.Fatalf("output = %q, want trim run", got)
|
t.Fatalf("output = %q, want trim run", got)
|
||||||
}
|
}
|
||||||
if !strings.Contains(got, "totals: run=7 skip=2") {
|
if !strings.Contains(got, "totals: run=9 skip=2") {
|
||||||
t.Fatalf("output = %q, want totals run=7 skip=2", got)
|
t.Fatalf("output = %q, want totals run=9 skip=2", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -118,6 +118,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`
|
`
|
||||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||||
t.Fatalf("write pipeline.yml: %v", err)
|
t.Fatalf("write pipeline.yml: %v", err)
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ import "testing"
|
|||||||
|
|
||||||
func TestBuildFullPlanOrder(t *testing.T) {
|
func TestBuildFullPlanOrder(t *testing.T) {
|
||||||
got := BuildFullPlan()
|
got := BuildFullPlan()
|
||||||
want := []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "publish", "notify"}
|
want := []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze", "publish", "notify"}
|
||||||
if len(got) != len(want) {
|
if len(got) != len(want) {
|
||||||
t.Fatalf("len(plan) = %d, want %d", len(got), len(want))
|
t.Fatalf("len(plan) = %d, want %d", len(got), len(want))
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -355,7 +355,7 @@ func publishStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Load() error = %v", err)
|
t.Fatalf("Load() error = %v", err)
|
||||||
}
|
}
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze"} {
|
||||||
seedManifest.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
seedManifest.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
||||||
}
|
}
|
||||||
seedManifest.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
|
seedManifest.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
|||||||
@@ -55,7 +55,7 @@ func loadRemoteLockStore(ctx context.Context, cfg *config.Config, store storage.
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, key, fmt.Errorf("read remote locks %q: %w", key, err)
|
return nil, key, fmt.Errorf("read remote locks %q: %w", key, err)
|
||||||
}
|
}
|
||||||
lockStore, err := config.LoadPublishLockStoreBytes("s3://"+s3BucketName(cfg.Pipeline)+"/"+key, data, cfg.Pipeline.Scriptorium)
|
lockStore, err := config.LoadPublishLockStoreBytes("s3://"+s3BucketName(cfg.Pipeline)+"/"+key, data, cfg.Pipeline.Scriptorium, cfg.Pipeline.Notarius)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, key, err
|
return nil, key, err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -34,7 +34,7 @@ func Restore(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
addCommonConfigFlags(fs, &flags)
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.BoolVar(&dryRun, "dry-run", false, "plan restore actions without writing local files")
|
fs.BoolVar(&dryRun, "dry-run", false, "plan restore actions without writing local files")
|
||||||
fs.BoolVar(&force, "force", false, "overwrite local conflicts with remote state")
|
fs.BoolVar(&force, "force", false, "overwrite local conflicts with remote state")
|
||||||
fs.BoolVar(&includeAudio, "include-audio", false, "include archived session-level audio objects")
|
fs.BoolVar(&includeAudio, "include-audio", false, "include remote session-level audio objects")
|
||||||
fs.Usage = func() {
|
fs.Usage = func() {
|
||||||
_, _ = fmt.Fprintln(out, "Usage: narratio session restore <session_id> [--config <path>] [--campaign <id>] [--campaign-file <path>] [--session <path>] [--previous-session-id <value>] [--dry-run] [--force] [--include-audio]")
|
_, _ = fmt.Fprintln(out, "Usage: narratio session restore <session_id> [--config <path>] [--campaign <id>] [--campaign-file <path>] [--session <path>] [--previous-session-id <value>] [--dry-run] [--force] [--include-audio]")
|
||||||
_, _ = fmt.Fprintln(out)
|
_, _ = fmt.Fprintln(out)
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/audio"
|
"gitea.maximumdirect.net/eric/narratio/internal/audio"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/fileops"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -114,10 +115,7 @@ func executeRestoreDownloadAction(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := os.Chmod(tmpPath, 0o644); err != nil {
|
if err := fileops.InstallDownloadedTempFile(tmpPath, safeLocalPath, 0o644); err != nil {
|
||||||
return fmt.Errorf("set file permissions: %w", err)
|
|
||||||
}
|
|
||||||
if err := os.Rename(tmpPath, safeLocalPath); err != nil {
|
|
||||||
return fmt.Errorf("install file atomically: %w", err)
|
return fmt.Errorf("install file atomically: %w", err)
|
||||||
}
|
}
|
||||||
removeTmp = false
|
removeTmp = false
|
||||||
|
|||||||
@@ -9,8 +9,10 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
@@ -39,7 +41,7 @@ func TestExecuteRestoreNonDryRunRestoresDurableFiles(t *testing.T) {
|
|||||||
if stderr.Len() != 0 {
|
if stderr.Len() != 0 {
|
||||||
t.Fatalf("stderr = %q, want empty", stderr.String())
|
t.Fatalf("stderr = %q, want empty", stderr.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "Restored session archive for sample-campaign/2026-05-03") {
|
if !strings.Contains(stdout.String(), "Restored session state for sample-campaign/2026-05-03") {
|
||||||
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -62,6 +64,55 @@ func TestExecuteRestoreNonDryRunRestoresDurableFiles(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteRestoreRoundTripsPublishedExtractionAndManifestMetadata(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
fake := &storage.FakeBackend{}
|
||||||
|
cfg, sessionPrefix, manifestKey, runIDKey := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
|
||||||
|
seedRestoreObject(fake, sessionPrefix+"artifacts/encounters.json", []byte(`{"encounters":[]}`))
|
||||||
|
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
|
||||||
|
|
||||||
|
remoteManifest := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
||||||
|
remoteManifest.Campaign = cfg.Session.Campaign
|
||||||
|
remoteManifest.RunID = "20260519T010203Z-a1b2c3d4"
|
||||||
|
remoteManifest.Stages["extract"] = &manifest.StageRecord{
|
||||||
|
Name: "extract", Status: manifest.StatusSucceeded,
|
||||||
|
Outputs: []manifest.ArtifactRecord{{
|
||||||
|
Kind: "notarius_lane", SourceID: artifacts.ExtractionArtifactSourceID("encounters"),
|
||||||
|
LocalPath: "/prior/workspace/artifacts/notarius/extract-run-1/lanes/encounters.json",
|
||||||
|
Contract: &artifactmodel.ContractMetadata{
|
||||||
|
MediaType: "application/json", SchemaID: "encounters", SchemaVersion: "1",
|
||||||
|
},
|
||||||
|
ExternalProvenance: &artifactmodel.ExternalProvenance{
|
||||||
|
System: "notarius", RunID: "notarius-run-1", PipelineID: "campaign.extract", ArtifactID: "encounters",
|
||||||
|
},
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
manifestBody, err := json.Marshal(remoteManifest)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
seedRestoreObject(fake, manifestKey, manifestBody)
|
||||||
|
restoreWithStoreAndRealPhases(t, fake)
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
code := Execute([]string{"session", "restore", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
|
||||||
|
if code != 0 {
|
||||||
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
|
}
|
||||||
|
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
|
||||||
|
mustReadEquals(t, filepath.Join(sessionRoot, "artifacts", "encounters.json"), `{"encounters":[]}`)
|
||||||
|
restored, err := (&manifest.LocalStore{}).Load(context.Background(), filepath.Join(sessionRoot, "manifest.json"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("load restored manifest: %v", err)
|
||||||
|
}
|
||||||
|
lane := restored.Stages["extract"].Outputs[0]
|
||||||
|
if lane.Contract == nil || lane.Contract.SchemaID != "encounters" || lane.ExternalProvenance == nil || lane.ExternalProvenance.RunID != "notarius-run-1" {
|
||||||
|
t.Fatalf("restored extraction metadata = %#v", lane)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteRestoreIncludeAudioRestoresAudio(t *testing.T) {
|
func TestExecuteRestoreIncludeAudioRestoresAudio(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package app
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
@@ -13,6 +14,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/pathsafe"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/previouscache"
|
"gitea.maximumdirect.net/eric/narratio/internal/previouscache"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -215,19 +217,17 @@ func joinWithinSessionRoot(sessionRoot, relative string) (string, error) {
|
|||||||
if strings.TrimSpace(sessionRoot) == "" {
|
if strings.TrimSpace(sessionRoot) == "" {
|
||||||
return "", fmt.Errorf("session root is required")
|
return "", fmt.Errorf("session root is required")
|
||||||
}
|
}
|
||||||
cleanRel := path.Clean(strings.TrimSpace(relative))
|
joined, err := pathsafe.JoinSlashRelativeUnderRoot(sessionRoot, filepath.ToSlash(strings.TrimSpace(relative)))
|
||||||
if cleanRel == "." || cleanRel == "" {
|
if err != nil {
|
||||||
return "", fmt.Errorf("relative path is required")
|
if errors.Is(err, pathsafe.ErrRelativePathRequired) {
|
||||||
|
return "", fmt.Errorf("relative path is required")
|
||||||
|
}
|
||||||
|
if errors.Is(err, pathsafe.ErrRelativePathEscape) || errors.Is(err, pathsafe.ErrRelativePathAbsolute) {
|
||||||
|
return "", fmt.Errorf("relative path escapes session root")
|
||||||
|
}
|
||||||
|
return "", fmt.Errorf("join relative path under session root: %w", err)
|
||||||
}
|
}
|
||||||
if cleanRel == ".." || strings.HasPrefix(cleanRel, "../") || strings.HasPrefix(cleanRel, "/") {
|
return joined, nil
|
||||||
return "", fmt.Errorf("relative path escapes session root")
|
|
||||||
}
|
|
||||||
abs := filepath.Clean(filepath.Join(sessionRoot, filepath.FromSlash(cleanRel)))
|
|
||||||
root := filepath.Clean(sessionRoot)
|
|
||||||
if abs != root && !strings.HasPrefix(abs, root+string(filepath.Separator)) {
|
|
||||||
return "", fmt.Errorf("resolved local path escapes session root")
|
|
||||||
}
|
|
||||||
return abs, nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func buildPreviousCacheRestoreActions(
|
func buildPreviousCacheRestoreActions(
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ func TestRestorePlanDefaultScope(t *testing.T) {
|
|||||||
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("# recap\n"))
|
seedRestoreObject(store, current.SessionPrefix+"artifacts/session_recap.md", []byte("# recap\n"))
|
||||||
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
|
seedRestoreObject(store, current.SessionPrefix+"audio/alice.flac", []byte("audio"))
|
||||||
seedRestoreObject(store, current.SessionPrefix+"runs/20260519T010203Z-a1b2/manifest.json", []byte("{}"))
|
seedRestoreObject(store, current.SessionPrefix+"runs/20260519T010203Z-a1b2/manifest.json", []byte("{}"))
|
||||||
seedRestoreObject(store, current.SessionPrefix+"logs/archive.log", []byte("log"))
|
seedRestoreObject(store, current.SessionPrefix+"logs/publish.log", []byte("log"))
|
||||||
|
|
||||||
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
plan, err := buildRestorePlan(context.Background(), cfg, current, store, RestorePlanOptions{})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -201,7 +201,7 @@ func writeRestoreSuccessSummary(out io.Writer, report *RestoreReport) error {
|
|||||||
if report == nil {
|
if report == nil {
|
||||||
return fmt.Errorf("restore report is required")
|
return fmt.Errorf("restore report is required")
|
||||||
}
|
}
|
||||||
if _, err := fmt.Fprintf(out, "Restored session archive for %s/%s\n", report.Campaign, report.SessionID); err != nil {
|
if _, err := fmt.Fprintf(out, "Restored session state for %s/%s\n", report.Campaign, report.SessionID); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if _, err := fmt.Fprintf(out, "Remote run: %s\n", report.RunID); err != nil {
|
if _, err := fmt.Fprintf(out, "Remote run: %s\n", report.RunID); err != nil {
|
||||||
|
|||||||
@@ -369,7 +369,7 @@ func TestExecuteRestoreNonDryRunForceExecutesPlan(t *testing.T) {
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
if !strings.Contains(stdout.String(), "Restored session archive for sample-campaign/2026-05-03") {
|
if !strings.Contains(stdout.String(), "Restored session state for sample-campaign/2026-05-03") {
|
||||||
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
t.Fatalf("stdout = %q, want completion summary", stdout.String())
|
||||||
}
|
}
|
||||||
if stderr.Len() != 0 {
|
if stderr.Len() != 0 {
|
||||||
@@ -397,6 +397,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`
|
`
|
||||||
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
|
||||||
t.Fatalf("write pipeline config: %v", err)
|
t.Fatalf("write pipeline config: %v", err)
|
||||||
@@ -407,6 +409,8 @@ inputs:
|
|||||||
mustWriteTestFile(t, filepath.Join(dir, "speakers.yml"), "alice: alice.flac\n")
|
mustWriteTestFile(t, filepath.Join(dir, "speakers.yml"), "alice: alice.flac\n")
|
||||||
mustWriteTestFile(t, filepath.Join(dir, "autocorrect.yml"), "[]\n")
|
mustWriteTestFile(t, filepath.Join(dir, "autocorrect.yml"), "[]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(dir, "glossary.yml"), "[]\n")
|
mustWriteTestFile(t, filepath.Join(dir, "glossary.yml"), "[]\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(dir, "players.yml"), "[]\n")
|
||||||
|
mustWriteTestFile(t, filepath.Join(dir, "party.yml"), "[]\n")
|
||||||
mustWriteTestFile(t, filepath.Join(dir, "audio", "alice.flac"), "audio-bytes")
|
mustWriteTestFile(t, filepath.Join(dir, "audio", "alice.flac"), "audio-bytes")
|
||||||
|
|
||||||
return pipelinePath, campaignPath, sessionPath
|
return pipelinePath, campaignPath, sessionPath
|
||||||
|
|||||||
@@ -1,102 +0,0 @@
|
|||||||
package app
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"flag"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
|
||||||
)
|
|
||||||
|
|
||||||
// Resume continues execution from the first non-succeeded stage in the manifest.
|
|
||||||
func Resume(ctx context.Context, args []string, out io.Writer) error {
|
|
||||||
fs := flag.NewFlagSet("resume", flag.ContinueOnError)
|
|
||||||
fs.SetOutput(io.Discard)
|
|
||||||
|
|
||||||
var flags commonConfigFlags
|
|
||||||
var force bool
|
|
||||||
var selectedArtifacts artifactSelectionFlag
|
|
||||||
addCommonConfigFlags(fs, &flags)
|
|
||||||
fs.BoolVar(&force, "force", false, "force stage execution")
|
|
||||||
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute and publish (comma-separated or repeatable)")
|
|
||||||
|
|
||||||
if err := parseSessionAwareFlags("resume", fs, args, &flags.sessionID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if flags.sessionID == "" {
|
|
||||||
return fmt.Errorf("resume: session_id is required")
|
|
||||||
}
|
|
||||||
cfg, err := loadCommandConfig(ctx, flags.pipelinePath, flags.campaignPath, flags.campaignFilePath, flags.sessionPath, flags.sessionOptions())
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("resume: %w", err)
|
|
||||||
}
|
|
||||||
if err := config.Validate(cfg); err != nil {
|
|
||||||
return fmt.Errorf("resume: %w", err)
|
|
||||||
}
|
|
||||||
normalizedArtifacts, err := selectedArtifacts.Normalize()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("resume: invalid --artifacts: %w", err)
|
|
||||||
}
|
|
||||||
if err := validateSelectedArtifacts(cfg, normalizedArtifacts); err != nil {
|
|
||||||
return fmt.Errorf("resume: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
full := BuildFullPlan()
|
|
||||||
selected := full
|
|
||||||
if !force {
|
|
||||||
m, err := loadManifestIfPresent(ctx, cfg)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("resume: %w", err)
|
|
||||||
}
|
|
||||||
if m != nil {
|
|
||||||
start := firstNonSucceededIndex(full, m)
|
|
||||||
if start >= len(full) {
|
|
||||||
_, err := fmt.Fprintf(out, "narratio resume: session %s has no remaining stages\n", cfg.Session.SessionID)
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
selected = full[start:]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
summary, err := executeStagesFn(ctx, cfg, selected, RunOptions{
|
|
||||||
Force: force,
|
|
||||||
SelectedArtifacts: normalizedArtifacts,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("resume: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = fmt.Fprintf(
|
|
||||||
out,
|
|
||||||
"narratio resume: session %s; executed=%d skipped=%d; manifest=%s\n",
|
|
||||||
summary.SessionID,
|
|
||||||
len(summary.Executed),
|
|
||||||
len(summary.Skipped),
|
|
||||||
summary.ManifestPath,
|
|
||||||
)
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
func loadManifestIfPresent(ctx context.Context, cfg *config.Config) (*manifest.Manifest, error) {
|
|
||||||
path := artifacts.SessionManifestPathForCampaign(
|
|
||||||
cfg.Pipeline.Workspace.Root,
|
|
||||||
cfg.Session.Campaign,
|
|
||||||
cfg.Session.SessionID,
|
|
||||||
)
|
|
||||||
exists, err := fileExists(path)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("check manifest %q: %w", path, err)
|
|
||||||
}
|
|
||||||
if !exists {
|
|
||||||
return nil, nil
|
|
||||||
}
|
|
||||||
store := &manifest.LocalStore{}
|
|
||||||
m, err := store.Load(ctx, path)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("load manifest %q: %w", path, err)
|
|
||||||
}
|
|
||||||
return m, nil
|
|
||||||
}
|
|
||||||
@@ -18,7 +18,7 @@ func Run(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
var force bool
|
var force bool
|
||||||
var selectedArtifacts artifactSelectionFlag
|
var selectedArtifacts artifactSelectionFlag
|
||||||
addCommonConfigFlags(fs, &flags)
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
fs.BoolVar(&force, "force", false, "rerun stages even when already succeeded")
|
||||||
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute and publish (comma-separated or repeatable)")
|
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute and publish (comma-separated or repeatable)")
|
||||||
|
|
||||||
if err := parseSessionAwareFlags("run", fs, args, &flags.sessionID); err != nil {
|
if err := parseSessionAwareFlags("run", fs, args, &flags.sessionID); err != nil {
|
||||||
|
|||||||
@@ -1,8 +1,13 @@
|
|||||||
package app
|
package app
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
"gitea.maximumdirect.net/eric/narratio/internal/stage"
|
||||||
)
|
)
|
||||||
@@ -19,22 +24,40 @@ type stageDecision struct {
|
|||||||
Action stageAction
|
Action stageAction
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const (
|
||||||
|
staleReasonForcedReplacement = "upstream stage was force-run"
|
||||||
|
staleReasonChangedResult = "upstream stage result changed"
|
||||||
|
staleReasonFailure = "upstream stage failed"
|
||||||
|
staleReasonSelfSkip = "upstream stage self-skipped"
|
||||||
|
staleReasonNotResumable = "upstream stage result was not resumable"
|
||||||
|
)
|
||||||
|
|
||||||
|
type priorStageOutcome struct {
|
||||||
|
exists bool
|
||||||
|
status manifest.StageStatus
|
||||||
|
skipReason string
|
||||||
|
outputs int
|
||||||
|
}
|
||||||
|
|
||||||
func decideStageActions(stages []stage.Stage, m *manifest.Manifest, force bool) []stageDecision {
|
func decideStageActions(stages []stage.Stage, m *manifest.Manifest, force bool) []stageDecision {
|
||||||
out := make([]stageDecision, 0, len(stages))
|
out := make([]stageDecision, 0, len(stages))
|
||||||
for _, s := range stages {
|
for _, s := range stages {
|
||||||
action := stageActionRun
|
|
||||||
// TODO: incorporate stale detection once checksum/input change tracking is implemented.
|
|
||||||
if !force && stageSucceeded(m, s.Name()) {
|
|
||||||
action = stageActionSkip
|
|
||||||
}
|
|
||||||
out = append(out, stageDecision{
|
out = append(out, stageDecision{
|
||||||
Stage: s,
|
Stage: s,
|
||||||
Action: action,
|
Action: decideStageAction(s, m, force),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func decideStageAction(s stage.Stage, m *manifest.Manifest, force bool) stageAction {
|
||||||
|
// TODO: incorporate stale detection once checksum/input change tracking is implemented.
|
||||||
|
if !force && stageSucceeded(m, s.Name()) {
|
||||||
|
return stageActionSkip
|
||||||
|
}
|
||||||
|
return stageActionRun
|
||||||
|
}
|
||||||
|
|
||||||
func stageSucceeded(m *manifest.Manifest, name string) bool {
|
func stageSucceeded(m *manifest.Manifest, name string) bool {
|
||||||
if m == nil || m.Stages == nil {
|
if m == nil || m.Stages == nil {
|
||||||
return false
|
return false
|
||||||
@@ -43,13 +66,48 @@ func stageSucceeded(m *manifest.Manifest, name string) bool {
|
|||||||
return sr != nil && sr.Status == manifest.StatusSucceeded
|
return sr != nil && sr.Status == manifest.StatusSucceeded
|
||||||
}
|
}
|
||||||
|
|
||||||
func firstNonSucceededIndex(stages []stage.Stage, m *manifest.Manifest) int {
|
func capturePriorStageOutcome(m *manifest.Manifest, name string) priorStageOutcome {
|
||||||
for i, s := range stages {
|
if m == nil || m.Stages == nil || m.Stages[name] == nil {
|
||||||
if !stageSucceeded(m, s.Name()) {
|
return priorStageOutcome{}
|
||||||
return i
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
return len(stages)
|
record := m.Stages[name]
|
||||||
|
outcome := priorStageOutcome{
|
||||||
|
exists: true,
|
||||||
|
status: record.Status,
|
||||||
|
outputs: len(record.Outputs),
|
||||||
|
}
|
||||||
|
if record.Error != nil && record.Error.Code == "skipped" {
|
||||||
|
outcome.skipReason = record.Error.Message
|
||||||
|
}
|
||||||
|
return outcome
|
||||||
|
}
|
||||||
|
|
||||||
|
func (o priorStageOutcome) isSameSelfSkip(reason string) bool {
|
||||||
|
return o.exists &&
|
||||||
|
o.status == manifest.StatusSkipped &&
|
||||||
|
o.outputs == 0 &&
|
||||||
|
o.skipReason == strings.TrimSpace(reason)
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadManifestIfPresent(ctx context.Context, cfg *config.Config) (*manifest.Manifest, error) {
|
||||||
|
path := artifacts.SessionManifestPathForCampaign(
|
||||||
|
cfg.Pipeline.Workspace.Root,
|
||||||
|
cfg.Session.Campaign,
|
||||||
|
cfg.Session.SessionID,
|
||||||
|
)
|
||||||
|
exists, err := fileExists(path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("check manifest %q: %w", path, err)
|
||||||
|
}
|
||||||
|
if !exists {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
store := &manifest.LocalStore{}
|
||||||
|
m, err := store.Load(ctx, path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("load manifest %q: %w", path, err)
|
||||||
|
}
|
||||||
|
return m, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func canonicalStageNames() []string {
|
func canonicalStageNames() []string {
|
||||||
@@ -75,7 +133,7 @@ func downstreamStageNames(stageName string) []string {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func invalidateDownstreamSucceededStages(m *manifest.Manifest, upstreamStage string, at time.Time) []string {
|
func invalidateDownstreamSucceededStagesWithReason(m *manifest.Manifest, upstreamStage string, at time.Time, reason string) []string {
|
||||||
if m == nil || m.Stages == nil {
|
if m == nil || m.Stages == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -86,7 +144,7 @@ func invalidateDownstreamSucceededStages(m *manifest.Manifest, upstreamStage str
|
|||||||
if sr == nil || sr.Status != manifest.StatusSucceeded {
|
if sr == nil || sr.Status != manifest.StatusSucceeded {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
m.MarkStageStale(downstream, at, "upstream stage rerun with force")
|
m.MarkStageStale(downstream, at, reason)
|
||||||
invalidated = append(invalidated, downstream)
|
invalidated = append(invalidated, downstream)
|
||||||
}
|
}
|
||||||
return invalidated
|
return invalidated
|
||||||
|
|||||||
@@ -8,18 +8,6 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestFirstNonSucceededIndex(t *testing.T) {
|
|
||||||
stages := BuildFullPlan()
|
|
||||||
m := manifest.New("2026-05-03", time.Now().UTC())
|
|
||||||
m.MarkStageSucceeded("prepare", time.Now().UTC(), nil)
|
|
||||||
m.MarkStageSucceeded("transcribe", time.Now().UTC(), nil)
|
|
||||||
|
|
||||||
got := firstNonSucceededIndex(stages, m)
|
|
||||||
if got != 2 {
|
|
||||||
t.Fatalf("firstNonSucceededIndex() = %d, want 2", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestDecideStageActions(t *testing.T) {
|
func TestDecideStageActions(t *testing.T) {
|
||||||
stages := BuildFullPlan()[:2]
|
stages := BuildFullPlan()[:2]
|
||||||
m := manifest.New("2026-05-03", time.Now().UTC())
|
m := manifest.New("2026-05-03", time.Now().UTC())
|
||||||
@@ -44,7 +32,7 @@ func TestDecideStageActions(t *testing.T) {
|
|||||||
|
|
||||||
func TestDownstreamStageNames(t *testing.T) {
|
func TestDownstreamStageNames(t *testing.T) {
|
||||||
got := downstreamStageNames("polish")
|
got := downstreamStageNames("polish")
|
||||||
want := []string{"normalize", "trim", "analyze", "publish", "notify"}
|
want := []string{"normalize", "trim", "extract", "render", "analyze", "publish", "notify"}
|
||||||
if !reflect.DeepEqual(got, want) {
|
if !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("downstreamStageNames(polish) = %#v, want %#v", got, want)
|
t.Fatalf("downstreamStageNames(polish) = %#v, want %#v", got, want)
|
||||||
}
|
}
|
||||||
@@ -55,7 +43,7 @@ func TestDownstreamStageNames(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestInvalidateDownstreamSucceededStages(t *testing.T) {
|
func TestInvalidateDownstreamSucceededStagesWithReason(t *testing.T) {
|
||||||
now := time.Now().UTC()
|
now := time.Now().UTC()
|
||||||
m := manifest.New("2026-05-03", now)
|
m := manifest.New("2026-05-03", now)
|
||||||
m.MarkStageSucceeded("prepare", now, nil)
|
m.MarkStageSucceeded("prepare", now, nil)
|
||||||
@@ -64,14 +52,16 @@ func TestInvalidateDownstreamSucceededStages(t *testing.T) {
|
|||||||
m.MarkStageSucceeded("polish", now, nil)
|
m.MarkStageSucceeded("polish", now, nil)
|
||||||
m.MarkStageSucceeded("normalize", now, nil)
|
m.MarkStageSucceeded("normalize", now, nil)
|
||||||
m.MarkStageSucceeded("trim", now, nil)
|
m.MarkStageSucceeded("trim", now, nil)
|
||||||
|
m.MarkStageSucceeded("extract", now, nil)
|
||||||
|
m.MarkStageSucceeded("render", now, nil)
|
||||||
m.MarkStageFailed("analyze", now, "analysis failed")
|
m.MarkStageFailed("analyze", now, "analysis failed")
|
||||||
m.MarkStageSucceeded("publish", now, nil)
|
m.MarkStageSucceeded("publish", now, nil)
|
||||||
m.MarkStageSucceeded("notify", now, nil)
|
m.MarkStageSucceeded("notify", now, nil)
|
||||||
|
|
||||||
got := invalidateDownstreamSucceededStages(m, "polish", now.Add(1*time.Second))
|
got := invalidateDownstreamSucceededStagesWithReason(m, "polish", now.Add(1*time.Second), staleReasonChangedResult)
|
||||||
want := []string{"normalize", "trim", "publish", "notify"}
|
want := []string{"normalize", "trim", "extract", "render", "publish", "notify"}
|
||||||
if !reflect.DeepEqual(got, want) {
|
if !reflect.DeepEqual(got, want) {
|
||||||
t.Fatalf("invalidateDownstreamSucceededStages() = %#v, want %#v", got, want)
|
t.Fatalf("invalidateDownstreamSucceededStagesWithReason() = %#v, want %#v", got, want)
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, stageName := range want {
|
for _, stageName := range want {
|
||||||
@@ -86,3 +76,30 @@ func TestInvalidateDownstreamSucceededStages(t *testing.T) {
|
|||||||
t.Fatalf("prepare status = %q, want succeeded", m.Stages["prepare"].Status)
|
t.Fatalf("prepare status = %q, want succeeded", m.Stages["prepare"].Status)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExtractionPositionControlsForceInvalidation(t *testing.T) {
|
||||||
|
now := time.Now().UTC()
|
||||||
|
tests := []struct {
|
||||||
|
upstream string
|
||||||
|
want []string
|
||||||
|
}{
|
||||||
|
{upstream: "trim", want: []string{"extract", "render", "analyze", "publish", "notify"}},
|
||||||
|
{upstream: "extract", want: []string{"render", "analyze", "publish", "notify"}},
|
||||||
|
{upstream: "render", want: []string{"analyze", "publish", "notify"}},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.upstream, func(t *testing.T) {
|
||||||
|
m := manifest.New("2026-05-03", now)
|
||||||
|
for _, name := range canonicalStageNames() {
|
||||||
|
m.MarkStageSucceeded(name, now, nil)
|
||||||
|
}
|
||||||
|
got := invalidateDownstreamSucceededStagesWithReason(m, test.upstream, now.Add(time.Second), staleReasonForcedReplacement)
|
||||||
|
if !reflect.DeepEqual(got, test.want) {
|
||||||
|
t.Fatalf("invalidated = %#v, want %#v", got, test.want)
|
||||||
|
}
|
||||||
|
if test.upstream == "render" && m.Stages["extract"].Status != manifest.StatusSucceeded {
|
||||||
|
t.Fatalf("forcing render changed extract: %#v", m.Stages["extract"])
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
|
|||||||
var force bool
|
var force bool
|
||||||
var selectedArtifacts artifactSelectionFlag
|
var selectedArtifacts artifactSelectionFlag
|
||||||
addCommonConfigFlags(fs, &flags)
|
addCommonConfigFlags(fs, &flags)
|
||||||
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
|
fs.BoolVar(&force, "force", false, "rerun the stage even when already succeeded")
|
||||||
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute or publish (comma-separated or repeatable)")
|
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute or publish (comma-separated or repeatable)")
|
||||||
|
|
||||||
if err := fs.Parse(args); err != nil {
|
if err := fs.Parse(args); err != nil {
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
func TestRunContinuesAfterCompletedStages(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||||
@@ -32,12 +32,12 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
|||||||
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
mustWriteTestFile(t, filepath.Join(workRoot, "inputs", "glossary.yml"), "terms: []\n")
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
err := Resume(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
err := Run(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "executed=7 skipped=0") {
|
if !strings.Contains(out.String(), "executed=9 skipped=3") {
|
||||||
t.Fatalf("output = %q, want executed=7 skipped=0", out.String())
|
t.Fatalf("output = %q, want executed=9 skipped=3", out.String())
|
||||||
}
|
}
|
||||||
|
|
||||||
loaded, err := store.Load(context.Background(), manifestPath)
|
loaded, err := store.Load(context.Background(), manifestPath)
|
||||||
@@ -49,35 +49,43 @@ func TestResumeStartsAfterCompletedStages(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResumeNoRemainingStages(t *testing.T) {
|
func TestRunNoRemainingStagesRecordsSkippedStages(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||||
|
|
||||||
store := &manifest.LocalStore{}
|
store := &manifest.LocalStore{}
|
||||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "publish", "notify"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
|
||||||
m.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
m.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||||
}
|
}
|
||||||
|
m.MarkStageSkipped("extract", time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), "notarius_disabled")
|
||||||
if err := store.Save(context.Background(), manifestPath, m); err != nil {
|
if err := store.Save(context.Background(), manifestPath, m); err != nil {
|
||||||
t.Fatalf("save manifest: %v", err)
|
t.Fatalf("save manifest: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
err := Resume(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
err := Run(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "has no remaining stages") {
|
if !strings.Contains(out.String(), "executed=1 skipped=11") {
|
||||||
t.Fatalf("output = %q, want no remaining stages", out.String())
|
t.Fatalf("output = %q, want disabled extraction to self-skip", out.String())
|
||||||
|
}
|
||||||
|
loaded, err := store.Load(context.Background(), manifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("load migrated manifest: %v", err)
|
||||||
|
}
|
||||||
|
if loaded.Stages["extract"] == nil || loaded.Stages["extract"].Status != manifest.StatusSkipped {
|
||||||
|
t.Fatalf("extract record = %#v, want stable disabled skip", loaded.Stages["extract"])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResumeForceRerunsSucceeded(t *testing.T) {
|
func TestRunForceRerunsSucceeded(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
_, _ = w.Write([]byte(`{"source":"resume-force-test","segments":[{"speaker":"alice"}]}`))
|
_, _ = w.Write([]byte(`{"source":"run-force-test","segments":[{"speaker":"alice"}]}`))
|
||||||
}))
|
}))
|
||||||
defer srv.Close()
|
defer srv.Close()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot, srv.URL)
|
||||||
@@ -85,7 +93,7 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
|
|||||||
|
|
||||||
store := &manifest.LocalStore{}
|
store := &manifest.LocalStore{}
|
||||||
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "publish", "notify"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze", "publish", "notify"} {
|
||||||
m.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
m.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||||
}
|
}
|
||||||
if err := store.Save(context.Background(), manifestPath, m); err != nil {
|
if err := store.Save(context.Background(), manifestPath, m); err != nil {
|
||||||
@@ -93,11 +101,11 @@ func TestResumeForceRerunsSucceeded(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
err := Resume(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--force"}, &out)
|
err := Run(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--force"}, &out)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "executed=9 skipped=0") {
|
if !strings.Contains(out.String(), "executed=11 skipped=1") {
|
||||||
t.Fatalf("output = %q, want forced full rerun", out.String())
|
t.Fatalf("output = %q, want forced full rerun", out.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -132,6 +140,27 @@ func TestRunStageExecutesOnlySelectedStage(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunStageExtractIsAcceptedAndSelfSkipsWhenDisabled(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
|
||||||
|
var out bytes.Buffer
|
||||||
|
err := RunStage(context.Background(), []string{"extract", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("RunStage(extract) error = %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(out.String(), "stage=extract executed=1 skipped=1 force=false") {
|
||||||
|
t.Fatalf("output = %q, want disabled extraction self-skip", out.String())
|
||||||
|
}
|
||||||
|
loaded, err := (&manifest.LocalStore{}).Load(context.Background(), filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() error = %v", err)
|
||||||
|
}
|
||||||
|
if loaded.Stages["extract"] == nil || loaded.Stages["extract"].Status != manifest.StatusSkipped {
|
||||||
|
t.Fatalf("extract record = %#v, want skipped", loaded.Stages["extract"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunStageSkipAndForce(t *testing.T) {
|
func TestRunStageSkipAndForce(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
@@ -166,7 +195,7 @@ func TestRunStageSkipAndForce(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunStageForceMarksDownstreamStaleAndResumeContinuesFromStale(t *testing.T) {
|
func TestRunStageForceMarksDownstreamStaleAndRunContinuesFromStale(t *testing.T) {
|
||||||
workspaceRoot := t.TempDir()
|
workspaceRoot := t.TempDir()
|
||||||
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||||
@@ -176,7 +205,7 @@ func TestRunStageForceMarksDownstreamStaleAndResumeContinuesFromStale(t *testing
|
|||||||
|
|
||||||
store := &manifest.LocalStore{}
|
store := &manifest.LocalStore{}
|
||||||
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "publish", "notify"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze", "publish", "notify"} {
|
||||||
seed.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
seed.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
|
||||||
}
|
}
|
||||||
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||||
@@ -196,19 +225,19 @@ func TestRunStageForceMarksDownstreamStaleAndResumeContinuesFromStale(t *testing
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("load manifest after force: %v", err)
|
t.Fatalf("load manifest after force: %v", err)
|
||||||
}
|
}
|
||||||
for _, name := range []string{"normalize", "trim", "analyze", "publish", "notify"} {
|
for _, name := range []string{"normalize", "trim", "extract", "render", "analyze", "publish", "notify"} {
|
||||||
if afterForce.Stages[name] == nil || afterForce.Stages[name].Status != manifest.StatusStale {
|
if afterForce.Stages[name] == nil || afterForce.Stages[name].Status != manifest.StatusStale {
|
||||||
t.Fatalf("stage %q = %#v, want stale", name, afterForce.Stages[name])
|
t.Fatalf("stage %q = %#v, want stale", name, afterForce.Stages[name])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
out.Reset()
|
out.Reset()
|
||||||
err = Resume(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
err = Run(context.Background(), []string{"2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("Resume() error = %v", err)
|
t.Fatalf("Run() error = %v", err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(out.String(), "executed=5 skipped=0") {
|
if !strings.Contains(out.String(), "executed=7 skipped=5") {
|
||||||
t.Fatalf("output = %q, want resume to execute normalize..notify", out.String())
|
t.Fatalf("output = %q, want run to execute stale downstream stages", out.String())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -236,8 +265,8 @@ func TestRunStageTrimExecutes(t *testing.T) {
|
|||||||
if m.Stages["trim"] == nil || m.Stages["trim"].Status != manifest.StatusSucceeded {
|
if m.Stages["trim"] == nil || m.Stages["trim"].Status != manifest.StatusSucceeded {
|
||||||
t.Fatalf("trim stage = %#v, want succeeded", m.Stages["trim"])
|
t.Fatalf("trim stage = %#v, want succeeded", m.Stages["trim"])
|
||||||
}
|
}
|
||||||
if m.Stages["trim"].Metadata == nil || m.Stages["trim"].Metadata["trim_action"] != "copy_disabled" {
|
if m.Stages["trim"].Metadata == nil || m.Stages["trim"].Metadata["trim_action"] != "copy" {
|
||||||
t.Fatalf("trim stage metadata = %#v, want trim_action=copy_disabled", m.Stages["trim"].Metadata)
|
t.Fatalf("trim stage metadata = %#v, want trim_action=copy", m.Stages["trim"].Metadata)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -266,3 +295,30 @@ func TestRunStageNormalizeExecutes(t *testing.T) {
|
|||||||
t.Fatalf("normalize stage = %#v, want succeeded", m.Stages["normalize"])
|
t.Fatalf("normalize stage = %#v, want succeeded", m.Stages["normalize"])
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunStageRenderExecutes(t *testing.T) {
|
||||||
|
workspaceRoot := t.TempDir()
|
||||||
|
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
|
||||||
|
manifestPath := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json")
|
||||||
|
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
|
||||||
|
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.json"), `{"segments":[{"id":1}]}`)
|
||||||
|
mustWriteTestFile(t, filepath.Join(workRoot, "transcripts", "final.trimmed.json"), `{"segments":[{"id":2}]}`)
|
||||||
|
|
||||||
|
var out bytes.Buffer
|
||||||
|
err := RunStage(context.Background(), []string{"render", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("RunStage(render) error = %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(out.String(), "stage=render executed=1 skipped=0") {
|
||||||
|
t.Fatalf("output = %q, want stage=render executed", out.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
store := &manifest.LocalStore{}
|
||||||
|
m, err := store.Load(context.Background(), manifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("load manifest: %v", err)
|
||||||
|
}
|
||||||
|
if m.Stages["render"] == nil || m.Stages["render"].Status != manifest.StatusSucceeded {
|
||||||
|
t.Fatalf("render stage = %#v, want succeeded", m.Stages["render"])
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -8,11 +8,12 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/audita"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/audita"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/notarius"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/notify"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/notify"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/scriptorium"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/scriptorium"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/seriatim"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/seriatim"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/whisperx"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/whisperx"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/logging"
|
"gitea.maximumdirect.net/eric/narratio/internal/logging"
|
||||||
@@ -80,12 +81,12 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
|||||||
}
|
}
|
||||||
env.Audita = runner
|
env.Audita = runner
|
||||||
}
|
}
|
||||||
|
if env.Notarius == nil && needsNotariusForRun(env.Config, stages) {
|
||||||
|
env.Notarius = notarius.NewSubprocessRunner()
|
||||||
|
}
|
||||||
if env.Scriptorium == nil {
|
if env.Scriptorium == nil {
|
||||||
env.Scriptorium = scriptorium.NewSubprocessRunner()
|
env.Scriptorium = scriptorium.NewSubprocessRunner()
|
||||||
}
|
}
|
||||||
if env.Storage == nil {
|
|
||||||
env.Storage = &storage.NoopBackend{}
|
|
||||||
}
|
|
||||||
if env.ObjectStore == nil && needsObjectStoreForRun(env.Config, stages) {
|
if env.ObjectStore == nil && needsObjectStoreForRun(env.Config, stages) {
|
||||||
objectStore, err := newCommandObjectStore(ctx, env.Config, nil)
|
objectStore, err := newCommandObjectStore(ctx, env.Config, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -170,6 +171,29 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
|||||||
for _, d := range decisions {
|
for _, d := range decisions {
|
||||||
s := d.Stage
|
s := d.Stage
|
||||||
runNames = append(runNames, s.Name())
|
runNames = append(runNames, s.Name())
|
||||||
|
d.Action = decideStageAction(s, m, opts.Force)
|
||||||
|
|
||||||
|
if d.Action == stageActionSkip {
|
||||||
|
if validator, ok := s.(stage.ResumeValidator); ok {
|
||||||
|
validation, err := validator.ValidateResume(ctx, stageEnv, m)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("validate resume for stage %q: %w", s.Name(), err)
|
||||||
|
}
|
||||||
|
validation = validation.Normalized()
|
||||||
|
if !validation.Resumable {
|
||||||
|
staleAt := nowUTC()
|
||||||
|
m.MarkStageStale(s.Name(), staleAt, validation.Reason)
|
||||||
|
invalidateDownstreamSucceededStagesWithReason(
|
||||||
|
m, s.Name(), staleAt, staleReasonNotResumable,
|
||||||
|
)
|
||||||
|
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
|
||||||
|
return nil, fmt.Errorf("save manifest after resume validation for stage %q: %w", s.Name(), err)
|
||||||
|
}
|
||||||
|
env.Logger.Info("stage result is not resumable", "stage", s.Name(), "reason", validation.Reason)
|
||||||
|
d.Action = stageActionRun
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
if d.Action == stageActionSkip {
|
if d.Action == stageActionSkip {
|
||||||
skipped = append(skipped, s.Name())
|
skipped = append(skipped, s.Name())
|
||||||
@@ -183,6 +207,7 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
executed = append(executed, s.Name())
|
executed = append(executed, s.Name())
|
||||||
|
priorOutcome := capturePriorStageOutcome(m, s.Name())
|
||||||
|
|
||||||
now := nowUTC()
|
now := nowUTC()
|
||||||
runManifest.SetStageAction(s.Name(), manifest.RunStageActionRun, now)
|
runManifest.SetStageAction(s.Name(), manifest.RunStageActionRun, now)
|
||||||
@@ -191,6 +216,9 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
|||||||
return nil, fmt.Errorf("save run manifest before stage %q: %w", s.Name(), err)
|
return nil, fmt.Errorf("save run manifest before stage %q: %w", s.Name(), err)
|
||||||
}
|
}
|
||||||
m.MarkStageRunning(s.Name(), now)
|
m.MarkStageRunning(s.Name(), now)
|
||||||
|
if opts.Force {
|
||||||
|
invalidateDownstreamSucceededStagesWithReason(m, s.Name(), now, staleReasonForcedReplacement)
|
||||||
|
}
|
||||||
env.Logger.Info("starting stage", "stage", s.Name())
|
env.Logger.Info("starting stage", "stage", s.Name())
|
||||||
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
|
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
|
||||||
return nil, fmt.Errorf("save manifest before stage %q: %w", s.Name(), err)
|
return nil, fmt.Errorf("save manifest before stage %q: %w", s.Name(), err)
|
||||||
@@ -198,9 +226,13 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
|||||||
env.Logger.Debug("manifest saved", "stage", s.Name(), "transition", "running", "path", manifestPath)
|
env.Logger.Debug("manifest saved", "stage", s.Name(), "transition", "running", "path", manifestPath)
|
||||||
|
|
||||||
result, err := s.Run(ctx, stageEnv, m)
|
result, err := s.Run(ctx, stageEnv, m)
|
||||||
|
if err == nil {
|
||||||
|
err = validateStageResult(result)
|
||||||
|
}
|
||||||
if err != nil {
|
if err != nil {
|
||||||
failedAt := nowUTC()
|
failedAt := nowUTC()
|
||||||
m.MarkStageFailed(s.Name(), failedAt, err.Error())
|
m.MarkStageFailed(s.Name(), failedAt, err.Error())
|
||||||
|
invalidateDownstreamSucceededStagesWithReason(m, s.Name(), failedAt, staleReasonFailure)
|
||||||
if saveErr := env.ManifestStore.Save(ctx, manifestPath, m); saveErr != nil {
|
if saveErr := env.ManifestStore.Save(ctx, manifestPath, m); saveErr != nil {
|
||||||
return nil, fmt.Errorf("stage %q failed (%v) and manifest save failed (%v)", s.Name(), err, saveErr)
|
return nil, fmt.Errorf("stage %q failed (%v) and manifest save failed (%v)", s.Name(), err, saveErr)
|
||||||
}
|
}
|
||||||
@@ -212,13 +244,34 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
|
|||||||
env.Logger.Info("stage failed", "stage", s.Name(), "error", err)
|
env.Logger.Info("stage failed", "stage", s.Name(), "error", err)
|
||||||
return nil, fmt.Errorf("stage %q failed: %w", s.Name(), err)
|
return nil, fmt.Errorf("stage %q failed: %w", s.Name(), err)
|
||||||
}
|
}
|
||||||
|
if result != nil && result.Disposition == stage.StageDispositionSkipped {
|
||||||
|
skipped = append(skipped, s.Name())
|
||||||
|
skippedAt := nowUTC()
|
||||||
|
m.MarkStageSkipped(s.Name(), skippedAt, result.SkipReason)
|
||||||
|
applyStageResultToManifest(m, s.Name(), result)
|
||||||
|
if !priorOutcome.isSameSelfSkip(result.SkipReason) {
|
||||||
|
invalidateDownstreamSucceededStagesWithReason(m, s.Name(), skippedAt, staleReasonSelfSkip)
|
||||||
|
}
|
||||||
|
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
|
||||||
|
return nil, fmt.Errorf("save manifest after self-skip %q: %w", s.Name(), err)
|
||||||
|
}
|
||||||
|
runManifest.MarkStageSkipped(s.Name(), skippedAt, result.SkipReason)
|
||||||
|
applyStageResultToRunManifest(runManifest, s.Name(), result)
|
||||||
|
syncRunManifestIdentityFromSession(m, runManifest)
|
||||||
|
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
|
||||||
|
return nil, fmt.Errorf("save run manifest after self-skip %q: %w", s.Name(), err)
|
||||||
|
}
|
||||||
|
env.Logger.Debug("manifest saved", "stage", s.Name(), "transition", "skipped", "path", manifestPath)
|
||||||
|
env.Logger.Info("stage skipped", "stage", s.Name(), "reason", result.SkipReason)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
outputs := mapResultOutputs(s.Name(), result, runID)
|
outputs := mapResultOutputs(s.Name(), result, runID)
|
||||||
succeededAt := nowUTC()
|
succeededAt := nowUTC()
|
||||||
m.MarkStageSucceeded(s.Name(), succeededAt, outputs)
|
m.MarkStageSucceeded(s.Name(), succeededAt, outputs)
|
||||||
applyStageResultToManifest(m, s.Name(), result)
|
applyStageResultToManifest(m, s.Name(), result)
|
||||||
if opts.Force {
|
if !priorOutcome.exists || priorOutcome.status != manifest.StatusSucceeded {
|
||||||
invalidateDownstreamSucceededStages(m, s.Name(), succeededAt)
|
invalidateDownstreamSucceededStagesWithReason(m, s.Name(), succeededAt, staleReasonChangedResult)
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
|
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
|
||||||
@@ -410,24 +463,85 @@ func mapResultOutputs(stageName string, result *stage.StageResult, runID string)
|
|||||||
localPath = ref.RelativePath
|
localPath = ref.RelativePath
|
||||||
}
|
}
|
||||||
kind := ref.Kind
|
kind := ref.Kind
|
||||||
sourceID := ""
|
sourceID := strings.TrimSpace(ref.SourceID)
|
||||||
if stageName == "analyze" {
|
if sourceID == "" {
|
||||||
sourceID = artifacts.ConfiguredArtifactSourceID(ref.Kind)
|
if stageName == "analyze" {
|
||||||
kind = "scriptorium_artifact"
|
sourceID = artifacts.ConfiguredArtifactSourceID(ref.Kind)
|
||||||
|
kind = "scriptorium_artifact"
|
||||||
|
} else {
|
||||||
|
sourceID = sourceIDForOutputKind(kind)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
out = append(out, manifest.ArtifactRecord{
|
out = append(out, manifest.ArtifactRecord{
|
||||||
Kind: kind,
|
Kind: kind,
|
||||||
SourceID: sourceID,
|
SourceID: sourceID,
|
||||||
LocalPath: localPath,
|
LocalPath: localPath,
|
||||||
ProducerRunID: runID,
|
Contract: cloneContractMetadata(ref.Contract),
|
||||||
RemoteKey: ref.RemoteKey,
|
ExternalProvenance: cloneExternalProvenance(ref.ExternalProvenance),
|
||||||
Checksum: ref.Checksum,
|
ProducerRunID: runID,
|
||||||
|
RemoteKey: ref.RemoteKey,
|
||||||
|
Checksum: ref.Checksum,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validateStageResult(result *stage.StageResult) error {
|
||||||
|
if result == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
switch result.Disposition {
|
||||||
|
case stage.StageDispositionSucceeded:
|
||||||
|
if strings.TrimSpace(result.SkipReason) != "" {
|
||||||
|
return fmt.Errorf("successful result contains a skip reason")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
case stage.StageDispositionSkipped:
|
||||||
|
if strings.TrimSpace(result.SkipReason) == "" {
|
||||||
|
return fmt.Errorf("skipped result requires a skip reason")
|
||||||
|
}
|
||||||
|
if len(result.Outputs) != 0 {
|
||||||
|
return fmt.Errorf("skipped result contains %d output(s)", len(result.Outputs))
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
default:
|
||||||
|
return fmt.Errorf("unsupported stage result disposition %q", result.Disposition)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneContractMetadata(value *artifactmodel.ContractMetadata) *artifactmodel.ContractMetadata {
|
||||||
|
if value == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
cloned := *value
|
||||||
|
return &cloned
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneExternalProvenance(value *artifactmodel.ExternalProvenance) *artifactmodel.ExternalProvenance {
|
||||||
|
if value == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
cloned := *value
|
||||||
|
return &cloned
|
||||||
|
}
|
||||||
|
|
||||||
|
func sourceIDForOutputKind(kind string) string {
|
||||||
|
trimmed := strings.TrimSpace(kind)
|
||||||
|
if trimmed == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
if trimmed == "session_bounds" {
|
||||||
|
return artifacts.ArtifactBoundsSession
|
||||||
|
}
|
||||||
|
for _, spec := range artifactmodel.RuntimeTranscriptArtifacts() {
|
||||||
|
if spec.OutputKind == trimmed {
|
||||||
|
return spec.SourceID
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
func applyStageResultToManifest(m *manifest.Manifest, stageName string, result *stage.StageResult) {
|
func applyStageResultToManifest(m *manifest.Manifest, stageName string, result *stage.StageResult) {
|
||||||
if m == nil || result == nil {
|
if m == nil || result == nil {
|
||||||
return
|
return
|
||||||
@@ -583,6 +697,18 @@ func needsObjectStoreForRun(cfg *config.Config, stages []stage.Stage) bool {
|
|||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func needsNotariusForRun(cfg *config.Config, stages []stage.Stage) bool {
|
||||||
|
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Notarius == nil || !cfg.Pipeline.Notarius.Enabled {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for _, candidate := range stages {
|
||||||
|
if candidate != nil && candidate.Name() == "extract" {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
func needsRemoteLocksForRun(cfg *config.Config, stages []stage.Stage) bool {
|
func needsRemoteLocksForRun(cfg *config.Config, stages []stage.Stage) bool {
|
||||||
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
|
||||||
return false
|
return false
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package app
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
@@ -16,6 +17,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/seriatim"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/seriatim"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/adapters/whisperx"
|
"gitea.maximumdirect.net/eric/narratio/internal/adapters/whisperx"
|
||||||
|
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
"gitea.maximumdirect.net/eric/narratio/internal/config"
|
||||||
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
|
||||||
@@ -38,6 +40,25 @@ type countingStage struct {
|
|||||||
runs *int
|
runs *int
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type resultStage struct {
|
||||||
|
name string
|
||||||
|
result *stage.StageResult
|
||||||
|
runs *int
|
||||||
|
order *[]string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s resultStage) Name() string { return s.name }
|
||||||
|
func (s resultStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||||
|
func (s resultStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
||||||
|
if s.runs != nil {
|
||||||
|
*s.runs = *s.runs + 1
|
||||||
|
}
|
||||||
|
if s.order != nil {
|
||||||
|
*s.order = append(*s.order, s.name)
|
||||||
|
}
|
||||||
|
return s.result, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (s countingStage) Name() string { return s.name }
|
func (s countingStage) Name() string { return s.name }
|
||||||
func (s countingStage) Declares() stage.IODecl { return stage.IODecl{} }
|
func (s countingStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||||
func (s countingStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
func (s countingStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
||||||
@@ -50,6 +71,34 @@ type captureSelectedArtifactsStage struct {
|
|||||||
captured *[]string
|
captured *[]string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type captureNotariusStage struct {
|
||||||
|
captured *bool
|
||||||
|
}
|
||||||
|
|
||||||
|
type resumeCheckingStage struct {
|
||||||
|
name string
|
||||||
|
validation stage.ResumeValidation
|
||||||
|
validateErr error
|
||||||
|
runs *int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s resumeCheckingStage) Name() string { return s.name }
|
||||||
|
func (s resumeCheckingStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||||
|
func (s resumeCheckingStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
||||||
|
*s.runs++
|
||||||
|
return &stage.StageResult{}, nil
|
||||||
|
}
|
||||||
|
func (s resumeCheckingStage) ValidateResume(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (stage.ResumeValidation, error) {
|
||||||
|
return s.validation, s.validateErr
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s captureNotariusStage) Name() string { return "extract" }
|
||||||
|
func (s captureNotariusStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||||
|
func (s captureNotariusStage) Run(_ context.Context, env *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
||||||
|
*s.captured = env.Notarius != nil
|
||||||
|
return &stage.StageResult{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (s captureSelectedArtifactsStage) Name() string { return s.name }
|
func (s captureSelectedArtifactsStage) Name() string { return s.name }
|
||||||
func (s captureSelectedArtifactsStage) Declares() stage.IODecl { return stage.IODecl{} }
|
func (s captureSelectedArtifactsStage) Declares() stage.IODecl { return stage.IODecl{} }
|
||||||
func (s captureSelectedArtifactsStage) Run(_ context.Context, env *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
func (s captureSelectedArtifactsStage) Run(_ context.Context, env *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
|
||||||
@@ -134,6 +183,28 @@ func TestExecuteStagesPropagatesSelectedArtifactsToEnv(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteStagesComposesNotariusOnlyForEnabledExtraction(t *testing.T) {
|
||||||
|
cfg := testConfig(t)
|
||||||
|
cfg.Pipeline.Notarius = &config.NotariusConfig{Enabled: true}
|
||||||
|
captured := false
|
||||||
|
|
||||||
|
_, err := executeStages(context.Background(), cfg, []stage.Stage{captureNotariusStage{captured: &captured}}, RunOptions{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if !captured {
|
||||||
|
t.Fatal("extract stage did not receive the default Notarius runner")
|
||||||
|
}
|
||||||
|
|
||||||
|
if needsNotariusForRun(cfg, []stage.Stage{countingStage{name: "analyze", runs: new(int)}}) {
|
||||||
|
t.Fatal("Notarius runner requested without extract in the selected plan")
|
||||||
|
}
|
||||||
|
cfg.Pipeline.Notarius.Enabled = false
|
||||||
|
if needsNotariusForRun(cfg, []stage.Stage{captureNotariusStage{captured: new(bool)}}) {
|
||||||
|
t.Fatal("Notarius runner requested while extraction is disabled")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteStagesAnalyzeOutputsPersistAsScriptoriumArtifacts(t *testing.T) {
|
func TestExecuteStagesAnalyzeOutputsPersistAsScriptoriumArtifacts(t *testing.T) {
|
||||||
cfg := testConfig(t)
|
cfg := testConfig(t)
|
||||||
storeForPaths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
|
storeForPaths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
|
||||||
@@ -199,6 +270,61 @@ func TestExecuteStagesAnalyzeOutputsPersistAsScriptoriumArtifacts(t *testing.T)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestMapResultOutputsPrefersExplicitSourceAndCopiesMetadata(t *testing.T) {
|
||||||
|
contract := &artifactmodel.ContractMetadata{
|
||||||
|
MediaType: "application/json",
|
||||||
|
SchemaID: "notarius.dnd.npc_registry",
|
||||||
|
SchemaVersion: "v1",
|
||||||
|
}
|
||||||
|
provenance := &artifactmodel.ExternalProvenance{
|
||||||
|
System: "notarius",
|
||||||
|
RunID: "external-run",
|
||||||
|
PipelineID: "dnd-session",
|
||||||
|
ArtifactID: "npc-registry",
|
||||||
|
}
|
||||||
|
result := &stage.StageResult{Outputs: []artifacts.Ref{{
|
||||||
|
Kind: "structured_data",
|
||||||
|
SourceID: "narratio.example.npcs",
|
||||||
|
RelativePath: "artifacts/npcs.json",
|
||||||
|
Contract: contract,
|
||||||
|
ExternalProvenance: provenance,
|
||||||
|
}}}
|
||||||
|
|
||||||
|
got := mapResultOutputs("analyze", result, "narratio-run")
|
||||||
|
if len(got) != 1 {
|
||||||
|
t.Fatalf("outputs len = %d, want 1", len(got))
|
||||||
|
}
|
||||||
|
if got[0].SourceID != "narratio.example.npcs" {
|
||||||
|
t.Fatalf("source_id = %q, want explicit source", got[0].SourceID)
|
||||||
|
}
|
||||||
|
if got[0].Kind != "structured_data" {
|
||||||
|
t.Fatalf("kind = %q, want explicit output kind preserved", got[0].Kind)
|
||||||
|
}
|
||||||
|
if got[0].Contract == nil || *got[0].Contract != *contract {
|
||||||
|
t.Fatalf("contract = %#v, want %#v", got[0].Contract, contract)
|
||||||
|
}
|
||||||
|
if got[0].ExternalProvenance == nil || *got[0].ExternalProvenance != *provenance {
|
||||||
|
t.Fatalf("external provenance = %#v, want %#v", got[0].ExternalProvenance, provenance)
|
||||||
|
}
|
||||||
|
if got[0].Contract == contract || got[0].ExternalProvenance == provenance {
|
||||||
|
t.Fatal("mapped metadata should not alias the stage result")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMapResultOutputsRetainsFallbackInference(t *testing.T) {
|
||||||
|
transcript := mapResultOutputs("trim", &stage.StageResult{Outputs: []artifacts.Ref{{
|
||||||
|
Kind: artifactmodel.TranscriptOutputKindFinalTrimmed,
|
||||||
|
}}}, "run-id")
|
||||||
|
if len(transcript) != 1 || transcript[0].SourceID != artifacts.ArtifactTranscriptFinalTrimmed {
|
||||||
|
t.Fatalf("transcript fallback = %#v, want final-trimmed source", transcript)
|
||||||
|
}
|
||||||
|
|
||||||
|
analyze := mapResultOutputs("analyze", &stage.StageResult{Outputs: []artifacts.Ref{{Kind: "session_recap"}}}, "run-id")
|
||||||
|
if len(analyze) != 1 || analyze[0].SourceID != "narratio.artifact.session_recap" || analyze[0].Kind != "scriptorium_artifact" {
|
||||||
|
t.Fatalf("analyze fallback = %#v, want configured artifact inference", analyze)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestNeedsObjectStoreForRunPrepareWithPreviousRequirements(t *testing.T) {
|
func TestNeedsObjectStoreForRunPrepareWithPreviousRequirements(t *testing.T) {
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -273,7 +399,7 @@ func TestExecuteStagesPublishSkipsRequiredUnselectedConfiguredOutput(t *testing.
|
|||||||
manifestPath := manifestPathFor(cfg)
|
manifestPath := manifestPathFor(cfg)
|
||||||
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
||||||
seed.Campaign = cfg.Session.Campaign
|
seed.Campaign = cfg.Session.Campaign
|
||||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim"} {
|
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render"} {
|
||||||
seed.MarkStageSucceeded(stageName, time.Now().UTC(), nil)
|
seed.MarkStageSucceeded(stageName, time.Now().UTC(), nil)
|
||||||
}
|
}
|
||||||
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||||
@@ -332,8 +458,8 @@ func TestExecuteStagesPlaceholderSuccessUpdatesManifest(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("executeStages() error = %v", err)
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
}
|
}
|
||||||
if len(summary.StageNames) != 9 || len(summary.Executed) != 9 || len(summary.Skipped) != 0 {
|
if len(summary.StageNames) != 11 || len(summary.Executed) != 11 || len(summary.Skipped) != 1 || summary.Skipped[0] != "extract" {
|
||||||
t.Fatalf("summary = %#v, want all 9 executed", summary)
|
t.Fatalf("summary = %#v, want full plan with disabled extraction self-skip", summary)
|
||||||
}
|
}
|
||||||
|
|
||||||
store := &manifest.LocalStore{}
|
store := &manifest.LocalStore{}
|
||||||
@@ -342,11 +468,17 @@ func TestExecuteStagesPlaceholderSuccessUpdatesManifest(t *testing.T) {
|
|||||||
t.Fatalf("Load manifest error = %v", err)
|
t.Fatalf("Load manifest error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze", "publish", "notify"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze", "publish", "notify"} {
|
||||||
sr := m.Stages[name]
|
sr := m.Stages[name]
|
||||||
if sr == nil {
|
if sr == nil {
|
||||||
t.Fatalf("missing stage record %q", name)
|
t.Fatalf("missing stage record %q", name)
|
||||||
}
|
}
|
||||||
|
if name == "extract" {
|
||||||
|
if sr.Status != manifest.StatusSkipped || sr.Error == nil || sr.Error.Message != "notarius_disabled" {
|
||||||
|
t.Fatalf("extract stage = %#v, want disabled skip", sr)
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
if sr.Status != manifest.StatusSucceeded {
|
if sr.Status != manifest.StatusSucceeded {
|
||||||
t.Fatalf("stage %q status = %q, want %q", name, sr.Status, manifest.StatusSucceeded)
|
t.Fatalf("stage %q status = %q, want %q", name, sr.Status, manifest.StatusSucceeded)
|
||||||
}
|
}
|
||||||
@@ -425,6 +557,15 @@ func TestExecuteStagesPlaceholderSuccessUpdatesManifest(t *testing.T) {
|
|||||||
}
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
if name == "render" {
|
||||||
|
if sr.Metadata == nil || sr.Metadata["stage"] != "render" {
|
||||||
|
t.Fatalf("render metadata missing stage=render: %#v", sr.Metadata)
|
||||||
|
}
|
||||||
|
if len(sr.Outputs) == 0 {
|
||||||
|
t.Fatalf("render outputs missing")
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
if name == "publish" {
|
if name == "publish" {
|
||||||
if sr.Metadata == nil || sr.Metadata["stage"] != "publish" {
|
if sr.Metadata == nil || sr.Metadata["stage"] != "publish" {
|
||||||
t.Fatalf("publish metadata missing stage=publish: %#v", sr.Metadata)
|
t.Fatalf("publish metadata missing stage=publish: %#v", sr.Metadata)
|
||||||
@@ -488,6 +629,91 @@ func TestExecuteStagesSkipSucceededWhenNotForced(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteStagesUsesOptionalResumeValidation(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
validation stage.ResumeValidation
|
||||||
|
wantRuns int
|
||||||
|
wantSkipped int
|
||||||
|
}{
|
||||||
|
{name: "resumable", validation: stage.Resumable(), wantSkipped: 1},
|
||||||
|
{name: "rerun", validation: stage.NonResumable("durable output changed"), wantRuns: 1},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
cfg := testConfig(t)
|
||||||
|
store := &manifest.LocalStore{}
|
||||||
|
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
||||||
|
seed.MarkStageSucceeded("checked", time.Now().UTC(), nil)
|
||||||
|
if err := store.Save(context.Background(), manifestPathFor(cfg), seed); err != nil {
|
||||||
|
t.Fatalf("Save() error = %v", err)
|
||||||
|
}
|
||||||
|
runs := 0
|
||||||
|
candidate := resumeCheckingStage{name: "checked", validation: test.validation, runs: &runs}
|
||||||
|
summary, err := executeStages(context.Background(), cfg, []stage.Stage{candidate}, RunOptions{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if runs != test.wantRuns || len(summary.Skipped) != test.wantSkipped {
|
||||||
|
t.Fatalf("runs = %d summary = %#v", runs, summary)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteStagesNonResumableResultRerunsSucceededDownstream(t *testing.T) {
|
||||||
|
cfg := testConfig(t)
|
||||||
|
store := &manifest.LocalStore{}
|
||||||
|
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
||||||
|
seed.MarkStageSucceeded("extract", time.Now().UTC(), nil)
|
||||||
|
seed.MarkStageSucceeded("render", time.Now().UTC(), nil)
|
||||||
|
if err := store.Save(context.Background(), manifestPathFor(cfg), seed); err != nil {
|
||||||
|
t.Fatalf("Save() error = %v", err)
|
||||||
|
}
|
||||||
|
extractRuns, renderRuns := 0, 0
|
||||||
|
stages := []stage.Stage{
|
||||||
|
resumeCheckingStage{name: "extract", validation: stage.NonResumable("checksum changed"), runs: &extractRuns},
|
||||||
|
countingStage{name: "render", runs: &renderRuns},
|
||||||
|
}
|
||||||
|
summary, err := executeStages(context.Background(), cfg, stages, RunOptions{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if extractRuns != 1 || renderRuns != 1 || len(summary.Executed) != 2 || len(summary.Skipped) != 0 {
|
||||||
|
t.Fatalf("extract runs=%d render runs=%d summary=%#v", extractRuns, renderRuns, summary)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteStagesResumeValidationErrorPreservesSucceededRecord(t *testing.T) {
|
||||||
|
cfg := testConfig(t)
|
||||||
|
store := &manifest.LocalStore{}
|
||||||
|
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
|
||||||
|
seed.MarkStageSucceeded("checked", time.Now().UTC(), []manifest.ArtifactRecord{{Kind: "kept", LocalPath: "kept.json"}})
|
||||||
|
seed.Stages["checked"].Metadata = map[string]any{"kept": true}
|
||||||
|
if err := store.Save(context.Background(), manifestPathFor(cfg), seed); err != nil {
|
||||||
|
t.Fatalf("Save() error = %v", err)
|
||||||
|
}
|
||||||
|
before, err := json.Marshal(seed.Stages["checked"])
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Marshal(before) error = %v", err)
|
||||||
|
}
|
||||||
|
runs := 0
|
||||||
|
candidate := resumeCheckingStage{name: "checked", validateErr: errors.New("inspection unavailable"), runs: &runs}
|
||||||
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{candidate}, RunOptions{}); err == nil || !strings.Contains(err.Error(), "inspection unavailable") {
|
||||||
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
loaded, err := store.Load(context.Background(), manifestPathFor(cfg))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() error = %v", err)
|
||||||
|
}
|
||||||
|
after, err := json.Marshal(loaded.Stages["checked"])
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Marshal(after) error = %v", err)
|
||||||
|
}
|
||||||
|
if string(after) != string(before) || runs != 0 {
|
||||||
|
t.Fatalf("succeeded record changed: before=%s after=%s runs=%d", before, after, runs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteStagesForceRerunsSucceeded(t *testing.T) {
|
func TestExecuteStagesForceRerunsSucceeded(t *testing.T) {
|
||||||
cfg := testConfig(t)
|
cfg := testConfig(t)
|
||||||
manifestPath := manifestPathFor(cfg)
|
manifestPath := manifestPathFor(cfg)
|
||||||
@@ -516,13 +742,48 @@ func TestExecuteStagesForceRerunsSucceeded(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteStagesSuccessfulReplacementDoesNotInheritResultDetails(t *testing.T) {
|
||||||
|
cfg := testConfig(t)
|
||||||
|
manifestPath := manifestPathFor(cfg)
|
||||||
|
store := &manifest.LocalStore{}
|
||||||
|
|
||||||
|
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||||
|
existing.MarkStageSucceeded("transcribe", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), []manifest.ArtifactRecord{{
|
||||||
|
Kind: "transcript_raw", LocalPath: "transcripts/old.json",
|
||||||
|
}})
|
||||||
|
existingStage := existing.Stages["transcribe"]
|
||||||
|
existingStage.Logs = []string{"logs/old.log"}
|
||||||
|
existingStage.GeneratedConfigs = []string{"generated/old.yaml"}
|
||||||
|
existingStage.Metadata = map[string]any{"old_result": true}
|
||||||
|
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
|
||||||
|
t.Fatalf("Save manifest error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
runs := 0
|
||||||
|
replacement := resultStage{name: "transcribe", result: &stage.StageResult{}, runs: &runs}
|
||||||
|
if _, err := executeStages(context.Background(), cfg, []stage.Stage{replacement}, RunOptions{Force: true}); err != nil {
|
||||||
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
loaded, err := store.Load(context.Background(), manifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load manifest error = %v", err)
|
||||||
|
}
|
||||||
|
record := loaded.Stages["transcribe"]
|
||||||
|
if runs != 1 || record == nil || record.Status != manifest.StatusSucceeded {
|
||||||
|
t.Fatalf("runs = %d, record = %#v, want one successful replacement", runs, record)
|
||||||
|
}
|
||||||
|
if len(record.Outputs) != 0 || len(record.Logs) != 0 || len(record.GeneratedConfigs) != 0 || len(record.Metadata) != 0 {
|
||||||
|
t.Fatalf("replacement inherited result details: %#v", record)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteStagesForceSuccessInvalidatesDownstreamSucceededStages(t *testing.T) {
|
func TestExecuteStagesForceSuccessInvalidatesDownstreamSucceededStages(t *testing.T) {
|
||||||
cfg := testConfig(t)
|
cfg := testConfig(t)
|
||||||
manifestPath := manifestPathFor(cfg)
|
manifestPath := manifestPathFor(cfg)
|
||||||
store := &manifest.LocalStore{}
|
store := &manifest.LocalStore{}
|
||||||
|
|
||||||
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||||
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "publish", "notify"} {
|
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "publish", "notify"} {
|
||||||
existing.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
existing.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
|
||||||
}
|
}
|
||||||
existing.MarkStageFailed("analyze", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), "previous analyze failure")
|
existing.MarkStageFailed("analyze", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), "previous analyze failure")
|
||||||
@@ -553,7 +814,7 @@ func TestExecuteStagesForceSuccessInvalidatesDownstreamSucceededStages(t *testin
|
|||||||
if loaded.Stages["polish"] == nil || loaded.Stages["polish"].Status != manifest.StatusSucceeded {
|
if loaded.Stages["polish"] == nil || loaded.Stages["polish"].Status != manifest.StatusSucceeded {
|
||||||
t.Fatalf("polish status = %#v, want succeeded", loaded.Stages["polish"])
|
t.Fatalf("polish status = %#v, want succeeded", loaded.Stages["polish"])
|
||||||
}
|
}
|
||||||
for _, stageName := range []string{"normalize", "trim", "publish", "notify"} {
|
for _, stageName := range []string{"normalize", "trim", "extract", "render", "publish", "notify"} {
|
||||||
if loaded.Stages[stageName] == nil || loaded.Stages[stageName].Status != manifest.StatusStale {
|
if loaded.Stages[stageName] == nil || loaded.Stages[stageName].Status != manifest.StatusStale {
|
||||||
t.Fatalf("%s status = %#v, want stale", stageName, loaded.Stages[stageName])
|
t.Fatalf("%s status = %#v, want stale", stageName, loaded.Stages[stageName])
|
||||||
}
|
}
|
||||||
@@ -729,6 +990,126 @@ func TestExecuteStagesRunManifestRecordsSkippedStage(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteStagesPersistsSelfSkipAndContinues(t *testing.T) {
|
||||||
|
cfg := testConfig(t)
|
||||||
|
store := &manifest.LocalStore{}
|
||||||
|
manifestPath := manifestPathFor(cfg)
|
||||||
|
seed := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
|
||||||
|
seed.MarkStageSucceeded("optional", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), []manifest.ArtifactRecord{{
|
||||||
|
Kind: "old_output",
|
||||||
|
SourceID: "narratio.example.old",
|
||||||
|
LocalPath: "artifacts/old.json",
|
||||||
|
}})
|
||||||
|
seed.Stages["optional"].Logs = []string{"old.log"}
|
||||||
|
seed.Stages["optional"].GeneratedConfigs = []string{"old.yml"}
|
||||||
|
seed.Stages["optional"].Metadata = map[string]any{"old": true}
|
||||||
|
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
|
||||||
|
t.Fatalf("MkdirAll() error = %v", err)
|
||||||
|
}
|
||||||
|
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
|
||||||
|
t.Fatalf("Save() seed manifest error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
order := []string{}
|
||||||
|
optionalRuns := 0
|
||||||
|
stages := []stage.Stage{
|
||||||
|
resultStage{
|
||||||
|
name: "optional",
|
||||||
|
runs: &optionalRuns,
|
||||||
|
order: &order,
|
||||||
|
result: &stage.StageResult{
|
||||||
|
Disposition: stage.StageDispositionSkipped,
|
||||||
|
SkipReason: "integration_disabled",
|
||||||
|
Logs: []string{"runs/current/optional.log"},
|
||||||
|
GeneratedConfigs: []string{"runs/current/optional.yml"},
|
||||||
|
Metadata: map[string]any{"enabled": false},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
resultStage{name: "later", order: &order, result: &stage.StageResult{}},
|
||||||
|
}
|
||||||
|
summary, err := executeStages(context.Background(), cfg, stages, RunOptions{Force: true})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if strings.Join(order, ",") != "optional,later" {
|
||||||
|
t.Fatalf("execution order = %v, want optional then later", order)
|
||||||
|
}
|
||||||
|
if len(summary.Executed) != 2 || len(summary.Skipped) != 1 || summary.Skipped[0] != "optional" {
|
||||||
|
t.Fatalf("summary = %#v, want optional executed and self-skipped before later", summary)
|
||||||
|
}
|
||||||
|
|
||||||
|
sessionManifest, err := store.Load(context.Background(), summary.ManifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Load() session manifest error = %v", err)
|
||||||
|
}
|
||||||
|
selfSkipped := sessionManifest.Stages["optional"]
|
||||||
|
if selfSkipped == nil || selfSkipped.Status != manifest.StatusSkipped {
|
||||||
|
t.Fatalf("optional stage = %#v, want skipped", selfSkipped)
|
||||||
|
}
|
||||||
|
if len(selfSkipped.Outputs) != 0 {
|
||||||
|
t.Fatalf("optional outputs = %#v, want old outputs cleared", selfSkipped.Outputs)
|
||||||
|
}
|
||||||
|
if selfSkipped.Error == nil || selfSkipped.Error.Message != "integration_disabled" {
|
||||||
|
t.Fatalf("optional skip reason = %#v, want integration_disabled", selfSkipped.Error)
|
||||||
|
}
|
||||||
|
if len(selfSkipped.Logs) != 1 || selfSkipped.Logs[0] != "runs/current/optional.log" ||
|
||||||
|
len(selfSkipped.GeneratedConfigs) != 1 || selfSkipped.GeneratedConfigs[0] != "runs/current/optional.yml" ||
|
||||||
|
selfSkipped.Metadata["enabled"] != false || selfSkipped.Metadata["old"] != nil {
|
||||||
|
t.Fatalf("optional result details = %#v, want current bounded diagnostics and metadata", selfSkipped)
|
||||||
|
}
|
||||||
|
if later := sessionManifest.Stages["later"]; later == nil || later.Status != manifest.StatusSucceeded {
|
||||||
|
t.Fatalf("later stage = %#v, want succeeded", later)
|
||||||
|
}
|
||||||
|
|
||||||
|
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("LoadRun() error = %v", err)
|
||||||
|
}
|
||||||
|
runStage := runManifest.Stages["optional"]
|
||||||
|
if runStage == nil || runStage.Action != manifest.RunStageActionRun || runStage.Status != manifest.StatusSkipped {
|
||||||
|
t.Fatalf("run optional stage = %#v, want run action with skipped status", runStage)
|
||||||
|
}
|
||||||
|
if len(runStage.Logs) != 1 || runStage.Metadata["enabled"] != false {
|
||||||
|
t.Fatalf("run optional stage details = %#v, want result diagnostics and metadata", runStage)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = executeStages(context.Background(), cfg, []stage.Stage{stages[0]}, RunOptions{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("second executeStages() error = %v", err)
|
||||||
|
}
|
||||||
|
if optionalRuns != 2 {
|
||||||
|
t.Fatalf("optional runs = %d, want self-skipped stage reconsidered", optionalRuns)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteStagesRejectsSkippedResultWithOutputs(t *testing.T) {
|
||||||
|
cfg := testConfig(t)
|
||||||
|
invalid := resultStage{name: "optional", result: &stage.StageResult{
|
||||||
|
Disposition: stage.StageDispositionSkipped,
|
||||||
|
SkipReason: "integration_disabled",
|
||||||
|
Outputs: []artifacts.Ref{{Kind: "unexpected"}},
|
||||||
|
}}
|
||||||
|
|
||||||
|
summary, err := executeStages(context.Background(), cfg, []stage.Stage{invalid}, RunOptions{})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("executeStages() error = nil, want invalid skipped result failure")
|
||||||
|
}
|
||||||
|
if summary != nil {
|
||||||
|
t.Fatalf("summary = %#v, want nil", summary)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "skipped result contains 1 output") {
|
||||||
|
t.Fatalf("error = %q, want skipped-output validation", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
loaded, loadErr := (&manifest.LocalStore{}).Load(context.Background(), manifestPathFor(cfg))
|
||||||
|
if loadErr != nil {
|
||||||
|
t.Fatalf("Load() session manifest error = %v", loadErr)
|
||||||
|
}
|
||||||
|
if got := loaded.Stages["optional"]; got == nil || got.Status != manifest.StatusFailed {
|
||||||
|
t.Fatalf("optional stage = %#v, want failed", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteStagesRunLocalArtifactsAndCanonicalSync(t *testing.T) {
|
func TestExecuteStagesRunLocalArtifactsAndCanonicalSync(t *testing.T) {
|
||||||
cfg := testConfig(t)
|
cfg := testConfig(t)
|
||||||
stages := []stage.Stage{
|
stages := []stage.Stage{
|
||||||
@@ -977,7 +1358,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
|
|||||||
seed.S3Bucket = "my-dnd-archive"
|
seed.S3Bucket = "my-dnd-archive"
|
||||||
seed.S3SessionPrefix = "dnd/campaigns/" + cfg.Session.Campaign + "/sessions/" + cfg.Session.SessionID + "/"
|
seed.S3SessionPrefix = "dnd/campaigns/" + cfg.Session.Campaign + "/sessions/" + cfg.Session.SessionID + "/"
|
||||||
seed.S3RunPrefix = seed.S3SessionPrefix + "runs/" + runID + "/"
|
seed.S3RunPrefix = seed.S3SessionPrefix + "runs/" + runID + "/"
|
||||||
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "analyze"} {
|
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze"} {
|
||||||
seed.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
seed.MarkStageSucceeded(name, time.Now().UTC(), nil)
|
||||||
}
|
}
|
||||||
if err := tc.env.ManifestStore.Save(context.Background(), manifestPathFor(cfg), seed); err != nil {
|
if err := tc.env.ManifestStore.Save(context.Background(), manifestPathFor(cfg), seed); err != nil {
|
||||||
@@ -1015,11 +1396,13 @@ func testConfig(t *testing.T) *config.Config {
|
|||||||
pipelinePath := filepath.Join(cfgDir, "pipeline.yml")
|
pipelinePath := filepath.Join(cfgDir, "pipeline.yml")
|
||||||
|
|
||||||
mustWriteFile(t, pipelinePath, "workspace:\n root: "+workspace+"\n")
|
mustWriteFile(t, pipelinePath, "workspace:\n root: "+workspace+"\n")
|
||||||
mustWriteFile(t, campaignPath, "campaign_id: sample-campaign\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n")
|
mustWriteFile(t, campaignPath, "campaign_id: sample-campaign\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n players_file: ./players.yml\n party_file: ./party.yml\n")
|
||||||
mustWriteFile(t, sessionPath, "session_id: 2026-05-03\ncampaign: sample-campaign\ninputs:\n audio_dir: ./audio\n")
|
mustWriteFile(t, sessionPath, "session_id: 2026-05-03\ncampaign: sample-campaign\ninputs:\n audio_dir: ./audio\n")
|
||||||
mustWriteFile(t, filepath.Join(cfgDir, "speakers.yml"), "alice: alice.flac\n")
|
mustWriteFile(t, filepath.Join(cfgDir, "speakers.yml"), "alice: alice.flac\n")
|
||||||
mustWriteFile(t, filepath.Join(cfgDir, "autocorrect.yml"), "[]\n")
|
mustWriteFile(t, filepath.Join(cfgDir, "autocorrect.yml"), "[]\n")
|
||||||
mustWriteFile(t, filepath.Join(cfgDir, "glossary.yml"), "[]\n")
|
mustWriteFile(t, filepath.Join(cfgDir, "glossary.yml"), "[]\n")
|
||||||
|
mustWriteFile(t, filepath.Join(cfgDir, "players.yml"), "[]\n")
|
||||||
|
mustWriteFile(t, filepath.Join(cfgDir, "party.yml"), "[]\n")
|
||||||
mustWriteFile(t, filepath.Join(cfgDir, "audio", "alice.flac"), "audio")
|
mustWriteFile(t, filepath.Join(cfgDir, "audio", "alice.flac"), "audio")
|
||||||
|
|
||||||
return &config.Config{
|
return &config.Config{
|
||||||
@@ -1044,6 +1427,16 @@ func testConfig(t *testing.T) *config.Config {
|
|||||||
ConfigPath: campaignPath,
|
ConfigPath: campaignPath,
|
||||||
Source: "campaign_config",
|
Source: "campaign_config",
|
||||||
},
|
},
|
||||||
|
PlayersFile: config.ResolvedInputFile{
|
||||||
|
Path: "./players.yml",
|
||||||
|
ConfigPath: campaignPath,
|
||||||
|
Source: "campaign_config",
|
||||||
|
},
|
||||||
|
PartyFile: config.ResolvedInputFile{
|
||||||
|
Path: "./party.yml",
|
||||||
|
ConfigPath: campaignPath,
|
||||||
|
Source: "campaign_config",
|
||||||
|
},
|
||||||
},
|
},
|
||||||
Session: &config.SessionConfig{
|
Session: &config.SessionConfig{
|
||||||
SessionID: "2026-05-03",
|
SessionID: "2026-05-03",
|
||||||
@@ -1053,6 +1446,8 @@ func testConfig(t *testing.T) *config.Config {
|
|||||||
SpeakersFile: "./speakers.yml",
|
SpeakersFile: "./speakers.yml",
|
||||||
AutocorrectFile: "./autocorrect.yml",
|
AutocorrectFile: "./autocorrect.yml",
|
||||||
GlossaryFile: "./glossary.yml",
|
GlossaryFile: "./glossary.yml",
|
||||||
|
PlayersFile: "./players.yml",
|
||||||
|
PartyFile: "./party.yml",
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -1075,6 +1470,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`
|
`
|
||||||
sessionYAML := `session_id: 2026-05-03
|
sessionYAML := `session_id: 2026-05-03
|
||||||
campaign: sample-campaign
|
campaign: sample-campaign
|
||||||
@@ -1083,6 +1480,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`
|
`
|
||||||
mustWriteFile(t, pipelinePath, pipelineYAML)
|
mustWriteFile(t, pipelinePath, pipelineYAML)
|
||||||
mustWriteFile(t, campaignPath, campaignYAML)
|
mustWriteFile(t, campaignPath, campaignYAML)
|
||||||
|
|||||||
@@ -59,3 +59,37 @@ func parseSessionAwareFlags(command string, fs *flag.FlagSet, args []string, ses
|
|||||||
}
|
}
|
||||||
return resolveParsedSessionID(command, positionalSessionID, fs, sessionID)
|
return resolveParsedSessionID(command, positionalSessionID, fs, sessionID)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func parseSessionIDAndOnePositionalArg(command, argName string, fs *flag.FlagSet, args []string, sessionID *string) (string, error) {
|
||||||
|
var positionalSessionID string
|
||||||
|
value := ""
|
||||||
|
if len(args) >= 2 && !isCLIFlagToken(args[0]) && !isCLIFlagToken(args[1]) {
|
||||||
|
positionalSessionID = strings.TrimSpace(args[0])
|
||||||
|
value = strings.TrimSpace(args[1])
|
||||||
|
args = append([]string(nil), args[2:]...)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := fs.Parse(args); err != nil {
|
||||||
|
return "", fmt.Errorf("%s: invalid flags: %w", command, err)
|
||||||
|
}
|
||||||
|
if value == "" {
|
||||||
|
switch fs.NArg() {
|
||||||
|
case 2:
|
||||||
|
positionalSessionID = strings.TrimSpace(fs.Arg(0))
|
||||||
|
value = strings.TrimSpace(fs.Arg(1))
|
||||||
|
case 1:
|
||||||
|
if strings.TrimSpace(*sessionID) == "" {
|
||||||
|
return "", fmt.Errorf("%s: expected session_id and %s", command, argName)
|
||||||
|
}
|
||||||
|
value = strings.TrimSpace(fs.Arg(0))
|
||||||
|
default:
|
||||||
|
return "", fmt.Errorf("%s: expected session_id and %s", command, argName)
|
||||||
|
}
|
||||||
|
} else if fs.NArg() != 0 {
|
||||||
|
return "", fmt.Errorf("%s: unexpected positional arguments", command)
|
||||||
|
}
|
||||||
|
if err := applyPositionalSessionID(command, positionalSessionID, sessionID); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return value, nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -22,6 +22,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`
|
`
|
||||||
if err := os.WriteFile(sessionPath, []byte(sessionTemplate), 0o644); err != nil {
|
if err := os.WriteFile(sessionPath, []byte(sessionTemplate), 0o644); err != nil {
|
||||||
t.Fatalf("write session template: %v", err)
|
t.Fatalf("write session template: %v", err)
|
||||||
@@ -75,6 +77,8 @@ inputs:
|
|||||||
speakers_file: ./speakers.yml
|
speakers_file: ./speakers.yml
|
||||||
autocorrect_file: ./autocorrect.yml
|
autocorrect_file: ./autocorrect.yml
|
||||||
glossary_file: ./glossary.yml
|
glossary_file: ./glossary.yml
|
||||||
|
players_file: ./players.yml
|
||||||
|
party_file: ./party.yml
|
||||||
`
|
`
|
||||||
if err := os.WriteFile(sessionPath, []byte(sessionYAML), 0o644); err != nil {
|
if err := os.WriteFile(sessionPath, []byte(sessionYAML), 0o644); err != nil {
|
||||||
t.Fatalf("write session.yml: %v", err)
|
t.Fatalf("write session.yml: %v", err)
|
||||||
|
|||||||
@@ -162,8 +162,8 @@ func TestExecuteWorkflowCommandsAcceptPositionalSessionID(t *testing.T) {
|
|||||||
wantForce bool
|
wantForce bool
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "resume",
|
name: "run",
|
||||||
args: []string{"resume", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath},
|
args: []string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath},
|
||||||
wantStage: "prepare",
|
wantStage: "prepare",
|
||||||
wantForce: false,
|
wantForce: false,
|
||||||
},
|
},
|
||||||
@@ -346,7 +346,7 @@ func TestExecuteSessionLocksMutationAcceptsPositionalSessionID(t *testing.T) {
|
|||||||
if code != 0 {
|
if code != 0 {
|
||||||
t.Fatalf("locks remove exit code = %d, want 0; stderr=%q", code, stderr.String())
|
t.Fatalf("locks remove exit code = %d, want 0; stderr=%q", code, stderr.String())
|
||||||
}
|
}
|
||||||
store, err := config.LoadPublishLockStoreBytes("locks.yml", fake.Objects[key].Data, nil)
|
store, err := config.LoadPublishLockStoreBytes("locks.yml", fake.Objects[key].Data, nil, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("LoadPublishLockStoreBytes() error = %v", err)
|
t.Fatalf("LoadPublishLockStoreBytes() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user