45 Commits

Author SHA1 Message Date
a7ec195587 Prepare a staged implementation plan to address the audit findings. 2026-08-10 17:08:57 +00:00
13de820931 Finalize codebase audit findings 2026-08-10 15:14:54 +00:00
14ef59aaed Document test suite policy audit conclusions 2026-08-10 15:04:05 +00:00
f387222fce Document maintainability audit conclusions 2026-08-10 14:53:19 +00:00
9cb9008dfc Document analyze dependency audit findings 2026-08-10 14:38:49 +00:00
083decc5b4 Document extraction audit findings 2026-08-10 14:24:03 +00:00
57cac5d3f7 Document ordinary stage audit findings 2026-08-10 14:09:54 +00:00
0a772e03b4 Document external adapter audit findings 2026-08-10 13:53:49 +00:00
0920062a38 Document configuration and composition audit findings 2026-08-10 13:33:59 +00:00
39afe644eb Document restore and previous-state audit findings 2026-08-10 13:16:35 +00:00
e3ee3de10a Audit publish commit and cleanup behavior 2026-08-10 12:55:10 +00:00
9c72db56e9 Audit artifact paths and filesystem safety 2026-08-10 12:42:49 +00:00
bb2d606dbb Audit runner and manifest lifecycle behavior 2026-08-10 12:28:36 +00:00
9850767a8a Establish the codebase audit baseline and contract map 2026-08-10 12:12:48 +00:00
74e2d21de5 Close the completed roadmap documents 2026-08-10 03:24:49 +00:00
7cb18a1a40 Reconcile promotion and manifest documentation 2026-08-10 03:04:20 +00:00
b556fc2f4f Clear superseded session stage result details 2026-08-10 02:53:28 +00:00
b99bd38eb4 Harden bundle promotion against symlink replacement 2026-08-10 02:45:51 +00:00
701b6726d7 Reconcile Notarius extraction documentation 2026-08-10 02:10:37 +00:00
665039f4dc Support atomic directory promotion across platforms 2026-08-10 01:58:52 +00:00
ef8dae776e Enforce canonical Notarius bundle paths 2026-08-10 01:50:31 +00:00
d01775b68a Exclude staged Notarius bundles from publish uploads 2026-08-10 01:43:43 +00:00
0d6f2dd0ce Invalidate downstream results when stages are replaced 2026-08-10 01:38:24 +00:00
df40cbec6e Document and validate Notarius extraction workflows 2026-08-10 00:42:44 +00:00
0341e0c7c0 Publish and inspect configured extraction artifacts 2026-08-10 00:32:24 +00:00
39af7d4f3c Integrate extraction artifacts into analysis catalog 2026-08-10 00:24:02 +00:00
bba582b4ca Integrate extraction lifecycle and resume validation 2026-08-10 00:14:46 +00:00
1f16a85330 Implement direct Notarius extraction execution 2026-08-09 23:59:50 +00:00
f9482639d4 Add the Notarius subprocess adapter 2026-08-09 23:47:38 +00:00
dce721cdbd Add safe immutable directory promotion 2026-08-09 23:37:02 +00:00
98734644d6 Add Notarius configuration and extraction source policy 2026-08-09 23:30:32 +00:00
951383226c Add artifact provenance and stage skip outcomes 2026-08-09 23:20:32 +00:00
df58595d1e Remove completed documentation alignment roadmaps 2026-08-09 22:02:07 +00:00
c3c14e7468 Complete documentation alignment roadmap 2026-08-09 21:53:04 +00:00
e7319ea016 Align internal documentation and maintained examples 2026-08-09 21:50:21 +00:00
bd2d5e2496 Clarify user and integration documentation contracts 2026-08-09 21:41:23 +00:00
115a44f629 Align documentation entry points and internal overview 2026-08-09 21:32:11 +00:00
18411dc5b5 Move contributor guidance to its canonical location 2026-08-09 21:27:58 +00:00
e23dc1ab6e Refocus the Narratio architecture policy 2026-08-09 21:26:18 +00:00
e1359ea227 Adopt canonical documentation ownership policy 2026-08-09 21:23:50 +00:00
7fdd99ec27 Prepare roadmap for documentation policy update 2026-08-09 21:21:00 +00:00
a90231ce0c Implement support for passing a session_id variable to scriptorium to support sticky routing 2026-07-02 21:04:37 -05:00
ed879b8bb0 Clean up obsolete placeholder code 2026-07-02 20:47:08 -05:00
717451512a Implemented new campaign/session stable inputs and corresponding input source references
All checks were successful
ci/woodpecker/tag/release Pipeline was successful
2026-05-27 09:34:05 -05:00
3ddb3a947b Update pipeline defaults so trim is enabled when omitted 2026-05-27 08:35:02 -05:00
139 changed files with 16827 additions and 1192 deletions

View File

@@ -1,22 +1,42 @@
# narratio
Narratio is a stage-driven Go orchestrator for turning D&D session audio into polished transcripts and generated artifacts.
Narratio is a stage-driven Go orchestrator for turning D&D session audio into
polished transcripts, validated Notarius extraction lanes, and generated
artifacts.
It runs a deterministic workflow across `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `render`, `analyze`, and `publish`, with manifest-driven continuation and restore support.
It runs a deterministic workflow with manifest-driven continuation, remote
publish, and restore support.
```bash
```sh
narratio run 2026-04-04
```
This requires resolvable `pipeline.yml`, `campaign.yml`, and concrete `session.yml` (or explicit config flags).
This requires resolvable `pipeline.yml`, `campaign.yml`, and concrete
`session.yml` files or their explicit command-line alternatives.
## Documentation
- [CLI Reference](docs/cli.md)
- [Configuration](docs/config.md)
- [Operations](docs/operations.md)
- [Troubleshooting](docs/troubleshooting.md)
- [Internal Component Contracts](docs/internal/README.md)
- [Development Guide](docs/policy/development.md)
- [Architecture Principles](docs/policy/architecture.md)
- [Maintained Examples](examples/)
- [CLI reference](docs/cli.md) — commands, arguments, flags, and invocation
behavior.
- [Configuration](docs/config.md) — discovery, fields, defaults, and
validation.
- [Operations](docs/operations.md) — runtime workflow, state, publishing,
recovery, and cleanup.
- [Troubleshooting](docs/troubleshooting.md) — symptom-driven diagnosis and
safe remedies.
- [Integration contracts](docs/integrations/) — external tools, formats, and
compatibility expectations.
- [Maintained examples](examples/README.md) — complete copyable configuration
and input files.
## Maintainer Documentation
- [Development guide](docs/development.md) — first-read orientation and
task-specific reading routes.
- [Internal overview](docs/internal/overview.md) — implemented component map.
- [Architecture](docs/policy/architecture.md) — normative boundaries and
invariants.
- [Documentation policy](docs/policy/documentation.md) — canonical ownership
and maintenance rules.
- [Testing policy](docs/policy/testing.md) — test value, boundaries, and
sufficiency.

View File

@@ -75,7 +75,10 @@ narratio run <session_id> [--force] [--artifacts <name[,name...]>] [...common co
Behavior:
- evaluates full stage order;
- skips already-succeeded stages unless `--force` is set;
- runs `extract` between `trim` and `render`; an omitted or disabled Notarius
configuration records an explicit `notarius_disabled` self-skip;
- skips already-succeeded stages unless `--force` is set or a stage-specific
resume check finds its durable result obsolete;
- continues interrupted or partially completed sessions by running non-succeeded stages;
- writes session and run manifests.
@@ -93,6 +96,7 @@ Valid stage names:
- `polish`
- `normalize`
- `trim`
- `extract`
- `render`
- `analyze`
- `publish`
@@ -135,14 +139,13 @@ narratio clean --all [--dry-run] [--clear-cache] [--config <pipeline.yml>]
Behavior:
- session mode removes:
- `{workspace.root}/work/{campaign}/{session_id}`
- `{spool.root}/{campaign}/{session_id}`
- `--all` removes:
- `{workspace.root}/work/*`
- direct children under `{spool.root}`
- session mode removes the selected session's local work and spool state;
- `--all` removes all local session work and spool state;
- cache remains unless `--clear-cache` is provided.
See [Operations: Cleanup](./operations.md#cleanup) for deletion scope and
post-publish cleanup behavior.
### `session plan`
```bash
@@ -207,17 +210,11 @@ Behavior:
- discovers committed remote current state;
- plans local restores;
- writes `reports/restore-latest.json` on execution;
- writes an execution report;
- blocks conflicting overwrites unless `--force` is set.
Default restore scope:
- `manifest.json`
- `transcripts/**`
- `artifacts/**`
- `previous/**` when required by configured previous-session inputs
`audio/**` is included only with `--include-audio`.
See [Operations: Restore Workflow](./operations.md#restore-workflow) for the
default restore scope, report location, and conflict-handling workflow.
### `session artifacts`
@@ -225,7 +222,10 @@ Default restore scope:
narratio session artifacts <session_id> [--remote] [...common config flags]
```
Lists effective built-in and configured artifact sources, publish rules, lock state, and optional remote published-state availability.
Lists effective built-in, configured Scriptorium, and configured extraction
sources; reports planned, available, unavailable, and published state without
reading payload bodies; and includes publish rules, lock state, and optional
remote published-state availability.
### `session locks`
@@ -237,10 +237,13 @@ narratio session locks remove <session_id> <source> [...common config flags]
Behavior:
- list mode merges static `pipeline.publish.locks` with remote `{session_prefix}/locks.yml`;
- list mode reports the effective merge of static and remote locks;
- add/remove mutate only remote locks;
- static locks from pipeline config cannot be removed by CLI commands.
See [Operations: Publish Locks](./operations.md#publish-locks) for lock storage
and precedence.
## `--artifacts` Selection Rules
- accepted on `run`, `run-stage`, `analyze`, and `publish`;
@@ -252,7 +255,9 @@ Effects:
- filters analyze execution to selected configured artifacts;
- filters publish rules that source `narratio.artifact.<name>`;
- does not filter built-in transcript/bounds publish sources.
- does not filter built-in transcript/bounds or explicitly configured
`narratio.extraction.<name>` publish sources; and
- does not select or filter Notarius lanes.
## Common Workflows
@@ -279,3 +284,18 @@ Force publish only:
```bash
narratio publish 2026-04-04
```
## Output And Exit Behavior
- Successful commands write their result or summary to standard output and
exit with status `0`.
- Command failures and invalid invocations write an error to standard error and
exit with status `1`.
- An unknown top-level command also prints the top-level usage summary to
standard error.
- `session restore --help` prints its command-specific usage and exits with
status `0`.
Output is intended for operator inspection. Narratio does not currently offer
a machine-readable CLI output mode; durable machine-readable state is recorded
in manifests and reports described in [Operations](./operations.md).

View File

@@ -44,7 +44,7 @@ using configured object storage.
- Session files must be concrete; unresolved `{{ ... }}` placeholders fail load.
- Pipeline defaults are applied before validation.
- Campaign and session identities must agree.
- Stable files (`speakers_file`, `autocorrect_file`, `glossary_file`) resolve from session overrides when provided, otherwise from campaign defaults.
- Stable files (`speakers_file`, `autocorrect_file`, `glossary_file`, `players_file`, `party_file`) resolve from session overrides when provided, otherwise from campaign defaults.
- Exactly one audio mode must be configured in session input:
- local (`audio_dir` or `audio_files`), or
- S3 (`audio_s3.prefix`).
@@ -69,6 +69,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
```
`session.yml` (local audio)
@@ -116,6 +118,9 @@ Rules:
- `outputs[].source` is required.
- `outputs[].dest` may be omitted when derivable from source.
- extraction sources require an explicit `outputs[].dest` and publish only when
a rule names that source; the Notarius index and complete bundle are not
publish sources.
- `outputs[].required` defaults to `true`.
- static locks (`pipeline.publish.locks`) merge with remote locks (`{session_prefix}/locks.yml`), with static locks taking precedence on duplicates.
@@ -184,16 +189,23 @@ Rules:
| `pipeline.normalize.output_path` | string | No | `transcripts/final.json` |
| `pipeline.normalize.output_schema` | string | No | `seriatim-intermediate` |
| `pipeline.normalize.report` | bool | No | `true` |
| `pipeline.trim.enabled` | bool | No | `false` |
| `pipeline.trim.output_path` | string | Conditional | required when trim enabled |
| `pipeline.trim.bounds.prompt_id` | string | Conditional | required when trim enabled |
| `pipeline.trim.enabled` | bool | No | `true` |
| `pipeline.trim.output_path` | string | No | `transcripts/final.trimmed.json` |
| `pipeline.trim.bounds.prompt_id` | string | No | `dnd.session_bounds` |
| `pipeline.trim.bounds.profile_id` | string | No | empty |
| `pipeline.trim.bounds.transcript_input_name` | string | Conditional | required when trim enabled |
| `pipeline.trim.bounds.output_path` | string | Conditional | required when trim enabled |
| `pipeline.trim.bounds.transcript_input_name` | string | No | `transcript` |
| `pipeline.trim.bounds.output_path` | string | No | `artifacts/session_bounds.json` |
| `pipeline.trim.bounds.timeout` | duration | No | `10m` |
| `pipeline.trim.bounds.render_debug` | bool | No | `false` |
| `pipeline.trim.bounds.render_output_path` | string | Conditional | required when `render_debug` is true |
| `pipeline.trim.seriatim.report` | bool | No | `false` |
| `pipeline.notarius.enabled` | bool | No | `false` |
| `pipeline.notarius.binary` | string | No | `notarius` |
| `pipeline.notarius.config_path` | string | Conditional | required when enabled; relative paths resolve from the pipeline file directory |
| `pipeline.notarius.pipeline_id` | string | Conditional | required when enabled |
| `pipeline.notarius.timeout` | duration | No | `3h`; must be positive |
| `pipeline.notarius.working_directory` | string | No | directory containing resolved `config_path`; relative paths resolve from the pipeline file directory |
| `pipeline.notarius.outputs` | map | Conditional | at least one entry when enabled |
| `pipeline.render.enabled` | bool | No | `true` |
| `pipeline.render.format` | string | No | `markdown` (only supported value) |
| `pipeline.render.title` | string | No | empty (falls back to `session.title` when set) |
@@ -207,7 +219,27 @@ Rules:
| `pipeline.scriptorium.artifacts` | map | No | empty |
| `pipeline.notification.backend` | string | No | empty |
| `pipeline.notification.recipient` | string | No | empty |
| `pipeline.notification.timeout` | duration | No | `30s` |
| `pipeline.notification.timeout` | duration | No | empty |
### Notarius Output Entries
For each `pipeline.notarius.outputs.<name>`:
| Field | Type | Required | Rule |
| --- | --- | --- | --- |
| `lane_id` | string | Yes | unique Notarius lane ID |
| `media_type` | string | Yes | exact accepted descriptor media type |
| `schema_id` | string | Yes | exact accepted descriptor schema ID |
| `schema_version` | string | Yes | exact accepted descriptor schema version |
| `module_key` | string | No | exact accepted module key when set |
Output names must match `^[a-z][a-z0-9_]*$` and become selectable sources named
`narratio.extraction.<name>`. Lane IDs must be unique. Every declared output is
required from a successful Notarius result; a missing, rejected, duplicate, or
contract-incompatible lane fails extraction. See the
[complete maintained example](../examples/pipeline.full.annotated.yml) for the
current ten-lane D&D mapping and the [Notarius contract](./integrations/notarius.md)
for compatibility ownership.
### Scriptorium Artifact Entries
@@ -223,13 +255,15 @@ For each `pipeline.scriptorium.artifacts.<name>`:
| `output_path` | string | Conditional | required when enabled; also required when referenced by publish/output/input rules |
| `timeout` | duration | No | artifact override |
| `inputs` | map | No | input key names must be non-empty |
| `vars` | map | No | values must be string or bool |
| `vars` | map | No | values must be string or bool; `session_id` is reserved and overwritten by Narratio |
Narratio adds `session_id=narratio-session-<session_id>` to every Scriptorium request for sticky upstream LLM routing. If an artifact config sets `vars.session_id`, Narratio replaces that value before invoking Scriptorium. Use a different variable name if a prompt needs the raw Narratio session ID as content.
For each artifact input `pipeline.scriptorium.artifacts.<name>.inputs.<input_name>`:
| Field | Type | Required | Rule |
| --- | --- | --- | --- |
| `source` | string | Yes | built-in runtime source, `narratio.artifact.<name>`, or `narratio.previous_session.artifact.<name>` |
| `source` | string | Yes | built-in runtime source, prepared input source, `narratio.extraction.<name>`, `narratio.artifact.<name>`, or `narratio.previous_session.artifact.<name>` |
| `artifact` | string | No | optional passthrough adapter field |
| `path` | string | No | optional passthrough adapter field |
| `required` | bool | No | optional input requirement |
@@ -243,6 +277,8 @@ For each artifact input `pipeline.scriptorium.artifacts.<name>.inputs.<input_nam
| `inputs.speakers_file` | string | Yes | stable input default |
| `inputs.autocorrect_file` | string | Yes | stable input default |
| `inputs.glossary_file` | string | Yes | stable input default |
| `inputs.players_file` | string | Yes | stable input default |
| `inputs.party_file` | string | Yes | stable input default |
### Session
@@ -256,6 +292,8 @@ For each artifact input `pipeline.scriptorium.artifacts.<name>.inputs.<input_nam
| `inputs.speakers_file` | string | No | overrides campaign stable input |
| `inputs.autocorrect_file` | string | No | overrides campaign stable input |
| `inputs.glossary_file` | string | No | overrides campaign stable input |
| `inputs.players_file` | string | No | overrides campaign stable input |
| `inputs.party_file` | string | No | overrides campaign stable input |
| `inputs.audio_dir` | string | Conditional | local audio mode |
| `inputs.audio_files[]` | list[string] | Conditional | local audio mode |
| `inputs.audio_s3.prefix` | string | Conditional | S3 audio mode |
@@ -266,10 +304,6 @@ Audio rules:
## Maintained Examples
- `examples/pipeline.minimal.yml`
- `examples/pipeline.production.yml`
- `examples/pipeline.full.annotated.yml`
- `examples/campaigns/sample-campaign/campaign.yml`
- `examples/session.local-audio.yml`
- `examples/session.s3-audio.yml`
- `examples/session.template.yml`
See the [maintained examples index](../examples/README.md) for complete pipeline,
campaign, session, template, and input fixtures. Keep complete copyable files
there rather than duplicating them in this reference.

43
docs/development.md Normal file
View File

@@ -0,0 +1,43 @@
# Development
This is the first-read landing page for people and LLM coding agents working on
Narratio. It provides a concise repository orientation and routes each kind of
change to its canonical documentation.
Narratio is a stage-driven Go orchestrator for turning D&D session audio into
polished transcripts and generated artifacts. Start with the
[README](../README.md) for product context,
[Architecture](policy/architecture.md) for normative system boundaries, and the
[Internal Overview](internal/overview.md) for implemented component ownership.
## What To Read
| When working on | Read | Why |
| --- | --- | --- |
| Finding the package or component that owns current behavior | [Internal Overview](internal/overview.md) | It is the implemented component inventory and routes to focused internal documents. |
| Application shape, boundaries, dependency direction, runtime invariants, safety properties, or dependencies | [Architecture](policy/architecture.md) | It defines the intended system shape, ownership, and non-goals. |
| Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical owners, audiences, current-behavior rules, and maintenance requirements. |
| Adding, changing, reviewing, rewriting, or deleting tests | [Testing Policy](policy/testing.md) | It defines risk-based sufficiency, durable test boundaries, test-double guidance, and test lifecycle decisions. |
| CLI composition or command behavior | [Internal Overview](internal/overview.md) and [CLI Reference](cli.md) | The overview routes to command ownership; the reference owns public syntax and invocation behavior. |
| Configuration loading, resolution, or user-visible configuration | [Internal Overview](internal/overview.md) and [Configuration](config.md) | The overview routes to implementation ownership; the reference owns fields, defaults, discovery, and validation. |
| Session workflow, status, restore, cleanup, or object storage | [Restore Internals](internal/command-restore.md), [Workspace Internals](internal/workspace.md), [Storage Internals](internal/storage.md), [Operations](operations.md), and [Troubleshooting](troubleshooting.md) | These separate implementation mechanics, operator procedures, and symptom-driven recovery. |
| Pipeline sequencing or the behavior of a stage | [Internal Overview](internal/overview.md) and its focused stage documents | The overview owns the implemented stage inventory and routes to each stage contract. |
| Adapters or external tool contracts | [Adapter Internals](internal/adapters.md) and [Integration Contracts](integrations/README.md) | The internal guide owns adapter composition and mechanics; integration documents own external formats and protocols. |
| Manifests, artifacts, workspace paths, or publish behavior | [Manifest Internals](internal/manifest.md), [Artifact Internals](internal/artifacts.md), [Workspace Internals](internal/workspace.md), [Publish Internals](internal/stage-publish.md), and [Operations](operations.md) | These separate implementation state and resolution from operator-visible layout and lifecycle. |
| Maintained configuration or input examples | [Configuration](config.md) and [Examples](../examples/README.md) | The reference owns field meanings; the examples directory owns complete copyable files. |
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
For an existing subsystem, also inspect its focused tests and package-level
contracts before changing behavior.
## Validation
Use focused package tests while iterating. Run the repository-wide checks when a
change affects shared contracts, application behavior, or maintained
documentation examples:
```sh
go test ./...
go vet ./...
go build ./cmd/narratio
```

View File

@@ -1,19 +1,35 @@
# Integrations Index
## Audience
Developers and coding agents changing Narratio's external integration boundaries.
Operators, developers, and coding agents who need to understand Narratio's
externally observable integration boundaries.
## Scope
`docs/integrations/` is the implementation-level reference for downstream tool adapter contracts.
These docs cover what Narratio expects from external tools and what each adapter guarantees back to stage code.
`docs/integrations/` is the canonical reference for protocols, invocation and
data contracts, logical outputs, and compatibility behavior at external tool
boundaries.
These documents describe what Narratio sends or invokes, what it accepts in
return, and how failures are surfaced. Internal composition and stage mechanics
belong in [the adapter implementation guide](../internal/adapters.md) and the
focused stage documents.
## Integration Contracts
- `audita.md`: transcript polishing adapter (`audita process`).
- `seriatim.md`: merge/normalize/trim/render adapter (`seriatim`).
- `scriptorium.md`: artifact run/render adapter (`scriptorium run|render`).
- [Audita](./audita.md): transcript polishing (`audita process`).
- [Notarius](./notarius.md): complete pipeline execution and safe JSON bundle
discovery (`notarius run`).
- [Seriatim](./seriatim.md): merge, normalize, trim, and render operations.
- [Scriptorium](./scriptorium.md): artifact generation and debug rendering
(`scriptorium run|render`).
- [WhisperX](./whisperx.md): speaker-audio transcription over HTTP.
## Related Canonical Docs
- `docs/config.md`: operator-facing configuration reference.
- `docs/internal/adapters.md`: shared adapter boundary and runner wiring.
- `docs/internal/stage-*.md`: stage-specific integration usage.
- [Configuration](../config.md): operator-facing configuration reference.
- [Adapter implementation](../internal/adapters.md): shared adapter boundary and
runner wiring.
- [Internal documentation](../internal/overview.md): stage-specific integration
usage and component ownership.

View File

@@ -3,16 +3,12 @@
## Purpose
Define the Audita adapter contract used by the `polish` stage.
## Adapter Boundary
Interface:
- `audita.Runner`
- method: `Run(ctx, PolishRequest) (PolishResult, error)`
## External Boundary
Primary implementation:
- `internal/adapters/audita/SubprocessRunner`
Execution mode:
- subprocess invocation of `audita process`
Narratio invokes `audita process` as a subprocess for each polish operation.
The configured timeout and parent cancellation bound the invocation. Internal
runner composition is documented in
[the adapter implementation guide](../internal/adapters.md).
## Request Contract
`PolishRequest` carries:
@@ -52,9 +48,12 @@ Failure results still include output/log/config/exit metadata for diagnostics.
- Generated invocation YAML (`audita.generated.v1`) is emitted when requested.
- Manifest writes are stage-owned; adapter itself is stateless.
## Config Mapping
Config fields consumed through runner/stage wiring are under `pipeline.audita.*`.
## Configuration
Operator-selected values are defined under `pipeline.audita.*` in the
[configuration reference](../config.md#pipeline).
Maintained example with Audita config:
- `examples/pipeline.full.annotated.yml`
- `examples/pipeline.production.yml`
- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml)
- [Production-shaped pipeline](../../examples/pipeline.production.yml)

View File

@@ -0,0 +1,83 @@
# Notarius Integration Contract
## Boundary
Narratio uses Notarius as a subprocess to extract configured structured JSON
lanes from the final trimmed Seriatim transcript. Narratio owns invocation,
safe bundle discovery, lane selection, and its own artifact metadata. Notarius
owns pipeline definitions, lane schemas, the receipt, and bundle formats.
Canonical Notarius references:
- [Subprocess consumer contract](https://gitea.maximumdirect.net/eric/notarius/src/branch/main/docs/consumers/subprocess.md)
- [D&D pipeline and lane contracts](https://gitea.maximumdirect.net/eric/notarius/src/branch/main/docs/consumers/dnd-pipeline.md)
- [Run-result receipt](https://gitea.maximumdirect.net/eric/notarius/src/branch/main/docs/integrations/run-result.md)
- [JSON output bundle](https://gitea.maximumdirect.net/eric/notarius/src/branch/main/docs/integrations/json-output.md)
The [complete Narratio example](../../examples/pipeline.full.annotated.yml)
records the exact current constraints for all ten D&D lanes. Treat the linked
Notarius documents as canonical when changing those values; Narratio does not
duplicate the complete schemas.
## Invocation
When `pipeline.notarius.enabled` is true, Narratio resolves the executable,
configuration path, input path, output directory, and working directory to
absolute paths and invokes:
```text
notarius run <pipeline_id> --config <config_path> --input <trimmed_json> --output-dir <staging_dir> --json
```
Standard output is reserved for the JSON receipt. Standard error is captured
separately as diagnostic output. Narratio applies the configured timeout and
does not interpret stdout as a receipt unless the subprocess exits successfully.
It does not pass a Narratio session ID or run `notarius config validate`
automatically; the configured working directory and inherited environment
apply to the subprocess.
## Accepted Result
Narratio currently accepts receipt schema `notarius.run-result.v1`. The receipt
must identify the configured pipeline, and its `index_file` must be exactly
`index.json` beneath the reported bundle root. The production index must name
the management files exactly as `manifest.json`, `rejected.json`, and
`warnings.json`. All receipt, index, and lane paths must stay inside that
bundle; symlinks and non-regular lane payloads are rejected.
Supported receipt and index shapes tolerate unknown fields for forward
compatibility, while required identity, validation, count, manifest,
rejection, warning, and lane-list fields remain mandatory. Narratio applies
bounded reads to the receipt, index, rejection, and warning documents. Optional
chunk-map and evidence-context descriptors must carry their complete generic
contract metadata when present.
For every entry in `pipeline.notarius.outputs`, Narratio requires exactly one
index descriptor with the configured lane ID, media type, schema ID, schema
version, and, when configured, module key. Missing, duplicate, rejected, or
incompatible required lanes fail extraction even if Notarius exited zero.
Unconfigured lanes may remain in the preserved bundle but do not become
selectable Narratio sources.
Each accepted configured lane is registered as
`narratio.extraction.<output_key>`. The bundle index is retained for audit and
resume validation but is not selectable. Scriptorium and publish rules consume
only explicitly named lane sources; `--artifacts` never selects Notarius lanes.
## Failure And Compatibility Behavior
- Startup and nonzero-exit errors fail extraction and retain captured diagnostics.
- Invalid receipt JSON or an unsupported receipt schema fails before bundle use.
- Unsafe or incompatible index data and required-lane rejection fail before the
staged bundle is promoted to durable storage.
- Contract and external provenance metadata are preserved on lane artifact
records and through explicit publication.
Rejection and warning summaries retain structured stage, scope, lane, and
reason-code fields for diagnostics without exposing free-form external messages
or reading lane payload bodies.
Configuration fields and defaults are in [Configuration](../config.md).
Operator paths, rerun procedures, and bundle retention are in
[Operations](../operations.md). See [Troubleshooting](../troubleshooting.md)
for failure recovery.

View File

@@ -3,20 +3,17 @@
## Purpose
Define the Scriptorium adapter contract used by `analyze` and trim-bounds generation in `trim`.
## Adapter Boundary
Interface:
- `scriptorium.Runner`
- methods:
- `RunArtifact(ctx, RunArtifactRequest)`
- `RenderArtifact(ctx, RenderArtifactRequest)`
## External Boundary
Primary implementation:
- `internal/adapters/scriptorium/SubprocessRunner`
Narratio invokes Scriptorium as a subprocess in these modes:
Execution modes:
- `scriptorium run`
- `scriptorium render`
The request timeout and parent cancellation bound each invocation. Internal
runner composition is documented in
[the adapter implementation guide](../internal/adapters.md).
## Request Contract
Both request types carry:
- binary/config/prompt/profile IDs;
@@ -55,12 +52,17 @@ Render behavior:
## Deterministic Behavior
- input and var maps are sorted into deterministic `--input` and `--var` CLI args.
- stage wiring adds `session_id=narratio-session-<session_id>` to every Scriptorium request for sticky upstream routing, overriding any configured `vars.session_id`.
- generated invocation YAML (`scriptorium.generated.v1`) is emitted when requested.
- adapter is stateless and does not own artifact-selection policy.
## Config Mapping
Config fields consumed through runner/stage wiring are under `pipeline.scriptorium.*` plus per-artifact settings under `pipeline.scriptorium.artifacts.*`.
## Configuration
Operator-selected values are defined under `pipeline.scriptorium.*`, including
per-artifact settings under `pipeline.scriptorium.artifacts.*`, in the
[configuration reference](../config.md#pipeline).
Maintained examples with Scriptorium config:
- `examples/pipeline.full.annotated.yml`
- `examples/pipeline.production.yml`
- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml)
- [Production-shaped pipeline](../../examples/pipeline.production.yml)

View File

@@ -3,24 +3,19 @@
## Purpose
Define the Seriatim adapter contract used by `merge`, `normalize`, `trim`, and `render`.
## Adapter Boundary
Interface:
- `seriatim.Runner`
- methods:
- `Run(ctx, MergeRequest)`
- `Normalize(ctx, NormalizeRequest)`
- `Trim(ctx, TrimRequest)`
- `Render(ctx, RenderRequest)`
## External Boundary
Primary implementation:
- `internal/adapters/seriatim/SubprocessRunner`
Narratio invokes Seriatim as a subprocess in these modes:
Execution modes:
- `seriatim merge`
- `seriatim normalize`
- `seriatim trim`
- `seriatim render`
The configured timeout and parent cancellation bound each invocation. Internal
runner composition is documented in
[the adapter implementation guide](../internal/adapters.md).
## Request/Result Contracts
- `MergeRequest`/`MergeResult`: multi-input merge to base transcript, optional report.
- `NormalizeRequest`/`NormalizeResult`: transcript normalization with explicit schema.
@@ -53,9 +48,13 @@ When report paths are provided/enabled, report files must parse as JSON.
- generated invocation YAML (`seriatim.generated.v1`) is emitted when requested.
- adapter does not write manifests or choose stage inputs.
## Config Mapping
Config fields consumed through runner/stage wiring are under `pipeline.seriatim.*` and `pipeline.render.*`.
## Configuration
Operator-selected values are defined under `pipeline.seriatim.*` and
`pipeline.render.*` in the
[configuration reference](../config.md#pipeline).
Maintained examples with Seriatim config:
- `examples/pipeline.full.annotated.yml`
- `examples/pipeline.production.yml`
- [Full annotated pipeline](../../examples/pipeline.full.annotated.yml)
- [Production-shaped pipeline](../../examples/pipeline.production.yml)

View File

@@ -0,0 +1,66 @@
# Integration: WhisperX
## Purpose
WhisperX transcribes each prepared speaker audio file for Narratio's
`transcribe` stage. Narratio uses an HTTP boundary and installs each successful
response as that speaker's raw transcript JSON.
## HTTP Boundary
Narratio sends an HTTP `POST` to the configured transcription URL using
`multipart/form-data` with:
- `file`: the audio file, retaining its base filename; and
- `language`: the configured language string.
The server must return a `2xx` response whose body is valid JSON. Narratio does
not currently require a more specific response schema at this boundary.
## Request And Result Contract
Each adapter request identifies a speaker, a readable audio file, and the
destination for the raw transcript. The HTTP request carries the audio and
language; the speaker identifier remains Narratio orchestration metadata.
On success, Narratio atomically writes the response body to the requested
destination. The adapter result reports that logical output together with the
attempt count, final HTTP status when available, elapsed duration, and adapter
identity metadata. A failed or invalid response is not installed as the
transcript output.
## Retry, Timeout, And Cancellation
- The configured timeout applies independently to each HTTP attempt.
- `retries` means additional attempts after the first.
- HTTP `429`, HTTP `5xx`, attempt timeouts, and network errors are retryable.
- Other HTTP `4xx` responses and explicit cancellation are not retryable.
- Narratio waits the configured retry delay between attempts and aborts that
wait when the parent context is canceled.
## Validation And Failure Semantics
Client construction rejects a missing or invalid absolute transcription URL,
a missing language, a non-positive timeout, negative retries, or a negative
retry delay. A request fails before transmission when its audio or output path
is missing.
Non-`2xx` status, transport failure, response-size overflow, invalid JSON, or
failure to install the output causes the transcription to fail. Errors include
attempt context, and the result retains attempts, final status when available,
and elapsed duration for diagnostics.
## Determinism And Concurrency
Each audio request has stable multipart field names, and successful bytes are
installed atomically. The transcribe stage may process speaker files in
parallel, bounded by the configured concurrency. It records results in stable
speaker order after all work completes; any speaker failure fails the stage.
## Related Canonical Docs
- [Configuration](../config.md#pipeline) defines the operator-selected
WhisperX URL, language, timeouts, retry policy, and concurrency.
- [Adapter implementation](../internal/adapters.md) describes internal wiring.
- [Transcribe stage](../internal/stage-transcribe.md) describes stage mechanics,
durable artifacts, and manifests.

View File

@@ -1,45 +0,0 @@
# Internal Documentation Index
## Audience
Developers and coding agents changing Narratio internals.
## Scope
`docs/internal/` documents implemented internal contracts: stage boundaries, manifest/state behavior, artifact resolution, restore behavior, storage boundaries, and workspace invariants.
User and operator behavior belongs in:
- `docs/cli.md`
- `docs/config.md`
- `docs/operations.md`
- `docs/troubleshooting.md`
## Pipeline Stage Set
Canonical stage order from `internal/stage.All()`:
1. `prepare`
2. `transcribe`
3. `merge`
4. `polish`
5. `normalize`
6. `trim`
7. `render`
8. `analyze`
9. `publish`
10. `notify` (placeholder)
`notify` is currently a placeholder stage with optional notifier call behavior; it has no persisted pipeline outputs.
## Internal Component Docs
- `adapters.md`: external adapter boundaries and default runtime wiring.
- `artifacts.md`: canonical source IDs, runtime catalog behavior, and resolution rules.
- `manifest.md`: session and run manifest contracts.
- `storage.md`: object-store interface and S3 implementation behavior.
- `workspace.md`: local session layout, run-local layout, and cleanup guardrails.
- `command-restore.md`: restore discovery, planning, execution, and reporting.
- `stage-prepare.md`
- `stage-transcribe.md`
- `stage-merge.md`
- `stage-polish.md`
- `stage-normalize.md`
- `stage-trim.md`
- `stage-render.md`
- `stage-analyze.md`
- `stage-publish.md`

View File

@@ -1,49 +1,78 @@
# Internal: Adapters
## Purpose
Define external integration boundaries and default adapter wiring used by app/stage orchestration.
Explain the adapter interfaces and production composition used by application
and stage orchestration. Externally observable protocols and formats belong in
the [integration contracts](../integrations/).
## Adapter Boundaries
Narratio stage logic depends on adapter interfaces, not transport-specific details.
Primary adapters:
- `whisperx.Client`
- `seriatim.Runner`
- `audita.Runner`
- `scriptorium.Runner`
- `notarius.Runner`
- `storage.ObjectStore`
- `notify.Sender`
## Ownership
Adapters own:
- HTTP/subprocess/SDK argument and transport details.
- Backend-specific request/response mapping.
Adapters do not own:
- stage ordering/skip/force logic;
- manifest transitions;
- canonical path policy.
## Default Wiring
`internal/app/runner.go` initializes default adapters when not injected:
- WhisperX HTTP client from pipeline config.
- Seriatim subprocess runner.
- Audita subprocess runner.
- Scriptorium subprocess runner.
- Notarius subprocess runner when extraction is enabled.
- Noop notifier (`notify.NoopSender`).
- Object store only when required by selected stages/config.
Object-store construction goes through `newCommandObjectStore`, which loads configured filesystem secrets before adapter initialization.
Notarius is composed only when extraction is enabled; the extract stage owns
receipt, bundle, and configured-lane policy rather than the adapter.
Object-store construction goes through `newCommandObjectStore`, which loads
configured filesystem secrets before adapter initialization.
## Failure Semantics
- Constructor errors fail stage execution setup early.
- Runtime adapter errors propagate to stage code and then manifest failure handling.
- Subprocess adapters persist stage logs/generated configs through stage-managed paths.
## Test Surfaces
## Implementation And Tests
- Composition: `internal/app/runner.go`, `internal/app/object_store.go`
- Shared subprocess mechanics: `internal/adapters/subprocess`
- Focused adapters: `internal/adapters/{whisperx,seriatim,audita,scriptorium,notarius,storage,notify}`
- `internal/adapters/whisperx/http_test.go`
- `internal/adapters/seriatim/subprocess_test.go`
- `internal/adapters/audita/subprocess_test.go`
- `internal/adapters/scriptorium/subprocess_test.go`
- `internal/adapters/notarius/subprocess_test.go`
- `internal/adapters/storage/*_test.go`
- `internal/app/runner_test.go`
See the [WhisperX](../integrations/whisperx.md),
[Seriatim](../integrations/seriatim.md), [Audita](../integrations/audita.md),
[Scriptorium](../integrations/scriptorium.md), and
[Notarius](../integrations/notarius.md) contracts before changing an
externally visible boundary. Operator-selected values belong in
[Configuration](../config.md).

View File

@@ -1,24 +1,38 @@
# Internal: Artifacts
## Purpose
Define canonical artifact IDs, runtime catalog behavior, source resolution rules, and shared current-state mechanics used by app and previous-cache code.
Explain the artifact registry, runtime catalog, resolver, previous-input
requirements, and shared remote current-state mechanics implemented by
`internal/artifacts`. Configuration fields that accept source IDs belong in
[Configuration](../config.md); physical placement belongs in
[Operations](../operations.md).
## Built-in Source IDs
- `narratio.transcript.base` -> `transcripts/base.json` (`merge`)
- `narratio.transcript.polished` -> `transcripts/polished.json` (`polish`)
- `narratio.transcript.final` -> `transcripts/final.json` (`normalize`)
- `narratio.transcript.final_trimmed` -> `transcripts/final.trimmed.json` (`trim`)
- `narratio.transcript.final_markdown` -> `transcripts/final.md` (`render`)
- `narratio.transcript.final_trimmed_markdown` -> `transcripts/final.trimmed.md` (`render`)
- `narratio.bounds.session` -> `artifacts/session_bounds.json` (`trim`)
The internal registry recognizes these stable built-in source IDs:
## Configured and Previous-Session Sources
- `narratio.transcript.base`
- `narratio.transcript.polished`
- `narratio.transcript.final`
- `narratio.transcript.final_trimmed`
- `narratio.transcript.final_markdown`
- `narratio.transcript.final_trimmed_markdown`
- `narratio.bounds.session`
Registry entries bind each ID to its producer, output kind, canonical fallback,
and content validator. The focused stage documents own their input/output flow;
[Configuration](../config.md) owns where operators may select these IDs.
## Configured, Extraction, And Previous-Session Sources
- configured source ID format: `narratio.artifact.<artifact_key>`
- extraction source ID format: `narratio.extraction.<output_key>`
- previous-session source ID format: `narratio.previous_session.artifact.<artifact_key>`
Both formats are validated by strict source-policy rules.
All formats are validated by strict source-policy rules. Extraction sources are
registered only from `pipeline.notarius.outputs`; the Notarius index has no
selectable source ID.
## Runtime Catalog
@@ -47,6 +61,15 @@ Configured sources (`narratio.artifact.*`):
- resolve only through runtime catalog availability.
Extraction sources (`narratio.extraction.*`):
- use the shared registration and manifest hydration path in
`extraction_catalog.go`;
- require a current successful extract record with the exact configured source,
compatible contract and Notarius provenance, a confined regular durable
payload, and matching checksum; and
- are never inferred by scanning the Notarius bundle directory.
Previous-session sources (`narratio.previous_session.artifact.*`):
- resolve only from local `previous/` cache state;
@@ -72,7 +95,8 @@ Validation by content type:
## Current-State Helpers
Artifacts package owns shared remote current-state loading mechanics used by restore, status/validate checks, and previous-cache planning.
Artifacts package owns shared remote current-state loading mechanics used by
restore, status and validation checks, and previous-cache planning.
Core helpers:
@@ -107,9 +131,31 @@ Caller policy is intentionally outside artifacts helpers:
- spool/cache paths;
- S3 session/run/current-state key layout.
See [Workspace Internals](workspace.md) for how callers consume local helpers
and [Operations](../operations.md#local-state-layout) for the authoritative
physical layout.
## Invariants
- source ID formats are stable contracts;
- artifact resolution is deterministic and manifest-aware;
- extraction sources are available only from a compatible successful manifest
record;
- previous-session source resolution in `analyze` is local-only;
- remote current-state key construction remains centralized in artifacts helpers.
## Implementation And Tests
- Registry and resolution: `internal/artifacts/artifact_resolver.go`,
`internal/artifacts/catalog.go`, `internal/artifacts/transcripts.go`,
`internal/artifacts/extraction_catalog.go`
- Current state: `internal/artifacts/current_state.go`
- Paths and keys: `internal/artifacts/paths.go`,
`internal/artifacts/s3_keys.go`
- Previous requirements: `internal/artifacts/previous_requirements.go`
- Tests: `internal/artifacts/artifact_resolver_test.go`,
`internal/artifacts/catalog_test.go`,
`internal/artifacts/extraction_catalog_test.go`,
`internal/artifacts/current_state_test.go`,
`internal/artifacts/paths_model_test.go`,
`internal/artifacts/previous_requirements_test.go`

View File

@@ -1,21 +1,20 @@
# Internal: Command Restore
## Purpose
Define the implemented `narratio session restore` command contract:
- committed remote current-state discovery;
- deterministic restore planning;
- safe local install semantics;
- durable restore reporting.
Explain the implemented restore discovery, planning, installation, and
reporting flow in `internal/app`. User invocation belongs in
[CLI](../cli.md#session-restore), and the operator recovery procedure and
physical restore scope belong in
[Operations](../operations.md#restore-workflow).
Restore is split into explicit phases so remote authority, local conflict
policy, and filesystem mutation can be tested independently.
## Discovery Contract
Restore resolves remote committed state from the session publish current pointers:
- `current/run_id.txt` (required, non-empty);
- `current/manifest.json` (required, decodable).
Current-state discovery uses shared artifacts-level mechanics and validates identity against the resolved request config:
Discovery delegates current-state pointer and manifest loading to
`internal/artifacts`, then validates the result against the resolved request:
- campaign must match;
- session ID must match.
@@ -34,26 +33,11 @@ Planner behavior:
- remote list scope is the resolved session prefix;
- remote-to-local mapping is traversal-safe;
- actions are sorted deterministically by local relative path.
- actions are sorted by local relative path and then remote key;
- force converts differing local targets from conflicts to downloads.
Restore scope from current remote state:
- include `manifest.json`;
- include `transcripts/**`;
- include `artifacts/**`;
- include `audio/**` only with `--include-audio`.
Explicit exclusions from current remote state mapping:
- `current/**`;
- `runs/**`;
- `logs/**`;
- `reports/**`;
- `config/**`;
- `inputs/**`;
- `previous/**`.
Previous-cache files are planned separately through `previouscache.BuildPlan` when configured previous-session requirements exist.
Previous-cache files are planned separately through `previouscache.BuildPlan`
when configured previous-session requirements exist.
## Execution Contract
@@ -73,8 +57,9 @@ Audio restore path:
## Reporting Contract
- `--dry-run`: prints summary only; no local writes.
- non-dry-run: writes `reports/restore-latest.json`.
- dry-run mode prints a summary and performs no local writes;
- execution mode persists the canonical restore report described in
[Operations](../operations.md#restore-workflow);
- report includes plan counts, per-action status, and execution failures.
## Invariants
@@ -82,3 +67,15 @@ Audio restore path:
- restore uses committed remote current state as authority;
- `current/run_id.txt` is the remote publish commit marker;
- restore does not execute pipeline stages.
## Implementation And Tests
- Discovery: `internal/app/restore_discovery.go`
- Planning: `internal/app/restore_plan.go`, `internal/previouscache`
- Execution: `internal/app/restore_execute.go`
- Reporting and command coordination: `internal/app/restore_report.go`,
`internal/app/restore.go`
- Tests: `internal/app/restore_discovery_test.go`,
`internal/app/restore_plan_test.go`,
`internal/app/restore_execution_test.go`,
`internal/app/restore_workflow_test.go`

View File

@@ -1,13 +1,15 @@
# Internal: Manifest
## Purpose
Define durable session state (`manifest.json`) and invocation state (`runs/{run_id}/manifest.json`) contracts.
Explain the session-progress and invocation-audit models implemented by
`internal/manifest`. Physical manifest placement belongs in
[Operations](../operations.md#local-state-layout).
## Session Manifest
Path:
- `{workspace.root}/work/{campaign}/{session_id}/manifest.json`
Primary model (`manifest.Manifest`):
`manifest.Manifest` records:
- identity (`session_id`, `campaign`, `run_id`)
- local path metadata (`local_workdir`, `local_spool_dir`)
- remote identity metadata (`s3_bucket`, `s3_session_prefix`, `s3_run_prefix`)
@@ -15,7 +17,8 @@ Primary model (`manifest.Manifest`):
- durable `artifacts` records
- per-stage `stages` map
Stage status enum:
The model admits these stage states:
- `pending`
- `running`
- `succeeded`
@@ -25,10 +28,9 @@ Stage status enum:
- `interrupted`
## Run Manifest
Path:
- `{workspace.root}/work/{campaign}/{session_id}/runs/{run_id}/manifest.json`
Run model (`manifest.RunManifest`):
`manifest.RunManifest` is created for each invocation and records:
- invocation identity and `force` flag
- requested stages
- per-stage action (`run` or `skip`)
@@ -36,22 +38,63 @@ Run model (`manifest.RunManifest`):
- overall run status (`running`, `succeeded`, `failed`)
## Persistence Semantics
`manifest.LocalStore`:
- validates loaded documents;
- normalizes missing maps/stage records;
- writes atomically via temp file + rename;
- updates `updated_at` on save.
## Execution Semantics
Runner updates both manifests per stage transition:
- mark running
- mark succeeded/failed/skipped
- persist logs/generated config refs and metadata
The application runner marks an executing stage running and then succeeded or
failed in both manifests, persisting each transition. On success it records
outputs, logs, generated configuration references, and metadata. Artifact
records may include optional contract and external provenance objects; old
manifests remain compatible when those fields are absent. A successful forced
rerun marks only succeeded downstream session-stage records stale.
Starting an execution clears the current session-stage record's prior outputs,
logs, generated configuration references, and metadata. Failed and skipped
transitions enforce the same clearing rule directly, while success repopulates
only fields returned by the new result. Marking a record stale does not clear
those details because resume validation and diagnosis may still require them
before execution begins. Invocation run manifests remain immutable audit
records of their own outcomes.
A stage may explicitly return a skipped disposition and stable reason. The
runner persists that outcome in both manifests, clears older outputs for the
session-stage record along with older logs, generated configuration references,
and metadata, then applies any bounded details from the current skip and
continues. This self-skip is distinct from deciding not to execute an
already-succeeded stage and is reconsidered on later runs. Skipped results
cannot contain outputs.
When an already-succeeded stage is skipped, the invocation run manifest records
the `skip` action and reason. The session manifest deliberately retains its
existing succeeded record because it remains the cross-invocation progress
authority. Stages with a resume validator, currently extraction, may reject an
otherwise eligible skip when the recorded durable result is obsolete; the
runner marks it stale and executes it.
Session manifest is the authoritative stage-progress ledger across invocations.
Run manifest is invocation-scoped audit state.
## Invariants
- stage resume/skip decisions are session-manifest driven.
- running, failed, and self-skipped stages do not retain result payloads from
an earlier success.
- stale stages retain prior details until replacement execution starts.
- force reruns stale downstream succeeded stages.
- run manifest does not replace session manifest as progress authority.
## Implementation And Tests
- Models and transitions: `internal/manifest/manifest.go`,
`internal/manifest/run_manifest.go`
- Persistence and validation: `internal/manifest/store.go`
- Package tests: `internal/manifest/*_test.go`
- Assembled execution behavior: `internal/app/runner_test.go`,
`internal/app/run_stage_test.go`

98
docs/internal/overview.md Normal file
View File

@@ -0,0 +1,98 @@
# Internal Overview
This document is the implemented component map for Narratio. Normative system
boundaries and dependency direction belong in
[Architecture](../policy/architecture.md). User and operator contracts belong
in the [CLI](../cli.md), [Configuration](../config.md),
[Operations](../operations.md), and [Troubleshooting](../troubleshooting.md).
Externally observable tool and format contracts belong under
[Integrations](../integrations/).
## Execution Path
```text
cmd/narratio -> internal/app -> configuration and production composition
-> internal/stage -> adapters and external systems
-> manifests and artifact resolution -> durable local/remote output
```
The executable delegates process behavior to the application boundary. The
application resolves configuration, composes concrete collaborators, acquires
session safety controls, and runs commands. Pipeline commands execute the
canonical stage sequence through adapter interfaces, while manifests record
progress and artifact services resolve durable inputs and outputs.
## Components
| Area | Implemented owners | Responsibility |
| --- | --- | --- |
| Executable | `cmd/narratio` | Process entry, standard stream wiring, argument handoff, and exit status. |
| Application orchestration | `internal/app` | Command dispatch, configuration selection, secret-file environment loading, production composition, session locking, planning, execution, restore, cleanup gates, and user-facing reporting. |
| Configuration | `internal/config` | Strict YAML loading, discovery, defaults, normalization, session templating, and validation. |
| Pipeline stages | `internal/stage` | Canonical stage registry, shared stage contract, execution dependencies, and implemented stage behavior. |
| External boundaries | `internal/adapters`, `internal/audio` | WhisperX HTTP, downstream subprocesses, notification, object storage, and S3 audio materialization behind Narratio contracts. |
| Manifests | `internal/manifest` | Durable session progress, invocation audit state, stage transitions, validation, and atomic persistence. |
| Artifacts and paths | `internal/artifacts`, `internal/pathsafe` | Artifact identities and resolution, local and remote path/key models, current-state discovery, and confined relative destinations. |
| Previous-session cache | `internal/previouscache` | Deterministic planning and materialization requirements for configured previous-session inputs. |
| Artifact policy | `internal/artifactpolicy` | Source and destination policy, configured artifact identity validation, and publish destination safety. |
| Shared models and file operations | `internal/artifactmodel`, `internal/contracts`, `internal/fileops` | Transcript and artifact data contracts plus narrow atomic filesystem helpers. |
| Logging | `internal/logging` | Application logger construction and shared structured logging behavior. |
The application boundary composes concrete implementations. Stages depend on
Narratio-level contracts; external transport and SDK details remain in
adapters. The normative rules for these relationships remain in
[Architecture](../policy/architecture.md).
## Pipeline Stage Set
The implemented canonical order is:
1. [`prepare`](stage-prepare.md)
2. [`transcribe`](stage-transcribe.md)
3. [`merge`](stage-merge.md)
4. [`polish`](stage-polish.md)
5. [`normalize`](stage-normalize.md)
6. [`trim`](stage-trim.md)
7. [`extract`](stage-extract.md)
8. [`render`](stage-render.md)
9. [`analyze`](stage-analyze.md)
10. [`publish`](stage-publish.md)
11. `notify` (placeholder)
`notify` currently has optional notifier call behavior and no persisted pipeline
outputs; its default collaborator is a no-op sender. The focused stage
documents own implementation mechanics. The
[CLI](../cli.md) and [Operations](../operations.md) own user-visible invocation
and execution semantics.
## Focused Documentation
- [Adapter Internals](adapters.md): external adapter boundaries, composition,
failure behavior, and test surfaces.
- [Artifact Internals](artifacts.md): source identities, runtime catalog,
resolution, previous requirements, and current-state helpers.
- [Manifest Internals](manifest.md): session and run records, persistence, and
execution transitions.
- [Storage Internals](storage.md): object-store interface and S3 behavior.
- [Workspace Internals](workspace.md): local layout, locking, and cleanup
guardrails.
- [Restore Internals](command-restore.md): discovery, planning, execution, and
reporting.
- [`prepare`](stage-prepare.md)
- [`transcribe`](stage-transcribe.md)
- [`merge`](stage-merge.md)
- [`polish`](stage-polish.md)
- [`normalize`](stage-normalize.md)
- [`trim`](stage-trim.md)
- [`extract`](stage-extract.md)
- [`render`](stage-render.md)
- [`analyze`](stage-analyze.md)
- [`publish`](stage-publish.md)
Use this map to find an owner, then read its focused documentation and tests
before changing behavior.
The stage registry is implemented in `internal/stage/placeholders.go` and its
ordering is protected by `internal/app/planner_test.go`. Cross-invocation skip,
force, failure, and invalidation behavior is exercised in
`internal/app/runner_test.go` and `internal/app/run_stage_test.go`.

View File

@@ -1,39 +1,60 @@
# Stage: analyze
## Purpose
Execute selected configured Scriptorium artifacts in dependency order and materialize outputs.
## Inputs
- configured artifacts from `pipeline.scriptorium.artifacts`
- optional selected artifact filter (`--artifacts`)
- optional selected artifact keys supplied through the stage environment
- built-in/configured/previous-session source references in artifact inputs
Supported source families:
- built-ins: `narratio.transcript.*`, `narratio.bounds.session`
- prepared stable inputs: `narratio.input.players`, `narratio.input.party`,
`narratio.input.glossary`
- configured artifacts: `narratio.artifact.<key>`
- previous-session cache: `narratio.previous_session.artifact.<key>`
## Outputs
- one materialized output per executed configured artifact (`output_path`)
- stage metadata describing selected/generated/reused artifacts
## Key Behavior
- skips with metadata when Scriptorium config is missing or no executable artifacts remain.
- builds runtime artifact catalog (built-ins + configured artifacts).
- marks non-executable configured artifacts as reusable when output files already exist.
- validates selected artifact dependency order (cycle-safe topo ordering).
- resolves required/optional inputs per artifact source definition.
- resolves prepared stable input sources from `inputs/*.yml` materialized by `prepare`.
- resolves previous-session sources from local `previous/` cache only.
- runs optional render-debug, then artifact execution.
- validates non-empty output files and materializes canonical outputs.
## Failure Semantics
- required missing configured/previous-session inputs fail.
- missing required prepared stable input source includes prepare rerun guidance.
- missing required previous-session source includes prepare rerun guidance.
- missing required `narratio.transcript.final_markdown` or `narratio.transcript.final_trimmed_markdown` inputs includes render rerun guidance.
- missing required `narratio.transcript.final_markdown` or
`narratio.transcript.final_trimmed_markdown` inputs includes render rerun
guidance.
- dependency cycles or unavailable required dependencies fail.
- adapter validation failures fail stage.
## Invariants
- `analyze` performs no remote storage calls for previous-session source resolution.
- output provenance and metadata are deterministic per execution.
## Related Contracts And Tests
- [Configuration](../config.md#scriptorium-artifact-entries) owns artifact
fields and source-selection rules.
- [CLI](../cli.md) owns user-visible artifact selection.
- [Scriptorium](../integrations/scriptorium.md) owns the subprocess contract.
- Implementation and tests: `internal/stage/analyze.go`,
`internal/stage/analyze_test.go`

View File

@@ -0,0 +1,84 @@
# Internal: Extract Stage
## Responsibility
`extract` runs after `trim` and before `render`. It converts the canonical
`narratio.transcript.final_trimmed` JSON into configured Notarius lane artifacts.
An omitted or disabled Notarius section makes the stage explicitly self-skip
with reason `notarius_disabled`, no outputs, and no Notarius runner.
The external protocol is documented in the
[Notarius integration contract](../integrations/notarius.md). Configuration
fields belong in [Configuration](../config.md), and physical paths and force
procedures belong in [Operations](../operations.md).
## Lifecycle
`internal/stage/extract.go`:
1. resolves the final trimmed transcript from the shared artifact catalog;
2. resolves and fingerprints the Notarius invocation contract;
3. creates a run-local staging directory and invokes the injected
`notarius.Runner`;
4. validates the successful receipt, confined index, configured required lane
descriptors, and regular payload files;
5. atomically promotes the complete bundle to its immutable durable location;
6. records one non-selectable `notarius_index` output and one selectable
`notarius_lane` output per configured lane; and
7. registers each lane as `narratio.extraction.<output_key>` for downstream
Scriptorium and publish resolution.
Lane records retain checksum, contract, producer run ID, and Notarius system,
run, pipeline, and lane provenance. Stage metadata retains the durable bundle
root, receipt, diagnostic paths, rejection/warning summaries, producing
Narratio run ID, and invocation fingerprint. Validation completes before
promotion, so a rejected result cannot expose a partial durable bundle.
Any executed extraction outcome that replaces a different effective outcome
marks succeeded downstream stages stale. Repeating the same disabled self-skip
with no outputs is stable and does not repeatedly invalidate downstream stages.
## Resume Validation
`internal/stage/extract_resume.go` permits a skip only when the existing stage
record succeeded and still matches the current invocation fingerprint. The
fingerprint covers the resolved executable and config paths, pipeline ID,
timeout, working directory, and sorted configured output contracts.
The validator then checks the producing run identity, canonical immutable
bundle root, path confinement and absence of symlink components, receipt
identity, exactly one canonical index, the exact configured source set,
contracts and provenance, regular-file status, and stored checksums. Missing or
obsolete results are non-resumable and run again; unsafe filesystem conditions
return an error rather than silently accepting or replacing data.
The fingerprint cannot observe files imported by Notarius configuration,
profile contents, prompt/module definitions, or other transitive inputs.
Operators must force extraction after changing any such input.
## Failure Behavior
Adapter startup, timeout, nonzero exit, receipt decoding, path confinement,
index compatibility, required-lane rejection, payload inspection, checksum, or
promotion errors fail the stage through ordinary manifest transition handling.
Stdout receipt and stderr diagnostics remain separate. Downstream stages are
not given selectable extraction sources unless the complete configured result
has passed validation and promotion.
When a replacement attempt begins, the current session-stage record no longer
advertises payload from the previous success. A failed replacement therefore
has no current outputs, logs, generated configuration references, or metadata,
while the earlier invocation manifest and immutable promoted bundle remain
available for audit and recovery.
## Implementation And Focused Tests
- Stage execution, selection, and resume validation: `internal/stage/extract.go`,
`internal/stage/extract_resume.go`,
`internal/stage/extract_test.go`
- Subprocess boundary: `internal/adapters/notarius/subprocess.go`,
`internal/adapters/notarius/subprocess_test.go`
- Catalog hydration: `internal/artifacts/extraction_catalog.go`,
`internal/artifacts/extraction_catalog_test.go`
- Composition and downstream behavior: `internal/app/runner_test.go`,
`internal/stage/analyze_test.go`, `internal/stage/publish_test.go`

View File

@@ -1,18 +1,22 @@
# Stage: merge
## Purpose
Normalize raw transcript inputs and merge into base transcript via Seriatim.
## Inputs
- `transcripts/raw/*.json`
- `inputs/speakers.yml`
- `inputs/autocorrect.yml`
## Outputs
- `transcripts/base.json`
- optional `artifacts/seriatim.report.json`
## Key Behavior
- discovers and validates raw transcript inputs.
- normalizes each raw transcript (`seriatim.Normalize`) into run-local scratch output.
- merges normalized inputs (`seriatim.Run`) into base transcript.
@@ -20,6 +24,14 @@ Normalize raw transcript inputs and merge into base transcript via Seriatim.
- materializes canonical outputs and records stage logs/generated configs.
## Invariants
- merge always consumes normalized forms of raw inputs.
- base transcript must validate before stage success.
- report output is config-gated.
## Related Contracts And Tests
- [Seriatim](../integrations/seriatim.md) owns subprocess and output semantics.
- [Configuration](../config.md#pipeline) owns operator-selected Seriatim values.
- Implementation and tests: `internal/stage/merge.go`,
`internal/stage/merge_test.go`

View File

@@ -1,16 +1,20 @@
# Stage: normalize
## Purpose
Normalize polished transcript into final transcript using Seriatim.
## Inputs
- `transcripts/polished.json`
## Outputs
- `transcripts/final.json` (or configured normalize output path)
- optional `artifacts/seriatim.normalize.report.json`
## Key Behavior
- resolves polished transcript from manifest outputs/canonical fallback.
- applies `pipeline.normalize` config or default normalize config.
- runs Seriatim normalize with configured timeout/binary.
@@ -18,5 +22,13 @@ Normalize polished transcript into final transcript using Seriatim.
- materializes canonical outputs and records logs/generated configs.
## Invariants
- final transcript must validate as processed transcript JSON (`segments` array).
- normalize defaults are applied when `pipeline.normalize` is unset.
## Related Contracts And Tests
- [Seriatim](../integrations/seriatim.md) owns subprocess and output semantics.
- [Configuration](../config.md#pipeline) owns normalize fields and defaults.
- Implementation and tests: `internal/stage/normalize.go`,
`internal/stage/normalize_test.go`

View File

@@ -1,17 +1,21 @@
# Stage: polish
## Purpose
Run Audita polishing on base transcript and produce polished transcript.
## Inputs
- `transcripts/base.json`
- `inputs/glossary.yml`
## Outputs
- `transcripts/polished.json`
- optional `artifacts/audita.report.json`
## Key Behavior
- resolves base transcript from merge outputs/canonical fallback.
- invokes Audita with configured model/module/runtime options.
- validates processed transcript structure (`segments` array required).
@@ -19,5 +23,14 @@ Run Audita polishing on base transcript and produce polished transcript.
- materializes canonical outputs; records logs/generated config and adapter metadata.
## Invariants
- polished transcript schema validation is mandatory.
- report output is config-gated.
## Related Contracts And Tests
- [Audita](../integrations/audita.md) owns subprocess, validation, and failure
semantics.
- [Configuration](../config.md#pipeline) owns operator-selected Audita values.
- Implementation and tests: `internal/stage/polish.go`,
`internal/stage/polish_test.go`

View File

@@ -1,29 +1,33 @@
# Stage: prepare
## Purpose
Materialize canonical current-session inputs before processing stages.
## Inputs
- resolved `campaign.yml`, `session.yml`, and pipeline config
- stable input files (`speakers`, `autocorrect`, `glossary`)
- audio source:
- local `audio_dir`/`audio_files`, or
- S3 `audio_s3.prefix`
- resolved campaign, session, and pipeline configuration
- stable input files (`speakers`, `autocorrect`, `glossary`, `players`, `party`)
- one resolved local or S3 audio source
- enabled configured artifact input requirements for previous-session sources
## Outputs
- `inputs/campaign.yml`
- `inputs/session.yml`
- `inputs/pipeline.resolved.yml`
- `inputs/speakers.yml`
- `inputs/autocorrect.yml`
- `inputs/glossary.yml`
- `inputs/players.yml`
- `inputs/party.yml`
- `audio/*.flac`
- optional `previous/manifest.json`
- optional `previous/artifacts/**`
- deterministic `manifest.inputs` entries (checksums + provenance)
## Key Behavior
- validates required config/store state.
- enforces local audio vs S3 audio mutual exclusivity.
- materializes S3 audio through spool/cache-aware logic.
@@ -37,6 +41,20 @@ Materialize canonical current-session inputs before processing stages.
Required previous-session inputs fail when unavailable; optional missing inputs are skipped.
## Invariants
- only `prepare` hydrates canonical `previous/` cache state.
- managed previous artifacts are stored under `previous/artifacts/**` without duplicate `artifacts/artifacts/` nesting.
- managed previous artifacts are stored under `previous/artifacts/**` without
duplicate `artifacts/artifacts/` nesting.
- `manifest.inputs` ordering is deterministic (`kind`, `path`).
## Related Contracts And Tests
- [Configuration](../config.md) owns audio selection, stable input fields, and
previous-session settings.
- [Operations](../operations.md) owns physical input, audio, spool, cache, and
previous-state layout.
- [Storage Internals](storage.md) and [Artifact Internals](artifacts.md) explain
the internal collaborators.
- Implementation and tests: `internal/stage/prepare.go`,
`internal/stage/prepare_test.go`, `internal/audio/s3_audio_test.go`,
`internal/previouscache/*_test.go`

View File

@@ -1,33 +1,45 @@
# Stage: publish
## Purpose
Upload run/session outputs to object storage and atomically advance remote current state.
## Inputs
- successful prerequisite stages: `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `render`, `analyze`
- run root `runs/{run_id}/**`
- publish output rules (`pipeline.publish.outputs`)
- successful preceding stages from the [canonical stage set](overview.md#pipeline-stage-set)
- invocation-scoped run files
- resolved publish output rules
- effective publish locks (static + remote merged lock set)
- local `previous/**` files when present
- durable previous-session cache files when present
## Outputs
- uploaded run files under remote `runs/{run_id}/...` (excluding `audio/**`)
- uploaded selected publish outputs under session prefix
- uploaded `previous/**` files under session prefix when present
- uploaded `current/manifest.json`
- uploaded `current/run_id.txt` written last
- uploaded invocation record and selected publish outputs;
- uploaded durable previous-session cache files when present;
- updated remote current manifest; and
- remote current-run commit marker, written last.
Exact remote placement and the operator workflow belong in
[Operations](../operations.md#publish-workflow).
## Key Behavior
- stage can self-skip when publish disabled or run upload disabled.
- validates prerequisite stage success and object-store availability.
- collects deterministic run file list plus run `manifest.json`.
- collects a deterministic run file list plus run `manifest.json`, excluding
`audio/**` and the run-local `extract/notarius-output/**` staging bundle.
- keeps run-local Notarius receipt and stderr diagnostics eligible for the run
archive.
- resolves publish output sources through runtime artifact catalog and manifest-aware resolution.
- publishes extraction lanes only through explicit configured output rules;
neither run-local nor durable Notarius bundles are scanned or uploaded wholesale.
- selected artifact filter applies to configured artifact sources only.
- locked outputs are skipped intentionally (including required ones).
- optional missing outputs are skipped; required missing unlocked outputs fail.
- writes remote current manifest before current run pointer.
## Metadata Signals
Includes counts/lists for:
- run uploads
- published output uploads
@@ -39,6 +51,24 @@ Includes counts/lists for:
- `current_pointer_written`
## Invariants
- `current/run_id.txt` is the remote commit marker and is written last.
- run upload excludes `audio/**`.
- run upload excludes `audio/**` and `extract/notarius-output/**`.
- `extract/notarius.receipt.json` and `extract/notarius.stderr.log` remain
eligible run-record diagnostics.
- publish locks are not overridden by `--force`.
The commit boundary and cleanup gate are normative architecture invariants; see
[Architecture](../policy/architecture.md#publish-commit-boundary).
## Related Contracts And Tests
- [Configuration](../config.md#publish-configuration-summary) owns output and
static-lock fields.
- [Operations](../operations.md#publish-locks) owns remote lock lifecycle and
physical remote state.
- [Artifact Internals](artifacts.md) explains source resolution and current-state
helpers.
- Implementation and tests: `internal/stage/publish.go`,
`internal/stage/publish_test.go`, `internal/app/operator_helpers_test.go`,
`internal/app/post_publish_cleanup_test.go`

View File

@@ -1,17 +1,21 @@
# Stage: render
## Purpose
Render Markdown transcript artifacts from normalized JSON transcripts via Seriatim.
## Inputs
- `narratio.transcript.final` (`transcripts/final.json`)
- `narratio.transcript.final_trimmed` (`transcripts/final.trimmed.json`)
## Outputs
- `narratio.transcript.final_markdown` -> `transcripts/final.md`
- `narratio.transcript.final_trimmed_markdown` -> `transcripts/final.trimmed.md`
## Key Behavior
- uses `pipeline.render` settings (enabled/format/title/booleans).
- resolves inputs manifest-first, then canonical fallback.
- writes run-local outputs first, then materializes canonical session outputs.
@@ -19,11 +23,20 @@ Render Markdown transcript artifacts from normalized JSON transcripts via Seriat
- skips with stage metadata when `pipeline.render.enabled=false`.
## Failure Semantics
- missing normalized input fails with normalize rerun guidance.
- missing trimmed input fails with trim rerun guidance.
- adapter/subprocess failure fails stage.
- empty render output files fail validation.
## Invariants
- only `format: markdown` is supported.
- render stage owns production of built-in Markdown transcript sources.
## Related Contracts And Tests
- [Seriatim](../integrations/seriatim.md) owns render subprocess behavior.
- [Configuration](../config.md#pipeline) owns render fields and defaults.
- Implementation and tests: `internal/stage/render.go`,
`internal/stage/render_test.go`

View File

@@ -1,22 +1,36 @@
# Stage: transcribe
## Purpose
Generate raw per-speaker transcripts from prepared audio using WhisperX.
## Inputs
- `audio/*.flac` from `prepare`
## Outputs
- `transcripts/raw/<speaker>.json`
## Key Behavior
- discovers prepared audio from manifest inputs or canonical audio directory.
- derives speaker ID from `.flac` basename.
- runs WhisperX with configured concurrency/retry settings.
- dispatches WhisperX requests through a bounded worker pool.
- validates each output as JSON.
- writes run-local outputs then materializes canonical transcript outputs.
## Invariants
- speaker basenames must be unique.
- output path returned by adapter must match requested output path.
- each successful output is validated before stage success.
## Related Contracts And Tests
- [WhisperX](../integrations/whisperx.md) owns HTTP, retry, timeout, and
cancellation semantics.
- [Configuration](../config.md#pipeline) owns concurrency and other
operator-selected values.
- Implementation and tests: `internal/stage/transcribe.go`,
`internal/stage/transcribe_test.go`

View File

@@ -1,18 +1,20 @@
# Stage: trim
## Purpose
Produce a final-trimmed transcript; optionally generate bounds-driven trim.
Produce a final-trimmed transcript. By default, the stage generates bounds and
applies a bounds-driven trim.
## Inputs
- `transcripts/final.json`
## Outputs
- `transcripts/final.trimmed.json` (or configured trim output path)
- when trim enabled: `artifacts/session_bounds.json`
## Key Behavior
When `trim.enabled=false`:
- copies normalized transcript to trimmed output.
When `trim.enabled=true`:
- runs Scriptorium bounds artifact generation;
@@ -22,7 +24,20 @@ When `trim.enabled=true`:
- either copies unchanged transcript or runs Seriatim trim;
- validates trimmed transcript and materializes bounds output.
When `trim.enabled=false`:
- copies normalized transcript to trimmed output.
## Invariants
- normalized transcript is required input.
- bounds output exists only in enabled trim path.
- render-debug output is diagnostic and not a declared stage output.
## Related Contracts And Tests
- [Scriptorium](../integrations/scriptorium.md) owns bounds generation and
debug-render subprocess behavior.
- [Seriatim](../integrations/seriatim.md) owns transcript trimming behavior.
- [Configuration](../config.md#pipeline) owns trim fields and defaults.
- Implementation and tests: `internal/stage/trim.go`,
`internal/stage/trim_test.go`

View File

@@ -1,10 +1,16 @@
# Internal: Storage
## Purpose
Document remote object-store contracts and S3 implementation behavior.
Explain the object-store interface and S3 implementation used by Narratio.
Remote key layout and lifecycle belong in [Operations](../operations.md), while
operator-selected storage fields and credential mechanisms belong in
[Configuration](../config.md).
## Primary Contract
`storage.ObjectStore` interface:
- `List(ctx, prefix)`
- `Download(ctx, key, localPath)`
- `Upload(ctx, localPath, key, opts)`
@@ -14,15 +20,15 @@ Key invariant:
- callers pass full bucket-relative keys;
- storage implementations do not infer campaign/session/run prefixes.
## Configuration
`NewObjectStoreFromConfig` currently supports S3-backed stores from `pipeline.storage.*` config.
## Composition
S3 constructor behavior:
- requires configured bucket;
- uses region/endpoint/path-style options when set;
- resolves credentials from configured env var names (with defaults).
`NewObjectStoreFromConfig` constructs the S3-backed implementation from
resolved configuration. The application loads configured filesystem secrets
before calling it. The storage adapter consumes already-resolved values; it does
not own discovery, defaults, or configuration validation.
## S3 Backend Behavior
- normalizes object keys.
- `List` paginates and returns normalized `ObjectInfo`.
- `Download` writes local files with parent directory creation.
@@ -30,5 +36,13 @@ S3 constructor behavior:
- `Exists` maps not-found responses to `false`.
## Invariants
- storage layer is stateless regarding manifest/stage progression.
- publish ordering semantics are owned by stage/app code, not storage adapters.
## Implementation And Tests
- Contract and S3 adapter: `internal/adapters/storage`
- Composition: `internal/app/object_store.go`
- Tests: `internal/adapters/storage/*_test.go`,
`internal/app/object_store_test.go`

View File

@@ -1,57 +1,74 @@
# Internal: Workspace
## Purpose
Define local session layout, run-local stage layout, and cleanup guardrails.
## Canonical Session Layout
Session root:
- `{workspace.root}/work/{campaign}/{session_id}`
Explain the helpers that construct local session and run paths, coordinate
single-writer access, and confine cleanup. The authoritative physical layout and
retention workflow belong in [Operations](../operations.md#local-state-layout).
Core directories/files:
- `inputs/`
- `audio/`
- `transcripts/`
- `artifacts/`
- `reports/`
- `logs/`
- `config/`
- `current/`
- `runs/`
- `previous/`
- `manifest.json`
- `.lock`
## Path Ownership
`previous/` reserved files:
- `previous/manifest.json`
- `previous/artifacts/**`
`internal/artifacts` owns canonical session, run, spool, cache, and
previous-cache path construction. `SessionPathsFor` provides the session-scoped
path model, and layout creation goes through `EnsureLayoutFor`. Callers should
consume those helpers instead of rebuilding relative paths.
`internal/pathsafe` and application cleanup helpers enforce confinement for
relative destinations and deletion targets.
## Run-Local Stage Layout
When run context is available, stages use:
- `runs/{run_id}/{stage}/outputs/`
- `runs/{run_id}/{stage}/logs/`
- `runs/{run_id}/{stage}/reports/`
- `runs/{run_id}/{stage}/config/`
- `runs/{run_id}/{stage}/scratch/`
Run-local outputs are materialized back into canonical session paths before stage success.
`previous/**` writes are never redirected to run-local output paths.
`internal/stage/run_local.go` maps stage outputs and diagnostics into an
invocation-scoped layout. Successful outputs are validated and atomically
materialized into canonical session paths before stage success. Managed
previous-session cache paths remain session-durable and are never redirected
into run-local output space.
Extraction uses run-local receipt, stderr, and output-root helpers, then
promotes the validated external bundle to the unique immutable Notarius bundle
path supplied by `internal/artifacts`. `internal/fileops.PromoteDirectory`
copies only regular files and directories to a same-filesystem temporary
sibling. Source traversal uses confined directory handles and identity checks
so replacing an inspected root, directory, or file is rejected rather than
followed. The completed tree is atomically renamed without replacing an
existing destination. Exact physical paths belong in
[Operations](../operations.md#extraction-workflow).
## Locking
`artifacts.LocalStore` enforces single-writer session lock via `.lock` file (`ErrLockConflict` on contention).
`artifacts.LocalStore` enforces the single-writer session lock via `.lock`
(`ErrLockConflict` on contention).
## Cleanup Semantics
Automatic post-publish cleanup:
- only runs when publish actually executed and succeeded;
- requires `uploaded=true` and `current_pointer_written=true` metadata;
- respects `pipeline.spool.delete_audio_after_publish` and `pipeline.workspace.cleanup_after_publish`;
- consumes the resolved cleanup policy described in
[Configuration](../config.md);
- refuses unsafe deletes (root delete, out-of-root delete, symlink paths).
Manual clean command:
- `clean <session_id>` removes session work and spool subtree.
- `clean --all` removes all workspace work and spool children.
- durable cache is preserved unless `--clear-cache` is requested.
Manual cleanup uses the same scoped-target checks. Invocation syntax and exact
deletion scope belong in [CLI](../cli.md#clean) and
[Operations](../operations.md#cleanup).
## Invariants
- campaign-aware session root is mandatory.
- manifest-driven stage state is durable across runs.
- cleanup guardrails prevent destructive root/out-of-scope deletion.
## Implementation And Tests
- Path model and local store: `internal/artifacts/paths.go`,
`internal/artifacts/local.go`
- Run-local materialization: `internal/stage/run_local.go`
- Immutable bundle promotion: `internal/fileops/directory.go`
- Cleanup confinement: `internal/app/cleanup_targets.go`,
`internal/app/post_publish_cleanup.go`
- Tests: `internal/artifacts/paths_model_test.go`,
`internal/artifacts/local_test.go`, `internal/stage/run_local_test.go`,
`internal/fileops/directory_test.go`,
`internal/app/cleanup_targets_test.go`,
`internal/app/post_publish_cleanup_test.go`

View File

@@ -36,6 +36,8 @@ narratio session init 2026-04-04 --remote --force
If `campaign.yml` sets `session_template_file`, `session init` renders it. Template variables must resolve to concrete values.
Campaigns must provide stable input files for speakers, autocorrect, glossary, players, and party. Session files may override those paths for one session. The `prepare` stage materializes them under `inputs/`; configured Scriptorium artifacts can reference prepared `players`, `party`, and `glossary` files with `narratio.input.players`, `narratio.input.party`, and `narratio.input.glossary`.
## Standard Session Workflow
1. Select pipeline/campaign/session config.
@@ -73,16 +75,22 @@ Canonical stage order:
4. `polish`
5. `normalize`
6. `trim`
7. `render`
8. `analyze`
9. `publish`
10. `notify`
7. `extract`
8. `render`
9. `analyze`
10. `publish`
11. `notify`
Execution rules:
- succeeded stages are skipped unless `--force` is set;
- `run` continues interrupted or partially completed sessions by running non-succeeded stages;
- force rerunning a succeeded upstream stage marks succeeded downstream stages as `stale`.
- forcing an upstream stage marks succeeded downstream stages as `stale` before
the replacement runs; and
- an executed failure, changed self-skip, or success that replaces a different
effective upstream outcome also marks succeeded downstream stages stale. A
repeated self-skip with the same reason and no outputs is stable and does not
perpetually rerun downstream work.
Single-stage execution:
@@ -99,7 +107,64 @@ Selection behavior:
- validates names against `pipeline.scriptorium.artifacts`;
- filters analyze execution to selected configured artifacts;
- filters publish rules for `narratio.artifact.<name>` sources only;
- does not suppress built-in transcript or bounds publish sources.
- does not suppress built-in transcript, bounds, or explicitly configured
`narratio.extraction.<name>` publish sources; and
- never partially selects Notarius lanes.
## Extraction Workflow
When Notarius is omitted or disabled, `extract` records an explicit skipped
outcome with reason `notarius_disabled` and no outputs. A later invocation
reconsiders the skipped stage, so enabling Notarius does not require force.
When Notarius extraction is enabled, the stage consumes the final trimmed JSON
and preserves the complete validated Notarius bundle at:
- `artifacts/notarius/{narratio_run_id}/`
The directory is immutable once promoted. Configured lanes become
`narratio.extraction.<name>` sources for Scriptorium and explicit publish rules;
the bundle and `index.json` are retained for audit and resume validation but
are not selectable or published implicitly.
Starting a replacement clears the previous extraction payload from the current
session-stage record. If that replacement fails or self-skips, the current
record does not fall back to the earlier outputs. The earlier run manifest and
immutable bundle remain available for inspection, but downstream resolution
requires a new current successful extraction record.
Atomic Notarius bundle promotion is supported on Linux, macOS, and Windows.
On other operating systems, extraction fails before copying the bundle into a
temporary promotion tree because Narratio has no verified atomic no-replace
directory primitive there. This is an extraction limitation, not a broader
platform-support guarantee for every Narratio workflow.
Run-local diagnostics are:
- `runs/{run_id}/extract/notarius.receipt.json`
- `runs/{run_id}/extract/notarius.stderr.log`
- `runs/{run_id}/extract/notarius-output/` before durable promotion
The run-record upload excludes the complete
`extract/notarius-output/**` subtree. The receipt and stderr files remain
eligible run-record diagnostics. The durable bundle is never scanned for
implicit publication; only lanes named by explicit `pipeline.publish.outputs`
rules are uploaded.
To intentionally replace the current extraction result, run:
```bash
narratio run-stage extract 2026-04-04 --force
```
Narratio automatically reruns extraction when its recorded invocation contract
or durable output validation changes. It cannot fingerprint configuration
files, profiles, prompts, modules, or references loaded transitively by
Notarius. Force extraction after changing any of those inputs, even when the
top-level Narratio and Notarius config paths remain the same. A forced extract
marks successful downstream stages stale. Ordinary extraction failures or
outcome changes also stale affected downstream stages, while an identical
repeated `notarius_disabled` self-skip does not repeatedly invalidate them.
## Publish Workflow
@@ -117,8 +182,10 @@ narratio run-stage publish 2026-04-04 --force
Publish commit model:
- uploads run files under `{session_prefix}/runs/{run_id}/`;
- uploads configured published outputs;
- uploads eligible run files under `{session_prefix}/runs/{run_id}/`, excluding
audio and the run-local Notarius staging bundle;
- uploads configured published outputs, including only explicitly configured
extraction lanes;
- uploads `previous/**` cache files when present;
- writes `current/manifest.json`;
- writes `current/run_id.txt` last.
@@ -196,6 +263,10 @@ Durable session paths:
- `config/**`
- `runs/**`
Validated Notarius bundles live below `artifacts/notarius/{run_id}/`; receipt,
stderr, and pre-promotion output remain in the producing run's `extract`
directory as described in [Extraction Workflow](#extraction-workflow).
Run-local layout:
- `runs/{run_id}/{stage}/outputs`

View File

@@ -1,202 +1,241 @@
# Narratio Architecture
# Architecture
## Purpose
This document defines Narratio's intended high-level architecture and the
invariants that changes must preserve. Implemented component details belong in
the [Internal Overview](../internal/overview.md) and its linked documents.
Significant architectural decision history belongs under `docs/adr/` when such
records exist.
`narratio` is a Go orchestration application for processing D&D session audio into polished transcripts and generated session artifacts.
## System Shape
This document defines the development principles for the project. It is inward-facing: its audience is developers and LLM coding agents. It should guide future changes, not serve as a complete implementation reference.
Narratio is a small Go application that turns D&D session audio into polished
transcripts and generated session artifacts. It is an explicit, stage-driven
orchestrator, not a general workflow engine.
Implemented component details belong under `docs/internal/`.
Narratio coordinates specialized external systems rather than reimplementing
their domains:
## Project Shape
- WhisperX performs transcription;
- Seriatim performs deterministic transcript processing and rendering;
- Audita performs transcript correction and polishing;
- Notarius extracts validated structured artifact bundles; and
- Scriptorium executes prompts and produces configured artifacts.
Narratio is a modular, stage-driven orchestrator.
Narratio owns orchestration, configuration resolution, session and run state,
artifact and path modeling, manifest persistence, stage sequencing, resume,
restore, cleanup gates, and publish semantics. External contracts are defined
in the [integration documentation](../integrations/).
It coordinates specialized downstream systems rather than reimplementing their domains:
The pipeline has one canonical ordered stage set. Configuration may enable,
disable, or parameterize supported behavior, but it must not turn that sequence
into an arbitrary DAG or hide orchestration in generic workflow abstractions.
The implemented stage inventory belongs in the
[Internal Overview](../internal/overview.md).
- WhisperX handles transcription.
- Seriatim handles deterministic transcript merge/normalization/trim behavior.
- Audita handles transcript correction and polishing.
- Scriptorium handles prompt execution and generated artifacts.
Narratio is contract-first without being abstraction-heavy. Interfaces and
extension points should protect demonstrated boundaries. New abstraction is not
itself an architectural goal.
Narratio owns orchestration, configuration loading, session/run state, local and remote path modeling, manifest persistence, stage sequencing, resume behavior, and publish semantics.
## Ownership And Dependency Direction
Narratio should remain explicit and comprehensible. It is not intended to become a generic workflow engine.
The application boundary owns command dispatch, configuration selection,
production composition, session locking, and top-level lifecycle. It may depend
on concrete implementations to assemble a run.
## Core Principles
Stage orchestration expresses intent in Narratio-level data and interfaces.
Stages may depend on configuration, manifest, artifact, path, and adapter
contracts, but they must not depend on transport-specific request types,
subprocess argument construction, cloud SDK types, or downstream tool internals.
### Modular and composable
Adapters translate between Narratio contracts and external systems. They own
HTTP, subprocess, notification, and object-storage mechanics, including command
construction, transport behavior, provider response handling, and external
error adaptation. External dependency types must remain inside the adapter that
owns them unless that dependency is the adapter's explicit public contract.
WhisperX HTTP behavior, Seriatim, Audita, Notarius, and Scriptorium command
construction, notification transport, and object-storage SDK details remain
behind these boundaries.
Code should be organized around clear responsibilities. Stages, adapters, config loading, manifest persistence, path construction, and storage behavior should remain separable and independently testable.
State and path services must not infer stage policy. Storage implementations
receive explicit bucket-relative keys and do not infer campaign, session, run,
or root-prefix semantics. Manifest persistence records transitions but does not
choose orchestration policy. Artifact resolution identifies and validates
artifacts but does not execute producers.
### Hexagonal boundaries
Dependencies should remain narrow and point toward Narratio-owned contracts.
Prefer the Go standard library. Add an external dependency only when it provides
a clear correctness, security, interoperability, or complexity benefit, and
confine it to the boundary that needs it.
External systems should be isolated behind narrow adapters. Stage logic should depend on Narratio-level interfaces and data structures, not on external SDK types, subprocess argument construction, or transport-specific details.
## Stage Boundaries
### Standard library preference
Each stage has one explicit responsibility and declares:
Prefer the Go standard library. Add dependencies only when they provide substantial value, are necessary for an external integration, or are a widely used de facto standard.
Accepted examples include a YAML library for configuration and the AWS SDK for S3-compatible storage.
### Explicit orchestration
The pipeline should remain stage-driven and explicit. New behavior should be added through clear stage, adapter, config, or manifest contracts rather than implicit side effects or generic workflow abstraction.
## Stage Design
Each stage should have a clear scope of responsibility.
A stage should define:
- its purpose;
- required input state;
- produced output state;
- config fields it consumes;
- configuration it consumes;
- external adapters it uses;
- manifest refs it reads or writes;
- skip, force, and resume behavior;
- failure behavior;
- tests that protect its contract.
- manifest references and metadata it reads or writes;
- skip, force, invalidation, and resume behavior; and
- failure behavior.
Stages should avoid reaching across boundaries. If shared behavior is needed, prefer a helper or service with a narrow interface over duplicating ad hoc logic between stages.
Stages write and validate run-local results before materializing canonical
outputs where that distinction applies. A stage is complete only after its
required outputs have been written, validated, and recorded in durable manifest
state. Later stages depend on recorded success and artifact resolution, not
merely on incidental files existing on disk.
## Transactionality and Resume
A failed or interrupted stage must not be presented as successful. Failure
should preserve enough local state and diagnostics for inspection, recovery,
and resume. Forcing an upstream stage invalidates succeeded downstream work
according to the canonical stage order.
A stage should behave transactionally.
A stage may explicitly self-skip with a stable reason and no outputs. That
outcome is persisted, clears older outputs owned by the stage, and is
reconsidered on a later invocation. A stage may also validate whether an
otherwise successful recorded result is still resumable; an obsolete result
is staled and rerun, while an unsafe condition that prevents a sound decision
stops execution.
A stage is complete only when its outputs have been written, validated, and recorded in the manifest. If a stage fails, Narratio should preserve enough local state for inspection, recovery, and resume.
Shared behavior should live behind a narrow service or helper with one clear
owner. Stages must not reach across boundaries or reproduce adapter, manifest,
artifact, or path policy ad hoc.
A failed or incomplete run must not be treated as successful. Later stages should depend on manifest-recorded success, not merely on incidental files existing on disk.
## Manifest, Resume, And Restore
## Manifest Model
The session manifest is the durable ledger for progress across invocations. It
records session and run identity, stage state, input and output references,
diagnostic references, checksums or provenance where useful, and non-secret
adapter and publish metadata.
The manifest is the durable local ledger for a run.
Resume and skip decisions are manifest-driven. Filesystem state may be
inspected and validated, but file presence alone does not replace recorded
stage state. Invocation-scoped run records provide an audit of one execution;
they do not replace the session manifest as progress authority.
It should record:
Restore treats committed remote current state as its authority. It must plan
deterministically, confine remote-to-local paths, protect local conflicts, and
install the validated session manifest after other restored durable files. The
physical workflow and recovery procedures belong in
[Operations](../operations.md).
- run identity;
- stage status;
- input and output refs;
- logs and generated config refs;
- checksums or provenance where useful;
- non-secret adapter and publish metadata.
## Configuration
Resume behavior should be manifest-driven. Filesystem state may be inspected and validated, but it should not replace manifest stage state as the source of run progress.
Configuration is strict, explicit, centralized, and operator-oriented.
## Adapter Boundaries
- YAML decoding rejects unknown fields.
- Defaults are centralized and testable.
- Empty configured values do not silently replace meaningful defaults.
- Validation rejects invalid composition before stage execution where
practical.
- Session templating remains narrow and deterministic rather than becoming a
general configuration language.
- Secret values are supplied indirectly and are not persisted in ordinary
configuration.
Adapters own external integration details.
Narratio must not become a second configuration system for downstream tools.
External systems own their runtime defaults wherever practical; Narratio passes
the paths required by its stage contracts and explicit operator overrides. The
field-level contract and credential-supply mechanisms belong in
[Configuration](../config.md).
Expected boundaries:
## Artifacts, Paths, And Storage
- WhisperX HTTP details stay in the WhisperX adapter.
- Seriatim CLI construction stays in the Seriatim adapter.
- Audita CLI construction stays in the Audita adapter.
- Scriptorium CLI construction stays in the Scriptorium adapter.
- Object-storage details stay behind the storage adapter interface.
- AWS SDK types stay inside the S3 storage implementation.
Artifact identities and local and remote paths are application contracts.
Canonical helpers own workspace, spool, cache, session, run, input, transcript,
artifact, log, report, configuration, and publish-current paths. Callers must
not reconstruct canonical paths through scattered string concatenation.
Stage code should express intent in Narratio terms and call adapters through narrow contracts.
Artifact resolution is deterministic and manifest-aware. Producers materialize
canonical outputs before reporting success, and consumers resolve declared
artifact identities rather than infer files from unrelated directory contents.
External artifact bundles become current only through validated immutable
promotion and manifest records; directory presence alone never establishes
availability.
## Configuration Philosophy
Writes, moves, replacements, and deletions must use narrow, explicit,
root-confined destinations. Symlinks, traversal, broad roots, and ambiguous
relative destinations must not expand the scope of an operation. Cleanup is
permitted only through explicit operator action or configured post-publish
gates, and it must preserve durable cache unless cache removal is explicitly
requested.
Configuration should be strict, explicit, and operator-friendly.
Physical layout, retention, and operational lifecycle belong in
[Operations](../operations.md). Logical external formats and durable integration
contracts belong under [Integrations](../integrations/).
Principles:
## Publish Commit Boundary
- YAML decoding should reject unknown fields.
- Defaults should be centralized and testable.
- Empty configured values should not silently override meaningful defaults.
- Session templating should remain narrow and deterministic.
- Template support should serve operator convenience, not become a general configuration language.
Publish has one explicit remote commit boundary. A remote run becomes current
only after Narratio has successfully uploaded the run record, required published
outputs, `current/manifest.json`, and finally `current/run_id.txt`.
Narratio should not become a secondary configuration system for downstream tools. Seriatim, Audita, and Scriptorium should own their runtime defaults wherever practical. Narratio should pass required stage-contract paths and explicit operator overrides.
`current/run_id.txt` is the commit marker and must be written last. Failed,
incomplete, skipped, or uncommitted publish attempts must not be presented as
current remote state. Publish locks remain authoritative and are not bypassed by
a forced run.
## Path and Storage Discipline
Automatic local cleanup is permitted only after a successful publish commit,
only when explicitly configured, and only through the path-safety guardrails.
Local and remote paths are part of Narratios application contract.
## Security, Privacy, And Diagnostics
Code should use centralized path helpers for workspace, spool, session, run, artifact, log, config, and publish/current paths. Stages should avoid reconstructing canonical paths through scattered string concatenation.
Narratio handles private campaign material. Transcripts, prompts, generated
artifacts, reports, logs, manifests, and diagnostic files are potentially
sensitive.
Storage backends should receive explicit bucket-relative keys. Storage implementations should not infer campaign, session, run, or root-prefix semantics.
Raw secrets must not be stored in pipeline, campaign, or session YAML or written
to manifests, logs, generated configuration, reports, publish metadata,
documentation, or examples. Secrets enter through configured environment
variable names or secret-file references. Diagnostics should avoid transcript
and prompt content unless a deliberate, bounded inspection mechanism requires
it.
## Publish Invariants
Logs, reports, generated invocation files, generated configuration, and render
debug files are diagnostics, not canonical pipeline products. They should be
durable and discoverable where configured, and manifest references must preserve
the distinction between diagnostics and artifacts.
Publish behavior must preserve a clear commit boundary.
Documentation security rules belong in the
[Documentation Policy](documentation.md). Credential supply belongs in
[Configuration](../config.md), while permissions, sensitive runtime-artifact
handling, and recovery belong in [Operations](../operations.md).
A remote run is current only after the publish stage has successfully uploaded the run record, required published outputs, `current/manifest.json`, and finally `current/run_id.txt`.
## Determinism And Testability
`current/run_id.txt` is the final remote commit marker and must be written last.
Narratio prefers deterministic behavior where practical, including stable local
and remote layouts, sorted operation order, predictable generated
configuration, repeatable command construction, deterministic artifact
resolution, and reproducible planning.
Failed, incomplete, skipped, or uncommitted publish attempts must not be presented as current remote state. Local cleanup is permitted only after successful publish commit and only when explicitly configured.
Run IDs and timestamps may be intentionally variable, but surrounding behavior
must remain controllable in tests. Core behavior should be testable without live
external services; expensive, nondeterministic, destructive, or external
boundaries should be replaceable with focused test doubles. General testing
philosophy and sufficiency rules belong in the [Testing Policy](testing.md).
## Security and Privacy
## Documentation And Decision Records
Narratio handles private campaign material.
Documentation follows the [Documentation Policy](documentation.md). Current
behavior belongs in its canonical user, operator, integration, architecture, or
internal owner. Proposed behavior and implementation status belong under
`docs/roadmap/`.
Rules:
Significant architectural decisions may be recorded under `docs/adr/` using the
format and lifecycle defined by the documentation policy. ADR acceptance does
not establish that a decision has been implemented.
- Do not store raw secrets in pipeline or session YAML.
- Use environment variable names or secret-file references for secret handling.
- Do not write raw secret values to manifests, logs, generated configs, or publish metadata.
- Treat transcripts, generated artifacts, prompts, reports, and logs as potentially sensitive.
- Avoid logging transcript or prompt content unless there is a deliberate diagnostic reason.
## Architectural Non-Goals
## Diagnostics
Diagnostics should be durable and discoverable, but distinct from canonical outputs.
Logs, reports, generated invocation/config files, and render-debug files support debugging. Transcript tiers and configured artifacts are pipeline products.
Manifest refs should preserve that distinction.
## Determinism
Where practical, Narratio should prefer deterministic behavior:
- stable local path layout;
- stable remote key layout;
- sorted upload order;
- predictable generated config files;
- repeatable command construction;
- tests that do not depend on live external services.
Run IDs and timestamps may be intentionally variable, but surrounding behavior should remain testable.
## Testing Expectations
Core behavior should be testable without live external services.
Tests should cover:
- config loading, defaults, and validation;
- CLI parsing and command construction;
- path helpers;
- manifest transitions;
- stage success, failure, skip, and resume behavior;
- adapter command construction;
- fake storage behavior;
- publish commit ordering;
- example config validity where practical.
Live S3, WhisperX, LLM, or subprocess integration tests should be explicit integration tests, not required for ordinary unit test runs.
## Documentation Expectations
Documentation must follow `docs/documentation/policy.md`.
Current behavior belongs in user-facing docs and `docs/internal/`. Future, planned, aspirational, experimental, or unimplemented work belongs only under `docs/roadmap/`.
`docs/architecture.md` should remain concise and principle-focused. It should not duplicate the full config reference, CLI reference, operations guide, or internal stage documentation.
## Non-Goals
Narratio is not:
Narratio does not aim to provide:
- a generic DAG or workflow engine;
- a replacement configuration layer for Seriatim, Audita, or Scriptorium;
- a storage backend abstraction beyond the needs of this pipeline;
- a place to embed raw secrets;
- a place for stage logic to depend directly on AWS SDK types or downstream tool internals;
- a replacement configuration layer for WhisperX, Seriatim, Audita,
Scriptorium, or other downstream tools;
- a storage abstraction broader than the needs of this pipeline;
- stage logic coupled directly to cloud SDKs, transports, subprocess details,
or downstream implementation internals;
- raw-secret persistence;
- implicit cross-stage behavior that bypasses manifest and artifact contracts;
or
- a prompt-authoring system.

View File

@@ -1,94 +0,0 @@
# Development Guide
## Purpose
Canonical contributor workflow and engineering conventions for implemented Narratio behavior.
## Repository layout
- `cmd/narratio/`: CLI entrypoint.
- `internal/app/`: command handlers, run/stage orchestration, cleanup gates, secrets loading.
- `internal/config/`: strict YAML loading, defaults, and validation.
- `internal/stage/`: stage implementations and stage registry/order.
- `internal/adapters/`: external boundary adapters (WhisperX, Seriatim, Audita, Scriptorium, storage, notify).
- `internal/manifest/`: session/run manifest types and persistence.
- `internal/artifacts/`: canonical local/remote path helpers and local artifact store.
- `docs/`: canonical documentation set.
- `examples/`: maintained config examples used by tests.
## Build and test commands
- Run focused CLI behavior checks:
```bash
go test ./internal/app -run TestExecute -v
```
- Run config example load/validate checks:
```bash
go test ./internal/config -run TestExamplesLoadAndValidate -v
```
- Run full test suite:
```bash
go test ./...
```
## Coding conventions
- Keep orchestration explicit and stage-driven; do not introduce generic workflow/DAG abstractions.
- Keep external-system details inside adapter packages; stages should consume Narratio-level contracts only.
- Use centralized path helpers from `internal/artifacts` rather than ad hoc path concatenation.
- Preserve manifest-driven state transitions (`running`, `succeeded`, `failed`, `skipped`, `stale`) as the source of run progress.
- Keep user/operator docs implementation-accurate; planned work belongs only under `docs/roadmap/`.
For design principles and invariants, see [docs/architecture.md](./architecture.md). For stage/adapter contracts, see [docs/internal/README.md](./internal/README.md).
## Dependency policy
- Prefer Go standard library where practical.
- Add third-party dependencies only when they provide clear value for required behavior.
- Keep dependency additions narrow to the boundary package that needs them.
## Change playbooks
### Add config fields
1. Add fields to config structs in `internal/config`.
2. Set defaults in `internal/config/defaults.go` when appropriate.
3. Add validation rules in `internal/config/validate.go`.
4. Add or update load/validate tests in `internal/config/*_test.go`.
5. Update canonical config docs and examples:
- [docs/config.md](./config.md)
- relevant files under `examples/`
### Add CLI flags or commands
1. Update command parsing and behavior in `internal/app`.
2. Add or update command tests (`TestExecute` and command-specific tests).
3. Update [docs/cli.md](./cli.md) and, if operator workflow changes, [docs/operations.md](./operations.md).
Remote-storage commands must obtain object storage through the app-level command object-store helper. Do not call `storage.NewObjectStoreFromConfig` directly from command handlers; the helper loads configured filesystem secrets before constructing the storage adapter.
### Add or modify stages/adapters
1. Implement stage behavior in `internal/stage` with clear input/output boundaries.
2. Keep external transport/subprocess details in `internal/adapters`.
3. Preserve manifest and publish-output semantics expected by runner and publish logic.
4. Add/update stage and adapter tests.
5. Update internal component contracts in `docs/internal/`.
### Update examples
1. Keep canonical examples only in `examples/`.
2. Ensure examples load and validate through runtime config paths.
3. Update `internal/config/load_validate_test.go` as needed.
4. Update links in `docs/config.md` if example filenames change.
### Update docs and roadmap
1. Keep implemented behavior in canonical docs (`README`, `docs/*.md`, `docs/internal/`).
2. Keep planned/unimplemented behavior only in `docs/roadmap/`.
3. After completing roadmap items, remove or mark them complete in `docs/roadmap/documentation.md`.
4. Run a link/path sweep before finalizing changes.

View File

@@ -1,356 +1,148 @@
# Go Project Documentation Policy
# Documentation Policy
## Purpose
Project documentation must help four audiences:
1. users who need to run the application;
2. administrators/operators who need to configure and operate it;
3. developers who need to understand and change it safely;
4. LLM coding agents that need clear scope, boundaries, and invariants.
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
This policy assigns each documentation topic to one canonical owner. Its goal is
to keep Narratio documentation accurate, concise, discoverable, and resistant
to drift for users, operators, developers, integrators, and LLM coding agents.
## Core Rules
### 1. Keep docs concise
### One Canonical Owner
Each authoritative fact belongs in one document. A non-owning document may give
a short, stable summary for orientation, but it must link to the canonical owner
instead of repeating volatile details.
Volatile details include commands, flags, configuration fields and defaults,
stage or integration keys, schemas, file names, paths, status codes, retry
behavior, and runtime guarantees. If readers could reasonably treat a statement
as a contract, maintain it only in the owning document.
Each document should cover a defined scope and only the essentials for that scope.
### Current And Future Behavior
Outside `docs/roadmap/`, documentation describes implemented behavior only.
Partial features may be described only to their implemented boundary.
ADRs are the narrow exception: an ADR may record an accepted architectural
decision before implementation, but acceptance must not be presented as proof
that the behavior exists. The roadmap owns implementation status and sequencing
until the decision is implemented. Current architecture, user, operator,
integration, and internal documentation are updated when the behavior lands.
### Audience And Detail
Write for the document's stated audience and include only the detail needed for
its owned topic. User and operator docs should not expose implementation detail.
Developer docs should link to user-facing and external contracts rather than
restate them.
### Examples
Avoid:
- long background explanations;
- repeated reference material;
- implementation detail in user-facing docs;
- aspirational language outside roadmap docs;
- verbose examples where one minimal example is clearer.
### 2. Document only implemented behavior outside roadmap files
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
- `docs/roadmap/`
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
### 3. Use canonical homes
Each type of information should have one canonical location.
Canonical homes:
- project purpose and quickstart: `README.md`
- development principles: `docs/architecture.md`
- configuration reference: `docs/config.md`
- CLI reference: `docs/cli.md`
- operations and recovery: `docs/operations.md`
- troubleshooting: `docs/troubleshooting.md`
- implemented internals: `docs/internal/`
- future work: `docs/roadmap/`
- contributor workflow: `docs/development.md`
- copyable examples: `examples/`
Other files should summarize briefly and link to the canonical source.
### 4. Keep examples real
Examples should be valid, maintained, and free of secrets.
Where practical:
- example configs should load successfully;
- example commands should match real CLI syntax;
- important examples should be covered by tests.
## Documentation Profiles
All projects require:
- `README.md`
- `docs/architecture.md`
Additional docs depend on the project.
### Small library
Recommended:
- `docs/development.md`, if contributor conventions are non-obvious
### Simple CLI
Required:
- `docs/cli.md`
Recommended:
- `docs/development.md`
### Config-driven CLI
Required:
- `docs/cli.md`
- `docs/config.md`
Recommended:
- `examples/`
- `docs/development.md`
### Stateful or operator-facing application
Required:
- `docs/cli.md`, if CLI-based
- `docs/config.md`, if config-driven
- `docs/operations.md`
Recommended:
- `docs/troubleshooting.md`
- `examples/`
- `docs/development.md`
### Modular, staged, service-oriented, or orchestration application
Required:
- `docs/cli.md`, if CLI-based
- `docs/config.md`, if config-driven
- `docs/operations.md`
- `docs/internal/`
- `docs/development.md`
Recommended:
- `docs/troubleshooting.md`
- validated examples under `examples/`
## Required Documents
### README.md
**Audience:** users, administrators, operators
The README is the outward-facing project orientation page.
It should include, in order:
1. concise description;
2. elevator pitch;
3. shortest useful command or usage example;
4. links to targeted docs.
The README should be short. It is not a manual.
The “shortest useful command” means the simplest command that performs the projects core use case. (It does not mean `app --help`.)
### docs/architecture.md
**Audience:** developers, LLM coding agents
`docs/architecture.md` is required for every project.
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
It should include:
- project shape;
- core design principles;
- package and boundary philosophy;
- state/persistence philosophy, if applicable;
- external integration philosophy, if applicable;
- error-handling and logging principles;
- testing expectations;
- documentation expectations;
- architectural invariants;
- explicit non-goals, if useful.
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
### docs/config.md
**Audience:** administrators, operators, advanced users
Required for applications with configuration files.
It should include, in order:
1. config file locations and discovery precedence;
2. minimal working config;
3. production-oriented config;
4. full configuration reference;
5. secrets handling, if applicable;
6. links to maintained examples.
The full configuration reference should be canonical.
### docs/cli.md
**Audience:** users, administrators, operators
Required for CLI applications.
It should include, in order:
1. shortest useful command;
2. command overview;
3. complete flag reference;
4. common workflows;
5. diagnostic or recovery commands, if applicable.
Explain when commands are useful, not just their syntax.
### docs/operations.md
**Audience:** administrators, operators
Required for applications that maintain state, support resume behavior, run multiple stages, write durable artifacts, use remote storage, or require recovery procedures.
It should cover:
- normal workflow;
- filesystem layout;
- remote storage layout, if applicable;
- logs and manifests;
- resume/retry behavior;
- cleanup behavior;
- archive/backup behavior;
- safe recovery procedures;
- operational caveats.
### docs/troubleshooting.md
**Audience:** administrators, operators
Recommended once recurring failure modes exist.
Each entry should include:
- symptom;
- likely cause;
- diagnostic command or inspection step;
- safe fix;
- relevant links.
### docs/development.md
**Audience:** developers, LLM coding agents
Required for projects maintained by humans and LLM coding agents.
It should include:
- repository layout;
- build/test commands;
- coding conventions;
- dependency policy;
- how to add config fields;
- how to add CLI flags;
- how to add stages/modules/adapters, if applicable;
- how to update examples;
- documentation update expectations.
### docs/internal/
**Audience:** developers, LLM coding agents
Required for modular, staged, service-oriented, or orchestration projects.
This directory describes implemented internal components. It is not the roadmap.
Use one file per major component where useful.
Each component doc should include:
1. purpose;
2. inputs and outputs;
3. boundaries;
4. config fields used;
5. external adapters used;
6. state or manifest behavior, if applicable;
7. skip/resume behavior, if applicable;
8. failure behavior;
9. tests to inspect before changing;
10. architectural invariants.
### docs/roadmap/
**Audience:** maintainers, developers, LLM coding agents
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
Roadmap docs should clearly distinguish:
- proposed work;
- accepted plans;
- deferred ideas;
- rejected ideas;
- implementation prompts or task breakdowns, if useful.
Roadmap docs should not be confused with current behavior.
### docs/integrations/
**Audience:** developers, LLM coding agents
Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses.
Use one file per integration where useful.
## Examples Directory
Projects with non-trivial configuration or workflows should include `examples/`.
Useful examples include:
- minimal working config;
- production-oriented config;
- full annotated config;
- local development config;
- remote/object-storage config;
- minimal session/input file.
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
## Security and Privacy
Docs and examples must not include:
- real API keys;
- tokens;
- passwords;
- private keys;
- private environment dumps;
- sensitive user data;
- raw private transcripts;
- private infrastructure details unless intentionally public.
Document secret-handling mechanisms, not actual secret values.
## Maintenance Rules
When docs change, verify the affected behavior.
Where practical:
- load example config files in tests;
- test CLI examples or command parser behavior;
- validate documented flags against real flags;
- remove stale references;
- update links after renames;
- keep roadmap content out of non-roadmap docs.
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
Documentation is complete only when it matches the current code.
## Documentation Change Checklist
Before merging documentation changes, verify:
- README is concise and orientation-focused.
- `docs/architecture.md` describes development principles.
- Future work appears only under `docs/roadmap/`.
- User-facing docs avoid unnecessary internals.
- Developer-facing docs preserve boundaries and invariants.
- Config examples match the schema.
- CLI examples match real commands and flags.
- Defaults appear in the canonical config reference.
- No secrets or private data are included.
- Links are accurate.
Complete copyable files belong in `examples/`. Documentation may use the
smallest illustrative snippet needed to explain its owned topic, but should link
to maintained examples instead of embedding a second complete copy.
Examples must be valid, secret-free, and tested where practical. Commands and
configuration used in documentation should match the application.
### Security And Privacy
Documentation and examples must not contain real credentials, private keys,
private environment dumps, sensitive source material, or private infrastructure
details unless intentionally public. Document secret-handling mechanisms, not
secret values.
## Canonical Ownership
| Topic | Canonical owner | Owned content | Content owned elsewhere |
| --- | --- | --- | --- |
| Product orientation and minimal end-to-end quickstart | `README.md` | What Narratio is, why it is useful, one shortest successful invocation, and links onward. | Complete command reference, configuration reference, operational procedures, implementation detail. |
| Contributor entry point | `docs/development.md` | Task-oriented reading guide, minimal contributor orientation, baseline validation commands, and links to canonical docs. | Package inventory, architecture rules, subsystem behavior, detailed change recipes. |
| Current application architecture | `docs/policy/architecture.md` | System shape, normative ownership, dependency direction, architectural boundaries, invariants, safety properties, and non-goals. | Concrete package inventory, implementation mechanics, contributor procedures, decision history, future work. |
| Documentation organization | `docs/policy/documentation.md` | Documentation ownership, audience boundaries, maintenance rules, and ADR/document lifecycle. | Application architecture or product behavior. |
| Testing policy | `docs/policy/testing.md` | Test philosophy, risk-based sufficiency, test boundaries, doubles, coverage guidance, regression-test policy, and criteria for adding, rewriting, or deleting tests. | Subsystem behavior, application contracts, subsystem-specific test inventories, and implementation plans. |
| CLI contract | `docs/cli.md` | Commands, arguments, flags, invocation semantics, output conventions, and exit behavior. | End-to-end operating procedures, configuration field definitions, runtime filesystem layout, stage implementation details. |
| Configuration contract | `docs/config.md` | Discovery and precedence, file schemas, fields, defaults, environment overrides, validation rules, and user-selectable stage or integration settings. | Complete example files, CLI syntax, runtime state lifecycle, implementation details. |
| Operations | `docs/operations.md` | Runtime workflows, physical filesystem and remote-state layout, output and diagnostic handling, resume, cleanup, permissions, recovery, and operational limits. | CLI flag syntax, configuration field definitions, logical artifact schemas, implementation mechanics. |
| Troubleshooting | `docs/troubleshooting.md` | Symptom-driven diagnosis, likely causes, safe inspection steps and remedies, and links to relevant contracts. | CLI syntax, configuration definitions, operational procedures, integration contracts, implementation mechanics. |
| Public HTTP contract, if introduced | `docs/api.md` | Routes, authentication, media types, request and response schemas, status codes, pagination, caching, idempotency, rate limits, and HTTP retry semantics. | Client walkthroughs, upstream or downstream integration internals, implementation detail. |
| Consumer guidance, if a public package or API is introduced | `docs/consumers/` | Task-oriented use of the public interface, minimal client examples, and consumer responsibilities. | HTTP wire semantics, external protocol contracts, internal implementation detail. |
| External and durable integration contracts | `docs/integrations/` | External file formats and protocols, upstream and downstream contracts, logical artifact paths and schemas, media types, and compatibility behavior. | Physical runtime placement and lifecycle, internal transformations, CLI syntax, configuration defaults. |
| Implemented component inventory | `docs/internal/overview.md` | Current packages and components, their implemented responsibilities, and links to focused internal docs. | Normative architecture, contributor reading policy, external contracts. |
| Internal component behavior | Other files under `docs/internal/` | Implementation flow, internal collaborators and state transitions, package-local guarantees and failures, and relevant tests. | Global architecture invariants, configuration definitions and defaults, external schemas, operator procedures. |
| Architectural decision history | `docs/adr/` | Significant decisions, context, alternatives, rationale, consequences, and supersession history. | Current behavior reference, implementation status, task sequencing. |
| Future work and implementation status | `docs/roadmap/` | Proposed, accepted, deferred, or rejected work; implementation status; sequencing; and task breakdowns. | Implemented behavior reference and architectural decision rationale. |
| Complete copyable artifacts | `examples/` | Maintained configuration, inputs, and other files intended to be copied or run. | Field-by-field reference, command reference, prose explanation. |
Documents that do not exist are required only when the corresponding interface
or responsibility exists. Do not create placeholder API, consumer, integration,
or operations documents for behavior the application does not have.
## Boundary Rules
### Orientation
The README owns product orientation. The developer guide routes contributors.
Architecture owns normative structure. Internal overview owns the current
concrete component map. These documents may link to one another but should not
maintain parallel package or behavior descriptions.
### Commands, Configuration, Operations, And Troubleshooting
CLI documentation answers how to invoke the application. Configuration
documentation answers what settings mean. Operations answers what happens to
runtime state and how to operate or recover the application. Troubleshooting
starts from observable symptoms and links readers to the owning command,
configuration, operational, or integration contract. When a workflow crosses
these topics, choose the document that owns the task and link to the other
contracts.
### Contracts And Implementation
Integration and API documents define externally observable shapes and
semantics. Internal documents explain how Narratio implements or consumes those
contracts. Internal docs may name a field, file, or protocol to identify a
dependency, but must link to its canonical contract for the definition.
### Security Topics
This policy owns what documentation and examples may contain. Architecture owns
application security invariants. Configuration owns credential-supply
mechanisms. Operations owns permissions and handling of sensitive runtime
artifacts. Troubleshooting owns safe diagnostic and remediation guidance.
Internal docs own implementation mechanisms only.
## Architecture Decision Records
Use sequentially numbered ADR filenames such as
`0001-record-architecture-decisions.md`. Follow the lightweight Nygard format:
1. title;
2. status;
3. date;
4. context;
5. decision;
6. alternatives considered;
7. consequences.
Treat the decision content of an accepted ADR as immutable. When a decision
changes, create a new ADR and update the earlier ADR's status to superseded.
Rejected architectural alternatives belong in the ADR; rejected product ideas
belong in the roadmap.
## Maintenance
When behavior changes, update its canonical owner in the same change. If
ownership moves, remove the old definition and replace it with a link where
navigation remains useful.
Before completing documentation work:
- verify affected behavior and examples;
- check commands, flags, fields, defaults, schemas, and paths against their
implementation;
- keep unimplemented behavior in the roadmap, subject to the ADR exception;
- remove stale references and validate links;
- confirm that non-owning documents summarize and link rather than redefine;
- confirm that no secrets or sensitive private data were added.

296
docs/policy/testing.md Normal file
View File

@@ -0,0 +1,296 @@
# Testing Policy
## Purpose
Our tests exist to make **incorrect changes expensive and correct changes cheap**.
We do not optimize for test count, line coverage, exhaustive isolation, or the fewest possible tests. We optimize for sufficient confidence in important behavior while imposing as little unnecessary friction as possible on future development.
## Every test has a cost
Testing is not an unqualified good. Every test imposes both an immediate cost and a continuing lifetime cost.
A test must be:
- written and reviewed;
- understood by future maintainers and coding agents;
- executed in local and CI workflows;
- diagnosed when it fails;
- updated when legitimate behavior changes;
- maintained as fixtures, APIs, and dependencies evolve; and
- removed or rewritten when it becomes redundant, brittle, misleading, or obsolete.
Tests also create cognitive and architectural friction. They can constrain refactoring, duplicate policy, slow feedback loops, add noise to failures, and cause harmless implementation changes to require unrelated edits across the suite.
A test is warranted only when the confidence it provides justifies these costs.
Apply this cost-benefit analysis at two levels:
1. **Per test:** What realistic defect does this test detect, how consequential would that defect be, and is that protection worth the test's lifetime cost?
2. **Across the suite:** Does this collection provide materially more confidence than a smaller, simpler suite would?
The preferred test suite is a **lean suite that provides sufficient confidence in the risks that matter, without redundant or low-value tests**. We seek sufficient confidence with the least unnecessary testing friction, not the fewest possible tests.
Some friction is intentional. Tests should make dangerous changes—such as breaking compatibility, corrupting data, violating security boundaries, or reintroducing subtle bugs—require deliberate review. They should not make ordinary internal changes needlessly expensive.
The cost of a test is not a reason to omit testing by default. Do not cite maintenance cost abstractly. When omitting a plausible test, be able to state why the protected failure is low-risk, already covered, obvious, reversible, or cheaper to detect elsewhere. For consequential, subtle, or difficult-to-observe behavior, the presumption should favor testing.
## Default testing style
Use a **classical/Detroit-style** approach:
- Test observable behavior, resulting state, contracts, and invariants.
- Use real internal collaborators when they are fast and deterministic.
- Use fakes, stubs, or mocks primarily at expensive, nondeterministic, destructive, or external boundaries.
- Prefer package-level behavioral tests over tests coupled to private helpers or internal call sequences.
- Treat exact collaborator interactions as testable behavior only when the interaction itself is a requirement.
Examples of appropriate seams include clocks, randomness, subprocesses, remote APIs, object storage, email, and paid LLM calls.
## Test execution requirements
Tests in the default suite must be deterministic, offline, and independent of real credentials. They must not invoke paid APIs or depend on mutable external services. Tests that require live infrastructure must be explicitly opt-in and clearly separated from the default suite.
Control clocks, randomness, environment variables, and other process-global or machine-specific state when they affect behavior. Tests should be safe to run repeatedly and alongside other tests without depending on execution order or state left by an earlier test.
## What deserves tests
Prioritize tests for:
1. Public and package-level contracts.
2. Domain rules and important invariants.
3. Boundary conditions and malformed input.
4. Failure handling, cancellation, retries, recovery, and partial success.
5. Serialization, schemas, compatibility, and round trips.
6. Previously observed or plausible regressions.
7. Representative integration and end-to-end workflows.
A package-level contract is behavior relied upon by another package or major collaborator, not every observable detail of a package implementation.
For behavior involving **data integrity, destructive operations, compatibility, security, concurrency, idempotency, or recovery**, presume that durable tests are required unless the behavior is already credibly protected at another layer.
Do not add tests merely because a function, branch, or line exists. Do not add a test when the same meaningful risk is already adequately protected elsewhere.
## Choose the right test boundary
Test through the narrowest stable boundary that expresses the behavior clearly.
This is often the package API, but it may instead be:
- a smaller pure function when dense domain logic is most clearly isolated there;
- a package-level operation when several internal collaborators jointly produce the behavior; or
- a larger integration boundary when correctness emerges from interaction with a real dependency.
Do not force all behavior through oversized end-to-end tests. Do not test every private helper merely because it exists. Choose the boundary that gives durable confidence with the least incidental coupling.
## Test behavior, not implementation
A test should protect a decision, contract, or invariant—not memorialize the current implementation.
Before adding or retaining a test, ask:
> What realistic defect would this test catch?
A test is suspect when its main purpose is to detect that someone:
- changed an internal constant;
- renamed or split a private helper;
- reordered equivalent internal operations;
- changed incidental formatting;
- replaced one correct algorithm with another; or
- refactored internal object structure without changing behavior.
Refactoring should normally require no test edits unless the refactored structure is itself part of the contract.
A test can be factually correct and still have negative value. Accurately describing current behavior is not enough; the protected behavior must be important enough to justify the future friction.
## Expected effects of different changes
Use the following expectations when evaluating test failures and test maintenance:
| Change | Expected effect on tests |
|---|---|
| Internal refactor that preserves behavior | Existing tests should normally remain unchanged and continue to pass. |
| Change to an internal default with no contractual significance | Behavioral tests should normally remain unchanged; tests should derive expectations from configuration or relationships rather than duplicate the old value. |
| Intentional change to public behavior, policy, schema, or compatibility guarantees | The relevant tests should be reviewed and changed deliberately. |
| Accidental violation of a contract or invariant | Tests should fail; fix the production code rather than rewriting the tests to accept the defect. |
A test failing is not the same as a test needing to be edited. Many tests may correctly fail because of one production defect. The maintenance smell is a correct internal change that requires unrelated expectation updates throughout the suite.
## Separate mechanism from policy
Configurable thresholds and defaults must not be duplicated throughout the test suite.
For example, do not encode an internal concurrency limit indirectly:
```go
// Production policy:
const maxConcurrency = 4
// Brittle test:
err := startProcesses(5)
require.Error(t, err)
```
Instead, test the mechanism relationally:
```go
const limit = 2
runner := NewRunner(limit)
require.NoError(t, runner.Start(limit))
require.ErrorIs(t, runner.Start(limit+1), ErrTooMuchConcurrency)
```
The test should prove:
- the configured limit is accepted; and
- one beyond the configured limit is rejected.
The production default should be tested exactly only when its literal value is itself a public, operational, safety, protocol, or compatibility requirement.
Apply the same rule to limits, timeouts, capacities, retry counts, and ranges: test relationships and behavior, not duplicated literals.
For concurrency limits, test both kinds of behavior when relevant:
1. **Configuration enforcement:** invalid or excessive requested values are handled correctly.
2. **Runtime enforcement:** observed peak concurrency never exceeds the configured limit.
Use a test-controlled limit and measure the behavior relative to that limit. Do not merely assert today's default value.
## Avoid semantic duplication across layers
Each behavior should have a clear test owner.
- Parser tests own parsing cases.
- Validator tests own validation rules.
- Domain tests own transformations and invariants.
- Adapter tests own external integration behavior.
- Orchestrator tests own coordination and failure propagation.
- CLI tests own argument and configuration mapping.
- End-to-end tests prove that representative assembled workflows work.
Higher-level tests should not repeat every lower-level case. A single intentional policy change should not require unrelated edits across many test files.
Tests that are individually reasonable may still be collectively redundant. Evaluate the marginal value of each additional test in light of the protection already provided by the rest of the suite.
## Use test doubles deliberately
Choose the least elaborate test double that provides the required control or observation.
As a default:
1. Prefer real collaborators when they are fast and deterministic.
2. Use small in-memory fakes when realistic stateful behavior is helpful.
3. Use stubs when a dependency only needs to provide controlled responses.
4. Use mocks when the interaction itself is contractual.
Mocks are appropriate when the contract includes facts such as:
- a notification is sent exactly once;
- a transaction is committed only after successful writes;
- cancellation reaches a subprocess;
- an expensive API is called no more than once; or
- a security audit event is emitted.
Do not use mocks merely to isolate every object or reproduce the implementation's call graph.
## Go-specific guidance
Use:
- table-driven tests for meaningful behavioral categories and boundaries;
- `t.TempDir()` for real filesystem behavior;
- `httptest.Server` for realistic HTTP interactions;
- fuzz tests for parsers, normalization, path handling, and broad input spaces;
- golden files only when the complete output is intentionally stable;
- integration tests where correctness depends on component interaction; and
- a small number of representative end-to-end tests.
Avoid exact error-string assertions unless the wording is itself contractual. Prefer `errors.Is`, `errors.As`, typed errors, or structured error fields.
At CLI boundaries, prefer exit classifications, structured output, and the smallest stable semantic fragment needed to identify the error. Do not snapshot complete diagnostic wording unless it is contractual.
Golden-file updates must require an explicit local flag. CI must not update golden files automatically, and reviewers must inspect the semantic diff before accepting an update.
Keep tests readable and direct. Test helpers and fixture frameworks must earn their own maintenance cost; do not build elaborate test infrastructure for small or isolated needs.
## Coverage
Coverage is a diagnostic, not a target.
Use it to find untested critical branches and unexpectedly weak packages. Do not write low-value tests solely to increase a percentage, and do not infer test quality from coverage alone.
Pure domain logic will often warrant higher coverage than CLI wiring or external adapters. Uneven coverage is acceptable when it reflects risk.
Increasing coverage is valuable only when the newly covered behavior protects a meaningful risk at an acceptable cost.
## Regression tests
A bug fix should normally include a regression test that fails before the fix and passes afterward.
Retain the test when the defect could realistically recur and its consequences justify the ongoing cost. Prefer the narrowest durable test of the violated contract or invariant; do not preserve accidental implementation details from the original bug.
Not every historical bug requires a permanent test. If the underlying design has made recurrence impossible, the test has become redundant, or a stronger invariant test now subsumes it, remove or consolidate it.
## Deleting or rewriting tests
Tests are maintained code, not permanent historical artifacts.
Delete or rewrite a test when its maintenance cost exceeds the confidence it provides.
Strong candidates include tests that:
- require updates after harmless internal changes;
- directly assert private constants without protecting a real contract;
- duplicate the same policy across several layers;
- verify mock choreography rather than outcomes;
- snapshot large amounts of incidental output;
- test trivial private helpers already exercised through stable package behavior;
- protect risks already covered more effectively elsewhere;
- are flaky, misleading, obsolete, or disproportionately expensive to diagnose; or
- no longer correspond to a plausible failure mode.
Several brittle tests may encode one genuine requirement. Replace them with one durable behavior-level or invariant test rather than preserving all of them.
Deleting a low-value test can improve the quality of the suite by reducing noise, maintenance burden, and friction around legitimate change.
## Reviewing a proposed test
Use the following questions when the value, boundary, or durability of a proposed test is not self-evident. Significant test additions should be reviewable against them, but written answers are not required for every routine test.
1. What realistic defect would it catch?
2. How likely is that defect?
3. How consequential would it be?
4. Is the behavior already protected elsewhere?
5. At which layer should this behavior be owned?
6. Does the test assert a durable contract or an incidental implementation detail?
7. Could the implementation be refactored without changing the behavior and without editing this test?
8. What should cause this test to fail?
9. What legitimate changes should not cause this test to fail?
10. What ongoing maintenance, execution, and diagnostic cost will the test impose?
11. Is there a smaller or more direct test that protects the same risk?
Do not add the test when its expected lifetime cost exceeds its expected protective value.
When deciding not to test plausible behavior, record or be able to explain why the risk is low, already protected, obvious, reversible, or cheaper to detect elsewhere.
## Definition of sufficient
A test suite is sufficient when:
- important contracts and invariants are protected;
- meaningful boundaries and failure modes are exercised;
- realistic and consequential regressions are credibly protected against silent recurrence;
- behavior involving data integrity, destructive operations, compatibility, security, concurrency, idempotency, and recovery is credibly protected;
- important external boundaries have realistic integration coverage;
- representative complete workflows are tested;
- failures provide useful signal rather than redundant noise;
- legitimate internal changes usually do not require test edits; and
- additional tests would mostly repeat existing protection or preserve inconsequential implementation details.
Sufficiency is a risk judgment, not a coverage percentage or test count. Reassess it as the application, its users, and the consequences of failure evolve.
The governing rule is:
> Test heavily where failure is consequential, subtle, or difficult to detect after the fact. Test lightly where failure is obvious, reversible, and inexpensive—and retain no test whose lifetime cost exceeds the confidence it provides.

File diff suppressed because it is too large Load Diff

332
docs/roadmap/audit-plan.md Normal file
View File

@@ -0,0 +1,332 @@
# Codebase Audit Plan
Status: proposed
## Purpose
This audit will evaluate Narratio for correctness, efficiency, maintainability,
and test-suite value. It will identify defects and credible risks, duplicated or
near-duplicated behavior, code that can be made smaller or more idiomatic, and
complex code whose remaining invariants need focused explanation.
The audit is investigative. It should produce evidence-backed findings and a
prioritized remediation backlog, not make opportunistic production changes as
it proceeds. The [Audit Sequence](audit-sequence.md) assigns this scope to
concrete execution stages.
## Authoritative Baseline
Review implemented behavior against its canonical owner rather than treating
the current implementation or tests as the specification:
- [Architecture](../policy/architecture.md) for system boundaries, dependency
direction, state and path ownership, safety properties, and pipeline
invariants;
- [Internal Overview](../internal/overview.md) and its focused internal
documents for implemented ownership and mechanics;
- [Testing Policy](../policy/testing.md) for risk-based sufficiency, durable
boundaries, test-double guidance, and test lifecycle decisions;
- the [CLI](../cli.md), [Configuration](../config.md),
[Operations](../operations.md), and [integration contracts](../integrations/)
for externally observable behavior; and
- the [Documentation Policy](../policy/documentation.md) for canonical ownership
and the distinction between current and proposed behavior.
Where code, tests, and documentation disagree, record the disagreement. Do not
assume which one is wrong until the canonical contract and caller expectations
have been traced.
## Audit Principles
1. Review correctness before cleanup. A shorter implementation is not an
improvement if it weakens a state transition, safety check, or external
contract.
2. Trace behavior across boundaries. Narratio's most important properties often
emerge from the interaction of application orchestration, stages, manifests,
artifact resolution, filesystem operations, and adapters.
3. Distinguish repeated syntax from repeated policy. Extract a helper only when
the behavior has one stable owner and the shared abstraction makes that
ownership clearer. Similar stage code may be intentionally explicit.
4. Prefer narrow, idiomatic Go over generic frameworks. In particular, proposed
refactors must preserve the explicit canonical stage sequence and must not
turn Narratio into a workflow engine or a second configuration system for
downstream tools.
5. Optimize credible work. Flag repeated I/O, hashing, serialization, remote
calls, subprocess work, allocation, or poor asymptotic behavior when the
relevant path can matter. Require a benchmark or workload argument for
performance changes whose benefit is not evident.
6. Treat comments as explanations of intent. Recommend comments for invariants,
ordering constraints, non-obvious failure policy, or security reasoning—not
as narration of ordinary Go or a substitute for simplifying code.
7. Judge tests as a suite. A test can be locally reasonable and still add no
marginal protection, while a compact test can be inadequate for a
consequential cross-component failure.
## Evidence And Finding Standard
Begin from a cleanly identified revision and record toolchain and platform
assumptions. Use the code knowledge graph to find ownership, callers, callees,
similarity candidates, high-complexity functions, and weakly protected
boundaries. Confirm every candidate by reading the implementation, its focused
tests, and the applicable contract. Text search and static analysis supplement
the graph for literals, configuration, generated files, and patterns that are
not modeled reliably.
Each finding should record:
- category: correctness defect, correctness risk, duplication, simplification,
efficiency, architectural boundary, comment/clarity, or test-suite issue;
- source locations and the affected contract or invariant;
- concrete evidence and a realistic failure or maintenance scenario;
- impact, likelihood, confidence, and estimated remediation scope separately;
- the smallest plausible improvement and its intended owner;
- tests that already protect the behavior, tests that should change or be
added, and tests that may become redundant; and
- dependencies on, or conflicts with, other findings.
Do not report a metric alone as a finding. Complexity, similarity, coverage,
fan-in, file size, and test count are prioritization signals that require manual
confirmation. Consolidate findings that share one root cause.
## Cross-Cutting Review Lenses
### Correctness And Pipeline Semantics
Construct an explicit lifecycle matrix for every stage outcome: first run,
already-succeeded skip, self-skip, failure, interruption, forced replacement,
non-resumable result, and successful rerun. Trace how each outcome changes the
session manifest, invocation manifest, downstream stage state, artifacts,
diagnostics, and cleanup eligibility.
Across the pipeline, verify:
- the registry exposes one deterministic canonical order;
- each stage's declared inputs, outputs, configuration, adapters, and manifest
effects agree with its implementation and focused documentation;
- inputs are resolved through manifest and artifact contracts rather than
incidental directory contents;
- run-local outputs are fully validated before canonical materialization;
- failure, cancellation, or process interruption cannot advertise partial work
as successful;
- force and changed outcomes invalidate exactly the intended succeeded
downstream work;
- repeated execution is idempotent where promised, and ordering is stable
wherever maps, directory reads, remote listings, or dependency graphs are
involved;
- session, campaign, run, source, checksum, contract, and external provenance
identities cannot be confused across runs; and
- errors preserve useful causes and do not expose secrets or private content.
Use fault-oriented reasoning at durability boundaries: fail immediately before
and after manifest saves, canonical renames, external process completion,
uploads, current-manifest publication, the current-run commit marker, restore
manifest installation, and cleanup. Determine which state is authoritative and
whether the next invocation recovers safely.
### Duplication And Helper Ownership
Search for exact and semantic duplication in production and tests, including:
- repeated stage setup, input resolution, output validation, run-local
materialization, metadata construction, and error adaptation;
- repeated manifest create/load/save and session/run transition handling;
- repeated adapter construction, timeout parsing, command execution, generated
configuration, log handling, and output checks;
- repeated source-ID, destination, remote-key, and path validation policy;
- repeated sorting, deduplication, checksum, copy, and atomic-write mechanics;
and
- repeated test fixtures and assertions that encode the same policy at several
layers.
For each candidate, decide whether it is coincidental similarity, a repeated
mechanism, or duplicated policy. Recommend extraction only when the helper can
have a clear package owner, a narrow contract, and callers that become easier
to understand. Prefer an unexported local helper when sharing is package-local.
Do not create a broad utility package, force unlike stage results into one data
model, or move policy into storage/file-operation helpers.
Initial similarity and complexity signals should seed, but not predetermine,
inspection of the single-stage command wrappers, session/run manifest
persistence pairs, adapter constructors, Scriptorium operations, stage fakes,
and common stage materialization paths.
### Simplification, Go Idioms, And Efficiency
Review long or branch-heavy functions for separable decisions, state
transitions, or data transformations. Pay particular attention to orchestration,
configuration validation, artifact dependency resolution, resume verification,
restore/previous-cache planning, and analyze/publish selection logic. A useful
refactor should reduce cognitive load while leaving the important ordering
visible.
Check for:
- unnecessary nesting, defensive branches made unreachable by earlier
validation, repeated normalization, and overly wide parameter lists;
- interfaces defined for hypothetical extensibility rather than a demonstrated
consumer boundary;
- manual slice, map, string, error, and filesystem logic with a clearer standard
library form;
- incorrect or inconsistent `errors.Is`/`errors.As`, wrapping, context
propagation, deferred cleanup, response-body closure, process waiting, and
goroutine/channel ownership;
- redundant filesystem scans, `stat`/checksum passes, whole-file buffering,
copying, YAML/JSON round trips, sorting, remote listings, downloads, uploads,
or adapter initialization;
- linear searches nested in loops and repeated dependency or artifact lookup
that should use an indexed map or a single planning pass;
- unbounded concurrency, leaked work after cancellation, serialized independent
work, and nondeterministic result collection; and
- obsolete dependencies, portability assumptions, and platform-sensitive path
or atomic-rename behavior.
Keep correctness and diagnosability ahead of micro-optimization. When a simpler
algorithm changes performance characteristics, specify the representative
input size and validation method.
### Comments And Local Explanation
Review high fan-in, high-complexity, security-sensitive, and commit-boundary
code after likely simplifications have been identified. Add a comment
recommendation when a maintainer needs to know why:
- state transitions or persistence operations occur in a specific order;
- a stale record intentionally retains data while another transition clears it;
- a path is checked more than once to resist traversal, symlink replacement, or
time-of-check/time-of-use hazards;
- an artifact is accepted only with particular manifest, checksum, contract, or
provenance evidence;
- a partial operation is intentionally not rolled back;
- a remote pointer or local manifest must be installed last; or
- concurrency, cancellation, compatibility, or downstream-tool behavior makes
an apparently simpler approach unsafe.
Prefer a named helper, typed state, or smaller control flow when that removes the
need for explanation. Check existing comments for stale claims as well as
missing rationale.
### Test Suite Against The Canonical Policy
Build a risk-to-test matrix rather than auditing tests file by file in
isolation. For each important behavior, identify its proper owner—parser,
validator, domain package, adapter, orchestrator, CLI, integration, or end to
end—and identify all tests that claim to protect it.
Evaluate:
- protection of data integrity, destructive operations, compatibility,
security, concurrency, idempotency, recovery, and partial failure;
- manifest transitions, force/invalidation, resume validation, atomic
materialization, publish commit order, restore install order, and cleanup
gates as assembled behaviors;
- realistic HTTP, subprocess, filesystem, and object-store boundary behavior,
including cancellation and malformed responses;
- whether higher-level tests intentionally sample lower-level behavior or
redundantly reproduce its full policy;
- whether tests assert durable outcomes or private constants, exact error text,
incidental paths, call choreography, or oversized snapshots;
- whether real fast collaborators could replace elaborate doubles, and whether
stateful fakes are realistic enough for the risk they protect;
- fixture/helper duplication, oversized test cases, and setup that obscures the
behavior under test without introducing a heavyweight test framework;
- deterministic, offline, credential-free, order-independent execution and
safe handling of environment and process-global state;
- focused fuzz candidates in parsing, normalization, source IDs, remote/local
path mapping, manifest decoding, and configuration boundaries; and
- the presence and value of a small number of representative assembled
workflows.
Use coverage only to locate unexpectedly weak consequential branches. Also
inspect packages with extensive coverage for redundant tests and refactoring
friction. For every proposed addition, deletion, or consolidation, state the
realistic defect and marginal confidence involved.
The audit baseline should include the repository's canonical commands plus
targeted diagnostic runs where supported:
```sh
go test ./...
go test -race ./...
go vet ./...
go build ./cmd/narratio
```
Use focused repeated or shuffled runs to investigate state leakage and
flakiness, and collect package/branch coverage for diagnosis. Review continuous
integration to determine whether the appropriate offline validation is enforced;
do not turn coverage percentage into a gate merely for this audit.
## Area-By-Area Inspection Map
| Area | Primary locations | What to inspect |
| --- | --- | --- |
| Process and application boundary | `cmd/narratio`, `internal/app` | Command dispatch, configuration selection, production composition, secret loading, lock lifetime, object-store initialization, context/error propagation, and separation of CLI reporting from orchestration policy. Review operator commands for consistent current-state authority and shared read-only mechanics. |
| Stage registry and runner | `internal/stage/placeholders.go`, `internal/stage/stage.go`, `internal/app/planner.go`, `internal/app/runner.go`, `internal/app/run_stage.go` | Canonical order, action decisions, resume/force/self-skip/failure transitions, downstream invalidation, session/run manifest consistency, resource lifecycle, cleanup triggering, and opportunities to decompose the runner without hiding its state machine. |
| Configuration | `internal/config` | Strict decoding, discovery and precedence, centralized defaults, normalization, templating, validation order, unknown fields, empty-value behavior, secret references, cross-field constraints, path confinement, deterministic errors, duplicated validator policy, and compatibility with maintained examples. |
| Prepare and audio | `internal/stage/prepare.go`, `internal/audio`, `internal/previouscache` | Local/S3 exclusivity, cache and spool identity, partial downloads, checksum/reuse policy, previous-session required/optional planning, deterministic input records, clearing semantics, traversal safety, and avoiding repeated remote or filesystem work. |
| Transcript stages | `internal/stage/transcribe.go`, `merge.go`, `polish.go`, `normalize.go`, `trim.go`, `render.go` | Contract parity across similar stages, bounded concurrency and cancellation, deterministic speaker/input ordering, run-local validation and canonical promotion, report/diagnostic classification, disabled behavior, and narrow opportunities for shared mechanics. |
| Extraction | `internal/stage/extract.go`, `extract_resume.go`, `internal/adapters/notarius`, `internal/fileops/directory.go` | External receipt and lane validation, configuration fingerprint limits, immutable promotion, symlink/root replacement defenses, provenance and checksum checks, immediate and cross-invocation reuse, obsolete versus unsafe outcomes, failure residue, and whether dense verification logic can be clarified without weakening it. |
| Analyze and artifact dependencies | `internal/stage/analyze.go`, `internal/artifacts`, `internal/artifactpolicy` | Source-family validation, runtime catalog state, enabled/selected/reused distinctions, topological ordering and cycle handling, required/optional inputs, local-only previous sources, deterministic metadata, repeated lookup/scanning, and ownership shared with config and publish. |
| Publish and cleanup | `internal/stage/publish.go`, `internal/app/post_publish_cleanup.go`, `internal/app/cleanup_targets.go` | Prerequisite success, output selection, locks, required/optional behavior, exclusion rules, deterministic upload set, retry/idempotency implications, current-manifest then commit-marker ordering, metadata gates, and destructive path confinement. |
| Manifest state | `internal/manifest` | Validation and backward compatibility, atomic persistence, timestamps, session/run identity, transition truth table, clearing versus retaining payload, create/load/save duplication, failure during dual-manifest updates, and whether state mutation has a single owner. |
| Artifacts, paths, and policy | `internal/artifacts`, `internal/artifactpolicy`, `internal/pathsafe` | Canonical helper coverage, ad hoc reconstruction by callers, source-ID ownership, manifest-first resolution, extraction/current-state identity, destination normalization, stable ordering, typed missing-state errors, symlink/traversal defenses, and duplicate policy across config/stages/app. |
| Restore | `internal/app/restore*.go`, `internal/previouscache`, `internal/audio` | Remote authority, confined mapping, deterministic plan actions, local conflict and force behavior, dry-run purity, temp-file installation, manifest-last ordering, partial failure/retry behavior, report accuracy, cache reuse, and shared current-state mechanics. |
| File operations | `internal/fileops`, `internal/pathsafe`, local-store code in `internal/artifacts` | Atomic-write and promotion guarantees, permissions, close/sync/rename error handling, temp cleanup, same-filesystem assumptions, replacement policy, regular-file-only traversal, symlink and root-swap resistance, lock cleanup, and portability. |
| External adapters and storage | `internal/adapters`, `internal/audio` | Transport isolation, shared subprocess mechanics versus adapter-specific policy, command/config duplication, quoting and working directories, timeouts/cancellation, stdout/stderr separation, HTTP body and retry behavior, S3 pagination/streaming/not-found mapping, credential independence, and external error adaptation. |
| Shared models and diagnostics | `internal/artifactmodel`, `internal/contracts`, `internal/logging` | Serialization and validation invariants, unnecessary conversions, ownership of shared types, stable diagnostic structure, redaction, and whether small shared packages remain cohesive. |
| Tests, examples, and automation | all `*_test.go`, `examples/`, `.woodpecker/` | Risk ownership, semantic duplication, fixture cost, policy-coupled assertions, realistic boundary tests, end-to-end sufficiency, default-suite isolation, example validation, diagnostic coverage, flakiness, runtime cost, and enforcement of canonical validation. |
## Narratio-Specific Cross-Boundary Scenarios
In addition to package-local review, trace these complete scenarios because a
modular pipeline can look correct within every package while violating an
end-to-end invariant:
1. A stage succeeds, its result becomes non-resumable, the rerun fails, and a
later invocation decides what remains usable.
2. An upstream forced or changed outcome interacts with already-succeeded,
self-skipped, and disabled downstream stages.
3. Extraction produces a valid immutable bundle, then configuration or
transitive Notarius inputs change before analyze or publish.
4. Previous-session state is published, restored or prepared into the local
cache, and consumed by analyze without an unintended remote read.
5. Publish fails at each upload boundary, especially between current manifest
and current-run pointer, followed by status, restore, and retry.
6. Restore encounters identical files, conflicting files, unsafe remote keys,
cache hits, and a failure immediately before manifest installation.
7. Automatic or manual cleanup is requested after skipped, failed, locked,
partially uploaded, and fully committed publish outcomes.
8. Cancellation reaches bounded transcription work, HTTP requests,
subprocesses, object storage, and manifest reporting without leaks or false
success.
9. A configured artifact is disabled, unselected, reused, generated from
another artifact, sourced from extraction, or sourced from a previous
session, then filtered for publish.
10. The same session is invoked concurrently, including lock contention and
cleanup/release failures.
## Completion Criteria
The audit is complete when:
- every area in the inspection map has been reviewed against its canonical
contracts and focused tests;
- the stage lifecycle matrix and cross-boundary scenarios have explicit
conclusions;
- duplication candidates have been classified rather than merely counted;
- simplification and performance recommendations explain their correctness
constraints and expected benefit;
- comment recommendations identify the non-obvious rationale to preserve;
- the test suite has a risk-based sufficiency assessment, including gaps,
redundancy, durability, execution properties, and automation;
- findings are deduplicated, evidence-backed, and ranked by risk and dependency;
and
- unresolved questions and intentionally accepted risks are recorded rather
than silently omitted.
## Execution
The [Audit Sequence](audit-sequence.md) is the canonical owner of execution
order, stage boundaries, checkpoints, validation, and audit deliverables. This
document remains the canonical owner of audit scope, review criteria, and the
finding standard.

View File

@@ -0,0 +1,734 @@
# Codebase Audit Sequence
Status: proposed
## Purpose And Relationship To The Audit Plan
This document turns the [Codebase Audit Plan](audit-plan.md) into a bounded,
execution-ready sequence. The plan owns scope, review criteria, and the finding
standard. This document owns ordering, dependencies, working records,
validation, and exit gates.
The sequence is for investigation only. Do not mix production refactors or bug
fixes into the audit. A confirmed urgent defect may justify stopping to request
a separate remediation change, but its fix is not part of this sequence.
## Audit Run Records
Create `docs/roadmap/audit-findings.md` when the audit begins. It is the single
working ledger and final audit report. Initialize it with:
- the audited revision, branch/worktree state, Go version, platform, and audit
date;
- baseline command results and timings;
- an area coverage ledger;
- the stage lifecycle matrix;
- the cross-boundary scenario matrix from the audit plan;
- a risk-to-test matrix;
- candidate and confirmed finding registers; and
- unresolved questions, accepted risks, and final conclusions.
Track each execution stage in the coverage ledger with one of `not_started`,
`in_progress`, `complete`, or `blocked`. For a completed stage, record:
- contracts, packages, files, and important symbols reviewed;
- graph traces, commands, tests, or other evidence used;
- finding and candidate IDs produced;
- explicit no-finding conclusions for reviewed high-risk behavior; and
- follow-up questions assigned to later stages.
Use stable finding IDs with these prefixes:
| Prefix | Category |
| --- | --- |
| `COR` | Confirmed correctness defect |
| `RSK` | Correctness or operational risk |
| `ARC` | Ownership or architectural-boundary issue |
| `DUP` | Duplicated mechanism or policy |
| `SIM` | Simplification or idiomatic-Go opportunity |
| `EFF` | Efficiency or resource-use issue |
| `COM` | Missing, misleading, or stale explanatory comment |
| `TST` | Test-suite gap, redundancy, brittleness, or execution issue |
Candidate IDs remain candidates until manual inspection confirms the behavior,
contract, realistic scenario, and affected callers. Rejected candidates remain
in a short classification log so later stages do not reopen them without new
evidence.
## Execution Rules
1. Pin the audit to the revision recorded in Stage 0. If the worktree or HEAD
changes, record the change and rerun every affected stage; do not silently
combine evidence from different implementations.
2. Use codebase graph search and call/data-flow traces before broad source
search. Read the exact implementation, focused tests, and canonical contract
before confirming a finding.
3. Record test-policy observations during every behavior pass. Stage 12 owns the
suite-wide conclusion but must not rediscover the suite from scratch.
4. Record cross-area observations as candidates for the stage that owns the
conclusion. Avoid producing duplicate findings from several review passes.
5. Treat baseline failures as evidence, not automatic blockers. Continue when
read-only inspection remains sound, and state the limitation. Stop only when
the repository cannot be identified, required sources are unavailable, or a
failure makes later evidence unreliable.
6. Do not exercise a suspected destructive, credentialed, paid, or live-service
path merely to prove a defect. Use source reasoning, existing safe fakes, or
a narrowly controlled offline reproduction.
7. Escalate a credible active data-loss, secret-exposure, or unsafe-cleanup
defect immediately. Preserve the evidence and do not wait for final
synthesis before reporting it.
8. A stage is complete only when its exit gate is met. A package test passing is
evidence, not proof that the review is complete.
## Sequence Overview
| Stage | Focus | Depends on | Primary result |
| --- | --- | --- | --- |
| 0 | Pin revision and establish baseline | None | Reproducible audit record |
| 1 | Contract, boundary, and lifecycle map | 0 | Review matrices and ownership map |
| 2 | Runner and manifest state machine | 1 | Lifecycle and dual-ledger conclusions |
| 3 | Paths, artifacts, and filesystem safety | 1-2 | State/path authority and mutation conclusions |
| 4 | Publish, remote commit, and cleanup | 2-3 | Commit-boundary and destructive-operation conclusions |
| 5 | Restore and remote/previous state | 2-4 | Restore authority and recovery conclusions |
| 6 | Configuration and application composition | 1-5 | Validation and wiring conclusions |
| 7 | External adapters and shared support | 3, 6 | Boundary, cancellation, and resource conclusions |
| 8 | Prepare and transcript-processing stages | 2-3, 6-7 | Ordinary stage-contract conclusions |
| 9 | Extraction vertical slice | 2-3, 6-7 | Promotion, provenance, and resume conclusions |
| 10 | Analyze and artifact dependency slice | 3, 6, 8-9 | Dependency and source-resolution conclusions |
| 11 | Cross-codebase duplication, simplicity, efficiency, and comments | 2-10 | Classified maintainability candidates |
| 12 | Test-suite policy audit | 2-11 | Risk-based suite sufficiency assessment |
| 13 | Synthesis and audit closeout | 0-12 | Final deduplicated audit report |
Stages are intentionally ordered. Later stages may resolve candidates raised by
earlier ones, but they must not invalidate an earlier stage silently. Return to
the owning stage, update its coverage record, and note the new evidence.
## Stage 0: Pin Revision And Establish Baseline
### Entry
- Repository root and `docs/development.md` are available.
- The audit plan and canonical policy documents can be read.
### Execute
1. Record `git rev-parse HEAD`, branch/detached state, `git status --short`,
`go version`, `go env GOOS GOARCH`, and the current date.
2. Confirm that the code knowledge graph represents the recorded repository and
revision; refresh the index if it is missing or stale.
3. Capture the package/file/test inventory, entry points, architecture
boundaries, high fan-in symbols, complexity signals, and similarity signals.
4. Run the default offline baseline and record wall time and failures:
```sh
go test -count=1 ./...
go test -race -count=1 ./...
go vet ./...
```
5. Build into an external temporary directory so validation does not add a
workspace binary:
```sh
audit_build_dir="$(mktemp -d)"
go build -o "$audit_build_dir/narratio" ./cmd/narratio
go test -coverprofile="$audit_build_dir/coverage.out" ./...
```
6. Inventory the repository's CI/release validation, maintained examples, fuzz
tests, golden data, opt-in tests, and generated-test update mechanisms.
### Output
- Baseline and inventory sections in `audit-findings.md`.
- Initial coverage ledger containing Stages 0-13.
- Unconfirmed metric-driven candidates, clearly labeled as such.
### Exit Gate
- Revision and environment are reproducible.
- Every baseline command has a recorded result.
- Graph freshness is known.
- Any limitation that affects later stages has an owner and disposition.
## Stage 1: Build The Contract, Boundary, And Lifecycle Map
### Entry
- Stage 0 is complete.
### Execute
1. Read the architecture, internal overview, testing policy, focused internal
documents, and the relevant CLI/configuration/operations/integration
contracts using the development guide's routing rules.
2. Map each package and important interface to its owned policy. Mark every
cross-package dependency that appears to reverse or blur the intended
direction for later confirmation.
3. Build a stage-contract matrix with canonical order, declared inputs,
outputs, configuration, adapters, skip behavior, resume validation,
materialization boundary, manifest effects, and downstream invalidation.
4. Build the lifecycle matrix required by the audit plan: first run,
already-succeeded skip, self-skip, failure, interruption, forced replacement,
non-resumable result, and successful rerun.
5. Assign each of the ten cross-boundary scenarios in the audit plan to its
primary execution stage and list supporting packages/tests.
6. Seed the risk-to-test matrix with the intended test owner for each
architectural invariant. Do not judge sufficiency yet.
### Output
- Package ownership, stage-contract, lifecycle, scenario, and preliminary
risk-to-test matrices.
- `ARC` and `RSK` candidates for apparent disagreements, without deciding from
documentation alone which artifact is wrong.
### Exit Gate
- Every area in the audit plan's inspection map has an assigned stage.
- Every architectural invariant has an implementation owner and intended test
owner.
- Unknown or contradictory contracts are explicitly recorded.
## Stage 2: Audit The Runner And Manifest State Machine
### Entry
- Stage 1 matrices are complete.
### Execute
1. Trace the entry paths into full-run and single-stage execution through
`internal/app/planner.go`, `runner.go`, `run_stage.go`, and related helpers.
2. Inspect `internal/manifest` models, validation, session/run creation,
loading, normalization, atomic saves, and all transition methods.
3. Walk every lifecycle-matrix cell through both manifests. Verify clearing
versus retention of outputs, diagnostics, generated configuration, metadata,
errors, actions, timestamps, and downstream state.
4. Reason about failures before and after each session-manifest and run-manifest
save. Determine which disagreement states are possible and how a later
invocation interprets them.
5. Review force, changed-result, self-skip, failed-result, and non-resumable
invalidation separately. Confirm behavior at the first and last canonical
stage.
6. Review session lock acquisition/release and concurrent invocation behavior,
while leaving path implementation details to Stage 3.
7. Classify the runner's complexity and repeated session/run persistence paths:
state-machine clarity, justified explicitness, candidate local helpers, and
comments that preserve ordering rationale.
8. Review focused app/manifest tests against the matrix and add observations to
the risk-to-test ledger.
### Validation
```sh
go test -count=1 ./internal/app ./internal/manifest
go test -race -count=1 ./internal/app ./internal/manifest
```
### Exit Gate
- Every lifecycle cell has a source-backed conclusion for both manifests.
- Cross-boundary scenarios 1, 2, and the lock portion of 10 are resolved or
carry explicit questions.
- All runner/manifest candidates are confirmed, rejected, or assigned to a
named later stage.
## Stage 3: Audit Paths, Artifacts, And Filesystem Safety
### Entry
- Stages 1-2 are complete.
### Execute
1. Review `internal/artifacts`, `internal/artifactpolicy`, `internal/pathsafe`,
`internal/fileops`, and local-store filesystem code.
2. Inventory canonical path and key helpers, then search callers for ad hoc
reconstruction, double normalization, mixed slash/filesystem semantics, or
policy implemented outside its owner.
3. Trace built-in, configured, extraction, previous-session, and current-state
artifact resolution. Verify identity, checksum, contract, provenance,
deterministic ordering, and typed missing-state behavior.
4. Review atomic file writes, copies, directory promotion, temp cleanup,
permission preservation, close/sync/rename errors, existing-destination
behavior, same-filesystem assumptions, and platform sensitivity.
5. Walk traversal, absolute path, broad root, symlink component, inspected-root
replacement, non-regular file, and time-of-check/time-of-use scenarios.
6. Confirm that low-level file/storage helpers receive explicit destinations
and do not infer stage, campaign, session, run, or publish policy.
7. Inspect lock-file implementation and cleanup errors to finish scenario 10.
8. Record focused test ownership and gaps without duplicating Stage 2's state
conclusions.
### Validation
```sh
go test -count=1 ./internal/artifacts ./internal/artifactpolicy ./internal/pathsafe ./internal/fileops
go test -race -count=1 ./internal/artifacts ./internal/fileops
```
### Exit Gate
- Every canonical path/key family has one identified owner.
- Every material filesystem mutation has documented confinement and atomicity
conclusions.
- Scenario 10 is resolved.
- Safety checks that appear repetitive are classified before any simplification
recommendation is made.
## Stage 4: Audit Publish, Remote Commit, And Cleanup
### Entry
- Stages 2-3 are complete.
### Execute
1. Trace publish from stage selection through object-store calls, manifest
metadata, commit-marker publication, run completion, and post-publish
cleanup.
2. Verify prerequisite stage-state checks, selected/configured/extraction
output resolution, required versus optional outputs, static and remote
locks, run-file exclusions, previous-cache inclusion, and deterministic
upload order.
3. Enumerate failures before and after every upload. Prove that
`current/run_id.txt` is written last and is the only remote-current commit
point.
4. Review retry/idempotency behavior, existing remote objects, partial uploads,
pointer/manifest disagreement, and status/restore interpretation after each
partial outcome.
5. Trace automatic and manual cleanup gates. Confirm publish execution,
`uploaded`, `current_pointer_written`, explicit policy, and confined targets
are all required at the correct boundary.
6. Confirm that `--force` cannot override publish locks or cleanup safety.
7. Review duplication between publish planning, artifact destination policy,
operator views, and cleanup metadata only after ownership is established.
### Validation
```sh
go test -count=1 ./internal/stage ./internal/app ./internal/artifacts ./internal/adapters/storage
```
### Exit Gate
- Cross-boundary scenarios 5 and 7 are resolved for every relevant failure
boundary.
- Remote-current authority and local-cleanup eligibility have explicit truth
tables.
- Publish findings distinguish stage policy from storage mechanics.
## Stage 5: Audit Restore And Remote/Previous State
### Entry
- Stages 2-4 are complete.
### Execute
1. Trace restore discovery, planning, execution, reporting, audio
materialization, and previous-cache planning through `internal/app`,
`internal/artifacts`, `internal/previouscache`, `internal/audio`, and storage.
2. Confirm remote pointer/manifest identity and campaign/session/run authority,
including missing and inconsistent current state.
3. Verify remote-to-local confinement, deterministic action ordering,
`download`/`skip_same`/`conflict` decisions, force semantics, and dry-run
purity.
4. Walk failures during download, checksum or manifest validation, atomic
install, report persistence, and the manifest-last boundary. Record the
intentional lack of rollback and retry consequences.
5. Review audio spool/cache identity, cache-hit verification, partial download
behavior, and duplicate remote/filesystem work.
6. Review previous-session requirement planning, required/optional behavior,
identity checks, published-path fallback, and deterministic local mapping.
7. Confirm which mechanics are shared with status/validate/operator commands
and which caller-specific missing-state policies must remain separate.
### Validation
```sh
go test -count=1 ./internal/app ./internal/previouscache ./internal/audio ./internal/artifacts ./internal/adapters/storage
```
### Exit Gate
- Cross-boundary scenarios 4 and 6 are resolved through retry/recovery.
- Restore authority, manifest-last installation, and partial-write behavior are
explicit.
- Previous-cache conclusions are ready for the prepare and analyze passes.
## Stage 6: Audit Configuration And Application Composition
### Entry
- Stage 1 is complete and Stages 2-5 have identified the policies that
configuration and composition must supply.
### Execute
1. Review `internal/config`, `cmd/narratio`, application command dispatch,
configuration selection, secret-file environment loading, and production
collaborator construction.
2. Trace discovery, precedence, strict YAML decoding, defaults, empty values,
normalization, session templating, and validation order across pipeline,
campaign, and session configuration.
3. Verify cross-field constraints for stage enablement, paths, timeouts,
concurrency, artifacts, Notarius, Scriptorium, publish, storage, cleanup,
audio, and previous-session behavior.
4. Compare validation logic with maintained examples and the public
configuration contract. Record contract drift rather than silently choosing
code or docs.
5. Check that filesystem secrets are loaded before the boundary that consumes
them and are excluded from logs, manifests, reports, generated files, and
errors.
6. Review conditional construction of expensive/external collaborators and
cleanup of anything with a lifecycle. Confirm test injection cannot create a
behavior different from production composition.
7. Classify repeated validators, path checks, timeout parsing, constructor
wrappers, and single-stage command wrappers by policy owner.
### Validation
```sh
go test -count=1 ./internal/config ./internal/app ./cmd/narratio
go vet ./...
```
### Exit Gate
- Every operator-visible field used by audited behavior has a traced default,
normalization, validation, and consumer.
- Composition conclusions cover enabled and disabled stages without requiring
live services or credentials.
- Maintained examples have an explicit validity conclusion.
## Stage 7: Audit External Adapters And Shared Support
### Entry
- Stages 3 and 6 are complete.
### Execute
1. Review `internal/adapters`, `internal/audio`, `internal/logging`,
`internal/contracts`, and `internal/artifactmodel` at their public package
boundaries.
2. For each HTTP, subprocess, notification, and object-storage adapter, compare
implementation with its integration contract and trace all production
callers.
3. Verify context cancellation, timeout ownership, process termination and
waiting, goroutine/channel closure, HTTP response-body closure, retries,
malformed responses, streaming, pagination, not-found mapping, and local
file cleanup.
4. Confirm command argument construction, working directory, environment,
generated configuration, stdout/stderr separation, output validation, and
external error adaptation stay inside the owning adapter.
5. Compare subprocess implementations to the shared subprocess package. Classify
repeated constructor/config/log/output mechanics separately from
adapter-specific protocol policy.
6. Review fakes for realistic state and concurrency behavior, but defer their
suite-wide value judgment to Stage 12.
7. Check shared models for avoidable conversions, stable serialization,
validation ownership, and redaction-sensitive diagnostic fields.
### Validation
```sh
go test -count=1 ./internal/adapters/... ./internal/audio ./internal/logging ./internal/contracts ./internal/artifactmodel
go test -race -count=1 ./internal/adapters/... ./internal/audio
```
### Exit Gate
- Every external resource has an explicit acquisition, cancellation, and
release conclusion.
- Transport types and protocol policy have not leaked into stages.
- Adapter duplication candidates identify the correct shared or specific
owner.
## Stage 8: Audit Prepare And Transcript-Processing Stages
### Entry
- Stages 2-3 and 6-7 are complete.
### Execute
1. Review `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, and
`render` as vertical slices from resolved configuration and manifest input
through adapter call, run-local output, validation, canonical
materialization, and recorded result.
2. Verify each implementation against the Stage 1 contract matrix and focused
internal document. Record any undeclared input, output, diagnostic, config,
adapter, or skip/failure behavior.
3. For prepare, confirm local/S3 exclusivity, stable input copying,
previous-cache clearing/hydration, and deterministic manifest input records.
4. For transcribe, confirm unique speaker identities, bounded runtime
concurrency, cancellation, deterministic result ordering, adapter-returned
path identity, and partial failure behavior.
5. For transformation/render stages, confirm manifest-first resolution,
run-local paths, schema/report validation, disabled/default behavior,
canonical promotion, and diagnostic-versus-artifact classification.
6. Compare similar stage implementations for shared mechanisms only after
listing meaningful differences. Avoid a generic stage framework.
7. Add stage-focused test ownership, gaps, and redundancy candidates to the
risk-to-test matrix.
### Validation
```sh
go test -count=1 ./internal/stage ./internal/audio ./internal/previouscache ./internal/adapters/whisperx ./internal/adapters/seriatim ./internal/adapters/audita ./internal/adapters/scriptorium
go test -race -count=1 ./internal/stage ./internal/audio
```
### Exit Gate
- Every reviewed stage has a completed contract-matrix row.
- Cross-boundary scenario 8 is resolved for transcription and subprocess-backed
transformation stages.
- Similarity candidates are classified as intentional explicitness, local
helper candidates, or shared-owner findings.
## Stage 9: Audit The Extraction Vertical Slice
### Entry
- Stages 2-3 and 6-7 are complete.
### Execute
1. Trace extraction from configuration validation and composition through
transcript resolution, invocation fingerprint, Notarius execution, receipt
and lane validation, directory promotion, manifest recording, catalog
hydration, resume validation, analyze, and publish consumers.
2. Verify run-local isolation, regular-file and confined-index requirements,
required-lane policy, contract/provenance construction, checksum timing,
immutable destination identity, and no-replacement promotion.
3. Enumerate failures before and after subprocess completion, receipt parsing,
payload inspection, promotion, and manifest persistence. Determine what
remains diagnostic, durable, advertised, and reusable.
4. Walk every resume validation branch. Distinguish obsolete/missing outcomes
that trigger rerun from unsafe conditions that must stop execution.
5. Evaluate the fingerprint's intentionally observable and unobservable inputs
against documentation and force guidance.
6. Review the dense validation code for named sub-decisions and comments while
preserving the visible security proof and check ordering.
7. Confirm focused tests cover immediate reuse, cross-invocation reuse,
configuration change, payload tampering, provenance mismatch, symlinks/root
replacement, failure residue, and downstream invalidation at the correct
layers.
### Validation
```sh
go test -count=1 ./internal/stage ./internal/artifacts ./internal/fileops ./internal/adapters/notarius ./internal/app
```
### Exit Gate
- Cross-boundary scenario 3 is resolved, including transitive-input limits.
- Promotion, advertisement, and resume each have a distinct authority and
failure conclusion.
- Every proposed simplification states which security or compatibility checks
it preserves.
## Stage 10: Audit Analyze And Artifact Dependencies
### Entry
- Stages 3, 6, 8, and 9 are complete.
### Execute
1. Trace all analyze source families from configuration validation through
runtime catalog registration, availability, resolution, Scriptorium
execution/reuse, materialization, metadata, and publish selection.
2. Verify enabled, selected, executable, reused, generated, and unavailable
states are distinct and deterministic.
3. Review configured-artifact dependency validation and runtime topological
ordering for cycles, missing dependencies, stable ordering, and consistency
between configuration and execution.
4. Confirm required/optional behavior and guidance for built-in transcripts,
prepared stable inputs, configured artifacts, extraction sources, and
previous-session sources.
5. Prove previous-session resolution is local-only during analyze and that
disabled artifacts are reused only under the documented conditions.
6. Inspect repeated resolution branches, parameter width, nested lookup, and
ordering work for a smaller representation or indexed plan without merging
distinct source policies.
7. Review tests for each state transition and source family at the narrowest
stable owner, noting semantic duplication across config, artifacts, stage,
publish, and assembled runner tests.
### Validation
```sh
go test -count=1 ./internal/stage ./internal/artifacts ./internal/artifactpolicy ./internal/config ./internal/adapters/scriptorium ./internal/app
```
### Exit Gate
- Cross-boundary scenario 9 is resolved for every source family and selection
state.
- Dependency ordering and source availability have explicit determinism and
complexity conclusions.
- Config, artifact-policy, catalog, stage, and publish ownership is unambiguous
or represented by an `ARC` finding.
## Stage 11: Audit Duplication, Simplicity, Efficiency, And Comments
### Entry
- Behavior stages 2-10 are complete, so structural candidates can be judged
against known contracts.
### Execute
1. Rerun graph similarity, complexity, fan-in/fan-out, call-path, loop-depth,
scan-in-loop, and change-coupling analyses on production code. Add targeted
text/static searches for patterns the graph cannot represent.
2. Revisit all `DUP`, `SIM`, `EFF`, and `COM` candidates collected earlier.
Search for additional occurrences and trace all callers before assigning an
owner.
3. For duplication, classify coincidental syntax, shared mechanism, duplicated
policy, or deliberately explicit security/state logic. Propose only the
narrowest helper that improves ownership and comprehension.
4. For complexity, sketch the smaller control flow or data model and verify it
leaves state transitions, validation order, and commit boundaries visible.
5. For efficiency, state the input scale or call frequency, current and proposed
complexity/I/O behavior, expected benefit, and benchmark or measurement
needed. Reject micro-optimizations without a credible workload.
6. Review standard-library usage, errors, slices/maps, allocations, copying,
sorting, serialization, filesystem passes, adapter initialization, remote
calls, goroutines/channels, and interface breadth across the complete codebase.
7. Review comments only after simplification decisions. Recommend why-comments
for remaining invariants, compatibility limits, safety checks, partial
failure, and ordering; flag comments that restate code or no longer match it.
8. Check dependencies and platform assumptions for clear correctness,
portability, complexity, or maintenance consequences.
### Validation
- Run focused package tests for any behavior used to disprove or confirm a
candidate.
- Run existing benchmarks where relevant. Propose a benchmark rather than
inventing performance claims when representative measurement is absent.
### Exit Gate
- Every structural candidate is confirmed, rejected with a reason, or merged
into a stronger root-cause finding.
- No helper recommendation creates a generic workflow abstraction or moves
policy into a low-level utility.
- Every efficiency finding has a credible workload and validation method.
- Every comment finding states the non-obvious rationale that should be
preserved.
## Stage 12: Audit The Test Suite Against Policy
### Entry
- Stages 2-11 have populated the risk-to-test matrix and test observations.
### Execute
1. Complete the risk-to-test matrix. For every consequential invariant, list
the current tests, proper owner, protected defect, missing failure modes, and
overlap with other layers.
2. Review tests by behavior cluster rather than filename: parsing/validation,
domain/state, filesystem, adapters, orchestration, CLI, integration, and
representative assembled workflows.
3. Classify gaps for data integrity, destructive operations, compatibility,
security, concurrency, idempotency, recovery, cancellation, and partial
success. Confirm the gap is not credibly protected elsewhere.
4. Classify redundancy and brittleness: private constants/defaults, exact error
wording, incidental formatting/paths, mock choreography, oversized
snapshots, helper-level duplication, and the same policy repeated across
layers.
5. Review doubles using the policy order: real deterministic collaborator,
stateful fake, stub, then mock when interaction is contractual. Check that
fakes model the failure and state semantics used by the tests.
6. Inspect test helpers and large test functions for simplification and
meaningful table-driven boundaries without creating a fixture framework
whose maintenance cost exceeds its value.
7. Review determinism and isolation: credentials, network access, paid APIs,
environment, working directory, clocks, randomness, ports, temp paths,
process-global state, ordering, cleanup, and parallel execution.
8. Use coverage to investigate consequential weak branches, not as a score.
Review heavily covered behavior for marginal-value duplication as well.
9. Identify focused fuzz opportunities for parsers, YAML/JSON normalization,
source IDs, confined paths, remote/local mapping, and manifest decoding.
10. Compare local requirements with `.woodpecker/` and other automation. Record
missing enforcement as a risk/cost decision, not an assumption that every
diagnostic command belongs in CI.
11. Investigate order dependence and flakiness with bounded runs, recording
runtime and any reproducible seed:
```sh
go test -shuffle=on -count=3 ./...
go test -race -shuffle=on -count=1 ./...
```
### Exit Gate
- Every important risk has a sufficiency conclusion and one intended test
owner.
- Every proposed test addition names the realistic defect and marginal value.
- Every deletion/consolidation names the stronger remaining protection.
- Default-suite determinism, offline behavior, runtime, flakiness, and CI
enforcement have explicit conclusions.
## Stage 13: Synthesize And Close The Audit
### Entry
- Stages 0-12 meet their exit gates or have explicitly accepted limitations.
### Execute
1. Reconcile candidates and findings across stages. Merge shared root causes and
remove repeated symptoms while retaining all affected locations and
contracts.
2. Recheck every confirmed finding against current source, callers, tests, and
canonical documentation. Downgrade or reject anything supported only by a
metric or hypothetical preference.
3. Rank impact, likelihood, confidence, and remediation scope separately. Order
the recommended backlog by dependency: correctness/data safety first,
architectural ownership next, then simplification/duplication, tests,
efficiency, and comments where they remain necessary.
4. Record positive conclusions for high-risk areas where the current design and
tests are sufficient. The report should not imply that only defective areas
were reviewed.
5. Reconcile the area coverage ledger, lifecycle matrix, cross-boundary scenario
matrix, and risk-to-test matrix with the audit plan's completion criteria.
6. Record any accepted risks, ambiguous contracts, environmental limitations,
and deferred investigations with an explicit rationale and owner.
7. Check whether HEAD or the worktree changed since Stage 0. Rerun affected
stages or clearly pin the report to the original revision.
8. Validate the report and roadmap document links and run `git diff --check`.
If implementation changed during the audit, rerun the full Stage 0 validation
baseline against the final audited revision.
### Final Deliverable
`docs/roadmap/audit-findings.md` must contain:
- an executive assessment without unsupported quality scores;
- the audited revision and validation baseline;
- coverage and scenario completion summaries;
- confirmed findings ordered by dependency and risk;
- rejected candidate themes where their recurrence would otherwise waste work;
- the test-suite sufficiency assessment;
- positive conclusions and accepted risks; and
- a recommended remediation order, without implementing the remediation.
### Exit Gate
- Every completion criterion in the audit plan is satisfied or explicitly
marked limited with rationale.
- Every finding is evidence-backed, deduplicated, actionable, and assigned a
stable ID.
- No production change is included in the audit output.
- The report is sufficient to prepare a separate remediation sequence without
repeating discovery.

104
docs/roadmap/audit.md Normal file
View File

@@ -0,0 +1,104 @@
# Implementation Plan Summary
This is a concise, plain-language summary of
[`implementation.md`](implementation.md). It describes the intended real-world
outcome of each stage without reproducing its implementation details.
1. **Stage 1 — Collaborative workspace permissions:** Treat ordinary Narratio and
Notarius data as shareable and make managed workspaces group-writable, while
retaining private handling for API keys.
2. **Stage 2 — Safe identifiers:** Reject campaign, session, run, artifact, and
source identifiers that could escape their intended filesystem namespace, and
use fuzz tests to cover platform-specific path tricks.
3. **Stage 3 — Durable file replacement:** Consolidate duplicated atomic-write
functions into one shared mechanism that fully persists a replacement before
reporting success.
4. **Stage 4 — Confined writes and downloads:** Rewrite destination mutations so
symlinks or concurrent directory replacement cannot redirect writes,
promotions, or downloads outside the intended root.
5. **Stage 5 — Safe deletion and crash-recoverable locks:** Confine recursive
cleanup to its authorized root and replace stale lock-file existence checks
with operating-system locks released automatically after process death.
6. **Stage 6 — Protected API-key reads:** Read API keys only from private,
bounded, regular files without following symlinks or exposing key material in
errors.
7. **Stage 7 — Bounded external results:** Prevent external adapters from causing
memory or disk exhaustion by validating regular result files and enforcing
generous, clearly reported per-adapter size limits.
8. **Stage 8 — Complete subprocess termination:** Ensure cancellation, timeout,
or a safety-limit failure terminates and reaps an external command's entire
process tree rather than only its parent process.
9. **Stage 9 — Safe subprocess diagnostics:** Redact known credentials from
stdout/stderr, cap persisted diagnostics, and terminate runaway producers when
those caps are reached.
10. **Stage 10 — Safe publish inputs:** Ensure publishing reads and uploads only
verified regular files declared within the selected run, even during
filesystem races.
11. **Stage 11 — Consistent run identity:** Resolve one campaign/session/run
identity for an invocation, reject conflicting authorities, and prevent stale
identity fields from leaking into a later run.
12. **Stage 12 — Reliable failure recording:** Consolidate terminal-failure
persistence so handled errors reliably update authoritative session state and
preserve any secondary persistence failures.
13. **Stage 13 — Immutable remote-state model:** Define a versioned immutable
remote snapshot selected by a small pointer, while isolating old-format read
compatibility so it can be removed after migration.
14. **Stage 14 — Transactional publication:** Upload and verify a complete
immutable snapshot before one final pointer change makes it current, using an
exact source-to-destination mapping instead of basename guesses.
15. **Stage 15 — Safe remote locking and pagination:** Use provider-enforced
conditional writes so publishers cannot overwrite another owner's lock, and
fail instead of looping when object-store pagination stops making progress.
16. **Stage 16 — Retryable cleanup:** Persist post-publication cleanup as a
durable obligation so interrupted or failed deletion is retried and never
mistaken for completed cleanup.
17. **Stage 17 — Snapshot-consistent restore:** Make restore and status use one
selected immutable snapshot throughout the operation, and prevent `--force`
from overwriting unsafe directory or non-file conflicts.
18. **Stage 18 — Race-safe, portable restore:** Serialize restore against runner
reuse, leave durable evidence of incomplete restores, and replace unsafe
producer-machine absolute paths with validated local references.
19. **Stage 19 — Correct audio-cache reuse:** Reuse downloaded audio only when its
local bytes and recorded metadata match the selected remote object version.
20. **Stage 20 — One previous-session resolver:** Give restore, prepare, run, and
dry-run one consistent view of required and optional previous-session inputs,
while avoiding ambiguous matches and duplicate downloads.
21. **Stage 21 — Strict configuration:** Reject multiple YAML documents, invalid
durations, implicit storage backends, and unmet previous-session expectations,
while separating configuration tests by responsibility.
22. **Stage 22 — Truthful product settings and temp-file ownership:** Remove
configuration fields that do nothing, reject unsupported notification
settings, and guarantee cleanup of remote-configuration temporary files.
23. **Stage 23 — Streaming WhisperX transport:** Stream uploads instead of
buffering entire audio files, reject unsupported endpoint schemes, and make
retries, cancellation, fake-server recording, and race tests reliable.
24. **Stage 24 — Correct prepare/transcribe transitions:** Prevent stale previous
data, cancelled or partial transcription work, and duplicate source identity
from being recorded as successful current output.
25. **Stage 25 — Authoritative output paths:** Require adapters to honor the
stage-requested output destination and consolidate duplicate singleton
transcript resolution without confusing it with multi-source discovery.
26. **Stage 26 — Shared extraction evidence:** Consolidate duplicated
extraction-bundle validation into one typed proof while allowing resume and
catalog consumers to apply their distinct policies.
27. **Stage 27 — Transcript-aware extraction reuse:** Include the direct
transcript's identity in extraction freshness checks so changed input cannot
reuse stale structured artifacts.
28. **Stage 28 — One effective artifact selection:** Resolve configured and
explicitly selected artifacts once, then use that same typed set for
prerequisites, extraction catalogs, analyze inputs, and execution planning.
29. **Stage 29 — Predictable analyze planning:** Represent optional and required
analyze inputs explicitly, produce deterministic dependency errors, and give
operators correct remediation commands.
30. **Stage 30 — Contract cleanup:** Remove dead or misleading interfaces and
helpers, move static Audita configuration to its proper owner, and correct
stale contract comments.
31. **Stage 31 — Enforced automated validation:** Require tests, race checks, vet,
builds, and example validation for changes and releases, while consolidating
redundant broad tests without losing focused coverage.
32. **Stage 32 — Documentation and closure:** Reconcile normative documentation
with the completed behavior and verify that every planned remediation has one
completed, traceable implementation stage.
Every stage's purpose was readily determinable from the implementation plan; no
stage required an uncertainty note.

File diff suppressed because it is too large Load Diff

View File

@@ -24,6 +24,8 @@ Safe fix:
- pass explicit `--config`, `--campaign` or `--campaign-file`, and `--session`.
Relevant reference: [Configuration discovery](./config.md#discovery-and-selection).
## Session template placeholders rejected
Symptom:
@@ -44,6 +46,8 @@ Safe fix:
- generate concrete session YAML with `narratio session init`.
Relevant reference: [Operations: Session Initialization](./operations.md#session-initialization).
## Strict decode or schema validation failure
Symptom:
@@ -62,7 +66,10 @@ narratio session plan 2026-04-04 --config /path/pipeline.yml --campaign-file /pa
Safe fix:
- align config with [docs/config.md](./config.md) and maintained files under `examples/`.
- align config with [Configuration](./config.md) and the
[maintained examples](../examples/README.md).
Relevant reference: [Configuration](./config.md).
## Audio mode conflict
@@ -74,10 +81,18 @@ Likely cause:
- configured both local and S3 session audio inputs.
Diagnostics:
```bash
narratio session validate 2026-04-04
```
Safe fix:
- use local mode (`audio_dir` or `audio_files`) or S3 mode (`audio_s3.prefix`), not both.
Relevant reference: [Session configuration](./config.md#session).
## `--artifacts` selection error
Symptom:
@@ -90,10 +105,159 @@ Likely causes:
- empty list entry (for example trailing comma);
- `run-stage` used with non-`analyze`/`publish` target.
Diagnostics:
```bash
narratio session artifacts 2026-04-04
```
Safe fix:
- provide only configured keys and use `--artifacts` with supported commands/stages.
Relevant reference: [CLI artifact selection](./cli.md).
## Notarius executable missing
Symptom:
- extraction fails while resolving or starting the Notarius executable.
Likely causes:
- `pipeline.notarius.binary` is not installed, executable, or on `PATH`;
- a configured executable path is wrong.
Safe fix:
- install a compatible Notarius release or correct the binary setting, then
rerun extraction.
Relevant references: [Notarius configuration](./config.md#notarius-output-entries)
and [Notarius integration](./integrations/notarius.md).
## Notarius exits nonzero
Symptom:
- extraction reports a Notarius exit error instead of a receipt.
Diagnostics:
- inspect `runs/{run_id}/extract/notarius.stderr.log`; stdout is reserved for
the receipt and is not merged with diagnostics.
Safe fix:
- correct the reported Notarius pipeline, input, provider, or configuration
failure and rerun extraction. Do not edit a staged output bundle into place.
After a failed replacement, an older immutable bundle may still exist even
though the current session manifest has no successful extraction payload. This
is expected audit state, not a signal to relink the old bundle manually.
Relevant reference: [Operations: Extraction Workflow](./operations.md#extraction-workflow).
## Atomic Notarius promotion unsupported
Symptom:
- extraction fails with `atomic no-replace directory promotion is unsupported`
before a durable bundle or temporary promotion tree is created.
Likely cause:
- Narratio is running on an operating system other than Linux, macOS, or
Windows, where the required atomic no-replace directory primitive has not
been implemented and verified.
Safe fix:
- run extraction on Linux, macOS, or Windows. Do not replace the atomic commit
with a manual copy or move; the session manifest must never observe a partial
or overwritten bundle.
This is an extraction-specific platform boundary, not a support statement for
unrelated Narratio workflows. See
[Operations: Extraction Workflow](./operations.md#extraction-workflow).
## Notarius receipt or index incompatible
Symptom:
- extraction rejects the receipt schema, pipeline identity, bundle/index path,
lane descriptor, or payload path even though Notarius exited successfully.
Likely causes:
- Narratio and Notarius versions disagree on their consumer contract;
- the configured pipeline or lane constraints are stale;
- output paths escape the bundle or traverse symlinks.
Safe fix:
- compare installed Notarius output with the canonical Notarius contracts,
including receipt `index_file: index.json` and index management names
`manifest.json`, `rejected.json`, and `warnings.json`; align
`pipeline.notarius` constraints and rerun. Do not bypass confinement or schema
checks.
Relevant reference: [Notarius integration](./integrations/notarius.md).
## Required Notarius lane rejected or missing
Symptom:
- extraction fails because a configured lane is rejected, missing, duplicated,
or incompatible, including after a zero exit.
Safe fix:
- inspect the Notarius diagnostic log and bundle rejection/warning information;
- correct the Notarius module or the exact declared lane contract;
- remove an output declaration only if downstream consumers genuinely no longer
require that source, then rerun extraction.
Every configured output is required. Narratio does not promote a partial result.
## Extraction resume invalidated
Symptom:
- a previously successful extraction runs again during ordinary continuation.
Likely causes:
- the executable/config path, pipeline ID, timeout, working directory, or
configured output contracts changed;
- the durable bundle, index, lane set, provenance, regular-file status, or
checksum no longer validates.
Safe fix:
- allow the automatic rerun after verifying the current configuration. Treat
an unsafe path or symlink error as filesystem corruption or tampering and
investigate it rather than replacing files manually.
## Notarius transitive configuration changed
Symptom:
- Notarius profiles, prompts, modules, imported files, or references changed,
but Narratio still considers the previous extraction resumable.
Safe fix:
```bash
narratio run-stage extract 2026-04-04 --force
```
Narratio fingerprints its invocation contract, not the contents of transitive
Notarius inputs. Always force extraction after changing them; downstream
successful stages are then marked stale normally.
Relevant reference: [Operations: Extraction Workflow](./operations.md#extraction-workflow).
## Previous-session artifact input missing
Symptom:
@@ -124,6 +288,8 @@ or rerun prepare after correcting session config:
narratio run-stage prepare 2026-04-04 --force
```
Relevant reference: [Operations: Restore Workflow](./operations.md#restore-workflow).
## Session lock conflict (`.lock`)
Symptom:
@@ -147,6 +313,8 @@ Safe fix:
- wait for active process completion;
- remove stale lock only after confirming no live process owns it.
Relevant reference: [Operations: Local State Layout](./operations.md#local-state-layout).
## Restore conflict without `--force`
Symptom:
@@ -168,6 +336,8 @@ Safe fix:
- review conflicts;
- rerun with `--force` only when remote state should overwrite local.
Relevant reference: [Operations: Restore Workflow](./operations.md#restore-workflow).
## Restore current-state discovery failure
Symptom:
@@ -191,6 +361,8 @@ Safe fix:
- resolve storage/auth issue;
- republish from healthy local state if current pointer is missing.
Relevant reference: [Operations: Publish Workflow](./operations.md#publish-workflow).
## Publish output failure
Symptom:
@@ -217,6 +389,8 @@ Safe fix:
- correct publish source/destination rules;
- retry after storage failure is resolved.
Relevant reference: [Publish configuration](./config.md#publish-configuration-summary).
## Render markdown source missing
Symptom:
@@ -232,7 +406,6 @@ Diagnostics:
```bash
narratio session status 2026-04-04
narratio run-stage render 2026-04-04 --force
```
Safe fix:
@@ -244,6 +417,8 @@ narratio run-stage render 2026-04-04 --force
narratio run-stage analyze 2026-04-04 --force
```
Relevant reference: [Operations: Stage Execution](./operations.md#stage-execution-and-continuation-behavior).
## Secrets or storage credential failure
Symptom:
@@ -269,6 +444,8 @@ Safe fix:
- provide required env vars;
- keep secret values out of YAML.
Relevant reference: [Secrets](./config.md#secrets-handling).
## S3 audio prepare failure
Symptom:
@@ -284,7 +461,7 @@ Likely causes:
Diagnostics:
```bash
narratio run-stage prepare 2026-04-04 --force
narratio session validate 2026-04-04
```
Safe fix:
@@ -292,6 +469,8 @@ Safe fix:
- verify prefix contents and storage access;
- keep session audio mode consistent.
Relevant reference: [Operations](./operations.md).
## References
- [docs/cli.md](./cli.md)

48
examples/README.md Normal file
View File

@@ -0,0 +1,48 @@
# Maintained Examples
These files are safe, copyable starting points for Narratio configuration and
input structure. Replace placeholder identifiers, storage names, integration
URLs, and paths for the target environment. Field meanings and defaults belong
in the [configuration reference](../docs/config.md).
## Pipeline Configuration
- [Minimal pipeline](pipeline.minimal.yml): campaign discovery plus the required
WhisperX URL.
- [Production-shaped pipeline](pipeline.production.yml): S3 storage, publish,
external tools, and configured Scriptorium artifacts.
- [Full annotated pipeline](pipeline.full.annotated.yml): every implemented
pipeline section with explanatory comments.
- [Extraction subset pipeline](pipeline.extraction-subset.yml): a focused
Scriptorium artifact consuming only three declared Notarius lanes.
The existing `internal/config` example test loads and validates each pipeline
with the sample campaign and a compatible local- or S3-audio session.
## Campaign And Session Configuration
- [Sample campaign](campaigns/sample-campaign/campaign.yml), its
[session template](campaigns/sample-campaign/session.template.yml), and its
adjacent stable inputs provide a complete campaign directory shape.
- [Local-audio session](session.local-audio.yml) and
[S3-audio session](session.s3-audio.yml) are concrete session files.
- [Session template](session.template.yml) and the campaign-local equivalent
demonstrate the narrow placeholder syntax consumed by `session init`; they
are templates, not runtime session files.
## Input Fixtures
- [Speakers](speakers.yml), [autocorrect](autocorrect.yml), and
[glossary](glossary.yml) show the standalone input shapes.
- The sample campaign references its local
[speakers](campaigns/sample-campaign/speakers.yml),
[autocorrect](campaigns/sample-campaign/autocorrect.yml),
[glossary](campaigns/sample-campaign/glossary.yml),
[players](campaigns/sample-campaign/players.yml), and
[party](campaigns/sample-campaign/party.yml) fixtures.
- [Sample speaker audio](audio/sample-speaker.flac) is a text placeholder that
reserves the expected filename and directory shape. Replace it with a real
FLAC file before running transcription.
The examples contain environment-variable names but no credential values. They
use fictional campaign content and reserved example domains.

View File

@@ -4,3 +4,5 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml

View File

@@ -0,0 +1,2 @@
- name: Example Hero
type: pc

View File

@@ -0,0 +1,2 @@
- name: Example Player
role: player

View File

@@ -1,5 +1,5 @@
match:
- speaker: "Eric Rakestraw"
- speaker: "Example Speaker"
match:
- "Eric_Rakestraw"
- "Eric"
- "Example_Speaker"
- "Example"

View File

@@ -0,0 +1,55 @@
# Purpose-specific extraction example: a Scriptorium session brief consumes
# only the three Notarius lanes it needs.
campaigns:
root: /usr/local/share/narratio/campaigns
default_campaign_id: sample-campaign
whisperx:
transcribe_url: "https://transcription.example.com/transcribe"
notarius:
enabled: true
binary: notarius
config_path: /usr/local/etc/notarius/config.yml
pipeline_id: dnd-session
timeout: 3h
outputs:
npc_registry:
lane_id: npc-registry
media_type: application/json
schema_id: notarius.dnd.npc_registry
schema_version: v1
module_key: dnd/npc-registry
location_registry:
lane_id: location-registry
media_type: application/json
schema_id: notarius.dnd.location_registry
schema_version: v1
module_key: dnd/location-registry
scene_descriptions:
lane_id: scene-descriptions
media_type: application/json
schema_id: notarius.dnd.scene_descriptions
schema_version: v1
module_key: dnd/scene-descriptions
scriptorium:
binary: scriptorium
config_path: /usr/local/etc/scriptorium/config.yml
artifacts:
session_brief:
enabled: true
prompt_id: dnd.session_brief
output_path: artifacts/session_brief.md
inputs:
npcs:
source: narratio.extraction.npc_registry
required: true
locations:
source: narratio.extraction.location_registry
required: true
scenes:
source: narratio.extraction.scene_descriptions
required: true

View File

@@ -60,6 +60,11 @@ publish:
- source: narratio.artifact.player_handout
dest: artifacts/player_handout.md
required: false
# Extraction lanes publish only when named explicitly; the bundle and index
# are never implicit publish sources.
- source: narratio.extraction.npc_registry
dest: artifacts/extraction/npc-registry.json
required: true
whisperx:
# Required.
@@ -110,20 +115,91 @@ normalize:
report: true
trim:
# Keep disabled unless bounds prompt integration is configured.
enabled: false
# Optional; defaults shown explicitly.
enabled: true
output_path: transcripts/final.trimmed.json
bounds:
prompt_id: dnd.session_bounds
profile_id: local-fast
profile_id: ""
transcript_input_name: transcript
output_path: reports/session_bounds.json
output_path: artifacts/session_bounds.json
timeout: 10m
render_debug: false
render_output_path: reports/session_bounds.render.json
seriatim:
report: false
notarius:
# Optional structured extraction between trim and render.
enabled: true
binary: notarius
config_path: /usr/local/etc/notarius/config.yml
pipeline_id: dnd-session
timeout: 3h
working_directory: /usr/local/etc/notarius
# Each key creates source narratio.extraction.<key>. These constraints match
# the current Notarius D&D lane contracts; update them with Notarius.
outputs:
item_registry:
lane_id: item-registry
media_type: application/json
schema_id: notarius.dnd.item_registry
schema_version: v1
module_key: dnd/item-registry
npc_registry:
lane_id: npc-registry
media_type: application/json
schema_id: notarius.dnd.npc_registry
schema_version: v1
module_key: dnd/npc-registry
location_registry:
lane_id: location-registry
media_type: application/json
schema_id: notarius.dnd.location_registry
schema_version: v1
module_key: dnd/location-registry
scene_descriptions:
lane_id: scene-descriptions
media_type: application/json
schema_id: notarius.dnd.scene_descriptions
schema_version: v1
module_key: dnd/scene-descriptions
item_occurrences:
lane_id: item-occurrences
media_type: application/json
schema_id: notarius.dnd.item_occurrences
schema_version: v1
module_key: dnd/item-occurrences
spells:
lane_id: spells
media_type: application/json
schema_id: notarius.dnd.spells
schema_version: v1
module_key: dnd/spells
combat_turns:
lane_id: combat-turns
media_type: application/json
schema_id: notarius.dnd.combat_turns
schema_version: v1
module_key: dnd/combat-turns
npc_occurrences:
lane_id: npc-occurrences
media_type: application/json
schema_id: notarius.dnd.npc_occurrences
schema_version: v1
module_key: dnd/npc-occurrences
location_occurrences:
lane_id: location-occurrences
media_type: application/json
schema_id: notarius.dnd.location_occurrences
schema_version: v1
module_key: dnd/location-occurrences
enemy_events:
lane_id: enemy-events
media_type: application/json
schema_id: notarius.dnd.enemy_events
schema_version: v1
module_key: dnd/enemy-events
scriptorium:
binary: scriptorium
config_path: /usr/local/etc/scriptorium/config.yml
@@ -144,6 +220,15 @@ scriptorium:
previous_recap:
source: narratio.previous_session.artifact.session_recap
required: false
players:
source: narratio.input.players
required: true
party:
source: narratio.input.party
required: true
glossary:
source: narratio.input.glossary
required: false
vars:
session_id: true
session_date: true

View File

@@ -71,9 +71,6 @@ normalize:
output_schema: seriatim-intermediate
report: true
trim:
enabled: false
scriptorium:
binary: scriptorium
config_path: /usr/local/etc/scriptorium/config.yml
@@ -93,6 +90,15 @@ scriptorium:
previous_recap:
source: narratio.previous_session.artifact.session_recap
required: false
players:
source: narratio.input.players
required: true
party:
source: narratio.input.party
required: true
glossary:
source: narratio.input.glossary
required: false
vars:
session_id: true
session_date: true

View File

@@ -1,5 +1,5 @@
match:
- speaker: "Eric Rakestraw"
- speaker: "Example Speaker"
match:
- "Eric_Rakestraw"
- "Eric"
- "Example_Speaker"
- "Example"

1
go.mod
View File

@@ -7,6 +7,7 @@ require (
github.com/aws/aws-sdk-go-v2/credentials v1.19.16
github.com/aws/aws-sdk-go-v2/service/s3 v1.101.0
github.com/aws/smithy-go v1.25.1
golang.org/x/sys v0.47.0
gopkg.in/yaml.v3 v3.0.1
)

2
go.sum
View File

@@ -34,6 +34,8 @@ github.com/aws/aws-sdk-go-v2/service/sts v1.42.1 h1:F/M5Y9I3nwr2IEpshZgh1GeHpOIt
github.com/aws/aws-sdk-go-v2/service/sts v1.42.1/go.mod h1:mTNxImtovCOEEuD65mKW7DCsL+2gjEH+RPEAexAzAio=
github.com/aws/smithy-go v1.25.1 h1:J8ERsGSU7d+aCmdQur5Txg6bVoYelvQJgtZehD12GkI=
github.com/aws/smithy-go v1.25.1/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=

View File

@@ -0,0 +1,22 @@
package notarius
import "context"
// FakeRunner is a configurable in-memory runner for stage tests.
type FakeRunner struct {
Requests []RunRequest
Result RunResult
Err error
}
// Run records the request and returns the configured result or error.
func (f *FakeRunner) Run(ctx context.Context, req RunRequest) (RunResult, error) {
if err := ctx.Err(); err != nil {
return RunResult{}, err
}
f.Requests = append(f.Requests, req)
if f.Err != nil {
return RunResult{}, f.Err
}
return f.Result, nil
}

View File

@@ -0,0 +1,108 @@
// Package notarius declares the adapter contract for Notarius CLI invocations.
package notarius
import (
"context"
"time"
)
const ReceiptSchemaVersion = "notarius.run-result.v1"
// Runner is the adapter boundary for a complete Notarius pipeline invocation.
type Runner interface {
Run(ctx context.Context, req RunRequest) (RunResult, error)
}
// RunRequest contains the resolved inputs and diagnostic destinations for one invocation.
type RunRequest struct {
Binary string
ConfigPath string
PipelineID string
InputPath string
OutputRoot string
WorkingDirectory string
ReceiptPath string
LogPath string
Timeout time.Duration
}
// Receipt is the transport-neutral successful run receipt.
type Receipt struct {
SchemaVersion string
RunID string
PipelineID string
OutputDirectory string
IndexFile string
NormalizedOutputCount int
RejectedOutputCount int
WarningCount int
ValidationStatus string
DebugDirectory string
}
// LaneDescriptor identifies one normalized lane payload discovered through the index.
type LaneDescriptor struct {
LaneID string
File string
Path string
MediaType string
ModuleKey string
SchemaID string
SchemaName string
SchemaVersion string
}
// PipelineDescriptor identifies a pipeline-wide artifact discovered through the index.
type PipelineDescriptor struct {
ArtifactKind string
File string
Path string
MediaType string
SchemaID string
SchemaName string
SchemaVersion string
}
// Index describes the validated bundle-management and artifact paths.
type Index struct {
Path string
ManifestFile string
ManifestPath string
RejectedFile string
RejectedPath string
WarningsFile string
WarningsPath string
Lanes []LaneDescriptor
ChunkMap *PipelineDescriptor
EvidenceContext *PipelineDescriptor
}
// RejectionSummary retains structured rejection identity without free-form messages.
type RejectionSummary struct {
Stage string
StepID string
LaneID string
ModuleKey string
ChunkID string
ValidatorName string
ReasonCode string
}
// WarningSummary retains structured warning identity without free-form messages.
type WarningSummary struct {
Scope string
ReasonCode string
}
// RunResult describes a successfully decoded and validated Notarius bundle.
type RunResult struct {
Receipt Receipt
Index Index
BundleRoot string
ReceiptPath string
LogPath string
ExitCode int
Duration time.Duration
Rejections []RejectionSummary
Warnings []WarningSummary
}

View File

@@ -0,0 +1,524 @@
package notarius
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/subprocess"
"gitea.maximumdirect.net/eric/narratio/internal/pathsafe"
)
const (
maxReceiptBytes = 1 << 20
maxIndexBytes = 4 << 20
maxSummaryBytes = 4 << 20
canonicalIndexFile = "index.json"
canonicalManifestFile = "manifest.json"
canonicalRejectedFile = "rejected.json"
canonicalWarningsFile = "warnings.json"
)
type subprocessRun func(context.Context, subprocess.RunRequest) (subprocess.RunResult, error)
// SubprocessRunner invokes Notarius through its public CLI.
type SubprocessRunner struct {
run subprocessRun
}
// NewSubprocessRunner constructs a production Notarius subprocess runner.
func NewSubprocessRunner() *SubprocessRunner {
return &SubprocessRunner{run: subprocess.Run}
}
// Run executes a complete Notarius pipeline and discovers its published bundle.
func (r *SubprocessRunner) Run(ctx context.Context, req RunRequest) (RunResult, error) {
if r == nil || r.run == nil {
return RunResult{}, fmt.Errorf("notarius subprocess runner is nil")
}
if err := validateRunRequest(req); err != nil {
return RunResult{}, err
}
args := []string{
"run", req.PipelineID,
"--config", req.ConfigPath,
"--input", req.InputPath,
"--output-dir", req.OutputRoot,
"--json",
}
processResult, err := r.run(ctx, subprocess.RunRequest{
Executable: req.Binary,
Args: args,
WorkingDir: req.WorkingDirectory,
Timeout: req.Timeout,
StdoutLogPath: req.ReceiptPath,
StderrLogPath: req.LogPath,
})
baseResult := RunResult{
ReceiptPath: req.ReceiptPath,
LogPath: req.LogPath,
ExitCode: processResult.ExitCode,
Duration: processResult.Duration,
}
if err != nil {
return baseResult, fmt.Errorf("run notarius pipeline %q: %w", req.PipelineID, err)
}
receipt, err := loadReceipt(req.ReceiptPath, req.PipelineID)
if err != nil {
return baseResult, err
}
bundleRoot, err := validateBundleRoot(req.OutputRoot, receipt.OutputDirectory)
if err != nil {
return baseResult, err
}
indexPath, err := resolveRegularFile(bundleRoot, receipt.IndexFile)
if err != nil {
return baseResult, fmt.Errorf("resolve receipt index file: %w", err)
}
index, err := loadIndex(bundleRoot, indexPath)
if err != nil {
return baseResult, err
}
rejections, err := loadRejections(index.RejectedPath)
if err != nil {
return baseResult, err
}
warnings, err := loadWarnings(index.WarningsPath)
if err != nil {
return baseResult, err
}
baseResult.Receipt = receipt
baseResult.Index = index
baseResult.BundleRoot = bundleRoot
baseResult.Rejections = rejections
baseResult.Warnings = warnings
return baseResult, nil
}
func validateRunRequest(req RunRequest) error {
if strings.TrimSpace(req.Binary) == "" {
return fmt.Errorf("notarius binary is required")
}
if strings.TrimSpace(req.PipelineID) == "" {
return fmt.Errorf("notarius pipeline id is required")
}
if req.Timeout <= 0 {
return fmt.Errorf("notarius timeout must be positive")
}
for label, path := range map[string]string{
"config": req.ConfigPath,
"input": req.InputPath,
"output root": req.OutputRoot,
"working directory": req.WorkingDirectory,
"receipt": req.ReceiptPath,
"log": req.LogPath,
} {
if strings.TrimSpace(path) == "" {
return fmt.Errorf("notarius %s path is required", label)
}
if !filepath.IsAbs(path) {
return fmt.Errorf("notarius %s path must be absolute", label)
}
}
if filepath.Clean(req.ReceiptPath) == filepath.Clean(req.LogPath) {
return fmt.Errorf("notarius receipt and log paths must be different")
}
if err := requireRegularFile(req.ConfigPath); err != nil {
return fmt.Errorf("validate notarius config path: %w", err)
}
if err := requireRegularFile(req.InputPath); err != nil {
return fmt.Errorf("validate notarius input path: %w", err)
}
if err := requireDirectory(req.OutputRoot); err != nil {
return fmt.Errorf("validate notarius output root: %w", err)
}
if err := requireDirectory(req.WorkingDirectory); err != nil {
return fmt.Errorf("validate notarius working directory: %w", err)
}
if err := validateLogDestination(req.ReceiptPath); err != nil {
return fmt.Errorf("validate notarius receipt path: %w", err)
}
if err := validateLogDestination(req.LogPath); err != nil {
return fmt.Errorf("validate notarius log path: %w", err)
}
return nil
}
type receiptDocument struct {
SchemaVersion string `json:"schema_version"`
RunID string `json:"run_id"`
PipelineID string `json:"pipeline_id"`
OutputDirectory string `json:"output_directory"`
IndexFile string `json:"index_file"`
NormalizedOutputCount *int `json:"normalized_output_count"`
RejectedOutputCount *int `json:"rejected_output_count"`
WarningCount *int `json:"warning_count"`
ValidationStatus string `json:"validation_status"`
DebugDirectory string `json:"debug_directory"`
}
func loadReceipt(path, pipelineID string) (Receipt, error) {
var document receiptDocument
if err := decodeBoundedJSON(path, maxReceiptBytes, &document); err != nil {
return Receipt{}, fmt.Errorf("decode notarius receipt: %w", err)
}
if document.SchemaVersion != ReceiptSchemaVersion {
return Receipt{}, fmt.Errorf("unsupported notarius receipt schema version %q", document.SchemaVersion)
}
if strings.TrimSpace(document.RunID) == "" || strings.TrimSpace(document.PipelineID) == "" ||
strings.TrimSpace(document.OutputDirectory) == "" || strings.TrimSpace(document.ValidationStatus) == "" ||
document.NormalizedOutputCount == nil ||
document.RejectedOutputCount == nil || document.WarningCount == nil {
return Receipt{}, fmt.Errorf("notarius receipt is missing required fields")
}
if document.IndexFile != canonicalIndexFile {
return Receipt{}, fmt.Errorf("notarius receipt index_file %q is incompatible; want %q", document.IndexFile, canonicalIndexFile)
}
if document.PipelineID != pipelineID {
return Receipt{}, fmt.Errorf("notarius receipt pipeline id %q does not match requested pipeline %q", document.PipelineID, pipelineID)
}
if *document.NormalizedOutputCount < 0 || *document.RejectedOutputCount < 0 || *document.WarningCount < 0 {
return Receipt{}, fmt.Errorf("notarius receipt counts must be non-negative")
}
if !filepath.IsAbs(document.OutputDirectory) {
return Receipt{}, fmt.Errorf("notarius receipt output directory must be absolute")
}
if document.DebugDirectory != "" && !filepath.IsAbs(document.DebugDirectory) {
return Receipt{}, fmt.Errorf("notarius receipt debug directory must be absolute when present")
}
return Receipt{
SchemaVersion: document.SchemaVersion,
RunID: document.RunID,
PipelineID: document.PipelineID,
OutputDirectory: filepath.Clean(document.OutputDirectory),
IndexFile: document.IndexFile,
NormalizedOutputCount: *document.NormalizedOutputCount,
RejectedOutputCount: *document.RejectedOutputCount,
WarningCount: *document.WarningCount,
ValidationStatus: document.ValidationStatus,
DebugDirectory: document.DebugDirectory,
}, nil
}
type indexDocument struct {
ManifestFile string `json:"manifest_file"`
OutputFiles *[]laneDocument `json:"output_files"`
RejectedFile string `json:"rejected_file"`
WarningsFile string `json:"warnings_file"`
ChunkMap *pipelineDocument `json:"chunk_map"`
EvidenceContext *pipelineDocument `json:"evidence_context"`
}
type laneDocument struct {
LaneID string `json:"lane_id"`
File string `json:"file"`
MediaType string `json:"media_type"`
ModuleKey string `json:"module_key"`
SchemaID string `json:"schema_id"`
SchemaName string `json:"schema_name"`
SchemaVersion string `json:"schema_version"`
}
type pipelineDocument struct {
ArtifactKind string `json:"artifact_kind"`
File string `json:"file"`
MediaType string `json:"media_type"`
SchemaID string `json:"schema_id"`
SchemaName string `json:"schema_name"`
SchemaVersion string `json:"schema_version"`
}
func loadIndex(bundleRoot, indexPath string) (Index, error) {
var document indexDocument
if err := decodeBoundedJSON(indexPath, maxIndexBytes, &document); err != nil {
return Index{}, fmt.Errorf("decode notarius index: %w", err)
}
for _, field := range []struct {
name string
got string
want string
}{
{name: "manifest_file", got: document.ManifestFile, want: canonicalManifestFile},
{name: "rejected_file", got: document.RejectedFile, want: canonicalRejectedFile},
{name: "warnings_file", got: document.WarningsFile, want: canonicalWarningsFile},
} {
if field.got != field.want {
return Index{}, fmt.Errorf("notarius index %s %q is incompatible; want %q", field.name, field.got, field.want)
}
}
if document.OutputFiles == nil {
return Index{}, fmt.Errorf("notarius index is missing required output_files")
}
index := Index{
Path: indexPath,
ManifestFile: document.ManifestFile,
RejectedFile: document.RejectedFile,
WarningsFile: document.WarningsFile,
}
var err error
if index.ManifestPath, err = resolveRegularFile(bundleRoot, index.ManifestFile); err != nil {
return Index{}, fmt.Errorf("resolve notarius manifest file: %w", err)
}
if index.RejectedPath, err = resolveRegularFile(bundleRoot, index.RejectedFile); err != nil {
return Index{}, fmt.Errorf("resolve notarius rejection file: %w", err)
}
if index.WarningsPath, err = resolveRegularFile(bundleRoot, index.WarningsFile); err != nil {
return Index{}, fmt.Errorf("resolve notarius warning file: %w", err)
}
seenLanes := make(map[string]struct{}, len(*document.OutputFiles))
for _, lane := range *document.OutputFiles {
if strings.TrimSpace(lane.LaneID) == "" || strings.TrimSpace(lane.File) == "" {
return Index{}, fmt.Errorf("notarius lane descriptors require lane_id and file")
}
if _, exists := seenLanes[lane.LaneID]; exists {
return Index{}, fmt.Errorf("notarius index contains duplicate lane id %q", lane.LaneID)
}
seenLanes[lane.LaneID] = struct{}{}
path, err := resolveRegularFile(bundleRoot, lane.File)
if err != nil {
return Index{}, fmt.Errorf("resolve notarius lane %q file: %w", lane.LaneID, err)
}
index.Lanes = append(index.Lanes, LaneDescriptor{
LaneID: lane.LaneID, File: lane.File, Path: path, MediaType: lane.MediaType,
ModuleKey: lane.ModuleKey, SchemaID: lane.SchemaID, SchemaName: lane.SchemaName,
SchemaVersion: lane.SchemaVersion,
})
}
if document.ChunkMap != nil {
index.ChunkMap, err = resolvePipelineDescriptor(bundleRoot, "chunk_map", *document.ChunkMap)
if err != nil {
return Index{}, err
}
}
if document.EvidenceContext != nil {
index.EvidenceContext, err = resolvePipelineDescriptor(bundleRoot, "evidence_context", *document.EvidenceContext)
if err != nil {
return Index{}, err
}
}
return index, nil
}
func resolvePipelineDescriptor(bundleRoot, label string, document pipelineDocument) (*PipelineDescriptor, error) {
if strings.TrimSpace(document.ArtifactKind) == "" || strings.TrimSpace(document.File) == "" ||
strings.TrimSpace(document.MediaType) == "" || strings.TrimSpace(document.SchemaID) == "" ||
strings.TrimSpace(document.SchemaName) == "" || strings.TrimSpace(document.SchemaVersion) == "" {
return nil, fmt.Errorf("notarius %s descriptor is missing required fields", label)
}
path, err := resolveRegularFile(bundleRoot, document.File)
if err != nil {
return nil, fmt.Errorf("resolve notarius %s file: %w", label, err)
}
return &PipelineDescriptor{
ArtifactKind: document.ArtifactKind, File: document.File, Path: path,
MediaType: document.MediaType, SchemaID: document.SchemaID,
SchemaName: document.SchemaName, SchemaVersion: document.SchemaVersion,
}, nil
}
type rejectionDocument struct {
Rejected *[]struct {
Stage string `json:"stage"`
StepID string `json:"step_id"`
LaneID string `json:"lane_id"`
ModuleKey string `json:"module_key"`
ChunkID string `json:"chunk_id"`
ValidatorName string `json:"validator_name"`
ReasonCode string `json:"reason_code"`
Message string `json:"message"`
} `json:"rejected"`
}
func loadRejections(path string) ([]RejectionSummary, error) {
var document rejectionDocument
if err := decodeBoundedJSON(path, maxSummaryBytes, &document); err != nil {
return nil, fmt.Errorf("decode notarius rejections: %w", err)
}
if document.Rejected == nil {
return nil, fmt.Errorf("notarius rejection document is missing rejected array")
}
summaries := make([]RejectionSummary, 0, len(*document.Rejected))
for _, item := range *document.Rejected {
if strings.TrimSpace(item.Stage) == "" || strings.TrimSpace(item.Message) == "" {
return nil, fmt.Errorf("notarius rejection entries require stage and message")
}
summaries = append(summaries, RejectionSummary{
Stage: item.Stage, StepID: item.StepID, LaneID: item.LaneID,
ModuleKey: item.ModuleKey, ChunkID: item.ChunkID,
ValidatorName: item.ValidatorName, ReasonCode: item.ReasonCode,
})
}
return summaries, nil
}
type warningDocument struct {
Warnings *[]struct {
Scope string `json:"scope"`
ReasonCode string `json:"reason_code"`
Message string `json:"message"`
} `json:"warnings"`
}
func loadWarnings(path string) ([]WarningSummary, error) {
var document warningDocument
if err := decodeBoundedJSON(path, maxSummaryBytes, &document); err != nil {
return nil, fmt.Errorf("decode notarius warnings: %w", err)
}
if document.Warnings == nil {
return nil, fmt.Errorf("notarius warning document is missing warnings array")
}
summaries := make([]WarningSummary, 0, len(*document.Warnings))
for _, item := range *document.Warnings {
if strings.TrimSpace(item.ReasonCode) == "" || strings.TrimSpace(item.Message) == "" {
return nil, fmt.Errorf("notarius warning entries require reason_code and message")
}
summaries = append(summaries, WarningSummary{Scope: item.Scope, ReasonCode: item.ReasonCode})
}
return summaries, nil
}
func decodeBoundedJSON(path string, limit int64, destination any) error {
inspected, err := os.Lstat(path)
if err != nil {
return err
}
if inspected.Mode()&os.ModeSymlink != 0 || !inspected.Mode().IsRegular() {
return fmt.Errorf("path %q must be a regular file without symlinks", path)
}
file, err := os.Open(path)
if err != nil {
return err
}
defer func() { _ = file.Close() }()
opened, err := file.Stat()
if err != nil {
return err
}
if !opened.Mode().IsRegular() || !os.SameFile(inspected, opened) {
return fmt.Errorf("file %q changed before it could be read", path)
}
reader := io.LimitReader(file, limit+1)
data, err := io.ReadAll(reader)
if err != nil {
return err
}
if int64(len(data)) > limit {
return fmt.Errorf("file %q exceeds %d-byte limit", path, limit)
}
if err := json.Unmarshal(data, destination); err != nil {
return err
}
return nil
}
func validateBundleRoot(outputRoot, bundleRoot string) (string, error) {
root := filepath.Clean(outputRoot)
bundle := filepath.Clean(bundleRoot)
relative, err := filepath.Rel(root, bundle)
if err != nil {
return "", fmt.Errorf("compare notarius output paths: %w", err)
}
if relative == "." || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
return "", fmt.Errorf("notarius output directory %q is not beneath output root %q", bundleRoot, outputRoot)
}
if err := requireDirectoryTree(root, relative); err != nil {
return "", fmt.Errorf("validate notarius output directory: %w", err)
}
return bundle, nil
}
func resolveRegularFile(root, logicalPath string) (string, error) {
resolved, err := pathsafe.JoinSlashRelativeUnderRoot(root, logicalPath)
if err != nil {
return "", err
}
relative, err := filepath.Rel(root, resolved)
if err != nil {
return "", err
}
if err := requireRegularFileTree(root, relative); err != nil {
return "", err
}
return resolved, nil
}
func requireDirectoryTree(root, relative string) error {
if err := requireDirectory(root); err != nil {
return err
}
current := root
for _, component := range strings.Split(relative, string(filepath.Separator)) {
current = filepath.Join(current, component)
if err := requireDirectory(current); err != nil {
return err
}
}
return nil
}
func requireRegularFileTree(root, relative string) error {
components := strings.Split(relative, string(filepath.Separator))
if len(components) == 0 {
return fmt.Errorf("regular file path is required")
}
if err := requireDirectory(root); err != nil {
return err
}
current := root
for _, component := range components[:len(components)-1] {
current = filepath.Join(current, component)
if err := requireDirectory(current); err != nil {
return err
}
}
return requireRegularFile(filepath.Join(current, components[len(components)-1]))
}
func requireDirectory(path string) error {
info, err := os.Lstat(path)
if err != nil {
return err
}
if info.Mode()&os.ModeSymlink != 0 || !info.IsDir() {
return fmt.Errorf("path %q must be a directory without symlinks", path)
}
return nil
}
func requireRegularFile(path string) error {
info, err := os.Lstat(path)
if err != nil {
return err
}
if info.Mode()&os.ModeSymlink != 0 || !info.Mode().IsRegular() {
return fmt.Errorf("path %q must be a regular file without symlinks", path)
}
return nil
}
func validateLogDestination(path string) error {
if err := requireDirectory(filepath.Dir(path)); err != nil {
return err
}
info, err := os.Lstat(path)
if errors.Is(err, os.ErrNotExist) {
return nil
}
if err != nil {
return err
}
if info.Mode()&os.ModeSymlink != 0 || !info.Mode().IsRegular() {
return fmt.Errorf("path %q must be absent or a regular file without symlinks", path)
}
return nil
}

View File

@@ -0,0 +1,572 @@
package notarius
import (
"context"
"encoding/json"
"errors"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
sharedsubprocess "gitea.maximumdirect.net/eric/narratio/internal/adapters/subprocess"
)
func TestSubprocessRunnerBuildsExactInvocationAndDiscoversBundle(t *testing.T) {
req := validRunRequest(t)
var captured sharedsubprocess.RunRequest
runner := &SubprocessRunner{run: func(_ context.Context, processReq sharedsubprocess.RunRequest) (sharedsubprocess.RunResult, error) {
captured = processReq
writeValidBundleAndReceipt(t, req, true)
return sharedsubprocess.RunResult{ExitCode: 0, Duration: 2 * time.Second}, nil
}}
result, err := runner.Run(context.Background(), req)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
wantArgs := []string{
"run", "dnd-session", "--config", req.ConfigPath, "--input", req.InputPath,
"--output-dir", req.OutputRoot, "--json",
}
if !reflect.DeepEqual(captured.Args, wantArgs) {
t.Fatalf("subprocess args = %#v, want %#v", captured.Args, wantArgs)
}
if captured.Executable != req.Binary || captured.WorkingDir != req.WorkingDirectory || captured.Timeout != req.Timeout {
t.Fatalf("subprocess request = %#v", captured)
}
if captured.StdoutLogPath != req.ReceiptPath || captured.StderrLogPath != req.LogPath {
t.Fatalf("stream paths = stdout %q stderr %q", captured.StdoutLogPath, captured.StderrLogPath)
}
if captured.EnvOverrides != nil {
t.Fatalf("environment overrides = %#v, want inherited environment only", captured.EnvOverrides)
}
for _, arg := range captured.Args {
if arg == "--session-id" {
t.Fatal("subprocess args unexpectedly contain --session-id")
}
}
if result.Receipt.SchemaVersion != ReceiptSchemaVersion || result.Receipt.RunID != "notarius-run-1" {
t.Fatalf("receipt = %#v", result.Receipt)
}
if len(result.Index.Lanes) != 1 || result.Index.Lanes[0].LaneID != "npc-registry" {
t.Fatalf("lanes = %#v", result.Index.Lanes)
}
if result.Index.ChunkMap == nil || result.Index.ChunkMap.ArtifactKind != "chunk_map" {
t.Fatalf("chunk map = %#v", result.Index.ChunkMap)
}
if result.Index.EvidenceContext == nil || result.Index.EvidenceContext.ArtifactKind != "evidence_context" {
t.Fatalf("evidence context = %#v", result.Index.EvidenceContext)
}
if len(result.Rejections) != 1 || result.Rejections[0].LaneID != "spells" || result.Rejections[0].ReasonCode != "invalid_spell" {
t.Fatalf("rejections = %#v", result.Rejections)
}
if len(result.Warnings) != 1 || result.Warnings[0].Scope != "lane:npc-registry" || result.Warnings[0].ReasonCode != "normalized_name" {
t.Fatalf("warnings = %#v", result.Warnings)
}
}
func TestSubprocessRunnerInheritsEnvironmentAndSeparatesStreams(t *testing.T) {
req := validRunRequest(t)
writeValidBundleAndReceipt(t, req, false)
receiptFixture := req.ReceiptPath + ".fixture"
data, err := os.ReadFile(req.ReceiptPath)
if err != nil {
t.Fatalf("ReadFile(receipt) error = %v", err)
}
if err := os.WriteFile(receiptFixture, data, 0o644); err != nil {
t.Fatalf("WriteFile(receipt fixture) error = %v", err)
}
if err := os.Remove(req.ReceiptPath); err != nil {
t.Fatalf("Remove(receipt) error = %v", err)
}
captureDir := filepath.Join(filepath.Dir(req.ReceiptPath), "capture")
if err := os.Mkdir(captureDir, 0o755); err != nil {
t.Fatalf("Mkdir(capture) error = %v", err)
}
script := writeShellScript(t, `#!/bin/sh
pwd > "$NOTARIUS_CAPTURE_DIR/working-directory"
printf '%s' "$NOTARIUS_INHERITED_VALUE" > "$NOTARIUS_CAPTURE_DIR/environment"
printf 'diagnostic stream\n' >&2
cat "$NOTARIUS_RECEIPT_FIXTURE"
`)
req.Binary = script
t.Setenv("NOTARIUS_CAPTURE_DIR", captureDir)
t.Setenv("NOTARIUS_INHERITED_VALUE", "inherited-value")
t.Setenv("NOTARIUS_RECEIPT_FIXTURE", receiptFixture)
if _, err := NewSubprocessRunner().Run(context.Background(), req); err != nil {
t.Fatalf("Run() error = %v", err)
}
assertTextFile(t, filepath.Join(captureDir, "working-directory"), req.WorkingDirectory+"\n")
assertTextFile(t, filepath.Join(captureDir, "environment"), "inherited-value")
assertTextFile(t, req.LogPath, "diagnostic stream\n")
receiptBytes, err := os.ReadFile(req.ReceiptPath)
if err != nil {
t.Fatalf("ReadFile(receipt) error = %v", err)
}
if strings.Contains(string(receiptBytes), "diagnostic stream") {
t.Fatal("receipt contains stderr output")
}
}
func TestSubprocessRunnerReturnsProcessFailuresWithoutParsingStdout(t *testing.T) {
tests := []struct {
name string
scriptBody string
timeout time.Duration
cancel bool
want string
}{
{name: "nonzero", scriptBody: "printf '{malformed receipt'; printf 'failed\\n' >&2; exit 7\n", timeout: time.Second, want: "exit code 7"},
{name: "timeout", scriptBody: "sleep 5\n", timeout: 20 * time.Millisecond, want: "timed out"},
{name: "cancellation", scriptBody: "sleep 5\n", timeout: time.Second, cancel: true, want: "canceled"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
req := validRunRequest(t)
req.Binary = writeShellScript(t, "#!/bin/sh\n"+test.scriptBody)
req.Timeout = test.timeout
ctx := context.Background()
if test.cancel {
cancelCtx, cancel := context.WithCancel(ctx)
ctx = cancelCtx
time.AfterFunc(20*time.Millisecond, cancel)
}
_, err := NewSubprocessRunner().Run(ctx, req)
if err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("Run() error = %v, want fragment %q", err, test.want)
}
if strings.Contains(err.Error(), "decode notarius receipt") {
t.Fatalf("Run() parsed stdout after process failure: %v", err)
}
})
}
}
func TestSubprocessRunnerReturnsSharedSubprocessErrorWithoutReadingReceipt(t *testing.T) {
req := validRunRequest(t)
if err := os.WriteFile(req.ReceiptPath, []byte("not json"), 0o644); err != nil {
t.Fatalf("WriteFile(receipt) error = %v", err)
}
wantErr := errors.New("process failed")
runner := &SubprocessRunner{run: func(context.Context, sharedsubprocess.RunRequest) (sharedsubprocess.RunResult, error) {
return sharedsubprocess.RunResult{ExitCode: 9}, wantErr
}}
_, err := runner.Run(context.Background(), req)
if !errors.Is(err, wantErr) {
t.Fatalf("Run() error = %v, want wrapped process error", err)
}
if strings.Contains(err.Error(), "decode") {
t.Fatalf("Run() parsed receipt after failure: %v", err)
}
}
func TestLoadReceiptValidation(t *testing.T) {
root := t.TempDir()
valid := map[string]any{
"schema_version": ReceiptSchemaVersion, "run_id": "run-1", "pipeline_id": "pipeline-1",
"output_directory": filepath.Join(root, "outputs", "run-1"), "index_file": "index.json",
"normalized_output_count": 1, "rejected_output_count": 0, "warning_count": 0,
"validation_status": "approved", "future_field": true,
}
tests := []struct {
name string
mutate func(map[string]any)
raw []byte
wantOK bool
wantError string
}{
{name: "unknown fields tolerated", wantOK: true},
{name: "malformed", raw: []byte("{")},
{name: "unsupported version", mutate: func(v map[string]any) { v["schema_version"] = "notarius.run-result.v2" }},
{name: "missing field", mutate: func(v map[string]any) { delete(v, "run_id") }},
{name: "pipeline mismatch", mutate: func(v map[string]any) { v["pipeline_id"] = "other" }},
{name: "relative output", mutate: func(v map[string]any) { v["output_directory"] = "run-1" }},
{name: "negative count", mutate: func(v map[string]any) { v["warning_count"] = -1 }},
{
name: "nested index", mutate: func(v map[string]any) { v["index_file"] = "nested/index.json" },
wantError: `index_file "nested/index.json"`,
},
{
name: "cleanable index", mutate: func(v map[string]any) { v["index_file"] = "./index.json" },
wantError: `index_file "./index.json"`,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
path := filepath.Join(root, strings.ReplaceAll(test.name, " ", "-")+".json")
values := cloneMap(valid)
if test.mutate != nil {
test.mutate(values)
}
if test.raw != nil {
if err := os.WriteFile(path, test.raw, 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
} else {
writeJSONFile(t, path, values)
}
_, err := loadReceipt(path, "pipeline-1")
if test.wantOK && err != nil {
t.Fatalf("loadReceipt() error = %v", err)
}
if !test.wantOK && err == nil {
t.Fatal("loadReceipt() error = nil, want validation failure")
}
if test.wantError != "" && !strings.Contains(err.Error(), test.wantError) {
t.Fatalf("loadReceipt() error = %v, want fragment %q", err, test.wantError)
}
})
}
oversized := filepath.Join(root, "oversized.json")
if err := os.WriteFile(oversized, []byte(strings.Repeat("x", maxReceiptBytes+1)), 0o644); err != nil {
t.Fatalf("WriteFile(oversized) error = %v", err)
}
if _, err := loadReceipt(oversized, "pipeline-1"); err == nil || !strings.Contains(err.Error(), "exceeds") {
t.Fatalf("loadReceipt(oversized) error = %v", err)
}
}
func TestValidateBundleRootRejectsEscapesAndSymlinks(t *testing.T) {
root := t.TempDir()
outputRoot := filepath.Join(root, "output")
if err := os.Mkdir(outputRoot, 0o755); err != nil {
t.Fatalf("Mkdir(output root) error = %v", err)
}
validBundle := filepath.Join(outputRoot, "run-1")
if err := os.Mkdir(validBundle, 0o755); err != nil {
t.Fatalf("Mkdir(bundle) error = %v", err)
}
if _, err := validateBundleRoot(outputRoot, validBundle); err != nil {
t.Fatalf("validateBundleRoot(valid) error = %v", err)
}
outside := filepath.Join(root, "output-other")
if err := os.Mkdir(outside, 0o755); err != nil {
t.Fatalf("Mkdir(outside) error = %v", err)
}
for name, candidate := range map[string]string{"equal root": outputRoot, "escape": root, "prefix confusion": outside} {
t.Run(name, func(t *testing.T) {
if _, err := validateBundleRoot(outputRoot, candidate); err == nil {
t.Fatalf("validateBundleRoot(%q) error = nil", candidate)
}
})
}
symlink := filepath.Join(outputRoot, "linked")
if err := os.Symlink(outside, symlink); err != nil {
t.Skipf("Symlink() unavailable: %v", err)
}
if _, err := validateBundleRoot(outputRoot, symlink); err == nil {
t.Fatal("validateBundleRoot(symlink) error = nil")
}
}
func TestLoadIndexRejectsMalformedUnsafeAndUnsupportedDocuments(t *testing.T) {
tests := []struct {
name string
indexValue any
prepare func(*testing.T, string)
wantError string
}{
{name: "malformed", indexValue: json.RawMessage(`{"manifest_file":`)},
{name: "unsupported output shape", indexValue: map[string]any{"manifest_file": "manifest.json", "output_files": map[string]any{}, "rejected_file": "rejected.json", "warnings_file": "warnings.json"}},
{name: "missing management path", indexValue: map[string]any{"output_files": []any{}, "rejected_file": "rejected.json", "warnings_file": "warnings.json"}},
{name: "renamed manifest", indexValue: func() any {
value := validIndexValue([]any{})
value["manifest_file"] = "metadata.json"
return value
}(), wantError: `manifest_file "metadata.json"`},
{name: "cleanable manifest", indexValue: func() any {
value := validIndexValue([]any{})
value["manifest_file"] = "./manifest.json"
return value
}(), wantError: `manifest_file "./manifest.json"`},
{name: "renamed rejections", indexValue: func() any {
value := validIndexValue([]any{})
value["rejected_file"] = "rejections.json"
return value
}(), wantError: `rejected_file "rejections.json"`},
{name: "renamed warnings", indexValue: func() any {
value := validIndexValue([]any{})
value["warnings_file"] = "diagnostics/warnings.json"
return value
}(), wantError: `warnings_file "diagnostics/warnings.json"`},
{name: "duplicate lane", indexValue: validIndexValue([]any{
map[string]any{"lane_id": "npc", "file": "lanes/npc.json"},
map[string]any{"lane_id": "npc", "file": "lanes/npc.json"},
})},
{name: "absolute logical path", indexValue: validIndexValue([]any{map[string]any{"lane_id": "npc", "file": "/tmp/npc.json"}})},
{name: "lexical traversal", indexValue: validIndexValue([]any{map[string]any{"lane_id": "npc", "file": "../outside.json"}})},
{name: "root prefix confusion", indexValue: validIndexValue([]any{map[string]any{"lane_id": "npc", "file": "../bundle-other/npc.json"}})},
{name: "file symlink", indexValue: validIndexValue([]any{map[string]any{"lane_id": "npc", "file": "lanes/npc.json"}}), prepare: func(t *testing.T, bundle string) {
if err := os.Symlink(filepath.Join(bundle, "manifest.json"), filepath.Join(bundle, "lanes", "npc.json")); err != nil {
t.Skipf("Symlink() unavailable: %v", err)
}
}},
{name: "directory symlink", indexValue: validIndexValue([]any{map[string]any{"lane_id": "npc", "file": "linked/npc.json"}}), prepare: func(t *testing.T, bundle string) {
if err := os.Symlink(filepath.Join(bundle, "lanes"), filepath.Join(bundle, "linked")); err != nil {
t.Skipf("Symlink() unavailable: %v", err)
}
}},
{name: "missing management file", indexValue: validIndexValue([]any{}), prepare: func(t *testing.T, bundle string) {
if err := os.Remove(filepath.Join(bundle, "manifest.json")); err != nil {
t.Fatalf("Remove(manifest) error = %v", err)
}
}},
{name: "incomplete pipeline descriptor", indexValue: func() any {
value := validIndexValue([]any{})
value["chunk_map"] = map[string]any{"artifact_kind": "chunk_map", "file": "chunk-map.json"}
return value
}()},
{name: "pipeline descriptor escape", indexValue: func() any {
value := validIndexValue([]any{})
value["evidence_context"] = map[string]any{
"artifact_kind": "evidence_context", "file": "../evidence.json", "media_type": "application/json",
"schema_id": "evidence", "schema_name": "Evidence", "schema_version": "v1",
}
return value
}()},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
bundle := createBundleSkeleton(t)
indexPath := filepath.Join(bundle, "index.json")
if raw, ok := test.indexValue.(json.RawMessage); ok {
if err := os.WriteFile(indexPath, raw, 0o644); err != nil {
t.Fatalf("WriteFile(index) error = %v", err)
}
} else {
writeJSONFile(t, indexPath, test.indexValue)
}
if test.prepare != nil {
test.prepare(t, bundle)
}
if _, err := loadIndex(bundle, indexPath); err == nil {
t.Fatal("loadIndex() error = nil, want failure")
} else if test.wantError != "" && !strings.Contains(err.Error(), test.wantError) {
t.Fatalf("loadIndex() error = %v, want fragment %q", err, test.wantError)
}
})
}
bundle := createBundleSkeleton(t)
oversizedIndex := filepath.Join(bundle, "index.json")
if err := os.WriteFile(oversizedIndex, []byte(strings.Repeat("x", maxIndexBytes+1)), 0o644); err != nil {
t.Fatalf("WriteFile(oversized index) error = %v", err)
}
if _, err := loadIndex(bundle, oversizedIndex); err == nil || !strings.Contains(err.Error(), "exceeds") {
t.Fatalf("loadIndex(oversized) error = %v", err)
}
}
func TestLoadDiagnosticSummariesValidateBoundsAndTolerateUnknownFields(t *testing.T) {
root := t.TempDir()
rejectedPath := filepath.Join(root, "rejected.json")
warningsPath := filepath.Join(root, "warnings.json")
writeJSONFile(t, rejectedPath, map[string]any{"rejected": []any{map[string]any{
"stage": "validate", "lane_id": "spells", "reason_code": "invalid", "message": "do not retain this", "future": true,
}}, "future": true})
writeJSONFile(t, warningsPath, map[string]any{"warnings": []any{map[string]any{
"scope": "lane:spells", "reason_code": "bounded", "message": "do not retain this", "future": true,
}}, "future": true})
rejections, err := loadRejections(rejectedPath)
if err != nil || len(rejections) != 1 || rejections[0].ReasonCode != "invalid" {
t.Fatalf("loadRejections() = %#v, %v", rejections, err)
}
warnings, err := loadWarnings(warningsPath)
if err != nil || len(warnings) != 1 || warnings[0].Scope != "lane:spells" {
t.Fatalf("loadWarnings() = %#v, %v", warnings, err)
}
for name, path := range map[string]string{"rejections": rejectedPath, "warnings": warningsPath} {
t.Run("malformed "+name, func(t *testing.T) {
if err := os.WriteFile(path, []byte("{"), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
var err error
if name == "rejections" {
_, err = loadRejections(path)
} else {
_, err = loadWarnings(path)
}
if err == nil {
t.Fatal("summary decoder error = nil")
}
})
}
oversized := filepath.Join(root, "oversized.json")
if err := os.WriteFile(oversized, []byte(strings.Repeat("x", maxSummaryBytes+1)), 0o644); err != nil {
t.Fatalf("WriteFile(oversized) error = %v", err)
}
if _, err := loadWarnings(oversized); err == nil || !strings.Contains(err.Error(), "exceeds") {
t.Fatalf("loadWarnings(oversized) error = %v", err)
}
if _, err := loadRejections(oversized); err == nil || !strings.Contains(err.Error(), "exceeds") {
t.Fatalf("loadRejections(oversized) error = %v", err)
}
}
func TestFakeRunnerCapturesRequestsAndHonorsContextAndError(t *testing.T) {
req := RunRequest{PipelineID: "pipeline"}
want := RunResult{BundleRoot: "/bundle"}
fake := &FakeRunner{Result: want}
got, err := fake.Run(context.Background(), req)
if err != nil || !reflect.DeepEqual(got, want) || !reflect.DeepEqual(fake.Requests, []RunRequest{req}) {
t.Fatalf("Run() = %#v, %v; requests = %#v", got, err, fake.Requests)
}
wantErr := errors.New("configured failure")
fake.Err = wantErr
if _, err := fake.Run(context.Background(), req); !errors.Is(err, wantErr) {
t.Fatalf("Run(configured error) = %v", err)
}
canceled, cancel := context.WithCancel(context.Background())
cancel()
before := len(fake.Requests)
if _, err := fake.Run(canceled, req); !errors.Is(err, context.Canceled) || len(fake.Requests) != before {
t.Fatalf("Run(canceled) error = %v; requests = %d", err, len(fake.Requests))
}
}
func validRunRequest(t *testing.T) RunRequest {
t.Helper()
root := t.TempDir()
configPath := filepath.Join(root, "notarius.yml")
inputPath := filepath.Join(root, "input.json")
outputRoot := filepath.Join(root, "outputs")
workingDirectory := filepath.Join(root, "work")
diagnostics := filepath.Join(root, "diagnostics")
for _, directory := range []string{outputRoot, workingDirectory, diagnostics} {
if err := os.Mkdir(directory, 0o755); err != nil {
t.Fatalf("Mkdir(%q) error = %v", directory, err)
}
}
if err := os.WriteFile(configPath, []byte("pipelines: {}\n"), 0o644); err != nil {
t.Fatalf("WriteFile(config) error = %v", err)
}
if err := os.WriteFile(inputPath, []byte("{}\n"), 0o644); err != nil {
t.Fatalf("WriteFile(input) error = %v", err)
}
return RunRequest{
Binary: "notarius", ConfigPath: configPath, PipelineID: "dnd-session", InputPath: inputPath,
OutputRoot: outputRoot, WorkingDirectory: workingDirectory,
ReceiptPath: filepath.Join(diagnostics, "receipt.json"), LogPath: filepath.Join(diagnostics, "stderr.log"),
Timeout: time.Second,
}
}
func writeValidBundleAndReceipt(t *testing.T, req RunRequest, includeUnknown bool) {
t.Helper()
bundle := filepath.Join(req.OutputRoot, "notarius-run-1")
if err := os.MkdirAll(filepath.Join(bundle, "lanes"), 0o755); err != nil {
t.Fatalf("MkdirAll(bundle) error = %v", err)
}
for path, data := range map[string]string{
"manifest.json": `{}`,
"lanes/npc.json": `{}`,
"chunk-map.json": `{}`,
"evidence-context.json": `{}`,
} {
if err := os.WriteFile(filepath.Join(bundle, filepath.FromSlash(path)), []byte(data), 0o644); err != nil {
t.Fatalf("WriteFile(%q) error = %v", path, err)
}
}
rejection := map[string]any{"stage": "validate", "lane_id": "spells", "reason_code": "invalid_spell", "message": strings.Repeat("external detail", 20)}
warning := map[string]any{"scope": "lane:npc-registry", "reason_code": "normalized_name", "message": strings.Repeat("external warning", 20)}
if includeUnknown {
rejection["future"] = true
warning["future"] = true
}
writeJSONFile(t, filepath.Join(bundle, "rejected.json"), map[string]any{"rejected": []any{rejection}, "future": true})
writeJSONFile(t, filepath.Join(bundle, "warnings.json"), map[string]any{"warnings": []any{warning}, "future": true})
index := validIndexValue([]any{map[string]any{
"lane_id": "npc-registry", "file": "lanes/npc.json", "media_type": "application/json",
"module_key": "dnd/npc-registry", "schema_id": "notarius.dnd.npc_registry",
"schema_name": "NPCRegistry", "schema_version": "v1", "future": true,
}})
index["chunk_map"] = map[string]any{
"artifact_kind": "chunk_map", "file": "chunk-map.json", "media_type": "application/json",
"schema_id": "notarius.chunk_map", "schema_name": "ChunkMap", "schema_version": "v1", "future": true,
}
index["evidence_context"] = map[string]any{
"artifact_kind": "evidence_context", "file": "evidence-context.json", "media_type": "application/json",
"schema_id": "notarius.evidence_context", "schema_name": "EvidenceContext", "schema_version": "v1", "future": true,
}
index["future"] = true
writeJSONFile(t, filepath.Join(bundle, "index.json"), index)
receipt := map[string]any{
"schema_version": ReceiptSchemaVersion, "run_id": "notarius-run-1", "pipeline_id": req.PipelineID,
"output_directory": bundle, "index_file": "index.json", "normalized_output_count": 1,
"rejected_output_count": 1, "warning_count": 1, "validation_status": "rejected",
}
if includeUnknown {
receipt["future"] = true
}
writeJSONFile(t, req.ReceiptPath, receipt)
}
func createBundleSkeleton(t *testing.T) string {
t.Helper()
bundle := filepath.Join(t.TempDir(), "bundle")
if err := os.MkdirAll(filepath.Join(bundle, "lanes"), 0o755); err != nil {
t.Fatalf("MkdirAll(bundle) error = %v", err)
}
for _, name := range []string{"manifest.json", "rejected.json", "warnings.json", "lanes/npc.json", "chunk-map.json"} {
if err := os.WriteFile(filepath.Join(bundle, filepath.FromSlash(name)), []byte("{}"), 0o644); err != nil {
t.Fatalf("WriteFile(%q) error = %v", name, err)
}
}
return bundle
}
func validIndexValue(lanes []any) map[string]any {
return map[string]any{
"manifest_file": "manifest.json", "output_files": lanes,
"rejected_file": "rejected.json", "warnings_file": "warnings.json",
}
}
func writeJSONFile(t *testing.T, path string, value any) {
t.Helper()
data, err := json.Marshal(value)
if err != nil {
t.Fatalf("json.Marshal() error = %v", err)
}
if err := os.WriteFile(path, data, 0o644); err != nil {
t.Fatalf("WriteFile(%q) error = %v", path, err)
}
}
func writeShellScript(t *testing.T, body string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "notarius-helper")
if err := os.WriteFile(path, []byte(body), 0o755); err != nil {
t.Fatalf("WriteFile(script) error = %v", err)
}
return path
}
func assertTextFile(t *testing.T, path, want string) {
t.Helper()
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("ReadFile(%q) error = %v", path, err)
}
if string(data) != want {
t.Fatalf("ReadFile(%q) = %q, want %q", path, string(data), want)
}
}
func cloneMap(source map[string]any) map[string]any {
result := make(map[string]any, len(source))
for key, value := range source {
result[key] = value
}
return result
}

View File

@@ -21,7 +21,7 @@ func TestExecuteRunStageArtifactsUnsupportedStageFails(t *testing.T) {
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute(
[]string{"run-stage", "polish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
[]string{"run-stage", "extract", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath, "--artifacts", "session_recap"},
&stdout,
&stderr,
)
@@ -130,6 +130,7 @@ func TestRunArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
seed.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
}
seed.MarkStageSkipped("extract", time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), "notarius_disabled")
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
t.Fatalf("save manifest: %v", err)
}
@@ -143,7 +144,7 @@ func TestRunArtifactsWithSucceededAnalyzeSkipsUnlessForced(t *testing.T) {
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if !strings.Contains(out.String(), "executed=0 skipped=10") {
if !strings.Contains(out.String(), "executed=1 skipped=11") {
t.Fatalf("output = %q, want all stages skipped", out.String())
}
}

View File

@@ -51,8 +51,3 @@ func Execute(args []string, stdout, stderr io.Writer) int {
func printUsage(w io.Writer) {
fmt.Fprintf(w, "Usage: narratio <%s>\n", strings.Join(supportedCommands, "|"))
}
func placeholder(out io.Writer, command string) error {
_, err := fmt.Fprintf(out, "narratio %s: not yet implemented\n", command)
return err
}

View File

@@ -31,8 +31,8 @@ func TestExecuteValidCommands(t *testing.T) {
args []string
wantOut string
}{
{name: "run", args: []string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run: session 2026-05-03; executed=10 skipped=0; manifest="},
{name: "session plan", args: []string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "prepare: skip\ntranscribe: skip\nmerge: skip\npolish: skip\nnormalize: skip\ntrim: skip\nrender: skip\nanalyze: skip\npublish: skip\nnotify: skip"},
{name: "run", args: []string{"run", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run: session 2026-05-03; executed=11 skipped=1; manifest="},
{name: "session plan", args: []string{"session", "plan", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "prepare: skip\ntranscribe: skip\nmerge: skip\npolish: skip\nnormalize: skip\ntrim: skip\nextract: run\nrender: skip\nanalyze: skip\npublish: skip\nnotify: skip"},
{name: "session status", args: []string{"session", "status", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "Session: 2026-05-03"},
{name: "run-stage", args: []string{"run-stage", "polish", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, wantOut: "narratio run-stage: stage=polish executed=0 skipped=1 force=false; manifest="},
}
@@ -227,6 +227,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
t.Fatalf("write pipeline.yml: %v", err)
@@ -290,6 +292,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
t.Fatalf("write pipeline.yml: %v", err)
@@ -331,7 +335,7 @@ func TestExecuteUsesDefaultPipelineConfigPathWhenConfigFlagOmitted(t *testing.T)
if code != 0 {
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
}
if !strings.Contains(stdout.String(), "narratio run: session 2026-05-03; executed=10 skipped=0; manifest=") {
if !strings.Contains(stdout.String(), "narratio run: session 2026-05-03; executed=11 skipped=1; manifest=") {
t.Fatalf("stdout = %q, want successful run output", stdout.String())
}
}
@@ -385,10 +389,14 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`)
mustWriteTestFile(t, filepath.Join(otherDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
mustWriteTestFile(t, filepath.Join(otherDir, "autocorrect.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(otherDir, "glossary.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(otherDir, "players.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(otherDir, "party.yml"), "[]\n")
var stdout bytes.Buffer
var stderr bytes.Buffer
@@ -466,10 +474,13 @@ func writeValidConfigFiles(t *testing.T, workspaceRoot string, transcribeURL ...
url = transcribeURL[0]
}
seriatimBinary := writeSeriatimAppTestWrapper(t)
scriptoriumBinary := writeScriptoriumAppTestWrapper(t)
auditaBinary := writeAuditaAppTestWrapper(t)
t.Setenv("GO_WANT_APP_SERIATIM_HELPER", "1")
t.Setenv("GO_WANT_APP_SCRIPTORIUM_HELPER", "1")
t.Setenv("GO_WANT_APP_AUDITA_HELPER", "1")
t.Setenv("AUDITA_LLM_API_KEY", "test-audita-key")
t.Setenv("PATH", filepath.Dir(scriptoriumBinary)+string(os.PathListSeparator)+os.Getenv("PATH"))
pipelineYAML := `workspace:
root: ` + workspaceRoot + `
@@ -514,6 +525,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
@@ -532,6 +545,8 @@ inputs:
mustWriteTestFile(t, filepath.Join(campaignDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
mustWriteTestFile(t, filepath.Join(campaignDir, "autocorrect.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(campaignDir, "glossary.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(campaignDir, "players.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(campaignDir, "party.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(dir, "audio", "alice.flac"), "audio-bytes")
return pipelinePath, campaignPath, sessionPath
@@ -545,6 +560,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`
if err := os.WriteFile(campaignPath, []byte(campaignYAML), 0o644); err != nil {
t.Fatalf("write campaign.yml: %v", err)
@@ -591,6 +608,60 @@ func writeSeriatimAppTestWrapper(t *testing.T) string {
return path
}
func writeScriptoriumAppTestWrapper(t *testing.T) string {
t.Helper()
exe, err := os.Executable()
if err != nil {
t.Fatalf("os.Executable() error = %v", err)
}
path := filepath.Join(t.TempDir(), "scriptorium")
content := "#!/bin/sh\nexec \"" + exe + "\" -test.run=TestScriptoriumAppHelper -- \"$@\"\n"
if err := os.WriteFile(path, []byte(content), 0o755); err != nil {
t.Fatalf("WriteFile(%q): %v", path, err)
}
return path
}
func TestScriptoriumAppHelper(t *testing.T) {
if os.Getenv("GO_WANT_APP_SCRIPTORIUM_HELPER") != "1" {
return
}
args := os.Args
start := -1
for i := range args {
if args[i] == "--" {
start = i + 1
break
}
}
if start < 0 || start >= len(args) {
_, _ = os.Stderr.WriteString("missing -- args separator\n")
os.Exit(2)
}
runArgs := args[start:]
outputPath := appSeriatimFlagValue(runArgs, "--out")
if strings.TrimSpace(outputPath) == "" {
outputPath = appSeriatimFlagValue(runArgs, "--output")
}
if strings.TrimSpace(outputPath) == "" {
_, _ = os.Stderr.WriteString("missing output flag\n")
os.Exit(2)
}
if err := os.MkdirAll(filepath.Dir(outputPath), 0o755); err != nil {
_, _ = os.Stderr.WriteString(fmt.Sprintf("mkdir output dir: %v\n", err))
os.Exit(2)
}
if err := os.WriteFile(outputPath, []byte(`{"trim_action":"copy","warnings":[]}`), 0o644); err != nil {
_, _ = os.Stderr.WriteString(fmt.Sprintf("write output: %v\n", err))
os.Exit(2)
}
_, _ = os.Stdout.WriteString("scriptorium helper stdout\n")
_, _ = os.Stderr.WriteString("scriptorium helper stderr\n")
os.Exit(0)
}
func TestSeriatimAppHelper(t *testing.T) {
if os.Getenv("GO_WANT_APP_SERIATIM_HELPER") != "1" {
return

View File

@@ -0,0 +1,416 @@
package app
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/notarius"
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
"gitea.maximumdirect.net/eric/narratio/internal/stage"
)
type materializingNotariusRunner struct {
cfg *config.NotariusConfig
requests []notarius.RunRequest
failuresRemaining int
}
func (r *materializingNotariusRunner) Run(_ context.Context, req notarius.RunRequest) (notarius.RunResult, error) {
r.requests = append(r.requests, req)
if r.failuresRemaining > 0 {
r.failuresRemaining--
return notarius.RunResult{}, errors.New("notarius execution failed")
}
externalRunID := fmt.Sprintf("notarius-run-%d", len(r.requests))
bundle := filepath.Join(req.OutputRoot, externalRunID)
lanesDir := filepath.Join(bundle, "lanes")
if err := os.MkdirAll(lanesDir, 0o755); err != nil {
return notarius.RunResult{}, err
}
for path, content := range map[string]string{
filepath.Join(bundle, "index.json"): `{"manifest_file":"manifest.json"}`,
filepath.Join(bundle, "manifest.json"): `{}`,
filepath.Join(bundle, "rejected.json"): `{"rejected":[]}`,
filepath.Join(bundle, "warnings.json"): `{"warnings":[]}`,
filepath.Join(lanesDir, "npcs.json"): `{"npcs":[]}`,
} {
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
return notarius.RunResult{}, err
}
}
output := r.cfg.Outputs["npc_registry"]
return notarius.RunResult{
Receipt: notarius.Receipt{
SchemaVersion: notarius.ReceiptSchemaVersion, RunID: externalRunID,
PipelineID: req.PipelineID, OutputDirectory: bundle, IndexFile: "index.json",
NormalizedOutputCount: 1, ValidationStatus: "valid",
},
BundleRoot: bundle,
Index: notarius.Index{
Path: filepath.Join(bundle, "index.json"), RejectedPath: filepath.Join(bundle, "rejected.json"),
WarningsPath: filepath.Join(bundle, "warnings.json"),
Lanes: []notarius.LaneDescriptor{{
LaneID: output.LaneID, File: "lanes/npcs.json", Path: filepath.Join(lanesDir, "npcs.json"),
MediaType: output.MediaType, SchemaID: output.SchemaID,
SchemaVersion: output.SchemaVersion, ModuleKey: output.ModuleKey,
}},
},
}, nil
}
func TestExtractLifecycleDisabledThenEnabled(t *testing.T) {
cfg, env, runner := extractionLifecycleFixture(t, false)
plan, err := BuildSingleStagePlan("extract")
if err != nil {
t.Fatalf("BuildSingleStagePlan(extract) error = %v", err)
}
first, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
if err != nil {
t.Fatalf("disabled executeStages() error = %v", err)
}
if len(first.Executed) != 1 || len(first.Skipped) != 1 || first.Skipped[0] != "extract" || len(runner.requests) != 0 {
t.Fatalf("disabled summary = %#v requests=%d", first, len(runner.requests))
}
cfg.Pipeline.Notarius.Enabled = true
second, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
if err != nil {
t.Fatalf("enabled executeStages() error = %v", err)
}
if len(second.Executed) != 1 || len(second.Skipped) != 0 || len(runner.requests) != 1 {
t.Fatalf("enabled summary = %#v requests=%d", second, len(runner.requests))
}
loaded, err := (&manifest.LocalStore{}).Load(context.Background(), second.ManifestPath)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
if loaded.Stages["extract"] == nil || loaded.Stages["extract"].Status != manifest.StatusSucceeded || len(loaded.Stages["extract"].Outputs) != 2 {
t.Fatalf("extract record = %#v, want succeeded manifest-ready outputs", loaded.Stages["extract"])
}
}
func TestExtractLifecycleChangedOutcomeRerunsSucceededDownstream(t *testing.T) {
cfg, env, runner := extractionLifecycleFixture(t, false)
analyzeRuns := 0
plan := extractionLifecyclePlan(t, &analyzeRuns)
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env}); err != nil {
t.Fatalf("disabled executeStages() error = %v", err)
}
if analyzeRuns != 1 || len(runner.requests) != 0 {
t.Fatalf("disabled run analyze=%d Notarius=%d, want 1 and 0", analyzeRuns, len(runner.requests))
}
cfg.Pipeline.Notarius.Enabled = true
summary, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
if err != nil {
t.Fatalf("enabled executeStages() error = %v", err)
}
if analyzeRuns != 2 || len(runner.requests) != 1 {
t.Fatalf("enabled run analyze=%d Notarius=%d, want 2 and 1", analyzeRuns, len(runner.requests))
}
if len(summary.Executed) != 2 || len(summary.Skipped) != 0 {
t.Fatalf("enabled summary = %#v, want extract and analyze executed", summary)
}
}
func TestExtractLifecycleFailureInvalidatesAndOrdinaryRetryRerunsDownstream(t *testing.T) {
cfg, env, runner := extractionLifecycleFixture(t, true)
runner.failuresRemaining = 1
markLifecycleStageSucceeded(t, cfg, "analyze")
analyzeRuns := 0
plan := extractionLifecyclePlan(t, &analyzeRuns)
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env}); err == nil || !strings.Contains(err.Error(), "notarius execution failed") {
t.Fatalf("failed executeStages() error = %v", err)
}
failed := loadLifecycleManifest(t, cfg)
if failed.Stages["extract"].Status != manifest.StatusFailed || failed.Stages["analyze"].Status != manifest.StatusStale {
t.Fatalf("failed lifecycle extract=%#v analyze=%#v", failed.Stages["extract"], failed.Stages["analyze"])
}
if failed.Stages["analyze"].Error == nil || failed.Stages["analyze"].Error.Message != staleReasonFailure {
t.Fatalf("analyze stale reason = %#v, want %q", failed.Stages["analyze"].Error, staleReasonFailure)
}
summary, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
if err != nil {
t.Fatalf("retry executeStages() error = %v", err)
}
if analyzeRuns != 1 || len(runner.requests) != 2 || len(summary.Executed) != 2 {
t.Fatalf("retry analyze=%d Notarius=%d summary=%#v", analyzeRuns, len(runner.requests), summary)
}
}
func TestExtractLifecycleForcedSelfSkipInvalidatesDownstream(t *testing.T) {
cfg, env, _ := extractionLifecycleFixture(t, false)
markLifecycleStageSucceeded(t, cfg, "analyze")
plan, _ := BuildSingleStagePlan("extract")
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env, Force: true}); err != nil {
t.Fatalf("executeStages() error = %v", err)
}
loaded := loadLifecycleManifest(t, cfg)
if loaded.Stages["extract"].Status != manifest.StatusSkipped || loaded.Stages["analyze"].Status != manifest.StatusStale {
t.Fatalf("forced self-skip extract=%#v analyze=%#v", loaded.Stages["extract"], loaded.Stages["analyze"])
}
if loaded.Stages["analyze"].Error == nil || loaded.Stages["analyze"].Error.Message != staleReasonForcedReplacement {
t.Fatalf("analyze stale reason = %#v, want %q", loaded.Stages["analyze"].Error, staleReasonForcedReplacement)
}
}
func TestExtractLifecycleForcedFailureInvalidatesDownstream(t *testing.T) {
cfg, env, runner := extractionLifecycleFixture(t, true)
plan, _ := BuildSingleStagePlan("extract")
first, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
if err != nil {
t.Fatalf("initial executeStages() error = %v", err)
}
succeeded := loadLifecycleManifest(t, cfg).Stages["extract"]
if succeeded == nil || succeeded.Status != manifest.StatusSucceeded || len(succeeded.Outputs) == 0 || len(succeeded.Logs) == 0 || len(succeeded.Metadata) == 0 {
t.Fatalf("initial extraction result = %#v, want succeeded result details", succeeded)
}
markLifecycleStageSucceeded(t, cfg, "analyze")
runner.failuresRemaining = 1
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env, Force: true}); err == nil {
t.Fatal("executeStages() error = nil, want forced extraction failure")
}
loaded := loadLifecycleManifest(t, cfg)
if loaded.Stages["extract"].Status != manifest.StatusFailed || loaded.Stages["analyze"].Status != manifest.StatusStale {
t.Fatalf("forced failure extract=%#v analyze=%#v", loaded.Stages["extract"], loaded.Stages["analyze"])
}
if loaded.Stages["analyze"].Error == nil || loaded.Stages["analyze"].Error.Message != staleReasonForcedReplacement {
t.Fatalf("analyze stale reason = %#v, want %q", loaded.Stages["analyze"].Error, staleReasonForcedReplacement)
}
failed := loaded.Stages["extract"]
if len(failed.Outputs) != 0 || len(failed.Logs) != 0 || len(failed.GeneratedConfigs) != 0 || len(failed.Metadata) != 0 {
t.Fatalf("failed replacement inherited extraction result details: %#v", failed)
}
historical, err := (&manifest.LocalStore{}).LoadRun(context.Background(), first.RunManifestPath)
if err != nil {
t.Fatalf("LoadRun(initial) error = %v", err)
}
historicalExtract := historical.Stages["extract"]
if historicalExtract == nil || historicalExtract.Status != manifest.StatusSucceeded || len(historicalExtract.Outputs) == 0 || len(historicalExtract.Logs) == 0 || len(historicalExtract.Metadata) == 0 {
t.Fatalf("historical extraction result = %#v, want preserved succeeded details", historicalExtract)
}
if _, err := os.Stat(succeeded.Outputs[0].LocalPath); err != nil {
t.Fatalf("durable extraction output was not preserved: %v", err)
}
}
func TestExtractLifecycleRepeatedSelfSkipPreservesSucceededDownstream(t *testing.T) {
cfg, env, runner := extractionLifecycleFixture(t, false)
analyzeRuns := 0
plan := extractionLifecyclePlan(t, &analyzeRuns)
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env}); err != nil {
t.Fatalf("first executeStages() error = %v", err)
}
second, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
if err != nil {
t.Fatalf("second executeStages() error = %v", err)
}
if analyzeRuns != 1 || len(runner.requests) != 0 {
t.Fatalf("repeated disabled run analyze=%d Notarius=%d, want 1 and 0", analyzeRuns, len(runner.requests))
}
if len(second.Executed) != 1 || len(second.Skipped) != 2 {
t.Fatalf("second summary = %#v, want executed self-skip and skipped analyze", second)
}
loaded := loadLifecycleManifest(t, cfg)
if loaded.Stages["analyze"].Status != manifest.StatusSucceeded {
t.Fatalf("analyze = %#v, want succeeded", loaded.Stages["analyze"])
}
}
func TestExtractLifecycleSkipsCurrentResumableResult(t *testing.T) {
cfg, env, runner := extractionLifecycleFixture(t, true)
analyzeRuns := 0
plan := extractionLifecyclePlan(t, &analyzeRuns)
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env}); err != nil {
t.Fatalf("first executeStages() error = %v", err)
}
resumed, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
if err != nil {
t.Fatalf("resume executeStages() error = %v", err)
}
if len(resumed.Executed) != 0 || len(resumed.Skipped) != 2 || len(runner.requests) != 1 || analyzeRuns != 1 {
t.Fatalf("resume summary = %#v requests=%d analyze=%d", resumed, len(runner.requests), analyzeRuns)
}
}
func TestExtractLifecycleResumesAndRerunsObsoleteResults(t *testing.T) {
for _, test := range []struct {
name string
mutate func(*testing.T, *config.Config, *manifest.Manifest)
}{
{name: "configuration changed", mutate: func(_ *testing.T, cfg *config.Config, _ *manifest.Manifest) {
output := cfg.Pipeline.Notarius.Outputs["npc_registry"]
output.SchemaVersion = "v2"
cfg.Pipeline.Notarius.Outputs["npc_registry"] = output
}},
{name: "payload missing", mutate: func(t *testing.T, _ *config.Config, m *manifest.Manifest) {
if err := os.Remove(m.Stages["extract"].Outputs[0].LocalPath); err != nil {
t.Fatalf("Remove() error = %v", err)
}
}},
{name: "payload tampered", mutate: func(t *testing.T, _ *config.Config, m *manifest.Manifest) {
if err := os.WriteFile(m.Stages["extract"].Outputs[0].LocalPath, []byte(`{"npcs":["tampered"]}`), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
}},
{name: "record incompatible", mutate: func(_ *testing.T, _ *config.Config, m *manifest.Manifest) {
m.Stages["extract"].Outputs[0].Contract.SchemaID = "incompatible"
}},
} {
t.Run(test.name, func(t *testing.T) {
cfg, env, runner := extractionLifecycleFixture(t, true)
plan, _ := BuildSingleStagePlan("extract")
first, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
if err != nil {
t.Fatalf("first executeStages() error = %v", err)
}
persisted, err := (&manifest.LocalStore{}).Load(context.Background(), first.ManifestPath)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
test.mutate(t, cfg, persisted)
if err := (&manifest.LocalStore{}).Save(context.Background(), first.ManifestPath, persisted); err != nil {
t.Fatalf("Save(mutated) error = %v", err)
}
rerun, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
if err != nil {
t.Fatalf("rerun executeStages() error = %v", err)
}
if len(rerun.Executed) != 1 || len(rerun.Skipped) != 0 || len(runner.requests) != 2 {
t.Fatalf("rerun summary = %#v requests=%d", rerun, len(runner.requests))
}
})
}
}
func TestExtractLifecycleUnsafeResumeErrorPreservesSuccess(t *testing.T) {
cfg, env, runner := extractionLifecycleFixture(t, true)
analyzeRuns := 0
plan := extractionLifecyclePlan(t, &analyzeRuns)
first, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env})
if err != nil {
t.Fatalf("first executeStages() error = %v", err)
}
store := &manifest.LocalStore{}
persisted, err := store.Load(context.Background(), first.ManifestPath)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
persisted.Stages["extract"].Outputs[0].LocalPath = filepath.Join(cfg.Pipeline.Workspace.Root, "outside.json")
if err := store.Save(context.Background(), first.ManifestPath, persisted); err != nil {
t.Fatalf("Save() error = %v", err)
}
before, _ := json.Marshal(map[string]*manifest.StageRecord{
"extract": persisted.Stages["extract"],
"analyze": persisted.Stages["analyze"],
})
if _, err := executeStages(context.Background(), cfg, plan, RunOptions{Env: env}); err == nil || !strings.Contains(err.Error(), "unsafe") {
t.Fatalf("executeStages() error = %v, want unsafe resume failure", err)
}
afterManifest, err := store.Load(context.Background(), first.ManifestPath)
if err != nil {
t.Fatalf("Load(after) error = %v", err)
}
after, _ := json.Marshal(map[string]*manifest.StageRecord{
"extract": afterManifest.Stages["extract"],
"analyze": afterManifest.Stages["analyze"],
})
if string(before) != string(after) || len(runner.requests) != 1 || analyzeRuns != 1 {
t.Fatalf("successful records changed: before=%s after=%s requests=%d analyze=%d", before, after, len(runner.requests), analyzeRuns)
}
}
func extractionLifecyclePlan(t *testing.T, analyzeRuns *int) []stage.Stage {
t.Helper()
plan, err := BuildSingleStagePlan("extract")
if err != nil {
t.Fatalf("BuildSingleStagePlan(extract) error = %v", err)
}
return append(plan, countingStage{name: "analyze", runs: analyzeRuns})
}
func loadLifecycleManifest(t *testing.T, cfg *config.Config) *manifest.Manifest {
t.Helper()
loaded, err := (&manifest.LocalStore{}).Load(context.Background(), manifestPathFor(cfg))
if err != nil {
t.Fatalf("Load() error = %v", err)
}
return loaded
}
func markLifecycleStageSucceeded(t *testing.T, cfg *config.Config, name string) {
t.Helper()
loaded := loadLifecycleManifest(t, cfg)
loaded.MarkStageSucceeded(name, time.Now().UTC(), nil)
if err := (&manifest.LocalStore{}).Save(context.Background(), manifestPathFor(cfg), loaded); err != nil {
t.Fatalf("Save() error = %v", err)
}
}
func extractionLifecycleFixture(t *testing.T, enabled bool) (*config.Config, *stage.Env, *materializingNotariusRunner) {
t.Helper()
cfg := testConfig(t)
root := cfg.Pipeline.Workspace.Root
binary := filepath.Join(root, "notarius")
configPath := filepath.Join(root, "notarius.yml")
workingDirectory := filepath.Join(root, "notarius-work")
if err := os.WriteFile(binary, []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil {
t.Fatalf("WriteFile(binary) error = %v", err)
}
if err := os.WriteFile(configPath, []byte("pipelines: {}\n"), 0o644); err != nil {
t.Fatalf("WriteFile(config) error = %v", err)
}
if err := os.Mkdir(workingDirectory, 0o755); err != nil {
t.Fatalf("Mkdir(working directory) error = %v", err)
}
cfg.Pipeline.Notarius = &config.NotariusConfig{
Enabled: enabled, Binary: binary, ConfigPath: configPath, PipelineID: "dnd-session",
Timeout: "45m", WorkingDirectory: workingDirectory,
Outputs: map[string]config.NotariusOutputConfig{
"npc_registry": {
LaneID: "npc-registry", MediaType: "application/json", SchemaID: "notarius.dnd.npc_registry",
SchemaVersion: "v1", ModuleKey: "dnd/npc-registry",
},
},
}
paths, err := artifacts.NewLocalStore(root).EnsureLayoutFor(cfg.Session.Campaign, cfg.Session.SessionID)
if err != nil {
t.Fatalf("EnsureLayoutFor() error = %v", err)
}
inputPath := filepath.Join(paths.ArtifactsDir, "trimmed.from-manifest.json")
if err := os.WriteFile(inputPath, []byte(`{"segments":[]}`), 0o644); err != nil {
t.Fatalf("WriteFile(input) error = %v", err)
}
m := manifest.New(cfg.Session.SessionID, time.Now().UTC())
m.Campaign = cfg.Session.Campaign
m.MarkStageSucceeded("trim", time.Now().UTC(), []manifest.ArtifactRecord{{
Kind: artifactmodel.TranscriptOutputKindFinalTrimmed, SourceID: artifactmodel.SourceTranscriptFinalTrimmed,
LocalPath: inputPath,
}})
if err := (&manifest.LocalStore{}).Save(context.Background(), paths.ManifestPath, m); err != nil {
t.Fatalf("Save(seed) error = %v", err)
}
runner := &materializingNotariusRunner{cfg: cfg.Pipeline.Notarius}
return cfg, &stage.Env{Notarius: runner}, runner
}

View File

@@ -10,9 +10,10 @@ import (
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
func buildHelperArtifactCatalog(cfg *config.Config) (*artifacts.ArtifactCatalog, error) {
func buildHelperArtifactCatalog(cfg *config.Config, m *manifest.Manifest) (*artifacts.ArtifactCatalog, error) {
catalog := artifacts.NewArtifactCatalog()
if err := catalog.RegisterBuiltIns(); err != nil {
return nil, err
@@ -26,6 +27,14 @@ func buildHelperArtifactCatalog(cfg *config.Config) (*artifacts.ArtifactCatalog,
if err := catalog.RegisterConfiguredArtifacts(configured, nil); err != nil {
return nil, err
}
extractionDefinitions := artifacts.ExtractionDefinitionsFromConfig(cfg.Pipeline.Notarius)
if err := catalog.RegisterExtractionArtifacts(extractionDefinitions); err != nil {
return nil, err
}
paths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root).SessionPathsFor(cfg.Session.Campaign, cfg.Session.SessionID)
if cfg.Pipeline.Notarius != nil && cfg.Pipeline.Notarius.Enabled {
catalog.HydrateExtractionArtifacts(paths, m, extractionDefinitions)
}
return catalog, nil
}
@@ -40,6 +49,14 @@ func writeArtifactList(out io.Writer, cfg *config.Config, catalog *artifacts.Art
for _, entry := range catalog.ListConfigured() {
writeArtifactLine(out, entry.SourceID, lockSet)
}
fmt.Fprintln(out, "Extraction:")
for _, entry := range catalog.ListExtraction() {
state := "unavailable"
if entry.Available {
state = "available"
}
writeExtractionArtifactLine(out, entry.SourceID, state, entry.Provenance, lockSet)
}
fmt.Fprintln(out, "Previous-session:")
for _, req := range artifacts.CollectPreviousArtifactRequirements(configuredScriptoriumArtifacts(cfg)) {
fmt.Fprintf(out, "- %s required=%t\n", artifactpolicy.PreviousSessionSourceID(req.Name), req.Required)
@@ -50,6 +67,17 @@ func writeArtifactList(out io.Writer, cfg *config.Config, catalog *artifacts.Art
}
}
func writeExtractionArtifactLine(out io.Writer, source, state, provenance string, lockSet map[string]config.PublishLockRule) {
parts := []string{source, "planned", state}
if strings.TrimSpace(provenance) != "" {
parts = append(parts, "provenance="+strings.TrimSpace(provenance))
}
if _, ok := lockSet[source]; ok {
parts = append(parts, "locked")
}
fmt.Fprintf(out, "- %s\n", strings.Join(parts, " "))
}
func writeArtifactLine(out io.Writer, source string, lockSet map[string]config.PublishLockRule) {
parts := []string{source}
if _, ok := lockSet[source]; ok {
@@ -103,7 +131,12 @@ func remotePublishedOutputAvailability(ctx context.Context, cfg *config.Config,
func helperPublishedOutputDest(rule config.PublishOutputRule, catalog *artifacts.ArtifactCatalog) (string, bool, error) {
source := strings.TrimSpace(rule.Source)
normalized, err := artifactpolicy.ResolvePublishedDestination(source, rule.Dest, helperConfiguredOutputPathMap(catalog))
normalized, err := artifactpolicy.ResolvePublishedDestinationWithExtractions(
source,
rule.Dest,
helperConfiguredOutputPathMap(catalog),
helperExtractionOutputSet(catalog),
)
if err != nil {
return "", false, err
}
@@ -112,6 +145,19 @@ func helperPublishedOutputDest(rule config.PublishOutputRule, catalog *artifacts
return normalized, showDest, nil
}
func helperExtractionOutputSet(catalog *artifacts.ArtifactCatalog) map[string]struct{} {
out := map[string]struct{}{}
if catalog == nil {
return out
}
for _, entry := range catalog.ListExtraction() {
if strings.TrimSpace(entry.ExtractionKey) != "" {
out[entry.ExtractionKey] = struct{}{}
}
}
return out
}
func helperConfiguredOutputPathMap(catalog *artifacts.ArtifactCatalog) map[string]string {
out := map[string]string{}
if catalog == nil {

View File

@@ -22,11 +22,11 @@ func ArtifactsList(ctx context.Context, args []string, out io.Writer) error {
if strings.TrimSpace(flags.sessionID) == "" {
return fmt.Errorf("artifacts list: session_id is required")
}
cfg, store, locks, _, err := loadHelperContext(ctx, flags, remote)
cfg, store, locks, m, err := loadHelperContext(ctx, flags, remote)
if err != nil {
return fmt.Errorf("artifacts list: %w", err)
}
catalog, err := buildHelperArtifactCatalog(cfg)
catalog, err := buildHelperArtifactCatalog(cfg, m)
if err != nil {
return fmt.Errorf("artifacts list: %w", err)
}

View File

@@ -8,8 +8,10 @@ import (
"path/filepath"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
@@ -109,12 +111,16 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`), 0o644); err != nil {
t.Fatalf("write explicit campaign: %v", err)
}
mustWriteTestFile(t, filepath.Join(explicitDir, "speakers.yml"), "match:\n - speaker: Alice\n match: [\"alice\"]\n")
mustWriteTestFile(t, filepath.Join(explicitDir, "autocorrect.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(explicitDir, "glossary.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(explicitDir, "players.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(explicitDir, "party.yml"), "[]\n")
fake := &storage.FakeBackend{}
var storeInitCalls int
@@ -445,6 +451,9 @@ inputs:
if !strings.Contains(stdout.String(), "OK audio") {
t.Fatalf("stdout = %q, want OK audio", stdout.String())
}
if !strings.Contains(stdout.String(), "OK inputs players:") || !strings.Contains(stdout.String(), "OK inputs party:") {
t.Fatalf("stdout = %q, want players and party input findings", stdout.String())
}
}
func TestExecuteLocksAddListAndRemoveUseRemoteLockStore(t *testing.T) {
@@ -501,7 +510,7 @@ func TestExecuteLocksAddListAndRemoveUseRemoteLockStore(t *testing.T) {
if code != 0 {
t.Fatalf("locks remove exit code = %d, want 0; stderr=%q", code, stderr.String())
}
store, err := config.LoadPublishLockStoreBytes("locks.yml", fake.Objects[key].Data, nil)
store, err := config.LoadPublishLockStoreBytes("locks.yml", fake.Objects[key].Data, nil, nil)
if err != nil {
t.Fatalf("LoadPublishLockStoreBytes() error = %v", err)
}
@@ -765,6 +774,71 @@ func TestExecuteArtifactsListRemoteReportsPublishedAvailability(t *testing.T) {
}
}
func TestExecuteArtifactsListReportsExtractionLifecycleWithoutPayload(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
addExtractionOutputToPipeline(t, pipelinePath)
addPublishOutputsToPipeline(t, pipelinePath, `
outputs:
- source: narratio.extraction.encounters
dest: artifacts/encounters.json
required: true
`)
lanePath := writeOperatorExtractionManifest(t, workspaceRoot)
fake := &storage.FakeBackend{}
publishedKey := artifacts.S3PublishedOutputKey(
artifacts.S3SessionPrefix("dnd", "sample-campaign", "2026-05-03"),
"artifacts/encounters.json",
)
fake.SeedObject(storage.FakeObject{Key: publishedKey, Data: []byte(`{"secret":"DO_NOT_PRINT"}`)})
var storeInitCalls int
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute([]string{
"session", "artifacts", "2026-05-03",
"--config", pipelinePath,
"--campaign-file", campaignPath,
"--session", sessionPath,
"--remote",
}, &stdout, &stderr)
if code != 0 {
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
}
out := stdout.String()
for _, want := range []string{
"Extraction:",
"narratio.extraction.encounters planned available provenance=manifest.current_extract_run",
"narratio.extraction.encounters dest=artifacts/encounters.json remote=published",
} {
if !strings.Contains(out, want) {
t.Fatalf("stdout = %q, want %q", out, want)
}
}
if strings.Contains(out, "DO_NOT_PRINT") {
t.Fatalf("operator output exposed extraction payload: %q", out)
}
if err := os.Remove(lanePath); err != nil {
t.Fatal(err)
}
stdout.Reset()
stderr.Reset()
code = Execute([]string{
"session", "artifacts", "2026-05-03",
"--config", pipelinePath,
"--campaign-file", campaignPath,
"--session", sessionPath,
}, &stdout, &stderr)
if code != 0 {
t.Fatalf("unavailable exit code = %d, want 0; stderr=%q", code, stderr.String())
}
if !strings.Contains(stdout.String(), "narratio.extraction.encounters planned unavailable") {
t.Fatalf("stdout = %q, want unavailable extraction state", stdout.String())
}
}
func TestExecuteArtifactsListRemoteUsesPublishOutputDestinations(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
@@ -862,6 +936,8 @@ func TestExecuteStatusReportsRemoteArtifactCatalog(t *testing.T) {
"narratio.transcript.final_trimmed locked",
"narratio.transcript.final_trimmed locked remote=published",
"narratio.transcript.final dest=transcripts/full.json remote=published",
"Stable input players:",
"Stable input party:",
} {
if !strings.Contains(out, want) {
t.Fatalf("stdout = %q, want %q", out, want)
@@ -997,7 +1073,7 @@ func TestExecutePublishLoadsRemoteLocks(t *testing.T) {
restoreAppConfigTestGlobals(t, fake, &storeInitCalls, []string{sessionPath})
workRoot := filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03")
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze"} {
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze"} {
// The publish stage only checks the manifest statuses and source files.
_ = stageName
}
@@ -1032,6 +1108,72 @@ func addPublishOutputsToPipeline(t *testing.T, pipelinePath, publishYAML string)
}
}
func addExtractionOutputToPipeline(t *testing.T, pipelinePath string) {
t.Helper()
data, err := os.ReadFile(pipelinePath)
if err != nil {
t.Fatal(err)
}
data = append(data, []byte(`notarius:
enabled: true
config_path: notarius.yml
pipeline_id: campaign.extract
outputs:
encounters:
lane_id: encounters
media_type: application/json
schema_id: encounters
schema_version: "1"
module_key: encounters
`)...)
if err := os.WriteFile(pipelinePath, data, 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(filepath.Dir(pipelinePath), "notarius.yml"), []byte("{}\n"), 0o644); err != nil {
t.Fatal(err)
}
}
func writeOperatorExtractionManifest(t *testing.T, workspaceRoot string) string {
t.Helper()
paths := artifacts.NewLocalStore(workspaceRoot).SessionPathsFor("sample-campaign", "2026-05-03")
bundleRoot := filepath.Join(paths.ArtifactsDir, "notarius", "extract-run-1")
lanePath := filepath.Join(bundleRoot, "lanes", "encounters.json")
indexPath := filepath.Join(bundleRoot, "index.json")
mustWriteTestFile(t, lanePath, `{"secret":"DO_NOT_PRINT"}`)
mustWriteTestFile(t, indexPath, `{"lanes":[]}`)
laneChecksum, err := artifacts.SHA256File(lanePath)
if err != nil {
t.Fatal(err)
}
indexChecksum, err := artifacts.SHA256File(indexPath)
if err != nil {
t.Fatal(err)
}
m := manifest.New("2026-05-03", time.Now().UTC())
m.Campaign = "sample-campaign"
m.Stages["extract"] = &manifest.StageRecord{
Name: "extract", Status: manifest.StatusSucceeded,
Metadata: map[string]any{
"narratio_run_id": "extract-run-1", "bundle_root": bundleRoot,
"receipt": map[string]any{"run_id": "notarius-run-1", "pipeline_id": "campaign.extract"},
},
Outputs: []manifest.ArtifactRecord{
{
Kind: "notarius_lane", SourceID: artifacts.ExtractionArtifactSourceID("encounters"), LocalPath: lanePath,
ProducerRunID: "extract-run-1", Checksum: laneChecksum,
Contract: &artifactmodel.ContractMetadata{MediaType: "application/json", SchemaID: "encounters", SchemaVersion: "1", ModuleKey: "encounters"},
ExternalProvenance: &artifactmodel.ExternalProvenance{System: "notarius", RunID: "notarius-run-1", PipelineID: "campaign.extract", ArtifactID: "encounters"},
},
{Kind: "notarius_index", LocalPath: indexPath, ProducerRunID: "extract-run-1", Checksum: indexChecksum},
},
}
if err := (&manifest.LocalStore{}).Save(context.Background(), paths.ManifestPath, m); err != nil {
t.Fatal(err)
}
return lanePath
}
func replaceInFileOrFatal(t *testing.T, path, old, new string) {
t.Helper()
data, err := os.ReadFile(path)
@@ -1067,7 +1209,7 @@ func writeValidPublishRunConfigFiles(t *testing.T, workspaceRoot string) (string
m := manifest.New("2026-05-03", nowUTC())
m.Campaign = "sample-campaign"
m.RunID = "20260521T160000Z-test"
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze"} {
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze"} {
m.MarkStageSucceeded(name, nowUTC(), nil)
}
path := artifacts.SessionManifestPathForCampaign(cfg.Pipeline.Workspace.Root, cfg.Session.Campaign, cfg.Session.SessionID)

View File

@@ -57,6 +57,8 @@ func inspectStableInputs(cfg *config.Config) []stableInputCheck {
{name: "speakers", in: cfg.StableInputs.SpeakersFile},
{name: "autocorrect", in: cfg.StableInputs.AutocorrectFile},
{name: "glossary", in: cfg.StableInputs.GlossaryFile},
{name: "players", in: cfg.StableInputs.PlayersFile},
{name: "party", in: cfg.StableInputs.PartyFile},
}
out := make([]stableInputCheck, 0, len(items))
for _, item := range items {

View File

@@ -67,7 +67,7 @@ func LocksAdd(ctx context.Context, args []string, out io.Writer) error {
if err != nil {
return fmt.Errorf("locks add: %w", err)
}
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, "locks add"); err != nil {
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, cfg.Pipeline.Notarius, "locks add"); err != nil {
return fmt.Errorf("locks add: %w", err)
}
if _, ok := lockSourceSet(locks.Static)[source]; ok {
@@ -79,7 +79,7 @@ func LocksAdd(ctx context.Context, args []string, out io.Writer) error {
}
remoteSet[source] = config.PublishLockRule{Source: source, Reason: strings.TrimSpace(reason)}
remoteLocks := lockMapValues(remoteSet)
if _, err := config.ValidatePublishLockRules(remoteLocks, cfg.Pipeline.Scriptorium, "locks"); err != nil {
if _, err := config.ValidatePublishLockRules(remoteLocks, cfg.Pipeline.Scriptorium, cfg.Pipeline.Notarius, "locks"); err != nil {
return fmt.Errorf("locks add: %w", err)
}
if err := uploadRemoteLockStore(ctx, store, locks.Key, &config.PublishLockStore{Locks: remoteLocks}); err != nil {
@@ -106,7 +106,7 @@ func LocksRemove(ctx context.Context, args []string, out io.Writer) error {
if err != nil {
return fmt.Errorf("locks remove: %w", err)
}
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, "locks remove"); err != nil {
if _, err := config.ValidatePublishLockRules([]config.PublishLockRule{{Source: source}}, cfg.Pipeline.Scriptorium, cfg.Pipeline.Notarius, "locks remove"); err != nil {
return fmt.Errorf("locks remove: %w", err)
}
remoteSet := lockSourceSet(locks.Remote)

View File

@@ -11,6 +11,7 @@ import (
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
// Status reports effective local/remote session state.
@@ -40,11 +41,13 @@ func Status(ctx context.Context, args []string, out io.Writer) error {
writeStatusStableInputs(out, inspectStableInputs(cfg))
writeStatusLocalAudio(out, inspectLocalAudioPresence(cfg))
var localManifest *manifest.Manifest
if m, err := loadLocalManifest(ctx, paths.ManifestPath); err != nil {
fmt.Fprintf(out, "Local manifest: error: %v\n", err)
} else if m == nil {
fmt.Fprintln(out, "Local manifest: missing")
} else {
localManifest = m
fmt.Fprintf(out, "Local manifest: %s\n", paths.ManifestPath)
writeStageStatuses(out, m)
}
@@ -72,7 +75,7 @@ func Status(ctx context.Context, args []string, out io.Writer) error {
lockChecks := inspectEffectiveLocks(ctx, cfg, store)
locks := lockChecks.Locks
lockErr := lockChecks.Err
if catalog, catalogErr := buildHelperArtifactCatalog(cfg); catalogErr != nil {
if catalog, catalogErr := buildHelperArtifactCatalog(cfg, localManifest); catalogErr != nil {
fmt.Fprintf(out, "Remote outputs: error: %v\n", catalogErr)
} else if storeErr == nil {
catalogLocks := locks

View File

@@ -22,7 +22,7 @@ func Plan(ctx context.Context, args []string, out io.Writer) error {
var flags commonConfigFlags
var force bool
addCommonConfigFlags(fs, &flags)
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
fs.BoolVar(&force, "force", false, "show all stages as scheduled to rerun")
if err := parseSessionAwareFlags("plan", fs, args, &flags.sessionID); err != nil {
return err

View File

@@ -27,12 +27,12 @@ func TestPlanCreatesAndReusesWorkdir(t *testing.T) {
if !strings.Contains(got, "narratio session plan: workdir prepared at") {
t.Fatalf("first output = %q, want workdir prepared", got)
}
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze", "publish", "notify"} {
if !strings.Contains(got, name+": run") {
t.Fatalf("first output = %q, missing stage %q", got, name)
}
}
if !strings.Contains(got, "totals: run=10 skip=0") {
if !strings.Contains(got, "totals: run=11 skip=0") {
t.Fatalf("first output = %q, want totals", got)
}
@@ -84,8 +84,8 @@ func TestPlanShowsRunAndSkipFromManifest(t *testing.T) {
if !strings.Contains(got, "trim: run") {
t.Fatalf("output = %q, want trim run", got)
}
if !strings.Contains(got, "totals: run=8 skip=2") {
t.Fatalf("output = %q, want totals run=8 skip=2", got)
if !strings.Contains(got, "totals: run=9 skip=2") {
t.Fatalf("output = %q, want totals run=9 skip=2", got)
}
}
@@ -118,6 +118,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
t.Fatalf("write pipeline.yml: %v", err)

View File

@@ -4,7 +4,7 @@ import "testing"
func TestBuildFullPlanOrder(t *testing.T) {
got := BuildFullPlan()
want := []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"}
want := []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze", "publish", "notify"}
if len(got) != len(want) {
t.Fatalf("len(plan) = %d, want %d", len(got), len(want))
}

View File

@@ -355,7 +355,7 @@ func publishStageCleanupFixture(t *testing.T) (*config.Config, cleanupSeed, stri
if err != nil {
t.Fatalf("Load() error = %v", err)
}
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze"} {
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze"} {
seedManifest.MarkStageSucceeded(name, time.Now().UTC(), nil)
}
seedManifest.S3SessionPrefix = artifacts.S3SessionPrefix("dnd", cfg.Session.Campaign, cfg.Session.SessionID)

View File

@@ -55,7 +55,7 @@ func loadRemoteLockStore(ctx context.Context, cfg *config.Config, store storage.
if err != nil {
return nil, key, fmt.Errorf("read remote locks %q: %w", key, err)
}
lockStore, err := config.LoadPublishLockStoreBytes("s3://"+s3BucketName(cfg.Pipeline)+"/"+key, data, cfg.Pipeline.Scriptorium)
lockStore, err := config.LoadPublishLockStoreBytes("s3://"+s3BucketName(cfg.Pipeline)+"/"+key, data, cfg.Pipeline.Scriptorium, cfg.Pipeline.Notarius)
if err != nil {
return nil, key, err
}

View File

@@ -9,8 +9,10 @@ import (
"path/filepath"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
@@ -62,6 +64,55 @@ func TestExecuteRestoreNonDryRunRestoresDurableFiles(t *testing.T) {
}
}
func TestExecuteRestoreRoundTripsPublishedExtractionAndManifestMetadata(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
fake := &storage.FakeBackend{}
cfg, sessionPrefix, manifestKey, runIDKey := seedRestoreCommittedState(t, fake, pipelinePath, campaignPath, sessionPath)
seedRestoreObject(fake, sessionPrefix+"artifacts/encounters.json", []byte(`{"encounters":[]}`))
seedRestoreObject(fake, runIDKey, []byte("20260519T010203Z-a1b2c3d4\n"))
remoteManifest := manifest.New(cfg.Session.SessionID, time.Now().UTC())
remoteManifest.Campaign = cfg.Session.Campaign
remoteManifest.RunID = "20260519T010203Z-a1b2c3d4"
remoteManifest.Stages["extract"] = &manifest.StageRecord{
Name: "extract", Status: manifest.StatusSucceeded,
Outputs: []manifest.ArtifactRecord{{
Kind: "notarius_lane", SourceID: artifacts.ExtractionArtifactSourceID("encounters"),
LocalPath: "/prior/workspace/artifacts/notarius/extract-run-1/lanes/encounters.json",
Contract: &artifactmodel.ContractMetadata{
MediaType: "application/json", SchemaID: "encounters", SchemaVersion: "1",
},
ExternalProvenance: &artifactmodel.ExternalProvenance{
System: "notarius", RunID: "notarius-run-1", PipelineID: "campaign.extract", ArtifactID: "encounters",
},
}},
}
manifestBody, err := json.Marshal(remoteManifest)
if err != nil {
t.Fatal(err)
}
seedRestoreObject(fake, manifestKey, manifestBody)
restoreWithStoreAndRealPhases(t, fake)
var stdout bytes.Buffer
var stderr bytes.Buffer
code := Execute([]string{"session", "restore", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &stdout, &stderr)
if code != 0 {
t.Fatalf("exit code = %d, want 0; stderr=%q", code, stderr.String())
}
sessionRoot := artifacts.SessionWorkDirForCampaign(workspaceRoot, cfg.Session.Campaign, cfg.Session.SessionID)
mustReadEquals(t, filepath.Join(sessionRoot, "artifacts", "encounters.json"), `{"encounters":[]}`)
restored, err := (&manifest.LocalStore{}).Load(context.Background(), filepath.Join(sessionRoot, "manifest.json"))
if err != nil {
t.Fatalf("load restored manifest: %v", err)
}
lane := restored.Stages["extract"].Outputs[0]
if lane.Contract == nil || lane.Contract.SchemaID != "encounters" || lane.ExternalProvenance == nil || lane.ExternalProvenance.RunID != "notarius-run-1" {
t.Fatalf("restored extraction metadata = %#v", lane)
}
}
func TestExecuteRestoreIncludeAudioRestoresAudio(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)

View File

@@ -397,6 +397,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
t.Fatalf("write pipeline config: %v", err)
@@ -407,6 +409,8 @@ inputs:
mustWriteTestFile(t, filepath.Join(dir, "speakers.yml"), "alice: alice.flac\n")
mustWriteTestFile(t, filepath.Join(dir, "autocorrect.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(dir, "glossary.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(dir, "players.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(dir, "party.yml"), "[]\n")
mustWriteTestFile(t, filepath.Join(dir, "audio", "alice.flac"), "audio-bytes")
return pipelinePath, campaignPath, sessionPath

View File

@@ -18,7 +18,7 @@ func Run(ctx context.Context, args []string, out io.Writer) error {
var force bool
var selectedArtifacts artifactSelectionFlag
addCommonConfigFlags(fs, &flags)
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
fs.BoolVar(&force, "force", false, "rerun stages even when already succeeded")
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute and publish (comma-separated or repeatable)")
if err := parseSessionAwareFlags("run", fs, args, &flags.sessionID); err != nil {

View File

@@ -3,6 +3,7 @@ package app
import (
"context"
"fmt"
"strings"
"time"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
@@ -23,22 +24,40 @@ type stageDecision struct {
Action stageAction
}
const (
staleReasonForcedReplacement = "upstream stage was force-run"
staleReasonChangedResult = "upstream stage result changed"
staleReasonFailure = "upstream stage failed"
staleReasonSelfSkip = "upstream stage self-skipped"
staleReasonNotResumable = "upstream stage result was not resumable"
)
type priorStageOutcome struct {
exists bool
status manifest.StageStatus
skipReason string
outputs int
}
func decideStageActions(stages []stage.Stage, m *manifest.Manifest, force bool) []stageDecision {
out := make([]stageDecision, 0, len(stages))
for _, s := range stages {
action := stageActionRun
// TODO: incorporate stale detection once checksum/input change tracking is implemented.
if !force && stageSucceeded(m, s.Name()) {
action = stageActionSkip
}
out = append(out, stageDecision{
Stage: s,
Action: action,
Action: decideStageAction(s, m, force),
})
}
return out
}
func decideStageAction(s stage.Stage, m *manifest.Manifest, force bool) stageAction {
// TODO: incorporate stale detection once checksum/input change tracking is implemented.
if !force && stageSucceeded(m, s.Name()) {
return stageActionSkip
}
return stageActionRun
}
func stageSucceeded(m *manifest.Manifest, name string) bool {
if m == nil || m.Stages == nil {
return false
@@ -47,6 +66,29 @@ func stageSucceeded(m *manifest.Manifest, name string) bool {
return sr != nil && sr.Status == manifest.StatusSucceeded
}
func capturePriorStageOutcome(m *manifest.Manifest, name string) priorStageOutcome {
if m == nil || m.Stages == nil || m.Stages[name] == nil {
return priorStageOutcome{}
}
record := m.Stages[name]
outcome := priorStageOutcome{
exists: true,
status: record.Status,
outputs: len(record.Outputs),
}
if record.Error != nil && record.Error.Code == "skipped" {
outcome.skipReason = record.Error.Message
}
return outcome
}
func (o priorStageOutcome) isSameSelfSkip(reason string) bool {
return o.exists &&
o.status == manifest.StatusSkipped &&
o.outputs == 0 &&
o.skipReason == strings.TrimSpace(reason)
}
func loadManifestIfPresent(ctx context.Context, cfg *config.Config) (*manifest.Manifest, error) {
path := artifacts.SessionManifestPathForCampaign(
cfg.Pipeline.Workspace.Root,
@@ -91,7 +133,7 @@ func downstreamStageNames(stageName string) []string {
return nil
}
func invalidateDownstreamSucceededStages(m *manifest.Manifest, upstreamStage string, at time.Time) []string {
func invalidateDownstreamSucceededStagesWithReason(m *manifest.Manifest, upstreamStage string, at time.Time, reason string) []string {
if m == nil || m.Stages == nil {
return nil
}
@@ -102,7 +144,7 @@ func invalidateDownstreamSucceededStages(m *manifest.Manifest, upstreamStage str
if sr == nil || sr.Status != manifest.StatusSucceeded {
continue
}
m.MarkStageStale(downstream, at, "upstream stage rerun with force")
m.MarkStageStale(downstream, at, reason)
invalidated = append(invalidated, downstream)
}
return invalidated

View File

@@ -32,7 +32,7 @@ func TestDecideStageActions(t *testing.T) {
func TestDownstreamStageNames(t *testing.T) {
got := downstreamStageNames("polish")
want := []string{"normalize", "trim", "render", "analyze", "publish", "notify"}
want := []string{"normalize", "trim", "extract", "render", "analyze", "publish", "notify"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("downstreamStageNames(polish) = %#v, want %#v", got, want)
}
@@ -43,7 +43,7 @@ func TestDownstreamStageNames(t *testing.T) {
}
}
func TestInvalidateDownstreamSucceededStages(t *testing.T) {
func TestInvalidateDownstreamSucceededStagesWithReason(t *testing.T) {
now := time.Now().UTC()
m := manifest.New("2026-05-03", now)
m.MarkStageSucceeded("prepare", now, nil)
@@ -52,15 +52,16 @@ func TestInvalidateDownstreamSucceededStages(t *testing.T) {
m.MarkStageSucceeded("polish", now, nil)
m.MarkStageSucceeded("normalize", now, nil)
m.MarkStageSucceeded("trim", now, nil)
m.MarkStageSucceeded("extract", now, nil)
m.MarkStageSucceeded("render", now, nil)
m.MarkStageFailed("analyze", now, "analysis failed")
m.MarkStageSucceeded("publish", now, nil)
m.MarkStageSucceeded("notify", now, nil)
got := invalidateDownstreamSucceededStages(m, "polish", now.Add(1*time.Second))
want := []string{"normalize", "trim", "render", "publish", "notify"}
got := invalidateDownstreamSucceededStagesWithReason(m, "polish", now.Add(1*time.Second), staleReasonChangedResult)
want := []string{"normalize", "trim", "extract", "render", "publish", "notify"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("invalidateDownstreamSucceededStages() = %#v, want %#v", got, want)
t.Fatalf("invalidateDownstreamSucceededStagesWithReason() = %#v, want %#v", got, want)
}
for _, stageName := range want {
@@ -75,3 +76,30 @@ func TestInvalidateDownstreamSucceededStages(t *testing.T) {
t.Fatalf("prepare status = %q, want succeeded", m.Stages["prepare"].Status)
}
}
func TestExtractionPositionControlsForceInvalidation(t *testing.T) {
now := time.Now().UTC()
tests := []struct {
upstream string
want []string
}{
{upstream: "trim", want: []string{"extract", "render", "analyze", "publish", "notify"}},
{upstream: "extract", want: []string{"render", "analyze", "publish", "notify"}},
{upstream: "render", want: []string{"analyze", "publish", "notify"}},
}
for _, test := range tests {
t.Run(test.upstream, func(t *testing.T) {
m := manifest.New("2026-05-03", now)
for _, name := range canonicalStageNames() {
m.MarkStageSucceeded(name, now, nil)
}
got := invalidateDownstreamSucceededStagesWithReason(m, test.upstream, now.Add(time.Second), staleReasonForcedReplacement)
if !reflect.DeepEqual(got, test.want) {
t.Fatalf("invalidated = %#v, want %#v", got, test.want)
}
if test.upstream == "render" && m.Stages["extract"].Status != manifest.StatusSucceeded {
t.Fatalf("forcing render changed extract: %#v", m.Stages["extract"])
}
})
}
}

View File

@@ -27,7 +27,7 @@ func RunStage(ctx context.Context, args []string, out io.Writer) error {
var force bool
var selectedArtifacts artifactSelectionFlag
addCommonConfigFlags(fs, &flags)
fs.BoolVar(&force, "force", false, "force stage execution (reserved for future behavior)")
fs.BoolVar(&force, "force", false, "rerun the stage even when already succeeded")
fs.Var(&selectedArtifacts, "artifacts", "configured artifact names to execute or publish (comma-separated or repeatable)")
if err := fs.Parse(args); err != nil {

View File

@@ -36,8 +36,8 @@ func TestRunContinuesAfterCompletedStages(t *testing.T) {
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if !strings.Contains(out.String(), "executed=8 skipped=2") {
t.Fatalf("output = %q, want executed=8 skipped=2", out.String())
if !strings.Contains(out.String(), "executed=9 skipped=3") {
t.Fatalf("output = %q, want executed=9 skipped=3", out.String())
}
loaded, err := store.Load(context.Background(), manifestPath)
@@ -59,6 +59,7 @@ func TestRunNoRemainingStagesRecordsSkippedStages(t *testing.T) {
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
m.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
}
m.MarkStageSkipped("extract", time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), "notarius_disabled")
if err := store.Save(context.Background(), manifestPath, m); err != nil {
t.Fatalf("save manifest: %v", err)
}
@@ -68,8 +69,15 @@ func TestRunNoRemainingStagesRecordsSkippedStages(t *testing.T) {
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if !strings.Contains(out.String(), "executed=0 skipped=10") {
t.Fatalf("output = %q, want executed=0 skipped=10", out.String())
if !strings.Contains(out.String(), "executed=1 skipped=11") {
t.Fatalf("output = %q, want disabled extraction to self-skip", out.String())
}
loaded, err := store.Load(context.Background(), manifestPath)
if err != nil {
t.Fatalf("load migrated manifest: %v", err)
}
if loaded.Stages["extract"] == nil || loaded.Stages["extract"].Status != manifest.StatusSkipped {
t.Fatalf("extract record = %#v, want stable disabled skip", loaded.Stages["extract"])
}
}
@@ -85,7 +93,7 @@ func TestRunForceRerunsSucceeded(t *testing.T) {
store := &manifest.LocalStore{}
m := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze", "publish", "notify"} {
m.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
}
if err := store.Save(context.Background(), manifestPath, m); err != nil {
@@ -97,7 +105,7 @@ func TestRunForceRerunsSucceeded(t *testing.T) {
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if !strings.Contains(out.String(), "executed=10 skipped=0") {
if !strings.Contains(out.String(), "executed=11 skipped=1") {
t.Fatalf("output = %q, want forced full rerun", out.String())
}
}
@@ -132,6 +140,27 @@ func TestRunStageExecutesOnlySelectedStage(t *testing.T) {
}
}
func TestRunStageExtractIsAcceptedAndSelfSkipsWhenDisabled(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
var out bytes.Buffer
err := RunStage(context.Background(), []string{"extract", "2026-05-03", "--config", pipelinePath, "--campaign-file", campaignPath, "--session", sessionPath}, &out)
if err != nil {
t.Fatalf("RunStage(extract) error = %v", err)
}
if !strings.Contains(out.String(), "stage=extract executed=1 skipped=1 force=false") {
t.Fatalf("output = %q, want disabled extraction self-skip", out.String())
}
loaded, err := (&manifest.LocalStore{}).Load(context.Background(), filepath.Join(workspaceRoot, "work", "sample-campaign", "2026-05-03", "manifest.json"))
if err != nil {
t.Fatalf("Load() error = %v", err)
}
if loaded.Stages["extract"] == nil || loaded.Stages["extract"].Status != manifest.StatusSkipped {
t.Fatalf("extract record = %#v, want skipped", loaded.Stages["extract"])
}
}
func TestRunStageSkipAndForce(t *testing.T) {
workspaceRoot := t.TempDir()
pipelinePath, campaignPath, sessionPath := writeValidConfigFiles(t, workspaceRoot)
@@ -176,7 +205,7 @@ func TestRunStageForceMarksDownstreamStaleAndRunContinuesFromStale(t *testing.T)
store := &manifest.LocalStore{}
seed := manifest.New("2026-05-03", time.Date(2026, 5, 3, 10, 0, 0, 0, time.UTC))
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze", "publish", "notify"} {
seed.MarkStageSucceeded(name, time.Date(2026, 5, 3, 10, 1, 0, 0, time.UTC), nil)
}
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
@@ -196,7 +225,7 @@ func TestRunStageForceMarksDownstreamStaleAndRunContinuesFromStale(t *testing.T)
if err != nil {
t.Fatalf("load manifest after force: %v", err)
}
for _, name := range []string{"normalize", "trim", "render", "analyze", "publish", "notify"} {
for _, name := range []string{"normalize", "trim", "extract", "render", "analyze", "publish", "notify"} {
if afterForce.Stages[name] == nil || afterForce.Stages[name].Status != manifest.StatusStale {
t.Fatalf("stage %q = %#v, want stale", name, afterForce.Stages[name])
}
@@ -207,7 +236,7 @@ func TestRunStageForceMarksDownstreamStaleAndRunContinuesFromStale(t *testing.T)
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if !strings.Contains(out.String(), "executed=6 skipped=4") {
if !strings.Contains(out.String(), "executed=7 skipped=5") {
t.Fatalf("output = %q, want run to execute stale downstream stages", out.String())
}
}
@@ -236,8 +265,8 @@ func TestRunStageTrimExecutes(t *testing.T) {
if m.Stages["trim"] == nil || m.Stages["trim"].Status != manifest.StatusSucceeded {
t.Fatalf("trim stage = %#v, want succeeded", m.Stages["trim"])
}
if m.Stages["trim"].Metadata == nil || m.Stages["trim"].Metadata["trim_action"] != "copy_disabled" {
t.Fatalf("trim stage metadata = %#v, want trim_action=copy_disabled", m.Stages["trim"].Metadata)
if m.Stages["trim"].Metadata == nil || m.Stages["trim"].Metadata["trim_action"] != "copy" {
t.Fatalf("trim stage metadata = %#v, want trim_action=copy", m.Stages["trim"].Metadata)
}
}

View File

@@ -8,6 +8,7 @@ import (
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/audita"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/notarius"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/notify"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/scriptorium"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/seriatim"
@@ -80,6 +81,9 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
}
env.Audita = runner
}
if env.Notarius == nil && needsNotariusForRun(env.Config, stages) {
env.Notarius = notarius.NewSubprocessRunner()
}
if env.Scriptorium == nil {
env.Scriptorium = scriptorium.NewSubprocessRunner()
}
@@ -167,6 +171,29 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
for _, d := range decisions {
s := d.Stage
runNames = append(runNames, s.Name())
d.Action = decideStageAction(s, m, opts.Force)
if d.Action == stageActionSkip {
if validator, ok := s.(stage.ResumeValidator); ok {
validation, err := validator.ValidateResume(ctx, stageEnv, m)
if err != nil {
return nil, fmt.Errorf("validate resume for stage %q: %w", s.Name(), err)
}
validation = validation.Normalized()
if !validation.Resumable {
staleAt := nowUTC()
m.MarkStageStale(s.Name(), staleAt, validation.Reason)
invalidateDownstreamSucceededStagesWithReason(
m, s.Name(), staleAt, staleReasonNotResumable,
)
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
return nil, fmt.Errorf("save manifest after resume validation for stage %q: %w", s.Name(), err)
}
env.Logger.Info("stage result is not resumable", "stage", s.Name(), "reason", validation.Reason)
d.Action = stageActionRun
}
}
}
if d.Action == stageActionSkip {
skipped = append(skipped, s.Name())
@@ -180,6 +207,7 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
continue
}
executed = append(executed, s.Name())
priorOutcome := capturePriorStageOutcome(m, s.Name())
now := nowUTC()
runManifest.SetStageAction(s.Name(), manifest.RunStageActionRun, now)
@@ -188,6 +216,9 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
return nil, fmt.Errorf("save run manifest before stage %q: %w", s.Name(), err)
}
m.MarkStageRunning(s.Name(), now)
if opts.Force {
invalidateDownstreamSucceededStagesWithReason(m, s.Name(), now, staleReasonForcedReplacement)
}
env.Logger.Info("starting stage", "stage", s.Name())
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
return nil, fmt.Errorf("save manifest before stage %q: %w", s.Name(), err)
@@ -195,9 +226,13 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
env.Logger.Debug("manifest saved", "stage", s.Name(), "transition", "running", "path", manifestPath)
result, err := s.Run(ctx, stageEnv, m)
if err == nil {
err = validateStageResult(result)
}
if err != nil {
failedAt := nowUTC()
m.MarkStageFailed(s.Name(), failedAt, err.Error())
invalidateDownstreamSucceededStagesWithReason(m, s.Name(), failedAt, staleReasonFailure)
if saveErr := env.ManifestStore.Save(ctx, manifestPath, m); saveErr != nil {
return nil, fmt.Errorf("stage %q failed (%v) and manifest save failed (%v)", s.Name(), err, saveErr)
}
@@ -209,13 +244,34 @@ func executeStages(ctx context.Context, cfg *config.Config, stages []stage.Stage
env.Logger.Info("stage failed", "stage", s.Name(), "error", err)
return nil, fmt.Errorf("stage %q failed: %w", s.Name(), err)
}
if result != nil && result.Disposition == stage.StageDispositionSkipped {
skipped = append(skipped, s.Name())
skippedAt := nowUTC()
m.MarkStageSkipped(s.Name(), skippedAt, result.SkipReason)
applyStageResultToManifest(m, s.Name(), result)
if !priorOutcome.isSameSelfSkip(result.SkipReason) {
invalidateDownstreamSucceededStagesWithReason(m, s.Name(), skippedAt, staleReasonSelfSkip)
}
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
return nil, fmt.Errorf("save manifest after self-skip %q: %w", s.Name(), err)
}
runManifest.MarkStageSkipped(s.Name(), skippedAt, result.SkipReason)
applyStageResultToRunManifest(runManifest, s.Name(), result)
syncRunManifestIdentityFromSession(m, runManifest)
if err := runManifestStore.SaveRun(ctx, runManifestPath, runManifest); err != nil {
return nil, fmt.Errorf("save run manifest after self-skip %q: %w", s.Name(), err)
}
env.Logger.Debug("manifest saved", "stage", s.Name(), "transition", "skipped", "path", manifestPath)
env.Logger.Info("stage skipped", "stage", s.Name(), "reason", result.SkipReason)
continue
}
outputs := mapResultOutputs(s.Name(), result, runID)
succeededAt := nowUTC()
m.MarkStageSucceeded(s.Name(), succeededAt, outputs)
applyStageResultToManifest(m, s.Name(), result)
if opts.Force {
invalidateDownstreamSucceededStages(m, s.Name(), succeededAt)
if !priorOutcome.exists || priorOutcome.status != manifest.StatusSucceeded {
invalidateDownstreamSucceededStagesWithReason(m, s.Name(), succeededAt, staleReasonChangedResult)
}
if err := env.ManifestStore.Save(ctx, manifestPath, m); err != nil {
@@ -407,26 +463,69 @@ func mapResultOutputs(stageName string, result *stage.StageResult, runID string)
localPath = ref.RelativePath
}
kind := ref.Kind
sourceID := ""
if stageName == "analyze" {
sourceID = artifacts.ConfiguredArtifactSourceID(ref.Kind)
kind = "scriptorium_artifact"
} else {
sourceID = sourceIDForOutputKind(kind)
sourceID := strings.TrimSpace(ref.SourceID)
if sourceID == "" {
if stageName == "analyze" {
sourceID = artifacts.ConfiguredArtifactSourceID(ref.Kind)
kind = "scriptorium_artifact"
} else {
sourceID = sourceIDForOutputKind(kind)
}
}
out = append(out, manifest.ArtifactRecord{
Kind: kind,
SourceID: sourceID,
LocalPath: localPath,
ProducerRunID: runID,
RemoteKey: ref.RemoteKey,
Checksum: ref.Checksum,
Kind: kind,
SourceID: sourceID,
LocalPath: localPath,
Contract: cloneContractMetadata(ref.Contract),
ExternalProvenance: cloneExternalProvenance(ref.ExternalProvenance),
ProducerRunID: runID,
RemoteKey: ref.RemoteKey,
Checksum: ref.Checksum,
})
}
return out
}
func validateStageResult(result *stage.StageResult) error {
if result == nil {
return nil
}
switch result.Disposition {
case stage.StageDispositionSucceeded:
if strings.TrimSpace(result.SkipReason) != "" {
return fmt.Errorf("successful result contains a skip reason")
}
return nil
case stage.StageDispositionSkipped:
if strings.TrimSpace(result.SkipReason) == "" {
return fmt.Errorf("skipped result requires a skip reason")
}
if len(result.Outputs) != 0 {
return fmt.Errorf("skipped result contains %d output(s)", len(result.Outputs))
}
return nil
default:
return fmt.Errorf("unsupported stage result disposition %q", result.Disposition)
}
}
func cloneContractMetadata(value *artifactmodel.ContractMetadata) *artifactmodel.ContractMetadata {
if value == nil {
return nil
}
cloned := *value
return &cloned
}
func cloneExternalProvenance(value *artifactmodel.ExternalProvenance) *artifactmodel.ExternalProvenance {
if value == nil {
return nil
}
cloned := *value
return &cloned
}
func sourceIDForOutputKind(kind string) string {
trimmed := strings.TrimSpace(kind)
if trimmed == "" {
@@ -598,6 +697,18 @@ func needsObjectStoreForRun(cfg *config.Config, stages []stage.Stage) bool {
return true
}
func needsNotariusForRun(cfg *config.Config, stages []stage.Stage) bool {
if cfg == nil || cfg.Pipeline == nil || cfg.Pipeline.Notarius == nil || !cfg.Pipeline.Notarius.Enabled {
return false
}
for _, candidate := range stages {
if candidate != nil && candidate.Name() == "extract" {
return true
}
}
return false
}
func needsRemoteLocksForRun(cfg *config.Config, stages []stage.Stage) bool {
if cfg == nil || cfg.Pipeline == nil || cfg.Session == nil {
return false

View File

@@ -2,6 +2,7 @@ package app
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
@@ -16,6 +17,7 @@ import (
"gitea.maximumdirect.net/eric/narratio/internal/adapters/seriatim"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/storage"
"gitea.maximumdirect.net/eric/narratio/internal/adapters/whisperx"
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
"gitea.maximumdirect.net/eric/narratio/internal/artifacts"
"gitea.maximumdirect.net/eric/narratio/internal/config"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
@@ -38,6 +40,25 @@ type countingStage struct {
runs *int
}
type resultStage struct {
name string
result *stage.StageResult
runs *int
order *[]string
}
func (s resultStage) Name() string { return s.name }
func (s resultStage) Declares() stage.IODecl { return stage.IODecl{} }
func (s resultStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
if s.runs != nil {
*s.runs = *s.runs + 1
}
if s.order != nil {
*s.order = append(*s.order, s.name)
}
return s.result, nil
}
func (s countingStage) Name() string { return s.name }
func (s countingStage) Declares() stage.IODecl { return stage.IODecl{} }
func (s countingStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
@@ -50,6 +71,34 @@ type captureSelectedArtifactsStage struct {
captured *[]string
}
type captureNotariusStage struct {
captured *bool
}
type resumeCheckingStage struct {
name string
validation stage.ResumeValidation
validateErr error
runs *int
}
func (s resumeCheckingStage) Name() string { return s.name }
func (s resumeCheckingStage) Declares() stage.IODecl { return stage.IODecl{} }
func (s resumeCheckingStage) Run(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
*s.runs++
return &stage.StageResult{}, nil
}
func (s resumeCheckingStage) ValidateResume(_ context.Context, _ *stage.Env, _ *manifest.Manifest) (stage.ResumeValidation, error) {
return s.validation, s.validateErr
}
func (s captureNotariusStage) Name() string { return "extract" }
func (s captureNotariusStage) Declares() stage.IODecl { return stage.IODecl{} }
func (s captureNotariusStage) Run(_ context.Context, env *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
*s.captured = env.Notarius != nil
return &stage.StageResult{}, nil
}
func (s captureSelectedArtifactsStage) Name() string { return s.name }
func (s captureSelectedArtifactsStage) Declares() stage.IODecl { return stage.IODecl{} }
func (s captureSelectedArtifactsStage) Run(_ context.Context, env *stage.Env, _ *manifest.Manifest) (*stage.StageResult, error) {
@@ -134,6 +183,28 @@ func TestExecuteStagesPropagatesSelectedArtifactsToEnv(t *testing.T) {
}
}
func TestExecuteStagesComposesNotariusOnlyForEnabledExtraction(t *testing.T) {
cfg := testConfig(t)
cfg.Pipeline.Notarius = &config.NotariusConfig{Enabled: true}
captured := false
_, err := executeStages(context.Background(), cfg, []stage.Stage{captureNotariusStage{captured: &captured}}, RunOptions{})
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
if !captured {
t.Fatal("extract stage did not receive the default Notarius runner")
}
if needsNotariusForRun(cfg, []stage.Stage{countingStage{name: "analyze", runs: new(int)}}) {
t.Fatal("Notarius runner requested without extract in the selected plan")
}
cfg.Pipeline.Notarius.Enabled = false
if needsNotariusForRun(cfg, []stage.Stage{captureNotariusStage{captured: new(bool)}}) {
t.Fatal("Notarius runner requested while extraction is disabled")
}
}
func TestExecuteStagesAnalyzeOutputsPersistAsScriptoriumArtifacts(t *testing.T) {
cfg := testConfig(t)
storeForPaths := artifacts.NewLocalStore(cfg.Pipeline.Workspace.Root)
@@ -199,6 +270,61 @@ func TestExecuteStagesAnalyzeOutputsPersistAsScriptoriumArtifacts(t *testing.T)
}
}
func TestMapResultOutputsPrefersExplicitSourceAndCopiesMetadata(t *testing.T) {
contract := &artifactmodel.ContractMetadata{
MediaType: "application/json",
SchemaID: "notarius.dnd.npc_registry",
SchemaVersion: "v1",
}
provenance := &artifactmodel.ExternalProvenance{
System: "notarius",
RunID: "external-run",
PipelineID: "dnd-session",
ArtifactID: "npc-registry",
}
result := &stage.StageResult{Outputs: []artifacts.Ref{{
Kind: "structured_data",
SourceID: "narratio.example.npcs",
RelativePath: "artifacts/npcs.json",
Contract: contract,
ExternalProvenance: provenance,
}}}
got := mapResultOutputs("analyze", result, "narratio-run")
if len(got) != 1 {
t.Fatalf("outputs len = %d, want 1", len(got))
}
if got[0].SourceID != "narratio.example.npcs" {
t.Fatalf("source_id = %q, want explicit source", got[0].SourceID)
}
if got[0].Kind != "structured_data" {
t.Fatalf("kind = %q, want explicit output kind preserved", got[0].Kind)
}
if got[0].Contract == nil || *got[0].Contract != *contract {
t.Fatalf("contract = %#v, want %#v", got[0].Contract, contract)
}
if got[0].ExternalProvenance == nil || *got[0].ExternalProvenance != *provenance {
t.Fatalf("external provenance = %#v, want %#v", got[0].ExternalProvenance, provenance)
}
if got[0].Contract == contract || got[0].ExternalProvenance == provenance {
t.Fatal("mapped metadata should not alias the stage result")
}
}
func TestMapResultOutputsRetainsFallbackInference(t *testing.T) {
transcript := mapResultOutputs("trim", &stage.StageResult{Outputs: []artifacts.Ref{{
Kind: artifactmodel.TranscriptOutputKindFinalTrimmed,
}}}, "run-id")
if len(transcript) != 1 || transcript[0].SourceID != artifacts.ArtifactTranscriptFinalTrimmed {
t.Fatalf("transcript fallback = %#v, want final-trimmed source", transcript)
}
analyze := mapResultOutputs("analyze", &stage.StageResult{Outputs: []artifacts.Ref{{Kind: "session_recap"}}}, "run-id")
if len(analyze) != 1 || analyze[0].SourceID != "narratio.artifact.session_recap" || analyze[0].Kind != "scriptorium_artifact" {
t.Fatalf("analyze fallback = %#v, want configured artifact inference", analyze)
}
}
func TestNeedsObjectStoreForRunPrepareWithPreviousRequirements(t *testing.T) {
tests := []struct {
name string
@@ -273,7 +399,7 @@ func TestExecuteStagesPublishSkipsRequiredUnselectedConfiguredOutput(t *testing.
manifestPath := manifestPathFor(cfg)
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
seed.Campaign = cfg.Session.Campaign
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render"} {
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render"} {
seed.MarkStageSucceeded(stageName, time.Now().UTC(), nil)
}
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
@@ -332,8 +458,8 @@ func TestExecuteStagesPlaceholderSuccessUpdatesManifest(t *testing.T) {
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
if len(summary.StageNames) != 10 || len(summary.Executed) != 10 || len(summary.Skipped) != 0 {
t.Fatalf("summary = %#v, want all 10 executed", summary)
if len(summary.StageNames) != 11 || len(summary.Executed) != 11 || len(summary.Skipped) != 1 || summary.Skipped[0] != "extract" {
t.Fatalf("summary = %#v, want full plan with disabled extraction self-skip", summary)
}
store := &manifest.LocalStore{}
@@ -342,11 +468,17 @@ func TestExecuteStagesPlaceholderSuccessUpdatesManifest(t *testing.T) {
t.Fatalf("Load manifest error = %v", err)
}
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze", "publish", "notify"} {
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze", "publish", "notify"} {
sr := m.Stages[name]
if sr == nil {
t.Fatalf("missing stage record %q", name)
}
if name == "extract" {
if sr.Status != manifest.StatusSkipped || sr.Error == nil || sr.Error.Message != "notarius_disabled" {
t.Fatalf("extract stage = %#v, want disabled skip", sr)
}
continue
}
if sr.Status != manifest.StatusSucceeded {
t.Fatalf("stage %q status = %q, want %q", name, sr.Status, manifest.StatusSucceeded)
}
@@ -497,6 +629,91 @@ func TestExecuteStagesSkipSucceededWhenNotForced(t *testing.T) {
}
}
func TestExecuteStagesUsesOptionalResumeValidation(t *testing.T) {
for _, test := range []struct {
name string
validation stage.ResumeValidation
wantRuns int
wantSkipped int
}{
{name: "resumable", validation: stage.Resumable(), wantSkipped: 1},
{name: "rerun", validation: stage.NonResumable("durable output changed"), wantRuns: 1},
} {
t.Run(test.name, func(t *testing.T) {
cfg := testConfig(t)
store := &manifest.LocalStore{}
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
seed.MarkStageSucceeded("checked", time.Now().UTC(), nil)
if err := store.Save(context.Background(), manifestPathFor(cfg), seed); err != nil {
t.Fatalf("Save() error = %v", err)
}
runs := 0
candidate := resumeCheckingStage{name: "checked", validation: test.validation, runs: &runs}
summary, err := executeStages(context.Background(), cfg, []stage.Stage{candidate}, RunOptions{})
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
if runs != test.wantRuns || len(summary.Skipped) != test.wantSkipped {
t.Fatalf("runs = %d summary = %#v", runs, summary)
}
})
}
}
func TestExecuteStagesNonResumableResultRerunsSucceededDownstream(t *testing.T) {
cfg := testConfig(t)
store := &manifest.LocalStore{}
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
seed.MarkStageSucceeded("extract", time.Now().UTC(), nil)
seed.MarkStageSucceeded("render", time.Now().UTC(), nil)
if err := store.Save(context.Background(), manifestPathFor(cfg), seed); err != nil {
t.Fatalf("Save() error = %v", err)
}
extractRuns, renderRuns := 0, 0
stages := []stage.Stage{
resumeCheckingStage{name: "extract", validation: stage.NonResumable("checksum changed"), runs: &extractRuns},
countingStage{name: "render", runs: &renderRuns},
}
summary, err := executeStages(context.Background(), cfg, stages, RunOptions{})
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
if extractRuns != 1 || renderRuns != 1 || len(summary.Executed) != 2 || len(summary.Skipped) != 0 {
t.Fatalf("extract runs=%d render runs=%d summary=%#v", extractRuns, renderRuns, summary)
}
}
func TestExecuteStagesResumeValidationErrorPreservesSucceededRecord(t *testing.T) {
cfg := testConfig(t)
store := &manifest.LocalStore{}
seed := manifest.New(cfg.Session.SessionID, time.Now().UTC())
seed.MarkStageSucceeded("checked", time.Now().UTC(), []manifest.ArtifactRecord{{Kind: "kept", LocalPath: "kept.json"}})
seed.Stages["checked"].Metadata = map[string]any{"kept": true}
if err := store.Save(context.Background(), manifestPathFor(cfg), seed); err != nil {
t.Fatalf("Save() error = %v", err)
}
before, err := json.Marshal(seed.Stages["checked"])
if err != nil {
t.Fatalf("Marshal(before) error = %v", err)
}
runs := 0
candidate := resumeCheckingStage{name: "checked", validateErr: errors.New("inspection unavailable"), runs: &runs}
if _, err := executeStages(context.Background(), cfg, []stage.Stage{candidate}, RunOptions{}); err == nil || !strings.Contains(err.Error(), "inspection unavailable") {
t.Fatalf("executeStages() error = %v", err)
}
loaded, err := store.Load(context.Background(), manifestPathFor(cfg))
if err != nil {
t.Fatalf("Load() error = %v", err)
}
after, err := json.Marshal(loaded.Stages["checked"])
if err != nil {
t.Fatalf("Marshal(after) error = %v", err)
}
if string(after) != string(before) || runs != 0 {
t.Fatalf("succeeded record changed: before=%s after=%s runs=%d", before, after, runs)
}
}
func TestExecuteStagesForceRerunsSucceeded(t *testing.T) {
cfg := testConfig(t)
manifestPath := manifestPathFor(cfg)
@@ -525,13 +742,48 @@ func TestExecuteStagesForceRerunsSucceeded(t *testing.T) {
}
}
func TestExecuteStagesSuccessfulReplacementDoesNotInheritResultDetails(t *testing.T) {
cfg := testConfig(t)
manifestPath := manifestPathFor(cfg)
store := &manifest.LocalStore{}
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
existing.MarkStageSucceeded("transcribe", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), []manifest.ArtifactRecord{{
Kind: "transcript_raw", LocalPath: "transcripts/old.json",
}})
existingStage := existing.Stages["transcribe"]
existingStage.Logs = []string{"logs/old.log"}
existingStage.GeneratedConfigs = []string{"generated/old.yaml"}
existingStage.Metadata = map[string]any{"old_result": true}
if err := store.Save(context.Background(), manifestPath, existing); err != nil {
t.Fatalf("Save manifest error = %v", err)
}
runs := 0
replacement := resultStage{name: "transcribe", result: &stage.StageResult{}, runs: &runs}
if _, err := executeStages(context.Background(), cfg, []stage.Stage{replacement}, RunOptions{Force: true}); err != nil {
t.Fatalf("executeStages() error = %v", err)
}
loaded, err := store.Load(context.Background(), manifestPath)
if err != nil {
t.Fatalf("Load manifest error = %v", err)
}
record := loaded.Stages["transcribe"]
if runs != 1 || record == nil || record.Status != manifest.StatusSucceeded {
t.Fatalf("runs = %d, record = %#v, want one successful replacement", runs, record)
}
if len(record.Outputs) != 0 || len(record.Logs) != 0 || len(record.GeneratedConfigs) != 0 || len(record.Metadata) != 0 {
t.Fatalf("replacement inherited result details: %#v", record)
}
}
func TestExecuteStagesForceSuccessInvalidatesDownstreamSucceededStages(t *testing.T) {
cfg := testConfig(t)
manifestPath := manifestPathFor(cfg)
store := &manifest.LocalStore{}
existing := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "publish", "notify"} {
for _, stageName := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "publish", "notify"} {
existing.MarkStageSucceeded(stageName, time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), nil)
}
existing.MarkStageFailed("analyze", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), "previous analyze failure")
@@ -562,7 +814,7 @@ func TestExecuteStagesForceSuccessInvalidatesDownstreamSucceededStages(t *testin
if loaded.Stages["polish"] == nil || loaded.Stages["polish"].Status != manifest.StatusSucceeded {
t.Fatalf("polish status = %#v, want succeeded", loaded.Stages["polish"])
}
for _, stageName := range []string{"normalize", "trim", "render", "publish", "notify"} {
for _, stageName := range []string{"normalize", "trim", "extract", "render", "publish", "notify"} {
if loaded.Stages[stageName] == nil || loaded.Stages[stageName].Status != manifest.StatusStale {
t.Fatalf("%s status = %#v, want stale", stageName, loaded.Stages[stageName])
}
@@ -738,6 +990,126 @@ func TestExecuteStagesRunManifestRecordsSkippedStage(t *testing.T) {
}
}
func TestExecuteStagesPersistsSelfSkipAndContinues(t *testing.T) {
cfg := testConfig(t)
store := &manifest.LocalStore{}
manifestPath := manifestPathFor(cfg)
seed := manifest.New(cfg.Session.SessionID, time.Date(2026, 5, 3, 1, 0, 0, 0, time.UTC))
seed.MarkStageSucceeded("optional", time.Date(2026, 5, 3, 1, 1, 0, 0, time.UTC), []manifest.ArtifactRecord{{
Kind: "old_output",
SourceID: "narratio.example.old",
LocalPath: "artifacts/old.json",
}})
seed.Stages["optional"].Logs = []string{"old.log"}
seed.Stages["optional"].GeneratedConfigs = []string{"old.yml"}
seed.Stages["optional"].Metadata = map[string]any{"old": true}
if err := os.MkdirAll(filepath.Dir(manifestPath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := store.Save(context.Background(), manifestPath, seed); err != nil {
t.Fatalf("Save() seed manifest error = %v", err)
}
order := []string{}
optionalRuns := 0
stages := []stage.Stage{
resultStage{
name: "optional",
runs: &optionalRuns,
order: &order,
result: &stage.StageResult{
Disposition: stage.StageDispositionSkipped,
SkipReason: "integration_disabled",
Logs: []string{"runs/current/optional.log"},
GeneratedConfigs: []string{"runs/current/optional.yml"},
Metadata: map[string]any{"enabled": false},
},
},
resultStage{name: "later", order: &order, result: &stage.StageResult{}},
}
summary, err := executeStages(context.Background(), cfg, stages, RunOptions{Force: true})
if err != nil {
t.Fatalf("executeStages() error = %v", err)
}
if strings.Join(order, ",") != "optional,later" {
t.Fatalf("execution order = %v, want optional then later", order)
}
if len(summary.Executed) != 2 || len(summary.Skipped) != 1 || summary.Skipped[0] != "optional" {
t.Fatalf("summary = %#v, want optional executed and self-skipped before later", summary)
}
sessionManifest, err := store.Load(context.Background(), summary.ManifestPath)
if err != nil {
t.Fatalf("Load() session manifest error = %v", err)
}
selfSkipped := sessionManifest.Stages["optional"]
if selfSkipped == nil || selfSkipped.Status != manifest.StatusSkipped {
t.Fatalf("optional stage = %#v, want skipped", selfSkipped)
}
if len(selfSkipped.Outputs) != 0 {
t.Fatalf("optional outputs = %#v, want old outputs cleared", selfSkipped.Outputs)
}
if selfSkipped.Error == nil || selfSkipped.Error.Message != "integration_disabled" {
t.Fatalf("optional skip reason = %#v, want integration_disabled", selfSkipped.Error)
}
if len(selfSkipped.Logs) != 1 || selfSkipped.Logs[0] != "runs/current/optional.log" ||
len(selfSkipped.GeneratedConfigs) != 1 || selfSkipped.GeneratedConfigs[0] != "runs/current/optional.yml" ||
selfSkipped.Metadata["enabled"] != false || selfSkipped.Metadata["old"] != nil {
t.Fatalf("optional result details = %#v, want current bounded diagnostics and metadata", selfSkipped)
}
if later := sessionManifest.Stages["later"]; later == nil || later.Status != manifest.StatusSucceeded {
t.Fatalf("later stage = %#v, want succeeded", later)
}
runManifest, err := store.LoadRun(context.Background(), summary.RunManifestPath)
if err != nil {
t.Fatalf("LoadRun() error = %v", err)
}
runStage := runManifest.Stages["optional"]
if runStage == nil || runStage.Action != manifest.RunStageActionRun || runStage.Status != manifest.StatusSkipped {
t.Fatalf("run optional stage = %#v, want run action with skipped status", runStage)
}
if len(runStage.Logs) != 1 || runStage.Metadata["enabled"] != false {
t.Fatalf("run optional stage details = %#v, want result diagnostics and metadata", runStage)
}
_, err = executeStages(context.Background(), cfg, []stage.Stage{stages[0]}, RunOptions{})
if err != nil {
t.Fatalf("second executeStages() error = %v", err)
}
if optionalRuns != 2 {
t.Fatalf("optional runs = %d, want self-skipped stage reconsidered", optionalRuns)
}
}
func TestExecuteStagesRejectsSkippedResultWithOutputs(t *testing.T) {
cfg := testConfig(t)
invalid := resultStage{name: "optional", result: &stage.StageResult{
Disposition: stage.StageDispositionSkipped,
SkipReason: "integration_disabled",
Outputs: []artifacts.Ref{{Kind: "unexpected"}},
}}
summary, err := executeStages(context.Background(), cfg, []stage.Stage{invalid}, RunOptions{})
if err == nil {
t.Fatal("executeStages() error = nil, want invalid skipped result failure")
}
if summary != nil {
t.Fatalf("summary = %#v, want nil", summary)
}
if !strings.Contains(err.Error(), "skipped result contains 1 output") {
t.Fatalf("error = %q, want skipped-output validation", err)
}
loaded, loadErr := (&manifest.LocalStore{}).Load(context.Background(), manifestPathFor(cfg))
if loadErr != nil {
t.Fatalf("Load() session manifest error = %v", loadErr)
}
if got := loaded.Stages["optional"]; got == nil || got.Status != manifest.StatusFailed {
t.Fatalf("optional stage = %#v, want failed", got)
}
}
func TestExecuteStagesRunLocalArtifactsAndCanonicalSync(t *testing.T) {
cfg := testConfig(t)
stages := []stage.Stage{
@@ -986,7 +1358,7 @@ func TestAdapterBackedStageFailureMarksManifestFailed(t *testing.T) {
seed.S3Bucket = "my-dnd-archive"
seed.S3SessionPrefix = "dnd/campaigns/" + cfg.Session.Campaign + "/sessions/" + cfg.Session.SessionID + "/"
seed.S3RunPrefix = seed.S3SessionPrefix + "runs/" + runID + "/"
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "render", "analyze"} {
for _, name := range []string{"prepare", "transcribe", "merge", "polish", "normalize", "trim", "extract", "render", "analyze"} {
seed.MarkStageSucceeded(name, time.Now().UTC(), nil)
}
if err := tc.env.ManifestStore.Save(context.Background(), manifestPathFor(cfg), seed); err != nil {
@@ -1024,11 +1396,13 @@ func testConfig(t *testing.T) *config.Config {
pipelinePath := filepath.Join(cfgDir, "pipeline.yml")
mustWriteFile(t, pipelinePath, "workspace:\n root: "+workspace+"\n")
mustWriteFile(t, campaignPath, "campaign_id: sample-campaign\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n")
mustWriteFile(t, campaignPath, "campaign_id: sample-campaign\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n players_file: ./players.yml\n party_file: ./party.yml\n")
mustWriteFile(t, sessionPath, "session_id: 2026-05-03\ncampaign: sample-campaign\ninputs:\n audio_dir: ./audio\n")
mustWriteFile(t, filepath.Join(cfgDir, "speakers.yml"), "alice: alice.flac\n")
mustWriteFile(t, filepath.Join(cfgDir, "autocorrect.yml"), "[]\n")
mustWriteFile(t, filepath.Join(cfgDir, "glossary.yml"), "[]\n")
mustWriteFile(t, filepath.Join(cfgDir, "players.yml"), "[]\n")
mustWriteFile(t, filepath.Join(cfgDir, "party.yml"), "[]\n")
mustWriteFile(t, filepath.Join(cfgDir, "audio", "alice.flac"), "audio")
return &config.Config{
@@ -1053,6 +1427,16 @@ func testConfig(t *testing.T) *config.Config {
ConfigPath: campaignPath,
Source: "campaign_config",
},
PlayersFile: config.ResolvedInputFile{
Path: "./players.yml",
ConfigPath: campaignPath,
Source: "campaign_config",
},
PartyFile: config.ResolvedInputFile{
Path: "./party.yml",
ConfigPath: campaignPath,
Source: "campaign_config",
},
},
Session: &config.SessionConfig{
SessionID: "2026-05-03",
@@ -1062,6 +1446,8 @@ func testConfig(t *testing.T) *config.Config {
SpeakersFile: "./speakers.yml",
AutocorrectFile: "./autocorrect.yml",
GlossaryFile: "./glossary.yml",
PlayersFile: "./players.yml",
PartyFile: "./party.yml",
},
},
}
@@ -1084,6 +1470,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`
sessionYAML := `session_id: 2026-05-03
campaign: sample-campaign
@@ -1092,6 +1480,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`
mustWriteFile(t, pipelinePath, pipelineYAML)
mustWriteFile(t, campaignPath, campaignYAML)

View File

@@ -22,6 +22,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`
if err := os.WriteFile(sessionPath, []byte(sessionTemplate), 0o644); err != nil {
t.Fatalf("write session template: %v", err)
@@ -75,6 +77,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`
if err := os.WriteFile(sessionPath, []byte(sessionYAML), 0o644); err != nil {
t.Fatalf("write session.yml: %v", err)

View File

@@ -346,7 +346,7 @@ func TestExecuteSessionLocksMutationAcceptsPositionalSessionID(t *testing.T) {
if code != 0 {
t.Fatalf("locks remove exit code = %d, want 0; stderr=%q", code, stderr.String())
}
store, err := config.LoadPublishLockStoreBytes("locks.yml", fake.Objects[key].Data, nil)
store, err := config.LoadPublishLockStoreBytes("locks.yml", fake.Objects[key].Data, nil, nil)
if err != nil {
t.Fatalf("LoadPublishLockStoreBytes() error = %v", err)
}

View File

@@ -0,0 +1,17 @@
package artifactmodel
// ContractMetadata identifies the data contract implemented by an artifact.
type ContractMetadata struct {
MediaType string `json:"media_type"`
SchemaID string `json:"schema_id"`
SchemaVersion string `json:"schema_version"`
ModuleKey string `json:"module_key,omitempty"`
}
// ExternalProvenance identifies an artifact produced by an external system.
type ExternalProvenance struct {
System string `json:"system"`
RunID string `json:"run_id"`
PipelineID string `json:"pipeline_id"`
ArtifactID string `json:"artifact_id"`
}

View File

@@ -0,0 +1,56 @@
package artifactmodel
import (
"encoding/json"
"strings"
"testing"
)
func TestArtifactMetadataJSON(t *testing.T) {
type envelope struct {
Contract *ContractMetadata `json:"contract,omitempty"`
ExternalProvenance *ExternalProvenance `json:"external_provenance,omitempty"`
}
complete, err := json.Marshal(envelope{
Contract: &ContractMetadata{
MediaType: "application/json",
SchemaID: "notarius.dnd.npc_registry",
SchemaVersion: "v1",
ModuleKey: "dnd/npc-registry",
},
ExternalProvenance: &ExternalProvenance{
System: "notarius",
RunID: "run-123",
PipelineID: "dnd-session",
ArtifactID: "npc-registry",
},
})
if err != nil {
t.Fatalf("Marshal() complete metadata error = %v", err)
}
wantComplete := `{"contract":{"media_type":"application/json","schema_id":"notarius.dnd.npc_registry","schema_version":"v1","module_key":"dnd/npc-registry"},"external_provenance":{"system":"notarius","run_id":"run-123","pipeline_id":"dnd-session","artifact_id":"npc-registry"}}`
if string(complete) != wantComplete {
t.Fatalf("complete metadata JSON = %s, want %s", complete, wantComplete)
}
omitted, err := json.Marshal(envelope{})
if err != nil {
t.Fatalf("Marshal() omitted metadata error = %v", err)
}
if string(omitted) != `{}` {
t.Fatalf("omitted metadata JSON = %s, want {}", omitted)
}
withoutModule, err := json.Marshal(envelope{Contract: &ContractMetadata{
MediaType: "application/json",
SchemaID: "notarius.dnd.npc_registry",
SchemaVersion: "v1",
}})
if err != nil {
t.Fatalf("Marshal() contract without module key error = %v", err)
}
if strings.Contains(string(withoutModule), "module_key") {
t.Fatalf("contract JSON unexpectedly contains omitted module_key: %s", withoutModule)
}
}

View File

@@ -13,12 +13,19 @@ import (
const (
SourceBoundsSession = "narratio.bounds.session"
SourceInputPlayers = "narratio.input.players"
SourceInputParty = "narratio.input.party"
SourceInputGlossary = "narratio.input.glossary"
configuredSourcePrefix = "narratio.artifact."
extractionSourcePrefix = "narratio.extraction."
previousConfiguredSrcPrefix = "narratio.previous_session.artifact."
)
var configuredSourceRE = regexp.MustCompile(`^narratio\.artifact\.([a-z][a-z0-9_]*)$`)
var extractionSourceRE = regexp.MustCompile(`^narratio\.extraction\.([a-z][a-z0-9_]*)$`)
var previousSourceRE = regexp.MustCompile(`^narratio\.previous_session\.artifact\.([a-z][a-z0-9_]*)$`)
var configuredKeyRE = regexp.MustCompile(`^[a-z][a-z0-9_]*$`)
var (
ErrUnsupportedScriptoriumInputSource = errors.New("unsupported scriptorium input source")
@@ -30,7 +37,9 @@ type SourceKind string
const (
SourceKindBuiltIn SourceKind = "built_in"
SourceKindConfiguredArtifact SourceKind = "configured_artifact"
SourceKindExtraction SourceKind = "extraction"
SourceKindPreviousArtifact SourceKind = "previous_session_configured_artifact"
SourceKindStableInput SourceKind = "stable_input"
)
// Source describes one normalized artifact source identifier.
@@ -62,11 +71,30 @@ func (e *UnknownConfiguredArtifactError) Error() string {
return fmt.Sprintf("references unknown artifact %q", e.ConfiguredKey)
}
// UnknownExtractionArtifactError reports a source that references an undefined extraction key.
type UnknownExtractionArtifactError struct {
ConfiguredKey string
}
func (e *UnknownExtractionArtifactError) Error() string {
return fmt.Sprintf("references unknown extraction output %q", e.ConfiguredKey)
}
// IsConfiguredKey reports whether a key follows the configured-artifact key grammar.
func IsConfiguredKey(key string) bool {
return configuredKeyRE.MatchString(strings.TrimSpace(key))
}
// ConfiguredSourceID converts a configured artifact key into source id form.
func ConfiguredSourceID(key string) string {
return configuredSourcePrefix + strings.TrimSpace(key)
}
// ExtractionSourceID converts an extraction output key into source id form.
func ExtractionSourceID(key string) string {
return extractionSourcePrefix + strings.TrimSpace(key)
}
// PreviousSessionSourceID converts a configured artifact key into previous-session source id form.
func PreviousSessionSourceID(key string) string {
return previousConfiguredSrcPrefix + strings.TrimSpace(key)
@@ -81,6 +109,15 @@ func ParseConfiguredSource(source string) (string, bool) {
return matches[1], true
}
// ParseExtractionSource extracts configured key from narratio.extraction.<key>.
func ParseExtractionSource(source string) (string, bool) {
matches := extractionSourceRE.FindStringSubmatch(strings.TrimSpace(source))
if len(matches) != 2 {
return "", false
}
return matches[1], true
}
// ParsePreviousSessionSource extracts configured key from narratio.previous_session.artifact.<key>.
func ParsePreviousSessionSource(source string) (string, bool) {
matches := previousSourceRE.FindStringSubmatch(strings.TrimSpace(source))
@@ -105,6 +142,9 @@ func ClassifySource(source string) (Source, error) {
if key, ok := ParseConfiguredSource(trimmed); ok {
return Source{ID: trimmed, Kind: SourceKindConfiguredArtifact, ConfiguredKey: key}, nil
}
if key, ok := ParseExtractionSource(trimmed); ok {
return Source{ID: trimmed, Kind: SourceKindExtraction, ConfiguredKey: key}, nil
}
if key, ok := ParsePreviousSessionSource(trimmed); ok {
return Source{ID: trimmed, Kind: SourceKindPreviousArtifact, ConfiguredKey: key}, nil
}
@@ -118,6 +158,11 @@ func DescribeScriptoriumInputSource(source string) (ScriptoriumInputSourceDescri
if trimmed == "" {
return ScriptoriumInputSourceDescriptor{}, ErrUnsupportedScriptoriumInputSource
}
if IsStableInputSource(trimmed) {
return ScriptoriumInputSourceDescriptor{
Source: Source{ID: trimmed, Kind: SourceKindStableInput},
}, nil
}
if strings.HasPrefix(trimmed, "narratio.previous_session.artifact") {
descriptor, err := DescribePreviousSessionSource(trimmed)
if err != nil {
@@ -140,6 +185,17 @@ func DescribeScriptoriumInputSource(source string) (ScriptoriumInputSourceDescri
return ScriptoriumInputSourceDescriptor{Source: classified}, nil
}
// IsStableInputSource reports whether source is a prepared stable input source
// available only to Scriptorium input resolution.
func IsStableInputSource(source string) bool {
switch strings.TrimSpace(source) {
case SourceInputPlayers, SourceInputParty, SourceInputGlossary:
return true
default:
return false
}
}
// DescribePreviousSessionSource validates a canonical previous-session source id
// and returns both previous and configured-source vocabulary descriptors.
func DescribePreviousSessionSource(source string) (PreviousSessionSourceDescriptor, error) {
@@ -165,24 +221,48 @@ func PreviousSessionSourceDescriptorForConfiguredKey(configuredKey string) (Prev
func ValidateInputConfiguredReference(
descriptor ScriptoriumInputSourceDescriptor,
configured map[string]struct{},
) error {
return ValidateInputReference(descriptor, configured, nil)
}
// ValidateInputReference checks that configured and extraction sources are declared
// by the effective pipeline configuration.
func ValidateInputReference(
descriptor ScriptoriumInputSourceDescriptor,
configured map[string]struct{},
extractions map[string]struct{},
) error {
switch descriptor.Source.Kind {
case SourceKindConfiguredArtifact, SourceKindPreviousArtifact:
if _, ok := configured[descriptor.Source.ConfiguredKey]; !ok {
return &UnknownConfiguredArtifactError{ConfiguredKey: descriptor.Source.ConfiguredKey}
}
case SourceKindExtraction:
if _, ok := extractions[descriptor.Source.ConfiguredKey]; !ok {
return &UnknownExtractionArtifactError{ConfiguredKey: descriptor.Source.ConfiguredKey}
}
}
return nil
}
// ValidatePublishSource validates that a source is publish-compatible and references a known configured artifact.
func ValidatePublishSource(source string, configured map[string]string) (Source, error) {
return ValidatePublishSourceWithExtractions(source, configured, nil)
}
// ValidatePublishSourceWithExtractions validates publish sources against the
// configured Scriptorium artifacts and extraction outputs.
func ValidatePublishSourceWithExtractions(
source string,
configured map[string]string,
extractions map[string]struct{},
) (Source, error) {
classified, err := ClassifySource(source)
if err != nil {
return Source{}, fmt.Errorf("must be a built-in source id or narratio.artifact.<name>")
return Source{}, fmt.Errorf("must be a built-in source id, narratio.artifact.<name>, or configured narratio.extraction.<name>")
}
if classified.Kind == SourceKindPreviousArtifact {
return Source{}, fmt.Errorf("must be a built-in source id or narratio.artifact.<name>")
return Source{}, fmt.Errorf("must be a built-in source id, narratio.artifact.<name>, or configured narratio.extraction.<name>")
}
if classified.Kind == SourceKindConfiguredArtifact {
if configured == nil {
@@ -192,6 +272,11 @@ func ValidatePublishSource(source string, configured map[string]string) (Source,
return Source{}, fmt.Errorf("configured artifact %q is not defined in pipeline.scriptorium.artifacts", classified.ConfiguredKey)
}
}
if classified.Kind == SourceKindExtraction {
if _, ok := extractions[classified.ConfiguredKey]; !ok {
return Source{}, fmt.Errorf("extraction output %q is not defined in pipeline.notarius.outputs", classified.ConfiguredKey)
}
}
return classified, nil
}
@@ -226,7 +311,17 @@ func DeriveDefaultPublishedDestination(source Source, configured map[string]stri
// ResolvePublishedDestination validates and normalizes an explicit destination,
// or derives one when omitted.
func ResolvePublishedDestination(sourceID, explicitDest string, configured map[string]string) (string, error) {
source, err := ValidatePublishSource(sourceID, configured)
return ResolvePublishedDestinationWithExtractions(sourceID, explicitDest, configured, nil)
}
// ResolvePublishedDestinationWithExtractions validates and normalizes a destination
// while accepting extraction sources declared by the effective Notarius configuration.
func ResolvePublishedDestinationWithExtractions(
sourceID, explicitDest string,
configured map[string]string,
extractions map[string]struct{},
) (string, error) {
source, err := ValidatePublishSourceWithExtractions(sourceID, configured, extractions)
if err != nil {
return "", err
}

View File

@@ -17,6 +17,7 @@ func TestClassifySource(t *testing.T) {
{name: "built in transcript", source: "narratio.transcript.final_trimmed", wantKind: SourceKindBuiltIn},
{name: "built in bounds", source: "narratio.bounds.session", wantKind: SourceKindBuiltIn},
{name: "configured artifact", source: "narratio.artifact.session_recap", wantKind: SourceKindConfiguredArtifact, wantKey: "session_recap"},
{name: "extraction", source: "narratio.extraction.npc_registry", wantKind: SourceKindExtraction, wantKey: "npc_registry"},
{name: "previous session configured", source: "narratio.previous_session.artifact.session_recap", wantKind: SourceKindPreviousArtifact, wantKey: "session_recap"},
{name: "unsupported", source: "narratio.unknown", wantErrLike: "unsupported artifact source"},
}
@@ -56,6 +57,60 @@ func TestValidatePublishSource(t *testing.T) {
}
}
func TestExtractionSourcePolicy(t *testing.T) {
if got := ExtractionSourceID(" npc_registry "); got != "narratio.extraction.npc_registry" {
t.Fatalf("ExtractionSourceID() = %q, want narratio.extraction.npc_registry", got)
}
if key, ok := ParseExtractionSource(" narratio.extraction.npc_registry "); !ok || key != "npc_registry" {
t.Fatalf("ParseExtractionSource() = %q, %t; want npc_registry, true", key, ok)
}
for _, source := range []string{
"narratio.extraction.",
"narratio.extraction.NPC",
"narratio.extraction.npc-registry",
"narratio.extraction.npc_registry.extra",
} {
if _, ok := ParseExtractionSource(source); ok {
t.Fatalf("ParseExtractionSource(%q) unexpectedly matched", source)
}
}
descriptor, err := DescribeScriptoriumInputSource("narratio.extraction.npc_registry")
if err != nil {
t.Fatalf("DescribeScriptoriumInputSource(extraction) error = %v", err)
}
if descriptor.Source.Kind != SourceKindExtraction || descriptor.Source.ConfiguredKey != "npc_registry" {
t.Fatalf("extraction descriptor = %#v", descriptor)
}
declared := map[string]struct{}{"npc_registry": {}}
if err := ValidateInputReference(descriptor, nil, declared); err != nil {
t.Fatalf("ValidateInputReference(declared extraction) error = %v", err)
}
if err := ValidateInputReference(descriptor, nil, nil); err == nil {
t.Fatal("ValidateInputReference(unknown extraction) error = nil, want error")
}
if _, err := ValidatePublishSourceWithExtractions("narratio.extraction.npc_registry", nil, declared); err != nil {
t.Fatalf("ValidatePublishSourceWithExtractions(declared) error = %v", err)
}
if _, err := ValidatePublishSourceWithExtractions("narratio.extraction.unknown", nil, declared); err == nil {
t.Fatal("ValidatePublishSourceWithExtractions(unknown) error = nil, want error")
}
identities := map[string]struct{}{}
for _, sourceID := range []string{
ExtractionSourceID("npc_registry"),
ConfiguredSourceID("npc_registry"),
PreviousSessionSourceID("npc_registry"),
SourceBoundsSession,
} {
if _, exists := identities[sourceID]; exists {
t.Fatalf("source identity collision at %q", sourceID)
}
identities[sourceID] = struct{}{}
}
}
func TestResolvePublishedDestination(t *testing.T) {
configured := map[string]string{"session_recap": "artifacts/session_recap.md"}
@@ -112,6 +167,9 @@ func TestDescribeScriptoriumInputSource(t *testing.T) {
}{
{name: "built in", source: "narratio.transcript.final_trimmed", wantKind: SourceKindBuiltIn},
{name: "built in markdown", source: "narratio.transcript.final_markdown", wantKind: SourceKindBuiltIn},
{name: "prepared players input", source: "narratio.input.players", wantKind: SourceKindStableInput},
{name: "prepared party input", source: "narratio.input.party", wantKind: SourceKindStableInput},
{name: "prepared glossary input", source: "narratio.input.glossary", wantKind: SourceKindStableInput},
{name: "configured", source: "narratio.artifact.session_recap", wantKind: SourceKindConfiguredArtifact, wantKey: "session_recap"},
{name: "previous", source: "narratio.previous_session.artifact.session_recap", wantKind: SourceKindPreviousArtifact, wantKey: "session_recap", wantPrev: true},
{name: "invalid previous", source: "narratio.previous_session.artifact.", wantErr: ErrInvalidPreviousSessionSource},

View File

@@ -144,6 +144,22 @@ func ConfiguredArtifactName(source string) (string, bool) {
return artifactpolicy.ParseConfiguredSource(source)
}
// ExtractionArtifactSourceID returns narratio.extraction.<name> for a configured key.
func ExtractionArtifactSourceID(key string) string {
return artifactpolicy.ExtractionSourceID(key)
}
// IsExtractionArtifactSource reports whether source is narratio.extraction.<name>.
func IsExtractionArtifactSource(source string) bool {
_, ok := artifactpolicy.ParseExtractionSource(source)
return ok
}
// ExtractionArtifactName extracts <name> from narratio.extraction.<name>.
func ExtractionArtifactName(source string) (string, bool) {
return artifactpolicy.ParseExtractionSource(source)
}
// IsPreviousSessionArtifactSource returns true when source is narratio.previous_session.artifact.<name>.
func IsPreviousSessionArtifactSource(source string) bool {
_, ok := artifactpolicy.ParsePreviousSessionSource(source)
@@ -204,17 +220,19 @@ func ResolveSessionArtifact(paths SessionPaths, m *manifest.Manifest, source str
}
// ResolveSessionArtifactWithCatalog resolves built-in sources using existing rules and resolves
// configured narratio.artifact.<name> sources through runtime catalog availability.
// configured artifact and extraction sources through runtime catalog availability.
func ResolveSessionArtifactWithCatalog(paths SessionPaths, m *manifest.Manifest, source string, catalog *ArtifactCatalog) (ResolvedSessionArtifact, error) {
normalized := strings.TrimSpace(source)
if IsPreviousSessionArtifactSource(normalized) {
return ResolvePreviousSessionArtifactWithCatalog(paths, m, normalized, catalog)
}
if !IsConfiguredArtifactSource(normalized) {
configuredSource := IsConfiguredArtifactSource(normalized)
extractionSource := IsExtractionArtifactSource(normalized)
if !configuredSource && !extractionSource {
return ResolveSessionArtifact(paths, m, normalized)
}
if catalog == nil {
return ResolvedSessionArtifact{}, fmt.Errorf("configured artifact source %q requires runtime artifact catalog", source)
return ResolvedSessionArtifact{}, fmt.Errorf("catalog-backed artifact source %q requires runtime artifact catalog", source)
}
entry, ok := catalog.Lookup(normalized)
if !ok {
@@ -223,7 +241,11 @@ func ResolveSessionArtifactWithCatalog(paths SessionPaths, m *manifest.Manifest,
if !entry.Available {
return ResolvedSessionArtifact{}, &SessionArtifactNotFoundError{ArtifactID: normalized}
}
if err := validateResolvedContent(entry.Path, contentText); err != nil {
contentKind := contentText
if extractionSource {
contentKind = contentJSON
}
if err := validateResolvedContent(entry.Path, contentKind); err != nil {
return ResolvedSessionArtifact{}, fmt.Errorf("validate %q: %w", normalized, err)
}
return ResolvedSessionArtifact{
@@ -231,6 +253,7 @@ func ResolveSessionArtifactWithCatalog(paths SessionPaths, m *manifest.Manifest,
Path: filepath.Clean(entry.Path),
ProducerStage: entry.ProducerStage,
OutputKind: entry.OutputKind,
ProducerRunID: entry.ProducerRunID,
Provenance: entry.Provenance,
}, nil
}

View File

@@ -150,6 +150,23 @@ func TestPreviousSessionArtifactSourceHelpers(t *testing.T) {
}
}
func TestExtractionArtifactSourceHelpers(t *testing.T) {
if got := ExtractionArtifactSourceID("npc_registry"); got != "narratio.extraction.npc_registry" {
t.Fatalf("ExtractionArtifactSourceID() = %q", got)
}
if !IsExtractionArtifactSource(" narratio.extraction.npc_registry ") {
t.Fatal("IsExtractionArtifactSource(valid) = false")
}
if key, ok := ExtractionArtifactName("narratio.extraction.npc_registry"); !ok || key != "npc_registry" {
t.Fatalf("ExtractionArtifactName() = %q, %t", key, ok)
}
for _, source := range []string{"narratio.extraction.", "narratio.extraction.npc-registry", "narratio.artifact.npc_registry"} {
if IsExtractionArtifactSource(source) {
t.Fatalf("IsExtractionArtifactSource(%q) = true, want false", source)
}
}
}
func TestResolveSessionArtifactPrefersManifestOutput(t *testing.T) {
workspace := t.TempDir()
paths := buildSessionPaths(workspace, "campaign", "session")

View File

@@ -6,11 +6,13 @@ import (
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/artifactpolicy"
"gitea.maximumdirect.net/eric/narratio/internal/config"
)
const (
ArtifactProvenanceGeneratedCurrentAnalyzeRun = "generated.current_analyze_run"
ArtifactProvenanceDisabledFromDisk = "filesystem.disabled_artifact_output"
ArtifactProvenanceCurrentExtractManifest = "manifest.current_extract_run"
)
// ConfiguredArtifactDefinition describes one configured analyze artifact.
@@ -19,10 +21,40 @@ type ConfiguredArtifactDefinition struct {
OutputPath string
}
// ExtractionArtifactDefinition describes one configured Notarius output lane.
type ExtractionArtifactDefinition struct {
LaneID string
PipelineID string
MediaType string
SchemaID string
SchemaVersion string
ModuleKey string
}
// ExtractionDefinitionsFromConfig converts the effective Notarius output map into catalog definitions.
func ExtractionDefinitionsFromConfig(cfg *config.NotariusConfig) map[string]ExtractionArtifactDefinition {
if cfg == nil || len(cfg.Outputs) == 0 {
return nil
}
definitions := make(map[string]ExtractionArtifactDefinition, len(cfg.Outputs))
for key, output := range cfg.Outputs {
definitions[key] = ExtractionArtifactDefinition{
LaneID: output.LaneID,
PipelineID: cfg.PipelineID,
MediaType: output.MediaType,
SchemaID: output.SchemaID,
SchemaVersion: output.SchemaVersion,
ModuleKey: output.ModuleKey,
}
}
return definitions
}
// CatalogEntry is one runtime catalog entry resolved by source ID.
type CatalogEntry struct {
SourceID string
ConfiguredKey string
ExtractionKey string
CanonicalRelPath string
ProducerStage string
OutputKind string
@@ -31,12 +63,14 @@ type CatalogEntry struct {
Available bool
Path string
Provenance string
ProducerRunID string
}
// ArtifactCatalog tracks built-in and configured artifact definitions and runtime state.
// ArtifactCatalog tracks built-in, configured, and extraction artifact definitions and runtime state.
type ArtifactCatalog struct {
entries map[string]CatalogEntry
configuredIndex map[string]string
extractionIndex map[string]string
}
// NewArtifactCatalog returns an empty runtime artifact catalog.
@@ -44,9 +78,41 @@ func NewArtifactCatalog() *ArtifactCatalog {
return &ArtifactCatalog{
entries: map[string]CatalogEntry{},
configuredIndex: map[string]string{},
extractionIndex: map[string]string{},
}
}
// RegisterExtractionArtifacts registers the configured Notarius output lanes.
func (c *ArtifactCatalog) RegisterExtractionArtifacts(configured map[string]ExtractionArtifactDefinition) error {
keys := make([]string, 0, len(configured))
for key := range configured {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
trimmed := strings.TrimSpace(key)
if trimmed == "" {
return fmt.Errorf("extraction artifact keys must be non-empty")
}
if _, exists := c.extractionIndex[trimmed]; exists {
return fmt.Errorf("duplicate extraction artifact key %q", trimmed)
}
sourceID := ExtractionArtifactSourceID(trimmed)
if err := c.addEntry(CatalogEntry{
SourceID: sourceID,
ExtractionKey: trimmed,
ProducerStage: "extract",
OutputKind: "notarius_lane",
Planned: true,
}); err != nil {
return fmt.Errorf("register extraction artifact %q: %w", trimmed, err)
}
c.extractionIndex[trimmed] = sourceID
}
return nil
}
// ConfiguredArtifactSourceID converts a configured artifact key into canonical source ID.
func ConfiguredArtifactSourceID(key string) string {
return artifactpolicy.ConfiguredSourceID(key)
@@ -151,6 +217,15 @@ func (c *ArtifactCatalog) SourceIDForConfiguredKey(key string) (string, bool) {
return sourceID, ok
}
// SourceIDForExtractionKey returns the canonical source ID for one Notarius output key.
func (c *ArtifactCatalog) SourceIDForExtractionKey(key string) (string, bool) {
if c == nil {
return "", false
}
sourceID, ok := c.extractionIndex[strings.TrimSpace(key)]
return sourceID, ok
}
// ListConfigured returns configured entries sorted by configured key.
func (c *ArtifactCatalog) ListConfigured() []CatalogEntry {
if c == nil || len(c.configuredIndex) == 0 {
@@ -169,6 +244,23 @@ func (c *ArtifactCatalog) ListConfigured() []CatalogEntry {
return out
}
// ListExtraction returns extraction entries sorted by configured output key.
func (c *ArtifactCatalog) ListExtraction() []CatalogEntry {
if c == nil || len(c.extractionIndex) == 0 {
return nil
}
keys := make([]string, 0, len(c.extractionIndex))
for key := range c.extractionIndex {
keys = append(keys, key)
}
sort.Strings(keys)
out := make([]CatalogEntry, 0, len(keys))
for _, key := range keys {
out = append(out, c.entries[c.extractionIndex[key]])
}
return out
}
// MarkAvailableGenerated marks one source as available in current analyze execution.
func (c *ArtifactCatalog) MarkAvailableGenerated(sourceID, path string) error {
return c.markAvailable(sourceID, path, ArtifactProvenanceGeneratedCurrentAnalyzeRun)
@@ -179,6 +271,16 @@ func (c *ArtifactCatalog) MarkAvailableFromDisk(sourceID, path string) error {
return c.markAvailable(sourceID, path, ArtifactProvenanceDisabledFromDisk)
}
func (c *ArtifactCatalog) markAvailableFromExtractManifest(sourceID, path, producerRunID string) error {
if err := c.markAvailable(sourceID, path, ArtifactProvenanceCurrentExtractManifest); err != nil {
return err
}
entry := c.entries[strings.TrimSpace(sourceID)]
entry.ProducerRunID = strings.TrimSpace(producerRunID)
c.entries[entry.SourceID] = entry
return nil
}
func (c *ArtifactCatalog) markAvailable(sourceID, path, provenance string) error {
if c == nil {
return fmt.Errorf("artifact catalog is nil")

View File

@@ -0,0 +1,180 @@
package artifacts
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
const (
extractStageName = "extract"
extractionLaneKind = "notarius_lane"
extractionIndexKind = "notarius_index"
extractionMetadataRun = "narratio_run_id"
extractionMetadataRoot = "bundle_root"
)
type hydratedExtraction struct {
sourceID string
path string
}
// HydrateExtractionArtifacts marks extraction sources available only when the current
// manifest contains one complete, internally consistent, succeeded extraction bundle.
// Invalid, stale, incomplete, or unsafe records leave every extraction source unavailable.
func (c *ArtifactCatalog) HydrateExtractionArtifacts(
paths SessionPaths,
m *manifest.Manifest,
configured map[string]ExtractionArtifactDefinition,
) {
if c == nil || m == nil || len(configured) == 0 {
return
}
record := m.Stages[extractStageName]
if record == nil || record.Name != extractStageName || record.Status != manifest.StatusSucceeded {
return
}
producerRunID := extractionMetadataString(record.Metadata, extractionMetadataRun)
if !safeExtractionPathSegment(producerRunID) {
return
}
bundleRoot := filepath.Clean(filepath.Join(paths.ArtifactsDir, "notarius", producerRunID))
if !filepath.IsAbs(bundleRoot) || extractionMetadataString(record.Metadata, extractionMetadataRoot) != bundleRoot {
return
}
if !safeExistingExtractionDirectory(paths.Root, bundleRoot) {
return
}
receiptRunID, receiptPipelineID := extractionReceiptIdentity(record.Metadata)
if receiptRunID == "" || receiptPipelineID == "" {
return
}
expected := make(map[string]ExtractionArtifactDefinition, len(configured))
for key, definition := range configured {
sourceID, ok := c.SourceIDForExtractionKey(key)
if !ok {
return
}
expected[sourceID] = definition
}
seen := make(map[string]struct{}, len(expected))
hydrated := make([]hydratedExtraction, 0, len(expected))
indexSeen := false
for _, output := range record.Outputs {
if strings.TrimSpace(output.ProducerRunID) != producerRunID {
return
}
if output.SourceID == "" {
if indexSeen || output.Kind != extractionIndexKind || filepath.Clean(output.LocalPath) != filepath.Join(bundleRoot, "index.json") ||
!validExtractionPayload(bundleRoot, output.LocalPath, output.Checksum) {
return
}
indexSeen = true
continue
}
definition, ok := expected[output.SourceID]
if !ok || output.Kind != extractionLaneKind {
return
}
if _, duplicate := seen[output.SourceID]; duplicate {
return
}
if !compatibleCatalogExtractionContract(output.Contract, definition) ||
!compatibleCatalogExtractionProvenance(output.ExternalProvenance, receiptRunID, receiptPipelineID, definition) ||
!validExtractionPayload(bundleRoot, output.LocalPath, output.Checksum) {
return
}
seen[output.SourceID] = struct{}{}
hydrated = append(hydrated, hydratedExtraction{sourceID: output.SourceID, path: output.LocalPath})
}
if !indexSeen || len(seen) != len(expected) || len(record.Outputs) != len(expected)+1 {
return
}
for _, item := range hydrated {
_ = c.markAvailableFromExtractManifest(item.sourceID, item.path, producerRunID)
}
}
func compatibleCatalogExtractionContract(got *artifactmodel.ContractMetadata, want ExtractionArtifactDefinition) bool {
return got != nil && got.MediaType == want.MediaType && got.SchemaID == want.SchemaID &&
got.SchemaVersion == want.SchemaVersion && (want.ModuleKey == "" || got.ModuleKey == want.ModuleKey)
}
func compatibleCatalogExtractionProvenance(
got *artifactmodel.ExternalProvenance,
runID, pipelineID string,
want ExtractionArtifactDefinition,
) bool {
return got != nil && got.System == "notarius" && got.RunID == runID && got.PipelineID == pipelineID &&
pipelineID == strings.TrimSpace(want.PipelineID) && got.ArtifactID == want.LaneID
}
func validExtractionPayload(bundleRoot, path, checksum string) bool {
if !filepath.IsAbs(path) || !pathWithinExtractionRoot(bundleRoot, path) || strings.TrimSpace(checksum) == "" {
return false
}
info, err := os.Lstat(path)
if err != nil || !info.Mode().IsRegular() || info.Mode()&os.ModeSymlink != 0 {
return false
}
if !safeExtractionComponents(bundleRoot, path) {
return false
}
actual, err := SHA256File(path)
if err != nil || actual != checksum {
return false
}
body, err := os.ReadFile(path)
return err == nil && json.Valid(body)
}
func safeExistingExtractionDirectory(sessionRoot, bundleRoot string) bool {
if !pathWithinExtractionRoot(sessionRoot, bundleRoot) || !safeExtractionComponents(sessionRoot, bundleRoot) {
return false
}
info, err := os.Lstat(bundleRoot)
return err == nil && info.IsDir() && info.Mode()&os.ModeSymlink == 0
}
func safeExtractionComponents(root, target string) bool {
relative, err := filepath.Rel(filepath.Clean(root), filepath.Clean(target))
if err != nil || relative == "." || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
return false
}
current := filepath.Clean(root)
for _, part := range strings.Split(relative, string(filepath.Separator)) {
current = filepath.Join(current, part)
info, err := os.Lstat(current)
if err != nil || info.Mode()&os.ModeSymlink != 0 {
return false
}
}
return true
}
func pathWithinExtractionRoot(root, target string) bool {
relative, err := filepath.Rel(filepath.Clean(root), filepath.Clean(target))
return err == nil && relative != "." && relative != ".." && !strings.HasPrefix(relative, ".."+string(filepath.Separator))
}
func safeExtractionPathSegment(value string) bool {
return value != "" && value != "." && value != ".." && filepath.Base(value) == value &&
!strings.ContainsAny(value, `/\\`)
}
func extractionMetadataString(metadata map[string]any, key string) string {
value, _ := metadata[key].(string)
return strings.TrimSpace(value)
}
func extractionReceiptIdentity(metadata map[string]any) (string, string) {
receipt, _ := metadata["receipt"].(map[string]any)
return extractionMetadataString(receipt, "run_id"), extractionMetadataString(receipt, "pipeline_id")
}

View File

@@ -0,0 +1,222 @@
package artifacts
import (
"errors"
"os"
"path/filepath"
"testing"
"time"
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
"gitea.maximumdirect.net/eric/narratio/internal/manifest"
)
func TestArtifactCatalogRegistersExtractionArtifactsDeterministically(t *testing.T) {
catalog := NewArtifactCatalog()
if err := catalog.RegisterConfiguredArtifacts(map[string]ConfiguredArtifactDefinition{
"summary": {Enabled: true},
}, nil); err != nil {
t.Fatal(err)
}
if err := catalog.RegisterExtractionArtifacts(map[string]ExtractionArtifactDefinition{
"zeta": {LaneID: "zeta"},
"summary": {LaneID: "summary"},
"alpha": {LaneID: "alpha"},
}); err != nil {
t.Fatalf("RegisterExtractionArtifacts() error = %v", err)
}
entries := catalog.ListExtraction()
if len(entries) != 3 || entries[0].ExtractionKey != "alpha" || entries[1].ExtractionKey != "summary" || entries[2].ExtractionKey != "zeta" {
t.Fatalf("ListExtraction() = %#v, want alpha, summary, zeta", entries)
}
extractionID, ok := catalog.SourceIDForExtractionKey("summary")
if !ok || extractionID != "narratio.extraction.summary" {
t.Fatalf("SourceIDForExtractionKey(summary) = %q, %v", extractionID, ok)
}
configuredID, _ := catalog.SourceIDForConfiguredKey("summary")
if configuredID == extractionID {
t.Fatalf("configured and extraction source families collided at %q", extractionID)
}
entry, ok := catalog.Lookup(extractionID)
if !ok || entry.ProducerStage != "extract" || entry.OutputKind != "notarius_lane" || !entry.Planned {
t.Fatalf("Lookup(%q) = %#v, %v", extractionID, entry, ok)
}
}
func TestArtifactCatalogRejectsDuplicateExtractionRegistration(t *testing.T) {
catalog := NewArtifactCatalog()
definitions := map[string]ExtractionArtifactDefinition{"summary": {LaneID: "summary"}}
if err := catalog.RegisterExtractionArtifacts(definitions); err != nil {
t.Fatal(err)
}
if err := catalog.RegisterExtractionArtifacts(definitions); err == nil {
t.Fatal("second RegisterExtractionArtifacts() error = nil, want collision error")
}
}
func TestHydrateExtractionArtifactsAcceptsOnlyCompleteCurrentBundle(t *testing.T) {
paths, currentManifest, definitions := validExtractionCatalogFixture(t)
catalog := registeredExtractionCatalog(t, definitions)
catalog.HydrateExtractionArtifacts(paths, currentManifest, definitions)
entry, ok := catalog.Lookup(ExtractionArtifactSourceID("encounters"))
if !ok || !entry.Available {
t.Fatalf("hydrated entry = %#v, %v; want available", entry, ok)
}
if entry.Provenance != ArtifactProvenanceCurrentExtractManifest || entry.ProducerRunID != "extract-run-1" {
t.Fatalf("hydrated provenance = %#v", entry)
}
resolved, err := ResolveSessionArtifactWithCatalog(paths, currentManifest, entry.SourceID, catalog)
if err != nil {
t.Fatalf("ResolveSessionArtifactWithCatalog() error = %v", err)
}
if resolved.Path != entry.Path || resolved.ProducerRunID != "extract-run-1" {
t.Fatalf("resolved = %#v", resolved)
}
}
func TestHydrateExtractionArtifactsRejectsUntrustedManifestState(t *testing.T) {
tests := []struct {
name string
mutate func(t *testing.T, paths SessionPaths, m *manifest.Manifest)
}{
{name: "missing stage", mutate: func(_ *testing.T, _ SessionPaths, m *manifest.Manifest) { delete(m.Stages, "extract") }},
{name: "skipped", mutate: setExtractionStatus(manifest.StatusSkipped)},
{name: "failed", mutate: setExtractionStatus(manifest.StatusFailed)},
{name: "stale", mutate: setExtractionStatus(manifest.StatusStale)},
{name: "interrupted", mutate: setExtractionStatus(manifest.StatusInterrupted)},
{name: "missing source", mutate: func(_ *testing.T, _ SessionPaths, m *manifest.Manifest) {
m.Stages["extract"].Outputs = m.Stages["extract"].Outputs[1:]
}},
{name: "inconsistent producer identity", mutate: func(_ *testing.T, _ SessionPaths, m *manifest.Manifest) {
m.Stages["extract"].Outputs[0].ProducerRunID = "another-run"
}},
{name: "incompatible contract", mutate: func(_ *testing.T, _ SessionPaths, m *manifest.Manifest) {
m.Stages["extract"].Outputs[0].Contract.SchemaVersion = "99"
}},
{name: "incompatible provenance", mutate: func(_ *testing.T, _ SessionPaths, m *manifest.Manifest) {
m.Stages["extract"].Outputs[0].ExternalProvenance.RunID = "another-notarius-run"
}},
{name: "missing file", mutate: func(t *testing.T, _ SessionPaths, m *manifest.Manifest) {
if err := os.Remove(m.Stages["extract"].Outputs[0].LocalPath); err != nil {
t.Fatal(err)
}
}},
{name: "tampered checksum", mutate: func(t *testing.T, _ SessionPaths, m *manifest.Manifest) {
if err := os.WriteFile(m.Stages["extract"].Outputs[0].LocalPath, []byte(`{"tampered":true}`), 0o644); err != nil {
t.Fatal(err)
}
}},
{name: "unsafe path", mutate: func(t *testing.T, paths SessionPaths, m *manifest.Manifest) {
outside := filepath.Join(paths.Root, "incidental.json")
writeExtractionFixtureFile(t, outside, `{"incidental":true}`)
m.Stages["extract"].Outputs[0].LocalPath = outside
m.Stages["extract"].Outputs[0].Checksum = extractionFixtureChecksum(t, outside)
}},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
paths, currentManifest, definitions := validExtractionCatalogFixture(t)
test.mutate(t, paths, currentManifest)
catalog := registeredExtractionCatalog(t, definitions)
catalog.HydrateExtractionArtifacts(paths, currentManifest, definitions)
entry, _ := catalog.Lookup(ExtractionArtifactSourceID("encounters"))
if entry.Available {
t.Fatalf("entry became available from %s manifest", test.name)
}
})
}
}
func TestResolveExtractionArtifactNeverDiscoversIncidentalBundleFile(t *testing.T) {
root := t.TempDir()
paths := SessionPaths{Root: root, ArtifactsDir: filepath.Join(root, "artifacts")}
incidental := filepath.Join(paths.ArtifactsDir, "notarius", "incidental", "lanes", "encounters.json")
writeExtractionFixtureFile(t, incidental, `{"encounters":[]}`)
definitions := extractionFixtureDefinitions()
catalog := registeredExtractionCatalog(t, definitions)
_, err := ResolveSessionArtifactWithCatalog(paths, manifest.New("session", fixtureTime), ExtractionArtifactSourceID("encounters"), catalog)
if err == nil || !errors.Is(err, ErrSessionArtifactNotFound) {
t.Fatalf("ResolveSessionArtifactWithCatalog() error = %v, want not found", err)
}
}
var fixtureTime = mustFixtureTime()
func mustFixtureTime() (value time.Time) {
return time.Date(2026, 8, 10, 0, 0, 0, 0, time.UTC)
}
func setExtractionStatus(status manifest.StageStatus) func(*testing.T, SessionPaths, *manifest.Manifest) {
return func(_ *testing.T, _ SessionPaths, m *manifest.Manifest) { m.Stages["extract"].Status = status }
}
func extractionFixtureDefinitions() map[string]ExtractionArtifactDefinition {
return map[string]ExtractionArtifactDefinition{
"encounters": {
LaneID: "encounters", PipelineID: "campaign.extract", MediaType: "application/json",
SchemaID: "encounters", SchemaVersion: "1", ModuleKey: "encounters",
},
}
}
func registeredExtractionCatalog(t *testing.T, definitions map[string]ExtractionArtifactDefinition) *ArtifactCatalog {
t.Helper()
catalog := NewArtifactCatalog()
if err := catalog.RegisterExtractionArtifacts(definitions); err != nil {
t.Fatal(err)
}
return catalog
}
func validExtractionCatalogFixture(t *testing.T) (SessionPaths, *manifest.Manifest, map[string]ExtractionArtifactDefinition) {
t.Helper()
root := t.TempDir()
paths := SessionPaths{Root: root, ArtifactsDir: filepath.Join(root, "artifacts")}
bundleRoot := filepath.Join(paths.ArtifactsDir, "notarius", "extract-run-1")
lanePath := filepath.Join(bundleRoot, "lanes", "encounters.json")
indexPath := filepath.Join(bundleRoot, "index.json")
writeExtractionFixtureFile(t, lanePath, `{"encounters":[]}`)
writeExtractionFixtureFile(t, indexPath, `{"lanes":[]}`)
definitions := extractionFixtureDefinitions()
m := manifest.New("session", fixtureTime)
m.Stages["extract"] = &manifest.StageRecord{
Name: "extract", Status: manifest.StatusSucceeded,
Metadata: map[string]any{
"narratio_run_id": "extract-run-1", "bundle_root": bundleRoot,
"receipt": map[string]any{"run_id": "notarius-run-1", "pipeline_id": "campaign.extract"},
},
Outputs: []manifest.ArtifactRecord{
{
Kind: "notarius_lane", SourceID: ExtractionArtifactSourceID("encounters"), LocalPath: lanePath,
ProducerRunID: "extract-run-1", Checksum: extractionFixtureChecksum(t, lanePath),
Contract: &artifactmodel.ContractMetadata{MediaType: "application/json", SchemaID: "encounters", SchemaVersion: "1", ModuleKey: "encounters"},
ExternalProvenance: &artifactmodel.ExternalProvenance{System: "notarius", RunID: "notarius-run-1", PipelineID: "campaign.extract", ArtifactID: "encounters"},
},
{Kind: "notarius_index", LocalPath: indexPath, ProducerRunID: "extract-run-1", Checksum: extractionFixtureChecksum(t, indexPath)},
},
}
return paths, m, definitions
}
func writeExtractionFixtureFile(t *testing.T, path, body string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
func extractionFixtureChecksum(t *testing.T, path string) string {
t.Helper()
checksum, err := SHA256File(path)
if err != nil {
t.Fatal(err)
}
return checksum
}

View File

@@ -86,6 +86,36 @@ func SessionRunStageDirForCampaign(rootDir, campaign, sessionID, runID, stageNam
return filepath.Join(SessionRunRootForCampaign(rootDir, campaign, sessionID, runID), stageName)
}
// SessionRunExtractDirForCampaign returns the invocation-local extraction directory.
func SessionRunExtractDirForCampaign(rootDir, campaign, sessionID, runID string) string {
return SessionRunStageDirForCampaign(rootDir, campaign, sessionID, runID, "extract")
}
// SessionRunNotariusReceiptPathForCampaign returns the invocation-local receipt path.
func SessionRunNotariusReceiptPathForCampaign(rootDir, campaign, sessionID, runID string) string {
return filepath.Join(SessionRunExtractDirForCampaign(rootDir, campaign, sessionID, runID), "notarius.receipt.json")
}
// SessionRunNotariusLogPathForCampaign returns the invocation-local stderr log path.
func SessionRunNotariusLogPathForCampaign(rootDir, campaign, sessionID, runID string) string {
return filepath.Join(SessionRunExtractDirForCampaign(rootDir, campaign, sessionID, runID), "notarius.stderr.log")
}
// SessionRunNotariusOutputRootForCampaign returns the invocation-local Notarius output root.
func SessionRunNotariusOutputRootForCampaign(rootDir, campaign, sessionID, runID string) string {
return filepath.Join(SessionRunExtractDirForCampaign(rootDir, campaign, sessionID, runID), "notarius-output")
}
// SessionNotariusBundleDirForCampaign returns one immutable durable bundle destination.
func SessionNotariusBundleDirForCampaign(rootDir, campaign, sessionID, runID string) string {
return filepath.Join(
SessionWorkDirForCampaign(rootDir, campaign, sessionID),
config.PathArtifactsDirSegment,
"notarius",
runID,
)
}
// SessionSpoolAudioDir returns the campaign/session/run scoped local spool audio path.
func SessionSpoolAudioDir(spoolRoot, campaign, sessionID, runID string) string {
return filepath.Join(spoolRoot, campaign, sessionID, runID, config.PathAudioDirSegment)

View File

@@ -49,6 +49,33 @@ func TestSessionRunManifestPathForCampaign(t *testing.T) {
}
}
func TestSessionNotariusPathsForCampaign(t *testing.T) {
root := "/tmp/workspace"
campaign := "forsaken"
sessionID := "2026-04-19"
runID := "20260515T031522Z-a1b2c3d4"
extractDir := filepath.Join(root, "work", campaign, sessionID, "runs", runID, "extract")
tests := []struct {
name string
got string
want string
}{
{name: "extract directory", got: SessionRunExtractDirForCampaign(root, campaign, sessionID, runID), want: extractDir},
{name: "receipt", got: SessionRunNotariusReceiptPathForCampaign(root, campaign, sessionID, runID), want: filepath.Join(extractDir, "notarius.receipt.json")},
{name: "stderr", got: SessionRunNotariusLogPathForCampaign(root, campaign, sessionID, runID), want: filepath.Join(extractDir, "notarius.stderr.log")},
{name: "output root", got: SessionRunNotariusOutputRootForCampaign(root, campaign, sessionID, runID), want: filepath.Join(extractDir, "notarius-output")},
{name: "durable bundle", got: SessionNotariusBundleDirForCampaign(root, campaign, sessionID, runID), want: filepath.Join(root, "work", campaign, sessionID, "artifacts", "notarius", runID)},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
if test.got != test.want {
t.Fatalf("path = %q, want %q", test.got, test.want)
}
})
}
}
func TestSessionPreviousPathsForCampaign(t *testing.T) {
root := "/tmp/workspace"
previousDir := SessionPreviousDirForCampaign(root, "forsaken", "2026-04-19")

View File

@@ -1,57 +0,0 @@
package artifacts
import (
"fmt"
"os"
)
// S3Store is a placeholder for future remote artifact persistence support.
type S3Store struct {
Bucket string
Prefix string
}
// SessionPathsFor is not implemented for S3-backed storage.
func (s *S3Store) SessionPathsFor(_, _ string) SessionPaths {
return SessionPaths{}
}
// EnsureLayoutFor returns a not-yet-implemented error in the scaffold.
func (s *S3Store) EnsureLayoutFor(_, _ string) (SessionPaths, error) {
return SessionPaths{}, fmt.Errorf("artifacts s3 ensure layout for campaign/session: not yet implemented")
}
// CopyInputFor returns a not-yet-implemented error in the scaffold.
func (s *S3Store) CopyInputFor(_, _, _, _ string) (Ref, error) {
return Ref{}, fmt.Errorf("artifacts s3 copy input for campaign/session: not yet implemented")
}
// Exists returns a not-yet-implemented error in the scaffold.
func (s *S3Store) Exists(_ string) (bool, error) {
return false, fmt.Errorf("artifacts s3 exists: not yet implemented")
}
// ExistsRef returns a not-yet-implemented error in the scaffold.
func (s *S3Store) ExistsRef(_ Ref) (bool, error) {
return false, fmt.Errorf("artifacts s3 exists ref: not yet implemented")
}
// WriteFileAtomic returns a not-yet-implemented error in the scaffold.
func (s *S3Store) WriteFileAtomic(_ string, _ []byte, _ os.FileMode) error {
return fmt.Errorf("artifacts s3 write file atomic: not yet implemented")
}
// Checksum returns a not-yet-implemented error in the scaffold.
func (s *S3Store) Checksum(_ string) (string, error) {
return "", fmt.Errorf("artifacts s3 checksum: not yet implemented")
}
// AcquireSessionLockFor returns a not-yet-implemented error in the scaffold.
func (s *S3Store) AcquireSessionLockFor(_, _ string) (*LockHandle, error) {
return nil, fmt.Errorf("artifacts s3 acquire lock for campaign/session: not yet implemented")
}
// ReleaseSessionLock returns a not-yet-implemented error in the scaffold.
func (s *S3Store) ReleaseSessionLock(_ *LockHandle) error {
return fmt.Errorf("artifacts s3 release lock: not yet implemented")
}

View File

@@ -1,16 +1,23 @@
package artifacts
import "os"
import (
"os"
"gitea.maximumdirect.net/eric/narratio/internal/artifactmodel"
)
// Ref identifies a pipeline artifact and its local/remote coordinates.
type Ref struct {
Kind string
Category string
SessionID string
RelativePath string
AbsolutePath string
RemoteKey string
Checksum string
Kind string
SourceID string
Category string
SessionID string
RelativePath string
AbsolutePath string
RemoteKey string
Checksum string
Contract *artifactmodel.ContractMetadata
ExternalProvenance *artifactmodel.ExternalProvenance
}
// Store is the local artifact/workdir abstraction used by orchestration code.

View File

@@ -37,7 +37,7 @@ notification:
func TestCampaignStrictDecodeAcceptsCampaignID(t *testing.T) {
pipelinePath, campaignPath, sessionPath := writeCampaignConfigTestFiles(t,
"campaign_id: sample-campaign\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n",
"campaign_id: sample-campaign\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n players_file: ./players.yml\n party_file: ./party.yml\n",
"session_id: 2026-05-03\ninputs:\n audio_dir: ./audio\n",
)
@@ -52,7 +52,7 @@ func TestCampaignStrictDecodeAcceptsCampaignID(t *testing.T) {
func TestCampaignStrictDecodeRejectsLegacyCampaignField(t *testing.T) {
pipelinePath, campaignPath, sessionPath := writeCampaignConfigTestFiles(t,
"campaign: sample-campaign\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n",
"campaign: sample-campaign\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n players_file: ./players.yml\n party_file: ./party.yml\n",
"session_id: 2026-05-03\ninputs:\n audio_dir: ./audio\n",
)
@@ -67,7 +67,7 @@ func TestCampaignStrictDecodeRejectsLegacyCampaignField(t *testing.T) {
func TestCampaignStrictDecodeRejectsUnknownFields(t *testing.T) {
pipelinePath, campaignPath, sessionPath := writeCampaignConfigTestFiles(t,
"campaign_id: sample-campaign\nunknown: true\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n",
"campaign_id: sample-campaign\nunknown: true\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n players_file: ./players.yml\n party_file: ./party.yml\n",
"session_id: 2026-05-03\ninputs:\n audio_dir: ./audio\n",
)
@@ -82,7 +82,7 @@ func TestCampaignStrictDecodeRejectsUnknownFields(t *testing.T) {
func TestCampaignStrictDecodeAcceptsSessionTemplateFile(t *testing.T) {
pipelinePath, campaignPath, sessionPath := writeCampaignConfigTestFiles(t,
"campaign_id: sample-campaign\nsession_template_file: ./session.template.yml\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n",
"campaign_id: sample-campaign\nsession_template_file: ./session.template.yml\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n players_file: ./players.yml\n party_file: ./party.yml\n",
"session_id: 2026-05-03\ninputs:\n audio_dir: ./audio\n",
)
@@ -97,7 +97,7 @@ func TestCampaignStrictDecodeAcceptsSessionTemplateFile(t *testing.T) {
func TestCampaignSessionMergeFillsStableInputs(t *testing.T) {
pipelinePath, campaignPath, sessionPath := writeCampaignConfigTestFiles(t,
"campaign_id: sample-campaign\ninputs:\n speakers_file: ./campaign-speakers.yml\n autocorrect_file: ./campaign-autocorrect.yml\n glossary_file: ./campaign-glossary.yml\n",
"campaign_id: sample-campaign\ninputs:\n speakers_file: ./campaign-speakers.yml\n autocorrect_file: ./campaign-autocorrect.yml\n glossary_file: ./campaign-glossary.yml\n players_file: ./campaign-players.yml\n party_file: ./campaign-party.yml\n",
"session_id: 2026-05-03\ninputs:\n audio_dir: ./audio\n",
)
@@ -114,12 +114,14 @@ func TestCampaignSessionMergeFillsStableInputs(t *testing.T) {
assertResolvedStableInput(t, cfg.StableInputs.SpeakersFile, "./campaign-speakers.yml", campaignPath, "campaign_config")
assertResolvedStableInput(t, cfg.StableInputs.AutocorrectFile, "./campaign-autocorrect.yml", campaignPath, "campaign_config")
assertResolvedStableInput(t, cfg.StableInputs.GlossaryFile, "./campaign-glossary.yml", campaignPath, "campaign_config")
assertResolvedStableInput(t, cfg.StableInputs.PlayersFile, "./campaign-players.yml", campaignPath, "campaign_config")
assertResolvedStableInput(t, cfg.StableInputs.PartyFile, "./campaign-party.yml", campaignPath, "campaign_config")
}
func TestCampaignSessionMergeSessionOverridesStableInputs(t *testing.T) {
pipelinePath, campaignPath, sessionPath := writeCampaignConfigTestFiles(t,
"campaign_id: sample-campaign\ninputs:\n speakers_file: ./campaign-speakers.yml\n autocorrect_file: ./campaign-autocorrect.yml\n glossary_file: ./campaign-glossary.yml\n",
"session_id: 2026-05-03\ncampaign: sample-campaign\ninputs:\n audio_dir: ./audio\n speakers_file: ./session-speakers.yml\n",
"campaign_id: sample-campaign\ninputs:\n speakers_file: ./campaign-speakers.yml\n autocorrect_file: ./campaign-autocorrect.yml\n glossary_file: ./campaign-glossary.yml\n players_file: ./campaign-players.yml\n party_file: ./campaign-party.yml\n",
"session_id: 2026-05-03\ncampaign: sample-campaign\ninputs:\n audio_dir: ./audio\n speakers_file: ./session-speakers.yml\n players_file: ./session-players.yml\n party_file: ./session-party.yml\n",
)
cfg, err := LoadWithSessionOptions(pipelinePath, campaignPath, sessionPath, SessionLoadOptions{})
@@ -132,11 +134,32 @@ func TestCampaignSessionMergeSessionOverridesStableInputs(t *testing.T) {
assertResolvedStableInput(t, cfg.StableInputs.SpeakersFile, "./session-speakers.yml", sessionPath, "session_config")
assertResolvedStableInput(t, cfg.StableInputs.AutocorrectFile, "./campaign-autocorrect.yml", campaignPath, "campaign_config")
assertResolvedStableInput(t, cfg.StableInputs.GlossaryFile, "./campaign-glossary.yml", campaignPath, "campaign_config")
assertResolvedStableInput(t, cfg.StableInputs.PlayersFile, "./session-players.yml", sessionPath, "session_config")
assertResolvedStableInput(t, cfg.StableInputs.PartyFile, "./session-party.yml", sessionPath, "session_config")
}
func TestCampaignRequiresPlayersAndPartyInputs(t *testing.T) {
pipelinePath, campaignPath, sessionPath := writeCampaignConfigTestFiles(t,
"campaign_id: sample-campaign\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n",
"session_id: 2026-05-03\ninputs:\n audio_dir: ./audio\n",
)
cfg, err := LoadWithSessionOptions(pipelinePath, campaignPath, sessionPath, SessionLoadOptions{})
if err != nil {
t.Fatalf("LoadWithSessionOptions() error = %v", err)
}
err = Validate(cfg)
if err == nil {
t.Fatal("expected validation error, got nil")
}
if !strings.Contains(err.Error(), "campaign.inputs.players_file is required") {
t.Fatalf("error = %q, want players_file required", err.Error())
}
}
func TestCampaignSessionMismatchFails(t *testing.T) {
pipelinePath, campaignPath, sessionPath := writeCampaignConfigTestFiles(t,
"campaign_id: sample-campaign\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n",
"campaign_id: sample-campaign\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n players_file: ./players.yml\n party_file: ./party.yml\n",
"session_id: 2026-05-03\ncampaign: other-campaign\ninputs:\n audio_dir: ./audio\n",
)
@@ -151,7 +174,7 @@ func TestCampaignSessionMismatchFails(t *testing.T) {
func TestLoadMissingCampaignFileFails(t *testing.T) {
pipelinePath, _, sessionPath := writeCampaignConfigTestFiles(t,
"campaign_id: sample-campaign\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n",
"campaign_id: sample-campaign\ninputs:\n speakers_file: ./speakers.yml\n autocorrect_file: ./autocorrect.yml\n glossary_file: ./glossary.yml\n players_file: ./players.yml\n party_file: ./party.yml\n",
"session_id: 2026-05-03\ninputs:\n audio_dir: ./audio\n",
)
missingCampaignPath := filepath.Join(filepath.Dir(sessionPath), "missing-campaign.yml")

View File

@@ -30,6 +30,7 @@ type PipelineConfig struct {
Trim *TrimConfig `yaml:"trim"`
Render *RenderConfig `yaml:"render"`
Scriptorium *ScriptoriumConfig `yaml:"scriptorium"`
Notarius *NotariusConfig `yaml:"notarius"`
Notification NotificationConfig `yaml:"notification"`
}
@@ -51,6 +52,8 @@ type CampaignInputsConfig struct {
SpeakersFile string `yaml:"speakers_file"`
AutocorrectFile string `yaml:"autocorrect_file"`
GlossaryFile string `yaml:"glossary_file"`
PlayersFile string `yaml:"players_file"`
PartyFile string `yaml:"party_file"`
}
// SessionConfig contains per-session inputs and metadata.
@@ -188,7 +191,7 @@ type NormalizeConfig struct {
// TrimConfig configures trim-stage transcript boundary behavior.
type TrimConfig struct {
Enabled bool `yaml:"enabled"`
Enabled *bool `yaml:"enabled"`
OutputPath string `yaml:"output_path"`
Bounds TrimBoundsConfig `yaml:"bounds"`
Seriatim TrimSeriatimConfig `yaml:"seriatim"`
@@ -250,6 +253,26 @@ type ScriptoriumInputConfig struct {
Required bool `yaml:"required"`
}
// NotariusConfig configures structured artifact extraction by Notarius.
type NotariusConfig struct {
Enabled bool `yaml:"enabled"`
Binary string `yaml:"binary"`
ConfigPath string `yaml:"config_path"`
PipelineID string `yaml:"pipeline_id"`
Timeout string `yaml:"timeout"`
WorkingDirectory string `yaml:"working_directory"`
Outputs map[string]NotariusOutputConfig `yaml:"outputs"`
}
// NotariusOutputConfig declares one required extraction lane contract.
type NotariusOutputConfig struct {
LaneID string `yaml:"lane_id"`
MediaType string `yaml:"media_type"`
SchemaID string `yaml:"schema_id"`
SchemaVersion string `yaml:"schema_version"`
ModuleKey string `yaml:"module_key"`
}
// NotificationConfig configures notification backend settings.
type NotificationConfig struct {
Backend string `yaml:"backend"`
@@ -265,6 +288,8 @@ type SessionInputsConfig struct {
SpeakersFile string `yaml:"speakers_file"`
AutocorrectFile string `yaml:"autocorrect_file"`
GlossaryFile string `yaml:"glossary_file"`
PlayersFile string `yaml:"players_file"`
PartyFile string `yaml:"party_file"`
}
// SessionAudioS3Input configures S3 session-audio input discovery.
@@ -278,6 +303,8 @@ type ResolvedStableInputs struct {
SpeakersFile ResolvedInputFile
AutocorrectFile ResolvedInputFile
GlossaryFile ResolvedInputFile
PlayersFile ResolvedInputFile
PartyFile ResolvedInputFile
}
// ResolvedInputFile records one merged config path and its source config file.

View File

@@ -30,22 +30,31 @@ const (
DefaultSeriatimCoalesceGap = 3.0
DefaultSeriatimReport = true
DefaultAuditaBinary = "audita"
DefaultAuditaTimeout = "3h"
DefaultAuditaReport = true
DefaultAuditaBinary = "audita"
DefaultAuditaTimeout = "3h"
DefaultAuditaReport = true
DefaultNotariusBinary = "notarius"
DefaultNotariusTimeout = "3h"
DefaultScriptoriumBinary = "scriptorium"
DefaultScriptoriumTimeout = "10m"
DefaultScriptoriumArtifactOutputRoot = "artifacts"
DefaultScriptoriumBinary = "scriptorium"
DefaultScriptoriumTimeout = "10m"
DefaultScriptoriumArtifactOutputRoot = "artifacts"
DefaultScriptoriumStickySessionVarName = "session_id"
DefaultScriptoriumStickySessionVarPrefix = "narratio-session-"
DefaultTrimBoundsTimeout = "10m"
DefaultTrimSeriatimReport = false
DefaultRenderEnabled = true
DefaultRenderFormat = "markdown"
DefaultRenderTitle = ""
DefaultRenderTimestamps = true
DefaultRenderSegmentIDs = true
DefaultRenderMetadata = false
DefaultTrimEnabled = true
DefaultTrimOutputPath = artifactmodel.TranscriptPathFinalTrimmed
DefaultTrimBoundsPromptID = "dnd.session_bounds"
DefaultTrimBoundsTranscriptInputName = "transcript"
DefaultTrimBoundsOutputPath = "artifacts/session_bounds.json"
DefaultTrimBoundsTimeout = "10m"
DefaultTrimSeriatimReport = false
DefaultRenderEnabled = true
DefaultRenderFormat = "markdown"
DefaultRenderTitle = ""
DefaultRenderTimestamps = true
DefaultRenderSegmentIDs = true
DefaultRenderMetadata = false
DefaultNormalizeOutputPath = artifactmodel.TranscriptPathFinal
DefaultNormalizeOutputSchema = "seriatim-intermediate"

View File

@@ -19,6 +19,9 @@ func LoadPipeline(path string) (*PipelineConfig, error) {
return nil, fmt.Errorf("load pipeline config: %w", err)
}
applyPipelineDefaults(&cfg)
if err := resolveNotariusPaths(&cfg, path); err != nil {
return nil, fmt.Errorf("load pipeline config: %w", err)
}
return &cfg, nil
}
@@ -86,12 +89,17 @@ func LoadSessionBytesWithOptions(label string, data []byte, opts SessionLoadOpti
// LoadPublishLockStoreBytes loads a mutable session lock store with strict
// field checking and source validation.
func LoadPublishLockStoreBytes(label string, data []byte, scriptorium *ScriptoriumConfig) (*PublishLockStore, error) {
func LoadPublishLockStoreBytes(
label string,
data []byte,
scriptorium *ScriptoriumConfig,
notarius *NotariusConfig,
) (*PublishLockStore, error) {
var store PublishLockStore
if err := decodeStrictYAMLFromReader("publish lock store", label, strings.NewReader(string(data)), &store); err != nil {
return nil, fmt.Errorf("load publish lock store: %w", err)
}
locks, err := ValidatePublishLockRules(store.Locks, scriptorium, "locks")
locks, err := ValidatePublishLockRules(store.Locks, scriptorium, notarius, "locks")
if err != nil {
return nil, fmt.Errorf("load publish lock store: %w", err)
}
@@ -226,11 +234,25 @@ func mergeCampaignSession(campaignCfg *CampaignConfig, sessionCfg *SessionConfig
campaignPath,
sessionPath,
),
PlayersFile: selectStableInput(
campaignCfg.Inputs.PlayersFile,
sessionCfg.Inputs.PlayersFile,
campaignPath,
sessionPath,
),
PartyFile: selectStableInput(
campaignCfg.Inputs.PartyFile,
sessionCfg.Inputs.PartyFile,
campaignPath,
sessionPath,
),
}
sessionCfg.Inputs.SpeakersFile = stable.SpeakersFile.Path
sessionCfg.Inputs.AutocorrectFile = stable.AutocorrectFile.Path
sessionCfg.Inputs.GlossaryFile = stable.GlossaryFile.Path
sessionCfg.Inputs.PlayersFile = stable.PlayersFile.Path
sessionCfg.Inputs.PartyFile = stable.PartyFile.Path
return stable, nil
}
@@ -336,9 +358,16 @@ func applyPipelineDefaults(cfg *PipelineConfig) {
cfg.Normalize = &NormalizeConfig{}
}
applyNormalizeDefaults(cfg.Normalize)
if cfg.Trim == nil {
cfg.Trim = &TrimConfig{}
}
applyTrimDefaults(cfg.Trim)
if trimEnabled(cfg.Trim) && cfg.Scriptorium == nil {
cfg.Scriptorium = &ScriptoriumConfig{}
}
applyRenderDefaults(&cfg.Render)
applyScriptoriumDefaults(cfg.Scriptorium)
applyNotariusDefaults(cfg.Notarius)
}
func applyCampaignsDefaults(cfg *CampaignsConfig) {
@@ -496,10 +525,76 @@ func applyScriptoriumDefaults(cfg *ScriptoriumConfig) {
}
}
func applyNotariusDefaults(cfg *NotariusConfig) {
if cfg == nil {
return
}
if cfg.Binary == "" {
cfg.Binary = DefaultNotariusBinary
}
if cfg.Timeout == "" {
cfg.Timeout = DefaultNotariusTimeout
}
}
func resolveNotariusPaths(cfg *PipelineConfig, pipelinePath string) error {
if cfg == nil || cfg.Notarius == nil || !cfg.Notarius.Enabled {
return nil
}
pipelineAbs, err := filepath.Abs(pipelinePath)
if err != nil {
return fmt.Errorf("resolve pipeline config path %q: %w", pipelinePath, err)
}
baseDir := filepath.Dir(pipelineAbs)
resolve := func(value string) (string, error) {
trimmed := strings.TrimSpace(value)
if trimmed == "" {
return "", nil
}
if !filepath.IsAbs(trimmed) {
trimmed = filepath.Join(baseDir, trimmed)
}
return filepath.Abs(trimmed)
}
resolvedConfigPath, err := resolve(cfg.Notarius.ConfigPath)
if err != nil {
return fmt.Errorf("resolve pipeline.notarius.config_path: %w", err)
}
cfg.Notarius.ConfigPath = resolvedConfigPath
if strings.TrimSpace(cfg.Notarius.WorkingDirectory) == "" {
if resolvedConfigPath != "" {
cfg.Notarius.WorkingDirectory = filepath.Dir(resolvedConfigPath)
}
return nil
}
workingDirectory, err := resolve(cfg.Notarius.WorkingDirectory)
if err != nil {
return fmt.Errorf("resolve pipeline.notarius.working_directory: %w", err)
}
cfg.Notarius.WorkingDirectory = workingDirectory
return nil
}
func applyTrimDefaults(cfg *TrimConfig) {
if cfg == nil {
return
}
if cfg.Enabled == nil {
cfg.Enabled = boolPtr(DefaultTrimEnabled)
}
if strings.TrimSpace(cfg.OutputPath) == "" {
cfg.OutputPath = DefaultTrimOutputPath
}
if strings.TrimSpace(cfg.Bounds.PromptID) == "" {
cfg.Bounds.PromptID = DefaultTrimBoundsPromptID
}
if strings.TrimSpace(cfg.Bounds.TranscriptInputName) == "" {
cfg.Bounds.TranscriptInputName = DefaultTrimBoundsTranscriptInputName
}
if strings.TrimSpace(cfg.Bounds.OutputPath) == "" {
cfg.Bounds.OutputPath = DefaultTrimBoundsOutputPath
}
if cfg.Bounds.Timeout == "" {
cfg.Bounds.Timeout = DefaultTrimBoundsTimeout
}
@@ -508,6 +603,10 @@ func applyTrimDefaults(cfg *TrimConfig) {
}
}
func trimEnabled(cfg *TrimConfig) bool {
return cfg != nil && cfg.Enabled != nil && *cfg.Enabled
}
func applyRenderDefaults(cfg **RenderConfig) {
if cfg == nil {
return

View File

@@ -3,6 +3,7 @@ package config
import (
"os"
"path/filepath"
"reflect"
"strings"
"testing"
)
@@ -36,6 +37,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
checkDefault: true,
wantRoot: "/tmp/narratio",
@@ -55,6 +58,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
checkDefault: true,
wantRoot: "/tmp/narratio",
@@ -72,6 +77,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
checkDefault: true,
wantRoot: DefaultWorkspaceRoot,
@@ -88,6 +95,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantLoadErr: "pipeline file",
},
@@ -106,6 +115,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantLoadErr: "strict decode failed",
},
@@ -123,6 +134,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantLoadErr: "strict decode failed",
},
@@ -143,6 +156,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantLoadErr: "strict decode failed",
},
@@ -163,6 +178,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.secrets.env_dir must be non-empty when pipeline.secrets is configured",
},
@@ -177,6 +194,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
unknown_field: true
`,
wantLoadErr: "session file",
@@ -196,6 +215,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "session config \"session.yml\" invalid: session.session_id is required",
},
@@ -215,6 +236,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
},
{
@@ -233,6 +256,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "session config \"session.yml\" invalid: session.previous_session_id must not equal session.session_id",
},
@@ -250,6 +275,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.whisperx.transcribe_url is required",
},
@@ -268,6 +295,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.whisperx.transcribe_url must be a valid URL",
},
@@ -287,6 +316,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.whisperx.timeout must be a valid duration",
},
@@ -306,6 +337,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.whisperx.retry_delay must be a valid duration",
},
@@ -325,6 +358,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.whisperx.retries must be >= 0",
},
@@ -344,6 +379,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.whisperx.concurrency must be > 0",
},
@@ -363,6 +400,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantLoadErr: "strict decode failed",
},
@@ -383,6 +422,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantLoadErr: "strict decode failed",
},
@@ -401,6 +442,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
},
{
@@ -419,6 +462,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.seriatim.timeout must be a valid duration",
},
@@ -438,6 +483,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.seriatim.output_schema must be one of: seriatim-minimal, seriatim-intermediate, seriatim-full",
},
@@ -457,6 +504,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.seriatim.coalesce_gap must be >= 0",
},
@@ -477,6 +526,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.seriatim.env.overlap_word_run_gap must be > 0 when provided",
},
@@ -498,6 +549,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantLoadErr: "strict decode failed",
},
@@ -518,6 +571,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
},
{
@@ -538,6 +593,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.timeout must be a valid duration",
},
@@ -559,6 +616,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
},
{
@@ -581,6 +640,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.modules[1] must be non-empty",
},
@@ -604,6 +665,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.modules[1] must be one of: glossary, homophones, spoken_word, grammar",
},
@@ -625,6 +688,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.base_url must be a valid URL",
},
@@ -646,6 +711,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantLoadErr: "strict decode failed",
},
@@ -667,6 +734,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.total_llm_concurrency must be > 0",
},
@@ -688,6 +757,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.proposal_llm_concurrency must be > 0",
},
@@ -709,6 +780,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.validation_llm_concurrency must be > 0",
},
@@ -730,6 +803,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.output_schema must be one of: bare-segments, audita-v1",
},
@@ -751,6 +826,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`,
wantValidate: "pipeline config \"pipeline.yml\" invalid: pipeline.audita.work_dir_retention must be one of: always, auto, never",
},
@@ -916,7 +993,16 @@ func TestValidateMissingAudioSource(t *testing.T) {
Report: boolPtr(true),
},
},
Campaign: &CampaignConfig{CampaignID: "sample-campaign"},
Campaign: &CampaignConfig{
CampaignID: "sample-campaign",
Inputs: CampaignInputsConfig{
SpeakersFile: "speakers.yml",
AutocorrectFile: "autocorrect.yml",
GlossaryFile: "glossary.yml",
PlayersFile: "players.yml",
PartyFile: "party.yml",
},
},
Session: &SessionConfig{
SessionID: "2026-05-03",
Campaign: "sample-campaign",
@@ -924,6 +1010,8 @@ func TestValidateMissingAudioSource(t *testing.T) {
SpeakersFile: "speakers.yml",
AutocorrectFile: "autocorrect.yml",
GlossaryFile: "glossary.yml",
PlayersFile: "players.yml",
PartyFile: "party.yml",
},
},
}
@@ -962,6 +1050,11 @@ func TestExamplesLoadAndValidate(t *testing.T) {
pipelineFile: "pipeline.full.annotated.yml",
sessionFile: "session.local-audio.yml",
},
{
name: "extraction subset pipeline with local audio session",
pipelineFile: "pipeline.extraction-subset.yml",
sessionFile: "session.local-audio.yml",
},
}
for _, tt := range tests {
@@ -981,6 +1074,47 @@ func TestExamplesLoadAndValidate(t *testing.T) {
}
}
func TestMaintainedExtractionExamplesPreservePublishedContracts(t *testing.T) {
examplesDir := filepath.Join("..", "..", "examples")
full, err := LoadPipeline(filepath.Join(examplesDir, "pipeline.full.annotated.yml"))
if err != nil {
t.Fatalf("load full example error = %v", err)
}
want := map[string]NotariusOutputConfig{
"item_registry": {LaneID: "item-registry", MediaType: "application/json", SchemaID: "notarius.dnd.item_registry", SchemaVersion: "v1", ModuleKey: "dnd/item-registry"},
"npc_registry": {LaneID: "npc-registry", MediaType: "application/json", SchemaID: "notarius.dnd.npc_registry", SchemaVersion: "v1", ModuleKey: "dnd/npc-registry"},
"location_registry": {LaneID: "location-registry", MediaType: "application/json", SchemaID: "notarius.dnd.location_registry", SchemaVersion: "v1", ModuleKey: "dnd/location-registry"},
"scene_descriptions": {LaneID: "scene-descriptions", MediaType: "application/json", SchemaID: "notarius.dnd.scene_descriptions", SchemaVersion: "v1", ModuleKey: "dnd/scene-descriptions"},
"item_occurrences": {LaneID: "item-occurrences", MediaType: "application/json", SchemaID: "notarius.dnd.item_occurrences", SchemaVersion: "v1", ModuleKey: "dnd/item-occurrences"},
"spells": {LaneID: "spells", MediaType: "application/json", SchemaID: "notarius.dnd.spells", SchemaVersion: "v1", ModuleKey: "dnd/spells"},
"combat_turns": {LaneID: "combat-turns", MediaType: "application/json", SchemaID: "notarius.dnd.combat_turns", SchemaVersion: "v1", ModuleKey: "dnd/combat-turns"},
"npc_occurrences": {LaneID: "npc-occurrences", MediaType: "application/json", SchemaID: "notarius.dnd.npc_occurrences", SchemaVersion: "v1", ModuleKey: "dnd/npc-occurrences"},
"location_occurrences": {LaneID: "location-occurrences", MediaType: "application/json", SchemaID: "notarius.dnd.location_occurrences", SchemaVersion: "v1", ModuleKey: "dnd/location-occurrences"},
"enemy_events": {LaneID: "enemy-events", MediaType: "application/json", SchemaID: "notarius.dnd.enemy_events", SchemaVersion: "v1", ModuleKey: "dnd/enemy-events"},
}
if full.Notarius == nil || !reflect.DeepEqual(full.Notarius.Outputs, want) {
t.Fatalf("full example outputs = %#v, want %#v", full.Notarius, want)
}
subset, err := LoadPipeline(filepath.Join(examplesDir, "pipeline.extraction-subset.yml"))
if err != nil {
t.Fatalf("load subset example error = %v", err)
}
brief := subset.Scriptorium.Artifacts["session_brief"]
wantSources := map[string]string{
"npcs": "narratio.extraction.npc_registry",
"locations": "narratio.extraction.location_registry",
"scenes": "narratio.extraction.scene_descriptions",
}
gotSources := make(map[string]string, len(brief.Inputs))
for name, input := range brief.Inputs {
gotSources[name] = input.Source
}
if !reflect.DeepEqual(gotSources, wantSources) {
t.Fatalf("subset example sources = %#v, want %#v", gotSources, wantSources)
}
}
func writeConfigFiles(t *testing.T, pipelineYAML, sessionYAML string) (string, string) {
t.Helper()
if !strings.Contains(pipelineYAML, "\naudita:") && !strings.HasPrefix(pipelineYAML, "audita:") {
@@ -1009,6 +1143,8 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`
if err := os.WriteFile(campaignPath, []byte(campaignYAML), 0o644); err != nil {
t.Fatalf("write campaign.yml: %v", err)

View File

@@ -0,0 +1,283 @@
package config
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestNotariusOmittedAndDisabledBehavior(t *testing.T) {
dir := t.TempDir()
omittedPath := filepath.Join(dir, "omitted.yml")
if err := os.WriteFile(omittedPath, []byte(testPipelineBaseYAML), 0o644); err != nil {
t.Fatalf("write omitted pipeline: %v", err)
}
omitted, err := LoadPipeline(omittedPath)
if err != nil {
t.Fatalf("LoadPipeline(omitted) error = %v", err)
}
if omitted.Notarius != nil {
t.Fatalf("Notarius = %#v, want nil when omitted", omitted.Notarius)
}
disabledPath := filepath.Join(dir, "disabled.yml")
disabledYAML := testPipelineBaseYAML + `
notarius:
enabled: false
config_path: relative/notarius.yml
`
if err := os.WriteFile(disabledPath, []byte(disabledYAML), 0o644); err != nil {
t.Fatalf("write disabled pipeline: %v", err)
}
disabled, err := LoadPipeline(disabledPath)
if err != nil {
t.Fatalf("LoadPipeline(disabled) error = %v", err)
}
if disabled.Notarius == nil {
t.Fatal("Notarius = nil, want configured disabled section")
}
if disabled.Notarius.Enabled {
t.Fatal("Notarius.Enabled = true, want false")
}
if disabled.Notarius.Binary != DefaultNotariusBinary || disabled.Notarius.Timeout != DefaultNotariusTimeout {
t.Fatalf("disabled defaults = %#v", disabled.Notarius)
}
if disabled.Notarius.ConfigPath != "relative/notarius.yml" || disabled.Notarius.WorkingDirectory != "" {
t.Fatalf("disabled paths were resolved unexpectedly: %#v", disabled.Notarius)
}
}
func TestNotariusEnabledDefaultsAndPathResolution(t *testing.T) {
dir := t.TempDir()
pipelinePath := filepath.Join(dir, "deployment", "pipeline.yml")
if err := os.MkdirAll(filepath.Dir(pipelinePath), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
pipelineYAML := testPipelineBaseYAML + `
notarius:
enabled: true
config_path: notarius/config.yml
pipeline_id: dnd-session
outputs:
npc_registry:
lane_id: npc-registry
media_type: application/json
schema_id: notarius.dnd.npc_registry
schema_version: v1
`
if err := os.WriteFile(pipelinePath, []byte(pipelineYAML), 0o644); err != nil {
t.Fatalf("write pipeline: %v", err)
}
cfg, err := LoadPipeline(pipelinePath)
if err != nil {
t.Fatalf("LoadPipeline() error = %v", err)
}
wantConfigPath := filepath.Join(filepath.Dir(pipelinePath), "notarius", "config.yml")
if cfg.Notarius.Binary != "notarius" || cfg.Notarius.Timeout != "3h" {
t.Fatalf("defaults = %#v", cfg.Notarius)
}
if cfg.Notarius.ConfigPath != wantConfigPath {
t.Fatalf("config_path = %q, want %q", cfg.Notarius.ConfigPath, wantConfigPath)
}
if cfg.Notarius.WorkingDirectory != filepath.Dir(wantConfigPath) {
t.Fatalf("working_directory = %q, want %q", cfg.Notarius.WorkingDirectory, filepath.Dir(wantConfigPath))
}
explicitPath := filepath.Join(dir, "explicit.yml")
explicitYAML := strings.Replace(pipelineYAML, " pipeline_id: dnd-session\n", " pipeline_id: dnd-session\n working_directory: runtime\n", 1)
if err := os.WriteFile(explicitPath, []byte(explicitYAML), 0o644); err != nil {
t.Fatalf("write explicit pipeline: %v", err)
}
explicit, err := LoadPipeline(explicitPath)
if err != nil {
t.Fatalf("LoadPipeline(explicit working directory) error = %v", err)
}
if explicit.Notarius.WorkingDirectory != filepath.Join(dir, "runtime") {
t.Fatalf("explicit working_directory = %q, want %q", explicit.Notarius.WorkingDirectory, filepath.Join(dir, "runtime"))
}
}
func TestNotariusStrictYAML(t *testing.T) {
tests := []struct {
name string
yaml string
}{
{name: "unknown section field", yaml: "notarius:\n unknown: true\n"},
{name: "unknown output field", yaml: "notarius:\n outputs:\n npc_registry:\n lane_id: npc-registry\n unknown: true\n"},
{name: "unsupported session id", yaml: "notarius:\n session_id: forbidden\n"},
{name: "unsupported model", yaml: "notarius:\n model: forbidden\n"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "pipeline.yml")
if err := os.WriteFile(path, []byte(testPipelineBaseYAML+"\n"+tt.yaml), 0o644); err != nil {
t.Fatalf("write pipeline: %v", err)
}
if _, err := LoadPipeline(path); err == nil || !strings.Contains(err.Error(), "strict decode failed") {
t.Fatalf("LoadPipeline() error = %v, want strict decode failure", err)
}
})
}
}
func TestNotariusEnabledValidation(t *testing.T) {
valid := `notarius:
enabled: true
config_path: ./notarius.yml
pipeline_id: dnd-session
timeout: 45m
outputs:
npc_registry:
lane_id: npc-registry
media_type: application/json
schema_id: notarius.dnd.npc_registry
schema_version: v1
module_key: dnd/npc-registry
`
tests := []struct {
name string
section string
wantErr string
}{
{name: "valid", section: valid},
{name: "blank binary", section: strings.Replace(valid, " enabled: true\n", " enabled: true\n binary: \" \"\n", 1), wantErr: "pipeline.notarius.binary is required"},
{name: "missing config path", section: strings.Replace(valid, " config_path: ./notarius.yml\n", "", 1), wantErr: "pipeline.notarius.config_path is required"},
{name: "missing pipeline id", section: strings.Replace(valid, " pipeline_id: dnd-session\n", "", 1), wantErr: "pipeline.notarius.pipeline_id is required"},
{name: "missing outputs", section: strings.Split(valid, " outputs:\n")[0], wantErr: "pipeline.notarius.outputs must contain at least one output"},
{name: "zero timeout", section: strings.Replace(valid, " timeout: 45m", " timeout: 0s", 1), wantErr: "pipeline.notarius.timeout must be positive"},
{name: "negative timeout", section: strings.Replace(valid, " timeout: 45m", " timeout: -1s", 1), wantErr: "pipeline.notarius.timeout must be positive"},
{name: "invalid timeout", section: strings.Replace(valid, " timeout: 45m", " timeout: later", 1), wantErr: "pipeline.notarius.timeout must be a valid duration"},
{name: "invalid output key", section: strings.Replace(valid, " npc_registry:", " npc-registry:", 1), wantErr: "outputs keys must match"},
{name: "missing lane", section: strings.Replace(valid, " lane_id: npc-registry\n", "", 1), wantErr: "lane_id is required"},
{name: "missing media type", section: strings.Replace(valid, " media_type: application/json\n", "", 1), wantErr: "media_type is required"},
{name: "missing schema id", section: strings.Replace(valid, " schema_id: notarius.dnd.npc_registry\n", "", 1), wantErr: "schema_id is required"},
{name: "missing schema version", section: strings.Replace(valid, " schema_version: v1\n", "", 1), wantErr: "schema_version is required"},
{
name: "normalized key collision",
section: strings.Replace(valid, " module_key: dnd/npc-registry\n", ` module_key: dnd/npc-registry
" npc_registry ":
lane_id: npc-registry-two
media_type: application/json
schema_id: two
schema_version: v1
`, 1),
wantErr: "normalize to duplicate source",
},
{
name: "duplicate normalized lane",
section: strings.Replace(valid, " module_key: dnd/npc-registry\n", ` module_key: dnd/npc-registry
spells:
lane_id: " npc-registry "
media_type: application/json
schema_id: two
schema_version: v1
`, 1),
wantErr: "duplicates pipeline.notarius.outputs.npc_registry.lane_id",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
pipelinePath, sessionPath := writeConfigFiles(t, testPipelineBaseYAML+"\n"+tt.section, testSessionBaseYAML)
cfg, err := Load(pipelinePath, sessionPath)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
err = Validate(cfg)
if tt.wantErr == "" {
if err != nil {
t.Fatalf("Validate() error = %v", err)
}
output := cfg.Pipeline.Notarius.Outputs["npc_registry"]
if output.LaneID != "npc-registry" || output.ModuleKey != "dnd/npc-registry" {
t.Fatalf("normalized output = %#v", output)
}
return
}
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("Validate() error = %v, want containing %q", err, tt.wantErr)
}
})
}
}
func TestExtractionReferencesRequireDeclaredOutput(t *testing.T) {
declared := `notarius:
enabled: false
outputs:
npc_registry: {}
`
tests := []struct {
name string
body string
wantErr string
}{
{
name: "scriptorium declared extraction",
body: declared + `scriptorium:
artifacts:
recap:
inputs:
npcs:
source: narratio.extraction.npc_registry
`,
},
{
name: "scriptorium unknown extraction",
body: declared + `scriptorium:
artifacts:
recap:
inputs:
npcs:
source: narratio.extraction.unknown
`,
wantErr: `references unknown extraction output "unknown"`,
},
{
name: "publish declared extraction",
body: declared + `publish:
outputs:
- source: narratio.extraction.npc_registry
dest: artifacts/npc-registry.json
`,
},
{
name: "publish unknown extraction",
body: declared + `publish:
outputs:
- source: narratio.extraction.unknown
dest: artifacts/unknown.json
`,
wantErr: `extraction output "unknown" is not defined`,
},
{
name: "publish lock declared extraction",
body: declared + `publish:
locks:
- source: narratio.extraction.npc_registry
`,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
pipelinePath, sessionPath := writeConfigFiles(t, testPipelineBaseYAML+"\n"+tt.body, testSessionBaseYAML)
cfg, err := Load(pipelinePath, sessionPath)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
err = Validate(cfg)
if tt.wantErr == "" {
if err != nil {
t.Fatalf("Validate() error = %v", err)
}
return
}
if err == nil || !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("Validate() error = %v, want containing %q", err, tt.wantErr)
}
})
}
}

View File

@@ -215,6 +215,27 @@ func TestScriptoriumLoadAndValidate(t *testing.T) {
transcript_markdown:
source: narratio.transcript.final_markdown
required: true
`,
},
{
name: "prepared stable input sources are accepted",
scriptoriumYAML: `scriptorium:
binary: scriptorium
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
inputs:
players:
source: narratio.input.players
required: true
party:
source: narratio.input.party
required: true
glossary:
source: narratio.input.glossary
required: false
`,
},
{
@@ -575,4 +596,6 @@ inputs:
speakers_file: ./speakers.yml
autocorrect_file: ./autocorrect.yml
glossary_file: ./glossary.yml
players_file: ./players.yml
party_file: ./party.yml
`

Some files were not shown because too many files have changed in this diff Show More