Compare commits

...

10 Commits

29 changed files with 3081 additions and 4038 deletions

View File

@@ -1,34 +1,43 @@
# Notarius # Notarius
Notarius is a Go CLI for extracting structured artifacts from source material Notarius is a Go CLI for turning source material into structured artifacts with
with explicit, configurable pipeline modules. configured extraction pipelines. The implemented D&D workflow reads Seriatim
transcript JSON and can produce scene descriptions, item and currency events,
NPC identities, combat turns, NPC interactions, and spell casts.
The current implementation reads Seriatim transcript JSON, chunks the source ## Quickstart
units, extracts D&D spell-cast artifacts with a Scriptorium-backed LLM runtime,
and writes JSON output. Add `--debug` when a per-run inspection bundle is
needed.
```sh Provide an OpenRouter API key through the environment, then run the maintained
OPENROUTER_API_KEY=... \ minimal example:
~~~
OPENROUTER_API_KEY=your-api-key \
go run ./cmd/notarius run dnd-session \ go run ./cmd/notarius run dnd-session \
--config examples/dnd-minimal.config.yml \ --config examples/dnd-minimal.config.yml \
--input examples/seriatim-minimal-transcript.json --input examples/seriatim-minimal-transcript.json
``` ~~~
This invocation uses the maintained example configuration and input. See the The command publishes a JSON output bundle. Its command syntax and exit
configuration and operations references for profile selection, credentials, and behavior are documented in the [CLI reference](docs/cli.md); configuration,
run artifacts. credentials, and module selection are owned by the
[configuration reference](docs/config.md).
Useful references: For the complete ordered D&D workflow, use
[the complete configuration](examples/dnd-complete.config.yml) with
[its synthetic transcript](examples/dnd-complete-transcript.json). It
demonstrates all implemented D&D lanes and the supporting campaign references.
- [CLI reference](docs/cli.md) ## Documentation
- [Configuration reference](docs/config.md)
- [Operations](docs/operations.md) - [CLI reference](docs/cli.md) — commands, flags, output streams, and exits.
- [Seriatim input contract](docs/integrations/seriatim.md) - [Configuration reference](docs/config.md) — configuration files, profiles,
- [JSON output contract](docs/integrations/json-output.md) validation, and module selection.
- [D&D spell artifact contract](docs/integrations/dnd-spell-artifacts.md) - [Operations](docs/operations.md) — output, state, recovery, and debug
- [Developer guide](docs/development.md) handling.
- [Internal implementation docs](docs/internal/overview.md) - [Integration contracts](docs/integrations/) — Seriatim input and published
- [Minimal D&D configuration](examples/dnd-minimal.config.yml) artifact formats.
- [Complete D&D configuration](examples/dnd-complete.config.yml) - [Internal overview](docs/internal/overview.md) — implemented component map
- [Maintained example input](examples/seriatim-minimal-transcript.json) for maintainers.
- [Developer guide](docs/development.md) — contributor orientation and
validation guidance.
- [Future work](docs/roadmap/future.md) — unimplemented ideas and priorities.

View File

@@ -1,278 +1,146 @@
# CLI Reference # CLI Reference
This is the canonical reference for the implemented Notarius command-line This is the canonical reference for the implemented Notarius command-line
interface. interface. For the shortest successful run, see the [README](../README.md).
Configuration fields, discovery rules, and selectable module keys are defined
in [Configuration](config.md); runtime state and recovery procedures are
defined in [Operations](operations.md).
For the minimal end-to-end invocation, see the [README](../README.md). ## Command Summary
## Commands ~~~
```text
notarius help notarius help
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--chunk_cache auto|bypass|refresh] [--output-dir path] [--resume] [--recompute-step step-id] [--debug [--debug-dir path]] [--llm-profile id] [--session-id id] [--reference selector=path] [--without-reference selector] notarius run <pipeline-id> --input path/to/source.json [flags]
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b] notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list [--config path/to/config.yml] [--json] notarius pipelines list [--config path/to/config.yml] [--json]
``` ~~~
Running `notarius` with no arguments, `notarius help`, `notarius --help`, or Running Notarius without arguments, or with **help**, **--help**, or **-h**,
`notarius -h` prints usage and exits successfully. writes the command summary to standard output and exits with status 0.
## `run` ## run
`notarius run <pipeline-id>` executes a configured pipeline against one input ~~~
file. notarius run <pipeline-id> --input path/to/source.json [flags]
~~~
Flags: The **run** command executes the named pipeline for one input file. The
pipeline ID and **--input** are required.
- `--input path`: required source input file. | Flag | Meaning |
- `--config path`: config file path. If omitted, Notarius uses the discovery | --- | --- |
rules in [Configuration](config.md#discovery). | **--config path** | Use this configuration file. When omitted, configuration discovery applies; see [Configuration](config.md). |
- `--only lane-a,lane-b`: run only the named artifact lanes. Values are | **--input path** | Source input file to process. Required. |
comma-separated and must be non-empty. This retains its existing behavior for | **--output-dir path** | Override the configured output root for this run. |
implicit single-step pipelines; explicit multi-step pipelines reject it | **--chunk_cache auto\|bypass\|refresh** | Override chunk-plan cache handling for this run. |
rather than inferring dependency closure. | **--resume** | Reuse compatible recorded checkpoints when checkpoint recording is enabled. |
- `--resume`: request checkpoint reuse for this invocation. Checkpoint recording | **--recompute-step step-id** | With **--resume**, recompute the selected ordered step and its dependent lanes. It cannot be combined with **--only**. |
must be enabled in configuration. See | **--debug** | Retain a debug bundle for this run. |
[Operations](operations.md#checkpoint-cache) for prerequisites and reuse | **--debug-dir path** | Override the debug-bundle root. Requires **--debug**. |
behavior. | **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. |
- `--recompute-step step-id`: with `--resume` and checkpoint recording enabled, | **--llm-profile id** | Override effective LLM-capable module bindings with one configured profile. |
force the named ordered step and every transitive dependent lane to execute. | **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. |
Compatible required predecessors and unrelated lanes remain reusable. The | **--reference selector=path** | Add or replace a file reference binding. Repeatable. |
value may identify an explicit step or the implicit single-step ID `default`; | **--without-reference selector** | Remove a configured optional reference binding. Repeatable. |
it cannot be combined with `--only`.
- `--chunk_cache auto|bypass|refresh`: select chunk-plan reuse for this
invocation. `auto` reuses a valid plan by canonical source digest, `bypass`
performs no plan-cache I/O, and `refresh` regenerates and replaces a valid
plan only after chunk validation succeeds. See
[Configuration](config.md#state-surfaces) for the persistent setting, precedence,
and cache-root selection.
- `--output-dir path`: output root. Defaults to `./notarius-output`.
- `--debug`: allocate and retain one debug bundle for this invocation.
- `--debug-dir path`: debug-bundle root override. This flag requires `--debug`.
- `--llm-profile id`: override every effective LLM-capable pipeline module
binding with one Scriptorium profile ID. Validator-specific profiles are not
overridden.
- `--session-id id`: pass a stable prompt session identifier through LLM-backed
module calls.
- `--reference selector=path`: bind a reference path to a chunk, extractor,
merger, or normalizer reference slot. Repeatable.
- `--without-reference selector`: remove a configured optional reference binding.
Repeatable. It accepts the same selector forms as `--reference`, without
`=path`.
On success, the command prints the completed pipeline ID, normalized output and **--chunk_cache** accepts only **auto**, **bypass**, or **refresh**.
rejected output counts, and the output directory. A debug-enabled run also **--debug-dir**, **--output-dir**, **--session-id**, and
prints `debug=<bundle-path>`. If the run completes with warnings, the warning **--recompute-step** reject explicit empty values. **--recompute-step**
count is printed to stderr. requires **--resume**; checkpoint requirements and reuse behavior are
documented in [Operations](operations.md).
Reference flags are external file bindings resolved against selected chunk, ### Reference selectors
extractor, merger, and normalizer targets before the run starts. Generated
artifact bindings are configured in ordered steps and cannot be introduced by a
CLI path flag. Flat slot names are accepted only
when exactly one selected target declares that slot. For configured reference
bindings, precedence, path resolution, and validation, see
[Configuration](config.md#pipelines).
`--reference` binds or replaces one slot for one selected target. Selectors are: Use **--reference** only for a reference slot declared by the selected
configured target. The accepted selector forms are:
- `slot=path`: valid when exactly one selected target declares `slot`; | Form | Target |
- `chunk.slot=path`: target the chunker; | --- | --- |
- `merge.slot=path`: valid when exactly one selected merger declares `slot`; | slot=path | The unique selected target that declares slot. |
- `lane.slot=path`: valid when exactly one selected extractor, merger, or | chunk.slot=path | The chunker. |
normalizer in that lane declares `slot`; | merge.slot=path | The unique selected merger that declares slot. |
- `lane.extract.slot=path`: target a lane extractor; | lane.slot=path | The unique extractor, merger, or normalizer in lane that declares slot. |
- `lane.merge.slot=path`: target a lane merger; | lane.extract.slot=path | The extractor in lane. |
- `lane.normalize.slot=path`: target a lane normalizer. | lane.merge.slot=path | The merger in lane. |
| lane.normalize.slot=path | The normalizer in lane. |
Use `slot=path` when the selected targets declare the slot unambiguously: **--without-reference** uses the same selector forms without =path. Slot
names, requiredness, and configured bindings are part of the
[configuration contract](config.md).
```sh ### Run output
On success, standard output contains the completed pipeline ID, counts of
normalized and rejected outputs, and the output directory. A debug-enabled run
also prints its debug-bundle path to standard output. A successful run with
warnings reports the warning count to standard error. The published JSON
envelope is defined by the [JSON output contract](integrations/json-output.md).
Example:
~~~
OPENROUTER_API_KEY=your-api-key \
go run ./cmd/notarius run dnd-session \ go run ./cmd/notarius run dnd-session \
--config examples/dnd-minimal.config.yml \ --config examples/dnd-minimal.config.yml \
--input examples/seriatim-minimal-transcript.json \ --input examples/seriatim-minimal-transcript.json
--reference roster=./campaign-roster.txt ~~~
```
Use an explicit selector when multiple selected targets declare the same slot or ## config validate
when you want to target a specific target:
```sh ~~~
go run ./cmd/notarius run dnd-session \ notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
--config examples/dnd-minimal.config.yml \ ~~~
--input examples/seriatim-minimal-transcript.json \
--reference spells.extract.glossary=./campaign-glossary.txt
```
For the maintained ordered D&D workflow, use the explicit pipeline. The first This command loads and validates a configuration. With **--pipeline**, it also
step produces normalized NPC and scene-description artifacts; the second step resolves that pipeline against the production module catalog. **--only** selects
receives the NPC artifact in memory for spell extraction, combat extraction, and lanes during that resolution and requires **--pipeline**.
combat normalization, and receives the required scene-description artifact for
combat eligibility:
```sh Success is written to standard output as either config "<path>" is valid or
go run ./cmd/notarius run dnd-session \ config "<path>" is valid for pipeline "<pipeline-id>".
--config examples/dnd-complete.config.yml \
--input examples/seriatim-minimal-transcript.json \
--output-dir ./npc-grounded-output
```
The generated NPC content remains contextual grounding, not spell or combat
evidence. It is represented in manifests and debug summaries by bounded
identity and producer provenance, not by payload content or a filesystem path.
The scene-description artifact is control context: combat extraction calls its
LLM only for an exact `combat` scene match. See the
[D&D combat-turn artifact contract](integrations/dnd-combat-turn-artifacts.md)
for the resulting empty-output and warning behavior.
The same grammar can target chunk, merge, and normalize slots when the configured
modules declare them:
```sh
go run ./cmd/notarius run dnd-session \
--config path/to/config.yml \
--input examples/seriatim-minimal-transcript.json \
--reference chunk.scene_guide=./campaign-scenes.txt \
--reference spells.merge.merge_notes=./merge-notes.txt \
--reference spells.normalize.normalization_notes=./normalization-notes.txt
```
Use `--without-reference` to remove a configured optional binding for a run:
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-minimal.config.yml \
--input examples/seriatim-minimal-transcript.json \
--without-reference glossary
```
Use `--session-id` when an external orchestrator needs all prompt calls from one
run to share an identifier:
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-minimal.config.yml \
--input examples/seriatim-minimal-transcript.json \
--session-id campaign-17-session-04
```
When `cache.checkpoints.enabled` is `true`, runs record checkpoints whether or
not `--resume` is present. Add the resume flag to load and reuse compatible
recorded work; using it while checkpoint recording is disabled is an error:
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-minimal.config.yml \
--input examples/seriatim-minimal-transcript.json \
--resume
```
To selectively rerun one ordered step and its dependent lanes, use the step ID
from the configuration. The selected step and dependents are reported as
`forced_recompute`; reusable predecessors are reported as `reused`:
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-complete.config.yml \
--input examples/seriatim-minimal-transcript.json \
--resume --recompute-step extract-events
```
Checkpoint decisions use these categories: `reused`, `executed`,
`forced_recompute`, and `dependency_invalidated`. The reason code and bounded
detail identify the decision without exposing reference content, local paths,
or secrets. `--recompute-step` requires checkpoint recording and `--resume`;
unknown step IDs, empty values, and combinations with `--only` are rejected.
The operator meanings of checkpoint reason codes are maintained in
[Operations](operations.md#resume-and-selective-recompute).
Use `--debug` to retain the redacted summary and trace bundle for one run. The
bundle is allocated before pipeline resolution; once allocated, its path is
also printed to stderr if the command fails. Debug-write failures cause exit
code `1`.
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-minimal.config.yml \
--input examples/seriatim-minimal-transcript.json \
--debug --debug-dir ./notarius-debug
```
Use `refresh` when intentionally replacing the cached plan for the same source:
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-minimal.config.yml \
--input examples/seriatim-minimal-transcript.json \
--chunk_cache refresh
```
Use `bypass` for a one-off run that must not inspect or create plan-cache state:
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-minimal.config.yml \
--input examples/seriatim-minimal-transcript.json \
--chunk_cache bypass
```
`--diagnostics-dir` has been removed. For checkpoint behavior, durable output,
debug-bundle lifecycle, and failure inspection, see [Operations](operations.md).
## `config validate`
`notarius config validate` loads and validates configuration.
Flags:
- `--config path`: config file path. If omitted, Notarius uses the discovery
rules in [Configuration](config.md#discovery).
- `--pipeline pipeline-id`: additionally resolve one configured pipeline against
the production module catalog.
- `--only lane-a,lane-b`: validate resolution for selected artifact lanes. This
flag requires `--pipeline`.
Examples: Examples:
```sh ~~~
go run ./cmd/notarius config validate \
--config examples/dnd-minimal.config.yml
go run ./cmd/notarius config validate \ go run ./cmd/notarius config validate \
--config examples/dnd-minimal.config.yml \ --config examples/dnd-minimal.config.yml \
--pipeline dnd-session \ --pipeline dnd-session
--only spells
```
## `pipelines list` OPENROUTER_API_KEY=validation-placeholder \
go run ./cmd/notarius config validate \
--config examples/dnd-complete.config.yml \
--pipeline dnd-session
~~~
`notarius pipelines list` prints configured pipeline IDs in sorted order. The placeholder in the second command is sufficient only for offline
validation; it cannot run a provider-backed pipeline.
Flags: ## pipelines list
- `--config path`: config file path. If omitted, Notarius uses the discovery ~~~
rules in [Configuration](config.md#discovery). notarius pipelines list [--config path/to/config.yml] [--json]
- `--json`: print `{"pipelines":[...]}` instead of one ID per line. ~~~
Examples: This command lists configured pipeline IDs in sorted order. By default, it
writes one ID per line to standard output. **--json** writes an object shaped as
{"pipelines":[...]} instead.
```sh ~~~
go run ./cmd/notarius pipelines list \ go run ./cmd/notarius pipelines list \
--config examples/dnd-minimal.config.yml --config examples/dnd-minimal.config.yml
~~~
go run ./cmd/notarius pipelines list \ ## Output Streams And Exit Statuses
--config examples/dnd-minimal.config.yml \
--json
```
## Exit Codes Successful commands write their primary result to standard output. Warnings and
errors are written to standard error.
- `0`: command succeeded. | Status | Meaning |
- `1`: command syntax was valid, but loading config, resolving modules, running | --- | --- |
the pipeline, calling the provider, writing output, or writing a requested | 0 | The command completed successfully, including root help. |
debug bundle failed. | 1 | Command syntax was valid but configuration loading or validation, pipeline resolution or execution, provider use, output, or requested debug handling failed. |
- `2`: command syntax was invalid, a command was unknown, a required argument | 2 | The command or flag syntax was invalid, including unknown commands, missing required arguments, invalid flag values, or invalid flag combinations. |
was missing, or a flag value was malformed.
For YAML structure, defaults, Scriptorium profile sources, environment The root help spellings are the supported help path. Invoking **--help** on
overrides, and selectable module and validator keys, see **run**, **config validate**, or **pipelines list** is handled by the flag
[Configuration](config.md). parser as a usage error: it writes an error to standard error and exits with
status 2.

File diff suppressed because it is too large Load Diff

View File

@@ -17,11 +17,12 @@ implemented component map.
| Application shape, package boundaries, contracts, dependency direction, runtime guarantees, or safety properties | [Architecture](policy/architecture.md) and relevant [ADRs](adr/) | Architecture defines the intended system and its invariants; ADRs preserve significant decision rationale. | | Application shape, package boundaries, contracts, dependency direction, runtime guarantees, or safety properties | [Architecture](policy/architecture.md) and relevant [ADRs](adr/) | Architecture defines the intended system and its invariants; ADRs preserve significant decision rationale. |
| Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical homes, audiences, current-behavior rules, and maintenance requirements. | | Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical homes, audiences, current-behavior rules, and maintenance requirements. |
| Adding, changing, reviewing, or deleting tests | [Testing Policy](policy/testing.md) | It defines risk-based sufficiency, durable test boundaries, test-double guidance, and criteria for retaining tests. | | Adding, changing, reviewing, or deleting tests | [Testing Policy](policy/testing.md) | It defines risk-based sufficiency, durable test boundaries, test-double guidance, and criteria for retaining tests. |
| CLI composition or command behavior | [CLI Internals](internal/cli.md) and [CLI Reference](cli.md) | The internal guide owns composition and command flow; the reference owns public syntax. |
| Configuration loading, resolution, or user-visible configuration behavior | [Configuration Internals](internal/configuration.md) and [Configuration](config.md) | The internal guide owns loading and resolution mechanics; the reference owns the configuration contract. |
| Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. | | Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
| Production modules or validators | [Module Internals](internal/modules.md) | It documents implemented module contracts, capabilities, assets, and registration. | | Production modules or validators | [Module Internals](internal/modules.md), [D&D Module Internals](internal/dnd.md), and [D&D integration contracts](integrations/) | The generic guide owns extension mechanics, the D&D guide owns shared family conventions, and the contracts own durable output shapes. |
| LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and Scriptorium integration. | | LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and Scriptorium integration. |
| Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. | | Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. |
| CLI or user-visible configuration behavior | [CLI Reference](cli.md) and [Configuration](config.md) | These are the canonical user and operator references. |
| External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. | | External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. |
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. | | Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |

View File

@@ -1,83 +1,81 @@
# Accepted Chunk Map # Accepted Chunk Map
This document defines the durable accepted chunk-map artifact that the JSON This document defines the optional durable `chunk-map.json` artifact in a
output encoder can write as `chunk-map.json`. It describes the exact accepted, [published JSON bundle](json-output.md). It describes the accepted,
materialized chunks used by a run; it is not a lane artifact and is never an materialized chunk plan used by one run. It is not a lane payload and is never
input to later pipeline steps. Enable it with the JSON output option described an input to a later pipeline step.
in [Configuration](../config.md#json-output-options).
## Identity ## Contract Identity
- Artifact kind: `source/chunk-map` | Property | Value |
- Logical file: `chunk-map.json` | --- | --- |
- Schema ID: `notarius.source.chunk_map` | Artifact kind | `source/chunk-map` |
- Schema name: `notarius_source_chunk_map_v1` | Logical file | `chunk-map.json` |
- Schema version: `v1` | Media type | `application/json` |
- Media type: `application/json` | Schema ID | `notarius.source.chunk_map` |
| Schema name | `notarius_source_chunk_map_v1` |
| Schema version | `v1` |
The checked-in [JSON Schema](../../internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json) The optional `chunk_map` descriptor in `index.json` identifies this artifact.
defines the strict wire shape. `chunk-map.json` is listed by the optional Export is controlled by the JSON output binding described in
`chunk_map` descriptor in [the JSON output index](json-output.md#indexjson), [Configuration](../config.md#module-bindings-and-validators).
not by the lane-oriented `output_files` collection.
## Payload ## Wire Shape
The payload has these required fields: Every payload has these required fields:
- `source_id`: accepted source-document identity. | Field | Meaning |
- `source_digest`: canonical lower-case `sha256:` digest of that document. | --- | --- |
- `plan_digest`: canonical lower-case `sha256:` digest of the accepted logical | `source_id` | Accepted source-document identity. |
plan. | `source_digest` | Lower-case `sha256:` digest of that source document. |
- `requested_chunker`: chunk module selected by the current resolved pipeline. | `plan_digest` | Lower-case `sha256:` digest of the logical chunk plan. |
- `producer`: the original accepted-plan producer, with required | `requested_chunker` | Chunk module selected by the resolved pipeline. |
`input_module` and `chunk_module`; `llm_profile` is present only for an | `producer` | Original accepted-plan producer. `input_module` and `chunk_module` are required; `llm_profile` is optional. |
LLM-backed producer. | `plan_annotations` | Plan-level annotation namespace map; `{}` when none are present. |
- `plan_annotations`: accepted plan-level annotation namespace map. It is | `chunks` | Non-empty execution-order chunk collection. |
`{}` when no namespaces are present.
- `chunks`: non-empty execution-order collection of accepted chunks.
Each chunk has `id`, zero-based `index`, `source_ref`, positive `unit_count`, Each `chunks` entry contains non-empty `id`, zero-based `index`, `source_ref`,
and an explicit `annotations` namespace map. A source reference has the source positive `unit_count`, and an explicit `annotations` map. `source_ref` contains
ID and inclusive positive `start_unit_id` and `end_unit_id` endpoints. the same `source_id` as the top-level value plus positive inclusive
Annotation values are arbitrary valid JSON under non-empty namespaces. They `start_unit_id` and `end_unit_id` values. Endpoints identify source units; their
are preserved as canonical JSON without interpreting any module-specific numeric values do not by themselves establish source-document order.
namespace.
## Invariants Annotation namespaces are non-empty trimmed strings. Their values are arbitrary
valid JSON and are retained without interpreting a module-specific namespace.
The framework constructs this artifact only after materializing the selected ## Ordering And Validation
logical plan and accepting it through the configured chunk validator chain.
Construction proves the source and plan digests, source-document range order,
materialized chunk IDs and indexes, source references, unit membership and
counts, and plan/range annotations agree exactly. Chunk IDs are unique and
indexes are contiguous and agree with array order.
The codec rejects unknown fixed-object fields, malformed identities or `chunks` are in execution order. Their indexes are contiguous, start at zero,
digests, invalid annotation JSON, trailing JSON content, and any payload whose and equal their array positions; chunk IDs are unique. The emitted map is built
reconstructed logical plan does not reproduce `plan_digest`. It makes only after the selected plan has been accepted and materialized against the
defensive copies at serialization and decoding boundaries. source document, so its ranges, unit counts, annotations, and digests describe
that exact plan.
## Acceptance And Provenance The codec rejects malformed JSON, trailing content, unknown fixed-object
fields, invalid identities or digests, invalid annotations, duplicate chunk
IDs, non-contiguous indexes, and a `plan_digest` that does not match the
reconstructed logical plan. The checked-in
[schema](../../internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json)
defines the strict JSON shape.
The artifact is available only when the chunk plan was accepted. It remains ## Valid Example
available when a later extraction, merge, or normalization result is rejected;
it is absent when chunk validation rejects the candidate plan.
`requested_chunker` describes the current pipeline selection. `producer` The compact
describes who originally produced the accepted plan. On a cache hit these can [source chunk-map fixture](../../internal/framework/chunkmap/testdata/source_chunk_map.v1.json)
differ: the accepted ranges, annotations, digests, and stable materialized IDs is decoded by the production codec and demonstrates an accepted map with
are reused, while the producer remains the stored producer. Cache paths, annotations, producer identity, and ordered chunks.
actions, references, metadata, warnings, timestamps, and detailed provenance
remain in the run manifest rather than this payload.
## Data Handling ## Publication And Compatibility
The map contains structure, not source content. It excludes transcript bytes, The map is present only when a chunk plan was accepted and its export is
materialized units, source-unit metadata, chunk content, private model enabled. It remains publishable if a later lane is rejected, but is absent when
responses, rejected proposals, debug data, external-reference content, and chunk-plan validation rejects the plan. `requested_chunker` identifies the
filesystem paths. current pipeline selection, while `producer` identifies the component that
originally produced the accepted plan; they may differ when an accepted plan is
reused.
Annotations can nevertheless be source- or model-derived. Treat an enabled The map contains structure rather than source content: it excludes transcript
`chunk-map.json` with the same sensitivity and retention expectations as lane bytes, source-unit metadata, chunk text, private model output, reference
output. Physical placement, confined atomic writing, and permissions follow content, debug data, and filesystem paths. Treat the exported map with the
the ordinary [output operation](../operations.md#output). same care as other published output. Publication location and retention are
defined in [Operations](../operations.md#output-bundles).

View File

@@ -1,10 +1,10 @@
# D&D Combat-Turn Artifact Contract # D&D Combat-Turn Artifact
This document defines the durable artifact, serialization, extraction, This contract defines the durable combat-action occurrence list produced by
candidate-validation, normalization, and production lane boundaries for D&D `dnd/combat-turns`. It records source-grounded turns and actions; it is not a
combat turns. complete initiative tracker, combat summary, or state model.
## Artifact identity ## Identity and compatibility
| Property | Value | | Property | Value |
| --- | --- | | --- | --- |
@@ -14,167 +14,56 @@ combat turns.
| Schema version | `v1` | | Schema version | `v1` |
| Media type | `application/json` | | Media type | `application/json` |
The top-level JSON object contains the required `combat_turns` array, which `v1` is a strict JSON object with required `combat_turns`; the array may be
may be empty. Every object rejects unknown fields. empty. Turn and source-reference objects reject unknown fields. A future
incompatible shape requires a new schema version.
## JSON shape ## Wire shape
Each combat turn contains these required fields: Each combat turn has these required fields:
| Field | Shape | | Field | Contract |
| --- | --- | | --- | --- |
| `actor` | Non-empty string. | | `actor` | Non-empty acting character or creature name. |
| `turn_kind` | One of `turn`, `reaction`, `legendary_action`, `lair_action`, or `other`. | | `turn_kind` | `turn`, `reaction`, `legendary_action`, `lair_action`, or `other`. |
| `source_refs` | Required array with at least one source reference. | | `source_refs` | One or more transcript evidence ranges. |
Source references use the shared source-reference shape: Each source reference has exactly `source_id`, `start_unit_id`, and
`end_unit_id`. It identifies an inclusive current-transcript range; unit IDs
are positive and the start may not follow the end.
```json ```json
{ {
"source_id": "session-alpha", "combat_turns": [
"start_unit_id": 1, {
"end_unit_id": 2 "actor": "Mira Thorn",
"turn_kind": "turn",
"source_refs": [
{"source_id": "session-7", "start_unit_id": 31, "end_unit_id": 32}
]
}
]
} }
``` ```
`source_id` must be non-empty and both unit IDs must be positive integers. The ## Eligibility, evidence, and normalized form
codec does not resolve references against a source document or enforce source
range ordering; those checks belong to the later source-reference validation
boundary.
## Codec behavior The extractor requires an approved [scene-description artifact](dnd-scene-description-artifacts.md).
It emits combat turns only for a chunk with an exact matching scene classified
`combat`; an exact non-combat scene produces an accepted empty list. The scene
record controls eligibility only: its title, summary, and reference do not
become turn evidence. No exact matching scene also produces an empty list and
the `scene_classification_unavailable` warning.
The codec exposes two representations of the same typed artifact: An optional normalized [NPC artifact](dnd-npc-artifacts.md) can ground an
actor name. Its registry references are provenance, never combat evidence.
Normalization trims and, where possible, canonicalizes actor names; orders and
deduplicates exact source references; orders valid-evidence turns by source
chronology; and collapses only duplicates with the same actor identity, turn
kind, and complete valid evidence. It does not infer turns, initiative, or
actions from registry or scene data.
- Candidate encode/decode preserves invalid actor and turn-kind values, The [NPC-interaction artifact](dnd-npc-interaction-artifacts.md) records
collection presence, and source references so later validators can report broader NPC occurrences. The [JSON output contract](json-output.md) defines
them. Candidate decoding still requires valid JSON, one JSON value, known publication, and [D&D module internals](../internal/dnd.md) describes routing
fields, and compatible JSON types. and validation mechanics.
- Approved encode/decode enforces the structural rules in this contract.
The codec owns the durable JSON Schema, whose object layers all set
`additionalProperties` to `false`. Codec metadata contains only
`combat_turn_count`.
The maintained compact fixture is
`internal/modules/dnd/codec/combatturns/testdata/dnd_combat_turns.v1.json`.
## Extraction boundary
The standalone extractor uses these identities:
| Property | Value |
| --- | --- |
| Extractor key | `dnd/combat-turns` |
| Capability | `dnd.combat_turns` |
| Prompt ID | `dnd.combat_turns` |
| Prompt version | `v1` |
| Private response-schema key | `dnd_combat_turns_llm` |
| Private response-schema ID | `notarius.dnd.combat_turns.llm` |
| Default profile | `gemini-2-flash` |
It requires `chunks` and `source.transcript`, accepts no options, and requires
one `scene_descriptions` reference. That reference must be exactly one approved
`dnd/scene-description-list` artifact with media type `application/json` and a
maximum size of 1 MiB (1048576 bytes). It may be an external file validated
during preparation or a canonical generated artifact supplied at an ordered
step handoff. An unbound slot is a configuration error.
The scene artifact controls eligibility, not evidence. The extractor calls the
LLM only when exactly one record has the current chunk's ID, source ID, start
unit ID, and end unit ID, and that record has `kind: combat`. An exact
`narrative`, `recap`, or `meta` record returns an accepted empty
`combat_turns` array without an LLM call, warning, or retry attempt. A missing
or mismatched exact record returns the same accepted empty result without an
LLM call or retry and emits one content-safe
`scene_classification_unavailable` warning. The scene artifact, its title and
summary, and its source references are never copied into combat turns.
For an eligible combat chunk, the prompt receives the chunk-scoped transcript
plus the existing `players`, `party`, and `glossary` inputs, and optionally the
deprecated `roster` reference through the shared party mapping. The optional
`npcs` reference is an approved normalized NPC artifact used only for identity
grounding; it never supplies combat evidence. `scene_descriptions` is never a
combat prompt input.
The private response envelope has the same turn fields and JSON types as the
durable shape except that source references contain only `start_unit_id`
and `end_unit_id`. It enforces required field presence, types, and
unknown-field rejection, while deterministic validators own enum membership,
non-empty values and collections, and positive-number requirements. The
extractor assigns the current source ID, removes exact duplicate ranges, and
stable-sorts turns by the earliest valid source-document position. Numeric unit
IDs are identifiers; source-document slice position determines chronology.
Semantically malformed candidate fields remain in the typed result for the
configured validation and retry boundary.
## Deterministic candidate validation
The standalone validator keys are:
| Validator | Responsibility |
| --- | --- |
| `extract/dnd/combat-turns/shape` | Required list, actor, turn kind, and source references, plus supported turn-kind values. |
| `extract/dnd/combat-turns/source_refs` | Source identity, source-unit existence, and range order through the source document. |
| `extract/dnd/combat-turns/source_relatedness` | At most one advisory warning per turn when the actor is not related to cited transcript text. |
Source-reference and relatedness validators defer malformed shape to the shape
validator. Relatedness also defers when any cited source range is invalid. It
combines overlapping cited ranges once in document order and compares actors
with the shared Unicode-aware NPC identity policy.
The production D&D registrar exposes the extractor and these validators. Its
default extraction chain preserves this order: JSON syntax, combat shape,
source references, private response schema, then source relatedness.
## Normalization boundary
The standalone normalizer uses key `dnd/combat-turns`, requires `merged`,
provides `normalized`, accepts no options, and accepts only the optional
structured `npcs` reference. Campaign references are LLM extraction context and
are not normalizer inputs. For an external file, the NPC registry is resolved
during preparation; for a generated binding, it is resolved at the operation-
time handoff. Runtime normalization uses that immutable prepared or handed-off
view.
Normalization policy is `dnd.combat_turns.normalize.v1`. It display-normalizes
the actor, canonicalizes exact registry actor matches, orders and deduplicates
exact source references, stable-sorts records by earliest valid source-document
position, and collapses only records with the same actor identity, turn kind,
and complete valid evidence set. The first normalized record is retained.
Invalid evidence is never eligible for duplicate collapse. Every mutation and
collapse emits a bounded warning using the merged input index in its scope.
The normalizer reports `normalization_policy` and `identity_policy` metadata
and fingerprints. An external registry may additionally contribute
`npc_registry_digest` and `npc_count`; generated registry identity is retained
in framework handoff provenance and dependency fingerprints. The
normalized-invariants validator is
`normalize/dnd/combat-turns/invariants`; it defers shape and source-reference
failures, then checks actor display normalization, canonical evidence ordering,
chronology, and duplicate identity. It rejects
with `invalid_combat_turn_normalization` under policy
`dnd.combat_turns.validator.normalized.v1`.
The production D&D registrar exposes the normalizer and normalized-invariants
validator. Its default normalization chain is JSON syntax, combat shape,
normalized invariants, source references, durable schema, then source
relatedness. The lane uses the framework's typed append-order merger and has no
merge validator chain.
## Production manifest and references
The selectable lane uses extractor and normalizer key `dnd/combat-turns`,
`appendorder` for the typed merger, and the durable codec above. Bound external
references contribute raw-file provenance to the run manifest. Generated
bindings contribute artifact kind, schema identity, media type, canonical
digest, size, and bounded producer provenance. This includes the generated
scene-description artifact for the combat extractor. Consumer metadata and
checkpoint fingerprints contain no reference names, content, paths, source
ranges, scene titles, or scene summaries. For an external scene artifact,
component metadata records the gate policy plus a semantic eligibility digest
and record count; generated scene identity remains in framework handoff
provenance and dependencies. The eligibility digest changes with scene ID,
exact source range, or kind, but not with title or summary. The normalized lane
is emitted as `lanes/<lane-id>.json` by the JSON output module, and warnings
and rejection summaries remain in their shared companion files.

View File

@@ -1,133 +1,78 @@
# D&D Item-Event Artifact Contract # D&D Item-Event Artifact
This document defines the durable D&D item-event artifact and its production This contract defines the durable item and currency occurrence list produced by
boundaries. It records source-grounded discoveries and possession changes; it `dnd/item-events`. It records source-grounded discoveries and possession
does not maintain an inventory or ledger. changes; it does not maintain an inventory, balance, or ledger.
## Artifact Identity ## Identity and compatibility
| Property | Value | | Property | Value |
| --- | --- | | --- | --- |
| Extractor key | `dnd/item-events` |
| Extract capability | `dnd.item_events` |
| Artifact kind | `dnd/item-event-list` | | Artifact kind | `dnd/item-event-list` |
| Schema ID | `notarius.dnd.item_events` | | Schema ID | `notarius.dnd.item_events` |
| Schema name | `notarius_dnd_item_events_v1` | | Schema name | `notarius_dnd_item_events_v1` |
| Schema version | `v1` | | Schema version | `v1` |
| Media type | `application/json` | | Media type | `application/json` |
| Normalizer key | `dnd/item-events` |
The payload is one strict JSON object containing a required `events` array, `v1` is a strict JSON object with required `events`; the array may be empty.
which may be empty. Objects reject unknown fields. Event and source-reference objects reject unknown fields. A future incompatible
shape requires a new schema version.
## Event Shape And Categories ## Wire shape
Every event has `name`, `kind`, and a non-empty `source_refs` array. `quantity` Every event has required `name`, `kind`, and `source_refs`. `quantity`, `from`,
is optional and, when present, is a positive integer. `from` and `to` are and `to` are optional where the event kind permits them.
optional display values whose presence depends on `kind`.
| Field | Rule | | Field | Contract |
| --- | --- | | --- | --- |
| `name` | Non-empty, transcript-supported display value. | | `name` | Non-empty item or currency display name. |
| `kind` | One of the categories below. | | `kind` | `discovered`, `acquired`, `lost`, `consumed`, or `transferred`. |
| `quantity` | Optional positive integer; omitted when the source does not establish a count. | | `quantity` | Optional positive integer; omit it when no count is established. |
| `from` | Holder that loses possession when the category permits it. | | `from` | Optional non-empty losing holder, when allowed by `kind`. |
| `to` | Holder that gains possession when the category permits it. | | `to` | Optional non-empty gaining holder, when allowed by `kind`. |
| `source_refs` | One or more current-source references. | | `source_refs` | One or more transcript evidence ranges. |
| Kind | Meaning and holder rule | Each source reference has exactly `source_id`, `start_unit_id`, and
| --- | --- | `end_unit_id`. It identifies an inclusive current-transcript range; unit IDs
| `discovered` | The party learns of or encounters an item without established possession; neither holder is present. | are positive and the start may not follow the end.
| `acquired` | A party member or `party` gains possession; `to` is required and `from` is absent. |
| `lost` | A party member or `party` ceases to possess an item without consuming it; `from` is required and `to` is absent. |
| `consumed` | Use depletes, expends, or destroys an item; `from` is required and `to` is absent. |
| `transferred` | Possession moves between distinct party members; both holders are required and neither may be `party`. |
`party` is the reserved display holder for collective party possession when an
individual holder is not established. Its comparison is case- and
Unicode-insensitive. Transfers require distinct normalized holder values.
Giving an item to an NPC, spending currency, selling an item, or another move
outside party possession is `lost`, not `transferred` or `consumed`. Monetary
spending, purchases, and payments are always `lost`. Currency is `consumed` only
when the source explicitly describes its physical destruction or expenditure as
a non-payment component. Ordinary non-depleting use is not an event.
Currency is represented as an ordinary event name plus an explicit quantity
when the transcript supplies one. Each denomination remains separate. The
artifact never converts denominations, infers a missing count, calculates a
balance, or sums nearby events.
## Source Evidence And Normalization
Each source reference contains `source_id`, `start_unit_id`, and `end_unit_id`.
It must identify an ordered range in the current source document. During
extraction, every cited range must also be wholly contained in the current
accepted chunk. Campaign references may disambiguate names, but never provide
event evidence.
The deterministic normalizer trims only display-edge whitespace in names and
holders, canonicalizes source-reference order and exact duplicate references,
then orders events by earliest valid source position and stable tie-breakers
over name, kind, holders, quantity, and complete evidence. It removes only
events with the same normalized fields and complete valid evidence sequence.
Invalid evidence is never collapsed. It does not rename, singularize, resolve
aliases, infer holders or quantities, or merge nearby events.
The default extraction validation chain is JSON syntax, item-event shape,
source references, durable JSON Schema, then advisory source relatedness. The
normalization chain adds normalized invariants after shape validation and before
source references and schema validation. Relatedness warnings are advisory so
contextual names and currency notation do not reject otherwise valid evidence.
## Representative JSON
```json ```json
{ {
"events": [ "events": [
{
"name": "Hidden Cache",
"kind": "discovered",
"source_refs": [{"source_id": "session-7", "start_unit_id": 1, "end_unit_id": 1}]
},
{ {
"name": "Silver Pieces", "name": "Silver Pieces",
"kind": "acquired", "kind": "acquired",
"quantity": 20, "quantity": 20,
"to": "party", "to": "party",
"source_refs": [{"source_id": "session-7", "start_unit_id": 2, "end_unit_id": 2}] "source_refs": [
}, {"source_id": "session-7", "start_unit_id": 2, "end_unit_id": 2}
{ ]
"name": "Torch",
"kind": "lost",
"from": "party",
"source_refs": [{"source_id": "session-7", "start_unit_id": 3, "end_unit_id": 3}]
},
{
"name": "Healing Potion",
"kind": "consumed",
"from": "Aria",
"source_refs": [{"source_id": "session-7", "start_unit_id": 4, "end_unit_id": 4}]
},
{
"name": "Moonblade",
"kind": "transferred",
"from": "Aria",
"to": "Borin",
"source_refs": [{"source_id": "session-7", "start_unit_id": 5, "end_unit_id": 5}]
} }
] ]
} }
``` ```
## Production Boundary ## Holder rules and minimal extraction
The extractor requires `chunks` and `source.transcript`, and accepts optional `discovered` has neither holder; `acquired` requires `to` and forbids `from`;
`glossary`, `party`, `players`, and deprecated `roster` campaign references for `lost` and `consumed` require `from` and forbid `to`; `transferred` requires
disambiguation only. It has no generated NPC, scene-description, or item-registry both holders. `party` denotes collective possession. A transfer cannot use
dependency. The append-order merger preserves chunk order, and the normalizer `party` for either holder and its two normalized holders must differ.
has no reference slots.
The normalized lane is emitted as `lanes/<lane-id>.json` by the JSON output Only an evidenced discovery or possession change belongs in this artifact.
module. See [Configuration](../config.md#implemented-production-modules) for It does not infer quantities or holders, convert currency denominations,
the selectable keys and default chains, and the calculate balances, or merge nearby events. Campaign references may
[JSON output contract](json-output.md) for bundle paths. disambiguate names but are never event evidence. Currency uses the ordinary
`name` field and an explicit `quantity` only when the transcript establishes
one; each denomination remains a separate event.
Normalization trims display whitespace, orders and removes exact duplicate
source references, then orders events by valid source chronology, name identity
and display value, kind, holders, quantity, and reference sequence. It
collapses only entries with the same normalized durable fields and complete
valid evidence.
The [JSON output contract](json-output.md) defines publication. See
[D&D module internals](../internal/dnd.md) for implementation details and the
[NPC-interaction artifact](dnd-npc-interaction-artifacts.md) for a distinct
kind of occurrence.

View File

@@ -1,160 +1,69 @@
# D&D NPC Artifact # D&D NPC Artifact
This document defines the durable D&D NPC-list artifact, its JSON codec, and This contract defines the durable NPC registry produced by `dnd/npcs`. It is a
the selectable production NPC pipeline. The normalized JSON payload can be minimal, source-grounded identity registry for other D&D artifacts, not a
passed explicitly to the spell extractor as an optional caster-name registry character sheet or a relationship summary.
or to the combat extractor and normalizer as an actor registry. It
remains a reference, not spell or combat evidence.
The NPC interaction extractor and normalizer also consume this registry for
canonical identity; registry source references remain provenance and never
become interaction evidence. Their occurrence contract is defined in the
[D&D NPC interaction artifact](dnd-npc-interaction-artifacts.md).
## Identity ## Identity and compatibility
- Artifact kind: `dnd/npc-list` | Property | Value |
- Durable schema ID: `notarius.dnd.npcs` | --- | --- |
- Durable schema name: `notarius_dnd_npcs_v1` | Artifact kind | `dnd/npc-list` |
- Durable schema version: `v1` | Schema ID | `notarius.dnd.npcs` |
- Media type: `application/json` | Schema name | `notarius_dnd_npcs_v1` |
- Identity policy: `dnd.npcs.identity.v1` | Schema version | `v1` |
| Media type | `application/json` |
| Identity policy | `dnd.npcs.identity.v1` |
The durable JSON Schema is owned by the D&D NPC codec. NPC IDs are derived from `v1` accepts one strict JSON object with required `npcs`; the array may be
the Unicode-normalized, case-folded canonical name using the identity policy. empty. NPC and source-reference objects reject unknown fields. A future
The durable codec enforces the artifact shape and ID syntax; registry identity incompatible artifact shape or identity policy uses a new version or policy.
validation remains a separate deterministic concern.
The extractor's private LLM response schema is a separate structural transport ## Wire shape and identity
contract. It omits framework-assigned NPC and source IDs and admits semantic
candidates for the deterministic shape and source-reference validators; it is
not part of this durable contract.
## Output Shape Each NPC has these required fields:
The payload is one object with a required top-level `npcs` array: | Field | Contract |
| --- | --- |
| `id` | `npc:sha256:` followed by 64 lowercase hexadecimal characters. |
| `name` | Non-empty canonical display name. |
| `source_refs` | One or more transcript evidence ranges for the identity. |
A source reference has exactly `source_id`, `start_unit_id`, and `end_unit_id`.
The source ID identifies the transcript, unit IDs are positive inclusive unit
identifiers, and the start may not follow the end.
```json ```json
{"npcs": []} {
"npcs": [
{
"id": "npc:sha256:99a16589618a04f535a7d21fdcc71a0b1c05d22f752cd492065b1086d97bc3d7",
"name": "Mira Thorn",
"source_refs": [
{"source_id": "session-7", "start_unit_id": 4, "end_unit_id": 5}
]
}
]
}
``` ```
The array may be empty. Every object and nested object rejects unknown fields. The ID is deterministic: normalize the name to Unicode NFKC, normalize the
supported apostrophe forms, collapse whitespace, case-fold it, SHA-256 the
result, then prefix the lowercase hexadecimal digest with `npc:sha256:`. Each
canonical identity and ID appears at most once. Normalization collapses records
with the same canonical identity, retains their earliest position, and merges
their canonicalized evidence; it does not add aliases, roles, descriptions, or
relationship fields.
## NPC Fields ## Scope and consumers
Each NPC contains exactly these required fields: Only individually identifiable NPC names with transcript evidence belong in
this artifact. Groups, generic roles, invented labels, and descriptive
enrichment are excluded. Its source references prove registry provenance; they
do not become evidence for a spell, interaction, or combat occurrence.
- `id`: `npc:sha256:` followed by 64 lowercase hexadecimal characters; This registry can ground actor or caster names in the [spell](dnd-spell-artifacts.md)
- `name`: the canonical display name; and [combat-turn](dnd-combat-turn-artifacts.md) artifacts. It is required to
- `source_refs`: at least one source reference supporting the NPC record. resolve the canonical `name` in an [NPC interaction](dnd-npc-interaction-artifacts.md).
The [JSON output contract](json-output.md) defines publication, and
Each source reference contains required `source_id`, `start_unit_id`, and [D&D module internals](../internal/dnd.md) owns pipeline mechanics.
`end_unit_id`; unit IDs are positive integers. Source document identity, unit
existence, and range ordering are validated by the source-reference validator
when the artifact is used by a pipeline.
## Codec Boundary
`EncodeCandidate` and `DecodeCandidate` provide strict single-value JSON
serialization while preserving typed values that still need semantic
validation. `Encode` and `Decode` are the approved-artifact boundary and
require all durable structural fields, non-empty required strings, valid source
reference shapes, and the NPC ID pattern.
Codec metadata contains only `npc_count`. Schema bytes and returned metadata
are independent values so callers cannot mutate codec-owned state.
## Production Pipeline
The production identities are:
- extractor: `dnd/npcs`;
- artifact kind: `dnd/npc-list`;
- normalizer: `dnd/npcs`; and
- durable schema: `notarius.dnd.npcs`, version `v1`, media type
`application/json`.
The extractor maps private model records to the current source identity and
assigns deterministic IDs. Extraction validation checks shape, source
references, and source relatedness. The normalizer first consolidates equal
canonical-name matches, then may make one document-level LLM-assisted identity
decision per configured normalize attempt for eligible distinctly named
records. Consolidation is name-based: it retains a supplied canonical display
name, keeps the earliest affected output position, derives its ID again, and
unions exact evidence in canonical order. Ambiguous, invalid, or conflicting
proposals are not applied; independently safe matches may still be retained.
After the retry budget is exhausted, the safe result is accepted with bounded
normalization warnings and the usual validation. The durable v1 artifact shape
does not add aliases, proposal fields, or any other semantic-normalization
representation.
The extraction prompt asks only for individually identifiable NPC names backed
by source evidence. Groups, generic roles, invented labels, and descriptive or
relationship enrichment are outside the contract.
The default extraction chain is `generic/valid_json`,
`extract/dnd/npcs/shape`, `extract/dnd/npcs/source_refs`,
`generic/valid_json_schema`, and `extract/dnd/npcs/source_relatedness`. The
default normalize chain is `generic/valid_json`, `extract/dnd/npcs/shape`,
`normalize/dnd/npcs/identity`, `extract/dnd/npcs/source_refs`,
`generic/valid_json_schema`, and `extract/dnd/npcs/source_relatedness`.
Relatedness emits bounded warnings when an NPC canonical name is not present
near its cited transcript text; opaque campaign
references may explain such a warning but do not become evidence.
## Manifest And Artifact Handoff
The NPC extractor records prompt and response-schema identities. The durable
codec records only `npc_count`; raw names, source references, and payload bytes
stay in the lane file rather than manifest
metadata. The normalized lane can be consumed by a later ordered step through
the registered canonical codec:
```yaml
steps:
- id: identify-npcs
artifacts:
npcs:
extract: dnd/npcs
normalize: dnd/npcs
scene-descriptions:
extract: dnd/scene-descriptions
normalize: dnd/scene-descriptions
- id: grounded-events
references:
npcs:
artifact:
step: identify-npcs
lane: npcs
scene_descriptions:
artifact:
step: identify-npcs
lane: scene-descriptions
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells
combat:
extract: dnd/combat-turns
normalize: dnd/combat-turns
```
The framework hands only accepted normalized artifacts across the barrier. It
validates the canonical bytes against each consumer slot and clones the NPC
operation-time reference for spell and combat consumers. The accompanying
scene-description reference is required by the combat extractor for eligibility
only; its consumer contract is defined in the
[D&D combat-turn artifact contract](dnd-combat-turn-artifacts.md). Generated
provenance records the artifact kind, schema identity, media type, canonical
digest, size, and producer step/lane/module, but not names, source ranges, or
payload bytes. External normalized files remain supported as explicit references
and retain their file provenance.
NPC source references are registry provenance and are never accepted as spell
or combat evidence. Current transcript units remain the only event evidence.
Consumers receive a separate names-only projection in normalized registry
order, for example `{"npcs":[{"name":"Mira Thorn"}]}`. The projection omits
IDs and evidence. Its digest covers the exact projected bytes and is used for
consumer-local checkpoint identity, while the full durable artifact digest
remains the manifest and generated-reference provenance identity. The unbound
projection is exactly `{"npcs":[]}` and also has a projection digest.

View File

@@ -1,20 +1,38 @@
# D&D NPC Interaction Artifact # D&D NPC Interaction Artifact
This document defines the durable D&D NPC-interaction-list artifact and its This contract defines the durable occurrence list produced by
two-step production pipeline. It records discrete, source-grounded occurrences `dnd/npc-interactions`. It records discrete, source-grounded interactions with
for NPCs already accepted into a normalized NPC registry; it does not expand NPCs already present in a normalized registry; it does not extend that registry
the registry or summarize events. or summarize the session.
## Identity And JSON ## Identity and compatibility
- Artifact kind: `dnd/npc-interaction-list` | Property | Value |
- Durable schema ID: `notarius.dnd.npc_interactions` | --- | --- |
- Durable schema name: `notarius_dnd_npc_interactions_v1` | Artifact kind | `dnd/npc-interaction-list` |
- Durable schema version: `v1` | Schema ID | `notarius.dnd.npc_interactions` |
- Media type: `application/json` | Schema name | `notarius_dnd_npc_interactions_v1` |
| Schema version | `v1` |
| Media type | `application/json` |
The payload is one strict JSON object with only an `interactions` array. The `v1` is a strict JSON object with required `interactions`; the array may be
array may be empty. Each item has exactly `name`, `kind`, and `source_refs`: empty. Interaction and source-reference objects reject unknown fields. A future
incompatible shape requires a new schema version.
## Wire shape
Each interaction has these required fields:
| Field | Contract |
| --- | --- |
| `name` | Non-empty canonical display name from the required NPC registry. |
| `kind` | One of the interaction categories below. |
| `source_refs` | One or more transcript evidence ranges. |
Each source reference has exactly `source_id`, `start_unit_id`, and
`end_unit_id`. It identifies an inclusive range in the current transcript;
unit IDs are positive and the start may not follow the end. Extraction evidence
for an interaction is confined to its accepted chunk.
```json ```json
{ {
@@ -30,119 +48,31 @@ array may be empty. Each item has exactly `name`, `kind`, and `source_refs`:
} }
``` ```
`name` is the canonical display name from the required NPC registry. ## Interaction categories
`source_refs` contains one or more current-source ranges with required
`source_id`, `start_unit_id`, and `end_unit_id`; unit IDs are positive integers.
During extraction, every range must be wholly contained in the current accepted
chunk. This prevents a candidate from citing valid units that were not presented
to that extraction call.
Unknown fields are rejected.
## Interaction Categories
`kind` is exactly one of:
| Kind | Meaning | | Kind | Meaning |
| --- | --- | | --- | --- |
| `mentioned` | The NPC is referred to, but is not established as present or communicating in the evidenced passage. | | `mentioned` | The NPC is referred to but is not established as present or communicating. |
| `noncombat_presence` | The NPC is present and relevant to the passage but does not meaningfully participate in dialogue or combat. | | `noncombat_presence` | The NPC is present and relevant without meaningful dialogue or combat participation. |
| `dialogue` | The NPC speaks, responds, or is directly engaged in a meaningful non-combat exchange. | | `dialogue` | The NPC speaks, responds, or meaningfully participates in a non-combat exchange. |
| `combat_ally` | The NPC actively participates in combat on the party's side. | | `combat_ally` | The NPC actively participates in combat on the party's side. |
| `combat_opponent` | The NPC actively participates in combat against the party. | | `combat_opponent` | The NPC actively participates in combat against the party. |
| `other` | The transcript clearly establishes a direct NPC occurrence that fits none of the preceding kinds. | | `other` | A clearly evidenced direct occurrence not covered by another category. |
`other` is a residual category for positively evidenced activity, not a fallback The categories do not represent motives, relationships, state, or events that
for uncertain classification. When activities overlap, active combat the cited transcript does not establish. An `other` entry is not a substitute
participation outranks dialogue, presence, and mention; dialogue outranks for uncertain classification.
non-combat presence and mention; and non-combat presence outranks mention.
Combat alignment is not resolved by precedence: a meaningful change between
ally and opponent creates separate occurrences.
These categories do not encode summaries, relationships, state, motives, or ## Identity, evidence, and order
unobserved events.
## Occurrence Boundaries And Ordering The required normalized [NPC artifact](dnd-npc-artifacts.md) resolves `name`.
Registry references are provenance only and never replace an interaction's own
evidence. Normalization canonicalizes recognized registry names, orders and
deduplicates exact source references, then orders interactions by valid source
chronology, NPC comparison identity, display name, kind, and reference sequence.
Only entries with the same canonical name, kind, and complete valid evidence
sequence are collapsed; distinct categories or evidence remain separate.
One occurrence represents one NPC, one kind, and one locally coherent passage See the [combat-turn artifact](dnd-combat-turn-artifacts.md) for combat-action
within one accepted chunk. Repeated evidence belongs to the same occurrence occurrences and the [JSON output contract](json-output.md) for publication.
only while it supports the same uninterrupted activity. A kind change, combat Pipeline mechanics are described in [D&D module internals](../internal/dnd.md).
alignment change, intervening scene or meaningful absence, or transition from
mention to presence starts a new occurrence. Occurrences never span chunks, and
merge or normalization never semantically combines nearby, overlapping, or
cross-chunk records.
Normalization orders records by:
1. earliest valid source-document position;
2. the NPC identity comparison key;
3. the exact canonical NPC display name;
4. interaction kind in lexical order; and
5. the complete canonical source-reference sequence, ordered by source ID and
the source-document positions of each range's start and end.
Only records with identical canonical names, kinds, and complete valid evidence
sequences are duplicates. Different categories, ranges, or separately grounded
occurrences remain separate.
## Evidence, Registry, And Normalization
The registry proves only the canonical NPC identity. Its source references are
registry provenance and are never interaction evidence. Every durable
interaction must cite current transcript units supporting both the name and its
classification.
The extractor receives a names-only registry projection such as
`{"npcs":[{"name":"Mira Thorn"}]}`. The normalizer uses the full immutable
registry for exact canonical-name lookup. It canonicalizes source references
and applies the ordering and exact-duplicate rules above.
## Production Pipeline
The extractor and normalizer key is `dnd/npc-interactions`. Both require the
structured `npcs` slot, so an accepted normalized registry must come from an
earlier step:
```yaml
pipelines:
dnd-npc-interactions:
input: seriatim
steps:
- id: identify-npcs
artifacts:
npcs:
extract: dnd/npcs
normalize: dnd/npcs
- id: extract-interactions
references:
npcs:
artifact:
step: identify-npcs
lane: npcs
artifacts:
interactions:
extract: dnd/npc-interactions
normalize: dnd/npc-interactions
```
The framework passes only the accepted normalized producer. A missing, rejected,
or incompatible NPC artifact prevents the consumer step from executing. It
records generated artifact identity and bounded producer provenance without
copying registry names, source ranges, or payload content into the manifest.
## Validation And Metadata
The default extract chain is `generic/valid_json`, interaction shape, registry,
and source-reference validation, `generic/valid_json_schema`, then warning-only
source relatedness. The normalize chain runs normalized invariants after
registry validation and before source-reference and schema validation, followed
by relatedness. Normalizer and relatedness warnings are bounded and end with an
omission summary when necessary. The codec metadata contains only
`interaction_count`. Extractor metadata identifies its prompt and private
response schema; component-local checkpoint identities include the names-only
registry projection where relevant. Generated registry identity stays in
framework provenance and dependency fingerprints.
See [Configuration](../config.md#implemented-production-modules) for selectable
keys and chains, [the NPC artifact contract](dnd-npc-artifacts.md) for the
registry boundary, and the copyable
[complete D&D example](../../examples/dnd-complete.config.yml).

View File

@@ -1,9 +1,10 @@
# D&D Scene Description Artifact Contract # D&D Scene-Description Artifact
This document defines the durable `dnd/scene-description-list` artifact This contract defines the durable output of `dnd/scene-descriptions`. Each
emitted by the D&D scene-description lane. record classifies one accepted transcript chunk and gives it a minimal
source-grounded title and summary.
## Artifact identity ## Identity and compatibility
| Property | Value | | Property | Value |
| --- | --- | | --- | --- |
@@ -13,38 +14,32 @@ emitted by the D&D scene-description lane.
| Schema version | `v1` | | Schema version | `v1` |
| Media type | `application/json` | | Media type | `application/json` |
The normalized payload is written by the JSON output module to `v1` is a strict JSON object with required non-empty `scenes`. Scene and
`lanes/<lane-id>.json`. See [JSON output](json-output.md) for the surrounding source-reference objects reject unknown fields. A future incompatible shape
output bundle. requires a new schema version.
## JSON shape ## Wire shape
The payload is a JSON object containing exactly one required field, `scenes`. Each scene has exactly these required fields:
Each scene object contains exactly these required fields:
| Field | Shape and ownership | | Field | Contract |
| --- | --- | | --- | --- |
| `id` | Non-empty accepted chunk ID, assigned by Notarius. | | `id` | Non-empty accepted chunk ID, assigned by Notarius. |
| `source_ref` | Exact inclusive accepted chunk range, assigned by Notarius. | | `source_ref` | The assigned inclusive source range for that chunk. |
| `kind` | One of `combat`, `narrative`, `recap`, or `meta`. | | `kind` | `combat`, `narrative`, `recap`, or `meta`. |
| `title` | Non-empty, trimmed, source-grounded title. | | `title` | Non-empty, trimmed, source-grounded title. |
| `summary` | Non-empty, trimmed, source-grounded summary. | | `summary` | Non-empty, trimmed, source-grounded summary. |
All object layers reject unknown fields. The `scenes` array must be present and `source_ref` has exactly `source_id`, `start_unit_id`, and `end_unit_id`.
non-empty. `source_ref` has exactly `source_id`, `start_unit_id`, and Its source ID identifies the input transcript; its positive unit IDs identify
`end_unit_id`; its source ID is non-empty and its unit IDs are positive the chunk's inclusive range, with the start no later than the end.
integers.
```json ```json
{ {
"scenes": [ "scenes": [
{ {
"id": "chunk-000001", "id": "chunk-000001",
"source_ref": { "source_ref": {"source_id": "session-7", "start_unit_id": 1, "end_unit_id": 3},
"source_id": "session-alpha",
"start_unit_id": 1,
"end_unit_id": 3
},
"kind": "narrative", "kind": "narrative",
"title": "Arrival at the watchtower", "title": "Arrival at the watchtower",
"summary": "The party reaches the ruined watchtower and begins to investigate it." "summary": "The party reaches the ruined watchtower and begins to investigate it."
@@ -53,78 +48,22 @@ integers.
} }
``` ```
`id` and `source_ref` are application-owned identity and evidence. The model ## Meaning and normalized form
provides only `kind`, `title`, and `summary`; it is not asked for chunk IDs,
source IDs, unit IDs, ranges, participants, or confidence.
## Scene kinds `combat` identifies a chunk where active combat is the central activity.
`narrative` is current in-world play that is not principally combat, recap, or
meta discussion. `recap` is primarily a recounting of an earlier session, and
`meta` is primarily out-of-character discussion. The artifact does not add
participants, confidence, events, or information absent from the chunk.
| Kind | Meaning | Normalization trims title and summary, orders scenes by source position and
| --- | --- | then ID, and removes exact duplicate records. A reused ID with different
| `combat` | Active combat is a substantive central activity. | durable fields, or the same source range with different kind, title, or
| `narrative` | Current-session in-world play that is not principally combat, recap, or meta discussion. | summary, is invalid. It does not merge adjacent ranges, alter prose, or infer
| `recap` | The table is primarily recounting a previous session. | missing scenes.
| `meta` | Sustained out-of-character discussion is the scene's primary purpose. |
For a mixed accepted chunk, classification prefers `combat`, then `recap`, The [combat-turn artifact](dnd-combat-turn-artifacts.md) uses an exact matching
then `meta`, then `narrative`. Brief table talk, rules clarification, or a `combat` scene only as eligibility control; scene title, summary, and source
short recollection does not replace the main current-session activity. reference never become combat evidence. Publication is defined by the
[JSON output contract](json-output.md); implementation details live in
## Extraction and evidence [D&D module internals](../internal/dnd.md).
The extractor runs once for each accepted chunk and maps one successful model
response to one scene record. It copies the current chunk ID and exact chunk
range, preserves the model kind without repair, and trims only surrounding
whitespace from title and summary. A model response cannot represent an empty
result; extraction failure follows the configured retry and rejection policy.
Optional `players`, `party`, and `glossary` campaign references can help
disambiguate names or setting terms. They never supply scene evidence or add
events absent from the accepted chunk. The lane requires no NPC registry or
other generated artifact reference.
## Merge and normalization
Accepted per-chunk lists are appended in chunk order. Normalization then:
1. validates the current-source range, non-empty ID and prose, and closed kind;
2. trims only title and summary whitespace;
3. sorts records by source-document start position, then ID;
4. removes records only when all five durable fields are identical;
5. rejects a reused ID when any remaining durable field differs; and
6. rejects the same exact range when `kind`, `title`, or `summary` differs.
Two different IDs with the same range and identical model-owned content remain
separate records. Normalization does not join adjacent ranges, rewrite prose,
repair kinds, infer missing scenes, or use chunk annotations.
## Validation and warnings
Extraction validation requires exactly one record with an ID and range exactly
equal to its current chunk. Later validation checks source membership without a
current chunk. Shape, source-range, kind, ID, and normalized-invariant failures
reject the artifact.
Relatedness checks are advisory. They separately warn when a scene title or
summary has no significant lexical token in its cited transcript range. The
check ignores short tokens and common function words, uses transcript text only,
and does not treat campaign references as evidence. Warning diagnostics are
bounded and do not copy transcript or campaign-reference content.
## Configuration
Use `dnd/scenes` to form accepted scene chunks, then select
`dnd/scene-descriptions` for extraction and normalization. The maintained
[complete D&D configuration](../../examples/dnd-complete.config.yml) includes
this lane alongside the other D&D artifacts. Selectable keys and default
validator chains are defined in [Configuration](../config.md).
## Downstream combat use
The combat-turn extractor consumes this approved artifact as required
eligibility control context through an explicit ordered reference binding. It
uses only an exact chunk ID and source-range match with `kind: combat` to permit
combat extraction; titles, summaries, and scene references do not become combat
prompt material or evidence. The complete downstream behavior, including empty
results and warnings for unavailable coverage, is defined in the
[D&D combat-turn artifact contract](dnd-combat-turn-artifacts.md).

View File

@@ -1,204 +1,73 @@
# D&D Spell Artifact # D&D Spell Artifact
This document is the durable serialized artifact contract for the production This contract defines the durable output of the `dnd/spells` extractor and
D&D spell extractor. Selectable extractor keys are cataloged in normalizer. It records source-grounded spell-casting occurrences; it is not a
[Configuration](../config.md#implemented-production-modules). spellbook, a rules lookup result, or a record of hypothetical casts.
## Identity ## Identity and compatibility
- Artifact kind: `dnd/spell-list` | Property | Value |
- Prompt ID: `dnd.spells`
- Response schema key: `dnd_spells`
- Response schema ID: `notarius.dnd.spells`
- Response schema name: `notarius_dnd_spells_v1`
- Response schema version: `v1`
- Media type: `application/json`
The durable JSON Schema is owned by the D&D spell artifact codec. The
extractor's private LLM response schema is a separate transport contract: its
source-reference objects omit `source_id`, which the extractor assigns while
mapping the response to the canonical artifact. The LLM DTO and transport
schema are not part of this durable contract. The private schema owns required
fields, JSON types, object and array shapes, and unknown-field rejection;
deterministic validators own the durable artifact's semantic constraints.
The output contains canonical spell casts derived from transcript evidence.
Source IDs are assigned from the input identity; source-unit ranges identify
the evidence location.
## Output Shape
The extractor payload is a JSON object with one required top-level array. Its
structure is:
```text
{"spell_casts": [<spell-cast object>, ...]}
```
`spell_casts` must be present. It may be empty when no spell casts are found.
When multiple chunk results are combined, spell casts remain in chunk order.
When the payload is written as durable output, its logical path is derived from
the configured artifact lane ID as defined by the
[JSON output contract](json-output.md#output-payload-files).
## Spell-Cast Fields
Each spell cast contains exactly these required fields:
- `caster`: in-world character or creature casting the spell;
- `spell`: spell name;
- `source_refs`: transcript source references with extractor-assigned source
IDs and evidence unit ranges. It must contain at least one entry.
Both string fields must be non-empty. `caster` is the canonical in-world
caster, not the human player, transcript speaker, or GM when the associated
character or creature can be identified. Player and party references may
disambiguate that identity, but do not independently establish that a cast
occurred. The `spell` value must resolve through the effective SRD-plus-overlay
catalog as either a canonical name or alias. Catalog validation accepts aliases
but does not rewrite them; unknown fields are rejected.
The artifact includes an actual casting event or an unambiguous declared
casting attempt. Spell mentions, hypothetical plans, rules discussion, and
catalog matches without a casting event are excluded. The spell catalog is a
name-recognition policy and never evidence that a cast occurred.
## Source References
Each source reference contains exactly three required fields: `source_id`,
`start_unit_id`, and `end_unit_id`. The source ID must match the input identity.
The unit IDs must be positive integers present in the input, and the start unit
must not appear after the end unit. Unknown fields are rejected.
For each cast, the complete `source_refs` collection identifies the transcript
evidence for the caster, spell name, and occurrence of the cast or declared
attempt. The deterministic validators establish that ranges are structurally
valid and that the spell name is related to cited text. Semantic evidence
sufficiency is an extraction policy and remains subject to evaluation rather
than deterministic proof.
Reference slot keys and accepted file types are defined in
[Configuration](../config.md#implemented-production-modules). References are
supporting disambiguation material, not source evidence, and are not
addressable through `source_refs`.
## Optional NPC Grounding
The `dnd/spells` extractor accepts an optional `npcs` reference containing one
normalized NPC artifact as `application/json`, up to 1 MiB. An external file is
validated during preparation; an ordered generated binding is validated at the
step handoff. Both paths use the approved NPC codec and identity policy,
re-encode canonical durable JSON for registry provenance, and supply only the
registry's ordered names as the operation-time spell prompt input. It helps the
model prefer canonical caster names; it does not establish that a spell was
cast.
NPC source references may identify the run that produced the registry or any
other session. They remain registry provenance and are never copied into a
spell cast's `source_refs`; every spell evidence range must still identify the
current transcript. Generated provenance records producer and canonical
artifact identity without payload content or a path. When the slot is absent,
the prompt receives exactly `{"npcs":[]}` with its projection digest, and the
run has no NPC reference provenance.
## Normalization Behavior
When the `dnd/spells` normalizer is selected, each recognized spell name is
rewritten to the effective catalog's canonical display name. Lookup uses the
catalog's case-insensitive, whitespace-normalizing, apostrophe-normalizing, and
alias rules. Unknown names are preserved exactly for the normalize validators;
the normalizer does not guess or apply fuzzy matching.
Each cast's `source_refs` is copied, sorted by exact `source_id`,
`start_unit_id`, and `end_unit_id`, and stripped of exact structural
duplicates. Adjacent or overlapping ranges are not merged, and the normalizer
does not synthesize references or change their boundaries.
After those per-cast changes, duplicate identity requires the same canonical
spell name, the same caster after case folding and whitespace normalization,
and the same complete, non-empty set of source references valid for the source
document. Only the first occurrence is retained, in stable order. Its caster
and canonical references are preserved. Unknown names, empty or invalid
evidence, and casts with different evidence remain separate.
Mutation and duplicate decisions are returned through the normal warnings
surface. Warning scopes use the merged input index, such as `spell_casts[0]`,
so they remain meaningful even when a later duplicate is removed. The
normalizer uses these reason codes:
| Reason code | Meaning |
| --- | --- | | --- | --- |
| `spell_name_canonicalized` | A catalog lookup replaced an input name with its canonical display name. | | Artifact kind | `dnd/spell-list` |
| `spell_name_unresolved` | A name was not found in the effective catalog and was retained unchanged. | | Schema ID | `notarius.dnd.spells` |
| `source_references_normalized` | Reference order changed or exact duplicate references were removed. | | Schema name | `notarius_dnd_spells_v1` |
| `duplicate_spell_cast_collapsed` | A later cast matched the retained cast's complete duplicate identity. | | Schema version | `v1` |
| Media type | `application/json` |
Only warnings from an accepted normalize attempt are promoted to `v1` is a single strict JSON object. It requires `spell_casts`; the array may
`warnings.json`. If an unresolved name reaches the default normalize validator be empty. Each spell-cast object and source-reference object rejects unknown
chain, the catalog validator rejects the candidate with `unknown_spell`; the fields. A future incompatible shape requires a new schema version.
`spell_name_unresolved` warning remains in the attempt's debug artifact. An
explicit validator override that accepts the candidate promotes the unresolved
warning normally.
The default extraction and normalization chains both preserve this registered ## Wire shape
order: JSON syntax, spell shape, catalog membership, source references, JSON
Schema, then source relatedness. Extraction validates the private response
schema; normalization validates the durable artifact schema.
## Manifest Metadata Each `spell_casts` entry has these required fields:
The extractor adds prompt and response-schema provenance under the artifact lane | Field | Contract |
manifest metadata: | --- | --- |
| `caster` | Non-empty in-world character or creature name. |
| `spell` | Non-empty spell name. |
| `source_refs` | One or more transcript evidence ranges. |
Every source reference has exactly `source_id`, `start_unit_id`, and
`end_unit_id`. The source ID identifies the input transcript; the unit IDs are
positive inclusive unit identifiers, and the start may not follow the end in
that source. References are evidence for the cast, not campaign-reference or
NPC-registry provenance.
```json ```json
{ {
"metadata": { "spell_casts": [
"extractor": { {
"prompt_id": "dnd.spells", "caster": "Mira Thorn",
"prompt_version": "v1", "spell": "Fireball",
"prompt_sha256": "sha256:...", "source_refs": [
"response_schema_key": "dnd_spells", {"source_id": "session-7", "start_unit_id": 12, "end_unit_id": 13}
"response_schema_id": "notarius.dnd.spells", ]
"response_schema_name": "notarius_dnd_spells_v1",
"response_schema_version": "v1",
"response_schema_sha256": "sha256:...",
"catalog_base_id": "dnd-5e-2014-srd-spells",
"catalog_digest": "sha256:...",
"catalog_overlay_ids": ["campaign.example"],
"npc_registry_digest": "sha256:...",
"npc_count": 3
},
"normalizer": {
"catalog_base_id": "dnd-5e-2014-srd-spells",
"catalog_digest": "sha256:...",
"catalog_overlay_ids": ["campaign.example"]
}
} }
]
} }
``` ```
`catalog_digest` identifies the effective semantic catalog, while ## Evidence and normalized form
`catalog_overlay_ids` is sorted and empty for a base-only configuration. Raw
prompt, schema, catalog, alias, and local overlay-file content are not
included in manifest metadata. The `normalizer` metadata uses the same catalog
identity fields when that module is selected. Overlay origin, media type, byte
size, and raw digest are recorded separately in the manifest's reference
provenance; see the [JSON output contract](json-output.md#manifestjson).
The `npc_registry_digest` and `npc_count` fields in the example are present for An entry represents an actual cast or an unambiguous declared attempt. A spell
an external NPC registry when the extractor publishes its prepared module mention, rules discussion, plan, or catalog match alone is not an occurrence.
metadata. They contain no NPC names, source references, paths, or raw The configured catalog checks the name; it does not establish evidence.
bytes. A generated registry's identity is instead represented by the framework
handoff provenance and dependency fingerprint, so the consumer module metadata
does not duplicate it.
The extractor's prompt hash, private response-schema hash, and effective catalog When normalization is selected, recognized spell names use the effective
digest also contribute independently scoped semantic checkpoint fingerprints. catalog's canonical display name. Source references are put in canonical source
Changing any of those prepared contracts intentionally produces a cold order and exact duplicate references are removed. A later entry is collapsed
checkpoint miss. Fingerprints contain only digests, never prompt, schema, only when it has the same canonical spell, the same case- and
catalog, or reference content. When an NPC registry is bound, its semantic whitespace-insensitive caster identity, and the same complete valid reference
digest contributes an additional local `npc_registry` fingerprint for an sequence. Remaining entries retain their merged order.
external binding; the manifest metadata contains only that digest and
`npc_count`. Raw NPC file provenance remains independently recorded in the The optional normalized [NPC artifact](dnd-npc-artifacts.md) can ground a
manifest's `references` list. Generated bindings contribute the canonical caster name. Its own references remain registry provenance and are never copied
artifact dependency fingerprint and bounded producer provenance instead. into `source_refs`.
## Related contracts
The [spell-catalog overlay contract](dnd-spell-catalog-overlays.md) defines
the configured catalog additions. The [JSON output contract](json-output.md)
defines where this logical artifact is published; [D&D module internals](../internal/dnd.md)
describes extraction and validation mechanics.

View File

@@ -1,19 +1,27 @@
# D&D Spell-Catalog Overlay Contract # D&D Spell-Catalog Overlays
This document defines the JSON format accepted by the D&D spell catalog This document defines the optional JSON overlay consumed by the D&D spell
resolver. An overlay supplies campaign-specific spell names and aliases for extractor. An overlay contributes campaign spell names and aliases for
recognition. It does not supply spell rules, levels, classes, effects, or recognition. It does not define spell rules, effects, levels, classes, or
source evidence. transcript evidence. Bind the optional `spell_catalog` reference as described
in [Configuration](../config.md#references-and-ordered-handoffs).
The `dnd/spells` extractor accepts one optional UTF-8 `application/json` overlay ## Contract Identity
bundle through its `spell_catalog` reference slot. The framework materializes
that file relative to the configuration or command-line binding, enforces the
1 MiB slot limit, and records its origin and raw digest separately from the
effective catalog digest.
## Shape | Property | Value |
| --- | --- |
| Consumer | D&D spell extraction and normalization |
| Reference slot | `spell_catalog` |
| Media type | `application/json` |
| Required schema version | `notarius.dnd.spell-catalog-overlay.v1` |
| Base catalog | Embedded D&D 5e 2014 SRD catalog |
An overlay bundle has this shape: At most one overlay document may be bound. The maintained example is
[dnd-spell-catalog.json](../../examples/dnd-spell-catalog.json).
## Wire Shape
This is a minimal valid overlay:
```json ```json
{ {
@@ -22,49 +30,43 @@ An overlay bundle has this shape:
{ {
"id": "campaign.example", "id": "campaign.example",
"ruleset": "dnd-5e-2014", "ruleset": "dnd-5e-2014",
"source": { "source": {"title": "Example campaign spells"},
"title": "Example campaign spells", "spells": [{"name": "Aegis of Emberfall"}]
"version": "1",
"url": "",
"license": ""
},
"spells": [
{
"name": "Aegis of Emberfall",
"aliases": ["Emberfall Aegis"]
}
]
} }
] ]
} }
``` ```
The top-level `schema_version` and `catalogs` fields are required. The schema | Field | Required | Meaning and constraints |
version must be exactly `notarius.dnd.spell-catalog-overlay.v1`, and at least | --- | --- | --- |
one catalog is required. Catalogs require a unique, non-empty, trimmed `id`, | `schema_version` | Yes | Exactly `notarius.dnd.spell-catalog-overlay.v1`. |
the exact `dnd-5e-2014` `ruleset`, a `source`, and a non-empty `spells` array. | `catalogs` | Yes | Non-empty array of catalog objects with unique IDs. |
| `catalogs[].id` | Yes | Non-empty trimmed string. |
| `catalogs[].ruleset` | Yes | Exactly `dnd-5e-2014`. |
| `catalogs[].source.title` | Yes | Non-empty trimmed string. |
| `catalogs[].source.version` | No | String when present. |
| `catalogs[].source.url` | No | String when present. |
| `catalogs[].source.license` | No | String when present. |
| `catalogs[].spells` | Yes | Non-empty array of spell objects. |
| `catalogs[].spells[].name` | Yes | Non-empty trimmed string. |
| `catalogs[].spells[].aliases` | No | Array of non-empty trimmed strings when present. |
`source.title` is required and must be non-empty and trimmed. `source.version`, Unknown fields are rejected at every object level. The document must contain
`source.url`, and `source.license` are optional strings and may be empty. one JSON value; `null` is not accepted for optional strings or aliases.
Each spell requires a non-empty, trimmed `name`. `aliases` may be omitted or
may be an array of trimmed, non-empty strings; JSON `null` is not an alias
array. Overlay objects contain no other supported spell fields.
Decoding is strict: unknown fields, malformed JSON, trailing JSON values, and ## Composition And Compatibility
non-string optional source fields are rejected.
## Composition Notarius starts with the embedded base catalog, then applies overlay catalogs
in ascending catalog-ID order. A new canonical spell name adds a recognition
entry. If an overlay names an existing canonical spell, it augments that spell
with aliases while retaining the established display spelling.
The resolver always starts with the embedded D&D 5e 2014 SRD catalog. Overlay Repeated aliases for the same spell are accepted. A canonical-name, canonical-
catalogs are sorted by `id` before composition, so the input order does not to-alias, or alias-to-alias collision between different spells is rejected,
affect the result. A new canonical name adds a recognition entry. A canonical including a collision with the embedded catalog. Matching uses the catalogs
name matching an existing canonical name augments that spell and keeps the case, whitespace, and apostrophe normalization, so authors should avoid names
established canonical display spelling. Repeated aliases for the same spell or aliases that normalize to another spell.
are idempotent.
Canonical-name display conflicts and canonical/alias or alias/alias collisions The overlay is a recognition aid only. The durable spell-artifact schema and
between different spells are errors, including collisions with the embedded source-evidence rules are defined by the
catalog. Canonical names and aliases use the catalog's case, whitespace, and [D&D spell artifact contract](dnd-spell-artifacts.md).
common-apostrophe normalization rules. The effective catalog returns canonical
names in sorted order and produces a semantic SHA-256 digest that is stable
under JSON formatting, object-key, catalog, spell, and alias reordering.

View File

@@ -1,201 +1,111 @@
# JSON Output # Published JSON Output
This document is the durable JSON output file-format contract produced by the This document defines the logical JSON bundle emitted by the production JSON
production JSON encoder and written by the CLI. Selectable output-encoder keys output encoder. The bundles physical destination, atomic publication, and
are cataloged in retention are operational concerns; see [Operations](../operations.md#output-bundles).
[Configuration](../config.md#implemented-production-modules). Output configuration, including chunk-map export, belongs in
[Configuration](../config.md#module-bindings-and-validators).
The output module produces the logical bundle described here. The CLI's ## Bundle Layout
physical placement and lifecycle for that bundle are defined in
[Operations](../operations.md#output-directory).
## Files All paths below are logical, relative, slash-separated bundle paths. The
encoder always emits the first four JSON files below and adds lane or chunk-map
files when their corresponding artifacts are available:
The encoder writes: | Path | Purpose |
| --- | --- |
| `index.json` | Entry point that names the other published files and lane payloads. |
| `manifest.json` | Run provenance and result summaries. |
| `rejected.json` | Rejected pipeline outputs. |
| `warnings.json` | Accepted-output and run warnings. |
| `lanes/<safe-lane-id>.json` | One normalized artifact payload for each lane. |
| `chunk-map.json` | Optional accepted chunk map, when its export is enabled and available. |
- `index.json` JSON files are pretty-printed with a trailing newline. Lane payloads are
- `manifest.json` accepted only when their media type is `application/json`.
- `lanes/<lane-id>.json`, one file per normalized serialized artifact
- `rejected.json`
- `warnings.json`
- `chunk-map.json`, only when the JSON output binding enables
`include_chunk_map` and the run has an accepted chunk map
Files are pretty-printed JSON with a trailing newline when the payload is JSON.
Logical file paths are relative, slash-separated, and may not contain `..`.
## `index.json` ## `index.json`
Shape: `index.json` is the bundles discovery document. An approved run with no
normalized lanes has this valid minimal index:
```json ```json
{ {
"manifest_file": "manifest.json", "manifest_file": "manifest.json",
"output_files": [ "output_files": [],
{
"lane_id": "spells",
"media_type": "application/json",
"file": "lanes/spells.json",
"module_key": "noop",
"schema_id": "notarius.dnd.spells",
"schema_name": "notarius_dnd_spells_v1",
"schema_version": "v1"
}
],
"rejected_file": "rejected.json", "rejected_file": "rejected.json",
"warnings_file": "warnings.json" "warnings_file": "warnings.json"
} }
``` ```
`output_files` is sorted by lane ID. Output file names are produced by | Field | Required | Meaning |
sanitizing the lane ID: | --- | --- | --- |
| `manifest_file` | Yes | Always `manifest.json`. |
| `output_files` | Yes | Lane descriptors sorted by `lane_id`. |
| `rejected_file` | Yes | Always `rejected.json`. |
| `warnings_file` | Yes | Always `warnings.json`. |
| `chunk_map` | No | Descriptor for the pipeline-wide `chunk-map.json`; never a lane descriptor. |
- characters outside `A-Z`, `a-z`, `0-9`, `.`, `_`, and `-` become `_`; Each lane descriptor has required `lane_id` and `file`. It may also include
- repeated `..` sequences are replaced; `media_type`, `module_key`, `schema_id`, `schema_name`, and `schema_version`
- leading and trailing `.`, `_`, and `-` are trimmed; when supplied by the normalized artifact. A `chunk_map` descriptor contains
- empty sanitized names are rejected;
- two lanes that sanitize to the same output file are rejected.
`manifest_file`, `rejected_file`, and `warnings_file` contain the fixed paths
shown above. Each `output_files` entry requires `lane_id` and `file`. It also
contains the normalized payload `media_type`, normalizer `module_key`, and
response `schema_id`, `schema_name`, and `schema_version` when those values are
available.
When present, the top-level optional `chunk_map` descriptor contains exactly
`artifact_kind`, `file`, `media_type`, `schema_id`, `schema_name`, and `artifact_kind`, `file`, `media_type`, `schema_id`, `schema_name`, and
`schema_version`. It identifies the pipeline-wide `chunk-map.json`; it is not `schema_version`; its payload is defined by the
a lane output and never appears in `output_files`. The descriptor and file are [Accepted Chunk Map contract](chunk-map.md).
both absent when export is disabled or no chunk plan was accepted. Its payload
contract is defined by [Accepted Chunk Map](chunk-map.md). The lane path is derived from its lane ID. Characters outside letters, digits,
periods, underscores, and hyphens become underscores; `..` sequences are
neutralized; leading and trailing periods and underscores are removed. A lane
that produces an empty name, or two lanes that produce the same path, makes
output encoding fail.
## Lane Payloads
Each `lanes/<safe-lane-id>.json` file is the codec-owned normalized JSON for
that lane. Consumers should use the index descriptors schema identity rather
than infer a lane schema from its name. The current D&D payload contracts are
[spells](dnd-spell-artifacts.md), [NPCs](dnd-npc-artifacts.md),
[NPC interactions](dnd-npc-interaction-artifacts.md),
[combat turns](dnd-combat-turn-artifacts.md),
[item events](dnd-item-event-artifacts.md), and
[scene descriptions](dnd-scene-description-artifacts.md).
## `manifest.json` ## `manifest.json`
`manifest.json` contains a run manifest. This abridged example shows its core `manifest.json` is published provenance, not a copy of lane payloads or a
structure: checkpoint store. Fields without a value may be omitted. Its top-level fields
group into the following externally observable summaries:
```json | Group | Fields |
{ | --- | --- |
"run_id": "run-123", | Run identity and result | `run_id`, `pipeline_id`, `pipeline_digest`, `schema_version`, `validation_status`, `started_at`, `completed_at` |
"pipeline_id": "dnd-session", | Resolved components | `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, `output_encoder`, `artifact_lanes`, `validator_chains`, `module_metadata` |
"artifact_lanes": [ | Source and references | `source_digests`, `references` |
{ | Published result summaries | `normalized_outputs`, `rejected_outputs` |
"id": "spells", | Execution summaries | `chunk_plan`, `checkpoint_decisions`, `llm_profiles`, `metadata` |
"extractor": "dnd/spells",
"merger": "appendorder",
"normalizer": "noop"
}
],
"validation_status": "approved",
"started_at": "2026-01-01T00:00:00Z",
"completed_at": "2026-01-01T00:00:01Z"
}
```
Fields with empty values may be omitted by JSON encoding. `references` records provenance such as the target, slot, origin, digest,
media type, size, and generated-artifact identity. It does not contain
reference content. `normalized_outputs` and `rejected_outputs` likewise
summarize results without embedding lane payload bytes. A chunk-plan summary is
provenance for the plan used by this run; cache records, debug artifacts, and
other operational state are not published as bundle files.
The manifest fields are: ## Rejections And Warnings
- `run_id`, `pipeline_id`, and `pipeline_digest`: run and resolved-pipeline `rejected.json` is always an object with a `rejected` array. Each entry has
identity; required `stage` and `message`; `step_id`, `lane_id`, `module_key`, `chunk_id`,
- `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, and `chunk_index`, `validator_name`, `reason_code`, `attempt_count`, and
`output_encoder`: resolved module keys; `diagnostic_artifact_path` are present only when applicable.
- `chunk_plan`: payload-free provenance for the effective chunk plan. `mode`
is the effective cache mode; `action` is `reused`, `generated`,
`refreshed`, or `bypassed` when a plan was materialized. `requested_module`
is the current pipeline chunker, while `producer_input_module`,
`producer_module`, `producer_llm_profile`, `producer_references`,
`producer_metadata`, `source_digest`, `plan_digest`, `plan_schema_version`,
and `created_at` describe the stored or generated producer when available.
A cached plan can therefore identify a producer different from the requested
module. This object never embeds ranges, units, annotations, prompts,
responses, or reference content;
- `module_metadata` and `artifact_lanes`: module and per-lane provenance,
including prompt and response-schema provenance when provided;
- `validator_chains`: resolved validation points and validators;
- `source_digests` and `references`: source and reference provenance;
- `normalized_outputs` and `rejected_outputs`: payload-free result summaries;
- `llm_profiles`: selected profile IDs and provider or model names when
available;
- `metadata`: the effective prompt `session_id`;
- `validation_status`: `approved` or `rejected`;
- `started_at` and `completed_at`: UTC run timestamps.
`source_digests` contains source document digests only. Bound references are `warnings.json` is always an object with a `warnings` array. Each warning has
recorded separately under `references`, which contains provenance only: target `reason_code` and `message`; `scope` is optional. Both arrays are empty when
stage, lane ID when present, slot name, origin type and URI, digest, media there is nothing to report.
type, byte size, and binding source. Reference content is not written to
durable output.
Reference `stage` is `chunk`, `extract`, `merge`, or `normalize`. `lane_id` is ## Compatibility
omitted for chunk references and present for extract, merge, and normalize
references.
`validation_status` is `approved` when no outputs were rejected and `rejected` The index is the authoritative map from a logical lane to its published
when one or more outputs were rejected. payload. Consumers must tolerate omitted optional manifest and descriptor
fields, and should rely on the linked artifact contract for each lanes JSON
Producer warnings and the current run's chunk-validation warnings remain in shape. This contract describes the published logical bundle only; it does not
`warnings.json`. The manifest records only provenance and decision summaries; promise a filesystem layout or expose internal state formats.
empty producer-only values are omitted for compatibility with existing readers.
`validator_chains` records the resolved validator chain for each validation
point. Entries include stage, lane ID when applicable, module key, and validators
with key and execution class. Empty chains are recorded with an empty
`validators` array, including chains resolved from explicit empty config
overrides.
`normalized_outputs` summarizes each normalized lane output without embedding
payload bytes. Entries include lane ID, normalizer module key, source ID, media
type, and response schema provenance where available.
`rejected_outputs` summarizes rejected module outputs without embedding raw
payload bytes. Entries include stage, lane, module, chunk, validator or reason,
message, attempt count, and optional diagnostic artifact path.
## Output Payload Files
Each normalized serialized artifact is written to
`lanes/<sanitized-lane-id>.json`. The JSON output encoder is domain-neutral and
accepts only artifacts whose codec media type is `application/json`. The file
contains the codec-owned JSON bytes pretty-printed.
The schema of each lane payload is owned by that artifact contract. For the
current D&D lanes, see [D&D Spell Artifact](dnd-spell-artifacts.md),
[D&D NPC Artifact](dnd-npc-artifacts.md), and
[D&D Combat-Turn Artifact](dnd-combat-turn-artifacts.md), and
[D&D Scene Description Artifact](dnd-scene-description-artifacts.md).
## `rejected.json`
Shape:
```json
{
"rejected": []
}
```
When output validation rejects an output, each entry contains `stage` and
`message`. It includes `lane_id`, `module_key`, `chunk_id`, `chunk_index`,
`validator_name`, `reason_code`, `attempt_count`, and
`diagnostic_artifact_path` when applicable.
## `warnings.json`
Shape:
```json
{
"warnings": [
{
"scope": "extract",
"reason_code": "example",
"message": "human-readable warning"
}
]
}
```
`warnings` is an empty array when no warnings are reported.
Each warning requires `reason_code` and `message`; `scope` is omitted when it is
empty.

View File

@@ -1,69 +1,73 @@
# Seriatim Transcript JSON # Seriatim Transcript Input
This document is the external input contract consumed by the production This document defines the JSON transcript accepted by the production Seriatim
Seriatim input adapter. Selectable input-adapter keys are cataloged in input adapter. It is a source input, not a durable lane artifact. Configure the
[Configuration](../config.md#implemented-production-modules). input adapter through [Configuration](../config.md#production-module-keys).
## Adapter ## Contract Identity
- Source format: `application/vnd.seriatim+json` | Property | Value |
| --- | --- |
| Consumer | Seriatim input adapter |
| Media type | `application/vnd.seriatim+json` |
| Source document kind | `transcript` |
| Source-unit kind | `transcript_segment` |
## Accepted Shape ## Accepted Shape
The input must be one JSON object with top-level `metadata` and `segments` The input is one JSON object containing `metadata` and a non-empty `segments`
fields. This covers the maintained minimal fixture and Seriatim intermediate array. This minimal document is valid:
output that provides the same required segment fields.
The maintained example is ```json
[examples/seriatim-minimal-transcript.json](../../examples/seriatim-minimal-transcript.json). {
"metadata": {"id": "session-alpha"},
"segments": [
{
"id": 1,
"start": 0,
"end": 4,
"speaker": "Aria",
"text": "Aria casts Cure Wounds."
}
]
}
```
Required top-level fields: The maintained two-segment input is
[seriatim-minimal-transcript.json](../../examples/seriatim-minimal-transcript.json).
- `metadata`: an object. Its entries are accepted as source metadata. | Field | Required | Meaning and constraints |
- `segments`: a non-empty array of segment objects. | --- | --- | --- |
| `metadata` | Yes | JSON object. Its entries become source metadata; no particular metadata key is otherwise required. |
| `segments` | Yes | Non-empty array of segment objects, kept in input order. |
| `segments[].id` | Yes | Positive canonical decimal integer, supplied as a JSON number or string. IDs must be unique. |
| `segments[].start` | Yes | Finite, non-negative numeric value, supplied as a JSON number or string. |
| `segments[].end` | Yes | Finite, non-negative numeric value that is not earlier than `start`. |
| `segments[].speaker` | Yes | String that is non-empty after trimming. |
| `segments[].text` | Yes | String that is non-empty after trimming. Its original text is retained. |
Required segment fields: Additional top-level and segment fields are ignored. A missing required field,
`null` in place of an object or array, malformed JSON, or more than one
top-level JSON value is rejected.
- `id`: a positive integer JSON number or canonical decimal string without ## Source Identity And References
leading zeros or surrounding whitespace;
- `start`: a finite, non-negative JSON number or numeric string;
- `end`: a finite, non-negative JSON number or numeric string that is not less
than `start`;
- `speaker`: a non-empty string;
- `text`: a non-empty string.
Other top-level and segment fields, such as `categories`, are ignored. The adapter chooses the source ID in this order:
Multiple top-level JSON values are rejected. 1. a non-empty source ID supplied by the calling request;
2. non-empty string `metadata.id`;
3. non-empty string `metadata.source_id`;
4. `seriatim:` followed by the first 16 hexadecimal characters of the raw
inputs SHA-256 digest.
## Validation Each accepted segment becomes one source unit whose unit ID is `segments[].id`.
Its self-reference uses the derived source ID and the same segment ID for both
range endpoints. Artifact contracts use those segment IDs when they cite
transcript evidence.
The adapter rejects empty input, malformed JSON, multiple top-level JSON values, ## Compatibility
non-object segment values, duplicate segment IDs, and any violation of the
shape or field constraints above.
Segment text is preserved as provided, but it must not be empty after trimming. This adapter accepts only the shape described here. A broader Seriatim export
is usable only when it supplies this object, metadata, and segment shape with
## Derived Identity the stated types and constraints. Unknown additional fields do not add
Notarius behavior.
Notarius identifies the parsed source in this order:
1. `metadata.id`, when it is a non-empty string after trimming;
2. `metadata.source_id`, when it is a non-empty string after trimming;
3. `seriatim:<first-16-hex-chars-of-raw-sha256>`.
The exact raw input SHA-256 remains the basis of the fallback source ID. The
source digest recorded in output provenance is instead the SHA-256 of the
canonical generic source document, excluding the digest field itself. It covers
the derived source identity, document kind and format, ordered units and their
self-references, and accepted metadata. Segment IDs become the unit IDs used by
artifact source references; each produced unit carries a self-reference whose
source ID is the derived document ID and whose start and end IDs both equal the
segment ID.
## Compatibility Limit
This contract covers only Seriatim transcript JSON with the top-level
`metadata` object and `segments` array described here. Broader Seriatim output
schemas are compatible only when they provide these required fields with the
accepted types.

133
docs/internal/cli.md Normal file
View File

@@ -0,0 +1,133 @@
# CLI Internals
This document describes **internal/cli**, Notarius's production composition
root. The [CLI reference](../cli.md) owns command syntax and exit statuses;
[Configuration](../config.md) owns configuration values; and
[Operations](../operations.md) owns filesystem layout, recovery, and operator
procedures.
## Inputs, Outputs, And Boundaries
The CLI accepts process arguments, standard streams, and injectable options
used by tests and embedding code. It writes command results to the supplied
streams and returns a process exit status. For a run, it also creates the
production catalog and runtime collaborators, hands a prepared pipeline and
source bytes to the framework, and places the logical files returned by the
runner.
It is the only boundary allowed to compose concrete registries, LLM clients,
cache/checkpoint collaborators, debug recorders, and physical output paths.
Pipeline modules receive interfaces and request data rather than CLI streams or
filesystem roots. The [Architecture](../policy/architecture.md) defines this
composition-root boundary; [Pipeline Internals](pipeline.md) owns resolution,
preparation, and runner mechanics after their inputs are supplied.
## Dispatch And Configuration Handoff
The root dispatcher handles help, configuration validation, pipeline listing,
and a pipeline run. It normalizes injectable options before dispatch so that a
missing production dependency fails as a command error rather than reaching
execution.
Commands that need configuration use one shared loader. The CLI discovers the
file, parses it through **internal/core/config**, starts from defaults, applies
the file and supported environment overrides, and then validates it for the
command. The configured discovery and precedence contract is in
[Configuration](../config.md), while the loading and resolution mechanics are
in [Configuration Internals](configuration.md).
Configuration validation without a selected pipeline checks structural
configuration only. Validation with a selected pipeline also builds the
effective catalog, resolves the pipeline, and verifies explicitly selected
Scriptorium profiles. Pipeline listing validates configuration before returning
normalized, sorted identifiers.
## Production Composition
The production composition helper allocates every framework registry and the
prompt-asset registry, then registers the generic, Seriatim, and D&D module
families in that order. The resulting registries provide both the module
catalog used for resolution and the concrete constructors used for preparation.
Tests may provide a catalog or registries instead; production code must not
silently merge an injected partial catalog with production registrations.
The production LLM factory builds the Scriptorium-backed client from resolved
configuration, creates one scheduler from the effective global LLM limit, and
wraps the client before it reaches modules. Registration and LLM construction
errors are returned before a pipeline is prepared. Concrete module keys and
validator chains are public configuration choices and remain documented in
[Configuration](../config.md).
## Run Orchestration
After parsing and validating a run invocation, the CLI performs this ordered
handoff:
1. load and validate configuration, then apply command-level operational
overrides;
2. create and validate a safe run identity, then allocate a debug bundle only
when requested;
3. build the effective catalog, resolve requested reference changes, resolve
the effective pipeline, and verify explicit Scriptorium profiles;
4. materialize external or generated references and record redacted invocation
and resolution provenance when debug capture is enabled;
5. construct registries, the scheduled LLM client, prepared modules, and the
requested cache/checkpoint collaborators;
6. read the source input and invoke the framework runner; and
7. write the runner's logical output files only after a successful run, then
complete the command report and user-facing result.
Preparation happens before source parsing, so module construction and
dependency failures cannot begin stage execution. The CLI also preserves the
framework's result and warning information when it writes summaries and the
final command result. Detailed state lifecycle, resume handling, and physical
path confinement are maintained in [Run State Internals](state.md) and
[Operations](../operations.md).
## Failure Mapping And Terminal Reporting
Argument, flag, and invocation-combination failures are reported to standard
error before runtime composition and use the syntax error class. Once an
invocation is syntactically valid, configuration loading and validation,
resolution, registration, profile checks, reference materialization, module
construction, input reads, runner failures, output publication, and requested
debug handling use the runtime failure class. The public status numbers and
stream contract are defined in the [CLI reference](../cli.md#output-streams-and-exit-statuses).
When debug capture has been allocated, one command-state value records the
known run result. Guarded terminalization writes a success report once, or
attempts a failure report and error record once. A persistence failure is
reported in addition to the original failure and never replaces it. If a debug
path exists, failure output includes that path so the retained diagnostic data
is discoverable.
## Invariants To Preserve
- Only the CLI composes production implementations and physical runtime roots.
- Configuration and resolved composition failures occur before module
preparation or source parsing.
- A runner's logical files are published only after a successful run.
- Production registries and a caller-supplied catalog or registries are
alternative composition sources, not an implicit mixture.
- A requested debug bundle has one terminal report attempt; its persistence
errors supplement rather than obscure the primary command error.
- User-facing flags, paths, exit codes, and configuration fields are defined
by their public documentation, not duplicated here.
## Focused Tests
- **internal/cli/command_contract_test.go** covers dispatch, help, syntax and
runtime error classes, discovery, validation, and listing.
- **internal/cli/run_contract_test.go** covers the run handoff, publication,
debug reporting, and command-owned state collaborators.
- **internal/cli/production_contract_test.go** covers registrar composition,
production catalog contents, assets, and representative configuration
validation.
- **internal/cli/reference_contract_test.go** covers CLI reference overrides,
origin separation, and materialization boundaries.
- **internal/cli/state_hardening_test.go** covers safe run identity, state
roots, and failure ordering.
Run **go test ./internal/cli** after changing command composition or command
behavior. Pair it with **go test ./internal/core/config** when the configuration
handoff changes.

View File

@@ -0,0 +1,129 @@
# Configuration Internals
This document describes the maintainer-facing configuration boundary in
**internal/core/config**. The [Configuration](../config.md) reference owns the
file format, fields, defaults, precedence contract, and selectable keys. The
[CLI reference](../cli.md) owns command syntax; this document does not redefine
either interface.
## Boundary
The configuration package turns a selected YAML file and supported environment
values into a validated, independently owned configuration. It then resolves a
requested pipeline against a module catalog before the framework prepares or
runs anything.
| Boundary | Inputs | Outputs | Does not own |
| --- | --- | --- | --- |
| Loading | Selected file path and environment lookup | Parsed file model and a populated **Config** | Choosing the file path or reporting a command result. |
| Validation | **Config** | Structural configuration errors with pipeline, lane, or binding context | Module availability, capabilities, or construction. |
| Resolution | Valid **Config**, selected pipeline and lanes, runtime reference changes, LLM override, and module catalog | **EffectiveConfig** with a **ResolvedPipeline** | Materializing reference bytes, preparing modules, execution, or filesystem state. |
| Summary | **Config** or **EffectiveConfig** | Detached redacted payload suitable for debug summaries | Redacting arbitrary process state or provider traffic. |
The CLI discovers a configuration file, invokes this package, and supplies the
result to the framework. Configuration never reads an input file, constructs a
module, or creates output, cache, or debug paths. Those responsibilities remain
at their respective [CLI](cli.md), [pipeline](pipeline.md), and
[run-state](state.md) boundaries.
## Loading And Validation
The CLI loads configuration in this order:
1. parse the selected YAML file strictly into the file model;
2. start from **Default**;
3. apply the file model; and
4. apply the supported environment overrides.
This establishes the public precedence order without giving environment input a
second file schema. Loading and application reject malformed YAML, unsupported
file versions, unknown fields, invalid values, and identifiers that are empty
or collide after whitespace normalization. The file application also makes the
effective extraction-worker default follow the effective LLM limit.
**Config.Validate** checks configuration-only invariants before resolution. It
rejects incompatible profile sources, invalid state-surface values, unsupported
concurrency settings, malformed bindings and references, invalid retries, and
invalid pipeline, step, or lane structure. Its errors retain the closest known
pipeline, lane, and binding context. It deliberately does not require modules
to be registered: that requires a catalog and belongs to resolution.
The exact user-selectable values and validation rules are defined in
[Configuration](../config.md). Keep additions to the file model, an
environment override, its validation, and that reference in the same change.
## Effective Resolution
**Config.Resolve** first recomputes derived concurrency defaults and validates
the configuration. It normalizes the requested pipeline ID, copies the selected
profile, applies a non-empty command-level LLM profile override to the
LLM-capable stage bindings, and calls the framework resolver with the requested
lane selection and reference changes.
The command-level override does not replace an explicitly selected validator
profile. Validator bindings remain part of the resolved validator chain and
are resolved under their own declared configuration.
The framework resolver supplies defaults, selects lanes, resolves validator
chains, checks registered module and artifact compatibility, validates module
options, and returns the fixed ordered pipeline shape. The resulting
**EffectiveConfig** retains the selected ID, requested selection and reference
changes, a clone of the input configuration, and the resolved pipeline.
Callers may therefore retain or modify their input slices and maps without
changing the resolved result, and later consumers cannot mutate the original
configuration through the effective value.
Resolution failures stop before module construction and source parsing. They
include an error path for an unconfigured pipeline, missing module, missing
capability, incompatible artifact variant, invalid option, invalid reference,
or invalid lane selection. CLI code maps these valid-invocation failures to the
runtime error class described in the [CLI reference](../cli.md#output-streams-and-exit-statuses).
## Resolved Identity And Redaction
The framework assigns the resolved pipeline a deterministic SHA-256 digest
after defaults, lane selection, module bindings, reference bindings, validator
chains, and artifact schema identity have been resolved. The digest excludes
its own stored value. It identifies resolved composition rather than raw YAML
bytes, a debug payload, or all runtime state. The CLI records it as invocation
provenance before execution; cache and checkpoint identity have additional
owners in [Run State Internals](state.md).
Configuration summaries must use **Redacted**, **RedactedSummaryPayload**, or
**RedactedResolvedPipelinePayload**, never a direct configuration marshal.
Those methods copy every binding and nested option container, replace values
whose key is credential-shaped with **[REDACTED]**, and omit materialized
reference content while retaining safe binding and reference provenance. The
payload must not alias the source configuration or resolved pipeline. This
redaction is deliberately narrow: it protects configuration summaries and does
not authorize recording arbitrary environment values or provider requests.
## Invariants To Preserve
- Defaults, YAML values, and environment values are applied in one direction;
later sources may override only their supported operational settings.
- A configuration is structurally valid before it is resolved, and a resolved
pipeline is compatible with the supplied catalog before preparation begins.
- Whitespace-normalized identifiers are unique wherever they identify a
pipeline, step, lane, worker, or reference slot.
- Resolution and summary generation return detached data. Redaction must cover
every configured and resolved binding, including nested validator bindings.
- The resolved digest changes when resolved composition changes and never
includes itself.
## Focused Tests
- **internal/core/config/file_config_contract_test.go** covers strict file
parsing, normalization, file application, and structural rejection.
- **internal/core/config/env_contract_test.go** covers supported operational
overrides and their precedence.
- **internal/core/config/validation_contract_test.go** covers configuration
invariants and contextual failures.
- **internal/core/config/effective_config_contract_test.go** covers defaults,
selections, overrides, resolution context, digest changes, and ownership.
- **internal/core/config/redaction_test.go** covers recursive credential
redaction, reference-content exclusion, and non-aliasing payloads.
Run **go test ./internal/core/config** after changing this boundary. Changes to
the handoff or resolved-composition semantics also need the focused framework
pipeline tests.

119
docs/internal/dnd.md Normal file
View File

@@ -0,0 +1,119 @@
# D&D Module Internals
This guide records the conventions shared by the production D&D module family.
It complements [Module Internals](modules.md), which owns generic registration
and extension mechanics, and [Configuration](../config.md), which owns the
selectable keys, bindings, reference syntax, and default validator chains.
## Durable Artifact Contracts
The six lanes have separate durable wire contracts. This guide deliberately
does not repeat their JSON shapes or schemas.
| Lane | Durable contract |
| --- | --- |
| Spells | [spell artifacts](../integrations/dnd-spell-artifacts.md) |
| NPCs | [NPC artifacts](../integrations/dnd-npc-artifacts.md) |
| Combat turns | [combat-turn artifacts](../integrations/dnd-combat-turn-artifacts.md) |
| Item events | [item-event artifacts](../integrations/dnd-item-event-artifacts.md) |
| NPC interactions | [NPC-interaction artifacts](../integrations/dnd-npc-interaction-artifacts.md) |
| Scene descriptions | [scene-description artifacts](../integrations/dnd-scene-description-artifacts.md) |
## Family Composition
The D&D registrar registers the familys artifact codecs, extractors, typed
append-order mergers, normalizers, validators, prompt assets, and default
validator chains. Each extractor and normalizer has a stable module spec,
strict option decoding, and a typed builder. Configuration remains the
canonical owner of the exact keys and validator order.
Private structured-LLM response schemas are deliberately minimal. They reject
invalid JSON structure, missing required fields, incompatible types, and
unknown fields, while preserving semantic candidates for deterministic
validation. Do not promote a private response envelope into a durable schema;
the contracts above define durable data.
## Prompt Construction
D&D extractors assemble prompts from an ordered manifest of shared and
module-owned assets. Reuse the shared D&D system, evidence, identity,
reference, and transcript assets instead of copying their text into individual
modules. A manifests declared sequence, including cache-control placement, is
part of the prompt behavior, and the chunk transcript is the final message.
Preserve that order when changing an extractor or its assets so prompt-cache
behavior remains stable.
All extractors use the shared prompt-input preparation rules. The current chunk
is copied into transcript material; player, party, glossary, and compatible
campaign references are context for disambiguation, not source evidence.
Reference prompt material is canonically ordered before it is rendered, which
keeps equivalent inputs stable across runs.
## Evidence, Candidates, And Normalization
The current transcript is the only durable evidence source. Extractors assign
the current source identity, preserve candidate evidence ranges for validators,
and canonically order or remove exact duplicate ranges without asking the
model to repair semantic errors. Campaign context and generated artifacts may
ground names or control routing, but they never establish evidence for a D&D
result.
Default chains keep responsibilities separate: structural validators assess the
candidate, source-reference validators resolve cited ranges against the current
source, durable-schema validation checks an approved representation, and
relatedness validators report advisory evidence concerns. The configured order
is documented in
[Configuration](../config.md#production-validator-keys-and-default-chains).
Normalizers are deterministic for spells, combat turns, item events, NPC
interactions, and scene descriptions. They canonicalize display values and
evidence, use source-document order for stable output, and issue bounded
warnings for changes or collapsed duplicates. The NPC normalizer is the
intentional exception: it first produces a deterministic candidate set, then
uses a bounded structured-LLM proposal to reconcile identity groups. Invalid
or unusable proposals retain the deterministic result and surface retry or
fallback diagnostics; the model does not directly replace durable records.
## Generated References And Grounding
Normalized D&D artifacts can be handed to a later step through a generated
reference binding. The framework verifies artifact compatibility and retains
producer provenance; consumers resolve the handed-off artifact into an
immutable, validated projection for each operation. External files are checked
during preparation, while generated artifacts are resolved at the handoff.
NPC registries are names-only grounding projections: they may canonicalize
actors for spells and combat turns and are required for NPC interactions, but
they do not supply evidence. Scene-description registries are eligibility-only
projections: they retain the current chunks classification data, not scene
prose or evidence, and exist to route combat extraction.
## Lane-Specific Rules
The following differences are intentional and should remain explicit when a
shared helper changes.
| Lane | Intentional behavior |
| --- | --- |
| Spells | May use a spell-catalog overlay and optional NPC grounding; the catalog validator supplies domain-specific semantic checks. |
| NPCs | Does not consume an NPC registry. Its normalizer is the LLM-assisted reconciliation exception described above. |
| Combat turns | Requires a scene-description artifact. It calls the LLM only for an exact `combat` classification; exact non-combat classifications return an accepted empty result, while missing or mismatched classifications return an empty result with a bounded warning. Optional NPC grounding never becomes evidence. |
| Item events | Uses campaign context for disambiguation but has no NPC-registry or scene-description dependency. |
| NPC interactions | Requires the normalized NPC registry at extraction and normalization, using it for canonical actor grounding only. |
| Scene descriptions | Produces the classifications consumed by combat routing; it does not consume an NPC registry or provide evidence for combat artifacts. |
The combat and scene-description contracts describe their exact handoff and
empty-result behavior in more detail:
[combat turns](../integrations/dnd-combat-turn-artifacts.md) and
[scene descriptions](../integrations/dnd-scene-description-artifacts.md).
## Focused Verification
When changing D&D behavior, test the affected codec, extractor, normalizer,
validator, prompt-asset manifest, and registry projection. Also test generated
handoffs at the integration boundary and run the full D&D module suite:
~~~sh
go test ./internal/modules/dnd/...
go test ./internal/modules/integration/...
~~~

View File

@@ -1,252 +1,149 @@
# LLM Runtime Internals # LLM Runtime Internals
`internal/framework/llm` implements Notarius's transport boundary for structured `internal/framework/llm` is Notariuss provider-independent structured
completion. It contains the Scriptorium adapter, concurrency scheduler, completion boundary. It adapts framework requests to Scriptorium, bounds
prompt/schema registries, selected-profile recording, and provider-error provider calls, assembles registered prompt and schema assets, records selected
redaction. profiles, and redacts provider errors. The architectural boundary is defined in
[Architecture](../policy/architecture.md#llm-boundary); profile sources,
credentials, and concurrency settings belong in
[Configuration](../config.md#scriptorium-profiles) and
[Configuration](../config.md#concurrency-output-cache-and-debug).
Provider-neutral ownership rules are defined in ## Structured Completion Boundary
[Architecture](../policy/architecture.md#llm-boundary). Profile sources,
credentials, and concurrency settings are defined in
[Configuration](../config.md).
## Structured Contract Modules and LLM-backed validators depend only on
`contracts.StructuredLLMClient`. A completion request supplies a prompt ID and
version, optional profile and session IDs, named input material, variables, and
a caller-owned decode target. The successful response returns the validated raw
structured bytes together with non-secret provider, model, profile, and token
metadata.
Modules and LLM-backed validators depend on The caller owns the domain behavior: it chooses the prompt, prepares inputs,
`contracts.StructuredLLMClient.CompleteStructured`. A request identifies a selects the private response schema, and interprets the decoded result. The
prompt and optional profile/session, supplies named input materials and adapter does not own source evidence, artifact conversion, normalization, or
variables, and provides a caller-owned decoding target. A successful response durable schemas. Those responsibilities remain with the module and its
contains the validated raw structured bytes plus non-secret provider, model, [integration contract](../integrations/).
profile, and token metadata.
The caller owns prompt selection, response-schema selection, and interpretation `ScriptoriumClient` validates the request target and prompt identity, maps each
of the decoded result. `LLMInputMaterial` keeps source and reference bytes with named material to a Scriptorium inline artifact while preserving its origin URI,
their origin metadata so the adapter can pass named artifacts to Scriptorium forwards session and profile selection, then prepares and runs the prompt. It
without exposing Scriptorium types through stage contracts. returns Scriptoriums validated raw bytes rather than re-encoding the decoded
target. An empty optional material is represented as one space so its named
input is retained by Scriptorium.
## Production Construction An empty request profile lets the prompt select its configured default. The CLI
prepares every explicitly selected binding profile before a run begins, so a
missing explicit profile fails before stage execution. Calls record the profile
actually selected by Scriptorium; the recorder deduplicates non-secret profile
identity, provider, and model values for manifest use.
`internal/cli` constructs the production runtime by: ## Shared Provider-Call Limit
1. allocating the asset registry populated by the generic, Seriatim, and D&D Production construction creates one Scriptorium client and wraps it in one
package-family registrars; scheduled client. The scheduler has a fixed, positive permit limit, serves
2. creating a `ScriptoriumClient` from the effective profile source; queued calls in FIFO order, and removes a queued call when its context is
3. attaching an `LLMProfileRecorder`; cancelled. A granted permit is released exactly once on every completion path.
4. creating a scheduler from the effective concurrency limit;
5. returning a `ScheduledClient` wrapper;
6. decorating that shared client before preparation when debug recording is
enabled; and
7. injecting that one shared client into complete pipeline preparation before
the source file is read or the runner is invoked.
The D&D scene chunker and spell, NPC, combat-turn, item-event, NPC-interaction, and The scheduled wrapper surrounds every `CompleteStructured` call, so concurrent
scene-description extractors retain this lanes, pipeline retries, and LLM-backed validators share the same provider-call
injected client and use it for every structured completion. Operation requests ceiling. This ceiling is independent of pipeline worker concurrency; changing
do not carry an LLM client. worker counts cannot exceed the configured LLM limit. The configuration field
and its effective default are owned by
The CLI separately gathers explicit profile IDs from resolved LLM-capable stage [Configuration](../config.md#concurrency-output-cache-and-debug).
and validator bindings. It prepares a small internal check prompt for each ID so
missing or invalid profiles fail before pipeline execution. The runtime profile
override syntax and scope are defined in the
[CLI reference](../cli.md#run); binding rules are defined in
[Configuration](../config.md#module-bindings).
## Scriptorium Adapter
`ScriptoriumClient` converts a Notarius request into a Scriptorium `RunRequest`.
It validates the decoding target and prompt identity, maps named input materials
to inline artifacts, forwards explicit profile and session context, delegates
rendering/provider execution/structured validation, and unmarshals successful
JSON into the caller target.
Empty optional input material is represented by a single space so Scriptorium
retains the named input. The client returns Scriptorium's validated structured
bytes rather than re-encoding the caller target, allowing modules to preserve
the runtime result exactly.
Selected profile, provider, model, and token metadata are mapped into the
Notarius response. The recorder deduplicates profiles by identity and supplies
manifest-safe profile summaries after actual calls; manifest population does
not guess the selected prompt default in advance.
Generated-output validation failures and provider failures are wrapped with
prompt context. Error strings pass through bearer-token redaction before they
cross the runtime boundary.
## Scheduling
`Scheduler` uses a bounded permit count and a FIFO waiter queue. Immediate
acquisition increments the in-flight count; queued acquisition waits for a
permit or context cancellation. Cancellation removes a queued waiter, while a
cancelled waiter that has already received a permit releases it.
`ScheduledClient` acquires a permit around each structured completion and
defers release on every result path. The effective limit and default are
configuration facts in [Configuration](../config.md#defaults).
This provider-call ceiling is independent of the pipeline's extract worker
limit. Concurrent lanes, retries, and validators all use the same scheduled
client, so increasing framework workers cannot exceed `total_llm`. Pipeline
dispatch and cancellation mechanics are documented in
[Pipeline Internals](pipeline.md#execution-flow).
## Prompt And Schema Assets ## Prompt And Schema Assets
`AssetRegistry` combines caller-owned prompt filesystems under stable prefixes An `AssetRegistry` collects prompt and schema filesystems from production module
and rejects invalid or conflicting registrations. Production module packages families. It flattens registered roots into the Scriptorium filesystems and
register their own prompt and schema assets; generic framework code contains no rejects invalid roots, unreadable assets, duplicate paths, and missing prompt
D&D prompt content. `internal/framework/promptfs` provides the domain-neutral or schema files during preparation. The frameworks `promptfs` helper combines
filesystem composition helper used to combine module-owned files with shared module-owned prompt files with reusable domain fragments without making the
domain prompt fragments. framework depend on D&D content.
The D&D scene chunker and spell, NPC, combat-turn, item-event, NPC-interaction, and Each LLM-backed module owns its prompt declaration, package-specific assets,
scene-description extractors each declare an and private response schema. Shared D&D wording is owned by the D&D shared
ordered prompt asset manifest. The manifest lists the package-owned YAML and asset package; the detailed D&D conventions are in
Markdown files, then the exact shared fragments rendered by that prompt; the [D&D Module Internals](dnd.md). The mounted prompt assets used by a module also
same ordered list drives both filesystem mounting and the prompt fingerprint. determine its prompt fingerprint. Schema loaders validate JSON, attach identity
Unused shared assets are neither mounted nor fingerprinted. Universal and digest metadata, make defensive copies, and expose diagnostics without raw
extraction-evidence and output policy lives only in the shared extraction schema bytes.
assets; package-owned prompt files retain artifact-specific rules. The scene
prompt keeps its separate output rule because it does not render the
extraction-evidence asset.
### D&D Extraction Prompt Ordering And Cache Boundaries Private response schemas validate a model transport envelope. They are not the
durable artifact schema and should not be documented as an external wire
contract. Durable formats and compatibility rules remain in the
[integration contracts](../integrations/).
D&D extraction prompts order messages from the most reusable content to the ## Prompt Maintenance And Backend Caching
most variable content. New extraction lanes use these tiers in order:
1. universal shared content, including the system, extraction-evidence, and Prompt message order and shared asset bytes are runtime behavior. Backend cache
in-world identity messages; reuse depends on the same preceding messages and content, not merely equivalent
2. stable campaign or run context shared across lanes, including campaign meaning. Keep reusable shared assets byte-identical and keep stable material
references; before the inputs that vary per request wherever a prompts declared sequence
3. stable subset- and lane-specific context and instructions, including an NPC supports caching. Preserve the existing manifest order and cache-control hints
registry, catalog, task, or extraction instructions when applicable; when editing a prompt.
4. the chunk transcript as the final user message.
This ordering lets requests reuse the longest identical prefix before the D&D extraction manifests place the changing chunk transcript at the end of the
per-chunk transcript changes. Cache reuse requires the preceding message prompt after their reusable context. Scene chunking and NPC normalization use
sequence and content to be exactly identical; semantic similarity is not their own declared message sequences because their inputs and work differ. The
sufficient. Cache boundaries belong at the ends of reusable stable tiers, family-specific asset and ordering rules belong in [D&D Module Internals](dnd.md).
subject to the provider's cache-boundary limit. The shared identity and Do not add tests that enforce a fixed message-prefix length; prompt-asset tests
campaign-reference messages form the first two extraction boundaries. Spell, should instead verify the meaningful asset sequence, inputs, and cache controls
combat, and interaction prompts add a boundary at the shared NPC registry. Each extraction of the prompt being changed.
prompt places its final boundary on its lane-specific instructions, immediately
before the transcript. The transcript does not carry cache control because no
reusable content follows it.
Accordingly, the common prefix of the spell, NPC, combat, item-event, and interaction ## Validation, Repair, And Retries
extraction prompts is system,
extraction evidence, identity, and campaign references. The NPC prompt then
renders task, instructions, and transcript. Spell renders the NPC registry,
catalog, task, instructions, and transcript. Combat renders the NPC registry,
task, instructions, and transcript. Item-event renders task, instructions, and
transcript without a generated-artifact input. NPC interaction renders the names-only NPC
registry, task, instructions, and transcript. The
scene chunker is not an extraction lane: it retains its separate system,
transcript, campaign-reference, task, and instruction order and marks its
transcript and campaign-reference messages ephemeral.
The scene-description extractor deliberately omits the citation-oriented Scriptorium performs prompt rendering, provider execution, and the prompts
`common-dnd-extraction-evidence.md` asset because Notarius attaches the whole structured-output validation. The adapter reports an empty result, validation
accepted chunk range itself. Its manifest is system, shared identity, shared failure, empty structured body, or decode failure as
campaign references, lane task, lane instructions, then the transcript. The `ErrInvalidStructuredOutput`, while retaining the returned raw bytes and debug
identity, campaign-reference, and instruction messages are ephemeral cache material when they exist. Provider failures remain operational errors rather
boundaries; the transcript is last and has no cache control. Compatible shared than output-validation failures.
messages remain canonical shared assets rather than copied package text.
### D&D NPC Normalization Prompt Ordering And Cache Boundaries Prompt-declared repair is executed within Scriptoriums structured-output flow.
The current production D&D prompt manifests set repair attempts to zero. That
setting does not replace pipeline retry behavior: a bindings configured retry
count reruns its stage attempt after an error or rejection, and an exhausted
rejection is a recorded output rather than a provider error. The pipeline owns
attempt lifecycle, validation chains, and retry diagnostics; see
[Pipeline Internals](pipeline.md#validation-retries-and-output) and the
[binding reference](../config.md#module-bindings-and-validators).
NPC normalization has a distinct prompt and response-schema identity from NPC ## Observability And Redaction
extraction. Its stable message tiers are the common D&D system asset, followed
by package-owned task and normalization instructions. Cache boundaries follow
the shared system tier and the package instructions. The variable tail contains
the private candidate-name-and-range input and a windowed transcript input
whose cited units provide local context; neither has a cache boundary because
it changes with the document.
This prompt intentionally omits shared identity guidance, When debug recording is enabled, the pipeline decorates the shared client. The
extraction-evidence, and campaign-reference assets: it reconciles existing wrapper records prepared prompt and response material, timing, selected profile
records rather than extracting events or adding evidence. Its package-owned and model, and call identifiers in the runs debug bundle, including material
manifest and schema identity are fingerprinted separately, so a normalization available from a failed structured completion. For a successful completion, a
prompt or schema change cannot reuse a prior normalization checkpoint. debug-write failure is surfaced; when the completion already failed, its call
error remains the result. Debug-bundle location, retention, and handling are
operational concerns documented in [Operations](../operations.md#debug-bundles).
Shared wording belongs in the canonical assets under Run manifests receive selected profile summaries and component identities, not
`internal/modules/dnd/shared`; extraction packages reference those assets in prompt, schema, source, reference, or response content. Provider error text is
their manifests instead of copying similar text into package-local files. wrapped with prompt context and bearer credentials are redacted before it
Package-local assets contain only lane-specific content. An extraction lane may crosses the runtime boundary. Known-secret redaction is available to other
depart from the tier order only when prompt-quality evidence or a provider runtime collaborators; it does not make prompt or response contents safe for
constraint makes the exception necessary; document the exception and rationale general logging.
here when it becomes implemented behavior.
Schema helpers load embedded JSON Schema with identity and digest metadata, ## Failure Boundaries
return defensive copies, and expose a diagnostics map that omits schema bytes.
The small framework registry contains only generic test schemas; production
schemas remain package-owned.
The spell, NPC, combat, item-event, NPC-interaction, and scene-description extractors' - Construction fails for missing asset registries, mutually exclusive profile
package-owned prompts declare their sources, invalid asset registration, or a non-positive scheduler limit.
structured JSON inputs and private response schemas. Each private response - Preparation failures, unavailable explicit profiles, provider failures, and
schema remains separate from its durable artifact codec schema; this work does context cancellation propagate to the calling stage with context.
not use shared schema fragments or schema generation. Those private schemas own - Malformed or schema-invalid provider output is classified separately as
the transport envelope—required fields, JSON types, nullability, and invalid structured output so the module or pipeline can apply its own retry
unknown-field rejection—while deterministic validators own semantic constraints and rejection policy.
such as enum membership, non-empty values and collections, and positive - Domain semantic checks, evidence decisions, and deterministic normalization
numbers. The spell extractor's prompt declares a required run outside the provider adapter.
`application/json` `spell_catalog` input and an optional `application/json`
`npcs` input. The extractor generates
the catalog input from its prepared
effective catalog as `{"spell_names":[...]}` using sorted canonical names only.
The shared D&D prompt assets include a generic NPC grounding fragment directly
after the campaign reference message for spell, combat, and interaction prompts. When an NPC
registry is bound, the
domain registry boundary strictly decodes and identity-validates one durable
artifact, re-encodes canonical JSON for provenance, and separately generates a
names-only prompt projection. The unbound projection is exactly `{"npcs":[]}`.
Prompt input and component-local checkpoint digests cover the projected bytes;
manifests retain the optional full registry digest/count rather than names,
overlay bytes, registry paths, or source metadata. Combat and interaction prompt,
response-schema, mapping, normalization, identity, and registry-projection
fingerprints remain separate semantic inputs to checkpoint identity.
## Debug And Redaction Boundaries ## Focused Verification
The pipeline may wrap the client with a debug recorder that captures prepared Read the LLM adapter, scheduler, asset registry, schema loader, and redaction
prompt/response material for an explicitly requested debug run. Debug summaries tests when changing this boundary. Prompt changes also require the owning
and manifests receive identities, hashes, usage, and selected profile summaries modules asset tests, and retry or debug changes require focused pipeline or
rather than prompt, source, reference, schema, or response content. CLI coverage. The focused runtime and D&D checks are:
The Scriptorium error wrapper removes bearer credential values from surfaced ~~~sh
provider errors; `RedactSecrets` and `ErrorWithSecretsRedacted` support known go test ./internal/framework/llm/... ./internal/modules/dnd/...
secret values elsewhere in the runtime. Config summaries use a separate ~~~
clone-and-redact path in `internal/core/config`. These mechanisms implement the
security invariant in
[Architecture](../policy/architecture.md#state-output-and-safety); operator
handling of debug data is defined in [Operations](../operations.md#debug).
## Failure Behavior
- Invalid targets, missing prompt IDs, malformed structured output, and
Scriptorium failures return contextual errors to the calling module.
- Scheduler construction rejects non-positive limits; acquisition respects
context cancellation.
- Asset registration rejects invalid roots, missing content, and path conflicts.
- Schema loading distinguishes missing assets, invalid JSON, and invalid
metadata.
- Profile validation errors occur during CLI preparation when an explicit
selected ID cannot be prepared.
## Tests To Inspect
- `internal/framework/llm/scriptorium_client_test.go`: adapter mapping and local
HTTP integration.
- `internal/framework/llm/scheduler_test.go` and
`scheduled_client_test.go`: permits, FIFO behavior, cancellation, and wrapper
release.
- `internal/framework/llm/asset_registry_test.go` and
`schema_registry_test.go`: asset composition, validation, and defensive
copies.
- `internal/framework/llm/secrets_test.go`: provider-error redaction.
- `internal/cli/run_contract_test.go`: profile validation, production client
wiring, manifest recording, and debug integration.
- Module-local `scriptorium_assets_test.go` files: prompt inputs and package
asset registration.

View File

@@ -1,728 +1,99 @@
# Module And Validator Internals # Module Internals
Production module and validator implementations live under their domain-first This guide owns the mechanics for implementing and registering production
trees in `internal/modules`. modules. [Configuration](../config.md) owns selectable keys, binding syntax,
The selectable keys, configuration options, reference slots, and default reference configuration, and default validator chains. Durable input and output
validator chain are canonical in the shapes belong in [integration contracts](../integrations/).
[module](../config.md#implemented-production-modules) and
[validator](../config.md#implemented-production-validators) catalogs in The D&D family has additional shared conventions and domain-specific
Configuration. exceptions. See [D&D Module Internals](dnd.md) rather than adding them here.
## Extension Pattern ## Module Boundary
A stage module package provides a stable key, constructor, contract A module is a typed implementation registered for one pipeline stage. Its
implementation, `ModuleSpec`, `Register`, and focused behavior and registration `ModuleSpec` is the public-to-the-framework declaration of its stable key,
tests. A validator package follows the same pattern with `ValidatorSpec` and the stage, required and provided capabilities, artifact kind, and accepted
validator registry. Package-family registrars compose those leaf registrations reference slots. The framework uses that declaration to resolve a configured
into the production catalog and own family-level policy such as default binding before it builds the implementation.
validator chains and prompt asset collection.
Implementations that accept options must provide both an option validator and
Production input, chunk, output, and D&D spell-, NPC-, combat-, item-event-, interaction-, and scene-description-extract packages a builder. The validator is used while resolving configuration; the builder
register strict option decoders and run-local builders. Preparation decodes their options into decodes the same options and constructs the implementation from the prepared
implementation-owned values and injects dependencies plus the materialized `BuildRequest`. Reject unknown options in both paths. A builder receives only
reference set for the selected target. Each builder receives an isolated clone the dependencies and materialized references that the framework prepared for
of that set; input and output builders receive no references. The spell, NPC, that operation, so it must not re-read configuration or files.
combat, item-event, interaction, and scene-description extractors are typed over the canonical D&D model. D&D validators, merge,
and normalize use typed variants; JSON representation validators use serialized Registry helpers register the typed builder for a stage-specific registry.
requests; and unconditional validators expose separate chunk and typed They are preferable to hand-written untyped registration because they retain
variants. The D&D production registrar registers the canonical typed spell, the artifact type at the framework boundary. Registrars validate the registries
NPC, combat, item-event, interaction, and scene-description implementations, including their kind-specific merge and they need, register each leaf implementation, and add any family-owned assets
normalize behavior. or default validator chains. They return contextual errors so production
composition fails at startup rather than at the first run.
For D&D artifact defaults, generic JSON syntax validation runs first. Rejecting
domain validators then own semantic diagnostics before generic JSON Schema ## Production Composition
validation provides the final rejecting representation backstop; warning-only
relatedness validators run last. This default composition does not reorder an Production composition is intentionally split by family:
explicitly configured validator chain.
- The generic registrar provides the unit chunker, generic JSON validators,
Prepared extractors, extract validators, and codecs may be reused concurrently and JSON output encoder.
by the run-wide extract pool. Production implementations are immutable after - The Seriatim registrar provides the transcript input adapter. Its external
construction: they retain only typed options, immutable assets, or the shared input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
concurrency-safe LLM client. Implementations that introduce mutable state must - The D&D registrar provides its codecs, extractors, mergers, normalizers,
synchronize that state without creating a separate provider scheduler. validators, prompt assets, and default chains. Its behavioral conventions
are documented in [D&D Module Internals](dnd.md).
Specs expose capability and execution metadata without constructing an
implementation. Registry entries separately expose option validation and The CLI owns the composition that invokes these registrars. A module package
run-local construction. Chunk, extract, merge, and normalize modules that accept may register its own family but must not assemble the CLI or make framework
auxiliary material declare identical reference slots from both packages depend on production extensions.
`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
that agreement. Runtime delivery uses the corresponding stage request's ## Adding Or Changing A Module
`References` field.
1. Choose the pipeline stage and the typed artifact boundary. Put external
LLM-backed extensions own their prompt definitions and response schemas under input or durable artifact formats in the relevant integration contract,
package-local embedded assets. Shared filesystem composition belongs in not in this guide or in a private LLM response type.
`internal/framework/promptfs`; reusable D&D prompt fragments, reference 2. Define a stable `ModuleSpec` with the exact capabilities and reference
declarations, prompt-input assembly, and source-unit/citation helpers belong in slots needed for the operation. Model a producer/consumer handoff as an
`internal/modules/dnd/shared`, which owns operation-scoped indexed artifact-compatible slot; configuration then chooses an external file or a
source-reference validation, citation traversal, ordering and canonicalization, generated binding.
plus bounded D&D diagnostics. The 3. Implement strict option decoding, construction, and the typed stage
D&D scene chunker and spell, NPC, combat-turn, item-event, NPC-interaction, and scene-description extractors use ordered interface. Preserve caller ownership: do not retain mutable request data
package-local prompt manifests for both rendering and prompt fingerprinting, so and return defensive copies where an implementation exposes stored data.
only the shared fragments each prompt actually renders participate in either 4. Register the module through its typed registry helper and add it to the
operation. Extraction prompts place stable shared and lane-specific context owning family registrar. Add a default validator chain only when that
before the variable transcript and use shared assets for wording common across family owns the behavior; otherwise require an explicit compatible chain.
lanes. The canonical ordering and cache-boundary policy is documented in 5. Update the selectable-key and chain reference in
[LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries). Stage [Configuration](../config.md#production-module-keys), the applicable
contracts expose only Notarius structured-completion types, not Scriptorium integration contract, and focused tests. Keep the configuration document
public types. as the sole list of production keys and validator order.
The shared `PrepareChunkExtraction` helper owns common extraction preflight and ## Validation And References
transcript material preparation for the spell, NPC, combat-turn, item-event,
NPC-interaction, and scene-description extractors. It validates common request Validators operate on the value produced at their configured stage. A default
state, clones supplied source metadata, falls back to the materialized chunk chain is ordered behavior, not a set: JSON parsing, structural checks,
when content is absent, checks that content remains chunk-identical, and fills domain-specific checks, durable-schema checks, and advisory checks may have
only the common default fields. Extractors retain receiver, dependency, and different responsibilities and failure handling. The active default chains and
lane-specific checks locally and wrap helper errors with their module context. override rules are maintained in
[Configuration](../config.md#production-validator-keys-and-default-chains).
Reference material may inform a module or prompt but must not become source
evidence. The resolver and materializer behavior is described in Reference slots are part of the module specification. They describe the
[Pipeline Internals](pipeline.md#reference-materialization). accepted artifact kind, media type, size, and whether a binding is required;
the framework validates those constraints before construction. An external
## Domain Reference Data reference is materialized during preparation. A generated reference is a
compatible normalized artifact handed from an earlier pipeline step at
### `internal/modules/dnd/spells/catalog` operation time. The configuration reference rules, including precedence and
ordered-handoff requirements, are maintained in
The spell catalog package owns the embedded, versioned D&D 5e 2014 SRD spell [Configuration](../config.md#references-and-ordered-handoffs).
reference data. Its strict JSON asset contains one canonical record per spell,
including spell level and all applicable class memberships. `LoadSRD5E2014` ## Focused Verification
validates catalog identity, provenance metadata, ordering, uniqueness, levels,
classes, aliases, and lookup-key collisions before exposing immutable copies. Exercise the leaf implementation and its registration path when changing a
module. Registry and registrar tests cover duplicate keys, required registries,
Lookup is case-insensitive and normalizes whitespace and common apostrophe and typed construction; pipeline resolution tests cover capabilities, options,
variants while preserving source punctuation in canonical display names. The and reference compatibility. Domain packages should additionally test their
catalog contains 319 unique spells and 779 class memberships. Source and codecs, validators, normalizers, and any integration handoffs they own.
license details live beside the asset in `SOURCES.md`. This domain-owned data is
separate from `internal/modules/dnd/shared`, which is reserved for reusable Run the affected package tests while iterating. The complete module suite is:
prompt and source-reference machinery.
~~~sh
`ResolveEffectiveCatalog` builds the immutable recognition view used by the go test ./internal/modules/...
spell extractor and catalog validator. It starts with the embedded SRD catalog ~~~
and optionally applies one strict JSON overlay from the `spell_catalog` item in
a materialized reference set. Overlay catalogs are ordered by ID, may add names
and aliases, and may augment an existing canonical spell without replacing its
display name. Cross-spell lookup collisions are errors. The effective view
exposes sorted canonical names, normalized lookup, overlay identities, and a
semantic digest; overlay content remains contextual reference material rather
than source evidence. Its external JSON contract is defined in the
[spell-catalog overlay contract](../integrations/dnd-spell-catalog-overlays.md).
### `internal/modules/dnd/npcs/identity`, `internal/modules/dnd/npcs/registry`, and `internal/modules/dnd/codec/npcs`
The NPC identity package owns Unicode comparison keys, deterministic
`npc:sha256:` IDs, display normalization, and whole-registry collision issues.
The registry package resolves one optional normalized artifact through the
strict codec, validates whole-registry identity, canonicalizes its JSON, and
provides immutable records, a names-only prompt projection, distinct durable
and projection digests, count, and exact canonical-name lookup. External files cross this boundary during
preparation; generated artifacts cross it at the ordered step handoff. It owns
the `npcs` slot and its bounded, content-safe validation failures. NPC source
references are durable provenance and are not treated as evidence for a
consuming pipeline. The NPC codec owns the strict durable `dnd/npc-list` JSON
boundary and exposes candidate versus approved encode/decode operations. The
shared `internal/modules/dnd/codec/candidatejson` package supplies strict typed
candidate JSON mechanics; each artifact codec retains its own durable schema
and approved-value policy.
### `internal/modules/dnd/scenedescriptions/registry`
The scene-description registry owns the required `scene_descriptions` control
reference used by combat extraction. It decodes exactly one approved scene-list
artifact through the scene-description codec and retains only scene ID, exact
source reference, and kind. Titles, summaries, original bytes, paths, and
prompt material do not cross this domain boundary.
An external reference is validated during preparation; an unbound seed is
permitted only while a configured generated reference awaits the ordered
handoff. At operation time, a generated artifact overrides the seed and is
resolved into an immutable view safe for concurrent extract jobs. Matching is
strictly exact by chunk ID, source ID, start unit ID, and end unit ID, producing
an exact, missing, or mismatched result. Only an exact result exposes kind.
The registry's semantic eligibility digest is derived from a sorted projection
of ID, exact range, and kind. It ignores titles, summaries, and input order;
the unbound view has a stable empty projection digest. Combat extractor
metadata and checkpoint identity use this semantic boundary for external
references, while generated artifact identity and dependencies remain owned by
the framework handoff.
The `internal/modules/dnd/codec/combatturns` package owns the durable
`dnd/combat-turn-list` schema and candidate versus approved JSON boundary. It
is registered by the production D&D family registrar for the selectable combat
lane.
The `internal/modules/dnd/codec/itemevents` package owns the durable
`dnd/item-event-list` schema and candidate versus approved JSON boundary. It is
registered by the production D&D family registrar. Its external contract is
defined in the [D&D item-event artifact contract](../integrations/dnd-item-event-artifacts.md).
The `internal/modules/dnd/codec/npcinteractions` package owns the durable
`dnd/npc-interaction-list` schema and candidate versus approved JSON boundary.
It is registered by the production D&D family registrar for the selectable
interaction lane. Its external contract is documented in the
[D&D NPC interaction artifact contract](../integrations/dnd-npc-interaction-artifacts.md).
The `internal/modules/dnd/codec/scenedescriptions` package owns the durable
`dnd/scene-description-list` schema and candidate versus approved JSON boundary.
It is registered by the production D&D family registrar. Its external contract
is documented in the
[D&D scene-description artifact contract](../integrations/dnd-scene-description-artifacts.md).
## Input Adapter
### `internal/modules/seriatim/input/transcript`
The adapter decodes the supported transcript JSON, selects the source identity,
computes canonical source provenance, validates segments, and maps each segment
into a generic source unit with a self-reference plus speaker and timestamp
metadata. It accepts no module options. Its spec advertises the transcript
capabilities consumed by D&D modules.
Parsing is strict about required values and duplicate unit IDs but deliberately
ignores unrelated Seriatim fields. The external format and derived-identity
rules are defined in the
[Seriatim contract](../integrations/seriatim.md).
## Chunkers
Chunkers implement `contracts.Chunker.Plan`. A plan identifies ordered source
unit ranges and may carry optional namespaced JSON annotations; it does not
contain materialized chunk content. The framework canonicalizes annotations,
validates ranges against the current source, and materializes chunk IDs,
indexes, references, content, units, and generic metadata. Materialized source
unit metadata is independently owned. Annotation
namespaces remain optional data: generic framework code and downstream modules
must not require D&D scene annotations or import `dnd/scenes`.
### `internal/modules/generic/chunk/units`
The generic chunker validates the source document and returns ranges over units
in configured windows. Overlap changes the next window start but never reorders
units. Framework materialization derives the resulting chunk identity and
generic metadata from those ranges.
The accepted options and defaults are defined in
[Configuration](../config.md#implemented-production-modules). Generic
framework validation canonicalizes the returned unit slices before extraction.
The chunker decodes its options during construction and retains only the typed
window settings used by `Plan`.
### `internal/modules/dnd/chunk/scenes`
The scene chunker prepares a structured Scriptorium request from the full
transcript, session, and optional D&D reference inputs. It validates the model's
inclusive source-unit endpoints against document position and converts them
into deterministic plan ranges. Preparation injects the shared structured LLM
client into the chunker; `Plan` supplies only the run-specific profile, session,
source, references, and metadata.
Scene validation requires sequential, contiguous, non-overlapping coverage from
the first source unit through the last. Its private response contains only the
boundary endpoints; the accepted plan has no D&D-specific annotations and
produces no boundary warnings. Malformed structured output is returned as an
error; there is no fallback chunker.
The package embeds its prompt and response schema and reports their non-secret
identity and hashes through singleton module metadata. Shared D&D assets supply
reference declarations and prompt inputs; their user-facing keys and accepted
file types remain canonical in [Configuration](../config.md).
## Extractor
### `internal/modules/dnd/extract/spells`
The spell extractor prepares a structured request from one chunk, the
chunk-scoped source input, the session, and optional D&D reference inputs. It
decodes the model response, assigns the generic source identity to every source
reference, canonicalizes duplicate references, orders spell casts by their
earliest valid source-document position, and returns `dnd.SpellList`.
Its private response schema admits only the structural transport envelope:
required fields, JSON types, array and object shapes, and unknown-field
rejection. It maps integer source-unit candidates directly without repairing
semantic values, so the deterministic shape, catalog, and source-reference
validators own blank values, empty evidence, and invalid or unresolved ranges.
The extractor owns its private model-response DTO, embedded prompt, LLM response
schema, strict option decoder, injected shared LLM client, and prompt/schema
manifest metadata. During preparation it resolves the optional `spell_catalog`
reference into an immutable effective catalog and adds a generated
canonical-name-only JSON input to every structured completion request. Overlay
failures therefore stop construction before source parsing or an LLM call;
campaign references remain separate disambiguation inputs and never become
source evidence.
The prompt includes only actual casting events and unambiguous declared casting
attempts. Spell mentions, plans, rules discussion, and catalog matches without
a casting event are excluded. Shared extraction-evidence and identity rules
require transcript-supported caster and spell facts, while the catalog,
campaign references, and NPC names only disambiguate source text. Structural
source validation remains deterministic; semantic evidence sufficiency is
enforced through extraction policy and evaluation.
Both the extractor and deterministic catalog validator expose
the effective base-plus-overlay semantic digest as scoped prepared-component
checkpoint identity. Raw overlay provenance independently covers file-byte
changes, while the semantic digest also invalidates reuse when the embedded
catalog or catalog composition changes. The extractor additionally fingerprints
its complete prompt assets and private response schema, so either semantic
contract changing invalidates previously recorded extraction checkpoints. The
separate `internal/modules/dnd/codec/spells` package
owns the durable schema and stable JSON representation for artifact kind
`dnd/spell-list`. The runner keeps the result typed through validators and later
stages, using the codec only for checkpoint, debug, and output boundaries.
Shared D&D helpers keep prompt input names and source-unit reference conversion
consistent with the scene chunker.
The extractor also declares the optional `npcs` registry slot and consumes the
immutable registry boundary from `internal/modules/dnd/npcs/registry`. An
external registry is prepared before execution; a generated registry is
validated and supplied at operation time. Bound external registries add only
the full `npc_registry_digest` and `npc_count` to module metadata. The local
`npc_registry` checkpoint fingerprint always covers the names-only projection,
including its exact unbound value. Generated bindings are represented by
framework handoff provenance and dependency fingerprints. The unbound prompt
input is exactly `{"npcs":[]}` and has no registry provenance.
The shared NPC grounding fragment is placed immediately after the common
campaign reference message and is included in the spell prompt fingerprint.
The durable payload and manifest metadata shapes are defined in the
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
### `internal/modules/dnd/extract/npcs`
The NPC extractor maps private model output to the canonical `dnd.NPCList`,
assigns source identity and deterministic NPC IDs, and preserves source
references for deterministic validation. It uses the shared campaign
references only for disambiguation and does not consume the optional NPC
registry slot. Its prompt and private response schema are package-owned. The
private response contains only a name and model-facing evidence ranges for each
record; anonymous groups, generic roles, invented labels, descriptions,
aliases, and relationships are outside its contract. The
prompt follows the shared D&D extraction ordering and cache policy documented
in [LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries).
The private response schema owns only structural transport validation and maps
integer source-unit candidates unchanged. Required semantic content, non-empty
evidence, and valid source ranges are rejected by the deterministic shape and
source-reference validators.
### `internal/modules/dnd/extract/scenedescriptions`
The scene-description extractor makes one structured completion for each
accepted chunk and maps its private `kind`, `title`, and `summary` response to
one `dnd.SceneDescription`. It assigns the current chunk ID and exact range,
preserves kind without repair, and trims only title and summary whitespace.
Optional players, party, and glossary references can disambiguate prompt terms
but do not supply evidence. The package owns its private schema, prompt assets,
and mapping fingerprint; deterministic validators own the durable semantic
checks. The durable contract is defined in the
[D&D scene-description artifact contract](../integrations/dnd-scene-description-artifacts.md).
### `internal/modules/dnd/extract/combatturns`
The combat extractor requires the `scene_descriptions` reference and resolves
it through the immutable scene-description registry before it resolves NPC
grounding or constructs prompt inputs. It calls the LLM only for an exact
current-chunk match whose kind is `combat`. Exact `narrative`, `recap`, and
`meta` matches return an accepted empty `dnd.CombatTurnList`; missing or
mismatched coverage returns the same result with one bounded unavailable-
classification warning. These deterministic results do not consume retry
attempts. Scene descriptions are control context only and are not passed to the
combat prompt or copied into combat evidence.
For eligible chunks, the extractor prepares one structured request using the
shared extraction-evidence, identity, campaign-reference, NPC-grounding, and
transcript prompt inputs. It maps the private response to
`dnd.CombatTurnList`, assigns the current source identity, removes exact
duplicate source ranges, and orders turns by valid source-document position
while preserving malformed candidate fields for deterministic validators. Its
package-owned private response schema enforces only the structural JSON
envelope; semantic artifact constraints remain with the validator chain.
Prepared metadata and checkpoint fingerprints include prompt, response-schema,
mapping, and scene-gate identities. An external scene reference additionally
reports its semantic eligibility digest and count; generated identity remains
framework handoff provenance and dependency state. Neither surface retains
scene prose or payload bytes. The prompt follows the shared D&D extraction
ordering and cache policy documented in
[LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries). The
package exposes typed registration and is included in the production D&D
registrar with the default combat extraction chain.
The combat normalizer accepts only the optional structured NPC registry.
Campaign references remain extractor-only LLM context and are not materialized
for deterministic normalization.
### `internal/modules/dnd/extract/itemevents`
The item-event extractor prepares one structured request from the accepted
chunk and optional campaign references, then maps private records to
`dnd.ItemEventList` with the current source identity. It declares only optional
`glossary`, `party`, `players`, and deprecated `roster` reference slots; these
can disambiguate names but never supply evidence. It has no NPC,
scene-description, or item-registry dependency.
The private response schema owns structural transport validation. The extractor
preserves candidate category, holder, quantity, and source-range values for the
deterministic validators, removes exact duplicate ranges, and source-orders
events. The source-reference validator requires citations to fit the current
accepted chunk. Prompt, response-schema, and mapping identities participate in
checkpoint identity. The durable schema is owned separately by
`internal/modules/dnd/codec/itemevents`.
### `internal/modules/dnd/extract/npcinteractions`
The NPC interaction extractor requires the structured `npcs` registry slot. It
uses the registry's names-only prompt projection with shared extraction
evidence, identity, and transcript material, then maps private model records to
`dnd.NPCInteractionList` with the current source identity. Registry source
references are never reused as interaction evidence. The private response
schema carries only name, bounded interaction kind, and source-unit ranges;
deterministic validators own registry membership, source validity, and
relatedness. Extract-stage source validation additionally requires every cited
range to be wholly contained in the current materialized chunk. Prompt, schema,
mapping, and the names-only registry projection
participate in checkpoint identity, while generated producer identity remains
framework provenance.
The shared D&D source-reference order defines canonical evidence ordering. The
domain-owned `internal/modules/dnd/npcinteractions` package defines occurrence
ordering, valid-evidence eligibility, and collision-safe exact identity. The
interaction normalizer and normalized invariants validator consume those
rules, so their production and checking paths cannot drift. Normalizer and
relatedness warning lists use the shared D&D diagnostic cap and emit a final
omission-summary warning when truncated.
### `internal/modules/dnd/normalize/npcs`
The NPC normalizer deterministically trims display names, recomputes IDs,
canonicalizes evidence, and consolidates equal comparison keys before semantic
work. Records are eligible for the document-level identity call only when they
have a non-empty comparison key and wholly valid current-document references.
It sends private candidate names and source ranges plus coalesced, cited
transcript windows to its own prompt; stable NPC IDs and the durable artifact
shape are not prompt inputs.
The private structured response proposes groups of supplied names and a
canonical supplied name. Deterministic comparison-key resolution validates each
group, discards unsafe or overlapping groups, and independently applies safe
ones. Application preserves earliest record order, unions canonical evidence,
and derives the final canonical ID. Invalid structured output and discarded
groups request framework retry with a safe fallback; bounded diagnostics become
durable only on final fallback exhaustion.
The normalizer records prompt and response-schema identities and digests,
identity and normalization policies, and semantic-context policy and radius as
manifest metadata. Its local checkpoint fingerprints cover the prompt, response
schema, identity policy, normalization policy, and semantic-context policy so a
meaningful behavior change invalidates prior normalize reuse.
## Merger And Normalizer
### `internal/modules/generic/merge/appendorder`
The merger passes typed values to an injected combine function in framework
source-chunk order. The D&D registrar specializes it for all six artifact
lists; each append merger preserves collection presence and order while giving
the result independently owned nested source-reference slices.
### `internal/modules/generic/normalize/noop`
The normalizer returns the merged domain value unchanged and is reusable for
any registered artifact type.
### `internal/modules/dnd/normalize/spells`
The typed spell normalizer resolves the optional `spell_catalog` reference into
the same immutable SRD-plus-overlay effective catalog used by spell extraction
and catalog validation. It performs no LLM calls. For each spell cast it
canonicalizes recognized names using the catalog's case, whitespace,
apostrophe, and alias rules; canonicalizes source references with the shared
document-aware order; removes only exact reference duplicates; and emits
bounded, scoped warnings for each mutation or unresolved name.
After those per-cast changes, it collapses only casts with the same canonical
spell, case-folded and whitespace-normalized caster, and complete non-empty
valid source-reference set. It retains the first occurrence and its caster,
source references, and stable order. Unknown names, empty or invalid evidence,
and adjacent or overlapping but different ranges remain unchanged for
validation.
The normalizer exposes the effective catalog digest as its independently scoped
`effective_catalog` checkpoint fingerprint and reports catalog base ID, digest,
and overlay IDs as manifest metadata. Catalog contents, reference paths, and
raw overlay bytes are not included in either surface. The normalize-stage
reference is stage-local, so an overlay-capable pipeline binds the catalog
independently for extraction and normalization.
### `internal/modules/dnd/normalize/combatturns`
The combat normalizer prepares an external NPC registry before execution or
receives a generated registry at the ordered step handoff, then uses the
immutable view during runtime. It display-normalizes actors,
rewrites canonical-name matches for actors, orders and deduplicates source
references, stable-sorts records by source-document position, and collapses
only exact duplicate identities with fully valid evidence. It deep-clones
output storage and emits bounded warnings scoped to merged input indexes. Its
metadata and fingerprints identify the normalization and NPC identity policies.
External bindings may contribute registry
digest/count metadata; generated identity is retained in framework provenance
and dependency fingerprints. The normalizer is included in the production D&D
registrar with the default combat normalization chain.
### `internal/modules/dnd/normalize/itemevents`
The item-event normalizer accepts no options or references and makes no LLM
calls. It trims display-edge whitespace in names and holders, canonicalizes
source references, source-orders events, and collapses only exact duplicates
with complete valid evidence. It does not create a ledger, calculate balances,
resolve aliases, infer quantities or holders, or reconcile nearby events. Its
policy fingerprint and bounded warnings identify deterministic normalization;
the matching invariant validator checks the resulting order and duplicate rule.
### `internal/modules/dnd/normalize/npcinteractions`
The interaction normalizer requires the same immutable NPC registry. It
canonicalizes exact registry-name matches, orders and de-duplicates source
references, stable-sorts occurrences by source-document position, and collapses
only exact interaction identities with valid evidence. It does not infer,
merge, or summarize distinct occurrences. Its metadata and fingerprints expose
the normalization and NPC identity policies; generated registry identity stays
in framework provenance and checkpoint dependencies.
### `internal/modules/dnd/normalize/scenedescriptions`
The scene-description normalizer has no options or references. It validates
each source range against the source document, trims title and summary
whitespace, orders records by source position then ID, removes only exactly
identical records, and rejects conflicting reused IDs or ranges. Its policy
fingerprint identifies this deterministic behavior; the matching invariant
validator checks the normalized result in the production chain.
## Output Encoder
### `internal/modules/generic/output/json`
The JSON encoder sorts normalized results by lane, derives collision-checked
safe logical names, pretty-prints JSON payloads, and assembles the logical index,
manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported
media types, unsafe names, and sanitized-name collisions are errors.
Its strict `include_chunk_map` option is disabled by default. When enabled, it
validates the framework-supplied accepted chunk map through its codec and adds
the pipeline-wide `chunk-map.json` plus its index descriptor; it does not treat
the map as a lane payload. The external shape is owned by the
[Accepted Chunk Map contract](../integrations/chunk-map.md).
The encoder returns logical files only. The CLI places them on disk, and the
[JSON output contract](../integrations/json-output.md) defines their external
paths and schemas.
## Generic Validators
The generic validator implementations live under
`internal/modules/generic/validate`.
The unconditional accept and reject validators provide explicit chunk and
typed-artifact variants used primarily for controlled composition and tests.
The serialized JSON syntax validator uses `encoding/json` to reject malformed
representation bytes. The serialized JSON Schema validator requires schema
bytes, parses the instance and schema with `jsonschema`, and distinguishes
payload rejection from schema loading or compilation errors. The framework
serialized-validation request carries either canonical chunk bytes or artifact
codec bytes according to its target context. Neither validator calls the LLM.
## D&D Spell Validators
All four validators receive `dnd.SpellList` directly. The shape validator
rejects a missing list, blank caster or spell names, and empty reference lists.
The catalog validator defers when shape is invalid, then checks every non-empty
spell name against the immutable effective SRD and overlay catalog. It accepts
normalized canonical names and aliases without rewriting the artifact; unknown
names reject the complete result with bounded, stable index/name diagnostics. The
source-reference validator defers malformed shapes, validates every cited
range, and reports all range defects through a bounded aggregate while
preserving `invalid_source_refs`. The relatedness validator resolves all cited
ranges through the shared document-order traversal, then warns when a normalized
consecutive spell-name token sequence is absent from the cited source text.
Invalid shape
or cited ranges produce no relatedness warnings; the shape and source-reference
validators own those defects.
These validators are deterministic. Shape, source-reference, and relatedness
each expose a local semantic `policy` checkpoint fingerprint. The catalog
validator instead exposes its effective catalog digest as its semantic
checkpoint identity and does not add a separate policy fingerprint. Their
selectable keys and production order are defined in
[Configuration](../config.md#implemented-production-validators); their durable
payload rules are defined in the
[artifact contract](../integrations/dnd-spell-artifacts.md).
## D&D NPC Validators
NPC shape validation checks the required ID and name strings, list presence, and source-reference
shape. The source-reference validator defers malformed shapes, checks
current-document identity, unit existence, and range ordering, and reports all
defects through bounded aggregates. Source relatedness uses the shared
document-order traversal and normalized consecutive-token matching, emitting at
most one bounded warning per record when the canonical name does not occur near
its cited text. Invalid shape or cited ranges produce no relatedness warnings.
Normalize identity validation checks deterministic IDs, canonical names, and
duplicate canonical-name or ID ownership.
All are deterministic and expose the policy fingerprints used by the
production chains.
## D&D Combat Validators
Combat shape validation owns the required list, actor, supported turn kind, and
non-empty source-reference collection. Combat source-reference validation defers invalid
shape, checks source identity, unit existence, and range order, and reports all
defects through bounded aggregates. Combat source-relatedness defers invalid
shape or ranges, uses the shared traversal to combine overlapping cited units
in document order, and emits at most one bounded advisory warning per turn for
an unrelated actor. Actors use normalized consecutive-token matching. The
normalized-invariants validator owns actor display normalization, canonical
source-reference order, chronology, and exact duplicate identity; it defers
shape and source-reference failures. All four validators are deterministic and
expose local policy fingerprints. In the registered defaults, JSON syntax runs
first; combat shape, normalized invariants when applicable, and source-reference
validation precede JSON Schema validation; warning-only relatedness runs last.
## D&D Item-Event Validators
Item-event shape validation owns the required list, non-empty name, supported
category, category-and-holder combination, positive optional quantity, and
non-empty source-reference collection. Source-reference validation defers
malformed shapes, checks current-source identity and ordered ranges, and during
extraction requires every citation to fit the accepted chunk. Relatedness is
advisory and warning-only: it checks the event name against cited transcript
text while deferring malformed candidates and invalid ranges to their blocking
owners. The normalized-invariants validator owns display normalization,
canonical source-reference order, chronology, and exact duplicate identity.
All four validators are deterministic and expose policy fingerprints. The
registered chains run syntax and blocking checks before durable JSON Schema;
relatedness remains last.
## D&D NPC Interaction Validators
Interaction shape validation owns the required list, registry name, supported
kind, and non-empty source-reference collection. Registry validation checks
exact membership in the required immutable NPC registry. Source-reference and
relatedness validation use the current transcript only; malformed candidates
are deferred by later validators and produce no relatedness warning. The
normalized-invariants validator owns canonical registry names, source-reference
order, chronology, and exact duplicate identity. The production chains run
shape, registry, and source-reference checks before JSON Schema validation;
relatedness remains warning-only and last.
## D&D Scene Description Validators
Scene-description shape validation owns the non-empty list, trimmed ID and
prose, closed kind, and basic source-reference shape. Extract-stage source
validation additionally requires the one record to attach exactly to the
current accepted chunk; later source validation checks source membership.
Relatedness checks the title and summary independently against only their cited
transcript range and emits bounded advisory warnings. The normalized-invariants
validator owns ordering, exact duplicate elimination, and conflicting ID or
range detection. The production chains run shape and source-reference checks
before JSON Schema validation; the warning-only relatedness check is last.
## Production Registration
Production composition occurs through family registrars. The CLI allocates one
complete framework registry set and one LLM asset registry. It invokes
`internal/modules/generic/register`,
`internal/modules/seriatim/register`, and `internal/modules/dnd/register` in
that order, then exposes the matching catalog for resolution. The generic and
Seriatim registrars own their production leaf registrations. The D&D registrar
owns D&D leaf registrations, typed spell, NPC, combat, item-event, interaction, and scene-description default-validator
chains, typed append-order specializations, and D&D prompt/schema asset
collection. Its registration helpers group module, validator, prompt-asset, and
chain composition while retaining artifact-specific merge and clone behavior in
the registrar.
Concrete implementation packages do not import generic implementation
packages directly. A concrete family's `register` package is its composition
point for specializing reusable generic implementations, while the generic
registrar composes only generic children.
Core and framework production packages do not import production extensions.
CLI production code is the sole application composition root for extensions
and imports only exact family registrar packages. Other production packages,
including commands and newly introduced package trees, do not import module
packages directly. Compatibility tests in the CLI, core, and framework trees
may import roots and implementation leaves directly. Other non-module tests do
not receive that exemption. White-box tests within module families retain the
production family boundaries. `internal/modules/integration` is test
infrastructure: its black-box tests may compose multiple families, but it is
not a production module family or production dependency target.
## Adding An Extension
When adding a production module or validator:
1. implement the stage or validator contract and package-local key;
2. expose and test its spec, constructor, and registration function;
3. keep format or domain parsing inside the concrete package;
4. add package-owned prompt/schema assets when the extension is LLM-backed;
new LLM-backed D&D extraction modules must follow the stable-to-variable
prompt ordering, shared-asset ownership, and cache-boundary policy in
[LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries), or
document the implemented exception and its evidence there;
5. register it through its package-family registrar and add a default chain
there only when production policy requires one;
6. add resolution and composition coverage for capabilities, options,
references, and validation behavior;
7. update the selectable-key catalog in [Configuration](../config.md), the
relevant external contract, this inventory, and maintained examples when
user-visible behavior changes.
Do not add the extension to `docs/development.md`; that file routes by task and
does not inventory implementations.
### D&D Extractor Contract
New D&D extractors preserve these package-owned responsibilities:
- Reject unknown options unless an option namespace is intentionally
extensible, and use shared common preflight while retaining receiver,
dependency, and lane-specific checks locally.
- Return independently owned results and exposed metadata that callers may
safely mutate.
- Keep the private response DTO, structural response schema and its identity,
provider-response mapping, durable artifact conversion, and lane diagnostics
in the owning package.
- Include every stable semantic input that can change durable output in
checkpoint identity. Consider prompt, schema, mapping, canonicalization,
prepared reference projections, identity, normalization, and trimming where
applicable.
- Add focused behavioral coverage where the lane's risks warrant it, including
construction and registration, option rejection, preflight, provider
failures, structured decoding, mapping and ownership, prompt
role/input/cache order, and checkpoint invalidation.
Prompt ordering, shared-asset ownership, cache boundaries, and private-schema
rules are defined in [LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries).
[Pipeline Internals](pipeline.md#reference-materialization) owns reference
materialization, and its [checkpoint hooks](pipeline.md#checkpoint-and-debug-hooks)
define checkpoint behavior. Follow [Architecture](../policy/architecture.md#source-and-domain-boundaries)
for ownership boundaries and the [Testing Policy](../policy/testing.md) when
selecting durable coverage. This contract intentionally does not prescribe
prompt prose or length, hashes, test counts, filenames, fixture layouts, or
generic implementation builders.
## Tests To Inspect
- Package-local `*_test.go` files under the module or validator being changed.
- `internal/framework/pipeline/typed_resolution_test.go`: typed registry, spec,
and heterogeneous artifact composition.
- `internal/framework/pipeline/profile_test.go`: framework binding defaults and
profile resolution.
- `internal/cli/production_contract_test.go`: production catalog, config
resolution, and composition smoke coverage.
- `internal/cli/example_contract_test.go`: maintained example ownership.
- `internal/framework/promptfs/*_test.go` and
`internal/modules/dnd/shared/*_test.go`: shared prompt and reference assembly.
- `internal/modules/integration/*_test.go`: black-box composition across
production extension domains.

View File

@@ -1,191 +1,58 @@
# Internal Overview # Internal Overview
This document inventories the implemented Notarius components. Normative This document is the implemented component map for Notarius. Normative
boundaries and dependency direction belong in boundaries and dependency direction belong in
[Architecture](../policy/architecture.md); external behavior belongs in the [Architecture](../policy/architecture.md). User and operator contracts belong
[CLI](../cli.md), [Configuration](../config.md), in the [CLI](../cli.md), [Configuration](../config.md),
[Operations](../operations.md), and [integration contracts](../integrations/). [Operations](../operations.md), and [integration contracts](../integrations/).
## Execution Path ## Execution Path
`cmd/notarius` delegates to `internal/cli`, the production composition root. ~~~
The CLI loads configuration, builds the production catalogs and runtime cmd/notarius -> internal/cli -> configuration and production composition
collaborators, invokes `internal/framework/pipeline`, and places the logical -> internal/framework/pipeline -> logical output files
output files returned by the runner. Cache and debug collaborators are supplied -> internal/cli -> durable output and optional state/debug data
at this boundary. ~~~
Resolution produces a fixed ordered workflow of steps and globally unique, The CLI is the application boundary: it discovers configuration, composes
sorted artifact lanes. Preparation constructs the complete module and validator production registries and runtime collaborators, invokes the framework, and
set before the runner receives source bytes. Source parsing and chunking are places returned files. The framework resolves and prepares a fixed extraction
serial. Each step then uses a bounded run-wide extraction pool followed by pipeline, then returns logical results without owning process behavior or
serial per-lane merge and normalize continuations. A step barrier prevents physical state roots.
later consumers from starting until all earlier lanes are terminal and their
required normalized artifacts have crossed the typed handoff.
## Application Boundary ## Components
| Package | Implemented responsibility | | Area | Implemented owners | Responsibility |
| --- | --- |
| `cmd/notarius` | Executable entry point and process exit delegation. |
| `internal/cli` | Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, state collaborator setup, durable writes, and user-facing results. |
## Core Packages
| Package | Implemented responsibility |
| --- | --- |
| `internal/core/artifacts` | Run-manifest and provenance models. |
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
| `internal/core/debugbundle` | Explicit per-run debug-bundle allocation and redacted summary writing. |
| `internal/core/fileio` | Generic confined atomic file and JSON writes with caller-selected permissions. |
| `internal/core/source` | Generic source documents, units, chunks, canonical references, validation, deterministic source digests, and independent metadata materialization. |
## Framework Packages
| Package | Implemented responsibility |
| --- | --- |
| `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. |
| `internal/framework/pipeline` | Module and artifact-codec registries, ordered-step and generated-reference resolution, option validation, profile resolution, capability checks, external reference materialization, complete pipeline preparation, typed handoff, retries, orchestration, warnings, checkpoint decisions, and manifest population. |
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
| `internal/framework/checkpoint` | Root-based checkpoint loading, recording, identity, and payload serialization. |
| `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. |
| `internal/framework/chunkmap` | Strict durable accepted chunk-map construction, schema, validation, cloning, and serialization. |
| `internal/framework/debug` | Root-based framework and LLM debug recording. |
Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
serialized-validator, and
typed-validator interfaces. The runner owns handoff provenance, validation
sequencing, rejection handling, checkpoint and debug boundaries, and final
manifest assembly.
Artifact registries support heterogeneous typed extraction entries and
kind-specific merger, normalizer, and validator variants. Resolution derives a
lane's kind from its extractor, requires the matching codec, verifies exact Go
type equality across the lane, and records schema identity in the resolved lane
and pipeline digest. Registry entries carry separate option-validation and
run-local construction closures. Preparation injects shared dependencies and
constructs input, chunk, validators, ordered lanes, and output before source
parsing. Production modules use strict construction-time option decoding, and
LLM-backed modules retain the injected shared client. The D&D family registers
the canonical `dnd/spell-list`, `dnd/npc-list`, `dnd/combat-turn-list`,
`dnd/item-event-list`, `dnd/npc-interaction-list`, and
`dnd/scene-description-list` codecs, typed spell, NPC, combat, item-event,
interaction, and scene-description extractors and normalizers, validators,
plus kind-specific generic merge strategies; generic JSON validators use the
serialized-validation contract. The runner executes lanes through
private exact-type-checked closures, coordinates extract results independently
of completion timing, and serializes artifacts only through their codec at
checkpoint, debug, and output boundaries.
## Production Extensions
The canonical catalogs of user-selectable
[module](../config.md#implemented-production-modules) and
[validator](../config.md#implemented-production-validators) keys are in
Configuration. The implemented module packages are:
| Package | Implemented responsibility |
| --- | --- |
| `internal/modules/seriatim/input/transcript` | Parses the supported Seriatim transcript format into the generic source model. |
| `internal/modules/generic/chunk/units` | Splits ordered source units by unit count and overlap. |
| `internal/modules/dnd/chunk/scenes` | Produces contiguous D&D scene chunks from structured model output. |
| `internal/modules/dnd` | Owns the canonical D&D spell-list, spell-cast, NPC-list, NPC, combat-turn-list, combat-turn, item-event-list, item-event, NPC-interaction-list, and scene-description-list artifact types. |
| `internal/modules/dnd/codec/spells` | Strictly decodes and stably encodes the durable D&D spell-list representation. |
| `internal/modules/dnd/codec/npcs` | Strictly decodes and stably encodes the durable D&D NPC-list representation. |
| `internal/modules/dnd/codec/combatturns` | Strictly decodes and stably encodes the durable D&D combat-turn-list representation. |
| `internal/modules/dnd/codec/itemevents` | Strictly decodes and stably encodes the durable D&D item-event-list representation. |
| `internal/modules/dnd/codec/npcinteractions` | Strictly decodes and stably encodes the durable D&D NPC-interaction-list representation. |
| `internal/modules/dnd/codec/scenedescriptions` | Strictly decodes and stably encodes the durable D&D scene-description-list representation. |
| `internal/modules/dnd/extract/spells` | Maps private structured model output to canonical source-grounded D&D spell lists. |
| `internal/modules/dnd/extract/npcs` | Maps private structured model output to canonical source-grounded D&D NPC lists. |
| `internal/modules/dnd/extract/combatturns` | Uses exact scene eligibility to select combat chunks, then maps private structured model output to source-grounded D&D combat-turn candidates. |
| `internal/modules/dnd/extract/itemevents` | Maps private structured model output to source-grounded D&D item-event candidates. |
| `internal/modules/dnd/extract/npcinteractions` | Maps private structured model output to current-source NPC interaction candidates grounded by a required registry. |
| `internal/modules/dnd/extract/scenedescriptions` | Maps one private scene description to the current accepted chunk's ID and exact range. |
| `internal/modules/dnd/npcinteractions` | Owns interaction occurrence ordering, valid-evidence checks, and exact interaction identity shared by normalization and invariant validation. |
| `internal/modules/dnd/normalize/combatturns` | Canonicalizes and orders merged combat turns, applies exact NPC identity matches, and collapses only exact valid-evidence duplicates. |
| `internal/modules/dnd/normalize/itemevents` | Trims, source-orders, and removes only exact valid-evidence item-event duplicates. |
| `internal/modules/dnd/normalize/npcinteractions` | Canonicalizes required-registry names, orders interaction occurrences, and collapses only exact valid-evidence duplicates. |
| `internal/modules/dnd/normalize/scenedescriptions` | Trims, source-orders, and removes only exactly identical scene descriptions while rejecting ID and range conflicts. |
| `internal/modules/dnd/validate/combatturns` | Provides deterministic shape, source-reference, source-relatedness, and normalized-invariant validation for the production combat chains. |
| `internal/modules/dnd/validate/itemevents` | Provides deterministic shape, source-reference, source-relatedness, and normalized-invariant validation for item-event chains. |
| `internal/modules/dnd/validate/npcinteractions` | Provides deterministic shape, registry, source-reference, source-relatedness, and normalized-invariant validation for interaction chains. |
| `internal/modules/dnd/validate/scenedescriptions` | Provides deterministic shape, exact extraction attachment, source-relatedness, and normalized-invariant validation for scene-description chains. |
| `internal/modules/dnd/npcs/registry` | Resolves validated normalized NPC references into immutable grounding data and exact identity lookup. |
| `internal/modules/dnd/scenedescriptions/registry` | Resolves approved scene descriptions into immutable exact-match combat eligibility data without retaining scene prose. |
| `internal/modules/dnd/npcs/identity` | Owns Unicode-aware NPC identity, ID derivation, and registry collision validation. |
| `internal/modules/dnd/spells/catalog` | Embeds and validates the versioned D&D 5e 2014 SRD catalog, composes optional overlays, and provides immutable effective lookup. |
| `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. |
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
| `internal/modules/dnd/normalize/spells` | Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity. |
| `internal/modules/dnd/normalize/npcs` | Deterministically prepares and safely applies document-level LLM-assisted NPC identity consolidation, preserving canonical evidence, order, and diagnostics. |
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, rejections, and an explicitly enabled accepted chunk map as logical JSON files. |
`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
reference declarations, prompt input assembly, document-aware source-reference
ordering and canonicalization, and bounded diagnostics under
`internal/modules/dnd/shared/diagnostics`.
The shared NPC grounding fragment is mounted for D&D prompts and is owned by
this package. Domain-neutral prompt filesystem composition lives in
`internal/framework/promptfs`.
The `dnd/npcs/registry` package owns the optional `npcs` registry boundary.
External references are strictly decoded and identity-validated during
preparation; generated references are decoded and identity-validated at the
ordered step handoff. Both paths retain canonical registry JSON for provenance
and emit a names-only projection to operation-time spell, combat, and
interaction prompts. Combat and interaction normalization use the canonical
registry for exact name lookup. The
framework records generated identity and bounded producer provenance, while
the raw external reference remains independently tracked by pipeline
provenance. An absent registry is represented only by the empty prompt value
`{"npcs":[]}`. Spell
and combat consumers use this shared boundary without changing their public
module contracts. Interaction consumers require it and retain only current
transcript references as durable evidence.
Generic validators under `internal/modules/generic/validate` provide
unconditional test decisions, JSON syntax validation, and JSON Schema
validation. D&D spell validators under `internal/modules/dnd/validate/spells`
consume the canonical spell-list type directly to provide shape,
effective-catalog, source-reference, and source-relatedness decisions.
Production composition is grouped behind package-family registrars, and every
implemented production extension uses its domain-first tree:
| Package | Implemented responsibility |
| --- | --- |
| `internal/modules/generic/register` | Registers domain-neutral chunk, merge, normalize, output, and validator implementations. |
| `internal/modules/seriatim/register` | Registers the Seriatim input adapter. |
| `internal/modules/dnd/register` | Registers D&D modules, validators, default validator policy, and prompt/schema assets. |
The CLI allocates the framework registries and asset registry, then invokes
these registrars in generic, Seriatim, and D&D order.
Implementation details for all production extensions are in
[Module Internals](modules.md).
## Run-State Components
| Surface | Implemented owners | Internal purpose |
| --- | --- | --- | | --- | --- | --- |
| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. | | Executable and command boundary | **cmd/notarius**, **internal/cli** | Process entry, command dispatch, configuration discovery, production composition, runtime collaborator setup, durable file placement, and user-facing reporting. |
| Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes, including ordered-step scope and generated-artifact dependency decisions. | | Configuration | **internal/core/config** | Defaults, strict YAML parsing, environment overrides, structural validation, effective resolution, redaction, and resolved-composition summaries. |
| Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. | | Generic models | **internal/core/source**, **internal/core/artifacts**, **internal/framework/contracts** | Source documents and chunks, manifests and provenance, plus typed artifact, reference, validation, output, and structured-completion contracts. |
| Debug bundles | `internal/core/debugbundle`, `internal/framework/debug`, and pipeline instrumentation | Persist redacted summaries and application-owned traces. | | Pipeline framework | **internal/framework/pipeline** | Registries, profile and reference resolution, typed preparation, validation, retry coordination, ordered execution, handoff, and result assembly. |
| LLM and prompt runtime | **internal/framework/llm**, **internal/framework/promptfs** | Provider-neutral structured completions, scheduling, profile recording, prompt assets, schema registration, and credential-shaped-value redaction. |
| Runtime state | **internal/core/fileio**, **internal/core/debugbundle**, **internal/framework/checkpoint**, **internal/framework/chunkplan**, **internal/framework/chunkmap**, **internal/framework/debug** | Confined atomic files, debug bundles, checkpoint and chunk-plan state, accepted chunk maps, and pipeline-facing debug recording. |
| Production extensions | **internal/modules/generic**, **internal/modules/seriatim**, **internal/modules/dnd** | Domain-neutral extensions, Seriatim input support, and D&D extraction families registered into the production catalog. |
Physical layout, cleanup, recovery, and sensitive-data handling are defined Generic core and framework packages do not depend on production extensions.
in [Operations](../operations.md). Concrete modules receive recorder Concrete extensions depend inward on their contracts and are registered only at
interfaces and request data, not physical state roots. the CLI composition boundary.
## Focused Documentation ## Focused Documentation
- [Pipeline Internals](pipeline.md): resolution, execution, validation, retries, - [Configuration Internals](configuration.md): loading, validation, effective
checkpoint/debug hooks, and result assembly. resolution, redaction, and resolved-composition identity.
- [Module Internals](modules.md): production modules, validators, assets, - [CLI Internals](cli.md): command dispatch, production composition, run
registration, and the contributor recipe for adding an extension. orchestration, and terminal reporting.
- [LLM Runtime](llm.md): structured completion contracts, Scriptorium adapter, - [Pipeline Internals](pipeline.md): resolution, preparation, execution,
assets, scheduling, profile recording, and redaction. validation, typed handoff, and framework state hooks.
- [Run State Internals](state.md): output, cache, debug collaborator - [Run State Internals](state.md): output, cache, debug collaborator
composition, and path safety. composition, and path safety.
- [LLM Runtime](llm.md): structured completion, scheduling, prompt assets,
profiles, and secret handling.
- [Module Internals](modules.md): generic extension registration, module
construction, validation, and reference mechanics.
- [D&D Module Internals](dnd.md): shared D&D extractor conventions, generated
reference projections, and lane-specific exceptions. Durable D&D and
Seriatim data shapes remain in the [integration contracts](../integrations/).
Use this map to find an owner, then read the focused document and its tests
before changing behavior.

View File

@@ -1,491 +1,168 @@
# Pipeline Internals # Pipeline Internals
The implemented resolver and runner live in `internal/framework/pipeline`. This document describes the framework-owned pipeline mechanics in
Their fixed workflow and ownership boundaries are defined by **internal/framework/pipeline**. [Configuration](../config.md) owns selectable
[Architecture](../policy/architecture.md#system-shape). Configuration fields, profiles, bindings, and retry settings; [Operations](../operations.md) owns
defaults, and selectable keys are defined in state lifecycle and recovery; and the [integration contracts](../integrations/)
[Configuration](../config.md#pipelines). own durable output shapes. Concrete production extensions are covered by
[Module Internals](modules.md).
Resolution fixes the ordered steps, selected lanes, and all stage bindings; ## Boundary
preparation constructs every selected implementation before the runner begins
source work. After serial input parsing and plan selection or generation, the
runner materializes chunks and executes one step at a time. Within a step,
extract work uses one bounded run-wide worker pool in chunk-first, lane-second
order. Each lane's merge and normalize operations remain serial, and lanes in
the same step may overlap once their extracts are terminal. A later step cannot
start across its barrier until every earlier lane is terminal and each required
generated artifact has been accepted and handed off.
## Resolution The pipeline framework accepts a resolved composition, registries, shared
dependencies, input bytes, and state/debug collaborators. It returns logical
output files, normalized artifacts, recorded rejections and warnings, manifest
provenance, and checkpoint decisions. The CLI owns process arguments,
configuration discovery, physical roots, and placement of returned output
files.
`internal/core/config.Config.Resolve` validates the loaded configuration, The framework has one fixed shape:
selects the named profile, applies the runtime inputs supplied by the CLI, and
calls `pipeline.ResolvePipeline`.
`ResolvePipeline`: ~~~
input -> chunk -> extract -> merge -> normalize -> output
~~~
1. selects the explicit ordered steps, or creates the implicit `default` step Input and chunking are pipeline-wide. A selected artifact lane owns extract,
from the legacy top-level `artifacts` map; merge, and normalize; output aggregates the terminal lane outcomes. A pipeline
2. selects and sorts artifact lanes within each step while enforcing global lane is an ordered list of steps, not an arbitrary workflow graph.
identity;
3. completes omitted bindings using the documented configuration defaults;
4. looks up each module and validator spec without constructing it;
5. for a typed extractor, derives its artifact kind, requires the codec, and
selects exact-type merger, normalizer, and validator variants;
6. checks required and provided capabilities in workflow order;
7. resolves external and generated target-aware reference bindings and
validates producer order, consumer slot declarations, and artifact-kind
compatibility;
8. validates each selected module and validator option set through its registry
entry; and
9. calculates a digest over the resolved structure, including step order, step
IDs, lane membership, generated topology, producer and consumer identities,
typed artifact kind and schema identity, and the effective validator policy
in its resolved execution order.
Resolution returns a `ResolvedPipeline` containing ordered steps, lanes, ## Resolve, Materialize, Prepare
concrete bindings, validator chains, reference targets, and the digest. It does
not read external reference bytes or construct runtime modules. CLI lane and
reference selector syntax is defined in the [CLI reference](../cli.md#run).
The digest includes each resolved step's ID and lane membership, generated Resolution turns a configured pipeline profile into a **ResolvedPipeline**.
producer/consumer topology, and each validator chain's stage, lane, owning It normalizes the pipeline and lane identities, applies stage defaults, selects
module, ordered validator bindings, execution classes, targets, and artifact requested lanes where that is supported, resolves validator chains, checks
kinds. Changing step order, a dependency, a default chain, or an explicit module capabilities and typed artifact compatibility, validates options, and
override therefore changes pipeline identity whenever it changes effective assigns a deterministic resolved-composition digest. The resolved pipeline
execution policy. contains bindings and declared reference targets, not external reference bytes.
Configuration resolution supplies the selected profile and catalog; see
[Configuration Internals](configuration.md).
## Reference Materialization External reference materialization happens before preparation. The materializer
checks that each slot is declared by the selected module, resolves a file path
relative to the correct configuration or working-directory origin, reads
UTF-8 text, verifies media type and size limits, and retains bounded
provenance. A generated-artifact selector remains declared but has no bytes
until its producing step completes.
The CLI calls `MaterializeReferences` after resolution and before constructing Preparation is the construction boundary. It validates the resolved shape and
the LLM client or running the pipeline. For external bindings, the materializer registry set, clones the resolved data, then constructs the input adapter,
checks each binding against its resolved target declaration, reads and validates chunker, stage-local validators, every typed lane, and output encoder with
the file, and builds both a `contracts.ReferenceSet` and provenance-only cloned options, references, and shared dependencies. It also collects stable
metadata on the corresponding `ResolvedReferenceTarget`. A structured checkpoint fingerprints. Missing registrations, incompatible typed entries,
generated binding is declaration-only at this point: its producer bytes do not nil implementations, and constructor failures are reported before source
exist until the producer lane reaches an accepted normalized result. parsing or any stage operation begins.
Preparation delivers the materialized external set for each target through ## Typed Lanes And References
`pipeline.BuildRequest`: chunkers and chunk validators receive the chunk target;
extractors and extract validators receive the lane extract target; mergers and
merge validators receive the lane merge target; and normalizers and normalize
validators receive the lane normalize target. Input and output builders receive
an empty set because those stages cannot declare references. Every builder gets
an isolated deep clone of its target set, so construction-time mutation cannot
change another builder, the resolved pipeline, or later runtime requests.
Prepared consumers do not need to be reconstructed when generated content is Each resolved lane has one artifact kind, codec, and exact Go type. The
available. At the step boundary, the runner encodes the accepted producer value framework uses private type erasure only around those typed operations; every
through its registered canonical codec, validates the generated bytes against handoff checks exact type and codec identity and reports incompatibility as an
each target slot's kind, schema, media type, and size, and clones one immutable error rather than panicking. Encoding through the registered codec is the
reference item into the operation request. The item includes canonical digest, boundary for output, checkpoints, debug records, and generated references.
size, and bounded producer provenance but no filesystem URI. A handoff failure
is a framework dependency error and prevents every consumer in that step from
starting.
The runner continues to clone the resulting set into the chunk, extract, merge, Reference targets are stage- and lane-specific. External reference bytes are
or normalize request that owns the target. LLM-backed extensions may convert cloned into the operation request. Generated references are built at the next
those items into named prompt inputs. Reference content remains separate from step boundary from exactly one accepted normalized producer output. The
source evidence and source digests, whether the item came from a file or a framework decodes and re-encodes that output with the registered producer
generated handoff. codec, checks its complete schema and media identity, and records a content
digest plus bounded producer provenance. A missing, ambiguous, invalid, or
incompatible producer prevents the consumer step from starting.
Binding precedence, path resolution, accepted content, and media-type behavior ## Execution And Ordering
are configuration contracts; see [Configuration](../config.md#pipelines).
Durable provenance is defined in the
[JSON output contract](../integrations/json-output.md#manifestjson), while
runtime sensitive-data handling belongs in [Operations](../operations.md).
## Registries And Specs The runner validates its input, installs no-op state collaborators when none
were supplied, and serially performs source parsing and chunk-plan selection.
An accepted plan is materialized into source-addressed chunks and passes the
configured chunk validators before any lane runs. A chunk rejection is a
recorded pipeline outcome: lanes do not start, but the output stage can encode
the terminal result.
`pipeline.Registries` holds option validators and run-local builders used during For each ordered step, the runner first builds generated reference sets from
resolution and preparation. the accepted normalized outputs of earlier steps. It then executes the step's
`pipeline.ModuleCatalog` exposes their specs during configuration validation and lanes. Later steps do not begin until the current step is terminal and its
resolution. Separate registries exist for every stage and for validators; generated handoffs have succeeded.
`ValidatorChainRegistry` stores production default-chain mappings. Both
containers also carry an `ArtifactCodecRegistry`. Generic registration records
one codec per stable artifact kind, validates its schema metadata and JSON
Schema, retains the exact schema digest and Go type, and safely encodes or
decodes framework-erased values with typed errors on incompatibility.
Typed extractor entries are keyed by module key and declare one artifact kind. Within a step, the lane engine dispatches extraction jobs in deterministic
Merger, normalizer, and typed-validator variants are keyed by module or chunk-first, lane-second order to a bounded worker group. When all extraction
validator key plus artifact kind. Chunk and serialized validators occupy jobs for one lane are terminal, a bounded continuation group can run that
separate target namespaces; serialized registrations declare whether they lane's merge and normalize work while extraction for other lanes continues.
support chunks, artifacts, or both. Duplicate variants and exact Go-type The framework does not create an unbounded goroutine per chunk or lane.
mismatches are rejected deterministically.
Lane-sensitive merger and normalizer spec discovery always supplies the Completion timing does not determine public results. The coordinator restores
extractor's artifact kind, so variants under one reusable key may declare lane and chunk order before merging results, and selects a framework error by
different capabilities and reference slots. Kind-neutral registry inspection stable stage, lane, and chunk position. A validator rejection records a lane
selects the first registered artifact kind in sorted order. outcome without cancelling unrelated work. A framework error or parent
cancellation cancels derived work, prevents queued work from starting, waits
for started workers, and prevents output encoding.
Production composition registers the D&D spell-list, NPC-list, combat-turn-list, ## Validation, Retries, And Output
NPC-interaction-list, and scene-description-list codecs and typed lane
variants, plus serialized JSON validators. Every artifact lane resolves through
the typed registries and a matching codec.
A `ModuleSpec` declares its stage plus required and provided capabilities. Every chunk, extract, merge, and normalize candidate passes its resolved
Chunk, extract, merge, and normalize specs may also declare reference slots. validator chain. Validators receive immutable canonical input appropriate to
Registry implementations defensively copy spec metadata, reject duplicate keys, their target: chunks, typed values, or serialized codec bytes. They may
and verify that a constructed implementation reports the registered key. approve, approve with warnings, reject, or fail. A rejection is an ordinary
Builder registrations accept `ModuleDependencies` and cloned configuration pipeline result; a validator error is a framework error.
options through one `BuildRequest`. Builders decode those options and retain
typed values or injected dependencies in the constructed implementation.
Extractors declare their artifact kind, and merger, normalizer, and validator
resolution selects the matching typed variant.
A `ValidatorSpec` declares a validator key and execution class. Resolution uses The runner applies the binding's retry policy around a stage operation and its
the execution class to reject incompatible profile bindings before execution. complete validation chain. It preserves warnings only from the final accepted
The current production catalog and default chain are listed only in or rejected attempt. Cancellation stops retries. Normalizer-specific retry
[Configuration](../config.md#implemented-production-validators). directives consume this same budget and validate any final safe fallback through
the normalizer chain.
## Preparation And Runner Boundary After terminal lane work, the runner assembles manifest provenance, normalized
artifacts, rejections, warnings, and an optional accepted chunk map. The output
`pipeline.Prepare` receives a resolved pipeline, the registries, and shared encoder returns logical files; it does not choose a physical directory. The CLI
module dependencies. It constructs input; chunk and its validators; every publishes those files only after the runner returns without a framework error.
step's lane extract, merge, and normalize modules and validator chains in Logical file names and schemas are defined by the
resolved order; then output. It stops at the first error with pipeline, step, [output integration contracts](../integrations/).
stage, lane, module, and validator context as applicable. It never invokes an
operation method. Generated references are not available during preparation;
the operation request is the handoff boundary.
`PreparedPipeline` keeps private constructed executors and exposes cloned
resolved input, chunk, lane, and output identities. Prepared components may
implement `pipeline.CheckpointFingerprintProvider` to contribute explicit
semantic identities to checkpoint reuse. Preparation trims and validates each
non-secret name and value, prefixes it with the component's stage, lane,
module, and validator scope, rejects duplicates, and retains the resulting
sorted collection behind a defensive-copy accessor. Fingerprints must be
stable and must not contain source content, credentials, local paths,
timestamps, or other invocation-specific values.
`pipeline.RunInput` carries that prepared pipeline, raw source input, run identity and timing, optional
session and profile metadata, a chunk-plan store and mode, a checkpoint
execution policy, and checkpoint/debug collaborators. The runner
parses source bytes through the already constructed input adapter. Later stage
requests receive the generic source model; extract requests receive
chunk-scoped input material, while chunk, merge, and normalize requests retain
access to the original source material. Input, chunk, and output operation
requests do not carry raw module options. The chunk request also does not carry
an LLM client; an LLM-backed chunker receives the shared client during
preparation. Their operation requests retain run-specific source, reference,
profile, session, metadata, and step-handoff context as applicable. A generated
reference is cloned into each compatible consumer request and is never exposed
as a path.
Prepared lanes retain exact-type-checked erased operation closures. The runner
uses those closures to keep each value typed through extraction, validation,
merge, and normalization.
Source validation requires every unit to carry a canonical self-reference to
its containing document and its own unit ID. Explicit clone, checkpoint, and
debug boundaries retain that reference, and the canonical source digest covers
it deterministically. Chunks use the same source model and carry one canonical
reference spanning the first selected unit through the last.
`pipeline.RunOutput` carries the run manifest, accepted normalized serialized
artifacts with lane and normalizer provenance,
rejected results, warnings, checkpoint events, and logical files returned by the
output encoder. The CLI owns debug-summary and durable filesystem writes after
the runner returns.
## Execution Flow
The pipeline-wide coordinator owns the ordered step loop, generated-reference
sets at each barrier, and deterministic merging of step outcomes. For one step,
one run-local lane engine owns worker lifecycle, cancellation, dispatch,
continuation queues, and result collection. It initializes checkpoint state in
lane order, dispatches bounded extract work, advances terminal lanes through
serial merge and normalize work, selects failures by stable pipeline scope, and
merges lane-local outcomes back in resolved order. Completion timing never
becomes public ordering.
The runner:
1. validates its prepared input;
2. parses the raw input with the prepared adapter and validates the generic
source document;
3. selects a stored plan or executes the configured chunker's `Plan` operation;
4. canonicalizes and materializes the plan, then validates the resulting
chunks;
5. builds the framework-owned accepted chunk map from the accepted source,
logical plan, and exact materialized chunks, then supplies it to the output
request independently of output-module options;
6. executes each resolved step in configuration order. For one step, it
dispatches extract jobs in source-chunk then resolved-lane order, starts a
bounded lane continuation when all extracts for that lane are terminal, and
waits for every lane to become terminal;
7. encodes and validates each accepted normalized producer artifact, then
builds the immutable generated reference sets for the next step;
8. invokes the prepared output encoder only after every step succeeds and
validates its logical file results;
9. returns the assembled manifest, outcomes, warnings, and files.
Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
and validators while performing these transitions:
1. extract once per accepted chunk and add runner-owned lane, source, and chunk
provenance;
2. validate each extract result and omit rejected results from merge input;
3. skip the rest of the lane when no extract result is accepted;
4. merge accepted extract results in their existing order;
5. validate the merge result and skip normalization on rejection;
6. normalize the accepted merge result;
7. validate and append the accepted normalized result.
At a step barrier, a lane with no accepted normalized output is still a regular
rejection unless a later generated binding names that lane as a required
producer. In that case the runner raises a deterministic dependency error and
does not start the consumer step. One accepted typed artifact may fan out to
multiple compatible target slots. Consumers in the same step may run
concurrently after the handoff; no work crosses the barrier early.
Module-provided warnings and payload warnings are promoted only from attempts
whose results are accepted and used.
## Chunk Plans And Reuse
`Chunker.Plan` returns a `source.ChunkPlan`: the canonical source digest,
ordered unit-ID ranges, and optional plan or range annotations. The framework
owns plan canonicalization and materialization. It creates the generic chunks
and therefore owns their IDs, indexes, source references, JSON content, units,
media type, and generic metadata. Plan and range annotations are independently
owned raw JSON and become `Chunk.PlanAnnotations` and `Chunk.Annotations`.
In `auto`, the runner looks up the source digest before invoking the chunker. A
valid hit is materialized and sent through the current run's configured chunk
validators; it does not invoke the chunk module, consume its retry budget, or
make a chunk-stage LLM call. A missing, invalid, or unmaterializable record
generates a candidate. `refresh` generates without lookup; `bypass` generates
without cache access. Generated plans are published only after the full chunk
validator chain approves them. A validator rejection is a regular rejected
pipeline outcome and never replaces a cached plan.
The store is source-addressed, not pipeline-addressed. Changes to pipeline
configuration, requested chunker, options, references, lanes, validators, or
LLM profile do not prevent a source-digest hit. The manifest records both the
currently requested chunker and the effective plan producer. Cache state and
paths are configured and operated outside the runner; see
[Configuration](../config.md#state-surfaces) and [Operations](../operations.md).
For an accepted plan, the runner also constructs the strict framework-owned
[Accepted Chunk Map](../integrations/chunk-map.md) before lane execution. It
uses the current resolved chunker as `requested_chunker` and the stored or
generated record as `producer`, preserving that distinction on reuse. Chunk
rejection supplies no map; later lane rejection does not discard it. Output
encoders receive a defensively owned serialized value and may explicitly
ignore it.
The extract job channel has the same capacity as the effective extract worker
count, so dispatch applies backpressure. A fixed continuation executor prevents
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
continuations publish lane-local results; the coordinator is the only writer of
aggregate output and merges those results in resolved lane and source-chunk
order.
## Plan Canonicalization And Chunk Materialization
Plan canonicalization requires canonical JSON annotations, a matching source
digest, at least one range, existing ordered boundaries, and increasing range
starts. Ranges may overlap or leave gaps; a chunker may impose stricter policy.
Materialization deterministically reconstructs each range from the current
source document, deep-clones JSON-shaped source-unit metadata, and copies
annotations without interpreting their namespaces. Materialized chunks and
separate materializations do not share mutable unit metadata; unsupported or
cyclic metadata fails materialization with context.
Before lane execution, generic chunk validation checks the materialized chunks'
identities, order, source references, content, media type, units, and metadata.
No chunk checkpoint participates in plan selection: plan storage is the only
chunk-reuse mechanism. Extract, merge, and normalize checkpoints continue to
use materialized chunk digests as their dependencies.
## Validation And Retries
Chunk, extract, merge, and normalize results pass through the resolved validator
chain for their stage and module. Chunk validators receive canonical chunks;
typed validators receive the domain value; and serialized validators receive
canonical chunk JSON or artifact codec bytes. Validators execute in resolved
order and stop at the first error or rejection. An empty chain approves the
result.
Production D&D artifact chains keep generic JSON syntax validation first, then
run every rejecting domain validator before generic JSON Schema validation. The
domain validator therefore owns expected semantic diagnostics; the generic
schema validator remains the final rejecting representation backstop, before
warning-only relatedness validation. Explicitly configured validator chains
retain their configured order.
`runWithRetry` applies the effective retry policy around module execution and
its complete validation chain. A module or validator error becomes a framework
error when attempts are exhausted. A rejection becomes a recorded
`RejectedOutput` when attempts are exhausted. Cancellation stops retry
processing immediately.
Structured-completion adapters classify malformed or undecodable provider
output with the provider-neutral `contracts.ErrInvalidStructuredOutput` error.
A typed normalizer may turn that condition, or another unsafe proposal, into a
normalize retry directive with a module-supplied safe candidate, stable
diagnostic, and fallback warnings. The directive consumes the same configured
normalize retry budget: `retries` permits that many additional attempts after
the initial attempt. It neither creates a normalizer-local retry loop nor
records an accepted checkpoint for the discarded attempt.
Before adding a normalize retry directive to attempt debug data, the runner
requires a nonblank, valid UTF-8 reason code of at most 128 bytes and a
nonblank, valid UTF-8 message of at most 4,096 bytes. These are encoded-byte
limits. The framework rejects an invalid directive without truncating or
rewriting either field. It validates only this mechanical contract; normalizers
remain responsible for ensuring their otherwise valid diagnostics do not expose
source material, credentials, paths, names, or other sensitive content.
The framework treats a module-supplied candidate as opaque. The normalizer owns
its safety determination, and the configured normalizer validator chain remains
the acceptance boundary for the final fallback.
When a later normalize attempt succeeds, its candidate alone proceeds through
the usual validation and checkpoint path. When the final attempt still returns
a directive, the runner validates its supplied safe fallback through that same
normalizer validator chain before accepting or rejecting it. Ordinary
attempt-local warnings and fallback warnings remain unpromoted while another
attempt is available; only final exhaustion promotes the supplied fallback
warnings. Rejected output is a non-fatal pipeline outcome and does not advance.
Configuration owns retry counts and validator overrides; see
[Module Bindings](../config.md#module-bindings).
## Checkpoint And Debug Hooks ## Checkpoint And Debug Hooks
The runner depends on recorder and loader interfaces, using no-op The runner receives checkpoint and debug interfaces rather than roots. It
implementations when collaborators are absent. Each checkpointed workflow records workflow transitions and reuse decisions through the supplied
boundary records a running, succeeded, or failed transition. Reuse decisions collaborators, and clones reusable artifacts before they re-enter normal typed
are consulted in workflow order and accepted payloads are cloned before handoff. Generated-reference dependencies participate in checkpoint decisions.
entering the normal handoff path. Typed extract, merge, and normalize Selective recomputation can require a canonical accepted normalized predecessor
checkpoints store codec bytes with artifact kind, schema ID, name, version and before a dependent lane starts.
exact digest, and media type. Reuse compares that identity with the prepared
codec and decodes through the codec; missing identity, mismatches, corrupt
bytes, and decode failures become explicit reuse misses and execute the lane
normally. Dependency fingerprints and debug content digests use the same stable
codec bytes that cross those boundaries.
That progressive extract, merge, and normalize reuse is the ordinary resume Debug recording is attempt-scoped and application-owned. A failure to persist
path. A lane marked as a required predecessor for selective recomputation takes required debug data is a framework error. State roots, persistence, reason-code
a separate accepted-output path before extract scheduling. The loader reads the meanings, resume, and cleanup are intentionally owned by
existing successful normalize manifest and payload by step, lane, and [Run State Internals](state.md) and [Operations](../operations.md).
normalizer, without consulting extract or merge dependencies. It requires the
current non-empty checkpoint identity to match, so the invocation identity
still binds the input, resolved topology and configuration, references, runtime
overrides, profiles, and component fingerprints.
The runner decodes and canonically re-encodes each reusable artifact once with ## Invariants To Preserve
the prepared codec, requiring exact kind, schema identity and digest, media
type, canonical bytes, content digest, and producer provenance. A valid accepted
producer becomes a runner-owned cloned normalized output, restores only
normalize-checkpoint warnings, and records one `accepted_artifact_reused`
normalize decision. It does not invoke or record extract, merge, normalize, or
their validators. Invalid or unavailable accepted state records its decision
and fails the producer step; the dependent step never starts and the producer
is not implicitly rerun. If a later required lane fails during initialization,
already hydrated terminal lanes remain in the failed output in resolved order.
Generated references add downstream dependencies containing the producer's - The six fixed stages remain explicit; a pipeline is not a general DAG.
artifact kind, complete schema identity, media type, canonical content digest, - Resolution and preparation reject statically discoverable incompatibility
and size. Compatible accepted producer outputs may therefore feed a later step before parsing or execution.
without re-executing the producer. Forced lanes bypass accepted-output - Every typed lane uses one compatible artifact kind, codec, and exact Go type.
hydration and execute normally. A missing, rejected, corrupt, incompatible, or - Generated references come only from one earlier accepted normalized producer
changed producer blocks its dependent while leaving independent work eligible and carry canonical identity rather than an unverified value.
for reuse. The runner records bounded decision - Rejections are recorded outcomes; framework errors cancel derived work and
categories: `reused`, `executed`, `forced_recompute`, and prevent output encoding.
`dependency_invalidated`. Operator meanings for the stable reason codes belong - Public ordering and selected errors are independent of goroutine completion
to [Operations](../operations.md#resume-and-selective-recompute). order.
- Pipeline modules receive collaborators and data, never CLI streams or
physical output, cache, or debug roots.
The CLI includes prepared-component fingerprints in the run-wide checkpoint ## Focused Tests
identity alongside resolved configuration, raw input, reference provenance,
runtime overrides, and LLM-profile fingerprints. Module metadata is not used
implicitly for cache identity: components opt in only with stable semantic
values that can change accepted output. Adding or changing a component
fingerprint intentionally produces a cold cache miss. Existing checkpoint
schemas and paths remain unchanged.
The CLI's `--recompute-step` policy forces the selected step and all transitive - **internal/framework/pipeline/profile_test.go** and
dependents, but requires accepted normalized artifacts for every unselected **typed_resolution_test.go** cover resolution, defaults, ordered steps,
producer on which that closure depends. It changes execution policy only; it compatibility, validators, references, and resolved identity.
does not alter persistent checkpoint identity. - **internal/framework/pipeline/preparation_test.go** covers complete
construction before execution and contextual construction failures.
- **internal/framework/pipeline/references_test.go** and **handoff_test.go**
cover external materialization, generated references, provenance, and typed
producer checks.
- **internal/framework/pipeline/runner_concurrency_test.go** covers bounded
execution, ordered steps, stable error selection, rejections, and
cancellation.
- **internal/framework/pipeline/runner_chunk_plan_test.go**,
**runner_typed_checkpoint_test.go**, and
**runner_accepted_checkpoint_test.go** cover state hooks and reuse behavior.
- **internal/framework/pipeline/runner_attempt_debug_test.go** and
**runner_terminal_debug_test.go** cover attempt and terminal debug behavior.
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM Run **go test ./internal/framework/pipeline ./internal/cli** after changing a
boundaries. Every executed chunk, extract, merge, and normalize attempt writes pipeline boundary. Use the more focused tests above while iterating.
one terminal envelope for acceptance, validator rejection, module or validator
error, or applicable candidate or final serialization error. The envelope
contains its attempt-local warnings, any available candidate and rejection,
and terminal error text; normalize retry directives retain their attempt-local
candidate and diagnostic, while only the final safe fallback reaches validation.
Failures before a candidate exists omit that payload.
Only LLM calls made by the module operation belong to the module attempt.
Validator calls retain independent scopes under `validate/` and are not
duplicated into the module envelope. A failed terminal-envelope write is a
non-retryable framework error and is joined with any primary attempt error.
Debug data is never used as a checkpoint source. Typed artifact debug envelopes
are domain-neutral, redact sensitive metadata and bytes through the common
debug policy, and record codec identity plus schema and content digests.
Merge and normalize attempts serialize their in-memory candidate with the
codec's required candidate encoder before typed validation. Serialized
validators and attempt debug use that candidate representation, which carries
the codec media type and schema identity but is never checkpointed or passed
downstream. Only a validator-approved value is encoded through the strict final
codec and made eligible for a checkpoint or stage output.
Checkpoint identity, physical layout, reuse behavior, and debug artifact
handling are operator contracts in [Operations](../operations.md). Serialization
and recorder implementation are inventoried in
[Internal Overview](overview.md#run-state-components).
## Results And Failures
The runner owns manifest assembly and handoff summaries but not the durable JSON
schema. It records resolved module and lane provenance, validator chains,
source/reference identities, selected LLM profiles, normalized and rejected
summaries, status, and timing. Serialized artifact content remains outside the manifest.
Module metadata providers may add non-secret singleton or lane-scoped metadata.
Execution errors include stage, module, lane, or validator context. Once a
manifest exists, a failing run returns it with failed status and completion
time. Successful status reflects whether any result was rejected. The
durable manifest and logical file schemas are defined in the
[JSON output contract](../integrations/json-output.md).
On a framework failure, the runner cancels its derived context, stops submitting
new extract work, drains started tasks, and skips the output encoder. Parent
cancellation takes precedence. Otherwise context-cancellation fallout is
discarded when a substantive error exists, and the primary error is selected by
stage, resolved lane, and source chunk rather than completion time.
## Tests To Inspect
- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
- `internal/framework/pipeline/profile_test.go`: selection, defaults,
capabilities, validator chains, and digest behavior.
- `internal/framework/pipeline/artifact_codec_registry_test.go`: typed codec
metadata, registration, erasure safety, strict decoding, and cloning.
- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed
lane resolution and preparation, target-specific validators,
incompatibilities, ordering, and schema-sensitive pipeline identity.
- `internal/framework/pipeline/runner_concurrency_test.go`: bounded dispatch and
continuations, reverse completion, stable errors, rejection, cancellation,
retries, and independent provider-call limits.
- `internal/framework/pipeline/preparation_test.go`: option validation,
construction order, dependency failures, and the before-source-work boundary.
- `internal/framework/pipeline/references_test.go`: target resolution and
materialization.
- `internal/cli/run_contract_test.go`: production run transitions, retries,
rejections, warnings, CLI recomputation controls, debug hooks, and manifests.
- `internal/cli/recompute_execution_contract_test.go`: filesystem-backed
selective recomputation and accepted-producer recovery.
- `internal/cli/production_contract_test.go`: production composition and
configuration-resolution smoke coverage.
- `internal/cli/example_contract_test.go`: maintained example resolution and
execution ownership.
- `internal/modules/integration/*_test.go` and
`internal/modules/seriatim/input/transcript/runner_test.go`: typed runner
composition across concrete module families.
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse
collaborators.

View File

@@ -2,7 +2,8 @@
This document describes the implementation collaborators behind output, cache, This document describes the implementation collaborators behind output, cache,
and debug state. User-visible fields belong in [Configuration](../config.md), and debug state. User-visible fields belong in [Configuration](../config.md),
and layouts and lifecycle belong in [Operations](../operations.md). and physical layout, retention, recovery, reason codes, and cleanup belong in
[Operations](../operations.md).
## Composition ## Composition
@@ -12,6 +13,12 @@ constructs cache collaborators, writes logical output files, and reports paths.
Pipeline modules receive interfaces and request data, never output, cache, or Pipeline modules receive interfaces and request data, never output, cache, or
debug roots. debug roots.
The CLI creates no chunk-plan store in bypass mode. It creates a checkpoint
recorder only when recording is enabled and a checkpoint loader only for a
resume invocation. It allocates debug state only after a safe run identity has
been generated and only when debug capture was requested. These choices keep
the three state families independently composable.
## Output And Cache ## Output And Cache
The pipeline runner returns logical output files. After validating every The pipeline runner returns logical output files. After validating every
@@ -68,12 +75,19 @@ names only the step, lane, and stable reason code. Decision detail is selected
from code-owned descriptions by reason code and then UTF-8 normalized and from code-owned descriptions by reason code and then UTF-8 normalized and
bounded; callers cannot supply arbitrary diagnostic prose. Typed categories and bounded; callers cannot supply arbitrary diagnostic prose. Typed categories and
codes remain intact through pipeline events and become strings only in manifest codes remain intact through pipeline events and become strings only in manifest
and debug-summary JSON. [Operations](../operations.md#resume-and-selective-recompute) and debug-summary JSON.
is the canonical operator-facing reason-code reference. [Operations](../operations.md#checkpoint-recording-resume-and-recompute) is the
canonical operator-facing reason-code reference.
`internal/core/fileio` provides confined atomic file writes used by state `internal/core/fileio` provides confined atomic file writes used by state
collaborators. The chunk-plan store retains its stronger entry validation. collaborators. The chunk-plan store retains its stronger entry validation.
The CLI constructs selective-recomputation policy from resolved generated
artifact dependencies. It forces the selected step and transitive consumers,
while marking unforced producers as required reusable inputs. The runner owns
the actual hydration and rejection decisions; the [Operations guide](../operations.md#checkpoint-recording-resume-and-recompute)
owns the operator workflow and stable reason-code meanings.
## Debug Bundles ## Debug Bundles
`internal/core/debugbundle` allocates an explicitly requested per-run bundle `internal/core/debugbundle` allocates an explicitly requested per-run bundle
@@ -98,12 +112,27 @@ operation writes the success report, or makes one attempt each to write the
failure report and error log. Terminal persistence failures are reported failure report and error log. Terminal persistence failures are reported
separately and never replace the command's primary error. separately and never replace the command's primary error.
## Invariants To Preserve
- Modules receive state collaborators and request data, never physical roots.
- Output logical paths are validated before a run directory is allocated, and
files are atomically written within that directory.
- Chunk-plan publication occurs only for accepted plans; bypass does not
construct or touch a plan store.
- Checkpoint recording and checkpoint loading remain separate collaborators.
- Debug state is opt-in, is not cache input, and terminal reporting does not
obscure the command's primary failure.
## Tests To Inspect ## Tests To Inspect
- `internal/cli/run_contract_test.go`: command-owned state allocation, - `internal/cli/run_contract_test.go`: command-owned state allocation,
terminalization, and output/report boundaries. terminalization, and output/report boundaries.
- `internal/cli/cache_contract_test.go`: cache-mode precedence, root selection,
and resume collaborator construction.
- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures, - `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
permissions, cleanup, and redaction. permissions, cleanup, and redaction.
- `internal/cli/recompute_policy_test.go`: forced dependents and required
reusable predecessors for selective recomputation.
- `internal/cli/recompute_execution_contract_test.go`: selective recomputation, - `internal/cli/recompute_execution_contract_test.go`: selective recomputation,
filesystem recovery, deterministic decisions, and failed predecessor state. filesystem recovery, deterministic decisions, and failed predecessor state.
- `internal/cli/production_contract_test.go`: production composition and - `internal/cli/production_contract_test.go`: production composition and

View File

@@ -1,322 +1,206 @@
# Operations # Operations
This is the canonical guide to operating Notarius filesystem state. Command This is the canonical guide for operating Notarius runtime state. The
syntax is in the [CLI reference](cli.md); field definitions and precedence are [CLI reference](cli.md) owns command syntax and exit statuses, while
in [Configuration](config.md). [Configuration](config.md) owns fields, defaults, and precedence. Maintainers
who need implementation mechanics should read [Run State Internals](internal/state.md).
## State Model ## State Surfaces
Notarius uses three independent filesystem surfaces: Each run can use independent roots with different retention and access-control
needs.
- output is durable user data; | Surface | Purpose | Created when | Retention |
- cache is reconstructible chunk-plan and checkpoint state; and | --- | --- | --- | --- |
- debug is explicitly requested inspection data. | Output | Durable user-facing result bundle | A pipeline completes and returns logical output files | Keep until consumers no longer need it. |
| Chunk-plan cache | Reconstructible source-addressed plan | The configured cache mode permits cache I/O | Keep while reuse is useful. |
| Checkpoint cache | Reconstructible execution and recovery state | Checkpoint recording is enabled | Keep only while recovery or reuse is useful. |
| Debug bundle | Explicit diagnostic record | A run requests debug collection | Keep only under an intentional sensitive-data retention policy. |
Choose separate roots and access controls for each surface. A normal run writes Output, cache, and debug roots are never merged or cleaned automatically. Use
durable output, may use the chunk-plan cache, and records checkpoints when separate locations and permissions for operators or services that must not
`cache.checkpoints.enabled` is true. It does not create debug state unless its share application data.
invocation includes `--debug`.
## Output ## Roots And Permissions
Durable logical files are written under: The configured output and debug directories are exact roots. An empty cache
directory selects a per-user root:
```text ~~~
<os.UserCacheDir>/notarius/chunk-plans
<os.UserCacheDir>/notarius/checkpoints
~~~
The field definitions and configuration examples are in [Configuration](config.md).
On supported Unix systems, output directories and files are created with
requested modes **0755** and **0644**. Chunk-plan, checkpoint, and debug
directories and files use **0700** and **0600**. The operating system's umask
may impose stricter output modes. Cache and debug roots may contain sensitive
source-derived data, so provision them for one trusted account or service.
## Run Lifecycle
Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads
and resolves configuration before module preparation and source parsing. It
then performs any permitted cache lookup, executes the pipeline, and publishes
logical output files only after a successful runner result.
On success, the command reports the output bundle path. A warning-bearing run
still succeeds and reports its warning count on standard error. Errors and
their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses).
## Output Bundles
Each successful run receives a generated safe run identifier and writes beneath:
~~~
<output-root>/<run-id>/ <output-root>/<run-id>/
``` ~~~
The CLI generates one run ID in the form The [JSON output contract](integrations/json-output.md) owns the logical files
`run-<started-at-unix-nanoseconds>-<32-lowercase-hex-characters>` and uses it and their schemas. Before creating the run directory, Notarius validates every
for output, manifests, and any requested debug bundle. It validates every logical output path. It refuses an existing run directory without changing it.
logical output name before exclusively creating the run directory. If that Files are written atomically; if a later write fails, the newly created partial
directory already exists, the invocation fails without changing it. run directory remains for inspection and is never removed automatically.
Each output file is written atomically. A later file-write failure leaves the Treat an output bundle as durable user data. Do not use cache-cleanup policy to
newly allocated partial run directory in place for inspection; Notarius never remove it. An optional accepted chunk map is also durable output and can carry
automatically removes output. The source- or model-derived annotations; its content and compatibility contract
[JSON output contract](integrations/json-output.md) owns the logical file are defined in [Accepted Chunk Map](integrations/chunk-map.md).
names, schemas, and media types inside a run directory.
An enabled JSON `include_chunk_map` option adds an accepted chunk map to durable
output. Its annotations may contain source- or model-derived data, so retain
and protect it like lane output. The map is opt-in and does not alter existing
bundles; its payload exclusions are defined in the
[Accepted Chunk Map contract](integrations/chunk-map.md).
Remove an output run directory only after its consumer data is no longer
needed. This is data deletion, not cache cleanup.
## Ordered D&D Workflow
The maintained [complete D&D configuration](../examples/dnd-complete.config.yml)
contains one pipeline with two ordered steps. The first step extracts and
normalizes independent item events, NPCs, and scene descriptions. Only after
the NPC and scene-description lanes reach accepted terminal results does the
second step begin; its generated NPC reference is
supplied in memory to spell, combat-turn, and NPC-interaction extraction and
the applicable normalizers, while its generated scene-description reference is
supplied to combat-turn extraction. The item-event lane has no generated
reference dependency and retains only current-transcript evidence.
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-complete.config.yml \
--input examples/seriatim-minimal-transcript.json \
--output-dir ./npc-grounded-output
```
The NPC artifact grounds canonical names through a names-only prompt projection, not spell or combat
evidence. Current-transcript source ranges remain the only event evidence. The
manifest records generated-reference identity and bounded producer provenance;
it does not record generated payload content, and no generated content is
exposed through a filesystem path. The same producer artifact may fan out to
compatible consumers, while a missing or rejected producer prevents the later
step from starting.
Combat-turn extraction calls its LLM only when a scene record exactly matches
the accepted chunk and has kind `combat`. Exact `narrative`, `recap`, and
`meta` records produce an accepted empty combat-turn result without an LLM call.
Missing or mismatched scene coverage also produces an accepted empty result and
a bounded warning. Scene descriptions remain control context rather than combat
evidence or prompt material; the complete behavior is defined by the
[D&D combat-turn artifact contract](integrations/dnd-combat-turn-artifacts.md).
Standalone module configurations continue to support external NPC files when a
workflow intentionally crosses a process or session boundary. Those files are
validated against the consumer slot and must be protected as sensitive
campaign data. They are not part of the maintained ordered handoff workflow.
### NPC Semantic Normalization
Before the first step can release its accepted NPC artifact across the ordered
generated-reference barrier, `dnd/npcs` performs one document-level semantic
normalization call for each configured normalize attempt when eligible distinct
names remain. The normalize binding's `retries` setting controls additional
attempts. If an invalid or unsafe identity proposal exhausts that budget, the
run safely accepts the deterministic base result and any independently
validated, model-proposed partial consolidation, with a bounded warning;
ordinary validation still applies before the artifact can cross the barrier.
An accepted normalized NPC checkpoint can be reused on `--resume` just like
other accepted normalize work. A changed normalization prompt, response schema,
or policy identity produces a cold cache miss, so the current reconciliation is
recomputed rather than silently reusing incompatible state.
## Chunk-Plan Cache ## Chunk-Plan Cache
Chunk plans are stored at: Chunk plans live beneath the selected chunk-plan root:
```text ~~~
<chunk-plan-root>/<source-sha256-hex>/plan.json <chunk-plan-root>/<source-sha256-hex>/plan.json
``` ~~~
`auto` reuses a complete valid plan or regenerates missing or invalid state. One validated canonical plan is active for each source digest. The plan stores
`refresh` regenerates and atomically replaces a plan after chunk validation. boundaries and provenance, not a second copy of the entire source. This
`bypass` performs no plan-cache I/O and does not resolve or create the root. source-addressed policy is recorded in [ADR-0005](adr/0005-cache-canonical-chunk-plans-by-source.md).
Plan selection is source-addressed and independent of checkpoint and debug
roots.
When its directory is empty in configuration, the root is The configured cache mode controls one invocation:
`<os.UserCacheDir>/notarius/chunk-plans`. A configured directory is the exact
root; no suffix is appended. Directories and files created by the store use
`0700` and `0600` permissions on supported Unix systems. The configured root
is a trust boundary: do not share it among mutually untrusted users.
Remove an exact digest directory or the configured root only when accepting the - **auto** looks for a valid active plan. Missing or invalid state causes a new
cost of recomputing plans and any chunk-stage work. Cache publication is atomic; plan to be generated; an accepted new plan is atomically published.
there is no history, locking, garbage collection, or rollback facility. - **refresh** skips lookup, generates a plan with the configured chunker, and
atomically replaces the active plan after it is accepted.
- **bypass** performs no chunk-plan cache I/O. It does not resolve or create a
chunk-plan root.
For a Linux service account, provision a dedicated restrictive root such as: A reused plan is still materialized and validated against the current source.
If a prior plan no longer gives acceptable results, use a refresh run rather
than editing cache files. Deleting a plan is recoverable but can repeat costly
chunking work.
```yaml ## Checkpoint Recording, Resume, And Recompute
cache:
chunk_plans:
directory: /var/cache/notarius/chunk-plans
```
## Checkpoint Cache Checkpoint recording is an explicit configuration choice and is disabled by
default. When enabled, each run records stage transitions and the state needed
for compatible recovery. A run records checkpoints even when it does not ask
to reuse them. Checkpoint payloads can contain source-derived and intermediate
application data, so treat the entire root as sensitive.
Checkpoint recording is controlled by `cache.checkpoints.enabled`, which Checkpoint loading is separate: [**--resume**](cli.md#run) asks a run to reuse
defaults to `false`. When enabled, every run records running, succeeded, and compatible recorded work. A resume request fails when checkpoint recording is
failed transitions and reusable validator-approved results. Successful, disabled. Without **--resume**, a recording-enabled run executes normally and
rejected, and failed runs may therefore all leave checkpoint state. The does not load checkpoint state. Compatibility includes the resolved pipeline,
`--resume` flag additionally loads compatible completed work before executing input, selected lanes, runtime overrides, reference provenance, LLM-profile
missing or incompatible stages. Without `--resume`, a recording-enabled run provenance, and prepared-component fingerprints. A changed identity produces a
never loads checkpoints. Using `--resume` while recording is disabled is an cold miss; Notarius does not migrate, rewrite, or delete older checkpoint
error. directories.
Checkpoints use the selected root and the existing identity hierarchy: Checkpoint state is confined below an identity-specific path:
```text ~~~
<checkpoint-root>/<pipeline-id>/<input-key>-<source-or-input-digest>/<pipeline-digest>/<identity-digest>/... <checkpoint-root>/<pipeline-id>/<input-key>-<source-or-input-digest-prefix>/<pipeline-digest-prefix>/<identity-digest-prefix>/
``` ~~~
The final identity digest includes stable semantic fingerprints explicitly ### Selective Recompute
contributed by prepared modules and validators. Adding or changing one of
these fingerprints intentionally causes a cold cache miss; old checkpoint
directories are left in place and are never migrated or deleted automatically.
An empty configured directory selects [**--recompute-step**](cli.md#run) requires both **--resume** and enabled
`<os.UserCacheDir>/notarius/checkpoints`. The root is exact when configured. checkpoint recording. It forces the selected ordered step and every lane that
Created directories and files use `0700` and `0600` permissions on supported depends on it through generated artifact references. Unrelated lanes remain
Unix systems. eligible for reuse.
Checkpoint payloads can contain source text, intermediate artifacts, metadata, For an earlier producer required by a forced consumer, Notarius requires a
warnings, and content digests. Treat them as sensitive derived application compatible accepted normalized artifact. It validates that artifact before
data. Compatible files from a former checkpoint root remain reusable when hydrating it and does not silently rerun the producer. If that state is
`cache.checkpoints.directory` names that exact existing root. They are not missing, rejected, corrupt, non-canonical, or incompatible, the run stops
moved, migrated, or deleted automatically. The frozen serialized identifier before its dependent starts. Rerun the required producer deliberately instead
`workspace_schema_version` remains part of checkpoint compatibility; it is not of copying or editing checkpoint files.
a configuration setting.
For a Linux service account, independently provision: ## Checkpoint Decisions And Recovery
```yaml Checkpoint events classify work as **executed**, **reused**,
cache: **forced_recompute**, or **dependency_invalidated**. Their stable reason codes
checkpoints: are written to run diagnostics and provenance. Use the code, not a copied
enabled: true error message, to decide what to repair.
directory: /var/cache/notarius/checkpoints
```
Remove an exact checkpoint identity directory or the configured root only when | Reason code | Recovery meaning |
recomputation is acceptable.
### Resume And Selective Recompute
`--resume` loads compatible accepted work only when checkpoint recording is
enabled. A normal resumed run may reuse source, extract, merge, and normalize
checkpoints independently and may recompute a stage after a cache miss.
Generated references add a dependency fingerprint
for the producer's artifact kind, schema identity, media type, canonical
content digest, and size. If that fingerprint changes or the producer is
missing, dependent checkpoints are invalidated; unrelated work remains eligible
for reuse.
`--recompute-step <step-id>` requires both `--resume` and
`cache.checkpoints.enabled: true`. It forces the named step and all transitive
dependents to execute, while compatible predecessors and unrelated lanes remain
reusable. The ID may be an explicit configured step or `default` for an
implicit single-step pipeline. It cannot be combined with `--only`, and it does
not change the persistent identity of otherwise identical checkpoints.
Decisions are bounded and categorized as `reused`, `executed`,
`forced_recompute`, or `dependency_invalidated`.
For an unselected producer required by a recomputed step, Notarius loads the
accepted normalized artifact directly. Valid normalize state is sufficient even
when that producer's extract or merge checkpoint is missing or corrupt. The
normalize manifest must be successful and match workspace schema v3, the exact
current invocation identity, step, lane, and normalizer; its payload digest and
canonical codec representation must also validate. A forced producer bypasses
this lookup and executes.
If a required predecessor's accepted normalized artifact is missing, rejected,
corrupt, non-canonical, or incompatible, the run fails before the dependent
step starts. It does not fall back to rerunning that predecessor. The failure
manifest retains completed upstream outcomes and dependency context but not
generated reference content. For diagnosis, first check the producer step and
lane in the manifest, then inspect checkpoint decision categories and reason
codes. Rerun the producer explicitly rather than copying an artifact into the
checkpoint root.
The decision that caused a required-predecessor failure is retained before the
run returns, and the CLI error identifies its step, lane, and reason code.
Checkpoint reason codes are stable diagnostic identifiers:
| Reason code | Operator meaning |
| --- | --- | | --- | --- |
| `loading_disabled` | This invocation did not enable checkpoint loading. | | **loading_disabled** | This invocation did not permit checkpoint loading. |
| `checkpoint_missing` | The requested checkpoint file does not exist. | | **checkpoint_missing**, **checkpoint_path_invalid**, **checkpoint_read_failed**, **checkpoint_decode_failed** | The stored checkpoint could not be located or read safely; normal resume work can execute again. |
| `checkpoint_path_invalid` | The requested checkpoint location failed confinement validation. | | **workspace_schema_incompatible**, **identity_mismatch**, **stage_mismatch**, **step_mismatch**, **lane_mismatch**, **module_mismatch** | Stored state belongs to a different compatible scope or identity; allow a fresh run to create new state. |
| `checkpoint_read_failed` | An existing checkpoint could not be read. | | **status_not_reusable** | The recorded operation did not end in reusable state. |
| `checkpoint_decode_failed` | Checkpoint JSON could not be decoded. | | **dependency_mismatch** | A dependency changed; dependent work is invalidated rather than reused. |
| `workspace_schema_incompatible` | The stored workspace schema is not supported by this build. | | **artifact_payload_invalid**, **artifact_digest_mismatch**, **artifact_codec_incompatible**, **artifact_not_canonical** | A stored artifact cannot safely be hydrated; rerun the producer instead of modifying the cache. |
| `identity_mismatch` | The stored invocation identity differs from the current invocation. | | **checkpoint_reused** | A normal checkpoint passed compatibility checks. |
| `stage_mismatch`, `step_mismatch`, `lane_mismatch`, `module_mismatch` | Stored scope does not match the requested pipeline scope. | | **accepted_artifact_reused** | A required predecessor's accepted normalized artifact was safely hydrated. |
| `status_not_reusable` | The stored operation did not finish in a reusable status. | | **recompute_step** | Selective recomputation deliberately forced this work. |
| `dependency_mismatch` | Stored dependencies differ; the category is `dependency_invalidated`. |
| `artifact_payload_invalid` | Stored artifact payload structure or encoding is invalid. |
| `artifact_digest_mismatch` | Stored artifact bytes do not match their recorded digest. |
| `artifact_codec_incompatible` | Stored artifact identity is incomplete or incompatible with the codec contract. |
| `artifact_not_canonical` | The codec can decode the artifact, but its bytes are not canonical. |
| `checkpoint_reused` | The stored checkpoint passed validation and was reused. |
| `accepted_artifact_reused` | A required producer's accepted normalized artifact was canonically validated and hydrated. |
| `recompute_step` | Selective recomputation forced execution of this lane. |
Decision detail is bounded explanatory text derived from the stable reason code, Reason detail is bounded code-owned text. It is diagnostic information, not a
not caller-supplied prose or a data-recovery channel. It never contains path-discovery or data-recovery mechanism, and does not contain checkpoint,
checkpoint paths, artifact or reference content, source content, credentials, source, reference, credential, or environment content.
or environment values.
## Debug Bundles ## Debug Bundles
Only `notarius run --debug` enables debug collection. The selected root contains Only a [debug-enabled run](cli.md#run) creates a bundle:
one retained bundle per invocation:
```text ~~~
<debug-root>/<run-id>/ <debug-root>/<run-id>/
summary/ summary/
trace/ trace/
``` ~~~
`summary/` contains redacted invocation, effective-configuration, resolved The summary contains redacted invocation and resolution information plus run,
pipeline and reference provenance, checkpoint and chunk-plan decisions, run warning, checkpoint, chunk-plan, and terminal reporting artifacts. The trace
manifest, warnings, report, and any available error text. It excludes raw contains allowlisted application diagnostic records and can include source or
source, references, annotations, prompts, model responses, credentials, and derived application data. Neither surface is a cache input. Do not treat a
malformed cache bytes. debug bundle as safe to share merely because its configuration summary is
redacted.
`trace/` contains application-owned execution detail, including source and Notarius never creates debug state without an explicit request and never
stage material, plans, chunks, validator attempts, prompts, model responses, automatically deletes a requested bundle. If allocation succeeds, the command
timing, and serialized artifacts. It may retain application data omitted from reports its path on both success and later failure. A summary, trace, or
output. Credentials, credential-shaped values, sensitive metadata, unrelated terminal-report persistence failure fails the command while preserving any
environment values, and unrelated filesystem content are not captured. already-written diagnostic data for inspection.
Bundles inherit the sensitivity of the application data they capture. Their
additional risk comes from copying and aggregating that data, so restrict
access, avoid shared roots between untrusted users, and define retention outside
Notarius. Created bundle directories use `0700` and files use `0600` on
supported Unix systems.
Notarius never automatically deletes a requested bundle. If allocation
succeeds, its path is reported on success and failure. A requested summary or
trace write failure makes the command fail, preserving whatever bundle data was
already written for inspection. Every allocated bundle makes one best-effort
attempt to record a terminal `run-report.json`.
## Failures And Warnings
Failures before debug allocation are reported on stderr without a bundle.
Failures after allocation report the bundle path on stderr and make independent
attempts to write a failure `run-report.json` and `error.log`. The report retains
the paths and pipeline outcome fields known at the failure point. If either
terminal write fails, the original command error remains first on stderr,
followed by the persistence error and bundle path. An output-write failure
leaves the allocated bundle in place. A successful run with warnings exits `0`,
reports a warning count on stderr, and records warnings in durable output and
any requested debug summary.
## Cleanup ## Cleanup
Use exact paths for manual cleanup. Examples: Cleanup is manual and destructive. First inspect the exact leaf directory,
then remove only that leaf; do not use a glob or a parent root as the target.
```sh ~~~
rm -rf ./notarius-output/run-1721300000000000000-0123456789abcdef0123456789abcdef rm -rf -- /srv/notarius/output/run-1721300000000000000-0123456789abcdef0123456789abcdef
rm -rf /var/cache/notarius/chunk-plans/0123abcd rm -rf -- /srv/notarius/chunk-plans/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
rm -rf /var/cache/notarius/checkpoints/pipeline/input-0123/pipeline-4567/identity-89ab rm -rf -- /srv/notarius/checkpoints/example/seriatim-0123456789abcdef/0123456789abcdef/0123456789abcdef
rm -rf ./notarius-debug/run-1721300000000000000-0123456789abcdef0123456789abcdef rm -rf -- /srv/notarius/debug/run-1721300000000000000-0123456789abcdef0123456789abcdef
``` ~~~
Avoid broad recursive cleanup against a parent root unless it is an explicit Deleting output permanently removes user data. Deleting chunk plans or
operator policy. Output deletion is permanent user-data loss. Cache deletion is checkpoints is recoverable but may repeat expensive provider or pipeline work.
recoverable but can repeat expensive work. Debug deletion removes troubleshooting Deleting a debug bundle removes troubleshooting evidence and a retained copy of
evidence and any retained application-data copy. application data. Notarius has no cache garbage collector, rollback operation,
or automatic cleanup command.
## Operational Limits ## Operational Limits
Provider retries and timeouts are handled by Scriptorium according to the Provider retries and timeouts are supplied by the selected Scriptorium profile.
selected execution profile. Pipeline module retry settings are defined in Module retry settings and concurrency limits are configuration contracts; see
[Configuration](config.md#module-bindings). Extract worker concurrency and [module bindings](config.md#module-bindings-and-validators) and
actual provider-call concurrency are separate limits; their fields and [concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
validation are defined in [Configuration](config.md#concurrency). Notarius limits and actual provider-call limits are independent. Notarius writes local
writes local files only; remote storage and archive management are outside the filesystem state only; remote storage, archival, and retention automation are
implemented CLI. outside the implemented CLI.

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,85 @@
{
"metadata": {
"id": "session-ravenfall",
"title": "The Ravenfall Watchtower"
},
"segments": [
{
"id": 1,
"start": 0,
"end": 14,
"speaker": "DM",
"text": "Recap: last session, the party learned that Elder Rowan vanished near the Ravenfall watchtower."
},
{
"id": 2,
"start": 14,
"end": 25,
"speaker": "Player",
"text": "Out of character, we agree to investigate the watchtower before the next game."
},
{
"id": 3,
"start": 25,
"end": 39,
"speaker": "DM",
"text": "Aria and Borin arrive at the ruined Ravenfall watchtower as dusk settles over the road."
},
{
"id": 4,
"start": 39,
"end": 55,
"speaker": "Mira Thorn",
"text": "Mira Thorn steps from the doorway and says, \"Elder Rowan warned me that Kesh would return for the relic.\""
},
{
"id": 5,
"start": 55,
"end": 70,
"speaker": "DM",
"text": "Mira leads the party to a hidden cache. The party discovers a moonblade and acquires 20 silver pieces."
},
{
"id": 6,
"start": 70,
"end": 83,
"speaker": "Aria",
"text": "Aria hands her healing potion to Borin so he can carry it into the tower."
},
{
"id": 7,
"start": 83,
"end": 96,
"speaker": "DM",
"text": "Kesh, the goblin captain, orders the raiders to attack. Roll initiative."
},
{
"id": 8,
"start": 96,
"end": 110,
"speaker": "DM",
"text": "On Kesh's turn, he strikes Borin with his scimitar. Borin drinks the healing potion on his turn."
},
{
"id": 9,
"start": 110,
"end": 124,
"speaker": "Aria",
"text": "Aria casts Cure Wounds on Borin, then invokes Aegis of Emberfall as Kesh closes in."
},
{
"id": 10,
"start": 124,
"end": 137,
"speaker": "DM",
"text": "Kesh casts Shield as a reaction against Borin's counterattack, but the party drives the raiders away."
},
{
"id": 11,
"start": 137,
"end": 150,
"speaker": "DM",
"text": "After the battle, Aria pays 5 silver pieces to repair the watchtower gate."
}
]
}

View File

@@ -1,2 +1,8 @@
Cure Wounds: healing spell cast by touch. Ravenfall watchtower: a ruined watchtower near the party's current route.
Shield: defensive reaction spell. Mira Thorn: the watchtower's keeper.
Elder Rowan: a missing local scholar.
Kesh: a goblin captain leading raiders.
Moonblade: a blade found in the watchtower's hidden cache.
Cure Wounds: a healing spell.
Shield: a defensive reaction spell.
Aegis of Emberfall: a campaign spell recorded in the supplied catalog overlay.

View File

@@ -1,3 +1,2 @@
Aria: party cleric and recurring healer. Aria: party cleric and recurring healer.
Borin: fighter ally. Borin: fighter ally.
Bandit mage: hostile spellcaster.

View File

@@ -1,6 +1,7 @@
package cli package cli
import ( import (
"context"
"encoding/json" "encoding/json"
"os" "os"
"path/filepath" "path/filepath"
@@ -11,9 +12,11 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts" "gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config" "gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/debugbundle" "gitea.maximumdirect.net/eric/notarius/internal/core/debugbundle"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline" "gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd" "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells" spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
"gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/input/transcript"
) )
func TestMaintainedExamplesLoadResolveAndList(t *testing.T) { func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
@@ -21,6 +24,20 @@ func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
for _, example := range maintainedExampleFiles(t) { for _, example := range maintainedExampleFiles(t) {
t.Run(example.name, func(t *testing.T) { t.Run(example.name, func(t *testing.T) {
cfg := loadMaintainedExample(t, example.path) cfg := loadMaintainedExample(t, example.path)
raw, err := os.ReadFile(example.transcriptPath)
if err != nil {
t.Fatalf("read maintained transcript %q: %v", example.transcriptPath, err)
}
document, err := transcript.New().Parse(context.Background(), contracts.ParseRequest{
Path: example.transcriptPath,
Raw: raw,
})
if err != nil {
t.Fatalf("parse maintained transcript %q: %v", example.transcriptPath, err)
}
if len(document.Units) == 0 {
t.Fatalf("maintained transcript %q has no parsed units", example.transcriptPath)
}
for _, pipelineID := range example.pipelineIDs { for _, pipelineID := range example.pipelineIDs {
effective, err := cfg.Resolve(resolveInputForMaintainedExample(components, pipelineID)) effective, err := cfg.Resolve(resolveInputForMaintainedExample(components, pipelineID))
if err != nil { if err != nil {

View File

@@ -569,14 +569,15 @@ func TestProductionSceneRunRecordsAnnotationFreeChunkPlanAndProvenance(t *testin
type maintainedExample struct { type maintainedExample struct {
name string name string
path string path string
transcriptPath string
pipelineIDs []string pipelineIDs []string
} }
func maintainedExampleFiles(t *testing.T) []maintainedExample { func maintainedExampleFiles(t *testing.T) []maintainedExample {
t.Helper() t.Helper()
return []maintainedExample{ return []maintainedExample{
{name: "minimal", path: repositoryPath("examples", "dnd-minimal.config.yml"), pipelineIDs: []string{"dnd-session"}}, {name: "minimal", path: repositoryPath("examples", "dnd-minimal.config.yml"), transcriptPath: repositoryPath("examples", "seriatim-minimal-transcript.json"), pipelineIDs: []string{"dnd-session"}},
{name: "complete", path: repositoryPath("examples", "dnd-complete.config.yml"), pipelineIDs: []string{"dnd-session"}}, {name: "complete", path: repositoryPath("examples", "dnd-complete.config.yml"), transcriptPath: repositoryPath("examples", "dnd-complete-transcript.json"), pipelineIDs: []string{"dnd-session"}},
} }
} }