Compare commits
10 Commits
5a968b64eb
...
f6981e2264
| Author | SHA1 | Date | |
|---|---|---|---|
| f6981e2264 | |||
| 2f506f4985 | |||
| fdf8c4afd4 | |||
| 74c793e6a1 | |||
| b5835fbc37 | |||
| fd3f7b85cc | |||
| d86b74f485 | |||
| 59cbf1eb27 | |||
| ee43add75c | |||
| 46761706a2 |
57
README.md
57
README.md
@@ -1,34 +1,43 @@
|
|||||||
# Notarius
|
# Notarius
|
||||||
|
|
||||||
Notarius is a Go CLI for extracting structured artifacts from source material
|
Notarius is a Go CLI for turning source material into structured artifacts with
|
||||||
with explicit, configurable pipeline modules.
|
configured extraction pipelines. The implemented D&D workflow reads Seriatim
|
||||||
|
transcript JSON and can produce scene descriptions, item and currency events,
|
||||||
|
NPC identities, combat turns, NPC interactions, and spell casts.
|
||||||
|
|
||||||
The current implementation reads Seriatim transcript JSON, chunks the source
|
## Quickstart
|
||||||
units, extracts D&D spell-cast artifacts with a Scriptorium-backed LLM runtime,
|
|
||||||
and writes JSON output. Add `--debug` when a per-run inspection bundle is
|
|
||||||
needed.
|
|
||||||
|
|
||||||
```sh
|
Provide an OpenRouter API key through the environment, then run the maintained
|
||||||
OPENROUTER_API_KEY=... \
|
minimal example:
|
||||||
|
|
||||||
|
~~~
|
||||||
|
OPENROUTER_API_KEY=your-api-key \
|
||||||
go run ./cmd/notarius run dnd-session \
|
go run ./cmd/notarius run dnd-session \
|
||||||
--config examples/dnd-minimal.config.yml \
|
--config examples/dnd-minimal.config.yml \
|
||||||
--input examples/seriatim-minimal-transcript.json
|
--input examples/seriatim-minimal-transcript.json
|
||||||
```
|
~~~
|
||||||
|
|
||||||
This invocation uses the maintained example configuration and input. See the
|
The command publishes a JSON output bundle. Its command syntax and exit
|
||||||
configuration and operations references for profile selection, credentials, and
|
behavior are documented in the [CLI reference](docs/cli.md); configuration,
|
||||||
run artifacts.
|
credentials, and module selection are owned by the
|
||||||
|
[configuration reference](docs/config.md).
|
||||||
|
|
||||||
Useful references:
|
For the complete ordered D&D workflow, use
|
||||||
|
[the complete configuration](examples/dnd-complete.config.yml) with
|
||||||
|
[its synthetic transcript](examples/dnd-complete-transcript.json). It
|
||||||
|
demonstrates all implemented D&D lanes and the supporting campaign references.
|
||||||
|
|
||||||
- [CLI reference](docs/cli.md)
|
## Documentation
|
||||||
- [Configuration reference](docs/config.md)
|
|
||||||
- [Operations](docs/operations.md)
|
- [CLI reference](docs/cli.md) — commands, flags, output streams, and exits.
|
||||||
- [Seriatim input contract](docs/integrations/seriatim.md)
|
- [Configuration reference](docs/config.md) — configuration files, profiles,
|
||||||
- [JSON output contract](docs/integrations/json-output.md)
|
validation, and module selection.
|
||||||
- [D&D spell artifact contract](docs/integrations/dnd-spell-artifacts.md)
|
- [Operations](docs/operations.md) — output, state, recovery, and debug
|
||||||
- [Developer guide](docs/development.md)
|
handling.
|
||||||
- [Internal implementation docs](docs/internal/overview.md)
|
- [Integration contracts](docs/integrations/) — Seriatim input and published
|
||||||
- [Minimal D&D configuration](examples/dnd-minimal.config.yml)
|
artifact formats.
|
||||||
- [Complete D&D configuration](examples/dnd-complete.config.yml)
|
- [Internal overview](docs/internal/overview.md) — implemented component map
|
||||||
- [Maintained example input](examples/seriatim-minimal-transcript.json)
|
for maintainers.
|
||||||
|
- [Developer guide](docs/development.md) — contributor orientation and
|
||||||
|
validation guidance.
|
||||||
|
- [Future work](docs/roadmap/future.md) — unimplemented ideas and priorities.
|
||||||
|
|||||||
340
docs/cli.md
340
docs/cli.md
@@ -1,278 +1,146 @@
|
|||||||
# CLI Reference
|
# CLI Reference
|
||||||
|
|
||||||
This is the canonical reference for the implemented Notarius command-line
|
This is the canonical reference for the implemented Notarius command-line
|
||||||
interface.
|
interface. For the shortest successful run, see the [README](../README.md).
|
||||||
|
Configuration fields, discovery rules, and selectable module keys are defined
|
||||||
|
in [Configuration](config.md); runtime state and recovery procedures are
|
||||||
|
defined in [Operations](operations.md).
|
||||||
|
|
||||||
For the minimal end-to-end invocation, see the [README](../README.md).
|
## Command Summary
|
||||||
|
|
||||||
## Commands
|
~~~
|
||||||
|
|
||||||
```text
|
|
||||||
notarius help
|
notarius help
|
||||||
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--chunk_cache auto|bypass|refresh] [--output-dir path] [--resume] [--recompute-step step-id] [--debug [--debug-dir path]] [--llm-profile id] [--session-id id] [--reference selector=path] [--without-reference selector]
|
notarius run <pipeline-id> --input path/to/source.json [flags]
|
||||||
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
|
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
|
||||||
notarius pipelines list [--config path/to/config.yml] [--json]
|
notarius pipelines list [--config path/to/config.yml] [--json]
|
||||||
```
|
~~~
|
||||||
|
|
||||||
Running `notarius` with no arguments, `notarius help`, `notarius --help`, or
|
Running Notarius without arguments, or with **help**, **--help**, or **-h**,
|
||||||
`notarius -h` prints usage and exits successfully.
|
writes the command summary to standard output and exits with status 0.
|
||||||
|
|
||||||
## `run`
|
## run
|
||||||
|
|
||||||
`notarius run <pipeline-id>` executes a configured pipeline against one input
|
~~~
|
||||||
file.
|
notarius run <pipeline-id> --input path/to/source.json [flags]
|
||||||
|
~~~
|
||||||
|
|
||||||
Flags:
|
The **run** command executes the named pipeline for one input file. The
|
||||||
|
pipeline ID and **--input** are required.
|
||||||
|
|
||||||
- `--input path`: required source input file.
|
| Flag | Meaning |
|
||||||
- `--config path`: config file path. If omitted, Notarius uses the discovery
|
| --- | --- |
|
||||||
rules in [Configuration](config.md#discovery).
|
| **--config path** | Use this configuration file. When omitted, configuration discovery applies; see [Configuration](config.md). |
|
||||||
- `--only lane-a,lane-b`: run only the named artifact lanes. Values are
|
| **--input path** | Source input file to process. Required. |
|
||||||
comma-separated and must be non-empty. This retains its existing behavior for
|
| **--output-dir path** | Override the configured output root for this run. |
|
||||||
implicit single-step pipelines; explicit multi-step pipelines reject it
|
| **--chunk_cache auto\|bypass\|refresh** | Override chunk-plan cache handling for this run. |
|
||||||
rather than inferring dependency closure.
|
| **--resume** | Reuse compatible recorded checkpoints when checkpoint recording is enabled. |
|
||||||
- `--resume`: request checkpoint reuse for this invocation. Checkpoint recording
|
| **--recompute-step step-id** | With **--resume**, recompute the selected ordered step and its dependent lanes. It cannot be combined with **--only**. |
|
||||||
must be enabled in configuration. See
|
| **--debug** | Retain a debug bundle for this run. |
|
||||||
[Operations](operations.md#checkpoint-cache) for prerequisites and reuse
|
| **--debug-dir path** | Override the debug-bundle root. Requires **--debug**. |
|
||||||
behavior.
|
| **--only lane-a,lane-b** | Run only the selected comma-separated artifact lanes when that selection is valid for the configured pipeline. |
|
||||||
- `--recompute-step step-id`: with `--resume` and checkpoint recording enabled,
|
| **--llm-profile id** | Override effective LLM-capable module bindings with one configured profile. |
|
||||||
force the named ordered step and every transitive dependent lane to execute.
|
| **--session-id id** | Supply a non-empty prompt session identifier to LLM-backed module calls. |
|
||||||
Compatible required predecessors and unrelated lanes remain reusable. The
|
| **--reference selector=path** | Add or replace a file reference binding. Repeatable. |
|
||||||
value may identify an explicit step or the implicit single-step ID `default`;
|
| **--without-reference selector** | Remove a configured optional reference binding. Repeatable. |
|
||||||
it cannot be combined with `--only`.
|
|
||||||
- `--chunk_cache auto|bypass|refresh`: select chunk-plan reuse for this
|
|
||||||
invocation. `auto` reuses a valid plan by canonical source digest, `bypass`
|
|
||||||
performs no plan-cache I/O, and `refresh` regenerates and replaces a valid
|
|
||||||
plan only after chunk validation succeeds. See
|
|
||||||
[Configuration](config.md#state-surfaces) for the persistent setting, precedence,
|
|
||||||
and cache-root selection.
|
|
||||||
- `--output-dir path`: output root. Defaults to `./notarius-output`.
|
|
||||||
- `--debug`: allocate and retain one debug bundle for this invocation.
|
|
||||||
- `--debug-dir path`: debug-bundle root override. This flag requires `--debug`.
|
|
||||||
- `--llm-profile id`: override every effective LLM-capable pipeline module
|
|
||||||
binding with one Scriptorium profile ID. Validator-specific profiles are not
|
|
||||||
overridden.
|
|
||||||
- `--session-id id`: pass a stable prompt session identifier through LLM-backed
|
|
||||||
module calls.
|
|
||||||
- `--reference selector=path`: bind a reference path to a chunk, extractor,
|
|
||||||
merger, or normalizer reference slot. Repeatable.
|
|
||||||
- `--without-reference selector`: remove a configured optional reference binding.
|
|
||||||
Repeatable. It accepts the same selector forms as `--reference`, without
|
|
||||||
`=path`.
|
|
||||||
|
|
||||||
On success, the command prints the completed pipeline ID, normalized output and
|
**--chunk_cache** accepts only **auto**, **bypass**, or **refresh**.
|
||||||
rejected output counts, and the output directory. A debug-enabled run also
|
**--debug-dir**, **--output-dir**, **--session-id**, and
|
||||||
prints `debug=<bundle-path>`. If the run completes with warnings, the warning
|
**--recompute-step** reject explicit empty values. **--recompute-step**
|
||||||
count is printed to stderr.
|
requires **--resume**; checkpoint requirements and reuse behavior are
|
||||||
|
documented in [Operations](operations.md).
|
||||||
|
|
||||||
Reference flags are external file bindings resolved against selected chunk,
|
### Reference selectors
|
||||||
extractor, merger, and normalizer targets before the run starts. Generated
|
|
||||||
artifact bindings are configured in ordered steps and cannot be introduced by a
|
|
||||||
CLI path flag. Flat slot names are accepted only
|
|
||||||
when exactly one selected target declares that slot. For configured reference
|
|
||||||
bindings, precedence, path resolution, and validation, see
|
|
||||||
[Configuration](config.md#pipelines).
|
|
||||||
|
|
||||||
`--reference` binds or replaces one slot for one selected target. Selectors are:
|
Use **--reference** only for a reference slot declared by the selected
|
||||||
|
configured target. The accepted selector forms are:
|
||||||
|
|
||||||
- `slot=path`: valid when exactly one selected target declares `slot`;
|
| Form | Target |
|
||||||
- `chunk.slot=path`: target the chunker;
|
| --- | --- |
|
||||||
- `merge.slot=path`: valid when exactly one selected merger declares `slot`;
|
| slot=path | The unique selected target that declares slot. |
|
||||||
- `lane.slot=path`: valid when exactly one selected extractor, merger, or
|
| chunk.slot=path | The chunker. |
|
||||||
normalizer in that lane declares `slot`;
|
| merge.slot=path | The unique selected merger that declares slot. |
|
||||||
- `lane.extract.slot=path`: target a lane extractor;
|
| lane.slot=path | The unique extractor, merger, or normalizer in lane that declares slot. |
|
||||||
- `lane.merge.slot=path`: target a lane merger;
|
| lane.extract.slot=path | The extractor in lane. |
|
||||||
- `lane.normalize.slot=path`: target a lane normalizer.
|
| lane.merge.slot=path | The merger in lane. |
|
||||||
|
| lane.normalize.slot=path | The normalizer in lane. |
|
||||||
|
|
||||||
Use `slot=path` when the selected targets declare the slot unambiguously:
|
**--without-reference** uses the same selector forms without =path. Slot
|
||||||
|
names, requiredness, and configured bindings are part of the
|
||||||
|
[configuration contract](config.md).
|
||||||
|
|
||||||
```sh
|
### Run output
|
||||||
|
|
||||||
|
On success, standard output contains the completed pipeline ID, counts of
|
||||||
|
normalized and rejected outputs, and the output directory. A debug-enabled run
|
||||||
|
also prints its debug-bundle path to standard output. A successful run with
|
||||||
|
warnings reports the warning count to standard error. The published JSON
|
||||||
|
envelope is defined by the [JSON output contract](integrations/json-output.md).
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
~~~
|
||||||
|
OPENROUTER_API_KEY=your-api-key \
|
||||||
go run ./cmd/notarius run dnd-session \
|
go run ./cmd/notarius run dnd-session \
|
||||||
--config examples/dnd-minimal.config.yml \
|
--config examples/dnd-minimal.config.yml \
|
||||||
--input examples/seriatim-minimal-transcript.json \
|
--input examples/seriatim-minimal-transcript.json
|
||||||
--reference roster=./campaign-roster.txt
|
~~~
|
||||||
```
|
|
||||||
|
|
||||||
Use an explicit selector when multiple selected targets declare the same slot or
|
## config validate
|
||||||
when you want to target a specific target:
|
|
||||||
|
|
||||||
```sh
|
~~~
|
||||||
go run ./cmd/notarius run dnd-session \
|
notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b]
|
||||||
--config examples/dnd-minimal.config.yml \
|
~~~
|
||||||
--input examples/seriatim-minimal-transcript.json \
|
|
||||||
--reference spells.extract.glossary=./campaign-glossary.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
For the maintained ordered D&D workflow, use the explicit pipeline. The first
|
This command loads and validates a configuration. With **--pipeline**, it also
|
||||||
step produces normalized NPC and scene-description artifacts; the second step
|
resolves that pipeline against the production module catalog. **--only** selects
|
||||||
receives the NPC artifact in memory for spell extraction, combat extraction, and
|
lanes during that resolution and requires **--pipeline**.
|
||||||
combat normalization, and receives the required scene-description artifact for
|
|
||||||
combat eligibility:
|
|
||||||
|
|
||||||
```sh
|
Success is written to standard output as either config "<path>" is valid or
|
||||||
go run ./cmd/notarius run dnd-session \
|
config "<path>" is valid for pipeline "<pipeline-id>".
|
||||||
--config examples/dnd-complete.config.yml \
|
|
||||||
--input examples/seriatim-minimal-transcript.json \
|
|
||||||
--output-dir ./npc-grounded-output
|
|
||||||
```
|
|
||||||
|
|
||||||
The generated NPC content remains contextual grounding, not spell or combat
|
|
||||||
evidence. It is represented in manifests and debug summaries by bounded
|
|
||||||
identity and producer provenance, not by payload content or a filesystem path.
|
|
||||||
The scene-description artifact is control context: combat extraction calls its
|
|
||||||
LLM only for an exact `combat` scene match. See the
|
|
||||||
[D&D combat-turn artifact contract](integrations/dnd-combat-turn-artifacts.md)
|
|
||||||
for the resulting empty-output and warning behavior.
|
|
||||||
|
|
||||||
The same grammar can target chunk, merge, and normalize slots when the configured
|
|
||||||
modules declare them:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go run ./cmd/notarius run dnd-session \
|
|
||||||
--config path/to/config.yml \
|
|
||||||
--input examples/seriatim-minimal-transcript.json \
|
|
||||||
--reference chunk.scene_guide=./campaign-scenes.txt \
|
|
||||||
--reference spells.merge.merge_notes=./merge-notes.txt \
|
|
||||||
--reference spells.normalize.normalization_notes=./normalization-notes.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
Use `--without-reference` to remove a configured optional binding for a run:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go run ./cmd/notarius run dnd-session \
|
|
||||||
--config examples/dnd-minimal.config.yml \
|
|
||||||
--input examples/seriatim-minimal-transcript.json \
|
|
||||||
--without-reference glossary
|
|
||||||
```
|
|
||||||
|
|
||||||
Use `--session-id` when an external orchestrator needs all prompt calls from one
|
|
||||||
run to share an identifier:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go run ./cmd/notarius run dnd-session \
|
|
||||||
--config examples/dnd-minimal.config.yml \
|
|
||||||
--input examples/seriatim-minimal-transcript.json \
|
|
||||||
--session-id campaign-17-session-04
|
|
||||||
```
|
|
||||||
|
|
||||||
When `cache.checkpoints.enabled` is `true`, runs record checkpoints whether or
|
|
||||||
not `--resume` is present. Add the resume flag to load and reuse compatible
|
|
||||||
recorded work; using it while checkpoint recording is disabled is an error:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go run ./cmd/notarius run dnd-session \
|
|
||||||
--config examples/dnd-minimal.config.yml \
|
|
||||||
--input examples/seriatim-minimal-transcript.json \
|
|
||||||
--resume
|
|
||||||
```
|
|
||||||
|
|
||||||
To selectively rerun one ordered step and its dependent lanes, use the step ID
|
|
||||||
from the configuration. The selected step and dependents are reported as
|
|
||||||
`forced_recompute`; reusable predecessors are reported as `reused`:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go run ./cmd/notarius run dnd-session \
|
|
||||||
--config examples/dnd-complete.config.yml \
|
|
||||||
--input examples/seriatim-minimal-transcript.json \
|
|
||||||
--resume --recompute-step extract-events
|
|
||||||
```
|
|
||||||
|
|
||||||
Checkpoint decisions use these categories: `reused`, `executed`,
|
|
||||||
`forced_recompute`, and `dependency_invalidated`. The reason code and bounded
|
|
||||||
detail identify the decision without exposing reference content, local paths,
|
|
||||||
or secrets. `--recompute-step` requires checkpoint recording and `--resume`;
|
|
||||||
unknown step IDs, empty values, and combinations with `--only` are rejected.
|
|
||||||
The operator meanings of checkpoint reason codes are maintained in
|
|
||||||
[Operations](operations.md#resume-and-selective-recompute).
|
|
||||||
|
|
||||||
Use `--debug` to retain the redacted summary and trace bundle for one run. The
|
|
||||||
bundle is allocated before pipeline resolution; once allocated, its path is
|
|
||||||
also printed to stderr if the command fails. Debug-write failures cause exit
|
|
||||||
code `1`.
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go run ./cmd/notarius run dnd-session \
|
|
||||||
--config examples/dnd-minimal.config.yml \
|
|
||||||
--input examples/seriatim-minimal-transcript.json \
|
|
||||||
--debug --debug-dir ./notarius-debug
|
|
||||||
```
|
|
||||||
|
|
||||||
Use `refresh` when intentionally replacing the cached plan for the same source:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go run ./cmd/notarius run dnd-session \
|
|
||||||
--config examples/dnd-minimal.config.yml \
|
|
||||||
--input examples/seriatim-minimal-transcript.json \
|
|
||||||
--chunk_cache refresh
|
|
||||||
```
|
|
||||||
|
|
||||||
Use `bypass` for a one-off run that must not inspect or create plan-cache state:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go run ./cmd/notarius run dnd-session \
|
|
||||||
--config examples/dnd-minimal.config.yml \
|
|
||||||
--input examples/seriatim-minimal-transcript.json \
|
|
||||||
--chunk_cache bypass
|
|
||||||
```
|
|
||||||
|
|
||||||
`--diagnostics-dir` has been removed. For checkpoint behavior, durable output,
|
|
||||||
debug-bundle lifecycle, and failure inspection, see [Operations](operations.md).
|
|
||||||
|
|
||||||
## `config validate`
|
|
||||||
|
|
||||||
`notarius config validate` loads and validates configuration.
|
|
||||||
|
|
||||||
Flags:
|
|
||||||
|
|
||||||
- `--config path`: config file path. If omitted, Notarius uses the discovery
|
|
||||||
rules in [Configuration](config.md#discovery).
|
|
||||||
- `--pipeline pipeline-id`: additionally resolve one configured pipeline against
|
|
||||||
the production module catalog.
|
|
||||||
- `--only lane-a,lane-b`: validate resolution for selected artifact lanes. This
|
|
||||||
flag requires `--pipeline`.
|
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
|
|
||||||
```sh
|
~~~
|
||||||
go run ./cmd/notarius config validate \
|
|
||||||
--config examples/dnd-minimal.config.yml
|
|
||||||
|
|
||||||
go run ./cmd/notarius config validate \
|
go run ./cmd/notarius config validate \
|
||||||
--config examples/dnd-minimal.config.yml \
|
--config examples/dnd-minimal.config.yml \
|
||||||
--pipeline dnd-session \
|
--pipeline dnd-session
|
||||||
--only spells
|
|
||||||
```
|
|
||||||
|
|
||||||
## `pipelines list`
|
OPENROUTER_API_KEY=validation-placeholder \
|
||||||
|
go run ./cmd/notarius config validate \
|
||||||
|
--config examples/dnd-complete.config.yml \
|
||||||
|
--pipeline dnd-session
|
||||||
|
~~~
|
||||||
|
|
||||||
`notarius pipelines list` prints configured pipeline IDs in sorted order.
|
The placeholder in the second command is sufficient only for offline
|
||||||
|
validation; it cannot run a provider-backed pipeline.
|
||||||
|
|
||||||
Flags:
|
## pipelines list
|
||||||
|
|
||||||
- `--config path`: config file path. If omitted, Notarius uses the discovery
|
~~~
|
||||||
rules in [Configuration](config.md#discovery).
|
notarius pipelines list [--config path/to/config.yml] [--json]
|
||||||
- `--json`: print `{"pipelines":[...]}` instead of one ID per line.
|
~~~
|
||||||
|
|
||||||
Examples:
|
This command lists configured pipeline IDs in sorted order. By default, it
|
||||||
|
writes one ID per line to standard output. **--json** writes an object shaped as
|
||||||
|
{"pipelines":[...]} instead.
|
||||||
|
|
||||||
```sh
|
~~~
|
||||||
go run ./cmd/notarius pipelines list \
|
go run ./cmd/notarius pipelines list \
|
||||||
--config examples/dnd-minimal.config.yml
|
--config examples/dnd-minimal.config.yml
|
||||||
|
~~~
|
||||||
|
|
||||||
go run ./cmd/notarius pipelines list \
|
## Output Streams And Exit Statuses
|
||||||
--config examples/dnd-minimal.config.yml \
|
|
||||||
--json
|
|
||||||
```
|
|
||||||
|
|
||||||
## Exit Codes
|
Successful commands write their primary result to standard output. Warnings and
|
||||||
|
errors are written to standard error.
|
||||||
|
|
||||||
- `0`: command succeeded.
|
| Status | Meaning |
|
||||||
- `1`: command syntax was valid, but loading config, resolving modules, running
|
| --- | --- |
|
||||||
the pipeline, calling the provider, writing output, or writing a requested
|
| 0 | The command completed successfully, including root help. |
|
||||||
debug bundle failed.
|
| 1 | Command syntax was valid but configuration loading or validation, pipeline resolution or execution, provider use, output, or requested debug handling failed. |
|
||||||
- `2`: command syntax was invalid, a command was unknown, a required argument
|
| 2 | The command or flag syntax was invalid, including unknown commands, missing required arguments, invalid flag values, or invalid flag combinations. |
|
||||||
was missing, or a flag value was malformed.
|
|
||||||
|
|
||||||
For YAML structure, defaults, Scriptorium profile sources, environment
|
The root help spellings are the supported help path. Invoking **--help** on
|
||||||
overrides, and selectable module and validator keys, see
|
**run**, **config validate**, or **pipelines list** is handled by the flag
|
||||||
[Configuration](config.md).
|
parser as a usage error: it writes an error to standard error and exits with
|
||||||
|
status 2.
|
||||||
|
|||||||
1065
docs/config.md
1065
docs/config.md
File diff suppressed because it is too large
Load Diff
@@ -17,11 +17,12 @@ implemented component map.
|
|||||||
| Application shape, package boundaries, contracts, dependency direction, runtime guarantees, or safety properties | [Architecture](policy/architecture.md) and relevant [ADRs](adr/) | Architecture defines the intended system and its invariants; ADRs preserve significant decision rationale. |
|
| Application shape, package boundaries, contracts, dependency direction, runtime guarantees, or safety properties | [Architecture](policy/architecture.md) and relevant [ADRs](adr/) | Architecture defines the intended system and its invariants; ADRs preserve significant decision rationale. |
|
||||||
| Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical homes, audiences, current-behavior rules, and maintenance requirements. |
|
| Any documentation addition or revision | [Documentation Policy](policy/documentation.md) | It defines canonical homes, audiences, current-behavior rules, and maintenance requirements. |
|
||||||
| Adding, changing, reviewing, or deleting tests | [Testing Policy](policy/testing.md) | It defines risk-based sufficiency, durable test boundaries, test-double guidance, and criteria for retaining tests. |
|
| Adding, changing, reviewing, or deleting tests | [Testing Policy](policy/testing.md) | It defines risk-based sufficiency, durable test boundaries, test-double guidance, and criteria for retaining tests. |
|
||||||
|
| CLI composition or command behavior | [CLI Internals](internal/cli.md) and [CLI Reference](cli.md) | The internal guide owns composition and command flow; the reference owns public syntax. |
|
||||||
|
| Configuration loading, resolution, or user-visible configuration behavior | [Configuration Internals](internal/configuration.md) and [Configuration](config.md) | The internal guide owns loading and resolution mechanics; the reference owns the configuration contract. |
|
||||||
| Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
|
| Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
|
||||||
| Production modules or validators | [Module Internals](internal/modules.md) | It documents implemented module contracts, capabilities, assets, and registration. |
|
| Production modules or validators | [Module Internals](internal/modules.md), [D&D Module Internals](internal/dnd.md), and [D&D integration contracts](integrations/) | The generic guide owns extension mechanics, the D&D guide owns shared family conventions, and the contracts own durable output shapes. |
|
||||||
| LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and Scriptorium integration. |
|
| LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and Scriptorium integration. |
|
||||||
| Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. |
|
| Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. |
|
||||||
| CLI or user-visible configuration behavior | [CLI Reference](cli.md) and [Configuration](config.md) | These are the canonical user and operator references. |
|
|
||||||
| External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. |
|
| External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. |
|
||||||
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
|
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
|
||||||
|
|
||||||
|
|||||||
@@ -1,83 +1,81 @@
|
|||||||
# Accepted Chunk Map
|
# Accepted Chunk Map
|
||||||
|
|
||||||
This document defines the durable accepted chunk-map artifact that the JSON
|
This document defines the optional durable `chunk-map.json` artifact in a
|
||||||
output encoder can write as `chunk-map.json`. It describes the exact accepted,
|
[published JSON bundle](json-output.md). It describes the accepted,
|
||||||
materialized chunks used by a run; it is not a lane artifact and is never an
|
materialized chunk plan used by one run. It is not a lane payload and is never
|
||||||
input to later pipeline steps. Enable it with the JSON output option described
|
an input to a later pipeline step.
|
||||||
in [Configuration](../config.md#json-output-options).
|
|
||||||
|
|
||||||
## Identity
|
## Contract Identity
|
||||||
|
|
||||||
- Artifact kind: `source/chunk-map`
|
| Property | Value |
|
||||||
- Logical file: `chunk-map.json`
|
| --- | --- |
|
||||||
- Schema ID: `notarius.source.chunk_map`
|
| Artifact kind | `source/chunk-map` |
|
||||||
- Schema name: `notarius_source_chunk_map_v1`
|
| Logical file | `chunk-map.json` |
|
||||||
- Schema version: `v1`
|
| Media type | `application/json` |
|
||||||
- Media type: `application/json`
|
| Schema ID | `notarius.source.chunk_map` |
|
||||||
|
| Schema name | `notarius_source_chunk_map_v1` |
|
||||||
|
| Schema version | `v1` |
|
||||||
|
|
||||||
The checked-in [JSON Schema](../../internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json)
|
The optional `chunk_map` descriptor in `index.json` identifies this artifact.
|
||||||
defines the strict wire shape. `chunk-map.json` is listed by the optional
|
Export is controlled by the JSON output binding described in
|
||||||
`chunk_map` descriptor in [the JSON output index](json-output.md#indexjson),
|
[Configuration](../config.md#module-bindings-and-validators).
|
||||||
not by the lane-oriented `output_files` collection.
|
|
||||||
|
|
||||||
## Payload
|
## Wire Shape
|
||||||
|
|
||||||
The payload has these required fields:
|
Every payload has these required fields:
|
||||||
|
|
||||||
- `source_id`: accepted source-document identity.
|
| Field | Meaning |
|
||||||
- `source_digest`: canonical lower-case `sha256:` digest of that document.
|
| --- | --- |
|
||||||
- `plan_digest`: canonical lower-case `sha256:` digest of the accepted logical
|
| `source_id` | Accepted source-document identity. |
|
||||||
plan.
|
| `source_digest` | Lower-case `sha256:` digest of that source document. |
|
||||||
- `requested_chunker`: chunk module selected by the current resolved pipeline.
|
| `plan_digest` | Lower-case `sha256:` digest of the logical chunk plan. |
|
||||||
- `producer`: the original accepted-plan producer, with required
|
| `requested_chunker` | Chunk module selected by the resolved pipeline. |
|
||||||
`input_module` and `chunk_module`; `llm_profile` is present only for an
|
| `producer` | Original accepted-plan producer. `input_module` and `chunk_module` are required; `llm_profile` is optional. |
|
||||||
LLM-backed producer.
|
| `plan_annotations` | Plan-level annotation namespace map; `{}` when none are present. |
|
||||||
- `plan_annotations`: accepted plan-level annotation namespace map. It is
|
| `chunks` | Non-empty execution-order chunk collection. |
|
||||||
`{}` when no namespaces are present.
|
|
||||||
- `chunks`: non-empty execution-order collection of accepted chunks.
|
|
||||||
|
|
||||||
Each chunk has `id`, zero-based `index`, `source_ref`, positive `unit_count`,
|
Each `chunks` entry contains non-empty `id`, zero-based `index`, `source_ref`,
|
||||||
and an explicit `annotations` namespace map. A source reference has the source
|
positive `unit_count`, and an explicit `annotations` map. `source_ref` contains
|
||||||
ID and inclusive positive `start_unit_id` and `end_unit_id` endpoints.
|
the same `source_id` as the top-level value plus positive inclusive
|
||||||
Annotation values are arbitrary valid JSON under non-empty namespaces. They
|
`start_unit_id` and `end_unit_id` values. Endpoints identify source units; their
|
||||||
are preserved as canonical JSON without interpreting any module-specific
|
numeric values do not by themselves establish source-document order.
|
||||||
namespace.
|
|
||||||
|
|
||||||
## Invariants
|
Annotation namespaces are non-empty trimmed strings. Their values are arbitrary
|
||||||
|
valid JSON and are retained without interpreting a module-specific namespace.
|
||||||
|
|
||||||
The framework constructs this artifact only after materializing the selected
|
## Ordering And Validation
|
||||||
logical plan and accepting it through the configured chunk validator chain.
|
|
||||||
Construction proves the source and plan digests, source-document range order,
|
|
||||||
materialized chunk IDs and indexes, source references, unit membership and
|
|
||||||
counts, and plan/range annotations agree exactly. Chunk IDs are unique and
|
|
||||||
indexes are contiguous and agree with array order.
|
|
||||||
|
|
||||||
The codec rejects unknown fixed-object fields, malformed identities or
|
`chunks` are in execution order. Their indexes are contiguous, start at zero,
|
||||||
digests, invalid annotation JSON, trailing JSON content, and any payload whose
|
and equal their array positions; chunk IDs are unique. The emitted map is built
|
||||||
reconstructed logical plan does not reproduce `plan_digest`. It makes
|
only after the selected plan has been accepted and materialized against the
|
||||||
defensive copies at serialization and decoding boundaries.
|
source document, so its ranges, unit counts, annotations, and digests describe
|
||||||
|
that exact plan.
|
||||||
|
|
||||||
## Acceptance And Provenance
|
The codec rejects malformed JSON, trailing content, unknown fixed-object
|
||||||
|
fields, invalid identities or digests, invalid annotations, duplicate chunk
|
||||||
|
IDs, non-contiguous indexes, and a `plan_digest` that does not match the
|
||||||
|
reconstructed logical plan. The checked-in
|
||||||
|
[schema](../../internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json)
|
||||||
|
defines the strict JSON shape.
|
||||||
|
|
||||||
The artifact is available only when the chunk plan was accepted. It remains
|
## Valid Example
|
||||||
available when a later extraction, merge, or normalization result is rejected;
|
|
||||||
it is absent when chunk validation rejects the candidate plan.
|
|
||||||
|
|
||||||
`requested_chunker` describes the current pipeline selection. `producer`
|
The compact
|
||||||
describes who originally produced the accepted plan. On a cache hit these can
|
[source chunk-map fixture](../../internal/framework/chunkmap/testdata/source_chunk_map.v1.json)
|
||||||
differ: the accepted ranges, annotations, digests, and stable materialized IDs
|
is decoded by the production codec and demonstrates an accepted map with
|
||||||
are reused, while the producer remains the stored producer. Cache paths,
|
annotations, producer identity, and ordered chunks.
|
||||||
actions, references, metadata, warnings, timestamps, and detailed provenance
|
|
||||||
remain in the run manifest rather than this payload.
|
|
||||||
|
|
||||||
## Data Handling
|
## Publication And Compatibility
|
||||||
|
|
||||||
The map contains structure, not source content. It excludes transcript bytes,
|
The map is present only when a chunk plan was accepted and its export is
|
||||||
materialized units, source-unit metadata, chunk content, private model
|
enabled. It remains publishable if a later lane is rejected, but is absent when
|
||||||
responses, rejected proposals, debug data, external-reference content, and
|
chunk-plan validation rejects the plan. `requested_chunker` identifies the
|
||||||
filesystem paths.
|
current pipeline selection, while `producer` identifies the component that
|
||||||
|
originally produced the accepted plan; they may differ when an accepted plan is
|
||||||
|
reused.
|
||||||
|
|
||||||
Annotations can nevertheless be source- or model-derived. Treat an enabled
|
The map contains structure rather than source content: it excludes transcript
|
||||||
`chunk-map.json` with the same sensitivity and retention expectations as lane
|
bytes, source-unit metadata, chunk text, private model output, reference
|
||||||
output. Physical placement, confined atomic writing, and permissions follow
|
content, debug data, and filesystem paths. Treat the exported map with the
|
||||||
the ordinary [output operation](../operations.md#output).
|
same care as other published output. Publication location and retention are
|
||||||
|
defined in [Operations](../operations.md#output-bundles).
|
||||||
|
|||||||
@@ -1,10 +1,10 @@
|
|||||||
# D&D Combat-Turn Artifact Contract
|
# D&D Combat-Turn Artifact
|
||||||
|
|
||||||
This document defines the durable artifact, serialization, extraction,
|
This contract defines the durable combat-action occurrence list produced by
|
||||||
candidate-validation, normalization, and production lane boundaries for D&D
|
`dnd/combat-turns`. It records source-grounded turns and actions; it is not a
|
||||||
combat turns.
|
complete initiative tracker, combat summary, or state model.
|
||||||
|
|
||||||
## Artifact identity
|
## Identity and compatibility
|
||||||
|
|
||||||
| Property | Value |
|
| Property | Value |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
@@ -14,167 +14,56 @@ combat turns.
|
|||||||
| Schema version | `v1` |
|
| Schema version | `v1` |
|
||||||
| Media type | `application/json` |
|
| Media type | `application/json` |
|
||||||
|
|
||||||
The top-level JSON object contains the required `combat_turns` array, which
|
`v1` is a strict JSON object with required `combat_turns`; the array may be
|
||||||
may be empty. Every object rejects unknown fields.
|
empty. Turn and source-reference objects reject unknown fields. A future
|
||||||
|
incompatible shape requires a new schema version.
|
||||||
|
|
||||||
## JSON shape
|
## Wire shape
|
||||||
|
|
||||||
Each combat turn contains these required fields:
|
Each combat turn has these required fields:
|
||||||
|
|
||||||
| Field | Shape |
|
| Field | Contract |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `actor` | Non-empty string. |
|
| `actor` | Non-empty acting character or creature name. |
|
||||||
| `turn_kind` | One of `turn`, `reaction`, `legendary_action`, `lair_action`, or `other`. |
|
| `turn_kind` | `turn`, `reaction`, `legendary_action`, `lair_action`, or `other`. |
|
||||||
| `source_refs` | Required array with at least one source reference. |
|
| `source_refs` | One or more transcript evidence ranges. |
|
||||||
|
|
||||||
Source references use the shared source-reference shape:
|
Each source reference has exactly `source_id`, `start_unit_id`, and
|
||||||
|
`end_unit_id`. It identifies an inclusive current-transcript range; unit IDs
|
||||||
|
are positive and the start may not follow the end.
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"source_id": "session-alpha",
|
"combat_turns": [
|
||||||
"start_unit_id": 1,
|
{
|
||||||
"end_unit_id": 2
|
"actor": "Mira Thorn",
|
||||||
|
"turn_kind": "turn",
|
||||||
|
"source_refs": [
|
||||||
|
{"source_id": "session-7", "start_unit_id": 31, "end_unit_id": 32}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
`source_id` must be non-empty and both unit IDs must be positive integers. The
|
## Eligibility, evidence, and normalized form
|
||||||
codec does not resolve references against a source document or enforce source
|
|
||||||
range ordering; those checks belong to the later source-reference validation
|
|
||||||
boundary.
|
|
||||||
|
|
||||||
## Codec behavior
|
The extractor requires an approved [scene-description artifact](dnd-scene-description-artifacts.md).
|
||||||
|
It emits combat turns only for a chunk with an exact matching scene classified
|
||||||
|
`combat`; an exact non-combat scene produces an accepted empty list. The scene
|
||||||
|
record controls eligibility only: its title, summary, and reference do not
|
||||||
|
become turn evidence. No exact matching scene also produces an empty list and
|
||||||
|
the `scene_classification_unavailable` warning.
|
||||||
|
|
||||||
The codec exposes two representations of the same typed artifact:
|
An optional normalized [NPC artifact](dnd-npc-artifacts.md) can ground an
|
||||||
|
actor name. Its registry references are provenance, never combat evidence.
|
||||||
|
Normalization trims and, where possible, canonicalizes actor names; orders and
|
||||||
|
deduplicates exact source references; orders valid-evidence turns by source
|
||||||
|
chronology; and collapses only duplicates with the same actor identity, turn
|
||||||
|
kind, and complete valid evidence. It does not infer turns, initiative, or
|
||||||
|
actions from registry or scene data.
|
||||||
|
|
||||||
- Candidate encode/decode preserves invalid actor and turn-kind values,
|
The [NPC-interaction artifact](dnd-npc-interaction-artifacts.md) records
|
||||||
collection presence, and source references so later validators can report
|
broader NPC occurrences. The [JSON output contract](json-output.md) defines
|
||||||
them. Candidate decoding still requires valid JSON, one JSON value, known
|
publication, and [D&D module internals](../internal/dnd.md) describes routing
|
||||||
fields, and compatible JSON types.
|
and validation mechanics.
|
||||||
- Approved encode/decode enforces the structural rules in this contract.
|
|
||||||
|
|
||||||
The codec owns the durable JSON Schema, whose object layers all set
|
|
||||||
`additionalProperties` to `false`. Codec metadata contains only
|
|
||||||
`combat_turn_count`.
|
|
||||||
|
|
||||||
The maintained compact fixture is
|
|
||||||
`internal/modules/dnd/codec/combatturns/testdata/dnd_combat_turns.v1.json`.
|
|
||||||
|
|
||||||
## Extraction boundary
|
|
||||||
|
|
||||||
The standalone extractor uses these identities:
|
|
||||||
|
|
||||||
| Property | Value |
|
|
||||||
| --- | --- |
|
|
||||||
| Extractor key | `dnd/combat-turns` |
|
|
||||||
| Capability | `dnd.combat_turns` |
|
|
||||||
| Prompt ID | `dnd.combat_turns` |
|
|
||||||
| Prompt version | `v1` |
|
|
||||||
| Private response-schema key | `dnd_combat_turns_llm` |
|
|
||||||
| Private response-schema ID | `notarius.dnd.combat_turns.llm` |
|
|
||||||
| Default profile | `gemini-2-flash` |
|
|
||||||
|
|
||||||
It requires `chunks` and `source.transcript`, accepts no options, and requires
|
|
||||||
one `scene_descriptions` reference. That reference must be exactly one approved
|
|
||||||
`dnd/scene-description-list` artifact with media type `application/json` and a
|
|
||||||
maximum size of 1 MiB (1048576 bytes). It may be an external file validated
|
|
||||||
during preparation or a canonical generated artifact supplied at an ordered
|
|
||||||
step handoff. An unbound slot is a configuration error.
|
|
||||||
|
|
||||||
The scene artifact controls eligibility, not evidence. The extractor calls the
|
|
||||||
LLM only when exactly one record has the current chunk's ID, source ID, start
|
|
||||||
unit ID, and end unit ID, and that record has `kind: combat`. An exact
|
|
||||||
`narrative`, `recap`, or `meta` record returns an accepted empty
|
|
||||||
`combat_turns` array without an LLM call, warning, or retry attempt. A missing
|
|
||||||
or mismatched exact record returns the same accepted empty result without an
|
|
||||||
LLM call or retry and emits one content-safe
|
|
||||||
`scene_classification_unavailable` warning. The scene artifact, its title and
|
|
||||||
summary, and its source references are never copied into combat turns.
|
|
||||||
|
|
||||||
For an eligible combat chunk, the prompt receives the chunk-scoped transcript
|
|
||||||
plus the existing `players`, `party`, and `glossary` inputs, and optionally the
|
|
||||||
deprecated `roster` reference through the shared party mapping. The optional
|
|
||||||
`npcs` reference is an approved normalized NPC artifact used only for identity
|
|
||||||
grounding; it never supplies combat evidence. `scene_descriptions` is never a
|
|
||||||
combat prompt input.
|
|
||||||
|
|
||||||
The private response envelope has the same turn fields and JSON types as the
|
|
||||||
durable shape except that source references contain only `start_unit_id`
|
|
||||||
and `end_unit_id`. It enforces required field presence, types, and
|
|
||||||
unknown-field rejection, while deterministic validators own enum membership,
|
|
||||||
non-empty values and collections, and positive-number requirements. The
|
|
||||||
extractor assigns the current source ID, removes exact duplicate ranges, and
|
|
||||||
stable-sorts turns by the earliest valid source-document position. Numeric unit
|
|
||||||
IDs are identifiers; source-document slice position determines chronology.
|
|
||||||
Semantically malformed candidate fields remain in the typed result for the
|
|
||||||
configured validation and retry boundary.
|
|
||||||
|
|
||||||
## Deterministic candidate validation
|
|
||||||
|
|
||||||
The standalone validator keys are:
|
|
||||||
|
|
||||||
| Validator | Responsibility |
|
|
||||||
| --- | --- |
|
|
||||||
| `extract/dnd/combat-turns/shape` | Required list, actor, turn kind, and source references, plus supported turn-kind values. |
|
|
||||||
| `extract/dnd/combat-turns/source_refs` | Source identity, source-unit existence, and range order through the source document. |
|
|
||||||
| `extract/dnd/combat-turns/source_relatedness` | At most one advisory warning per turn when the actor is not related to cited transcript text. |
|
|
||||||
|
|
||||||
Source-reference and relatedness validators defer malformed shape to the shape
|
|
||||||
validator. Relatedness also defers when any cited source range is invalid. It
|
|
||||||
combines overlapping cited ranges once in document order and compares actors
|
|
||||||
with the shared Unicode-aware NPC identity policy.
|
|
||||||
|
|
||||||
The production D&D registrar exposes the extractor and these validators. Its
|
|
||||||
default extraction chain preserves this order: JSON syntax, combat shape,
|
|
||||||
source references, private response schema, then source relatedness.
|
|
||||||
|
|
||||||
## Normalization boundary
|
|
||||||
|
|
||||||
The standalone normalizer uses key `dnd/combat-turns`, requires `merged`,
|
|
||||||
provides `normalized`, accepts no options, and accepts only the optional
|
|
||||||
structured `npcs` reference. Campaign references are LLM extraction context and
|
|
||||||
are not normalizer inputs. For an external file, the NPC registry is resolved
|
|
||||||
during preparation; for a generated binding, it is resolved at the operation-
|
|
||||||
time handoff. Runtime normalization uses that immutable prepared or handed-off
|
|
||||||
view.
|
|
||||||
|
|
||||||
Normalization policy is `dnd.combat_turns.normalize.v1`. It display-normalizes
|
|
||||||
the actor, canonicalizes exact registry actor matches, orders and deduplicates
|
|
||||||
exact source references, stable-sorts records by earliest valid source-document
|
|
||||||
position, and collapses only records with the same actor identity, turn kind,
|
|
||||||
and complete valid evidence set. The first normalized record is retained.
|
|
||||||
Invalid evidence is never eligible for duplicate collapse. Every mutation and
|
|
||||||
collapse emits a bounded warning using the merged input index in its scope.
|
|
||||||
|
|
||||||
The normalizer reports `normalization_policy` and `identity_policy` metadata
|
|
||||||
and fingerprints. An external registry may additionally contribute
|
|
||||||
`npc_registry_digest` and `npc_count`; generated registry identity is retained
|
|
||||||
in framework handoff provenance and dependency fingerprints. The
|
|
||||||
normalized-invariants validator is
|
|
||||||
`normalize/dnd/combat-turns/invariants`; it defers shape and source-reference
|
|
||||||
failures, then checks actor display normalization, canonical evidence ordering,
|
|
||||||
chronology, and duplicate identity. It rejects
|
|
||||||
with `invalid_combat_turn_normalization` under policy
|
|
||||||
`dnd.combat_turns.validator.normalized.v1`.
|
|
||||||
|
|
||||||
The production D&D registrar exposes the normalizer and normalized-invariants
|
|
||||||
validator. Its default normalization chain is JSON syntax, combat shape,
|
|
||||||
normalized invariants, source references, durable schema, then source
|
|
||||||
relatedness. The lane uses the framework's typed append-order merger and has no
|
|
||||||
merge validator chain.
|
|
||||||
|
|
||||||
## Production manifest and references
|
|
||||||
|
|
||||||
The selectable lane uses extractor and normalizer key `dnd/combat-turns`,
|
|
||||||
`appendorder` for the typed merger, and the durable codec above. Bound external
|
|
||||||
references contribute raw-file provenance to the run manifest. Generated
|
|
||||||
bindings contribute artifact kind, schema identity, media type, canonical
|
|
||||||
digest, size, and bounded producer provenance. This includes the generated
|
|
||||||
scene-description artifact for the combat extractor. Consumer metadata and
|
|
||||||
checkpoint fingerprints contain no reference names, content, paths, source
|
|
||||||
ranges, scene titles, or scene summaries. For an external scene artifact,
|
|
||||||
component metadata records the gate policy plus a semantic eligibility digest
|
|
||||||
and record count; generated scene identity remains in framework handoff
|
|
||||||
provenance and dependencies. The eligibility digest changes with scene ID,
|
|
||||||
exact source range, or kind, but not with title or summary. The normalized lane
|
|
||||||
is emitted as `lanes/<lane-id>.json` by the JSON output module, and warnings
|
|
||||||
and rejection summaries remain in their shared companion files.
|
|
||||||
|
|||||||
@@ -1,133 +1,78 @@
|
|||||||
# D&D Item-Event Artifact Contract
|
# D&D Item-Event Artifact
|
||||||
|
|
||||||
This document defines the durable D&D item-event artifact and its production
|
This contract defines the durable item and currency occurrence list produced by
|
||||||
boundaries. It records source-grounded discoveries and possession changes; it
|
`dnd/item-events`. It records source-grounded discoveries and possession
|
||||||
does not maintain an inventory or ledger.
|
changes; it does not maintain an inventory, balance, or ledger.
|
||||||
|
|
||||||
## Artifact Identity
|
## Identity and compatibility
|
||||||
|
|
||||||
| Property | Value |
|
| Property | Value |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| Extractor key | `dnd/item-events` |
|
|
||||||
| Extract capability | `dnd.item_events` |
|
|
||||||
| Artifact kind | `dnd/item-event-list` |
|
| Artifact kind | `dnd/item-event-list` |
|
||||||
| Schema ID | `notarius.dnd.item_events` |
|
| Schema ID | `notarius.dnd.item_events` |
|
||||||
| Schema name | `notarius_dnd_item_events_v1` |
|
| Schema name | `notarius_dnd_item_events_v1` |
|
||||||
| Schema version | `v1` |
|
| Schema version | `v1` |
|
||||||
| Media type | `application/json` |
|
| Media type | `application/json` |
|
||||||
| Normalizer key | `dnd/item-events` |
|
|
||||||
|
|
||||||
The payload is one strict JSON object containing a required `events` array,
|
`v1` is a strict JSON object with required `events`; the array may be empty.
|
||||||
which may be empty. Objects reject unknown fields.
|
Event and source-reference objects reject unknown fields. A future incompatible
|
||||||
|
shape requires a new schema version.
|
||||||
|
|
||||||
## Event Shape And Categories
|
## Wire shape
|
||||||
|
|
||||||
Every event has `name`, `kind`, and a non-empty `source_refs` array. `quantity`
|
Every event has required `name`, `kind`, and `source_refs`. `quantity`, `from`,
|
||||||
is optional and, when present, is a positive integer. `from` and `to` are
|
and `to` are optional where the event kind permits them.
|
||||||
optional display values whose presence depends on `kind`.
|
|
||||||
|
|
||||||
| Field | Rule |
|
| Field | Contract |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `name` | Non-empty, transcript-supported display value. |
|
| `name` | Non-empty item or currency display name. |
|
||||||
| `kind` | One of the categories below. |
|
| `kind` | `discovered`, `acquired`, `lost`, `consumed`, or `transferred`. |
|
||||||
| `quantity` | Optional positive integer; omitted when the source does not establish a count. |
|
| `quantity` | Optional positive integer; omit it when no count is established. |
|
||||||
| `from` | Holder that loses possession when the category permits it. |
|
| `from` | Optional non-empty losing holder, when allowed by `kind`. |
|
||||||
| `to` | Holder that gains possession when the category permits it. |
|
| `to` | Optional non-empty gaining holder, when allowed by `kind`. |
|
||||||
| `source_refs` | One or more current-source references. |
|
| `source_refs` | One or more transcript evidence ranges. |
|
||||||
|
|
||||||
| Kind | Meaning and holder rule |
|
Each source reference has exactly `source_id`, `start_unit_id`, and
|
||||||
| --- | --- |
|
`end_unit_id`. It identifies an inclusive current-transcript range; unit IDs
|
||||||
| `discovered` | The party learns of or encounters an item without established possession; neither holder is present. |
|
are positive and the start may not follow the end.
|
||||||
| `acquired` | A party member or `party` gains possession; `to` is required and `from` is absent. |
|
|
||||||
| `lost` | A party member or `party` ceases to possess an item without consuming it; `from` is required and `to` is absent. |
|
|
||||||
| `consumed` | Use depletes, expends, or destroys an item; `from` is required and `to` is absent. |
|
|
||||||
| `transferred` | Possession moves between distinct party members; both holders are required and neither may be `party`. |
|
|
||||||
|
|
||||||
`party` is the reserved display holder for collective party possession when an
|
|
||||||
individual holder is not established. Its comparison is case- and
|
|
||||||
Unicode-insensitive. Transfers require distinct normalized holder values.
|
|
||||||
Giving an item to an NPC, spending currency, selling an item, or another move
|
|
||||||
outside party possession is `lost`, not `transferred` or `consumed`. Monetary
|
|
||||||
spending, purchases, and payments are always `lost`. Currency is `consumed` only
|
|
||||||
when the source explicitly describes its physical destruction or expenditure as
|
|
||||||
a non-payment component. Ordinary non-depleting use is not an event.
|
|
||||||
|
|
||||||
Currency is represented as an ordinary event name plus an explicit quantity
|
|
||||||
when the transcript supplies one. Each denomination remains separate. The
|
|
||||||
artifact never converts denominations, infers a missing count, calculates a
|
|
||||||
balance, or sums nearby events.
|
|
||||||
|
|
||||||
## Source Evidence And Normalization
|
|
||||||
|
|
||||||
Each source reference contains `source_id`, `start_unit_id`, and `end_unit_id`.
|
|
||||||
It must identify an ordered range in the current source document. During
|
|
||||||
extraction, every cited range must also be wholly contained in the current
|
|
||||||
accepted chunk. Campaign references may disambiguate names, but never provide
|
|
||||||
event evidence.
|
|
||||||
|
|
||||||
The deterministic normalizer trims only display-edge whitespace in names and
|
|
||||||
holders, canonicalizes source-reference order and exact duplicate references,
|
|
||||||
then orders events by earliest valid source position and stable tie-breakers
|
|
||||||
over name, kind, holders, quantity, and complete evidence. It removes only
|
|
||||||
events with the same normalized fields and complete valid evidence sequence.
|
|
||||||
Invalid evidence is never collapsed. It does not rename, singularize, resolve
|
|
||||||
aliases, infer holders or quantities, or merge nearby events.
|
|
||||||
|
|
||||||
The default extraction validation chain is JSON syntax, item-event shape,
|
|
||||||
source references, durable JSON Schema, then advisory source relatedness. The
|
|
||||||
normalization chain adds normalized invariants after shape validation and before
|
|
||||||
source references and schema validation. Relatedness warnings are advisory so
|
|
||||||
contextual names and currency notation do not reject otherwise valid evidence.
|
|
||||||
|
|
||||||
## Representative JSON
|
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"events": [
|
"events": [
|
||||||
{
|
|
||||||
"name": "Hidden Cache",
|
|
||||||
"kind": "discovered",
|
|
||||||
"source_refs": [{"source_id": "session-7", "start_unit_id": 1, "end_unit_id": 1}]
|
|
||||||
},
|
|
||||||
{
|
{
|
||||||
"name": "Silver Pieces",
|
"name": "Silver Pieces",
|
||||||
"kind": "acquired",
|
"kind": "acquired",
|
||||||
"quantity": 20,
|
"quantity": 20,
|
||||||
"to": "party",
|
"to": "party",
|
||||||
"source_refs": [{"source_id": "session-7", "start_unit_id": 2, "end_unit_id": 2}]
|
"source_refs": [
|
||||||
},
|
{"source_id": "session-7", "start_unit_id": 2, "end_unit_id": 2}
|
||||||
{
|
]
|
||||||
"name": "Torch",
|
|
||||||
"kind": "lost",
|
|
||||||
"from": "party",
|
|
||||||
"source_refs": [{"source_id": "session-7", "start_unit_id": 3, "end_unit_id": 3}]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "Healing Potion",
|
|
||||||
"kind": "consumed",
|
|
||||||
"from": "Aria",
|
|
||||||
"source_refs": [{"source_id": "session-7", "start_unit_id": 4, "end_unit_id": 4}]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "Moonblade",
|
|
||||||
"kind": "transferred",
|
|
||||||
"from": "Aria",
|
|
||||||
"to": "Borin",
|
|
||||||
"source_refs": [{"source_id": "session-7", "start_unit_id": 5, "end_unit_id": 5}]
|
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
## Production Boundary
|
## Holder rules and minimal extraction
|
||||||
|
|
||||||
The extractor requires `chunks` and `source.transcript`, and accepts optional
|
`discovered` has neither holder; `acquired` requires `to` and forbids `from`;
|
||||||
`glossary`, `party`, `players`, and deprecated `roster` campaign references for
|
`lost` and `consumed` require `from` and forbid `to`; `transferred` requires
|
||||||
disambiguation only. It has no generated NPC, scene-description, or item-registry
|
both holders. `party` denotes collective possession. A transfer cannot use
|
||||||
dependency. The append-order merger preserves chunk order, and the normalizer
|
`party` for either holder and its two normalized holders must differ.
|
||||||
has no reference slots.
|
|
||||||
|
|
||||||
The normalized lane is emitted as `lanes/<lane-id>.json` by the JSON output
|
Only an evidenced discovery or possession change belongs in this artifact.
|
||||||
module. See [Configuration](../config.md#implemented-production-modules) for
|
It does not infer quantities or holders, convert currency denominations,
|
||||||
the selectable keys and default chains, and the
|
calculate balances, or merge nearby events. Campaign references may
|
||||||
[JSON output contract](json-output.md) for bundle paths.
|
disambiguate names but are never event evidence. Currency uses the ordinary
|
||||||
|
`name` field and an explicit `quantity` only when the transcript establishes
|
||||||
|
one; each denomination remains a separate event.
|
||||||
|
|
||||||
|
Normalization trims display whitespace, orders and removes exact duplicate
|
||||||
|
source references, then orders events by valid source chronology, name identity
|
||||||
|
and display value, kind, holders, quantity, and reference sequence. It
|
||||||
|
collapses only entries with the same normalized durable fields and complete
|
||||||
|
valid evidence.
|
||||||
|
|
||||||
|
The [JSON output contract](json-output.md) defines publication. See
|
||||||
|
[D&D module internals](../internal/dnd.md) for implementation details and the
|
||||||
|
[NPC-interaction artifact](dnd-npc-interaction-artifacts.md) for a distinct
|
||||||
|
kind of occurrence.
|
||||||
|
|||||||
@@ -1,160 +1,69 @@
|
|||||||
# D&D NPC Artifact
|
# D&D NPC Artifact
|
||||||
|
|
||||||
This document defines the durable D&D NPC-list artifact, its JSON codec, and
|
This contract defines the durable NPC registry produced by `dnd/npcs`. It is a
|
||||||
the selectable production NPC pipeline. The normalized JSON payload can be
|
minimal, source-grounded identity registry for other D&D artifacts, not a
|
||||||
passed explicitly to the spell extractor as an optional caster-name registry
|
character sheet or a relationship summary.
|
||||||
or to the combat extractor and normalizer as an actor registry. It
|
|
||||||
remains a reference, not spell or combat evidence.
|
|
||||||
The NPC interaction extractor and normalizer also consume this registry for
|
|
||||||
canonical identity; registry source references remain provenance and never
|
|
||||||
become interaction evidence. Their occurrence contract is defined in the
|
|
||||||
[D&D NPC interaction artifact](dnd-npc-interaction-artifacts.md).
|
|
||||||
|
|
||||||
## Identity
|
## Identity and compatibility
|
||||||
|
|
||||||
- Artifact kind: `dnd/npc-list`
|
| Property | Value |
|
||||||
- Durable schema ID: `notarius.dnd.npcs`
|
| --- | --- |
|
||||||
- Durable schema name: `notarius_dnd_npcs_v1`
|
| Artifact kind | `dnd/npc-list` |
|
||||||
- Durable schema version: `v1`
|
| Schema ID | `notarius.dnd.npcs` |
|
||||||
- Media type: `application/json`
|
| Schema name | `notarius_dnd_npcs_v1` |
|
||||||
- Identity policy: `dnd.npcs.identity.v1`
|
| Schema version | `v1` |
|
||||||
|
| Media type | `application/json` |
|
||||||
|
| Identity policy | `dnd.npcs.identity.v1` |
|
||||||
|
|
||||||
The durable JSON Schema is owned by the D&D NPC codec. NPC IDs are derived from
|
`v1` accepts one strict JSON object with required `npcs`; the array may be
|
||||||
the Unicode-normalized, case-folded canonical name using the identity policy.
|
empty. NPC and source-reference objects reject unknown fields. A future
|
||||||
The durable codec enforces the artifact shape and ID syntax; registry identity
|
incompatible artifact shape or identity policy uses a new version or policy.
|
||||||
validation remains a separate deterministic concern.
|
|
||||||
|
|
||||||
The extractor's private LLM response schema is a separate structural transport
|
## Wire shape and identity
|
||||||
contract. It omits framework-assigned NPC and source IDs and admits semantic
|
|
||||||
candidates for the deterministic shape and source-reference validators; it is
|
|
||||||
not part of this durable contract.
|
|
||||||
|
|
||||||
## Output Shape
|
Each NPC has these required fields:
|
||||||
|
|
||||||
The payload is one object with a required top-level `npcs` array:
|
| Field | Contract |
|
||||||
|
| --- | --- |
|
||||||
|
| `id` | `npc:sha256:` followed by 64 lowercase hexadecimal characters. |
|
||||||
|
| `name` | Non-empty canonical display name. |
|
||||||
|
| `source_refs` | One or more transcript evidence ranges for the identity. |
|
||||||
|
|
||||||
|
A source reference has exactly `source_id`, `start_unit_id`, and `end_unit_id`.
|
||||||
|
The source ID identifies the transcript, unit IDs are positive inclusive unit
|
||||||
|
identifiers, and the start may not follow the end.
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{"npcs": []}
|
{
|
||||||
|
"npcs": [
|
||||||
|
{
|
||||||
|
"id": "npc:sha256:99a16589618a04f535a7d21fdcc71a0b1c05d22f752cd492065b1086d97bc3d7",
|
||||||
|
"name": "Mira Thorn",
|
||||||
|
"source_refs": [
|
||||||
|
{"source_id": "session-7", "start_unit_id": 4, "end_unit_id": 5}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
The array may be empty. Every object and nested object rejects unknown fields.
|
The ID is deterministic: normalize the name to Unicode NFKC, normalize the
|
||||||
|
supported apostrophe forms, collapse whitespace, case-fold it, SHA-256 the
|
||||||
|
result, then prefix the lowercase hexadecimal digest with `npc:sha256:`. Each
|
||||||
|
canonical identity and ID appears at most once. Normalization collapses records
|
||||||
|
with the same canonical identity, retains their earliest position, and merges
|
||||||
|
their canonicalized evidence; it does not add aliases, roles, descriptions, or
|
||||||
|
relationship fields.
|
||||||
|
|
||||||
## NPC Fields
|
## Scope and consumers
|
||||||
|
|
||||||
Each NPC contains exactly these required fields:
|
Only individually identifiable NPC names with transcript evidence belong in
|
||||||
|
this artifact. Groups, generic roles, invented labels, and descriptive
|
||||||
|
enrichment are excluded. Its source references prove registry provenance; they
|
||||||
|
do not become evidence for a spell, interaction, or combat occurrence.
|
||||||
|
|
||||||
- `id`: `npc:sha256:` followed by 64 lowercase hexadecimal characters;
|
This registry can ground actor or caster names in the [spell](dnd-spell-artifacts.md)
|
||||||
- `name`: the canonical display name;
|
and [combat-turn](dnd-combat-turn-artifacts.md) artifacts. It is required to
|
||||||
- `source_refs`: at least one source reference supporting the NPC record.
|
resolve the canonical `name` in an [NPC interaction](dnd-npc-interaction-artifacts.md).
|
||||||
|
The [JSON output contract](json-output.md) defines publication, and
|
||||||
Each source reference contains required `source_id`, `start_unit_id`, and
|
[D&D module internals](../internal/dnd.md) owns pipeline mechanics.
|
||||||
`end_unit_id`; unit IDs are positive integers. Source document identity, unit
|
|
||||||
existence, and range ordering are validated by the source-reference validator
|
|
||||||
when the artifact is used by a pipeline.
|
|
||||||
|
|
||||||
## Codec Boundary
|
|
||||||
|
|
||||||
`EncodeCandidate` and `DecodeCandidate` provide strict single-value JSON
|
|
||||||
serialization while preserving typed values that still need semantic
|
|
||||||
validation. `Encode` and `Decode` are the approved-artifact boundary and
|
|
||||||
require all durable structural fields, non-empty required strings, valid source
|
|
||||||
reference shapes, and the NPC ID pattern.
|
|
||||||
|
|
||||||
Codec metadata contains only `npc_count`. Schema bytes and returned metadata
|
|
||||||
are independent values so callers cannot mutate codec-owned state.
|
|
||||||
|
|
||||||
## Production Pipeline
|
|
||||||
|
|
||||||
The production identities are:
|
|
||||||
|
|
||||||
- extractor: `dnd/npcs`;
|
|
||||||
- artifact kind: `dnd/npc-list`;
|
|
||||||
- normalizer: `dnd/npcs`; and
|
|
||||||
- durable schema: `notarius.dnd.npcs`, version `v1`, media type
|
|
||||||
`application/json`.
|
|
||||||
|
|
||||||
The extractor maps private model records to the current source identity and
|
|
||||||
assigns deterministic IDs. Extraction validation checks shape, source
|
|
||||||
references, and source relatedness. The normalizer first consolidates equal
|
|
||||||
canonical-name matches, then may make one document-level LLM-assisted identity
|
|
||||||
decision per configured normalize attempt for eligible distinctly named
|
|
||||||
records. Consolidation is name-based: it retains a supplied canonical display
|
|
||||||
name, keeps the earliest affected output position, derives its ID again, and
|
|
||||||
unions exact evidence in canonical order. Ambiguous, invalid, or conflicting
|
|
||||||
proposals are not applied; independently safe matches may still be retained.
|
|
||||||
After the retry budget is exhausted, the safe result is accepted with bounded
|
|
||||||
normalization warnings and the usual validation. The durable v1 artifact shape
|
|
||||||
does not add aliases, proposal fields, or any other semantic-normalization
|
|
||||||
representation.
|
|
||||||
|
|
||||||
The extraction prompt asks only for individually identifiable NPC names backed
|
|
||||||
by source evidence. Groups, generic roles, invented labels, and descriptive or
|
|
||||||
relationship enrichment are outside the contract.
|
|
||||||
|
|
||||||
The default extraction chain is `generic/valid_json`,
|
|
||||||
`extract/dnd/npcs/shape`, `extract/dnd/npcs/source_refs`,
|
|
||||||
`generic/valid_json_schema`, and `extract/dnd/npcs/source_relatedness`. The
|
|
||||||
default normalize chain is `generic/valid_json`, `extract/dnd/npcs/shape`,
|
|
||||||
`normalize/dnd/npcs/identity`, `extract/dnd/npcs/source_refs`,
|
|
||||||
`generic/valid_json_schema`, and `extract/dnd/npcs/source_relatedness`.
|
|
||||||
Relatedness emits bounded warnings when an NPC canonical name is not present
|
|
||||||
near its cited transcript text; opaque campaign
|
|
||||||
references may explain such a warning but do not become evidence.
|
|
||||||
|
|
||||||
## Manifest And Artifact Handoff
|
|
||||||
|
|
||||||
The NPC extractor records prompt and response-schema identities. The durable
|
|
||||||
codec records only `npc_count`; raw names, source references, and payload bytes
|
|
||||||
stay in the lane file rather than manifest
|
|
||||||
metadata. The normalized lane can be consumed by a later ordered step through
|
|
||||||
the registered canonical codec:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
steps:
|
|
||||||
- id: identify-npcs
|
|
||||||
artifacts:
|
|
||||||
npcs:
|
|
||||||
extract: dnd/npcs
|
|
||||||
normalize: dnd/npcs
|
|
||||||
scene-descriptions:
|
|
||||||
extract: dnd/scene-descriptions
|
|
||||||
normalize: dnd/scene-descriptions
|
|
||||||
- id: grounded-events
|
|
||||||
references:
|
|
||||||
npcs:
|
|
||||||
artifact:
|
|
||||||
step: identify-npcs
|
|
||||||
lane: npcs
|
|
||||||
scene_descriptions:
|
|
||||||
artifact:
|
|
||||||
step: identify-npcs
|
|
||||||
lane: scene-descriptions
|
|
||||||
artifacts:
|
|
||||||
spells:
|
|
||||||
extract: dnd/spells
|
|
||||||
normalize: dnd/spells
|
|
||||||
combat:
|
|
||||||
extract: dnd/combat-turns
|
|
||||||
normalize: dnd/combat-turns
|
|
||||||
```
|
|
||||||
|
|
||||||
The framework hands only accepted normalized artifacts across the barrier. It
|
|
||||||
validates the canonical bytes against each consumer slot and clones the NPC
|
|
||||||
operation-time reference for spell and combat consumers. The accompanying
|
|
||||||
scene-description reference is required by the combat extractor for eligibility
|
|
||||||
only; its consumer contract is defined in the
|
|
||||||
[D&D combat-turn artifact contract](dnd-combat-turn-artifacts.md). Generated
|
|
||||||
provenance records the artifact kind, schema identity, media type, canonical
|
|
||||||
digest, size, and producer step/lane/module, but not names, source ranges, or
|
|
||||||
payload bytes. External normalized files remain supported as explicit references
|
|
||||||
and retain their file provenance.
|
|
||||||
|
|
||||||
NPC source references are registry provenance and are never accepted as spell
|
|
||||||
or combat evidence. Current transcript units remain the only event evidence.
|
|
||||||
|
|
||||||
Consumers receive a separate names-only projection in normalized registry
|
|
||||||
order, for example `{"npcs":[{"name":"Mira Thorn"}]}`. The projection omits
|
|
||||||
IDs and evidence. Its digest covers the exact projected bytes and is used for
|
|
||||||
consumer-local checkpoint identity, while the full durable artifact digest
|
|
||||||
remains the manifest and generated-reference provenance identity. The unbound
|
|
||||||
projection is exactly `{"npcs":[]}` and also has a projection digest.
|
|
||||||
|
|||||||
@@ -1,20 +1,38 @@
|
|||||||
# D&D NPC Interaction Artifact
|
# D&D NPC Interaction Artifact
|
||||||
|
|
||||||
This document defines the durable D&D NPC-interaction-list artifact and its
|
This contract defines the durable occurrence list produced by
|
||||||
two-step production pipeline. It records discrete, source-grounded occurrences
|
`dnd/npc-interactions`. It records discrete, source-grounded interactions with
|
||||||
for NPCs already accepted into a normalized NPC registry; it does not expand
|
NPCs already present in a normalized registry; it does not extend that registry
|
||||||
the registry or summarize events.
|
or summarize the session.
|
||||||
|
|
||||||
## Identity And JSON
|
## Identity and compatibility
|
||||||
|
|
||||||
- Artifact kind: `dnd/npc-interaction-list`
|
| Property | Value |
|
||||||
- Durable schema ID: `notarius.dnd.npc_interactions`
|
| --- | --- |
|
||||||
- Durable schema name: `notarius_dnd_npc_interactions_v1`
|
| Artifact kind | `dnd/npc-interaction-list` |
|
||||||
- Durable schema version: `v1`
|
| Schema ID | `notarius.dnd.npc_interactions` |
|
||||||
- Media type: `application/json`
|
| Schema name | `notarius_dnd_npc_interactions_v1` |
|
||||||
|
| Schema version | `v1` |
|
||||||
|
| Media type | `application/json` |
|
||||||
|
|
||||||
The payload is one strict JSON object with only an `interactions` array. The
|
`v1` is a strict JSON object with required `interactions`; the array may be
|
||||||
array may be empty. Each item has exactly `name`, `kind`, and `source_refs`:
|
empty. Interaction and source-reference objects reject unknown fields. A future
|
||||||
|
incompatible shape requires a new schema version.
|
||||||
|
|
||||||
|
## Wire shape
|
||||||
|
|
||||||
|
Each interaction has these required fields:
|
||||||
|
|
||||||
|
| Field | Contract |
|
||||||
|
| --- | --- |
|
||||||
|
| `name` | Non-empty canonical display name from the required NPC registry. |
|
||||||
|
| `kind` | One of the interaction categories below. |
|
||||||
|
| `source_refs` | One or more transcript evidence ranges. |
|
||||||
|
|
||||||
|
Each source reference has exactly `source_id`, `start_unit_id`, and
|
||||||
|
`end_unit_id`. It identifies an inclusive range in the current transcript;
|
||||||
|
unit IDs are positive and the start may not follow the end. Extraction evidence
|
||||||
|
for an interaction is confined to its accepted chunk.
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
@@ -30,119 +48,31 @@ array may be empty. Each item has exactly `name`, `kind`, and `source_refs`:
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
`name` is the canonical display name from the required NPC registry.
|
## Interaction categories
|
||||||
`source_refs` contains one or more current-source ranges with required
|
|
||||||
`source_id`, `start_unit_id`, and `end_unit_id`; unit IDs are positive integers.
|
|
||||||
During extraction, every range must be wholly contained in the current accepted
|
|
||||||
chunk. This prevents a candidate from citing valid units that were not presented
|
|
||||||
to that extraction call.
|
|
||||||
Unknown fields are rejected.
|
|
||||||
|
|
||||||
## Interaction Categories
|
|
||||||
|
|
||||||
`kind` is exactly one of:
|
|
||||||
|
|
||||||
| Kind | Meaning |
|
| Kind | Meaning |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `mentioned` | The NPC is referred to, but is not established as present or communicating in the evidenced passage. |
|
| `mentioned` | The NPC is referred to but is not established as present or communicating. |
|
||||||
| `noncombat_presence` | The NPC is present and relevant to the passage but does not meaningfully participate in dialogue or combat. |
|
| `noncombat_presence` | The NPC is present and relevant without meaningful dialogue or combat participation. |
|
||||||
| `dialogue` | The NPC speaks, responds, or is directly engaged in a meaningful non-combat exchange. |
|
| `dialogue` | The NPC speaks, responds, or meaningfully participates in a non-combat exchange. |
|
||||||
| `combat_ally` | The NPC actively participates in combat on the party's side. |
|
| `combat_ally` | The NPC actively participates in combat on the party's side. |
|
||||||
| `combat_opponent` | The NPC actively participates in combat against the party. |
|
| `combat_opponent` | The NPC actively participates in combat against the party. |
|
||||||
| `other` | The transcript clearly establishes a direct NPC occurrence that fits none of the preceding kinds. |
|
| `other` | A clearly evidenced direct occurrence not covered by another category. |
|
||||||
|
|
||||||
`other` is a residual category for positively evidenced activity, not a fallback
|
The categories do not represent motives, relationships, state, or events that
|
||||||
for uncertain classification. When activities overlap, active combat
|
the cited transcript does not establish. An `other` entry is not a substitute
|
||||||
participation outranks dialogue, presence, and mention; dialogue outranks
|
for uncertain classification.
|
||||||
non-combat presence and mention; and non-combat presence outranks mention.
|
|
||||||
Combat alignment is not resolved by precedence: a meaningful change between
|
|
||||||
ally and opponent creates separate occurrences.
|
|
||||||
|
|
||||||
These categories do not encode summaries, relationships, state, motives, or
|
## Identity, evidence, and order
|
||||||
unobserved events.
|
|
||||||
|
|
||||||
## Occurrence Boundaries And Ordering
|
The required normalized [NPC artifact](dnd-npc-artifacts.md) resolves `name`.
|
||||||
|
Registry references are provenance only and never replace an interaction's own
|
||||||
|
evidence. Normalization canonicalizes recognized registry names, orders and
|
||||||
|
deduplicates exact source references, then orders interactions by valid source
|
||||||
|
chronology, NPC comparison identity, display name, kind, and reference sequence.
|
||||||
|
Only entries with the same canonical name, kind, and complete valid evidence
|
||||||
|
sequence are collapsed; distinct categories or evidence remain separate.
|
||||||
|
|
||||||
One occurrence represents one NPC, one kind, and one locally coherent passage
|
See the [combat-turn artifact](dnd-combat-turn-artifacts.md) for combat-action
|
||||||
within one accepted chunk. Repeated evidence belongs to the same occurrence
|
occurrences and the [JSON output contract](json-output.md) for publication.
|
||||||
only while it supports the same uninterrupted activity. A kind change, combat
|
Pipeline mechanics are described in [D&D module internals](../internal/dnd.md).
|
||||||
alignment change, intervening scene or meaningful absence, or transition from
|
|
||||||
mention to presence starts a new occurrence. Occurrences never span chunks, and
|
|
||||||
merge or normalization never semantically combines nearby, overlapping, or
|
|
||||||
cross-chunk records.
|
|
||||||
|
|
||||||
Normalization orders records by:
|
|
||||||
|
|
||||||
1. earliest valid source-document position;
|
|
||||||
2. the NPC identity comparison key;
|
|
||||||
3. the exact canonical NPC display name;
|
|
||||||
4. interaction kind in lexical order; and
|
|
||||||
5. the complete canonical source-reference sequence, ordered by source ID and
|
|
||||||
the source-document positions of each range's start and end.
|
|
||||||
|
|
||||||
Only records with identical canonical names, kinds, and complete valid evidence
|
|
||||||
sequences are duplicates. Different categories, ranges, or separately grounded
|
|
||||||
occurrences remain separate.
|
|
||||||
|
|
||||||
## Evidence, Registry, And Normalization
|
|
||||||
|
|
||||||
The registry proves only the canonical NPC identity. Its source references are
|
|
||||||
registry provenance and are never interaction evidence. Every durable
|
|
||||||
interaction must cite current transcript units supporting both the name and its
|
|
||||||
classification.
|
|
||||||
|
|
||||||
The extractor receives a names-only registry projection such as
|
|
||||||
`{"npcs":[{"name":"Mira Thorn"}]}`. The normalizer uses the full immutable
|
|
||||||
registry for exact canonical-name lookup. It canonicalizes source references
|
|
||||||
and applies the ordering and exact-duplicate rules above.
|
|
||||||
|
|
||||||
## Production Pipeline
|
|
||||||
|
|
||||||
The extractor and normalizer key is `dnd/npc-interactions`. Both require the
|
|
||||||
structured `npcs` slot, so an accepted normalized registry must come from an
|
|
||||||
earlier step:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
pipelines:
|
|
||||||
dnd-npc-interactions:
|
|
||||||
input: seriatim
|
|
||||||
steps:
|
|
||||||
- id: identify-npcs
|
|
||||||
artifacts:
|
|
||||||
npcs:
|
|
||||||
extract: dnd/npcs
|
|
||||||
normalize: dnd/npcs
|
|
||||||
- id: extract-interactions
|
|
||||||
references:
|
|
||||||
npcs:
|
|
||||||
artifact:
|
|
||||||
step: identify-npcs
|
|
||||||
lane: npcs
|
|
||||||
artifacts:
|
|
||||||
interactions:
|
|
||||||
extract: dnd/npc-interactions
|
|
||||||
normalize: dnd/npc-interactions
|
|
||||||
```
|
|
||||||
|
|
||||||
The framework passes only the accepted normalized producer. A missing, rejected,
|
|
||||||
or incompatible NPC artifact prevents the consumer step from executing. It
|
|
||||||
records generated artifact identity and bounded producer provenance without
|
|
||||||
copying registry names, source ranges, or payload content into the manifest.
|
|
||||||
|
|
||||||
## Validation And Metadata
|
|
||||||
|
|
||||||
The default extract chain is `generic/valid_json`, interaction shape, registry,
|
|
||||||
and source-reference validation, `generic/valid_json_schema`, then warning-only
|
|
||||||
source relatedness. The normalize chain runs normalized invariants after
|
|
||||||
registry validation and before source-reference and schema validation, followed
|
|
||||||
by relatedness. Normalizer and relatedness warnings are bounded and end with an
|
|
||||||
omission summary when necessary. The codec metadata contains only
|
|
||||||
`interaction_count`. Extractor metadata identifies its prompt and private
|
|
||||||
response schema; component-local checkpoint identities include the names-only
|
|
||||||
registry projection where relevant. Generated registry identity stays in
|
|
||||||
framework provenance and dependency fingerprints.
|
|
||||||
|
|
||||||
See [Configuration](../config.md#implemented-production-modules) for selectable
|
|
||||||
keys and chains, [the NPC artifact contract](dnd-npc-artifacts.md) for the
|
|
||||||
registry boundary, and the copyable
|
|
||||||
[complete D&D example](../../examples/dnd-complete.config.yml).
|
|
||||||
|
|||||||
@@ -1,9 +1,10 @@
|
|||||||
# D&D Scene Description Artifact Contract
|
# D&D Scene-Description Artifact
|
||||||
|
|
||||||
This document defines the durable `dnd/scene-description-list` artifact
|
This contract defines the durable output of `dnd/scene-descriptions`. Each
|
||||||
emitted by the D&D scene-description lane.
|
record classifies one accepted transcript chunk and gives it a minimal
|
||||||
|
source-grounded title and summary.
|
||||||
|
|
||||||
## Artifact identity
|
## Identity and compatibility
|
||||||
|
|
||||||
| Property | Value |
|
| Property | Value |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
@@ -13,38 +14,32 @@ emitted by the D&D scene-description lane.
|
|||||||
| Schema version | `v1` |
|
| Schema version | `v1` |
|
||||||
| Media type | `application/json` |
|
| Media type | `application/json` |
|
||||||
|
|
||||||
The normalized payload is written by the JSON output module to
|
`v1` is a strict JSON object with required non-empty `scenes`. Scene and
|
||||||
`lanes/<lane-id>.json`. See [JSON output](json-output.md) for the surrounding
|
source-reference objects reject unknown fields. A future incompatible shape
|
||||||
output bundle.
|
requires a new schema version.
|
||||||
|
|
||||||
## JSON shape
|
## Wire shape
|
||||||
|
|
||||||
The payload is a JSON object containing exactly one required field, `scenes`.
|
Each scene has exactly these required fields:
|
||||||
Each scene object contains exactly these required fields:
|
|
||||||
|
|
||||||
| Field | Shape and ownership |
|
| Field | Contract |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `id` | Non-empty accepted chunk ID, assigned by Notarius. |
|
| `id` | Non-empty accepted chunk ID, assigned by Notarius. |
|
||||||
| `source_ref` | Exact inclusive accepted chunk range, assigned by Notarius. |
|
| `source_ref` | The assigned inclusive source range for that chunk. |
|
||||||
| `kind` | One of `combat`, `narrative`, `recap`, or `meta`. |
|
| `kind` | `combat`, `narrative`, `recap`, or `meta`. |
|
||||||
| `title` | Non-empty, trimmed, source-grounded title. |
|
| `title` | Non-empty, trimmed, source-grounded title. |
|
||||||
| `summary` | Non-empty, trimmed, source-grounded summary. |
|
| `summary` | Non-empty, trimmed, source-grounded summary. |
|
||||||
|
|
||||||
All object layers reject unknown fields. The `scenes` array must be present and
|
`source_ref` has exactly `source_id`, `start_unit_id`, and `end_unit_id`.
|
||||||
non-empty. `source_ref` has exactly `source_id`, `start_unit_id`, and
|
Its source ID identifies the input transcript; its positive unit IDs identify
|
||||||
`end_unit_id`; its source ID is non-empty and its unit IDs are positive
|
the chunk's inclusive range, with the start no later than the end.
|
||||||
integers.
|
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"scenes": [
|
"scenes": [
|
||||||
{
|
{
|
||||||
"id": "chunk-000001",
|
"id": "chunk-000001",
|
||||||
"source_ref": {
|
"source_ref": {"source_id": "session-7", "start_unit_id": 1, "end_unit_id": 3},
|
||||||
"source_id": "session-alpha",
|
|
||||||
"start_unit_id": 1,
|
|
||||||
"end_unit_id": 3
|
|
||||||
},
|
|
||||||
"kind": "narrative",
|
"kind": "narrative",
|
||||||
"title": "Arrival at the watchtower",
|
"title": "Arrival at the watchtower",
|
||||||
"summary": "The party reaches the ruined watchtower and begins to investigate it."
|
"summary": "The party reaches the ruined watchtower and begins to investigate it."
|
||||||
@@ -53,78 +48,22 @@ integers.
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
`id` and `source_ref` are application-owned identity and evidence. The model
|
## Meaning and normalized form
|
||||||
provides only `kind`, `title`, and `summary`; it is not asked for chunk IDs,
|
|
||||||
source IDs, unit IDs, ranges, participants, or confidence.
|
|
||||||
|
|
||||||
## Scene kinds
|
`combat` identifies a chunk where active combat is the central activity.
|
||||||
|
`narrative` is current in-world play that is not principally combat, recap, or
|
||||||
|
meta discussion. `recap` is primarily a recounting of an earlier session, and
|
||||||
|
`meta` is primarily out-of-character discussion. The artifact does not add
|
||||||
|
participants, confidence, events, or information absent from the chunk.
|
||||||
|
|
||||||
| Kind | Meaning |
|
Normalization trims title and summary, orders scenes by source position and
|
||||||
| --- | --- |
|
then ID, and removes exact duplicate records. A reused ID with different
|
||||||
| `combat` | Active combat is a substantive central activity. |
|
durable fields, or the same source range with different kind, title, or
|
||||||
| `narrative` | Current-session in-world play that is not principally combat, recap, or meta discussion. |
|
summary, is invalid. It does not merge adjacent ranges, alter prose, or infer
|
||||||
| `recap` | The table is primarily recounting a previous session. |
|
missing scenes.
|
||||||
| `meta` | Sustained out-of-character discussion is the scene's primary purpose. |
|
|
||||||
|
|
||||||
For a mixed accepted chunk, classification prefers `combat`, then `recap`,
|
The [combat-turn artifact](dnd-combat-turn-artifacts.md) uses an exact matching
|
||||||
then `meta`, then `narrative`. Brief table talk, rules clarification, or a
|
`combat` scene only as eligibility control; scene title, summary, and source
|
||||||
short recollection does not replace the main current-session activity.
|
reference never become combat evidence. Publication is defined by the
|
||||||
|
[JSON output contract](json-output.md); implementation details live in
|
||||||
## Extraction and evidence
|
[D&D module internals](../internal/dnd.md).
|
||||||
|
|
||||||
The extractor runs once for each accepted chunk and maps one successful model
|
|
||||||
response to one scene record. It copies the current chunk ID and exact chunk
|
|
||||||
range, preserves the model kind without repair, and trims only surrounding
|
|
||||||
whitespace from title and summary. A model response cannot represent an empty
|
|
||||||
result; extraction failure follows the configured retry and rejection policy.
|
|
||||||
|
|
||||||
Optional `players`, `party`, and `glossary` campaign references can help
|
|
||||||
disambiguate names or setting terms. They never supply scene evidence or add
|
|
||||||
events absent from the accepted chunk. The lane requires no NPC registry or
|
|
||||||
other generated artifact reference.
|
|
||||||
|
|
||||||
## Merge and normalization
|
|
||||||
|
|
||||||
Accepted per-chunk lists are appended in chunk order. Normalization then:
|
|
||||||
|
|
||||||
1. validates the current-source range, non-empty ID and prose, and closed kind;
|
|
||||||
2. trims only title and summary whitespace;
|
|
||||||
3. sorts records by source-document start position, then ID;
|
|
||||||
4. removes records only when all five durable fields are identical;
|
|
||||||
5. rejects a reused ID when any remaining durable field differs; and
|
|
||||||
6. rejects the same exact range when `kind`, `title`, or `summary` differs.
|
|
||||||
|
|
||||||
Two different IDs with the same range and identical model-owned content remain
|
|
||||||
separate records. Normalization does not join adjacent ranges, rewrite prose,
|
|
||||||
repair kinds, infer missing scenes, or use chunk annotations.
|
|
||||||
|
|
||||||
## Validation and warnings
|
|
||||||
|
|
||||||
Extraction validation requires exactly one record with an ID and range exactly
|
|
||||||
equal to its current chunk. Later validation checks source membership without a
|
|
||||||
current chunk. Shape, source-range, kind, ID, and normalized-invariant failures
|
|
||||||
reject the artifact.
|
|
||||||
|
|
||||||
Relatedness checks are advisory. They separately warn when a scene title or
|
|
||||||
summary has no significant lexical token in its cited transcript range. The
|
|
||||||
check ignores short tokens and common function words, uses transcript text only,
|
|
||||||
and does not treat campaign references as evidence. Warning diagnostics are
|
|
||||||
bounded and do not copy transcript or campaign-reference content.
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
Use `dnd/scenes` to form accepted scene chunks, then select
|
|
||||||
`dnd/scene-descriptions` for extraction and normalization. The maintained
|
|
||||||
[complete D&D configuration](../../examples/dnd-complete.config.yml) includes
|
|
||||||
this lane alongside the other D&D artifacts. Selectable keys and default
|
|
||||||
validator chains are defined in [Configuration](../config.md).
|
|
||||||
|
|
||||||
## Downstream combat use
|
|
||||||
|
|
||||||
The combat-turn extractor consumes this approved artifact as required
|
|
||||||
eligibility control context through an explicit ordered reference binding. It
|
|
||||||
uses only an exact chunk ID and source-range match with `kind: combat` to permit
|
|
||||||
combat extraction; titles, summaries, and scene references do not become combat
|
|
||||||
prompt material or evidence. The complete downstream behavior, including empty
|
|
||||||
results and warnings for unavailable coverage, is defined in the
|
|
||||||
[D&D combat-turn artifact contract](dnd-combat-turn-artifacts.md).
|
|
||||||
|
|||||||
@@ -1,204 +1,73 @@
|
|||||||
# D&D Spell Artifact
|
# D&D Spell Artifact
|
||||||
|
|
||||||
This document is the durable serialized artifact contract for the production
|
This contract defines the durable output of the `dnd/spells` extractor and
|
||||||
D&D spell extractor. Selectable extractor keys are cataloged in
|
normalizer. It records source-grounded spell-casting occurrences; it is not a
|
||||||
[Configuration](../config.md#implemented-production-modules).
|
spellbook, a rules lookup result, or a record of hypothetical casts.
|
||||||
|
|
||||||
## Identity
|
## Identity and compatibility
|
||||||
|
|
||||||
- Artifact kind: `dnd/spell-list`
|
| Property | Value |
|
||||||
- Prompt ID: `dnd.spells`
|
|
||||||
- Response schema key: `dnd_spells`
|
|
||||||
- Response schema ID: `notarius.dnd.spells`
|
|
||||||
- Response schema name: `notarius_dnd_spells_v1`
|
|
||||||
- Response schema version: `v1`
|
|
||||||
- Media type: `application/json`
|
|
||||||
|
|
||||||
The durable JSON Schema is owned by the D&D spell artifact codec. The
|
|
||||||
extractor's private LLM response schema is a separate transport contract: its
|
|
||||||
source-reference objects omit `source_id`, which the extractor assigns while
|
|
||||||
mapping the response to the canonical artifact. The LLM DTO and transport
|
|
||||||
schema are not part of this durable contract. The private schema owns required
|
|
||||||
fields, JSON types, object and array shapes, and unknown-field rejection;
|
|
||||||
deterministic validators own the durable artifact's semantic constraints.
|
|
||||||
|
|
||||||
The output contains canonical spell casts derived from transcript evidence.
|
|
||||||
Source IDs are assigned from the input identity; source-unit ranges identify
|
|
||||||
the evidence location.
|
|
||||||
|
|
||||||
## Output Shape
|
|
||||||
|
|
||||||
The extractor payload is a JSON object with one required top-level array. Its
|
|
||||||
structure is:
|
|
||||||
|
|
||||||
```text
|
|
||||||
{"spell_casts": [<spell-cast object>, ...]}
|
|
||||||
```
|
|
||||||
|
|
||||||
`spell_casts` must be present. It may be empty when no spell casts are found.
|
|
||||||
When multiple chunk results are combined, spell casts remain in chunk order.
|
|
||||||
When the payload is written as durable output, its logical path is derived from
|
|
||||||
the configured artifact lane ID as defined by the
|
|
||||||
[JSON output contract](json-output.md#output-payload-files).
|
|
||||||
|
|
||||||
## Spell-Cast Fields
|
|
||||||
|
|
||||||
Each spell cast contains exactly these required fields:
|
|
||||||
|
|
||||||
- `caster`: in-world character or creature casting the spell;
|
|
||||||
- `spell`: spell name;
|
|
||||||
- `source_refs`: transcript source references with extractor-assigned source
|
|
||||||
IDs and evidence unit ranges. It must contain at least one entry.
|
|
||||||
|
|
||||||
Both string fields must be non-empty. `caster` is the canonical in-world
|
|
||||||
caster, not the human player, transcript speaker, or GM when the associated
|
|
||||||
character or creature can be identified. Player and party references may
|
|
||||||
disambiguate that identity, but do not independently establish that a cast
|
|
||||||
occurred. The `spell` value must resolve through the effective SRD-plus-overlay
|
|
||||||
catalog as either a canonical name or alias. Catalog validation accepts aliases
|
|
||||||
but does not rewrite them; unknown fields are rejected.
|
|
||||||
|
|
||||||
The artifact includes an actual casting event or an unambiguous declared
|
|
||||||
casting attempt. Spell mentions, hypothetical plans, rules discussion, and
|
|
||||||
catalog matches without a casting event are excluded. The spell catalog is a
|
|
||||||
name-recognition policy and never evidence that a cast occurred.
|
|
||||||
|
|
||||||
## Source References
|
|
||||||
|
|
||||||
Each source reference contains exactly three required fields: `source_id`,
|
|
||||||
`start_unit_id`, and `end_unit_id`. The source ID must match the input identity.
|
|
||||||
The unit IDs must be positive integers present in the input, and the start unit
|
|
||||||
must not appear after the end unit. Unknown fields are rejected.
|
|
||||||
|
|
||||||
For each cast, the complete `source_refs` collection identifies the transcript
|
|
||||||
evidence for the caster, spell name, and occurrence of the cast or declared
|
|
||||||
attempt. The deterministic validators establish that ranges are structurally
|
|
||||||
valid and that the spell name is related to cited text. Semantic evidence
|
|
||||||
sufficiency is an extraction policy and remains subject to evaluation rather
|
|
||||||
than deterministic proof.
|
|
||||||
|
|
||||||
Reference slot keys and accepted file types are defined in
|
|
||||||
[Configuration](../config.md#implemented-production-modules). References are
|
|
||||||
supporting disambiguation material, not source evidence, and are not
|
|
||||||
addressable through `source_refs`.
|
|
||||||
|
|
||||||
## Optional NPC Grounding
|
|
||||||
|
|
||||||
The `dnd/spells` extractor accepts an optional `npcs` reference containing one
|
|
||||||
normalized NPC artifact as `application/json`, up to 1 MiB. An external file is
|
|
||||||
validated during preparation; an ordered generated binding is validated at the
|
|
||||||
step handoff. Both paths use the approved NPC codec and identity policy,
|
|
||||||
re-encode canonical durable JSON for registry provenance, and supply only the
|
|
||||||
registry's ordered names as the operation-time spell prompt input. It helps the
|
|
||||||
model prefer canonical caster names; it does not establish that a spell was
|
|
||||||
cast.
|
|
||||||
|
|
||||||
NPC source references may identify the run that produced the registry or any
|
|
||||||
other session. They remain registry provenance and are never copied into a
|
|
||||||
spell cast's `source_refs`; every spell evidence range must still identify the
|
|
||||||
current transcript. Generated provenance records producer and canonical
|
|
||||||
artifact identity without payload content or a path. When the slot is absent,
|
|
||||||
the prompt receives exactly `{"npcs":[]}` with its projection digest, and the
|
|
||||||
run has no NPC reference provenance.
|
|
||||||
|
|
||||||
## Normalization Behavior
|
|
||||||
|
|
||||||
When the `dnd/spells` normalizer is selected, each recognized spell name is
|
|
||||||
rewritten to the effective catalog's canonical display name. Lookup uses the
|
|
||||||
catalog's case-insensitive, whitespace-normalizing, apostrophe-normalizing, and
|
|
||||||
alias rules. Unknown names are preserved exactly for the normalize validators;
|
|
||||||
the normalizer does not guess or apply fuzzy matching.
|
|
||||||
|
|
||||||
Each cast's `source_refs` is copied, sorted by exact `source_id`,
|
|
||||||
`start_unit_id`, and `end_unit_id`, and stripped of exact structural
|
|
||||||
duplicates. Adjacent or overlapping ranges are not merged, and the normalizer
|
|
||||||
does not synthesize references or change their boundaries.
|
|
||||||
|
|
||||||
After those per-cast changes, duplicate identity requires the same canonical
|
|
||||||
spell name, the same caster after case folding and whitespace normalization,
|
|
||||||
and the same complete, non-empty set of source references valid for the source
|
|
||||||
document. Only the first occurrence is retained, in stable order. Its caster
|
|
||||||
and canonical references are preserved. Unknown names, empty or invalid
|
|
||||||
evidence, and casts with different evidence remain separate.
|
|
||||||
|
|
||||||
Mutation and duplicate decisions are returned through the normal warnings
|
|
||||||
surface. Warning scopes use the merged input index, such as `spell_casts[0]`,
|
|
||||||
so they remain meaningful even when a later duplicate is removed. The
|
|
||||||
normalizer uses these reason codes:
|
|
||||||
|
|
||||||
| Reason code | Meaning |
|
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `spell_name_canonicalized` | A catalog lookup replaced an input name with its canonical display name. |
|
| Artifact kind | `dnd/spell-list` |
|
||||||
| `spell_name_unresolved` | A name was not found in the effective catalog and was retained unchanged. |
|
| Schema ID | `notarius.dnd.spells` |
|
||||||
| `source_references_normalized` | Reference order changed or exact duplicate references were removed. |
|
| Schema name | `notarius_dnd_spells_v1` |
|
||||||
| `duplicate_spell_cast_collapsed` | A later cast matched the retained cast's complete duplicate identity. |
|
| Schema version | `v1` |
|
||||||
|
| Media type | `application/json` |
|
||||||
|
|
||||||
Only warnings from an accepted normalize attempt are promoted to
|
`v1` is a single strict JSON object. It requires `spell_casts`; the array may
|
||||||
`warnings.json`. If an unresolved name reaches the default normalize validator
|
be empty. Each spell-cast object and source-reference object rejects unknown
|
||||||
chain, the catalog validator rejects the candidate with `unknown_spell`; the
|
fields. A future incompatible shape requires a new schema version.
|
||||||
`spell_name_unresolved` warning remains in the attempt's debug artifact. An
|
|
||||||
explicit validator override that accepts the candidate promotes the unresolved
|
|
||||||
warning normally.
|
|
||||||
|
|
||||||
The default extraction and normalization chains both preserve this registered
|
## Wire shape
|
||||||
order: JSON syntax, spell shape, catalog membership, source references, JSON
|
|
||||||
Schema, then source relatedness. Extraction validates the private response
|
|
||||||
schema; normalization validates the durable artifact schema.
|
|
||||||
|
|
||||||
## Manifest Metadata
|
Each `spell_casts` entry has these required fields:
|
||||||
|
|
||||||
The extractor adds prompt and response-schema provenance under the artifact lane
|
| Field | Contract |
|
||||||
manifest metadata:
|
| --- | --- |
|
||||||
|
| `caster` | Non-empty in-world character or creature name. |
|
||||||
|
| `spell` | Non-empty spell name. |
|
||||||
|
| `source_refs` | One or more transcript evidence ranges. |
|
||||||
|
|
||||||
|
Every source reference has exactly `source_id`, `start_unit_id`, and
|
||||||
|
`end_unit_id`. The source ID identifies the input transcript; the unit IDs are
|
||||||
|
positive inclusive unit identifiers, and the start may not follow the end in
|
||||||
|
that source. References are evidence for the cast, not campaign-reference or
|
||||||
|
NPC-registry provenance.
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"metadata": {
|
"spell_casts": [
|
||||||
"extractor": {
|
{
|
||||||
"prompt_id": "dnd.spells",
|
"caster": "Mira Thorn",
|
||||||
"prompt_version": "v1",
|
"spell": "Fireball",
|
||||||
"prompt_sha256": "sha256:...",
|
"source_refs": [
|
||||||
"response_schema_key": "dnd_spells",
|
{"source_id": "session-7", "start_unit_id": 12, "end_unit_id": 13}
|
||||||
"response_schema_id": "notarius.dnd.spells",
|
]
|
||||||
"response_schema_name": "notarius_dnd_spells_v1",
|
|
||||||
"response_schema_version": "v1",
|
|
||||||
"response_schema_sha256": "sha256:...",
|
|
||||||
"catalog_base_id": "dnd-5e-2014-srd-spells",
|
|
||||||
"catalog_digest": "sha256:...",
|
|
||||||
"catalog_overlay_ids": ["campaign.example"],
|
|
||||||
"npc_registry_digest": "sha256:...",
|
|
||||||
"npc_count": 3
|
|
||||||
},
|
|
||||||
"normalizer": {
|
|
||||||
"catalog_base_id": "dnd-5e-2014-srd-spells",
|
|
||||||
"catalog_digest": "sha256:...",
|
|
||||||
"catalog_overlay_ids": ["campaign.example"]
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
]
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
`catalog_digest` identifies the effective semantic catalog, while
|
## Evidence and normalized form
|
||||||
`catalog_overlay_ids` is sorted and empty for a base-only configuration. Raw
|
|
||||||
prompt, schema, catalog, alias, and local overlay-file content are not
|
|
||||||
included in manifest metadata. The `normalizer` metadata uses the same catalog
|
|
||||||
identity fields when that module is selected. Overlay origin, media type, byte
|
|
||||||
size, and raw digest are recorded separately in the manifest's reference
|
|
||||||
provenance; see the [JSON output contract](json-output.md#manifestjson).
|
|
||||||
|
|
||||||
The `npc_registry_digest` and `npc_count` fields in the example are present for
|
An entry represents an actual cast or an unambiguous declared attempt. A spell
|
||||||
an external NPC registry when the extractor publishes its prepared module
|
mention, rules discussion, plan, or catalog match alone is not an occurrence.
|
||||||
metadata. They contain no NPC names, source references, paths, or raw
|
The configured catalog checks the name; it does not establish evidence.
|
||||||
bytes. A generated registry's identity is instead represented by the framework
|
|
||||||
handoff provenance and dependency fingerprint, so the consumer module metadata
|
|
||||||
does not duplicate it.
|
|
||||||
|
|
||||||
The extractor's prompt hash, private response-schema hash, and effective catalog
|
When normalization is selected, recognized spell names use the effective
|
||||||
digest also contribute independently scoped semantic checkpoint fingerprints.
|
catalog's canonical display name. Source references are put in canonical source
|
||||||
Changing any of those prepared contracts intentionally produces a cold
|
order and exact duplicate references are removed. A later entry is collapsed
|
||||||
checkpoint miss. Fingerprints contain only digests, never prompt, schema,
|
only when it has the same canonical spell, the same case- and
|
||||||
catalog, or reference content. When an NPC registry is bound, its semantic
|
whitespace-insensitive caster identity, and the same complete valid reference
|
||||||
digest contributes an additional local `npc_registry` fingerprint for an
|
sequence. Remaining entries retain their merged order.
|
||||||
external binding; the manifest metadata contains only that digest and
|
|
||||||
`npc_count`. Raw NPC file provenance remains independently recorded in the
|
The optional normalized [NPC artifact](dnd-npc-artifacts.md) can ground a
|
||||||
manifest's `references` list. Generated bindings contribute the canonical
|
caster name. Its own references remain registry provenance and are never copied
|
||||||
artifact dependency fingerprint and bounded producer provenance instead.
|
into `source_refs`.
|
||||||
|
|
||||||
|
## Related contracts
|
||||||
|
|
||||||
|
The [spell-catalog overlay contract](dnd-spell-catalog-overlays.md) defines
|
||||||
|
the configured catalog additions. The [JSON output contract](json-output.md)
|
||||||
|
defines where this logical artifact is published; [D&D module internals](../internal/dnd.md)
|
||||||
|
describes extraction and validation mechanics.
|
||||||
|
|||||||
@@ -1,19 +1,27 @@
|
|||||||
# D&D Spell-Catalog Overlay Contract
|
# D&D Spell-Catalog Overlays
|
||||||
|
|
||||||
This document defines the JSON format accepted by the D&D spell catalog
|
This document defines the optional JSON overlay consumed by the D&D spell
|
||||||
resolver. An overlay supplies campaign-specific spell names and aliases for
|
extractor. An overlay contributes campaign spell names and aliases for
|
||||||
recognition. It does not supply spell rules, levels, classes, effects, or
|
recognition. It does not define spell rules, effects, levels, classes, or
|
||||||
source evidence.
|
transcript evidence. Bind the optional `spell_catalog` reference as described
|
||||||
|
in [Configuration](../config.md#references-and-ordered-handoffs).
|
||||||
|
|
||||||
The `dnd/spells` extractor accepts one optional UTF-8 `application/json` overlay
|
## Contract Identity
|
||||||
bundle through its `spell_catalog` reference slot. The framework materializes
|
|
||||||
that file relative to the configuration or command-line binding, enforces the
|
|
||||||
1 MiB slot limit, and records its origin and raw digest separately from the
|
|
||||||
effective catalog digest.
|
|
||||||
|
|
||||||
## Shape
|
| Property | Value |
|
||||||
|
| --- | --- |
|
||||||
|
| Consumer | D&D spell extraction and normalization |
|
||||||
|
| Reference slot | `spell_catalog` |
|
||||||
|
| Media type | `application/json` |
|
||||||
|
| Required schema version | `notarius.dnd.spell-catalog-overlay.v1` |
|
||||||
|
| Base catalog | Embedded D&D 5e 2014 SRD catalog |
|
||||||
|
|
||||||
An overlay bundle has this shape:
|
At most one overlay document may be bound. The maintained example is
|
||||||
|
[dnd-spell-catalog.json](../../examples/dnd-spell-catalog.json).
|
||||||
|
|
||||||
|
## Wire Shape
|
||||||
|
|
||||||
|
This is a minimal valid overlay:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
@@ -22,49 +30,43 @@ An overlay bundle has this shape:
|
|||||||
{
|
{
|
||||||
"id": "campaign.example",
|
"id": "campaign.example",
|
||||||
"ruleset": "dnd-5e-2014",
|
"ruleset": "dnd-5e-2014",
|
||||||
"source": {
|
"source": {"title": "Example campaign spells"},
|
||||||
"title": "Example campaign spells",
|
"spells": [{"name": "Aegis of Emberfall"}]
|
||||||
"version": "1",
|
|
||||||
"url": "",
|
|
||||||
"license": ""
|
|
||||||
},
|
|
||||||
"spells": [
|
|
||||||
{
|
|
||||||
"name": "Aegis of Emberfall",
|
|
||||||
"aliases": ["Emberfall Aegis"]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
The top-level `schema_version` and `catalogs` fields are required. The schema
|
| Field | Required | Meaning and constraints |
|
||||||
version must be exactly `notarius.dnd.spell-catalog-overlay.v1`, and at least
|
| --- | --- | --- |
|
||||||
one catalog is required. Catalogs require a unique, non-empty, trimmed `id`,
|
| `schema_version` | Yes | Exactly `notarius.dnd.spell-catalog-overlay.v1`. |
|
||||||
the exact `dnd-5e-2014` `ruleset`, a `source`, and a non-empty `spells` array.
|
| `catalogs` | Yes | Non-empty array of catalog objects with unique IDs. |
|
||||||
|
| `catalogs[].id` | Yes | Non-empty trimmed string. |
|
||||||
|
| `catalogs[].ruleset` | Yes | Exactly `dnd-5e-2014`. |
|
||||||
|
| `catalogs[].source.title` | Yes | Non-empty trimmed string. |
|
||||||
|
| `catalogs[].source.version` | No | String when present. |
|
||||||
|
| `catalogs[].source.url` | No | String when present. |
|
||||||
|
| `catalogs[].source.license` | No | String when present. |
|
||||||
|
| `catalogs[].spells` | Yes | Non-empty array of spell objects. |
|
||||||
|
| `catalogs[].spells[].name` | Yes | Non-empty trimmed string. |
|
||||||
|
| `catalogs[].spells[].aliases` | No | Array of non-empty trimmed strings when present. |
|
||||||
|
|
||||||
`source.title` is required and must be non-empty and trimmed. `source.version`,
|
Unknown fields are rejected at every object level. The document must contain
|
||||||
`source.url`, and `source.license` are optional strings and may be empty.
|
one JSON value; `null` is not accepted for optional strings or aliases.
|
||||||
Each spell requires a non-empty, trimmed `name`. `aliases` may be omitted or
|
|
||||||
may be an array of trimmed, non-empty strings; JSON `null` is not an alias
|
|
||||||
array. Overlay objects contain no other supported spell fields.
|
|
||||||
|
|
||||||
Decoding is strict: unknown fields, malformed JSON, trailing JSON values, and
|
## Composition And Compatibility
|
||||||
non-string optional source fields are rejected.
|
|
||||||
|
|
||||||
## Composition
|
Notarius starts with the embedded base catalog, then applies overlay catalogs
|
||||||
|
in ascending catalog-ID order. A new canonical spell name adds a recognition
|
||||||
|
entry. If an overlay names an existing canonical spell, it augments that spell
|
||||||
|
with aliases while retaining the established display spelling.
|
||||||
|
|
||||||
The resolver always starts with the embedded D&D 5e 2014 SRD catalog. Overlay
|
Repeated aliases for the same spell are accepted. A canonical-name, canonical-
|
||||||
catalogs are sorted by `id` before composition, so the input order does not
|
to-alias, or alias-to-alias collision between different spells is rejected,
|
||||||
affect the result. A new canonical name adds a recognition entry. A canonical
|
including a collision with the embedded catalog. Matching uses the catalog’s
|
||||||
name matching an existing canonical name augments that spell and keeps the
|
case, whitespace, and apostrophe normalization, so authors should avoid names
|
||||||
established canonical display spelling. Repeated aliases for the same spell
|
or aliases that normalize to another spell.
|
||||||
are idempotent.
|
|
||||||
|
|
||||||
Canonical-name display conflicts and canonical/alias or alias/alias collisions
|
The overlay is a recognition aid only. The durable spell-artifact schema and
|
||||||
between different spells are errors, including collisions with the embedded
|
source-evidence rules are defined by the
|
||||||
catalog. Canonical names and aliases use the catalog's case, whitespace, and
|
[D&D spell artifact contract](dnd-spell-artifacts.md).
|
||||||
common-apostrophe normalization rules. The effective catalog returns canonical
|
|
||||||
names in sorted order and produces a semantic SHA-256 digest that is stable
|
|
||||||
under JSON formatting, object-key, catalog, spell, and alias reordering.
|
|
||||||
|
|||||||
@@ -1,201 +1,111 @@
|
|||||||
# JSON Output
|
# Published JSON Output
|
||||||
|
|
||||||
This document is the durable JSON output file-format contract produced by the
|
This document defines the logical JSON bundle emitted by the production JSON
|
||||||
production JSON encoder and written by the CLI. Selectable output-encoder keys
|
output encoder. The bundle’s physical destination, atomic publication, and
|
||||||
are cataloged in
|
retention are operational concerns; see [Operations](../operations.md#output-bundles).
|
||||||
[Configuration](../config.md#implemented-production-modules).
|
Output configuration, including chunk-map export, belongs in
|
||||||
|
[Configuration](../config.md#module-bindings-and-validators).
|
||||||
|
|
||||||
The output module produces the logical bundle described here. The CLI's
|
## Bundle Layout
|
||||||
physical placement and lifecycle for that bundle are defined in
|
|
||||||
[Operations](../operations.md#output-directory).
|
|
||||||
|
|
||||||
## Files
|
All paths below are logical, relative, slash-separated bundle paths. The
|
||||||
|
encoder always emits the first four JSON files below and adds lane or chunk-map
|
||||||
|
files when their corresponding artifacts are available:
|
||||||
|
|
||||||
The encoder writes:
|
| Path | Purpose |
|
||||||
|
| --- | --- |
|
||||||
|
| `index.json` | Entry point that names the other published files and lane payloads. |
|
||||||
|
| `manifest.json` | Run provenance and result summaries. |
|
||||||
|
| `rejected.json` | Rejected pipeline outputs. |
|
||||||
|
| `warnings.json` | Accepted-output and run warnings. |
|
||||||
|
| `lanes/<safe-lane-id>.json` | One normalized artifact payload for each lane. |
|
||||||
|
| `chunk-map.json` | Optional accepted chunk map, when its export is enabled and available. |
|
||||||
|
|
||||||
- `index.json`
|
JSON files are pretty-printed with a trailing newline. Lane payloads are
|
||||||
- `manifest.json`
|
accepted only when their media type is `application/json`.
|
||||||
- `lanes/<lane-id>.json`, one file per normalized serialized artifact
|
|
||||||
- `rejected.json`
|
|
||||||
- `warnings.json`
|
|
||||||
- `chunk-map.json`, only when the JSON output binding enables
|
|
||||||
`include_chunk_map` and the run has an accepted chunk map
|
|
||||||
|
|
||||||
Files are pretty-printed JSON with a trailing newline when the payload is JSON.
|
|
||||||
Logical file paths are relative, slash-separated, and may not contain `..`.
|
|
||||||
|
|
||||||
## `index.json`
|
## `index.json`
|
||||||
|
|
||||||
Shape:
|
`index.json` is the bundle’s discovery document. An approved run with no
|
||||||
|
normalized lanes has this valid minimal index:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"manifest_file": "manifest.json",
|
"manifest_file": "manifest.json",
|
||||||
"output_files": [
|
"output_files": [],
|
||||||
{
|
|
||||||
"lane_id": "spells",
|
|
||||||
"media_type": "application/json",
|
|
||||||
"file": "lanes/spells.json",
|
|
||||||
"module_key": "noop",
|
|
||||||
"schema_id": "notarius.dnd.spells",
|
|
||||||
"schema_name": "notarius_dnd_spells_v1",
|
|
||||||
"schema_version": "v1"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"rejected_file": "rejected.json",
|
"rejected_file": "rejected.json",
|
||||||
"warnings_file": "warnings.json"
|
"warnings_file": "warnings.json"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
`output_files` is sorted by lane ID. Output file names are produced by
|
| Field | Required | Meaning |
|
||||||
sanitizing the lane ID:
|
| --- | --- | --- |
|
||||||
|
| `manifest_file` | Yes | Always `manifest.json`. |
|
||||||
|
| `output_files` | Yes | Lane descriptors sorted by `lane_id`. |
|
||||||
|
| `rejected_file` | Yes | Always `rejected.json`. |
|
||||||
|
| `warnings_file` | Yes | Always `warnings.json`. |
|
||||||
|
| `chunk_map` | No | Descriptor for the pipeline-wide `chunk-map.json`; never a lane descriptor. |
|
||||||
|
|
||||||
- characters outside `A-Z`, `a-z`, `0-9`, `.`, `_`, and `-` become `_`;
|
Each lane descriptor has required `lane_id` and `file`. It may also include
|
||||||
- repeated `..` sequences are replaced;
|
`media_type`, `module_key`, `schema_id`, `schema_name`, and `schema_version`
|
||||||
- leading and trailing `.`, `_`, and `-` are trimmed;
|
when supplied by the normalized artifact. A `chunk_map` descriptor contains
|
||||||
- empty sanitized names are rejected;
|
|
||||||
- two lanes that sanitize to the same output file are rejected.
|
|
||||||
|
|
||||||
`manifest_file`, `rejected_file`, and `warnings_file` contain the fixed paths
|
|
||||||
shown above. Each `output_files` entry requires `lane_id` and `file`. It also
|
|
||||||
contains the normalized payload `media_type`, normalizer `module_key`, and
|
|
||||||
response `schema_id`, `schema_name`, and `schema_version` when those values are
|
|
||||||
available.
|
|
||||||
|
|
||||||
When present, the top-level optional `chunk_map` descriptor contains exactly
|
|
||||||
`artifact_kind`, `file`, `media_type`, `schema_id`, `schema_name`, and
|
`artifact_kind`, `file`, `media_type`, `schema_id`, `schema_name`, and
|
||||||
`schema_version`. It identifies the pipeline-wide `chunk-map.json`; it is not
|
`schema_version`; its payload is defined by the
|
||||||
a lane output and never appears in `output_files`. The descriptor and file are
|
[Accepted Chunk Map contract](chunk-map.md).
|
||||||
both absent when export is disabled or no chunk plan was accepted. Its payload
|
|
||||||
contract is defined by [Accepted Chunk Map](chunk-map.md).
|
The lane path is derived from its lane ID. Characters outside letters, digits,
|
||||||
|
periods, underscores, and hyphens become underscores; `..` sequences are
|
||||||
|
neutralized; leading and trailing periods and underscores are removed. A lane
|
||||||
|
that produces an empty name, or two lanes that produce the same path, makes
|
||||||
|
output encoding fail.
|
||||||
|
|
||||||
|
## Lane Payloads
|
||||||
|
|
||||||
|
Each `lanes/<safe-lane-id>.json` file is the codec-owned normalized JSON for
|
||||||
|
that lane. Consumers should use the index descriptor’s schema identity rather
|
||||||
|
than infer a lane schema from its name. The current D&D payload contracts are
|
||||||
|
[spells](dnd-spell-artifacts.md), [NPCs](dnd-npc-artifacts.md),
|
||||||
|
[NPC interactions](dnd-npc-interaction-artifacts.md),
|
||||||
|
[combat turns](dnd-combat-turn-artifacts.md),
|
||||||
|
[item events](dnd-item-event-artifacts.md), and
|
||||||
|
[scene descriptions](dnd-scene-description-artifacts.md).
|
||||||
|
|
||||||
## `manifest.json`
|
## `manifest.json`
|
||||||
|
|
||||||
`manifest.json` contains a run manifest. This abridged example shows its core
|
`manifest.json` is published provenance, not a copy of lane payloads or a
|
||||||
structure:
|
checkpoint store. Fields without a value may be omitted. Its top-level fields
|
||||||
|
group into the following externally observable summaries:
|
||||||
|
|
||||||
```json
|
| Group | Fields |
|
||||||
{
|
| --- | --- |
|
||||||
"run_id": "run-123",
|
| Run identity and result | `run_id`, `pipeline_id`, `pipeline_digest`, `schema_version`, `validation_status`, `started_at`, `completed_at` |
|
||||||
"pipeline_id": "dnd-session",
|
| Resolved components | `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, `output_encoder`, `artifact_lanes`, `validator_chains`, `module_metadata` |
|
||||||
"artifact_lanes": [
|
| Source and references | `source_digests`, `references` |
|
||||||
{
|
| Published result summaries | `normalized_outputs`, `rejected_outputs` |
|
||||||
"id": "spells",
|
| Execution summaries | `chunk_plan`, `checkpoint_decisions`, `llm_profiles`, `metadata` |
|
||||||
"extractor": "dnd/spells",
|
|
||||||
"merger": "appendorder",
|
|
||||||
"normalizer": "noop"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"validation_status": "approved",
|
|
||||||
"started_at": "2026-01-01T00:00:00Z",
|
|
||||||
"completed_at": "2026-01-01T00:00:01Z"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Fields with empty values may be omitted by JSON encoding.
|
`references` records provenance such as the target, slot, origin, digest,
|
||||||
|
media type, size, and generated-artifact identity. It does not contain
|
||||||
|
reference content. `normalized_outputs` and `rejected_outputs` likewise
|
||||||
|
summarize results without embedding lane payload bytes. A chunk-plan summary is
|
||||||
|
provenance for the plan used by this run; cache records, debug artifacts, and
|
||||||
|
other operational state are not published as bundle files.
|
||||||
|
|
||||||
The manifest fields are:
|
## Rejections And Warnings
|
||||||
|
|
||||||
- `run_id`, `pipeline_id`, and `pipeline_digest`: run and resolved-pipeline
|
`rejected.json` is always an object with a `rejected` array. Each entry has
|
||||||
identity;
|
required `stage` and `message`; `step_id`, `lane_id`, `module_key`, `chunk_id`,
|
||||||
- `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, and
|
`chunk_index`, `validator_name`, `reason_code`, `attempt_count`, and
|
||||||
`output_encoder`: resolved module keys;
|
`diagnostic_artifact_path` are present only when applicable.
|
||||||
- `chunk_plan`: payload-free provenance for the effective chunk plan. `mode`
|
|
||||||
is the effective cache mode; `action` is `reused`, `generated`,
|
|
||||||
`refreshed`, or `bypassed` when a plan was materialized. `requested_module`
|
|
||||||
is the current pipeline chunker, while `producer_input_module`,
|
|
||||||
`producer_module`, `producer_llm_profile`, `producer_references`,
|
|
||||||
`producer_metadata`, `source_digest`, `plan_digest`, `plan_schema_version`,
|
|
||||||
and `created_at` describe the stored or generated producer when available.
|
|
||||||
A cached plan can therefore identify a producer different from the requested
|
|
||||||
module. This object never embeds ranges, units, annotations, prompts,
|
|
||||||
responses, or reference content;
|
|
||||||
- `module_metadata` and `artifact_lanes`: module and per-lane provenance,
|
|
||||||
including prompt and response-schema provenance when provided;
|
|
||||||
- `validator_chains`: resolved validation points and validators;
|
|
||||||
- `source_digests` and `references`: source and reference provenance;
|
|
||||||
- `normalized_outputs` and `rejected_outputs`: payload-free result summaries;
|
|
||||||
- `llm_profiles`: selected profile IDs and provider or model names when
|
|
||||||
available;
|
|
||||||
- `metadata`: the effective prompt `session_id`;
|
|
||||||
- `validation_status`: `approved` or `rejected`;
|
|
||||||
- `started_at` and `completed_at`: UTC run timestamps.
|
|
||||||
|
|
||||||
`source_digests` contains source document digests only. Bound references are
|
`warnings.json` is always an object with a `warnings` array. Each warning has
|
||||||
recorded separately under `references`, which contains provenance only: target
|
`reason_code` and `message`; `scope` is optional. Both arrays are empty when
|
||||||
stage, lane ID when present, slot name, origin type and URI, digest, media
|
there is nothing to report.
|
||||||
type, byte size, and binding source. Reference content is not written to
|
|
||||||
durable output.
|
|
||||||
|
|
||||||
Reference `stage` is `chunk`, `extract`, `merge`, or `normalize`. `lane_id` is
|
## Compatibility
|
||||||
omitted for chunk references and present for extract, merge, and normalize
|
|
||||||
references.
|
|
||||||
|
|
||||||
`validation_status` is `approved` when no outputs were rejected and `rejected`
|
The index is the authoritative map from a logical lane to its published
|
||||||
when one or more outputs were rejected.
|
payload. Consumers must tolerate omitted optional manifest and descriptor
|
||||||
|
fields, and should rely on the linked artifact contract for each lane’s JSON
|
||||||
Producer warnings and the current run's chunk-validation warnings remain in
|
shape. This contract describes the published logical bundle only; it does not
|
||||||
`warnings.json`. The manifest records only provenance and decision summaries;
|
promise a filesystem layout or expose internal state formats.
|
||||||
empty producer-only values are omitted for compatibility with existing readers.
|
|
||||||
|
|
||||||
`validator_chains` records the resolved validator chain for each validation
|
|
||||||
point. Entries include stage, lane ID when applicable, module key, and validators
|
|
||||||
with key and execution class. Empty chains are recorded with an empty
|
|
||||||
`validators` array, including chains resolved from explicit empty config
|
|
||||||
overrides.
|
|
||||||
|
|
||||||
`normalized_outputs` summarizes each normalized lane output without embedding
|
|
||||||
payload bytes. Entries include lane ID, normalizer module key, source ID, media
|
|
||||||
type, and response schema provenance where available.
|
|
||||||
|
|
||||||
`rejected_outputs` summarizes rejected module outputs without embedding raw
|
|
||||||
payload bytes. Entries include stage, lane, module, chunk, validator or reason,
|
|
||||||
message, attempt count, and optional diagnostic artifact path.
|
|
||||||
|
|
||||||
## Output Payload Files
|
|
||||||
|
|
||||||
Each normalized serialized artifact is written to
|
|
||||||
`lanes/<sanitized-lane-id>.json`. The JSON output encoder is domain-neutral and
|
|
||||||
accepts only artifacts whose codec media type is `application/json`. The file
|
|
||||||
contains the codec-owned JSON bytes pretty-printed.
|
|
||||||
|
|
||||||
The schema of each lane payload is owned by that artifact contract. For the
|
|
||||||
current D&D lanes, see [D&D Spell Artifact](dnd-spell-artifacts.md),
|
|
||||||
[D&D NPC Artifact](dnd-npc-artifacts.md), and
|
|
||||||
[D&D Combat-Turn Artifact](dnd-combat-turn-artifacts.md), and
|
|
||||||
[D&D Scene Description Artifact](dnd-scene-description-artifacts.md).
|
|
||||||
|
|
||||||
## `rejected.json`
|
|
||||||
|
|
||||||
Shape:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"rejected": []
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
When output validation rejects an output, each entry contains `stage` and
|
|
||||||
`message`. It includes `lane_id`, `module_key`, `chunk_id`, `chunk_index`,
|
|
||||||
`validator_name`, `reason_code`, `attempt_count`, and
|
|
||||||
`diagnostic_artifact_path` when applicable.
|
|
||||||
|
|
||||||
## `warnings.json`
|
|
||||||
|
|
||||||
Shape:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"warnings": [
|
|
||||||
{
|
|
||||||
"scope": "extract",
|
|
||||||
"reason_code": "example",
|
|
||||||
"message": "human-readable warning"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
`warnings` is an empty array when no warnings are reported.
|
|
||||||
Each warning requires `reason_code` and `message`; `scope` is omitted when it is
|
|
||||||
empty.
|
|
||||||
|
|||||||
@@ -1,69 +1,73 @@
|
|||||||
# Seriatim Transcript JSON
|
# Seriatim Transcript Input
|
||||||
|
|
||||||
This document is the external input contract consumed by the production
|
This document defines the JSON transcript accepted by the production Seriatim
|
||||||
Seriatim input adapter. Selectable input-adapter keys are cataloged in
|
input adapter. It is a source input, not a durable lane artifact. Configure the
|
||||||
[Configuration](../config.md#implemented-production-modules).
|
input adapter through [Configuration](../config.md#production-module-keys).
|
||||||
|
|
||||||
## Adapter
|
## Contract Identity
|
||||||
|
|
||||||
- Source format: `application/vnd.seriatim+json`
|
| Property | Value |
|
||||||
|
| --- | --- |
|
||||||
|
| Consumer | Seriatim input adapter |
|
||||||
|
| Media type | `application/vnd.seriatim+json` |
|
||||||
|
| Source document kind | `transcript` |
|
||||||
|
| Source-unit kind | `transcript_segment` |
|
||||||
|
|
||||||
## Accepted Shape
|
## Accepted Shape
|
||||||
|
|
||||||
The input must be one JSON object with top-level `metadata` and `segments`
|
The input is one JSON object containing `metadata` and a non-empty `segments`
|
||||||
fields. This covers the maintained minimal fixture and Seriatim intermediate
|
array. This minimal document is valid:
|
||||||
output that provides the same required segment fields.
|
|
||||||
|
|
||||||
The maintained example is
|
```json
|
||||||
[examples/seriatim-minimal-transcript.json](../../examples/seriatim-minimal-transcript.json).
|
{
|
||||||
|
"metadata": {"id": "session-alpha"},
|
||||||
|
"segments": [
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"start": 0,
|
||||||
|
"end": 4,
|
||||||
|
"speaker": "Aria",
|
||||||
|
"text": "Aria casts Cure Wounds."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
Required top-level fields:
|
The maintained two-segment input is
|
||||||
|
[seriatim-minimal-transcript.json](../../examples/seriatim-minimal-transcript.json).
|
||||||
|
|
||||||
- `metadata`: an object. Its entries are accepted as source metadata.
|
| Field | Required | Meaning and constraints |
|
||||||
- `segments`: a non-empty array of segment objects.
|
| --- | --- | --- |
|
||||||
|
| `metadata` | Yes | JSON object. Its entries become source metadata; no particular metadata key is otherwise required. |
|
||||||
|
| `segments` | Yes | Non-empty array of segment objects, kept in input order. |
|
||||||
|
| `segments[].id` | Yes | Positive canonical decimal integer, supplied as a JSON number or string. IDs must be unique. |
|
||||||
|
| `segments[].start` | Yes | Finite, non-negative numeric value, supplied as a JSON number or string. |
|
||||||
|
| `segments[].end` | Yes | Finite, non-negative numeric value that is not earlier than `start`. |
|
||||||
|
| `segments[].speaker` | Yes | String that is non-empty after trimming. |
|
||||||
|
| `segments[].text` | Yes | String that is non-empty after trimming. Its original text is retained. |
|
||||||
|
|
||||||
Required segment fields:
|
Additional top-level and segment fields are ignored. A missing required field,
|
||||||
|
`null` in place of an object or array, malformed JSON, or more than one
|
||||||
|
top-level JSON value is rejected.
|
||||||
|
|
||||||
- `id`: a positive integer JSON number or canonical decimal string without
|
## Source Identity And References
|
||||||
leading zeros or surrounding whitespace;
|
|
||||||
- `start`: a finite, non-negative JSON number or numeric string;
|
|
||||||
- `end`: a finite, non-negative JSON number or numeric string that is not less
|
|
||||||
than `start`;
|
|
||||||
- `speaker`: a non-empty string;
|
|
||||||
- `text`: a non-empty string.
|
|
||||||
|
|
||||||
Other top-level and segment fields, such as `categories`, are ignored.
|
The adapter chooses the source ID in this order:
|
||||||
|
|
||||||
Multiple top-level JSON values are rejected.
|
1. a non-empty source ID supplied by the calling request;
|
||||||
|
2. non-empty string `metadata.id`;
|
||||||
|
3. non-empty string `metadata.source_id`;
|
||||||
|
4. `seriatim:` followed by the first 16 hexadecimal characters of the raw
|
||||||
|
input’s SHA-256 digest.
|
||||||
|
|
||||||
## Validation
|
Each accepted segment becomes one source unit whose unit ID is `segments[].id`.
|
||||||
|
Its self-reference uses the derived source ID and the same segment ID for both
|
||||||
|
range endpoints. Artifact contracts use those segment IDs when they cite
|
||||||
|
transcript evidence.
|
||||||
|
|
||||||
The adapter rejects empty input, malformed JSON, multiple top-level JSON values,
|
## Compatibility
|
||||||
non-object segment values, duplicate segment IDs, and any violation of the
|
|
||||||
shape or field constraints above.
|
|
||||||
|
|
||||||
Segment text is preserved as provided, but it must not be empty after trimming.
|
This adapter accepts only the shape described here. A broader Seriatim export
|
||||||
|
is usable only when it supplies this object, metadata, and segment shape with
|
||||||
## Derived Identity
|
the stated types and constraints. Unknown additional fields do not add
|
||||||
|
Notarius behavior.
|
||||||
Notarius identifies the parsed source in this order:
|
|
||||||
|
|
||||||
1. `metadata.id`, when it is a non-empty string after trimming;
|
|
||||||
2. `metadata.source_id`, when it is a non-empty string after trimming;
|
|
||||||
3. `seriatim:<first-16-hex-chars-of-raw-sha256>`.
|
|
||||||
|
|
||||||
The exact raw input SHA-256 remains the basis of the fallback source ID. The
|
|
||||||
source digest recorded in output provenance is instead the SHA-256 of the
|
|
||||||
canonical generic source document, excluding the digest field itself. It covers
|
|
||||||
the derived source identity, document kind and format, ordered units and their
|
|
||||||
self-references, and accepted metadata. Segment IDs become the unit IDs used by
|
|
||||||
artifact source references; each produced unit carries a self-reference whose
|
|
||||||
source ID is the derived document ID and whose start and end IDs both equal the
|
|
||||||
segment ID.
|
|
||||||
|
|
||||||
## Compatibility Limit
|
|
||||||
|
|
||||||
This contract covers only Seriatim transcript JSON with the top-level
|
|
||||||
`metadata` object and `segments` array described here. Broader Seriatim output
|
|
||||||
schemas are compatible only when they provide these required fields with the
|
|
||||||
accepted types.
|
|
||||||
|
|||||||
133
docs/internal/cli.md
Normal file
133
docs/internal/cli.md
Normal file
@@ -0,0 +1,133 @@
|
|||||||
|
# CLI Internals
|
||||||
|
|
||||||
|
This document describes **internal/cli**, Notarius's production composition
|
||||||
|
root. The [CLI reference](../cli.md) owns command syntax and exit statuses;
|
||||||
|
[Configuration](../config.md) owns configuration values; and
|
||||||
|
[Operations](../operations.md) owns filesystem layout, recovery, and operator
|
||||||
|
procedures.
|
||||||
|
|
||||||
|
## Inputs, Outputs, And Boundaries
|
||||||
|
|
||||||
|
The CLI accepts process arguments, standard streams, and injectable options
|
||||||
|
used by tests and embedding code. It writes command results to the supplied
|
||||||
|
streams and returns a process exit status. For a run, it also creates the
|
||||||
|
production catalog and runtime collaborators, hands a prepared pipeline and
|
||||||
|
source bytes to the framework, and places the logical files returned by the
|
||||||
|
runner.
|
||||||
|
|
||||||
|
It is the only boundary allowed to compose concrete registries, LLM clients,
|
||||||
|
cache/checkpoint collaborators, debug recorders, and physical output paths.
|
||||||
|
Pipeline modules receive interfaces and request data rather than CLI streams or
|
||||||
|
filesystem roots. The [Architecture](../policy/architecture.md) defines this
|
||||||
|
composition-root boundary; [Pipeline Internals](pipeline.md) owns resolution,
|
||||||
|
preparation, and runner mechanics after their inputs are supplied.
|
||||||
|
|
||||||
|
## Dispatch And Configuration Handoff
|
||||||
|
|
||||||
|
The root dispatcher handles help, configuration validation, pipeline listing,
|
||||||
|
and a pipeline run. It normalizes injectable options before dispatch so that a
|
||||||
|
missing production dependency fails as a command error rather than reaching
|
||||||
|
execution.
|
||||||
|
|
||||||
|
Commands that need configuration use one shared loader. The CLI discovers the
|
||||||
|
file, parses it through **internal/core/config**, starts from defaults, applies
|
||||||
|
the file and supported environment overrides, and then validates it for the
|
||||||
|
command. The configured discovery and precedence contract is in
|
||||||
|
[Configuration](../config.md), while the loading and resolution mechanics are
|
||||||
|
in [Configuration Internals](configuration.md).
|
||||||
|
|
||||||
|
Configuration validation without a selected pipeline checks structural
|
||||||
|
configuration only. Validation with a selected pipeline also builds the
|
||||||
|
effective catalog, resolves the pipeline, and verifies explicitly selected
|
||||||
|
Scriptorium profiles. Pipeline listing validates configuration before returning
|
||||||
|
normalized, sorted identifiers.
|
||||||
|
|
||||||
|
## Production Composition
|
||||||
|
|
||||||
|
The production composition helper allocates every framework registry and the
|
||||||
|
prompt-asset registry, then registers the generic, Seriatim, and D&D module
|
||||||
|
families in that order. The resulting registries provide both the module
|
||||||
|
catalog used for resolution and the concrete constructors used for preparation.
|
||||||
|
Tests may provide a catalog or registries instead; production code must not
|
||||||
|
silently merge an injected partial catalog with production registrations.
|
||||||
|
|
||||||
|
The production LLM factory builds the Scriptorium-backed client from resolved
|
||||||
|
configuration, creates one scheduler from the effective global LLM limit, and
|
||||||
|
wraps the client before it reaches modules. Registration and LLM construction
|
||||||
|
errors are returned before a pipeline is prepared. Concrete module keys and
|
||||||
|
validator chains are public configuration choices and remain documented in
|
||||||
|
[Configuration](../config.md).
|
||||||
|
|
||||||
|
## Run Orchestration
|
||||||
|
|
||||||
|
After parsing and validating a run invocation, the CLI performs this ordered
|
||||||
|
handoff:
|
||||||
|
|
||||||
|
1. load and validate configuration, then apply command-level operational
|
||||||
|
overrides;
|
||||||
|
2. create and validate a safe run identity, then allocate a debug bundle only
|
||||||
|
when requested;
|
||||||
|
3. build the effective catalog, resolve requested reference changes, resolve
|
||||||
|
the effective pipeline, and verify explicit Scriptorium profiles;
|
||||||
|
4. materialize external or generated references and record redacted invocation
|
||||||
|
and resolution provenance when debug capture is enabled;
|
||||||
|
5. construct registries, the scheduled LLM client, prepared modules, and the
|
||||||
|
requested cache/checkpoint collaborators;
|
||||||
|
6. read the source input and invoke the framework runner; and
|
||||||
|
7. write the runner's logical output files only after a successful run, then
|
||||||
|
complete the command report and user-facing result.
|
||||||
|
|
||||||
|
Preparation happens before source parsing, so module construction and
|
||||||
|
dependency failures cannot begin stage execution. The CLI also preserves the
|
||||||
|
framework's result and warning information when it writes summaries and the
|
||||||
|
final command result. Detailed state lifecycle, resume handling, and physical
|
||||||
|
path confinement are maintained in [Run State Internals](state.md) and
|
||||||
|
[Operations](../operations.md).
|
||||||
|
|
||||||
|
## Failure Mapping And Terminal Reporting
|
||||||
|
|
||||||
|
Argument, flag, and invocation-combination failures are reported to standard
|
||||||
|
error before runtime composition and use the syntax error class. Once an
|
||||||
|
invocation is syntactically valid, configuration loading and validation,
|
||||||
|
resolution, registration, profile checks, reference materialization, module
|
||||||
|
construction, input reads, runner failures, output publication, and requested
|
||||||
|
debug handling use the runtime failure class. The public status numbers and
|
||||||
|
stream contract are defined in the [CLI reference](../cli.md#output-streams-and-exit-statuses).
|
||||||
|
|
||||||
|
When debug capture has been allocated, one command-state value records the
|
||||||
|
known run result. Guarded terminalization writes a success report once, or
|
||||||
|
attempts a failure report and error record once. A persistence failure is
|
||||||
|
reported in addition to the original failure and never replaces it. If a debug
|
||||||
|
path exists, failure output includes that path so the retained diagnostic data
|
||||||
|
is discoverable.
|
||||||
|
|
||||||
|
## Invariants To Preserve
|
||||||
|
|
||||||
|
- Only the CLI composes production implementations and physical runtime roots.
|
||||||
|
- Configuration and resolved composition failures occur before module
|
||||||
|
preparation or source parsing.
|
||||||
|
- A runner's logical files are published only after a successful run.
|
||||||
|
- Production registries and a caller-supplied catalog or registries are
|
||||||
|
alternative composition sources, not an implicit mixture.
|
||||||
|
- A requested debug bundle has one terminal report attempt; its persistence
|
||||||
|
errors supplement rather than obscure the primary command error.
|
||||||
|
- User-facing flags, paths, exit codes, and configuration fields are defined
|
||||||
|
by their public documentation, not duplicated here.
|
||||||
|
|
||||||
|
## Focused Tests
|
||||||
|
|
||||||
|
- **internal/cli/command_contract_test.go** covers dispatch, help, syntax and
|
||||||
|
runtime error classes, discovery, validation, and listing.
|
||||||
|
- **internal/cli/run_contract_test.go** covers the run handoff, publication,
|
||||||
|
debug reporting, and command-owned state collaborators.
|
||||||
|
- **internal/cli/production_contract_test.go** covers registrar composition,
|
||||||
|
production catalog contents, assets, and representative configuration
|
||||||
|
validation.
|
||||||
|
- **internal/cli/reference_contract_test.go** covers CLI reference overrides,
|
||||||
|
origin separation, and materialization boundaries.
|
||||||
|
- **internal/cli/state_hardening_test.go** covers safe run identity, state
|
||||||
|
roots, and failure ordering.
|
||||||
|
|
||||||
|
Run **go test ./internal/cli** after changing command composition or command
|
||||||
|
behavior. Pair it with **go test ./internal/core/config** when the configuration
|
||||||
|
handoff changes.
|
||||||
129
docs/internal/configuration.md
Normal file
129
docs/internal/configuration.md
Normal file
@@ -0,0 +1,129 @@
|
|||||||
|
# Configuration Internals
|
||||||
|
|
||||||
|
This document describes the maintainer-facing configuration boundary in
|
||||||
|
**internal/core/config**. The [Configuration](../config.md) reference owns the
|
||||||
|
file format, fields, defaults, precedence contract, and selectable keys. The
|
||||||
|
[CLI reference](../cli.md) owns command syntax; this document does not redefine
|
||||||
|
either interface.
|
||||||
|
|
||||||
|
## Boundary
|
||||||
|
|
||||||
|
The configuration package turns a selected YAML file and supported environment
|
||||||
|
values into a validated, independently owned configuration. It then resolves a
|
||||||
|
requested pipeline against a module catalog before the framework prepares or
|
||||||
|
runs anything.
|
||||||
|
|
||||||
|
| Boundary | Inputs | Outputs | Does not own |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| Loading | Selected file path and environment lookup | Parsed file model and a populated **Config** | Choosing the file path or reporting a command result. |
|
||||||
|
| Validation | **Config** | Structural configuration errors with pipeline, lane, or binding context | Module availability, capabilities, or construction. |
|
||||||
|
| Resolution | Valid **Config**, selected pipeline and lanes, runtime reference changes, LLM override, and module catalog | **EffectiveConfig** with a **ResolvedPipeline** | Materializing reference bytes, preparing modules, execution, or filesystem state. |
|
||||||
|
| Summary | **Config** or **EffectiveConfig** | Detached redacted payload suitable for debug summaries | Redacting arbitrary process state or provider traffic. |
|
||||||
|
|
||||||
|
The CLI discovers a configuration file, invokes this package, and supplies the
|
||||||
|
result to the framework. Configuration never reads an input file, constructs a
|
||||||
|
module, or creates output, cache, or debug paths. Those responsibilities remain
|
||||||
|
at their respective [CLI](cli.md), [pipeline](pipeline.md), and
|
||||||
|
[run-state](state.md) boundaries.
|
||||||
|
|
||||||
|
## Loading And Validation
|
||||||
|
|
||||||
|
The CLI loads configuration in this order:
|
||||||
|
|
||||||
|
1. parse the selected YAML file strictly into the file model;
|
||||||
|
2. start from **Default**;
|
||||||
|
3. apply the file model; and
|
||||||
|
4. apply the supported environment overrides.
|
||||||
|
|
||||||
|
This establishes the public precedence order without giving environment input a
|
||||||
|
second file schema. Loading and application reject malformed YAML, unsupported
|
||||||
|
file versions, unknown fields, invalid values, and identifiers that are empty
|
||||||
|
or collide after whitespace normalization. The file application also makes the
|
||||||
|
effective extraction-worker default follow the effective LLM limit.
|
||||||
|
|
||||||
|
**Config.Validate** checks configuration-only invariants before resolution. It
|
||||||
|
rejects incompatible profile sources, invalid state-surface values, unsupported
|
||||||
|
concurrency settings, malformed bindings and references, invalid retries, and
|
||||||
|
invalid pipeline, step, or lane structure. Its errors retain the closest known
|
||||||
|
pipeline, lane, and binding context. It deliberately does not require modules
|
||||||
|
to be registered: that requires a catalog and belongs to resolution.
|
||||||
|
|
||||||
|
The exact user-selectable values and validation rules are defined in
|
||||||
|
[Configuration](../config.md). Keep additions to the file model, an
|
||||||
|
environment override, its validation, and that reference in the same change.
|
||||||
|
|
||||||
|
## Effective Resolution
|
||||||
|
|
||||||
|
**Config.Resolve** first recomputes derived concurrency defaults and validates
|
||||||
|
the configuration. It normalizes the requested pipeline ID, copies the selected
|
||||||
|
profile, applies a non-empty command-level LLM profile override to the
|
||||||
|
LLM-capable stage bindings, and calls the framework resolver with the requested
|
||||||
|
lane selection and reference changes.
|
||||||
|
|
||||||
|
The command-level override does not replace an explicitly selected validator
|
||||||
|
profile. Validator bindings remain part of the resolved validator chain and
|
||||||
|
are resolved under their own declared configuration.
|
||||||
|
|
||||||
|
The framework resolver supplies defaults, selects lanes, resolves validator
|
||||||
|
chains, checks registered module and artifact compatibility, validates module
|
||||||
|
options, and returns the fixed ordered pipeline shape. The resulting
|
||||||
|
**EffectiveConfig** retains the selected ID, requested selection and reference
|
||||||
|
changes, a clone of the input configuration, and the resolved pipeline.
|
||||||
|
Callers may therefore retain or modify their input slices and maps without
|
||||||
|
changing the resolved result, and later consumers cannot mutate the original
|
||||||
|
configuration through the effective value.
|
||||||
|
|
||||||
|
Resolution failures stop before module construction and source parsing. They
|
||||||
|
include an error path for an unconfigured pipeline, missing module, missing
|
||||||
|
capability, incompatible artifact variant, invalid option, invalid reference,
|
||||||
|
or invalid lane selection. CLI code maps these valid-invocation failures to the
|
||||||
|
runtime error class described in the [CLI reference](../cli.md#output-streams-and-exit-statuses).
|
||||||
|
|
||||||
|
## Resolved Identity And Redaction
|
||||||
|
|
||||||
|
The framework assigns the resolved pipeline a deterministic SHA-256 digest
|
||||||
|
after defaults, lane selection, module bindings, reference bindings, validator
|
||||||
|
chains, and artifact schema identity have been resolved. The digest excludes
|
||||||
|
its own stored value. It identifies resolved composition rather than raw YAML
|
||||||
|
bytes, a debug payload, or all runtime state. The CLI records it as invocation
|
||||||
|
provenance before execution; cache and checkpoint identity have additional
|
||||||
|
owners in [Run State Internals](state.md).
|
||||||
|
|
||||||
|
Configuration summaries must use **Redacted**, **RedactedSummaryPayload**, or
|
||||||
|
**RedactedResolvedPipelinePayload**, never a direct configuration marshal.
|
||||||
|
Those methods copy every binding and nested option container, replace values
|
||||||
|
whose key is credential-shaped with **[REDACTED]**, and omit materialized
|
||||||
|
reference content while retaining safe binding and reference provenance. The
|
||||||
|
payload must not alias the source configuration or resolved pipeline. This
|
||||||
|
redaction is deliberately narrow: it protects configuration summaries and does
|
||||||
|
not authorize recording arbitrary environment values or provider requests.
|
||||||
|
|
||||||
|
## Invariants To Preserve
|
||||||
|
|
||||||
|
- Defaults, YAML values, and environment values are applied in one direction;
|
||||||
|
later sources may override only their supported operational settings.
|
||||||
|
- A configuration is structurally valid before it is resolved, and a resolved
|
||||||
|
pipeline is compatible with the supplied catalog before preparation begins.
|
||||||
|
- Whitespace-normalized identifiers are unique wherever they identify a
|
||||||
|
pipeline, step, lane, worker, or reference slot.
|
||||||
|
- Resolution and summary generation return detached data. Redaction must cover
|
||||||
|
every configured and resolved binding, including nested validator bindings.
|
||||||
|
- The resolved digest changes when resolved composition changes and never
|
||||||
|
includes itself.
|
||||||
|
|
||||||
|
## Focused Tests
|
||||||
|
|
||||||
|
- **internal/core/config/file_config_contract_test.go** covers strict file
|
||||||
|
parsing, normalization, file application, and structural rejection.
|
||||||
|
- **internal/core/config/env_contract_test.go** covers supported operational
|
||||||
|
overrides and their precedence.
|
||||||
|
- **internal/core/config/validation_contract_test.go** covers configuration
|
||||||
|
invariants and contextual failures.
|
||||||
|
- **internal/core/config/effective_config_contract_test.go** covers defaults,
|
||||||
|
selections, overrides, resolution context, digest changes, and ownership.
|
||||||
|
- **internal/core/config/redaction_test.go** covers recursive credential
|
||||||
|
redaction, reference-content exclusion, and non-aliasing payloads.
|
||||||
|
|
||||||
|
Run **go test ./internal/core/config** after changing this boundary. Changes to
|
||||||
|
the handoff or resolved-composition semantics also need the focused framework
|
||||||
|
pipeline tests.
|
||||||
119
docs/internal/dnd.md
Normal file
119
docs/internal/dnd.md
Normal file
@@ -0,0 +1,119 @@
|
|||||||
|
# D&D Module Internals
|
||||||
|
|
||||||
|
This guide records the conventions shared by the production D&D module family.
|
||||||
|
It complements [Module Internals](modules.md), which owns generic registration
|
||||||
|
and extension mechanics, and [Configuration](../config.md), which owns the
|
||||||
|
selectable keys, bindings, reference syntax, and default validator chains.
|
||||||
|
|
||||||
|
## Durable Artifact Contracts
|
||||||
|
|
||||||
|
The six lanes have separate durable wire contracts. This guide deliberately
|
||||||
|
does not repeat their JSON shapes or schemas.
|
||||||
|
|
||||||
|
| Lane | Durable contract |
|
||||||
|
| --- | --- |
|
||||||
|
| Spells | [spell artifacts](../integrations/dnd-spell-artifacts.md) |
|
||||||
|
| NPCs | [NPC artifacts](../integrations/dnd-npc-artifacts.md) |
|
||||||
|
| Combat turns | [combat-turn artifacts](../integrations/dnd-combat-turn-artifacts.md) |
|
||||||
|
| Item events | [item-event artifacts](../integrations/dnd-item-event-artifacts.md) |
|
||||||
|
| NPC interactions | [NPC-interaction artifacts](../integrations/dnd-npc-interaction-artifacts.md) |
|
||||||
|
| Scene descriptions | [scene-description artifacts](../integrations/dnd-scene-description-artifacts.md) |
|
||||||
|
|
||||||
|
## Family Composition
|
||||||
|
|
||||||
|
The D&D registrar registers the family’s artifact codecs, extractors, typed
|
||||||
|
append-order mergers, normalizers, validators, prompt assets, and default
|
||||||
|
validator chains. Each extractor and normalizer has a stable module spec,
|
||||||
|
strict option decoding, and a typed builder. Configuration remains the
|
||||||
|
canonical owner of the exact keys and validator order.
|
||||||
|
|
||||||
|
Private structured-LLM response schemas are deliberately minimal. They reject
|
||||||
|
invalid JSON structure, missing required fields, incompatible types, and
|
||||||
|
unknown fields, while preserving semantic candidates for deterministic
|
||||||
|
validation. Do not promote a private response envelope into a durable schema;
|
||||||
|
the contracts above define durable data.
|
||||||
|
|
||||||
|
## Prompt Construction
|
||||||
|
|
||||||
|
D&D extractors assemble prompts from an ordered manifest of shared and
|
||||||
|
module-owned assets. Reuse the shared D&D system, evidence, identity,
|
||||||
|
reference, and transcript assets instead of copying their text into individual
|
||||||
|
modules. A manifest’s declared sequence, including cache-control placement, is
|
||||||
|
part of the prompt behavior, and the chunk transcript is the final message.
|
||||||
|
Preserve that order when changing an extractor or its assets so prompt-cache
|
||||||
|
behavior remains stable.
|
||||||
|
|
||||||
|
All extractors use the shared prompt-input preparation rules. The current chunk
|
||||||
|
is copied into transcript material; player, party, glossary, and compatible
|
||||||
|
campaign references are context for disambiguation, not source evidence.
|
||||||
|
Reference prompt material is canonically ordered before it is rendered, which
|
||||||
|
keeps equivalent inputs stable across runs.
|
||||||
|
|
||||||
|
## Evidence, Candidates, And Normalization
|
||||||
|
|
||||||
|
The current transcript is the only durable evidence source. Extractors assign
|
||||||
|
the current source identity, preserve candidate evidence ranges for validators,
|
||||||
|
and canonically order or remove exact duplicate ranges without asking the
|
||||||
|
model to repair semantic errors. Campaign context and generated artifacts may
|
||||||
|
ground names or control routing, but they never establish evidence for a D&D
|
||||||
|
result.
|
||||||
|
|
||||||
|
Default chains keep responsibilities separate: structural validators assess the
|
||||||
|
candidate, source-reference validators resolve cited ranges against the current
|
||||||
|
source, durable-schema validation checks an approved representation, and
|
||||||
|
relatedness validators report advisory evidence concerns. The configured order
|
||||||
|
is documented in
|
||||||
|
[Configuration](../config.md#production-validator-keys-and-default-chains).
|
||||||
|
|
||||||
|
Normalizers are deterministic for spells, combat turns, item events, NPC
|
||||||
|
interactions, and scene descriptions. They canonicalize display values and
|
||||||
|
evidence, use source-document order for stable output, and issue bounded
|
||||||
|
warnings for changes or collapsed duplicates. The NPC normalizer is the
|
||||||
|
intentional exception: it first produces a deterministic candidate set, then
|
||||||
|
uses a bounded structured-LLM proposal to reconcile identity groups. Invalid
|
||||||
|
or unusable proposals retain the deterministic result and surface retry or
|
||||||
|
fallback diagnostics; the model does not directly replace durable records.
|
||||||
|
|
||||||
|
## Generated References And Grounding
|
||||||
|
|
||||||
|
Normalized D&D artifacts can be handed to a later step through a generated
|
||||||
|
reference binding. The framework verifies artifact compatibility and retains
|
||||||
|
producer provenance; consumers resolve the handed-off artifact into an
|
||||||
|
immutable, validated projection for each operation. External files are checked
|
||||||
|
during preparation, while generated artifacts are resolved at the handoff.
|
||||||
|
|
||||||
|
NPC registries are names-only grounding projections: they may canonicalize
|
||||||
|
actors for spells and combat turns and are required for NPC interactions, but
|
||||||
|
they do not supply evidence. Scene-description registries are eligibility-only
|
||||||
|
projections: they retain the current chunk’s classification data, not scene
|
||||||
|
prose or evidence, and exist to route combat extraction.
|
||||||
|
|
||||||
|
## Lane-Specific Rules
|
||||||
|
|
||||||
|
The following differences are intentional and should remain explicit when a
|
||||||
|
shared helper changes.
|
||||||
|
|
||||||
|
| Lane | Intentional behavior |
|
||||||
|
| --- | --- |
|
||||||
|
| Spells | May use a spell-catalog overlay and optional NPC grounding; the catalog validator supplies domain-specific semantic checks. |
|
||||||
|
| NPCs | Does not consume an NPC registry. Its normalizer is the LLM-assisted reconciliation exception described above. |
|
||||||
|
| Combat turns | Requires a scene-description artifact. It calls the LLM only for an exact `combat` classification; exact non-combat classifications return an accepted empty result, while missing or mismatched classifications return an empty result with a bounded warning. Optional NPC grounding never becomes evidence. |
|
||||||
|
| Item events | Uses campaign context for disambiguation but has no NPC-registry or scene-description dependency. |
|
||||||
|
| NPC interactions | Requires the normalized NPC registry at extraction and normalization, using it for canonical actor grounding only. |
|
||||||
|
| Scene descriptions | Produces the classifications consumed by combat routing; it does not consume an NPC registry or provide evidence for combat artifacts. |
|
||||||
|
|
||||||
|
The combat and scene-description contracts describe their exact handoff and
|
||||||
|
empty-result behavior in more detail:
|
||||||
|
[combat turns](../integrations/dnd-combat-turn-artifacts.md) and
|
||||||
|
[scene descriptions](../integrations/dnd-scene-description-artifacts.md).
|
||||||
|
|
||||||
|
## Focused Verification
|
||||||
|
|
||||||
|
When changing D&D behavior, test the affected codec, extractor, normalizer,
|
||||||
|
validator, prompt-asset manifest, and registry projection. Also test generated
|
||||||
|
handoffs at the integration boundary and run the full D&D module suite:
|
||||||
|
|
||||||
|
~~~sh
|
||||||
|
go test ./internal/modules/dnd/...
|
||||||
|
go test ./internal/modules/integration/...
|
||||||
|
~~~
|
||||||
@@ -1,252 +1,149 @@
|
|||||||
# LLM Runtime Internals
|
# LLM Runtime Internals
|
||||||
|
|
||||||
`internal/framework/llm` implements Notarius's transport boundary for structured
|
`internal/framework/llm` is Notarius’s provider-independent structured
|
||||||
completion. It contains the Scriptorium adapter, concurrency scheduler,
|
completion boundary. It adapts framework requests to Scriptorium, bounds
|
||||||
prompt/schema registries, selected-profile recording, and provider-error
|
provider calls, assembles registered prompt and schema assets, records selected
|
||||||
redaction.
|
profiles, and redacts provider errors. The architectural boundary is defined in
|
||||||
|
[Architecture](../policy/architecture.md#llm-boundary); profile sources,
|
||||||
|
credentials, and concurrency settings belong in
|
||||||
|
[Configuration](../config.md#scriptorium-profiles) and
|
||||||
|
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||||||
|
|
||||||
Provider-neutral ownership rules are defined in
|
## Structured Completion Boundary
|
||||||
[Architecture](../policy/architecture.md#llm-boundary). Profile sources,
|
|
||||||
credentials, and concurrency settings are defined in
|
|
||||||
[Configuration](../config.md).
|
|
||||||
|
|
||||||
## Structured Contract
|
Modules and LLM-backed validators depend only on
|
||||||
|
`contracts.StructuredLLMClient`. A completion request supplies a prompt ID and
|
||||||
|
version, optional profile and session IDs, named input material, variables, and
|
||||||
|
a caller-owned decode target. The successful response returns the validated raw
|
||||||
|
structured bytes together with non-secret provider, model, profile, and token
|
||||||
|
metadata.
|
||||||
|
|
||||||
Modules and LLM-backed validators depend on
|
The caller owns the domain behavior: it chooses the prompt, prepares inputs,
|
||||||
`contracts.StructuredLLMClient.CompleteStructured`. A request identifies a
|
selects the private response schema, and interprets the decoded result. The
|
||||||
prompt and optional profile/session, supplies named input materials and
|
adapter does not own source evidence, artifact conversion, normalization, or
|
||||||
variables, and provides a caller-owned decoding target. A successful response
|
durable schemas. Those responsibilities remain with the module and its
|
||||||
contains the validated raw structured bytes plus non-secret provider, model,
|
[integration contract](../integrations/).
|
||||||
profile, and token metadata.
|
|
||||||
|
|
||||||
The caller owns prompt selection, response-schema selection, and interpretation
|
`ScriptoriumClient` validates the request target and prompt identity, maps each
|
||||||
of the decoded result. `LLMInputMaterial` keeps source and reference bytes with
|
named material to a Scriptorium inline artifact while preserving its origin URI,
|
||||||
their origin metadata so the adapter can pass named artifacts to Scriptorium
|
forwards session and profile selection, then prepares and runs the prompt. It
|
||||||
without exposing Scriptorium types through stage contracts.
|
returns Scriptorium’s validated raw bytes rather than re-encoding the decoded
|
||||||
|
target. An empty optional material is represented as one space so its named
|
||||||
|
input is retained by Scriptorium.
|
||||||
|
|
||||||
## Production Construction
|
An empty request profile lets the prompt select its configured default. The CLI
|
||||||
|
prepares every explicitly selected binding profile before a run begins, so a
|
||||||
|
missing explicit profile fails before stage execution. Calls record the profile
|
||||||
|
actually selected by Scriptorium; the recorder deduplicates non-secret profile
|
||||||
|
identity, provider, and model values for manifest use.
|
||||||
|
|
||||||
`internal/cli` constructs the production runtime by:
|
## Shared Provider-Call Limit
|
||||||
|
|
||||||
1. allocating the asset registry populated by the generic, Seriatim, and D&D
|
Production construction creates one Scriptorium client and wraps it in one
|
||||||
package-family registrars;
|
scheduled client. The scheduler has a fixed, positive permit limit, serves
|
||||||
2. creating a `ScriptoriumClient` from the effective profile source;
|
queued calls in FIFO order, and removes a queued call when its context is
|
||||||
3. attaching an `LLMProfileRecorder`;
|
cancelled. A granted permit is released exactly once on every completion path.
|
||||||
4. creating a scheduler from the effective concurrency limit;
|
|
||||||
5. returning a `ScheduledClient` wrapper;
|
|
||||||
6. decorating that shared client before preparation when debug recording is
|
|
||||||
enabled; and
|
|
||||||
7. injecting that one shared client into complete pipeline preparation before
|
|
||||||
the source file is read or the runner is invoked.
|
|
||||||
|
|
||||||
The D&D scene chunker and spell, NPC, combat-turn, item-event, NPC-interaction, and
|
The scheduled wrapper surrounds every `CompleteStructured` call, so concurrent
|
||||||
scene-description extractors retain this
|
lanes, pipeline retries, and LLM-backed validators share the same provider-call
|
||||||
injected client and use it for every structured completion. Operation requests
|
ceiling. This ceiling is independent of pipeline worker concurrency; changing
|
||||||
do not carry an LLM client.
|
worker counts cannot exceed the configured LLM limit. The configuration field
|
||||||
|
and its effective default are owned by
|
||||||
The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
|
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||||||
and validator bindings. It prepares a small internal check prompt for each ID so
|
|
||||||
missing or invalid profiles fail before pipeline execution. The runtime profile
|
|
||||||
override syntax and scope are defined in the
|
|
||||||
[CLI reference](../cli.md#run); binding rules are defined in
|
|
||||||
[Configuration](../config.md#module-bindings).
|
|
||||||
|
|
||||||
## Scriptorium Adapter
|
|
||||||
|
|
||||||
`ScriptoriumClient` converts a Notarius request into a Scriptorium `RunRequest`.
|
|
||||||
It validates the decoding target and prompt identity, maps named input materials
|
|
||||||
to inline artifacts, forwards explicit profile and session context, delegates
|
|
||||||
rendering/provider execution/structured validation, and unmarshals successful
|
|
||||||
JSON into the caller target.
|
|
||||||
|
|
||||||
Empty optional input material is represented by a single space so Scriptorium
|
|
||||||
retains the named input. The client returns Scriptorium's validated structured
|
|
||||||
bytes rather than re-encoding the caller target, allowing modules to preserve
|
|
||||||
the runtime result exactly.
|
|
||||||
|
|
||||||
Selected profile, provider, model, and token metadata are mapped into the
|
|
||||||
Notarius response. The recorder deduplicates profiles by identity and supplies
|
|
||||||
manifest-safe profile summaries after actual calls; manifest population does
|
|
||||||
not guess the selected prompt default in advance.
|
|
||||||
|
|
||||||
Generated-output validation failures and provider failures are wrapped with
|
|
||||||
prompt context. Error strings pass through bearer-token redaction before they
|
|
||||||
cross the runtime boundary.
|
|
||||||
|
|
||||||
## Scheduling
|
|
||||||
|
|
||||||
`Scheduler` uses a bounded permit count and a FIFO waiter queue. Immediate
|
|
||||||
acquisition increments the in-flight count; queued acquisition waits for a
|
|
||||||
permit or context cancellation. Cancellation removes a queued waiter, while a
|
|
||||||
cancelled waiter that has already received a permit releases it.
|
|
||||||
|
|
||||||
`ScheduledClient` acquires a permit around each structured completion and
|
|
||||||
defers release on every result path. The effective limit and default are
|
|
||||||
configuration facts in [Configuration](../config.md#defaults).
|
|
||||||
|
|
||||||
This provider-call ceiling is independent of the pipeline's extract worker
|
|
||||||
limit. Concurrent lanes, retries, and validators all use the same scheduled
|
|
||||||
client, so increasing framework workers cannot exceed `total_llm`. Pipeline
|
|
||||||
dispatch and cancellation mechanics are documented in
|
|
||||||
[Pipeline Internals](pipeline.md#execution-flow).
|
|
||||||
|
|
||||||
## Prompt And Schema Assets
|
## Prompt And Schema Assets
|
||||||
|
|
||||||
`AssetRegistry` combines caller-owned prompt filesystems under stable prefixes
|
An `AssetRegistry` collects prompt and schema filesystems from production module
|
||||||
and rejects invalid or conflicting registrations. Production module packages
|
families. It flattens registered roots into the Scriptorium filesystems and
|
||||||
register their own prompt and schema assets; generic framework code contains no
|
rejects invalid roots, unreadable assets, duplicate paths, and missing prompt
|
||||||
D&D prompt content. `internal/framework/promptfs` provides the domain-neutral
|
or schema files during preparation. The framework’s `promptfs` helper combines
|
||||||
filesystem composition helper used to combine module-owned files with shared
|
module-owned prompt files with reusable domain fragments without making the
|
||||||
domain prompt fragments.
|
framework depend on D&D content.
|
||||||
|
|
||||||
The D&D scene chunker and spell, NPC, combat-turn, item-event, NPC-interaction, and
|
Each LLM-backed module owns its prompt declaration, package-specific assets,
|
||||||
scene-description extractors each declare an
|
and private response schema. Shared D&D wording is owned by the D&D shared
|
||||||
ordered prompt asset manifest. The manifest lists the package-owned YAML and
|
asset package; the detailed D&D conventions are in
|
||||||
Markdown files, then the exact shared fragments rendered by that prompt; the
|
[D&D Module Internals](dnd.md). The mounted prompt assets used by a module also
|
||||||
same ordered list drives both filesystem mounting and the prompt fingerprint.
|
determine its prompt fingerprint. Schema loaders validate JSON, attach identity
|
||||||
Unused shared assets are neither mounted nor fingerprinted. Universal
|
and digest metadata, make defensive copies, and expose diagnostics without raw
|
||||||
extraction-evidence and output policy lives only in the shared extraction
|
schema bytes.
|
||||||
assets; package-owned prompt files retain artifact-specific rules. The scene
|
|
||||||
prompt keeps its separate output rule because it does not render the
|
|
||||||
extraction-evidence asset.
|
|
||||||
|
|
||||||
### D&D Extraction Prompt Ordering And Cache Boundaries
|
Private response schemas validate a model transport envelope. They are not the
|
||||||
|
durable artifact schema and should not be documented as an external wire
|
||||||
|
contract. Durable formats and compatibility rules remain in the
|
||||||
|
[integration contracts](../integrations/).
|
||||||
|
|
||||||
D&D extraction prompts order messages from the most reusable content to the
|
## Prompt Maintenance And Backend Caching
|
||||||
most variable content. New extraction lanes use these tiers in order:
|
|
||||||
|
|
||||||
1. universal shared content, including the system, extraction-evidence, and
|
Prompt message order and shared asset bytes are runtime behavior. Backend cache
|
||||||
in-world identity messages;
|
reuse depends on the same preceding messages and content, not merely equivalent
|
||||||
2. stable campaign or run context shared across lanes, including campaign
|
meaning. Keep reusable shared assets byte-identical and keep stable material
|
||||||
references;
|
before the inputs that vary per request wherever a prompt’s declared sequence
|
||||||
3. stable subset- and lane-specific context and instructions, including an NPC
|
supports caching. Preserve the existing manifest order and cache-control hints
|
||||||
registry, catalog, task, or extraction instructions when applicable;
|
when editing a prompt.
|
||||||
4. the chunk transcript as the final user message.
|
|
||||||
|
|
||||||
This ordering lets requests reuse the longest identical prefix before the
|
D&D extraction manifests place the changing chunk transcript at the end of the
|
||||||
per-chunk transcript changes. Cache reuse requires the preceding message
|
prompt after their reusable context. Scene chunking and NPC normalization use
|
||||||
sequence and content to be exactly identical; semantic similarity is not
|
their own declared message sequences because their inputs and work differ. The
|
||||||
sufficient. Cache boundaries belong at the ends of reusable stable tiers,
|
family-specific asset and ordering rules belong in [D&D Module Internals](dnd.md).
|
||||||
subject to the provider's cache-boundary limit. The shared identity and
|
Do not add tests that enforce a fixed message-prefix length; prompt-asset tests
|
||||||
campaign-reference messages form the first two extraction boundaries. Spell,
|
should instead verify the meaningful asset sequence, inputs, and cache controls
|
||||||
combat, and interaction prompts add a boundary at the shared NPC registry. Each extraction
|
of the prompt being changed.
|
||||||
prompt places its final boundary on its lane-specific instructions, immediately
|
|
||||||
before the transcript. The transcript does not carry cache control because no
|
|
||||||
reusable content follows it.
|
|
||||||
|
|
||||||
Accordingly, the common prefix of the spell, NPC, combat, item-event, and interaction
|
## Validation, Repair, And Retries
|
||||||
extraction prompts is system,
|
|
||||||
extraction evidence, identity, and campaign references. The NPC prompt then
|
|
||||||
renders task, instructions, and transcript. Spell renders the NPC registry,
|
|
||||||
catalog, task, instructions, and transcript. Combat renders the NPC registry,
|
|
||||||
task, instructions, and transcript. Item-event renders task, instructions, and
|
|
||||||
transcript without a generated-artifact input. NPC interaction renders the names-only NPC
|
|
||||||
registry, task, instructions, and transcript. The
|
|
||||||
scene chunker is not an extraction lane: it retains its separate system,
|
|
||||||
transcript, campaign-reference, task, and instruction order and marks its
|
|
||||||
transcript and campaign-reference messages ephemeral.
|
|
||||||
|
|
||||||
The scene-description extractor deliberately omits the citation-oriented
|
Scriptorium performs prompt rendering, provider execution, and the prompt’s
|
||||||
`common-dnd-extraction-evidence.md` asset because Notarius attaches the whole
|
structured-output validation. The adapter reports an empty result, validation
|
||||||
accepted chunk range itself. Its manifest is system, shared identity, shared
|
failure, empty structured body, or decode failure as
|
||||||
campaign references, lane task, lane instructions, then the transcript. The
|
`ErrInvalidStructuredOutput`, while retaining the returned raw bytes and debug
|
||||||
identity, campaign-reference, and instruction messages are ephemeral cache
|
material when they exist. Provider failures remain operational errors rather
|
||||||
boundaries; the transcript is last and has no cache control. Compatible shared
|
than output-validation failures.
|
||||||
messages remain canonical shared assets rather than copied package text.
|
|
||||||
|
|
||||||
### D&D NPC Normalization Prompt Ordering And Cache Boundaries
|
Prompt-declared repair is executed within Scriptorium’s structured-output flow.
|
||||||
|
The current production D&D prompt manifests set repair attempts to zero. That
|
||||||
|
setting does not replace pipeline retry behavior: a binding’s configured retry
|
||||||
|
count reruns its stage attempt after an error or rejection, and an exhausted
|
||||||
|
rejection is a recorded output rather than a provider error. The pipeline owns
|
||||||
|
attempt lifecycle, validation chains, and retry diagnostics; see
|
||||||
|
[Pipeline Internals](pipeline.md#validation-retries-and-output) and the
|
||||||
|
[binding reference](../config.md#module-bindings-and-validators).
|
||||||
|
|
||||||
NPC normalization has a distinct prompt and response-schema identity from NPC
|
## Observability And Redaction
|
||||||
extraction. Its stable message tiers are the common D&D system asset, followed
|
|
||||||
by package-owned task and normalization instructions. Cache boundaries follow
|
|
||||||
the shared system tier and the package instructions. The variable tail contains
|
|
||||||
the private candidate-name-and-range input and a windowed transcript input
|
|
||||||
whose cited units provide local context; neither has a cache boundary because
|
|
||||||
it changes with the document.
|
|
||||||
|
|
||||||
This prompt intentionally omits shared identity guidance,
|
When debug recording is enabled, the pipeline decorates the shared client. The
|
||||||
extraction-evidence, and campaign-reference assets: it reconciles existing
|
wrapper records prepared prompt and response material, timing, selected profile
|
||||||
records rather than extracting events or adding evidence. Its package-owned
|
and model, and call identifiers in the run’s debug bundle, including material
|
||||||
manifest and schema identity are fingerprinted separately, so a normalization
|
available from a failed structured completion. For a successful completion, a
|
||||||
prompt or schema change cannot reuse a prior normalization checkpoint.
|
debug-write failure is surfaced; when the completion already failed, its call
|
||||||
|
error remains the result. Debug-bundle location, retention, and handling are
|
||||||
|
operational concerns documented in [Operations](../operations.md#debug-bundles).
|
||||||
|
|
||||||
Shared wording belongs in the canonical assets under
|
Run manifests receive selected profile summaries and component identities, not
|
||||||
`internal/modules/dnd/shared`; extraction packages reference those assets in
|
prompt, schema, source, reference, or response content. Provider error text is
|
||||||
their manifests instead of copying similar text into package-local files.
|
wrapped with prompt context and bearer credentials are redacted before it
|
||||||
Package-local assets contain only lane-specific content. An extraction lane may
|
crosses the runtime boundary. Known-secret redaction is available to other
|
||||||
depart from the tier order only when prompt-quality evidence or a provider
|
runtime collaborators; it does not make prompt or response contents safe for
|
||||||
constraint makes the exception necessary; document the exception and rationale
|
general logging.
|
||||||
here when it becomes implemented behavior.
|
|
||||||
|
|
||||||
Schema helpers load embedded JSON Schema with identity and digest metadata,
|
## Failure Boundaries
|
||||||
return defensive copies, and expose a diagnostics map that omits schema bytes.
|
|
||||||
The small framework registry contains only generic test schemas; production
|
|
||||||
schemas remain package-owned.
|
|
||||||
|
|
||||||
The spell, NPC, combat, item-event, NPC-interaction, and scene-description extractors'
|
- Construction fails for missing asset registries, mutually exclusive profile
|
||||||
package-owned prompts declare their
|
sources, invalid asset registration, or a non-positive scheduler limit.
|
||||||
structured JSON inputs and private response schemas. Each private response
|
- Preparation failures, unavailable explicit profiles, provider failures, and
|
||||||
schema remains separate from its durable artifact codec schema; this work does
|
context cancellation propagate to the calling stage with context.
|
||||||
not use shared schema fragments or schema generation. Those private schemas own
|
- Malformed or schema-invalid provider output is classified separately as
|
||||||
the transport envelope—required fields, JSON types, nullability, and
|
invalid structured output so the module or pipeline can apply its own retry
|
||||||
unknown-field rejection—while deterministic validators own semantic constraints
|
and rejection policy.
|
||||||
such as enum membership, non-empty values and collections, and positive
|
- Domain semantic checks, evidence decisions, and deterministic normalization
|
||||||
numbers. The spell extractor's prompt declares a required
|
run outside the provider adapter.
|
||||||
`application/json` `spell_catalog` input and an optional `application/json`
|
|
||||||
`npcs` input. The extractor generates
|
|
||||||
the catalog input from its prepared
|
|
||||||
effective catalog as `{"spell_names":[...]}` using sorted canonical names only.
|
|
||||||
The shared D&D prompt assets include a generic NPC grounding fragment directly
|
|
||||||
after the campaign reference message for spell, combat, and interaction prompts. When an NPC
|
|
||||||
registry is bound, the
|
|
||||||
domain registry boundary strictly decodes and identity-validates one durable
|
|
||||||
artifact, re-encodes canonical JSON for provenance, and separately generates a
|
|
||||||
names-only prompt projection. The unbound projection is exactly `{"npcs":[]}`.
|
|
||||||
Prompt input and component-local checkpoint digests cover the projected bytes;
|
|
||||||
manifests retain the optional full registry digest/count rather than names,
|
|
||||||
overlay bytes, registry paths, or source metadata. Combat and interaction prompt,
|
|
||||||
response-schema, mapping, normalization, identity, and registry-projection
|
|
||||||
fingerprints remain separate semantic inputs to checkpoint identity.
|
|
||||||
|
|
||||||
## Debug And Redaction Boundaries
|
## Focused Verification
|
||||||
|
|
||||||
The pipeline may wrap the client with a debug recorder that captures prepared
|
Read the LLM adapter, scheduler, asset registry, schema loader, and redaction
|
||||||
prompt/response material for an explicitly requested debug run. Debug summaries
|
tests when changing this boundary. Prompt changes also require the owning
|
||||||
and manifests receive identities, hashes, usage, and selected profile summaries
|
module’s asset tests, and retry or debug changes require focused pipeline or
|
||||||
rather than prompt, source, reference, schema, or response content.
|
CLI coverage. The focused runtime and D&D checks are:
|
||||||
|
|
||||||
The Scriptorium error wrapper removes bearer credential values from surfaced
|
~~~sh
|
||||||
provider errors; `RedactSecrets` and `ErrorWithSecretsRedacted` support known
|
go test ./internal/framework/llm/... ./internal/modules/dnd/...
|
||||||
secret values elsewhere in the runtime. Config summaries use a separate
|
~~~
|
||||||
clone-and-redact path in `internal/core/config`. These mechanisms implement the
|
|
||||||
security invariant in
|
|
||||||
[Architecture](../policy/architecture.md#state-output-and-safety); operator
|
|
||||||
handling of debug data is defined in [Operations](../operations.md#debug).
|
|
||||||
|
|
||||||
## Failure Behavior
|
|
||||||
|
|
||||||
- Invalid targets, missing prompt IDs, malformed structured output, and
|
|
||||||
Scriptorium failures return contextual errors to the calling module.
|
|
||||||
- Scheduler construction rejects non-positive limits; acquisition respects
|
|
||||||
context cancellation.
|
|
||||||
- Asset registration rejects invalid roots, missing content, and path conflicts.
|
|
||||||
- Schema loading distinguishes missing assets, invalid JSON, and invalid
|
|
||||||
metadata.
|
|
||||||
- Profile validation errors occur during CLI preparation when an explicit
|
|
||||||
selected ID cannot be prepared.
|
|
||||||
|
|
||||||
## Tests To Inspect
|
|
||||||
|
|
||||||
- `internal/framework/llm/scriptorium_client_test.go`: adapter mapping and local
|
|
||||||
HTTP integration.
|
|
||||||
- `internal/framework/llm/scheduler_test.go` and
|
|
||||||
`scheduled_client_test.go`: permits, FIFO behavior, cancellation, and wrapper
|
|
||||||
release.
|
|
||||||
- `internal/framework/llm/asset_registry_test.go` and
|
|
||||||
`schema_registry_test.go`: asset composition, validation, and defensive
|
|
||||||
copies.
|
|
||||||
- `internal/framework/llm/secrets_test.go`: provider-error redaction.
|
|
||||||
- `internal/cli/run_contract_test.go`: profile validation, production client
|
|
||||||
wiring, manifest recording, and debug integration.
|
|
||||||
- Module-local `scriptorium_assets_test.go` files: prompt inputs and package
|
|
||||||
asset registration.
|
|
||||||
|
|||||||
@@ -1,728 +1,99 @@
|
|||||||
# Module And Validator Internals
|
# Module Internals
|
||||||
|
|
||||||
Production module and validator implementations live under their domain-first
|
This guide owns the mechanics for implementing and registering production
|
||||||
trees in `internal/modules`.
|
modules. [Configuration](../config.md) owns selectable keys, binding syntax,
|
||||||
The selectable keys, configuration options, reference slots, and default
|
reference configuration, and default validator chains. Durable input and output
|
||||||
validator chain are canonical in the
|
shapes belong in [integration contracts](../integrations/).
|
||||||
[module](../config.md#implemented-production-modules) and
|
|
||||||
[validator](../config.md#implemented-production-validators) catalogs in
|
The D&D family has additional shared conventions and domain-specific
|
||||||
Configuration.
|
exceptions. See [D&D Module Internals](dnd.md) rather than adding them here.
|
||||||
|
|
||||||
## Extension Pattern
|
## Module Boundary
|
||||||
|
|
||||||
A stage module package provides a stable key, constructor, contract
|
A module is a typed implementation registered for one pipeline stage. Its
|
||||||
implementation, `ModuleSpec`, `Register`, and focused behavior and registration
|
`ModuleSpec` is the public-to-the-framework declaration of its stable key,
|
||||||
tests. A validator package follows the same pattern with `ValidatorSpec` and the
|
stage, required and provided capabilities, artifact kind, and accepted
|
||||||
validator registry. Package-family registrars compose those leaf registrations
|
reference slots. The framework uses that declaration to resolve a configured
|
||||||
into the production catalog and own family-level policy such as default
|
binding before it builds the implementation.
|
||||||
validator chains and prompt asset collection.
|
|
||||||
|
Implementations that accept options must provide both an option validator and
|
||||||
Production input, chunk, output, and D&D spell-, NPC-, combat-, item-event-, interaction-, and scene-description-extract packages
|
a builder. The validator is used while resolving configuration; the builder
|
||||||
register strict option decoders and run-local builders. Preparation decodes their options into
|
decodes the same options and constructs the implementation from the prepared
|
||||||
implementation-owned values and injects dependencies plus the materialized
|
`BuildRequest`. Reject unknown options in both paths. A builder receives only
|
||||||
reference set for the selected target. Each builder receives an isolated clone
|
the dependencies and materialized references that the framework prepared for
|
||||||
of that set; input and output builders receive no references. The spell, NPC,
|
that operation, so it must not re-read configuration or files.
|
||||||
combat, item-event, interaction, and scene-description extractors are typed over the canonical D&D model. D&D validators, merge,
|
|
||||||
and normalize use typed variants; JSON representation validators use serialized
|
Registry helpers register the typed builder for a stage-specific registry.
|
||||||
requests; and unconditional validators expose separate chunk and typed
|
They are preferable to hand-written untyped registration because they retain
|
||||||
variants. The D&D production registrar registers the canonical typed spell,
|
the artifact type at the framework boundary. Registrars validate the registries
|
||||||
NPC, combat, item-event, interaction, and scene-description implementations, including their kind-specific merge and
|
they need, register each leaf implementation, and add any family-owned assets
|
||||||
normalize behavior.
|
or default validator chains. They return contextual errors so production
|
||||||
|
composition fails at startup rather than at the first run.
|
||||||
For D&D artifact defaults, generic JSON syntax validation runs first. Rejecting
|
|
||||||
domain validators then own semantic diagnostics before generic JSON Schema
|
## Production Composition
|
||||||
validation provides the final rejecting representation backstop; warning-only
|
|
||||||
relatedness validators run last. This default composition does not reorder an
|
Production composition is intentionally split by family:
|
||||||
explicitly configured validator chain.
|
|
||||||
|
- The generic registrar provides the unit chunker, generic JSON validators,
|
||||||
Prepared extractors, extract validators, and codecs may be reused concurrently
|
and JSON output encoder.
|
||||||
by the run-wide extract pool. Production implementations are immutable after
|
- The Seriatim registrar provides the transcript input adapter. Its external
|
||||||
construction: they retain only typed options, immutable assets, or the shared
|
input behavior is defined by the [Seriatim contract](../integrations/seriatim.md).
|
||||||
concurrency-safe LLM client. Implementations that introduce mutable state must
|
- The D&D registrar provides its codecs, extractors, mergers, normalizers,
|
||||||
synchronize that state without creating a separate provider scheduler.
|
validators, prompt assets, and default chains. Its behavioral conventions
|
||||||
|
are documented in [D&D Module Internals](dnd.md).
|
||||||
Specs expose capability and execution metadata without constructing an
|
|
||||||
implementation. Registry entries separately expose option validation and
|
The CLI owns the composition that invokes these registrars. A module package
|
||||||
run-local construction. Chunk, extract, merge, and normalize modules that accept
|
may register its own family but must not assemble the CLI or make framework
|
||||||
auxiliary material declare identical reference slots from both
|
packages depend on production extensions.
|
||||||
`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
|
|
||||||
that agreement. Runtime delivery uses the corresponding stage request's
|
## Adding Or Changing A Module
|
||||||
`References` field.
|
|
||||||
|
1. Choose the pipeline stage and the typed artifact boundary. Put external
|
||||||
LLM-backed extensions own their prompt definitions and response schemas under
|
input or durable artifact formats in the relevant integration contract,
|
||||||
package-local embedded assets. Shared filesystem composition belongs in
|
not in this guide or in a private LLM response type.
|
||||||
`internal/framework/promptfs`; reusable D&D prompt fragments, reference
|
2. Define a stable `ModuleSpec` with the exact capabilities and reference
|
||||||
declarations, prompt-input assembly, and source-unit/citation helpers belong in
|
slots needed for the operation. Model a producer/consumer handoff as an
|
||||||
`internal/modules/dnd/shared`, which owns operation-scoped indexed
|
artifact-compatible slot; configuration then chooses an external file or a
|
||||||
source-reference validation, citation traversal, ordering and canonicalization,
|
generated binding.
|
||||||
plus bounded D&D diagnostics. The
|
3. Implement strict option decoding, construction, and the typed stage
|
||||||
D&D scene chunker and spell, NPC, combat-turn, item-event, NPC-interaction, and scene-description extractors use ordered
|
interface. Preserve caller ownership: do not retain mutable request data
|
||||||
package-local prompt manifests for both rendering and prompt fingerprinting, so
|
and return defensive copies where an implementation exposes stored data.
|
||||||
only the shared fragments each prompt actually renders participate in either
|
4. Register the module through its typed registry helper and add it to the
|
||||||
operation. Extraction prompts place stable shared and lane-specific context
|
owning family registrar. Add a default validator chain only when that
|
||||||
before the variable transcript and use shared assets for wording common across
|
family owns the behavior; otherwise require an explicit compatible chain.
|
||||||
lanes. The canonical ordering and cache-boundary policy is documented in
|
5. Update the selectable-key and chain reference in
|
||||||
[LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries). Stage
|
[Configuration](../config.md#production-module-keys), the applicable
|
||||||
contracts expose only Notarius structured-completion types, not Scriptorium
|
integration contract, and focused tests. Keep the configuration document
|
||||||
public types.
|
as the sole list of production keys and validator order.
|
||||||
|
|
||||||
The shared `PrepareChunkExtraction` helper owns common extraction preflight and
|
## Validation And References
|
||||||
transcript material preparation for the spell, NPC, combat-turn, item-event,
|
|
||||||
NPC-interaction, and scene-description extractors. It validates common request
|
Validators operate on the value produced at their configured stage. A default
|
||||||
state, clones supplied source metadata, falls back to the materialized chunk
|
chain is ordered behavior, not a set: JSON parsing, structural checks,
|
||||||
when content is absent, checks that content remains chunk-identical, and fills
|
domain-specific checks, durable-schema checks, and advisory checks may have
|
||||||
only the common default fields. Extractors retain receiver, dependency, and
|
different responsibilities and failure handling. The active default chains and
|
||||||
lane-specific checks locally and wrap helper errors with their module context.
|
override rules are maintained in
|
||||||
|
[Configuration](../config.md#production-validator-keys-and-default-chains).
|
||||||
Reference material may inform a module or prompt but must not become source
|
|
||||||
evidence. The resolver and materializer behavior is described in
|
Reference slots are part of the module specification. They describe the
|
||||||
[Pipeline Internals](pipeline.md#reference-materialization).
|
accepted artifact kind, media type, size, and whether a binding is required;
|
||||||
|
the framework validates those constraints before construction. An external
|
||||||
## Domain Reference Data
|
reference is materialized during preparation. A generated reference is a
|
||||||
|
compatible normalized artifact handed from an earlier pipeline step at
|
||||||
### `internal/modules/dnd/spells/catalog`
|
operation time. The configuration reference rules, including precedence and
|
||||||
|
ordered-handoff requirements, are maintained in
|
||||||
The spell catalog package owns the embedded, versioned D&D 5e 2014 SRD spell
|
[Configuration](../config.md#references-and-ordered-handoffs).
|
||||||
reference data. Its strict JSON asset contains one canonical record per spell,
|
|
||||||
including spell level and all applicable class memberships. `LoadSRD5E2014`
|
## Focused Verification
|
||||||
validates catalog identity, provenance metadata, ordering, uniqueness, levels,
|
|
||||||
classes, aliases, and lookup-key collisions before exposing immutable copies.
|
Exercise the leaf implementation and its registration path when changing a
|
||||||
|
module. Registry and registrar tests cover duplicate keys, required registries,
|
||||||
Lookup is case-insensitive and normalizes whitespace and common apostrophe
|
and typed construction; pipeline resolution tests cover capabilities, options,
|
||||||
variants while preserving source punctuation in canonical display names. The
|
and reference compatibility. Domain packages should additionally test their
|
||||||
catalog contains 319 unique spells and 779 class memberships. Source and
|
codecs, validators, normalizers, and any integration handoffs they own.
|
||||||
license details live beside the asset in `SOURCES.md`. This domain-owned data is
|
|
||||||
separate from `internal/modules/dnd/shared`, which is reserved for reusable
|
Run the affected package tests while iterating. The complete module suite is:
|
||||||
prompt and source-reference machinery.
|
|
||||||
|
~~~sh
|
||||||
`ResolveEffectiveCatalog` builds the immutable recognition view used by the
|
go test ./internal/modules/...
|
||||||
spell extractor and catalog validator. It starts with the embedded SRD catalog
|
~~~
|
||||||
and optionally applies one strict JSON overlay from the `spell_catalog` item in
|
|
||||||
a materialized reference set. Overlay catalogs are ordered by ID, may add names
|
|
||||||
and aliases, and may augment an existing canonical spell without replacing its
|
|
||||||
display name. Cross-spell lookup collisions are errors. The effective view
|
|
||||||
exposes sorted canonical names, normalized lookup, overlay identities, and a
|
|
||||||
semantic digest; overlay content remains contextual reference material rather
|
|
||||||
than source evidence. Its external JSON contract is defined in the
|
|
||||||
[spell-catalog overlay contract](../integrations/dnd-spell-catalog-overlays.md).
|
|
||||||
|
|
||||||
### `internal/modules/dnd/npcs/identity`, `internal/modules/dnd/npcs/registry`, and `internal/modules/dnd/codec/npcs`
|
|
||||||
|
|
||||||
The NPC identity package owns Unicode comparison keys, deterministic
|
|
||||||
`npc:sha256:` IDs, display normalization, and whole-registry collision issues.
|
|
||||||
The registry package resolves one optional normalized artifact through the
|
|
||||||
strict codec, validates whole-registry identity, canonicalizes its JSON, and
|
|
||||||
provides immutable records, a names-only prompt projection, distinct durable
|
|
||||||
and projection digests, count, and exact canonical-name lookup. External files cross this boundary during
|
|
||||||
preparation; generated artifacts cross it at the ordered step handoff. It owns
|
|
||||||
the `npcs` slot and its bounded, content-safe validation failures. NPC source
|
|
||||||
references are durable provenance and are not treated as evidence for a
|
|
||||||
consuming pipeline. The NPC codec owns the strict durable `dnd/npc-list` JSON
|
|
||||||
boundary and exposes candidate versus approved encode/decode operations. The
|
|
||||||
shared `internal/modules/dnd/codec/candidatejson` package supplies strict typed
|
|
||||||
candidate JSON mechanics; each artifact codec retains its own durable schema
|
|
||||||
and approved-value policy.
|
|
||||||
|
|
||||||
### `internal/modules/dnd/scenedescriptions/registry`
|
|
||||||
|
|
||||||
The scene-description registry owns the required `scene_descriptions` control
|
|
||||||
reference used by combat extraction. It decodes exactly one approved scene-list
|
|
||||||
artifact through the scene-description codec and retains only scene ID, exact
|
|
||||||
source reference, and kind. Titles, summaries, original bytes, paths, and
|
|
||||||
prompt material do not cross this domain boundary.
|
|
||||||
|
|
||||||
An external reference is validated during preparation; an unbound seed is
|
|
||||||
permitted only while a configured generated reference awaits the ordered
|
|
||||||
handoff. At operation time, a generated artifact overrides the seed and is
|
|
||||||
resolved into an immutable view safe for concurrent extract jobs. Matching is
|
|
||||||
strictly exact by chunk ID, source ID, start unit ID, and end unit ID, producing
|
|
||||||
an exact, missing, or mismatched result. Only an exact result exposes kind.
|
|
||||||
|
|
||||||
The registry's semantic eligibility digest is derived from a sorted projection
|
|
||||||
of ID, exact range, and kind. It ignores titles, summaries, and input order;
|
|
||||||
the unbound view has a stable empty projection digest. Combat extractor
|
|
||||||
metadata and checkpoint identity use this semantic boundary for external
|
|
||||||
references, while generated artifact identity and dependencies remain owned by
|
|
||||||
the framework handoff.
|
|
||||||
|
|
||||||
The `internal/modules/dnd/codec/combatturns` package owns the durable
|
|
||||||
`dnd/combat-turn-list` schema and candidate versus approved JSON boundary. It
|
|
||||||
is registered by the production D&D family registrar for the selectable combat
|
|
||||||
lane.
|
|
||||||
|
|
||||||
The `internal/modules/dnd/codec/itemevents` package owns the durable
|
|
||||||
`dnd/item-event-list` schema and candidate versus approved JSON boundary. It is
|
|
||||||
registered by the production D&D family registrar. Its external contract is
|
|
||||||
defined in the [D&D item-event artifact contract](../integrations/dnd-item-event-artifacts.md).
|
|
||||||
|
|
||||||
The `internal/modules/dnd/codec/npcinteractions` package owns the durable
|
|
||||||
`dnd/npc-interaction-list` schema and candidate versus approved JSON boundary.
|
|
||||||
It is registered by the production D&D family registrar for the selectable
|
|
||||||
interaction lane. Its external contract is documented in the
|
|
||||||
[D&D NPC interaction artifact contract](../integrations/dnd-npc-interaction-artifacts.md).
|
|
||||||
|
|
||||||
The `internal/modules/dnd/codec/scenedescriptions` package owns the durable
|
|
||||||
`dnd/scene-description-list` schema and candidate versus approved JSON boundary.
|
|
||||||
It is registered by the production D&D family registrar. Its external contract
|
|
||||||
is documented in the
|
|
||||||
[D&D scene-description artifact contract](../integrations/dnd-scene-description-artifacts.md).
|
|
||||||
|
|
||||||
## Input Adapter
|
|
||||||
|
|
||||||
### `internal/modules/seriatim/input/transcript`
|
|
||||||
|
|
||||||
The adapter decodes the supported transcript JSON, selects the source identity,
|
|
||||||
computes canonical source provenance, validates segments, and maps each segment
|
|
||||||
into a generic source unit with a self-reference plus speaker and timestamp
|
|
||||||
metadata. It accepts no module options. Its spec advertises the transcript
|
|
||||||
capabilities consumed by D&D modules.
|
|
||||||
|
|
||||||
Parsing is strict about required values and duplicate unit IDs but deliberately
|
|
||||||
ignores unrelated Seriatim fields. The external format and derived-identity
|
|
||||||
rules are defined in the
|
|
||||||
[Seriatim contract](../integrations/seriatim.md).
|
|
||||||
|
|
||||||
## Chunkers
|
|
||||||
|
|
||||||
Chunkers implement `contracts.Chunker.Plan`. A plan identifies ordered source
|
|
||||||
unit ranges and may carry optional namespaced JSON annotations; it does not
|
|
||||||
contain materialized chunk content. The framework canonicalizes annotations,
|
|
||||||
validates ranges against the current source, and materializes chunk IDs,
|
|
||||||
indexes, references, content, units, and generic metadata. Materialized source
|
|
||||||
unit metadata is independently owned. Annotation
|
|
||||||
namespaces remain optional data: generic framework code and downstream modules
|
|
||||||
must not require D&D scene annotations or import `dnd/scenes`.
|
|
||||||
|
|
||||||
### `internal/modules/generic/chunk/units`
|
|
||||||
|
|
||||||
The generic chunker validates the source document and returns ranges over units
|
|
||||||
in configured windows. Overlap changes the next window start but never reorders
|
|
||||||
units. Framework materialization derives the resulting chunk identity and
|
|
||||||
generic metadata from those ranges.
|
|
||||||
|
|
||||||
The accepted options and defaults are defined in
|
|
||||||
[Configuration](../config.md#implemented-production-modules). Generic
|
|
||||||
framework validation canonicalizes the returned unit slices before extraction.
|
|
||||||
The chunker decodes its options during construction and retains only the typed
|
|
||||||
window settings used by `Plan`.
|
|
||||||
|
|
||||||
### `internal/modules/dnd/chunk/scenes`
|
|
||||||
|
|
||||||
The scene chunker prepares a structured Scriptorium request from the full
|
|
||||||
transcript, session, and optional D&D reference inputs. It validates the model's
|
|
||||||
inclusive source-unit endpoints against document position and converts them
|
|
||||||
into deterministic plan ranges. Preparation injects the shared structured LLM
|
|
||||||
client into the chunker; `Plan` supplies only the run-specific profile, session,
|
|
||||||
source, references, and metadata.
|
|
||||||
|
|
||||||
Scene validation requires sequential, contiguous, non-overlapping coverage from
|
|
||||||
the first source unit through the last. Its private response contains only the
|
|
||||||
boundary endpoints; the accepted plan has no D&D-specific annotations and
|
|
||||||
produces no boundary warnings. Malformed structured output is returned as an
|
|
||||||
error; there is no fallback chunker.
|
|
||||||
|
|
||||||
The package embeds its prompt and response schema and reports their non-secret
|
|
||||||
identity and hashes through singleton module metadata. Shared D&D assets supply
|
|
||||||
reference declarations and prompt inputs; their user-facing keys and accepted
|
|
||||||
file types remain canonical in [Configuration](../config.md).
|
|
||||||
|
|
||||||
## Extractor
|
|
||||||
|
|
||||||
### `internal/modules/dnd/extract/spells`
|
|
||||||
|
|
||||||
The spell extractor prepares a structured request from one chunk, the
|
|
||||||
chunk-scoped source input, the session, and optional D&D reference inputs. It
|
|
||||||
decodes the model response, assigns the generic source identity to every source
|
|
||||||
reference, canonicalizes duplicate references, orders spell casts by their
|
|
||||||
earliest valid source-document position, and returns `dnd.SpellList`.
|
|
||||||
|
|
||||||
Its private response schema admits only the structural transport envelope:
|
|
||||||
required fields, JSON types, array and object shapes, and unknown-field
|
|
||||||
rejection. It maps integer source-unit candidates directly without repairing
|
|
||||||
semantic values, so the deterministic shape, catalog, and source-reference
|
|
||||||
validators own blank values, empty evidence, and invalid or unresolved ranges.
|
|
||||||
|
|
||||||
The extractor owns its private model-response DTO, embedded prompt, LLM response
|
|
||||||
schema, strict option decoder, injected shared LLM client, and prompt/schema
|
|
||||||
manifest metadata. During preparation it resolves the optional `spell_catalog`
|
|
||||||
reference into an immutable effective catalog and adds a generated
|
|
||||||
canonical-name-only JSON input to every structured completion request. Overlay
|
|
||||||
failures therefore stop construction before source parsing or an LLM call;
|
|
||||||
campaign references remain separate disambiguation inputs and never become
|
|
||||||
source evidence.
|
|
||||||
|
|
||||||
The prompt includes only actual casting events and unambiguous declared casting
|
|
||||||
attempts. Spell mentions, plans, rules discussion, and catalog matches without
|
|
||||||
a casting event are excluded. Shared extraction-evidence and identity rules
|
|
||||||
require transcript-supported caster and spell facts, while the catalog,
|
|
||||||
campaign references, and NPC names only disambiguate source text. Structural
|
|
||||||
source validation remains deterministic; semantic evidence sufficiency is
|
|
||||||
enforced through extraction policy and evaluation.
|
|
||||||
|
|
||||||
Both the extractor and deterministic catalog validator expose
|
|
||||||
the effective base-plus-overlay semantic digest as scoped prepared-component
|
|
||||||
checkpoint identity. Raw overlay provenance independently covers file-byte
|
|
||||||
changes, while the semantic digest also invalidates reuse when the embedded
|
|
||||||
catalog or catalog composition changes. The extractor additionally fingerprints
|
|
||||||
its complete prompt assets and private response schema, so either semantic
|
|
||||||
contract changing invalidates previously recorded extraction checkpoints. The
|
|
||||||
separate `internal/modules/dnd/codec/spells` package
|
|
||||||
owns the durable schema and stable JSON representation for artifact kind
|
|
||||||
`dnd/spell-list`. The runner keeps the result typed through validators and later
|
|
||||||
stages, using the codec only for checkpoint, debug, and output boundaries.
|
|
||||||
Shared D&D helpers keep prompt input names and source-unit reference conversion
|
|
||||||
consistent with the scene chunker.
|
|
||||||
|
|
||||||
The extractor also declares the optional `npcs` registry slot and consumes the
|
|
||||||
immutable registry boundary from `internal/modules/dnd/npcs/registry`. An
|
|
||||||
external registry is prepared before execution; a generated registry is
|
|
||||||
validated and supplied at operation time. Bound external registries add only
|
|
||||||
the full `npc_registry_digest` and `npc_count` to module metadata. The local
|
|
||||||
`npc_registry` checkpoint fingerprint always covers the names-only projection,
|
|
||||||
including its exact unbound value. Generated bindings are represented by
|
|
||||||
framework handoff provenance and dependency fingerprints. The unbound prompt
|
|
||||||
input is exactly `{"npcs":[]}` and has no registry provenance.
|
|
||||||
The shared NPC grounding fragment is placed immediately after the common
|
|
||||||
campaign reference message and is included in the spell prompt fingerprint.
|
|
||||||
|
|
||||||
The durable payload and manifest metadata shapes are defined in the
|
|
||||||
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
|
|
||||||
|
|
||||||
### `internal/modules/dnd/extract/npcs`
|
|
||||||
|
|
||||||
The NPC extractor maps private model output to the canonical `dnd.NPCList`,
|
|
||||||
assigns source identity and deterministic NPC IDs, and preserves source
|
|
||||||
references for deterministic validation. It uses the shared campaign
|
|
||||||
references only for disambiguation and does not consume the optional NPC
|
|
||||||
registry slot. Its prompt and private response schema are package-owned. The
|
|
||||||
private response contains only a name and model-facing evidence ranges for each
|
|
||||||
record; anonymous groups, generic roles, invented labels, descriptions,
|
|
||||||
aliases, and relationships are outside its contract. The
|
|
||||||
prompt follows the shared D&D extraction ordering and cache policy documented
|
|
||||||
in [LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries).
|
|
||||||
|
|
||||||
The private response schema owns only structural transport validation and maps
|
|
||||||
integer source-unit candidates unchanged. Required semantic content, non-empty
|
|
||||||
evidence, and valid source ranges are rejected by the deterministic shape and
|
|
||||||
source-reference validators.
|
|
||||||
|
|
||||||
### `internal/modules/dnd/extract/scenedescriptions`
|
|
||||||
|
|
||||||
The scene-description extractor makes one structured completion for each
|
|
||||||
accepted chunk and maps its private `kind`, `title`, and `summary` response to
|
|
||||||
one `dnd.SceneDescription`. It assigns the current chunk ID and exact range,
|
|
||||||
preserves kind without repair, and trims only title and summary whitespace.
|
|
||||||
Optional players, party, and glossary references can disambiguate prompt terms
|
|
||||||
but do not supply evidence. The package owns its private schema, prompt assets,
|
|
||||||
and mapping fingerprint; deterministic validators own the durable semantic
|
|
||||||
checks. The durable contract is defined in the
|
|
||||||
[D&D scene-description artifact contract](../integrations/dnd-scene-description-artifacts.md).
|
|
||||||
|
|
||||||
### `internal/modules/dnd/extract/combatturns`
|
|
||||||
|
|
||||||
The combat extractor requires the `scene_descriptions` reference and resolves
|
|
||||||
it through the immutable scene-description registry before it resolves NPC
|
|
||||||
grounding or constructs prompt inputs. It calls the LLM only for an exact
|
|
||||||
current-chunk match whose kind is `combat`. Exact `narrative`, `recap`, and
|
|
||||||
`meta` matches return an accepted empty `dnd.CombatTurnList`; missing or
|
|
||||||
mismatched coverage returns the same result with one bounded unavailable-
|
|
||||||
classification warning. These deterministic results do not consume retry
|
|
||||||
attempts. Scene descriptions are control context only and are not passed to the
|
|
||||||
combat prompt or copied into combat evidence.
|
|
||||||
|
|
||||||
For eligible chunks, the extractor prepares one structured request using the
|
|
||||||
shared extraction-evidence, identity, campaign-reference, NPC-grounding, and
|
|
||||||
transcript prompt inputs. It maps the private response to
|
|
||||||
`dnd.CombatTurnList`, assigns the current source identity, removes exact
|
|
||||||
duplicate source ranges, and orders turns by valid source-document position
|
|
||||||
while preserving malformed candidate fields for deterministic validators. Its
|
|
||||||
package-owned private response schema enforces only the structural JSON
|
|
||||||
envelope; semantic artifact constraints remain with the validator chain.
|
|
||||||
|
|
||||||
Prepared metadata and checkpoint fingerprints include prompt, response-schema,
|
|
||||||
mapping, and scene-gate identities. An external scene reference additionally
|
|
||||||
reports its semantic eligibility digest and count; generated identity remains
|
|
||||||
framework handoff provenance and dependency state. Neither surface retains
|
|
||||||
scene prose or payload bytes. The prompt follows the shared D&D extraction
|
|
||||||
ordering and cache policy documented in
|
|
||||||
[LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries). The
|
|
||||||
package exposes typed registration and is included in the production D&D
|
|
||||||
registrar with the default combat extraction chain.
|
|
||||||
|
|
||||||
The combat normalizer accepts only the optional structured NPC registry.
|
|
||||||
Campaign references remain extractor-only LLM context and are not materialized
|
|
||||||
for deterministic normalization.
|
|
||||||
|
|
||||||
### `internal/modules/dnd/extract/itemevents`
|
|
||||||
|
|
||||||
The item-event extractor prepares one structured request from the accepted
|
|
||||||
chunk and optional campaign references, then maps private records to
|
|
||||||
`dnd.ItemEventList` with the current source identity. It declares only optional
|
|
||||||
`glossary`, `party`, `players`, and deprecated `roster` reference slots; these
|
|
||||||
can disambiguate names but never supply evidence. It has no NPC,
|
|
||||||
scene-description, or item-registry dependency.
|
|
||||||
|
|
||||||
The private response schema owns structural transport validation. The extractor
|
|
||||||
preserves candidate category, holder, quantity, and source-range values for the
|
|
||||||
deterministic validators, removes exact duplicate ranges, and source-orders
|
|
||||||
events. The source-reference validator requires citations to fit the current
|
|
||||||
accepted chunk. Prompt, response-schema, and mapping identities participate in
|
|
||||||
checkpoint identity. The durable schema is owned separately by
|
|
||||||
`internal/modules/dnd/codec/itemevents`.
|
|
||||||
|
|
||||||
### `internal/modules/dnd/extract/npcinteractions`
|
|
||||||
|
|
||||||
The NPC interaction extractor requires the structured `npcs` registry slot. It
|
|
||||||
uses the registry's names-only prompt projection with shared extraction
|
|
||||||
evidence, identity, and transcript material, then maps private model records to
|
|
||||||
`dnd.NPCInteractionList` with the current source identity. Registry source
|
|
||||||
references are never reused as interaction evidence. The private response
|
|
||||||
schema carries only name, bounded interaction kind, and source-unit ranges;
|
|
||||||
deterministic validators own registry membership, source validity, and
|
|
||||||
relatedness. Extract-stage source validation additionally requires every cited
|
|
||||||
range to be wholly contained in the current materialized chunk. Prompt, schema,
|
|
||||||
mapping, and the names-only registry projection
|
|
||||||
participate in checkpoint identity, while generated producer identity remains
|
|
||||||
framework provenance.
|
|
||||||
|
|
||||||
The shared D&D source-reference order defines canonical evidence ordering. The
|
|
||||||
domain-owned `internal/modules/dnd/npcinteractions` package defines occurrence
|
|
||||||
ordering, valid-evidence eligibility, and collision-safe exact identity. The
|
|
||||||
interaction normalizer and normalized invariants validator consume those
|
|
||||||
rules, so their production and checking paths cannot drift. Normalizer and
|
|
||||||
relatedness warning lists use the shared D&D diagnostic cap and emit a final
|
|
||||||
omission-summary warning when truncated.
|
|
||||||
|
|
||||||
### `internal/modules/dnd/normalize/npcs`
|
|
||||||
|
|
||||||
The NPC normalizer deterministically trims display names, recomputes IDs,
|
|
||||||
canonicalizes evidence, and consolidates equal comparison keys before semantic
|
|
||||||
work. Records are eligible for the document-level identity call only when they
|
|
||||||
have a non-empty comparison key and wholly valid current-document references.
|
|
||||||
It sends private candidate names and source ranges plus coalesced, cited
|
|
||||||
transcript windows to its own prompt; stable NPC IDs and the durable artifact
|
|
||||||
shape are not prompt inputs.
|
|
||||||
|
|
||||||
The private structured response proposes groups of supplied names and a
|
|
||||||
canonical supplied name. Deterministic comparison-key resolution validates each
|
|
||||||
group, discards unsafe or overlapping groups, and independently applies safe
|
|
||||||
ones. Application preserves earliest record order, unions canonical evidence,
|
|
||||||
and derives the final canonical ID. Invalid structured output and discarded
|
|
||||||
groups request framework retry with a safe fallback; bounded diagnostics become
|
|
||||||
durable only on final fallback exhaustion.
|
|
||||||
|
|
||||||
The normalizer records prompt and response-schema identities and digests,
|
|
||||||
identity and normalization policies, and semantic-context policy and radius as
|
|
||||||
manifest metadata. Its local checkpoint fingerprints cover the prompt, response
|
|
||||||
schema, identity policy, normalization policy, and semantic-context policy so a
|
|
||||||
meaningful behavior change invalidates prior normalize reuse.
|
|
||||||
|
|
||||||
## Merger And Normalizer
|
|
||||||
|
|
||||||
### `internal/modules/generic/merge/appendorder`
|
|
||||||
|
|
||||||
The merger passes typed values to an injected combine function in framework
|
|
||||||
source-chunk order. The D&D registrar specializes it for all six artifact
|
|
||||||
lists; each append merger preserves collection presence and order while giving
|
|
||||||
the result independently owned nested source-reference slices.
|
|
||||||
|
|
||||||
### `internal/modules/generic/normalize/noop`
|
|
||||||
|
|
||||||
The normalizer returns the merged domain value unchanged and is reusable for
|
|
||||||
any registered artifact type.
|
|
||||||
|
|
||||||
### `internal/modules/dnd/normalize/spells`
|
|
||||||
|
|
||||||
The typed spell normalizer resolves the optional `spell_catalog` reference into
|
|
||||||
the same immutable SRD-plus-overlay effective catalog used by spell extraction
|
|
||||||
and catalog validation. It performs no LLM calls. For each spell cast it
|
|
||||||
canonicalizes recognized names using the catalog's case, whitespace,
|
|
||||||
apostrophe, and alias rules; canonicalizes source references with the shared
|
|
||||||
document-aware order; removes only exact reference duplicates; and emits
|
|
||||||
bounded, scoped warnings for each mutation or unresolved name.
|
|
||||||
|
|
||||||
After those per-cast changes, it collapses only casts with the same canonical
|
|
||||||
spell, case-folded and whitespace-normalized caster, and complete non-empty
|
|
||||||
valid source-reference set. It retains the first occurrence and its caster,
|
|
||||||
source references, and stable order. Unknown names, empty or invalid evidence,
|
|
||||||
and adjacent or overlapping but different ranges remain unchanged for
|
|
||||||
validation.
|
|
||||||
|
|
||||||
The normalizer exposes the effective catalog digest as its independently scoped
|
|
||||||
`effective_catalog` checkpoint fingerprint and reports catalog base ID, digest,
|
|
||||||
and overlay IDs as manifest metadata. Catalog contents, reference paths, and
|
|
||||||
raw overlay bytes are not included in either surface. The normalize-stage
|
|
||||||
reference is stage-local, so an overlay-capable pipeline binds the catalog
|
|
||||||
independently for extraction and normalization.
|
|
||||||
|
|
||||||
### `internal/modules/dnd/normalize/combatturns`
|
|
||||||
|
|
||||||
The combat normalizer prepares an external NPC registry before execution or
|
|
||||||
receives a generated registry at the ordered step handoff, then uses the
|
|
||||||
immutable view during runtime. It display-normalizes actors,
|
|
||||||
rewrites canonical-name matches for actors, orders and deduplicates source
|
|
||||||
references, stable-sorts records by source-document position, and collapses
|
|
||||||
only exact duplicate identities with fully valid evidence. It deep-clones
|
|
||||||
output storage and emits bounded warnings scoped to merged input indexes. Its
|
|
||||||
metadata and fingerprints identify the normalization and NPC identity policies.
|
|
||||||
External bindings may contribute registry
|
|
||||||
digest/count metadata; generated identity is retained in framework provenance
|
|
||||||
and dependency fingerprints. The normalizer is included in the production D&D
|
|
||||||
registrar with the default combat normalization chain.
|
|
||||||
|
|
||||||
### `internal/modules/dnd/normalize/itemevents`
|
|
||||||
|
|
||||||
The item-event normalizer accepts no options or references and makes no LLM
|
|
||||||
calls. It trims display-edge whitespace in names and holders, canonicalizes
|
|
||||||
source references, source-orders events, and collapses only exact duplicates
|
|
||||||
with complete valid evidence. It does not create a ledger, calculate balances,
|
|
||||||
resolve aliases, infer quantities or holders, or reconcile nearby events. Its
|
|
||||||
policy fingerprint and bounded warnings identify deterministic normalization;
|
|
||||||
the matching invariant validator checks the resulting order and duplicate rule.
|
|
||||||
|
|
||||||
### `internal/modules/dnd/normalize/npcinteractions`
|
|
||||||
|
|
||||||
The interaction normalizer requires the same immutable NPC registry. It
|
|
||||||
canonicalizes exact registry-name matches, orders and de-duplicates source
|
|
||||||
references, stable-sorts occurrences by source-document position, and collapses
|
|
||||||
only exact interaction identities with valid evidence. It does not infer,
|
|
||||||
merge, or summarize distinct occurrences. Its metadata and fingerprints expose
|
|
||||||
the normalization and NPC identity policies; generated registry identity stays
|
|
||||||
in framework provenance and checkpoint dependencies.
|
|
||||||
|
|
||||||
### `internal/modules/dnd/normalize/scenedescriptions`
|
|
||||||
|
|
||||||
The scene-description normalizer has no options or references. It validates
|
|
||||||
each source range against the source document, trims title and summary
|
|
||||||
whitespace, orders records by source position then ID, removes only exactly
|
|
||||||
identical records, and rejects conflicting reused IDs or ranges. Its policy
|
|
||||||
fingerprint identifies this deterministic behavior; the matching invariant
|
|
||||||
validator checks the normalized result in the production chain.
|
|
||||||
|
|
||||||
## Output Encoder
|
|
||||||
|
|
||||||
### `internal/modules/generic/output/json`
|
|
||||||
|
|
||||||
The JSON encoder sorts normalized results by lane, derives collision-checked
|
|
||||||
safe logical names, pretty-prints JSON payloads, and assembles the logical index,
|
|
||||||
manifest, rejected-result, warning, and lane files. Invalid JSON, unsupported
|
|
||||||
media types, unsafe names, and sanitized-name collisions are errors.
|
|
||||||
|
|
||||||
Its strict `include_chunk_map` option is disabled by default. When enabled, it
|
|
||||||
validates the framework-supplied accepted chunk map through its codec and adds
|
|
||||||
the pipeline-wide `chunk-map.json` plus its index descriptor; it does not treat
|
|
||||||
the map as a lane payload. The external shape is owned by the
|
|
||||||
[Accepted Chunk Map contract](../integrations/chunk-map.md).
|
|
||||||
|
|
||||||
The encoder returns logical files only. The CLI places them on disk, and the
|
|
||||||
[JSON output contract](../integrations/json-output.md) defines their external
|
|
||||||
paths and schemas.
|
|
||||||
|
|
||||||
## Generic Validators
|
|
||||||
|
|
||||||
The generic validator implementations live under
|
|
||||||
`internal/modules/generic/validate`.
|
|
||||||
|
|
||||||
The unconditional accept and reject validators provide explicit chunk and
|
|
||||||
typed-artifact variants used primarily for controlled composition and tests.
|
|
||||||
|
|
||||||
The serialized JSON syntax validator uses `encoding/json` to reject malformed
|
|
||||||
representation bytes. The serialized JSON Schema validator requires schema
|
|
||||||
bytes, parses the instance and schema with `jsonschema`, and distinguishes
|
|
||||||
payload rejection from schema loading or compilation errors. The framework
|
|
||||||
serialized-validation request carries either canonical chunk bytes or artifact
|
|
||||||
codec bytes according to its target context. Neither validator calls the LLM.
|
|
||||||
|
|
||||||
## D&D Spell Validators
|
|
||||||
|
|
||||||
All four validators receive `dnd.SpellList` directly. The shape validator
|
|
||||||
rejects a missing list, blank caster or spell names, and empty reference lists.
|
|
||||||
The catalog validator defers when shape is invalid, then checks every non-empty
|
|
||||||
spell name against the immutable effective SRD and overlay catalog. It accepts
|
|
||||||
normalized canonical names and aliases without rewriting the artifact; unknown
|
|
||||||
names reject the complete result with bounded, stable index/name diagnostics. The
|
|
||||||
source-reference validator defers malformed shapes, validates every cited
|
|
||||||
range, and reports all range defects through a bounded aggregate while
|
|
||||||
preserving `invalid_source_refs`. The relatedness validator resolves all cited
|
|
||||||
ranges through the shared document-order traversal, then warns when a normalized
|
|
||||||
consecutive spell-name token sequence is absent from the cited source text.
|
|
||||||
Invalid shape
|
|
||||||
or cited ranges produce no relatedness warnings; the shape and source-reference
|
|
||||||
validators own those defects.
|
|
||||||
|
|
||||||
These validators are deterministic. Shape, source-reference, and relatedness
|
|
||||||
each expose a local semantic `policy` checkpoint fingerprint. The catalog
|
|
||||||
validator instead exposes its effective catalog digest as its semantic
|
|
||||||
checkpoint identity and does not add a separate policy fingerprint. Their
|
|
||||||
selectable keys and production order are defined in
|
|
||||||
[Configuration](../config.md#implemented-production-validators); their durable
|
|
||||||
payload rules are defined in the
|
|
||||||
[artifact contract](../integrations/dnd-spell-artifacts.md).
|
|
||||||
|
|
||||||
## D&D NPC Validators
|
|
||||||
|
|
||||||
NPC shape validation checks the required ID and name strings, list presence, and source-reference
|
|
||||||
shape. The source-reference validator defers malformed shapes, checks
|
|
||||||
current-document identity, unit existence, and range ordering, and reports all
|
|
||||||
defects through bounded aggregates. Source relatedness uses the shared
|
|
||||||
document-order traversal and normalized consecutive-token matching, emitting at
|
|
||||||
most one bounded warning per record when the canonical name does not occur near
|
|
||||||
its cited text. Invalid shape or cited ranges produce no relatedness warnings.
|
|
||||||
Normalize identity validation checks deterministic IDs, canonical names, and
|
|
||||||
duplicate canonical-name or ID ownership.
|
|
||||||
All are deterministic and expose the policy fingerprints used by the
|
|
||||||
production chains.
|
|
||||||
|
|
||||||
## D&D Combat Validators
|
|
||||||
|
|
||||||
Combat shape validation owns the required list, actor, supported turn kind, and
|
|
||||||
non-empty source-reference collection. Combat source-reference validation defers invalid
|
|
||||||
shape, checks source identity, unit existence, and range order, and reports all
|
|
||||||
defects through bounded aggregates. Combat source-relatedness defers invalid
|
|
||||||
shape or ranges, uses the shared traversal to combine overlapping cited units
|
|
||||||
in document order, and emits at most one bounded advisory warning per turn for
|
|
||||||
an unrelated actor. Actors use normalized consecutive-token matching. The
|
|
||||||
normalized-invariants validator owns actor display normalization, canonical
|
|
||||||
source-reference order, chronology, and exact duplicate identity; it defers
|
|
||||||
shape and source-reference failures. All four validators are deterministic and
|
|
||||||
expose local policy fingerprints. In the registered defaults, JSON syntax runs
|
|
||||||
first; combat shape, normalized invariants when applicable, and source-reference
|
|
||||||
validation precede JSON Schema validation; warning-only relatedness runs last.
|
|
||||||
|
|
||||||
## D&D Item-Event Validators
|
|
||||||
|
|
||||||
Item-event shape validation owns the required list, non-empty name, supported
|
|
||||||
category, category-and-holder combination, positive optional quantity, and
|
|
||||||
non-empty source-reference collection. Source-reference validation defers
|
|
||||||
malformed shapes, checks current-source identity and ordered ranges, and during
|
|
||||||
extraction requires every citation to fit the accepted chunk. Relatedness is
|
|
||||||
advisory and warning-only: it checks the event name against cited transcript
|
|
||||||
text while deferring malformed candidates and invalid ranges to their blocking
|
|
||||||
owners. The normalized-invariants validator owns display normalization,
|
|
||||||
canonical source-reference order, chronology, and exact duplicate identity.
|
|
||||||
All four validators are deterministic and expose policy fingerprints. The
|
|
||||||
registered chains run syntax and blocking checks before durable JSON Schema;
|
|
||||||
relatedness remains last.
|
|
||||||
|
|
||||||
## D&D NPC Interaction Validators
|
|
||||||
|
|
||||||
Interaction shape validation owns the required list, registry name, supported
|
|
||||||
kind, and non-empty source-reference collection. Registry validation checks
|
|
||||||
exact membership in the required immutable NPC registry. Source-reference and
|
|
||||||
relatedness validation use the current transcript only; malformed candidates
|
|
||||||
are deferred by later validators and produce no relatedness warning. The
|
|
||||||
normalized-invariants validator owns canonical registry names, source-reference
|
|
||||||
order, chronology, and exact duplicate identity. The production chains run
|
|
||||||
shape, registry, and source-reference checks before JSON Schema validation;
|
|
||||||
relatedness remains warning-only and last.
|
|
||||||
|
|
||||||
## D&D Scene Description Validators
|
|
||||||
|
|
||||||
Scene-description shape validation owns the non-empty list, trimmed ID and
|
|
||||||
prose, closed kind, and basic source-reference shape. Extract-stage source
|
|
||||||
validation additionally requires the one record to attach exactly to the
|
|
||||||
current accepted chunk; later source validation checks source membership.
|
|
||||||
Relatedness checks the title and summary independently against only their cited
|
|
||||||
transcript range and emits bounded advisory warnings. The normalized-invariants
|
|
||||||
validator owns ordering, exact duplicate elimination, and conflicting ID or
|
|
||||||
range detection. The production chains run shape and source-reference checks
|
|
||||||
before JSON Schema validation; the warning-only relatedness check is last.
|
|
||||||
|
|
||||||
## Production Registration
|
|
||||||
|
|
||||||
Production composition occurs through family registrars. The CLI allocates one
|
|
||||||
complete framework registry set and one LLM asset registry. It invokes
|
|
||||||
`internal/modules/generic/register`,
|
|
||||||
`internal/modules/seriatim/register`, and `internal/modules/dnd/register` in
|
|
||||||
that order, then exposes the matching catalog for resolution. The generic and
|
|
||||||
Seriatim registrars own their production leaf registrations. The D&D registrar
|
|
||||||
owns D&D leaf registrations, typed spell, NPC, combat, item-event, interaction, and scene-description default-validator
|
|
||||||
chains, typed append-order specializations, and D&D prompt/schema asset
|
|
||||||
collection. Its registration helpers group module, validator, prompt-asset, and
|
|
||||||
chain composition while retaining artifact-specific merge and clone behavior in
|
|
||||||
the registrar.
|
|
||||||
|
|
||||||
Concrete implementation packages do not import generic implementation
|
|
||||||
packages directly. A concrete family's `register` package is its composition
|
|
||||||
point for specializing reusable generic implementations, while the generic
|
|
||||||
registrar composes only generic children.
|
|
||||||
|
|
||||||
Core and framework production packages do not import production extensions.
|
|
||||||
CLI production code is the sole application composition root for extensions
|
|
||||||
and imports only exact family registrar packages. Other production packages,
|
|
||||||
including commands and newly introduced package trees, do not import module
|
|
||||||
packages directly. Compatibility tests in the CLI, core, and framework trees
|
|
||||||
may import roots and implementation leaves directly. Other non-module tests do
|
|
||||||
not receive that exemption. White-box tests within module families retain the
|
|
||||||
production family boundaries. `internal/modules/integration` is test
|
|
||||||
infrastructure: its black-box tests may compose multiple families, but it is
|
|
||||||
not a production module family or production dependency target.
|
|
||||||
|
|
||||||
## Adding An Extension
|
|
||||||
|
|
||||||
When adding a production module or validator:
|
|
||||||
|
|
||||||
1. implement the stage or validator contract and package-local key;
|
|
||||||
2. expose and test its spec, constructor, and registration function;
|
|
||||||
3. keep format or domain parsing inside the concrete package;
|
|
||||||
4. add package-owned prompt/schema assets when the extension is LLM-backed;
|
|
||||||
new LLM-backed D&D extraction modules must follow the stable-to-variable
|
|
||||||
prompt ordering, shared-asset ownership, and cache-boundary policy in
|
|
||||||
[LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries), or
|
|
||||||
document the implemented exception and its evidence there;
|
|
||||||
5. register it through its package-family registrar and add a default chain
|
|
||||||
there only when production policy requires one;
|
|
||||||
6. add resolution and composition coverage for capabilities, options,
|
|
||||||
references, and validation behavior;
|
|
||||||
7. update the selectable-key catalog in [Configuration](../config.md), the
|
|
||||||
relevant external contract, this inventory, and maintained examples when
|
|
||||||
user-visible behavior changes.
|
|
||||||
|
|
||||||
Do not add the extension to `docs/development.md`; that file routes by task and
|
|
||||||
does not inventory implementations.
|
|
||||||
|
|
||||||
### D&D Extractor Contract
|
|
||||||
|
|
||||||
New D&D extractors preserve these package-owned responsibilities:
|
|
||||||
|
|
||||||
- Reject unknown options unless an option namespace is intentionally
|
|
||||||
extensible, and use shared common preflight while retaining receiver,
|
|
||||||
dependency, and lane-specific checks locally.
|
|
||||||
- Return independently owned results and exposed metadata that callers may
|
|
||||||
safely mutate.
|
|
||||||
- Keep the private response DTO, structural response schema and its identity,
|
|
||||||
provider-response mapping, durable artifact conversion, and lane diagnostics
|
|
||||||
in the owning package.
|
|
||||||
- Include every stable semantic input that can change durable output in
|
|
||||||
checkpoint identity. Consider prompt, schema, mapping, canonicalization,
|
|
||||||
prepared reference projections, identity, normalization, and trimming where
|
|
||||||
applicable.
|
|
||||||
- Add focused behavioral coverage where the lane's risks warrant it, including
|
|
||||||
construction and registration, option rejection, preflight, provider
|
|
||||||
failures, structured decoding, mapping and ownership, prompt
|
|
||||||
role/input/cache order, and checkpoint invalidation.
|
|
||||||
|
|
||||||
Prompt ordering, shared-asset ownership, cache boundaries, and private-schema
|
|
||||||
rules are defined in [LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries).
|
|
||||||
[Pipeline Internals](pipeline.md#reference-materialization) owns reference
|
|
||||||
materialization, and its [checkpoint hooks](pipeline.md#checkpoint-and-debug-hooks)
|
|
||||||
define checkpoint behavior. Follow [Architecture](../policy/architecture.md#source-and-domain-boundaries)
|
|
||||||
for ownership boundaries and the [Testing Policy](../policy/testing.md) when
|
|
||||||
selecting durable coverage. This contract intentionally does not prescribe
|
|
||||||
prompt prose or length, hashes, test counts, filenames, fixture layouts, or
|
|
||||||
generic implementation builders.
|
|
||||||
|
|
||||||
## Tests To Inspect
|
|
||||||
|
|
||||||
- Package-local `*_test.go` files under the module or validator being changed.
|
|
||||||
- `internal/framework/pipeline/typed_resolution_test.go`: typed registry, spec,
|
|
||||||
and heterogeneous artifact composition.
|
|
||||||
- `internal/framework/pipeline/profile_test.go`: framework binding defaults and
|
|
||||||
profile resolution.
|
|
||||||
- `internal/cli/production_contract_test.go`: production catalog, config
|
|
||||||
resolution, and composition smoke coverage.
|
|
||||||
- `internal/cli/example_contract_test.go`: maintained example ownership.
|
|
||||||
- `internal/framework/promptfs/*_test.go` and
|
|
||||||
`internal/modules/dnd/shared/*_test.go`: shared prompt and reference assembly.
|
|
||||||
- `internal/modules/integration/*_test.go`: black-box composition across
|
|
||||||
production extension domains.
|
|
||||||
|
|||||||
@@ -1,191 +1,58 @@
|
|||||||
# Internal Overview
|
# Internal Overview
|
||||||
|
|
||||||
This document inventories the implemented Notarius components. Normative
|
This document is the implemented component map for Notarius. Normative
|
||||||
boundaries and dependency direction belong in
|
boundaries and dependency direction belong in
|
||||||
[Architecture](../policy/architecture.md); external behavior belongs in the
|
[Architecture](../policy/architecture.md). User and operator contracts belong
|
||||||
[CLI](../cli.md), [Configuration](../config.md),
|
in the [CLI](../cli.md), [Configuration](../config.md),
|
||||||
[Operations](../operations.md), and [integration contracts](../integrations/).
|
[Operations](../operations.md), and [integration contracts](../integrations/).
|
||||||
|
|
||||||
## Execution Path
|
## Execution Path
|
||||||
|
|
||||||
`cmd/notarius` delegates to `internal/cli`, the production composition root.
|
~~~
|
||||||
The CLI loads configuration, builds the production catalogs and runtime
|
cmd/notarius -> internal/cli -> configuration and production composition
|
||||||
collaborators, invokes `internal/framework/pipeline`, and places the logical
|
-> internal/framework/pipeline -> logical output files
|
||||||
output files returned by the runner. Cache and debug collaborators are supplied
|
-> internal/cli -> durable output and optional state/debug data
|
||||||
at this boundary.
|
~~~
|
||||||
|
|
||||||
Resolution produces a fixed ordered workflow of steps and globally unique,
|
The CLI is the application boundary: it discovers configuration, composes
|
||||||
sorted artifact lanes. Preparation constructs the complete module and validator
|
production registries and runtime collaborators, invokes the framework, and
|
||||||
set before the runner receives source bytes. Source parsing and chunking are
|
places returned files. The framework resolves and prepares a fixed extraction
|
||||||
serial. Each step then uses a bounded run-wide extraction pool followed by
|
pipeline, then returns logical results without owning process behavior or
|
||||||
serial per-lane merge and normalize continuations. A step barrier prevents
|
physical state roots.
|
||||||
later consumers from starting until all earlier lanes are terminal and their
|
|
||||||
required normalized artifacts have crossed the typed handoff.
|
|
||||||
|
|
||||||
## Application Boundary
|
## Components
|
||||||
|
|
||||||
| Package | Implemented responsibility |
|
| Area | Implemented owners | Responsibility |
|
||||||
| --- | --- |
|
|
||||||
| `cmd/notarius` | Executable entry point and process exit delegation. |
|
|
||||||
| `internal/cli` | Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, state collaborator setup, durable writes, and user-facing results. |
|
|
||||||
|
|
||||||
## Core Packages
|
|
||||||
|
|
||||||
| Package | Implemented responsibility |
|
|
||||||
| --- | --- |
|
|
||||||
| `internal/core/artifacts` | Run-manifest and provenance models. |
|
|
||||||
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
|
|
||||||
| `internal/core/debugbundle` | Explicit per-run debug-bundle allocation and redacted summary writing. |
|
|
||||||
| `internal/core/fileio` | Generic confined atomic file and JSON writes with caller-selected permissions. |
|
|
||||||
| `internal/core/source` | Generic source documents, units, chunks, canonical references, validation, deterministic source digests, and independent metadata materialization. |
|
|
||||||
|
|
||||||
## Framework Packages
|
|
||||||
|
|
||||||
| Package | Implemented responsibility |
|
|
||||||
| --- | --- |
|
|
||||||
| `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. |
|
|
||||||
| `internal/framework/pipeline` | Module and artifact-codec registries, ordered-step and generated-reference resolution, option validation, profile resolution, capability checks, external reference materialization, complete pipeline preparation, typed handoff, retries, orchestration, warnings, checkpoint decisions, and manifest population. |
|
|
||||||
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
|
|
||||||
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
|
|
||||||
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
|
|
||||||
| `internal/framework/checkpoint` | Root-based checkpoint loading, recording, identity, and payload serialization. |
|
|
||||||
| `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. |
|
|
||||||
| `internal/framework/chunkmap` | Strict durable accepted chunk-map construction, schema, validation, cloning, and serialization. |
|
|
||||||
| `internal/framework/debug` | Root-based framework and LLM debug recording. |
|
|
||||||
|
|
||||||
Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
|
|
||||||
serialized-validator, and
|
|
||||||
typed-validator interfaces. The runner owns handoff provenance, validation
|
|
||||||
sequencing, rejection handling, checkpoint and debug boundaries, and final
|
|
||||||
manifest assembly.
|
|
||||||
|
|
||||||
Artifact registries support heterogeneous typed extraction entries and
|
|
||||||
kind-specific merger, normalizer, and validator variants. Resolution derives a
|
|
||||||
lane's kind from its extractor, requires the matching codec, verifies exact Go
|
|
||||||
type equality across the lane, and records schema identity in the resolved lane
|
|
||||||
and pipeline digest. Registry entries carry separate option-validation and
|
|
||||||
run-local construction closures. Preparation injects shared dependencies and
|
|
||||||
constructs input, chunk, validators, ordered lanes, and output before source
|
|
||||||
parsing. Production modules use strict construction-time option decoding, and
|
|
||||||
LLM-backed modules retain the injected shared client. The D&D family registers
|
|
||||||
the canonical `dnd/spell-list`, `dnd/npc-list`, `dnd/combat-turn-list`,
|
|
||||||
`dnd/item-event-list`, `dnd/npc-interaction-list`, and
|
|
||||||
`dnd/scene-description-list` codecs, typed spell, NPC, combat, item-event,
|
|
||||||
interaction, and scene-description extractors and normalizers, validators,
|
|
||||||
plus kind-specific generic merge strategies; generic JSON validators use the
|
|
||||||
serialized-validation contract. The runner executes lanes through
|
|
||||||
private exact-type-checked closures, coordinates extract results independently
|
|
||||||
of completion timing, and serializes artifacts only through their codec at
|
|
||||||
checkpoint, debug, and output boundaries.
|
|
||||||
|
|
||||||
## Production Extensions
|
|
||||||
|
|
||||||
The canonical catalogs of user-selectable
|
|
||||||
[module](../config.md#implemented-production-modules) and
|
|
||||||
[validator](../config.md#implemented-production-validators) keys are in
|
|
||||||
Configuration. The implemented module packages are:
|
|
||||||
|
|
||||||
| Package | Implemented responsibility |
|
|
||||||
| --- | --- |
|
|
||||||
| `internal/modules/seriatim/input/transcript` | Parses the supported Seriatim transcript format into the generic source model. |
|
|
||||||
| `internal/modules/generic/chunk/units` | Splits ordered source units by unit count and overlap. |
|
|
||||||
| `internal/modules/dnd/chunk/scenes` | Produces contiguous D&D scene chunks from structured model output. |
|
|
||||||
| `internal/modules/dnd` | Owns the canonical D&D spell-list, spell-cast, NPC-list, NPC, combat-turn-list, combat-turn, item-event-list, item-event, NPC-interaction-list, and scene-description-list artifact types. |
|
|
||||||
| `internal/modules/dnd/codec/spells` | Strictly decodes and stably encodes the durable D&D spell-list representation. |
|
|
||||||
| `internal/modules/dnd/codec/npcs` | Strictly decodes and stably encodes the durable D&D NPC-list representation. |
|
|
||||||
| `internal/modules/dnd/codec/combatturns` | Strictly decodes and stably encodes the durable D&D combat-turn-list representation. |
|
|
||||||
| `internal/modules/dnd/codec/itemevents` | Strictly decodes and stably encodes the durable D&D item-event-list representation. |
|
|
||||||
| `internal/modules/dnd/codec/npcinteractions` | Strictly decodes and stably encodes the durable D&D NPC-interaction-list representation. |
|
|
||||||
| `internal/modules/dnd/codec/scenedescriptions` | Strictly decodes and stably encodes the durable D&D scene-description-list representation. |
|
|
||||||
| `internal/modules/dnd/extract/spells` | Maps private structured model output to canonical source-grounded D&D spell lists. |
|
|
||||||
| `internal/modules/dnd/extract/npcs` | Maps private structured model output to canonical source-grounded D&D NPC lists. |
|
|
||||||
| `internal/modules/dnd/extract/combatturns` | Uses exact scene eligibility to select combat chunks, then maps private structured model output to source-grounded D&D combat-turn candidates. |
|
|
||||||
| `internal/modules/dnd/extract/itemevents` | Maps private structured model output to source-grounded D&D item-event candidates. |
|
|
||||||
| `internal/modules/dnd/extract/npcinteractions` | Maps private structured model output to current-source NPC interaction candidates grounded by a required registry. |
|
|
||||||
| `internal/modules/dnd/extract/scenedescriptions` | Maps one private scene description to the current accepted chunk's ID and exact range. |
|
|
||||||
| `internal/modules/dnd/npcinteractions` | Owns interaction occurrence ordering, valid-evidence checks, and exact interaction identity shared by normalization and invariant validation. |
|
|
||||||
| `internal/modules/dnd/normalize/combatturns` | Canonicalizes and orders merged combat turns, applies exact NPC identity matches, and collapses only exact valid-evidence duplicates. |
|
|
||||||
| `internal/modules/dnd/normalize/itemevents` | Trims, source-orders, and removes only exact valid-evidence item-event duplicates. |
|
|
||||||
| `internal/modules/dnd/normalize/npcinteractions` | Canonicalizes required-registry names, orders interaction occurrences, and collapses only exact valid-evidence duplicates. |
|
|
||||||
| `internal/modules/dnd/normalize/scenedescriptions` | Trims, source-orders, and removes only exactly identical scene descriptions while rejecting ID and range conflicts. |
|
|
||||||
| `internal/modules/dnd/validate/combatturns` | Provides deterministic shape, source-reference, source-relatedness, and normalized-invariant validation for the production combat chains. |
|
|
||||||
| `internal/modules/dnd/validate/itemevents` | Provides deterministic shape, source-reference, source-relatedness, and normalized-invariant validation for item-event chains. |
|
|
||||||
| `internal/modules/dnd/validate/npcinteractions` | Provides deterministic shape, registry, source-reference, source-relatedness, and normalized-invariant validation for interaction chains. |
|
|
||||||
| `internal/modules/dnd/validate/scenedescriptions` | Provides deterministic shape, exact extraction attachment, source-relatedness, and normalized-invariant validation for scene-description chains. |
|
|
||||||
| `internal/modules/dnd/npcs/registry` | Resolves validated normalized NPC references into immutable grounding data and exact identity lookup. |
|
|
||||||
| `internal/modules/dnd/scenedescriptions/registry` | Resolves approved scene descriptions into immutable exact-match combat eligibility data without retaining scene prose. |
|
|
||||||
| `internal/modules/dnd/npcs/identity` | Owns Unicode-aware NPC identity, ID derivation, and registry collision validation. |
|
|
||||||
| `internal/modules/dnd/spells/catalog` | Embeds and validates the versioned D&D 5e 2014 SRD catalog, composes optional overlays, and provides immutable effective lookup. |
|
|
||||||
| `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. |
|
|
||||||
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
|
|
||||||
| `internal/modules/dnd/normalize/spells` | Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity. |
|
|
||||||
| `internal/modules/dnd/normalize/npcs` | Deterministically prepares and safely applies document-level LLM-assisted NPC identity consolidation, preserving canonical evidence, order, and diagnostics. |
|
|
||||||
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, rejections, and an explicitly enabled accepted chunk map as logical JSON files. |
|
|
||||||
|
|
||||||
`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
|
|
||||||
reference declarations, prompt input assembly, document-aware source-reference
|
|
||||||
ordering and canonicalization, and bounded diagnostics under
|
|
||||||
`internal/modules/dnd/shared/diagnostics`.
|
|
||||||
The shared NPC grounding fragment is mounted for D&D prompts and is owned by
|
|
||||||
this package. Domain-neutral prompt filesystem composition lives in
|
|
||||||
`internal/framework/promptfs`.
|
|
||||||
|
|
||||||
The `dnd/npcs/registry` package owns the optional `npcs` registry boundary.
|
|
||||||
External references are strictly decoded and identity-validated during
|
|
||||||
preparation; generated references are decoded and identity-validated at the
|
|
||||||
ordered step handoff. Both paths retain canonical registry JSON for provenance
|
|
||||||
and emit a names-only projection to operation-time spell, combat, and
|
|
||||||
interaction prompts. Combat and interaction normalization use the canonical
|
|
||||||
registry for exact name lookup. The
|
|
||||||
framework records generated identity and bounded producer provenance, while
|
|
||||||
the raw external reference remains independently tracked by pipeline
|
|
||||||
provenance. An absent registry is represented only by the empty prompt value
|
|
||||||
`{"npcs":[]}`. Spell
|
|
||||||
and combat consumers use this shared boundary without changing their public
|
|
||||||
module contracts. Interaction consumers require it and retain only current
|
|
||||||
transcript references as durable evidence.
|
|
||||||
|
|
||||||
Generic validators under `internal/modules/generic/validate` provide
|
|
||||||
unconditional test decisions, JSON syntax validation, and JSON Schema
|
|
||||||
validation. D&D spell validators under `internal/modules/dnd/validate/spells`
|
|
||||||
consume the canonical spell-list type directly to provide shape,
|
|
||||||
effective-catalog, source-reference, and source-relatedness decisions.
|
|
||||||
|
|
||||||
Production composition is grouped behind package-family registrars, and every
|
|
||||||
implemented production extension uses its domain-first tree:
|
|
||||||
|
|
||||||
| Package | Implemented responsibility |
|
|
||||||
| --- | --- |
|
|
||||||
| `internal/modules/generic/register` | Registers domain-neutral chunk, merge, normalize, output, and validator implementations. |
|
|
||||||
| `internal/modules/seriatim/register` | Registers the Seriatim input adapter. |
|
|
||||||
| `internal/modules/dnd/register` | Registers D&D modules, validators, default validator policy, and prompt/schema assets. |
|
|
||||||
|
|
||||||
The CLI allocates the framework registries and asset registry, then invokes
|
|
||||||
these registrars in generic, Seriatim, and D&D order.
|
|
||||||
|
|
||||||
Implementation details for all production extensions are in
|
|
||||||
[Module Internals](modules.md).
|
|
||||||
|
|
||||||
## Run-State Components
|
|
||||||
|
|
||||||
| Surface | Implemented owners | Internal purpose |
|
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
|
| Executable and command boundary | **cmd/notarius**, **internal/cli** | Process entry, command dispatch, configuration discovery, production composition, runtime collaborator setup, durable file placement, and user-facing reporting. |
|
||||||
| Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes, including ordered-step scope and generated-artifact dependency decisions. |
|
| Configuration | **internal/core/config** | Defaults, strict YAML parsing, environment overrides, structural validation, effective resolution, redaction, and resolved-composition summaries. |
|
||||||
| Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. |
|
| Generic models | **internal/core/source**, **internal/core/artifacts**, **internal/framework/contracts** | Source documents and chunks, manifests and provenance, plus typed artifact, reference, validation, output, and structured-completion contracts. |
|
||||||
| Debug bundles | `internal/core/debugbundle`, `internal/framework/debug`, and pipeline instrumentation | Persist redacted summaries and application-owned traces. |
|
| Pipeline framework | **internal/framework/pipeline** | Registries, profile and reference resolution, typed preparation, validation, retry coordination, ordered execution, handoff, and result assembly. |
|
||||||
|
| LLM and prompt runtime | **internal/framework/llm**, **internal/framework/promptfs** | Provider-neutral structured completions, scheduling, profile recording, prompt assets, schema registration, and credential-shaped-value redaction. |
|
||||||
|
| Runtime state | **internal/core/fileio**, **internal/core/debugbundle**, **internal/framework/checkpoint**, **internal/framework/chunkplan**, **internal/framework/chunkmap**, **internal/framework/debug** | Confined atomic files, debug bundles, checkpoint and chunk-plan state, accepted chunk maps, and pipeline-facing debug recording. |
|
||||||
|
| Production extensions | **internal/modules/generic**, **internal/modules/seriatim**, **internal/modules/dnd** | Domain-neutral extensions, Seriatim input support, and D&D extraction families registered into the production catalog. |
|
||||||
|
|
||||||
Physical layout, cleanup, recovery, and sensitive-data handling are defined
|
Generic core and framework packages do not depend on production extensions.
|
||||||
in [Operations](../operations.md). Concrete modules receive recorder
|
Concrete extensions depend inward on their contracts and are registered only at
|
||||||
interfaces and request data, not physical state roots.
|
the CLI composition boundary.
|
||||||
|
|
||||||
## Focused Documentation
|
## Focused Documentation
|
||||||
|
|
||||||
- [Pipeline Internals](pipeline.md): resolution, execution, validation, retries,
|
- [Configuration Internals](configuration.md): loading, validation, effective
|
||||||
checkpoint/debug hooks, and result assembly.
|
resolution, redaction, and resolved-composition identity.
|
||||||
- [Module Internals](modules.md): production modules, validators, assets,
|
- [CLI Internals](cli.md): command dispatch, production composition, run
|
||||||
registration, and the contributor recipe for adding an extension.
|
orchestration, and terminal reporting.
|
||||||
- [LLM Runtime](llm.md): structured completion contracts, Scriptorium adapter,
|
- [Pipeline Internals](pipeline.md): resolution, preparation, execution,
|
||||||
assets, scheduling, profile recording, and redaction.
|
validation, typed handoff, and framework state hooks.
|
||||||
- [Run State Internals](state.md): output, cache, debug collaborator
|
- [Run State Internals](state.md): output, cache, debug collaborator
|
||||||
composition, and path safety.
|
composition, and path safety.
|
||||||
|
- [LLM Runtime](llm.md): structured completion, scheduling, prompt assets,
|
||||||
|
profiles, and secret handling.
|
||||||
|
- [Module Internals](modules.md): generic extension registration, module
|
||||||
|
construction, validation, and reference mechanics.
|
||||||
|
- [D&D Module Internals](dnd.md): shared D&D extractor conventions, generated
|
||||||
|
reference projections, and lane-specific exceptions. Durable D&D and
|
||||||
|
Seriatim data shapes remain in the [integration contracts](../integrations/).
|
||||||
|
|
||||||
|
Use this map to find an owner, then read the focused document and its tests
|
||||||
|
before changing behavior.
|
||||||
|
|||||||
@@ -1,491 +1,168 @@
|
|||||||
# Pipeline Internals
|
# Pipeline Internals
|
||||||
|
|
||||||
The implemented resolver and runner live in `internal/framework/pipeline`.
|
This document describes the framework-owned pipeline mechanics in
|
||||||
Their fixed workflow and ownership boundaries are defined by
|
**internal/framework/pipeline**. [Configuration](../config.md) owns selectable
|
||||||
[Architecture](../policy/architecture.md#system-shape). Configuration fields,
|
profiles, bindings, and retry settings; [Operations](../operations.md) owns
|
||||||
defaults, and selectable keys are defined in
|
state lifecycle and recovery; and the [integration contracts](../integrations/)
|
||||||
[Configuration](../config.md#pipelines).
|
own durable output shapes. Concrete production extensions are covered by
|
||||||
|
[Module Internals](modules.md).
|
||||||
|
|
||||||
Resolution fixes the ordered steps, selected lanes, and all stage bindings;
|
## Boundary
|
||||||
preparation constructs every selected implementation before the runner begins
|
|
||||||
source work. After serial input parsing and plan selection or generation, the
|
|
||||||
runner materializes chunks and executes one step at a time. Within a step,
|
|
||||||
extract work uses one bounded run-wide worker pool in chunk-first, lane-second
|
|
||||||
order. Each lane's merge and normalize operations remain serial, and lanes in
|
|
||||||
the same step may overlap once their extracts are terminal. A later step cannot
|
|
||||||
start across its barrier until every earlier lane is terminal and each required
|
|
||||||
generated artifact has been accepted and handed off.
|
|
||||||
|
|
||||||
## Resolution
|
The pipeline framework accepts a resolved composition, registries, shared
|
||||||
|
dependencies, input bytes, and state/debug collaborators. It returns logical
|
||||||
|
output files, normalized artifacts, recorded rejections and warnings, manifest
|
||||||
|
provenance, and checkpoint decisions. The CLI owns process arguments,
|
||||||
|
configuration discovery, physical roots, and placement of returned output
|
||||||
|
files.
|
||||||
|
|
||||||
`internal/core/config.Config.Resolve` validates the loaded configuration,
|
The framework has one fixed shape:
|
||||||
selects the named profile, applies the runtime inputs supplied by the CLI, and
|
|
||||||
calls `pipeline.ResolvePipeline`.
|
|
||||||
|
|
||||||
`ResolvePipeline`:
|
~~~
|
||||||
|
input -> chunk -> extract -> merge -> normalize -> output
|
||||||
|
~~~
|
||||||
|
|
||||||
1. selects the explicit ordered steps, or creates the implicit `default` step
|
Input and chunking are pipeline-wide. A selected artifact lane owns extract,
|
||||||
from the legacy top-level `artifacts` map;
|
merge, and normalize; output aggregates the terminal lane outcomes. A pipeline
|
||||||
2. selects and sorts artifact lanes within each step while enforcing global lane
|
is an ordered list of steps, not an arbitrary workflow graph.
|
||||||
identity;
|
|
||||||
3. completes omitted bindings using the documented configuration defaults;
|
|
||||||
4. looks up each module and validator spec without constructing it;
|
|
||||||
5. for a typed extractor, derives its artifact kind, requires the codec, and
|
|
||||||
selects exact-type merger, normalizer, and validator variants;
|
|
||||||
6. checks required and provided capabilities in workflow order;
|
|
||||||
7. resolves external and generated target-aware reference bindings and
|
|
||||||
validates producer order, consumer slot declarations, and artifact-kind
|
|
||||||
compatibility;
|
|
||||||
8. validates each selected module and validator option set through its registry
|
|
||||||
entry; and
|
|
||||||
9. calculates a digest over the resolved structure, including step order, step
|
|
||||||
IDs, lane membership, generated topology, producer and consumer identities,
|
|
||||||
typed artifact kind and schema identity, and the effective validator policy
|
|
||||||
in its resolved execution order.
|
|
||||||
|
|
||||||
Resolution returns a `ResolvedPipeline` containing ordered steps, lanes,
|
## Resolve, Materialize, Prepare
|
||||||
concrete bindings, validator chains, reference targets, and the digest. It does
|
|
||||||
not read external reference bytes or construct runtime modules. CLI lane and
|
|
||||||
reference selector syntax is defined in the [CLI reference](../cli.md#run).
|
|
||||||
|
|
||||||
The digest includes each resolved step's ID and lane membership, generated
|
Resolution turns a configured pipeline profile into a **ResolvedPipeline**.
|
||||||
producer/consumer topology, and each validator chain's stage, lane, owning
|
It normalizes the pipeline and lane identities, applies stage defaults, selects
|
||||||
module, ordered validator bindings, execution classes, targets, and artifact
|
requested lanes where that is supported, resolves validator chains, checks
|
||||||
kinds. Changing step order, a dependency, a default chain, or an explicit
|
module capabilities and typed artifact compatibility, validates options, and
|
||||||
override therefore changes pipeline identity whenever it changes effective
|
assigns a deterministic resolved-composition digest. The resolved pipeline
|
||||||
execution policy.
|
contains bindings and declared reference targets, not external reference bytes.
|
||||||
|
Configuration resolution supplies the selected profile and catalog; see
|
||||||
|
[Configuration Internals](configuration.md).
|
||||||
|
|
||||||
## Reference Materialization
|
External reference materialization happens before preparation. The materializer
|
||||||
|
checks that each slot is declared by the selected module, resolves a file path
|
||||||
|
relative to the correct configuration or working-directory origin, reads
|
||||||
|
UTF-8 text, verifies media type and size limits, and retains bounded
|
||||||
|
provenance. A generated-artifact selector remains declared but has no bytes
|
||||||
|
until its producing step completes.
|
||||||
|
|
||||||
The CLI calls `MaterializeReferences` after resolution and before constructing
|
Preparation is the construction boundary. It validates the resolved shape and
|
||||||
the LLM client or running the pipeline. For external bindings, the materializer
|
registry set, clones the resolved data, then constructs the input adapter,
|
||||||
checks each binding against its resolved target declaration, reads and validates
|
chunker, stage-local validators, every typed lane, and output encoder with
|
||||||
the file, and builds both a `contracts.ReferenceSet` and provenance-only
|
cloned options, references, and shared dependencies. It also collects stable
|
||||||
metadata on the corresponding `ResolvedReferenceTarget`. A structured
|
checkpoint fingerprints. Missing registrations, incompatible typed entries,
|
||||||
generated binding is declaration-only at this point: its producer bytes do not
|
nil implementations, and constructor failures are reported before source
|
||||||
exist until the producer lane reaches an accepted normalized result.
|
parsing or any stage operation begins.
|
||||||
|
|
||||||
Preparation delivers the materialized external set for each target through
|
## Typed Lanes And References
|
||||||
`pipeline.BuildRequest`: chunkers and chunk validators receive the chunk target;
|
|
||||||
extractors and extract validators receive the lane extract target; mergers and
|
|
||||||
merge validators receive the lane merge target; and normalizers and normalize
|
|
||||||
validators receive the lane normalize target. Input and output builders receive
|
|
||||||
an empty set because those stages cannot declare references. Every builder gets
|
|
||||||
an isolated deep clone of its target set, so construction-time mutation cannot
|
|
||||||
change another builder, the resolved pipeline, or later runtime requests.
|
|
||||||
|
|
||||||
Prepared consumers do not need to be reconstructed when generated content is
|
Each resolved lane has one artifact kind, codec, and exact Go type. The
|
||||||
available. At the step boundary, the runner encodes the accepted producer value
|
framework uses private type erasure only around those typed operations; every
|
||||||
through its registered canonical codec, validates the generated bytes against
|
handoff checks exact type and codec identity and reports incompatibility as an
|
||||||
each target slot's kind, schema, media type, and size, and clones one immutable
|
error rather than panicking. Encoding through the registered codec is the
|
||||||
reference item into the operation request. The item includes canonical digest,
|
boundary for output, checkpoints, debug records, and generated references.
|
||||||
size, and bounded producer provenance but no filesystem URI. A handoff failure
|
|
||||||
is a framework dependency error and prevents every consumer in that step from
|
|
||||||
starting.
|
|
||||||
|
|
||||||
The runner continues to clone the resulting set into the chunk, extract, merge,
|
Reference targets are stage- and lane-specific. External reference bytes are
|
||||||
or normalize request that owns the target. LLM-backed extensions may convert
|
cloned into the operation request. Generated references are built at the next
|
||||||
those items into named prompt inputs. Reference content remains separate from
|
step boundary from exactly one accepted normalized producer output. The
|
||||||
source evidence and source digests, whether the item came from a file or a
|
framework decodes and re-encodes that output with the registered producer
|
||||||
generated handoff.
|
codec, checks its complete schema and media identity, and records a content
|
||||||
|
digest plus bounded producer provenance. A missing, ambiguous, invalid, or
|
||||||
|
incompatible producer prevents the consumer step from starting.
|
||||||
|
|
||||||
Binding precedence, path resolution, accepted content, and media-type behavior
|
## Execution And Ordering
|
||||||
are configuration contracts; see [Configuration](../config.md#pipelines).
|
|
||||||
Durable provenance is defined in the
|
|
||||||
[JSON output contract](../integrations/json-output.md#manifestjson), while
|
|
||||||
runtime sensitive-data handling belongs in [Operations](../operations.md).
|
|
||||||
|
|
||||||
## Registries And Specs
|
The runner validates its input, installs no-op state collaborators when none
|
||||||
|
were supplied, and serially performs source parsing and chunk-plan selection.
|
||||||
|
An accepted plan is materialized into source-addressed chunks and passes the
|
||||||
|
configured chunk validators before any lane runs. A chunk rejection is a
|
||||||
|
recorded pipeline outcome: lanes do not start, but the output stage can encode
|
||||||
|
the terminal result.
|
||||||
|
|
||||||
`pipeline.Registries` holds option validators and run-local builders used during
|
For each ordered step, the runner first builds generated reference sets from
|
||||||
resolution and preparation.
|
the accepted normalized outputs of earlier steps. It then executes the step's
|
||||||
`pipeline.ModuleCatalog` exposes their specs during configuration validation and
|
lanes. Later steps do not begin until the current step is terminal and its
|
||||||
resolution. Separate registries exist for every stage and for validators;
|
generated handoffs have succeeded.
|
||||||
`ValidatorChainRegistry` stores production default-chain mappings. Both
|
|
||||||
containers also carry an `ArtifactCodecRegistry`. Generic registration records
|
|
||||||
one codec per stable artifact kind, validates its schema metadata and JSON
|
|
||||||
Schema, retains the exact schema digest and Go type, and safely encodes or
|
|
||||||
decodes framework-erased values with typed errors on incompatibility.
|
|
||||||
|
|
||||||
Typed extractor entries are keyed by module key and declare one artifact kind.
|
Within a step, the lane engine dispatches extraction jobs in deterministic
|
||||||
Merger, normalizer, and typed-validator variants are keyed by module or
|
chunk-first, lane-second order to a bounded worker group. When all extraction
|
||||||
validator key plus artifact kind. Chunk and serialized validators occupy
|
jobs for one lane are terminal, a bounded continuation group can run that
|
||||||
separate target namespaces; serialized registrations declare whether they
|
lane's merge and normalize work while extraction for other lanes continues.
|
||||||
support chunks, artifacts, or both. Duplicate variants and exact Go-type
|
The framework does not create an unbounded goroutine per chunk or lane.
|
||||||
mismatches are rejected deterministically.
|
|
||||||
|
|
||||||
Lane-sensitive merger and normalizer spec discovery always supplies the
|
Completion timing does not determine public results. The coordinator restores
|
||||||
extractor's artifact kind, so variants under one reusable key may declare
|
lane and chunk order before merging results, and selects a framework error by
|
||||||
different capabilities and reference slots. Kind-neutral registry inspection
|
stable stage, lane, and chunk position. A validator rejection records a lane
|
||||||
selects the first registered artifact kind in sorted order.
|
outcome without cancelling unrelated work. A framework error or parent
|
||||||
|
cancellation cancels derived work, prevents queued work from starting, waits
|
||||||
|
for started workers, and prevents output encoding.
|
||||||
|
|
||||||
Production composition registers the D&D spell-list, NPC-list, combat-turn-list,
|
## Validation, Retries, And Output
|
||||||
NPC-interaction-list, and scene-description-list codecs and typed lane
|
|
||||||
variants, plus serialized JSON validators. Every artifact lane resolves through
|
|
||||||
the typed registries and a matching codec.
|
|
||||||
|
|
||||||
A `ModuleSpec` declares its stage plus required and provided capabilities.
|
Every chunk, extract, merge, and normalize candidate passes its resolved
|
||||||
Chunk, extract, merge, and normalize specs may also declare reference slots.
|
validator chain. Validators receive immutable canonical input appropriate to
|
||||||
Registry implementations defensively copy spec metadata, reject duplicate keys,
|
their target: chunks, typed values, or serialized codec bytes. They may
|
||||||
and verify that a constructed implementation reports the registered key.
|
approve, approve with warnings, reject, or fail. A rejection is an ordinary
|
||||||
Builder registrations accept `ModuleDependencies` and cloned configuration
|
pipeline result; a validator error is a framework error.
|
||||||
options through one `BuildRequest`. Builders decode those options and retain
|
|
||||||
typed values or injected dependencies in the constructed implementation.
|
|
||||||
Extractors declare their artifact kind, and merger, normalizer, and validator
|
|
||||||
resolution selects the matching typed variant.
|
|
||||||
|
|
||||||
A `ValidatorSpec` declares a validator key and execution class. Resolution uses
|
The runner applies the binding's retry policy around a stage operation and its
|
||||||
the execution class to reject incompatible profile bindings before execution.
|
complete validation chain. It preserves warnings only from the final accepted
|
||||||
The current production catalog and default chain are listed only in
|
or rejected attempt. Cancellation stops retries. Normalizer-specific retry
|
||||||
[Configuration](../config.md#implemented-production-validators).
|
directives consume this same budget and validate any final safe fallback through
|
||||||
|
the normalizer chain.
|
||||||
|
|
||||||
## Preparation And Runner Boundary
|
After terminal lane work, the runner assembles manifest provenance, normalized
|
||||||
|
artifacts, rejections, warnings, and an optional accepted chunk map. The output
|
||||||
`pipeline.Prepare` receives a resolved pipeline, the registries, and shared
|
encoder returns logical files; it does not choose a physical directory. The CLI
|
||||||
module dependencies. It constructs input; chunk and its validators; every
|
publishes those files only after the runner returns without a framework error.
|
||||||
step's lane extract, merge, and normalize modules and validator chains in
|
Logical file names and schemas are defined by the
|
||||||
resolved order; then output. It stops at the first error with pipeline, step,
|
[output integration contracts](../integrations/).
|
||||||
stage, lane, module, and validator context as applicable. It never invokes an
|
|
||||||
operation method. Generated references are not available during preparation;
|
|
||||||
the operation request is the handoff boundary.
|
|
||||||
|
|
||||||
`PreparedPipeline` keeps private constructed executors and exposes cloned
|
|
||||||
resolved input, chunk, lane, and output identities. Prepared components may
|
|
||||||
implement `pipeline.CheckpointFingerprintProvider` to contribute explicit
|
|
||||||
semantic identities to checkpoint reuse. Preparation trims and validates each
|
|
||||||
non-secret name and value, prefixes it with the component's stage, lane,
|
|
||||||
module, and validator scope, rejects duplicates, and retains the resulting
|
|
||||||
sorted collection behind a defensive-copy accessor. Fingerprints must be
|
|
||||||
stable and must not contain source content, credentials, local paths,
|
|
||||||
timestamps, or other invocation-specific values.
|
|
||||||
|
|
||||||
`pipeline.RunInput` carries that prepared pipeline, raw source input, run identity and timing, optional
|
|
||||||
session and profile metadata, a chunk-plan store and mode, a checkpoint
|
|
||||||
execution policy, and checkpoint/debug collaborators. The runner
|
|
||||||
parses source bytes through the already constructed input adapter. Later stage
|
|
||||||
requests receive the generic source model; extract requests receive
|
|
||||||
chunk-scoped input material, while chunk, merge, and normalize requests retain
|
|
||||||
access to the original source material. Input, chunk, and output operation
|
|
||||||
requests do not carry raw module options. The chunk request also does not carry
|
|
||||||
an LLM client; an LLM-backed chunker receives the shared client during
|
|
||||||
preparation. Their operation requests retain run-specific source, reference,
|
|
||||||
profile, session, metadata, and step-handoff context as applicable. A generated
|
|
||||||
reference is cloned into each compatible consumer request and is never exposed
|
|
||||||
as a path.
|
|
||||||
|
|
||||||
Prepared lanes retain exact-type-checked erased operation closures. The runner
|
|
||||||
uses those closures to keep each value typed through extraction, validation,
|
|
||||||
merge, and normalization.
|
|
||||||
|
|
||||||
Source validation requires every unit to carry a canonical self-reference to
|
|
||||||
its containing document and its own unit ID. Explicit clone, checkpoint, and
|
|
||||||
debug boundaries retain that reference, and the canonical source digest covers
|
|
||||||
it deterministically. Chunks use the same source model and carry one canonical
|
|
||||||
reference spanning the first selected unit through the last.
|
|
||||||
|
|
||||||
`pipeline.RunOutput` carries the run manifest, accepted normalized serialized
|
|
||||||
artifacts with lane and normalizer provenance,
|
|
||||||
rejected results, warnings, checkpoint events, and logical files returned by the
|
|
||||||
output encoder. The CLI owns debug-summary and durable filesystem writes after
|
|
||||||
the runner returns.
|
|
||||||
|
|
||||||
## Execution Flow
|
|
||||||
|
|
||||||
The pipeline-wide coordinator owns the ordered step loop, generated-reference
|
|
||||||
sets at each barrier, and deterministic merging of step outcomes. For one step,
|
|
||||||
one run-local lane engine owns worker lifecycle, cancellation, dispatch,
|
|
||||||
continuation queues, and result collection. It initializes checkpoint state in
|
|
||||||
lane order, dispatches bounded extract work, advances terminal lanes through
|
|
||||||
serial merge and normalize work, selects failures by stable pipeline scope, and
|
|
||||||
merges lane-local outcomes back in resolved order. Completion timing never
|
|
||||||
becomes public ordering.
|
|
||||||
|
|
||||||
The runner:
|
|
||||||
|
|
||||||
1. validates its prepared input;
|
|
||||||
2. parses the raw input with the prepared adapter and validates the generic
|
|
||||||
source document;
|
|
||||||
3. selects a stored plan or executes the configured chunker's `Plan` operation;
|
|
||||||
4. canonicalizes and materializes the plan, then validates the resulting
|
|
||||||
chunks;
|
|
||||||
5. builds the framework-owned accepted chunk map from the accepted source,
|
|
||||||
logical plan, and exact materialized chunks, then supplies it to the output
|
|
||||||
request independently of output-module options;
|
|
||||||
6. executes each resolved step in configuration order. For one step, it
|
|
||||||
dispatches extract jobs in source-chunk then resolved-lane order, starts a
|
|
||||||
bounded lane continuation when all extracts for that lane are terminal, and
|
|
||||||
waits for every lane to become terminal;
|
|
||||||
7. encodes and validates each accepted normalized producer artifact, then
|
|
||||||
builds the immutable generated reference sets for the next step;
|
|
||||||
8. invokes the prepared output encoder only after every step succeeds and
|
|
||||||
validates its logical file results;
|
|
||||||
9. returns the assembled manifest, outcomes, warnings, and files.
|
|
||||||
|
|
||||||
Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
|
|
||||||
and validators while performing these transitions:
|
|
||||||
|
|
||||||
1. extract once per accepted chunk and add runner-owned lane, source, and chunk
|
|
||||||
provenance;
|
|
||||||
2. validate each extract result and omit rejected results from merge input;
|
|
||||||
3. skip the rest of the lane when no extract result is accepted;
|
|
||||||
4. merge accepted extract results in their existing order;
|
|
||||||
5. validate the merge result and skip normalization on rejection;
|
|
||||||
6. normalize the accepted merge result;
|
|
||||||
7. validate and append the accepted normalized result.
|
|
||||||
|
|
||||||
At a step barrier, a lane with no accepted normalized output is still a regular
|
|
||||||
rejection unless a later generated binding names that lane as a required
|
|
||||||
producer. In that case the runner raises a deterministic dependency error and
|
|
||||||
does not start the consumer step. One accepted typed artifact may fan out to
|
|
||||||
multiple compatible target slots. Consumers in the same step may run
|
|
||||||
concurrently after the handoff; no work crosses the barrier early.
|
|
||||||
|
|
||||||
Module-provided warnings and payload warnings are promoted only from attempts
|
|
||||||
whose results are accepted and used.
|
|
||||||
|
|
||||||
## Chunk Plans And Reuse
|
|
||||||
|
|
||||||
`Chunker.Plan` returns a `source.ChunkPlan`: the canonical source digest,
|
|
||||||
ordered unit-ID ranges, and optional plan or range annotations. The framework
|
|
||||||
owns plan canonicalization and materialization. It creates the generic chunks
|
|
||||||
and therefore owns their IDs, indexes, source references, JSON content, units,
|
|
||||||
media type, and generic metadata. Plan and range annotations are independently
|
|
||||||
owned raw JSON and become `Chunk.PlanAnnotations` and `Chunk.Annotations`.
|
|
||||||
|
|
||||||
In `auto`, the runner looks up the source digest before invoking the chunker. A
|
|
||||||
valid hit is materialized and sent through the current run's configured chunk
|
|
||||||
validators; it does not invoke the chunk module, consume its retry budget, or
|
|
||||||
make a chunk-stage LLM call. A missing, invalid, or unmaterializable record
|
|
||||||
generates a candidate. `refresh` generates without lookup; `bypass` generates
|
|
||||||
without cache access. Generated plans are published only after the full chunk
|
|
||||||
validator chain approves them. A validator rejection is a regular rejected
|
|
||||||
pipeline outcome and never replaces a cached plan.
|
|
||||||
|
|
||||||
The store is source-addressed, not pipeline-addressed. Changes to pipeline
|
|
||||||
configuration, requested chunker, options, references, lanes, validators, or
|
|
||||||
LLM profile do not prevent a source-digest hit. The manifest records both the
|
|
||||||
currently requested chunker and the effective plan producer. Cache state and
|
|
||||||
paths are configured and operated outside the runner; see
|
|
||||||
[Configuration](../config.md#state-surfaces) and [Operations](../operations.md).
|
|
||||||
|
|
||||||
For an accepted plan, the runner also constructs the strict framework-owned
|
|
||||||
[Accepted Chunk Map](../integrations/chunk-map.md) before lane execution. It
|
|
||||||
uses the current resolved chunker as `requested_chunker` and the stored or
|
|
||||||
generated record as `producer`, preserving that distinction on reuse. Chunk
|
|
||||||
rejection supplies no map; later lane rejection does not discard it. Output
|
|
||||||
encoders receive a defensively owned serialized value and may explicitly
|
|
||||||
ignore it.
|
|
||||||
|
|
||||||
The extract job channel has the same capacity as the effective extract worker
|
|
||||||
count, so dispatch applies backpressure. A fixed continuation executor prevents
|
|
||||||
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
|
|
||||||
continuations publish lane-local results; the coordinator is the only writer of
|
|
||||||
aggregate output and merges those results in resolved lane and source-chunk
|
|
||||||
order.
|
|
||||||
|
|
||||||
## Plan Canonicalization And Chunk Materialization
|
|
||||||
|
|
||||||
Plan canonicalization requires canonical JSON annotations, a matching source
|
|
||||||
digest, at least one range, existing ordered boundaries, and increasing range
|
|
||||||
starts. Ranges may overlap or leave gaps; a chunker may impose stricter policy.
|
|
||||||
Materialization deterministically reconstructs each range from the current
|
|
||||||
source document, deep-clones JSON-shaped source-unit metadata, and copies
|
|
||||||
annotations without interpreting their namespaces. Materialized chunks and
|
|
||||||
separate materializations do not share mutable unit metadata; unsupported or
|
|
||||||
cyclic metadata fails materialization with context.
|
|
||||||
|
|
||||||
Before lane execution, generic chunk validation checks the materialized chunks'
|
|
||||||
identities, order, source references, content, media type, units, and metadata.
|
|
||||||
No chunk checkpoint participates in plan selection: plan storage is the only
|
|
||||||
chunk-reuse mechanism. Extract, merge, and normalize checkpoints continue to
|
|
||||||
use materialized chunk digests as their dependencies.
|
|
||||||
|
|
||||||
## Validation And Retries
|
|
||||||
|
|
||||||
Chunk, extract, merge, and normalize results pass through the resolved validator
|
|
||||||
chain for their stage and module. Chunk validators receive canonical chunks;
|
|
||||||
typed validators receive the domain value; and serialized validators receive
|
|
||||||
canonical chunk JSON or artifact codec bytes. Validators execute in resolved
|
|
||||||
order and stop at the first error or rejection. An empty chain approves the
|
|
||||||
result.
|
|
||||||
|
|
||||||
Production D&D artifact chains keep generic JSON syntax validation first, then
|
|
||||||
run every rejecting domain validator before generic JSON Schema validation. The
|
|
||||||
domain validator therefore owns expected semantic diagnostics; the generic
|
|
||||||
schema validator remains the final rejecting representation backstop, before
|
|
||||||
warning-only relatedness validation. Explicitly configured validator chains
|
|
||||||
retain their configured order.
|
|
||||||
|
|
||||||
`runWithRetry` applies the effective retry policy around module execution and
|
|
||||||
its complete validation chain. A module or validator error becomes a framework
|
|
||||||
error when attempts are exhausted. A rejection becomes a recorded
|
|
||||||
`RejectedOutput` when attempts are exhausted. Cancellation stops retry
|
|
||||||
processing immediately.
|
|
||||||
|
|
||||||
Structured-completion adapters classify malformed or undecodable provider
|
|
||||||
output with the provider-neutral `contracts.ErrInvalidStructuredOutput` error.
|
|
||||||
A typed normalizer may turn that condition, or another unsafe proposal, into a
|
|
||||||
normalize retry directive with a module-supplied safe candidate, stable
|
|
||||||
diagnostic, and fallback warnings. The directive consumes the same configured
|
|
||||||
normalize retry budget: `retries` permits that many additional attempts after
|
|
||||||
the initial attempt. It neither creates a normalizer-local retry loop nor
|
|
||||||
records an accepted checkpoint for the discarded attempt.
|
|
||||||
|
|
||||||
Before adding a normalize retry directive to attempt debug data, the runner
|
|
||||||
requires a nonblank, valid UTF-8 reason code of at most 128 bytes and a
|
|
||||||
nonblank, valid UTF-8 message of at most 4,096 bytes. These are encoded-byte
|
|
||||||
limits. The framework rejects an invalid directive without truncating or
|
|
||||||
rewriting either field. It validates only this mechanical contract; normalizers
|
|
||||||
remain responsible for ensuring their otherwise valid diagnostics do not expose
|
|
||||||
source material, credentials, paths, names, or other sensitive content.
|
|
||||||
|
|
||||||
The framework treats a module-supplied candidate as opaque. The normalizer owns
|
|
||||||
its safety determination, and the configured normalizer validator chain remains
|
|
||||||
the acceptance boundary for the final fallback.
|
|
||||||
|
|
||||||
When a later normalize attempt succeeds, its candidate alone proceeds through
|
|
||||||
the usual validation and checkpoint path. When the final attempt still returns
|
|
||||||
a directive, the runner validates its supplied safe fallback through that same
|
|
||||||
normalizer validator chain before accepting or rejecting it. Ordinary
|
|
||||||
attempt-local warnings and fallback warnings remain unpromoted while another
|
|
||||||
attempt is available; only final exhaustion promotes the supplied fallback
|
|
||||||
warnings. Rejected output is a non-fatal pipeline outcome and does not advance.
|
|
||||||
Configuration owns retry counts and validator overrides; see
|
|
||||||
[Module Bindings](../config.md#module-bindings).
|
|
||||||
|
|
||||||
## Checkpoint And Debug Hooks
|
## Checkpoint And Debug Hooks
|
||||||
|
|
||||||
The runner depends on recorder and loader interfaces, using no-op
|
The runner receives checkpoint and debug interfaces rather than roots. It
|
||||||
implementations when collaborators are absent. Each checkpointed workflow
|
records workflow transitions and reuse decisions through the supplied
|
||||||
boundary records a running, succeeded, or failed transition. Reuse decisions
|
collaborators, and clones reusable artifacts before they re-enter normal typed
|
||||||
are consulted in workflow order and accepted payloads are cloned before
|
handoff. Generated-reference dependencies participate in checkpoint decisions.
|
||||||
entering the normal handoff path. Typed extract, merge, and normalize
|
Selective recomputation can require a canonical accepted normalized predecessor
|
||||||
checkpoints store codec bytes with artifact kind, schema ID, name, version and
|
before a dependent lane starts.
|
||||||
exact digest, and media type. Reuse compares that identity with the prepared
|
|
||||||
codec and decodes through the codec; missing identity, mismatches, corrupt
|
|
||||||
bytes, and decode failures become explicit reuse misses and execute the lane
|
|
||||||
normally. Dependency fingerprints and debug content digests use the same stable
|
|
||||||
codec bytes that cross those boundaries.
|
|
||||||
|
|
||||||
That progressive extract, merge, and normalize reuse is the ordinary resume
|
Debug recording is attempt-scoped and application-owned. A failure to persist
|
||||||
path. A lane marked as a required predecessor for selective recomputation takes
|
required debug data is a framework error. State roots, persistence, reason-code
|
||||||
a separate accepted-output path before extract scheduling. The loader reads the
|
meanings, resume, and cleanup are intentionally owned by
|
||||||
existing successful normalize manifest and payload by step, lane, and
|
[Run State Internals](state.md) and [Operations](../operations.md).
|
||||||
normalizer, without consulting extract or merge dependencies. It requires the
|
|
||||||
current non-empty checkpoint identity to match, so the invocation identity
|
|
||||||
still binds the input, resolved topology and configuration, references, runtime
|
|
||||||
overrides, profiles, and component fingerprints.
|
|
||||||
|
|
||||||
The runner decodes and canonically re-encodes each reusable artifact once with
|
## Invariants To Preserve
|
||||||
the prepared codec, requiring exact kind, schema identity and digest, media
|
|
||||||
type, canonical bytes, content digest, and producer provenance. A valid accepted
|
|
||||||
producer becomes a runner-owned cloned normalized output, restores only
|
|
||||||
normalize-checkpoint warnings, and records one `accepted_artifact_reused`
|
|
||||||
normalize decision. It does not invoke or record extract, merge, normalize, or
|
|
||||||
their validators. Invalid or unavailable accepted state records its decision
|
|
||||||
and fails the producer step; the dependent step never starts and the producer
|
|
||||||
is not implicitly rerun. If a later required lane fails during initialization,
|
|
||||||
already hydrated terminal lanes remain in the failed output in resolved order.
|
|
||||||
|
|
||||||
Generated references add downstream dependencies containing the producer's
|
- The six fixed stages remain explicit; a pipeline is not a general DAG.
|
||||||
artifact kind, complete schema identity, media type, canonical content digest,
|
- Resolution and preparation reject statically discoverable incompatibility
|
||||||
and size. Compatible accepted producer outputs may therefore feed a later step
|
before parsing or execution.
|
||||||
without re-executing the producer. Forced lanes bypass accepted-output
|
- Every typed lane uses one compatible artifact kind, codec, and exact Go type.
|
||||||
hydration and execute normally. A missing, rejected, corrupt, incompatible, or
|
- Generated references come only from one earlier accepted normalized producer
|
||||||
changed producer blocks its dependent while leaving independent work eligible
|
and carry canonical identity rather than an unverified value.
|
||||||
for reuse. The runner records bounded decision
|
- Rejections are recorded outcomes; framework errors cancel derived work and
|
||||||
categories: `reused`, `executed`, `forced_recompute`, and
|
prevent output encoding.
|
||||||
`dependency_invalidated`. Operator meanings for the stable reason codes belong
|
- Public ordering and selected errors are independent of goroutine completion
|
||||||
to [Operations](../operations.md#resume-and-selective-recompute).
|
order.
|
||||||
|
- Pipeline modules receive collaborators and data, never CLI streams or
|
||||||
|
physical output, cache, or debug roots.
|
||||||
|
|
||||||
The CLI includes prepared-component fingerprints in the run-wide checkpoint
|
## Focused Tests
|
||||||
identity alongside resolved configuration, raw input, reference provenance,
|
|
||||||
runtime overrides, and LLM-profile fingerprints. Module metadata is not used
|
|
||||||
implicitly for cache identity: components opt in only with stable semantic
|
|
||||||
values that can change accepted output. Adding or changing a component
|
|
||||||
fingerprint intentionally produces a cold cache miss. Existing checkpoint
|
|
||||||
schemas and paths remain unchanged.
|
|
||||||
|
|
||||||
The CLI's `--recompute-step` policy forces the selected step and all transitive
|
- **internal/framework/pipeline/profile_test.go** and
|
||||||
dependents, but requires accepted normalized artifacts for every unselected
|
**typed_resolution_test.go** cover resolution, defaults, ordered steps,
|
||||||
producer on which that closure depends. It changes execution policy only; it
|
compatibility, validators, references, and resolved identity.
|
||||||
does not alter persistent checkpoint identity.
|
- **internal/framework/pipeline/preparation_test.go** covers complete
|
||||||
|
construction before execution and contextual construction failures.
|
||||||
|
- **internal/framework/pipeline/references_test.go** and **handoff_test.go**
|
||||||
|
cover external materialization, generated references, provenance, and typed
|
||||||
|
producer checks.
|
||||||
|
- **internal/framework/pipeline/runner_concurrency_test.go** covers bounded
|
||||||
|
execution, ordered steps, stable error selection, rejections, and
|
||||||
|
cancellation.
|
||||||
|
- **internal/framework/pipeline/runner_chunk_plan_test.go**,
|
||||||
|
**runner_typed_checkpoint_test.go**, and
|
||||||
|
**runner_accepted_checkpoint_test.go** cover state hooks and reuse behavior.
|
||||||
|
- **internal/framework/pipeline/runner_attempt_debug_test.go** and
|
||||||
|
**runner_terminal_debug_test.go** cover attempt and terminal debug behavior.
|
||||||
|
|
||||||
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
|
Run **go test ./internal/framework/pipeline ./internal/cli** after changing a
|
||||||
boundaries. Every executed chunk, extract, merge, and normalize attempt writes
|
pipeline boundary. Use the more focused tests above while iterating.
|
||||||
one terminal envelope for acceptance, validator rejection, module or validator
|
|
||||||
error, or applicable candidate or final serialization error. The envelope
|
|
||||||
contains its attempt-local warnings, any available candidate and rejection,
|
|
||||||
and terminal error text; normalize retry directives retain their attempt-local
|
|
||||||
candidate and diagnostic, while only the final safe fallback reaches validation.
|
|
||||||
Failures before a candidate exists omit that payload.
|
|
||||||
Only LLM calls made by the module operation belong to the module attempt.
|
|
||||||
Validator calls retain independent scopes under `validate/` and are not
|
|
||||||
duplicated into the module envelope. A failed terminal-envelope write is a
|
|
||||||
non-retryable framework error and is joined with any primary attempt error.
|
|
||||||
Debug data is never used as a checkpoint source. Typed artifact debug envelopes
|
|
||||||
are domain-neutral, redact sensitive metadata and bytes through the common
|
|
||||||
debug policy, and record codec identity plus schema and content digests.
|
|
||||||
|
|
||||||
Merge and normalize attempts serialize their in-memory candidate with the
|
|
||||||
codec's required candidate encoder before typed validation. Serialized
|
|
||||||
validators and attempt debug use that candidate representation, which carries
|
|
||||||
the codec media type and schema identity but is never checkpointed or passed
|
|
||||||
downstream. Only a validator-approved value is encoded through the strict final
|
|
||||||
codec and made eligible for a checkpoint or stage output.
|
|
||||||
|
|
||||||
Checkpoint identity, physical layout, reuse behavior, and debug artifact
|
|
||||||
handling are operator contracts in [Operations](../operations.md). Serialization
|
|
||||||
and recorder implementation are inventoried in
|
|
||||||
[Internal Overview](overview.md#run-state-components).
|
|
||||||
|
|
||||||
## Results And Failures
|
|
||||||
|
|
||||||
The runner owns manifest assembly and handoff summaries but not the durable JSON
|
|
||||||
schema. It records resolved module and lane provenance, validator chains,
|
|
||||||
source/reference identities, selected LLM profiles, normalized and rejected
|
|
||||||
summaries, status, and timing. Serialized artifact content remains outside the manifest.
|
|
||||||
Module metadata providers may add non-secret singleton or lane-scoped metadata.
|
|
||||||
|
|
||||||
Execution errors include stage, module, lane, or validator context. Once a
|
|
||||||
manifest exists, a failing run returns it with failed status and completion
|
|
||||||
time. Successful status reflects whether any result was rejected. The
|
|
||||||
durable manifest and logical file schemas are defined in the
|
|
||||||
[JSON output contract](../integrations/json-output.md).
|
|
||||||
|
|
||||||
On a framework failure, the runner cancels its derived context, stops submitting
|
|
||||||
new extract work, drains started tasks, and skips the output encoder. Parent
|
|
||||||
cancellation takes precedence. Otherwise context-cancellation fallout is
|
|
||||||
discarded when a substantive error exists, and the primary error is selected by
|
|
||||||
stage, resolved lane, and source chunk rather than completion time.
|
|
||||||
|
|
||||||
## Tests To Inspect
|
|
||||||
|
|
||||||
- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
|
|
||||||
- `internal/framework/pipeline/profile_test.go`: selection, defaults,
|
|
||||||
capabilities, validator chains, and digest behavior.
|
|
||||||
- `internal/framework/pipeline/artifact_codec_registry_test.go`: typed codec
|
|
||||||
metadata, registration, erasure safety, strict decoding, and cloning.
|
|
||||||
- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed
|
|
||||||
lane resolution and preparation, target-specific validators,
|
|
||||||
incompatibilities, ordering, and schema-sensitive pipeline identity.
|
|
||||||
- `internal/framework/pipeline/runner_concurrency_test.go`: bounded dispatch and
|
|
||||||
continuations, reverse completion, stable errors, rejection, cancellation,
|
|
||||||
retries, and independent provider-call limits.
|
|
||||||
- `internal/framework/pipeline/preparation_test.go`: option validation,
|
|
||||||
construction order, dependency failures, and the before-source-work boundary.
|
|
||||||
- `internal/framework/pipeline/references_test.go`: target resolution and
|
|
||||||
materialization.
|
|
||||||
- `internal/cli/run_contract_test.go`: production run transitions, retries,
|
|
||||||
rejections, warnings, CLI recomputation controls, debug hooks, and manifests.
|
|
||||||
- `internal/cli/recompute_execution_contract_test.go`: filesystem-backed
|
|
||||||
selective recomputation and accepted-producer recovery.
|
|
||||||
- `internal/cli/production_contract_test.go`: production composition and
|
|
||||||
configuration-resolution smoke coverage.
|
|
||||||
- `internal/cli/example_contract_test.go`: maintained example resolution and
|
|
||||||
execution ownership.
|
|
||||||
- `internal/modules/integration/*_test.go` and
|
|
||||||
`internal/modules/seriatim/input/transcript/runner_test.go`: typed runner
|
|
||||||
composition across concrete module families.
|
|
||||||
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse
|
|
||||||
collaborators.
|
|
||||||
|
|||||||
@@ -2,7 +2,8 @@
|
|||||||
|
|
||||||
This document describes the implementation collaborators behind output, cache,
|
This document describes the implementation collaborators behind output, cache,
|
||||||
and debug state. User-visible fields belong in [Configuration](../config.md),
|
and debug state. User-visible fields belong in [Configuration](../config.md),
|
||||||
and layouts and lifecycle belong in [Operations](../operations.md).
|
and physical layout, retention, recovery, reason codes, and cleanup belong in
|
||||||
|
[Operations](../operations.md).
|
||||||
|
|
||||||
## Composition
|
## Composition
|
||||||
|
|
||||||
@@ -12,6 +13,12 @@ constructs cache collaborators, writes logical output files, and reports paths.
|
|||||||
Pipeline modules receive interfaces and request data, never output, cache, or
|
Pipeline modules receive interfaces and request data, never output, cache, or
|
||||||
debug roots.
|
debug roots.
|
||||||
|
|
||||||
|
The CLI creates no chunk-plan store in bypass mode. It creates a checkpoint
|
||||||
|
recorder only when recording is enabled and a checkpoint loader only for a
|
||||||
|
resume invocation. It allocates debug state only after a safe run identity has
|
||||||
|
been generated and only when debug capture was requested. These choices keep
|
||||||
|
the three state families independently composable.
|
||||||
|
|
||||||
## Output And Cache
|
## Output And Cache
|
||||||
|
|
||||||
The pipeline runner returns logical output files. After validating every
|
The pipeline runner returns logical output files. After validating every
|
||||||
@@ -68,12 +75,19 @@ names only the step, lane, and stable reason code. Decision detail is selected
|
|||||||
from code-owned descriptions by reason code and then UTF-8 normalized and
|
from code-owned descriptions by reason code and then UTF-8 normalized and
|
||||||
bounded; callers cannot supply arbitrary diagnostic prose. Typed categories and
|
bounded; callers cannot supply arbitrary diagnostic prose. Typed categories and
|
||||||
codes remain intact through pipeline events and become strings only in manifest
|
codes remain intact through pipeline events and become strings only in manifest
|
||||||
and debug-summary JSON. [Operations](../operations.md#resume-and-selective-recompute)
|
and debug-summary JSON.
|
||||||
is the canonical operator-facing reason-code reference.
|
[Operations](../operations.md#checkpoint-recording-resume-and-recompute) is the
|
||||||
|
canonical operator-facing reason-code reference.
|
||||||
|
|
||||||
`internal/core/fileio` provides confined atomic file writes used by state
|
`internal/core/fileio` provides confined atomic file writes used by state
|
||||||
collaborators. The chunk-plan store retains its stronger entry validation.
|
collaborators. The chunk-plan store retains its stronger entry validation.
|
||||||
|
|
||||||
|
The CLI constructs selective-recomputation policy from resolved generated
|
||||||
|
artifact dependencies. It forces the selected step and transitive consumers,
|
||||||
|
while marking unforced producers as required reusable inputs. The runner owns
|
||||||
|
the actual hydration and rejection decisions; the [Operations guide](../operations.md#checkpoint-recording-resume-and-recompute)
|
||||||
|
owns the operator workflow and stable reason-code meanings.
|
||||||
|
|
||||||
## Debug Bundles
|
## Debug Bundles
|
||||||
|
|
||||||
`internal/core/debugbundle` allocates an explicitly requested per-run bundle
|
`internal/core/debugbundle` allocates an explicitly requested per-run bundle
|
||||||
@@ -98,12 +112,27 @@ operation writes the success report, or makes one attempt each to write the
|
|||||||
failure report and error log. Terminal persistence failures are reported
|
failure report and error log. Terminal persistence failures are reported
|
||||||
separately and never replace the command's primary error.
|
separately and never replace the command's primary error.
|
||||||
|
|
||||||
|
## Invariants To Preserve
|
||||||
|
|
||||||
|
- Modules receive state collaborators and request data, never physical roots.
|
||||||
|
- Output logical paths are validated before a run directory is allocated, and
|
||||||
|
files are atomically written within that directory.
|
||||||
|
- Chunk-plan publication occurs only for accepted plans; bypass does not
|
||||||
|
construct or touch a plan store.
|
||||||
|
- Checkpoint recording and checkpoint loading remain separate collaborators.
|
||||||
|
- Debug state is opt-in, is not cache input, and terminal reporting does not
|
||||||
|
obscure the command's primary failure.
|
||||||
|
|
||||||
## Tests To Inspect
|
## Tests To Inspect
|
||||||
|
|
||||||
- `internal/cli/run_contract_test.go`: command-owned state allocation,
|
- `internal/cli/run_contract_test.go`: command-owned state allocation,
|
||||||
terminalization, and output/report boundaries.
|
terminalization, and output/report boundaries.
|
||||||
|
- `internal/cli/cache_contract_test.go`: cache-mode precedence, root selection,
|
||||||
|
and resume collaborator construction.
|
||||||
- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
|
- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
|
||||||
permissions, cleanup, and redaction.
|
permissions, cleanup, and redaction.
|
||||||
|
- `internal/cli/recompute_policy_test.go`: forced dependents and required
|
||||||
|
reusable predecessors for selective recomputation.
|
||||||
- `internal/cli/recompute_execution_contract_test.go`: selective recomputation,
|
- `internal/cli/recompute_execution_contract_test.go`: selective recomputation,
|
||||||
filesystem recovery, deterministic decisions, and failed predecessor state.
|
filesystem recovery, deterministic decisions, and failed predecessor state.
|
||||||
- `internal/cli/production_contract_test.go`: production composition and
|
- `internal/cli/production_contract_test.go`: production composition and
|
||||||
|
|||||||
@@ -1,322 +1,206 @@
|
|||||||
# Operations
|
# Operations
|
||||||
|
|
||||||
This is the canonical guide to operating Notarius filesystem state. Command
|
This is the canonical guide for operating Notarius runtime state. The
|
||||||
syntax is in the [CLI reference](cli.md); field definitions and precedence are
|
[CLI reference](cli.md) owns command syntax and exit statuses, while
|
||||||
in [Configuration](config.md).
|
[Configuration](config.md) owns fields, defaults, and precedence. Maintainers
|
||||||
|
who need implementation mechanics should read [Run State Internals](internal/state.md).
|
||||||
|
|
||||||
## State Model
|
## State Surfaces
|
||||||
|
|
||||||
Notarius uses three independent filesystem surfaces:
|
Each run can use independent roots with different retention and access-control
|
||||||
|
needs.
|
||||||
|
|
||||||
- output is durable user data;
|
| Surface | Purpose | Created when | Retention |
|
||||||
- cache is reconstructible chunk-plan and checkpoint state; and
|
| --- | --- | --- | --- |
|
||||||
- debug is explicitly requested inspection data.
|
| Output | Durable user-facing result bundle | A pipeline completes and returns logical output files | Keep until consumers no longer need it. |
|
||||||
|
| Chunk-plan cache | Reconstructible source-addressed plan | The configured cache mode permits cache I/O | Keep while reuse is useful. |
|
||||||
|
| Checkpoint cache | Reconstructible execution and recovery state | Checkpoint recording is enabled | Keep only while recovery or reuse is useful. |
|
||||||
|
| Debug bundle | Explicit diagnostic record | A run requests debug collection | Keep only under an intentional sensitive-data retention policy. |
|
||||||
|
|
||||||
Choose separate roots and access controls for each surface. A normal run writes
|
Output, cache, and debug roots are never merged or cleaned automatically. Use
|
||||||
durable output, may use the chunk-plan cache, and records checkpoints when
|
separate locations and permissions for operators or services that must not
|
||||||
`cache.checkpoints.enabled` is true. It does not create debug state unless its
|
share application data.
|
||||||
invocation includes `--debug`.
|
|
||||||
|
|
||||||
## Output
|
## Roots And Permissions
|
||||||
|
|
||||||
Durable logical files are written under:
|
The configured output and debug directories are exact roots. An empty cache
|
||||||
|
directory selects a per-user root:
|
||||||
|
|
||||||
```text
|
~~~
|
||||||
|
<os.UserCacheDir>/notarius/chunk-plans
|
||||||
|
<os.UserCacheDir>/notarius/checkpoints
|
||||||
|
~~~
|
||||||
|
|
||||||
|
The field definitions and configuration examples are in [Configuration](config.md).
|
||||||
|
On supported Unix systems, output directories and files are created with
|
||||||
|
requested modes **0755** and **0644**. Chunk-plan, checkpoint, and debug
|
||||||
|
directories and files use **0700** and **0600**. The operating system's umask
|
||||||
|
may impose stricter output modes. Cache and debug roots may contain sensitive
|
||||||
|
source-derived data, so provision them for one trusted account or service.
|
||||||
|
|
||||||
|
## Run Lifecycle
|
||||||
|
|
||||||
|
Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads
|
||||||
|
and resolves configuration before module preparation and source parsing. It
|
||||||
|
then performs any permitted cache lookup, executes the pipeline, and publishes
|
||||||
|
logical output files only after a successful runner result.
|
||||||
|
|
||||||
|
On success, the command reports the output bundle path. A warning-bearing run
|
||||||
|
still succeeds and reports its warning count on standard error. Errors and
|
||||||
|
their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses).
|
||||||
|
|
||||||
|
## Output Bundles
|
||||||
|
|
||||||
|
Each successful run receives a generated safe run identifier and writes beneath:
|
||||||
|
|
||||||
|
~~~
|
||||||
<output-root>/<run-id>/
|
<output-root>/<run-id>/
|
||||||
```
|
~~~
|
||||||
|
|
||||||
The CLI generates one run ID in the form
|
The [JSON output contract](integrations/json-output.md) owns the logical files
|
||||||
`run-<started-at-unix-nanoseconds>-<32-lowercase-hex-characters>` and uses it
|
and their schemas. Before creating the run directory, Notarius validates every
|
||||||
for output, manifests, and any requested debug bundle. It validates every
|
logical output path. It refuses an existing run directory without changing it.
|
||||||
logical output name before exclusively creating the run directory. If that
|
Files are written atomically; if a later write fails, the newly created partial
|
||||||
directory already exists, the invocation fails without changing it.
|
run directory remains for inspection and is never removed automatically.
|
||||||
|
|
||||||
Each output file is written atomically. A later file-write failure leaves the
|
Treat an output bundle as durable user data. Do not use cache-cleanup policy to
|
||||||
newly allocated partial run directory in place for inspection; Notarius never
|
remove it. An optional accepted chunk map is also durable output and can carry
|
||||||
automatically removes output. The
|
source- or model-derived annotations; its content and compatibility contract
|
||||||
[JSON output contract](integrations/json-output.md) owns the logical file
|
are defined in [Accepted Chunk Map](integrations/chunk-map.md).
|
||||||
names, schemas, and media types inside a run directory.
|
|
||||||
|
|
||||||
An enabled JSON `include_chunk_map` option adds an accepted chunk map to durable
|
|
||||||
output. Its annotations may contain source- or model-derived data, so retain
|
|
||||||
and protect it like lane output. The map is opt-in and does not alter existing
|
|
||||||
bundles; its payload exclusions are defined in the
|
|
||||||
[Accepted Chunk Map contract](integrations/chunk-map.md).
|
|
||||||
|
|
||||||
Remove an output run directory only after its consumer data is no longer
|
|
||||||
needed. This is data deletion, not cache cleanup.
|
|
||||||
|
|
||||||
## Ordered D&D Workflow
|
|
||||||
|
|
||||||
The maintained [complete D&D configuration](../examples/dnd-complete.config.yml)
|
|
||||||
contains one pipeline with two ordered steps. The first step extracts and
|
|
||||||
normalizes independent item events, NPCs, and scene descriptions. Only after
|
|
||||||
the NPC and scene-description lanes reach accepted terminal results does the
|
|
||||||
second step begin; its generated NPC reference is
|
|
||||||
supplied in memory to spell, combat-turn, and NPC-interaction extraction and
|
|
||||||
the applicable normalizers, while its generated scene-description reference is
|
|
||||||
supplied to combat-turn extraction. The item-event lane has no generated
|
|
||||||
reference dependency and retains only current-transcript evidence.
|
|
||||||
|
|
||||||
```sh
|
|
||||||
go run ./cmd/notarius run dnd-session \
|
|
||||||
--config examples/dnd-complete.config.yml \
|
|
||||||
--input examples/seriatim-minimal-transcript.json \
|
|
||||||
--output-dir ./npc-grounded-output
|
|
||||||
```
|
|
||||||
|
|
||||||
The NPC artifact grounds canonical names through a names-only prompt projection, not spell or combat
|
|
||||||
evidence. Current-transcript source ranges remain the only event evidence. The
|
|
||||||
manifest records generated-reference identity and bounded producer provenance;
|
|
||||||
it does not record generated payload content, and no generated content is
|
|
||||||
exposed through a filesystem path. The same producer artifact may fan out to
|
|
||||||
compatible consumers, while a missing or rejected producer prevents the later
|
|
||||||
step from starting.
|
|
||||||
|
|
||||||
Combat-turn extraction calls its LLM only when a scene record exactly matches
|
|
||||||
the accepted chunk and has kind `combat`. Exact `narrative`, `recap`, and
|
|
||||||
`meta` records produce an accepted empty combat-turn result without an LLM call.
|
|
||||||
Missing or mismatched scene coverage also produces an accepted empty result and
|
|
||||||
a bounded warning. Scene descriptions remain control context rather than combat
|
|
||||||
evidence or prompt material; the complete behavior is defined by the
|
|
||||||
[D&D combat-turn artifact contract](integrations/dnd-combat-turn-artifacts.md).
|
|
||||||
|
|
||||||
Standalone module configurations continue to support external NPC files when a
|
|
||||||
workflow intentionally crosses a process or session boundary. Those files are
|
|
||||||
validated against the consumer slot and must be protected as sensitive
|
|
||||||
campaign data. They are not part of the maintained ordered handoff workflow.
|
|
||||||
|
|
||||||
### NPC Semantic Normalization
|
|
||||||
|
|
||||||
Before the first step can release its accepted NPC artifact across the ordered
|
|
||||||
generated-reference barrier, `dnd/npcs` performs one document-level semantic
|
|
||||||
normalization call for each configured normalize attempt when eligible distinct
|
|
||||||
names remain. The normalize binding's `retries` setting controls additional
|
|
||||||
attempts. If an invalid or unsafe identity proposal exhausts that budget, the
|
|
||||||
run safely accepts the deterministic base result and any independently
|
|
||||||
validated, model-proposed partial consolidation, with a bounded warning;
|
|
||||||
ordinary validation still applies before the artifact can cross the barrier.
|
|
||||||
|
|
||||||
An accepted normalized NPC checkpoint can be reused on `--resume` just like
|
|
||||||
other accepted normalize work. A changed normalization prompt, response schema,
|
|
||||||
or policy identity produces a cold cache miss, so the current reconciliation is
|
|
||||||
recomputed rather than silently reusing incompatible state.
|
|
||||||
|
|
||||||
## Chunk-Plan Cache
|
## Chunk-Plan Cache
|
||||||
|
|
||||||
Chunk plans are stored at:
|
Chunk plans live beneath the selected chunk-plan root:
|
||||||
|
|
||||||
```text
|
~~~
|
||||||
<chunk-plan-root>/<source-sha256-hex>/plan.json
|
<chunk-plan-root>/<source-sha256-hex>/plan.json
|
||||||
```
|
~~~
|
||||||
|
|
||||||
`auto` reuses a complete valid plan or regenerates missing or invalid state.
|
One validated canonical plan is active for each source digest. The plan stores
|
||||||
`refresh` regenerates and atomically replaces a plan after chunk validation.
|
boundaries and provenance, not a second copy of the entire source. This
|
||||||
`bypass` performs no plan-cache I/O and does not resolve or create the root.
|
source-addressed policy is recorded in [ADR-0005](adr/0005-cache-canonical-chunk-plans-by-source.md).
|
||||||
Plan selection is source-addressed and independent of checkpoint and debug
|
|
||||||
roots.
|
|
||||||
|
|
||||||
When its directory is empty in configuration, the root is
|
The configured cache mode controls one invocation:
|
||||||
`<os.UserCacheDir>/notarius/chunk-plans`. A configured directory is the exact
|
|
||||||
root; no suffix is appended. Directories and files created by the store use
|
|
||||||
`0700` and `0600` permissions on supported Unix systems. The configured root
|
|
||||||
is a trust boundary: do not share it among mutually untrusted users.
|
|
||||||
|
|
||||||
Remove an exact digest directory or the configured root only when accepting the
|
- **auto** looks for a valid active plan. Missing or invalid state causes a new
|
||||||
cost of recomputing plans and any chunk-stage work. Cache publication is atomic;
|
plan to be generated; an accepted new plan is atomically published.
|
||||||
there is no history, locking, garbage collection, or rollback facility.
|
- **refresh** skips lookup, generates a plan with the configured chunker, and
|
||||||
|
atomically replaces the active plan after it is accepted.
|
||||||
|
- **bypass** performs no chunk-plan cache I/O. It does not resolve or create a
|
||||||
|
chunk-plan root.
|
||||||
|
|
||||||
For a Linux service account, provision a dedicated restrictive root such as:
|
A reused plan is still materialized and validated against the current source.
|
||||||
|
If a prior plan no longer gives acceptable results, use a refresh run rather
|
||||||
|
than editing cache files. Deleting a plan is recoverable but can repeat costly
|
||||||
|
chunking work.
|
||||||
|
|
||||||
```yaml
|
## Checkpoint Recording, Resume, And Recompute
|
||||||
cache:
|
|
||||||
chunk_plans:
|
|
||||||
directory: /var/cache/notarius/chunk-plans
|
|
||||||
```
|
|
||||||
|
|
||||||
## Checkpoint Cache
|
Checkpoint recording is an explicit configuration choice and is disabled by
|
||||||
|
default. When enabled, each run records stage transitions and the state needed
|
||||||
|
for compatible recovery. A run records checkpoints even when it does not ask
|
||||||
|
to reuse them. Checkpoint payloads can contain source-derived and intermediate
|
||||||
|
application data, so treat the entire root as sensitive.
|
||||||
|
|
||||||
Checkpoint recording is controlled by `cache.checkpoints.enabled`, which
|
Checkpoint loading is separate: [**--resume**](cli.md#run) asks a run to reuse
|
||||||
defaults to `false`. When enabled, every run records running, succeeded, and
|
compatible recorded work. A resume request fails when checkpoint recording is
|
||||||
failed transitions and reusable validator-approved results. Successful,
|
disabled. Without **--resume**, a recording-enabled run executes normally and
|
||||||
rejected, and failed runs may therefore all leave checkpoint state. The
|
does not load checkpoint state. Compatibility includes the resolved pipeline,
|
||||||
`--resume` flag additionally loads compatible completed work before executing
|
input, selected lanes, runtime overrides, reference provenance, LLM-profile
|
||||||
missing or incompatible stages. Without `--resume`, a recording-enabled run
|
provenance, and prepared-component fingerprints. A changed identity produces a
|
||||||
never loads checkpoints. Using `--resume` while recording is disabled is an
|
cold miss; Notarius does not migrate, rewrite, or delete older checkpoint
|
||||||
error.
|
directories.
|
||||||
|
|
||||||
Checkpoints use the selected root and the existing identity hierarchy:
|
Checkpoint state is confined below an identity-specific path:
|
||||||
|
|
||||||
```text
|
~~~
|
||||||
<checkpoint-root>/<pipeline-id>/<input-key>-<source-or-input-digest>/<pipeline-digest>/<identity-digest>/...
|
<checkpoint-root>/<pipeline-id>/<input-key>-<source-or-input-digest-prefix>/<pipeline-digest-prefix>/<identity-digest-prefix>/
|
||||||
```
|
~~~
|
||||||
|
|
||||||
The final identity digest includes stable semantic fingerprints explicitly
|
### Selective Recompute
|
||||||
contributed by prepared modules and validators. Adding or changing one of
|
|
||||||
these fingerprints intentionally causes a cold cache miss; old checkpoint
|
|
||||||
directories are left in place and are never migrated or deleted automatically.
|
|
||||||
|
|
||||||
An empty configured directory selects
|
[**--recompute-step**](cli.md#run) requires both **--resume** and enabled
|
||||||
`<os.UserCacheDir>/notarius/checkpoints`. The root is exact when configured.
|
checkpoint recording. It forces the selected ordered step and every lane that
|
||||||
Created directories and files use `0700` and `0600` permissions on supported
|
depends on it through generated artifact references. Unrelated lanes remain
|
||||||
Unix systems.
|
eligible for reuse.
|
||||||
|
|
||||||
Checkpoint payloads can contain source text, intermediate artifacts, metadata,
|
For an earlier producer required by a forced consumer, Notarius requires a
|
||||||
warnings, and content digests. Treat them as sensitive derived application
|
compatible accepted normalized artifact. It validates that artifact before
|
||||||
data. Compatible files from a former checkpoint root remain reusable when
|
hydrating it and does not silently rerun the producer. If that state is
|
||||||
`cache.checkpoints.directory` names that exact existing root. They are not
|
missing, rejected, corrupt, non-canonical, or incompatible, the run stops
|
||||||
moved, migrated, or deleted automatically. The frozen serialized identifier
|
before its dependent starts. Rerun the required producer deliberately instead
|
||||||
`workspace_schema_version` remains part of checkpoint compatibility; it is not
|
of copying or editing checkpoint files.
|
||||||
a configuration setting.
|
|
||||||
|
|
||||||
For a Linux service account, independently provision:
|
## Checkpoint Decisions And Recovery
|
||||||
|
|
||||||
```yaml
|
Checkpoint events classify work as **executed**, **reused**,
|
||||||
cache:
|
**forced_recompute**, or **dependency_invalidated**. Their stable reason codes
|
||||||
checkpoints:
|
are written to run diagnostics and provenance. Use the code, not a copied
|
||||||
enabled: true
|
error message, to decide what to repair.
|
||||||
directory: /var/cache/notarius/checkpoints
|
|
||||||
```
|
|
||||||
|
|
||||||
Remove an exact checkpoint identity directory or the configured root only when
|
| Reason code | Recovery meaning |
|
||||||
recomputation is acceptable.
|
|
||||||
|
|
||||||
### Resume And Selective Recompute
|
|
||||||
|
|
||||||
`--resume` loads compatible accepted work only when checkpoint recording is
|
|
||||||
enabled. A normal resumed run may reuse source, extract, merge, and normalize
|
|
||||||
checkpoints independently and may recompute a stage after a cache miss.
|
|
||||||
Generated references add a dependency fingerprint
|
|
||||||
for the producer's artifact kind, schema identity, media type, canonical
|
|
||||||
content digest, and size. If that fingerprint changes or the producer is
|
|
||||||
missing, dependent checkpoints are invalidated; unrelated work remains eligible
|
|
||||||
for reuse.
|
|
||||||
|
|
||||||
`--recompute-step <step-id>` requires both `--resume` and
|
|
||||||
`cache.checkpoints.enabled: true`. It forces the named step and all transitive
|
|
||||||
dependents to execute, while compatible predecessors and unrelated lanes remain
|
|
||||||
reusable. The ID may be an explicit configured step or `default` for an
|
|
||||||
implicit single-step pipeline. It cannot be combined with `--only`, and it does
|
|
||||||
not change the persistent identity of otherwise identical checkpoints.
|
|
||||||
Decisions are bounded and categorized as `reused`, `executed`,
|
|
||||||
`forced_recompute`, or `dependency_invalidated`.
|
|
||||||
|
|
||||||
For an unselected producer required by a recomputed step, Notarius loads the
|
|
||||||
accepted normalized artifact directly. Valid normalize state is sufficient even
|
|
||||||
when that producer's extract or merge checkpoint is missing or corrupt. The
|
|
||||||
normalize manifest must be successful and match workspace schema v3, the exact
|
|
||||||
current invocation identity, step, lane, and normalizer; its payload digest and
|
|
||||||
canonical codec representation must also validate. A forced producer bypasses
|
|
||||||
this lookup and executes.
|
|
||||||
|
|
||||||
If a required predecessor's accepted normalized artifact is missing, rejected,
|
|
||||||
corrupt, non-canonical, or incompatible, the run fails before the dependent
|
|
||||||
step starts. It does not fall back to rerunning that predecessor. The failure
|
|
||||||
manifest retains completed upstream outcomes and dependency context but not
|
|
||||||
generated reference content. For diagnosis, first check the producer step and
|
|
||||||
lane in the manifest, then inspect checkpoint decision categories and reason
|
|
||||||
codes. Rerun the producer explicitly rather than copying an artifact into the
|
|
||||||
checkpoint root.
|
|
||||||
|
|
||||||
The decision that caused a required-predecessor failure is retained before the
|
|
||||||
run returns, and the CLI error identifies its step, lane, and reason code.
|
|
||||||
|
|
||||||
Checkpoint reason codes are stable diagnostic identifiers:
|
|
||||||
|
|
||||||
| Reason code | Operator meaning |
|
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| `loading_disabled` | This invocation did not enable checkpoint loading. |
|
| **loading_disabled** | This invocation did not permit checkpoint loading. |
|
||||||
| `checkpoint_missing` | The requested checkpoint file does not exist. |
|
| **checkpoint_missing**, **checkpoint_path_invalid**, **checkpoint_read_failed**, **checkpoint_decode_failed** | The stored checkpoint could not be located or read safely; normal resume work can execute again. |
|
||||||
| `checkpoint_path_invalid` | The requested checkpoint location failed confinement validation. |
|
| **workspace_schema_incompatible**, **identity_mismatch**, **stage_mismatch**, **step_mismatch**, **lane_mismatch**, **module_mismatch** | Stored state belongs to a different compatible scope or identity; allow a fresh run to create new state. |
|
||||||
| `checkpoint_read_failed` | An existing checkpoint could not be read. |
|
| **status_not_reusable** | The recorded operation did not end in reusable state. |
|
||||||
| `checkpoint_decode_failed` | Checkpoint JSON could not be decoded. |
|
| **dependency_mismatch** | A dependency changed; dependent work is invalidated rather than reused. |
|
||||||
| `workspace_schema_incompatible` | The stored workspace schema is not supported by this build. |
|
| **artifact_payload_invalid**, **artifact_digest_mismatch**, **artifact_codec_incompatible**, **artifact_not_canonical** | A stored artifact cannot safely be hydrated; rerun the producer instead of modifying the cache. |
|
||||||
| `identity_mismatch` | The stored invocation identity differs from the current invocation. |
|
| **checkpoint_reused** | A normal checkpoint passed compatibility checks. |
|
||||||
| `stage_mismatch`, `step_mismatch`, `lane_mismatch`, `module_mismatch` | Stored scope does not match the requested pipeline scope. |
|
| **accepted_artifact_reused** | A required predecessor's accepted normalized artifact was safely hydrated. |
|
||||||
| `status_not_reusable` | The stored operation did not finish in a reusable status. |
|
| **recompute_step** | Selective recomputation deliberately forced this work. |
|
||||||
| `dependency_mismatch` | Stored dependencies differ; the category is `dependency_invalidated`. |
|
|
||||||
| `artifact_payload_invalid` | Stored artifact payload structure or encoding is invalid. |
|
|
||||||
| `artifact_digest_mismatch` | Stored artifact bytes do not match their recorded digest. |
|
|
||||||
| `artifact_codec_incompatible` | Stored artifact identity is incomplete or incompatible with the codec contract. |
|
|
||||||
| `artifact_not_canonical` | The codec can decode the artifact, but its bytes are not canonical. |
|
|
||||||
| `checkpoint_reused` | The stored checkpoint passed validation and was reused. |
|
|
||||||
| `accepted_artifact_reused` | A required producer's accepted normalized artifact was canonically validated and hydrated. |
|
|
||||||
| `recompute_step` | Selective recomputation forced execution of this lane. |
|
|
||||||
|
|
||||||
Decision detail is bounded explanatory text derived from the stable reason code,
|
Reason detail is bounded code-owned text. It is diagnostic information, not a
|
||||||
not caller-supplied prose or a data-recovery channel. It never contains
|
path-discovery or data-recovery mechanism, and does not contain checkpoint,
|
||||||
checkpoint paths, artifact or reference content, source content, credentials,
|
source, reference, credential, or environment content.
|
||||||
or environment values.
|
|
||||||
|
|
||||||
## Debug Bundles
|
## Debug Bundles
|
||||||
|
|
||||||
Only `notarius run --debug` enables debug collection. The selected root contains
|
Only a [debug-enabled run](cli.md#run) creates a bundle:
|
||||||
one retained bundle per invocation:
|
|
||||||
|
|
||||||
```text
|
~~~
|
||||||
<debug-root>/<run-id>/
|
<debug-root>/<run-id>/
|
||||||
summary/
|
summary/
|
||||||
trace/
|
trace/
|
||||||
```
|
~~~
|
||||||
|
|
||||||
`summary/` contains redacted invocation, effective-configuration, resolved
|
The summary contains redacted invocation and resolution information plus run,
|
||||||
pipeline and reference provenance, checkpoint and chunk-plan decisions, run
|
warning, checkpoint, chunk-plan, and terminal reporting artifacts. The trace
|
||||||
manifest, warnings, report, and any available error text. It excludes raw
|
contains allowlisted application diagnostic records and can include source or
|
||||||
source, references, annotations, prompts, model responses, credentials, and
|
derived application data. Neither surface is a cache input. Do not treat a
|
||||||
malformed cache bytes.
|
debug bundle as safe to share merely because its configuration summary is
|
||||||
|
redacted.
|
||||||
|
|
||||||
`trace/` contains application-owned execution detail, including source and
|
Notarius never creates debug state without an explicit request and never
|
||||||
stage material, plans, chunks, validator attempts, prompts, model responses,
|
automatically deletes a requested bundle. If allocation succeeds, the command
|
||||||
timing, and serialized artifacts. It may retain application data omitted from
|
reports its path on both success and later failure. A summary, trace, or
|
||||||
output. Credentials, credential-shaped values, sensitive metadata, unrelated
|
terminal-report persistence failure fails the command while preserving any
|
||||||
environment values, and unrelated filesystem content are not captured.
|
already-written diagnostic data for inspection.
|
||||||
|
|
||||||
Bundles inherit the sensitivity of the application data they capture. Their
|
|
||||||
additional risk comes from copying and aggregating that data, so restrict
|
|
||||||
access, avoid shared roots between untrusted users, and define retention outside
|
|
||||||
Notarius. Created bundle directories use `0700` and files use `0600` on
|
|
||||||
supported Unix systems.
|
|
||||||
|
|
||||||
Notarius never automatically deletes a requested bundle. If allocation
|
|
||||||
succeeds, its path is reported on success and failure. A requested summary or
|
|
||||||
trace write failure makes the command fail, preserving whatever bundle data was
|
|
||||||
already written for inspection. Every allocated bundle makes one best-effort
|
|
||||||
attempt to record a terminal `run-report.json`.
|
|
||||||
|
|
||||||
## Failures And Warnings
|
|
||||||
|
|
||||||
Failures before debug allocation are reported on stderr without a bundle.
|
|
||||||
Failures after allocation report the bundle path on stderr and make independent
|
|
||||||
attempts to write a failure `run-report.json` and `error.log`. The report retains
|
|
||||||
the paths and pipeline outcome fields known at the failure point. If either
|
|
||||||
terminal write fails, the original command error remains first on stderr,
|
|
||||||
followed by the persistence error and bundle path. An output-write failure
|
|
||||||
leaves the allocated bundle in place. A successful run with warnings exits `0`,
|
|
||||||
reports a warning count on stderr, and records warnings in durable output and
|
|
||||||
any requested debug summary.
|
|
||||||
|
|
||||||
## Cleanup
|
## Cleanup
|
||||||
|
|
||||||
Use exact paths for manual cleanup. Examples:
|
Cleanup is manual and destructive. First inspect the exact leaf directory,
|
||||||
|
then remove only that leaf; do not use a glob or a parent root as the target.
|
||||||
|
|
||||||
```sh
|
~~~
|
||||||
rm -rf ./notarius-output/run-1721300000000000000-0123456789abcdef0123456789abcdef
|
rm -rf -- /srv/notarius/output/run-1721300000000000000-0123456789abcdef0123456789abcdef
|
||||||
rm -rf /var/cache/notarius/chunk-plans/0123abcd
|
rm -rf -- /srv/notarius/chunk-plans/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
|
||||||
rm -rf /var/cache/notarius/checkpoints/pipeline/input-0123/pipeline-4567/identity-89ab
|
rm -rf -- /srv/notarius/checkpoints/example/seriatim-0123456789abcdef/0123456789abcdef/0123456789abcdef
|
||||||
rm -rf ./notarius-debug/run-1721300000000000000-0123456789abcdef0123456789abcdef
|
rm -rf -- /srv/notarius/debug/run-1721300000000000000-0123456789abcdef0123456789abcdef
|
||||||
```
|
~~~
|
||||||
|
|
||||||
Avoid broad recursive cleanup against a parent root unless it is an explicit
|
Deleting output permanently removes user data. Deleting chunk plans or
|
||||||
operator policy. Output deletion is permanent user-data loss. Cache deletion is
|
checkpoints is recoverable but may repeat expensive provider or pipeline work.
|
||||||
recoverable but can repeat expensive work. Debug deletion removes troubleshooting
|
Deleting a debug bundle removes troubleshooting evidence and a retained copy of
|
||||||
evidence and any retained application-data copy.
|
application data. Notarius has no cache garbage collector, rollback operation,
|
||||||
|
or automatic cleanup command.
|
||||||
|
|
||||||
## Operational Limits
|
## Operational Limits
|
||||||
|
|
||||||
Provider retries and timeouts are handled by Scriptorium according to the
|
Provider retries and timeouts are supplied by the selected Scriptorium profile.
|
||||||
selected execution profile. Pipeline module retry settings are defined in
|
Module retry settings and concurrency limits are configuration contracts; see
|
||||||
[Configuration](config.md#module-bindings). Extract worker concurrency and
|
[module bindings](config.md#module-bindings-and-validators) and
|
||||||
actual provider-call concurrency are separate limits; their fields and
|
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
|
||||||
validation are defined in [Configuration](config.md#concurrency). Notarius
|
limits and actual provider-call limits are independent. Notarius writes local
|
||||||
writes local files only; remote storage and archive management are outside the
|
filesystem state only; remote storage, archival, and retention automation are
|
||||||
implemented CLI.
|
outside the implemented CLI.
|
||||||
|
|||||||
1031
docs/roadmap/documentation.md
Normal file
1031
docs/roadmap/documentation.md
Normal file
File diff suppressed because it is too large
Load Diff
85
examples/dnd-complete-transcript.json
Normal file
85
examples/dnd-complete-transcript.json
Normal file
@@ -0,0 +1,85 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"id": "session-ravenfall",
|
||||||
|
"title": "The Ravenfall Watchtower"
|
||||||
|
},
|
||||||
|
"segments": [
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"start": 0,
|
||||||
|
"end": 14,
|
||||||
|
"speaker": "DM",
|
||||||
|
"text": "Recap: last session, the party learned that Elder Rowan vanished near the Ravenfall watchtower."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 2,
|
||||||
|
"start": 14,
|
||||||
|
"end": 25,
|
||||||
|
"speaker": "Player",
|
||||||
|
"text": "Out of character, we agree to investigate the watchtower before the next game."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 3,
|
||||||
|
"start": 25,
|
||||||
|
"end": 39,
|
||||||
|
"speaker": "DM",
|
||||||
|
"text": "Aria and Borin arrive at the ruined Ravenfall watchtower as dusk settles over the road."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 4,
|
||||||
|
"start": 39,
|
||||||
|
"end": 55,
|
||||||
|
"speaker": "Mira Thorn",
|
||||||
|
"text": "Mira Thorn steps from the doorway and says, \"Elder Rowan warned me that Kesh would return for the relic.\""
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 5,
|
||||||
|
"start": 55,
|
||||||
|
"end": 70,
|
||||||
|
"speaker": "DM",
|
||||||
|
"text": "Mira leads the party to a hidden cache. The party discovers a moonblade and acquires 20 silver pieces."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 6,
|
||||||
|
"start": 70,
|
||||||
|
"end": 83,
|
||||||
|
"speaker": "Aria",
|
||||||
|
"text": "Aria hands her healing potion to Borin so he can carry it into the tower."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 7,
|
||||||
|
"start": 83,
|
||||||
|
"end": 96,
|
||||||
|
"speaker": "DM",
|
||||||
|
"text": "Kesh, the goblin captain, orders the raiders to attack. Roll initiative."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 8,
|
||||||
|
"start": 96,
|
||||||
|
"end": 110,
|
||||||
|
"speaker": "DM",
|
||||||
|
"text": "On Kesh's turn, he strikes Borin with his scimitar. Borin drinks the healing potion on his turn."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 9,
|
||||||
|
"start": 110,
|
||||||
|
"end": 124,
|
||||||
|
"speaker": "Aria",
|
||||||
|
"text": "Aria casts Cure Wounds on Borin, then invokes Aegis of Emberfall as Kesh closes in."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 10,
|
||||||
|
"start": 124,
|
||||||
|
"end": 137,
|
||||||
|
"speaker": "DM",
|
||||||
|
"text": "Kesh casts Shield as a reaction against Borin's counterattack, but the party drives the raiders away."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 11,
|
||||||
|
"start": 137,
|
||||||
|
"end": 150,
|
||||||
|
"speaker": "DM",
|
||||||
|
"text": "After the battle, Aria pays 5 silver pieces to repair the watchtower gate."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -1,2 +1,8 @@
|
|||||||
Cure Wounds: healing spell cast by touch.
|
Ravenfall watchtower: a ruined watchtower near the party's current route.
|
||||||
Shield: defensive reaction spell.
|
Mira Thorn: the watchtower's keeper.
|
||||||
|
Elder Rowan: a missing local scholar.
|
||||||
|
Kesh: a goblin captain leading raiders.
|
||||||
|
Moonblade: a blade found in the watchtower's hidden cache.
|
||||||
|
Cure Wounds: a healing spell.
|
||||||
|
Shield: a defensive reaction spell.
|
||||||
|
Aegis of Emberfall: a campaign spell recorded in the supplied catalog overlay.
|
||||||
|
|||||||
@@ -1,3 +1,2 @@
|
|||||||
Aria: party cleric and recurring healer.
|
Aria: party cleric and recurring healer.
|
||||||
Borin: fighter ally.
|
Borin: fighter ally.
|
||||||
Bandit mage: hostile spellcaster.
|
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package cli
|
package cli
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
@@ -11,9 +12,11 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/core/debugbundle"
|
"gitea.maximumdirect.net/eric/notarius/internal/core/debugbundle"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/input/transcript"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
|
func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
|
||||||
@@ -21,6 +24,20 @@ func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
|
|||||||
for _, example := range maintainedExampleFiles(t) {
|
for _, example := range maintainedExampleFiles(t) {
|
||||||
t.Run(example.name, func(t *testing.T) {
|
t.Run(example.name, func(t *testing.T) {
|
||||||
cfg := loadMaintainedExample(t, example.path)
|
cfg := loadMaintainedExample(t, example.path)
|
||||||
|
raw, err := os.ReadFile(example.transcriptPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read maintained transcript %q: %v", example.transcriptPath, err)
|
||||||
|
}
|
||||||
|
document, err := transcript.New().Parse(context.Background(), contracts.ParseRequest{
|
||||||
|
Path: example.transcriptPath,
|
||||||
|
Raw: raw,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse maintained transcript %q: %v", example.transcriptPath, err)
|
||||||
|
}
|
||||||
|
if len(document.Units) == 0 {
|
||||||
|
t.Fatalf("maintained transcript %q has no parsed units", example.transcriptPath)
|
||||||
|
}
|
||||||
for _, pipelineID := range example.pipelineIDs {
|
for _, pipelineID := range example.pipelineIDs {
|
||||||
effective, err := cfg.Resolve(resolveInputForMaintainedExample(components, pipelineID))
|
effective, err := cfg.Resolve(resolveInputForMaintainedExample(components, pipelineID))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -569,14 +569,15 @@ func TestProductionSceneRunRecordsAnnotationFreeChunkPlanAndProvenance(t *testin
|
|||||||
type maintainedExample struct {
|
type maintainedExample struct {
|
||||||
name string
|
name string
|
||||||
path string
|
path string
|
||||||
|
transcriptPath string
|
||||||
pipelineIDs []string
|
pipelineIDs []string
|
||||||
}
|
}
|
||||||
|
|
||||||
func maintainedExampleFiles(t *testing.T) []maintainedExample {
|
func maintainedExampleFiles(t *testing.T) []maintainedExample {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
return []maintainedExample{
|
return []maintainedExample{
|
||||||
{name: "minimal", path: repositoryPath("examples", "dnd-minimal.config.yml"), pipelineIDs: []string{"dnd-session"}},
|
{name: "minimal", path: repositoryPath("examples", "dnd-minimal.config.yml"), transcriptPath: repositoryPath("examples", "seriatim-minimal-transcript.json"), pipelineIDs: []string{"dnd-session"}},
|
||||||
{name: "complete", path: repositoryPath("examples", "dnd-complete.config.yml"), pipelineIDs: []string{"dnd-session"}},
|
{name: "complete", path: repositoryPath("examples", "dnd-complete.config.yml"), transcriptPath: repositoryPath("examples", "dnd-complete-transcript.json"), pipelineIDs: []string{"dnd-session"}},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user