825 lines
36 KiB
Markdown
825 lines
36 KiB
Markdown
# Configuration
|
|
|
|
This is the canonical reference for implemented Notarius configuration.
|
|
|
|
Notarius reads YAML config files with `version: 3`. File configuration is
|
|
applied over built-in defaults, then environment overrides are applied. Explicit
|
|
CLI overrides are applied last where the command supports them.
|
|
|
|
## Discovery
|
|
|
|
Commands that load configuration use this order:
|
|
|
|
1. an explicit path supplied through the CLI, when provided;
|
|
2. `NOTARIUS_CONFIG`, when set to a non-empty path;
|
|
3. `/usr/local/etc/notarius/config.yml`.
|
|
|
|
If none is available, the command fails with a config file not found error.
|
|
The explicit-path option is defined in the [CLI reference](cli.md).
|
|
|
|
## Maintained Examples
|
|
|
|
- [Minimal D&D configuration](../examples/dnd-minimal.config.yml) shows the
|
|
shortest useful single-step, single-lane pipeline.
|
|
- [Complete D&D configuration](../examples/dnd-complete.config.yml) shows scene
|
|
chunking, ordered steps, all six D&D artifact lanes, generated NPC and
|
|
scene-description handoffs, campaign and spell-catalog references, checkpoint
|
|
storage, and chunk-map output.
|
|
|
|
Both are complete version 3 files. The fragments below illustrate individual
|
|
fields and are not alternate complete configurations.
|
|
|
|
## Top-Level Fields
|
|
|
|
- `version`: required. The only supported value is `3`.
|
|
- `scriptorium`: optional Scriptorium profile source settings.
|
|
- `pipelines`: optional map of pipeline IDs to pipeline definitions.
|
|
- `concurrency`: optional global concurrency settings.
|
|
- `output`: optional durable output placement.
|
|
- `cache`: optional chunk-plan and checkpoint cache placement.
|
|
- `debug`: optional debug-bundle placement. It does not enable debug capture.
|
|
|
|
Unknown YAML fields are rejected. The removed top-level `llm_profiles` field is
|
|
rejected; execution profiles now come from Scriptorium.
|
|
|
|
## Defaults
|
|
|
|
Built-in defaults:
|
|
|
|
- `concurrency.total_llm`: `1`
|
|
- `concurrency.stage_workers.extract`: effective `concurrency.total_llm`
|
|
- `output.directory`: `./notarius-output`
|
|
- `cache.chunk_plans.mode`: `auto`
|
|
- `cache.chunk_plans.directory`: unset, selecting
|
|
`<os.UserCacheDir>/notarius/chunk-plans`
|
|
- `cache.checkpoints.enabled`: `false`
|
|
- `cache.checkpoints.directory`: unset, selecting
|
|
`<os.UserCacheDir>/notarius/checkpoints`
|
|
- `debug.directory`: `./notarius-debug`
|
|
|
|
No pipelines are built in. A run requires a configured pipeline.
|
|
|
|
If `scriptorium` is omitted, Notarius uses Scriptorium's built-in profile
|
|
catalog. Prompt definitions may also name default profile IDs. The current D&D
|
|
scene, extraction, and NPC-normalization prompts default to `gemini-2-flash`
|
|
when a module binding does not set `llm_profile`. The selected built-in profile
|
|
reads its credentials from its Scriptorium profile definition.
|
|
|
|
## Scriptorium Profiles
|
|
|
|
`scriptorium` fields:
|
|
|
|
- `profile_dir`: optional directory containing Scriptorium profile YAML files.
|
|
- `profile_file`: optional Scriptorium profile YAML file.
|
|
|
|
`profile_dir` and `profile_file` are mutually exclusive. Custom profiles
|
|
overlay Scriptorium built-in profiles by profile ID.
|
|
|
|
Scriptorium profile files use Scriptorium's profile schema. A minimal profile
|
|
looks like:
|
|
|
|
```yaml
|
|
id: local-fast
|
|
endpoint: http://127.0.0.1:8080/v1
|
|
model: your-model
|
|
api_key_env: SCRIPTORIUM_API_KEY
|
|
timeout_seconds: 180
|
|
```
|
|
|
|
Notarius does not accept raw API keys in Notarius config. For file-backed
|
|
Scriptorium profiles, store the environment variable name in `api_key_env` and
|
|
set that variable in the run environment. Scriptorium rejects raw `api_key`
|
|
fields in profile YAML.
|
|
|
|
## Environment Overrides
|
|
|
|
These environment variables are applied after the config file:
|
|
|
|
- `NOTARIUS_CONFIG`: config discovery path.
|
|
- `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency.
|
|
- `NOTARIUS_STAGE_WORKERS_EXTRACT`: integer extract worker limit.
|
|
- `NOTARIUS_OUTPUT_DIR`: durable output root.
|
|
- `NOTARIUS_CACHE_CHUNK_PLANS_MODE`: chunk-plan cache mode.
|
|
- `NOTARIUS_CACHE_CHUNK_PLANS_DIR`: chunk-plan cache root.
|
|
- `NOTARIUS_CACHE_CHECKPOINTS_DIR`: checkpoint cache root.
|
|
- `NOTARIUS_DEBUG_DIR`: debug-bundle root.
|
|
|
|
Integer environment values must parse as base-10 integers. Directory overrides
|
|
must be non-empty after trimming. Cache-directory fields in a file may be
|
|
empty, which deliberately selects the corresponding per-user default.
|
|
|
|
The removed `NOTARIUS_LLM_DEFAULT_*` variables are not read. Configure provider
|
|
endpoint, model, and credential environment variable names through Scriptorium
|
|
profiles.
|
|
|
|
## Concurrency
|
|
|
|
`concurrency` fields:
|
|
|
|
- `total_llm`: positive integer ceiling on concurrent provider calls.
|
|
- `stage_workers`: optional map of framework worker limits. The only supported
|
|
key is `extract`.
|
|
|
|
`stage_workers.extract` defaults to the effective `total_llm` value after file
|
|
and environment precedence. It must be between `1` and `total_llm`, inclusive.
|
|
Unknown or empty stage-worker keys are rejected. The environment override
|
|
`NOTARIUS_STAGE_WORKERS_EXTRACT` takes precedence over the file value, as does
|
|
`NOTARIUS_TOTAL_LLM_CONCURRENCY` for the global ceiling.
|
|
|
|
The worker value is present in effective and redacted configuration. It bounds
|
|
the fixed run-wide extract pool and its bounded dispatch queue. Extract jobs are
|
|
submitted by source chunk and then resolved lane; `total_llm` independently
|
|
bounds actual provider calls made by extracts, retries, and validators.
|
|
|
|
## Pipelines
|
|
|
|
A pipeline selects implementations for the fixed workflow defined by
|
|
[Architecture](policy/architecture.md#system-shape).
|
|
|
|
Pipeline fields:
|
|
|
|
- `input`: required module binding.
|
|
- `chunk`: optional module binding. Default module is `generic`.
|
|
- `artifacts`: the artifact lane map for a single-step pipeline. It is treated
|
|
as an implicit step with the stable ID `default`.
|
|
- `steps`: an ordered, non-empty list of step definitions. A pipeline may use
|
|
`steps` or `artifacts`, but not both. Step IDs must be unique after trimming.
|
|
- `output`: optional module binding. Default module is `json`.
|
|
- `references`: optional map of reference slot names to reference paths. These
|
|
bindings are defaults for eligible pipeline targets that declare the matching
|
|
slot.
|
|
|
|
Each explicit step contains an `id`, an optional `references` map, and a
|
|
non-empty `artifacts` map. Steps share the pipeline input, chunk plan, worker
|
|
budget, output encoder, manifest, and failure boundary. Lanes within a step
|
|
retain the fixed extract, validate, merge, validate, normalize, and validate
|
|
workflow; the next step starts only after the current step is terminal.
|
|
|
|
Generated references use the structured `artifact` source form to identify one
|
|
accepted normalized lane from an earlier step:
|
|
|
|
```yaml
|
|
pipelines:
|
|
dnd-npc-grounded:
|
|
input: seriatim
|
|
steps:
|
|
- id: identify-npcs
|
|
artifacts:
|
|
npcs:
|
|
extract: dnd/npcs
|
|
normalize: dnd/npcs
|
|
scene-descriptions:
|
|
extract: dnd/scene-descriptions
|
|
normalize: dnd/scene-descriptions
|
|
- id: grounded-events
|
|
references:
|
|
npcs:
|
|
artifact:
|
|
step: identify-npcs
|
|
lane: npcs
|
|
scene_descriptions:
|
|
artifact:
|
|
step: identify-npcs
|
|
lane: scene-descriptions
|
|
artifacts:
|
|
spells:
|
|
extract: dnd/spells
|
|
normalize: dnd/spells
|
|
combat:
|
|
extract: dnd/combat-turns
|
|
normalize: dnd/combat-turns
|
|
```
|
|
|
|
The generated binding is explicit and typed; it is not inferred from module,
|
|
lane, or slot names. It may be declared at step scope, applying to every
|
|
selected target in that step that declares the slot, or at one target's
|
|
`references` map. A producer may fan out to compatible target slots, but a
|
|
slot accepts only one producer. A producer must be in an earlier step, and a
|
|
configured generated dependency is required even when the consumer slot is
|
|
otherwise optional. Aggregating several producer artifacts is unsupported.
|
|
|
|
The producer codec supplies the artifact kind, complete schema identity, media
|
|
type, canonical content digest, and size used for compatibility and checkpoint
|
|
dependency checks. Only one accepted normalized artifact crosses the boundary;
|
|
raw extraction results, rejected output, intermediate values, and validator
|
|
diagnostics do not. Generated content is supplied in memory and is never
|
|
represented by a filesystem path.
|
|
|
|
Artifact lane fields:
|
|
|
|
- `extract`: required module binding.
|
|
- `merge`: optional module binding. Default module is `appendorder`.
|
|
- `normalize`: optional module binding. Default module is `noop`.
|
|
- `validators`: deprecated lane-level validator list. Non-empty lists are
|
|
rejected; use `extract.validators`, `merge.validators`, or
|
|
`normalize.validators`.
|
|
- `references`: optional compatibility alias for extractor reference bindings.
|
|
Lane bindings override pipeline-level bindings for the same slot.
|
|
|
|
Commands that resolve a pipeline fail for unknown or incompatible module keys.
|
|
See [CLI Reference](cli.md) for command syntax.
|
|
|
|
Reference bindings are validated against reference slots declared by eligible
|
|
chunk, extract, merge, and normalize targets during pipeline resolution. Required slots
|
|
must be bound after config defaults, target-local references, lane-level
|
|
compatibility bindings, step-local references, and command-line reference
|
|
overrides are applied. Config-relative paths are resolved relative to the
|
|
config file; command-line reference paths are resolved relative to the current
|
|
working directory. Bound files must be UTF-8 text. Reference media types are
|
|
inferred from file extensions and checked when a module restricts accepted
|
|
types; unknown extensions use `application/octet-stream`. See [CLI Reference](cli.md#run)
|
|
for command-line selectors and [Operations](operations.md) for recorded
|
|
provenance and sensitive-data handling.
|
|
|
|
Pipeline-level `references` are defaults. They are valid when at least one
|
|
eligible target in the full configured pipeline declares the slot, including
|
|
chunk, extractor, merger, and normalizer targets. During a run, they apply only
|
|
to the selected targets that declare the slot. For external bindings, a
|
|
step-local binding overrides a pipeline-level default and a target-local
|
|
binding retains the existing most-specific precedence. Generated and external
|
|
bindings may not resolve to the same target slot, and a step-scoped generated
|
|
binding may not be duplicated by a target-local generated binding.
|
|
|
|
```yaml
|
|
pipelines:
|
|
dnd-session:
|
|
input: seriatim
|
|
references:
|
|
players: ./campaign/players.txt
|
|
party: ./campaign/party-roster.txt
|
|
glossary: ./campaign/glossary.txt
|
|
artifacts:
|
|
spells:
|
|
extract: dnd/spells
|
|
```
|
|
|
|
Extractor binding `references` are the canonical lane-local location. The
|
|
legacy lane-level `references` field remains supported as an alias; when both
|
|
bind the same slot, `extract.references` wins:
|
|
|
|
```yaml
|
|
pipelines:
|
|
dnd-session:
|
|
input: seriatim
|
|
references:
|
|
glossary: ./campaign/glossary.txt
|
|
artifacts:
|
|
spells:
|
|
references:
|
|
roster: ./campaign/legacy-roster.txt
|
|
extract:
|
|
module: dnd/spells
|
|
references:
|
|
party: ./campaign/session-party.txt
|
|
```
|
|
|
|
`chunk.references`, `merge.references`, and `normalize.references` are accepted
|
|
in object-form bindings. They override pipeline-level defaults for slots
|
|
declared by that target module. Extractor-local references apply only to the
|
|
extractor, merger-local references apply only to the merger, and
|
|
normalizer-local references apply only to the normalizer.
|
|
|
|
Target-local reference fields use the same map shape at:
|
|
|
|
- `pipelines.<id>.chunk.references`
|
|
- `pipelines.<id>.artifacts.<lane>.extract.references`
|
|
- `pipelines.<id>.artifacts.<lane>.merge.references`
|
|
- `pipelines.<id>.artifacts.<lane>.normalize.references`
|
|
|
|
Each binding is valid only when that target module declares the slot.
|
|
|
|
Reference source forms are:
|
|
|
|
- a scalar string, which is an external file path; or
|
|
- an object with only `artifact`, containing trimmed `step` and `lane` IDs for
|
|
an earlier producer lane.
|
|
|
|
Pipeline-level references accept only external paths. Generated references are
|
|
valid at step scope or on a target-local `references` map. Their producer's
|
|
registered codec is authoritative for schema, media type, and canonical
|
|
content identity; an external file is not treated as generated merely because
|
|
its bytes decode as the same artifact.
|
|
|
|
## Module Bindings
|
|
|
|
Every module binding may use shorthand:
|
|
|
|
```yaml
|
|
input: seriatim
|
|
```
|
|
|
|
or object form:
|
|
|
|
```yaml
|
|
chunk:
|
|
module: dnd/scenes
|
|
llm_profile: local-fast
|
|
```
|
|
|
|
Binding fields:
|
|
|
|
- `module`: module key.
|
|
- `llm_profile`: optional Scriptorium profile ID. Empty or omitted lets the
|
|
Scriptorium prompt default select the profile.
|
|
- `retries`: non-negative retry count for extra runtime attempts after the
|
|
first attempt. Default: `0`, which permits one initial attempt and no
|
|
additional attempts. Supported on `chunk`, `extract`, `merge`, and
|
|
`normalize` bindings.
|
|
- `options`: optional module-specific settings.
|
|
- `references`: optional reference bindings. Supported only for `chunk`,
|
|
`extract`, `merge`, and `normalize` bindings. `input` and `output` bindings
|
|
reject this field during validation.
|
|
- `validators`: optional stage-local validator chain override. Supported only
|
|
for `chunk`, `extract`, `merge`, and `normalize` bindings. Omit the field to
|
|
use the production default chain; set `validators: []` to force an empty
|
|
chain; set a non-empty list to use exactly those validators in configured
|
|
order.
|
|
|
|
During resolution, each selected module's registered option validator runs.
|
|
Production input, chunk, and output bindings reject unknown or invalid options
|
|
with the affected binding context.
|
|
|
|
### NPC Semantic Normalization
|
|
|
|
The `dnd/npcs` normalizer combines deterministic canonical-name consolidation
|
|
with a document-level LLM identity decision for eligible differently named NPC
|
|
records. Its `llm_profile` and `retries` use the ordinary `normalize` binding
|
|
fields above; `retries: 0` therefore still makes one normalization attempt.
|
|
The normalizer declares no references and accepts no options. Context-window
|
|
selection is current module policy, not configuration.
|
|
|
|
### JSON Output Options
|
|
|
|
The `json` output module accepts only `include_chunk_map`, a boolean that
|
|
defaults to `false`. When `true`, it adds the accepted pipeline-wide chunk map
|
|
to the logical output bundle when one exists. It does not create a CLI flag or
|
|
change output placement.
|
|
|
|
```yaml
|
|
output:
|
|
module: json
|
|
options:
|
|
include_chunk_map: true
|
|
```
|
|
|
|
The payload and index descriptor are defined by the
|
|
[Accepted Chunk Map contract](integrations/chunk-map.md). See the
|
|
[complete D&D configuration](../examples/dnd-complete.config.yml) for a
|
|
copyable pipeline that enables this option.
|
|
|
|
Validator bindings use the same shorthand or object module-binding form, but
|
|
only these fields are supported:
|
|
|
|
- `module`: validator key.
|
|
- `llm_profile`: optional Scriptorium profile ID for LLM-backed validators.
|
|
- `options`: optional validator-specific settings.
|
|
|
|
Validator bindings reject `references`, `retries`, and nested `validators`.
|
|
During resolution, deterministic validators reject explicit `llm_profile`
|
|
values.
|
|
|
|
Configured LLM-backed validators with explicit `llm_profile` values are
|
|
validated against the configured Scriptorium profile source. Deterministic
|
|
production validators do not call the LLM and must not set `llm_profile`.
|
|
|
|
## Implemented Production Modules
|
|
|
|
| Slot | Key | Notes |
|
|
| --- | --- | --- |
|
|
| input | `seriatim` | Reads Seriatim transcript JSON. |
|
|
| chunk | `generic` | Splits source units into ordered chunks. |
|
|
| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. |
|
|
| extract | `dnd/spells` | Extracts typed D&D spell-list artifacts. |
|
|
| extract | `dnd/npcs` | Extracts typed D&D NPC-list artifacts. |
|
|
| extract | `dnd/combat-turns` | Extracts typed D&D combat-turn-list artifacts only for exact combat scene matches. |
|
|
| extract | `dnd/item-events` | Extracts typed, source-grounded D&D item-event-list artifacts. |
|
|
| extract | `dnd/npc-interactions` | Extracts typed D&D NPC-interaction-list artifacts from a supplied NPC registry. |
|
|
| extract | `dnd/scene-descriptions` | Extracts one typed D&D scene description for each accepted chunk. |
|
|
| merge | `appendorder` | Combines typed artifacts in chunk order. |
|
|
| normalize | `noop` | Passes merged typed artifacts through unchanged. |
|
|
| normalize | `dnd/spells` | Deterministically canonicalizes and de-duplicates typed D&D spell-list artifacts. |
|
|
| normalize | `dnd/npcs` | Deterministically prepares typed D&D NPC-list artifacts, then uses an LLM-assisted document-level identity decision to consolidate safe name-based matches and union exact evidence. |
|
|
| normalize | `dnd/combat-turns` | Deterministically canonicalizes, orders, and de-duplicates typed D&D combat-turn artifacts. |
|
|
| normalize | `dnd/item-events` | Deterministically trims, source-orders, and removes only exact D&D item-event duplicates. |
|
|
| normalize | `dnd/npc-interactions` | Canonicalizes registry NPC names, orders interaction occurrences, and removes only exact duplicates. |
|
|
| normalize | `dnd/scene-descriptions` | Validates, orders, trims, and de-duplicates typed D&D scene-description artifacts. |
|
|
| output | `json` | Produces JSON output files for normalized `application/json` lanes and can opt in to an accepted chunk map. |
|
|
|
|
## Implemented Production Validators
|
|
|
|
| Key | Execution | Notes |
|
|
| --- | --- | --- |
|
|
| `generic/always_accept` | deterministic | Accepts returned module output. |
|
|
| `generic/always_reject` | deterministic | Rejects returned module output with reason `always_reject`. |
|
|
| `generic/valid_json` | deterministic | Rejects payloads that are not syntactically valid JSON. |
|
|
| `generic/valid_json_schema` | deterministic | Rejects invalid JSON or JSON that does not conform to the module response schema. |
|
|
| `extract/dnd/spells/shape` | deterministic | Rejects malformed D&D spell-list artifacts. |
|
|
| `extract/dnd/spells/catalog` | deterministic | Rejects spell-list artifacts containing names outside the effective SRD and overlay catalog. |
|
|
| `extract/dnd/spells/source_refs` | deterministic | Rejects missing or invalid D&D spell source references. |
|
|
| `extract/dnd/spells/source_relatedness` | deterministic | Emits warnings when a spell name is not found near its cited source text. |
|
|
| `extract/dnd/npcs/shape` | deterministic | Rejects malformed D&D NPC-list artifacts. |
|
|
| `extract/dnd/npcs/source_refs` | deterministic | Rejects missing or invalid D&D NPC source references. |
|
|
| `extract/dnd/npcs/source_relatedness` | deterministic | Emits warnings when an NPC name is not found near its cited source text. |
|
|
| `normalize/dnd/npcs/identity` | deterministic | Rejects invalid canonical IDs and duplicate canonical-name or ID ownership. |
|
|
| `extract/dnd/combat-turns/shape` | deterministic | Rejects malformed D&D combat-turn artifacts. |
|
|
| `extract/dnd/combat-turns/source_refs` | deterministic | Rejects missing or invalid D&D combat-turn source references. |
|
|
| `extract/dnd/combat-turns/source_relatedness` | deterministic | Emits warnings when an actor is not found near cited source text. |
|
|
| `normalize/dnd/combat-turns/invariants` | deterministic | Rejects normalized combat-turn identity, evidence-order, and chronology violations. |
|
|
| `extract/dnd/item-events/shape` | deterministic | Rejects malformed D&D item-event-list artifacts and invalid category, holder, or quantity combinations. |
|
|
| `extract/dnd/item-events/source_refs` | deterministic | Rejects missing, invalid, or extract-chunk-external D&D item-event source references. |
|
|
| `extract/dnd/item-events/source_relatedness` | deterministic | Emits bounded warnings when an item-event name is not found in cited source text. |
|
|
| `normalize/dnd/item-events/invariants` | deterministic | Rejects non-normalized D&D item-event order, source references, or exact duplicates. |
|
|
| `extract/dnd/npc-interactions/shape` | deterministic | Rejects malformed D&D NPC-interaction-list artifacts. |
|
|
| `extract/dnd/npc-interactions/registry` | deterministic | Rejects interaction names absent from the supplied NPC registry. |
|
|
| `extract/dnd/npc-interactions/source_refs` | deterministic | Rejects missing, invalid, or extract-chunk-external D&D interaction source references. |
|
|
| `extract/dnd/npc-interactions/source_relatedness` | deterministic | Emits bounded warnings when an interaction name is not found in its cited source text. |
|
|
| `normalize/dnd/npc-interactions/invariants` | deterministic | Rejects normalized interaction identity, evidence-order, and chronology violations. |
|
|
| `extract/dnd/scene-descriptions/shape` | deterministic | Rejects malformed D&D scene-description-list artifacts. |
|
|
| `extract/dnd/scene-descriptions/source_refs` | deterministic | Rejects invalid current-source references and extract records not exactly attached to their chunk. |
|
|
| `extract/dnd/scene-descriptions/source_relatedness` | deterministic | Emits bounded advisory warnings for title or summary text not grounded in its cited transcript range. |
|
|
| `normalize/dnd/scene-descriptions/invariants` | deterministic | Rejects non-normalized scene order, exact duplicates, and ID or range conflicts. |
|
|
|
|
The production default chain for `dnd/spells` is used for both its extract and
|
|
normalize stages:
|
|
|
|
```yaml
|
|
validators:
|
|
- generic/valid_json
|
|
- generic/valid_json_schema
|
|
- extract/dnd/spells/shape
|
|
- extract/dnd/spells/catalog
|
|
- extract/dnd/spells/source_refs
|
|
- extract/dnd/spells/source_relatedness
|
|
```
|
|
|
|
The production default chain for `dnd/npcs` uses the extraction chain for the
|
|
extract stage and the identity chain for normalize-stage output:
|
|
|
|
```yaml
|
|
extract:
|
|
validators:
|
|
- generic/valid_json
|
|
- generic/valid_json_schema
|
|
- extract/dnd/npcs/shape
|
|
- extract/dnd/npcs/source_refs
|
|
- extract/dnd/npcs/source_relatedness
|
|
normalize:
|
|
validators:
|
|
- generic/valid_json
|
|
- generic/valid_json_schema
|
|
- normalize/dnd/npcs/identity
|
|
- extract/dnd/npcs/source_refs
|
|
- extract/dnd/npcs/source_relatedness
|
|
```
|
|
|
|
The production default chains for `dnd/combat-turns` are:
|
|
|
|
```yaml
|
|
extract:
|
|
validators:
|
|
- generic/valid_json
|
|
- generic/valid_json_schema
|
|
- extract/dnd/combat-turns/shape
|
|
- extract/dnd/combat-turns/source_refs
|
|
- extract/dnd/combat-turns/source_relatedness
|
|
normalize:
|
|
validators:
|
|
- generic/valid_json
|
|
- generic/valid_json_schema
|
|
- extract/dnd/combat-turns/shape
|
|
- normalize/dnd/combat-turns/invariants
|
|
- extract/dnd/combat-turns/source_refs
|
|
- extract/dnd/combat-turns/source_relatedness
|
|
```
|
|
|
|
The production default chains for `dnd/item-events` are:
|
|
|
|
```yaml
|
|
extract:
|
|
validators:
|
|
- generic/valid_json
|
|
- extract/dnd/item-events/shape
|
|
- extract/dnd/item-events/source_refs
|
|
- generic/valid_json_schema
|
|
- extract/dnd/item-events/source_relatedness
|
|
normalize:
|
|
validators:
|
|
- generic/valid_json
|
|
- extract/dnd/item-events/shape
|
|
- normalize/dnd/item-events/invariants
|
|
- extract/dnd/item-events/source_refs
|
|
- generic/valid_json_schema
|
|
- extract/dnd/item-events/source_relatedness
|
|
```
|
|
|
|
The production default chains for `dnd/npc-interactions` are:
|
|
|
|
```yaml
|
|
extract:
|
|
validators:
|
|
- generic/valid_json
|
|
- extract/dnd/npc-interactions/shape
|
|
- extract/dnd/npc-interactions/registry
|
|
- extract/dnd/npc-interactions/source_refs
|
|
- generic/valid_json_schema
|
|
- extract/dnd/npc-interactions/source_relatedness
|
|
normalize:
|
|
validators:
|
|
- generic/valid_json
|
|
- extract/dnd/npc-interactions/shape
|
|
- extract/dnd/npc-interactions/registry
|
|
- normalize/dnd/npc-interactions/invariants
|
|
- extract/dnd/npc-interactions/source_refs
|
|
- generic/valid_json_schema
|
|
- extract/dnd/npc-interactions/source_relatedness
|
|
```
|
|
|
|
The production default chains for `dnd/scene-descriptions` are:
|
|
|
|
```yaml
|
|
extract:
|
|
validators:
|
|
- generic/valid_json
|
|
- extract/dnd/scene-descriptions/shape
|
|
- extract/dnd/scene-descriptions/source_refs
|
|
- generic/valid_json_schema
|
|
- extract/dnd/scene-descriptions/source_relatedness
|
|
normalize:
|
|
validators:
|
|
- generic/valid_json
|
|
- extract/dnd/scene-descriptions/shape
|
|
- normalize/dnd/scene-descriptions/invariants
|
|
- extract/dnd/scene-descriptions/source_refs
|
|
- generic/valid_json_schema
|
|
- extract/dnd/scene-descriptions/source_relatedness
|
|
```
|
|
|
|
Empty chains approve output by default.
|
|
|
|
The `generic` chunker accepts:
|
|
|
|
- `max_units`: positive integer, default `50`;
|
|
- `overlap_units`: non-negative integer, default `0`, and must be less than
|
|
`max_units`.
|
|
|
|
The `dnd/scenes` chunker requires transcript source capabilities, calls the
|
|
configured structured LLM runtime, and does not accept module options. It
|
|
declares optional `players`, `party`, and `glossary` references for scene
|
|
disambiguation, and accepts `roster` as a deprecated compatibility alias for
|
|
`party`.
|
|
|
|
The `dnd/spells` extractor declares optional campaign reference slots:
|
|
|
|
- `players`
|
|
- `party`
|
|
- `glossary`
|
|
- `roster` as a deprecated compatibility alias for `party`
|
|
|
|
These campaign slots accept UTF-8 plain text, Markdown, YAML, or JSON reference
|
|
files. The extractor also declares an optional `spell_catalog` slot that accepts
|
|
one UTF-8 `application/json` overlay bundle no larger than 1 MiB. The slot does
|
|
not allow multiple files. Its format is defined in the
|
|
[spell-catalog overlay contract](integrations/dnd-spell-catalog-overlays.md).
|
|
The extractor uses campaign references only as supporting disambiguation
|
|
material; spell casts still must be present in the source transcript.
|
|
|
|
It also declares an optional `npcs` slot for a normalized NPC artifact. The
|
|
slot accepts exactly one `application/json` artifact no larger than 1 MiB. An
|
|
external file is decoded and identity-validated during preparation. A
|
|
generated binding is validated at the step handoff and is provided to the
|
|
operation through the same reference contract. In both cases, the model
|
|
receives a names-only JSON projection for caster-name grounding. Registry source references
|
|
may belong to the NPC-producing session and are provenance only; they are not
|
|
spell evidence. Generated reference identity and bounded producer provenance
|
|
are recorded by the framework; NPC names, content, and paths are not copied
|
|
into manifests. Consumer-local checkpoint identity uses the names-only
|
|
projection digest. When absent, the prompt receives the exact empty value
|
|
`{"npcs":[]}` with its projection digest and no registry provenance.
|
|
|
|
The `dnd/spells` normalizer declares the same optional `spell_catalog` slot.
|
|
When an overlay is used, bind it independently under
|
|
`artifacts.<lane>.normalize.references.spell_catalog`; normalize-stage
|
|
references are local to that stage and are not inherited from extraction. The
|
|
normalizer uses the embedded SRD catalog when no normalize-stage overlay is
|
|
bound.
|
|
|
|
The `dnd/npcs` extractor declares the same optional campaign slots as the spell
|
|
extractor, but it does not declare the `npcs` registry slot. Its normalizer
|
|
accepts no references. The
|
|
[complete D&D example](../examples/dnd-complete.config.yml) binds its accepted
|
|
normalized output to later spell, combat-turn, and NPC-interaction targets
|
|
through an explicit ordered step.
|
|
|
|
The `dnd/item-events` extractor accepts optional `glossary`, `party`, `players`,
|
|
and deprecated `roster` campaign slots for disambiguation only. It requires no
|
|
generated artifact and its normalizer accepts no references, so the complete
|
|
example keeps this independent lane in the first step. Current transcript
|
|
references remain the only item-event evidence. Its complete durable behavior
|
|
is defined in the [D&D item-event artifact contract](integrations/dnd-item-event-artifacts.md).
|
|
|
|
The `dnd/scene-descriptions` extractor accepts only the optional `players`,
|
|
`party`, and `glossary` campaign slots. They can disambiguate terms in the
|
|
model-facing title and summary but are never source evidence. Its normalizer
|
|
accepts no references. No NPC registry or generated artifact is required. See
|
|
the [D&D scene-description artifact contract](integrations/dnd-scene-description-artifacts.md)
|
|
and the copyable [complete D&D configuration](../examples/dnd-complete.config.yml).
|
|
|
|
The `dnd/combat-turns` extractor declares the optional campaign slots, the
|
|
optional structured `npcs` slot, and the required `scene_descriptions` slot.
|
|
Campaign references guide only the LLM extraction stage. The deterministic
|
|
normalizer declares only `npcs`, whose operation-time registry supports the
|
|
same actor canonicalization. Each `npcs` slot accepts exactly one UTF-8
|
|
`application/json` artifact no larger than 1 MiB. The registry's source ranges
|
|
remain provenance for the reference and never become combat evidence.
|
|
|
|
`scene_descriptions` accepts exactly one approved
|
|
`dnd/scene-description-list` artifact with media type `application/json`, no
|
|
larger than 1 MiB (1048576 bytes). Pipeline resolution rejects a combat
|
|
extractor whose required slot is unbound. The scene artifact is eligibility
|
|
control context only; it is neither prompt input nor combat evidence, and the
|
|
combat normalizer does not accept it.
|
|
|
|
An ordered step binds the generated NPC artifact to extraction and normalization
|
|
and the generated scene-description artifact to combat extraction:
|
|
|
|
```yaml
|
|
references:
|
|
npcs:
|
|
artifact:
|
|
step: identify-npcs
|
|
lane: npcs
|
|
scene_descriptions:
|
|
artifact:
|
|
step: identify-npcs
|
|
lane: scene-descriptions
|
|
```
|
|
|
|
When `npcs` is bound, the combat extractor and normalizer receive the generated
|
|
NPC registry at operation time. Framework provenance and checkpoint dependencies
|
|
contain its kind, schema identity, media type, canonical digest, size, and
|
|
bounded producer identity; names, content, and paths are not recorded there.
|
|
When absent, the combat prompt receives the exact empty NPC registry value
|
|
`{"npcs":[]}` with its projection digest and no registry provenance. The
|
|
generated `scene_descriptions` artifact is supplied only to combat extraction.
|
|
The complete example uses the same explicit scene binding; an external approved
|
|
scene-description artifact may be used instead when the workflow crosses a
|
|
process or session boundary.
|
|
|
|
Both `dnd/npc-interactions` stages require the structured `npcs` slot. Bind one
|
|
accepted normalized NPC artifact from an earlier ordered step; the interaction
|
|
extractor receives only its names-only projection, while the normalizer uses
|
|
the immutable registry for exact canonical-name lookup. A missing, rejected,
|
|
or incompatible producer prevents the interaction step from running. Registry
|
|
source references remain identity provenance and never supply interaction
|
|
evidence: every interaction must cite current transcript units. The complete
|
|
durable contract and two-step configuration are defined in the
|
|
[D&D NPC interaction artifact contract](integrations/dnd-npc-interaction-artifacts.md).
|
|
|
|
## State Surfaces
|
|
|
|
The `output`, `cache`, and `debug` top-level fields select independent physical
|
|
roots. Their layout, permissions, lifecycle, and sensitive-data handling are
|
|
defined in [Operations](operations.md).
|
|
|
|
```yaml
|
|
output:
|
|
directory: ./notarius-output
|
|
cache:
|
|
chunk_plans:
|
|
directory: ""
|
|
mode: auto
|
|
checkpoints:
|
|
enabled: false
|
|
directory: ""
|
|
debug:
|
|
directory: ./notarius-debug
|
|
```
|
|
|
|
`output.directory` is the durable output root. Its precedence is
|
|
`--output-dir`, `NOTARIUS_OUTPUT_DIR`, the file value, then the default.
|
|
|
|
`cache.chunk_plans.mode` accepts `auto`, `bypass`, or `refresh`. Its precedence
|
|
is `--chunk_cache`, `NOTARIUS_CACHE_CHUNK_PLANS_MODE`, the file value, then
|
|
`auto`. `auto` reuses a valid source-addressed plan and regenerates missing or
|
|
invalid records; `bypass` performs no plan-cache I/O; `refresh` regenerates and
|
|
publishes a plan after chunk validation.
|
|
|
|
`cache.chunk_plans.directory` and `cache.checkpoints.directory` each name an
|
|
exact cache-family root. Their precedence is the corresponding environment
|
|
variable, the file value, then the family-specific per-user default. There is
|
|
no CLI cache-root override. The defaults are
|
|
`<os.UserCacheDir>/notarius/chunk-plans` and
|
|
`<os.UserCacheDir>/notarius/checkpoints`; on Unix, `os.UserCacheDir` ordinarily
|
|
uses an absolute `$XDG_CACHE_HOME` or falls back to `$HOME/.cache`. A relative
|
|
`XDG_CACHE_HOME` is an error.
|
|
|
|
`cache.checkpoints.enabled` defaults to `false`. When `true`, every run records
|
|
checkpoint transitions and reusable approved results. When `false`, Notarius
|
|
does not resolve or create the checkpoint root, and `--resume` is rejected.
|
|
The `--resume` flag authorizes loading compatible checkpoints; it does not
|
|
control recording.
|
|
|
|
`debug.directory` chooses a root but never enables debug capture. Its precedence
|
|
is `--debug-dir`, `NOTARIUS_DEBUG_DIR`, the file value, then the default.
|
|
Only `--debug` requests a bundle; `--debug-dir` is valid only with `--debug`.
|
|
|
|
Every supplied file, environment, and CLI value is validated even when a
|
|
higher-precedence value wins.
|
|
|
|
## Version 2 To Version 3 Migration
|
|
|
|
Version 2 files are rejected. Move each setting to the surface it controls and
|
|
remove obsolete enablement and retention controls. This complete before/after
|
|
example preserves an existing chunk-plan cache and checkpoint directory while
|
|
choosing an output and debug root explicitly.
|
|
|
|
```yaml
|
|
# Version 2 (no longer accepted)
|
|
version: 2
|
|
workspace:
|
|
directory: /srv/notarius/state
|
|
resume:
|
|
enabled: true
|
|
debug:
|
|
enabled: true
|
|
chunk_cache:
|
|
directory: /srv/notarius/chunk-plans
|
|
mode: auto
|
|
diagnostics:
|
|
retention: always
|
|
diagnostics:
|
|
work_dir: /srv/notarius/inspection
|
|
```
|
|
|
|
```yaml
|
|
# Version 3
|
|
version: 3
|
|
output:
|
|
directory: /srv/notarius/output
|
|
cache:
|
|
chunk_plans:
|
|
directory: /srv/notarius/chunk-plans
|
|
mode: auto
|
|
checkpoints:
|
|
enabled: true
|
|
directory: /srv/notarius/state/checkpoints
|
|
debug:
|
|
directory: /srv/notarius/debug
|
|
```
|
|
|
|
Run the migrated configuration with `--resume` when checkpoint reuse is wanted,
|
|
and with `--debug` when a debug bundle is wanted. Enabled checkpoint recording
|
|
occurs with or without `--resume`.
|
|
|
|
The removed fields are `workspace.directory`, `workspace.resume.enabled`,
|
|
`workspace.debug.enabled`, `workspace.chunk_cache.mode`,
|
|
`workspace.chunk_cache.directory`, `workspace.diagnostics.enabled`,
|
|
`workspace.diagnostics.retention`, `diagnostics.work_dir`, and
|
|
`diagnostics.retention`. The removed environment variables are `NOTARIUS_WORKSPACE_DIR`,
|
|
`NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`,
|
|
`NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`,
|
|
`NOTARIUS_WORKSPACE_RESUME_ENABLED`, `NOTARIUS_WORKSPACE_DEBUG_ENABLED`,
|
|
`NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`,
|
|
`NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`, `NOTARIUS_WORK_DIR`, and
|
|
`NOTARIUS_DIAGNOSTICS_RETENTION`. The chunk-cache variables are replaced by
|
|
`NOTARIUS_CACHE_CHUNK_PLANS_MODE` and
|
|
`NOTARIUS_CACHE_CHUNK_PLANS_DIR`; the former shared directory has no direct
|
|
replacement.
|
|
|
|
## Validation
|
|
|
|
Configuration validation checks:
|
|
|
|
- supported config version and known YAML fields;
|
|
- mutually exclusive `scriptorium.profile_dir` and `scriptorium.profile_file`;
|
|
- non-empty, non-duplicated IDs after trimming;
|
|
- positive global LLM concurrency;
|
|
- supported stage-worker keys and an effective extract worker count in the
|
|
inclusive range `1..concurrency.total_llm`;
|
|
- non-empty output and debug directories;
|
|
- a supported chunk-cache mode, Boolean checkpoint enablement, and state-surface
|
|
directories without NUL bytes;
|
|
- stale removed fields such as `llm_profiles`.
|
|
|
|
Pipeline resolution additionally checks:
|
|
|
|
- the pipeline ID exists;
|
|
- at least one artifact lane is declared and selected in each explicit step;
|
|
- `artifacts` and `steps` are mutually exclusive, explicit steps are non-empty,
|
|
and step IDs are unique after trimming;
|
|
- lanes selected through the CLI exist in the resolved pipeline;
|
|
- lane IDs are globally unique across ordered steps;
|
|
- required module keys are present;
|
|
- module keys are registered for the expected slot;
|
|
- module capability requirements are satisfied;
|
|
- non-empty validator overrides reference registered validator keys;
|
|
- deterministic validators do not set `llm_profile`;
|
|
- LLM-backed validators with explicit `llm_profile` values reference configured
|
|
Scriptorium profile IDs;
|
|
- bound reference slots are declared by selected chunk, extractor, merger, or
|
|
normalizer targets;
|
|
- generated references identify one lane in an earlier step, use a declared
|
|
compatible artifact kind, and do not conflict with external or target-local
|
|
generated bindings;
|
|
- required reference slots are bound for selected targets.
|