From 59cbf1eb27cebfb65823c5234e049ec10a3aecc1 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 26 Jul 2026 13:16:06 +0000 Subject: [PATCH] Rewrite configuration reference --- docs/config.md | 1065 ++++++++++++++---------------------------------- 1 file changed, 295 insertions(+), 770 deletions(-) diff --git a/docs/config.md b/docs/config.md index 6e2686f..9c2f027 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1,824 +1,349 @@ # Configuration -This is the canonical reference for implemented Notarius configuration. +This is the canonical reference for Notarius configuration. Configuration files +are YAML and must declare version 3. They select pipelines and their modules; +the [CLI reference](cli.md) owns invocation syntax, and +[Operations](operations.md) owns run-state procedures. -Notarius reads YAML config files with `version: 3`. File configuration is -applied over built-in defaults, then environment overrides are applied. Explicit -CLI overrides are applied last where the command supports them. +## Configuration Discovery And Precedence -## Discovery +Commands that load configuration choose a file in this order: -Commands that load configuration use this order: +1. a non-empty **--config** CLI value; +2. a non-empty **NOTARIUS_CONFIG** environment value; +3. the installed default file at **/usr/local/etc/notarius/config.yml**, when + it exists. -1. an explicit path supplied through the CLI, when provided; -2. `NOTARIUS_CONFIG`, when set to a non-empty path; -3. `/usr/local/etc/notarius/config.yml`. +The command fails if none of these paths provides a configuration file. -If none is available, the command fails with a config file not found error. -The explicit-path option is defined in the [CLI reference](cli.md). +For configuration values, precedence is: + +1. built-in defaults; +2. the selected YAML file; +3. supported operational environment variables; and +4. the CLI run overrides that apply to a command. + +Environment variables do not provide a second configuration schema. They only +override the fields listed below. ## Maintained Examples -- [Minimal D&D configuration](../examples/dnd-minimal.config.yml) shows the - shortest useful single-step, single-lane pipeline. -- [Complete D&D configuration](../examples/dnd-complete.config.yml) shows scene - chunking, ordered steps, all six D&D artifact lanes, generated NPC and - scene-description handoffs, campaign and spell-catalog references, checkpoint - storage, and chunk-map output. +- [Minimal D&D configuration](../examples/dnd-minimal.config.yml) is a + single-lane Seriatim-to-spell pipeline. +- [Complete D&D configuration](../examples/dnd-complete.config.yml) uses + ordered steps, all implemented D&D lanes, generated references, state + settings, and bounded LLM concurrency. -Both are complete version 3 files. The fragments below illustrate individual -fields and are not alternate complete configurations. +Use these complete files as starting points rather than combining the +illustrative fragments in this reference. -## Top-Level Fields +## File Shape And Defaults -- `version`: required. The only supported value is `3`. -- `scriptorium`: optional Scriptorium profile source settings. -- `pipelines`: optional map of pipeline IDs to pipeline definitions. -- `concurrency`: optional global concurrency settings. -- `output`: optional durable output placement. -- `cache`: optional chunk-plan and checkpoint cache placement. -- `debug`: optional debug-bundle placement. It does not enable debug capture. +Unknown fields, duplicate mapping keys, empty identifiers, and identifiers that +become duplicates after trimming whitespace are rejected. Every top-level field +other than **version** is optional. -Unknown YAML fields are rejected. The removed top-level `llm_profiles` field is -rejected; execution profiles now come from Scriptorium. +| Field | Type | Default | Rules | +| --- | --- | --- | --- | +| **version** | integer | none | Required; must be 3. | +| **scriptorium** | object | none | Profile source configuration. | +| **pipelines** | map | empty | Maps pipeline IDs to pipeline definitions. | +| **concurrency** | object | see below | Global LLM and extraction limits. | +| **output** | object | see below | Published output settings. | +| **cache** | object | see below | Chunk-plan and checkpoint settings. | +| **debug** | object | see below | Debug-bundle root only; it does not enable capture. | -## Defaults +Built-in defaults are: -Built-in defaults: +| Field | Default | +| --- | --- | +| **concurrency.total_llm** | 1 | +| **concurrency.stage_workers.extract** | Effective **total_llm** | +| **output.directory** | **./notarius-output** | +| **cache.chunk_plans.mode** | **auto** | +| **cache.chunk_plans.directory** | Empty, selecting the per-user chunk-plan root | +| **cache.checkpoints.enabled** | false | +| **cache.checkpoints.directory** | Empty, selecting the per-user checkpoint root | +| **debug.directory** | **./notarius-debug** | -- `concurrency.total_llm`: `1` -- `concurrency.stage_workers.extract`: effective `concurrency.total_llm` -- `output.directory`: `./notarius-output` -- `cache.chunk_plans.mode`: `auto` -- `cache.chunk_plans.directory`: unset, selecting - `/notarius/chunk-plans` -- `cache.checkpoints.enabled`: `false` -- `cache.checkpoints.directory`: unset, selecting - `/notarius/checkpoints` -- `debug.directory`: `./notarius-debug` - -No pipelines are built in. A run requires a configured pipeline. - -If `scriptorium` is omitted, Notarius uses Scriptorium's built-in profile -catalog. Prompt definitions may also name default profile IDs. The current D&D -scene, extraction, and NPC-normalization prompts default to `gemini-2-flash` -when a module binding does not set `llm_profile`. The selected built-in profile -reads its credentials from its Scriptorium profile definition. +An empty cache directory in YAML deliberately selects the corresponding +per-user root. An explicit empty output or debug directory is invalid. ## Scriptorium Profiles -`scriptorium` fields: +The optional **scriptorium** object selects one source of profile definitions: -- `profile_dir`: optional directory containing Scriptorium profile YAML files. -- `profile_file`: optional Scriptorium profile YAML file. - -`profile_dir` and `profile_file` are mutually exclusive. Custom profiles -overlay Scriptorium built-in profiles by profile ID. - -Scriptorium profile files use Scriptorium's profile schema. A minimal profile -looks like: - -```yaml -id: local-fast -endpoint: http://127.0.0.1:8080/v1 -model: your-model -api_key_env: SCRIPTORIUM_API_KEY -timeout_seconds: 180 -``` - -Notarius does not accept raw API keys in Notarius config. For file-backed -Scriptorium profiles, store the environment variable name in `api_key_env` and -set that variable in the run environment. Scriptorium rejects raw `api_key` -fields in profile YAML. - -## Environment Overrides - -These environment variables are applied after the config file: - -- `NOTARIUS_CONFIG`: config discovery path. -- `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency. -- `NOTARIUS_STAGE_WORKERS_EXTRACT`: integer extract worker limit. -- `NOTARIUS_OUTPUT_DIR`: durable output root. -- `NOTARIUS_CACHE_CHUNK_PLANS_MODE`: chunk-plan cache mode. -- `NOTARIUS_CACHE_CHUNK_PLANS_DIR`: chunk-plan cache root. -- `NOTARIUS_CACHE_CHECKPOINTS_DIR`: checkpoint cache root. -- `NOTARIUS_DEBUG_DIR`: debug-bundle root. - -Integer environment values must parse as base-10 integers. Directory overrides -must be non-empty after trimming. Cache-directory fields in a file may be -empty, which deliberately selects the corresponding per-user default. - -The removed `NOTARIUS_LLM_DEFAULT_*` variables are not read. Configure provider -endpoint, model, and credential environment variable names through Scriptorium -profiles. - -## Concurrency - -`concurrency` fields: - -- `total_llm`: positive integer ceiling on concurrent provider calls. -- `stage_workers`: optional map of framework worker limits. The only supported - key is `extract`. - -`stage_workers.extract` defaults to the effective `total_llm` value after file -and environment precedence. It must be between `1` and `total_llm`, inclusive. -Unknown or empty stage-worker keys are rejected. The environment override -`NOTARIUS_STAGE_WORKERS_EXTRACT` takes precedence over the file value, as does -`NOTARIUS_TOTAL_LLM_CONCURRENCY` for the global ceiling. - -The worker value is present in effective and redacted configuration. It bounds -the fixed run-wide extract pool and its bounded dispatch queue. Extract jobs are -submitted by source chunk and then resolved lane; `total_llm` independently -bounds actual provider calls made by extracts, retries, and validators. - -## Pipelines - -A pipeline selects implementations for the fixed workflow defined by -[Architecture](policy/architecture.md#system-shape). - -Pipeline fields: - -- `input`: required module binding. -- `chunk`: optional module binding. Default module is `generic`. -- `artifacts`: the artifact lane map for a single-step pipeline. It is treated - as an implicit step with the stable ID `default`. -- `steps`: an ordered, non-empty list of step definitions. A pipeline may use - `steps` or `artifacts`, but not both. Step IDs must be unique after trimming. -- `output`: optional module binding. Default module is `json`. -- `references`: optional map of reference slot names to reference paths. These - bindings are defaults for eligible pipeline targets that declare the matching - slot. - -Each explicit step contains an `id`, an optional `references` map, and a -non-empty `artifacts` map. Steps share the pipeline input, chunk plan, worker -budget, output encoder, manifest, and failure boundary. Lanes within a step -retain the fixed extract, validate, merge, validate, normalize, and validate -workflow; the next step starts only after the current step is terminal. - -Generated references use the structured `artifact` source form to identify one -accepted normalized lane from an earlier step: - -```yaml -pipelines: - dnd-npc-grounded: - input: seriatim - steps: - - id: identify-npcs - artifacts: - npcs: - extract: dnd/npcs - normalize: dnd/npcs - scene-descriptions: - extract: dnd/scene-descriptions - normalize: dnd/scene-descriptions - - id: grounded-events - references: - npcs: - artifact: - step: identify-npcs - lane: npcs - scene_descriptions: - artifact: - step: identify-npcs - lane: scene-descriptions - artifacts: - spells: - extract: dnd/spells - normalize: dnd/spells - combat: - extract: dnd/combat-turns - normalize: dnd/combat-turns -``` - -The generated binding is explicit and typed; it is not inferred from module, -lane, or slot names. It may be declared at step scope, applying to every -selected target in that step that declares the slot, or at one target's -`references` map. A producer may fan out to compatible target slots, but a -slot accepts only one producer. A producer must be in an earlier step, and a -configured generated dependency is required even when the consumer slot is -otherwise optional. Aggregating several producer artifacts is unsupported. - -The producer codec supplies the artifact kind, complete schema identity, media -type, canonical content digest, and size used for compatibility and checkpoint -dependency checks. Only one accepted normalized artifact crosses the boundary; -raw extraction results, rejected output, intermediate values, and validator -diagnostics do not. Generated content is supplied in memory and is never -represented by a filesystem path. - -Artifact lane fields: - -- `extract`: required module binding. -- `merge`: optional module binding. Default module is `appendorder`. -- `normalize`: optional module binding. Default module is `noop`. -- `validators`: deprecated lane-level validator list. Non-empty lists are - rejected; use `extract.validators`, `merge.validators`, or - `normalize.validators`. -- `references`: optional compatibility alias for extractor reference bindings. - Lane bindings override pipeline-level bindings for the same slot. - -Commands that resolve a pipeline fail for unknown or incompatible module keys. -See [CLI Reference](cli.md) for command syntax. - -Reference bindings are validated against reference slots declared by eligible -chunk, extract, merge, and normalize targets during pipeline resolution. Required slots -must be bound after config defaults, target-local references, lane-level -compatibility bindings, step-local references, and command-line reference -overrides are applied. Config-relative paths are resolved relative to the -config file; command-line reference paths are resolved relative to the current -working directory. Bound files must be UTF-8 text. Reference media types are -inferred from file extensions and checked when a module restricts accepted -types; unknown extensions use `application/octet-stream`. See [CLI Reference](cli.md#run) -for command-line selectors and [Operations](operations.md) for recorded -provenance and sensitive-data handling. - -Pipeline-level `references` are defaults. They are valid when at least one -eligible target in the full configured pipeline declares the slot, including -chunk, extractor, merger, and normalizer targets. During a run, they apply only -to the selected targets that declare the slot. For external bindings, a -step-local binding overrides a pipeline-level default and a target-local -binding retains the existing most-specific precedence. Generated and external -bindings may not resolve to the same target slot, and a step-scoped generated -binding may not be duplicated by a target-local generated binding. - -```yaml -pipelines: - dnd-session: - input: seriatim - references: - players: ./campaign/players.txt - party: ./campaign/party-roster.txt - glossary: ./campaign/glossary.txt - artifacts: - spells: - extract: dnd/spells -``` - -Extractor binding `references` are the canonical lane-local location. The -legacy lane-level `references` field remains supported as an alias; when both -bind the same slot, `extract.references` wins: - -```yaml -pipelines: - dnd-session: - input: seriatim - references: - glossary: ./campaign/glossary.txt - artifacts: - spells: - references: - roster: ./campaign/legacy-roster.txt - extract: - module: dnd/spells - references: - party: ./campaign/session-party.txt -``` - -`chunk.references`, `merge.references`, and `normalize.references` are accepted -in object-form bindings. They override pipeline-level defaults for slots -declared by that target module. Extractor-local references apply only to the -extractor, merger-local references apply only to the merger, and -normalizer-local references apply only to the normalizer. - -Target-local reference fields use the same map shape at: - -- `pipelines..chunk.references` -- `pipelines..artifacts..extract.references` -- `pipelines..artifacts..merge.references` -- `pipelines..artifacts..normalize.references` - -Each binding is valid only when that target module declares the slot. - -Reference source forms are: - -- a scalar string, which is an external file path; or -- an object with only `artifact`, containing trimmed `step` and `lane` IDs for - an earlier producer lane. - -Pipeline-level references accept only external paths. Generated references are -valid at step scope or on a target-local `references` map. Their producer's -registered codec is authoritative for schema, media type, and canonical -content identity; an external file is not treated as generated merely because -its bytes decode as the same artifact. - -## Module Bindings - -Every module binding may use shorthand: - -```yaml -input: seriatim -``` - -or object form: - -```yaml -chunk: - module: dnd/scenes - llm_profile: local-fast -``` - -Binding fields: - -- `module`: module key. -- `llm_profile`: optional Scriptorium profile ID. Empty or omitted lets the - Scriptorium prompt default select the profile. -- `retries`: non-negative retry count for extra runtime attempts after the - first attempt. Default: `0`, which permits one initial attempt and no - additional attempts. Supported on `chunk`, `extract`, `merge`, and - `normalize` bindings. -- `options`: optional module-specific settings. -- `references`: optional reference bindings. Supported only for `chunk`, - `extract`, `merge`, and `normalize` bindings. `input` and `output` bindings - reject this field during validation. -- `validators`: optional stage-local validator chain override. Supported only - for `chunk`, `extract`, `merge`, and `normalize` bindings. Omit the field to - use the production default chain; set `validators: []` to force an empty - chain; set a non-empty list to use exactly those validators in configured - order. - -During resolution, each selected module's registered option validator runs. -Production input, chunk, and output bindings reject unknown or invalid options -with the affected binding context. - -### NPC Semantic Normalization - -The `dnd/npcs` normalizer combines deterministic canonical-name consolidation -with a document-level LLM identity decision for eligible differently named NPC -records. Its `llm_profile` and `retries` use the ordinary `normalize` binding -fields above; `retries: 0` therefore still makes one normalization attempt. -The normalizer declares no references and accepts no options. Context-window -selection is current module policy, not configuration. - -### JSON Output Options - -The `json` output module accepts only `include_chunk_map`, a boolean that -defaults to `false`. When `true`, it adds the accepted pipeline-wide chunk map -to the logical output bundle when one exists. It does not create a CLI flag or -change output placement. - -```yaml -output: - module: json - options: - include_chunk_map: true -``` - -The payload and index descriptor are defined by the -[Accepted Chunk Map contract](integrations/chunk-map.md). See the -[complete D&D configuration](../examples/dnd-complete.config.yml) for a -copyable pipeline that enables this option. - -Validator bindings use the same shorthand or object module-binding form, but -only these fields are supported: - -- `module`: validator key. -- `llm_profile`: optional Scriptorium profile ID for LLM-backed validators. -- `options`: optional validator-specific settings. - -Validator bindings reject `references`, `retries`, and nested `validators`. -During resolution, deterministic validators reject explicit `llm_profile` -values. - -Configured LLM-backed validators with explicit `llm_profile` values are -validated against the configured Scriptorium profile source. Deterministic -production validators do not call the LLM and must not set `llm_profile`. - -## Implemented Production Modules - -| Slot | Key | Notes | +| Field | Type | Rules | | --- | --- | --- | -| input | `seriatim` | Reads Seriatim transcript JSON. | -| chunk | `generic` | Splits source units into ordered chunks. | -| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. | -| extract | `dnd/spells` | Extracts typed D&D spell-list artifacts. | -| extract | `dnd/npcs` | Extracts typed D&D NPC-list artifacts. | -| extract | `dnd/combat-turns` | Extracts typed D&D combat-turn-list artifacts only for exact combat scene matches. | -| extract | `dnd/item-events` | Extracts typed, source-grounded D&D item-event-list artifacts. | -| extract | `dnd/npc-interactions` | Extracts typed D&D NPC-interaction-list artifacts from a supplied NPC registry. | -| extract | `dnd/scene-descriptions` | Extracts one typed D&D scene description for each accepted chunk. | -| merge | `appendorder` | Combines typed artifacts in chunk order. | -| normalize | `noop` | Passes merged typed artifacts through unchanged. | -| normalize | `dnd/spells` | Deterministically canonicalizes and de-duplicates typed D&D spell-list artifacts. | -| normalize | `dnd/npcs` | Deterministically prepares typed D&D NPC-list artifacts, then uses an LLM-assisted document-level identity decision to consolidate safe name-based matches and union exact evidence. | -| normalize | `dnd/combat-turns` | Deterministically canonicalizes, orders, and de-duplicates typed D&D combat-turn artifacts. | -| normalize | `dnd/item-events` | Deterministically trims, source-orders, and removes only exact D&D item-event duplicates. | -| normalize | `dnd/npc-interactions` | Canonicalizes registry NPC names, orders interaction occurrences, and removes only exact duplicates. | -| normalize | `dnd/scene-descriptions` | Validates, orders, trims, and de-duplicates typed D&D scene-description artifacts. | -| output | `json` | Produces JSON output files for normalized `application/json` lanes and can opt in to an accepted chunk map. | +| **profile_dir** | string | Non-empty directory containing profile files. | +| **profile_file** | string | Non-empty profile file. | -## Implemented Production Validators +Set at most one of these fields. Profile IDs used by a binding must be available +from the selected Scriptorium profile source when the pipeline is resolved. +Keep credentials out of this file: configure a profile to read its credential +from an environment variable, then set that environment variable only in the +run environment. -| Key | Execution | Notes | +## Operational Environment Variables + +These variables are applied after YAML values: + +| Variable | Overrides | Rules | | --- | --- | --- | -| `generic/always_accept` | deterministic | Accepts returned module output. | -| `generic/always_reject` | deterministic | Rejects returned module output with reason `always_reject`. | -| `generic/valid_json` | deterministic | Rejects payloads that are not syntactically valid JSON. | -| `generic/valid_json_schema` | deterministic | Rejects invalid JSON or JSON that does not conform to the module response schema. | -| `extract/dnd/spells/shape` | deterministic | Rejects malformed D&D spell-list artifacts. | -| `extract/dnd/spells/catalog` | deterministic | Rejects spell-list artifacts containing names outside the effective SRD and overlay catalog. | -| `extract/dnd/spells/source_refs` | deterministic | Rejects missing or invalid D&D spell source references. | -| `extract/dnd/spells/source_relatedness` | deterministic | Emits warnings when a spell name is not found near its cited source text. | -| `extract/dnd/npcs/shape` | deterministic | Rejects malformed D&D NPC-list artifacts. | -| `extract/dnd/npcs/source_refs` | deterministic | Rejects missing or invalid D&D NPC source references. | -| `extract/dnd/npcs/source_relatedness` | deterministic | Emits warnings when an NPC name is not found near its cited source text. | -| `normalize/dnd/npcs/identity` | deterministic | Rejects invalid canonical IDs and duplicate canonical-name or ID ownership. | -| `extract/dnd/combat-turns/shape` | deterministic | Rejects malformed D&D combat-turn artifacts. | -| `extract/dnd/combat-turns/source_refs` | deterministic | Rejects missing or invalid D&D combat-turn source references. | -| `extract/dnd/combat-turns/source_relatedness` | deterministic | Emits warnings when an actor is not found near cited source text. | -| `normalize/dnd/combat-turns/invariants` | deterministic | Rejects normalized combat-turn identity, evidence-order, and chronology violations. | -| `extract/dnd/item-events/shape` | deterministic | Rejects malformed D&D item-event-list artifacts and invalid category, holder, or quantity combinations. | -| `extract/dnd/item-events/source_refs` | deterministic | Rejects missing, invalid, or extract-chunk-external D&D item-event source references. | -| `extract/dnd/item-events/source_relatedness` | deterministic | Emits bounded warnings when an item-event name is not found in cited source text. | -| `normalize/dnd/item-events/invariants` | deterministic | Rejects non-normalized D&D item-event order, source references, or exact duplicates. | -| `extract/dnd/npc-interactions/shape` | deterministic | Rejects malformed D&D NPC-interaction-list artifacts. | -| `extract/dnd/npc-interactions/registry` | deterministic | Rejects interaction names absent from the supplied NPC registry. | -| `extract/dnd/npc-interactions/source_refs` | deterministic | Rejects missing, invalid, or extract-chunk-external D&D interaction source references. | -| `extract/dnd/npc-interactions/source_relatedness` | deterministic | Emits bounded warnings when an interaction name is not found in its cited source text. | -| `normalize/dnd/npc-interactions/invariants` | deterministic | Rejects normalized interaction identity, evidence-order, and chronology violations. | -| `extract/dnd/scene-descriptions/shape` | deterministic | Rejects malformed D&D scene-description-list artifacts. | -| `extract/dnd/scene-descriptions/source_refs` | deterministic | Rejects invalid current-source references and extract records not exactly attached to their chunk. | -| `extract/dnd/scene-descriptions/source_relatedness` | deterministic | Emits bounded advisory warnings for title or summary text not grounded in its cited transcript range. | -| `normalize/dnd/scene-descriptions/invariants` | deterministic | Rejects non-normalized scene order, exact duplicates, and ID or range conflicts. | +| **NOTARIUS_TOTAL_LLM_CONCURRENCY** | **concurrency.total_llm** | Integer. | +| **NOTARIUS_STAGE_WORKERS_EXTRACT** | **concurrency.stage_workers.extract** | Integer. | +| **NOTARIUS_OUTPUT_DIR** | **output.directory** | Non-empty path. | +| **NOTARIUS_CACHE_CHUNK_PLANS_MODE** | **cache.chunk_plans.mode** | **auto**, **bypass**, or **refresh**. | +| **NOTARIUS_CACHE_CHUNK_PLANS_DIR** | **cache.chunk_plans.directory** | Non-empty path. | +| **NOTARIUS_CACHE_CHECKPOINTS_DIR** | **cache.checkpoints.directory** | Non-empty path. | +| **NOTARIUS_DEBUG_DIR** | **debug.directory** | Non-empty path. | -The production default chain for `dnd/spells` is used for both its extract and -normalize stages: +Integer values are trimmed then parsed as base-10 integers. Directory and +output values reject NUL characters. **NOTARIUS_CONFIG** participates only in +configuration discovery. -```yaml -validators: - - generic/valid_json - - generic/valid_json_schema - - extract/dnd/spells/shape - - extract/dnd/spells/catalog - - extract/dnd/spells/source_refs - - extract/dnd/spells/source_relatedness -``` +## Concurrency, Output, Cache, And Debug -The production default chain for `dnd/npcs` uses the extraction chain for the -extract stage and the identity chain for normalize-stage output: - -```yaml -extract: - validators: - - generic/valid_json - - generic/valid_json_schema - - extract/dnd/npcs/shape - - extract/dnd/npcs/source_refs - - extract/dnd/npcs/source_relatedness -normalize: - validators: - - generic/valid_json - - generic/valid_json_schema - - normalize/dnd/npcs/identity - - extract/dnd/npcs/source_refs - - extract/dnd/npcs/source_relatedness -``` - -The production default chains for `dnd/combat-turns` are: - -```yaml -extract: - validators: - - generic/valid_json - - generic/valid_json_schema - - extract/dnd/combat-turns/shape - - extract/dnd/combat-turns/source_refs - - extract/dnd/combat-turns/source_relatedness -normalize: - validators: - - generic/valid_json - - generic/valid_json_schema - - extract/dnd/combat-turns/shape - - normalize/dnd/combat-turns/invariants - - extract/dnd/combat-turns/source_refs - - extract/dnd/combat-turns/source_relatedness -``` - -The production default chains for `dnd/item-events` are: - -```yaml -extract: - validators: - - generic/valid_json - - extract/dnd/item-events/shape - - extract/dnd/item-events/source_refs - - generic/valid_json_schema - - extract/dnd/item-events/source_relatedness -normalize: - validators: - - generic/valid_json - - extract/dnd/item-events/shape - - normalize/dnd/item-events/invariants - - extract/dnd/item-events/source_refs - - generic/valid_json_schema - - extract/dnd/item-events/source_relatedness -``` - -The production default chains for `dnd/npc-interactions` are: - -```yaml -extract: - validators: - - generic/valid_json - - extract/dnd/npc-interactions/shape - - extract/dnd/npc-interactions/registry - - extract/dnd/npc-interactions/source_refs - - generic/valid_json_schema - - extract/dnd/npc-interactions/source_relatedness -normalize: - validators: - - generic/valid_json - - extract/dnd/npc-interactions/shape - - extract/dnd/npc-interactions/registry - - normalize/dnd/npc-interactions/invariants - - extract/dnd/npc-interactions/source_refs - - generic/valid_json_schema - - extract/dnd/npc-interactions/source_relatedness -``` - -The production default chains for `dnd/scene-descriptions` are: - -```yaml -extract: - validators: - - generic/valid_json - - extract/dnd/scene-descriptions/shape - - extract/dnd/scene-descriptions/source_refs - - generic/valid_json_schema - - extract/dnd/scene-descriptions/source_relatedness -normalize: - validators: - - generic/valid_json - - extract/dnd/scene-descriptions/shape - - normalize/dnd/scene-descriptions/invariants - - extract/dnd/scene-descriptions/source_refs - - generic/valid_json_schema - - extract/dnd/scene-descriptions/source_relatedness -``` - -Empty chains approve output by default. - -The `generic` chunker accepts: - -- `max_units`: positive integer, default `50`; -- `overlap_units`: non-negative integer, default `0`, and must be less than - `max_units`. - -The `dnd/scenes` chunker requires transcript source capabilities, calls the -configured structured LLM runtime, and does not accept module options. It -declares optional `players`, `party`, and `glossary` references for scene -disambiguation, and accepts `roster` as a deprecated compatibility alias for -`party`. - -The `dnd/spells` extractor declares optional campaign reference slots: - -- `players` -- `party` -- `glossary` -- `roster` as a deprecated compatibility alias for `party` - -These campaign slots accept UTF-8 plain text, Markdown, YAML, or JSON reference -files. The extractor also declares an optional `spell_catalog` slot that accepts -one UTF-8 `application/json` overlay bundle no larger than 1 MiB. The slot does -not allow multiple files. Its format is defined in the -[spell-catalog overlay contract](integrations/dnd-spell-catalog-overlays.md). -The extractor uses campaign references only as supporting disambiguation -material; spell casts still must be present in the source transcript. - -It also declares an optional `npcs` slot for a normalized NPC artifact. The -slot accepts exactly one `application/json` artifact no larger than 1 MiB. An -external file is decoded and identity-validated during preparation. A -generated binding is validated at the step handoff and is provided to the -operation through the same reference contract. In both cases, the model -receives a names-only JSON projection for caster-name grounding. Registry source references -may belong to the NPC-producing session and are provenance only; they are not -spell evidence. Generated reference identity and bounded producer provenance -are recorded by the framework; NPC names, content, and paths are not copied -into manifests. Consumer-local checkpoint identity uses the names-only -projection digest. When absent, the prompt receives the exact empty value -`{"npcs":[]}` with its projection digest and no registry provenance. - -The `dnd/spells` normalizer declares the same optional `spell_catalog` slot. -When an overlay is used, bind it independently under -`artifacts..normalize.references.spell_catalog`; normalize-stage -references are local to that stage and are not inherited from extraction. The -normalizer uses the embedded SRD catalog when no normalize-stage overlay is -bound. - -The `dnd/npcs` extractor declares the same optional campaign slots as the spell -extractor, but it does not declare the `npcs` registry slot. Its normalizer -accepts no references. The -[complete D&D example](../examples/dnd-complete.config.yml) binds its accepted -normalized output to later spell, combat-turn, and NPC-interaction targets -through an explicit ordered step. - -The `dnd/item-events` extractor accepts optional `glossary`, `party`, `players`, -and deprecated `roster` campaign slots for disambiguation only. It requires no -generated artifact and its normalizer accepts no references, so the complete -example keeps this independent lane in the first step. Current transcript -references remain the only item-event evidence. Its complete durable behavior -is defined in the [D&D item-event artifact contract](integrations/dnd-item-event-artifacts.md). - -The `dnd/scene-descriptions` extractor accepts only the optional `players`, -`party`, and `glossary` campaign slots. They can disambiguate terms in the -model-facing title and summary but are never source evidence. Its normalizer -accepts no references. No NPC registry or generated artifact is required. See -the [D&D scene-description artifact contract](integrations/dnd-scene-description-artifacts.md) -and the copyable [complete D&D configuration](../examples/dnd-complete.config.yml). - -The `dnd/combat-turns` extractor declares the optional campaign slots, the -optional structured `npcs` slot, and the required `scene_descriptions` slot. -Campaign references guide only the LLM extraction stage. The deterministic -normalizer declares only `npcs`, whose operation-time registry supports the -same actor canonicalization. Each `npcs` slot accepts exactly one UTF-8 -`application/json` artifact no larger than 1 MiB. The registry's source ranges -remain provenance for the reference and never become combat evidence. - -`scene_descriptions` accepts exactly one approved -`dnd/scene-description-list` artifact with media type `application/json`, no -larger than 1 MiB (1048576 bytes). Pipeline resolution rejects a combat -extractor whose required slot is unbound. The scene artifact is eligibility -control context only; it is neither prompt input nor combat evidence, and the -combat normalizer does not accept it. - -An ordered step binds the generated NPC artifact to extraction and normalization -and the generated scene-description artifact to combat extraction: - -```yaml -references: - npcs: - artifact: - step: identify-npcs - lane: npcs - scene_descriptions: - artifact: - step: identify-npcs - lane: scene-descriptions -``` - -When `npcs` is bound, the combat extractor and normalizer receive the generated -NPC registry at operation time. Framework provenance and checkpoint dependencies -contain its kind, schema identity, media type, canonical digest, size, and -bounded producer identity; names, content, and paths are not recorded there. -When absent, the combat prompt receives the exact empty NPC registry value -`{"npcs":[]}` with its projection digest and no registry provenance. The -generated `scene_descriptions` artifact is supplied only to combat extraction. -The complete example uses the same explicit scene binding; an external approved -scene-description artifact may be used instead when the workflow crosses a -process or session boundary. - -Both `dnd/npc-interactions` stages require the structured `npcs` slot. Bind one -accepted normalized NPC artifact from an earlier ordered step; the interaction -extractor receives only its names-only projection, while the normalizer uses -the immutable registry for exact canonical-name lookup. A missing, rejected, -or incompatible producer prevents the interaction step from running. Registry -source references remain identity provenance and never supply interaction -evidence: every interaction must cite current transcript units. The complete -durable contract and two-step configuration are defined in the -[D&D NPC interaction artifact contract](integrations/dnd-npc-interaction-artifacts.md). - -## State Surfaces - -The `output`, `cache`, and `debug` top-level fields select independent physical -roots. Their layout, permissions, lifecycle, and sensitive-data handling are -defined in [Operations](operations.md). - -```yaml +~~~yaml +concurrency: + total_llm: 2 + stage_workers: + extract: 2 output: directory: ./notarius-output cache: chunk_plans: - directory: "" mode: auto + directory: ./notarius-cache/chunk-plans checkpoints: - enabled: false - directory: "" + enabled: true + directory: ./notarius-cache/checkpoints debug: directory: ./notarius-debug -``` +~~~ -`output.directory` is the durable output root. Its precedence is -`--output-dir`, `NOTARIUS_OUTPUT_DIR`, the file value, then the default. +**concurrency.total_llm** must be greater than zero. The only supported +**concurrency.stage_workers** key is **extract**; its value must be from 1 +through **total_llm**. When omitted, it is recalculated from the effective +**total_llm** after YAML and environment precedence. -`cache.chunk_plans.mode` accepts `auto`, `bypass`, or `refresh`. Its precedence -is `--chunk_cache`, `NOTARIUS_CACHE_CHUNK_PLANS_MODE`, the file value, then -`auto`. `auto` reuses a valid source-addressed plan and regenerates missing or -invalid records; `bypass` performs no plan-cache I/O; `refresh` regenerates and -publishes a plan after chunk validation. +**cache.chunk_plans.mode** accepts **auto**, **bypass**, or **refresh**. +**cache.checkpoints.enabled** is a boolean. The CLI can override the output +directory and chunk-plan mode for one run; see [CLI reference](cli.md#run). -`cache.chunk_plans.directory` and `cache.checkpoints.directory` each name an -exact cache-family root. Their precedence is the corresponding environment -variable, the file value, then the family-specific per-user default. There is -no CLI cache-root override. The defaults are -`/notarius/chunk-plans` and -`/notarius/checkpoints`; on Unix, `os.UserCacheDir` ordinarily -uses an absolute `$XDG_CACHE_HOME` or falls back to `$HOME/.cache`. A relative -`XDG_CACHE_HOME` is an error. +## Pipelines -`cache.checkpoints.enabled` defaults to `false`. When `true`, every run records -checkpoint transitions and reusable approved results. When `false`, Notarius -does not resolve or create the checkpoint root, and `--resume` is rejected. -The `--resume` flag authorizes loading compatible checkpoints; it does not -control recording. +Each **pipelines** entry has a unique, non-empty ID and the following shape: -`debug.directory` chooses a root but never enables debug capture. Its precedence -is `--debug-dir`, `NOTARIUS_DEBUG_DIR`, the file value, then the default. -Only `--debug` requests a bundle; `--debug-dir` is valid only with `--debug`. +~~~yaml +pipelines: + dnd-session: + input: seriatim + chunk: generic + output: json + artifacts: + spells: + extract: dnd/spells + merge: appendorder + normalize: dnd/spells +~~~ -Every supplied file, environment, and CLI value is validated even when a -higher-precedence value wins. +| Field | Type | Default | Rules | +| --- | --- | --- | --- | +| **input** | module binding | none | Required. | +| **chunk** | module binding | **generic** | Optional. | +| **output** | module binding | **json** | Optional. | +| **artifacts** | map | none | Compact single-step lane map. | +| **steps** | list | none | Ordered lane definitions. Mutually exclusive with **artifacts**. | +| **references** | map | none | External reference defaults for eligible targets. | -## Version 2 To Version 3 Migration +Use either **artifacts** or **steps**. The compact **artifacts** form is an +implicit single step. An explicit **steps** list must be non-empty; every step +needs a unique non-empty **id**, an **artifacts** map, and may have +**references**. A lane ID must not appear more than once in a pipeline, +including across explicit steps. -Version 2 files are rejected. Move each setting to the surface it controls and -remove obsolete enablement and retention controls. This complete before/after -example preserves an existing chunk-plan cache and checkpoint directory while -choosing an output and debug root explicitly. +A lane has these fields: -```yaml -# Version 2 (no longer accepted) -version: 2 -workspace: - directory: /srv/notarius/state - resume: - enabled: true - debug: - enabled: true - chunk_cache: - directory: /srv/notarius/chunk-plans - mode: auto - diagnostics: - retention: always -diagnostics: - work_dir: /srv/notarius/inspection -``` +| Field | Type | Default | Rules | +| --- | --- | --- | --- | +| **extract** | module binding | none | Required. | +| **merge** | module binding | **appendorder** | Optional. | +| **normalize** | module binding | **noop** | Optional. | +| **references** | map | none | Supported compatibility alias for **extract.references**. | +| **validators** | list | none | Non-empty lane-level lists are rejected. Set validator overrides on a binding instead. | -```yaml -# Version 3 -version: 3 +The lane-level **references** alias remains accepted. When the alias and +**extract.references** bind the same slot, **extract.references** wins. Use +the binding-local form in new configurations. + +## Module Bindings And Validators + +Use a module key directly when no other binding fields are needed: + +~~~yaml +input: seriatim +~~~ + +Use an object for fields: + +~~~yaml +extract: + module: dnd/spells + llm_profile: gemini-2-flash + retries: 2 + references: + spell_catalog: ./dnd-spell-catalog.json +~~~ + +| Binding field | Type | Default | Rules | +| --- | --- | --- | --- | +| **module** | string | none | Required for an object binding. Must be a registered compatible key. | +| **llm_profile** | string | none | Optional non-empty Scriptorium profile ID. | +| **retries** | integer | 0 | Non-negative additional attempts for chunk, extract, merge, and normalize bindings. | +| **options** | object | none | Must satisfy the selected module. | +| **references** | map | none | Valid only on chunk, extract, merge, and normalize bindings. | +| **validators** | list | production chain | Valid only on chunk, extract, merge, and normalize bindings. | + +Omitting **validators** uses the registered chain. **validators: []** selects +an empty chain; a non-empty list replaces the chain in the listed order. +Validator bindings accept only **module**, **llm_profile**, and **options**. +They reject **references**, **retries**, and nested **validators**. Deterministic +validators reject an explicit **llm_profile**. + +The **json** output module accepts one option: + +~~~yaml output: - directory: /srv/notarius/output -cache: - chunk_plans: - directory: /srv/notarius/chunk-plans - mode: auto - checkpoints: - enabled: true - directory: /srv/notarius/state/checkpoints -debug: - directory: /srv/notarius/debug -``` + module: json + options: + include_chunk_map: true +~~~ -Run the migrated configuration with `--resume` when checkpoint reuse is wanted, -and with `--debug` when a debug bundle is wanted. Enabled checkpoint recording -occurs with or without `--resume`. +**include_chunk_map** is a boolean and defaults to false. It adds the accepted +chunk map when one exists; its wire format is defined in the +[chunk-map contract](integrations/chunk-map.md). -The removed fields are `workspace.directory`, `workspace.resume.enabled`, -`workspace.debug.enabled`, `workspace.chunk_cache.mode`, -`workspace.chunk_cache.directory`, `workspace.diagnostics.enabled`, -`workspace.diagnostics.retention`, `diagnostics.work_dir`, and -`diagnostics.retention`. The removed environment variables are `NOTARIUS_WORKSPACE_DIR`, -`NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`, -`NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`, -`NOTARIUS_WORKSPACE_RESUME_ENABLED`, `NOTARIUS_WORKSPACE_DEBUG_ENABLED`, -`NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`, -`NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`, `NOTARIUS_WORK_DIR`, and -`NOTARIUS_DIAGNOSTICS_RETENTION`. The chunk-cache variables are replaced by -`NOTARIUS_CACHE_CHUNK_PLANS_MODE` and -`NOTARIUS_CACHE_CHUNK_PLANS_DIR`; the former shared directory has no direct -replacement. +## References And Ordered Handoffs + +Reference maps bind named slots that the selected target declares. A scalar is +an external path. Pipeline-level maps accept only external paths; step-local +and binding-local maps may also select a normalized artifact from an earlier +step: + +~~~yaml +steps: + - id: describe-session + artifacts: + npcs: + extract: dnd/npcs + normalize: dnd/npcs + - id: extract-events + references: + npcs: + artifact: + step: describe-session + lane: npcs + artifacts: + spells: + extract: dnd/spells + normalize: dnd/spells +~~~ + +An artifact selector contains only **step** and **lane**. The producer must be +an earlier step and the selected artifact must be compatible with the consumer +slot. A generated binding supplies one accepted normalized artifact; it does +not name a file. A configured generated dependency remains required even when +that consumer slot is otherwise optional. + +Pipeline references are defaults. A matching step-local or binding-local +external path overrides a pipeline default. Required slots must be bound after +these configuration values and any CLI reference overrides are applied. +Reference paths in YAML are resolved relative to the configuration file. + +### D&D Reference Slots + +The following slot names are accepted by the implemented D&D modules when the +selected target declares them: + +| Slot | Source and use | +| --- | --- | +| **party** | Optional text campaign context. This is the canonical party-roster spelling. | +| **roster** | Accepted compatibility alias for **party**. | +| **players** | Optional text player context. | +| **glossary** | Optional text campaign glossary. | +| **spell_catalog** | Optional JSON spell-catalog overlay for spell extraction and normalization. See [spell-catalog overlays](integrations/dnd-spell-catalog-overlays.md). | +| **npcs** | Normalized NPC registry. Optional for spells and combat turns; required for NPC interactions. | +| **scene_descriptions** | Required normalized scene-description artifact for combat-turn extraction. | + +Scene descriptions accept **party**, **players**, and **glossary**, but not +**roster**. NPC interactions require **npcs** for both extraction and +normalization. Combat turns require **scene_descriptions** for extraction; the +normalized combat-turn module may use optional **npcs**. The complete example +shows generated **npcs** and **scene_descriptions** bindings. + +## Production Module Keys + +| Kind | Keys | +| --- | --- | +| Input | **seriatim** | +| Chunk | **generic**, **dnd/scenes** | +| Extract | **dnd/spells**, **dnd/npcs**, **dnd/combat-turns**, **dnd/item-events**, **dnd/npc-interactions**, **dnd/scene-descriptions** | +| Merge | **appendorder** | +| Normalize | **noop**, **dnd/spells**, **dnd/npcs**, **dnd/combat-turns**, **dnd/item-events**, **dnd/npc-interactions**, **dnd/scene-descriptions** | +| Output | **json** | + +The D&D artifact contracts define each emitted schema: +[spells](integrations/dnd-spell-artifacts.md), +[NPCs](integrations/dnd-npc-artifacts.md), +[NPC interactions](integrations/dnd-npc-interaction-artifacts.md), +[combat turns](integrations/dnd-combat-turn-artifacts.md), +[item events](integrations/dnd-item-event-artifacts.md), and +[scene descriptions](integrations/dnd-scene-description-artifacts.md). + +## Production Validator Keys And Default Chains + +Available validator keys are: + +| Family | Keys | +| --- | --- | +| Generic | **generic/always_accept**, **generic/always_reject**, **generic/valid_json**, **generic/valid_json_schema** | +| Spells | **extract/dnd/spells/shape**, **extract/dnd/spells/catalog**, **extract/dnd/spells/source_refs**, **extract/dnd/spells/source_relatedness** | +| NPCs | **extract/dnd/npcs/shape**, **extract/dnd/npcs/source_refs**, **extract/dnd/npcs/source_relatedness**, **normalize/dnd/npcs/identity** | +| Combat turns | **extract/dnd/combat-turns/shape**, **extract/dnd/combat-turns/source_refs**, **extract/dnd/combat-turns/source_relatedness**, **normalize/dnd/combat-turns/invariants** | +| Item events | **extract/dnd/item-events/shape**, **extract/dnd/item-events/source_refs**, **extract/dnd/item-events/source_relatedness**, **normalize/dnd/item-events/invariants** | +| NPC interactions | **extract/dnd/npc-interactions/shape**, **extract/dnd/npc-interactions/registry**, **extract/dnd/npc-interactions/source_refs**, **extract/dnd/npc-interactions/source_relatedness**, **normalize/dnd/npc-interactions/invariants** | +| Scene descriptions | **extract/dnd/scene-descriptions/shape**, **extract/dnd/scene-descriptions/source_refs**, **extract/dnd/scene-descriptions/source_relatedness**, **normalize/dnd/scene-descriptions/invariants** | + +When no override is configured, production D&D bindings use the following +ordered chains. Each row lists extract then normalize; spell chains are the +same at both stages. + +| Lane | Extract | Normalize | +| --- | --- | --- | +| Spells | generic/valid_json, extract/dnd/spells/shape, extract/dnd/spells/catalog, extract/dnd/spells/source_refs, generic/valid_json_schema, extract/dnd/spells/source_relatedness | Same as extract | +| NPCs | generic/valid_json, extract/dnd/npcs/shape, extract/dnd/npcs/source_refs, generic/valid_json_schema, extract/dnd/npcs/source_relatedness | generic/valid_json, extract/dnd/npcs/shape, normalize/dnd/npcs/identity, extract/dnd/npcs/source_refs, generic/valid_json_schema, extract/dnd/npcs/source_relatedness | +| Combat turns | generic/valid_json, extract/dnd/combat-turns/shape, extract/dnd/combat-turns/source_refs, generic/valid_json_schema, extract/dnd/combat-turns/source_relatedness | generic/valid_json, extract/dnd/combat-turns/shape, normalize/dnd/combat-turns/invariants, extract/dnd/combat-turns/source_refs, generic/valid_json_schema, extract/dnd/combat-turns/source_relatedness | +| Item events | generic/valid_json, extract/dnd/item-events/shape, extract/dnd/item-events/source_refs, generic/valid_json_schema, extract/dnd/item-events/source_relatedness | generic/valid_json, extract/dnd/item-events/shape, normalize/dnd/item-events/invariants, extract/dnd/item-events/source_refs, generic/valid_json_schema, extract/dnd/item-events/source_relatedness | +| NPC interactions | generic/valid_json, extract/dnd/npc-interactions/shape, extract/dnd/npc-interactions/registry, extract/dnd/npc-interactions/source_refs, generic/valid_json_schema, extract/dnd/npc-interactions/source_relatedness | generic/valid_json, extract/dnd/npc-interactions/shape, extract/dnd/npc-interactions/registry, normalize/dnd/npc-interactions/invariants, extract/dnd/npc-interactions/source_refs, generic/valid_json_schema, extract/dnd/npc-interactions/source_relatedness | +| Scene descriptions | generic/valid_json, extract/dnd/scene-descriptions/shape, extract/dnd/scene-descriptions/source_refs, generic/valid_json_schema, extract/dnd/scene-descriptions/source_relatedness | generic/valid_json, extract/dnd/scene-descriptions/shape, normalize/dnd/scene-descriptions/invariants, extract/dnd/scene-descriptions/source_refs, generic/valid_json_schema, extract/dnd/scene-descriptions/source_relatedness | + +Chains are only registered for the D&D extract and normalize modules shown +above; select an explicit override when a different compatible chain is +required. ## Validation -Configuration validation checks: +Validate a file and one pipeline before running it: -- supported config version and known YAML fields; -- mutually exclusive `scriptorium.profile_dir` and `scriptorium.profile_file`; -- non-empty, non-duplicated IDs after trimming; -- positive global LLM concurrency; -- supported stage-worker keys and an effective extract worker count in the - inclusive range `1..concurrency.total_llm`; -- non-empty output and debug directories; -- a supported chunk-cache mode, Boolean checkpoint enablement, and state-surface - directories without NUL bytes; -- stale removed fields such as `llm_profiles`. +~~~sh +go run ./cmd/notarius config validate \ + --config examples/dnd-minimal.config.yml \ + --pipeline dnd-session +~~~ -Pipeline resolution additionally checks: - -- the pipeline ID exists; -- at least one artifact lane is declared and selected in each explicit step; -- `artifacts` and `steps` are mutually exclusive, explicit steps are non-empty, - and step IDs are unique after trimming; -- lanes selected through the CLI exist in the resolved pipeline; -- lane IDs are globally unique across ordered steps; -- required module keys are present; -- module keys are registered for the expected slot; -- module capability requirements are satisfied; -- non-empty validator overrides reference registered validator keys; -- deterministic validators do not set `llm_profile`; -- LLM-backed validators with explicit `llm_profile` values reference configured - Scriptorium profile IDs; -- bound reference slots are declared by selected chunk, extractor, merger, or - normalizer targets; -- generated references identify one lane in an earlier step, use a declared - compatible artifact kind, and do not conflict with external or target-local - generated bindings; -- required reference slots are bound for selected targets. +Configuration validation rejects invalid YAML, unsupported fields, invalid +defaults or environment overrides, incompatible module keys, unknown options, +invalid reference bindings, missing required reference slots, invalid validator +overrides, and incompatible generated artifact handoffs. Use +[pipelines list](cli.md#pipelines-list) to inspect configured IDs.