notarius/docs/config.md

# Configuration

This is the canonical reference for implemented Notarius configuration.

Notarius reads YAML config files with `version: 3`. File configuration is
applied over built-in defaults, then environment overrides are applied. Explicit
CLI overrides are applied last where the command supports them.

## Discovery

Commands that load configuration use this order:

1. an explicit path supplied through the CLI, when provided;
2. `NOTARIUS_CONFIG`, when set to a non-empty path;
3. `/usr/local/etc/notarius/config.yml`.

If none is available, the command fails with a config file not found error.
The explicit-path option is defined in the [CLI reference](cli.md).

## Maintained Examples

- [Minimal D&D spell configuration](../examples/dnd-spells.config.yml)
- [Production-oriented D&D spell configuration](../examples/dnd-spells-production.config.yml)
- [D&D NPC configuration](../examples/dnd-npcs.config.yml)
- [Sequential D&D NPC and spell configuration](../examples/dnd-npc-spell-sequential.config.yml)
- [D&D combat-turn configuration](../examples/dnd-combat-turns.config.yml)
- [Sequential D&D NPC and combat-turn configuration](../examples/dnd-npc-combat-sequential.config.yml)

All are complete version 3 files. The fragments below illustrate individual
fields and are not alternate complete configurations.

## Top-Level Fields

- `version`: required. The only supported value is `3`.
- `scriptorium`: optional Scriptorium profile source settings.
- `pipelines`: optional map of pipeline IDs to pipeline definitions.
- `concurrency`: optional global concurrency settings.
- `output`: optional durable output placement.
- `cache`: optional chunk-plan and checkpoint cache placement.
- `debug`: optional debug-bundle placement. It does not enable debug capture.

Unknown YAML fields are rejected. The removed top-level `llm_profiles` field is
rejected; execution profiles now come from Scriptorium.

## Defaults

Built-in defaults:

- `concurrency.total_llm`: `1`
- `concurrency.stage_workers.extract`: effective `concurrency.total_llm`
- `output.directory`: `./notarius-output`
- `cache.chunk_plans.mode`: `auto`
- `cache.chunk_plans.directory`: unset, selecting
  `<os.UserCacheDir>/notarius/chunk-plans`
- `cache.checkpoints.enabled`: `false`
- `cache.checkpoints.directory`: unset, selecting
  `<os.UserCacheDir>/notarius/checkpoints`
- `debug.directory`: `./notarius-debug`

No pipelines are built in. A run requires a configured pipeline.

If `scriptorium` is omitted, Notarius uses Scriptorium's built-in profile
catalog. Prompt definitions may also name default profile IDs. The current D&D
scene and spell prompts default to the built-in `mistral-small-3` profile when a
module binding does not set `llm_profile`. That built-in profile reads its
credential from `OPENROUTER_API_KEY`.

## Scriptorium Profiles

`scriptorium` fields:

- `profile_dir`: optional directory containing Scriptorium profile YAML files.
- `profile_file`: optional Scriptorium profile YAML file.

`profile_dir` and `profile_file` are mutually exclusive. Custom profiles
overlay Scriptorium built-in profiles by profile ID.

Scriptorium profile files use Scriptorium's profile schema. A minimal profile
looks like:

```yaml
id: local-fast
endpoint: http://127.0.0.1:8080/v1
model: your-model
api_key_env: SCRIPTORIUM_API_KEY
timeout_seconds: 180
```

Notarius does not accept raw API keys in Notarius config. For file-backed
Scriptorium profiles, store the environment variable name in `api_key_env` and
set that variable in the run environment. Scriptorium rejects raw `api_key`
fields in profile YAML.

## Environment Overrides

These environment variables are applied after the config file:

- `NOTARIUS_CONFIG`: config discovery path.
- `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency.
- `NOTARIUS_STAGE_WORKERS_EXTRACT`: integer extract worker limit.
- `NOTARIUS_OUTPUT_DIR`: durable output root.
- `NOTARIUS_CACHE_CHUNK_PLANS_MODE`: chunk-plan cache mode.
- `NOTARIUS_CACHE_CHUNK_PLANS_DIR`: chunk-plan cache root.
- `NOTARIUS_CACHE_CHECKPOINTS_DIR`: checkpoint cache root.
- `NOTARIUS_DEBUG_DIR`: debug-bundle root.

Integer environment values must parse as base-10 integers. Directory overrides
must be non-empty after trimming. Cache-directory fields in a file may be
empty, which deliberately selects the corresponding per-user default.

The removed `NOTARIUS_LLM_DEFAULT_*` variables are not read. Configure provider
endpoint, model, and credential environment variable names through Scriptorium
profiles.

## Concurrency

`concurrency` fields:

- `total_llm`: positive integer ceiling on concurrent provider calls.
- `stage_workers`: optional map of framework worker limits. The only supported
  key is `extract`.

`stage_workers.extract` defaults to the effective `total_llm` value after file
and environment precedence. It must be between `1` and `total_llm`, inclusive.
Unknown or empty stage-worker keys are rejected. The environment override
`NOTARIUS_STAGE_WORKERS_EXTRACT` takes precedence over the file value, as does
`NOTARIUS_TOTAL_LLM_CONCURRENCY` for the global ceiling.

The worker value is present in effective and redacted configuration. It bounds
the fixed run-wide extract pool and its bounded dispatch queue. Extract jobs are
submitted by source chunk and then resolved lane; `total_llm` independently
bounds actual provider calls made by extracts, retries, and validators.

## Pipelines

A pipeline selects implementations for the fixed workflow defined by
[Architecture](policy/architecture.md#system-shape).

Pipeline fields:

- `input`: required module binding.
- `chunk`: optional module binding. Default module is `generic`.
- `artifacts`: required for pipeline resolution. It maps artifact lane IDs to
  lane definitions.
- `output`: optional module binding. Default module is `json`.
- `references`: optional map of reference slot names to reference paths. These
  bindings are defaults for eligible pipeline targets that declare the matching
  slot.

Artifact lane fields:

- `extract`: required module binding.
- `merge`: optional module binding. Default module is `appendorder`.
- `normalize`: optional module binding. Default module is `noop`.
- `validators`: deprecated lane-level validator list. Non-empty lists are
  rejected; use `extract.validators`, `merge.validators`, or
  `normalize.validators`.
- `references`: optional compatibility alias for extractor reference bindings.
  Lane bindings override pipeline-level bindings for the same slot.

Commands that resolve a pipeline fail for unknown or incompatible module keys.
See [CLI Reference](cli.md) for command syntax.

Reference bindings are validated against reference slots declared by eligible
chunk, extract, merge, and normalize targets during pipeline resolution. Required slots
must be bound after config defaults, target-local references, lane-level
compatibility bindings, and command-line reference overrides are applied.
Config-relative paths are resolved relative to the config file; command-line
reference paths are resolved relative to the current working directory. Bound
files must be UTF-8 text. Reference media types are inferred from file
extensions and checked when a module restricts accepted types; unknown
extensions use `application/octet-stream`. See [CLI Reference](cli.md#run) for
command-line selectors and [Operations](operations.md) for recorded provenance
and sensitive-data handling.

Pipeline-level `references` are defaults. They are valid when at least one
eligible target in the full configured pipeline declares the slot, including
chunk, extractor, merger, and normalizer targets. During a run, they apply only
to the selected targets that declare the slot:

```yaml
pipelines:
  dnd-session:
    input: seriatim
    references:
      players: ./campaign/players.txt
      party: ./campaign/party-roster.txt
      glossary: ./campaign/glossary.txt
    artifacts:
      spells:
        extract: dnd/spells
```

Extractor binding `references` are the canonical lane-local location. The
legacy lane-level `references` field remains supported as an alias; when both
bind the same slot, `extract.references` wins:

```yaml
pipelines:
  dnd-session:
    input: seriatim
    references:
      glossary: ./campaign/glossary.txt
    artifacts:
      spells:
        references:
          roster: ./campaign/legacy-roster.txt
        extract:
          module: dnd/spells
          references:
            party: ./campaign/session-party.txt
```

`chunk.references`, `merge.references`, and `normalize.references` are accepted
in object-form bindings. They override pipeline-level defaults for slots
declared by that target module. Extractor-local references apply only to the
extractor, merger-local references apply only to the merger, and
normalizer-local references apply only to the normalizer.

Target-local reference fields use the same map shape at:

- `pipelines.<id>.chunk.references`
- `pipelines.<id>.artifacts.<lane>.extract.references`
- `pipelines.<id>.artifacts.<lane>.merge.references`
- `pipelines.<id>.artifacts.<lane>.normalize.references`

Each binding is valid only when that target module declares the slot.

## Module Bindings

Every module binding may use shorthand:

```yaml
input: seriatim
```

or object form:

```yaml
chunk:
  module: dnd/scenes
  llm_profile: local-fast
```

Binding fields:

- `module`: module key.
- `llm_profile`: optional Scriptorium profile ID. Empty or omitted lets the
  Scriptorium prompt default select the profile.
- `retries`: non-negative retry count for extra runtime attempts after the
  first attempt. Default: `0`. Supported on `chunk`, `extract`, `merge`, and
  `normalize` bindings.
- `options`: optional module-specific settings.
- `references`: optional reference bindings. Supported only for `chunk`,
  `extract`, `merge`, and `normalize` bindings. `input` and `output` bindings
  reject this field during validation.
- `validators`: optional stage-local validator chain override. Supported only
  for `chunk`, `extract`, `merge`, and `normalize` bindings. Omit the field to
  use the production default chain; set `validators: []` to force an empty
  chain; set a non-empty list to use exactly those validators in configured
  order.

During resolution, each selected module's registered option validator runs.
Production input, chunk, and output bindings reject unknown or invalid options
with the affected binding context.

Validator bindings use the same shorthand or object module-binding form, but
only these fields are supported:

- `module`: validator key.
- `llm_profile`: optional Scriptorium profile ID for LLM-backed validators.
- `options`: optional validator-specific settings.

Validator bindings reject `references`, `retries`, and nested `validators`.
During resolution, deterministic validators reject explicit `llm_profile`
values.

Configured LLM-backed validators with explicit `llm_profile` values are
validated against the configured Scriptorium profile source. Deterministic
production validators do not call the LLM and must not set `llm_profile`.

## Implemented Production Modules

| Slot | Key | Notes |
| --- | --- | --- |
| input | `seriatim` | Reads Seriatim transcript JSON. |
| chunk | `generic` | Splits source units into ordered chunks. |
| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. |
| extract | `dnd/spells` | Extracts typed D&D spell-list artifacts. |
| extract | `dnd/npcs` | Extracts typed D&D NPC-list artifacts. |
| extract | `dnd/combat-turns` | Extracts typed D&D combat-turn-list artifacts. |
| merge | `appendorder` | Combines typed artifacts in chunk order. |
| normalize | `noop` | Passes merged typed artifacts through unchanged. |
| normalize | `dnd/spells` | Deterministically canonicalizes and de-duplicates typed D&D spell-list artifacts. |
| normalize | `dnd/npcs` | Deterministically consolidates typed D&D NPC-list artifacts by canonical identity and aliases. |
| normalize | `dnd/combat-turns` | Deterministically canonicalizes, orders, and de-duplicates typed D&D combat-turn artifacts. |
| output | `json` | Produces JSON output files for normalized `application/json` lanes. |

## Implemented Production Validators

| Key | Execution | Notes |
| --- | --- | --- |
| `generic/always_accept` | deterministic | Accepts returned module output. |
| `generic/always_reject` | deterministic | Rejects returned module output with reason `always_reject`. |
| `generic/valid_json` | deterministic | Rejects payloads that are not syntactically valid JSON. |
| `generic/valid_json_schema` | deterministic | Rejects invalid JSON or JSON that does not conform to the module response schema. |
| `extract/dnd/spells/shape` | deterministic | Rejects malformed D&D spell-list artifacts. |
| `extract/dnd/spells/catalog` | deterministic | Rejects spell-list artifacts containing names outside the effective SRD and overlay catalog. |
| `extract/dnd/spells/source_refs` | deterministic | Rejects missing or invalid D&D spell source references. |
| `extract/dnd/spells/source_relatedness` | deterministic | Emits warnings when a spell name is not found near its cited source text. |
| `extract/dnd/npcs/shape` | deterministic | Rejects malformed D&D NPC-list artifacts. |
| `extract/dnd/npcs/source_refs` | deterministic | Rejects missing or invalid D&D NPC source references. |
| `extract/dnd/npcs/source_relatedness` | deterministic | Emits warnings when an NPC name or alias is not found near its cited source text. |
| `normalize/dnd/npcs/identity` | deterministic | Rejects invalid canonical IDs, aliases, and cross-record identity collisions. |
| `extract/dnd/combat-turns/shape` | deterministic | Rejects malformed D&D combat-turn artifacts. |
| `extract/dnd/combat-turns/source_refs` | deterministic | Rejects missing or invalid D&D combat-turn source references. |
| `extract/dnd/combat-turns/source_relatedness` | deterministic | Emits warnings when an actor or declared action is not found near cited source text. |
| `normalize/dnd/combat-turns/invariants` | deterministic | Rejects normalized combat-turn identity, target, evidence-order, and chronology violations. |

The production default chain for `dnd/spells` is used for both its extract and
normalize stages:

```yaml
validators:
  - generic/valid_json
  - generic/valid_json_schema
  - extract/dnd/spells/shape
  - extract/dnd/spells/catalog
  - extract/dnd/spells/source_refs
  - extract/dnd/spells/source_relatedness
```

The production default chain for `dnd/npcs` uses the extraction chain for the
extract stage and the identity chain for normalize-stage output:

```yaml
extract:
  validators:
    - generic/valid_json
    - generic/valid_json_schema
    - extract/dnd/npcs/shape
    - extract/dnd/npcs/source_refs
    - extract/dnd/npcs/source_relatedness
normalize:
  validators:
    - generic/valid_json
    - generic/valid_json_schema
    - normalize/dnd/npcs/identity
    - extract/dnd/npcs/source_refs
    - extract/dnd/npcs/source_relatedness
```

The production default chains for `dnd/combat-turns` are:

```yaml
extract:
  validators:
    - generic/valid_json
    - generic/valid_json_schema
    - extract/dnd/combat-turns/shape
    - extract/dnd/combat-turns/source_refs
    - extract/dnd/combat-turns/source_relatedness
normalize:
  validators:
    - generic/valid_json
    - generic/valid_json_schema
    - extract/dnd/combat-turns/shape
    - normalize/dnd/combat-turns/invariants
    - extract/dnd/combat-turns/source_refs
    - extract/dnd/combat-turns/source_relatedness
```

Empty chains approve output by default.

The `generic` chunker accepts:

- `max_units`: positive integer, default `50`;
- `overlap_units`: non-negative integer, default `0`, and must be less than
  `max_units`.

The `dnd/scenes` chunker requires transcript source capabilities, calls the
configured structured LLM runtime, and does not accept module options. It
declares optional `players`, `party`, and `glossary` references for scene
disambiguation, and accepts `roster` as a deprecated compatibility alias for
`party`.

The `dnd/spells` extractor declares optional campaign reference slots:

- `players`
- `party`
- `glossary`
- `roster` as a deprecated compatibility alias for `party`

These campaign slots accept UTF-8 plain text, Markdown, YAML, or JSON reference
files. The extractor also declares an optional `spell_catalog` slot that accepts
one UTF-8 `application/json` overlay bundle no larger than 1 MiB. The slot does
not allow multiple files. Its format is defined in the
[spell-catalog overlay contract](integrations/dnd-spell-catalog-overlays.md).
The extractor uses campaign references only as supporting disambiguation
material; spell casts still must be present in the source transcript.

It also declares an optional `npcs` slot for a normalized NPC artifact. The
slot accepts exactly one `application/json` file no larger than 1 MiB. During
extractor preparation Notarius strictly decodes and identity-validates the
artifact, then gives the model canonical JSON for caster-name grounding.
Registry source references may belong to the NPC-producing session and are
provenance only; they are not spell evidence. The bound registry contributes a
semantic digest and NPC count to extractor metadata and checkpoint identity,
while its names, aliases, content, and path do not appear there. When absent,
the prompt receives the exact empty value `{"npcs":[]}` and no registry
provenance or fingerprint is recorded.

The `dnd/spells` normalizer declares the same optional `spell_catalog` slot.
When an overlay is used, bind it independently under
`artifacts.<lane>.normalize.references.spell_catalog`; normalize-stage
references are local to that stage and are not inherited from extraction. The
normalizer uses the embedded SRD catalog when no normalize-stage overlay is
bound.

The `dnd/npcs` extractor declares the same optional campaign slots as the spell
extractor, but it does not declare the `npcs` registry slot. Its normalizer
accepts no references. To pass an NPC result to a later spell run, bind the
normalized payload explicitly at runtime; the maintained sequential example
documents that operator workflow.

The `dnd/combat-turns` extractor declares the optional campaign slots and the
structured `npcs` slot. Campaign references guide only the LLM extraction
stage. The deterministic normalizer declares only `npcs`, whose prepared
immutable registry supports the same actor and target canonicalization. Each
`npcs` slot accepts exactly one UTF-8 `application/json` file no larger than 1
MiB. The registry's source ranges remain provenance for the reference and never
become combat evidence. Binding `npcs` to extraction and normalization is
stage-local, so an operator-driven combat run uses two explicit selectors:

```text
combat.extract.npcs=<npc-run>/lanes/npcs.json
combat.normalize.npcs=<npc-run>/lanes/npcs.json
```

When bound, the combat extractor and normalizer record only the registry's
semantic digest and count in their metadata and checkpoint fingerprints; names,
aliases, content, and paths are not recorded there. When absent, the combat
prompt receives the exact empty registry value `{"npcs":[]}` and no registry
provenance or fingerprint is recorded.

## State Surfaces

The `output`, `cache`, and `debug` top-level fields select independent physical
roots. Their layout, permissions, lifecycle, and sensitive-data handling are
defined in [Operations](operations.md).

```yaml
output:
  directory: ./notarius-output
cache:
  chunk_plans:
    directory: ""
    mode: auto
  checkpoints:
    enabled: false
    directory: ""
debug:
  directory: ./notarius-debug
```

`output.directory` is the durable output root. Its precedence is
`--output-dir`, `NOTARIUS_OUTPUT_DIR`, the file value, then the default.

`cache.chunk_plans.mode` accepts `auto`, `bypass`, or `refresh`. Its precedence
is `--chunk_cache`, `NOTARIUS_CACHE_CHUNK_PLANS_MODE`, the file value, then
`auto`. `auto` reuses a valid source-addressed plan and regenerates missing or
invalid records; `bypass` performs no plan-cache I/O; `refresh` regenerates and
publishes a plan after chunk validation.

`cache.chunk_plans.directory` and `cache.checkpoints.directory` each name an
exact cache-family root. Their precedence is the corresponding environment
variable, the file value, then the family-specific per-user default. There is
no CLI cache-root override. The defaults are
`<os.UserCacheDir>/notarius/chunk-plans` and
`<os.UserCacheDir>/notarius/checkpoints`; on Unix, `os.UserCacheDir` ordinarily
uses an absolute `$XDG_CACHE_HOME` or falls back to `$HOME/.cache`. A relative
`XDG_CACHE_HOME` is an error.

`cache.checkpoints.enabled` defaults to `false`. When `true`, every run records
checkpoint transitions and reusable approved results. When `false`, Notarius
does not resolve or create the checkpoint root, and `--resume` is rejected.
The `--resume` flag authorizes loading compatible checkpoints; it does not
control recording.

`debug.directory` chooses a root but never enables debug capture. Its precedence
is `--debug-dir`, `NOTARIUS_DEBUG_DIR`, the file value, then the default.
Only `--debug` requests a bundle; `--debug-dir` is valid only with `--debug`.

Every supplied file, environment, and CLI value is validated even when a
higher-precedence value wins.

## Version 2 To Version 3 Migration

Version 2 files are rejected. Move each setting to the surface it controls and
remove obsolete enablement and retention controls. This complete before/after
example preserves an existing chunk-plan cache and checkpoint directory while
choosing an output and debug root explicitly.

```yaml
# Version 2 (no longer accepted)
version: 2
workspace:
  directory: /srv/notarius/state
  resume:
    enabled: true
  debug:
    enabled: true
  chunk_cache:
    directory: /srv/notarius/chunk-plans
    mode: auto
  diagnostics:
    retention: always
diagnostics:
  work_dir: /srv/notarius/inspection
```

```yaml
# Version 3
version: 3
output:
  directory: /srv/notarius/output
cache:
  chunk_plans:
    directory: /srv/notarius/chunk-plans
    mode: auto
  checkpoints:
    enabled: true
    directory: /srv/notarius/state/checkpoints
debug:
  directory: /srv/notarius/debug
```

Run the migrated configuration with `--resume` when checkpoint reuse is wanted,
and with `--debug` when a debug bundle is wanted. Enabled checkpoint recording
occurs with or without `--resume`.

The removed fields are `workspace.directory`, `workspace.resume.enabled`,
`workspace.debug.enabled`, `workspace.chunk_cache.mode`,
`workspace.chunk_cache.directory`, `workspace.diagnostics.enabled`,
`workspace.diagnostics.retention`, `diagnostics.work_dir`, and
`diagnostics.retention`. The removed environment variables are `NOTARIUS_WORKSPACE_DIR`,
`NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`,
`NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`,
`NOTARIUS_WORKSPACE_RESUME_ENABLED`, `NOTARIUS_WORKSPACE_DEBUG_ENABLED`,
`NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`,
`NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`, `NOTARIUS_WORK_DIR`, and
`NOTARIUS_DIAGNOSTICS_RETENTION`. The chunk-cache variables are replaced by
`NOTARIUS_CACHE_CHUNK_PLANS_MODE` and
`NOTARIUS_CACHE_CHUNK_PLANS_DIR`; the former shared directory has no direct
replacement.

## Validation

Configuration validation checks:

- supported config version and known YAML fields;
- mutually exclusive `scriptorium.profile_dir` and `scriptorium.profile_file`;
- non-empty, non-duplicated IDs after trimming;
- positive global LLM concurrency;
- supported stage-worker keys and an effective extract worker count in the
  inclusive range `1..concurrency.total_llm`;
- non-empty output and debug directories;
- a supported chunk-cache mode, Boolean checkpoint enablement, and state-surface
  directories without NUL bytes;
- stale removed fields such as `llm_profiles`.

Pipeline resolution additionally checks:

- the pipeline ID exists;
- at least one artifact lane is declared and selected;
- lanes selected through the CLI exist in the resolved pipeline;
- required module keys are present;
- module keys are registered for the expected slot;
- module capability requirements are satisfied;
- non-empty validator overrides reference registered validator keys;
- deterministic validators do not set `llm_profile`;
- LLM-backed validators with explicit `llm_profile` values reference configured
  Scriptorium profile IDs;
- bound reference slots are declared by selected chunk, extractor, merger, or
  normalizer targets;
- required reference slots are bound for selected targets.