notarius/docs/config.md

# Configuration

This is the canonical reference for implemented Notarius configuration.

Notarius reads YAML config files with `version: 2`. File config is applied over
built-in defaults, then environment overrides are applied.

## Discovery

Commands that load configuration use this order:

1. an explicit path supplied through the CLI, when provided;
2. `NOTARIUS_CONFIG`, when set to a non-empty path;
3. `/usr/local/etc/notarius/config.yml`.

If none is available, the command fails with a config file not found error.
The explicit-path option is defined in the [CLI reference](cli.md).

## Maintained Examples

- [Minimal D&D spell configuration](../examples/dnd-spells.config.yml)
- [Production-oriented D&D spell configuration](../examples/dnd-spells-production.config.yml)

Both complete files are validated by the CLI test suite. The fragments below
illustrate individual fields and are not alternate complete configurations.

## Top-Level Fields

- `version`: required. The only supported value is `2`.
- `scriptorium`: optional Scriptorium profile source settings.
- `pipelines`: optional map of pipeline IDs to pipeline definitions.
- `concurrency`: optional global concurrency settings.
- `workspace`: optional workspace settings for Notarius-owned local state.
- `diagnostics`: optional diagnostics settings.

Unknown YAML fields are rejected. The removed top-level `llm_profiles` field is
rejected; execution profiles now come from Scriptorium.

## Defaults

Built-in defaults:

- `concurrency.total_llm`: `1`
- `concurrency.stage_workers.extract`: effective `concurrency.total_llm`
- `diagnostics.work_dir`: `/tmp/notarius`
- `diagnostics.retention`: `auto`
- `workspace.directory`: unset
- `workspace.diagnostics.enabled`: `true`
- `workspace.resume.enabled`: `false`
- `workspace.debug.enabled`: `false`
- `workspace.chunk_cache.mode`: `auto`
- `workspace.chunk_cache.directory`: unset

No pipelines are built in. A run requires a configured pipeline.

If `scriptorium` is omitted, Notarius uses Scriptorium's built-in profile
catalog. Prompt definitions may also name default profile IDs. The current D&D
scene and spell prompts default to the built-in `mistral-small-3` profile when a
module binding does not set `llm_profile`. That built-in profile reads its
credential from `OPENROUTER_API_KEY`.

## Scriptorium Profiles

`scriptorium` fields:

- `profile_dir`: optional directory containing Scriptorium profile YAML files.
- `profile_file`: optional Scriptorium profile YAML file.

`profile_dir` and `profile_file` are mutually exclusive. Custom profiles
overlay Scriptorium built-in profiles by profile ID.

Scriptorium profile files use Scriptorium's profile schema. A minimal profile
looks like:

```yaml
id: local-fast
endpoint: http://127.0.0.1:8080/v1
model: your-model
api_key_env: SCRIPTORIUM_API_KEY
timeout_seconds: 180
```

Notarius does not accept raw API keys in Notarius config. For file-backed
Scriptorium profiles, store the environment variable name in `api_key_env` and
set that variable in the run environment. Scriptorium rejects raw `api_key`
fields in profile YAML.

## Environment Overrides

These environment variables are applied after the config file:

- `NOTARIUS_CONFIG`: config discovery path.
- `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency.
- `NOTARIUS_STAGE_WORKERS_EXTRACT`: integer extract worker limit.
- `NOTARIUS_WORKSPACE_DIR`: workspace directory.
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`: boolean diagnostics enablement.
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`: workspace diagnostics retention
  mode.
- `NOTARIUS_WORKSPACE_RESUME_ENABLED`: boolean resume checkpointing
  enablement.
- `NOTARIUS_WORKSPACE_DEBUG_ENABLED`: boolean debug artifact enablement.
- `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`: chunk-plan cache mode.
- `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`: chunk-plan cache root.
- `NOTARIUS_WORK_DIR`: deprecated diagnostics work directory compatibility
  override.
- `NOTARIUS_DIAGNOSTICS_RETENTION`: deprecated diagnostics retention
  compatibility override.

Integer environment values must parse as base-10 integers. Boolean environment
values must parse as Go booleans such as `true`, `false`, `1`, or `0`.

The removed `NOTARIUS_LLM_DEFAULT_*` variables are not read. Configure provider
endpoint, model, and credential environment variable names through Scriptorium
profiles.

## Concurrency

`concurrency` fields:

- `total_llm`: positive integer ceiling on concurrent provider calls.
- `stage_workers`: optional map of framework worker limits. The only supported
  key is `extract`.

`stage_workers.extract` defaults to the effective `total_llm` value after file
and environment precedence. It must be between `1` and `total_llm`, inclusive.
Unknown or empty stage-worker keys are rejected. The environment override
`NOTARIUS_STAGE_WORKERS_EXTRACT` takes precedence over the file value, as does
`NOTARIUS_TOTAL_LLM_CONCURRENCY` for the global ceiling.

The worker value is present in effective and redacted configuration. It bounds
the fixed run-wide extract pool and its bounded dispatch queue. Extract jobs are
submitted by source chunk and then resolved lane; `total_llm` independently
bounds actual provider calls made by extracts, retries, and validators.

## Pipelines

A pipeline selects implementations for the fixed workflow defined by
[Architecture](policy/architecture.md#system-shape).

Pipeline fields:

- `input`: required module binding.
- `chunk`: optional module binding. Default module is `generic`.
- `artifacts`: required for pipeline resolution. It maps artifact lane IDs to
  lane definitions.
- `output`: optional module binding. Default module is `json`.
- `references`: optional map of reference slot names to reference paths. These
  bindings are defaults for eligible pipeline targets that declare the matching
  slot.

Artifact lane fields:

- `extract`: required module binding.
- `merge`: optional module binding. Default module is `appendorder`.
- `normalize`: optional module binding. Default module is `noop`.
- `validators`: deprecated lane-level validator list. Non-empty lists are
  rejected; use `extract.validators`, `merge.validators`, or
  `normalize.validators`.
- `references`: optional compatibility alias for extractor reference bindings.
  Lane bindings override pipeline-level bindings for the same slot.

Commands that resolve a pipeline fail for unknown or incompatible module keys.
See [CLI Reference](cli.md) for command syntax.

Reference bindings are validated against reference slots declared by eligible
chunk, extract, merge, and normalize targets during pipeline resolution. Required slots
must be bound after config defaults, target-local references, lane-level
compatibility bindings, and command-line reference overrides are applied.
Config-relative paths are resolved relative to the config file; command-line
reference paths are resolved relative to the current working directory. Bound
files must be UTF-8 text. Reference media types are inferred from file
extensions and checked when a module restricts accepted types; unknown
extensions use `application/octet-stream`. See [CLI Reference](cli.md#run) for
command-line selectors and [Operations](operations.md) for recorded provenance
and sensitive-data handling.

Pipeline-level `references` are defaults. They are valid when at least one
eligible target in the full configured pipeline declares the slot, including
chunk, extractor, merger, and normalizer targets. During a run, they apply only
to the selected targets that declare the slot:

```yaml
pipelines:
  dnd-session:
    input: seriatim
    references:
      players: ./campaign/players.txt
      party: ./campaign/party-roster.txt
      glossary: ./campaign/glossary.txt
    artifacts:
      spells:
        extract: dnd/spells
```

Extractor binding `references` are the canonical lane-local location. The
legacy lane-level `references` field remains supported as an alias; when both
bind the same slot, `extract.references` wins:

```yaml
pipelines:
  dnd-session:
    input: seriatim
    references:
      glossary: ./campaign/glossary.txt
    artifacts:
      spells:
        references:
          roster: ./campaign/legacy-roster.txt
        extract:
          module: dnd/spells
          references:
            party: ./campaign/session-party.txt
```

`chunk.references`, `merge.references`, and `normalize.references` are accepted
in object-form bindings. They override pipeline-level defaults for slots
declared by that target module. Extractor-local references apply only to the
extractor, merger-local references apply only to the merger, and
normalizer-local references apply only to the normalizer.

Target-local reference fields use the same map shape at:

- `pipelines.<id>.chunk.references`
- `pipelines.<id>.artifacts.<lane>.extract.references`
- `pipelines.<id>.artifacts.<lane>.merge.references`
- `pipelines.<id>.artifacts.<lane>.normalize.references`

Each binding is valid only when that target module declares the slot.

## Module Bindings

Every module binding may use shorthand:

```yaml
input: seriatim
```

or object form:

```yaml
chunk:
  module: dnd/scenes
  llm_profile: local-fast
```

Binding fields:

- `module`: module key.
- `llm_profile`: optional Scriptorium profile ID. Empty or omitted lets the
  Scriptorium prompt default select the profile.
- `retries`: non-negative retry count for extra runtime attempts after the
  first attempt. Default: `0`. Supported on `chunk`, `extract`, `merge`, and
  `normalize` bindings.
- `options`: optional module-specific settings.
- `references`: optional reference bindings. Supported only for `chunk`,
  `extract`, `merge`, and `normalize` bindings. `input` and `output` bindings
  reject this field during validation.
- `validators`: optional stage-local validator chain override. Supported only
  for `chunk`, `extract`, `merge`, and `normalize` bindings. Omit the field to
  use the production default chain; set `validators: []` to force an empty
  chain; set a non-empty list to use exactly those validators in configured
  order.

During resolution, each selected module's registered option validator runs.
Production input, chunk, and output bindings reject unknown or invalid options
with the affected binding context.

Validator bindings use the same shorthand or object module-binding form, but
only these fields are supported:

- `module`: validator key.
- `llm_profile`: optional Scriptorium profile ID for LLM-backed validators.
- `options`: optional validator-specific settings.

Validator bindings reject `references`, `retries`, and nested `validators`.
During resolution, deterministic validators reject explicit `llm_profile`
values.

Configured LLM-backed validators with explicit `llm_profile` values are
validated against the configured Scriptorium profile source. Deterministic
production validators do not call the LLM and must not set `llm_profile`.

## Implemented Production Modules

| Slot | Key | Notes |
| --- | --- | --- |
| input | `seriatim` | Reads Seriatim transcript JSON. |
| chunk | `generic` | Splits source units into ordered chunks. |
| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. |
| extract | `dnd/spells` | Extracts typed D&D spell-list artifacts. |
| merge | `appendorder` | Combines typed artifacts in chunk order. |
| normalize | `noop` | Passes merged typed artifacts through unchanged. |
| output | `json` | Produces JSON output files for normalized `application/json` lanes. |

## Implemented Production Validators

| Key | Execution | Notes |
| --- | --- | --- |
| `generic/always_accept` | deterministic | Accepts returned module output. |
| `generic/always_reject` | deterministic | Rejects returned module output with reason `always_reject`. |
| `generic/valid_json` | deterministic | Rejects payloads that are not syntactically valid JSON. |
| `generic/valid_json_schema` | deterministic | Rejects invalid JSON or JSON that does not conform to the module response schema. |
| `extract/dnd/spells/shape` | deterministic | Rejects malformed D&D spell-list artifacts. |
| `extract/dnd/spells/source_refs` | deterministic | Rejects missing or invalid D&D spell source references. |
| `extract/dnd/spells/source_relatedness` | deterministic | Emits warnings when a spell name is not found near its cited source text. |

The production default chain for the `dnd/spells` extractor is:

```yaml
validators:
  - generic/valid_json
  - generic/valid_json_schema
  - extract/dnd/spells/shape
  - extract/dnd/spells/source_refs
  - extract/dnd/spells/source_relatedness
```

No other production module currently has a default validator chain. Empty
chains approve output by default.

The `generic` chunker accepts:

- `max_units`: positive integer, default `50`;
- `overlap_units`: non-negative integer, default `0`, and must be less than
  `max_units`.

The `dnd/scenes` chunker requires transcript source capabilities, calls the
configured structured LLM runtime, and does not accept module options. It
declares optional `players`, `party`, and `glossary` references for scene
disambiguation, and accepts `roster` as a deprecated compatibility alias for
`party`.

The `dnd/spells` extractor declares optional reference slots:

- `players`
- `party`
- `glossary`
- `roster` as a deprecated compatibility alias for `party`

Both modules accept UTF-8 plain text, Markdown, YAML, or JSON reference files.
The extractor uses references only as supporting disambiguation material; spell
casts still must be present in the source transcript.

## Workspace

`workspace` fields:

- `directory`: optional workspace root for Notarius-owned local state.
- `resume.enabled`: boolean resume checkpointing setting.
- `debug.enabled`: boolean debug artifact setting.
- `chunk_cache.mode`: persistent chunk-plan cache mode: `auto`, `bypass`, or
  `refresh`. The default is `auto`.
- `chunk_cache.directory`: optional chunk-plan cache root. This value is the
  root itself; Notarius does not append `chunk-plans` to it.
- `diagnostics`: optional diagnostics settings defined below.

`workspace.resume.enabled`, `workspace.debug.enabled`, and
`workspace.chunk_cache` are independent. `workspace.directory` does not affect
chunk-plan placement. For directory layout, state lifecycle, permissions, and
sensitive content, see [Operations](operations.md).

`chunk_cache.mode` accepts only `auto`, `bypass`, and `refresh`. In `auto`, a
valid source-addressed plan is reused and a missing or invalid record is
regenerated and published after chunk validation. `bypass` neither reads nor
writes plan-cache state. `refresh` always generates a new plan and publishes it
only after validation succeeds.

Configuration values are applied in file then environment order; an explicit
`--chunk_cache` CLI value has highest precedence for the mode. The cache root
is selected from environment, file, then the per-user default; there is no CLI
root override. Every supplied value is parsed strictly even when a higher
precedence value wins, so malformed configuration is still an error.

When `chunk_cache.directory` is unset, the root is
`<os.UserCacheDir>/notarius/chunk-plans`. On Unix this is ordinarily
`$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is an absolute
path, or `$HOME/.cache/notarius/chunk-plans` when it is unset. A relative
`XDG_CACHE_HOME` is rejected by `os.UserCacheDir`; Notarius reports that as a
configuration error and does not fall back to another directory.

## Diagnostics

Preferred workspace diagnostics fields:

- `workspace.diagnostics.enabled`: set to `false` to skip creating diagnostics
  run directories and diagnostics artifacts.
- `workspace.diagnostics.retention`: `auto`, `always`, or `never`.

Defaults for workspace and diagnostics fields are listed in
[Defaults](#defaults).

`workspace.diagnostics.retention` overrides legacy diagnostics retention when
set.

`diagnostics` fields:

- `work_dir`: deprecated compatibility directory for per-run diagnostics.
- `retention`: deprecated compatibility retention mode. `auto`, `always`, or
  `never`.

Existing `diagnostics.work_dir`, `diagnostics.retention`, `NOTARIUS_WORK_DIR`,
and `NOTARIUS_DIAGNOSTICS_RETENTION` inputs remain supported for compatibility.
New configuration should use `workspace.directory` and
`workspace.diagnostics.retention` instead.

For retention behavior and the physical diagnostics layout, see
[Operations](operations.md#retention). For the invocation-specific diagnostics
override, see [CLI Reference](cli.md#run).

## Validation

Configuration validation checks:

- supported config version and known YAML fields;
- mutually exclusive `scriptorium.profile_dir` and `scriptorium.profile_file`;
- non-empty, non-duplicated IDs after trimming;
- positive global LLM concurrency;
- supported stage-worker keys and an effective extract worker count in the
  inclusive range `1..concurrency.total_llm`;
- supported diagnostics retention and non-empty work directory;
- a supported chunk-cache mode and a chunk-cache directory without NUL bytes;
- stale removed fields such as `llm_profiles`.

Pipeline resolution additionally checks:

- the pipeline ID exists;
- at least one artifact lane is declared and selected;
- lanes selected through the CLI exist in the resolved pipeline;
- required module keys are present;
- module keys are registered for the expected slot;
- module capability requirements are satisfied;
- non-empty validator overrides reference registered validator keys;
- deterministic validators do not set `llm_profile`;
- LLM-backed validators with explicit `llm_profile` values reference configured
  Scriptorium profile IDs;
- bound reference slots are declared by selected chunk, extractor, merger, or
  normalizer targets;
- required reference slots are bound for selected targets.