454 lines
16 KiB
Markdown
454 lines
16 KiB
Markdown
# Configuration
|
|
|
|
This is the canonical reference for implemented Notarius configuration.
|
|
|
|
Notarius reads YAML config files with `version: 2`. File config is applied over
|
|
built-in defaults, then environment overrides are applied.
|
|
|
|
## Discovery
|
|
|
|
Commands that accept `--config` load configuration in this order:
|
|
|
|
1. the `--config` path, when provided;
|
|
2. `NOTARIUS_CONFIG`, when set to a non-empty path;
|
|
3. `/usr/local/etc/notarius/config.yml`.
|
|
|
|
If none is available, the command fails with a config file not found error.
|
|
|
|
## Minimal Example
|
|
|
|
```yaml
|
|
version: 2
|
|
pipelines:
|
|
dnd-session:
|
|
input: seriatim
|
|
references:
|
|
party: ./dnd-spells-roster.txt
|
|
glossary: ./dnd-spells-glossary.txt
|
|
chunk:
|
|
module: generic
|
|
options:
|
|
max_units: 50
|
|
artifacts:
|
|
spells:
|
|
extract: dnd/spells
|
|
```
|
|
|
|
The maintained fixture is [examples/dnd-spells.config.yml](../examples/dnd-spells.config.yml).
|
|
|
|
## Top-Level Fields
|
|
|
|
- `version`: required. The only supported value is `2`.
|
|
- `scriptorium`: optional Scriptorium profile source settings.
|
|
- `pipelines`: optional map of pipeline IDs to pipeline definitions.
|
|
- `concurrency`: optional global concurrency settings.
|
|
- `workspace`: optional workspace settings for Notarius-owned local state.
|
|
- `diagnostics`: optional diagnostics settings.
|
|
|
|
Unknown YAML fields are rejected. The removed top-level `llm_profiles` field is
|
|
rejected; execution profiles now come from Scriptorium.
|
|
|
|
## Defaults
|
|
|
|
Built-in defaults:
|
|
|
|
```yaml
|
|
concurrency:
|
|
total_llm: 1
|
|
diagnostics:
|
|
work_dir: /tmp/notarius
|
|
retention: auto
|
|
workspace:
|
|
diagnostics:
|
|
enabled: true
|
|
resume:
|
|
enabled: false
|
|
debug:
|
|
enabled: false
|
|
```
|
|
|
|
`workspace.directory` is unset by default. Without a workspace directory,
|
|
diagnostics continue to use `/tmp/notarius`, and checkpoint and debug workspace
|
|
features have no storage root.
|
|
|
|
No pipelines are built in. A run requires a configured pipeline.
|
|
|
|
If `scriptorium` is omitted, Notarius uses Scriptorium's built-in profile
|
|
catalog. Prompt definitions may also name default profile IDs. The current D&D
|
|
scene and spell prompts use Scriptorium prompt defaults when a module binding
|
|
does not set `llm_profile`.
|
|
|
|
## Scriptorium Profiles
|
|
|
|
`scriptorium` fields:
|
|
|
|
- `profile_dir`: optional directory containing Scriptorium profile YAML files.
|
|
- `profile_file`: optional Scriptorium profile YAML file.
|
|
|
|
`profile_dir` and `profile_file` are mutually exclusive. Custom profiles
|
|
overlay Scriptorium built-in profiles by profile ID.
|
|
|
|
Scriptorium profile files use Scriptorium's profile schema. A minimal profile
|
|
looks like:
|
|
|
|
```yaml
|
|
id: local-fast
|
|
endpoint: http://127.0.0.1:8080/v1
|
|
model: your-model
|
|
api_key_env: SCRIPTORIUM_API_KEY
|
|
timeout_seconds: 180
|
|
```
|
|
|
|
Notarius does not accept raw API keys in Notarius config. For file-backed
|
|
Scriptorium profiles, store the environment variable name in `api_key_env` and
|
|
set that variable in the run environment. Scriptorium rejects raw `api_key`
|
|
fields in profile YAML.
|
|
|
|
## Environment Overrides
|
|
|
|
These environment variables are applied after the config file:
|
|
|
|
- `NOTARIUS_CONFIG`: config discovery path.
|
|
- `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency.
|
|
- `NOTARIUS_WORKSPACE_DIR`: workspace directory.
|
|
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`: boolean diagnostics enablement.
|
|
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`: workspace diagnostics retention
|
|
mode.
|
|
- `NOTARIUS_WORKSPACE_RESUME_ENABLED`: boolean resume checkpointing
|
|
enablement.
|
|
- `NOTARIUS_WORKSPACE_DEBUG_ENABLED`: boolean debug artifact enablement.
|
|
- `NOTARIUS_WORK_DIR`: deprecated diagnostics work directory compatibility
|
|
override.
|
|
- `NOTARIUS_DIAGNOSTICS_RETENTION`: deprecated diagnostics retention
|
|
compatibility override.
|
|
|
|
Integer environment values must parse as base-10 integers. Boolean environment
|
|
values must parse as Go booleans such as `true`, `false`, `1`, or `0`.
|
|
|
|
The removed `NOTARIUS_LLM_DEFAULT_*` variables are not read. Configure provider
|
|
endpoint, model, and credential environment variable names through Scriptorium
|
|
profiles.
|
|
|
|
## Pipelines
|
|
|
|
A pipeline defines the fixed Notarius workflow:
|
|
|
|
```text
|
|
input -> chunk -> extract -> merge -> normalize -> output
|
|
```
|
|
|
|
Pipeline fields:
|
|
|
|
- `input`: required module binding.
|
|
- `chunk`: optional module binding. Default module is `generic`.
|
|
- `artifacts`: required for pipeline resolution. It maps artifact lane IDs to
|
|
lane definitions.
|
|
- `output`: optional module binding. Default module is `json`.
|
|
- `references`: optional map of reference slot names to reference paths. These
|
|
bindings are defaults for eligible pipeline targets that declare the matching
|
|
slot.
|
|
|
|
Artifact lane fields:
|
|
|
|
- `extract`: required module binding.
|
|
- `merge`: optional module binding. Default module is `appendorder`.
|
|
- `normalize`: optional module binding. Default module is `noop`.
|
|
- `validators`: deprecated lane-level validator list. Non-empty lists are
|
|
rejected; use `extract.validators`, `merge.validators`, or
|
|
`normalize.validators`.
|
|
- `references`: optional compatibility alias for extractor reference bindings.
|
|
Lane bindings override pipeline-level bindings for the same slot.
|
|
|
|
`notarius run` and `notarius config validate --pipeline` resolve the pipeline
|
|
against the production module catalog and fail fast for unknown or incompatible
|
|
module keys.
|
|
|
|
Reference bindings are validated against reference slots declared by eligible
|
|
chunk, extract, merge, and normalize targets during pipeline resolution. Required slots
|
|
must be bound after config defaults, target-local references, lane-level
|
|
compatibility bindings, and run-time `--reference` or `--without-reference`
|
|
overrides are applied. Config-relative paths are resolved relative to the config
|
|
file; CLI reference paths are resolved relative to the current working
|
|
directory. Materialized bound files must be UTF-8 text. Materialized reference
|
|
provenance is recorded for chunk, extractor, merger, and normalizer targets, and runtime
|
|
reference content is passed to the target that declares the slot. Reference
|
|
media types are inferred from file extensions, recorded as canonical base media
|
|
types, and checked only when a module declares `AcceptedMediaTypes`; unknown
|
|
extensions are recorded as `application/octet-stream`. Reference content is not
|
|
written to diagnostics, logs, errors, or manifests.
|
|
|
|
Pipeline-level `references` are defaults. They are valid when at least one
|
|
eligible target in the full configured pipeline declares the slot, including
|
|
chunk, extractor, merger, and normalizer targets. During a run, they apply only
|
|
to the selected targets that declare the slot:
|
|
|
|
```yaml
|
|
pipelines:
|
|
dnd-session:
|
|
input: seriatim
|
|
references:
|
|
players: ./campaign/players.txt
|
|
party: ./campaign/party-roster.txt
|
|
glossary: ./campaign/glossary.txt
|
|
artifacts:
|
|
spells:
|
|
extract: dnd/spells
|
|
```
|
|
|
|
Extractor binding `references` are the canonical lane-local location. The
|
|
legacy lane-level `references` field remains supported as an alias; when both
|
|
bind the same slot, `extract.references` wins:
|
|
|
|
```yaml
|
|
pipelines:
|
|
dnd-session:
|
|
input: seriatim
|
|
references:
|
|
glossary: ./campaign/glossary.txt
|
|
artifacts:
|
|
spells:
|
|
references:
|
|
roster: ./campaign/legacy-roster.txt
|
|
extract:
|
|
module: dnd/spells
|
|
references:
|
|
party: ./campaign/session-party.txt
|
|
```
|
|
|
|
`chunk.references`, `merge.references`, and `normalize.references` are accepted
|
|
in object-form bindings. They override pipeline-level defaults for slots
|
|
declared by that target module. Extractor-local references apply only to the
|
|
extractor, merger-local references apply only to the merger, and
|
|
normalizer-local references apply only to the normalizer.
|
|
|
|
Target-local reference fields use the same map shape at:
|
|
|
|
- `pipelines.<id>.chunk.references`
|
|
- `pipelines.<id>.artifacts.<lane>.extract.references`
|
|
- `pipelines.<id>.artifacts.<lane>.merge.references`
|
|
- `pipelines.<id>.artifacts.<lane>.normalize.references`
|
|
|
|
Each binding is valid only when that target module declares the slot.
|
|
|
|
## Module Bindings
|
|
|
|
Every module binding may use shorthand:
|
|
|
|
```yaml
|
|
input: seriatim
|
|
```
|
|
|
|
or object form:
|
|
|
|
```yaml
|
|
chunk:
|
|
module: dnd/scenes
|
|
llm_profile: local-fast
|
|
```
|
|
|
|
Binding fields:
|
|
|
|
- `module`: module key.
|
|
- `llm_profile`: optional Scriptorium profile ID. Empty or omitted lets the
|
|
Scriptorium prompt default select the profile.
|
|
- `retries`: non-negative retry count for extra runtime attempts after the
|
|
first attempt. The runner applies retries to `chunk`, `extract`, `merge`, and
|
|
`normalize` bindings.
|
|
- `options`: optional module-specific settings.
|
|
- `references`: optional reference bindings. Supported only for `chunk`,
|
|
`extract`, `merge`, and `normalize` bindings. `input` and `output` bindings
|
|
reject this field during validation.
|
|
- `validators`: optional stage-local validator chain override. Supported only
|
|
for `chunk`, `extract`, `merge`, and `normalize` bindings. Omit the field to
|
|
use the production default chain; set `validators: []` to force an empty
|
|
chain; set a non-empty list to use exactly those validators in configured
|
|
order.
|
|
|
|
Validator bindings use the same shorthand or object module-binding form, but
|
|
only these fields are supported:
|
|
|
|
- `module`: validator key.
|
|
- `llm_profile`: optional Scriptorium profile ID for LLM-backed validators.
|
|
- `options`: optional validator-specific settings.
|
|
|
|
Validator bindings reject `references`, `retries`, and nested `validators`.
|
|
During resolution, deterministic validators reject explicit `llm_profile`
|
|
values.
|
|
|
|
The `--llm-profile` run flag overrides every effective LLM-capable module
|
|
binding to use one Scriptorium profile ID: chunk, every selected lane extract,
|
|
merge, and normalize binding. It does not override validator-specific
|
|
`llm_profile` values.
|
|
|
|
Configured LLM-backed validators with explicit `llm_profile` values are
|
|
validated against the configured Scriptorium profile source. Deterministic
|
|
production validators do not call the LLM and must not set `llm_profile`.
|
|
|
|
## Implemented Production Modules
|
|
|
|
| Slot | Key | Notes |
|
|
| --- | --- | --- |
|
|
| input | `seriatim` | Reads Seriatim transcript JSON. |
|
|
| chunk | `generic` | Splits source units into ordered chunks. |
|
|
| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. |
|
|
| extract | `dnd/spells` | Extracts D&D spell raw outputs. |
|
|
| merge | `appendorder` | Merges JSON raw extract outputs in chunk order. |
|
|
| normalize | `noop` | Passes merged raw outputs through unchanged. |
|
|
| output | `json` | Produces JSON output files for normalized `application/json` lanes. |
|
|
|
|
## Implemented Production Validators
|
|
|
|
| Key | Execution | Notes |
|
|
| --- | --- | --- |
|
|
| `generic/always_accept` | deterministic | Accepts returned module output. |
|
|
| `generic/always_reject` | deterministic | Rejects returned module output with reason `always_reject`. |
|
|
| `generic/valid_json` | deterministic | Rejects payloads that are not syntactically valid JSON. |
|
|
| `generic/valid_json_schema` | deterministic | Rejects invalid JSON or JSON that does not conform to the module response schema. |
|
|
| `extract/dnd/spells/shape` | deterministic | Rejects malformed D&D spell-cast JSON payloads. |
|
|
| `extract/dnd/spells/source_refs` | deterministic | Rejects missing or invalid D&D spell source references. |
|
|
| `extract/dnd/spells/source_relatedness` | deterministic | Emits warnings when a spell name is not found near its cited source text. |
|
|
|
|
The production default chain for the `dnd/spells` extractor is:
|
|
|
|
```yaml
|
|
validators:
|
|
- generic/valid_json
|
|
- generic/valid_json_schema
|
|
- extract/dnd/spells/shape
|
|
- extract/dnd/spells/source_refs
|
|
- extract/dnd/spells/source_relatedness
|
|
```
|
|
|
|
No other production module currently has a default validator chain. Empty
|
|
chains approve output by default.
|
|
|
|
The `generic` chunker accepts:
|
|
|
|
- `max_units`: positive integer, default `50`;
|
|
- `overlap_units`: non-negative integer, default `0`, and must be less than
|
|
`max_units`.
|
|
|
|
The `dnd/scenes` chunker requires transcript source capabilities, calls the
|
|
configured structured LLM runtime, and does not accept module options. It
|
|
declares optional `players`, `party`, and `glossary` references for scene
|
|
disambiguation, and accepts `roster` as a deprecated compatibility alias for
|
|
`party`.
|
|
|
|
The `dnd/spells` extractor declares optional reference slots:
|
|
|
|
- `players`
|
|
- `party`
|
|
- `glossary`
|
|
- `roster` as a deprecated compatibility alias for `party`
|
|
|
|
Both modules accept UTF-8 plain text, Markdown, YAML, or JSON reference files.
|
|
The extractor uses references only as supporting disambiguation material; spell
|
|
casts still must be present in the source transcript.
|
|
|
|
## Workspace
|
|
|
|
`workspace` fields:
|
|
|
|
- `directory`: optional workspace root for Notarius-owned local state.
|
|
- `diagnostics.enabled`: set to `false` to skip diagnostics run directories and
|
|
diagnostics artifact writes. Default: `true`.
|
|
- `diagnostics.retention`: `auto`, `always`, or `never`.
|
|
- `resume.enabled`: boolean resume checkpointing setting. Default: `false`.
|
|
- `debug.enabled`: boolean debug artifact setting. Default: `false`.
|
|
|
|
Use `/var/lib/notarius` as the standard production workspace directory. For
|
|
local development, prefer a project-local ignored path such as
|
|
`./.notarius/workspace`.
|
|
|
|
```yaml
|
|
workspace:
|
|
directory: /var/lib/notarius
|
|
diagnostics:
|
|
enabled: true
|
|
retention: auto
|
|
resume:
|
|
enabled: false
|
|
debug:
|
|
enabled: false
|
|
```
|
|
|
|
When `workspace.directory` is set, diagnostics are written under
|
|
`<workspace.directory>/diagnostics/`.
|
|
|
|
When both `workspace.directory` and `workspace.resume.enabled` are set, runs
|
|
write stage-owned checkpoint artifacts under
|
|
`<workspace.directory>/checkpoints/`. `notarius run --resume` can reuse valid
|
|
checkpoints from a compatible invocation. Checkpoints may contain source text,
|
|
intermediate raw outputs, rejected outputs, metadata, and warnings. Protect the
|
|
workspace as sensitive local state.
|
|
|
|
When both `workspace.directory` and `workspace.debug.enabled` are set, runs
|
|
write per-invocation debug artifacts under
|
|
`<workspace.directory>/debug/<run-id>/`. Debug artifacts may contain source
|
|
material, reference material, prompt inputs, model outputs, validation payloads,
|
|
and other sensitive content. Debug is disabled by default.
|
|
|
|
`workspace.resume.enabled` and `workspace.debug.enabled` are independent.
|
|
Enabling one does not enable the other.
|
|
|
|
## Diagnostics
|
|
|
|
Preferred workspace diagnostics fields:
|
|
|
|
- `workspace.directory`: workspace root for Notarius-owned local state.
|
|
- `workspace.diagnostics.enabled`: set to `false` to skip creating diagnostics
|
|
run directories and diagnostics artifacts. Default: `true`.
|
|
- `workspace.diagnostics.retention`: `auto`, `always`, or `never`.
|
|
|
|
When `workspace.directory` is set, diagnostics use
|
|
`<workspace.directory>/diagnostics` as their work directory.
|
|
`workspace.diagnostics.retention` overrides legacy diagnostics retention when
|
|
set.
|
|
|
|
`diagnostics` fields:
|
|
|
|
- `work_dir`: deprecated compatibility directory for per-run diagnostics.
|
|
Default: `/tmp/notarius`.
|
|
- `retention`: deprecated compatibility retention mode. `auto`, `always`, or
|
|
`never`. Empty uses `auto`.
|
|
|
|
Existing `diagnostics.work_dir`, `diagnostics.retention`, `NOTARIUS_WORK_DIR`,
|
|
and `NOTARIUS_DIAGNOSTICS_RETENTION` inputs remain supported for compatibility.
|
|
New configuration should use `workspace.directory` and
|
|
`workspace.diagnostics.retention` instead.
|
|
|
|
`auto` retains diagnostics for failed runs and successful runs with warnings.
|
|
`always` retains diagnostics for every run. `never` removes diagnostics for
|
|
successful runs without regard to warnings; failed runs are retained.
|
|
|
|
The `--diagnostics-dir` run flag overrides the effective diagnostics work
|
|
directory for that invocation. It affects diagnostics only and does not change
|
|
the workspace directory.
|
|
|
|
## Validation
|
|
|
|
Configuration validation checks:
|
|
|
|
- supported config version and known YAML fields;
|
|
- mutually exclusive `scriptorium.profile_dir` and `scriptorium.profile_file`;
|
|
- non-empty, non-duplicated IDs after trimming;
|
|
- positive global LLM concurrency;
|
|
- supported diagnostics retention and non-empty work directory;
|
|
- stale removed fields such as `llm_profiles`.
|
|
|
|
Pipeline resolution additionally checks:
|
|
|
|
- the pipeline ID exists;
|
|
- at least one artifact lane is declared and selected;
|
|
- selected lanes exist when `--only` is used;
|
|
- required module keys are present;
|
|
- module keys are registered for the expected slot;
|
|
- module capability requirements are satisfied;
|
|
- non-empty validator overrides reference registered validator keys;
|
|
- deterministic validators do not set `llm_profile`;
|
|
- LLM-backed validators with explicit `llm_profile` values reference configured
|
|
Scriptorium profile IDs;
|
|
- bound reference slots are declared by selected chunk, extractor, merger, or
|
|
normalizer targets;
|
|
- required reference slots are bound for selected targets.
|