eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document output cache and debug state model

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-18 13:08:27 +00:00
parent 8cb11e60e4
commit 8824948910
14 changed files with 354 additions and 1478 deletions
Download Patch File Download Diff File Expand all files Collapse all files

200
docs/config.md
View File

@@ -2,8 +2,9 @@
This is the canonical reference for implemented Notarius configuration.
Notarius reads YAML config files with `version: 2`. File config is applied over
built-in defaults, then environment overrides are applied.
Notarius reads YAML config files with `version: 3`. File configuration is
applied over built-in defaults, then environment overrides are applied. Explicit
CLI overrides are applied last where the command supports them.
## Discovery
@@ -21,17 +22,18 @@ The explicit-path option is defined in the [CLI reference](cli.md).
- [Minimal D&D spell configuration](../examples/dnd-spells.config.yml)
- [Production-oriented D&D spell configuration](../examples/dnd-spells-production.config.yml)
Both complete files are validated by the CLI test suite. The fragments below
illustrate individual fields and are not alternate complete configurations.
Both are complete version 3 files. The fragments below illustrate individual
fields and are not alternate complete configurations.
## Top-Level Fields
- `version`: required. The only supported value is `2`.
- `version`: required. The only supported value is `3`.
- `scriptorium`: optional Scriptorium profile source settings.
- `pipelines`: optional map of pipeline IDs to pipeline definitions.
- `concurrency`: optional global concurrency settings.
- `workspace`: optional workspace settings for Notarius-owned local state.
- `diagnostics`: optional diagnostics settings.
- `output`: optional durable output placement.
- `cache`: optional chunk-plan and checkpoint cache placement.
- `debug`: optional debug-bundle placement. It does not enable debug capture.
Unknown YAML fields are rejected. The removed top-level `llm_profiles` field is
rejected; execution profiles now come from Scriptorium.
@@ -42,14 +44,13 @@ Built-in defaults:
- `concurrency.total_llm`: `1`
- `concurrency.stage_workers.extract`: effective `concurrency.total_llm`
- `diagnostics.work_dir`: `/tmp/notarius`
- `diagnostics.retention`: `auto`
- `workspace.directory`: unset
- `workspace.diagnostics.enabled`: `true`
- `workspace.resume.enabled`: `false`
- `workspace.debug.enabled`: `false`
- `workspace.chunk_cache.mode`: `auto`
- `workspace.chunk_cache.directory`: unset
- `output.directory`: `./notarius-output`
- `cache.chunk_plans.mode`: `auto`
- `cache.chunk_plans.directory`: unset, selecting
`<os.UserCacheDir>/notarius/chunk-plans`
- `cache.checkpoints.directory`: unset, selecting
`<os.UserCacheDir>/notarius/checkpoints`
- `debug.directory`: `./notarius-debug`
No pipelines are built in. A run requires a configured pipeline.
@@ -92,22 +93,15 @@ These environment variables are applied after the config file:
- `NOTARIUS_CONFIG`: config discovery path.
- `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency.
- `NOTARIUS_STAGE_WORKERS_EXTRACT`: integer extract worker limit.
- `NOTARIUS_WORKSPACE_DIR`: workspace directory.
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`: boolean diagnostics enablement.
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`: workspace diagnostics retention
mode.
- `NOTARIUS_WORKSPACE_RESUME_ENABLED`: boolean resume checkpointing
enablement.
- `NOTARIUS_WORKSPACE_DEBUG_ENABLED`: boolean debug artifact enablement.
- `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`: chunk-plan cache mode.
- `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`: chunk-plan cache root.
- `NOTARIUS_WORK_DIR`: deprecated diagnostics work directory compatibility
override.
- `NOTARIUS_DIAGNOSTICS_RETENTION`: deprecated diagnostics retention
compatibility override.
- `NOTARIUS_OUTPUT_DIR`: durable output root.
- `NOTARIUS_CACHE_CHUNK_PLANS_MODE`: chunk-plan cache mode.
- `NOTARIUS_CACHE_CHUNK_PLANS_DIR`: chunk-plan cache root.
- `NOTARIUS_CACHE_CHECKPOINTS_DIR`: checkpoint cache root.
- `NOTARIUS_DEBUG_DIR`: debug-bundle root.
Integer environment values must parse as base-10 integers. Boolean environment
values must parse as Go booleans such as `true`, `false`, `1`, or `0`.
Integer environment values must parse as base-10 integers. Directory overrides
must be non-empty after trimming. Cache-directory fields in a file may be
empty, which deliberately selects the corresponding per-user default.
The removed `NOTARIUS_LLM_DEFAULT_*` variables are not read. Configure provider
endpoint, model, and credential environment variable names through Scriptorium
@@ -341,71 +335,111 @@ Both modules accept UTF-8 plain text, Markdown, YAML, or JSON reference files.
The extractor uses references only as supporting disambiguation material; spell
casts still must be present in the source transcript.
## Workspace
## State Surfaces
`workspace` fields:
The `output`, `cache`, and `debug` top-level fields select independent physical
roots. Their layout, permissions, lifecycle, and sensitive-data handling are
defined in [Operations](operations.md).
- `directory`: optional workspace root for Notarius-owned local state.
- `resume.enabled`: boolean resume checkpointing setting.
- `debug.enabled`: boolean debug artifact setting.
- `chunk_cache.mode`: persistent chunk-plan cache mode: `auto`, `bypass`, or
`refresh`. The default is `auto`.
- `chunk_cache.directory`: optional chunk-plan cache root. This value is the
root itself; Notarius does not append `chunk-plans` to it.
- `diagnostics`: optional diagnostics settings defined below.
```yaml
output:
directory: ./notarius-output
cache:
chunk_plans:
directory: ""
mode: auto
checkpoints:
directory: ""
debug:
directory: ./notarius-debug
```
`workspace.resume.enabled`, `workspace.debug.enabled`, and
`workspace.chunk_cache` are independent. `workspace.directory` does not affect
chunk-plan placement. For directory layout, state lifecycle, permissions, and
sensitive content, see [Operations](operations.md).
`output.directory` is the durable output root. Its precedence is
`--output-dir`, `NOTARIUS_OUTPUT_DIR`, the file value, then the default.
`chunk_cache.mode` accepts only `auto`, `bypass`, and `refresh`. In `auto`, a
valid source-addressed plan is reused and a missing or invalid record is
regenerated and published after chunk validation. `bypass` neither reads nor
writes plan-cache state. `refresh` always generates a new plan and publishes it
only after validation succeeds.
`cache.chunk_plans.mode` accepts `auto`, `bypass`, or `refresh`. Its precedence
is `--chunk_cache`, `NOTARIUS_CACHE_CHUNK_PLANS_MODE`, the file value, then
`auto`. `auto` reuses a valid source-addressed plan and regenerates missing or
invalid records; `bypass` performs no plan-cache I/O; `refresh` regenerates and
publishes a plan after chunk validation.
Configuration values are applied in file then environment order; an explicit
`--chunk_cache` CLI value has highest precedence for the mode. The cache root
is selected from environment, file, then the per-user default; there is no CLI
root override. Every supplied value is parsed strictly even when a higher
precedence value wins, so malformed configuration is still an error.
`cache.chunk_plans.directory` and `cache.checkpoints.directory` each name an
exact cache-family root. Their precedence is the corresponding environment
variable, the file value, then the family-specific per-user default. There is
no CLI cache-root override. The defaults are
`<os.UserCacheDir>/notarius/chunk-plans` and
`<os.UserCacheDir>/notarius/checkpoints`; on Unix, `os.UserCacheDir` ordinarily
uses an absolute `$XDG_CACHE_HOME` or falls back to `$HOME/.cache`. A relative
`XDG_CACHE_HOME` is an error.
When `chunk_cache.directory` is unset, the root is
`<os.UserCacheDir>/notarius/chunk-plans`. On Unix this is ordinarily
`$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is an absolute
path, or `$HOME/.cache/notarius/chunk-plans` when it is unset. A relative
`XDG_CACHE_HOME` is rejected by `os.UserCacheDir`; Notarius reports that as a
configuration error and does not fall back to another directory.
Checkpoint I/O occurs only for `notarius run --resume`. That invocation loads
compatible checkpoints and records work it executes. Without `--resume`,
Notarius does not resolve, create, load, or record the checkpoint root.
## Diagnostics
`debug.directory` chooses a root but never enables debug capture. Its precedence
is `--debug-dir`, `NOTARIUS_DEBUG_DIR`, the file value, then the default.
Only `--debug` requests a bundle; `--debug-dir` is valid only with `--debug`.
Preferred workspace diagnostics fields:
Every supplied file, environment, and CLI value is validated even when a
higher-precedence value wins.
- `workspace.diagnostics.enabled`: set to `false` to skip creating diagnostics
run directories and diagnostics artifacts.
- `workspace.diagnostics.retention`: `auto`, `always`, or `never`.
## Version 2 To Version 3 Migration
Defaults for workspace and diagnostics fields are listed in
[Defaults](#defaults).
Version 2 files are rejected. Move each setting to the surface it controls and
remove obsolete enablement and retention controls. This complete before/after
example preserves an existing chunk-plan cache and checkpoint directory while
choosing an output and debug root explicitly.
`workspace.diagnostics.retention` overrides legacy diagnostics retention when
set.
```yaml
# Version 2 (no longer accepted)
version: 2
workspace:
directory: /srv/notarius/state
resume:
enabled: true
debug:
enabled: true
chunk_cache:
directory: /srv/notarius/chunk-plans
mode: auto
diagnostics:
retention: always
diagnostics:
work_dir: /srv/notarius/inspection
```
`diagnostics` fields:
```yaml
# Version 3
version: 3
output:
directory: /srv/notarius/output
cache:
chunk_plans:
directory: /srv/notarius/chunk-plans
mode: auto
checkpoints:
directory: /srv/notarius/state/checkpoints
debug:
directory: /srv/notarius/debug
```
- `work_dir`: deprecated compatibility directory for per-run diagnostics.
- `retention`: deprecated compatibility retention mode. `auto`, `always`, or
`never`.
Run the migrated configuration with `--resume` when checkpoint reuse or
recording is wanted, and with `--debug` when a debug bundle is wanted.
Existing `diagnostics.work_dir`, `diagnostics.retention`, `NOTARIUS_WORK_DIR`,
and `NOTARIUS_DIAGNOSTICS_RETENTION` inputs remain supported for compatibility.
New configuration should use `workspace.directory` and
`workspace.diagnostics.retention` instead.
For retention behavior and the physical diagnostics layout, see
[Operations](operations.md#retention). For the invocation-specific diagnostics
override, see [CLI Reference](cli.md#run).
The removed fields are `workspace.directory`, `workspace.resume.enabled`,
`workspace.debug.enabled`, `workspace.chunk_cache.mode`,
`workspace.chunk_cache.directory`, `workspace.diagnostics.enabled`,
`workspace.diagnostics.retention`, `diagnostics.work_dir`, and
`diagnostics.retention`. The removed environment variables are `NOTARIUS_WORKSPACE_DIR`,
`NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`,
`NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`,
`NOTARIUS_WORKSPACE_RESUME_ENABLED`, `NOTARIUS_WORKSPACE_DEBUG_ENABLED`,
`NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`,
`NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`, `NOTARIUS_WORK_DIR`, and
`NOTARIUS_DIAGNOSTICS_RETENTION`. The chunk-cache variables are replaced by
`NOTARIUS_CACHE_CHUNK_PLANS_MODE` and
`NOTARIUS_CACHE_CHUNK_PLANS_DIR`; the former shared directory has no direct
replacement.
## Validation
@@ -417,8 +451,8 @@ Configuration validation checks:
- positive global LLM concurrency;
- supported stage-worker keys and an effective extract worker count in the
inclusive range `1..concurrency.total_llm`;
- supported diagnostics retention and non-empty work directory;
- a supported chunk-cache mode and a chunk-cache directory without NUL bytes;
- non-empty output and debug directories;
- a supported chunk-cache mode and state-surface directories without NUL bytes;
- stale removed fields such as `llm_profiles`.
Pipeline resolution additionally checks: