eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document output cache and debug state model

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-18 13:08:27 +00:00
parent 8cb11e60e4
commit 8824948910
14 changed files with 354 additions and 1478 deletions
Download Patch File Download Diff File Expand all files Collapse all files

347
docs/operations.md
View File

@@ -1,303 +1,170 @@
# Operations
This is the canonical reference for operating implemented Notarius runs.
This is the canonical guide to operating Notarius filesystem state. Command
syntax is in the [CLI reference](cli.md); field definitions and precedence are
in [Configuration](config.md).
## Normal Run
## State Model
A run reads one source file, resolves one configured pipeline, executes its
modules, writes durable output, and writes diagnostics when enabled. Start with
the [README quickstart](../README.md), then use the [CLI reference](cli.md) for
invocation options.
Notarius uses three independent filesystem surfaces:
For production, configure an application-owned workspace such as
`/var/lib/notarius` and ensure the Notarius process can create files below it.
For local development, prefer an ignored project-local workspace such as
`./.notarius/workspace`. See [Configuration](config.md#workspace) for workspace
fields.
- output is durable user data;
- cache is reconstructible chunk-plan and checkpoint state; and
- debug is explicitly requested inspection data.
## Output Directory
Choose separate roots and access controls for each surface. A normal run writes
durable output and may use the chunk-plan cache. It does not create checkpoint
or debug state unless its invocation includes `--resume` or `--debug`.
Durable output is written to:
## Output
Durable logical files are written under:
```text
<output-root>/<run-id>/
```
The output root and its invocation-specific override are defined in the
[CLI reference](cli.md#run). Output writes are atomic per file. The
[JSON output contract](integrations/json-output.md) defines the logical files,
paths, schemas, and media types inside each run directory.
Each output file is written atomically. Notarius never automatically removes
output. The [JSON output contract](integrations/json-output.md) owns the
logical file names, schemas, and media types inside a run directory.
## Diagnostics Directory
Diagnostics are written under:
```text
<diagnostics-work-dir>/<run-id>/
```
When a workspace directory is configured, diagnostics are written under
`<workspace.directory>/diagnostics/<run-id>/`. An invocation-specific override
changes only the diagnostics root, not the workspace root. Configuration and
environment controls are defined in [Configuration](config.md); the override
flag is defined in the [CLI reference](cli.md#run).
Diagnostics can be disabled through configuration. When disabled, Notarius
does not create a diagnostics run directory or write diagnostics artifacts;
concise failures are still printed to stderr.
Implemented diagnostics artifacts:
- `invocation.json`: command metadata such as operation, config path, input
path, selected lanes, run ID, and pipeline digest when available.
- `effective-config.json`: resolved config without raw API keys.
- `resolved-pipeline.json`: resolved module bindings and pipeline digest.
- `resolved-references.json`: resolved reference provenance, including target
stage, lane ID when present, origin, digest, media type, byte size, and
binding source, without reference content.
- `checkpoint-events.json`: checkpoint steps that were reused or executed
during an explicit resume invocation.
- `chunk-plan.json`: redacted plan-cache lookup, validation, and publication
summary. It contains identifiers and decisions, never source units, plan
annotations, reference content, prompts, model responses, or invalid-file
bytes.
- `run-manifest.json`: the same run manifest written to durable output when it
is available, including top-level module metadata when present.
- `warnings.json`: warning list.
- `run-report.json`: counts, status, output path, diagnostics path, and run ID.
- `error.log`: failure message, written after diagnostics directory creation
when a run fails.
Remove an output run directory only after its consumer data is no longer
needed. This is data deletion, not cache cleanup.
## Chunk-Plan Cache
The chunk-plan cache is independent of the workspace and checkpoints. Its
configuration and selection precedence are defined in
[Configuration](config.md#workspace); the invocation override is documented in
the [CLI reference](cli.md#run).
When no root is configured, a normal Linux user uses
`$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is a valid absolute
path, or `$HOME/.cache/notarius/chunk-plans` when it is unset. A relative
`XDG_CACHE_HOME` is a configuration error. A configured
`workspace.chunk_cache.directory` is the root itself, not a parent to which
Notarius adds a suffix.
Each source digest has one file:
Chunk plans are stored at:
```text
<chunk-plan-root>/<source-sha256-hex>/plan.json
```
Directories are created with `0700` permissions and plan files with `0600`.
`auto` reuses a complete valid plan or regenerates an absent or invalid one;
`refresh` deliberately regenerates; `bypass` performs no cache I/O. A stored
plan is still validated and materialized against the current source before use,
and the current run's chunk validators always run. Invalid state is recoverable:
an `auto` run regenerates and atomically replaces it only after validation
succeeds. Delete an exact cache root or digest directory only when regeneration
cost is acceptable.
`auto` reuses a complete valid plan or regenerates missing or invalid state.
`refresh` regenerates and atomically replaces a plan after chunk validation.
`bypass` performs no plan-cache I/O and does not resolve or create the root.
Plan selection is source-addressed and independent of checkpoint and debug
roots.
The configured root is the cache trust boundary. An operator-supplied root path
may itself resolve through a symlink, but cache-owned digest directories and
plan files must be real directory and regular-file entries. Links or other
unexpected entry types are rejected rather than followed.
When its directory is empty in configuration, the root is
`<os.UserCacheDir>/notarius/chunk-plans`. A configured directory is the exact
root; no suffix is appended. Directories and files created by the store use
`0700` and `0600` permissions on supported Unix systems. The configured root
is a trust boundary: do not share it among mutually untrusted users.
Publication uses atomic replacement. Concurrent readers observe a complete old
or new plan, and concurrent writers leave one complete valid winner; there is
no history, lock protocol, or rollback facility. Do not share a cache root
between mutually untrusted users because plans can contain source-derived
structure and annotations.
Remove an exact digest directory or the configured root only when accepting the
cost of recomputing plans and any chunk-stage work. Cache publication is atomic;
there is no history, locking, garbage collection, or rollback facility.
For a system-wide Linux deployment under a dedicated service account, configure
and provision a separate restrictive root such as:
For a Linux service account, provision a dedicated restrictive root such as:
```yaml
workspace:
chunk_cache:
cache:
chunk_plans:
directory: /var/cache/notarius/chunk-plans
```
`/var/cache/notarius/chunk-plans` is a recommended configured service root, not
the unprivileged default. The operator or package installer must create it with
restrictive service-account ownership and permissions before use.
## Checkpoint Cache
## Checkpoints
Checkpoint state is used only by an invocation with `--resume`. That invocation
loads compatible completed work and records checkpoints for work it executes.
Without `--resume`, Notarius neither resolves nor creates the checkpoint root,
and neither loads nor records checkpoints.
When checkpoint writing is enabled for a configured workspace, runs write
checkpoints under:
Checkpoints use the selected root and the existing identity hierarchy:
```text
<workspace.directory>/checkpoints/<pipeline-id>/<input-key>-<source-or-input-digest>/<pipeline-digest>/<identity-digest>/
<checkpoint-root>/<pipeline-id>/<input-key>-<source-or-input-digest>/<pipeline-digest>/<identity-digest>/...
```
Each workflow step owns its own manifest and payload files. There is no
root-level checkpoint summary. Ordinary invocations execute the pipeline
normally and refresh checkpoints. An explicit resume invocation reuses valid
checkpoints and executes any missing, invalid, or incompatible step normally.
Configuration controls checkpoint writing, while the explicit resume option is
defined in the [Configuration](config.md#workspace) and
[CLI](cli.md#run) references.
An empty configured directory selects
`<os.UserCacheDir>/notarius/checkpoints`. The root is exact when configured.
Created directories and files use `0700` and `0600` permissions on supported
Unix systems.
Checkpoints do not include raw prompts, raw reference contents, raw LLM request
payloads, or debug traces. They can still contain source text, intermediate
extracted content, rejected outputs, metadata, warnings, and content digests.
Treat checkpoint directories as sensitive local state.
Checkpoint payloads can contain source text, intermediate artifacts, metadata,
warnings, and content digests. Treat them as sensitive derived application
data. Compatible files from a former checkpoint root remain reusable when
`cache.checkpoints.directory` names that exact existing root. They are not
moved, migrated, or deleted automatically. The frozen serialized identifier
`workspace_schema_version` remains part of checkpoint compatibility; it is not
a configuration setting.
A checkpoint is reused only when its stored status, dependencies, payloads, and
digests match the current invocation. Changes to input bytes, the resolved
pipeline, selected lanes, the runtime LLM profile override, or bound reference
content invalidate reuse. The resolved pipeline identity includes effective
default and explicitly overridden validator chains, so adding, removing,
reordering, or reconfiguring a validator invalidates checkpoints even when the
pipeline profile itself is unchanged.
For a Linux service account, independently provision:
Typed artifact checkpoints additionally record codec-owned bytes, artifact
kind, schema ID and version, exact schema digest, and media type. A missing or
mismatched codec identity, or bytes the current codec cannot decode, is reported
as a checkpoint reuse miss. The affected operation executes normally and, when
checkpoint writing is enabled, replaces the incompatible checkpoint.
```yaml
cache:
checkpoints:
directory: /var/cache/notarius/checkpoints
```
Current checkpoint manifests use workspace schema `notarius.workspace.v2`.
Manifests written with `notarius.workspace.v1` are incompatible because their
chunk provenance has an older shape. On the first explicit resume after an
upgrade, each affected checkpoint is treated as a reuse miss and its workflow
step executes normally. The compatibility check does not migrate or delete the
v1 files; when checkpoint writing is enabled, normal execution refreshes the
affected checkpoint files in the current schema.
Remove an exact checkpoint identity directory or the configured root only when
recomputation is acceptable.
Runs do not reuse checkpoints unless explicitly requested. Without reuse, the
workflow executes normally and refreshes checkpoint files when checkpointing is
enabled.
## Debug Bundles
## Debug
When debug recording is enabled for a configured workspace, runs write debug
artifacts under:
Only `notarius run --debug` enables debug collection. The selected root contains
one retained bundle per invocation:
```text
<workspace.directory>/debug/<run-id>/
<debug-root>/<run-id>/
summary/
trace/
```
Debug output is per invocation. It is independent of checkpointing and is not
used for resume. Enabling debug does not write checkpoints, and enabling resume
checkpointing does not write debug output.
`summary/` contains redacted invocation, effective-configuration, resolved
pipeline and reference provenance, checkpoint and chunk-plan decisions, run
manifest, warnings, report, and any available error text. It excludes raw
source, references, annotations, prompts, model responses, credentials, and
malformed cache bytes.
Debug artifacts include inputs and outputs for source, chunk, extract, merge,
normalize, and output work, structured LLM request and response data, validator
requests and results, timing, and retry attempt metadata. LLM calls made inside
a module retry write `prompt-000N.json`, `response-000N.json`, and
`response-content-000N.*` files under that attempt directory and are linked
from its `llm_calls` array. Validator calls use separate attempt scopes under
`validate/` and are not duplicated into the module attempt. Prompt content is
written inline in the prompt artifact. The response metadata and body use the
paired files described above; the body is pretty-printed JSON when possible
and raw text otherwise. Retrying stages use these stable module-attempt paths:
`trace/` contains application-owned execution detail, including source and
stage material, plans, chunks, validator attempts, prompts, model responses,
timing, and serialized artifacts. It may retain application data omitted from
output. Credentials, credential-shaped values, sensitive metadata, unrelated
environment values, and unrelated filesystem content are not captured.
```text
chunk/attempt-<NN>.json
extract/<lane-id>/chunk-<NNNNNN>/attempt-<NN>.json
merge/<lane-id>/attempt-<NN>.json
merge/<lane-id>/attempt-<NN>/prompt-<NNNN>.json
merge/<lane-id>/attempt-<NN>/response-<NNNN>.json
merge/<lane-id>/attempt-<NN>/response-content-<NNNN>.<ext>
Bundles inherit the sensitivity of the application data they capture. Their
additional risk comes from copying and aggregating that data, so restrict
access, avoid shared roots between untrusted users, and define retention outside
Notarius. Created bundle directories use `0700` and files use `0600` on
supported Unix systems.
normalize/<lane-id>/attempt-<NN>.json
normalize/<lane-id>/attempt-<NN>/prompt-<NNNN>.json
normalize/<lane-id>/attempt-<NN>/response-<NNNN>.json
normalize/<lane-id>/attempt-<NN>/response-content-<NNNN>.<ext>
```
Notarius never automatically deletes a requested bundle. If allocation
succeeds, its path is reported on success and failure. A requested summary or
trace write failure makes the command fail, preserving whatever bundle data was
already written for inspection.
Every executed chunk, extract, merge, and normalize attempt has one terminal
envelope recording acceptance, validator rejection, or a module, validator,
candidate-serialization, or final-serialization error as applicable. It
includes attempt-local warnings and any available candidate or rejection. A
failure before a candidate exists has no candidate payload. If the envelope
cannot be persisted, the run does not retry that module attempt and reports the
debug failure together with any primary attempt error.
## Failures And Warnings
Chunk-plan candidates, materialized chunks, annotations, and chunk-attempt
details appear only in these opt-in debug artifacts. They are intentionally not
included in normal manifests or the `chunk-plan.json` diagnostics summary.
Checkpoint-reused extract, merge, and normalize work retains the stage-level
input and output artifacts but has no retry-attempt artifacts
because no module attempt executed. Debug artifacts may contain source
material, reference material, prompt inputs, model outputs, and other sensitive
data. Typed artifact
envelopes include domain-neutral codec identity, redacted metadata and content,
and digests of the stable codec bytes. API keys are not written, and obvious
credential-shaped values and sensitive map keys are redacted, but debug
directories should still be protected as sensitive local state.
## Retention
Diagnostics retention uses the effective mode selected through configuration;
see [Configuration](config.md#diagnostics) for the fields, environment
overrides, precedence, and default.
- `auto`: keep failed runs and successful runs with warnings; remove successful
warning-free runs.
- `always`: keep every diagnostics run directory.
- `never`: remove successful run directories; failed runs are still retained.
## Failures
Failures before diagnostics directory creation, such as a missing config file or
an unusable diagnostics work directory, are printed to stderr and may not have a
diagnostics run directory.
Failures after diagnostics directory creation are printed to stderr and written
to `error.log`. Depending on where the failure occurred, the directory may also
contain artifacts written before the failure.
If durable output writing fails after the pipeline completes, diagnostics are
retained for inspection.
## Warnings
A successful run with warnings exits with code `0`, prints a warning count to
stderr, and writes warnings to durable output and diagnostics when retained.
The [JSON output contract](integrations/json-output.md) defines durable warning
and validation-status fields.
Failures before debug allocation are reported on stderr without a bundle.
Failures after allocation report the bundle path on stderr and write `error.log`
when that summary write succeeds. An output-write failure leaves the allocated
bundle in place. A successful run with warnings exits `0`, reports a warning
count on stderr, and records warnings in durable output and any requested debug
summary.
## Cleanup
It is safe to remove specific old run directories after their output and
diagnostics are no longer needed:
Use exact paths for manual cleanup. Examples:
```sh
rm -rf /tmp/notarius/run-1234567890
rm -rf ./notarius-output/run-1234567890
rm -rf /var/cache/notarius/chunk-plans/0123abcd
rm -rf /var/cache/notarius/checkpoints/pipeline/input-0123/pipeline-4567/identity-89ab
rm -rf ./notarius-debug/run-1234567890
```
Workspace checkpoint and debug directories can also be removed when no longer
needed. Remove exact identity or run directories, for example:
```sh
rm -rf /var/lib/notarius/checkpoints/dnd-session/seriatim-abcdef123456/7890abcd1234/identityabcd1234
rm -rf /var/lib/notarius/debug/run-1234567890
```
Chunk-plan cache entries can likewise be removed by exact digest directory or
configured root. Removal is recoverable, but the next non-bypass run may need
to regenerate plans and repeat any chunk-stage LLM work.
Use exact run-directory paths. Avoid broad cleanup commands against parent
directories unless they are part of your own operational policy.
Avoid broad recursive cleanup against a parent root unless it is an explicit
operator policy. Output deletion is permanent user-data loss. Cache deletion is
recoverable but can repeat expensive work. Debug deletion removes troubleshooting
evidence and any retained application-data copy.
## Operational Limits
Provider retries and timeouts are handled by Scriptorium according to the
selected execution profile. Pipeline module retry settings are defined in
[Configuration](config.md#module-bindings). There is no separate CLI retry
command.
Extract worker concurrency and actual provider-call concurrency are separate
limits. Their configuration, defaults, and validation are defined in
[Configuration](config.md#concurrency). Cancellation stops undispatched extract
work; already started work is allowed to finish or observe cancellation before
the run reports failure.
Notarius writes local files only. Remote storage and archive management are not
part of the implemented CLI.
[Configuration](config.md#module-bindings). Extract worker concurrency and
actual provider-call concurrency are separate limits; their fields and
validation are defined in [Configuration](config.md#concurrency). Notarius
writes local files only; remote storage and archive management are outside the
implemented CLI.