notarius/docs/operations.md

Operations

This is the canonical reference for operating implemented Notarius runs.

Normal Run

A run reads one source file, resolves one configured pipeline, executes its modules, writes durable output, and writes diagnostics when enabled. Start with the README quickstart, then use the CLI reference for invocation options.

For production, configure an application-owned workspace such as /var/lib/notarius and ensure the Notarius process can create files below it. For local development, prefer an ignored project-local workspace such as ./.notarius/workspace. See Configuration for workspace fields.

Output Directory

Durable output is written to:

<output-root>/<run-id>/

The output root and its invocation-specific override are defined in the CLI reference. Output writes are atomic per file. The JSON output contract defines the logical files, paths, schemas, and media types inside each run directory.

Diagnostics Directory

Diagnostics are written under:

<diagnostics-work-dir>/<run-id>/

When a workspace directory is configured, diagnostics are written under <workspace.directory>/diagnostics/<run-id>/. An invocation-specific override changes only the diagnostics root, not the workspace root. Configuration and environment controls are defined in Configuration; the override flag is defined in the CLI reference.

Diagnostics can be disabled through configuration. When disabled, Notarius does not create a diagnostics run directory or write diagnostics artifacts; concise failures are still printed to stderr.

Implemented diagnostics artifacts:

• invocation.json: command metadata such as operation, config path, input path, selected lanes, run ID, and pipeline digest when available.
• effective-config.json: resolved config without raw API keys.
• resolved-pipeline.json: resolved module bindings and pipeline digest.
• resolved-references.json: resolved reference provenance, including target stage, lane ID when present, origin, digest, media type, byte size, and binding source, without reference content.
• checkpoint-events.json: checkpoint steps that were reused or executed during an explicit resume invocation.
• run-manifest.json: the same run manifest written to durable output when it is available, including top-level module metadata when present.
• warnings.json: warning list.
• run-report.json: counts, status, output path, diagnostics path, and run ID.
• error.log: failure message, written after diagnostics directory creation when a run fails.

Checkpoints

When checkpoint writing is enabled for a configured workspace, runs write checkpoints under:

<workspace.directory>/checkpoints/<pipeline-id>/<input-key>-<source-or-input-digest>/<pipeline-digest>/<identity-digest>/

Each workflow step owns its own manifest and payload files. There is no root-level checkpoint summary. Ordinary invocations execute the pipeline normally and refresh checkpoints. An explicit resume invocation reuses valid checkpoints and executes any missing, invalid, or incompatible step normally. Configuration controls checkpoint writing, while the explicit resume option is defined in the Configuration and CLI references.

Checkpoints do not include raw prompts, raw reference contents, raw LLM request payloads, or debug traces. They can still contain source text, intermediate extracted content, rejected outputs, metadata, warnings, and content digests. Treat checkpoint directories as sensitive local state.

A checkpoint is reused only when its stored status, dependencies, payloads, and digests match the current invocation. Changes to input bytes, the resolved pipeline, selected lanes, the runtime LLM profile override, or bound reference content invalidate reuse. The resolved pipeline identity includes effective default and explicitly overridden validator chains, so adding, removing, reordering, or reconfiguring a validator invalidates checkpoints even when the pipeline profile itself is unchanged.

Typed artifact checkpoints additionally record codec-owned bytes, artifact kind, schema ID and version, exact schema digest, and media type. A missing or mismatched codec identity, or bytes the current codec cannot decode, is reported as a checkpoint reuse miss. The affected operation executes normally and, when checkpoint writing is enabled, replaces the incompatible checkpoint.

Current checkpoint manifests use workspace schema notarius.workspace.v2. Manifests written with notarius.workspace.v1 are incompatible because their chunk provenance has an older shape. On the first explicit resume after an upgrade, each affected checkpoint is treated as a reuse miss and its workflow step executes normally. The compatibility check does not migrate or delete the v1 files; when checkpoint writing is enabled, normal execution refreshes the affected checkpoint files in the current schema.

Runs do not reuse checkpoints unless explicitly requested. Without reuse, the workflow executes normally and refreshes checkpoint files when checkpointing is enabled.

Debug

When debug recording is enabled for a configured workspace, runs write debug artifacts under:

<workspace.directory>/debug/<run-id>/

Debug output is per invocation. It is independent of checkpointing and is not used for resume. Enabling debug does not write checkpoints, and enabling resume checkpointing does not write debug output.

Debug artifacts include inputs and outputs for source, chunk, extract, merge, normalize, and output work, structured LLM request and response data, validator requests and results, timing, and retry attempt metadata. LLM calls made inside a module retry write prompt-000N.json, response-000N.json, and response-content-000N.* files under that attempt directory and are linked from its llm_calls array. Validator calls use separate attempt scopes under validate/ and are not duplicated into the module attempt. Prompt content is written inline in the prompt artifact. The response metadata and body use the paired files described above; the body is pretty-printed JSON when possible and raw text otherwise. Retrying stages use these stable module-attempt paths:

chunk/attempt-<NN>.json
extract/<lane-id>/chunk-<NNNNNN>/attempt-<NN>.json
merge/<lane-id>/attempt-<NN>.json
merge/<lane-id>/attempt-<NN>/prompt-<NNNN>.json
merge/<lane-id>/attempt-<NN>/response-<NNNN>.json
merge/<lane-id>/attempt-<NN>/response-content-<NNNN>.<ext>

normalize/<lane-id>/attempt-<NN>.json
normalize/<lane-id>/attempt-<NN>/prompt-<NNNN>.json
normalize/<lane-id>/attempt-<NN>/response-<NNNN>.json
normalize/<lane-id>/attempt-<NN>/response-content-<NNNN>.<ext>

Every executed chunk, extract, merge, and normalize attempt has one terminal envelope recording acceptance, validator rejection, or a module, validator, candidate-serialization, or final-serialization error as applicable. It includes attempt-local warnings and any available candidate or rejection. A failure before a candidate exists has no candidate payload. If the envelope cannot be persisted, the run does not retry that module attempt and reports the debug failure together with any primary attempt error.

Checkpoint-reused chunk, extract, merge, and normalize work retains the stage-level input and output artifacts but has no retry-attempt artifacts because no module attempt executed. Debug artifacts may contain source material, reference material, prompt inputs, model outputs, and other sensitive data. Typed artifact envelopes include domain-neutral codec identity, redacted metadata and content, and digests of the stable codec bytes. API keys are not written, and obvious credential-shaped values and sensitive map keys are redacted, but debug directories should still be protected as sensitive local state.

Retention

Diagnostics retention uses the effective mode selected through configuration; see Configuration for the fields, environment overrides, precedence, and default.

• auto: keep failed runs and successful runs with warnings; remove successful warning-free runs.
• always: keep every diagnostics run directory.
• never: remove successful run directories; failed runs are still retained.

Failures

Failures before diagnostics directory creation, such as a missing config file or an unusable diagnostics work directory, are printed to stderr and may not have a diagnostics run directory.

Failures after diagnostics directory creation are printed to stderr and written to error.log. Depending on where the failure occurred, the directory may also contain artifacts written before the failure.

If durable output writing fails after the pipeline completes, diagnostics are retained for inspection.

Warnings

A successful run with warnings exits with code 0, prints a warning count to stderr, and writes warnings to durable output and diagnostics when retained.

The JSON output contract defines durable warning and validation-status fields.

Cleanup

It is safe to remove specific old run directories after their output and diagnostics are no longer needed:

rm -rf /tmp/notarius/run-1234567890
rm -rf ./notarius-output/run-1234567890

Workspace checkpoint and debug directories can also be removed when no longer needed. Remove exact identity or run directories, for example:

rm -rf /var/lib/notarius/checkpoints/dnd-session/seriatim-abcdef123456/7890abcd1234/identityabcd1234
rm -rf /var/lib/notarius/debug/run-1234567890

Use exact run-directory paths. Avoid broad cleanup commands against parent directories unless they are part of your own operational policy.

Operational Limits

Provider retries and timeouts are handled by Scriptorium according to the selected execution profile. Pipeline module retry settings are defined in Configuration. There is no separate CLI retry command.

Extract worker concurrency and actual provider-call concurrency are separate limits. Their configuration, defaults, and validation are defined in Configuration. Cancellation stops undispatched extract work; already started work is allowed to finish or observe cancellation before the run reports failure.

Notarius writes local files only. Remote storage and archive management are not part of the implemented CLI.