notarius/docs/operations.md

# Operations

This is the canonical reference for operating implemented Notarius runs.

## Normal Run

A run reads one source file, resolves one configured pipeline, executes its
modules, writes durable output, and writes diagnostics when enabled. Start with
the [README quickstart](../README.md), then use the [CLI reference](cli.md) for
invocation options.

For production, configure an application-owned workspace such as
`/var/lib/notarius` and ensure the Notarius process can create files below it.
For local development, prefer an ignored project-local workspace such as
`./.notarius/workspace`. See [Configuration](config.md#workspace) for workspace
fields.

## Output Directory

Durable output is written to:

```text
<output-root>/<run-id>/
```

The output root and its invocation-specific override are defined in the
[CLI reference](cli.md#run). Output writes are atomic per file. The
[JSON output contract](integrations/json-output.md) defines the logical files,
paths, schemas, and media types inside each run directory.

## Diagnostics Directory

Diagnostics are written under:

```text
<diagnostics-work-dir>/<run-id>/
```

When a workspace directory is configured, diagnostics are written under
`<workspace.directory>/diagnostics/<run-id>/`. An invocation-specific override
changes only the diagnostics root, not the workspace root. Configuration and
environment controls are defined in [Configuration](config.md); the override
flag is defined in the [CLI reference](cli.md#run).

Diagnostics can be disabled through configuration. When disabled, Notarius
does not create a diagnostics run directory or write diagnostics artifacts;
concise failures are still printed to stderr.

Implemented diagnostics artifacts:

- `invocation.json`: command metadata such as operation, config path, input
  path, selected lanes, run ID, and pipeline digest when available.
- `effective-config.json`: resolved config without raw API keys.
- `resolved-pipeline.json`: resolved module bindings and pipeline digest.
- `resolved-references.json`: resolved reference provenance, including target
  stage, lane ID when present, origin, digest, media type, byte size, and
  binding source, without reference content.
- `checkpoint-events.json`: checkpoint steps that were reused or executed
  during an explicit resume invocation.
- `run-manifest.json`: the same run manifest written to durable output when it
  is available, including top-level module metadata when present.
- `warnings.json`: warning list.
- `run-report.json`: counts, status, output path, diagnostics path, and run ID.
- `error.log`: failure message, written after diagnostics directory creation
  when a run fails.

## Checkpoints

When `workspace.resume.enabled: true` and `workspace.directory` is set, runs
write checkpoints under:

```text
<workspace.directory>/checkpoints/<pipeline-id>/<input-key>-<source-or-input-digest>/<pipeline-digest>/<identity-digest>/
```

Each workflow step owns its own manifest and payload files. There is no
root-level checkpoint summary. Ordinary `notarius run` invocations execute the
pipeline normally and refresh checkpoints. `notarius run --resume` reuses valid
checkpoints and executes any missing, invalid, or incompatible step normally.

Checkpoints do not include raw prompts, raw reference contents, raw LLM request
payloads, or debug traces. They can still contain source text, intermediate
extracted content, rejected outputs, metadata, warnings, and content digests.
Treat checkpoint directories as sensitive local state.

A checkpoint is reused only when its stored status, dependencies, payloads, and
digests match the current invocation. Changes to input bytes, the resolved
pipeline, selected lanes, the runtime LLM profile override, or bound reference
content invalidate reuse.

Plain `notarius run` does not reuse checkpoints. It executes the workflow and
refreshes checkpoint files when checkpointing is enabled. `notarius run
--resume` is the explicit reuse path.

## Debug

When `workspace.debug.enabled: true` and `workspace.directory` is set, runs
write debug artifacts under:

```text
<workspace.directory>/debug/<run-id>/
```

Debug output is per invocation. It is independent of checkpointing and is not
used for resume. Enabling debug does not write checkpoints, and enabling resume
checkpointing does not write debug output.

Debug artifacts include inputs and outputs for source, chunk, extract, merge,
normalize, and output work, structured LLM request and response data, validator
requests and results, timing, and retry attempt metadata. LLM calls made inside
a retry or validator attempt
write `prompt-000N.json`, `response-000N.json`, and
`response-content-000N.*` files under that attempt directory and are linked from
the attempt `llm_calls` array. Prompt content is written inline in the prompt
artifact. Response metadata is written to `response-000N.json`, while the
response body is written separately as pretty-printed JSON when possible or as
raw text otherwise. Debug artifacts may contain source material, reference
material, prompt inputs, model outputs, and other sensitive data. API keys are
not written, and obvious credential-shaped values and sensitive map keys are
redacted, but debug directories should still be protected as sensitive local
state.

## Retention

Diagnostics retention uses the effective mode selected through configuration;
see [Configuration](config.md#diagnostics) for the fields, environment
overrides, precedence, and default.

- `auto`: keep failed runs and successful runs with warnings; remove successful
  warning-free runs.
- `always`: keep every diagnostics run directory.
- `never`: remove successful run directories; failed runs are still retained.

## Failures

Failures before diagnostics directory creation, such as a missing config file or
an unusable diagnostics work directory, are printed to stderr and may not have a
diagnostics run directory.

Failures after diagnostics directory creation are printed to stderr and written
to `error.log`. Depending on where the failure occurred, diagnostics may also
include invocation metadata, redacted effective config, resolved pipeline data,
the run manifest, warnings, and a run report.

If durable output writing fails after the pipeline completes, diagnostics are
retained for inspection and may include `run-manifest.json`, `warnings.json`,
`run-report.json`, and `error.log`.

## Warnings

A successful run with warnings exits with code `0`, prints a warning count to
stderr, and writes warnings to durable output and diagnostics when retained.

The [JSON output contract](integrations/json-output.md) defines durable warning
and validation-status fields.

## Cleanup

It is safe to remove specific old run directories after their output and
diagnostics are no longer needed:

```sh
rm -rf /tmp/notarius/run-1234567890
rm -rf ./notarius-output/run-1234567890
```

Workspace checkpoint and debug directories can also be removed when no longer
needed. Remove exact identity or run directories, for example:

```sh
rm -rf /var/lib/notarius/checkpoints/dnd-session/seriatim-abcdef123456/7890abcd1234/identityabcd1234
rm -rf /var/lib/notarius/debug/run-1234567890
```

Use exact run-directory paths. Avoid broad cleanup commands against parent
directories unless they are part of your own operational policy.

## Operational Limits

If `--resume` cannot reuse a checkpoint, Notarius executes that step and writes
a fresh checkpoint when checkpointing is enabled.

Provider retries and timeouts are handled by Scriptorium according to the
selected execution profile. Pipeline module retry settings are defined in
[Configuration](config.md#module-bindings). There is no separate CLI retry
command.

Notarius writes local files only. Remote storage and archive management are not
part of the implemented CLI.