187 lines
7.4 KiB
Markdown
187 lines
7.4 KiB
Markdown
# Operations
|
|
|
|
This is the canonical reference for operating implemented Notarius runs.
|
|
|
|
## Normal Run
|
|
|
|
A run reads one source file, resolves one configured pipeline, executes its
|
|
modules, writes durable output, and writes diagnostics when enabled. Start with
|
|
the [README quickstart](../README.md), then use the [CLI reference](cli.md) for
|
|
invocation options.
|
|
|
|
For production, configure an application-owned workspace such as
|
|
`/var/lib/notarius` and ensure the Notarius process can create files below it.
|
|
For local development, prefer an ignored project-local workspace such as
|
|
`./.notarius/workspace`. See [Configuration](config.md#workspace) for workspace
|
|
fields.
|
|
|
|
## Output Directory
|
|
|
|
Durable output is written to:
|
|
|
|
```text
|
|
<output-root>/<run-id>/
|
|
```
|
|
|
|
The output root and its invocation-specific override are defined in the
|
|
[CLI reference](cli.md#run). Output writes are atomic per file. The
|
|
[JSON output contract](integrations/json-output.md) defines the logical files,
|
|
paths, schemas, and media types inside each run directory.
|
|
|
|
## Diagnostics Directory
|
|
|
|
Diagnostics are written under:
|
|
|
|
```text
|
|
<diagnostics-work-dir>/<run-id>/
|
|
```
|
|
|
|
When a workspace directory is configured, diagnostics are written under
|
|
`<workspace.directory>/diagnostics/<run-id>/`. An invocation-specific override
|
|
changes only the diagnostics root, not the workspace root. Configuration and
|
|
environment controls are defined in [Configuration](config.md); the override
|
|
flag is defined in the [CLI reference](cli.md#run).
|
|
|
|
Diagnostics can be disabled through configuration. When disabled, Notarius
|
|
does not create a diagnostics run directory or write diagnostics artifacts;
|
|
concise failures are still printed to stderr.
|
|
|
|
Implemented diagnostics artifacts:
|
|
|
|
- `invocation.json`: command metadata such as operation, config path, input
|
|
path, selected lanes, run ID, and pipeline digest when available.
|
|
- `effective-config.json`: resolved config without raw API keys.
|
|
- `resolved-pipeline.json`: resolved module bindings and pipeline digest.
|
|
- `resolved-references.json`: resolved reference provenance, including target
|
|
stage, lane ID when present, origin, digest, media type, byte size, and
|
|
binding source, without reference content.
|
|
- `checkpoint-events.json`: checkpoint steps that were reused or executed
|
|
during an explicit resume invocation.
|
|
- `run-manifest.json`: the same run manifest written to durable output when it
|
|
is available, including top-level module metadata when present.
|
|
- `warnings.json`: warning list.
|
|
- `run-report.json`: counts, status, output path, diagnostics path, and run ID.
|
|
- `error.log`: failure message, written after diagnostics directory creation
|
|
when a run fails.
|
|
|
|
## Checkpoints
|
|
|
|
When checkpoint writing is enabled for a configured workspace, runs write
|
|
checkpoints under:
|
|
|
|
```text
|
|
<workspace.directory>/checkpoints/<pipeline-id>/<input-key>-<source-or-input-digest>/<pipeline-digest>/<identity-digest>/
|
|
```
|
|
|
|
Each workflow step owns its own manifest and payload files. There is no
|
|
root-level checkpoint summary. Ordinary invocations execute the pipeline
|
|
normally and refresh checkpoints. An explicit resume invocation reuses valid
|
|
checkpoints and executes any missing, invalid, or incompatible step normally.
|
|
Configuration controls checkpoint writing, while the explicit resume option is
|
|
defined in the [Configuration](config.md#workspace) and
|
|
[CLI](cli.md#run) references.
|
|
|
|
Checkpoints do not include raw prompts, raw reference contents, raw LLM request
|
|
payloads, or debug traces. They can still contain source text, intermediate
|
|
extracted content, rejected outputs, metadata, warnings, and content digests.
|
|
Treat checkpoint directories as sensitive local state.
|
|
|
|
A checkpoint is reused only when its stored status, dependencies, payloads, and
|
|
digests match the current invocation. Changes to input bytes, the resolved
|
|
pipeline, selected lanes, the runtime LLM profile override, or bound reference
|
|
content invalidate reuse.
|
|
|
|
Runs do not reuse checkpoints unless explicitly requested. Without reuse, the
|
|
workflow executes normally and refreshes checkpoint files when checkpointing is
|
|
enabled.
|
|
|
|
## Debug
|
|
|
|
When debug recording is enabled for a configured workspace, runs write debug
|
|
artifacts under:
|
|
|
|
```text
|
|
<workspace.directory>/debug/<run-id>/
|
|
```
|
|
|
|
Debug output is per invocation. It is independent of checkpointing and is not
|
|
used for resume. Enabling debug does not write checkpoints, and enabling resume
|
|
checkpointing does not write debug output.
|
|
|
|
Debug artifacts include inputs and outputs for source, chunk, extract, merge,
|
|
normalize, and output work, structured LLM request and response data, validator
|
|
requests and results, timing, and retry attempt metadata. LLM calls made inside
|
|
a retry or validator attempt
|
|
write `prompt-000N.json`, `response-000N.json`, and
|
|
`response-content-000N.*` files under that attempt directory and are linked from
|
|
the attempt `llm_calls` array. Prompt content is written inline in the prompt
|
|
artifact. The response metadata and body use the paired files described above;
|
|
the body is pretty-printed JSON when possible and raw text otherwise. Debug
|
|
artifacts may contain source material, reference material, prompt inputs, model
|
|
outputs, and other sensitive data. API keys are not written, and obvious
|
|
credential-shaped values and sensitive map keys are redacted, but debug
|
|
directories should still be protected as sensitive local state.
|
|
|
|
## Retention
|
|
|
|
Diagnostics retention uses the effective mode selected through configuration;
|
|
see [Configuration](config.md#diagnostics) for the fields, environment
|
|
overrides, precedence, and default.
|
|
|
|
- `auto`: keep failed runs and successful runs with warnings; remove successful
|
|
warning-free runs.
|
|
- `always`: keep every diagnostics run directory.
|
|
- `never`: remove successful run directories; failed runs are still retained.
|
|
|
|
## Failures
|
|
|
|
Failures before diagnostics directory creation, such as a missing config file or
|
|
an unusable diagnostics work directory, are printed to stderr and may not have a
|
|
diagnostics run directory.
|
|
|
|
Failures after diagnostics directory creation are printed to stderr and written
|
|
to `error.log`. Depending on where the failure occurred, the directory may also
|
|
contain artifacts written before the failure.
|
|
|
|
If durable output writing fails after the pipeline completes, diagnostics are
|
|
retained for inspection.
|
|
|
|
## Warnings
|
|
|
|
A successful run with warnings exits with code `0`, prints a warning count to
|
|
stderr, and writes warnings to durable output and diagnostics when retained.
|
|
|
|
The [JSON output contract](integrations/json-output.md) defines durable warning
|
|
and validation-status fields.
|
|
|
|
## Cleanup
|
|
|
|
It is safe to remove specific old run directories after their output and
|
|
diagnostics are no longer needed:
|
|
|
|
```sh
|
|
rm -rf /tmp/notarius/run-1234567890
|
|
rm -rf ./notarius-output/run-1234567890
|
|
```
|
|
|
|
Workspace checkpoint and debug directories can also be removed when no longer
|
|
needed. Remove exact identity or run directories, for example:
|
|
|
|
```sh
|
|
rm -rf /var/lib/notarius/checkpoints/dnd-session/seriatim-abcdef123456/7890abcd1234/identityabcd1234
|
|
rm -rf /var/lib/notarius/debug/run-1234567890
|
|
```
|
|
|
|
Use exact run-directory paths. Avoid broad cleanup commands against parent
|
|
directories unless they are part of your own operational policy.
|
|
|
|
## Operational Limits
|
|
|
|
Provider retries and timeouts are handled by Scriptorium according to the
|
|
selected execution profile. Pipeline module retry settings are defined in
|
|
[Configuration](config.md#module-bindings). There is no separate CLI retry
|
|
command.
|
|
|
|
Notarius writes local files only. Remote storage and archive management are not
|
|
part of the implemented CLI.
|