221 lines
8.9 KiB
Markdown
221 lines
8.9 KiB
Markdown
# Operations
|
|
|
|
This is the canonical reference for operating implemented Notarius runs.
|
|
|
|
## Normal Run
|
|
|
|
A run reads one source file, resolves one configured pipeline, calls the
|
|
configured Scriptorium-backed LLM runtime, writes durable JSON output, and
|
|
writes diagnostics for inspection.
|
|
|
|
```sh
|
|
go run ./cmd/notarius run dnd-session \
|
|
--config examples/dnd-spells.config.yml \
|
|
--input examples/seriatim-minimal-transcript.json \
|
|
--output-dir ./notarius-output \
|
|
--diagnostics-dir /tmp/notarius
|
|
```
|
|
|
|
The command prints a success line with the pipeline ID, normalized output count,
|
|
rejected output count, and the output path.
|
|
|
|
For production, configure a workspace such as `/var/lib/notarius` and ensure the
|
|
Notarius process can create files below it. For local development, prefer an
|
|
ignored project-local workspace such as `./.notarius/workspace`.
|
|
|
|
## Output Directory
|
|
|
|
Durable output is written to:
|
|
|
|
```text
|
|
<output-root>/<run-id>/
|
|
```
|
|
|
|
The default output root is `./notarius-output`. Use `--output-dir` to choose a
|
|
different root.
|
|
|
|
The `json` output module writes these files:
|
|
|
|
- `index.json`: file index with paths to the manifest, lane output files,
|
|
rejected outputs, and warnings.
|
|
- `manifest.json`: run manifest with resolved pipeline provenance, top-level
|
|
module metadata, module keys, reference provenance, validation status, and
|
|
timing.
|
|
- `lanes/<lane-id>.json`: normalized raw JSON output payloads, one file per
|
|
lane. For the current D&D spell extractor, this includes `lanes/spells.json`.
|
|
- `rejected.json`: rejected raw output records.
|
|
- `warnings.json`: warnings reported by pipeline modules or the output encoder.
|
|
|
|
Output writes are atomic per file. Logical output file names must be clean,
|
|
relative, slash-separated paths and must not contain `..`.
|
|
|
|
## Diagnostics Directory
|
|
|
|
Diagnostics are written under:
|
|
|
|
```text
|
|
<diagnostics-work-dir>/<run-id>/
|
|
```
|
|
|
|
The default diagnostics work directory is `/tmp/notarius`. It can be set with
|
|
`workspace.directory`, `NOTARIUS_WORKSPACE_DIR`, legacy
|
|
`diagnostics.work_dir`, legacy `NOTARIUS_WORK_DIR`, or `--diagnostics-dir`.
|
|
When a workspace directory is set, diagnostics are written under
|
|
`<workspace.directory>/diagnostics/<run-id>/` unless `--diagnostics-dir`
|
|
overrides the diagnostics work directory for that invocation.
|
|
|
|
Set `workspace.diagnostics.enabled: false` or
|
|
`NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED=false` to skip diagnostics directory
|
|
creation and diagnostics artifact writes. Concise failures are still printed to
|
|
stderr.
|
|
|
|
Implemented diagnostics artifacts:
|
|
|
|
- `invocation.json`: command metadata such as operation, config path, input
|
|
path, selected lanes, run ID, and pipeline digest when available.
|
|
- `effective-config.json`: resolved config without raw API keys.
|
|
- `resolved-pipeline.json`: resolved module bindings and pipeline digest.
|
|
- `resolved-references.json`: resolved reference provenance, including target
|
|
stage, lane ID when present, origin, digest, media type, byte size, and
|
|
binding source, without reference content.
|
|
- `checkpoint-events.json`: checkpoint steps that were reused or executed
|
|
during an explicit resume invocation.
|
|
- `run-manifest.json`: the same run manifest written to durable output when it
|
|
is available, including top-level module metadata when present.
|
|
- `warnings.json`: warning list.
|
|
- `run-report.json`: counts, status, output path, diagnostics path, and run ID.
|
|
- `error.log`: failure message, written after diagnostics directory creation
|
|
when a run fails.
|
|
|
|
`source-document.json` is supported by the diagnostics writer but is not written
|
|
by the current CLI run workflow.
|
|
|
|
## Checkpoints
|
|
|
|
When `workspace.resume.enabled: true` and `workspace.directory` is set, runs
|
|
write checkpoints under:
|
|
|
|
```text
|
|
<workspace.directory>/checkpoints/<pipeline-id>/<input-key>-<source-or-input-digest>/<pipeline-digest>/<identity-digest>/
|
|
```
|
|
|
|
Each workflow step owns its own manifest and payload files. There is no
|
|
root-level checkpoint summary. Ordinary `notarius run` invocations execute the
|
|
pipeline normally and refresh checkpoints. `notarius run --resume` reuses valid
|
|
checkpoints and executes any missing, invalid, or incompatible step normally.
|
|
|
|
Checkpoint payloads preserve byte content with base64 envelopes, media type,
|
|
metadata, warnings, and content digests where applicable. Checkpoints do not
|
|
include raw prompts, raw reference contents, raw LLM request payloads, or debug
|
|
traces. They can still contain source text, intermediate extracted content,
|
|
rejected outputs, metadata, and warnings. Treat checkpoint directories as
|
|
sensitive local state.
|
|
|
|
A checkpoint is reused only when its workspace schema version, checkpoint
|
|
identity digest, step status, dependency fingerprints, payload files, and
|
|
payload digests match the current invocation. Changes to input bytes, resolved
|
|
pipeline digest, selected lanes, runtime LLM profile override, materialized
|
|
reference digests, or other identity material invalidate reuse and use a
|
|
separate checkpoint directory.
|
|
|
|
Plain `notarius run` does not reuse checkpoints. It executes the workflow and
|
|
refreshes checkpoint files when checkpointing is enabled. `notarius run
|
|
--resume` is the explicit reuse path.
|
|
|
|
## Debug
|
|
|
|
When `workspace.debug.enabled: true` and `workspace.directory` is set, runs
|
|
write debug artifacts under:
|
|
|
|
```text
|
|
<workspace.directory>/debug/<run-id>/
|
|
```
|
|
|
|
Debug output is per invocation. It is independent of checkpointing and is not
|
|
used for resume. Enabling debug does not write checkpoints, and enabling resume
|
|
checkpointing does not write debug output.
|
|
|
|
Debug artifacts include framework-boundary inputs and outputs for source,
|
|
chunk, extract, merge, normalize, and output work, structured LLM request and
|
|
response data from Notarius contracts, validator requests and results, timing,
|
|
and retry attempt metadata. Canonical LLM call artifacts are written under
|
|
`llm/call-000N.json`; LLM calls made inside a retry or validator attempt are
|
|
also copied under that attempt directory and linked from the attempt
|
|
`llm_calls` array. Debug artifacts may contain source material, reference
|
|
material, prompt inputs, model outputs, and other sensitive data. Obvious
|
|
credential-shaped values and sensitive map keys are redacted, but debug
|
|
directories should still be protected as sensitive local state.
|
|
|
|
## Retention
|
|
|
|
Diagnostics retention is configured with `workspace.diagnostics.retention`,
|
|
`NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`, legacy `diagnostics.retention`,
|
|
legacy `NOTARIUS_DIAGNOSTICS_RETENTION`, or the default `auto`.
|
|
|
|
- `auto`: keep failed runs and successful runs with warnings; remove successful
|
|
warning-free runs.
|
|
- `always`: keep every diagnostics run directory.
|
|
- `never`: remove successful run directories; failed runs are still retained.
|
|
|
|
Unknown retention values are rejected during config validation.
|
|
|
|
## Failures
|
|
|
|
Failures before diagnostics directory creation, such as a missing config file or
|
|
an unusable diagnostics work directory, are printed to stderr and may not have a
|
|
diagnostics run directory.
|
|
|
|
Failures after diagnostics directory creation are printed to stderr and written
|
|
to `error.log`. Depending on where the failure occurred, diagnostics may also
|
|
include invocation metadata, redacted effective config, resolved pipeline data,
|
|
the run manifest, warnings, and a run report.
|
|
|
|
If durable output writing fails after the pipeline completes, diagnostics are
|
|
retained for inspection and may include `run-manifest.json`, `warnings.json`,
|
|
`run-report.json`, and `error.log`.
|
|
|
|
## Warnings
|
|
|
|
A successful run with warnings exits with code `0`, prints a warning count to
|
|
stderr, and writes warnings to durable output and diagnostics when retained.
|
|
|
|
The run manifest `validation_status` indicates whether raw outputs were
|
|
approved or rejected after validation.
|
|
|
|
Reference-related warnings include empty bound reference files. Empty references
|
|
are still passed to extractors so optional slots can be intentionally blank.
|
|
|
|
## Cleanup
|
|
|
|
It is safe to remove specific old run directories after their output and
|
|
diagnostics are no longer needed:
|
|
|
|
```sh
|
|
rm -rf /tmp/notarius/run-1234567890
|
|
rm -rf ./notarius-output/run-1234567890
|
|
```
|
|
|
|
Workspace checkpoint and debug directories can also be removed when no longer
|
|
needed. Remove exact identity or run directories, for example:
|
|
|
|
```sh
|
|
rm -rf /var/lib/notarius/checkpoints/dnd-session/seriatim-abcdef123456/7890abcd1234/identityabcd1234
|
|
rm -rf /var/lib/notarius/debug/run-1234567890
|
|
```
|
|
|
|
Use exact run-directory paths. Avoid broad cleanup commands against parent
|
|
directories unless they are part of your own operational policy.
|
|
|
|
## Operational Limits
|
|
|
|
If `--resume` cannot reuse a checkpoint, Notarius executes that step and writes
|
|
a fresh checkpoint when checkpointing is enabled.
|
|
|
|
Provider retries and timeouts are handled by Scriptorium according to the
|
|
selected execution profile. Pipeline module retries are controlled by module
|
|
binding `retries` values in config for chunk, extract, merge, and normalize.
|
|
There is no separate CLI retry command.
|
|
|
|
Notarius writes local files only. Remote storage and archive management are not
|
|
part of the implemented CLI.
|