eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add operations and troubleshooting documentation

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-04 03:04:01 +00:00
parent 4dd00347e0
commit 2d75f6ad13
3 changed files with 350 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docs/cli.md
View File

@@ -51,6 +51,9 @@ On success, the command prints the completed pipeline ID, approved and rejected
artifact counts, and the output directory. If the run completes with warnings,
the warning count is printed to stderr.
For durable output, diagnostics, retention, and failure inspection, see
[Operations](operations.md).
The current `run` command requires the resolved pipeline to use exactly one
distinct LLM profile after defaults and overrides are applied.

132
docs/operations.md Normal file
View File

@@ -0,0 +1,132 @@
# Operations
This is the canonical reference for operating implemented Notarius runs.
## Normal Run
A run reads one source file, resolves one configured pipeline, calls the
configured OpenAI-compatible LLM profile, writes durable JSON output, and writes
diagnostics for inspection.
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-spells.config.yml \
--input examples/seriatim-minimal-transcript.json \
--output-dir ./notarius-output \
--diagnostics-dir /tmp/notarius
```
The command prints a success line with the pipeline ID, approved and rejected
artifact counts, and the output path.
## Output Directory
Durable output is written to:
```text
<output-root>/<run-id>/
```
The default output root is `./notarius-output`. Use `--output-dir` to choose a
different root.
The `json` output module writes these files:
- `index.json`: file index with paths to the manifest, artifact files,
rejected artifacts, and warnings.
- `manifest.json`: run manifest with resolved pipeline provenance, module keys,
validation status, and timing.
- `artifacts/<artifact-type>.json`: approved artifacts grouped by artifact
type. For the current D&D spell extractor, this includes
`artifacts/dnd.spell_cast.json` when spell-cast artifacts are approved.
- `rejected.json`: rejected candidates and validator decisions.
- `warnings.json`: warnings reported by pipeline modules or the output encoder.
Output writes are atomic per file. Logical output file names must be clean,
relative, slash-separated paths and must not contain `..`.
## Diagnostics Directory
Diagnostics are written under:
```text
<diagnostics-work-dir>/<run-id>/
```
The default diagnostics work directory is `/tmp/notarius`. It can be set with
`diagnostics.work_dir`, `NOTARIUS_WORK_DIR`, or `--diagnostics-dir`.
Implemented diagnostics artifacts:
- `invocation.json`: command metadata such as operation, config path, input
path, selected lanes, run ID, and pipeline digest when available.
- `effective-config.json`: resolved config with API keys redacted.
- `resolved-pipeline.json`: resolved module bindings and pipeline digest.
- `run-manifest.json`: the same run manifest written to durable output when it
is available.
- `warnings.json`: warning list.
- `run-report.json`: counts, status, output path, diagnostics path, and run ID.
- `error.log`: failure message, written after diagnostics directory creation
when a run fails.
`source-document.json` is supported by the diagnostics writer but is not written
by the current CLI run workflow.
## Retention
Diagnostics retention is configured with `diagnostics.retention`,
`NOTARIUS_DIAGNOSTICS_RETENTION`, or the default `auto`.
- `auto`: keep failed runs and successful runs with warnings; remove successful
warning-free runs.
- `always`: keep every diagnostics run directory.
- `never`: remove successful run directories; failed runs are still retained.
Unknown retention values are rejected during config validation.
## Failures
Failures before diagnostics directory creation, such as a missing config file or
an unusable diagnostics work directory, are printed to stderr and may not have a
diagnostics run directory.
Failures after diagnostics directory creation are printed to stderr and written
to `error.log`. Depending on where the failure occurred, diagnostics may also
include invocation metadata, redacted effective config, resolved pipeline data,
the run manifest, warnings, and a run report.
If durable output writing fails after the pipeline completes, diagnostics are
retained for inspection and may include `run-manifest.json`, `warnings.json`,
`run-report.json`, and `error.log`.
## Warnings
A successful run with warnings exits with code `0`, prints a warning count to
stderr, and writes warnings to durable output and diagnostics when retained.
The run manifest `validation_status` indicates whether final artifacts were
approved or rejected after validation.
## Cleanup
It is safe to remove specific old run directories after their output and
diagnostics are no longer needed:
```sh
rm -rf /tmp/notarius/run-1234567890
rm -rf ./notarius-output/run-1234567890
```
Use exact run-directory paths. Avoid broad cleanup commands against parent
directories unless they are part of your own operational policy.
## Operational Limits
There is no command to resume a failed run. Re-run `notarius run` after fixing
the cause.
Provider retries are limited to the OpenAI-compatible client retry behavior
configured by the effective LLM profile. There is no separate CLI retry command.
Notarius writes local files only. Remote storage and archive management are not
part of the implemented CLI.

215
docs/troubleshooting.md Normal file
View File

@@ -0,0 +1,215 @@
# Troubleshooting
This guide maps common implemented failure modes to inspection steps and fixes.
For command syntax, see [CLI Reference](cli.md). For YAML fields and
environment overrides, see [Configuration](config.md). For output and
diagnostics layout, see [Operations](operations.md).
## Config File Not Found
Symptom:
```text
notarius: config file not found; pass --config or set NOTARIUS_CONFIG
```
Fix:
- Pass `--config path/to/config.yml`.
- Or set `NOTARIUS_CONFIG` to a readable file.
- Or install a config at `/usr/local/etc/notarius/config.yml`.
If the message says the config path is a directory or is not available, correct
the path or file permissions.
## Unsupported Or Invalid Config
Symptoms include:
- `unsupported config version`
- `config version is required`
- `field <name> not found`
- `total LLM concurrency must be greater than zero`
- `diagnostics retention "<value>" is not supported`
Fix:
- Use `version: 1`.
- Remove unknown YAML fields.
- Validate with:
```sh
go run ./cmd/notarius config validate --config path/to/config.yml
```
## Unknown Pipeline
Symptom:
```text
notarius: pipeline "..." is not configured
```
Fix:
- List configured pipeline IDs:
```sh
go run ./cmd/notarius pipelines list --config path/to/config.yml
```
- Use one of those IDs in `notarius run <pipeline-id>`.
- Check indentation under the top-level `pipelines` map.
## Unknown Or Incompatible Module
Symptoms mention a module key, pipeline slot, lane, capability, or `not
registered`.
Fix:
- Validate the pipeline against the production module catalog:
```sh
go run ./cmd/notarius config validate \
--config path/to/config.yml \
--pipeline dnd-session
```
- Use only implemented production module keys listed in
[Configuration](config.md#implemented-production-modules).
- Check that artifact lanes include an `extract` binding.
## Invalid `--only`
Symptoms include:
- `--only must contain comma-separated non-empty artifact lane IDs`
- `--only requires --pipeline`
- `selected artifact lane`
Fix:
- Use comma-separated lane IDs with no empty entries:
```sh
go run ./cmd/notarius run dnd-session \
--config path/to/config.yml \
--input path/to/input.json \
--only spells
```
- For `config validate`, include `--pipeline` when using `--only`.
- Confirm the lane ID exists under `pipelines.<id>.artifacts`.
## Seriatim Input Validation Failure
Symptoms include `seriatim input`, `parse JSON`, `segments must not be empty`,
or validation errors naming a segment field.
Fix:
- Compare the input to
[examples/seriatim-minimal-transcript.json](../examples/seriatim-minimal-transcript.json).
- Ensure the JSON has a `metadata` object and a non-empty `segments` array.
- Each segment needs a non-empty `id`, non-empty `speaker`, non-empty `text`,
non-negative numeric `start`, and non-negative numeric `end`.
- Segment IDs must be unique and must not contain leading or trailing
whitespace.
- `end` must be greater than or equal to `start`.
## Missing LLM Base URL Or Model
Symptoms include:
- `LLM profile "default" base URL must not be empty`
- `LLM profile "default" model must not be empty`
- `base URL must be valid`
Fix:
- Set `base_url` and `model` in `llm_profiles.default`.
- Or set `NOTARIUS_LLM_DEFAULT_BASE_URL` and
`NOTARIUS_LLM_DEFAULT_MODEL`.
- If a profile needs authentication, set `api_key_env` in YAML or set
`NOTARIUS_LLM_DEFAULT_API_KEY`.
## LLM Profile Override Failure
Symptom:
```text
notarius: LLM profile override "..." is not configured
```
Fix:
- Add the profile under `llm_profiles`.
- Or use an existing profile ID with `--llm-profile`.
Current runs require exactly one distinct effective LLM profile. If a pipeline
uses several profiles, run with `--llm-profile <id>` or align the bindings in
configuration.
## Provider HTTP Or Response Failure
Symptoms include:
- `provider request failed`
- `provider returned status 400`
- `provider returned status 403`
- `provider response missing choices`
- `provider response assistant message content is not valid JSON`
- `decode structured output`
Fix:
- Confirm the `base_url` points to an OpenAI-compatible endpoint root. Notarius
posts to `<base_url>/chat/completions`.
- Check `model` and provider credentials.
- Inspect the retained diagnostics `error.log`.
- For 400 and 403 responses, fix the request configuration or credentials.
- For 429 and 5xx responses, the client retries according to `max_retries`; if
the failure persists, inspect the provider response and adjust capacity,
credentials, or model settings.
- The assistant message content must decode as JSON matching the extractor's
structured response schema.
Provider error messages are redacted for configured API key values.
## Output Write Failure
Symptoms include:
- `create output directory`
- `write output file`
- `output file name must`
Fix:
- Ensure `--output-dir` points to a directory path or a path that can be
created.
- Check filesystem permissions and available disk space.
- If diagnostics were retained, inspect `run-report.json`, `run-manifest.json`,
and `error.log`.
The CLI rejects unsafe logical output paths before writing files.
## Diagnostics Directory Surprise
Symptom: the diagnostics directory is missing after a successful run.
Fix:
- Check `diagnostics.retention`.
- With `auto`, successful runs without warnings are removed.
- Use `diagnostics.retention: always` when every diagnostics run directory
should be kept.
- Use `--diagnostics-dir` to override the configured work directory for a run.
Symptom: diagnostics exist even with `retention: never`.
Explanation:
- Failed runs are retained so that `error.log` and available context can be
inspected.