From 2d75f6ad1336d60462c016531e79ae1afd3c1e29 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 4 Jul 2026 03:04:01 +0000 Subject: [PATCH] Add operations and troubleshooting documentation --- docs/cli.md | 3 + docs/operations.md | 132 ++++++++++++++++++++++++ docs/troubleshooting.md | 215 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 350 insertions(+) create mode 100644 docs/operations.md create mode 100644 docs/troubleshooting.md diff --git a/docs/cli.md b/docs/cli.md index 54a27e9..38b5643 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -51,6 +51,9 @@ On success, the command prints the completed pipeline ID, approved and rejected artifact counts, and the output directory. If the run completes with warnings, the warning count is printed to stderr. +For durable output, diagnostics, retention, and failure inspection, see +[Operations](operations.md). + The current `run` command requires the resolved pipeline to use exactly one distinct LLM profile after defaults and overrides are applied. diff --git a/docs/operations.md b/docs/operations.md new file mode 100644 index 0000000..130c3bb --- /dev/null +++ b/docs/operations.md @@ -0,0 +1,132 @@ +# Operations + +This is the canonical reference for operating implemented Notarius runs. + +## Normal Run + +A run reads one source file, resolves one configured pipeline, calls the +configured OpenAI-compatible LLM profile, writes durable JSON output, and writes +diagnostics for inspection. + +```sh +go run ./cmd/notarius run dnd-session \ + --config examples/dnd-spells.config.yml \ + --input examples/seriatim-minimal-transcript.json \ + --output-dir ./notarius-output \ + --diagnostics-dir /tmp/notarius +``` + +The command prints a success line with the pipeline ID, approved and rejected +artifact counts, and the output path. + +## Output Directory + +Durable output is written to: + +```text +// +``` + +The default output root is `./notarius-output`. Use `--output-dir` to choose a +different root. + +The `json` output module writes these files: + +- `index.json`: file index with paths to the manifest, artifact files, + rejected artifacts, and warnings. +- `manifest.json`: run manifest with resolved pipeline provenance, module keys, + validation status, and timing. +- `artifacts/.json`: approved artifacts grouped by artifact + type. For the current D&D spell extractor, this includes + `artifacts/dnd.spell_cast.json` when spell-cast artifacts are approved. +- `rejected.json`: rejected candidates and validator decisions. +- `warnings.json`: warnings reported by pipeline modules or the output encoder. + +Output writes are atomic per file. Logical output file names must be clean, +relative, slash-separated paths and must not contain `..`. + +## Diagnostics Directory + +Diagnostics are written under: + +```text +// +``` + +The default diagnostics work directory is `/tmp/notarius`. It can be set with +`diagnostics.work_dir`, `NOTARIUS_WORK_DIR`, or `--diagnostics-dir`. + +Implemented diagnostics artifacts: + +- `invocation.json`: command metadata such as operation, config path, input + path, selected lanes, run ID, and pipeline digest when available. +- `effective-config.json`: resolved config with API keys redacted. +- `resolved-pipeline.json`: resolved module bindings and pipeline digest. +- `run-manifest.json`: the same run manifest written to durable output when it + is available. +- `warnings.json`: warning list. +- `run-report.json`: counts, status, output path, diagnostics path, and run ID. +- `error.log`: failure message, written after diagnostics directory creation + when a run fails. + +`source-document.json` is supported by the diagnostics writer but is not written +by the current CLI run workflow. + +## Retention + +Diagnostics retention is configured with `diagnostics.retention`, +`NOTARIUS_DIAGNOSTICS_RETENTION`, or the default `auto`. + +- `auto`: keep failed runs and successful runs with warnings; remove successful + warning-free runs. +- `always`: keep every diagnostics run directory. +- `never`: remove successful run directories; failed runs are still retained. + +Unknown retention values are rejected during config validation. + +## Failures + +Failures before diagnostics directory creation, such as a missing config file or +an unusable diagnostics work directory, are printed to stderr and may not have a +diagnostics run directory. + +Failures after diagnostics directory creation are printed to stderr and written +to `error.log`. Depending on where the failure occurred, diagnostics may also +include invocation metadata, redacted effective config, resolved pipeline data, +the run manifest, warnings, and a run report. + +If durable output writing fails after the pipeline completes, diagnostics are +retained for inspection and may include `run-manifest.json`, `warnings.json`, +`run-report.json`, and `error.log`. + +## Warnings + +A successful run with warnings exits with code `0`, prints a warning count to +stderr, and writes warnings to durable output and diagnostics when retained. + +The run manifest `validation_status` indicates whether final artifacts were +approved or rejected after validation. + +## Cleanup + +It is safe to remove specific old run directories after their output and +diagnostics are no longer needed: + +```sh +rm -rf /tmp/notarius/run-1234567890 +rm -rf ./notarius-output/run-1234567890 +``` + +Use exact run-directory paths. Avoid broad cleanup commands against parent +directories unless they are part of your own operational policy. + +## Operational Limits + +There is no command to resume a failed run. Re-run `notarius run` after fixing +the cause. + +Provider retries are limited to the OpenAI-compatible client retry behavior +configured by the effective LLM profile. There is no separate CLI retry command. + +Notarius writes local files only. Remote storage and archive management are not +part of the implemented CLI. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 0000000..0d5e2c9 --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,215 @@ +# Troubleshooting + +This guide maps common implemented failure modes to inspection steps and fixes. +For command syntax, see [CLI Reference](cli.md). For YAML fields and +environment overrides, see [Configuration](config.md). For output and +diagnostics layout, see [Operations](operations.md). + +## Config File Not Found + +Symptom: + +```text +notarius: config file not found; pass --config or set NOTARIUS_CONFIG +``` + +Fix: + +- Pass `--config path/to/config.yml`. +- Or set `NOTARIUS_CONFIG` to a readable file. +- Or install a config at `/usr/local/etc/notarius/config.yml`. + +If the message says the config path is a directory or is not available, correct +the path or file permissions. + +## Unsupported Or Invalid Config + +Symptoms include: + +- `unsupported config version` +- `config version is required` +- `field not found` +- `total LLM concurrency must be greater than zero` +- `diagnostics retention "" is not supported` + +Fix: + +- Use `version: 1`. +- Remove unknown YAML fields. +- Validate with: + +```sh +go run ./cmd/notarius config validate --config path/to/config.yml +``` + +## Unknown Pipeline + +Symptom: + +```text +notarius: pipeline "..." is not configured +``` + +Fix: + +- List configured pipeline IDs: + +```sh +go run ./cmd/notarius pipelines list --config path/to/config.yml +``` + +- Use one of those IDs in `notarius run `. +- Check indentation under the top-level `pipelines` map. + +## Unknown Or Incompatible Module + +Symptoms mention a module key, pipeline slot, lane, capability, or `not +registered`. + +Fix: + +- Validate the pipeline against the production module catalog: + +```sh +go run ./cmd/notarius config validate \ + --config path/to/config.yml \ + --pipeline dnd-session +``` + +- Use only implemented production module keys listed in + [Configuration](config.md#implemented-production-modules). +- Check that artifact lanes include an `extract` binding. + +## Invalid `--only` + +Symptoms include: + +- `--only must contain comma-separated non-empty artifact lane IDs` +- `--only requires --pipeline` +- `selected artifact lane` + +Fix: + +- Use comma-separated lane IDs with no empty entries: + +```sh +go run ./cmd/notarius run dnd-session \ + --config path/to/config.yml \ + --input path/to/input.json \ + --only spells +``` + +- For `config validate`, include `--pipeline` when using `--only`. +- Confirm the lane ID exists under `pipelines..artifacts`. + +## Seriatim Input Validation Failure + +Symptoms include `seriatim input`, `parse JSON`, `segments must not be empty`, +or validation errors naming a segment field. + +Fix: + +- Compare the input to + [examples/seriatim-minimal-transcript.json](../examples/seriatim-minimal-transcript.json). +- Ensure the JSON has a `metadata` object and a non-empty `segments` array. +- Each segment needs a non-empty `id`, non-empty `speaker`, non-empty `text`, + non-negative numeric `start`, and non-negative numeric `end`. +- Segment IDs must be unique and must not contain leading or trailing + whitespace. +- `end` must be greater than or equal to `start`. + +## Missing LLM Base URL Or Model + +Symptoms include: + +- `LLM profile "default" base URL must not be empty` +- `LLM profile "default" model must not be empty` +- `base URL must be valid` + +Fix: + +- Set `base_url` and `model` in `llm_profiles.default`. +- Or set `NOTARIUS_LLM_DEFAULT_BASE_URL` and + `NOTARIUS_LLM_DEFAULT_MODEL`. +- If a profile needs authentication, set `api_key_env` in YAML or set + `NOTARIUS_LLM_DEFAULT_API_KEY`. + +## LLM Profile Override Failure + +Symptom: + +```text +notarius: LLM profile override "..." is not configured +``` + +Fix: + +- Add the profile under `llm_profiles`. +- Or use an existing profile ID with `--llm-profile`. + +Current runs require exactly one distinct effective LLM profile. If a pipeline +uses several profiles, run with `--llm-profile ` or align the bindings in +configuration. + +## Provider HTTP Or Response Failure + +Symptoms include: + +- `provider request failed` +- `provider returned status 400` +- `provider returned status 403` +- `provider response missing choices` +- `provider response assistant message content is not valid JSON` +- `decode structured output` + +Fix: + +- Confirm the `base_url` points to an OpenAI-compatible endpoint root. Notarius + posts to `/chat/completions`. +- Check `model` and provider credentials. +- Inspect the retained diagnostics `error.log`. +- For 400 and 403 responses, fix the request configuration or credentials. +- For 429 and 5xx responses, the client retries according to `max_retries`; if + the failure persists, inspect the provider response and adjust capacity, + credentials, or model settings. +- The assistant message content must decode as JSON matching the extractor's + structured response schema. + +Provider error messages are redacted for configured API key values. + +## Output Write Failure + +Symptoms include: + +- `create output directory` +- `write output file` +- `output file name must` + +Fix: + +- Ensure `--output-dir` points to a directory path or a path that can be + created. +- Check filesystem permissions and available disk space. +- If diagnostics were retained, inspect `run-report.json`, `run-manifest.json`, + and `error.log`. + +The CLI rejects unsafe logical output paths before writing files. + +## Diagnostics Directory Surprise + +Symptom: the diagnostics directory is missing after a successful run. + +Fix: + +- Check `diagnostics.retention`. +- With `auto`, successful runs without warnings are removed. +- Use `diagnostics.retention: always` when every diagnostics run directory + should be kept. +- Use `--diagnostics-dir` to override the configured work directory for a run. + +Symptom: diagnostics exist even with `retention: never`. + +Explanation: + +- Failed runs are retained so that `error.log` and available context can be + inspected.