diff --git a/README.md b/README.md index 24a9e4e..d3b624d 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,29 @@ -# go-application-template +# Notarius +Notarius is a Go CLI for extracting structured artifacts from source material +with explicit, configurable pipeline modules. + +The current implementation reads Seriatim minimal transcript JSON, chunks the +source units, extracts D&D spell-cast artifacts with an OpenAI-compatible LLM, +and writes JSON output plus diagnostics for each run. + +```sh +NOTARIUS_LLM_DEFAULT_BASE_URL=http://127.0.0.1:8080/v1 \ +NOTARIUS_LLM_DEFAULT_MODEL=your-model \ +go run ./cmd/notarius run dnd-session \ + --config examples/dnd-spells.config.yml \ + --input examples/seriatim-minimal-transcript.json +``` + +If the provider requires authentication, set +`NOTARIUS_LLM_DEFAULT_API_KEY` in the environment before running the command. +Outputs are written under `./notarius-output//` unless `--output-dir` +is provided. + +Useful references: + +- [CLI reference](docs/cli.md) +- [Configuration reference](docs/config.md) +- [Seriatim input contract](docs/integrations/seriatim.md) +- [Maintained example config](examples/dnd-spells.config.yml) +- [Maintained example input](examples/seriatim-minimal-transcript.json) diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 0000000..54a27e9 --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,126 @@ +# CLI Reference + +This is the canonical reference for the implemented Notarius command-line +interface. + +## Quick Run + +```sh +NOTARIUS_LLM_DEFAULT_BASE_URL=http://127.0.0.1:8080/v1 \ +NOTARIUS_LLM_DEFAULT_MODEL=your-model \ +go run ./cmd/notarius run dnd-session \ + --config examples/dnd-spells.config.yml \ + --input examples/seriatim-minimal-transcript.json +``` + +Set `NOTARIUS_LLM_DEFAULT_API_KEY` if the OpenAI-compatible provider requires +a bearer token. + +## Commands + +```text +notarius help +notarius run --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] +notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b] +notarius pipelines list --config path/to/config.yml [--json] +``` + +Running `notarius` with no arguments, `notarius help`, `notarius --help`, or +`notarius -h` prints usage and exits successfully. + +## `run` + +`notarius run ` executes a configured pipeline against one input +file. + +Flags: + +- `--input path`: required source input file. +- `--config path`: config file path. If omitted, Notarius checks + `NOTARIUS_CONFIG`, then `/usr/local/etc/notarius/config.yml`. +- `--only lane-a,lane-b`: run only the named artifact lanes. Values are + comma-separated and must be non-empty. +- `--output-dir path`: output root. The run writes to `//`. + Defaults to `./notarius-output`. +- `--diagnostics-dir path`: diagnostics work directory override for this + invocation. +- `--llm-profile id`: override every effective module binding to use one LLM + profile. + +On success, the command prints the completed pipeline ID, approved and rejected +artifact counts, and the output directory. If the run completes with warnings, +the warning count is printed to stderr. + +The current `run` command requires the resolved pipeline to use exactly one +distinct LLM profile after defaults and overrides are applied. + +## `config validate` + +`notarius config validate` loads and validates configuration. + +Flags: + +- `--config path`: config file path. If omitted, discovery uses + `NOTARIUS_CONFIG`, then `/usr/local/etc/notarius/config.yml`. +- `--pipeline pipeline-id`: additionally resolve one configured pipeline against + the production module catalog. +- `--only lane-a,lane-b`: validate resolution for selected artifact lanes. This + flag requires `--pipeline`. + +Examples: + +```sh +go run ./cmd/notarius config validate \ + --config examples/dnd-spells.config.yml + +go run ./cmd/notarius config validate \ + --config examples/dnd-spells.config.yml \ + --pipeline dnd-session \ + --only spells +``` + +## `pipelines list` + +`notarius pipelines list` prints configured pipeline IDs in sorted order. + +Flags: + +- `--config path`: config file path. If omitted, discovery uses + `NOTARIUS_CONFIG`, then `/usr/local/etc/notarius/config.yml`. +- `--json`: print `{"pipelines":[...]}` instead of one ID per line. + +Examples: + +```sh +go run ./cmd/notarius pipelines list \ + --config examples/dnd-spells.config.yml + +go run ./cmd/notarius pipelines list \ + --config examples/dnd-spells.config.yml \ + --json +``` + +## Exit Codes + +- `0`: command succeeded. +- `1`: command syntax was valid, but loading config, resolving modules, running + the pipeline, calling the provider, writing output, or writing diagnostics + failed. +- `2`: command syntax was invalid, a command was unknown, a required argument + was missing, or a flag value was malformed. + +## Implemented Production Pipeline Modules + +The production CLI currently registers these module keys: + +- input: `seriatim` +- chunk: `generic` +- extract: `dnd/spells` +- merge: `appendorder` +- normalize: `noop` +- output: `json` + +The production CLI does not currently register validator modules. + +For YAML structure, defaults, environment overrides, and module binding syntax, +see [Configuration](config.md). diff --git a/docs/config.md b/docs/config.md new file mode 100644 index 0000000..7670b6d --- /dev/null +++ b/docs/config.md @@ -0,0 +1,213 @@ +# Configuration + +This is the canonical reference for implemented Notarius configuration. + +Notarius reads YAML config files with `version: 1`. File config is applied over +built-in defaults, then environment overrides are applied. + +## Discovery + +Commands that accept `--config` load configuration in this order: + +1. the `--config` path, when provided; +2. `NOTARIUS_CONFIG`, when set to a non-empty path; +3. `/usr/local/etc/notarius/config.yml`. + +If none is available, the command fails with a config file not found error. + +## Minimal Example + +```yaml +version: 1 +llm_profiles: + default: + provider: openai-compatible + base_url: http://127.0.0.1:8080/v1 + model: your-model +pipelines: + dnd-session: + input: seriatim + chunk: + module: generic + options: + max_units: 50 + artifacts: + spells: + extract: dnd/spells +``` + +The maintained fixture is [examples/dnd-spells.config.yml](../examples/dnd-spells.config.yml). + +## Top-Level Fields + +- `version`: required. The only supported value is `1`. +- `llm_profiles`: optional map of LLM profile IDs to profile settings. +- `pipelines`: optional map of pipeline IDs to pipeline definitions. +- `concurrency`: optional global concurrency settings. +- `diagnostics`: optional diagnostics settings. + +Unknown YAML fields are rejected. + +## Defaults + +Built-in defaults: + +```yaml +llm_profiles: + default: + provider: openai-compatible + timeout: 600 + max_retries: 3 + max_concurrency: 1 +concurrency: + total_llm: 1 +diagnostics: + work_dir: /tmp/notarius + retention: auto +``` + +No pipelines are built in. A run requires a configured pipeline. + +## LLM Profiles + +Each `llm_profiles` entry may contain: + +- `provider`: optional provider key. Empty means `openai-compatible`; any other + non-empty value must be `openai-compatible`. +- `base_url`: provider base URL. Required for actual LLM calls. +- `model`: provider model name. Required for actual LLM calls. +- `api_key_env`: environment variable name to read for the API key. +- `timeout`: request timeout as whole seconds or a Go-style duration string such + as `10m`. +- `max_retries`: retry count for provider calls. Must be zero or greater. +- `max_concurrency`: per-profile LLM concurrency. Must be zero or greater; when + zero, Notarius uses `concurrency.total_llm`. + +Raw API keys are not accepted as file config fields. Use `api_key_env` or an +environment override. + +## Environment Overrides + +These environment variables are applied after the config file: + +- `NOTARIUS_CONFIG`: config discovery path. +- `NOTARIUS_LLM_DEFAULT_API_KEY`: API key for the `default` LLM profile. +- `NOTARIUS_LLM_DEFAULT_BASE_URL`: base URL for the `default` LLM profile. +- `NOTARIUS_LLM_DEFAULT_MODEL`: model for the `default` LLM profile. +- `NOTARIUS_LLM_DEFAULT_TIMEOUT_SECONDS`: integer timeout seconds for the + `default` LLM profile. +- `NOTARIUS_LLM_DEFAULT_MAX_RETRIES`: integer retry count for the `default` LLM + profile. +- `NOTARIUS_LLM_DEFAULT_MAX_CONCURRENCY`: integer max concurrency for the + `default` LLM profile. +- `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency. +- `NOTARIUS_WORK_DIR`: diagnostics work directory. +- `NOTARIUS_DIAGNOSTICS_RETENTION`: diagnostics retention mode. + +Integer environment values must parse as base-10 integers. + +## Pipelines + +A pipeline defines the fixed Notarius workflow: + +```text +input -> chunk -> extract -> merge -> normalize -> output +``` + +Pipeline fields: + +- `input`: required module binding. +- `chunk`: optional module binding. Default module is `generic`. +- `artifacts`: required for pipeline resolution. It maps artifact lane IDs to + lane definitions. +- `output`: optional module binding. Default module is `json`. + +Artifact lane fields: + +- `extract`: required module binding. +- `merge`: optional module binding. Default module is `appendorder`. +- `normalize`: optional module binding. Default module is `noop`. +- `validators`: optional list of module bindings. The production CLI currently + does not register validator modules. + +`notarius run` and `notarius config validate --pipeline` resolve the pipeline +against the production module catalog and fail fast for unknown or incompatible +module keys. + +## Module Bindings + +Every module binding may use shorthand: + +```yaml +input: seriatim +``` + +or object form: + +```yaml +chunk: + module: generic + llm_profile: default + options: + max_units: 50 +``` + +Binding fields: + +- `module`: module key. +- `llm_profile`: optional LLM profile ID. Empty means `default`. +- `options`: optional module-specific settings. + +The `--llm-profile` run flag overrides every effective module binding to use +one configured profile. + +## Implemented Production Modules + +| Slot | Key | Notes | +| --- | --- | --- | +| input | `seriatim` | Reads Seriatim minimal transcript JSON. | +| chunk | `generic` | Splits source units into ordered chunks. | +| extract | `dnd/spells` | Extracts `dnd.spell_cast` artifacts. | +| merge | `appendorder` | Keeps candidates in append order. | +| normalize | `noop` | Passes merged artifacts through unchanged. | +| output | `json` | Produces JSON output files. | + +The `generic` chunker accepts: + +- `max_units`: positive integer, default `50`; +- `overlap_units`: non-negative integer, default `0`, and must be less than + `max_units`. + +## Diagnostics + +`diagnostics` fields: + +- `work_dir`: directory for per-run diagnostics. Default: `/tmp/notarius`. +- `retention`: `auto`, `always`, or `never`. Empty uses `auto`. + +`auto` retains diagnostics for failed runs and successful runs with warnings. +`always` retains diagnostics for every run. `never` removes diagnostics for +successful runs without regard to warnings; failed runs are retained. + +The `--diagnostics-dir` run flag overrides `diagnostics.work_dir` for that +invocation. + +## Validation + +Configuration validation checks: + +- supported config version and known YAML fields; +- non-empty, non-duplicated IDs after trimming; +- supported LLM provider and non-negative profile limits; +- positive global LLM concurrency; +- supported diagnostics retention and non-empty work directory; +- module binding LLM profiles refer to configured profiles. + +Pipeline resolution additionally checks: + +- the pipeline ID exists; +- at least one artifact lane is declared and selected; +- selected lanes exist when `--only` is used; +- required module keys are present; +- module keys are registered for the expected slot; +- module capability requirements are satisfied.