17 Commits

Author SHA1 Message Date
b85e826c1c Bugfix in the seriatim input adapter 2026-07-03 22:27:54 -05:00
4d0b2c69e6 Clean up completed documentation roadmaps 2026-07-04 03:13:27 +00:00
30e98a4d99 Add integration and artifact contract documentation 2026-07-04 03:11:29 +00:00
6ceabf40bb Add developer and internal architecture documentation 2026-07-04 03:07:56 +00:00
2d75f6ad13 Add operations and troubleshooting documentation 2026-07-04 03:04:01 +00:00
4dd00347e0 Add user documentation for Notarius CLI and config 2026-07-04 03:01:38 +00:00
5fab9936d0 Add a roadmap for a full documentation pass 2026-07-03 21:55:58 -05:00
b738dbc1eb Update policy documentation in advance of a full documentation pass 2026-07-03 21:46:27 -05:00
4db7805a95 Clean up MVP output handling 2026-07-03 21:26:05 -05:00
5424aae3de Mark MVP implementation complete in roadmap 2026-07-04 01:17:51 +00:00
6310e49fce Add fixture-driven MVP acceptance coverage 2026-07-04 01:15:59 +00:00
7de41eb3bd Write run outputs and diagnostics 2026-07-04 01:11:57 +00:00
ae218d7c57 Add in-memory pipeline run command 2026-07-04 01:06:04 +00:00
361b1f53f4 Wire production CLI catalog and scheduled LLM client 2026-07-04 00:59:32 +00:00
0ad96618fc Add production default pipeline modules 2026-07-04 00:55:18 +00:00
bc4203a264 Add run manifest and logical output file contracts 2026-07-04 00:45:21 +00:00
ac14667797 Move D&D spell assets into extractor module 2026-07-04 00:39:31 +00:00
77 changed files with 6524 additions and 2961 deletions

View File

@@ -1,2 +1,36 @@
# go-application-template
# Notarius
Notarius is a Go CLI for extracting structured artifacts from source material
with explicit, configurable pipeline modules.
The current implementation reads Seriatim transcript JSON, chunks the source
units, extracts D&D spell-cast artifacts with an OpenAI-compatible LLM, and
writes JSON output plus diagnostics for each run.
```sh
NOTARIUS_LLM_DEFAULT_BASE_URL=http://127.0.0.1:8080/v1 \
NOTARIUS_LLM_DEFAULT_MODEL=your-model \
go run ./cmd/notarius run dnd-session \
--config examples/dnd-spells.config.yml \
--input examples/seriatim-minimal-transcript.json
```
If the provider requires authentication, set
`NOTARIUS_LLM_DEFAULT_API_KEY` in the environment before running the command.
Outputs are written under `./notarius-output/<run-id>/` unless `--output-dir`
is provided.
Useful references:
- [CLI reference](docs/cli.md)
- [Configuration reference](docs/config.md)
- [Operations](docs/operations.md)
- [Troubleshooting](docs/troubleshooting.md)
- [Seriatim input contract](docs/integrations/seriatim.md)
- [OpenAI-compatible provider contract](docs/integrations/openai-compatible.md)
- [JSON output contract](docs/integrations/json-output.md)
- [D&D spell artifact contract](docs/integrations/dnd-spell-artifacts.md)
- [Developer workflow](docs/policy/development.md)
- [Internal architecture docs](docs/internal/overview.md)
- [Maintained example config](examples/dnd-spells.config.yml)
- [Maintained example input](examples/seriatim-minimal-transcript.json)

129
docs/cli.md Normal file
View File

@@ -0,0 +1,129 @@
# CLI Reference
This is the canonical reference for the implemented Notarius command-line
interface.
## Quick Run
```sh
NOTARIUS_LLM_DEFAULT_BASE_URL=http://127.0.0.1:8080/v1 \
NOTARIUS_LLM_DEFAULT_MODEL=your-model \
go run ./cmd/notarius run dnd-session \
--config examples/dnd-spells.config.yml \
--input examples/seriatim-minimal-transcript.json
```
Set `NOTARIUS_LLM_DEFAULT_API_KEY` if the OpenAI-compatible provider requires
a bearer token.
## Commands
```text
notarius help
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b]
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list --config path/to/config.yml [--json]
```
Running `notarius` with no arguments, `notarius help`, `notarius --help`, or
`notarius -h` prints usage and exits successfully.
## `run`
`notarius run <pipeline-id>` executes a configured pipeline against one input
file.
Flags:
- `--input path`: required source input file.
- `--config path`: config file path. If omitted, Notarius checks
`NOTARIUS_CONFIG`, then `/usr/local/etc/notarius/config.yml`.
- `--only lane-a,lane-b`: run only the named artifact lanes. Values are
comma-separated and must be non-empty.
- `--output-dir path`: output root. The run writes to `<path>/<run-id>/`.
Defaults to `./notarius-output`.
- `--diagnostics-dir path`: diagnostics work directory override for this
invocation.
- `--llm-profile id`: override every effective module binding to use one LLM
profile.
On success, the command prints the completed pipeline ID, approved and rejected
artifact counts, and the output directory. If the run completes with warnings,
the warning count is printed to stderr.
For durable output, diagnostics, retention, and failure inspection, see
[Operations](operations.md).
The current `run` command requires the resolved pipeline to use exactly one
distinct LLM profile after defaults and overrides are applied.
## `config validate`
`notarius config validate` loads and validates configuration.
Flags:
- `--config path`: config file path. If omitted, discovery uses
`NOTARIUS_CONFIG`, then `/usr/local/etc/notarius/config.yml`.
- `--pipeline pipeline-id`: additionally resolve one configured pipeline against
the production module catalog.
- `--only lane-a,lane-b`: validate resolution for selected artifact lanes. This
flag requires `--pipeline`.
Examples:
```sh
go run ./cmd/notarius config validate \
--config examples/dnd-spells.config.yml
go run ./cmd/notarius config validate \
--config examples/dnd-spells.config.yml \
--pipeline dnd-session \
--only spells
```
## `pipelines list`
`notarius pipelines list` prints configured pipeline IDs in sorted order.
Flags:
- `--config path`: config file path. If omitted, discovery uses
`NOTARIUS_CONFIG`, then `/usr/local/etc/notarius/config.yml`.
- `--json`: print `{"pipelines":[...]}` instead of one ID per line.
Examples:
```sh
go run ./cmd/notarius pipelines list \
--config examples/dnd-spells.config.yml
go run ./cmd/notarius pipelines list \
--config examples/dnd-spells.config.yml \
--json
```
## Exit Codes
- `0`: command succeeded.
- `1`: command syntax was valid, but loading config, resolving modules, running
the pipeline, calling the provider, writing output, or writing diagnostics
failed.
- `2`: command syntax was invalid, a command was unknown, a required argument
was missing, or a flag value was malformed.
## Implemented Production Pipeline Modules
The production CLI currently registers these module keys:
- input: `seriatim`
- chunk: `generic`
- extract: `dnd/spells`
- merge: `appendorder`
- normalize: `noop`
- output: `json`
The production CLI does not currently register validator modules.
For YAML structure, defaults, environment overrides, and module binding syntax,
see [Configuration](config.md).

213
docs/config.md Normal file
View File

@@ -0,0 +1,213 @@
# Configuration
This is the canonical reference for implemented Notarius configuration.
Notarius reads YAML config files with `version: 1`. File config is applied over
built-in defaults, then environment overrides are applied.
## Discovery
Commands that accept `--config` load configuration in this order:
1. the `--config` path, when provided;
2. `NOTARIUS_CONFIG`, when set to a non-empty path;
3. `/usr/local/etc/notarius/config.yml`.
If none is available, the command fails with a config file not found error.
## Minimal Example
```yaml
version: 1
llm_profiles:
default:
provider: openai-compatible
base_url: http://127.0.0.1:8080/v1
model: your-model
pipelines:
dnd-session:
input: seriatim
chunk:
module: generic
options:
max_units: 50
artifacts:
spells:
extract: dnd/spells
```
The maintained fixture is [examples/dnd-spells.config.yml](../examples/dnd-spells.config.yml).
## Top-Level Fields
- `version`: required. The only supported value is `1`.
- `llm_profiles`: optional map of LLM profile IDs to profile settings.
- `pipelines`: optional map of pipeline IDs to pipeline definitions.
- `concurrency`: optional global concurrency settings.
- `diagnostics`: optional diagnostics settings.
Unknown YAML fields are rejected.
## Defaults
Built-in defaults:
```yaml
llm_profiles:
default:
provider: openai-compatible
timeout: 600
max_retries: 3
max_concurrency: 1
concurrency:
total_llm: 1
diagnostics:
work_dir: /tmp/notarius
retention: auto
```
No pipelines are built in. A run requires a configured pipeline.
## LLM Profiles
Each `llm_profiles` entry may contain:
- `provider`: optional provider key. Empty means `openai-compatible`; any other
non-empty value must be `openai-compatible`.
- `base_url`: provider base URL. Required for actual LLM calls.
- `model`: provider model name. Required for actual LLM calls.
- `api_key_env`: environment variable name to read for the API key.
- `timeout`: request timeout as whole seconds or a Go-style duration string such
as `10m`.
- `max_retries`: retry count for provider calls. Must be zero or greater.
- `max_concurrency`: per-profile LLM concurrency. Must be zero or greater; when
zero, Notarius uses `concurrency.total_llm`.
Raw API keys are not accepted as file config fields. Use `api_key_env` or an
environment override.
## Environment Overrides
These environment variables are applied after the config file:
- `NOTARIUS_CONFIG`: config discovery path.
- `NOTARIUS_LLM_DEFAULT_API_KEY`: API key for the `default` LLM profile.
- `NOTARIUS_LLM_DEFAULT_BASE_URL`: base URL for the `default` LLM profile.
- `NOTARIUS_LLM_DEFAULT_MODEL`: model for the `default` LLM profile.
- `NOTARIUS_LLM_DEFAULT_TIMEOUT_SECONDS`: integer timeout seconds for the
`default` LLM profile.
- `NOTARIUS_LLM_DEFAULT_MAX_RETRIES`: integer retry count for the `default` LLM
profile.
- `NOTARIUS_LLM_DEFAULT_MAX_CONCURRENCY`: integer max concurrency for the
`default` LLM profile.
- `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency.
- `NOTARIUS_WORK_DIR`: diagnostics work directory.
- `NOTARIUS_DIAGNOSTICS_RETENTION`: diagnostics retention mode.
Integer environment values must parse as base-10 integers.
## Pipelines
A pipeline defines the fixed Notarius workflow:
```text
input -> chunk -> extract -> merge -> normalize -> output
```
Pipeline fields:
- `input`: required module binding.
- `chunk`: optional module binding. Default module is `generic`.
- `artifacts`: required for pipeline resolution. It maps artifact lane IDs to
lane definitions.
- `output`: optional module binding. Default module is `json`.
Artifact lane fields:
- `extract`: required module binding.
- `merge`: optional module binding. Default module is `appendorder`.
- `normalize`: optional module binding. Default module is `noop`.
- `validators`: optional list of module bindings. The production CLI currently
does not register validator modules.
`notarius run` and `notarius config validate --pipeline` resolve the pipeline
against the production module catalog and fail fast for unknown or incompatible
module keys.
## Module Bindings
Every module binding may use shorthand:
```yaml
input: seriatim
```
or object form:
```yaml
chunk:
module: generic
llm_profile: default
options:
max_units: 50
```
Binding fields:
- `module`: module key.
- `llm_profile`: optional LLM profile ID. Empty means `default`.
- `options`: optional module-specific settings.
The `--llm-profile` run flag overrides every effective module binding to use
one configured profile.
## Implemented Production Modules
| Slot | Key | Notes |
| --- | --- | --- |
| input | `seriatim` | Reads Seriatim transcript JSON. |
| chunk | `generic` | Splits source units into ordered chunks. |
| extract | `dnd/spells` | Extracts `dnd.spell_cast` artifacts. |
| merge | `appendorder` | Keeps candidates in append order. |
| normalize | `noop` | Passes merged artifacts through unchanged. |
| output | `json` | Produces JSON output files. |
The `generic` chunker accepts:
- `max_units`: positive integer, default `50`;
- `overlap_units`: non-negative integer, default `0`, and must be less than
`max_units`.
## Diagnostics
`diagnostics` fields:
- `work_dir`: directory for per-run diagnostics. Default: `/tmp/notarius`.
- `retention`: `auto`, `always`, or `never`. Empty uses `auto`.
`auto` retains diagnostics for failed runs and successful runs with warnings.
`always` retains diagnostics for every run. `never` removes diagnostics for
successful runs without regard to warnings; failed runs are retained.
The `--diagnostics-dir` run flag overrides `diagnostics.work_dir` for that
invocation.
## Validation
Configuration validation checks:
- supported config version and known YAML fields;
- non-empty, non-duplicated IDs after trimming;
- supported LLM provider and non-negative profile limits;
- positive global LLM concurrency;
- supported diagnostics retention and non-empty work directory;
- module binding LLM profiles refer to configured profiles.
Pipeline resolution additionally checks:
- the pipeline ID exists;
- at least one artifact lane is declared and selected;
- selected lanes exist when `--only` is used;
- required module keys are present;
- module keys are registered for the expected slot;
- module capability requirements are satisfied.

View File

@@ -0,0 +1,152 @@
# D&D Spell-Cast Artifacts
This document is the durable artifact contract for approved
`dnd.spell_cast` artifacts produced by the implemented `dnd/spells` extractor.
## Artifact Identity
- Extractor key: `dnd/spells`
- Artifact type: `dnd.spell_cast`
- Schema version: `v1`
- Prompt ID: `dnd.spells`
- Response schema key: `dnd_spells`
- Response schema ID: `notarius.dnd.spells`
- Response schema name: `notarius_dnd_spells_v1`
The extractor requires source chunks and transcript source capability. It
returns generic artifact candidates that are serialized by the JSON output
module.
## Artifact Envelope
Approved artifacts use the generic artifact envelope documented in
[JSON Output](json-output.md#artifact-files):
```json
{
"extractor_key": "dnd/spells",
"artifact_type": "dnd.spell_cast",
"schema_version": "v1",
"payload": {
"caster": "Aria",
"spell": "Cure Wounds",
"effect": "heals an injured ally",
"narrative_description": "Aria raises her holy symbol and casts Cure Wounds."
},
"source_refs": [
{
"source_id": "session-alpha",
"start_unit_id": "seg-001",
"end_unit_id": "seg-001"
}
]
}
```
## Payload Fields
The `payload` object contains:
- `caster`: in-world character or creature casting the spell;
- `spell`: spell name;
- `effect`: concise spell effect in the scene;
- `narrative_description`: short description of the spell cast in context.
All payload fields are strings and must be non-empty after trimming.
`caster` is the in-world caster, not the transcript speaker.
## Source References
Source references live on the artifact envelope as `source_refs`; they are not
duplicated inside the `payload`.
Each source reference uses the generic source-reference shape:
- `source_id`
- `start_unit_id`
- `end_unit_id`
Validation requires:
- at least one source reference;
- non-empty source ID and unit IDs;
- source ID matching the source document ID;
- start and end unit IDs existing in the source document;
- start unit appearing before or at the same position as end unit.
## Structured LLM Response Shape
The extractor asks the LLM for this top-level response shape:
```json
{
"spell_casts": [
{
"caster": "Aria",
"spell": "Cure Wounds",
"effect": "heals an injured ally",
"narrative_description": "Aria raises her holy symbol and casts Cure Wounds.",
"source_refs": [
{
"source_id": "session-alpha",
"start_unit_id": "seg-001",
"end_unit_id": "seg-001"
}
]
}
]
}
```
`spell_casts` must be present. It may be empty when no spell casts are found.
The response schema asset is embedded at
`internal/modules/extract/dnd/spells/assets/schemas/dnd_spells.v1.json`.
## Validators
The extractor supplies two deterministic validators by default:
- `dnd/spells/shape`
- `dnd/spells/source_refs`
Rejection reason codes:
- `invalid_payload`: payload JSON cannot be decoded as a spell-cast payload.
- `missing_required_field`: `caster`, `spell`, `effect`, or
`narrative_description` is blank.
- `missing_source_ref`: candidate has no source references.
- `invalid_source_ref`: at least one source reference fails generic source
reference validation.
Rejected candidates are written to `rejected.json` by the JSON output module.
## Manifest Metadata
The extractor adds prompt and response-schema provenance under the artifact lane
manifest metadata:
```json
{
"metadata": {
"extractor": {
"prompt_id": "dnd.spells",
"prompt_version": "v1",
"prompt_sha256": "sha256:...",
"response_schema_key": "dnd_spells",
"response_schema_id": "notarius.dnd.spells",
"response_schema_name": "notarius_dnd_spells_v1",
"response_schema_version": "v1",
"response_schema_sha256": "sha256:..."
}
}
}
```
Raw prompt and schema content are not included in manifest metadata.
## Compatibility Limit
This contract covers only `dnd.spell_cast` artifacts produced by the
implemented spell-cast extractor.

View File

@@ -1,116 +0,0 @@
# D&D Spell Cast Extraction
This document describes the D&D spell-cast extractor currently implemented in
Notarius.
## Module
- Module key: `dnd/spells`
- Artifact type: `dnd.spell_cast`
- Schema version: `v1`
- Prompt ID: `dnd.spells`
- Response schema key: `dnd_spells`
- Response schema ID: `notarius.dnd.spells`
- Response schema name: `notarius_dnd_spells_v1`
The module is an extract module. It reads a generic source chunk, renders the
`dnd.spells` prompt, calls the configured structured LLM client, and returns
spell-cast artifact candidates.
## Source Expectations
The extractor expects a generic `SourceDocument` and active source chunk. It
does not depend on concrete Seriatim package types.
Pipeline resolution must provide these capabilities before the extractor runs:
- `chunks`
- `source.transcript`
Source units may include transcript metadata such as speaker and timestamps.
That metadata is optional prompt context. It is not part of the durable spell
payload.
## Artifact Payload
Each approved artifact payload is a JSON object with these fields:
- `caster`: in-world character or creature casting the spell;
- `spell`: spell name;
- `effect`: concise spell effect in the scene;
- `narrative_description`: short description of the spell cast in context.
`caster` is not the table speaker. NPCs, monsters, and other DM-voiced
characters can be casters.
## Source References
The structured LLM response must include `source_refs` for each spell cast.
Each source reference uses the generic source-reference shape:
- `source_id`
- `start_unit_id`
- `end_unit_id`
The extractor copies those references into the generic artifact envelope
`source_refs` field. The durable `dnd.spell_cast` payload does not duplicate
source references.
Source-reference IDs must match the source document and source-unit IDs
exactly. The validator chain rejects unknown source IDs, unknown unit IDs, and
reversed unit ranges.
## Validators
The extractor provides these deterministic validators by default, in order:
- `dnd/spells/shape`
- `dnd/spells/source_refs`
`dnd/spells/shape` rejects:
- malformed JSON payloads with reason code `invalid_payload`;
- blank `caster`, `spell`, `effect`, or `narrative_description` fields with
reason code `missing_required_field`.
`dnd/spells/source_refs` rejects:
- candidates with no source references using reason code `missing_source_ref`;
- invalid source references using reason code `invalid_source_ref`.
The source-reference validator uses the core `source.ValidateRef` behavior, so
its rejection message includes the underlying source-reference validation
error.
## Capabilities
The module declares these required capabilities:
- `chunks`
- `source.transcript`
The module declares this provided capability:
- `dnd.spell_casts`
A pipeline artifact lane can reference the extractor with:
```yaml
artifacts:
spells:
extract: dnd/spells
merge: appendorder
normalize: noop
```
## Limits
The current implementation covers only D&D spell-cast extraction. It does not
yet implement:
- item extraction;
- NPC extraction;
- combat extraction;
- encounter extraction;
- broad D&D rules validation;
- a CLI `run` workflow.

View File

@@ -0,0 +1,192 @@
# JSON Output
This document is the durable JSON output file-format contract produced by the
implemented `json` output module and written by the CLI.
## Output Directory
The CLI writes logical output files under:
```text
<output-root>/<run-id>/
```
The default output root is `./notarius-output`. Operational behavior is covered
in [Operations](../operations.md).
## Files
The `json` output module writes:
- `index.json`
- `manifest.json`
- `artifacts/<artifact-type>.json`, one file per approved artifact type
- `rejected.json`
- `warnings.json`
Files are pretty-printed JSON with a trailing newline.
## `index.json`
Shape:
```json
{
"manifest_file": "manifest.json",
"artifact_files": [
{
"artifact_type": "dnd.spell_cast",
"file": "artifacts/dnd.spell_cast.json"
}
],
"rejected_file": "rejected.json",
"warnings_file": "warnings.json"
}
```
`artifact_files` is sorted by artifact type. It is empty when no artifacts are
approved.
## `manifest.json`
`manifest.json` contains a run manifest:
```json
{
"run_id": "run-123",
"pipeline_id": "dnd-session",
"pipeline_digest": "sha256:...",
"input_module": "seriatim",
"chunker": "generic",
"source_digests": ["sha256:..."],
"extractors": ["dnd/spells"],
"merger": "appendorder",
"normalizer": "noop",
"output_encoder": "json",
"artifact_lanes": [
{
"id": "spells",
"extractor": "dnd/spells",
"merger": "appendorder",
"normalizer": "noop"
}
],
"llm_profiles": [
{
"id": "default",
"provider": "openai-compatible",
"model": "configured-model"
}
],
"validation_status": "approved",
"started_at": "2026-01-01T00:00:00Z",
"completed_at": "2026-01-01T00:00:01Z"
}
```
Fields with empty values may be omitted by JSON encoding.
`validation_status` is `approved` when no candidates were rejected and
`rejected` when one or more candidates were rejected.
## Artifact Files
Each artifact file has this shape:
```json
{
"artifact_type": "dnd.spell_cast",
"artifacts": [
{
"extractor_key": "dnd/spells",
"artifact_type": "dnd.spell_cast",
"schema_version": "v1",
"payload": {},
"source_refs": [
{
"source_id": "session-alpha",
"start_unit_id": "seg-001",
"end_unit_id": "seg-001"
}
]
}
]
}
```
Artifact envelope fields:
- `extractor_key`: extractor module key.
- `artifact_type`: artifact type.
- `schema_version`: artifact schema version.
- `payload`: artifact-type-specific JSON payload.
- `source_refs`: optional generic source references.
- `metadata`: optional artifact metadata.
Artifact file names are produced by sanitizing the artifact type:
- characters outside `A-Z`, `a-z`, `0-9`, `.`, `_`, and `-` become `_`;
- repeated `..` sequences are replaced;
- leading and trailing `.`, `_`, and `-` are trimmed;
- empty sanitized names are rejected.
For current D&D spell-cast artifacts, the file is
`artifacts/dnd.spell_cast.json`.
## `rejected.json`
Shape:
```json
{
"rejected": [
{
"candidate": {
"index": 0,
"extractor_key": "dnd/spells",
"artifact_type": "dnd.spell_cast",
"schema_version": "v1",
"payload": {},
"source_refs": []
},
"validator_name": "dnd/spells/source_refs",
"reason_code": "missing_source_ref",
"message": "spell cast candidate must include at least one source ref"
}
]
}
```
`rejected` is an empty array when no candidates are rejected.
## `warnings.json`
Shape:
```json
{
"warnings": [
{
"scope": "output",
"reason_code": "example_warning",
"message": "warning message"
}
]
}
```
`warnings` is an empty array when no warnings are reported.
## Path Safety
The output module returns slash-separated logical paths. The CLI also validates
logical output names before writing:
- names must be non-empty;
- names must be relative;
- names must be clean;
- names must use `/`, not `\`;
- names must not contain `..`;
- resolved paths must stay under the run output directory.
Durable writes are atomic per file.

View File

@@ -0,0 +1,128 @@
# OpenAI-Compatible Structured Output
This document describes the external LLM provider contract implemented by the
production Notarius LLM client.
## Provider
- Provider key: `openai-compatible`
- HTTP method: `POST`
- Endpoint: `<base_url>/chat/completions`
- Request body: JSON
- Response mode: chat completions with structured JSON schema output
`base_url` is trimmed of trailing slashes before `/chat/completions` is
appended. Configure provider settings in [Configuration](../config.md).
## Request
The client sends a JSON object with:
```json
{
"model": "configured-model",
"messages": [
{
"role": "system",
"content": "..."
},
{
"role": "user",
"content": "..."
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "schema_name",
"strict": true,
"schema": {}
}
}
}
```
Implemented request behavior:
- `model` comes from the structured completion request when set, otherwise from
the configured LLM profile.
- `messages` must be non-empty; each role and content must be non-empty after
trimming.
- `response_format.type` is always `json_schema`.
- `response_format.json_schema.strict` is always `true`.
- `response_format.json_schema.name` and `schema` come from the extractor or
validator making the call.
If an API key is configured, the client sends:
```text
Authorization: Bearer <api-key>
```
The client always sends `Content-Type: application/json`.
## Response
The client expects a JSON response with at least one choice:
```json
{
"model": "provider-model",
"choices": [
{
"message": {
"content": "{\"field\":\"value\"}"
}
}
],
"usage": {
"prompt_tokens": 10,
"completion_tokens": 5,
"total_tokens": 15
}
}
```
`choices[0].message.content` may be either:
- a JSON string whose contents are valid JSON; or
- raw JSON.
The decoded content is unmarshaled into the caller-provided structured output
target. If `usage` is present, prompt, completion, and total token counts are
copied into the completion response.
## Errors And Retries
The client validates base URL, model, response schema name, response schema
JSON, messages, and output target before or during the call.
Retryable failures:
- HTTP request failure;
- response body read failure;
- HTTP `429`;
- HTTP `5xx`;
- malformed provider response envelope;
- missing choices;
- missing, empty, or invalid assistant JSON content;
- structured-output decode failure.
Non-retryable provider status codes include non-`429` `4xx` responses.
Provider error bodies are parsed for `error.message` or `message` when present.
Configured API key values and bearer-token values are redacted from returned
provider errors.
## Timeouts And Concurrency
The configured profile timeout is applied per provider request when greater
than zero. Context cancellation is respected.
The production CLI wraps the provider client with the LLM scheduler. Effective
concurrency is described in [LLM runtime internals](../internal/llm.md).
## Limits
This contract documents only the fields the implemented client sends and reads.
Provider-specific extensions are ignored unless they affect those fields.

View File

@@ -1,42 +1,50 @@
# Seriatim Minimal Transcript JSON
# Seriatim Transcript JSON
This document describes the Seriatim input format currently accepted by the
`seriatim` input adapter.
This document is the external input contract for the implemented `seriatim`
input adapter.
## Adapter
- Module key: `seriatim`
- Document kind: `transcript`
- Unit kind: `transcript_segment`
- Source format: `application/vnd.seriatim.minimal+json`
- Source format: `application/vnd.seriatim+json`
The adapter parses raw Seriatim JSON into a generic `SourceDocument`. It does
not add transcript-specific fields to core source or runner contracts.
The adapter parses raw Seriatim JSON into a generic source document. It owns
transcript-specific JSON parsing and metadata mapping; core source and pipeline
code stay source-format agnostic.
## Accepted Shape
The input must be a JSON object with top-level `metadata` and `segments` fields:
The input must be one JSON object with top-level `metadata` and `segments`
fields. This covers the maintained minimal fixture and Seriatim intermediate
output that provides the same required segment fields.
```json
{
"metadata": {
"id": "session-alpha",
"title": "Synthetic session transcript"
"title": "Synthetic D&D spell session"
},
"segments": [
{
"id": "seg-001",
"start": 0,
"end": 4.5,
"speaker": "Narrator",
"text": "The stone door opens."
"end": 4,
"speaker": "Aria",
"text": "Aria raises her holy symbol and casts Cure Wounds."
}
]
}
```
Extra compatible fields are ignored. Multiple top-level JSON values are
rejected.
The maintained example is
[examples/seriatim-minimal-transcript.json](../../examples/seriatim-minimal-transcript.json).
Top-level metadata entries are preserved. Other segment fields, such as
`categories`, are ignored.
Multiple top-level JSON values are rejected.
## Validation
@@ -44,8 +52,12 @@ The adapter rejects:
- empty raw input;
- malformed JSON;
- top-level JSON that is not an object;
- missing, null, or non-object `metadata`;
- missing, null, non-array, or empty `segments`;
- segment values that are not objects;
- segment `id` values that are neither strings nor numbers;
- non-string `speaker` or `text`;
- empty segment IDs;
- segment IDs with leading or trailing whitespace;
- duplicate segment IDs;
@@ -55,19 +67,16 @@ The adapter rejects:
- `end` values before `start`;
- missing or empty `text`.
Segment text may keep leading or trailing whitespace, but it must not be empty
after trimming.
Segment text is preserved as provided, but it must not be empty after trimming.
## Source Mapping
The adapter maps Seriatim input into the source model as follows:
The adapter maps input to `SourceDocument`:
- top-level `metadata` becomes `SourceDocument.Metadata`;
- `SourceDocument.Digest` is `sha256:<hex>` of the exact raw input bytes;
- `segment.id` becomes `SourceUnit.ID`;
- `segment.text` becomes `SourceUnit.Text`;
- each source unit has kind `transcript_segment`;
- segment `speaker`, `start`, and `end` are stored in source-unit metadata.
- `metadata` becomes `SourceDocument.Metadata`;
- `SourceDocument.Kind` is `transcript`;
- `SourceDocument.Format` is `application/vnd.seriatim+json`;
- `SourceDocument.Digest` is `sha256:<hex>` of the exact raw input bytes.
`SourceDocument.ID` is selected in this order:
@@ -76,6 +85,14 @@ The adapter maps Seriatim input into the source model as follows:
3. `metadata.source_id`, when it is a non-empty string after trimming;
4. `seriatim:<first-16-hex-chars-of-raw-sha256>`.
Each segment becomes one `SourceUnit`:
- `segment.id` becomes `SourceUnit.ID`; numeric IDs are converted to their JSON
number text, so `1` becomes `"1"`;
- `segment.text` becomes `SourceUnit.Text`;
- `SourceUnit.Kind` is `transcript_segment`;
- `speaker`, `start`, and `end` are stored in source-unit metadata.
## Metadata Keys
Seriatim unit metadata uses these keys:
@@ -84,18 +101,20 @@ Seriatim unit metadata uses these keys:
- `start`: `json.Number` start value;
- `end`: `json.Number` end value.
The `internal/modules/input/seriatim` package provides typed accessors for
these metadata values.
The `internal/modules/input/seriatim` package exposes typed accessors for these
values.
## Capabilities
The module declares these provided capabilities for pipeline validation:
The module declares these provided capabilities:
- `source.transcript`
- `transcript.speaker`
- `transcript.timestamps`
## Limits
## Compatibility Limit
Only the Seriatim minimal transcript shape described here is supported. Broader
Seriatim schema variants are not currently accepted as a compatibility contract.
This contract covers only Seriatim transcript JSON with the top-level
`metadata` object and `segments` array described here. Broader Seriatim output
schemas are compatible only when they provide these required fields with the
accepted types.

View File

@@ -0,0 +1,88 @@
# Diagnostics Internals
Diagnostics internals live in `internal/core/diagnostics`. Operator-facing run
behavior is documented in [Operations](../operations.md).
## Purpose
Diagnostics provide local inspection artifacts for a run without becoming the
durable output contract. Durable user output is produced by output modules and
written by the CLI.
Diagnostics must not expose secrets.
## Run Directory
`NewRunDirectory(workDir, retention)` creates:
```text
<workDir>/run-<unix-nanoseconds>/
```
If `workDir` is empty, it defaults to `/tmp/notarius`. Empty retention defaults
to `auto`.
The writer makes the work directory if needed, then attempts to create a unique
run directory. It retries run ID creation a bounded number of times if a
collision occurs.
## Artifact Writers
Implemented artifact names:
- `invocation.json`
- `effective-config.json`
- `resolved-pipeline.json`
- `source-document.json`
- `run-manifest.json`
- `run-report.json`
- `warnings.json`
- `error.log`
JSON artifacts are encoded with indentation and a trailing newline. Writes are
atomic through a temporary file in the target directory followed by rename.
Artifact names must be single relative file names. Absolute paths, path
separators, and names resolving outside the run directory are rejected.
## Redacted Effective Config
Diagnostics writers accept payloads that implement
`RedactedDiagnosticsPayload`. `internal/core/config` uses this to redact API
keys in effective config diagnostics while preserving resolved pipeline context.
The redaction path clones config data before replacing secret values.
## Retention
Retention is decided by `ShouldRetainRunDirectory`.
- Failed runs are always retained.
- `always` retains successful runs.
- `never` removes successful runs.
- `auto` retains successful runs only when warnings exist.
- Unknown retention values are treated as retain by the retention decision, but
config validation rejects unsupported values before normal runs.
`ApplyRetention` removes only the specific run directory.
## CLI Failure Behavior
The CLI creates the diagnostics run directory after config loading and before
pipeline resolution. Failures before that point do not have diagnostics.
After diagnostics creation, run failures call `WriteErrorLog` and apply
retention with `RunSucceeded: false`, so the run directory remains available.
When the pipeline returns a partial manifest on failure, the CLI writes that
manifest before logging the failure.
## Invariants
- Diagnostics paths must be narrow and run-directory scoped.
- Writes should be atomic where practical.
- Secrets must be redacted.
- Diagnostics write failures are command failures because they can hide the
information needed for recovery.
- Durable output file contracts belong to output modules and integration docs,
not to diagnostics.

116
docs/internal/llm.md Normal file
View File

@@ -0,0 +1,116 @@
# LLM Runtime
The implemented LLM runtime lives in `internal/framework/llm`. It provides
transport-neutral structured completion contracts, an OpenAI-compatible HTTP
adapter, concurrency scheduling, schema registry helpers, retry behavior, and
secret redaction.
## Contract
Modules depend on `contracts.StructuredLLMClient`:
```go
CompleteStructured(ctx, request, out) (response, error)
```
The request contains messages, optional model override, response schema name,
and response schema JSON. The caller supplies a pointer target for decoded
structured output.
Extractors own prompts and schemas. Provider adapters should not contain
domain-specific prompt logic.
## Production Client Construction
`internal/cli` builds the production LLM client from the effective config:
1. find the effective LLM profile;
2. build `OpenAICompatibleClientConfig`;
3. create an OpenAI-compatible client;
4. create a scheduler from profile or global concurrency;
5. wrap the client with `NewScheduledClient`;
6. return non-secret LLM profile manifest metadata.
The current run command requires exactly one distinct effective LLM profile for
the resolved pipeline.
## OpenAI-Compatible Adapter
`OpenAICompatibleClient` posts JSON to:
```text
<base_url>/chat/completions
```
It sends:
- `model`
- `messages`
- `response_format.type = "json_schema"`
- `response_format.json_schema.name`
- `response_format.json_schema.strict = true`
- `response_format.json_schema.schema`
If an API key is configured, the adapter sends an `Authorization: Bearer ...`
header.
The adapter accepts assistant content either as a JSON string containing JSON or
as raw JSON content. It then unmarshals that content into the caller-provided
target.
External wire-contract details belong in the
[OpenAI-compatible integration doc](../integrations/openai-compatible.md).
## Retries And Timeouts
The adapter retries:
- provider request failures;
- response read failures;
- HTTP `429`;
- HTTP `5xx`;
- malformed provider envelopes;
- malformed assistant JSON;
- structured-output decode failures.
Non-retryable `4xx` responses are returned without retry. Request timeout comes
from the effective LLM profile. Context cancellation is respected.
## Scheduler
`Scheduler` bounds concurrent provider calls. It tracks in-flight calls and a
FIFO queue of waiters. Cancellation removes queued waiters or releases granted
permits.
`NewScheduledClient` wraps any structured LLM client and runs each completion
inside the scheduler.
Effective concurrency is:
1. `llm_profiles.<id>.max_concurrency`, when greater than zero;
2. `concurrency.total_llm`, when greater than zero;
3. `1`.
## Schema Registry
The framework schema registry embeds generic test schemas. It also exposes
helpers for caller-owned schemas:
- `LoadResponseSchema`
- `LookupResponseSchema`
- `MustLookupResponseSchema`
- `ResponseSchema.DiagnosticsMap`
`DiagnosticsMap` omits raw schema content and includes metadata such as key,
ID, version, name, and SHA-256.
The D&D spell extractor owns and loads its own embedded response schema.
## Secret Redaction
Provider errors are passed through `ErrorWithSecretsRedacted` with the API key
and bearer-token value. Config diagnostics use redacted effective config
payloads.
Do not add raw provider request bodies, response bodies, API keys, or prompt
payloads to diagnostics by default.

165
docs/internal/modules.md Normal file
View File

@@ -0,0 +1,165 @@
# Modules
Production modules live under `internal/modules`. Each module implements one
contract from `internal/framework/contracts`, exposes a `ModuleSpec`, and
registers itself with the matching pipeline registry.
The CLI production catalog currently registers only the modules listed here.
## Contract Pattern
A production module package should provide:
- a stable module key;
- a constructor such as `New`;
- the relevant contract implementation;
- `ModuleSpec`;
- `Register`;
- focused tests for registration, options, contract behavior, and errors.
Module specs should describe capabilities accurately. Resolution uses specs to
reject incompatible pipelines before execution.
## `seriatim` Input
Package: `internal/modules/input/seriatim`
The `seriatim` adapter parses Seriatim transcript JSON into a generic source
document. It owns transcript JSON details, source ID selection, source digest
creation, transcript segment validation, and segment metadata mapping.
Provides:
- `source.transcript`
- `transcript.speaker`
- `transcript.timestamps`
External JSON shape belongs in the Seriatim integration doc.
## `generic` Chunker
Package: `internal/modules/chunk/generic`
The `generic` chunker splits source units into ordered chunks. It validates the
source document, clones source units, assigns chunk IDs such as `chunk-000001`,
and records chunk metadata for start unit, end unit, and unit count.
Options:
- `max_units`: positive integer, default `50`;
- `overlap_units`: non-negative integer, default `0`, and less than
`max_units`.
Provides:
- `chunks`
## `dnd/spells` Extractor
Package: `internal/modules/extract/dnd/spells`
The `dnd/spells` extractor owns D&D spell-cast artifact semantics. It renders
embedded prompts, loads the embedded structured response schema, calls the
structured LLM client, converts spell-cast responses into artifact candidates,
and supplies deterministic validators.
Requires:
- `chunks`
- `source.transcript`
Provides:
- `dnd.spell_casts`
Artifact type and schema version:
- artifact type: `dnd.spell_cast`
- schema version: `v1`
The extractor adds prompt and response-schema provenance to lane manifest
metadata. Durable artifact payload details belong in the
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
## D&D Spell Validators
The spell extractor returns two built-in validators:
- `dnd/spells/shape`: rejects malformed payloads and missing required fields.
- `dnd/spells/source_refs`: rejects candidates without valid source references.
Reason codes include:
- `invalid_payload`
- `missing_required_field`
- `missing_source_ref`
- `invalid_source_ref`
These validators are supplied by the extractor when no validators are configured
for the lane.
## `appendorder` Merger
Package: `internal/modules/merge/appendorder`
The `appendorder` merger clones and appends candidates in chunk order. It does
not deduplicate or reconcile candidates.
Provides:
- `merged`
## `noop` Normalizer
Package: `internal/modules/normalize/noop`
The `noop` normalizer clones merged candidates and returns them unchanged.
Requires:
- `merged`
Provides:
- `normalized`
## `json` Output
Package: `internal/modules/output/json`
The `json` output encoder converts approved artifacts, rejected artifacts,
warnings, and the run manifest into logical JSON output files. It groups
approved artifacts by artifact type and sanitizes artifact-type file names.
Requires:
- `normalized`
Provides:
- `encoded`
Durable output file shapes belong in the
[JSON output contract](../integrations/json-output.md). Operator behavior
belongs in [Operations](../operations.md).
## Production Registration
Production registration is centralized in `internal/cli/catalog.go`.
Do not make framework code import production modules. The CLI wires production
modules at the application boundary; tests may provide fake registries or fake
catalogs directly.
## Adding A Module
When adding a module, keep source-format and extraction-domain boundaries clear:
- input modules may know external source formats;
- extract modules may know artifact semantics and prompt/schema assets;
- merge and normalize modules own candidate combination and reconciliation;
- output modules own serialization, not diagnostics or CLI reporting.
Update [Development](../policy/development.md), [Configuration](../config.md),
internal docs, integration docs, and examples when the new module becomes
implemented production behavior.

86
docs/internal/overview.md Normal file
View File

@@ -0,0 +1,86 @@
# Internal Overview
This directory documents implemented Notarius internals for developers and LLM
coding agents. It complements [Architecture](../policy/architecture.md), which
is the durable policy for boundaries and invariants.
## Executable And CLI
`cmd/notarius` calls the CLI package. `internal/cli` owns:
- command parsing and usage;
- config discovery and loading;
- production module catalog and registry wiring;
- production LLM client construction;
- run directory creation;
- durable output writes;
- user-facing stdout, stderr, and exit codes.
The CLI should stay thin around framework contracts. Domain extraction behavior
belongs in modules, not in command handlers.
## Core Packages
- `internal/core/artifacts`: artifact candidates, approved artifacts, rejected
artifacts, validation decisions, and run manifests.
- `internal/core/config`: defaults, YAML config parsing, environment overrides,
validation, redaction, and resolved pipeline config.
- `internal/core/diagnostics`: per-run diagnostics directory creation,
diagnostics artifact writers, atomic writes, and retention decisions.
- `internal/core/source`: source documents, source units, source references, and
validation.
Core packages should remain deterministic and concrete. They should not import
production modules.
## Framework Packages
- `internal/framework/contracts`: interfaces and request/result structs for
input adapters, chunkers, extractors, mergers, normalizers, validators, output
encoders, and structured LLM clients.
- `internal/framework/pipeline`: module registries, module specs, profile
resolution, capability checks, run orchestration, warnings, validation, and
manifest population.
- `internal/framework/llm`: OpenAI-compatible structured-output client,
scheduler, schema registry, retries, and secret redaction.
- `internal/framework/prompt`: embedded prompt registry and template rendering.
- `internal/framework/validate`: validator decision helpers and cardinality
enforcement.
Framework code should stay source-agnostic and domain-agnostic.
## Module Packages
Production module packages live under `internal/modules`:
- `input/seriatim`
- `chunk/generic`
- `extract/dnd/spells`
- `merge/appendorder`
- `normalize/noop`
- `output/json`
Each module package owns its contract implementation, module spec,
registration, options, focused tests, and module-specific errors.
## Fixtures And Tests
The repository uses focused package tests plus a fixture-driven CLI workflow.
- CLI acceptance tests cover maintained examples under `examples/`.
- Pipeline tests cover registry composition and end-to-end framework behavior
with fakes.
- Module tests cover implemented module contracts without requiring real
provider calls.
- LLM tests use local test servers and fakes.
Do not use real external services in tests. Use fakes, fixtures, or local test
servers.
## Boundary Reminders
- Source-format details stay in input modules and integration docs.
- Extraction-domain details stay in extract modules and artifact docs.
- Provider wire details stay in the LLM runtime and provider integration docs.
- Durable output contracts belong in integration docs.
- Operator procedures belong in `docs/operations.md`, not internal docs.

127
docs/internal/pipeline.md Normal file
View File

@@ -0,0 +1,127 @@
# Pipeline Internals
The implemented pipeline runner lives in `internal/framework/pipeline`. It
executes the fixed workflow defined by the architecture policy:
```text
input -> chunk -> extract -> merge -> normalize -> output
```
Pipeline execution is serial. The runner executes the resolved lanes one after
another in the fixed workflow order.
## Profile Resolution
Config loading produces `pipeline.PipelineProfile` values. Resolution happens
before execution:
1. `internal/core/config.Config.Resolve` validates config and finds the named
pipeline.
2. The optional lane selection is passed to `pipeline.ResolvePipeline`.
3. Module bindings are defaulted:
- chunk: `generic`
- merge: `appendorder`
- normalize: `noop`
- output: `json`
- LLM profile: `default`
4. The module catalog is checked for each bound module key.
5. Module capabilities are checked in workflow order.
6. A digest is calculated from the resolved pipeline without the digest field.
The CLI writes the resolved pipeline and digest to diagnostics.
## Registries And Module Specs
`pipeline.Registries` holds concrete constructors for execution. A
`pipeline.ModuleCatalog` exposes module specs for config validation and
resolution.
Every production module registers a `ModuleSpec` with:
- `Key`: module key used in config;
- `Stage`: module kind such as input, chunk, extract, merge, normalize,
validate, or output;
- `Provides`: capabilities added after that module runs;
- `Requires`: capabilities that must already be available.
Capability checks prevent incompatible pipeline composition before a run starts.
## Runner Input And Output
`pipeline.RunInput` carries:
- a `ResolvedPipeline`;
- optional source ID, input path, and raw input bytes;
- a structured LLM client;
- run ID, start time, LLM profile manifest metadata, and CLI metadata.
`pipeline.RunOutput` carries:
- run manifest;
- approved artifacts;
- rejected artifacts;
- warnings;
- logical output files returned by the output encoder.
The CLI owns durable file writes and diagnostics writes after the runner returns.
## Execution
The runner:
1. validates run input and registries;
2. builds the input adapter and parses the raw input into a source document;
3. validates the source document;
4. builds the chunker and produces source chunks;
5. runs each selected artifact lane in sorted resolved order;
6. builds the output encoder and validates logical output file names.
Within an artifact lane, the runner:
1. builds the extractor, merger, and normalizer;
2. records module manifest metadata when modules provide it;
3. extracts candidates from each chunk;
4. normalizes candidate envelope fields such as index, extractor key, artifact
type, and schema version;
5. merges candidates;
6. normalizes merged candidates;
7. validates candidate envelope consistency;
8. runs validators;
9. converts approved candidates to artifacts.
## Validators
If a lane declares validators in config, the runner builds those validators from
the validator registry. Otherwise it uses validators returned by the extractor.
Each validator must return exactly one decision for each eligible candidate. The
runner enforces decision cardinality with `internal/framework/validate`.
Rejected candidates are removed before the next validator runs. Approved
candidates continue through the chain.
The production CLI currently registers no standalone validator modules. The
current D&D spell extractor supplies deterministic shape and source-reference
validators.
## Warnings And Failures
Warnings from chunking, extraction, merging, normalization, validation, and
output encoding are accumulated in `RunOutput.Warnings`.
Errors wrap the operation and module key or lane context. If execution fails
after a manifest exists, the returned manifest is marked `failed` and receives a
completion timestamp.
On successful execution, the manifest validation status is:
- `approved` when no candidates were rejected;
- `rejected` when at least one candidate was rejected.
## Manifest Population
The manifest records run ID, pipeline ID, pipeline digest, module keys, artifact
lanes, LLM profile metadata, source digest, validation status, and timing.
Modules can add non-secret manifest metadata by implementing
`contracts.ManifestMetadataProvider`. The D&D spell extractor uses this for
prompt and response-schema provenance.

132
docs/operations.md Normal file
View File

@@ -0,0 +1,132 @@
# Operations
This is the canonical reference for operating implemented Notarius runs.
## Normal Run
A run reads one source file, resolves one configured pipeline, calls the
configured OpenAI-compatible LLM profile, writes durable JSON output, and writes
diagnostics for inspection.
```sh
go run ./cmd/notarius run dnd-session \
--config examples/dnd-spells.config.yml \
--input examples/seriatim-minimal-transcript.json \
--output-dir ./notarius-output \
--diagnostics-dir /tmp/notarius
```
The command prints a success line with the pipeline ID, approved and rejected
artifact counts, and the output path.
## Output Directory
Durable output is written to:
```text
<output-root>/<run-id>/
```
The default output root is `./notarius-output`. Use `--output-dir` to choose a
different root.
The `json` output module writes these files:
- `index.json`: file index with paths to the manifest, artifact files,
rejected artifacts, and warnings.
- `manifest.json`: run manifest with resolved pipeline provenance, module keys,
validation status, and timing.
- `artifacts/<artifact-type>.json`: approved artifacts grouped by artifact
type. For the current D&D spell extractor, this includes
`artifacts/dnd.spell_cast.json` when spell-cast artifacts are approved.
- `rejected.json`: rejected candidates and validator decisions.
- `warnings.json`: warnings reported by pipeline modules or the output encoder.
Output writes are atomic per file. Logical output file names must be clean,
relative, slash-separated paths and must not contain `..`.
## Diagnostics Directory
Diagnostics are written under:
```text
<diagnostics-work-dir>/<run-id>/
```
The default diagnostics work directory is `/tmp/notarius`. It can be set with
`diagnostics.work_dir`, `NOTARIUS_WORK_DIR`, or `--diagnostics-dir`.
Implemented diagnostics artifacts:
- `invocation.json`: command metadata such as operation, config path, input
path, selected lanes, run ID, and pipeline digest when available.
- `effective-config.json`: resolved config with API keys redacted.
- `resolved-pipeline.json`: resolved module bindings and pipeline digest.
- `run-manifest.json`: the same run manifest written to durable output when it
is available.
- `warnings.json`: warning list.
- `run-report.json`: counts, status, output path, diagnostics path, and run ID.
- `error.log`: failure message, written after diagnostics directory creation
when a run fails.
`source-document.json` is supported by the diagnostics writer but is not written
by the current CLI run workflow.
## Retention
Diagnostics retention is configured with `diagnostics.retention`,
`NOTARIUS_DIAGNOSTICS_RETENTION`, or the default `auto`.
- `auto`: keep failed runs and successful runs with warnings; remove successful
warning-free runs.
- `always`: keep every diagnostics run directory.
- `never`: remove successful run directories; failed runs are still retained.
Unknown retention values are rejected during config validation.
## Failures
Failures before diagnostics directory creation, such as a missing config file or
an unusable diagnostics work directory, are printed to stderr and may not have a
diagnostics run directory.
Failures after diagnostics directory creation are printed to stderr and written
to `error.log`. Depending on where the failure occurred, diagnostics may also
include invocation metadata, redacted effective config, resolved pipeline data,
the run manifest, warnings, and a run report.
If durable output writing fails after the pipeline completes, diagnostics are
retained for inspection and may include `run-manifest.json`, `warnings.json`,
`run-report.json`, and `error.log`.
## Warnings
A successful run with warnings exits with code `0`, prints a warning count to
stderr, and writes warnings to durable output and diagnostics when retained.
The run manifest `validation_status` indicates whether final artifacts were
approved or rejected after validation.
## Cleanup
It is safe to remove specific old run directories after their output and
diagnostics are no longer needed:
```sh
rm -rf /tmp/notarius/run-1234567890
rm -rf ./notarius-output/run-1234567890
```
Use exact run-directory paths. Avoid broad cleanup commands against parent
directories unless they are part of your own operational policy.
## Operational Limits
There is no command to resume a failed run. Re-run `notarius run` after fixing
the cause.
Provider retries are limited to the OpenAI-compatible client retry behavior
configured by the effective LLM profile. There is no separate CLI retry command.
Notarius writes local files only. Remote storage and archive management are not
part of the implemented CLI.

View File

@@ -1,19 +1,22 @@
# Architecture
This document defines the development principles for Notarius. It is
inward-facing: developers and LLM coding agents should use it to preserve the
project's shape, boundaries, and invariants as the code evolves.
This document defines Notarius development policy. It is inward-facing:
developers and LLM coding agents should use it to preserve the project's shape,
boundaries, and invariants as the code evolves.
Keep this document concise. It should describe durable architectural rules, not
CLI syntax, configuration reference material, module catalogs, or roadmap items.
## Project Shape
Notarius is a small, explicit, dependency-light Go application for extracting
structured artifacts from source material using modular extractors.
structured artifacts from source material using modular pipeline stages.
The application should be contract-first but not abstraction-heavy. Add
interfaces and extension points when they protect a real boundary:
The application is contract-first but not abstraction-heavy. Add interfaces and
extension points when they protect a real boundary:
- external source formats;
- extractor modules;
- pipeline stage modules;
- validators;
- LLM providers and runtime plumbing;
- output schemas and embedded assets.
@@ -23,95 +26,43 @@ contracts that can be exercised by tests and real modules.
## Core Invariants
The core framework must remain source-agnostic and domain-agnostic.
The framework must remain source-agnostic and domain-agnostic.
Source-format details belong in input modules. Transcript-specific concepts
such as segments, speakers, timestamps, and transcript schemas must not spread
into runner, extractor, or validator framework code.
Source-format details belong in input modules. Transcript-specific concepts such
as segments, speakers, timestamps, and transcript schemas must not spread into
runner, extractor, validator, or LLM framework code.
Extraction-domain details belong in extract modules. D&D-specific concepts
such as spells, NPCs, items, combat turns, and encounters must not spread into
core source, runner, or LLM framework packages.
Extraction-domain details belong in domain modules. D&D-specific concepts such
as spells, NPCs, items, combat turns, and encounters must not spread into core
source, runner, or LLM framework packages.
Extracted facts should be grounded with source references. Source references
should point to generic source units, not to transcript-only structures.
Artifact records should require source references by default unless their schema
explicitly opts into ungrounded fields. Generic pipeline code should preserve
source-reference ranges exactly and should not merge or rewrite overlapping
ranges.
should point to generic source units, not transcript-only structures. Framework
code should preserve source-reference ranges exactly and should not merge or
rewrite overlapping ranges unless a module explicitly owns that behavior.
The application workflow is:
The application workflow is fixed:
```text
input -> chunk -> extract -> merge -> normalize -> output
```
These stages should remain explicit in the architecture. Chunking, merging, and
normalization must not be hidden inside domain extract modules when they represent
normalization must not be hidden inside domain extractors when they represent
general pipeline behavior.
## Dependency Policy
Pipelines are fixed-shape templates for this workflow, not arbitrary DAGs or a
general workflow language. Module selection should be configuration- and
registry-driven, not scattered through conditionals.
Prefer the Go standard library where practical.
## Package Boundaries
Use external dependencies only when justified by correctness, security,
interoperability, or substantial complexity reduction. Good reasons include
widely used file formats, complex validation behavior, or secure transport
handling.
Prefer fewer, larger framework packages until a boundary proves itself through
import direction, ownership, test seams, or substantial file size.
Avoid dependencies for small conveniences. Do not let external dependency types
leak across internal package boundaries unless the dependency is itself the
explicit public contract of that package.
## Package Layout
Use this layout unless a change documents a better project-specific reason.
CLI and executable entrypoint:
- `cmd/notarius`: executable entrypoint.
- `internal/cli`: CLI command definitions, flags, argument parsing, and command wiring.
Core deterministic model and policy:
- `internal/core/source`: source document, source unit, and source reference types.
- `internal/core/artifacts`: artifact envelope, artifact candidates, rejected artifacts, and manifests.
- `internal/core/config`: configuration structs, defaults, loading, precedence, and validation, once config exists.
Reusable framework plumbing:
- `internal/framework/contracts`: core interfaces and transport-neutral request/response contracts.
- `internal/framework/pipeline`: runner, pipeline-stage orchestration, registries, and small shared stage helpers.
- `internal/framework/validate`: shared validator runtime behavior and decision checks.
- `internal/framework/llm`: LLM clients, scheduling, structured-output parsing, and response-schema registry, once LLM runtime exists.
- `internal/framework/prompt`: embedded prompt assets, prompt registry, and prompt rendering helpers, once prompt assets exist.
Domain implementations:
- `internal/modules/input/<name>`: input-stage modules that parse external input into core source documents.
- `internal/modules/chunk/<name>`: chunk-stage modules.
- `internal/modules/extract/<domain>/<name>`: extract-stage extractor modules.
- `internal/modules/merge/<name>` or `internal/modules/merge/<domain>/<name>`: merge-stage modules.
- `internal/modules/normalize/<name>` or `internal/modules/normalize/<domain>/<name>`: normalize-stage modules.
- `internal/modules/output/<name>`: output-stage modules.
- `internal/validators/<validator>`: built-in validator implementations.
- `internal/transport/http`: shared HTTP client code, if needed by provider integrations.
Package-private implementation constants may live near the package that owns
them, preferably in `constants.go` when useful.
Start with fewer, larger framework packages. Split a package only when a real
boundary proves itself through import direction, ownership, test seams, or
substantial file size. Do not create catalog, diagnostics, reporting,
structured-output, response-schema, output, merge, normalize, extraction, or
warnings packages merely because the concepts exist in the architecture.
## Stage Modules
Concrete business logic should live under `internal/modules/<stage>/...`.
Stage-oriented module layout is preferred because it makes the application
workflow visible in the filesystem:
Core packages should contain deterministic models and policy. Framework
packages should contain reusable orchestration and provider plumbing. Concrete
business logic should live under stage-oriented module packages:
```text
internal/modules/input/...
@@ -122,120 +73,43 @@ internal/modules/normalize/...
internal/modules/output/...
```
Use short, lowercase, idiomatic Go package names. Prefer names such as
`dndtranscript`, `appendorder`, and `spells` over names like `dnd_transcript`,
`serial_merge`, or `spell_extractor` that repeat parent-stage context.
## Input Modules
Use a hexagonal architecture style for source input.
Use short, lowercase, idiomatic Go package names. Avoid package names that repeat
parent-stage context.
Input modules translate external source formats into the core source model.
They may know about external schema details, source-specific metadata, and
format-specific validation rules. They should not own extraction-domain
decisions.
Other packages should interact with source input through adapter contracts and
core source types. Input module implementation details and external dependency
types must not leak into framework or extract module packages.
Extract modules own artifact semantics, prompt usage, structured response schema
selection, validator defaults, and domain-specific interpretation. They should
depend on framework contracts and core source/artifact types, not concrete input
module packages.
Input module metadata may preserve source-specific facts such as transcript
speaker, timestamps, Markdown heading path, page number, or block ID. Framework
code may carry metadata through, but should not require a specific adapter's
metadata shape.
Merge modules combine extracted candidates. Normalize modules reconcile merged
candidates for semantic consistency. Generic behavior may exist for simple
artifact types, but domain-specific behavior belongs in modules for the relevant
stage.
Core source metadata should remain `map[string]any`. Document well-known keys
as conventions, and let input modules expose typed accessor helpers for their
own metadata when useful.
Output modules serialize final artifacts and may report warnings out of band.
CLI, diagnostics, and reporting layers are responsible for surfacing those
warnings.
## Extractors
Extractors are independent modules that process source chunks or whole source
documents and produce one kind of structured artifact candidate.
Each extract module owns:
- its artifact semantics;
- its prompt usage;
- its structured response schema selection;
- its validator chain;
- any domain-specific mapping or interpretation.
Extract modules should depend on framework contracts and core source/artifact
types. They should not depend on concrete input module packages.
Extraction requests should carry the active source chunk plus optional ambient
context, such as a document synopsis, prior-chunk summaries, known entities, or
other module-provided state. The context may be empty for simple modules, but
the contract should not assume extraction is always chunk-local.
Extractors should not be the only place where chunking, merging, or
normalization happens. They may choose processing mode or provide domain-specific
merge/normalization behavior when generic behavior is insufficient, but the
pipeline stages themselves are framework concepts.
The runner should be able to compose, skip, resume, or run individual extractors
when their prerequisites are satisfied. Ordering should be explicit through
configuration, a default sequence, or documented orchestration rules.
Extractor selection must go through a registry or equivalent mechanism rather
than scattered conditionals.
## Pipeline Stages
The pipeline has six conceptual stages:
1. input: external source material becomes a `SourceDocument`;
2. chunk: a `SourceDocument` becomes ordered source chunks;
3. extract: extractors produce artifact candidates from chunks or whole documents;
4. merge: per-chunk candidates become a merged candidate collection;
5. normalize: merged candidates are reconciled for duplicates, aliases, consistency, or cross-chunk issues;
6. output: final artifacts are serialized.
Chunking is first-class because source documents may be too large for a single
LLM pass. Chunkers should preserve source-unit order and produce stable chunk
metadata.
Merge and normalize are separate concerns. Merge combines per-chunk results into
a deterministic collection. Normalize performs semantic reconciliation after
merge. Generic append-in-chunk-order merge and no-op normalization should be
available for simple artifact types, while domain-specific behavior can be
provided where needed.
The framework should allow serial and parallel chunk processing. The first
implementation may execute chunks serially for determinism, but contracts should
not prevent later parallel execution.
Final durable output should use one artifact file per artifact type plus a
run-level manifest/index file. Framework artifact flow should use a generic
envelope with `json.RawMessage` payloads; extract modules should own typed Go
structs at their boundaries and encode into that generic envelope before
returning to framework code.
Output-stage warnings are intentionally out-of-band. An `OutputEncoder` may
return warnings about serialization, formatting, truncation, or destination
concerns, but those warnings are appended to the runner result after encoding
and are not expected to appear inside the encoded artifact bytes. CLI,
diagnostic, or reporting layers should surface output-stage warnings from the
runner result.
Schemas should be versioned per extractor, with a separate envelope/manifest
format version.
## Validators
## Validation
Validators should be independently testable and composable.
Deterministic validators should run before LLM-backed validators when both are
present. Validator decision semantics should be explicit: each candidate
artifact should receive exactly one decision from each validator that evaluates
it.
artifact evaluated by a validator should receive exactly one decision from that
validator.
LLM-backed review belongs in module-owned validator chains, not in a separate
global review phase. Extract modules and normalize modules may both use
deterministic and LLM-backed validators.
LLM-backed review belongs in module-owned validator chains, not in an implicit
global review phase. Extract and normalize modules may both use deterministic
and LLM-backed validators.
Shared validator runtime mechanics belong under `internal/framework/validate`.
Concrete validator behavior belongs under `internal/validators/<validator>`.
Shared validator runtime mechanics belong in framework code. Concrete validator
behavior belongs in module or validator implementation packages.
## LLM Runtime
@@ -243,131 +117,64 @@ LLM provider details belong behind transport-neutral framework contracts.
Provider-specific HTTP request and response types should stay inside the LLM
runtime package. Prompt construction should stay in extractors, validators, or
shared prompt-context helpers; provider adapters should not own domain prompt
logic.
shared prompt helpers; provider adapters should not own domain prompt logic.
Errors, diagnostics, reports, and redacted config must not expose secrets.
Errors, diagnostics, reports, manifests, and redacted configuration must not
expose secrets.
## Configuration
Centralize configuration loading, processing, precedence, defaults, and
validation in `internal/core/config`.
Configuration should make pipeline composition explicit and discoverable.
The goal is to make configuration discoverable and avoid implicit or hidden
operational values. User-visible defaults and cross-package operational defaults
should be defined in config code.
Centralize configuration loading, precedence, defaults, and validation. Structural
pipeline choices should come from named pipeline definitions, not ad hoc command
flags. Operational overrides may be handled separately when they do not obscure
the configured pipeline structure.
Configuration should be organized around named pipeline profiles. A pipeline is
a fixed-shape template for the application workflow, not a free-form DAG or
general workflow program. The six-stage flow remains fixed:
Module registries should expose module metadata and capabilities without
requiring module construction. Configuration validation should fail fast when a
pipeline binds incompatible or unknown modules.
```text
input -> chunk -> extract -> merge -> normalize -> output
```
Run manifests should record enough resolved pipeline provenance to make a run
auditable after named configuration changes over time.
A pipeline profile should bind registered modules to those stage slots:
## Dependencies
- one shared input module;
- one shared chunk module by default;
- one or more artifact lanes, each with extract, merge, normalize, and
validator behavior;
- one output module.
Prefer the Go standard library where practical.
The MVP should use one shared chunk module per pipeline. Per-lane chunk
overrides are a future extension and should be added only if a real artifact
lane needs different chunking.
Use external dependencies only when justified by correctness, security,
interoperability, or substantial complexity reduction. Good reasons include
widely used file formats, complex validation behavior, or secure transport
handling.
The CLI should select a named pipeline by ID, such as
`notarius run dnd-session --input session.json`. Structural module selection
should come from configuration, not ad hoc CLI flags. CLI flags may select a
subset of configured artifact lanes, such as `--only spells,npcs`, and may
override operational settings such as model, concurrency, output directory, or
diagnostics directory.
Pipeline definitions should support compact defaults:
- `chunk`: `generic`;
- lane `merge`: `appendorder`;
- lane `normalize`: `noop`;
- `output`: `json`;
- `llm_profile`: `default` where an LLM profile is needed.
Module bindings should support both string shorthand and object form. For
example, `extract: dnd/spells` and
`extract: {module: dnd/spells, llm_profile: fast}` should normalize to the same
internal binding type.
Module registries should expose module metadata, including flat string
capabilities, without requiring module construction. Config validation should
fail fast on unknown module keys, unknown pipeline IDs, missing required slots,
missing capabilities, unknown LLM profiles, empty artifact-lane sets, or
`--only` lane names that do not exist in the selected pipeline.
Keep capabilities as a flat string set. Do not evolve capabilities into a type
system unless real module interactions prove the need.
Unless documented otherwise, precedence from lowest to highest is:
1. built-in defaults
2. configuration file
3. environment variables
4. CLI flags
Prefer YAML configuration unless the project has a strong reason to use another
format. Config files should be discoverable at
`/usr/local/etc/notarius/config.yml`, with a CLI override via `--config`.
Configuration files should not contain raw secrets unless the application is
explicitly designed for that. Prefer environment variables or secret files for
secrets.
Stage-module-specific configuration should remain inline with the pipeline slot
that owns it. Do not add named module instances until repeated inline settings
create real drift or duplication. LLM profiles are the justified top-level
exception because model settings are cross-cutting.
The run manifest should record the selected `pipeline_id` and a digest of the
resolved pipeline definition after defaults and lane selection are applied.
`pipeline_id` alone is not sufficient provenance because a named pipeline can
change over time.
## Embedded Assets
Store embedded JSON schemas, Markdown prompts, templates, and similar assets as
separate files, not inline string literals, unless there is a strong reason
otherwise.
Embedded prompts and response schemas should have stable IDs, versions, source
metadata, and hashes suitable for diagnostics and run manifests.
## Errors and Logging
Errors should be actionable and preserve context. Wrap errors with operation and
path/resource context. CLI code should convert internal errors into concise
user-facing messages.
Errors and logs must not expose secrets.
Use structured logging where practical. Logs should describe operations, paths,
external calls, retries, and failure causes, but should not include large source
or artifact payloads by default.
## Context, Timeouts, and Cancellation
Long-running operations should accept `context.Context`. External calls,
subprocesses, HTTP requests, storage operations, LLM calls, and multi-stage
workflows should respect cancellation and timeouts.
Avoid dependencies for small conveniences. Do not let external dependency types
leak across internal package boundaries unless the dependency is itself the
explicit contract of that package.
## State, Files, and Safety
If the application writes durable state, writes should be atomic where
practical. Multi-step workflows should preserve enough state to support
inspection, retry, or resume after failure.
practical. Multi-step workflows should preserve enough diagnostics to support
inspection after failure.
Code that deletes, moves, or overwrites files must use narrow, explicit paths.
Avoid broad parent-directory operations. Cleanup that can cause data loss must
be opt-in.
## Errors and Logging
Errors should be actionable and preserve context. Wrap errors with operation and
path or resource context. CLI code should convert internal errors into concise
user-facing messages.
Errors and logs must not expose secrets. Logs should describe operations,
external calls, retries, and failure causes, but should not include large source
or artifact payloads by default.
Long-running operations should accept `context.Context`. External calls,
subprocesses, HTTP requests, storage operations, LLM calls, and multi-stage
workflows should respect cancellation and timeouts.
## Testing
Core logic should be testable without real external services. Use fakes,
@@ -375,18 +182,15 @@ fixtures, or local test doubles for input modules, extract modules, validators,
and LLM clients where practical.
Contract-first work should include fake implementations that prove interfaces
compose before real input modules or extract modules depend on them.
compose before real modules depend on them.
Once the pipeline-stage contracts exist, maintain a fixture-driven walking
skeleton that exercises input, chunk, extract, merge, normalize, and output
stages with fake modules and fake external clients. This test should protect
stage composition continuously while real modules are introduced over later
checkpoints.
Maintain a fixture-driven walking skeleton that exercises the full pipeline with
fake modules and fake external clients. This protects stage composition as real
modules evolve.
Config examples should be load-tested once config files exist. Important CLI
workflows should have parser or command tests. Adapter, extractor, and validator
contracts should have focused tests that do not require running the full
application unless end-to-end coverage is intentional.
Important CLI and configuration workflows should have tests. Adapter, extractor,
validator, and stage contracts should have focused tests that do not require
running the full application unless end-to-end coverage is intentional.
## Documentation

139
docs/policy/development.md Normal file
View File

@@ -0,0 +1,139 @@
# Development
This document defines contributor workflow for Notarius. For architectural
invariants and package boundaries, read [Architecture](architecture.md) first.
## Required Reading
Before changing the repository, review:
- [Architecture](architecture.md)
- [Documentation Policy](documentation.md)
Keep current-behavior documentation limited to implemented behavior. Put planned
or deferred behavior under `docs/roadmap/`.
## Repository Layout
- `cmd/notarius`: executable entry point.
- `internal/cli`: CLI parsing, production catalog wiring, config loading, run
command orchestration, output writes, and user-facing errors.
- `internal/core`: deterministic models and policy for artifacts, source
documents, config, and diagnostics.
- `internal/framework`: reusable contracts, pipeline orchestration, prompt
helpers, validation helpers, and LLM runtime plumbing.
- `internal/modules`: concrete input, chunk, extract, merge, normalize, and
output modules.
- `docs`: policy, user/operator docs, internal docs, integration docs, and
roadmap files.
- `examples`: maintained, secret-free examples covered by tests where practical.
## Validation Commands
Run focused tests for the area changed, then run the broader checks when the
change affects shared contracts, CLI behavior, or documentation examples.
```sh
go test ./...
go vet ./...
go build ./cmd/notarius
```
Useful focused checks:
```sh
go test ./internal/cli
go test ./internal/core/config
go test ./internal/framework/pipeline
go test ./internal/framework/llm
go test ./internal/modules/input/seriatim
go test ./internal/modules/extract/dnd/spells
go test ./internal/modules/output/json
```
## Go Conventions
- Prefer the standard library unless a dependency is justified by correctness,
security, interoperability, or substantial complexity reduction.
- Keep package names short, lowercase, and idiomatic.
- Preserve import direction: framework and core code must not depend on concrete
production modules.
- Use `context.Context` for long-running operations and external calls.
- Return contextual errors that name the operation and relevant module, path, or
resource.
- Do not include secrets in errors, logs, diagnostics, manifests, or docs.
## Adding Config Fields
Config behavior is centralized under `internal/core/config`.
When adding a file config field:
1. Update file config structs and YAML parsing in `file_config.go`.
2. Apply the field over defaults in config application code.
3. Add validation in `validation.go` when the field has constraints.
4. Add environment override support in `env.go` only for operational overrides.
5. Update redaction if the field can contain secrets.
6. Add focused config tests.
7. Update [Configuration](../config.md) and maintained examples when behavior
changes.
Pipeline composition should remain config-driven. Do not add command flags that
silently replace structural pipeline definitions.
## Adding CLI Flags Or Commands
CLI behavior lives in `internal/cli`.
When adding CLI surface:
1. Keep syntax explicit and update usage text.
2. Validate arguments before running expensive work.
3. Convert internal errors into concise user-facing messages.
4. Add CLI tests for success, syntax errors, and failure modes.
5. Update [CLI Reference](../cli.md), and update
[Operations](../operations.md) or [Troubleshooting](../troubleshooting.md)
if run behavior changes.
## Adding Modules Or Adapters
Concrete modules live under `internal/modules/<kind>/...` and implement the
interfaces in `internal/framework/contracts`.
For a new production module:
1. Implement the relevant contract.
2. Expose a `ModuleSpec` with the correct module key, module kind, provided
capabilities, and required capabilities.
3. Expose a `Register` function that registers the module with its registry.
4. Add focused module tests for contract behavior, registration, options,
validation, and errors.
5. Register the module in `internal/cli/catalog.go` only when it is production
ready.
6. Update internal docs and user-facing docs only for implemented behavior.
Source-format behavior belongs in input modules and integration docs.
Extraction-domain behavior belongs in extract modules and artifact docs.
## Updating Examples
Examples must be valid, secret-free, and small.
- Prefer environment-based secret configuration.
- Keep `examples/dnd-spells.config.yml` loadable by CLI tests.
- Keep `examples/seriatim-minimal-transcript.json` compatible with the Seriatim
adapter.
- Do not add expected-output fixtures unless they are validated or have a clear
regeneration procedure.
## Documentation Updates
Update docs in the same change when behavior changes.
- CLI syntax: `docs/cli.md`
- Config fields and defaults: `docs/config.md`
- Output, diagnostics, retention, or recovery: `docs/operations.md`
- Common user-facing failures: `docs/troubleshooting.md`
- Internal architecture and contracts: `docs/internal/`
- External file formats and durable integration contracts: `docs/integrations/`
- Future or planned work only: `docs/roadmap/`

View File

@@ -188,7 +188,9 @@ It should include:
- architectural invariants;
- explicit non-goals, if useful.
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
Notably, this file should prescribe a core development *policy* that should remain unchanged as the application evolves. It is not a place for details (e.g., CLI flags) that could change over time.
The contents of `architecture.md` should be trim and concise. LLMs may be directed to review it routinely via AGENTS.md, CLAUDE.md, or similar.
### docs/api.md

View File

@@ -1,199 +0,0 @@
# Documentation Roadmap
## Status
This document captures planned documentation decisions for Notarius. It records
policy choices while the application architecture is still being shaped. It does
not describe implemented behavior.
## Documentation Goals
Notarius documentation should make three boundaries obvious:
- source-format support belongs to input-stage modules;
- extraction-domain behavior belongs to extract-stage modules;
- core framework behavior is source-agnostic and domain-agnostic.
Documentation should avoid making the MVP look more transcript-specific or
D&D-specific than the architecture intends.
## Current Policy Decisions
### Planned Work Stays In Roadmap Docs
Until code exists, planned behavior belongs under `docs/roadmap/`.
Implemented behavior should later move into canonical docs. Roadmap files may
then link to those docs or be reduced to remaining future work.
### Core Docs Should Use Generic Terms
Core architecture docs should prefer:
- source document;
- source unit;
- source reference;
- input adapter;
- extractor;
- chunker;
- merger;
- normalizer;
- output encoder;
- artifact;
- validator;
- run manifest.
Core docs should avoid transcript-specific terms such as segment, speaker,
timestamp, and transcript range unless discussing an input adapter or an example.
Core docs should avoid D&D-specific terms such as spell, NPC, item, combat, and
encounter unless discussing extract modules, artifact docs, or examples.
### Input Module Docs Own Source Formats
Each implemented input-stage module should have a canonical integration
document.
Likely future files:
```text
docs/integrations/seriatim-transcript.md
docs/integrations/markdown-source.md
```
Input module docs should cover:
- accepted external schema or file shape;
- mapping into `SourceDocument` and `SourceUnit`;
- metadata preserved by the module;
- validation rules and failure behavior;
- examples.
The Seriatim input module doc should reference the Seriatim schema it supports and
explain how transcript segment IDs become source-unit IDs.
### Stage Module Docs Own Business Logic
Each implemented stage-module family should have canonical internal or
integration docs.
Likely future files:
```text
docs/internal/stage-modules.md
docs/integrations/artifacts-dnd.md
```
Stage module docs should cover:
- module key;
- stage;
- artifact type;
- schema version;
- required source-reference behavior;
- validator chain;
- prompt and response-schema ownership;
- examples.
D&D concepts should be documented in D&D extract-module or artifact docs, not in
generic runner or framework docs.
### CLI Docs Should Reflect Extensibility
The CLI reference should present named pipeline profiles as the primary
user-facing abstraction. Individual stage modules should be visible through
pipeline configuration and discovery commands, not through ad hoc structural
run flags.
Provisional command shape:
```sh
notarius run dnd-session --input ./source.json
notarius run dnd-session --input ./source.json --only spells,npcs
notarius config validate
notarius pipelines list
```
Once implemented, `docs/cli.md` should document:
- pipeline ID selection;
- required input path flags;
- `--only` artifact-lane selection;
- config path behavior;
- operational overrides such as output path, model, concurrency, and diagnostics
directory;
- output path behavior;
- diagnostics and report behavior;
- exit codes.
### Config Docs Should Separate Framework And Plugin-Like Options
`docs/config.md` should describe named pipeline profiles and the resolved
pipeline model.
It should cover:
- config file locations and precedence;
- `llm_profiles`;
- `pipelines.<pipeline_id>.input`;
- `pipelines.<pipeline_id>.chunk`;
- `pipelines.<pipeline_id>.artifacts.<lane>.extract`;
- lane `merge`, `normalize`, and validator settings;
- output module selection;
- string shorthand versus inline module-binding object form;
- defaults for omitted slots;
- capability validation;
- pipeline digest and manifest provenance.
Module-specific config should stay inline with the pipeline slot that owns it.
Top-level named module instances should not be introduced until repeated inline
settings create real drift. `llm_profiles` are the cross-cutting exception.
### Examples Should Stay Real
Examples should be added only when the matching behavior exists and should be
covered by tests where practical.
Likely future examples:
```text
examples/seriatim-minimal-transcript.json
examples/minimal-config.yml
examples/dnd-spells.artifacts.json
examples/dnd-session.config.yml
```
Examples should be secret-free and should use the same command shapes documented
in `docs/cli.md`.
## Canonical Documentation Targets
When the first vertical slice is implemented, add or update:
- `README.md`: concise purpose, shortest useful command, links.
- `docs/cli.md`: implemented command behavior.
- `docs/config.md`: implemented config behavior.
- `docs/operations.md`: diagnostics, retention, failure inspection.
- `docs/troubleshooting.md`: common failures.
- `docs/internal/overview.md`: implemented package map.
- `docs/internal/pipeline.md`: implemented extraction flow.
- `docs/internal/stage-modules.md`: stage contracts and implemented modules.
- `docs/internal/input-modules.md`: input adapter contract and implemented input modules.
- `docs/internal/validators.md`: validator contract and built-ins.
- `docs/integrations/seriatim-transcript.md`: Seriatim input contract.
- `docs/integrations/artifacts.md`: output artifact envelope.
## Review Checklist For Future Documentation Changes
Before merging docs, check:
- Does the document describe implemented behavior outside `docs/roadmap/`?
- Are source-format details isolated to input module or integration docs?
- Are D&D details isolated to extract module or artifact docs?
- Is there one canonical home for the topic?
- Do command examples match implemented CLI syntax?
- Do config examples use named pipeline profiles rather than ad hoc module
flags?
- Are examples valid, maintained, and free of secrets?
- Did any architecture, config, CLI, stage module, validator, or artifact
contract change require a docs update?

View File

@@ -1,888 +0,0 @@
# Implementation Plan: MVP
## Status
This is the staged implementation plan for the active MVP roadmap:
[`mvp.md`](mvp.md).
The target audience is an LLM coding agent. Implement the stages in order.
Each stage should leave the repository compiling and tested. Do not skip ahead
to later stages unless the current stage's done criteria are satisfied.
## Policy Context
Follow:
- [`../policy/architecture.md`](../policy/architecture.md)
- [`../policy/documentation.md`](../policy/documentation.md)
- [`mvp.md`](mvp.md)
- [`initial-architecture.md`](initial-architecture.md)
Required boundaries:
- framework packages must remain source-agnostic and domain-agnostic;
- source-format behavior belongs in input modules;
- D&D spell behavior, prompt assets, response schema assets, and stable
prompt/schema identifiers belong in `internal/modules/extract/dnd/spells`;
- stage business logic belongs under `internal/modules/<stage>/...` unless it
is genuinely tiny shared framework plumbing;
- structural pipeline selection must remain config-driven;
- `--only` may select artifact lanes but must not alter pipeline structure;
- output-stage warnings are out-of-band from artifact payloads and must be
available to CLI/diagnostics;
- keep planned documentation in `docs/roadmap/` until MVP behavior exists.
## Global Implementation Decisions
- Add no new third-party dependencies.
- Keep YAML config version `1` unless a user-visible config syntax change is
unavoidable. New module options can use existing binding `options`.
- Use the existing six-stage workflow:
`input -> chunk -> extract -> merge -> normalize -> output`.
- Use production CLI wiring in `internal/cli` for the MVP instead of adding a
new app package. The CLI may compose modules, but it must not own module
business logic.
- Keep `notarius run` serial over chunks for the MVP. The contracts and LLM
scheduler should still permit later parallel execution.
- Use one effective LLM profile per MVP run. The current runner accepts one
`StructuredLLMClient`, so a selected pipeline with multiple distinct effective
LLM profile IDs should fail clearly until multi-client runtime support is
intentionally added.
- Use the existing OpenAI-compatible client for real runs.
- Add a scheduled LLM client wrapper so every structured completion passes
through the configured scheduler.
- Use the existing `seriatim` input module and `dnd/spells` extractor module.
- Implement production default modules with these keys:
- `generic` chunker;
- `appendorder` merger;
- `noop` normalizer;
- `json` output encoder.
- Put production default modules under:
- `internal/modules/chunk/generic`;
- `internal/modules/merge/appendorder`;
- `internal/modules/normalize/noop`;
- `internal/modules/output/json`.
- The output encoder should return logical output files; the CLI/application
layer should write those files to disk. Encoders should not own filesystem
side effects.
- Use an output directory per run. The MVP default output root should be
`./notarius-output`, overrideable by `--output-dir`.
- File writes for durable output should be atomic where practical: write to a
temporary file in the target directory, then rename.
- Use synthetic fixtures only. Do not add private campaign transcript content,
real API keys, or private infrastructure values.
## Stage 1: Move D&D Prompt And Schema Assets Into The Spells Module
### Goal
Restore the intended framework/domain boundary before building additional MVP
functionality.
`internal/framework/llm` and `internal/framework/prompt` should provide generic
asset loading, metadata, and rendering primitives. They must not define
D&D-specific prompt IDs, response schema keys, asset paths, or tests.
### Files To Update Or Move
Expected files:
- `internal/framework/llm/schema_registry.go`
- `internal/framework/llm/schema_registry_test.go`
- `internal/framework/llm/assets/schemas/dnd_spells.v1.json`
- `internal/framework/prompt/registry.go`
- `internal/framework/prompt/render.go`
- `internal/framework/prompt/render_test.go`
- `internal/framework/prompt/assets/dnd/spells/system.md`
- `internal/framework/prompt/assets/dnd/spells/user.md`
- `internal/modules/extract/dnd/spells/extractor.go`
- `internal/modules/extract/dnd/spells/prompt.go`
- `internal/modules/extract/dnd/spells/prompt_test.go`
- `internal/modules/extract/dnd/spells/schema_test.go`
- `docs/integrations/dnd-spells.md`, only if prompt/schema ownership text needs
to be corrected.
### Required Design
Refactor `internal/framework/llm` so it can load schemas from caller-owned
embedded files.
Add or expose a generic constructor similar to:
```go
func LoadResponseSchema(fsys fs.FS, def ResponseSchemaDefinition) (ResponseSchema, error)
```
where `ResponseSchemaDefinition` carries:
- key;
- ID;
- version;
- name;
- asset path.
The existing framework registry may keep test schemas, but it must not include
`DNDSpellsSchemaKey` or `dnd_spells.v1.json`.
Refactor `internal/framework/prompt` so it can compile/render prompt pairs from
caller-owned embedded files.
Add or expose a generic constructor/rendering type similar to:
```go
type Bundle struct { ... }
func LoadBundle(fsys fs.FS, def Definition) (*Bundle, error)
func (b *Bundle) RenderUserSystem(data any) (system string, user string, metadata Metadata, err error)
```
The framework prompt package may continue to own shared hardening text if that
is useful, but it must not include `DNDSpellsPromptID` or D&D prompt paths.
Move D&D prompt and schema assets under `internal/modules/extract/dnd/spells`.
Recommended paths:
```text
internal/modules/extract/dnd/spells/assets/prompts/system.md
internal/modules/extract/dnd/spells/assets/prompts/user.md
internal/modules/extract/dnd/spells/assets/schemas/dnd_spells.v1.json
```
The spells package should define its own stable identifiers:
```go
const PromptID = "dnd.spells"
const ResponseSchemaKey = "dnd_spells"
const ResponseSchemaID = "notarius.dnd.spells"
const ResponseSchemaName = "notarius_dnd_spells_v1"
```
The spells extractor must call module-owned prompt/schema helpers and pass only
generic framework values into the LLM client.
### Required Tests
- Framework LLM schema tests prove test schemas still load, sort, clone, and
omit raw schema content from diagnostics.
- Framework LLM schema tests prove looking up `dnd_spells` in the framework
registry fails.
- Framework prompt tests prove test prompts still render and missing template
data still errors.
- Framework prompt tests contain no D&D prompt assertions.
- Spells package schema tests load the module-owned D&D schema and verify:
- key;
- ID;
- version;
- response schema name;
- valid JSON;
- clone/mutation safety;
- diagnostics omit raw schema content.
- Spells package prompt tests render the module-owned prompt and verify
hardening text and prompt metadata.
- Existing spells extractor tests still pass without importing framework-owned
D&D constants.
### Validation
Run:
```sh
gofmt -w internal/framework/llm internal/framework/prompt internal/modules/extract/dnd/spells
go test ./internal/framework/llm ./internal/framework/prompt ./internal/modules/extract/dnd/spells
go test ./...
```
## Stage 2: Add MVP Manifest And Logical Output File Contracts
### Goal
Make output and manifest contracts capable of representing the MVP's durable
run output before implementing the production JSON encoder or CLI writing.
### Files To Update
Expected files:
- `internal/framework/contracts/contracts.go`
- `internal/core/artifacts/*.go`
- `internal/framework/pipeline/runner.go`
- `internal/framework/pipeline/runner_test.go`
- `internal/framework/contracts/contracts_test.go`
- `docs/policy/architecture.md`, only if the implemented output contract
requires clarifying policy text.
### Required Design
Extend the output contract to support logical files:
```go
type OutputFile struct {
Name string `json:"name"`
ContentType string `json:"content_type,omitempty"`
Bytes []byte `json:"-"`
}
type OutputResult struct {
Files []OutputFile `json:"files,omitempty"`
Warnings []Warning `json:"warnings,omitempty"`
}
```
Remove or stop using the single `OutputResult.Bytes` / `ContentType` path once
all tests are updated. If keeping those fields temporarily reduces churn, mark
them as legacy in comments and make the runner prefer `Files`.
Add matching fields to `pipeline.RunOutput`:
```go
OutputFiles []contracts.OutputFile `json:"-"`
```
The runner should collect output-stage warnings exactly as it does now, after
calling the output encoder.
Define safe logical file names:
- names are slash-separated relative paths;
- names must not be empty, absolute, contain `..`, or contain `\`;
- names are validated before the runner returns them;
- file names are sorted deterministically by the encoder that creates them.
Extend manifest data enough for MVP provenance:
- add `RunManifest.LLMProfiles []LLMProfileManifest`;
- add `ArtifactLaneManifest.Metadata map[string]any`;
- add `RunManifest.StartedAt`, `CompletedAt`, and `RunID` population support
in the runner input/output path.
Recommended structs:
```go
type LLMProfileManifest struct {
ID string `json:"id"`
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
}
```
Add an optional metadata interface for modules:
```go
type ManifestMetadataProvider interface {
ManifestMetadata() map[string]any
}
```
When a stage module implements the interface, the runner should include that
metadata in the appropriate manifest area. For the MVP, the D&D spells extractor
will use this to report prompt and response schema identifiers/hashes on its
artifact lane.
Update `pipeline.RunInput` to accept:
- `RunID string`;
- `StartedAt time.Time`;
- `LLMProfiles []artifacts.LLMProfileManifest`.
The runner should set `CompletedAt` when the run finishes or fails after a
manifest has been initialized.
### Required Tests
- `contracts.OutputFile` JSON shape omits bytes and includes name/content type.
- Runner rejects unsafe output file names returned by an encoder.
- Runner preserves output warnings out-of-band.
- Runner output contains logical files returned by the encoder.
- Manifest includes run ID, started/completed timestamps when supplied or
generated.
- Manifest includes LLM profile metadata supplied in `RunInput`.
- Manifest includes extractor-provided lane metadata when the extractor
implements `ManifestMetadataProvider`.
### Validation
Run:
```sh
gofmt -w internal/framework/contracts internal/core/artifacts internal/framework/pipeline
go test ./internal/framework/contracts ./internal/core/artifacts ./internal/framework/pipeline
go test ./...
```
## Stage 3: Implement Production Default Stage Modules
### Goal
Make pipeline defaults real production modules instead of test-only fakes or
framework-only helpers.
### Files To Add Or Update
Expected packages:
- `internal/modules/chunk/generic`
- `internal/modules/merge/appendorder`
- `internal/modules/normalize/noop`
- `internal/modules/output/json`
Expected framework cleanup:
- `internal/framework/pipeline/generic_stages.go`
- pipeline tests that currently instantiate framework `AppendOrderMerger` or
`NoopNormalizer`.
### Required Design
#### `generic` chunker
Package: `internal/modules/chunk/generic`
Key: `generic`
Module spec:
- stage: `chunk`;
- requires: `source.transcript` is **not** required;
- provides: `chunks`.
Behavior:
- accepts any valid `SourceDocument`;
- preserves source-unit order;
- returns stable chunk IDs: `chunk-000001`, `chunk-000002`, and so on;
- copies source units defensively;
- adds chunk metadata:
- `start_unit_id`;
- `end_unit_id`;
- `unit_count`.
Options:
- `max_units`: positive integer, default `50`;
- `overlap_units`: non-negative integer, default `0`, must be less than
`max_units`.
If the source has no units, return a clear error. If options have the wrong type
or invalid values, return a clear module-specific error.
#### `appendorder` merger
Package: `internal/modules/merge/appendorder`
Key: `appendorder`
Module spec:
- stage: `merge`;
- requires: no artifact-type-specific capability;
- provides: `merged`.
Behavior:
- preserves chunk order as provided by the runner;
- preserves candidate order within each chunk;
- defensively copies candidates, payloads, source refs, and metadata;
- does not merge, deduplicate, or rewrite source references.
#### `noop` normalizer
Package: `internal/modules/normalize/noop`
Key: `noop`
Module spec:
- stage: `normalize`;
- requires: `merged`;
- provides: `normalized`.
Behavior:
- defensively copies candidates;
- does not deduplicate, rewrite, or validate domain content.
#### `json` output encoder
Package: `internal/modules/output/json`
Key: `json`
Module spec:
- stage: `output`;
- requires: `normalized`;
- provides: `encoded`.
Behavior:
- returns logical output files:
- `index.json`;
- `manifest.json`;
- `artifacts/<artifact_type>.json` for each approved artifact type;
- `rejected.json`;
- `warnings.json`.
- groups approved artifacts by `Artifact.ArtifactType`;
- sorts artifact-type file names by artifact type;
- preserves artifact order within each artifact type according to runner order;
- pretty-prints JSON with two-space indentation and trailing newline;
- uses content type `application/json`;
- includes rejected artifacts and warnings even when the arrays are empty;
- does not include output warnings inside artifact payloads.
File-name safety:
- artifact type may contain dots and hyphens;
- replace any character outside `[A-Za-z0-9._-]` with `_` for artifact file
names;
- if sanitization produces an empty name, return an error.
### Required Tests
- Generic chunker tests cover defaults, exact chunk boundaries, overlap,
invalid options, empty source, defensive copies, and stable IDs.
- Append-order merge tests cover ordering and defensive copies.
- Noop normalizer tests cover pass-through behavior and defensive copies.
- JSON output tests cover all logical files, grouping, sorted filenames,
rejected/warnings presence, pretty JSON, unsafe artifact type sanitization,
and no mutation of inputs.
- Pipeline config tests using defaults resolve when these module specs are
registered.
### Validation
Run:
```sh
gofmt -w internal/modules/chunk/generic internal/modules/merge/appendorder internal/modules/normalize/noop internal/modules/output/json internal/framework/pipeline
go test ./internal/modules/chunk/generic ./internal/modules/merge/appendorder ./internal/modules/normalize/noop ./internal/modules/output/json
go test ./internal/framework/pipeline
go test ./...
```
## Stage 4: Add Production CLI Catalog And Runtime Wiring
### Goal
Make implemented modules selectable by real CLI commands without test-injected
catalogs.
### Files To Add Or Update
Expected files:
- `internal/cli/run.go`
- new `internal/cli/catalog.go` or equivalent;
- `internal/cli/run_test.go`;
- module registry tests as needed.
### Required Design
Add production wiring in `internal/cli`:
```go
func productionRegistries() (pipeline.Registries, error)
func productionCatalog() (pipeline.ModuleCatalog, error)
```
The production wiring must register:
- input: `seriatim`;
- chunk: `generic`;
- extract: `dnd/spells`;
- merge: `appendorder`;
- normalize: `noop`;
- output: `json`.
Keep all business logic in module packages. `internal/cli` should only compose
registries/catalogs and command behavior.
Update `cli.Options` so tests may inject registries/catalog/runtime without
disabling production defaults unintentionally.
Recommended option fields:
```go
type Options struct {
Catalog pipeline.ModuleCatalog
Registries pipeline.Registries
LLMClientFactory LLMClientFactory
LookupEnv func(string) (string, bool)
Now func() time.Time
}
```
If `Catalog` or `Registries` is empty in normal `Run`, use production wiring.
If tests provide either, use the provided value.
Define `LLMClientFactory` in `internal/cli` or a small local file:
```go
type LLMClientFactory func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error)
```
The production factory should:
- read the selected LLM profile from effective config;
- construct `llm.OpenAICompatibleClient`;
- construct `llm.Scheduler` using the most specific configured concurrency:
profile `max_concurrency` if set, otherwise global `concurrency.total_llm`,
otherwise `1`;
- wrap the client in a scheduled client so every completion acquires/releases a
scheduler permit;
- return manifest-safe LLM metadata with profile ID, provider, and model.
Add a scheduled client wrapper in `internal/framework/llm` if it does not
already exist:
```go
func NewScheduledClient(client contracts.StructuredLLMClient, scheduler *Scheduler) contracts.StructuredLLMClient
```
### Required CLI Behavior
- `notarius config validate --config <file> --pipeline <id>` uses the
production catalog by default.
- `notarius pipelines list --config <file>` still lists configured pipeline
IDs and validates config shape.
- `notarius pipelines list --config <file> --json` remains stable.
Do not implement `notarius run` in this stage.
### Required Tests
- Production catalog includes the six MVP modules and their module specs.
- `config validate --pipeline` succeeds for a real MVP config fixture using no
injected catalog.
- Unknown module keys still fail with stage/pipeline context.
- Production LLM client factory rejects missing/invalid LLM profiles with clear
errors.
- Scheduled client wrapper enforces scheduler use and propagates errors.
- Existing CLI tests using injected catalogs still pass.
### Validation
Run:
```sh
gofmt -w internal/cli internal/framework/llm
go test ./internal/cli ./internal/framework/llm
go test ./...
go vet ./...
go build ./cmd/notarius
```
## Stage 5: Implement `notarius run` Without Durable File Writing
### Goal
Add the user-facing run command and prove it can drive the configured pipeline
with injected fake runtime pieces. This stage should return/run data in memory
or through test buffers, but durable file writing may be completed in Stage 6.
### Files To Update
Expected files:
- `internal/cli/run.go`
- `internal/cli/run_test.go`
- `cmd/notarius/main.go`, only if command wiring requires it.
### Required Command Shape
Support:
```sh
notarius run <pipeline-id> --input path/to/source.json
notarius run <pipeline-id> --input path/to/source.json --only spells
```
Supported flags:
- `--config path`;
- `--input path`, required;
- `--only lane-a,lane-b`;
- `--output-dir path`, parsed and passed through metadata for Stage 6;
- `--diagnostics-dir path`, overrides config diagnostics work dir for this run;
- `--llm-profile profile-id`, operational override for MVP runs.
Do not add flags for structural module selection, such as `--extractor`,
`--chunker`, `--merge`, or `--output`.
### Required Behavior
- Missing pipeline ID returns exit code `2`.
- Missing `--input` returns exit code `2`.
- Unknown flags return exit code `2`.
- Config/load/resolve/runtime failures return exit code `1`.
- Successful runs return exit code `0`.
- `--only` uses existing lane selection behavior.
- Extend `config.ResolveInput` with `LLMProfileOverride string` or an
equivalent option. When `--llm-profile` is provided, apply it to every
resolved module binding before the resolved pipeline digest is computed. This
keeps the override operational rather than structural while still making the
effective pipeline digest truthful.
- After resolution, collect the distinct effective LLM profile IDs used by the
selected pipeline. For the MVP, require exactly one distinct profile ID and
fail clearly if more than one is present.
- Input file is read as raw bytes and passed to the runner with path metadata.
- Source ID defaults to the input path or basename only if the input adapter
needs one; do not invent transcript-specific source IDs in the CLI.
- The command resolves the selected pipeline with the production catalog.
- The command constructs the LLM client through `LLMClientFactory`.
- The command invokes `pipeline.New(registries).Run(...)`.
- The command prints a concise success message that includes at least:
- pipeline ID;
- approved artifact count;
- rejected artifact count.
- If warnings exist, print a concise warning count to stderr.
### Required Tests
Use fake LLM/runtime injection; do not call external services.
- Missing pipeline ID.
- Missing input flag.
- Unknown pipeline.
- Unknown `--only` lane.
- Invalid input file path.
- Successful run invokes runner path through real registries and fake LLM.
- `--only spells` runs only the selected lane.
- LLM factory failure is reported clearly.
- Validation rejection produces a failed/non-zero or successful-with-rejections
behavior according to current runner semantics. For MVP, keep runner
semantics: a run with rejected artifacts completes successfully with
`ValidationStatus` set to `rejected`, unless an error occurs.
### Validation
Run:
```sh
gofmt -w internal/cli
go test ./internal/cli
go test ./...
go vet ./...
go build ./cmd/notarius
```
## Stage 6: Write Durable Output And Diagnostics For `notarius run`
### Goal
Complete the MVP run workflow by writing output files and diagnostics.
### Files To Update
Expected files:
- `internal/cli/run.go`
- `internal/cli/run_test.go`
- `internal/core/diagnostics/*.go`, only if helper methods are needed.
### Required Design
Output directory behavior:
- default root: `./notarius-output`;
- override: `--output-dir`;
- each run writes to `<output-root>/<run-id>/`;
- run ID comes from diagnostics run directory when available or from a
generated UTC nanosecond timestamp using the same style as diagnostics;
- create directories with `0755`;
- write files with `0644`;
- write each file atomically where practical.
Logical output files from `pipeline.RunOutput.OutputFiles` should be written
under the run output directory. Reject unsafe logical file names before writing:
- empty;
- absolute;
- contains `..`;
- contains backslash;
- escapes the run output directory after path cleaning.
Diagnostics behavior:
- create a diagnostics run directory at command start unless retention is
`never` and the implementation can still reliably capture failures; simplest
MVP behavior is to create it and then apply retention at the end;
- write invocation metadata;
- write redacted effective config;
- write resolved pipeline;
- write run manifest;
- write warnings;
- write run report containing output path, counts, and validation status;
- write error log on failure;
- apply retention with existing diagnostics policy.
`--diagnostics-dir` should override `Config.Diagnostics.WorkDir` after file and
environment config have been applied, without changing structural pipeline
definition or pipeline digest.
Success output:
- stdout includes the durable output run directory path;
- stderr includes warning count when warnings are present;
- no raw prompt text, raw API keys, or large source payloads should be printed.
### Required Tests
- Successful `notarius run` writes output files under a temp output directory.
- Output write rejects unsafe logical file names from a fake encoder.
- Writes are atomic enough that no temporary files remain after success.
- Diagnostics artifacts are written on success.
- Error log is written on failure after diagnostics directory creation.
- Retention `never` removes successful warning-free diagnostics directories.
- Warnings are present in diagnostics and are reported to stderr.
- `--diagnostics-dir` overrides config diagnostics directory.
### Validation
Run:
```sh
gofmt -w internal/cli internal/core/diagnostics
go test ./internal/cli ./internal/core/diagnostics
go test ./...
go vet ./...
go build ./cmd/notarius
```
## Stage 7: Add MVP Fixtures And End-To-End Acceptance Coverage
### Goal
Make the MVP path continuously testable without network access.
### Files To Add Or Update
Expected fixtures:
- `examples/seriatim-minimal-transcript.json`, if the example can be kept
accurate before the deferred documentation pass;
- `examples/dnd-spells.config.yml`, if config examples are tested in this
stage;
- or equivalent `internal/cli/testdata/...` fixtures if examples are deferred.
Expected tests:
- `internal/cli/run_test.go`
- `internal/modules/extract/dnd/spells/runner_test.go`
- config tests as needed.
### Required Design
Add a maintained MVP config fixture:
```yaml
version: 1
llm_profiles:
default:
provider: openai-compatible
base_url: http://127.0.0.1:1
model: fake-model
pipelines:
dnd-session:
input: seriatim
chunk:
module: generic
options:
max_units: 50
artifacts:
spells:
extract: dnd/spells
```
The fixture may use a fake base URL because tests should inject a fake LLM
client factory. Do not require a real network call.
Acceptance tests should execute the public CLI entry path with:
```sh
notarius run dnd-session --config <fixture> --input <fixture> --output-dir <tmp>
notarius run dnd-session --config <fixture> --input <fixture> --only spells --output-dir <tmp>
notarius config validate --config <fixture> --pipeline dnd-session
notarius pipelines list --config <fixture>
```
The fake LLM should return deterministic D&D spell output with valid source
references. The resulting output files should be parsed as JSON and checked for:
- manifest pipeline ID and digest;
- spell artifact payload;
- source references;
- prompt/schema metadata in manifest or artifact metadata;
- validation status;
- warning behavior.
### Required Failure Coverage
Add fixture-driven tests for:
- missing config;
- unknown pipeline;
- invalid Seriatim input;
- invalid `--only` lane;
- fake LLM failure;
- malformed LLM response;
- invalid source reference rejection.
### Validation
Run:
```sh
gofmt -w internal/cli internal/modules/extract/dnd/spells
go test ./internal/cli ./internal/modules/extract/dnd/spells
go test ./...
go vet ./...
go build ./cmd/notarius
```
## Stage 8: MVP Final Review And Roadmap Cleanup
### Goal
Confirm the MVP is complete enough to trigger the deferred documentation pass.
### Required Review
Perform a code review against:
- [`mvp.md`](mvp.md);
- [`../policy/architecture.md`](../policy/architecture.md);
- [`../policy/documentation.md`](../policy/documentation.md).
Check specifically:
- no D&D prompt/schema assets or constants remain in framework packages;
- production CLI commands use production wiring by default;
- `config validate --pipeline` works with the MVP fixture;
- `pipelines list` works with the MVP fixture;
- `notarius run` writes durable output and diagnostics;
- default modules resolve without test-only registration;
- output warnings remain out-of-band from artifact payloads;
- no private data or secrets appear in fixtures;
- docs outside `docs/roadmap/` describe only implemented behavior.
### Required Validation
Run:
```sh
go test ./...
go vet ./...
go build ./cmd/notarius
```
### Required Roadmap Update
After the MVP is implemented and reviewed:
- update [`mvp.md`](mvp.md) to mark MVP functionality complete or reduce it to
remaining release/documentation work;
- keep the full documentation pass deferred until this review passes;
- do not tag alpha `0.1.0` until the documentation pass is complete.
## Open Questions
None. The plan above makes the required MVP implementation choices explicitly.

View File

@@ -1,632 +0,0 @@
# Initial Architecture Roadmap
## Status
This document captures proposed architecture and implementation sequencing for
Notarius. It describes planned work, not implemented behavior.
## Goal
Notarius should extract structured JSON artifacts from primary source inputs
using modular, LLM-backed extractors.
The first MVP should target audio transcripts generated by Seriatim. That
choice should be implemented as an input-stage module, not as a
transcript-specific assumption in the application core. Later input sources,
such as unstructured Markdown notes or Obsidian documents, should be addable
through new input and extract modules without reshaping the framework.
The first extraction domain should be D&D session analysis, starting with spell
casts. That domain should live in extract-stage modules and related schemas, not
in core framework packages.
The application should follow the same broad architecture as Audita:
- deterministic core packages for source documents, artifacts, and configuration once needed;
- input-stage modules that translate external source formats into a small internal source model;
- reusable framework packages for contracts, orchestration, LLM runtime, structured output, and validation;
- independent extract-stage modules that own domain-specific behavior;
- independent validator packages;
- embedded prompt and JSON schema assets;
- CLI orchestration that wires the pieces together without owning domain logic.
The main domain difference from Audita is that Notarius emits extracted
artifacts rather than proposing and applying transcript corrections.
## Architectural Principles
- Keep the core input model generic: ordered text units plus metadata.
- Keep source-format details in hexagonal input modules.
- Keep extraction-domain details in extract modules.
- Treat evidence as source references, not transcript references.
- Prefer narrow, useful abstractions over a universal document model.
- Preserve enough provenance for validation, replay, and downstream inspection.
## Proposed Package Shape
```text
cmd/notarius
internal/cli
internal/core/source
internal/core/artifacts
internal/framework/contracts
internal/framework/pipeline
internal/framework/validate
internal/framework/llm
internal/framework/prompt
internal/modules/input/seriatim
internal/modules/input/markdown
internal/modules/chunk/generic
internal/modules/chunk/dndtranscript
internal/modules/extract/dnd/spells
internal/modules/extract/dnd/items
internal/modules/extract/dnd/npcs
internal/modules/extract/dnd/combat
internal/modules/merge/appendorder
internal/modules/merge/dnd/spells
internal/modules/normalize/noop
internal/modules/normalize/dnd/spells
internal/modules/output/json
internal/validators/source_refs
internal/validators/schema_validity
internal/validators/domain_consistency
internal/validators/llm_review
examples
docs/internal
```
The `markdown` input module and D&D-specific chunk, merge, normalize, and
output modules are listed as likely future packages. The MVP should implement
only the stage modules needed by the checkpoint sequence.
`internal/core/config` should be added when production configuration exists.
The framework package list is intentionally consolidated. `pipeline` should own
runner orchestration, stage registries, and small merge/normalize/output helpers
until those boundaries prove they need separate packages. `llm` should own
structured output and response-schema mechanics until those concerns become too
large or import-heavy. `prompt` should own prompt assets and rendering helpers
once prompt assets exist.
## Core Concepts
### SourceDocument
Canonical internal representation of source material. This should be the object
extractors receive, regardless of whether the original input was a transcript,
Markdown file, note export, or another source type.
```go
type SourceDocument struct {
ID string `json:"id"`
Kind string `json:"kind"`
Format string `json:"format"`
Digest string `json:"digest"`
Units []SourceUnit `json:"units"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type SourceUnit struct {
ID string `json:"id"`
Kind string `json:"kind"`
Text string `json:"text"`
Metadata map[string]any `json:"metadata,omitempty"`
}
```
Initial source-unit assumptions:
- units are ordered;
- unit IDs are stable within a source document;
- each unit has extractable text;
- adapter-specific metadata may carry speaker, timestamps, heading paths, page
numbers, or other source details.
Core source metadata should remain `map[string]any`. Notarius should not define
a universal document model. Instead, the project should document well-known
metadata keys, such as `speaker`, `start`, `end`, and `heading_path`, as
conventions. Input modules may export typed accessor helpers for their own
metadata, such as `seriatim.SpeakerOf(unit)`, without leaking those helpers into
core framework contracts.
### Input Module / Adapter Contract
Hexagonal boundary for external source formats.
```go
type InputAdapter interface {
Key() string
Parse(ctx context.Context, req ParseRequest) (*source.SourceDocument, error)
}
```
The MVP input module should target Seriatim minimal transcript JSON. Seriatim segment
fields should map as follows:
- `id` becomes `SourceUnit.ID`;
- `text` becomes `SourceUnit.Text`;
- `speaker`, `start`, and `end` become unit metadata;
- Seriatim output metadata becomes document metadata.
The core runner should not know that these units came from transcript segments.
### SourceRef
Grounding reference from an extracted fact back to source units.
```go
type SourceRef struct {
SourceID string `json:"source_id"`
StartUnitID string `json:"start_unit_id"`
EndUnitID string `json:"end_unit_id"`
}
```
Initial source-reference validation should require:
- source ID exists for the current run;
- start and end unit IDs exist;
- start is less than or equal to end in document order;
- the referenced range is contiguous within the source document;
- every extracted fact has at least one source reference unless its schema
explicitly allows ungrounded metadata.
Transcript-oriented output can still present these as transcript segment ranges
when the adapter metadata makes that interpretation available.
Source references should preserve the exact ranges produced by extractors and
validators. Overlapping ranges should not be merged or rewritten by generic
pipeline code. If a domain module wants a derived compact range later, that
should be additional output, not a replacement for the original evidence.
### Extractor
Reusable module contract for producing one artifact type.
```go
type Extractor interface {
Key() string
ArtifactType() string
SchemaVersion() string
Validators() []Validator
Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error)
}
```
An extractor should receive either a whole source document or a source chunk,
depending on processing mode. It should return typed artifact candidates plus
warnings. It should not mutate the source document.
`ExtractionRequest` should be designed now to carry both the active chunk and
optional ambient context, even if the MVP leaves that context empty. Useful
ambient context may include a document synopsis, prior-chunk summaries, known
entities, or other module-provided state. D&D spell extraction can likely work
per chunk, but combat, NPC, and identity-oriented extraction will need broader
context. Adding the field later would force churn across every extractor.
Extract modules own domain concepts. For example, D&D spell extraction should
live under `internal/modules/extract/dnd/spells`; a future to-do extractor for
notes should live under a different extract-module path and use the same
framework contract.
### Chunker
Reusable stage contract for splitting a source document into ordered source
chunks.
Chunking is a first-class pipeline concern because source documents may exceed a
single LLM extraction pass. Chunkers should preserve source-unit order and
produce stable chunk metadata suitable for diagnostics and replay.
### Merger
Reusable stage contract for combining per-chunk artifact candidates into one
merged candidate collection.
Merge should combine outputs without doing semantic reconciliation. A generic
append-in-chunk-order merger should be sufficient for many artifact streams,
including the likely first D&D spell-cast extractor.
### Normalizer
Reusable stage contract for reconciling merged artifact candidates.
Normalize is distinct from merge. Normalizers may deduplicate repeated facts,
resolve aliases, reconcile conflicting fields, check cross-chunk consistency,
or attach normalization warnings.
### Validator
Reusable validation contract for artifact candidates.
Validators should cover:
- JSON/schema validity;
- source-reference validity;
- required-field and shape checks;
- domain consistency;
- optional LLM review for high-risk or ambiguous artifacts.
Validator output should follow Audita's decision-cardinality model: each
candidate artifact receives exactly one decision per validator.
LLM-backed review should be modeled as part of a module's validator chain, not
as a separate global review phase. Extract modules should be able to attach one
or more deterministic or LLM-backed validators. Normalize-stage modules may also
run validator chains, including LLM-backed validators, when semantic
reconciliation needs review.
### Artifact
Final approved JSON output from one or more extractors.
Artifacts should preserve enough metadata to support downstream validation,
debugging, and replay.
The pipeline should carry artifact candidates through a generic envelope with a
`json.RawMessage` payload. Extract modules should own typed Go structs at their
module boundary, then encode those typed records into the generic artifact
candidate envelope before returning to framework code. This keeps stage
contracts simple and avoids generic type plumbing across unrelated artifact
families.
Final durable output should be one file per artifact type plus a run-level
manifest/index file. This supports partial success and lets downstream consumers
read only the artifact types they need. Each artifact file should include its
artifact type, extractor key, extractor schema version, envelope format version,
records, source references, and enough provenance to connect it to the run
manifest.
Every artifact record should require source references unless that artifact
schema explicitly opts into ungrounded fields. Artifact-level metadata, counts,
run information, and other derived summary fields are exempt from the per-record
grounding rule.
Schemas should be versioned per extractor, with a separate envelope/manifest
format version. A single global schema version would couple unrelated extractor
release cadence.
### RunManifest
Per-run provenance record.
```go
type RunManifest struct {
EnvelopeVersion string `json:"envelope_version"`
PipelineID string `json:"pipeline_id"`
PipelineDigest string `json:"pipeline_digest"`
InputModule string `json:"input_module"`
Chunker string `json:"chunker"`
SourceDigests []string `json:"source_digests"`
Extractors []string `json:"extractors"`
Merger string `json:"merger"`
Normalizer string `json:"normalizer"`
OutputEncoder string `json:"output_encoder"`
SchemaVersion string `json:"schema_version"`
ValidationStatus string `json:"validation_status"`
}
```
The manifest should eventually include model names, prompt IDs, prompt hashes,
response schema versions, config source, redacted resolved config digest,
started/completed timestamps, and diagnostics paths.
## Initial Extractor Targets
### D&D Spells
Recommended first vertical slice because it is narrow but representative.
```go
type SpellCast struct {
Player string `json:"player"`
Spell string `json:"spell"`
Effect string `json:"effect"`
NarrativeDescription string `json:"narrative_description"`
SourceRefs []SourceRef `json:"source_refs"`
}
```
The spell extractor should be D&D-specific. The framework should not know what a
spell is.
### D&D Items
Tracks items gained, lost, transferred, consumed, or transformed.
Open questions:
- Should currency be represented as items or as its own artifact type?
- Should item ownership be a required field?
- How should ambiguous ownership changes be represented?
### D&D NPCs
Tracks NPCs interacted with, newly introduced, renamed, described, or otherwise
made relevant to campaign state.
Open questions:
- Should NPC identity resolution happen inside this extractor or in a later
deduplication stage?
- Should location/faction/relationship facts be separate artifact types?
### D&D Combat
Likely warrants a dedicated schema rather than a generic event list.
Proposed first shape:
```go
type CombatTurn struct {
Actor string `json:"actor"`
Action string `json:"action"`
Outcome string `json:"outcome"`
NarrativeDescription string `json:"narrative_description"`
SourceRefs []SourceRef `json:"source_refs"`
}
```
Open questions:
- Should combat be extracted as turns, rounds, encounters, or all three?
- Should mechanical fields such as damage, conditions, saves, attacks, and spell
slots be normalized immediately or added later?
- How should uncertain initiative order be represented?
### Future Non-D&D Extractors
The architecture should support extractors outside the D&D domain. Examples:
- to-do items from Markdown or Obsidian notes;
- decisions and action items from meeting transcripts;
- named people, places, and dates from research notes.
These should be addable as extract modules without changing runner,
validator, source-reference, or LLM framework contracts.
## Configuration Model
Notarius should use named pipeline profiles selected by ID at the CLI. A
pipeline is a fixed-shape template for the known application workflow, not a
free-form list of steps:
```text
input -> chunk -> extract -> merge -> normalize -> output
```
A pipeline profile should define one shared front end and one or more artifact
lanes:
- shared input module;
- shared chunk module by default;
- artifact lanes containing extract, merge, normalize, and validator behavior;
- shared output module.
The MVP should use one shared chunk module per pipeline. Per-lane chunk
overrides can be added later if an artifact lane, such as combat, proves it
needs a different chunking strategy.
Example shape:
```yaml
llm_profiles:
default:
model: example-model
max_concurrency: 4
pipelines:
dnd-session:
input: seriatim
chunk: dnd/transcript
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells
npcs:
extract: dnd/npcs
items:
extract: dnd/items
```
The CLI should run named pipelines:
```sh
notarius run dnd-session --input session-014.json
notarius run dnd-session --input session-014.json --only spells,npcs
```
`--only` should select configured artifact lanes. It should not create an
ad hoc pipeline. Structural module selection should come from config, while CLI
flags may override operational knobs such as model, concurrency, output
directory, and diagnostics directory.
Initial defaults:
- `chunk`: `generic`;
- lane `merge`: `appendorder`;
- lane `normalize`: `noop`;
- `output`: `json`;
- `llm_profile`: `default` where an LLM profile is needed.
Module bindings should support both string shorthand and object form:
```yaml
extract: dnd/spells
```
```yaml
extract:
module: dnd/spells
llm_profile: fast
prompt_version: v1
```
Both forms should normalize into a single internal `ModuleBinding` shape before
validation and manifest hashing.
Pipeline validation should use module metadata declared through registries.
Modules should expose flat string capability metadata, such as `speaker` or
`timestamps`, without requiring module construction. Config validation should
fail fast for:
- unknown pipeline IDs;
- unknown module keys;
- missing required slots;
- missing required capabilities;
- unknown LLM profiles;
- empty artifact-lane sets;
- `--only` lane names that do not exist in the selected pipeline.
The MVP should keep pipelines config-file-only. Built-in pipeline profiles can
be added later if the project needs embedded defaults, but that introduces
merge/override semantics that the MVP does not need.
The resolved pipeline definition should be hashed after defaults and lane
selection are applied. The run manifest should record both `pipeline_id` and
`pipeline_digest`; a pipeline ID alone is not stable provenance.
## Proposed Pipeline Flow
The application workflow should be first-class:
```text
input -> chunk -> extract -> merge -> normalize -> output
```
Proposed runner flow:
1. Load effective config.
2. Resolve the selected pipeline profile by ID.
3. Apply defaults and `--only` lane selection.
4. Validate module keys, lane definitions, LLM profiles, and capabilities.
5. Hash the resolved pipeline definition.
6. Create diagnostics run directory.
7. Resolve the configured input module through the input adapter registry.
8. Read source input.
9. Parse source input into a `SourceDocument`.
10. Validate source-document invariants.
11. Resolve the configured chunker.
12. Chunk source units into deterministic source chunks.
13. Resolve configured artifact lanes through registries.
14. Extract, merge, normalize, and validate each selected artifact lane.
15. Retain approved artifacts and rejected-artifact diagnostics.
16. Serialize output files and run-level manifest/index.
17. Write diagnostics and optional report JSON.
The runner should operate on source documents and source chunks only. Any
transcript-specific behavior should happen before the runner, inside the input
adapter, or after the runner, inside output rendering that understands source
metadata.
## Audita Patterns To Reuse
Reuse these architectural patterns:
- deterministic parsing and schema validation style;
- deterministic chunking of ordered source units;
- explicit extractor registry;
- explicit pipeline stage contracts;
- `contracts` package for transport-neutral interfaces;
- OpenAI-compatible structured LLM client;
- scheduler for bounded LLM concurrency;
- embedded prompt registry with prompt metadata and hashes;
- embedded response-schema registry with schema metadata and hashes;
- diagnostics run directory with redacted effective config;
- validator decision cardinality and deterministic validator ordering;
- CLI tests and fixture-driven integration tests.
The fixture-driven integration-test pattern should remain part of the codebase:
walking skeleton tests over fake modules and fake LLM clients should be
preserved as real Seriatim, runtime, and D&D modules are added, so the
end-to-end contract coverage is not lost.
Avoid copying these Audita concepts directly:
- transcript-specific core types;
- correction proposals;
- replacement policies;
- deterministic transcript mutation;
- correction ledger terminology.
Those concepts are specific to Audita's transcript-editing role and should be
replaced with source-document, artifact-candidate, artifact-validation, and
extraction-report concepts.
## Checkpoint Roadmap
The initial six checkpoint roadmap has been implemented and retired. The
checkpoint files have been removed from `docs/roadmap/` so the active roadmap
does not compete with completed implementation history.
The remaining work needed to reach the first functional MVP is tracked in
[`mvp.md`](mvp.md). Future staged implementation plans should be written to
[`implementation.md`](implementation.md) from that active MVP roadmap.
## Architecture Decisions
- Final durable output should use one artifact file per artifact type plus a
run-level manifest/index file.
- Framework artifact flow should use a generic envelope with `json.RawMessage`
payloads. Extract modules should use typed Go structs at their own boundaries.
- Schemas should be versioned per extractor, with a separate envelope/manifest
format version.
- Artifact records should require source references by default. Individual
schemas may explicitly opt into ungrounded fields. Artifact-level metadata is
exempt.
- Source-reference ranges should be preserved exactly. Generic pipeline code
should not merge or rewrite overlapping ranges.
- `ExtractionRequest` should carry the active chunk plus optional ambient
context for document synopsis, prior-chunk summaries, known entities, or
similar module-provided state.
- LLM-backed review should be part of module-owned validator chains. Extract
modules and normalize modules may both use deterministic and LLM-backed
validators.
- The Seriatim MVP should support only the minimal Seriatim schema. Broader
Seriatim schema support should be added later without changing core source
contracts.
- Core source metadata should remain `map[string]any`. Well-known metadata keys
should be documented as conventions, and input modules may expose typed
accessor helpers for their own metadata.
- Configuration should use named pipeline profiles selected by ID at the CLI.
- A pipeline profile should be a fixed template, not a free-form DAG: shared
input and chunk stages, one or more artifact lanes, and shared output.
- `--only` should select configured artifact lanes without creating ad hoc
pipelines.
- Module bindings should support string shorthand and inline object settings,
normalized into one internal binding shape.
- Registries should expose flat capability metadata so config can fail fast on
invalid module combinations.
- The run manifest should record both `pipeline_id` and a digest of the resolved
pipeline definition after defaults and lane selection.
## Open Design Questions
- Which artifact types should use generic append-in-chunk-order merge, and which
should use domain-specific merge?
- Which artifact types need domain-specific normalization for deduplication,
identity resolution, or consistency?
- Which operational settings should be allowed as CLI/environment overrides
without weakening pipeline provenance?
## Near-Term Documentation Tasks
Once behavior is implemented, move implemented contracts out of roadmap docs and
into canonical docs:
- `README.md` for purpose and shortest useful command;
- `docs/cli.md` for CLI behavior;
- `docs/config.md` for config fields and precedence;
- `docs/internal/` for implemented architecture and package boundaries;
- `docs/integrations/` for source input and artifact file formats;
- `examples/` for maintained source, config, and artifact examples.

View File

@@ -1,249 +1,29 @@
# MVP Roadmap
# Future Work
## Status
Current Notarius behavior is documented in the canonical README, CLI,
configuration, operations, internal, and integration docs. This roadmap records
future work only.
This is the active roadmap for reaching the first functional Notarius MVP.
## Candidate Product Work
The previous numbered checkpoint roadmaps have been implemented and retired.
This document captures the remaining work needed to turn the implemented
architecture into a usable MVP, with the current architectural review findings
folded in as first-class work.
- Additional input adapters, such as Markdown or note-export formats.
- Additional D&D extractors beyond spell casts.
- Cross-lane entity normalization.
- Cross-chunk semantic deduplication.
- Configurable validator chains with production validator modules.
- Multiple effective LLM profiles in one run.
- Parallel execution where it preserves deterministic manifests and diagnostics.
- Additional output encoders.
Implementation staging belongs in [`implementation.md`](implementation.md).
## Candidate Operational Work
## MVP Goal
- Packaged release artifacts for alpha distribution.
- A documented versioning and release process.
- Optional generated example output fixtures with a regeneration procedure.
- Additional diagnostics or reporting views if operator workflows need them.
The MVP should let a user run Notarius against a Seriatim minimal transcript
JSON file, select a configured pipeline profile, extract D&D spell-cast
artifacts with an LLM-backed extractor, validate those artifacts, and write
durable JSON output plus diagnostics.
## Non-Goals To Revisit Deliberately
The intended command shape is:
```sh
notarius run dnd-session --input session-014.json
notarius run dnd-session --input session-014.json --only spells
```
The MVP remains intentionally narrow:
- one production input module: `seriatim`;
- one production extract module: `dnd/spells`;
- one generic chunk module sufficient for transcript-scale processing;
- generic append-order merge;
- generic noop normalization;
- generic JSON output;
- config-driven pipeline profiles;
- OpenAI-compatible structured LLM execution through the existing LLM client.
## MVP Work Areas
### Framework/Domain Asset Boundaries
This is the highest-priority remaining architecture correction.
Framework packages must remain source-agnostic and domain-agnostic. D&D spell
prompt assets, response schema assets, prompt IDs, response schema keys, and
domain-specific prompt/schema tests should not live in `internal/framework/llm`
or `internal/framework/prompt`.
Target state:
- `internal/framework/llm` provides generic structured-output client,
scheduler, schema metadata, schema loading, and schema lookup/registration
primitives.
- `internal/framework/prompt` provides generic prompt metadata, prompt loading,
rendering, hardening, and lookup/registration primitives.
- `internal/modules/extract/dnd/spells` owns the D&D spell prompt assets,
response schema assets, stable prompt ID, stable response schema key, and
module-specific prompt/schema tests.
- Framework tests use placeholder/test assets only.
- The D&D spells extractor depends on generic framework APIs, not
framework-owned D&D constants.
This work should not change the external artifact shape or module key. It is an
ownership and package-boundary correction.
### Production Application Catalog Wiring
The implemented modules and registries are currently exercised mostly through
tests that inject catalogs. The MVP needs a production assembly point that
builds the catalog and stage registries used by real CLI commands.
Target state:
- a small app-level package or CLI wiring function constructs the production
`pipeline.ModuleCatalog`;
- production wiring registers `seriatim`;
- production wiring registers `dnd/spells`;
- production wiring registers the default `generic`, `appendorder`, `noop`, and
`json` modules;
- `notarius config validate --pipeline ...` validates real configured
pipelines without test-only catalog injection;
- `notarius pipelines list` reports production-registered modules where useful
for discoverability.
The production wiring should not move domain behavior into the CLI. The CLI may
compose modules, but module packages should continue to own their own behavior
and metadata.
### Default Production Stage Modules
Pipeline defaults are already part of the architecture:
- `chunk: generic`;
- `merge: appendorder`;
- `normalize: noop`;
- `output: json`.
The MVP should make those defaults real production modules rather than
test-only conveniences.
Target state:
- `generic` chunking creates ordered chunks over generic source units and is
configurable enough for transcript MVP use;
- `appendorder` merge serializes artifact candidates in deterministic source
and chunk order;
- `noop` normalize passes merged artifacts through unchanged while preserving
diagnostics;
- `json` output encodes approved artifacts, rejected artifacts, warnings,
manifest data, and relevant run metadata in a durable JSON shape;
- each default module declares module specs and capabilities compatible with
pipeline validation;
- default modules are registered by production app wiring.
If a default module remains implemented in `internal/framework/pipeline`, its
production registration still needs to be explicit and discoverable. If its
logic grows beyond a small generic helper, move it under `internal/modules`.
### `notarius run`
The MVP needs a functional run command that drives the already-implemented
pipeline runner.
Target state:
- command shape:
```sh
notarius run <pipeline-id> --input path/to/source.json
notarius run <pipeline-id> --input path/to/source.json --only spells
```
- required flags and arguments produce clear usage errors;
- `--config` selects the config file;
- `--only` filters artifact lanes without changing structural pipeline config;
- operational overrides may cover output directory, work directory,
concurrency, and LLM profile/model settings where already supported by config;
- structural stage selection remains config-driven;
- the command parses input through the configured input adapter;
- the command constructs the configured LLM client and scheduler;
- the command invokes the pipeline runner;
- the command writes durable output and diagnostics;
- failures return stable non-zero exit codes and useful error messages.
The command should be covered by fixture-driven CLI tests with fake LLM behavior
where network calls would otherwise be required.
### MVP Output And Diagnostics Behavior
The MVP should produce inspectable files that are stable enough for downstream
experiments, without pretending to be a final public artifact contract.
Target state:
- output path behavior is deterministic and documented in code/tests;
- JSON output includes approved artifacts grouped or ordered predictably;
- each artifact includes its generic source references;
- rejected artifacts and validation decisions remain inspectable;
- output-stage warnings remain out-of-band from the durable artifact payload but
are captured for CLI reporting and diagnostics;
- run manifest data includes source digest, resolved pipeline digest, relevant
model/profile information, prompt/schema identifiers, and validation status;
- diagnostics redact secrets and include the resolved effective configuration
needed to debug a run.
### MVP Fixtures And Acceptance Tests
The MVP should be continuously testable without external services.
Target state:
- maintained Seriatim transcript fixture for the D&D spells MVP;
- maintained minimal config fixture for the MVP pipeline;
- fake LLM path for deterministic CLI and runner tests;
- config validation tests using the production catalog;
- `notarius run` fixture test from input file to output JSON;
- failure tests for missing config, unknown pipeline, invalid input,
invalid lane selection, LLM failure, and validation rejection;
- `go test ./...` is sufficient to exercise the MVP path without network
access.
### Documentation Pass Preparation
The full documentation pass is intentionally deferred until MVP functionality
exists. It should happen before tagging alpha `0.1.0`.
The MVP implementation should still leave clear hooks for the documentation
rewrite:
- command behavior should be stable enough to document in `docs/cli.md`;
- config behavior should be stable enough to document in `docs/config.md`;
- output behavior should be stable enough to document in integration docs;
- examples should be generated from or validated against maintained fixtures
where practical.
## Out Of Scope For MVP
- D&D item extraction;
- NPC extraction;
- combat extraction;
- D&D rules validation beyond the spell extractor's deterministic checks;
- Markdown or Obsidian input;
- cross-lane entity normalization;
- cross-chunk semantic deduplication beyond whatever a simple normalizer can
safely support;
- a general DAG or workflow engine;
- ad hoc CLI flags for structural module selection;
- release-quality documentation before the MVP behavior is implemented.
## MVP Done Criteria
- D&D prompt and response schema assets are owned by the D&D spells module, not
by framework packages.
- Production CLI commands use a real app catalog rather than test-injected
module catalogs.
- A config profile can bind `input: seriatim` and an artifact lane with
`extract: dnd/spells`.
- Default `generic`, `appendorder`, `noop`, and `json` modules resolve through
production wiring.
- `notarius config validate --config <file> --pipeline <id>` works with the
MVP config.
- `notarius pipelines list --config <file>` works with the MVP config.
- `notarius run <pipeline-id> --input <file>` reads a Seriatim transcript,
extracts D&D spell artifacts, validates them, and writes JSON output.
- `notarius run <pipeline-id> --input <file> --only spells` runs only the
selected artifact lane.
- The run manifest records source digest, resolved pipeline digest, LLM profile
and model, prompt/schema identifiers, and validation status.
- Output warnings are available to CLI/diagnostics without becoming artifact
payload fields.
- MVP fixture tests cover the full path without network access.
- `go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` pass.
## Deferred Documentation Pass
After MVP behavior is implemented and before alpha `0.1.0`, complete a full
documentation pass/rewrite. That pass should move implemented behavior out of
roadmap documents and into canonical docs required by
[`../policy/documentation.md`](../policy/documentation.md), including at least:
- `README.md`;
- `docs/cli.md`;
- `docs/config.md`;
- `docs/operations.md`, if diagnostics/run recovery behavior warrants it;
- `docs/internal/` architecture and package-boundary docs;
- `docs/integrations/` updates for Seriatim input, D&D spell artifacts, and
JSON output;
- maintained `examples/` files.
- A general workflow language.
- Structural module selection through ad hoc run flags.
- Storing secrets in config files, diagnostics, manifests, or examples.

215
docs/troubleshooting.md Normal file
View File

@@ -0,0 +1,215 @@
# Troubleshooting
This guide maps common implemented failure modes to inspection steps and fixes.
For command syntax, see [CLI Reference](cli.md). For YAML fields and
environment overrides, see [Configuration](config.md). For output and
diagnostics layout, see [Operations](operations.md).
## Config File Not Found
Symptom:
```text
notarius: config file not found; pass --config or set NOTARIUS_CONFIG
```
Fix:
- Pass `--config path/to/config.yml`.
- Or set `NOTARIUS_CONFIG` to a readable file.
- Or install a config at `/usr/local/etc/notarius/config.yml`.
If the message says the config path is a directory or is not available, correct
the path or file permissions.
## Unsupported Or Invalid Config
Symptoms include:
- `unsupported config version`
- `config version is required`
- `field <name> not found`
- `total LLM concurrency must be greater than zero`
- `diagnostics retention "<value>" is not supported`
Fix:
- Use `version: 1`.
- Remove unknown YAML fields.
- Validate with:
```sh
go run ./cmd/notarius config validate --config path/to/config.yml
```
## Unknown Pipeline
Symptom:
```text
notarius: pipeline "..." is not configured
```
Fix:
- List configured pipeline IDs:
```sh
go run ./cmd/notarius pipelines list --config path/to/config.yml
```
- Use one of those IDs in `notarius run <pipeline-id>`.
- Check indentation under the top-level `pipelines` map.
## Unknown Or Incompatible Module
Symptoms mention a module key, pipeline slot, lane, capability, or `not
registered`.
Fix:
- Validate the pipeline against the production module catalog:
```sh
go run ./cmd/notarius config validate \
--config path/to/config.yml \
--pipeline dnd-session
```
- Use only implemented production module keys listed in
[Configuration](config.md#implemented-production-modules).
- Check that artifact lanes include an `extract` binding.
## Invalid `--only`
Symptoms include:
- `--only must contain comma-separated non-empty artifact lane IDs`
- `--only requires --pipeline`
- `selected artifact lane`
Fix:
- Use comma-separated lane IDs with no empty entries:
```sh
go run ./cmd/notarius run dnd-session \
--config path/to/config.yml \
--input path/to/input.json \
--only spells
```
- For `config validate`, include `--pipeline` when using `--only`.
- Confirm the lane ID exists under `pipelines.<id>.artifacts`.
## Seriatim Input Validation Failure
Symptoms include `seriatim input`, `parse JSON`, `segments must not be empty`,
or validation errors naming a segment field.
Fix:
- Compare the input to
[examples/seriatim-minimal-transcript.json](../examples/seriatim-minimal-transcript.json).
- Ensure the JSON has a `metadata` object and a non-empty `segments` array.
- Each segment needs a non-empty `id`, non-empty `speaker`, non-empty `text`,
non-negative numeric `start`, and non-negative numeric `end`.
- Segment IDs must be unique and must not contain leading or trailing
whitespace.
- `end` must be greater than or equal to `start`.
## Missing LLM Base URL Or Model
Symptoms include:
- `LLM profile "default" base URL must not be empty`
- `LLM profile "default" model must not be empty`
- `base URL must be valid`
Fix:
- Set `base_url` and `model` in `llm_profiles.default`.
- Or set `NOTARIUS_LLM_DEFAULT_BASE_URL` and
`NOTARIUS_LLM_DEFAULT_MODEL`.
- If a profile needs authentication, set `api_key_env` in YAML or set
`NOTARIUS_LLM_DEFAULT_API_KEY`.
## LLM Profile Override Failure
Symptom:
```text
notarius: LLM profile override "..." is not configured
```
Fix:
- Add the profile under `llm_profiles`.
- Or use an existing profile ID with `--llm-profile`.
Current runs require exactly one distinct effective LLM profile. If a pipeline
uses several profiles, run with `--llm-profile <id>` or align the bindings in
configuration.
## Provider HTTP Or Response Failure
Symptoms include:
- `provider request failed`
- `provider returned status 400`
- `provider returned status 403`
- `provider response missing choices`
- `provider response assistant message content is not valid JSON`
- `decode structured output`
Fix:
- Confirm the `base_url` points to an OpenAI-compatible endpoint root. Notarius
posts to `<base_url>/chat/completions`.
- Check `model` and provider credentials.
- Inspect the retained diagnostics `error.log`.
- For 400 and 403 responses, fix the request configuration or credentials.
- For 429 and 5xx responses, the client retries according to `max_retries`; if
the failure persists, inspect the provider response and adjust capacity,
credentials, or model settings.
- The assistant message content must decode as JSON matching the extractor's
structured response schema.
Provider error messages are redacted for configured API key values.
## Output Write Failure
Symptoms include:
- `create output directory`
- `write output file`
- `output file name must`
Fix:
- Ensure `--output-dir` points to a directory path or a path that can be
created.
- Check filesystem permissions and available disk space.
- If diagnostics were retained, inspect `run-report.json`, `run-manifest.json`,
and `error.log`.
The CLI rejects unsafe logical output paths before writing files.
## Diagnostics Directory Surprise
Symptom: the diagnostics directory is missing after a successful run.
Fix:
- Check `diagnostics.retention`.
- With `auto`, successful runs without warnings are removed.
- Use `diagnostics.retention: always` when every diagnostics run directory
should be kept.
- Use `--diagnostics-dir` to override the configured work directory for a run.
Symptom: diagnostics exist even with `retention: never`.
Explanation:
- Failed runs are retained so that `error.log` and available context can be
inspected.

View File

@@ -0,0 +1,16 @@
version: 1
llm_profiles:
default:
provider: openai-compatible
base_url: http://127.0.0.1:1
model: fake-model
pipelines:
dnd-session:
input: seriatim
chunk:
module: generic
options:
max_units: 50
artifacts:
spells:
extract: dnd/spells

View File

@@ -0,0 +1,22 @@
{
"metadata": {
"id": "session-alpha",
"title": "Synthetic D&D spell session"
},
"segments": [
{
"id": "seg-001",
"start": 0,
"end": 4,
"speaker": "Aria",
"text": "Aria raises her holy symbol and casts Cure Wounds."
},
{
"id": "seg-002",
"start": 4,
"end": 8,
"speaker": "DM",
"text": "The bandit mage casts Shield as the blow lands."
}
]
}

172
internal/cli/catalog.go Normal file
View File

@@ -0,0 +1,172 @@
package cli
import (
"context"
"fmt"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
"gitea.maximumdirect.net/eric/notarius/internal/modules/extract/dnd/spells"
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
)
func productionRegistries() (pipeline.Registries, error) {
registries := pipeline.Registries{
Inputs: pipeline.NewInputAdapterRegistry(),
Chunkers: pipeline.NewChunkerRegistry(),
Extractors: pipeline.NewExtractorRegistry(),
Mergers: pipeline.NewMergerRegistry(),
Normalizers: pipeline.NewNormalizerRegistry(),
Validators: pipeline.NewValidatorRegistry(),
Outputs: pipeline.NewOutputEncoderRegistry(),
}
if err := seriatim.Register(registries.Inputs); err != nil {
return pipeline.Registries{}, fmt.Errorf("register seriatim input: %w", err)
}
if err := generic.Register(registries.Chunkers); err != nil {
return pipeline.Registries{}, fmt.Errorf("register generic chunker: %w", err)
}
if err := spells.Register(registries.Extractors); err != nil {
return pipeline.Registries{}, fmt.Errorf("register dnd spells extractor: %w", err)
}
if err := appendorder.Register(registries.Mergers); err != nil {
return pipeline.Registries{}, fmt.Errorf("register appendorder merger: %w", err)
}
if err := noop.Register(registries.Normalizers); err != nil {
return pipeline.Registries{}, fmt.Errorf("register noop normalizer: %w", err)
}
if err := jsonoutput.Register(registries.Outputs); err != nil {
return pipeline.Registries{}, fmt.Errorf("register json output encoder: %w", err)
}
return registries, nil
}
func productionCatalog() (pipeline.ModuleCatalog, error) {
registries, err := productionRegistries()
if err != nil {
return pipeline.ModuleCatalog{}, err
}
return catalogFromRegistries(registries), nil
}
func effectiveCatalog(opts Options) (pipeline.ModuleCatalog, error) {
if !isEmptyCatalog(opts.Catalog) {
return opts.Catalog, nil
}
if !isEmptyRegistries(opts.Registries) {
return catalogFromRegistries(opts.Registries), nil
}
return productionCatalog()
}
func effectiveRegistries(opts Options) (pipeline.Registries, error) {
if !isEmptyRegistries(opts.Registries) {
return opts.Registries, nil
}
if !isEmptyCatalog(opts.Catalog) {
return registriesFromCatalog(opts.Catalog), nil
}
return productionRegistries()
}
func catalogFromRegistries(registries pipeline.Registries) pipeline.ModuleCatalog {
return pipeline.ModuleCatalog{
Inputs: registries.Inputs,
Chunkers: registries.Chunkers,
Extractors: registries.Extractors,
Mergers: registries.Mergers,
Normalizers: registries.Normalizers,
Validators: registries.Validators,
Outputs: registries.Outputs,
}
}
func registriesFromCatalog(catalog pipeline.ModuleCatalog) pipeline.Registries {
return pipeline.Registries{
Inputs: catalog.Inputs,
Chunkers: catalog.Chunkers,
Extractors: catalog.Extractors,
Mergers: catalog.Mergers,
Normalizers: catalog.Normalizers,
Validators: catalog.Validators,
Outputs: catalog.Outputs,
}
}
func isEmptyCatalog(catalog pipeline.ModuleCatalog) bool {
return catalog.Inputs == nil &&
catalog.Chunkers == nil &&
catalog.Extractors == nil &&
catalog.Mergers == nil &&
catalog.Normalizers == nil &&
catalog.Validators == nil &&
catalog.Outputs == nil
}
func isEmptyRegistries(registries pipeline.Registries) bool {
return registries.Inputs == nil &&
registries.Chunkers == nil &&
registries.Extractors == nil &&
registries.Mergers == nil &&
registries.Normalizers == nil &&
registries.Validators == nil &&
registries.Outputs == nil
}
func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
if err := ctx.Err(); err != nil {
return nil, nil, err
}
trimmedID := strings.TrimSpace(profileID)
if trimmedID == "" {
trimmedID = pipeline.DefaultLLMProfile
}
profile, ok := cfg.LLMProfile(trimmedID)
if !ok {
return nil, nil, fmt.Errorf("LLM profile %q is not configured", trimmedID)
}
clientCfg, err := cfg.OpenAICompatibleClientConfig(trimmedID)
if err != nil {
return nil, nil, err
}
client, err := llm.NewOpenAICompatibleClient(clientCfg)
if err != nil {
return nil, nil, fmt.Errorf("create LLM client for profile %q: %w", trimmedID, err)
}
scheduler, err := llm.NewScheduler(effectiveLLMConcurrency(cfg, profile))
if err != nil {
return nil, nil, fmt.Errorf("create LLM scheduler for profile %q: %w", trimmedID, err)
}
provider := strings.TrimSpace(profile.Provider)
if provider == "" {
provider = "openai-compatible"
}
metadata := []artifacts.LLMProfileManifest{
{
ID: trimmedID,
Provider: provider,
Model: strings.TrimSpace(profile.Model),
},
}
return llm.NewScheduledClient(client, scheduler), metadata, nil
}
func effectiveLLMConcurrency(cfg config.Config, profile config.LLMProfile) int {
if profile.MaxConcurrency > 0 {
return profile.MaxConcurrency
}
if cfg.Concurrency.TotalLLM > 0 {
return cfg.Concurrency.TotalLLM
}
return 1
}

View File

@@ -1,31 +1,45 @@
package cli
import (
"context"
"encoding/json"
"flag"
"fmt"
"io"
"os"
"path"
"path/filepath"
"sort"
"strings"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const defaultConfigPath = "/usr/local/etc/notarius/config.yml"
const defaultOutputRoot = "./notarius-output"
const usage = `Usage:
notarius help
notarius run <pipeline-id> --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b]
notarius config validate --config path/to/config.yml [--pipeline pipeline-id] [--only lane-a,lane-b]
notarius pipelines list --config path/to/config.yml [--json]
`
type Options struct {
Catalog pipeline.ModuleCatalog
LookupEnv func(string) (string, bool)
Catalog pipeline.ModuleCatalog
Registries pipeline.Registries
LLMClientFactory LLMClientFactory
LookupEnv func(string) (string, bool)
Now func() time.Time
}
type LLMClientFactory func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error)
// Run executes the command-line interface and returns a process exit code.
func Run(args []string, stdout, stderr io.Writer) int {
return RunWithOptions(args, stdout, stderr, Options{})
@@ -46,6 +60,8 @@ func RunWithOptions(args []string, stdout, stderr io.Writer, opts Options) int {
return runConfig(args[1:], stdout, stderr, opts)
case "pipelines":
return runPipelines(args[1:], stdout, stderr, opts)
case "run":
return runPipelineCommand(args[1:], stdout, stderr, opts)
default:
fmt.Fprintf(stderr, "notarius: unknown command %q\n", args[0])
writeUsage(stderr)
@@ -61,9 +77,389 @@ func normalizeOptions(opts Options) Options {
if opts.LookupEnv == nil {
opts.LookupEnv = os.LookupEnv
}
if opts.Now == nil {
opts.Now = time.Now
}
if opts.LLMClientFactory == nil {
opts.LLMClientFactory = productionLLMClientFactory
}
return opts
}
func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) int {
fs := flag.NewFlagSet("run", flag.ContinueOnError)
fs.SetOutput(io.Discard)
configPath := fs.String("config", "", "config file path")
inputPath := fs.String("input", "", "source input file path")
onlyRaw := fs.String("only", "", "comma-separated artifact lanes")
outputDir := fs.String("output-dir", "", "output directory")
diagnosticsDir := fs.String("diagnostics-dir", "", "diagnostics directory")
llmProfile := fs.String("llm-profile", "", "LLM profile override")
if err := fs.Parse(reorderRunArgs(args)); err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 2
}
if fs.NArg() == 0 {
fmt.Fprintln(stderr, "notarius: run requires a pipeline ID")
return 2
}
if fs.NArg() > 1 {
fmt.Fprintf(stderr, "notarius: unexpected argument %q\n", fs.Arg(1))
return 2
}
pipelineID := strings.TrimSpace(fs.Arg(0))
if pipelineID == "" {
fmt.Fprintln(stderr, "notarius: run requires a pipeline ID")
return 2
}
if strings.TrimSpace(*inputPath) == "" {
fmt.Fprintln(stderr, "notarius: run requires --input")
return 2
}
only, err := parseOnly(*onlyRaw)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 2
}
cfg, loadedConfigPath, err := loadConfig(*configPath, opts)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
if dir := strings.TrimSpace(*diagnosticsDir); dir != "" {
cfg.Diagnostics.WorkDir = dir
}
startedAt := opts.Now().UTC()
runDir, err := diagnostics.NewRunDirectory(cfg.Diagnostics.WorkDir, cfg.Diagnostics.Retention)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
invocation := diagnostics.InvocationMetadata{
Operation: "run",
PipelineID: pipelineID,
InputPath: strings.TrimSpace(*inputPath),
ConfigPath: loadedConfigPath,
ConfigSource: configSource(*configPath),
OnlyLanes: append([]string(nil), only...),
RunID: runDir.RunID(),
StartedAt: startedAt,
}
if err := runDir.WriteInvocationMetadata(invocation); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics invocation metadata: %w", err))
}
catalog, err := effectiveCatalog(opts)
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
}
effective, err := cfg.Resolve(config.ResolveInput{
PipelineID: pipelineID,
Only: only,
Catalog: catalog,
LLMProfileOverride: *llmProfile,
})
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
}
invocation.PipelineDigest = effective.ResolvedPipeline.Digest
if err := runDir.WriteInvocationMetadata(invocation); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics invocation metadata: %w", err))
}
if err := runDir.WriteRedactedEffectiveConfig(effective); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics effective config: %w", err))
}
if err := runDir.WriteResolvedPipeline(effective.ResolvedPipeline); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics resolved pipeline: %w", err))
}
profileIDs := effectiveLLMProfileIDs(effective.ResolvedPipeline)
if len(profileIDs) != 1 {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("pipeline %q uses %d distinct LLM profiles; current runs require exactly one: %s", pipelineID, len(profileIDs), strings.Join(profileIDs, ", ")))
}
rawInput, err := os.ReadFile(strings.TrimSpace(*inputPath))
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("read input %q: %w", strings.TrimSpace(*inputPath), err))
}
registries, err := effectiveRegistries(opts)
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
}
ctx := context.Background()
llmClient, llmProfiles, err := opts.LLMClientFactory(ctx, effective.Config, profileIDs[0])
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("create LLM client for profile %q: %w", profileIDs[0], err))
}
output, err := pipeline.New(registries).Run(ctx, pipeline.RunInput{
Pipeline: effective.ResolvedPipeline,
Path: strings.TrimSpace(*inputPath),
RawInput: rawInput,
LLMClient: llmClient,
RunID: runDir.RunID(),
StartedAt: startedAt,
LLMProfiles: llmProfiles,
Metadata: runMetadata(*outputDir, *diagnosticsDir),
})
if err != nil {
if output.Manifest.PipelineID != "" {
_ = runDir.WriteRunManifest(output.Manifest)
}
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("run pipeline %q: %w", pipelineID, err))
}
runOutputDir := filepath.Join(outputRoot(*outputDir), runDir.RunID())
if err := runDir.WriteRunManifest(output.Manifest); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics run manifest: %w", err))
}
if err := runDir.WriteWarnings(output.Warnings); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics warnings: %w", err))
}
if err := runDir.WriteRunReport(runReport{
RunID: runDir.RunID(),
PipelineID: effective.PipelineID,
OutputPath: runOutputDir,
DiagnosticsPath: runDir.Path(),
ApprovedCount: len(output.Approved),
RejectedCount: len(output.Rejected),
WarningCount: len(output.Warnings),
ValidationStatus: output.Manifest.ValidationStatus,
}); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics run report: %w", err))
}
if err := writeOutputFiles(runOutputDir, output.OutputFiles); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
}
if err := runDir.ApplyRetention(diagnostics.RetentionDecisionInput{
RetentionMode: cfg.Diagnostics.Retention,
RunSucceeded: true,
HasWarnings: len(output.Warnings) > 0,
}); err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("apply diagnostics retention: %w", err))
}
fmt.Fprintf(stdout, "pipeline %q complete: approved=%d rejected=%d output=%s\n", effective.PipelineID, len(output.Approved), len(output.Rejected), runOutputDir)
if len(output.Warnings) > 0 {
fmt.Fprintf(stderr, "notarius: run completed with %d warning(s)\n", len(output.Warnings))
}
return 0
}
type runReport struct {
RunID string `json:"run_id"`
PipelineID string `json:"pipeline_id"`
OutputPath string `json:"output_path"`
DiagnosticsPath string `json:"diagnostics_path,omitempty"`
ApprovedCount int `json:"approved_count"`
RejectedCount int `json:"rejected_count"`
WarningCount int `json:"warning_count"`
ValidationStatus string `json:"validation_status,omitempty"`
}
func failPipelineCommand(stderr io.Writer, runDir *diagnostics.RunDirectory, retention diagnostics.RetentionMode, err error) int {
fmt.Fprintf(stderr, "notarius: %v\n", err)
if runDir != nil {
if logErr := runDir.WriteErrorLog(err.Error()); logErr != nil {
fmt.Fprintf(stderr, "notarius: write diagnostics error log: %v\n", logErr)
}
if retentionErr := runDir.ApplyRetention(diagnostics.RetentionDecisionInput{
RetentionMode: retention,
RunSucceeded: false,
}); retentionErr != nil {
fmt.Fprintf(stderr, "notarius: apply diagnostics retention: %v\n", retentionErr)
}
}
return 1
}
func configSource(configPath string) string {
if strings.TrimSpace(configPath) != "" {
return "flag"
}
return "discovered"
}
func outputRoot(outputDir string) string {
if dir := strings.TrimSpace(outputDir); dir != "" {
return dir
}
return defaultOutputRoot
}
func writeOutputFiles(runOutputDir string, files []contracts.OutputFile) error {
type outputTarget struct {
path string
file contracts.OutputFile
}
targets := make([]outputTarget, 0, len(files))
for _, file := range files {
targetPath, err := outputFilePath(runOutputDir, file.Name)
if err != nil {
return err
}
targets = append(targets, outputTarget{path: targetPath, file: file})
}
if err := os.MkdirAll(runOutputDir, 0o755); err != nil {
return fmt.Errorf("create output directory %q: %w", runOutputDir, err)
}
for _, target := range targets {
if err := os.MkdirAll(filepath.Dir(target.path), 0o755); err != nil {
return fmt.Errorf("create output directory %q: %w", filepath.Dir(target.path), err)
}
if err := writeFileAtomic(target.path, target.file.Bytes, 0o644); err != nil {
return fmt.Errorf("write output file %q: %w", target.file.Name, err)
}
}
return nil
}
func outputFilePath(runOutputDir, logicalName string) (string, error) {
name := strings.TrimSpace(logicalName)
if name == "" {
return "", fmt.Errorf("output file name must not be empty")
}
if strings.Contains(name, `\`) {
return "", fmt.Errorf("output file name %q must use slash-separated relative paths", name)
}
if path.IsAbs(name) || filepath.IsAbs(name) {
return "", fmt.Errorf("output file name %q must be relative", name)
}
if strings.Contains(name, "..") {
return "", fmt.Errorf("output file name %q must not contain ..", name)
}
cleaned := path.Clean(name)
if cleaned == "." || cleaned != name {
return "", fmt.Errorf("output file name %q must be clean", name)
}
root, err := filepath.Abs(runOutputDir)
if err != nil {
return "", fmt.Errorf("resolve output directory %q: %w", runOutputDir, err)
}
target, err := filepath.Abs(filepath.Join(root, filepath.FromSlash(cleaned)))
if err != nil {
return "", fmt.Errorf("resolve output file %q: %w", name, err)
}
rel, err := filepath.Rel(root, target)
if err != nil {
return "", fmt.Errorf("resolve output file %q: %w", name, err)
}
if rel == "." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || rel == ".." {
return "", fmt.Errorf("output file name %q resolves outside output directory", name)
}
return target, nil
}
func writeFileAtomic(path string, data []byte, perm os.FileMode) error {
dir := filepath.Dir(path)
temp, err := os.CreateTemp(dir, "."+filepath.Base(path)+".tmp-*")
if err != nil {
return err
}
tempPath := temp.Name()
removeTemp := true
defer func() {
if removeTemp {
_ = os.Remove(tempPath)
}
}()
if _, err := temp.Write(data); err != nil {
_ = temp.Close()
return err
}
if err := temp.Chmod(perm); err != nil {
_ = temp.Close()
return err
}
if err := temp.Close(); err != nil {
return err
}
if err := os.Rename(tempPath, path); err != nil {
return err
}
removeTemp = false
return nil
}
func reorderRunArgs(args []string) []string {
var flags []string
var positionals []string
for i := 0; i < len(args); i++ {
arg := args[i]
if arg == "--" {
positionals = append(positionals, args[i+1:]...)
break
}
if strings.HasPrefix(arg, "-") {
flags = append(flags, arg)
if runFlagTakesValue(arg) && !strings.Contains(arg, "=") && i+1 < len(args) {
i++
flags = append(flags, args[i])
}
continue
}
positionals = append(positionals, arg)
}
return append(flags, positionals...)
}
func runFlagTakesValue(arg string) bool {
switch arg {
case "--config", "--input", "--only", "--output-dir", "--diagnostics-dir", "--llm-profile":
return true
default:
return false
}
}
func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
seen := make(map[string]struct{})
add := func(binding pipeline.ModuleBinding) {
id := strings.TrimSpace(binding.LLMProfile)
if id != "" {
seen[id] = struct{}{}
}
}
add(resolved.Input)
add(resolved.Chunk)
add(resolved.Output)
for _, lane := range resolved.ArtifactLanes {
add(lane.Extract)
add(lane.Merge)
add(lane.Normalize)
for _, validator := range lane.Validators {
add(validator)
}
}
ids := make([]string, 0, len(seen))
for id := range seen {
ids = append(ids, id)
}
sort.Strings(ids)
return ids
}
func runMetadata(outputDir, diagnosticsDir string) map[string]any {
metadata := make(map[string]any)
if dir := strings.TrimSpace(outputDir); dir != "" {
metadata["output_dir"] = dir
}
if dir := strings.TrimSpace(diagnosticsDir); dir != "" {
metadata["diagnostics_dir"] = dir
}
if len(metadata) == 0 {
return nil
}
return metadata
}
func runConfig(args []string, stdout, stderr io.Writer, opts Options) int {
if len(args) == 0 {
fmt.Fprintln(stderr, "notarius: config requires a subcommand")
@@ -111,10 +507,15 @@ func runConfigValidate(args []string, stdout, stderr io.Writer, opts Options) in
}
if strings.TrimSpace(*pipelineID) != "" {
catalog, err := effectiveCatalog(opts)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
if _, err := cfg.Resolve(config.ResolveInput{
PipelineID: *pipelineID,
Only: only,
Catalog: opts.Catalog,
Catalog: catalog,
}); err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,6 @@
{
"metadata": {
"id": "session-alpha"
},
"segments": []
}

View File

@@ -34,11 +34,18 @@ type RejectedArtifact struct {
}
type ArtifactLaneManifest struct {
ID string `json:"id"`
Extractor string `json:"extractor"`
Merger string `json:"merger"`
Normalizer string `json:"normalizer"`
Validators []string `json:"validators,omitempty"`
ID string `json:"id"`
Extractor string `json:"extractor"`
Merger string `json:"merger"`
Normalizer string `json:"normalizer"`
Validators []string `json:"validators,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type LLMProfileManifest struct {
ID string `json:"id"`
Provider string `json:"provider,omitempty"`
Model string `json:"model,omitempty"`
}
type RunManifest struct {
@@ -53,6 +60,7 @@ type RunManifest struct {
Normalizer string `json:"normalizer,omitempty"`
OutputEncoder string `json:"output_encoder,omitempty"`
ArtifactLanes []ArtifactLaneManifest `json:"artifact_lanes,omitempty"`
LLMProfiles []LLMProfileManifest `json:"llm_profiles,omitempty"`
SchemaVersion string `json:"schema_version,omitempty"`
ValidationStatus string `json:"validation_status,omitempty"`
StartedAt *time.Time `json:"started_at,omitempty"`

View File

@@ -127,6 +127,9 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
manifest := RunManifest{
PipelineID: "pipeline-1",
PipelineDigest: "sha256:abc123",
LLMProfiles: []LLMProfileManifest{
{ID: "default", Provider: "openai-compatible", Model: "model-a"},
},
ArtifactLanes: []ArtifactLaneManifest{
{
ID: "events",
@@ -134,6 +137,9 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
Merger: "appendorder",
Normalizer: "noop",
Validators: []string{"grounded"},
Metadata: map[string]any{
"extractor": map[string]any{"prompt_id": "test.prompt"},
},
},
},
}
@@ -148,7 +154,20 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
t.Fatalf("json.Unmarshal() error = %v", err)
}
assertHasKeys(t, got, "pipeline_id", "pipeline_digest", "artifact_lanes")
assertHasKeys(t, got, "pipeline_id", "pipeline_digest", "artifact_lanes", "llm_profiles")
profiles, ok := got["llm_profiles"].([]any)
if !ok {
t.Fatalf("llm_profiles = %#v, want array", got["llm_profiles"])
}
if len(profiles) != 1 {
t.Fatalf("len(llm_profiles) = %d, want 1", len(profiles))
}
profile, ok := profiles[0].(map[string]any)
if !ok {
t.Fatalf("llm_profiles[0] = %#v, want object", profiles[0])
}
assertHasKeys(t, profile, "id", "provider", "model")
lanes, ok := got["artifact_lanes"].([]any)
if !ok {
@@ -161,7 +180,7 @@ func TestRunManifestIncludesPipelineAndArtifactLaneFields(t *testing.T) {
if !ok {
t.Fatalf("artifact_lanes[0] = %#v, want object", lanes[0])
}
assertHasKeys(t, lane, "id", "extractor", "merger", "normalizer", "validators")
assertHasKeys(t, lane, "id", "extractor", "merger", "normalizer", "validators", "metadata")
}
func assertHasKeys(t *testing.T, values map[string]any, keys ...string) {

View File

@@ -10,9 +10,10 @@ import (
)
type ResolveInput struct {
PipelineID string
Only []string
Catalog pipeline.ModuleCatalog
PipelineID string
Only []string
Catalog pipeline.ModuleCatalog
LLMProfileOverride string
}
type EffectiveConfig struct {
@@ -38,6 +39,12 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
}
profile = clonePipelineProfile(profile)
profile.ID = pipelineID
if override := strings.TrimSpace(input.LLMProfileOverride); override != "" {
if !hasLLMProfile(c.LLMProfiles, override) {
return EffectiveConfig{}, fmt.Errorf("LLM profile override %q is not configured", override)
}
applyLLMProfileOverride(&profile, override)
}
resolved, err := pipeline.ResolvePipeline(profile, pipeline.ResolveOptions{Only: input.Only}, input.Catalog)
if err != nil {
@@ -52,6 +59,21 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
}, nil
}
func applyLLMProfileOverride(profile *pipeline.PipelineProfile, profileID string) {
profile.Input.LLMProfile = profileID
profile.Chunk.LLMProfile = profileID
profile.Output.LLMProfile = profileID
for laneID, lane := range profile.Artifacts {
lane.Extract.LLMProfile = profileID
lane.Merge.LLMProfile = profileID
lane.Normalize.LLMProfile = profileID
for i := range lane.Validators {
lane.Validators[i].LLMProfile = profileID
}
profile.Artifacts[laneID] = lane
}
}
func lookupPipelineProfile(profiles map[string]pipeline.PipelineProfile, pipelineID string) (pipeline.PipelineProfile, bool) {
pipelineID = strings.TrimSpace(pipelineID)
for rawID, profile := range profiles {

View File

@@ -118,6 +118,51 @@ func TestResolveDigestChangesWhenEffectiveConfigChanges(t *testing.T) {
}
}
func TestResolveLLMProfileOverrideAppliesBeforeDigest(t *testing.T) {
cfg := validConfig()
cfg.LLMProfiles["runtime"] = LLMProfile{Provider: "openai-compatible"}
base, err := cfg.Resolve(ResolveInput{PipelineID: "example", Catalog: fakeCatalog(t)})
if err != nil {
t.Fatalf("Resolve base: %v", err)
}
effective, err := cfg.Resolve(ResolveInput{
PipelineID: "example",
Catalog: fakeCatalog(t),
LLMProfileOverride: "runtime",
})
if err != nil {
t.Fatalf("Resolve override: %v", err)
}
if base.ResolvedPipeline.Digest == effective.ResolvedPipeline.Digest {
t.Fatalf("expected digest to change after LLM profile override")
}
for _, binding := range resolvedBindings(effective.ResolvedPipeline) {
if binding.LLMProfile != "runtime" {
t.Fatalf("binding profile = %q, want runtime", binding.LLMProfile)
}
}
_, err = cfg.Resolve(ResolveInput{
PipelineID: "example",
Catalog: fakeCatalog(t),
LLMProfileOverride: "missing",
})
if err == nil || !strings.Contains(err.Error(), "LLM profile override") {
t.Fatalf("expected override profile error, got %v", err)
}
}
func resolvedBindings(resolved pipeline.ResolvedPipeline) []pipeline.ModuleBinding {
bindings := []pipeline.ModuleBinding{resolved.Input, resolved.Chunk, resolved.Output}
for _, lane := range resolved.ArtifactLanes {
bindings = append(bindings, lane.Extract, lane.Merge, lane.Normalize)
bindings = append(bindings, lane.Validators...)
}
return bindings
}
func TestOpenAICompatibleClientConfigRejectsIncompleteDefaultProfile(t *testing.T) {
cfg := Default()

View File

@@ -173,7 +173,7 @@ func (r *RunDirectory) WriteErrorLog(errorMessage string) error {
if err != nil {
return err
}
if err := os.WriteFile(path, []byte(errorMessage+"\n"), 0o644); err != nil {
if err := writeFileAtomic(path, []byte(errorMessage+"\n"), 0o644); err != nil {
return fmt.Errorf("write diagnostics artifact %q: %w", ArtifactErrorLog, err)
}
return nil
@@ -193,7 +193,7 @@ func (r *RunDirectory) WriteJSONArtifact(name string, payload any) error {
return fmt.Errorf("marshal diagnostics artifact %q: %w", name, err)
}
data = append(data, '\n')
if err := os.WriteFile(path, data, 0o644); err != nil {
if err := writeFileAtomic(path, data, 0o644); err != nil {
return fmt.Errorf("write diagnostics artifact %q: %w", name, err)
}
return nil
@@ -241,3 +241,39 @@ func (r *RunDirectory) artifactPath(name string) (string, error) {
}
return artifactPath, nil
}
func writeFileAtomic(path string, data []byte, perm os.FileMode) error {
dir := filepath.Dir(path)
if err := os.MkdirAll(dir, 0o755); err != nil {
return err
}
temp, err := os.CreateTemp(dir, "."+filepath.Base(path)+".tmp-*")
if err != nil {
return err
}
tempPath := temp.Name()
removeTemp := true
defer func() {
if removeTemp {
_ = os.Remove(tempPath)
}
}()
if _, err := temp.Write(data); err != nil {
_ = temp.Close()
return err
}
if err := temp.Chmod(perm); err != nil {
_ = temp.Close()
return err
}
if err := temp.Close(); err != nil {
return err
}
if err := os.Rename(tempPath, path); err != nil {
return err
}
removeTemp = false
return nil
}

View File

@@ -118,6 +118,24 @@ func TestWriteJSONArtifactWritesIndentedNewlineTerminatedJSON(t *testing.T) {
}
}
func TestWriteJSONArtifactLeavesNoTemporaryFiles(t *testing.T) {
runDir := newTestRunDirectory(t)
if err := runDir.WriteJSONArtifact("artifact.json", map[string]any{"value": "ok"}); err != nil {
t.Fatalf("WriteJSONArtifact: %v", err)
}
entries, err := os.ReadDir(runDir.Path())
if err != nil {
t.Fatalf("read run directory: %v", err)
}
for _, entry := range entries {
if strings.Contains(entry.Name(), ".tmp-") {
t.Fatalf("temporary diagnostics file remains after success: %s", entry.Name())
}
}
}
func TestWriteInvocationMetadataFillsMissingRunIDAndStartTime(t *testing.T) {
runDir := newTestRunDirectory(t)

View File

@@ -124,10 +124,13 @@ func TestContractsComposeAcrossPackages(t *testing.T) {
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
if output.ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", output.ContentType)
if len(output.Files) != 1 {
t.Fatalf("len(Files) = %d, want 1", len(output.Files))
}
if len(output.Bytes) == 0 {
if output.Files[0].ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", output.Files[0].ContentType)
}
if len(output.Files[0].Bytes) == 0 {
t.Fatal("len(Bytes) = 0, want encoded bytes")
}
}
@@ -293,7 +296,12 @@ func (encoder compositionOutputEncoder) Encode(ctx context.Context, req contract
}
return contracts.OutputResult{
Bytes: encoded,
ContentType: "application/json",
Files: []contracts.OutputFile{
{
Name: "artifacts/generic.json",
ContentType: "application/json",
Bytes: encoded,
},
},
}, nil
}

View File

@@ -182,13 +182,22 @@ type OutputRequest struct {
Metadata map[string]any `json:"metadata,omitempty"`
}
type OutputFile struct {
Name string `json:"name"`
ContentType string `json:"content_type,omitempty"`
Bytes []byte `json:"-"`
}
type OutputResult struct {
Bytes []byte `json:"-"`
ContentType string `json:"content_type,omitempty"`
Warnings []Warning `json:"warnings,omitempty"`
Files []OutputFile `json:"files,omitempty"`
Warnings []Warning `json:"warnings,omitempty"`
}
type OutputEncoder interface {
Key() string
Encode(ctx context.Context, req OutputRequest) (OutputResult, error)
}
type ManifestMetadataProvider interface {
ManifestMetadata() map[string]any
}

View File

@@ -230,11 +230,44 @@ func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
if encoder.Key() != "generic-output" {
t.Fatalf("OutputEncoder.Key() = %q, want generic-output", encoder.Key())
}
if encoded.ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", encoded.ContentType)
if len(encoded.Files) != 1 {
t.Fatalf("len(Files) = %d, want 1", len(encoded.Files))
}
if string(encoded.Bytes) != `{"run_id":"run-1","approved_count":1}` {
t.Fatalf("Bytes = %s, want encoded output", encoded.Bytes)
if encoded.Files[0].ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", encoded.Files[0].ContentType)
}
if string(encoded.Files[0].Bytes) != `{"run_id":"run-1","approved_count":1}` {
t.Fatalf("Bytes = %s, want encoded output", encoded.Files[0].Bytes)
}
}
func TestOutputFileJSONShapeOmitsBytes(t *testing.T) {
file := OutputFile{
Name: "artifacts/events.json",
ContentType: "application/json",
Bytes: []byte(`{"ignored":true}`),
}
encoded, err := json.Marshal(file)
if err != nil {
t.Fatalf("json.Marshal() error = %v, want nil", err)
}
var got map[string]any
if err := json.Unmarshal(encoded, &got); err != nil {
t.Fatalf("json.Unmarshal() error = %v, want nil", err)
}
if got["name"] != "artifacts/events.json" {
t.Fatalf("name = %#v, want logical file name", got["name"])
}
if got["content_type"] != "application/json" {
t.Fatalf("content_type = %#v, want application/json", got["content_type"])
}
if _, ok := got["Bytes"]; ok {
t.Fatalf("encoded output file leaked Bytes: %s", encoded)
}
if _, ok := got["bytes"]; ok {
t.Fatalf("encoded output file leaked bytes: %s", encoded)
}
}
@@ -397,7 +430,12 @@ func (encoder fakeOutputEncoder) Key() string {
func (encoder fakeOutputEncoder) Encode(ctx context.Context, req OutputRequest) (OutputResult, error) {
return OutputResult{
Bytes: []byte(`{"run_id":"` + req.Manifest.RunID + `","approved_count":1}`),
ContentType: "application/json",
Files: []OutputFile{
{
Name: "artifacts/generic.json",
ContentType: "application/json",
Bytes: []byte(`{"run_id":"` + req.Manifest.RunID + `","approved_count":1}`),
},
},
}, nil
}

View File

@@ -0,0 +1,43 @@
package llm
import (
"context"
"fmt"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type scheduledClient struct {
client contracts.StructuredLLMClient
scheduler *Scheduler
}
func NewScheduledClient(client contracts.StructuredLLMClient, scheduler *Scheduler) contracts.StructuredLLMClient {
return &scheduledClient{
client: client,
scheduler: scheduler,
}
}
func (c *scheduledClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
if c == nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("scheduled LLM client must not be nil")
}
if c.client == nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("scheduled LLM client inner client must not be nil")
}
if c.scheduler == nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("scheduled LLM client scheduler must not be nil")
}
var response contracts.StructuredCompletionResponse
err := c.scheduler.Run(ctx, func(ctx context.Context) error {
var callErr error
response, callErr = c.client.CompleteStructured(ctx, req, out)
return callErr
})
if err != nil {
return contracts.StructuredCompletionResponse{}, err
}
return response, nil
}

View File

@@ -0,0 +1,115 @@
package llm
import (
"context"
"encoding/json"
"errors"
"sync"
"sync/atomic"
"testing"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestScheduledClientEnforcesSchedulerLimit(t *testing.T) {
scheduler, err := NewScheduler(1)
if err != nil {
t.Fatalf("NewScheduler() error = %v, want nil", err)
}
inner := &blockingStructuredClient{
release: make(chan struct{}),
}
client := NewScheduledClient(inner, scheduler)
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func() {
defer wg.Done()
var out map[string]any
if _, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{}, &out); err != nil {
t.Errorf("CompleteStructured() error = %v, want nil", err)
}
}()
}
waitForAtomicAtLeast(t, &inner.calls, 1)
time.Sleep(20 * time.Millisecond)
if got := atomic.LoadInt32(&inner.maxInFlight); got > 1 {
t.Fatalf("max in-flight calls = %d, want <= 1", got)
}
close(inner.release)
wg.Wait()
if got := atomic.LoadInt32(&inner.calls); got != 3 {
t.Fatalf("calls = %d, want 3", got)
}
}
func TestScheduledClientPropagatesClientError(t *testing.T) {
scheduler, err := NewScheduler(1)
if err != nil {
t.Fatalf("NewScheduler() error = %v, want nil", err)
}
expected := errors.New("provider unavailable")
client := NewScheduledClient(&errorStructuredClient{err: expected}, scheduler)
_, err = client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{}, &struct{}{})
if !errors.Is(err, expected) {
t.Fatalf("CompleteStructured() error = %v, want %v", err, expected)
}
}
func TestScheduledClientPropagatesSchedulerError(t *testing.T) {
scheduler, err := NewScheduler(1)
if err != nil {
t.Fatalf("NewScheduler() error = %v, want nil", err)
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
client := NewScheduledClient(&errorStructuredClient{}, scheduler)
_, err = client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{}, &struct{}{})
if !errors.Is(err, context.Canceled) {
t.Fatalf("CompleteStructured() error = %v, want context canceled", err)
}
}
type blockingStructuredClient struct {
release chan struct{}
inFlight int32
maxInFlight int32
calls int32
}
func (c *blockingStructuredClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
atomic.AddInt32(&c.calls, 1)
current := atomic.AddInt32(&c.inFlight, 1)
for {
seen := atomic.LoadInt32(&c.maxInFlight)
if current <= seen || atomic.CompareAndSwapInt32(&c.maxInFlight, seen, current) {
break
}
}
defer atomic.AddInt32(&c.inFlight, -1)
select {
case <-c.release:
case <-ctx.Done():
return contracts.StructuredCompletionResponse{}, ctx.Err()
}
if target, ok := out.(*map[string]any); ok {
*target = map[string]any{"ok": true}
}
return contracts.StructuredCompletionResponse{
Content: json.RawMessage(`{"ok":true}`),
}, nil
}
type errorStructuredClient struct {
err error
}
func (c *errorStructuredClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
return contracts.StructuredCompletionResponse{}, c.err
}

View File

@@ -6,6 +6,7 @@ import (
"encoding/hex"
"encoding/json"
"fmt"
"io/fs"
"sort"
"strings"
)
@@ -17,13 +18,21 @@ var schemaAssets embed.FS
type ResponseSchemaKey string
const (
DNDSpellsSchemaKey ResponseSchemaKey = "dnd_spells"
TestArtifactSchemaKey ResponseSchemaKey = "test_artifact"
TestValidatorDecisionSchemaKey ResponseSchemaKey = "test_validator_decision"
schemaVersionV1 = "v1"
)
// ResponseSchemaDefinition identifies a caller-owned structured response schema asset.
type ResponseSchemaDefinition struct {
Key ResponseSchemaKey
ID string
Version string
Name string
AssetPath string
}
// ResponseSchema describes one registered structured response schema.
type ResponseSchema struct {
Key ResponseSchemaKey `json:"key"`
@@ -35,27 +44,20 @@ type ResponseSchema struct {
}
var responseSchemaRegistry = map[ResponseSchemaKey]ResponseSchema{
DNDSpellsSchemaKey: mustLoadResponseSchema(
DNDSpellsSchemaKey,
"notarius.dnd.spells",
schemaVersionV1,
"notarius_dnd_spells_v1",
"assets/schemas/dnd_spells.v1.json",
),
TestArtifactSchemaKey: mustLoadResponseSchema(
TestArtifactSchemaKey,
"notarius.test_artifact",
schemaVersionV1,
"notarius_test_artifact_v1",
"assets/schemas/test_artifact.v1.json",
),
TestValidatorDecisionSchemaKey: mustLoadResponseSchema(
TestValidatorDecisionSchemaKey,
"notarius.test_validator_decision",
schemaVersionV1,
"notarius_test_validator_decision_v1",
"assets/schemas/test_validator_decision.v1.json",
),
TestArtifactSchemaKey: mustLoadResponseSchema(schemaAssets, ResponseSchemaDefinition{
Key: TestArtifactSchemaKey,
ID: "notarius.test_artifact",
Version: schemaVersionV1,
Name: "notarius_test_artifact_v1",
AssetPath: "assets/schemas/test_artifact.v1.json",
}),
TestValidatorDecisionSchemaKey: mustLoadResponseSchema(schemaAssets, ResponseSchemaDefinition{
Key: TestValidatorDecisionSchemaKey,
ID: "notarius.test_validator_decision",
Version: schemaVersionV1,
Name: "notarius_test_validator_decision_v1",
AssetPath: "assets/schemas/test_validator_decision.v1.json",
}),
}
// RegisteredResponseSchemas returns all registered response schemas sorted by key.
@@ -102,40 +104,35 @@ func (s ResponseSchema) DiagnosticsMap() map[string]any {
}
}
func mustLoadResponseSchema(
key ResponseSchemaKey,
id string,
version string,
name string,
path string,
) ResponseSchema {
key = ResponseSchemaKey(strings.TrimSpace(string(key)))
id = strings.TrimSpace(id)
version = strings.TrimSpace(version)
name = strings.TrimSpace(name)
path = strings.TrimSpace(path)
// LoadResponseSchema loads a structured response schema from a caller-owned filesystem.
func LoadResponseSchema(fsys fs.FS, def ResponseSchemaDefinition) (ResponseSchema, error) {
key := ResponseSchemaKey(strings.TrimSpace(string(def.Key)))
id := strings.TrimSpace(def.ID)
version := strings.TrimSpace(def.Version)
name := strings.TrimSpace(def.Name)
path := strings.TrimSpace(def.AssetPath)
if key == "" {
panic("response schema key must not be empty")
return ResponseSchema{}, fmt.Errorf("response schema key must not be empty")
}
if id == "" {
panic("response schema id must not be empty")
return ResponseSchema{}, fmt.Errorf("response schema id must not be empty")
}
if version == "" {
panic("response schema version must not be empty")
return ResponseSchema{}, fmt.Errorf("response schema version must not be empty")
}
if name == "" {
panic("response schema name must not be empty")
return ResponseSchema{}, fmt.Errorf("response schema name must not be empty")
}
if path == "" {
panic("response schema asset path must not be empty")
return ResponseSchema{}, fmt.Errorf("response schema asset path must not be empty")
}
rawSchema, err := schemaAssets.ReadFile(path)
rawSchema, err := fs.ReadFile(fsys, path)
if err != nil {
panic(fmt.Sprintf("read response schema %s: %v", path, err))
return ResponseSchema{}, fmt.Errorf("read response schema %s: %w", path, err)
}
if !json.Valid(rawSchema) {
panic(fmt.Sprintf("response schema %s is not valid JSON", path))
return ResponseSchema{}, fmt.Errorf("response schema %s is not valid JSON", path)
}
hash := sha256.Sum256(rawSchema)
@@ -146,7 +143,15 @@ func mustLoadResponseSchema(
Name: name,
JSONSchema: append(json.RawMessage(nil), rawSchema...),
SHA256: "sha256:" + hex.EncodeToString(hash[:]),
}, nil
}
func mustLoadResponseSchema(fsys fs.FS, def ResponseSchemaDefinition) ResponseSchema {
schema, err := LoadResponseSchema(fsys, def)
if err != nil {
panic(err)
}
return schema
}
func cloneResponseSchema(in ResponseSchema) ResponseSchema {

View File

@@ -9,7 +9,6 @@ import (
func TestLookupResponseSchemaSucceedsForRegisteredSchemas(t *testing.T) {
tests := []ResponseSchemaKey{
DNDSpellsSchemaKey,
TestArtifactSchemaKey,
TestValidatorDecisionSchemaKey,
}
@@ -39,6 +38,12 @@ func TestLookupResponseSchemaUnknownReturnsFalse(t *testing.T) {
}
}
func TestLookupResponseSchemaDNDSpellsIsNotFrameworkRegistered(t *testing.T) {
if schema, ok := LookupResponseSchema("dnd_spells"); ok {
t.Fatalf("expected D&D spells schema lookup to fail in framework registry, got %+v", schema)
}
}
func TestMustLookupResponseSchemaPanicsForUnknownKey(t *testing.T) {
defer func() {
if recover() == nil {
@@ -51,8 +56,8 @@ func TestMustLookupResponseSchemaPanicsForUnknownKey(t *testing.T) {
func TestRegisteredResponseSchemasSortedByKey(t *testing.T) {
schemas := RegisteredResponseSchemas()
if len(schemas) != 3 {
t.Fatalf("expected three schemas, got %d", len(schemas))
if len(schemas) != 2 {
t.Fatalf("expected two schemas, got %d", len(schemas))
}
keys := make([]string, len(schemas))
@@ -64,8 +69,8 @@ func TestRegisteredResponseSchemasSortedByKey(t *testing.T) {
if !sort.StringsAreSorted(keys) {
t.Fatalf("expected sorted keys, got %v", keys)
}
if !seen[DNDSpellsSchemaKey] {
t.Fatalf("registered schemas = %v, want %q", keys, DNDSpellsSchemaKey)
if !seen[TestArtifactSchemaKey] || !seen[TestValidatorDecisionSchemaKey] {
t.Fatalf("registered schemas = %v, want test schemas", keys)
}
}
@@ -78,7 +83,7 @@ func TestResponseSchemaContentIsValidJSON(t *testing.T) {
}
func TestResponseSchemaJSONIsMutationSafe(t *testing.T) {
for _, key := range []ResponseSchemaKey{DNDSpellsSchemaKey, TestArtifactSchemaKey} {
for _, key := range []ResponseSchemaKey{TestArtifactSchemaKey, TestValidatorDecisionSchemaKey} {
t.Run(string(key), func(t *testing.T) {
first := MustLookupResponseSchema(key)
first.JSONSchema[0] = '['

View File

@@ -0,0 +1,124 @@
package pipeline_test
import (
"context"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
)
func TestPipelineConfigResolvesWithProductionDefaultsRegistered(t *testing.T) {
cfg := config.Default()
cfg.Pipelines = map[string]pipeline.PipelineProfile{
"defaults": {
Input: pipeline.Binding("input"),
Artifacts: map[string]pipeline.ArtifactLaneProfile{
"events": {Extract: pipeline.Binding("extract")},
},
},
}
resolved, err := cfg.Resolve(config.ResolveInput{
PipelineID: "defaults",
Catalog: defaultModuleCatalog(t),
})
if err != nil {
t.Fatalf("Resolve() error = %v, want nil", err)
}
pipeline := resolved.ResolvedPipeline
if pipeline.Chunk.Module != generic.Key {
t.Fatalf("Chunk.Module = %q, want %q", pipeline.Chunk.Module, generic.Key)
}
if pipeline.Output.Module != jsonoutput.Key {
t.Fatalf("Output.Module = %q, want %q", pipeline.Output.Module, jsonoutput.Key)
}
lane := pipeline.ArtifactLanes[0]
if lane.Merge.Module != appendorder.Key {
t.Fatalf("Merge.Module = %q, want %q", lane.Merge.Module, appendorder.Key)
}
if lane.Normalize.Module != noop.Key {
t.Fatalf("Normalize.Module = %q, want %q", lane.Normalize.Module, noop.Key)
}
}
func defaultModuleCatalog(t *testing.T) pipeline.ModuleCatalog {
t.Helper()
inputs := pipeline.NewInputAdapterRegistry()
chunkers := pipeline.NewChunkerRegistry()
extractors := pipeline.NewExtractorRegistry()
mergers := pipeline.NewMergerRegistry()
normalizers := pipeline.NewNormalizerRegistry()
outputs := pipeline.NewOutputEncoderRegistry()
if err := inputs.RegisterWithSpec(pipeline.ModuleSpec{
Key: "input",
Stage: pipeline.StageInput,
Provides: []string{"source"},
}, func() (contracts.InputAdapter, error) {
return defaultInput{}, nil
}); err != nil {
t.Fatalf("register input: %v", err)
}
if err := generic.Register(chunkers); err != nil {
t.Fatalf("register generic chunker: %v", err)
}
if err := extractors.RegisterWithSpec(pipeline.ModuleSpec{
Key: "extract",
Stage: pipeline.StageExtract,
Requires: []string{"chunks"},
Provides: []string{"records"},
}, func() (contracts.Extractor, error) {
return defaultExtractor{}, nil
}); err != nil {
t.Fatalf("register extractor: %v", err)
}
if err := appendorder.Register(mergers); err != nil {
t.Fatalf("register appendorder merger: %v", err)
}
if err := noop.Register(normalizers); err != nil {
t.Fatalf("register noop normalizer: %v", err)
}
if err := jsonoutput.Register(outputs); err != nil {
t.Fatalf("register json output: %v", err)
}
return pipeline.ModuleCatalog{
Inputs: inputs,
Chunkers: chunkers,
Extractors: extractors,
Mergers: mergers,
Normalizers: normalizers,
Outputs: outputs,
}
}
type defaultInput struct{}
func (defaultInput) Key() string { return "input" }
func (defaultInput) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
return nil, nil
}
type defaultExtractor struct{}
func (defaultExtractor) Key() string { return "extract" }
func (defaultExtractor) ArtifactType() string { return "record" }
func (defaultExtractor) SchemaVersion() string { return "v1" }
func (defaultExtractor) Validators() []contracts.Validator { return nil }
func (defaultExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
return contracts.ExtractionResult{}, nil
}

View File

@@ -1,70 +0,0 @@
package pipeline
import (
"context"
"encoding/json"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type AppendOrderMerger struct{}
func (m AppendOrderMerger) Key() string {
return DefaultMergeModule
}
func (m AppendOrderMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
var candidates []artifacts.ArtifactCandidate
for _, chunkArtifacts := range req.ChunkArtifacts {
candidates = append(candidates, copyArtifactCandidates(chunkArtifacts.Candidates)...)
}
return contracts.MergeResult{Candidates: candidates}, nil
}
type NoopNormalizer struct{}
func (n NoopNormalizer) Key() string {
return DefaultNormalizeModule
}
func (n NoopNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
return contracts.NormalizeResult{Candidates: copyArtifactCandidates(req.Candidates)}, nil
}
func copyArtifactCandidates(candidates []artifacts.ArtifactCandidate) []artifacts.ArtifactCandidate {
if len(candidates) == 0 {
return nil
}
copied := make([]artifacts.ArtifactCandidate, 0, len(candidates))
for _, candidate := range candidates {
copied = append(copied, copyArtifactCandidate(candidate))
}
return copied
}
func copyArtifactCandidate(candidate artifacts.ArtifactCandidate) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: candidate.Index,
ExtractorKey: candidate.ExtractorKey,
ArtifactType: candidate.ArtifactType,
SchemaVersion: candidate.SchemaVersion,
Payload: append(json.RawMessage(nil), candidate.Payload...),
SourceRefs: append([]source.SourceRef(nil), candidate.SourceRefs...),
Metadata: copyArtifactMetadata(candidate.Metadata),
}
}
func copyArtifactMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
copied := make(map[string]any, len(metadata))
for key, value := range metadata {
copied[key] = value
}
return copied
}

View File

@@ -1,221 +0,0 @@
package pipeline
import (
"context"
"encoding/json"
"reflect"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestGenericMergeAndNormalizeKeys(t *testing.T) {
merger := AppendOrderMerger{}
normalizer := NoopNormalizer{}
if merger.Key() != DefaultMergeModule {
t.Fatalf("AppendOrderMerger.Key() = %q, want %q", merger.Key(), DefaultMergeModule)
}
if normalizer.Key() != DefaultNormalizeModule {
t.Fatalf("NoopNormalizer.Key() = %q, want %q", normalizer.Key(), DefaultNormalizeModule)
}
}
func TestAppendOrderMergerConcatenatesByChunkAndCandidateOrder(t *testing.T) {
merger := AppendOrderMerger{}
chunks := []contracts.ChunkArtifacts{
{
Chunk: sourceChunk(0),
Candidates: []artifacts.ArtifactCandidate{
candidate(2, "first-b"),
candidate(1, "first-a"),
},
},
{
Chunk: sourceChunk(1),
Candidates: []artifacts.ArtifactCandidate{
candidate(4, "second-b"),
candidate(3, "second-a"),
},
},
}
result, err := merger.Merge(context.Background(), contracts.MergeRequest{ChunkArtifacts: chunks})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
}
got := candidateNames(result.Candidates)
want := []string{"first-b", "first-a", "second-b", "second-a"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("candidate order = %#v, want %#v", got, want)
}
}
func TestAppendOrderMergerReturnsMutationSafeCandidates(t *testing.T) {
merger := AppendOrderMerger{}
input := []contracts.ChunkArtifacts{
{
Chunk: sourceChunk(0),
Candidates: []artifacts.ArtifactCandidate{
candidate(1, "original"),
},
},
}
result, err := merger.Merge(context.Background(), contracts.MergeRequest{ChunkArtifacts: input})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
if len(result.Candidates) != 1 {
t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
}
input[0].Candidates[0].Index = 99
input[0].Candidates[0].Payload[0] = '['
input[0].Candidates[0].SourceRefs[0].StartUnitID = "changed"
input[0].Candidates[0].Metadata["name"] = "changed"
got := result.Candidates[0]
if got.Index != 1 {
t.Fatalf("Index = %d, want 1", got.Index)
}
if string(got.Payload) != `{"name":"original"}` {
t.Fatalf("Payload = %s, want original payload", got.Payload)
}
if got.SourceRefs[0].StartUnitID != "u1" {
t.Fatalf("SourceRefs = %#v, want original source ref", got.SourceRefs)
}
if got.Metadata["name"] != "original" {
t.Fatalf("Metadata = %#v, want original metadata", got.Metadata)
}
}
func TestNoopNormalizerPreservesOrderAndValues(t *testing.T) {
normalizer := NoopNormalizer{}
input := []artifacts.ArtifactCandidate{
candidate(3, "third"),
candidate(1, "first"),
candidate(2, "second"),
}
result, err := normalizer.Normalize(context.Background(), contracts.NormalizeRequest{Candidates: input})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
}
got := candidateNames(result.Candidates)
want := []string{"third", "first", "second"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("candidate order = %#v, want %#v", got, want)
}
if !reflect.DeepEqual(result.Candidates[0].SourceRefs, input[0].SourceRefs) {
t.Fatalf("SourceRefs = %#v, want %#v", result.Candidates[0].SourceRefs, input[0].SourceRefs)
}
if !reflect.DeepEqual(result.Candidates[0].Metadata, input[0].Metadata) {
t.Fatalf("Metadata = %#v, want %#v", result.Candidates[0].Metadata, input[0].Metadata)
}
}
func TestNoopNormalizerReturnsMutationSafeCandidates(t *testing.T) {
normalizer := NoopNormalizer{}
input := []artifacts.ArtifactCandidate{candidate(1, "original")}
result, err := normalizer.Normalize(context.Background(), contracts.NormalizeRequest{Candidates: input})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(result.Candidates) != 1 {
t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
}
input[0].Index = 99
input[0].Payload[0] = '['
input[0].SourceRefs[0].EndUnitID = "changed"
input[0].Metadata["name"] = "changed"
got := result.Candidates[0]
if got.Index != 1 {
t.Fatalf("Index = %d, want 1", got.Index)
}
if string(got.Payload) != `{"name":"original"}` {
t.Fatalf("Payload = %s, want original payload", got.Payload)
}
if got.SourceRefs[0].EndUnitID != "u1" {
t.Fatalf("SourceRefs = %#v, want original source ref", got.SourceRefs)
}
if got.Metadata["name"] != "original" {
t.Fatalf("Metadata = %#v, want original metadata", got.Metadata)
}
}
func TestGenericMergeAndNormalizeHandleEmptyInput(t *testing.T) {
merger := AppendOrderMerger{}
normalizer := NoopNormalizer{}
mergeResult, err := merger.Merge(context.Background(), contracts.MergeRequest{})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
if len(mergeResult.Candidates) != 0 {
t.Fatalf("len(mergeResult.Candidates) = %d, want 0", len(mergeResult.Candidates))
}
if len(mergeResult.Warnings) != 0 {
t.Fatalf("merge warnings = %#v, want none", mergeResult.Warnings)
}
normalizeResult, err := normalizer.Normalize(context.Background(), contracts.NormalizeRequest{})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(normalizeResult.Candidates) != 0 {
t.Fatalf("len(normalizeResult.Candidates) = %d, want 0", len(normalizeResult.Candidates))
}
if len(normalizeResult.Warnings) != 0 {
t.Fatalf("normalize warnings = %#v, want none", normalizeResult.Warnings)
}
}
func candidate(index int, name string) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: index,
ExtractorKey: "generic-extractor",
ArtifactType: "generic-artifact",
SchemaVersion: "v1",
Payload: json.RawMessage(`{"name":"` + name + `"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
},
Metadata: map[string]any{
"name": name,
},
}
}
func candidateNames(candidates []artifacts.ArtifactCandidate) []string {
names := make([]string, 0, len(candidates))
for _, candidate := range candidates {
names = append(names, candidate.Metadata["name"].(string))
}
return names
}
func sourceChunk(index int) contracts.SourceChunk {
return contracts.SourceChunk{
ID: "chunk",
SourceID: "source-1",
Index: index,
Units: []source.SourceUnit{
{ID: "u1", Kind: "unit", Text: "Source unit."},
},
}
}

View File

@@ -193,7 +193,11 @@ func (output integrationOutput) Key() string {
}
func (output integrationOutput) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
return contracts.OutputResult{Bytes: []byte(`{}`), ContentType: "application/json"}, nil
return contracts.OutputResult{
Files: []contracts.OutputFile{
{Name: "output.json", ContentType: "application/json", Bytes: []byte(`{}`)},
},
}, nil
}
type integrationValidator struct {

View File

@@ -3,6 +3,9 @@ package pipeline
import (
"context"
"fmt"
"path"
"strings"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
@@ -29,21 +32,23 @@ func New(registries Registries) *Runner {
}
type RunInput struct {
Pipeline ResolvedPipeline
SourceID string
Path string
RawInput []byte
LLMClient contracts.StructuredLLMClient
Metadata map[string]any
Pipeline ResolvedPipeline
SourceID string
Path string
RawInput []byte
LLMClient contracts.StructuredLLMClient
RunID string
StartedAt time.Time
LLMProfiles []artifacts.LLMProfileManifest
Metadata map[string]any
}
type RunOutput struct {
Manifest artifacts.RunManifest `json:"manifest"`
Approved []artifacts.Artifact `json:"approved,omitempty"`
Rejected []artifacts.RejectedArtifact `json:"rejected,omitempty"`
Warnings []contracts.Warning `json:"warnings,omitempty"`
EncodedOutput []byte `json:"-"`
ContentType string `json:"content_type,omitempty"`
Manifest artifacts.RunManifest `json:"manifest"`
Approved []artifacts.Artifact `json:"approved,omitempty"`
Rejected []artifacts.RejectedArtifact `json:"rejected,omitempty"`
Warnings []contracts.Warning `json:"warnings,omitempty"`
OutputFiles []contracts.OutputFile `json:"-"`
}
func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error) {
@@ -58,7 +63,7 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error) {
return output, err
}
output.Manifest = manifestFromPipeline(input.Pipeline)
output.Manifest = manifestFromPipeline(input)
adapter, err := r.registries.Inputs.Build(input.Pipeline.Input.Module)
if err != nil {
@@ -110,6 +115,7 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error) {
} else {
output.Manifest.ValidationStatus = "approved"
}
output.Manifest.CompletedAt = timePtr(time.Now().UTC())
encoder, err := r.registries.Outputs.Build(input.Pipeline.Output.Module)
if err != nil {
@@ -128,8 +134,11 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (RunOutput, error) {
if err != nil {
return failOutput(output), fmt.Errorf("encode output with encoder %q: %w", encoder.Key(), err)
}
output.EncodedOutput = encoded.Bytes
output.ContentType = encoded.ContentType
files, err := outputFilesFromResult(encoded)
if err != nil {
return failOutput(output), fmt.Errorf("validate output files from encoder %q: %w", encoder.Key(), err)
}
output.OutputFiles = files
return output, nil
}
@@ -147,6 +156,7 @@ func (r *Runner) runLane(ctx context.Context, input RunInput, doc *source.Source
if err != nil {
return fmt.Errorf("build normalizer %q for lane %q: %w", lane.Normalize.Module, lane.ID, err)
}
setLaneManifestMetadata(output, lane.ID, extractor, merger, normalizer)
var validators []validatorExecution
if len(lane.Validators) > 0 {
@@ -315,7 +325,17 @@ func validateRunInput(input RunInput) error {
return nil
}
func manifestFromPipeline(pipeline ResolvedPipeline) artifacts.RunManifest {
func manifestFromPipeline(input RunInput) artifacts.RunManifest {
startedAt := input.StartedAt
if startedAt.IsZero() {
startedAt = time.Now().UTC()
}
runID := strings.TrimSpace(input.RunID)
if runID == "" {
runID = fmt.Sprintf("run-%d", startedAt.UnixNano())
}
pipeline := input.Pipeline
manifest := artifacts.RunManifest{
PipelineID: pipeline.ID,
PipelineDigest: pipeline.Digest,
@@ -323,6 +343,9 @@ func manifestFromPipeline(pipeline ResolvedPipeline) artifacts.RunManifest {
Chunker: pipeline.Chunk.Module,
OutputEncoder: pipeline.Output.Module,
ArtifactLanes: make([]artifacts.ArtifactLaneManifest, 0, len(pipeline.ArtifactLanes)),
RunID: runID,
StartedAt: timePtr(startedAt),
LLMProfiles: cloneLLMProfiles(input.LLMProfiles),
}
for _, lane := range pipeline.ArtifactLanes {
@@ -343,10 +366,113 @@ func manifestFromPipeline(pipeline ResolvedPipeline) artifacts.RunManifest {
func failOutput(output RunOutput) RunOutput {
if output.Manifest.PipelineID != "" {
output.Manifest.ValidationStatus = "failed"
output.Manifest.CompletedAt = timePtr(time.Now().UTC())
}
return output
}
func setLaneManifestMetadata(output *RunOutput, laneID string, modules ...any) {
if output == nil {
return
}
for i := range output.Manifest.ArtifactLanes {
if output.Manifest.ArtifactLanes[i].ID != laneID {
continue
}
metadata := make(map[string]any)
for _, module := range modules {
provider, ok := module.(contracts.ManifestMetadataProvider)
if !ok {
continue
}
moduleMetadata := cloneMetadata(provider.ManifestMetadata())
if len(moduleMetadata) == 0 {
continue
}
key := manifestMetadataKey(module)
if key == "" {
continue
}
metadata[key] = moduleMetadata
}
if len(metadata) > 0 {
output.Manifest.ArtifactLanes[i].Metadata = metadata
}
return
}
}
func manifestMetadataKey(module any) string {
switch module.(type) {
case contracts.Extractor:
return "extractor"
case contracts.Merger:
return "merger"
case contracts.Normalizer:
return "normalizer"
default:
return ""
}
}
func outputFilesFromResult(result contracts.OutputResult) ([]contracts.OutputFile, error) {
out := make([]contracts.OutputFile, 0, len(result.Files))
for _, file := range result.Files {
if err := validateOutputFileName(file.Name); err != nil {
return nil, err
}
out = append(out, contracts.OutputFile{
Name: file.Name,
ContentType: file.ContentType,
Bytes: append([]byte(nil), file.Bytes...),
})
}
return out, nil
}
func validateOutputFileName(name string) error {
if strings.TrimSpace(name) == "" {
return fmt.Errorf("output file name must not be empty")
}
if strings.Contains(name, "\\") {
return fmt.Errorf("output file name %q must use slash-separated relative paths", name)
}
if path.IsAbs(name) {
return fmt.Errorf("output file name %q must be relative", name)
}
if strings.Contains(name, "..") {
return fmt.Errorf("output file name %q must not contain ..", name)
}
cleaned := path.Clean(name)
if cleaned == "." || cleaned != name {
return fmt.Errorf("output file name %q must be clean", name)
}
return nil
}
func cloneMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
out := make(map[string]any, len(metadata))
for key, value := range metadata {
out[key] = value
}
return out
}
func cloneLLMProfiles(profiles []artifacts.LLMProfileManifest) []artifacts.LLMProfileManifest {
if len(profiles) == 0 {
return nil
}
return append([]artifacts.LLMProfileManifest(nil), profiles...)
}
func timePtr(t time.Time) *time.Time {
return &t
}
func pipelineUsesConfiguredValidators(pipeline ResolvedPipeline) bool {
for _, lane := range pipeline.ArtifactLanes {
if len(lane.Validators) > 0 {

View File

@@ -6,6 +6,7 @@ import (
"reflect"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
@@ -28,18 +29,17 @@ func TestNewAndDataTypes(t *testing.T) {
Metadata: map[string]any{"request": "test"},
}
output := RunOutput{
Manifest: artifacts.RunManifest{PipelineID: "pipeline-1"},
Approved: []artifacts.Artifact{{ExtractorKey: "extract-alpha"}},
Rejected: []artifacts.RejectedArtifact{{ValidatorName: "validator"}},
Warnings: []contracts.Warning{{ReasonCode: "note", Message: "message"}},
EncodedOutput: []byte(`{}`),
ContentType: "application/json",
Manifest: artifacts.RunManifest{PipelineID: "pipeline-1"},
Approved: []artifacts.Artifact{{ExtractorKey: "extract-alpha"}},
Rejected: []artifacts.RejectedArtifact{{ValidatorName: "validator"}},
Warnings: []contracts.Warning{{ReasonCode: "note", Message: "message"}},
OutputFiles: []contracts.OutputFile{{Name: "artifacts/generic.json", ContentType: "application/json", Bytes: []byte(`{}`)}},
}
if input.Pipeline.ID != "pipeline-1" || input.SourceID != "source-1" {
t.Fatalf("RunInput = %#v, want constructed fields", input)
}
if output.Manifest.PipelineID != "pipeline-1" || len(output.Approved) != 1 || len(output.Rejected) != 1 || len(output.Warnings) != 1 {
if output.Manifest.PipelineID != "pipeline-1" || len(output.Approved) != 1 || len(output.Rejected) != 1 || len(output.Warnings) != 1 || len(output.OutputFiles) != 1 {
t.Fatalf("RunOutput = %#v, want constructed fields", output)
}
}
@@ -604,11 +604,18 @@ func TestRunOutputEncoderReceivesManifestAndArtifacts(t *testing.T) {
t.Fatalf("Run() error = %v, want nil", err)
}
if output.ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", output.ContentType)
if len(output.OutputFiles) != 1 {
t.Fatalf("len(OutputFiles) = %d, want 1", len(output.OutputFiles))
}
if string(output.EncodedOutput) != `{"encoded":true}` {
t.Fatalf("EncodedOutput = %s, want encoded payload", output.EncodedOutput)
file := output.OutputFiles[0]
if file.Name != "artifacts/generic.json" {
t.Fatalf("OutputFiles[0].Name = %q, want artifacts/generic.json", file.Name)
}
if file.ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", file.ContentType)
}
if string(file.Bytes) != `{"encoded":true}` {
t.Fatalf("OutputFiles[0].Bytes = %s, want encoded payload", file.Bytes)
}
if len(modules.output.requests) != 1 {
t.Fatalf("len(output requests) = %d, want 1", len(modules.output.requests))
@@ -622,6 +629,35 @@ func TestRunOutputEncoderReceivesManifestAndArtifacts(t *testing.T) {
}
}
func TestRunRejectsUnsafeOutputFileNames(t *testing.T) {
tests := []struct {
name string
fileName string
}{
{name: "empty", fileName: ""},
{name: "absolute", fileName: "/tmp/output.json"},
{name: "parent", fileName: "artifacts/../manifest.json"},
{name: "backslash", fileName: `artifacts\manifest.json`},
{name: "unclean", fileName: "artifacts//manifest.json"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
modules := defaultRunnerModules()
modules.output.files = []contracts.OutputFile{
{Name: test.fileName, ContentType: "application/json", Bytes: []byte(`{}`)},
}
output, err := New(newRunnerRegistries(t, modules)).Run(context.Background(), RunInput{Pipeline: resolvedPipeline()})
assertRunError(t, err, "output file name")
if output.Manifest.ValidationStatus != "failed" {
t.Fatalf("ValidationStatus = %q, want failed", output.Manifest.ValidationStatus)
}
})
}
}
func TestRunReturnsFailedManifestWhenOutputEncoderFails(t *testing.T) {
modules := defaultRunnerModules()
modules.output.err = errors.New("encode failed")
@@ -632,6 +668,9 @@ func TestRunReturnsFailedManifestWhenOutputEncoderFails(t *testing.T) {
if output.Manifest.ValidationStatus != "failed" {
t.Fatalf("ValidationStatus = %q, want failed", output.Manifest.ValidationStatus)
}
if output.Manifest.CompletedAt == nil {
t.Fatal("CompletedAt = nil, want failed run completion timestamp")
}
if len(output.Approved) != 2 {
t.Fatalf("len(Approved) = %d, want partial approved output", len(output.Approved))
}
@@ -668,6 +707,76 @@ func TestRunManifestIncludesPipelineAndLaneDetails(t *testing.T) {
}
}
func TestRunManifestIncludesRunTimingAndLLMProfiles(t *testing.T) {
startedAt := time.Now().Add(-time.Minute).UTC()
profiles := []artifacts.LLMProfileManifest{
{ID: "default", Provider: "openai-compatible", Model: "model-a"},
}
output, err := New(newRunnerRegistries(t, nil)).Run(context.Background(), RunInput{
Pipeline: resolvedPipeline(),
RunID: "run-test",
StartedAt: startedAt,
LLMProfiles: profiles,
})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
manifest := output.Manifest
if manifest.RunID != "run-test" {
t.Fatalf("RunID = %q, want run-test", manifest.RunID)
}
if manifest.StartedAt == nil || !manifest.StartedAt.Equal(startedAt) {
t.Fatalf("StartedAt = %v, want %s", manifest.StartedAt, startedAt)
}
if manifest.CompletedAt == nil || manifest.CompletedAt.Before(startedAt) {
t.Fatalf("CompletedAt = %v, want timestamp after start", manifest.CompletedAt)
}
if !reflect.DeepEqual(manifest.LLMProfiles, profiles) {
t.Fatalf("LLMProfiles = %#v, want %#v", manifest.LLMProfiles, profiles)
}
}
func TestRunManifestGeneratesRunIDAndTimestamps(t *testing.T) {
output, err := New(newRunnerRegistries(t, nil)).Run(context.Background(), RunInput{Pipeline: resolvedPipeline()})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if !strings.HasPrefix(output.Manifest.RunID, "run-") {
t.Fatalf("RunID = %q, want generated run ID", output.Manifest.RunID)
}
if output.Manifest.StartedAt == nil {
t.Fatal("StartedAt = nil, want generated timestamp")
}
if output.Manifest.CompletedAt == nil {
t.Fatal("CompletedAt = nil, want generated timestamp")
}
}
func TestRunManifestIncludesExtractorMetadata(t *testing.T) {
modules := defaultRunnerModules()
modules.extractors["extract-alpha"].manifestMetadata = map[string]any{
"prompt_id": "test.prompt",
"response_schema_name": "test_schema",
}
output, err := New(newRunnerRegistries(t, modules)).Run(context.Background(), RunInput{Pipeline: resolvedPipeline()})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
lane := output.Manifest.ArtifactLanes[0]
extractorMetadata, ok := lane.Metadata["extractor"].(map[string]any)
if !ok {
t.Fatalf("lane metadata = %#v, want extractor metadata", lane.Metadata)
}
if extractorMetadata["prompt_id"] != "test.prompt" || extractorMetadata["response_schema_name"] != "test_schema" {
t.Fatalf("extractor metadata = %#v, want prompt and schema metadata", extractorMetadata)
}
}
func TestRunReturnsPartialOutputWhenLaterLaneFails(t *testing.T) {
modules := defaultRunnerModules()
modules.extractors["extract-beta"] = &runnerExtractor{key: "extract-beta", artifactType: "artifact", schemaVersion: "v1", err: errors.New("extract failed")}
@@ -783,7 +892,12 @@ func defaultRunnerModules() *runnerModules {
"configured": {name: "configured", decisions: approveAll},
"second-validator": {name: "second-validator", decisions: approveAll},
},
output: &runnerOutputEncoder{key: "output", bytes: []byte(`{"encoded":true}`), contentType: "application/json"},
output: &runnerOutputEncoder{
key: "output",
files: []contracts.OutputFile{
{Name: "artifacts/generic.json", ContentType: "application/json", Bytes: []byte(`{"encoded":true}`)},
},
},
}
}
@@ -885,17 +999,18 @@ func (chunker *runnerChunker) Chunk(ctx context.Context, req contracts.ChunkRequ
}
type runnerExtractor struct {
key string
artifactType string
schemaVersion string
candidates []artifacts.ArtifactCandidate
validators []contracts.Validator
warnings []contracts.Warning
err error
requests []contracts.ExtractionRequest
seenChunkIDs []string
seenLLMClients []contracts.StructuredLLMClient
seenMetadata []map[string]any
key string
artifactType string
schemaVersion string
manifestMetadata map[string]any
candidates []artifacts.ArtifactCandidate
validators []contracts.Validator
warnings []contracts.Warning
err error
requests []contracts.ExtractionRequest
seenChunkIDs []string
seenLLMClients []contracts.StructuredLLMClient
seenMetadata []map[string]any
}
func (extractor *runnerExtractor) Key() string {
@@ -910,6 +1025,10 @@ func (extractor *runnerExtractor) SchemaVersion() string {
return extractor.schemaVersion
}
func (extractor *runnerExtractor) ManifestMetadata() map[string]any {
return extractor.manifestMetadata
}
func (extractor *runnerExtractor) Validators() []contracts.Validator {
return extractor.validators
}
@@ -1019,12 +1138,11 @@ func (validator *runnerValidator) Validate(ctx context.Context, req contracts.Va
}
type runnerOutputEncoder struct {
key string
bytes []byte
contentType string
warnings []contracts.Warning
err error
requests []contracts.OutputRequest
key string
files []contracts.OutputFile
warnings []contracts.Warning
err error
requests []contracts.OutputRequest
}
func (encoder *runnerOutputEncoder) Key() string {
@@ -1034,9 +1152,8 @@ func (encoder *runnerOutputEncoder) Key() string {
func (encoder *runnerOutputEncoder) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
encoder.requests = append(encoder.requests, req)
return contracts.OutputResult{
Bytes: encoder.bytes,
ContentType: encoder.contentType,
Warnings: encoder.warnings,
Files: encoder.files,
Warnings: encoder.warnings,
}, encoder.err
}

View File

@@ -36,10 +36,13 @@ func TestWalkingSkeletonFixture(t *testing.T) {
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if output.ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", output.ContentType)
if len(output.OutputFiles) != 1 {
t.Fatalf("len(OutputFiles) = %d, want 1", len(output.OutputFiles))
}
assertStructuralJSONEqual(t, output.EncodedOutput, expectedBytes)
if output.OutputFiles[0].ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", output.OutputFiles[0].ContentType)
}
assertStructuralJSONEqual(t, output.OutputFiles[0].Bytes, expectedBytes)
if llmClient.calls != 2 {
t.Fatalf("LLM calls = %d, want chunk count 2", llmClient.calls)
}
@@ -135,7 +138,7 @@ func walkingSkeletonCatalog(t *testing.T) ModuleCatalog {
Stage: StageMerge,
Requires: []string{"fake_artifacts"},
}, func() (contracts.Merger, error) {
return AppendOrderMerger{}, nil
return walkingSkeletonMerger{}, nil
}); err != nil {
t.Fatalf("register append-order merger: %v", err)
}
@@ -143,7 +146,7 @@ func walkingSkeletonCatalog(t *testing.T) ModuleCatalog {
Key: DefaultNormalizeModule,
Stage: StageNormalize,
}, func() (contracts.Normalizer, error) {
return NoopNormalizer{}, nil
return walkingSkeletonNormalizer{}, nil
}); err != nil {
t.Fatalf("register no-op normalizer: %v", err)
}
@@ -309,6 +312,30 @@ func (client *walkingSkeletonLLMClient) CompleteStructured(ctx context.Context,
}, nil
}
type walkingSkeletonMerger struct{}
func (merger walkingSkeletonMerger) Key() string {
return DefaultMergeModule
}
func (merger walkingSkeletonMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
var candidates []artifacts.ArtifactCandidate
for _, chunkArtifacts := range req.ChunkArtifacts {
candidates = append(candidates, chunkArtifacts.Candidates...)
}
return contracts.MergeResult{Candidates: candidates}, nil
}
type walkingSkeletonNormalizer struct{}
func (normalizer walkingSkeletonNormalizer) Key() string {
return DefaultNormalizeModule
}
func (normalizer walkingSkeletonNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
return contracts.NormalizeResult{Candidates: req.Candidates}, nil
}
type walkingSkeletonOutput struct{}
func (output walkingSkeletonOutput) Key() string {
@@ -332,8 +359,9 @@ func (output walkingSkeletonOutput) Encode(ctx context.Context, req contracts.Ou
return contracts.OutputResult{}, err
}
return contracts.OutputResult{
Bytes: encoded,
ContentType: "application/json",
Files: []contracts.OutputFile{
{Name: "output.json", ContentType: "application/json", Bytes: encoded},
},
}, nil
}

View File

@@ -5,6 +5,7 @@ import (
"embed"
"encoding/hex"
"fmt"
"io/fs"
"path"
"sort"
"strings"
@@ -17,7 +18,6 @@ var embeddedAssets embed.FS
const (
SourceBuiltin = "builtin"
VersionV1 = "v1"
DNDSpellsPromptID = "dnd.spells"
TestGenericPromptID = "test.generic"
)
@@ -41,21 +41,31 @@ func (m Metadata) DiagnosticsMap() map[string]any {
}
}
type definition struct {
id string
version string
embeddedDir string
systemPath string
userPath string
// Definition identifies a caller-owned system/user prompt bundle.
type Definition struct {
PromptID string
Version string
EmbeddedPath string
SystemPath string
UserPath string
}
type compiledPrompt struct {
// Bundle is a compiled system/user prompt pair.
type Bundle struct {
systemTmpl *template.Template
userTmpl *template.Template
metadata Metadata
}
var promptRegistry map[string]compiledPrompt
// Metadata returns metadata for the compiled prompt bundle.
func (b *Bundle) Metadata() Metadata {
if b == nil {
return Metadata{}
}
return b.metadata
}
var promptRegistry map[string]*Bundle
var sharedHardening string
func init() {
@@ -65,30 +75,23 @@ func init() {
panic(err)
}
defs := []definition{
defs := []Definition{
{
id: DNDSpellsPromptID,
version: VersionV1,
embeddedDir: "assets/dnd/spells",
systemPath: "assets/dnd/spells/system.md",
userPath: "assets/dnd/spells/user.md",
},
{
id: TestGenericPromptID,
version: VersionV1,
embeddedDir: "assets/test/generic",
systemPath: "assets/test/generic/system.md",
userPath: "assets/test/generic/user.md",
PromptID: TestGenericPromptID,
Version: VersionV1,
EmbeddedPath: "assets/test/generic",
SystemPath: "assets/test/generic/system.md",
UserPath: "assets/test/generic/user.md",
},
}
promptRegistry = make(map[string]compiledPrompt, len(defs))
promptRegistry = make(map[string]*Bundle, len(defs))
for _, def := range defs {
compiled, compileErr := compilePrompt(def)
compiled, compileErr := LoadBundle(embeddedAssets, def)
if compileErr != nil {
panic(compileErr)
}
promptRegistry[def.id] = compiled
promptRegistry[compiled.metadata.PromptID] = compiled
}
}
@@ -138,51 +141,68 @@ func readAsset(assetPath string) (string, error) {
return string(content), nil
}
func compilePrompt(def definition) (compiledPrompt, error) {
if strings.TrimSpace(def.id) == "" {
return compiledPrompt{}, fmt.Errorf("prompt id must not be empty")
// LoadBundle compiles a system/user prompt bundle from a caller-owned filesystem.
func LoadBundle(fsys fs.FS, def Definition) (*Bundle, error) {
promptID := strings.TrimSpace(def.PromptID)
version := strings.TrimSpace(def.Version)
embeddedPath := strings.TrimSpace(def.EmbeddedPath)
systemPath := strings.TrimSpace(def.SystemPath)
userPath := strings.TrimSpace(def.UserPath)
if promptID == "" {
return nil, fmt.Errorf("prompt id must not be empty")
}
if strings.TrimSpace(def.version) == "" {
return compiledPrompt{}, fmt.Errorf("prompt version must not be empty")
if version == "" {
return nil, fmt.Errorf("prompt version must not be empty")
}
if strings.TrimSpace(def.embeddedDir) == "" {
return compiledPrompt{}, fmt.Errorf("prompt embedded path must not be empty")
if embeddedPath == "" {
return nil, fmt.Errorf("prompt embedded path must not be empty")
}
systemSource, err := readAsset(def.systemPath)
systemSource, err := readPromptAsset(fsys, systemPath)
if err != nil {
return compiledPrompt{}, err
return nil, err
}
userSource, err := readAsset(def.userPath)
userSource, err := readPromptAsset(fsys, userPath)
if err != nil {
return compiledPrompt{}, err
return nil, err
}
funcs := template.FuncMap{
"hardening": func() string { return sharedHardening },
}
systemTmpl, err := template.New(path.Base(def.systemPath)).Option("missingkey=error").Funcs(funcs).Parse(systemSource)
systemTmpl, err := template.New(path.Base(systemPath)).Option("missingkey=error").Funcs(funcs).Parse(systemSource)
if err != nil {
return compiledPrompt{}, fmt.Errorf("parse embedded system prompt %q: %w", def.systemPath, err)
return nil, fmt.Errorf("parse embedded system prompt %q: %w", systemPath, err)
}
userTmpl, err := template.New(path.Base(def.userPath)).Option("missingkey=error").Funcs(funcs).Parse(userSource)
userTmpl, err := template.New(path.Base(userPath)).Option("missingkey=error").Funcs(funcs).Parse(userSource)
if err != nil {
return compiledPrompt{}, fmt.Errorf("parse embedded user prompt %q: %w", def.userPath, err)
return nil, fmt.Errorf("parse embedded user prompt %q: %w", userPath, err)
}
hashInput := systemSource + "\n\n" + userSource
hash := sha256.Sum256([]byte(hashInput))
metadata := Metadata{
PromptID: strings.TrimSpace(def.id),
PromptVersion: strings.TrimSpace(def.version),
PromptID: promptID,
PromptVersion: version,
PromptSource: SourceBuiltin,
EmbeddedPath: strings.TrimSpace(def.embeddedDir),
EmbeddedPath: embeddedPath,
SHA256: "sha256:" + hex.EncodeToString(hash[:]),
}
return compiledPrompt{
return &Bundle{
systemTmpl: systemTmpl,
userTmpl: userTmpl,
metadata: metadata,
}, nil
}
func readPromptAsset(fsys fs.FS, assetPath string) (string, error) {
if strings.TrimSpace(assetPath) == "" {
return "", fmt.Errorf("prompt asset path must not be empty")
}
content, err := fs.ReadFile(fsys, assetPath)
if err != nil {
return "", fmt.Errorf("read embedded prompt asset %q: %w", assetPath, err)
}
return string(content), nil
}

View File

@@ -11,7 +11,6 @@ func TestLookupMetadataSucceedsForRegisteredPrompts(t *testing.T) {
promptID string
embeddedPath string
}{
{promptID: DNDSpellsPromptID, embeddedPath: "assets/dnd/spells"},
{promptID: TestGenericPromptID, embeddedPath: "assets/test/generic"},
}
@@ -59,8 +58,8 @@ func TestMustLookupMetadataPanicsForUnknownPromptID(t *testing.T) {
func TestRegisteredMetadataSortedByPromptID(t *testing.T) {
registered := RegisteredMetadata()
if len(registered) != 2 {
t.Fatalf("expected two registered prompts, got %d", len(registered))
if len(registered) != 1 {
t.Fatalf("expected one registered prompt, got %d", len(registered))
}
ids := make([]string, len(registered))
@@ -72,8 +71,8 @@ func TestRegisteredMetadataSortedByPromptID(t *testing.T) {
if !sort.StringsAreSorted(ids) {
t.Fatalf("expected sorted prompt IDs, got %v", ids)
}
if !seen[DNDSpellsPromptID] {
t.Fatalf("registered prompt IDs = %v, want %q", ids, DNDSpellsPromptID)
if !seen[TestGenericPromptID] {
t.Fatalf("registered prompt IDs = %v, want %q", ids, TestGenericPromptID)
}
}

View File

@@ -13,16 +13,23 @@ func RenderUserSystem(promptID string, data any) (system string, user string, me
if !ok {
return "", "", Metadata{}, fmt.Errorf("unknown prompt id %q", promptID)
}
return compiled.RenderUserSystem(data)
}
// RenderUserSystem renders the bundle's system and user prompts.
func (b *Bundle) RenderUserSystem(data any) (system string, user string, metadata Metadata, err error) {
if b == nil {
return "", "", Metadata{}, fmt.Errorf("prompt bundle must not be nil")
}
var systemBuf bytes.Buffer
if err := compiled.systemTmpl.Execute(&systemBuf, data); err != nil {
return "", "", Metadata{}, fmt.Errorf("render system prompt %q: %w", trimmedID, err)
if err := b.systemTmpl.Execute(&systemBuf, data); err != nil {
return "", "", Metadata{}, fmt.Errorf("render system prompt %q: %w", b.metadata.PromptID, err)
}
var userBuf bytes.Buffer
if err := compiled.userTmpl.Execute(&userBuf, data); err != nil {
return "", "", Metadata{}, fmt.Errorf("render user prompt %q: %w", trimmedID, err)
if err := b.userTmpl.Execute(&userBuf, data); err != nil {
return "", "", Metadata{}, fmt.Errorf("render user prompt %q: %w", b.metadata.PromptID, err)
}
return strings.TrimSpace(systemBuf.String()), strings.TrimSpace(userBuf.String()), compiled.metadata, nil
return strings.TrimSpace(systemBuf.String()), strings.TrimSpace(userBuf.String()), b.metadata, nil
}

View File

@@ -64,47 +64,3 @@ func TestRenderUserSystemIncludesHardeningText(t *testing.T) {
t.Fatalf("expected rendered system prompt to include hardening text: %q", system)
}
}
func TestRenderDNDSpellsPromptIncludesHardeningText(t *testing.T) {
system, user, metadata, err := RenderUserSystem(DNDSpellsPromptID, map[string]any{
"SourceID": "session-alpha",
"HasChunk": true,
"ChunkID": "session-alpha:chunk:0",
"ChunkIndex": 0,
"Units": []map[string]any{
{
"ID": "seg-001",
"Text": "Aria casts Cure Wounds.",
"Metadata": []map[string]string{
{"Key": "speaker", "Value": "Alice"},
},
},
},
})
if err != nil {
t.Fatalf("RenderUserSystem: %v", err)
}
hardening := strings.TrimSpace(HardeningText())
if hardening == "" {
t.Fatalf("expected hardening text")
}
if !strings.Contains(system, hardening) {
t.Fatalf("expected rendered system prompt to include hardening text: %q", system)
}
for _, want := range []string{"session-alpha", "session-alpha:chunk:0", "seg-001", "Aria casts Cure Wounds.", "speaker: Alice"} {
if !strings.Contains(user, want) {
t.Fatalf("rendered user prompt = %q, want substring %q", user, want)
}
}
if metadata.PromptID != DNDSpellsPromptID {
t.Fatalf("unexpected metadata: %+v", metadata)
}
}
func TestRenderDNDSpellsPromptMissingTemplateDataReturnsError(t *testing.T) {
_, _, _, err := RenderUserSystem(DNDSpellsPromptID, map[string]any{})
if err == nil || !strings.Contains(err.Error(), "SourceID") {
t.Fatalf("expected missing SourceID error, got %v", err)
}
}

View File

@@ -0,0 +1,241 @@
package generic
import (
"context"
"encoding/json"
"fmt"
"math"
"strconv"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const Key = "generic"
const (
defaultMaxUnits = 50
defaultOverlapUnits = 0
)
var _ contracts.Chunker = (*Chunker)(nil)
type Chunker struct{}
func New() *Chunker {
return &Chunker{}
}
func (c *Chunker) Key() string {
return Key
}
func (c *Chunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
if c == nil {
return contracts.ChunkResult{}, chunkerErrorf("chunker must not be nil")
}
if ctx == nil {
return contracts.ChunkResult{}, chunkerErrorf("context must not be nil")
}
if err := ctx.Err(); err != nil {
return contracts.ChunkResult{}, chunkerErrorf("context error before chunking: %w", err)
}
if req.Source == nil {
return contracts.ChunkResult{}, chunkerErrorf("source must not be nil")
}
if len(req.Source.Units) == 0 {
return contracts.ChunkResult{}, chunkerErrorf("source units must not be empty")
}
if err := source.ValidateDocument(req.Source); err != nil {
return contracts.ChunkResult{}, chunkerErrorf("validate source document: %w", err)
}
opts, err := chunkOptionsFrom(req.Options)
if err != nil {
return contracts.ChunkResult{}, err
}
step := opts.maxUnits - opts.overlapUnits
chunks := make([]contracts.SourceChunk, 0, (len(req.Source.Units)+step-1)/step)
for start := 0; start < len(req.Source.Units); start += step {
end := start + opts.maxUnits
if end > len(req.Source.Units) {
end = len(req.Source.Units)
}
units := cloneUnits(req.Source.Units[start:end])
chunks = append(chunks, contracts.SourceChunk{
ID: fmt.Sprintf("chunk-%06d", len(chunks)+1),
SourceID: req.Source.ID,
Index: len(chunks),
Units: units,
Metadata: map[string]any{
"start_unit_id": units[0].ID,
"end_unit_id": units[len(units)-1].ID,
"unit_count": len(units),
},
})
if end == len(req.Source.Units) {
break
}
}
return contracts.ChunkResult{Chunks: chunks}, nil
}
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageChunk,
Provides: []string{"chunks"},
}
}
func Register(registry *pipeline.ChunkerRegistry) error {
return registry.RegisterWithSpec(ModuleSpec(), func() (contracts.Chunker, error) {
return New(), nil
})
}
type chunkOptions struct {
maxUnits int
overlapUnits int
}
func chunkOptionsFrom(options map[string]any) (chunkOptions, error) {
opts := chunkOptions{
maxUnits: defaultMaxUnits,
overlapUnits: defaultOverlapUnits,
}
var err error
if value, ok := options["max_units"]; ok {
opts.maxUnits, err = positiveIntOption("max_units", value)
if err != nil {
return chunkOptions{}, err
}
}
if value, ok := options["overlap_units"]; ok {
opts.overlapUnits, err = nonNegativeIntOption("overlap_units", value)
if err != nil {
return chunkOptions{}, err
}
}
if opts.overlapUnits >= opts.maxUnits {
return chunkOptions{}, chunkerErrorf("overlap_units must be less than max_units")
}
return opts, nil
}
func positiveIntOption(name string, value any) (int, error) {
got, err := intOption(name, value)
if err != nil {
return 0, err
}
if got <= 0 {
return 0, chunkerErrorf("%s must be positive", name)
}
return got, nil
}
func nonNegativeIntOption(name string, value any) (int, error) {
got, err := intOption(name, value)
if err != nil {
return 0, err
}
if got < 0 {
return 0, chunkerErrorf("%s must be non-negative", name)
}
return got, nil
}
func intOption(name string, value any) (int, error) {
switch typed := value.(type) {
case int:
return typed, nil
case int8:
return int(typed), nil
case int16:
return int(typed), nil
case int32:
return int(typed), nil
case int64:
if typed > maxInt() || typed < minInt() {
return 0, chunkerErrorf("%s is outside supported integer range", name)
}
return int(typed), nil
case uint:
if uint64(typed) > uint64(maxInt()) {
return 0, chunkerErrorf("%s is outside supported integer range", name)
}
return int(typed), nil
case uint8:
return int(typed), nil
case uint16:
return int(typed), nil
case uint32:
if uint64(typed) > uint64(maxInt()) {
return 0, chunkerErrorf("%s is outside supported integer range", name)
}
return int(typed), nil
case uint64:
if typed > uint64(maxInt()) {
return 0, chunkerErrorf("%s is outside supported integer range", name)
}
return int(typed), nil
case float64:
if typed != math.Trunc(typed) {
return 0, chunkerErrorf("%s must be an integer", name)
}
if typed > float64(maxInt()) || typed < float64(minInt()) {
return 0, chunkerErrorf("%s is outside supported integer range", name)
}
return int(typed), nil
case json.Number:
parsed, err := typed.Int64()
if err != nil {
return 0, chunkerErrorf("%s must be an integer", name)
}
if parsed > maxInt() || parsed < minInt() {
return 0, chunkerErrorf("%s is outside supported integer range", name)
}
return int(parsed), nil
default:
return 0, chunkerErrorf("%s must be an integer", name)
}
}
func maxInt() int64 {
return int64(1<<(strconv.IntSize-1) - 1)
}
func minInt() int64 {
return -maxInt() - 1
}
func cloneUnits(units []source.SourceUnit) []source.SourceUnit {
out := make([]source.SourceUnit, 0, len(units))
for _, unit := range units {
out = append(out, source.SourceUnit{
ID: unit.ID,
Kind: unit.Kind,
Text: unit.Text,
Metadata: cloneMetadata(unit.Metadata),
})
}
return out
}
func cloneMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
out := make(map[string]any, len(metadata))
for key, value := range metadata {
out[key] = value
}
return out
}
func chunkerErrorf(format string, args ...any) error {
return fmt.Errorf("generic chunker: "+format, args...)
}

View File

@@ -0,0 +1,213 @@
package generic
import (
"context"
"encoding/json"
"fmt"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestModuleSpecAndRegister(t *testing.T) {
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageChunk,
Provides: []string{"chunks"},
}
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
}
registry := pipeline.NewChunkerRegistry()
if err := Register(registry); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
spec, ok := registry.Spec(Key)
if !ok {
t.Fatalf("Spec(%q) ok = false, want true", Key)
}
if !reflect.DeepEqual(spec, want) {
t.Fatalf("registered spec = %#v, want %#v", spec, want)
}
chunker, err := registry.Build(Key)
if err != nil {
t.Fatalf("Build(%q) error = %v, want nil", Key, err)
}
if chunker.Key() != Key {
t.Fatalf("Key() = %q, want %q", chunker.Key(), Key)
}
}
func TestChunkUsesDefaultsForSingleChunk(t *testing.T) {
result, err := New().Chunk(context.Background(), contracts.ChunkRequest{Source: testSource(3), Options: nil})
if err != nil {
t.Fatalf("Chunk() error = %v, want nil", err)
}
if got := chunkIDs(result.Chunks); !reflect.DeepEqual(got, []string{"chunk-000001"}) {
t.Fatalf("chunk IDs = %#v, want one stable ID", got)
}
chunk := result.Chunks[0]
if chunk.Index != 0 || chunk.SourceID != "source-1" {
t.Fatalf("chunk = %#v, want source and index fields", chunk)
}
if got := unitIDs(chunk.Units); !reflect.DeepEqual(got, []string{"u001", "u002", "u003"}) {
t.Fatalf("unit IDs = %#v, want all units", got)
}
if chunk.Metadata["start_unit_id"] != "u001" || chunk.Metadata["end_unit_id"] != "u003" || chunk.Metadata["unit_count"] != 3 {
t.Fatalf("metadata = %#v, want chunk bounds", chunk.Metadata)
}
}
func TestChunkExactBoundaries(t *testing.T) {
result, err := New().Chunk(context.Background(), contracts.ChunkRequest{
Source: testSource(6),
Options: map[string]any{"max_units": 2},
})
if err != nil {
t.Fatalf("Chunk() error = %v, want nil", err)
}
if got := chunkIDs(result.Chunks); !reflect.DeepEqual(got, []string{"chunk-000001", "chunk-000002", "chunk-000003"}) {
t.Fatalf("chunk IDs = %#v, want stable IDs", got)
}
gotUnits := [][]string{unitIDs(result.Chunks[0].Units), unitIDs(result.Chunks[1].Units), unitIDs(result.Chunks[2].Units)}
wantUnits := [][]string{{"u001", "u002"}, {"u003", "u004"}, {"u005", "u006"}}
if !reflect.DeepEqual(gotUnits, wantUnits) {
t.Fatalf("chunk units = %#v, want %#v", gotUnits, wantUnits)
}
}
func TestChunkOverlap(t *testing.T) {
result, err := New().Chunk(context.Background(), contracts.ChunkRequest{
Source: testSource(7),
Options: map[string]any{"max_units": 3, "overlap_units": 1},
})
if err != nil {
t.Fatalf("Chunk() error = %v, want nil", err)
}
gotUnits := make([][]string, 0, len(result.Chunks))
for _, chunk := range result.Chunks {
gotUnits = append(gotUnits, unitIDs(chunk.Units))
}
wantUnits := [][]string{{"u001", "u002", "u003"}, {"u003", "u004", "u005"}, {"u005", "u006", "u007"}}
if !reflect.DeepEqual(gotUnits, wantUnits) {
t.Fatalf("chunk units = %#v, want %#v", gotUnits, wantUnits)
}
}
func TestChunkRejectsInvalidOptions(t *testing.T) {
tests := []struct {
name string
options map[string]any
want string
}{
{name: "max wrong type", options: map[string]any{"max_units": "2"}, want: "max_units"},
{name: "max fractional", options: map[string]any{"max_units": 1.5}, want: "integer"},
{name: "max zero", options: map[string]any{"max_units": 0}, want: "positive"},
{name: "overlap negative", options: map[string]any{"overlap_units": -1}, want: "non-negative"},
{name: "overlap too large", options: map[string]any{"max_units": 2, "overlap_units": 2}, want: "less than"},
{name: "json number", options: map[string]any{"max_units": json.Number("bad")}, want: "integer"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
_, err := New().Chunk(context.Background(), contracts.ChunkRequest{
Source: testSource(3),
Options: test.options,
})
if err == nil {
t.Fatal("Chunk() error = nil, want error")
}
if !strings.Contains(err.Error(), "generic chunker") || !strings.Contains(err.Error(), test.want) {
t.Fatalf("Chunk() error = %q, want module context and %q", err.Error(), test.want)
}
})
}
}
func TestChunkRejectsEmptySource(t *testing.T) {
doc := testSource(1)
doc.Units = nil
_, err := New().Chunk(context.Background(), contracts.ChunkRequest{Source: doc})
if err == nil {
t.Fatal("Chunk() error = nil, want empty source error")
}
if !strings.Contains(err.Error(), "generic chunker") || !strings.Contains(err.Error(), "units") {
t.Fatalf("Chunk() error = %q, want empty source context", err.Error())
}
}
func TestChunkDefensivelyCopiesUnits(t *testing.T) {
doc := testSource(2)
result, err := New().Chunk(context.Background(), contracts.ChunkRequest{
Source: doc,
Options: map[string]any{"max_units": 1},
})
if err != nil {
t.Fatalf("Chunk() error = %v, want nil", err)
}
if len(result.Chunks) != 2 {
t.Fatalf("len(Chunks) = %d, want 2", len(result.Chunks))
}
doc.Units[0].ID = "changed"
doc.Units[0].Metadata["speaker"] = "changed"
if result.Chunks[0].Units[0].ID != "u001" {
t.Fatalf("chunk unit ID changed after source mutation: %#v", result.Chunks[0].Units[0])
}
if result.Chunks[0].Units[0].Metadata["speaker"] != "speaker-001" {
t.Fatalf("chunk unit metadata changed after source mutation: %#v", result.Chunks[0].Units[0].Metadata)
}
}
func testSource(count int) *source.SourceDocument {
units := make([]source.SourceUnit, 0, count)
for i := 1; i <= count; i++ {
id := "u" + zeroPad3(i)
units = append(units, source.SourceUnit{
ID: id,
Kind: "unit",
Text: "Text for " + id,
Metadata: map[string]any{
"speaker": "speaker-" + zeroPad3(i),
},
})
}
return &source.SourceDocument{
ID: "source-1",
Kind: "document",
Format: "text/plain",
Digest: "sha256:source",
Units: units,
}
}
func zeroPad3(value int) string {
return fmt.Sprintf("%03d", value)
}
func chunkIDs(chunks []contracts.SourceChunk) []string {
ids := make([]string, 0, len(chunks))
for _, chunk := range chunks {
ids = append(ids, chunk.ID)
}
return ids
}
func unitIDs(units []source.SourceUnit) []string {
ids := make([]string, 0, len(units))
for _, unit := range units {
ids = append(ids, unit.ID)
}
return ids
}

View File

@@ -0,0 +1,6 @@
package spells
import "embed"
//go:embed assets/prompts/*.md assets/schemas/*.json
var embeddedAssets embed.FS

View File

@@ -11,6 +11,8 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
)
func TestPipelineConfigLoadsAndResolvesWithDNDSpellsExtractor(t *testing.T) {
@@ -188,7 +190,7 @@ func dndSpellsTestCatalog(t *testing.T, specs dndSpellsCatalogSpecs) pipeline.Mo
Stage: pipeline.StageMerge,
Requires: []string{"dnd.spell_casts"},
}, func() (contracts.Merger, error) {
return pipeline.AppendOrderMerger{}, nil
return appendorder.New(), nil
}); err != nil {
t.Fatalf("register merger: %v", err)
}
@@ -196,7 +198,7 @@ func dndSpellsTestCatalog(t *testing.T, specs dndSpellsCatalogSpecs) pipeline.Mo
Key: pipeline.DefaultNormalizeModule,
Stage: pipeline.StageNormalize,
}, func() (contracts.Normalizer, error) {
return pipeline.NoopNormalizer{}, nil
return noop.New(), nil
}); err != nil {
t.Fatalf("register normalizer: %v", err)
}
@@ -255,8 +257,9 @@ func (dndSpellsOutput) Key() string {
func (dndSpellsOutput) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
return contracts.OutputResult{
Bytes: []byte(`{"encoded":true}`),
ContentType: "application/json",
Files: []contracts.OutputFile{
{Name: "output.json", ContentType: "application/json", Bytes: []byte(`{"encoded":true}`)},
},
}, nil
}

View File

@@ -9,7 +9,6 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
@@ -46,6 +45,23 @@ func (e *Extractor) SchemaVersion() string {
return SchemaVersion
}
func (e *Extractor) ManifestMetadata() map[string]any {
promptMetadata := spellsPromptBundle.Metadata()
metadata := map[string]any{
"prompt_id": PromptID,
"prompt_version": promptMetadata.PromptVersion,
"prompt_sha256": promptMetadata.SHA256,
"response_schema_key": string(ResponseSchemaKey),
"response_schema_id": ResponseSchemaID,
"response_schema_name": ResponseSchemaName,
}
if schema, err := loadResponseSchema(); err == nil {
metadata["response_schema_version"] = schema.Version
metadata["response_schema_sha256"] = schema.SHA256
}
return metadata
}
func (e *Extractor) Validators() []contracts.Validator {
return []contracts.Validator{
ShapeValidator{},
@@ -80,9 +96,9 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.ExtractionRequest
if err != nil {
return contracts.ExtractionResult{}, extractorErrorf("render prompt: %w", err)
}
schema, ok := llm.LookupResponseSchema(llm.DNDSpellsSchemaKey)
if !ok {
return contracts.ExtractionResult{}, extractorErrorf("lookup response schema %q", llm.DNDSpellsSchemaKey)
schema, err := loadResponseSchema()
if err != nil {
return contracts.ExtractionResult{}, extractorErrorf("load response schema %q: %w", ResponseSchemaKey, err)
}
var response extractionResponse

View File

@@ -10,7 +10,6 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
)
func TestExtractReturnsSpellCandidateFromStructuredOutput(t *testing.T) {
@@ -42,7 +41,10 @@ func TestExtractReturnsSpellCandidateFromStructuredOutput(t *testing.T) {
if req.StageName != Key {
t.Fatalf("StageName = %q, want %q", req.StageName, Key)
}
schema := llm.MustLookupResponseSchema(llm.DNDSpellsSchemaKey)
schema, err := loadResponseSchema()
if err != nil {
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
}
if req.ResponseSchemaName != schema.Name {
t.Fatalf("ResponseSchemaName = %q, want %q", req.ResponseSchemaName, schema.Name)
}
@@ -90,6 +92,30 @@ func TestExtractReturnsSpellCandidateFromStructuredOutput(t *testing.T) {
}
}
func TestExtractorManifestMetadataIncludesPromptAndSchemaProvenance(t *testing.T) {
metadata := New().ManifestMetadata()
tests := map[string]string{
"prompt_id": PromptID,
"prompt_version": SchemaVersion,
"response_schema_key": string(ResponseSchemaKey),
"response_schema_id": ResponseSchemaID,
"response_schema_name": ResponseSchemaName,
"response_schema_version": SchemaVersion,
}
for key, want := range tests {
if metadata[key] != want {
t.Fatalf("metadata[%q] = %#v, want %q", key, metadata[key], want)
}
}
for _, key := range []string{"prompt_sha256", "response_schema_sha256"} {
value, ok := metadata[key].(string)
if !ok || !strings.HasPrefix(value, "sha256:") {
t.Fatalf("metadata[%q] = %#v, want sha256 value", key, metadata[key])
}
}
}
func TestExtractReturnsNoCandidatesForEmptyResponse(t *testing.T) {
client := &fakeSpellsLLMClient{response: extractionResponse{SpellCasts: []spellCastResponse{}}}

View File

@@ -28,6 +28,22 @@ type promptMetadata struct {
Value string
}
var spellsPromptBundle = mustLoadPromptBundle()
func mustLoadPromptBundle() *prompt.Bundle {
bundle, err := prompt.LoadBundle(embeddedAssets, prompt.Definition{
PromptID: PromptID,
Version: SchemaVersion,
EmbeddedPath: "assets/prompts",
SystemPath: "assets/prompts/system.md",
UserPath: "assets/prompts/user.md",
})
if err != nil {
panic(err)
}
return bundle
}
func buildPromptData(req contracts.ExtractionRequest) (promptData, error) {
if req.Source == nil {
return promptData{}, fmt.Errorf("dnd spells prompt: source must not be nil")
@@ -58,7 +74,7 @@ func renderPrompt(req contracts.ExtractionRequest) (system string, user string,
if err != nil {
return "", "", prompt.Metadata{}, err
}
system, user, metadata, err = prompt.RenderUserSystem(prompt.DNDSpellsPromptID, data)
system, user, metadata, err = spellsPromptBundle.RenderUserSystem(data)
if err != nil {
return "", "", prompt.Metadata{}, fmt.Errorf("dnd spells prompt: %w", err)
}

View File

@@ -92,8 +92,14 @@ func TestRenderPromptIncludesSourceContext(t *testing.T) {
t.Fatalf("user prompt = %q, want substring %q", user, want)
}
}
if metadata.PromptID != prompt.DNDSpellsPromptID {
t.Fatalf("metadata.PromptID = %q, want %q", metadata.PromptID, prompt.DNDSpellsPromptID)
if metadata.PromptID != PromptID {
t.Fatalf("metadata.PromptID = %q, want %q", metadata.PromptID, PromptID)
}
if metadata.PromptVersion != SchemaVersion {
t.Fatalf("metadata.PromptVersion = %q, want %q", metadata.PromptVersion, SchemaVersion)
}
if metadata.EmbeddedPath != "assets/prompts" {
t.Fatalf("metadata.EmbeddedPath = %q, want assets/prompts", metadata.EmbeddedPath)
}
}

View File

@@ -93,8 +93,20 @@ func TestRunnerProcessesSeriatimInputWithDNDSpellsExtractor(t *testing.T) {
if lane.ID != "spells" || lane.Extractor != Key {
t.Fatalf("manifest lane = %#v, want spells lane with dnd/spells extractor", lane)
}
if output.ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", output.ContentType)
extractorMetadata, ok := lane.Metadata["extractor"].(map[string]any)
if !ok {
t.Fatalf("manifest lane metadata = %#v, want extractor metadata", lane.Metadata)
}
if extractorMetadata["prompt_id"] != PromptID ||
extractorMetadata["response_schema_key"] != string(ResponseSchemaKey) ||
extractorMetadata["response_schema_name"] != ResponseSchemaName {
t.Fatalf("extractor metadata = %#v, want prompt/schema identifiers", extractorMetadata)
}
if len(output.OutputFiles) != 1 {
t.Fatalf("len(OutputFiles) = %d, want 1", len(output.OutputFiles))
}
if output.OutputFiles[0].ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", output.OutputFiles[0].ContentType)
}
}

View File

@@ -0,0 +1,20 @@
package spells
import "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
const (
PromptID = "dnd.spells"
ResponseSchemaKey = llm.ResponseSchemaKey("dnd_spells")
ResponseSchemaID = "notarius.dnd.spells"
ResponseSchemaName = "notarius_dnd_spells_v1"
)
func loadResponseSchema() (llm.ResponseSchema, error) {
return llm.LoadResponseSchema(embeddedAssets, llm.ResponseSchemaDefinition{
Key: ResponseSchemaKey,
ID: ResponseSchemaID,
Version: SchemaVersion,
Name: ResponseSchemaName,
AssetPath: "assets/schemas/dnd_spells.v1.json",
})
}

View File

@@ -4,23 +4,24 @@ import (
"encoding/json"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
)
func TestLookupResponseSchemaForSpells(t *testing.T) {
schema, ok := llm.LookupResponseSchema(llm.DNDSpellsSchemaKey)
if !ok {
t.Fatalf("LookupResponseSchema(%q) ok = false, want true", llm.DNDSpellsSchemaKey)
func TestLoadResponseSchemaForSpells(t *testing.T) {
schema, err := loadResponseSchema()
if err != nil {
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
}
if schema.ID != "notarius.dnd.spells" {
t.Fatalf("schema.ID = %q, want notarius.dnd.spells", schema.ID)
if schema.Key != ResponseSchemaKey {
t.Fatalf("schema.Key = %q, want %q", schema.Key, ResponseSchemaKey)
}
if schema.ID != ResponseSchemaID {
t.Fatalf("schema.ID = %q, want %q", schema.ID, ResponseSchemaID)
}
if schema.Version != SchemaVersion {
t.Fatalf("schema.Version = %q, want %q", schema.Version, SchemaVersion)
}
if schema.Name != "notarius_dnd_spells_v1" {
t.Fatalf("schema.Name = %q, want notarius_dnd_spells_v1", schema.Name)
if schema.Name != ResponseSchemaName {
t.Fatalf("schema.Name = %q, want %q", schema.Name, ResponseSchemaName)
}
if !strings.HasPrefix(schema.SHA256, "sha256:") {
t.Fatalf("schema.SHA256 = %q, want sha256 prefix", schema.SHA256)
@@ -30,12 +31,34 @@ func TestLookupResponseSchemaForSpells(t *testing.T) {
}
}
func TestResponseSchemaJSONIsMutationSafe(t *testing.T) {
first, err := loadResponseSchema()
if err != nil {
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
}
first.JSONSchema[0] = '['
second, err := loadResponseSchema()
if err != nil {
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
}
if !json.Valid(second.JSONSchema) {
t.Fatalf("schema JSON was mutated: %s", second.JSONSchema)
}
if len(second.JSONSchema) > 0 && second.JSONSchema[0] == '[' {
t.Fatalf("schema JSON did not use defensive copy")
}
}
func TestResponseSchemaDiagnosticsOmitRawSchema(t *testing.T) {
schema := llm.MustLookupResponseSchema(llm.DNDSpellsSchemaKey)
schema, err := loadResponseSchema()
if err != nil {
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
}
diagnostics := schema.DiagnosticsMap()
if diagnostics["key"] != llm.DNDSpellsSchemaKey {
t.Fatalf("diagnostics[key] = %#v, want %q", diagnostics["key"], llm.DNDSpellsSchemaKey)
if diagnostics["key"] != ResponseSchemaKey {
t.Fatalf("diagnostics[key] = %#v, want %q", diagnostics["key"], ResponseSchemaKey)
}
for _, key := range []string{"id", "version", "name", "sha256"} {
if diagnostics[key] == "" {
@@ -45,4 +68,7 @@ func TestResponseSchemaDiagnosticsOmitRawSchema(t *testing.T) {
if _, ok := diagnostics["json_schema"]; ok {
t.Fatalf("diagnostics should omit raw schema content: %#v", diagnostics)
}
if _, ok := diagnostics["JSONSchema"]; ok {
t.Fatalf("diagnostics should omit raw schema content: %#v", diagnostics)
}
}

View File

@@ -20,7 +20,7 @@ const Key = "seriatim"
const (
DocumentKind = "transcript"
UnitKind = "transcript_segment"
Format = "application/vnd.seriatim.minimal+json"
Format = "application/vnd.seriatim+json"
)
var providedCapabilities = []string{

View File

@@ -70,6 +70,35 @@ func TestParseValidMinimalTranscript(t *testing.T) {
}
}
func TestParseAcceptsNumericSegmentIDs(t *testing.T) {
raw := []byte(`{"metadata":{"application":"seriatim","version":"v1.5.0","output_schema":"seriatim-intermediate"},"segments":[{"id":1,"start":451.821,"end":469.685,"speaker":"Narrator","text":"The stone door opens.","categories":["scene"]},{"id":2,"start":470,"end":471,"speaker":"Player","text":"I step inside."}]}`)
doc, err := New().Parse(context.Background(), contracts.ParseRequest{Raw: raw})
if err != nil {
t.Fatalf("Parse() error = %v, want nil", err)
}
if got, want := doc.Metadata["output_schema"], "seriatim-intermediate"; got != want {
t.Fatalf("doc.Metadata[output_schema] = %#v, want %q", got, want)
}
if doc.Format != Format {
t.Fatalf("doc.Format = %q, want %q", doc.Format, Format)
}
if len(doc.Units) != 2 {
t.Fatalf("len(doc.Units) = %d, want 2", len(doc.Units))
}
if doc.Units[0].ID != "1" || doc.Units[1].ID != "2" {
t.Fatalf("unit IDs = %#v, want numeric IDs normalized to strings", []string{doc.Units[0].ID, doc.Units[1].ID})
}
ref := source.SourceRef{
SourceID: doc.ID,
StartUnitID: "1",
EndUnitID: "2",
}
if err := source.ValidateRef(doc, ref); err != nil {
t.Fatalf("ValidateRef() error = %v, want nil", err)
}
}
func TestParseRequestSourceIDOverridesMetadataIDs(t *testing.T) {
raw := readFixture(t, "testdata/valid_minimal.json")
@@ -176,6 +205,11 @@ func TestParseRejectsInvalidInput(t *testing.T) {
raw: validJSONWithSegment(`"start":0,"end":1,"speaker":"Narrator","text":"Synthetic text."`),
wantErr: []string{"id", "empty"},
},
{
name: "invalid segment id type",
raw: validJSONWithSegment(`"id":{},"start":0,"end":1,"speaker":"Narrator","text":"Synthetic text."`),
wantErr: []string{"id", "string or number"},
},
{
name: "whitespace segment id",
raw: validJSONWithSegment(`"id":" s1 ","start":0,"end":1,"speaker":"Narrator","text":"Synthetic text."`),

View File

@@ -10,6 +10,8 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
)
func TestPipelineConfigLoadsAndResolvesWithSeriatimInput(t *testing.T) {
@@ -179,7 +181,7 @@ func mustRegisterExtractor(t *testing.T, registry *pipeline.ExtractorRegistry, s
func mustRegisterMerger(t *testing.T, registry *pipeline.MergerRegistry, spec pipeline.ModuleSpec) {
t.Helper()
if err := registry.RegisterWithSpec(spec, func() (contracts.Merger, error) {
return pipeline.AppendOrderMerger{}, nil
return appendorder.New(), nil
}); err != nil {
t.Fatalf("register merger: %v", err)
}
@@ -188,7 +190,7 @@ func mustRegisterMerger(t *testing.T, registry *pipeline.MergerRegistry, spec pi
func mustRegisterNormalizer(t *testing.T, registry *pipeline.NormalizerRegistry, spec pipeline.ModuleSpec) {
t.Helper()
if err := registry.RegisterWithSpec(spec, func() (contracts.Normalizer, error) {
return pipeline.NoopNormalizer{}, nil
return noop.New(), nil
}); err != nil {
t.Fatalf("register normalizer: %v", err)
}

View File

@@ -77,8 +77,8 @@ func decodeSegment(raw []byte, index int) (segment, error) {
}
var decoded segment
if err := decodeOptionalString(fields, "id", &decoded.ID); err != nil {
return segment{}, fmt.Errorf("segment[%d] id must be a string: %w", index, err)
if err := decodeOptionalSegmentID(fields, "id", &decoded.ID); err != nil {
return segment{}, fmt.Errorf("segment[%d] id must be a string or number: %w", index, err)
}
if err := decodeOptionalNumber(fields, "start", &decoded.Start); err != nil {
return segment{}, fmt.Errorf("segment[%d] start must be a number: %w", index, err)
@@ -103,6 +103,27 @@ func decodeOptionalString(fields map[string]json.RawMessage, key string, out *st
return decodeJSON(raw, out)
}
func decodeOptionalSegmentID(fields map[string]json.RawMessage, key string, out *string) error {
raw, ok := fields[key]
if !ok {
return nil
}
var text string
if err := decodeJSON(raw, &text); err == nil {
*out = text
return nil
}
var number json.Number
if err := decodeJSON(raw, &number); err == nil {
*out = number.String()
return nil
}
return fmt.Errorf("must be a string or number")
}
func decodeOptionalNumber(fields map[string]json.RawMessage, key string, out *json.Number) error {
raw, ok := fields[key]
if !ok {

View File

@@ -12,6 +12,8 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
)
func TestRunnerProcessesSeriatimInputWithFakeModules(t *testing.T) {
@@ -61,8 +63,11 @@ func TestRunnerProcessesSeriatimInputWithFakeModules(t *testing.T) {
if extractor.calls != 1 {
t.Fatalf("extractor calls = %d, want 1", extractor.calls)
}
if output.ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", output.ContentType)
if len(output.OutputFiles) != 1 {
t.Fatalf("len(OutputFiles) = %d, want 1", len(output.OutputFiles))
}
if output.OutputFiles[0].ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", output.OutputFiles[0].ContentType)
}
}
@@ -121,12 +126,12 @@ func seriatimRunnerRegistries(t *testing.T, extractor contracts.Extractor) pipel
t.Fatalf("register extractor: %v", err)
}
if err := mergers.Register(pipeline.DefaultMergeModule, func() (contracts.Merger, error) {
return pipeline.AppendOrderMerger{}, nil
return appendorder.New(), nil
}); err != nil {
t.Fatalf("register merger: %v", err)
}
if err := normalizers.Register(pipeline.DefaultNormalizeModule, func() (contracts.Normalizer, error) {
return pipeline.NoopNormalizer{}, nil
return noop.New(), nil
}); err != nil {
t.Fatalf("register normalizer: %v", err)
}
@@ -235,8 +240,9 @@ func (runnerSeriatimOutput) Key() string {
func (runnerSeriatimOutput) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
return contracts.OutputResult{
Bytes: []byte(`{"encoded":true}`),
ContentType: "application/json",
Files: []contracts.OutputFile{
{Name: "output.json", ContentType: "application/json", Bytes: []byte(`{"encoded":true}`)},
},
}, nil
}

View File

@@ -0,0 +1,97 @@
package appendorder
import (
"context"
"encoding/json"
"fmt"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const Key = "appendorder"
var _ contracts.Merger = (*Merger)(nil)
type Merger struct{}
func New() *Merger {
return &Merger{}
}
func (m *Merger) Key() string {
return Key
}
func (m *Merger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
if m == nil {
return contracts.MergeResult{}, mergerErrorf("merger must not be nil")
}
if ctx == nil {
return contracts.MergeResult{}, mergerErrorf("context must not be nil")
}
if err := ctx.Err(); err != nil {
return contracts.MergeResult{}, mergerErrorf("context error before merge: %w", err)
}
var candidates []artifacts.ArtifactCandidate
for _, chunkArtifacts := range req.ChunkArtifacts {
candidates = append(candidates, cloneCandidates(chunkArtifacts.Candidates)...)
}
return contracts.MergeResult{Candidates: candidates}, nil
}
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageMerge,
Provides: []string{"merged"},
}
}
func Register(registry *pipeline.MergerRegistry) error {
return registry.RegisterWithSpec(ModuleSpec(), func() (contracts.Merger, error) {
return New(), nil
})
}
func cloneCandidates(candidates []artifacts.ArtifactCandidate) []artifacts.ArtifactCandidate {
if len(candidates) == 0 {
return nil
}
out := make([]artifacts.ArtifactCandidate, 0, len(candidates))
for _, candidate := range candidates {
out = append(out, cloneCandidate(candidate))
}
return out
}
func cloneCandidate(candidate artifacts.ArtifactCandidate) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: candidate.Index,
ExtractorKey: candidate.ExtractorKey,
ArtifactType: candidate.ArtifactType,
SchemaVersion: candidate.SchemaVersion,
Payload: append(json.RawMessage(nil), candidate.Payload...),
SourceRefs: append([]source.SourceRef(nil), candidate.SourceRefs...),
Metadata: cloneMetadata(candidate.Metadata),
}
}
func cloneMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
out := make(map[string]any, len(metadata))
for key, value := range metadata {
out[key] = value
}
return out
}
func mergerErrorf(format string, args ...any) error {
return fmt.Errorf("appendorder merger: "+format, args...)
}

View File

@@ -0,0 +1,147 @@
package appendorder
import (
"context"
"encoding/json"
"reflect"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestModuleSpecAndRegister(t *testing.T) {
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageMerge,
Provides: []string{"merged"},
}
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
}
registry := pipeline.NewMergerRegistry()
if err := Register(registry); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
spec, ok := registry.Spec(Key)
if !ok {
t.Fatalf("Spec(%q) ok = false, want true", Key)
}
if !reflect.DeepEqual(spec, want) {
t.Fatalf("registered spec = %#v, want %#v", spec, want)
}
}
func TestMergePreservesChunkAndCandidateOrder(t *testing.T) {
result, err := New().Merge(context.Background(), contracts.MergeRequest{
ChunkArtifacts: []contracts.ChunkArtifacts{
{
Chunk: sourceChunk(0),
Candidates: []artifacts.ArtifactCandidate{candidate(2, "first-b"), candidate(1, "first-a")},
},
{
Chunk: sourceChunk(1),
Candidates: []artifacts.ArtifactCandidate{candidate(4, "second-b"), candidate(3, "second-a")},
},
},
})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
got := candidateNames(result.Candidates)
want := []string{"first-b", "first-a", "second-b", "second-a"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("candidate order = %#v, want %#v", got, want)
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
}
}
func TestMergeDefensivelyCopiesCandidates(t *testing.T) {
input := []contracts.ChunkArtifacts{
{
Chunk: sourceChunk(0),
Candidates: []artifacts.ArtifactCandidate{candidate(1, "original")},
},
}
result, err := New().Merge(context.Background(), contracts.MergeRequest{ChunkArtifacts: input})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
if len(result.Candidates) != 1 {
t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
}
input[0].Candidates[0].Index = 99
input[0].Candidates[0].Payload[0] = '['
input[0].Candidates[0].SourceRefs[0].StartUnitID = "changed"
input[0].Candidates[0].Metadata["name"] = "changed"
got := result.Candidates[0]
if got.Index != 1 {
t.Fatalf("Index = %d, want 1", got.Index)
}
if string(got.Payload) != `{"name":"original"}` {
t.Fatalf("Payload = %s, want original payload", got.Payload)
}
if got.SourceRefs[0].StartUnitID != "u1" {
t.Fatalf("SourceRefs = %#v, want original source ref", got.SourceRefs)
}
if got.Metadata["name"] != "original" {
t.Fatalf("Metadata = %#v, want original metadata", got.Metadata)
}
}
func TestMergeHandlesEmptyInput(t *testing.T) {
result, err := New().Merge(context.Background(), contracts.MergeRequest{})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
if len(result.Candidates) != 0 {
t.Fatalf("len(Candidates) = %d, want 0", len(result.Candidates))
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
}
}
func candidate(index int, name string) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: index,
ExtractorKey: "generic-extractor",
ArtifactType: "generic-artifact",
SchemaVersion: "v1",
Payload: json.RawMessage(`{"name":"` + name + `"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
},
Metadata: map[string]any{
"name": name,
},
}
}
func candidateNames(candidates []artifacts.ArtifactCandidate) []string {
names := make([]string, 0, len(candidates))
for _, candidate := range candidates {
names = append(names, candidate.Metadata["name"].(string))
}
return names
}
func sourceChunk(index int) contracts.SourceChunk {
return contracts.SourceChunk{
ID: "chunk",
SourceID: "source-1",
Index: index,
Units: []source.SourceUnit{
{ID: "u1", Kind: "unit", Text: "Source unit."},
},
}
}

View File

@@ -0,0 +1,89 @@
package noop
import (
"context"
"encoding/json"
"fmt"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const Key = "noop"
var _ contracts.Normalizer = (*Normalizer)(nil)
type Normalizer struct{}
func New() *Normalizer {
return &Normalizer{}
}
func (n *Normalizer) Key() string {
return Key
}
func (n *Normalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
if n == nil {
return contracts.NormalizeResult{}, normalizerErrorf("normalizer must not be nil")
}
if ctx == nil {
return contracts.NormalizeResult{}, normalizerErrorf("context must not be nil")
}
if err := ctx.Err(); err != nil {
return contracts.NormalizeResult{}, normalizerErrorf("context error before normalize: %w", err)
}
return contracts.NormalizeResult{Candidates: cloneCandidates(req.Candidates)}, nil
}
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageNormalize,
Requires: []string{"merged"},
Provides: []string{"normalized"},
}
}
func Register(registry *pipeline.NormalizerRegistry) error {
return registry.RegisterWithSpec(ModuleSpec(), func() (contracts.Normalizer, error) {
return New(), nil
})
}
func cloneCandidates(candidates []artifacts.ArtifactCandidate) []artifacts.ArtifactCandidate {
if len(candidates) == 0 {
return nil
}
out := make([]artifacts.ArtifactCandidate, 0, len(candidates))
for _, candidate := range candidates {
out = append(out, artifacts.ArtifactCandidate{
Index: candidate.Index,
ExtractorKey: candidate.ExtractorKey,
ArtifactType: candidate.ArtifactType,
SchemaVersion: candidate.SchemaVersion,
Payload: append(json.RawMessage(nil), candidate.Payload...),
SourceRefs: append([]source.SourceRef(nil), candidate.SourceRefs...),
Metadata: cloneMetadata(candidate.Metadata),
})
}
return out
}
func cloneMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
out := make(map[string]any, len(metadata))
for key, value := range metadata {
out[key] = value
}
return out
}
func normalizerErrorf(format string, args ...any) error {
return fmt.Errorf("noop normalizer: "+format, args...)
}

View File

@@ -0,0 +1,133 @@
package noop
import (
"context"
"encoding/json"
"reflect"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestModuleSpecAndRegister(t *testing.T) {
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageNormalize,
Requires: []string{"merged"},
Provides: []string{"normalized"},
}
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
}
registry := pipeline.NewNormalizerRegistry()
if err := Register(registry); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
spec, ok := registry.Spec(Key)
if !ok {
t.Fatalf("Spec(%q) ok = false, want true", Key)
}
if !reflect.DeepEqual(spec, want) {
t.Fatalf("registered spec = %#v, want %#v", spec, want)
}
}
func TestNormalizePassesThroughOrderAndValues(t *testing.T) {
input := []artifacts.ArtifactCandidate{
candidate(3, "third"),
candidate(1, "first"),
candidate(2, "second"),
}
result, err := New().Normalize(context.Background(), contracts.NormalizeRequest{Candidates: input})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
}
got := candidateNames(result.Candidates)
want := []string{"third", "first", "second"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("candidate order = %#v, want %#v", got, want)
}
if !reflect.DeepEqual(result.Candidates[0].SourceRefs, input[0].SourceRefs) {
t.Fatalf("SourceRefs = %#v, want %#v", result.Candidates[0].SourceRefs, input[0].SourceRefs)
}
if !reflect.DeepEqual(result.Candidates[0].Metadata, input[0].Metadata) {
t.Fatalf("Metadata = %#v, want %#v", result.Candidates[0].Metadata, input[0].Metadata)
}
}
func TestNormalizeDefensivelyCopiesCandidates(t *testing.T) {
input := []artifacts.ArtifactCandidate{candidate(1, "original")}
result, err := New().Normalize(context.Background(), contracts.NormalizeRequest{Candidates: input})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(result.Candidates) != 1 {
t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
}
input[0].Index = 99
input[0].Payload[0] = '['
input[0].SourceRefs[0].EndUnitID = "changed"
input[0].Metadata["name"] = "changed"
got := result.Candidates[0]
if got.Index != 1 {
t.Fatalf("Index = %d, want 1", got.Index)
}
if string(got.Payload) != `{"name":"original"}` {
t.Fatalf("Payload = %s, want original payload", got.Payload)
}
if got.SourceRefs[0].EndUnitID != "u1" {
t.Fatalf("SourceRefs = %#v, want original source ref", got.SourceRefs)
}
if got.Metadata["name"] != "original" {
t.Fatalf("Metadata = %#v, want original metadata", got.Metadata)
}
}
func TestNormalizeHandlesEmptyInput(t *testing.T) {
result, err := New().Normalize(context.Background(), contracts.NormalizeRequest{})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(result.Candidates) != 0 {
t.Fatalf("len(Candidates) = %d, want 0", len(result.Candidates))
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
}
}
func candidate(index int, name string) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: index,
ExtractorKey: "generic-extractor",
ArtifactType: "generic-artifact",
SchemaVersion: "v1",
Payload: json.RawMessage(`{"name":"` + name + `"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
},
Metadata: map[string]any{
"name": name,
},
}
}
func candidateNames(candidates []artifacts.ArtifactCandidate) []string {
names := make([]string, 0, len(candidates))
for _, candidate := range candidates {
names = append(names, candidate.Metadata["name"].(string))
}
return names
}

View File

@@ -0,0 +1,253 @@
package json
import (
"context"
stdjson "encoding/json"
"fmt"
"regexp"
"sort"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
const Key = "json"
const contentTypeJSON = "application/json"
var safeArtifactFileChar = regexp.MustCompile(`[^A-Za-z0-9._-]`)
var _ contracts.OutputEncoder = (*Encoder)(nil)
type Encoder struct{}
func New() *Encoder {
return &Encoder{}
}
func (e *Encoder) Key() string {
return Key
}
func (e *Encoder) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
if e == nil {
return contracts.OutputResult{}, encoderErrorf("encoder must not be nil")
}
if ctx == nil {
return contracts.OutputResult{}, encoderErrorf("context must not be nil")
}
if err := ctx.Err(); err != nil {
return contracts.OutputResult{}, encoderErrorf("context error before encoding: %w", err)
}
files, err := logicalFiles(req)
if err != nil {
return contracts.OutputResult{}, err
}
return contracts.OutputResult{Files: files}, nil
}
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageOutput,
Requires: []string{"normalized"},
Provides: []string{"encoded"},
}
}
func Register(registry *pipeline.OutputEncoderRegistry) error {
return registry.RegisterWithSpec(ModuleSpec(), func() (contracts.OutputEncoder, error) {
return New(), nil
})
}
type indexFile struct {
ManifestFile string `json:"manifest_file"`
ArtifactFiles []artifactFileIndex `json:"artifact_files"`
RejectedFile string `json:"rejected_file"`
WarningsFile string `json:"warnings_file"`
}
type artifactFileIndex struct {
ArtifactType string `json:"artifact_type"`
File string `json:"file"`
}
type artifactFile struct {
ArtifactType string `json:"artifact_type"`
Artifacts []artifacts.Artifact `json:"artifacts"`
}
type rejectedFile struct {
Rejected []artifacts.RejectedArtifact `json:"rejected"`
}
type warningsFile struct {
Warnings []contracts.Warning `json:"warnings"`
}
func logicalFiles(req contracts.OutputRequest) ([]contracts.OutputFile, error) {
artifactsByType := make(map[string][]artifacts.Artifact)
for _, artifact := range req.Approved {
artifactsByType[artifact.ArtifactType] = append(artifactsByType[artifact.ArtifactType], cloneArtifact(artifact))
}
artifactTypes := make([]string, 0, len(artifactsByType))
for artifactType := range artifactsByType {
artifactTypes = append(artifactTypes, artifactType)
}
sort.Strings(artifactTypes)
artifactIndexes := make([]artifactFileIndex, 0, len(artifactTypes))
files := make([]contracts.OutputFile, 0, len(artifactTypes)+4)
manifestFile, err := jsonFile("manifest.json", req.Manifest)
if err != nil {
return nil, err
}
files = append(files, manifestFile)
usedArtifactFiles := make(map[string]string, len(artifactTypes))
for _, artifactType := range artifactTypes {
name, err := artifactFileName(artifactType)
if err != nil {
return nil, err
}
if existingType, ok := usedArtifactFiles[name]; ok {
return nil, encoderErrorf("artifact types %q and %q produce duplicate output file %q", existingType, artifactType, name)
}
usedArtifactFiles[name] = artifactType
artifactIndexes = append(artifactIndexes, artifactFileIndex{
ArtifactType: artifactType,
File: name,
})
file, err := jsonFile(name, artifactFile{
ArtifactType: artifactType,
Artifacts: artifactsByType[artifactType],
})
if err != nil {
return nil, err
}
files = append(files, file)
}
index := indexFile{
ManifestFile: "manifest.json",
ArtifactFiles: artifactIndexes,
RejectedFile: "rejected.json",
WarningsFile: "warnings.json",
}
indexOutput, err := jsonFile("index.json", index)
if err != nil {
return nil, err
}
rejectedOutput, err := jsonFile("rejected.json", rejectedFile{Rejected: cloneRejected(req.Rejected)})
if err != nil {
return nil, err
}
warningsOutput, err := jsonFile("warnings.json", warningsFile{Warnings: cloneWarnings(req.Warnings)})
if err != nil {
return nil, err
}
files = append(files, indexOutput, rejectedOutput, warningsOutput)
sort.Slice(files, func(i, j int) bool {
return files[i].Name < files[j].Name
})
return files, nil
}
func jsonFile(name string, value any) (contracts.OutputFile, error) {
data, err := marshalPretty(value)
if err != nil {
return contracts.OutputFile{}, encoderErrorf("encode %s: %w", name, err)
}
return contracts.OutputFile{
Name: name,
ContentType: contentTypeJSON,
Bytes: data,
}, nil
}
func marshalPretty(value any) ([]byte, error) {
data, err := stdjson.MarshalIndent(value, "", " ")
if err != nil {
return nil, err
}
return append(data, '\n'), nil
}
func artifactFileName(artifactType string) (string, error) {
sanitized := safeArtifactFileChar.ReplaceAllString(strings.TrimSpace(artifactType), "_")
for strings.Contains(sanitized, "..") {
sanitized = strings.ReplaceAll(sanitized, "..", "__")
}
sanitized = strings.Trim(sanitized, "._")
if sanitized == "" {
return "", encoderErrorf("artifact type %q cannot produce a safe file name", artifactType)
}
return "artifacts/" + sanitized + ".json", nil
}
func cloneArtifact(artifact artifacts.Artifact) artifacts.Artifact {
return artifacts.Artifact{
ExtractorKey: artifact.ExtractorKey,
ArtifactType: artifact.ArtifactType,
SchemaVersion: artifact.SchemaVersion,
Payload: append(stdjson.RawMessage(nil), artifact.Payload...),
SourceRefs: append([]source.SourceRef(nil), artifact.SourceRefs...),
Metadata: cloneMetadata(artifact.Metadata),
}
}
func cloneRejected(rejected []artifacts.RejectedArtifact) []artifacts.RejectedArtifact {
if len(rejected) == 0 {
return []artifacts.RejectedArtifact{}
}
out := make([]artifacts.RejectedArtifact, 0, len(rejected))
for _, item := range rejected {
out = append(out, artifacts.RejectedArtifact{
Candidate: cloneCandidate(item.Candidate),
ValidatorName: item.ValidatorName,
ReasonCode: item.ReasonCode,
Message: item.Message,
})
}
return out
}
func cloneCandidate(candidate artifacts.ArtifactCandidate) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: candidate.Index,
ExtractorKey: candidate.ExtractorKey,
ArtifactType: candidate.ArtifactType,
SchemaVersion: candidate.SchemaVersion,
Payload: append(stdjson.RawMessage(nil), candidate.Payload...),
SourceRefs: append([]source.SourceRef(nil), candidate.SourceRefs...),
Metadata: cloneMetadata(candidate.Metadata),
}
}
func cloneWarnings(warnings []contracts.Warning) []contracts.Warning {
if len(warnings) == 0 {
return []contracts.Warning{}
}
return append([]contracts.Warning(nil), warnings...)
}
func cloneMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
out := make(map[string]any, len(metadata))
for key, value := range metadata {
out[key] = value
}
return out
}
func encoderErrorf(format string, args ...any) error {
return fmt.Errorf("json output encoder: "+format, args...)
}

View File

@@ -0,0 +1,316 @@
package json
import (
"context"
stdjson "encoding/json"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestModuleSpecAndRegister(t *testing.T) {
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageOutput,
Requires: []string{"normalized"},
Provides: []string{"encoded"},
}
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
}
registry := pipeline.NewOutputEncoderRegistry()
if err := Register(registry); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
spec, ok := registry.Spec(Key)
if !ok {
t.Fatalf("Spec(%q) ok = false, want true", Key)
}
if !reflect.DeepEqual(spec, want) {
t.Fatalf("registered spec = %#v, want %#v", spec, want)
}
}
func TestEncodeReturnsLogicalFilesGroupedByArtifactType(t *testing.T) {
req := contracts.OutputRequest{
Manifest: artifacts.RunManifest{RunID: "run-1", PipelineID: "pipeline-1"},
Approved: []artifacts.Artifact{
artifact("dnd.spell-cast", "first"),
artifact("notes/item", "item"),
artifact("dnd.spell-cast", "second"),
},
Rejected: []artifacts.RejectedArtifact{
{
Candidate: candidate("bad type", "bad"),
ValidatorName: "validator",
ReasonCode: "invalid",
Message: "not accepted",
},
},
Warnings: []contracts.Warning{{ReasonCode: "warning", Message: "check source"}},
}
result, err := New().Encode(context.Background(), req)
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
wantNames := []string{
"artifacts/dnd.spell-cast.json",
"artifacts/notes_item.json",
"index.json",
"manifest.json",
"rejected.json",
"warnings.json",
}
if got := outputFileNames(result.Files); !reflect.DeepEqual(got, wantNames) {
t.Fatalf("file names = %#v, want %#v", got, wantNames)
}
for _, file := range result.Files {
if file.ContentType != contentTypeJSON {
t.Fatalf("%s ContentType = %q, want %q", file.Name, file.ContentType, contentTypeJSON)
}
if !strings.HasSuffix(string(file.Bytes), "\n") {
t.Fatalf("%s does not end with newline: %q", file.Name, string(file.Bytes))
}
if !stdjson.Valid(file.Bytes) {
t.Fatalf("%s has invalid JSON: %s", file.Name, file.Bytes)
}
}
spellFile := decodeObject(t, fileBytes(t, result.Files, "artifacts/dnd.spell-cast.json"))
if spellFile["artifact_type"] != "dnd.spell-cast" {
t.Fatalf("artifact_type = %#v, want dnd.spell-cast", spellFile["artifact_type"])
}
spells := spellFile["artifacts"].([]any)
if len(spells) != 2 {
t.Fatalf("len(spells) = %d, want 2", len(spells))
}
firstPayload := spells[0].(map[string]any)["payload"].(map[string]any)
secondPayload := spells[1].(map[string]any)["payload"].(map[string]any)
if firstPayload["name"] != "first" || secondPayload["name"] != "second" {
t.Fatalf("spell order payloads = %#v then %#v, want runner order", firstPayload, secondPayload)
}
index := decodeObject(t, fileBytes(t, result.Files, "index.json"))
artifactFiles := index["artifact_files"].([]any)
if len(artifactFiles) != 2 {
t.Fatalf("len(index artifact_files) = %d, want 2", len(artifactFiles))
}
firstIndex := artifactFiles[0].(map[string]any)
secondIndex := artifactFiles[1].(map[string]any)
if firstIndex["artifact_type"] != "dnd.spell-cast" || secondIndex["artifact_type"] != "notes/item" {
t.Fatalf("artifact_files = %#v, want sorted by artifact type", artifactFiles)
}
}
func TestEncodeIncludesRejectedAndWarningsWhenEmpty(t *testing.T) {
result, err := New().Encode(context.Background(), contracts.OutputRequest{
Manifest: artifacts.RunManifest{RunID: "run-1"},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
rejected := decodeObject(t, fileBytes(t, result.Files, "rejected.json"))
if got := rejected["rejected"].([]any); len(got) != 0 {
t.Fatalf("rejected = %#v, want empty array", got)
}
warnings := decodeObject(t, fileBytes(t, result.Files, "warnings.json"))
if got := warnings["warnings"].([]any); len(got) != 0 {
t.Fatalf("warnings = %#v, want empty array", got)
}
}
func TestEncodePrettyPrintsJSON(t *testing.T) {
result, err := New().Encode(context.Background(), contracts.OutputRequest{
Manifest: artifacts.RunManifest{RunID: "run-1"},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
manifest := string(fileBytes(t, result.Files, "manifest.json"))
if !strings.Contains(manifest, "\n \"run_id\": \"run-1\"\n") {
t.Fatalf("manifest JSON = %q, want two-space indentation", manifest)
}
}
func TestEncodeRejectsArtifactTypeWithoutSafeFileName(t *testing.T) {
_, err := New().Encode(context.Background(), contracts.OutputRequest{
Approved: []artifacts.Artifact{artifact("///", "unsafe")},
})
if err == nil {
t.Fatal("Encode() error = nil, want unsafe artifact type error")
}
if !strings.Contains(err.Error(), "json output encoder") || !strings.Contains(err.Error(), "safe file name") {
t.Fatalf("Encode() error = %q, want safe file name context", err.Error())
}
}
func TestEncodeSanitizesParentPathSequences(t *testing.T) {
result, err := New().Encode(context.Background(), contracts.OutputRequest{
Approved: []artifacts.Artifact{artifact("dnd..spell.", "spell")},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
if got := outputFileNames(result.Files); !containsString(got, "artifacts/dnd__spell.json") {
t.Fatalf("file names = %#v, want sanitized artifact filename", got)
}
}
func TestEncodeRejectsSanitizedFilenameCollisions(t *testing.T) {
_, err := New().Encode(context.Background(), contracts.OutputRequest{
Approved: []artifacts.Artifact{
artifact("a/b", "slash"),
artifact("a?b", "question"),
},
})
if err == nil {
t.Fatal("Encode() error = nil, want duplicate file error")
}
if !strings.Contains(err.Error(), "duplicate output file") {
t.Fatalf("Encode() error = %q, want duplicate file context", err.Error())
}
}
func TestEncodeDoesNotMutateInputs(t *testing.T) {
req := contracts.OutputRequest{
Manifest: artifacts.RunManifest{RunID: "run-1"},
Approved: []artifacts.Artifact{
artifact("dnd.spell", "original"),
},
Rejected: []artifacts.RejectedArtifact{
{
Candidate: candidate("bad", "rejected"),
ValidatorName: "validator",
ReasonCode: "invalid",
Message: "not accepted",
},
},
Warnings: []contracts.Warning{{ReasonCode: "warning", Message: "message"}},
}
before := mustMarshal(t, req)
result, err := New().Encode(context.Background(), req)
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
after := mustMarshal(t, req)
if before != after {
t.Fatalf("request mutated:\nbefore: %s\nafter: %s", before, after)
}
req.Approved[0].Payload[0] = '['
req.Approved[0].SourceRefs[0].StartUnitID = "changed"
req.Approved[0].Metadata["name"] = "changed"
req.Rejected[0].Candidate.Payload[0] = '['
req.Warnings[0].Message = "changed"
if !stdjson.Valid(fileBytes(t, result.Files, "artifacts/dnd.spell.json")) {
t.Fatal("artifact output changed after request mutation")
}
warnings := decodeObject(t, fileBytes(t, result.Files, "warnings.json"))
gotWarnings := warnings["warnings"].([]any)
if gotWarnings[0].(map[string]any)["message"] != "message" {
t.Fatalf("warnings output changed after request mutation: %#v", gotWarnings)
}
}
func TestArtifactFilesDoNotContainWarnings(t *testing.T) {
result, err := New().Encode(context.Background(), contracts.OutputRequest{
Approved: []artifacts.Artifact{artifact("dnd.spell", "spell")},
Warnings: []contracts.Warning{
{ReasonCode: "pipeline-warning", Message: "warning"},
},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
artifactFile := decodeObject(t, fileBytes(t, result.Files, "artifacts/dnd.spell.json"))
if _, ok := artifactFile["warnings"]; ok {
t.Fatalf("artifact file contains warnings: %#v", artifactFile)
}
}
func artifact(artifactType, name string) artifacts.Artifact {
return artifacts.Artifact{
ExtractorKey: "extractor",
ArtifactType: artifactType,
SchemaVersion: "v1",
Payload: stdjson.RawMessage(`{"name":"` + name + `"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
},
Metadata: map[string]any{"name": name},
}
}
func candidate(artifactType, name string) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: 1,
ExtractorKey: "extractor",
ArtifactType: artifactType,
SchemaVersion: "v1",
Payload: stdjson.RawMessage(`{"name":"` + name + `"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
},
Metadata: map[string]any{"name": name},
}
}
func outputFileNames(files []contracts.OutputFile) []string {
names := make([]string, 0, len(files))
for _, file := range files {
names = append(names, file.Name)
}
return names
}
func containsString(values []string, want string) bool {
for _, value := range values {
if value == want {
return true
}
}
return false
}
func fileBytes(t *testing.T, files []contracts.OutputFile, name string) []byte {
t.Helper()
for _, file := range files {
if file.Name == name {
return file.Bytes
}
}
t.Fatalf("file %q not found in %#v", name, outputFileNames(files))
return nil
}
func decodeObject(t *testing.T, data []byte) map[string]any {
t.Helper()
var got map[string]any
if err := stdjson.Unmarshal(data, &got); err != nil {
t.Fatalf("Unmarshal() error = %v, want nil\n%s", err, data)
}
return got
}
func mustMarshal(t *testing.T, value any) string {
t.Helper()
data, err := stdjson.Marshal(value)
if err != nil {
t.Fatalf("Marshal() error = %v, want nil", err)
}
return string(data)
}