From 30e98a4d9947ab215b536b766f65ca9e7027891e Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 4 Jul 2026 03:11:29 +0000 Subject: [PATCH] Add integration and artifact contract documentation --- docs/integrations/dnd-spell-artifacts.md | 152 ++++++++++++++++++ docs/integrations/dnd-spells.md | 116 -------------- docs/integrations/json-output.md | 192 +++++++++++++++++++++++ docs/integrations/openai-compatible.md | 128 +++++++++++++++ docs/integrations/seriatim.md | 64 +++++--- docs/internal/llm.md | 3 +- docs/internal/modules.md | 9 +- 7 files changed, 517 insertions(+), 147 deletions(-) create mode 100644 docs/integrations/dnd-spell-artifacts.md delete mode 100644 docs/integrations/dnd-spells.md create mode 100644 docs/integrations/json-output.md create mode 100644 docs/integrations/openai-compatible.md diff --git a/docs/integrations/dnd-spell-artifacts.md b/docs/integrations/dnd-spell-artifacts.md new file mode 100644 index 0000000..5de9e72 --- /dev/null +++ b/docs/integrations/dnd-spell-artifacts.md @@ -0,0 +1,152 @@ +# D&D Spell-Cast Artifacts + +This document is the durable artifact contract for approved +`dnd.spell_cast` artifacts produced by the implemented `dnd/spells` extractor. + +## Artifact Identity + +- Extractor key: `dnd/spells` +- Artifact type: `dnd.spell_cast` +- Schema version: `v1` +- Prompt ID: `dnd.spells` +- Response schema key: `dnd_spells` +- Response schema ID: `notarius.dnd.spells` +- Response schema name: `notarius_dnd_spells_v1` + +The extractor requires source chunks and transcript source capability. It +returns generic artifact candidates that are serialized by the JSON output +module. + +## Artifact Envelope + +Approved artifacts use the generic artifact envelope documented in +[JSON Output](json-output.md#artifact-files): + +```json +{ + "extractor_key": "dnd/spells", + "artifact_type": "dnd.spell_cast", + "schema_version": "v1", + "payload": { + "caster": "Aria", + "spell": "Cure Wounds", + "effect": "heals an injured ally", + "narrative_description": "Aria raises her holy symbol and casts Cure Wounds." + }, + "source_refs": [ + { + "source_id": "session-alpha", + "start_unit_id": "seg-001", + "end_unit_id": "seg-001" + } + ] +} +``` + +## Payload Fields + +The `payload` object contains: + +- `caster`: in-world character or creature casting the spell; +- `spell`: spell name; +- `effect`: concise spell effect in the scene; +- `narrative_description`: short description of the spell cast in context. + +All payload fields are strings and must be non-empty after trimming. + +`caster` is the in-world caster, not the transcript speaker. + +## Source References + +Source references live on the artifact envelope as `source_refs`; they are not +duplicated inside the `payload`. + +Each source reference uses the generic source-reference shape: + +- `source_id` +- `start_unit_id` +- `end_unit_id` + +Validation requires: + +- at least one source reference; +- non-empty source ID and unit IDs; +- source ID matching the source document ID; +- start and end unit IDs existing in the source document; +- start unit appearing before or at the same position as end unit. + +## Structured LLM Response Shape + +The extractor asks the LLM for this top-level response shape: + +```json +{ + "spell_casts": [ + { + "caster": "Aria", + "spell": "Cure Wounds", + "effect": "heals an injured ally", + "narrative_description": "Aria raises her holy symbol and casts Cure Wounds.", + "source_refs": [ + { + "source_id": "session-alpha", + "start_unit_id": "seg-001", + "end_unit_id": "seg-001" + } + ] + } + ] +} +``` + +`spell_casts` must be present. It may be empty when no spell casts are found. + +The response schema asset is embedded at +`internal/modules/extract/dnd/spells/assets/schemas/dnd_spells.v1.json`. + +## Validators + +The extractor supplies two deterministic validators by default: + +- `dnd/spells/shape` +- `dnd/spells/source_refs` + +Rejection reason codes: + +- `invalid_payload`: payload JSON cannot be decoded as a spell-cast payload. +- `missing_required_field`: `caster`, `spell`, `effect`, or + `narrative_description` is blank. +- `missing_source_ref`: candidate has no source references. +- `invalid_source_ref`: at least one source reference fails generic source + reference validation. + +Rejected candidates are written to `rejected.json` by the JSON output module. + +## Manifest Metadata + +The extractor adds prompt and response-schema provenance under the artifact lane +manifest metadata: + +```json +{ + "metadata": { + "extractor": { + "prompt_id": "dnd.spells", + "prompt_version": "v1", + "prompt_sha256": "sha256:...", + "response_schema_key": "dnd_spells", + "response_schema_id": "notarius.dnd.spells", + "response_schema_name": "notarius_dnd_spells_v1", + "response_schema_version": "v1", + "response_schema_sha256": "sha256:..." + } + } +} +``` + +Raw prompt and schema content are not included in manifest metadata. + +## Compatibility Limit + +This contract covers only `dnd.spell_cast` artifacts produced by the +implemented spell-cast extractor. diff --git a/docs/integrations/dnd-spells.md b/docs/integrations/dnd-spells.md deleted file mode 100644 index abf441e..0000000 --- a/docs/integrations/dnd-spells.md +++ /dev/null @@ -1,116 +0,0 @@ -# D&D Spell Cast Extraction - -This document describes the D&D spell-cast extractor currently implemented in -Notarius. - -## Module - -- Module key: `dnd/spells` -- Artifact type: `dnd.spell_cast` -- Schema version: `v1` -- Prompt ID: `dnd.spells` -- Response schema key: `dnd_spells` -- Response schema ID: `notarius.dnd.spells` -- Response schema name: `notarius_dnd_spells_v1` - -The module is an extract module. It reads a generic source chunk, renders the -`dnd.spells` prompt, calls the configured structured LLM client, and returns -spell-cast artifact candidates. - -## Source Expectations - -The extractor expects a generic `SourceDocument` and active source chunk. It -does not depend on concrete Seriatim package types. - -Pipeline resolution must provide these capabilities before the extractor runs: - -- `chunks` -- `source.transcript` - -Source units may include transcript metadata such as speaker and timestamps. -That metadata is optional prompt context. It is not part of the durable spell -payload. - -## Artifact Payload - -Each approved artifact payload is a JSON object with these fields: - -- `caster`: in-world character or creature casting the spell; -- `spell`: spell name; -- `effect`: concise spell effect in the scene; -- `narrative_description`: short description of the spell cast in context. - -`caster` is not the table speaker. NPCs, monsters, and other DM-voiced -characters can be casters. - -## Source References - -The structured LLM response must include `source_refs` for each spell cast. -Each source reference uses the generic source-reference shape: - -- `source_id` -- `start_unit_id` -- `end_unit_id` - -The extractor copies those references into the generic artifact envelope -`source_refs` field. The durable `dnd.spell_cast` payload does not duplicate -source references. - -Source-reference IDs must match the source document and source-unit IDs -exactly. The validator chain rejects unknown source IDs, unknown unit IDs, and -reversed unit ranges. - -## Validators - -The extractor provides these deterministic validators by default, in order: - -- `dnd/spells/shape` -- `dnd/spells/source_refs` - -`dnd/spells/shape` rejects: - -- malformed JSON payloads with reason code `invalid_payload`; -- blank `caster`, `spell`, `effect`, or `narrative_description` fields with - reason code `missing_required_field`. - -`dnd/spells/source_refs` rejects: - -- candidates with no source references using reason code `missing_source_ref`; -- invalid source references using reason code `invalid_source_ref`. - -The source-reference validator uses the core `source.ValidateRef` behavior, so -its rejection message includes the underlying source-reference validation -error. - -## Capabilities - -The module declares these required capabilities: - -- `chunks` -- `source.transcript` - -The module declares this provided capability: - -- `dnd.spell_casts` - -A pipeline artifact lane can reference the extractor with: - -```yaml -artifacts: - spells: - extract: dnd/spells - merge: appendorder - normalize: noop -``` - -## Limits - -The current implementation covers only D&D spell-cast extraction. It does not -yet implement: - -- item extraction; -- NPC extraction; -- combat extraction; -- encounter extraction; -- broad D&D rules validation; -- a CLI `run` workflow. diff --git a/docs/integrations/json-output.md b/docs/integrations/json-output.md new file mode 100644 index 0000000..10f8ff5 --- /dev/null +++ b/docs/integrations/json-output.md @@ -0,0 +1,192 @@ +# JSON Output + +This document is the durable JSON output file-format contract produced by the +implemented `json` output module and written by the CLI. + +## Output Directory + +The CLI writes logical output files under: + +```text +// +``` + +The default output root is `./notarius-output`. Operational behavior is covered +in [Operations](../operations.md). + +## Files + +The `json` output module writes: + +- `index.json` +- `manifest.json` +- `artifacts/.json`, one file per approved artifact type +- `rejected.json` +- `warnings.json` + +Files are pretty-printed JSON with a trailing newline. + +## `index.json` + +Shape: + +```json +{ + "manifest_file": "manifest.json", + "artifact_files": [ + { + "artifact_type": "dnd.spell_cast", + "file": "artifacts/dnd.spell_cast.json" + } + ], + "rejected_file": "rejected.json", + "warnings_file": "warnings.json" +} +``` + +`artifact_files` is sorted by artifact type. It is empty when no artifacts are +approved. + +## `manifest.json` + +`manifest.json` contains a run manifest: + +```json +{ + "run_id": "run-123", + "pipeline_id": "dnd-session", + "pipeline_digest": "sha256:...", + "input_module": "seriatim", + "chunker": "generic", + "source_digests": ["sha256:..."], + "extractors": ["dnd/spells"], + "merger": "appendorder", + "normalizer": "noop", + "output_encoder": "json", + "artifact_lanes": [ + { + "id": "spells", + "extractor": "dnd/spells", + "merger": "appendorder", + "normalizer": "noop" + } + ], + "llm_profiles": [ + { + "id": "default", + "provider": "openai-compatible", + "model": "configured-model" + } + ], + "validation_status": "approved", + "started_at": "2026-01-01T00:00:00Z", + "completed_at": "2026-01-01T00:00:01Z" +} +``` + +Fields with empty values may be omitted by JSON encoding. + +`validation_status` is `approved` when no candidates were rejected and +`rejected` when one or more candidates were rejected. + +## Artifact Files + +Each artifact file has this shape: + +```json +{ + "artifact_type": "dnd.spell_cast", + "artifacts": [ + { + "extractor_key": "dnd/spells", + "artifact_type": "dnd.spell_cast", + "schema_version": "v1", + "payload": {}, + "source_refs": [ + { + "source_id": "session-alpha", + "start_unit_id": "seg-001", + "end_unit_id": "seg-001" + } + ] + } + ] +} +``` + +Artifact envelope fields: + +- `extractor_key`: extractor module key. +- `artifact_type`: artifact type. +- `schema_version`: artifact schema version. +- `payload`: artifact-type-specific JSON payload. +- `source_refs`: optional generic source references. +- `metadata`: optional artifact metadata. + +Artifact file names are produced by sanitizing the artifact type: + +- characters outside `A-Z`, `a-z`, `0-9`, `.`, `_`, and `-` become `_`; +- repeated `..` sequences are replaced; +- leading and trailing `.`, `_`, and `-` are trimmed; +- empty sanitized names are rejected. + +For current D&D spell-cast artifacts, the file is +`artifacts/dnd.spell_cast.json`. + +## `rejected.json` + +Shape: + +```json +{ + "rejected": [ + { + "candidate": { + "index": 0, + "extractor_key": "dnd/spells", + "artifact_type": "dnd.spell_cast", + "schema_version": "v1", + "payload": {}, + "source_refs": [] + }, + "validator_name": "dnd/spells/source_refs", + "reason_code": "missing_source_ref", + "message": "spell cast candidate must include at least one source ref" + } + ] +} +``` + +`rejected` is an empty array when no candidates are rejected. + +## `warnings.json` + +Shape: + +```json +{ + "warnings": [ + { + "scope": "output", + "reason_code": "example_warning", + "message": "warning message" + } + ] +} +``` + +`warnings` is an empty array when no warnings are reported. + +## Path Safety + +The output module returns slash-separated logical paths. The CLI also validates +logical output names before writing: + +- names must be non-empty; +- names must be relative; +- names must be clean; +- names must use `/`, not `\`; +- names must not contain `..`; +- resolved paths must stay under the run output directory. + +Durable writes are atomic per file. diff --git a/docs/integrations/openai-compatible.md b/docs/integrations/openai-compatible.md new file mode 100644 index 0000000..48b4819 --- /dev/null +++ b/docs/integrations/openai-compatible.md @@ -0,0 +1,128 @@ +# OpenAI-Compatible Structured Output + +This document describes the external LLM provider contract implemented by the +production Notarius LLM client. + +## Provider + +- Provider key: `openai-compatible` +- HTTP method: `POST` +- Endpoint: `/chat/completions` +- Request body: JSON +- Response mode: chat completions with structured JSON schema output + +`base_url` is trimmed of trailing slashes before `/chat/completions` is +appended. Configure provider settings in [Configuration](../config.md). + +## Request + +The client sends a JSON object with: + +```json +{ + "model": "configured-model", + "messages": [ + { + "role": "system", + "content": "..." + }, + { + "role": "user", + "content": "..." + } + ], + "response_format": { + "type": "json_schema", + "json_schema": { + "name": "schema_name", + "strict": true, + "schema": {} + } + } +} +``` + +Implemented request behavior: + +- `model` comes from the structured completion request when set, otherwise from + the configured LLM profile. +- `messages` must be non-empty; each role and content must be non-empty after + trimming. +- `response_format.type` is always `json_schema`. +- `response_format.json_schema.strict` is always `true`. +- `response_format.json_schema.name` and `schema` come from the extractor or + validator making the call. + +If an API key is configured, the client sends: + +```text +Authorization: Bearer +``` + +The client always sends `Content-Type: application/json`. + +## Response + +The client expects a JSON response with at least one choice: + +```json +{ + "model": "provider-model", + "choices": [ + { + "message": { + "content": "{\"field\":\"value\"}" + } + } + ], + "usage": { + "prompt_tokens": 10, + "completion_tokens": 5, + "total_tokens": 15 + } +} +``` + +`choices[0].message.content` may be either: + +- a JSON string whose contents are valid JSON; or +- raw JSON. + +The decoded content is unmarshaled into the caller-provided structured output +target. If `usage` is present, prompt, completion, and total token counts are +copied into the completion response. + +## Errors And Retries + +The client validates base URL, model, response schema name, response schema +JSON, messages, and output target before or during the call. + +Retryable failures: + +- HTTP request failure; +- response body read failure; +- HTTP `429`; +- HTTP `5xx`; +- malformed provider response envelope; +- missing choices; +- missing, empty, or invalid assistant JSON content; +- structured-output decode failure. + +Non-retryable provider status codes include non-`429` `4xx` responses. + +Provider error bodies are parsed for `error.message` or `message` when present. +Configured API key values and bearer-token values are redacted from returned +provider errors. + +## Timeouts And Concurrency + +The configured profile timeout is applied per provider request when greater +than zero. Context cancellation is respected. + +The production CLI wraps the provider client with the LLM scheduler. Effective +concurrency is described in [LLM runtime internals](../internal/llm.md). + +## Limits + +This contract documents only the fields the implemented client sends and reads. +Provider-specific extensions are ignored unless they affect those fields. diff --git a/docs/integrations/seriatim.md b/docs/integrations/seriatim.md index 69b98b1..0587362 100644 --- a/docs/integrations/seriatim.md +++ b/docs/integrations/seriatim.md @@ -1,7 +1,7 @@ # Seriatim Minimal Transcript JSON -This document describes the Seriatim input format currently accepted by the -`seriatim` input adapter. +This document is the external input contract for the implemented `seriatim` +input adapter. ## Adapter @@ -10,33 +10,39 @@ This document describes the Seriatim input format currently accepted by the - Unit kind: `transcript_segment` - Source format: `application/vnd.seriatim.minimal+json` -The adapter parses raw Seriatim JSON into a generic `SourceDocument`. It does -not add transcript-specific fields to core source or runner contracts. +The adapter parses raw Seriatim JSON into a generic source document. It owns +transcript-specific JSON parsing and metadata mapping; core source and pipeline +code stay source-format agnostic. ## Accepted Shape -The input must be a JSON object with top-level `metadata` and `segments` fields: +The input must be one JSON object with top-level `metadata` and `segments` +fields: ```json { "metadata": { "id": "session-alpha", - "title": "Synthetic session transcript" + "title": "Synthetic D&D spell session" }, "segments": [ { "id": "seg-001", "start": 0, - "end": 4.5, - "speaker": "Narrator", - "text": "The stone door opens." + "end": 4, + "speaker": "Aria", + "text": "Aria raises her holy symbol and casts Cure Wounds." } ] } ``` -Extra compatible fields are ignored. Multiple top-level JSON values are -rejected. +The maintained example is +[examples/seriatim-minimal-transcript.json](../../examples/seriatim-minimal-transcript.json). + +Top-level metadata entries are preserved. Other segment fields are ignored. + +Multiple top-level JSON values are rejected. ## Validation @@ -44,8 +50,11 @@ The adapter rejects: - empty raw input; - malformed JSON; +- top-level JSON that is not an object; - missing, null, or non-object `metadata`; - missing, null, non-array, or empty `segments`; +- segment values that are not objects; +- non-string `id`, `speaker`, or `text`; - empty segment IDs; - segment IDs with leading or trailing whitespace; - duplicate segment IDs; @@ -55,19 +64,16 @@ The adapter rejects: - `end` values before `start`; - missing or empty `text`. -Segment text may keep leading or trailing whitespace, but it must not be empty -after trimming. +Segment text is preserved as provided, but it must not be empty after trimming. ## Source Mapping -The adapter maps Seriatim input into the source model as follows: +The adapter maps input to `SourceDocument`: -- top-level `metadata` becomes `SourceDocument.Metadata`; -- `SourceDocument.Digest` is `sha256:` of the exact raw input bytes; -- `segment.id` becomes `SourceUnit.ID`; -- `segment.text` becomes `SourceUnit.Text`; -- each source unit has kind `transcript_segment`; -- segment `speaker`, `start`, and `end` are stored in source-unit metadata. +- `metadata` becomes `SourceDocument.Metadata`; +- `SourceDocument.Kind` is `transcript`; +- `SourceDocument.Format` is `application/vnd.seriatim.minimal+json`; +- `SourceDocument.Digest` is `sha256:` of the exact raw input bytes. `SourceDocument.ID` is selected in this order: @@ -76,6 +82,13 @@ The adapter maps Seriatim input into the source model as follows: 3. `metadata.source_id`, when it is a non-empty string after trimming; 4. `seriatim:`. +Each segment becomes one `SourceUnit`: + +- `segment.id` becomes `SourceUnit.ID`; +- `segment.text` becomes `SourceUnit.Text`; +- `SourceUnit.Kind` is `transcript_segment`; +- `speaker`, `start`, and `end` are stored in source-unit metadata. + ## Metadata Keys Seriatim unit metadata uses these keys: @@ -84,18 +97,17 @@ Seriatim unit metadata uses these keys: - `start`: `json.Number` start value; - `end`: `json.Number` end value. -The `internal/modules/input/seriatim` package provides typed accessors for -these metadata values. +The `internal/modules/input/seriatim` package exposes typed accessors for these +values. ## Capabilities -The module declares these provided capabilities for pipeline validation: +The module declares these provided capabilities: - `source.transcript` - `transcript.speaker` - `transcript.timestamps` -## Limits +## Compatibility Limit -Only the Seriatim minimal transcript shape described here is supported. Broader -Seriatim schema variants are not currently accepted as a compatibility contract. +This contract covers only the minimal transcript JSON shape described here. diff --git a/docs/internal/llm.md b/docs/internal/llm.md index 90284a8..949ce1c 100644 --- a/docs/internal/llm.md +++ b/docs/internal/llm.md @@ -58,7 +58,8 @@ The adapter accepts assistant content either as a JSON string containing JSON or as raw JSON content. It then unmarshals that content into the caller-provided target. -External wire-contract details belong in the OpenAI-compatible integration doc. +External wire-contract details belong in the +[OpenAI-compatible integration doc](../integrations/openai-compatible.md). ## Retries And Timeouts diff --git a/docs/internal/modules.md b/docs/internal/modules.md index 102df36..be818c2 100644 --- a/docs/internal/modules.md +++ b/docs/internal/modules.md @@ -78,8 +78,8 @@ Artifact type and schema version: - schema version: `v1` The extractor adds prompt and response-schema provenance to lane manifest -metadata. Durable artifact payload details belong in the D&D spell artifact -integration doc. +metadata. Durable artifact payload details belong in the +[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md). ## D&D Spell Validators @@ -139,8 +139,9 @@ Provides: - `encoded` -Durable output file shapes belong in the JSON output integration doc. Operator -behavior belongs in [Operations](../operations.md). +Durable output file shapes belong in the +[JSON output contract](../integrations/json-output.md). Operator behavior +belongs in [Operations](../operations.md). ## Production Registration