eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement raw module output contracts

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-07 18:58:23 +00:00
parent 9e3f8809b3
commit c05ecb58d8
33 changed files with 1166 additions and 1780 deletions
Download Patch File Download Diff File Expand all files Collapse all files

168
docs/integrations/json-output.md
View File

@@ -20,11 +20,11 @@ The `json` output module writes:
- `index.json`
- `manifest.json`
- `artifacts/<artifact-type>.json`, one file per approved artifact type
- `outputs/<lane-id>.json`, one file per normalized raw lane output
- `rejected.json`
- `warnings.json`
Files are pretty-printed JSON with a trailing newline.
Files are pretty-printed JSON with a trailing newline when the payload is JSON.
## `index.json`
@@ -33,10 +33,15 @@ Shape:
```json
{
"manifest_file": "manifest.json",
"artifact_files": [
"output_files": [
{
"artifact_type": "dnd.spell_cast",
"file": "artifacts/dnd.spell_cast.json"
"lane_id": "spells",
"media_type": "application/json",
"file": "outputs/spells.json",
"module_key": "noop",
"schema_id": "notarius.dnd.spells",
"schema_name": "notarius_dnd_spells_v1",
"schema_version": "v1"
}
],
"rejected_file": "rejected.json",
@@ -44,8 +49,14 @@ Shape:
}
```
`artifact_files` is sorted by artifact type. It is empty when no artifacts are
approved.
`output_files` is in normalized output order. Output file names are produced by
sanitizing the lane ID:
- characters outside `A-Z`, `a-z`, `0-9`, `.`, `_`, and `-` become `_`;
- repeated `..` sequences are replaced;
- leading and trailing `.`, `_`, and `-` are trimmed;
- empty sanitized names are rejected;
- two lanes that sanitize to the same output file are rejected.
## `manifest.json`
@@ -58,18 +69,6 @@ approved.
"pipeline_digest": "sha256:...",
"input_module": "seriatim",
"chunker": "dnd/scenes",
"module_metadata": {
"chunker": {
"prompt_id": "dnd.scenes",
"prompt_version": "v1",
"prompt_sha256": "sha256:...",
"response_schema_key": "dnd_scenes",
"response_schema_id": "notarius.dnd.scenes",
"response_schema_name": "notarius_dnd_scenes_v1",
"response_schema_version": "v1",
"response_schema_sha256": "sha256:..."
}
},
"source_digests": ["sha256:..."],
"extractors": ["dnd/spells"],
"merger": "appendorder",
@@ -83,13 +82,6 @@ approved.
"normalizer": "noop"
}
],
"llm_profiles": [
{
"id": "mistral-small-3",
"provider": "scriptorium",
"model": "configured-model"
}
],
"validation_status": "approved",
"started_at": "2026-01-01T00:00:00Z",
"completed_at": "2026-01-01T00:00:01Z"
@@ -99,60 +91,34 @@ approved.
Fields with empty values may be omitted by JSON encoding.
`source_digests` contains source document digests only. Bound references are
recorded separately under `references`, which contains provenance only:
target stage, lane ID when present, slot name, origin type and URI, digest,
media type, byte size, and binding source. Reference content is not written to
recorded separately under `references`, which contains provenance only: target
stage, lane ID when present, slot name, origin type and URI, digest, media
type, byte size, and binding source. Reference content is not written to
durable output.
Reference `stage` is `chunk`, `extract`, or `normalize`. `lane_id` is omitted
for chunk references and present for extract and normalize references.
When references are bound, the manifest section has this shape:
`validation_status` is `approved` when no raw outputs were rejected and
`rejected` when one or more raw outputs were rejected.
## Output Payload Files
Each normalized raw output is written to `outputs/<sanitized-lane-id>.json`.
For `application/json` payloads, the file contains the raw JSON payload
pretty-printed. Other media types are written as raw bytes with the media type
reported in `index.json`.
For the current D&D spell extractor, `outputs/spells.json` has this shape:
```json
{
"references": [
"spell_casts": [
{
"stage": "extract",
"lane_id": "spells",
"slot_name": "roster",
"origin_type": "file",
"origin_uri": "file:///absolute/path/roster.txt",
"digest": "sha256:...",
"media_type": "text/plain",
"size_bytes": 123,
"binding_source": "config"
}
]
}
```
Reference media types are inferred from file extensions and recorded as
canonical base media types. Unknown extensions are recorded as
`application/octet-stream`.
`module_metadata` is omitted when no singleton module provides metadata.
`validation_status` is `approved` when no candidates were rejected and
`rejected` when one or more candidates were rejected.
Top-level `module_metadata` is reserved for singleton pipeline modules
(`input`, `chunker`, and `output`). Lane-owned module metadata remains under
`artifact_lanes[].metadata`.
## Artifact Files
Each artifact file has this shape:
```json
{
"artifact_type": "dnd.spell_cast",
"artifacts": [
{
"extractor_key": "dnd/spells",
"artifact_type": "dnd.spell_cast",
"schema_version": "v1",
"payload": {},
"caster": "Aria",
"spell": "Cure Wounds",
"effect": "heals an injured ally",
"narrative_description": "Aria raises her holy symbol and casts Cure Wounds.",
"source_refs": [
{
"source_id": "session-alpha",
@@ -165,50 +131,20 @@ Each artifact file has this shape:
}
```
Artifact envelope fields:
- `extractor_key`: extractor module key.
- `artifact_type`: artifact type.
- `schema_version`: artifact schema version.
- `payload`: artifact-type-specific JSON payload.
- `source_refs`: optional generic source references.
- `metadata`: optional artifact metadata.
Artifact file names are produced by sanitizing the artifact type:
- characters outside `A-Z`, `a-z`, `0-9`, `.`, `_`, and `-` become `_`;
- repeated `..` sequences are replaced;
- leading and trailing `.`, `_`, and `-` are trimmed;
- empty sanitized names are rejected.
For current D&D spell-cast artifacts, the file is
`artifacts/dnd.spell_cast.json`.
## `rejected.json`
Shape:
```json
{
"rejected": [
{
"candidate": {
"index": 0,
"extractor_key": "dnd/spells",
"artifact_type": "dnd.spell_cast",
"schema_version": "v1",
"payload": {},
"source_refs": []
},
"validator_name": "dnd/spells/source_refs",
"reason_code": "missing_source_ref",
"message": "spell cast candidate must include at least one source ref"
}
]
"rejected": []
}
```
`rejected` is an empty array when no candidates are rejected.
When raw output validation rejects an output, entries use the
`contracts.RejectedOutput` shape, including stage, lane ID, module key,
validator name, reason code, message, attempt count, and optional diagnostic
artifact path.
## `warnings.json`
@@ -218,26 +154,12 @@ Shape:
{
"warnings": [
{
"scope": "output",
"reason_code": "example_warning",
"message": "warning message"
"scope": "extract",
"reason_code": "example",
"message": "human-readable warning"
}
]
}
```
`warnings` is an empty array when no warnings are reported.
## Path Safety
The output module returns slash-separated logical paths. The CLI also validates
logical output names before writing:
- names must be non-empty;
- names must be relative;
- names must be clean;
- names must use `/`, not `\`;
- names must not contain `..`;
- resolved paths must stay under the run output directory.
Durable writes are atomic per file.