eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: eric:4f057b99accf40de348f3be22ad915311859f228
Branches Tags
eric:main
eric:v0.4.0 eric:v0.3.0 eric:v0.2.0 eric:v0.1.0
...
compare: eric:a9d8505cdbc54fbab7c5112cdb6a0bd1af38281b
Branches Tags
eric:main
eric:v0.4.0 eric:v0.3.0 eric:v0.2.0 eric:v0.1.0

7 Commits
4f057b99ac ... a9d8505cdb

Author SHA1 Message Date
Eric Rakestraw
a9d8505cdb Document raw pipeline completion 2026-07-07 19:32:15 +00:00
Eric Rakestraw
7c95791e94 Update raw output files and manifests 2026-07-07 19:27:28 +00:00
Eric Rakestraw
aa14faa3cb Migrate production modules to raw outputs 2026-07-07 19:23:35 +00:00
Eric Rakestraw
cc6b050367 Implement runner retries and raw validation 2026-07-07 19:19:04 +00:00
Eric Rakestraw
bcedf19a08 Add merge references and retry config 2026-07-07 19:10:55 +00:00
Eric Rakestraw
c05ecb58d8 Implement raw module output contracts 2026-07-07 18:58:23 +00:00
Eric Rakestraw
9e3f8809b3 Implement integer source units and chunk payloads 2026-07-07 18:34:23 +00:00
78 changed files with 3494 additions and 3254 deletions
Expand all files Collapse all files

21
docs/cli.md
View File

@@ -49,22 +49,22 @@ Flags:
use one Scriptorium profile ID.
- `--session-id id`: pass a stable prompt session identifier through LLM-backed
module calls.
- `--reference selector=path`: bind a reference path to a chunk, extractor, or
normalizer reference slot. Repeatable.
- `--reference selector=path`: bind a reference path to a chunk, extractor,
merger, or normalizer reference slot. Repeatable.
- `--without-reference selector`: remove a configured optional reference binding.
Repeatable. It accepts the same selector forms as `--reference`, without
`=path`.
On success, the command prints the completed pipeline ID, approved and rejected
artifact counts, and the output directory. If the run completes with warnings,
On success, the command prints the completed pipeline ID, normalized output and
rejected output counts, and the output directory. If the run completes with warnings,
the warning count is printed to stderr.
Reference flags are resolved against selected chunk, extractor, and normalizer
Reference flags are resolved against selected chunk, extractor, merger, and normalizer
targets before the run starts. Flat slot names are accepted only when exactly
one selected target declares that slot. Bound reference files are read before
pipeline work starts, validated as UTF-8 text, and recorded as provenance for
the target that declares the slot. Runtime reference content is passed to the
chunker, extractor, or normalizer target that declares the slot. Notarius infers
chunker, extractor, merger, or normalizer target that declares the slot. Notarius infers
reference media types from file extensions for provenance and for optional slot
checks. Reference content is not written to diagnostics, logs, errors, or
manifests.
@@ -81,9 +81,11 @@ Reference binding precedence is:
- `slot=path`: valid when exactly one selected target declares `slot`;
- `chunk.slot=path`: target the chunker;
- `lane.slot=path`: valid when exactly one selected extractor or normalizer in
that lane declares `slot`;
- `merge.slot=path`: valid when exactly one selected merger declares `slot`;
- `lane.slot=path`: valid when exactly one selected extractor, merger, or
normalizer in that lane declares `slot`;
- `lane.extract.slot=path`: target a lane extractor;
- `lane.merge.slot=path`: target a lane merger;
- `lane.normalize.slot=path`: target a lane normalizer.
Use `slot=path` when the selected targets declare the slot unambiguously:
@@ -105,7 +107,7 @@ go run ./cmd/notarius run dnd-session \
--reference spells.extract.glossary=./campaign-glossary.txt
```
The same grammar can target chunk and normalize slots when the configured
The same grammar can target chunk, merge, and normalize slots when the configured
modules declare them:
```sh
@@ -113,6 +115,7 @@ go run ./cmd/notarius run dnd-session \
--config path/to/config.yml \
--input examples/seriatim-minimal-transcript.json \
--reference chunk.scene_guide=./campaign-scenes.txt \
--reference spells.merge.merge_notes=./merge-notes.txt \
--reference spells.normalize.normalization_notes=./normalization-notes.txt
```

36
docs/config.md
View File

@@ -141,13 +141,13 @@ against the production module catalog and fail fast for unknown or incompatible
module keys.
Reference bindings are validated against reference slots declared by eligible
chunk, extract, and normalize targets during pipeline resolution. Required slots
chunk, extract, merge, and normalize targets during pipeline resolution. Required slots
must be bound after config defaults, target-local references, lane-level
compatibility bindings, and run-time `--reference` or `--without-reference`
overrides are applied. Config-relative paths are resolved relative to the config
file; CLI reference paths are resolved relative to the current working
directory. Materialized bound files must be UTF-8 text. Materialized reference
provenance is recorded for chunk, extractor, and normalizer targets, and runtime
provenance is recorded for chunk, extractor, merger, and normalizer targets, and runtime
reference content is passed to the target that declares the slot. Reference
media types are inferred from file extensions, recorded as canonical base media
types, and checked only when a module declares `AcceptedMediaTypes`; unknown
@@ -156,8 +156,8 @@ written to diagnostics, logs, errors, or manifests.
Pipeline-level `references` are defaults. They are valid when at least one
eligible target in the full configured pipeline declares the slot, including
chunk, extractor, and normalizer targets. During a run, they apply only to the
selected targets that declare the slot:
chunk, extractor, merger, and normalizer targets. During a run, they apply only
to the selected targets that declare the slot:
```yaml
pipelines:
@@ -192,15 +192,17 @@ pipelines:
party: ./campaign/session-party.txt
```
`chunk.references` and `normalize.references` are accepted in object-form
bindings. They override pipeline-level defaults for slots declared by the chunk
or normalizer module. Extractor-local references apply only to the extractor,
and normalizer-local references apply only to the normalizer.
`chunk.references`, `merge.references`, and `normalize.references` are accepted
in object-form bindings. They override pipeline-level defaults for slots
declared by that target module. Extractor-local references apply only to the
extractor, merger-local references apply only to the merger, and
normalizer-local references apply only to the normalizer.
Target-local reference fields use the same map shape at:
- `pipelines.<id>.chunk.references`
- `pipelines.<id>.artifacts.<lane>.extract.references`
- `pipelines.<id>.artifacts.<lane>.merge.references`
- `pipelines.<id>.artifacts.<lane>.normalize.references`
Each binding is valid only when that target module declares the slot.
@@ -226,13 +228,17 @@ Binding fields:
- `module`: module key.
- `llm_profile`: optional Scriptorium profile ID. Empty or omitted lets the
Scriptorium prompt default select the profile.
- `retries`: non-negative retry count for extra runtime attempts after the
first attempt. The runner applies retries to `chunk`, `extract`, `merge`, and
`normalize` bindings.
- `options`: optional module-specific settings.
- `references`: optional reference bindings. Supported only for `chunk`,
`extract`, and `normalize` bindings. `input`, `merge`, validator, and
`extract`, `merge`, and `normalize` bindings. `input`, validator, and
`output` bindings reject this field during validation.
The `--llm-profile` run flag overrides every effective LLM-capable module
binding to use one Scriptorium profile ID.
binding to use one Scriptorium profile ID: chunk, every selected lane extract,
merge, and normalize binding.
## Implemented Production Modules
@@ -241,10 +247,10 @@ binding to use one Scriptorium profile ID.
| input | `seriatim` | Reads Seriatim transcript JSON. |
| chunk | `generic` | Splits source units into ordered chunks. |
| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. |
| extract | `dnd/spells` | Extracts `dnd.spell_cast` artifacts. |
| merge | `appendorder` | Keeps candidates in append order. |
| normalize | `noop` | Passes merged artifacts through unchanged. |
| output | `json` | Produces JSON output files. |
| extract | `dnd/spells` | Extracts D&D spell raw outputs. |
| merge | `appendorder` | Merges JSON raw extract outputs in chunk order. |
| normalize | `noop` | Passes merged raw outputs through unchanged. |
| output | `json` | Produces JSON output files for normalized `application/json` lanes. |
The `generic` chunker accepts:
@@ -302,6 +308,6 @@ Pipeline resolution additionally checks:
- required module keys are present;
- module keys are registered for the expected slot;
- module capability requirements are satisfied;
- bound reference slots are declared by selected chunk, extractor, or
- bound reference slots are declared by selected chunk, extractor, merger, or
normalizer targets;
- required reference slots are bound for selected targets.

163
docs/integrations/dnd-spell-artifacts.md
View File

@@ -1,91 +1,27 @@
# D&D Spell-Cast Artifacts
# D&D Spell Raw Output
This document is the durable artifact contract for approved
`dnd.spell_cast` artifacts produced by the implemented `dnd/spells` extractor.
This document is the durable raw output contract for the implemented
`dnd/spells` extractor.
## Artifact Identity
## Identity
- Extractor key: `dnd/spells`
- Artifact type: `dnd.spell_cast`
- Schema version: `v1`
- Prompt ID: `dnd.spells`
- Response schema key: `dnd_spells`
- Response schema ID: `notarius.dnd.spells`
- Response schema name: `notarius_dnd_spells_v1`
- Response schema version: `v1`
- Media type: `application/json`
The extractor requires source chunks and transcript source capability. It
returns generic artifact candidates that are serialized by the JSON output
module.
returns the structured LLM response as raw JSON. The default `appendorder`
merger passes a single chunk output through and concatenates multiple
`spell_casts` arrays in chunk order. The default `noop` normalizer passes the
merge output through unchanged.
The extractor accepts optional UTF-8 text references:
## Output Shape
- `roster`: campaign roster or player-character notes.
- `glossary`: campaign glossary or spell/name notes.
References are supporting disambiguation material only. They are not source
evidence and are not addressable through `source_refs`.
## Artifact Envelope
Approved artifacts use the generic artifact envelope documented in
[JSON Output](json-output.md#artifact-files):
```json
{
"extractor_key": "dnd/spells",
"artifact_type": "dnd.spell_cast",
"schema_version": "v1",
"payload": {
"caster": "Aria",
"spell": "Cure Wounds",
"effect": "heals an injured ally",
"narrative_description": "Aria raises her holy symbol and casts Cure Wounds."
},
"source_refs": [
{
"source_id": "session-alpha",
"start_unit_id": "seg-001",
"end_unit_id": "seg-001"
}
]
}
```
## Payload Fields
The `payload` object contains:
- `caster`: in-world character or creature casting the spell;
- `spell`: spell name;
- `effect`: concise spell effect in the scene;
- `narrative_description`: short description of the spell cast in context.
All payload fields are strings and must be non-empty after trimming.
`caster` is the in-world caster, not the transcript speaker.
## Source References
Source references live on the artifact envelope as `source_refs`; they are not
duplicated inside the `payload`.
Each source reference uses the generic source-reference shape:
- `source_id`
- `start_unit_id`
- `end_unit_id`
Validation requires:
- at least one source reference;
- non-empty source ID and unit IDs;
- source ID matching the source document ID;
- start and end unit IDs existing in the source document;
- start unit appearing before or at the same position as end unit.
## Structured LLM Response Shape
The extractor asks the LLM for this top-level response shape:
For a single chunk, `lanes/spells.json` has this shape:
```json
{
@@ -98,8 +34,8 @@ The extractor asks the LLM for this top-level response shape:
"source_refs": [
{
"source_id": "session-alpha",
"start_unit_id": "seg-001",
"end_unit_id": "seg-001"
"start_unit_id": 1,
"end_unit_id": 1
}
]
}
@@ -109,31 +45,63 @@ The extractor asks the LLM for this top-level response shape:
`spell_casts` must be present. It may be empty when no spell casts are found.
The response schema asset is embedded at
`internal/modules/extract/dnd/spells/assets/schemas/dnd_spells.v1.json`.
For multiple chunks with the default merger, the lane output keeps the same
top-level shape and concatenates `spell_casts` in chunk order:
## Validators
```json
{
"spell_casts": [
{
"caster": "Aria",
"spell": "Cure Wounds",
"effect": "heals an injured ally",
"narrative_description": "Aria raises her holy symbol and casts Cure Wounds.",
"source_refs": [
{
"source_id": "session-alpha",
"start_unit_id": 1,
"end_unit_id": 1
}
]
}
]
}
```
The extractor supplies two deterministic validators by default:
## Spell-Cast Fields
- `dnd/spells/shape`
- `dnd/spells/source_refs`
Each spell cast contains:
Rejection reason codes:
- `caster`: in-world character or creature casting the spell;
- `spell`: spell name;
- `effect`: concise spell effect in the scene;
- `narrative_description`: short description of the spell cast in context;
- `source_refs`: transcript source references supplied by the model.
- `invalid_payload`: payload JSON cannot be decoded as a spell-cast payload.
- `missing_required_field`: `caster`, `spell`, `effect`, or
`narrative_description` is blank.
- `missing_source_ref`: candidate has no source references.
- `invalid_source_ref`: at least one source reference fails generic source
reference validation.
`caster` is the in-world caster, not the transcript speaker.
Warning reason codes:
## Source References
- `spell_not_near_source`: the extracted spell name was not found in the cited
source text.
Each source reference uses the generic source-reference shape:
Rejected candidates are written to `rejected.json` by the JSON output module.
- `source_id`
- `start_unit_id`
- `end_unit_id`
The extractor prompt and schema use integer `start_unit_id` and `end_unit_id`
values matching source-unit IDs.
## References
The extractor accepts optional UTF-8 text references:
- `players`
- `party`
- `glossary`
- `roster`, a deprecated compatibility alias for `party`
References are supporting disambiguation material only. They are not source
evidence and are not addressable through `source_refs`.
## Manifest Metadata
@@ -158,8 +126,3 @@ manifest metadata:
```
Raw prompt and schema content are not included in manifest metadata.
## Compatibility Limit
This contract covers only `dnd.spell_cast` artifacts produced by the
implemented spell-cast extractor.

184
docs/integrations/json-output.md
View File

@@ -20,11 +20,11 @@ The `json` output module writes:
- `index.json`
- `manifest.json`
- `artifacts/<artifact-type>.json`, one file per approved artifact type
- `lanes/<lane-id>.json`, one file per normalized raw lane output
- `rejected.json`
- `warnings.json`
Files are pretty-printed JSON with a trailing newline.
Files are pretty-printed JSON with a trailing newline when the payload is JSON.
## `index.json`
@@ -33,10 +33,15 @@ Shape:
```json
{
"manifest_file": "manifest.json",
"artifact_files": [
"output_files": [
{
"artifact_type": "dnd.spell_cast",
"file": "artifacts/dnd.spell_cast.json"
"lane_id": "spells",
"media_type": "application/json",
"file": "lanes/spells.json",
"module_key": "noop",
"schema_id": "notarius.dnd.spells",
"schema_name": "notarius_dnd_spells_v1",
"schema_version": "v1"
}
],
"rejected_file": "rejected.json",
@@ -44,8 +49,14 @@ Shape:
}
```
`artifact_files` is sorted by artifact type. It is empty when no artifacts are
approved.
`output_files` is sorted by lane ID. Output file names are produced by
sanitizing the lane ID:
- characters outside `A-Z`, `a-z`, `0-9`, `.`, `_`, and `-` become `_`;
- repeated `..` sequences are replaced;
- leading and trailing `.`, `_`, and `-` are trimmed;
- empty sanitized names are rejected;
- two lanes that sanitize to the same output file are rejected.
## `manifest.json`
@@ -58,18 +69,6 @@ approved.
"pipeline_digest": "sha256:...",
"input_module": "seriatim",
"chunker": "dnd/scenes",
"module_metadata": {
"chunker": {
"prompt_id": "dnd.scenes",
"prompt_version": "v1",
"prompt_sha256": "sha256:...",
"response_schema_key": "dnd_scenes",
"response_schema_id": "notarius.dnd.scenes",
"response_schema_name": "notarius_dnd_scenes_v1",
"response_schema_version": "v1",
"response_schema_sha256": "sha256:..."
}
},
"source_digests": ["sha256:..."],
"extractors": ["dnd/spells"],
"merger": "appendorder",
@@ -83,13 +82,6 @@ approved.
"normalizer": "noop"
}
],
"llm_profiles": [
{
"id": "mistral-small-3",
"provider": "scriptorium",
"model": "configured-model"
}
],
"validation_status": "approved",
"started_at": "2026-01-01T00:00:00Z",
"completed_at": "2026-01-01T00:00:01Z"
@@ -99,65 +91,47 @@ approved.
Fields with empty values may be omitted by JSON encoding.
`source_digests` contains source document digests only. Bound references are
recorded separately under `references`, which contains provenance only:
target stage, lane ID when present, slot name, origin type and URI, digest,
media type, byte size, and binding source. Reference content is not written to
recorded separately under `references`, which contains provenance only: target
stage, lane ID when present, slot name, origin type and URI, digest, media
type, byte size, and binding source. Reference content is not written to
durable output.
Reference `stage` is `chunk`, `extract`, or `normalize`. `lane_id` is omitted
for chunk references and present for extract and normalize references.
Reference `stage` is `chunk`, `extract`, `merge`, or `normalize`. `lane_id` is
omitted for chunk references and present for extract, merge, and normalize
references.
When references are bound, the manifest section has this shape:
`validation_status` is `approved` when no raw outputs were rejected and
`rejected` when one or more raw outputs were rejected.
`normalized_outputs` summarizes each normalized lane output without embedding
payload bytes. Entries include lane ID, normalizer module key, source ID, media
type, and response schema provenance where available.
`rejected_outputs` summarizes rejected module outputs without embedding raw
payload bytes. Entries include stage, lane, module, chunk, validator or reason,
message, attempt count, and optional diagnostic artifact path.
## Output Payload Files
Each normalized raw output is written to `lanes/<sanitized-lane-id>.json`.
The JSON output encoder accepts only `application/json` normalized outputs. The
file contains the raw JSON payload pretty-printed.
For the current D&D spell extractor, `lanes/spells.json` has this shape:
```json
{
"references": [
"spell_casts": [
{
"stage": "extract",
"lane_id": "spells",
"slot_name": "roster",
"origin_type": "file",
"origin_uri": "file:///absolute/path/roster.txt",
"digest": "sha256:...",
"media_type": "text/plain",
"size_bytes": 123,
"binding_source": "config"
}
]
}
```
Reference media types are inferred from file extensions and recorded as
canonical base media types. Unknown extensions are recorded as
`application/octet-stream`.
`module_metadata` is omitted when no singleton module provides metadata.
`validation_status` is `approved` when no candidates were rejected and
`rejected` when one or more candidates were rejected.
Top-level `module_metadata` is reserved for singleton pipeline modules
(`input`, `chunker`, and `output`). Lane-owned module metadata remains under
`artifact_lanes[].metadata`.
## Artifact Files
Each artifact file has this shape:
```json
{
"artifact_type": "dnd.spell_cast",
"artifacts": [
{
"extractor_key": "dnd/spells",
"artifact_type": "dnd.spell_cast",
"schema_version": "v1",
"payload": {},
"caster": "Aria",
"spell": "Cure Wounds",
"effect": "heals an injured ally",
"narrative_description": "Aria raises her holy symbol and casts Cure Wounds.",
"source_refs": [
{
"source_id": "session-alpha",
"start_unit_id": "seg-001",
"end_unit_id": "seg-001"
"start_unit_id": 1,
"end_unit_id": 1
}
]
}
@@ -165,50 +139,20 @@ Each artifact file has this shape:
}
```
Artifact envelope fields:
- `extractor_key`: extractor module key.
- `artifact_type`: artifact type.
- `schema_version`: artifact schema version.
- `payload`: artifact-type-specific JSON payload.
- `source_refs`: optional generic source references.
- `metadata`: optional artifact metadata.
Artifact file names are produced by sanitizing the artifact type:
- characters outside `A-Z`, `a-z`, `0-9`, `.`, `_`, and `-` become `_`;
- repeated `..` sequences are replaced;
- leading and trailing `.`, `_`, and `-` are trimmed;
- empty sanitized names are rejected.
For current D&D spell-cast artifacts, the file is
`artifacts/dnd.spell_cast.json`.
## `rejected.json`
Shape:
```json
{
"rejected": [
{
"candidate": {
"index": 0,
"extractor_key": "dnd/spells",
"artifact_type": "dnd.spell_cast",
"schema_version": "v1",
"payload": {},
"source_refs": []
},
"validator_name": "dnd/spells/source_refs",
"reason_code": "missing_source_ref",
"message": "spell cast candidate must include at least one source ref"
}
]
"rejected": []
}
```
`rejected` is an empty array when no candidates are rejected.
When raw output validation rejects an output, entries use the
`contracts.RejectedOutput` shape, including stage, lane ID, module key,
validator name, reason code, message, attempt count, and optional diagnostic
artifact path.
## `warnings.json`
@@ -218,26 +162,12 @@ Shape:
{
"warnings": [
{
"scope": "output",
"reason_code": "example_warning",
"message": "warning message"
"scope": "extract",
"reason_code": "example",
"message": "human-readable warning"
}
]
}
```
`warnings` is an empty array when no warnings are reported.
## Path Safety
The output module returns slash-separated logical paths. The CLI also validates
logical output names before writing:
- names must be non-empty;
- names must be relative;
- names must be clean;
- names must use `/`, not `\`;
- names must not contain `..`;
- resolved paths must stay under the run output directory.
Durable writes are atomic per file.

10
docs/integrations/seriatim.md
View File

@@ -28,7 +28,7 @@ output that provides the same required segment fields.
},
"segments": [
{
"id": "seg-001",
"id": 1,
"start": 0,
"end": 4,
"speaker": "Aria",
@@ -56,10 +56,9 @@ The adapter rejects:
- missing, null, or non-object `metadata`;
- missing, null, non-array, or empty `segments`;
- segment values that are not objects;
- segment `id` values that are neither strings nor numbers;
- segment `id` values that are not positive integer JSON numbers or numeric
strings;
- non-string `speaker` or `text`;
- empty segment IDs;
- segment IDs with leading or trailing whitespace;
- duplicate segment IDs;
- missing or empty `speaker`;
- missing, empty, invalid, non-finite, or negative `start`;
@@ -87,8 +86,7 @@ The adapter maps input to `SourceDocument`:
Each segment becomes one `SourceUnit`:
- `segment.id` becomes `SourceUnit.ID`; numeric IDs are converted to their JSON
number text, so `1` becomes `"1"`;
- `segment.id` becomes integer `SourceUnit.ID`;
- `segment.text` becomes `SourceUnit.Text`;
- `SourceUnit.Kind` is `transcript_segment`;
- `speaker`, `start`, and `end` are stored in source-unit metadata.

9
docs/internal/llm.md
View File

@@ -15,7 +15,9 @@ CompleteStructured(ctx, request, out) (response, error)
The request contains prompt ID/version, profile ID, session ID, prompt input
materials, and variables. The caller supplies a pointer target for decoded
structured output.
structured output. The response also carries the raw structured output bytes
returned by the runtime so modules can preserve raw payloads in pipeline stage
outputs.
Modules that call the LLM own their prompts, schemas, prompt IDs, validators,
and domain-specific interpretation. Provider adapters should not contain
@@ -41,6 +43,10 @@ The runtime records the actual selected Scriptorium profile, provider, and model
used during execution. Manifest population does not rely on a precomputed
profile ID before pipeline execution.
Explicit profile validation and `--llm-profile` overrides apply to LLM-capable
pipeline stages: chunk, extract, merge, and normalize. Input, output, and
validator bindings are not part of the current production LLM profile scope.
## Scriptorium Adapter
`ScriptoriumClient` implements `contracts.StructuredLLMClient` by converting
@@ -55,6 +61,7 @@ Notarius prompt requests into Scriptorium `RunRequest` values. It:
- lets Scriptorium render prompts, call the configured provider, and validate
structured output;
- unmarshals successful JSON into the caller-provided target;
- returns the validated raw structured output bytes to the caller;
- maps token usage and selected profile/model metadata into the Notarius
response and manifest profile recorder.

92
docs/internal/modules.md
View File

@@ -20,7 +20,7 @@ A production module package should provide:
Module specs should describe capabilities accurately. Resolution uses specs to
reject incompatible pipelines before execution.
Chunk, extract, and normalize modules that accept auxiliary reference material
Chunk, extract, merge, and normalize modules that accept auxiliary reference material
must declare slots through both `ReferenceSlots()` and
`ModuleSpec().ReferenceSlots`. The runtime slot list and registry metadata
should match so config validation can inspect slots without constructing module
@@ -31,10 +31,10 @@ must still be UTF-8 text. When a slot declares accepted media types, Notarius
compares the canonical base media type inferred from the file extension,
case-insensitively and without parameters.
The resolver materializes reference content for chunk, extractor, and
The resolver materializes reference content for chunk, extractor, merger, and
normalizer targets. Runtime delivery uses `contracts.ChunkRequest.References`,
`contracts.ExtractionRequest.References`, and
`contracts.NormalizeRequest.References`. Reference material is not source
`contracts.ExtractionRequest.References`, `contracts.MergeRequest.References`,
and `contracts.NormalizeRequest.References`. Reference material is not source
evidence and must not be converted into `SourceRef` values. If a module prompt
uses references, pass them as prompt input materials through the structured LLM
request. Prompt metadata hashes remain based on prompt asset source, not
@@ -48,7 +48,7 @@ Common D&D prompt fragments, reference slot helpers, prompt input assembly, and
reference rendering live under `internal/modules/sharedassets/dnd`. Module
contracts should expose prompt IDs, versions, input material names, and
non-secret prompt/schema hashes through manifest metadata; they should not
expose Scriptorium public types through chunk, extract, or normalize contracts.
expose Scriptorium public types through chunk, extract, merge, or normalize contracts.
Chunk modules receive the structured LLM client, configured Scriptorium profile
ID, prompt session ID, and raw source input material through
@@ -60,6 +60,10 @@ Normalize modules receive the structured LLM client, configured Scriptorium
profile ID, prompt session ID, and reference material through
`contracts.NormalizeRequest` when they need model-backed reconciliation.
Merge modules receive the structured LLM client, configured Scriptorium profile
ID, prompt session ID, raw source input material, and reference material through
`contracts.MergeRequest` when they need model-backed merge behavior.
## `seriatim` Input
Package: `internal/modules/input/seriatim`
@@ -84,9 +88,10 @@ The `generic` chunker splits source units into ordered chunks. It validates the
source document, clones source units, assigns chunk IDs such as `chunk-000001`,
and records chunk metadata for start unit, end unit, and unit count.
The pipeline runner canonicalizes chunk units from the source document by ID
before extractors and mergers run. Chunker-owned context should stay in
`SourceChunk.Metadata`.
The pipeline runner canonicalizes chunk units from the source document by
integer ID before extractors and mergers run. Chunkers also populate chunk
start and end unit IDs, content bytes, and media type. Chunker-owned context
should stay in `SourceChunk.Metadata`.
Options:
@@ -126,12 +131,11 @@ Options: none. Non-empty options are rejected.
The chunker enforces full source-unit coverage from the first source unit to the
last, sequential contiguous scenes, and no overlap. Its LLM-facing schema uses
integer `start_unit_id` and `end_unit_id` values as 1-based source-unit numbers;
the module canonicalizes valid integer references to source-unit IDs before
producing chunks. It assigns chunk IDs such as `scene-000001` and stores scene
metadata including title, primary mode, participants, summary, boundary note,
confidence, boundary unit IDs, and unit count. Boundary caveats become warnings
with reason code
integer `start_unit_id` and `end_unit_id` values matching source-unit IDs. It
assigns chunk IDs such as `scene-000001`, emits JSON chunk content, and stores
scene metadata including title, primary mode, participants, summary, boundary
note, confidence, boundary unit IDs, and unit count. Boundary caveats become
warnings with reason code
`scene_boundary_caveat`. Whitespace-only caveats are treated as malformed
structured output rather than silently dropped.
@@ -144,14 +148,12 @@ text, or secrets.
Package: `internal/modules/extract/dnd/spells`
The `dnd/spells` extractor owns D&D spell-cast artifact semantics. It supplies
the embedded Scriptorium prompt ID, prompt version, transcript and reference
input materials, response schema, and session ID to the runtime; converts
spell-cast responses into artifact candidates; and supplies deterministic
validators.
The `dnd/spells` extractor owns D&D spell-cast extraction semantics. It
supplies the embedded Scriptorium prompt ID, prompt version, transcript and
reference input materials, response schema, and session ID to the runtime; then
returns the structured LLM `spell_casts` response as raw JSON.
Its LLM-facing source-reference schema uses integer `start_unit_id` and
`end_unit_id` values as 1-based source-unit numbers; the module canonicalizes
valid integer references to source-unit IDs before validation and output.
`end_unit_id` values matching source-unit IDs.
Its prompt definition lives under `assets/prompts` and its schema under
`assets/schemas`. Shared reusable D&D prompt fragments are provided by
@@ -167,15 +169,16 @@ Provides:
- `dnd.spell_casts`
Artifact type and schema version:
Response schema identity:
- artifact type: `dnd.spell_cast`
- schema ID: `notarius.dnd.spells`
- schema name: `notarius_dnd_spells_v1`
- schema version: `v1`
The extractor adds prompt and response-schema provenance to lane manifest
metadata under `artifact_lanes[].metadata.extractor`. Durable artifact payload
metadata under `artifact_lanes[].metadata.extractor`. Durable raw output
details belong in the
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
[D&D spell raw output contract](../integrations/dnd-spell-artifacts.md).
The `dnd/scenes` chunker and `dnd/spells` extractor declare optional `players`,
`party`, and `glossary` reference slots accepting UTF-8 plain text, Markdown,
@@ -183,32 +186,16 @@ YAML, or JSON. They also accept `roster` as a deprecated compatibility alias for
`party`. Their prompts frame references as supporting disambiguation material
only; spell-cast artifacts must still be grounded in the source transcript.
## D&D Spell Validators
The spell extractor returns two built-in validators:
- `dnd/spells/shape`: rejects malformed payloads and missing required fields.
- `dnd/spells/source_refs`: rejects candidates without valid source references.
It also emits a warning when the extracted spell name is not found in the
cited source text.
Reason codes include:
- `invalid_payload`
- `missing_required_field`
- `missing_source_ref`
- `invalid_source_ref`
- `spell_not_near_source`
These validators are supplied by the extractor when no validators are configured
for the lane.
## `appendorder` Merger
Package: `internal/modules/merge/appendorder`
The `appendorder` merger clones and appends candidates in chunk order. It does
not deduplicate or reconcile candidates.
The `appendorder` merger preserves chunk order for raw extract outputs. A
single JSON extract output is passed through as the merge output. Multiple JSON
object outputs with one common top-level array field are merged by concatenating
that array field in chunk order. Other valid JSON shapes are merged as a JSON
array of decoded values in chunk order. Non-JSON media types and invalid JSON
are rejected.
Provides:
@@ -218,7 +205,7 @@ Provides:
Package: `internal/modules/normalize/noop`
The `noop` normalizer clones merged candidates and returns them unchanged.
The `noop` normalizer clones the raw merge output and returns it unchanged.
Requires:
@@ -232,9 +219,10 @@ Provides:
Package: `internal/modules/output/json`
The `json` output encoder converts approved artifacts, rejected artifacts,
warnings, and the run manifest into logical JSON output files. It groups
approved artifacts by artifact type and sanitizes artifact-type file names.
The `json` output encoder converts normalized raw outputs, rejected raw outputs,
warnings, and the run manifest into logical JSON output files. It writes one
payload file per lane under `lanes/` and sanitizes lane IDs for file names.
Normalized output payloads must be valid `application/json`.
Requires:
@@ -262,7 +250,7 @@ When adding a module, keep source-format and extraction-domain boundaries clear:
- input modules may know external source formats;
- extract modules may know artifact semantics and prompt/schema assets;
- merge and normalize modules own candidate combination and reconciliation;
- merge and normalize modules own raw output combination and reconciliation;
- output modules own serialization, not diagnostics or CLI reporting.
Update [Development](../policy/development.md), [Configuration](../config.md),

4
docs/internal/overview.md
View File

@@ -21,8 +21,8 @@ belongs in modules, not in command handlers.
## Core Packages
- `internal/core/artifacts`: artifact candidates, approved artifacts, rejected
artifacts, validation decisions, and run manifests.
- `internal/core/artifacts`: run manifests and legacy artifact serialization
shapes retained while pipeline handoff contracts use raw outputs.
- `internal/core/config`: defaults, YAML config parsing, environment overrides,
validation, redaction, and resolved pipeline config.
- `internal/core/diagnostics`: per-run diagnostics directory creation,

126
docs/internal/pipeline.md
View File

@@ -33,18 +33,19 @@ The CLI writes the resolved pipeline and digest to diagnostics.
Pipeline profiles and artifact lanes may include reference binding maps keyed by
reference slot name. During resolution, pipeline-level bindings act as defaults
for selected chunk, extractor, and normalizer targets that declare the slot;
target-local bindings override or add bindings for that target. Runtime
for selected chunk, extractor, merger, and normalizer targets that declare the
slot; target-local bindings override or add bindings for that target. Runtime
`--reference` requests override target config bindings, and runtime unbinds
remove optional target bindings. Flat runtime slot names are resolved only when
exactly one selected target declares the slot; otherwise the CLI requires a more
specific selector such as `chunk.slot`, `lane.extract.slot`, or
`lane.normalize.slot`. Resolution validates bindings against the declaring
target specs and stores the bindings in target-aware resolved reference holders.
It does not read reference files or include reference bytes in source digests.
specific selector such as `chunk.slot`, `lane.extract.slot`,
`lane.merge.slot`, or `lane.normalize.slot`. Resolution validates bindings
against the declaring target specs and stores the bindings in target-aware
resolved reference holders. It does not read reference files or include
reference bytes in source digests.
During run preparation, resolved file references for chunk, extractor, and
normalizer targets are materialized before any LLM-backed pipeline work. Config
During run preparation, resolved file references for chunk, extractor, merger,
and normalizer targets are materialized before any LLM-backed pipeline work. Config
bindings resolve relative to the config file, and CLI bindings resolve relative
to the current working directory. Materialization accepts UTF-8 text files,
computes `sha256:` content digests, records file origins, infers canonical base
@@ -55,8 +56,8 @@ empty bound files. Media-type acceptance is checked only when a slot declares
manifests. The CLI writes provenance-only resolved reference diagnostics, and
the run manifest records target-stage reference provenance separately from
source digests. Runtime reference content is passed to the matching chunker,
extractor, or normalizer request. LLM-backed modules pass that material onward
as named Scriptorium prompt inputs.
extractor, merger, or normalizer request. LLM-backed modules pass that material
onward as named Scriptorium prompt inputs.
The CLI carries raw input bytes into `pipeline.RunInput`. Input adapters parse
those bytes into the source document, while LLM-backed modules that need the
@@ -65,9 +66,9 @@ origin metadata. The raw input payload is not written to manifests or default
diagnostics.
The CLI also carries an optional run `session_id`. The runner makes it available
to chunk, extract, and normalize requests; LLM-backed modules forward it through
their structured completion requests so Scriptorium can include it in prompt
execution metadata.
to chunk, extract, merge, and normalize requests; LLM-backed modules forward it
through their structured completion requests so Scriptorium can include it in
prompt execution metadata.
## Registries And Module Specs
@@ -83,10 +84,9 @@ Every production module registers a `ModuleSpec` with:
- `Provides`: capabilities added after that module runs;
- `Requires`: capabilities that must already be available.
Chunk, extract, and normalize specs may also declare reference slots. Slot
Chunk, extract, merge, and normalize specs may also declare reference slots. Slot
declarations are available from registry metadata without constructing module
instances. Input, merge, validate, and output specs must not declare reference
slots.
instances. Input, validate, and output specs must not declare reference slots.
Capability checks prevent incompatible pipeline composition before a run starts.
@@ -102,8 +102,8 @@ Capability checks prevent incompatible pipeline composition before a run starts.
`pipeline.RunOutput` carries:
- run manifest;
- approved artifacts;
- rejected artifacts;
- normalized raw outputs;
- rejected raw outputs;
- warnings;
- logical output files returned by the output encoder.
@@ -116,10 +116,13 @@ The runner:
1. validates run input and registries;
2. builds the input adapter and parses the raw input into a source document;
3. validates the source document;
4. builds the chunker and produces source chunks;
5. validates source chunks against framework invariants;
4. builds the chunker and produces source chunks, retrying when configured;
5. validates source chunks against framework invariants and any registered raw
chunk validators;
6. runs each selected artifact lane in sorted resolved order;
7. builds the output encoder and validates logical output file names.
8. passes accepted normalized raw outputs, rejected output records, warnings,
and the manifest to the output encoder.
## Chunk Results
@@ -129,22 +132,29 @@ configured LLM profile, module options, and run metadata. Deterministic and
LLM-backed chunkers use the same contract; provider construction stays outside
chunk modules.
After `Chunk` returns, the runner appends chunker warnings before returning any
chunker error. When chunking succeeds, the runner validates generic chunk
invariants before running extractors:
When chunking succeeds, the runner validates generic chunk invariants before
running extractors:
- chunk IDs must be non-empty and unique in the chunk result;
- each chunk `SourceID` must match the source document ID;
- each chunk `Index` must match its zero-based returned order;
- each chunk start and end unit ID must exist in the source document, with the
start unit at or before the end unit;
- each chunk must include non-empty extraction content and media type;
- each chunk must contain at least one source unit;
- a chunk must not repeat a source unit;
- every chunk source unit must exist in the source document;
- source units inside each chunk must appear in source-document order.
After validation, the runner rebuilds each chunk from source-document units by
ID, preserving the chunk boundary order and cloning chunk metadata. Extractors
and downstream stages therefore see canonical source units, while
`SourceChunk.Metadata` remains the supported place for chunker-owned context.
integer ID, preserving chunk boundaries, content bytes, media type, and cloned
chunk metadata. Extractors and downstream stages therefore see canonical source
units, while `SourceChunk.Metadata` remains the supported place for
chunker-owned context.
If chunk validation rejects a chunk after configured retries, the runner records
a rejected raw output and skips downstream lane execution. Framework-level
chunking or validation errors that remain after configured retries fail the run.
The framework does not require complete source-unit coverage and does not reject
overlap between different chunks. Stricter policies, such as full coverage or
@@ -155,33 +165,37 @@ Within an artifact lane, the runner:
1. builds the extractor, merger, and normalizer;
2. records module manifest metadata when modules provide it;
3. extracts candidates from each chunk;
4. normalizes candidate envelope fields such as index, extractor key, artifact
type, and schema version;
5. merges candidates;
6. normalizes merged candidates;
7. validates candidate envelope consistency;
8. runs validators;
9. converts approved candidates to artifacts.
3. extracts one raw `ExtractOutput` from each accepted chunk, retrying when
configured;
4. fills runner-owned provenance on each extract output, including lane ID,
extractor key, source ID, chunk ID, and chunk index;
5. validates raw extract outputs and omits rejected outputs from merge input;
6. merges ordered accepted extract outputs into one raw `MergeOutput`, retrying
when configured;
7. validates raw merge output and skips normalization for rejected merge output;
8. normalizes the accepted merge output into one raw `NormalizeOutput`,
retrying when configured;
9. validates raw normalize output and appends accepted normalized raw output to
`RunOutput.NormalizeOutputs`.
## Validators
If a lane declares validators in config, the runner builds those validators from
the validator registry. Otherwise it uses validators returned by the extractor.
The current runner handoff is raw-output based. Extractors, mergers, and
normalizers do not advertise validator chains through their module interfaces.
Runner-side raw validation chains receive the raw module output plus
stage, lane, module, source, and chunk provenance. Empty raw validation chains
approve output by default.
Each validator must return exactly one decision for each eligible candidate. The
runner enforces decision cardinality with `internal/framework/validate`.
Rejected candidates are removed before the next validator runs. Approved
candidates continue through the chain.
The production CLI currently registers no standalone validator modules. The
current D&D spell extractor supplies deterministic shape and source-reference
validators.
Validator rejection is a non-fatal run outcome: the rejected output is recorded
in `RunOutput.Rejected` and does not pass to the next stage. Validator execution
errors are framework-level errors and retry according to the relevant binding.
## Warnings And Failures
Warnings from chunking, extraction, merging, normalization, validation, and
output encoding are accumulated in `RunOutput.Warnings`.
Warnings from the successful chunking, extraction, merging, and normalization
attempts whose outputs are used are accumulated in `RunOutput.Warnings`, along
with output encoder warnings. Warnings from discarded retry attempts are not
promoted to final warnings.
Errors wrap the operation and module key or lane context. If execution fails
after a manifest exists, the returned manifest is marked `failed` and receives a
@@ -189,14 +203,19 @@ completion timestamp.
On successful execution, the manifest validation status is:
- `approved` when no candidates were rejected;
- `rejected` when at least one candidate was rejected.
- `approved` when no raw outputs were rejected;
- `rejected` when at least one raw output was rejected.
## Manifest Population
The manifest records run ID, pipeline ID, pipeline digest, module keys, top-level
module metadata, artifact lanes, LLM profile metadata, source digest,
reference provenance, validation status, and timing.
reference provenance, normalized raw output summaries, rejected output
summaries, validation status, and timing. Raw output summaries include lane ID,
normalizer module key, media type, source ID, and response-schema provenance
when present. Rejected output summaries include stage, lane, module, chunk,
validator or reason, message, attempt count, and optional diagnostic artifact
path. The manifest does not include raw output payload bytes.
Singleton pipeline modules may add non-secret metadata by implementing
`contracts.ManifestMetadataProvider`. The runner records that metadata under
@@ -206,3 +225,12 @@ Lane-owned modules may add non-secret metadata through
`artifact_lanes[].metadata`. The runner records extractor, merger, and
normalizer metadata there. The D&D spell extractor uses lane metadata for
prompt and response-schema provenance.
## JSON Output
The production JSON output encoder writes `manifest.json`, `index.json`,
`warnings.json`, `rejected.json`, and one pretty-printed JSON file per accepted
normalized lane output under `lanes/`. It accepts only normalized outputs with
valid `application/json` payloads. Unsupported media types, invalid JSON, unsafe
logical paths, and duplicate sanitized lane file names fail the run before
durable output files are written.

21
docs/operations.md
View File

@@ -16,8 +16,8 @@ go run ./cmd/notarius run dnd-session \
--diagnostics-dir /tmp/notarius
```
The command prints a success line with the pipeline ID, approved and rejected
artifact counts, and the output path.
The command prints a success line with the pipeline ID, normalized output count,
rejected output count, and the output path.
## Output Directory
@@ -32,15 +32,14 @@ different root.
The `json` output module writes these files:
- `index.json`: file index with paths to the manifest, artifact files,
rejected artifacts, and warnings.
- `index.json`: file index with paths to the manifest, lane output files,
rejected outputs, and warnings.
- `manifest.json`: run manifest with resolved pipeline provenance, top-level
module metadata, module keys, reference provenance, validation status, and
timing.
- `artifacts/<artifact-type>.json`: approved artifacts grouped by artifact
type. For the current D&D spell extractor, this includes
`artifacts/dnd.spell_cast.json` when spell-cast artifacts are approved.
- `rejected.json`: rejected candidates and validator decisions.
- `lanes/<lane-id>.json`: normalized raw JSON output payloads, one file per
lane. For the current D&D spell extractor, this includes `lanes/spells.json`.
- `rejected.json`: rejected raw output records.
- `warnings.json`: warnings reported by pipeline modules or the output encoder.
Output writes are atomic per file. Logical output file names must be clean,
@@ -108,7 +107,7 @@ retained for inspection and may include `run-manifest.json`, `warnings.json`,
A successful run with warnings exits with code `0`, prints a warning count to
stderr, and writes warnings to durable output and diagnostics when retained.
The run manifest `validation_status` indicates whether final artifacts were
The run manifest `validation_status` indicates whether raw outputs were
approved or rejected after validation.
Reference-related warnings include empty bound reference files and D&D spell
@@ -134,7 +133,9 @@ There is no command to resume a failed run. Re-run `notarius run` after fixing
the cause.
Provider retries and timeouts are handled by Scriptorium according to the
selected execution profile. There is no separate CLI retry command.
selected execution profile. Pipeline module retries are controlled by module
binding `retries` values in config for chunk, extract, merge, and normalize.
There is no separate CLI retry command.
Notarius writes local files only. Remote storage and archive management are not
part of the implemented CLI.

429
docs/roadmap/implementation.md
View File

@@ -1,419 +1,18 @@
# Raw Pipeline Data Model Implementation Plan
# Raw Pipeline Implementation
This plan implements the target state in
[`pipeline.md`](pipeline.md). It is intentionally staged for multiple coding
passes. Do not skip stages: later work assumes the contracts and tests from
earlier stages are already in place.
The raw pipeline migration described here has been implemented.
The desired end state is a fixed-shape pipeline:
Current behavior is documented in:
```text
input -> chunk -> extract -> merge -> normalize -> output
```
- [Pipeline Internals](../internal/pipeline.md)
- [Modules](../internal/modules.md)
- [LLM Runtime](../internal/llm.md)
- [Configuration](../config.md)
- [CLI Reference](../cli.md)
- [Operations](../operations.md)
- [Troubleshooting](../troubleshooting.md)
- [JSON Output](../integrations/json-output.md)
- [D&D Spell Raw Output](../integrations/dnd-spell-artifacts.md)
`chunk` produces typed chunk envelopes. `extract`, `merge`, and `normalize`
produce raw byte payload envelopes with media type and provenance. Rejected
outputs do not pass to the next stage.
## Stage 1: Integer Source Units And Chunk Payloads
Update the core source and chunk contracts before changing downstream stages.
Implementation tasks:
- Change `internal/core/source.SourceUnit.ID` from `string` to `int`.
- Change `internal/core/source.SourceRef.StartUnitID` and `EndUnitID` from
`string` to `int`.
- Treat source-unit IDs as positive, stable, input-adapter-owned integers.
`0` and negative IDs should be invalid.
- Update source validation helpers, including unit lookup and source-ref
ordering checks, to use integer IDs.
- Update `contracts.SourceChunk` to include:
- `StartUnitID int`;
- `EndUnitID int`;
- `Content []byte`;
- `MediaType string`.
- Keep `SourceChunk.Units []source.SourceUnit` for framework scheduling,
diagnostics, and modules that need unit metadata.
- Update chunk canonicalization in `internal/framework/pipeline` to enforce
minimal scheduling invariants:
- at least one chunk;
- non-empty chunk ID;
- source ID matches the source document;
- indexes are deterministic and contiguous from zero;
- start/end unit IDs exist and are ordered;
- `Units` are canonical copies from the source document;
- content is non-empty;
- media type is non-empty.
- Update the Seriatim input adapter to parse segment IDs as integers.
Accept JSON numbers and numeric strings only when they are positive integers.
Reject non-numeric IDs such as `seg-001`.
- Update Seriatim examples, fixtures, tests, and integration docs to use
integer segment IDs.
- Update D&D source-reference helper code under
`internal/modules/sharedassets/dnd` so source refs remain integer-valued all
the way into `source.SourceRef`.
- Remove fallback behavior that converts integer LLM unit refs back into string
source-unit IDs.
- Update the generic chunker and D&D scenes chunker to populate
`StartUnitID`, `EndUnitID`, `Content`, and `MediaType`.
- For current transcript chunkers, use `application/json` content containing a
canonical JSON encoding of the chunk's source units. Keep the original
`Units` field populated as framework-owned typed metadata.
Tests to update or add:
- `go test ./internal/core/source`
- `go test ./internal/modules/input/seriatim`
- `go test ./internal/modules/chunk/generic`
- `go test ./internal/modules/chunk/dnd/scenes`
- `go test ./internal/framework/contracts`
- `go test ./internal/framework/pipeline`
Completion criteria:
- All source refs and source-unit IDs in core/framework types are integers.
- Chunkers return extraction-ready content bytes plus media type.
- Framework chunk validation still allows partial coverage and overlap unless a
validator rejects them.
## Stage 2: Raw Module Output Contracts
Replace artifact-candidate stage contracts with raw payload envelopes.
Implementation tasks:
- Add contract types in `internal/framework/contracts` for raw stage payloads.
The exact names may differ, but the contracts must preserve:
- raw bytes;
- media type;
- lane ID;
- stage module key;
- source ID;
- chunk ID and chunk index where applicable;
- response schema ID, name, and version when applicable;
- metadata;
- warnings.
- Recommended shape:
- `ExtractOutput`
- `MergeOutput`
- `NormalizeOutput`
- a small shared payload/provenance helper if it reduces duplication.
- Update `Extractor` so `Extract` returns one raw `ExtractOutput` for one input
chunk instead of `[]artifacts.ArtifactCandidate`.
- Remove `Extractor.ArtifactType`, `Extractor.SchemaVersion`, and
`Extractor.Validators` from the generic extractor contract. Schema and prompt
provenance should come from the returned output and manifest metadata, not
artifact-specific methods.
- Update `Merger` so `Merge` receives ordered accepted `[]ExtractOutput` and
returns one raw `MergeOutput`.
- Update `Normalizer` so `Normalize` receives one accepted `MergeOutput` and
returns one raw `NormalizeOutput`.
- Update `OutputRequest` so output encoders receive the validated normalized
lane outputs rather than approved `Artifact` values.
- Add a raw rejected-output record type for manifests/output files. It should
identify stage, lane, module, chunk when relevant, validator or error reason,
message, attempt count, and diagnostic artifact path if present. It must not
include large raw content by default.
- Leave old artifact/candidate types in `internal/core/artifacts` only if
temporary compatibility code still needs them during the migration. They
should not remain in the final stage contracts.
Tests to update or add:
- Contract composition tests proving fake chunk, extract, merge, normalize, and
output modules compose using raw envelopes.
- Defensive-copy tests for raw content and metadata maps.
- Registry integration tests proving the new interfaces build and run.
Completion criteria:
- Framework contracts no longer require extractors, mergers, normalizers, or
output encoders to use `ArtifactCandidate` or `Artifact`.
- Stage outputs carry raw bytes and media type.
## Stage 3: Config, References, Profiles, And Retries
Make `merge` a first-class LLM/profile/reference stage and add retry policy.
Implementation tasks:
- Update `pipeline.referenceSlotStage` so `merge` modules may declare reference
slots.
- Add `MergeReferences` to `ResolvedArtifactLane`.
- Resolve and materialize merge references using the same declared-slot model
used by `chunk`, `extract`, and `normalize`.
- Update reference provenance so merge-stage references appear in manifests and
resolved reference diagnostics.
- Update config parsing, config validation, redaction, cloning, and examples so
`artifacts.<lane>.merge.references` is valid.
- Update CLI `--reference` selector parsing and ambiguity checks to support:
- `merge.<slot>=<path>` where unambiguous;
- `<lane>.merge.<slot>=<path>`;
- lane-qualified shorthand only when it resolves unambiguously.
- Update `--without-reference` behavior to support merge-stage slots.
- Update `ModuleStage` LLM-capable helpers so `chunk`, `extract`, `merge`, and
`normalize` are all considered LLM-capable.
- Update `--llm-profile` override handling so it applies to `chunk`, every lane
`extract`, every lane `merge`, and every lane `normalize`.
- Update explicit Scriptorium profile validation so it inspects only those four
LLM-capable stages.
- Add `retries` to `pipeline.ModuleBinding` as a non-negative integer count of
extra attempts after the first attempt. Default is `0`.
- Apply `retries` only to `chunk`, `extract`, `merge`, and `normalize` runtime
execution. Keep input and output retry behavior out of scope.
- Validate `retries >= 0` in config validation.
- Ensure redacted config preserves retry counts.
Tests to update or add:
- Config accepts merge references and rejects unknown/undeclared merge slots.
- CLI reference selectors work for merge and remain strict for ambiguous slots.
- `--llm-profile` overrides merge bindings as well as chunk/extract/normalize.
- Explicit profile validation includes merge and ignores non-LLM stages.
- Negative retries are rejected.
Completion criteria:
- Merge has the same runtime plumbing class as chunk/extract/normalize.
- Retry policy is representable in pipeline config and resolved bindings.
## Stage 4: Runner Orchestration And Retry Semantics
Rewrite pipeline execution around raw envelopes.
Implementation tasks:
- Introduce a small retry helper in `internal/framework/pipeline` that reruns
the same module with the same input when:
- the module returns a framework-level error;
- the module returns output that is rejected by that stage's validator chain.
- Preserve context cancellation: never retry after `ctx.Err()` is non-nil.
- Record attempt count and compact attempt diagnostics for manifests/output.
- Do not log or manifest raw payload bytes by default.
- Run chunk once per source input, with retries from the chunk binding.
- After successful chunk execution, run chunk validation when validator mapping
support exists. A rejected chunk result after final retry should stop
downstream execution and produce a rejected run record.
- Run extract once per accepted chunk, with per-chunk retries from the extract
binding.
- Omit rejected extract outputs from merge input.
- If a lane has no accepted extract outputs after retries, skip merge and
normalize for that lane, record rejected output details, and continue to the
next lane.
- Run merge once per lane with accepted extract outputs, with retries from the
merge binding.
- If merge output is rejected after final retry, omit that lane from normalize
and output.
- Run normalize once per accepted merge output, with retries from the normalize
binding.
- If normalize output is rejected after final retry, omit that lane from output.
- Treat validator rejection as non-fatal run outcome. Treat unrecovered
framework-level execution errors as run failures.
- Preserve deterministic ordering:
- chunks by chunk index;
- extract outputs by chunk index;
- lane outputs by resolved lane order.
- Continue to collect warnings from every successful module attempt whose output
is used. For failed attempts, record compact attempt diagnostics rather than
promoting warnings as final-stage warnings unless the implementation already
has a clear warning policy.
- Set manifest validation status to:
- `approved` when all produced normalized lane outputs pass;
- `rejected` when one or more module outputs are rejected;
- failed-run status through existing failure handling for unrecovered runtime
errors.
Validation seam scope:
- Add the runner-side raw validation seam needed by this data-model refactor.
- Represent the validation boundary as raw module output plus stage provenance,
consistent with [`validation.md`](validation.md).
- Support empty validator chains as approval.
- Add fake validator/test-only coverage for approval, rejection, and
retry-on-rejection behavior.
- Do not implement the full production validator package migration, default
validator mappings, or concrete generic validators in this plan. Those remain
owned by the validation roadmap.
Tests to update or add:
- Runner passes chunk content and media type to extractors.
- Runner passes ordered accepted extract outputs to merge.
- Runner passes accepted merge output to normalize.
- Rejected extract output is omitted from merge input.
- A lane with no accepted extracts is omitted and recorded.
- Rejected merge output prevents normalize for that lane.
- Rejected normalize output prevents output for that lane.
- Retry reruns the same module input after framework error.
- Retry reruns the same module input after validator rejection.
- Retry stops after configured attempts and records attempt count.
- Context cancellation stops retries.
Completion criteria:
- The runner no longer depends on candidate materialization between extract,
merge, normalize, and output.
- Rejected outputs never pass to the next stage.
## Stage 5: Production Module Migration
Update concrete modules to the new contracts.
Implementation tasks:
- Update `internal/modules/extract/dnd/spells`:
- call Scriptorium with the existing prompt/schema;
- capture the returned raw JSON response bytes;
- return one `ExtractOutput` with `application/json` media type and response
schema provenance;
- do not convert spell casts into `ArtifactCandidate`;
- do not run shape/source-ref/source-relatedness validation inside the module.
- Keep D&D spell prompt and schema asset metadata in manifest metadata.
- Update `internal/modules/merge/appendorder` to merge raw JSON extract outputs
deterministically:
- preserve input order by chunk index;
- if all extract outputs are JSON objects with one common top-level array
field, concatenate those arrays into one JSON object under that field;
- otherwise produce a JSON array containing each extract output's decoded JSON
value in order;
- reject non-JSON media types with a clear error unless a future option
deliberately supports them.
- Update `internal/modules/normalize/noop` to pass accepted merge output through
unchanged as normalized output.
- Update chunk modules from Stage 1 as needed after interface changes.
- Remove or quarantine old candidate validators from D&D module packages if
they no longer compile. Their replacement belongs under the validation
roadmap in `internal/validators`.
- Update fake/test modules throughout the repository to the raw contracts.
Tests to update or add:
- D&D spells extractor returns raw JSON with schema provenance and no candidate
materialization.
- Append-order merger concatenates common top-level JSON arrays.
- Append-order merger falls back to ordered JSON value arrays when shapes differ.
- Append-order merger rejects invalid JSON and non-JSON media types clearly.
- No-op normalizer returns defensive copies of merge output bytes and metadata.
- Production catalog still registers all modules successfully.
Completion criteria:
- Production modules compile against raw contracts.
- D&D spell extraction no longer rejects spell candidates inside the extractor.
## Stage 6: Output, Manifest, Diagnostics, And Files
Update durable output around normalized lane outputs.
Implementation tasks:
- Update `RunOutput` to carry normalized lane outputs and rejected module-output
records instead of approved/rejected artifacts.
- Update `artifacts.RunManifest` or introduce a more accurately named manifest
package if the old artifact naming becomes misleading. Prefer the smallest
rename that keeps manifest output clear and avoids a broad unrelated refactor.
- Preserve existing manifest fields that remain meaningful:
- pipeline ID and digest;
- module keys;
- lane IDs;
- source digest;
- LLM profile provenance;
- reference provenance;
- module metadata;
- validation status.
- Add manifest/reporting fields needed for raw outputs:
- normalized lane output media type;
- output schema provenance where present;
- rejected stage/lane/chunk/module information;
- retry attempt counts.
- Update `internal/modules/output/json`:
- write `manifest.json`;
- write `index.json`;
- write `warnings.json`;
- write `rejected.json`;
- write one file per normalized lane output.
- For the JSON output encoder, accept `application/json` normalized outputs and
write them as raw JSON files. Recommended file path:
`lanes/<safe-lane-id>.json`.
- Reject non-JSON normalized output media types in the JSON encoder with a clear
error. Future text/Markdown/binary encoders can support other media types.
- Keep output file name validation strict.
- Ensure diagnostics redaction still prevents raw source, prompt, reference,
schema, and payload bytes from appearing in ordinary errors or manifests.
Tests to update or add:
- JSON output writes lane output files and index entries deterministically.
- JSON output rejects invalid JSON bytes and unsupported media types.
- Rejected output records are written without raw payload bytes.
- Manifest includes raw-output provenance and retry attempt counts.
- Diagnostics tests still prove sensitive/large content is not emitted.
Completion criteria:
- Durable output no longer assumes artifact candidates.
- JSON output remains deterministic and safe.
## Stage 7: Documentation, Examples, And Full Validation
After behavior is implemented, update canonical current-behavior docs.
Implementation tasks:
- Update `docs/internal/pipeline.md` for the raw data model.
- Update `docs/internal/modules.md` for new module contracts.
- Update `docs/internal/llm.md` for merge-stage LLM/profile plumbing.
- Update `docs/config.md` for:
- integer source-unit expectations where config examples include source refs;
- merge references;
- module `retries`;
- LLM profile override scope.
- Update `docs/cli.md` for merge reference selectors and profile behavior.
- Update `docs/integrations/json-output.md` for lane output files.
- Update `docs/integrations/dnd-spell-artifacts.md`; if the artifact-specific
contract is no longer durable, rename or replace it with a D&D spell raw-output
integration doc.
- Update `docs/troubleshooting.md` for common raw-output, media-type, retry, and
validation failures.
- Update examples so Seriatim segment IDs are positive integers and expected
output shape matches lane-normalized output.
- Replace `docs/roadmap/implementation.md` with a completed-note document only
after this implementation is finished and reviewed.
- Leave `docs/roadmap/pipeline.md` as target-state roadmap context until the
feature is implemented, then move implemented behavior into canonical docs and
remove stale roadmap language.
Validation commands:
```sh
go test ./internal/core/source
go test ./internal/framework/contracts
go test ./internal/framework/pipeline
go test ./internal/core/config
go test ./internal/cli
go test ./internal/modules/input/seriatim
go test ./internal/modules/chunk/generic
go test ./internal/modules/chunk/dnd/scenes
go test ./internal/modules/extract/dnd/spells
go test ./internal/modules/merge/appendorder
go test ./internal/modules/normalize/noop
go test ./internal/modules/output/json
go test ./...
go vet ./...
go build ./cmd/notarius
```
Documentation inspection:
```sh
rg -n "ArtifactCandidate|approved artifacts|spell artifact|start_unit_id\": \"|end_unit_id\": \"" docs examples internal
rg -n "chunk`, `extract`, and `normalize|deterministic-only stage|json.RawMessage" docs/roadmap docs/internal docs/config.md docs/cli.md
```
Expected result:
- Remaining `ArtifactCandidate` mentions are only in deliberately retained
compatibility code or historical roadmap context.
- Current-behavior docs describe raw lane outputs, integer source-unit IDs,
merge LLM/reference support, and retry behavior.
Follow-up validator work remains tracked separately in
[Validation Roadmap](validation.md).

405
docs/roadmap/pipeline.md
View File

@@ -1,403 +1,12 @@
# Raw Pipeline Data Model
This roadmap defines the target data model for the pipeline stages after input.
It is paired with the validation system roadmap in
[`validation.md`](validation.md): validators should evaluate module outputs, but
the pipeline first needs a raw-output handoff model that does not require every
module response shape to be represented by bespoke Go structs.
The raw pipeline data model has been implemented.
## Goals
Current behavior is documented in:
- Make typed chunk envelopes and raw module-output envelopes first-class
pipeline handoffs.
- Keep framework-owned provenance around raw payloads so runs remain
deterministic, ordered, and auditable.
- Avoid requiring each extractor, merger, or normalizer response schema to have
matching Go structs.
- Let extract modules produce one raw output per input chunk.
- Let merge modules receive accepted extract outputs and merge them into one raw
payload.
- Let normalize modules optionally post-process accepted merge output.
- Let output modules write normalized output bytes directly, along with
manifests, warnings, rejected outputs, and diagnostics as appropriate.
- Preserve the fixed workflow shape:
- [Pipeline Internals](../internal/pipeline.md)
- [Modules](../internal/modules.md)
- [JSON Output](../integrations/json-output.md)
```text
input -> chunk -> extract -> merge -> normalize -> output
```
## Non-Goals
- Do not turn the pipeline into an arbitrary DAG or general workflow language.
- Do not make validators mutate, materialize, or rewrite module outputs.
- Do not require a generic source-reference convention for every raw payload in
this pass.
- Do not eliminate typed framework data where the framework genuinely needs it,
such as input source documents and chunk boundaries.
## Cross-Stage Runtime Plumbing
LLM/profile/reference plumbing should be available to every stage that may need
LLM-backed or reference-aware module behavior: `chunk`, `extract`, `merge`, and
`normalize`.
For all four stages, the framework should provide the same categories of runtime
support where the concrete module contract needs them:
- configured LLM profile and profile override handling;
- Scriptorium client access through framework-owned LLM contracts;
- session ID propagation;
- declared reference slots and resolved reference bindings;
- module options and metadata;
- prompt, schema, profile, and reference provenance for manifests and
diagnostics.
`merge` must not be treated as a deterministic-only stage. A merge module may be
simple and deterministic, but it may also be LLM-backed, reference-aware, and
validator-gated in the same way as `chunk`, `extract`, and `normalize`.
## Chunk Stage Target Model
The chunk stage receives the canonical `SourceDocument` from the input stage,
plus any original source input material needed for prompt construction. The
`SourceDocument` remains the source of truth for source identity, source-unit
ordering, source-unit IDs, and source provenance. Original raw input material
may be supplied to LLM-backed chunkers, but it should not replace the
`SourceDocument` as the framework handoff.
Input adapters are responsible for assigning stable integer source-unit IDs. How
those IDs are assigned depends on the input format. A numbered JSON transcript
may map source units directly to transcript segment numbers; a PDF input adapter
may assign page or extracted-text unit numbers; an adapter for unordered source
material may assign deterministic IDs as part of input parsing. Downstream
framework code should treat those IDs as opaque integers owned by the input
adapter.
The chunk module returns one or more ordered `SourceChunk` envelopes. A single
chunk representing the whole source is valid and should be supported.
Each chunk should include framework-readable provenance and ordering metadata,
plus content suitable for extraction. The content may be JSON, plain text,
Markdown, PDF page text, or another module/input-specific representation, as
long as the framework can still associate the chunk with the source and preserve
deterministic order.
Conceptually:
```go
type SourceChunk struct {
ID string
SourceID string
Index int
StartUnitID int
EndUnitID int
Content []byte
MediaType string
Units []source.SourceUnit
Metadata map[string]any
}
```
The exact implementation shape can differ, but it should preserve:
- chunk identity;
- source identity;
- deterministic chunk order;
- source locator or range, normally integer start/end unit IDs;
- chunk content and media type;
- optional source-unit projection when useful;
- metadata needed by extractors, validators, manifests, diagnostics, and output
encoders.
The framework owns the minimal invariants required to schedule extract work:
- the chunker returns at least one chunk;
- chunk IDs and indexes are stable and non-empty;
- chunk source IDs match the source document;
- chunk ordering is deterministic;
- chunk provenance is sufficient to trace the chunk back to the input source.
Domain-specific chunk acceptability belongs in validators. For example, full
coverage, no gaps, no overlap, scene metadata quality, expected media type, and
D&D scene-boundary policy should be explicit validator concerns rather than
hidden framework rules, except where a minimal invariant is required for
extraction to run safely.
## Extract Stage Target Model
The extract stage receives one input chunk from the chunk stage and produces one
raw extract output for that chunk.
The extractor owns:
- prompt selection;
- input material assembly;
- Scriptorium request construction;
- response-schema selection;
- LLM profile and session usage;
- returned raw payload metadata.
The framework owns:
- chunk iteration;
- deterministic ordering;
- association between each extract output and its input chunk;
- execution errors when an extractor does not return output;
- handoff of returned output to validation and later stages.
An extract output should be an envelope, not just raw bytes. Conceptually:
```go
type ExtractOutput struct {
LaneID string
ExtractorKey string
ChunkID string
ChunkIndex int
SourceID string
RawContent []byte
MediaType string
SchemaID string
SchemaName string
SchemaVersion string
Metadata map[string]any
Warnings []contracts.Warning
}
```
The exact implementation shape can differ, but it should preserve:
- raw returned content;
- chunk provenance;
- chunk order;
- source identity;
- module identity;
- response schema provenance;
- metadata needed by validators, mergers, manifests, diagnostics, and output
encoders.
Extractor modules should not be required to convert raw LLM output into
`ArtifactCandidate` values. Domain-specific Go projection may still exist for
specific modules when it is useful, but it should not be the generic pipeline
contract.
Rejected extract outputs are not passed to merge. This keeps downstream
contracts simple and makes rejection behavior explicit. Support for passing
rejected outputs forward as marked data is deferred future work.
## Merge Stage Target Model
The merge stage receives the accepted extract outputs for a lane. Each extract
output corresponds to one input chunk and carries enough metadata to recover the
original chunk order.
The merger owns:
- merge/reconciliation strategy;
- deterministic or LLM-backed merge logic;
- prompt and schema usage when LLM-backed;
- the shape of merged raw output.
The framework owns:
- passing the ordered accepted extract-output set to the merger;
- lane identity;
- source context;
- references;
- runtime LLM plumbing;
- validation handoff for merged output;
- manifest provenance.
Simple merge modules may concatenate raw extract outputs in chunk order. Other
merge modules may deterministically merge JSON documents, reconcile duplicates,
or use an LLM to produce a more coherent merged result.
Conceptually:
```go
type MergeRequest struct {
LaneID string
Source *source.SourceDocument
ExtractOutputs []ExtractOutput
SourceInput contracts.LLMInputMaterial
SessionID string
References contracts.ReferenceSet
LLMClient contracts.StructuredLLMClient
LLMProfile string
Options map[string]any
Metadata map[string]any
}
type MergeOutput struct {
LaneID string
MergerKey string
RawContent []byte
MediaType string
SchemaID string
SchemaName string
SchemaVersion string
Metadata map[string]any
Warnings []contracts.Warning
}
```
The exact implementation shape can differ, but the key contract is that
merge consumes ordered accepted extract outputs and produces one merged raw
output for the lane.
If no accepted extract outputs remain for a lane, the framework should not pass
rejected outputs to the merger. The lane should produce no merge output and
should be reported as rejected or omitted according to run reporting policy.
## Normalize Stage Target Model
The normalize stage receives accepted merge output for a lane and optionally
post-processes it into the final normalized raw payload for that lane.
Normalization may be a no-op, deterministic cleanup, schema conversion, or an
LLM-backed post-processing pass.
The normalizer owns:
- post-merge processing strategy;
- deterministic or LLM-backed normalization logic;
- prompt and schema usage when LLM-backed;
- the shape of normalized raw output.
The framework owns:
- passing accepted merge output to the normalizer;
- lane identity;
- source context;
- references;
- runtime LLM plumbing;
- validation handoff for normalized output;
- manifest provenance.
Conceptually:
```go
type NormalizeRequest struct {
LaneID string
Source *source.SourceDocument
MergeOutput MergeOutput
SourceInput contracts.LLMInputMaterial
SessionID string
References contracts.ReferenceSet
LLMClient contracts.StructuredLLMClient
LLMProfile string
Options map[string]any
Metadata map[string]any
}
type NormalizeOutput struct {
LaneID string
NormalizerKey string
RawContent []byte
MediaType string
SchemaID string
SchemaName string
SchemaVersion string
Metadata map[string]any
Warnings []contracts.Warning
}
```
The exact implementation shape can differ, but the key contract is that
normalize consumes one accepted merged output and produces one normalized raw
output for the lane.
## Output Stage Target Model
The output stage receives validated normalized output bytes and writes them to
its configured destination.
Output modules should not require normalized output to be converted into
`Artifact` values. Output modules should use the normalized output media type to
decide how to serialize or wrap the payload. A JSON output module can write raw
normalized JSON directly, while a text or Markdown output module can write text
payloads directly. Output modules may also write run manifests, warnings,
rejected outputs, and indexes.
Output modules may still choose to provide convenience layouts, grouping, or file
naming conventions, but those should be output concerns rather than constraints
on extractor or normalizer response schemas.
## Validation Relationship
Validation chains should attach to returned module outputs, not to hidden
module-internal conversions.
For chunk:
```text
chunk(input) -> SourceChunk set -> chunk validators -> extract input
```
For extract:
```text
extract(chunk) -> raw ExtractOutput -> extract validators -> merge input
```
For merge:
```text
merge(extract outputs) -> raw MergeOutput -> merge validators -> normalize input
```
For normalize:
```text
normalize(merge output) -> raw NormalizeOutput -> normalize validators -> output
```
Validators must be read-only. They inspect raw output and metadata, return
accept/reject decisions and warnings, and do not rewrite output.
An empty validator chain approves returned output for that validation point.
Framework/runtime errors remain separate from validation rejections: if a module
or Scriptorium call fails before output is returned, the pipeline reports an
execution error rather than asking validators to evaluate nonexistent output.
Rejected outputs do not pass to the next stage. An empty validator chain still
approves returned output for that validation point.
## Retry Policy
A retry means re-running the same module with the same input after that module
fails to produce valid output. Failure to produce valid output includes both:
- framework-level execution errors, such as module errors, Scriptorium errors,
provider errors, or missing returned output;
- validator rejection of returned output.
Retries should be configurable at the pipeline or lane level. Chunk retries are
per source input. Extract retries are per chunk. Merge and normalize retries,
when configured, are per lane. Retry attempts should preserve deterministic
reporting: the final accepted or rejected output should record attempt count and
enough diagnostics/provenance to understand prior failures without leaking
secrets or large payloads by default.
## Relationship To Validation Roadmap
This roadmap defines the data model that validation should evaluate. The
validation system roadmap in [`validation.md`](validation.md) defines validator
registration, mapping, execution classes, and concrete validator behavior.
The shared boundary between the roadmaps is a returned module output: validators
inspect typed chunk output or raw module-output envelopes and decide whether
that output may continue through the pipeline.
Validator mapping and concrete validator behavior remain tracked separately in
[Validation Roadmap](validation.md).

58
docs/troubleshooting.md
View File

@@ -118,16 +118,18 @@ Symptoms include:
Fix:
- Confirm the selected chunker, extractor, or normalizer declares the slot. The
- Confirm the selected chunker, extractor, merger, or normalizer declares the slot. The
implemented `dnd/scenes` chunker and `dnd/spells` extractor declare optional
`roster` and `glossary` slots.
- Use a specific selector when more than one selected target declares the same
slot. Examples include `chunk.context=./context.txt`,
`spells.extract.context=./extract-context.txt`, and
slot: `chunk.context=./context.txt`,
`spells.extract.context=./extract-context.txt`,
`spells.merge.context=./merge-context.txt`, or
`spells.normalize.context=./normalize-context.txt`.
- `lane.slot=path` is valid only when exactly one selected extractor or
normalizer in that lane declares the slot. If both do, use
`lane.extract.slot=path` or `lane.normalize.slot=path`.
- `lane.slot=path` is valid only when exactly one selected extractor, merger,
or normalizer in that lane declares the slot. If more than one does, use
`lane.extract.slot=path`, `lane.merge.slot=path`, or
`lane.normalize.slot=path`.
- Use `--without-reference selector` to remove optional config bindings; do not
pass an empty `--reference selector=`.
- Check whether a path came from config or CLI. Config paths are relative to
@@ -198,7 +200,8 @@ Fix:
- Or use an existing Scriptorium profile ID with `--llm-profile`.
Use `--llm-profile <id>` when one run should force every LLM-backed binding to
the same Scriptorium profile.
the same Scriptorium profile. The override applies to effective chunk, extract,
merge, and normalize bindings.
## Missing API Key Environment Variable
@@ -283,17 +286,58 @@ Symptoms include:
- `create output directory`
- `write output file`
- `output file name must`
- `unsupported media type`
- `invalid JSON`
Fix:
- Ensure `--output-dir` points to a directory path or a path that can be
created.
- Check filesystem permissions and available disk space.
- The production JSON output encoder writes lane payloads under `lanes/` and
accepts only valid `application/json` normalized outputs. If an error names an
unsupported media type or invalid JSON, inspect the lane's merge and normalize
module output.
- If diagnostics were retained, inspect `run-report.json`, `run-manifest.json`,
and `error.log`.
The CLI rejects unsafe logical output paths before writing files.
## Raw Output Rejection
Symptoms include a successful run with:
- `validation_status` set to `rejected`;
- non-empty `rejected.json`;
- `rejected_outputs` entries in `manifest.json`.
Explanation and fixes:
- Validator rejection is a non-fatal run outcome. Rejected module outputs do not
pass to the next pipeline stage.
- Check `rejected.json` for the stage, lane, module, chunk, validator, reason,
message, and attempt count.
- Increase a module binding's `retries` only when re-running the same module
input can reasonably produce an acceptable output.
- If rejection is deterministic, fix the source input, module configuration, or
validator configuration rather than adding retries.
## Retry Exhaustion
Symptoms include:
- errors containing `failed after ... attempt(s)`;
- rejected output records with `attempt_count` greater than `1`.
Fix:
- `retries` is the number of extra attempts after the first attempt for chunk,
extract, merge, and normalize bindings.
- Framework-level errors after the last attempt fail the run.
- Validator rejections after the last attempt are recorded as rejected outputs.
- Check retained `error.log`, `run-manifest.json`, and `rejected.json` for the
operation, module key, lane, chunk, and attempt count.
## Diagnostics Directory Surprise
Symptom: the diagnostics directory is missing after a successful run.

4
examples/seriatim-minimal-transcript.json
View File

@@ -5,14 +5,14 @@
},
"segments": [
{
"id": "seg-001",
"id": 1,
"start": 0,
"end": 4,
"speaker": "Aria",
"text": "Aria raises her holy symbol and casts Cure Wounds."
},
{
"id": "seg-002",
"id": 2,
"start": 4,
"end": 8,
"speaker": "DM",

73
internal/cli/run.go
View File

@@ -99,7 +99,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
referenceFlags := stringListFlag{}
withoutReferenceFlags := stringListFlag{}
fs.Var(&sessionID, "session-id", "prompt session identifier")
fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, lane.slot=path, lane.extract.slot=path, or lane.normalize.slot=path")
fs.Var(&referenceFlags, "reference", "reference binding, as slot=path, chunk.slot=path, merge.slot=path, lane.slot=path, lane.extract.slot=path, lane.merge.slot=path, or lane.normalize.slot=path")
fs.Var(&withoutReferenceFlags, "without-reference", "unbind a reference, using the same selector forms as --reference")
if err := validateRunFlagValues(args); err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
@@ -275,7 +275,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
PipelineID: effective.PipelineID,
OutputPath: runOutputDir,
DiagnosticsPath: runDir.Path(),
ApprovedCount: len(output.Approved),
OutputCount: len(output.NormalizeOutputs),
RejectedCount: len(output.Rejected),
WarningCount: len(output.Warnings),
ValidationStatus: output.Manifest.ValidationStatus,
@@ -293,7 +293,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("apply diagnostics retention: %w", err))
}
fmt.Fprintf(stdout, "pipeline %q complete: approved=%d rejected=%d output=%s\n", effective.PipelineID, len(output.Approved), len(output.Rejected), runOutputDir)
fmt.Fprintf(stdout, "pipeline %q complete: outputs=%d rejected=%d output=%s\n", effective.PipelineID, len(output.NormalizeOutputs), len(output.Rejected), runOutputDir)
if len(output.Warnings) > 0 {
fmt.Fprintf(stderr, "notarius: run completed with %d warning(s)\n", len(output.Warnings))
}
@@ -305,7 +305,7 @@ type runReport struct {
PipelineID string `json:"pipeline_id"`
OutputPath string `json:"output_path"`
DiagnosticsPath string `json:"diagnostics_path,omitempty"`
ApprovedCount int `json:"approved_count"`
OutputCount int `json:"output_count"`
RejectedCount int `json:"rejected_count"`
WarningCount int `json:"warning_count"`
ValidationStatus string `json:"validation_status,omitempty"`
@@ -492,6 +492,7 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
add(resolved.Chunk)
for _, lane := range resolved.ArtifactLanes {
add(lane.Extract)
add(lane.Merge)
add(lane.Normalize)
}
ids := make([]string, 0, len(seen))
@@ -834,17 +835,20 @@ func parseReferenceSelector(raw string, flagName string) (cliReferenceSelector,
if first == string(pipeline.StageChunk) {
return cliReferenceSelector{Stage: pipeline.StageChunk, SlotName: slotName}, nil
}
if first == string(pipeline.StageMerge) {
return cliReferenceSelector{Stage: pipeline.StageMerge, SlotName: slotName}, nil
}
return cliReferenceSelector{LaneID: first, SlotName: slotName}, nil
case 3:
laneID := strings.TrimSpace(parts[0])
stage := pipeline.ModuleStage(strings.TrimSpace(parts[1]))
slotName := strings.TrimSpace(parts[2])
if stage != pipeline.StageExtract && stage != pipeline.StageNormalize {
return cliReferenceSelector{}, fmt.Errorf("%s lane-qualified selector must use lane.extract.slot or lane.normalize.slot", flagName)
if stage != pipeline.StageExtract && stage != pipeline.StageMerge && stage != pipeline.StageNormalize {
return cliReferenceSelector{}, fmt.Errorf("%s lane-qualified selector must use lane.extract.slot, lane.merge.slot, or lane.normalize.slot", flagName)
}
return cliReferenceSelector{LaneID: laneID, Stage: stage, SlotName: slotName}, nil
default:
return cliReferenceSelector{}, fmt.Errorf("%s must use slot, chunk.slot, lane.slot, lane.extract.slot, or lane.normalize.slot", flagName)
return cliReferenceSelector{}, fmt.Errorf("%s must use slot, chunk.slot, merge.slot, lane.slot, lane.extract.slot, lane.merge.slot, or lane.normalize.slot", flagName)
}
}
@@ -944,7 +948,7 @@ func selectedReferenceTargets(cfg config.Config, pipelineID string, only []strin
}
sort.Strings(selectedIDs)
targets := make([]selectedReferenceTarget, 0, 1+len(selectedIDs)*2)
targets := make([]selectedReferenceTarget, 0, 1+len(selectedIDs)*3)
chunk := pipeline.Binding(profile.Chunk.Module)
chunk.Module = strings.TrimSpace(profile.Chunk.Module)
if chunk.Module == "" {
@@ -977,6 +981,21 @@ func selectedReferenceTargets(cfg config.Config, pipelineID string, only []strin
slots: referenceSlotSet(extractSpec.ReferenceSlots),
})
mergeModule := strings.TrimSpace(lane.Merge.Module)
if mergeModule == "" {
mergeModule = pipeline.DefaultMergeModule
}
mergeSpec, err := cliReferenceMergerSpec(catalog, mergeModule)
if err != nil {
return nil, fmt.Errorf("pipeline %q lane %q merge module %q: %w", strings.TrimSpace(pipelineID), laneID, mergeModule, err)
}
targets = append(targets, selectedReferenceTarget{
laneID: laneID,
stage: pipeline.StageMerge,
module: mergeModule,
slots: referenceSlotSet(mergeSpec.ReferenceSlots),
})
normalizeModule := strings.TrimSpace(lane.Normalize.Module)
if normalizeModule == "" {
normalizeModule = pipeline.DefaultNormalizeModule
@@ -1027,6 +1046,17 @@ func cliReferenceExtractorSpec(catalog pipeline.ModuleCatalog, module string) (p
return spec, nil
}
func cliReferenceMergerSpec(catalog pipeline.ModuleCatalog, module string) (pipeline.ModuleSpec, error) {
if catalog.Mergers == nil {
return pipeline.ModuleSpec{}, fmt.Errorf("module %q is not registered", module)
}
spec, ok := catalog.Mergers.Spec(module)
if !ok {
return pipeline.ModuleSpec{}, fmt.Errorf("module %q is not registered", module)
}
return spec, nil
}
func cliReferenceNormalizerSpec(catalog pipeline.ModuleCatalog, module string) (pipeline.ModuleSpec, error) {
if catalog.Normalizers == nil {
return pipeline.ModuleSpec{}, fmt.Errorf("module %q is not registered", module)
@@ -1063,7 +1093,10 @@ func resolveCLIReferenceTarget(targets []selectedReferenceTarget, selector cliRe
}
return selectedReferenceTarget{}, fmt.Errorf("reference chunk target is not selected")
}
if selector.Stage == pipeline.StageExtract || selector.Stage == pipeline.StageNormalize {
if selector.Stage == pipeline.StageExtract || selector.Stage == pipeline.StageMerge || selector.Stage == pipeline.StageNormalize {
if selector.LaneID == "" && selector.Stage == pipeline.StageMerge {
return resolveCLIReferenceStageTarget(targets, selector.Stage, slotName)
}
for _, target := range targets {
if target.laneID == selector.LaneID && target.stage == selector.Stage {
if _, ok := target.slots[slotName]; !ok {
@@ -1080,6 +1113,26 @@ func resolveCLIReferenceTarget(targets []selectedReferenceTarget, selector cliRe
return resolveCLIReferenceFlatTarget(targets, slotName)
}
func resolveCLIReferenceStageTarget(targets []selectedReferenceTarget, stage pipeline.ModuleStage, slotName string) (selectedReferenceTarget, error) {
matches := make([]selectedReferenceTarget, 0, 2)
for _, target := range targets {
if target.stage != stage {
continue
}
if _, ok := target.slots[slotName]; ok {
matches = append(matches, target)
}
}
switch len(matches) {
case 0:
return selectedReferenceTarget{}, fmt.Errorf("reference slot %q is not declared by any selected %s target", slotName, stage)
case 1:
return matches[0], nil
default:
return selectedReferenceTarget{}, fmt.Errorf("reference slot %q is declared by multiple selected %s targets (%s); use a more specific selector such as %s", slotName, stage, targetList(matches), selectorSuggestions(matches, slotName))
}
}
func resolveCLIReferenceLaneTarget(targets []selectedReferenceTarget, laneID string, slotName string) (selectedReferenceTarget, error) {
laneSelected := false
matches := make([]selectedReferenceTarget, 0, 2)
@@ -1101,7 +1154,7 @@ func resolveCLIReferenceLaneTarget(targets []selectedReferenceTarget, laneID str
case 1:
return matches[0], nil
default:
return selectedReferenceTarget{}, fmt.Errorf("reference slot %q is declared by multiple selected targets in lane %q (%s); use %s.extract.%s or %s.normalize.%s", slotName, laneID, targetList(matches), laneID, slotName, laneID, slotName)
return selectedReferenceTarget{}, fmt.Errorf("reference slot %q is declared by multiple selected targets in lane %q (%s); use a more specific selector such as %s", slotName, laneID, targetList(matches), selectorSuggestions(matches, slotName))
}
}

229
internal/cli/run_test.go
View File

@@ -690,7 +690,7 @@ func TestRunPipelineSuccessUsesProductionRegistriesAndFakeLLM(t *testing.T) {
if client.calls != 1 {
t.Fatalf("LLM calls = %d, want 1", client.calls)
}
for _, want := range []string{"dnd-session", "approved=1", "rejected=0", outputDir} {
for _, want := range []string{"dnd-session", "outputs=1", "rejected=0", outputDir} {
if !strings.Contains(stdout.String(), want) {
t.Fatalf("stdout = %q, want substring %q", stdout.String(), want)
}
@@ -719,8 +719,8 @@ func TestRunPipelineOnlySelectsRequestedLane(t *testing.T) {
if client.calls != 1 {
t.Fatalf("LLM calls = %d, want only selected lane to run once", client.calls)
}
if !strings.Contains(stdout.String(), "approved=1") {
t.Fatalf("stdout = %q, want approved count", stdout.String())
if !strings.Contains(stdout.String(), "outputs=1") {
t.Fatalf("stdout = %q, want output count", stdout.String())
}
}
@@ -743,7 +743,7 @@ func TestRunPipelineLLMFactoryFailure(t *testing.T) {
}
}
func TestRunPipelineValidationRejectionCompletesSuccessfully(t *testing.T) {
func TestRunPipelineCarriesInvalidLLMSourceRefsAsRawOutput(t *testing.T) {
configPath := writeTestConfig(t, mvpConfigYAML("dnd-session", "dnd/spells"))
inputPath := writeSeriatimInput(t)
outputDir := t.TempDir()
@@ -759,8 +759,8 @@ func TestRunPipelineValidationRejectionCompletesSuccessfully(t *testing.T) {
if code != 0 {
t.Fatalf("RunWithOptions() code = %d, stderr=%q", code, stderr.String())
}
if !strings.Contains(stdout.String(), "approved=0") || !strings.Contains(stdout.String(), "rejected=1") {
t.Fatalf("stdout = %q, want rejection counts", stdout.String())
if !strings.Contains(stdout.String(), "outputs=1") || !strings.Contains(stdout.String(), "rejected=0") {
t.Fatalf("stdout = %q, want raw output count", stdout.String())
}
}
@@ -816,7 +816,7 @@ pipelines:
}
}
func TestRunConfigValidateIgnoresNonLLMStageScriptoriumProfiles(t *testing.T) {
func TestRunConfigValidateIncludesMergeAndIgnoresNonLLMStageScriptoriumProfiles(t *testing.T) {
profilePath := writeScriptoriumProfileFile(t, "known", "http://profile.test/v1", "test-model")
configPath := writeTestConfig(t, `version: 2
scriptorium:
@@ -843,11 +843,11 @@ pipelines:
Catalog: fakeCatalog(t),
})
if code != 0 {
t.Fatalf("RunWithOptions() code = %d, stderr=%q, want success", code, stderr.String())
if code != 1 {
t.Fatalf("RunWithOptions() code = %d, want failure for missing merge profile", code)
}
if strings.Contains(stderr.String(), "missing-") {
t.Fatalf("stderr = %q, want non-LLM stage profiles ignored", stderr.String())
if !strings.Contains(stderr.String(), "Scriptorium profile") || !strings.Contains(stderr.String(), "missing-merge") {
t.Fatalf("stderr = %q, want missing merge profile error", stderr.String())
}
}
@@ -867,7 +867,7 @@ func TestEffectiveLLMProfileIDsUsesLLMCapableStagesOnly(t *testing.T) {
}
got := effectiveLLMProfileIDs(resolved)
want := []string{"chunk-profile", "extract-profile", "normalize-profile"}
want := []string{"chunk-profile", "extract-profile", "merge-profile", "normalize-profile"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("effectiveLLMProfileIDs() = %#v, want %#v", got, want)
}
@@ -1166,6 +1166,81 @@ func TestRunPipelineReferenceFlagBindsExplicitNormalizeSlot(t *testing.T) {
}
}
func TestRunPipelineReferenceFlagBindsExplicitMergeSlot(t *testing.T) {
configPath := writeTestConfig(t, testConfigYAML("example", "events"))
inputPath := filepath.Join(t.TempDir(), "missing.json")
referencePath := writeFile(t, "merge.md", "Merge notes\n")
diagnosticsDir := t.TempDir()
var stdout bytes.Buffer
var stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "example",
"--config", configPath,
"--input", inputPath,
"--diagnostics-dir", diagnosticsDir,
"--reference", "events.merge.notes=" + referencePath,
}, &stdout, &stderr, Options{
Catalog: fakeCatalog(t, pipeline.ModuleSpec{
Key: "appendorder",
Stage: pipeline.StageMerge,
Requires: []string{"artifact"},
Provides: []string{"merged"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "notes"},
},
}),
})
if code != 1 || !strings.Contains(stderr.String(), "read input") {
t.Fatalf("RunWithOptions() code = %d stderr=%q, want read input failure after resolution", code, stderr.String())
}
resolved := readResolvedPipeline(t, diagnosticsDir)
refs := resolved.ArtifactLanes[0].MergeReferences.Bindings
want := []pipeline.ReferenceBinding{
{LaneID: "events", SlotName: "notes", Source: referencePath, BindingSource: contracts.ReferenceBindingSourceCLI},
}
if !reflect.DeepEqual(refs, want) {
t.Fatalf("merge references = %#v, want %#v", refs, want)
}
}
func TestRunPipelineReferenceFlagBindsUnambiguousMergeSlot(t *testing.T) {
configPath := writeTestConfig(t, testConfigYAML("example", "events"))
inputPath := filepath.Join(t.TempDir(), "missing.json")
referencePath := writeFile(t, "merge.md", "Merge notes\n")
diagnosticsDir := t.TempDir()
var stdout bytes.Buffer
var stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "example",
"--config", configPath,
"--input", inputPath,
"--diagnostics-dir", diagnosticsDir,
"--reference", "merge.notes=" + referencePath,
}, &stdout, &stderr, Options{
Catalog: fakeCatalog(t, pipeline.ModuleSpec{
Key: "appendorder",
Stage: pipeline.StageMerge,
Requires: []string{"artifact"},
Provides: []string{"merged"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "notes"},
},
}),
})
if code != 1 || !strings.Contains(stderr.String(), "read input") {
t.Fatalf("RunWithOptions() code = %d stderr=%q, want read input failure after resolution", code, stderr.String())
}
resolved := readResolvedPipeline(t, diagnosticsDir)
refs := resolved.ArtifactLanes[0].MergeReferences.Bindings
if len(refs) != 1 || refs[0].Source != referencePath || refs[0].LaneID != "events" {
t.Fatalf("merge references = %#v, want unambiguous merge binding", refs)
}
}
func TestRunPipelineReferenceFlagBindsFlatSlotAcrossOneTarget(t *testing.T) {
configPath := writeTestConfig(t, testConfigYAML("example", "events"))
inputPath := filepath.Join(t.TempDir(), "missing.json")
@@ -1366,8 +1441,8 @@ func TestRunPipelineReferenceFlagsRejectMalformedValues(t *testing.T) {
{name: "missing equals", args: []string{"--reference", "roster"}, want: "slot=path"},
{name: "empty path", args: []string{"--reference", "roster="}, want: "path must not be empty"},
{name: "empty slot", args: []string{"--reference", "=./roster.yml"}, want: "slot must not be empty"},
{name: "unsupported explicit stage", args: []string{"--reference", "a.b.c=./roster.yml"}, want: "lane.extract.slot or lane.normalize.slot"},
{name: "too many selector parts", args: []string{"--reference", "a.b.c.d=./roster.yml"}, want: "slot, chunk.slot, lane.slot, lane.extract.slot, or lane.normalize.slot"},
{name: "unsupported explicit stage", args: []string{"--reference", "a.b.c=./roster.yml"}, want: "lane.extract.slot, lane.merge.slot, or lane.normalize.slot"},
{name: "too many selector parts", args: []string{"--reference", "a.b.c.d=./roster.yml"}, want: "slot, chunk.slot, merge.slot, lane.slot, lane.extract.slot, lane.merge.slot, or lane.normalize.slot"},
{name: "unbind with equals", args: []string{"--without-reference", "roster=./roster.yml"}, want: "without =path"},
}
@@ -1394,6 +1469,7 @@ func TestRunPipelineReferenceFlagsRejectMalformedValues(t *testing.T) {
func TestRunPipelineWithoutReferenceRemovesConfigBindingsForEligibleTargets(t *testing.T) {
configPath := writeTestConfig(t, testConfigYAMLWithPipelineReferences("example", "events", map[string]string{
"context": "./config-context.md",
"merge_notes": "./config-merge.md",
"notes": "./config-notes.md",
"roster": "./config-roster.yml",
}))
@@ -1409,6 +1485,7 @@ func TestRunPipelineWithoutReferenceRemovesConfigBindingsForEligibleTargets(t *t
"--diagnostics-dir", diagnosticsDir,
"--without-reference", "chunk.context",
"--without-reference", "events.extract.roster",
"--without-reference", "events.merge.merge_notes",
"--without-reference", "events.normalize.notes",
}, &stdout, &stderr, Options{
Catalog: fakeCatalog(t,
@@ -1421,6 +1498,15 @@ func TestRunPipelineWithoutReferenceRemovesConfigBindingsForEligibleTargets(t *t
{Name: "context"},
},
},
pipeline.ModuleSpec{
Key: "appendorder",
Stage: pipeline.StageMerge,
Requires: []string{"artifact"},
Provides: []string{"merged"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "merge_notes"},
},
},
pipeline.ModuleSpec{
Key: "fake/extract",
Stage: pipeline.StageExtract,
@@ -1452,6 +1538,9 @@ func TestRunPipelineWithoutReferenceRemovesConfigBindingsForEligibleTargets(t *t
if refs := resolved.ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
t.Fatalf("extract references = %#v, want none", refs)
}
if refs := resolved.ArtifactLanes[0].MergeReferences.Bindings; len(refs) != 0 {
t.Fatalf("merge references = %#v, want none", refs)
}
if refs := resolved.ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 {
t.Fatalf("normalize references = %#v, want none", refs)
}
@@ -1643,8 +1732,8 @@ func TestRunPipelineWritesDurableOutputFiles(t *testing.T) {
runOutputDir := onlyChildDir(t, outputDir)
for _, name := range []string{
"index.json",
"lanes/spells.json",
"manifest.json",
"artifacts/dnd.spell_cast.json",
"rejected.json",
"warnings.json",
} {
@@ -1819,7 +1908,7 @@ func TestRunPipelineWritesDiagnosticsArtifactsOnSuccess(t *testing.T) {
}
}
report := string(readFile(t, filepath.Join(runDir, diagnostics.ArtifactRunReport)))
if !strings.Contains(report, `"approved_count": 1`) || !strings.Contains(report, `"validation_status": "approved"`) || !strings.Contains(report, outputDir) {
if !strings.Contains(report, `"output_count": 1`) || !strings.Contains(report, `"validation_status": "approved"`) || !strings.Contains(report, outputDir) {
t.Fatalf("unexpected run report: %s", report)
}
}
@@ -1989,30 +2078,27 @@ func TestExampleFixtureRunWritesExpectedJSON(t *testing.T) {
t.Fatalf("extractor metadata = %#v, want prompt/schema identifiers", extractorMetadata)
}
var artifactFile struct {
ArtifactType string `json:"artifact_type"`
Artifacts []artifacts.Artifact `json:"artifacts"`
}
readJSONFile(t, filepath.Join(runOutputDir, "artifacts", "dnd.spell_cast.json"), &artifactFile)
if artifactFile.ArtifactType != "dnd.spell_cast" || len(artifactFile.Artifacts) != 1 {
t.Fatalf("artifact file = %#v, want one spell artifact", artifactFile)
}
var payload struct {
var spellOutput struct {
SpellCasts []struct {
Caster string `json:"caster"`
Spell string `json:"spell"`
Effect string `json:"effect"`
SourceRefs []source.SourceRef `json:"source_refs"`
} `json:"spell_casts"`
}
if err := json.Unmarshal(artifactFile.Artifacts[0].Payload, &payload); err != nil {
t.Fatalf("unmarshal spell payload: %v", err)
readJSONFile(t, filepath.Join(runOutputDir, "lanes", "spells.json"), &spellOutput)
if len(spellOutput.SpellCasts) != 1 {
t.Fatalf("spell output = %#v, want one spell cast", spellOutput)
}
payload := spellOutput.SpellCasts[0]
if payload.Caster != "Aria" || payload.Spell != "Cure Wounds" || payload.Effect == "" {
t.Fatalf("payload = %#v, want deterministic spell output", payload)
}
if len(artifactFile.Artifacts[0].SourceRefs) != 1 {
t.Fatalf("source refs = %#v, want one source ref", artifactFile.Artifacts[0].SourceRefs)
if len(payload.SourceRefs) != 1 {
t.Fatalf("source refs = %#v, want one source ref", payload.SourceRefs)
}
ref := artifactFile.Artifacts[0].SourceRefs[0]
if ref.SourceID != "session-alpha" || ref.StartUnitID != "seg-001" || ref.EndUnitID != "seg-001" {
ref := payload.SourceRefs[0]
if ref.SourceID != "session-alpha" || ref.StartUnitID != 1 || ref.EndUnitID != 1 {
t.Fatalf("source ref = %#v, want fixture source ref", ref)
}
@@ -2168,18 +2254,18 @@ func TestExampleFixtureFailureCoverage(t *testing.T) {
wantStderr: "completion unavailable",
},
{
name: "malformed LLM response",
name: "malformed LLM response carried as raw output",
args: []string{"run", "dnd-session", "--config", configPath, "--input", inputPath},
factory: fakeLLMFactory(newMalformedRunLLMClient(), nil),
wantCode: 1,
wantStderr: "spell_casts",
wantCode: 0,
wantOutputStatus: "approved",
},
{
name: "invalid source reference rejection",
name: "invalid source reference raw output",
args: []string{"run", "dnd-session", "--config", configPath, "--input", inputPath},
factory: fakeLLMFactory(newFakeRunLLMClient(true), nil),
wantCode: 0,
wantOutputStatus: "rejected",
wantOutputStatus: "approved",
},
}
@@ -2213,10 +2299,6 @@ func TestExampleFixtureFailureCoverage(t *testing.T) {
if manifest.ValidationStatus != test.wantOutputStatus {
t.Fatalf("validation status = %q, want %q", manifest.ValidationStatus, test.wantOutputStatus)
}
rejected := string(readFile(t, filepath.Join(runOutputDir, "rejected.json")))
if !strings.Contains(rejected, "invalid_source_ref") {
t.Fatalf("rejected output = %s, want invalid source ref rejection", rejected)
}
}
})
}
@@ -2411,7 +2493,7 @@ func writeSeriatimInput(t *testing.T) string {
},
"segments": [
{
"id": "seg-001",
"id": 1,
"start": 0,
"end": 1,
"speaker": "Aria",
@@ -2454,8 +2536,8 @@ func (client *fakeRunLLMClient) CompleteStructured(ctx context.Context, req cont
payload := map[string]any{
"scenes": []map[string]any{
{
"start_unit_id": "seg-001",
"end_unit_id": "seg-002",
"start_unit_id": 1,
"end_unit_id": 2,
"short_title": "Opening spell",
"primary_mode": "Narrative",
"main_participants": []string{"Aria"},
@@ -2478,9 +2560,9 @@ func (client *fakeRunLLMClient) CompleteStructured(ctx context.Context, req cont
}
return contracts.StructuredCompletionResponse{Content: encoded}, nil
}
startUnitID := "seg-001"
startUnitID := 1
if client.invalidSourceRef {
startUnitID = "missing-segment"
startUnitID = 999
}
payload := client.payload
if payload == nil {
@@ -2491,11 +2573,11 @@ func (client *fakeRunLLMClient) CompleteStructured(ctx context.Context, req cont
"spell": "Cure Wounds",
"effect": "Heals a wounded ally.",
"narrative_description": "Aria casts Cure Wounds.",
"source_refs": []map[string]string{
"source_refs": []map[string]any{
{
"source_id": "session-alpha",
"start_unit_id": startUnitID,
"end_unit_id": "seg-001",
"end_unit_id": 1,
},
},
},
@@ -2653,7 +2735,7 @@ func (fakeRunInputAdapter) Parse(ctx context.Context, req contracts.ParseRequest
Format: "test",
Digest: "sha256:source",
Units: []source.SourceUnit{
{ID: "unit-1", Kind: "text", Text: string(req.Raw)},
{ID: 1, Kind: "text", Text: string(req.Raw)},
},
}, nil
}
@@ -2671,7 +2753,7 @@ func (fakeRunChunker) ReferenceSlots() []contracts.ReferenceSlot {
func (fakeRunChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
return contracts.ChunkResult{
Chunks: []contracts.SourceChunk{
{ID: "chunk-1", SourceID: req.Source.ID, Index: 0, Units: req.Source.Units},
{ID: "chunk-1", SourceID: req.Source.ID, Index: 0, StartUnitID: 1, EndUnitID: 1, Content: []byte(`{"units":[1]}`), MediaType: "application/json", Units: req.Source.Units},
},
}, nil
}
@@ -2682,30 +2764,17 @@ func (fakeRunExtractor) Key() string {
return "fake/extract"
}
func (fakeRunExtractor) ArtifactType() string {
return "fake.artifact"
}
func (fakeRunExtractor) SchemaVersion() string {
return "v1"
}
func (fakeRunExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return []contracts.ReferenceSlot{{Name: "roster"}}
}
func (fakeRunExtractor) Validators() []contracts.Validator {
return nil
}
func (fakeRunExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
return contracts.ExtractionResult{
Candidates: []artifacts.ArtifactCandidate{
{
Payload: []byte(`{"value":true}`),
SourceRefs: []source.SourceRef{
{SourceID: "source", StartUnitID: "unit-1", EndUnitID: "unit-1"},
},
Output: contracts.ExtractOutput{
Schema: contracts.ResponseSchema{ID: "fake.artifact", Name: "fake_artifact", Version: "v1"},
Payload: contracts.RawPayload{
Content: []byte(`{"value":true}`),
MediaType: "application/json",
},
},
}, nil
@@ -2718,11 +2787,20 @@ func (fakeRunMerger) Key() string {
}
func (fakeRunMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
var candidates []artifacts.ArtifactCandidate
for _, chunkArtifacts := range req.ChunkArtifacts {
candidates = append(candidates, chunkArtifacts.Candidates...)
output := contracts.MergeOutput{
LaneID: req.LaneID,
SourceID: req.Source.ID,
Schema: contracts.ResponseSchema{ID: "fake.artifact", Name: "fake_artifact", Version: "v1"},
Payload: contracts.RawPayload{
Content: []byte(`{"merged":true}`),
MediaType: "application/json",
},
}
return contracts.MergeResult{Candidates: candidates}, nil
if len(req.ExtractOutputs) > 0 {
output.Schema = req.ExtractOutputs[0].Schema
output.Payload = req.ExtractOutputs[0].Payload
}
return contracts.MergeResult{Output: output}, nil
}
type fakeRunNormalizer struct{}
@@ -2736,7 +2814,14 @@ func (fakeRunNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
}
func (fakeRunNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
return contracts.NormalizeResult{Candidates: append([]artifacts.ArtifactCandidate(nil), req.Candidates...)}, nil
return contracts.NormalizeResult{
Output: contracts.NormalizeOutput{
LaneID: req.LaneID,
SourceID: req.MergeOutput.SourceID,
Schema: req.MergeOutput.Schema,
Payload: req.MergeOutput.Payload,
},
}, nil
}
func onlyChildDir(t *testing.T, root string) string {

29
internal/core/artifacts/artifacts.go
View File

@@ -60,6 +60,33 @@ type ReferenceProvenance struct {
BindingSource string `json:"binding_source,omitempty"`
}
type OutputSchemaProvenance struct {
ID string `json:"id,omitempty"`
Name string `json:"name,omitempty"`
Version string `json:"version,omitempty"`
}
type NormalizedOutputManifest struct {
LaneID string `json:"lane_id"`
ModuleKey string `json:"module_key,omitempty"`
SourceID string `json:"source_id,omitempty"`
MediaType string `json:"media_type,omitempty"`
Schema OutputSchemaProvenance `json:"schema,omitempty"`
}
type RejectedOutputManifest struct {
Stage string `json:"stage"`
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"`
ChunkID string `json:"chunk_id,omitempty"`
ChunkIndex int `json:"chunk_index,omitempty"`
ValidatorName string `json:"validator_name,omitempty"`
ReasonCode string `json:"reason_code,omitempty"`
Message string `json:"message,omitempty"`
AttemptCount int `json:"attempt_count,omitempty"`
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
}
type RunManifest struct {
RunID string `json:"run_id,omitempty"`
PipelineID string `json:"pipeline_id,omitempty"`
@@ -74,6 +101,8 @@ type RunManifest struct {
ModuleMetadata map[string]map[string]any `json:"module_metadata,omitempty"`
ArtifactLanes []ArtifactLaneManifest `json:"artifact_lanes,omitempty"`
References []ReferenceProvenance `json:"references,omitempty"`
NormalizedOutputs []NormalizedOutputManifest `json:"normalized_outputs,omitempty"`
RejectedOutputs []RejectedOutputManifest `json:"rejected_outputs,omitempty"`
LLMProfiles []LLMProfileManifest `json:"llm_profiles,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
SchemaVersion string `json:"schema_version,omitempty"`

8
internal/core/artifacts/artifacts_test.go
View File

@@ -16,7 +16,7 @@ func TestArtifactFromCandidatePreservesCandidateFields(t *testing.T) {
SchemaVersion: "v1",
Payload: json.RawMessage(`{"name":"example"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u2"},
{SourceID: "source-1", StartUnitID: 1, EndUnitID: 2},
},
Metadata: map[string]any{
"confidence": 0.75,
@@ -45,13 +45,13 @@ func TestArtifactFromCandidatePreservesCandidateFields(t *testing.T) {
}
candidate.Payload[0] = '['
candidate.SourceRefs[0].StartUnitID = "changed"
candidate.SourceRefs[0].StartUnitID = 99
candidate.Metadata["confidence"] = 0.5
if string(artifact.Payload) != `{"name":"example"}` {
t.Fatalf("Payload changed after candidate mutation: %s", artifact.Payload)
}
if artifact.SourceRefs[0].StartUnitID != "u1" {
if artifact.SourceRefs[0].StartUnitID != 1 {
t.Fatalf("SourceRefs changed after candidate mutation: %#v", artifact.SourceRefs)
}
if artifact.Metadata["confidence"] != 0.75 {
@@ -67,7 +67,7 @@ func TestJSONMarshalUsesExpectedFieldNames(t *testing.T) {
SchemaVersion: "v1",
Payload: json.RawMessage(`{"value":true}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1},
},
Metadata: map[string]any{
"reviewed": true,

1
internal/core/config/effective_config.go
View File

@@ -68,6 +68,7 @@ func applyLLMProfileOverride(profile *pipeline.PipelineProfile, profileID string
profile.Chunk.LLMProfile = profileID
for laneID, lane := range profile.Artifacts {
lane.Extract.LLMProfile = profileID
lane.Merge.LLMProfile = profileID
lane.Normalize.LLMProfile = profileID
profile.Artifacts[laneID] = lane
}

6
internal/core/config/effective_config_test.go
View File

@@ -193,8 +193,8 @@ func TestResolveLLMProfileOverrideAppliesBeforeDigest(t *testing.T) {
t.Fatalf("output profile = %q, want original output-profile", effective.ResolvedPipeline.Output.LLMProfile)
}
eventLane := effective.ResolvedPipeline.ArtifactLanes[0]
if eventLane.Merge.LLMProfile != "merge-profile" {
t.Fatalf("merge profile = %q, want original merge-profile", eventLane.Merge.LLMProfile)
if eventLane.Merge.LLMProfile != "runtime" {
t.Fatalf("merge profile = %q, want runtime", eventLane.Merge.LLMProfile)
}
if len(eventLane.Validators) != 1 || eventLane.Validators[0].LLMProfile != "validator-profile" {
t.Fatalf("validator profiles = %#v, want original validator-profile", eventLane.Validators)
@@ -204,7 +204,7 @@ func TestResolveLLMProfileOverrideAppliesBeforeDigest(t *testing.T) {
func llmCapableBindings(resolved pipeline.ResolvedPipeline) []pipeline.ModuleBinding {
bindings := []pipeline.ModuleBinding{resolved.Chunk}
for _, lane := range resolved.ArtifactLanes {
bindings = append(bindings, lane.Extract, lane.Normalize)
bindings = append(bindings, lane.Extract, lane.Merge, lane.Normalize)
}
return bindings
}

8
internal/core/config/file_config.go
View File

@@ -53,6 +53,7 @@ type FileDiagnosticsConfig struct {
type fileModuleBinding struct {
Module string
LLMProfile string
Retries int
Options map[string]any
References map[string]string
}
@@ -83,6 +84,12 @@ func (b *fileModuleBinding) UnmarshalYAML(node *yaml.Node) error {
return err
}
b.LLMProfile = strings.TrimSpace(llmProfile)
case "retries":
var retries int
if err := valueNode.Decode(&retries); err != nil {
return err
}
b.Retries = retries
case "options":
var options map[string]any
if err := valueNode.Decode(&options); err != nil {
@@ -109,6 +116,7 @@ func (b fileModuleBinding) toPipelineBinding() pipeline.ModuleBinding {
return pipeline.ModuleBinding{
Module: strings.TrimSpace(b.Module),
LLMProfile: strings.TrimSpace(b.LLMProfile),
Retries: b.Retries,
Options: cloneOptions(b.Options),
References: normalizedStringMap(b.References),
}

19
internal/core/config/file_config_test.go
View File

@@ -127,6 +127,7 @@ pipelines:
input: fake/input
chunk:
module: generic
retries: 2
options:
size: 10
flags:
@@ -138,9 +139,12 @@ pipelines:
extract:
module: fake/extract
llm_profile: fast
retries: 3
options:
temperature: 0
merge: appendorder
merge:
module: appendorder
retries: 1
normalize:
module: noop
output: json
@@ -153,6 +157,9 @@ pipelines:
if profile.Chunk.Module != "generic" {
t.Fatalf("unexpected chunk binding: %+v", profile.Chunk)
}
if profile.Chunk.Retries != 2 {
t.Fatalf("chunk retries = %d, want 2", profile.Chunk.Retries)
}
if profile.Chunk.Options["size"] != 10 {
t.Fatalf("expected chunk options to preserve scalar, got %#v", profile.Chunk.Options)
}
@@ -168,6 +175,9 @@ pipelines:
if lane.Extract.Module != "fake/extract" || lane.Extract.LLMProfile != "fast" {
t.Fatalf("unexpected extract binding: %+v", lane.Extract)
}
if lane.Extract.Retries != 3 || lane.Merge.Retries != 1 {
t.Fatalf("unexpected retries: extract=%d merge=%d", lane.Extract.Retries, lane.Merge.Retries)
}
if lane.Extract.Options["temperature"] != 0 {
t.Fatalf("expected object options, got %#v", lane.Extract.Options)
}
@@ -224,6 +234,10 @@ pipelines:
references:
roster: ./legacy-roster.yml
lore: ./lore.md
merge:
module: appendorder
references:
" merge_notes ": " ./merge.md "
normalize:
module: noop
references:
@@ -246,6 +260,9 @@ pipelines:
if !reflect.DeepEqual(lane.Extract.References, wantExtract) {
t.Fatalf("extract references = %#v, want legacy merged with extract override %#v", lane.Extract.References, wantExtract)
}
if !reflect.DeepEqual(lane.Merge.References, map[string]string{"merge_notes": "./merge.md"}) {
t.Fatalf("merge references = %#v, want trimmed map", lane.Merge.References)
}
if !reflect.DeepEqual(lane.Normalize.References, map[string]string{"normalization_notes": "./normalization.md"}) {
t.Fatalf("normalize references = %#v, want trimmed map", lane.Normalize.References)
}

1
internal/core/config/redaction.go
View File

@@ -42,6 +42,7 @@ func cloneResolvedArtifactLane(in pipeline.ResolvedArtifactLane) pipeline.Resolv
out.Merge = cloneModuleBinding(in.Merge)
out.Normalize = cloneModuleBinding(in.Normalize)
out.ExtractReferences = pipeline.CloneReferenceTarget(in.ExtractReferences)
out.MergeReferences = pipeline.CloneReferenceTarget(in.MergeReferences)
out.NormalizeReferences = pipeline.CloneReferenceTarget(in.NormalizeReferences)
if len(in.Validators) > 0 {
out.Validators = make([]pipeline.ModuleBinding, len(in.Validators))

8
internal/core/config/validation.go
View File

@@ -78,7 +78,7 @@ func validatePipelineProfiles(profiles map[string]pipeline.PipelineProfile) erro
if err := validateBinding(id, laneID, "extract", lane.Extract, true); err != nil {
return err
}
if err := validateBinding(id, laneID, "merge", lane.Merge, false); err != nil {
if err := validateBinding(id, laneID, "merge", lane.Merge, true); err != nil {
return err
}
if err := validateBinding(id, laneID, "normalize", lane.Normalize, true); err != nil {
@@ -104,6 +104,12 @@ func validateBinding(
if err := validateBindingLLMProfile(pipelineID, laneID, slot, binding); err != nil {
return err
}
if binding.Retries < 0 {
if laneID != "" {
return fmt.Errorf("pipeline %q lane %q %s retries must be greater than or equal to zero", pipelineID, laneID, slot)
}
return fmt.Errorf("pipeline %q %s retries must be greater than or equal to zero", pipelineID, slot)
}
if len(binding.References) == 0 {
return nil
}

24
internal/core/config/validation_test.go
View File

@@ -54,6 +54,18 @@ func TestValidateRejectsInvalidNumericFields(t *testing.T) {
},
want: "total LLM concurrency",
},
{
name: "negative retries",
mutate: func(cfg Config) Config {
profile := cfg.Pipelines["example"]
lane := profile.Artifacts["events"]
lane.Merge.Retries = -1
profile.Artifacts["events"] = lane
cfg.Pipelines["example"] = profile
return cfg
},
want: "retries",
},
}
for _, tc := range tests {
@@ -238,18 +250,6 @@ func TestValidateRejectsReferencesOnUnsupportedBindings(t *testing.T) {
},
want: []string{"example", "input", "references", "not supported"},
},
{
name: "merge",
mutate: func(cfg Config) Config {
profile := cfg.Pipelines["example"]
lane := profile.Artifacts["events"]
lane.Merge.References = map[string]string{"roster": "./roster.yml"}
profile.Artifacts["events"] = lane
cfg.Pipelines["example"] = profile
return cfg
},
want: []string{"example", "events", "merge", "references", "not supported"},
},
{
name: "validator",
mutate: func(cfg Config) Config {

6
internal/core/source/source.go
View File

@@ -10,7 +10,7 @@ type SourceDocument struct {
}
type SourceUnit struct {
ID string `json:"id"`
ID int `json:"id"`
Kind string `json:"kind"`
Text string `json:"text"`
Metadata map[string]any `json:"metadata,omitempty"`
@@ -18,6 +18,6 @@ type SourceUnit struct {
type SourceRef struct {
SourceID string `json:"source_id"`
StartUnitID string `json:"start_unit_id"`
EndUnitID string `json:"end_unit_id"`
StartUnitID int `json:"start_unit_id"`
EndUnitID int `json:"end_unit_id"`
}

63
internal/core/source/source_test.go
View File

@@ -96,13 +96,8 @@ func TestValidateDocumentMissingUnitFields(t *testing.T) {
}{
{
name: "id",
mutate: func(doc *SourceDocument) { doc.Units[1].ID = "" },
wantErr: "source unit[1].id must not be empty",
},
{
name: "id surrounding whitespace",
mutate: func(doc *SourceDocument) { doc.Units[1].ID = " u2 " },
wantErr: "source unit[1].id \" u2 \" must not contain leading or trailing whitespace",
mutate: func(doc *SourceDocument) { doc.Units[1].ID = 0 },
wantErr: "source unit[1].id must be positive",
},
{
name: "kind",
@@ -135,14 +130,14 @@ func TestValidateDocumentMissingUnitFields(t *testing.T) {
func TestValidateDocumentDuplicateUnitIDs(t *testing.T) {
doc := validDocument()
doc.Units[1].ID = "u1"
doc.Units[1].ID = 1
err := ValidateDocument(doc)
if err == nil {
t.Fatal("ValidateDocument() error = nil, want error")
}
if err.Error() != "source unit id \"u1\" is duplicated" {
if err.Error() != "source unit id 1 is duplicated" {
t.Fatalf("ValidateDocument() error = %q", err.Error())
}
}
@@ -151,8 +146,8 @@ func TestValidateRefValid(t *testing.T) {
doc := validDocument()
ref := SourceRef{
SourceID: "source-1",
StartUnitID: "u1",
EndUnitID: "u2",
StartUnitID: 1,
EndUnitID: 2,
}
if err := ValidateRef(doc, ref); err != nil {
@@ -164,8 +159,8 @@ func TestValidateRefSourceIDMismatch(t *testing.T) {
doc := validDocument()
ref := SourceRef{
SourceID: "source-2",
StartUnitID: "u1",
EndUnitID: "u2",
StartUnitID: 1,
EndUnitID: 2,
}
err := ValidateRef(doc, ref)
@@ -186,43 +181,33 @@ func TestValidateRefMissingUnitIDs(t *testing.T) {
}{
{
name: "missing source id",
ref: SourceRef{StartUnitID: "u1", EndUnitID: "u2"},
ref: SourceRef{StartUnitID: 1, EndUnitID: 2},
wantErr: "source ref source_id must not be empty",
},
{
name: "source id surrounding whitespace",
ref: SourceRef{SourceID: " source-1 ", StartUnitID: "u1", EndUnitID: "u2"},
ref: SourceRef{SourceID: " source-1 ", StartUnitID: 1, EndUnitID: 2},
wantErr: "source ref source_id \" source-1 \" must not contain leading or trailing whitespace",
},
{
name: "missing start id",
ref: SourceRef{SourceID: "source-1", EndUnitID: "u2"},
wantErr: "source ref start_unit_id must not be empty",
},
{
name: "start id surrounding whitespace",
ref: SourceRef{SourceID: "source-1", StartUnitID: " u1 ", EndUnitID: "u2"},
wantErr: "source ref start_unit_id \" u1 \" must not contain leading or trailing whitespace",
ref: SourceRef{SourceID: "source-1", EndUnitID: 2},
wantErr: "source ref start_unit_id must be positive",
},
{
name: "missing end id",
ref: SourceRef{SourceID: "source-1", StartUnitID: "u1"},
wantErr: "source ref end_unit_id must not be empty",
},
{
name: "end id surrounding whitespace",
ref: SourceRef{SourceID: "source-1", StartUnitID: "u1", EndUnitID: " u2 "},
wantErr: "source ref end_unit_id \" u2 \" must not contain leading or trailing whitespace",
ref: SourceRef{SourceID: "source-1", StartUnitID: 1},
wantErr: "source ref end_unit_id must be positive",
},
{
name: "unknown start id",
ref: SourceRef{SourceID: "source-1", StartUnitID: "u9", EndUnitID: "u2"},
wantErr: "source ref start_unit_id \"u9\" was not found",
ref: SourceRef{SourceID: "source-1", StartUnitID: 9, EndUnitID: 2},
wantErr: "source ref start_unit_id 9 was not found",
},
{
name: "unknown end id",
ref: SourceRef{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u9"},
wantErr: "source ref end_unit_id \"u9\" was not found",
ref: SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 9},
wantErr: "source ref end_unit_id 9 was not found",
},
}
@@ -244,8 +229,8 @@ func TestValidateRefReversedUnitOrder(t *testing.T) {
doc := validDocument()
ref := SourceRef{
SourceID: "source-1",
StartUnitID: "u2",
EndUnitID: "u1",
StartUnitID: 2,
EndUnitID: 1,
}
err := ValidateRef(doc, ref)
@@ -261,7 +246,7 @@ func TestValidateRefReversedUnitOrder(t *testing.T) {
func TestUnitIndex(t *testing.T) {
doc := validDocument()
index, ok := UnitIndex(doc, "u2")
index, ok := UnitIndex(doc, 2)
if !ok {
t.Fatal("UnitIndex() ok = false, want true")
}
@@ -269,7 +254,7 @@ func TestUnitIndex(t *testing.T) {
t.Fatalf("UnitIndex() index = %d, want 1", index)
}
index, ok = UnitIndex(doc, "u9")
index, ok = UnitIndex(doc, 9)
if ok {
t.Fatal("UnitIndex() ok = true, want false")
}
@@ -286,12 +271,12 @@ func validDocument() *SourceDocument {
Digest: "sha256:abc123",
Units: []SourceUnit{
{
ID: "u1",
ID: 1,
Kind: "paragraph",
Text: "First unit.",
},
{
ID: "u2",
ID: 2,
Kind: "paragraph",
Text: "Second unit.",
},

33
internal/core/source/validation.go
View File

@@ -28,13 +28,10 @@ func ValidateDocument(doc *SourceDocument) error {
return fmt.Errorf("source document units must not be empty")
}
seenUnitIDs := make(map[string]struct{}, len(doc.Units))
seenUnitIDs := make(map[int]struct{}, len(doc.Units))
for i, unit := range doc.Units {
if isBlank(unit.ID) {
return fmt.Errorf("source unit[%d].id must not be empty", i)
}
if hasSurroundingWhitespace(unit.ID) {
return fmt.Errorf("source unit[%d].id %q must not contain leading or trailing whitespace", i, unit.ID)
if unit.ID <= 0 {
return fmt.Errorf("source unit[%d].id must be positive", i)
}
if isBlank(unit.Kind) {
return fmt.Errorf("source unit[%d].kind must not be empty", i)
@@ -43,7 +40,7 @@ func ValidateDocument(doc *SourceDocument) error {
return fmt.Errorf("source unit[%d].text must not be empty", i)
}
if _, ok := seenUnitIDs[unit.ID]; ok {
return fmt.Errorf("source unit id %q is duplicated", unit.ID)
return fmt.Errorf("source unit id %d is duplicated", unit.ID)
}
seenUnitIDs[unit.ID] = struct{}{}
}
@@ -61,17 +58,11 @@ func ValidateRef(doc *SourceDocument, ref SourceRef) error {
if hasSurroundingWhitespace(ref.SourceID) {
return fmt.Errorf("source ref source_id %q must not contain leading or trailing whitespace", ref.SourceID)
}
if isBlank(ref.StartUnitID) {
return fmt.Errorf("source ref start_unit_id must not be empty")
if ref.StartUnitID <= 0 {
return fmt.Errorf("source ref start_unit_id must be positive")
}
if hasSurroundingWhitespace(ref.StartUnitID) {
return fmt.Errorf("source ref start_unit_id %q must not contain leading or trailing whitespace", ref.StartUnitID)
}
if isBlank(ref.EndUnitID) {
return fmt.Errorf("source ref end_unit_id must not be empty")
}
if hasSurroundingWhitespace(ref.EndUnitID) {
return fmt.Errorf("source ref end_unit_id %q must not contain leading or trailing whitespace", ref.EndUnitID)
if ref.EndUnitID <= 0 {
return fmt.Errorf("source ref end_unit_id must be positive")
}
if ref.SourceID != doc.ID {
return fmt.Errorf("source ref source_id %q does not match document id %q", ref.SourceID, doc.ID)
@@ -79,20 +70,20 @@ func ValidateRef(doc *SourceDocument, ref SourceRef) error {
startIndex, ok := UnitIndex(doc, ref.StartUnitID)
if !ok {
return fmt.Errorf("source ref start_unit_id %q was not found", ref.StartUnitID)
return fmt.Errorf("source ref start_unit_id %d was not found", ref.StartUnitID)
}
endIndex, ok := UnitIndex(doc, ref.EndUnitID)
if !ok {
return fmt.Errorf("source ref end_unit_id %q was not found", ref.EndUnitID)
return fmt.Errorf("source ref end_unit_id %d was not found", ref.EndUnitID)
}
if startIndex > endIndex {
return fmt.Errorf("source ref start_unit_id %q appears after end_unit_id %q", ref.StartUnitID, ref.EndUnitID)
return fmt.Errorf("source ref start_unit_id %d appears after end_unit_id %d", ref.StartUnitID, ref.EndUnitID)
}
return nil
}
func UnitIndex(doc *SourceDocument, unitID string) (int, bool) {
func UnitIndex(doc *SourceDocument, unitID int) (int, bool) {
if doc == nil {
return 0, false
}

145
internal/framework/contracts/composition_test.go
View File

@@ -27,7 +27,6 @@ func TestContractsComposeAcrossPackages(t *testing.T) {
extractor := compositionExtractor{}
merger := compositionMerger{}
normalizer := compositionNormalizer{}
validator := compositionValidator{}
encoder := compositionOutputEncoder{}
doc, err := adapter.Parse(ctx, contracts.ParseRequest{SourceID: "source-1"})
@@ -58,70 +57,37 @@ func TestContractsComposeAcrossPackages(t *testing.T) {
if err != nil {
t.Fatalf("Extract() error = %v, want nil", err)
}
if len(extraction.Candidates) != 1 {
t.Fatalf("len(Candidates) = %d, want 1", len(extraction.Candidates))
}
candidate := extraction.Candidates[0]
for _, ref := range candidate.SourceRefs {
if err := source.ValidateRef(doc, ref); err != nil {
t.Fatalf("ValidateRef() error = %v, want nil", err)
}
if extraction.Output.Payload.MediaType != "application/json" {
t.Fatalf("extract media type = %q, want application/json", extraction.Output.Payload.MediaType)
}
merge, err := merger.Merge(ctx, contracts.MergeRequest{
Source: doc,
LaneID: candidate.ArtifactType,
ChunkArtifacts: []contracts.ChunkArtifacts{
{
Chunk: chunking.Chunks[0],
Candidates: extraction.Candidates,
},
},
LaneID: "generic-lane",
ExtractOutputs: []contracts.ExtractOutput{extraction.Output},
})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
if len(merge.Candidates) != 1 {
t.Fatalf("len(merge.Candidates) = %d, want 1", len(merge.Candidates))
if string(merge.Output.Payload.Content) != `{"value":"example"}` {
t.Fatalf("merge output = %s, want extract payload", merge.Output.Payload.Content)
}
normalize, err := normalizer.Normalize(ctx, contracts.NormalizeRequest{
Source: doc,
LaneID: candidate.ArtifactType,
Candidates: merge.Candidates,
LaneID: "generic-lane",
MergeOutput: merge.Output,
})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(normalize.Candidates) != 1 {
t.Fatalf("len(normalize.Candidates) = %d, want 1", len(normalize.Candidates))
}
validation, err := validator.Validate(ctx, contracts.ValidationRequest{
Source: doc,
Candidates: normalize.Candidates,
})
if err != nil {
t.Fatalf("Validate() error = %v, want nil", err)
}
if len(validation.Decisions) != 1 {
t.Fatalf("len(Decisions) = %d, want 1", len(validation.Decisions))
}
decision := validation.Decisions[0]
if !decision.Approved {
t.Fatal("Approved = false, want true")
}
if decision.CandidateIndex != candidate.Index {
t.Fatalf("CandidateIndex = %d, want %d", decision.CandidateIndex, candidate.Index)
if string(normalize.Output.Payload.Content) != `{"value":"example"}` {
t.Fatalf("normalize output = %s, want merge payload", normalize.Output.Payload.Content)
}
output, err := encoder.Encode(ctx, contracts.OutputRequest{
Manifest: artifacts.RunManifest{RunID: "run-1"},
Approved: []artifacts.Artifact{
artifacts.ArtifactFromCandidate(normalize.Candidates[0]),
},
NormalizeOutputs: []contracts.NormalizeOutput{normalize.Output},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
@@ -150,8 +116,8 @@ func (adapter compositionAdapter) Parse(ctx context.Context, req contracts.Parse
Format: "text/plain",
Digest: "sha256:abc123",
Units: []source.SourceUnit{
{ID: "u1", Kind: "unit", Text: "First source unit."},
{ID: "u2", Kind: "unit", Text: "Second source unit."},
{ID: 1, Kind: "unit", Text: "First source unit."},
{ID: 2, Kind: "unit", Text: "Second source unit."},
},
}, nil
}
@@ -180,6 +146,10 @@ func (chunker compositionChunker) Chunk(ctx context.Context, req contracts.Chunk
ID: req.Source.ID + ":chunk:0",
SourceID: req.Source.ID,
Index: 0,
StartUnitID: req.Source.Units[0].ID,
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
Content: []byte(`{"units":[{"id":1,"kind":"unit","text":"First source unit."},{"id":2,"kind":"unit","text":"Second source unit."}]}`),
MediaType: "application/json",
Units: append([]source.SourceUnit(nil), req.Source.Units...),
Metadata: map[string]any{"strategy": "whole-document"},
},
@@ -199,49 +169,24 @@ func (extractor compositionExtractor) Key() string {
return "generic-extractor"
}
func (extractor compositionExtractor) ArtifactType() string {
return "generic-artifact"
}
func (extractor compositionExtractor) SchemaVersion() string {
return "v1"
}
func (extractor compositionExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (extractor compositionExtractor) Validators() []contracts.Validator {
return []contracts.Validator{compositionValidator{}}
}
func (extractor compositionExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
if req.Source == nil {
return contracts.ExtractionResult{}, errors.New("source document is required")
}
units := req.Source.Units
if req.Chunk != nil {
units = req.Chunk.Units
}
if req.AmbientContext["synopsis"] == "" {
return contracts.ExtractionResult{}, errors.New("ambient synopsis is required")
}
return contracts.ExtractionResult{
Candidates: []artifacts.ArtifactCandidate{
{
Index: 0,
ExtractorKey: extractor.Key(),
ArtifactType: extractor.ArtifactType(),
SchemaVersion: extractor.SchemaVersion(),
Payload: json.RawMessage(`{"value":"example"}`),
SourceRefs: []source.SourceRef{
{
SourceID: req.Source.ID,
StartUnitID: units[0].ID,
EndUnitID: units[len(units)-1].ID,
},
},
Output: contracts.ExtractOutput{
Schema: contracts.ResponseSchema{ID: "schema-id", Name: "schema-name", Version: "v1"},
Payload: contracts.RawPayload{
Content: []byte(`{"value":"example"}`),
MediaType: "application/json",
},
},
}, nil
@@ -254,12 +199,14 @@ func (merger compositionMerger) Key() string {
}
func (merger compositionMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
var candidates []artifacts.ArtifactCandidate
for _, chunkArtifacts := range req.ChunkArtifacts {
candidates = append(candidates, chunkArtifacts.Candidates...)
}
return contracts.MergeResult{Candidates: candidates}, nil
output := req.ExtractOutputs[0]
return contracts.MergeResult{Output: contracts.MergeOutput{
LaneID: req.LaneID,
MergerKey: merger.Key(),
SourceID: output.SourceID,
Schema: output.Schema,
Payload: cloneCompositionPayload(output.Payload),
}}, nil
}
type compositionNormalizer struct{}
@@ -273,7 +220,33 @@ func (normalizer compositionNormalizer) ReferenceSlots() []contracts.ReferenceSl
}
func (normalizer compositionNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
return contracts.NormalizeResult{Candidates: req.Candidates}, nil
return contracts.NormalizeResult{Output: contracts.NormalizeOutput{
LaneID: req.LaneID,
NormalizerKey: normalizer.Key(),
SourceID: req.MergeOutput.SourceID,
Schema: req.MergeOutput.Schema,
Payload: cloneCompositionPayload(req.MergeOutput.Payload),
}}, nil
}
func cloneCompositionPayload(payload contracts.RawPayload) contracts.RawPayload {
return contracts.RawPayload{
Content: append([]byte(nil), payload.Content...),
MediaType: payload.MediaType,
Metadata: cloneCompositionMetadata(payload.Metadata),
Warnings: append([]contracts.Warning(nil), payload.Warnings...),
}
}
func cloneCompositionMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
out := make(map[string]any, len(metadata))
for key, value := range metadata {
out[key] = value
}
return out
}
type compositionValidator struct{}
@@ -308,10 +281,10 @@ func (encoder compositionOutputEncoder) Key() string {
func (encoder compositionOutputEncoder) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
payload := struct {
RunID string `json:"run_id"`
ApprovedCount int `json:"approved_count"`
OutputCount int `json:"output_count"`
}{
RunID: req.Manifest.RunID,
ApprovedCount: len(req.Approved),
OutputCount: len(req.NormalizeOutputs),
}
encoded, err := json.Marshal(payload)
if err != nil {

104
internal/framework/contracts/contracts.go
View File

@@ -92,6 +92,10 @@ type SourceChunk struct {
ID string `json:"id"`
SourceID string `json:"source_id"`
Index int `json:"index"`
StartUnitID int `json:"start_unit_id"`
EndUnitID int `json:"end_unit_id"`
Content []byte `json:"-"`
MediaType string `json:"media_type"`
Units []source.SourceUnit `json:"units"`
Metadata map[string]any `json:"metadata,omitempty"`
}
@@ -182,38 +186,91 @@ type ExtractionRequest struct {
}
type ExtractionResult struct {
Candidates []artifacts.ArtifactCandidate `json:"candidates,omitempty"`
Output ExtractOutput `json:"output"`
Warnings []Warning `json:"warnings,omitempty"`
}
type Extractor interface {
Key() string
ArtifactType() string
SchemaVersion() string
ReferenceSlots() []ReferenceSlot
Validators() []Validator
Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error)
}
type ChunkArtifacts struct {
Chunk SourceChunk `json:"chunk"`
Candidates []artifacts.ArtifactCandidate `json:"candidates"`
type RawPayload struct {
Content []byte `json:"-"`
MediaType string `json:"media_type"`
Metadata map[string]any `json:"metadata,omitempty"`
Warnings []Warning `json:"warnings,omitempty"`
}
type RawValidationRequest struct {
Stage string `json:"stage"`
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key"`
Source *source.SourceDocument `json:"-"`
SourceID string `json:"source_id,omitempty"`
ChunkID string `json:"chunk_id,omitempty"`
ChunkIndex int `json:"chunk_index,omitempty"`
Schema ResponseSchema `json:"schema,omitempty"`
Payload RawPayload `json:"payload"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type RawValidationResult struct {
Approved bool `json:"approved"`
ReasonCode string `json:"reason_code,omitempty"`
Message string `json:"message,omitempty"`
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
Warnings []Warning `json:"warnings,omitempty"`
}
type RawValidator interface {
Name() string
ValidateRaw(ctx context.Context, req RawValidationRequest) (RawValidationResult, error)
}
type ResponseSchema struct {
ID string `json:"id,omitempty"`
Name string `json:"name,omitempty"`
Version string `json:"version,omitempty"`
}
type ExtractOutput struct {
LaneID string `json:"lane_id"`
ExtractorKey string `json:"extractor_key"`
SourceID string `json:"source_id"`
ChunkID string `json:"chunk_id"`
ChunkIndex int `json:"chunk_index"`
Schema ResponseSchema `json:"schema,omitempty"`
Payload RawPayload `json:"payload"`
}
type MergeRequest struct {
Source *source.SourceDocument `json:"-"`
LaneID string `json:"lane_id"`
ChunkArtifacts []ChunkArtifacts `json:"chunk_artifacts"`
ExtractOutputs []ExtractOutput `json:"extract_outputs"`
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
SessionID string `json:"session_id,omitempty"`
References ReferenceSet `json:"references,omitempty"`
LLMClient StructuredLLMClient `json:"-"`
LLMProfile string `json:"llm_profile,omitempty"`
Options map[string]any `json:"options,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type MergeResult struct {
Candidates []artifacts.ArtifactCandidate `json:"candidates"`
Output MergeOutput `json:"output"`
Warnings []Warning `json:"warnings,omitempty"`
}
type MergeOutput struct {
LaneID string `json:"lane_id"`
MergerKey string `json:"merger_key"`
SourceID string `json:"source_id,omitempty"`
Schema ResponseSchema `json:"schema,omitempty"`
Payload RawPayload `json:"payload"`
}
type Merger interface {
Key() string
Merge(ctx context.Context, req MergeRequest) (MergeResult, error)
@@ -222,7 +279,7 @@ type Merger interface {
type NormalizeRequest struct {
Source *source.SourceDocument `json:"-"`
LaneID string `json:"lane_id"`
Candidates []artifacts.ArtifactCandidate `json:"candidates"`
MergeOutput MergeOutput `json:"merge_output"`
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
SessionID string `json:"session_id,omitempty"`
References ReferenceSet `json:"references,omitempty"`
@@ -233,10 +290,18 @@ type NormalizeRequest struct {
}
type NormalizeResult struct {
Candidates []artifacts.ArtifactCandidate `json:"candidates"`
Output NormalizeOutput `json:"output"`
Warnings []Warning `json:"warnings,omitempty"`
}
type NormalizeOutput struct {
LaneID string `json:"lane_id"`
NormalizerKey string `json:"normalizer_key"`
SourceID string `json:"source_id,omitempty"`
Schema ResponseSchema `json:"schema,omitempty"`
Payload RawPayload `json:"payload"`
}
type Normalizer interface {
Key() string
ReferenceSlots() []ReferenceSlot
@@ -278,8 +343,8 @@ type Warning struct {
type OutputRequest struct {
Manifest artifacts.RunManifest `json:"manifest"`
Approved []artifacts.Artifact `json:"approved,omitempty"`
Rejected []artifacts.RejectedArtifact `json:"rejected,omitempty"`
NormalizeOutputs []NormalizeOutput `json:"normalize_outputs,omitempty"`
Rejected []RejectedOutput `json:"rejected,omitempty"`
Warnings []Warning `json:"warnings,omitempty"`
LLMProfile string `json:"llm_profile,omitempty"`
Options map[string]any `json:"options,omitempty"`
@@ -302,6 +367,19 @@ type OutputEncoder interface {
Encode(ctx context.Context, req OutputRequest) (OutputResult, error)
}
type RejectedOutput struct {
Stage string `json:"stage"`
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key,omitempty"`
ChunkID string `json:"chunk_id,omitempty"`
ChunkIndex int `json:"chunk_index,omitempty"`
ValidatorName string `json:"validator_name,omitempty"`
ReasonCode string `json:"reason_code,omitempty"`
Message string `json:"message"`
AttemptCount int `json:"attempt_count,omitempty"`
DiagnosticArtifactPath string `json:"diagnostic_artifact_path,omitempty"`
}
type ManifestMetadataProvider interface {
ManifestMetadata() map[string]any
}

217
internal/framework/contracts/contracts_test.go
View File

@@ -19,13 +19,9 @@ var _ Validator = fakeValidator{}
var _ StructuredLLMClient = fakeLLMClient{}
var _ OutputEncoder = fakeOutputEncoder{}
func TestFakeExtractorReturnsCandidateAndValidator(t *testing.T) {
validator := fakeValidator{name: "generic-validator"}
func TestFakeExtractorReturnsRawOutput(t *testing.T) {
extractor := fakeExtractor{
key: "generic-extractor",
artifactType: "generic-artifact",
schemaVersion: "v1",
validators: []Validator{validator},
}
doc := &source.SourceDocument{
ID: "source-1",
@@ -33,7 +29,7 @@ func TestFakeExtractorReturnsCandidateAndValidator(t *testing.T) {
Format: "text/plain",
Digest: "sha256:abc123",
Units: []source.SourceUnit{
{ID: "u1", Kind: "section", Text: "Source text."},
{ID: 1, Kind: "section", Text: "Source text."},
},
}
@@ -45,37 +41,14 @@ func TestFakeExtractorReturnsCandidateAndValidator(t *testing.T) {
if extractor.Key() != "generic-extractor" {
t.Fatalf("Key() = %q, want generic-extractor", extractor.Key())
}
if extractor.ArtifactType() != "generic-artifact" {
t.Fatalf("ArtifactType() = %q, want generic-artifact", extractor.ArtifactType())
if result.Output.ExtractorKey != "" {
t.Fatalf("ExtractorKey = %q, want runner-owned empty value", result.Output.ExtractorKey)
}
if extractor.SchemaVersion() != "v1" {
t.Fatalf("SchemaVersion() = %q, want v1", extractor.SchemaVersion())
if result.Output.Schema.Version != "v1" {
t.Fatalf("Schema.Version = %q, want v1", result.Output.Schema.Version)
}
if len(extractor.Validators()) != 1 {
t.Fatalf("len(Validators()) = %d, want 1", len(extractor.Validators()))
}
if extractor.Validators()[0].Name() != "generic-validator" {
t.Fatalf("Validators()[0].Name() = %q, want generic-validator", extractor.Validators()[0].Name())
}
if len(result.Candidates) != 1 {
t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
}
candidate := result.Candidates[0]
if candidate.Index != 0 {
t.Fatalf("ArtifactCandidate.Index = %d, want 0", candidate.Index)
}
if candidate.ExtractorKey != extractor.Key() {
t.Fatalf("ArtifactCandidate.ExtractorKey = %q, want %q", candidate.ExtractorKey, extractor.Key())
}
if candidate.ArtifactType != extractor.ArtifactType() {
t.Fatalf("ArtifactCandidate.ArtifactType = %q, want %q", candidate.ArtifactType, extractor.ArtifactType())
}
if candidate.SchemaVersion != extractor.SchemaVersion() {
t.Fatalf("ArtifactCandidate.SchemaVersion = %q, want %q", candidate.SchemaVersion, extractor.SchemaVersion())
}
if string(candidate.Payload) != `{"value":"example"}` {
t.Fatalf("ArtifactCandidate.Payload = %s, want example payload", candidate.Payload)
if result.Output.Payload.MediaType != "application/json" || string(result.Output.Payload.Content) != `{"value":"example"}` {
t.Fatalf("payload = %q %s, want JSON raw output", result.Output.Payload.MediaType, result.Output.Payload.Content)
}
}
@@ -86,7 +59,7 @@ func TestFakeChunkerReturnsSourceChunks(t *testing.T) {
Format: "text/plain",
Digest: "sha256:abc123",
Units: []source.SourceUnit{
{ID: "u1", Kind: "section", Text: "Source text."},
{ID: 1, Kind: "section", Text: "Source text."},
},
}
chunker := fakeChunker{key: "generic-chunker"}
@@ -113,6 +86,12 @@ func TestFakeChunkerReturnsSourceChunks(t *testing.T) {
if chunk.Index != 0 {
t.Fatalf("SourceChunk.Index = %d, want 0", chunk.Index)
}
if chunk.StartUnitID != 1 || chunk.EndUnitID != 1 {
t.Fatalf("SourceChunk boundaries = %d-%d, want 1-1", chunk.StartUnitID, chunk.EndUnitID)
}
if chunk.MediaType != "application/json" || string(chunk.Content) != `{"units":[{"id":1,"kind":"section","text":"Source text."}]}` {
t.Fatalf("SourceChunk payload = %q %s, want JSON units", chunk.MediaType, chunk.Content)
}
if len(chunk.Units) != 1 {
t.Fatalf("len(SourceChunk.Units) = %d, want 1", len(chunk.Units))
}
@@ -125,7 +104,7 @@ func TestFakeChunkerReceivesLLMClient(t *testing.T) {
Format: "text/plain",
Digest: "sha256:abc123",
Units: []source.SourceUnit{
{ID: "u1", Kind: "section", Text: "Source text."},
{ID: 1, Kind: "section", Text: "Source text."},
},
}
client := fakeLLMClient{}
@@ -140,25 +119,25 @@ func TestFakeChunkerReceivesLLMClient(t *testing.T) {
}
func TestFakeExtractorReceivesChunkAndAmbientContext(t *testing.T) {
extractor := fakeExtractor{
key: "generic-extractor",
artifactType: "generic-artifact",
schemaVersion: "v1",
}
extractor := fakeExtractor{key: "generic-extractor"}
doc := &source.SourceDocument{
ID: "source-1",
Kind: "document",
Format: "text/plain",
Digest: "sha256:abc123",
Units: []source.SourceUnit{
{ID: "u1", Kind: "section", Text: "First source text."},
{ID: "u2", Kind: "section", Text: "Second source text."},
{ID: 1, Kind: "section", Text: "First source text."},
{ID: 2, Kind: "section", Text: "Second source text."},
},
}
chunk := SourceChunk{
ID: "source-1:chunk:1",
SourceID: doc.ID,
Index: 1,
StartUnitID: 2,
EndUnitID: 2,
Content: []byte(`{"units":[{"id":2,"kind":"section","text":"Second source text."}]}`),
MediaType: "application/json",
Units: []source.SourceUnit{doc.Units[1]},
}
@@ -170,20 +149,11 @@ func TestFakeExtractorReceivesChunkAndAmbientContext(t *testing.T) {
if err != nil {
t.Fatalf("Extract() error = %v, want nil", err)
}
if len(result.Candidates) != 1 {
t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
if result.Output.ChunkID != "" || result.Output.ChunkIndex != 0 {
t.Fatalf("chunk provenance = %q/%d, want runner-owned zero values", result.Output.ChunkID, result.Output.ChunkIndex)
}
candidate := result.Candidates[0]
if string(candidate.Payload) != `{"value":"chunked"}` {
t.Fatalf("ArtifactCandidate.Payload = %s, want chunked payload", candidate.Payload)
}
if len(candidate.SourceRefs) != 1 {
t.Fatalf("len(SourceRefs) = %d, want 1", len(candidate.SourceRefs))
}
ref := candidate.SourceRefs[0]
if ref.StartUnitID != "u2" || ref.EndUnitID != "u2" {
t.Fatalf("SourceRef = %+v, want u2 range", ref)
if string(result.Output.Payload.Content) != `{"value":"chunked"}` {
t.Fatalf("Payload.Content = %s, want chunked payload", result.Output.Payload.Content)
}
}
@@ -353,19 +323,17 @@ func TestLLMInputSetCloneCopiesContent(t *testing.T) {
}
func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
candidate := artifacts.ArtifactCandidate{
Index: 0,
extractOutput := ExtractOutput{
LaneID: "generic-lane",
ExtractorKey: "generic-extractor",
ArtifactType: "generic-artifact",
SchemaVersion: "v1",
Payload: json.RawMessage(`{"value":"example"}`),
}
chunk := SourceChunk{
ID: "source-1:chunk:0",
SourceID: "source-1",
Index: 0,
Units: []source.SourceUnit{
{ID: "u1", Kind: "section", Text: "Source text."},
ChunkID: "source-1:chunk:0",
ChunkIndex: 0,
Schema: ResponseSchema{ID: "schema-id", Name: "schema-name", Version: "v1"},
Payload: RawPayload{
Content: []byte(`{"value":"example"}`),
MediaType: "application/json",
Metadata: map[string]any{"confidence": 0.75},
},
}
merger := fakeMerger{key: "generic-merger"}
@@ -373,13 +341,8 @@ func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
encoder := fakeOutputEncoder{key: "generic-output"}
merged, err := merger.Merge(context.Background(), MergeRequest{
LaneID: "generic-artifact",
ChunkArtifacts: []ChunkArtifacts{
{
Chunk: chunk,
Candidates: []artifacts.ArtifactCandidate{candidate},
},
},
LaneID: "generic-lane",
ExtractOutputs: []ExtractOutput{extractOutput},
})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
@@ -387,13 +350,13 @@ func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
if merger.Key() != "generic-merger" {
t.Fatalf("Merger.Key() = %q, want generic-merger", merger.Key())
}
if len(merged.Candidates) != 1 {
t.Fatalf("len(merged.Candidates) = %d, want 1", len(merged.Candidates))
if string(merged.Output.Payload.Content) != `{"value":"example"}` {
t.Fatalf("merged content = %s, want raw extract content", merged.Output.Payload.Content)
}
normalized, err := normalizer.Normalize(context.Background(), NormalizeRequest{
LaneID: "generic-artifact",
Candidates: merged.Candidates,
LaneID: "generic-lane",
MergeOutput: merged.Output,
})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
@@ -401,15 +364,13 @@ func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
if normalizer.Key() != "generic-normalizer" {
t.Fatalf("Normalizer.Key() = %q, want generic-normalizer", normalizer.Key())
}
if len(normalized.Candidates) != 1 {
t.Fatalf("len(normalized.Candidates) = %d, want 1", len(normalized.Candidates))
if string(normalized.Output.Payload.Content) != `{"value":"example"}` {
t.Fatalf("normalized content = %s, want raw merge content", normalized.Output.Payload.Content)
}
encoded, err := encoder.Encode(context.Background(), OutputRequest{
Manifest: artifacts.RunManifest{RunID: "run-1"},
Approved: []artifacts.Artifact{
artifacts.ArtifactFromCandidate(normalized.Candidates[0]),
},
NormalizeOutputs: []NormalizeOutput{normalized.Output},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
@@ -423,7 +384,7 @@ func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
if encoded.Files[0].ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", encoded.Files[0].ContentType)
}
if string(encoded.Files[0].Bytes) != `{"run_id":"run-1","approved_count":1}` {
if string(encoded.Files[0].Bytes) != `{"run_id":"run-1","output_count":1}` {
t.Fatalf("Bytes = %s, want encoded output", encoded.Files[0].Bytes)
}
}
@@ -490,6 +451,10 @@ func (chunker fakeChunker) Chunk(ctx context.Context, req ChunkRequest) (ChunkRe
ID: req.Source.ID + ":chunk:0",
SourceID: req.Source.ID,
Index: 0,
StartUnitID: req.Source.Units[0].ID,
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
Content: []byte(`{"units":[{"id":1,"kind":"section","text":"Source text."}]}`),
MediaType: "application/json",
Units: append([]source.SourceUnit(nil), req.Source.Units...),
},
},
@@ -516,56 +481,28 @@ func (chunker *recordingChunker) Chunk(ctx context.Context, req ChunkRequest) (C
type fakeExtractor struct {
key string
artifactType string
schemaVersion string
validators []Validator
}
func (extractor fakeExtractor) Key() string {
return extractor.key
}
func (extractor fakeExtractor) ArtifactType() string {
return extractor.artifactType
}
func (extractor fakeExtractor) SchemaVersion() string {
return extractor.schemaVersion
}
func (extractor fakeExtractor) ReferenceSlots() []ReferenceSlot {
return nil
}
func (extractor fakeExtractor) Validators() []Validator {
return extractor.validators
}
func (extractor fakeExtractor) Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error) {
units := req.Source.Units
if req.Chunk != nil {
units = req.Chunk.Units
}
payload := json.RawMessage(`{"value":"example"}`)
if req.AmbientContext["mode"] == "chunked" {
payload = json.RawMessage(`{"value":"chunked"}`)
}
return ExtractionResult{
Candidates: []artifacts.ArtifactCandidate{
{
Index: 0,
ExtractorKey: extractor.key,
ArtifactType: extractor.artifactType,
SchemaVersion: extractor.schemaVersion,
Payload: payload,
SourceRefs: []source.SourceRef{
{
SourceID: req.Source.ID,
StartUnitID: units[0].ID,
EndUnitID: units[len(units)-1].ID,
},
},
Output: ExtractOutput{
Schema: ResponseSchema{ID: "schema-id", Name: "schema-name", Version: "v1"},
Payload: RawPayload{
Content: append([]byte(nil), payload...),
MediaType: "application/json",
},
},
}, nil
@@ -580,12 +517,14 @@ func (merger fakeMerger) Key() string {
}
func (merger fakeMerger) Merge(ctx context.Context, req MergeRequest) (MergeResult, error) {
var candidates []artifacts.ArtifactCandidate
for _, chunkArtifacts := range req.ChunkArtifacts {
candidates = append(candidates, chunkArtifacts.Candidates...)
}
return MergeResult{Candidates: candidates}, nil
output := req.ExtractOutputs[0]
return MergeResult{Output: MergeOutput{
LaneID: req.LaneID,
MergerKey: merger.key,
SourceID: output.SourceID,
Schema: output.Schema,
Payload: cloneTestRawPayload(output.Payload),
}}, nil
}
type fakeNormalizer struct {
@@ -601,7 +540,33 @@ func (normalizer fakeNormalizer) ReferenceSlots() []ReferenceSlot {
}
func (normalizer fakeNormalizer) Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error) {
return NormalizeResult{Candidates: req.Candidates}, nil
return NormalizeResult{Output: NormalizeOutput{
LaneID: req.LaneID,
NormalizerKey: normalizer.key,
SourceID: req.MergeOutput.SourceID,
Schema: req.MergeOutput.Schema,
Payload: cloneTestRawPayload(req.MergeOutput.Payload),
}}, nil
}
func cloneTestRawPayload(payload RawPayload) RawPayload {
return RawPayload{
Content: append([]byte(nil), payload.Content...),
MediaType: payload.MediaType,
Metadata: cloneTestMetadata(payload.Metadata),
Warnings: append([]Warning(nil), payload.Warnings...),
}
}
func cloneTestMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
out := make(map[string]any, len(metadata))
for key, value := range metadata {
out[key] = value
}
return out
}
type fakeValidator struct {
@@ -651,7 +616,7 @@ func (encoder fakeOutputEncoder) Encode(ctx context.Context, req OutputRequest)
{
Name: "artifacts/generic.json",
ContentType: "application/json",
Bytes: []byte(`{"run_id":"` + req.Manifest.RunID + `","approved_count":1}`),
Bytes: []byte(`{"run_id":"` + req.Manifest.RunID + `","output_count":1}`),
},
},
}, nil

39
internal/framework/pipeline/chunk_validation.go
View File

@@ -9,8 +9,12 @@ import (
)
func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []contracts.SourceChunk) ([]contracts.SourceChunk, error) {
sourceUnitIndexes := make(map[string]int, len(doc.Units))
sourceUnits := make(map[string]source.SourceUnit, len(doc.Units))
if len(chunks) == 0 {
return nil, fmt.Errorf("chunks must not be empty")
}
sourceUnitIndexes := make(map[int]int, len(doc.Units))
sourceUnits := make(map[int]source.SourceUnit, len(doc.Units))
for index, unit := range doc.Units {
sourceUnitIndexes[unit.ID] = index
sourceUnits[unit.ID] = unit
@@ -33,25 +37,42 @@ func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []con
if chunk.Index != chunkIndex {
return nil, fmt.Errorf("chunk %q index %d does not match returned order %d", chunk.ID, chunk.Index, chunkIndex)
}
startIndex, ok := sourceUnitIndexes[chunk.StartUnitID]
if !ok {
return nil, fmt.Errorf("chunk %q start_unit_id %d was not found in source document %q", chunk.ID, chunk.StartUnitID, doc.ID)
}
endIndex, ok := sourceUnitIndexes[chunk.EndUnitID]
if !ok {
return nil, fmt.Errorf("chunk %q end_unit_id %d was not found in source document %q", chunk.ID, chunk.EndUnitID, doc.ID)
}
if startIndex > endIndex {
return nil, fmt.Errorf("chunk %q start_unit_id %d appears after end_unit_id %d", chunk.ID, chunk.StartUnitID, chunk.EndUnitID)
}
if len(chunk.Units) == 0 {
return nil, fmt.Errorf("chunk %q units must not be empty", chunk.ID)
}
if len(chunk.Content) == 0 {
return nil, fmt.Errorf("chunk %q content must not be empty", chunk.ID)
}
if strings.TrimSpace(chunk.MediaType) == "" {
return nil, fmt.Errorf("chunk %q media_type must not be empty", chunk.ID)
}
seenUnitIDs := make(map[string]struct{}, len(chunk.Units))
seenUnitIDs := make(map[int]struct{}, len(chunk.Units))
previousSourceIndex := -1
canonicalUnits := make([]source.SourceUnit, 0, len(chunk.Units))
for unitIndex, unit := range chunk.Units {
if strings.TrimSpace(unit.ID) == "" {
return nil, fmt.Errorf("chunk %q unit[%d].id must not be empty", chunk.ID, unitIndex)
if unit.ID <= 0 {
return nil, fmt.Errorf("chunk %q unit[%d].id must be positive", chunk.ID, unitIndex)
}
if _, ok := seenUnitIDs[unit.ID]; ok {
return nil, fmt.Errorf("chunk %q repeats source unit %q", chunk.ID, unit.ID)
return nil, fmt.Errorf("chunk %q repeats source unit %d", chunk.ID, unit.ID)
}
seenUnitIDs[unit.ID] = struct{}{}
sourceIndex, ok := sourceUnitIndexes[unit.ID]
if !ok {
return nil, fmt.Errorf("chunk %q source unit %q was not found in source document %q", chunk.ID, unit.ID, doc.ID)
return nil, fmt.Errorf("chunk %q source unit %d was not found in source document %q", chunk.ID, unit.ID, doc.ID)
}
if sourceIndex <= previousSourceIndex {
return nil, fmt.Errorf("chunk %q source units must appear in source document order", chunk.ID)
@@ -64,6 +85,10 @@ func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []con
ID: chunk.ID,
SourceID: chunk.SourceID,
Index: chunk.Index,
StartUnitID: chunk.StartUnitID,
EndUnitID: chunk.EndUnitID,
Content: append([]byte(nil), chunk.Content...),
MediaType: chunk.MediaType,
Units: canonicalUnits,
Metadata: cloneMetadata(chunk.Metadata),
})

6
internal/framework/pipeline/default_modules_test.go
View File

@@ -113,14 +113,8 @@ type defaultExtractor struct{}
func (defaultExtractor) Key() string { return "extract" }
func (defaultExtractor) ArtifactType() string { return "record" }
func (defaultExtractor) SchemaVersion() string { return "v1" }
func (defaultExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (defaultExtractor) Validators() []contracts.Validator { return nil }
func (defaultExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
return contracts.ExtractionResult{}, nil
}

12
internal/framework/pipeline/extractor_registry_test.go
View File

@@ -367,22 +367,10 @@ func (extractor registryFakeExtractor) Key() string {
return extractor.key
}
func (extractor registryFakeExtractor) ArtifactType() string {
return "generic-artifact"
}
func (extractor registryFakeExtractor) SchemaVersion() string {
return "v1"
}
func (extractor registryFakeExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (extractor registryFakeExtractor) Validators() []contracts.Validator {
return nil
}
func (extractor registryFakeExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
return contracts.ExtractionResult{}, nil
}

2
internal/framework/pipeline/input_registry_test.go
View File

@@ -301,7 +301,7 @@ func (adapter fakeAdapter) Parse(ctx context.Context, req contracts.ParseRequest
Format: "text/plain",
Digest: "sha256:abc123",
Units: []source.SourceUnit{
{ID: "u1", Kind: "unit", Text: "Source unit."},
{ID: 1, Kind: "unit", Text: "Source unit."},
},
}, nil
}

2
internal/framework/pipeline/module.go
View File

@@ -97,7 +97,7 @@ func validateModuleSpec(kind string, expectedStage ModuleStage, spec ModuleSpec)
}
func referenceSlotStage(stage ModuleStage) bool {
return stage == StageChunk || stage == StageExtract || stage == StageNormalize
return stage == StageChunk || stage == StageExtract || stage == StageMerge || stage == StageNormalize
}
func sortedRegistryKeys[C any](constructors map[string]C) []string {

3
internal/framework/pipeline/module_test.go
View File

@@ -15,6 +15,7 @@ func TestValidateModuleSpecAllowsReferenceSlotsForEligibleStages(t *testing.T) {
}{
{name: "chunker", kind: "chunker", stage: StageChunk},
{name: "extractor", kind: "extractor", stage: StageExtract},
{name: "merger", kind: "merger", stage: StageMerge},
{name: "normalizer", kind: "normalizer", stage: StageNormalize},
}
@@ -42,7 +43,6 @@ func TestValidateModuleSpecRejectsReferenceSlotsForIneligibleStages(t *testing.T
stage ModuleStage
}{
{name: "input", kind: "input adapter", stage: StageInput},
{name: "merge", kind: "merger", stage: StageMerge},
{name: "validate", kind: "validator", stage: StageValidate},
{name: "output", kind: "output encoder", stage: StageOutput},
}
@@ -99,6 +99,7 @@ func TestValidateModuleSpecRejectsInvalidReferenceSlotsForEligibleStages(t *test
}{
{name: "chunk", kind: "chunker", stage: StageChunk},
{name: "extract", kind: "extractor", stage: StageExtract},
{name: "merge", kind: "merger", stage: StageMerge},
{name: "normalize", kind: "normalizer", stage: StageNormalize},
}

28
internal/framework/pipeline/profile.go
View File

@@ -22,6 +22,7 @@ const (
type ModuleBinding struct {
Module string `json:"module"`
LLMProfile string `json:"llm_profile,omitempty"`
Retries int `json:"retries,omitempty"`
Options map[string]any `json:"options,omitempty"`
References map[string]string `json:"references,omitempty"`
}
@@ -78,6 +79,7 @@ type ResolvedArtifactLane struct {
Normalize ModuleBinding
Validators []ModuleBinding
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
}
@@ -251,6 +253,20 @@ func resolveArtifactLane(
if missing, ok := capabilities.missing(mergeSpec.Requires); ok {
return ResolvedArtifactLane{}, nil, capabilityError(pipelineID, laneID, StageMerge, lane.Merge.Module, missing)
}
mergeReferences, err := resolveReferenceTargetBindings(referenceResolutionTarget{
PipelineID: pipelineID,
LaneID: laneID,
Stage: StageMerge,
Module: lane.Merge.Module,
Slots: mergeSpec.ReferenceSlots,
PipelineReferences: pipelineReferences,
LocalReferences: lane.Merge.References,
Options: options,
})
if err != nil {
return ResolvedArtifactLane{}, nil, err
}
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
capabilities.add(mergeSpec.Provides...)
normalizeSpec, err := normalizerSpec(catalog, lane.Normalize.Module)
@@ -346,6 +362,15 @@ func validatePipelineReferenceDefaults(
declaredByAnyTarget[slot.Name] = struct{}{}
}
merge := resolveBinding(laneProfile.Merge, DefaultMergeModule)
mergeSpec, err := mergerSpec(catalog, merge.Module)
if err != nil {
return moduleLookupError(pipelineID, laneID, StageMerge, merge.Module, err)
}
for _, slot := range mergeSpec.ReferenceSlots {
declaredByAnyTarget[slot.Name] = struct{}{}
}
normalize := resolveBinding(laneProfile.Normalize, DefaultNormalizeModule)
normalizeSpec, err := normalizerSpec(catalog, normalize.Module)
if err != nil {
@@ -500,7 +525,7 @@ func normalizeReferenceOptionTarget(pipelineID string, operation string, stage M
if laneID != "" {
return "", "", fmt.Errorf("pipeline %q reference %s for chunk must not include a lane id", pipelineID, operation)
}
case StageExtract, StageNormalize:
case StageExtract, StageMerge, StageNormalize:
if laneID == "" {
return "", "", fmt.Errorf("pipeline %q reference %s lane id must not be empty", pipelineID, operation)
}
@@ -600,6 +625,7 @@ func resolveBinding(binding ModuleBinding, defaultModule string) ModuleBinding {
return ModuleBinding{
Module: module,
LLMProfile: llmProfile,
Retries: binding.Retries,
Options: cloneOptions(binding.Options),
References: normalizeReferenceMap(binding.References),
}

58
internal/framework/pipeline/profile_test.go
View File

@@ -160,6 +160,9 @@ func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
if events.NormalizeReferences.Stage != StageNormalize || events.NormalizeReferences.LaneID != "events" || events.NormalizeReferences.Module != DefaultNormalizeModule {
t.Fatalf("normalize reference target = %#v, want event normalizer target", events.NormalizeReferences)
}
if events.MergeReferences.Stage != StageMerge || events.MergeReferences.LaneID != "events" || events.MergeReferences.Module != DefaultMergeModule {
t.Fatalf("merge reference target = %#v, want event merger target", events.MergeReferences)
}
if resolved.ChunkReferences.Stage != StageChunk || resolved.ChunkReferences.Module != DefaultChunkModule {
t.Fatalf("chunk reference target = %#v, want chunk target", resolved.ChunkReferences)
}
@@ -260,12 +263,44 @@ func TestResolvePipelineAppliesPipelineReferenceDefaultToNormalizerTarget(t *tes
}
}
func TestResolvePipelineAppliesPipelineReferenceDefaultToMergeTarget(t *testing.T) {
profile := baselineProfile()
profile.References = map[string]string{"merge_notes": "./merge.md"}
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "appendorder",
Stage: StageMerge,
Requires: []string{"candidate"},
Provides: []string{"merged"},
ReferenceSlots: []contracts.ReferenceSlot{{Name: "merge_notes"}},
})
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
want := []ReferenceBinding{{LaneID: "events", SlotName: "merge_notes", Source: "./merge.md", BindingSource: contracts.ReferenceBindingSourceConfig}}
if !reflect.DeepEqual(resolved.ArtifactLanes[0].MergeReferences.Bindings, want) {
t.Fatalf("merge references = %#v, want %#v", resolved.ArtifactLanes[0].MergeReferences.Bindings, want)
}
if refs := resolved.ChunkReferences.Bindings; len(refs) != 0 {
t.Fatalf("chunk references = %#v, want none", refs)
}
if refs := resolved.ArtifactLanes[0].ExtractReferences.Bindings; len(refs) != 0 {
t.Fatalf("extract references = %#v, want none", refs)
}
if refs := resolved.ArtifactLanes[0].NormalizeReferences.Bindings; len(refs) != 0 {
t.Fatalf("normalize references = %#v, want none", refs)
}
}
func TestResolvePipelineAppliesOnePipelineReferenceDefaultToMultipleTargets(t *testing.T) {
profile := baselineProfile()
profile.References = map[string]string{"context": "./context.md"}
catalog := newProfileCatalogWithOverrides(t,
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ReferenceSlots: []contracts.ReferenceSlot{{Name: "context"}}},
)
@@ -276,6 +311,7 @@ func TestResolvePipelineAppliesOnePipelineReferenceDefaultToMultipleTargets(t *t
assertBindingSource(t, resolved.ChunkReferences.Bindings, "context", "./context.md")
assertBindingSource(t, resolved.ArtifactLanes[0].ExtractReferences.Bindings, "context", "./context.md")
assertBindingSource(t, resolved.ArtifactLanes[0].MergeReferences.Bindings, "context", "./context.md")
assertBindingSource(t, resolved.ArtifactLanes[0].NormalizeReferences.Bindings, "context", "./context.md")
}
@@ -373,6 +409,28 @@ func TestResolvePipelineRejectsExtractLocalReferenceDeclaredOnlyByNormalizer(t *
assertErrorContains(t, err, "baseline", "events", "extract", "event-extractor", "normalization_notes", "not declared")
}
func TestResolvePipelineRejectsMergeLocalReferenceDeclaredOnlyByNormalizer(t *testing.T) {
profile := baselineProfile()
lane := profile.Artifacts["events"]
lane.Merge.References = map[string]string{"normalization_notes": "./normalize.md"}
profile.Artifacts["events"] = lane
catalog := newProfileCatalogWithOverrides(t, ModuleSpec{
Key: "noop",
Stage: StageNormalize,
Requires: []string{"merged"},
Provides: []string{"normalized"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "normalization_notes"},
},
})
_, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
if err == nil {
t.Fatal("ResolvePipeline() error = nil, want error")
}
assertErrorContains(t, err, "merge", "normalization_notes", "not declared")
}
func TestResolvePipelineRejectsNormalizeLocalReferenceDeclaredOnlyByExtractor(t *testing.T) {
profile := baselineProfile()
lane := profile.Artifacts["events"]

73
internal/framework/pipeline/raw_validation_registry.go Normal file
View File

@@ -0,0 +1,73 @@
package pipeline
import (
"fmt"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type rawValidationKey struct {
stage ModuleStage
module string
}
type RawValidationRegistry struct {
chains map[rawValidationKey][]contracts.RawValidator
}
func NewRawValidationRegistry() *RawValidationRegistry {
return &RawValidationRegistry{
chains: make(map[rawValidationKey][]contracts.RawValidator),
}
}
func (r *RawValidationRegistry) Register(stage ModuleStage, module string, validators ...contracts.RawValidator) error {
if r == nil {
return fmt.Errorf("raw validation registry must not be nil")
}
normalizedModule := strings.TrimSpace(module)
if normalizedModule == "" {
return fmt.Errorf("raw validation module key must not be empty")
}
switch stage {
case StageChunk, StageExtract, StageMerge, StageNormalize:
default:
return fmt.Errorf("raw validation stage %q is not supported", stage)
}
if len(validators) == 0 {
return fmt.Errorf("raw validation chain for %q %q must not be empty", stage, normalizedModule)
}
chain := make([]contracts.RawValidator, 0, len(validators))
for i, validator := range validators {
if validator == nil {
return fmt.Errorf("raw validator %d for %q %q must not be nil", i, stage, normalizedModule)
}
if strings.TrimSpace(validator.Name()) == "" {
return fmt.Errorf("raw validator %d for %q %q must not have an empty name", i, stage, normalizedModule)
}
chain = append(chain, validator)
}
if r.chains == nil {
r.chains = make(map[rawValidationKey][]contracts.RawValidator)
}
key := rawValidationKey{stage: stage, module: normalizedModule}
if _, exists := r.chains[key]; exists {
return fmt.Errorf("raw validation chain for %q %q is already registered", stage, normalizedModule)
}
r.chains[key] = append([]contracts.RawValidator(nil), chain...)
return nil
}
func (r *RawValidationRegistry) Validators(stage ModuleStage, module string) []contracts.RawValidator {
if r == nil {
return nil
}
chain := r.chains[rawValidationKey{stage: stage, module: strings.TrimSpace(module)}]
if len(chain) == 0 {
return nil
}
return append([]contracts.RawValidator(nil), chain...)
}

11
internal/framework/pipeline/references.go
View File

@@ -44,6 +44,7 @@ func MaterializeReferences(resolved ResolvedPipeline, catalog ModuleCatalog, opt
for i, lane := range resolved.ArtifactLanes {
materializedLane := lane
materializedLane.ExtractReferences = CloneReferenceTarget(lane.ExtractReferences)
materializedLane.MergeReferences = CloneReferenceTarget(lane.MergeReferences)
materializedLane.NormalizeReferences = CloneReferenceTarget(lane.NormalizeReferences)
extractReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.ExtractReferences, catalog, options)
if err != nil {
@@ -52,6 +53,13 @@ func MaterializeReferences(resolved ResolvedPipeline, catalog ModuleCatalog, opt
materializedLane.ExtractReferences.ReferenceSet = extractReferenceSet
warnings = append(warnings, laneWarnings...)
mergeReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.MergeReferences, catalog, options)
if err != nil {
return ResolvedPipeline{}, nil, err
}
materializedLane.MergeReferences.ReferenceSet = mergeReferenceSet
warnings = append(warnings, laneWarnings...)
normalizeReferenceSet, laneWarnings, err := materializeReferenceTarget(resolved.ID, lane.NormalizeReferences, catalog, options)
if err != nil {
return ResolvedPipeline{}, nil, err
@@ -140,6 +148,8 @@ func referenceTargetSpec(target ResolvedReferenceTarget, catalog ModuleCatalog)
return registrySpec(catalog.Chunkers, target.Module)
case StageExtract:
return registrySpec(catalog.Extractors, target.Module)
case StageMerge:
return registrySpec(catalog.Mergers, target.Module)
case StageNormalize:
return registrySpec(catalog.Normalizers, target.Module)
default:
@@ -291,6 +301,7 @@ func ReferenceProvenance(resolved ResolvedPipeline) []artifacts.ReferenceProvena
provenance = append(provenance, referenceTargetProvenance(resolved.ChunkReferences)...)
for _, lane := range resolved.ArtifactLanes {
provenance = append(provenance, referenceTargetProvenance(lane.ExtractReferences)...)
provenance = append(provenance, referenceTargetProvenance(lane.MergeReferences)...)
provenance = append(provenance, referenceTargetProvenance(lane.NormalizeReferences)...)
}
return provenance

44
internal/framework/pipeline/references_test.go
View File

@@ -109,17 +109,20 @@ func TestMaterializeReferencesStoresSetsAndProvenanceForAllTargets(t *testing.T)
configDir := t.TempDir()
writeReferenceFile(t, filepath.Join(configDir, "chunk.txt"), []byte("chunk text"))
writeReferenceFile(t, filepath.Join(configDir, "extract.txt"), []byte("extract text"))
writeReferenceFile(t, filepath.Join(configDir, "merge.txt"), []byte("merge text"))
writeReferenceFile(t, filepath.Join(configDir, "normalize.txt"), []byte("normalize text"))
profile := baselineProfile()
profile.References = map[string]string{
"scene_guide": "chunk.txt",
"roster": "extract.txt",
"merge_notes": "merge.txt",
"normalization_notes": "normalize.txt",
}
catalog := referenceCatalogForTargets(t,
[]contracts.ReferenceSlot{{Name: "scene_guide"}},
[]contracts.ReferenceSlot{{Name: "roster"}},
[]contracts.ReferenceSlot{{Name: "merge_notes"}},
[]contracts.ReferenceSlot{{Name: "normalization_notes"}},
)
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
@@ -145,14 +148,18 @@ func TestMaterializeReferencesStoresSetsAndProvenanceForAllTargets(t *testing.T)
if string(extractItem.Content) != "extract text" {
t.Fatalf("extract content = %q, want extract text", extractItem.Content)
}
mergeItem := materialized.ArtifactLanes[0].MergeReferences.ReferenceSet.Slots["merge_notes"].Items[0]
if string(mergeItem.Content) != "merge text" {
t.Fatalf("merge content = %q, want merge text", mergeItem.Content)
}
normalizeItem := materialized.ArtifactLanes[0].NormalizeReferences.ReferenceSet.Slots["normalization_notes"].Items[0]
if string(normalizeItem.Content) != "normalize text" {
t.Fatalf("normalize content = %q, want normalize text", normalizeItem.Content)
}
provenance := ReferenceProvenance(materialized)
if len(provenance) != 3 {
t.Fatalf("ReferenceProvenance() = %#v, want three entries", provenance)
if len(provenance) != 4 {
t.Fatalf("ReferenceProvenance() = %#v, want four entries", provenance)
}
if provenance[0].Stage != string(StageChunk) || provenance[0].LaneID != "" || provenance[0].SlotName != "scene_guide" || provenance[0].Digest != chunkItem.Digest {
t.Fatalf("ReferenceProvenance()[0] = %#v, want chunk scene guide provenance", provenance[0])
@@ -160,7 +167,10 @@ func TestMaterializeReferencesStoresSetsAndProvenanceForAllTargets(t *testing.T)
if provenance[1].Stage != string(StageExtract) || provenance[1].LaneID != "events" || provenance[1].SlotName != "roster" || provenance[1].Digest != extractItem.Digest {
t.Fatalf("ReferenceProvenance()[1] = %#v, want extract roster provenance", provenance[1])
}
if provenance[2].Stage != string(StageNormalize) || provenance[2].LaneID != "events" || provenance[2].SlotName != "normalization_notes" || provenance[2].Digest != normalizeItem.Digest {
if provenance[2].Stage != string(StageMerge) || provenance[2].LaneID != "events" || provenance[2].SlotName != "merge_notes" || provenance[2].Digest != mergeItem.Digest {
t.Fatalf("ReferenceProvenance()[2] = %#v, want merge notes provenance", provenance[2])
}
if provenance[3].Stage != string(StageNormalize) || provenance[3].LaneID != "events" || provenance[3].SlotName != "normalization_notes" || provenance[3].Digest != normalizeItem.Digest {
t.Fatalf("ReferenceProvenance()[2] = %#v, want normalize notes provenance", provenance[2])
}
}
@@ -185,7 +195,7 @@ func TestMaterializeReferencesRejectsNonUTF8ContentForChunkTarget(t *testing.T)
writeReferenceFile(t, path, []byte{0xff, 0xfe})
resolved := resolvedPipelineWithTargetReference(t, StageChunk, "", "scene_guide", "bad.txt", contracts.ReferenceBindingSourceConfig, contracts.ReferenceSlot{Name: "scene_guide"})
_, _, err := MaterializeReferences(resolved, referenceCatalogForTargets(t, []contracts.ReferenceSlot{{Name: "scene_guide"}}, nil, nil), ReferenceMaterializationOptions{
_, _, err := MaterializeReferences(resolved, referenceCatalogForTargets(t, []contracts.ReferenceSlot{{Name: "scene_guide"}}, nil, nil, nil), ReferenceMaterializationOptions{
ConfigPath: filepath.Join(configDir, "config.yml"),
})
if err == nil || !strings.Contains(err.Error(), "chunk") || !strings.Contains(err.Error(), "UTF-8") || !strings.Contains(err.Error(), "scene_guide") || !strings.Contains(err.Error(), path) {
@@ -309,7 +319,7 @@ func TestMaterializeReferencesRejectsUnacceptedMediaTypeForNormalizeTarget(t *te
slot := contracts.ReferenceSlot{Name: "normalization_notes", AcceptedMediaTypes: []string{"text/markdown"}}
resolved := resolvedPipelineWithTargetReference(t, StageNormalize, "events", "normalization_notes", "notes.json", contracts.ReferenceBindingSourceConfig, slot)
_, _, err := MaterializeReferences(resolved, referenceCatalogForTargets(t, nil, nil, []contracts.ReferenceSlot{slot}), ReferenceMaterializationOptions{
_, _, err := MaterializeReferences(resolved, referenceCatalogForTargets(t, nil, nil, nil, []contracts.ReferenceSlot{slot}), ReferenceMaterializationOptions{
ConfigPath: filepath.Join(configDir, "config.yml"),
})
if err == nil || !strings.Contains(err.Error(), "normalize") || !strings.Contains(err.Error(), "media type") || !strings.Contains(err.Error(), "application/json") || !strings.Contains(err.Error(), "normalization_notes") {
@@ -353,6 +363,7 @@ func TestMaterializeReferencesWarningScopesIncludeTargetContext(t *testing.T) {
catalog := referenceCatalogForTargets(t,
[]contracts.ReferenceSlot{{Name: "scene_guide"}},
[]contracts.ReferenceSlot{{Name: "roster"}},
nil,
[]contracts.ReferenceSlot{{Name: "normalization_notes"}},
)
resolved, err := ResolvePipeline(profile, ResolveOptions{}, catalog)
@@ -407,6 +418,10 @@ func resolvedPipelineWithTargetReference(t *testing.T, stage ModuleStage, laneID
lane := profile.Artifacts[laneID]
lane.References = map[string]string{slotName: source}
profile.Artifacts[laneID] = lane
case StageMerge:
lane := profile.Artifacts[laneID]
lane.Merge.References = map[string]string{slotName: source}
profile.Artifacts[laneID] = lane
case StageNormalize:
lane := profile.Artifacts[laneID]
lane.Normalize.References = map[string]string{slotName: source}
@@ -425,6 +440,8 @@ func resolvedPipelineWithTargetReference(t *testing.T, stage ModuleStage, laneID
resolved.ChunkReferences.Bindings[0].BindingSource = bindingSource
case StageExtract:
resolved.ArtifactLanes[0].ExtractReferences.Bindings[0].BindingSource = bindingSource
case StageMerge:
resolved.ArtifactLanes[0].MergeReferences.Bindings[0].BindingSource = bindingSource
case StageNormalize:
resolved.ArtifactLanes[0].NormalizeReferences.Bindings[0].BindingSource = bindingSource
}
@@ -441,18 +458,20 @@ func referenceCatalogForStage(t *testing.T, stage ModuleStage, slots []contracts
t.Helper()
switch stage {
case StageChunk:
return referenceCatalogForTargets(t, slots, nil, nil)
return referenceCatalogForTargets(t, slots, nil, nil, nil)
case StageExtract:
return referenceCatalogForTargets(t, nil, slots, nil)
return referenceCatalogForTargets(t, nil, slots, nil, nil)
case StageMerge:
return referenceCatalogForTargets(t, nil, nil, slots, nil)
case StageNormalize:
return referenceCatalogForTargets(t, nil, nil, slots)
return referenceCatalogForTargets(t, nil, nil, nil, slots)
default:
t.Fatalf("unsupported reference target stage %q", stage)
return ModuleCatalog{}
}
}
func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, normalizeSlots []contracts.ReferenceSlot) ModuleCatalog {
func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, mergeSlots, normalizeSlots []contracts.ReferenceSlot) ModuleCatalog {
t.Helper()
return newProfileCatalogWithOverrides(t,
ModuleSpec{
@@ -469,6 +488,13 @@ func referenceCatalogForTargets(t *testing.T, chunkSlots, extractSlots, normaliz
Provides: []string{"candidate"},
ReferenceSlots: extractSlots,
},
ModuleSpec{
Key: "appendorder",
Stage: StageMerge,
Requires: []string{"candidate"},
Provides: []string{"merged"},
ReferenceSlots: mergeSlots,
},
ModuleSpec{
Key: "noop",
Stage: StageNormalize,

116
internal/framework/pipeline/registry_integration_test.go
View File

@@ -5,11 +5,8 @@ import (
"reflect"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
validate "gitea.maximumdirect.net/eric/notarius/internal/framework/validate"
)
func TestRunnerUsesRegistries(t *testing.T) {
@@ -33,11 +30,11 @@ func TestRunnerUsesRegistries(t *testing.T) {
if !reflect.DeepEqual(executed, []string{"extract-first:chunk-0", "extract-second:chunk-0"}) {
t.Fatalf("executed = %#v, want extractor chunk execution", executed)
}
if got := artifactKeys(output.Approved); !reflect.DeepEqual(got, []string{"extract-first"}) {
t.Fatalf("approved keys = %#v, want [extract-first]", got)
if got := normalizeOutputKeys(output.NormalizeOutputs); !reflect.DeepEqual(got, []string{"normalize", "normalize"}) {
t.Fatalf("normalize output keys = %#v, want one output from each lane", got)
}
if got := rejectedKeys(output.Rejected); !reflect.DeepEqual(got, []string{"extract-second"}) {
t.Fatalf("rejected keys = %#v, want [extract-second]", got)
if len(output.Rejected) != 0 {
t.Fatalf("len(Rejected) = %d, want none", len(output.Rejected))
}
}
@@ -64,12 +61,8 @@ func integrationRegistries(t *testing.T, built, executed *[]string) Registries {
}); err != nil {
t.Fatalf("register chunker: %v", err)
}
registerIntegrationExtractor(t, registries.Extractors, "extract-first", built, executed, []contracts.Validator{
integrationValidator{name: "approve-first", approve: true},
})
registerIntegrationExtractor(t, registries.Extractors, "extract-second", built, executed, []contracts.Validator{
integrationValidator{name: "reject-second", approve: false},
})
registerIntegrationExtractor(t, registries.Extractors, "extract-first", built, executed)
registerIntegrationExtractor(t, registries.Extractors, "extract-second", built, executed)
if err := registries.Mergers.Register("merge", func() (contracts.Merger, error) {
*built = append(*built, "merge")
return integrationMerger{}, nil
@@ -91,12 +84,12 @@ func integrationRegistries(t *testing.T, built, executed *[]string) Registries {
return registries
}
func registerIntegrationExtractor(t *testing.T, registry *ExtractorRegistry, key string, built, executed *[]string, validators []contracts.Validator) {
func registerIntegrationExtractor(t *testing.T, registry *ExtractorRegistry, key string, built, executed *[]string) {
t.Helper()
if err := registry.Register(key, func() (contracts.Extractor, error) {
*built = append(*built, key)
return integrationExtractor{key: key, executed: executed, validators: validators}, nil
return integrationExtractor{key: key, executed: executed}, nil
}); err != nil {
t.Fatalf("Register(%q) error = %v, want nil", key, err)
}
@@ -129,6 +122,10 @@ func (chunker integrationChunker) Chunk(ctx context.Context, req contracts.Chunk
ID: "chunk-0",
SourceID: req.Source.ID,
Index: 0,
StartUnitID: req.Source.Units[0].ID,
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
Content: []byte(`{"units":[1]}`),
MediaType: "application/json",
Units: req.Source.Units,
},
},
@@ -138,34 +135,25 @@ func (chunker integrationChunker) Chunk(ctx context.Context, req contracts.Chunk
type integrationExtractor struct {
key string
executed *[]string
validators []contracts.Validator
}
func (extractor integrationExtractor) Key() string {
return extractor.key
}
func (extractor integrationExtractor) ArtifactType() string {
return "generic-artifact"
}
func (extractor integrationExtractor) SchemaVersion() string {
return "v1"
}
func (extractor integrationExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (extractor integrationExtractor) Validators() []contracts.Validator {
return extractor.validators
}
func (extractor integrationExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
*extractor.executed = append(*extractor.executed, extractor.key+":"+req.Chunk.ID)
return contracts.ExtractionResult{
Candidates: []artifacts.ArtifactCandidate{
{Payload: []byte(`{"value":true}`)},
Output: contracts.ExtractOutput{
Schema: contracts.ResponseSchema{ID: "integration", Name: "integration", Version: "v1"},
Payload: contracts.RawPayload{
Content: []byte(`{"value":true}`),
MediaType: "application/json",
},
},
}, nil
}
@@ -179,11 +167,20 @@ func (merger integrationMerger) Key() string {
}
func (merger integrationMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
var candidates []artifacts.ArtifactCandidate
for _, chunkArtifacts := range req.ChunkArtifacts {
candidates = append(candidates, chunkArtifacts.Candidates...)
output := contracts.MergeOutput{
LaneID: req.LaneID,
Schema: contracts.ResponseSchema{ID: "integration", Name: "integration", Version: "v1"},
Payload: contracts.RawPayload{
Content: []byte(`{"merged":true}`),
MediaType: "application/json",
},
}
return contracts.MergeResult{Candidates: candidates}, nil
if len(req.ExtractOutputs) > 0 {
output.SourceID = req.ExtractOutputs[0].SourceID
output.Schema = req.ExtractOutputs[0].Schema
output.Payload = req.ExtractOutputs[0].Payload
}
return contracts.MergeResult{Output: output}, nil
}
func (normalizer integrationNormalizer) Key() string {
@@ -195,7 +192,14 @@ func (normalizer integrationNormalizer) ReferenceSlots() []contracts.ReferenceSl
}
func (normalizer integrationNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
return contracts.NormalizeResult{Candidates: req.Candidates}, nil
return contracts.NormalizeResult{
Output: contracts.NormalizeOutput{
LaneID: req.LaneID,
SourceID: req.MergeOutput.SourceID,
Schema: req.MergeOutput.Schema,
Payload: req.MergeOutput.Payload,
},
}, nil
}
type integrationOutput struct{}
@@ -212,30 +216,6 @@ func (output integrationOutput) Encode(ctx context.Context, req contracts.Output
}, nil
}
type integrationValidator struct {
name string
approve bool
}
func (validator integrationValidator) Name() string {
return validator.name
}
func (validator integrationValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
decisions := make([]contracts.ValidationDecision, 0, len(req.Candidates))
for _, candidate := range req.Candidates {
if validator.approve {
decisions = append(decisions, validate.Approved(candidate.Index))
} else {
decisions = append(decisions, validate.Rejected(candidate.Index, "invalid", "not accepted"))
}
}
return contracts.ValidationResult{
ValidatorName: validator.name,
Decisions: decisions,
}, nil
}
func integrationPipeline() ResolvedPipeline {
return ResolvedPipeline{
ID: "pipeline-1",
@@ -267,23 +247,15 @@ func integrationSourceDocument() *source.SourceDocument {
Format: "text/plain",
Digest: "sha256:abc123",
Units: []source.SourceUnit{
{ID: "u1", Kind: "unit", Text: "Source unit."},
{ID: 1, Kind: "unit", Text: "Source unit."},
},
}
}
func artifactKeys(approved []artifacts.Artifact) []string {
keys := make([]string, 0, len(approved))
for _, artifact := range approved {
keys = append(keys, artifact.ExtractorKey)
}
return keys
}
func rejectedKeys(rejected []artifacts.RejectedArtifact) []string {
keys := make([]string, 0, len(rejected))
for _, artifact := range rejected {
keys = append(keys, artifact.Candidate.ExtractorKey)
func normalizeOutputKeys(outputs []contracts.NormalizeOutput) []string {
keys := make([]string, 0, len(outputs))
for _, output := range outputs {
keys = append(keys, output.NormalizerKey)
}
return keys
}

554
internal/framework/pipeline/runner.go
View File

@@ -15,7 +15,6 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/validate"
)
type Registries struct {
@@ -25,6 +24,7 @@ type Registries struct {
Mergers *MergerRegistry
Normalizers *NormalizerRegistry
Validators *ValidatorRegistry
RawValidators *RawValidationRegistry
Outputs *OutputEncoderRegistry
}
@@ -52,8 +52,8 @@ type RunInput struct {
type RunOutput struct {
Manifest artifacts.RunManifest `json:"manifest"`
Approved []artifacts.Artifact `json:"approved,omitempty"`
Rejected []artifacts.RejectedArtifact `json:"rejected,omitempty"`
NormalizeOutputs []contracts.NormalizeOutput `json:"normalize_outputs,omitempty"`
Rejected []contracts.RejectedOutput `json:"rejected,omitempty"`
Warnings []contracts.Warning `json:"warnings,omitempty"`
OutputFiles []contracts.OutputFile `json:"-"`
}
@@ -104,6 +104,9 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
return failOutput(output), fmt.Errorf("build chunker %q: %w", input.Pipeline.Chunk.Module, err)
}
attachModuleManifestMetadata(&output, "chunker", chunker)
var canonicalChunks []contracts.SourceChunk
var chunkWarnings []contracts.Warning
chunksAccepted, chunkRejection, err := runWithRetry(ctx, input.Pipeline.Chunk.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
chunkResult, err := chunker.Chunk(ctx, contracts.ChunkRequest{
Source: doc,
SourceInput: sourceInput.Clone(),
@@ -114,30 +117,47 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
Options: cloneOptions(input.Pipeline.Chunk.Options),
Metadata: input.Metadata,
})
output.Warnings = append(output.Warnings, chunkResult.Warnings...)
if err != nil {
return failOutput(output), fmt.Errorf("chunk source with chunker %q: %w", chunker.Key(), err)
return false, nil, fmt.Errorf("chunk source with chunker %q: %w", chunker.Key(), err)
}
if len(chunkResult.Chunks) == 0 {
return failOutput(output), fmt.Errorf("chunker %q returned no chunks", chunker.Key())
return false, nil, fmt.Errorf("chunker %q returned no chunks", chunker.Key())
}
canonicalChunks, err := validateAndCanonicalizeChunkResult(doc, chunkResult.Chunks)
chunks, err := validateAndCanonicalizeChunkResult(doc, chunkResult.Chunks)
if err != nil {
return failOutput(output), fmt.Errorf("validate chunks from chunker %q: %w", chunker.Key(), err)
return false, nil, fmt.Errorf("validate chunks from chunker %q: %w", chunker.Key(), err)
}
rejection, err := r.validateChunksRaw(ctx, doc, chunker.Key(), chunks, input.Metadata, attempt)
if err != nil || rejection != nil {
return false, rejection, err
}
canonicalChunks = chunks
chunkWarnings = cloneWarnings(chunkResult.Warnings)
return true, nil, nil
})
if err != nil {
return failOutput(output), err
}
if !chunksAccepted {
output.Rejected = append(output.Rejected, *chunkRejection)
} else {
output.Warnings = append(output.Warnings, chunkWarnings...)
}
nextCandidateIndex := 0
if chunksAccepted {
for _, lane := range input.Pipeline.ArtifactLanes {
if err := r.runLane(ctx, input, doc, sourceInput, sessionID, canonicalChunks, lane, &output, &nextCandidateIndex); err != nil {
if err := r.runLane(ctx, input, doc, sourceInput, sessionID, canonicalChunks, lane, &output); err != nil {
return failOutput(output), err
}
}
}
if len(output.Rejected) > 0 {
output.Manifest.ValidationStatus = "rejected"
} else {
output.Manifest.ValidationStatus = "approved"
}
populateRawOutputManifest(&output)
output.Manifest.CompletedAt = timePtr(time.Now().UTC())
encoder, err := r.registries.Outputs.Build(input.Pipeline.Output.Module)
@@ -147,8 +167,8 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
attachModuleManifestMetadata(&output, "output", encoder)
encoded, err := encoder.Encode(ctx, contracts.OutputRequest{
Manifest: output.Manifest,
Approved: output.Approved,
Rejected: output.Rejected,
NormalizeOutputs: cloneNormalizeOutputs(output.NormalizeOutputs),
Rejected: cloneRejectedOutputs(output.Rejected),
Warnings: output.Warnings,
LLMProfile: input.Pipeline.Output.LLMProfile,
Options: cloneOptions(input.Pipeline.Output.Options),
@@ -167,7 +187,7 @@ func (r *Runner) Run(ctx context.Context, input RunInput) (output RunOutput, err
return output, nil
}
func (r *Runner) runLane(ctx context.Context, input RunInput, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []contracts.SourceChunk, lane ResolvedArtifactLane, output *RunOutput, nextCandidateIndex *int) error {
func (r *Runner) runLane(ctx context.Context, input RunInput, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []contracts.SourceChunk, lane ResolvedArtifactLane, output *RunOutput) error {
extractor, err := r.registries.Extractors.Build(lane.Extract.Module)
if err != nil {
return fmt.Errorf("build extractor %q for lane %q: %w", lane.Extract.Module, lane.ID, err)
@@ -182,21 +202,12 @@ func (r *Runner) runLane(ctx context.Context, input RunInput, doc *source.Source
}
setLaneManifestMetadata(output, lane.ID, extractor, merger, normalizer)
var validators []validatorExecution
if len(lane.Validators) > 0 {
validators, err = r.buildConfiguredValidators(lane)
if err != nil {
return err
}
} else {
for _, validator := range extractor.Validators() {
validators = append(validators, validatorExecution{validator: validator})
}
}
chunkArtifacts := make([]contracts.ChunkArtifacts, 0, len(chunks))
extractOutputs := make([]contracts.ExtractOutput, 0, len(chunks))
for index := range chunks {
chunk := chunks[index]
var acceptedOutput contracts.ExtractOutput
var acceptedWarnings []contracts.Warning
accepted, rejection, err := runWithRetry(ctx, lane.Extract.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
result, err := extractor.Extract(ctx, contracts.ExtractionRequest{
Source: doc,
Chunk: &chunk,
@@ -208,38 +219,108 @@ func (r *Runner) runLane(ctx context.Context, input RunInput, doc *source.Source
Options: cloneOptions(lane.Extract.Options),
Metadata: input.Metadata,
})
output.Warnings = append(output.Warnings, result.Warnings...)
if err != nil {
return fmt.Errorf("extract lane %q chunk %q with extractor %q: %w", lane.ID, chunk.ID, extractor.Key(), err)
return false, nil, fmt.Errorf("extract lane %q chunk %q with extractor %q: %w", lane.ID, chunk.ID, extractor.Key(), err)
}
candidates, err := normalizeCandidates(extractor, result.Candidates, nextCandidateIndex)
extractOutput := result.Output
extractOutput.LaneID = lane.ID
extractOutput.ExtractorKey = extractor.Key()
extractOutput.SourceID = doc.ID
extractOutput.ChunkID = chunk.ID
extractOutput.ChunkIndex = chunk.Index
extractOutput.Payload.Warnings = append(extractOutput.Payload.Warnings, result.Warnings...)
validationWarnings, rejection, err := r.validateRaw(ctx, rawValidationTarget{
stage: StageExtract,
laneID: lane.ID,
moduleKey: extractor.Key(),
source: doc,
sourceID: doc.ID,
chunkID: chunk.ID,
chunkIndex: chunk.Index,
schema: extractOutput.Schema,
payload: extractOutput.Payload,
metadata: input.Metadata,
attempt: attempt,
})
if err != nil || rejection != nil {
return false, rejection, err
}
acceptedOutput = cloneExtractOutput(extractOutput)
acceptedWarnings = append(cloneWarnings(result.Warnings), validationWarnings...)
return true, nil, nil
})
if err != nil {
return err
}
chunkArtifacts = append(chunkArtifacts, contracts.ChunkArtifacts{
Chunk: chunk,
Candidates: candidates,
})
if !accepted {
output.Rejected = append(output.Rejected, *rejection)
continue
}
output.Warnings = append(output.Warnings, acceptedWarnings...)
extractOutputs = append(extractOutputs, acceptedOutput)
}
if len(extractOutputs) == 0 {
return nil
}
var acceptedMerge contracts.MergeOutput
var mergeWarnings []contracts.Warning
mergeAccepted, mergeRejection, err := runWithRetry(ctx, lane.Merge.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
mergeResult, err := merger.Merge(ctx, contracts.MergeRequest{
Source: doc,
LaneID: lane.ID,
ChunkArtifacts: chunkArtifacts,
ExtractOutputs: cloneExtractOutputs(extractOutputs),
SourceInput: sourceInput.Clone(),
SessionID: sessionID,
References: CloneReferenceSet(lane.MergeReferences.ReferenceSet),
LLMClient: input.LLMClient,
LLMProfile: lane.Merge.LLMProfile,
Options: cloneOptions(lane.Merge.Options),
Metadata: input.Metadata,
})
output.Warnings = append(output.Warnings, mergeResult.Warnings...)
if err != nil {
return fmt.Errorf("merge lane %q with merger %q: %w", lane.ID, merger.Key(), err)
return false, nil, fmt.Errorf("merge lane %q with merger %q: %w", lane.ID, merger.Key(), err)
}
mergeOutput := mergeResult.Output
mergeOutput.LaneID = lane.ID
mergeOutput.MergerKey = merger.Key()
mergeOutput.SourceID = doc.ID
mergeOutput.Payload.Warnings = append(mergeOutput.Payload.Warnings, mergeResult.Warnings...)
validationWarnings, rejection, err := r.validateRaw(ctx, rawValidationTarget{
stage: StageMerge,
laneID: lane.ID,
moduleKey: merger.Key(),
source: doc,
sourceID: doc.ID,
schema: mergeOutput.Schema,
payload: mergeOutput.Payload,
metadata: input.Metadata,
attempt: attempt,
})
if err != nil || rejection != nil {
return false, rejection, err
}
acceptedMerge = cloneMergeOutput(mergeOutput)
mergeWarnings = append(cloneWarnings(mergeResult.Warnings), validationWarnings...)
return true, nil, nil
})
if err != nil {
return err
}
if !mergeAccepted {
output.Rejected = append(output.Rejected, *mergeRejection)
return nil
}
output.Warnings = append(output.Warnings, mergeWarnings...)
var acceptedNormalize contracts.NormalizeOutput
var normalizeWarnings []contracts.Warning
normalizeAccepted, normalizeRejection, err := runWithRetry(ctx, lane.Normalize.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
normalizeResult, err := normalizer.Normalize(ctx, contracts.NormalizeRequest{
Source: doc,
LaneID: lane.ID,
Candidates: mergeResult.Candidates,
MergeOutput: cloneMergeOutput(acceptedMerge),
SourceInput: sourceInput.Clone(),
SessionID: sessionID,
References: CloneReferenceSet(lane.NormalizeReferences.ReferenceSet),
@@ -248,25 +329,41 @@ func (r *Runner) runLane(ctx context.Context, input RunInput, doc *source.Source
Options: cloneOptions(lane.Normalize.Options),
Metadata: input.Metadata,
})
output.Warnings = append(output.Warnings, normalizeResult.Warnings...)
if err != nil {
return fmt.Errorf("normalize lane %q with normalizer %q: %w", lane.ID, normalizer.Key(), err)
return false, nil, fmt.Errorf("normalize lane %q with normalizer %q: %w", lane.ID, normalizer.Key(), err)
}
if err := validateCandidateEnvelope(extractor, normalizeResult.Candidates); err != nil {
return fmt.Errorf("validate normalized candidates for lane %q: %w", lane.ID, err)
normalizeOutput := normalizeResult.Output
normalizeOutput.LaneID = lane.ID
normalizeOutput.NormalizerKey = normalizer.Key()
normalizeOutput.SourceID = doc.ID
normalizeOutput.Payload.Warnings = append(normalizeOutput.Payload.Warnings, normalizeResult.Warnings...)
validationWarnings, rejection, err := r.validateRaw(ctx, rawValidationTarget{
stage: StageNormalize,
laneID: lane.ID,
moduleKey: normalizer.Key(),
source: doc,
sourceID: doc.ID,
schema: normalizeOutput.Schema,
payload: normalizeOutput.Payload,
metadata: input.Metadata,
attempt: attempt,
})
if err != nil || rejection != nil {
return false, rejection, err
}
approved, rejected, warnings, err := runValidators(ctx, extractor.Key(), validators, doc, normalizeResult.Candidates, input.Metadata)
output.Warnings = append(output.Warnings, warnings...)
output.Rejected = append(output.Rejected, rejected...)
acceptedNormalize = cloneNormalizeOutput(normalizeOutput)
normalizeWarnings = append(cloneWarnings(normalizeResult.Warnings), validationWarnings...)
return true, nil, nil
})
if err != nil {
return err
}
for _, candidate := range approved {
output.Approved = append(output.Approved, artifacts.ArtifactFromCandidate(candidate))
if !normalizeAccepted {
output.Rejected = append(output.Rejected, *normalizeRejection)
return nil
}
output.Warnings = append(output.Warnings, normalizeWarnings...)
output.NormalizeOutputs = append(output.NormalizeOutputs, acceptedNormalize)
return nil
}
@@ -275,6 +372,146 @@ type validatorExecution struct {
binding ModuleBinding
}
type rawValidationTarget struct {
stage ModuleStage
laneID string
moduleKey string
source *source.SourceDocument
sourceID string
chunkID string
chunkIndex int
schema contracts.ResponseSchema
payload contracts.RawPayload
metadata map[string]any
attempt int
}
func runWithRetry(ctx context.Context, retries int, run func(attempt int) (bool, *contracts.RejectedOutput, error)) (bool, *contracts.RejectedOutput, error) {
attempts := 1
if retries > 0 {
attempts += retries
}
var lastRejection *contracts.RejectedOutput
for attempt := 1; attempt <= attempts; attempt++ {
if err := ctx.Err(); err != nil {
return false, nil, err
}
accepted, rejection, err := run(attempt)
if err != nil {
if ctxErr := ctx.Err(); ctxErr != nil {
return false, nil, ctxErr
}
if attempt == attempts {
return false, nil, fmt.Errorf("failed after %d attempt(s): %w", attempt, err)
}
continue
}
if accepted {
return true, nil, nil
}
if rejection != nil {
rejection.AttemptCount = attempt
lastRejection = rejection
}
if ctxErr := ctx.Err(); ctxErr != nil {
return false, nil, ctxErr
}
if attempt == attempts {
if lastRejection == nil {
lastRejection = &contracts.RejectedOutput{
ReasonCode: "raw_output_rejected",
Message: "raw output rejected",
AttemptCount: attempt,
}
}
return false, lastRejection, nil
}
}
return false, lastRejection, nil
}
func (r *Runner) validateChunksRaw(ctx context.Context, doc *source.SourceDocument, moduleKey string, chunks []contracts.SourceChunk, metadata map[string]any, attempt int) (*contracts.RejectedOutput, error) {
for _, chunk := range chunks {
_, rejection, err := r.validateRaw(ctx, rawValidationTarget{
stage: StageChunk,
moduleKey: moduleKey,
source: doc,
sourceID: doc.ID,
chunkID: chunk.ID,
chunkIndex: chunk.Index,
payload: contracts.RawPayload{
Content: append([]byte(nil), chunk.Content...),
MediaType: chunk.MediaType,
Metadata: cloneMetadata(chunk.Metadata),
},
metadata: metadata,
attempt: attempt,
})
if err != nil {
return nil, err
}
if rejection != nil {
return rejection, nil
}
}
return nil, nil
}
func (r *Runner) validateRaw(ctx context.Context, target rawValidationTarget) ([]contracts.Warning, *contracts.RejectedOutput, error) {
validators := r.registries.RawValidators.Validators(target.stage, target.moduleKey)
if len(validators) == 0 {
return nil, nil, nil
}
request := contracts.RawValidationRequest{
Stage: string(target.stage),
LaneID: target.laneID,
ModuleKey: target.moduleKey,
Source: target.source,
SourceID: target.sourceID,
ChunkID: target.chunkID,
ChunkIndex: target.chunkIndex,
Schema: target.schema,
Payload: cloneRawPayload(target.payload),
Metadata: cloneMetadata(target.metadata),
}
var warnings []contracts.Warning
for _, validator := range validators {
result, err := validator.ValidateRaw(ctx, request)
if err != nil {
return nil, nil, fmt.Errorf("validate raw %s output with validator %q: %w", target.stage, validator.Name(), err)
}
if !result.Approved {
reasonCode := strings.TrimSpace(result.ReasonCode)
if reasonCode == "" {
reasonCode = "raw_output_rejected"
}
message := strings.TrimSpace(result.Message)
if message == "" {
message = "raw output rejected"
}
return nil, &contracts.RejectedOutput{
Stage: string(target.stage),
LaneID: target.laneID,
ModuleKey: target.moduleKey,
ChunkID: target.chunkID,
ChunkIndex: target.chunkIndex,
ValidatorName: validator.Name(),
ReasonCode: reasonCode,
Message: message,
AttemptCount: target.attempt,
DiagnosticArtifactPath: result.DiagnosticArtifactPath,
}, nil
}
warnings = append(warnings, result.Warnings...)
}
return warnings, nil, nil
}
func (r *Runner) buildConfiguredValidators(lane ResolvedArtifactLane) ([]validatorExecution, error) {
validators := make([]validatorExecution, 0, len(lane.Validators))
for _, binding := range lane.Validators {
@@ -400,12 +637,64 @@ func manifestFromPipeline(input RunInput) artifacts.RunManifest {
func failOutput(output RunOutput) RunOutput {
if output.Manifest.PipelineID != "" {
populateRawOutputManifest(&output)
output.Manifest.ValidationStatus = "failed"
output.Manifest.CompletedAt = timePtr(time.Now().UTC())
}
return output
}
func populateRawOutputManifest(output *RunOutput) {
if output == nil {
return
}
output.Manifest.NormalizedOutputs = normalizedOutputManifests(output.NormalizeOutputs)
output.Manifest.RejectedOutputs = rejectedOutputManifests(output.Rejected)
}
func normalizedOutputManifests(outputs []contracts.NormalizeOutput) []artifacts.NormalizedOutputManifest {
if len(outputs) == 0 {
return nil
}
manifests := make([]artifacts.NormalizedOutputManifest, 0, len(outputs))
for _, output := range outputs {
manifests = append(manifests, artifacts.NormalizedOutputManifest{
LaneID: output.LaneID,
ModuleKey: output.NormalizerKey,
SourceID: output.SourceID,
MediaType: output.Payload.MediaType,
Schema: artifacts.OutputSchemaProvenance{
ID: output.Schema.ID,
Name: output.Schema.Name,
Version: output.Schema.Version,
},
})
}
return manifests
}
func rejectedOutputManifests(rejected []contracts.RejectedOutput) []artifacts.RejectedOutputManifest {
if len(rejected) == 0 {
return nil
}
manifests := make([]artifacts.RejectedOutputManifest, 0, len(rejected))
for _, output := range rejected {
manifests = append(manifests, artifacts.RejectedOutputManifest{
Stage: output.Stage,
LaneID: output.LaneID,
ModuleKey: output.ModuleKey,
ChunkID: output.ChunkID,
ChunkIndex: output.ChunkIndex,
ValidatorName: output.ValidatorName,
ReasonCode: output.ReasonCode,
Message: output.Message,
AttemptCount: output.AttemptCount,
DiagnosticArtifactPath: output.DiagnosticArtifactPath,
})
}
return manifests
}
func setLaneManifestMetadata(output *RunOutput, laneID string, modules ...any) {
if output == nil {
return
@@ -628,6 +917,59 @@ func cloneWarnings(warnings []contracts.Warning) []contracts.Warning {
return append([]contracts.Warning(nil), warnings...)
}
func cloneRawPayload(payload contracts.RawPayload) contracts.RawPayload {
return contracts.RawPayload{
Content: append([]byte(nil), payload.Content...),
MediaType: payload.MediaType,
Metadata: cloneMetadata(payload.Metadata),
Warnings: cloneWarnings(payload.Warnings),
}
}
func cloneExtractOutput(output contracts.ExtractOutput) contracts.ExtractOutput {
output.Payload = cloneRawPayload(output.Payload)
return output
}
func cloneExtractOutputs(outputs []contracts.ExtractOutput) []contracts.ExtractOutput {
if len(outputs) == 0 {
return nil
}
out := make([]contracts.ExtractOutput, 0, len(outputs))
for _, output := range outputs {
out = append(out, cloneExtractOutput(output))
}
return out
}
func cloneMergeOutput(output contracts.MergeOutput) contracts.MergeOutput {
output.Payload = cloneRawPayload(output.Payload)
return output
}
func cloneNormalizeOutput(output contracts.NormalizeOutput) contracts.NormalizeOutput {
output.Payload = cloneRawPayload(output.Payload)
return output
}
func cloneNormalizeOutputs(outputs []contracts.NormalizeOutput) []contracts.NormalizeOutput {
if len(outputs) == 0 {
return nil
}
out := make([]contracts.NormalizeOutput, 0, len(outputs))
for _, output := range outputs {
out = append(out, cloneNormalizeOutput(output))
}
return out
}
func cloneRejectedOutputs(rejected []contracts.RejectedOutput) []contracts.RejectedOutput {
if len(rejected) == 0 {
return nil
}
return append([]contracts.RejectedOutput(nil), rejected...)
}
func timePtr(t time.Time) *time.Time {
return &t
}
@@ -640,115 +982,3 @@ func pipelineUsesConfiguredValidators(pipeline ResolvedPipeline) bool {
}
return false
}
func normalizeCandidates(extractor contracts.Extractor, candidates []artifacts.ArtifactCandidate, nextIndex *int) ([]artifacts.ArtifactCandidate, error) {
normalized := make([]artifacts.ArtifactCandidate, 0, len(candidates))
for _, candidate := range candidates {
candidate.Index = *nextIndex
*nextIndex = *nextIndex + 1
if candidate.ExtractorKey == "" {
candidate.ExtractorKey = extractor.Key()
} else if candidate.ExtractorKey != extractor.Key() {
return nil, fmt.Errorf("candidate extractor_key %q does not match extractor %q", candidate.ExtractorKey, extractor.Key())
}
if candidate.ArtifactType == "" {
candidate.ArtifactType = extractor.ArtifactType()
} else if candidate.ArtifactType != extractor.ArtifactType() {
return nil, fmt.Errorf("candidate artifact_type %q does not match extractor %q artifact type %q", candidate.ArtifactType, extractor.Key(), extractor.ArtifactType())
}
if candidate.SchemaVersion == "" {
candidate.SchemaVersion = extractor.SchemaVersion()
} else if candidate.SchemaVersion != extractor.SchemaVersion() {
return nil, fmt.Errorf("candidate schema_version %q does not match extractor %q schema version %q", candidate.SchemaVersion, extractor.Key(), extractor.SchemaVersion())
}
normalized = append(normalized, candidate)
}
return normalized, nil
}
func validateCandidateEnvelope(extractor contracts.Extractor, candidates []artifacts.ArtifactCandidate) error {
seen := make(map[int]struct{}, len(candidates))
for _, candidate := range candidates {
if _, ok := seen[candidate.Index]; ok {
return fmt.Errorf("candidate index %d is duplicated", candidate.Index)
}
seen[candidate.Index] = struct{}{}
if candidate.ExtractorKey == "" {
return fmt.Errorf("candidate index %d extractor_key must not be empty", candidate.Index)
}
if candidate.ExtractorKey != extractor.Key() {
return fmt.Errorf("candidate index %d extractor_key %q does not match extractor %q", candidate.Index, candidate.ExtractorKey, extractor.Key())
}
if candidate.ArtifactType == "" {
return fmt.Errorf("candidate index %d artifact_type must not be empty", candidate.Index)
}
if candidate.ArtifactType != extractor.ArtifactType() {
return fmt.Errorf("candidate index %d artifact_type %q does not match extractor %q artifact type %q", candidate.Index, candidate.ArtifactType, extractor.Key(), extractor.ArtifactType())
}
if candidate.SchemaVersion == "" {
return fmt.Errorf("candidate index %d schema_version must not be empty", candidate.Index)
}
if candidate.SchemaVersion != extractor.SchemaVersion() {
return fmt.Errorf("candidate index %d schema_version %q does not match extractor %q schema version %q", candidate.Index, candidate.SchemaVersion, extractor.Key(), extractor.SchemaVersion())
}
}
return nil
}
func runValidators(ctx context.Context, extractorKey string, validators []validatorExecution, doc *source.SourceDocument, candidates []artifacts.ArtifactCandidate, metadata map[string]any) ([]artifacts.ArtifactCandidate, []artifacts.RejectedArtifact, []contracts.Warning, error) {
eligible := candidates
var rejected []artifacts.RejectedArtifact
var warnings []contracts.Warning
for validatorIndex, execution := range validators {
validator := execution.validator
if validator == nil {
return nil, rejected, warnings, fmt.Errorf("extractor %q validator[%d] must not be nil", extractorKey, validatorIndex)
}
result, err := validator.Validate(ctx, contracts.ValidationRequest{
Source: doc,
Candidates: eligible,
LLMProfile: execution.binding.LLMProfile,
Options: cloneOptions(execution.binding.Options),
Metadata: metadata,
})
warnings = append(warnings, result.Warnings...)
if err != nil {
return nil, rejected, warnings, fmt.Errorf("validate extractor %q with validator %q: %w", extractorKey, validator.Name(), err)
}
if result.ValidatorName != validator.Name() {
return nil, rejected, warnings, fmt.Errorf("validator %q returned result for %q", validator.Name(), result.ValidatorName)
}
if err := validate.EnforceDecisionCardinality(eligible, result.Decisions); err != nil {
return nil, rejected, warnings, fmt.Errorf("validate extractor %q with validator %q: %w", extractorKey, validator.Name(), err)
}
decisions := make(map[int]contracts.ValidationDecision, len(result.Decisions))
for _, decision := range result.Decisions {
decisions[decision.CandidateIndex] = decision
}
nextEligible := make([]artifacts.ArtifactCandidate, 0, len(eligible))
for _, candidate := range eligible {
decision := decisions[candidate.Index]
if decision.Approved {
nextEligible = append(nextEligible, candidate)
continue
}
rejected = append(rejected, artifacts.RejectedArtifact{
Candidate: candidate,
ValidatorName: result.ValidatorName,
ReasonCode: decision.ReasonCode,
Message: decision.Message,
})
}
eligible = nextEligible
}
return eligible, rejected, warnings, nil
}

816
internal/framework/pipeline/runner_test.go
View File

File diff suppressed because it is too large Load Diff

6
internal/framework/pipeline/testdata/walking_skeleton_input.json vendored
View File

@@ -2,15 +2,15 @@
"id": "fixture-source",
"units": [
{
"id": "u1",
"id": 1,
"text": "First event."
},
{
"id": "u2",
"id": 2,
"text": "Second event."
},
{
"id": "u3",
"id": 3,
"text": "Third event."
}
]

39
internal/framework/pipeline/testdata/walking_skeleton_output.json vendored
View File

@@ -1,7 +1,7 @@
{
"manifest": {
"pipeline_id": "walking-skeleton",
"pipeline_digest": "sha256:437d7ff486c336ddba38654ea00b7ef9dd6e49ce09f468698202ad8fc459bdcd",
"pipeline_digest": "sha256:75c1f6d64d86666734ccf175cdb82ae90e0ba5be3194be952850d7c00c66e615",
"validation_status": "approved",
"artifact_lanes": [
{
@@ -12,40 +12,31 @@
}
]
},
"approved": [
"normalize_outputs": [
{
"lane_id": "events",
"normalizer_key": "noop",
"source_id": "fixture-source",
"schema": {
"id": "fake_event",
"name": "fake_event",
"version": "v1"
},
"media_type": "application/json",
"content": {
"outputs": [
{
"extractor_key": "fake/extract",
"artifact_type": "fake_event",
"schema_version": "v1",
"payload": {
"chunk_id": "fixture-source:chunk:0",
"llm_call": 1,
"text": "First event. Second event."
},
"source_refs": [
{
"source_id": "fixture-source",
"start_unit_id": "u1",
"end_unit_id": "u2"
}
]
},
{
"extractor_key": "fake/extract",
"artifact_type": "fake_event",
"schema_version": "v1",
"payload": {
"chunk_id": "fixture-source:chunk:1",
"llm_call": 2,
"text": "Third event."
},
"source_refs": [
{
"source_id": "fixture-source",
"start_unit_id": "u3",
"end_unit_id": "u3"
}
]
}
}
]
}

91
internal/framework/pipeline/walking_skeleton_test.go
View File

@@ -185,7 +185,7 @@ func (input walkingSkeletonInput) Parse(ctx context.Context, req contracts.Parse
var fixture struct {
ID string `json:"id"`
Units []struct {
ID string `json:"id"`
ID int `json:"id"`
Text string `json:"text"`
} `json:"units"`
}
@@ -230,12 +230,20 @@ func (chunker walkingSkeletonChunker) Chunk(ctx context.Context, req contracts.C
ID: req.Source.ID + ":chunk:0",
SourceID: req.Source.ID,
Index: 0,
StartUnitID: req.Source.Units[0].ID,
EndUnitID: req.Source.Units[1].ID,
Content: []byte(`{"units":[1,2]}`),
MediaType: "application/json",
Units: append([]source.SourceUnit(nil), req.Source.Units[:2]...),
},
{
ID: req.Source.ID + ":chunk:1",
SourceID: req.Source.ID,
Index: 1,
StartUnitID: req.Source.Units[2].ID,
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
Content: []byte(`{"units":[3]}`),
MediaType: "application/json",
Units: append([]source.SourceUnit(nil), req.Source.Units[2:]...),
},
},
@@ -248,22 +256,10 @@ func (extractor walkingSkeletonExtractor) Key() string {
return "fake/extract"
}
func (extractor walkingSkeletonExtractor) ArtifactType() string {
return "fake_event"
}
func (extractor walkingSkeletonExtractor) SchemaVersion() string {
return "v1"
}
func (extractor walkingSkeletonExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (extractor walkingSkeletonExtractor) Validators() []contracts.Validator {
return nil
}
func (extractor walkingSkeletonExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
var response struct {
Call int `json:"call"`
@@ -286,16 +282,11 @@ func (extractor walkingSkeletonExtractor) Extract(ctx context.Context, req contr
}
return contracts.ExtractionResult{
Candidates: []artifacts.ArtifactCandidate{
{
Payload: payload,
SourceRefs: []source.SourceRef{
{
SourceID: req.Source.ID,
StartUnitID: req.Chunk.Units[0].ID,
EndUnitID: req.Chunk.Units[len(req.Chunk.Units)-1].ID,
},
},
Output: contracts.ExtractOutput{
Schema: contracts.ResponseSchema{ID: "fake_event", Name: "fake_event", Version: "v1"},
Payload: contracts.RawPayload{
Content: payload,
MediaType: "application/json",
},
},
}, nil
@@ -328,11 +319,25 @@ func (merger walkingSkeletonMerger) Key() string {
}
func (merger walkingSkeletonMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
var candidates []artifacts.ArtifactCandidate
for _, chunkArtifacts := range req.ChunkArtifacts {
candidates = append(candidates, chunkArtifacts.Candidates...)
outputs := make([]json.RawMessage, 0, len(req.ExtractOutputs))
for _, output := range req.ExtractOutputs {
outputs = append(outputs, json.RawMessage(output.Payload.Content))
}
return contracts.MergeResult{Candidates: candidates}, nil
content, err := json.Marshal(map[string]any{"outputs": outputs})
if err != nil {
return contracts.MergeResult{}, err
}
return contracts.MergeResult{
Output: contracts.MergeOutput{
LaneID: req.LaneID,
SourceID: req.Source.ID,
Schema: contracts.ResponseSchema{ID: "fake_event", Name: "fake_event", Version: "v1"},
Payload: contracts.RawPayload{
Content: content,
MediaType: "application/json",
},
},
}, nil
}
type walkingSkeletonNormalizer struct{}
@@ -356,7 +361,14 @@ func (normalizer walkingSkeletonNormalizer) Normalize(ctx context.Context, req c
}, &response); err != nil {
return contracts.NormalizeResult{}, err
}
return contracts.NormalizeResult{Candidates: req.Candidates}, nil
return contracts.NormalizeResult{
Output: contracts.NormalizeOutput{
LaneID: req.LaneID,
SourceID: req.MergeOutput.SourceID,
Schema: req.MergeOutput.Schema,
Payload: req.MergeOutput.Payload,
},
}, nil
}
type walkingSkeletonOutput struct{}
@@ -366,9 +378,28 @@ func (output walkingSkeletonOutput) Key() string {
}
func (output walkingSkeletonOutput) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
type rawOutput struct {
LaneID string `json:"lane_id"`
NormalizerKey string `json:"normalizer_key"`
SourceID string `json:"source_id"`
Schema contracts.ResponseSchema `json:"schema"`
MediaType string `json:"media_type"`
Content json.RawMessage `json:"content"`
}
rawOutputs := make([]rawOutput, 0, len(req.NormalizeOutputs))
for _, output := range req.NormalizeOutputs {
rawOutputs = append(rawOutputs, rawOutput{
LaneID: output.LaneID,
NormalizerKey: output.NormalizerKey,
SourceID: output.SourceID,
Schema: output.Schema,
MediaType: output.Payload.MediaType,
Content: json.RawMessage(output.Payload.Content),
})
}
encoded, err := json.Marshal(struct {
Manifest artifacts.RunManifest `json:"manifest"`
Approved []artifacts.Artifact `json:"approved"`
NormalizeOutputs []rawOutput `json:"normalize_outputs"`
}{
Manifest: artifacts.RunManifest{
PipelineID: req.Manifest.PipelineID,
@@ -376,7 +407,7 @@ func (output walkingSkeletonOutput) Encode(ctx context.Context, req contracts.Ou
ArtifactLanes: req.Manifest.ArtifactLanes,
ValidationStatus: req.Manifest.ValidationStatus,
},
Approved: req.Approved,
NormalizeOutputs: rawOutputs,
})
if err != nil {
return contracts.OutputResult{}, err

3
internal/modules/chunk/dnd/scenes/assets/prompts/instructions.md
View File

@@ -22,8 +22,7 @@ dnd/scenes boundary policy:
- return sequential scenes with no gaps;
- do not overlap scenes;
- preserve source-unit order;
- use 1-based integer source-unit numbers from the transcript, where 1 is the
first provided source unit;
- use integer source-unit IDs from the transcript;
- each scene must have start_unit_id and end_unit_id;
- do not include final chunk IDs or chunk indexes.

42
internal/modules/chunk/dnd/scenes/chunker.go
View File

@@ -2,6 +2,7 @@ package scenes
import (
"context"
"encoding/json"
"fmt"
"strings"
@@ -142,7 +143,7 @@ func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]c
return nil, fmt.Errorf("scenes must not be empty")
}
unitIndexes := make(map[string]int, len(doc.Units))
unitIndexes := make(map[int]int, len(doc.Units))
for i, unit := range doc.Units {
unitIndexes[unit.ID] = i
}
@@ -157,18 +158,18 @@ func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]c
startIndex, ok := unitIndexes[normalized.StartUnitID]
if !ok {
return nil, fmt.Errorf("scene[%d] start_unit_id %q was not found", i, normalized.StartUnitID)
return nil, fmt.Errorf("scene[%d] start_unit_id %d was not found", i, normalized.StartUnitID)
}
endIndex, ok := unitIndexes[normalized.EndUnitID]
if !ok {
return nil, fmt.Errorf("scene[%d] end_unit_id %q was not found", i, normalized.EndUnitID)
return nil, fmt.Errorf("scene[%d] end_unit_id %d was not found", i, normalized.EndUnitID)
}
if startIndex > endIndex {
return nil, fmt.Errorf("scene[%d] start_unit_id %q appears after end_unit_id %q", i, normalized.StartUnitID, normalized.EndUnitID)
return nil, fmt.Errorf("scene[%d] start_unit_id %d appears after end_unit_id %d", i, normalized.StartUnitID, normalized.EndUnitID)
}
if i == 0 && startIndex != 0 {
return nil, fmt.Errorf("first scene must start at first source unit %q", doc.Units[0].ID)
return nil, fmt.Errorf("first scene must start at first source unit %d", doc.Units[0].ID)
}
if i > 0 {
if startIndex <= previousEnd {
@@ -181,10 +182,18 @@ func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]c
previousEnd = endIndex
units := cloneUnits(doc.Units[startIndex : endIndex+1])
content, err := chunkContent(units)
if err != nil {
return nil, err
}
chunks = append(chunks, contracts.SourceChunk{
ID: fmt.Sprintf("scene-%06d", i+1),
SourceID: doc.ID,
Index: i,
StartUnitID: units[0].ID,
EndUnitID: units[len(units)-1].ID,
Content: content,
MediaType: "application/json",
Units: units,
Metadata: map[string]any{
"scene_title": normalized.ShortTitle,
@@ -201,11 +210,23 @@ func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]c
}
if previousEnd != len(doc.Units)-1 {
return nil, fmt.Errorf("final scene must end at final source unit %q", doc.Units[len(doc.Units)-1].ID)
return nil, fmt.Errorf("final scene must end at final source unit %d", doc.Units[len(doc.Units)-1].ID)
}
return chunks, nil
}
func chunkContent(units []source.SourceUnit) ([]byte, error) {
content, err := json.Marshal(struct {
Units []source.SourceUnit `json:"units"`
}{
Units: units,
})
if err != nil {
return nil, fmt.Errorf("encode chunk content: %w", err)
}
return content, nil
}
func normalizeScene(doc *source.SourceDocument, index int, scene sceneResponse) (normalizedScene, error) {
startUnitID, err := dnd.ResolveUnitID(doc, "start_unit_id", scene.StartUnitID)
if err != nil {
@@ -226,9 +247,16 @@ func normalizeScene(doc *source.SourceDocument, index int, scene sceneResponse)
BoundaryConfidence: strings.TrimSpace(scene.BoundaryConfidence),
}
required := map[string]string{
requiredInts := map[string]int{
"start_unit_id": out.StartUnitID,
"end_unit_id": out.EndUnitID,
}
for field, value := range requiredInts {
if value <= 0 {
return normalizedScene{}, fmt.Errorf("scene[%d] %s must be positive", index, field)
}
}
required := map[string]string{
"short_title": out.ShortTitle,
"primary_mode": out.PrimaryMode,
"summary": out.Summary,

64
internal/modules/chunk/dnd/scenes/chunker_test.go
View File

@@ -170,8 +170,8 @@ func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
if got := chunkIDs(result.Chunks); !reflect.DeepEqual(got, []string{"scene-000001", "scene-000002"}) {
t.Fatalf("chunk IDs = %#v, want deterministic scene IDs", got)
}
gotUnits := [][]string{unitIDs(result.Chunks[0].Units), unitIDs(result.Chunks[1].Units)}
wantUnits := [][]string{{"seg-001", "seg-002"}, {"seg-003", "seg-004"}}
gotUnits := [][]int{unitIDs(result.Chunks[0].Units), unitIDs(result.Chunks[1].Units)}
wantUnits := [][]int{{1, 2}, {3, 4}}
if !reflect.DeepEqual(gotUnits, wantUnits) {
t.Fatalf("chunk units = %#v, want %#v", gotUnits, wantUnits)
}
@@ -179,13 +179,19 @@ func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
if first.SourceID != "session-alpha" || first.Index != 0 {
t.Fatalf("first chunk = %#v, want source and index fields", first)
}
if first.StartUnitID != 1 || first.EndUnitID != 2 {
t.Fatalf("first boundaries = %d-%d, want 1-2", first.StartUnitID, first.EndUnitID)
}
if first.MediaType != "application/json" || len(first.Content) == 0 {
t.Fatalf("first payload = media type %q length %d, want JSON content", first.MediaType, len(first.Content))
}
if first.Metadata["scene_title"] != "Goblin parley" ||
first.Metadata["primary_mode"] != "Discussion" ||
first.Metadata["summary"] != "The party negotiates with a scout." ||
first.Metadata["boundary_note"] != "The scene covers the discussion before fighting starts." ||
first.Metadata["boundary_confidence"] != "High" ||
first.Metadata["start_unit_id"] != "seg-001" ||
first.Metadata["end_unit_id"] != "seg-002" ||
first.Metadata["start_unit_id"] != 1 ||
first.Metadata["end_unit_id"] != 2 ||
first.Metadata["unit_count"] != 2 {
t.Fatalf("first metadata = %#v, want scene metadata", first.Metadata)
}
@@ -311,11 +317,11 @@ func TestChunkDefensivelyCopiesSourceUnitsAndMetadata(t *testing.T) {
t.Fatalf("Chunk() error = %v, want nil", err)
}
doc.Units[0].ID = "mutated"
doc.Units[0].ID = 99
doc.Units[0].Metadata["speaker"] = "mutated"
client.response.Scenes[0].MainParticipants[0] = "mutated"
if result.Chunks[0].Units[0].ID != "seg-001" {
if result.Chunks[0].Units[0].ID != 1 {
t.Fatalf("chunk unit ID changed after source mutation: %#v", result.Chunks[0].Units[0])
}
if result.Chunks[0].Units[0].Metadata["speaker"] != "Alice" {
@@ -362,7 +368,7 @@ func TestChunkRejectsInvalidRequests(t *testing.T) {
canceledCtx, cancel := context.WithCancel(context.Background())
cancel()
invalidDoc := sceneSourceDocument()
invalidDoc.Units[0].ID = ""
invalidDoc.Units[0].ID = 0
emptyDoc := sceneSourceDocument()
emptyDoc.Units = nil
@@ -407,37 +413,37 @@ func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
{
name: "unknown boundary id",
response: replaceScenes(validSceneResponse(), []sceneResponse{
scene("seg-001", "seg-999"),
scene(1, 999),
}),
want: "was not found",
},
{
name: "out of order boundaries",
response: replaceScenes(validSceneResponse(), []sceneResponse{
scene("seg-003", "seg-002"),
scene(3, 2),
}),
want: "appears after",
},
{
name: "gap",
response: replaceScenes(validSceneResponse(), []sceneResponse{
scene("seg-001", "seg-001"),
scene("seg-003", "seg-004"),
scene(1, 1),
scene(3, 4),
}),
want: "gap",
},
{
name: "overlap",
response: replaceScenes(validSceneResponse(), []sceneResponse{
scene("seg-001", "seg-002"),
scene("seg-002", "seg-004"),
scene(1, 2),
scene(2, 4),
}),
want: "overlap",
},
{
name: "incomplete coverage",
response: replaceScenes(validSceneResponse(), []sceneResponse{
scene("seg-001", "seg-003"),
scene(1, 3),
}),
want: "final scene",
},
@@ -445,8 +451,8 @@ func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
name: "empty metadata field",
response: replaceScenes(validSceneResponse(), []sceneResponse{
{
StartUnitID: dnd.UnitRefFromString("seg-001"),
EndUnitID: dnd.UnitRefFromString("seg-004"),
StartUnitID: dnd.UnitRefFromInt(1),
EndUnitID: dnd.UnitRefFromInt(4),
ShortTitle: " ",
PrimaryMode: "Narrative",
MainParticipants: []string{"Aria"},
@@ -461,8 +467,8 @@ func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
name: "empty participant",
response: replaceScenes(validSceneResponse(), []sceneResponse{
{
StartUnitID: dnd.UnitRefFromString("seg-001"),
EndUnitID: dnd.UnitRefFromString("seg-004"),
StartUnitID: dnd.UnitRefFromInt(1),
EndUnitID: dnd.UnitRefFromInt(4),
ShortTitle: "Title",
PrimaryMode: "Narrative",
MainParticipants: []string{"Aria", " "},
@@ -511,7 +517,7 @@ func chunkRequestWithClient(client contracts.StructuredLLMClient) contracts.Chun
}
}
const sceneTranscriptJSON = `{"id":"session-alpha","segments":[{"id":"seg-001","text":"Aria asks whether the goblin will parley."}]}`
const sceneTranscriptJSON = `{"id":"session-alpha","segments":[{"id":1,"text":"Aria asks whether the goblin will parley."}]}`
func sceneSourceInput() contracts.LLMInputMaterial {
return contracts.NewLLMInputMaterial("source", "application/json", []byte(sceneTranscriptJSON), "sha256:transcript", "file:///session-alpha.json")
@@ -529,10 +535,10 @@ func sceneSourceDocument() *source.SourceDocument {
Format: "application/vnd.seriatim.minimal+json",
Digest: "sha256:source",
Units: []source.SourceUnit{
{ID: "seg-001", Kind: "transcript_segment", Text: "Aria asks whether the goblin will parley.", Metadata: map[string]any{"speaker": "Alice"}},
{ID: "seg-002", Kind: "transcript_segment", Text: "The goblin scout describes the gate guards."},
{ID: "seg-003", Kind: "transcript_segment", Text: "The guards rush out with blades drawn."},
{ID: "seg-004", Kind: "transcript_segment", Text: "The party defeats the ambushers."},
{ID: 1, Kind: "transcript_segment", Text: "Aria asks whether the goblin will parley.", Metadata: map[string]any{"speaker": "Alice"}},
{ID: 2, Kind: "transcript_segment", Text: "The goblin scout describes the gate guards."},
{ID: 3, Kind: "transcript_segment", Text: "The guards rush out with blades drawn."},
{ID: 4, Kind: "transcript_segment", Text: "The party defeats the ambushers."},
},
}
}
@@ -540,7 +546,7 @@ func sceneSourceDocument() *source.SourceDocument {
func validSceneResponse() chunkResponse {
return chunkResponse{
Scenes: []sceneResponse{
scene("seg-001", "seg-004"),
scene(1, 4),
},
BoundaryCaveats: []string{},
}
@@ -551,10 +557,10 @@ func replaceScenes(response chunkResponse, scenes []sceneResponse) chunkResponse
return response
}
func scene(startUnitID string, endUnitID string) sceneResponse {
func scene(startUnitID int, endUnitID int) sceneResponse {
return sceneResponse{
StartUnitID: dnd.UnitRefFromString(startUnitID),
EndUnitID: dnd.UnitRefFromString(endUnitID),
StartUnitID: dnd.UnitRefFromInt(startUnitID),
EndUnitID: dnd.UnitRefFromInt(endUnitID),
ShortTitle: "Scene title",
PrimaryMode: "Narrative",
MainParticipants: []string{"Aria"},
@@ -572,8 +578,8 @@ func chunkIDs(chunks []contracts.SourceChunk) []string {
return ids
}
func unitIDs(units []source.SourceUnit) []string {
ids := make([]string, 0, len(units))
func unitIDs(units []source.SourceUnit) []int {
ids := make([]int, 0, len(units))
for _, unit := range units {
ids = append(ids, unit.ID)
}

4
internal/modules/chunk/dnd/scenes/model.go
View File

@@ -19,8 +19,8 @@ type sceneResponse struct {
}
type normalizedScene struct {
StartUnitID string
EndUnitID string
StartUnitID int
EndUnitID int
ShortTitle string
PrimaryMode string
MainParticipants []string

20
internal/modules/chunk/generic/chunker.go
View File

@@ -68,10 +68,18 @@ func (c *Chunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contra
end = len(req.Source.Units)
}
units := cloneUnits(req.Source.Units[start:end])
content, err := chunkContent(units)
if err != nil {
return contracts.ChunkResult{}, err
}
chunks = append(chunks, contracts.SourceChunk{
ID: fmt.Sprintf("chunk-%06d", len(chunks)+1),
SourceID: req.Source.ID,
Index: len(chunks),
StartUnitID: units[0].ID,
EndUnitID: units[len(units)-1].ID,
Content: content,
MediaType: "application/json",
Units: units,
Metadata: map[string]any{
"start_unit_id": units[0].ID,
@@ -87,6 +95,18 @@ func (c *Chunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contra
return contracts.ChunkResult{Chunks: chunks}, nil
}
func chunkContent(units []source.SourceUnit) ([]byte, error) {
content, err := json.Marshal(struct {
Units []source.SourceUnit `json:"units"`
}{
Units: units,
})
if err != nil {
return nil, chunkerErrorf("encode chunk content: %w", err)
}
return content, nil
}
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,

31
internal/modules/chunk/generic/chunker_test.go
View File

@@ -59,10 +59,16 @@ func TestChunkUsesDefaultsForSingleChunk(t *testing.T) {
if chunk.Index != 0 || chunk.SourceID != "source-1" {
t.Fatalf("chunk = %#v, want source and index fields", chunk)
}
if got := unitIDs(chunk.Units); !reflect.DeepEqual(got, []string{"u001", "u002", "u003"}) {
if got := unitIDs(chunk.Units); !reflect.DeepEqual(got, []int{1, 2, 3}) {
t.Fatalf("unit IDs = %#v, want all units", got)
}
if chunk.Metadata["start_unit_id"] != "u001" || chunk.Metadata["end_unit_id"] != "u003" || chunk.Metadata["unit_count"] != 3 {
if chunk.StartUnitID != 1 || chunk.EndUnitID != 3 {
t.Fatalf("chunk boundaries = %d-%d, want 1-3", chunk.StartUnitID, chunk.EndUnitID)
}
if chunk.MediaType != "application/json" || len(chunk.Content) == 0 {
t.Fatalf("chunk payload = media type %q length %d, want JSON content", chunk.MediaType, len(chunk.Content))
}
if chunk.Metadata["start_unit_id"] != 1 || chunk.Metadata["end_unit_id"] != 3 || chunk.Metadata["unit_count"] != 3 {
t.Fatalf("metadata = %#v, want chunk bounds", chunk.Metadata)
}
}
@@ -79,8 +85,8 @@ func TestChunkExactBoundaries(t *testing.T) {
if got := chunkIDs(result.Chunks); !reflect.DeepEqual(got, []string{"chunk-000001", "chunk-000002", "chunk-000003"}) {
t.Fatalf("chunk IDs = %#v, want stable IDs", got)
}
gotUnits := [][]string{unitIDs(result.Chunks[0].Units), unitIDs(result.Chunks[1].Units), unitIDs(result.Chunks[2].Units)}
wantUnits := [][]string{{"u001", "u002"}, {"u003", "u004"}, {"u005", "u006"}}
gotUnits := [][]int{unitIDs(result.Chunks[0].Units), unitIDs(result.Chunks[1].Units), unitIDs(result.Chunks[2].Units)}
wantUnits := [][]int{{1, 2}, {3, 4}, {5, 6}}
if !reflect.DeepEqual(gotUnits, wantUnits) {
t.Fatalf("chunk units = %#v, want %#v", gotUnits, wantUnits)
}
@@ -95,11 +101,11 @@ func TestChunkOverlap(t *testing.T) {
t.Fatalf("Chunk() error = %v, want nil", err)
}
gotUnits := make([][]string, 0, len(result.Chunks))
gotUnits := make([][]int, 0, len(result.Chunks))
for _, chunk := range result.Chunks {
gotUnits = append(gotUnits, unitIDs(chunk.Units))
}
wantUnits := [][]string{{"u001", "u002", "u003"}, {"u003", "u004", "u005"}, {"u005", "u006", "u007"}}
wantUnits := [][]int{{1, 2, 3}, {3, 4, 5}, {5, 6, 7}}
if !reflect.DeepEqual(gotUnits, wantUnits) {
t.Fatalf("chunk units = %#v, want %#v", gotUnits, wantUnits)
}
@@ -162,10 +168,10 @@ func TestChunkDefensivelyCopiesUnits(t *testing.T) {
t.Fatalf("len(Chunks) = %d, want 2", len(result.Chunks))
}
doc.Units[0].ID = "changed"
doc.Units[0].ID = 99
doc.Units[0].Metadata["speaker"] = "changed"
if result.Chunks[0].Units[0].ID != "u001" {
if result.Chunks[0].Units[0].ID != 1 {
t.Fatalf("chunk unit ID changed after source mutation: %#v", result.Chunks[0].Units[0])
}
if result.Chunks[0].Units[0].Metadata["speaker"] != "speaker-001" {
@@ -176,11 +182,10 @@ func TestChunkDefensivelyCopiesUnits(t *testing.T) {
func testSource(count int) *source.SourceDocument {
units := make([]source.SourceUnit, 0, count)
for i := 1; i <= count; i++ {
id := "u" + zeroPad3(i)
units = append(units, source.SourceUnit{
ID: id,
ID: i,
Kind: "unit",
Text: "Text for " + id,
Text: "Text for " + zeroPad3(i),
Metadata: map[string]any{
"speaker": "speaker-" + zeroPad3(i),
},
@@ -207,8 +212,8 @@ func chunkIDs(chunks []contracts.SourceChunk) []string {
return ids
}
func unitIDs(units []source.SourceUnit) []string {
ids := make([]string, 0, len(units))
func unitIDs(units []source.SourceUnit) []int {
ids := make([]int, 0, len(units))
for _, unit := range units {
ids = append(ids, unit.ID)
}

3
internal/modules/extract/dnd/spells/assets/prompts/instructions.md
View File

@@ -1,5 +1,4 @@
Source references must use 1-based integer source-unit numbers from the
transcript, where 1 is the first provided source unit.
Source references must use integer source-unit IDs from the transcript.
Return only D&D spell-cast artifacts. For each spell cast, identify the in-world
caster, spell name, effect, narrative description, and source references using

4
internal/modules/extract/dnd/spells/config_test.go
View File

@@ -247,6 +247,10 @@ func (dndSpellsChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (
ID: req.Source.ID + ":chunk:0",
SourceID: req.Source.ID,
Index: 0,
StartUnitID: req.Source.Units[0].ID,
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
Content: []byte(`{"units":[1,2,3]}`),
MediaType: "application/json",
Units: append([]source.SourceUnit(nil), req.Source.Units...),
},
},

79
internal/modules/extract/dnd/spells/extractor.go
View File

@@ -6,8 +6,6 @@ import (
"fmt"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/sharedassets/dnd"
@@ -45,14 +43,6 @@ func (e *Extractor) Key() string {
return Key
}
func (e *Extractor) ArtifactType() string {
return ArtifactType
}
func (e *Extractor) SchemaVersion() string {
return SchemaVersion
}
func (e *Extractor) ReferenceSlots() []contracts.ReferenceSlot {
return dnd.ReferenceSlots(referenceSlotDescriptions)
}
@@ -77,13 +67,6 @@ func (e *Extractor) ManifestMetadata() map[string]any {
return metadata
}
func (e *Extractor) Validators() []contracts.Validator {
return []contracts.Validator{
ShapeValidator{},
SourceRefValidator{},
}
}
func (e *Extractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
if e == nil {
return contracts.ExtractionResult{}, extractorErrorf("extractor must not be nil")
@@ -108,35 +91,41 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.ExtractionRequest
}
var response extractionResponse
if _, err := req.LLMClient.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
completion, err := req.LLMClient.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
StageName: Key,
PromptID: PromptID,
PromptVersion: SchemaVersion,
ProfileID: req.LLMProfile,
SessionID: req.SessionID,
Inputs: dnd.PromptInputs(req.SourceInput, req.References),
}, &response); err != nil {
}, &response)
if err != nil {
return contracts.ExtractionResult{}, extractorErrorf("complete structured output: %w", err)
}
if response.SpellCasts == nil {
return contracts.ExtractionResult{}, extractorErrorf("malformed structured output: spell_casts must be present")
}
if len(response.SpellCasts) == 0 {
return contracts.ExtractionResult{}, nil
}
candidates := make([]artifacts.ArtifactCandidate, 0, len(response.SpellCasts))
for i, spellCast := range response.SpellCasts {
payload, err := spellCastPayload(spellCast)
content := append([]byte(nil), completion.Content...)
if len(strings.TrimSpace(string(content))) == 0 {
var err error
content, err = json.Marshal(response)
if err != nil {
return contracts.ExtractionResult{}, extractorErrorf("marshal spell cast[%d]: %w", i, err)
return contracts.ExtractionResult{}, extractorErrorf("marshal raw output: %w", err)
}
candidates = append(candidates, artifacts.ArtifactCandidate{
Payload: payload,
SourceRefs: sourceRefCandidates(req.Source, spellCast.SourceRefs),
})
}
return contracts.ExtractionResult{Candidates: candidates}, nil
return contracts.ExtractionResult{
Output: contracts.ExtractOutput{
Schema: contracts.ResponseSchema{
ID: ResponseSchemaID,
Name: ResponseSchemaName,
Version: SchemaVersion,
},
Payload: contracts.RawPayload{
Content: content,
MediaType: "application/json",
Metadata: map[string]any{
"spell_cast_count": len(response.SpellCasts),
},
},
},
}, nil
}
func ModuleSpec() pipeline.ModuleSpec {
@@ -155,26 +144,6 @@ func Register(registry *pipeline.ExtractorRegistry) error {
})
}
func spellCastPayload(spellCast spellCastResponse) (json.RawMessage, error) {
return json.Marshal(SpellCast{
Caster: strings.TrimSpace(spellCast.Caster),
Spell: strings.TrimSpace(spellCast.Spell),
Effect: strings.TrimSpace(spellCast.Effect),
NarrativeDescription: strings.TrimSpace(spellCast.NarrativeDescription),
})
}
func sourceRefCandidates(doc *source.SourceDocument, refs []dnd.SourceRefResponse) []source.SourceRef {
if len(refs) == 0 {
return nil
}
out := make([]source.SourceRef, 0, len(refs))
for _, ref := range refs {
out = append(out, dnd.SourceRefCandidate(doc, ref))
}
return out
}
func extractorErrorf(format string, args ...any) error {
return fmt.Errorf("dnd spells extractor: "+format, args...)
}

101
internal/modules/extract/dnd/spells/extractor_test.go
View File

@@ -7,12 +7,11 @@ import (
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/modules/sharedassets/dnd"
)
func TestExtractReturnsSpellCandidateFromStructuredOutput(t *testing.T) {
func TestExtractReturnsRawOutputFromStructuredResponse(t *testing.T) {
client := &fakeSpellsLLMClient{
response: extractionResponse{
SpellCasts: []spellCastResponse{
@@ -25,6 +24,7 @@ func TestExtractReturnsSpellCandidateFromStructuredOutput(t *testing.T) {
},
},
},
content: []byte(`{"spell_casts":[{"caster":" Aria ","spell":" Cure Wounds ","effect":" Heals an injured ally. ","narrative_description":" Aria restores the fighter after the fight. ","source_refs":[{"source_id":"session-alpha","start_unit_id":1,"end_unit_id":2}]}],"raw_marker":true}`),
}
result, err := New().Extract(context.Background(), extractionRequestWithClient(client))
@@ -53,29 +53,22 @@ func TestExtractReturnsSpellCandidateFromStructuredOutput(t *testing.T) {
t.Fatalf("transcript content = %q, want original source input", got)
}
if len(result.Candidates) != 1 {
t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
if result.Output.Payload.MediaType != "application/json" {
t.Fatalf("MediaType = %q, want application/json", result.Output.Payload.MediaType)
}
candidate := result.Candidates[0]
if candidate.Index != 0 || candidate.ExtractorKey != "" || candidate.ArtifactType != "" || candidate.SchemaVersion != "" {
t.Fatalf("candidate envelope fields = %#v, want runner-normalized zero values", candidate)
if result.Output.Schema.ID != ResponseSchemaID || result.Output.Schema.Name != ResponseSchemaName || result.Output.Schema.Version != SchemaVersion {
t.Fatalf("schema = %#v, want response schema provenance", result.Output.Schema)
}
var payload SpellCast
if err := json.Unmarshal(candidate.Payload, &payload); err != nil {
t.Fatalf("Unmarshal(Payload) error = %v, want nil", err)
if got := string(result.Output.Payload.Content); got != string(client.content) {
t.Fatalf("content = %q, want exact raw completion content", got)
}
wantPayload := SpellCast{
Caster: "Aria",
Spell: "Cure Wounds",
Effect: "Heals an injured ally.",
NarrativeDescription: "Aria restores the fighter after the fight.",
var payload extractionResponse
if err := json.Unmarshal(result.Output.Payload.Content, &payload); err != nil {
t.Fatalf("Unmarshal(Content) error = %v, want nil", err)
}
if payload != wantPayload {
t.Fatalf("payload = %#v, want %#v", payload, wantPayload)
}
wantRef := source.SourceRef{SourceID: "session-alpha", StartUnitID: "seg-001", EndUnitID: "seg-002"}
if len(candidate.SourceRefs) != 1 || candidate.SourceRefs[0] != wantRef {
t.Fatalf("SourceRefs = %#v, want %#v", candidate.SourceRefs, []source.SourceRef{wantRef})
if len(payload.SpellCasts) != 1 || payload.SpellCasts[0].Spell != " Cure Wounds " {
t.Fatalf("payload = %#v, want raw structured response", payload)
}
}
@@ -174,27 +167,31 @@ func TestPromptInputsMapLegacyRosterReferenceToParty(t *testing.T) {
}
}
func TestExtractReturnsNoCandidatesForEmptyResponse(t *testing.T) {
func TestExtractReturnsRawOutputForEmptyResponse(t *testing.T) {
client := &fakeSpellsLLMClient{response: extractionResponse{SpellCasts: []spellCastResponse{}}}
result, err := New().Extract(context.Background(), extractionRequestWithClient(client))
if err != nil {
t.Fatalf("Extract() error = %v, want nil", err)
}
if len(result.Candidates) != 0 {
t.Fatalf("Candidates = %#v, want none", result.Candidates)
var payload extractionResponse
if err := json.Unmarshal(result.Output.Payload.Content, &payload); err != nil {
t.Fatalf("Unmarshal(Content) error = %v, want nil", err)
}
if len(payload.SpellCasts) != 0 {
t.Fatalf("SpellCasts = %#v, want none", payload.SpellCasts)
}
}
func TestExtractRejectsMissingSpellCasts(t *testing.T) {
func TestExtractCarriesMalformedStructuredContentAsRawOutput(t *testing.T) {
client := &fakeSpellsLLMClient{response: extractionResponse{}}
_, err := New().Extract(context.Background(), extractionRequestWithClient(client))
if err == nil {
t.Fatal("Extract() error = nil, want malformed output error")
result, err := New().Extract(context.Background(), extractionRequestWithClient(client))
if err != nil {
t.Fatalf("Extract() error = %v, want nil", err)
}
if !strings.Contains(err.Error(), "dnd spells") || !strings.Contains(err.Error(), "spell_casts") {
t.Fatalf("Extract() error = %q, want spell_casts context", err.Error())
if string(result.Output.Payload.Content) != `{"spell_casts":null}` {
t.Fatalf("content = %s, want raw structured output", result.Output.Payload.Content)
}
}
@@ -254,14 +251,14 @@ func TestExtractPreservesResponseOrder(t *testing.T) {
Spell: "Cure Wounds",
Effect: "Heals.",
NarrativeDescription: "First spell.",
SourceRefs: responseSourceRefs("session-alpha", "seg-001", "seg-001"),
SourceRefs: responseSourceRefs("session-alpha", 1, 1),
},
{
Caster: "Bandit Shaman",
Spell: "Fire Bolt",
Effect: "Burns.",
NarrativeDescription: "Second spell.",
SourceRefs: responseSourceRefs("session-alpha", "seg-002", "seg-002"),
SourceRefs: responseSourceRefs("session-alpha", 2, 2),
},
},
},
@@ -271,23 +268,16 @@ func TestExtractPreservesResponseOrder(t *testing.T) {
if err != nil {
t.Fatalf("Extract() error = %v, want nil", err)
}
if len(result.Candidates) != 2 {
t.Fatalf("len(Candidates) = %d, want 2", len(result.Candidates))
var payload extractionResponse
if err := json.Unmarshal(result.Output.Payload.Content, &payload); err != nil {
t.Fatalf("Unmarshal(Content) error = %v, want nil", err)
}
var first, second SpellCast
if err := json.Unmarshal(result.Candidates[0].Payload, &first); err != nil {
t.Fatalf("Unmarshal(first) error = %v, want nil", err)
}
if err := json.Unmarshal(result.Candidates[1].Payload, &second); err != nil {
t.Fatalf("Unmarshal(second) error = %v, want nil", err)
}
if first.Spell != "Cure Wounds" || second.Spell != "Fire Bolt" {
t.Fatalf("candidate order = %q, %q; want response order", first.Spell, second.Spell)
if len(payload.SpellCasts) != 2 || payload.SpellCasts[0].Spell != "Cure Wounds" || payload.SpellCasts[1].Spell != "Fire Bolt" {
t.Fatalf("spell order = %#v, want response order", payload.SpellCasts)
}
}
func TestExtractCopiesCandidateSourceRefs(t *testing.T) {
func TestExtractDefensivelyCopiesRawContent(t *testing.T) {
client := &fakeSpellsLLMClient{
response: extractionResponse{
SpellCasts: []spellCastResponse{
@@ -296,7 +286,7 @@ func TestExtractCopiesCandidateSourceRefs(t *testing.T) {
Spell: "Cure Wounds",
Effect: "Heals.",
NarrativeDescription: "Aria heals.",
SourceRefs: responseSourceRefs("session-alpha", "seg-001", "seg-002"),
SourceRefs: responseSourceRefs("session-alpha", 1, 2),
},
},
},
@@ -306,10 +296,14 @@ func TestExtractCopiesCandidateSourceRefs(t *testing.T) {
if err != nil {
t.Fatalf("Extract() error = %v, want nil", err)
}
client.response.SpellCasts[0].SourceRefs[0].StartUnitID = dnd.UnitRefFromString("mutated")
client.response.SpellCasts[0].SourceRefs[0].StartUnitID = dnd.UnitRefFromInt(99)
if got := result.Candidates[0].SourceRefs[0].StartUnitID; got != "seg-001" {
t.Fatalf("candidate source ref start = %q, want copied seg-001", got)
var payload extractionResponse
if err := json.Unmarshal(result.Output.Payload.Content, &payload); err != nil {
t.Fatalf("Unmarshal(Content) error = %v, want nil", err)
}
if got := payload.SpellCasts[0].SourceRefs[0].StartUnitID.String(); got != "1" {
t.Fatalf("source ref start = %q, want copied 1", got)
}
}
@@ -322,7 +316,7 @@ func extractionRequestWithClient(client contracts.StructuredLLMClient) contracts
return req
}
const spellTranscriptJSON = `{"id":"session-alpha","segments":[{"id":"seg-001","text":"Aria raises her hand and casts Cure Wounds."}]}`
const spellTranscriptJSON = `{"id":"session-alpha","segments":[{"id":1,"text":"Aria raises her hand and casts Cure Wounds."}]}`
func spellSourceInput() contracts.LLMInputMaterial {
return contracts.NewLLMInputMaterial("source", "application/json", []byte(spellTranscriptJSON), "sha256:transcript", "file:///session-alpha.json")
@@ -339,6 +333,7 @@ func emptyChunkRequest(req contracts.ExtractionRequest) contracts.ExtractionRequ
type fakeSpellsLLMClient struct {
response extractionResponse
content []byte
err error
requests []contracts.StructuredCompletionRequest
}
@@ -354,10 +349,14 @@ func (client *fakeSpellsLLMClient) CompleteStructured(ctx context.Context, req c
return contracts.StructuredCompletionResponse{}, errors.New("unexpected output target")
}
*target = client.response
content, err := json.Marshal(client.response)
content := append([]byte(nil), client.content...)
if len(content) == 0 {
var err error
content, err = json.Marshal(client.response)
if err != nil {
return contracts.StructuredCompletionResponse{}, err
}
}
return contracts.StructuredCompletionResponse{Content: content}, nil
}

6
internal/modules/extract/dnd/spells/registry_test.go
View File

@@ -17,12 +17,6 @@ func TestNewReturnsExtractorWithMetadata(t *testing.T) {
if extractor.Key() != Key {
t.Fatalf("extractor.Key() = %q, want %q", extractor.Key(), Key)
}
if extractor.ArtifactType() != ArtifactType {
t.Fatalf("extractor.ArtifactType() = %q, want %q", extractor.ArtifactType(), ArtifactType)
}
if extractor.SchemaVersion() != SchemaVersion {
t.Fatalf("extractor.SchemaVersion() = %q, want %q", extractor.SchemaVersion(), SchemaVersion)
}
}
func TestModuleSpec(t *testing.T) {

106
internal/modules/extract/dnd/spells/runner_test.go
View File

@@ -26,14 +26,14 @@ func TestRunnerProcessesSeriatimInputWithDNDSpellsExtractor(t *testing.T) {
Spell: "Cure Wounds",
Effect: "Heals an injured ally.",
NarrativeDescription: "Aria restores the fighter after the fight.",
SourceRefs: responseSourceRefs(expectedDoc.ID, "seg-001", "seg-001"),
SourceRefs: responseSourceRefs(expectedDoc.ID, 1, 1),
},
{
Caster: "Borin",
Spell: "Fire Bolt",
Effect: "Scorches the wight.",
NarrativeDescription: "Borin hurls fire at the wight.",
SourceRefs: responseSourceRefs(expectedDoc.ID, "seg-003", "seg-003"),
SourceRefs: responseSourceRefs(expectedDoc.ID, 3, 3),
},
},
},
@@ -48,31 +48,30 @@ func TestRunnerProcessesSeriatimInputWithDNDSpellsExtractor(t *testing.T) {
t.Fatalf("Run() error = %v, want nil", err)
}
if len(output.Approved) != 2 {
t.Fatalf("len(Approved) = %d, want 2", len(output.Approved))
if len(output.NormalizeOutputs) != 1 {
t.Fatalf("len(NormalizeOutputs) = %d, want 1", len(output.NormalizeOutputs))
}
var first, second SpellCast
if err := json.Unmarshal(output.Approved[0].Payload, &first); err != nil {
t.Fatalf("Unmarshal(first payload) error = %v, want nil", err)
rawOutput := output.NormalizeOutputs[0]
if rawOutput.LaneID != "spells" || rawOutput.Schema.ID != ResponseSchemaID || rawOutput.Schema.Version != SchemaVersion {
t.Fatalf("raw output envelope = %#v, want dnd spells schema on spells lane", rawOutput)
}
if err := json.Unmarshal(output.Approved[1].Payload, &second); err != nil {
t.Fatalf("Unmarshal(second payload) error = %v, want nil", err)
response := decodeRunnerSpellResponse(t, rawOutput.Payload.Content)
if len(response.SpellCasts) != 2 {
t.Fatalf("len(spell_casts) = %d, want 2", len(response.SpellCasts))
}
first, second := response.SpellCasts[0], response.SpellCasts[1]
if first.Spell != "Cure Wounds" || second.Spell != "Fire Bolt" {
t.Fatalf("approved spell order = %q, %q; want response order", first.Spell, second.Spell)
t.Fatalf("spell order = %q, %q; want response order", first.Spell, second.Spell)
}
if first.Caster != "Aria" || second.Caster != "Borin" {
t.Fatalf("approved casters = %q, %q; want spell data", first.Caster, second.Caster)
t.Fatalf("casters = %q, %q; want spell data", first.Caster, second.Caster)
}
for _, artifact := range output.Approved {
if artifact.ExtractorKey != Key || artifact.ArtifactType != ArtifactType || artifact.SchemaVersion != SchemaVersion {
t.Fatalf("approved artifact envelope = %#v, want dnd spells envelope", artifact)
for _, spell := range response.SpellCasts {
if len(spell.SourceRefs) != 1 {
t.Fatalf("len(SourceRefs) = %d, want 1", len(spell.SourceRefs))
}
if len(artifact.SourceRefs) != 1 {
t.Fatalf("len(SourceRefs) = %d, want 1", len(artifact.SourceRefs))
}
if err := source.ValidateRef(expectedDoc, artifact.SourceRefs[0]); err != nil {
t.Fatalf("ValidateRef() error = %v, want nil", err)
if spell.SourceRefs[0].SourceID != expectedDoc.ID {
t.Fatalf("SourceID = %q, want fixture document ID", spell.SourceRefs[0].SourceID)
}
}
@@ -122,7 +121,7 @@ func TestRunnerPassesPartyAndGlossaryReferencesToDNDSpellsPrompt(t *testing.T) {
Spell: "Fire Bolt",
Effect: "Scorches the wight.",
NarrativeDescription: "Borin hurls fire at the wight.",
SourceRefs: responseSourceRefs(expectedDoc.ID, "seg-003", "seg-003"),
SourceRefs: responseSourceRefs(expectedDoc.ID, 3, 3),
},
},
},
@@ -137,8 +136,8 @@ func TestRunnerPassesPartyAndGlossaryReferencesToDNDSpellsPrompt(t *testing.T) {
t.Fatalf("Run() error = %v, want nil", err)
}
if len(output.Approved) != 1 {
t.Fatalf("len(Approved) = %d, want 1", len(output.Approved))
if len(output.NormalizeOutputs) != 1 {
t.Fatalf("len(NormalizeOutputs) = %d, want 1", len(output.NormalizeOutputs))
}
if len(output.Manifest.References) != 2 {
t.Fatalf("manifest references = %#v, want party and glossary provenance", output.Manifest.References)
@@ -178,8 +177,12 @@ func TestRunnerDoesNotExtractSpellMentionedOnlyInPartyReference(t *testing.T) {
t.Fatalf("Run() error = %v, want nil", err)
}
if len(output.Approved) != 0 {
t.Fatalf("approved artifacts = %#v, want no party-reference-only spell casts", output.Approved)
if len(output.NormalizeOutputs) != 1 {
t.Fatalf("len(NormalizeOutputs) = %d, want empty spell response output", len(output.NormalizeOutputs))
}
response := decodeRunnerSpellResponse(t, output.NormalizeOutputs[0].Payload.Content)
if len(response.SpellCasts) != 0 {
t.Fatalf("spell_casts = %#v, want no party-reference-only spell casts", response.SpellCasts)
}
if len(llmClient.requests) != 1 {
t.Fatalf("LLM calls = %d, want 1", len(llmClient.requests))
@@ -196,7 +199,7 @@ func TestRunnerDoesNotExtractSpellMentionedOnlyInPartyReference(t *testing.T) {
}
}
func TestRunnerRejectsDNDSpellCastWithInvalidSourceRef(t *testing.T) {
func TestRunnerCarriesDNDSpellCastWithInvalidSourceRefAsRawOutput(t *testing.T) {
raw := readDNDSpellsFixture(t)
resolved := resolveDNDSpellsPipeline(t)
llmClient := &fakeSpellsLLMClient{
@@ -207,7 +210,7 @@ func TestRunnerRejectsDNDSpellCastWithInvalidSourceRef(t *testing.T) {
Spell: "Cure Wounds",
Effect: "Heals an injured ally.",
NarrativeDescription: "Aria restores the fighter after the fight.",
SourceRefs: responseSourceRefs("spell-session", "seg-999", "seg-999"),
SourceRefs: responseSourceRefs("spell-session", 999, 999),
},
},
},
@@ -221,21 +224,21 @@ func TestRunnerRejectsDNDSpellCastWithInvalidSourceRef(t *testing.T) {
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if len(output.Approved) != 0 {
t.Fatalf("len(Approved) = %d, want 0", len(output.Approved))
if len(output.NormalizeOutputs) != 1 {
t.Fatalf("len(NormalizeOutputs) = %d, want 1", len(output.NormalizeOutputs))
}
if len(output.Rejected) != 1 {
t.Fatalf("len(Rejected) = %d, want 1", len(output.Rejected))
response := decodeRunnerSpellResponse(t, output.NormalizeOutputs[0].Payload.Content)
if len(response.SpellCasts) != 1 {
t.Fatalf("len(spell_casts) = %d, want 1", len(response.SpellCasts))
}
rejected := output.Rejected[0]
if rejected.ValidatorName != sourceRefValidatorName {
t.Fatalf("ValidatorName = %q, want %q", rejected.ValidatorName, sourceRefValidatorName)
if response.SpellCasts[0].SourceRefs[0].SourceID != "spell-session" {
t.Fatalf("SourceID = %q, want raw invalid source ref preserved", response.SpellCasts[0].SourceRefs[0].SourceID)
}
if rejected.ReasonCode != reasonInvalidSourceRef {
t.Fatalf("ReasonCode = %q, want %q", rejected.ReasonCode, reasonInvalidSourceRef)
if len(output.Rejected) != 0 {
t.Fatalf("len(Rejected) = %d, want 0", len(output.Rejected))
}
if output.Manifest.ValidationStatus != "rejected" {
t.Fatalf("ValidationStatus = %q, want rejected", output.Manifest.ValidationStatus)
if output.Manifest.ValidationStatus != "approved" {
t.Fatalf("ValidationStatus = %q, want approved", output.Manifest.ValidationStatus)
}
}
@@ -276,7 +279,7 @@ func dndSpellsReferenceSet(party string, glossary string) contracts.ReferenceSet
return contracts.ReferenceSet{Slots: slots}
}
func TestRunnerFailsWhenDNDSpellsExtractorReturnsMalformedOutput(t *testing.T) {
func TestRunnerCarriesMalformedDNDSpellsExtractorOutput(t *testing.T) {
raw := readDNDSpellsFixture(t)
resolved := resolveDNDSpellsPipeline(t)
llmClient := &fakeSpellsLLMClient{response: extractionResponse{}}
@@ -286,16 +289,17 @@ func TestRunnerFailsWhenDNDSpellsExtractorReturnsMalformedOutput(t *testing.T) {
RawInput: raw,
LLMClient: llmClient,
})
if err == nil {
t.Fatal("Run() error = nil, want malformed extraction error")
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if !strings.Contains(err.Error(), "extract lane") ||
!strings.Contains(err.Error(), "dnd spells") ||
!strings.Contains(err.Error(), "spell_casts") {
t.Fatalf("Run() error = %q, want D&D spells extraction context", err.Error())
if len(output.NormalizeOutputs) != 1 {
t.Fatalf("len(NormalizeOutputs) = %d, want raw output", len(output.NormalizeOutputs))
}
if output.Manifest.ValidationStatus != "failed" {
t.Fatalf("ValidationStatus = %q, want failed", output.Manifest.ValidationStatus)
if string(output.NormalizeOutputs[0].Payload.Content) != `{"spell_casts":null}` {
t.Fatalf("content = %s, want raw structured output", output.NormalizeOutputs[0].Payload.Content)
}
if output.Manifest.ValidationStatus != "approved" {
t.Fatalf("ValidationStatus = %q, want approved", output.Manifest.ValidationStatus)
}
}
@@ -345,3 +349,13 @@ func parseDNDSpellsFixture(t *testing.T, raw []byte) *source.SourceDocument {
}
return doc
}
func decodeRunnerSpellResponse(t *testing.T, raw []byte) extractionResponse {
t.Helper()
var response extractionResponse
if err := json.Unmarshal(raw, &response); err != nil {
t.Fatalf("Unmarshal(raw output) error = %v, want nil", err)
}
return response
}

14
internal/modules/extract/dnd/spells/test_helpers_test.go
View File

@@ -15,6 +15,10 @@ func promptExtractionRequest() contracts.ExtractionRequest {
ID: "session-alpha:chunk:0",
SourceID: doc.ID,
Index: 0,
StartUnitID: doc.Units[0].ID,
EndUnitID: doc.Units[len(doc.Units)-1].ID,
Content: []byte(`{"units":[1,2]}`),
MediaType: "application/json",
Units: append([]source.SourceUnit(nil), doc.Units...),
Metadata: map[string]any{"ignored": "chunk metadata"},
}
@@ -32,7 +36,7 @@ func promptSourceDocument() *source.SourceDocument {
Digest: "sha256:test",
Units: []source.SourceUnit{
{
ID: "seg-001",
ID: 1,
Kind: "transcript_segment",
Text: "Aria raises her hand and casts Cure Wounds.",
Metadata: map[string]any{
@@ -43,7 +47,7 @@ func promptSourceDocument() *source.SourceDocument {
},
},
{
ID: "seg-002",
ID: 2,
Kind: "transcript_segment",
Text: "The fighter's wounds begin to close.",
Metadata: map[string]any{"ignored": "not rendered"},
@@ -61,12 +65,12 @@ func mustJSON(t *testing.T, value any) string {
return string(encoded)
}
func responseSourceRefs(sourceID string, startUnitID string, endUnitID string) []dnd.SourceRefResponse {
func responseSourceRefs(sourceID string, startUnitID int, endUnitID int) []dnd.SourceRefResponse {
return []dnd.SourceRefResponse{
{
SourceID: sourceID,
StartUnitID: dnd.UnitRefFromString(startUnitID),
EndUnitID: dnd.UnitRefFromString(endUnitID),
StartUnitID: dnd.UnitRefFromInt(startUnitID),
EndUnitID: dnd.UnitRefFromInt(endUnitID),
},
}
}

6
internal/modules/extract/dnd/spells/testdata/seriatim_spell_session.json vendored
View File

@@ -5,21 +5,21 @@
},
"segments": [
{
"id": "seg-001",
"id": 1,
"start": 0,
"end": 4,
"speaker": "Alice",
"text": "Aria raises her holy symbol and casts Cure Wounds."
},
{
"id": "seg-002",
"id": 2,
"start": 4,
"end": 8,
"speaker": "DM",
"text": "The bandit mage casts Shield as the blow lands."
},
{
"id": "seg-003",
"id": 3,
"start": 8,
"end": 12,
"speaker": "Bob",

30
internal/modules/extract/dnd/spells/validator_test.go
View File

@@ -12,26 +12,6 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/framework/validate"
)
func TestExtractorValidatorsReturnsExpectedChain(t *testing.T) {
validators := New().Validators()
if len(validators) != 2 {
t.Fatalf("len(Validators()) = %d, want 2", len(validators))
}
if validators[0].Name() != shapeValidatorName {
t.Fatalf("Validators()[0].Name() = %q, want %q", validators[0].Name(), shapeValidatorName)
}
if validators[1].Name() != sourceRefValidatorName {
t.Fatalf("Validators()[1].Name() = %q, want %q", validators[1].Name(), sourceRefValidatorName)
}
validators[0] = nil
again := New().Validators()
if len(again) != 2 || again[0] == nil || again[0].Name() != shapeValidatorName {
t.Fatalf("Validators() after caller mutation = %#v, want fresh validators", again)
}
}
func TestValidatorsApproveValidCandidate(t *testing.T) {
candidate := validSpellCandidate(7)
@@ -130,22 +110,22 @@ func TestSourceRefValidatorRejectsInvalidRefs(t *testing.T) {
}{
{
name: "unknown source id",
ref: source.SourceRef{SourceID: "session-beta", StartUnitID: "seg-001", EndUnitID: "seg-002"},
ref: source.SourceRef{SourceID: "session-beta", StartUnitID: 1, EndUnitID: 2},
want: "does not match",
},
{
name: "unknown start unit",
ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: "seg-999", EndUnitID: "seg-002"},
ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 999, EndUnitID: 2},
want: "start_unit_id",
},
{
name: "unknown end unit",
ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: "seg-001", EndUnitID: "seg-999"},
ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 999},
want: "end_unit_id",
},
{
name: "reversed unit range",
ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: "seg-002", EndUnitID: "seg-001"},
ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 1},
want: "appears after",
},
}
@@ -234,7 +214,7 @@ func validSpellCandidate(index int) artifacts.ArtifactCandidate {
Index: index,
Payload: spellPayload(validSpellPayload()),
SourceRefs: []source.SourceRef{
{SourceID: "session-alpha", StartUnitID: "seg-001", EndUnitID: "seg-002"},
{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 2},
},
}
}

24
internal/modules/input/seriatim/adapter.go
View File

@@ -69,7 +69,7 @@ func (a *Adapter) Parse(ctx context.Context, req contracts.ParseRequest) (*sourc
Metadata: copyMetadata(parsed.Metadata),
}
seenSegmentIDs := make(map[string]struct{}, len(parsed.Segments))
seenSegmentIDs := make(map[int]struct{}, len(parsed.Segments))
for i, segment := range parsed.Segments {
unit, err := sourceUnit(segment, i, seenSegmentIDs)
if err != nil {
@@ -98,39 +98,35 @@ func Register(registry *pipeline.InputAdapterRegistry) error {
})
}
func sourceUnit(segment segment, index int, seen map[string]struct{}) (source.SourceUnit, error) {
func sourceUnit(segment segment, index int, seen map[int]struct{}) (source.SourceUnit, error) {
segmentLabel := fmt.Sprintf("segment[%d]", index)
segmentID := strings.TrimSpace(segment.ID)
if segmentID == "" {
return source.SourceUnit{}, inputErrorf("%s id must not be empty", segmentLabel)
}
if segmentID != segment.ID {
return source.SourceUnit{}, inputErrorf("%s id %q must not contain leading or trailing whitespace", segmentLabel, segment.ID)
if segment.ID <= 0 {
return source.SourceUnit{}, inputErrorf("%s id must be positive", segmentLabel)
}
if _, ok := seen[segment.ID]; ok {
return source.SourceUnit{}, inputErrorf("segment id %q is duplicated", segment.ID)
return source.SourceUnit{}, inputErrorf("segment id %d is duplicated", segment.ID)
}
seen[segment.ID] = struct{}{}
speaker := strings.TrimSpace(segment.Speaker)
if speaker == "" {
return source.SourceUnit{}, inputErrorf("segment %q speaker must not be empty", segment.ID)
return source.SourceUnit{}, inputErrorf("segment %d speaker must not be empty", segment.ID)
}
start, err := validTimestamp(segment.Start, fmt.Sprintf("segment %q start", segment.ID))
start, err := validTimestamp(segment.Start, fmt.Sprintf("segment %d start", segment.ID))
if err != nil {
return source.SourceUnit{}, err
}
end, err := validTimestamp(segment.End, fmt.Sprintf("segment %q end", segment.ID))
end, err := validTimestamp(segment.End, fmt.Sprintf("segment %d end", segment.ID))
if err != nil {
return source.SourceUnit{}, err
}
if end.Cmp(start) < 0 {
return source.SourceUnit{}, inputErrorf("segment %q end must be greater than or equal to start", segment.ID)
return source.SourceUnit{}, inputErrorf("segment %d end must be greater than or equal to start", segment.ID)
}
if strings.TrimSpace(segment.Text) == "" {
return source.SourceUnit{}, inputErrorf("segment %q text must not be empty", segment.ID)
return source.SourceUnit{}, inputErrorf("segment %d text must not be empty", segment.ID)
}
return source.SourceUnit{

40
internal/modules/input/seriatim/adapter_test.go
View File

@@ -41,8 +41,8 @@ func TestParseValidMinimalTranscript(t *testing.T) {
}
first := doc.Units[0]
if first.ID != "seg-001" {
t.Fatalf("first.ID = %q, want seg-001", first.ID)
if first.ID != 1 {
t.Fatalf("first.ID = %d, want 1", first.ID)
}
if first.Kind != UnitKind {
t.Fatalf("first.Kind = %q, want %q", first.Kind, UnitKind)
@@ -86,13 +86,13 @@ func TestParseAcceptsNumericSegmentIDs(t *testing.T) {
if len(doc.Units) != 2 {
t.Fatalf("len(doc.Units) = %d, want 2", len(doc.Units))
}
if doc.Units[0].ID != "1" || doc.Units[1].ID != "2" {
t.Fatalf("unit IDs = %#v, want numeric IDs normalized to strings", []string{doc.Units[0].ID, doc.Units[1].ID})
if doc.Units[0].ID != 1 || doc.Units[1].ID != 2 {
t.Fatalf("unit IDs = %#v, want numeric IDs", []int{doc.Units[0].ID, doc.Units[1].ID})
}
ref := source.SourceRef{
SourceID: doc.ID,
StartUnitID: "1",
EndUnitID: "2",
StartUnitID: 1,
EndUnitID: 2,
}
if err := source.ValidateRef(doc, ref); err != nil {
t.Fatalf("ValidateRef() error = %v, want nil", err)
@@ -115,7 +115,7 @@ func TestParseRequestSourceIDOverridesMetadataIDs(t *testing.T) {
}
func TestParseFallbackDocumentIDIsDeterministic(t *testing.T) {
raw := []byte(`{"metadata":{},"segments":[{"id":"s1","start":0,"end":1,"speaker":"Narrator","text":"Synthetic text."}]}`)
raw := []byte(`{"metadata":{},"segments":[{"id":1,"start":0,"end":1,"speaker":"Narrator","text":"Synthetic text."}]}`)
first, err := New().Parse(context.Background(), contracts.ParseRequest{Raw: raw})
if err != nil {
@@ -138,7 +138,7 @@ func TestParseFallbackDocumentIDIsDeterministic(t *testing.T) {
}
func TestParseUsesMetadataSourceIDWhenMetadataIDIsAbsent(t *testing.T) {
raw := []byte(`{"metadata":{"source_id":" source-from-metadata "},"segments":[{"id":"s1","start":0,"end":1,"speaker":"Narrator","text":"Synthetic text."}]}`)
raw := []byte(`{"metadata":{"source_id":" source-from-metadata "},"segments":[{"id":1,"start":0,"end":1,"speaker":"Narrator","text":"Synthetic text."}]}`)
doc, err := New().Parse(context.Background(), contracts.ParseRequest{Raw: raw})
if err != nil {
@@ -203,7 +203,7 @@ func TestParseRejectsInvalidInput(t *testing.T) {
{
name: "missing segment id",
raw: validJSONWithSegment(`"start":0,"end":1,"speaker":"Narrator","text":"Synthetic text."`),
wantErr: []string{"id", "empty"},
wantErr: []string{"id", "positive"},
},
{
name: "invalid segment id type",
@@ -212,52 +212,52 @@ func TestParseRejectsInvalidInput(t *testing.T) {
},
{
name: "whitespace segment id",
raw: validJSONWithSegment(`"id":" s1 ","start":0,"end":1,"speaker":"Narrator","text":"Synthetic text."`),
raw: validJSONWithSegment(`"id":" 1 ","start":0,"end":1,"speaker":"Narrator","text":"Synthetic text."`),
wantErr: []string{"id", "whitespace"},
},
{
name: "empty text",
raw: validJSONWithSegment(`"id":"s1","start":0,"end":1,"speaker":"Narrator","text":" "`),
raw: validJSONWithSegment(`"id":1,"start":0,"end":1,"speaker":"Narrator","text":" "`),
wantErr: []string{"text", "empty"},
},
{
name: "missing speaker",
raw: validJSONWithSegment(`"id":"s1","start":0,"end":1,"text":"Synthetic text."`),
raw: validJSONWithSegment(`"id":1,"start":0,"end":1,"text":"Synthetic text."`),
wantErr: []string{"speaker", "empty"},
},
{
name: "missing start",
raw: validJSONWithSegment(`"id":"s1","end":1,"speaker":"Narrator","text":"Synthetic text."`),
raw: validJSONWithSegment(`"id":1,"end":1,"speaker":"Narrator","text":"Synthetic text."`),
wantErr: []string{"start", "empty"},
},
{
name: "missing end",
raw: validJSONWithSegment(`"id":"s1","start":0,"speaker":"Narrator","text":"Synthetic text."`),
raw: validJSONWithSegment(`"id":1,"start":0,"speaker":"Narrator","text":"Synthetic text."`),
wantErr: []string{"end", "empty"},
},
{
name: "negative start",
raw: validJSONWithSegment(`"id":"s1","start":-1,"end":1,"speaker":"Narrator","text":"Synthetic text."`),
raw: validJSONWithSegment(`"id":1,"start":-1,"end":1,"speaker":"Narrator","text":"Synthetic text."`),
wantErr: []string{"start", "negative"},
},
{
name: "non-numeric end",
raw: validJSONWithSegment(`"id":"s1","start":0,"end":"late","speaker":"Narrator","text":"Synthetic text."`),
raw: validJSONWithSegment(`"id":1,"start":0,"end":"late","speaker":"Narrator","text":"Synthetic text."`),
wantErr: []string{"segment[0]", "end", "number"},
},
{
name: "non-finite timestamp",
raw: validJSONWithSegment(`"id":"s1","start":1e10000,"end":1e10000,"speaker":"Narrator","text":"Synthetic text."`),
raw: validJSONWithSegment(`"id":1,"start":1e10000,"end":1e10000,"speaker":"Narrator","text":"Synthetic text."`),
wantErr: []string{"start", "valid number"},
},
{
name: "end before start",
raw: validJSONWithSegment(`"id":"s1","start":2,"end":1,"speaker":"Narrator","text":"Synthetic text."`),
raw: validJSONWithSegment(`"id":1,"start":2,"end":1,"speaker":"Narrator","text":"Synthetic text."`),
wantErr: []string{"end", "start"},
},
{
name: "end before start beyond float precision",
raw: validJSONWithSegment(`"id":"s1","start":9007199254740993,"end":9007199254740992,"speaker":"Narrator","text":"Synthetic text."`),
raw: validJSONWithSegment(`"id":1,"start":9007199254740993,"end":9007199254740992,"speaker":"Narrator","text":"Synthetic text."`),
wantErr: []string{"end", "start"},
},
}
@@ -284,7 +284,7 @@ func TestParseRejectsDuplicateSegmentIDs(t *testing.T) {
if err == nil {
t.Fatal("Parse() error = nil, want duplicate ID error")
}
if !strings.Contains(err.Error(), "duplicated") || !strings.Contains(err.Error(), "seg-001") {
if !strings.Contains(err.Error(), "duplicated") || !strings.Contains(err.Error(), "1") {
t.Fatalf("Parse() error = %q, want duplicate segment context", err.Error())
}
}

6
internal/modules/input/seriatim/config_test.go
View File

@@ -219,14 +219,8 @@ type fakeExtractor struct{}
func (fakeExtractor) Key() string { return "fake/extract" }
func (fakeExtractor) ArtifactType() string { return "fake" }
func (fakeExtractor) SchemaVersion() string { return "v1" }
func (fakeExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (fakeExtractor) Validators() []contracts.Validator { return nil }
func (fakeExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
return contracts.ExtractionResult{}, nil
}

43
internal/modules/input/seriatim/model.go
View File

@@ -5,6 +5,8 @@ import (
"encoding/json"
"fmt"
"io"
"strconv"
"strings"
)
type transcript struct {
@@ -13,7 +15,7 @@ type transcript struct {
}
type segment struct {
ID string `json:"id"`
ID int `json:"id"`
Start json.Number `json:"start"`
End json.Number `json:"end"`
Speaker string `json:"speaker"`
@@ -78,7 +80,7 @@ func decodeSegment(raw []byte, index int) (segment, error) {
var decoded segment
if err := decodeOptionalSegmentID(fields, "id", &decoded.ID); err != nil {
return segment{}, fmt.Errorf("segment[%d] id must be a string or number: %w", index, err)
return segment{}, fmt.Errorf("segment[%d] id must be a positive integer string or number: %w", index, err)
}
if err := decodeOptionalNumber(fields, "start", &decoded.Start); err != nil {
return segment{}, fmt.Errorf("segment[%d] start must be a number: %w", index, err)
@@ -103,7 +105,7 @@ func decodeOptionalString(fields map[string]json.RawMessage, key string, out *st
return decodeJSON(raw, out)
}
func decodeOptionalSegmentID(fields map[string]json.RawMessage, key string, out *string) error {
func decodeOptionalSegmentID(fields map[string]json.RawMessage, key string, out *int) error {
raw, ok := fields[key]
if !ok {
return nil
@@ -111,17 +113,46 @@ func decodeOptionalSegmentID(fields map[string]json.RawMessage, key string, out
var text string
if err := decodeJSON(raw, &text); err == nil {
*out = text
parsed, err := parsePositiveInt(text)
if err != nil {
return err
}
*out = parsed
return nil
}
var number json.Number
if err := decodeJSON(raw, &number); err == nil {
*out = number.String()
parsed, err := parsePositiveInt(number.String())
if err != nil {
return err
}
*out = parsed
return nil
}
return fmt.Errorf("must be a string or number")
return fmt.Errorf("must be a positive integer string or number")
}
func parsePositiveInt(value string) (int, error) {
trimmed := strings.TrimSpace(value)
if trimmed == "" {
return 0, fmt.Errorf("must not be empty")
}
if trimmed != value {
return 0, fmt.Errorf("must not contain leading or trailing whitespace")
}
parsed, err := strconv.Atoi(value)
if err != nil {
return 0, fmt.Errorf("must be an integer")
}
if parsed <= 0 {
return 0, fmt.Errorf("must be positive")
}
if strconv.Itoa(parsed) != value {
return 0, fmt.Errorf("must be a canonical positive integer")
}
return parsed, nil
}
func decodeOptionalNumber(fields map[string]json.RawMessage, key string, out *json.Number) error {

80
internal/modules/input/seriatim/runner_test.go
View File

@@ -7,7 +7,6 @@ import (
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
@@ -43,22 +42,29 @@ func TestRunnerProcessesSeriatimInputWithFakeModules(t *testing.T) {
if got := output.Manifest.SourceDigests; len(got) != 1 || got[0] != expectedDoc.Digest {
t.Fatalf("manifest source digests = %#v, want %q", got, expectedDoc.Digest)
}
if len(output.Approved) != 1 {
t.Fatalf("len(Approved) = %d, want 1", len(output.Approved))
if len(output.NormalizeOutputs) != 1 {
t.Fatalf("len(NormalizeOutputs) = %d, want 1", len(output.NormalizeOutputs))
}
artifact := output.Approved[0]
if artifact.ExtractorKey != "fake/extract" || artifact.ArtifactType != "fake.event" || artifact.SchemaVersion != "v1" {
t.Fatalf("approved artifact envelope = %#v, want fake extractor envelope", artifact)
rawOutput := output.NormalizeOutputs[0]
if rawOutput.LaneID != "events" || rawOutput.NormalizerKey != pipeline.DefaultNormalizeModule || rawOutput.Schema.ID != "fake.event" || rawOutput.Schema.Version != "v1" {
t.Fatalf("raw output envelope = %#v, want fake extractor envelope", rawOutput)
}
if len(artifact.SourceRefs) != 1 {
t.Fatalf("len(SourceRefs) = %d, want 1", len(artifact.SourceRefs))
var payload struct {
Value string `json:"value"`
SourceRefs []source.SourceRef `json:"source_refs"`
}
if err := source.ValidateRef(expectedDoc, artifact.SourceRefs[0]); err != nil {
if err := json.Unmarshal(rawOutput.Payload.Content, &payload); err != nil {
t.Fatalf("Unmarshal(raw output) error = %v, want nil", err)
}
if len(payload.SourceRefs) != 1 {
t.Fatalf("len(SourceRefs) = %d, want 1", len(payload.SourceRefs))
}
if err := source.ValidateRef(expectedDoc, payload.SourceRefs[0]); err != nil {
t.Fatalf("ValidateRef() error = %v, want nil", err)
}
if artifact.SourceRefs[0].StartUnitID != "seg-001" || artifact.SourceRefs[0].EndUnitID != "seg-002" {
t.Fatalf("SourceRefs[0] = %#v, want Seriatim unit IDs", artifact.SourceRefs[0])
if payload.SourceRefs[0].StartUnitID != 1 || payload.SourceRefs[0].EndUnitID != 2 {
t.Fatalf("SourceRefs[0] = %#v, want Seriatim unit IDs", payload.SourceRefs[0])
}
if extractor.calls != 1 {
t.Fatalf("extractor calls = %d, want 1", extractor.calls)
@@ -168,6 +174,10 @@ func (runnerSeriatimChunker) Chunk(ctx context.Context, req contracts.ChunkReque
ID: req.Source.ID + ":chunk:0",
SourceID: req.Source.ID,
Index: 0,
StartUnitID: req.Source.Units[0].ID,
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
Content: []byte(`{"units":[1,2]}`),
MediaType: "application/json",
Units: append([]source.SourceUnit(nil), req.Source.Units...),
},
},
@@ -182,22 +192,10 @@ func (e *runnerSeriatimExtractor) Key() string {
return "fake/extract"
}
func (e *runnerSeriatimExtractor) ArtifactType() string {
return "fake.event"
}
func (e *runnerSeriatimExtractor) SchemaVersion() string {
return "v1"
}
func (e *runnerSeriatimExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (e *runnerSeriatimExtractor) Validators() []contracts.Validator {
return nil
}
func (e *runnerSeriatimExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
e.calls++
if req.Source == nil {
@@ -206,28 +204,29 @@ func (e *runnerSeriatimExtractor) Extract(ctx context.Context, req contracts.Ext
if req.Chunk == nil {
return contracts.ExtractionResult{}, fmt.Errorf("chunk must not be nil")
}
if got := unitIDs(req.Source.Units); !equalStrings(got, []string{"seg-001", "seg-002"}) {
if got := unitIDs(req.Source.Units); !equalInts(got, []int{1, 2}) {
return contracts.ExtractionResult{}, fmt.Errorf("source unit IDs = %#v, want Seriatim segment IDs", got)
}
if got := unitIDs(req.Chunk.Units); !equalStrings(got, []string{"seg-001", "seg-002"}) {
if got := unitIDs(req.Chunk.Units); !equalInts(got, []int{1, 2}) {
return contracts.ExtractionResult{}, fmt.Errorf("chunk unit IDs = %#v, want Seriatim segment IDs", got)
}
for _, unit := range req.Chunk.Units {
if speaker, ok := Speaker(unit); !ok || speaker == "" {
return contracts.ExtractionResult{}, fmt.Errorf("unit %q missing speaker metadata", unit.ID)
return contracts.ExtractionResult{}, fmt.Errorf("unit %d missing speaker metadata", unit.ID)
}
if _, ok := Start(unit); !ok {
return contracts.ExtractionResult{}, fmt.Errorf("unit %q missing start metadata", unit.ID)
return contracts.ExtractionResult{}, fmt.Errorf("unit %d missing start metadata", unit.ID)
}
if _, ok := End(unit); !ok {
return contracts.ExtractionResult{}, fmt.Errorf("unit %q missing end metadata", unit.ID)
return contracts.ExtractionResult{}, fmt.Errorf("unit %d missing end metadata", unit.ID)
}
}
return contracts.ExtractionResult{
Candidates: []artifacts.ArtifactCandidate{
{
Payload: json.RawMessage(`{"value":"seriatim-source-ref"}`),
payload, err := json.Marshal(struct {
Value string `json:"value"`
SourceRefs []source.SourceRef `json:"source_refs"`
}{
Value: "seriatim-source-ref",
SourceRefs: []source.SourceRef{
{
SourceID: req.Source.ID,
@@ -235,6 +234,17 @@ func (e *runnerSeriatimExtractor) Extract(ctx context.Context, req contracts.Ext
EndUnitID: req.Chunk.Units[len(req.Chunk.Units)-1].ID,
},
},
})
if err != nil {
return contracts.ExtractionResult{}, err
}
return contracts.ExtractionResult{
Output: contracts.ExtractOutput{
Schema: contracts.ResponseSchema{ID: "fake.event", Name: "fake_event", Version: "v1"},
Payload: contracts.RawPayload{
Content: payload,
MediaType: "application/json",
},
},
}, nil
@@ -254,15 +264,15 @@ func (runnerSeriatimOutput) Encode(ctx context.Context, req contracts.OutputRequ
}, nil
}
func unitIDs(units []source.SourceUnit) []string {
ids := make([]string, 0, len(units))
func unitIDs(units []source.SourceUnit) []int {
ids := make([]int, 0, len(units))
for _, unit := range units {
ids = append(ids, unit.ID)
}
return ids
}
func equalStrings(a, b []string) bool {
func equalInts(a, b []int) bool {
if len(a) != len(b) {
return false
}

4
internal/modules/input/seriatim/testdata/duplicate_segment_id.json vendored
View File

@@ -4,14 +4,14 @@
},
"segments": [
{
"id": "seg-001",
"id": 1,
"start": 0,
"end": 1,
"speaker": "Narrator",
"text": "First segment."
},
{
"id": "seg-001",
"id": 1,
"start": 1,
"end": 2,
"speaker": "Player",

4
internal/modules/input/seriatim/testdata/valid_minimal.json vendored
View File

@@ -6,14 +6,14 @@
},
"segments": [
{
"id": "seg-001",
"id": 1,
"start": 0,
"end": 4.5,
"speaker": "Narrator",
"text": "The stone door opens."
},
{
"id": "seg-002",
"id": 2,
"start": 4.5,
"end": 8,
"speaker": "Player",

173
internal/modules/merge/appendorder/merger.go
View File

@@ -4,9 +4,10 @@ import (
"context"
"encoding/json"
"fmt"
"mime"
"sort"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
@@ -36,11 +37,39 @@ func (m *Merger) Merge(ctx context.Context, req contracts.MergeRequest) (contrac
return contracts.MergeResult{}, mergerErrorf("context error before merge: %w", err)
}
var candidates []artifacts.ArtifactCandidate
for _, chunkArtifacts := range req.ChunkArtifacts {
candidates = append(candidates, cloneCandidates(chunkArtifacts.Candidates)...)
outputs, err := orderedOutputs(req.ExtractOutputs)
if err != nil {
return contracts.MergeResult{}, err
}
return contracts.MergeResult{Candidates: candidates}, nil
if len(outputs) == 1 {
payload := cloneRawPayload(outputs[0].Payload)
return contracts.MergeResult{
Output: contracts.MergeOutput{
LaneID: req.LaneID,
MergerKey: Key,
SourceID: outputs[0].SourceID,
Schema: outputs[0].Schema,
Payload: payload,
},
}, nil
}
content, err := mergedContent(outputs)
if err != nil {
return contracts.MergeResult{}, err
}
return contracts.MergeResult{
Output: contracts.MergeOutput{
LaneID: req.LaneID,
MergerKey: Key,
SourceID: sourceID(outputs),
Schema: commonSchema(outputs),
Payload: contracts.RawPayload{
Content: content,
MediaType: "application/json",
},
},
}, nil
}
func ModuleSpec() pipeline.ModuleSpec {
@@ -57,27 +86,127 @@ func Register(registry *pipeline.MergerRegistry) error {
})
}
func cloneCandidates(candidates []artifacts.ArtifactCandidate) []artifacts.ArtifactCandidate {
if len(candidates) == 0 {
return nil
func orderedOutputs(outputs []contracts.ExtractOutput) ([]contracts.ExtractOutput, error) {
ordered := make([]contracts.ExtractOutput, 0, len(outputs))
for _, output := range outputs {
if !isJSONMediaType(output.Payload.MediaType) {
return nil, mergerErrorf("extract output for chunk %q has unsupported media type %q", output.ChunkID, output.Payload.MediaType)
}
if !json.Valid(output.Payload.Content) {
return nil, mergerErrorf("extract output for chunk %q contains invalid JSON", output.ChunkID)
}
ordered = append(ordered, cloneExtractOutput(output))
}
sort.SliceStable(ordered, func(i, j int) bool {
return ordered[i].ChunkIndex < ordered[j].ChunkIndex
})
return ordered, nil
}
out := make([]artifacts.ArtifactCandidate, 0, len(candidates))
for _, candidate := range candidates {
out = append(out, cloneCandidate(candidate))
func mergedContent(outputs []contracts.ExtractOutput) ([]byte, error) {
values := make([]any, 0, len(outputs))
objects := make([]map[string]any, 0, len(outputs))
for _, output := range outputs {
var value any
if err := json.Unmarshal(output.Payload.Content, &value); err != nil {
return nil, mergerErrorf("decode extract output for chunk %q: %w", output.ChunkID, err)
}
return out
values = append(values, value)
object, ok := value.(map[string]any)
if !ok {
continue
}
objects = append(objects, object)
}
func cloneCandidate(candidate artifacts.ArtifactCandidate) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: candidate.Index,
ExtractorKey: candidate.ExtractorKey,
ArtifactType: candidate.ArtifactType,
SchemaVersion: candidate.SchemaVersion,
Payload: append(json.RawMessage(nil), candidate.Payload...),
SourceRefs: append([]source.SourceRef(nil), candidate.SourceRefs...),
Metadata: cloneMetadata(candidate.Metadata),
if len(objects) == len(outputs) {
if field, ok := commonArrayField(objects); ok {
merged := make([]any, 0)
for _, object := range objects {
items := object[field].([]any)
merged = append(merged, items...)
}
return marshalMerged(map[string]any{field: merged})
}
}
return marshalMerged(values)
}
func commonArrayField(objects []map[string]any) (string, bool) {
if len(objects) == 0 {
return "", false
}
candidates := map[string]struct{}{}
for key, value := range objects[0] {
if _, ok := value.([]any); ok {
candidates[key] = struct{}{}
}
}
for _, object := range objects[1:] {
for key := range candidates {
if _, ok := object[key].([]any); !ok {
delete(candidates, key)
}
}
}
if len(candidates) != 1 {
return "", false
}
for key := range candidates {
return key, true
}
return "", false
}
func marshalMerged(value any) ([]byte, error) {
content, err := json.Marshal(value)
if err != nil {
return nil, mergerErrorf("encode merged output: %w", err)
}
return content, nil
}
func isJSONMediaType(mediaType string) bool {
base, _, err := mime.ParseMediaType(strings.TrimSpace(mediaType))
if err != nil {
base = strings.TrimSpace(mediaType)
}
return strings.EqualFold(base, "application/json")
}
func sourceID(outputs []contracts.ExtractOutput) string {
for _, output := range outputs {
if output.SourceID != "" {
return output.SourceID
}
}
return ""
}
func commonSchema(outputs []contracts.ExtractOutput) contracts.ResponseSchema {
if len(outputs) == 0 {
return contracts.ResponseSchema{}
}
schema := outputs[0].Schema
for _, output := range outputs[1:] {
if output.Schema != schema {
return contracts.ResponseSchema{}
}
}
return schema
}
func cloneExtractOutput(output contracts.ExtractOutput) contracts.ExtractOutput {
output.Payload = cloneRawPayload(output.Payload)
return output
}
func cloneRawPayload(payload contracts.RawPayload) contracts.RawPayload {
return contracts.RawPayload{
Content: append([]byte(nil), payload.Content...),
MediaType: payload.MediaType,
Metadata: cloneMetadata(payload.Metadata),
Warnings: append([]contracts.Warning(nil), payload.Warnings...),
}
}

218
internal/modules/merge/appendorder/merger_test.go
View File

@@ -4,10 +4,9 @@ import (
"context"
"encoding/json"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
@@ -35,113 +34,158 @@ func TestModuleSpecAndRegister(t *testing.T) {
}
}
func TestMergePreservesChunkAndCandidateOrder(t *testing.T) {
func TestMergePassesThroughSingleExtractOutput(t *testing.T) {
input := extractOutput("chunk-0", 0, `{"name":"original"}`)
result, err := New().Merge(context.Background(), contracts.MergeRequest{
ChunkArtifacts: []contracts.ChunkArtifacts{
{
Chunk: sourceChunk(0),
Candidates: []artifacts.ArtifactCandidate{candidate(2, "first-b"), candidate(1, "first-a")},
},
{
Chunk: sourceChunk(1),
Candidates: []artifacts.ArtifactCandidate{candidate(4, "second-b"), candidate(3, "second-a")},
LaneID: "events",
ExtractOutputs: []contracts.ExtractOutput{input},
})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
if result.Output.LaneID != "events" || result.Output.MergerKey != Key {
t.Fatalf("output provenance = %#v, want lane and merger", result.Output)
}
if string(result.Output.Payload.Content) != `{"name":"original"}` {
t.Fatalf("content = %s, want original content", result.Output.Payload.Content)
}
if result.Output.Payload.Metadata["name"] != "chunk-0" {
t.Fatalf("metadata = %#v, want original metadata", result.Output.Payload.Metadata)
}
}
func TestMergeDefensivelyCopiesRawPayload(t *testing.T) {
input := extractOutput("chunk-0", 0, `{"name":"original"}`)
result, err := New().Merge(context.Background(), contracts.MergeRequest{
LaneID: "events",
ExtractOutputs: []contracts.ExtractOutput{input},
})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
input.Payload.Content[0] = '['
input.Payload.Metadata["name"] = "changed"
if string(result.Output.Payload.Content) != `{"name":"original"}` {
t.Fatalf("content changed after input mutation: %s", result.Output.Payload.Content)
}
if result.Output.Payload.Metadata["name"] != "chunk-0" {
t.Fatalf("metadata changed after input mutation: %#v", result.Output.Payload.Metadata)
}
}
func TestMergeConcatenatesCommonTopLevelArrayFieldInChunkOrder(t *testing.T) {
result, err := New().Merge(context.Background(), contracts.MergeRequest{
LaneID: "events",
ExtractOutputs: []contracts.ExtractOutput{
extractOutput("chunk-1", 1, `{"events":[{"name":"second"}]}`),
extractOutput("chunk-0", 0, `{"events":[{"name":"first"}]}`),
},
})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
if result.Output.Payload.MediaType != "application/json" {
t.Fatalf("MediaType = %q, want application/json", result.Output.Payload.MediaType)
}
var decoded struct {
Events []struct {
Name string `json:"name"`
} `json:"events"`
}
if err := json.Unmarshal(result.Output.Payload.Content, &decoded); err != nil {
t.Fatalf("Unmarshal() error = %v, want nil", err)
}
if len(decoded.Events) != 2 || decoded.Events[0].Name != "first" || decoded.Events[1].Name != "second" {
t.Fatalf("events = %#v, want concatenated chunk order", decoded.Events)
}
if result.Output.Schema.ID != "schema-id" {
t.Fatalf("schema = %#v, want common extract schema", result.Output.Schema)
}
}
func TestMergeFallsBackToOrderedJSONValueArrayWhenShapesDiffer(t *testing.T) {
result, err := New().Merge(context.Background(), contracts.MergeRequest{
LaneID: "events",
ExtractOutputs: []contracts.ExtractOutput{
extractOutput("chunk-1", 1, `{"notes":["second"]}`),
extractOutput("chunk-0", 0, `{"events":[{"name":"first"}]}`),
},
})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
got := candidateNames(result.Candidates)
want := []string{"first-b", "first-a", "second-b", "second-a"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("candidate order = %#v, want %#v", got, want)
var decoded []map[string]any
if err := json.Unmarshal(result.Output.Payload.Content, &decoded); err != nil {
t.Fatalf("Unmarshal() error = %v, want nil", err)
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
if len(decoded) != 2 {
t.Fatalf("len(decoded) = %d, want 2", len(decoded))
}
if _, ok := decoded[0]["events"]; !ok {
t.Fatalf("decoded[0] = %#v, want first chunk value", decoded[0])
}
if _, ok := decoded[1]["notes"]; !ok {
t.Fatalf("decoded[1] = %#v, want second chunk value", decoded[1])
}
}
func TestMergeDefensivelyCopiesCandidates(t *testing.T) {
input := []contracts.ChunkArtifacts{
func TestMergeRejectsInvalidJSONAndNonJSONMediaTypes(t *testing.T) {
tests := []struct {
name string
output contracts.ExtractOutput
want string
}{
{
Chunk: sourceChunk(0),
Candidates: []artifacts.ArtifactCandidate{candidate(1, "original")},
name: "invalid JSON",
output: extractOutput("chunk-0", 0, `{"events":[`),
want: "invalid JSON",
},
{
name: "non JSON media type",
output: func() contracts.ExtractOutput {
output := extractOutput("chunk-0", 0, `{"events":[]}`)
output.Payload.MediaType = "text/plain"
return output
}(),
want: "unsupported media type",
},
}
result, err := New().Merge(context.Background(), contracts.MergeRequest{ChunkArtifacts: input})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
_, err := New().Merge(context.Background(), contracts.MergeRequest{
LaneID: "events",
ExtractOutputs: []contracts.ExtractOutput{test.output},
})
if err == nil {
t.Fatal("Merge() error = nil, want error")
}
if len(result.Candidates) != 1 {
t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
if !strings.Contains(err.Error(), test.want) {
t.Fatalf("Merge() error = %q, want %q", err.Error(), test.want)
}
input[0].Candidates[0].Index = 99
input[0].Candidates[0].Payload[0] = '['
input[0].Candidates[0].SourceRefs[0].StartUnitID = "changed"
input[0].Candidates[0].Metadata["name"] = "changed"
got := result.Candidates[0]
if got.Index != 1 {
t.Fatalf("Index = %d, want 1", got.Index)
}
if string(got.Payload) != `{"name":"original"}` {
t.Fatalf("Payload = %s, want original payload", got.Payload)
}
if got.SourceRefs[0].StartUnitID != "u1" {
t.Fatalf("SourceRefs = %#v, want original source ref", got.SourceRefs)
}
if got.Metadata["name"] != "original" {
t.Fatalf("Metadata = %#v, want original metadata", got.Metadata)
})
}
}
func TestMergeHandlesEmptyInput(t *testing.T) {
result, err := New().Merge(context.Background(), contracts.MergeRequest{})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
if len(result.Candidates) != 0 {
t.Fatalf("len(Candidates) = %d, want 0", len(result.Candidates))
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
}
}
func candidate(index int, name string) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: index,
ExtractorKey: "generic-extractor",
ArtifactType: "generic-artifact",
SchemaVersion: "v1",
Payload: json.RawMessage(`{"name":"` + name + `"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
},
Metadata: map[string]any{
"name": name,
},
}
}
func candidateNames(candidates []artifacts.ArtifactCandidate) []string {
names := make([]string, 0, len(candidates))
for _, candidate := range candidates {
names = append(names, candidate.Metadata["name"].(string))
}
return names
}
func sourceChunk(index int) contracts.SourceChunk {
return contracts.SourceChunk{
ID: "chunk",
func extractOutput(chunkID string, chunkIndex int, content string) contracts.ExtractOutput {
return contracts.ExtractOutput{
LaneID: "events",
ExtractorKey: "extract",
SourceID: "source-1",
Index: index,
Units: []source.SourceUnit{
{ID: "u1", Kind: "unit", Text: "Source unit."},
ChunkID: chunkID,
ChunkIndex: chunkIndex,
Schema: contracts.ResponseSchema{ID: "schema-id", Name: "schema-name", Version: "v1"},
Payload: contracts.RawPayload{
Content: []byte(content),
MediaType: "application/json",
Metadata: map[string]any{"name": chunkID},
},
}
}

36
internal/modules/normalize/noop/normalizer.go
View File

@@ -2,11 +2,8 @@ package noop
import (
"context"
"encoding/json"
"fmt"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
@@ -39,7 +36,15 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.NormalizeReque
if err := ctx.Err(); err != nil {
return contracts.NormalizeResult{}, normalizerErrorf("context error before normalize: %w", err)
}
return contracts.NormalizeResult{Candidates: cloneCandidates(req.Candidates)}, nil
return contracts.NormalizeResult{
Output: contracts.NormalizeOutput{
LaneID: req.LaneID,
NormalizerKey: Key,
SourceID: req.MergeOutput.SourceID,
Schema: req.MergeOutput.Schema,
Payload: cloneRawPayload(req.MergeOutput.Payload),
},
}, nil
}
func ModuleSpec() pipeline.ModuleSpec {
@@ -57,24 +62,13 @@ func Register(registry *pipeline.NormalizerRegistry) error {
})
}
func cloneCandidates(candidates []artifacts.ArtifactCandidate) []artifacts.ArtifactCandidate {
if len(candidates) == 0 {
return nil
func cloneRawPayload(payload contracts.RawPayload) contracts.RawPayload {
return contracts.RawPayload{
Content: append([]byte(nil), payload.Content...),
MediaType: payload.MediaType,
Metadata: cloneMetadata(payload.Metadata),
Warnings: append([]contracts.Warning(nil), payload.Warnings...),
}
out := make([]artifacts.ArtifactCandidate, 0, len(candidates))
for _, candidate := range candidates {
out = append(out, artifacts.ArtifactCandidate{
Index: candidate.Index,
ExtractorKey: candidate.ExtractorKey,
ArtifactType: candidate.ArtifactType,
SchemaVersion: candidate.SchemaVersion,
Payload: append(json.RawMessage(nil), candidate.Payload...),
SourceRefs: append([]source.SourceRef(nil), candidate.SourceRefs...),
Metadata: cloneMetadata(candidate.Metadata),
})
}
return out
}
func cloneMetadata(metadata map[string]any) map[string]any {

109
internal/modules/normalize/noop/normalizer_test.go
View File

@@ -2,12 +2,9 @@ package noop
import (
"context"
"encoding/json"
"reflect"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
@@ -43,98 +40,60 @@ func TestModuleSpecAndRegister(t *testing.T) {
}
}
func TestNormalizePassesThroughOrderAndValues(t *testing.T) {
input := []artifacts.ArtifactCandidate{
candidate(3, "third"),
candidate(1, "first"),
candidate(2, "second"),
}
func TestNormalizePassesThroughMergeOutput(t *testing.T) {
input := mergeOutput(`{"name":"original"}`)
result, err := New().Normalize(context.Background(), contracts.NormalizeRequest{Candidates: input})
result, err := New().Normalize(context.Background(), contracts.NormalizeRequest{
LaneID: "events",
MergeOutput: input,
})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
}
got := candidateNames(result.Candidates)
want := []string{"third", "first", "second"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("candidate order = %#v, want %#v", got, want)
if result.Output.LaneID != "events" || result.Output.NormalizerKey != Key {
t.Fatalf("output provenance = %#v, want lane and normalizer", result.Output)
}
if !reflect.DeepEqual(result.Candidates[0].SourceRefs, input[0].SourceRefs) {
t.Fatalf("SourceRefs = %#v, want %#v", result.Candidates[0].SourceRefs, input[0].SourceRefs)
if string(result.Output.Payload.Content) != `{"name":"original"}` {
t.Fatalf("content = %s, want original content", result.Output.Payload.Content)
}
if !reflect.DeepEqual(result.Candidates[0].Metadata, input[0].Metadata) {
t.Fatalf("Metadata = %#v, want %#v", result.Candidates[0].Metadata, input[0].Metadata)
if result.Output.Payload.Metadata["name"] != "original" {
t.Fatalf("metadata = %#v, want original metadata", result.Output.Payload.Metadata)
}
}
func TestNormalizeDefensivelyCopiesCandidates(t *testing.T) {
input := []artifacts.ArtifactCandidate{candidate(1, "original")}
func TestNormalizeDefensivelyCopiesRawPayload(t *testing.T) {
input := mergeOutput(`{"name":"original"}`)
result, err := New().Normalize(context.Background(), contracts.NormalizeRequest{Candidates: input})
result, err := New().Normalize(context.Background(), contracts.NormalizeRequest{
LaneID: "events",
MergeOutput: input,
})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(result.Candidates) != 1 {
t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
}
input[0].Index = 99
input[0].Payload[0] = '['
input[0].SourceRefs[0].EndUnitID = "changed"
input[0].Metadata["name"] = "changed"
input.Payload.Content[0] = '['
input.Payload.Metadata["name"] = "changed"
got := result.Candidates[0]
if got.Index != 1 {
t.Fatalf("Index = %d, want 1", got.Index)
if string(result.Output.Payload.Content) != `{"name":"original"}` {
t.Fatalf("content changed after input mutation: %s", result.Output.Payload.Content)
}
if string(got.Payload) != `{"name":"original"}` {
t.Fatalf("Payload = %s, want original payload", got.Payload)
}
if got.SourceRefs[0].EndUnitID != "u1" {
t.Fatalf("SourceRefs = %#v, want original source ref", got.SourceRefs)
}
if got.Metadata["name"] != "original" {
t.Fatalf("Metadata = %#v, want original metadata", got.Metadata)
if result.Output.Payload.Metadata["name"] != "original" {
t.Fatalf("metadata changed after input mutation: %#v", result.Output.Payload.Metadata)
}
}
func TestNormalizeHandlesEmptyInput(t *testing.T) {
result, err := New().Normalize(context.Background(), contracts.NormalizeRequest{})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if len(result.Candidates) != 0 {
t.Fatalf("len(Candidates) = %d, want 0", len(result.Candidates))
}
if len(result.Warnings) != 0 {
t.Fatalf("Warnings = %#v, want none", result.Warnings)
}
}
func candidate(index int, name string) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: index,
ExtractorKey: "generic-extractor",
ArtifactType: "generic-artifact",
SchemaVersion: "v1",
Payload: json.RawMessage(`{"name":"` + name + `"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
},
Metadata: map[string]any{
"name": name,
func mergeOutput(content string) contracts.MergeOutput {
return contracts.MergeOutput{
LaneID: "events",
MergerKey: "merge",
SourceID: "source-1",
Schema: contracts.ResponseSchema{ID: "schema-id", Name: "schema-name", Version: "v1"},
Payload: contracts.RawPayload{
Content: []byte(content),
MediaType: "application/json",
Metadata: map[string]any{"name": "original"},
},
}
}
func candidateNames(candidates []artifacts.ArtifactCandidate) []string {
names := make([]string, 0, len(candidates))
for _, candidate := range candidates {
names = append(names, candidate.Metadata["name"].(string))
}
return names
}

163
internal/modules/output/json/encoder.go
View File

@@ -4,12 +4,11 @@ import (
"context"
stdjson "encoding/json"
"fmt"
"mime"
"regexp"
"sort"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
@@ -18,7 +17,7 @@ const Key = "json"
const contentTypeJSON = "application/json"
var safeArtifactFileChar = regexp.MustCompile(`[^A-Za-z0-9._-]`)
var safeOutputFileChar = regexp.MustCompile(`[^A-Za-z0-9._-]`)
var _ contracts.OutputEncoder = (*Encoder)(nil)
@@ -67,23 +66,23 @@ func Register(registry *pipeline.OutputEncoderRegistry) error {
type indexFile struct {
ManifestFile string `json:"manifest_file"`
ArtifactFiles []artifactFileIndex `json:"artifact_files"`
OutputFiles []outputFileIndex `json:"output_files"`
RejectedFile string `json:"rejected_file"`
WarningsFile string `json:"warnings_file"`
}
type artifactFileIndex struct {
ArtifactType string `json:"artifact_type"`
type outputFileIndex struct {
LaneID string `json:"lane_id"`
MediaType string `json:"media_type,omitempty"`
File string `json:"file"`
}
type artifactFile struct {
ArtifactType string `json:"artifact_type"`
Artifacts []artifacts.Artifact `json:"artifacts"`
ModuleKey string `json:"module_key,omitempty"`
SchemaID string `json:"schema_id,omitempty"`
SchemaName string `json:"schema_name,omitempty"`
SchemaVer string `json:"schema_version,omitempty"`
}
type rejectedFile struct {
Rejected []artifacts.RejectedArtifact `json:"rejected"`
Rejected []contracts.RejectedOutput `json:"rejected"`
}
type warningsFile struct {
@@ -91,43 +90,39 @@ type warningsFile struct {
}
func logicalFiles(req contracts.OutputRequest) ([]contracts.OutputFile, error) {
artifactsByType := make(map[string][]artifacts.Artifact)
for _, artifact := range req.Approved {
artifactsByType[artifact.ArtifactType] = append(artifactsByType[artifact.ArtifactType], cloneArtifact(artifact))
}
outputs := cloneNormalizeOutputs(req.NormalizeOutputs)
sort.SliceStable(outputs, func(i, j int) bool {
return outputs[i].LaneID < outputs[j].LaneID
})
artifactTypes := make([]string, 0, len(artifactsByType))
for artifactType := range artifactsByType {
artifactTypes = append(artifactTypes, artifactType)
}
sort.Strings(artifactTypes)
artifactIndexes := make([]artifactFileIndex, 0, len(artifactTypes))
files := make([]contracts.OutputFile, 0, len(artifactTypes)+4)
outputIndexes := make([]outputFileIndex, 0, len(outputs))
files := make([]contracts.OutputFile, 0, len(outputs)+4)
manifestFile, err := jsonFile("manifest.json", req.Manifest)
if err != nil {
return nil, err
}
files = append(files, manifestFile)
usedArtifactFiles := make(map[string]string, len(artifactTypes))
for _, artifactType := range artifactTypes {
name, err := artifactFileName(artifactType)
usedOutputFiles := make(map[string]string, len(outputs))
for _, output := range outputs {
name, err := outputFileName(output.LaneID)
if err != nil {
return nil, err
}
if existingType, ok := usedArtifactFiles[name]; ok {
return nil, encoderErrorf("artifact types %q and %q produce duplicate output file %q", existingType, artifactType, name)
if existingLane, ok := usedOutputFiles[name]; ok {
return nil, encoderErrorf("lanes %q and %q produce duplicate output file %q", existingLane, output.LaneID, name)
}
usedArtifactFiles[name] = artifactType
artifactIndexes = append(artifactIndexes, artifactFileIndex{
ArtifactType: artifactType,
usedOutputFiles[name] = output.LaneID
outputIndexes = append(outputIndexes, outputFileIndex{
LaneID: output.LaneID,
MediaType: output.Payload.MediaType,
File: name,
ModuleKey: output.NormalizerKey,
SchemaID: output.Schema.ID,
SchemaName: output.Schema.Name,
SchemaVer: output.Schema.Version,
})
file, err := jsonFile(name, artifactFile{
ArtifactType: artifactType,
Artifacts: artifactsByType[artifactType],
})
file, err := rawOutputFile(name, output.Payload)
if err != nil {
return nil, err
}
@@ -136,7 +131,7 @@ func logicalFiles(req contracts.OutputRequest) ([]contracts.OutputFile, error) {
index := indexFile{
ManifestFile: "manifest.json",
ArtifactFiles: artifactIndexes,
OutputFiles: outputIndexes,
RejectedFile: "rejected.json",
WarningsFile: "warnings.json",
}
@@ -159,6 +154,41 @@ func logicalFiles(req contracts.OutputRequest) ([]contracts.OutputFile, error) {
return files, nil
}
func rawOutputFile(name string, payload contracts.RawPayload) (contracts.OutputFile, error) {
content := append([]byte(nil), payload.Content...)
if len(content) == 0 {
content = []byte("null")
}
mediaType := strings.TrimSpace(payload.MediaType)
if mediaType == "" {
mediaType = "application/octet-stream"
}
if !isJSONMediaType(mediaType) {
return contracts.OutputFile{}, encoderErrorf("normalized output %q has unsupported media type %q", name, mediaType)
}
var decoded any
if err := stdjson.Unmarshal(content, &decoded); err != nil {
return contracts.OutputFile{}, encoderErrorf("normalized output %q contains invalid JSON: %w", name, err)
}
pretty, err := marshalPretty(decoded)
if err != nil {
return contracts.OutputFile{}, err
}
return contracts.OutputFile{
Name: name,
ContentType: mediaType,
Bytes: pretty,
}, nil
}
func isJSONMediaType(mediaType string) bool {
base, _, err := mime.ParseMediaType(strings.TrimSpace(mediaType))
if err != nil {
base = strings.TrimSpace(mediaType)
}
return strings.EqualFold(base, contentTypeJSON)
}
func jsonFile(name string, value any) (contracts.OutputFile, error) {
data, err := marshalPretty(value)
if err != nil {
@@ -179,57 +209,46 @@ func marshalPretty(value any) ([]byte, error) {
return append(data, '\n'), nil
}
func artifactFileName(artifactType string) (string, error) {
sanitized := safeArtifactFileChar.ReplaceAllString(strings.TrimSpace(artifactType), "_")
func outputFileName(laneID string) (string, error) {
sanitized := safeOutputFileChar.ReplaceAllString(strings.TrimSpace(laneID), "_")
for strings.Contains(sanitized, "..") {
sanitized = strings.ReplaceAll(sanitized, "..", "__")
}
sanitized = strings.Trim(sanitized, "._")
if sanitized == "" {
return "", encoderErrorf("artifact type %q cannot produce a safe file name", artifactType)
return "", encoderErrorf("lane id %q cannot produce a safe file name", laneID)
}
return "artifacts/" + sanitized + ".json", nil
return "lanes/" + sanitized + ".json", nil
}
func cloneArtifact(artifact artifacts.Artifact) artifacts.Artifact {
return artifacts.Artifact{
ExtractorKey: artifact.ExtractorKey,
ArtifactType: artifact.ArtifactType,
SchemaVersion: artifact.SchemaVersion,
Payload: append(stdjson.RawMessage(nil), artifact.Payload...),
SourceRefs: append([]source.SourceRef(nil), artifact.SourceRefs...),
Metadata: cloneMetadata(artifact.Metadata),
func cloneNormalizeOutputs(outputs []contracts.NormalizeOutput) []contracts.NormalizeOutput {
if len(outputs) == 0 {
return nil
}
}
func cloneRejected(rejected []artifacts.RejectedArtifact) []artifacts.RejectedArtifact {
if len(rejected) == 0 {
return []artifacts.RejectedArtifact{}
}
out := make([]artifacts.RejectedArtifact, 0, len(rejected))
for _, item := range rejected {
out = append(out, artifacts.RejectedArtifact{
Candidate: cloneCandidate(item.Candidate),
ValidatorName: item.ValidatorName,
ReasonCode: item.ReasonCode,
Message: item.Message,
})
out := make([]contracts.NormalizeOutput, 0, len(outputs))
for _, output := range outputs {
output.Payload = cloneRawPayload(output.Payload)
out = append(out, output)
}
return out
}
func cloneCandidate(candidate artifacts.ArtifactCandidate) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: candidate.Index,
ExtractorKey: candidate.ExtractorKey,
ArtifactType: candidate.ArtifactType,
SchemaVersion: candidate.SchemaVersion,
Payload: append(stdjson.RawMessage(nil), candidate.Payload...),
SourceRefs: append([]source.SourceRef(nil), candidate.SourceRefs...),
Metadata: cloneMetadata(candidate.Metadata),
func cloneRawPayload(payload contracts.RawPayload) contracts.RawPayload {
return contracts.RawPayload{
Content: append([]byte(nil), payload.Content...),
MediaType: payload.MediaType,
Metadata: cloneMetadata(payload.Metadata),
Warnings: append([]contracts.Warning(nil), payload.Warnings...),
}
}
func cloneRejected(rejected []contracts.RejectedOutput) []contracts.RejectedOutput {
if len(rejected) == 0 {
return []contracts.RejectedOutput{}
}
return append([]contracts.RejectedOutput(nil), rejected...)
}
func cloneWarnings(warnings []contracts.Warning) []contracts.Warning {
if len(warnings) == 0 {
return []contracts.Warning{}

242
internal/modules/output/json/encoder_test.go
View File

@@ -8,7 +8,6 @@ import (
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
@@ -37,20 +36,24 @@ func TestModuleSpecAndRegister(t *testing.T) {
}
}
func TestEncodeReturnsLogicalFilesGroupedByArtifactType(t *testing.T) {
func TestEncodeReturnsLogicalFilesForNormalizedOutputs(t *testing.T) {
req := contracts.OutputRequest{
Manifest: artifacts.RunManifest{RunID: "run-1", PipelineID: "pipeline-1"},
Approved: []artifacts.Artifact{
artifact("dnd.spell-cast", "first"),
artifact("notes/item", "item"),
artifact("dnd.spell-cast", "second"),
NormalizeOutputs: []contracts.NormalizeOutput{
normalizeOutput("spells", `{"spell_casts":[{"spell":"Cure Wounds"}]}`),
normalizeOutput("notes/items", `{"items":[{"name":"Torch"}]}`),
},
Rejected: []artifacts.RejectedArtifact{
Rejected: []contracts.RejectedOutput{
{
Candidate: candidate("bad type", "bad"),
ValidatorName: "validator",
Stage: "extract",
LaneID: "spells",
ModuleKey: "dnd/spells",
ChunkID: "chunk-1",
ChunkIndex: 1,
ReasonCode: "invalid",
Message: "not accepted",
AttemptCount: 1,
ValidatorName: "validator",
},
},
Warnings: []contracts.Warning{{ReasonCode: "warning", Message: "check source"}},
@@ -62,9 +65,9 @@ func TestEncodeReturnsLogicalFilesGroupedByArtifactType(t *testing.T) {
}
wantNames := []string{
"artifacts/dnd.spell-cast.json",
"artifacts/notes_item.json",
"index.json",
"lanes/notes_items.json",
"lanes/spells.json",
"manifest.json",
"rejected.json",
"warnings.json",
@@ -72,41 +75,36 @@ func TestEncodeReturnsLogicalFilesGroupedByArtifactType(t *testing.T) {
if got := outputFileNames(result.Files); !reflect.DeepEqual(got, wantNames) {
t.Fatalf("file names = %#v, want %#v", got, wantNames)
}
for _, file := range result.Files {
if file.ContentType != contentTypeJSON {
t.Fatalf("%s ContentType = %q, want %q", file.Name, file.ContentType, contentTypeJSON)
}
if !strings.HasSuffix(string(file.Bytes), "\n") {
t.Fatalf("%s does not end with newline: %q", file.Name, string(file.Bytes))
}
if !stdjson.Valid(file.Bytes) {
if file.ContentType == contentTypeJSON && !stdjson.Valid(file.Bytes) {
t.Fatalf("%s has invalid JSON: %s", file.Name, file.Bytes)
}
}
spellFile := decodeObject(t, fileBytes(t, result.Files, "artifacts/dnd.spell-cast.json"))
if spellFile["artifact_type"] != "dnd.spell-cast" {
t.Fatalf("artifact_type = %#v, want dnd.spell-cast", spellFile["artifact_type"])
}
spells := spellFile["artifacts"].([]any)
if len(spells) != 2 {
t.Fatalf("len(spells) = %d, want 2", len(spells))
}
firstPayload := spells[0].(map[string]any)["payload"].(map[string]any)
secondPayload := spells[1].(map[string]any)["payload"].(map[string]any)
if firstPayload["name"] != "first" || secondPayload["name"] != "second" {
t.Fatalf("spell order payloads = %#v then %#v, want runner order", firstPayload, secondPayload)
spells := decodeObject(t, fileBytes(t, result.Files, "lanes/spells.json"))
spellCasts := spells["spell_casts"].([]any)
if spellCasts[0].(map[string]any)["spell"] != "Cure Wounds" {
t.Fatalf("spells output = %#v, want raw normalized content", spells)
}
index := decodeObject(t, fileBytes(t, result.Files, "index.json"))
artifactFiles := index["artifact_files"].([]any)
if len(artifactFiles) != 2 {
t.Fatalf("len(index artifact_files) = %d, want 2", len(artifactFiles))
outputFiles := index["output_files"].([]any)
if len(outputFiles) != 2 {
t.Fatalf("len(index output_files) = %d, want 2", len(outputFiles))
}
firstIndex := artifactFiles[0].(map[string]any)
secondIndex := artifactFiles[1].(map[string]any)
if firstIndex["artifact_type"] != "dnd.spell-cast" || secondIndex["artifact_type"] != "notes/item" {
t.Fatalf("artifact_files = %#v, want sorted by artifact type", artifactFiles)
firstIndex := outputFiles[0].(map[string]any)
secondIndex := outputFiles[1].(map[string]any)
if firstIndex["lane_id"] != "notes/items" || secondIndex["lane_id"] != "spells" {
t.Fatalf("output_files = %#v, want sorted by lane id", outputFiles)
}
rejected := decodeObject(t, fileBytes(t, result.Files, "rejected.json"))
if got := rejected["rejected"].([]any); len(got) != 1 {
t.Fatalf("rejected = %#v, want one rejected output", got)
}
}
@@ -179,12 +177,64 @@ func TestEncodeIncludesManifestReferences(t *testing.T) {
}
}
func TestEncodeRejectsArtifactTypeWithoutSafeFileName(t *testing.T) {
func TestEncodeIncludesManifestRawOutputProvenance(t *testing.T) {
result, err := New().Encode(context.Background(), contracts.OutputRequest{
Manifest: artifacts.RunManifest{
RunID: "run-1",
NormalizedOutputs: []artifacts.NormalizedOutputManifest{
{
LaneID: "spells",
ModuleKey: "noop",
SourceID: "source-1",
MediaType: contentTypeJSON,
Schema: artifacts.OutputSchemaProvenance{
ID: "schema-id",
Name: "schema-name",
Version: "v1",
},
},
},
RejectedOutputs: []artifacts.RejectedOutputManifest{
{
Stage: "extract",
LaneID: "spells",
ModuleKey: "dnd/spells",
ChunkID: "chunk-0",
ReasonCode: "raw_output_rejected",
AttemptCount: 2,
},
},
},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
manifest := decodeObject(t, fileBytes(t, result.Files, "manifest.json"))
normalized := manifest["normalized_outputs"].([]any)
if len(normalized) != 1 {
t.Fatalf("normalized_outputs = %#v, want one entry", normalized)
}
normalizedEntry := normalized[0].(map[string]any)
if normalizedEntry["lane_id"] != "spells" || normalizedEntry["media_type"] != contentTypeJSON {
t.Fatalf("normalized output manifest = %#v, want lane and media type", normalizedEntry)
}
rejected := manifest["rejected_outputs"].([]any)
if len(rejected) != 1 {
t.Fatalf("rejected_outputs = %#v, want one entry", rejected)
}
rejectedEntry := rejected[0].(map[string]any)
if rejectedEntry["attempt_count"] != float64(2) || rejectedEntry["chunk_id"] != "chunk-0" {
t.Fatalf("rejected output manifest = %#v, want attempt count and chunk", rejectedEntry)
}
}
func TestEncodeRejectsLaneIDWithoutSafeFileName(t *testing.T) {
_, err := New().Encode(context.Background(), contracts.OutputRequest{
Approved: []artifacts.Artifact{artifact("///", "unsafe")},
NormalizeOutputs: []contracts.NormalizeOutput{normalizeOutput("///", `{"value":true}`)},
})
if err == nil {
t.Fatal("Encode() error = nil, want unsafe artifact type error")
t.Fatal("Encode() error = nil, want unsafe lane id error")
}
if !strings.Contains(err.Error(), "json output encoder") || !strings.Contains(err.Error(), "safe file name") {
t.Fatalf("Encode() error = %q, want safe file name context", err.Error())
@@ -193,22 +243,59 @@ func TestEncodeRejectsArtifactTypeWithoutSafeFileName(t *testing.T) {
func TestEncodeSanitizesParentPathSequences(t *testing.T) {
result, err := New().Encode(context.Background(), contracts.OutputRequest{
Approved: []artifacts.Artifact{artifact("dnd..spell.", "spell")},
NormalizeOutputs: []contracts.NormalizeOutput{normalizeOutput("dnd..spell.", `{"value":true}`)},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
if got := outputFileNames(result.Files); !containsString(got, "artifacts/dnd__spell.json") {
t.Fatalf("file names = %#v, want sanitized artifact filename", got)
if got := outputFileNames(result.Files); !containsString(got, "lanes/dnd__spell.json") {
t.Fatalf("file names = %#v, want sanitized output filename", got)
}
}
func TestEncodeRejectsInvalidJSONAndUnsupportedMediaTypes(t *testing.T) {
tests := []struct {
name string
output contracts.NormalizeOutput
want string
}{
{
name: "invalid JSON",
output: normalizeOutput("spells", `{"spell_casts":[`),
want: "invalid JSON",
},
{
name: "unsupported media type",
output: func() contracts.NormalizeOutput {
output := normalizeOutput("spells", `{"spell_casts":[]}`)
output.Payload.MediaType = "text/plain"
return output
}(),
want: "unsupported media type",
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
_, err := New().Encode(context.Background(), contracts.OutputRequest{
NormalizeOutputs: []contracts.NormalizeOutput{test.output},
})
if err == nil {
t.Fatal("Encode() error = nil, want error")
}
if !strings.Contains(err.Error(), test.want) {
t.Fatalf("Encode() error = %q, want %q", err.Error(), test.want)
}
})
}
}
func TestEncodeRejectsSanitizedFilenameCollisions(t *testing.T) {
_, err := New().Encode(context.Background(), contracts.OutputRequest{
Approved: []artifacts.Artifact{
artifact("a/b", "slash"),
artifact("a?b", "question"),
NormalizeOutputs: []contracts.NormalizeOutput{
normalizeOutput("a/b", `{"value":"slash"}`),
normalizeOutput("a?b", `{"value":"question"}`),
},
})
if err == nil {
@@ -222,16 +309,11 @@ func TestEncodeRejectsSanitizedFilenameCollisions(t *testing.T) {
func TestEncodeDoesNotMutateInputs(t *testing.T) {
req := contracts.OutputRequest{
Manifest: artifacts.RunManifest{RunID: "run-1"},
Approved: []artifacts.Artifact{
artifact("dnd.spell", "original"),
},
Rejected: []artifacts.RejectedArtifact{
{
Candidate: candidate("bad", "rejected"),
ValidatorName: "validator",
ReasonCode: "invalid",
Message: "not accepted",
NormalizeOutputs: []contracts.NormalizeOutput{
normalizeOutput("spells", `{"name":"original"}`),
},
Rejected: []contracts.RejectedOutput{
{Stage: "extract", LaneID: "spells", Message: "not accepted"},
},
Warnings: []contracts.Warning{{ReasonCode: "warning", Message: "message"}},
}
@@ -246,14 +328,13 @@ func TestEncodeDoesNotMutateInputs(t *testing.T) {
t.Fatalf("request mutated:\nbefore: %s\nafter: %s", before, after)
}
req.Approved[0].Payload[0] = '['
req.Approved[0].SourceRefs[0].StartUnitID = "changed"
req.Approved[0].Metadata["name"] = "changed"
req.Rejected[0].Candidate.Payload[0] = '['
req.NormalizeOutputs[0].Payload.Content[0] = '['
req.NormalizeOutputs[0].Payload.Metadata["name"] = "changed"
req.Rejected[0].Message = "changed"
req.Warnings[0].Message = "changed"
if !stdjson.Valid(fileBytes(t, result.Files, "artifacts/dnd.spell.json")) {
t.Fatal("artifact output changed after request mutation")
if !stdjson.Valid(fileBytes(t, result.Files, "lanes/spells.json")) {
t.Fatal("output changed after request mutation")
}
warnings := decodeObject(t, fileBytes(t, result.Files, "warnings.json"))
gotWarnings := warnings["warnings"].([]any)
@@ -262,9 +343,9 @@ func TestEncodeDoesNotMutateInputs(t *testing.T) {
}
}
func TestArtifactFilesDoNotContainWarnings(t *testing.T) {
func TestOutputFilesDoNotContainWarnings(t *testing.T) {
result, err := New().Encode(context.Background(), contracts.OutputRequest{
Approved: []artifacts.Artifact{artifact("dnd.spell", "spell")},
NormalizeOutputs: []contracts.NormalizeOutput{normalizeOutput("spells", `{"spell":"Shield"}`)},
Warnings: []contracts.Warning{
{ReasonCode: "pipeline-warning", Message: "warning"},
},
@@ -273,36 +354,27 @@ func TestArtifactFilesDoNotContainWarnings(t *testing.T) {
t.Fatalf("Encode() error = %v, want nil", err)
}
artifactFile := decodeObject(t, fileBytes(t, result.Files, "artifacts/dnd.spell.json"))
if _, ok := artifactFile["warnings"]; ok {
t.Fatalf("artifact file contains warnings: %#v", artifactFile)
outputFile := decodeObject(t, fileBytes(t, result.Files, "lanes/spells.json"))
if _, ok := outputFile["warnings"]; ok {
t.Fatalf("output file contains warnings: %#v", outputFile)
}
}
func artifact(artifactType, name string) artifacts.Artifact {
return artifacts.Artifact{
ExtractorKey: "extractor",
ArtifactType: artifactType,
SchemaVersion: "v1",
Payload: stdjson.RawMessage(`{"name":"` + name + `"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
func normalizeOutput(laneID string, content string) contracts.NormalizeOutput {
return contracts.NormalizeOutput{
LaneID: laneID,
NormalizerKey: "noop",
SourceID: "source-1",
Schema: contracts.ResponseSchema{
ID: "schema-id",
Name: "schema-name",
Version: "v1",
},
Metadata: map[string]any{"name": name},
}
}
func candidate(artifactType, name string) artifacts.ArtifactCandidate {
return artifacts.ArtifactCandidate{
Index: 1,
ExtractorKey: "extractor",
ArtifactType: artifactType,
SchemaVersion: "v1",
Payload: stdjson.RawMessage(`{"name":"` + name + `"}`),
SourceRefs: []source.SourceRef{
{SourceID: "source-1", StartUnitID: "u1", EndUnitID: "u1"},
Payload: contracts.RawPayload{
Content: []byte(content),
MediaType: contentTypeJSON,
Metadata: map[string]any{"name": laneID},
},
Metadata: map[string]any{"name": name},
}
}

95
internal/modules/sharedassets/dnd/unit_refs.go
View File

@@ -11,9 +11,8 @@ import (
)
type UnitRef struct {
value string
value int
fromNumber bool
number int
}
type SourceRefResponse struct {
@@ -23,19 +22,22 @@ type SourceRefResponse struct {
}
func UnitRefFromString(value string) UnitRef {
return UnitRef{value: value}
parsed, _ := parseUnitRefNumber(value)
return UnitRef{value: parsed}
}
func UnitRefFromInt(value int) UnitRef {
return UnitRef{
value: strconv.Itoa(value),
value: value,
fromNumber: true,
number: value,
}
}
func (ref UnitRef) String() string {
return ref.value
if ref.value == 0 {
return ""
}
return strconv.Itoa(ref.value)
}
func (ref *UnitRef) UnmarshalJSON(raw []byte) error {
@@ -48,13 +50,17 @@ func (ref *UnitRef) UnmarshalJSON(raw []byte) error {
if err := json.Unmarshal(raw, &value); err != nil {
return err
}
*ref = UnitRefFromString(value)
number, err := parseUnitRefNumber(value)
if err != nil {
return err
}
*ref = UnitRef{value: number}
return nil
}
number, err := strconv.Atoi(string(raw))
number, err := parseUnitRefNumber(string(raw))
if err != nil {
return fmt.Errorf("unit ref must be a string or integer")
return err
}
*ref = UnitRefFromInt(number)
return nil
@@ -62,72 +68,47 @@ func (ref *UnitRef) UnmarshalJSON(raw []byte) error {
func (ref UnitRef) MarshalJSON() ([]byte, error) {
if ref.fromNumber {
return []byte(strconv.Itoa(ref.number)), nil
return []byte(strconv.Itoa(ref.value)), nil
}
return json.Marshal(ref.value)
return json.Marshal(ref.String())
}
func ResolveUnitID(doc *source.SourceDocument, field string, ref UnitRef) (string, error) {
value := strings.TrimSpace(ref.value)
if value == "" {
return "", fmt.Errorf("%s must not be empty", field)
func ResolveUnitID(doc *source.SourceDocument, field string, ref UnitRef) (int, error) {
if ref.value <= 0 {
return 0, fmt.Errorf("%s must be positive", field)
}
if id, ok := canonicalUnitID(doc, value); ok {
return id, nil
if _, ok := source.UnitIndex(doc, ref.value); !ok {
return 0, fmt.Errorf("%s %d was not found", field, ref.value)
}
if number, ok := unitNumber(value); ok {
if id, ok := unitIDByNumber(doc, number); ok {
return id, nil
}
return "", fmt.Errorf("%s %d was not found as a source-unit ID or 1-based unit number", field, number)
}
return "", fmt.Errorf("%s %q was not found", field, value)
return ref.value, nil
}
func SourceRefCandidate(doc *source.SourceDocument, ref SourceRefResponse) source.SourceRef {
return source.SourceRef{
SourceID: strings.TrimSpace(ref.SourceID),
StartUnitID: unitIDCandidate(doc, ref.StartUnitID),
EndUnitID: unitIDCandidate(doc, ref.EndUnitID),
StartUnitID: unitIDCandidate(ref.StartUnitID),
EndUnitID: unitIDCandidate(ref.EndUnitID),
}
}
func unitIDCandidate(doc *source.SourceDocument, ref UnitRef) string {
value := strings.TrimSpace(ref.value)
if id, ok := canonicalUnitID(doc, value); ok {
return id
}
if number, ok := unitNumber(value); ok {
if id, ok := unitIDByNumber(doc, number); ok {
return id
}
}
return value
func unitIDCandidate(ref UnitRef) int {
return ref.value
}
func canonicalUnitID(doc *source.SourceDocument, value string) (string, bool) {
if doc == nil {
return "", false
func parseUnitRefNumber(value string) (int, error) {
trimmed := strings.TrimSpace(value)
if trimmed == "" {
return 0, fmt.Errorf("unit ref must not be empty")
}
for _, unit := range doc.Units {
if unit.ID == value {
return unit.ID, true
if trimmed != value {
return 0, fmt.Errorf("unit ref must not contain leading or trailing whitespace")
}
}
return "", false
}
func unitIDByNumber(doc *source.SourceDocument, number int) (string, bool) {
if doc == nil || number < 1 || number > len(doc.Units) {
return "", false
}
return doc.Units[number-1].ID, true
}
func unitNumber(value string) (int, bool) {
number, err := strconv.Atoi(value)
if err != nil {
return 0, false
return 0, fmt.Errorf("unit ref must be an integer")
}
return number, true
if number <= 0 {
return 0, fmt.Errorf("unit ref must be positive")
}
return number, nil
}

46
internal/modules/sharedassets/dnd/unit_refs_test.go
View File

@@ -8,7 +8,7 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
)
func TestUnitRefUnmarshalAcceptsIntegerAndString(t *testing.T) {
func TestUnitRefUnmarshalAcceptsIntegerAndNumericString(t *testing.T) {
var integerRef UnitRef
if err := json.Unmarshal([]byte(`12`), &integerRef); err != nil {
t.Fatalf("Unmarshal(integer) error = %v, want nil", err)
@@ -18,55 +18,49 @@ func TestUnitRefUnmarshalAcceptsIntegerAndString(t *testing.T) {
}
var stringRef UnitRef
if err := json.Unmarshal([]byte(`"seg-001"`), &stringRef); err != nil {
if err := json.Unmarshal([]byte(`"12"`), &stringRef); err != nil {
t.Fatalf("Unmarshal(string) error = %v, want nil", err)
}
if got := stringRef.String(); got != "seg-001" {
t.Fatalf("string ref = %q, want seg-001", got)
if got := stringRef.String(); got != "12" {
t.Fatalf("string ref = %q, want 12", got)
}
}
func TestUnitRefUnmarshalRejectsNonIntegerTypes(t *testing.T) {
for _, raw := range []string{`true`, `null`, `1.5`, `{}`} {
func TestUnitRefUnmarshalRejectsNonIntegerValues(t *testing.T) {
for _, raw := range []string{`true`, `null`, `1.5`, `{}`, `"seg-001"`, `" 1 "`, `0`, `-1`} {
t.Run(raw, func(t *testing.T) {
var ref UnitRef
err := json.Unmarshal([]byte(raw), &ref)
if err == nil {
t.Fatal("Unmarshal() error = nil, want error")
}
if !strings.Contains(err.Error(), "string or integer") {
t.Fatalf("Unmarshal() error = %q, want type context", err.Error())
}
})
}
}
func TestResolveUnitIDPrefersExactSourceUnitID(t *testing.T) {
doc := unitRefSourceDocument("2", "10")
func TestResolveUnitIDReturnsExistingIntegerSourceUnitID(t *testing.T) {
doc := unitRefSourceDocument(2, 10)
got, err := ResolveUnitID(doc, "start_unit_id", UnitRefFromInt(2))
if err != nil {
t.Fatalf("ResolveUnitID() error = %v, want nil", err)
}
if got != "2" {
t.Fatalf("ResolveUnitID() = %q, want exact source unit ID", got)
if got != 2 {
t.Fatalf("ResolveUnitID() = %d, want exact source unit ID", got)
}
}
func TestResolveUnitIDFallsBackToOneBasedUnitNumber(t *testing.T) {
doc := unitRefSourceDocument("seg-001", "seg-002")
func TestResolveUnitIDDoesNotFallbackToOneBasedUnitNumber(t *testing.T) {
doc := unitRefSourceDocument(10, 20)
got, err := ResolveUnitID(doc, "end_unit_id", UnitRefFromInt(2))
if err != nil {
t.Fatalf("ResolveUnitID() error = %v, want nil", err)
}
if got != "seg-002" {
t.Fatalf("ResolveUnitID() = %q, want second source unit ID", got)
_, err := ResolveUnitID(doc, "end_unit_id", UnitRefFromInt(2))
if err == nil {
t.Fatal("ResolveUnitID() error = nil, want missing source-unit ID")
}
}
func TestResolveUnitIDRejectsMissingUnit(t *testing.T) {
doc := unitRefSourceDocument("seg-001")
doc := unitRefSourceDocument(1)
_, err := ResolveUnitID(doc, "start_unit_id", UnitRefFromInt(9))
if err == nil {
@@ -78,14 +72,14 @@ func TestResolveUnitIDRejectsMissingUnit(t *testing.T) {
}
func TestSourceRefCandidateCanonicalizesValidRefsAndPreservesInvalidRefs(t *testing.T) {
doc := unitRefSourceDocument("seg-001", "seg-002")
doc := unitRefSourceDocument(1, 2)
valid := SourceRefCandidate(doc, SourceRefResponse{
SourceID: " session-alpha ",
StartUnitID: UnitRefFromInt(1),
EndUnitID: UnitRefFromInt(2),
})
if valid != (source.SourceRef{SourceID: "session-alpha", StartUnitID: "seg-001", EndUnitID: "seg-002"}) {
if valid != (source.SourceRef{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 2}) {
t.Fatalf("valid candidate = %#v, want canonical source ref", valid)
}
@@ -94,12 +88,12 @@ func TestSourceRefCandidateCanonicalizesValidRefsAndPreservesInvalidRefs(t *test
StartUnitID: UnitRefFromInt(9),
EndUnitID: UnitRefFromString("missing"),
})
if invalid != (source.SourceRef{SourceID: "session-alpha", StartUnitID: "9", EndUnitID: "missing"}) {
if invalid != (source.SourceRef{SourceID: "session-alpha", StartUnitID: 9, EndUnitID: 0}) {
t.Fatalf("invalid candidate = %#v, want unresolved values for validator", invalid)
}
}
func unitRefSourceDocument(ids ...string) *source.SourceDocument {
func unitRefSourceDocument(ids ...int) *source.SourceDocument {
doc := &source.SourceDocument{
ID: "session-alpha",
Kind: "transcript",