Compare commits

..

18 Commits

Author SHA1 Message Date
3013ee044d Finalize bounded typed pipeline implementation 2026-07-17 08:49:08 +00:00
adfd3bd052 Add bounded concurrent pipeline execution 2026-07-17 08:37:52 +00:00
4023c66508 Add extract worker configuration controls 2026-07-17 08:21:24 +00:00
adfe3825ee Remove legacy raw pipeline contracts 2026-07-17 08:13:08 +00:00
814fcdc6ba Serialize typed artifacts at durable boundaries 2026-07-17 07:33:49 +00:00
66de1a5520 Run D&D spell lanes through typed artifacts 2026-07-17 07:19:58 +00:00
52e6b31408 Add typed spell validation strategies 2026-07-17 07:02:30 +00:00
142ba36695 Introduce typed D&D spell artifacts 2026-07-17 06:50:08 +00:00
b949e9bbc0 Construct universal modules with decoded options 2026-07-17 06:30:56 +00:00
ce3a07512f Prepare pipelines before source execution 2026-07-17 06:18:46 +00:00
1c84d19e5f Add type-safe artifact lane resolution 2026-07-17 05:57:31 +00:00
fc1b57bde2 Add typed artifact codec foundation 2026-07-17 05:41:20 +00:00
075888c97f Move chunks into the canonical source model 2026-07-17 05:32:11 +00:00
40709e4ad8 Add canonical source unit provenance 2026-07-17 05:19:57 +00:00
15c369c509 Organize D&D extensions by domain 2026-07-17 05:05:48 +00:00
a81b9f1e1f Organize generic and Seriatim modules by domain 2026-07-17 04:57:47 +00:00
0327659355 Compose production modules through family registrars 2026-07-17 04:53:09 +00:00
c99bad19ae Add domain pipeline compatibility baselines 2026-07-17 04:45:01 +00:00
199 changed files with 10173 additions and 10059 deletions

View File

@@ -1,6 +1,6 @@
# ADR-0002: Linear pipes-and-filters pipeline, not a general DAG
**Status:** Proposed
**Status:** Accepted
**Date:** 2026-07-13
## Context

View File

@@ -1,6 +1,6 @@
# ADR-0003: Strongly typed stage interfaces with a two-zone data model
**Status:** Proposed
**Status:** Accepted
**Date:** 2026-07-13
## Context

View File

@@ -1,6 +1,6 @@
# ADR-0004: Package modules by domain, not by stage
**Status:** Proposed
**Status:** Accepted
**Date:** 2026-07-13
## Context

View File

@@ -41,6 +41,7 @@ rejected; execution profiles now come from Scriptorium.
Built-in defaults:
- `concurrency.total_llm`: `1`
- `concurrency.stage_workers.extract`: effective `concurrency.total_llm`
- `diagnostics.work_dir`: `/tmp/notarius`
- `diagnostics.retention`: `auto`
- `workspace.directory`: unset
@@ -88,6 +89,7 @@ These environment variables are applied after the config file:
- `NOTARIUS_CONFIG`: config discovery path.
- `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency.
- `NOTARIUS_STAGE_WORKERS_EXTRACT`: integer extract worker limit.
- `NOTARIUS_WORKSPACE_DIR`: workspace directory.
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`: boolean diagnostics enablement.
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`: workspace diagnostics retention
@@ -107,6 +109,25 @@ The removed `NOTARIUS_LLM_DEFAULT_*` variables are not read. Configure provider
endpoint, model, and credential environment variable names through Scriptorium
profiles.
## Concurrency
`concurrency` fields:
- `total_llm`: positive integer ceiling on concurrent provider calls.
- `stage_workers`: optional map of framework worker limits. The only supported
key is `extract`.
`stage_workers.extract` defaults to the effective `total_llm` value after file
and environment precedence. It must be between `1` and `total_llm`, inclusive.
Unknown or empty stage-worker keys are rejected. The environment override
`NOTARIUS_STAGE_WORKERS_EXTRACT` takes precedence over the file value, as does
`NOTARIUS_TOTAL_LLM_CONCURRENCY` for the global ceiling.
The worker value is present in effective and redacted configuration. It bounds
the fixed run-wide extract pool and its bounded dispatch queue. Extract jobs are
submitted by source chunk and then resolved lane; `total_llm` independently
bounds actual provider calls made by extracts, retries, and validators.
## Pipelines
A pipeline selects implementations for the fixed workflow defined by
@@ -236,6 +257,10 @@ Binding fields:
chain; set a non-empty list to use exactly those validators in configured
order.
During resolution, each selected module's registered option validator runs.
Production input, chunk, and output bindings reject unknown or invalid options
with the affected binding context.
Validator bindings use the same shorthand or object module-binding form, but
only these fields are supported:
@@ -258,9 +283,9 @@ production validators do not call the LLM and must not set `llm_profile`.
| input | `seriatim` | Reads Seriatim transcript JSON. |
| chunk | `generic` | Splits source units into ordered chunks. |
| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. |
| extract | `dnd/spells` | Extracts D&D spell raw outputs. |
| merge | `appendorder` | Merges JSON raw extract outputs in chunk order. |
| normalize | `noop` | Passes merged raw outputs through unchanged. |
| extract | `dnd/spells` | Extracts typed D&D spell-list artifacts. |
| merge | `appendorder` | Combines typed artifacts in chunk order. |
| normalize | `noop` | Passes merged typed artifacts through unchanged. |
| output | `json` | Produces JSON output files for normalized `application/json` lanes. |
## Implemented Production Validators
@@ -271,7 +296,7 @@ production validators do not call the LLM and must not set `llm_profile`.
| `generic/always_reject` | deterministic | Rejects returned module output with reason `always_reject`. |
| `generic/valid_json` | deterministic | Rejects payloads that are not syntactically valid JSON. |
| `generic/valid_json_schema` | deterministic | Rejects invalid JSON or JSON that does not conform to the module response schema. |
| `extract/dnd/spells/shape` | deterministic | Rejects malformed D&D spell-cast JSON payloads. |
| `extract/dnd/spells/shape` | deterministic | Rejects malformed D&D spell-list artifacts. |
| `extract/dnd/spells/source_refs` | deterministic | Rejects missing or invalid D&D spell source references. |
| `extract/dnd/spells/source_relatedness` | deterministic | Emits warnings when a spell name is not found near its cited source text. |
@@ -362,6 +387,8 @@ Configuration validation checks:
- mutually exclusive `scriptorium.profile_dir` and `scriptorium.profile_file`;
- non-empty, non-duplicated IDs after trimming;
- positive global LLM concurrency;
- supported stage-worker keys and an effective extract worker count in the
inclusive range `1..concurrency.total_llm`;
- supported diagnostics retention and non-empty work directory;
- stale removed fields such as `llm_profiles`.

View File

@@ -1,11 +1,12 @@
# D&D Spell Raw Output
# D&D Spell Artifact
This document is the durable raw output contract for the production D&D spell
extractor. Selectable extractor keys are cataloged in
This document is the durable serialized artifact contract for the production
D&D spell extractor. Selectable extractor keys are cataloged in
[Configuration](../config.md#implemented-production-modules).
## Identity
- Artifact kind: `dnd/spell-list`
- Prompt ID: `dnd.spells`
- Response schema key: `dnd_spells`
- Response schema ID: `notarius.dnd.spells`
@@ -13,6 +14,12 @@ extractor. Selectable extractor keys are cataloged in
- Response schema version: `v1`
- Media type: `application/json`
The durable JSON Schema is owned by the D&D spell artifact codec. The
extractor's private LLM response schema is a separate transport contract: its
source-reference objects omit `source_id`, which the extractor assigns while
mapping the response to the canonical artifact. The LLM DTO and transport
schema are not part of this durable contract.
The output contains canonical spell casts derived from transcript evidence.
Source IDs are assigned from the input identity; source-unit ranges identify
the evidence location.

View File

@@ -15,7 +15,7 @@ The encoder writes:
- `index.json`
- `manifest.json`
- `lanes/<lane-id>.json`, one file per normalized raw lane output
- `lanes/<lane-id>.json`, one file per normalized serialized artifact
- `rejected.json`
- `warnings.json`
@@ -112,8 +112,8 @@ Reference `stage` is `chunk`, `extract`, `merge`, or `normalize`. `lane_id` is
omitted for chunk references and present for extract, merge, and normalize
references.
`validation_status` is `approved` when no raw outputs were rejected and
`rejected` when one or more raw outputs were rejected.
`validation_status` is `approved` when no outputs were rejected and `rejected`
when one or more outputs were rejected.
`validator_chains` records the resolved validator chain for each validation
point. Entries include stage, lane ID when applicable, module key, and validators
@@ -131,12 +131,13 @@ message, attempt count, and optional diagnostic artifact path.
## Output Payload Files
Each normalized raw output is written to `lanes/<sanitized-lane-id>.json`.
The JSON output encoder accepts only `application/json` normalized outputs. The
file contains the raw JSON payload pretty-printed.
Each normalized serialized artifact is written to
`lanes/<sanitized-lane-id>.json`. The JSON output encoder is domain-neutral and
accepts only artifacts whose codec media type is `application/json`. The file
contains the codec-owned JSON bytes pretty-printed.
The schema of each lane payload is owned by that artifact contract. For the
current D&D spell lane, see [D&D Spell Raw Output](dnd-spell-artifacts.md).
current D&D spell lane, see [D&D Spell Artifact](dnd-spell-artifacts.md).
## `rejected.json`
@@ -148,7 +149,7 @@ Shape:
}
```
When raw output validation rejects an output, each entry contains `stage` and
When output validation rejects an output, each entry contains `stage` and
`message`. It includes `lane_id`, `module_key`, `chunk_id`, `chunk_index`,
`validator_name`, `reason_code`, `attempt_count`, and
`diagnostic_artifact_path` when applicable.

View File

@@ -52,9 +52,14 @@ Notarius identifies the parsed source in this order:
2. `metadata.source_id`, when it is a non-empty string after trimming;
3. `seriatim:<first-16-hex-chars-of-raw-sha256>`.
The source digest recorded in output provenance is `sha256:<hex>` of the exact
raw input bytes. Segment IDs become the unit IDs used by artifact source
references.
The exact raw input SHA-256 remains the basis of the fallback source ID. The
source digest recorded in output provenance is instead the SHA-256 of the
canonical generic source document, excluding the digest field itself. It covers
the derived source identity, document kind and format, ordered units and their
self-references, and accepted metadata. Segment IDs become the unit IDs used by
artifact source references; each produced unit carries a self-reference whose
source ID is the derived document ID and whose start and end IDs both equal the
segment ID.
## Compatibility Limit

View File

@@ -28,12 +28,20 @@ without exposing Scriptorium types through stage contracts.
`internal/cli` constructs the production runtime by:
1. collecting embedded prompt and response-schema assets from production module
packages;
1. allocating the asset registry populated by the generic, Seriatim, and D&D
package-family registrars;
2. creating a `ScriptoriumClient` from the effective profile source;
3. attaching an `LLMProfileRecorder`;
4. creating a scheduler from the effective concurrency limit;
5. returning a `ScheduledClient` wrapper.
5. returning a `ScheduledClient` wrapper;
6. decorating that shared client before preparation when debug recording is
enabled; and
7. injecting that one shared client into complete pipeline preparation before
the source file is read or the runner is invoked.
The D&D scene chunker and spell extractor retain this injected client and use
it for every structured completion. Operation requests do not carry an LLM
client.
The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
and validator bindings. It prepares a small internal check prompt for each ID so
@@ -75,12 +83,20 @@ cancelled waiter that has already received a permit releases it.
defers release on every result path. The effective limit and default are
configuration facts in [Configuration](../config.md#defaults).
This provider-call ceiling is independent of the pipeline's extract worker
limit. Concurrent lanes, retries, and validators all use the same scheduled
client, so increasing framework workers cannot exceed `total_llm`. Pipeline
dispatch and cancellation mechanics are documented in
[Pipeline Internals](pipeline.md#execution-flow).
## Prompt And Schema Assets
`AssetRegistry` combines caller-owned prompt filesystems under stable prefixes
and rejects invalid or conflicting registrations. Production module packages
register their own prompt and schema assets; generic framework code contains no
D&D prompt content.
D&D prompt content. `internal/framework/promptfs` provides the domain-neutral
filesystem composition helper used to combine module-owned files with shared
domain prompt fragments.
Schema helpers load embedded JSON Schema with identity and digest metadata,
return defensive copies, and expose a diagnostics map that omits schema bytes.

View File

@@ -1,8 +1,9 @@
# Module And Validator Internals
Production stage implementations live under `internal/modules`; production
validators live under `internal/validators`. The selectable keys, configuration
options, reference slots, and default validator chain are canonical in the
Production module and validator implementations live under their domain-first
trees in `internal/modules`.
The selectable keys, configuration options, reference slots, and default
validator chain are canonical in the
[module](../config.md#implemented-production-modules) and
[validator](../config.md#implemented-production-validators) catalogs in
Configuration.
@@ -12,10 +13,27 @@ Configuration.
A stage module package provides a stable key, constructor, contract
implementation, `ModuleSpec`, `Register`, and focused behavior and registration
tests. A validator package follows the same pattern with `ValidatorSpec` and the
validator registry.
validator registry. Package-family registrars compose those leaf registrations
into the production catalog and own family-level policy such as default
validator chains and prompt asset collection.
Production input, chunk, output, and D&D spell-extract packages register strict
option decoders and run-local builders. Preparation decodes their options into
implementation-owned values and injects dependencies. The spell extractor is
typed over the canonical D&D model. D&D validators, merge, and normalize use
typed variants; JSON representation validators use serialized requests; and
unconditional validators expose separate chunk and typed variants. The D&D
production registrar registers only the canonical typed spell implementations.
Prepared extractors, extract validators, and codecs may be reused concurrently
by the run-wide extract pool. Production implementations are immutable after
construction: they retain only typed options, immutable assets, or the shared
concurrency-safe LLM client. Implementations that introduce mutable state must
synchronize that state without creating a separate provider scheduler.
Specs expose capability and execution metadata without constructing an
implementation. Chunk, extract, merge, and normalize modules that accept
implementation. Registry entries separately expose option validation and
run-local construction. Chunk, extract, merge, and normalize modules that accept
auxiliary material declare identical reference slots from both
`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
that agreement. Runtime delivery uses the corresponding stage request's
@@ -23,10 +41,10 @@ that agreement. Runtime delivery uses the corresponding stage request's
LLM-backed extensions own their prompt definitions and response schemas under
package-local embedded assets. Shared filesystem composition belongs in
`internal/modules/sharedassets`; reusable D&D prompt fragments, reference
`internal/framework/promptfs`; reusable D&D prompt fragments, reference
declarations, prompt-input assembly, and source-unit helpers belong in
`internal/modules/sharedassets/dnd`. Stage contracts expose only Notarius
structured-completion types, not Scriptorium public types.
`internal/modules/dnd/shared`. Stage contracts expose only Notarius structured-
completion types, not Scriptorium public types.
Reference material may inform a module or prompt but must not become source
evidence. The resolver and materializer behavior is described in
@@ -34,12 +52,13 @@ evidence. The resolver and materializer behavior is described in
## Input Adapter
### `internal/modules/input/seriatim`
### `internal/modules/seriatim/input/transcript`
The adapter decodes the supported transcript JSON, selects the source identity,
computes the raw-input digest, validates segments, and maps each segment into a
generic source unit with speaker and timestamp metadata. Its spec advertises the
transcript capabilities consumed by D&D modules.
computes canonical source provenance, validates segments, and maps each segment
into a generic source unit with a self-reference plus speaker and timestamp
metadata. It accepts no module options. Its spec advertises the transcript
capabilities consumed by D&D modules.
Parsing is strict about required values and duplicate unit IDs but deliberately
ignores unrelated Seriatim fields. The external format and derived-identity
@@ -48,23 +67,29 @@ rules are defined in the
## Chunkers
### `internal/modules/chunk/generic`
### `internal/modules/generic/chunk/units`
The generic chunker validates the source document, walks units in configured
windows, clones each selected unit, and emits deterministic ordered chunk IDs.
Overlap changes the next window start but never reorders units. It records the
first and last unit and unit count in chunk metadata.
first and last unit and unit count in chunk metadata, and derives the chunk's
canonical source reference from those unit references.
The accepted options and defaults are defined in
[Configuration](../config.md#implemented-production-modules). Generic
framework validation canonicalizes the returned unit slices before extraction.
The chunker decodes its options during construction and retains only the typed
window settings used by `Chunk`.
### `internal/modules/chunk/dnd/scenes`
### `internal/modules/dnd/chunk/scenes`
The scene chunker prepares a structured Scriptorium request from the full
transcript, session, and optional D&D reference inputs. It validates the model's
scene boundaries against source-unit IDs and converts them into deterministic
chunks.
chunks with canonical source references spanning each scene's units.
Preparation injects the shared structured LLM client into the chunker; `Chunk`
supplies only the run-specific profile, session, source, references, and
metadata.
Scene validation requires sequential, contiguous, non-overlapping coverage from
the first source unit through the last. Each chunk contains JSON scene content
@@ -79,40 +104,42 @@ file types remain canonical in [Configuration](../config.md).
## Extractor
### `internal/modules/extract/dnd/spells`
### `internal/modules/dnd/extract/spells`
The spell extractor prepares a structured request from one chunk, the
chunk-scoped source input, the session, and optional D&D reference inputs. It
decodes the model response, assigns the generic source identity to every source
reference, canonicalizes duplicate references, orders spell casts by their
earliest cited unit, and returns raw JSON plus response-schema provenance.
earliest cited unit, and returns `dnd.SpellList`.
The package owns its embedded prompt, response schemas, and prompt/schema
manifest metadata. Shared D&D helpers keep prompt input names and source-unit
reference conversion consistent with the scene chunker. The extractor produces
raw output; production validators own approval policy.
The extractor owns its private model-response DTO, embedded prompt, LLM response
schema, strict option decoder, injected shared LLM client, and prompt/schema
manifest metadata. The separate `internal/modules/dnd/codec/spells` package
owns the durable schema and stable JSON representation for artifact kind
`dnd/spell-list`. The runner keeps the result typed through validators and later
stages, using the codec only for checkpoint, debug, and output boundaries.
Shared D&D helpers keep prompt input
names and source-unit reference conversion consistent with the scene chunker.
The durable payload and manifest metadata shapes are defined in the
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
## Merger And Normalizer
### `internal/modules/merge/appendorder`
### `internal/modules/generic/merge/appendorder`
The merger preserves extract-result order. It passes through one JSON result,
concatenates a common top-level array field across multiple JSON objects, and
otherwise emits an array of the decoded values. It rejects invalid JSON and
non-JSON media types, and it preserves compatible schema provenance.
The merger passes typed values to an injected combine function in framework
source-chunk order. The D&D registrar specializes it with a spell-list append
function.
### `internal/modules/normalize/noop`
### `internal/modules/generic/normalize/noop`
The normalizer defensively clones the accepted merge result, including payload
bytes, metadata, warnings, and schema provenance, without changing its logical
content.
The normalizer returns the merged domain value unchanged and is reusable for
any registered artifact type.
## Output Encoder
### `internal/modules/output/json`
### `internal/modules/generic/output/json`
The JSON encoder sorts normalized results by lane, derives collision-checked
safe logical names, pretty-prints JSON payloads, and assembles the logical index,
@@ -125,26 +152,26 @@ paths and schemas.
## Generic Validators
The unconditional accept and reject validators provide deterministic production
registrations used primarily for controlled composition and tests.
The generic validator implementations live under
`internal/modules/generic/validate`.
The JSON syntax validator uses `encoding/json` to reject malformed payloads. The
JSON Schema validator requires schema bytes on the validation request, parses
the instance and schema with `jsonschema`, and distinguishes payload rejection
from schema loading or compilation errors. Neither validator calls the LLM.
The unconditional accept and reject validators provide explicit chunk and
typed-artifact variants used primarily for controlled composition and tests.
The serialized JSON syntax validator uses `encoding/json` to reject malformed
representation bytes. The serialized JSON Schema validator requires schema
bytes, parses the instance and schema with `jsonschema`, and distinguishes
payload rejection from schema loading or compilation errors. The framework
serialized-validation request carries either canonical chunk bytes or artifact
codec bytes according to its target context. Neither validator calls the LLM.
## D&D Spell Validators
`internal/validators/extract/dnd/spells/spellpayload` provides strict decoding,
shape checks, source-reference candidates, and cited-text lookup shared by the
three validators.
The shape validator rejects malformed JSON, unknown fields, missing or empty
spell fields, and empty reference lists. The source-reference validator applies
generic source-reference validation to every cited range. The relatedness
validator approves structurally valid payloads but warns when a case-insensitive
spell name is absent from all cited source text. It leaves malformed payloads to
the earlier validators in the configured chain.
All three validators receive `dnd.SpellList` directly. The shape validator
rejects missing or empty spell fields and empty reference lists. The
source-reference validator applies generic source-reference validation to every
cited range. The relatedness validator warns when a case-insensitive spell name
is absent from all cited source text.
These validators are deterministic. Their selectable keys and production order
are defined in
@@ -154,10 +181,13 @@ payload rules are defined in the
## Production Registration
`internal/cli/catalog.go` builds the production registries, registers module and
validator constructors, installs default validator-chain mappings, and exposes
the matching catalog for resolution. It also collects prompt assets from
LLM-backed packages before constructing the production client.
The CLI allocates one complete framework registry set and one LLM asset
registry. It invokes `internal/modules/generic/register`,
`internal/modules/seriatim/register`, and `internal/modules/dnd/register` in
that order, then exposes the matching catalog for resolution. The generic and
Seriatim registrars own their production leaf registrations. The D&D registrar
owns D&D leaf registrations, the spell default-validator chain, and D&D
prompt/schema asset collection.
Framework packages must not import production extensions. Tests may compose
registries and catalogs directly with fakes.
@@ -170,8 +200,8 @@ When adding a production module or validator:
2. expose and test its spec, constructor, and registration function;
3. keep format or domain parsing inside the concrete package;
4. add package-owned prompt/schema assets when the extension is LLM-backed;
5. register it in `internal/cli/catalog.go` and add a default chain only when
production policy requires one;
5. register it through its package-family registrar and add a default chain
there only when production policy requires one;
6. add resolution and composition coverage for capabilities, options,
references, and validation behavior;
7. update the selectable-key catalog in [Configuration](../config.md), the
@@ -184,11 +214,13 @@ does not inventory implementations.
## Tests To Inspect
- Package-local `*_test.go` files under the module or validator being changed.
- `internal/framework/pipeline/registry_integration_test.go`: registry and spec
composition.
- `internal/framework/pipeline/typed_resolution_test.go`: typed registry, spec,
and heterogeneous artifact composition.
- `internal/framework/pipeline/default_modules_test.go`: framework binding
defaults.
- `internal/cli/run_test.go`: production catalog, config resolution, and
end-to-end CLI composition.
- `internal/modules/sharedassets/**/*_test.go`: shared prompt and reference
assembly.
- `internal/framework/promptfs/*_test.go` and
`internal/modules/dnd/shared/*_test.go`: shared prompt and reference assembly.
- `internal/modules/integration/*_test.go`: black-box composition across
production extension domains.

View File

@@ -14,15 +14,18 @@ collaborators, invokes `internal/framework/pipeline`, and places the logical
output files returned by the runner. Diagnostics, checkpoints, and debug
recorders are optional side-channel collaborators supplied at this boundary.
Pipeline execution is serial. Resolution produces a fixed ordered workflow and
a sorted set of artifact lanes before the runner constructs any stage module.
Resolution produces a fixed ordered workflow and a sorted set of artifact
lanes. Preparation constructs the complete module and validator set before the
runner receives source bytes. Source parsing and chunking are serial; extraction
uses a bounded run-wide worker pool, followed by serial per-lane merge and
normalize continuations that may overlap across lanes.
## Application Boundary
| Package | Implemented responsibility |
| --- | --- |
| `cmd/notarius` | Executable entry point and process exit delegation. |
| `internal/cli` | Command parsing, config discovery, production registration, prompt asset collection, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results. |
| `internal/cli` | Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results. |
## Core Packages
@@ -31,23 +34,42 @@ a sorted set of artifact lanes before the runner constructs any stage module.
| `internal/core/artifacts` | Run-manifest and provenance models. |
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
| `internal/core/source` | Generic source documents, units, references, lookup, and validation. |
| `internal/core/source` | Generic source documents, units, chunks, canonical references, lookup, validation, and deterministic source digests. |
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, checkpoint identity, and checkpoint manifest models. |
## Framework Packages
| Package | Implemented responsibility |
| --- | --- |
| `internal/framework/contracts` | Stage, validator, reference, output, and structured-completion interfaces and data types. |
| `internal/framework/pipeline` | Registries, profile resolution, capability checks, reference materialization, validator-chain resolution, retries, orchestration, warnings, and manifest population. |
| `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. |
| `internal/framework/pipeline` | Module and artifact-codec registries, option validation, profile resolution, capability checks, reference materialization, complete pipeline preparation, retries, orchestration, warnings, and manifest population. |
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. |
| `internal/framework/debug` | Workspace-backed framework and LLM debug recording. |
Framework contracts carry raw stage results between implementations. The
runner owns handoff provenance, validation sequencing, rejection handling,
checkpoint and debug boundaries, and final manifest assembly.
Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
serialized-validator, and
typed-validator interfaces. The runner owns handoff provenance, validation
sequencing, rejection handling, checkpoint and debug boundaries, and final
manifest assembly.
Artifact registries support heterogeneous typed extraction entries and
kind-specific merger, normalizer, and validator variants. Resolution derives a
lane's kind from its extractor, requires the matching codec, verifies exact Go
type equality across the lane, and records schema identity in the resolved lane
and pipeline digest. Registry entries carry separate option-validation and
run-local construction closures. Preparation injects shared dependencies and
constructs input, chunk, validators, ordered lanes, and output before source
parsing. Production modules use strict construction-time option decoding, and
LLM-backed modules retain the injected shared client. The D&D family registers
the canonical `dnd/spell-list` codec, typed spell extractor and validators, and
kind-specific generic merge and normalize strategies; generic JSON validators
use the serialized-validation contract. The runner executes lanes through
private exact-type-checked closures, coordinates extract results independently
of completion timing, and serializes artifacts only through their codec at
checkpoint, debug, and output boundaries.
## Production Extensions
@@ -58,24 +80,38 @@ Configuration. The implemented module packages are:
| Package | Implemented responsibility |
| --- | --- |
| `internal/modules/input/seriatim` | Parses the supported Seriatim transcript format into the generic source model. |
| `internal/modules/chunk/generic` | Splits ordered source units by unit count and overlap. |
| `internal/modules/chunk/dnd/scenes` | Produces contiguous D&D scene chunks from structured model output. |
| `internal/modules/extract/dnd/spells` | Produces source-grounded D&D spell-cast raw output. |
| `internal/modules/merge/appendorder` | Combines accepted extraction results in chunk order. |
| `internal/modules/normalize/noop` | Preserves accepted merged output. |
| `internal/modules/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
| `internal/modules/seriatim/input/transcript` | Parses the supported Seriatim transcript format into the generic source model. |
| `internal/modules/generic/chunk/units` | Splits ordered source units by unit count and overlap. |
| `internal/modules/dnd/chunk/scenes` | Produces contiguous D&D scene chunks from structured model output. |
| `internal/modules/dnd` | Owns the canonical D&D spell-list and spell-cast artifact types. |
| `internal/modules/dnd/codec/spells` | Strictly decodes and stably encodes the durable D&D spell-list representation. |
| `internal/modules/dnd/extract/spells` | Maps private structured model output to canonical source-grounded D&D spell lists. |
| `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. |
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
`internal/modules/sharedassets` composes shared prompt filesystems.
`internal/modules/sharedassets/dnd` owns reusable D&D prompt fragments,
`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
reference declarations, prompt input assembly, and source-unit reference
helpers.
helpers. Domain-neutral prompt filesystem composition lives in
`internal/framework/promptfs`.
Concrete validators live under `internal/validators`. Generic packages provide
Generic validators under `internal/modules/generic/validate` provide
unconditional test decisions, JSON syntax validation, and JSON Schema
validation. D&D spell packages provide shape, source-reference, and
source-relatedness decisions, with `spellpayload` holding their shared parser
and lookup helpers. Production chain composition is owned by `internal/cli`.
validation. D&D spell validators under `internal/modules/dnd/validate/spells`
consume the canonical spell-list type directly to provide shape,
source-reference, and source-relatedness decisions.
Production composition is grouped behind package-family registrars, and every
implemented production extension uses its domain-first tree:
| Package | Implemented responsibility |
| --- | --- |
| `internal/modules/generic/register` | Registers domain-neutral chunk, merge, normalize, output, and validator implementations. |
| `internal/modules/seriatim/register` | Registers the Seriatim input adapter. |
| `internal/modules/dnd/register` | Registers D&D modules, validators, default validator policy, and prompt/schema assets. |
The CLI allocates the framework registries and asset registry, then invokes
these registrars in generic, Seriatim, and D&D order.
Implementation details for all production extensions are in
[Module Internals](modules.md).

View File

@@ -6,8 +6,12 @@ Their fixed workflow and ownership boundaries are defined by
defaults, and selectable keys are defined in
[Configuration](../config.md#pipelines).
Pipeline execution is serial. Resolution fixes the selected lanes and all
stage bindings before the runner constructs stage implementations.
Resolution fixes the selected lanes and all stage bindings; preparation
constructs every selected implementation before the runner begins source work.
After serial input parsing and chunking, the runner dispatches extract work to
one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's
merge and normalize operations remain serial and may overlap other lanes once
all extracts for that lane are terminal.
## Resolution
@@ -20,9 +24,14 @@ calls `pipeline.ResolvePipeline`.
1. selects and sorts artifact lanes;
2. completes omitted bindings using the documented configuration defaults;
3. looks up each module and validator spec without constructing it;
4. checks required and provided capabilities in workflow order;
5. resolves target-aware reference bindings and validator chains;
6. calculates a digest over the resolved structure.
4. for a typed extractor, derives its artifact kind, requires the codec, and
selects exact-type merger, normalizer, and validator variants;
5. checks required and provided capabilities in workflow order;
6. resolves target-aware reference bindings and validator chains;
7. validates each selected module and validator option set through its registry
entry; and
8. calculates a digest over the resolved structure, including typed artifact
kind and schema identity.
Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
bindings, validator chains, reference targets, and the digest. It does not read
@@ -50,31 +59,76 @@ runtime sensitive-data handling belongs in [Operations](../operations.md).
## Registries And Specs
`pipeline.Registries` holds constructors used during execution.
`pipeline.Registries` holds option validators and run-local builders used during
resolution and preparation.
`pipeline.ModuleCatalog` exposes their specs during configuration validation and
resolution. Separate registries exist for every stage and for validators;
`ValidatorChainRegistry` stores production default-chain mappings.
`ValidatorChainRegistry` stores production default-chain mappings. Both
containers also carry an `ArtifactCodecRegistry`. Generic registration records
one codec per stable artifact kind, validates its schema metadata and JSON
Schema, retains the exact schema digest and Go type, and safely encodes or
decodes framework-erased values with typed errors on incompatibility.
Typed extractor entries are keyed by module key and declare one artifact kind.
Merger, normalizer, and typed-validator variants are keyed by module or
validator key plus artifact kind. Chunk and serialized validators occupy
separate target namespaces; serialized registrations declare whether they
support chunks, artifacts, or both. Duplicate variants and exact Go-type
mismatches are rejected deterministically.
Production composition registers the D&D spell-list codec and typed extractor,
matching typed merge, normalize, and semantic-validator variants, and
serialized JSON validators. Every artifact lane resolves through the typed
registries and a matching codec.
A `ModuleSpec` declares its stage plus required and provided capabilities.
Chunk, extract, merge, and normalize specs may also declare reference slots.
Registry implementations defensively copy spec metadata, reject duplicate keys,
and verify that a constructed implementation reports the registered key.
Builder registrations accept `ModuleDependencies` and cloned configuration
options through one `BuildRequest`. Builders decode those options and retain
typed values or injected dependencies in the constructed implementation.
Extractors declare their artifact kind, and merger, normalizer, and validator
resolution selects the matching typed variant.
A `ValidatorSpec` declares a validator key and execution class. Resolution uses
the execution class to reject incompatible profile bindings before execution.
The current production catalog and default chain are listed only in
[Configuration](../config.md#implemented-production-validators).
## Runner Boundary
## Preparation And Runner Boundary
`pipeline.RunInput` carries the resolved pipeline, raw source input, structured
LLM client, run identity and timing, optional session and profile metadata, and
checkpoint/debug collaborators. The runner parses source bytes through the
selected input adapter. Later stage requests receive the generic source model;
extract requests receive chunk-scoped input material, while chunk, merge, and
normalize requests retain access to the original source material.
`pipeline.Prepare` receives a resolved pipeline, the registries, and shared
module dependencies. It constructs input; chunk and its validators; each lane's
extract, merge, and normalize modules and validator chains in resolved order;
then output. It stops at the first error with pipeline, stage, lane, module, and
validator context as applicable. It never invokes an operation method.
`pipeline.RunOutput` carries the run manifest, accepted normalized results,
`PreparedPipeline` keeps private constructed executors and exposes cloned
resolved input, chunk, lane, and output identities. `pipeline.RunInput` carries
that prepared pipeline, raw source input, run identity and timing, optional
session and profile metadata, and checkpoint/debug collaborators. The runner
parses source bytes through the already constructed input adapter. Later stage
requests receive the generic source model; extract requests receive
chunk-scoped input material, while chunk, merge, and normalize requests retain
access to the original source material. Input, chunk, and output operation
requests do not carry raw module options. The chunk request also does not carry
an LLM client; an LLM-backed chunker receives the shared client during
preparation. Their operation requests retain run-specific source, reference,
profile, session, and metadata context as applicable.
Prepared lanes retain exact-type-checked erased operation closures. The runner
uses those closures to keep each value typed through extraction, validation,
merge, and normalization.
Source validation requires every unit to carry a canonical self-reference to
its containing document and its own unit ID. Explicit clone, checkpoint, and
debug boundaries retain that reference, and the canonical source digest covers
it deterministically. Chunks use the same source model and carry one canonical
reference spanning the first selected unit through the last.
`pipeline.RunOutput` carries the run manifest, accepted normalized serialized
artifacts with lane and normalizer provenance,
rejected results, warnings, checkpoint events, and logical files returned by the
output encoder. The CLI owns diagnostics and durable filesystem writes after the
runner returns.
@@ -83,21 +137,22 @@ runner returns.
The runner:
1. validates its input and registries;
2. builds the input adapter, parses the raw input, and validates the generic
1. validates its prepared input;
2. parses the raw input with the prepared adapter and validates the generic
source document;
3. obtains or executes the chunk result;
4. validates and canonicalizes chunks;
5. executes each resolved artifact lane in order;
6. builds the output encoder and validates its logical file results;
5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
bounded lane continuation when all extracts for that lane are terminal;
6. invokes the prepared output encoder and validates its logical file results;
7. returns the assembled manifest, outcomes, warnings, and files.
Within each artifact lane, it builds the extractor, merger, and normalizer,
then performs these transitions:
Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
and validators while performing these transitions:
1. extract once per accepted chunk and add runner-owned lane, source, and chunk
provenance;
2. validate each raw extract result and omit rejected results from merge input;
2. validate each extract result and omit rejected results from merge input;
3. skip the rest of the lane when no extract result is accepted;
4. merge accepted extract results in their existing order;
5. validate the merge result and skip normalization on rejection;
@@ -107,25 +162,35 @@ then performs these transitions:
Module-provided warnings and payload warnings are promoted only from attempts
whose results are accepted and used.
The extract job channel has the same capacity as the effective extract worker
count, so dispatch applies backpressure. A fixed continuation executor prevents
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
continuations publish lane-local results; the coordinator is the only writer of
aggregate output and merges those results in resolved lane and source-chunk
order.
## Chunk Canonicalization
Before lane execution, generic validation requires unique chunk IDs, matching
source identity, indexes matching returned order, valid ordered boundaries,
source identity, indexes matching returned order, a valid canonical reference,
non-empty content and media type, and at least one valid source unit per chunk.
Units may not repeat inside a chunk and must preserve source-document order.
Units may not repeat inside a chunk and must form a contiguous range in
source-document order. The chunk reference must exactly match the source and
the first and last unit references.
The runner then rebuilds each chunk's unit slice from the source document by
unit ID. It preserves the module-owned boundaries, content, media type, and
cloned metadata. The framework permits gaps and overlap between separate
unit ID. It preserves the canonical reference, content, media type, and cloned
metadata. The framework permits gaps and overlap between separate
chunks; stricter coverage policy belongs to the chunk implementation.
## Validation And Retries
Chunk, extract, merge, and normalize results pass through the resolved validator
chain for their stage and module. Each validator receives the raw payload plus
the relevant source, chunk, prior-stage, schema, reference, session, LLM, option,
and run context. Validators execute in resolved order and stop at the first
error or rejection. An empty chain approves the result.
chain for their stage and module. Chunk validators receive canonical chunks;
typed validators receive the domain value; and serialized validators receive
canonical chunk JSON or artifact codec bytes. Validators execute in resolved
order and stop at the first error or rejection. An empty chain approves the
result.
`runWithRetry` applies the effective retry policy around module execution and
its complete validation chain. A module or validator error becomes a framework
@@ -143,13 +208,20 @@ The runner depends on recorder and loader interfaces, using no-op
implementations when collaborators are absent. Each checkpointed workflow
boundary records a running, succeeded, or failed transition. Reuse decisions
are consulted in workflow order and accepted payloads are cloned before
entering the normal handoff path. Dependency fingerprints connect later
checkpoints to the exact accepted results on which they depend.
entering the normal handoff path. Typed extract, merge, and normalize
checkpoints store codec bytes with artifact kind, schema ID and version, exact
schema digest, and media type. Reuse compares that identity with the prepared
codec and decodes through the codec; missing identity, mismatches, corrupt
bytes, and decode failures become explicit reuse misses and execute the step
normally. Dependency fingerprints and debug content digests use the same stable
codec bytes that cross those boundaries.
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
boundaries. Context scopes associate nested LLM calls with the module or
validator attempt that made them. Debug-write failures are framework errors;
debug data is never used as a checkpoint source.
debug data is never used as a checkpoint source. Typed artifact debug envelopes
are domain-neutral, redact sensitive metadata and bytes through the common
debug policy, and record codec identity plus schema and content digests.
Checkpoint identity, physical layout, reuse behavior, and debug artifact
handling are operator contracts in [Operations](../operations.md). Serialization
@@ -161,25 +233,42 @@ and recorder implementation are inventoried in
The runner owns manifest assembly and handoff summaries but not the durable JSON
schema. It records resolved module and lane provenance, validator chains,
source/reference identities, selected LLM profiles, normalized and rejected
summaries, status, and timing. Raw payload bytes remain outside the manifest.
summaries, status, and timing. Serialized artifact content remains outside the manifest.
Module metadata providers may add non-secret singleton or lane-scoped metadata.
Execution errors include stage, module, lane, or validator context. Once a
manifest exists, a failing run returns it with failed status and completion
time. Successful status reflects whether any raw result was rejected. The
time. Successful status reflects whether any result was rejected. The
durable manifest and logical file schemas are defined in the
[JSON output contract](../integrations/json-output.md).
On a framework failure, the runner cancels its derived context, stops submitting
new extract work, drains started tasks, and skips the output encoder. Parent
cancellation takes precedence. Otherwise context-cancellation fallout is
discarded when a substantive error exists, and the primary error is selected by
stage, resolved lane, and source chunk rather than completion time.
## Tests To Inspect
- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
- `internal/framework/pipeline/profile_test.go`: selection, defaults,
capabilities, validator chains, and digest behavior.
- `internal/framework/pipeline/artifact_codec_registry_test.go`: typed codec
metadata, registration, erasure safety, strict decoding, and cloning.
- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed
lane resolution and preparation, target-specific validators,
incompatibilities, ordering, and schema-sensitive pipeline identity.
- `internal/framework/pipeline/runner_concurrency_test.go`: bounded dispatch and
continuations, reverse completion, stable errors, rejection, cancellation,
retries, and independent provider-call limits.
- `internal/framework/pipeline/preparation_test.go`: option validation,
construction order, dependency failures, and the before-source-work boundary.
- `internal/framework/pipeline/references_test.go`: target resolution and
materialization.
- `internal/framework/pipeline/runner_test.go`: stage transitions, retries,
rejections, warnings, checkpoints, debug hooks, and manifests.
- `internal/framework/pipeline/walking_skeleton_test.go`: fake-backed complete
workflow composition.
- `internal/cli/run_test.go`: production stage transitions, retries, rejections,
warnings, debug hooks, manifests, and end-to-end composition.
- `internal/modules/integration/*_test.go` and
`internal/modules/seriatim/input/transcript/runner_test.go`: typed runner
composition across concrete module families.
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse
collaborators.

View File

@@ -91,6 +91,20 @@ digests match the current invocation. Changes to input bytes, the resolved
pipeline, selected lanes, the runtime LLM profile override, or bound reference
content invalidate reuse.
Typed artifact checkpoints additionally record codec-owned bytes, artifact
kind, schema ID and version, exact schema digest, and media type. A missing or
mismatched codec identity, or bytes the current codec cannot decode, is reported
as a checkpoint reuse miss. The affected operation executes normally and, when
checkpoint writing is enabled, replaces the incompatible checkpoint.
Current checkpoint manifests use workspace schema `notarius.workspace.v2`.
Manifests written with `notarius.workspace.v1` are incompatible because their
chunk provenance has an older shape. On the first explicit resume after an
upgrade, each affected checkpoint is treated as a reuse miss and its workflow
step executes normally. The compatibility check does not migrate or delete the
v1 files; when checkpoint writing is enabled, normal execution refreshes the
affected checkpoint files in the current schema.
Runs do not reuse checkpoints unless explicitly requested. Without reuse, the
workflow executes normally and refreshes checkpoint files when checkpointing is
enabled.
@@ -118,7 +132,9 @@ the attempt `llm_calls` array. Prompt content is written inline in the prompt
artifact. The response metadata and body use the paired files described above;
the body is pretty-printed JSON when possible and raw text otherwise. Debug
artifacts may contain source material, reference material, prompt inputs, model
outputs, and other sensitive data. API keys are not written, and obvious
outputs, and other sensitive data. Typed artifact envelopes include
domain-neutral codec identity, redacted metadata and content, and digests of
the stable codec bytes. API keys are not written, and obvious
credential-shaped values and sensitive map keys are redacted, but debug
directories should still be protected as sensitive local state.
@@ -182,5 +198,11 @@ selected execution profile. Pipeline module retry settings are defined in
[Configuration](config.md#module-bindings). There is no separate CLI retry
command.
Extract worker concurrency and actual provider-call concurrency are separate
limits. Their configuration, defaults, and validation are defined in
[Configuration](config.md#concurrency). Cancellation stops undispatched extract
work; already started work is allowed to finish or observe cancellation before
the run reports failure.
Notarius writes local files only. Remote storage and archive management are not
part of the implemented CLI.

View File

@@ -66,12 +66,19 @@ Framework stages operate on source documents, source units, and source
references rather than format-specific structures. A source reference identifies
an ordered range of generic source units. Framework code preserves those ranges
and does not merge or rewrite them unless a stage module explicitly owns that
behavior.
behavior. Every source unit carries a validated self-reference to its containing
document and its own unit ID.
Extract modules own artifact semantics, prompt use, response schemas, and
domain interpretation. Domain-specific concepts remain in the relevant module,
validator, shared domain helper, and artifact contract.
Typed artifact registrations declare one stable artifact kind and exact Go
type from extraction through merge, normalization, and semantic validation.
Pipeline resolution requires a compatible codec and matching kind-specific
variants before a typed lane can be accepted. Framework-owned erasure remains
private and must report type incompatibility as an error rather than a panic.
Auxiliary references provide context or disambiguation. They are not source
evidence and must not be converted into source references.
@@ -83,6 +90,12 @@ and verifies module availability and capabilities before execution. Structural
pipeline choices must not be scattered through conditionals or hidden behind
ad hoc command flags.
Resolution validates every selected module and validator option set. A separate
preparation boundary then constructs the complete input, chunk, lane,
validation, and output implementation set in pipeline order. The runner accepts
only that prepared set, so construction and dependency failures occur before
source parsing or any other module operation.
Stage ownership is explicit:
- input modules convert external material into the generic source model;
@@ -96,13 +109,32 @@ The framework owns orchestration and handoff provenance. Modules return logical
results and warnings; they do not own CLI reporting, workspace paths, durable
file placement, checkpoints, or diagnostics.
After pipeline-wide chunking, extraction uses bounded framework concurrency.
One run-wide worker pool receives chunk-scoped lane jobs in deterministic
chunk-first, lane-second order. A lane may begin its merge and normalize
continuation only after all of its extract jobs are terminal; that continuation
remains serial within the lane, while bounded continuations for different lanes
may overlap. The framework must not create unbounded goroutines per lane or
chunk.
Completion timing does not choose public ordering or errors. The coordinator
orders accepted artifacts, warnings, rejections, checkpoint events, and
framework errors by stable pipeline scope. Rejections do not cancel unrelated
work. A framework error cancels derived work, prevents undispatched work from
starting, waits for started work, and prevents output encoding.
## Validation
Validation is a framework-managed boundary around raw outputs from chunk,
extract, merge, and normalize stages. Validators receive immutable stage output
Validation is a framework-managed boundary around outputs from chunk, extract,
merge, and normalize stages. Validators receive immutable stage output
and make an explicit whole-output decision: approve, approve with warnings, or
reject.
Typed artifact validators receive the domain value directly. Chunk validators
receive source-zone chunks, while serialized validators receive immutable
representation bytes and declared schema metadata. A validator registered for
one target or artifact kind cannot satisfy an incompatible selection.
Rejection is a recorded pipeline outcome, not a framework execution error.
Validator execution failures are framework errors. Rejected output does not
advance to the next stage.
@@ -126,6 +158,11 @@ LLM calls and other external operations accept cancellation and respect
timeouts. Concurrency control belongs in shared runtime plumbing rather than in
individual modules.
The application-wide LLM scheduler bounds actual provider calls independently
of framework worker limits. Every LLM-backed module, retry, and validator uses
the single injected scheduled client, including work performed by overlapping
lanes.
## Configuration And Provenance
Configuration loading, precedence, defaults, environment overrides, redaction,

View File

@@ -2,16 +2,16 @@
## Status
Decision-complete; implementation pending. This roadmap defines the desired end
state for [ADR-0002](../adr/0002-linear-pipes-and-filters-pipeline.md),
Implemented on 2026-07-17. This roadmap records the design delivered for
[ADR-0002](../adr/0002-linear-pipes-and-filters-pipeline.md),
[ADR-0003](../adr/0003-typed-interfaces-with-two-zone-data-model.md), and
[ADR-0004](../adr/0004-package-modules-by-domain.md). The work needed to reach
that state is owned by the
[domain pipeline implementation plan](implementation.md).
[ADR-0004](../adr/0004-package-modules-by-domain.md). Its implementation history
is summarized in the [completion record](implementation.md).
Until that plan is complete, current behavior remains defined by the
architecture, configuration, integration, operations, and internal
documentation outside `docs/roadmap/`.
This file is historical design context, not a current-behavior reference.
Implemented contracts and mechanics are defined by the architecture,
configuration, integration, operations, and internal documentation outside
`docs/roadmap/`.
## User Intent

View File

@@ -4,12 +4,6 @@ Current Notarius behavior is documented in the canonical README, CLI,
configuration, operations, internal, and integration docs. This roadmap records
future work only.
## Focused Roadmaps
- [Domain-Typed Pipeline Implementation](domain.md): proposed migration to
domain-owned typed artifact lanes, domain-first packages, explicit
serialization boundaries, and bounded deterministic extract execution.
## Candidate Product Work
- Additional input adapters, such as Markdown or note-export formats.

View File

@@ -1,836 +1,48 @@
# Domain-Typed Pipeline Implementation Plan
# Domain-Typed Pipeline Completion Record
## Purpose
## Status
This document is the executable implementation plan for the target state in the
[domain-typed pipeline feature roadmap](domain.md). It assumes the decisions in
Implemented on 2026-07-17.
This file is a concise historical record. Current behavior is owned by
[Architecture](../policy/architecture.md),
[Configuration](../config.md), [Operations](../operations.md),
[Pipeline Internals](../internal/pipeline.md),
[Module Internals](../internal/modules.md),
[LLM Runtime Internals](../internal/llm.md), and the durable
[integration contracts](../integrations/). The accepted decisions remain in
[ADR-0002](../adr/0002-linear-pipes-and-filters-pipeline.md),
[ADR-0003](../adr/0003-typed-interfaces-with-two-zone-data-model.md), and
[ADR-0004](../adr/0004-package-modules-by-domain.md).
The intended operator is an LLM coding agent working through one stage per
implementation prompt. Complete the stages in order. Each stage must leave the
repository buildable and tested; do not defer a broken intermediate state to a
later stage.
## Implementation Rules
For every stage:
1. Read `docs/development.md` and follow its task-specific reading guide. Read
the current implementation and focused tests for every touched subsystem.
2. Treat the feature roadmap as the canonical owner of desired behavior and
this document as the canonical owner of task sequencing. Do not restate
future behavior in current-behavior documentation before it exists.
3. Preserve unrelated user changes. Use mechanical moves where possible so file
history and test intent remain legible.
4. Add focused tests with the change. Run those tests while iterating, then run
`go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` before ending
the stage.
5. Run `go test -race ./...` in stages that introduce or change concurrency and
in the final stage.
6. Update the canonical current-behavior documents in the same stage in which
behavior changes. At minimum, reconsider `docs/policy/architecture.md`,
`docs/internal/overview.md`, `docs/internal/pipeline.md`,
`docs/internal/modules.md`, `docs/internal/llm.md`, `docs/config.md`,
`docs/operations.md`, and `docs/integrations/` according to the documentation
policy; edit only the documents whose owned facts changed.
7. Do not change user-visible module keys, validator keys, output paths, durable
D&D JSON, prompt/schema identities, default chains, or rejection semantics
unless this plan explicitly requires it.
8. Stop after a stage if an exit criterion cannot be met. Record the concrete
blocker rather than implementing a second architecture alongside this one.
## Fixed Technical Decisions
The following choices are inputs to implementation, not questions to reopen in
individual stages.
### Source types
Keep the existing names `source.SourceDocument`, `source.SourceUnit`, and
`source.SourceRef`. Add `Ref source.SourceRef` to `SourceUnit`. Move
`contracts.SourceChunk` to `internal/core/source` as `source.Chunk`, with this
logical shape:
```go
type Chunk struct {
ID string
SourceID string
Index int
Ref SourceRef
Content []byte
MediaType string
Units []SourceUnit
Metadata map[string]any
}
```
Remove `StartUnitID` and `EndUnitID`; `Ref` is the only chunk-boundary
representation. A Seriatim unit's self-reference is
`{SourceID: document ID, StartUnitID: unit ID, EndUnitID: unit ID}`. A chunk
reference spans its first and last included units. Advance persisted workspace
state from `notarius.workspace.v1` to `notarius.workspace.v2`; v1 state is
incompatible and must be recomputed, but never deleted automatically.
### Artifact contracts
Place engine-owned artifact primitives with the other universal contracts under
`internal/framework/contracts`:
```go
type ArtifactKind string
type ArtifactSchema struct {
ID string
Name string
Version string
JSONSchema []byte
}
type SerializedArtifact struct {
Kind ArtifactKind
Schema ArtifactSchema
MediaType string
Content []byte
Metadata map[string]any
}
type ArtifactCodec[T any] interface {
Kind() ArtifactKind
Schema() ArtifactSchema
MediaType() string
Encode(T) ([]byte, error)
Decode([]byte) (T, error)
}
```
Use strict JSON decoding for JSON codecs: reject unknown fields and trailing
tokens. Encoding must be deterministic for equal canonical values. Clone byte
slices and maps at framework ownership boundaries. Validate non-empty artifact
kind, schema ID, schema name, schema version, media type, and JSON Schema during
registration. Compute and retain a SHA-256 digest of the JSON Schema bytes.
Use generic `Extractor[T]`, `Merger[T]`, `Normalizer[T]`, and
`TypedValidator[T]` contracts. The exact request/result structs may retain
existing names where that reduces churn, but they must satisfy these rules:
- the extractor returns `T`, warnings, and framework-owned chunk provenance;
- merge receives accepted per-chunk typed values carrying lane ID, source ID,
chunk ID, chunk index, and chunk reference, already sorted by chunk index;
- normalize receives and returns `T`;
- typed validators receive `T` plus the relevant universal source, chunk,
reference, lane, stage, profile, and metadata context;
- the LLM client and decoded module options are held by constructed
implementations, not passed in operation requests; and
- no module-facing Zone-B request or result contains `RawPayload`, `any`, or
serialized JSON as its artifact value.
Retain separate framework wrappers around `T` for extract, merge, and normalize
provenance. Do not put lane IDs, module keys, or framework warnings into the D&D
domain value itself.
Support a second `SerializedValidator` contract for representation-level
generic validators. Its request contains immutable bytes, media type, and
optional schema metadata. For a Zone-B value, the framework produces that
request with the lane codec. Keep a separate non-generic `ChunkValidator`
contract for semantic validation of immutable `[]source.Chunk`; when a
representation validator is selected at chunk, the framework instead supplies
its canonical JSON chunk encoding. `valid_json` and `valid_json_schema` use the
serialized path, domain validators use `TypedValidator[T]`, and generic
approve/reject validators register explicit chunk and typed-artifact variants.
### Typed registry model
Because Go methods cannot introduce type parameters, expose free generic
registration functions in `internal/framework/pipeline`, backed by private
non-generic registry entries. Use `reflect.TypeFor[T]()` only inside registration
and framework assembly to prove exact type equality.
- Codec registry key: artifact kind. Exactly one codec may be registered per
kind.
- Extractor registry key: existing module key. Each entry declares one artifact
kind and exact Go type.
- Merger and normalizer registry key: `(existing module key, artifact kind)`.
- Typed validator registry key: `(existing validator key, artifact kind)`.
- Chunk-validator registry key: existing validator key in the distinct chunk
target namespace.
- Serialized validators retain their existing validator key and declare whether
they support chunk values, artifact values, or both.
- Duplicate keys/variants, missing codecs, Go-type mismatches, and incompatible
selected variants are errors.
The extractor selected for a lane establishes the lane artifact kind. During
resolution, look up merger, normalizer, and validators against that kind.
Record artifact kind, schema ID, schema version, and schema digest on the
resolved lane and in the resolved-pipeline digest. A resolved pipeline with an
incompatible lane must fail before preparation or source execution.
The private erased lane entry owns closures for construction and execution of
one concrete `T`. It may store a value as `any` internally, but it must verify
the exact registered `reflect.Type` at every erased boundary and return a
descriptive framework error rather than panic.
### Construction and preparation
Use one uniform construction context:
```go
type ModuleDependencies struct {
LLM contracts.StructuredLLMClient
}
type BuildRequest struct {
Dependencies ModuleDependencies
Options map[string]any
}
```
Each registry entry stores both an option-validation closure and a constructor.
Implementations own concrete option structs and one decoder used by both
closures. Resolution/configuration validation calls the decoder and discards
the value; preparation calls it once and supplies the decoded value to the
constructor. Reject unknown option fields. Empty options produce the
implementation's explicit defaults.
Add:
```go
func Prepare(
resolved ResolvedPipeline,
registries Registries,
deps ModuleDependencies,
) (*PreparedPipeline, error)
```
`PreparedPipeline` retains explicit resolved input, chunk, artifact-lane, and
output fields and all constructed validators. Construct in stable pipeline
order: input; chunk and its validators; each resolved lane in order with extract,
merge, normalize, and their validator chains in stage order; then output. On the
first failure, return an error identifying stage, lane if any, module or
validator key, and cause. No operation method may have run.
Create the one scheduled production LLM client first, inject it into
preparation, and then run only the prepared pipeline. Test-only deterministic
modules may accept a nil LLM dependency; any implementation that declares or
uses LLM-backed execution must reject a nil client at preparation. Remove LLM
clients and raw option maps from operation requests after every production
implementation has migrated.
Constructed modules and validators are reused for a run. Anything callable from
parallel extract workers must be concurrency-safe; production implementations
should be immutable after construction.
### Package registration
Each package-family registrar exposes:
```go
func Register(registries pipeline.Registries, assets *llm.AssetRegistry) error
```
The generic and Seriatim registrars ignore the asset argument until they need
it. Registrars validate the registry pointers they use and return contextual
errors. The CLI creates one complete registry set and one asset registry, then
calls registrars in this order: generic, Seriatim, D&D. The D&D registrar owns
D&D codecs, implementations, typed generic specializations, prompt/schema
assets, and default validator chains.
`internal/modules/dnd` owns D&D shared types. Its `register` sibling may import
children and generic strategies; the root package must not import its children.
`internal/modules/generic` never imports D&D. `internal/modules/seriatim` does not
import D&D. Move domain-neutral embedded prompt-filesystem helpers to
`internal/framework/promptfs`.
### D&D typed model
Define `dnd.SpellList`, `dnd.SpellCast`, and evidence/source-reference fields at
the D&D package root. Use `source.SourceRef`; do not create another D&D unit-ref
type for artifact provenance. The stable artifact kind is
`dnd/spell-list`.
The spell extractor owns a private LLM DTO and the existing
`dnd_spells_llm.v1.json` response schema. It canonicalizes and maps that DTO to
`dnd.SpellList`. The codec package owns `dnd_spells.v1.json` and the durable
encoding. Keep those schemas separate. Remove the validator-only
`spellpayload` model after all three D&D validators consume `dnd.SpellList`.
Make `appendorder` a generic strategy that accepts a typed combine function at
registration/construction. The D&D registrar supplies a function that appends
spell casts in already-sorted source-chunk order. Make `noop` a generic typed
strategy. Neither generic package imports D&D.
### Serialized boundaries
After normalize, encode `T` once to `SerializedArtifact` for final output.
Output remains domain-neutral and receives serialized artifacts. Preserve the
existing output bundle and index contract.
Extract, merge, and normalize checkpoints encode and decode through the same
lane codec. Checkpoint identity includes artifact kind, schema ID, schema
version, and schema digest. A missing codec or any mismatch invalidates reuse
and recomputes the stage; it is not a fatal run error by itself. Codec decode
failure also invalidates that checkpoint and records the reason. Never pass
serialized checkpoint content directly to the next typed stage.
Debug recording uses the codec for typed artifact values and preserves existing
opt-in/sensitive-content rules. Artifact/checkpoint digests use the stable codec
bytes. Aggregate manifests, checkpoint indexes, warning slices, and rejection
slices have one coordinator writer.
### Concurrency
Version-2 configuration gains:
```yaml
concurrency:
total_llm: 4
stage_workers:
extract: 4
```
Represent effective stage worker limits as `map[string]int`. Initially accept
only `extract`; reject unknown keys. Missing extract defaults to `total_llm` and
its valid range is `1..total_llm`. Add the environment override
`NOTARIUS_STAGE_WORKERS_EXTRACT`. Preserve precedence rules and keep the file
configuration version at 2.
Use one run-wide fixed extract worker pool. Dispatch jobs in round-robin order
with source chunk as the outer loop and resolved lane as the inner loop. Use one
job channel whose capacity equals the effective extract worker count, so the
dispatcher applies bounded backpressure. Do not create one goroutine per job. A
job includes extract retries and extract-stage validators. Store results by
lane index and chunk index.
When all extract jobs for a lane are terminal, run that lane's merge and then
normalize serially. Lane continuations may overlap. All LLM calls at all stages,
including retries and validators, use the single injected scheduled client, so
`total_llm` remains the authoritative process-wide provider-call ceiling.
Rejections are terminal results and do not cancel other work. A framework error
cancels a derived run context, stops dispatching jobs not yet started, and waits
for started tasks to finish or observe cancellation. Choose the returned error
as follows:
1. if the parent context is canceled, return its error;
2. otherwise discard internal `context.Canceled`/`DeadlineExceeded` errors when
at least one non-context framework error exists; and
3. choose the earliest remaining error by stage order (`extract`, `merge`,
`normalize`), resolved lane index, chunk index for chunk-scoped work, and
configured validator/operation index.
Use a sentinel chunk index after all real chunks for lane-scoped merge and
normalize errors. Retain other started-task errors only in opt-in diagnostics.
Sort accepted artifacts, warnings, and rejections by resolved lane index, source
chunk index where applicable, stage order, validator order, and original
within-result order. Completion timing must not affect public output. Run output
only if every lane has a successful or rejection-only terminal outcome and no
framework error occurred.
## Staged Implementation
### Stage 1: Compatibility Baselines and ADR Acceptance
Goal: lock down behavior that subsequent internal migrations must preserve and
record the architectural decisions as accepted.
Tasks:
- Add semantic or golden compatibility tests for the maintained Seriatim-to-D&D
path: durable output files and JSON, output index, manifest provenance,
warnings, rejections, and stable lane/chunk ordering.
- Snapshot production module keys, validator keys, default validator chains,
prompt/schema identities, and maintained example/profile resolution in tests.
- Strengthen runner tests for fixed topology, validator rejection as a nonfatal
outcome, framework-error abort, retries, parent cancellation, checkpoint reuse
and invalidation, diagnostics, and opt-in debug behavior.
- Add an instrumented scheduled-client test showing that all existing production
LLM callers share `concurrency.total_llm`. It need not demonstrate parallel
lanes yet.
- Review the three ADRs against this decision-complete plan, set their status to
`Accepted`, and update their dates only if ADR policy requires an acceptance
date. Do not rewrite accepted decision text after this stage.
Exit criteria:
- compatibility tests fail on an unintended durable-output, key, chain,
provenance, or outcome-semantics change; and
- ADR-0002, ADR-0003, and ADR-0004 are accepted.
### Stage 2: Registrar Composition Without Package Moves
Goal: replace CLI leaf-by-leaf registration with package-family composition
before changing imports.
Tasks:
- Add `internal/modules/generic/register`, `internal/modules/seriatim/register`,
and `internal/modules/dnd/register` using the fixed registrar signature.
- Initially let those registrars import the existing stage-oriented packages.
Move ownership of production validators, default chains, and prompt assets out
of `internal/cli/catalog.go` and into the appropriate registrar.
- Have the CLI allocate complete registries and the asset registry once, invoke
generic, Seriatim, then D&D registration, and retain its existing test
injection paths.
- Test nil registry handling, duplicate registration errors, stable registered
keys, default chains, and asset identities.
Exit criteria:
- the CLI composition root names only the three registrar packages, framework
registry types, and asset registry; and
- no production key, chain, prompt, schema, or runtime behavior changes.
### Stage 3: Mechanical Generic and Seriatim Package Moves
Goal: establish the domain-first generic and source-format trees without
changing contracts.
Tasks:
- Move the Seriatim adapter and tests to
`internal/modules/seriatim/input/transcript`.
- Move the generic chunker to `internal/modules/generic/chunk/units`, retaining
the configured key `generic`.
- Move append-order merge, no-op normalize, JSON output, and all generic
validators to their target paths under `internal/modules/generic`.
- Update only registrar imports and affected black-box tests. Preserve package
behavior and all public registry keys.
- Remove the emptied old directories.
Exit criteria:
- generic and Seriatim production implementations exist only under their target
trees; and
- compatibility baselines remain green.
### Stage 4: Mechanical D&D Package Move and Import Guard
Goal: establish the D&D tree and enforce ADR-0004 dependency direction while
legacy contracts are still intact.
Tasks:
- Move the D&D scenes chunker, spell extractor, validators, schemas, prompt
assets, and D&D shared helpers into the target D&D tree. Do not create the
typed root model or codec yet.
- Move domain-neutral prompt filesystem helpers from
`internal/modules/sharedassets` to `internal/framework/promptfs`.
- Move tests with their owning implementation. Relocate tests that intentionally
compose domains to a black-box integration-test package rather than creating
peer-domain production imports.
- Add a Go-parser-based import-boundary test. It must reject concrete
D&D-to-Seriatim and Seriatim-to-D&D imports, all generic-to-D&D imports, and
root-domain imports of child implementations. Allow domain registrars, the CLI
composition root, and designated external integration tests to compose
packages.
- Remove old empty stage-oriented and validator directories.
Exit criteria:
- all production extensions use the target domain-first package layout except
the not-yet-created typed codec/model pieces;
- the import guard detects a deliberate fixture violation; and
- behavior and keys remain unchanged.
### Stage 5: Source-Unit Provenance
Goal: add canonical provenance to engine-owned source units without yet changing
the chunk type.
Tasks:
- Add `Ref source.SourceRef` to `source.SourceUnit`, including clone and debug
representations.
- Make the Seriatim adapter assign the fixed self-reference for every unit.
- Extend `source.ValidateDocument` to require the unit reference's source ID to
match the document, require start and end IDs to equal the unit ID, reject
missing/invalid/reversed references, and preserve the existing unit-order and
uniqueness checks.
- Make source digests and source checkpoint serialization include the new
reference deterministically.
- Add focused tests for valid refs and missing, foreign, non-self, and reversed
refs, plus source checkpoint/debug round trips.
Exit criteria:
- every produced source unit has a validated self-reference; and
- source state preserves it through clone, debug, digest, and checkpoint paths.
### Stage 6: Engine-Owned Chunks and Workspace v2
Goal: finish the Zone-A source model and make its persisted compatibility break
explicit.
Tasks:
- Add `source.Chunk` with the fixed shape and update universal chunk contracts,
modules, validators, runner code, checkpoints, debug envelopes, and tests to
use it.
- Derive `Chunk.Ref` from the first and last included unit references. Validate
source identity, non-empty ordered units, contiguous boundary agreement, and
exact correspondence between the chunk ref and first/last unit refs.
- Remove `contracts.SourceChunk`, `StartUnitID`, and `EndUnitID` after all
consumers migrate. Do not keep aliases.
- Advance `workspace.WorkspaceSchemaVersion` to `notarius.workspace.v2`. Make
loader behavior explicitly classify v1 as incompatible and recompute while
leaving files untouched.
- Update checkpoint identity/digest tests and operations documentation for the
one-time v1 resume miss.
Exit criteria:
- no production code imports a framework-owned chunk type;
- chunk provenance round-trips exactly; and
- v1 workspaces are safely ignored while v2 workspaces reuse successfully.
### Stage 7: Artifact and Codec Foundation
Goal: add the typed primitives and prove strict serialization independently of
production lanes.
Tasks:
- Add the fixed artifact types, codec interface, schema digest helper, and clone
helpers under `internal/framework/contracts`.
- Add `ArtifactCodecRegistry` to `pipeline.Registries` and `ModuleCatalog`.
- Implement generic codec registration and private erasure/type tracking.
- Validate registration metadata and duplicates. Ensure erased encode/decode
returns typed errors, never reflection panics.
- Use two small test artifact types to cover registration, exact type identity,
deterministic encoding, strict decoding, cloning, duplicate kind rejection,
and schema metadata/digest behavior.
- Wire the new empty registry through CLI/test registry constructors without
changing production lane execution.
Exit criteria:
- codecs for heterogeneous test types can coexist and safely round-trip through
erased framework storage; and
- current production behavior remains on the legacy raw path and unchanged.
### Stage 8: Typed Contracts, Variants, and Resolution
Goal: resolve a complete type-compatible lane before executing it.
Tasks:
- Add the typed stage, typed validator, chunk validator, serialized validator,
and provenance wrapper contracts from the fixed decisions.
- Extend extractor specs with artifact kind/type. Convert merger, normalizer,
and typed validator registries to artifact-kind variants while retaining
serialized-validator registration by key.
- Implement free generic registration helpers and private erased entries.
- Extend lane resolution to derive kind from extractor, require its codec,
select exact merger/normalizer/validator variants, and include artifact/schema
identity in resolved lanes and pipeline digest.
- Keep legacy registration helpers only as explicitly named transitional APIs;
do not let a raw registration satisfy a typed lane.
- Add composition tests with two artifact types and heterogeneous lanes. Cover
missing codec, missing variant, Go-type mismatch, duplicate variant, wrong
validator kind, stable resolution order, and digest changes on schema identity
or schema digest changes.
Exit criteria:
- heterogeneous typed test lanes resolve without module-facing erasure;
- every incompatible selection fails before execution; and
- existing raw production lanes continue to resolve only through their visible
transitional path.
### Stage 9: Preparation and Construction Foundation
Goal: construct and validate an entire resolved pipeline before source work.
Tasks:
- Add `ModuleDependencies`, `BuildRequest`, `PreparedPipeline`, and `Prepare` as
specified.
- Extend registry entries/specs with option validation and construction
functions. Supply adapters for legacy zero-argument constructors during the
migration.
- Call option validation for every selected module and validator during
resolution/config validation. Reject unknown fields and contextualize errors.
- Have preparation construct all selected components in fixed order and retain
immutable prepared lane executors.
- Update the runner API so `Run` receives a prepared pipeline. At the CLI, create
the shared scheduled LLM client, prepare, and only then invoke the runner.
- Prove with fakes that malformed options, missing required LLM dependencies,
and late-component construction failures occur before the input adapter's
`Parse` method.
Exit criteria:
- all components are constructed before source work;
- preparation errors identify exact scope and perform no operations; and
- legacy production modules still run through temporary construction adapters.
### Stage 10: Migrate Universal Modules to Construction
Goal: remove legacy option/dependency handling from input, chunk, and output.
Tasks:
- Give Seriatim input, generic units chunking, D&D scenes chunking, generic JSON
output, and all applicable chunk validators implementation-owned option
structs and strict decoders.
- Build those implementations with decoded options and injected dependencies.
Require the LLM client for D&D scenes; keep deterministic implementations
independent of it.
- Remove `Options` and `LLMClient` from the corresponding operation requests.
Retain per-run source, reference, profile, session, and metadata fields.
- Update registrars and focused tests. Verify options are decoded once during
preparation and operation methods do not inspect raw maps.
Exit criteria:
- no universal production module parses raw options during execution; and
- every universal LLM call uses the injected shared client.
### Stage 11: Canonical D&D Model, Codec, and Typed Extractor
Goal: establish the first production `T` and its extraction boundary.
Tasks:
- Add canonical spell types at the D&D root using engine-owned source refs.
- Add `internal/modules/dnd/codec/spells`, register kind `dnd/spell-list`, and
make it own the existing durable `dnd_spells.v1.json` schema and strict stable
encoding.
- Keep the private spell-extraction LLM DTO and
`dnd_spells_llm.v1.json` in the extractor package. Map canonicalized DTO values
to `dnd.SpellList` and do not expose the DTO to validators or the codec.
- Convert the extractor to `Extractor[dnd.SpellList]`, construction-time options
and dependency injection. Preserve prompt assets, retries, warnings, evidence,
and LLM response validation.
- Register the codec and typed extractor from the D&D registrar.
- Add codec compatibility tests against existing durable fixtures and tests
proving LLM schema ownership is separate from durable schema ownership.
Exit criteria:
- the typed extractor returns the canonical domain model;
- codec output is semantically identical to the maintained durable spell JSON;
and
- no downstream production consumer is switched until the next stages.
### Stage 12: Typed Validators and Generic Typed Strategies
Goal: complete all typed components required by the D&D lane.
Tasks:
- Convert D&D spell shape, source-reference, and source-relatedness validators
to `TypedValidator[dnd.SpellList]` with construction-time options/dependencies.
- Remove JSON reparsing from those validators. Remove the duplicate
`spellpayload` package after its final consumer migrates.
- Convert `valid_json` and `valid_json_schema` to serialized validators. Register
them so the framework uses the D&D codec when they occur in the spell chain.
- Implement generic typed append-order merge and no-op normalize. In the D&D
registrar, register D&D variants using a spell-list append function and
`noop[dnd.SpellList]`.
- Convert always-accept/reject into explicit chunk and typed variants and
register the D&D variants without changing their keys. Verify serialized
validators operate on the framework encoding at chunk and the codec encoding
at artifact stages.
- Test typed validator requests, source refs, relatedness LLM injection, generic
strategy reuse with a second test type, default chain order, and rejection
behavior.
Exit criteria:
- every selected spell-lane component has a compatible D&D typed variant;
- generic packages import no D&D code; and
- the duplicate validator payload model and inter-validator JSON parsing are
gone.
### Stage 13: Typed D&D Runner Vertical Slice
Goal: execute one complete production lane as `dnd.SpellList` while retaining
temporary raw output/checkpoint adapters.
Tasks:
- Implement the erased typed lane executor and runner path for extract, stage
retries, typed/serialized validation, merge, and normalize.
- Keep accepted extract values indexed by chunk and pass them to merge in source
order. Preserve warnings and rejections in stable scope order.
- Add narrow transitional adapters from typed stage outputs to the existing raw
checkpoint/debug/output envelopes. These adapters must use the registered
codec and be named/commented as migration-only.
- Route the production D&D lane through the typed path; leave no production raw
D&D stage module registered in parallel.
- Add end-to-end and checkpoint-disabled tests proving the maintained D&D output
and outcome semantics are unchanged. Add incompatible-lane tests proving
failure occurs before input.
Exit criteria:
- D&D values remain typed from extractor through normalize and typed validation;
- the runner's only D&D erasure is its private lane adapter and explicit codec
boundary; and
- durable output remains unchanged through the transitional adapter.
### Stage 14: Typed Checkpoints, Debugging, and Output
Goal: move every serialization side effect and the final Zone-C boundary to the
codec model.
Tasks:
- Change runner/output contracts so final normalized results are
`SerializedArtifact` values. Update generic JSON output without importing D&D.
- Preserve logical file names, index fields, media types, manifest contents, and
durable spell JSON.
- Change extract, merge, and normalize checkpoints to store codec bytes plus
artifact kind, schema ID, version, and schema digest. Decode reused values
back to `T` before the next stage.
- Implement safe invalidation for missing/mismatched codecs and decode failure,
with explicit checkpoint-event reasons.
- Change typed debug envelopes to serialize through the codec, preserving opt-in
and redaction behavior. Use stable codec bytes for artifact digests.
- Remove the transitional raw output/checkpoint/debug adapters introduced in
Stage 13 after all tests use the typed boundaries.
Exit criteria:
- typed values round-trip at every stage checkpoint;
- incompatible checkpoint artifacts recompute safely;
- output and debug code are domain-neutral; and
- no D&D typed lane depends on a raw-boundary adapter.
### Stage 15: Finish Construction Migration and Remove Raw Contracts
Goal: leave one production extension system rather than parallel raw and typed
models.
Tasks:
- Migrate any remaining merge, normalize, extract, and validator implementations
to construction-time option decoding and dependency injection.
- Remove LLM clients and raw option maps from all remaining operation requests.
- Remove legacy raw extractor/merger/normalizer/validator contracts,
constructors, registry entries, `RawPayload`, `ResponseSchema` if superseded,
raw clone helpers, raw checkpoint envelopes, and migration-only adapters.
- Remove dead duplicate models and compatibility helpers. Search for production
references to old stage-oriented paths and raw Zone-B types.
- Keep serialized artifacts only at codec, checkpoint/debug, and output
boundaries.
Exit criteria:
- no production Zone-B handoff uses JSON bytes, `RawPayload`, or `any`;
- all production configuration options are validated before execution;
- all production LLM users receive the one injected client; and
- there is no legacy production registration path.
### Stage 16: Stage-Worker Configuration
Goal: add the decided scheduling control without changing runner execution yet.
Tasks:
- Add `StageWorkers map[string]int` to effective concurrency configuration and
`stage_workers` to the version-2 YAML shape. Deep-clone the map.
- Accept only `extract`, reject unknown or empty keys, default missing extract to
effective `total_llm`, and validate the inclusive range
`1..total_llm` after file/environment precedence resolves.
- Add `NOTARIUS_STAGE_WORKERS_EXTRACT` with the existing environment precedence
and integer error style.
- Preserve redaction/effective-config diagnostics and version 2.
- Update `docs/config.md` and maintained examples that intentionally demonstrate
concurrency. Do not add the field to every example when the default conveys
the intended behavior.
- Add file, default, merge/precedence, environment, unknown-key, boundary, and
redacted-effective-config tests.
Exit criteria:
- every run has a validated effective extract worker count;
- omitted configuration preserves current effective behavior at the default
`total_llm: 1`; and
- current configuration documentation owns the implemented contract.
### Stage 17: Concurrent Lane and Extract Scheduling
Goal: implement bounded concurrent lanes while preserving deterministic public
behavior and the global LLM invariant.
Tasks:
- Add the fixed-size run-wide worker pool and central round-robin dispatcher.
Bound queued work so dispatch applies backpressure; do not enqueue the whole
run into unbounded memory.
- Treat extract, retries, and extract validators as one job. Publish immutable
results to a coordinator indexed by lane and chunk.
- Start each lane's serial merge/normalize continuation only after all its
extract jobs are terminal. Permit different lane continuations to overlap.
- Make the coordinator the sole writer of aggregate output, manifest,
checkpoint-event collection, warnings, and rejections. Use attempt-specific
debug paths and synchronize any recorder state that remains shared.
- Implement rejection, cancellation, stable sorting, deterministic primary
error selection, and output gating exactly as specified under Concurrency.
- Audit every concurrently reused extractor, validator, codec, LLM/debug wrapper,
checkpoint loader/recorder, and manifest metadata provider. Make production
implementations immutable or add narrowly scoped synchronization.
- Add deterministic barrier-controlled tests that force reverse completion
order, simultaneous failures, parent cancellation, rejections mixed with
successes, lane continuation overlap, and undispatched-job cancellation.
- Add instrumented integration tests issuing calls from multiple lanes, retries,
and LLM-backed validators. Assert provider calls never exceed
`total_llm`, extract jobs never exceed the effective extract worker count, and
both limits are exercised independently.
- Run `go test -race ./...` and eliminate races rather than weakening tests.
Exit criteria:
- lanes and extract jobs actually overlap when configured above one;
- job and provider-call limits are independently enforced;
- public results and primary errors are identical across forced completion
orders; and
- full race testing passes.
### Stage 18: Documentation, Cleanup, and Final Verification
Goal: make the implemented repository and its canonical documentation agree,
then close the focused roadmap.
Tasks:
- Review every current-behavior document routed by `docs/development.md` and
update only its owned facts: architecture and dependency direction, package
inventory, pipeline resolution/preparation/execution, typed module contracts,
LLM scheduling, configuration, checkpoint compatibility, diagnostics,
operations, and durable integration contracts.
- Verify maintained examples and copyable files against the implementation.
- Remove stale old package paths, raw-contract terminology, and superseded
future-work entries. Validate all changed documentation links.
- Update ADR consequences only in ways permitted for accepted ADR metadata; do
not edit accepted decision text. Record a new superseding ADR if final code
required an architectural change.
- Mark the feature roadmap implemented and reduce this implementation plan to a
concise completion record, or move it to the repository's established
completed-roadmap location if one exists. Do not let this file become a second
current-behavior reference.
- Run final checks:
```sh
go test ./...
go test -race ./...
go vet ./...
go build ./cmd/notarius
```
Exit criteria:
- all feature-roadmap completion criteria are met;
- no stale production paths or legacy typed/raw bridge remain;
- current documentation and examples describe only implemented behavior; and
- all final validation commands pass.
## Open Questions
None. The feature-policy decisions and the implementation choices required to
begin each stage are resolved above. If implementation evidence contradicts one
of the accepted architectural decisions, stop and handle that as an ADR change
or supersession rather than treating it as an implicit implementation choice.
## Delivered
- Production extensions use domain-first packages and package-family
registrars without changing selectable module or validator keys.
- Source units and chunks carry canonical engine-owned provenance using
`source.SourceRef`; workspace checkpoints use schema v2.
- Artifact lanes keep one domain-owned Go type through extraction, merge,
normalization, and typed validation.
- Artifact codecs own stable schema-aware checkpoint, debug, and output
serialization; generic representation validators consume serialized views.
- Resolution verifies artifact-kind, codec, typed variant, capability,
reference, validator, and option compatibility before source work.
- Preparation constructs the full run-local implementation set and injects one
shared scheduled LLM client.
- Extraction uses bounded chunk-first, lane-second dispatch. Lane continuations
are also bounded, public results are deterministic, and framework failures
cancel undispatched work without treating rejections as errors.
- Maintained D&D payloads, logical output paths, prompt and schema identities,
default validators, warnings, rejections, and manifest provenance remain
covered by compatibility tests.
## Completion Evidence
Focused tests cover durable compatibility, typed resolution and preparation,
codec boundaries, checkpoint invalidation, debug recording, bounded scheduling,
reverse completion, cancellation, rejection, retries, and independent extract
worker and provider-call limits. Maintained example configurations and input are
validated and exercised by the CLI test suite.
Repository validation is defined in [Development](../development.md). The Go
race suite requires a CGO-capable toolchain in the execution environment.

View File

@@ -1,6 +1,8 @@
version: 2
concurrency:
total_llm: 1
stage_workers:
extract: 1
workspace:
directory: /var/lib/notarius
diagnostics:

View File

@@ -9,26 +9,21 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/dnd/scenes"
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
"gitea.maximumdirect.net/eric/notarius/internal/modules/extract/dnd/spells"
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
spellshape "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/shape"
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_refs"
spellrelatedness "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_relatedness"
alwaysaccept "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_accept"
alwaysreject "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_reject"
validjson "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json"
validjsonschema "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json_schema"
dndregister "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/register"
genericregister "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/register"
seriatimregister "gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/register"
)
func productionRegistries() (pipeline.Registries, error) {
type productionComponents struct {
registries pipeline.Registries
assets *llm.AssetRegistry
}
func newProductionComponents() (productionComponents, error) {
registries := pipeline.Registries{
Inputs: pipeline.NewInputAdapterRegistry(),
Chunkers: pipeline.NewChunkerRegistry(),
ArtifactCodecs: pipeline.NewArtifactCodecRegistry(),
Extractors: pipeline.NewExtractorRegistry(),
Mergers: pipeline.NewMergerRegistry(),
Normalizers: pipeline.NewNormalizerRegistry(),
@@ -36,72 +31,26 @@ func productionRegistries() (pipeline.Registries, error) {
ValidatorChains: pipeline.NewValidatorChainRegistry(),
Outputs: pipeline.NewOutputEncoderRegistry(),
}
if err := seriatim.Register(registries.Inputs); err != nil {
return pipeline.Registries{}, fmt.Errorf("register seriatim input: %w", err)
}
if err := generic.Register(registries.Chunkers); err != nil {
return pipeline.Registries{}, fmt.Errorf("register generic chunker: %w", err)
}
if err := scenes.Register(registries.Chunkers); err != nil {
return pipeline.Registries{}, fmt.Errorf("register dnd scenes chunker: %w", err)
}
if err := spells.Register(registries.Extractors); err != nil {
return pipeline.Registries{}, fmt.Errorf("register dnd spells extractor: %w", err)
}
if err := appendorder.Register(registries.Mergers); err != nil {
return pipeline.Registries{}, fmt.Errorf("register appendorder merger: %w", err)
}
if err := noop.Register(registries.Normalizers); err != nil {
return pipeline.Registries{}, fmt.Errorf("register noop normalizer: %w", err)
}
if err := registerProductionValidators(registries.Validators); err != nil {
return pipeline.Registries{}, err
}
if err := registerProductionValidatorChains(registries.ValidatorChains); err != nil {
return pipeline.Registries{}, err
}
if err := jsonoutput.Register(registries.Outputs); err != nil {
return pipeline.Registries{}, fmt.Errorf("register json output encoder: %w", err)
}
return registries, nil
}
func registerProductionValidators(registry *pipeline.ValidatorRegistry) error {
registrations := []struct {
assets := llm.NewAssetRegistry()
registrars := []struct {
name string
register func(*pipeline.ValidatorRegistry) error
register func(pipeline.Registries, *llm.AssetRegistry) error
}{
{name: "generic always accept validator", register: alwaysaccept.Register},
{name: "generic always reject validator", register: alwaysreject.Register},
{name: "generic valid json validator", register: validjson.Register},
{name: "generic valid json schema validator", register: validjsonschema.Register},
{name: "dnd spell shape validator", register: spellshape.Register},
{name: "dnd spell source references validator", register: spellsourcerefs.Register},
{name: "dnd spell source relatedness validator", register: spellrelatedness.Register},
{name: "generic", register: genericregister.Register},
{name: "seriatim", register: seriatimregister.Register},
{name: "dnd", register: dndregister.Register},
}
for _, registration := range registrations {
if err := registration.register(registry); err != nil {
return fmt.Errorf("register %s: %w", registration.name, err)
for _, registrar := range registrars {
if err := registrar.register(registries, assets); err != nil {
return productionComponents{}, fmt.Errorf("register %s module family: %w", registrar.name, err)
}
}
return nil
return productionComponents{registries: registries, assets: assets}, nil
}
func registerProductionValidatorChains(registry *pipeline.ValidatorChainRegistry) error {
if err := registry.Register(pipeline.ValidatorChainMapping{
Stage: pipeline.StageExtract,
Module: spells.Key,
Validators: []pipeline.ModuleBinding{
pipeline.Binding(validjson.Key),
pipeline.Binding(validjsonschema.Key),
pipeline.Binding(spellshape.Key),
pipeline.Binding(spellsourcerefs.Key),
pipeline.Binding(spellrelatedness.Key),
},
}); err != nil {
return fmt.Errorf("register dnd spells validator chain: %w", err)
}
return nil
func productionRegistries() (pipeline.Registries, error) {
components, err := newProductionComponents()
return components.registries, err
}
func productionCatalog() (pipeline.ModuleCatalog, error) {
@@ -113,14 +62,8 @@ func productionCatalog() (pipeline.ModuleCatalog, error) {
}
func productionPromptAssets() (*llm.AssetRegistry, error) {
registry := llm.NewAssetRegistry()
if err := scenes.RegisterPromptAssets(registry); err != nil {
return nil, fmt.Errorf("register dnd scenes prompt assets: %w", err)
}
if err := spells.RegisterPromptAssets(registry); err != nil {
return nil, fmt.Errorf("register dnd spells prompt assets: %w", err)
}
return registry, nil
components, err := newProductionComponents()
return components.assets, err
}
func effectiveCatalog(opts Options) (pipeline.ModuleCatalog, error) {
@@ -147,6 +90,7 @@ func catalogFromRegistries(registries pipeline.Registries) pipeline.ModuleCatalo
return pipeline.ModuleCatalog{
Inputs: registries.Inputs,
Chunkers: registries.Chunkers,
ArtifactCodecs: registries.ArtifactCodecs,
Extractors: registries.Extractors,
Mergers: registries.Mergers,
Normalizers: registries.Normalizers,
@@ -160,6 +104,7 @@ func registriesFromCatalog(catalog pipeline.ModuleCatalog) pipeline.Registries {
return pipeline.Registries{
Inputs: catalog.Inputs,
Chunkers: catalog.Chunkers,
ArtifactCodecs: catalog.ArtifactCodecs,
Extractors: catalog.Extractors,
Mergers: catalog.Mergers,
Normalizers: catalog.Normalizers,
@@ -172,6 +117,7 @@ func registriesFromCatalog(catalog pipeline.ModuleCatalog) pipeline.Registries {
func isEmptyCatalog(catalog pipeline.ModuleCatalog) bool {
return catalog.Inputs == nil &&
catalog.Chunkers == nil &&
catalog.ArtifactCodecs == nil &&
catalog.Extractors == nil &&
catalog.Mergers == nil &&
catalog.Normalizers == nil &&
@@ -183,6 +129,7 @@ func isEmptyCatalog(catalog pipeline.ModuleCatalog) bool {
func isEmptyRegistries(registries pipeline.Registries) bool {
return registries.Inputs == nil &&
registries.Chunkers == nil &&
registries.ArtifactCodecs == nil &&
registries.Extractors == nil &&
registries.Mergers == nil &&
registries.Normalizers == nil &&
@@ -199,6 +146,22 @@ func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileI
if err != nil {
return nil, nil, err
}
return buildProductionLLMClient(ctx, cfg, profileID, assets)
}
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
return func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
return buildProductionLLMClient(ctx, cfg, profileID, assets)
}
}
func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
if err := ctx.Err(); err != nil {
return nil, nil, err
}
if assets == nil {
return nil, nil, fmt.Errorf("production asset registry must not be nil")
}
recorder := llm.NewLLMProfileRecorder()
client, err := llm.NewScriptoriumClient(llm.ScriptoriumClientConfig{
ProfileDir: cfg.Scriptorium.ProfileDir,

View File

@@ -0,0 +1,600 @@
package cli
import (
"bytes"
"context"
"encoding/json"
"io/fs"
"path/filepath"
"reflect"
"sort"
"strings"
"sync"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
frameworkllm "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
spellshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/shape"
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/source_refs"
spellrelatedness "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/source_relatedness"
validjson "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/valid_json"
validjsonschema "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/valid_json_schema"
)
func TestProductionCompatibilitySnapshot(t *testing.T) {
normalizedOptions, err := normalizeOptions(Options{})
if err != nil {
t.Fatalf("normalizeOptions() error = %v, want nil", err)
}
if normalizedOptions.Catalog.Inputs != normalizedOptions.Registries.Inputs ||
normalizedOptions.Catalog.Chunkers != normalizedOptions.Registries.Chunkers ||
normalizedOptions.Catalog.Extractors != normalizedOptions.Registries.Extractors ||
normalizedOptions.Catalog.Mergers != normalizedOptions.Registries.Mergers ||
normalizedOptions.Catalog.Normalizers != normalizedOptions.Registries.Normalizers ||
normalizedOptions.Catalog.Validators != normalizedOptions.Registries.Validators ||
normalizedOptions.Catalog.ValidatorChains != normalizedOptions.Registries.ValidatorChains ||
normalizedOptions.Catalog.Outputs != normalizedOptions.Registries.Outputs {
t.Fatal("production catalog and execution registries do not share one composition")
}
if normalizedOptions.LLMClientFactory == nil {
t.Fatal("production LLM client factory is nil")
}
registries, err := productionRegistries()
if err != nil {
t.Fatalf("productionRegistries() error = %v, want nil", err)
}
keySnapshots := []struct {
name string
got []string
want []string
}{
{name: "inputs", got: registries.Inputs.RegisteredKeys(), want: []string{"seriatim"}},
{name: "chunkers", got: registries.Chunkers.RegisteredKeys(), want: []string{"dnd/scenes", "generic"}},
{name: "extractors", got: registries.Extractors.RegisteredKeys(), want: []string{"dnd/spells"}},
{name: "mergers", got: registries.Mergers.RegisteredKeys(), want: []string{"appendorder"}},
{name: "normalizers", got: registries.Normalizers.RegisteredKeys(), want: []string{"noop"}},
{name: "validators", got: registries.Validators.RegisteredKeys(), want: []string{
"extract/dnd/spells/shape", "extract/dnd/spells/source_refs", "extract/dnd/spells/source_relatedness",
"generic/always_accept", "generic/always_reject", "generic/valid_json", "generic/valid_json_schema",
}},
{name: "outputs", got: registries.Outputs.RegisteredKeys(), want: []string{"json"}},
}
for _, snapshot := range keySnapshots {
t.Run(snapshot.name, func(t *testing.T) {
if !reflect.DeepEqual(snapshot.got, snapshot.want) {
t.Fatalf("registered keys = %#v, want compatibility snapshot %#v", snapshot.got, snapshot.want)
}
})
}
wantChain := []pipeline.ModuleBinding{
pipeline.Binding(validjson.Key),
pipeline.Binding(validjsonschema.Key),
pipeline.Binding(spellshape.Key),
pipeline.Binding(spellsourcerefs.Key),
pipeline.Binding(spellrelatedness.Key),
}
if got := registries.ValidatorChains.Validators(pipeline.StageExtract, spells.Key); !reflect.DeepEqual(got, wantChain) {
t.Fatalf("spell validator chain = %#v, want compatibility snapshot %#v", got, wantChain)
}
assets, err := productionPromptAssets()
if err != nil {
t.Fatalf("productionPromptAssets() error = %v, want nil", err)
}
assertAssetNames(t, assets.PromptFS, []string{
"dnd.scenes/dnd.scenes.yaml",
"dnd.scenes/instructions.md",
"dnd.scenes/sharedassets/common-dnd-references.md",
"dnd.scenes/sharedassets/common-dnd-system.md",
"dnd.scenes/sharedassets/common-dnd-transcript.md",
"dnd.scenes/task.md",
"dnd.spells/dnd.spells.yaml",
"dnd.spells/instructions.md",
"dnd.spells/sharedassets/common-dnd-references.md",
"dnd.spells/sharedassets/common-dnd-system.md",
"dnd.spells/sharedassets/common-dnd-transcript.md",
"dnd.spells/task.md",
})
assertAssetNames(t, assets.SchemaFS, []string{
"dnd_scenes.v1.json",
"dnd_spells_llm.v1.json",
})
identitySnapshot := map[string]map[string]any{
"scenes": sceneManifestMetadata(t),
"spells": spellManifestMetadata(t),
}
for name, metadata := range identitySnapshot {
for _, key := range []string{"prompt_id", "prompt_version", "prompt_sha256", "response_schema_key", "response_schema_id", "response_schema_name", "response_schema_version", "response_schema_sha256"} {
if value, ok := metadata[key].(string); !ok || value == "" {
t.Fatalf("%s metadata[%q] = %#v, want non-empty identity", name, key, metadata[key])
}
}
}
if got := []any{
identitySnapshot["scenes"]["prompt_id"], identitySnapshot["scenes"]["prompt_version"],
identitySnapshot["scenes"]["response_schema_key"], identitySnapshot["scenes"]["response_schema_id"], identitySnapshot["scenes"]["response_schema_name"], identitySnapshot["scenes"]["response_schema_version"],
}; !reflect.DeepEqual(got, []any{"dnd.scenes", "v1", "dnd_scenes", "notarius.dnd.scenes", "notarius_dnd_scenes_v1", "v1"}) {
t.Fatalf("scene identities = %#v, want compatibility snapshot", got)
}
if got := []any{
identitySnapshot["spells"]["prompt_id"], identitySnapshot["spells"]["prompt_version"],
identitySnapshot["spells"]["response_schema_key"], identitySnapshot["spells"]["response_schema_id"], identitySnapshot["spells"]["response_schema_name"], identitySnapshot["spells"]["response_schema_version"],
}; !reflect.DeepEqual(got, []any{"dnd.spells", "v1", "dnd_spells", "notarius.dnd.spells", "notarius_dnd_spells_v1", "v1"}) {
t.Fatalf("spell identities = %#v, want compatibility snapshot", got)
}
fileConfig, err := config.LoadFileConfig(fixturePath(t, "examples/dnd-spells.config.yml"))
if err != nil {
t.Fatalf("LoadFileConfig() error = %v, want nil", err)
}
cfg := config.Default()
if err := cfg.ApplyFileConfig(fileConfig); err != nil {
t.Fatalf("ApplyFileConfig() error = %v, want nil", err)
}
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-session", Catalog: catalogFromRegistries(registries)})
if err != nil {
t.Fatalf("Resolve() error = %v, want nil", err)
}
resolved := effective.ResolvedPipeline
if resolved.Input.Module != "seriatim" || resolved.Chunk.Module != "generic" || resolved.Output.Module != "json" || len(resolved.ArtifactLanes) != 1 {
t.Fatalf("resolved example = %#v, want maintained production topology", resolved)
}
lane := resolved.ArtifactLanes[0]
if lane.ID != "spells" || lane.Extract.Module != "dnd/spells" || lane.Merge.Module != "appendorder" || lane.Normalize.Module != "noop" {
t.Fatalf("resolved lane = %#v, want maintained spell lane", lane)
}
if got := resolvedValidatorKeys(resolved.ValidatorChains, pipeline.StageExtract, "spells", spells.Key); !reflect.DeepEqual(got, []string{
"generic/valid_json", "generic/valid_json_schema", "extract/dnd/spells/shape", "extract/dnd/spells/source_refs", "extract/dnd/spells/source_relatedness",
}) {
t.Fatalf("resolved validator keys = %#v, want compatibility snapshot", got)
}
productionFileConfig, err := config.LoadFileConfig(fixturePath(t, "examples/dnd-spells-production.config.yml"))
if err != nil {
t.Fatalf("LoadFileConfig(production example) error = %v, want nil", err)
}
productionConfig := config.Default()
if err := productionConfig.ApplyFileConfig(productionFileConfig); err != nil {
t.Fatalf("ApplyFileConfig(production example) error = %v, want nil", err)
}
productionEffective, err := productionConfig.Resolve(config.ResolveInput{PipelineID: "dnd-session", Catalog: catalogFromRegistries(registries)})
if err != nil {
t.Fatalf("Resolve(production example) error = %v, want nil", err)
}
productionResolved := productionEffective.ResolvedPipeline
if productionConfig.Concurrency.TotalLLM != 1 || productionConfig.Concurrency.StageWorkers["extract"] != 1 || !reflect.DeepEqual(productionResolved.Chunk.Options, map[string]any{"max_units": 50}) {
t.Fatalf("production example concurrency/options = %#v/%#v, want compatibility snapshot", productionConfig.Concurrency, productionResolved.Chunk.Options)
}
bindings := productionResolved.ArtifactLanes[0].ExtractReferences.Bindings
if len(bindings) != 2 || bindings[0].SlotName != "glossary" || bindings[0].Source != "./dnd-spells-glossary.txt" || bindings[1].SlotName != "party" || bindings[1].Source != "./dnd-spells-roster.txt" {
t.Fatalf("production example reference bindings = %#v, want maintained glossary and party bindings", bindings)
}
}
func TestMaintainedSeriatimToDNDCompatibilityBundle(t *testing.T) {
tests := []struct {
name string
client contracts.StructuredLLMClient
wantStatus string
wantLaneFile bool
wantRejectedCount int
}{
{name: "approved", client: newFakeRunLLMClient(false), wantStatus: "approved", wantLaneFile: true},
{name: "validator rejection is nonfatal", client: newFakeRunLLMClient(true), wantStatus: "rejected", wantRejectedCount: 1},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
outputDir := t.TempDir()
var stdout bytes.Buffer
var stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "dnd-session",
"--config", fixturePath(t, "examples/dnd-spells.config.yml"),
"--input", fixturePath(t, "examples/seriatim-minimal-transcript.json"),
"--output-dir", outputDir,
"--diagnostics-dir", t.TempDir(),
}, &stdout, &stderr, Options{LLMClientFactory: fakeLLMFactory(test.client, nil)})
if code != 0 {
t.Fatalf("RunWithOptions() code = %d, stderr=%q", code, stderr.String())
}
runDir := onlyChildDir(t, outputDir)
wantFiles := []string{"index.json", "manifest.json", "rejected.json", "warnings.json"}
if test.wantLaneFile {
wantFiles = append(wantFiles, "lanes/spells.json")
}
sort.Strings(wantFiles)
if got := relativeFileNames(t, runDir); !reflect.DeepEqual(got, wantFiles) {
t.Fatalf("durable files = %#v, want compatibility snapshot %#v", got, wantFiles)
}
var manifest artifacts.RunManifest
readJSONFile(t, filepath.Join(runDir, "manifest.json"), &manifest)
if manifest.PipelineID != "dnd-session" || manifest.InputModule != "seriatim" || manifest.Chunker != "generic" || manifest.OutputEncoder != "json" {
t.Fatalf("manifest module provenance = %#v, want maintained production modules", manifest)
}
if len(manifest.ArtifactLanes) != 1 || manifest.ArtifactLanes[0].ID != "spells" {
t.Fatalf("artifact lanes = %#v, want one spells lane", manifest.ArtifactLanes)
}
laneManifest := manifest.ArtifactLanes[0]
if laneManifest.Extractor != "dnd/spells" || laneManifest.Merger != "appendorder" || laneManifest.Normalizer != "noop" {
t.Fatalf("manifest lane module provenance = %#v, want maintained production modules", laneManifest)
}
if len(manifest.Extractors) != 0 || manifest.Merger != "" || manifest.Normalizer != "" {
t.Fatalf("legacy top-level lane summaries = %#v/%q/%q, want empty compatibility snapshot", manifest.Extractors, manifest.Merger, manifest.Normalizer)
}
if manifest.ValidationStatus != test.wantStatus || len(manifest.RejectedOutputs) != test.wantRejectedCount {
t.Fatalf("manifest outcome = status %q rejected %#v, want %q/%d", manifest.ValidationStatus, manifest.RejectedOutputs, test.wantStatus, test.wantRejectedCount)
}
if !reflect.DeepEqual(manifest.SourceDigests, []string{"sha256:1c98d94ae632fb10a2b56f684cd4fb1019cedb1a629e57dc0977cf4a54135be0"}) {
t.Fatalf("source digests = %#v, want maintained fixture provenance", manifest.SourceDigests)
}
if got := manifestValidatorKeys(manifestValidatorChain(t, manifest, pipeline.StageExtract, "spells", spells.Key)); !reflect.DeepEqual(got, []string{
"generic/valid_json", "generic/valid_json_schema", "extract/dnd/spells/shape", "extract/dnd/spells/source_refs", "extract/dnd/spells/source_relatedness",
}) {
t.Fatalf("manifest validator chain = %#v, want compatibility snapshot", got)
}
var index struct {
ManifestFile string `json:"manifest_file"`
OutputFiles []struct {
LaneID string `json:"lane_id"`
MediaType string `json:"media_type"`
File string `json:"file"`
ModuleKey string `json:"module_key"`
SchemaID string `json:"schema_id"`
SchemaName string `json:"schema_name"`
SchemaVersion string `json:"schema_version"`
} `json:"output_files"`
RejectedFile string `json:"rejected_file"`
WarningsFile string `json:"warnings_file"`
}
readJSONFile(t, filepath.Join(runDir, "index.json"), &index)
if index.ManifestFile != "manifest.json" || index.RejectedFile != "rejected.json" || index.WarningsFile != "warnings.json" {
t.Fatalf("output index fixed files = %#v, want compatibility snapshot", index)
}
if test.wantLaneFile {
if len(index.OutputFiles) != 1 {
t.Fatalf("output index entries = %#v, want one", index.OutputFiles)
}
wantOutput := struct {
LaneID, MediaType, File, ModuleKey, SchemaID, SchemaName, SchemaVersion string
}{"spells", "application/json", "lanes/spells.json", "noop", "notarius.dnd.spells", "notarius_dnd_spells_v1", "v1"}
gotOutput := index.OutputFiles[0]
got := struct {
LaneID, MediaType, File, ModuleKey, SchemaID, SchemaName, SchemaVersion string
}{gotOutput.LaneID, gotOutput.MediaType, gotOutput.File, gotOutput.ModuleKey, gotOutput.SchemaID, gotOutput.SchemaName, gotOutput.SchemaVersion}
if got != wantOutput {
t.Fatalf("output index entries = %#v, want compatibility snapshot %#v", index.OutputFiles, wantOutput)
}
assertJSONEqual(t, readFile(t, filepath.Join(runDir, "lanes/spells.json")), []byte(`{
"spell_casts": [{
"caster": "Aria",
"spell": "Cure Wounds",
"effect": "Heals a wounded ally.",
"narrative_description": "Aria casts Cure Wounds.",
"source_refs": [{"source_id": "session-alpha", "start_unit_id": 1, "end_unit_id": 1}]
}]
}`))
} else if len(index.OutputFiles) != 0 {
t.Fatalf("output index entries = %#v, want none for rejected lane", index.OutputFiles)
}
var warnings struct {
Warnings []contracts.Warning `json:"warnings"`
}
readJSONFile(t, filepath.Join(runDir, "warnings.json"), &warnings)
if len(warnings.Warnings) != 0 {
t.Fatalf("warnings = %#v, want empty compatibility snapshot", warnings.Warnings)
}
var rejected struct {
Rejected []contracts.RejectedOutput `json:"rejected"`
}
readJSONFile(t, filepath.Join(runDir, "rejected.json"), &rejected)
if len(rejected.Rejected) != test.wantRejectedCount {
t.Fatalf("rejected outputs = %#v, want %d", rejected.Rejected, test.wantRejectedCount)
}
if test.wantRejectedCount == 1 {
got := rejected.Rejected[0]
if got.Stage != "extract" || got.LaneID != "spells" || got.ModuleKey != "dnd/spells" || got.ChunkID != "chunk-000001" || got.ChunkIndex != 0 || got.ValidatorName != "extract/dnd/spells/source_refs" || got.ReasonCode != "invalid_source_refs" || got.AttemptCount != 1 {
t.Fatalf("rejection = %#v, want maintained nonfatal validator outcome", got)
}
}
})
}
}
func TestProductionLLMCallersShareScheduledClient(t *testing.T) {
underlying := newBlockingProductionLLMClient()
scheduler, err := frameworkllm.NewScheduler(1)
if err != nil {
t.Fatalf("NewScheduler() error = %v, want nil", err)
}
client := frameworkllm.NewScheduledClient(underlying, scheduler)
doc := &source.SourceDocument{
ID: "session-alpha", Kind: "transcript", Format: "application/json", Digest: "sha256:source",
Units: []source.SourceUnit{
{ID: 1, Kind: "segment", Text: "Aria casts Cure Wounds.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 1}},
{ID: 2, Kind: "segment", Text: "The spell takes effect.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 2}},
},
}
chunk := source.Chunk{
ID: "session-alpha:chunk:0", SourceID: doc.ID, Index: 0, Ref: source.SourceRef{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 2},
Content: []byte(`{"scene":"Aria casts Cure Wounds."}`), MediaType: "application/json", Units: append([]source.SourceUnit(nil), doc.Units...),
}
var started sync.WaitGroup
started.Add(2)
errs := make(chan error, 2)
go func() {
started.Done()
chunker, err := scenes.New(client, scenes.Options{})
if err == nil {
_, err = chunker.Chunk(context.Background(), contracts.ChunkRequest{Source: doc})
}
errs <- err
}()
go func() {
started.Done()
extractor, err := spells.New(client, spells.Options{})
if err == nil {
_, err = extractor.Extract(context.Background(), contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk})
}
errs <- err
}()
started.Wait()
for i := 0; i < 2; i++ {
<-underlying.entered
underlying.release <- struct{}{}
}
for i := 0; i < 2; i++ {
if err := <-errs; err != nil {
t.Fatalf("production LLM caller error = %v, want nil", err)
}
}
if underlying.maxActive != 1 {
t.Fatalf("maximum concurrent provider calls = %d, want total_llm limit 1", underlying.maxActive)
}
sort.Strings(underlying.stageNames)
if !reflect.DeepEqual(underlying.stageNames, []string{"dnd/scenes", "dnd/spells"}) {
t.Fatalf("scheduled stage names = %#v, want both production LLM callers", underlying.stageNames)
}
}
func TestProductionBundlePreservesLaneAndChunkOrder(t *testing.T) {
configPath := writeTestConfig(t, `version: 2
pipelines:
dnd-session:
input: seriatim
chunk:
module: generic
options:
max_units: 1
artifacts:
zeta:
extract: dnd/spells
alpha:
extract: dnd/spells
`)
outputDir := t.TempDir()
var stdout bytes.Buffer
var stderr bytes.Buffer
code := RunWithOptions([]string{
"run", "dnd-session",
"--config", configPath,
"--input", fixturePath(t, "examples/seriatim-minimal-transcript.json"),
"--output-dir", outputDir,
"--diagnostics-dir", t.TempDir(),
}, &stdout, &stderr, Options{LLMClientFactory: fakeLLMFactory(orderingProductionLLMClient{}, nil)})
if code != 0 {
t.Fatalf("RunWithOptions() code = %d, stderr=%q", code, stderr.String())
}
runDir := onlyChildDir(t, outputDir)
var index struct {
OutputFiles []struct {
LaneID string `json:"lane_id"`
} `json:"output_files"`
}
readJSONFile(t, filepath.Join(runDir, "index.json"), &index)
if len(index.OutputFiles) != 2 {
t.Fatalf("output index entries = %#v, want two lanes", index.OutputFiles)
}
if got := []string{index.OutputFiles[0].LaneID, index.OutputFiles[1].LaneID}; !reflect.DeepEqual(got, []string{"alpha", "zeta"}) {
t.Fatalf("output lane order = %#v, want resolved lane order", got)
}
for _, laneID := range []string{"alpha", "zeta"} {
var payload struct {
SpellCasts []struct {
Spell string `json:"spell"`
SourceRefs []source.SourceRef `json:"source_refs"`
} `json:"spell_casts"`
}
readJSONFile(t, filepath.Join(runDir, "lanes", laneID+".json"), &payload)
if len(payload.SpellCasts) != 2 {
t.Fatalf("lane %q spell casts = %#v, want one per source chunk", laneID, payload.SpellCasts)
}
got := []any{
payload.SpellCasts[0].Spell, payload.SpellCasts[0].SourceRefs[0].StartUnitID,
payload.SpellCasts[1].Spell, payload.SpellCasts[1].SourceRefs[0].StartUnitID,
}
if !reflect.DeepEqual(got, []any{"Cure Wounds", 1, "Shield", 2}) {
t.Fatalf("lane %q chunk handoff order = %#v, want source chunk order", laneID, got)
}
}
}
type blockingProductionLLMClient struct {
mu sync.Mutex
active int
maxActive int
stageNames []string
entered chan struct{}
release chan struct{}
}
type orderingProductionLLMClient struct{}
func (orderingProductionLLMClient) CompleteStructured(_ context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
material := req.Inputs["transcript"]
unitID := 1
spellName := "Cure Wounds"
if strings.Contains(string(material.Content), "Shield") {
unitID = 2
spellName = "Shield"
}
payload, err := json.Marshal(map[string]any{
"spell_casts": []map[string]any{{
"caster": "Aria",
"spell": spellName,
"effect": "Fixture effect.",
"narrative_description": "Fixture spell cast.",
"source_refs": []map[string]any{{
"start_unit_id": unitID,
"end_unit_id": unitID,
}},
}},
})
if err != nil {
return contracts.StructuredCompletionResponse{}, err
}
if err := json.Unmarshal(payload, out); err != nil {
return contracts.StructuredCompletionResponse{}, err
}
return contracts.StructuredCompletionResponse{Content: payload}, nil
}
func newBlockingProductionLLMClient() *blockingProductionLLMClient {
return &blockingProductionLLMClient{entered: make(chan struct{}, 2), release: make(chan struct{}, 2)}
}
func (client *blockingProductionLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
client.mu.Lock()
client.active++
if client.active > client.maxActive {
client.maxActive = client.active
}
client.stageNames = append(client.stageNames, req.StageName)
client.mu.Unlock()
client.entered <- struct{}{}
select {
case <-ctx.Done():
return contracts.StructuredCompletionResponse{}, ctx.Err()
case <-client.release:
}
client.mu.Lock()
client.active--
client.mu.Unlock()
var payload []byte
switch req.StageName {
case scenes.Key:
payload = []byte(`{"scenes":[{"start_unit_id":1,"end_unit_id":2,"short_title":"Spell","primary_mode":"Narrative","main_participants":["Aria"],"summary":"Aria casts a spell.","boundary_note":"Complete source.","boundary_confidence":"High"}],"boundary_caveats":[]}`)
case spells.Key:
payload = []byte(`{"spell_casts":[{"caster":"Aria","spell":"Cure Wounds","effect":"Healing","narrative_description":"Aria casts Cure Wounds.","source_refs":[{"start_unit_id":1,"end_unit_id":1}]}]}`)
}
if err := json.Unmarshal(payload, out); err != nil {
return contracts.StructuredCompletionResponse{}, err
}
return contracts.StructuredCompletionResponse{Content: payload}, nil
}
func sceneManifestMetadata(t *testing.T) map[string]any {
t.Helper()
chunker, err := scenes.New(orderingProductionLLMClient{}, scenes.Options{})
if err != nil {
t.Fatalf("construct scene chunker: %v", err)
}
return chunker.ManifestMetadata()
}
func spellManifestMetadata(t *testing.T) map[string]any {
t.Helper()
extractor, err := spells.New(orderingProductionLLMClient{}, spells.Options{})
if err != nil {
t.Fatalf("construct spell extractor: %v", err)
}
return extractor.ManifestMetadata()
}
func assertAssetNames(t *testing.T, getFS func() (fs.FS, error), want []string) {
t.Helper()
fSys, err := getFS()
if err != nil {
t.Fatalf("asset filesystem error = %v, want nil", err)
}
var got []string
if err := fs.WalkDir(fSys, ".", func(path string, entry fs.DirEntry, err error) error {
if err == nil && !entry.IsDir() {
got = append(got, path)
}
return err
}); err != nil {
t.Fatalf("walk assets: %v", err)
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("asset names = %#v, want compatibility snapshot %#v", got, want)
}
}
func resolvedValidatorKeys(chains []pipeline.ResolvedValidatorChain, stage pipeline.ModuleStage, laneID, moduleKey string) []string {
for _, chain := range chains {
if chain.Stage == stage && chain.LaneID == laneID && chain.ModuleKey == moduleKey {
keys := make([]string, 0, len(chain.Validators))
for _, validator := range chain.Validators {
keys = append(keys, validator.Binding.Module)
}
return keys
}
}
return nil
}
func relativeFileNames(t *testing.T, root string) []string {
t.Helper()
var names []string
if err := filepath.WalkDir(root, func(path string, entry fs.DirEntry, err error) error {
if err != nil || entry.IsDir() {
return err
}
rel, err := filepath.Rel(root, path)
if err != nil {
return err
}
names = append(names, filepath.ToSlash(rel))
return nil
}); err != nil {
t.Fatalf("walk durable output: %v", err)
}
sort.Strings(names)
return names
}
func assertJSONEqual(t *testing.T, got, want []byte) {
t.Helper()
var gotValue any
var wantValue any
if err := json.Unmarshal(got, &gotValue); err != nil {
t.Fatalf("unmarshal actual JSON: %v", err)
}
if err := json.Unmarshal(want, &wantValue); err != nil {
t.Fatalf("unmarshal expected JSON: %v", err)
}
if !reflect.DeepEqual(gotValue, wantValue) {
t.Fatalf("JSON = %#v, want compatibility snapshot %#v", gotValue, wantValue)
}
}

View File

@@ -51,7 +51,12 @@ func Run(args []string, stdout, stderr io.Writer) int {
}
func RunWithOptions(args []string, stdout, stderr io.Writer, opts Options) int {
opts = normalizeOptions(opts)
var err error
opts, err = normalizeOptions(opts)
if err != nil {
fmt.Fprintf(stderr, "notarius: %v\n", err)
return 1
}
if len(args) == 0 {
writeUsage(stdout)
return 0
@@ -78,17 +83,28 @@ func writeUsage(w io.Writer) {
fmt.Fprint(w, usage)
}
func normalizeOptions(opts Options) Options {
func normalizeOptions(opts Options) (Options, error) {
if opts.LookupEnv == nil {
opts.LookupEnv = os.LookupEnv
}
if opts.Now == nil {
opts.Now = time.Now
}
if isEmptyCatalog(opts.Catalog) && isEmptyRegistries(opts.Registries) {
components, err := newProductionComponents()
if err != nil {
return Options{}, err
}
opts.Registries = components.registries
opts.Catalog = catalogFromRegistries(components.registries)
if opts.LLMClientFactory == nil {
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(components.assets)
}
}
if opts.LLMClientFactory == nil {
opts.LLMClientFactory = productionLLMClientFactory
}
return opts
return opts, nil
}
func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) int {
@@ -195,6 +211,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("create debug recorder: %w", err))
}
debugRecorder = pipeline.SynchronizedDebugRecorder(debugRecorder)
catalog, err := effectiveCatalog(opts)
if err != nil {
@@ -247,11 +264,6 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics resolved references: %w", err))
}
rawInput, err := os.ReadFile(strings.TrimSpace(*inputPath))
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("read input %q: %w", strings.TrimSpace(*inputPath), err))
}
registries, err := effectiveRegistries(opts)
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
@@ -266,25 +278,34 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("create LLM client for profile %q: %w", factoryProfileID, err))
}
llmClient = pipeline.WithDebugLLMRecording(llmClient, debugRecorder)
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, registries, pipeline.ModuleDependencies{LLM: llmClient})
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("prepare pipeline %q: %w", pipelineID, err))
}
rawInput, err := os.ReadFile(strings.TrimSpace(*inputPath))
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("read input %q: %w", strings.TrimSpace(*inputPath), err))
}
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(workspaceSettings, effective.ResolvedPipeline, rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), *resume)
if err != nil {
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
}
output, err := pipeline.New(registries).Run(ctx, pipeline.RunInput{
Pipeline: effective.ResolvedPipeline,
Path: strings.TrimSpace(*inputPath),
RawInput: rawInput,
LLMClient: llmClient,
SessionID: strings.TrimSpace(sessionID.value),
RunID: runID,
StartedAt: startedAt,
LLMProfiles: llmProfiles,
Metadata: runMetadata(*outputDir, *diagnosticsDir),
Warnings: referenceWarnings,
Checkpoints: checkpointRecorder,
Checkpoint: checkpointLoader,
Debug: debugRecorder,
output, err := pipeline.New().Run(ctx, pipeline.RunInput{
Prepared: prepared,
Path: strings.TrimSpace(*inputPath),
RawInput: rawInput,
SessionID: strings.TrimSpace(sessionID.value),
RunID: runID,
StartedAt: startedAt,
LLMProfiles: llmProfiles,
Metadata: runMetadata(*outputDir, *diagnosticsDir),
Warnings: referenceWarnings,
Checkpoints: checkpointRecorder,
Checkpoint: checkpointLoader,
Debug: debugRecorder,
ExtractWorkers: cfg.Concurrency.StageWorkers["extract"],
})
if err != nil {
if output.Manifest.PipelineID != "" && runDir != nil {

View File

@@ -22,20 +22,20 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/dnd/scenes"
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
"gitea.maximumdirect.net/eric/notarius/internal/modules/extract/dnd/spells"
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
spellshape "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/shape"
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_refs"
spellrelatedness "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_relatedness"
alwaysaccept "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_accept"
alwaysreject "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_reject"
validjson "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json"
validjsonschema "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json_schema"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
spellshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/shape"
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/source_refs"
spellrelatedness "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/source_relatedness"
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/chunk/units"
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/merge/appendorder"
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/output/json"
alwaysaccept "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/always_accept"
alwaysreject "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/always_reject"
validjson "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/valid_json"
validjsonschema "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/valid_json_schema"
"gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/input/transcript"
"gitea.maximumdirect.net/eric/scriptorium"
)
@@ -135,6 +135,12 @@ func TestProductionCatalogIncludesProductionModulesValidatorsAndDefaults(t *test
if err != nil {
t.Fatalf("productionCatalog() error = %v, want nil", err)
}
if catalog.ArtifactCodecs == nil {
t.Fatal("production artifact codec registry = nil, want initialized empty registry")
}
if got := catalog.ArtifactCodecs.RegisteredKinds(); !reflect.DeepEqual(got, []contracts.ArtifactKind{"dnd/spell-list"}) {
t.Fatalf("production artifact codec kinds = %#v, want dnd/spell-list", got)
}
moduleTests := []struct {
name string
@@ -143,13 +149,13 @@ func TestProductionCatalogIncludesProductionModulesValidatorsAndDefaults(t *test
}{
{
name: "seriatim input",
got: func() (pipeline.ModuleSpec, bool) { return catalog.Inputs.Spec(seriatim.Key) },
want: seriatim.ModuleSpec(),
got: func() (pipeline.ModuleSpec, bool) { return catalog.Inputs.Spec(transcript.Key) },
want: transcript.ModuleSpec(),
},
{
name: "generic chunker",
got: func() (pipeline.ModuleSpec, bool) { return catalog.Chunkers.Spec(generic.Key) },
want: generic.ModuleSpec(),
got: func() (pipeline.ModuleSpec, bool) { return catalog.Chunkers.Spec(units.Key) },
want: units.ModuleSpec(),
},
{
name: "dnd scenes chunker",
@@ -164,12 +170,12 @@ func TestProductionCatalogIncludesProductionModulesValidatorsAndDefaults(t *test
{
name: "appendorder merger",
got: func() (pipeline.ModuleSpec, bool) { return catalog.Mergers.Spec(appendorder.Key) },
want: appendorder.ModuleSpec(),
want: appendorder.TypedModuleSpec(spells.ModuleSpec().ArtifactKind),
},
{
name: "noop normalizer",
got: func() (pipeline.ModuleSpec, bool) { return catalog.Normalizers.Spec(noop.Key) },
want: noop.ModuleSpec(),
want: noop.TypedModuleSpec(spells.ModuleSpec().ArtifactKind),
},
{
name: "json output",
@@ -222,11 +228,28 @@ func TestProductionCatalogIncludesProductionModulesValidatorsAndDefaults(t *test
if !reflect.DeepEqual(gotChain, wantChain) {
t.Fatalf("dnd spell default validator chain = %#v, want %#v", gotChain, wantChain)
}
if got := catalog.ValidatorChains.Validators(pipeline.StageChunk, generic.Key); len(got) != 0 {
if got := catalog.ValidatorChains.Validators(pipeline.StageChunk, units.Key); len(got) != 0 {
t.Fatalf("generic chunker default validator chain = %#v, want empty", got)
}
}
func TestCatalogConversionsPreserveArtifactCodecRegistry(t *testing.T) {
codecs := pipeline.NewArtifactCodecRegistry()
registries := pipeline.Registries{ArtifactCodecs: codecs}
if isEmptyRegistries(registries) {
t.Fatal("registries with artifact codecs reported empty")
}
catalog := catalogFromRegistries(registries)
if catalog.ArtifactCodecs != codecs || isEmptyCatalog(catalog) {
t.Fatalf("catalog artifact codecs = %p empty=%t, want %p and non-empty", catalog.ArtifactCodecs, isEmptyCatalog(catalog), codecs)
}
converted := registriesFromCatalog(catalog)
if converted.ArtifactCodecs != codecs {
t.Fatalf("converted artifact codecs = %p, want %p", converted.ArtifactCodecs, codecs)
}
}
func TestProductionPromptAssetsRegisterAndPrepareDndPrompts(t *testing.T) {
registry, err := productionPromptAssets()
if err != nil {
@@ -3533,6 +3556,10 @@ func fakeExecutionRegistries(t *testing.T) pipeline.Registries {
mergers := pipeline.NewMergerRegistry()
normalizers := pipeline.NewNormalizerRegistry()
outputs := pipeline.NewOutputEncoderRegistry()
codecs := pipeline.NewArtifactCodecRegistry()
if err := pipeline.RegisterArtifactCodec(codecs, fakeRunCodec{}); err != nil {
t.Fatal(err)
}
if err := inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "fake/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
return fakeRunInputAdapter{}, nil
@@ -3544,25 +3571,25 @@ func fakeExecutionRegistries(t *testing.T) pipeline.Registries {
}); err != nil {
t.Fatalf("register fake chunker: %v", err)
}
if err := extractors.RegisterWithSpec(pipeline.ModuleSpec{
if err := pipeline.RegisterExtractor(extractors, pipeline.ModuleSpec{
Key: "fake/extract",
Stage: pipeline.StageExtract,
Requires: []string{"chunks"},
Provides: []string{"artifact"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster"},
},
}, func() (contracts.Extractor, error) {
}, ArtifactKind: fakeRunArtifactKind,
}, func() (contracts.Extractor[fakeRunArtifact], error) {
return fakeRunExtractor{}, nil
}); err != nil {
t.Fatalf("register fake extractor: %v", err)
}
if err := mergers.RegisterWithSpec(pipeline.ModuleSpec{Key: "appendorder", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}}, func() (contracts.Merger, error) {
if err := pipeline.RegisterMerger(mergers, pipeline.ModuleSpec{Key: "appendorder", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: fakeRunArtifactKind}, func() (contracts.Merger[fakeRunArtifact], error) {
return fakeRunMerger{}, nil
}); err != nil {
t.Fatalf("register fake merger: %v", err)
}
if err := normalizers.RegisterWithSpec(pipeline.ModuleSpec{Key: "noop", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer, error) {
if err := pipeline.RegisterNormalizer(normalizers, pipeline.ModuleSpec{Key: "noop", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: fakeRunArtifactKind}, func() (contracts.Normalizer[fakeRunArtifact], error) {
return fakeRunNormalizer{}, nil
}); err != nil {
t.Fatalf("register fake normalizer: %v", err)
@@ -3572,12 +3599,13 @@ func fakeExecutionRegistries(t *testing.T) pipeline.Registries {
}
return pipeline.Registries{
Inputs: inputs,
Chunkers: chunkers,
Extractors: extractors,
Mergers: mergers,
Normalizers: normalizers,
Outputs: outputs,
Inputs: inputs,
Chunkers: chunkers,
ArtifactCodecs: codecs,
Extractors: extractors,
Mergers: mergers,
Normalizers: normalizers,
Outputs: outputs,
}
}
@@ -3594,7 +3622,7 @@ func (fakeRunInputAdapter) Parse(ctx context.Context, req contracts.ParseRequest
Format: "test",
Digest: "sha256:source",
Units: []source.SourceUnit{
{ID: 1, Kind: "text", Text: string(req.Raw)},
{ID: 1, Kind: "text", Text: string(req.Raw), Ref: source.SourceRef{SourceID: "source", StartUnitID: 1, EndUnitID: 1}},
},
}, nil
}
@@ -3611,14 +3639,33 @@ func (fakeRunChunker) ReferenceSlots() []contracts.ReferenceSlot {
func (fakeRunChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
return contracts.ChunkResult{
Chunks: []contracts.SourceChunk{
{ID: "chunk-1", SourceID: req.Source.ID, Index: 0, StartUnitID: 1, EndUnitID: 1, Content: []byte(`{"units":[1]}`), MediaType: "application/json", Units: req.Source.Units},
Chunks: []source.Chunk{
{ID: "chunk-1", SourceID: req.Source.ID, Index: 0, Ref: source.SourceRef{SourceID: req.Source.ID, StartUnitID: 1, EndUnitID: 1}, Content: []byte(`{"units":[1]}`), MediaType: "application/json", Units: req.Source.Units},
},
}, nil
}
type fakeRunExtractor struct{}
const fakeRunArtifactKind contracts.ArtifactKind = "test/fake"
type fakeRunArtifact struct {
Value bool `json:"value"`
}
type fakeRunCodec struct{}
func (fakeRunCodec) Kind() contracts.ArtifactKind { return fakeRunArtifactKind }
func (fakeRunCodec) Schema() contracts.ArtifactSchema {
return contracts.ArtifactSchema{ID: "fake.artifact", Name: "fake_artifact", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
}
func (fakeRunCodec) MediaType() string { return "application/json" }
func (fakeRunCodec) Encode(v fakeRunArtifact) ([]byte, error) { return json.Marshal(v) }
func (fakeRunCodec) Decode(b []byte) (fakeRunArtifact, error) {
var v fakeRunArtifact
err := json.Unmarshal(b, &v)
return v, err
}
func (fakeRunExtractor) Key() string {
return "fake/extract"
}
@@ -3627,16 +3674,8 @@ func (fakeRunExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return []contracts.ReferenceSlot{{Name: "roster"}}
}
func (fakeRunExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
return contracts.ExtractionResult{
Output: contracts.ExtractOutput{
Schema: contracts.ResponseSchema{ID: "fake.artifact", Name: "fake_artifact", Version: "v1"},
Payload: contracts.RawPayload{
Content: []byte(`{"value":true}`),
MediaType: "application/json",
},
},
}, nil
func (fakeRunExtractor) Extract(ctx context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[fakeRunArtifact], error) {
return contracts.TypedExtractionResult[fakeRunArtifact]{Value: fakeRunArtifact{Value: true}}, nil
}
type fakeRunMerger struct{}
@@ -3645,21 +3684,11 @@ func (fakeRunMerger) Key() string {
return "appendorder"
}
func (fakeRunMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
output := contracts.MergeOutput{
LaneID: req.LaneID,
SourceID: req.Source.ID,
Schema: contracts.ResponseSchema{ID: "fake.artifact", Name: "fake_artifact", Version: "v1"},
Payload: contracts.RawPayload{
Content: []byte(`{"merged":true}`),
MediaType: "application/json",
},
}
func (fakeRunMerger) Merge(ctx context.Context, req contracts.TypedMergeRequest[fakeRunArtifact]) (contracts.TypedMergeResult[fakeRunArtifact], error) {
if len(req.ExtractOutputs) > 0 {
output.Schema = req.ExtractOutputs[0].Schema
output.Payload = req.ExtractOutputs[0].Payload
return contracts.TypedMergeResult[fakeRunArtifact]{Value: req.ExtractOutputs[0].Value}, nil
}
return contracts.MergeResult{Output: output}, nil
return contracts.TypedMergeResult[fakeRunArtifact]{}, nil
}
type fakeRunNormalizer struct{}
@@ -3672,15 +3701,8 @@ func (fakeRunNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (fakeRunNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
return contracts.NormalizeResult{
Output: contracts.NormalizeOutput{
LaneID: req.LaneID,
SourceID: req.MergeOutput.SourceID,
Schema: req.MergeOutput.Schema,
Payload: req.MergeOutput.Payload,
},
}, nil
func (fakeRunNormalizer) Normalize(ctx context.Context, req contracts.TypedNormalizeRequest[fakeRunArtifact]) (contracts.TypedNormalizeResult[fakeRunArtifact], error) {
return contracts.TypedNormalizeResult[fakeRunArtifact]{Value: req.MergeOutput.Value}, nil
}
func onlyChildDir(t *testing.T, root string) string {
@@ -3942,9 +3964,18 @@ func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.Module
for _, override := range overrides {
specs[override.Key] = override
}
for _, key := range []string{"fake/extract", "appendorder", "noop"} {
spec := specs[key]
spec.ArtifactKind = fakeRunArtifactKind
specs[key] = spec
}
mustRegisterInput(t, inputs, specs["fake/input"])
mustRegisterChunker(t, chunkers, specs["generic"])
codecs := pipeline.NewArtifactCodecRegistry()
if err := pipeline.RegisterArtifactCodec(codecs, fakeRunCodec{}); err != nil {
t.Fatal(err)
}
mustRegisterExtractor(t, extractors, specs["fake/extract"])
mustRegisterMerger(t, mergers, specs["appendorder"])
mustRegisterNormalizer(t, normalizers, specs["noop"])
@@ -3953,6 +3984,7 @@ func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.Module
return pipeline.ModuleCatalog{
Inputs: inputs,
Chunkers: chunkers,
ArtifactCodecs: codecs,
Extractors: extractors,
Mergers: mergers,
Normalizers: normalizers,
@@ -3964,53 +3996,58 @@ func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.Module
func mustRegisterInput(t *testing.T, registry *pipeline.InputAdapterRegistry, spec pipeline.ModuleSpec) {
t.Helper()
if err := registry.RegisterWithSpec(spec, func() (contracts.InputAdapter, error) { return nil, nil }); err != nil {
if err := registry.RegisterWithSpec(spec, func() (contracts.InputAdapter, error) { return fakeRunInputAdapter{}, nil }); err != nil {
t.Fatalf("register input: %v", err)
}
}
func mustRegisterChunker(t *testing.T, registry *pipeline.ChunkerRegistry, spec pipeline.ModuleSpec) {
t.Helper()
if err := registry.RegisterWithSpec(spec, func() (contracts.Chunker, error) { return nil, nil }); err != nil {
if err := registry.RegisterWithSpec(spec, func() (contracts.Chunker, error) { return fakeRunChunker{}, nil }); err != nil {
t.Fatalf("register chunker: %v", err)
}
}
func mustRegisterExtractor(t *testing.T, registry *pipeline.ExtractorRegistry, spec pipeline.ModuleSpec) {
t.Helper()
if err := registry.RegisterWithSpec(spec, func() (contracts.Extractor, error) { return nil, nil }); err != nil {
if err := pipeline.RegisterExtractor(registry, spec, func() (contracts.Extractor[fakeRunArtifact], error) { return fakeRunExtractor{}, nil }); err != nil {
t.Fatalf("register extractor: %v", err)
}
}
func mustRegisterMerger(t *testing.T, registry *pipeline.MergerRegistry, spec pipeline.ModuleSpec) {
t.Helper()
if err := registry.RegisterWithSpec(spec, func() (contracts.Merger, error) { return nil, nil }); err != nil {
if err := pipeline.RegisterMerger(registry, spec, func() (contracts.Merger[fakeRunArtifact], error) { return fakeRunMerger{}, nil }); err != nil {
t.Fatalf("register merger: %v", err)
}
}
func mustRegisterNormalizer(t *testing.T, registry *pipeline.NormalizerRegistry, spec pipeline.ModuleSpec) {
t.Helper()
if err := registry.RegisterWithSpec(spec, func() (contracts.Normalizer, error) { return nil, nil }); err != nil {
if err := pipeline.RegisterNormalizer(registry, spec, func() (contracts.Normalizer[fakeRunArtifact], error) { return fakeRunNormalizer{}, nil }); err != nil {
t.Fatalf("register normalizer: %v", err)
}
}
func mustRegisterOutput(t *testing.T, registry *pipeline.OutputEncoderRegistry, spec pipeline.ModuleSpec) {
t.Helper()
if err := registry.RegisterWithSpec(spec, func() (contracts.OutputEncoder, error) { return nil, nil }); err != nil {
if err := registry.RegisterWithSpec(spec, func() (contracts.OutputEncoder, error) { return jsonoutput.New(), nil }); err != nil {
t.Fatalf("register output: %v", err)
}
}
func mustRegisterValidator(t *testing.T, registry *pipeline.ValidatorRegistry, spec pipeline.ValidatorSpec) {
t.Helper()
if err := registry.RegisterWithSpec(spec, func() (contracts.Validator, error) {
if err := pipeline.RegisterChunkValidator(registry, spec, func() (contracts.ChunkValidator, error) {
return fakeConfigValidator{name: spec.Key, executionClass: spec.ExecutionClass}, nil
}); err != nil {
t.Fatalf("register validator: %v", err)
}
if err := pipeline.RegisterTypedValidator[fakeRunArtifact](registry, fakeRunArtifactKind, spec, func() (contracts.TypedValidator[fakeRunArtifact], error) {
return fakeConfigTypedValidator{fakeConfigValidator{name: spec.Key, executionClass: spec.ExecutionClass}}, nil
}); err != nil {
t.Fatalf("register typed validator: %v", err)
}
}
type fakeConfigValidator struct {
@@ -4026,7 +4063,13 @@ func (validator fakeConfigValidator) ExecutionClass() contracts.ExecutionClass {
return validator.executionClass
}
func (validator fakeConfigValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
func (validator fakeConfigValidator) Validate(ctx context.Context, req contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: true}, nil
}
type fakeConfigTypedValidator struct{ fakeConfigValidator }
func (validator fakeConfigTypedValidator) Validate(ctx context.Context, req contracts.TypedValidationRequest[fakeRunArtifact]) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: true}, nil
}

View File

@@ -24,7 +24,11 @@ type ScriptoriumConfig struct {
}
type ConcurrencyConfig struct {
TotalLLM int `json:"total_llm"`
TotalLLM int `json:"total_llm"`
StageWorkers map[string]int `json:"stage_workers"`
extractWorkersConfigured bool
defaultedExtractWorkers int
}
type DiagnosticsConfig struct {
@@ -58,7 +62,9 @@ func Default() Config {
return Config{
Pipelines: map[string]pipeline.PipelineProfile{},
Concurrency: ConcurrencyConfig{
TotalLLM: 1,
TotalLLM: 1,
StageWorkers: map[string]int{"extract": 1},
defaultedExtractWorkers: 1,
},
Diagnostics: DiagnosticsConfig{
WorkDir: "/tmp/notarius",
@@ -101,6 +107,7 @@ func (c Config) workspaceDirectory() string {
func cloneConfig(in Config) Config {
out := in
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
for key, profile := range in.Pipelines {
out.Pipelines[key] = clonePipelineProfile(profile)
@@ -108,6 +115,34 @@ func cloneConfig(in Config) Config {
return out
}
func cloneIntMap(in map[string]int) map[string]int {
if len(in) == 0 {
return nil
}
out := make(map[string]int, len(in))
for key, value := range in {
out[key] = value
}
return out
}
func (c *ConcurrencyConfig) recomputeStageWorkerDefaults() {
if c == nil {
return
}
if c.StageWorkers == nil {
c.StageWorkers = make(map[string]int)
}
if !c.extractWorkersConfigured {
if value, ok := c.StageWorkers["extract"]; ok && (c.defaultedExtractWorkers == 0 || value != c.defaultedExtractWorkers) {
c.extractWorkersConfigured = true
return
}
c.StageWorkers["extract"] = c.TotalLLM
c.defaultedExtractWorkers = c.TotalLLM
}
}
func clonePipelineProfile(in pipeline.PipelineProfile) pipeline.PipelineProfile {
out := in
out.Input = cloneModuleBinding(in.Input)

View File

@@ -18,6 +18,9 @@ func TestDefaultValues(t *testing.T) {
if cfg.Concurrency.TotalLLM != 1 {
t.Fatalf("unexpected total LLM concurrency: %d", cfg.Concurrency.TotalLLM)
}
if got := cfg.Concurrency.StageWorkers["extract"]; got != 1 {
t.Fatalf("unexpected extract workers: %d", got)
}
if cfg.Diagnostics.WorkDir != "/tmp/notarius" {
t.Fatalf("unexpected diagnostics work dir: %q", cfg.Diagnostics.WorkDir)
}
@@ -68,6 +71,9 @@ pipelines:
if cfg.Concurrency.TotalLLM != 1 {
t.Fatalf("expected default concurrency preserved, got %d", cfg.Concurrency.TotalLLM)
}
if got := cfg.Concurrency.StageWorkers["extract"]; got != 1 {
t.Fatalf("expected default extract workers preserved, got %d", got)
}
if cfg.Diagnostics.Retention != diagnostics.RetentionAuto {
t.Fatalf("expected default diagnostics retention preserved, got %q", cfg.Diagnostics.Retention)
}

View File

@@ -26,6 +26,7 @@ type EffectiveConfig struct {
}
func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
c.Concurrency.recomputeStageWorkerDefaults()
if err := c.Validate(); err != nil {
return EffectiveConfig{}, err
}

View File

@@ -27,6 +27,19 @@ func TestResolveRejectsEmptyAndUnknownPipelineID(t *testing.T) {
}
}
func TestResolveMaterializesDefaultExtractWorkersFromEffectiveTotal(t *testing.T) {
cfg := validConfig()
cfg.Concurrency.TotalLLM = 4
effective, err := cfg.Resolve(ResolveInput{PipelineID: "example", Catalog: fakeCatalog(t)})
if err != nil {
t.Fatalf("Resolve() error = %v, want nil", err)
}
if got := effective.Config.Concurrency.StageWorkers["extract"]; got != 4 {
t.Fatalf("effective extract workers = %d, want total concurrency 4", got)
}
}
func TestResolveLaneFilteringSuccessAndFailure(t *testing.T) {
effective, err := validConfig().Resolve(ResolveInput{
PipelineID: " example ",

View File

@@ -36,6 +36,18 @@ func (c *Config) applyEnvOverridesWithLookup(lookup func(string) (string, bool))
}
c.Concurrency.TotalLLM = value
}
if raw, ok := lookup("NOTARIUS_STAGE_WORKERS_EXTRACT"); ok {
value, err := parseIntEnv("NOTARIUS_STAGE_WORKERS_EXTRACT", raw)
if err != nil {
return err
}
if c.Concurrency.StageWorkers == nil {
c.Concurrency.StageWorkers = make(map[string]int)
}
c.Concurrency.StageWorkers["extract"] = value
c.Concurrency.extractWorkersConfigured = true
}
c.Concurrency.recomputeStageWorkerDefaults()
if raw, ok := lookup("NOTARIUS_WORK_DIR"); ok {
c.Diagnostics.WorkDir = strings.TrimSpace(raw)
}

View File

@@ -14,6 +14,7 @@ func TestApplyEnvOverridesOperationalValues(t *testing.T) {
err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "3",
"NOTARIUS_STAGE_WORKERS_EXTRACT": "2",
"NOTARIUS_WORK_DIR": "/tmp/notarius-env",
"NOTARIUS_DIAGNOSTICS_RETENTION": "never",
"NOTARIUS_WORKSPACE_DIR": "/var/lib/notarius-env",
@@ -33,6 +34,9 @@ func TestApplyEnvOverridesOperationalValues(t *testing.T) {
if cfg.Concurrency.TotalLLM != 3 {
t.Fatalf("unexpected total concurrency: %d", cfg.Concurrency.TotalLLM)
}
if got := cfg.Concurrency.StageWorkers["extract"]; got != 2 {
t.Fatalf("extract workers = %d, want 2", got)
}
if cfg.Workspace.Directory != "/var/lib/notarius-env" {
t.Fatalf("unexpected workspace directory: %q", cfg.Workspace.Directory)
}
@@ -54,12 +58,74 @@ func TestApplyEnvOverridesOperationalValues(t *testing.T) {
}
func TestApplyEnvOverridesRejectsInvalidIntegers(t *testing.T) {
for _, name := range []string{"NOTARIUS_TOTAL_LLM_CONCURRENCY", "NOTARIUS_STAGE_WORKERS_EXTRACT"} {
t.Run(name, func(t *testing.T) {
cfg := Default()
err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{name: "many"}))
if err == nil || !strings.Contains(err.Error(), name) {
t.Fatalf("expected named integer error, got %v", err)
}
})
}
}
func TestStageWorkerEnvironmentPrecedenceAndDefaulting(t *testing.T) {
fileCfg, err := ParseFileConfigYAML([]byte(`
version: 2
concurrency:
total_llm: 4
stage_workers:
extract: 2
`))
if err != nil {
t.Fatalf("ParseFileConfigYAML() error = %v", err)
}
cfg := Default()
err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "many",
}))
if err == nil || !strings.Contains(err.Error(), "NOTARIUS_TOTAL_LLM_CONCURRENCY") {
t.Fatalf("expected named integer error, got %v", err)
if err := cfg.applyFileConfigWithLookup(fileCfg, emptyLookup); err != nil {
t.Fatalf("ApplyFileConfig() error = %v", err)
}
if err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "5",
"NOTARIUS_STAGE_WORKERS_EXTRACT": "3",
})); err != nil {
t.Fatalf("ApplyEnvOverrides() error = %v", err)
}
if cfg.Concurrency.TotalLLM != 5 || cfg.Concurrency.StageWorkers["extract"] != 3 {
t.Fatalf("effective concurrency = %#v, want total 5 and extract 3", cfg.Concurrency)
}
if err := cfg.Validate(); err != nil {
t.Fatalf("Validate(overridden) error = %v, want nil", err)
}
defaulted := Default()
if err := defaulted.applyEnvOverridesWithLookup(mapLookup(map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6"})); err != nil {
t.Fatalf("ApplyEnvOverrides(defaulted) error = %v", err)
}
if got := defaulted.Concurrency.StageWorkers["extract"]; got != 6 {
t.Fatalf("defaulted extract workers = %d, want effective total 6", got)
}
}
func TestStageWorkerRangeValidationUsesFinalEnvironmentTotal(t *testing.T) {
fileCfg, err := ParseFileConfigYAML([]byte(`
version: 2
concurrency:
total_llm: 4
stage_workers:
extract: 5
`))
if err != nil {
t.Fatalf("ParseFileConfigYAML() error = %v", err)
}
cfg := Default()
if err := cfg.applyFileConfigWithLookup(fileCfg, emptyLookup); err != nil {
t.Fatalf("ApplyFileConfig() error = %v", err)
}
if err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6"})); err != nil {
t.Fatalf("ApplyEnvOverrides() error = %v", err)
}
if err := cfg.Validate(); err != nil {
t.Fatalf("Validate() error = %v, want final total to make extract workers valid", err)
}
}
@@ -111,4 +177,7 @@ func TestLoadFromEnvUsesDefaultConfig(t *testing.T) {
if cfg.Concurrency.TotalLLM != 2 {
t.Fatalf("expected env concurrency override, got %+v", cfg.Concurrency)
}
if got := cfg.Concurrency.StageWorkers["extract"]; got != 2 {
t.Fatalf("expected extract workers to default to total, got %d", got)
}
}

View File

@@ -43,7 +43,8 @@ type FileArtifactLaneProfile struct {
}
type FileConcurrencyConfig struct {
TotalLLM *int `yaml:"total_llm,omitempty"`
TotalLLM *int `yaml:"total_llm,omitempty"`
StageWorkers map[string]int `yaml:"stage_workers,omitempty"`
}
type FileDiagnosticsConfig struct {
@@ -316,6 +317,15 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
if fileCfg.Concurrency != nil && fileCfg.Concurrency.TotalLLM != nil {
c.Concurrency.TotalLLM = *fileCfg.Concurrency.TotalLLM
}
if fileCfg.Concurrency != nil && fileCfg.Concurrency.StageWorkers != nil {
workers, configured, err := normalizeStageWorkers(fileCfg.Concurrency.StageWorkers)
if err != nil {
return err
}
c.Concurrency.StageWorkers = workers
c.Concurrency.extractWorkersConfigured = configured
}
c.Concurrency.recomputeStageWorkerDefaults()
if fileCfg.Diagnostics != nil {
if fileCfg.Diagnostics.WorkDir != nil {
c.Diagnostics.WorkDir = strings.TrimSpace(*fileCfg.Diagnostics.WorkDir)
@@ -350,6 +360,26 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
return nil
}
func normalizeStageWorkers(values map[string]int) (map[string]int, bool, error) {
workers := make(map[string]int, len(values))
configured := false
for rawKey, value := range values {
key := strings.TrimSpace(rawKey)
if key == "" {
return nil, false, fmt.Errorf("concurrency.stage_workers key must not be empty")
}
if key != "extract" {
return nil, false, fmt.Errorf("concurrency.stage_workers key %q is not supported", rawKey)
}
if _, exists := workers[key]; exists {
return nil, false, fmt.Errorf("concurrency.stage_workers key %q is duplicated after trimming", key)
}
workers[key] = value
configured = true
}
return workers, configured, nil
}
func normalizedMapKeys[T any](values map[string]T, keyName string) ([]string, map[string]string, error) {
keys := make([]string, 0, len(values))
rawByNormalized := make(map[string]string, len(values))

View File

@@ -250,7 +250,7 @@ pipelines:
}
lane := profile.Artifacts["events"]
if !reflect.DeepEqual(lane.References, map[string]string{"lore": "./lore.md", "roster": "./legacy-roster.yml"}) {
t.Fatalf("legacy lane references = %#v, want trimmed map", lane.References)
t.Fatalf("lane references = %#v, want trimmed map", lane.References)
}
wantExtract := map[string]string{
"glossary": "./glossary.md",
@@ -546,6 +546,9 @@ diagnostics:
if cfg.Concurrency.TotalLLM != 4 {
t.Fatalf("unexpected total concurrency: %d", cfg.Concurrency.TotalLLM)
}
if got := cfg.Concurrency.StageWorkers["extract"]; got != 4 {
t.Fatalf("default extract workers = %d, want total concurrency", got)
}
if cfg.Diagnostics.WorkDir != "/tmp/notarius-test" {
t.Fatalf("unexpected work dir: %q", cfg.Diagnostics.WorkDir)
}
@@ -554,6 +557,58 @@ diagnostics:
}
}
func TestApplyFileConfigStageWorkers(t *testing.T) {
cfg := parseAndApplyConfig(t, `
version: 2
concurrency:
total_llm: 4
stage_workers:
extract: 3
`)
if got := cfg.Concurrency.StageWorkers["extract"]; got != 3 {
t.Fatalf("extract workers = %d, want 3", got)
}
if err := cfg.Validate(); err != nil {
t.Fatalf("Validate() error = %v, want nil", err)
}
}
func TestApplyFileConfigEmptyStageWorkersDefaultsExtractToTotal(t *testing.T) {
cfg := parseAndApplyConfig(t, `
version: 2
concurrency:
total_llm: 4
stage_workers: {}
`)
if got := cfg.Concurrency.StageWorkers["extract"]; got != 4 {
t.Fatalf("extract workers = %d, want total concurrency 4", got)
}
}
func TestApplyFileConfigRejectsUnsupportedStageWorkerKeys(t *testing.T) {
for _, test := range []struct {
name string
key string
want string
}{
{name: "empty", key: "' '", want: "must not be empty"},
{name: "unknown", key: "merge", want: "not supported"},
} {
t.Run(test.name, func(t *testing.T) {
fileCfg, err := ParseFileConfigYAML([]byte("version: 2\nconcurrency:\n stage_workers:\n " + test.key + ": 1\n"))
if err != nil {
t.Fatalf("ParseFileConfigYAML() error = %v", err)
}
cfg := Default()
err = cfg.applyFileConfigWithLookup(fileCfg, emptyLookup)
if err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("ApplyFileConfig() error = %v, want %q", err, test.want)
}
})
}
}
func TestApplyFileConfigWorkspaceSection(t *testing.T) {
cfg := parseAndApplyConfig(t, `
version: 2

View File

@@ -50,6 +50,8 @@ func cloneResolvedValidatorChain(in pipeline.ResolvedValidatorChain) pipeline.Re
out.Validators[i] = pipeline.ResolvedValidator{
Binding: cloneModuleBinding(validator.Binding),
ExecutionClass: validator.ExecutionClass,
Target: validator.Target,
ArtifactKind: validator.ArtifactKind,
}
}
}

View File

@@ -12,6 +12,7 @@ func TestRedactedConfigCopiesScriptoriumConfig(t *testing.T) {
cfg.Scriptorium.ProfileDir = "./profiles"
cfg.Workspace.Directory = "/var/lib/notarius"
cfg.Workspace.Resume.Enabled = true
cfg.Concurrency.StageWorkers["extract"] = 1
redacted := cfg.Redacted()
@@ -29,6 +30,10 @@ func TestRedactedConfigCopiesScriptoriumConfig(t *testing.T) {
if cfg.Workspace.Directory != "/var/lib/notarius" {
t.Fatalf("redaction mutated original workspace config")
}
redacted.Concurrency.StageWorkers["extract"] = 9
if cfg.Concurrency.StageWorkers["extract"] != 1 {
t.Fatalf("redaction aliased stage worker map")
}
}
func TestConfigRedactedDiagnosticsPayloadCopiesConfig(t *testing.T) {
@@ -46,6 +51,8 @@ func TestConfigRedactedDiagnosticsPayloadCopiesConfig(t *testing.T) {
func TestEffectiveConfigRedactedDiagnosticsPayloadCopies(t *testing.T) {
cfg := validConfig()
cfg.Concurrency.TotalLLM = 4
cfg.Concurrency.StageWorkers["extract"] = 2
lane := cfg.Pipelines["example"].Artifacts["events"]
lane.Extract.Options = map[string]any{"temperature": 0.2}
lane.References = map[string]string{"roster": "./roster.yml"}
@@ -101,6 +108,10 @@ func TestEffectiveConfigRedactedDiagnosticsPayloadCopies(t *testing.T) {
if payload.PipelineID != effective.PipelineID || payload.ResolvedPipeline.Digest != effective.ResolvedPipeline.Digest {
t.Fatalf("expected pipeline metadata preserved, got %+v", payload)
}
payload.Config.Concurrency.StageWorkers["extract"] = 4
if effective.Config.Concurrency.StageWorkers["extract"] != 2 {
t.Fatalf("expected effective stage worker map to be copied")
}
payload.Only[0] = "changed"
if effective.Only[0] != "events" {

View File

@@ -2,6 +2,7 @@ package config
import (
"fmt"
"sort"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
@@ -9,6 +10,7 @@ import (
)
func (c Config) Validate() error {
c.Concurrency.recomputeStageWorkerDefaults()
if err := validateScriptorium(c.Scriptorium); err != nil {
return err
}
@@ -21,9 +23,36 @@ func (c Config) Validate() error {
if c.Concurrency.TotalLLM <= 0 {
return fmt.Errorf("total LLM concurrency must be greater than zero")
}
if err := validateStageWorkers(c.Concurrency); err != nil {
return err
}
return validatePipelineProfiles(c.Pipelines)
}
func validateStageWorkers(cfg ConcurrencyConfig) error {
keys := make([]string, 0, len(cfg.StageWorkers))
for key := range cfg.StageWorkers {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
if strings.TrimSpace(key) == "" {
return fmt.Errorf("concurrency.stage_workers key must not be empty")
}
if key != "extract" {
return fmt.Errorf("concurrency.stage_workers key %q is not supported", key)
}
}
extractWorkers, ok := cfg.StageWorkers["extract"]
if !ok {
extractWorkers = cfg.TotalLLM
}
if extractWorkers < 1 || extractWorkers > cfg.TotalLLM {
return fmt.Errorf("concurrency.stage_workers.extract must be between 1 and concurrency.total_llm (%d)", cfg.TotalLLM)
}
return nil
}
func validateScriptorium(cfg ScriptoriumConfig) error {
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
return fmt.Errorf("scriptorium profile_dir and profile_file are mutually exclusive")

View File

@@ -1,6 +1,7 @@
package config
import (
"fmt"
"strings"
"testing"
@@ -78,6 +79,41 @@ func TestValidateRejectsInvalidNumericFields(t *testing.T) {
}
}
func TestValidateStageWorkerBoundaries(t *testing.T) {
for _, test := range []struct {
name string
workers int
wantErr bool
}{
{name: "below minimum", workers: 0, wantErr: true},
{name: "minimum", workers: 1},
{name: "maximum", workers: 4},
{name: "above maximum", workers: 5, wantErr: true},
} {
t.Run(test.name, func(t *testing.T) {
cfg := validConfig()
cfg.Concurrency.TotalLLM = 4
cfg.Concurrency.StageWorkers["extract"] = test.workers
err := cfg.Validate()
if test.wantErr && (err == nil || !strings.Contains(err.Error(), "stage_workers.extract")) {
t.Fatalf("Validate() error = %v, want extract worker range error", err)
}
if !test.wantErr && err != nil {
t.Fatalf("Validate() error = %v, want nil", err)
}
})
}
}
func TestValidateRejectsUnknownEffectiveStageWorkerKey(t *testing.T) {
cfg := validConfig()
cfg.Concurrency.StageWorkers["merge"] = 1
err := cfg.Validate()
if err == nil || !strings.Contains(err.Error(), "stage_workers key") || !strings.Contains(err.Error(), "merge") {
t.Fatalf("Validate() error = %v, want unknown stage worker key", err)
}
}
func TestValidateRejectsMutuallyExclusiveScriptoriumProfileSources(t *testing.T) {
cfg := validConfig()
cfg.Scriptorium.ProfileDir = "./profiles"
@@ -497,6 +533,11 @@ func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.Module
for _, override := range overrides {
specs[override.Key] = override
}
for _, key := range []string{"fake/extract", "appendorder", "noop"} {
spec := specs[key]
spec.ArtifactKind = fakeArtifactKind
specs[key] = spec
}
inputs := pipeline.NewInputAdapterRegistry()
chunkers := pipeline.NewChunkerRegistry()
@@ -515,9 +556,15 @@ func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.Module
mustRegisterValidator(t, validators, specs["fake/llm-validator"])
mustRegisterOutput(t, outputs, specs["json"])
codecs := pipeline.NewArtifactCodecRegistry()
if err := pipeline.RegisterArtifactCodec(codecs, fakeArtifactCodec{}); err != nil {
t.Fatalf("register artifact codec: %v", err)
}
return pipeline.ModuleCatalog{
Inputs: inputs,
Chunkers: chunkers,
ArtifactCodecs: codecs,
Extractors: extractors,
Mergers: mergers,
Normalizers: normalizers,
@@ -543,21 +590,32 @@ func mustRegisterChunker(t *testing.T, registry *pipeline.ChunkerRegistry, spec
func mustRegisterExtractor(t *testing.T, registry *pipeline.ExtractorRegistry, spec pipeline.ModuleSpec) {
t.Helper()
if err := registry.RegisterWithSpec(spec, func() (contracts.Extractor, error) { return nil, nil }); err != nil {
validateOptions := func(options map[string]any) error {
if err := pipeline.RejectUnknownOptions(options, "temperature"); err != nil {
return err
}
if value, ok := options["temperature"]; ok {
if _, ok := value.(float64); !ok {
return fmt.Errorf("temperature must be a number")
}
}
return nil
}
if err := pipeline.RegisterExtractorBuilder[fakeArtifact](registry, spec, validateOptions, func(pipeline.BuildRequest) (contracts.Extractor[fakeArtifact], error) { return nil, nil }); err != nil {
t.Fatalf("register extractor: %v", err)
}
}
func mustRegisterMerger(t *testing.T, registry *pipeline.MergerRegistry, spec pipeline.ModuleSpec) {
t.Helper()
if err := registry.RegisterWithSpec(spec, func() (contracts.Merger, error) { return nil, nil }); err != nil {
if err := pipeline.RegisterMerger[fakeArtifact](registry, spec, func() (contracts.Merger[fakeArtifact], error) { return nil, nil }); err != nil {
t.Fatalf("register merger: %v", err)
}
}
func mustRegisterNormalizer(t *testing.T, registry *pipeline.NormalizerRegistry, spec pipeline.ModuleSpec) {
t.Helper()
if err := registry.RegisterWithSpec(spec, func() (contracts.Normalizer, error) { return nil, nil }); err != nil {
if err := pipeline.RegisterNormalizer[fakeArtifact](registry, spec, func() (contracts.Normalizer[fakeArtifact], error) { return nil, nil }); err != nil {
t.Fatalf("register normalizer: %v", err)
}
}
@@ -569,11 +627,32 @@ func mustRegisterValidator(t *testing.T, registry *pipeline.ValidatorRegistry, s
executionClass = contracts.ExecutionClassLLMBacked
}
validatorSpec := pipeline.ValidatorSpec{Key: spec.Key, ExecutionClass: executionClass}
if err := registry.RegisterWithSpec(validatorSpec, func() (contracts.Validator, error) { return nil, nil }); err != nil {
if err := pipeline.RegisterTypedValidator[fakeArtifact](registry, fakeArtifactKind, validatorSpec, func() (contracts.TypedValidator[fakeArtifact], error) { return nil, nil }); err != nil {
t.Fatalf("register validator: %v", err)
}
}
const fakeArtifactKind contracts.ArtifactKind = "test/artifact"
type fakeArtifact string
type fakeArtifactCodec struct{}
func (fakeArtifactCodec) Kind() contracts.ArtifactKind { return fakeArtifactKind }
func (fakeArtifactCodec) Schema() contracts.ArtifactSchema {
return contracts.ArtifactSchema{ID: "urn:notarius:test:artifact", Name: "Test artifact", Version: "1", JSONSchema: []byte(`{"type":"string"}`)}
}
func (fakeArtifactCodec) MediaType() string { return "application/json" }
func (fakeArtifactCodec) Encode(value fakeArtifact) ([]byte, error) {
return []byte(fmt.Sprintf("%q", value)), nil
}
func (fakeArtifactCodec) Decode(content []byte) (fakeArtifact, error) {
if len(content) < 2 {
return "", fmt.Errorf("invalid test artifact")
}
return fakeArtifact(content[1 : len(content)-1]), nil
}
func mustRegisterOutput(t *testing.T, registry *pipeline.OutputEncoderRegistry, spec pipeline.ModuleSpec) {
t.Helper()
if err := registry.RegisterWithSpec(spec, func() (contracts.OutputEncoder, error) { return nil, nil }); err != nil {

View File

@@ -0,0 +1,65 @@
package source
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
)
// DigestDocument returns a deterministic digest of the canonical source
// document content. The existing Digest field is excluded from its own digest.
func DigestDocument(doc *SourceDocument) (string, error) {
if doc == nil {
return "", fmt.Errorf("source document must not be nil")
}
payload := struct {
ID string `json:"id"`
Kind string `json:"kind"`
Format string `json:"format"`
Units []SourceUnit `json:"units"`
Metadata map[string]any `json:"metadata,omitempty"`
}{
ID: doc.ID,
Kind: doc.Kind,
Format: doc.Format,
Units: doc.Units,
Metadata: doc.Metadata,
}
encoded, err := json.Marshal(payload)
if err != nil {
return "", fmt.Errorf("encode source document for digest: %w", err)
}
sum := sha256.Sum256(encoded)
return "sha256:" + hex.EncodeToString(sum[:]), nil
}
// DigestChunk returns a deterministic digest of a chunk, including its source
// provenance, content, units, and metadata.
func DigestChunk(chunk Chunk) (string, error) {
payload := struct {
ID string `json:"id"`
SourceID string `json:"source_id"`
Index int `json:"index"`
Ref SourceRef `json:"ref"`
Content []byte `json:"content"`
MediaType string `json:"media_type"`
Units []SourceUnit `json:"units"`
Metadata map[string]any `json:"metadata,omitempty"`
}{
ID: chunk.ID,
SourceID: chunk.SourceID,
Index: chunk.Index,
Ref: chunk.Ref,
Content: chunk.Content,
MediaType: chunk.MediaType,
Units: chunk.Units,
Metadata: chunk.Metadata,
}
encoded, err := json.Marshal(payload)
if err != nil {
return "", fmt.Errorf("encode source chunk for digest: %w", err)
}
sum := sha256.Sum256(encoded)
return "sha256:" + hex.EncodeToString(sum[:]), nil
}

View File

@@ -13,6 +13,7 @@ type SourceUnit struct {
ID int `json:"id"`
Kind string `json:"kind"`
Text string `json:"text"`
Ref SourceRef `json:"ref"`
Metadata map[string]any `json:"metadata,omitempty"`
}
@@ -21,3 +22,14 @@ type SourceRef struct {
StartUnitID int `json:"start_unit_id"`
EndUnitID int `json:"end_unit_id"`
}
type Chunk struct {
ID string `json:"id"`
SourceID string `json:"source_id"`
Index int `json:"index"`
Ref SourceRef `json:"ref"`
Content []byte `json:"-"`
MediaType string `json:"media_type"`
Units []SourceUnit `json:"units"`
Metadata map[string]any `json:"metadata,omitempty"`
}

View File

@@ -142,6 +142,122 @@ func TestValidateDocumentDuplicateUnitIDs(t *testing.T) {
}
}
func TestValidateDocumentUnitReferences(t *testing.T) {
tests := []struct {
name string
mutate func(*SourceDocument)
wantErr string
}{
{
name: "missing",
mutate: func(doc *SourceDocument) { doc.Units[0].Ref = SourceRef{} },
wantErr: "source unit[0].ref: source ref source_id must not be empty",
},
{
name: "foreign source",
mutate: func(doc *SourceDocument) { doc.Units[0].Ref.SourceID = "source-2" },
wantErr: "source unit[0].ref: source ref source_id \"source-2\" does not match document id \"source-1\"",
},
{
name: "non-self range",
mutate: func(doc *SourceDocument) {
doc.Units[0].Ref.StartUnitID = 2
doc.Units[0].Ref.EndUnitID = 2
},
wantErr: "source unit[0].ref must identify source unit id 1",
},
{
name: "reversed range",
mutate: func(doc *SourceDocument) {
doc.Units[0].Ref.StartUnitID = 2
doc.Units[0].Ref.EndUnitID = 1
},
wantErr: "source unit[0].ref: source ref start_unit_id 2 appears after end_unit_id 1",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
doc := validDocument()
tt.mutate(doc)
err := ValidateDocument(doc)
if err == nil {
t.Fatal("ValidateDocument() error = nil, want unit reference error")
}
if err.Error() != tt.wantErr {
t.Fatalf("ValidateDocument() error = %q, want %q", err.Error(), tt.wantErr)
}
})
}
}
func TestDigestDocumentIsDeterministicAndIncludesUnitReference(t *testing.T) {
doc := validDocument()
doc.Metadata = map[string]any{"second": "value", "first": true}
first, err := DigestDocument(doc)
if err != nil {
t.Fatalf("DigestDocument() error = %v, want nil", err)
}
reordered := validDocument()
reordered.Metadata = map[string]any{"first": true, "second": "value"}
second, err := DigestDocument(reordered)
if err != nil {
t.Fatalf("DigestDocument(reordered) error = %v, want nil", err)
}
if first != second {
t.Fatalf("digests = %q and %q, want deterministic map ordering", first, second)
}
changed := validDocument()
changed.Metadata = map[string]any{"first": true, "second": "value"}
changed.Units[0].Ref.SourceID = "different-source"
changedDigest, err := DigestDocument(changed)
if err != nil {
t.Fatalf("DigestDocument(changed) error = %v, want nil", err)
}
if first == changedDigest {
t.Fatalf("digest = %q after reference change, want different digest", changedDigest)
}
}
func TestDigestChunkIsDeterministicAndIncludesReference(t *testing.T) {
doc := validDocument()
chunk := Chunk{
ID: "chunk-1",
SourceID: doc.ID,
Index: 0,
Ref: SourceRef{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 2},
Content: []byte("chunk content"),
MediaType: "text/plain",
Units: doc.Units,
Metadata: map[string]any{"second": "value", "first": true},
}
first, err := DigestChunk(chunk)
if err != nil {
t.Fatalf("DigestChunk() error = %v, want nil", err)
}
chunk.Metadata = map[string]any{"first": true, "second": "value"}
second, err := DigestChunk(chunk)
if err != nil {
t.Fatalf("DigestChunk(reordered metadata) error = %v, want nil", err)
}
if first != second {
t.Fatalf("digests = %q and %q, want deterministic map ordering", first, second)
}
chunk.Ref.EndUnitID = 1
changed, err := DigestChunk(chunk)
if err != nil {
t.Fatalf("DigestChunk(changed ref) error = %v, want nil", err)
}
if first == changed {
t.Fatalf("digest = %q after reference change, want different digest", changed)
}
}
func TestValidateRefValid(t *testing.T) {
doc := validDocument()
ref := SourceRef{
@@ -274,11 +390,13 @@ func validDocument() *SourceDocument {
ID: 1,
Kind: "paragraph",
Text: "First unit.",
Ref: SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1},
},
{
ID: 2,
Kind: "paragraph",
Text: "Second unit.",
Ref: SourceRef{SourceID: "source-1", StartUnitID: 2, EndUnitID: 2},
},
},
}

View File

@@ -44,6 +44,14 @@ func ValidateDocument(doc *SourceDocument) error {
}
seenUnitIDs[unit.ID] = struct{}{}
}
for i, unit := range doc.Units {
if err := ValidateRef(doc, unit.Ref); err != nil {
return fmt.Errorf("source unit[%d].ref: %w", i, err)
}
if unit.Ref.StartUnitID != unit.ID || unit.Ref.EndUnitID != unit.ID {
return fmt.Errorf("source unit[%d].ref must identify source unit id %d", i, unit.ID)
}
}
return nil
}

View File

@@ -2,7 +2,10 @@ package workspace
import "time"
const WorkspaceSchemaVersion = "notarius.workspace.v1"
const (
WorkspaceSchemaVersion = "notarius.workspace.v2"
WorkspaceSchemaVersionV1 = "notarius.workspace.v1"
)
type StageName string

View File

@@ -7,6 +7,12 @@ import (
)
func TestStageManifestDefaults(t *testing.T) {
if WorkspaceSchemaVersion != "notarius.workspace.v2" {
t.Fatalf("current schema version = %q, want notarius.workspace.v2", WorkspaceSchemaVersion)
}
if WorkspaceSchemaVersionV1 != "notarius.workspace.v1" {
t.Fatalf("legacy schema version = %q, want notarius.workspace.v1", WorkspaceSchemaVersionV1)
}
manifest := NewStageManifest(StageExtract, StatusRunning)
if manifest.WorkspaceSchemaVersion != WorkspaceSchemaVersion {

View File

@@ -78,80 +78,98 @@ func (l *WorkspaceLoader) Chunk(moduleKey string, sourceDigest string) (pipeline
if len(chunks) == 0 {
return pipeline.ChunkCheckpoint{}, invalidDecision("chunk checkpoint payload has no chunks")
}
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), chunkOutputDigests(chunks)) {
outputDigests, err := chunkOutputDigests(chunks)
if err != nil {
return pipeline.ChunkCheckpoint{}, invalidDecision("chunk checkpoint output cannot be digested: %v", err)
}
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), outputDigests) {
return pipeline.ChunkCheckpoint{}, invalidDecision("chunk checkpoint output digests do not match payload")
}
return pipeline.ChunkCheckpoint{Chunks: chunks, Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
}
func (l *WorkspaceLoader) Extract(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.ExtractCheckpoint, pipeline.CheckpointDecision) {
func (l *WorkspaceLoader) Extract(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.ExtractCheckpoint, pipeline.CheckpointDecision) {
var manifest coreworkspace.ExtractLaneManifest
if decision := l.readJSON(laneManifestPath("extract", laneID), &manifest); !decision.Reused {
return pipeline.ExtractCheckpoint{}, decision
if d := l.readJSON(laneManifestPath("extract", laneID), &manifest); !d.Reused {
return pipeline.ExtractCheckpoint{}, d
}
if decision := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageExtract, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded, coreworkspace.StatusSucceededWithRejections); !decision.Reused {
return pipeline.ExtractCheckpoint{}, decision
if d := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageExtract, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded, coreworkspace.StatusSucceededWithRejections); !d.Reused {
return pipeline.ExtractCheckpoint{}, d
}
var payload extractOutputsEnvelope
if decision := l.readJSON(lanePayloadPath("extract", laneID, "outputs.json"), &payload); !decision.Reused {
return pipeline.ExtractCheckpoint{}, decision
var payload artifactExtractEnvelope
if d := l.readJSON(lanePayloadPath("extract", laneID, "outputs.json"), &payload); !d.Reused {
return pipeline.ExtractCheckpoint{}, d
}
outputs, err := extractOutputsFromEnvelope(payload.Outputs)
outputs, err := artifactCheckpointOutputs(payload.Outputs)
if err != nil {
return pipeline.ExtractCheckpoint{}, invalidDecision("extract checkpoint payload is invalid: %v", err)
return pipeline.ExtractCheckpoint{}, invalidDecision("extract artifact checkpoint payload is invalid: %v", err)
}
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), rawOutputDigests(extractPayloads(outputs))) {
return pipeline.ExtractCheckpoint{}, invalidDecision("extract checkpoint output digests do not match payload")
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(outputs)) {
return pipeline.ExtractCheckpoint{}, invalidDecision("extract artifact checkpoint output digests do not match payload")
}
return pipeline.ExtractCheckpoint{
Outputs: outputs,
Rejected: cloneRejectedOutputs(payload.Rejected),
Warnings: cloneWarnings(payload.Warnings),
}, reusedDecision()
return pipeline.ExtractCheckpoint{Outputs: outputs, Rejected: cloneRejectedOutputs(payload.Rejected), Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
}
func (l *WorkspaceLoader) Merge(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.MergeCheckpoint, pipeline.CheckpointDecision) {
func (l *WorkspaceLoader) Merge(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.MergeCheckpoint, pipeline.CheckpointDecision) {
var manifest coreworkspace.MergeLaneManifest
if decision := l.readJSON(laneManifestPath("merge", laneID), &manifest); !decision.Reused {
return pipeline.MergeCheckpoint{}, decision
if d := l.readJSON(laneManifestPath("merge", laneID), &manifest); !d.Reused {
return pipeline.MergeCheckpoint{}, d
}
if decision := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageMerge, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded); !decision.Reused {
return pipeline.MergeCheckpoint{}, decision
if d := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageMerge, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded); !d.Reused {
return pipeline.MergeCheckpoint{}, d
}
var payload mergeOutputEnvelope
if decision := l.readJSON(lanePayloadPath("merge", laneID, "output.json"), &payload); !decision.Reused {
return pipeline.MergeCheckpoint{}, decision
var payload artifactSingleEnvelope
if d := l.readJSON(lanePayloadPath("merge", laneID, "output.json"), &payload); !d.Reused {
return pipeline.MergeCheckpoint{}, d
}
output, err := mergeOutputFromEnvelope(payload.Output)
values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output})
if err != nil {
return pipeline.MergeCheckpoint{}, invalidDecision("merge checkpoint payload is invalid: %v", err)
return pipeline.MergeCheckpoint{}, invalidDecision("merge artifact checkpoint payload is invalid: %v", err)
}
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), rawOutputDigests([]contracts.RawPayload{output.Payload})) {
return pipeline.MergeCheckpoint{}, invalidDecision("merge checkpoint output digest does not match payload")
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
return pipeline.MergeCheckpoint{}, invalidDecision("merge artifact checkpoint output digest does not match payload")
}
return pipeline.MergeCheckpoint{Output: output, Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
return pipeline.MergeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
}
func (l *WorkspaceLoader) Normalize(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
func (l *WorkspaceLoader) Normalize(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
var manifest coreworkspace.NormalizeLaneManifest
if decision := l.readJSON(laneManifestPath("normalize", laneID), &manifest); !decision.Reused {
return pipeline.NormalizeCheckpoint{}, decision
if d := l.readJSON(laneManifestPath("normalize", laneID), &manifest); !d.Reused {
return pipeline.NormalizeCheckpoint{}, d
}
if decision := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageNormalize, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded); !decision.Reused {
return pipeline.NormalizeCheckpoint{}, decision
if d := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageNormalize, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded); !d.Reused {
return pipeline.NormalizeCheckpoint{}, d
}
var payload normalizeOutputEnvelope
if decision := l.readJSON(lanePayloadPath("normalize", laneID, "output.json"), &payload); !decision.Reused {
return pipeline.NormalizeCheckpoint{}, decision
var payload artifactSingleEnvelope
if d := l.readJSON(lanePayloadPath("normalize", laneID, "output.json"), &payload); !d.Reused {
return pipeline.NormalizeCheckpoint{}, d
}
output, err := normalizeOutputFromEnvelope(payload.Output)
values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output})
if err != nil {
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize checkpoint payload is invalid: %v", err)
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize artifact checkpoint payload is invalid: %v", err)
}
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), rawOutputDigests([]contracts.RawPayload{output.Payload})) {
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize checkpoint output digest does not match payload")
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize artifact checkpoint output digest does not match payload")
}
return pipeline.NormalizeCheckpoint{Output: output, Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
return pipeline.NormalizeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
}
func artifactCheckpointOutputs(values []artifactCheckpointEnvelope) ([]pipeline.CheckpointArtifact, error) {
if len(values) == 0 {
return nil, nil
}
out := make([]pipeline.CheckpointArtifact, 0, len(values))
for _, v := range values {
content, err := contentFromEnvelope(v.Content)
if err != nil {
return nil, err
}
if strings.TrimSpace(string(v.Kind)) == "" || strings.TrimSpace(v.Schema.ID) == "" || strings.TrimSpace(v.Schema.Version) == "" || strings.TrimSpace(v.SchemaDigest) == "" {
return nil, fmt.Errorf("artifact codec identity is incomplete")
}
out = append(out, pipeline.CheckpointArtifact{LaneID: v.LaneID, ModuleKey: v.ModuleKey, SourceID: v.SourceID, ChunkID: v.ChunkID, ChunkIndex: v.ChunkIndex, ChunkRef: v.ChunkRef, SchemaDigest: v.SchemaDigest, Artifact: contracts.SerializedArtifact{Kind: v.Kind, Schema: v.Schema, MediaType: v.Content.MediaType, Content: content, Metadata: cloneMetadata(v.Content.Metadata)}})
}
return out, nil
}
func (l *WorkspaceLoader) readJSON(name string, out any) pipeline.CheckpointDecision {
@@ -180,6 +198,9 @@ func (l *WorkspaceLoader) validateManifest(manifest coreworkspace.StageManifest,
}
func (l *WorkspaceLoader) validateLaneManifest(manifest coreworkspace.StageManifest, stage coreworkspace.StageName, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, statuses ...coreworkspace.StageStatus) pipeline.CheckpointDecision {
if manifest.WorkspaceSchemaVersion == coreworkspace.WorkspaceSchemaVersionV1 {
return invalidDecision("checkpoint workspace schema version %q is incompatible with %q and must be recomputed", manifest.WorkspaceSchemaVersion, coreworkspace.WorkspaceSchemaVersion)
}
if manifest.WorkspaceSchemaVersion != coreworkspace.WorkspaceSchemaVersion {
return invalidDecision("checkpoint workspace schema version %q is not supported", manifest.WorkspaceSchemaVersion)
}
@@ -211,95 +232,30 @@ func (l *WorkspaceLoader) validateLaneManifest(manifest coreworkspace.StageManif
return reusedDecision()
}
func sourceChunksFromEnvelope(values []chunkEnvelope) ([]contracts.SourceChunk, error) {
func sourceChunksFromEnvelope(values []chunkEnvelope) ([]source.Chunk, error) {
if len(values) == 0 {
return nil, nil
}
out := make([]contracts.SourceChunk, 0, len(values))
out := make([]source.Chunk, 0, len(values))
for _, value := range values {
content, err := contentFromEnvelope(value.Content)
if err != nil {
return nil, err
}
out = append(out, contracts.SourceChunk{
ID: value.ID,
SourceID: value.SourceID,
Index: value.Index,
StartUnitID: value.StartUnitID,
EndUnitID: value.EndUnitID,
Content: content,
MediaType: value.Content.MediaType,
Units: cloneSourceUnits(value.Units),
Metadata: cloneMetadata(value.Metadata),
out = append(out, source.Chunk{
ID: value.ID,
SourceID: value.SourceID,
Index: value.Index,
Ref: value.Ref,
Content: content,
MediaType: value.Content.MediaType,
Units: cloneSourceUnits(value.Units),
Metadata: cloneMetadata(value.Metadata),
})
}
return out, nil
}
func extractOutputsFromEnvelope(values []extractOutputEnvelope) ([]contracts.ExtractOutput, error) {
if len(values) == 0 {
return nil, nil
}
out := make([]contracts.ExtractOutput, 0, len(values))
for _, value := range values {
payload, err := rawPayloadFromEnvelope(value.Payload)
if err != nil {
return nil, err
}
out = append(out, contracts.ExtractOutput{
LaneID: value.LaneID,
ExtractorKey: value.ExtractorKey,
SourceID: value.SourceID,
ChunkID: value.ChunkID,
ChunkIndex: value.ChunkIndex,
Schema: value.Schema,
Payload: payload,
})
}
return out, nil
}
func mergeOutputFromEnvelope(value mergeOutputPayload) (contracts.MergeOutput, error) {
payload, err := rawPayloadFromEnvelope(value.Payload)
if err != nil {
return contracts.MergeOutput{}, err
}
return contracts.MergeOutput{
LaneID: value.LaneID,
MergerKey: value.MergerKey,
SourceID: value.SourceID,
Schema: value.Schema,
Payload: payload,
}, nil
}
func normalizeOutputFromEnvelope(value normalizeOutputPayload) (contracts.NormalizeOutput, error) {
payload, err := rawPayloadFromEnvelope(value.Payload)
if err != nil {
return contracts.NormalizeOutput{}, err
}
return contracts.NormalizeOutput{
LaneID: value.LaneID,
NormalizerKey: value.NormalizerKey,
SourceID: value.SourceID,
Schema: value.Schema,
Payload: payload,
}, nil
}
func rawPayloadFromEnvelope(value binaryEnvelope) (contracts.RawPayload, error) {
content, err := contentFromEnvelope(value)
if err != nil {
return contracts.RawPayload{}, err
}
return contracts.RawPayload{
Content: content,
MediaType: value.MediaType,
Metadata: cloneMetadata(value.Metadata),
Warnings: cloneWarnings(value.Warnings),
}, nil
}
func contentFromEnvelope(value binaryEnvelope) ([]byte, error) {
content, err := base64.StdEncoding.DecodeString(value.ContentBase64)
if err != nil {

View File

@@ -73,7 +73,11 @@ func (r *WorkspaceRecorder) ChunkRunning(moduleKey string, sourceDigest string)
return r.writeManifest("chunk/manifest.json", coreworkspace.ChunkManifest{StageManifest: manifest})
}
func (r *WorkspaceRecorder) ChunkSucceeded(moduleKey string, sourceDigest string, chunks []contracts.SourceChunk, warnings []contracts.Warning) error {
func (r *WorkspaceRecorder) ChunkSucceeded(moduleKey string, sourceDigest string, chunks []source.Chunk, warnings []contracts.Warning) error {
outputDigests, err := chunkOutputDigests(chunks)
if err != nil {
return fmt.Errorf("digest chunk checkpoint output: %w", err)
}
payload := chunksEnvelope{Chunks: chunkEnvelopes(chunks), Warnings: cloneWarnings(warnings)}
if err := r.writePayload("chunk/chunks.json", payload); err != nil {
return err
@@ -81,7 +85,7 @@ func (r *WorkspaceRecorder) ChunkSucceeded(moduleKey string, sourceDigest string
manifest := r.newStageManifest(coreworkspace.StageChunk, coreworkspace.StatusSucceeded)
manifest.ModuleKey = moduleKey
manifest.DependencyFingerprints = workspaceFingerprints(digestFingerprints("source_document", sourceDigest))
manifest.OutputDigests = workspaceFingerprints(chunkOutputDigests(chunks))
manifest.OutputDigests = workspaceFingerprints(outputDigests)
manifest.ValidationStatus = validationStatusString(warnings, nil)
manifest.CompletedAt = timePtr(r.timestamp())
return r.writeManifest("chunk/manifest.json", coreworkspace.ChunkManifest{
@@ -115,25 +119,17 @@ func (r *WorkspaceRecorder) ExtractRunning(laneID string, moduleKey string, depe
return r.writeManifest(laneManifestPath("extract", laneID), coreworkspace.ExtractLaneManifest{StageManifest: manifest})
}
func (r *WorkspaceRecorder) ExtractSucceeded(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []contracts.ExtractOutput, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
payload := extractOutputsEnvelope{
Outputs: extractOutputEnvelopes(outputs),
Rejected: cloneRejectedOutputs(rejected),
Warnings: cloneWarnings(warnings),
}
func (r *WorkspaceRecorder) ExtractSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []pipeline.CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
payload := artifactExtractEnvelope{Outputs: artifactCheckpointEnvelopes(outputs), Rejected: cloneRejectedOutputs(rejected), Warnings: cloneWarnings(warnings)}
if err := r.writePayload(lanePayloadPath("extract", laneID, "outputs.json"), payload); err != nil {
return err
}
manifest := r.laneManifest(coreworkspace.StageExtract, statusForRejected(rejected), laneID, moduleKey, dependencies)
manifest.OutputDigests = workspaceFingerprints(rawOutputDigests(extractPayloads(outputs)))
manifest.OutputDigests = workspaceFingerprints(artifactOutputDigests(outputs))
manifest.ValidationStatus = validationStatusString(warnings, rejected)
manifest.Rejections = rejectionSummaries(rejected)
manifest.CompletedAt = timePtr(r.timestamp())
return r.writeManifest(laneManifestPath("extract", laneID), coreworkspace.ExtractLaneManifest{
StageManifest: manifest,
ChunkCount: len(outputs) + len(rejected),
OutputCount: len(outputs),
})
return r.writeManifest(laneManifestPath("extract", laneID), coreworkspace.ExtractLaneManifest{StageManifest: manifest, ChunkCount: len(outputs) + len(rejected), OutputCount: len(outputs)})
}
func (r *WorkspaceRecorder) ExtractFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
@@ -149,19 +145,15 @@ func (r *WorkspaceRecorder) MergeRunning(laneID string, moduleKey string, depend
return r.writeManifest(laneManifestPath("merge", laneID), coreworkspace.MergeLaneManifest{StageManifest: manifest})
}
func (r *WorkspaceRecorder) MergeSucceeded(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output contracts.MergeOutput, warnings []contracts.Warning) error {
payload := mergeOutputEnvelope{Output: mergeOutputEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}
if err := r.writePayload(lanePayloadPath("merge", laneID, "output.json"), payload); err != nil {
func (r *WorkspaceRecorder) MergeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
if err := r.writePayload(lanePayloadPath("merge", laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
return err
}
manifest := r.laneManifest(coreworkspace.StageMerge, coreworkspace.StatusSucceeded, laneID, moduleKey, dependencies)
manifest.OutputDigests = workspaceFingerprints(rawOutputDigests([]contracts.RawPayload{output.Payload}))
manifest.OutputDigests = workspaceFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
manifest.ValidationStatus = validationStatusString(warnings, nil)
manifest.CompletedAt = timePtr(r.timestamp())
return r.writeManifest(laneManifestPath("merge", laneID), coreworkspace.MergeLaneManifest{
StageManifest: manifest,
InputCount: len(dependencies),
})
return r.writeManifest(laneManifestPath("merge", laneID), coreworkspace.MergeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
}
func (r *WorkspaceRecorder) MergeRejected(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error {
@@ -185,13 +177,12 @@ func (r *WorkspaceRecorder) NormalizeRunning(laneID string, moduleKey string, de
return r.writeManifest(laneManifestPath("normalize", laneID), coreworkspace.NormalizeLaneManifest{StageManifest: manifest})
}
func (r *WorkspaceRecorder) NormalizeSucceeded(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output contracts.NormalizeOutput, warnings []contracts.Warning) error {
payload := normalizeOutputEnvelope{Output: normalizeOutputEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}
if err := r.writePayload(lanePayloadPath("normalize", laneID, "output.json"), payload); err != nil {
func (r *WorkspaceRecorder) NormalizeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
if err := r.writePayload(lanePayloadPath("normalize", laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
return err
}
manifest := r.laneManifest(coreworkspace.StageNormalize, coreworkspace.StatusSucceeded, laneID, moduleKey, dependencies)
manifest.OutputDigests = workspaceFingerprints(rawOutputDigests([]contracts.RawPayload{output.Payload}))
manifest.OutputDigests = workspaceFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
manifest.ValidationStatus = validationStatusString(warnings, nil)
manifest.CompletedAt = timePtr(r.timestamp())
return r.writeManifest(laneManifestPath("normalize", laneID), coreworkspace.NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
@@ -260,56 +251,13 @@ type chunksEnvelope struct {
}
type chunkEnvelope struct {
ID string `json:"id"`
SourceID string `json:"source_id"`
Index int `json:"index"`
StartUnitID int `json:"start_unit_id"`
EndUnitID int `json:"end_unit_id"`
Content binaryEnvelope `json:"content"`
Units []source.SourceUnit `json:"units,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type extractOutputsEnvelope struct {
Outputs []extractOutputEnvelope `json:"outputs"`
Rejected []contracts.RejectedOutput `json:"rejected,omitempty"`
Warnings []contracts.Warning `json:"warnings,omitempty"`
}
type extractOutputEnvelope struct {
LaneID string `json:"lane_id"`
ExtractorKey string `json:"extractor_key"`
SourceID string `json:"source_id"`
ChunkID string `json:"chunk_id"`
ChunkIndex int `json:"chunk_index"`
Schema contracts.ResponseSchema `json:"schema,omitempty"`
Payload binaryEnvelope `json:"payload"`
}
type mergeOutputEnvelope struct {
Output mergeOutputPayload `json:"output"`
Warnings []contracts.Warning `json:"warnings,omitempty"`
}
type mergeOutputPayload struct {
LaneID string `json:"lane_id"`
MergerKey string `json:"merger_key"`
SourceID string `json:"source_id,omitempty"`
Schema contracts.ResponseSchema `json:"schema,omitempty"`
Payload binaryEnvelope `json:"payload"`
}
type normalizeOutputEnvelope struct {
Output normalizeOutputPayload `json:"output"`
Warnings []contracts.Warning `json:"warnings,omitempty"`
}
type normalizeOutputPayload struct {
LaneID string `json:"lane_id"`
NormalizerKey string `json:"normalizer_key"`
SourceID string `json:"source_id,omitempty"`
Schema contracts.ResponseSchema `json:"schema,omitempty"`
Payload binaryEnvelope `json:"payload"`
ID string `json:"id"`
SourceID string `json:"source_id"`
Index int `json:"index"`
Ref source.SourceRef `json:"ref"`
Content binaryEnvelope `json:"content"`
Units []source.SourceUnit `json:"units,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type binaryEnvelope struct {
@@ -320,74 +268,70 @@ type binaryEnvelope struct {
Warnings []contracts.Warning `json:"warnings,omitempty"`
}
func chunkEnvelopes(chunks []contracts.SourceChunk) []chunkEnvelope {
type artifactCheckpointEnvelope struct {
LaneID string `json:"lane_id"`
ModuleKey string `json:"module_key"`
SourceID string `json:"source_id,omitempty"`
ChunkID string `json:"chunk_id,omitempty"`
ChunkIndex int `json:"chunk_index,omitempty"`
ChunkRef source.SourceRef `json:"chunk_ref,omitempty"`
Kind contracts.ArtifactKind `json:"artifact_kind"`
Schema contracts.ArtifactSchema `json:"schema"`
SchemaDigest string `json:"schema_digest"`
Content binaryEnvelope `json:"content"`
}
type artifactExtractEnvelope struct {
Outputs []artifactCheckpointEnvelope `json:"outputs"`
Rejected []contracts.RejectedOutput `json:"rejected,omitempty"`
Warnings []contracts.Warning `json:"warnings,omitempty"`
}
type artifactSingleEnvelope struct {
Output artifactCheckpointEnvelope `json:"output"`
Warnings []contracts.Warning `json:"warnings,omitempty"`
}
func artifactCheckpointEnvelopeFromOutput(output pipeline.CheckpointArtifact) artifactCheckpointEnvelope {
schema := contracts.CloneArtifactSchema(output.Artifact.Schema)
schema.JSONSchema = nil
return artifactCheckpointEnvelope{LaneID: output.LaneID, ModuleKey: output.ModuleKey, SourceID: output.SourceID, ChunkID: output.ChunkID, ChunkIndex: output.ChunkIndex, ChunkRef: output.ChunkRef, Kind: output.Artifact.Kind, Schema: schema, SchemaDigest: output.SchemaDigest, Content: binaryEnvelopeFromContent(output.Artifact.Content, output.Artifact.MediaType, output.Artifact.Metadata, nil)}
}
func artifactCheckpointEnvelopes(outputs []pipeline.CheckpointArtifact) []artifactCheckpointEnvelope {
if len(outputs) == 0 {
return nil
}
out := make([]artifactCheckpointEnvelope, 0, len(outputs))
for _, v := range outputs {
out = append(out, artifactCheckpointEnvelopeFromOutput(v))
}
return out
}
func artifactOutputDigests(outputs []pipeline.CheckpointArtifact) []pipeline.CheckpointFingerprint {
values := make([]pipeline.CheckpointFingerprint, 0, len(outputs))
for i, v := range outputs {
values = append(values, pipeline.CheckpointFingerprint{Name: fmt.Sprintf("artifact[%d]", i), Value: contentDigest(v.Artifact.Content)})
}
return normalizeFingerprints(values)
}
func chunkEnvelopes(chunks []source.Chunk) []chunkEnvelope {
if len(chunks) == 0 {
return nil
}
out := make([]chunkEnvelope, 0, len(chunks))
for _, chunk := range chunks {
out = append(out, chunkEnvelope{
ID: chunk.ID,
SourceID: chunk.SourceID,
Index: chunk.Index,
StartUnitID: chunk.StartUnitID,
EndUnitID: chunk.EndUnitID,
Content: binaryEnvelopeFromContent(chunk.Content, chunk.MediaType, chunk.Metadata, nil),
Units: cloneSourceUnits(chunk.Units),
Metadata: cloneMetadata(chunk.Metadata),
ID: chunk.ID,
SourceID: chunk.SourceID,
Index: chunk.Index,
Ref: chunk.Ref,
Content: binaryEnvelopeFromContent(chunk.Content, chunk.MediaType, chunk.Metadata, nil),
Units: cloneSourceUnits(chunk.Units),
Metadata: cloneMetadata(chunk.Metadata),
})
}
return out
}
func extractOutputEnvelopes(outputs []contracts.ExtractOutput) []extractOutputEnvelope {
if len(outputs) == 0 {
return nil
}
out := make([]extractOutputEnvelope, 0, len(outputs))
for _, output := range outputs {
out = append(out, extractOutputEnvelope{
LaneID: output.LaneID,
ExtractorKey: output.ExtractorKey,
SourceID: output.SourceID,
ChunkID: output.ChunkID,
ChunkIndex: output.ChunkIndex,
Schema: schemaEnvelope(output.Schema),
Payload: binaryEnvelopeFromPayload(output.Payload),
})
}
return out
}
func mergeOutputEnvelopeFromOutput(output contracts.MergeOutput) mergeOutputPayload {
return mergeOutputPayload{
LaneID: output.LaneID,
MergerKey: output.MergerKey,
SourceID: output.SourceID,
Schema: schemaEnvelope(output.Schema),
Payload: binaryEnvelopeFromPayload(output.Payload),
}
}
func normalizeOutputEnvelopeFromOutput(output contracts.NormalizeOutput) normalizeOutputPayload {
return normalizeOutputPayload{
LaneID: output.LaneID,
NormalizerKey: output.NormalizerKey,
SourceID: output.SourceID,
Schema: schemaEnvelope(output.Schema),
Payload: binaryEnvelopeFromPayload(output.Payload),
}
}
func schemaEnvelope(schema contracts.ResponseSchema) contracts.ResponseSchema {
schema.JSONSchema = nil
return schema
}
func binaryEnvelopeFromPayload(payload contracts.RawPayload) binaryEnvelope {
return binaryEnvelopeFromContent(payload.Content, payload.MediaType, payload.Metadata, payload.Warnings)
}
func binaryEnvelopeFromContent(content []byte, mediaType string, metadata map[string]any, warnings []contracts.Warning) binaryEnvelope {
return binaryEnvelope{
ContentBase64: base64.StdEncoding.EncodeToString(content),
@@ -414,6 +358,7 @@ func cloneSourceUnits(units []source.SourceUnit) []source.SourceUnit {
ID: unit.ID,
Kind: unit.Kind,
Text: unit.Text,
Ref: unit.Ref,
Metadata: cloneMetadata(unit.Metadata),
})
}
@@ -445,37 +390,19 @@ func cloneMetadata(metadata map[string]any) map[string]any {
return out
}
func rawOutputDigests(payloads []contracts.RawPayload) []pipeline.CheckpointFingerprint {
values := make([]pipeline.CheckpointFingerprint, 0, len(payloads))
for i, payload := range payloads {
values = append(values, pipeline.CheckpointFingerprint{
Name: fmt.Sprintf("payload[%d]", i),
Value: contentDigest(payload.Content),
})
}
return normalizeFingerprints(values)
}
func extractPayloads(outputs []contracts.ExtractOutput) []contracts.RawPayload {
if len(outputs) == 0 {
return nil
}
payloads := make([]contracts.RawPayload, 0, len(outputs))
for _, output := range outputs {
payloads = append(payloads, output.Payload)
}
return payloads
}
func chunkOutputDigests(chunks []contracts.SourceChunk) []pipeline.CheckpointFingerprint {
func chunkOutputDigests(chunks []source.Chunk) ([]pipeline.CheckpointFingerprint, error) {
values := make([]pipeline.CheckpointFingerprint, 0, len(chunks))
for _, chunk := range chunks {
digest, err := source.DigestChunk(chunk)
if err != nil {
return nil, fmt.Errorf("chunk %q: %w", chunk.ID, err)
}
values = append(values, pipeline.CheckpointFingerprint{
Name: chunk.ID,
Value: contentDigest(chunk.Content),
Value: digest,
})
}
return normalizeFingerprints(values)
return normalizeFingerprints(values), nil
}
func digestFingerprints(name string, digest string) []pipeline.CheckpointFingerprint {

View File

@@ -22,18 +22,17 @@ func TestWorkspaceRecorderWritesSuccessfulCheckpointFiles(t *testing.T) {
Kind: "document",
Format: "text/plain",
Digest: "sha256:source",
Units: []source.SourceUnit{{ID: 1, Kind: "line", Text: "hello"}},
Units: []source.SourceUnit{{ID: 1, Kind: "line", Text: "hello", Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1}}},
}
chunks := []contracts.SourceChunk{
chunks := []source.Chunk{
{
ID: "chunk-1",
SourceID: "source-1",
Index: 0,
StartUnitID: 1,
EndUnitID: 1,
Content: []byte("chunk content"),
MediaType: "text/plain",
Units: doc.Units,
ID: "chunk-1",
SourceID: "source-1",
Index: 0,
Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1},
Content: []byte("chunk content"),
MediaType: "text/plain",
Units: doc.Units,
},
}
@@ -80,96 +79,43 @@ func TestWorkspaceRecorderWritesSuccessfulCheckpointFiles(t *testing.T) {
}
}
func TestWorkspaceLoaderReusesSuccessfulCheckpointFiles(t *testing.T) {
func TestWorkspaceArtifactCheckpointsRoundTripCodecIdentityAndBytes(t *testing.T) {
root := t.TempDir()
recorder := newTestRecorder(t, root)
loader := &WorkspaceLoader{root: root}
doc := &source.SourceDocument{
ID: "source-1",
Kind: "document",
Format: "text/plain",
Digest: "sha256:source",
Units: []source.SourceUnit{{ID: 1, Kind: "line", Text: "hello"}},
}
chunks := []contracts.SourceChunk{
{
ID: "chunk-1",
SourceID: "source-1",
Index: 0,
StartUnitID: 1,
EndUnitID: 1,
Content: []byte("chunk content"),
MediaType: "text/plain",
Units: doc.Units,
},
}
extractOutput := contracts.ExtractOutput{
LaneID: "spells",
ExtractorKey: "dnd/spells",
SourceID: doc.ID,
ChunkID: "chunk-1",
Payload: contracts.RawPayload{
Content: []byte(`{"spell":"cure wounds"}`),
MediaType: "application/json",
},
}
mergeOutput := contracts.MergeOutput{
LaneID: "spells",
MergerKey: "appendorder",
SourceID: doc.ID,
Payload: contracts.RawPayload{
Content: []byte(`{"merged":true}`),
MediaType: "application/json",
},
}
normalizeOutput := contracts.NormalizeOutput{
LaneID: "spells",
NormalizerKey: "noop",
SourceID: doc.ID,
Payload: contracts.RawPayload{
Content: []byte(`{"normalized":true}`),
MediaType: "application/json",
},
}
schema := contracts.ArtifactSchema{ID: "dnd.spell_response", Name: "spell response", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
artifact := contracts.SerializedArtifact{Kind: "dnd.spells", Schema: schema, MediaType: "application/json", Content: []byte(`{"spell_casts":[]}`), Metadata: map[string]any{"spell_cast_count": float64(0)}}
stored := pipeline.CheckpointArtifact{LaneID: "spells", ModuleKey: "dnd/spells", SourceID: "source-1", ChunkID: "chunk-1", ChunkIndex: 2, ChunkRef: source.SourceRef{SourceID: "source-1", StartUnitID: 4, EndUnitID: 8}, Artifact: artifact, SchemaDigest: contracts.DigestArtifactSchema(schema)}
if err := recorder.SourceSucceeded("seriatim", doc); err != nil {
t.Fatalf("SourceSucceeded: %v", err)
}
if err := recorder.ChunkSucceeded("generic", doc.Digest, chunks, nil); err != nil {
t.Fatalf("ChunkSucceeded: %v", err)
}
extractDeps := []pipeline.CheckpointFingerprint{{Name: "chunks", Value: "sha256:chunks"}}
if err := recorder.ExtractSucceeded("spells", "dnd/spells", extractDeps, []contracts.ExtractOutput{extractOutput}, nil, nil); err != nil {
if err := recorder.ExtractSucceeded("spells", "dnd/spells", extractDeps, []pipeline.CheckpointArtifact{stored}, nil, nil); err != nil {
t.Fatalf("ExtractSucceeded: %v", err)
}
mergeDeps := rawOutputDigests([]contracts.RawPayload{extractOutput.Payload})
if err := recorder.MergeSucceeded("spells", "appendorder", mergeDeps, mergeOutput, nil); err != nil {
t.Fatalf("MergeSucceeded: %v", err)
extracted, decision := loader.Extract("spells", "dnd/spells", extractDeps)
if !decision.Reused || len(extracted.Outputs) != 1 {
t.Fatalf("extract decision=%#v checkpoint=%#v, want reused", decision, extracted)
}
normalizeDeps := rawOutputDigests([]contracts.RawPayload{mergeOutput.Payload})
if err := recorder.NormalizeSucceeded("spells", "noop", normalizeDeps, normalizeOutput, nil); err != nil {
t.Fatalf("NormalizeSucceeded: %v", err)
got := extracted.Outputs[0]
if got.Artifact.Kind != artifact.Kind || got.Artifact.Schema.ID != schema.ID || got.Artifact.Schema.Version != schema.Version || got.SchemaDigest != stored.SchemaDigest || string(got.Artifact.Content) != string(artifact.Content) || got.ChunkRef != stored.ChunkRef {
t.Fatalf("artifact checkpoint = %#v, want codec identity, bytes, and provenance", got)
}
sourceCheckpoint, decision := loader.Source("seriatim")
if !decision.Reused || sourceCheckpoint.Document.ID != "source-1" {
t.Fatalf("source decision = %#v checkpoint=%#v, want reused", decision, sourceCheckpoint)
mergeDeps := artifactOutputDigests([]pipeline.CheckpointArtifact{stored})
if err := recorder.MergeSucceeded("spells", "merge", mergeDeps, stored, nil); err != nil {
t.Fatalf("MergeSucceeded: %v", err)
}
chunkCheckpoint, decision := loader.Chunk("generic", doc.Digest)
if !decision.Reused || len(chunkCheckpoint.Chunks) != 1 || string(chunkCheckpoint.Chunks[0].Content) != "chunk content" {
t.Fatalf("chunk decision = %#v checkpoint=%#v, want reused", decision, chunkCheckpoint)
merged, decision := loader.Merge("spells", "merge", mergeDeps)
if !decision.Reused || string(merged.Output.Artifact.Content) != string(artifact.Content) {
t.Fatalf("merge decision=%#v checkpoint=%#v, want reused", decision, merged)
}
extractCheckpoint, decision := loader.Extract("spells", "dnd/spells", extractDeps)
if !decision.Reused || len(extractCheckpoint.Outputs) != 1 || string(extractCheckpoint.Outputs[0].Payload.Content) != `{"spell":"cure wounds"}` {
t.Fatalf("extract decision = %#v checkpoint=%#v, want reused", decision, extractCheckpoint)
normalizeDeps := artifactOutputDigests([]pipeline.CheckpointArtifact{stored})
if err := recorder.NormalizeSucceeded("spells", "normalize", normalizeDeps, stored, nil); err != nil {
t.Fatalf("NormalizeSucceeded: %v", err)
}
mergeCheckpoint, decision := loader.Merge("spells", "appendorder", mergeDeps)
if !decision.Reused || string(mergeCheckpoint.Output.Payload.Content) != `{"merged":true}` {
t.Fatalf("merge decision = %#v checkpoint=%#v, want reused", decision, mergeCheckpoint)
}
normalizeCheckpoint, decision := loader.Normalize("spells", "noop", normalizeDeps)
if !decision.Reused || string(normalizeCheckpoint.Output.Payload.Content) != `{"normalized":true}` {
t.Fatalf("normalize decision = %#v checkpoint=%#v, want reused", decision, normalizeCheckpoint)
normalized, decision := loader.Normalize("spells", "normalize", normalizeDeps)
if !decision.Reused || normalized.Output.SchemaDigest != stored.SchemaDigest {
t.Fatalf("normalize decision=%#v checkpoint=%#v, want reused", decision, normalized)
}
}
@@ -184,7 +130,7 @@ func TestWorkspaceLoaderInvalidatesMissingCorruptAndMismatchedCheckpoints(t *tes
t.Run("dependency mismatch", func(t *testing.T) {
root := t.TempDir()
recorder := newTestRecorder(t, root)
chunks := []contracts.SourceChunk{{
chunks := []source.Chunk{{
ID: "chunk-1",
SourceID: "source-1",
Content: []byte("chunk content"),
@@ -199,10 +145,41 @@ func TestWorkspaceLoaderInvalidatesMissingCorruptAndMismatchedCheckpoints(t *tes
}
})
t.Run("incompatible workspace schema remains untouched", func(t *testing.T) {
root := t.TempDir()
recorder := newTestRecorder(t, root)
doc := &source.SourceDocument{
ID: "source-1", Kind: "document", Format: "text/plain", Digest: "sha256:source",
Units: []source.SourceUnit{{ID: 1, Kind: "line", Text: "hello", Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1}}},
}
if err := recorder.SourceSucceeded("seriatim", doc); err != nil {
t.Fatalf("SourceSucceeded: %v", err)
}
manifestPath := filepath.Join(root, "source", "manifest.json")
manifest := strings.Replace(string(readFile(t, manifestPath)), coreworkspace.WorkspaceSchemaVersion, coreworkspace.WorkspaceSchemaVersionV1, 1)
if err := os.WriteFile(manifestPath, []byte(manifest), 0o644); err != nil {
t.Fatalf("write legacy manifest: %v", err)
}
beforeManifest := readFile(t, manifestPath)
payloadPath := filepath.Join(root, "source", "source-document.json")
beforePayload := readFile(t, payloadPath)
loader := &WorkspaceLoader{root: root}
if _, decision := loader.Source("seriatim"); decision.Reused || !strings.Contains(decision.Reason, "incompatible") || !strings.Contains(decision.Reason, coreworkspace.WorkspaceSchemaVersionV1) {
t.Fatalf("decision = %#v, want incompatible legacy schema invalidation", decision)
}
if got := readFile(t, manifestPath); string(got) != string(beforeManifest) {
t.Fatal("legacy manifest changed during reuse decision")
}
if got := readFile(t, payloadPath); string(got) != string(beforePayload) {
t.Fatal("legacy payload changed during reuse decision")
}
})
t.Run("corrupt payload", func(t *testing.T) {
root := t.TempDir()
recorder := newTestRecorder(t, root)
chunks := []contracts.SourceChunk{{
chunks := []source.Chunk{{
ID: "chunk-1",
SourceID: "source-1",
Content: []byte("chunk content"),
@@ -286,23 +263,20 @@ func TestWorkspaceRecorderRecordsFailedStages(t *testing.T) {
func TestWorkspaceRecorderRecordsWarningOnlyValidation(t *testing.T) {
root := t.TempDir()
recorder := newTestRecorder(t, root)
output := contracts.NormalizeOutput{
LaneID: "spells",
NormalizerKey: "noop",
SourceID: "source-1",
Payload: contracts.RawPayload{
Content: []byte(`{"ok":true}`),
MediaType: "application/json",
},
schema := contracts.ArtifactSchema{ID: "test.artifact", Name: "test_artifact", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
output := pipeline.CheckpointArtifact{
LaneID: "events", ModuleKey: "noop", SourceID: "source-1",
Artifact: contracts.SerializedArtifact{Kind: "test/artifact", Schema: schema, MediaType: "application/json", Content: []byte(`{"ok":true}`)},
SchemaDigest: contracts.DigestArtifactSchema(schema),
}
warnings := []contracts.Warning{{ReasonCode: "note", Message: "warning"}}
if err := recorder.NormalizeSucceeded("spells", "noop", nil, output, warnings); err != nil {
if err := recorder.NormalizeSucceeded("events", "noop", nil, output, warnings); err != nil {
t.Fatalf("NormalizeSucceeded: %v", err)
}
var manifest coreworkspace.NormalizeLaneManifest
readJSON(t, filepath.Join(root, "normalize", "spells", "manifest.json"), &manifest)
readJSON(t, filepath.Join(root, "normalize", "events", "manifest.json"), &manifest)
if manifest.Status != coreworkspace.StatusSucceeded || manifest.ValidationStatus != "approved_with_warnings" {
t.Fatalf("normalize manifest status = %q validation=%q", manifest.Status, manifest.ValidationStatus)
}

View File

@@ -0,0 +1,81 @@
package contracts
import (
"crypto/sha256"
"encoding/hex"
)
// ArtifactKind is the stable logical identity of a domain artifact.
type ArtifactKind string
// ArtifactSchema describes the durable representation owned by an artifact
// codec. JSONSchema is cloned whenever framework ownership changes.
type ArtifactSchema struct {
ID string `json:"id"`
Name string `json:"name"`
Version string `json:"version"`
JSONSchema []byte `json:"-"`
}
// SerializedArtifact is the domain-neutral representation of a typed
// artifact at an explicit serialization boundary.
type SerializedArtifact struct {
Kind ArtifactKind `json:"kind"`
Schema ArtifactSchema `json:"schema"`
MediaType string `json:"media_type"`
Content []byte `json:"-"`
Metadata map[string]any `json:"metadata,omitempty"`
}
// SerializedOutput associates a domain-neutral artifact with the pipeline
// operation that produced it. Provenance remains outside codec-owned bytes.
type SerializedOutput struct {
LaneID string `json:"lane_id"`
NormalizerKey string `json:"normalizer_key"`
SourceID string `json:"source_id,omitempty"`
Artifact SerializedArtifact `json:"artifact"`
}
// ArtifactCodec owns the stable encoding for one concrete artifact type.
type ArtifactCodec[T any] interface {
Kind() ArtifactKind
Schema() ArtifactSchema
MediaType() string
Encode(T) ([]byte, error)
Decode([]byte) (T, error)
}
// DigestArtifactSchema returns the SHA-256 digest of the exact JSON Schema
// bytes. Schema formatting is therefore part of the registered identity.
func DigestArtifactSchema(schema ArtifactSchema) string {
sum := sha256.Sum256(schema.JSONSchema)
return "sha256:" + hex.EncodeToString(sum[:])
}
func CloneArtifactSchema(schema ArtifactSchema) ArtifactSchema {
schema.JSONSchema = append([]byte(nil), schema.JSONSchema...)
return schema
}
func CloneSerializedArtifact(artifact SerializedArtifact) SerializedArtifact {
artifact.Schema = CloneArtifactSchema(artifact.Schema)
artifact.Content = append([]byte(nil), artifact.Content...)
artifact.Metadata = cloneArtifactMetadata(artifact.Metadata)
return artifact
}
func CloneSerializedOutput(output SerializedOutput) SerializedOutput {
output.Artifact = CloneSerializedArtifact(output.Artifact)
return output
}
func cloneArtifactMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
out := make(map[string]any, len(metadata))
for key, value := range metadata {
out[key] = value
}
return out
}

View File

@@ -0,0 +1,45 @@
package contracts
import "testing"
func TestDigestArtifactSchemaUsesExactBytes(t *testing.T) {
first := ArtifactSchema{JSONSchema: []byte(`{"type":"object"}`)}
second := ArtifactSchema{JSONSchema: []byte("{\n \"type\": \"object\"\n}")}
if got := DigestArtifactSchema(first); got != "sha256:a2c799262a3ce3c19ef5cdd983bf3d12b43ab3c426227091b909dcb7054738c0" {
t.Fatalf("DigestArtifactSchema() = %q, want stable SHA-256", got)
}
if DigestArtifactSchema(first) == DigestArtifactSchema(second) {
t.Fatal("schema digests match for different exact bytes")
}
}
func TestArtifactCloneHelpersOwnSlicesAndMaps(t *testing.T) {
schema := ArtifactSchema{ID: "notes.v1", Name: "notes", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
artifact := SerializedArtifact{
Kind: "test/notes",
Schema: schema,
MediaType: "application/json",
Content: []byte(`{"items":["one"]}`),
Metadata: map[string]any{"origin": "test"},
}
clonedSchema := CloneArtifactSchema(schema)
cloned := CloneSerializedArtifact(artifact)
schema.JSONSchema[0] = '['
artifact.Content[0] = '['
artifact.Metadata["origin"] = "changed"
if string(clonedSchema.JSONSchema) != `{"type":"object"}` {
t.Fatalf("cloned schema = %q, want original bytes", clonedSchema.JSONSchema)
}
if string(cloned.Schema.JSONSchema) != `{"type":"object"}` {
t.Fatalf("serialized artifact schema = %q, want original bytes", cloned.Schema.JSONSchema)
}
if string(cloned.Content) != `{"items":["one"]}` {
t.Fatalf("cloned content = %q, want original bytes", cloned.Content)
}
if cloned.Metadata["origin"] != "test" {
t.Fatalf("cloned metadata = %#v, want independent map", cloned.Metadata)
}
}

View File

@@ -1,298 +0,0 @@
package contracts_test
import (
"context"
"encoding/json"
"errors"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
var _ contracts.InputAdapter = compositionAdapter{}
var _ contracts.Chunker = compositionChunker{}
var _ contracts.Extractor = compositionExtractor{}
var _ contracts.Merger = compositionMerger{}
var _ contracts.Normalizer = compositionNormalizer{}
var _ contracts.Validator = compositionValidator{}
var _ contracts.StructuredLLMClient = compositionLLMClient{}
var _ contracts.OutputEncoder = compositionOutputEncoder{}
func TestContractsComposeAcrossPackages(t *testing.T) {
ctx := context.Background()
adapter := compositionAdapter{}
chunker := compositionChunker{}
extractor := compositionExtractor{}
merger := compositionMerger{}
normalizer := compositionNormalizer{}
encoder := compositionOutputEncoder{}
doc, err := adapter.Parse(ctx, contracts.ParseRequest{SourceID: "source-1"})
if err != nil {
t.Fatalf("Parse() error = %v, want nil", err)
}
if err := source.ValidateDocument(doc); err != nil {
t.Fatalf("ValidateDocument() error = %v, want nil", err)
}
chunking, err := chunker.Chunk(ctx, contracts.ChunkRequest{
Source: doc,
LLMClient: compositionLLMClient{},
Metadata: map[string]any{"max_units": 2},
})
if err != nil {
t.Fatalf("Chunk() error = %v, want nil", err)
}
if len(chunking.Chunks) != 1 {
t.Fatalf("len(Chunks) = %d, want 1", len(chunking.Chunks))
}
extraction, err := extractor.Extract(ctx, contracts.ExtractionRequest{
Source: doc,
Chunk: &chunking.Chunks[0],
AmbientContext: map[string]any{"synopsis": "example synopsis"},
})
if err != nil {
t.Fatalf("Extract() error = %v, want nil", err)
}
if extraction.Output.Payload.MediaType != "application/json" {
t.Fatalf("extract media type = %q, want application/json", extraction.Output.Payload.MediaType)
}
merge, err := merger.Merge(ctx, contracts.MergeRequest{
Source: doc,
LaneID: "generic-lane",
ExtractOutputs: []contracts.ExtractOutput{extraction.Output},
})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
}
if string(merge.Output.Payload.Content) != `{"value":"example"}` {
t.Fatalf("merge output = %s, want extract payload", merge.Output.Payload.Content)
}
normalize, err := normalizer.Normalize(ctx, contracts.NormalizeRequest{
Source: doc,
LaneID: "generic-lane",
MergeOutput: merge.Output,
})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if string(normalize.Output.Payload.Content) != `{"value":"example"}` {
t.Fatalf("normalize output = %s, want merge payload", normalize.Output.Payload.Content)
}
output, err := encoder.Encode(ctx, contracts.OutputRequest{
Manifest: artifacts.RunManifest{RunID: "run-1"},
NormalizeOutputs: []contracts.NormalizeOutput{normalize.Output},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
if len(output.Files) != 1 {
t.Fatalf("len(Files) = %d, want 1", len(output.Files))
}
if output.Files[0].ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", output.Files[0].ContentType)
}
if len(output.Files[0].Bytes) == 0 {
t.Fatal("len(Bytes) = 0, want encoded bytes")
}
}
type compositionAdapter struct{}
func (adapter compositionAdapter) Key() string {
return "generic-input"
}
func (adapter compositionAdapter) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
return &source.SourceDocument{
ID: req.SourceID,
Kind: "document",
Format: "text/plain",
Digest: "sha256:abc123",
Units: []source.SourceUnit{
{ID: 1, Kind: "unit", Text: "First source unit."},
{ID: 2, Kind: "unit", Text: "Second source unit."},
},
}, nil
}
type compositionChunker struct{}
func (chunker compositionChunker) Key() string {
return "generic-chunker"
}
func (chunker compositionChunker) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (chunker compositionChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
if req.Source == nil {
return contracts.ChunkResult{}, errors.New("source document is required")
}
if req.LLMClient == nil {
return contracts.ChunkResult{}, errors.New("structured llm client is required")
}
return contracts.ChunkResult{
Chunks: []contracts.SourceChunk{
{
ID: req.Source.ID + ":chunk:0",
SourceID: req.Source.ID,
Index: 0,
StartUnitID: req.Source.Units[0].ID,
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
Content: []byte(`{"units":[{"id":1,"kind":"unit","text":"First source unit."},{"id":2,"kind":"unit","text":"Second source unit."}]}`),
MediaType: "application/json",
Units: append([]source.SourceUnit(nil), req.Source.Units...),
Metadata: map[string]any{"strategy": "whole-document"},
},
},
}, nil
}
type compositionLLMClient struct{}
func (client compositionLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
return contracts.StructuredCompletionResponse{}, nil
}
type compositionExtractor struct{}
func (extractor compositionExtractor) Key() string {
return "generic-extractor"
}
func (extractor compositionExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (extractor compositionExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
if req.Source == nil {
return contracts.ExtractionResult{}, errors.New("source document is required")
}
if req.AmbientContext["synopsis"] == "" {
return contracts.ExtractionResult{}, errors.New("ambient synopsis is required")
}
return contracts.ExtractionResult{
Output: contracts.ExtractOutput{
Schema: contracts.ResponseSchema{ID: "schema-id", Name: "schema-name", Version: "v1"},
Payload: contracts.RawPayload{
Content: []byte(`{"value":"example"}`),
MediaType: "application/json",
},
},
}, nil
}
type compositionMerger struct{}
func (merger compositionMerger) Key() string {
return "generic-merger"
}
func (merger compositionMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
output := req.ExtractOutputs[0]
return contracts.MergeResult{Output: contracts.MergeOutput{
LaneID: req.LaneID,
MergerKey: merger.Key(),
SourceID: output.SourceID,
Schema: output.Schema,
Payload: cloneCompositionPayload(output.Payload),
}}, nil
}
type compositionNormalizer struct{}
func (normalizer compositionNormalizer) Key() string {
return "generic-normalizer"
}
func (normalizer compositionNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (normalizer compositionNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
return contracts.NormalizeResult{Output: contracts.NormalizeOutput{
LaneID: req.LaneID,
NormalizerKey: normalizer.Key(),
SourceID: req.MergeOutput.SourceID,
Schema: req.MergeOutput.Schema,
Payload: cloneCompositionPayload(req.MergeOutput.Payload),
}}, nil
}
func cloneCompositionPayload(payload contracts.RawPayload) contracts.RawPayload {
return contracts.RawPayload{
Content: append([]byte(nil), payload.Content...),
MediaType: payload.MediaType,
Metadata: cloneCompositionMetadata(payload.Metadata),
Warnings: append([]contracts.Warning(nil), payload.Warnings...),
}
}
func cloneCompositionMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
out := make(map[string]any, len(metadata))
for key, value := range metadata {
out[key] = value
}
return out
}
type compositionValidator struct{}
func (validator compositionValidator) Name() string {
return "generic-validator"
}
func (validator compositionValidator) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassDeterministic
}
func (validator compositionValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
return contracts.ValidationResult{
Approved: true,
ReasonCode: "accepted",
Message: "output accepted",
}, nil
}
type compositionOutputEncoder struct{}
func (encoder compositionOutputEncoder) Key() string {
return "generic-output"
}
func (encoder compositionOutputEncoder) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
payload := struct {
RunID string `json:"run_id"`
OutputCount int `json:"output_count"`
}{
RunID: req.Manifest.RunID,
OutputCount: len(req.NormalizeOutputs),
}
encoded, err := json.Marshal(payload)
if err != nil {
return contracts.OutputResult{}, err
}
return contracts.OutputResult{
Files: []contracts.OutputFile{
{
Name: "artifacts/generic.json",
ContentType: "application/json",
Bytes: encoded,
},
},
}, nil
}

View File

@@ -129,7 +129,6 @@ type ParseRequest struct {
Path string `json:"path,omitempty"`
Raw []byte `json:"-"`
LLMProfile string `json:"llm_profile,omitempty"`
Options map[string]any `json:"options,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
@@ -138,32 +137,18 @@ type InputAdapter interface {
Parse(ctx context.Context, req ParseRequest) (*source.SourceDocument, error)
}
type SourceChunk struct {
ID string `json:"id"`
SourceID string `json:"source_id"`
Index int `json:"index"`
StartUnitID int `json:"start_unit_id"`
EndUnitID int `json:"end_unit_id"`
Content []byte `json:"-"`
MediaType string `json:"media_type"`
Units []source.SourceUnit `json:"units"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type ChunkRequest struct {
Source *source.SourceDocument `json:"-"`
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
SessionID string `json:"session_id,omitempty"`
References ReferenceSet `json:"references,omitempty"`
LLMClient StructuredLLMClient `json:"-"`
LLMProfile string `json:"llm_profile,omitempty"`
Options map[string]any `json:"options,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type ChunkResult struct {
Chunks []SourceChunk `json:"chunks"`
Warnings []Warning `json:"warnings,omitempty"`
Chunks []source.Chunk `json:"chunks"`
Warnings []Warning `json:"warnings,omitempty"`
}
type Chunker interface {
@@ -222,37 +207,6 @@ type ReferenceSet struct {
Slots map[string]ResolvedReferenceSlot `json:"slots,omitempty"`
}
type ExtractionRequest struct {
Source *source.SourceDocument `json:"-"`
Chunk *SourceChunk `json:"chunk,omitempty"`
AmbientContext map[string]any `json:"ambient_context,omitempty"`
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
SessionID string `json:"session_id,omitempty"`
References ReferenceSet `json:"references,omitempty"`
LLMClient StructuredLLMClient `json:"-"`
LLMProfile string `json:"llm_profile,omitempty"`
Options map[string]any `json:"options,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type ExtractionResult struct {
Output ExtractOutput `json:"output"`
Warnings []Warning `json:"warnings,omitempty"`
}
type Extractor interface {
Key() string
ReferenceSlots() []ReferenceSlot
Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error)
}
type RawPayload struct {
Content []byte `json:"-"`
MediaType string `json:"media_type"`
Metadata map[string]any `json:"metadata,omitempty"`
Warnings []Warning `json:"warnings,omitempty"`
}
type ExecutionClass string
const (
@@ -260,29 +214,6 @@ const (
ExecutionClassLLMBacked ExecutionClass = "llm_backed"
)
type ValidationRequest struct {
Stage string `json:"stage"`
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key"`
Source *source.SourceDocument `json:"-"`
SourceID string `json:"source_id,omitempty"`
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
SessionID string `json:"session_id,omitempty"`
References ReferenceSet `json:"references,omitempty"`
LLMClient StructuredLLMClient `json:"-"`
LLMProfile string `json:"llm_profile,omitempty"`
Options map[string]any `json:"options,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
Schema ResponseSchema `json:"schema,omitempty"`
Payload RawPayload `json:"payload"`
ChunkID string `json:"chunk_id,omitempty"`
ChunkIndex int `json:"chunk_index,omitempty"`
Chunk *SourceChunk `json:"chunk,omitempty"`
Chunks []SourceChunk `json:"chunks,omitempty"`
ExtractOutputs []ExtractOutput `json:"extract_outputs,omitempty"`
MergeOutput MergeOutput `json:"merge_output,omitempty"`
}
type ValidationResult struct {
Approved bool `json:"approved"`
ReasonCode string `json:"reason_code,omitempty"`
@@ -291,92 +222,6 @@ type ValidationResult struct {
Warnings []Warning `json:"warnings,omitempty"`
}
type Validator interface {
Name() string
ExecutionClass() ExecutionClass
Validate(ctx context.Context, req ValidationRequest) (ValidationResult, error)
}
type ResponseSchema struct {
ID string `json:"id,omitempty"`
Name string `json:"name,omitempty"`
Version string `json:"version,omitempty"`
JSONSchema []byte `json:"-"`
}
type ExtractOutput struct {
LaneID string `json:"lane_id"`
ExtractorKey string `json:"extractor_key"`
SourceID string `json:"source_id"`
ChunkID string `json:"chunk_id"`
ChunkIndex int `json:"chunk_index"`
Schema ResponseSchema `json:"schema,omitempty"`
Payload RawPayload `json:"payload"`
}
type MergeRequest struct {
Source *source.SourceDocument `json:"-"`
LaneID string `json:"lane_id"`
ExtractOutputs []ExtractOutput `json:"extract_outputs"`
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
SessionID string `json:"session_id,omitempty"`
References ReferenceSet `json:"references,omitempty"`
LLMClient StructuredLLMClient `json:"-"`
LLMProfile string `json:"llm_profile,omitempty"`
Options map[string]any `json:"options,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type MergeResult struct {
Output MergeOutput `json:"output"`
Warnings []Warning `json:"warnings,omitempty"`
}
type MergeOutput struct {
LaneID string `json:"lane_id"`
MergerKey string `json:"merger_key"`
SourceID string `json:"source_id,omitempty"`
Schema ResponseSchema `json:"schema,omitempty"`
Payload RawPayload `json:"payload"`
}
type Merger interface {
Key() string
Merge(ctx context.Context, req MergeRequest) (MergeResult, error)
}
type NormalizeRequest struct {
Source *source.SourceDocument `json:"-"`
LaneID string `json:"lane_id"`
MergeOutput MergeOutput `json:"merge_output"`
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
SessionID string `json:"session_id,omitempty"`
References ReferenceSet `json:"references,omitempty"`
LLMClient StructuredLLMClient `json:"-"`
LLMProfile string `json:"llm_profile,omitempty"`
Options map[string]any `json:"options,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type NormalizeResult struct {
Output NormalizeOutput `json:"output"`
Warnings []Warning `json:"warnings,omitempty"`
}
type NormalizeOutput struct {
LaneID string `json:"lane_id"`
NormalizerKey string `json:"normalizer_key"`
SourceID string `json:"source_id,omitempty"`
Schema ResponseSchema `json:"schema,omitempty"`
Payload RawPayload `json:"payload"`
}
type Normalizer interface {
Key() string
ReferenceSlots() []ReferenceSlot
Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error)
}
type Warning struct {
Scope string `json:"scope,omitempty"`
ReasonCode string `json:"reason_code"`
@@ -385,11 +230,10 @@ type Warning struct {
type OutputRequest struct {
Manifest artifacts.RunManifest `json:"manifest"`
NormalizeOutputs []NormalizeOutput `json:"normalize_outputs,omitempty"`
NormalizeOutputs []SerializedOutput `json:"normalize_outputs,omitempty"`
Rejected []RejectedOutput `json:"rejected,omitempty"`
Warnings []Warning `json:"warnings,omitempty"`
LLMProfile string `json:"llm_profile,omitempty"`
Options map[string]any `json:"options,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}

View File

@@ -12,14 +12,13 @@ import (
var _ InputAdapter = fakeAdapter{}
var _ Chunker = fakeChunker{}
var _ Extractor = fakeExtractor{}
var _ Merger = fakeMerger{}
var _ Normalizer = fakeNormalizer{}
var _ Validator = fakeValidator{}
var _ Extractor[fakeArtifact] = fakeExtractor{}
var _ Merger[fakeArtifact] = fakeMerger{}
var _ Normalizer[fakeArtifact] = fakeNormalizer{}
var _ StructuredLLMClient = fakeLLMClient{}
var _ OutputEncoder = fakeOutputEncoder{}
func TestFakeExtractorReturnsRawOutput(t *testing.T) {
func TestFakeExtractorReturnsTypedOutput(t *testing.T) {
extractor := fakeExtractor{
key: "generic-extractor",
}
@@ -33,7 +32,7 @@ func TestFakeExtractorReturnsRawOutput(t *testing.T) {
},
}
result, err := extractor.Extract(context.Background(), ExtractionRequest{Source: doc})
result, err := extractor.Extract(context.Background(), TypedExtractionRequest{Source: doc})
if err != nil {
t.Fatalf("Extract() error = %v, want nil", err)
}
@@ -41,14 +40,8 @@ func TestFakeExtractorReturnsRawOutput(t *testing.T) {
if extractor.Key() != "generic-extractor" {
t.Fatalf("Key() = %q, want generic-extractor", extractor.Key())
}
if result.Output.ExtractorKey != "" {
t.Fatalf("ExtractorKey = %q, want runner-owned empty value", result.Output.ExtractorKey)
}
if result.Output.Schema.Version != "v1" {
t.Fatalf("Schema.Version = %q, want v1", result.Output.Schema.Version)
}
if result.Output.Payload.MediaType != "application/json" || string(result.Output.Payload.Content) != `{"value":"example"}` {
t.Fatalf("payload = %q %s, want JSON raw output", result.Output.Payload.MediaType, result.Output.Payload.Content)
if result.Value.Value != "example" {
t.Fatalf("Value = %q, want example", result.Value.Value)
}
}
@@ -78,26 +71,26 @@ func TestFakeChunkerReturnsSourceChunks(t *testing.T) {
chunk := result.Chunks[0]
if chunk.ID != "source-1:chunk:0" {
t.Fatalf("SourceChunk.ID = %q, want source-1:chunk:0", chunk.ID)
t.Fatalf("source.Chunk.ID = %q, want source-1:chunk:0", chunk.ID)
}
if chunk.SourceID != doc.ID {
t.Fatalf("SourceChunk.SourceID = %q, want %q", chunk.SourceID, doc.ID)
t.Fatalf("source.Chunk.SourceID = %q, want %q", chunk.SourceID, doc.ID)
}
if chunk.Index != 0 {
t.Fatalf("SourceChunk.Index = %d, want 0", chunk.Index)
t.Fatalf("source.Chunk.Index = %d, want 0", chunk.Index)
}
if chunk.StartUnitID != 1 || chunk.EndUnitID != 1 {
t.Fatalf("SourceChunk boundaries = %d-%d, want 1-1", chunk.StartUnitID, chunk.EndUnitID)
if chunk.Ref.StartUnitID != 1 || chunk.Ref.EndUnitID != 1 {
t.Fatalf("source.Chunk.Ref = %#v, want source-1:1-1", chunk.Ref)
}
if chunk.MediaType != "application/json" || string(chunk.Content) != `{"units":[{"id":1,"kind":"section","text":"Source text."}]}` {
t.Fatalf("SourceChunk payload = %q %s, want JSON units", chunk.MediaType, chunk.Content)
t.Fatalf("source.Chunk payload = %q %s, want JSON units", chunk.MediaType, chunk.Content)
}
if len(chunk.Units) != 1 {
t.Fatalf("len(SourceChunk.Units) = %d, want 1", len(chunk.Units))
t.Fatalf("len(source.Chunk.Units) = %d, want 1", len(chunk.Units))
}
}
func TestFakeChunkerReceivesLLMClient(t *testing.T) {
func TestFakeChunkerReceivesPerRunContext(t *testing.T) {
doc := &source.SourceDocument{
ID: "source-1",
Kind: "document",
@@ -107,14 +100,13 @@ func TestFakeChunkerReceivesLLMClient(t *testing.T) {
{ID: 1, Kind: "section", Text: "Source text."},
},
}
client := fakeLLMClient{}
chunker := &recordingChunker{key: "llm-chunker"}
if _, err := chunker.Chunk(context.Background(), ChunkRequest{Source: doc, LLMClient: client}); err != nil {
if _, err := chunker.Chunk(context.Background(), ChunkRequest{Source: doc, SessionID: "session", LLMProfile: "profile"}); err != nil {
t.Fatalf("Chunk() error = %v, want nil", err)
}
if chunker.request.LLMClient == nil {
t.Fatal("ChunkRequest.LLMClient = nil, want structured LLM client")
if chunker.request.SessionID != "session" || chunker.request.LLMProfile != "profile" {
t.Fatalf("ChunkRequest = %#v, want per-run session and profile", chunker.request)
}
}
@@ -130,18 +122,17 @@ func TestFakeExtractorReceivesChunkAndAmbientContext(t *testing.T) {
{ID: 2, Kind: "section", Text: "Second source text."},
},
}
chunk := SourceChunk{
ID: "source-1:chunk:1",
SourceID: doc.ID,
Index: 1,
StartUnitID: 2,
EndUnitID: 2,
Content: []byte(`{"units":[{"id":2,"kind":"section","text":"Second source text."}]}`),
MediaType: "application/json",
Units: []source.SourceUnit{doc.Units[1]},
chunk := source.Chunk{
ID: "source-1:chunk:1",
SourceID: doc.ID,
Index: 1,
Ref: source.SourceRef{SourceID: doc.ID, StartUnitID: 2, EndUnitID: 2},
Content: []byte(`{"units":[{"id":2,"kind":"section","text":"Second source text."}]}`),
MediaType: "application/json",
Units: []source.SourceUnit{doc.Units[1]},
}
result, err := extractor.Extract(context.Background(), ExtractionRequest{
result, err := extractor.Extract(context.Background(), TypedExtractionRequest{
Source: doc,
Chunk: &chunk,
AmbientContext: map[string]any{"mode": "chunked"},
@@ -149,11 +140,8 @@ func TestFakeExtractorReceivesChunkAndAmbientContext(t *testing.T) {
if err != nil {
t.Fatalf("Extract() error = %v, want nil", err)
}
if result.Output.ChunkID != "" || result.Output.ChunkIndex != 0 {
t.Fatalf("chunk provenance = %q/%d, want runner-owned zero values", result.Output.ChunkID, result.Output.ChunkIndex)
}
if string(result.Output.Payload.Content) != `{"value":"chunked"}` {
t.Fatalf("Payload.Content = %s, want chunked payload", result.Output.Payload.Content)
if result.Value.Value != "chunked" {
t.Fatalf("Value = %q, want chunked", result.Value.Value)
}
}
@@ -322,8 +310,8 @@ func TestLLMInputSetCloneCopiesContent(t *testing.T) {
}
}
func TestResponseSchemaJSONOmitRawSchemaContent(t *testing.T) {
schema := ResponseSchema{
func TestArtifactSchemaJSONOmitsSchemaContent(t *testing.T) {
schema := ArtifactSchema{
ID: "schema-id",
Name: "schema-name",
Version: "v1",
@@ -349,26 +337,14 @@ func TestResponseSchemaJSONOmitRawSchemaContent(t *testing.T) {
}
func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
extractOutput := ExtractOutput{
LaneID: "generic-lane",
ExtractorKey: "generic-extractor",
SourceID: "source-1",
ChunkID: "source-1:chunk:0",
ChunkIndex: 0,
Schema: ResponseSchema{ID: "schema-id", Name: "schema-name", Version: "v1"},
Payload: RawPayload{
Content: []byte(`{"value":"example"}`),
MediaType: "application/json",
Metadata: map[string]any{"confidence": 0.75},
},
}
extractOutput := ExtractArtifact[fakeArtifact]{LaneID: "generic-lane", ExtractorKey: "generic-extractor", SourceID: "source-1", ChunkID: "source-1:chunk:0", ChunkIndex: 0, Value: fakeArtifact{Value: "example"}}
merger := fakeMerger{key: "generic-merger"}
normalizer := fakeNormalizer{key: "generic-normalizer"}
encoder := fakeOutputEncoder{key: "generic-output"}
merged, err := merger.Merge(context.Background(), MergeRequest{
merged, err := merger.Merge(context.Background(), TypedMergeRequest[fakeArtifact]{
LaneID: "generic-lane",
ExtractOutputs: []ExtractOutput{extractOutput},
ExtractOutputs: []ExtractArtifact[fakeArtifact]{extractOutput},
})
if err != nil {
t.Fatalf("Merge() error = %v, want nil", err)
@@ -376,13 +352,13 @@ func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
if merger.Key() != "generic-merger" {
t.Fatalf("Merger.Key() = %q, want generic-merger", merger.Key())
}
if string(merged.Output.Payload.Content) != `{"value":"example"}` {
t.Fatalf("merged content = %s, want raw extract content", merged.Output.Payload.Content)
if merged.Value.Value != "example" {
t.Fatalf("merged value = %q, want example", merged.Value.Value)
}
normalized, err := normalizer.Normalize(context.Background(), NormalizeRequest{
normalized, err := normalizer.Normalize(context.Background(), TypedNormalizeRequest[fakeArtifact]{
LaneID: "generic-lane",
MergeOutput: merged.Output,
MergeOutput: MergeArtifact[fakeArtifact]{LaneID: "generic-lane", MergerKey: merger.Key(), SourceID: "source-1", Value: merged.Value},
})
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
@@ -390,13 +366,13 @@ func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
if normalizer.Key() != "generic-normalizer" {
t.Fatalf("Normalizer.Key() = %q, want generic-normalizer", normalizer.Key())
}
if string(normalized.Output.Payload.Content) != `{"value":"example"}` {
t.Fatalf("normalized content = %s, want raw merge content", normalized.Output.Payload.Content)
if normalized.Value.Value != "example" {
t.Fatalf("normalized value = %q, want example", normalized.Value.Value)
}
encoded, err := encoder.Encode(context.Background(), OutputRequest{
Manifest: artifacts.RunManifest{RunID: "run-1"},
NormalizeOutputs: []NormalizeOutput{normalized.Output},
NormalizeOutputs: []SerializedOutput{{LaneID: "generic-lane", NormalizerKey: normalizer.Key(), SourceID: "source-1", Artifact: SerializedArtifact{Kind: "test/artifact", Schema: ArtifactSchema{ID: "schema-id", Name: "schema-name", Version: "v1"}, MediaType: "application/json", Content: []byte(`{"value":"example"}`)}}},
})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
@@ -472,16 +448,19 @@ func (chunker fakeChunker) ReferenceSlots() []ReferenceSlot {
func (chunker fakeChunker) Chunk(ctx context.Context, req ChunkRequest) (ChunkResult, error) {
return ChunkResult{
Chunks: []SourceChunk{
Chunks: []source.Chunk{
{
ID: req.Source.ID + ":chunk:0",
SourceID: req.Source.ID,
Index: 0,
StartUnitID: req.Source.Units[0].ID,
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
Content: []byte(`{"units":[{"id":1,"kind":"section","text":"Source text."}]}`),
MediaType: "application/json",
Units: append([]source.SourceUnit(nil), req.Source.Units...),
ID: req.Source.ID + ":chunk:0",
SourceID: req.Source.ID,
Index: 0,
Ref: source.SourceRef{
SourceID: req.Source.ID,
StartUnitID: req.Source.Units[0].ID,
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
},
Content: []byte(`{"units":[{"id":1,"kind":"section","text":"Source text."}]}`),
MediaType: "application/json",
Units: append([]source.SourceUnit(nil), req.Source.Units...),
},
},
}, nil
@@ -509,6 +488,8 @@ type fakeExtractor struct {
key string
}
type fakeArtifact struct{ Value string }
func (extractor fakeExtractor) Key() string {
return extractor.key
}
@@ -517,21 +498,12 @@ func (extractor fakeExtractor) ReferenceSlots() []ReferenceSlot {
return nil
}
func (extractor fakeExtractor) Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error) {
payload := json.RawMessage(`{"value":"example"}`)
func (extractor fakeExtractor) Extract(ctx context.Context, req TypedExtractionRequest) (TypedExtractionResult[fakeArtifact], error) {
value := "example"
if req.AmbientContext["mode"] == "chunked" {
payload = json.RawMessage(`{"value":"chunked"}`)
value = "chunked"
}
return ExtractionResult{
Output: ExtractOutput{
Schema: ResponseSchema{ID: "schema-id", Name: "schema-name", Version: "v1"},
Payload: RawPayload{
Content: append([]byte(nil), payload...),
MediaType: "application/json",
},
},
}, nil
return TypedExtractionResult[fakeArtifact]{Value: fakeArtifact{Value: value}}, nil
}
type fakeMerger struct {
@@ -542,15 +514,8 @@ func (merger fakeMerger) Key() string {
return merger.key
}
func (merger fakeMerger) Merge(ctx context.Context, req MergeRequest) (MergeResult, error) {
output := req.ExtractOutputs[0]
return MergeResult{Output: MergeOutput{
LaneID: req.LaneID,
MergerKey: merger.key,
SourceID: output.SourceID,
Schema: output.Schema,
Payload: cloneTestRawPayload(output.Payload),
}}, nil
func (merger fakeMerger) Merge(ctx context.Context, req TypedMergeRequest[fakeArtifact]) (TypedMergeResult[fakeArtifact], error) {
return TypedMergeResult[fakeArtifact]{Value: req.ExtractOutputs[0].Value}, nil
}
type fakeNormalizer struct {
@@ -565,54 +530,8 @@ func (normalizer fakeNormalizer) ReferenceSlots() []ReferenceSlot {
return nil
}
func (normalizer fakeNormalizer) Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error) {
return NormalizeResult{Output: NormalizeOutput{
LaneID: req.LaneID,
NormalizerKey: normalizer.key,
SourceID: req.MergeOutput.SourceID,
Schema: req.MergeOutput.Schema,
Payload: cloneTestRawPayload(req.MergeOutput.Payload),
}}, nil
}
func cloneTestRawPayload(payload RawPayload) RawPayload {
return RawPayload{
Content: append([]byte(nil), payload.Content...),
MediaType: payload.MediaType,
Metadata: cloneTestMetadata(payload.Metadata),
Warnings: append([]Warning(nil), payload.Warnings...),
}
}
func cloneTestMetadata(metadata map[string]any) map[string]any {
if len(metadata) == 0 {
return nil
}
out := make(map[string]any, len(metadata))
for key, value := range metadata {
out[key] = value
}
return out
}
type fakeValidator struct {
name string
}
func (validator fakeValidator) Name() string {
return validator.name
}
func (validator fakeValidator) ExecutionClass() ExecutionClass {
return ExecutionClassDeterministic
}
func (validator fakeValidator) Validate(ctx context.Context, req ValidationRequest) (ValidationResult, error) {
return ValidationResult{
Approved: true,
ReasonCode: "accepted",
Message: "output accepted",
}, nil
func (normalizer fakeNormalizer) Normalize(ctx context.Context, req TypedNormalizeRequest[fakeArtifact]) (TypedNormalizeResult[fakeArtifact], error) {
return TypedNormalizeResult[fakeArtifact]{Value: req.MergeOutput.Value}, nil
}
type fakeLLMClient struct{}

View File

@@ -0,0 +1,164 @@
package contracts
import (
"context"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
)
// ExtractArtifact carries a typed per-chunk value with framework provenance.
type ExtractArtifact[T any] struct {
LaneID string
ExtractorKey string
SourceID string
ChunkID string
ChunkIndex int
ChunkRef source.SourceRef
Value T
}
// MergeArtifact carries a typed merged value with framework provenance.
type MergeArtifact[T any] struct {
LaneID string
MergerKey string
SourceID string
Value T
}
// NormalizeArtifact carries a typed normalized value with framework provenance.
type NormalizeArtifact[T any] struct {
LaneID string
NormalizerKey string
SourceID string
Value T
}
type TypedExtractionRequest struct {
Source *source.SourceDocument
Chunk *source.Chunk
AmbientContext map[string]any
SourceInput LLMInputMaterial
SessionID string
References ReferenceSet
LLMProfile string
Metadata map[string]any
}
type TypedExtractionResult[T any] struct {
Value T
Warnings []Warning
}
type Extractor[T any] interface {
Key() string
ReferenceSlots() []ReferenceSlot
Extract(context.Context, TypedExtractionRequest) (TypedExtractionResult[T], error)
}
type TypedMergeRequest[T any] struct {
Source *source.SourceDocument
LaneID string
ExtractOutputs []ExtractArtifact[T]
SourceInput LLMInputMaterial
SessionID string
References ReferenceSet
LLMProfile string
Metadata map[string]any
}
type TypedMergeResult[T any] struct {
Value T
Warnings []Warning
}
type Merger[T any] interface {
Key() string
Merge(context.Context, TypedMergeRequest[T]) (TypedMergeResult[T], error)
}
type TypedNormalizeRequest[T any] struct {
Source *source.SourceDocument
LaneID string
MergeOutput MergeArtifact[T]
SourceInput LLMInputMaterial
SessionID string
References ReferenceSet
LLMProfile string
Metadata map[string]any
}
type TypedNormalizeResult[T any] struct {
Value T
Warnings []Warning
}
type Normalizer[T any] interface {
Key() string
ReferenceSlots() []ReferenceSlot
Normalize(context.Context, TypedNormalizeRequest[T]) (TypedNormalizeResult[T], error)
}
type TypedValidationRequest[T any] struct {
Stage string
LaneID string
ModuleKey string
Source *source.SourceDocument
SourceID string
SourceInput LLMInputMaterial
SessionID string
References ReferenceSet
LLMProfile string
Metadata map[string]any
Chunk *source.Chunk
Chunks []source.Chunk
Ref source.SourceRef
Value T
}
type TypedValidator[T any] interface {
Name() string
ExecutionClass() ExecutionClass
Validate(context.Context, TypedValidationRequest[T]) (ValidationResult, error)
}
type ChunkValidationRequest struct {
ModuleKey string
Source *source.SourceDocument
SourceID string
SourceInput LLMInputMaterial
SessionID string
References ReferenceSet
LLMProfile string
Metadata map[string]any
Chunks []source.Chunk
}
type ChunkValidator interface {
Name() string
ExecutionClass() ExecutionClass
Validate(context.Context, ChunkValidationRequest) (ValidationResult, error)
}
type SerializedValidationRequest struct {
Stage string
LaneID string
ModuleKey string
Source *source.SourceDocument
SourceID string
SourceInput LLMInputMaterial
SessionID string
References ReferenceSet
LLMProfile string
Metadata map[string]any
Chunk *source.Chunk
Chunks []source.Chunk
Schema ArtifactSchema
MediaType string
Content []byte
}
type SerializedValidator interface {
Name() string
ExecutionClass() ExecutionClass
Validate(context.Context, SerializedValidationRequest) (ValidationResult, error)
}

View File

@@ -0,0 +1,318 @@
package pipeline
import (
"bytes"
"encoding/json"
"fmt"
"reflect"
"sort"
"strings"
"github.com/santhosh-tekuri/jsonschema/v6"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type ArtifactCodecSpec struct {
Kind contracts.ArtifactKind
Schema contracts.ArtifactSchema
SchemaDigest string
MediaType string
}
// ArtifactCodecTypeError reports a value that does not have the exact Go type
// registered for an artifact kind.
type ArtifactCodecTypeError struct {
Operation string
Kind contracts.ArtifactKind
ExpectedType string
ActualType string
}
func (e *ArtifactCodecTypeError) Error() string {
return fmt.Sprintf("%s artifact %q: expected exact Go type %s, got %s", e.Operation, e.Kind, e.ExpectedType, e.ActualType)
}
// ArtifactCodecCompatibilityError reports serialized metadata that does not
// identify the registered representation for an artifact kind.
type ArtifactCodecCompatibilityError struct {
Kind contracts.ArtifactKind
Reason string
}
func (e *ArtifactCodecCompatibilityError) Error() string {
return fmt.Sprintf("decode artifact %q: %s", e.Kind, e.Reason)
}
// ArtifactCodecOperationError preserves an encode or decode failure from the
// registered domain codec.
type ArtifactCodecOperationError struct {
Operation string
Kind contracts.ArtifactKind
Err error
}
func (e *ArtifactCodecOperationError) Error() string {
return fmt.Sprintf("%s artifact %q: %v", e.Operation, e.Kind, e.Err)
}
func (e *ArtifactCodecOperationError) Unwrap() error { return e.Err }
type ArtifactCodecRegistry struct {
entries map[contracts.ArtifactKind]artifactCodecEntry
}
type artifactCodecEntry struct {
spec ArtifactCodecSpec
valueType reflect.Type
encode func(any) ([]byte, error)
encodeCandidate func(any) ([]byte, error)
metadata func(any) map[string]any
decode func([]byte) (any, error)
}
func NewArtifactCodecRegistry() *ArtifactCodecRegistry {
return &ArtifactCodecRegistry{entries: make(map[contracts.ArtifactKind]artifactCodecEntry)}
}
// RegisterArtifactCodec registers one codec for T. The concrete type is kept
// private and checked at every erased encode boundary.
func RegisterArtifactCodec[T any](registry *ArtifactCodecRegistry, codec contracts.ArtifactCodec[T]) error {
if registry == nil {
return fmt.Errorf("artifact codec registry must not be nil")
}
if nilInterface(codec) {
return fmt.Errorf("artifact codec must not be nil")
}
spec, err := artifactCodecSpec(codec.Kind(), codec.Schema(), codec.MediaType())
if err != nil {
return err
}
if _, ok := registry.entries[spec.Kind]; ok {
return fmt.Errorf("artifact codec %q is already registered", spec.Kind)
}
valueType := reflect.TypeFor[T]()
entry := artifactCodecEntry{
spec: cloneArtifactCodecSpec(spec),
valueType: valueType,
encode: func(value any) ([]byte, error) {
actualType := reflect.TypeOf(value)
if actualType != valueType {
return nil, newArtifactCodecTypeError("encode", spec.Kind, valueType, actualType)
}
typed, ok := value.(T)
if !ok {
return nil, newArtifactCodecTypeError("encode", spec.Kind, valueType, actualType)
}
encoded, err := codec.Encode(typed)
if err != nil {
return nil, &ArtifactCodecOperationError{Operation: "encode", Kind: spec.Kind, Err: err}
}
return append([]byte(nil), encoded...), nil
},
decode: func(content []byte) (any, error) {
decoded, err := codec.Decode(append([]byte(nil), content...))
if err != nil {
return nil, &ArtifactCodecOperationError{Operation: "decode", Kind: spec.Kind, Err: err}
}
return decoded, nil
},
}
entry.encodeCandidate = entry.encode
if candidate, ok := any(codec).(interface{ EncodeCandidate(T) ([]byte, error) }); ok {
entry.encodeCandidate = func(value any) ([]byte, error) {
typed, err := exactTypedValue[T]("encode candidate artifact", value)
if err != nil {
return nil, err
}
content, err := candidate.EncodeCandidate(typed)
if err != nil {
return nil, &ArtifactCodecOperationError{Operation: "encode", Kind: spec.Kind, Err: err}
}
return append([]byte(nil), content...), nil
}
}
if provider, ok := any(codec).(interface{ Metadata(T) map[string]any }); ok {
entry.metadata = func(value any) map[string]any {
typed, err := exactTypedValue[T]("artifact metadata", value)
if err != nil {
return nil
}
return cloneMetadata(provider.Metadata(typed))
}
}
if registry.entries == nil {
registry.entries = make(map[contracts.ArtifactKind]artifactCodecEntry)
}
registry.entries[spec.Kind] = entry
return nil
}
func (r *ArtifactCodecRegistry) Spec(kind contracts.ArtifactKind) (ArtifactCodecSpec, bool) {
if r == nil {
return ArtifactCodecSpec{}, false
}
entry, ok := r.entries[normalizeArtifactKind(kind)]
if !ok {
return ArtifactCodecSpec{}, false
}
return cloneArtifactCodecSpec(entry.spec), true
}
func (r *ArtifactCodecRegistry) valueType(kind contracts.ArtifactKind) (reflect.Type, bool) {
if r == nil {
return nil, false
}
entry, ok := r.entries[normalizeArtifactKind(kind)]
if !ok {
return nil, false
}
return entry.valueType, true
}
func (r *ArtifactCodecRegistry) RegisteredKinds() []contracts.ArtifactKind {
if r == nil || len(r.entries) == 0 {
return nil
}
kinds := make([]contracts.ArtifactKind, 0, len(r.entries))
for kind := range r.entries {
kinds = append(kinds, kind)
}
sort.Slice(kinds, func(i, j int) bool { return kinds[i] < kinds[j] })
return kinds
}
// Encode serializes an erased framework value after proving its exact
// registered Go type.
func (r *ArtifactCodecRegistry) Encode(kind contracts.ArtifactKind, value any) (contracts.SerializedArtifact, error) {
entry, normalizedKind, err := r.entry(kind)
if err != nil {
return contracts.SerializedArtifact{}, err
}
content, err := entry.encode(value)
if err != nil {
return contracts.SerializedArtifact{}, err
}
return contracts.SerializedArtifact{
Kind: normalizedKind,
Schema: contracts.CloneArtifactSchema(entry.spec.Schema),
MediaType: entry.spec.MediaType,
Content: content,
}, nil
}
// Decode verifies serialized identity before invoking the registered codec.
func (r *ArtifactCodecRegistry) Decode(artifact contracts.SerializedArtifact) (any, error) {
entry, normalizedKind, err := r.entry(artifact.Kind)
if err != nil {
return nil, err
}
if artifact.Schema.ID != entry.spec.Schema.ID || artifact.Schema.Name != entry.spec.Schema.Name || artifact.Schema.Version != entry.spec.Schema.Version {
return nil, &ArtifactCodecCompatibilityError{Kind: normalizedKind, Reason: "schema identity does not match registered codec"}
}
if contracts.DigestArtifactSchema(artifact.Schema) != entry.spec.SchemaDigest {
return nil, &ArtifactCodecCompatibilityError{Kind: normalizedKind, Reason: "schema digest does not match registered codec"}
}
if strings.TrimSpace(artifact.MediaType) != entry.spec.MediaType {
return nil, &ArtifactCodecCompatibilityError{Kind: normalizedKind, Reason: "media type does not match registered codec"}
}
return entry.decode(artifact.Content)
}
func (r *ArtifactCodecRegistry) entry(kind contracts.ArtifactKind) (artifactCodecEntry, contracts.ArtifactKind, error) {
if r == nil {
return artifactCodecEntry{}, "", fmt.Errorf("artifact codec registry must not be nil")
}
normalizedKind := normalizeArtifactKind(kind)
if normalizedKind == "" {
return artifactCodecEntry{}, "", fmt.Errorf("artifact kind must not be empty")
}
entry, ok := r.entries[normalizedKind]
if !ok {
return artifactCodecEntry{}, normalizedKind, fmt.Errorf("artifact codec %q is not registered", normalizedKind)
}
return entry, normalizedKind, nil
}
func artifactCodecSpec(kind contracts.ArtifactKind, schema contracts.ArtifactSchema, mediaType string) (ArtifactCodecSpec, error) {
kind = normalizeArtifactKind(kind)
schema.ID = strings.TrimSpace(schema.ID)
schema.Name = strings.TrimSpace(schema.Name)
schema.Version = strings.TrimSpace(schema.Version)
mediaType = strings.TrimSpace(mediaType)
switch {
case kind == "":
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec kind must not be empty")
case schema.ID == "":
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q schema id must not be empty", kind)
case schema.Name == "":
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q schema name must not be empty", kind)
case schema.Version == "":
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q schema version must not be empty", kind)
case mediaType == "":
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q media type must not be empty", kind)
case len(bytes.TrimSpace(schema.JSONSchema)) == 0:
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema must not be empty", kind)
case !json.Valid(schema.JSONSchema):
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema must be valid JSON", kind)
}
schemaDocument, err := jsonschema.UnmarshalJSON(bytes.NewReader(schema.JSONSchema))
if err != nil {
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema is invalid: %w", kind, err)
}
compiler := jsonschema.NewCompiler()
if err := compiler.AddResource("artifact-schema.json", schemaDocument); err != nil {
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema is invalid: %w", kind, err)
}
if _, err := compiler.Compile("artifact-schema.json"); err != nil {
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema is invalid: %w", kind, err)
}
schema = contracts.CloneArtifactSchema(schema)
return ArtifactCodecSpec{
Kind: kind,
Schema: schema,
SchemaDigest: contracts.DigestArtifactSchema(schema),
MediaType: mediaType,
}, nil
}
func normalizeArtifactKind(kind contracts.ArtifactKind) contracts.ArtifactKind {
return contracts.ArtifactKind(strings.TrimSpace(string(kind)))
}
func cloneArtifactCodecSpec(spec ArtifactCodecSpec) ArtifactCodecSpec {
spec.Schema = contracts.CloneArtifactSchema(spec.Schema)
return spec
}
func nilInterface(value any) bool {
if value == nil {
return true
}
reflected := reflect.ValueOf(value)
switch reflected.Kind() {
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:
return reflected.IsNil()
default:
return false
}
}
func newArtifactCodecTypeError(operation string, kind contracts.ArtifactKind, expected reflect.Type, actual reflect.Type) error {
expectedName := "<nil>"
if expected != nil {
expectedName = expected.String()
}
actualName := "<nil>"
if actual != nil {
actualName = actual.String()
}
return &ArtifactCodecTypeError{
Operation: operation,
Kind: kind,
ExpectedType: expectedName,
ActualType: actualName,
}
}

View File

@@ -0,0 +1,307 @@
package pipeline
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type codecNotes struct {
Items []string `json:"items"`
}
type codecScore struct {
Value int `json:"value"`
}
type codecNotesAlias codecNotes
type testArtifactCodec[T any] struct {
kind contracts.ArtifactKind
schema contracts.ArtifactSchema
mediaType string
encodeFunc func(T) ([]byte, error)
decodeFunc func([]byte) (T, error)
}
func (c testArtifactCodec[T]) Kind() contracts.ArtifactKind { return c.kind }
func (c testArtifactCodec[T]) Schema() contracts.ArtifactSchema { return c.schema }
func (c testArtifactCodec[T]) MediaType() string { return c.mediaType }
func (c testArtifactCodec[T]) Encode(value T) ([]byte, error) { return c.encodeFunc(value) }
func (c testArtifactCodec[T]) Decode(content []byte) (T, error) { return c.decodeFunc(content) }
func TestArtifactCodecRegistryStoresHeterogeneousExactTypes(t *testing.T) {
registry := NewArtifactCodecRegistry()
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
t.Fatalf("RegisterArtifactCodec(notes) error = %v, want nil", err)
}
if err := RegisterArtifactCodec(registry, scoreCodec()); err != nil {
t.Fatalf("RegisterArtifactCodec(score) error = %v, want nil", err)
}
if got, want := registry.RegisteredKinds(), []contracts.ArtifactKind{"test/notes", "test/score"}; !reflect.DeepEqual(got, want) {
t.Fatalf("RegisteredKinds() = %#v, want %#v", got, want)
}
notes := codecNotes{Items: []string{"second", "first"}}
first, err := registry.Encode("test/notes", notes)
if err != nil {
t.Fatalf("Encode(notes) error = %v, want nil", err)
}
second, err := registry.Encode(" test/notes ", notes)
if err != nil {
t.Fatalf("Encode(notes again) error = %v, want nil", err)
}
if !bytes.Equal(first.Content, second.Content) {
t.Fatalf("equal values encoded as %q and %q, want deterministic bytes", first.Content, second.Content)
}
decodedNotes, err := registry.Decode(first)
if err != nil {
t.Fatalf("Decode(notes) error = %v, want nil", err)
}
if !reflect.DeepEqual(decodedNotes, notes) {
t.Fatalf("Decode(notes) = %#v, want %#v", decodedNotes, notes)
}
score := codecScore{Value: 17}
serializedScore, err := registry.Encode("test/score", score)
if err != nil {
t.Fatalf("Encode(score) error = %v, want nil", err)
}
decodedScore, err := registry.Decode(serializedScore)
if err != nil {
t.Fatalf("Decode(score) error = %v, want nil", err)
}
if decodedScore != score {
t.Fatalf("Decode(score) = %#v, want %#v", decodedScore, score)
}
_, err = registry.Encode("test/notes", codecNotesAlias(notes))
var typeErr *ArtifactCodecTypeError
if !errors.As(err, &typeErr) {
t.Fatalf("Encode(alias) error = %T %v, want ArtifactCodecTypeError", err, err)
}
if typeErr.ExpectedType == typeErr.ActualType {
t.Fatalf("type error = %#v, want distinct exact types", typeErr)
}
}
func TestArtifactCodecRegistryStoresValidatedSchemaMetadata(t *testing.T) {
registry := NewArtifactCodecRegistry()
codec := notesCodec()
codec.kind = " test/notes "
codec.schema.ID = " notes.v1 "
codec.schema.Name = " notes "
codec.schema.Version = " v1 "
codec.mediaType = " application/json "
if err := RegisterArtifactCodec(registry, codec); err != nil {
t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
}
codec.schema.JSONSchema[0] = '['
spec, ok := registry.Spec("test/notes")
if !ok {
t.Fatal("Spec() ok = false, want true")
}
if spec.Kind != "test/notes" || spec.Schema.ID != "notes.v1" || spec.Schema.Name != "notes" || spec.Schema.Version != "v1" || spec.MediaType != "application/json" {
t.Fatalf("Spec() = %#v, want normalized metadata", spec)
}
if spec.SchemaDigest != contracts.DigestArtifactSchema(spec.Schema) {
t.Fatalf("schema digest = %q, want %q", spec.SchemaDigest, contracts.DigestArtifactSchema(spec.Schema))
}
spec.Schema.JSONSchema[0] = '['
again, _ := registry.Spec("test/notes")
if string(again.Schema.JSONSchema) != `{"additionalProperties":false,"properties":{"items":{"items":{"type":"string"},"type":"array"}},"required":["items"],"type":"object"}` {
t.Fatalf("stored JSON Schema changed through Spec result: %q", again.Schema.JSONSchema)
}
}
func TestArtifactCodecRegistryRejectsInvalidRegistration(t *testing.T) {
tests := []struct {
name string
mutate func(*testArtifactCodec[codecNotes])
want string
}{
{name: "kind", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.kind = " " }, want: "kind"},
{name: "schema id", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.ID = "" }, want: "schema id"},
{name: "schema name", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.Name = "" }, want: "schema name"},
{name: "schema version", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.Version = "" }, want: "schema version"},
{name: "media type", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.mediaType = "" }, want: "media type"},
{name: "empty JSON Schema", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.JSONSchema = nil }, want: "JSON Schema"},
{name: "invalid JSON Schema", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.JSONSchema = []byte(`{"type":`) }, want: "valid JSON"},
{name: "non-schema JSON", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.JSONSchema = []byte(`[]`) }, want: "JSON Schema is invalid"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
registry := NewArtifactCodecRegistry()
codec := notesCodec()
test.mutate(&codec)
if err := RegisterArtifactCodec(registry, codec); err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("RegisterArtifactCodec() error = %v, want %q", err, test.want)
}
})
}
}
func TestArtifactCodecRegistryRejectsDuplicateKind(t *testing.T) {
registry := NewArtifactCodecRegistry()
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
}
duplicate := notesCodec()
duplicate.kind = " test/notes "
if err := RegisterArtifactCodec(registry, duplicate); err == nil || !strings.Contains(err.Error(), "already registered") {
t.Fatalf("duplicate registration error = %v, want duplicate kind error", err)
}
}
func TestArtifactCodecRegistryRejectsNilRegistryAndCodec(t *testing.T) {
codec := notesCodec()
if err := RegisterArtifactCodec[codecNotes](nil, codec); err == nil || !strings.Contains(err.Error(), "registry") {
t.Fatalf("nil registry error = %v, want registry error", err)
}
var nilCodec *testArtifactCodec[codecNotes]
if err := RegisterArtifactCodec(NewArtifactCodecRegistry(), nilCodec); err == nil || !strings.Contains(err.Error(), "must not be nil") {
t.Fatalf("nil codec error = %v, want codec error", err)
}
}
func TestArtifactCodecRegistryStrictDecodeAndTypedFailures(t *testing.T) {
registry := NewArtifactCodecRegistry()
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
}
valid, err := registry.Encode("test/notes", codecNotes{Items: []string{"one"}})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
for _, content := range [][]byte{
[]byte(`{"items":["one"],"unknown":true}`),
[]byte(`{"items":["one"]} {}`),
} {
candidate := contracts.CloneSerializedArtifact(valid)
candidate.Content = content
_, err := registry.Decode(candidate)
var operationErr *ArtifactCodecOperationError
if !errors.As(err, &operationErr) || operationErr.Operation != "decode" {
t.Fatalf("Decode(%q) error = %T %v, want typed decode error", content, err, err)
}
}
wrongSchema := contracts.CloneSerializedArtifact(valid)
wrongSchema.Schema.Version = "v2"
_, err = registry.Decode(wrongSchema)
var compatibilityErr *ArtifactCodecCompatibilityError
if !errors.As(err, &compatibilityErr) {
t.Fatalf("Decode(wrong schema) error = %T %v, want ArtifactCodecCompatibilityError", err, err)
}
}
func TestArtifactCodecRegistryWrapsEncodeFailure(t *testing.T) {
codec := notesCodec()
cause := errors.New("cannot encode notes")
codec.encodeFunc = func(codecNotes) ([]byte, error) { return nil, cause }
registry := NewArtifactCodecRegistry()
if err := RegisterArtifactCodec(registry, codec); err != nil {
t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
}
_, err := registry.Encode("test/notes", codecNotes{})
var operationErr *ArtifactCodecOperationError
if !errors.As(err, &operationErr) || operationErr.Operation != "encode" || !errors.Is(err, cause) {
t.Fatalf("Encode() error = %T %v, want typed wrapping encode error", err, err)
}
}
func TestArtifactCodecRegistryClonesCodecBytes(t *testing.T) {
shared := []byte(`{"items":["one"]}`)
codec := notesCodec()
codec.encodeFunc = func(codecNotes) ([]byte, error) { return shared, nil }
codec.decodeFunc = func(content []byte) (codecNotes, error) {
content[0] = '['
return codecNotes{Items: []string{"one"}}, nil
}
registry := NewArtifactCodecRegistry()
if err := RegisterArtifactCodec(registry, codec); err != nil {
t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
}
artifact, err := registry.Encode("test/notes", codecNotes{})
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
shared[0] = '['
if string(artifact.Content) != `{"items":["one"]}` {
t.Fatalf("encoded content = %q after codec buffer mutation, want owned bytes", artifact.Content)
}
before := append([]byte(nil), artifact.Content...)
if _, err := registry.Decode(artifact); err != nil {
t.Fatalf("Decode() error = %v, want nil", err)
}
if !bytes.Equal(artifact.Content, before) {
t.Fatalf("serialized content changed during decode: %q", artifact.Content)
}
}
func notesCodec() testArtifactCodec[codecNotes] {
return testArtifactCodec[codecNotes]{
kind: "test/notes",
schema: contracts.ArtifactSchema{
ID: "notes.v1",
Name: "notes",
Version: "v1",
JSONSchema: []byte(`{"additionalProperties":false,"properties":{"items":{"items":{"type":"string"},"type":"array"}},"required":["items"],"type":"object"}`),
},
mediaType: "application/json",
encodeFunc: func(value codecNotes) ([]byte, error) {
return json.Marshal(value)
},
decodeFunc: func(content []byte) (codecNotes, error) {
var value codecNotes
return value, decodeStrictJSON(content, &value)
},
}
}
func scoreCodec() testArtifactCodec[codecScore] {
return testArtifactCodec[codecScore]{
kind: "test/score",
schema: contracts.ArtifactSchema{
ID: "score.v1",
Name: "score",
Version: "v1",
JSONSchema: []byte(`{"additionalProperties":false,"properties":{"value":{"type":"integer"}},"required":["value"],"type":"object"}`),
},
mediaType: "application/json",
encodeFunc: func(value codecScore) ([]byte, error) {
return json.Marshal(value)
},
decodeFunc: func(content []byte) (codecScore, error) {
var value codecScore
return value, decodeStrictJSON(content, &value)
},
}
}
func decodeStrictJSON(content []byte, out any) error {
decoder := json.NewDecoder(bytes.NewReader(content))
decoder.DisallowUnknownFields()
if err := decoder.Decode(out); err != nil {
return err
}
if err := decoder.Decode(&struct{}{}); err != io.EOF {
if err == nil {
return fmt.Errorf("unexpected trailing JSON value")
}
return fmt.Errorf("decode trailing JSON: %w", err)
}
return nil
}

View File

@@ -21,18 +21,18 @@ type CheckpointRecorder interface {
SourceSucceeded(moduleKey string, doc *source.SourceDocument) error
SourceFailed(moduleKey string, err error) error
ChunkRunning(moduleKey string, sourceDigest string) error
ChunkSucceeded(moduleKey string, sourceDigest string, chunks []contracts.SourceChunk, warnings []contracts.Warning) error
ChunkSucceeded(moduleKey string, sourceDigest string, chunks []source.Chunk, warnings []contracts.Warning) error
ChunkRejected(moduleKey string, sourceDigest string, rejected contracts.RejectedOutput) error
ChunkFailed(moduleKey string, sourceDigest string, err error) error
ExtractRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
ExtractSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, outputs []contracts.ExtractOutput, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error
ExtractSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error
ExtractFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
MergeRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
MergeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output contracts.MergeOutput, warnings []contracts.Warning) error
MergeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error
MergeRejected(laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
MergeFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
NormalizeRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
NormalizeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output contracts.NormalizeOutput, warnings []contracts.Warning) error
NormalizeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error
NormalizeRejected(laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
NormalizeFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
}
@@ -55,23 +55,35 @@ type SourceCheckpoint struct {
}
type ChunkCheckpoint struct {
Chunks []contracts.SourceChunk
Chunks []source.Chunk
Warnings []contracts.Warning
}
// CheckpointArtifact is the durable, domain-neutral value stored at a lane
// checkpoint boundary.
type CheckpointArtifact struct {
LaneID string
ModuleKey string
SourceID string
ChunkID string
ChunkIndex int
ChunkRef source.SourceRef
Artifact contracts.SerializedArtifact
SchemaDigest string
}
type ExtractCheckpoint struct {
Outputs []contracts.ExtractOutput
Outputs []CheckpointArtifact
Rejected []contracts.RejectedOutput
Warnings []contracts.Warning
}
type MergeCheckpoint struct {
Output contracts.MergeOutput
Output CheckpointArtifact
Warnings []contracts.Warning
}
type NormalizeCheckpoint struct {
Output contracts.NormalizeOutput
Output CheckpointArtifact
Warnings []contracts.Warning
}
@@ -94,7 +106,7 @@ func (noopCheckpointRecorder) SourceRunning(string) error
func (noopCheckpointRecorder) SourceSucceeded(string, *source.SourceDocument) error { return nil }
func (noopCheckpointRecorder) SourceFailed(string, error) error { return nil }
func (noopCheckpointRecorder) ChunkRunning(string, string) error { return nil }
func (noopCheckpointRecorder) ChunkSucceeded(string, string, []contracts.SourceChunk, []contracts.Warning) error {
func (noopCheckpointRecorder) ChunkSucceeded(string, string, []source.Chunk, []contracts.Warning) error {
return nil
}
func (noopCheckpointRecorder) ChunkRejected(string, string, contracts.RejectedOutput) error {
@@ -104,14 +116,14 @@ func (noopCheckpointRecorder) ChunkFailed(string, string, error) error { return
func (noopCheckpointRecorder) ExtractRunning(string, string, []CheckpointFingerprint) error {
return nil
}
func (noopCheckpointRecorder) ExtractSucceeded(string, string, []CheckpointFingerprint, []contracts.ExtractOutput, []contracts.RejectedOutput, []contracts.Warning) error {
func (noopCheckpointRecorder) ExtractSucceeded(string, string, []CheckpointFingerprint, []CheckpointArtifact, []contracts.RejectedOutput, []contracts.Warning) error {
return nil
}
func (noopCheckpointRecorder) ExtractFailed(string, string, []CheckpointFingerprint, error) error {
return nil
}
func (noopCheckpointRecorder) MergeRunning(string, string, []CheckpointFingerprint) error { return nil }
func (noopCheckpointRecorder) MergeSucceeded(string, string, []CheckpointFingerprint, contracts.MergeOutput, []contracts.Warning) error {
func (noopCheckpointRecorder) MergeSucceeded(string, string, []CheckpointFingerprint, CheckpointArtifact, []contracts.Warning) error {
return nil
}
func (noopCheckpointRecorder) MergeRejected(string, string, []CheckpointFingerprint, contracts.RejectedOutput) error {
@@ -123,7 +135,7 @@ func (noopCheckpointRecorder) MergeFailed(string, string, []CheckpointFingerprin
func (noopCheckpointRecorder) NormalizeRunning(string, string, []CheckpointFingerprint) error {
return nil
}
func (noopCheckpointRecorder) NormalizeSucceeded(string, string, []CheckpointFingerprint, contracts.NormalizeOutput, []contracts.Warning) error {
func (noopCheckpointRecorder) NormalizeSucceeded(string, string, []CheckpointFingerprint, CheckpointArtifact, []contracts.Warning) error {
return nil
}
func (noopCheckpointRecorder) NormalizeRejected(string, string, []CheckpointFingerprint, contracts.RejectedOutput) error {
@@ -150,28 +162,6 @@ func (noopCheckpointLoader) Normalize(string, string, []CheckpointFingerprint) (
return NormalizeCheckpoint{}, CheckpointDecision{Reason: "checkpoint loading disabled"}
}
func rawOutputDigests(payloads []contracts.RawPayload) []CheckpointFingerprint {
values := make([]CheckpointFingerprint, 0, len(payloads))
for i, payload := range payloads {
values = append(values, CheckpointFingerprint{
Name: fmt.Sprintf("payload[%d]", i),
Value: checkpointContentDigest(payload.Content),
})
}
return normalizeCheckpointFingerprints(values)
}
func extractPayloads(outputs []contracts.ExtractOutput) []contracts.RawPayload {
if len(outputs) == 0 {
return nil
}
payloads := make([]contracts.RawPayload, 0, len(outputs))
for _, output := range outputs {
payloads = append(payloads, output.Payload)
}
return payloads
}
func digestFingerprints(name string, digest string) []CheckpointFingerprint {
digest = strings.TrimSpace(digest)
if digest == "" {
@@ -180,17 +170,21 @@ func digestFingerprints(name string, digest string) []CheckpointFingerprint {
return []CheckpointFingerprint{{Name: name, Value: digest}}
}
func joinedChunkDigest(chunks []contracts.SourceChunk) string {
func joinedChunkDigest(chunks []source.Chunk) (string, error) {
if len(chunks) == 0 {
return ""
return "", nil
}
values := make([]string, 0, len(chunks))
for _, chunk := range chunks {
values = append(values, chunk.ID+"="+checkpointContentDigest(chunk.Content))
digest, err := source.DigestChunk(chunk)
if err != nil {
return "", fmt.Errorf("digest chunk %q: %w", chunk.ID, err)
}
values = append(values, chunk.ID+"="+digest)
}
sort.Strings(values)
sum := sha256.Sum256([]byte(strings.Join(values, "\n")))
return "sha256:" + hex.EncodeToString(sum[:])
return "sha256:" + hex.EncodeToString(sum[:]), nil
}
func normalizeCheckpointFingerprints(values []CheckpointFingerprint) []CheckpointFingerprint {

View File

@@ -5,10 +5,9 @@ import (
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []contracts.SourceChunk) ([]contracts.SourceChunk, error) {
func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []source.Chunk) ([]source.Chunk, error) {
if len(chunks) == 0 {
return nil, fmt.Errorf("chunks must not be empty")
}
@@ -20,7 +19,7 @@ func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []con
sourceUnits[unit.ID] = unit
}
canonicalChunks := make([]contracts.SourceChunk, 0, len(chunks))
canonicalChunks := make([]source.Chunk, 0, len(chunks))
seenChunkIDs := make(map[string]struct{}, len(chunks))
for chunkIndex, chunk := range chunks {
if strings.TrimSpace(chunk.ID) == "" {
@@ -37,17 +36,6 @@ func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []con
if chunk.Index != chunkIndex {
return nil, fmt.Errorf("chunk %q index %d does not match returned order %d", chunk.ID, chunk.Index, chunkIndex)
}
startIndex, ok := sourceUnitIndexes[chunk.StartUnitID]
if !ok {
return nil, fmt.Errorf("chunk %q start_unit_id %d was not found in source document %q", chunk.ID, chunk.StartUnitID, doc.ID)
}
endIndex, ok := sourceUnitIndexes[chunk.EndUnitID]
if !ok {
return nil, fmt.Errorf("chunk %q end_unit_id %d was not found in source document %q", chunk.ID, chunk.EndUnitID, doc.ID)
}
if startIndex > endIndex {
return nil, fmt.Errorf("chunk %q start_unit_id %d appears after end_unit_id %d", chunk.ID, chunk.StartUnitID, chunk.EndUnitID)
}
if len(chunk.Units) == 0 {
return nil, fmt.Errorf("chunk %q units must not be empty", chunk.ID)
}
@@ -57,6 +45,9 @@ func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []con
if strings.TrimSpace(chunk.MediaType) == "" {
return nil, fmt.Errorf("chunk %q media_type must not be empty", chunk.ID)
}
if err := source.ValidateRef(doc, chunk.Ref); err != nil {
return nil, fmt.Errorf("chunk %q ref: %w", chunk.ID, err)
}
seenUnitIDs := make(map[int]struct{}, len(chunk.Units))
previousSourceIndex := -1
@@ -74,23 +65,33 @@ func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []con
if !ok {
return nil, fmt.Errorf("chunk %q source unit %d was not found in source document %q", chunk.ID, unit.ID, doc.ID)
}
if sourceIndex <= previousSourceIndex {
return nil, fmt.Errorf("chunk %q source units must appear in source document order", chunk.ID)
if previousSourceIndex >= 0 && sourceIndex != previousSourceIndex+1 {
return nil, fmt.Errorf("chunk %q source units must form a contiguous range in source document order", chunk.ID)
}
if unit.Ref != sourceUnits[unit.ID].Ref {
return nil, fmt.Errorf("chunk %q source unit %d ref does not match source document", chunk.ID, unit.ID)
}
previousSourceIndex = sourceIndex
canonicalUnits = append(canonicalUnits, cloneSourceUnit(sourceUnits[unit.ID]))
}
expectedRef := source.SourceRef{
SourceID: doc.ID,
StartUnitID: canonicalUnits[0].Ref.StartUnitID,
EndUnitID: canonicalUnits[len(canonicalUnits)-1].Ref.EndUnitID,
}
if chunk.Ref != expectedRef {
return nil, fmt.Errorf("chunk %q ref %#v does not match unit span %#v", chunk.ID, chunk.Ref, expectedRef)
}
canonicalChunks = append(canonicalChunks, contracts.SourceChunk{
ID: chunk.ID,
SourceID: chunk.SourceID,
Index: chunk.Index,
StartUnitID: chunk.StartUnitID,
EndUnitID: chunk.EndUnitID,
Content: append([]byte(nil), chunk.Content...),
MediaType: chunk.MediaType,
Units: canonicalUnits,
Metadata: cloneMetadata(chunk.Metadata),
canonicalChunks = append(canonicalChunks, source.Chunk{
ID: chunk.ID,
SourceID: chunk.SourceID,
Index: chunk.Index,
Ref: expectedRef,
Content: append([]byte(nil), chunk.Content...),
MediaType: chunk.MediaType,
Units: canonicalUnits,
Metadata: cloneMetadata(chunk.Metadata),
})
}
@@ -102,6 +103,7 @@ func cloneSourceUnit(unit source.SourceUnit) source.SourceUnit {
ID: unit.ID,
Kind: unit.Kind,
Text: unit.Text,
Ref: unit.Ref,
Metadata: cloneMetadata(unit.Metadata),
}
}

View File

@@ -8,16 +8,19 @@ import (
)
type ChunkerConstructor func() (contracts.Chunker, error)
type ChunkerBuilder func(BuildRequest) (contracts.Chunker, error)
type ChunkerRegistry struct {
constructors map[string]ChunkerConstructor
specs map[string]ModuleSpec
builders map[string]ChunkerBuilder
optionValidators map[string]OptionValidator
specs map[string]ModuleSpec
}
func NewChunkerRegistry() *ChunkerRegistry {
return &ChunkerRegistry{
constructors: make(map[string]ChunkerConstructor),
specs: make(map[string]ModuleSpec),
builders: make(map[string]ChunkerBuilder),
optionValidators: make(map[string]OptionValidator),
specs: make(map[string]ModuleSpec),
}
}
@@ -26,6 +29,15 @@ func (r *ChunkerRegistry) Register(key string, constructor ChunkerConstructor) e
}
func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerConstructor) error {
if constructor == nil {
return fmt.Errorf("chunker constructor for %q must not be nil", strings.TrimSpace(spec.Key))
}
return r.RegisterBuilderWithSpec(spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.Chunker, error) {
return constructor()
})
}
func (r *ChunkerRegistry) RegisterBuilderWithSpec(spec ModuleSpec, validateOptions OptionValidator, builder ChunkerBuilder) error {
if r == nil {
return fmt.Errorf("chunker registry must not be nil")
}
@@ -34,25 +46,36 @@ func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerC
if err := validateModuleSpec("chunker", StageChunk, normalizedSpec); err != nil {
return err
}
if constructor == nil {
return fmt.Errorf("chunker constructor for %q must not be nil", normalizedSpec.Key)
if validateOptions == nil {
return fmt.Errorf("chunker option validator for %q must not be nil", normalizedSpec.Key)
}
if _, ok := r.constructors[normalizedSpec.Key]; ok {
if builder == nil {
return fmt.Errorf("chunker builder for %q must not be nil", normalizedSpec.Key)
}
if _, ok := r.builders[normalizedSpec.Key]; ok {
return fmt.Errorf("chunker %q is already registered", normalizedSpec.Key)
}
if r.constructors == nil {
r.constructors = make(map[string]ChunkerConstructor)
if r.builders == nil {
r.builders = make(map[string]ChunkerBuilder)
}
if r.optionValidators == nil {
r.optionValidators = make(map[string]OptionValidator)
}
if r.specs == nil {
r.specs = make(map[string]ModuleSpec)
}
r.constructors[normalizedSpec.Key] = constructor
r.builders[normalizedSpec.Key] = builder
r.optionValidators[normalizedSpec.Key] = validateOptions
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
return nil
}
func (r *ChunkerRegistry) Build(key string) (contracts.Chunker, error) {
return r.BuildWithRequest(key, BuildRequest{})
}
func (r *ChunkerRegistry) BuildWithRequest(key string, request BuildRequest) (contracts.Chunker, error) {
if r == nil {
return nil, fmt.Errorf("chunker registry must not be nil")
}
@@ -62,12 +85,12 @@ func (r *ChunkerRegistry) Build(key string) (contracts.Chunker, error) {
return nil, fmt.Errorf("chunker key must not be empty")
}
constructor, ok := r.constructors[normalizedKey]
builder, ok := r.builders[normalizedKey]
if !ok {
return nil, fmt.Errorf("chunker %q is not registered", normalizedKey)
}
chunker, err := constructor()
chunker, err := builder(cloneBuildRequest(request))
if err != nil {
return nil, fmt.Errorf("build chunker %q: %w", normalizedKey, err)
}
@@ -81,6 +104,18 @@ func (r *ChunkerRegistry) Build(key string) (contracts.Chunker, error) {
return chunker, nil
}
func (r *ChunkerRegistry) ValidateOptions(key string, options map[string]any) error {
if r == nil {
return fmt.Errorf("chunker registry must not be nil")
}
normalizedKey := strings.TrimSpace(key)
validator, ok := r.optionValidators[normalizedKey]
if !ok {
return fmt.Errorf("chunker %q is not registered", normalizedKey)
}
return validateRegisteredOptions(validator, options)
}
func (r *ChunkerRegistry) Spec(key string) (ModuleSpec, bool) {
if r == nil {
return ModuleSpec{}, false
@@ -98,5 +133,5 @@ func (r *ChunkerRegistry) RegisteredKeys() []string {
return nil
}
return sortedRegistryKeys(r.constructors)
return sortedRegistryKeys(r.builders)
}

View File

@@ -338,34 +338,6 @@ func (chunker registryChunker) Chunk(ctx context.Context, req contracts.ChunkReq
return contracts.ChunkResult{}, nil
}
type registryMerger struct {
key string
}
func (merger registryMerger) Key() string {
return merger.key
}
func (merger registryMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
return contracts.MergeResult{}, nil
}
type registryNormalizer struct {
key string
}
func (normalizer registryNormalizer) Key() string {
return normalizer.key
}
func (normalizer registryNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (normalizer registryNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
return contracts.NormalizeResult{}, nil
}
type registryOutputEncoder struct {
key string
}
@@ -377,19 +349,3 @@ func (encoder registryOutputEncoder) Key() string {
func (encoder registryOutputEncoder) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
return contracts.OutputResult{}, nil
}
type registryValidator struct {
name string
}
func (validator registryValidator) Name() string {
return validator.name
}
func (validator registryValidator) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassDeterministic
}
func (validator registryValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: true}, nil
}

View File

@@ -0,0 +1,69 @@
package pipeline
import (
"fmt"
"sort"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
// ModuleDependencies contains run-scoped collaborators shared by constructed
// modules. Implementations retain only the dependencies they use.
type ModuleDependencies struct {
LLM contracts.StructuredLLMClient
}
// BuildRequest contains the stable dependencies and configured options used to
// construct one module or validator for a run.
type BuildRequest struct {
Dependencies ModuleDependencies
Options map[string]any
}
// OptionValidator validates one module binding without constructing it.
type OptionValidator func(map[string]any) error
func rejectUnconfiguredOptions(options map[string]any) error {
return RejectUnknownOptions(options)
}
func validateRegisteredOptions(validator OptionValidator, options map[string]any) error {
if validator == nil {
return fmt.Errorf("option validator must not be nil")
}
return validator(cloneOptions(options))
}
func cloneBuildRequest(request BuildRequest) BuildRequest {
return BuildRequest{
Dependencies: request.Dependencies,
Options: cloneOptions(request.Options),
}
}
// RejectUnknownOptions provides the common strict-map check used by module-
// owned option decoders. Values remain the implementation's responsibility.
func RejectUnknownOptions(options map[string]any, allowed ...string) error {
known := make(map[string]struct{}, len(allowed))
for _, key := range allowed {
key = strings.TrimSpace(key)
if key != "" {
known[key] = struct{}{}
}
}
unknown := make([]string, 0)
for key := range options {
if _, ok := known[key]; !ok {
unknown = append(unknown, key)
}
}
if len(unknown) == 0 {
return nil
}
sort.Strings(unknown)
if len(unknown) == 1 {
return fmt.Errorf("unknown option %q", unknown[0])
}
return fmt.Errorf("unknown options %q", unknown)
}

View File

@@ -82,10 +82,6 @@ type debugBinaryEnvelope struct {
Warnings []contracts.Warning `json:"warnings,omitempty"`
}
type debugRawPayload struct {
Content debugBinaryEnvelope `json:"content"`
}
type debugSourceInput struct {
SourceID string `json:"source_id,omitempty"`
Path string `json:"path,omitempty"`
@@ -104,40 +100,23 @@ type debugSourceDocument struct {
}
type debugSourceChunk struct {
ID string `json:"id"`
SourceID string `json:"source_id"`
Index int `json:"index"`
StartUnitID int `json:"start_unit_id"`
EndUnitID int `json:"end_unit_id"`
Content debugBinaryEnvelope `json:"content"`
Units []source.SourceUnit `json:"units,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
ID string `json:"id"`
SourceID string `json:"source_id"`
Index int `json:"index"`
Ref source.SourceRef `json:"ref"`
Content debugBinaryEnvelope `json:"content"`
Units []source.SourceUnit `json:"units,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
}
type debugExtractOutput struct {
LaneID string `json:"lane_id"`
ExtractorKey string `json:"extractor_key"`
SourceID string `json:"source_id"`
ChunkID string `json:"chunk_id"`
ChunkIndex int `json:"chunk_index"`
Schema contracts.ResponseSchema `json:"schema,omitempty"`
Payload debugBinaryEnvelope `json:"payload"`
}
type debugMergeOutput struct {
LaneID string `json:"lane_id"`
MergerKey string `json:"merger_key"`
SourceID string `json:"source_id,omitempty"`
Schema contracts.ResponseSchema `json:"schema,omitempty"`
Payload debugBinaryEnvelope `json:"payload"`
}
type debugNormalizeOutput struct {
type debugSerializedOutput struct {
LaneID string `json:"lane_id"`
NormalizerKey string `json:"normalizer_key"`
SourceID string `json:"source_id,omitempty"`
Schema contracts.ResponseSchema `json:"schema,omitempty"`
Payload debugBinaryEnvelope `json:"payload"`
Kind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
Schema contracts.ArtifactSchema `json:"schema"`
SchemaDigest string `json:"schema_digest"`
Content debugBinaryEnvelope `json:"content"`
}
type debugLLMInputMaterial struct {
@@ -193,28 +172,9 @@ type debugLLMCallReference struct {
Error bool `json:"error,omitempty"`
}
type debugValidationRequest struct {
Stage string `json:"stage"`
LaneID string `json:"lane_id,omitempty"`
ModuleKey string `json:"module_key"`
SourceID string `json:"source_id,omitempty"`
SessionID string `json:"session_id,omitempty"`
LLMProfile string `json:"llm_profile,omitempty"`
Options map[string]any `json:"options,omitempty"`
Metadata map[string]any `json:"metadata,omitempty"`
Schema contracts.ResponseSchema `json:"schema,omitempty"`
Payload *debugBinaryEnvelope `json:"payload,omitempty"`
ChunkID string `json:"chunk_id,omitempty"`
ChunkIndex int `json:"chunk_index,omitempty"`
Chunk *debugSourceChunk `json:"chunk,omitempty"`
Chunks []debugSourceChunk `json:"chunks,omitempty"`
ExtractOutputs []debugExtractOutput `json:"extract_outputs,omitempty"`
MergeOutput *debugMergeOutput `json:"merge_output,omitempty"`
}
type debugValidationCall struct {
ValidatorName string `json:"validator_name"`
Request debugValidationRequest `json:"request"`
Request any `json:"request"`
Result contracts.ValidationResult `json:"result,omitempty"`
Error string `json:"error,omitempty"`
}
@@ -239,7 +199,16 @@ func wrapDebugLLMClient(client contracts.StructuredLLMClient, recorder DebugReco
if client == nil || recorder == nil || !recorder.Enabled() {
return client
}
return &debugLLMClient{inner: client, recorder: recorder}
if _, ok := client.(*debugLLMClient); ok {
return client
}
return &debugLLMClient{inner: client, recorder: synchronizedDebugRecorder(recorder)}
}
// WithDebugLLMRecording decorates a shared LLM client so calls made by
// construction-injected modules participate in the run's debug recording.
func WithDebugLLMRecording(client contracts.StructuredLLMClient, recorder DebugRecorder) contracts.StructuredLLMClient {
return wrapDebugLLMClient(client, recorder)
}
func (client *debugLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
@@ -430,10 +399,6 @@ func debugContentEnvelope(content []byte, mediaType string, metadata map[string]
}
}
func debugPayloadEnvelope(payload contracts.RawPayload) debugBinaryEnvelope {
return debugContentEnvelope(payload.Content, payload.MediaType, payload.Metadata, payload.Warnings)
}
func debugSourceDocumentEnvelope(doc *source.SourceDocument) *debugSourceDocument {
if doc == nil {
return nil
@@ -448,20 +413,19 @@ func debugSourceDocumentEnvelope(doc *source.SourceDocument) *debugSourceDocumen
}
}
func debugSourceChunkEnvelope(chunk contracts.SourceChunk) debugSourceChunk {
func debugSourceChunkEnvelope(chunk source.Chunk) debugSourceChunk {
return debugSourceChunk{
ID: chunk.ID,
SourceID: chunk.SourceID,
Index: chunk.Index,
StartUnitID: chunk.StartUnitID,
EndUnitID: chunk.EndUnitID,
Content: debugContentEnvelope(chunk.Content, chunk.MediaType, chunk.Metadata, nil),
Units: cloneSourceUnits(chunk.Units),
Metadata: redactSensitiveMap(chunk.Metadata),
ID: chunk.ID,
SourceID: chunk.SourceID,
Index: chunk.Index,
Ref: chunk.Ref,
Content: debugContentEnvelope(chunk.Content, chunk.MediaType, chunk.Metadata, nil),
Units: cloneSourceUnits(chunk.Units),
Metadata: redactSensitiveMap(chunk.Metadata),
}
}
func debugSourceChunkEnvelopes(chunks []contracts.SourceChunk) []debugSourceChunk {
func debugSourceChunkEnvelopes(chunks []source.Chunk) []debugSourceChunk {
if len(chunks) == 0 {
return nil
}
@@ -472,59 +436,26 @@ func debugSourceChunkEnvelopes(chunks []contracts.SourceChunk) []debugSourceChun
return out
}
func debugExtractOutputEnvelope(output contracts.ExtractOutput) debugExtractOutput {
output.Schema.JSONSchema = nil
return debugExtractOutput{
LaneID: output.LaneID,
ExtractorKey: output.ExtractorKey,
SourceID: output.SourceID,
ChunkID: output.ChunkID,
ChunkIndex: output.ChunkIndex,
Schema: output.Schema,
Payload: debugPayloadEnvelope(output.Payload),
func debugSerializedOutputEnvelope(output contracts.SerializedOutput) debugSerializedOutput {
schema := contracts.CloneArtifactSchema(output.Artifact.Schema)
digest := contracts.DigestArtifactSchema(schema)
schema.JSONSchema = nil
content := debugContentEnvelope(output.Artifact.Content, output.Artifact.MediaType, output.Artifact.Metadata, nil)
content.ContentDigest = debugContentDigest(output.Artifact.Content)
return debugSerializedOutput{
LaneID: output.LaneID, NormalizerKey: output.NormalizerKey, SourceID: output.SourceID,
Kind: output.Artifact.Kind, Schema: schema, SchemaDigest: digest,
Content: content,
}
}
func debugExtractOutputEnvelopes(outputs []contracts.ExtractOutput) []debugExtractOutput {
func debugSerializedOutputEnvelopes(outputs []contracts.SerializedOutput) []debugSerializedOutput {
if len(outputs) == 0 {
return nil
}
out := make([]debugExtractOutput, 0, len(outputs))
out := make([]debugSerializedOutput, 0, len(outputs))
for _, output := range outputs {
out = append(out, debugExtractOutputEnvelope(output))
}
return out
}
func debugMergeOutputEnvelope(output contracts.MergeOutput) debugMergeOutput {
output.Schema.JSONSchema = nil
return debugMergeOutput{
LaneID: output.LaneID,
MergerKey: output.MergerKey,
SourceID: output.SourceID,
Schema: output.Schema,
Payload: debugPayloadEnvelope(output.Payload),
}
}
func debugNormalizeOutputEnvelope(output contracts.NormalizeOutput) debugNormalizeOutput {
output.Schema.JSONSchema = nil
return debugNormalizeOutput{
LaneID: output.LaneID,
NormalizerKey: output.NormalizerKey,
SourceID: output.SourceID,
Schema: output.Schema,
Payload: debugPayloadEnvelope(output.Payload),
}
}
func debugNormalizeOutputEnvelopes(outputs []contracts.NormalizeOutput) []debugNormalizeOutput {
if len(outputs) == 0 {
return nil
}
out := make([]debugNormalizeOutput, 0, len(outputs))
for _, output := range outputs {
out = append(out, debugNormalizeOutputEnvelope(output))
out = append(out, debugSerializedOutputEnvelope(output))
}
return out
}
@@ -673,36 +604,6 @@ func debugResponseModel(response contracts.StructuredCompletionResponse) string
return response.Debug.Response.ModelName
}
func debugValidationRequestEnvelope(req contracts.ValidationRequest) debugValidationRequest {
req.Schema.JSONSchema = nil
out := debugValidationRequest{
Stage: req.Stage,
LaneID: req.LaneID,
ModuleKey: req.ModuleKey,
SourceID: req.SourceID,
SessionID: req.SessionID,
LLMProfile: req.LLMProfile,
Options: redactSensitiveMap(req.Options),
Metadata: redactSensitiveMap(req.Metadata),
Schema: req.Schema,
ChunkID: req.ChunkID,
ChunkIndex: req.ChunkIndex,
}
payload := debugPayloadEnvelope(req.Payload)
out.Payload = &payload
if req.Chunk != nil {
chunk := debugSourceChunkEnvelope(*req.Chunk)
out.Chunk = &chunk
}
out.Chunks = debugSourceChunkEnvelopes(req.Chunks)
out.ExtractOutputs = debugExtractOutputEnvelopes(req.ExtractOutputs)
if len(req.MergeOutput.Payload.Content) > 0 || req.MergeOutput.LaneID != "" {
merge := debugMergeOutputEnvelope(req.MergeOutput)
out.MergeOutput = &merge
}
return out
}
func debugValidationResultEnvelope(result contracts.ValidationResult) contracts.ValidationResult {
result.Message = string(redactSecretBytes([]byte(result.Message)))
result.DiagnosticArtifactPath = string(redactSecretBytes([]byte(result.DiagnosticArtifactPath)))
@@ -712,6 +613,14 @@ func debugValidationResultEnvelope(result contracts.ValidationResult) contracts.
return result
}
func debugWarningEnvelopes(warnings []contracts.Warning) []contracts.Warning {
out := cloneWarnings(warnings)
for i := range out {
out[i].Message = string(redactSecretBytes([]byte(out[i].Message)))
}
return out
}
func debugRejectedOutputEnvelope(rejected contracts.RejectedOutput) contracts.RejectedOutput {
rejected.Message = string(redactSecretBytes([]byte(rejected.Message)))
rejected.DiagnosticArtifactPath = string(redactSecretBytes([]byte(rejected.DiagnosticArtifactPath)))

View File

@@ -0,0 +1,51 @@
package pipeline
import (
"encoding/json"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
)
func TestDebugSourceDocumentPreservesUnitReferences(t *testing.T) {
doc := validSourceDocument()
envelope := debugSourceDocumentEnvelope(doc)
encoded, err := json.Marshal(envelope)
if err != nil {
t.Fatalf("Marshal(debug source document) error = %v, want nil", err)
}
var decoded debugSourceDocument
if err := json.Unmarshal(encoded, &decoded); err != nil {
t.Fatalf("Unmarshal(debug source document) error = %v, want nil", err)
}
if got, want := decoded.Units[0].Ref, (source.SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1}); got != want {
t.Fatalf("debug source unit ref = %#v, want %#v", got, want)
}
envelope.Units[0].Ref.SourceID = "mutated"
if got := doc.Units[0].Ref.SourceID; got != "source-1" {
t.Fatalf("source document ref = %q after debug mutation, want source-1", got)
}
}
func TestDebugSourceChunkPreservesReference(t *testing.T) {
doc := validSourceDocument()
chunk := source.Chunk{
ID: "chunk-1", SourceID: doc.ID, Index: 0,
Ref: source.SourceRef{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 1},
Content: []byte("chunk content"), MediaType: "text/plain", Units: doc.Units[:1],
}
envelope := debugSourceChunkEnvelope(chunk)
encoded, err := json.Marshal(envelope)
if err != nil {
t.Fatalf("Marshal(debug source chunk) error = %v, want nil", err)
}
var decoded debugSourceChunk
if err := json.Unmarshal(encoded, &decoded); err != nil {
t.Fatalf("Unmarshal(debug source chunk) error = %v, want nil", err)
}
if decoded.Ref != chunk.Ref {
t.Fatalf("debug chunk ref = %#v, want %#v", decoded.Ref, chunk.Ref)
}
}

View File

@@ -2,16 +2,17 @@ package pipeline_test
import (
"context"
"encoding/json"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/chunk/units"
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/merge/appendorder"
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/output/json"
)
func TestPipelineConfigResolvesWithProductionDefaultsRegistered(t *testing.T) {
@@ -34,8 +35,8 @@ func TestPipelineConfigResolvesWithProductionDefaultsRegistered(t *testing.T) {
}
pipeline := resolved.ResolvedPipeline
if pipeline.Chunk.Module != generic.Key {
t.Fatalf("Chunk.Module = %q, want %q", pipeline.Chunk.Module, generic.Key)
if pipeline.Chunk.Module != units.Key {
t.Fatalf("Chunk.Module = %q, want %q", pipeline.Chunk.Module, units.Key)
}
if pipeline.Output.Module != jsonoutput.Key {
t.Fatalf("Output.Module = %q, want %q", pipeline.Output.Module, jsonoutput.Key)
@@ -68,32 +69,43 @@ func defaultModuleCatalog(t *testing.T) pipeline.ModuleCatalog {
}); err != nil {
t.Fatalf("register input: %v", err)
}
if err := generic.Register(chunkers); err != nil {
if err := units.Register(chunkers); err != nil {
t.Fatalf("register generic chunker: %v", err)
}
if err := extractors.RegisterWithSpec(pipeline.ModuleSpec{
Key: "extract",
Stage: pipeline.StageExtract,
Requires: []string{"chunks"},
Provides: []string{"records"},
}, func() (contracts.Extractor, error) {
if err := pipeline.RegisterExtractor[defaultArtifact](extractors, pipeline.ModuleSpec{
Key: "extract",
Stage: pipeline.StageExtract,
ArtifactKind: defaultArtifactKind,
Requires: []string{"chunks"},
Provides: []string{"records"},
}, func() (contracts.Extractor[defaultArtifact], error) {
return defaultExtractor{}, nil
}); err != nil {
t.Fatalf("register extractor: %v", err)
}
if err := appendorder.Register(mergers); err != nil {
if err := appendorder.RegisterTyped(mergers, defaultArtifactKind, func(values []defaultArtifact) (defaultArtifact, error) {
if len(values) == 0 {
return defaultArtifact{}, nil
}
return values[0], nil
}); err != nil {
t.Fatalf("register appendorder merger: %v", err)
}
if err := noop.Register(normalizers); err != nil {
if err := noop.RegisterTyped[defaultArtifact](normalizers, defaultArtifactKind); err != nil {
t.Fatalf("register noop normalizer: %v", err)
}
if err := jsonoutput.Register(outputs); err != nil {
t.Fatalf("register json output: %v", err)
}
codecs := pipeline.NewArtifactCodecRegistry()
if err := pipeline.RegisterArtifactCodec(codecs, defaultArtifactCodec{}); err != nil {
t.Fatalf("register artifact codec: %v", err)
}
return pipeline.ModuleCatalog{
Inputs: inputs,
Chunkers: chunkers,
ArtifactCodecs: codecs,
Extractors: extractors,
Mergers: mergers,
Normalizers: normalizers,
@@ -116,6 +128,25 @@ func (defaultExtractor) Key() string { return "extract" }
func (defaultExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (defaultExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
return contracts.ExtractionResult{}, nil
func (defaultExtractor) Extract(ctx context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[defaultArtifact], error) {
return contracts.TypedExtractionResult[defaultArtifact]{}, nil
}
const defaultArtifactKind contracts.ArtifactKind = "test/default"
type defaultArtifact struct {
Value string `json:"value"`
}
type defaultArtifactCodec struct{}
func (defaultArtifactCodec) Kind() contracts.ArtifactKind { return defaultArtifactKind }
func (defaultArtifactCodec) Schema() contracts.ArtifactSchema {
return contracts.ArtifactSchema{ID: "urn:notarius:test:default", Name: "default", Version: "1", JSONSchema: []byte(`{"type":"object"}`)}
}
func (defaultArtifactCodec) MediaType() string { return "application/json" }
func (defaultArtifactCodec) Encode(value defaultArtifact) ([]byte, error) { return json.Marshal(value) }
func (defaultArtifactCodec) Decode(content []byte) (defaultArtifact, error) {
var value defaultArtifact
err := json.Unmarshal(content, &value)
return value, err
}

View File

@@ -1,102 +1,108 @@
package pipeline
import (
"context"
"fmt"
"reflect"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type ExtractorConstructor func() (contracts.Extractor, error)
type ExtractorRegistry struct {
constructors map[string]ExtractorConstructor
typedEntries map[string]typedExtractorEntry
specs map[string]ModuleSpec
}
type typedExtractorEntry struct {
spec ModuleSpec
valueType reflect.Type
validateOptions OptionValidator
builder func(BuildRequest) (any, error)
extract typedExtractOperation
}
func NewExtractorRegistry() *ExtractorRegistry {
return &ExtractorRegistry{
constructors: make(map[string]ExtractorConstructor),
specs: make(map[string]ModuleSpec),
return &ExtractorRegistry{typedEntries: map[string]typedExtractorEntry{}, specs: map[string]ModuleSpec{}}
}
func RegisterExtractor[T any](registry *ExtractorRegistry, spec ModuleSpec, constructor func() (contracts.Extractor[T], error)) error {
if constructor == nil {
return fmt.Errorf("extractor constructor for %q must not be nil", strings.TrimSpace(spec.Key))
}
return RegisterExtractorBuilder(registry, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.Extractor[T], error) { return constructor() })
}
func (r *ExtractorRegistry) Register(key string, constructor ExtractorConstructor) error {
return r.RegisterWithSpec(defaultModuleSpec(key, StageExtract), constructor)
}
func (r *ExtractorRegistry) RegisterWithSpec(spec ModuleSpec, constructor ExtractorConstructor) error {
if r == nil {
func RegisterExtractorBuilder[T any](registry *ExtractorRegistry, spec ModuleSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.Extractor[T], error)) error {
if registry == nil {
return fmt.Errorf("extractor registry must not be nil")
}
normalizedSpec := normalizeModuleSpec(spec)
if err := validateModuleSpec("extractor", StageExtract, normalizedSpec); err != nil {
normalized := normalizeModuleSpec(spec)
if err := validateModuleSpec("extractor", StageExtract, normalized); err != nil {
return err
}
if constructor == nil {
return fmt.Errorf("extractor constructor for %q must not be nil", normalizedSpec.Key)
if normalized.ArtifactKind == "" {
return fmt.Errorf("typed extractor %q artifact kind must not be empty", normalized.Key)
}
if _, ok := r.constructors[normalizedSpec.Key]; ok {
return fmt.Errorf("extractor %q is already registered", normalizedSpec.Key)
if validateOptions == nil {
return fmt.Errorf("extractor option validator for %q must not be nil", normalized.Key)
}
if r.constructors == nil {
r.constructors = make(map[string]ExtractorConstructor)
if builder == nil {
return fmt.Errorf("extractor builder for %q must not be nil", normalized.Key)
}
if r.specs == nil {
r.specs = make(map[string]ModuleSpec)
if _, ok := registry.specs[normalized.Key]; ok {
return fmt.Errorf("extractor %q is already registered", normalized.Key)
}
r.constructors[normalizedSpec.Key] = constructor
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
entry := typedExtractorEntry{spec: cloneModuleSpec(normalized), valueType: reflect.TypeFor[T](), validateOptions: validateOptions, builder: func(request BuildRequest) (any, error) { return builder(cloneBuildRequest(request)) }, extract: func(ctx context.Context, implementation any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
extractor, ok := implementation.(contracts.Extractor[T])
if !ok {
return erasedTypedResult{}, fmt.Errorf("extractor %q has incompatible implementation %T", normalized.Key, implementation)
}
result, err := extractor.Extract(ctx, request)
if err != nil {
return erasedTypedResult{}, err
}
return erasedTypedResult{Value: result.Value, Warnings: result.Warnings}, nil
}}
if registry.typedEntries == nil {
registry.typedEntries = map[string]typedExtractorEntry{}
}
if registry.specs == nil {
registry.specs = map[string]ModuleSpec{}
}
registry.typedEntries[normalized.Key] = entry
registry.specs[normalized.Key] = cloneModuleSpec(normalized)
return nil
}
func (r *ExtractorRegistry) Build(key string) (contracts.Extractor, error) {
func (r *ExtractorRegistry) validateOptions(key string, options map[string]any) error {
if r == nil {
return nil, fmt.Errorf("extractor registry must not be nil")
return fmt.Errorf("extractor registry must not be nil")
}
normalizedKey := strings.TrimSpace(key)
if normalizedKey == "" {
return nil, fmt.Errorf("extractor key must not be empty")
}
constructor, ok := r.constructors[normalizedKey]
normalized := strings.TrimSpace(key)
entry, ok := r.typedEntries[normalized]
if !ok {
return nil, fmt.Errorf("extractor %q is not registered", normalizedKey)
return fmt.Errorf("extractor %q is not registered", normalized)
}
extractor, err := constructor()
if err != nil {
return nil, fmt.Errorf("build extractor %q: %w", normalizedKey, err)
}
if extractor == nil {
return nil, fmt.Errorf("extractor %q constructor returned nil", normalizedKey)
}
if extractor.Key() != normalizedKey {
return nil, fmt.Errorf("extractor %q returned key %q", normalizedKey, extractor.Key())
}
return extractor, nil
return validateRegisteredOptions(entry.validateOptions, options)
}
func (r *ExtractorRegistry) Spec(key string) (ModuleSpec, bool) {
if r == nil {
return ModuleSpec{}, false
}
spec, ok := r.specs[strings.TrimSpace(key)]
if !ok {
return ModuleSpec{}, false
}
return cloneModuleSpec(spec), true
return cloneModuleSpec(spec), ok
}
func (r *ExtractorRegistry) typedEntry(key string) (typedExtractorEntry, bool) {
if r == nil {
return typedExtractorEntry{}, false
}
entry, ok := r.typedEntries[strings.TrimSpace(key)]
return entry, ok
}
func (r *ExtractorRegistry) RegisteredKeys() []string {
if r == nil {
return nil
}
return sortedRegistryKeys(r.constructors)
return sortedRegistryKeys(r.specs)
}

View File

@@ -1,376 +0,0 @@
package pipeline
import (
"context"
"errors"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestExtractorRegistryRegisterAndBuild(t *testing.T) {
registry := NewExtractorRegistry()
if err := registry.Register("generic-extractor", fakeExtractorConstructor("generic-extractor")); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
extractor, err := registry.Build("generic-extractor")
if err != nil {
t.Fatalf("Build() error = %v, want nil", err)
}
if extractor.Key() != "generic-extractor" {
t.Fatalf("extractor.Key() = %q, want generic-extractor", extractor.Key())
}
}
func TestExtractorRegistryRegisterAndBuildTrimKeys(t *testing.T) {
registry := NewExtractorRegistry()
if err := registry.Register(" generic-extractor ", fakeExtractorConstructor("generic-extractor")); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
extractor, err := registry.Build("\tgeneric-extractor\n")
if err != nil {
t.Fatalf("Build() error = %v, want nil", err)
}
if extractor.Key() != "generic-extractor" {
t.Fatalf("extractor.Key() = %q, want generic-extractor", extractor.Key())
}
}
func TestExtractorRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
registry := NewExtractorRegistry()
spec := ModuleSpec{
Key: " generic-extractor ",
Stage: StageExtract,
Provides: []string{" generic-artifact ", "source-citations", "generic-artifact", ""},
Requires: []string{" source-document ", "source-document", ""},
ReferenceSlots: []contracts.ReferenceSlot{
{
Name: " glossary ",
Description: " Supporting terms ",
AcceptedMediaTypes: []string{" text/plain ", "text/markdown", "text/plain", ""},
MaxBytes: 1024,
},
{
Name: " roster ",
Description: " Characters ",
Required: true,
Multiple: true,
},
},
}
if err := registry.RegisterWithSpec(spec, fakeExtractorConstructor("generic-extractor")); err != nil {
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
}
got, ok := registry.Spec("\tgeneric-extractor\n")
if !ok {
t.Fatal("Spec() ok = false, want true")
}
want := ModuleSpec{
Key: "generic-extractor",
Stage: StageExtract,
Provides: []string{"generic-artifact", "source-citations"},
Requires: []string{"source-document"},
ReferenceSlots: []contracts.ReferenceSlot{
{
Name: "glossary",
Description: "Supporting terms",
AcceptedMediaTypes: []string{"text/markdown", "text/plain"},
MaxBytes: 1024,
},
{
Name: "roster",
Description: "Characters",
Required: true,
Multiple: true,
},
},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("Spec() = %#v, want %#v", got, want)
}
got.Provides[0] = "changed"
got.ReferenceSlots[0].Name = "changed"
got.ReferenceSlots[0].AcceptedMediaTypes[0] = "changed"
again, ok := registry.Spec("generic-extractor")
if !ok {
t.Fatal("Spec() after caller mutation ok = false, want true")
}
if !reflect.DeepEqual(again, want) {
t.Fatalf("Spec() after caller mutation = %#v, want %#v", again, want)
}
}
func TestExtractorRegistryRegisterStoresDefaultSpec(t *testing.T) {
registry := NewExtractorRegistry()
if err := registry.Register(" generic-extractor ", fakeExtractorConstructor("generic-extractor")); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
got, ok := registry.Spec("generic-extractor")
if !ok {
t.Fatal("Spec() ok = false, want true")
}
want := ModuleSpec{Key: "generic-extractor", Stage: StageExtract}
if !reflect.DeepEqual(got, want) {
t.Fatalf("Spec() = %#v, want %#v", got, want)
}
}
func TestExtractorRegistryRegisterWithSpecRejectsWrongStage(t *testing.T) {
registry := NewExtractorRegistry()
err := registry.RegisterWithSpec(ModuleSpec{Key: "generic-extractor", Stage: StageInput}, fakeExtractorConstructor("generic-extractor"))
if err == nil {
t.Fatal("RegisterWithSpec() error = nil, want error")
}
if !strings.Contains(err.Error(), "stage") {
t.Fatalf("RegisterWithSpec() error = %q, want stage error", err.Error())
}
}
func TestExtractorRegistryRejectsInvalidReferenceSlots(t *testing.T) {
tests := []struct {
name string
slots []contracts.ReferenceSlot
want string
}{
{
name: "empty name",
slots: []contracts.ReferenceSlot{{Name: " "}},
want: "name",
},
{
name: "duplicate name after trim",
slots: []contracts.ReferenceSlot{
{Name: "roster"},
{Name: " roster "},
},
want: "duplicated",
},
{
name: "negative max bytes",
slots: []contracts.ReferenceSlot{{Name: "roster", MaxBytes: -1}},
want: "max_bytes",
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
registry := NewExtractorRegistry()
err := registry.RegisterWithSpec(ModuleSpec{
Key: "generic-extractor",
Stage: StageExtract,
ReferenceSlots: test.slots,
}, fakeExtractorConstructor("generic-extractor"))
if err == nil {
t.Fatal("RegisterWithSpec() error = nil, want error")
}
if !strings.Contains(err.Error(), test.want) {
t.Fatalf("RegisterWithSpec() error = %q, want %q", err.Error(), test.want)
}
})
}
}
func TestExtractorRegistrySpecRejectsUnknownKey(t *testing.T) {
registry := NewExtractorRegistry()
if _, ok := registry.Spec("missing-extractor"); ok {
t.Fatal("Spec() ok = true, want false")
}
}
func TestExtractorRegistryRegisterRejectsEmptyKey(t *testing.T) {
registry := NewExtractorRegistry()
err := registry.Register(" \t", fakeExtractorConstructor("generic-extractor"))
if err == nil {
t.Fatal("Register() error = nil, want error")
}
if !strings.Contains(err.Error(), "key must not be empty") {
t.Fatalf("Register() error = %q, want empty key error", err.Error())
}
}
func TestExtractorRegistryRegisterRejectsDuplicateKey(t *testing.T) {
registry := NewExtractorRegistry()
if err := registry.Register("generic-extractor", fakeExtractorConstructor("generic-extractor")); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
err := registry.Register(" generic-extractor ", fakeExtractorConstructor("generic-extractor"))
if err == nil {
t.Fatal("Register() error = nil, want error")
}
if !strings.Contains(err.Error(), "already registered") {
t.Fatalf("Register() error = %q, want duplicate key error", err.Error())
}
}
func TestExtractorRegistryRegisterRejectsNilConstructor(t *testing.T) {
registry := NewExtractorRegistry()
err := registry.Register("generic-extractor", nil)
if err == nil {
t.Fatal("Register() error = nil, want error")
}
if !strings.Contains(err.Error(), "constructor") {
t.Fatalf("Register() error = %q, want constructor error", err.Error())
}
}
func TestExtractorRegistryBuildRejectsUnknownKey(t *testing.T) {
registry := NewExtractorRegistry()
_, err := registry.Build("missing-extractor")
if err == nil {
t.Fatal("Build() error = nil, want error")
}
if !strings.Contains(err.Error(), "not registered") {
t.Fatalf("Build() error = %q, want unknown key error", err.Error())
}
}
func TestExtractorRegistryBuildWrapsConstructorError(t *testing.T) {
registry := NewExtractorRegistry()
constructorErr := errors.New("constructor failed")
if err := registry.Register("generic-extractor", func() (contracts.Extractor, error) {
return nil, constructorErr
}); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
_, err := registry.Build("generic-extractor")
if err == nil {
t.Fatal("Build() error = nil, want error")
}
if !errors.Is(err, constructorErr) {
t.Fatalf("Build() error = %v, want wrapped constructor error", err)
}
if !strings.Contains(err.Error(), "generic-extractor") {
t.Fatalf("Build() error = %q, want key context", err.Error())
}
}
func TestExtractorRegistryBuildRejectsNilExtractor(t *testing.T) {
registry := NewExtractorRegistry()
if err := registry.Register("generic-extractor", func() (contracts.Extractor, error) {
return nil, nil
}); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
_, err := registry.Build("generic-extractor")
if err == nil {
t.Fatal("Build() error = nil, want error")
}
if !strings.Contains(err.Error(), "returned nil") {
t.Fatalf("Build() error = %q, want nil extractor error", err.Error())
}
}
func TestExtractorRegistryBuildRejectsExtractorKeyMismatch(t *testing.T) {
registry := NewExtractorRegistry()
if err := registry.Register("generic-extractor", fakeExtractorConstructor("other-extractor")); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
_, err := registry.Build("generic-extractor")
if err == nil {
t.Fatal("Build() error = nil, want error")
}
if !strings.Contains(err.Error(), "returned key") {
t.Fatalf("Build() error = %q, want key mismatch error", err.Error())
}
}
func TestExtractorRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
registry := NewExtractorRegistry()
for _, key := range []string{"zeta", "alpha", "middle"} {
if err := registry.Register(key, fakeExtractorConstructor(key)); err != nil {
t.Fatalf("Register(%q) error = %v, want nil", key, err)
}
}
keys := registry.RegisteredKeys()
want := []string{"alpha", "middle", "zeta"}
if !reflect.DeepEqual(keys, want) {
t.Fatalf("RegisteredKeys() = %#v, want %#v", keys, want)
}
keys[0] = "changed"
if got := registry.RegisteredKeys(); !reflect.DeepEqual(got, want) {
t.Fatalf("RegisteredKeys() after caller mutation = %#v, want %#v", got, want)
}
}
func TestExtractorRegistryNilRegistryBehavior(t *testing.T) {
var registry *ExtractorRegistry
if err := registry.Register("generic-extractor", fakeExtractorConstructor("generic-extractor")); err == nil {
t.Fatal("Register() error = nil, want error")
}
if _, err := registry.Build("generic-extractor"); err == nil {
t.Fatal("Build() error = nil, want error")
}
if _, ok := registry.Spec("generic-extractor"); ok {
t.Fatal("Spec() ok = true, want false")
}
if keys := registry.RegisteredKeys(); keys != nil {
t.Fatalf("RegisteredKeys() = %#v, want nil", keys)
}
}
func TestExtractorRegistryBuildRejectsEmptyKey(t *testing.T) {
registry := NewExtractorRegistry()
_, err := registry.Build(" \n")
if err == nil {
t.Fatal("Build() error = nil, want error")
}
if !strings.Contains(err.Error(), "key must not be empty") {
t.Fatalf("Build() error = %q, want empty key error", err.Error())
}
}
type registryFakeExtractor struct {
key string
}
func fakeExtractorConstructor(key string) ExtractorConstructor {
return func() (contracts.Extractor, error) {
return registryFakeExtractor{key: key}, nil
}
}
func (extractor registryFakeExtractor) Key() string {
return extractor.key
}
func (extractor registryFakeExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (extractor registryFakeExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
return contracts.ExtractionResult{}, nil
}

View File

@@ -8,16 +8,19 @@ import (
)
type InputAdapterConstructor func() (contracts.InputAdapter, error)
type InputAdapterBuilder func(BuildRequest) (contracts.InputAdapter, error)
type InputAdapterRegistry struct {
constructors map[string]InputAdapterConstructor
specs map[string]ModuleSpec
builders map[string]InputAdapterBuilder
optionValidators map[string]OptionValidator
specs map[string]ModuleSpec
}
func NewInputAdapterRegistry() *InputAdapterRegistry {
return &InputAdapterRegistry{
constructors: make(map[string]InputAdapterConstructor),
specs: make(map[string]ModuleSpec),
builders: make(map[string]InputAdapterBuilder),
optionValidators: make(map[string]OptionValidator),
specs: make(map[string]ModuleSpec),
}
}
@@ -26,6 +29,15 @@ func (r *InputAdapterRegistry) Register(key string, constructor InputAdapterCons
}
func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor InputAdapterConstructor) error {
if constructor == nil {
return fmt.Errorf("input adapter constructor for %q must not be nil", strings.TrimSpace(spec.Key))
}
return r.RegisterBuilderWithSpec(spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.InputAdapter, error) {
return constructor()
})
}
func (r *InputAdapterRegistry) RegisterBuilderWithSpec(spec ModuleSpec, validateOptions OptionValidator, builder InputAdapterBuilder) error {
if r == nil {
return fmt.Errorf("input adapter registry must not be nil")
}
@@ -34,25 +46,36 @@ func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor Inp
if err := validateModuleSpec("input adapter", StageInput, normalizedSpec); err != nil {
return err
}
if constructor == nil {
return fmt.Errorf("input adapter constructor for %q must not be nil", normalizedSpec.Key)
if validateOptions == nil {
return fmt.Errorf("input adapter option validator for %q must not be nil", normalizedSpec.Key)
}
if _, ok := r.constructors[normalizedSpec.Key]; ok {
if builder == nil {
return fmt.Errorf("input adapter builder for %q must not be nil", normalizedSpec.Key)
}
if _, ok := r.builders[normalizedSpec.Key]; ok {
return fmt.Errorf("input adapter %q is already registered", normalizedSpec.Key)
}
if r.constructors == nil {
r.constructors = make(map[string]InputAdapterConstructor)
if r.builders == nil {
r.builders = make(map[string]InputAdapterBuilder)
}
if r.optionValidators == nil {
r.optionValidators = make(map[string]OptionValidator)
}
if r.specs == nil {
r.specs = make(map[string]ModuleSpec)
}
r.constructors[normalizedSpec.Key] = constructor
r.builders[normalizedSpec.Key] = builder
r.optionValidators[normalizedSpec.Key] = validateOptions
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
return nil
}
func (r *InputAdapterRegistry) Build(key string) (contracts.InputAdapter, error) {
return r.BuildWithRequest(key, BuildRequest{})
}
func (r *InputAdapterRegistry) BuildWithRequest(key string, request BuildRequest) (contracts.InputAdapter, error) {
if r == nil {
return nil, fmt.Errorf("input adapter registry must not be nil")
}
@@ -62,12 +85,12 @@ func (r *InputAdapterRegistry) Build(key string) (contracts.InputAdapter, error)
return nil, fmt.Errorf("input adapter key must not be empty")
}
constructor, ok := r.constructors[normalizedKey]
builder, ok := r.builders[normalizedKey]
if !ok {
return nil, fmt.Errorf("input adapter %q is not registered", normalizedKey)
}
adapter, err := constructor()
adapter, err := builder(cloneBuildRequest(request))
if err != nil {
return nil, fmt.Errorf("build input adapter %q: %w", normalizedKey, err)
}
@@ -81,6 +104,18 @@ func (r *InputAdapterRegistry) Build(key string) (contracts.InputAdapter, error)
return adapter, nil
}
func (r *InputAdapterRegistry) ValidateOptions(key string, options map[string]any) error {
if r == nil {
return fmt.Errorf("input adapter registry must not be nil")
}
normalizedKey := strings.TrimSpace(key)
validator, ok := r.optionValidators[normalizedKey]
if !ok {
return fmt.Errorf("input adapter %q is not registered", normalizedKey)
}
return validateRegisteredOptions(validator, options)
}
func (r *InputAdapterRegistry) Spec(key string) (ModuleSpec, bool) {
if r == nil {
return ModuleSpec{}, false
@@ -98,5 +133,5 @@ func (r *InputAdapterRegistry) RegisteredKeys() []string {
return nil
}
return sortedRegistryKeys(r.constructors)
return sortedRegistryKeys(r.builders)
}

View File

@@ -301,7 +301,7 @@ func (adapter fakeAdapter) Parse(ctx context.Context, req contracts.ParseRequest
Format: "text/plain",
Digest: "sha256:abc123",
Units: []source.SourceUnit{
{ID: 1, Kind: "unit", Text: "Source unit."},
{ID: 1, Kind: "unit", Text: "Source unit.", Ref: source.SourceRef{SourceID: req.SourceID, StartUnitID: 1, EndUnitID: 1}},
},
}, nil
}

View File

@@ -1,102 +1,155 @@
package pipeline
import (
"context"
"fmt"
"reflect"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type MergerConstructor func() (contracts.Merger, error)
type artifactVariantKey struct {
module string
kind contracts.ArtifactKind
}
type MergerRegistry struct {
constructors map[string]MergerConstructor
specs map[string]ModuleSpec
typedEntries map[artifactVariantKey]typedMergerEntry
}
type typedMergerEntry struct {
spec ModuleSpec
valueType reflect.Type
validateOptions OptionValidator
builder func(BuildRequest) (any, error)
merge typedMergeOperation
}
func NewMergerRegistry() *MergerRegistry {
return &MergerRegistry{
constructors: make(map[string]MergerConstructor),
specs: make(map[string]ModuleSpec),
typedEntries: make(map[artifactVariantKey]typedMergerEntry),
}
}
func (r *MergerRegistry) Register(key string, constructor MergerConstructor) error {
return r.RegisterWithSpec(defaultModuleSpec(key, StageMerge), constructor)
func RegisterMerger[T any](registry *MergerRegistry, spec ModuleSpec, constructor func() (contracts.Merger[T], error)) error {
if constructor == nil {
return fmt.Errorf("merger constructor for %q must not be nil", strings.TrimSpace(spec.Key))
}
return RegisterMergerBuilder(registry, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.Merger[T], error) {
return constructor()
})
}
func (r *MergerRegistry) RegisterWithSpec(spec ModuleSpec, constructor MergerConstructor) error {
if r == nil {
func RegisterMergerBuilder[T any](registry *MergerRegistry, spec ModuleSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.Merger[T], error)) error {
if registry == nil {
return fmt.Errorf("merger registry must not be nil")
}
normalizedSpec := normalizeModuleSpec(spec)
if err := validateModuleSpec("merger", StageMerge, normalizedSpec); err != nil {
return err
}
if constructor == nil {
return fmt.Errorf("merger constructor for %q must not be nil", normalizedSpec.Key)
if normalizedSpec.ArtifactKind == "" {
return fmt.Errorf("typed merger %q artifact kind must not be empty", normalizedSpec.Key)
}
if _, ok := r.constructors[normalizedSpec.Key]; ok {
return fmt.Errorf("merger %q is already registered", normalizedSpec.Key)
if validateOptions == nil {
return fmt.Errorf("merger option validator for %q must not be nil", normalizedSpec.Key)
}
if r.constructors == nil {
r.constructors = make(map[string]MergerConstructor)
if builder == nil {
return fmt.Errorf("merger builder for %q must not be nil", normalizedSpec.Key)
}
if r.specs == nil {
r.specs = make(map[string]ModuleSpec)
key := artifactVariantKey{module: normalizedSpec.Key, kind: normalizedSpec.ArtifactKind}
if _, ok := registry.typedEntries[key]; ok {
return fmt.Errorf("merger %q variant for artifact kind %q is already registered", key.module, key.kind)
}
if registry.typedEntries == nil {
registry.typedEntries = make(map[artifactVariantKey]typedMergerEntry)
}
registry.typedEntries[key] = typedMergerEntry{
spec: cloneModuleSpec(normalizedSpec),
valueType: reflect.TypeFor[T](),
validateOptions: validateOptions,
builder: func(request BuildRequest) (any, error) {
return builder(cloneBuildRequest(request))
},
merge: func(ctx context.Context, implementation any, request contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
merger, ok := implementation.(contracts.Merger[T])
if !ok {
return erasedTypedResult{}, fmt.Errorf("merger %q has incompatible implementation %T", normalizedSpec.Key, implementation)
}
outputs := make([]contracts.ExtractArtifact[T], len(request.ExtractOutputs))
for i, output := range request.ExtractOutputs {
value, err := exactTypedValue[T]("merge extract value", output.Value)
if err != nil {
return erasedTypedResult{}, err
}
outputs[i] = contracts.ExtractArtifact[T]{LaneID: output.LaneID, ExtractorKey: output.ExtractorKey, SourceID: output.SourceID, ChunkID: output.ChunkID, ChunkIndex: output.ChunkIndex, ChunkRef: output.ChunkRef, Value: value}
}
result, err := merger.Merge(ctx, contracts.TypedMergeRequest[T]{Source: request.Source, LaneID: request.LaneID, ExtractOutputs: outputs, SourceInput: request.SourceInput, SessionID: request.SessionID, References: request.References, LLMProfile: request.LLMProfile, Metadata: request.Metadata})
if err != nil {
return erasedTypedResult{}, err
}
return erasedTypedResult{Value: result.Value, Warnings: result.Warnings}, nil
},
}
r.constructors[normalizedSpec.Key] = constructor
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
return nil
}
func (r *MergerRegistry) Build(key string) (contracts.Merger, error) {
func (r *MergerRegistry) validateOptions(key string, kind contracts.ArtifactKind, options map[string]any) error {
if r == nil {
return nil, fmt.Errorf("merger registry must not be nil")
return fmt.Errorf("merger registry must not be nil")
}
normalizedKey := strings.TrimSpace(key)
if normalizedKey == "" {
return nil, fmt.Errorf("merger key must not be empty")
}
constructor, ok := r.constructors[normalizedKey]
entry, ok := r.typedEntry(normalizedKey, kind)
if !ok {
return nil, fmt.Errorf("merger %q is not registered", normalizedKey)
return fmt.Errorf("merger %q variant for artifact kind %q is not registered", normalizedKey, kind)
}
merger, err := constructor()
if err != nil {
return nil, fmt.Errorf("build merger %q: %w", normalizedKey, err)
}
if merger == nil {
return nil, fmt.Errorf("merger %q constructor returned nil", normalizedKey)
}
if merger.Key() != normalizedKey {
return nil, fmt.Errorf("merger %q returned key %q", normalizedKey, merger.Key())
}
return merger, nil
return validateRegisteredOptions(entry.validateOptions, options)
}
func (r *MergerRegistry) Spec(key string) (ModuleSpec, bool) {
if r == nil {
return ModuleSpec{}, false
}
spec, ok := r.specs[strings.TrimSpace(key)]
if !ok {
return ModuleSpec{}, false
module := strings.TrimSpace(key)
for variant, entry := range r.typedEntries {
if variant.module == module {
return cloneModuleSpec(entry.spec), true
}
}
return cloneModuleSpec(spec), true
return ModuleSpec{}, false
}
func (r *MergerRegistry) typedEntry(key string, kind contracts.ArtifactKind) (typedMergerEntry, bool) {
if r == nil {
return typedMergerEntry{}, false
}
entry, ok := r.typedEntries[artifactVariantKey{module: strings.TrimSpace(key), kind: normalizeArtifactKind(kind)}]
return entry, ok
}
func (r *MergerRegistry) registeredKinds(key string) []contracts.ArtifactKind {
if r == nil {
return nil
}
module := strings.TrimSpace(key)
kinds := make([]contracts.ArtifactKind, 0)
for variant := range r.typedEntries {
if variant.module == module {
kinds = append(kinds, variant.kind)
}
}
sortArtifactKinds(kinds)
return kinds
}
func (r *MergerRegistry) RegisteredKeys() []string {
if r == nil {
return nil
}
return sortedRegistryKeys(r.constructors)
keys := make(map[string]struct{}, len(r.typedEntries))
for key := range r.typedEntries {
keys[key.module] = struct{}{}
}
return sortedRegistryKeys(keys)
}

View File

@@ -1,58 +0,0 @@
package pipeline
import (
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestMergerRegistryBehavior(t *testing.T) {
runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Merger]{
name: "MergerRegistry",
key: "generic-merger",
stage: StageMerge,
wrongStage: StageExtract,
newRegistry: func() any {
return NewMergerRegistry()
},
register: func(registry any, key string, constructor func() (contracts.Merger, error)) error {
return registry.(*MergerRegistry).Register(key, constructor)
},
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Merger, error)) error {
return registry.(*MergerRegistry).RegisterWithSpec(spec, constructor)
},
build: func(registry any, key string) (contracts.Merger, error) {
return registry.(*MergerRegistry).Build(key)
},
spec: func(registry any, key string) (ModuleSpec, bool) {
return registry.(*MergerRegistry).Spec(key)
},
registeredKeys: func(registry any) []string {
return registry.(*MergerRegistry).RegisteredKeys()
},
nilRegister: func(key string, constructor func() (contracts.Merger, error)) error {
var registry *MergerRegistry
return registry.Register(key, constructor)
},
nilBuild: func(key string) (contracts.Merger, error) {
var registry *MergerRegistry
return registry.Build(key)
},
nilSpec: func(key string) (ModuleSpec, bool) {
var registry *MergerRegistry
return registry.Spec(key)
},
nilRegisteredKey: func() []string {
var registry *MergerRegistry
return registry.RegisteredKeys()
},
constructor: func(key string) func() (contracts.Merger, error) {
return func() (contracts.Merger, error) {
return registryMerger{key: key}, nil
}
},
moduleKey: func(module contracts.Merger) string {
return module.Key()
},
})
}

View File

@@ -23,6 +23,7 @@ const (
type ModuleSpec struct {
Key string
Stage ModuleStage
ArtifactKind contracts.ArtifactKind
Provides []string
Requires []string
ReferenceSlots []contracts.ReferenceSlot
@@ -39,6 +40,7 @@ func normalizeModuleSpec(spec ModuleSpec) ModuleSpec {
return ModuleSpec{
Key: strings.TrimSpace(spec.Key),
Stage: spec.Stage,
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
Provides: normalizeCapabilities(spec.Provides),
Requires: normalizeCapabilities(spec.Requires),
ReferenceSlots: normalizeReferenceSlots(spec.ReferenceSlots),
@@ -74,6 +76,7 @@ func cloneModuleSpec(spec ModuleSpec) ModuleSpec {
return ModuleSpec{
Key: spec.Key,
Stage: spec.Stage,
ArtifactKind: spec.ArtifactKind,
Provides: append([]string(nil), spec.Provides...),
Requires: append([]string(nil), spec.Requires...),
ReferenceSlots: contracts.CloneReferenceSlots(spec.ReferenceSlots),
@@ -87,6 +90,9 @@ func validateModuleSpec(kind string, expectedStage ModuleStage, spec ModuleSpec)
if spec.Stage != expectedStage {
return fmt.Errorf("%s %q must use %q stage, got %q", kind, spec.Key, expectedStage, spec.Stage)
}
if spec.ArtifactKind != "" && spec.Stage != StageExtract && spec.Stage != StageMerge && spec.Stage != StageNormalize {
return fmt.Errorf("%s %q must not declare an artifact kind", kind, spec.Key)
}
if !referenceSlotStage(spec.Stage) && len(spec.ReferenceSlots) > 0 {
return fmt.Errorf("%s %q must not declare reference slots", kind, spec.Key)
}
@@ -113,6 +119,10 @@ func sortedRegistryKeys[C any](constructors map[string]C) []string {
return keys
}
func sortArtifactKinds(kinds []contracts.ArtifactKind) {
sort.Slice(kinds, func(i, j int) bool { return kinds[i] < kinds[j] })
}
func normalizeReferenceSlots(slots []contracts.ReferenceSlot) []contracts.ReferenceSlot {
if len(slots) == 0 {
return nil

View File

@@ -1,102 +1,146 @@
package pipeline
import (
"context"
"fmt"
"reflect"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type NormalizerConstructor func() (contracts.Normalizer, error)
type NormalizerRegistry struct {
constructors map[string]NormalizerConstructor
specs map[string]ModuleSpec
typedEntries map[artifactVariantKey]typedNormalizerEntry
}
type typedNormalizerEntry struct {
spec ModuleSpec
valueType reflect.Type
validateOptions OptionValidator
builder func(BuildRequest) (any, error)
normalize typedNormalizeOperation
}
func NewNormalizerRegistry() *NormalizerRegistry {
return &NormalizerRegistry{
constructors: make(map[string]NormalizerConstructor),
specs: make(map[string]ModuleSpec),
typedEntries: make(map[artifactVariantKey]typedNormalizerEntry),
}
}
func (r *NormalizerRegistry) Register(key string, constructor NormalizerConstructor) error {
return r.RegisterWithSpec(defaultModuleSpec(key, StageNormalize), constructor)
func RegisterNormalizer[T any](registry *NormalizerRegistry, spec ModuleSpec, constructor func() (contracts.Normalizer[T], error)) error {
if constructor == nil {
return fmt.Errorf("normalizer constructor for %q must not be nil", strings.TrimSpace(spec.Key))
}
return RegisterNormalizerBuilder(registry, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.Normalizer[T], error) {
return constructor()
})
}
func (r *NormalizerRegistry) RegisterWithSpec(spec ModuleSpec, constructor NormalizerConstructor) error {
if r == nil {
func RegisterNormalizerBuilder[T any](registry *NormalizerRegistry, spec ModuleSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.Normalizer[T], error)) error {
if registry == nil {
return fmt.Errorf("normalizer registry must not be nil")
}
normalizedSpec := normalizeModuleSpec(spec)
if err := validateModuleSpec("normalizer", StageNormalize, normalizedSpec); err != nil {
return err
}
if constructor == nil {
return fmt.Errorf("normalizer constructor for %q must not be nil", normalizedSpec.Key)
if normalizedSpec.ArtifactKind == "" {
return fmt.Errorf("typed normalizer %q artifact kind must not be empty", normalizedSpec.Key)
}
if _, ok := r.constructors[normalizedSpec.Key]; ok {
return fmt.Errorf("normalizer %q is already registered", normalizedSpec.Key)
if validateOptions == nil {
return fmt.Errorf("normalizer option validator for %q must not be nil", normalizedSpec.Key)
}
if r.constructors == nil {
r.constructors = make(map[string]NormalizerConstructor)
if builder == nil {
return fmt.Errorf("normalizer builder for %q must not be nil", normalizedSpec.Key)
}
if r.specs == nil {
r.specs = make(map[string]ModuleSpec)
key := artifactVariantKey{module: normalizedSpec.Key, kind: normalizedSpec.ArtifactKind}
if _, ok := registry.typedEntries[key]; ok {
return fmt.Errorf("normalizer %q variant for artifact kind %q is already registered", key.module, key.kind)
}
if registry.typedEntries == nil {
registry.typedEntries = make(map[artifactVariantKey]typedNormalizerEntry)
}
registry.typedEntries[key] = typedNormalizerEntry{
spec: cloneModuleSpec(normalizedSpec),
valueType: reflect.TypeFor[T](),
validateOptions: validateOptions,
builder: func(request BuildRequest) (any, error) {
return builder(cloneBuildRequest(request))
},
normalize: func(ctx context.Context, implementation any, request contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
normalizer, ok := implementation.(contracts.Normalizer[T])
if !ok {
return erasedTypedResult{}, fmt.Errorf("normalizer %q has incompatible implementation %T", normalizedSpec.Key, implementation)
}
value, err := exactTypedValue[T]("normalize merge value", request.MergeOutput.Value)
if err != nil {
return erasedTypedResult{}, err
}
result, err := normalizer.Normalize(ctx, contracts.TypedNormalizeRequest[T]{Source: request.Source, LaneID: request.LaneID, MergeOutput: contracts.MergeArtifact[T]{LaneID: request.MergeOutput.LaneID, MergerKey: request.MergeOutput.MergerKey, SourceID: request.MergeOutput.SourceID, Value: value}, SourceInput: request.SourceInput, SessionID: request.SessionID, References: request.References, LLMProfile: request.LLMProfile, Metadata: request.Metadata})
if err != nil {
return erasedTypedResult{}, err
}
return erasedTypedResult{Value: result.Value, Warnings: result.Warnings}, nil
},
}
r.constructors[normalizedSpec.Key] = constructor
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
return nil
}
func (r *NormalizerRegistry) Build(key string) (contracts.Normalizer, error) {
func (r *NormalizerRegistry) validateOptions(key string, kind contracts.ArtifactKind, options map[string]any) error {
if r == nil {
return nil, fmt.Errorf("normalizer registry must not be nil")
return fmt.Errorf("normalizer registry must not be nil")
}
normalizedKey := strings.TrimSpace(key)
if normalizedKey == "" {
return nil, fmt.Errorf("normalizer key must not be empty")
}
constructor, ok := r.constructors[normalizedKey]
entry, ok := r.typedEntry(normalizedKey, kind)
if !ok {
return nil, fmt.Errorf("normalizer %q is not registered", normalizedKey)
return fmt.Errorf("normalizer %q variant for artifact kind %q is not registered", normalizedKey, kind)
}
normalizer, err := constructor()
if err != nil {
return nil, fmt.Errorf("build normalizer %q: %w", normalizedKey, err)
}
if normalizer == nil {
return nil, fmt.Errorf("normalizer %q constructor returned nil", normalizedKey)
}
if normalizer.Key() != normalizedKey {
return nil, fmt.Errorf("normalizer %q returned key %q", normalizedKey, normalizer.Key())
}
return normalizer, nil
return validateRegisteredOptions(entry.validateOptions, options)
}
func (r *NormalizerRegistry) Spec(key string) (ModuleSpec, bool) {
if r == nil {
return ModuleSpec{}, false
}
spec, ok := r.specs[strings.TrimSpace(key)]
if !ok {
return ModuleSpec{}, false
module := strings.TrimSpace(key)
for variant, entry := range r.typedEntries {
if variant.module == module {
return cloneModuleSpec(entry.spec), true
}
}
return cloneModuleSpec(spec), true
return ModuleSpec{}, false
}
func (r *NormalizerRegistry) typedEntry(key string, kind contracts.ArtifactKind) (typedNormalizerEntry, bool) {
if r == nil {
return typedNormalizerEntry{}, false
}
entry, ok := r.typedEntries[artifactVariantKey{module: strings.TrimSpace(key), kind: normalizeArtifactKind(kind)}]
return entry, ok
}
func (r *NormalizerRegistry) registeredKinds(key string) []contracts.ArtifactKind {
if r == nil {
return nil
}
module := strings.TrimSpace(key)
kinds := make([]contracts.ArtifactKind, 0)
for variant := range r.typedEntries {
if variant.module == module {
kinds = append(kinds, variant.kind)
}
}
sortArtifactKinds(kinds)
return kinds
}
func (r *NormalizerRegistry) RegisteredKeys() []string {
if r == nil {
return nil
}
return sortedRegistryKeys(r.constructors)
keys := make(map[string]struct{}, len(r.typedEntries))
for key := range r.typedEntries {
keys[key.module] = struct{}{}
}
return sortedRegistryKeys(keys)
}

View File

@@ -1,58 +0,0 @@
package pipeline
import (
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestNormalizerRegistryBehavior(t *testing.T) {
runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Normalizer]{
name: "NormalizerRegistry",
key: "generic-normalizer",
stage: StageNormalize,
wrongStage: StageExtract,
newRegistry: func() any {
return NewNormalizerRegistry()
},
register: func(registry any, key string, constructor func() (contracts.Normalizer, error)) error {
return registry.(*NormalizerRegistry).Register(key, constructor)
},
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Normalizer, error)) error {
return registry.(*NormalizerRegistry).RegisterWithSpec(spec, constructor)
},
build: func(registry any, key string) (contracts.Normalizer, error) {
return registry.(*NormalizerRegistry).Build(key)
},
spec: func(registry any, key string) (ModuleSpec, bool) {
return registry.(*NormalizerRegistry).Spec(key)
},
registeredKeys: func(registry any) []string {
return registry.(*NormalizerRegistry).RegisteredKeys()
},
nilRegister: func(key string, constructor func() (contracts.Normalizer, error)) error {
var registry *NormalizerRegistry
return registry.Register(key, constructor)
},
nilBuild: func(key string) (contracts.Normalizer, error) {
var registry *NormalizerRegistry
return registry.Build(key)
},
nilSpec: func(key string) (ModuleSpec, bool) {
var registry *NormalizerRegistry
return registry.Spec(key)
},
nilRegisteredKey: func() []string {
var registry *NormalizerRegistry
return registry.RegisteredKeys()
},
constructor: func(key string) func() (contracts.Normalizer, error) {
return func() (contracts.Normalizer, error) {
return registryNormalizer{key: key}, nil
}
},
moduleKey: func(module contracts.Normalizer) string {
return module.Key()
},
})
}

View File

@@ -0,0 +1,65 @@
package pipeline
import "fmt"
func validateResolvedOptions(resolved ResolvedPipeline, catalog ModuleCatalog) error {
if err := catalog.Inputs.ValidateOptions(resolved.Input.Module, resolved.Input.Options); err != nil {
return moduleOptionsError(resolved.ID, "", StageInput, resolved.Input.Module, err)
}
if err := catalog.Chunkers.ValidateOptions(resolved.Chunk.Module, resolved.Chunk.Options); err != nil {
return moduleOptionsError(resolved.ID, "", StageChunk, resolved.Chunk.Module, err)
}
if err := validateChainOptions(resolved, catalog, StageChunk, "", resolved.Chunk.Module); err != nil {
return err
}
for _, lane := range resolved.ArtifactLanes {
if err := catalog.Extractors.validateOptions(lane.Extract.Module, lane.Extract.Options); err != nil {
return moduleOptionsError(resolved.ID, lane.ID, StageExtract, lane.Extract.Module, err)
}
if err := validateChainOptions(resolved, catalog, StageExtract, lane.ID, lane.Extract.Module); err != nil {
return err
}
if err := catalog.Mergers.validateOptions(lane.Merge.Module, lane.ArtifactKind, lane.Merge.Options); err != nil {
return moduleOptionsError(resolved.ID, lane.ID, StageMerge, lane.Merge.Module, err)
}
if err := validateChainOptions(resolved, catalog, StageMerge, lane.ID, lane.Merge.Module); err != nil {
return err
}
if err := catalog.Normalizers.validateOptions(lane.Normalize.Module, lane.ArtifactKind, lane.Normalize.Options); err != nil {
return moduleOptionsError(resolved.ID, lane.ID, StageNormalize, lane.Normalize.Module, err)
}
if err := validateChainOptions(resolved, catalog, StageNormalize, lane.ID, lane.Normalize.Module); err != nil {
return err
}
}
if err := catalog.Outputs.ValidateOptions(resolved.Output.Module, resolved.Output.Options); err != nil {
return moduleOptionsError(resolved.ID, "", StageOutput, resolved.Output.Module, err)
}
return nil
}
func validateChainOptions(resolved ResolvedPipeline, catalog ModuleCatalog, stage ModuleStage, laneID, moduleKey string) error {
chain := resolvedValidatorChain(stage, laneID, moduleKey, resolved.ValidatorChains)
for _, validator := range chain.Validators {
if err := catalog.Validators.validateOptions(validator); err != nil {
return validatorOptionsError(resolved.ID, laneID, stage, moduleKey, validator.Binding.Module, err)
}
}
return nil
}
func moduleOptionsError(pipelineID, laneID string, stage ModuleStage, moduleKey string, cause error) error {
if laneID == "" {
return fmt.Errorf("pipeline %q %s module %q options: %w", pipelineID, stage, moduleKey, cause)
}
return fmt.Errorf("pipeline %q lane %q %s module %q options: %w", pipelineID, laneID, stage, moduleKey, cause)
}
func validatorOptionsError(pipelineID, laneID string, stage ModuleStage, moduleKey, validatorKey string, cause error) error {
if laneID == "" {
return fmt.Errorf("pipeline %q %s module %q validator %q options: %w", pipelineID, stage, moduleKey, validatorKey, cause)
}
return fmt.Errorf("pipeline %q lane %q %s module %q validator %q options: %w", pipelineID, laneID, stage, moduleKey, validatorKey, cause)
}

View File

@@ -8,16 +8,19 @@ import (
)
type OutputEncoderConstructor func() (contracts.OutputEncoder, error)
type OutputEncoderBuilder func(BuildRequest) (contracts.OutputEncoder, error)
type OutputEncoderRegistry struct {
constructors map[string]OutputEncoderConstructor
specs map[string]ModuleSpec
builders map[string]OutputEncoderBuilder
optionValidators map[string]OptionValidator
specs map[string]ModuleSpec
}
func NewOutputEncoderRegistry() *OutputEncoderRegistry {
return &OutputEncoderRegistry{
constructors: make(map[string]OutputEncoderConstructor),
specs: make(map[string]ModuleSpec),
builders: make(map[string]OutputEncoderBuilder),
optionValidators: make(map[string]OptionValidator),
specs: make(map[string]ModuleSpec),
}
}
@@ -26,6 +29,15 @@ func (r *OutputEncoderRegistry) Register(key string, constructor OutputEncoderCo
}
func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor OutputEncoderConstructor) error {
if constructor == nil {
return fmt.Errorf("output encoder constructor for %q must not be nil", strings.TrimSpace(spec.Key))
}
return r.RegisterBuilderWithSpec(spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.OutputEncoder, error) {
return constructor()
})
}
func (r *OutputEncoderRegistry) RegisterBuilderWithSpec(spec ModuleSpec, validateOptions OptionValidator, builder OutputEncoderBuilder) error {
if r == nil {
return fmt.Errorf("output encoder registry must not be nil")
}
@@ -34,25 +46,36 @@ func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor Ou
if err := validateModuleSpec("output encoder", StageOutput, normalizedSpec); err != nil {
return err
}
if constructor == nil {
return fmt.Errorf("output encoder constructor for %q must not be nil", normalizedSpec.Key)
if validateOptions == nil {
return fmt.Errorf("output encoder option validator for %q must not be nil", normalizedSpec.Key)
}
if _, ok := r.constructors[normalizedSpec.Key]; ok {
if builder == nil {
return fmt.Errorf("output encoder builder for %q must not be nil", normalizedSpec.Key)
}
if _, ok := r.builders[normalizedSpec.Key]; ok {
return fmt.Errorf("output encoder %q is already registered", normalizedSpec.Key)
}
if r.constructors == nil {
r.constructors = make(map[string]OutputEncoderConstructor)
if r.builders == nil {
r.builders = make(map[string]OutputEncoderBuilder)
}
if r.optionValidators == nil {
r.optionValidators = make(map[string]OptionValidator)
}
if r.specs == nil {
r.specs = make(map[string]ModuleSpec)
}
r.constructors[normalizedSpec.Key] = constructor
r.builders[normalizedSpec.Key] = builder
r.optionValidators[normalizedSpec.Key] = validateOptions
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
return nil
}
func (r *OutputEncoderRegistry) Build(key string) (contracts.OutputEncoder, error) {
return r.BuildWithRequest(key, BuildRequest{})
}
func (r *OutputEncoderRegistry) BuildWithRequest(key string, request BuildRequest) (contracts.OutputEncoder, error) {
if r == nil {
return nil, fmt.Errorf("output encoder registry must not be nil")
}
@@ -62,12 +85,12 @@ func (r *OutputEncoderRegistry) Build(key string) (contracts.OutputEncoder, erro
return nil, fmt.Errorf("output encoder key must not be empty")
}
constructor, ok := r.constructors[normalizedKey]
builder, ok := r.builders[normalizedKey]
if !ok {
return nil, fmt.Errorf("output encoder %q is not registered", normalizedKey)
}
encoder, err := constructor()
encoder, err := builder(cloneBuildRequest(request))
if err != nil {
return nil, fmt.Errorf("build output encoder %q: %w", normalizedKey, err)
}
@@ -81,6 +104,18 @@ func (r *OutputEncoderRegistry) Build(key string) (contracts.OutputEncoder, erro
return encoder, nil
}
func (r *OutputEncoderRegistry) ValidateOptions(key string, options map[string]any) error {
if r == nil {
return fmt.Errorf("output encoder registry must not be nil")
}
normalizedKey := strings.TrimSpace(key)
validator, ok := r.optionValidators[normalizedKey]
if !ok {
return fmt.Errorf("output encoder %q is not registered", normalizedKey)
}
return validateRegisteredOptions(validator, options)
}
func (r *OutputEncoderRegistry) Spec(key string) (ModuleSpec, bool) {
if r == nil {
return ModuleSpec{}, false
@@ -98,5 +133,5 @@ func (r *OutputEncoderRegistry) RegisteredKeys() []string {
return nil
}
return sortedRegistryKeys(r.constructors)
return sortedRegistryKeys(r.builders)
}

View File

@@ -0,0 +1,240 @@
package pipeline
import (
"context"
"errors"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestResolvePipelineValidatesModuleAndValidatorOptions(t *testing.T) {
tests := []struct {
name string
mutate func(*PipelineProfile)
want []string
}{
{
name: "module",
mutate: func(profile *PipelineProfile) {
profile.Input.Options = map[string]any{"surprise": true}
},
want: []string{`pipeline "construction" input module "input" options`, `unknown option "surprise"`},
},
{
name: "validator",
mutate: func(profile *PipelineProfile) {
profile.Chunk.Validators = ValidatorOverride{Set: true, Validators: []ModuleBinding{{Module: "configured", Options: map[string]any{"surprise": true}}}}
},
want: []string{`pipeline "construction" chunk module "chunk" validator "configured" options`, `unknown option "surprise"`},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
registries, _ := constructionRegistries(t, nil, nil)
profile := constructionProfile()
test.mutate(&profile)
_, err := ResolvePipeline(profile, ResolveOptions{}, registries.catalog())
if err == nil {
t.Fatal("ResolvePipeline() error = nil, want option validation error")
}
for _, want := range test.want {
if !strings.Contains(err.Error(), want) {
t.Fatalf("ResolvePipeline() error = %q, want substring %q", err, want)
}
}
})
}
}
func TestPrepareConstructsEverythingInStableOrder(t *testing.T) {
var built []string
registries, _ := constructionRegistries(t, &built, nil)
resolved, err := ResolvePipeline(constructionProfile(), ResolveOptions{}, registries.catalog())
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
prepared, err := Prepare(resolved, registries, ModuleDependencies{})
if err != nil {
t.Fatalf("Prepare() error = %v, want nil", err)
}
want := []string{"input", "chunk", "validator", "extract", "validator", "merge", "validator", "normalize", "validator", "output"}
if !reflect.DeepEqual(built, want) {
t.Fatalf("construction order = %#v, want %#v", built, want)
}
if prepared.Input.Module != "input" || prepared.Chunk.Module != "chunk" || prepared.Output.Module != "output" || len(prepared.ArtifactLanes) != 1 {
t.Fatalf("PreparedPipeline = %#v, want explicit resolved components", prepared)
}
}
func TestPrepareFailuresOccurBeforeInputParse(t *testing.T) {
tests := []struct {
name string
deps ModuleDependencies
configure func(*constructionFailure)
want string
wantBuilt []string
}{
{
name: "missing required llm dependency",
configure: func(failure *constructionFailure) {
failure.requireExtractorLLM = true
},
want: `lane "artifact" extract module "extract"`,
wantBuilt: []string{"input", "chunk", "validator", "extract"},
},
{
name: "late output construction",
configure: func(failure *constructionFailure) {
failure.output = errors.New("output unavailable")
},
want: `output module "output"`,
wantBuilt: []string{"input", "chunk", "validator", "extract", "validator", "merge", "validator", "normalize", "validator", "output"},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
failure := &constructionFailure{}
test.configure(failure)
var built []string
registries, input := constructionRegistries(t, &built, failure)
resolved, err := ResolvePipeline(constructionProfile(), ResolveOptions{}, registries.catalog())
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
_, err = Prepare(resolved, registries, test.deps)
if err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("Prepare() error = %v, want substring %q", err, test.want)
}
if len(input.requests) != 0 {
t.Fatalf("input Parse calls = %d, want zero", len(input.requests))
}
if !reflect.DeepEqual(built, test.wantBuilt) {
t.Fatalf("construction order = %#v, want %#v", built, test.wantBuilt)
}
})
}
}
type constructionFailure struct {
requireExtractorLLM bool
output error
}
func constructionProfile() PipelineProfile {
validators := ValidatorOverride{Set: true, Validators: []ModuleBinding{{Module: "configured"}}}
return PipelineProfile{
ID: "construction",
Input: Binding("input"),
Chunk: ModuleBinding{Module: "chunk", Validators: validators},
Artifacts: map[string]ArtifactLaneProfile{
"artifact": {
Extract: ModuleBinding{Module: "extract", Validators: validators},
Merge: ModuleBinding{Module: "merge", Validators: validators},
Normalize: ModuleBinding{Module: "normalize", Validators: validators},
},
},
Output: Binding("output"),
}
}
func constructionRegistries(t *testing.T, built *[]string, failure *constructionFailure) (Registries, *constructionInput) {
t.Helper()
if built == nil {
built = &[]string{}
}
if failure == nil {
failure = &constructionFailure{}
}
record := func(name string) { *built = append(*built, name) }
strict := func(options map[string]any) error { return RejectUnknownOptions(options, "known") }
input := &constructionInput{key: "input"}
registries := Registries{
Inputs: NewInputAdapterRegistry(), Chunkers: NewChunkerRegistry(), ArtifactCodecs: NewArtifactCodecRegistry(),
Extractors: NewExtractorRegistry(), Mergers: NewMergerRegistry(), Normalizers: NewNormalizerRegistry(),
Validators: NewValidatorRegistry(), ValidatorChains: NewValidatorChainRegistry(), Outputs: NewOutputEncoderRegistry(),
}
if err := RegisterArtifactCodec(registries.ArtifactCodecs, notesCodec()); err != nil {
t.Fatal(err)
}
if err := registries.Inputs.RegisterBuilderWithSpec(defaultModuleSpec("input", StageInput), strict, func(BuildRequest) (contracts.InputAdapter, error) {
record("input")
return input, nil
}); err != nil {
t.Fatal(err)
}
if err := registries.Chunkers.RegisterBuilderWithSpec(defaultModuleSpec("chunk", StageChunk), strict, func(BuildRequest) (contracts.Chunker, error) {
record("chunk")
return &typedTestChunker{key: "chunk"}, nil
}); err != nil {
t.Fatal(err)
}
extractSpec := defaultModuleSpec("extract", StageExtract)
extractSpec.ArtifactKind = "test/notes"
if err := RegisterExtractorBuilder(registries.Extractors, extractSpec, strict, func(request BuildRequest) (contracts.Extractor[codecNotes], error) {
record("extract")
if failure.requireExtractorLLM && request.Dependencies.LLM == nil {
return nil, errors.New("structured LLM client is required")
}
return typedTestExtractor[codecNotes]{key: "extract"}, nil
}); err != nil {
t.Fatal(err)
}
mergeSpec := defaultModuleSpec("merge", StageMerge)
mergeSpec.ArtifactKind = "test/notes"
if err := RegisterMergerBuilder(registries.Mergers, mergeSpec, strict, func(BuildRequest) (contracts.Merger[codecNotes], error) {
record("merge")
return typedTestMerger[codecNotes]{key: "merge"}, nil
}); err != nil {
t.Fatal(err)
}
normalizeSpec := defaultModuleSpec("normalize", StageNormalize)
normalizeSpec.ArtifactKind = "test/notes"
if err := RegisterNormalizerBuilder(registries.Normalizers, normalizeSpec, strict, func(BuildRequest) (contracts.Normalizer[codecNotes], error) {
record("normalize")
return typedTestNormalizer[codecNotes]{key: "normalize"}, nil
}); err != nil {
t.Fatal(err)
}
validatorSpec := ValidatorSpec{Key: "configured", ExecutionClass: contracts.ExecutionClassDeterministic}
if err := RegisterChunkValidatorBuilder(registries.Validators, validatorSpec, strict, func(BuildRequest) (contracts.ChunkValidator, error) {
record("validator")
return typedTestChunkValidator{key: "configured"}, nil
}); err != nil {
t.Fatal(err)
}
if err := RegisterTypedValidatorBuilder(registries.Validators, "test/notes", validatorSpec, strict, func(BuildRequest) (contracts.TypedValidator[codecNotes], error) {
record("validator")
return typedTestValidator[codecNotes]{key: "configured"}, nil
}); err != nil {
t.Fatal(err)
}
if err := registries.Outputs.RegisterBuilderWithSpec(defaultModuleSpec("output", StageOutput), strict, func(BuildRequest) (contracts.OutputEncoder, error) {
record("output")
if failure.output != nil {
return nil, failure.output
}
return &typedTestOutput{key: "output"}, nil
}); err != nil {
t.Fatal(err)
}
return registries, input
}
type constructionInput struct {
key string
requests []contracts.ParseRequest
}
func (input *constructionInput) Key() string { return input.key }
func (input *constructionInput) Parse(_ context.Context, request contracts.ParseRequest) (*source.SourceDocument, error) {
input.requests = append(input.requests, request)
return typedTestDocument(), nil
}

View File

@@ -0,0 +1,348 @@
package pipeline
import (
"fmt"
"reflect"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
// PreparedPipeline owns the constructed, run-local implementation set for one
// resolved pipeline. Its implementation values are private so execution cannot
// replace or reconfigure them after preparation.
type PreparedPipeline struct {
Input ModuleBinding
Chunk ModuleBinding
ArtifactLanes []PreparedArtifactLane
Output ModuleBinding
resolved ResolvedPipeline
dependencies ModuleDependencies
input contracts.InputAdapter
chunker contracts.Chunker
chunkValidators preparedValidatorChain
lanes []preparedLaneExecutor
output contracts.OutputEncoder
}
type PreparedArtifactLane struct {
Resolved ResolvedArtifactLane
}
type preparedLaneExecutor struct {
resolved ResolvedArtifactLane
typed *preparedTypedLane
extractValidators preparedValidatorChain
mergeValidators preparedValidatorChain
normalizeValidators preparedValidatorChain
}
type preparedTypedLane struct {
extractor any
merger any
normalizer any
extract typedExtractOperation
merge typedMergeOperation
normalize typedNormalizeOperation
codec artifactCodecEntry
}
type preparedValidatorChain struct {
resolved ResolvedValidatorChain
validators []preparedValidator
}
type preparedValidator struct {
resolved ResolvedValidator
typed any
typedValidate typedValidateOperation
chunk contracts.ChunkValidator
serialized contracts.SerializedValidator
}
// Prepare validates all configured options and constructs every selected
// module and validator before any operation method can run.
func Prepare(resolved ResolvedPipeline, registries Registries, deps ModuleDependencies) (*PreparedPipeline, error) {
if err := validateResolvedPipeline(resolved); err != nil {
return nil, err
}
if err := validateRegistrySet(resolved, registries); err != nil {
return nil, err
}
stable := cloneResolvedPipeline(resolved)
prepared := &PreparedPipeline{
Input: cloneModuleBinding(stable.Input),
Chunk: cloneModuleBinding(stable.Chunk),
Output: cloneModuleBinding(stable.Output),
resolved: stable,
dependencies: deps,
}
request := func(binding ModuleBinding) BuildRequest {
return BuildRequest{Dependencies: deps, Options: cloneOptions(binding.Options)}
}
input, err := registries.Inputs.BuildWithRequest(stable.Input.Module, request(stable.Input))
if err != nil {
return nil, constructionError(stable.ID, "", StageInput, stable.Input.Module, "", err)
}
prepared.input = input
chunker, err := registries.Chunkers.BuildWithRequest(stable.Chunk.Module, request(stable.Chunk))
if err != nil {
return nil, constructionError(stable.ID, "", StageChunk, stable.Chunk.Module, "", err)
}
prepared.chunker = chunker
prepared.chunkValidators, err = prepareValidatorChain(stable, registries, deps, StageChunk, "", stable.Chunk.Module)
if err != nil {
return nil, err
}
prepared.ArtifactLanes = make([]PreparedArtifactLane, 0, len(stable.ArtifactLanes))
prepared.lanes = make([]preparedLaneExecutor, 0, len(stable.ArtifactLanes))
for _, lane := range stable.ArtifactLanes {
executor, err := prepareLane(stable, lane, registries, deps)
if err != nil {
return nil, err
}
prepared.ArtifactLanes = append(prepared.ArtifactLanes, PreparedArtifactLane{Resolved: cloneResolvedArtifactLane(lane)})
prepared.lanes = append(prepared.lanes, executor)
}
output, err := registries.Outputs.BuildWithRequest(stable.Output.Module, request(stable.Output))
if err != nil {
return nil, constructionError(stable.ID, "", StageOutput, stable.Output.Module, "", err)
}
prepared.output = output
return prepared, nil
}
func prepareLane(pipeline ResolvedPipeline, lane ResolvedArtifactLane, registries Registries, deps ModuleDependencies) (preparedLaneExecutor, error) {
executor := preparedLaneExecutor{resolved: cloneResolvedArtifactLane(lane)}
request := func(binding ModuleBinding) BuildRequest {
return BuildRequest{Dependencies: deps, Options: cloneOptions(binding.Options)}
}
extractEntry, ok := registries.Extractors.typedEntry(lane.Extract.Module)
if !ok {
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageExtract, lane.Extract.Module, "", fmt.Errorf("typed construction entry is not registered"))
}
module, err := buildErasedModule(extractEntry.builder, request(lane.Extract), lane.Extract.Module, "extractor")
if err != nil {
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageExtract, lane.Extract.Module, "", err)
}
codec, _, codecErr := registries.ArtifactCodecs.entry(lane.ArtifactKind)
if codecErr != nil {
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageExtract, lane.Extract.Module, "", codecErr)
}
executor.typed = &preparedTypedLane{extractor: module, extract: extractEntry.extract, codec: codec}
executor.extractValidators, err = prepareValidatorChain(pipeline, registries, deps, StageExtract, lane.ID, lane.Extract.Module)
if err != nil {
return preparedLaneExecutor{}, err
}
mergeEntry, ok := registries.Mergers.typedEntry(lane.Merge.Module, lane.ArtifactKind)
if !ok {
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageMerge, lane.Merge.Module, "", fmt.Errorf("typed construction entry is not registered"))
}
module, err = buildErasedModule(mergeEntry.builder, request(lane.Merge), lane.Merge.Module, "merger")
if err != nil {
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageMerge, lane.Merge.Module, "", err)
}
executor.typed.merger = module
executor.typed.merge = mergeEntry.merge
executor.mergeValidators, err = prepareValidatorChain(pipeline, registries, deps, StageMerge, lane.ID, lane.Merge.Module)
if err != nil {
return preparedLaneExecutor{}, err
}
normalizeEntry, ok := registries.Normalizers.typedEntry(lane.Normalize.Module, lane.ArtifactKind)
if !ok {
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageNormalize, lane.Normalize.Module, "", fmt.Errorf("typed construction entry is not registered"))
}
module, err = buildErasedModule(normalizeEntry.builder, request(lane.Normalize), lane.Normalize.Module, "normalizer")
if err != nil {
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageNormalize, lane.Normalize.Module, "", err)
}
executor.typed.normalizer = module
executor.typed.normalize = normalizeEntry.normalize
executor.normalizeValidators, err = prepareValidatorChain(pipeline, registries, deps, StageNormalize, lane.ID, lane.Normalize.Module)
if err != nil {
return preparedLaneExecutor{}, err
}
return executor, nil
}
func prepareValidatorChain(pipeline ResolvedPipeline, registries Registries, deps ModuleDependencies, stage ModuleStage, laneID, moduleKey string) (preparedValidatorChain, error) {
resolved := resolvedValidatorChain(stage, laneID, moduleKey, pipeline.ValidatorChains)
prepared := preparedValidatorChain{resolved: resolved}
for _, validator := range resolved.Validators {
request := BuildRequest{Dependencies: deps, Options: cloneOptions(validator.Binding.Options)}
built, err := buildPreparedValidator(registries.Validators, validator, request)
if err != nil {
return preparedValidatorChain{}, constructionError(pipeline.ID, laneID, stage, moduleKey, validator.Binding.Module, err)
}
prepared.validators = append(prepared.validators, built)
}
return prepared, nil
}
func buildPreparedValidator(registry *ValidatorRegistry, resolved ResolvedValidator, request BuildRequest) (preparedValidator, error) {
prepared := preparedValidator{resolved: resolved}
key := resolved.Binding.Module
var implementation any
var err error
switch resolved.Target {
case ValidatorTargetTyped:
entry, ok := registry.typedEntry(key, resolved.ArtifactKind)
if !ok {
return preparedValidator{}, fmt.Errorf("typed construction entry is not registered")
}
implementation, err = entry.builder(cloneBuildRequest(request))
prepared.typed = implementation
prepared.typedValidate = entry.validate
case ValidatorTargetChunk:
entry, ok := registry.chunkEntry(key)
if !ok {
return preparedValidator{}, fmt.Errorf("chunk construction entry is not registered")
}
prepared.chunk, err = entry.builder(cloneBuildRequest(request))
implementation = prepared.chunk
case ValidatorTargetSerialized:
entry, ok := registry.serializedEntry(key)
if !ok {
return preparedValidator{}, fmt.Errorf("serialized construction entry is not registered")
}
prepared.serialized, err = entry.builder(cloneBuildRequest(request))
implementation = prepared.serialized
default:
return preparedValidator{}, fmt.Errorf("validator construction target %q is not supported", resolved.Target)
}
if err != nil {
return preparedValidator{}, err
}
if isNilImplementation(implementation) {
return preparedValidator{}, fmt.Errorf("validator %q builder returned nil", key)
}
identity, ok := implementation.(interface {
Name() string
ExecutionClass() contracts.ExecutionClass
})
if !ok {
return preparedValidator{}, fmt.Errorf("validator %q builder returned incompatible implementation %T", key, implementation)
}
if identity.Name() != key {
return preparedValidator{}, fmt.Errorf("validator %q returned name %q", key, identity.Name())
}
if identity.ExecutionClass() != resolved.ExecutionClass {
return preparedValidator{}, fmt.Errorf("validator %q returned execution class %q, want %q", key, identity.ExecutionClass(), resolved.ExecutionClass)
}
return prepared, nil
}
func buildErasedModule(builder func(BuildRequest) (any, error), request BuildRequest, key, kind string) (any, error) {
implementation, err := builder(cloneBuildRequest(request))
if err != nil {
return nil, err
}
if isNilImplementation(implementation) {
return nil, fmt.Errorf("%s %q builder returned nil", kind, key)
}
identity, ok := implementation.(interface{ Key() string })
if !ok {
return nil, fmt.Errorf("%s %q builder returned incompatible implementation %T", kind, key, implementation)
}
if identity.Key() != key {
return nil, fmt.Errorf("%s %q returned key %q", kind, key, identity.Key())
}
return implementation, nil
}
func isNilImplementation(value any) bool {
if value == nil {
return true
}
reflected := reflect.ValueOf(value)
switch reflected.Kind() {
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:
return reflected.IsNil()
default:
return false
}
}
func constructionError(pipelineID, laneID string, stage ModuleStage, moduleKey, validatorKey string, cause error) error {
scope := fmt.Sprintf("pipeline %q %s module %q", pipelineID, stage, moduleKey)
if laneID != "" {
scope = fmt.Sprintf("pipeline %q lane %q %s module %q", pipelineID, laneID, stage, moduleKey)
}
if strings.TrimSpace(validatorKey) != "" {
scope += fmt.Sprintf(" validator %q", validatorKey)
}
return fmt.Errorf("prepare %s: %w", scope, cause)
}
func (registries Registries) catalog() ModuleCatalog {
return ModuleCatalog{
Inputs: registries.Inputs, Chunkers: registries.Chunkers, ArtifactCodecs: registries.ArtifactCodecs,
Extractors: registries.Extractors, Mergers: registries.Mergers, Normalizers: registries.Normalizers,
Validators: registries.Validators, ValidatorChains: registries.ValidatorChains, Outputs: registries.Outputs,
}
}
func validateRegistrySet(resolved ResolvedPipeline, registries Registries) error {
if registries.Inputs == nil {
return fmt.Errorf("input registry must not be nil")
}
if registries.Chunkers == nil {
return fmt.Errorf("chunker registry must not be nil")
}
if registries.Extractors == nil {
return fmt.Errorf("extractor registry must not be nil")
}
if registries.Mergers == nil {
return fmt.Errorf("merger registry must not be nil")
}
if registries.Normalizers == nil {
return fmt.Errorf("normalizer registry must not be nil")
}
if registries.Validators == nil {
for _, chain := range resolved.ValidatorChains {
if len(chain.Validators) > 0 {
return fmt.Errorf("validator registry must not be nil")
}
}
}
if registries.Outputs == nil {
return fmt.Errorf("output encoder registry must not be nil")
}
return nil
}
func cloneResolvedPipeline(in ResolvedPipeline) ResolvedPipeline {
out := in
out.Input = cloneModuleBinding(in.Input)
out.Chunk = cloneModuleBinding(in.Chunk)
out.Output = cloneModuleBinding(in.Output)
out.ChunkReferences = CloneReferenceTarget(in.ChunkReferences)
out.ValidatorChains = cloneResolvedValidatorChains(in.ValidatorChains)
if len(in.ArtifactLanes) > 0 {
out.ArtifactLanes = make([]ResolvedArtifactLane, len(in.ArtifactLanes))
for i, lane := range in.ArtifactLanes {
out.ArtifactLanes[i] = cloneResolvedArtifactLane(lane)
}
}
return out
}
func cloneResolvedArtifactLane(in ResolvedArtifactLane) ResolvedArtifactLane {
out := in
out.Extract = cloneModuleBinding(in.Extract)
out.Merge = cloneModuleBinding(in.Merge)
out.Normalize = cloneModuleBinding(in.Normalize)
out.Validators = cloneModuleBindings(in.Validators)
out.ExtractReferences = CloneReferenceTarget(in.ExtractReferences)
out.MergeReferences = CloneReferenceTarget(in.MergeReferences)
out.NormalizeReferences = CloneReferenceTarget(in.NormalizeReferences)
return out
}

View File

@@ -5,6 +5,7 @@ import (
"encoding/hex"
"encoding/json"
"fmt"
"reflect"
"sort"
"strings"
@@ -102,14 +103,19 @@ type ResolvedReferenceTarget struct {
}
type ResolvedArtifactLane struct {
ID string
Extract ModuleBinding
Merge ModuleBinding
Normalize ModuleBinding
Validators []ModuleBinding
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
ID string
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
ArtifactSchemaName string `json:"artifact_schema_name,omitempty"`
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
Extract ModuleBinding
Merge ModuleBinding
Normalize ModuleBinding
Validators []ModuleBinding
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
}
type ResolvedValidatorChain struct {
@@ -122,6 +128,8 @@ type ResolvedValidatorChain struct {
type ResolvedValidator struct {
Binding ModuleBinding `json:"binding"`
ExecutionClass contracts.ExecutionClass `json:"execution_class"`
Target ValidatorTarget `json:"target,omitempty"`
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
}
type ResolvedPipeline struct {
@@ -138,6 +146,7 @@ type ResolvedPipeline struct {
type ModuleCatalog struct {
Inputs *InputAdapterRegistry
Chunkers *ChunkerRegistry
ArtifactCodecs *ArtifactCodecRegistry
Extractors *ExtractorRegistry
Mergers *MergerRegistry
Normalizers *NormalizerRegistry
@@ -215,7 +224,7 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
Output: resolveBinding(profile.Output, DefaultOutputModule),
}
chunkValidatorChain, err := resolveValidatorChain(pipelineID, "", StageChunk, chunk.Module, chunk.Validators, catalog)
chunkValidatorChain, err := resolveValidatorChain(pipelineID, "", StageChunk, chunk.Module, chunk.Validators, "", nil, catalog)
if err != nil {
return ResolvedPipeline{}, err
}
@@ -240,6 +249,9 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
if missing, ok := outputCapabilities.missing(outputSpec.Requires); ok {
return ResolvedPipeline{}, capabilityError(pipelineID, "", StageOutput, resolved.Output.Module, missing)
}
if err := validateResolvedOptions(resolved, catalog); err != nil {
return ResolvedPipeline{}, err
}
digest, err := resolvedPipelineDigest(resolved)
if err != nil {
@@ -278,6 +290,10 @@ func resolveArtifactLane(
if missing, ok := capabilities.missing(extractSpec.Requires); ok {
return ResolvedArtifactLane{}, nil, nil, capabilityError(pipelineID, laneID, StageExtract, lane.Extract.Module, missing)
}
artifactType, err := resolveArtifactIdentity(pipelineID, laneID, &lane, extractSpec, catalog)
if err != nil {
return ResolvedArtifactLane{}, nil, nil, err
}
extractReferences := mergeReferenceMaps(profile.References, lane.Extract.References)
references, err := resolveReferenceTargetBindings(referenceResolutionTarget{
PipelineID: pipelineID,
@@ -295,7 +311,7 @@ func resolveArtifactLane(
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
capabilities.add(extractSpec.Provides...)
mergeSpec, err := mergerSpec(catalog, lane.Merge.Module)
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
if err != nil {
return ResolvedArtifactLane{}, nil, nil, moduleLookupError(pipelineID, laneID, StageMerge, lane.Merge.Module, err)
}
@@ -318,7 +334,7 @@ func resolveArtifactLane(
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
capabilities.add(mergeSpec.Provides...)
normalizeSpec, err := normalizerSpec(catalog, lane.Normalize.Module)
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
if err != nil {
return ResolvedArtifactLane{}, nil, nil, moduleLookupError(pipelineID, laneID, StageNormalize, lane.Normalize.Module, err)
}
@@ -345,15 +361,15 @@ func resolveArtifactLane(
return ResolvedArtifactLane{}, nil, nil, configuredValidatorsError(pipelineID, laneID)
}
extractValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageExtract, lane.Extract.Module, lane.Extract.Validators, catalog)
extractValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageExtract, lane.Extract.Module, lane.Extract.Validators, lane.ArtifactKind, artifactType, catalog)
if err != nil {
return ResolvedArtifactLane{}, nil, nil, err
}
mergeValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageMerge, lane.Merge.Module, lane.Merge.Validators, catalog)
mergeValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageMerge, lane.Merge.Module, lane.Merge.Validators, lane.ArtifactKind, artifactType, catalog)
if err != nil {
return ResolvedArtifactLane{}, nil, nil, err
}
normalizeValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageNormalize, lane.Normalize.Module, lane.Normalize.Validators, catalog)
normalizeValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageNormalize, lane.Normalize.Module, lane.Normalize.Validators, lane.ArtifactKind, artifactType, catalog)
if err != nil {
return ResolvedArtifactLane{}, nil, nil, err
}
@@ -366,7 +382,131 @@ func configuredValidatorsError(pipelineID string, laneID string) error {
return fmt.Errorf("pipeline %q lane %q validators are not supported at artifact lane level; use extract.validators, merge.validators, or normalize.validators", pipelineID, laneID)
}
func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage, module string, override ValidatorOverride, catalog ModuleCatalog) (ResolvedValidatorChain, error) {
func resolveArtifactIdentity(pipelineID, laneID string, lane *ResolvedArtifactLane, extractSpec ModuleSpec, catalog ModuleCatalog) (reflect.Type, error) {
if extractSpec.ArtifactKind == "" {
return nil, fmt.Errorf("pipeline %q lane %q extract module %q does not declare an artifact kind", pipelineID, laneID, lane.Extract.Module)
}
if catalog.Extractors == nil {
return nil, fmt.Errorf("pipeline %q lane %q extractor registry must not be nil", pipelineID, laneID)
}
extractor, ok := catalog.Extractors.typedEntry(lane.Extract.Module)
if !ok {
return nil, fmt.Errorf("pipeline %q lane %q extract module %q declares artifact kind %q without a typed registration", pipelineID, laneID, lane.Extract.Module, extractSpec.ArtifactKind)
}
if catalog.ArtifactCodecs == nil {
return nil, fmt.Errorf("pipeline %q lane %q artifact codec registry must not be nil for kind %q", pipelineID, laneID, extractSpec.ArtifactKind)
}
codecSpec, ok := catalog.ArtifactCodecs.Spec(extractSpec.ArtifactKind)
if !ok {
return nil, fmt.Errorf("pipeline %q lane %q artifact codec %q is not registered", pipelineID, laneID, extractSpec.ArtifactKind)
}
codecType, ok := catalog.ArtifactCodecs.valueType(extractSpec.ArtifactKind)
if !ok {
return nil, fmt.Errorf("pipeline %q lane %q artifact codec %q has no Go type", pipelineID, laneID, extractSpec.ArtifactKind)
}
if extractor.valueType != codecType {
return nil, artifactTypeMismatchError(pipelineID, laneID, StageExtract, lane.Extract.Module, extractSpec.ArtifactKind, codecType, extractor.valueType)
}
lane.ArtifactKind = codecSpec.Kind
lane.ArtifactSchemaID = codecSpec.Schema.ID
lane.ArtifactSchemaName = codecSpec.Schema.Name
lane.ArtifactSchemaVersion = codecSpec.Schema.Version
lane.ArtifactSchemaDigest = codecSpec.SchemaDigest
return codecType, nil
}
func mergerSpecForArtifact(catalog ModuleCatalog, key string, kind contracts.ArtifactKind, expectedType reflect.Type) (ModuleSpec, error) {
if kind == "" {
return mergerSpec(catalog, key)
}
if catalog.Mergers == nil {
return ModuleSpec{}, fmt.Errorf("module %q is not registered", key)
}
entry, ok := catalog.Mergers.typedEntry(key, kind)
if !ok {
return ModuleSpec{}, missingArtifactVariantError("merger", key, kind, catalog.Mergers.registeredKinds(key))
}
if entry.valueType != expectedType {
return ModuleSpec{}, fmt.Errorf("artifact kind %q requires Go type %s, but merger %q variant uses %s", kind, typeName(expectedType), key, typeName(entry.valueType))
}
return cloneModuleSpec(entry.spec), nil
}
func normalizerSpecForArtifact(catalog ModuleCatalog, key string, kind contracts.ArtifactKind, expectedType reflect.Type) (ModuleSpec, error) {
if kind == "" {
return normalizerSpec(catalog, key)
}
if catalog.Normalizers == nil {
return ModuleSpec{}, fmt.Errorf("module %q is not registered", key)
}
entry, ok := catalog.Normalizers.typedEntry(key, kind)
if !ok {
return ModuleSpec{}, missingArtifactVariantError("normalizer", key, kind, catalog.Normalizers.registeredKinds(key))
}
if entry.valueType != expectedType {
return ModuleSpec{}, fmt.Errorf("artifact kind %q requires Go type %s, but normalizer %q variant uses %s", kind, typeName(expectedType), key, typeName(entry.valueType))
}
return cloneModuleSpec(entry.spec), nil
}
func validatorSpecForTarget(registry *ValidatorRegistry, stage ModuleStage, key string, kind contracts.ArtifactKind, expectedType reflect.Type) (ValidatorSpec, ValidatorTarget, error) {
key = strings.TrimSpace(key)
if stage == StageChunk {
if entry, ok := registry.chunkEntry(key); ok {
return entry.spec, ValidatorTargetChunk, nil
}
if entry, ok := registry.serializedEntry(key); ok && entry.spec.SupportsChunks {
return entry.spec.ValidatorSpec, ValidatorTargetSerialized, nil
}
if spec, ok := registry.Spec(key); ok {
return spec, "", nil
}
return ValidatorSpec{}, "", fmt.Errorf("references unknown validator %q for chunk target", key)
}
if kind == "" {
if spec, ok := registry.Spec(key); ok {
return spec, "", nil
}
return ValidatorSpec{}, "", fmt.Errorf("references unknown validator %q without an artifact kind", key)
}
if entry, ok := registry.typedEntry(key, kind); ok {
if entry.valueType != expectedType {
return ValidatorSpec{}, "", fmt.Errorf("artifact kind %q requires Go type %s, but validator %q variant uses %s", kind, typeName(expectedType), key, typeName(entry.valueType))
}
return entry.spec, ValidatorTargetTyped, nil
}
if entry, ok := registry.serializedEntry(key); ok && entry.spec.SupportsArtifacts {
return entry.spec.ValidatorSpec, ValidatorTargetSerialized, nil
}
if _, ok := registry.Spec(key); !ok {
return ValidatorSpec{}, "", fmt.Errorf("references unknown validator %q for artifact kind %q", key, kind)
}
return ValidatorSpec{}, "", missingArtifactVariantError("validator", key, kind, registry.registeredTypedKinds(key))
}
func missingArtifactVariantError(moduleType, key string, kind contracts.ArtifactKind, registered []contracts.ArtifactKind) error {
if len(registered) == 0 {
return fmt.Errorf("%s %q has no typed variant for artifact kind %q", moduleType, key, kind)
}
values := make([]string, len(registered))
for i, value := range registered {
values[i] = string(value)
}
return fmt.Errorf("%s %q has no typed variant for artifact kind %q; registered kinds: %s", moduleType, key, kind, strings.Join(values, ", "))
}
func artifactTypeMismatchError(pipelineID, laneID string, stage ModuleStage, module string, kind contracts.ArtifactKind, expected, actual reflect.Type) error {
return fmt.Errorf("pipeline %q lane %q %s module %q artifact kind %q requires Go type %s, got %s", pipelineID, laneID, stage, module, kind, typeName(expected), typeName(actual))
}
func typeName(value reflect.Type) string {
if value == nil {
return "<nil>"
}
return value.String()
}
func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage, module string, override ValidatorOverride, artifactKind contracts.ArtifactKind, artifactType reflect.Type, catalog ModuleCatalog) (ResolvedValidatorChain, error) {
chain := ResolvedValidatorChain{
Stage: stage,
LaneID: strings.TrimSpace(laneID),
@@ -395,9 +535,9 @@ func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage,
}
chain.Validators = make([]ResolvedValidator, 0, len(bindings))
for _, validator := range bindings {
spec, ok := catalog.Validators.Spec(validator.Module)
if !ok {
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q references unknown validator %q", pipelineID, stage, chain.ModuleKey, validator.Module)
spec, target, err := validatorSpecForTarget(catalog.Validators, stage, validator.Module, artifactKind, artifactType)
if err != nil {
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q: %w", pipelineID, stage, chain.ModuleKey, err)
}
if strings.TrimSpace(validator.LLMProfile) != "" && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q assigns llm_profile to deterministic validator %q", pipelineID, stage, chain.ModuleKey, validator.Module)
@@ -405,6 +545,8 @@ func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage,
chain.Validators = append(chain.Validators, ResolvedValidator{
Binding: cloneModuleBinding(validator),
ExecutionClass: spec.ExecutionClass,
Target: target,
ArtifactKind: artifactKind,
})
}
return chain, nil
@@ -435,6 +577,8 @@ func cloneResolvedValidators(validators []ResolvedValidator) []ResolvedValidator
out[i] = ResolvedValidator{
Binding: cloneModuleBinding(validator.Binding),
ExecutionClass: validator.ExecutionClass,
Target: validator.Target,
ArtifactKind: validator.ArtifactKind,
}
}
return out
@@ -497,7 +641,14 @@ func validatePipelineReferenceDefaults(
}
merge := resolveBinding(laneProfile.Merge, DefaultMergeModule)
mergeSpec, err := mergerSpec(catalog, merge.Module)
var artifactType reflect.Type
artifactKind := extractSpec.ArtifactKind
if artifactKind != "" && catalog.Extractors != nil {
if entry, ok := catalog.Extractors.typedEntry(extract.Module); ok {
artifactType = entry.valueType
}
}
mergeSpec, err := mergerSpecForArtifact(catalog, merge.Module, artifactKind, artifactType)
if err != nil {
return moduleLookupError(pipelineID, laneID, StageMerge, merge.Module, err)
}
@@ -506,7 +657,7 @@ func validatePipelineReferenceDefaults(
}
normalize := resolveBinding(laneProfile.Normalize, DefaultNormalizeModule)
normalizeSpec, err := normalizerSpec(catalog, normalize.Module)
normalizeSpec, err := normalizerSpecForArtifact(catalog, normalize.Module, artifactKind, artifactType)
if err != nil {
return moduleLookupError(pipelineID, laneID, StageNormalize, normalize.Module, err)
}

View File

@@ -766,20 +766,22 @@ func TestResolvePipelineUsesReferenceSlotsFromSpecWithoutConstructingExtractor(t
profile := baselineProfile()
profile.References = map[string]string{"roster": "./roster.yml"}
catalog := emptyProfileCatalog()
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, notesCodec())
for _, spec := range defaultProfileSpecs() {
if spec.Key != "event-extractor" {
registerProfileSpecs(t, catalog, spec)
}
}
if err := catalog.Extractors.RegisterWithSpec(ModuleSpec{
Key: "event-extractor",
Stage: StageExtract,
Requires: []string{"chunk"},
Provides: []string{"candidate"},
if err := RegisterExtractor[codecNotes](catalog.Extractors, ModuleSpec{
Key: "event-extractor",
Stage: StageExtract,
ArtifactKind: "test/notes",
Requires: []string{"chunk"},
Provides: []string{"candidate"},
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "roster", Required: true},
},
}, func() (contracts.Extractor, error) {
}, func() (contracts.Extractor[codecNotes], error) {
return nil, errors.New("constructor should not run")
}); err != nil {
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
@@ -1177,6 +1179,7 @@ func newProfileCatalog(t *testing.T) ModuleCatalog {
t.Helper()
catalog := emptyProfileCatalog()
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, notesCodec())
registerProfileSpecs(t, catalog, defaultProfileSpecs()...)
return catalog
}
@@ -1192,6 +1195,9 @@ func newProfileCatalogWithOverrides(t *testing.T, overrides ...ModuleSpec) Modul
specs := defaultProfileSpecs()
for _, override := range overrides {
if override.ArtifactKind == "" && (override.Stage == StageExtract || override.Stage == StageMerge || override.Stage == StageNormalize) {
override.ArtifactKind = "test/notes"
}
replaced := false
for index, spec := range specs {
if spec.Stage == override.Stage && spec.Key == override.Key {
@@ -1206,6 +1212,7 @@ func newProfileCatalogWithOverrides(t *testing.T, overrides ...ModuleSpec) Modul
}
catalog := emptyProfileCatalog()
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, notesCodec())
registerProfileSpecs(t, catalog, specs...)
return catalog
}
@@ -1214,6 +1221,7 @@ func emptyProfileCatalog() ModuleCatalog {
return ModuleCatalog{
Inputs: NewInputAdapterRegistry(),
Chunkers: NewChunkerRegistry(),
ArtifactCodecs: NewArtifactCodecRegistry(),
Extractors: NewExtractorRegistry(),
Mergers: NewMergerRegistry(),
Normalizers: NewNormalizerRegistry(),
@@ -1227,10 +1235,10 @@ func defaultProfileSpecs() []ModuleSpec {
return []ModuleSpec{
ModuleSpec{Key: "text", Stage: StageInput, Provides: []string{"source"}},
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}},
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "note-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}},
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}},
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
ModuleSpec{Key: "appendorder", Stage: StageMerge, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
ModuleSpec{Key: "noop", Stage: StageNormalize, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
ModuleSpec{Key: "grounded", Stage: StageValidate, Requires: []string{"normalized"}, Provides: []string{"validated"}},
ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
}
@@ -1240,30 +1248,40 @@ func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSp
t.Helper()
for _, spec := range specs {
if spec.ArtifactKind == "" && (spec.Stage == StageExtract || spec.Stage == StageMerge || spec.Stage == StageNormalize) {
spec.ArtifactKind = "test/notes"
}
switch spec.Stage {
case StageInput:
if err := catalog.Inputs.RegisterWithSpec(spec, profileInputConstructor(spec.Key)); err != nil {
t.Fatalf("register input spec %#v: %v", spec, err)
}
case StageChunk:
if err := catalog.Chunkers.RegisterWithSpec(spec, profileChunkerConstructor(spec.Key)); err != nil {
validateOptions := func(options map[string]any) error { return RejectUnknownOptions(options, "a", "b", "size") }
if err := catalog.Chunkers.RegisterBuilderWithSpec(spec, validateOptions, func(BuildRequest) (contracts.Chunker, error) { return &typedTestChunker{key: spec.Key}, nil }); err != nil {
t.Fatalf("register chunk spec %#v: %v", spec, err)
}
case StageExtract:
if err := catalog.Extractors.RegisterWithSpec(spec, profileExtractorConstructor(spec.Key)); err != nil {
if err := RegisterExtractor(catalog.Extractors, spec, func() (contracts.Extractor[codecNotes], error) {
return typedTestExtractor[codecNotes]{key: spec.Key}, nil
}); err != nil {
t.Fatalf("register extractor spec %#v: %v", spec, err)
}
case StageMerge:
if err := catalog.Mergers.RegisterWithSpec(spec, profileMergerConstructor(spec.Key)); err != nil {
if err := RegisterMerger(catalog.Mergers, spec, func() (contracts.Merger[codecNotes], error) { return typedTestMerger[codecNotes]{key: spec.Key}, nil }); err != nil {
t.Fatalf("register merger spec %#v: %v", spec, err)
}
case StageNormalize:
if err := catalog.Normalizers.RegisterWithSpec(spec, profileNormalizerConstructor(spec.Key)); err != nil {
if err := RegisterNormalizer(catalog.Normalizers, spec, func() (contracts.Normalizer[codecNotes], error) {
return typedTestNormalizer[codecNotes]{key: spec.Key}, nil
}); err != nil {
t.Fatalf("register normalizer spec %#v: %v", spec, err)
}
case StageValidate:
validatorSpec := ValidatorSpec{Key: spec.Key, ExecutionClass: contracts.ExecutionClassDeterministic}
if err := catalog.Validators.RegisterWithSpec(validatorSpec, profileValidatorConstructor(spec.Key)); err != nil {
if err := RegisterTypedValidator(catalog.Validators, "test/notes", validatorSpec, func() (contracts.TypedValidator[codecNotes], error) {
return typedTestValidator[codecNotes]{key: spec.Key}, nil
}); err != nil {
t.Fatalf("register validator spec %#v: %v", spec, err)
}
case StageOutput:
@@ -1278,7 +1296,9 @@ func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSp
func registerProfileValidatorSpec(t *testing.T, catalog ModuleCatalog, spec ValidatorSpec) {
t.Helper()
if err := catalog.Validators.RegisterWithSpec(spec, profileValidatorConstructor(spec.Key)); err != nil {
if err := RegisterTypedValidator(catalog.Validators, "test/notes", spec, func() (contracts.TypedValidator[codecNotes], error) {
return typedTestValidator[codecNotes]{key: spec.Key}, nil
}); err != nil {
t.Fatalf("register validator spec %#v: %v", spec, err)
}
}
@@ -1301,36 +1321,6 @@ func (adapter profileInputAdapter) Parse(ctx context.Context, req contracts.Pars
return &source.SourceDocument{}, nil
}
func profileChunkerConstructor(key string) ChunkerConstructor {
return func() (contracts.Chunker, error) {
return registryChunker{key: key}, nil
}
}
func profileExtractorConstructor(key string) ExtractorConstructor {
return func() (contracts.Extractor, error) {
return registryFakeExtractor{key: key}, nil
}
}
func profileMergerConstructor(key string) MergerConstructor {
return func() (contracts.Merger, error) {
return registryMerger{key: key}, nil
}
}
func profileNormalizerConstructor(key string) NormalizerConstructor {
return func() (contracts.Normalizer, error) {
return registryNormalizer{key: key}, nil
}
}
func profileValidatorConstructor(key string) ValidatorConstructor {
return func() (contracts.Validator, error) {
return registryValidator{name: key}, nil
}
}
func profileOutputConstructor(key string) OutputEncoderConstructor {
return func() (contracts.OutputEncoder, error) {
return registryOutputEncoder{key: key}, nil

View File

@@ -0,0 +1,14 @@
package pipeline
import "gitea.maximumdirect.net/eric/notarius/internal/core/source"
func validSourceDocument() *source.SourceDocument {
return &source.SourceDocument{
ID: "source-1", Kind: "document", Format: "text/plain", Digest: "sha256:source",
Units: []source.SourceUnit{
{ID: 1, Kind: "unit", Text: "Source unit.", Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1}},
{ID: 2, Kind: "unit", Text: "Second source unit.", Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 2, EndUnitID: 2}},
{ID: 3, Kind: "unit", Text: "Third source unit.", Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 3, EndUnitID: 3}},
},
}
}

View File

@@ -1,261 +0,0 @@
package pipeline
import (
"context"
"reflect"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestRunnerUsesRegistries(t *testing.T) {
var built []string
var executed []string
registries := integrationRegistries(t, &built, &executed)
output, err := New(registries).Run(context.Background(), RunInput{
Pipeline: integrationPipeline(),
SourceID: "source-1",
RawInput: []byte("source text"),
})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
wantBuilt := []string{"input", "chunk", "extract-first", "merge", "normalize", "extract-second", "merge", "normalize", "output"}
if !reflect.DeepEqual(built, wantBuilt) {
t.Fatalf("built = %#v, want %#v", built, wantBuilt)
}
if !reflect.DeepEqual(executed, []string{"extract-first:chunk-0", "extract-second:chunk-0"}) {
t.Fatalf("executed = %#v, want extractor chunk execution", executed)
}
if got := normalizeOutputKeys(output.NormalizeOutputs); !reflect.DeepEqual(got, []string{"normalize", "normalize"}) {
t.Fatalf("normalize output keys = %#v, want one output from each lane", got)
}
if len(output.Rejected) != 0 {
t.Fatalf("len(Rejected) = %d, want none", len(output.Rejected))
}
}
func integrationRegistries(t *testing.T, built, executed *[]string) Registries {
t.Helper()
registries := Registries{
Inputs: NewInputAdapterRegistry(),
Chunkers: NewChunkerRegistry(),
Extractors: NewExtractorRegistry(),
Mergers: NewMergerRegistry(),
Normalizers: NewNormalizerRegistry(),
Outputs: NewOutputEncoderRegistry(),
}
if err := registries.Inputs.Register("input", func() (contracts.InputAdapter, error) {
*built = append(*built, "input")
return integrationInput{}, nil
}); err != nil {
t.Fatalf("register input: %v", err)
}
if err := registries.Chunkers.Register("chunk", func() (contracts.Chunker, error) {
*built = append(*built, "chunk")
return integrationChunker{}, nil
}); err != nil {
t.Fatalf("register chunker: %v", err)
}
registerIntegrationExtractor(t, registries.Extractors, "extract-first", built, executed)
registerIntegrationExtractor(t, registries.Extractors, "extract-second", built, executed)
if err := registries.Mergers.Register("merge", func() (contracts.Merger, error) {
*built = append(*built, "merge")
return integrationMerger{}, nil
}); err != nil {
t.Fatalf("register merger: %v", err)
}
if err := registries.Normalizers.Register("normalize", func() (contracts.Normalizer, error) {
*built = append(*built, "normalize")
return integrationNormalizer{}, nil
}); err != nil {
t.Fatalf("register normalizer: %v", err)
}
if err := registries.Outputs.Register("output", func() (contracts.OutputEncoder, error) {
*built = append(*built, "output")
return integrationOutput{}, nil
}); err != nil {
t.Fatalf("register output: %v", err)
}
return registries
}
func registerIntegrationExtractor(t *testing.T, registry *ExtractorRegistry, key string, built, executed *[]string) {
t.Helper()
if err := registry.Register(key, func() (contracts.Extractor, error) {
*built = append(*built, key)
return integrationExtractor{key: key, executed: executed}, nil
}); err != nil {
t.Fatalf("Register(%q) error = %v, want nil", key, err)
}
}
type integrationInput struct{}
func (input integrationInput) Key() string {
return "input"
}
func (input integrationInput) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
return integrationSourceDocument(), nil
}
type integrationChunker struct{}
func (chunker integrationChunker) Key() string {
return "chunk"
}
func (chunker integrationChunker) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (chunker integrationChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
return contracts.ChunkResult{
Chunks: []contracts.SourceChunk{
{
ID: "chunk-0",
SourceID: req.Source.ID,
Index: 0,
StartUnitID: req.Source.Units[0].ID,
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
Content: []byte(`{"units":[1]}`),
MediaType: "application/json",
Units: req.Source.Units,
},
},
}, nil
}
type integrationExtractor struct {
key string
executed *[]string
}
func (extractor integrationExtractor) Key() string {
return extractor.key
}
func (extractor integrationExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (extractor integrationExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
*extractor.executed = append(*extractor.executed, extractor.key+":"+req.Chunk.ID)
return contracts.ExtractionResult{
Output: contracts.ExtractOutput{
Schema: contracts.ResponseSchema{ID: "integration", Name: "integration", Version: "v1"},
Payload: contracts.RawPayload{
Content: []byte(`{"value":true}`),
MediaType: "application/json",
},
},
}, nil
}
type integrationNormalizer struct{}
type integrationMerger struct{}
func (merger integrationMerger) Key() string {
return "merge"
}
func (merger integrationMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
output := contracts.MergeOutput{
LaneID: req.LaneID,
Schema: contracts.ResponseSchema{ID: "integration", Name: "integration", Version: "v1"},
Payload: contracts.RawPayload{
Content: []byte(`{"merged":true}`),
MediaType: "application/json",
},
}
if len(req.ExtractOutputs) > 0 {
output.SourceID = req.ExtractOutputs[0].SourceID
output.Schema = req.ExtractOutputs[0].Schema
output.Payload = req.ExtractOutputs[0].Payload
}
return contracts.MergeResult{Output: output}, nil
}
func (normalizer integrationNormalizer) Key() string {
return "normalize"
}
func (normalizer integrationNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (normalizer integrationNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
return contracts.NormalizeResult{
Output: contracts.NormalizeOutput{
LaneID: req.LaneID,
SourceID: req.MergeOutput.SourceID,
Schema: req.MergeOutput.Schema,
Payload: req.MergeOutput.Payload,
},
}, nil
}
type integrationOutput struct{}
func (output integrationOutput) Key() string {
return "output"
}
func (output integrationOutput) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
return contracts.OutputResult{
Files: []contracts.OutputFile{
{Name: "output.json", ContentType: "application/json", Bytes: []byte(`{}`)},
},
}, nil
}
func integrationPipeline() ResolvedPipeline {
return ResolvedPipeline{
ID: "pipeline-1",
Digest: "sha256:pipeline",
Input: Binding("input"),
Chunk: Binding("chunk"),
ArtifactLanes: []ResolvedArtifactLane{
{
ID: "first",
Extract: Binding("extract-first"),
Merge: Binding("merge"),
Normalize: Binding("normalize"),
},
{
ID: "second",
Extract: Binding("extract-second"),
Merge: Binding("merge"),
Normalize: Binding("normalize"),
},
},
Output: Binding("output"),
}
}
func integrationSourceDocument() *source.SourceDocument {
return &source.SourceDocument{
ID: "source-1",
Kind: "document",
Format: "text/plain",
Digest: "sha256:abc123",
Units: []source.SourceUnit{
{ID: 1, Kind: "unit", Text: "Source unit."},
},
}
}
func normalizeOutputKeys(outputs []contracts.NormalizeOutput) []string {
keys := make([]string, 0, len(outputs))
for _, output := range outputs {
keys = append(keys, output.NormalizerKey)
}
return keys
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,408 @@
package pipeline
import (
"context"
"errors"
"fmt"
"reflect"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
frameworkllm "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
)
type countingProvider struct {
active atomic.Int32
maximum atomic.Int32
calls atomic.Int32
}
func (p *countingProvider) CompleteStructured(ctx context.Context, _ contracts.StructuredCompletionRequest, _ any) (contracts.StructuredCompletionResponse, error) {
p.calls.Add(1)
current := p.active.Add(1)
defer p.active.Add(-1)
for {
seen := p.maximum.Load()
if current <= seen || p.maximum.CompareAndSwap(seen, current) {
break
}
}
select {
case <-time.After(5 * time.Millisecond):
return contracts.StructuredCompletionResponse{}, nil
case <-ctx.Done():
return contracts.StructuredCompletionResponse{}, ctx.Err()
}
}
func preparedConcurrentPipeline(t *testing.T, chunkCount int) *PreparedPipeline {
t.Helper()
catalog := typedResolutionCatalog(t, completeTypedCatalogOptions())
resolved, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v", err)
}
prepared, err := Prepare(resolved, registriesFromModuleCatalog(catalog), ModuleDependencies{})
if err != nil {
t.Fatalf("Prepare() error = %v", err)
}
doc := typedTestDocument()
chunks := make([]source.Chunk, chunkCount)
for i := range chunks {
chunks[i] = source.Chunk{ID: fmt.Sprintf("chunk-%d", i+1), SourceID: doc.ID, Index: i, Ref: doc.Units[0].Ref, Content: []byte(fmt.Sprintf(`{"chunk":%d}`, i+1)), MediaType: "application/json", Units: []source.SourceUnit{doc.Units[0]}}
}
prepared.chunker = &typedTestChunker{key: "typed/chunk", chunks: chunks}
return prepared
}
func installExtractOperation(prepared *PreparedPipeline, laneIndex int, operation func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error)) {
prepared.lanes[laneIndex].typed.extract = func(ctx context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return operation(ctx, request)
}
}
func typedValueForLane(laneIndex int, chunkIndex int) any {
if laneIndex == 0 {
return codecNotes{Items: []string{fmt.Sprintf("chunk-%d", chunkIndex)}}
}
return codecScore{Value: chunkIndex}
}
func TestRunnerBoundsExtractJobsAndStabilizesReverseCompletion(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 3)
started := make(chan int, 6)
release := make([]chan struct{}, 6)
for i := range release {
release[i] = make(chan struct{})
}
var active atomic.Int32
var maximum atomic.Int32
for laneIndex := range prepared.lanes {
lane := laneIndex
installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
jobIndex := request.Chunk.Index*2 + lane
current := active.Add(1)
defer active.Add(-1)
for {
seen := maximum.Load()
if current <= seen || maximum.CompareAndSwap(seen, current) {
break
}
}
started <- jobIndex
select {
case <-release[jobIndex]:
case <-ctx.Done():
return erasedTypedResult{}, ctx.Err()
}
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index), Warnings: []contracts.Warning{{Scope: fmt.Sprintf("lane-%d/chunk-%d", lane, request.Chunk.Index), ReasonCode: "observed", Message: "ordered"}}}, nil
})
}
type outcome struct {
output RunOutput
err error
}
done := make(chan outcome, 1)
go func() {
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
done <- outcome{output: output, err: err}
}()
for pair := 0; pair < 3; pair++ {
first, second := <-started, <-started
want := map[int]bool{pair * 2: true, pair*2 + 1: true}
if !want[first] || !want[second] || first == second {
t.Fatalf("started jobs = %d, %d; want round-robin pair %d", first, second, pair)
}
close(release[second])
close(release[first])
}
result := <-done
if result.err != nil {
t.Fatalf("Run() error = %v", result.err)
}
if got := maximum.Load(); got != 2 {
t.Fatalf("maximum concurrent extract jobs = %d, want 2", got)
}
wantScopes := []string{"lane-0/chunk-0", "lane-0/chunk-1", "lane-0/chunk-2", "lane-1/chunk-0", "lane-1/chunk-1", "lane-1/chunk-2"}
gotScopes := make([]string, len(result.output.Warnings))
for i := range result.output.Warnings {
gotScopes[i] = result.output.Warnings[i].Scope
}
if !reflect.DeepEqual(gotScopes, wantScopes) {
t.Fatalf("warning order = %#v, want %#v", gotScopes, wantScopes)
}
}
func TestRunnerStartsLaneContinuationWhileOtherLaneExtractsRemain(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 2)
releaseOtherLane := make(chan struct{})
mergeStarted := make(chan struct{}, 1)
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: typedValueForLane(0, request.Chunk.Index)}, nil
})
installExtractOperation(prepared, 1, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
select {
case <-releaseOtherLane:
return erasedTypedResult{Value: typedValueForLane(1, request.Chunk.Index)}, nil
case <-ctx.Done():
return erasedTypedResult{}, ctx.Err()
}
})
originalMerge := prepared.lanes[0].typed.merge
prepared.lanes[0].typed.merge = func(ctx context.Context, implementation any, request contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
select {
case mergeStarted <- struct{}{}:
default:
}
return originalMerge(ctx, implementation, request)
}
done := make(chan error, 1)
go func() {
_, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
done <- err
}()
select {
case <-mergeStarted:
close(releaseOtherLane)
case <-time.After(2 * time.Second):
t.Fatal("first lane merge did not start while the second lane remained in extract")
}
if err := <-done; err != nil {
t.Fatalf("Run() error = %v", err)
}
}
func TestRunnerBoundsConcurrentLaneContinuations(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 1)
base := prepared.lanes[0]
lanes := make([]preparedLaneExecutor, 8)
resolvedLanes := make([]ResolvedArtifactLane, len(lanes))
publicLanes := make([]PreparedArtifactLane, len(lanes))
var active atomic.Int32
var maximum atomic.Int32
for i := range lanes {
lane := base
lane.resolved.ID = fmt.Sprintf("lane-%02d", i)
lane.typed = &preparedTypedLane{
extractor: base.typed.extractor,
merger: base.typed.merger,
normalizer: base.typed.normalizer,
extract: func(_ context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: codecNotes{Items: []string{request.Chunk.ID}}}, nil
},
merge: func(_ context.Context, _ any, _ contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
current := active.Add(1)
defer active.Add(-1)
for {
seen := maximum.Load()
if current <= seen || maximum.CompareAndSwap(seen, current) {
break
}
}
time.Sleep(5 * time.Millisecond)
return erasedTypedResult{Value: codecNotes{}}, nil
},
normalize: base.typed.normalize,
codec: base.typed.codec,
}
lanes[i] = lane
resolvedLanes[i] = lane.resolved
publicLanes[i] = PreparedArtifactLane{Resolved: lane.resolved}
}
prepared.lanes = lanes
prepared.resolved.ArtifactLanes = resolvedLanes
prepared.ArtifactLanes = publicLanes
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(output.NormalizeOutputs) != len(lanes) {
t.Fatalf("normalize outputs = %d, want %d", len(output.NormalizeOutputs), len(lanes))
}
if got := maximum.Load(); got != 2 {
t.Fatalf("maximum concurrent lane continuations = %d, want 2", got)
}
}
func TestRunnerSelectsFrameworkErrorByStableLaneOrder(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 1)
ready := make(chan struct{}, 2)
release := make(chan struct{})
for laneIndex := range prepared.lanes {
lane := laneIndex
installExtractOperation(prepared, lane, func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
ready <- struct{}{}
<-release
return erasedTypedResult{}, fmt.Errorf("lane-%d failure", lane)
})
}
done := make(chan error, 1)
go func() {
_, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
done <- err
}()
<-ready
<-ready
close(release)
if err := <-done; err == nil || !strings.Contains(err.Error(), "lane-0 failure") {
t.Fatalf("Run() error = %v, want first resolved lane failure", err)
}
}
func TestRunnerSelectsMergeErrorBeforeEarlierLaneNormalizeError(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 1)
for laneIndex := range prepared.lanes {
lane := laneIndex
installExtractOperation(prepared, lane, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil
})
}
ready := make(chan struct{}, 2)
release := make(chan struct{})
prepared.lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
ready <- struct{}{}
<-release
return erasedTypedResult{}, errors.New("earlier lane normalize failure")
}
prepared.lanes[1].typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
ready <- struct{}{}
<-release
return erasedTypedResult{}, errors.New("later lane merge failure")
}
done := make(chan error, 1)
go func() {
_, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
done <- err
}()
<-ready
<-ready
close(release)
if err := <-done; err == nil || !strings.Contains(err.Error(), "later lane merge failure") {
t.Fatalf("Run() error = %v, want merge failure before normalize failure", err)
}
}
func TestRunnerKeepsSuccessfulChunksWhenAnotherExtractIsRejected(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 2)
for laneIndex := range prepared.lanes {
lane := laneIndex
installExtractOperation(prepared, lane, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil
})
}
validator := &prepared.lanes[0].extractValidators.validators[0]
validator.typedValidate = func(_ context.Context, _ any, target typedValidationTarget) (contracts.ValidationResult, error) {
if target.chunk != nil && target.chunk.Index == 0 {
return contracts.ValidationResult{Approved: false, ReasonCode: "expected_rejection", Message: "rejected by test"}, nil
}
return contracts.ValidationResult{Approved: true}, nil
}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(output.Rejected) != 1 || output.Rejected[0].LaneID != prepared.lanes[0].resolved.ID || output.Rejected[0].ChunkIndex != 0 {
t.Fatalf("rejections = %#v, want the first lane's first chunk", output.Rejected)
}
if len(output.NormalizeOutputs) != 2 {
t.Fatalf("normalize outputs = %d, want both lanes to continue", len(output.NormalizeOutputs))
}
if output.Manifest.ValidationStatus != "rejected" {
t.Fatalf("validation status = %q, want rejected", output.Manifest.ValidationStatus)
}
}
func TestRunnerSeparatelyBoundsExtractJobsAndSharedProviderCalls(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 3)
provider := &countingProvider{}
scheduler, err := frameworkllm.NewScheduler(2)
if err != nil {
t.Fatalf("NewScheduler() error = %v", err)
}
client := frameworkllm.NewScheduledClient(provider, scheduler)
var jobActive atomic.Int32
var jobMaximum atomic.Int32
var attempts sync.Map
for laneIndex := range prepared.lanes {
lane := laneIndex
prepared.lanes[lane].resolved.Extract.Retries = 1
installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
current := jobActive.Add(1)
defer jobActive.Add(-1)
for {
seen := jobMaximum.Load()
if current <= seen || jobMaximum.CompareAndSwap(seen, current) {
break
}
}
if _, callErr := client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{StageName: "extract"}, nil); callErr != nil {
return erasedTypedResult{}, callErr
}
key := fmt.Sprintf("%d/%d", lane, request.Chunk.Index)
countValue, _ := attempts.LoadOrStore(key, &atomic.Int32{})
count := countValue.(*atomic.Int32).Add(1)
if lane == 0 && request.Chunk.Index == 0 && count == 1 {
return erasedTypedResult{}, errors.New("retry extract")
}
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil
})
validator := &prepared.lanes[lane].extractValidators.validators[0]
validator.typedValidate = func(ctx context.Context, _ any, _ typedValidationTarget) (contracts.ValidationResult, error) {
if _, callErr := client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{StageName: "validate"}, nil); callErr != nil {
return contracts.ValidationResult{}, callErr
}
return contracts.ValidationResult{Approved: true}, nil
}
}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 3})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(output.NormalizeOutputs) != 2 {
t.Fatalf("normalize outputs = %d, want 2", len(output.NormalizeOutputs))
}
if got := jobMaximum.Load(); got != 3 {
t.Fatalf("maximum extract jobs = %d, want 3", got)
}
if got := provider.maximum.Load(); got != 2 {
t.Fatalf("maximum provider calls = %d, want 2", got)
}
if got := provider.calls.Load(); got != 13 {
t.Fatalf("provider calls = %d, want 13 including retry and validators", got)
}
}
func TestRunnerReturnsParentCancellationAndStopsQueuedExtracts(t *testing.T) {
prepared := preparedConcurrentPipeline(t, 4)
started := make(chan struct{}, 8)
for laneIndex := range prepared.lanes {
lane := laneIndex
installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
started <- struct{}{}
<-ctx.Done()
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, ctx.Err()
})
}
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() {
_, err := New().Run(ctx, RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
done <- err
}()
<-started
<-started
cancel()
if err := <-done; !errors.Is(err, context.Canceled) {
t.Fatalf("Run() error = %v, want context.Canceled", err)
}
if got := len(started); got != 0 {
t.Fatalf("additional started jobs = %d, want none after cancellation", got)
}
}

View File

@@ -0,0 +1,399 @@
package pipeline
import (
"context"
"errors"
"fmt"
"path"
"sort"
"strings"
"sync"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type laneExtractState struct {
index int
prepared preparedLaneExecutor
deps []CheckpointFingerprint
decision CheckpointDecision
values []erasedExtractArtifact
serialized []CheckpointArtifact
warnings []contracts.Warning
rejected []contracts.RejectedOutput
results map[int]extractJobResult
remaining int
failed bool
}
type extractJob struct {
lane *laneExtractState
chunk source.Chunk
}
type extractJobResult struct {
laneIndex int
chunkIndex int
value erasedExtractArtifact
serialized CheckpointArtifact
warnings []contracts.Warning
rejected *contracts.RejectedOutput
err error
}
type laneCompletion struct {
index int
output RunOutput
err error
}
type orderedRunError struct {
stage int
lane int
chunk int
err error
}
type completedExtractLoader struct {
CheckpointLoader
laneID string
checkpoint ExtractCheckpoint
}
func (l completedExtractLoader) Extract(laneID, _ string, _ []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
if laneID == l.laneID {
return l.checkpoint, CheckpointDecision{Reused: true, Reason: "coordinated extract result"}
}
return ExtractCheckpoint{}, CheckpointDecision{Reason: "extract result unavailable"}
}
func (r *Runner) runLanes(parent context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk) (RunOutput, error) {
output := RunOutput{Manifest: manifestFromPipeline(input)}
states := make([]*laneExtractState, len(input.Prepared.lanes))
for i, prepared := range input.Prepared.lanes {
if prepared.typed == nil {
return output, fmt.Errorf("typed lane %q executor is not prepared", prepared.resolved.ID)
}
setTypedLaneManifestMetadata(&output, prepared.resolved.ID, prepared.typed.extractor, prepared.typed.merger, prepared.typed.normalizer)
state, err := prepareLaneExtract(input, loader, doc, chunks, i, prepared)
if err != nil {
return output, err
}
if !state.decision.Reused {
if err := checkpoints.ExtractRunning(prepared.resolved.ID, prepared.resolved.Extract.Module, state.deps); err != nil {
return output, fmt.Errorf("write extract checkpoint for lane %q: %w", prepared.resolved.ID, err)
}
}
states[i] = state
}
workerCount := input.ExtractWorkers
if workerCount < 1 {
workerCount = 1
}
ctx, cancel := context.WithCancel(parent)
defer cancel()
jobs := make(chan extractJob, workerCount)
results := make(chan extractJobResult, workerCount)
completions := make(chan laneCompletion, len(states))
continuations := make(chan *laneExtractState, workerCount)
var workers sync.WaitGroup
for i := 0; i < workerCount; i++ {
workers.Add(1)
go func() {
defer workers.Done()
for job := range jobs {
if ctx.Err() != nil {
continue
}
result := r.runExtractJob(ctx, input, doc, sourceInput, sessionID, job)
results <- result
}
}()
}
go func() {
defer close(jobs)
for chunkIndex := range chunks {
for laneIndex := range states {
state := states[laneIndex]
if state.decision.Reused {
continue
}
select {
case jobs <- extractJob{lane: state, chunk: chunks[chunkIndex]}:
case <-ctx.Done():
return
}
}
}
}()
go func() { workers.Wait(); close(results) }()
var continuationWorkers sync.WaitGroup
for i := 0; i < workerCount; i++ {
continuationWorkers.Add(1)
go func() {
defer continuationWorkers.Done()
for state := range continuations {
if err := ctx.Err(); err != nil {
completions <- laneCompletion{index: state.index, err: err}
continue
}
laneOutput, err := r.continueLane(ctx, input, checkpoints, loader, doc, sourceInput, sessionID, chunks, state)
completions <- laneCompletion{index: state.index, output: laneOutput, err: err}
}
}()
}
completedOutputs := make([]RunOutput, len(states))
var runErrors []orderedRunError
var pendingContinuations []*laneExtractState
launched, completed := 0, 0
for _, state := range states {
if state.decision.Reused {
pendingContinuations = append(pendingContinuations, state)
}
}
resultChannel := results
for resultChannel != nil || len(pendingContinuations) > 0 || completed < launched {
var continuationChannel chan<- *laneExtractState
var nextContinuation *laneExtractState
if len(pendingContinuations) > 0 && ctx.Err() == nil {
continuationChannel = continuations
nextContinuation = pendingContinuations[0]
} else if ctx.Err() != nil {
pendingContinuations = nil
}
select {
case continuationChannel <- nextContinuation:
pendingContinuations = pendingContinuations[1:]
launched++
case result, ok := <-resultChannel:
if !ok {
resultChannel = nil
continue
}
state := states[result.laneIndex]
state.remaining--
if result.err != nil {
state.failed = true
runErrors = append(runErrors, orderedRunError{stage: 0, lane: state.index, chunk: result.chunkIndex, err: result.err})
_ = checkpoints.ExtractFailed(state.prepared.resolved.ID, state.prepared.resolved.Extract.Module, state.deps, result.err)
cancel()
} else {
state.results[result.chunkIndex] = result
}
if state.remaining == 0 && !state.failed && ctx.Err() == nil {
if err := finalizeLaneExtract(checkpoints, state); err != nil {
state.failed = true
runErrors = append(runErrors, orderedRunError{stage: 0, lane: state.index, chunk: len(chunks), err: err})
cancel()
} else {
pendingContinuations = append(pendingContinuations, state)
}
}
case completion := <-completions:
completed++
completedOutputs[completion.index] = completion.output
if completion.err != nil {
runErrors = append(runErrors, classifyLaneError(completion.index, len(chunks), completion.err))
cancel()
}
}
}
close(continuations)
continuationWorkers.Wait()
for i := range completedOutputs {
mergeLaneOutput(&output, completedOutputs[i])
}
if err := selectRunError(parent, runErrors); err != nil {
return output, err
}
return output, nil
}
func prepareLaneExtract(input RunInput, loader CheckpointLoader, doc *source.SourceDocument, chunks []source.Chunk, index int, prepared preparedLaneExecutor) (*laneExtractState, error) {
lane, typed := prepared.resolved, prepared.typed
digest, err := joinedChunkDigest(chunks)
if err != nil {
return nil, fmt.Errorf("digest chunks for lane %q: %w", lane.ID, err)
}
state := &laneExtractState{index: index, prepared: prepared, deps: digestFingerprints("chunks", digest), remaining: len(chunks), results: make(map[int]extractJobResult, len(chunks))}
cp, decision := loadExtract(loader, lane.ID, lane.Extract.Module, state.deps)
if decision.Reused {
for _, stored := range cp.Outputs {
if _, decodeErr := decodeCheckpointArtifact(typed.codec, stored); decodeErr != nil {
decision = CheckpointDecision{Reason: "extract artifact checkpoint codec is incompatible: " + decodeErr.Error()}
break
}
}
}
state.decision = decision
if decision.Reused {
state.remaining = 0
for _, stored := range cp.Outputs {
value, decodeErr := decodeCheckpointArtifact(typed.codec, stored)
if decodeErr != nil {
return nil, fmt.Errorf("decode extract checkpoint for lane %q: %w", lane.ID, decodeErr)
}
stored = hydrateCheckpointArtifact(typed.codec, stored, value)
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: stored.ChunkID, ChunkIndex: stored.ChunkIndex, ChunkRef: stored.ChunkRef, Value: value}
if stored.ChunkIndex >= 0 && stored.ChunkIndex < len(chunks) && artifact.ChunkRef == (source.SourceRef{}) {
artifact.ChunkRef = chunks[stored.ChunkIndex].Ref
}
state.values = append(state.values, artifact)
state.serialized = append(state.serialized, cloneCheckpointArtifact(stored))
}
state.warnings, state.rejected = cloneWarnings(cp.Warnings), cloneRejectedOutputs(cp.Rejected)
}
return state, nil
}
func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, job extractJob) extractJobResult {
state, chunk := job.lane, job.chunk
lane, typed := state.prepared.resolved, state.prepared.typed
result := extractJobResult{laneIndex: state.index, chunkIndex: chunk.Index}
var accepted erasedExtractArtifact
var serialized CheckpointArtifact
var acceptedWarnings []contracts.Warning
ok, rejection, err := runWithRetry(ctx, lane.Extract.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
started := time.Now().UTC()
attemptPath := path.Join("extract", debugPathComponent(lane.ID), fmt.Sprintf("chunk-%06d", chunk.Index+1), fmt.Sprintf("attempt-%02d", attempt))
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
extracted, callErr := typed.extract(attemptCtx, typed.extractor, contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk, SourceInput: chunkInputMaterial(sourceInput, chunk), SessionID: sessionID, References: CloneReferenceSet(lane.ExtractReferences.ReferenceSet), LLMProfile: lane.Extract.LLMProfile, Metadata: cloneMetadata(input.Metadata)})
if callErr != nil {
_ = writeDebugTimed(input.Debug, attemptPath+".json", debugEnvelopeWithLLMCalls(debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started, Error: callErr.Error()}, llmScope))
return false, nil, fmt.Errorf("extract lane %q chunk %q with extractor %q: %w", lane.ID, chunk.ID, lane.Extract.Module, callErr)
}
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: chunk.ID, ChunkIndex: chunk.Index, ChunkRef: chunk.Ref, Value: extracted.Value}
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageExtract, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: lane.ExtractReferences.ReferenceSet, metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: extracted.Value}, state.prepared.extractValidators, attempt, input.Debug)
if validateErr != nil || rejected != nil {
return false, rejected, validateErr
}
stored, encodeErr := checkpointArtifact(typed.codec, artifact.LaneID, artifact.ExtractorKey, artifact.SourceID, artifact.Value)
if encodeErr != nil {
return false, nil, encodeErr
}
stored.ChunkID, stored.ChunkIndex, stored.ChunkRef = artifact.ChunkID, artifact.ChunkIndex, artifact.ChunkRef
accepted, serialized = artifact, stored
acceptedWarnings = append(cloneWarnings(extracted.Warnings), warnings...)
if debugErr := writeDebugTimed(input.Debug, attemptPath+".json", debugEnvelopeWithLLMCalls(debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started, Payload: map[string]any{"output": debugCheckpointArtifact(stored), "warnings": debugWarningEnvelopes(acceptedWarnings)}}, llmScope)); debugErr != nil {
return false, nil, debugErr
}
return true, nil, nil
})
result.err = err
if err == nil && !ok {
result.rejected = rejection
return result
}
result.value, result.serialized, result.warnings = accepted, serialized, acceptedWarnings
return result
}
func finalizeLaneExtract(checkpoints CheckpointRecorder, state *laneExtractState) error {
lane := state.prepared.resolved
indexes := make([]int, 0, len(state.results))
for index := range state.results {
indexes = append(indexes, index)
}
sort.Ints(indexes)
for _, index := range indexes {
result := state.results[index]
if result.rejected != nil {
state.rejected = append(state.rejected, *result.rejected)
continue
}
state.values = append(state.values, result.value)
state.serialized = append(state.serialized, result.serialized)
state.warnings = append(state.warnings, result.warnings...)
}
sort.SliceStable(state.values, func(i, j int) bool { return state.values[i].ChunkIndex < state.values[j].ChunkIndex })
sort.SliceStable(state.serialized, func(i, j int) bool { return state.serialized[i].ChunkIndex < state.serialized[j].ChunkIndex })
sort.SliceStable(state.rejected, func(i, j int) bool { return state.rejected[i].ChunkIndex < state.rejected[j].ChunkIndex })
if !state.decision.Reused {
if err := recordExtract(checkpoints, lane.ID, lane.Extract.Module, state.deps, state.serialized, state.rejected, state.warnings); err != nil {
return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err)
}
}
return nil
}
func (r *Runner) continueLane(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk, state *laneExtractState) (RunOutput, error) {
lane := state.prepared.resolved
local := RunOutput{Manifest: manifestFromPipeline(input)}
checkpoint := ExtractCheckpoint{Outputs: state.serialized, Rejected: state.rejected, Warnings: state.warnings}
coordinatedLoader := completedExtractLoader{CheckpointLoader: loader, laneID: lane.ID, checkpoint: checkpoint}
input.extractDecision = &state.decision
err := r.runTypedLane(ctx, input, checkpoints, coordinatedLoader, doc, sourceInput, sessionID, chunks, state.prepared, &local)
return local, err
}
func classifyLaneError(lane, sentinel int, err error) orderedRunError {
stage := 1
var laneErr *laneRunError
if errors.As(err, &laneErr) {
switch laneErr.stage {
case StageExtract:
stage = 0
case StageNormalize:
stage = 2
}
} else if strings.Contains(strings.ToLower(err.Error()), "normalize") {
stage = 2
}
return orderedRunError{stage: stage, lane: lane, chunk: sentinel, err: err}
}
func selectRunError(parent context.Context, values []orderedRunError) error {
if err := parent.Err(); err != nil {
return err
}
if len(values) == 0 {
return nil
}
hasReal := false
for _, value := range values {
if !errors.Is(value.err, context.Canceled) && !errors.Is(value.err, context.DeadlineExceeded) {
hasReal = true
break
}
}
filtered := values[:0]
for _, value := range values {
if hasReal && (errors.Is(value.err, context.Canceled) || errors.Is(value.err, context.DeadlineExceeded)) {
continue
}
filtered = append(filtered, value)
}
sort.SliceStable(filtered, func(i, j int) bool {
if filtered[i].stage != filtered[j].stage {
return filtered[i].stage < filtered[j].stage
}
if filtered[i].lane != filtered[j].lane {
return filtered[i].lane < filtered[j].lane
}
return filtered[i].chunk < filtered[j].chunk
})
return filtered[0].err
}
func mergeLaneOutput(dst *RunOutput, src RunOutput) {
if dst == nil {
return
}
dst.NormalizeOutputs = append(dst.NormalizeOutputs, cloneSerializedOutputs(src.NormalizeOutputs)...)
dst.Rejected = append(dst.Rejected, cloneRejectedOutputs(src.Rejected)...)
dst.Warnings = append(dst.Warnings, cloneWarnings(src.Warnings)...)
dst.CheckpointEvents = append(dst.CheckpointEvents, src.CheckpointEvents...)
for i := range dst.Manifest.ArtifactLanes {
for j := range src.Manifest.ArtifactLanes {
if dst.Manifest.ArtifactLanes[i].ID == src.Manifest.ArtifactLanes[j].ID && src.Manifest.ArtifactLanes[j].Metadata != nil {
dst.Manifest.ArtifactLanes[i].Metadata = cloneMetadata(src.Manifest.ArtifactLanes[j].Metadata)
}
}
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,463 @@
package pipeline
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"path"
"sort"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func loadExtract(loader CheckpointLoader, laneID, moduleKey string, deps []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
return loader.Extract(laneID, moduleKey, deps)
}
func loadMerge(loader CheckpointLoader, laneID, moduleKey string, deps []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
return loader.Merge(laneID, moduleKey, deps)
}
func loadNormalize(loader CheckpointLoader, laneID, moduleKey string, deps []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
return loader.Normalize(laneID, moduleKey, deps)
}
func recordExtract(recorder CheckpointRecorder, laneID, moduleKey string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
return recorder.ExtractSucceeded(laneID, moduleKey, deps, outputs, rejected, warnings)
}
func recordMerge(recorder CheckpointRecorder, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
return recorder.MergeSucceeded(laneID, moduleKey, deps, output, warnings)
}
func recordNormalize(recorder CheckpointRecorder, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
return recorder.NormalizeSucceeded(laneID, moduleKey, deps, output, warnings)
}
func cloneCheckpointArtifact(output CheckpointArtifact) CheckpointArtifact {
output.Artifact = contracts.CloneSerializedArtifact(output.Artifact)
return output
}
func hydrateCheckpointArtifact(codec artifactCodecEntry, output CheckpointArtifact, value any) CheckpointArtifact {
output.Artifact.Schema = contracts.CloneArtifactSchema(codec.spec.Schema)
if codec.metadata != nil {
output.Artifact.Metadata = cloneMetadata(codec.metadata(value))
} else {
output.Artifact.Metadata = nil
}
return output
}
func artifactCheckpointDigests(outputs []CheckpointArtifact) []CheckpointFingerprint {
values := make([]CheckpointFingerprint, 0, len(outputs))
for i, output := range outputs {
sum := sha256.Sum256(output.Artifact.Content)
values = append(values, CheckpointFingerprint{Name: fmt.Sprintf("artifact[%d]", i), Value: "sha256:" + hex.EncodeToString(sum[:])})
}
return normalizeCheckpointFingerprints(values)
}
func debugCheckpointArtifact(output CheckpointArtifact) map[string]any {
artifact := output.Artifact
schema := contracts.CloneArtifactSchema(artifact.Schema)
digest := output.SchemaDigest
if digest == "" {
digest = contracts.DigestArtifactSchema(schema)
}
schema.JSONSchema = nil
content := debugContentEnvelope(artifact.Content, artifact.MediaType, artifact.Metadata, nil)
content.ContentDigest = debugContentDigest(artifact.Content)
return map[string]any{"lane_id": output.LaneID, "module_key": output.ModuleKey, "source_id": output.SourceID, "chunk_id": output.ChunkID, "chunk_index": output.ChunkIndex, "chunk_ref": output.ChunkRef, "artifact_kind": artifact.Kind, "schema": schema, "schema_digest": digest, "content": content}
}
func debugCheckpointArtifacts(outputs []CheckpointArtifact) []map[string]any {
if len(outputs) == 0 {
return nil
}
out := make([]map[string]any, 0, len(outputs))
for _, output := range outputs {
out = append(out, debugCheckpointArtifact(output))
}
return out
}
func serializeArtifact(codec artifactCodecEntry, value any, candidate bool) (contracts.SerializedArtifact, error) {
encode := codec.encode
if candidate {
encode = codec.encodeCandidate
}
content, err := encode(value)
if err != nil {
return contracts.SerializedArtifact{}, err
}
schema := codec.spec.Schema
metadata := map[string]any(nil)
if codec.metadata != nil {
metadata = codec.metadata(value)
}
return contracts.SerializedArtifact{Kind: codec.spec.Kind, Schema: contracts.CloneArtifactSchema(schema), MediaType: codec.spec.MediaType, Content: append([]byte(nil), content...), Metadata: cloneMetadata(metadata)}, nil
}
func decodeCheckpointArtifact(codec artifactCodecEntry, artifact CheckpointArtifact) (any, error) {
expectedDigest := contracts.DigestArtifactSchema(codec.spec.Schema)
if artifact.Artifact.Kind != codec.spec.Kind {
return nil, fmt.Errorf("artifact kind %q does not match codec %q", artifact.Artifact.Kind, codec.spec.Kind)
}
if artifact.Artifact.Schema.ID != codec.spec.Schema.ID || artifact.Artifact.Schema.Version != codec.spec.Schema.Version {
return nil, fmt.Errorf("artifact schema %q version %q does not match codec schema %q version %q", artifact.Artifact.Schema.ID, artifact.Artifact.Schema.Version, codec.spec.Schema.ID, codec.spec.Schema.Version)
}
if artifact.SchemaDigest != expectedDigest {
return nil, fmt.Errorf("artifact schema digest %q does not match codec schema digest %q", artifact.SchemaDigest, expectedDigest)
}
if artifact.Artifact.MediaType != codec.spec.MediaType {
return nil, fmt.Errorf("artifact media type %q does not match codec media type %q", artifact.Artifact.MediaType, codec.spec.MediaType)
}
return codec.decode(append([]byte(nil), artifact.Artifact.Content...))
}
func checkpointArtifact(codec artifactCodecEntry, laneID, moduleKey, sourceID string, value any) (CheckpointArtifact, error) {
serialized, err := serializeArtifact(codec, value, false)
if err != nil {
return CheckpointArtifact{}, err
}
return CheckpointArtifact{LaneID: laneID, ModuleKey: moduleKey, SourceID: sourceID, Artifact: serialized, SchemaDigest: contracts.DigestArtifactSchema(serialized.Schema)}, nil
}
type laneRunError struct {
stage ModuleStage
err error
}
func (e *laneRunError) Error() string { return e.err.Error() }
func (e *laneRunError) Unwrap() error { return e.err }
func (r *Runner) runTypedLane(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk, prepared preparedLaneExecutor, output *RunOutput) (err error) {
activeStage := StageExtract
defer func() {
if err != nil {
err = &laneRunError{stage: activeStage, err: err}
}
}()
lane, typed := prepared.resolved, prepared.typed
if typed == nil {
return fmt.Errorf("typed lane %q executor is not prepared", lane.ID)
}
setTypedLaneManifestMetadata(output, lane.ID, typed.extractor, typed.merger, typed.normalizer)
values := make([]erasedExtractArtifact, 0, len(chunks))
serializedExtracts := make([]CheckpointArtifact, 0, len(chunks))
extractWarnings := []contracts.Warning{}
rejectedStart := len(output.Rejected)
chunksDigest, err := joinedChunkDigest(chunks)
if err != nil {
return fmt.Errorf("digest chunks for lane %q: %w", lane.ID, err)
}
extractDeps := digestFingerprints("chunks", chunksDigest)
cp, decision := loadExtract(loader, lane.ID, lane.Extract.Module, extractDeps)
if decision.Reused {
for _, stored := range cp.Outputs {
if _, decodeErr := decodeCheckpointArtifact(typed.codec, stored); decodeErr != nil {
decision = CheckpointDecision{Reason: "extract artifact checkpoint codec is incompatible: " + decodeErr.Error()}
break
}
}
}
reportedDecision := decision
if input.extractDecision != nil {
reportedDecision = *input.extractDecision
}
recordCheckpointEvent(output, loader, string(StageExtract), lane.ID, lane.Extract.Module, reportedDecision)
if err := writeDebugTimed(input.Debug, path.Join("extract", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": reportedDecision.Reused, "decision": reportedDecision, "source": debugSourceDocumentEnvelope(doc), "chunks": debugSourceChunkEnvelopes(chunks), "options": redactSensitiveMap(lane.Extract.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
return err
}
if decision.Reused {
for _, stored := range cp.Outputs {
value, decodeErr := decodeCheckpointArtifact(typed.codec, stored)
if decodeErr != nil {
return fmt.Errorf("decode extract checkpoint for lane %q: %w", lane.ID, decodeErr)
}
stored = hydrateCheckpointArtifact(typed.codec, stored, value)
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: stored.ChunkID, ChunkIndex: stored.ChunkIndex, ChunkRef: stored.ChunkRef, Value: value}
if stored.ChunkIndex >= 0 && stored.ChunkIndex < len(chunks) && artifact.ChunkRef == (source.SourceRef{}) {
artifact.ChunkRef = chunks[stored.ChunkIndex].Ref
}
values = append(values, artifact)
serializedExtracts = append(serializedExtracts, cloneCheckpointArtifact(stored))
}
extractWarnings = cloneWarnings(cp.Warnings)
output.Warnings = append(output.Warnings, extractWarnings...)
output.Rejected = append(output.Rejected, cloneRejectedOutputs(cp.Rejected)...)
} else {
if err := checkpoints.ExtractRunning(lane.ID, lane.Extract.Module, extractDeps); err != nil {
return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err)
}
for i := range chunks {
chunk := chunks[i]
var accepted erasedExtractArtifact
var serializedAccepted CheckpointArtifact
var acceptedWarnings []contracts.Warning
ok, rejection, runErr := runWithRetry(ctx, lane.Extract.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
started := time.Now().UTC()
attemptPath := path.Join("extract", debugPathComponent(lane.ID), fmt.Sprintf("chunk-%06d", chunk.Index+1), fmt.Sprintf("attempt-%02d", attempt))
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
result, callErr := typed.extract(attemptCtx, typed.extractor, contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk, SourceInput: chunkInputMaterial(sourceInput, chunk), SessionID: sessionID, References: CloneReferenceSet(lane.ExtractReferences.ReferenceSet), LLMProfile: lane.Extract.LLMProfile, Metadata: cloneMetadata(input.Metadata)})
if callErr != nil {
_ = writeDebugTimed(input.Debug, attemptPath+".json", debugEnvelopeWithLLMCalls(debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started, Error: callErr.Error()}, llmScope))
return false, nil, fmt.Errorf("extract lane %q chunk %q with extractor %q: %w", lane.ID, chunk.ID, lane.Extract.Module, callErr)
}
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: chunk.ID, ChunkIndex: chunk.Index, ChunkRef: chunk.Ref, Value: result.Value}
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageExtract, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: lane.ExtractReferences.ReferenceSet, metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: result.Value}, prepared.extractValidators, attempt, input.Debug)
if validateErr != nil || rejected != nil {
return false, rejected, validateErr
}
stored, encodeErr := checkpointArtifact(typed.codec, artifact.LaneID, artifact.ExtractorKey, artifact.SourceID, artifact.Value)
if encodeErr != nil {
return false, nil, encodeErr
}
stored.ChunkID, stored.ChunkIndex, stored.ChunkRef = artifact.ChunkID, artifact.ChunkIndex, artifact.ChunkRef
accepted, serializedAccepted = artifact, stored
acceptedWarnings = append(cloneWarnings(result.Warnings), warnings...)
if debugErr := writeDebugTimed(input.Debug, attemptPath+".json", debugEnvelopeWithLLMCalls(debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started, Payload: map[string]any{"output": debugCheckpointArtifact(stored), "warnings": debugWarningEnvelopes(acceptedWarnings)}}, llmScope)); debugErr != nil {
return false, nil, debugErr
}
return true, nil, nil
})
if runErr != nil {
_ = checkpoints.ExtractFailed(lane.ID, lane.Extract.Module, extractDeps, runErr)
return runErr
}
if !ok {
output.Rejected = append(output.Rejected, *rejection)
continue
}
values = append(values, accepted)
serializedExtracts = append(serializedExtracts, serializedAccepted)
extractWarnings = append(extractWarnings, acceptedWarnings...)
output.Warnings = append(output.Warnings, acceptedWarnings...)
}
if err := recordExtract(checkpoints, lane.ID, lane.Extract.Module, extractDeps, serializedExtracts, cloneRejectedOutputs(output.Rejected[rejectedStart:]), extractWarnings); err != nil {
return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err)
}
}
sort.SliceStable(values, func(i, j int) bool { return values[i].ChunkIndex < values[j].ChunkIndex })
sort.SliceStable(serializedExtracts, func(i, j int) bool { return serializedExtracts[i].ChunkIndex < serializedExtracts[j].ChunkIndex })
if err := writeDebugTimed(input.Debug, path.Join("extract", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": reportedDecision.Reused, "outputs": debugCheckpointArtifacts(serializedExtracts), "rejected": debugRejectedOutputEnvelopes(output.Rejected[rejectedStart:]), "warnings": debugWarningEnvelopes(extractWarnings)}}); err != nil {
return err
}
if len(values) == 0 {
return nil
}
activeStage = StageMerge
mergeInputs := make([]contracts.ExtractArtifact[any], len(values))
for i, value := range values {
mergeInputs[i] = contracts.ExtractArtifact[any]{LaneID: value.LaneID, ExtractorKey: value.ExtractorKey, SourceID: value.SourceID, ChunkID: value.ChunkID, ChunkIndex: value.ChunkIndex, ChunkRef: value.ChunkRef, Value: value.Value}
}
mergeDeps := artifactCheckpointDigests(serializedExtracts)
mergeCP, mergeDecision := loadMerge(loader, lane.ID, lane.Merge.Module, mergeDeps)
if mergeDecision.Reused {
if _, decodeErr := decodeCheckpointArtifact(typed.codec, mergeCP.Output); decodeErr != nil {
mergeDecision = CheckpointDecision{Reason: "merge artifact checkpoint codec is incompatible: " + decodeErr.Error()}
}
}
recordCheckpointEvent(output, loader, string(StageMerge), lane.ID, lane.Merge.Module, mergeDecision)
if err := writeDebugTimed(input.Debug, path.Join("merge", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageMerge), LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "decision": mergeDecision, "source": debugSourceDocumentEnvelope(doc), "extract_outputs": debugCheckpointArtifacts(serializedExtracts), "options": redactSensitiveMap(lane.Merge.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
return err
}
var merged erasedMergeArtifact
var serializedMerge CheckpointArtifact
var mergeWarnings []contracts.Warning
if mergeDecision.Reused {
value, decodeErr := decodeCheckpointArtifact(typed.codec, mergeCP.Output)
if decodeErr != nil {
return fmt.Errorf("decode merge checkpoint for lane %q: %w", lane.ID, decodeErr)
}
merged = erasedMergeArtifact{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: value}
serializedMerge = hydrateCheckpointArtifact(typed.codec, cloneCheckpointArtifact(mergeCP.Output), value)
mergeWarnings = cloneWarnings(mergeCP.Warnings)
output.Warnings = append(output.Warnings, mergeWarnings...)
} else {
if err := checkpoints.MergeRunning(lane.ID, lane.Merge.Module, mergeDeps); err != nil {
return err
}
ok, rejection, runErr := runWithRetry(ctx, lane.Merge.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
result, callErr := typed.merge(ctx, typed.merger, contracts.TypedMergeRequest[any]{Source: doc, LaneID: lane.ID, ExtractOutputs: mergeInputs, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(lane.MergeReferences.ReferenceSet), LLMProfile: lane.Merge.LLMProfile, Metadata: cloneMetadata(input.Metadata)})
if callErr != nil {
return false, nil, fmt.Errorf("merge lane %q with merger %q: %w", lane.ID, lane.Merge.Module, callErr)
}
candidate := erasedMergeArtifact{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: result.Value}
warnings, rejected, validateErr := r.validateTypedArtifact(ctx, typed.codec, typedValidationTarget{stage: StageMerge, laneID: lane.ID, moduleKey: lane.Merge.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: lane.MergeReferences.ReferenceSet, metadata: input.Metadata, value: result.Value}, prepared.mergeValidators, attempt, input.Debug)
if validateErr != nil || rejected != nil {
return false, rejected, validateErr
}
stored, encodeErr := checkpointArtifact(typed.codec, candidate.LaneID, candidate.MergerKey, candidate.SourceID, candidate.Value)
if encodeErr != nil {
return false, nil, encodeErr
}
merged, serializedMerge = candidate, stored
mergeWarnings = append(cloneWarnings(result.Warnings), warnings...)
return true, nil, nil
})
if runErr != nil {
_ = checkpoints.MergeFailed(lane.ID, lane.Merge.Module, mergeDeps, runErr)
return runErr
}
if !ok {
output.Rejected = append(output.Rejected, *rejection)
if err := checkpoints.MergeRejected(lane.ID, lane.Merge.Module, mergeDeps, *rejection); err != nil {
return err
}
return nil
}
output.Warnings = append(output.Warnings, mergeWarnings...)
if err := recordMerge(checkpoints, lane.ID, lane.Merge.Module, mergeDeps, serializedMerge, mergeWarnings); err != nil {
return err
}
}
if err := writeDebugTimed(input.Debug, path.Join("merge", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageMerge), LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedMerge), "warnings": debugWarningEnvelopes(mergeWarnings)}}); err != nil {
return err
}
activeStage = StageNormalize
normalizeDeps := artifactCheckpointDigests([]CheckpointArtifact{serializedMerge})
normalizeCP, normalizeDecision := loadNormalize(loader, lane.ID, lane.Normalize.Module, normalizeDeps)
if normalizeDecision.Reused {
if _, decodeErr := decodeCheckpointArtifact(typed.codec, normalizeCP.Output); decodeErr != nil {
normalizeDecision = CheckpointDecision{Reason: "normalize artifact checkpoint codec is incompatible: " + decodeErr.Error()}
}
}
recordCheckpointEvent(output, loader, string(StageNormalize), lane.ID, lane.Normalize.Module, normalizeDecision)
if err := writeDebugTimed(input.Debug, path.Join("normalize", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageNormalize), LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "decision": normalizeDecision, "source": debugSourceDocumentEnvelope(doc), "merge_output": debugCheckpointArtifact(serializedMerge), "options": redactSensitiveMap(lane.Normalize.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
return err
}
var serializedNormalize CheckpointArtifact
var normalizeWarnings []contracts.Warning
if normalizeDecision.Reused {
value, decodeErr := decodeCheckpointArtifact(typed.codec, normalizeCP.Output)
if decodeErr != nil {
return fmt.Errorf("decode normalize checkpoint for lane %q: %w", lane.ID, decodeErr)
}
serializedNormalize, normalizeWarnings = hydrateCheckpointArtifact(typed.codec, cloneCheckpointArtifact(normalizeCP.Output), value), cloneWarnings(normalizeCP.Warnings)
output.Warnings = append(output.Warnings, normalizeWarnings...)
} else {
if err := checkpoints.NormalizeRunning(lane.ID, lane.Normalize.Module, normalizeDeps); err != nil {
return err
}
ok, rejection, runErr := runWithRetry(ctx, lane.Normalize.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
result, callErr := typed.normalize(ctx, typed.normalizer, contracts.TypedNormalizeRequest[any]{Source: doc, LaneID: lane.ID, MergeOutput: contracts.MergeArtifact[any]{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: merged.Value}, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(lane.NormalizeReferences.ReferenceSet), LLMProfile: lane.Normalize.LLMProfile, Metadata: cloneMetadata(input.Metadata)})
if callErr != nil {
return false, nil, fmt.Errorf("normalize lane %q with normalizer %q: %w", lane.ID, lane.Normalize.Module, callErr)
}
warnings, rejected, validateErr := r.validateTypedArtifact(ctx, typed.codec, typedValidationTarget{stage: StageNormalize, laneID: lane.ID, moduleKey: lane.Normalize.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: lane.NormalizeReferences.ReferenceSet, metadata: input.Metadata, value: result.Value}, prepared.normalizeValidators, attempt, input.Debug)
if validateErr != nil || rejected != nil {
return false, rejected, validateErr
}
stored, encodeErr := checkpointArtifact(typed.codec, lane.ID, lane.Normalize.Module, doc.ID, result.Value)
if encodeErr != nil {
return false, nil, encodeErr
}
serializedNormalize = stored
normalizeWarnings = append(cloneWarnings(result.Warnings), warnings...)
return true, nil, nil
})
if runErr != nil {
_ = checkpoints.NormalizeFailed(lane.ID, lane.Normalize.Module, normalizeDeps, runErr)
return runErr
}
if !ok {
output.Rejected = append(output.Rejected, *rejection)
if err := checkpoints.NormalizeRejected(lane.ID, lane.Normalize.Module, normalizeDeps, *rejection); err != nil {
return err
}
return nil
}
output.Warnings = append(output.Warnings, normalizeWarnings...)
if err := recordNormalize(checkpoints, lane.ID, lane.Normalize.Module, normalizeDeps, serializedNormalize, normalizeWarnings); err != nil {
return err
}
}
if err := writeDebugTimed(input.Debug, path.Join("normalize", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageNormalize), LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedNormalize), "warnings": debugWarningEnvelopes(normalizeWarnings)}}); err != nil {
return err
}
output.NormalizeOutputs = append(output.NormalizeOutputs, contracts.SerializedOutput{LaneID: lane.ID, NormalizerKey: lane.Normalize.Module, SourceID: doc.ID, Artifact: contracts.CloneSerializedArtifact(serializedNormalize.Artifact)})
return nil
}
func setTypedLaneManifestMetadata(output *RunOutput, laneID string, extractor, merger, normalizer any) {
if output == nil {
return
}
for i := range output.Manifest.ArtifactLanes {
if output.Manifest.ArtifactLanes[i].ID != laneID {
continue
}
metadata := make(map[string]any)
for _, item := range []struct {
name string
module any
}{{"extractor", extractor}, {"merger", merger}, {"normalizer", normalizer}} {
if value, ok := moduleManifestMetadata(item.module); ok {
metadata[item.name] = value
}
}
if len(metadata) > 0 {
output.Manifest.ArtifactLanes[i].Metadata = metadata
}
return
}
}
func (r *Runner) validateTypedArtifact(ctx context.Context, codec artifactCodecEntry, target typedValidationTarget, chain preparedValidatorChain, attempt int, debug DebugRecorder) ([]contracts.Warning, *contracts.RejectedOutput, error) {
var warnings []contracts.Warning
for index, item := range chain.validators {
binding := item.resolved.Binding
var result contracts.ValidationResult
var err error
started := time.Now().UTC()
attemptPath := path.Join("validate", debugPathComponent(string(target.stage)), debugPathComponent(target.laneID), debugPathComponent(target.moduleKey), fmt.Sprintf("%02d-%s-attempt-%02d", index+1, debugPathComponent(binding.Module), attempt))
validatorCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
switch item.resolved.Target {
case ValidatorTargetTyped:
target.llmProfile = binding.LLMProfile
result, err = item.typedValidate(validatorCtx, item.typed, target)
case ValidatorTargetSerialized:
artifact, encodeErr := serializeArtifact(codec, target.value, true)
if encodeErr != nil {
err = encodeErr
break
}
result, err = item.serialized.Validate(validatorCtx, contracts.SerializedValidationRequest{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, Source: target.source, SourceID: target.sourceID, SourceInput: target.sourceInput.Clone(), SessionID: target.sessionID, References: CloneReferenceSet(target.references), LLMProfile: binding.LLMProfile, Metadata: cloneMetadata(target.metadata), Chunk: cloneSourceChunkPtr(target.chunk), Chunks: cloneSourceChunks(target.chunks), Schema: contracts.CloneArtifactSchema(artifact.Schema), MediaType: artifact.MediaType, Content: append([]byte(nil), artifact.Content...)})
default:
return nil, nil, fmt.Errorf("validator %q is incompatible with typed artifact validation", binding.Module)
}
artifact, _ := serializeArtifact(codec, target.value, true)
debugCall := debugValidationCall{ValidatorName: binding.Module, Request: map[string]any{"stage": string(target.stage), "lane_id": target.laneID, "module_key": target.moduleKey, "source_id": target.sourceID, "artifact": debugCheckpointArtifact(CheckpointArtifact{Artifact: artifact, SchemaDigest: contracts.DigestArtifactSchema(artifact.Schema)}), "metadata": redactSensitiveMap(target.metadata)}, Result: debugValidationResultEnvelope(result)}
if err != nil {
debugCall.Error = err.Error()
}
if debugErr := writeDebugTimed(debug, attemptPath+".json", debugEnvelopeWithLLMCalls(debugTimedEnvelope{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall, Error: debugCall.Error}, llmScope)); debugErr != nil {
return nil, nil, fmt.Errorf("write validation debug artifact: %w", debugErr)
}
if err != nil {
return nil, nil, fmt.Errorf("validate typed %s output with validator %q: %w", target.stage, binding.Module, err)
}
if !result.Approved {
reason := result.ReasonCode
if reason == "" {
reason = "artifact_rejected"
}
message := result.Message
if message == "" {
message = "artifact rejected"
}
return nil, &contracts.RejectedOutput{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, ChunkID: func() string {
if target.chunk != nil {
return target.chunk.ID
}
return ""
}(), ChunkIndex: func() int {
if target.chunk != nil {
return target.chunk.Index
}
return 0
}(), ValidatorName: binding.Module, ReasonCode: reason, Message: message, AttemptCount: attempt, DiagnosticArtifactPath: result.DiagnosticArtifactPath}, nil
}
warnings = append(warnings, result.Warnings...)
}
return warnings, nil, nil
}

View File

@@ -0,0 +1,45 @@
package pipeline
import (
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestDecodeCheckpointArtifactRejectsIncompatibleCodecIdentityAndBytes(t *testing.T) {
registry := NewArtifactCodecRegistry()
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
t.Fatalf("RegisterArtifactCodec: %v", err)
}
codec, _, err := registry.entry("test/notes")
if err != nil {
t.Fatalf("entry: %v", err)
}
artifact, err := serializeArtifact(codec, codecNotes{Items: []string{"one"}}, false)
if err != nil {
t.Fatalf("serializeArtifact: %v", err)
}
base := CheckpointArtifact{Artifact: artifact, SchemaDigest: contracts.DigestArtifactSchema(artifact.Schema)}
tests := []struct {
name string
mutate func(*CheckpointArtifact)
want string
}{
{name: "missing kind", mutate: func(v *CheckpointArtifact) { v.Artifact.Kind = "" }, want: "artifact kind"},
{name: "schema version", mutate: func(v *CheckpointArtifact) { v.Artifact.Schema.Version = "v999" }, want: "does not match codec schema"},
{name: "schema digest", mutate: func(v *CheckpointArtifact) { v.SchemaDigest = "sha256:different" }, want: "schema digest"},
{name: "media type", mutate: func(v *CheckpointArtifact) { v.Artifact.MediaType = "text/plain" }, want: "media type"},
{name: "decode failure", mutate: func(v *CheckpointArtifact) { v.Artifact.Content = []byte(`{"items":[`) }, want: "unexpected EOF"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
stored := cloneCheckpointArtifact(base)
test.mutate(&stored)
if _, err := decodeCheckpointArtifact(codec, stored); err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("decode error = %v, want %q", err, test.want)
}
})
}
}

View File

@@ -0,0 +1,154 @@
package pipeline
import (
"sync"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type lockedDebugRecorder struct {
inner DebugRecorder
mu sync.Mutex
}
func synchronizedDebugRecorder(inner DebugRecorder) DebugRecorder {
if _, ok := inner.(*lockedDebugRecorder); ok {
return inner
}
return &lockedDebugRecorder{inner: inner}
}
// SynchronizedDebugRecorder serializes access when a recorder is shared by
// pipeline operations and construction-injected LLM clients.
func SynchronizedDebugRecorder(inner DebugRecorder) DebugRecorder {
return synchronizedDebugRecorder(inner)
}
func (r *lockedDebugRecorder) Enabled() bool {
r.mu.Lock()
defer r.mu.Unlock()
return r.inner.Enabled()
}
func (r *lockedDebugRecorder) WriteJSON(name string, payload any) error {
r.mu.Lock()
defer r.mu.Unlock()
return r.inner.WriteJSON(name, payload)
}
func (r *lockedDebugRecorder) WriteBytes(name string, data []byte) error {
r.mu.Lock()
defer r.mu.Unlock()
return r.inner.WriteBytes(name, data)
}
type lockedCheckpointLoader struct {
inner CheckpointLoader
mu sync.Mutex
}
func synchronizedCheckpointLoader(inner CheckpointLoader) CheckpointLoader {
if _, ok := inner.(*lockedCheckpointLoader); ok {
return inner
}
return &lockedCheckpointLoader{inner: inner}
}
func (l *lockedCheckpointLoader) Enabled() bool {
l.mu.Lock()
defer l.mu.Unlock()
return l.inner.Enabled()
}
func (l *lockedCheckpointLoader) Source(key string) (SourceCheckpoint, CheckpointDecision) {
l.mu.Lock()
defer l.mu.Unlock()
return l.inner.Source(key)
}
func (l *lockedCheckpointLoader) Chunk(key, digest string) (ChunkCheckpoint, CheckpointDecision) {
l.mu.Lock()
defer l.mu.Unlock()
return l.inner.Chunk(key, digest)
}
func (l *lockedCheckpointLoader) Extract(lane, key string, deps []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
l.mu.Lock()
defer l.mu.Unlock()
return l.inner.Extract(lane, key, deps)
}
func (l *lockedCheckpointLoader) Merge(lane, key string, deps []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
l.mu.Lock()
defer l.mu.Unlock()
return l.inner.Merge(lane, key, deps)
}
func (l *lockedCheckpointLoader) Normalize(lane, key string, deps []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
l.mu.Lock()
defer l.mu.Unlock()
return l.inner.Normalize(lane, key, deps)
}
type lockedCheckpointRecorder struct {
inner CheckpointRecorder
mu sync.Mutex
}
func synchronizedCheckpointRecorder(inner CheckpointRecorder) CheckpointRecorder {
if _, ok := inner.(*lockedCheckpointRecorder); ok {
return inner
}
return &lockedCheckpointRecorder{inner: inner}
}
func (r *lockedCheckpointRecorder) call(fn func() error) error {
r.mu.Lock()
defer r.mu.Unlock()
return fn()
}
func (r *lockedCheckpointRecorder) SourceRunning(key string) error {
return r.call(func() error { return r.inner.SourceRunning(key) })
}
func (r *lockedCheckpointRecorder) SourceSucceeded(key string, doc *source.SourceDocument) error {
return r.call(func() error { return r.inner.SourceSucceeded(key, doc) })
}
func (r *lockedCheckpointRecorder) SourceFailed(key string, err error) error {
return r.call(func() error { return r.inner.SourceFailed(key, err) })
}
func (r *lockedCheckpointRecorder) ChunkRunning(key, digest string) error {
return r.call(func() error { return r.inner.ChunkRunning(key, digest) })
}
func (r *lockedCheckpointRecorder) ChunkSucceeded(key, digest string, chunks []source.Chunk, warnings []contracts.Warning) error {
return r.call(func() error { return r.inner.ChunkSucceeded(key, digest, chunks, warnings) })
}
func (r *lockedCheckpointRecorder) ChunkRejected(key, digest string, rejected contracts.RejectedOutput) error {
return r.call(func() error { return r.inner.ChunkRejected(key, digest, rejected) })
}
func (r *lockedCheckpointRecorder) ChunkFailed(key, digest string, err error) error {
return r.call(func() error { return r.inner.ChunkFailed(key, digest, err) })
}
func (r *lockedCheckpointRecorder) ExtractRunning(lane, key string, deps []CheckpointFingerprint) error {
return r.call(func() error { return r.inner.ExtractRunning(lane, key, deps) })
}
func (r *lockedCheckpointRecorder) ExtractSucceeded(lane, key string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
return r.call(func() error { return r.inner.ExtractSucceeded(lane, key, deps, outputs, rejected, warnings) })
}
func (r *lockedCheckpointRecorder) ExtractFailed(lane, key string, deps []CheckpointFingerprint, err error) error {
return r.call(func() error { return r.inner.ExtractFailed(lane, key, deps, err) })
}
func (r *lockedCheckpointRecorder) MergeRunning(lane, key string, deps []CheckpointFingerprint) error {
return r.call(func() error { return r.inner.MergeRunning(lane, key, deps) })
}
func (r *lockedCheckpointRecorder) MergeSucceeded(lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
return r.call(func() error { return r.inner.MergeSucceeded(lane, key, deps, output, warnings) })
}
func (r *lockedCheckpointRecorder) MergeRejected(lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
return r.call(func() error { return r.inner.MergeRejected(lane, key, deps, rejected) })
}
func (r *lockedCheckpointRecorder) MergeFailed(lane, key string, deps []CheckpointFingerprint, err error) error {
return r.call(func() error { return r.inner.MergeFailed(lane, key, deps, err) })
}
func (r *lockedCheckpointRecorder) NormalizeRunning(lane, key string, deps []CheckpointFingerprint) error {
return r.call(func() error { return r.inner.NormalizeRunning(lane, key, deps) })
}
func (r *lockedCheckpointRecorder) NormalizeSucceeded(lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
return r.call(func() error { return r.inner.NormalizeSucceeded(lane, key, deps, output, warnings) })
}
func (r *lockedCheckpointRecorder) NormalizeRejected(lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
return r.call(func() error { return r.inner.NormalizeRejected(lane, key, deps, rejected) })
}
func (r *lockedCheckpointRecorder) NormalizeFailed(lane, key string, deps []CheckpointFingerprint, err error) error {
return r.call(func() error { return r.inner.NormalizeFailed(lane, key, deps, err) })
}

View File

@@ -0,0 +1,63 @@
package pipeline
import (
"context"
"fmt"
"reflect"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type erasedExtractArtifact struct {
LaneID, ExtractorKey, SourceID, ChunkID string
ChunkIndex int
ChunkRef source.SourceRef
Value any
}
type erasedMergeArtifact struct {
LaneID, MergerKey, SourceID string
Value any
}
type erasedTypedResult struct {
Value any
Warnings []contracts.Warning
}
type typedValidationTarget struct {
stage ModuleStage
laneID string
moduleKey string
source *source.SourceDocument
sourceID string
sourceInput contracts.LLMInputMaterial
sessionID string
references contracts.ReferenceSet
llmProfile string
metadata map[string]any
chunk *source.Chunk
chunks []source.Chunk
ref source.SourceRef
value any
}
func exactTypedValue[T any](operation string, value any) (T, error) {
want := reflect.TypeFor[T]()
if reflect.TypeOf(value) != want {
var zero T
return zero, fmt.Errorf("%s: expected exact Go type %s, got %T", operation, want, value)
}
typed, ok := value.(T)
if !ok {
var zero T
return zero, fmt.Errorf("%s: expected exact Go type %s, got %T", operation, want, value)
}
return typed, nil
}
type typedExtractOperation func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error)
type typedMergeOperation func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error)
type typedNormalizeOperation func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error)
type typedValidateOperation func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error)

View File

@@ -0,0 +1,477 @@
package pipeline
import (
"context"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type typedTestExtractor[T any] struct{ key string }
func (e typedTestExtractor[T]) Key() string { return e.key }
func (typedTestExtractor[T]) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (typedTestExtractor[T]) Extract(context.Context, contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[T], error) {
return contracts.TypedExtractionResult[T]{}, nil
}
type typedTestMerger[T any] struct{ key string }
func (m typedTestMerger[T]) Key() string { return m.key }
func (typedTestMerger[T]) Merge(context.Context, contracts.TypedMergeRequest[T]) (contracts.TypedMergeResult[T], error) {
return contracts.TypedMergeResult[T]{}, nil
}
type typedTestNormalizer[T any] struct{ key string }
func (n typedTestNormalizer[T]) Key() string { return n.key }
func (typedTestNormalizer[T]) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (typedTestNormalizer[T]) Normalize(context.Context, contracts.TypedNormalizeRequest[T]) (contracts.TypedNormalizeResult[T], error) {
return contracts.TypedNormalizeResult[T]{}, nil
}
type typedTestValidator[T any] struct{ key string }
func (v typedTestValidator[T]) Name() string { return v.key }
func (typedTestValidator[T]) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassDeterministic
}
func (typedTestValidator[T]) Validate(context.Context, contracts.TypedValidationRequest[T]) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: true}, nil
}
type typedTestChunkValidator struct{ key string }
func (v typedTestChunkValidator) Name() string { return v.key }
func (typedTestChunkValidator) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassDeterministic
}
func (typedTestChunkValidator) Validate(context.Context, contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: true}, nil
}
type typedTestSerializedValidator struct{ key string }
type typedTestInput struct {
key string
doc *source.SourceDocument
}
func (v *typedTestInput) Key() string { return v.key }
func (v *typedTestInput) Parse(context.Context, contracts.ParseRequest) (*source.SourceDocument, error) {
return v.doc, nil
}
type typedTestChunker struct {
key string
chunks []source.Chunk
}
func (v *typedTestChunker) Key() string { return v.key }
func (v *typedTestChunker) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (v *typedTestChunker) Chunk(context.Context, contracts.ChunkRequest) (contracts.ChunkResult, error) {
return contracts.ChunkResult{Chunks: v.chunks}, nil
}
type typedTestOutput struct{ key string }
func (v *typedTestOutput) Key() string { return v.key }
func (v *typedTestOutput) Encode(context.Context, contracts.OutputRequest) (contracts.OutputResult, error) {
return contracts.OutputResult{}, nil
}
func typedTestDocument() *source.SourceDocument {
ref := source.SourceRef{SourceID: "source", StartUnitID: 1, EndUnitID: 1}
doc := &source.SourceDocument{ID: "source", Kind: "document", Format: "text/plain", Units: []source.SourceUnit{{ID: 1, Kind: "line", Text: "text", Ref: ref}}}
doc.Digest, _ = source.DigestDocument(doc)
return doc
}
func (v typedTestSerializedValidator) Name() string { return v.key }
func (typedTestSerializedValidator) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassDeterministic
}
func (typedTestSerializedValidator) Validate(context.Context, contracts.SerializedValidationRequest) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: true}, nil
}
type typedCatalogOptions struct {
registerNotesCodec bool
registerScoreCodec bool
registerNotesMerger bool
registerScoreMerger bool
registerNotesNormalizer bool
registerScoreNormalizer bool
registerScoreValidator bool
scoreExtractorUsesNotes bool
notesCodec testArtifactCodec[codecNotes]
scoreCodec testArtifactCodec[codecScore]
}
func completeTypedCatalogOptions() typedCatalogOptions {
return typedCatalogOptions{
registerNotesCodec: true,
registerScoreCodec: true,
registerNotesMerger: true,
registerScoreMerger: true,
registerNotesNormalizer: true,
registerScoreNormalizer: true,
registerScoreValidator: true,
notesCodec: notesCodec(),
scoreCodec: scoreCodec(),
}
}
func TestResolveTypedHeterogeneousLanes(t *testing.T) {
catalog := typedResolutionCatalog(t, completeTypedCatalogOptions())
resolved, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
if got, want := resolvedLaneIDs(resolved.ArtifactLanes), []string{"notes", "score"}; !reflect.DeepEqual(got, want) {
t.Fatalf("lane order = %#v, want %#v", got, want)
}
assertResolvedArtifactIdentity(t, resolved.ArtifactLanes[0], "test/notes", "notes.v1")
assertResolvedArtifactIdentity(t, resolved.ArtifactLanes[1], "test/score", "score.v1")
chunkChain := resolved.ValidatorChains[0]
if got := resolvedValidatorTargets(chunkChain.Validators); !reflect.DeepEqual(got, []ValidatorTarget{ValidatorTargetChunk, ValidatorTargetSerialized}) {
t.Fatalf("chunk validator targets = %#v, want chunk then serialized", got)
}
notesExtract := resolved.ValidatorChains[1]
if got := resolvedValidatorTargets(notesExtract.Validators); !reflect.DeepEqual(got, []ValidatorTarget{ValidatorTargetTyped, ValidatorTargetSerialized}) {
t.Fatalf("notes extract validator targets = %#v, want typed then serialized", got)
}
if notesExtract.Validators[0].ArtifactKind != "test/notes" || resolved.ValidatorChains[4].Validators[0].ArtifactKind != "test/score" {
t.Fatalf("resolved validator kinds = %#v, want lane kinds", resolved.ValidatorChains)
}
}
func TestPrepareConstructsHeterogeneousTypedLanes(t *testing.T) {
catalog := typedResolutionCatalog(t, completeTypedCatalogOptions())
resolved, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
prepared, err := Prepare(resolved, registriesFromModuleCatalog(catalog), ModuleDependencies{})
if err != nil {
t.Fatalf("Prepare() error = %v, want nil", err)
}
if len(prepared.ArtifactLanes) != 2 || prepared.lanes[0].typed == nil || prepared.lanes[1].typed == nil {
t.Fatalf("PreparedPipeline lanes = %#v, want two typed executors", prepared.ArtifactLanes)
}
}
func TestRunExecutesHeterogeneousTypedLanesWithCheckpointsDisabled(t *testing.T) {
catalog := typedResolutionCatalog(t, completeTypedCatalogOptions())
resolved, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, catalog)
if err != nil {
t.Fatalf("ResolvePipeline() error = %v", err)
}
prepared, err := Prepare(resolved, registriesFromModuleCatalog(catalog), ModuleDependencies{})
if err != nil {
t.Fatalf("Prepare() error = %v", err)
}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte(`{"source":true}`), RunID: "typed-run"})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if len(output.NormalizeOutputs) != 2 {
t.Fatalf("normalize outputs = %#v, want two typed lane results", output.NormalizeOutputs)
}
if len(output.CheckpointEvents) != 0 {
t.Fatalf("checkpoint events = %#v, want none with checkpoint loading disabled", output.CheckpointEvents)
}
}
func TestResolveTypedLaneRejectsIncompatibleComposition(t *testing.T) {
tests := []struct {
name string
mutate func(*typedCatalogOptions)
want string
}{
{name: "missing codec", mutate: func(options *typedCatalogOptions) { options.registerScoreCodec = false }, want: `artifact codec "test/score" is not registered`},
{name: "missing merger variant", mutate: func(options *typedCatalogOptions) { options.registerScoreMerger = false }, want: `merger "typed/merge" has no typed variant for artifact kind "test/score"`},
{name: "missing normalizer variant", mutate: func(options *typedCatalogOptions) { options.registerScoreNormalizer = false }, want: `normalizer "typed/normalize" has no typed variant for artifact kind "test/score"`},
{name: "extractor Go type mismatch", mutate: func(options *typedCatalogOptions) { options.scoreExtractorUsesNotes = true }, want: `artifact kind "test/score" requires Go type pipeline.codecScore, got pipeline.codecNotes`},
{name: "wrong validator kind", mutate: func(options *typedCatalogOptions) { options.registerScoreValidator = false }, want: `validator "typed/check" has no typed variant for artifact kind "test/score"; registered kinds: test/notes`},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
options := completeTypedCatalogOptions()
test.mutate(&options)
_, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, typedResolutionCatalog(t, options))
if err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("ResolvePipeline() error = %v, want %q", err, test.want)
}
})
}
}
func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) {
registry := NewMergerRegistry()
spec := ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"}
constructor := func() (contracts.Merger[codecNotes], error) {
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
}
if err := RegisterMerger(registry, spec, constructor); err != nil {
t.Fatalf("RegisterMerger() error = %v, want nil", err)
}
if err := RegisterMerger(registry, spec, constructor); err == nil || !strings.Contains(err.Error(), "already registered") {
t.Fatalf("duplicate RegisterMerger() error = %v, want duplicate variant error", err)
}
}
func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
extractors := NewExtractorRegistry()
if err := RegisterExtractor(extractors, ModuleSpec{Key: "typed/extract", Stage: StageExtract, ArtifactKind: "test/notes"}, func() (contracts.Extractor[codecNotes], error) {
return typedTestExtractor[codecNotes]{key: "typed/extract"}, nil
}); err != nil {
t.Fatalf("RegisterExtractor() error = %v", err)
}
if err := extractors.validateOptions("typed/extract", map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("extractor option validation error = %v, want unknown option", err)
}
mergers := NewMergerRegistry()
if err := RegisterMerger(mergers, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"}, func() (contracts.Merger[codecNotes], error) {
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
}); err != nil {
t.Fatalf("RegisterMerger() error = %v", err)
}
if err := mergers.validateOptions("typed/merge", "test/notes", map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("merger option validation error = %v, want unknown option", err)
}
normalizers := NewNormalizerRegistry()
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
return typedTestNormalizer[codecNotes]{key: "typed/normalize"}, nil
}); err != nil {
t.Fatalf("RegisterNormalizer() error = %v", err)
}
if err := normalizers.validateOptions("typed/normalize", "test/notes", map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("normalizer option validation error = %v, want unknown option", err)
}
validators := NewValidatorRegistry()
if err := RegisterTypedValidator(validators, "test/notes", ValidatorSpec{Key: "typed/check", ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.TypedValidator[codecNotes], error) {
return typedTestValidator[codecNotes]{key: "typed/check"}, nil
}); err != nil {
t.Fatalf("RegisterTypedValidator() error = %v", err)
}
if err := validators.validateOptions(ResolvedValidator{Binding: ModuleBinding{Module: "typed/check", Options: map[string]any{"unexpected": true}}, Target: ValidatorTargetTyped, ArtifactKind: "test/notes"}); err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("validator option validation error = %v, want unknown option", err)
}
}
func TestResolvedPipelineDigestIncludesArtifactSchemaIdentity(t *testing.T) {
baseOptions := completeTypedCatalogOptions()
base, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, typedResolutionCatalog(t, baseOptions))
if err != nil {
t.Fatalf("ResolvePipeline(base) error = %v, want nil", err)
}
identityOptions := completeTypedCatalogOptions()
identityOptions.notesCodec.schema.ID = "notes-renamed.v1"
identity, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, typedResolutionCatalog(t, identityOptions))
if err != nil {
t.Fatalf("ResolvePipeline(identity) error = %v, want nil", err)
}
if base.Digest == identity.Digest {
t.Fatalf("pipeline digest = %q after schema identity change, want different digest", identity.Digest)
}
digestOptions := completeTypedCatalogOptions()
digestOptions.notesCodec.schema.JSONSchema = []byte(`{"additionalProperties":false,"description":"changed","properties":{"items":{"items":{"type":"string"},"type":"array"}},"required":["items"],"type":"object"}`)
changedSchema, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, typedResolutionCatalog(t, digestOptions))
if err != nil {
t.Fatalf("ResolvePipeline(schema bytes) error = %v, want nil", err)
}
if base.Digest == changedSchema.Digest {
t.Fatalf("pipeline digest = %q after schema digest change, want different digest", changedSchema.Digest)
}
}
func typedResolutionCatalog(t *testing.T, options typedCatalogOptions) ModuleCatalog {
t.Helper()
catalog := ModuleCatalog{
Inputs: NewInputAdapterRegistry(),
Chunkers: NewChunkerRegistry(),
ArtifactCodecs: NewArtifactCodecRegistry(),
Extractors: NewExtractorRegistry(),
Mergers: NewMergerRegistry(),
Normalizers: NewNormalizerRegistry(),
Validators: NewValidatorRegistry(),
ValidatorChains: NewValidatorChainRegistry(),
Outputs: NewOutputEncoderRegistry(),
}
mustRegisterTypedTestBase(t, catalog)
if options.registerNotesCodec {
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, options.notesCodec)
}
if options.registerScoreCodec {
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, options.scoreCodec)
}
mustRegisterTypedExtractor(t, catalog.Extractors, "typed/extract-notes", "test/notes", typedTestExtractor[codecNotes]{key: "typed/extract-notes"})
if options.scoreExtractorUsesNotes {
mustRegisterTypedExtractor(t, catalog.Extractors, "typed/extract-score", "test/score", typedTestExtractor[codecNotes]{key: "typed/extract-score"})
} else {
mustRegisterTypedExtractor(t, catalog.Extractors, "typed/extract-score", "test/score", typedTestExtractor[codecScore]{key: "typed/extract-score"})
}
if options.registerNotesMerger {
mustRegisterTypedMerger(t, catalog.Mergers, "test/notes", typedTestMerger[codecNotes]{key: "typed/merge"})
}
if options.registerScoreMerger {
mustRegisterTypedMerger(t, catalog.Mergers, "test/score", typedTestMerger[codecScore]{key: "typed/merge"})
}
if options.registerNotesNormalizer {
mustRegisterTypedNormalizer(t, catalog.Normalizers, "test/notes", typedTestNormalizer[codecNotes]{key: "typed/normalize"})
}
if options.registerScoreNormalizer {
mustRegisterTypedNormalizer(t, catalog.Normalizers, "test/score", typedTestNormalizer[codecScore]{key: "typed/normalize"})
}
mustRegisterTypedValidator(t, catalog.Validators, "test/notes", typedTestValidator[codecNotes]{key: "typed/check"})
if options.registerScoreValidator {
mustRegisterTypedValidator(t, catalog.Validators, "test/score", typedTestValidator[codecScore]{key: "typed/check"})
}
if err := RegisterChunkValidator(catalog.Validators, ValidatorSpec{Key: "chunk/check", ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.ChunkValidator, error) {
return typedTestChunkValidator{key: "chunk/check"}, nil
}); err != nil {
t.Fatalf("RegisterChunkValidator() error = %v", err)
}
if err := RegisterSerializedValidator(catalog.Validators, SerializedValidatorSpec{
ValidatorSpec: ValidatorSpec{Key: "serialized/check", ExecutionClass: contracts.ExecutionClassDeterministic},
SupportsArtifacts: true,
}, func() (contracts.SerializedValidator, error) {
return typedTestSerializedValidator{key: "serialized/check"}, nil
}); err != nil {
t.Fatalf("RegisterSerializedValidator() error = %v", err)
}
if err := RegisterSerializedValidator(catalog.Validators, SerializedValidatorSpec{
ValidatorSpec: ValidatorSpec{Key: "serialized/chunks", ExecutionClass: contracts.ExecutionClassDeterministic},
SupportsChunks: true,
}, func() (contracts.SerializedValidator, error) {
return typedTestSerializedValidator{key: "serialized/chunks"}, nil
}); err != nil {
t.Fatalf("RegisterSerializedValidator(chunks) error = %v", err)
}
return catalog
}
func mustRegisterTypedTestBase(t *testing.T, catalog ModuleCatalog) {
t.Helper()
if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{Key: "typed/input", Stage: StageInput}, func() (contracts.InputAdapter, error) {
return &typedTestInput{key: "typed/input", doc: typedTestDocument()}, nil
}); err != nil {
t.Fatalf("register input: %v", err)
}
if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{Key: "typed/chunk", Stage: StageChunk}, func() (contracts.Chunker, error) {
doc := typedTestDocument()
return &typedTestChunker{key: "typed/chunk", chunks: []source.Chunk{{ID: "chunk-1", SourceID: doc.ID, Index: 0, Ref: doc.Units[0].Ref, Content: []byte(`{"chunk":1}`), MediaType: "application/json", Units: []source.SourceUnit{doc.Units[0]}}}}, nil
}); err != nil {
t.Fatalf("register chunker: %v", err)
}
if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{Key: "typed/output", Stage: StageOutput}, func() (contracts.OutputEncoder, error) {
return &typedTestOutput{key: "typed/output"}, nil
}); err != nil {
t.Fatalf("register output: %v", err)
}
}
func registriesFromModuleCatalog(catalog ModuleCatalog) Registries {
return Registries{
Inputs: catalog.Inputs, Chunkers: catalog.Chunkers, ArtifactCodecs: catalog.ArtifactCodecs,
Extractors: catalog.Extractors, Mergers: catalog.Mergers, Normalizers: catalog.Normalizers,
Validators: catalog.Validators, ValidatorChains: catalog.ValidatorChains, Outputs: catalog.Outputs,
}
}
func mustRegisterArtifactCodec[T any](t *testing.T, registry *ArtifactCodecRegistry, codec contracts.ArtifactCodec[T]) {
t.Helper()
if err := RegisterArtifactCodec(registry, codec); err != nil {
t.Fatalf("RegisterArtifactCodec() error = %v", err)
}
}
func mustRegisterTypedExtractor[T any](t *testing.T, registry *ExtractorRegistry, key string, kind contracts.ArtifactKind, extractor contracts.Extractor[T]) {
t.Helper()
if err := RegisterExtractor(registry, ModuleSpec{Key: key, Stage: StageExtract, ArtifactKind: kind}, func() (contracts.Extractor[T], error) { return extractor, nil }); err != nil {
t.Fatalf("RegisterExtractor() error = %v", err)
}
}
func mustRegisterTypedMerger[T any](t *testing.T, registry *MergerRegistry, kind contracts.ArtifactKind, merger contracts.Merger[T]) {
t.Helper()
if err := RegisterMerger(registry, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: kind}, func() (contracts.Merger[T], error) { return merger, nil }); err != nil {
t.Fatalf("RegisterMerger() error = %v", err)
}
}
func mustRegisterTypedNormalizer[T any](t *testing.T, registry *NormalizerRegistry, kind contracts.ArtifactKind, normalizer contracts.Normalizer[T]) {
t.Helper()
if err := RegisterNormalizer(registry, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: kind}, func() (contracts.Normalizer[T], error) { return normalizer, nil }); err != nil {
t.Fatalf("RegisterNormalizer() error = %v", err)
}
}
func mustRegisterTypedValidator[T any](t *testing.T, registry *ValidatorRegistry, kind contracts.ArtifactKind, validator contracts.TypedValidator[T]) {
t.Helper()
if err := RegisterTypedValidator(registry, kind, ValidatorSpec{Key: "typed/check", ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.TypedValidator[T], error) { return validator, nil }); err != nil {
t.Fatalf("RegisterTypedValidator() error = %v", err)
}
}
func typedResolutionProfile() PipelineProfile {
validatorOverride := ValidatorOverride{Set: true, Validators: []ModuleBinding{Binding("typed/check"), Binding("serialized/check")}}
return PipelineProfile{
ID: "typed-pipeline",
Input: Binding("typed/input"),
Chunk: ModuleBinding{Module: "typed/chunk", Validators: ValidatorOverride{Set: true, Validators: []ModuleBinding{Binding("chunk/check"), Binding("serialized/chunks")}}},
Artifacts: map[string]ArtifactLaneProfile{
"score": typedLaneProfile("typed/extract-score", validatorOverride),
"notes": typedLaneProfile("typed/extract-notes", validatorOverride),
},
Output: Binding("typed/output"),
}
}
func typedLaneProfile(extractor string, validators ValidatorOverride) ArtifactLaneProfile {
return ArtifactLaneProfile{
Extract: ModuleBinding{Module: extractor, Validators: validators},
Merge: Binding("typed/merge"),
Normalize: Binding("typed/normalize"),
}
}
func assertResolvedArtifactIdentity(t *testing.T, lane ResolvedArtifactLane, kind contracts.ArtifactKind, schemaID string) {
t.Helper()
if lane.ArtifactKind != kind || lane.ArtifactSchemaID != schemaID || lane.ArtifactSchemaName == "" || lane.ArtifactSchemaVersion == "" || lane.ArtifactSchemaDigest == "" {
t.Fatalf("resolved lane identity = %#v, want kind %q schema %q with complete metadata", lane, kind, schemaID)
}
}
func resolvedLaneIDs(lanes []ResolvedArtifactLane) []string {
ids := make([]string, len(lanes))
for i, lane := range lanes {
ids[i] = lane.ID
}
return ids
}
func resolvedValidatorTargets(validators []ResolvedValidator) []ValidatorTarget {
targets := make([]ValidatorTarget, len(validators))
for i, validator := range validators {
targets[i] = validator.Target
}
return targets
}
var _ contracts.Extractor[codecNotes] = typedTestExtractor[codecNotes]{}
var _ contracts.Merger[codecNotes] = typedTestMerger[codecNotes]{}
var _ contracts.Normalizer[codecNotes] = typedTestNormalizer[codecNotes]{}
var _ contracts.TypedValidator[codecNotes] = typedTestValidator[codecNotes]{}
var _ contracts.ChunkValidator = typedTestChunkValidator{}
var _ contracts.SerializedValidator = typedTestSerializedValidator{}

View File

@@ -1,125 +1,292 @@
package pipeline
import (
"context"
"fmt"
"reflect"
"sort"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type ValidatorConstructor func() (contracts.Validator, error)
type ValidatorSpec struct {
Key string `json:"key"`
ExecutionClass contracts.ExecutionClass `json:"execution_class"`
}
type SerializedValidatorSpec struct {
ValidatorSpec
SupportsChunks bool `json:"supports_chunks,omitempty"`
SupportsArtifacts bool `json:"supports_artifacts,omitempty"`
}
type ValidatorTarget string
const (
ValidatorTargetChunk ValidatorTarget = "chunk"
ValidatorTargetSerialized ValidatorTarget = "serialized"
ValidatorTargetTyped ValidatorTarget = "typed"
)
type ValidatorRegistry struct {
constructors map[string]ValidatorConstructor
specs map[string]ValidatorSpec
typedEntries map[artifactVariantKey]typedValidatorEntry
chunkEntries map[string]chunkValidatorEntry
serializedEntries map[string]serializedValidatorEntry
}
type typedValidatorEntry struct {
spec ValidatorSpec
kind contracts.ArtifactKind
valueType reflect.Type
validateOptions OptionValidator
builder func(BuildRequest) (any, error)
validate typedValidateOperation
}
type chunkValidatorEntry struct {
spec ValidatorSpec
validateOptions OptionValidator
builder func(BuildRequest) (contracts.ChunkValidator, error)
}
type serializedValidatorEntry struct {
spec SerializedValidatorSpec
validateOptions OptionValidator
builder func(BuildRequest) (contracts.SerializedValidator, error)
}
func NewValidatorRegistry() *ValidatorRegistry {
return &ValidatorRegistry{
constructors: make(map[string]ValidatorConstructor),
specs: make(map[string]ValidatorSpec),
typedEntries: make(map[artifactVariantKey]typedValidatorEntry),
chunkEntries: make(map[string]chunkValidatorEntry),
serializedEntries: make(map[string]serializedValidatorEntry),
}
}
func (r *ValidatorRegistry) Register(key string, constructor ValidatorConstructor) error {
return r.RegisterWithSpec(ValidatorSpec{Key: key, ExecutionClass: contracts.ExecutionClassDeterministic}, constructor)
func RegisterTypedValidator[T any](registry *ValidatorRegistry, kind contracts.ArtifactKind, spec ValidatorSpec, constructor func() (contracts.TypedValidator[T], error)) error {
if constructor == nil {
return fmt.Errorf("validator constructor for %q must not be nil", strings.TrimSpace(spec.Key))
}
return RegisterTypedValidatorBuilder(registry, kind, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.TypedValidator[T], error) {
return constructor()
})
}
func (r *ValidatorRegistry) RegisterWithSpec(spec ValidatorSpec, constructor ValidatorConstructor) error {
if r == nil {
func RegisterTypedValidatorBuilder[T any](registry *ValidatorRegistry, kind contracts.ArtifactKind, spec ValidatorSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.TypedValidator[T], error)) error {
if registry == nil {
return fmt.Errorf("validator registry must not be nil")
}
normalizedSpec, err := normalizeValidatorSpec(spec)
if err != nil {
return err
}
if constructor == nil {
return fmt.Errorf("validator constructor for %q must not be nil", normalizedSpec.Key)
kind = normalizeArtifactKind(kind)
if kind == "" {
return fmt.Errorf("typed validator %q artifact kind must not be empty", normalizedSpec.Key)
}
if _, ok := r.constructors[normalizedSpec.Key]; ok {
return fmt.Errorf("validator %q is already registered", normalizedSpec.Key)
if validateOptions == nil {
return fmt.Errorf("validator option validator for %q must not be nil", normalizedSpec.Key)
}
if r.constructors == nil {
r.constructors = make(map[string]ValidatorConstructor)
if builder == nil {
return fmt.Errorf("validator builder for %q must not be nil", normalizedSpec.Key)
}
if r.specs == nil {
r.specs = make(map[string]ValidatorSpec)
key := artifactVariantKey{module: normalizedSpec.Key, kind: kind}
if _, ok := registry.typedEntries[key]; ok {
return fmt.Errorf("validator %q variant for artifact kind %q is already registered", key.module, key.kind)
}
if registry.typedEntries == nil {
registry.typedEntries = make(map[artifactVariantKey]typedValidatorEntry)
}
registry.typedEntries[key] = typedValidatorEntry{
spec: normalizedSpec,
kind: kind,
valueType: reflect.TypeFor[T](),
validateOptions: validateOptions,
builder: func(request BuildRequest) (any, error) {
return builder(cloneBuildRequest(request))
},
validate: func(ctx context.Context, implementation any, target typedValidationTarget) (contracts.ValidationResult, error) {
validator, ok := implementation.(contracts.TypedValidator[T])
if !ok {
return contracts.ValidationResult{}, fmt.Errorf("validator %q has incompatible implementation %T", normalizedSpec.Key, implementation)
}
value, err := exactTypedValue[T]("validate artifact value", target.value)
if err != nil {
return contracts.ValidationResult{}, err
}
return validator.Validate(ctx, contracts.TypedValidationRequest[T]{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, Source: target.source, SourceID: target.sourceID, SourceInput: target.sourceInput, SessionID: target.sessionID, References: target.references, LLMProfile: target.llmProfile, Metadata: target.metadata, Chunk: target.chunk, Chunks: target.chunks, Ref: target.ref, Value: value})
},
}
r.constructors[normalizedSpec.Key] = constructor
r.specs[normalizedSpec.Key] = normalizedSpec
return nil
}
func (r *ValidatorRegistry) Build(key string) (contracts.Validator, error) {
if r == nil {
return nil, fmt.Errorf("validator registry must not be nil")
func RegisterChunkValidator(registry *ValidatorRegistry, spec ValidatorSpec, constructor func() (contracts.ChunkValidator, error)) error {
if constructor == nil {
return fmt.Errorf("validator constructor for %q must not be nil", strings.TrimSpace(spec.Key))
}
return RegisterChunkValidatorBuilder(registry, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.ChunkValidator, error) {
return constructor()
})
}
normalizedKey := strings.TrimSpace(key)
if normalizedKey == "" {
return nil, fmt.Errorf("validator key must not be empty")
func RegisterChunkValidatorBuilder(registry *ValidatorRegistry, spec ValidatorSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.ChunkValidator, error)) error {
if registry == nil {
return fmt.Errorf("validator registry must not be nil")
}
constructor, ok := r.constructors[normalizedKey]
if !ok {
return nil, fmt.Errorf("validator %q is not registered", normalizedKey)
}
validator, err := constructor()
normalizedSpec, err := normalizeValidatorSpec(spec)
if err != nil {
return nil, fmt.Errorf("build validator %q: %w", normalizedKey, err)
return err
}
if validateOptions == nil {
return fmt.Errorf("validator option validator for %q must not be nil", normalizedSpec.Key)
}
if builder == nil {
return fmt.Errorf("validator builder for %q must not be nil", normalizedSpec.Key)
}
if _, ok := registry.chunkEntries[normalizedSpec.Key]; ok {
return fmt.Errorf("chunk validator %q is already registered", normalizedSpec.Key)
}
if registry.chunkEntries == nil {
registry.chunkEntries = make(map[string]chunkValidatorEntry)
}
registry.chunkEntries[normalizedSpec.Key] = chunkValidatorEntry{spec: normalizedSpec, validateOptions: validateOptions, builder: builder}
return nil
}
func RegisterSerializedValidator(registry *ValidatorRegistry, spec SerializedValidatorSpec, constructor func() (contracts.SerializedValidator, error)) error {
if constructor == nil {
return fmt.Errorf("validator constructor for %q must not be nil", strings.TrimSpace(spec.Key))
}
return RegisterSerializedValidatorBuilder(registry, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.SerializedValidator, error) {
return constructor()
})
}
func RegisterSerializedValidatorBuilder(registry *ValidatorRegistry, spec SerializedValidatorSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.SerializedValidator, error)) error {
if registry == nil {
return fmt.Errorf("validator registry must not be nil")
}
normalizedValidatorSpec, err := normalizeValidatorSpec(spec.ValidatorSpec)
if err != nil {
return err
}
spec.ValidatorSpec = normalizedValidatorSpec
if !spec.SupportsChunks && !spec.SupportsArtifacts {
return fmt.Errorf("serialized validator %q must support chunks, artifacts, or both", spec.Key)
}
if validateOptions == nil {
return fmt.Errorf("validator option validator for %q must not be nil", spec.Key)
}
if builder == nil {
return fmt.Errorf("validator builder for %q must not be nil", spec.Key)
}
if _, ok := registry.serializedEntries[spec.Key]; ok {
return fmt.Errorf("serialized validator %q is already registered", spec.Key)
}
if registry.serializedEntries == nil {
registry.serializedEntries = make(map[string]serializedValidatorEntry)
}
registry.serializedEntries[spec.Key] = serializedValidatorEntry{spec: spec, validateOptions: validateOptions, builder: builder}
return nil
}
func (r *ValidatorRegistry) validateOptions(resolved ResolvedValidator) error {
if r == nil {
return fmt.Errorf("validator registry must not be nil")
}
key := strings.TrimSpace(resolved.Binding.Module)
var validator OptionValidator
switch resolved.Target {
case ValidatorTargetTyped:
entry, ok := r.typedEntry(key, resolved.ArtifactKind)
if ok {
validator = entry.validateOptions
}
case ValidatorTargetChunk:
entry, ok := r.chunkEntry(key)
if ok {
validator = entry.validateOptions
}
case ValidatorTargetSerialized:
entry, ok := r.serializedEntry(key)
if ok {
validator = entry.validateOptions
}
}
if validator == nil {
return nil, fmt.Errorf("validator %q constructor returned nil", normalizedKey)
return fmt.Errorf("validator %q construction entry is not registered", key)
}
if validator.Name() != normalizedKey {
return nil, fmt.Errorf("validator %q returned name %q", normalizedKey, validator.Name())
}
spec, ok := r.specs[normalizedKey]
if !ok {
return nil, fmt.Errorf("validator %q spec is not registered", normalizedKey)
}
if validator.ExecutionClass() != spec.ExecutionClass {
return nil, fmt.Errorf("validator %q returned execution class %q, want %q", normalizedKey, validator.ExecutionClass(), spec.ExecutionClass)
}
return validator, nil
return validateRegisteredOptions(validator, resolved.Binding.Options)
}
func (r *ValidatorRegistry) Spec(key string) (ValidatorSpec, bool) {
if r == nil {
return ValidatorSpec{}, false
}
spec, ok := r.specs[strings.TrimSpace(key)]
if !ok {
return ValidatorSpec{}, false
normalized := strings.TrimSpace(key)
if entry, found := r.chunkEntries[normalized]; found {
return entry.spec, true
}
return spec, true
if entry, found := r.serializedEntries[normalized]; found {
return entry.spec.ValidatorSpec, true
}
if kinds := r.registeredTypedKinds(normalized); len(kinds) > 0 {
entry, found := r.typedEntry(normalized, kinds[0])
return entry.spec, found
}
return ValidatorSpec{}, false
}
func (r *ValidatorRegistry) typedEntry(key string, kind contracts.ArtifactKind) (typedValidatorEntry, bool) {
if r == nil {
return typedValidatorEntry{}, false
}
entry, ok := r.typedEntries[artifactVariantKey{module: strings.TrimSpace(key), kind: normalizeArtifactKind(kind)}]
return entry, ok
}
func (r *ValidatorRegistry) chunkEntry(key string) (chunkValidatorEntry, bool) {
if r == nil {
return chunkValidatorEntry{}, false
}
entry, ok := r.chunkEntries[strings.TrimSpace(key)]
return entry, ok
}
func (r *ValidatorRegistry) serializedEntry(key string) (serializedValidatorEntry, bool) {
if r == nil {
return serializedValidatorEntry{}, false
}
entry, ok := r.serializedEntries[strings.TrimSpace(key)]
return entry, ok
}
func (r *ValidatorRegistry) registeredTypedKinds(key string) []contracts.ArtifactKind {
if r == nil {
return nil
}
module := strings.TrimSpace(key)
kinds := make([]contracts.ArtifactKind, 0)
for variant := range r.typedEntries {
if variant.module == module {
kinds = append(kinds, variant.kind)
}
}
sortArtifactKinds(kinds)
return kinds
}
func (r *ValidatorRegistry) RegisteredSpecs() []ValidatorSpec {
if r == nil || len(r.specs) == 0 {
if r == nil {
return nil
}
keys := make([]string, 0, len(r.specs))
for key := range r.specs {
keys = append(keys, key)
}
sort.Strings(keys)
keys := r.RegisteredKeys()
specs := make([]ValidatorSpec, 0, len(keys))
for _, key := range keys {
specs = append(specs, r.specs[key])
if spec, ok := r.Spec(key); ok {
specs = append(specs, spec)
}
}
return specs
}
@@ -128,15 +295,21 @@ func (r *ValidatorRegistry) RegisteredKeys() []string {
if r == nil {
return nil
}
return sortedRegistryKeys(r.constructors)
keys := make(map[string]struct{})
for key := range r.typedEntries {
keys[key.module] = struct{}{}
}
for key := range r.chunkEntries {
keys[key] = struct{}{}
}
for key := range r.serializedEntries {
keys[key] = struct{}{}
}
return sortedRegistryKeys(keys)
}
func normalizeValidatorSpec(spec ValidatorSpec) (ValidatorSpec, error) {
normalized := ValidatorSpec{
Key: strings.TrimSpace(spec.Key),
ExecutionClass: spec.ExecutionClass,
}
normalized := ValidatorSpec{Key: strings.TrimSpace(spec.Key), ExecutionClass: spec.ExecutionClass}
if normalized.Key == "" {
return ValidatorSpec{}, fmt.Errorf("validator key must not be empty")
}
@@ -147,3 +320,7 @@ func normalizeValidatorSpec(spec ValidatorSpec) (ValidatorSpec, error) {
}
return normalized, nil
}
func sortValidatorSpecs(specs []ValidatorSpec) {
sort.Slice(specs, func(i, j int) bool { return specs[i].Key < specs[j].Key })
}

View File

@@ -1,117 +0,0 @@
package pipeline
import (
"context"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestValidatorRegistryBehavior(t *testing.T) {
registry := NewValidatorRegistry()
if err := registry.Register(" generic-validator ", validatorConstructor("generic-validator", contracts.ExecutionClassDeterministic)); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
validator, err := registry.Build("generic-validator")
if err != nil {
t.Fatalf("Build() error = %v, want nil", err)
}
if validator.Name() != "generic-validator" {
t.Fatalf("validator name = %q, want generic-validator", validator.Name())
}
spec, ok := registry.Spec(" generic-validator ")
if !ok {
t.Fatal("Spec() ok = false, want true")
}
want := ValidatorSpec{Key: "generic-validator", ExecutionClass: contracts.ExecutionClassDeterministic}
if !reflect.DeepEqual(spec, want) {
t.Fatalf("Spec() = %#v, want %#v", spec, want)
}
}
func TestValidatorRegistryRegistersSpecs(t *testing.T) {
registry := NewValidatorRegistry()
spec := ValidatorSpec{Key: " llm-validator ", ExecutionClass: contracts.ExecutionClassLLMBacked}
if err := registry.RegisterWithSpec(spec, validatorConstructor("llm-validator", contracts.ExecutionClassLLMBacked)); err != nil {
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
}
got, ok := registry.Spec("llm-validator")
if !ok {
t.Fatal("Spec() ok = false, want true")
}
want := ValidatorSpec{Key: "llm-validator", ExecutionClass: contracts.ExecutionClassLLMBacked}
if !reflect.DeepEqual(got, want) {
t.Fatalf("Spec() = %#v, want %#v", got, want)
}
}
func TestValidatorRegistryRegisteredSpecsAreSorted(t *testing.T) {
registry := NewValidatorRegistry()
for _, key := range []string{"zeta", "alpha"} {
if err := registry.Register(key, validatorConstructor(key, contracts.ExecutionClassDeterministic)); err != nil {
t.Fatalf("Register(%q) error = %v", key, err)
}
}
specs := registry.RegisteredSpecs()
if len(specs) != 2 || specs[0].Key != "alpha" || specs[1].Key != "zeta" {
t.Fatalf("RegisteredSpecs() = %#v, want sorted specs", specs)
}
}
func TestValidatorRegistryRejectsUnsupportedExecutionClass(t *testing.T) {
registry := NewValidatorRegistry()
err := registry.RegisterWithSpec(
ValidatorSpec{Key: "invalid-validator", ExecutionClass: contracts.ExecutionClass("unsupported")},
validatorConstructor("invalid-validator", contracts.ExecutionClass("unsupported")),
)
if err == nil {
t.Fatal("RegisterWithSpec() error = nil, want unsupported execution class error")
}
}
func TestValidatorRegistryRejectsConstructorExecutionClassMismatch(t *testing.T) {
registry := NewValidatorRegistry()
if err := registry.RegisterWithSpec(
ValidatorSpec{Key: "validator", ExecutionClass: contracts.ExecutionClassDeterministic},
validatorConstructor("validator", contracts.ExecutionClassLLMBacked),
); err != nil {
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
}
_, err := registry.Build("validator")
if err == nil {
t.Fatal("Build() error = nil, want execution class mismatch")
}
if !strings.Contains(err.Error(), "execution class") {
t.Fatalf("Build() error = %q, want execution class context", err.Error())
}
}
type testValidator struct {
name string
executionClass contracts.ExecutionClass
}
func validatorConstructor(name string, executionClass contracts.ExecutionClass) ValidatorConstructor {
return func() (contracts.Validator, error) {
return testValidator{name: name, executionClass: executionClass}, nil
}
}
func (validator testValidator) Name() string {
return validator.name
}
func (validator testValidator) ExecutionClass() contracts.ExecutionClass {
return validator.executionClass
}
func (validator testValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: true}, nil
}

View File

@@ -1,462 +0,0 @@
package pipeline
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"os"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestWalkingSkeletonFixture(t *testing.T) {
inputBytes := readTestFixture(t, "testdata/walking_skeleton_input.json")
expectedBytes := readTestFixture(t, "testdata/walking_skeleton_output.json")
llmClient := &walkingSkeletonLLMClient{}
resolved, err := ResolvePipeline(walkingSkeletonProfile(), ResolveOptions{}, walkingSkeletonCatalog(t))
if err != nil {
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
}
output, err := New(walkingSkeletonRegistries(t)).Run(context.Background(), RunInput{
Pipeline: resolved,
SourceID: "fixture-source",
Path: "walking_skeleton_input.json",
RawInput: inputBytes,
LLMClient: llmClient,
})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if len(output.OutputFiles) != 1 {
t.Fatalf("len(OutputFiles) = %d, want 1", len(output.OutputFiles))
}
if output.OutputFiles[0].ContentType != "application/json" {
t.Fatalf("ContentType = %q, want application/json", output.OutputFiles[0].ContentType)
}
assertStructuralJSONEqual(t, output.OutputFiles[0].Bytes, expectedBytes)
if llmClient.calls != 3 {
t.Fatalf("LLM calls = %d, want extractor calls plus normalizer call", llmClient.calls)
}
}
func TestWalkingSkeletonResolutionRejectsMissingCapability(t *testing.T) {
catalog := walkingSkeletonCatalog(t)
catalog.Extractors = NewExtractorRegistry()
if err := catalog.Extractors.RegisterWithSpec(ModuleSpec{
Key: "fake/extract",
Stage: StageExtract,
Requires: []string{"missing"},
Provides: []string{"fake_artifacts"},
}, func() (contracts.Extractor, error) {
return walkingSkeletonExtractor{}, nil
}); err != nil {
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
}
_, err := ResolvePipeline(walkingSkeletonProfile(), ResolveOptions{}, catalog)
if err == nil {
t.Fatal("ResolvePipeline() error = nil, want error")
}
if !strings.Contains(err.Error(), "missing") {
t.Fatalf("ResolvePipeline() error = %q, want missing capability", err.Error())
}
}
func TestWalkingSkeletonResolutionRejectsUnknownOnlyLane(t *testing.T) {
_, err := ResolvePipeline(walkingSkeletonProfile(), ResolveOptions{Only: []string{"missing"}}, walkingSkeletonCatalog(t))
if err == nil {
t.Fatal("ResolvePipeline() error = nil, want error")
}
if !strings.Contains(err.Error(), "missing") || !strings.Contains(err.Error(), "not declared") {
t.Fatalf("ResolvePipeline() error = %q, want unknown lane error", err.Error())
}
}
func walkingSkeletonProfile() PipelineProfile {
return PipelineProfile{
ID: "walking-skeleton",
Input: Binding("fake/input"),
Chunk: Binding("fake/chunk"),
Output: Binding("json"),
Artifacts: map[string]ArtifactLaneProfile{
"events": {Extract: Binding("fake/extract")},
},
}
}
func walkingSkeletonCatalog(t *testing.T) ModuleCatalog {
t.Helper()
catalog := ModuleCatalog{
Inputs: NewInputAdapterRegistry(),
Chunkers: NewChunkerRegistry(),
Extractors: NewExtractorRegistry(),
Mergers: NewMergerRegistry(),
Normalizers: NewNormalizerRegistry(),
ValidatorChains: NewValidatorChainRegistry(),
Outputs: NewOutputEncoderRegistry(),
}
if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{
Key: "fake/input",
Stage: StageInput,
Provides: []string{"plain_text"},
}, func() (contracts.InputAdapter, error) {
return walkingSkeletonInput{}, nil
}); err != nil {
t.Fatalf("register fake input: %v", err)
}
if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{
Key: "fake/chunk",
Stage: StageChunk,
Requires: []string{"plain_text"},
Provides: []string{"chunks"},
}, func() (contracts.Chunker, error) {
return walkingSkeletonChunker{}, nil
}); err != nil {
t.Fatalf("register fake chunker: %v", err)
}
if err := catalog.Extractors.RegisterWithSpec(ModuleSpec{
Key: "fake/extract",
Stage: StageExtract,
Requires: []string{"chunks"},
Provides: []string{"fake_artifacts"},
}, func() (contracts.Extractor, error) {
return walkingSkeletonExtractor{}, nil
}); err != nil {
t.Fatalf("register fake extractor: %v", err)
}
if err := catalog.Mergers.RegisterWithSpec(ModuleSpec{
Key: DefaultMergeModule,
Stage: StageMerge,
Requires: []string{"fake_artifacts"},
}, func() (contracts.Merger, error) {
return walkingSkeletonMerger{}, nil
}); err != nil {
t.Fatalf("register append-order merger: %v", err)
}
if err := catalog.Normalizers.RegisterWithSpec(ModuleSpec{
Key: DefaultNormalizeModule,
Stage: StageNormalize,
}, func() (contracts.Normalizer, error) {
return walkingSkeletonNormalizer{}, nil
}); err != nil {
t.Fatalf("register no-op normalizer: %v", err)
}
if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{
Key: "json",
Stage: StageOutput,
}, func() (contracts.OutputEncoder, error) {
return walkingSkeletonOutput{}, nil
}); err != nil {
t.Fatalf("register fake output: %v", err)
}
return catalog
}
func walkingSkeletonRegistries(t *testing.T) Registries {
t.Helper()
catalog := walkingSkeletonCatalog(t)
return Registries{
Inputs: catalog.Inputs,
Chunkers: catalog.Chunkers,
Extractors: catalog.Extractors,
Mergers: catalog.Mergers,
Normalizers: catalog.Normalizers,
Outputs: catalog.Outputs,
}
}
type walkingSkeletonInput struct{}
func (input walkingSkeletonInput) Key() string {
return "fake/input"
}
func (input walkingSkeletonInput) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
var fixture struct {
ID string `json:"id"`
Units []struct {
ID int `json:"id"`
Text string `json:"text"`
} `json:"units"`
}
if err := json.Unmarshal(req.Raw, &fixture); err != nil {
return nil, err
}
units := make([]source.SourceUnit, 0, len(fixture.Units))
for _, unit := range fixture.Units {
units = append(units, source.SourceUnit{
ID: unit.ID,
Kind: "unit",
Text: unit.Text,
})
}
return &source.SourceDocument{
ID: fixture.ID,
Kind: "fixture",
Format: "application/json",
Digest: rawDigest(req.Raw),
Units: units,
}, nil
}
type walkingSkeletonChunker struct{}
func (chunker walkingSkeletonChunker) Key() string {
return "fake/chunk"
}
func (chunker walkingSkeletonChunker) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (chunker walkingSkeletonChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
if len(req.Source.Units) < 3 {
return contracts.ChunkResult{}, fmt.Errorf("fixture source must contain at least three units")
}
return contracts.ChunkResult{
Chunks: []contracts.SourceChunk{
{
ID: req.Source.ID + ":chunk:0",
SourceID: req.Source.ID,
Index: 0,
StartUnitID: req.Source.Units[0].ID,
EndUnitID: req.Source.Units[1].ID,
Content: []byte(`{"units":[1,2]}`),
MediaType: "application/json",
Units: append([]source.SourceUnit(nil), req.Source.Units[:2]...),
},
{
ID: req.Source.ID + ":chunk:1",
SourceID: req.Source.ID,
Index: 1,
StartUnitID: req.Source.Units[2].ID,
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
Content: []byte(`{"units":[3]}`),
MediaType: "application/json",
Units: append([]source.SourceUnit(nil), req.Source.Units[2:]...),
},
},
}, nil
}
type walkingSkeletonExtractor struct{}
func (extractor walkingSkeletonExtractor) Key() string {
return "fake/extract"
}
func (extractor walkingSkeletonExtractor) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (extractor walkingSkeletonExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
var response struct {
Call int `json:"call"`
}
if _, err := req.LLMClient.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
StageName: "fake/extract",
PromptID: "fake.event",
PromptVersion: "v1",
}, &response); err != nil {
return contracts.ExtractionResult{}, err
}
payload, err := json.Marshal(map[string]any{
"chunk_id": req.Chunk.ID,
"llm_call": response.Call,
"text": chunkText(req.Chunk.Units),
})
if err != nil {
return contracts.ExtractionResult{}, err
}
return contracts.ExtractionResult{
Output: contracts.ExtractOutput{
Schema: contracts.ResponseSchema{ID: "fake_event", Name: "fake_event", Version: "v1"},
Payload: contracts.RawPayload{
Content: payload,
MediaType: "application/json",
},
},
}, nil
}
type walkingSkeletonLLMClient struct {
calls int
}
func (client *walkingSkeletonLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
client.calls++
if response, ok := out.(*struct {
Call int `json:"call"`
}); ok {
response.Call = client.calls
}
content, err := json.Marshal(map[string]any{"call": client.calls})
if err != nil {
return contracts.StructuredCompletionResponse{}, err
}
return contracts.StructuredCompletionResponse{
Content: content,
}, nil
}
type walkingSkeletonMerger struct{}
func (merger walkingSkeletonMerger) Key() string {
return DefaultMergeModule
}
func (merger walkingSkeletonMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
outputs := make([]json.RawMessage, 0, len(req.ExtractOutputs))
for _, output := range req.ExtractOutputs {
outputs = append(outputs, json.RawMessage(output.Payload.Content))
}
content, err := json.Marshal(map[string]any{"outputs": outputs})
if err != nil {
return contracts.MergeResult{}, err
}
return contracts.MergeResult{
Output: contracts.MergeOutput{
LaneID: req.LaneID,
SourceID: req.Source.ID,
Schema: contracts.ResponseSchema{ID: "fake_event", Name: "fake_event", Version: "v1"},
Payload: contracts.RawPayload{
Content: content,
MediaType: "application/json",
},
},
}, nil
}
type walkingSkeletonNormalizer struct{}
func (normalizer walkingSkeletonNormalizer) Key() string {
return DefaultNormalizeModule
}
func (normalizer walkingSkeletonNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
return nil
}
func (normalizer walkingSkeletonNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
var response struct {
Call int `json:"call"`
}
if _, err := req.LLMClient.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
StageName: "fake/normalize",
PromptID: "fake.normalize",
PromptVersion: "v1",
}, &response); err != nil {
return contracts.NormalizeResult{}, err
}
return contracts.NormalizeResult{
Output: contracts.NormalizeOutput{
LaneID: req.LaneID,
SourceID: req.MergeOutput.SourceID,
Schema: req.MergeOutput.Schema,
Payload: req.MergeOutput.Payload,
},
}, nil
}
type walkingSkeletonOutput struct{}
func (output walkingSkeletonOutput) Key() string {
return "json"
}
func (output walkingSkeletonOutput) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
type rawOutput struct {
LaneID string `json:"lane_id"`
NormalizerKey string `json:"normalizer_key"`
SourceID string `json:"source_id"`
Schema contracts.ResponseSchema `json:"schema"`
MediaType string `json:"media_type"`
Content json.RawMessage `json:"content"`
}
rawOutputs := make([]rawOutput, 0, len(req.NormalizeOutputs))
for _, output := range req.NormalizeOutputs {
rawOutputs = append(rawOutputs, rawOutput{
LaneID: output.LaneID,
NormalizerKey: output.NormalizerKey,
SourceID: output.SourceID,
Schema: output.Schema,
MediaType: output.Payload.MediaType,
Content: json.RawMessage(output.Payload.Content),
})
}
encoded, err := json.Marshal(struct {
Manifest artifacts.RunManifest `json:"manifest"`
NormalizeOutputs []rawOutput `json:"normalize_outputs"`
}{
Manifest: artifacts.RunManifest{
PipelineID: req.Manifest.PipelineID,
PipelineDigest: req.Manifest.PipelineDigest,
ArtifactLanes: req.Manifest.ArtifactLanes,
ValidationStatus: req.Manifest.ValidationStatus,
},
NormalizeOutputs: rawOutputs,
})
if err != nil {
return contracts.OutputResult{}, err
}
return contracts.OutputResult{
Files: []contracts.OutputFile{
{Name: "output.json", ContentType: "application/json", Bytes: encoded},
},
}, nil
}
func readTestFixture(t *testing.T, path string) []byte {
t.Helper()
bytes, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read fixture %q: %v", path, err)
}
return bytes
}
func assertStructuralJSONEqual(t *testing.T, gotBytes, wantBytes []byte) {
t.Helper()
var got any
if err := json.Unmarshal(gotBytes, &got); err != nil {
t.Fatalf("unmarshal actual JSON: %v\n%s", err, gotBytes)
}
var want any
if err := json.Unmarshal(wantBytes, &want); err != nil {
t.Fatalf("unmarshal expected JSON: %v\n%s", err, wantBytes)
}
if !reflect.DeepEqual(got, want) {
gotFormatted, _ := json.MarshalIndent(got, "", " ")
wantFormatted, _ := json.MarshalIndent(want, "", " ")
t.Fatalf("actual JSON:\n%s\nwant:\n%s", gotFormatted, wantFormatted)
}
}
func chunkText(units []source.SourceUnit) string {
parts := make([]string, 0, len(units))
for _, unit := range units {
parts = append(parts, unit.Text)
}
return strings.Join(parts, " ")
}
func rawDigest(raw []byte) string {
sum := sha256.Sum256(raw)
return "sha256:" + hex.EncodeToString(sum[:])
}

View File

@@ -1,4 +1,4 @@
package sharedassets
package promptfs
import (
"bytes"

View File

@@ -1,4 +1,4 @@
package sharedassets
package promptfs
import (
"io/fs"

View File

@@ -1,30 +0,0 @@
package scenes
import "gitea.maximumdirect.net/eric/notarius/internal/modules/sharedassets/dnd"
type chunkResponse struct {
Scenes []sceneResponse `json:"scenes"`
BoundaryCaveats []string `json:"boundary_caveats"`
}
type sceneResponse struct {
StartUnitID dnd.UnitRef `json:"start_unit_id"`
EndUnitID dnd.UnitRef `json:"end_unit_id"`
ShortTitle string `json:"short_title"`
PrimaryMode string `json:"primary_mode"`
MainParticipants []string `json:"main_participants"`
Summary string `json:"summary"`
BoundaryNote string `json:"boundary_note"`
BoundaryConfidence string `json:"boundary_confidence"`
}
type normalizedScene struct {
StartUnitID int
EndUnitID int
ShortTitle string
PrimaryMode string
MainParticipants []string
Summary string
BoundaryNote string
BoundaryConfidence string
}

View File

@@ -9,7 +9,7 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/sharedassets/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
)
const Key = "dnd/scenes"
@@ -23,7 +23,7 @@ var providedCapabilities = []string{
"chunks.scenes",
}
var referenceSlotDescriptions = dnd.ReferenceSlotDescriptions{
var referenceSlotDescriptions = shared.ReferenceSlotDescriptions{
Glossary: "Optional campaign glossary reference material used only for scene disambiguation.",
Party: "Optional party roster reference material used only for scene disambiguation.",
Players: "Optional player list reference material used only for scene disambiguation.",
@@ -33,10 +33,17 @@ var referenceSlotDescriptions = dnd.ReferenceSlotDescriptions{
var _ contracts.Chunker = (*Chunker)(nil)
var _ contracts.ManifestMetadataProvider = (*Chunker)(nil)
type Chunker struct{}
type Options struct{}
func New() *Chunker {
return &Chunker{}
type Chunker struct {
llm contracts.StructuredLLMClient
}
func New(llmClient contracts.StructuredLLMClient, _ Options) (*Chunker, error) {
if llmClient == nil {
return nil, chunkerErrorf("LLM client must not be nil")
}
return &Chunker{llm: llmClient}, nil
}
func (c *Chunker) Key() string {
@@ -44,7 +51,7 @@ func (c *Chunker) Key() string {
}
func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
return dnd.ReferenceSlots(referenceSlotDescriptions)
return shared.ReferenceSlots(referenceSlotDescriptions)
}
func (c *Chunker) ManifestMetadata() map[string]any {
@@ -71,6 +78,9 @@ func (c *Chunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contra
if c == nil {
return contracts.ChunkResult{}, chunkerErrorf("chunker must not be nil")
}
if c.llm == nil {
return contracts.ChunkResult{}, chunkerErrorf("LLM client must not be nil")
}
if ctx == nil {
return contracts.ChunkResult{}, chunkerErrorf("context must not be nil")
}
@@ -86,21 +96,14 @@ func (c *Chunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contra
if err := source.ValidateDocument(req.Source); err != nil {
return contracts.ChunkResult{}, chunkerErrorf("validate source document: %w", err)
}
if req.LLMClient == nil {
return contracts.ChunkResult{}, chunkerErrorf("LLM client must not be nil")
}
if len(req.Options) > 0 {
return contracts.ChunkResult{}, chunkerErrorf("options are not supported")
}
var response chunkResponse
if _, err := req.LLMClient.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
if _, err := c.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
StageName: Key,
PromptID: PromptID,
PromptVersion: ResponseSchemaVersion,
ProfileID: req.LLMProfile,
SessionID: req.SessionID,
Inputs: dnd.PromptInputs(req.SourceInput, req.References),
Inputs: shared.PromptInputs(req.SourceInput, req.References),
}, &response); err != nil {
return contracts.ChunkResult{}, chunkerErrorf("complete structured output: %w", err)
}
@@ -125,17 +128,33 @@ func ModuleSpec() pipeline.ModuleSpec {
Stage: pipeline.StageChunk,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ReferenceSlots: dnd.ReferenceSlots(referenceSlotDescriptions),
ReferenceSlots: shared.ReferenceSlots(referenceSlotDescriptions),
}
}
func Register(registry *pipeline.ChunkerRegistry) error {
return registry.RegisterWithSpec(ModuleSpec(), func() (contracts.Chunker, error) {
return New(), nil
return registry.RegisterBuilderWithSpec(ModuleSpec(), validateOptions, func(request pipeline.BuildRequest) (contracts.Chunker, error) {
options, err := DecodeOptions(request.Options)
if err != nil {
return nil, err
}
return New(request.Dependencies.LLM, options)
})
}
func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]contracts.SourceChunk, error) {
func validateOptions(options map[string]any) error {
_, err := DecodeOptions(options)
return err
}
func DecodeOptions(options map[string]any) (Options, error) {
if err := pipeline.RejectUnknownOptions(options); err != nil {
return Options{}, chunkerErrorf("%w", err)
}
return Options{}, nil
}
func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]source.Chunk, error) {
if response.Scenes == nil {
return nil, fmt.Errorf("scenes must be present")
}
@@ -148,7 +167,7 @@ func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]c
unitIndexes[unit.ID] = i
}
chunks := make([]contracts.SourceChunk, 0, len(response.Scenes))
chunks := make([]source.Chunk, 0, len(response.Scenes))
previousEnd := -1
for i, scene := range response.Scenes {
normalized, err := normalizeScene(doc, i, scene)
@@ -186,15 +205,18 @@ func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]c
if err != nil {
return nil, err
}
chunks = append(chunks, contracts.SourceChunk{
ID: fmt.Sprintf("scene-%06d", i+1),
SourceID: doc.ID,
Index: i,
StartUnitID: units[0].ID,
EndUnitID: units[len(units)-1].ID,
Content: content,
MediaType: "application/json",
Units: units,
chunks = append(chunks, source.Chunk{
ID: fmt.Sprintf("scene-%06d", i+1),
SourceID: doc.ID,
Index: i,
Ref: source.SourceRef{
SourceID: doc.ID,
StartUnitID: units[0].Ref.StartUnitID,
EndUnitID: units[len(units)-1].Ref.EndUnitID,
},
Content: content,
MediaType: "application/json",
Units: units,
Metadata: map[string]any{
"scene_title": normalized.ShortTitle,
"primary_mode": normalized.PrimaryMode,
@@ -228,11 +250,11 @@ func chunkContent(units []source.SourceUnit) ([]byte, error) {
}
func normalizeScene(doc *source.SourceDocument, index int, scene sceneResponse) (normalizedScene, error) {
startUnitID, err := dnd.ResolveUnitID(doc, "start_unit_id", scene.StartUnitID)
startUnitID, err := shared.ResolveUnitID(doc, "start_unit_id", scene.StartUnitID)
if err != nil {
return normalizedScene{}, fmt.Errorf("scene[%d] %w", index, err)
}
endUnitID, err := dnd.ResolveUnitID(doc, "end_unit_id", scene.EndUnitID)
endUnitID, err := shared.ResolveUnitID(doc, "end_unit_id", scene.EndUnitID)
if err != nil {
return normalizedScene{}, fmt.Errorf("scene[%d] %w", index, err)
}
@@ -332,6 +354,7 @@ func cloneUnits(units []source.SourceUnit) []source.SourceUnit {
ID: unit.ID,
Kind: unit.Kind,
Text: unit.Text,
Ref: unit.Ref,
Metadata: cloneMetadata(unit.Metadata),
})
}

View File

@@ -11,14 +11,12 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/sharedassets/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
)
func TestNewModuleSpecAndRegister(t *testing.T) {
chunker := New()
if chunker == nil {
t.Fatal("New() = nil, want chunker")
}
client := &fakeScenesLLMClient{}
chunker := newChunker(t, client)
if chunker.Key() != Key {
t.Fatalf("Key() = %q, want %q", chunker.Key(), Key)
}
@@ -52,7 +50,7 @@ func TestNewModuleSpecAndRegister(t *testing.T) {
if !reflect.DeepEqual(registered, want) {
t.Fatalf("registered spec = %#v, want %#v", registered, want)
}
built, err := registry.Build(Key)
built, err := registry.BuildWithRequest(Key, pipeline.BuildRequest{Dependencies: pipeline.ModuleDependencies{LLM: client}})
if err != nil {
t.Fatalf("Build(%q) error = %v, want nil", Key, err)
}
@@ -62,6 +60,18 @@ func TestNewModuleSpecAndRegister(t *testing.T) {
if slots := built.ReferenceSlots(); !reflect.DeepEqual(slots, want.ReferenceSlots) {
t.Fatalf("ReferenceSlots() = %#v, want %#v", slots, want.ReferenceSlots)
}
if err := registry.ValidateOptions(Key, map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("ValidateOptions() error = %v, want unknown option error", err)
}
}
func TestConstructionRejectsMissingDependencyAndUnknownOptions(t *testing.T) {
if _, err := New(nil, Options{}); err == nil || !strings.Contains(err.Error(), "LLM client") {
t.Fatalf("New(nil) error = %v, want LLM client error", err)
}
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("DecodeOptions() error = %v, want unknown option error", err)
}
}
func TestRegisterNilRegistryReturnsError(t *testing.T) {
@@ -105,8 +115,8 @@ func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
response: chunkResponse{
Scenes: []sceneResponse{
{
StartUnitID: dnd.UnitRefFromInt(1),
EndUnitID: dnd.UnitRefFromInt(2),
StartUnitID: shared.UnitRefFromInt(1),
EndUnitID: shared.UnitRefFromInt(2),
ShortTitle: " Goblin parley ",
PrimaryMode: "Discussion",
MainParticipants: []string{" Aria ", "Goblin scout"},
@@ -115,8 +125,8 @@ func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
BoundaryConfidence: "High",
},
{
StartUnitID: dnd.UnitRefFromInt(3),
EndUnitID: dnd.UnitRefFromInt(4),
StartUnitID: shared.UnitRefFromInt(3),
EndUnitID: shared.UnitRefFromInt(4),
ShortTitle: "Ambush at the gate",
PrimaryMode: "Combat",
MainParticipants: []string{"Aria", "Goblin ambushers"},
@@ -129,7 +139,7 @@ func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
},
}
result, err := New().Chunk(context.Background(), chunkRequestWithClient(client))
result, err := newChunker(t, client).Chunk(context.Background(), chunkRequest())
if err != nil {
t.Fatalf("Chunk() error = %v, want nil", err)
}
@@ -179,8 +189,8 @@ func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
if first.SourceID != "session-alpha" || first.Index != 0 {
t.Fatalf("first chunk = %#v, want source and index fields", first)
}
if first.StartUnitID != 1 || first.EndUnitID != 2 {
t.Fatalf("first boundaries = %d-%d, want 1-2", first.StartUnitID, first.EndUnitID)
if first.Ref != (source.SourceRef{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 2}) {
t.Fatalf("first ref = %#v, want session-alpha:1-2", first.Ref)
}
if first.MediaType != "application/json" || len(first.Content) == 0 {
t.Fatalf("first payload = media type %q length %d, want JSON content", first.MediaType, len(first.Content))
@@ -210,8 +220,8 @@ func TestChunkPassesReferencesAsPromptInputs(t *testing.T) {
client := &fakeScenesLLMClient{response: chunkResponse{
Scenes: []sceneResponse{
{
StartUnitID: dnd.UnitRefFromInt(1),
EndUnitID: dnd.UnitRefFromInt(4),
StartUnitID: shared.UnitRefFromInt(1),
EndUnitID: shared.UnitRefFromInt(4),
ShortTitle: "Ambush",
PrimaryMode: "Combat",
MainParticipants: []string{"Aria"},
@@ -221,7 +231,7 @@ func TestChunkPassesReferencesAsPromptInputs(t *testing.T) {
},
},
}}
req := chunkRequestWithClient(client)
req := chunkRequest()
req.References = contracts.ReferenceSet{
Slots: map[string]contracts.ResolvedReferenceSlot{
"players": {
@@ -245,7 +255,7 @@ func TestChunkPassesReferencesAsPromptInputs(t *testing.T) {
},
}
if _, err := New().Chunk(context.Background(), req); err != nil {
if _, err := newChunker(t, client).Chunk(context.Background(), req); err != nil {
t.Fatalf("Chunk() error = %v, want nil", err)
}
request := client.requests[0]
@@ -264,7 +274,7 @@ func TestChunkPassesReferencesAsPromptInputs(t *testing.T) {
}
func TestPromptInputsMapLegacyRosterReferenceToParty(t *testing.T) {
inputs := dnd.PromptInputs(sceneSourceInput(), contracts.ReferenceSet{
inputs := shared.PromptInputs(sceneSourceInput(), contracts.ReferenceSet{
Slots: map[string]contracts.ResolvedReferenceSlot{
"roster": {
Slot: contracts.ReferenceSlot{Name: "roster"},
@@ -293,7 +303,7 @@ func TestChunkRejectsWhitespaceOnlyBoundaryCaveats(t *testing.T) {
},
}
_, err := New().Chunk(context.Background(), chunkRequestWithClient(client))
_, err := newChunker(t, client).Chunk(context.Background(), chunkRequest())
if err == nil {
t.Fatal("Chunk() error = nil, want malformed structured output error")
}
@@ -306,24 +316,27 @@ func TestChunkDefensivelyCopiesSourceUnitsAndMetadata(t *testing.T) {
doc := sceneSourceDocument()
client := &fakeScenesLLMClient{response: validSceneResponse()}
result, err := New().Chunk(context.Background(), contracts.ChunkRequest{
result, err := newChunker(t, client).Chunk(context.Background(), contracts.ChunkRequest{
Source: doc,
SourceInput: sceneSourceInput(),
SessionID: "session-123",
LLMProfile: "profile-scenes",
LLMClient: client,
})
if err != nil {
t.Fatalf("Chunk() error = %v, want nil", err)
}
doc.Units[0].ID = 99
doc.Units[0].Ref.SourceID = "mutated"
doc.Units[0].Metadata["speaker"] = "mutated"
client.response.Scenes[0].MainParticipants[0] = "mutated"
if result.Chunks[0].Units[0].ID != 1 {
t.Fatalf("chunk unit ID changed after source mutation: %#v", result.Chunks[0].Units[0])
}
if got := result.Chunks[0].Units[0].Ref.SourceID; got != "session-alpha" {
t.Fatalf("chunk unit ref changed after source mutation: %#v", result.Chunks[0].Units[0].Ref)
}
if result.Chunks[0].Units[0].Metadata["speaker"] != "Alice" {
t.Fatalf("chunk unit metadata changed after source mutation: %#v", result.Chunks[0].Units[0].Metadata)
}
@@ -334,7 +347,7 @@ func TestChunkDefensivelyCopiesSourceUnitsAndMetadata(t *testing.T) {
}
func TestChunkerManifestMetadataIncludesPromptAndSchemaProvenance(t *testing.T) {
metadata := New().ManifestMetadata()
metadata := newChunker(t, &fakeScenesLLMClient{}).ManifestMetadata()
tests := map[string]string{
"prompt_id": PromptID,
@@ -364,7 +377,7 @@ func TestChunkerManifestMetadataIncludesPromptAndSchemaProvenance(t *testing.T)
func TestChunkRejectsInvalidRequests(t *testing.T) {
validClient := &fakeScenesLLMClient{response: validSceneResponse()}
validReq := chunkRequestWithClient(validClient)
validReq := chunkRequest()
canceledCtx, cancel := context.WithCancel(context.Background())
cancel()
invalidDoc := sceneSourceDocument()
@@ -380,13 +393,11 @@ func TestChunkRejectsInvalidRequests(t *testing.T) {
want string
}{
{name: "nil chunker", chunker: nil, ctx: context.Background(), req: validReq, want: "chunker"},
{name: "nil context", chunker: New(), ctx: nil, req: validReq, want: "context"},
{name: "canceled context", chunker: New(), ctx: canceledCtx, req: validReq, want: "context"},
{name: "nil source", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{LLMClient: validClient}, want: "source"},
{name: "empty source units", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{Source: emptyDoc, LLMClient: validClient}, want: "units"},
{name: "invalid source", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{Source: invalidDoc, LLMClient: validClient}, want: "validate source document"},
{name: "nil LLM client", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{Source: sceneSourceDocument()}, want: "LLM client"},
{name: "unsupported options", chunker: New(), ctx: context.Background(), req: requestWithOptions(validReq), want: "options"},
{name: "nil context", chunker: newChunker(t, validClient), ctx: nil, req: validReq, want: "context"},
{name: "canceled context", chunker: newChunker(t, validClient), ctx: canceledCtx, req: validReq, want: "context"},
{name: "nil source", chunker: newChunker(t, validClient), ctx: context.Background(), req: contracts.ChunkRequest{}, want: "source"},
{name: "empty source units", chunker: newChunker(t, validClient), ctx: context.Background(), req: contracts.ChunkRequest{Source: emptyDoc}, want: "units"},
{name: "invalid source", chunker: newChunker(t, validClient), ctx: context.Background(), req: contracts.ChunkRequest{Source: invalidDoc}, want: "validate source document"},
}
for _, tt := range tests {
@@ -451,8 +462,8 @@ func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
name: "empty metadata field",
response: replaceScenes(validSceneResponse(), []sceneResponse{
{
StartUnitID: dnd.UnitRefFromInt(1),
EndUnitID: dnd.UnitRefFromInt(4),
StartUnitID: shared.UnitRefFromInt(1),
EndUnitID: shared.UnitRefFromInt(4),
ShortTitle: " ",
PrimaryMode: "Narrative",
MainParticipants: []string{"Aria"},
@@ -467,8 +478,8 @@ func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
name: "empty participant",
response: replaceScenes(validSceneResponse(), []sceneResponse{
{
StartUnitID: dnd.UnitRefFromInt(1),
EndUnitID: dnd.UnitRefFromInt(4),
StartUnitID: shared.UnitRefFromInt(1),
EndUnitID: shared.UnitRefFromInt(4),
ShortTitle: "Title",
PrimaryMode: "Narrative",
MainParticipants: []string{"Aria", " "},
@@ -484,7 +495,7 @@ func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
client := &fakeScenesLLMClient{response: tt.response}
_, err := New().Chunk(context.Background(), chunkRequestWithClient(client))
_, err := newChunker(t, client).Chunk(context.Background(), chunkRequest())
if err == nil {
t.Fatal("Chunk() error = nil, want error")
}
@@ -498,7 +509,7 @@ func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
func TestChunkWrapsLLMClientError(t *testing.T) {
client := &fakeScenesLLMClient{err: errors.New("provider unavailable")}
_, err := New().Chunk(context.Background(), chunkRequestWithClient(client))
_, err := newChunker(t, client).Chunk(context.Background(), chunkRequest())
if err == nil {
t.Fatal("Chunk() error = nil, want LLM error")
}
@@ -507,13 +518,12 @@ func TestChunkWrapsLLMClientError(t *testing.T) {
}
}
func chunkRequestWithClient(client contracts.StructuredLLMClient) contracts.ChunkRequest {
func chunkRequest() contracts.ChunkRequest {
return contracts.ChunkRequest{
Source: sceneSourceDocument(),
SourceInput: sceneSourceInput(),
SessionID: "session-123",
LLMProfile: "profile-scenes",
LLMClient: client,
}
}
@@ -523,9 +533,13 @@ func sceneSourceInput() contracts.LLMInputMaterial {
return contracts.NewLLMInputMaterial("source", "application/json", []byte(sceneTranscriptJSON), "sha256:transcript", "file:///session-alpha.json")
}
func requestWithOptions(req contracts.ChunkRequest) contracts.ChunkRequest {
req.Options = map[string]any{"max_units": 2}
return req
func newChunker(t *testing.T, client contracts.StructuredLLMClient) *Chunker {
t.Helper()
chunker, err := New(client, Options{})
if err != nil {
t.Fatalf("New() error = %v, want nil", err)
}
return chunker
}
func sceneSourceDocument() *source.SourceDocument {
@@ -535,10 +549,10 @@ func sceneSourceDocument() *source.SourceDocument {
Format: "application/vnd.seriatim.minimal+json",
Digest: "sha256:source",
Units: []source.SourceUnit{
{ID: 1, Kind: "transcript_segment", Text: "Aria asks whether the goblin will parley.", Metadata: map[string]any{"speaker": "Alice"}},
{ID: 2, Kind: "transcript_segment", Text: "The goblin scout describes the gate guards."},
{ID: 3, Kind: "transcript_segment", Text: "The guards rush out with blades drawn."},
{ID: 4, Kind: "transcript_segment", Text: "The party defeats the ambushers."},
{ID: 1, Kind: "transcript_segment", Text: "Aria asks whether the goblin will parley.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 1}, Metadata: map[string]any{"speaker": "Alice"}},
{ID: 2, Kind: "transcript_segment", Text: "The goblin scout describes the gate guards.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 2}},
{ID: 3, Kind: "transcript_segment", Text: "The guards rush out with blades drawn.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 3, EndUnitID: 3}},
{ID: 4, Kind: "transcript_segment", Text: "The party defeats the ambushers.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 4, EndUnitID: 4}},
},
}
}
@@ -559,8 +573,8 @@ func replaceScenes(response chunkResponse, scenes []sceneResponse) chunkResponse
func scene(startUnitID int, endUnitID int) sceneResponse {
return sceneResponse{
StartUnitID: dnd.UnitRefFromInt(startUnitID),
EndUnitID: dnd.UnitRefFromInt(endUnitID),
StartUnitID: shared.UnitRefFromInt(startUnitID),
EndUnitID: shared.UnitRefFromInt(endUnitID),
ShortTitle: "Scene title",
PrimaryMode: "Narrative",
MainParticipants: []string{"Aria"},
@@ -570,7 +584,7 @@ func scene(startUnitID int, endUnitID int) sceneResponse {
}
}
func chunkIDs(chunks []contracts.SourceChunk) []string {
func chunkIDs(chunks []source.Chunk) []string {
ids := make([]string, 0, len(chunks))
for _, chunk := range chunks {
ids = append(ids, chunk.ID)

View File

@@ -0,0 +1,30 @@
package scenes
import "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
type chunkResponse struct {
Scenes []sceneResponse `json:"scenes"`
BoundaryCaveats []string `json:"boundary_caveats"`
}
type sceneResponse struct {
StartUnitID shared.UnitRef `json:"start_unit_id"`
EndUnitID shared.UnitRef `json:"end_unit_id"`
ShortTitle string `json:"short_title"`
PrimaryMode string `json:"primary_mode"`
MainParticipants []string `json:"main_participants"`
Summary string `json:"summary"`
BoundaryNote string `json:"boundary_note"`
BoundaryConfidence string `json:"boundary_confidence"`
}
type normalizedScene struct {
StartUnitID int
EndUnitID int
ShortTitle string
PrimaryMode string
MainParticipants []string
Summary string
BoundaryNote string
BoundaryConfidence string
}

Some files were not shown because too many files have changed in this diff Show More