Compare commits
18 Commits
35f9446ed8
...
3013ee044d
| Author | SHA1 | Date | |
|---|---|---|---|
| 3013ee044d | |||
| adfd3bd052 | |||
| 4023c66508 | |||
| adfe3825ee | |||
| 814fcdc6ba | |||
| 66de1a5520 | |||
| 52e6b31408 | |||
| 142ba36695 | |||
| b949e9bbc0 | |||
| ce3a07512f | |||
| 1c84d19e5f | |||
| fc1b57bde2 | |||
| 075888c97f | |||
| 40709e4ad8 | |||
| 15c369c509 | |||
| a81b9f1e1f | |||
| 0327659355 | |||
| c99bad19ae |
@@ -1,6 +1,6 @@
|
||||
# ADR-0002: Linear pipes-and-filters pipeline, not a general DAG
|
||||
|
||||
**Status:** Proposed
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-07-13
|
||||
|
||||
## Context
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# ADR-0003: Strongly typed stage interfaces with a two-zone data model
|
||||
|
||||
**Status:** Proposed
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-07-13
|
||||
|
||||
## Context
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# ADR-0004: Package modules by domain, not by stage
|
||||
|
||||
**Status:** Proposed
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-07-13
|
||||
|
||||
## Context
|
||||
|
||||
@@ -41,6 +41,7 @@ rejected; execution profiles now come from Scriptorium.
|
||||
Built-in defaults:
|
||||
|
||||
- `concurrency.total_llm`: `1`
|
||||
- `concurrency.stage_workers.extract`: effective `concurrency.total_llm`
|
||||
- `diagnostics.work_dir`: `/tmp/notarius`
|
||||
- `diagnostics.retention`: `auto`
|
||||
- `workspace.directory`: unset
|
||||
@@ -88,6 +89,7 @@ These environment variables are applied after the config file:
|
||||
|
||||
- `NOTARIUS_CONFIG`: config discovery path.
|
||||
- `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency.
|
||||
- `NOTARIUS_STAGE_WORKERS_EXTRACT`: integer extract worker limit.
|
||||
- `NOTARIUS_WORKSPACE_DIR`: workspace directory.
|
||||
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`: boolean diagnostics enablement.
|
||||
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`: workspace diagnostics retention
|
||||
@@ -107,6 +109,25 @@ The removed `NOTARIUS_LLM_DEFAULT_*` variables are not read. Configure provider
|
||||
endpoint, model, and credential environment variable names through Scriptorium
|
||||
profiles.
|
||||
|
||||
## Concurrency
|
||||
|
||||
`concurrency` fields:
|
||||
|
||||
- `total_llm`: positive integer ceiling on concurrent provider calls.
|
||||
- `stage_workers`: optional map of framework worker limits. The only supported
|
||||
key is `extract`.
|
||||
|
||||
`stage_workers.extract` defaults to the effective `total_llm` value after file
|
||||
and environment precedence. It must be between `1` and `total_llm`, inclusive.
|
||||
Unknown or empty stage-worker keys are rejected. The environment override
|
||||
`NOTARIUS_STAGE_WORKERS_EXTRACT` takes precedence over the file value, as does
|
||||
`NOTARIUS_TOTAL_LLM_CONCURRENCY` for the global ceiling.
|
||||
|
||||
The worker value is present in effective and redacted configuration. It bounds
|
||||
the fixed run-wide extract pool and its bounded dispatch queue. Extract jobs are
|
||||
submitted by source chunk and then resolved lane; `total_llm` independently
|
||||
bounds actual provider calls made by extracts, retries, and validators.
|
||||
|
||||
## Pipelines
|
||||
|
||||
A pipeline selects implementations for the fixed workflow defined by
|
||||
@@ -236,6 +257,10 @@ Binding fields:
|
||||
chain; set a non-empty list to use exactly those validators in configured
|
||||
order.
|
||||
|
||||
During resolution, each selected module's registered option validator runs.
|
||||
Production input, chunk, and output bindings reject unknown or invalid options
|
||||
with the affected binding context.
|
||||
|
||||
Validator bindings use the same shorthand or object module-binding form, but
|
||||
only these fields are supported:
|
||||
|
||||
@@ -258,9 +283,9 @@ production validators do not call the LLM and must not set `llm_profile`.
|
||||
| input | `seriatim` | Reads Seriatim transcript JSON. |
|
||||
| chunk | `generic` | Splits source units into ordered chunks. |
|
||||
| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. |
|
||||
| extract | `dnd/spells` | Extracts D&D spell raw outputs. |
|
||||
| merge | `appendorder` | Merges JSON raw extract outputs in chunk order. |
|
||||
| normalize | `noop` | Passes merged raw outputs through unchanged. |
|
||||
| extract | `dnd/spells` | Extracts typed D&D spell-list artifacts. |
|
||||
| merge | `appendorder` | Combines typed artifacts in chunk order. |
|
||||
| normalize | `noop` | Passes merged typed artifacts through unchanged. |
|
||||
| output | `json` | Produces JSON output files for normalized `application/json` lanes. |
|
||||
|
||||
## Implemented Production Validators
|
||||
@@ -271,7 +296,7 @@ production validators do not call the LLM and must not set `llm_profile`.
|
||||
| `generic/always_reject` | deterministic | Rejects returned module output with reason `always_reject`. |
|
||||
| `generic/valid_json` | deterministic | Rejects payloads that are not syntactically valid JSON. |
|
||||
| `generic/valid_json_schema` | deterministic | Rejects invalid JSON or JSON that does not conform to the module response schema. |
|
||||
| `extract/dnd/spells/shape` | deterministic | Rejects malformed D&D spell-cast JSON payloads. |
|
||||
| `extract/dnd/spells/shape` | deterministic | Rejects malformed D&D spell-list artifacts. |
|
||||
| `extract/dnd/spells/source_refs` | deterministic | Rejects missing or invalid D&D spell source references. |
|
||||
| `extract/dnd/spells/source_relatedness` | deterministic | Emits warnings when a spell name is not found near its cited source text. |
|
||||
|
||||
@@ -362,6 +387,8 @@ Configuration validation checks:
|
||||
- mutually exclusive `scriptorium.profile_dir` and `scriptorium.profile_file`;
|
||||
- non-empty, non-duplicated IDs after trimming;
|
||||
- positive global LLM concurrency;
|
||||
- supported stage-worker keys and an effective extract worker count in the
|
||||
inclusive range `1..concurrency.total_llm`;
|
||||
- supported diagnostics retention and non-empty work directory;
|
||||
- stale removed fields such as `llm_profiles`.
|
||||
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
# D&D Spell Raw Output
|
||||
# D&D Spell Artifact
|
||||
|
||||
This document is the durable raw output contract for the production D&D spell
|
||||
extractor. Selectable extractor keys are cataloged in
|
||||
This document is the durable serialized artifact contract for the production
|
||||
D&D spell extractor. Selectable extractor keys are cataloged in
|
||||
[Configuration](../config.md#implemented-production-modules).
|
||||
|
||||
## Identity
|
||||
|
||||
- Artifact kind: `dnd/spell-list`
|
||||
- Prompt ID: `dnd.spells`
|
||||
- Response schema key: `dnd_spells`
|
||||
- Response schema ID: `notarius.dnd.spells`
|
||||
@@ -13,6 +14,12 @@ extractor. Selectable extractor keys are cataloged in
|
||||
- Response schema version: `v1`
|
||||
- Media type: `application/json`
|
||||
|
||||
The durable JSON Schema is owned by the D&D spell artifact codec. The
|
||||
extractor's private LLM response schema is a separate transport contract: its
|
||||
source-reference objects omit `source_id`, which the extractor assigns while
|
||||
mapping the response to the canonical artifact. The LLM DTO and transport
|
||||
schema are not part of this durable contract.
|
||||
|
||||
The output contains canonical spell casts derived from transcript evidence.
|
||||
Source IDs are assigned from the input identity; source-unit ranges identify
|
||||
the evidence location.
|
||||
|
||||
@@ -15,7 +15,7 @@ The encoder writes:
|
||||
|
||||
- `index.json`
|
||||
- `manifest.json`
|
||||
- `lanes/<lane-id>.json`, one file per normalized raw lane output
|
||||
- `lanes/<lane-id>.json`, one file per normalized serialized artifact
|
||||
- `rejected.json`
|
||||
- `warnings.json`
|
||||
|
||||
@@ -112,8 +112,8 @@ Reference `stage` is `chunk`, `extract`, `merge`, or `normalize`. `lane_id` is
|
||||
omitted for chunk references and present for extract, merge, and normalize
|
||||
references.
|
||||
|
||||
`validation_status` is `approved` when no raw outputs were rejected and
|
||||
`rejected` when one or more raw outputs were rejected.
|
||||
`validation_status` is `approved` when no outputs were rejected and `rejected`
|
||||
when one or more outputs were rejected.
|
||||
|
||||
`validator_chains` records the resolved validator chain for each validation
|
||||
point. Entries include stage, lane ID when applicable, module key, and validators
|
||||
@@ -131,12 +131,13 @@ message, attempt count, and optional diagnostic artifact path.
|
||||
|
||||
## Output Payload Files
|
||||
|
||||
Each normalized raw output is written to `lanes/<sanitized-lane-id>.json`.
|
||||
The JSON output encoder accepts only `application/json` normalized outputs. The
|
||||
file contains the raw JSON payload pretty-printed.
|
||||
Each normalized serialized artifact is written to
|
||||
`lanes/<sanitized-lane-id>.json`. The JSON output encoder is domain-neutral and
|
||||
accepts only artifacts whose codec media type is `application/json`. The file
|
||||
contains the codec-owned JSON bytes pretty-printed.
|
||||
|
||||
The schema of each lane payload is owned by that artifact contract. For the
|
||||
current D&D spell lane, see [D&D Spell Raw Output](dnd-spell-artifacts.md).
|
||||
current D&D spell lane, see [D&D Spell Artifact](dnd-spell-artifacts.md).
|
||||
|
||||
## `rejected.json`
|
||||
|
||||
@@ -148,7 +149,7 @@ Shape:
|
||||
}
|
||||
```
|
||||
|
||||
When raw output validation rejects an output, each entry contains `stage` and
|
||||
When output validation rejects an output, each entry contains `stage` and
|
||||
`message`. It includes `lane_id`, `module_key`, `chunk_id`, `chunk_index`,
|
||||
`validator_name`, `reason_code`, `attempt_count`, and
|
||||
`diagnostic_artifact_path` when applicable.
|
||||
|
||||
@@ -52,9 +52,14 @@ Notarius identifies the parsed source in this order:
|
||||
2. `metadata.source_id`, when it is a non-empty string after trimming;
|
||||
3. `seriatim:<first-16-hex-chars-of-raw-sha256>`.
|
||||
|
||||
The source digest recorded in output provenance is `sha256:<hex>` of the exact
|
||||
raw input bytes. Segment IDs become the unit IDs used by artifact source
|
||||
references.
|
||||
The exact raw input SHA-256 remains the basis of the fallback source ID. The
|
||||
source digest recorded in output provenance is instead the SHA-256 of the
|
||||
canonical generic source document, excluding the digest field itself. It covers
|
||||
the derived source identity, document kind and format, ordered units and their
|
||||
self-references, and accepted metadata. Segment IDs become the unit IDs used by
|
||||
artifact source references; each produced unit carries a self-reference whose
|
||||
source ID is the derived document ID and whose start and end IDs both equal the
|
||||
segment ID.
|
||||
|
||||
## Compatibility Limit
|
||||
|
||||
|
||||
@@ -28,12 +28,20 @@ without exposing Scriptorium types through stage contracts.
|
||||
|
||||
`internal/cli` constructs the production runtime by:
|
||||
|
||||
1. collecting embedded prompt and response-schema assets from production module
|
||||
packages;
|
||||
1. allocating the asset registry populated by the generic, Seriatim, and D&D
|
||||
package-family registrars;
|
||||
2. creating a `ScriptoriumClient` from the effective profile source;
|
||||
3. attaching an `LLMProfileRecorder`;
|
||||
4. creating a scheduler from the effective concurrency limit;
|
||||
5. returning a `ScheduledClient` wrapper.
|
||||
5. returning a `ScheduledClient` wrapper;
|
||||
6. decorating that shared client before preparation when debug recording is
|
||||
enabled; and
|
||||
7. injecting that one shared client into complete pipeline preparation before
|
||||
the source file is read or the runner is invoked.
|
||||
|
||||
The D&D scene chunker and spell extractor retain this injected client and use
|
||||
it for every structured completion. Operation requests do not carry an LLM
|
||||
client.
|
||||
|
||||
The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
|
||||
and validator bindings. It prepares a small internal check prompt for each ID so
|
||||
@@ -75,12 +83,20 @@ cancelled waiter that has already received a permit releases it.
|
||||
defers release on every result path. The effective limit and default are
|
||||
configuration facts in [Configuration](../config.md#defaults).
|
||||
|
||||
This provider-call ceiling is independent of the pipeline's extract worker
|
||||
limit. Concurrent lanes, retries, and validators all use the same scheduled
|
||||
client, so increasing framework workers cannot exceed `total_llm`. Pipeline
|
||||
dispatch and cancellation mechanics are documented in
|
||||
[Pipeline Internals](pipeline.md#execution-flow).
|
||||
|
||||
## Prompt And Schema Assets
|
||||
|
||||
`AssetRegistry` combines caller-owned prompt filesystems under stable prefixes
|
||||
and rejects invalid or conflicting registrations. Production module packages
|
||||
register their own prompt and schema assets; generic framework code contains no
|
||||
D&D prompt content.
|
||||
D&D prompt content. `internal/framework/promptfs` provides the domain-neutral
|
||||
filesystem composition helper used to combine module-owned files with shared
|
||||
domain prompt fragments.
|
||||
|
||||
Schema helpers load embedded JSON Schema with identity and digest metadata,
|
||||
return defensive copies, and expose a diagnostics map that omits schema bytes.
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
# Module And Validator Internals
|
||||
|
||||
Production stage implementations live under `internal/modules`; production
|
||||
validators live under `internal/validators`. The selectable keys, configuration
|
||||
options, reference slots, and default validator chain are canonical in the
|
||||
Production module and validator implementations live under their domain-first
|
||||
trees in `internal/modules`.
|
||||
The selectable keys, configuration options, reference slots, and default
|
||||
validator chain are canonical in the
|
||||
[module](../config.md#implemented-production-modules) and
|
||||
[validator](../config.md#implemented-production-validators) catalogs in
|
||||
Configuration.
|
||||
@@ -12,10 +13,27 @@ Configuration.
|
||||
A stage module package provides a stable key, constructor, contract
|
||||
implementation, `ModuleSpec`, `Register`, and focused behavior and registration
|
||||
tests. A validator package follows the same pattern with `ValidatorSpec` and the
|
||||
validator registry.
|
||||
validator registry. Package-family registrars compose those leaf registrations
|
||||
into the production catalog and own family-level policy such as default
|
||||
validator chains and prompt asset collection.
|
||||
|
||||
Production input, chunk, output, and D&D spell-extract packages register strict
|
||||
option decoders and run-local builders. Preparation decodes their options into
|
||||
implementation-owned values and injects dependencies. The spell extractor is
|
||||
typed over the canonical D&D model. D&D validators, merge, and normalize use
|
||||
typed variants; JSON representation validators use serialized requests; and
|
||||
unconditional validators expose separate chunk and typed variants. The D&D
|
||||
production registrar registers only the canonical typed spell implementations.
|
||||
|
||||
Prepared extractors, extract validators, and codecs may be reused concurrently
|
||||
by the run-wide extract pool. Production implementations are immutable after
|
||||
construction: they retain only typed options, immutable assets, or the shared
|
||||
concurrency-safe LLM client. Implementations that introduce mutable state must
|
||||
synchronize that state without creating a separate provider scheduler.
|
||||
|
||||
Specs expose capability and execution metadata without constructing an
|
||||
implementation. Chunk, extract, merge, and normalize modules that accept
|
||||
implementation. Registry entries separately expose option validation and
|
||||
run-local construction. Chunk, extract, merge, and normalize modules that accept
|
||||
auxiliary material declare identical reference slots from both
|
||||
`ReferenceSlots()` and `ModuleSpec().ReferenceSlots`; registration tests enforce
|
||||
that agreement. Runtime delivery uses the corresponding stage request's
|
||||
@@ -23,10 +41,10 @@ that agreement. Runtime delivery uses the corresponding stage request's
|
||||
|
||||
LLM-backed extensions own their prompt definitions and response schemas under
|
||||
package-local embedded assets. Shared filesystem composition belongs in
|
||||
`internal/modules/sharedassets`; reusable D&D prompt fragments, reference
|
||||
`internal/framework/promptfs`; reusable D&D prompt fragments, reference
|
||||
declarations, prompt-input assembly, and source-unit helpers belong in
|
||||
`internal/modules/sharedassets/dnd`. Stage contracts expose only Notarius
|
||||
structured-completion types, not Scriptorium public types.
|
||||
`internal/modules/dnd/shared`. Stage contracts expose only Notarius structured-
|
||||
completion types, not Scriptorium public types.
|
||||
|
||||
Reference material may inform a module or prompt but must not become source
|
||||
evidence. The resolver and materializer behavior is described in
|
||||
@@ -34,12 +52,13 @@ evidence. The resolver and materializer behavior is described in
|
||||
|
||||
## Input Adapter
|
||||
|
||||
### `internal/modules/input/seriatim`
|
||||
### `internal/modules/seriatim/input/transcript`
|
||||
|
||||
The adapter decodes the supported transcript JSON, selects the source identity,
|
||||
computes the raw-input digest, validates segments, and maps each segment into a
|
||||
generic source unit with speaker and timestamp metadata. Its spec advertises the
|
||||
transcript capabilities consumed by D&D modules.
|
||||
computes canonical source provenance, validates segments, and maps each segment
|
||||
into a generic source unit with a self-reference plus speaker and timestamp
|
||||
metadata. It accepts no module options. Its spec advertises the transcript
|
||||
capabilities consumed by D&D modules.
|
||||
|
||||
Parsing is strict about required values and duplicate unit IDs but deliberately
|
||||
ignores unrelated Seriatim fields. The external format and derived-identity
|
||||
@@ -48,23 +67,29 @@ rules are defined in the
|
||||
|
||||
## Chunkers
|
||||
|
||||
### `internal/modules/chunk/generic`
|
||||
### `internal/modules/generic/chunk/units`
|
||||
|
||||
The generic chunker validates the source document, walks units in configured
|
||||
windows, clones each selected unit, and emits deterministic ordered chunk IDs.
|
||||
Overlap changes the next window start but never reorders units. It records the
|
||||
first and last unit and unit count in chunk metadata.
|
||||
first and last unit and unit count in chunk metadata, and derives the chunk's
|
||||
canonical source reference from those unit references.
|
||||
|
||||
The accepted options and defaults are defined in
|
||||
[Configuration](../config.md#implemented-production-modules). Generic
|
||||
framework validation canonicalizes the returned unit slices before extraction.
|
||||
The chunker decodes its options during construction and retains only the typed
|
||||
window settings used by `Chunk`.
|
||||
|
||||
### `internal/modules/chunk/dnd/scenes`
|
||||
### `internal/modules/dnd/chunk/scenes`
|
||||
|
||||
The scene chunker prepares a structured Scriptorium request from the full
|
||||
transcript, session, and optional D&D reference inputs. It validates the model's
|
||||
scene boundaries against source-unit IDs and converts them into deterministic
|
||||
chunks.
|
||||
chunks with canonical source references spanning each scene's units.
|
||||
Preparation injects the shared structured LLM client into the chunker; `Chunk`
|
||||
supplies only the run-specific profile, session, source, references, and
|
||||
metadata.
|
||||
|
||||
Scene validation requires sequential, contiguous, non-overlapping coverage from
|
||||
the first source unit through the last. Each chunk contains JSON scene content
|
||||
@@ -79,40 +104,42 @@ file types remain canonical in [Configuration](../config.md).
|
||||
|
||||
## Extractor
|
||||
|
||||
### `internal/modules/extract/dnd/spells`
|
||||
### `internal/modules/dnd/extract/spells`
|
||||
|
||||
The spell extractor prepares a structured request from one chunk, the
|
||||
chunk-scoped source input, the session, and optional D&D reference inputs. It
|
||||
decodes the model response, assigns the generic source identity to every source
|
||||
reference, canonicalizes duplicate references, orders spell casts by their
|
||||
earliest cited unit, and returns raw JSON plus response-schema provenance.
|
||||
earliest cited unit, and returns `dnd.SpellList`.
|
||||
|
||||
The package owns its embedded prompt, response schemas, and prompt/schema
|
||||
manifest metadata. Shared D&D helpers keep prompt input names and source-unit
|
||||
reference conversion consistent with the scene chunker. The extractor produces
|
||||
raw output; production validators own approval policy.
|
||||
The extractor owns its private model-response DTO, embedded prompt, LLM response
|
||||
schema, strict option decoder, injected shared LLM client, and prompt/schema
|
||||
manifest metadata. The separate `internal/modules/dnd/codec/spells` package
|
||||
owns the durable schema and stable JSON representation for artifact kind
|
||||
`dnd/spell-list`. The runner keeps the result typed through validators and later
|
||||
stages, using the codec only for checkpoint, debug, and output boundaries.
|
||||
Shared D&D helpers keep prompt input
|
||||
names and source-unit reference conversion consistent with the scene chunker.
|
||||
|
||||
The durable payload and manifest metadata shapes are defined in the
|
||||
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
|
||||
|
||||
## Merger And Normalizer
|
||||
|
||||
### `internal/modules/merge/appendorder`
|
||||
### `internal/modules/generic/merge/appendorder`
|
||||
|
||||
The merger preserves extract-result order. It passes through one JSON result,
|
||||
concatenates a common top-level array field across multiple JSON objects, and
|
||||
otherwise emits an array of the decoded values. It rejects invalid JSON and
|
||||
non-JSON media types, and it preserves compatible schema provenance.
|
||||
The merger passes typed values to an injected combine function in framework
|
||||
source-chunk order. The D&D registrar specializes it with a spell-list append
|
||||
function.
|
||||
|
||||
### `internal/modules/normalize/noop`
|
||||
### `internal/modules/generic/normalize/noop`
|
||||
|
||||
The normalizer defensively clones the accepted merge result, including payload
|
||||
bytes, metadata, warnings, and schema provenance, without changing its logical
|
||||
content.
|
||||
The normalizer returns the merged domain value unchanged and is reusable for
|
||||
any registered artifact type.
|
||||
|
||||
## Output Encoder
|
||||
|
||||
### `internal/modules/output/json`
|
||||
### `internal/modules/generic/output/json`
|
||||
|
||||
The JSON encoder sorts normalized results by lane, derives collision-checked
|
||||
safe logical names, pretty-prints JSON payloads, and assembles the logical index,
|
||||
@@ -125,26 +152,26 @@ paths and schemas.
|
||||
|
||||
## Generic Validators
|
||||
|
||||
The unconditional accept and reject validators provide deterministic production
|
||||
registrations used primarily for controlled composition and tests.
|
||||
The generic validator implementations live under
|
||||
`internal/modules/generic/validate`.
|
||||
|
||||
The JSON syntax validator uses `encoding/json` to reject malformed payloads. The
|
||||
JSON Schema validator requires schema bytes on the validation request, parses
|
||||
the instance and schema with `jsonschema`, and distinguishes payload rejection
|
||||
from schema loading or compilation errors. Neither validator calls the LLM.
|
||||
The unconditional accept and reject validators provide explicit chunk and
|
||||
typed-artifact variants used primarily for controlled composition and tests.
|
||||
|
||||
The serialized JSON syntax validator uses `encoding/json` to reject malformed
|
||||
representation bytes. The serialized JSON Schema validator requires schema
|
||||
bytes, parses the instance and schema with `jsonschema`, and distinguishes
|
||||
payload rejection from schema loading or compilation errors. The framework
|
||||
serialized-validation request carries either canonical chunk bytes or artifact
|
||||
codec bytes according to its target context. Neither validator calls the LLM.
|
||||
|
||||
## D&D Spell Validators
|
||||
|
||||
`internal/validators/extract/dnd/spells/spellpayload` provides strict decoding,
|
||||
shape checks, source-reference candidates, and cited-text lookup shared by the
|
||||
three validators.
|
||||
|
||||
The shape validator rejects malformed JSON, unknown fields, missing or empty
|
||||
spell fields, and empty reference lists. The source-reference validator applies
|
||||
generic source-reference validation to every cited range. The relatedness
|
||||
validator approves structurally valid payloads but warns when a case-insensitive
|
||||
spell name is absent from all cited source text. It leaves malformed payloads to
|
||||
the earlier validators in the configured chain.
|
||||
All three validators receive `dnd.SpellList` directly. The shape validator
|
||||
rejects missing or empty spell fields and empty reference lists. The
|
||||
source-reference validator applies generic source-reference validation to every
|
||||
cited range. The relatedness validator warns when a case-insensitive spell name
|
||||
is absent from all cited source text.
|
||||
|
||||
These validators are deterministic. Their selectable keys and production order
|
||||
are defined in
|
||||
@@ -154,10 +181,13 @@ payload rules are defined in the
|
||||
|
||||
## Production Registration
|
||||
|
||||
`internal/cli/catalog.go` builds the production registries, registers module and
|
||||
validator constructors, installs default validator-chain mappings, and exposes
|
||||
the matching catalog for resolution. It also collects prompt assets from
|
||||
LLM-backed packages before constructing the production client.
|
||||
The CLI allocates one complete framework registry set and one LLM asset
|
||||
registry. It invokes `internal/modules/generic/register`,
|
||||
`internal/modules/seriatim/register`, and `internal/modules/dnd/register` in
|
||||
that order, then exposes the matching catalog for resolution. The generic and
|
||||
Seriatim registrars own their production leaf registrations. The D&D registrar
|
||||
owns D&D leaf registrations, the spell default-validator chain, and D&D
|
||||
prompt/schema asset collection.
|
||||
|
||||
Framework packages must not import production extensions. Tests may compose
|
||||
registries and catalogs directly with fakes.
|
||||
@@ -170,8 +200,8 @@ When adding a production module or validator:
|
||||
2. expose and test its spec, constructor, and registration function;
|
||||
3. keep format or domain parsing inside the concrete package;
|
||||
4. add package-owned prompt/schema assets when the extension is LLM-backed;
|
||||
5. register it in `internal/cli/catalog.go` and add a default chain only when
|
||||
production policy requires one;
|
||||
5. register it through its package-family registrar and add a default chain
|
||||
there only when production policy requires one;
|
||||
6. add resolution and composition coverage for capabilities, options,
|
||||
references, and validation behavior;
|
||||
7. update the selectable-key catalog in [Configuration](../config.md), the
|
||||
@@ -184,11 +214,13 @@ does not inventory implementations.
|
||||
## Tests To Inspect
|
||||
|
||||
- Package-local `*_test.go` files under the module or validator being changed.
|
||||
- `internal/framework/pipeline/registry_integration_test.go`: registry and spec
|
||||
composition.
|
||||
- `internal/framework/pipeline/typed_resolution_test.go`: typed registry, spec,
|
||||
and heterogeneous artifact composition.
|
||||
- `internal/framework/pipeline/default_modules_test.go`: framework binding
|
||||
defaults.
|
||||
- `internal/cli/run_test.go`: production catalog, config resolution, and
|
||||
end-to-end CLI composition.
|
||||
- `internal/modules/sharedassets/**/*_test.go`: shared prompt and reference
|
||||
assembly.
|
||||
- `internal/framework/promptfs/*_test.go` and
|
||||
`internal/modules/dnd/shared/*_test.go`: shared prompt and reference assembly.
|
||||
- `internal/modules/integration/*_test.go`: black-box composition across
|
||||
production extension domains.
|
||||
|
||||
@@ -14,15 +14,18 @@ collaborators, invokes `internal/framework/pipeline`, and places the logical
|
||||
output files returned by the runner. Diagnostics, checkpoints, and debug
|
||||
recorders are optional side-channel collaborators supplied at this boundary.
|
||||
|
||||
Pipeline execution is serial. Resolution produces a fixed ordered workflow and
|
||||
a sorted set of artifact lanes before the runner constructs any stage module.
|
||||
Resolution produces a fixed ordered workflow and a sorted set of artifact
|
||||
lanes. Preparation constructs the complete module and validator set before the
|
||||
runner receives source bytes. Source parsing and chunking are serial; extraction
|
||||
uses a bounded run-wide worker pool, followed by serial per-lane merge and
|
||||
normalize continuations that may overlap across lanes.
|
||||
|
||||
## Application Boundary
|
||||
|
||||
| Package | Implemented responsibility |
|
||||
| --- | --- |
|
||||
| `cmd/notarius` | Executable entry point and process exit delegation. |
|
||||
| `internal/cli` | Command parsing, config discovery, production registration, prompt asset collection, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results. |
|
||||
| `internal/cli` | Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results. |
|
||||
|
||||
## Core Packages
|
||||
|
||||
@@ -31,23 +34,42 @@ a sorted set of artifact lanes before the runner constructs any stage module.
|
||||
| `internal/core/artifacts` | Run-manifest and provenance models. |
|
||||
| `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
|
||||
| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
|
||||
| `internal/core/source` | Generic source documents, units, references, lookup, and validation. |
|
||||
| `internal/core/source` | Generic source documents, units, chunks, canonical references, lookup, validation, and deterministic source digests. |
|
||||
| `internal/core/workspace` | Effective workspace settings, confined paths and writes, checkpoint identity, and checkpoint manifest models. |
|
||||
|
||||
## Framework Packages
|
||||
|
||||
| Package | Implemented responsibility |
|
||||
| --- | --- |
|
||||
| `internal/framework/contracts` | Stage, validator, reference, output, and structured-completion interfaces and data types. |
|
||||
| `internal/framework/pipeline` | Registries, profile resolution, capability checks, reference materialization, validator-chain resolution, retries, orchestration, warnings, and manifest population. |
|
||||
| `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. |
|
||||
| `internal/framework/pipeline` | Module and artifact-codec registries, option validation, profile resolution, capability checks, reference materialization, complete pipeline preparation, retries, orchestration, warnings, and manifest population. |
|
||||
| `internal/framework/validate` | Shared validator decision and cardinality helpers. |
|
||||
| `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
|
||||
| `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
|
||||
| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. |
|
||||
| `internal/framework/debug` | Workspace-backed framework and LLM debug recording. |
|
||||
|
||||
Framework contracts carry raw stage results between implementations. The
|
||||
runner owns handoff provenance, validation sequencing, rejection handling,
|
||||
checkpoint and debug boundaries, and final manifest assembly.
|
||||
Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
|
||||
serialized-validator, and
|
||||
typed-validator interfaces. The runner owns handoff provenance, validation
|
||||
sequencing, rejection handling, checkpoint and debug boundaries, and final
|
||||
manifest assembly.
|
||||
|
||||
Artifact registries support heterogeneous typed extraction entries and
|
||||
kind-specific merger, normalizer, and validator variants. Resolution derives a
|
||||
lane's kind from its extractor, requires the matching codec, verifies exact Go
|
||||
type equality across the lane, and records schema identity in the resolved lane
|
||||
and pipeline digest. Registry entries carry separate option-validation and
|
||||
run-local construction closures. Preparation injects shared dependencies and
|
||||
constructs input, chunk, validators, ordered lanes, and output before source
|
||||
parsing. Production modules use strict construction-time option decoding, and
|
||||
LLM-backed modules retain the injected shared client. The D&D family registers
|
||||
the canonical `dnd/spell-list` codec, typed spell extractor and validators, and
|
||||
kind-specific generic merge and normalize strategies; generic JSON validators
|
||||
use the serialized-validation contract. The runner executes lanes through
|
||||
private exact-type-checked closures, coordinates extract results independently
|
||||
of completion timing, and serializes artifacts only through their codec at
|
||||
checkpoint, debug, and output boundaries.
|
||||
|
||||
## Production Extensions
|
||||
|
||||
@@ -58,24 +80,38 @@ Configuration. The implemented module packages are:
|
||||
|
||||
| Package | Implemented responsibility |
|
||||
| --- | --- |
|
||||
| `internal/modules/input/seriatim` | Parses the supported Seriatim transcript format into the generic source model. |
|
||||
| `internal/modules/chunk/generic` | Splits ordered source units by unit count and overlap. |
|
||||
| `internal/modules/chunk/dnd/scenes` | Produces contiguous D&D scene chunks from structured model output. |
|
||||
| `internal/modules/extract/dnd/spells` | Produces source-grounded D&D spell-cast raw output. |
|
||||
| `internal/modules/merge/appendorder` | Combines accepted extraction results in chunk order. |
|
||||
| `internal/modules/normalize/noop` | Preserves accepted merged output. |
|
||||
| `internal/modules/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
|
||||
| `internal/modules/seriatim/input/transcript` | Parses the supported Seriatim transcript format into the generic source model. |
|
||||
| `internal/modules/generic/chunk/units` | Splits ordered source units by unit count and overlap. |
|
||||
| `internal/modules/dnd/chunk/scenes` | Produces contiguous D&D scene chunks from structured model output. |
|
||||
| `internal/modules/dnd` | Owns the canonical D&D spell-list and spell-cast artifact types. |
|
||||
| `internal/modules/dnd/codec/spells` | Strictly decodes and stably encodes the durable D&D spell-list representation. |
|
||||
| `internal/modules/dnd/extract/spells` | Maps private structured model output to canonical source-grounded D&D spell lists. |
|
||||
| `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. |
|
||||
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
|
||||
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
|
||||
|
||||
`internal/modules/sharedassets` composes shared prompt filesystems.
|
||||
`internal/modules/sharedassets/dnd` owns reusable D&D prompt fragments,
|
||||
`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
|
||||
reference declarations, prompt input assembly, and source-unit reference
|
||||
helpers.
|
||||
helpers. Domain-neutral prompt filesystem composition lives in
|
||||
`internal/framework/promptfs`.
|
||||
|
||||
Concrete validators live under `internal/validators`. Generic packages provide
|
||||
Generic validators under `internal/modules/generic/validate` provide
|
||||
unconditional test decisions, JSON syntax validation, and JSON Schema
|
||||
validation. D&D spell packages provide shape, source-reference, and
|
||||
source-relatedness decisions, with `spellpayload` holding their shared parser
|
||||
and lookup helpers. Production chain composition is owned by `internal/cli`.
|
||||
validation. D&D spell validators under `internal/modules/dnd/validate/spells`
|
||||
consume the canonical spell-list type directly to provide shape,
|
||||
source-reference, and source-relatedness decisions.
|
||||
|
||||
Production composition is grouped behind package-family registrars, and every
|
||||
implemented production extension uses its domain-first tree:
|
||||
|
||||
| Package | Implemented responsibility |
|
||||
| --- | --- |
|
||||
| `internal/modules/generic/register` | Registers domain-neutral chunk, merge, normalize, output, and validator implementations. |
|
||||
| `internal/modules/seriatim/register` | Registers the Seriatim input adapter. |
|
||||
| `internal/modules/dnd/register` | Registers D&D modules, validators, default validator policy, and prompt/schema assets. |
|
||||
|
||||
The CLI allocates the framework registries and asset registry, then invokes
|
||||
these registrars in generic, Seriatim, and D&D order.
|
||||
|
||||
Implementation details for all production extensions are in
|
||||
[Module Internals](modules.md).
|
||||
|
||||
@@ -6,8 +6,12 @@ Their fixed workflow and ownership boundaries are defined by
|
||||
defaults, and selectable keys are defined in
|
||||
[Configuration](../config.md#pipelines).
|
||||
|
||||
Pipeline execution is serial. Resolution fixes the selected lanes and all
|
||||
stage bindings before the runner constructs stage implementations.
|
||||
Resolution fixes the selected lanes and all stage bindings; preparation
|
||||
constructs every selected implementation before the runner begins source work.
|
||||
After serial input parsing and chunking, the runner dispatches extract work to
|
||||
one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's
|
||||
merge and normalize operations remain serial and may overlap other lanes once
|
||||
all extracts for that lane are terminal.
|
||||
|
||||
## Resolution
|
||||
|
||||
@@ -20,9 +24,14 @@ calls `pipeline.ResolvePipeline`.
|
||||
1. selects and sorts artifact lanes;
|
||||
2. completes omitted bindings using the documented configuration defaults;
|
||||
3. looks up each module and validator spec without constructing it;
|
||||
4. checks required and provided capabilities in workflow order;
|
||||
5. resolves target-aware reference bindings and validator chains;
|
||||
6. calculates a digest over the resolved structure.
|
||||
4. for a typed extractor, derives its artifact kind, requires the codec, and
|
||||
selects exact-type merger, normalizer, and validator variants;
|
||||
5. checks required and provided capabilities in workflow order;
|
||||
6. resolves target-aware reference bindings and validator chains;
|
||||
7. validates each selected module and validator option set through its registry
|
||||
entry; and
|
||||
8. calculates a digest over the resolved structure, including typed artifact
|
||||
kind and schema identity.
|
||||
|
||||
Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
|
||||
bindings, validator chains, reference targets, and the digest. It does not read
|
||||
@@ -50,31 +59,76 @@ runtime sensitive-data handling belongs in [Operations](../operations.md).
|
||||
|
||||
## Registries And Specs
|
||||
|
||||
`pipeline.Registries` holds constructors used during execution.
|
||||
`pipeline.Registries` holds option validators and run-local builders used during
|
||||
resolution and preparation.
|
||||
`pipeline.ModuleCatalog` exposes their specs during configuration validation and
|
||||
resolution. Separate registries exist for every stage and for validators;
|
||||
`ValidatorChainRegistry` stores production default-chain mappings.
|
||||
`ValidatorChainRegistry` stores production default-chain mappings. Both
|
||||
containers also carry an `ArtifactCodecRegistry`. Generic registration records
|
||||
one codec per stable artifact kind, validates its schema metadata and JSON
|
||||
Schema, retains the exact schema digest and Go type, and safely encodes or
|
||||
decodes framework-erased values with typed errors on incompatibility.
|
||||
|
||||
Typed extractor entries are keyed by module key and declare one artifact kind.
|
||||
Merger, normalizer, and typed-validator variants are keyed by module or
|
||||
validator key plus artifact kind. Chunk and serialized validators occupy
|
||||
separate target namespaces; serialized registrations declare whether they
|
||||
support chunks, artifacts, or both. Duplicate variants and exact Go-type
|
||||
mismatches are rejected deterministically.
|
||||
|
||||
Production composition registers the D&D spell-list codec and typed extractor,
|
||||
matching typed merge, normalize, and semantic-validator variants, and
|
||||
serialized JSON validators. Every artifact lane resolves through the typed
|
||||
registries and a matching codec.
|
||||
|
||||
A `ModuleSpec` declares its stage plus required and provided capabilities.
|
||||
Chunk, extract, merge, and normalize specs may also declare reference slots.
|
||||
Registry implementations defensively copy spec metadata, reject duplicate keys,
|
||||
and verify that a constructed implementation reports the registered key.
|
||||
Builder registrations accept `ModuleDependencies` and cloned configuration
|
||||
options through one `BuildRequest`. Builders decode those options and retain
|
||||
typed values or injected dependencies in the constructed implementation.
|
||||
Extractors declare their artifact kind, and merger, normalizer, and validator
|
||||
resolution selects the matching typed variant.
|
||||
|
||||
A `ValidatorSpec` declares a validator key and execution class. Resolution uses
|
||||
the execution class to reject incompatible profile bindings before execution.
|
||||
The current production catalog and default chain are listed only in
|
||||
[Configuration](../config.md#implemented-production-validators).
|
||||
|
||||
## Runner Boundary
|
||||
## Preparation And Runner Boundary
|
||||
|
||||
`pipeline.RunInput` carries the resolved pipeline, raw source input, structured
|
||||
LLM client, run identity and timing, optional session and profile metadata, and
|
||||
checkpoint/debug collaborators. The runner parses source bytes through the
|
||||
selected input adapter. Later stage requests receive the generic source model;
|
||||
extract requests receive chunk-scoped input material, while chunk, merge, and
|
||||
normalize requests retain access to the original source material.
|
||||
`pipeline.Prepare` receives a resolved pipeline, the registries, and shared
|
||||
module dependencies. It constructs input; chunk and its validators; each lane's
|
||||
extract, merge, and normalize modules and validator chains in resolved order;
|
||||
then output. It stops at the first error with pipeline, stage, lane, module, and
|
||||
validator context as applicable. It never invokes an operation method.
|
||||
|
||||
`pipeline.RunOutput` carries the run manifest, accepted normalized results,
|
||||
`PreparedPipeline` keeps private constructed executors and exposes cloned
|
||||
resolved input, chunk, lane, and output identities. `pipeline.RunInput` carries
|
||||
that prepared pipeline, raw source input, run identity and timing, optional
|
||||
session and profile metadata, and checkpoint/debug collaborators. The runner
|
||||
parses source bytes through the already constructed input adapter. Later stage
|
||||
requests receive the generic source model; extract requests receive
|
||||
chunk-scoped input material, while chunk, merge, and normalize requests retain
|
||||
access to the original source material. Input, chunk, and output operation
|
||||
requests do not carry raw module options. The chunk request also does not carry
|
||||
an LLM client; an LLM-backed chunker receives the shared client during
|
||||
preparation. Their operation requests retain run-specific source, reference,
|
||||
profile, session, and metadata context as applicable.
|
||||
|
||||
Prepared lanes retain exact-type-checked erased operation closures. The runner
|
||||
uses those closures to keep each value typed through extraction, validation,
|
||||
merge, and normalization.
|
||||
|
||||
Source validation requires every unit to carry a canonical self-reference to
|
||||
its containing document and its own unit ID. Explicit clone, checkpoint, and
|
||||
debug boundaries retain that reference, and the canonical source digest covers
|
||||
it deterministically. Chunks use the same source model and carry one canonical
|
||||
reference spanning the first selected unit through the last.
|
||||
|
||||
`pipeline.RunOutput` carries the run manifest, accepted normalized serialized
|
||||
artifacts with lane and normalizer provenance,
|
||||
rejected results, warnings, checkpoint events, and logical files returned by the
|
||||
output encoder. The CLI owns diagnostics and durable filesystem writes after the
|
||||
runner returns.
|
||||
@@ -83,21 +137,22 @@ runner returns.
|
||||
|
||||
The runner:
|
||||
|
||||
1. validates its input and registries;
|
||||
2. builds the input adapter, parses the raw input, and validates the generic
|
||||
1. validates its prepared input;
|
||||
2. parses the raw input with the prepared adapter and validates the generic
|
||||
source document;
|
||||
3. obtains or executes the chunk result;
|
||||
4. validates and canonicalizes chunks;
|
||||
5. executes each resolved artifact lane in order;
|
||||
6. builds the output encoder and validates its logical file results;
|
||||
5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
|
||||
bounded lane continuation when all extracts for that lane are terminal;
|
||||
6. invokes the prepared output encoder and validates its logical file results;
|
||||
7. returns the assembled manifest, outcomes, warnings, and files.
|
||||
|
||||
Within each artifact lane, it builds the extractor, merger, and normalizer,
|
||||
then performs these transitions:
|
||||
Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
|
||||
and validators while performing these transitions:
|
||||
|
||||
1. extract once per accepted chunk and add runner-owned lane, source, and chunk
|
||||
provenance;
|
||||
2. validate each raw extract result and omit rejected results from merge input;
|
||||
2. validate each extract result and omit rejected results from merge input;
|
||||
3. skip the rest of the lane when no extract result is accepted;
|
||||
4. merge accepted extract results in their existing order;
|
||||
5. validate the merge result and skip normalization on rejection;
|
||||
@@ -107,25 +162,35 @@ then performs these transitions:
|
||||
Module-provided warnings and payload warnings are promoted only from attempts
|
||||
whose results are accepted and used.
|
||||
|
||||
The extract job channel has the same capacity as the effective extract worker
|
||||
count, so dispatch applies backpressure. A fixed continuation executor prevents
|
||||
ready or checkpoint-reused lanes from creating one goroutine each. Workers and
|
||||
continuations publish lane-local results; the coordinator is the only writer of
|
||||
aggregate output and merges those results in resolved lane and source-chunk
|
||||
order.
|
||||
|
||||
## Chunk Canonicalization
|
||||
|
||||
Before lane execution, generic validation requires unique chunk IDs, matching
|
||||
source identity, indexes matching returned order, valid ordered boundaries,
|
||||
source identity, indexes matching returned order, a valid canonical reference,
|
||||
non-empty content and media type, and at least one valid source unit per chunk.
|
||||
Units may not repeat inside a chunk and must preserve source-document order.
|
||||
Units may not repeat inside a chunk and must form a contiguous range in
|
||||
source-document order. The chunk reference must exactly match the source and
|
||||
the first and last unit references.
|
||||
|
||||
The runner then rebuilds each chunk's unit slice from the source document by
|
||||
unit ID. It preserves the module-owned boundaries, content, media type, and
|
||||
cloned metadata. The framework permits gaps and overlap between separate
|
||||
unit ID. It preserves the canonical reference, content, media type, and cloned
|
||||
metadata. The framework permits gaps and overlap between separate
|
||||
chunks; stricter coverage policy belongs to the chunk implementation.
|
||||
|
||||
## Validation And Retries
|
||||
|
||||
Chunk, extract, merge, and normalize results pass through the resolved validator
|
||||
chain for their stage and module. Each validator receives the raw payload plus
|
||||
the relevant source, chunk, prior-stage, schema, reference, session, LLM, option,
|
||||
and run context. Validators execute in resolved order and stop at the first
|
||||
error or rejection. An empty chain approves the result.
|
||||
chain for their stage and module. Chunk validators receive canonical chunks;
|
||||
typed validators receive the domain value; and serialized validators receive
|
||||
canonical chunk JSON or artifact codec bytes. Validators execute in resolved
|
||||
order and stop at the first error or rejection. An empty chain approves the
|
||||
result.
|
||||
|
||||
`runWithRetry` applies the effective retry policy around module execution and
|
||||
its complete validation chain. A module or validator error becomes a framework
|
||||
@@ -143,13 +208,20 @@ The runner depends on recorder and loader interfaces, using no-op
|
||||
implementations when collaborators are absent. Each checkpointed workflow
|
||||
boundary records a running, succeeded, or failed transition. Reuse decisions
|
||||
are consulted in workflow order and accepted payloads are cloned before
|
||||
entering the normal handoff path. Dependency fingerprints connect later
|
||||
checkpoints to the exact accepted results on which they depend.
|
||||
entering the normal handoff path. Typed extract, merge, and normalize
|
||||
checkpoints store codec bytes with artifact kind, schema ID and version, exact
|
||||
schema digest, and media type. Reuse compares that identity with the prepared
|
||||
codec and decodes through the codec; missing identity, mismatches, corrupt
|
||||
bytes, and decode failures become explicit reuse misses and execute the step
|
||||
normally. Dependency fingerprints and debug content digests use the same stable
|
||||
codec bytes that cross those boundaries.
|
||||
|
||||
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
|
||||
boundaries. Context scopes associate nested LLM calls with the module or
|
||||
validator attempt that made them. Debug-write failures are framework errors;
|
||||
debug data is never used as a checkpoint source.
|
||||
debug data is never used as a checkpoint source. Typed artifact debug envelopes
|
||||
are domain-neutral, redact sensitive metadata and bytes through the common
|
||||
debug policy, and record codec identity plus schema and content digests.
|
||||
|
||||
Checkpoint identity, physical layout, reuse behavior, and debug artifact
|
||||
handling are operator contracts in [Operations](../operations.md). Serialization
|
||||
@@ -161,25 +233,42 @@ and recorder implementation are inventoried in
|
||||
The runner owns manifest assembly and handoff summaries but not the durable JSON
|
||||
schema. It records resolved module and lane provenance, validator chains,
|
||||
source/reference identities, selected LLM profiles, normalized and rejected
|
||||
summaries, status, and timing. Raw payload bytes remain outside the manifest.
|
||||
summaries, status, and timing. Serialized artifact content remains outside the manifest.
|
||||
Module metadata providers may add non-secret singleton or lane-scoped metadata.
|
||||
|
||||
Execution errors include stage, module, lane, or validator context. Once a
|
||||
manifest exists, a failing run returns it with failed status and completion
|
||||
time. Successful status reflects whether any raw result was rejected. The
|
||||
time. Successful status reflects whether any result was rejected. The
|
||||
durable manifest and logical file schemas are defined in the
|
||||
[JSON output contract](../integrations/json-output.md).
|
||||
|
||||
On a framework failure, the runner cancels its derived context, stops submitting
|
||||
new extract work, drains started tasks, and skips the output encoder. Parent
|
||||
cancellation takes precedence. Otherwise context-cancellation fallout is
|
||||
discarded when a substantive error exists, and the primary error is selected by
|
||||
stage, resolved lane, and source chunk rather than completion time.
|
||||
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/core/config/effective_config_test.go`: config-to-resolution boundary.
|
||||
- `internal/framework/pipeline/profile_test.go`: selection, defaults,
|
||||
capabilities, validator chains, and digest behavior.
|
||||
- `internal/framework/pipeline/artifact_codec_registry_test.go`: typed codec
|
||||
metadata, registration, erasure safety, strict decoding, and cloning.
|
||||
- `internal/framework/pipeline/typed_resolution_test.go`: heterogeneous typed
|
||||
lane resolution and preparation, target-specific validators,
|
||||
incompatibilities, ordering, and schema-sensitive pipeline identity.
|
||||
- `internal/framework/pipeline/runner_concurrency_test.go`: bounded dispatch and
|
||||
continuations, reverse completion, stable errors, rejection, cancellation,
|
||||
retries, and independent provider-call limits.
|
||||
- `internal/framework/pipeline/preparation_test.go`: option validation,
|
||||
construction order, dependency failures, and the before-source-work boundary.
|
||||
- `internal/framework/pipeline/references_test.go`: target resolution and
|
||||
materialization.
|
||||
- `internal/framework/pipeline/runner_test.go`: stage transitions, retries,
|
||||
rejections, warnings, checkpoints, debug hooks, and manifests.
|
||||
- `internal/framework/pipeline/walking_skeleton_test.go`: fake-backed complete
|
||||
workflow composition.
|
||||
- `internal/cli/run_test.go`: production stage transitions, retries, rejections,
|
||||
warnings, debug hooks, manifests, and end-to-end composition.
|
||||
- `internal/modules/integration/*_test.go` and
|
||||
`internal/modules/seriatim/input/transcript/runner_test.go`: typed runner
|
||||
composition across concrete module families.
|
||||
- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and reuse
|
||||
collaborators.
|
||||
|
||||
@@ -91,6 +91,20 @@ digests match the current invocation. Changes to input bytes, the resolved
|
||||
pipeline, selected lanes, the runtime LLM profile override, or bound reference
|
||||
content invalidate reuse.
|
||||
|
||||
Typed artifact checkpoints additionally record codec-owned bytes, artifact
|
||||
kind, schema ID and version, exact schema digest, and media type. A missing or
|
||||
mismatched codec identity, or bytes the current codec cannot decode, is reported
|
||||
as a checkpoint reuse miss. The affected operation executes normally and, when
|
||||
checkpoint writing is enabled, replaces the incompatible checkpoint.
|
||||
|
||||
Current checkpoint manifests use workspace schema `notarius.workspace.v2`.
|
||||
Manifests written with `notarius.workspace.v1` are incompatible because their
|
||||
chunk provenance has an older shape. On the first explicit resume after an
|
||||
upgrade, each affected checkpoint is treated as a reuse miss and its workflow
|
||||
step executes normally. The compatibility check does not migrate or delete the
|
||||
v1 files; when checkpoint writing is enabled, normal execution refreshes the
|
||||
affected checkpoint files in the current schema.
|
||||
|
||||
Runs do not reuse checkpoints unless explicitly requested. Without reuse, the
|
||||
workflow executes normally and refreshes checkpoint files when checkpointing is
|
||||
enabled.
|
||||
@@ -118,7 +132,9 @@ the attempt `llm_calls` array. Prompt content is written inline in the prompt
|
||||
artifact. The response metadata and body use the paired files described above;
|
||||
the body is pretty-printed JSON when possible and raw text otherwise. Debug
|
||||
artifacts may contain source material, reference material, prompt inputs, model
|
||||
outputs, and other sensitive data. API keys are not written, and obvious
|
||||
outputs, and other sensitive data. Typed artifact envelopes include
|
||||
domain-neutral codec identity, redacted metadata and content, and digests of
|
||||
the stable codec bytes. API keys are not written, and obvious
|
||||
credential-shaped values and sensitive map keys are redacted, but debug
|
||||
directories should still be protected as sensitive local state.
|
||||
|
||||
@@ -182,5 +198,11 @@ selected execution profile. Pipeline module retry settings are defined in
|
||||
[Configuration](config.md#module-bindings). There is no separate CLI retry
|
||||
command.
|
||||
|
||||
Extract worker concurrency and actual provider-call concurrency are separate
|
||||
limits. Their configuration, defaults, and validation are defined in
|
||||
[Configuration](config.md#concurrency). Cancellation stops undispatched extract
|
||||
work; already started work is allowed to finish or observe cancellation before
|
||||
the run reports failure.
|
||||
|
||||
Notarius writes local files only. Remote storage and archive management are not
|
||||
part of the implemented CLI.
|
||||
|
||||
@@ -66,12 +66,19 @@ Framework stages operate on source documents, source units, and source
|
||||
references rather than format-specific structures. A source reference identifies
|
||||
an ordered range of generic source units. Framework code preserves those ranges
|
||||
and does not merge or rewrite them unless a stage module explicitly owns that
|
||||
behavior.
|
||||
behavior. Every source unit carries a validated self-reference to its containing
|
||||
document and its own unit ID.
|
||||
|
||||
Extract modules own artifact semantics, prompt use, response schemas, and
|
||||
domain interpretation. Domain-specific concepts remain in the relevant module,
|
||||
validator, shared domain helper, and artifact contract.
|
||||
|
||||
Typed artifact registrations declare one stable artifact kind and exact Go
|
||||
type from extraction through merge, normalization, and semantic validation.
|
||||
Pipeline resolution requires a compatible codec and matching kind-specific
|
||||
variants before a typed lane can be accepted. Framework-owned erasure remains
|
||||
private and must report type incompatibility as an error rather than a panic.
|
||||
|
||||
Auxiliary references provide context or disambiguation. They are not source
|
||||
evidence and must not be converted into source references.
|
||||
|
||||
@@ -83,6 +90,12 @@ and verifies module availability and capabilities before execution. Structural
|
||||
pipeline choices must not be scattered through conditionals or hidden behind
|
||||
ad hoc command flags.
|
||||
|
||||
Resolution validates every selected module and validator option set. A separate
|
||||
preparation boundary then constructs the complete input, chunk, lane,
|
||||
validation, and output implementation set in pipeline order. The runner accepts
|
||||
only that prepared set, so construction and dependency failures occur before
|
||||
source parsing or any other module operation.
|
||||
|
||||
Stage ownership is explicit:
|
||||
|
||||
- input modules convert external material into the generic source model;
|
||||
@@ -96,13 +109,32 @@ The framework owns orchestration and handoff provenance. Modules return logical
|
||||
results and warnings; they do not own CLI reporting, workspace paths, durable
|
||||
file placement, checkpoints, or diagnostics.
|
||||
|
||||
After pipeline-wide chunking, extraction uses bounded framework concurrency.
|
||||
One run-wide worker pool receives chunk-scoped lane jobs in deterministic
|
||||
chunk-first, lane-second order. A lane may begin its merge and normalize
|
||||
continuation only after all of its extract jobs are terminal; that continuation
|
||||
remains serial within the lane, while bounded continuations for different lanes
|
||||
may overlap. The framework must not create unbounded goroutines per lane or
|
||||
chunk.
|
||||
|
||||
Completion timing does not choose public ordering or errors. The coordinator
|
||||
orders accepted artifacts, warnings, rejections, checkpoint events, and
|
||||
framework errors by stable pipeline scope. Rejections do not cancel unrelated
|
||||
work. A framework error cancels derived work, prevents undispatched work from
|
||||
starting, waits for started work, and prevents output encoding.
|
||||
|
||||
## Validation
|
||||
|
||||
Validation is a framework-managed boundary around raw outputs from chunk,
|
||||
extract, merge, and normalize stages. Validators receive immutable stage output
|
||||
Validation is a framework-managed boundary around outputs from chunk, extract,
|
||||
merge, and normalize stages. Validators receive immutable stage output
|
||||
and make an explicit whole-output decision: approve, approve with warnings, or
|
||||
reject.
|
||||
|
||||
Typed artifact validators receive the domain value directly. Chunk validators
|
||||
receive source-zone chunks, while serialized validators receive immutable
|
||||
representation bytes and declared schema metadata. A validator registered for
|
||||
one target or artifact kind cannot satisfy an incompatible selection.
|
||||
|
||||
Rejection is a recorded pipeline outcome, not a framework execution error.
|
||||
Validator execution failures are framework errors. Rejected output does not
|
||||
advance to the next stage.
|
||||
@@ -126,6 +158,11 @@ LLM calls and other external operations accept cancellation and respect
|
||||
timeouts. Concurrency control belongs in shared runtime plumbing rather than in
|
||||
individual modules.
|
||||
|
||||
The application-wide LLM scheduler bounds actual provider calls independently
|
||||
of framework worker limits. Every LLM-backed module, retry, and validator uses
|
||||
the single injected scheduled client, including work performed by overlapping
|
||||
lanes.
|
||||
|
||||
## Configuration And Provenance
|
||||
|
||||
Configuration loading, precedence, defaults, environment overrides, redaction,
|
||||
|
||||
@@ -2,16 +2,16 @@
|
||||
|
||||
## Status
|
||||
|
||||
Decision-complete; implementation pending. This roadmap defines the desired end
|
||||
state for [ADR-0002](../adr/0002-linear-pipes-and-filters-pipeline.md),
|
||||
Implemented on 2026-07-17. This roadmap records the design delivered for
|
||||
[ADR-0002](../adr/0002-linear-pipes-and-filters-pipeline.md),
|
||||
[ADR-0003](../adr/0003-typed-interfaces-with-two-zone-data-model.md), and
|
||||
[ADR-0004](../adr/0004-package-modules-by-domain.md). The work needed to reach
|
||||
that state is owned by the
|
||||
[domain pipeline implementation plan](implementation.md).
|
||||
[ADR-0004](../adr/0004-package-modules-by-domain.md). Its implementation history
|
||||
is summarized in the [completion record](implementation.md).
|
||||
|
||||
Until that plan is complete, current behavior remains defined by the
|
||||
architecture, configuration, integration, operations, and internal
|
||||
documentation outside `docs/roadmap/`.
|
||||
This file is historical design context, not a current-behavior reference.
|
||||
Implemented contracts and mechanics are defined by the architecture,
|
||||
configuration, integration, operations, and internal documentation outside
|
||||
`docs/roadmap/`.
|
||||
|
||||
## User Intent
|
||||
|
||||
|
||||
@@ -4,12 +4,6 @@ Current Notarius behavior is documented in the canonical README, CLI,
|
||||
configuration, operations, internal, and integration docs. This roadmap records
|
||||
future work only.
|
||||
|
||||
## Focused Roadmaps
|
||||
|
||||
- [Domain-Typed Pipeline Implementation](domain.md): proposed migration to
|
||||
domain-owned typed artifact lanes, domain-first packages, explicit
|
||||
serialization boundaries, and bounded deterministic extract execution.
|
||||
|
||||
## Candidate Product Work
|
||||
|
||||
- Additional input adapters, such as Markdown or note-export formats.
|
||||
|
||||
@@ -1,836 +1,48 @@
|
||||
# Domain-Typed Pipeline Implementation Plan
|
||||
# Domain-Typed Pipeline Completion Record
|
||||
|
||||
## Purpose
|
||||
## Status
|
||||
|
||||
This document is the executable implementation plan for the target state in the
|
||||
[domain-typed pipeline feature roadmap](domain.md). It assumes the decisions in
|
||||
Implemented on 2026-07-17.
|
||||
|
||||
This file is a concise historical record. Current behavior is owned by
|
||||
[Architecture](../policy/architecture.md),
|
||||
[Configuration](../config.md), [Operations](../operations.md),
|
||||
[Pipeline Internals](../internal/pipeline.md),
|
||||
[Module Internals](../internal/modules.md),
|
||||
[LLM Runtime Internals](../internal/llm.md), and the durable
|
||||
[integration contracts](../integrations/). The accepted decisions remain in
|
||||
[ADR-0002](../adr/0002-linear-pipes-and-filters-pipeline.md),
|
||||
[ADR-0003](../adr/0003-typed-interfaces-with-two-zone-data-model.md), and
|
||||
[ADR-0004](../adr/0004-package-modules-by-domain.md).
|
||||
|
||||
The intended operator is an LLM coding agent working through one stage per
|
||||
implementation prompt. Complete the stages in order. Each stage must leave the
|
||||
repository buildable and tested; do not defer a broken intermediate state to a
|
||||
later stage.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
For every stage:
|
||||
|
||||
1. Read `docs/development.md` and follow its task-specific reading guide. Read
|
||||
the current implementation and focused tests for every touched subsystem.
|
||||
2. Treat the feature roadmap as the canonical owner of desired behavior and
|
||||
this document as the canonical owner of task sequencing. Do not restate
|
||||
future behavior in current-behavior documentation before it exists.
|
||||
3. Preserve unrelated user changes. Use mechanical moves where possible so file
|
||||
history and test intent remain legible.
|
||||
4. Add focused tests with the change. Run those tests while iterating, then run
|
||||
`go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` before ending
|
||||
the stage.
|
||||
5. Run `go test -race ./...` in stages that introduce or change concurrency and
|
||||
in the final stage.
|
||||
6. Update the canonical current-behavior documents in the same stage in which
|
||||
behavior changes. At minimum, reconsider `docs/policy/architecture.md`,
|
||||
`docs/internal/overview.md`, `docs/internal/pipeline.md`,
|
||||
`docs/internal/modules.md`, `docs/internal/llm.md`, `docs/config.md`,
|
||||
`docs/operations.md`, and `docs/integrations/` according to the documentation
|
||||
policy; edit only the documents whose owned facts changed.
|
||||
7. Do not change user-visible module keys, validator keys, output paths, durable
|
||||
D&D JSON, prompt/schema identities, default chains, or rejection semantics
|
||||
unless this plan explicitly requires it.
|
||||
8. Stop after a stage if an exit criterion cannot be met. Record the concrete
|
||||
blocker rather than implementing a second architecture alongside this one.
|
||||
|
||||
## Fixed Technical Decisions
|
||||
|
||||
The following choices are inputs to implementation, not questions to reopen in
|
||||
individual stages.
|
||||
|
||||
### Source types
|
||||
|
||||
Keep the existing names `source.SourceDocument`, `source.SourceUnit`, and
|
||||
`source.SourceRef`. Add `Ref source.SourceRef` to `SourceUnit`. Move
|
||||
`contracts.SourceChunk` to `internal/core/source` as `source.Chunk`, with this
|
||||
logical shape:
|
||||
|
||||
```go
|
||||
type Chunk struct {
|
||||
ID string
|
||||
SourceID string
|
||||
Index int
|
||||
Ref SourceRef
|
||||
Content []byte
|
||||
MediaType string
|
||||
Units []SourceUnit
|
||||
Metadata map[string]any
|
||||
}
|
||||
```
|
||||
|
||||
Remove `StartUnitID` and `EndUnitID`; `Ref` is the only chunk-boundary
|
||||
representation. A Seriatim unit's self-reference is
|
||||
`{SourceID: document ID, StartUnitID: unit ID, EndUnitID: unit ID}`. A chunk
|
||||
reference spans its first and last included units. Advance persisted workspace
|
||||
state from `notarius.workspace.v1` to `notarius.workspace.v2`; v1 state is
|
||||
incompatible and must be recomputed, but never deleted automatically.
|
||||
|
||||
### Artifact contracts
|
||||
|
||||
Place engine-owned artifact primitives with the other universal contracts under
|
||||
`internal/framework/contracts`:
|
||||
|
||||
```go
|
||||
type ArtifactKind string
|
||||
|
||||
type ArtifactSchema struct {
|
||||
ID string
|
||||
Name string
|
||||
Version string
|
||||
JSONSchema []byte
|
||||
}
|
||||
|
||||
type SerializedArtifact struct {
|
||||
Kind ArtifactKind
|
||||
Schema ArtifactSchema
|
||||
MediaType string
|
||||
Content []byte
|
||||
Metadata map[string]any
|
||||
}
|
||||
|
||||
type ArtifactCodec[T any] interface {
|
||||
Kind() ArtifactKind
|
||||
Schema() ArtifactSchema
|
||||
MediaType() string
|
||||
Encode(T) ([]byte, error)
|
||||
Decode([]byte) (T, error)
|
||||
}
|
||||
```
|
||||
|
||||
Use strict JSON decoding for JSON codecs: reject unknown fields and trailing
|
||||
tokens. Encoding must be deterministic for equal canonical values. Clone byte
|
||||
slices and maps at framework ownership boundaries. Validate non-empty artifact
|
||||
kind, schema ID, schema name, schema version, media type, and JSON Schema during
|
||||
registration. Compute and retain a SHA-256 digest of the JSON Schema bytes.
|
||||
|
||||
Use generic `Extractor[T]`, `Merger[T]`, `Normalizer[T]`, and
|
||||
`TypedValidator[T]` contracts. The exact request/result structs may retain
|
||||
existing names where that reduces churn, but they must satisfy these rules:
|
||||
|
||||
- the extractor returns `T`, warnings, and framework-owned chunk provenance;
|
||||
- merge receives accepted per-chunk typed values carrying lane ID, source ID,
|
||||
chunk ID, chunk index, and chunk reference, already sorted by chunk index;
|
||||
- normalize receives and returns `T`;
|
||||
- typed validators receive `T` plus the relevant universal source, chunk,
|
||||
reference, lane, stage, profile, and metadata context;
|
||||
- the LLM client and decoded module options are held by constructed
|
||||
implementations, not passed in operation requests; and
|
||||
- no module-facing Zone-B request or result contains `RawPayload`, `any`, or
|
||||
serialized JSON as its artifact value.
|
||||
|
||||
Retain separate framework wrappers around `T` for extract, merge, and normalize
|
||||
provenance. Do not put lane IDs, module keys, or framework warnings into the D&D
|
||||
domain value itself.
|
||||
|
||||
Support a second `SerializedValidator` contract for representation-level
|
||||
generic validators. Its request contains immutable bytes, media type, and
|
||||
optional schema metadata. For a Zone-B value, the framework produces that
|
||||
request with the lane codec. Keep a separate non-generic `ChunkValidator`
|
||||
contract for semantic validation of immutable `[]source.Chunk`; when a
|
||||
representation validator is selected at chunk, the framework instead supplies
|
||||
its canonical JSON chunk encoding. `valid_json` and `valid_json_schema` use the
|
||||
serialized path, domain validators use `TypedValidator[T]`, and generic
|
||||
approve/reject validators register explicit chunk and typed-artifact variants.
|
||||
|
||||
### Typed registry model
|
||||
|
||||
Because Go methods cannot introduce type parameters, expose free generic
|
||||
registration functions in `internal/framework/pipeline`, backed by private
|
||||
non-generic registry entries. Use `reflect.TypeFor[T]()` only inside registration
|
||||
and framework assembly to prove exact type equality.
|
||||
|
||||
- Codec registry key: artifact kind. Exactly one codec may be registered per
|
||||
kind.
|
||||
- Extractor registry key: existing module key. Each entry declares one artifact
|
||||
kind and exact Go type.
|
||||
- Merger and normalizer registry key: `(existing module key, artifact kind)`.
|
||||
- Typed validator registry key: `(existing validator key, artifact kind)`.
|
||||
- Chunk-validator registry key: existing validator key in the distinct chunk
|
||||
target namespace.
|
||||
- Serialized validators retain their existing validator key and declare whether
|
||||
they support chunk values, artifact values, or both.
|
||||
- Duplicate keys/variants, missing codecs, Go-type mismatches, and incompatible
|
||||
selected variants are errors.
|
||||
|
||||
The extractor selected for a lane establishes the lane artifact kind. During
|
||||
resolution, look up merger, normalizer, and validators against that kind.
|
||||
Record artifact kind, schema ID, schema version, and schema digest on the
|
||||
resolved lane and in the resolved-pipeline digest. A resolved pipeline with an
|
||||
incompatible lane must fail before preparation or source execution.
|
||||
|
||||
The private erased lane entry owns closures for construction and execution of
|
||||
one concrete `T`. It may store a value as `any` internally, but it must verify
|
||||
the exact registered `reflect.Type` at every erased boundary and return a
|
||||
descriptive framework error rather than panic.
|
||||
|
||||
### Construction and preparation
|
||||
|
||||
Use one uniform construction context:
|
||||
|
||||
```go
|
||||
type ModuleDependencies struct {
|
||||
LLM contracts.StructuredLLMClient
|
||||
}
|
||||
|
||||
type BuildRequest struct {
|
||||
Dependencies ModuleDependencies
|
||||
Options map[string]any
|
||||
}
|
||||
```
|
||||
|
||||
Each registry entry stores both an option-validation closure and a constructor.
|
||||
Implementations own concrete option structs and one decoder used by both
|
||||
closures. Resolution/configuration validation calls the decoder and discards
|
||||
the value; preparation calls it once and supplies the decoded value to the
|
||||
constructor. Reject unknown option fields. Empty options produce the
|
||||
implementation's explicit defaults.
|
||||
|
||||
Add:
|
||||
|
||||
```go
|
||||
func Prepare(
|
||||
resolved ResolvedPipeline,
|
||||
registries Registries,
|
||||
deps ModuleDependencies,
|
||||
) (*PreparedPipeline, error)
|
||||
```
|
||||
|
||||
`PreparedPipeline` retains explicit resolved input, chunk, artifact-lane, and
|
||||
output fields and all constructed validators. Construct in stable pipeline
|
||||
order: input; chunk and its validators; each resolved lane in order with extract,
|
||||
merge, normalize, and their validator chains in stage order; then output. On the
|
||||
first failure, return an error identifying stage, lane if any, module or
|
||||
validator key, and cause. No operation method may have run.
|
||||
|
||||
Create the one scheduled production LLM client first, inject it into
|
||||
preparation, and then run only the prepared pipeline. Test-only deterministic
|
||||
modules may accept a nil LLM dependency; any implementation that declares or
|
||||
uses LLM-backed execution must reject a nil client at preparation. Remove LLM
|
||||
clients and raw option maps from operation requests after every production
|
||||
implementation has migrated.
|
||||
|
||||
Constructed modules and validators are reused for a run. Anything callable from
|
||||
parallel extract workers must be concurrency-safe; production implementations
|
||||
should be immutable after construction.
|
||||
|
||||
### Package registration
|
||||
|
||||
Each package-family registrar exposes:
|
||||
|
||||
```go
|
||||
func Register(registries pipeline.Registries, assets *llm.AssetRegistry) error
|
||||
```
|
||||
|
||||
The generic and Seriatim registrars ignore the asset argument until they need
|
||||
it. Registrars validate the registry pointers they use and return contextual
|
||||
errors. The CLI creates one complete registry set and one asset registry, then
|
||||
calls registrars in this order: generic, Seriatim, D&D. The D&D registrar owns
|
||||
D&D codecs, implementations, typed generic specializations, prompt/schema
|
||||
assets, and default validator chains.
|
||||
|
||||
`internal/modules/dnd` owns D&D shared types. Its `register` sibling may import
|
||||
children and generic strategies; the root package must not import its children.
|
||||
`internal/modules/generic` never imports D&D. `internal/modules/seriatim` does not
|
||||
import D&D. Move domain-neutral embedded prompt-filesystem helpers to
|
||||
`internal/framework/promptfs`.
|
||||
|
||||
### D&D typed model
|
||||
|
||||
Define `dnd.SpellList`, `dnd.SpellCast`, and evidence/source-reference fields at
|
||||
the D&D package root. Use `source.SourceRef`; do not create another D&D unit-ref
|
||||
type for artifact provenance. The stable artifact kind is
|
||||
`dnd/spell-list`.
|
||||
|
||||
The spell extractor owns a private LLM DTO and the existing
|
||||
`dnd_spells_llm.v1.json` response schema. It canonicalizes and maps that DTO to
|
||||
`dnd.SpellList`. The codec package owns `dnd_spells.v1.json` and the durable
|
||||
encoding. Keep those schemas separate. Remove the validator-only
|
||||
`spellpayload` model after all three D&D validators consume `dnd.SpellList`.
|
||||
|
||||
Make `appendorder` a generic strategy that accepts a typed combine function at
|
||||
registration/construction. The D&D registrar supplies a function that appends
|
||||
spell casts in already-sorted source-chunk order. Make `noop` a generic typed
|
||||
strategy. Neither generic package imports D&D.
|
||||
|
||||
### Serialized boundaries
|
||||
|
||||
After normalize, encode `T` once to `SerializedArtifact` for final output.
|
||||
Output remains domain-neutral and receives serialized artifacts. Preserve the
|
||||
existing output bundle and index contract.
|
||||
|
||||
Extract, merge, and normalize checkpoints encode and decode through the same
|
||||
lane codec. Checkpoint identity includes artifact kind, schema ID, schema
|
||||
version, and schema digest. A missing codec or any mismatch invalidates reuse
|
||||
and recomputes the stage; it is not a fatal run error by itself. Codec decode
|
||||
failure also invalidates that checkpoint and records the reason. Never pass
|
||||
serialized checkpoint content directly to the next typed stage.
|
||||
|
||||
Debug recording uses the codec for typed artifact values and preserves existing
|
||||
opt-in/sensitive-content rules. Artifact/checkpoint digests use the stable codec
|
||||
bytes. Aggregate manifests, checkpoint indexes, warning slices, and rejection
|
||||
slices have one coordinator writer.
|
||||
|
||||
### Concurrency
|
||||
|
||||
Version-2 configuration gains:
|
||||
|
||||
```yaml
|
||||
concurrency:
|
||||
total_llm: 4
|
||||
stage_workers:
|
||||
extract: 4
|
||||
```
|
||||
|
||||
Represent effective stage worker limits as `map[string]int`. Initially accept
|
||||
only `extract`; reject unknown keys. Missing extract defaults to `total_llm` and
|
||||
its valid range is `1..total_llm`. Add the environment override
|
||||
`NOTARIUS_STAGE_WORKERS_EXTRACT`. Preserve precedence rules and keep the file
|
||||
configuration version at 2.
|
||||
|
||||
Use one run-wide fixed extract worker pool. Dispatch jobs in round-robin order
|
||||
with source chunk as the outer loop and resolved lane as the inner loop. Use one
|
||||
job channel whose capacity equals the effective extract worker count, so the
|
||||
dispatcher applies bounded backpressure. Do not create one goroutine per job. A
|
||||
job includes extract retries and extract-stage validators. Store results by
|
||||
lane index and chunk index.
|
||||
|
||||
When all extract jobs for a lane are terminal, run that lane's merge and then
|
||||
normalize serially. Lane continuations may overlap. All LLM calls at all stages,
|
||||
including retries and validators, use the single injected scheduled client, so
|
||||
`total_llm` remains the authoritative process-wide provider-call ceiling.
|
||||
|
||||
Rejections are terminal results and do not cancel other work. A framework error
|
||||
cancels a derived run context, stops dispatching jobs not yet started, and waits
|
||||
for started tasks to finish or observe cancellation. Choose the returned error
|
||||
as follows:
|
||||
|
||||
1. if the parent context is canceled, return its error;
|
||||
2. otherwise discard internal `context.Canceled`/`DeadlineExceeded` errors when
|
||||
at least one non-context framework error exists; and
|
||||
3. choose the earliest remaining error by stage order (`extract`, `merge`,
|
||||
`normalize`), resolved lane index, chunk index for chunk-scoped work, and
|
||||
configured validator/operation index.
|
||||
|
||||
Use a sentinel chunk index after all real chunks for lane-scoped merge and
|
||||
normalize errors. Retain other started-task errors only in opt-in diagnostics.
|
||||
Sort accepted artifacts, warnings, and rejections by resolved lane index, source
|
||||
chunk index where applicable, stage order, validator order, and original
|
||||
within-result order. Completion timing must not affect public output. Run output
|
||||
only if every lane has a successful or rejection-only terminal outcome and no
|
||||
framework error occurred.
|
||||
|
||||
## Staged Implementation
|
||||
|
||||
### Stage 1: Compatibility Baselines and ADR Acceptance
|
||||
|
||||
Goal: lock down behavior that subsequent internal migrations must preserve and
|
||||
record the architectural decisions as accepted.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Add semantic or golden compatibility tests for the maintained Seriatim-to-D&D
|
||||
path: durable output files and JSON, output index, manifest provenance,
|
||||
warnings, rejections, and stable lane/chunk ordering.
|
||||
- Snapshot production module keys, validator keys, default validator chains,
|
||||
prompt/schema identities, and maintained example/profile resolution in tests.
|
||||
- Strengthen runner tests for fixed topology, validator rejection as a nonfatal
|
||||
outcome, framework-error abort, retries, parent cancellation, checkpoint reuse
|
||||
and invalidation, diagnostics, and opt-in debug behavior.
|
||||
- Add an instrumented scheduled-client test showing that all existing production
|
||||
LLM callers share `concurrency.total_llm`. It need not demonstrate parallel
|
||||
lanes yet.
|
||||
- Review the three ADRs against this decision-complete plan, set their status to
|
||||
`Accepted`, and update their dates only if ADR policy requires an acceptance
|
||||
date. Do not rewrite accepted decision text after this stage.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- compatibility tests fail on an unintended durable-output, key, chain,
|
||||
provenance, or outcome-semantics change; and
|
||||
- ADR-0002, ADR-0003, and ADR-0004 are accepted.
|
||||
|
||||
### Stage 2: Registrar Composition Without Package Moves
|
||||
|
||||
Goal: replace CLI leaf-by-leaf registration with package-family composition
|
||||
before changing imports.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Add `internal/modules/generic/register`, `internal/modules/seriatim/register`,
|
||||
and `internal/modules/dnd/register` using the fixed registrar signature.
|
||||
- Initially let those registrars import the existing stage-oriented packages.
|
||||
Move ownership of production validators, default chains, and prompt assets out
|
||||
of `internal/cli/catalog.go` and into the appropriate registrar.
|
||||
- Have the CLI allocate complete registries and the asset registry once, invoke
|
||||
generic, Seriatim, then D&D registration, and retain its existing test
|
||||
injection paths.
|
||||
- Test nil registry handling, duplicate registration errors, stable registered
|
||||
keys, default chains, and asset identities.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- the CLI composition root names only the three registrar packages, framework
|
||||
registry types, and asset registry; and
|
||||
- no production key, chain, prompt, schema, or runtime behavior changes.
|
||||
|
||||
### Stage 3: Mechanical Generic and Seriatim Package Moves
|
||||
|
||||
Goal: establish the domain-first generic and source-format trees without
|
||||
changing contracts.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Move the Seriatim adapter and tests to
|
||||
`internal/modules/seriatim/input/transcript`.
|
||||
- Move the generic chunker to `internal/modules/generic/chunk/units`, retaining
|
||||
the configured key `generic`.
|
||||
- Move append-order merge, no-op normalize, JSON output, and all generic
|
||||
validators to their target paths under `internal/modules/generic`.
|
||||
- Update only registrar imports and affected black-box tests. Preserve package
|
||||
behavior and all public registry keys.
|
||||
- Remove the emptied old directories.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- generic and Seriatim production implementations exist only under their target
|
||||
trees; and
|
||||
- compatibility baselines remain green.
|
||||
|
||||
### Stage 4: Mechanical D&D Package Move and Import Guard
|
||||
|
||||
Goal: establish the D&D tree and enforce ADR-0004 dependency direction while
|
||||
legacy contracts are still intact.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Move the D&D scenes chunker, spell extractor, validators, schemas, prompt
|
||||
assets, and D&D shared helpers into the target D&D tree. Do not create the
|
||||
typed root model or codec yet.
|
||||
- Move domain-neutral prompt filesystem helpers from
|
||||
`internal/modules/sharedassets` to `internal/framework/promptfs`.
|
||||
- Move tests with their owning implementation. Relocate tests that intentionally
|
||||
compose domains to a black-box integration-test package rather than creating
|
||||
peer-domain production imports.
|
||||
- Add a Go-parser-based import-boundary test. It must reject concrete
|
||||
D&D-to-Seriatim and Seriatim-to-D&D imports, all generic-to-D&D imports, and
|
||||
root-domain imports of child implementations. Allow domain registrars, the CLI
|
||||
composition root, and designated external integration tests to compose
|
||||
packages.
|
||||
- Remove old empty stage-oriented and validator directories.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- all production extensions use the target domain-first package layout except
|
||||
the not-yet-created typed codec/model pieces;
|
||||
- the import guard detects a deliberate fixture violation; and
|
||||
- behavior and keys remain unchanged.
|
||||
|
||||
### Stage 5: Source-Unit Provenance
|
||||
|
||||
Goal: add canonical provenance to engine-owned source units without yet changing
|
||||
the chunk type.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Add `Ref source.SourceRef` to `source.SourceUnit`, including clone and debug
|
||||
representations.
|
||||
- Make the Seriatim adapter assign the fixed self-reference for every unit.
|
||||
- Extend `source.ValidateDocument` to require the unit reference's source ID to
|
||||
match the document, require start and end IDs to equal the unit ID, reject
|
||||
missing/invalid/reversed references, and preserve the existing unit-order and
|
||||
uniqueness checks.
|
||||
- Make source digests and source checkpoint serialization include the new
|
||||
reference deterministically.
|
||||
- Add focused tests for valid refs and missing, foreign, non-self, and reversed
|
||||
refs, plus source checkpoint/debug round trips.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- every produced source unit has a validated self-reference; and
|
||||
- source state preserves it through clone, debug, digest, and checkpoint paths.
|
||||
|
||||
### Stage 6: Engine-Owned Chunks and Workspace v2
|
||||
|
||||
Goal: finish the Zone-A source model and make its persisted compatibility break
|
||||
explicit.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Add `source.Chunk` with the fixed shape and update universal chunk contracts,
|
||||
modules, validators, runner code, checkpoints, debug envelopes, and tests to
|
||||
use it.
|
||||
- Derive `Chunk.Ref` from the first and last included unit references. Validate
|
||||
source identity, non-empty ordered units, contiguous boundary agreement, and
|
||||
exact correspondence between the chunk ref and first/last unit refs.
|
||||
- Remove `contracts.SourceChunk`, `StartUnitID`, and `EndUnitID` after all
|
||||
consumers migrate. Do not keep aliases.
|
||||
- Advance `workspace.WorkspaceSchemaVersion` to `notarius.workspace.v2`. Make
|
||||
loader behavior explicitly classify v1 as incompatible and recompute while
|
||||
leaving files untouched.
|
||||
- Update checkpoint identity/digest tests and operations documentation for the
|
||||
one-time v1 resume miss.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- no production code imports a framework-owned chunk type;
|
||||
- chunk provenance round-trips exactly; and
|
||||
- v1 workspaces are safely ignored while v2 workspaces reuse successfully.
|
||||
|
||||
### Stage 7: Artifact and Codec Foundation
|
||||
|
||||
Goal: add the typed primitives and prove strict serialization independently of
|
||||
production lanes.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Add the fixed artifact types, codec interface, schema digest helper, and clone
|
||||
helpers under `internal/framework/contracts`.
|
||||
- Add `ArtifactCodecRegistry` to `pipeline.Registries` and `ModuleCatalog`.
|
||||
- Implement generic codec registration and private erasure/type tracking.
|
||||
- Validate registration metadata and duplicates. Ensure erased encode/decode
|
||||
returns typed errors, never reflection panics.
|
||||
- Use two small test artifact types to cover registration, exact type identity,
|
||||
deterministic encoding, strict decoding, cloning, duplicate kind rejection,
|
||||
and schema metadata/digest behavior.
|
||||
- Wire the new empty registry through CLI/test registry constructors without
|
||||
changing production lane execution.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- codecs for heterogeneous test types can coexist and safely round-trip through
|
||||
erased framework storage; and
|
||||
- current production behavior remains on the legacy raw path and unchanged.
|
||||
|
||||
### Stage 8: Typed Contracts, Variants, and Resolution
|
||||
|
||||
Goal: resolve a complete type-compatible lane before executing it.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Add the typed stage, typed validator, chunk validator, serialized validator,
|
||||
and provenance wrapper contracts from the fixed decisions.
|
||||
- Extend extractor specs with artifact kind/type. Convert merger, normalizer,
|
||||
and typed validator registries to artifact-kind variants while retaining
|
||||
serialized-validator registration by key.
|
||||
- Implement free generic registration helpers and private erased entries.
|
||||
- Extend lane resolution to derive kind from extractor, require its codec,
|
||||
select exact merger/normalizer/validator variants, and include artifact/schema
|
||||
identity in resolved lanes and pipeline digest.
|
||||
- Keep legacy registration helpers only as explicitly named transitional APIs;
|
||||
do not let a raw registration satisfy a typed lane.
|
||||
- Add composition tests with two artifact types and heterogeneous lanes. Cover
|
||||
missing codec, missing variant, Go-type mismatch, duplicate variant, wrong
|
||||
validator kind, stable resolution order, and digest changes on schema identity
|
||||
or schema digest changes.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- heterogeneous typed test lanes resolve without module-facing erasure;
|
||||
- every incompatible selection fails before execution; and
|
||||
- existing raw production lanes continue to resolve only through their visible
|
||||
transitional path.
|
||||
|
||||
### Stage 9: Preparation and Construction Foundation
|
||||
|
||||
Goal: construct and validate an entire resolved pipeline before source work.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Add `ModuleDependencies`, `BuildRequest`, `PreparedPipeline`, and `Prepare` as
|
||||
specified.
|
||||
- Extend registry entries/specs with option validation and construction
|
||||
functions. Supply adapters for legacy zero-argument constructors during the
|
||||
migration.
|
||||
- Call option validation for every selected module and validator during
|
||||
resolution/config validation. Reject unknown fields and contextualize errors.
|
||||
- Have preparation construct all selected components in fixed order and retain
|
||||
immutable prepared lane executors.
|
||||
- Update the runner API so `Run` receives a prepared pipeline. At the CLI, create
|
||||
the shared scheduled LLM client, prepare, and only then invoke the runner.
|
||||
- Prove with fakes that malformed options, missing required LLM dependencies,
|
||||
and late-component construction failures occur before the input adapter's
|
||||
`Parse` method.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- all components are constructed before source work;
|
||||
- preparation errors identify exact scope and perform no operations; and
|
||||
- legacy production modules still run through temporary construction adapters.
|
||||
|
||||
### Stage 10: Migrate Universal Modules to Construction
|
||||
|
||||
Goal: remove legacy option/dependency handling from input, chunk, and output.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Give Seriatim input, generic units chunking, D&D scenes chunking, generic JSON
|
||||
output, and all applicable chunk validators implementation-owned option
|
||||
structs and strict decoders.
|
||||
- Build those implementations with decoded options and injected dependencies.
|
||||
Require the LLM client for D&D scenes; keep deterministic implementations
|
||||
independent of it.
|
||||
- Remove `Options` and `LLMClient` from the corresponding operation requests.
|
||||
Retain per-run source, reference, profile, session, and metadata fields.
|
||||
- Update registrars and focused tests. Verify options are decoded once during
|
||||
preparation and operation methods do not inspect raw maps.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- no universal production module parses raw options during execution; and
|
||||
- every universal LLM call uses the injected shared client.
|
||||
|
||||
### Stage 11: Canonical D&D Model, Codec, and Typed Extractor
|
||||
|
||||
Goal: establish the first production `T` and its extraction boundary.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Add canonical spell types at the D&D root using engine-owned source refs.
|
||||
- Add `internal/modules/dnd/codec/spells`, register kind `dnd/spell-list`, and
|
||||
make it own the existing durable `dnd_spells.v1.json` schema and strict stable
|
||||
encoding.
|
||||
- Keep the private spell-extraction LLM DTO and
|
||||
`dnd_spells_llm.v1.json` in the extractor package. Map canonicalized DTO values
|
||||
to `dnd.SpellList` and do not expose the DTO to validators or the codec.
|
||||
- Convert the extractor to `Extractor[dnd.SpellList]`, construction-time options
|
||||
and dependency injection. Preserve prompt assets, retries, warnings, evidence,
|
||||
and LLM response validation.
|
||||
- Register the codec and typed extractor from the D&D registrar.
|
||||
- Add codec compatibility tests against existing durable fixtures and tests
|
||||
proving LLM schema ownership is separate from durable schema ownership.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- the typed extractor returns the canonical domain model;
|
||||
- codec output is semantically identical to the maintained durable spell JSON;
|
||||
and
|
||||
- no downstream production consumer is switched until the next stages.
|
||||
|
||||
### Stage 12: Typed Validators and Generic Typed Strategies
|
||||
|
||||
Goal: complete all typed components required by the D&D lane.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Convert D&D spell shape, source-reference, and source-relatedness validators
|
||||
to `TypedValidator[dnd.SpellList]` with construction-time options/dependencies.
|
||||
- Remove JSON reparsing from those validators. Remove the duplicate
|
||||
`spellpayload` package after its final consumer migrates.
|
||||
- Convert `valid_json` and `valid_json_schema` to serialized validators. Register
|
||||
them so the framework uses the D&D codec when they occur in the spell chain.
|
||||
- Implement generic typed append-order merge and no-op normalize. In the D&D
|
||||
registrar, register D&D variants using a spell-list append function and
|
||||
`noop[dnd.SpellList]`.
|
||||
- Convert always-accept/reject into explicit chunk and typed variants and
|
||||
register the D&D variants without changing their keys. Verify serialized
|
||||
validators operate on the framework encoding at chunk and the codec encoding
|
||||
at artifact stages.
|
||||
- Test typed validator requests, source refs, relatedness LLM injection, generic
|
||||
strategy reuse with a second test type, default chain order, and rejection
|
||||
behavior.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- every selected spell-lane component has a compatible D&D typed variant;
|
||||
- generic packages import no D&D code; and
|
||||
- the duplicate validator payload model and inter-validator JSON parsing are
|
||||
gone.
|
||||
|
||||
### Stage 13: Typed D&D Runner Vertical Slice
|
||||
|
||||
Goal: execute one complete production lane as `dnd.SpellList` while retaining
|
||||
temporary raw output/checkpoint adapters.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Implement the erased typed lane executor and runner path for extract, stage
|
||||
retries, typed/serialized validation, merge, and normalize.
|
||||
- Keep accepted extract values indexed by chunk and pass them to merge in source
|
||||
order. Preserve warnings and rejections in stable scope order.
|
||||
- Add narrow transitional adapters from typed stage outputs to the existing raw
|
||||
checkpoint/debug/output envelopes. These adapters must use the registered
|
||||
codec and be named/commented as migration-only.
|
||||
- Route the production D&D lane through the typed path; leave no production raw
|
||||
D&D stage module registered in parallel.
|
||||
- Add end-to-end and checkpoint-disabled tests proving the maintained D&D output
|
||||
and outcome semantics are unchanged. Add incompatible-lane tests proving
|
||||
failure occurs before input.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- D&D values remain typed from extractor through normalize and typed validation;
|
||||
- the runner's only D&D erasure is its private lane adapter and explicit codec
|
||||
boundary; and
|
||||
- durable output remains unchanged through the transitional adapter.
|
||||
|
||||
### Stage 14: Typed Checkpoints, Debugging, and Output
|
||||
|
||||
Goal: move every serialization side effect and the final Zone-C boundary to the
|
||||
codec model.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Change runner/output contracts so final normalized results are
|
||||
`SerializedArtifact` values. Update generic JSON output without importing D&D.
|
||||
- Preserve logical file names, index fields, media types, manifest contents, and
|
||||
durable spell JSON.
|
||||
- Change extract, merge, and normalize checkpoints to store codec bytes plus
|
||||
artifact kind, schema ID, version, and schema digest. Decode reused values
|
||||
back to `T` before the next stage.
|
||||
- Implement safe invalidation for missing/mismatched codecs and decode failure,
|
||||
with explicit checkpoint-event reasons.
|
||||
- Change typed debug envelopes to serialize through the codec, preserving opt-in
|
||||
and redaction behavior. Use stable codec bytes for artifact digests.
|
||||
- Remove the transitional raw output/checkpoint/debug adapters introduced in
|
||||
Stage 13 after all tests use the typed boundaries.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- typed values round-trip at every stage checkpoint;
|
||||
- incompatible checkpoint artifacts recompute safely;
|
||||
- output and debug code are domain-neutral; and
|
||||
- no D&D typed lane depends on a raw-boundary adapter.
|
||||
|
||||
### Stage 15: Finish Construction Migration and Remove Raw Contracts
|
||||
|
||||
Goal: leave one production extension system rather than parallel raw and typed
|
||||
models.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Migrate any remaining merge, normalize, extract, and validator implementations
|
||||
to construction-time option decoding and dependency injection.
|
||||
- Remove LLM clients and raw option maps from all remaining operation requests.
|
||||
- Remove legacy raw extractor/merger/normalizer/validator contracts,
|
||||
constructors, registry entries, `RawPayload`, `ResponseSchema` if superseded,
|
||||
raw clone helpers, raw checkpoint envelopes, and migration-only adapters.
|
||||
- Remove dead duplicate models and compatibility helpers. Search for production
|
||||
references to old stage-oriented paths and raw Zone-B types.
|
||||
- Keep serialized artifacts only at codec, checkpoint/debug, and output
|
||||
boundaries.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- no production Zone-B handoff uses JSON bytes, `RawPayload`, or `any`;
|
||||
- all production configuration options are validated before execution;
|
||||
- all production LLM users receive the one injected client; and
|
||||
- there is no legacy production registration path.
|
||||
|
||||
### Stage 16: Stage-Worker Configuration
|
||||
|
||||
Goal: add the decided scheduling control without changing runner execution yet.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Add `StageWorkers map[string]int` to effective concurrency configuration and
|
||||
`stage_workers` to the version-2 YAML shape. Deep-clone the map.
|
||||
- Accept only `extract`, reject unknown or empty keys, default missing extract to
|
||||
effective `total_llm`, and validate the inclusive range
|
||||
`1..total_llm` after file/environment precedence resolves.
|
||||
- Add `NOTARIUS_STAGE_WORKERS_EXTRACT` with the existing environment precedence
|
||||
and integer error style.
|
||||
- Preserve redaction/effective-config diagnostics and version 2.
|
||||
- Update `docs/config.md` and maintained examples that intentionally demonstrate
|
||||
concurrency. Do not add the field to every example when the default conveys
|
||||
the intended behavior.
|
||||
- Add file, default, merge/precedence, environment, unknown-key, boundary, and
|
||||
redacted-effective-config tests.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- every run has a validated effective extract worker count;
|
||||
- omitted configuration preserves current effective behavior at the default
|
||||
`total_llm: 1`; and
|
||||
- current configuration documentation owns the implemented contract.
|
||||
|
||||
### Stage 17: Concurrent Lane and Extract Scheduling
|
||||
|
||||
Goal: implement bounded concurrent lanes while preserving deterministic public
|
||||
behavior and the global LLM invariant.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Add the fixed-size run-wide worker pool and central round-robin dispatcher.
|
||||
Bound queued work so dispatch applies backpressure; do not enqueue the whole
|
||||
run into unbounded memory.
|
||||
- Treat extract, retries, and extract validators as one job. Publish immutable
|
||||
results to a coordinator indexed by lane and chunk.
|
||||
- Start each lane's serial merge/normalize continuation only after all its
|
||||
extract jobs are terminal. Permit different lane continuations to overlap.
|
||||
- Make the coordinator the sole writer of aggregate output, manifest,
|
||||
checkpoint-event collection, warnings, and rejections. Use attempt-specific
|
||||
debug paths and synchronize any recorder state that remains shared.
|
||||
- Implement rejection, cancellation, stable sorting, deterministic primary
|
||||
error selection, and output gating exactly as specified under Concurrency.
|
||||
- Audit every concurrently reused extractor, validator, codec, LLM/debug wrapper,
|
||||
checkpoint loader/recorder, and manifest metadata provider. Make production
|
||||
implementations immutable or add narrowly scoped synchronization.
|
||||
- Add deterministic barrier-controlled tests that force reverse completion
|
||||
order, simultaneous failures, parent cancellation, rejections mixed with
|
||||
successes, lane continuation overlap, and undispatched-job cancellation.
|
||||
- Add instrumented integration tests issuing calls from multiple lanes, retries,
|
||||
and LLM-backed validators. Assert provider calls never exceed
|
||||
`total_llm`, extract jobs never exceed the effective extract worker count, and
|
||||
both limits are exercised independently.
|
||||
- Run `go test -race ./...` and eliminate races rather than weakening tests.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- lanes and extract jobs actually overlap when configured above one;
|
||||
- job and provider-call limits are independently enforced;
|
||||
- public results and primary errors are identical across forced completion
|
||||
orders; and
|
||||
- full race testing passes.
|
||||
|
||||
### Stage 18: Documentation, Cleanup, and Final Verification
|
||||
|
||||
Goal: make the implemented repository and its canonical documentation agree,
|
||||
then close the focused roadmap.
|
||||
|
||||
Tasks:
|
||||
|
||||
- Review every current-behavior document routed by `docs/development.md` and
|
||||
update only its owned facts: architecture and dependency direction, package
|
||||
inventory, pipeline resolution/preparation/execution, typed module contracts,
|
||||
LLM scheduling, configuration, checkpoint compatibility, diagnostics,
|
||||
operations, and durable integration contracts.
|
||||
- Verify maintained examples and copyable files against the implementation.
|
||||
- Remove stale old package paths, raw-contract terminology, and superseded
|
||||
future-work entries. Validate all changed documentation links.
|
||||
- Update ADR consequences only in ways permitted for accepted ADR metadata; do
|
||||
not edit accepted decision text. Record a new superseding ADR if final code
|
||||
required an architectural change.
|
||||
- Mark the feature roadmap implemented and reduce this implementation plan to a
|
||||
concise completion record, or move it to the repository's established
|
||||
completed-roadmap location if one exists. Do not let this file become a second
|
||||
current-behavior reference.
|
||||
- Run final checks:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
```
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- all feature-roadmap completion criteria are met;
|
||||
- no stale production paths or legacy typed/raw bridge remain;
|
||||
- current documentation and examples describe only implemented behavior; and
|
||||
- all final validation commands pass.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The feature-policy decisions and the implementation choices required to
|
||||
begin each stage are resolved above. If implementation evidence contradicts one
|
||||
of the accepted architectural decisions, stop and handle that as an ADR change
|
||||
or supersession rather than treating it as an implicit implementation choice.
|
||||
## Delivered
|
||||
|
||||
- Production extensions use domain-first packages and package-family
|
||||
registrars without changing selectable module or validator keys.
|
||||
- Source units and chunks carry canonical engine-owned provenance using
|
||||
`source.SourceRef`; workspace checkpoints use schema v2.
|
||||
- Artifact lanes keep one domain-owned Go type through extraction, merge,
|
||||
normalization, and typed validation.
|
||||
- Artifact codecs own stable schema-aware checkpoint, debug, and output
|
||||
serialization; generic representation validators consume serialized views.
|
||||
- Resolution verifies artifact-kind, codec, typed variant, capability,
|
||||
reference, validator, and option compatibility before source work.
|
||||
- Preparation constructs the full run-local implementation set and injects one
|
||||
shared scheduled LLM client.
|
||||
- Extraction uses bounded chunk-first, lane-second dispatch. Lane continuations
|
||||
are also bounded, public results are deterministic, and framework failures
|
||||
cancel undispatched work without treating rejections as errors.
|
||||
- Maintained D&D payloads, logical output paths, prompt and schema identities,
|
||||
default validators, warnings, rejections, and manifest provenance remain
|
||||
covered by compatibility tests.
|
||||
|
||||
## Completion Evidence
|
||||
|
||||
Focused tests cover durable compatibility, typed resolution and preparation,
|
||||
codec boundaries, checkpoint invalidation, debug recording, bounded scheduling,
|
||||
reverse completion, cancellation, rejection, retries, and independent extract
|
||||
worker and provider-call limits. Maintained example configurations and input are
|
||||
validated and exercised by the CLI test suite.
|
||||
|
||||
Repository validation is defined in [Development](../development.md). The Go
|
||||
race suite requires a CGO-capable toolchain in the execution environment.
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
version: 2
|
||||
concurrency:
|
||||
total_llm: 1
|
||||
stage_workers:
|
||||
extract: 1
|
||||
workspace:
|
||||
directory: /var/lib/notarius
|
||||
diagnostics:
|
||||
|
||||
@@ -9,26 +9,21 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/dnd/scenes"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/extract/dnd/spells"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
|
||||
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
|
||||
spellshape "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/shape"
|
||||
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_refs"
|
||||
spellrelatedness "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_relatedness"
|
||||
alwaysaccept "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_accept"
|
||||
alwaysreject "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_reject"
|
||||
validjson "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json"
|
||||
validjsonschema "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json_schema"
|
||||
dndregister "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/register"
|
||||
genericregister "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/register"
|
||||
seriatimregister "gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/register"
|
||||
)
|
||||
|
||||
func productionRegistries() (pipeline.Registries, error) {
|
||||
type productionComponents struct {
|
||||
registries pipeline.Registries
|
||||
assets *llm.AssetRegistry
|
||||
}
|
||||
|
||||
func newProductionComponents() (productionComponents, error) {
|
||||
registries := pipeline.Registries{
|
||||
Inputs: pipeline.NewInputAdapterRegistry(),
|
||||
Chunkers: pipeline.NewChunkerRegistry(),
|
||||
ArtifactCodecs: pipeline.NewArtifactCodecRegistry(),
|
||||
Extractors: pipeline.NewExtractorRegistry(),
|
||||
Mergers: pipeline.NewMergerRegistry(),
|
||||
Normalizers: pipeline.NewNormalizerRegistry(),
|
||||
@@ -36,72 +31,26 @@ func productionRegistries() (pipeline.Registries, error) {
|
||||
ValidatorChains: pipeline.NewValidatorChainRegistry(),
|
||||
Outputs: pipeline.NewOutputEncoderRegistry(),
|
||||
}
|
||||
if err := seriatim.Register(registries.Inputs); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register seriatim input: %w", err)
|
||||
}
|
||||
if err := generic.Register(registries.Chunkers); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register generic chunker: %w", err)
|
||||
}
|
||||
if err := scenes.Register(registries.Chunkers); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register dnd scenes chunker: %w", err)
|
||||
}
|
||||
if err := spells.Register(registries.Extractors); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register dnd spells extractor: %w", err)
|
||||
}
|
||||
if err := appendorder.Register(registries.Mergers); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register appendorder merger: %w", err)
|
||||
}
|
||||
if err := noop.Register(registries.Normalizers); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register noop normalizer: %w", err)
|
||||
}
|
||||
if err := registerProductionValidators(registries.Validators); err != nil {
|
||||
return pipeline.Registries{}, err
|
||||
}
|
||||
if err := registerProductionValidatorChains(registries.ValidatorChains); err != nil {
|
||||
return pipeline.Registries{}, err
|
||||
}
|
||||
if err := jsonoutput.Register(registries.Outputs); err != nil {
|
||||
return pipeline.Registries{}, fmt.Errorf("register json output encoder: %w", err)
|
||||
}
|
||||
return registries, nil
|
||||
}
|
||||
|
||||
func registerProductionValidators(registry *pipeline.ValidatorRegistry) error {
|
||||
registrations := []struct {
|
||||
assets := llm.NewAssetRegistry()
|
||||
registrars := []struct {
|
||||
name string
|
||||
register func(*pipeline.ValidatorRegistry) error
|
||||
register func(pipeline.Registries, *llm.AssetRegistry) error
|
||||
}{
|
||||
{name: "generic always accept validator", register: alwaysaccept.Register},
|
||||
{name: "generic always reject validator", register: alwaysreject.Register},
|
||||
{name: "generic valid json validator", register: validjson.Register},
|
||||
{name: "generic valid json schema validator", register: validjsonschema.Register},
|
||||
{name: "dnd spell shape validator", register: spellshape.Register},
|
||||
{name: "dnd spell source references validator", register: spellsourcerefs.Register},
|
||||
{name: "dnd spell source relatedness validator", register: spellrelatedness.Register},
|
||||
{name: "generic", register: genericregister.Register},
|
||||
{name: "seriatim", register: seriatimregister.Register},
|
||||
{name: "dnd", register: dndregister.Register},
|
||||
}
|
||||
for _, registration := range registrations {
|
||||
if err := registration.register(registry); err != nil {
|
||||
return fmt.Errorf("register %s: %w", registration.name, err)
|
||||
for _, registrar := range registrars {
|
||||
if err := registrar.register(registries, assets); err != nil {
|
||||
return productionComponents{}, fmt.Errorf("register %s module family: %w", registrar.name, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
return productionComponents{registries: registries, assets: assets}, nil
|
||||
}
|
||||
|
||||
func registerProductionValidatorChains(registry *pipeline.ValidatorChainRegistry) error {
|
||||
if err := registry.Register(pipeline.ValidatorChainMapping{
|
||||
Stage: pipeline.StageExtract,
|
||||
Module: spells.Key,
|
||||
Validators: []pipeline.ModuleBinding{
|
||||
pipeline.Binding(validjson.Key),
|
||||
pipeline.Binding(validjsonschema.Key),
|
||||
pipeline.Binding(spellshape.Key),
|
||||
pipeline.Binding(spellsourcerefs.Key),
|
||||
pipeline.Binding(spellrelatedness.Key),
|
||||
},
|
||||
}); err != nil {
|
||||
return fmt.Errorf("register dnd spells validator chain: %w", err)
|
||||
}
|
||||
return nil
|
||||
func productionRegistries() (pipeline.Registries, error) {
|
||||
components, err := newProductionComponents()
|
||||
return components.registries, err
|
||||
}
|
||||
|
||||
func productionCatalog() (pipeline.ModuleCatalog, error) {
|
||||
@@ -113,14 +62,8 @@ func productionCatalog() (pipeline.ModuleCatalog, error) {
|
||||
}
|
||||
|
||||
func productionPromptAssets() (*llm.AssetRegistry, error) {
|
||||
registry := llm.NewAssetRegistry()
|
||||
if err := scenes.RegisterPromptAssets(registry); err != nil {
|
||||
return nil, fmt.Errorf("register dnd scenes prompt assets: %w", err)
|
||||
}
|
||||
if err := spells.RegisterPromptAssets(registry); err != nil {
|
||||
return nil, fmt.Errorf("register dnd spells prompt assets: %w", err)
|
||||
}
|
||||
return registry, nil
|
||||
components, err := newProductionComponents()
|
||||
return components.assets, err
|
||||
}
|
||||
|
||||
func effectiveCatalog(opts Options) (pipeline.ModuleCatalog, error) {
|
||||
@@ -147,6 +90,7 @@ func catalogFromRegistries(registries pipeline.Registries) pipeline.ModuleCatalo
|
||||
return pipeline.ModuleCatalog{
|
||||
Inputs: registries.Inputs,
|
||||
Chunkers: registries.Chunkers,
|
||||
ArtifactCodecs: registries.ArtifactCodecs,
|
||||
Extractors: registries.Extractors,
|
||||
Mergers: registries.Mergers,
|
||||
Normalizers: registries.Normalizers,
|
||||
@@ -160,6 +104,7 @@ func registriesFromCatalog(catalog pipeline.ModuleCatalog) pipeline.Registries {
|
||||
return pipeline.Registries{
|
||||
Inputs: catalog.Inputs,
|
||||
Chunkers: catalog.Chunkers,
|
||||
ArtifactCodecs: catalog.ArtifactCodecs,
|
||||
Extractors: catalog.Extractors,
|
||||
Mergers: catalog.Mergers,
|
||||
Normalizers: catalog.Normalizers,
|
||||
@@ -172,6 +117,7 @@ func registriesFromCatalog(catalog pipeline.ModuleCatalog) pipeline.Registries {
|
||||
func isEmptyCatalog(catalog pipeline.ModuleCatalog) bool {
|
||||
return catalog.Inputs == nil &&
|
||||
catalog.Chunkers == nil &&
|
||||
catalog.ArtifactCodecs == nil &&
|
||||
catalog.Extractors == nil &&
|
||||
catalog.Mergers == nil &&
|
||||
catalog.Normalizers == nil &&
|
||||
@@ -183,6 +129,7 @@ func isEmptyCatalog(catalog pipeline.ModuleCatalog) bool {
|
||||
func isEmptyRegistries(registries pipeline.Registries) bool {
|
||||
return registries.Inputs == nil &&
|
||||
registries.Chunkers == nil &&
|
||||
registries.ArtifactCodecs == nil &&
|
||||
registries.Extractors == nil &&
|
||||
registries.Mergers == nil &&
|
||||
registries.Normalizers == nil &&
|
||||
@@ -199,6 +146,22 @@ func productionLLMClientFactory(ctx context.Context, cfg config.Config, profileI
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
return buildProductionLLMClient(ctx, cfg, profileID, assets)
|
||||
}
|
||||
|
||||
func productionLLMClientFactoryWithAssets(assets *llm.AssetRegistry) LLMClientFactory {
|
||||
return func(ctx context.Context, cfg config.Config, profileID string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
return buildProductionLLMClient(ctx, cfg, profileID, assets)
|
||||
}
|
||||
}
|
||||
|
||||
func buildProductionLLMClient(ctx context.Context, cfg config.Config, profileID string, assets *llm.AssetRegistry) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if assets == nil {
|
||||
return nil, nil, fmt.Errorf("production asset registry must not be nil")
|
||||
}
|
||||
recorder := llm.NewLLMProfileRecorder()
|
||||
client, err := llm.NewScriptoriumClient(llm.ScriptoriumClientConfig{
|
||||
ProfileDir: cfg.Scriptorium.ProfileDir,
|
||||
|
||||
600
internal/cli/compatibility_test.go
Normal file
600
internal/cli/compatibility_test.go
Normal file
@@ -0,0 +1,600 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io/fs"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
frameworkllm "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
|
||||
spellshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/shape"
|
||||
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/source_refs"
|
||||
spellrelatedness "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/source_relatedness"
|
||||
validjson "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/valid_json"
|
||||
validjsonschema "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/valid_json_schema"
|
||||
)
|
||||
|
||||
func TestProductionCompatibilitySnapshot(t *testing.T) {
|
||||
normalizedOptions, err := normalizeOptions(Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("normalizeOptions() error = %v, want nil", err)
|
||||
}
|
||||
if normalizedOptions.Catalog.Inputs != normalizedOptions.Registries.Inputs ||
|
||||
normalizedOptions.Catalog.Chunkers != normalizedOptions.Registries.Chunkers ||
|
||||
normalizedOptions.Catalog.Extractors != normalizedOptions.Registries.Extractors ||
|
||||
normalizedOptions.Catalog.Mergers != normalizedOptions.Registries.Mergers ||
|
||||
normalizedOptions.Catalog.Normalizers != normalizedOptions.Registries.Normalizers ||
|
||||
normalizedOptions.Catalog.Validators != normalizedOptions.Registries.Validators ||
|
||||
normalizedOptions.Catalog.ValidatorChains != normalizedOptions.Registries.ValidatorChains ||
|
||||
normalizedOptions.Catalog.Outputs != normalizedOptions.Registries.Outputs {
|
||||
t.Fatal("production catalog and execution registries do not share one composition")
|
||||
}
|
||||
if normalizedOptions.LLMClientFactory == nil {
|
||||
t.Fatal("production LLM client factory is nil")
|
||||
}
|
||||
|
||||
registries, err := productionRegistries()
|
||||
if err != nil {
|
||||
t.Fatalf("productionRegistries() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
keySnapshots := []struct {
|
||||
name string
|
||||
got []string
|
||||
want []string
|
||||
}{
|
||||
{name: "inputs", got: registries.Inputs.RegisteredKeys(), want: []string{"seriatim"}},
|
||||
{name: "chunkers", got: registries.Chunkers.RegisteredKeys(), want: []string{"dnd/scenes", "generic"}},
|
||||
{name: "extractors", got: registries.Extractors.RegisteredKeys(), want: []string{"dnd/spells"}},
|
||||
{name: "mergers", got: registries.Mergers.RegisteredKeys(), want: []string{"appendorder"}},
|
||||
{name: "normalizers", got: registries.Normalizers.RegisteredKeys(), want: []string{"noop"}},
|
||||
{name: "validators", got: registries.Validators.RegisteredKeys(), want: []string{
|
||||
"extract/dnd/spells/shape", "extract/dnd/spells/source_refs", "extract/dnd/spells/source_relatedness",
|
||||
"generic/always_accept", "generic/always_reject", "generic/valid_json", "generic/valid_json_schema",
|
||||
}},
|
||||
{name: "outputs", got: registries.Outputs.RegisteredKeys(), want: []string{"json"}},
|
||||
}
|
||||
for _, snapshot := range keySnapshots {
|
||||
t.Run(snapshot.name, func(t *testing.T) {
|
||||
if !reflect.DeepEqual(snapshot.got, snapshot.want) {
|
||||
t.Fatalf("registered keys = %#v, want compatibility snapshot %#v", snapshot.got, snapshot.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
wantChain := []pipeline.ModuleBinding{
|
||||
pipeline.Binding(validjson.Key),
|
||||
pipeline.Binding(validjsonschema.Key),
|
||||
pipeline.Binding(spellshape.Key),
|
||||
pipeline.Binding(spellsourcerefs.Key),
|
||||
pipeline.Binding(spellrelatedness.Key),
|
||||
}
|
||||
if got := registries.ValidatorChains.Validators(pipeline.StageExtract, spells.Key); !reflect.DeepEqual(got, wantChain) {
|
||||
t.Fatalf("spell validator chain = %#v, want compatibility snapshot %#v", got, wantChain)
|
||||
}
|
||||
|
||||
assets, err := productionPromptAssets()
|
||||
if err != nil {
|
||||
t.Fatalf("productionPromptAssets() error = %v, want nil", err)
|
||||
}
|
||||
assertAssetNames(t, assets.PromptFS, []string{
|
||||
"dnd.scenes/dnd.scenes.yaml",
|
||||
"dnd.scenes/instructions.md",
|
||||
"dnd.scenes/sharedassets/common-dnd-references.md",
|
||||
"dnd.scenes/sharedassets/common-dnd-system.md",
|
||||
"dnd.scenes/sharedassets/common-dnd-transcript.md",
|
||||
"dnd.scenes/task.md",
|
||||
"dnd.spells/dnd.spells.yaml",
|
||||
"dnd.spells/instructions.md",
|
||||
"dnd.spells/sharedassets/common-dnd-references.md",
|
||||
"dnd.spells/sharedassets/common-dnd-system.md",
|
||||
"dnd.spells/sharedassets/common-dnd-transcript.md",
|
||||
"dnd.spells/task.md",
|
||||
})
|
||||
assertAssetNames(t, assets.SchemaFS, []string{
|
||||
"dnd_scenes.v1.json",
|
||||
"dnd_spells_llm.v1.json",
|
||||
})
|
||||
|
||||
identitySnapshot := map[string]map[string]any{
|
||||
"scenes": sceneManifestMetadata(t),
|
||||
"spells": spellManifestMetadata(t),
|
||||
}
|
||||
for name, metadata := range identitySnapshot {
|
||||
for _, key := range []string{"prompt_id", "prompt_version", "prompt_sha256", "response_schema_key", "response_schema_id", "response_schema_name", "response_schema_version", "response_schema_sha256"} {
|
||||
if value, ok := metadata[key].(string); !ok || value == "" {
|
||||
t.Fatalf("%s metadata[%q] = %#v, want non-empty identity", name, key, metadata[key])
|
||||
}
|
||||
}
|
||||
}
|
||||
if got := []any{
|
||||
identitySnapshot["scenes"]["prompt_id"], identitySnapshot["scenes"]["prompt_version"],
|
||||
identitySnapshot["scenes"]["response_schema_key"], identitySnapshot["scenes"]["response_schema_id"], identitySnapshot["scenes"]["response_schema_name"], identitySnapshot["scenes"]["response_schema_version"],
|
||||
}; !reflect.DeepEqual(got, []any{"dnd.scenes", "v1", "dnd_scenes", "notarius.dnd.scenes", "notarius_dnd_scenes_v1", "v1"}) {
|
||||
t.Fatalf("scene identities = %#v, want compatibility snapshot", got)
|
||||
}
|
||||
if got := []any{
|
||||
identitySnapshot["spells"]["prompt_id"], identitySnapshot["spells"]["prompt_version"],
|
||||
identitySnapshot["spells"]["response_schema_key"], identitySnapshot["spells"]["response_schema_id"], identitySnapshot["spells"]["response_schema_name"], identitySnapshot["spells"]["response_schema_version"],
|
||||
}; !reflect.DeepEqual(got, []any{"dnd.spells", "v1", "dnd_spells", "notarius.dnd.spells", "notarius_dnd_spells_v1", "v1"}) {
|
||||
t.Fatalf("spell identities = %#v, want compatibility snapshot", got)
|
||||
}
|
||||
|
||||
fileConfig, err := config.LoadFileConfig(fixturePath(t, "examples/dnd-spells.config.yml"))
|
||||
if err != nil {
|
||||
t.Fatalf("LoadFileConfig() error = %v, want nil", err)
|
||||
}
|
||||
cfg := config.Default()
|
||||
if err := cfg.ApplyFileConfig(fileConfig); err != nil {
|
||||
t.Fatalf("ApplyFileConfig() error = %v, want nil", err)
|
||||
}
|
||||
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-session", Catalog: catalogFromRegistries(registries)})
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve() error = %v, want nil", err)
|
||||
}
|
||||
resolved := effective.ResolvedPipeline
|
||||
if resolved.Input.Module != "seriatim" || resolved.Chunk.Module != "generic" || resolved.Output.Module != "json" || len(resolved.ArtifactLanes) != 1 {
|
||||
t.Fatalf("resolved example = %#v, want maintained production topology", resolved)
|
||||
}
|
||||
lane := resolved.ArtifactLanes[0]
|
||||
if lane.ID != "spells" || lane.Extract.Module != "dnd/spells" || lane.Merge.Module != "appendorder" || lane.Normalize.Module != "noop" {
|
||||
t.Fatalf("resolved lane = %#v, want maintained spell lane", lane)
|
||||
}
|
||||
if got := resolvedValidatorKeys(resolved.ValidatorChains, pipeline.StageExtract, "spells", spells.Key); !reflect.DeepEqual(got, []string{
|
||||
"generic/valid_json", "generic/valid_json_schema", "extract/dnd/spells/shape", "extract/dnd/spells/source_refs", "extract/dnd/spells/source_relatedness",
|
||||
}) {
|
||||
t.Fatalf("resolved validator keys = %#v, want compatibility snapshot", got)
|
||||
}
|
||||
|
||||
productionFileConfig, err := config.LoadFileConfig(fixturePath(t, "examples/dnd-spells-production.config.yml"))
|
||||
if err != nil {
|
||||
t.Fatalf("LoadFileConfig(production example) error = %v, want nil", err)
|
||||
}
|
||||
productionConfig := config.Default()
|
||||
if err := productionConfig.ApplyFileConfig(productionFileConfig); err != nil {
|
||||
t.Fatalf("ApplyFileConfig(production example) error = %v, want nil", err)
|
||||
}
|
||||
productionEffective, err := productionConfig.Resolve(config.ResolveInput{PipelineID: "dnd-session", Catalog: catalogFromRegistries(registries)})
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve(production example) error = %v, want nil", err)
|
||||
}
|
||||
productionResolved := productionEffective.ResolvedPipeline
|
||||
if productionConfig.Concurrency.TotalLLM != 1 || productionConfig.Concurrency.StageWorkers["extract"] != 1 || !reflect.DeepEqual(productionResolved.Chunk.Options, map[string]any{"max_units": 50}) {
|
||||
t.Fatalf("production example concurrency/options = %#v/%#v, want compatibility snapshot", productionConfig.Concurrency, productionResolved.Chunk.Options)
|
||||
}
|
||||
bindings := productionResolved.ArtifactLanes[0].ExtractReferences.Bindings
|
||||
if len(bindings) != 2 || bindings[0].SlotName != "glossary" || bindings[0].Source != "./dnd-spells-glossary.txt" || bindings[1].SlotName != "party" || bindings[1].Source != "./dnd-spells-roster.txt" {
|
||||
t.Fatalf("production example reference bindings = %#v, want maintained glossary and party bindings", bindings)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMaintainedSeriatimToDNDCompatibilityBundle(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
client contracts.StructuredLLMClient
|
||||
wantStatus string
|
||||
wantLaneFile bool
|
||||
wantRejectedCount int
|
||||
}{
|
||||
{name: "approved", client: newFakeRunLLMClient(false), wantStatus: "approved", wantLaneFile: true},
|
||||
{name: "validator rejection is nonfatal", client: newFakeRunLLMClient(true), wantStatus: "rejected", wantRejectedCount: 1},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
outputDir := t.TempDir()
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := RunWithOptions([]string{
|
||||
"run", "dnd-session",
|
||||
"--config", fixturePath(t, "examples/dnd-spells.config.yml"),
|
||||
"--input", fixturePath(t, "examples/seriatim-minimal-transcript.json"),
|
||||
"--output-dir", outputDir,
|
||||
"--diagnostics-dir", t.TempDir(),
|
||||
}, &stdout, &stderr, Options{LLMClientFactory: fakeLLMFactory(test.client, nil)})
|
||||
if code != 0 {
|
||||
t.Fatalf("RunWithOptions() code = %d, stderr=%q", code, stderr.String())
|
||||
}
|
||||
|
||||
runDir := onlyChildDir(t, outputDir)
|
||||
wantFiles := []string{"index.json", "manifest.json", "rejected.json", "warnings.json"}
|
||||
if test.wantLaneFile {
|
||||
wantFiles = append(wantFiles, "lanes/spells.json")
|
||||
}
|
||||
sort.Strings(wantFiles)
|
||||
if got := relativeFileNames(t, runDir); !reflect.DeepEqual(got, wantFiles) {
|
||||
t.Fatalf("durable files = %#v, want compatibility snapshot %#v", got, wantFiles)
|
||||
}
|
||||
|
||||
var manifest artifacts.RunManifest
|
||||
readJSONFile(t, filepath.Join(runDir, "manifest.json"), &manifest)
|
||||
if manifest.PipelineID != "dnd-session" || manifest.InputModule != "seriatim" || manifest.Chunker != "generic" || manifest.OutputEncoder != "json" {
|
||||
t.Fatalf("manifest module provenance = %#v, want maintained production modules", manifest)
|
||||
}
|
||||
if len(manifest.ArtifactLanes) != 1 || manifest.ArtifactLanes[0].ID != "spells" {
|
||||
t.Fatalf("artifact lanes = %#v, want one spells lane", manifest.ArtifactLanes)
|
||||
}
|
||||
laneManifest := manifest.ArtifactLanes[0]
|
||||
if laneManifest.Extractor != "dnd/spells" || laneManifest.Merger != "appendorder" || laneManifest.Normalizer != "noop" {
|
||||
t.Fatalf("manifest lane module provenance = %#v, want maintained production modules", laneManifest)
|
||||
}
|
||||
if len(manifest.Extractors) != 0 || manifest.Merger != "" || manifest.Normalizer != "" {
|
||||
t.Fatalf("legacy top-level lane summaries = %#v/%q/%q, want empty compatibility snapshot", manifest.Extractors, manifest.Merger, manifest.Normalizer)
|
||||
}
|
||||
if manifest.ValidationStatus != test.wantStatus || len(manifest.RejectedOutputs) != test.wantRejectedCount {
|
||||
t.Fatalf("manifest outcome = status %q rejected %#v, want %q/%d", manifest.ValidationStatus, manifest.RejectedOutputs, test.wantStatus, test.wantRejectedCount)
|
||||
}
|
||||
if !reflect.DeepEqual(manifest.SourceDigests, []string{"sha256:1c98d94ae632fb10a2b56f684cd4fb1019cedb1a629e57dc0977cf4a54135be0"}) {
|
||||
t.Fatalf("source digests = %#v, want maintained fixture provenance", manifest.SourceDigests)
|
||||
}
|
||||
if got := manifestValidatorKeys(manifestValidatorChain(t, manifest, pipeline.StageExtract, "spells", spells.Key)); !reflect.DeepEqual(got, []string{
|
||||
"generic/valid_json", "generic/valid_json_schema", "extract/dnd/spells/shape", "extract/dnd/spells/source_refs", "extract/dnd/spells/source_relatedness",
|
||||
}) {
|
||||
t.Fatalf("manifest validator chain = %#v, want compatibility snapshot", got)
|
||||
}
|
||||
|
||||
var index struct {
|
||||
ManifestFile string `json:"manifest_file"`
|
||||
OutputFiles []struct {
|
||||
LaneID string `json:"lane_id"`
|
||||
MediaType string `json:"media_type"`
|
||||
File string `json:"file"`
|
||||
ModuleKey string `json:"module_key"`
|
||||
SchemaID string `json:"schema_id"`
|
||||
SchemaName string `json:"schema_name"`
|
||||
SchemaVersion string `json:"schema_version"`
|
||||
} `json:"output_files"`
|
||||
RejectedFile string `json:"rejected_file"`
|
||||
WarningsFile string `json:"warnings_file"`
|
||||
}
|
||||
readJSONFile(t, filepath.Join(runDir, "index.json"), &index)
|
||||
if index.ManifestFile != "manifest.json" || index.RejectedFile != "rejected.json" || index.WarningsFile != "warnings.json" {
|
||||
t.Fatalf("output index fixed files = %#v, want compatibility snapshot", index)
|
||||
}
|
||||
if test.wantLaneFile {
|
||||
if len(index.OutputFiles) != 1 {
|
||||
t.Fatalf("output index entries = %#v, want one", index.OutputFiles)
|
||||
}
|
||||
wantOutput := struct {
|
||||
LaneID, MediaType, File, ModuleKey, SchemaID, SchemaName, SchemaVersion string
|
||||
}{"spells", "application/json", "lanes/spells.json", "noop", "notarius.dnd.spells", "notarius_dnd_spells_v1", "v1"}
|
||||
gotOutput := index.OutputFiles[0]
|
||||
got := struct {
|
||||
LaneID, MediaType, File, ModuleKey, SchemaID, SchemaName, SchemaVersion string
|
||||
}{gotOutput.LaneID, gotOutput.MediaType, gotOutput.File, gotOutput.ModuleKey, gotOutput.SchemaID, gotOutput.SchemaName, gotOutput.SchemaVersion}
|
||||
if got != wantOutput {
|
||||
t.Fatalf("output index entries = %#v, want compatibility snapshot %#v", index.OutputFiles, wantOutput)
|
||||
}
|
||||
assertJSONEqual(t, readFile(t, filepath.Join(runDir, "lanes/spells.json")), []byte(`{
|
||||
"spell_casts": [{
|
||||
"caster": "Aria",
|
||||
"spell": "Cure Wounds",
|
||||
"effect": "Heals a wounded ally.",
|
||||
"narrative_description": "Aria casts Cure Wounds.",
|
||||
"source_refs": [{"source_id": "session-alpha", "start_unit_id": 1, "end_unit_id": 1}]
|
||||
}]
|
||||
}`))
|
||||
} else if len(index.OutputFiles) != 0 {
|
||||
t.Fatalf("output index entries = %#v, want none for rejected lane", index.OutputFiles)
|
||||
}
|
||||
|
||||
var warnings struct {
|
||||
Warnings []contracts.Warning `json:"warnings"`
|
||||
}
|
||||
readJSONFile(t, filepath.Join(runDir, "warnings.json"), &warnings)
|
||||
if len(warnings.Warnings) != 0 {
|
||||
t.Fatalf("warnings = %#v, want empty compatibility snapshot", warnings.Warnings)
|
||||
}
|
||||
var rejected struct {
|
||||
Rejected []contracts.RejectedOutput `json:"rejected"`
|
||||
}
|
||||
readJSONFile(t, filepath.Join(runDir, "rejected.json"), &rejected)
|
||||
if len(rejected.Rejected) != test.wantRejectedCount {
|
||||
t.Fatalf("rejected outputs = %#v, want %d", rejected.Rejected, test.wantRejectedCount)
|
||||
}
|
||||
if test.wantRejectedCount == 1 {
|
||||
got := rejected.Rejected[0]
|
||||
if got.Stage != "extract" || got.LaneID != "spells" || got.ModuleKey != "dnd/spells" || got.ChunkID != "chunk-000001" || got.ChunkIndex != 0 || got.ValidatorName != "extract/dnd/spells/source_refs" || got.ReasonCode != "invalid_source_refs" || got.AttemptCount != 1 {
|
||||
t.Fatalf("rejection = %#v, want maintained nonfatal validator outcome", got)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestProductionLLMCallersShareScheduledClient(t *testing.T) {
|
||||
underlying := newBlockingProductionLLMClient()
|
||||
scheduler, err := frameworkllm.NewScheduler(1)
|
||||
if err != nil {
|
||||
t.Fatalf("NewScheduler() error = %v, want nil", err)
|
||||
}
|
||||
client := frameworkllm.NewScheduledClient(underlying, scheduler)
|
||||
doc := &source.SourceDocument{
|
||||
ID: "session-alpha", Kind: "transcript", Format: "application/json", Digest: "sha256:source",
|
||||
Units: []source.SourceUnit{
|
||||
{ID: 1, Kind: "segment", Text: "Aria casts Cure Wounds.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 1}},
|
||||
{ID: 2, Kind: "segment", Text: "The spell takes effect.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 2}},
|
||||
},
|
||||
}
|
||||
chunk := source.Chunk{
|
||||
ID: "session-alpha:chunk:0", SourceID: doc.ID, Index: 0, Ref: source.SourceRef{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 2},
|
||||
Content: []byte(`{"scene":"Aria casts Cure Wounds."}`), MediaType: "application/json", Units: append([]source.SourceUnit(nil), doc.Units...),
|
||||
}
|
||||
|
||||
var started sync.WaitGroup
|
||||
started.Add(2)
|
||||
errs := make(chan error, 2)
|
||||
go func() {
|
||||
started.Done()
|
||||
chunker, err := scenes.New(client, scenes.Options{})
|
||||
if err == nil {
|
||||
_, err = chunker.Chunk(context.Background(), contracts.ChunkRequest{Source: doc})
|
||||
}
|
||||
errs <- err
|
||||
}()
|
||||
go func() {
|
||||
started.Done()
|
||||
extractor, err := spells.New(client, spells.Options{})
|
||||
if err == nil {
|
||||
_, err = extractor.Extract(context.Background(), contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk})
|
||||
}
|
||||
errs <- err
|
||||
}()
|
||||
started.Wait()
|
||||
|
||||
for i := 0; i < 2; i++ {
|
||||
<-underlying.entered
|
||||
underlying.release <- struct{}{}
|
||||
}
|
||||
for i := 0; i < 2; i++ {
|
||||
if err := <-errs; err != nil {
|
||||
t.Fatalf("production LLM caller error = %v, want nil", err)
|
||||
}
|
||||
}
|
||||
if underlying.maxActive != 1 {
|
||||
t.Fatalf("maximum concurrent provider calls = %d, want total_llm limit 1", underlying.maxActive)
|
||||
}
|
||||
sort.Strings(underlying.stageNames)
|
||||
if !reflect.DeepEqual(underlying.stageNames, []string{"dnd/scenes", "dnd/spells"}) {
|
||||
t.Fatalf("scheduled stage names = %#v, want both production LLM callers", underlying.stageNames)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProductionBundlePreservesLaneAndChunkOrder(t *testing.T) {
|
||||
configPath := writeTestConfig(t, `version: 2
|
||||
pipelines:
|
||||
dnd-session:
|
||||
input: seriatim
|
||||
chunk:
|
||||
module: generic
|
||||
options:
|
||||
max_units: 1
|
||||
artifacts:
|
||||
zeta:
|
||||
extract: dnd/spells
|
||||
alpha:
|
||||
extract: dnd/spells
|
||||
`)
|
||||
outputDir := t.TempDir()
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
code := RunWithOptions([]string{
|
||||
"run", "dnd-session",
|
||||
"--config", configPath,
|
||||
"--input", fixturePath(t, "examples/seriatim-minimal-transcript.json"),
|
||||
"--output-dir", outputDir,
|
||||
"--diagnostics-dir", t.TempDir(),
|
||||
}, &stdout, &stderr, Options{LLMClientFactory: fakeLLMFactory(orderingProductionLLMClient{}, nil)})
|
||||
if code != 0 {
|
||||
t.Fatalf("RunWithOptions() code = %d, stderr=%q", code, stderr.String())
|
||||
}
|
||||
|
||||
runDir := onlyChildDir(t, outputDir)
|
||||
var index struct {
|
||||
OutputFiles []struct {
|
||||
LaneID string `json:"lane_id"`
|
||||
} `json:"output_files"`
|
||||
}
|
||||
readJSONFile(t, filepath.Join(runDir, "index.json"), &index)
|
||||
if len(index.OutputFiles) != 2 {
|
||||
t.Fatalf("output index entries = %#v, want two lanes", index.OutputFiles)
|
||||
}
|
||||
if got := []string{index.OutputFiles[0].LaneID, index.OutputFiles[1].LaneID}; !reflect.DeepEqual(got, []string{"alpha", "zeta"}) {
|
||||
t.Fatalf("output lane order = %#v, want resolved lane order", got)
|
||||
}
|
||||
for _, laneID := range []string{"alpha", "zeta"} {
|
||||
var payload struct {
|
||||
SpellCasts []struct {
|
||||
Spell string `json:"spell"`
|
||||
SourceRefs []source.SourceRef `json:"source_refs"`
|
||||
} `json:"spell_casts"`
|
||||
}
|
||||
readJSONFile(t, filepath.Join(runDir, "lanes", laneID+".json"), &payload)
|
||||
if len(payload.SpellCasts) != 2 {
|
||||
t.Fatalf("lane %q spell casts = %#v, want one per source chunk", laneID, payload.SpellCasts)
|
||||
}
|
||||
got := []any{
|
||||
payload.SpellCasts[0].Spell, payload.SpellCasts[0].SourceRefs[0].StartUnitID,
|
||||
payload.SpellCasts[1].Spell, payload.SpellCasts[1].SourceRefs[0].StartUnitID,
|
||||
}
|
||||
if !reflect.DeepEqual(got, []any{"Cure Wounds", 1, "Shield", 2}) {
|
||||
t.Fatalf("lane %q chunk handoff order = %#v, want source chunk order", laneID, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type blockingProductionLLMClient struct {
|
||||
mu sync.Mutex
|
||||
active int
|
||||
maxActive int
|
||||
stageNames []string
|
||||
entered chan struct{}
|
||||
release chan struct{}
|
||||
}
|
||||
|
||||
type orderingProductionLLMClient struct{}
|
||||
|
||||
func (orderingProductionLLMClient) CompleteStructured(_ context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
material := req.Inputs["transcript"]
|
||||
unitID := 1
|
||||
spellName := "Cure Wounds"
|
||||
if strings.Contains(string(material.Content), "Shield") {
|
||||
unitID = 2
|
||||
spellName = "Shield"
|
||||
}
|
||||
payload, err := json.Marshal(map[string]any{
|
||||
"spell_casts": []map[string]any{{
|
||||
"caster": "Aria",
|
||||
"spell": spellName,
|
||||
"effect": "Fixture effect.",
|
||||
"narrative_description": "Fixture spell cast.",
|
||||
"source_refs": []map[string]any{{
|
||||
"start_unit_id": unitID,
|
||||
"end_unit_id": unitID,
|
||||
}},
|
||||
}},
|
||||
})
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
if err := json.Unmarshal(payload, out); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{Content: payload}, nil
|
||||
}
|
||||
|
||||
func newBlockingProductionLLMClient() *blockingProductionLLMClient {
|
||||
return &blockingProductionLLMClient{entered: make(chan struct{}, 2), release: make(chan struct{}, 2)}
|
||||
}
|
||||
|
||||
func (client *blockingProductionLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
client.mu.Lock()
|
||||
client.active++
|
||||
if client.active > client.maxActive {
|
||||
client.maxActive = client.active
|
||||
}
|
||||
client.stageNames = append(client.stageNames, req.StageName)
|
||||
client.mu.Unlock()
|
||||
client.entered <- struct{}{}
|
||||
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return contracts.StructuredCompletionResponse{}, ctx.Err()
|
||||
case <-client.release:
|
||||
}
|
||||
|
||||
client.mu.Lock()
|
||||
client.active--
|
||||
client.mu.Unlock()
|
||||
|
||||
var payload []byte
|
||||
switch req.StageName {
|
||||
case scenes.Key:
|
||||
payload = []byte(`{"scenes":[{"start_unit_id":1,"end_unit_id":2,"short_title":"Spell","primary_mode":"Narrative","main_participants":["Aria"],"summary":"Aria casts a spell.","boundary_note":"Complete source.","boundary_confidence":"High"}],"boundary_caveats":[]}`)
|
||||
case spells.Key:
|
||||
payload = []byte(`{"spell_casts":[{"caster":"Aria","spell":"Cure Wounds","effect":"Healing","narrative_description":"Aria casts Cure Wounds.","source_refs":[{"start_unit_id":1,"end_unit_id":1}]}]}`)
|
||||
}
|
||||
if err := json.Unmarshal(payload, out); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{Content: payload}, nil
|
||||
}
|
||||
|
||||
func sceneManifestMetadata(t *testing.T) map[string]any {
|
||||
t.Helper()
|
||||
chunker, err := scenes.New(orderingProductionLLMClient{}, scenes.Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("construct scene chunker: %v", err)
|
||||
}
|
||||
return chunker.ManifestMetadata()
|
||||
}
|
||||
|
||||
func spellManifestMetadata(t *testing.T) map[string]any {
|
||||
t.Helper()
|
||||
extractor, err := spells.New(orderingProductionLLMClient{}, spells.Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("construct spell extractor: %v", err)
|
||||
}
|
||||
return extractor.ManifestMetadata()
|
||||
}
|
||||
|
||||
func assertAssetNames(t *testing.T, getFS func() (fs.FS, error), want []string) {
|
||||
t.Helper()
|
||||
fSys, err := getFS()
|
||||
if err != nil {
|
||||
t.Fatalf("asset filesystem error = %v, want nil", err)
|
||||
}
|
||||
var got []string
|
||||
if err := fs.WalkDir(fSys, ".", func(path string, entry fs.DirEntry, err error) error {
|
||||
if err == nil && !entry.IsDir() {
|
||||
got = append(got, path)
|
||||
}
|
||||
return err
|
||||
}); err != nil {
|
||||
t.Fatalf("walk assets: %v", err)
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("asset names = %#v, want compatibility snapshot %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func resolvedValidatorKeys(chains []pipeline.ResolvedValidatorChain, stage pipeline.ModuleStage, laneID, moduleKey string) []string {
|
||||
for _, chain := range chains {
|
||||
if chain.Stage == stage && chain.LaneID == laneID && chain.ModuleKey == moduleKey {
|
||||
keys := make([]string, 0, len(chain.Validators))
|
||||
for _, validator := range chain.Validators {
|
||||
keys = append(keys, validator.Binding.Module)
|
||||
}
|
||||
return keys
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func relativeFileNames(t *testing.T, root string) []string {
|
||||
t.Helper()
|
||||
var names []string
|
||||
if err := filepath.WalkDir(root, func(path string, entry fs.DirEntry, err error) error {
|
||||
if err != nil || entry.IsDir() {
|
||||
return err
|
||||
}
|
||||
rel, err := filepath.Rel(root, path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
names = append(names, filepath.ToSlash(rel))
|
||||
return nil
|
||||
}); err != nil {
|
||||
t.Fatalf("walk durable output: %v", err)
|
||||
}
|
||||
sort.Strings(names)
|
||||
return names
|
||||
}
|
||||
|
||||
func assertJSONEqual(t *testing.T, got, want []byte) {
|
||||
t.Helper()
|
||||
var gotValue any
|
||||
var wantValue any
|
||||
if err := json.Unmarshal(got, &gotValue); err != nil {
|
||||
t.Fatalf("unmarshal actual JSON: %v", err)
|
||||
}
|
||||
if err := json.Unmarshal(want, &wantValue); err != nil {
|
||||
t.Fatalf("unmarshal expected JSON: %v", err)
|
||||
}
|
||||
if !reflect.DeepEqual(gotValue, wantValue) {
|
||||
t.Fatalf("JSON = %#v, want compatibility snapshot %#v", gotValue, wantValue)
|
||||
}
|
||||
}
|
||||
@@ -51,7 +51,12 @@ func Run(args []string, stdout, stderr io.Writer) int {
|
||||
}
|
||||
|
||||
func RunWithOptions(args []string, stdout, stderr io.Writer, opts Options) int {
|
||||
opts = normalizeOptions(opts)
|
||||
var err error
|
||||
opts, err = normalizeOptions(opts)
|
||||
if err != nil {
|
||||
fmt.Fprintf(stderr, "notarius: %v\n", err)
|
||||
return 1
|
||||
}
|
||||
if len(args) == 0 {
|
||||
writeUsage(stdout)
|
||||
return 0
|
||||
@@ -78,17 +83,28 @@ func writeUsage(w io.Writer) {
|
||||
fmt.Fprint(w, usage)
|
||||
}
|
||||
|
||||
func normalizeOptions(opts Options) Options {
|
||||
func normalizeOptions(opts Options) (Options, error) {
|
||||
if opts.LookupEnv == nil {
|
||||
opts.LookupEnv = os.LookupEnv
|
||||
}
|
||||
if opts.Now == nil {
|
||||
opts.Now = time.Now
|
||||
}
|
||||
if isEmptyCatalog(opts.Catalog) && isEmptyRegistries(opts.Registries) {
|
||||
components, err := newProductionComponents()
|
||||
if err != nil {
|
||||
return Options{}, err
|
||||
}
|
||||
opts.Registries = components.registries
|
||||
opts.Catalog = catalogFromRegistries(components.registries)
|
||||
if opts.LLMClientFactory == nil {
|
||||
opts.LLMClientFactory = productionLLMClientFactoryWithAssets(components.assets)
|
||||
}
|
||||
}
|
||||
if opts.LLMClientFactory == nil {
|
||||
opts.LLMClientFactory = productionLLMClientFactory
|
||||
}
|
||||
return opts
|
||||
return opts, nil
|
||||
}
|
||||
|
||||
func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) int {
|
||||
@@ -195,6 +211,7 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
if err != nil {
|
||||
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("create debug recorder: %w", err))
|
||||
}
|
||||
debugRecorder = pipeline.SynchronizedDebugRecorder(debugRecorder)
|
||||
|
||||
catalog, err := effectiveCatalog(opts)
|
||||
if err != nil {
|
||||
@@ -247,11 +264,6 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("write diagnostics resolved references: %w", err))
|
||||
}
|
||||
|
||||
rawInput, err := os.ReadFile(strings.TrimSpace(*inputPath))
|
||||
if err != nil {
|
||||
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("read input %q: %w", strings.TrimSpace(*inputPath), err))
|
||||
}
|
||||
|
||||
registries, err := effectiveRegistries(opts)
|
||||
if err != nil {
|
||||
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
|
||||
@@ -266,25 +278,34 @@ func runPipelineCommand(args []string, stdout, stderr io.Writer, opts Options) i
|
||||
if err != nil {
|
||||
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("create LLM client for profile %q: %w", factoryProfileID, err))
|
||||
}
|
||||
llmClient = pipeline.WithDebugLLMRecording(llmClient, debugRecorder)
|
||||
prepared, err := pipeline.Prepare(effective.ResolvedPipeline, registries, pipeline.ModuleDependencies{LLM: llmClient})
|
||||
if err != nil {
|
||||
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("prepare pipeline %q: %w", pipelineID, err))
|
||||
}
|
||||
rawInput, err := os.ReadFile(strings.TrimSpace(*inputPath))
|
||||
if err != nil {
|
||||
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, fmt.Errorf("read input %q: %w", strings.TrimSpace(*inputPath), err))
|
||||
}
|
||||
checkpointRecorder, checkpointLoader, err := checkpointHandlersForRun(workspaceSettings, effective.ResolvedPipeline, rawInput, only, llmProfiles, strings.TrimSpace(*llmProfile), strings.TrimSpace(sessionID.value), *resume)
|
||||
if err != nil {
|
||||
return failPipelineCommand(stderr, runDir, cfg.Diagnostics.Retention, err)
|
||||
}
|
||||
|
||||
output, err := pipeline.New(registries).Run(ctx, pipeline.RunInput{
|
||||
Pipeline: effective.ResolvedPipeline,
|
||||
Path: strings.TrimSpace(*inputPath),
|
||||
RawInput: rawInput,
|
||||
LLMClient: llmClient,
|
||||
SessionID: strings.TrimSpace(sessionID.value),
|
||||
RunID: runID,
|
||||
StartedAt: startedAt,
|
||||
LLMProfiles: llmProfiles,
|
||||
Metadata: runMetadata(*outputDir, *diagnosticsDir),
|
||||
Warnings: referenceWarnings,
|
||||
Checkpoints: checkpointRecorder,
|
||||
Checkpoint: checkpointLoader,
|
||||
Debug: debugRecorder,
|
||||
output, err := pipeline.New().Run(ctx, pipeline.RunInput{
|
||||
Prepared: prepared,
|
||||
Path: strings.TrimSpace(*inputPath),
|
||||
RawInput: rawInput,
|
||||
SessionID: strings.TrimSpace(sessionID.value),
|
||||
RunID: runID,
|
||||
StartedAt: startedAt,
|
||||
LLMProfiles: llmProfiles,
|
||||
Metadata: runMetadata(*outputDir, *diagnosticsDir),
|
||||
Warnings: referenceWarnings,
|
||||
Checkpoints: checkpointRecorder,
|
||||
Checkpoint: checkpointLoader,
|
||||
Debug: debugRecorder,
|
||||
ExtractWorkers: cfg.Concurrency.StageWorkers["extract"],
|
||||
})
|
||||
if err != nil {
|
||||
if output.Manifest.PipelineID != "" && runDir != nil {
|
||||
|
||||
@@ -22,20 +22,20 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/dnd/scenes"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/extract/dnd/spells"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
|
||||
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
|
||||
spellshape "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/shape"
|
||||
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_refs"
|
||||
spellrelatedness "gitea.maximumdirect.net/eric/notarius/internal/validators/extract/dnd/spells/source_relatedness"
|
||||
alwaysaccept "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_accept"
|
||||
alwaysreject "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/always_reject"
|
||||
validjson "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json"
|
||||
validjsonschema "gitea.maximumdirect.net/eric/notarius/internal/validators/generic/valid_json_schema"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
|
||||
spellshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/shape"
|
||||
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/source_refs"
|
||||
spellrelatedness "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/source_relatedness"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/chunk/units"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/merge/appendorder"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
|
||||
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/output/json"
|
||||
alwaysaccept "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/always_accept"
|
||||
alwaysreject "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/always_reject"
|
||||
validjson "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/valid_json"
|
||||
validjsonschema "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/validate/valid_json_schema"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/input/transcript"
|
||||
"gitea.maximumdirect.net/eric/scriptorium"
|
||||
)
|
||||
|
||||
@@ -135,6 +135,12 @@ func TestProductionCatalogIncludesProductionModulesValidatorsAndDefaults(t *test
|
||||
if err != nil {
|
||||
t.Fatalf("productionCatalog() error = %v, want nil", err)
|
||||
}
|
||||
if catalog.ArtifactCodecs == nil {
|
||||
t.Fatal("production artifact codec registry = nil, want initialized empty registry")
|
||||
}
|
||||
if got := catalog.ArtifactCodecs.RegisteredKinds(); !reflect.DeepEqual(got, []contracts.ArtifactKind{"dnd/spell-list"}) {
|
||||
t.Fatalf("production artifact codec kinds = %#v, want dnd/spell-list", got)
|
||||
}
|
||||
|
||||
moduleTests := []struct {
|
||||
name string
|
||||
@@ -143,13 +149,13 @@ func TestProductionCatalogIncludesProductionModulesValidatorsAndDefaults(t *test
|
||||
}{
|
||||
{
|
||||
name: "seriatim input",
|
||||
got: func() (pipeline.ModuleSpec, bool) { return catalog.Inputs.Spec(seriatim.Key) },
|
||||
want: seriatim.ModuleSpec(),
|
||||
got: func() (pipeline.ModuleSpec, bool) { return catalog.Inputs.Spec(transcript.Key) },
|
||||
want: transcript.ModuleSpec(),
|
||||
},
|
||||
{
|
||||
name: "generic chunker",
|
||||
got: func() (pipeline.ModuleSpec, bool) { return catalog.Chunkers.Spec(generic.Key) },
|
||||
want: generic.ModuleSpec(),
|
||||
got: func() (pipeline.ModuleSpec, bool) { return catalog.Chunkers.Spec(units.Key) },
|
||||
want: units.ModuleSpec(),
|
||||
},
|
||||
{
|
||||
name: "dnd scenes chunker",
|
||||
@@ -164,12 +170,12 @@ func TestProductionCatalogIncludesProductionModulesValidatorsAndDefaults(t *test
|
||||
{
|
||||
name: "appendorder merger",
|
||||
got: func() (pipeline.ModuleSpec, bool) { return catalog.Mergers.Spec(appendorder.Key) },
|
||||
want: appendorder.ModuleSpec(),
|
||||
want: appendorder.TypedModuleSpec(spells.ModuleSpec().ArtifactKind),
|
||||
},
|
||||
{
|
||||
name: "noop normalizer",
|
||||
got: func() (pipeline.ModuleSpec, bool) { return catalog.Normalizers.Spec(noop.Key) },
|
||||
want: noop.ModuleSpec(),
|
||||
want: noop.TypedModuleSpec(spells.ModuleSpec().ArtifactKind),
|
||||
},
|
||||
{
|
||||
name: "json output",
|
||||
@@ -222,11 +228,28 @@ func TestProductionCatalogIncludesProductionModulesValidatorsAndDefaults(t *test
|
||||
if !reflect.DeepEqual(gotChain, wantChain) {
|
||||
t.Fatalf("dnd spell default validator chain = %#v, want %#v", gotChain, wantChain)
|
||||
}
|
||||
if got := catalog.ValidatorChains.Validators(pipeline.StageChunk, generic.Key); len(got) != 0 {
|
||||
if got := catalog.ValidatorChains.Validators(pipeline.StageChunk, units.Key); len(got) != 0 {
|
||||
t.Fatalf("generic chunker default validator chain = %#v, want empty", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCatalogConversionsPreserveArtifactCodecRegistry(t *testing.T) {
|
||||
codecs := pipeline.NewArtifactCodecRegistry()
|
||||
registries := pipeline.Registries{ArtifactCodecs: codecs}
|
||||
if isEmptyRegistries(registries) {
|
||||
t.Fatal("registries with artifact codecs reported empty")
|
||||
}
|
||||
|
||||
catalog := catalogFromRegistries(registries)
|
||||
if catalog.ArtifactCodecs != codecs || isEmptyCatalog(catalog) {
|
||||
t.Fatalf("catalog artifact codecs = %p empty=%t, want %p and non-empty", catalog.ArtifactCodecs, isEmptyCatalog(catalog), codecs)
|
||||
}
|
||||
converted := registriesFromCatalog(catalog)
|
||||
if converted.ArtifactCodecs != codecs {
|
||||
t.Fatalf("converted artifact codecs = %p, want %p", converted.ArtifactCodecs, codecs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProductionPromptAssetsRegisterAndPrepareDndPrompts(t *testing.T) {
|
||||
registry, err := productionPromptAssets()
|
||||
if err != nil {
|
||||
@@ -3533,6 +3556,10 @@ func fakeExecutionRegistries(t *testing.T) pipeline.Registries {
|
||||
mergers := pipeline.NewMergerRegistry()
|
||||
normalizers := pipeline.NewNormalizerRegistry()
|
||||
outputs := pipeline.NewOutputEncoderRegistry()
|
||||
codecs := pipeline.NewArtifactCodecRegistry()
|
||||
if err := pipeline.RegisterArtifactCodec(codecs, fakeRunCodec{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if err := inputs.RegisterWithSpec(pipeline.ModuleSpec{Key: "fake/input", Stage: pipeline.StageInput, Provides: []string{"source"}}, func() (contracts.InputAdapter, error) {
|
||||
return fakeRunInputAdapter{}, nil
|
||||
@@ -3544,25 +3571,25 @@ func fakeExecutionRegistries(t *testing.T) pipeline.Registries {
|
||||
}); err != nil {
|
||||
t.Fatalf("register fake chunker: %v", err)
|
||||
}
|
||||
if err := extractors.RegisterWithSpec(pipeline.ModuleSpec{
|
||||
if err := pipeline.RegisterExtractor(extractors, pipeline.ModuleSpec{
|
||||
Key: "fake/extract",
|
||||
Stage: pipeline.StageExtract,
|
||||
Requires: []string{"chunks"},
|
||||
Provides: []string{"artifact"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{Name: "roster"},
|
||||
},
|
||||
}, func() (contracts.Extractor, error) {
|
||||
}, ArtifactKind: fakeRunArtifactKind,
|
||||
}, func() (contracts.Extractor[fakeRunArtifact], error) {
|
||||
return fakeRunExtractor{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register fake extractor: %v", err)
|
||||
}
|
||||
if err := mergers.RegisterWithSpec(pipeline.ModuleSpec{Key: "appendorder", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}}, func() (contracts.Merger, error) {
|
||||
if err := pipeline.RegisterMerger(mergers, pipeline.ModuleSpec{Key: "appendorder", Stage: pipeline.StageMerge, Requires: []string{"artifact"}, Provides: []string{"merged"}, ArtifactKind: fakeRunArtifactKind}, func() (contracts.Merger[fakeRunArtifact], error) {
|
||||
return fakeRunMerger{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register fake merger: %v", err)
|
||||
}
|
||||
if err := normalizers.RegisterWithSpec(pipeline.ModuleSpec{Key: "noop", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}}, func() (contracts.Normalizer, error) {
|
||||
if err := pipeline.RegisterNormalizer(normalizers, pipeline.ModuleSpec{Key: "noop", Stage: pipeline.StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: fakeRunArtifactKind}, func() (contracts.Normalizer[fakeRunArtifact], error) {
|
||||
return fakeRunNormalizer{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register fake normalizer: %v", err)
|
||||
@@ -3572,12 +3599,13 @@ func fakeExecutionRegistries(t *testing.T) pipeline.Registries {
|
||||
}
|
||||
|
||||
return pipeline.Registries{
|
||||
Inputs: inputs,
|
||||
Chunkers: chunkers,
|
||||
Extractors: extractors,
|
||||
Mergers: mergers,
|
||||
Normalizers: normalizers,
|
||||
Outputs: outputs,
|
||||
Inputs: inputs,
|
||||
Chunkers: chunkers,
|
||||
ArtifactCodecs: codecs,
|
||||
Extractors: extractors,
|
||||
Mergers: mergers,
|
||||
Normalizers: normalizers,
|
||||
Outputs: outputs,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3594,7 +3622,7 @@ func (fakeRunInputAdapter) Parse(ctx context.Context, req contracts.ParseRequest
|
||||
Format: "test",
|
||||
Digest: "sha256:source",
|
||||
Units: []source.SourceUnit{
|
||||
{ID: 1, Kind: "text", Text: string(req.Raw)},
|
||||
{ID: 1, Kind: "text", Text: string(req.Raw), Ref: source.SourceRef{SourceID: "source", StartUnitID: 1, EndUnitID: 1}},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
@@ -3611,14 +3639,33 @@ func (fakeRunChunker) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
|
||||
func (fakeRunChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
|
||||
return contracts.ChunkResult{
|
||||
Chunks: []contracts.SourceChunk{
|
||||
{ID: "chunk-1", SourceID: req.Source.ID, Index: 0, StartUnitID: 1, EndUnitID: 1, Content: []byte(`{"units":[1]}`), MediaType: "application/json", Units: req.Source.Units},
|
||||
Chunks: []source.Chunk{
|
||||
{ID: "chunk-1", SourceID: req.Source.ID, Index: 0, Ref: source.SourceRef{SourceID: req.Source.ID, StartUnitID: 1, EndUnitID: 1}, Content: []byte(`{"units":[1]}`), MediaType: "application/json", Units: req.Source.Units},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type fakeRunExtractor struct{}
|
||||
|
||||
const fakeRunArtifactKind contracts.ArtifactKind = "test/fake"
|
||||
|
||||
type fakeRunArtifact struct {
|
||||
Value bool `json:"value"`
|
||||
}
|
||||
type fakeRunCodec struct{}
|
||||
|
||||
func (fakeRunCodec) Kind() contracts.ArtifactKind { return fakeRunArtifactKind }
|
||||
func (fakeRunCodec) Schema() contracts.ArtifactSchema {
|
||||
return contracts.ArtifactSchema{ID: "fake.artifact", Name: "fake_artifact", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
|
||||
}
|
||||
func (fakeRunCodec) MediaType() string { return "application/json" }
|
||||
func (fakeRunCodec) Encode(v fakeRunArtifact) ([]byte, error) { return json.Marshal(v) }
|
||||
func (fakeRunCodec) Decode(b []byte) (fakeRunArtifact, error) {
|
||||
var v fakeRunArtifact
|
||||
err := json.Unmarshal(b, &v)
|
||||
return v, err
|
||||
}
|
||||
|
||||
func (fakeRunExtractor) Key() string {
|
||||
return "fake/extract"
|
||||
}
|
||||
@@ -3627,16 +3674,8 @@ func (fakeRunExtractor) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return []contracts.ReferenceSlot{{Name: "roster"}}
|
||||
}
|
||||
|
||||
func (fakeRunExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
|
||||
return contracts.ExtractionResult{
|
||||
Output: contracts.ExtractOutput{
|
||||
Schema: contracts.ResponseSchema{ID: "fake.artifact", Name: "fake_artifact", Version: "v1"},
|
||||
Payload: contracts.RawPayload{
|
||||
Content: []byte(`{"value":true}`),
|
||||
MediaType: "application/json",
|
||||
},
|
||||
},
|
||||
}, nil
|
||||
func (fakeRunExtractor) Extract(ctx context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[fakeRunArtifact], error) {
|
||||
return contracts.TypedExtractionResult[fakeRunArtifact]{Value: fakeRunArtifact{Value: true}}, nil
|
||||
}
|
||||
|
||||
type fakeRunMerger struct{}
|
||||
@@ -3645,21 +3684,11 @@ func (fakeRunMerger) Key() string {
|
||||
return "appendorder"
|
||||
}
|
||||
|
||||
func (fakeRunMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
|
||||
output := contracts.MergeOutput{
|
||||
LaneID: req.LaneID,
|
||||
SourceID: req.Source.ID,
|
||||
Schema: contracts.ResponseSchema{ID: "fake.artifact", Name: "fake_artifact", Version: "v1"},
|
||||
Payload: contracts.RawPayload{
|
||||
Content: []byte(`{"merged":true}`),
|
||||
MediaType: "application/json",
|
||||
},
|
||||
}
|
||||
func (fakeRunMerger) Merge(ctx context.Context, req contracts.TypedMergeRequest[fakeRunArtifact]) (contracts.TypedMergeResult[fakeRunArtifact], error) {
|
||||
if len(req.ExtractOutputs) > 0 {
|
||||
output.Schema = req.ExtractOutputs[0].Schema
|
||||
output.Payload = req.ExtractOutputs[0].Payload
|
||||
return contracts.TypedMergeResult[fakeRunArtifact]{Value: req.ExtractOutputs[0].Value}, nil
|
||||
}
|
||||
return contracts.MergeResult{Output: output}, nil
|
||||
return contracts.TypedMergeResult[fakeRunArtifact]{}, nil
|
||||
}
|
||||
|
||||
type fakeRunNormalizer struct{}
|
||||
@@ -3672,15 +3701,8 @@ func (fakeRunNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (fakeRunNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
|
||||
return contracts.NormalizeResult{
|
||||
Output: contracts.NormalizeOutput{
|
||||
LaneID: req.LaneID,
|
||||
SourceID: req.MergeOutput.SourceID,
|
||||
Schema: req.MergeOutput.Schema,
|
||||
Payload: req.MergeOutput.Payload,
|
||||
},
|
||||
}, nil
|
||||
func (fakeRunNormalizer) Normalize(ctx context.Context, req contracts.TypedNormalizeRequest[fakeRunArtifact]) (contracts.TypedNormalizeResult[fakeRunArtifact], error) {
|
||||
return contracts.TypedNormalizeResult[fakeRunArtifact]{Value: req.MergeOutput.Value}, nil
|
||||
}
|
||||
|
||||
func onlyChildDir(t *testing.T, root string) string {
|
||||
@@ -3942,9 +3964,18 @@ func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.Module
|
||||
for _, override := range overrides {
|
||||
specs[override.Key] = override
|
||||
}
|
||||
for _, key := range []string{"fake/extract", "appendorder", "noop"} {
|
||||
spec := specs[key]
|
||||
spec.ArtifactKind = fakeRunArtifactKind
|
||||
specs[key] = spec
|
||||
}
|
||||
|
||||
mustRegisterInput(t, inputs, specs["fake/input"])
|
||||
mustRegisterChunker(t, chunkers, specs["generic"])
|
||||
codecs := pipeline.NewArtifactCodecRegistry()
|
||||
if err := pipeline.RegisterArtifactCodec(codecs, fakeRunCodec{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mustRegisterExtractor(t, extractors, specs["fake/extract"])
|
||||
mustRegisterMerger(t, mergers, specs["appendorder"])
|
||||
mustRegisterNormalizer(t, normalizers, specs["noop"])
|
||||
@@ -3953,6 +3984,7 @@ func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.Module
|
||||
return pipeline.ModuleCatalog{
|
||||
Inputs: inputs,
|
||||
Chunkers: chunkers,
|
||||
ArtifactCodecs: codecs,
|
||||
Extractors: extractors,
|
||||
Mergers: mergers,
|
||||
Normalizers: normalizers,
|
||||
@@ -3964,53 +3996,58 @@ func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.Module
|
||||
|
||||
func mustRegisterInput(t *testing.T, registry *pipeline.InputAdapterRegistry, spec pipeline.ModuleSpec) {
|
||||
t.Helper()
|
||||
if err := registry.RegisterWithSpec(spec, func() (contracts.InputAdapter, error) { return nil, nil }); err != nil {
|
||||
if err := registry.RegisterWithSpec(spec, func() (contracts.InputAdapter, error) { return fakeRunInputAdapter{}, nil }); err != nil {
|
||||
t.Fatalf("register input: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustRegisterChunker(t *testing.T, registry *pipeline.ChunkerRegistry, spec pipeline.ModuleSpec) {
|
||||
t.Helper()
|
||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Chunker, error) { return nil, nil }); err != nil {
|
||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Chunker, error) { return fakeRunChunker{}, nil }); err != nil {
|
||||
t.Fatalf("register chunker: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustRegisterExtractor(t *testing.T, registry *pipeline.ExtractorRegistry, spec pipeline.ModuleSpec) {
|
||||
t.Helper()
|
||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Extractor, error) { return nil, nil }); err != nil {
|
||||
if err := pipeline.RegisterExtractor(registry, spec, func() (contracts.Extractor[fakeRunArtifact], error) { return fakeRunExtractor{}, nil }); err != nil {
|
||||
t.Fatalf("register extractor: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustRegisterMerger(t *testing.T, registry *pipeline.MergerRegistry, spec pipeline.ModuleSpec) {
|
||||
t.Helper()
|
||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Merger, error) { return nil, nil }); err != nil {
|
||||
if err := pipeline.RegisterMerger(registry, spec, func() (contracts.Merger[fakeRunArtifact], error) { return fakeRunMerger{}, nil }); err != nil {
|
||||
t.Fatalf("register merger: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustRegisterNormalizer(t *testing.T, registry *pipeline.NormalizerRegistry, spec pipeline.ModuleSpec) {
|
||||
t.Helper()
|
||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Normalizer, error) { return nil, nil }); err != nil {
|
||||
if err := pipeline.RegisterNormalizer(registry, spec, func() (contracts.Normalizer[fakeRunArtifact], error) { return fakeRunNormalizer{}, nil }); err != nil {
|
||||
t.Fatalf("register normalizer: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustRegisterOutput(t *testing.T, registry *pipeline.OutputEncoderRegistry, spec pipeline.ModuleSpec) {
|
||||
t.Helper()
|
||||
if err := registry.RegisterWithSpec(spec, func() (contracts.OutputEncoder, error) { return nil, nil }); err != nil {
|
||||
if err := registry.RegisterWithSpec(spec, func() (contracts.OutputEncoder, error) { return jsonoutput.New(), nil }); err != nil {
|
||||
t.Fatalf("register output: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustRegisterValidator(t *testing.T, registry *pipeline.ValidatorRegistry, spec pipeline.ValidatorSpec) {
|
||||
t.Helper()
|
||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Validator, error) {
|
||||
if err := pipeline.RegisterChunkValidator(registry, spec, func() (contracts.ChunkValidator, error) {
|
||||
return fakeConfigValidator{name: spec.Key, executionClass: spec.ExecutionClass}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register validator: %v", err)
|
||||
}
|
||||
if err := pipeline.RegisterTypedValidator[fakeRunArtifact](registry, fakeRunArtifactKind, spec, func() (contracts.TypedValidator[fakeRunArtifact], error) {
|
||||
return fakeConfigTypedValidator{fakeConfigValidator{name: spec.Key, executionClass: spec.ExecutionClass}}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register typed validator: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
type fakeConfigValidator struct {
|
||||
@@ -4026,7 +4063,13 @@ func (validator fakeConfigValidator) ExecutionClass() contracts.ExecutionClass {
|
||||
return validator.executionClass
|
||||
}
|
||||
|
||||
func (validator fakeConfigValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
|
||||
func (validator fakeConfigValidator) Validate(ctx context.Context, req contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}
|
||||
|
||||
type fakeConfigTypedValidator struct{ fakeConfigValidator }
|
||||
|
||||
func (validator fakeConfigTypedValidator) Validate(ctx context.Context, req contracts.TypedValidationRequest[fakeRunArtifact]) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -24,7 +24,11 @@ type ScriptoriumConfig struct {
|
||||
}
|
||||
|
||||
type ConcurrencyConfig struct {
|
||||
TotalLLM int `json:"total_llm"`
|
||||
TotalLLM int `json:"total_llm"`
|
||||
StageWorkers map[string]int `json:"stage_workers"`
|
||||
|
||||
extractWorkersConfigured bool
|
||||
defaultedExtractWorkers int
|
||||
}
|
||||
|
||||
type DiagnosticsConfig struct {
|
||||
@@ -58,7 +62,9 @@ func Default() Config {
|
||||
return Config{
|
||||
Pipelines: map[string]pipeline.PipelineProfile{},
|
||||
Concurrency: ConcurrencyConfig{
|
||||
TotalLLM: 1,
|
||||
TotalLLM: 1,
|
||||
StageWorkers: map[string]int{"extract": 1},
|
||||
defaultedExtractWorkers: 1,
|
||||
},
|
||||
Diagnostics: DiagnosticsConfig{
|
||||
WorkDir: "/tmp/notarius",
|
||||
@@ -101,6 +107,7 @@ func (c Config) workspaceDirectory() string {
|
||||
|
||||
func cloneConfig(in Config) Config {
|
||||
out := in
|
||||
out.Concurrency.StageWorkers = cloneIntMap(in.Concurrency.StageWorkers)
|
||||
out.Pipelines = make(map[string]pipeline.PipelineProfile, len(in.Pipelines))
|
||||
for key, profile := range in.Pipelines {
|
||||
out.Pipelines[key] = clonePipelineProfile(profile)
|
||||
@@ -108,6 +115,34 @@ func cloneConfig(in Config) Config {
|
||||
return out
|
||||
}
|
||||
|
||||
func cloneIntMap(in map[string]int) map[string]int {
|
||||
if len(in) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]int, len(in))
|
||||
for key, value := range in {
|
||||
out[key] = value
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (c *ConcurrencyConfig) recomputeStageWorkerDefaults() {
|
||||
if c == nil {
|
||||
return
|
||||
}
|
||||
if c.StageWorkers == nil {
|
||||
c.StageWorkers = make(map[string]int)
|
||||
}
|
||||
if !c.extractWorkersConfigured {
|
||||
if value, ok := c.StageWorkers["extract"]; ok && (c.defaultedExtractWorkers == 0 || value != c.defaultedExtractWorkers) {
|
||||
c.extractWorkersConfigured = true
|
||||
return
|
||||
}
|
||||
c.StageWorkers["extract"] = c.TotalLLM
|
||||
c.defaultedExtractWorkers = c.TotalLLM
|
||||
}
|
||||
}
|
||||
|
||||
func clonePipelineProfile(in pipeline.PipelineProfile) pipeline.PipelineProfile {
|
||||
out := in
|
||||
out.Input = cloneModuleBinding(in.Input)
|
||||
|
||||
@@ -18,6 +18,9 @@ func TestDefaultValues(t *testing.T) {
|
||||
if cfg.Concurrency.TotalLLM != 1 {
|
||||
t.Fatalf("unexpected total LLM concurrency: %d", cfg.Concurrency.TotalLLM)
|
||||
}
|
||||
if got := cfg.Concurrency.StageWorkers["extract"]; got != 1 {
|
||||
t.Fatalf("unexpected extract workers: %d", got)
|
||||
}
|
||||
if cfg.Diagnostics.WorkDir != "/tmp/notarius" {
|
||||
t.Fatalf("unexpected diagnostics work dir: %q", cfg.Diagnostics.WorkDir)
|
||||
}
|
||||
@@ -68,6 +71,9 @@ pipelines:
|
||||
if cfg.Concurrency.TotalLLM != 1 {
|
||||
t.Fatalf("expected default concurrency preserved, got %d", cfg.Concurrency.TotalLLM)
|
||||
}
|
||||
if got := cfg.Concurrency.StageWorkers["extract"]; got != 1 {
|
||||
t.Fatalf("expected default extract workers preserved, got %d", got)
|
||||
}
|
||||
if cfg.Diagnostics.Retention != diagnostics.RetentionAuto {
|
||||
t.Fatalf("expected default diagnostics retention preserved, got %q", cfg.Diagnostics.Retention)
|
||||
}
|
||||
|
||||
@@ -26,6 +26,7 @@ type EffectiveConfig struct {
|
||||
}
|
||||
|
||||
func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
|
||||
c.Concurrency.recomputeStageWorkerDefaults()
|
||||
if err := c.Validate(); err != nil {
|
||||
return EffectiveConfig{}, err
|
||||
}
|
||||
|
||||
@@ -27,6 +27,19 @@ func TestResolveRejectsEmptyAndUnknownPipelineID(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveMaterializesDefaultExtractWorkersFromEffectiveTotal(t *testing.T) {
|
||||
cfg := validConfig()
|
||||
cfg.Concurrency.TotalLLM = 4
|
||||
|
||||
effective, err := cfg.Resolve(ResolveInput{PipelineID: "example", Catalog: fakeCatalog(t)})
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve() error = %v, want nil", err)
|
||||
}
|
||||
if got := effective.Config.Concurrency.StageWorkers["extract"]; got != 4 {
|
||||
t.Fatalf("effective extract workers = %d, want total concurrency 4", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveLaneFilteringSuccessAndFailure(t *testing.T) {
|
||||
effective, err := validConfig().Resolve(ResolveInput{
|
||||
PipelineID: " example ",
|
||||
|
||||
@@ -36,6 +36,18 @@ func (c *Config) applyEnvOverridesWithLookup(lookup func(string) (string, bool))
|
||||
}
|
||||
c.Concurrency.TotalLLM = value
|
||||
}
|
||||
if raw, ok := lookup("NOTARIUS_STAGE_WORKERS_EXTRACT"); ok {
|
||||
value, err := parseIntEnv("NOTARIUS_STAGE_WORKERS_EXTRACT", raw)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if c.Concurrency.StageWorkers == nil {
|
||||
c.Concurrency.StageWorkers = make(map[string]int)
|
||||
}
|
||||
c.Concurrency.StageWorkers["extract"] = value
|
||||
c.Concurrency.extractWorkersConfigured = true
|
||||
}
|
||||
c.Concurrency.recomputeStageWorkerDefaults()
|
||||
if raw, ok := lookup("NOTARIUS_WORK_DIR"); ok {
|
||||
c.Diagnostics.WorkDir = strings.TrimSpace(raw)
|
||||
}
|
||||
|
||||
@@ -14,6 +14,7 @@ func TestApplyEnvOverridesOperationalValues(t *testing.T) {
|
||||
|
||||
err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{
|
||||
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "3",
|
||||
"NOTARIUS_STAGE_WORKERS_EXTRACT": "2",
|
||||
"NOTARIUS_WORK_DIR": "/tmp/notarius-env",
|
||||
"NOTARIUS_DIAGNOSTICS_RETENTION": "never",
|
||||
"NOTARIUS_WORKSPACE_DIR": "/var/lib/notarius-env",
|
||||
@@ -33,6 +34,9 @@ func TestApplyEnvOverridesOperationalValues(t *testing.T) {
|
||||
if cfg.Concurrency.TotalLLM != 3 {
|
||||
t.Fatalf("unexpected total concurrency: %d", cfg.Concurrency.TotalLLM)
|
||||
}
|
||||
if got := cfg.Concurrency.StageWorkers["extract"]; got != 2 {
|
||||
t.Fatalf("extract workers = %d, want 2", got)
|
||||
}
|
||||
if cfg.Workspace.Directory != "/var/lib/notarius-env" {
|
||||
t.Fatalf("unexpected workspace directory: %q", cfg.Workspace.Directory)
|
||||
}
|
||||
@@ -54,12 +58,74 @@ func TestApplyEnvOverridesOperationalValues(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestApplyEnvOverridesRejectsInvalidIntegers(t *testing.T) {
|
||||
for _, name := range []string{"NOTARIUS_TOTAL_LLM_CONCURRENCY", "NOTARIUS_STAGE_WORKERS_EXTRACT"} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
cfg := Default()
|
||||
err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{name: "many"}))
|
||||
if err == nil || !strings.Contains(err.Error(), name) {
|
||||
t.Fatalf("expected named integer error, got %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestStageWorkerEnvironmentPrecedenceAndDefaulting(t *testing.T) {
|
||||
fileCfg, err := ParseFileConfigYAML([]byte(`
|
||||
version: 2
|
||||
concurrency:
|
||||
total_llm: 4
|
||||
stage_workers:
|
||||
extract: 2
|
||||
`))
|
||||
if err != nil {
|
||||
t.Fatalf("ParseFileConfigYAML() error = %v", err)
|
||||
}
|
||||
cfg := Default()
|
||||
err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{
|
||||
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "many",
|
||||
}))
|
||||
if err == nil || !strings.Contains(err.Error(), "NOTARIUS_TOTAL_LLM_CONCURRENCY") {
|
||||
t.Fatalf("expected named integer error, got %v", err)
|
||||
if err := cfg.applyFileConfigWithLookup(fileCfg, emptyLookup); err != nil {
|
||||
t.Fatalf("ApplyFileConfig() error = %v", err)
|
||||
}
|
||||
if err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{
|
||||
"NOTARIUS_TOTAL_LLM_CONCURRENCY": "5",
|
||||
"NOTARIUS_STAGE_WORKERS_EXTRACT": "3",
|
||||
})); err != nil {
|
||||
t.Fatalf("ApplyEnvOverrides() error = %v", err)
|
||||
}
|
||||
if cfg.Concurrency.TotalLLM != 5 || cfg.Concurrency.StageWorkers["extract"] != 3 {
|
||||
t.Fatalf("effective concurrency = %#v, want total 5 and extract 3", cfg.Concurrency)
|
||||
}
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Fatalf("Validate(overridden) error = %v, want nil", err)
|
||||
}
|
||||
|
||||
defaulted := Default()
|
||||
if err := defaulted.applyEnvOverridesWithLookup(mapLookup(map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6"})); err != nil {
|
||||
t.Fatalf("ApplyEnvOverrides(defaulted) error = %v", err)
|
||||
}
|
||||
if got := defaulted.Concurrency.StageWorkers["extract"]; got != 6 {
|
||||
t.Fatalf("defaulted extract workers = %d, want effective total 6", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStageWorkerRangeValidationUsesFinalEnvironmentTotal(t *testing.T) {
|
||||
fileCfg, err := ParseFileConfigYAML([]byte(`
|
||||
version: 2
|
||||
concurrency:
|
||||
total_llm: 4
|
||||
stage_workers:
|
||||
extract: 5
|
||||
`))
|
||||
if err != nil {
|
||||
t.Fatalf("ParseFileConfigYAML() error = %v", err)
|
||||
}
|
||||
cfg := Default()
|
||||
if err := cfg.applyFileConfigWithLookup(fileCfg, emptyLookup); err != nil {
|
||||
t.Fatalf("ApplyFileConfig() error = %v", err)
|
||||
}
|
||||
if err := cfg.applyEnvOverridesWithLookup(mapLookup(map[string]string{"NOTARIUS_TOTAL_LLM_CONCURRENCY": "6"})); err != nil {
|
||||
t.Fatalf("ApplyEnvOverrides() error = %v", err)
|
||||
}
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Fatalf("Validate() error = %v, want final total to make extract workers valid", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -111,4 +177,7 @@ func TestLoadFromEnvUsesDefaultConfig(t *testing.T) {
|
||||
if cfg.Concurrency.TotalLLM != 2 {
|
||||
t.Fatalf("expected env concurrency override, got %+v", cfg.Concurrency)
|
||||
}
|
||||
if got := cfg.Concurrency.StageWorkers["extract"]; got != 2 {
|
||||
t.Fatalf("expected extract workers to default to total, got %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -43,7 +43,8 @@ type FileArtifactLaneProfile struct {
|
||||
}
|
||||
|
||||
type FileConcurrencyConfig struct {
|
||||
TotalLLM *int `yaml:"total_llm,omitempty"`
|
||||
TotalLLM *int `yaml:"total_llm,omitempty"`
|
||||
StageWorkers map[string]int `yaml:"stage_workers,omitempty"`
|
||||
}
|
||||
|
||||
type FileDiagnosticsConfig struct {
|
||||
@@ -316,6 +317,15 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
||||
if fileCfg.Concurrency != nil && fileCfg.Concurrency.TotalLLM != nil {
|
||||
c.Concurrency.TotalLLM = *fileCfg.Concurrency.TotalLLM
|
||||
}
|
||||
if fileCfg.Concurrency != nil && fileCfg.Concurrency.StageWorkers != nil {
|
||||
workers, configured, err := normalizeStageWorkers(fileCfg.Concurrency.StageWorkers)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
c.Concurrency.StageWorkers = workers
|
||||
c.Concurrency.extractWorkersConfigured = configured
|
||||
}
|
||||
c.Concurrency.recomputeStageWorkerDefaults()
|
||||
if fileCfg.Diagnostics != nil {
|
||||
if fileCfg.Diagnostics.WorkDir != nil {
|
||||
c.Diagnostics.WorkDir = strings.TrimSpace(*fileCfg.Diagnostics.WorkDir)
|
||||
@@ -350,6 +360,26 @@ func (c *Config) applyFileConfigWithLookup(fileCfg FileConfig, lookup func(strin
|
||||
return nil
|
||||
}
|
||||
|
||||
func normalizeStageWorkers(values map[string]int) (map[string]int, bool, error) {
|
||||
workers := make(map[string]int, len(values))
|
||||
configured := false
|
||||
for rawKey, value := range values {
|
||||
key := strings.TrimSpace(rawKey)
|
||||
if key == "" {
|
||||
return nil, false, fmt.Errorf("concurrency.stage_workers key must not be empty")
|
||||
}
|
||||
if key != "extract" {
|
||||
return nil, false, fmt.Errorf("concurrency.stage_workers key %q is not supported", rawKey)
|
||||
}
|
||||
if _, exists := workers[key]; exists {
|
||||
return nil, false, fmt.Errorf("concurrency.stage_workers key %q is duplicated after trimming", key)
|
||||
}
|
||||
workers[key] = value
|
||||
configured = true
|
||||
}
|
||||
return workers, configured, nil
|
||||
}
|
||||
|
||||
func normalizedMapKeys[T any](values map[string]T, keyName string) ([]string, map[string]string, error) {
|
||||
keys := make([]string, 0, len(values))
|
||||
rawByNormalized := make(map[string]string, len(values))
|
||||
|
||||
@@ -250,7 +250,7 @@ pipelines:
|
||||
}
|
||||
lane := profile.Artifacts["events"]
|
||||
if !reflect.DeepEqual(lane.References, map[string]string{"lore": "./lore.md", "roster": "./legacy-roster.yml"}) {
|
||||
t.Fatalf("legacy lane references = %#v, want trimmed map", lane.References)
|
||||
t.Fatalf("lane references = %#v, want trimmed map", lane.References)
|
||||
}
|
||||
wantExtract := map[string]string{
|
||||
"glossary": "./glossary.md",
|
||||
@@ -546,6 +546,9 @@ diagnostics:
|
||||
if cfg.Concurrency.TotalLLM != 4 {
|
||||
t.Fatalf("unexpected total concurrency: %d", cfg.Concurrency.TotalLLM)
|
||||
}
|
||||
if got := cfg.Concurrency.StageWorkers["extract"]; got != 4 {
|
||||
t.Fatalf("default extract workers = %d, want total concurrency", got)
|
||||
}
|
||||
if cfg.Diagnostics.WorkDir != "/tmp/notarius-test" {
|
||||
t.Fatalf("unexpected work dir: %q", cfg.Diagnostics.WorkDir)
|
||||
}
|
||||
@@ -554,6 +557,58 @@ diagnostics:
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyFileConfigStageWorkers(t *testing.T) {
|
||||
cfg := parseAndApplyConfig(t, `
|
||||
version: 2
|
||||
concurrency:
|
||||
total_llm: 4
|
||||
stage_workers:
|
||||
extract: 3
|
||||
`)
|
||||
|
||||
if got := cfg.Concurrency.StageWorkers["extract"]; got != 3 {
|
||||
t.Fatalf("extract workers = %d, want 3", got)
|
||||
}
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Fatalf("Validate() error = %v, want nil", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyFileConfigEmptyStageWorkersDefaultsExtractToTotal(t *testing.T) {
|
||||
cfg := parseAndApplyConfig(t, `
|
||||
version: 2
|
||||
concurrency:
|
||||
total_llm: 4
|
||||
stage_workers: {}
|
||||
`)
|
||||
if got := cfg.Concurrency.StageWorkers["extract"]; got != 4 {
|
||||
t.Fatalf("extract workers = %d, want total concurrency 4", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyFileConfigRejectsUnsupportedStageWorkerKeys(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
key string
|
||||
want string
|
||||
}{
|
||||
{name: "empty", key: "' '", want: "must not be empty"},
|
||||
{name: "unknown", key: "merge", want: "not supported"},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
fileCfg, err := ParseFileConfigYAML([]byte("version: 2\nconcurrency:\n stage_workers:\n " + test.key + ": 1\n"))
|
||||
if err != nil {
|
||||
t.Fatalf("ParseFileConfigYAML() error = %v", err)
|
||||
}
|
||||
cfg := Default()
|
||||
err = cfg.applyFileConfigWithLookup(fileCfg, emptyLookup)
|
||||
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("ApplyFileConfig() error = %v, want %q", err, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyFileConfigWorkspaceSection(t *testing.T) {
|
||||
cfg := parseAndApplyConfig(t, `
|
||||
version: 2
|
||||
|
||||
@@ -50,6 +50,8 @@ func cloneResolvedValidatorChain(in pipeline.ResolvedValidatorChain) pipeline.Re
|
||||
out.Validators[i] = pipeline.ResolvedValidator{
|
||||
Binding: cloneModuleBinding(validator.Binding),
|
||||
ExecutionClass: validator.ExecutionClass,
|
||||
Target: validator.Target,
|
||||
ArtifactKind: validator.ArtifactKind,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@ func TestRedactedConfigCopiesScriptoriumConfig(t *testing.T) {
|
||||
cfg.Scriptorium.ProfileDir = "./profiles"
|
||||
cfg.Workspace.Directory = "/var/lib/notarius"
|
||||
cfg.Workspace.Resume.Enabled = true
|
||||
cfg.Concurrency.StageWorkers["extract"] = 1
|
||||
|
||||
redacted := cfg.Redacted()
|
||||
|
||||
@@ -29,6 +30,10 @@ func TestRedactedConfigCopiesScriptoriumConfig(t *testing.T) {
|
||||
if cfg.Workspace.Directory != "/var/lib/notarius" {
|
||||
t.Fatalf("redaction mutated original workspace config")
|
||||
}
|
||||
redacted.Concurrency.StageWorkers["extract"] = 9
|
||||
if cfg.Concurrency.StageWorkers["extract"] != 1 {
|
||||
t.Fatalf("redaction aliased stage worker map")
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigRedactedDiagnosticsPayloadCopiesConfig(t *testing.T) {
|
||||
@@ -46,6 +51,8 @@ func TestConfigRedactedDiagnosticsPayloadCopiesConfig(t *testing.T) {
|
||||
|
||||
func TestEffectiveConfigRedactedDiagnosticsPayloadCopies(t *testing.T) {
|
||||
cfg := validConfig()
|
||||
cfg.Concurrency.TotalLLM = 4
|
||||
cfg.Concurrency.StageWorkers["extract"] = 2
|
||||
lane := cfg.Pipelines["example"].Artifacts["events"]
|
||||
lane.Extract.Options = map[string]any{"temperature": 0.2}
|
||||
lane.References = map[string]string{"roster": "./roster.yml"}
|
||||
@@ -101,6 +108,10 @@ func TestEffectiveConfigRedactedDiagnosticsPayloadCopies(t *testing.T) {
|
||||
if payload.PipelineID != effective.PipelineID || payload.ResolvedPipeline.Digest != effective.ResolvedPipeline.Digest {
|
||||
t.Fatalf("expected pipeline metadata preserved, got %+v", payload)
|
||||
}
|
||||
payload.Config.Concurrency.StageWorkers["extract"] = 4
|
||||
if effective.Config.Concurrency.StageWorkers["extract"] != 2 {
|
||||
t.Fatalf("expected effective stage worker map to be copied")
|
||||
}
|
||||
|
||||
payload.Only[0] = "changed"
|
||||
if effective.Only[0] != "events" {
|
||||
|
||||
@@ -2,6 +2,7 @@ package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/diagnostics"
|
||||
@@ -9,6 +10,7 @@ import (
|
||||
)
|
||||
|
||||
func (c Config) Validate() error {
|
||||
c.Concurrency.recomputeStageWorkerDefaults()
|
||||
if err := validateScriptorium(c.Scriptorium); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -21,9 +23,36 @@ func (c Config) Validate() error {
|
||||
if c.Concurrency.TotalLLM <= 0 {
|
||||
return fmt.Errorf("total LLM concurrency must be greater than zero")
|
||||
}
|
||||
if err := validateStageWorkers(c.Concurrency); err != nil {
|
||||
return err
|
||||
}
|
||||
return validatePipelineProfiles(c.Pipelines)
|
||||
}
|
||||
|
||||
func validateStageWorkers(cfg ConcurrencyConfig) error {
|
||||
keys := make([]string, 0, len(cfg.StageWorkers))
|
||||
for key := range cfg.StageWorkers {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, key := range keys {
|
||||
if strings.TrimSpace(key) == "" {
|
||||
return fmt.Errorf("concurrency.stage_workers key must not be empty")
|
||||
}
|
||||
if key != "extract" {
|
||||
return fmt.Errorf("concurrency.stage_workers key %q is not supported", key)
|
||||
}
|
||||
}
|
||||
extractWorkers, ok := cfg.StageWorkers["extract"]
|
||||
if !ok {
|
||||
extractWorkers = cfg.TotalLLM
|
||||
}
|
||||
if extractWorkers < 1 || extractWorkers > cfg.TotalLLM {
|
||||
return fmt.Errorf("concurrency.stage_workers.extract must be between 1 and concurrency.total_llm (%d)", cfg.TotalLLM)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateScriptorium(cfg ScriptoriumConfig) error {
|
||||
if strings.TrimSpace(cfg.ProfileDir) != "" && strings.TrimSpace(cfg.ProfileFile) != "" {
|
||||
return fmt.Errorf("scriptorium profile_dir and profile_file are mutually exclusive")
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -78,6 +79,41 @@ func TestValidateRejectsInvalidNumericFields(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateStageWorkerBoundaries(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
workers int
|
||||
wantErr bool
|
||||
}{
|
||||
{name: "below minimum", workers: 0, wantErr: true},
|
||||
{name: "minimum", workers: 1},
|
||||
{name: "maximum", workers: 4},
|
||||
{name: "above maximum", workers: 5, wantErr: true},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
cfg := validConfig()
|
||||
cfg.Concurrency.TotalLLM = 4
|
||||
cfg.Concurrency.StageWorkers["extract"] = test.workers
|
||||
err := cfg.Validate()
|
||||
if test.wantErr && (err == nil || !strings.Contains(err.Error(), "stage_workers.extract")) {
|
||||
t.Fatalf("Validate() error = %v, want extract worker range error", err)
|
||||
}
|
||||
if !test.wantErr && err != nil {
|
||||
t.Fatalf("Validate() error = %v, want nil", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRejectsUnknownEffectiveStageWorkerKey(t *testing.T) {
|
||||
cfg := validConfig()
|
||||
cfg.Concurrency.StageWorkers["merge"] = 1
|
||||
err := cfg.Validate()
|
||||
if err == nil || !strings.Contains(err.Error(), "stage_workers key") || !strings.Contains(err.Error(), "merge") {
|
||||
t.Fatalf("Validate() error = %v, want unknown stage worker key", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRejectsMutuallyExclusiveScriptoriumProfileSources(t *testing.T) {
|
||||
cfg := validConfig()
|
||||
cfg.Scriptorium.ProfileDir = "./profiles"
|
||||
@@ -497,6 +533,11 @@ func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.Module
|
||||
for _, override := range overrides {
|
||||
specs[override.Key] = override
|
||||
}
|
||||
for _, key := range []string{"fake/extract", "appendorder", "noop"} {
|
||||
spec := specs[key]
|
||||
spec.ArtifactKind = fakeArtifactKind
|
||||
specs[key] = spec
|
||||
}
|
||||
|
||||
inputs := pipeline.NewInputAdapterRegistry()
|
||||
chunkers := pipeline.NewChunkerRegistry()
|
||||
@@ -515,9 +556,15 @@ func fakeCatalog(t *testing.T, overrides ...pipeline.ModuleSpec) pipeline.Module
|
||||
mustRegisterValidator(t, validators, specs["fake/llm-validator"])
|
||||
mustRegisterOutput(t, outputs, specs["json"])
|
||||
|
||||
codecs := pipeline.NewArtifactCodecRegistry()
|
||||
if err := pipeline.RegisterArtifactCodec(codecs, fakeArtifactCodec{}); err != nil {
|
||||
t.Fatalf("register artifact codec: %v", err)
|
||||
}
|
||||
|
||||
return pipeline.ModuleCatalog{
|
||||
Inputs: inputs,
|
||||
Chunkers: chunkers,
|
||||
ArtifactCodecs: codecs,
|
||||
Extractors: extractors,
|
||||
Mergers: mergers,
|
||||
Normalizers: normalizers,
|
||||
@@ -543,21 +590,32 @@ func mustRegisterChunker(t *testing.T, registry *pipeline.ChunkerRegistry, spec
|
||||
|
||||
func mustRegisterExtractor(t *testing.T, registry *pipeline.ExtractorRegistry, spec pipeline.ModuleSpec) {
|
||||
t.Helper()
|
||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Extractor, error) { return nil, nil }); err != nil {
|
||||
validateOptions := func(options map[string]any) error {
|
||||
if err := pipeline.RejectUnknownOptions(options, "temperature"); err != nil {
|
||||
return err
|
||||
}
|
||||
if value, ok := options["temperature"]; ok {
|
||||
if _, ok := value.(float64); !ok {
|
||||
return fmt.Errorf("temperature must be a number")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if err := pipeline.RegisterExtractorBuilder[fakeArtifact](registry, spec, validateOptions, func(pipeline.BuildRequest) (contracts.Extractor[fakeArtifact], error) { return nil, nil }); err != nil {
|
||||
t.Fatalf("register extractor: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustRegisterMerger(t *testing.T, registry *pipeline.MergerRegistry, spec pipeline.ModuleSpec) {
|
||||
t.Helper()
|
||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Merger, error) { return nil, nil }); err != nil {
|
||||
if err := pipeline.RegisterMerger[fakeArtifact](registry, spec, func() (contracts.Merger[fakeArtifact], error) { return nil, nil }); err != nil {
|
||||
t.Fatalf("register merger: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustRegisterNormalizer(t *testing.T, registry *pipeline.NormalizerRegistry, spec pipeline.ModuleSpec) {
|
||||
t.Helper()
|
||||
if err := registry.RegisterWithSpec(spec, func() (contracts.Normalizer, error) { return nil, nil }); err != nil {
|
||||
if err := pipeline.RegisterNormalizer[fakeArtifact](registry, spec, func() (contracts.Normalizer[fakeArtifact], error) { return nil, nil }); err != nil {
|
||||
t.Fatalf("register normalizer: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -569,11 +627,32 @@ func mustRegisterValidator(t *testing.T, registry *pipeline.ValidatorRegistry, s
|
||||
executionClass = contracts.ExecutionClassLLMBacked
|
||||
}
|
||||
validatorSpec := pipeline.ValidatorSpec{Key: spec.Key, ExecutionClass: executionClass}
|
||||
if err := registry.RegisterWithSpec(validatorSpec, func() (contracts.Validator, error) { return nil, nil }); err != nil {
|
||||
if err := pipeline.RegisterTypedValidator[fakeArtifact](registry, fakeArtifactKind, validatorSpec, func() (contracts.TypedValidator[fakeArtifact], error) { return nil, nil }); err != nil {
|
||||
t.Fatalf("register validator: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
const fakeArtifactKind contracts.ArtifactKind = "test/artifact"
|
||||
|
||||
type fakeArtifact string
|
||||
|
||||
type fakeArtifactCodec struct{}
|
||||
|
||||
func (fakeArtifactCodec) Kind() contracts.ArtifactKind { return fakeArtifactKind }
|
||||
func (fakeArtifactCodec) Schema() contracts.ArtifactSchema {
|
||||
return contracts.ArtifactSchema{ID: "urn:notarius:test:artifact", Name: "Test artifact", Version: "1", JSONSchema: []byte(`{"type":"string"}`)}
|
||||
}
|
||||
func (fakeArtifactCodec) MediaType() string { return "application/json" }
|
||||
func (fakeArtifactCodec) Encode(value fakeArtifact) ([]byte, error) {
|
||||
return []byte(fmt.Sprintf("%q", value)), nil
|
||||
}
|
||||
func (fakeArtifactCodec) Decode(content []byte) (fakeArtifact, error) {
|
||||
if len(content) < 2 {
|
||||
return "", fmt.Errorf("invalid test artifact")
|
||||
}
|
||||
return fakeArtifact(content[1 : len(content)-1]), nil
|
||||
}
|
||||
|
||||
func mustRegisterOutput(t *testing.T, registry *pipeline.OutputEncoderRegistry, spec pipeline.ModuleSpec) {
|
||||
t.Helper()
|
||||
if err := registry.RegisterWithSpec(spec, func() (contracts.OutputEncoder, error) { return nil, nil }); err != nil {
|
||||
|
||||
65
internal/core/source/digest.go
Normal file
65
internal/core/source/digest.go
Normal file
@@ -0,0 +1,65 @@
|
||||
package source
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// DigestDocument returns a deterministic digest of the canonical source
|
||||
// document content. The existing Digest field is excluded from its own digest.
|
||||
func DigestDocument(doc *SourceDocument) (string, error) {
|
||||
if doc == nil {
|
||||
return "", fmt.Errorf("source document must not be nil")
|
||||
}
|
||||
payload := struct {
|
||||
ID string `json:"id"`
|
||||
Kind string `json:"kind"`
|
||||
Format string `json:"format"`
|
||||
Units []SourceUnit `json:"units"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}{
|
||||
ID: doc.ID,
|
||||
Kind: doc.Kind,
|
||||
Format: doc.Format,
|
||||
Units: doc.Units,
|
||||
Metadata: doc.Metadata,
|
||||
}
|
||||
encoded, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("encode source document for digest: %w", err)
|
||||
}
|
||||
sum := sha256.Sum256(encoded)
|
||||
return "sha256:" + hex.EncodeToString(sum[:]), nil
|
||||
}
|
||||
|
||||
// DigestChunk returns a deterministic digest of a chunk, including its source
|
||||
// provenance, content, units, and metadata.
|
||||
func DigestChunk(chunk Chunk) (string, error) {
|
||||
payload := struct {
|
||||
ID string `json:"id"`
|
||||
SourceID string `json:"source_id"`
|
||||
Index int `json:"index"`
|
||||
Ref SourceRef `json:"ref"`
|
||||
Content []byte `json:"content"`
|
||||
MediaType string `json:"media_type"`
|
||||
Units []SourceUnit `json:"units"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}{
|
||||
ID: chunk.ID,
|
||||
SourceID: chunk.SourceID,
|
||||
Index: chunk.Index,
|
||||
Ref: chunk.Ref,
|
||||
Content: chunk.Content,
|
||||
MediaType: chunk.MediaType,
|
||||
Units: chunk.Units,
|
||||
Metadata: chunk.Metadata,
|
||||
}
|
||||
encoded, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("encode source chunk for digest: %w", err)
|
||||
}
|
||||
sum := sha256.Sum256(encoded)
|
||||
return "sha256:" + hex.EncodeToString(sum[:]), nil
|
||||
}
|
||||
@@ -13,6 +13,7 @@ type SourceUnit struct {
|
||||
ID int `json:"id"`
|
||||
Kind string `json:"kind"`
|
||||
Text string `json:"text"`
|
||||
Ref SourceRef `json:"ref"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
@@ -21,3 +22,14 @@ type SourceRef struct {
|
||||
StartUnitID int `json:"start_unit_id"`
|
||||
EndUnitID int `json:"end_unit_id"`
|
||||
}
|
||||
|
||||
type Chunk struct {
|
||||
ID string `json:"id"`
|
||||
SourceID string `json:"source_id"`
|
||||
Index int `json:"index"`
|
||||
Ref SourceRef `json:"ref"`
|
||||
Content []byte `json:"-"`
|
||||
MediaType string `json:"media_type"`
|
||||
Units []SourceUnit `json:"units"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
@@ -142,6 +142,122 @@ func TestValidateDocumentDuplicateUnitIDs(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateDocumentUnitReferences(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
mutate func(*SourceDocument)
|
||||
wantErr string
|
||||
}{
|
||||
{
|
||||
name: "missing",
|
||||
mutate: func(doc *SourceDocument) { doc.Units[0].Ref = SourceRef{} },
|
||||
wantErr: "source unit[0].ref: source ref source_id must not be empty",
|
||||
},
|
||||
{
|
||||
name: "foreign source",
|
||||
mutate: func(doc *SourceDocument) { doc.Units[0].Ref.SourceID = "source-2" },
|
||||
wantErr: "source unit[0].ref: source ref source_id \"source-2\" does not match document id \"source-1\"",
|
||||
},
|
||||
{
|
||||
name: "non-self range",
|
||||
mutate: func(doc *SourceDocument) {
|
||||
doc.Units[0].Ref.StartUnitID = 2
|
||||
doc.Units[0].Ref.EndUnitID = 2
|
||||
},
|
||||
wantErr: "source unit[0].ref must identify source unit id 1",
|
||||
},
|
||||
{
|
||||
name: "reversed range",
|
||||
mutate: func(doc *SourceDocument) {
|
||||
doc.Units[0].Ref.StartUnitID = 2
|
||||
doc.Units[0].Ref.EndUnitID = 1
|
||||
},
|
||||
wantErr: "source unit[0].ref: source ref start_unit_id 2 appears after end_unit_id 1",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
doc := validDocument()
|
||||
tt.mutate(doc)
|
||||
|
||||
err := ValidateDocument(doc)
|
||||
if err == nil {
|
||||
t.Fatal("ValidateDocument() error = nil, want unit reference error")
|
||||
}
|
||||
if err.Error() != tt.wantErr {
|
||||
t.Fatalf("ValidateDocument() error = %q, want %q", err.Error(), tt.wantErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDigestDocumentIsDeterministicAndIncludesUnitReference(t *testing.T) {
|
||||
doc := validDocument()
|
||||
doc.Metadata = map[string]any{"second": "value", "first": true}
|
||||
first, err := DigestDocument(doc)
|
||||
if err != nil {
|
||||
t.Fatalf("DigestDocument() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
reordered := validDocument()
|
||||
reordered.Metadata = map[string]any{"first": true, "second": "value"}
|
||||
second, err := DigestDocument(reordered)
|
||||
if err != nil {
|
||||
t.Fatalf("DigestDocument(reordered) error = %v, want nil", err)
|
||||
}
|
||||
if first != second {
|
||||
t.Fatalf("digests = %q and %q, want deterministic map ordering", first, second)
|
||||
}
|
||||
|
||||
changed := validDocument()
|
||||
changed.Metadata = map[string]any{"first": true, "second": "value"}
|
||||
changed.Units[0].Ref.SourceID = "different-source"
|
||||
changedDigest, err := DigestDocument(changed)
|
||||
if err != nil {
|
||||
t.Fatalf("DigestDocument(changed) error = %v, want nil", err)
|
||||
}
|
||||
if first == changedDigest {
|
||||
t.Fatalf("digest = %q after reference change, want different digest", changedDigest)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDigestChunkIsDeterministicAndIncludesReference(t *testing.T) {
|
||||
doc := validDocument()
|
||||
chunk := Chunk{
|
||||
ID: "chunk-1",
|
||||
SourceID: doc.ID,
|
||||
Index: 0,
|
||||
Ref: SourceRef{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 2},
|
||||
Content: []byte("chunk content"),
|
||||
MediaType: "text/plain",
|
||||
Units: doc.Units,
|
||||
Metadata: map[string]any{"second": "value", "first": true},
|
||||
}
|
||||
first, err := DigestChunk(chunk)
|
||||
if err != nil {
|
||||
t.Fatalf("DigestChunk() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
chunk.Metadata = map[string]any{"first": true, "second": "value"}
|
||||
second, err := DigestChunk(chunk)
|
||||
if err != nil {
|
||||
t.Fatalf("DigestChunk(reordered metadata) error = %v, want nil", err)
|
||||
}
|
||||
if first != second {
|
||||
t.Fatalf("digests = %q and %q, want deterministic map ordering", first, second)
|
||||
}
|
||||
|
||||
chunk.Ref.EndUnitID = 1
|
||||
changed, err := DigestChunk(chunk)
|
||||
if err != nil {
|
||||
t.Fatalf("DigestChunk(changed ref) error = %v, want nil", err)
|
||||
}
|
||||
if first == changed {
|
||||
t.Fatalf("digest = %q after reference change, want different digest", changed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRefValid(t *testing.T) {
|
||||
doc := validDocument()
|
||||
ref := SourceRef{
|
||||
@@ -274,11 +390,13 @@ func validDocument() *SourceDocument {
|
||||
ID: 1,
|
||||
Kind: "paragraph",
|
||||
Text: "First unit.",
|
||||
Ref: SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1},
|
||||
},
|
||||
{
|
||||
ID: 2,
|
||||
Kind: "paragraph",
|
||||
Text: "Second unit.",
|
||||
Ref: SourceRef{SourceID: "source-1", StartUnitID: 2, EndUnitID: 2},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
@@ -44,6 +44,14 @@ func ValidateDocument(doc *SourceDocument) error {
|
||||
}
|
||||
seenUnitIDs[unit.ID] = struct{}{}
|
||||
}
|
||||
for i, unit := range doc.Units {
|
||||
if err := ValidateRef(doc, unit.Ref); err != nil {
|
||||
return fmt.Errorf("source unit[%d].ref: %w", i, err)
|
||||
}
|
||||
if unit.Ref.StartUnitID != unit.ID || unit.Ref.EndUnitID != unit.ID {
|
||||
return fmt.Errorf("source unit[%d].ref must identify source unit id %d", i, unit.ID)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -2,7 +2,10 @@ package workspace
|
||||
|
||||
import "time"
|
||||
|
||||
const WorkspaceSchemaVersion = "notarius.workspace.v1"
|
||||
const (
|
||||
WorkspaceSchemaVersion = "notarius.workspace.v2"
|
||||
WorkspaceSchemaVersionV1 = "notarius.workspace.v1"
|
||||
)
|
||||
|
||||
type StageName string
|
||||
|
||||
|
||||
@@ -7,6 +7,12 @@ import (
|
||||
)
|
||||
|
||||
func TestStageManifestDefaults(t *testing.T) {
|
||||
if WorkspaceSchemaVersion != "notarius.workspace.v2" {
|
||||
t.Fatalf("current schema version = %q, want notarius.workspace.v2", WorkspaceSchemaVersion)
|
||||
}
|
||||
if WorkspaceSchemaVersionV1 != "notarius.workspace.v1" {
|
||||
t.Fatalf("legacy schema version = %q, want notarius.workspace.v1", WorkspaceSchemaVersionV1)
|
||||
}
|
||||
manifest := NewStageManifest(StageExtract, StatusRunning)
|
||||
|
||||
if manifest.WorkspaceSchemaVersion != WorkspaceSchemaVersion {
|
||||
|
||||
@@ -78,80 +78,98 @@ func (l *WorkspaceLoader) Chunk(moduleKey string, sourceDigest string) (pipeline
|
||||
if len(chunks) == 0 {
|
||||
return pipeline.ChunkCheckpoint{}, invalidDecision("chunk checkpoint payload has no chunks")
|
||||
}
|
||||
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), chunkOutputDigests(chunks)) {
|
||||
outputDigests, err := chunkOutputDigests(chunks)
|
||||
if err != nil {
|
||||
return pipeline.ChunkCheckpoint{}, invalidDecision("chunk checkpoint output cannot be digested: %v", err)
|
||||
}
|
||||
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), outputDigests) {
|
||||
return pipeline.ChunkCheckpoint{}, invalidDecision("chunk checkpoint output digests do not match payload")
|
||||
}
|
||||
return pipeline.ChunkCheckpoint{Chunks: chunks, Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
||||
}
|
||||
|
||||
func (l *WorkspaceLoader) Extract(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.ExtractCheckpoint, pipeline.CheckpointDecision) {
|
||||
func (l *WorkspaceLoader) Extract(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.ExtractCheckpoint, pipeline.CheckpointDecision) {
|
||||
var manifest coreworkspace.ExtractLaneManifest
|
||||
if decision := l.readJSON(laneManifestPath("extract", laneID), &manifest); !decision.Reused {
|
||||
return pipeline.ExtractCheckpoint{}, decision
|
||||
if d := l.readJSON(laneManifestPath("extract", laneID), &manifest); !d.Reused {
|
||||
return pipeline.ExtractCheckpoint{}, d
|
||||
}
|
||||
if decision := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageExtract, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded, coreworkspace.StatusSucceededWithRejections); !decision.Reused {
|
||||
return pipeline.ExtractCheckpoint{}, decision
|
||||
if d := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageExtract, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded, coreworkspace.StatusSucceededWithRejections); !d.Reused {
|
||||
return pipeline.ExtractCheckpoint{}, d
|
||||
}
|
||||
var payload extractOutputsEnvelope
|
||||
if decision := l.readJSON(lanePayloadPath("extract", laneID, "outputs.json"), &payload); !decision.Reused {
|
||||
return pipeline.ExtractCheckpoint{}, decision
|
||||
var payload artifactExtractEnvelope
|
||||
if d := l.readJSON(lanePayloadPath("extract", laneID, "outputs.json"), &payload); !d.Reused {
|
||||
return pipeline.ExtractCheckpoint{}, d
|
||||
}
|
||||
outputs, err := extractOutputsFromEnvelope(payload.Outputs)
|
||||
outputs, err := artifactCheckpointOutputs(payload.Outputs)
|
||||
if err != nil {
|
||||
return pipeline.ExtractCheckpoint{}, invalidDecision("extract checkpoint payload is invalid: %v", err)
|
||||
return pipeline.ExtractCheckpoint{}, invalidDecision("extract artifact checkpoint payload is invalid: %v", err)
|
||||
}
|
||||
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), rawOutputDigests(extractPayloads(outputs))) {
|
||||
return pipeline.ExtractCheckpoint{}, invalidDecision("extract checkpoint output digests do not match payload")
|
||||
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(outputs)) {
|
||||
return pipeline.ExtractCheckpoint{}, invalidDecision("extract artifact checkpoint output digests do not match payload")
|
||||
}
|
||||
return pipeline.ExtractCheckpoint{
|
||||
Outputs: outputs,
|
||||
Rejected: cloneRejectedOutputs(payload.Rejected),
|
||||
Warnings: cloneWarnings(payload.Warnings),
|
||||
}, reusedDecision()
|
||||
return pipeline.ExtractCheckpoint{Outputs: outputs, Rejected: cloneRejectedOutputs(payload.Rejected), Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
||||
}
|
||||
|
||||
func (l *WorkspaceLoader) Merge(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.MergeCheckpoint, pipeline.CheckpointDecision) {
|
||||
func (l *WorkspaceLoader) Merge(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.MergeCheckpoint, pipeline.CheckpointDecision) {
|
||||
var manifest coreworkspace.MergeLaneManifest
|
||||
if decision := l.readJSON(laneManifestPath("merge", laneID), &manifest); !decision.Reused {
|
||||
return pipeline.MergeCheckpoint{}, decision
|
||||
if d := l.readJSON(laneManifestPath("merge", laneID), &manifest); !d.Reused {
|
||||
return pipeline.MergeCheckpoint{}, d
|
||||
}
|
||||
if decision := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageMerge, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded); !decision.Reused {
|
||||
return pipeline.MergeCheckpoint{}, decision
|
||||
if d := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageMerge, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded); !d.Reused {
|
||||
return pipeline.MergeCheckpoint{}, d
|
||||
}
|
||||
var payload mergeOutputEnvelope
|
||||
if decision := l.readJSON(lanePayloadPath("merge", laneID, "output.json"), &payload); !decision.Reused {
|
||||
return pipeline.MergeCheckpoint{}, decision
|
||||
var payload artifactSingleEnvelope
|
||||
if d := l.readJSON(lanePayloadPath("merge", laneID, "output.json"), &payload); !d.Reused {
|
||||
return pipeline.MergeCheckpoint{}, d
|
||||
}
|
||||
output, err := mergeOutputFromEnvelope(payload.Output)
|
||||
values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output})
|
||||
if err != nil {
|
||||
return pipeline.MergeCheckpoint{}, invalidDecision("merge checkpoint payload is invalid: %v", err)
|
||||
return pipeline.MergeCheckpoint{}, invalidDecision("merge artifact checkpoint payload is invalid: %v", err)
|
||||
}
|
||||
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), rawOutputDigests([]contracts.RawPayload{output.Payload})) {
|
||||
return pipeline.MergeCheckpoint{}, invalidDecision("merge checkpoint output digest does not match payload")
|
||||
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
|
||||
return pipeline.MergeCheckpoint{}, invalidDecision("merge artifact checkpoint output digest does not match payload")
|
||||
}
|
||||
return pipeline.MergeCheckpoint{Output: output, Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
||||
return pipeline.MergeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
||||
}
|
||||
|
||||
func (l *WorkspaceLoader) Normalize(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
|
||||
func (l *WorkspaceLoader) Normalize(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint) (pipeline.NormalizeCheckpoint, pipeline.CheckpointDecision) {
|
||||
var manifest coreworkspace.NormalizeLaneManifest
|
||||
if decision := l.readJSON(laneManifestPath("normalize", laneID), &manifest); !decision.Reused {
|
||||
return pipeline.NormalizeCheckpoint{}, decision
|
||||
if d := l.readJSON(laneManifestPath("normalize", laneID), &manifest); !d.Reused {
|
||||
return pipeline.NormalizeCheckpoint{}, d
|
||||
}
|
||||
if decision := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageNormalize, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded); !decision.Reused {
|
||||
return pipeline.NormalizeCheckpoint{}, decision
|
||||
if d := l.validateLaneManifest(manifest.StageManifest, coreworkspace.StageNormalize, laneID, moduleKey, dependencies, coreworkspace.StatusSucceeded); !d.Reused {
|
||||
return pipeline.NormalizeCheckpoint{}, d
|
||||
}
|
||||
var payload normalizeOutputEnvelope
|
||||
if decision := l.readJSON(lanePayloadPath("normalize", laneID, "output.json"), &payload); !decision.Reused {
|
||||
return pipeline.NormalizeCheckpoint{}, decision
|
||||
var payload artifactSingleEnvelope
|
||||
if d := l.readJSON(lanePayloadPath("normalize", laneID, "output.json"), &payload); !d.Reused {
|
||||
return pipeline.NormalizeCheckpoint{}, d
|
||||
}
|
||||
output, err := normalizeOutputFromEnvelope(payload.Output)
|
||||
values, err := artifactCheckpointOutputs([]artifactCheckpointEnvelope{payload.Output})
|
||||
if err != nil {
|
||||
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize checkpoint payload is invalid: %v", err)
|
||||
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize artifact checkpoint payload is invalid: %v", err)
|
||||
}
|
||||
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), rawOutputDigests([]contracts.RawPayload{output.Payload})) {
|
||||
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize checkpoint output digest does not match payload")
|
||||
if !fingerprintsEqual(coreworkspaceToPipelineFingerprints(manifest.OutputDigests), artifactOutputDigests(values)) {
|
||||
return pipeline.NormalizeCheckpoint{}, invalidDecision("normalize artifact checkpoint output digest does not match payload")
|
||||
}
|
||||
return pipeline.NormalizeCheckpoint{Output: output, Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
||||
return pipeline.NormalizeCheckpoint{Output: values[0], Warnings: cloneWarnings(payload.Warnings)}, reusedDecision()
|
||||
}
|
||||
|
||||
func artifactCheckpointOutputs(values []artifactCheckpointEnvelope) ([]pipeline.CheckpointArtifact, error) {
|
||||
if len(values) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
out := make([]pipeline.CheckpointArtifact, 0, len(values))
|
||||
for _, v := range values {
|
||||
content, err := contentFromEnvelope(v.Content)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if strings.TrimSpace(string(v.Kind)) == "" || strings.TrimSpace(v.Schema.ID) == "" || strings.TrimSpace(v.Schema.Version) == "" || strings.TrimSpace(v.SchemaDigest) == "" {
|
||||
return nil, fmt.Errorf("artifact codec identity is incomplete")
|
||||
}
|
||||
out = append(out, pipeline.CheckpointArtifact{LaneID: v.LaneID, ModuleKey: v.ModuleKey, SourceID: v.SourceID, ChunkID: v.ChunkID, ChunkIndex: v.ChunkIndex, ChunkRef: v.ChunkRef, SchemaDigest: v.SchemaDigest, Artifact: contracts.SerializedArtifact{Kind: v.Kind, Schema: v.Schema, MediaType: v.Content.MediaType, Content: content, Metadata: cloneMetadata(v.Content.Metadata)}})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (l *WorkspaceLoader) readJSON(name string, out any) pipeline.CheckpointDecision {
|
||||
@@ -180,6 +198,9 @@ func (l *WorkspaceLoader) validateManifest(manifest coreworkspace.StageManifest,
|
||||
}
|
||||
|
||||
func (l *WorkspaceLoader) validateLaneManifest(manifest coreworkspace.StageManifest, stage coreworkspace.StageName, laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, statuses ...coreworkspace.StageStatus) pipeline.CheckpointDecision {
|
||||
if manifest.WorkspaceSchemaVersion == coreworkspace.WorkspaceSchemaVersionV1 {
|
||||
return invalidDecision("checkpoint workspace schema version %q is incompatible with %q and must be recomputed", manifest.WorkspaceSchemaVersion, coreworkspace.WorkspaceSchemaVersion)
|
||||
}
|
||||
if manifest.WorkspaceSchemaVersion != coreworkspace.WorkspaceSchemaVersion {
|
||||
return invalidDecision("checkpoint workspace schema version %q is not supported", manifest.WorkspaceSchemaVersion)
|
||||
}
|
||||
@@ -211,95 +232,30 @@ func (l *WorkspaceLoader) validateLaneManifest(manifest coreworkspace.StageManif
|
||||
return reusedDecision()
|
||||
}
|
||||
|
||||
func sourceChunksFromEnvelope(values []chunkEnvelope) ([]contracts.SourceChunk, error) {
|
||||
func sourceChunksFromEnvelope(values []chunkEnvelope) ([]source.Chunk, error) {
|
||||
if len(values) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
out := make([]contracts.SourceChunk, 0, len(values))
|
||||
out := make([]source.Chunk, 0, len(values))
|
||||
for _, value := range values {
|
||||
content, err := contentFromEnvelope(value.Content)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, contracts.SourceChunk{
|
||||
ID: value.ID,
|
||||
SourceID: value.SourceID,
|
||||
Index: value.Index,
|
||||
StartUnitID: value.StartUnitID,
|
||||
EndUnitID: value.EndUnitID,
|
||||
Content: content,
|
||||
MediaType: value.Content.MediaType,
|
||||
Units: cloneSourceUnits(value.Units),
|
||||
Metadata: cloneMetadata(value.Metadata),
|
||||
out = append(out, source.Chunk{
|
||||
ID: value.ID,
|
||||
SourceID: value.SourceID,
|
||||
Index: value.Index,
|
||||
Ref: value.Ref,
|
||||
Content: content,
|
||||
MediaType: value.Content.MediaType,
|
||||
Units: cloneSourceUnits(value.Units),
|
||||
Metadata: cloneMetadata(value.Metadata),
|
||||
})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func extractOutputsFromEnvelope(values []extractOutputEnvelope) ([]contracts.ExtractOutput, error) {
|
||||
if len(values) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
out := make([]contracts.ExtractOutput, 0, len(values))
|
||||
for _, value := range values {
|
||||
payload, err := rawPayloadFromEnvelope(value.Payload)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, contracts.ExtractOutput{
|
||||
LaneID: value.LaneID,
|
||||
ExtractorKey: value.ExtractorKey,
|
||||
SourceID: value.SourceID,
|
||||
ChunkID: value.ChunkID,
|
||||
ChunkIndex: value.ChunkIndex,
|
||||
Schema: value.Schema,
|
||||
Payload: payload,
|
||||
})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func mergeOutputFromEnvelope(value mergeOutputPayload) (contracts.MergeOutput, error) {
|
||||
payload, err := rawPayloadFromEnvelope(value.Payload)
|
||||
if err != nil {
|
||||
return contracts.MergeOutput{}, err
|
||||
}
|
||||
return contracts.MergeOutput{
|
||||
LaneID: value.LaneID,
|
||||
MergerKey: value.MergerKey,
|
||||
SourceID: value.SourceID,
|
||||
Schema: value.Schema,
|
||||
Payload: payload,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func normalizeOutputFromEnvelope(value normalizeOutputPayload) (contracts.NormalizeOutput, error) {
|
||||
payload, err := rawPayloadFromEnvelope(value.Payload)
|
||||
if err != nil {
|
||||
return contracts.NormalizeOutput{}, err
|
||||
}
|
||||
return contracts.NormalizeOutput{
|
||||
LaneID: value.LaneID,
|
||||
NormalizerKey: value.NormalizerKey,
|
||||
SourceID: value.SourceID,
|
||||
Schema: value.Schema,
|
||||
Payload: payload,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func rawPayloadFromEnvelope(value binaryEnvelope) (contracts.RawPayload, error) {
|
||||
content, err := contentFromEnvelope(value)
|
||||
if err != nil {
|
||||
return contracts.RawPayload{}, err
|
||||
}
|
||||
return contracts.RawPayload{
|
||||
Content: content,
|
||||
MediaType: value.MediaType,
|
||||
Metadata: cloneMetadata(value.Metadata),
|
||||
Warnings: cloneWarnings(value.Warnings),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func contentFromEnvelope(value binaryEnvelope) ([]byte, error) {
|
||||
content, err := base64.StdEncoding.DecodeString(value.ContentBase64)
|
||||
if err != nil {
|
||||
|
||||
@@ -73,7 +73,11 @@ func (r *WorkspaceRecorder) ChunkRunning(moduleKey string, sourceDigest string)
|
||||
return r.writeManifest("chunk/manifest.json", coreworkspace.ChunkManifest{StageManifest: manifest})
|
||||
}
|
||||
|
||||
func (r *WorkspaceRecorder) ChunkSucceeded(moduleKey string, sourceDigest string, chunks []contracts.SourceChunk, warnings []contracts.Warning) error {
|
||||
func (r *WorkspaceRecorder) ChunkSucceeded(moduleKey string, sourceDigest string, chunks []source.Chunk, warnings []contracts.Warning) error {
|
||||
outputDigests, err := chunkOutputDigests(chunks)
|
||||
if err != nil {
|
||||
return fmt.Errorf("digest chunk checkpoint output: %w", err)
|
||||
}
|
||||
payload := chunksEnvelope{Chunks: chunkEnvelopes(chunks), Warnings: cloneWarnings(warnings)}
|
||||
if err := r.writePayload("chunk/chunks.json", payload); err != nil {
|
||||
return err
|
||||
@@ -81,7 +85,7 @@ func (r *WorkspaceRecorder) ChunkSucceeded(moduleKey string, sourceDigest string
|
||||
manifest := r.newStageManifest(coreworkspace.StageChunk, coreworkspace.StatusSucceeded)
|
||||
manifest.ModuleKey = moduleKey
|
||||
manifest.DependencyFingerprints = workspaceFingerprints(digestFingerprints("source_document", sourceDigest))
|
||||
manifest.OutputDigests = workspaceFingerprints(chunkOutputDigests(chunks))
|
||||
manifest.OutputDigests = workspaceFingerprints(outputDigests)
|
||||
manifest.ValidationStatus = validationStatusString(warnings, nil)
|
||||
manifest.CompletedAt = timePtr(r.timestamp())
|
||||
return r.writeManifest("chunk/manifest.json", coreworkspace.ChunkManifest{
|
||||
@@ -115,25 +119,17 @@ func (r *WorkspaceRecorder) ExtractRunning(laneID string, moduleKey string, depe
|
||||
return r.writeManifest(laneManifestPath("extract", laneID), coreworkspace.ExtractLaneManifest{StageManifest: manifest})
|
||||
}
|
||||
|
||||
func (r *WorkspaceRecorder) ExtractSucceeded(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []contracts.ExtractOutput, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
||||
payload := extractOutputsEnvelope{
|
||||
Outputs: extractOutputEnvelopes(outputs),
|
||||
Rejected: cloneRejectedOutputs(rejected),
|
||||
Warnings: cloneWarnings(warnings),
|
||||
}
|
||||
func (r *WorkspaceRecorder) ExtractSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, outputs []pipeline.CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
||||
payload := artifactExtractEnvelope{Outputs: artifactCheckpointEnvelopes(outputs), Rejected: cloneRejectedOutputs(rejected), Warnings: cloneWarnings(warnings)}
|
||||
if err := r.writePayload(lanePayloadPath("extract", laneID, "outputs.json"), payload); err != nil {
|
||||
return err
|
||||
}
|
||||
manifest := r.laneManifest(coreworkspace.StageExtract, statusForRejected(rejected), laneID, moduleKey, dependencies)
|
||||
manifest.OutputDigests = workspaceFingerprints(rawOutputDigests(extractPayloads(outputs)))
|
||||
manifest.OutputDigests = workspaceFingerprints(artifactOutputDigests(outputs))
|
||||
manifest.ValidationStatus = validationStatusString(warnings, rejected)
|
||||
manifest.Rejections = rejectionSummaries(rejected)
|
||||
manifest.CompletedAt = timePtr(r.timestamp())
|
||||
return r.writeManifest(laneManifestPath("extract", laneID), coreworkspace.ExtractLaneManifest{
|
||||
StageManifest: manifest,
|
||||
ChunkCount: len(outputs) + len(rejected),
|
||||
OutputCount: len(outputs),
|
||||
})
|
||||
return r.writeManifest(laneManifestPath("extract", laneID), coreworkspace.ExtractLaneManifest{StageManifest: manifest, ChunkCount: len(outputs) + len(rejected), OutputCount: len(outputs)})
|
||||
}
|
||||
|
||||
func (r *WorkspaceRecorder) ExtractFailed(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, err error) error {
|
||||
@@ -149,19 +145,15 @@ func (r *WorkspaceRecorder) MergeRunning(laneID string, moduleKey string, depend
|
||||
return r.writeManifest(laneManifestPath("merge", laneID), coreworkspace.MergeLaneManifest{StageManifest: manifest})
|
||||
}
|
||||
|
||||
func (r *WorkspaceRecorder) MergeSucceeded(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output contracts.MergeOutput, warnings []contracts.Warning) error {
|
||||
payload := mergeOutputEnvelope{Output: mergeOutputEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}
|
||||
if err := r.writePayload(lanePayloadPath("merge", laneID, "output.json"), payload); err != nil {
|
||||
func (r *WorkspaceRecorder) MergeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
if err := r.writePayload(lanePayloadPath("merge", laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
|
||||
return err
|
||||
}
|
||||
manifest := r.laneManifest(coreworkspace.StageMerge, coreworkspace.StatusSucceeded, laneID, moduleKey, dependencies)
|
||||
manifest.OutputDigests = workspaceFingerprints(rawOutputDigests([]contracts.RawPayload{output.Payload}))
|
||||
manifest.OutputDigests = workspaceFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
|
||||
manifest.ValidationStatus = validationStatusString(warnings, nil)
|
||||
manifest.CompletedAt = timePtr(r.timestamp())
|
||||
return r.writeManifest(laneManifestPath("merge", laneID), coreworkspace.MergeLaneManifest{
|
||||
StageManifest: manifest,
|
||||
InputCount: len(dependencies),
|
||||
})
|
||||
return r.writeManifest(laneManifestPath("merge", laneID), coreworkspace.MergeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
||||
}
|
||||
|
||||
func (r *WorkspaceRecorder) MergeRejected(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||
@@ -185,13 +177,12 @@ func (r *WorkspaceRecorder) NormalizeRunning(laneID string, moduleKey string, de
|
||||
return r.writeManifest(laneManifestPath("normalize", laneID), coreworkspace.NormalizeLaneManifest{StageManifest: manifest})
|
||||
}
|
||||
|
||||
func (r *WorkspaceRecorder) NormalizeSucceeded(laneID string, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output contracts.NormalizeOutput, warnings []contracts.Warning) error {
|
||||
payload := normalizeOutputEnvelope{Output: normalizeOutputEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}
|
||||
if err := r.writePayload(lanePayloadPath("normalize", laneID, "output.json"), payload); err != nil {
|
||||
func (r *WorkspaceRecorder) NormalizeSucceeded(laneID, moduleKey string, dependencies []pipeline.CheckpointFingerprint, output pipeline.CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
if err := r.writePayload(lanePayloadPath("normalize", laneID, "output.json"), artifactSingleEnvelope{Output: artifactCheckpointEnvelopeFromOutput(output), Warnings: cloneWarnings(warnings)}); err != nil {
|
||||
return err
|
||||
}
|
||||
manifest := r.laneManifest(coreworkspace.StageNormalize, coreworkspace.StatusSucceeded, laneID, moduleKey, dependencies)
|
||||
manifest.OutputDigests = workspaceFingerprints(rawOutputDigests([]contracts.RawPayload{output.Payload}))
|
||||
manifest.OutputDigests = workspaceFingerprints(artifactOutputDigests([]pipeline.CheckpointArtifact{output}))
|
||||
manifest.ValidationStatus = validationStatusString(warnings, nil)
|
||||
manifest.CompletedAt = timePtr(r.timestamp())
|
||||
return r.writeManifest(laneManifestPath("normalize", laneID), coreworkspace.NormalizeLaneManifest{StageManifest: manifest, InputCount: len(dependencies)})
|
||||
@@ -260,56 +251,13 @@ type chunksEnvelope struct {
|
||||
}
|
||||
|
||||
type chunkEnvelope struct {
|
||||
ID string `json:"id"`
|
||||
SourceID string `json:"source_id"`
|
||||
Index int `json:"index"`
|
||||
StartUnitID int `json:"start_unit_id"`
|
||||
EndUnitID int `json:"end_unit_id"`
|
||||
Content binaryEnvelope `json:"content"`
|
||||
Units []source.SourceUnit `json:"units,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
type extractOutputsEnvelope struct {
|
||||
Outputs []extractOutputEnvelope `json:"outputs"`
|
||||
Rejected []contracts.RejectedOutput `json:"rejected,omitempty"`
|
||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
type extractOutputEnvelope struct {
|
||||
LaneID string `json:"lane_id"`
|
||||
ExtractorKey string `json:"extractor_key"`
|
||||
SourceID string `json:"source_id"`
|
||||
ChunkID string `json:"chunk_id"`
|
||||
ChunkIndex int `json:"chunk_index"`
|
||||
Schema contracts.ResponseSchema `json:"schema,omitempty"`
|
||||
Payload binaryEnvelope `json:"payload"`
|
||||
}
|
||||
|
||||
type mergeOutputEnvelope struct {
|
||||
Output mergeOutputPayload `json:"output"`
|
||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
type mergeOutputPayload struct {
|
||||
LaneID string `json:"lane_id"`
|
||||
MergerKey string `json:"merger_key"`
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
Schema contracts.ResponseSchema `json:"schema,omitempty"`
|
||||
Payload binaryEnvelope `json:"payload"`
|
||||
}
|
||||
|
||||
type normalizeOutputEnvelope struct {
|
||||
Output normalizeOutputPayload `json:"output"`
|
||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
type normalizeOutputPayload struct {
|
||||
LaneID string `json:"lane_id"`
|
||||
NormalizerKey string `json:"normalizer_key"`
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
Schema contracts.ResponseSchema `json:"schema,omitempty"`
|
||||
Payload binaryEnvelope `json:"payload"`
|
||||
ID string `json:"id"`
|
||||
SourceID string `json:"source_id"`
|
||||
Index int `json:"index"`
|
||||
Ref source.SourceRef `json:"ref"`
|
||||
Content binaryEnvelope `json:"content"`
|
||||
Units []source.SourceUnit `json:"units,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
type binaryEnvelope struct {
|
||||
@@ -320,74 +268,70 @@ type binaryEnvelope struct {
|
||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
func chunkEnvelopes(chunks []contracts.SourceChunk) []chunkEnvelope {
|
||||
type artifactCheckpointEnvelope struct {
|
||||
LaneID string `json:"lane_id"`
|
||||
ModuleKey string `json:"module_key"`
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
ChunkID string `json:"chunk_id,omitempty"`
|
||||
ChunkIndex int `json:"chunk_index,omitempty"`
|
||||
ChunkRef source.SourceRef `json:"chunk_ref,omitempty"`
|
||||
Kind contracts.ArtifactKind `json:"artifact_kind"`
|
||||
Schema contracts.ArtifactSchema `json:"schema"`
|
||||
SchemaDigest string `json:"schema_digest"`
|
||||
Content binaryEnvelope `json:"content"`
|
||||
}
|
||||
type artifactExtractEnvelope struct {
|
||||
Outputs []artifactCheckpointEnvelope `json:"outputs"`
|
||||
Rejected []contracts.RejectedOutput `json:"rejected,omitempty"`
|
||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
type artifactSingleEnvelope struct {
|
||||
Output artifactCheckpointEnvelope `json:"output"`
|
||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
func artifactCheckpointEnvelopeFromOutput(output pipeline.CheckpointArtifact) artifactCheckpointEnvelope {
|
||||
schema := contracts.CloneArtifactSchema(output.Artifact.Schema)
|
||||
schema.JSONSchema = nil
|
||||
return artifactCheckpointEnvelope{LaneID: output.LaneID, ModuleKey: output.ModuleKey, SourceID: output.SourceID, ChunkID: output.ChunkID, ChunkIndex: output.ChunkIndex, ChunkRef: output.ChunkRef, Kind: output.Artifact.Kind, Schema: schema, SchemaDigest: output.SchemaDigest, Content: binaryEnvelopeFromContent(output.Artifact.Content, output.Artifact.MediaType, output.Artifact.Metadata, nil)}
|
||||
}
|
||||
func artifactCheckpointEnvelopes(outputs []pipeline.CheckpointArtifact) []artifactCheckpointEnvelope {
|
||||
if len(outputs) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]artifactCheckpointEnvelope, 0, len(outputs))
|
||||
for _, v := range outputs {
|
||||
out = append(out, artifactCheckpointEnvelopeFromOutput(v))
|
||||
}
|
||||
return out
|
||||
}
|
||||
func artifactOutputDigests(outputs []pipeline.CheckpointArtifact) []pipeline.CheckpointFingerprint {
|
||||
values := make([]pipeline.CheckpointFingerprint, 0, len(outputs))
|
||||
for i, v := range outputs {
|
||||
values = append(values, pipeline.CheckpointFingerprint{Name: fmt.Sprintf("artifact[%d]", i), Value: contentDigest(v.Artifact.Content)})
|
||||
}
|
||||
return normalizeFingerprints(values)
|
||||
}
|
||||
|
||||
func chunkEnvelopes(chunks []source.Chunk) []chunkEnvelope {
|
||||
if len(chunks) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]chunkEnvelope, 0, len(chunks))
|
||||
for _, chunk := range chunks {
|
||||
out = append(out, chunkEnvelope{
|
||||
ID: chunk.ID,
|
||||
SourceID: chunk.SourceID,
|
||||
Index: chunk.Index,
|
||||
StartUnitID: chunk.StartUnitID,
|
||||
EndUnitID: chunk.EndUnitID,
|
||||
Content: binaryEnvelopeFromContent(chunk.Content, chunk.MediaType, chunk.Metadata, nil),
|
||||
Units: cloneSourceUnits(chunk.Units),
|
||||
Metadata: cloneMetadata(chunk.Metadata),
|
||||
ID: chunk.ID,
|
||||
SourceID: chunk.SourceID,
|
||||
Index: chunk.Index,
|
||||
Ref: chunk.Ref,
|
||||
Content: binaryEnvelopeFromContent(chunk.Content, chunk.MediaType, chunk.Metadata, nil),
|
||||
Units: cloneSourceUnits(chunk.Units),
|
||||
Metadata: cloneMetadata(chunk.Metadata),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func extractOutputEnvelopes(outputs []contracts.ExtractOutput) []extractOutputEnvelope {
|
||||
if len(outputs) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]extractOutputEnvelope, 0, len(outputs))
|
||||
for _, output := range outputs {
|
||||
out = append(out, extractOutputEnvelope{
|
||||
LaneID: output.LaneID,
|
||||
ExtractorKey: output.ExtractorKey,
|
||||
SourceID: output.SourceID,
|
||||
ChunkID: output.ChunkID,
|
||||
ChunkIndex: output.ChunkIndex,
|
||||
Schema: schemaEnvelope(output.Schema),
|
||||
Payload: binaryEnvelopeFromPayload(output.Payload),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func mergeOutputEnvelopeFromOutput(output contracts.MergeOutput) mergeOutputPayload {
|
||||
return mergeOutputPayload{
|
||||
LaneID: output.LaneID,
|
||||
MergerKey: output.MergerKey,
|
||||
SourceID: output.SourceID,
|
||||
Schema: schemaEnvelope(output.Schema),
|
||||
Payload: binaryEnvelopeFromPayload(output.Payload),
|
||||
}
|
||||
}
|
||||
|
||||
func normalizeOutputEnvelopeFromOutput(output contracts.NormalizeOutput) normalizeOutputPayload {
|
||||
return normalizeOutputPayload{
|
||||
LaneID: output.LaneID,
|
||||
NormalizerKey: output.NormalizerKey,
|
||||
SourceID: output.SourceID,
|
||||
Schema: schemaEnvelope(output.Schema),
|
||||
Payload: binaryEnvelopeFromPayload(output.Payload),
|
||||
}
|
||||
}
|
||||
|
||||
func schemaEnvelope(schema contracts.ResponseSchema) contracts.ResponseSchema {
|
||||
schema.JSONSchema = nil
|
||||
return schema
|
||||
}
|
||||
|
||||
func binaryEnvelopeFromPayload(payload contracts.RawPayload) binaryEnvelope {
|
||||
return binaryEnvelopeFromContent(payload.Content, payload.MediaType, payload.Metadata, payload.Warnings)
|
||||
}
|
||||
|
||||
func binaryEnvelopeFromContent(content []byte, mediaType string, metadata map[string]any, warnings []contracts.Warning) binaryEnvelope {
|
||||
return binaryEnvelope{
|
||||
ContentBase64: base64.StdEncoding.EncodeToString(content),
|
||||
@@ -414,6 +358,7 @@ func cloneSourceUnits(units []source.SourceUnit) []source.SourceUnit {
|
||||
ID: unit.ID,
|
||||
Kind: unit.Kind,
|
||||
Text: unit.Text,
|
||||
Ref: unit.Ref,
|
||||
Metadata: cloneMetadata(unit.Metadata),
|
||||
})
|
||||
}
|
||||
@@ -445,37 +390,19 @@ func cloneMetadata(metadata map[string]any) map[string]any {
|
||||
return out
|
||||
}
|
||||
|
||||
func rawOutputDigests(payloads []contracts.RawPayload) []pipeline.CheckpointFingerprint {
|
||||
values := make([]pipeline.CheckpointFingerprint, 0, len(payloads))
|
||||
for i, payload := range payloads {
|
||||
values = append(values, pipeline.CheckpointFingerprint{
|
||||
Name: fmt.Sprintf("payload[%d]", i),
|
||||
Value: contentDigest(payload.Content),
|
||||
})
|
||||
}
|
||||
return normalizeFingerprints(values)
|
||||
}
|
||||
|
||||
func extractPayloads(outputs []contracts.ExtractOutput) []contracts.RawPayload {
|
||||
if len(outputs) == 0 {
|
||||
return nil
|
||||
}
|
||||
payloads := make([]contracts.RawPayload, 0, len(outputs))
|
||||
for _, output := range outputs {
|
||||
payloads = append(payloads, output.Payload)
|
||||
}
|
||||
return payloads
|
||||
}
|
||||
|
||||
func chunkOutputDigests(chunks []contracts.SourceChunk) []pipeline.CheckpointFingerprint {
|
||||
func chunkOutputDigests(chunks []source.Chunk) ([]pipeline.CheckpointFingerprint, error) {
|
||||
values := make([]pipeline.CheckpointFingerprint, 0, len(chunks))
|
||||
for _, chunk := range chunks {
|
||||
digest, err := source.DigestChunk(chunk)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("chunk %q: %w", chunk.ID, err)
|
||||
}
|
||||
values = append(values, pipeline.CheckpointFingerprint{
|
||||
Name: chunk.ID,
|
||||
Value: contentDigest(chunk.Content),
|
||||
Value: digest,
|
||||
})
|
||||
}
|
||||
return normalizeFingerprints(values)
|
||||
return normalizeFingerprints(values), nil
|
||||
}
|
||||
|
||||
func digestFingerprints(name string, digest string) []pipeline.CheckpointFingerprint {
|
||||
|
||||
@@ -22,18 +22,17 @@ func TestWorkspaceRecorderWritesSuccessfulCheckpointFiles(t *testing.T) {
|
||||
Kind: "document",
|
||||
Format: "text/plain",
|
||||
Digest: "sha256:source",
|
||||
Units: []source.SourceUnit{{ID: 1, Kind: "line", Text: "hello"}},
|
||||
Units: []source.SourceUnit{{ID: 1, Kind: "line", Text: "hello", Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1}}},
|
||||
}
|
||||
chunks := []contracts.SourceChunk{
|
||||
chunks := []source.Chunk{
|
||||
{
|
||||
ID: "chunk-1",
|
||||
SourceID: "source-1",
|
||||
Index: 0,
|
||||
StartUnitID: 1,
|
||||
EndUnitID: 1,
|
||||
Content: []byte("chunk content"),
|
||||
MediaType: "text/plain",
|
||||
Units: doc.Units,
|
||||
ID: "chunk-1",
|
||||
SourceID: "source-1",
|
||||
Index: 0,
|
||||
Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1},
|
||||
Content: []byte("chunk content"),
|
||||
MediaType: "text/plain",
|
||||
Units: doc.Units,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -80,96 +79,43 @@ func TestWorkspaceRecorderWritesSuccessfulCheckpointFiles(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestWorkspaceLoaderReusesSuccessfulCheckpointFiles(t *testing.T) {
|
||||
func TestWorkspaceArtifactCheckpointsRoundTripCodecIdentityAndBytes(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
recorder := newTestRecorder(t, root)
|
||||
loader := &WorkspaceLoader{root: root}
|
||||
doc := &source.SourceDocument{
|
||||
ID: "source-1",
|
||||
Kind: "document",
|
||||
Format: "text/plain",
|
||||
Digest: "sha256:source",
|
||||
Units: []source.SourceUnit{{ID: 1, Kind: "line", Text: "hello"}},
|
||||
}
|
||||
chunks := []contracts.SourceChunk{
|
||||
{
|
||||
ID: "chunk-1",
|
||||
SourceID: "source-1",
|
||||
Index: 0,
|
||||
StartUnitID: 1,
|
||||
EndUnitID: 1,
|
||||
Content: []byte("chunk content"),
|
||||
MediaType: "text/plain",
|
||||
Units: doc.Units,
|
||||
},
|
||||
}
|
||||
extractOutput := contracts.ExtractOutput{
|
||||
LaneID: "spells",
|
||||
ExtractorKey: "dnd/spells",
|
||||
SourceID: doc.ID,
|
||||
ChunkID: "chunk-1",
|
||||
Payload: contracts.RawPayload{
|
||||
Content: []byte(`{"spell":"cure wounds"}`),
|
||||
MediaType: "application/json",
|
||||
},
|
||||
}
|
||||
mergeOutput := contracts.MergeOutput{
|
||||
LaneID: "spells",
|
||||
MergerKey: "appendorder",
|
||||
SourceID: doc.ID,
|
||||
Payload: contracts.RawPayload{
|
||||
Content: []byte(`{"merged":true}`),
|
||||
MediaType: "application/json",
|
||||
},
|
||||
}
|
||||
normalizeOutput := contracts.NormalizeOutput{
|
||||
LaneID: "spells",
|
||||
NormalizerKey: "noop",
|
||||
SourceID: doc.ID,
|
||||
Payload: contracts.RawPayload{
|
||||
Content: []byte(`{"normalized":true}`),
|
||||
MediaType: "application/json",
|
||||
},
|
||||
}
|
||||
schema := contracts.ArtifactSchema{ID: "dnd.spell_response", Name: "spell response", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
|
||||
artifact := contracts.SerializedArtifact{Kind: "dnd.spells", Schema: schema, MediaType: "application/json", Content: []byte(`{"spell_casts":[]}`), Metadata: map[string]any{"spell_cast_count": float64(0)}}
|
||||
stored := pipeline.CheckpointArtifact{LaneID: "spells", ModuleKey: "dnd/spells", SourceID: "source-1", ChunkID: "chunk-1", ChunkIndex: 2, ChunkRef: source.SourceRef{SourceID: "source-1", StartUnitID: 4, EndUnitID: 8}, Artifact: artifact, SchemaDigest: contracts.DigestArtifactSchema(schema)}
|
||||
|
||||
if err := recorder.SourceSucceeded("seriatim", doc); err != nil {
|
||||
t.Fatalf("SourceSucceeded: %v", err)
|
||||
}
|
||||
if err := recorder.ChunkSucceeded("generic", doc.Digest, chunks, nil); err != nil {
|
||||
t.Fatalf("ChunkSucceeded: %v", err)
|
||||
}
|
||||
extractDeps := []pipeline.CheckpointFingerprint{{Name: "chunks", Value: "sha256:chunks"}}
|
||||
if err := recorder.ExtractSucceeded("spells", "dnd/spells", extractDeps, []contracts.ExtractOutput{extractOutput}, nil, nil); err != nil {
|
||||
if err := recorder.ExtractSucceeded("spells", "dnd/spells", extractDeps, []pipeline.CheckpointArtifact{stored}, nil, nil); err != nil {
|
||||
t.Fatalf("ExtractSucceeded: %v", err)
|
||||
}
|
||||
mergeDeps := rawOutputDigests([]contracts.RawPayload{extractOutput.Payload})
|
||||
if err := recorder.MergeSucceeded("spells", "appendorder", mergeDeps, mergeOutput, nil); err != nil {
|
||||
t.Fatalf("MergeSucceeded: %v", err)
|
||||
extracted, decision := loader.Extract("spells", "dnd/spells", extractDeps)
|
||||
if !decision.Reused || len(extracted.Outputs) != 1 {
|
||||
t.Fatalf("extract decision=%#v checkpoint=%#v, want reused", decision, extracted)
|
||||
}
|
||||
normalizeDeps := rawOutputDigests([]contracts.RawPayload{mergeOutput.Payload})
|
||||
if err := recorder.NormalizeSucceeded("spells", "noop", normalizeDeps, normalizeOutput, nil); err != nil {
|
||||
t.Fatalf("NormalizeSucceeded: %v", err)
|
||||
got := extracted.Outputs[0]
|
||||
if got.Artifact.Kind != artifact.Kind || got.Artifact.Schema.ID != schema.ID || got.Artifact.Schema.Version != schema.Version || got.SchemaDigest != stored.SchemaDigest || string(got.Artifact.Content) != string(artifact.Content) || got.ChunkRef != stored.ChunkRef {
|
||||
t.Fatalf("artifact checkpoint = %#v, want codec identity, bytes, and provenance", got)
|
||||
}
|
||||
|
||||
sourceCheckpoint, decision := loader.Source("seriatim")
|
||||
if !decision.Reused || sourceCheckpoint.Document.ID != "source-1" {
|
||||
t.Fatalf("source decision = %#v checkpoint=%#v, want reused", decision, sourceCheckpoint)
|
||||
mergeDeps := artifactOutputDigests([]pipeline.CheckpointArtifact{stored})
|
||||
if err := recorder.MergeSucceeded("spells", "merge", mergeDeps, stored, nil); err != nil {
|
||||
t.Fatalf("MergeSucceeded: %v", err)
|
||||
}
|
||||
chunkCheckpoint, decision := loader.Chunk("generic", doc.Digest)
|
||||
if !decision.Reused || len(chunkCheckpoint.Chunks) != 1 || string(chunkCheckpoint.Chunks[0].Content) != "chunk content" {
|
||||
t.Fatalf("chunk decision = %#v checkpoint=%#v, want reused", decision, chunkCheckpoint)
|
||||
merged, decision := loader.Merge("spells", "merge", mergeDeps)
|
||||
if !decision.Reused || string(merged.Output.Artifact.Content) != string(artifact.Content) {
|
||||
t.Fatalf("merge decision=%#v checkpoint=%#v, want reused", decision, merged)
|
||||
}
|
||||
extractCheckpoint, decision := loader.Extract("spells", "dnd/spells", extractDeps)
|
||||
if !decision.Reused || len(extractCheckpoint.Outputs) != 1 || string(extractCheckpoint.Outputs[0].Payload.Content) != `{"spell":"cure wounds"}` {
|
||||
t.Fatalf("extract decision = %#v checkpoint=%#v, want reused", decision, extractCheckpoint)
|
||||
|
||||
normalizeDeps := artifactOutputDigests([]pipeline.CheckpointArtifact{stored})
|
||||
if err := recorder.NormalizeSucceeded("spells", "normalize", normalizeDeps, stored, nil); err != nil {
|
||||
t.Fatalf("NormalizeSucceeded: %v", err)
|
||||
}
|
||||
mergeCheckpoint, decision := loader.Merge("spells", "appendorder", mergeDeps)
|
||||
if !decision.Reused || string(mergeCheckpoint.Output.Payload.Content) != `{"merged":true}` {
|
||||
t.Fatalf("merge decision = %#v checkpoint=%#v, want reused", decision, mergeCheckpoint)
|
||||
}
|
||||
normalizeCheckpoint, decision := loader.Normalize("spells", "noop", normalizeDeps)
|
||||
if !decision.Reused || string(normalizeCheckpoint.Output.Payload.Content) != `{"normalized":true}` {
|
||||
t.Fatalf("normalize decision = %#v checkpoint=%#v, want reused", decision, normalizeCheckpoint)
|
||||
normalized, decision := loader.Normalize("spells", "normalize", normalizeDeps)
|
||||
if !decision.Reused || normalized.Output.SchemaDigest != stored.SchemaDigest {
|
||||
t.Fatalf("normalize decision=%#v checkpoint=%#v, want reused", decision, normalized)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -184,7 +130,7 @@ func TestWorkspaceLoaderInvalidatesMissingCorruptAndMismatchedCheckpoints(t *tes
|
||||
t.Run("dependency mismatch", func(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
recorder := newTestRecorder(t, root)
|
||||
chunks := []contracts.SourceChunk{{
|
||||
chunks := []source.Chunk{{
|
||||
ID: "chunk-1",
|
||||
SourceID: "source-1",
|
||||
Content: []byte("chunk content"),
|
||||
@@ -199,10 +145,41 @@ func TestWorkspaceLoaderInvalidatesMissingCorruptAndMismatchedCheckpoints(t *tes
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("incompatible workspace schema remains untouched", func(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
recorder := newTestRecorder(t, root)
|
||||
doc := &source.SourceDocument{
|
||||
ID: "source-1", Kind: "document", Format: "text/plain", Digest: "sha256:source",
|
||||
Units: []source.SourceUnit{{ID: 1, Kind: "line", Text: "hello", Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1}}},
|
||||
}
|
||||
if err := recorder.SourceSucceeded("seriatim", doc); err != nil {
|
||||
t.Fatalf("SourceSucceeded: %v", err)
|
||||
}
|
||||
manifestPath := filepath.Join(root, "source", "manifest.json")
|
||||
manifest := strings.Replace(string(readFile(t, manifestPath)), coreworkspace.WorkspaceSchemaVersion, coreworkspace.WorkspaceSchemaVersionV1, 1)
|
||||
if err := os.WriteFile(manifestPath, []byte(manifest), 0o644); err != nil {
|
||||
t.Fatalf("write legacy manifest: %v", err)
|
||||
}
|
||||
beforeManifest := readFile(t, manifestPath)
|
||||
payloadPath := filepath.Join(root, "source", "source-document.json")
|
||||
beforePayload := readFile(t, payloadPath)
|
||||
|
||||
loader := &WorkspaceLoader{root: root}
|
||||
if _, decision := loader.Source("seriatim"); decision.Reused || !strings.Contains(decision.Reason, "incompatible") || !strings.Contains(decision.Reason, coreworkspace.WorkspaceSchemaVersionV1) {
|
||||
t.Fatalf("decision = %#v, want incompatible legacy schema invalidation", decision)
|
||||
}
|
||||
if got := readFile(t, manifestPath); string(got) != string(beforeManifest) {
|
||||
t.Fatal("legacy manifest changed during reuse decision")
|
||||
}
|
||||
if got := readFile(t, payloadPath); string(got) != string(beforePayload) {
|
||||
t.Fatal("legacy payload changed during reuse decision")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("corrupt payload", func(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
recorder := newTestRecorder(t, root)
|
||||
chunks := []contracts.SourceChunk{{
|
||||
chunks := []source.Chunk{{
|
||||
ID: "chunk-1",
|
||||
SourceID: "source-1",
|
||||
Content: []byte("chunk content"),
|
||||
@@ -286,23 +263,20 @@ func TestWorkspaceRecorderRecordsFailedStages(t *testing.T) {
|
||||
func TestWorkspaceRecorderRecordsWarningOnlyValidation(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
recorder := newTestRecorder(t, root)
|
||||
output := contracts.NormalizeOutput{
|
||||
LaneID: "spells",
|
||||
NormalizerKey: "noop",
|
||||
SourceID: "source-1",
|
||||
Payload: contracts.RawPayload{
|
||||
Content: []byte(`{"ok":true}`),
|
||||
MediaType: "application/json",
|
||||
},
|
||||
schema := contracts.ArtifactSchema{ID: "test.artifact", Name: "test_artifact", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
|
||||
output := pipeline.CheckpointArtifact{
|
||||
LaneID: "events", ModuleKey: "noop", SourceID: "source-1",
|
||||
Artifact: contracts.SerializedArtifact{Kind: "test/artifact", Schema: schema, MediaType: "application/json", Content: []byte(`{"ok":true}`)},
|
||||
SchemaDigest: contracts.DigestArtifactSchema(schema),
|
||||
}
|
||||
warnings := []contracts.Warning{{ReasonCode: "note", Message: "warning"}}
|
||||
|
||||
if err := recorder.NormalizeSucceeded("spells", "noop", nil, output, warnings); err != nil {
|
||||
if err := recorder.NormalizeSucceeded("events", "noop", nil, output, warnings); err != nil {
|
||||
t.Fatalf("NormalizeSucceeded: %v", err)
|
||||
}
|
||||
|
||||
var manifest coreworkspace.NormalizeLaneManifest
|
||||
readJSON(t, filepath.Join(root, "normalize", "spells", "manifest.json"), &manifest)
|
||||
readJSON(t, filepath.Join(root, "normalize", "events", "manifest.json"), &manifest)
|
||||
if manifest.Status != coreworkspace.StatusSucceeded || manifest.ValidationStatus != "approved_with_warnings" {
|
||||
t.Fatalf("normalize manifest status = %q validation=%q", manifest.Status, manifest.ValidationStatus)
|
||||
}
|
||||
|
||||
81
internal/framework/contracts/artifact.go
Normal file
81
internal/framework/contracts/artifact.go
Normal file
@@ -0,0 +1,81 @@
|
||||
package contracts
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
)
|
||||
|
||||
// ArtifactKind is the stable logical identity of a domain artifact.
|
||||
type ArtifactKind string
|
||||
|
||||
// ArtifactSchema describes the durable representation owned by an artifact
|
||||
// codec. JSONSchema is cloned whenever framework ownership changes.
|
||||
type ArtifactSchema struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version"`
|
||||
JSONSchema []byte `json:"-"`
|
||||
}
|
||||
|
||||
// SerializedArtifact is the domain-neutral representation of a typed
|
||||
// artifact at an explicit serialization boundary.
|
||||
type SerializedArtifact struct {
|
||||
Kind ArtifactKind `json:"kind"`
|
||||
Schema ArtifactSchema `json:"schema"`
|
||||
MediaType string `json:"media_type"`
|
||||
Content []byte `json:"-"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
// SerializedOutput associates a domain-neutral artifact with the pipeline
|
||||
// operation that produced it. Provenance remains outside codec-owned bytes.
|
||||
type SerializedOutput struct {
|
||||
LaneID string `json:"lane_id"`
|
||||
NormalizerKey string `json:"normalizer_key"`
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
Artifact SerializedArtifact `json:"artifact"`
|
||||
}
|
||||
|
||||
// ArtifactCodec owns the stable encoding for one concrete artifact type.
|
||||
type ArtifactCodec[T any] interface {
|
||||
Kind() ArtifactKind
|
||||
Schema() ArtifactSchema
|
||||
MediaType() string
|
||||
Encode(T) ([]byte, error)
|
||||
Decode([]byte) (T, error)
|
||||
}
|
||||
|
||||
// DigestArtifactSchema returns the SHA-256 digest of the exact JSON Schema
|
||||
// bytes. Schema formatting is therefore part of the registered identity.
|
||||
func DigestArtifactSchema(schema ArtifactSchema) string {
|
||||
sum := sha256.Sum256(schema.JSONSchema)
|
||||
return "sha256:" + hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
func CloneArtifactSchema(schema ArtifactSchema) ArtifactSchema {
|
||||
schema.JSONSchema = append([]byte(nil), schema.JSONSchema...)
|
||||
return schema
|
||||
}
|
||||
|
||||
func CloneSerializedArtifact(artifact SerializedArtifact) SerializedArtifact {
|
||||
artifact.Schema = CloneArtifactSchema(artifact.Schema)
|
||||
artifact.Content = append([]byte(nil), artifact.Content...)
|
||||
artifact.Metadata = cloneArtifactMetadata(artifact.Metadata)
|
||||
return artifact
|
||||
}
|
||||
|
||||
func CloneSerializedOutput(output SerializedOutput) SerializedOutput {
|
||||
output.Artifact = CloneSerializedArtifact(output.Artifact)
|
||||
return output
|
||||
}
|
||||
|
||||
func cloneArtifactMetadata(metadata map[string]any) map[string]any {
|
||||
if len(metadata) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]any, len(metadata))
|
||||
for key, value := range metadata {
|
||||
out[key] = value
|
||||
}
|
||||
return out
|
||||
}
|
||||
45
internal/framework/contracts/artifact_test.go
Normal file
45
internal/framework/contracts/artifact_test.go
Normal file
@@ -0,0 +1,45 @@
|
||||
package contracts
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestDigestArtifactSchemaUsesExactBytes(t *testing.T) {
|
||||
first := ArtifactSchema{JSONSchema: []byte(`{"type":"object"}`)}
|
||||
second := ArtifactSchema{JSONSchema: []byte("{\n \"type\": \"object\"\n}")}
|
||||
|
||||
if got := DigestArtifactSchema(first); got != "sha256:a2c799262a3ce3c19ef5cdd983bf3d12b43ab3c426227091b909dcb7054738c0" {
|
||||
t.Fatalf("DigestArtifactSchema() = %q, want stable SHA-256", got)
|
||||
}
|
||||
if DigestArtifactSchema(first) == DigestArtifactSchema(second) {
|
||||
t.Fatal("schema digests match for different exact bytes")
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCloneHelpersOwnSlicesAndMaps(t *testing.T) {
|
||||
schema := ArtifactSchema{ID: "notes.v1", Name: "notes", Version: "v1", JSONSchema: []byte(`{"type":"object"}`)}
|
||||
artifact := SerializedArtifact{
|
||||
Kind: "test/notes",
|
||||
Schema: schema,
|
||||
MediaType: "application/json",
|
||||
Content: []byte(`{"items":["one"]}`),
|
||||
Metadata: map[string]any{"origin": "test"},
|
||||
}
|
||||
|
||||
clonedSchema := CloneArtifactSchema(schema)
|
||||
cloned := CloneSerializedArtifact(artifact)
|
||||
schema.JSONSchema[0] = '['
|
||||
artifact.Content[0] = '['
|
||||
artifact.Metadata["origin"] = "changed"
|
||||
|
||||
if string(clonedSchema.JSONSchema) != `{"type":"object"}` {
|
||||
t.Fatalf("cloned schema = %q, want original bytes", clonedSchema.JSONSchema)
|
||||
}
|
||||
if string(cloned.Schema.JSONSchema) != `{"type":"object"}` {
|
||||
t.Fatalf("serialized artifact schema = %q, want original bytes", cloned.Schema.JSONSchema)
|
||||
}
|
||||
if string(cloned.Content) != `{"items":["one"]}` {
|
||||
t.Fatalf("cloned content = %q, want original bytes", cloned.Content)
|
||||
}
|
||||
if cloned.Metadata["origin"] != "test" {
|
||||
t.Fatalf("cloned metadata = %#v, want independent map", cloned.Metadata)
|
||||
}
|
||||
}
|
||||
@@ -1,298 +0,0 @@
|
||||
package contracts_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
var _ contracts.InputAdapter = compositionAdapter{}
|
||||
var _ contracts.Chunker = compositionChunker{}
|
||||
var _ contracts.Extractor = compositionExtractor{}
|
||||
var _ contracts.Merger = compositionMerger{}
|
||||
var _ contracts.Normalizer = compositionNormalizer{}
|
||||
var _ contracts.Validator = compositionValidator{}
|
||||
var _ contracts.StructuredLLMClient = compositionLLMClient{}
|
||||
var _ contracts.OutputEncoder = compositionOutputEncoder{}
|
||||
|
||||
func TestContractsComposeAcrossPackages(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
adapter := compositionAdapter{}
|
||||
chunker := compositionChunker{}
|
||||
extractor := compositionExtractor{}
|
||||
merger := compositionMerger{}
|
||||
normalizer := compositionNormalizer{}
|
||||
encoder := compositionOutputEncoder{}
|
||||
|
||||
doc, err := adapter.Parse(ctx, contracts.ParseRequest{SourceID: "source-1"})
|
||||
if err != nil {
|
||||
t.Fatalf("Parse() error = %v, want nil", err)
|
||||
}
|
||||
if err := source.ValidateDocument(doc); err != nil {
|
||||
t.Fatalf("ValidateDocument() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
chunking, err := chunker.Chunk(ctx, contracts.ChunkRequest{
|
||||
Source: doc,
|
||||
LLMClient: compositionLLMClient{},
|
||||
Metadata: map[string]any{"max_units": 2},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Chunk() error = %v, want nil", err)
|
||||
}
|
||||
if len(chunking.Chunks) != 1 {
|
||||
t.Fatalf("len(Chunks) = %d, want 1", len(chunking.Chunks))
|
||||
}
|
||||
|
||||
extraction, err := extractor.Extract(ctx, contracts.ExtractionRequest{
|
||||
Source: doc,
|
||||
Chunk: &chunking.Chunks[0],
|
||||
AmbientContext: map[string]any{"synopsis": "example synopsis"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Extract() error = %v, want nil", err)
|
||||
}
|
||||
if extraction.Output.Payload.MediaType != "application/json" {
|
||||
t.Fatalf("extract media type = %q, want application/json", extraction.Output.Payload.MediaType)
|
||||
}
|
||||
|
||||
merge, err := merger.Merge(ctx, contracts.MergeRequest{
|
||||
Source: doc,
|
||||
LaneID: "generic-lane",
|
||||
ExtractOutputs: []contracts.ExtractOutput{extraction.Output},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Merge() error = %v, want nil", err)
|
||||
}
|
||||
if string(merge.Output.Payload.Content) != `{"value":"example"}` {
|
||||
t.Fatalf("merge output = %s, want extract payload", merge.Output.Payload.Content)
|
||||
}
|
||||
|
||||
normalize, err := normalizer.Normalize(ctx, contracts.NormalizeRequest{
|
||||
Source: doc,
|
||||
LaneID: "generic-lane",
|
||||
MergeOutput: merge.Output,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Normalize() error = %v, want nil", err)
|
||||
}
|
||||
if string(normalize.Output.Payload.Content) != `{"value":"example"}` {
|
||||
t.Fatalf("normalize output = %s, want merge payload", normalize.Output.Payload.Content)
|
||||
}
|
||||
|
||||
output, err := encoder.Encode(ctx, contracts.OutputRequest{
|
||||
Manifest: artifacts.RunManifest{RunID: "run-1"},
|
||||
NormalizeOutputs: []contracts.NormalizeOutput{normalize.Output},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Encode() error = %v, want nil", err)
|
||||
}
|
||||
if len(output.Files) != 1 {
|
||||
t.Fatalf("len(Files) = %d, want 1", len(output.Files))
|
||||
}
|
||||
if output.Files[0].ContentType != "application/json" {
|
||||
t.Fatalf("ContentType = %q, want application/json", output.Files[0].ContentType)
|
||||
}
|
||||
if len(output.Files[0].Bytes) == 0 {
|
||||
t.Fatal("len(Bytes) = 0, want encoded bytes")
|
||||
}
|
||||
}
|
||||
|
||||
type compositionAdapter struct{}
|
||||
|
||||
func (adapter compositionAdapter) Key() string {
|
||||
return "generic-input"
|
||||
}
|
||||
|
||||
func (adapter compositionAdapter) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
|
||||
return &source.SourceDocument{
|
||||
ID: req.SourceID,
|
||||
Kind: "document",
|
||||
Format: "text/plain",
|
||||
Digest: "sha256:abc123",
|
||||
Units: []source.SourceUnit{
|
||||
{ID: 1, Kind: "unit", Text: "First source unit."},
|
||||
{ID: 2, Kind: "unit", Text: "Second source unit."},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type compositionChunker struct{}
|
||||
|
||||
func (chunker compositionChunker) Key() string {
|
||||
return "generic-chunker"
|
||||
}
|
||||
|
||||
func (chunker compositionChunker) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (chunker compositionChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
|
||||
if req.Source == nil {
|
||||
return contracts.ChunkResult{}, errors.New("source document is required")
|
||||
}
|
||||
if req.LLMClient == nil {
|
||||
return contracts.ChunkResult{}, errors.New("structured llm client is required")
|
||||
}
|
||||
|
||||
return contracts.ChunkResult{
|
||||
Chunks: []contracts.SourceChunk{
|
||||
{
|
||||
ID: req.Source.ID + ":chunk:0",
|
||||
SourceID: req.Source.ID,
|
||||
Index: 0,
|
||||
StartUnitID: req.Source.Units[0].ID,
|
||||
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
|
||||
Content: []byte(`{"units":[{"id":1,"kind":"unit","text":"First source unit."},{"id":2,"kind":"unit","text":"Second source unit."}]}`),
|
||||
MediaType: "application/json",
|
||||
Units: append([]source.SourceUnit(nil), req.Source.Units...),
|
||||
Metadata: map[string]any{"strategy": "whole-document"},
|
||||
},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type compositionLLMClient struct{}
|
||||
|
||||
func (client compositionLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
return contracts.StructuredCompletionResponse{}, nil
|
||||
}
|
||||
|
||||
type compositionExtractor struct{}
|
||||
|
||||
func (extractor compositionExtractor) Key() string {
|
||||
return "generic-extractor"
|
||||
}
|
||||
|
||||
func (extractor compositionExtractor) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (extractor compositionExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
|
||||
if req.Source == nil {
|
||||
return contracts.ExtractionResult{}, errors.New("source document is required")
|
||||
}
|
||||
if req.AmbientContext["synopsis"] == "" {
|
||||
return contracts.ExtractionResult{}, errors.New("ambient synopsis is required")
|
||||
}
|
||||
|
||||
return contracts.ExtractionResult{
|
||||
Output: contracts.ExtractOutput{
|
||||
Schema: contracts.ResponseSchema{ID: "schema-id", Name: "schema-name", Version: "v1"},
|
||||
Payload: contracts.RawPayload{
|
||||
Content: []byte(`{"value":"example"}`),
|
||||
MediaType: "application/json",
|
||||
},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type compositionMerger struct{}
|
||||
|
||||
func (merger compositionMerger) Key() string {
|
||||
return "generic-merger"
|
||||
}
|
||||
|
||||
func (merger compositionMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
|
||||
output := req.ExtractOutputs[0]
|
||||
return contracts.MergeResult{Output: contracts.MergeOutput{
|
||||
LaneID: req.LaneID,
|
||||
MergerKey: merger.Key(),
|
||||
SourceID: output.SourceID,
|
||||
Schema: output.Schema,
|
||||
Payload: cloneCompositionPayload(output.Payload),
|
||||
}}, nil
|
||||
}
|
||||
|
||||
type compositionNormalizer struct{}
|
||||
|
||||
func (normalizer compositionNormalizer) Key() string {
|
||||
return "generic-normalizer"
|
||||
}
|
||||
|
||||
func (normalizer compositionNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (normalizer compositionNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
|
||||
return contracts.NormalizeResult{Output: contracts.NormalizeOutput{
|
||||
LaneID: req.LaneID,
|
||||
NormalizerKey: normalizer.Key(),
|
||||
SourceID: req.MergeOutput.SourceID,
|
||||
Schema: req.MergeOutput.Schema,
|
||||
Payload: cloneCompositionPayload(req.MergeOutput.Payload),
|
||||
}}, nil
|
||||
}
|
||||
|
||||
func cloneCompositionPayload(payload contracts.RawPayload) contracts.RawPayload {
|
||||
return contracts.RawPayload{
|
||||
Content: append([]byte(nil), payload.Content...),
|
||||
MediaType: payload.MediaType,
|
||||
Metadata: cloneCompositionMetadata(payload.Metadata),
|
||||
Warnings: append([]contracts.Warning(nil), payload.Warnings...),
|
||||
}
|
||||
}
|
||||
|
||||
func cloneCompositionMetadata(metadata map[string]any) map[string]any {
|
||||
if len(metadata) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]any, len(metadata))
|
||||
for key, value := range metadata {
|
||||
out[key] = value
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
type compositionValidator struct{}
|
||||
|
||||
func (validator compositionValidator) Name() string {
|
||||
return "generic-validator"
|
||||
}
|
||||
|
||||
func (validator compositionValidator) ExecutionClass() contracts.ExecutionClass {
|
||||
return contracts.ExecutionClassDeterministic
|
||||
}
|
||||
|
||||
func (validator compositionValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{
|
||||
Approved: true,
|
||||
ReasonCode: "accepted",
|
||||
Message: "output accepted",
|
||||
}, nil
|
||||
}
|
||||
|
||||
type compositionOutputEncoder struct{}
|
||||
|
||||
func (encoder compositionOutputEncoder) Key() string {
|
||||
return "generic-output"
|
||||
}
|
||||
|
||||
func (encoder compositionOutputEncoder) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
|
||||
payload := struct {
|
||||
RunID string `json:"run_id"`
|
||||
OutputCount int `json:"output_count"`
|
||||
}{
|
||||
RunID: req.Manifest.RunID,
|
||||
OutputCount: len(req.NormalizeOutputs),
|
||||
}
|
||||
encoded, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return contracts.OutputResult{}, err
|
||||
}
|
||||
|
||||
return contracts.OutputResult{
|
||||
Files: []contracts.OutputFile{
|
||||
{
|
||||
Name: "artifacts/generic.json",
|
||||
ContentType: "application/json",
|
||||
Bytes: encoded,
|
||||
},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
@@ -129,7 +129,6 @@ type ParseRequest struct {
|
||||
Path string `json:"path,omitempty"`
|
||||
Raw []byte `json:"-"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
@@ -138,32 +137,18 @@ type InputAdapter interface {
|
||||
Parse(ctx context.Context, req ParseRequest) (*source.SourceDocument, error)
|
||||
}
|
||||
|
||||
type SourceChunk struct {
|
||||
ID string `json:"id"`
|
||||
SourceID string `json:"source_id"`
|
||||
Index int `json:"index"`
|
||||
StartUnitID int `json:"start_unit_id"`
|
||||
EndUnitID int `json:"end_unit_id"`
|
||||
Content []byte `json:"-"`
|
||||
MediaType string `json:"media_type"`
|
||||
Units []source.SourceUnit `json:"units"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
type ChunkRequest struct {
|
||||
Source *source.SourceDocument `json:"-"`
|
||||
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
References ReferenceSet `json:"references,omitempty"`
|
||||
LLMClient StructuredLLMClient `json:"-"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
type ChunkResult struct {
|
||||
Chunks []SourceChunk `json:"chunks"`
|
||||
Warnings []Warning `json:"warnings,omitempty"`
|
||||
Chunks []source.Chunk `json:"chunks"`
|
||||
Warnings []Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
type Chunker interface {
|
||||
@@ -222,37 +207,6 @@ type ReferenceSet struct {
|
||||
Slots map[string]ResolvedReferenceSlot `json:"slots,omitempty"`
|
||||
}
|
||||
|
||||
type ExtractionRequest struct {
|
||||
Source *source.SourceDocument `json:"-"`
|
||||
Chunk *SourceChunk `json:"chunk,omitempty"`
|
||||
AmbientContext map[string]any `json:"ambient_context,omitempty"`
|
||||
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
References ReferenceSet `json:"references,omitempty"`
|
||||
LLMClient StructuredLLMClient `json:"-"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
type ExtractionResult struct {
|
||||
Output ExtractOutput `json:"output"`
|
||||
Warnings []Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
type Extractor interface {
|
||||
Key() string
|
||||
ReferenceSlots() []ReferenceSlot
|
||||
Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error)
|
||||
}
|
||||
|
||||
type RawPayload struct {
|
||||
Content []byte `json:"-"`
|
||||
MediaType string `json:"media_type"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
Warnings []Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
type ExecutionClass string
|
||||
|
||||
const (
|
||||
@@ -260,29 +214,6 @@ const (
|
||||
ExecutionClassLLMBacked ExecutionClass = "llm_backed"
|
||||
)
|
||||
|
||||
type ValidationRequest struct {
|
||||
Stage string `json:"stage"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key"`
|
||||
Source *source.SourceDocument `json:"-"`
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
References ReferenceSet `json:"references,omitempty"`
|
||||
LLMClient StructuredLLMClient `json:"-"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
Schema ResponseSchema `json:"schema,omitempty"`
|
||||
Payload RawPayload `json:"payload"`
|
||||
ChunkID string `json:"chunk_id,omitempty"`
|
||||
ChunkIndex int `json:"chunk_index,omitempty"`
|
||||
Chunk *SourceChunk `json:"chunk,omitempty"`
|
||||
Chunks []SourceChunk `json:"chunks,omitempty"`
|
||||
ExtractOutputs []ExtractOutput `json:"extract_outputs,omitempty"`
|
||||
MergeOutput MergeOutput `json:"merge_output,omitempty"`
|
||||
}
|
||||
|
||||
type ValidationResult struct {
|
||||
Approved bool `json:"approved"`
|
||||
ReasonCode string `json:"reason_code,omitempty"`
|
||||
@@ -291,92 +222,6 @@ type ValidationResult struct {
|
||||
Warnings []Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
type Validator interface {
|
||||
Name() string
|
||||
ExecutionClass() ExecutionClass
|
||||
Validate(ctx context.Context, req ValidationRequest) (ValidationResult, error)
|
||||
}
|
||||
|
||||
type ResponseSchema struct {
|
||||
ID string `json:"id,omitempty"`
|
||||
Name string `json:"name,omitempty"`
|
||||
Version string `json:"version,omitempty"`
|
||||
JSONSchema []byte `json:"-"`
|
||||
}
|
||||
|
||||
type ExtractOutput struct {
|
||||
LaneID string `json:"lane_id"`
|
||||
ExtractorKey string `json:"extractor_key"`
|
||||
SourceID string `json:"source_id"`
|
||||
ChunkID string `json:"chunk_id"`
|
||||
ChunkIndex int `json:"chunk_index"`
|
||||
Schema ResponseSchema `json:"schema,omitempty"`
|
||||
Payload RawPayload `json:"payload"`
|
||||
}
|
||||
|
||||
type MergeRequest struct {
|
||||
Source *source.SourceDocument `json:"-"`
|
||||
LaneID string `json:"lane_id"`
|
||||
ExtractOutputs []ExtractOutput `json:"extract_outputs"`
|
||||
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
References ReferenceSet `json:"references,omitempty"`
|
||||
LLMClient StructuredLLMClient `json:"-"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
type MergeResult struct {
|
||||
Output MergeOutput `json:"output"`
|
||||
Warnings []Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
type MergeOutput struct {
|
||||
LaneID string `json:"lane_id"`
|
||||
MergerKey string `json:"merger_key"`
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
Schema ResponseSchema `json:"schema,omitempty"`
|
||||
Payload RawPayload `json:"payload"`
|
||||
}
|
||||
|
||||
type Merger interface {
|
||||
Key() string
|
||||
Merge(ctx context.Context, req MergeRequest) (MergeResult, error)
|
||||
}
|
||||
|
||||
type NormalizeRequest struct {
|
||||
Source *source.SourceDocument `json:"-"`
|
||||
LaneID string `json:"lane_id"`
|
||||
MergeOutput MergeOutput `json:"merge_output"`
|
||||
SourceInput LLMInputMaterial `json:"source_input,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
References ReferenceSet `json:"references,omitempty"`
|
||||
LLMClient StructuredLLMClient `json:"-"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
type NormalizeResult struct {
|
||||
Output NormalizeOutput `json:"output"`
|
||||
Warnings []Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
type NormalizeOutput struct {
|
||||
LaneID string `json:"lane_id"`
|
||||
NormalizerKey string `json:"normalizer_key"`
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
Schema ResponseSchema `json:"schema,omitempty"`
|
||||
Payload RawPayload `json:"payload"`
|
||||
}
|
||||
|
||||
type Normalizer interface {
|
||||
Key() string
|
||||
ReferenceSlots() []ReferenceSlot
|
||||
Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error)
|
||||
}
|
||||
|
||||
type Warning struct {
|
||||
Scope string `json:"scope,omitempty"`
|
||||
ReasonCode string `json:"reason_code"`
|
||||
@@ -385,11 +230,10 @@ type Warning struct {
|
||||
|
||||
type OutputRequest struct {
|
||||
Manifest artifacts.RunManifest `json:"manifest"`
|
||||
NormalizeOutputs []NormalizeOutput `json:"normalize_outputs,omitempty"`
|
||||
NormalizeOutputs []SerializedOutput `json:"normalize_outputs,omitempty"`
|
||||
Rejected []RejectedOutput `json:"rejected,omitempty"`
|
||||
Warnings []Warning `json:"warnings,omitempty"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
|
||||
@@ -12,14 +12,13 @@ import (
|
||||
|
||||
var _ InputAdapter = fakeAdapter{}
|
||||
var _ Chunker = fakeChunker{}
|
||||
var _ Extractor = fakeExtractor{}
|
||||
var _ Merger = fakeMerger{}
|
||||
var _ Normalizer = fakeNormalizer{}
|
||||
var _ Validator = fakeValidator{}
|
||||
var _ Extractor[fakeArtifact] = fakeExtractor{}
|
||||
var _ Merger[fakeArtifact] = fakeMerger{}
|
||||
var _ Normalizer[fakeArtifact] = fakeNormalizer{}
|
||||
var _ StructuredLLMClient = fakeLLMClient{}
|
||||
var _ OutputEncoder = fakeOutputEncoder{}
|
||||
|
||||
func TestFakeExtractorReturnsRawOutput(t *testing.T) {
|
||||
func TestFakeExtractorReturnsTypedOutput(t *testing.T) {
|
||||
extractor := fakeExtractor{
|
||||
key: "generic-extractor",
|
||||
}
|
||||
@@ -33,7 +32,7 @@ func TestFakeExtractorReturnsRawOutput(t *testing.T) {
|
||||
},
|
||||
}
|
||||
|
||||
result, err := extractor.Extract(context.Background(), ExtractionRequest{Source: doc})
|
||||
result, err := extractor.Extract(context.Background(), TypedExtractionRequest{Source: doc})
|
||||
if err != nil {
|
||||
t.Fatalf("Extract() error = %v, want nil", err)
|
||||
}
|
||||
@@ -41,14 +40,8 @@ func TestFakeExtractorReturnsRawOutput(t *testing.T) {
|
||||
if extractor.Key() != "generic-extractor" {
|
||||
t.Fatalf("Key() = %q, want generic-extractor", extractor.Key())
|
||||
}
|
||||
if result.Output.ExtractorKey != "" {
|
||||
t.Fatalf("ExtractorKey = %q, want runner-owned empty value", result.Output.ExtractorKey)
|
||||
}
|
||||
if result.Output.Schema.Version != "v1" {
|
||||
t.Fatalf("Schema.Version = %q, want v1", result.Output.Schema.Version)
|
||||
}
|
||||
if result.Output.Payload.MediaType != "application/json" || string(result.Output.Payload.Content) != `{"value":"example"}` {
|
||||
t.Fatalf("payload = %q %s, want JSON raw output", result.Output.Payload.MediaType, result.Output.Payload.Content)
|
||||
if result.Value.Value != "example" {
|
||||
t.Fatalf("Value = %q, want example", result.Value.Value)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -78,26 +71,26 @@ func TestFakeChunkerReturnsSourceChunks(t *testing.T) {
|
||||
|
||||
chunk := result.Chunks[0]
|
||||
if chunk.ID != "source-1:chunk:0" {
|
||||
t.Fatalf("SourceChunk.ID = %q, want source-1:chunk:0", chunk.ID)
|
||||
t.Fatalf("source.Chunk.ID = %q, want source-1:chunk:0", chunk.ID)
|
||||
}
|
||||
if chunk.SourceID != doc.ID {
|
||||
t.Fatalf("SourceChunk.SourceID = %q, want %q", chunk.SourceID, doc.ID)
|
||||
t.Fatalf("source.Chunk.SourceID = %q, want %q", chunk.SourceID, doc.ID)
|
||||
}
|
||||
if chunk.Index != 0 {
|
||||
t.Fatalf("SourceChunk.Index = %d, want 0", chunk.Index)
|
||||
t.Fatalf("source.Chunk.Index = %d, want 0", chunk.Index)
|
||||
}
|
||||
if chunk.StartUnitID != 1 || chunk.EndUnitID != 1 {
|
||||
t.Fatalf("SourceChunk boundaries = %d-%d, want 1-1", chunk.StartUnitID, chunk.EndUnitID)
|
||||
if chunk.Ref.StartUnitID != 1 || chunk.Ref.EndUnitID != 1 {
|
||||
t.Fatalf("source.Chunk.Ref = %#v, want source-1:1-1", chunk.Ref)
|
||||
}
|
||||
if chunk.MediaType != "application/json" || string(chunk.Content) != `{"units":[{"id":1,"kind":"section","text":"Source text."}]}` {
|
||||
t.Fatalf("SourceChunk payload = %q %s, want JSON units", chunk.MediaType, chunk.Content)
|
||||
t.Fatalf("source.Chunk payload = %q %s, want JSON units", chunk.MediaType, chunk.Content)
|
||||
}
|
||||
if len(chunk.Units) != 1 {
|
||||
t.Fatalf("len(SourceChunk.Units) = %d, want 1", len(chunk.Units))
|
||||
t.Fatalf("len(source.Chunk.Units) = %d, want 1", len(chunk.Units))
|
||||
}
|
||||
}
|
||||
|
||||
func TestFakeChunkerReceivesLLMClient(t *testing.T) {
|
||||
func TestFakeChunkerReceivesPerRunContext(t *testing.T) {
|
||||
doc := &source.SourceDocument{
|
||||
ID: "source-1",
|
||||
Kind: "document",
|
||||
@@ -107,14 +100,13 @@ func TestFakeChunkerReceivesLLMClient(t *testing.T) {
|
||||
{ID: 1, Kind: "section", Text: "Source text."},
|
||||
},
|
||||
}
|
||||
client := fakeLLMClient{}
|
||||
chunker := &recordingChunker{key: "llm-chunker"}
|
||||
|
||||
if _, err := chunker.Chunk(context.Background(), ChunkRequest{Source: doc, LLMClient: client}); err != nil {
|
||||
if _, err := chunker.Chunk(context.Background(), ChunkRequest{Source: doc, SessionID: "session", LLMProfile: "profile"}); err != nil {
|
||||
t.Fatalf("Chunk() error = %v, want nil", err)
|
||||
}
|
||||
if chunker.request.LLMClient == nil {
|
||||
t.Fatal("ChunkRequest.LLMClient = nil, want structured LLM client")
|
||||
if chunker.request.SessionID != "session" || chunker.request.LLMProfile != "profile" {
|
||||
t.Fatalf("ChunkRequest = %#v, want per-run session and profile", chunker.request)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -130,18 +122,17 @@ func TestFakeExtractorReceivesChunkAndAmbientContext(t *testing.T) {
|
||||
{ID: 2, Kind: "section", Text: "Second source text."},
|
||||
},
|
||||
}
|
||||
chunk := SourceChunk{
|
||||
ID: "source-1:chunk:1",
|
||||
SourceID: doc.ID,
|
||||
Index: 1,
|
||||
StartUnitID: 2,
|
||||
EndUnitID: 2,
|
||||
Content: []byte(`{"units":[{"id":2,"kind":"section","text":"Second source text."}]}`),
|
||||
MediaType: "application/json",
|
||||
Units: []source.SourceUnit{doc.Units[1]},
|
||||
chunk := source.Chunk{
|
||||
ID: "source-1:chunk:1",
|
||||
SourceID: doc.ID,
|
||||
Index: 1,
|
||||
Ref: source.SourceRef{SourceID: doc.ID, StartUnitID: 2, EndUnitID: 2},
|
||||
Content: []byte(`{"units":[{"id":2,"kind":"section","text":"Second source text."}]}`),
|
||||
MediaType: "application/json",
|
||||
Units: []source.SourceUnit{doc.Units[1]},
|
||||
}
|
||||
|
||||
result, err := extractor.Extract(context.Background(), ExtractionRequest{
|
||||
result, err := extractor.Extract(context.Background(), TypedExtractionRequest{
|
||||
Source: doc,
|
||||
Chunk: &chunk,
|
||||
AmbientContext: map[string]any{"mode": "chunked"},
|
||||
@@ -149,11 +140,8 @@ func TestFakeExtractorReceivesChunkAndAmbientContext(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("Extract() error = %v, want nil", err)
|
||||
}
|
||||
if result.Output.ChunkID != "" || result.Output.ChunkIndex != 0 {
|
||||
t.Fatalf("chunk provenance = %q/%d, want runner-owned zero values", result.Output.ChunkID, result.Output.ChunkIndex)
|
||||
}
|
||||
if string(result.Output.Payload.Content) != `{"value":"chunked"}` {
|
||||
t.Fatalf("Payload.Content = %s, want chunked payload", result.Output.Payload.Content)
|
||||
if result.Value.Value != "chunked" {
|
||||
t.Fatalf("Value = %q, want chunked", result.Value.Value)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -322,8 +310,8 @@ func TestLLMInputSetCloneCopiesContent(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestResponseSchemaJSONOmitRawSchemaContent(t *testing.T) {
|
||||
schema := ResponseSchema{
|
||||
func TestArtifactSchemaJSONOmitsSchemaContent(t *testing.T) {
|
||||
schema := ArtifactSchema{
|
||||
ID: "schema-id",
|
||||
Name: "schema-name",
|
||||
Version: "v1",
|
||||
@@ -349,26 +337,14 @@ func TestResponseSchemaJSONOmitRawSchemaContent(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
|
||||
extractOutput := ExtractOutput{
|
||||
LaneID: "generic-lane",
|
||||
ExtractorKey: "generic-extractor",
|
||||
SourceID: "source-1",
|
||||
ChunkID: "source-1:chunk:0",
|
||||
ChunkIndex: 0,
|
||||
Schema: ResponseSchema{ID: "schema-id", Name: "schema-name", Version: "v1"},
|
||||
Payload: RawPayload{
|
||||
Content: []byte(`{"value":"example"}`),
|
||||
MediaType: "application/json",
|
||||
Metadata: map[string]any{"confidence": 0.75},
|
||||
},
|
||||
}
|
||||
extractOutput := ExtractArtifact[fakeArtifact]{LaneID: "generic-lane", ExtractorKey: "generic-extractor", SourceID: "source-1", ChunkID: "source-1:chunk:0", ChunkIndex: 0, Value: fakeArtifact{Value: "example"}}
|
||||
merger := fakeMerger{key: "generic-merger"}
|
||||
normalizer := fakeNormalizer{key: "generic-normalizer"}
|
||||
encoder := fakeOutputEncoder{key: "generic-output"}
|
||||
|
||||
merged, err := merger.Merge(context.Background(), MergeRequest{
|
||||
merged, err := merger.Merge(context.Background(), TypedMergeRequest[fakeArtifact]{
|
||||
LaneID: "generic-lane",
|
||||
ExtractOutputs: []ExtractOutput{extractOutput},
|
||||
ExtractOutputs: []ExtractArtifact[fakeArtifact]{extractOutput},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Merge() error = %v, want nil", err)
|
||||
@@ -376,13 +352,13 @@ func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
|
||||
if merger.Key() != "generic-merger" {
|
||||
t.Fatalf("Merger.Key() = %q, want generic-merger", merger.Key())
|
||||
}
|
||||
if string(merged.Output.Payload.Content) != `{"value":"example"}` {
|
||||
t.Fatalf("merged content = %s, want raw extract content", merged.Output.Payload.Content)
|
||||
if merged.Value.Value != "example" {
|
||||
t.Fatalf("merged value = %q, want example", merged.Value.Value)
|
||||
}
|
||||
|
||||
normalized, err := normalizer.Normalize(context.Background(), NormalizeRequest{
|
||||
normalized, err := normalizer.Normalize(context.Background(), TypedNormalizeRequest[fakeArtifact]{
|
||||
LaneID: "generic-lane",
|
||||
MergeOutput: merged.Output,
|
||||
MergeOutput: MergeArtifact[fakeArtifact]{LaneID: "generic-lane", MergerKey: merger.Key(), SourceID: "source-1", Value: merged.Value},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Normalize() error = %v, want nil", err)
|
||||
@@ -390,13 +366,13 @@ func TestFakeMergeNormalizeAndOutputContracts(t *testing.T) {
|
||||
if normalizer.Key() != "generic-normalizer" {
|
||||
t.Fatalf("Normalizer.Key() = %q, want generic-normalizer", normalizer.Key())
|
||||
}
|
||||
if string(normalized.Output.Payload.Content) != `{"value":"example"}` {
|
||||
t.Fatalf("normalized content = %s, want raw merge content", normalized.Output.Payload.Content)
|
||||
if normalized.Value.Value != "example" {
|
||||
t.Fatalf("normalized value = %q, want example", normalized.Value.Value)
|
||||
}
|
||||
|
||||
encoded, err := encoder.Encode(context.Background(), OutputRequest{
|
||||
Manifest: artifacts.RunManifest{RunID: "run-1"},
|
||||
NormalizeOutputs: []NormalizeOutput{normalized.Output},
|
||||
NormalizeOutputs: []SerializedOutput{{LaneID: "generic-lane", NormalizerKey: normalizer.Key(), SourceID: "source-1", Artifact: SerializedArtifact{Kind: "test/artifact", Schema: ArtifactSchema{ID: "schema-id", Name: "schema-name", Version: "v1"}, MediaType: "application/json", Content: []byte(`{"value":"example"}`)}}},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Encode() error = %v, want nil", err)
|
||||
@@ -472,16 +448,19 @@ func (chunker fakeChunker) ReferenceSlots() []ReferenceSlot {
|
||||
|
||||
func (chunker fakeChunker) Chunk(ctx context.Context, req ChunkRequest) (ChunkResult, error) {
|
||||
return ChunkResult{
|
||||
Chunks: []SourceChunk{
|
||||
Chunks: []source.Chunk{
|
||||
{
|
||||
ID: req.Source.ID + ":chunk:0",
|
||||
SourceID: req.Source.ID,
|
||||
Index: 0,
|
||||
StartUnitID: req.Source.Units[0].ID,
|
||||
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
|
||||
Content: []byte(`{"units":[{"id":1,"kind":"section","text":"Source text."}]}`),
|
||||
MediaType: "application/json",
|
||||
Units: append([]source.SourceUnit(nil), req.Source.Units...),
|
||||
ID: req.Source.ID + ":chunk:0",
|
||||
SourceID: req.Source.ID,
|
||||
Index: 0,
|
||||
Ref: source.SourceRef{
|
||||
SourceID: req.Source.ID,
|
||||
StartUnitID: req.Source.Units[0].ID,
|
||||
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
|
||||
},
|
||||
Content: []byte(`{"units":[{"id":1,"kind":"section","text":"Source text."}]}`),
|
||||
MediaType: "application/json",
|
||||
Units: append([]source.SourceUnit(nil), req.Source.Units...),
|
||||
},
|
||||
},
|
||||
}, nil
|
||||
@@ -509,6 +488,8 @@ type fakeExtractor struct {
|
||||
key string
|
||||
}
|
||||
|
||||
type fakeArtifact struct{ Value string }
|
||||
|
||||
func (extractor fakeExtractor) Key() string {
|
||||
return extractor.key
|
||||
}
|
||||
@@ -517,21 +498,12 @@ func (extractor fakeExtractor) ReferenceSlots() []ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (extractor fakeExtractor) Extract(ctx context.Context, req ExtractionRequest) (ExtractionResult, error) {
|
||||
payload := json.RawMessage(`{"value":"example"}`)
|
||||
func (extractor fakeExtractor) Extract(ctx context.Context, req TypedExtractionRequest) (TypedExtractionResult[fakeArtifact], error) {
|
||||
value := "example"
|
||||
if req.AmbientContext["mode"] == "chunked" {
|
||||
payload = json.RawMessage(`{"value":"chunked"}`)
|
||||
value = "chunked"
|
||||
}
|
||||
|
||||
return ExtractionResult{
|
||||
Output: ExtractOutput{
|
||||
Schema: ResponseSchema{ID: "schema-id", Name: "schema-name", Version: "v1"},
|
||||
Payload: RawPayload{
|
||||
Content: append([]byte(nil), payload...),
|
||||
MediaType: "application/json",
|
||||
},
|
||||
},
|
||||
}, nil
|
||||
return TypedExtractionResult[fakeArtifact]{Value: fakeArtifact{Value: value}}, nil
|
||||
}
|
||||
|
||||
type fakeMerger struct {
|
||||
@@ -542,15 +514,8 @@ func (merger fakeMerger) Key() string {
|
||||
return merger.key
|
||||
}
|
||||
|
||||
func (merger fakeMerger) Merge(ctx context.Context, req MergeRequest) (MergeResult, error) {
|
||||
output := req.ExtractOutputs[0]
|
||||
return MergeResult{Output: MergeOutput{
|
||||
LaneID: req.LaneID,
|
||||
MergerKey: merger.key,
|
||||
SourceID: output.SourceID,
|
||||
Schema: output.Schema,
|
||||
Payload: cloneTestRawPayload(output.Payload),
|
||||
}}, nil
|
||||
func (merger fakeMerger) Merge(ctx context.Context, req TypedMergeRequest[fakeArtifact]) (TypedMergeResult[fakeArtifact], error) {
|
||||
return TypedMergeResult[fakeArtifact]{Value: req.ExtractOutputs[0].Value}, nil
|
||||
}
|
||||
|
||||
type fakeNormalizer struct {
|
||||
@@ -565,54 +530,8 @@ func (normalizer fakeNormalizer) ReferenceSlots() []ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (normalizer fakeNormalizer) Normalize(ctx context.Context, req NormalizeRequest) (NormalizeResult, error) {
|
||||
return NormalizeResult{Output: NormalizeOutput{
|
||||
LaneID: req.LaneID,
|
||||
NormalizerKey: normalizer.key,
|
||||
SourceID: req.MergeOutput.SourceID,
|
||||
Schema: req.MergeOutput.Schema,
|
||||
Payload: cloneTestRawPayload(req.MergeOutput.Payload),
|
||||
}}, nil
|
||||
}
|
||||
|
||||
func cloneTestRawPayload(payload RawPayload) RawPayload {
|
||||
return RawPayload{
|
||||
Content: append([]byte(nil), payload.Content...),
|
||||
MediaType: payload.MediaType,
|
||||
Metadata: cloneTestMetadata(payload.Metadata),
|
||||
Warnings: append([]Warning(nil), payload.Warnings...),
|
||||
}
|
||||
}
|
||||
|
||||
func cloneTestMetadata(metadata map[string]any) map[string]any {
|
||||
if len(metadata) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]any, len(metadata))
|
||||
for key, value := range metadata {
|
||||
out[key] = value
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
type fakeValidator struct {
|
||||
name string
|
||||
}
|
||||
|
||||
func (validator fakeValidator) Name() string {
|
||||
return validator.name
|
||||
}
|
||||
|
||||
func (validator fakeValidator) ExecutionClass() ExecutionClass {
|
||||
return ExecutionClassDeterministic
|
||||
}
|
||||
|
||||
func (validator fakeValidator) Validate(ctx context.Context, req ValidationRequest) (ValidationResult, error) {
|
||||
return ValidationResult{
|
||||
Approved: true,
|
||||
ReasonCode: "accepted",
|
||||
Message: "output accepted",
|
||||
}, nil
|
||||
func (normalizer fakeNormalizer) Normalize(ctx context.Context, req TypedNormalizeRequest[fakeArtifact]) (TypedNormalizeResult[fakeArtifact], error) {
|
||||
return TypedNormalizeResult[fakeArtifact]{Value: req.MergeOutput.Value}, nil
|
||||
}
|
||||
|
||||
type fakeLLMClient struct{}
|
||||
|
||||
164
internal/framework/contracts/typed_pipeline.go
Normal file
164
internal/framework/contracts/typed_pipeline.go
Normal file
@@ -0,0 +1,164 @@
|
||||
package contracts
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
)
|
||||
|
||||
// ExtractArtifact carries a typed per-chunk value with framework provenance.
|
||||
type ExtractArtifact[T any] struct {
|
||||
LaneID string
|
||||
ExtractorKey string
|
||||
SourceID string
|
||||
ChunkID string
|
||||
ChunkIndex int
|
||||
ChunkRef source.SourceRef
|
||||
Value T
|
||||
}
|
||||
|
||||
// MergeArtifact carries a typed merged value with framework provenance.
|
||||
type MergeArtifact[T any] struct {
|
||||
LaneID string
|
||||
MergerKey string
|
||||
SourceID string
|
||||
Value T
|
||||
}
|
||||
|
||||
// NormalizeArtifact carries a typed normalized value with framework provenance.
|
||||
type NormalizeArtifact[T any] struct {
|
||||
LaneID string
|
||||
NormalizerKey string
|
||||
SourceID string
|
||||
Value T
|
||||
}
|
||||
|
||||
type TypedExtractionRequest struct {
|
||||
Source *source.SourceDocument
|
||||
Chunk *source.Chunk
|
||||
AmbientContext map[string]any
|
||||
SourceInput LLMInputMaterial
|
||||
SessionID string
|
||||
References ReferenceSet
|
||||
LLMProfile string
|
||||
Metadata map[string]any
|
||||
}
|
||||
|
||||
type TypedExtractionResult[T any] struct {
|
||||
Value T
|
||||
Warnings []Warning
|
||||
}
|
||||
|
||||
type Extractor[T any] interface {
|
||||
Key() string
|
||||
ReferenceSlots() []ReferenceSlot
|
||||
Extract(context.Context, TypedExtractionRequest) (TypedExtractionResult[T], error)
|
||||
}
|
||||
|
||||
type TypedMergeRequest[T any] struct {
|
||||
Source *source.SourceDocument
|
||||
LaneID string
|
||||
ExtractOutputs []ExtractArtifact[T]
|
||||
SourceInput LLMInputMaterial
|
||||
SessionID string
|
||||
References ReferenceSet
|
||||
LLMProfile string
|
||||
Metadata map[string]any
|
||||
}
|
||||
|
||||
type TypedMergeResult[T any] struct {
|
||||
Value T
|
||||
Warnings []Warning
|
||||
}
|
||||
|
||||
type Merger[T any] interface {
|
||||
Key() string
|
||||
Merge(context.Context, TypedMergeRequest[T]) (TypedMergeResult[T], error)
|
||||
}
|
||||
|
||||
type TypedNormalizeRequest[T any] struct {
|
||||
Source *source.SourceDocument
|
||||
LaneID string
|
||||
MergeOutput MergeArtifact[T]
|
||||
SourceInput LLMInputMaterial
|
||||
SessionID string
|
||||
References ReferenceSet
|
||||
LLMProfile string
|
||||
Metadata map[string]any
|
||||
}
|
||||
|
||||
type TypedNormalizeResult[T any] struct {
|
||||
Value T
|
||||
Warnings []Warning
|
||||
}
|
||||
|
||||
type Normalizer[T any] interface {
|
||||
Key() string
|
||||
ReferenceSlots() []ReferenceSlot
|
||||
Normalize(context.Context, TypedNormalizeRequest[T]) (TypedNormalizeResult[T], error)
|
||||
}
|
||||
|
||||
type TypedValidationRequest[T any] struct {
|
||||
Stage string
|
||||
LaneID string
|
||||
ModuleKey string
|
||||
Source *source.SourceDocument
|
||||
SourceID string
|
||||
SourceInput LLMInputMaterial
|
||||
SessionID string
|
||||
References ReferenceSet
|
||||
LLMProfile string
|
||||
Metadata map[string]any
|
||||
Chunk *source.Chunk
|
||||
Chunks []source.Chunk
|
||||
Ref source.SourceRef
|
||||
Value T
|
||||
}
|
||||
|
||||
type TypedValidator[T any] interface {
|
||||
Name() string
|
||||
ExecutionClass() ExecutionClass
|
||||
Validate(context.Context, TypedValidationRequest[T]) (ValidationResult, error)
|
||||
}
|
||||
|
||||
type ChunkValidationRequest struct {
|
||||
ModuleKey string
|
||||
Source *source.SourceDocument
|
||||
SourceID string
|
||||
SourceInput LLMInputMaterial
|
||||
SessionID string
|
||||
References ReferenceSet
|
||||
LLMProfile string
|
||||
Metadata map[string]any
|
||||
Chunks []source.Chunk
|
||||
}
|
||||
|
||||
type ChunkValidator interface {
|
||||
Name() string
|
||||
ExecutionClass() ExecutionClass
|
||||
Validate(context.Context, ChunkValidationRequest) (ValidationResult, error)
|
||||
}
|
||||
|
||||
type SerializedValidationRequest struct {
|
||||
Stage string
|
||||
LaneID string
|
||||
ModuleKey string
|
||||
Source *source.SourceDocument
|
||||
SourceID string
|
||||
SourceInput LLMInputMaterial
|
||||
SessionID string
|
||||
References ReferenceSet
|
||||
LLMProfile string
|
||||
Metadata map[string]any
|
||||
Chunk *source.Chunk
|
||||
Chunks []source.Chunk
|
||||
Schema ArtifactSchema
|
||||
MediaType string
|
||||
Content []byte
|
||||
}
|
||||
|
||||
type SerializedValidator interface {
|
||||
Name() string
|
||||
ExecutionClass() ExecutionClass
|
||||
Validate(context.Context, SerializedValidationRequest) (ValidationResult, error)
|
||||
}
|
||||
318
internal/framework/pipeline/artifact_codec_registry.go
Normal file
318
internal/framework/pipeline/artifact_codec_registry.go
Normal file
@@ -0,0 +1,318 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/santhosh-tekuri/jsonschema/v6"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type ArtifactCodecSpec struct {
|
||||
Kind contracts.ArtifactKind
|
||||
Schema contracts.ArtifactSchema
|
||||
SchemaDigest string
|
||||
MediaType string
|
||||
}
|
||||
|
||||
// ArtifactCodecTypeError reports a value that does not have the exact Go type
|
||||
// registered for an artifact kind.
|
||||
type ArtifactCodecTypeError struct {
|
||||
Operation string
|
||||
Kind contracts.ArtifactKind
|
||||
ExpectedType string
|
||||
ActualType string
|
||||
}
|
||||
|
||||
func (e *ArtifactCodecTypeError) Error() string {
|
||||
return fmt.Sprintf("%s artifact %q: expected exact Go type %s, got %s", e.Operation, e.Kind, e.ExpectedType, e.ActualType)
|
||||
}
|
||||
|
||||
// ArtifactCodecCompatibilityError reports serialized metadata that does not
|
||||
// identify the registered representation for an artifact kind.
|
||||
type ArtifactCodecCompatibilityError struct {
|
||||
Kind contracts.ArtifactKind
|
||||
Reason string
|
||||
}
|
||||
|
||||
func (e *ArtifactCodecCompatibilityError) Error() string {
|
||||
return fmt.Sprintf("decode artifact %q: %s", e.Kind, e.Reason)
|
||||
}
|
||||
|
||||
// ArtifactCodecOperationError preserves an encode or decode failure from the
|
||||
// registered domain codec.
|
||||
type ArtifactCodecOperationError struct {
|
||||
Operation string
|
||||
Kind contracts.ArtifactKind
|
||||
Err error
|
||||
}
|
||||
|
||||
func (e *ArtifactCodecOperationError) Error() string {
|
||||
return fmt.Sprintf("%s artifact %q: %v", e.Operation, e.Kind, e.Err)
|
||||
}
|
||||
|
||||
func (e *ArtifactCodecOperationError) Unwrap() error { return e.Err }
|
||||
|
||||
type ArtifactCodecRegistry struct {
|
||||
entries map[contracts.ArtifactKind]artifactCodecEntry
|
||||
}
|
||||
|
||||
type artifactCodecEntry struct {
|
||||
spec ArtifactCodecSpec
|
||||
valueType reflect.Type
|
||||
encode func(any) ([]byte, error)
|
||||
encodeCandidate func(any) ([]byte, error)
|
||||
metadata func(any) map[string]any
|
||||
decode func([]byte) (any, error)
|
||||
}
|
||||
|
||||
func NewArtifactCodecRegistry() *ArtifactCodecRegistry {
|
||||
return &ArtifactCodecRegistry{entries: make(map[contracts.ArtifactKind]artifactCodecEntry)}
|
||||
}
|
||||
|
||||
// RegisterArtifactCodec registers one codec for T. The concrete type is kept
|
||||
// private and checked at every erased encode boundary.
|
||||
func RegisterArtifactCodec[T any](registry *ArtifactCodecRegistry, codec contracts.ArtifactCodec[T]) error {
|
||||
if registry == nil {
|
||||
return fmt.Errorf("artifact codec registry must not be nil")
|
||||
}
|
||||
if nilInterface(codec) {
|
||||
return fmt.Errorf("artifact codec must not be nil")
|
||||
}
|
||||
|
||||
spec, err := artifactCodecSpec(codec.Kind(), codec.Schema(), codec.MediaType())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, ok := registry.entries[spec.Kind]; ok {
|
||||
return fmt.Errorf("artifact codec %q is already registered", spec.Kind)
|
||||
}
|
||||
|
||||
valueType := reflect.TypeFor[T]()
|
||||
entry := artifactCodecEntry{
|
||||
spec: cloneArtifactCodecSpec(spec),
|
||||
valueType: valueType,
|
||||
encode: func(value any) ([]byte, error) {
|
||||
actualType := reflect.TypeOf(value)
|
||||
if actualType != valueType {
|
||||
return nil, newArtifactCodecTypeError("encode", spec.Kind, valueType, actualType)
|
||||
}
|
||||
typed, ok := value.(T)
|
||||
if !ok {
|
||||
return nil, newArtifactCodecTypeError("encode", spec.Kind, valueType, actualType)
|
||||
}
|
||||
encoded, err := codec.Encode(typed)
|
||||
if err != nil {
|
||||
return nil, &ArtifactCodecOperationError{Operation: "encode", Kind: spec.Kind, Err: err}
|
||||
}
|
||||
return append([]byte(nil), encoded...), nil
|
||||
},
|
||||
decode: func(content []byte) (any, error) {
|
||||
decoded, err := codec.Decode(append([]byte(nil), content...))
|
||||
if err != nil {
|
||||
return nil, &ArtifactCodecOperationError{Operation: "decode", Kind: spec.Kind, Err: err}
|
||||
}
|
||||
return decoded, nil
|
||||
},
|
||||
}
|
||||
entry.encodeCandidate = entry.encode
|
||||
if candidate, ok := any(codec).(interface{ EncodeCandidate(T) ([]byte, error) }); ok {
|
||||
entry.encodeCandidate = func(value any) ([]byte, error) {
|
||||
typed, err := exactTypedValue[T]("encode candidate artifact", value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
content, err := candidate.EncodeCandidate(typed)
|
||||
if err != nil {
|
||||
return nil, &ArtifactCodecOperationError{Operation: "encode", Kind: spec.Kind, Err: err}
|
||||
}
|
||||
return append([]byte(nil), content...), nil
|
||||
}
|
||||
}
|
||||
if provider, ok := any(codec).(interface{ Metadata(T) map[string]any }); ok {
|
||||
entry.metadata = func(value any) map[string]any {
|
||||
typed, err := exactTypedValue[T]("artifact metadata", value)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return cloneMetadata(provider.Metadata(typed))
|
||||
}
|
||||
}
|
||||
if registry.entries == nil {
|
||||
registry.entries = make(map[contracts.ArtifactKind]artifactCodecEntry)
|
||||
}
|
||||
registry.entries[spec.Kind] = entry
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *ArtifactCodecRegistry) Spec(kind contracts.ArtifactKind) (ArtifactCodecSpec, bool) {
|
||||
if r == nil {
|
||||
return ArtifactCodecSpec{}, false
|
||||
}
|
||||
entry, ok := r.entries[normalizeArtifactKind(kind)]
|
||||
if !ok {
|
||||
return ArtifactCodecSpec{}, false
|
||||
}
|
||||
return cloneArtifactCodecSpec(entry.spec), true
|
||||
}
|
||||
|
||||
func (r *ArtifactCodecRegistry) valueType(kind contracts.ArtifactKind) (reflect.Type, bool) {
|
||||
if r == nil {
|
||||
return nil, false
|
||||
}
|
||||
entry, ok := r.entries[normalizeArtifactKind(kind)]
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
return entry.valueType, true
|
||||
}
|
||||
|
||||
func (r *ArtifactCodecRegistry) RegisteredKinds() []contracts.ArtifactKind {
|
||||
if r == nil || len(r.entries) == 0 {
|
||||
return nil
|
||||
}
|
||||
kinds := make([]contracts.ArtifactKind, 0, len(r.entries))
|
||||
for kind := range r.entries {
|
||||
kinds = append(kinds, kind)
|
||||
}
|
||||
sort.Slice(kinds, func(i, j int) bool { return kinds[i] < kinds[j] })
|
||||
return kinds
|
||||
}
|
||||
|
||||
// Encode serializes an erased framework value after proving its exact
|
||||
// registered Go type.
|
||||
func (r *ArtifactCodecRegistry) Encode(kind contracts.ArtifactKind, value any) (contracts.SerializedArtifact, error) {
|
||||
entry, normalizedKind, err := r.entry(kind)
|
||||
if err != nil {
|
||||
return contracts.SerializedArtifact{}, err
|
||||
}
|
||||
content, err := entry.encode(value)
|
||||
if err != nil {
|
||||
return contracts.SerializedArtifact{}, err
|
||||
}
|
||||
return contracts.SerializedArtifact{
|
||||
Kind: normalizedKind,
|
||||
Schema: contracts.CloneArtifactSchema(entry.spec.Schema),
|
||||
MediaType: entry.spec.MediaType,
|
||||
Content: content,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Decode verifies serialized identity before invoking the registered codec.
|
||||
func (r *ArtifactCodecRegistry) Decode(artifact contracts.SerializedArtifact) (any, error) {
|
||||
entry, normalizedKind, err := r.entry(artifact.Kind)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if artifact.Schema.ID != entry.spec.Schema.ID || artifact.Schema.Name != entry.spec.Schema.Name || artifact.Schema.Version != entry.spec.Schema.Version {
|
||||
return nil, &ArtifactCodecCompatibilityError{Kind: normalizedKind, Reason: "schema identity does not match registered codec"}
|
||||
}
|
||||
if contracts.DigestArtifactSchema(artifact.Schema) != entry.spec.SchemaDigest {
|
||||
return nil, &ArtifactCodecCompatibilityError{Kind: normalizedKind, Reason: "schema digest does not match registered codec"}
|
||||
}
|
||||
if strings.TrimSpace(artifact.MediaType) != entry.spec.MediaType {
|
||||
return nil, &ArtifactCodecCompatibilityError{Kind: normalizedKind, Reason: "media type does not match registered codec"}
|
||||
}
|
||||
return entry.decode(artifact.Content)
|
||||
}
|
||||
|
||||
func (r *ArtifactCodecRegistry) entry(kind contracts.ArtifactKind) (artifactCodecEntry, contracts.ArtifactKind, error) {
|
||||
if r == nil {
|
||||
return artifactCodecEntry{}, "", fmt.Errorf("artifact codec registry must not be nil")
|
||||
}
|
||||
normalizedKind := normalizeArtifactKind(kind)
|
||||
if normalizedKind == "" {
|
||||
return artifactCodecEntry{}, "", fmt.Errorf("artifact kind must not be empty")
|
||||
}
|
||||
entry, ok := r.entries[normalizedKind]
|
||||
if !ok {
|
||||
return artifactCodecEntry{}, normalizedKind, fmt.Errorf("artifact codec %q is not registered", normalizedKind)
|
||||
}
|
||||
return entry, normalizedKind, nil
|
||||
}
|
||||
|
||||
func artifactCodecSpec(kind contracts.ArtifactKind, schema contracts.ArtifactSchema, mediaType string) (ArtifactCodecSpec, error) {
|
||||
kind = normalizeArtifactKind(kind)
|
||||
schema.ID = strings.TrimSpace(schema.ID)
|
||||
schema.Name = strings.TrimSpace(schema.Name)
|
||||
schema.Version = strings.TrimSpace(schema.Version)
|
||||
mediaType = strings.TrimSpace(mediaType)
|
||||
switch {
|
||||
case kind == "":
|
||||
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec kind must not be empty")
|
||||
case schema.ID == "":
|
||||
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q schema id must not be empty", kind)
|
||||
case schema.Name == "":
|
||||
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q schema name must not be empty", kind)
|
||||
case schema.Version == "":
|
||||
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q schema version must not be empty", kind)
|
||||
case mediaType == "":
|
||||
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q media type must not be empty", kind)
|
||||
case len(bytes.TrimSpace(schema.JSONSchema)) == 0:
|
||||
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema must not be empty", kind)
|
||||
case !json.Valid(schema.JSONSchema):
|
||||
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema must be valid JSON", kind)
|
||||
}
|
||||
schemaDocument, err := jsonschema.UnmarshalJSON(bytes.NewReader(schema.JSONSchema))
|
||||
if err != nil {
|
||||
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema is invalid: %w", kind, err)
|
||||
}
|
||||
compiler := jsonschema.NewCompiler()
|
||||
if err := compiler.AddResource("artifact-schema.json", schemaDocument); err != nil {
|
||||
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema is invalid: %w", kind, err)
|
||||
}
|
||||
if _, err := compiler.Compile("artifact-schema.json"); err != nil {
|
||||
return ArtifactCodecSpec{}, fmt.Errorf("artifact codec %q JSON Schema is invalid: %w", kind, err)
|
||||
}
|
||||
schema = contracts.CloneArtifactSchema(schema)
|
||||
return ArtifactCodecSpec{
|
||||
Kind: kind,
|
||||
Schema: schema,
|
||||
SchemaDigest: contracts.DigestArtifactSchema(schema),
|
||||
MediaType: mediaType,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func normalizeArtifactKind(kind contracts.ArtifactKind) contracts.ArtifactKind {
|
||||
return contracts.ArtifactKind(strings.TrimSpace(string(kind)))
|
||||
}
|
||||
|
||||
func cloneArtifactCodecSpec(spec ArtifactCodecSpec) ArtifactCodecSpec {
|
||||
spec.Schema = contracts.CloneArtifactSchema(spec.Schema)
|
||||
return spec
|
||||
}
|
||||
|
||||
func nilInterface(value any) bool {
|
||||
if value == nil {
|
||||
return true
|
||||
}
|
||||
reflected := reflect.ValueOf(value)
|
||||
switch reflected.Kind() {
|
||||
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:
|
||||
return reflected.IsNil()
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
func newArtifactCodecTypeError(operation string, kind contracts.ArtifactKind, expected reflect.Type, actual reflect.Type) error {
|
||||
expectedName := "<nil>"
|
||||
if expected != nil {
|
||||
expectedName = expected.String()
|
||||
}
|
||||
actualName := "<nil>"
|
||||
if actual != nil {
|
||||
actualName = actual.String()
|
||||
}
|
||||
return &ArtifactCodecTypeError{
|
||||
Operation: operation,
|
||||
Kind: kind,
|
||||
ExpectedType: expectedName,
|
||||
ActualType: actualName,
|
||||
}
|
||||
}
|
||||
307
internal/framework/pipeline/artifact_codec_registry_test.go
Normal file
307
internal/framework/pipeline/artifact_codec_registry_test.go
Normal file
@@ -0,0 +1,307 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type codecNotes struct {
|
||||
Items []string `json:"items"`
|
||||
}
|
||||
|
||||
type codecScore struct {
|
||||
Value int `json:"value"`
|
||||
}
|
||||
|
||||
type codecNotesAlias codecNotes
|
||||
|
||||
type testArtifactCodec[T any] struct {
|
||||
kind contracts.ArtifactKind
|
||||
schema contracts.ArtifactSchema
|
||||
mediaType string
|
||||
encodeFunc func(T) ([]byte, error)
|
||||
decodeFunc func([]byte) (T, error)
|
||||
}
|
||||
|
||||
func (c testArtifactCodec[T]) Kind() contracts.ArtifactKind { return c.kind }
|
||||
func (c testArtifactCodec[T]) Schema() contracts.ArtifactSchema { return c.schema }
|
||||
func (c testArtifactCodec[T]) MediaType() string { return c.mediaType }
|
||||
func (c testArtifactCodec[T]) Encode(value T) ([]byte, error) { return c.encodeFunc(value) }
|
||||
func (c testArtifactCodec[T]) Decode(content []byte) (T, error) { return c.decodeFunc(content) }
|
||||
|
||||
func TestArtifactCodecRegistryStoresHeterogeneousExactTypes(t *testing.T) {
|
||||
registry := NewArtifactCodecRegistry()
|
||||
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
|
||||
t.Fatalf("RegisterArtifactCodec(notes) error = %v, want nil", err)
|
||||
}
|
||||
if err := RegisterArtifactCodec(registry, scoreCodec()); err != nil {
|
||||
t.Fatalf("RegisterArtifactCodec(score) error = %v, want nil", err)
|
||||
}
|
||||
|
||||
if got, want := registry.RegisteredKinds(), []contracts.ArtifactKind{"test/notes", "test/score"}; !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("RegisteredKinds() = %#v, want %#v", got, want)
|
||||
}
|
||||
notes := codecNotes{Items: []string{"second", "first"}}
|
||||
first, err := registry.Encode("test/notes", notes)
|
||||
if err != nil {
|
||||
t.Fatalf("Encode(notes) error = %v, want nil", err)
|
||||
}
|
||||
second, err := registry.Encode(" test/notes ", notes)
|
||||
if err != nil {
|
||||
t.Fatalf("Encode(notes again) error = %v, want nil", err)
|
||||
}
|
||||
if !bytes.Equal(first.Content, second.Content) {
|
||||
t.Fatalf("equal values encoded as %q and %q, want deterministic bytes", first.Content, second.Content)
|
||||
}
|
||||
decodedNotes, err := registry.Decode(first)
|
||||
if err != nil {
|
||||
t.Fatalf("Decode(notes) error = %v, want nil", err)
|
||||
}
|
||||
if !reflect.DeepEqual(decodedNotes, notes) {
|
||||
t.Fatalf("Decode(notes) = %#v, want %#v", decodedNotes, notes)
|
||||
}
|
||||
|
||||
score := codecScore{Value: 17}
|
||||
serializedScore, err := registry.Encode("test/score", score)
|
||||
if err != nil {
|
||||
t.Fatalf("Encode(score) error = %v, want nil", err)
|
||||
}
|
||||
decodedScore, err := registry.Decode(serializedScore)
|
||||
if err != nil {
|
||||
t.Fatalf("Decode(score) error = %v, want nil", err)
|
||||
}
|
||||
if decodedScore != score {
|
||||
t.Fatalf("Decode(score) = %#v, want %#v", decodedScore, score)
|
||||
}
|
||||
|
||||
_, err = registry.Encode("test/notes", codecNotesAlias(notes))
|
||||
var typeErr *ArtifactCodecTypeError
|
||||
if !errors.As(err, &typeErr) {
|
||||
t.Fatalf("Encode(alias) error = %T %v, want ArtifactCodecTypeError", err, err)
|
||||
}
|
||||
if typeErr.ExpectedType == typeErr.ActualType {
|
||||
t.Fatalf("type error = %#v, want distinct exact types", typeErr)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCodecRegistryStoresValidatedSchemaMetadata(t *testing.T) {
|
||||
registry := NewArtifactCodecRegistry()
|
||||
codec := notesCodec()
|
||||
codec.kind = " test/notes "
|
||||
codec.schema.ID = " notes.v1 "
|
||||
codec.schema.Name = " notes "
|
||||
codec.schema.Version = " v1 "
|
||||
codec.mediaType = " application/json "
|
||||
if err := RegisterArtifactCodec(registry, codec); err != nil {
|
||||
t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
|
||||
}
|
||||
codec.schema.JSONSchema[0] = '['
|
||||
|
||||
spec, ok := registry.Spec("test/notes")
|
||||
if !ok {
|
||||
t.Fatal("Spec() ok = false, want true")
|
||||
}
|
||||
if spec.Kind != "test/notes" || spec.Schema.ID != "notes.v1" || spec.Schema.Name != "notes" || spec.Schema.Version != "v1" || spec.MediaType != "application/json" {
|
||||
t.Fatalf("Spec() = %#v, want normalized metadata", spec)
|
||||
}
|
||||
if spec.SchemaDigest != contracts.DigestArtifactSchema(spec.Schema) {
|
||||
t.Fatalf("schema digest = %q, want %q", spec.SchemaDigest, contracts.DigestArtifactSchema(spec.Schema))
|
||||
}
|
||||
|
||||
spec.Schema.JSONSchema[0] = '['
|
||||
again, _ := registry.Spec("test/notes")
|
||||
if string(again.Schema.JSONSchema) != `{"additionalProperties":false,"properties":{"items":{"items":{"type":"string"},"type":"array"}},"required":["items"],"type":"object"}` {
|
||||
t.Fatalf("stored JSON Schema changed through Spec result: %q", again.Schema.JSONSchema)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCodecRegistryRejectsInvalidRegistration(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
mutate func(*testArtifactCodec[codecNotes])
|
||||
want string
|
||||
}{
|
||||
{name: "kind", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.kind = " " }, want: "kind"},
|
||||
{name: "schema id", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.ID = "" }, want: "schema id"},
|
||||
{name: "schema name", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.Name = "" }, want: "schema name"},
|
||||
{name: "schema version", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.Version = "" }, want: "schema version"},
|
||||
{name: "media type", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.mediaType = "" }, want: "media type"},
|
||||
{name: "empty JSON Schema", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.JSONSchema = nil }, want: "JSON Schema"},
|
||||
{name: "invalid JSON Schema", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.JSONSchema = []byte(`{"type":`) }, want: "valid JSON"},
|
||||
{name: "non-schema JSON", mutate: func(codec *testArtifactCodec[codecNotes]) { codec.schema.JSONSchema = []byte(`[]`) }, want: "JSON Schema is invalid"},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
registry := NewArtifactCodecRegistry()
|
||||
codec := notesCodec()
|
||||
test.mutate(&codec)
|
||||
if err := RegisterArtifactCodec(registry, codec); err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("RegisterArtifactCodec() error = %v, want %q", err, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCodecRegistryRejectsDuplicateKind(t *testing.T) {
|
||||
registry := NewArtifactCodecRegistry()
|
||||
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
|
||||
t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
|
||||
}
|
||||
duplicate := notesCodec()
|
||||
duplicate.kind = " test/notes "
|
||||
if err := RegisterArtifactCodec(registry, duplicate); err == nil || !strings.Contains(err.Error(), "already registered") {
|
||||
t.Fatalf("duplicate registration error = %v, want duplicate kind error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCodecRegistryRejectsNilRegistryAndCodec(t *testing.T) {
|
||||
codec := notesCodec()
|
||||
if err := RegisterArtifactCodec[codecNotes](nil, codec); err == nil || !strings.Contains(err.Error(), "registry") {
|
||||
t.Fatalf("nil registry error = %v, want registry error", err)
|
||||
}
|
||||
var nilCodec *testArtifactCodec[codecNotes]
|
||||
if err := RegisterArtifactCodec(NewArtifactCodecRegistry(), nilCodec); err == nil || !strings.Contains(err.Error(), "must not be nil") {
|
||||
t.Fatalf("nil codec error = %v, want codec error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCodecRegistryStrictDecodeAndTypedFailures(t *testing.T) {
|
||||
registry := NewArtifactCodecRegistry()
|
||||
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
|
||||
t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
|
||||
}
|
||||
valid, err := registry.Encode("test/notes", codecNotes{Items: []string{"one"}})
|
||||
if err != nil {
|
||||
t.Fatalf("Encode() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
for _, content := range [][]byte{
|
||||
[]byte(`{"items":["one"],"unknown":true}`),
|
||||
[]byte(`{"items":["one"]} {}`),
|
||||
} {
|
||||
candidate := contracts.CloneSerializedArtifact(valid)
|
||||
candidate.Content = content
|
||||
_, err := registry.Decode(candidate)
|
||||
var operationErr *ArtifactCodecOperationError
|
||||
if !errors.As(err, &operationErr) || operationErr.Operation != "decode" {
|
||||
t.Fatalf("Decode(%q) error = %T %v, want typed decode error", content, err, err)
|
||||
}
|
||||
}
|
||||
|
||||
wrongSchema := contracts.CloneSerializedArtifact(valid)
|
||||
wrongSchema.Schema.Version = "v2"
|
||||
_, err = registry.Decode(wrongSchema)
|
||||
var compatibilityErr *ArtifactCodecCompatibilityError
|
||||
if !errors.As(err, &compatibilityErr) {
|
||||
t.Fatalf("Decode(wrong schema) error = %T %v, want ArtifactCodecCompatibilityError", err, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCodecRegistryWrapsEncodeFailure(t *testing.T) {
|
||||
codec := notesCodec()
|
||||
cause := errors.New("cannot encode notes")
|
||||
codec.encodeFunc = func(codecNotes) ([]byte, error) { return nil, cause }
|
||||
registry := NewArtifactCodecRegistry()
|
||||
if err := RegisterArtifactCodec(registry, codec); err != nil {
|
||||
t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
_, err := registry.Encode("test/notes", codecNotes{})
|
||||
var operationErr *ArtifactCodecOperationError
|
||||
if !errors.As(err, &operationErr) || operationErr.Operation != "encode" || !errors.Is(err, cause) {
|
||||
t.Fatalf("Encode() error = %T %v, want typed wrapping encode error", err, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactCodecRegistryClonesCodecBytes(t *testing.T) {
|
||||
shared := []byte(`{"items":["one"]}`)
|
||||
codec := notesCodec()
|
||||
codec.encodeFunc = func(codecNotes) ([]byte, error) { return shared, nil }
|
||||
codec.decodeFunc = func(content []byte) (codecNotes, error) {
|
||||
content[0] = '['
|
||||
return codecNotes{Items: []string{"one"}}, nil
|
||||
}
|
||||
registry := NewArtifactCodecRegistry()
|
||||
if err := RegisterArtifactCodec(registry, codec); err != nil {
|
||||
t.Fatalf("RegisterArtifactCodec() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
artifact, err := registry.Encode("test/notes", codecNotes{})
|
||||
if err != nil {
|
||||
t.Fatalf("Encode() error = %v, want nil", err)
|
||||
}
|
||||
shared[0] = '['
|
||||
if string(artifact.Content) != `{"items":["one"]}` {
|
||||
t.Fatalf("encoded content = %q after codec buffer mutation, want owned bytes", artifact.Content)
|
||||
}
|
||||
before := append([]byte(nil), artifact.Content...)
|
||||
if _, err := registry.Decode(artifact); err != nil {
|
||||
t.Fatalf("Decode() error = %v, want nil", err)
|
||||
}
|
||||
if !bytes.Equal(artifact.Content, before) {
|
||||
t.Fatalf("serialized content changed during decode: %q", artifact.Content)
|
||||
}
|
||||
}
|
||||
|
||||
func notesCodec() testArtifactCodec[codecNotes] {
|
||||
return testArtifactCodec[codecNotes]{
|
||||
kind: "test/notes",
|
||||
schema: contracts.ArtifactSchema{
|
||||
ID: "notes.v1",
|
||||
Name: "notes",
|
||||
Version: "v1",
|
||||
JSONSchema: []byte(`{"additionalProperties":false,"properties":{"items":{"items":{"type":"string"},"type":"array"}},"required":["items"],"type":"object"}`),
|
||||
},
|
||||
mediaType: "application/json",
|
||||
encodeFunc: func(value codecNotes) ([]byte, error) {
|
||||
return json.Marshal(value)
|
||||
},
|
||||
decodeFunc: func(content []byte) (codecNotes, error) {
|
||||
var value codecNotes
|
||||
return value, decodeStrictJSON(content, &value)
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func scoreCodec() testArtifactCodec[codecScore] {
|
||||
return testArtifactCodec[codecScore]{
|
||||
kind: "test/score",
|
||||
schema: contracts.ArtifactSchema{
|
||||
ID: "score.v1",
|
||||
Name: "score",
|
||||
Version: "v1",
|
||||
JSONSchema: []byte(`{"additionalProperties":false,"properties":{"value":{"type":"integer"}},"required":["value"],"type":"object"}`),
|
||||
},
|
||||
mediaType: "application/json",
|
||||
encodeFunc: func(value codecScore) ([]byte, error) {
|
||||
return json.Marshal(value)
|
||||
},
|
||||
decodeFunc: func(content []byte) (codecScore, error) {
|
||||
var value codecScore
|
||||
return value, decodeStrictJSON(content, &value)
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func decodeStrictJSON(content []byte, out any) error {
|
||||
decoder := json.NewDecoder(bytes.NewReader(content))
|
||||
decoder.DisallowUnknownFields()
|
||||
if err := decoder.Decode(out); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := decoder.Decode(&struct{}{}); err != io.EOF {
|
||||
if err == nil {
|
||||
return fmt.Errorf("unexpected trailing JSON value")
|
||||
}
|
||||
return fmt.Errorf("decode trailing JSON: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -21,18 +21,18 @@ type CheckpointRecorder interface {
|
||||
SourceSucceeded(moduleKey string, doc *source.SourceDocument) error
|
||||
SourceFailed(moduleKey string, err error) error
|
||||
ChunkRunning(moduleKey string, sourceDigest string) error
|
||||
ChunkSucceeded(moduleKey string, sourceDigest string, chunks []contracts.SourceChunk, warnings []contracts.Warning) error
|
||||
ChunkSucceeded(moduleKey string, sourceDigest string, chunks []source.Chunk, warnings []contracts.Warning) error
|
||||
ChunkRejected(moduleKey string, sourceDigest string, rejected contracts.RejectedOutput) error
|
||||
ChunkFailed(moduleKey string, sourceDigest string, err error) error
|
||||
ExtractRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
||||
ExtractSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, outputs []contracts.ExtractOutput, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error
|
||||
ExtractSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error
|
||||
ExtractFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||
MergeRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
||||
MergeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output contracts.MergeOutput, warnings []contracts.Warning) error
|
||||
MergeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error
|
||||
MergeRejected(laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
||||
MergeFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||
NormalizeRunning(laneID string, moduleKey string, dependencies []CheckpointFingerprint) error
|
||||
NormalizeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output contracts.NormalizeOutput, warnings []contracts.Warning) error
|
||||
NormalizeSucceeded(laneID string, moduleKey string, dependencies []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error
|
||||
NormalizeRejected(laneID string, moduleKey string, dependencies []CheckpointFingerprint, rejected contracts.RejectedOutput) error
|
||||
NormalizeFailed(laneID string, moduleKey string, dependencies []CheckpointFingerprint, err error) error
|
||||
}
|
||||
@@ -55,23 +55,35 @@ type SourceCheckpoint struct {
|
||||
}
|
||||
|
||||
type ChunkCheckpoint struct {
|
||||
Chunks []contracts.SourceChunk
|
||||
Chunks []source.Chunk
|
||||
Warnings []contracts.Warning
|
||||
}
|
||||
|
||||
// CheckpointArtifact is the durable, domain-neutral value stored at a lane
|
||||
// checkpoint boundary.
|
||||
type CheckpointArtifact struct {
|
||||
LaneID string
|
||||
ModuleKey string
|
||||
SourceID string
|
||||
ChunkID string
|
||||
ChunkIndex int
|
||||
ChunkRef source.SourceRef
|
||||
Artifact contracts.SerializedArtifact
|
||||
SchemaDigest string
|
||||
}
|
||||
|
||||
type ExtractCheckpoint struct {
|
||||
Outputs []contracts.ExtractOutput
|
||||
Outputs []CheckpointArtifact
|
||||
Rejected []contracts.RejectedOutput
|
||||
Warnings []contracts.Warning
|
||||
}
|
||||
|
||||
type MergeCheckpoint struct {
|
||||
Output contracts.MergeOutput
|
||||
Output CheckpointArtifact
|
||||
Warnings []contracts.Warning
|
||||
}
|
||||
|
||||
type NormalizeCheckpoint struct {
|
||||
Output contracts.NormalizeOutput
|
||||
Output CheckpointArtifact
|
||||
Warnings []contracts.Warning
|
||||
}
|
||||
|
||||
@@ -94,7 +106,7 @@ func (noopCheckpointRecorder) SourceRunning(string) error
|
||||
func (noopCheckpointRecorder) SourceSucceeded(string, *source.SourceDocument) error { return nil }
|
||||
func (noopCheckpointRecorder) SourceFailed(string, error) error { return nil }
|
||||
func (noopCheckpointRecorder) ChunkRunning(string, string) error { return nil }
|
||||
func (noopCheckpointRecorder) ChunkSucceeded(string, string, []contracts.SourceChunk, []contracts.Warning) error {
|
||||
func (noopCheckpointRecorder) ChunkSucceeded(string, string, []source.Chunk, []contracts.Warning) error {
|
||||
return nil
|
||||
}
|
||||
func (noopCheckpointRecorder) ChunkRejected(string, string, contracts.RejectedOutput) error {
|
||||
@@ -104,14 +116,14 @@ func (noopCheckpointRecorder) ChunkFailed(string, string, error) error { return
|
||||
func (noopCheckpointRecorder) ExtractRunning(string, string, []CheckpointFingerprint) error {
|
||||
return nil
|
||||
}
|
||||
func (noopCheckpointRecorder) ExtractSucceeded(string, string, []CheckpointFingerprint, []contracts.ExtractOutput, []contracts.RejectedOutput, []contracts.Warning) error {
|
||||
func (noopCheckpointRecorder) ExtractSucceeded(string, string, []CheckpointFingerprint, []CheckpointArtifact, []contracts.RejectedOutput, []contracts.Warning) error {
|
||||
return nil
|
||||
}
|
||||
func (noopCheckpointRecorder) ExtractFailed(string, string, []CheckpointFingerprint, error) error {
|
||||
return nil
|
||||
}
|
||||
func (noopCheckpointRecorder) MergeRunning(string, string, []CheckpointFingerprint) error { return nil }
|
||||
func (noopCheckpointRecorder) MergeSucceeded(string, string, []CheckpointFingerprint, contracts.MergeOutput, []contracts.Warning) error {
|
||||
func (noopCheckpointRecorder) MergeSucceeded(string, string, []CheckpointFingerprint, CheckpointArtifact, []contracts.Warning) error {
|
||||
return nil
|
||||
}
|
||||
func (noopCheckpointRecorder) MergeRejected(string, string, []CheckpointFingerprint, contracts.RejectedOutput) error {
|
||||
@@ -123,7 +135,7 @@ func (noopCheckpointRecorder) MergeFailed(string, string, []CheckpointFingerprin
|
||||
func (noopCheckpointRecorder) NormalizeRunning(string, string, []CheckpointFingerprint) error {
|
||||
return nil
|
||||
}
|
||||
func (noopCheckpointRecorder) NormalizeSucceeded(string, string, []CheckpointFingerprint, contracts.NormalizeOutput, []contracts.Warning) error {
|
||||
func (noopCheckpointRecorder) NormalizeSucceeded(string, string, []CheckpointFingerprint, CheckpointArtifact, []contracts.Warning) error {
|
||||
return nil
|
||||
}
|
||||
func (noopCheckpointRecorder) NormalizeRejected(string, string, []CheckpointFingerprint, contracts.RejectedOutput) error {
|
||||
@@ -150,28 +162,6 @@ func (noopCheckpointLoader) Normalize(string, string, []CheckpointFingerprint) (
|
||||
return NormalizeCheckpoint{}, CheckpointDecision{Reason: "checkpoint loading disabled"}
|
||||
}
|
||||
|
||||
func rawOutputDigests(payloads []contracts.RawPayload) []CheckpointFingerprint {
|
||||
values := make([]CheckpointFingerprint, 0, len(payloads))
|
||||
for i, payload := range payloads {
|
||||
values = append(values, CheckpointFingerprint{
|
||||
Name: fmt.Sprintf("payload[%d]", i),
|
||||
Value: checkpointContentDigest(payload.Content),
|
||||
})
|
||||
}
|
||||
return normalizeCheckpointFingerprints(values)
|
||||
}
|
||||
|
||||
func extractPayloads(outputs []contracts.ExtractOutput) []contracts.RawPayload {
|
||||
if len(outputs) == 0 {
|
||||
return nil
|
||||
}
|
||||
payloads := make([]contracts.RawPayload, 0, len(outputs))
|
||||
for _, output := range outputs {
|
||||
payloads = append(payloads, output.Payload)
|
||||
}
|
||||
return payloads
|
||||
}
|
||||
|
||||
func digestFingerprints(name string, digest string) []CheckpointFingerprint {
|
||||
digest = strings.TrimSpace(digest)
|
||||
if digest == "" {
|
||||
@@ -180,17 +170,21 @@ func digestFingerprints(name string, digest string) []CheckpointFingerprint {
|
||||
return []CheckpointFingerprint{{Name: name, Value: digest}}
|
||||
}
|
||||
|
||||
func joinedChunkDigest(chunks []contracts.SourceChunk) string {
|
||||
func joinedChunkDigest(chunks []source.Chunk) (string, error) {
|
||||
if len(chunks) == 0 {
|
||||
return ""
|
||||
return "", nil
|
||||
}
|
||||
values := make([]string, 0, len(chunks))
|
||||
for _, chunk := range chunks {
|
||||
values = append(values, chunk.ID+"="+checkpointContentDigest(chunk.Content))
|
||||
digest, err := source.DigestChunk(chunk)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("digest chunk %q: %w", chunk.ID, err)
|
||||
}
|
||||
values = append(values, chunk.ID+"="+digest)
|
||||
}
|
||||
sort.Strings(values)
|
||||
sum := sha256.Sum256([]byte(strings.Join(values, "\n")))
|
||||
return "sha256:" + hex.EncodeToString(sum[:])
|
||||
return "sha256:" + hex.EncodeToString(sum[:]), nil
|
||||
}
|
||||
|
||||
func normalizeCheckpointFingerprints(values []CheckpointFingerprint) []CheckpointFingerprint {
|
||||
|
||||
@@ -5,10 +5,9 @@ import (
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []contracts.SourceChunk) ([]contracts.SourceChunk, error) {
|
||||
func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []source.Chunk) ([]source.Chunk, error) {
|
||||
if len(chunks) == 0 {
|
||||
return nil, fmt.Errorf("chunks must not be empty")
|
||||
}
|
||||
@@ -20,7 +19,7 @@ func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []con
|
||||
sourceUnits[unit.ID] = unit
|
||||
}
|
||||
|
||||
canonicalChunks := make([]contracts.SourceChunk, 0, len(chunks))
|
||||
canonicalChunks := make([]source.Chunk, 0, len(chunks))
|
||||
seenChunkIDs := make(map[string]struct{}, len(chunks))
|
||||
for chunkIndex, chunk := range chunks {
|
||||
if strings.TrimSpace(chunk.ID) == "" {
|
||||
@@ -37,17 +36,6 @@ func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []con
|
||||
if chunk.Index != chunkIndex {
|
||||
return nil, fmt.Errorf("chunk %q index %d does not match returned order %d", chunk.ID, chunk.Index, chunkIndex)
|
||||
}
|
||||
startIndex, ok := sourceUnitIndexes[chunk.StartUnitID]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("chunk %q start_unit_id %d was not found in source document %q", chunk.ID, chunk.StartUnitID, doc.ID)
|
||||
}
|
||||
endIndex, ok := sourceUnitIndexes[chunk.EndUnitID]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("chunk %q end_unit_id %d was not found in source document %q", chunk.ID, chunk.EndUnitID, doc.ID)
|
||||
}
|
||||
if startIndex > endIndex {
|
||||
return nil, fmt.Errorf("chunk %q start_unit_id %d appears after end_unit_id %d", chunk.ID, chunk.StartUnitID, chunk.EndUnitID)
|
||||
}
|
||||
if len(chunk.Units) == 0 {
|
||||
return nil, fmt.Errorf("chunk %q units must not be empty", chunk.ID)
|
||||
}
|
||||
@@ -57,6 +45,9 @@ func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []con
|
||||
if strings.TrimSpace(chunk.MediaType) == "" {
|
||||
return nil, fmt.Errorf("chunk %q media_type must not be empty", chunk.ID)
|
||||
}
|
||||
if err := source.ValidateRef(doc, chunk.Ref); err != nil {
|
||||
return nil, fmt.Errorf("chunk %q ref: %w", chunk.ID, err)
|
||||
}
|
||||
|
||||
seenUnitIDs := make(map[int]struct{}, len(chunk.Units))
|
||||
previousSourceIndex := -1
|
||||
@@ -74,23 +65,33 @@ func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []con
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("chunk %q source unit %d was not found in source document %q", chunk.ID, unit.ID, doc.ID)
|
||||
}
|
||||
if sourceIndex <= previousSourceIndex {
|
||||
return nil, fmt.Errorf("chunk %q source units must appear in source document order", chunk.ID)
|
||||
if previousSourceIndex >= 0 && sourceIndex != previousSourceIndex+1 {
|
||||
return nil, fmt.Errorf("chunk %q source units must form a contiguous range in source document order", chunk.ID)
|
||||
}
|
||||
if unit.Ref != sourceUnits[unit.ID].Ref {
|
||||
return nil, fmt.Errorf("chunk %q source unit %d ref does not match source document", chunk.ID, unit.ID)
|
||||
}
|
||||
previousSourceIndex = sourceIndex
|
||||
canonicalUnits = append(canonicalUnits, cloneSourceUnit(sourceUnits[unit.ID]))
|
||||
}
|
||||
expectedRef := source.SourceRef{
|
||||
SourceID: doc.ID,
|
||||
StartUnitID: canonicalUnits[0].Ref.StartUnitID,
|
||||
EndUnitID: canonicalUnits[len(canonicalUnits)-1].Ref.EndUnitID,
|
||||
}
|
||||
if chunk.Ref != expectedRef {
|
||||
return nil, fmt.Errorf("chunk %q ref %#v does not match unit span %#v", chunk.ID, chunk.Ref, expectedRef)
|
||||
}
|
||||
|
||||
canonicalChunks = append(canonicalChunks, contracts.SourceChunk{
|
||||
ID: chunk.ID,
|
||||
SourceID: chunk.SourceID,
|
||||
Index: chunk.Index,
|
||||
StartUnitID: chunk.StartUnitID,
|
||||
EndUnitID: chunk.EndUnitID,
|
||||
Content: append([]byte(nil), chunk.Content...),
|
||||
MediaType: chunk.MediaType,
|
||||
Units: canonicalUnits,
|
||||
Metadata: cloneMetadata(chunk.Metadata),
|
||||
canonicalChunks = append(canonicalChunks, source.Chunk{
|
||||
ID: chunk.ID,
|
||||
SourceID: chunk.SourceID,
|
||||
Index: chunk.Index,
|
||||
Ref: expectedRef,
|
||||
Content: append([]byte(nil), chunk.Content...),
|
||||
MediaType: chunk.MediaType,
|
||||
Units: canonicalUnits,
|
||||
Metadata: cloneMetadata(chunk.Metadata),
|
||||
})
|
||||
}
|
||||
|
||||
@@ -102,6 +103,7 @@ func cloneSourceUnit(unit source.SourceUnit) source.SourceUnit {
|
||||
ID: unit.ID,
|
||||
Kind: unit.Kind,
|
||||
Text: unit.Text,
|
||||
Ref: unit.Ref,
|
||||
Metadata: cloneMetadata(unit.Metadata),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -8,16 +8,19 @@ import (
|
||||
)
|
||||
|
||||
type ChunkerConstructor func() (contracts.Chunker, error)
|
||||
type ChunkerBuilder func(BuildRequest) (contracts.Chunker, error)
|
||||
|
||||
type ChunkerRegistry struct {
|
||||
constructors map[string]ChunkerConstructor
|
||||
specs map[string]ModuleSpec
|
||||
builders map[string]ChunkerBuilder
|
||||
optionValidators map[string]OptionValidator
|
||||
specs map[string]ModuleSpec
|
||||
}
|
||||
|
||||
func NewChunkerRegistry() *ChunkerRegistry {
|
||||
return &ChunkerRegistry{
|
||||
constructors: make(map[string]ChunkerConstructor),
|
||||
specs: make(map[string]ModuleSpec),
|
||||
builders: make(map[string]ChunkerBuilder),
|
||||
optionValidators: make(map[string]OptionValidator),
|
||||
specs: make(map[string]ModuleSpec),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -26,6 +29,15 @@ func (r *ChunkerRegistry) Register(key string, constructor ChunkerConstructor) e
|
||||
}
|
||||
|
||||
func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerConstructor) error {
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("chunker constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||
}
|
||||
return r.RegisterBuilderWithSpec(spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.Chunker, error) {
|
||||
return constructor()
|
||||
})
|
||||
}
|
||||
|
||||
func (r *ChunkerRegistry) RegisterBuilderWithSpec(spec ModuleSpec, validateOptions OptionValidator, builder ChunkerBuilder) error {
|
||||
if r == nil {
|
||||
return fmt.Errorf("chunker registry must not be nil")
|
||||
}
|
||||
@@ -34,25 +46,36 @@ func (r *ChunkerRegistry) RegisterWithSpec(spec ModuleSpec, constructor ChunkerC
|
||||
if err := validateModuleSpec("chunker", StageChunk, normalizedSpec); err != nil {
|
||||
return err
|
||||
}
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("chunker constructor for %q must not be nil", normalizedSpec.Key)
|
||||
if validateOptions == nil {
|
||||
return fmt.Errorf("chunker option validator for %q must not be nil", normalizedSpec.Key)
|
||||
}
|
||||
if _, ok := r.constructors[normalizedSpec.Key]; ok {
|
||||
if builder == nil {
|
||||
return fmt.Errorf("chunker builder for %q must not be nil", normalizedSpec.Key)
|
||||
}
|
||||
if _, ok := r.builders[normalizedSpec.Key]; ok {
|
||||
return fmt.Errorf("chunker %q is already registered", normalizedSpec.Key)
|
||||
}
|
||||
|
||||
if r.constructors == nil {
|
||||
r.constructors = make(map[string]ChunkerConstructor)
|
||||
if r.builders == nil {
|
||||
r.builders = make(map[string]ChunkerBuilder)
|
||||
}
|
||||
if r.optionValidators == nil {
|
||||
r.optionValidators = make(map[string]OptionValidator)
|
||||
}
|
||||
if r.specs == nil {
|
||||
r.specs = make(map[string]ModuleSpec)
|
||||
}
|
||||
r.constructors[normalizedSpec.Key] = constructor
|
||||
r.builders[normalizedSpec.Key] = builder
|
||||
r.optionValidators[normalizedSpec.Key] = validateOptions
|
||||
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *ChunkerRegistry) Build(key string) (contracts.Chunker, error) {
|
||||
return r.BuildWithRequest(key, BuildRequest{})
|
||||
}
|
||||
|
||||
func (r *ChunkerRegistry) BuildWithRequest(key string, request BuildRequest) (contracts.Chunker, error) {
|
||||
if r == nil {
|
||||
return nil, fmt.Errorf("chunker registry must not be nil")
|
||||
}
|
||||
@@ -62,12 +85,12 @@ func (r *ChunkerRegistry) Build(key string) (contracts.Chunker, error) {
|
||||
return nil, fmt.Errorf("chunker key must not be empty")
|
||||
}
|
||||
|
||||
constructor, ok := r.constructors[normalizedKey]
|
||||
builder, ok := r.builders[normalizedKey]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("chunker %q is not registered", normalizedKey)
|
||||
}
|
||||
|
||||
chunker, err := constructor()
|
||||
chunker, err := builder(cloneBuildRequest(request))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("build chunker %q: %w", normalizedKey, err)
|
||||
}
|
||||
@@ -81,6 +104,18 @@ func (r *ChunkerRegistry) Build(key string) (contracts.Chunker, error) {
|
||||
return chunker, nil
|
||||
}
|
||||
|
||||
func (r *ChunkerRegistry) ValidateOptions(key string, options map[string]any) error {
|
||||
if r == nil {
|
||||
return fmt.Errorf("chunker registry must not be nil")
|
||||
}
|
||||
normalizedKey := strings.TrimSpace(key)
|
||||
validator, ok := r.optionValidators[normalizedKey]
|
||||
if !ok {
|
||||
return fmt.Errorf("chunker %q is not registered", normalizedKey)
|
||||
}
|
||||
return validateRegisteredOptions(validator, options)
|
||||
}
|
||||
|
||||
func (r *ChunkerRegistry) Spec(key string) (ModuleSpec, bool) {
|
||||
if r == nil {
|
||||
return ModuleSpec{}, false
|
||||
@@ -98,5 +133,5 @@ func (r *ChunkerRegistry) RegisteredKeys() []string {
|
||||
return nil
|
||||
}
|
||||
|
||||
return sortedRegistryKeys(r.constructors)
|
||||
return sortedRegistryKeys(r.builders)
|
||||
}
|
||||
|
||||
@@ -338,34 +338,6 @@ func (chunker registryChunker) Chunk(ctx context.Context, req contracts.ChunkReq
|
||||
return contracts.ChunkResult{}, nil
|
||||
}
|
||||
|
||||
type registryMerger struct {
|
||||
key string
|
||||
}
|
||||
|
||||
func (merger registryMerger) Key() string {
|
||||
return merger.key
|
||||
}
|
||||
|
||||
func (merger registryMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
|
||||
return contracts.MergeResult{}, nil
|
||||
}
|
||||
|
||||
type registryNormalizer struct {
|
||||
key string
|
||||
}
|
||||
|
||||
func (normalizer registryNormalizer) Key() string {
|
||||
return normalizer.key
|
||||
}
|
||||
|
||||
func (normalizer registryNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (normalizer registryNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
|
||||
return contracts.NormalizeResult{}, nil
|
||||
}
|
||||
|
||||
type registryOutputEncoder struct {
|
||||
key string
|
||||
}
|
||||
@@ -377,19 +349,3 @@ func (encoder registryOutputEncoder) Key() string {
|
||||
func (encoder registryOutputEncoder) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
|
||||
return contracts.OutputResult{}, nil
|
||||
}
|
||||
|
||||
type registryValidator struct {
|
||||
name string
|
||||
}
|
||||
|
||||
func (validator registryValidator) Name() string {
|
||||
return validator.name
|
||||
}
|
||||
|
||||
func (validator registryValidator) ExecutionClass() contracts.ExecutionClass {
|
||||
return contracts.ExecutionClassDeterministic
|
||||
}
|
||||
|
||||
func (validator registryValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}
|
||||
|
||||
69
internal/framework/pipeline/construction.go
Normal file
69
internal/framework/pipeline/construction.go
Normal file
@@ -0,0 +1,69 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
// ModuleDependencies contains run-scoped collaborators shared by constructed
|
||||
// modules. Implementations retain only the dependencies they use.
|
||||
type ModuleDependencies struct {
|
||||
LLM contracts.StructuredLLMClient
|
||||
}
|
||||
|
||||
// BuildRequest contains the stable dependencies and configured options used to
|
||||
// construct one module or validator for a run.
|
||||
type BuildRequest struct {
|
||||
Dependencies ModuleDependencies
|
||||
Options map[string]any
|
||||
}
|
||||
|
||||
// OptionValidator validates one module binding without constructing it.
|
||||
type OptionValidator func(map[string]any) error
|
||||
|
||||
func rejectUnconfiguredOptions(options map[string]any) error {
|
||||
return RejectUnknownOptions(options)
|
||||
}
|
||||
|
||||
func validateRegisteredOptions(validator OptionValidator, options map[string]any) error {
|
||||
if validator == nil {
|
||||
return fmt.Errorf("option validator must not be nil")
|
||||
}
|
||||
return validator(cloneOptions(options))
|
||||
}
|
||||
|
||||
func cloneBuildRequest(request BuildRequest) BuildRequest {
|
||||
return BuildRequest{
|
||||
Dependencies: request.Dependencies,
|
||||
Options: cloneOptions(request.Options),
|
||||
}
|
||||
}
|
||||
|
||||
// RejectUnknownOptions provides the common strict-map check used by module-
|
||||
// owned option decoders. Values remain the implementation's responsibility.
|
||||
func RejectUnknownOptions(options map[string]any, allowed ...string) error {
|
||||
known := make(map[string]struct{}, len(allowed))
|
||||
for _, key := range allowed {
|
||||
key = strings.TrimSpace(key)
|
||||
if key != "" {
|
||||
known[key] = struct{}{}
|
||||
}
|
||||
}
|
||||
unknown := make([]string, 0)
|
||||
for key := range options {
|
||||
if _, ok := known[key]; !ok {
|
||||
unknown = append(unknown, key)
|
||||
}
|
||||
}
|
||||
if len(unknown) == 0 {
|
||||
return nil
|
||||
}
|
||||
sort.Strings(unknown)
|
||||
if len(unknown) == 1 {
|
||||
return fmt.Errorf("unknown option %q", unknown[0])
|
||||
}
|
||||
return fmt.Errorf("unknown options %q", unknown)
|
||||
}
|
||||
@@ -82,10 +82,6 @@ type debugBinaryEnvelope struct {
|
||||
Warnings []contracts.Warning `json:"warnings,omitempty"`
|
||||
}
|
||||
|
||||
type debugRawPayload struct {
|
||||
Content debugBinaryEnvelope `json:"content"`
|
||||
}
|
||||
|
||||
type debugSourceInput struct {
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
Path string `json:"path,omitempty"`
|
||||
@@ -104,40 +100,23 @@ type debugSourceDocument struct {
|
||||
}
|
||||
|
||||
type debugSourceChunk struct {
|
||||
ID string `json:"id"`
|
||||
SourceID string `json:"source_id"`
|
||||
Index int `json:"index"`
|
||||
StartUnitID int `json:"start_unit_id"`
|
||||
EndUnitID int `json:"end_unit_id"`
|
||||
Content debugBinaryEnvelope `json:"content"`
|
||||
Units []source.SourceUnit `json:"units,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
ID string `json:"id"`
|
||||
SourceID string `json:"source_id"`
|
||||
Index int `json:"index"`
|
||||
Ref source.SourceRef `json:"ref"`
|
||||
Content debugBinaryEnvelope `json:"content"`
|
||||
Units []source.SourceUnit `json:"units,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
}
|
||||
|
||||
type debugExtractOutput struct {
|
||||
LaneID string `json:"lane_id"`
|
||||
ExtractorKey string `json:"extractor_key"`
|
||||
SourceID string `json:"source_id"`
|
||||
ChunkID string `json:"chunk_id"`
|
||||
ChunkIndex int `json:"chunk_index"`
|
||||
Schema contracts.ResponseSchema `json:"schema,omitempty"`
|
||||
Payload debugBinaryEnvelope `json:"payload"`
|
||||
}
|
||||
|
||||
type debugMergeOutput struct {
|
||||
LaneID string `json:"lane_id"`
|
||||
MergerKey string `json:"merger_key"`
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
Schema contracts.ResponseSchema `json:"schema,omitempty"`
|
||||
Payload debugBinaryEnvelope `json:"payload"`
|
||||
}
|
||||
|
||||
type debugNormalizeOutput struct {
|
||||
type debugSerializedOutput struct {
|
||||
LaneID string `json:"lane_id"`
|
||||
NormalizerKey string `json:"normalizer_key"`
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
Schema contracts.ResponseSchema `json:"schema,omitempty"`
|
||||
Payload debugBinaryEnvelope `json:"payload"`
|
||||
Kind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
|
||||
Schema contracts.ArtifactSchema `json:"schema"`
|
||||
SchemaDigest string `json:"schema_digest"`
|
||||
Content debugBinaryEnvelope `json:"content"`
|
||||
}
|
||||
|
||||
type debugLLMInputMaterial struct {
|
||||
@@ -193,28 +172,9 @@ type debugLLMCallReference struct {
|
||||
Error bool `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type debugValidationRequest struct {
|
||||
Stage string `json:"stage"`
|
||||
LaneID string `json:"lane_id,omitempty"`
|
||||
ModuleKey string `json:"module_key"`
|
||||
SourceID string `json:"source_id,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
LLMProfile string `json:"llm_profile,omitempty"`
|
||||
Options map[string]any `json:"options,omitempty"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
Schema contracts.ResponseSchema `json:"schema,omitempty"`
|
||||
Payload *debugBinaryEnvelope `json:"payload,omitempty"`
|
||||
ChunkID string `json:"chunk_id,omitempty"`
|
||||
ChunkIndex int `json:"chunk_index,omitempty"`
|
||||
Chunk *debugSourceChunk `json:"chunk,omitempty"`
|
||||
Chunks []debugSourceChunk `json:"chunks,omitempty"`
|
||||
ExtractOutputs []debugExtractOutput `json:"extract_outputs,omitempty"`
|
||||
MergeOutput *debugMergeOutput `json:"merge_output,omitempty"`
|
||||
}
|
||||
|
||||
type debugValidationCall struct {
|
||||
ValidatorName string `json:"validator_name"`
|
||||
Request debugValidationRequest `json:"request"`
|
||||
Request any `json:"request"`
|
||||
Result contracts.ValidationResult `json:"result,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
@@ -239,7 +199,16 @@ func wrapDebugLLMClient(client contracts.StructuredLLMClient, recorder DebugReco
|
||||
if client == nil || recorder == nil || !recorder.Enabled() {
|
||||
return client
|
||||
}
|
||||
return &debugLLMClient{inner: client, recorder: recorder}
|
||||
if _, ok := client.(*debugLLMClient); ok {
|
||||
return client
|
||||
}
|
||||
return &debugLLMClient{inner: client, recorder: synchronizedDebugRecorder(recorder)}
|
||||
}
|
||||
|
||||
// WithDebugLLMRecording decorates a shared LLM client so calls made by
|
||||
// construction-injected modules participate in the run's debug recording.
|
||||
func WithDebugLLMRecording(client contracts.StructuredLLMClient, recorder DebugRecorder) contracts.StructuredLLMClient {
|
||||
return wrapDebugLLMClient(client, recorder)
|
||||
}
|
||||
|
||||
func (client *debugLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
@@ -430,10 +399,6 @@ func debugContentEnvelope(content []byte, mediaType string, metadata map[string]
|
||||
}
|
||||
}
|
||||
|
||||
func debugPayloadEnvelope(payload contracts.RawPayload) debugBinaryEnvelope {
|
||||
return debugContentEnvelope(payload.Content, payload.MediaType, payload.Metadata, payload.Warnings)
|
||||
}
|
||||
|
||||
func debugSourceDocumentEnvelope(doc *source.SourceDocument) *debugSourceDocument {
|
||||
if doc == nil {
|
||||
return nil
|
||||
@@ -448,20 +413,19 @@ func debugSourceDocumentEnvelope(doc *source.SourceDocument) *debugSourceDocumen
|
||||
}
|
||||
}
|
||||
|
||||
func debugSourceChunkEnvelope(chunk contracts.SourceChunk) debugSourceChunk {
|
||||
func debugSourceChunkEnvelope(chunk source.Chunk) debugSourceChunk {
|
||||
return debugSourceChunk{
|
||||
ID: chunk.ID,
|
||||
SourceID: chunk.SourceID,
|
||||
Index: chunk.Index,
|
||||
StartUnitID: chunk.StartUnitID,
|
||||
EndUnitID: chunk.EndUnitID,
|
||||
Content: debugContentEnvelope(chunk.Content, chunk.MediaType, chunk.Metadata, nil),
|
||||
Units: cloneSourceUnits(chunk.Units),
|
||||
Metadata: redactSensitiveMap(chunk.Metadata),
|
||||
ID: chunk.ID,
|
||||
SourceID: chunk.SourceID,
|
||||
Index: chunk.Index,
|
||||
Ref: chunk.Ref,
|
||||
Content: debugContentEnvelope(chunk.Content, chunk.MediaType, chunk.Metadata, nil),
|
||||
Units: cloneSourceUnits(chunk.Units),
|
||||
Metadata: redactSensitiveMap(chunk.Metadata),
|
||||
}
|
||||
}
|
||||
|
||||
func debugSourceChunkEnvelopes(chunks []contracts.SourceChunk) []debugSourceChunk {
|
||||
func debugSourceChunkEnvelopes(chunks []source.Chunk) []debugSourceChunk {
|
||||
if len(chunks) == 0 {
|
||||
return nil
|
||||
}
|
||||
@@ -472,59 +436,26 @@ func debugSourceChunkEnvelopes(chunks []contracts.SourceChunk) []debugSourceChun
|
||||
return out
|
||||
}
|
||||
|
||||
func debugExtractOutputEnvelope(output contracts.ExtractOutput) debugExtractOutput {
|
||||
output.Schema.JSONSchema = nil
|
||||
return debugExtractOutput{
|
||||
LaneID: output.LaneID,
|
||||
ExtractorKey: output.ExtractorKey,
|
||||
SourceID: output.SourceID,
|
||||
ChunkID: output.ChunkID,
|
||||
ChunkIndex: output.ChunkIndex,
|
||||
Schema: output.Schema,
|
||||
Payload: debugPayloadEnvelope(output.Payload),
|
||||
func debugSerializedOutputEnvelope(output contracts.SerializedOutput) debugSerializedOutput {
|
||||
schema := contracts.CloneArtifactSchema(output.Artifact.Schema)
|
||||
digest := contracts.DigestArtifactSchema(schema)
|
||||
schema.JSONSchema = nil
|
||||
content := debugContentEnvelope(output.Artifact.Content, output.Artifact.MediaType, output.Artifact.Metadata, nil)
|
||||
content.ContentDigest = debugContentDigest(output.Artifact.Content)
|
||||
return debugSerializedOutput{
|
||||
LaneID: output.LaneID, NormalizerKey: output.NormalizerKey, SourceID: output.SourceID,
|
||||
Kind: output.Artifact.Kind, Schema: schema, SchemaDigest: digest,
|
||||
Content: content,
|
||||
}
|
||||
}
|
||||
|
||||
func debugExtractOutputEnvelopes(outputs []contracts.ExtractOutput) []debugExtractOutput {
|
||||
func debugSerializedOutputEnvelopes(outputs []contracts.SerializedOutput) []debugSerializedOutput {
|
||||
if len(outputs) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]debugExtractOutput, 0, len(outputs))
|
||||
out := make([]debugSerializedOutput, 0, len(outputs))
|
||||
for _, output := range outputs {
|
||||
out = append(out, debugExtractOutputEnvelope(output))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func debugMergeOutputEnvelope(output contracts.MergeOutput) debugMergeOutput {
|
||||
output.Schema.JSONSchema = nil
|
||||
return debugMergeOutput{
|
||||
LaneID: output.LaneID,
|
||||
MergerKey: output.MergerKey,
|
||||
SourceID: output.SourceID,
|
||||
Schema: output.Schema,
|
||||
Payload: debugPayloadEnvelope(output.Payload),
|
||||
}
|
||||
}
|
||||
|
||||
func debugNormalizeOutputEnvelope(output contracts.NormalizeOutput) debugNormalizeOutput {
|
||||
output.Schema.JSONSchema = nil
|
||||
return debugNormalizeOutput{
|
||||
LaneID: output.LaneID,
|
||||
NormalizerKey: output.NormalizerKey,
|
||||
SourceID: output.SourceID,
|
||||
Schema: output.Schema,
|
||||
Payload: debugPayloadEnvelope(output.Payload),
|
||||
}
|
||||
}
|
||||
|
||||
func debugNormalizeOutputEnvelopes(outputs []contracts.NormalizeOutput) []debugNormalizeOutput {
|
||||
if len(outputs) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]debugNormalizeOutput, 0, len(outputs))
|
||||
for _, output := range outputs {
|
||||
out = append(out, debugNormalizeOutputEnvelope(output))
|
||||
out = append(out, debugSerializedOutputEnvelope(output))
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -673,36 +604,6 @@ func debugResponseModel(response contracts.StructuredCompletionResponse) string
|
||||
return response.Debug.Response.ModelName
|
||||
}
|
||||
|
||||
func debugValidationRequestEnvelope(req contracts.ValidationRequest) debugValidationRequest {
|
||||
req.Schema.JSONSchema = nil
|
||||
out := debugValidationRequest{
|
||||
Stage: req.Stage,
|
||||
LaneID: req.LaneID,
|
||||
ModuleKey: req.ModuleKey,
|
||||
SourceID: req.SourceID,
|
||||
SessionID: req.SessionID,
|
||||
LLMProfile: req.LLMProfile,
|
||||
Options: redactSensitiveMap(req.Options),
|
||||
Metadata: redactSensitiveMap(req.Metadata),
|
||||
Schema: req.Schema,
|
||||
ChunkID: req.ChunkID,
|
||||
ChunkIndex: req.ChunkIndex,
|
||||
}
|
||||
payload := debugPayloadEnvelope(req.Payload)
|
||||
out.Payload = &payload
|
||||
if req.Chunk != nil {
|
||||
chunk := debugSourceChunkEnvelope(*req.Chunk)
|
||||
out.Chunk = &chunk
|
||||
}
|
||||
out.Chunks = debugSourceChunkEnvelopes(req.Chunks)
|
||||
out.ExtractOutputs = debugExtractOutputEnvelopes(req.ExtractOutputs)
|
||||
if len(req.MergeOutput.Payload.Content) > 0 || req.MergeOutput.LaneID != "" {
|
||||
merge := debugMergeOutputEnvelope(req.MergeOutput)
|
||||
out.MergeOutput = &merge
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func debugValidationResultEnvelope(result contracts.ValidationResult) contracts.ValidationResult {
|
||||
result.Message = string(redactSecretBytes([]byte(result.Message)))
|
||||
result.DiagnosticArtifactPath = string(redactSecretBytes([]byte(result.DiagnosticArtifactPath)))
|
||||
@@ -712,6 +613,14 @@ func debugValidationResultEnvelope(result contracts.ValidationResult) contracts.
|
||||
return result
|
||||
}
|
||||
|
||||
func debugWarningEnvelopes(warnings []contracts.Warning) []contracts.Warning {
|
||||
out := cloneWarnings(warnings)
|
||||
for i := range out {
|
||||
out[i].Message = string(redactSecretBytes([]byte(out[i].Message)))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func debugRejectedOutputEnvelope(rejected contracts.RejectedOutput) contracts.RejectedOutput {
|
||||
rejected.Message = string(redactSecretBytes([]byte(rejected.Message)))
|
||||
rejected.DiagnosticArtifactPath = string(redactSecretBytes([]byte(rejected.DiagnosticArtifactPath)))
|
||||
|
||||
51
internal/framework/pipeline/debug_test.go
Normal file
51
internal/framework/pipeline/debug_test.go
Normal file
@@ -0,0 +1,51 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
)
|
||||
|
||||
func TestDebugSourceDocumentPreservesUnitReferences(t *testing.T) {
|
||||
doc := validSourceDocument()
|
||||
envelope := debugSourceDocumentEnvelope(doc)
|
||||
encoded, err := json.Marshal(envelope)
|
||||
if err != nil {
|
||||
t.Fatalf("Marshal(debug source document) error = %v, want nil", err)
|
||||
}
|
||||
|
||||
var decoded debugSourceDocument
|
||||
if err := json.Unmarshal(encoded, &decoded); err != nil {
|
||||
t.Fatalf("Unmarshal(debug source document) error = %v, want nil", err)
|
||||
}
|
||||
if got, want := decoded.Units[0].Ref, (source.SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1}); got != want {
|
||||
t.Fatalf("debug source unit ref = %#v, want %#v", got, want)
|
||||
}
|
||||
|
||||
envelope.Units[0].Ref.SourceID = "mutated"
|
||||
if got := doc.Units[0].Ref.SourceID; got != "source-1" {
|
||||
t.Fatalf("source document ref = %q after debug mutation, want source-1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDebugSourceChunkPreservesReference(t *testing.T) {
|
||||
doc := validSourceDocument()
|
||||
chunk := source.Chunk{
|
||||
ID: "chunk-1", SourceID: doc.ID, Index: 0,
|
||||
Ref: source.SourceRef{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 1},
|
||||
Content: []byte("chunk content"), MediaType: "text/plain", Units: doc.Units[:1],
|
||||
}
|
||||
envelope := debugSourceChunkEnvelope(chunk)
|
||||
encoded, err := json.Marshal(envelope)
|
||||
if err != nil {
|
||||
t.Fatalf("Marshal(debug source chunk) error = %v, want nil", err)
|
||||
}
|
||||
var decoded debugSourceChunk
|
||||
if err := json.Unmarshal(encoded, &decoded); err != nil {
|
||||
t.Fatalf("Unmarshal(debug source chunk) error = %v, want nil", err)
|
||||
}
|
||||
if decoded.Ref != chunk.Ref {
|
||||
t.Fatalf("debug chunk ref = %#v, want %#v", decoded.Ref, chunk.Ref)
|
||||
}
|
||||
}
|
||||
@@ -2,16 +2,17 @@ package pipeline_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/chunk/generic"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/merge/appendorder"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/normalize/noop"
|
||||
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/output/json"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/chunk/units"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/merge/appendorder"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
|
||||
jsonoutput "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/output/json"
|
||||
)
|
||||
|
||||
func TestPipelineConfigResolvesWithProductionDefaultsRegistered(t *testing.T) {
|
||||
@@ -34,8 +35,8 @@ func TestPipelineConfigResolvesWithProductionDefaultsRegistered(t *testing.T) {
|
||||
}
|
||||
|
||||
pipeline := resolved.ResolvedPipeline
|
||||
if pipeline.Chunk.Module != generic.Key {
|
||||
t.Fatalf("Chunk.Module = %q, want %q", pipeline.Chunk.Module, generic.Key)
|
||||
if pipeline.Chunk.Module != units.Key {
|
||||
t.Fatalf("Chunk.Module = %q, want %q", pipeline.Chunk.Module, units.Key)
|
||||
}
|
||||
if pipeline.Output.Module != jsonoutput.Key {
|
||||
t.Fatalf("Output.Module = %q, want %q", pipeline.Output.Module, jsonoutput.Key)
|
||||
@@ -68,32 +69,43 @@ func defaultModuleCatalog(t *testing.T) pipeline.ModuleCatalog {
|
||||
}); err != nil {
|
||||
t.Fatalf("register input: %v", err)
|
||||
}
|
||||
if err := generic.Register(chunkers); err != nil {
|
||||
if err := units.Register(chunkers); err != nil {
|
||||
t.Fatalf("register generic chunker: %v", err)
|
||||
}
|
||||
if err := extractors.RegisterWithSpec(pipeline.ModuleSpec{
|
||||
Key: "extract",
|
||||
Stage: pipeline.StageExtract,
|
||||
Requires: []string{"chunks"},
|
||||
Provides: []string{"records"},
|
||||
}, func() (contracts.Extractor, error) {
|
||||
if err := pipeline.RegisterExtractor[defaultArtifact](extractors, pipeline.ModuleSpec{
|
||||
Key: "extract",
|
||||
Stage: pipeline.StageExtract,
|
||||
ArtifactKind: defaultArtifactKind,
|
||||
Requires: []string{"chunks"},
|
||||
Provides: []string{"records"},
|
||||
}, func() (contracts.Extractor[defaultArtifact], error) {
|
||||
return defaultExtractor{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register extractor: %v", err)
|
||||
}
|
||||
if err := appendorder.Register(mergers); err != nil {
|
||||
if err := appendorder.RegisterTyped(mergers, defaultArtifactKind, func(values []defaultArtifact) (defaultArtifact, error) {
|
||||
if len(values) == 0 {
|
||||
return defaultArtifact{}, nil
|
||||
}
|
||||
return values[0], nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register appendorder merger: %v", err)
|
||||
}
|
||||
if err := noop.Register(normalizers); err != nil {
|
||||
if err := noop.RegisterTyped[defaultArtifact](normalizers, defaultArtifactKind); err != nil {
|
||||
t.Fatalf("register noop normalizer: %v", err)
|
||||
}
|
||||
if err := jsonoutput.Register(outputs); err != nil {
|
||||
t.Fatalf("register json output: %v", err)
|
||||
}
|
||||
|
||||
codecs := pipeline.NewArtifactCodecRegistry()
|
||||
if err := pipeline.RegisterArtifactCodec(codecs, defaultArtifactCodec{}); err != nil {
|
||||
t.Fatalf("register artifact codec: %v", err)
|
||||
}
|
||||
return pipeline.ModuleCatalog{
|
||||
Inputs: inputs,
|
||||
Chunkers: chunkers,
|
||||
ArtifactCodecs: codecs,
|
||||
Extractors: extractors,
|
||||
Mergers: mergers,
|
||||
Normalizers: normalizers,
|
||||
@@ -116,6 +128,25 @@ func (defaultExtractor) Key() string { return "extract" }
|
||||
|
||||
func (defaultExtractor) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||
|
||||
func (defaultExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
|
||||
return contracts.ExtractionResult{}, nil
|
||||
func (defaultExtractor) Extract(ctx context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[defaultArtifact], error) {
|
||||
return contracts.TypedExtractionResult[defaultArtifact]{}, nil
|
||||
}
|
||||
|
||||
const defaultArtifactKind contracts.ArtifactKind = "test/default"
|
||||
|
||||
type defaultArtifact struct {
|
||||
Value string `json:"value"`
|
||||
}
|
||||
type defaultArtifactCodec struct{}
|
||||
|
||||
func (defaultArtifactCodec) Kind() contracts.ArtifactKind { return defaultArtifactKind }
|
||||
func (defaultArtifactCodec) Schema() contracts.ArtifactSchema {
|
||||
return contracts.ArtifactSchema{ID: "urn:notarius:test:default", Name: "default", Version: "1", JSONSchema: []byte(`{"type":"object"}`)}
|
||||
}
|
||||
func (defaultArtifactCodec) MediaType() string { return "application/json" }
|
||||
func (defaultArtifactCodec) Encode(value defaultArtifact) ([]byte, error) { return json.Marshal(value) }
|
||||
func (defaultArtifactCodec) Decode(content []byte) (defaultArtifact, error) {
|
||||
var value defaultArtifact
|
||||
err := json.Unmarshal(content, &value)
|
||||
return value, err
|
||||
}
|
||||
|
||||
@@ -1,102 +1,108 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type ExtractorConstructor func() (contracts.Extractor, error)
|
||||
|
||||
type ExtractorRegistry struct {
|
||||
constructors map[string]ExtractorConstructor
|
||||
typedEntries map[string]typedExtractorEntry
|
||||
specs map[string]ModuleSpec
|
||||
}
|
||||
|
||||
type typedExtractorEntry struct {
|
||||
spec ModuleSpec
|
||||
valueType reflect.Type
|
||||
validateOptions OptionValidator
|
||||
builder func(BuildRequest) (any, error)
|
||||
extract typedExtractOperation
|
||||
}
|
||||
|
||||
func NewExtractorRegistry() *ExtractorRegistry {
|
||||
return &ExtractorRegistry{
|
||||
constructors: make(map[string]ExtractorConstructor),
|
||||
specs: make(map[string]ModuleSpec),
|
||||
return &ExtractorRegistry{typedEntries: map[string]typedExtractorEntry{}, specs: map[string]ModuleSpec{}}
|
||||
}
|
||||
|
||||
func RegisterExtractor[T any](registry *ExtractorRegistry, spec ModuleSpec, constructor func() (contracts.Extractor[T], error)) error {
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("extractor constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||
}
|
||||
return RegisterExtractorBuilder(registry, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.Extractor[T], error) { return constructor() })
|
||||
}
|
||||
|
||||
func (r *ExtractorRegistry) Register(key string, constructor ExtractorConstructor) error {
|
||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageExtract), constructor)
|
||||
}
|
||||
|
||||
func (r *ExtractorRegistry) RegisterWithSpec(spec ModuleSpec, constructor ExtractorConstructor) error {
|
||||
if r == nil {
|
||||
func RegisterExtractorBuilder[T any](registry *ExtractorRegistry, spec ModuleSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.Extractor[T], error)) error {
|
||||
if registry == nil {
|
||||
return fmt.Errorf("extractor registry must not be nil")
|
||||
}
|
||||
|
||||
normalizedSpec := normalizeModuleSpec(spec)
|
||||
if err := validateModuleSpec("extractor", StageExtract, normalizedSpec); err != nil {
|
||||
normalized := normalizeModuleSpec(spec)
|
||||
if err := validateModuleSpec("extractor", StageExtract, normalized); err != nil {
|
||||
return err
|
||||
}
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("extractor constructor for %q must not be nil", normalizedSpec.Key)
|
||||
if normalized.ArtifactKind == "" {
|
||||
return fmt.Errorf("typed extractor %q artifact kind must not be empty", normalized.Key)
|
||||
}
|
||||
if _, ok := r.constructors[normalizedSpec.Key]; ok {
|
||||
return fmt.Errorf("extractor %q is already registered", normalizedSpec.Key)
|
||||
if validateOptions == nil {
|
||||
return fmt.Errorf("extractor option validator for %q must not be nil", normalized.Key)
|
||||
}
|
||||
|
||||
if r.constructors == nil {
|
||||
r.constructors = make(map[string]ExtractorConstructor)
|
||||
if builder == nil {
|
||||
return fmt.Errorf("extractor builder for %q must not be nil", normalized.Key)
|
||||
}
|
||||
if r.specs == nil {
|
||||
r.specs = make(map[string]ModuleSpec)
|
||||
if _, ok := registry.specs[normalized.Key]; ok {
|
||||
return fmt.Errorf("extractor %q is already registered", normalized.Key)
|
||||
}
|
||||
r.constructors[normalizedSpec.Key] = constructor
|
||||
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
||||
entry := typedExtractorEntry{spec: cloneModuleSpec(normalized), valueType: reflect.TypeFor[T](), validateOptions: validateOptions, builder: func(request BuildRequest) (any, error) { return builder(cloneBuildRequest(request)) }, extract: func(ctx context.Context, implementation any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
extractor, ok := implementation.(contracts.Extractor[T])
|
||||
if !ok {
|
||||
return erasedTypedResult{}, fmt.Errorf("extractor %q has incompatible implementation %T", normalized.Key, implementation)
|
||||
}
|
||||
result, err := extractor.Extract(ctx, request)
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
return erasedTypedResult{Value: result.Value, Warnings: result.Warnings}, nil
|
||||
}}
|
||||
if registry.typedEntries == nil {
|
||||
registry.typedEntries = map[string]typedExtractorEntry{}
|
||||
}
|
||||
if registry.specs == nil {
|
||||
registry.specs = map[string]ModuleSpec{}
|
||||
}
|
||||
registry.typedEntries[normalized.Key] = entry
|
||||
registry.specs[normalized.Key] = cloneModuleSpec(normalized)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *ExtractorRegistry) Build(key string) (contracts.Extractor, error) {
|
||||
func (r *ExtractorRegistry) validateOptions(key string, options map[string]any) error {
|
||||
if r == nil {
|
||||
return nil, fmt.Errorf("extractor registry must not be nil")
|
||||
return fmt.Errorf("extractor registry must not be nil")
|
||||
}
|
||||
|
||||
normalizedKey := strings.TrimSpace(key)
|
||||
if normalizedKey == "" {
|
||||
return nil, fmt.Errorf("extractor key must not be empty")
|
||||
}
|
||||
|
||||
constructor, ok := r.constructors[normalizedKey]
|
||||
normalized := strings.TrimSpace(key)
|
||||
entry, ok := r.typedEntries[normalized]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("extractor %q is not registered", normalizedKey)
|
||||
return fmt.Errorf("extractor %q is not registered", normalized)
|
||||
}
|
||||
|
||||
extractor, err := constructor()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("build extractor %q: %w", normalizedKey, err)
|
||||
}
|
||||
if extractor == nil {
|
||||
return nil, fmt.Errorf("extractor %q constructor returned nil", normalizedKey)
|
||||
}
|
||||
if extractor.Key() != normalizedKey {
|
||||
return nil, fmt.Errorf("extractor %q returned key %q", normalizedKey, extractor.Key())
|
||||
}
|
||||
|
||||
return extractor, nil
|
||||
return validateRegisteredOptions(entry.validateOptions, options)
|
||||
}
|
||||
|
||||
func (r *ExtractorRegistry) Spec(key string) (ModuleSpec, bool) {
|
||||
if r == nil {
|
||||
return ModuleSpec{}, false
|
||||
}
|
||||
|
||||
spec, ok := r.specs[strings.TrimSpace(key)]
|
||||
if !ok {
|
||||
return ModuleSpec{}, false
|
||||
}
|
||||
return cloneModuleSpec(spec), true
|
||||
return cloneModuleSpec(spec), ok
|
||||
}
|
||||
func (r *ExtractorRegistry) typedEntry(key string) (typedExtractorEntry, bool) {
|
||||
if r == nil {
|
||||
return typedExtractorEntry{}, false
|
||||
}
|
||||
entry, ok := r.typedEntries[strings.TrimSpace(key)]
|
||||
return entry, ok
|
||||
}
|
||||
|
||||
func (r *ExtractorRegistry) RegisteredKeys() []string {
|
||||
if r == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
return sortedRegistryKeys(r.constructors)
|
||||
return sortedRegistryKeys(r.specs)
|
||||
}
|
||||
|
||||
@@ -1,376 +0,0 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func TestExtractorRegistryRegisterAndBuild(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
|
||||
if err := registry.Register("generic-extractor", fakeExtractorConstructor("generic-extractor")); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
extractor, err := registry.Build("generic-extractor")
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v, want nil", err)
|
||||
}
|
||||
if extractor.Key() != "generic-extractor" {
|
||||
t.Fatalf("extractor.Key() = %q, want generic-extractor", extractor.Key())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistryRegisterAndBuildTrimKeys(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
|
||||
if err := registry.Register(" generic-extractor ", fakeExtractorConstructor("generic-extractor")); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
extractor, err := registry.Build("\tgeneric-extractor\n")
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v, want nil", err)
|
||||
}
|
||||
if extractor.Key() != "generic-extractor" {
|
||||
t.Fatalf("extractor.Key() = %q, want generic-extractor", extractor.Key())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistryRegisterWithSpecStoresMetadata(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
spec := ModuleSpec{
|
||||
Key: " generic-extractor ",
|
||||
Stage: StageExtract,
|
||||
Provides: []string{" generic-artifact ", "source-citations", "generic-artifact", ""},
|
||||
Requires: []string{" source-document ", "source-document", ""},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{
|
||||
Name: " glossary ",
|
||||
Description: " Supporting terms ",
|
||||
AcceptedMediaTypes: []string{" text/plain ", "text/markdown", "text/plain", ""},
|
||||
MaxBytes: 1024,
|
||||
},
|
||||
{
|
||||
Name: " roster ",
|
||||
Description: " Characters ",
|
||||
Required: true,
|
||||
Multiple: true,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
if err := registry.RegisterWithSpec(spec, fakeExtractorConstructor("generic-extractor")); err != nil {
|
||||
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
got, ok := registry.Spec("\tgeneric-extractor\n")
|
||||
if !ok {
|
||||
t.Fatal("Spec() ok = false, want true")
|
||||
}
|
||||
want := ModuleSpec{
|
||||
Key: "generic-extractor",
|
||||
Stage: StageExtract,
|
||||
Provides: []string{"generic-artifact", "source-citations"},
|
||||
Requires: []string{"source-document"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{
|
||||
Name: "glossary",
|
||||
Description: "Supporting terms",
|
||||
AcceptedMediaTypes: []string{"text/markdown", "text/plain"},
|
||||
MaxBytes: 1024,
|
||||
},
|
||||
{
|
||||
Name: "roster",
|
||||
Description: "Characters",
|
||||
Required: true,
|
||||
Multiple: true,
|
||||
},
|
||||
},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||
}
|
||||
|
||||
got.Provides[0] = "changed"
|
||||
got.ReferenceSlots[0].Name = "changed"
|
||||
got.ReferenceSlots[0].AcceptedMediaTypes[0] = "changed"
|
||||
again, ok := registry.Spec("generic-extractor")
|
||||
if !ok {
|
||||
t.Fatal("Spec() after caller mutation ok = false, want true")
|
||||
}
|
||||
if !reflect.DeepEqual(again, want) {
|
||||
t.Fatalf("Spec() after caller mutation = %#v, want %#v", again, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistryRegisterStoresDefaultSpec(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
|
||||
if err := registry.Register(" generic-extractor ", fakeExtractorConstructor("generic-extractor")); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
got, ok := registry.Spec("generic-extractor")
|
||||
if !ok {
|
||||
t.Fatal("Spec() ok = false, want true")
|
||||
}
|
||||
want := ModuleSpec{Key: "generic-extractor", Stage: StageExtract}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistryRegisterWithSpecRejectsWrongStage(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
|
||||
err := registry.RegisterWithSpec(ModuleSpec{Key: "generic-extractor", Stage: StageInput}, fakeExtractorConstructor("generic-extractor"))
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("RegisterWithSpec() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "stage") {
|
||||
t.Fatalf("RegisterWithSpec() error = %q, want stage error", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistryRejectsInvalidReferenceSlots(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
slots []contracts.ReferenceSlot
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "empty name",
|
||||
slots: []contracts.ReferenceSlot{{Name: " "}},
|
||||
want: "name",
|
||||
},
|
||||
{
|
||||
name: "duplicate name after trim",
|
||||
slots: []contracts.ReferenceSlot{
|
||||
{Name: "roster"},
|
||||
{Name: " roster "},
|
||||
},
|
||||
want: "duplicated",
|
||||
},
|
||||
{
|
||||
name: "negative max bytes",
|
||||
slots: []contracts.ReferenceSlot{{Name: "roster", MaxBytes: -1}},
|
||||
want: "max_bytes",
|
||||
},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
err := registry.RegisterWithSpec(ModuleSpec{
|
||||
Key: "generic-extractor",
|
||||
Stage: StageExtract,
|
||||
ReferenceSlots: test.slots,
|
||||
}, fakeExtractorConstructor("generic-extractor"))
|
||||
if err == nil {
|
||||
t.Fatal("RegisterWithSpec() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("RegisterWithSpec() error = %q, want %q", err.Error(), test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistrySpecRejectsUnknownKey(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
|
||||
if _, ok := registry.Spec("missing-extractor"); ok {
|
||||
t.Fatal("Spec() ok = true, want false")
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistryRegisterRejectsEmptyKey(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
|
||||
err := registry.Register(" \t", fakeExtractorConstructor("generic-extractor"))
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("Register() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "key must not be empty") {
|
||||
t.Fatalf("Register() error = %q, want empty key error", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistryRegisterRejectsDuplicateKey(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
if err := registry.Register("generic-extractor", fakeExtractorConstructor("generic-extractor")); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
err := registry.Register(" generic-extractor ", fakeExtractorConstructor("generic-extractor"))
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("Register() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "already registered") {
|
||||
t.Fatalf("Register() error = %q, want duplicate key error", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistryRegisterRejectsNilConstructor(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
|
||||
err := registry.Register("generic-extractor", nil)
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("Register() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "constructor") {
|
||||
t.Fatalf("Register() error = %q, want constructor error", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistryBuildRejectsUnknownKey(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
|
||||
_, err := registry.Build("missing-extractor")
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("Build() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "not registered") {
|
||||
t.Fatalf("Build() error = %q, want unknown key error", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistryBuildWrapsConstructorError(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
constructorErr := errors.New("constructor failed")
|
||||
if err := registry.Register("generic-extractor", func() (contracts.Extractor, error) {
|
||||
return nil, constructorErr
|
||||
}); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
_, err := registry.Build("generic-extractor")
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("Build() error = nil, want error")
|
||||
}
|
||||
if !errors.Is(err, constructorErr) {
|
||||
t.Fatalf("Build() error = %v, want wrapped constructor error", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "generic-extractor") {
|
||||
t.Fatalf("Build() error = %q, want key context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistryBuildRejectsNilExtractor(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
if err := registry.Register("generic-extractor", func() (contracts.Extractor, error) {
|
||||
return nil, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
_, err := registry.Build("generic-extractor")
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("Build() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "returned nil") {
|
||||
t.Fatalf("Build() error = %q, want nil extractor error", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistryBuildRejectsExtractorKeyMismatch(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
if err := registry.Register("generic-extractor", fakeExtractorConstructor("other-extractor")); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
_, err := registry.Build("generic-extractor")
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("Build() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "returned key") {
|
||||
t.Fatalf("Build() error = %q, want key mismatch error", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistryRegisteredKeysReturnsSortedCopy(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
for _, key := range []string{"zeta", "alpha", "middle"} {
|
||||
if err := registry.Register(key, fakeExtractorConstructor(key)); err != nil {
|
||||
t.Fatalf("Register(%q) error = %v, want nil", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
keys := registry.RegisteredKeys()
|
||||
|
||||
want := []string{"alpha", "middle", "zeta"}
|
||||
if !reflect.DeepEqual(keys, want) {
|
||||
t.Fatalf("RegisteredKeys() = %#v, want %#v", keys, want)
|
||||
}
|
||||
|
||||
keys[0] = "changed"
|
||||
if got := registry.RegisteredKeys(); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("RegisteredKeys() after caller mutation = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistryNilRegistryBehavior(t *testing.T) {
|
||||
var registry *ExtractorRegistry
|
||||
|
||||
if err := registry.Register("generic-extractor", fakeExtractorConstructor("generic-extractor")); err == nil {
|
||||
t.Fatal("Register() error = nil, want error")
|
||||
}
|
||||
if _, err := registry.Build("generic-extractor"); err == nil {
|
||||
t.Fatal("Build() error = nil, want error")
|
||||
}
|
||||
if _, ok := registry.Spec("generic-extractor"); ok {
|
||||
t.Fatal("Spec() ok = true, want false")
|
||||
}
|
||||
if keys := registry.RegisteredKeys(); keys != nil {
|
||||
t.Fatalf("RegisteredKeys() = %#v, want nil", keys)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorRegistryBuildRejectsEmptyKey(t *testing.T) {
|
||||
registry := NewExtractorRegistry()
|
||||
|
||||
_, err := registry.Build(" \n")
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("Build() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "key must not be empty") {
|
||||
t.Fatalf("Build() error = %q, want empty key error", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
type registryFakeExtractor struct {
|
||||
key string
|
||||
}
|
||||
|
||||
func fakeExtractorConstructor(key string) ExtractorConstructor {
|
||||
return func() (contracts.Extractor, error) {
|
||||
return registryFakeExtractor{key: key}, nil
|
||||
}
|
||||
}
|
||||
|
||||
func (extractor registryFakeExtractor) Key() string {
|
||||
return extractor.key
|
||||
}
|
||||
|
||||
func (extractor registryFakeExtractor) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (extractor registryFakeExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
|
||||
return contracts.ExtractionResult{}, nil
|
||||
}
|
||||
@@ -8,16 +8,19 @@ import (
|
||||
)
|
||||
|
||||
type InputAdapterConstructor func() (contracts.InputAdapter, error)
|
||||
type InputAdapterBuilder func(BuildRequest) (contracts.InputAdapter, error)
|
||||
|
||||
type InputAdapterRegistry struct {
|
||||
constructors map[string]InputAdapterConstructor
|
||||
specs map[string]ModuleSpec
|
||||
builders map[string]InputAdapterBuilder
|
||||
optionValidators map[string]OptionValidator
|
||||
specs map[string]ModuleSpec
|
||||
}
|
||||
|
||||
func NewInputAdapterRegistry() *InputAdapterRegistry {
|
||||
return &InputAdapterRegistry{
|
||||
constructors: make(map[string]InputAdapterConstructor),
|
||||
specs: make(map[string]ModuleSpec),
|
||||
builders: make(map[string]InputAdapterBuilder),
|
||||
optionValidators: make(map[string]OptionValidator),
|
||||
specs: make(map[string]ModuleSpec),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -26,6 +29,15 @@ func (r *InputAdapterRegistry) Register(key string, constructor InputAdapterCons
|
||||
}
|
||||
|
||||
func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor InputAdapterConstructor) error {
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("input adapter constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||
}
|
||||
return r.RegisterBuilderWithSpec(spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.InputAdapter, error) {
|
||||
return constructor()
|
||||
})
|
||||
}
|
||||
|
||||
func (r *InputAdapterRegistry) RegisterBuilderWithSpec(spec ModuleSpec, validateOptions OptionValidator, builder InputAdapterBuilder) error {
|
||||
if r == nil {
|
||||
return fmt.Errorf("input adapter registry must not be nil")
|
||||
}
|
||||
@@ -34,25 +46,36 @@ func (r *InputAdapterRegistry) RegisterWithSpec(spec ModuleSpec, constructor Inp
|
||||
if err := validateModuleSpec("input adapter", StageInput, normalizedSpec); err != nil {
|
||||
return err
|
||||
}
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("input adapter constructor for %q must not be nil", normalizedSpec.Key)
|
||||
if validateOptions == nil {
|
||||
return fmt.Errorf("input adapter option validator for %q must not be nil", normalizedSpec.Key)
|
||||
}
|
||||
if _, ok := r.constructors[normalizedSpec.Key]; ok {
|
||||
if builder == nil {
|
||||
return fmt.Errorf("input adapter builder for %q must not be nil", normalizedSpec.Key)
|
||||
}
|
||||
if _, ok := r.builders[normalizedSpec.Key]; ok {
|
||||
return fmt.Errorf("input adapter %q is already registered", normalizedSpec.Key)
|
||||
}
|
||||
|
||||
if r.constructors == nil {
|
||||
r.constructors = make(map[string]InputAdapterConstructor)
|
||||
if r.builders == nil {
|
||||
r.builders = make(map[string]InputAdapterBuilder)
|
||||
}
|
||||
if r.optionValidators == nil {
|
||||
r.optionValidators = make(map[string]OptionValidator)
|
||||
}
|
||||
if r.specs == nil {
|
||||
r.specs = make(map[string]ModuleSpec)
|
||||
}
|
||||
r.constructors[normalizedSpec.Key] = constructor
|
||||
r.builders[normalizedSpec.Key] = builder
|
||||
r.optionValidators[normalizedSpec.Key] = validateOptions
|
||||
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *InputAdapterRegistry) Build(key string) (contracts.InputAdapter, error) {
|
||||
return r.BuildWithRequest(key, BuildRequest{})
|
||||
}
|
||||
|
||||
func (r *InputAdapterRegistry) BuildWithRequest(key string, request BuildRequest) (contracts.InputAdapter, error) {
|
||||
if r == nil {
|
||||
return nil, fmt.Errorf("input adapter registry must not be nil")
|
||||
}
|
||||
@@ -62,12 +85,12 @@ func (r *InputAdapterRegistry) Build(key string) (contracts.InputAdapter, error)
|
||||
return nil, fmt.Errorf("input adapter key must not be empty")
|
||||
}
|
||||
|
||||
constructor, ok := r.constructors[normalizedKey]
|
||||
builder, ok := r.builders[normalizedKey]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("input adapter %q is not registered", normalizedKey)
|
||||
}
|
||||
|
||||
adapter, err := constructor()
|
||||
adapter, err := builder(cloneBuildRequest(request))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("build input adapter %q: %w", normalizedKey, err)
|
||||
}
|
||||
@@ -81,6 +104,18 @@ func (r *InputAdapterRegistry) Build(key string) (contracts.InputAdapter, error)
|
||||
return adapter, nil
|
||||
}
|
||||
|
||||
func (r *InputAdapterRegistry) ValidateOptions(key string, options map[string]any) error {
|
||||
if r == nil {
|
||||
return fmt.Errorf("input adapter registry must not be nil")
|
||||
}
|
||||
normalizedKey := strings.TrimSpace(key)
|
||||
validator, ok := r.optionValidators[normalizedKey]
|
||||
if !ok {
|
||||
return fmt.Errorf("input adapter %q is not registered", normalizedKey)
|
||||
}
|
||||
return validateRegisteredOptions(validator, options)
|
||||
}
|
||||
|
||||
func (r *InputAdapterRegistry) Spec(key string) (ModuleSpec, bool) {
|
||||
if r == nil {
|
||||
return ModuleSpec{}, false
|
||||
@@ -98,5 +133,5 @@ func (r *InputAdapterRegistry) RegisteredKeys() []string {
|
||||
return nil
|
||||
}
|
||||
|
||||
return sortedRegistryKeys(r.constructors)
|
||||
return sortedRegistryKeys(r.builders)
|
||||
}
|
||||
|
||||
@@ -301,7 +301,7 @@ func (adapter fakeAdapter) Parse(ctx context.Context, req contracts.ParseRequest
|
||||
Format: "text/plain",
|
||||
Digest: "sha256:abc123",
|
||||
Units: []source.SourceUnit{
|
||||
{ID: 1, Kind: "unit", Text: "Source unit."},
|
||||
{ID: 1, Kind: "unit", Text: "Source unit.", Ref: source.SourceRef{SourceID: req.SourceID, StartUnitID: 1, EndUnitID: 1}},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -1,102 +1,155 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type MergerConstructor func() (contracts.Merger, error)
|
||||
type artifactVariantKey struct {
|
||||
module string
|
||||
kind contracts.ArtifactKind
|
||||
}
|
||||
|
||||
type MergerRegistry struct {
|
||||
constructors map[string]MergerConstructor
|
||||
specs map[string]ModuleSpec
|
||||
typedEntries map[artifactVariantKey]typedMergerEntry
|
||||
}
|
||||
|
||||
type typedMergerEntry struct {
|
||||
spec ModuleSpec
|
||||
valueType reflect.Type
|
||||
validateOptions OptionValidator
|
||||
builder func(BuildRequest) (any, error)
|
||||
merge typedMergeOperation
|
||||
}
|
||||
|
||||
func NewMergerRegistry() *MergerRegistry {
|
||||
return &MergerRegistry{
|
||||
constructors: make(map[string]MergerConstructor),
|
||||
specs: make(map[string]ModuleSpec),
|
||||
typedEntries: make(map[artifactVariantKey]typedMergerEntry),
|
||||
}
|
||||
}
|
||||
|
||||
func (r *MergerRegistry) Register(key string, constructor MergerConstructor) error {
|
||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageMerge), constructor)
|
||||
func RegisterMerger[T any](registry *MergerRegistry, spec ModuleSpec, constructor func() (contracts.Merger[T], error)) error {
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("merger constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||
}
|
||||
return RegisterMergerBuilder(registry, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.Merger[T], error) {
|
||||
return constructor()
|
||||
})
|
||||
}
|
||||
|
||||
func (r *MergerRegistry) RegisterWithSpec(spec ModuleSpec, constructor MergerConstructor) error {
|
||||
if r == nil {
|
||||
func RegisterMergerBuilder[T any](registry *MergerRegistry, spec ModuleSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.Merger[T], error)) error {
|
||||
if registry == nil {
|
||||
return fmt.Errorf("merger registry must not be nil")
|
||||
}
|
||||
|
||||
normalizedSpec := normalizeModuleSpec(spec)
|
||||
if err := validateModuleSpec("merger", StageMerge, normalizedSpec); err != nil {
|
||||
return err
|
||||
}
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("merger constructor for %q must not be nil", normalizedSpec.Key)
|
||||
if normalizedSpec.ArtifactKind == "" {
|
||||
return fmt.Errorf("typed merger %q artifact kind must not be empty", normalizedSpec.Key)
|
||||
}
|
||||
if _, ok := r.constructors[normalizedSpec.Key]; ok {
|
||||
return fmt.Errorf("merger %q is already registered", normalizedSpec.Key)
|
||||
if validateOptions == nil {
|
||||
return fmt.Errorf("merger option validator for %q must not be nil", normalizedSpec.Key)
|
||||
}
|
||||
|
||||
if r.constructors == nil {
|
||||
r.constructors = make(map[string]MergerConstructor)
|
||||
if builder == nil {
|
||||
return fmt.Errorf("merger builder for %q must not be nil", normalizedSpec.Key)
|
||||
}
|
||||
if r.specs == nil {
|
||||
r.specs = make(map[string]ModuleSpec)
|
||||
key := artifactVariantKey{module: normalizedSpec.Key, kind: normalizedSpec.ArtifactKind}
|
||||
if _, ok := registry.typedEntries[key]; ok {
|
||||
return fmt.Errorf("merger %q variant for artifact kind %q is already registered", key.module, key.kind)
|
||||
}
|
||||
if registry.typedEntries == nil {
|
||||
registry.typedEntries = make(map[artifactVariantKey]typedMergerEntry)
|
||||
}
|
||||
registry.typedEntries[key] = typedMergerEntry{
|
||||
spec: cloneModuleSpec(normalizedSpec),
|
||||
valueType: reflect.TypeFor[T](),
|
||||
validateOptions: validateOptions,
|
||||
builder: func(request BuildRequest) (any, error) {
|
||||
return builder(cloneBuildRequest(request))
|
||||
},
|
||||
merge: func(ctx context.Context, implementation any, request contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
merger, ok := implementation.(contracts.Merger[T])
|
||||
if !ok {
|
||||
return erasedTypedResult{}, fmt.Errorf("merger %q has incompatible implementation %T", normalizedSpec.Key, implementation)
|
||||
}
|
||||
outputs := make([]contracts.ExtractArtifact[T], len(request.ExtractOutputs))
|
||||
for i, output := range request.ExtractOutputs {
|
||||
value, err := exactTypedValue[T]("merge extract value", output.Value)
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
outputs[i] = contracts.ExtractArtifact[T]{LaneID: output.LaneID, ExtractorKey: output.ExtractorKey, SourceID: output.SourceID, ChunkID: output.ChunkID, ChunkIndex: output.ChunkIndex, ChunkRef: output.ChunkRef, Value: value}
|
||||
}
|
||||
result, err := merger.Merge(ctx, contracts.TypedMergeRequest[T]{Source: request.Source, LaneID: request.LaneID, ExtractOutputs: outputs, SourceInput: request.SourceInput, SessionID: request.SessionID, References: request.References, LLMProfile: request.LLMProfile, Metadata: request.Metadata})
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
return erasedTypedResult{Value: result.Value, Warnings: result.Warnings}, nil
|
||||
},
|
||||
}
|
||||
r.constructors[normalizedSpec.Key] = constructor
|
||||
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *MergerRegistry) Build(key string) (contracts.Merger, error) {
|
||||
func (r *MergerRegistry) validateOptions(key string, kind contracts.ArtifactKind, options map[string]any) error {
|
||||
if r == nil {
|
||||
return nil, fmt.Errorf("merger registry must not be nil")
|
||||
return fmt.Errorf("merger registry must not be nil")
|
||||
}
|
||||
|
||||
normalizedKey := strings.TrimSpace(key)
|
||||
if normalizedKey == "" {
|
||||
return nil, fmt.Errorf("merger key must not be empty")
|
||||
}
|
||||
|
||||
constructor, ok := r.constructors[normalizedKey]
|
||||
entry, ok := r.typedEntry(normalizedKey, kind)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("merger %q is not registered", normalizedKey)
|
||||
return fmt.Errorf("merger %q variant for artifact kind %q is not registered", normalizedKey, kind)
|
||||
}
|
||||
|
||||
merger, err := constructor()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("build merger %q: %w", normalizedKey, err)
|
||||
}
|
||||
if merger == nil {
|
||||
return nil, fmt.Errorf("merger %q constructor returned nil", normalizedKey)
|
||||
}
|
||||
if merger.Key() != normalizedKey {
|
||||
return nil, fmt.Errorf("merger %q returned key %q", normalizedKey, merger.Key())
|
||||
}
|
||||
|
||||
return merger, nil
|
||||
return validateRegisteredOptions(entry.validateOptions, options)
|
||||
}
|
||||
|
||||
func (r *MergerRegistry) Spec(key string) (ModuleSpec, bool) {
|
||||
if r == nil {
|
||||
return ModuleSpec{}, false
|
||||
}
|
||||
|
||||
spec, ok := r.specs[strings.TrimSpace(key)]
|
||||
if !ok {
|
||||
return ModuleSpec{}, false
|
||||
module := strings.TrimSpace(key)
|
||||
for variant, entry := range r.typedEntries {
|
||||
if variant.module == module {
|
||||
return cloneModuleSpec(entry.spec), true
|
||||
}
|
||||
}
|
||||
return cloneModuleSpec(spec), true
|
||||
return ModuleSpec{}, false
|
||||
}
|
||||
|
||||
func (r *MergerRegistry) typedEntry(key string, kind contracts.ArtifactKind) (typedMergerEntry, bool) {
|
||||
if r == nil {
|
||||
return typedMergerEntry{}, false
|
||||
}
|
||||
entry, ok := r.typedEntries[artifactVariantKey{module: strings.TrimSpace(key), kind: normalizeArtifactKind(kind)}]
|
||||
return entry, ok
|
||||
}
|
||||
|
||||
func (r *MergerRegistry) registeredKinds(key string) []contracts.ArtifactKind {
|
||||
if r == nil {
|
||||
return nil
|
||||
}
|
||||
module := strings.TrimSpace(key)
|
||||
kinds := make([]contracts.ArtifactKind, 0)
|
||||
for variant := range r.typedEntries {
|
||||
if variant.module == module {
|
||||
kinds = append(kinds, variant.kind)
|
||||
}
|
||||
}
|
||||
sortArtifactKinds(kinds)
|
||||
return kinds
|
||||
}
|
||||
|
||||
func (r *MergerRegistry) RegisteredKeys() []string {
|
||||
if r == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
return sortedRegistryKeys(r.constructors)
|
||||
keys := make(map[string]struct{}, len(r.typedEntries))
|
||||
for key := range r.typedEntries {
|
||||
keys[key.module] = struct{}{}
|
||||
}
|
||||
return sortedRegistryKeys(keys)
|
||||
}
|
||||
|
||||
@@ -1,58 +0,0 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func TestMergerRegistryBehavior(t *testing.T) {
|
||||
runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Merger]{
|
||||
name: "MergerRegistry",
|
||||
key: "generic-merger",
|
||||
stage: StageMerge,
|
||||
wrongStage: StageExtract,
|
||||
newRegistry: func() any {
|
||||
return NewMergerRegistry()
|
||||
},
|
||||
register: func(registry any, key string, constructor func() (contracts.Merger, error)) error {
|
||||
return registry.(*MergerRegistry).Register(key, constructor)
|
||||
},
|
||||
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Merger, error)) error {
|
||||
return registry.(*MergerRegistry).RegisterWithSpec(spec, constructor)
|
||||
},
|
||||
build: func(registry any, key string) (contracts.Merger, error) {
|
||||
return registry.(*MergerRegistry).Build(key)
|
||||
},
|
||||
spec: func(registry any, key string) (ModuleSpec, bool) {
|
||||
return registry.(*MergerRegistry).Spec(key)
|
||||
},
|
||||
registeredKeys: func(registry any) []string {
|
||||
return registry.(*MergerRegistry).RegisteredKeys()
|
||||
},
|
||||
nilRegister: func(key string, constructor func() (contracts.Merger, error)) error {
|
||||
var registry *MergerRegistry
|
||||
return registry.Register(key, constructor)
|
||||
},
|
||||
nilBuild: func(key string) (contracts.Merger, error) {
|
||||
var registry *MergerRegistry
|
||||
return registry.Build(key)
|
||||
},
|
||||
nilSpec: func(key string) (ModuleSpec, bool) {
|
||||
var registry *MergerRegistry
|
||||
return registry.Spec(key)
|
||||
},
|
||||
nilRegisteredKey: func() []string {
|
||||
var registry *MergerRegistry
|
||||
return registry.RegisteredKeys()
|
||||
},
|
||||
constructor: func(key string) func() (contracts.Merger, error) {
|
||||
return func() (contracts.Merger, error) {
|
||||
return registryMerger{key: key}, nil
|
||||
}
|
||||
},
|
||||
moduleKey: func(module contracts.Merger) string {
|
||||
return module.Key()
|
||||
},
|
||||
})
|
||||
}
|
||||
@@ -23,6 +23,7 @@ const (
|
||||
type ModuleSpec struct {
|
||||
Key string
|
||||
Stage ModuleStage
|
||||
ArtifactKind contracts.ArtifactKind
|
||||
Provides []string
|
||||
Requires []string
|
||||
ReferenceSlots []contracts.ReferenceSlot
|
||||
@@ -39,6 +40,7 @@ func normalizeModuleSpec(spec ModuleSpec) ModuleSpec {
|
||||
return ModuleSpec{
|
||||
Key: strings.TrimSpace(spec.Key),
|
||||
Stage: spec.Stage,
|
||||
ArtifactKind: normalizeArtifactKind(spec.ArtifactKind),
|
||||
Provides: normalizeCapabilities(spec.Provides),
|
||||
Requires: normalizeCapabilities(spec.Requires),
|
||||
ReferenceSlots: normalizeReferenceSlots(spec.ReferenceSlots),
|
||||
@@ -74,6 +76,7 @@ func cloneModuleSpec(spec ModuleSpec) ModuleSpec {
|
||||
return ModuleSpec{
|
||||
Key: spec.Key,
|
||||
Stage: spec.Stage,
|
||||
ArtifactKind: spec.ArtifactKind,
|
||||
Provides: append([]string(nil), spec.Provides...),
|
||||
Requires: append([]string(nil), spec.Requires...),
|
||||
ReferenceSlots: contracts.CloneReferenceSlots(spec.ReferenceSlots),
|
||||
@@ -87,6 +90,9 @@ func validateModuleSpec(kind string, expectedStage ModuleStage, spec ModuleSpec)
|
||||
if spec.Stage != expectedStage {
|
||||
return fmt.Errorf("%s %q must use %q stage, got %q", kind, spec.Key, expectedStage, spec.Stage)
|
||||
}
|
||||
if spec.ArtifactKind != "" && spec.Stage != StageExtract && spec.Stage != StageMerge && spec.Stage != StageNormalize {
|
||||
return fmt.Errorf("%s %q must not declare an artifact kind", kind, spec.Key)
|
||||
}
|
||||
if !referenceSlotStage(spec.Stage) && len(spec.ReferenceSlots) > 0 {
|
||||
return fmt.Errorf("%s %q must not declare reference slots", kind, spec.Key)
|
||||
}
|
||||
@@ -113,6 +119,10 @@ func sortedRegistryKeys[C any](constructors map[string]C) []string {
|
||||
return keys
|
||||
}
|
||||
|
||||
func sortArtifactKinds(kinds []contracts.ArtifactKind) {
|
||||
sort.Slice(kinds, func(i, j int) bool { return kinds[i] < kinds[j] })
|
||||
}
|
||||
|
||||
func normalizeReferenceSlots(slots []contracts.ReferenceSlot) []contracts.ReferenceSlot {
|
||||
if len(slots) == 0 {
|
||||
return nil
|
||||
|
||||
@@ -1,102 +1,146 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type NormalizerConstructor func() (contracts.Normalizer, error)
|
||||
|
||||
type NormalizerRegistry struct {
|
||||
constructors map[string]NormalizerConstructor
|
||||
specs map[string]ModuleSpec
|
||||
typedEntries map[artifactVariantKey]typedNormalizerEntry
|
||||
}
|
||||
|
||||
type typedNormalizerEntry struct {
|
||||
spec ModuleSpec
|
||||
valueType reflect.Type
|
||||
validateOptions OptionValidator
|
||||
builder func(BuildRequest) (any, error)
|
||||
normalize typedNormalizeOperation
|
||||
}
|
||||
|
||||
func NewNormalizerRegistry() *NormalizerRegistry {
|
||||
return &NormalizerRegistry{
|
||||
constructors: make(map[string]NormalizerConstructor),
|
||||
specs: make(map[string]ModuleSpec),
|
||||
typedEntries: make(map[artifactVariantKey]typedNormalizerEntry),
|
||||
}
|
||||
}
|
||||
|
||||
func (r *NormalizerRegistry) Register(key string, constructor NormalizerConstructor) error {
|
||||
return r.RegisterWithSpec(defaultModuleSpec(key, StageNormalize), constructor)
|
||||
func RegisterNormalizer[T any](registry *NormalizerRegistry, spec ModuleSpec, constructor func() (contracts.Normalizer[T], error)) error {
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("normalizer constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||
}
|
||||
return RegisterNormalizerBuilder(registry, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.Normalizer[T], error) {
|
||||
return constructor()
|
||||
})
|
||||
}
|
||||
|
||||
func (r *NormalizerRegistry) RegisterWithSpec(spec ModuleSpec, constructor NormalizerConstructor) error {
|
||||
if r == nil {
|
||||
func RegisterNormalizerBuilder[T any](registry *NormalizerRegistry, spec ModuleSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.Normalizer[T], error)) error {
|
||||
if registry == nil {
|
||||
return fmt.Errorf("normalizer registry must not be nil")
|
||||
}
|
||||
|
||||
normalizedSpec := normalizeModuleSpec(spec)
|
||||
if err := validateModuleSpec("normalizer", StageNormalize, normalizedSpec); err != nil {
|
||||
return err
|
||||
}
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("normalizer constructor for %q must not be nil", normalizedSpec.Key)
|
||||
if normalizedSpec.ArtifactKind == "" {
|
||||
return fmt.Errorf("typed normalizer %q artifact kind must not be empty", normalizedSpec.Key)
|
||||
}
|
||||
if _, ok := r.constructors[normalizedSpec.Key]; ok {
|
||||
return fmt.Errorf("normalizer %q is already registered", normalizedSpec.Key)
|
||||
if validateOptions == nil {
|
||||
return fmt.Errorf("normalizer option validator for %q must not be nil", normalizedSpec.Key)
|
||||
}
|
||||
|
||||
if r.constructors == nil {
|
||||
r.constructors = make(map[string]NormalizerConstructor)
|
||||
if builder == nil {
|
||||
return fmt.Errorf("normalizer builder for %q must not be nil", normalizedSpec.Key)
|
||||
}
|
||||
if r.specs == nil {
|
||||
r.specs = make(map[string]ModuleSpec)
|
||||
key := artifactVariantKey{module: normalizedSpec.Key, kind: normalizedSpec.ArtifactKind}
|
||||
if _, ok := registry.typedEntries[key]; ok {
|
||||
return fmt.Errorf("normalizer %q variant for artifact kind %q is already registered", key.module, key.kind)
|
||||
}
|
||||
if registry.typedEntries == nil {
|
||||
registry.typedEntries = make(map[artifactVariantKey]typedNormalizerEntry)
|
||||
}
|
||||
registry.typedEntries[key] = typedNormalizerEntry{
|
||||
spec: cloneModuleSpec(normalizedSpec),
|
||||
valueType: reflect.TypeFor[T](),
|
||||
validateOptions: validateOptions,
|
||||
builder: func(request BuildRequest) (any, error) {
|
||||
return builder(cloneBuildRequest(request))
|
||||
},
|
||||
normalize: func(ctx context.Context, implementation any, request contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
normalizer, ok := implementation.(contracts.Normalizer[T])
|
||||
if !ok {
|
||||
return erasedTypedResult{}, fmt.Errorf("normalizer %q has incompatible implementation %T", normalizedSpec.Key, implementation)
|
||||
}
|
||||
value, err := exactTypedValue[T]("normalize merge value", request.MergeOutput.Value)
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
result, err := normalizer.Normalize(ctx, contracts.TypedNormalizeRequest[T]{Source: request.Source, LaneID: request.LaneID, MergeOutput: contracts.MergeArtifact[T]{LaneID: request.MergeOutput.LaneID, MergerKey: request.MergeOutput.MergerKey, SourceID: request.MergeOutput.SourceID, Value: value}, SourceInput: request.SourceInput, SessionID: request.SessionID, References: request.References, LLMProfile: request.LLMProfile, Metadata: request.Metadata})
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
return erasedTypedResult{Value: result.Value, Warnings: result.Warnings}, nil
|
||||
},
|
||||
}
|
||||
r.constructors[normalizedSpec.Key] = constructor
|
||||
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *NormalizerRegistry) Build(key string) (contracts.Normalizer, error) {
|
||||
func (r *NormalizerRegistry) validateOptions(key string, kind contracts.ArtifactKind, options map[string]any) error {
|
||||
if r == nil {
|
||||
return nil, fmt.Errorf("normalizer registry must not be nil")
|
||||
return fmt.Errorf("normalizer registry must not be nil")
|
||||
}
|
||||
|
||||
normalizedKey := strings.TrimSpace(key)
|
||||
if normalizedKey == "" {
|
||||
return nil, fmt.Errorf("normalizer key must not be empty")
|
||||
}
|
||||
|
||||
constructor, ok := r.constructors[normalizedKey]
|
||||
entry, ok := r.typedEntry(normalizedKey, kind)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("normalizer %q is not registered", normalizedKey)
|
||||
return fmt.Errorf("normalizer %q variant for artifact kind %q is not registered", normalizedKey, kind)
|
||||
}
|
||||
|
||||
normalizer, err := constructor()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("build normalizer %q: %w", normalizedKey, err)
|
||||
}
|
||||
if normalizer == nil {
|
||||
return nil, fmt.Errorf("normalizer %q constructor returned nil", normalizedKey)
|
||||
}
|
||||
if normalizer.Key() != normalizedKey {
|
||||
return nil, fmt.Errorf("normalizer %q returned key %q", normalizedKey, normalizer.Key())
|
||||
}
|
||||
|
||||
return normalizer, nil
|
||||
return validateRegisteredOptions(entry.validateOptions, options)
|
||||
}
|
||||
|
||||
func (r *NormalizerRegistry) Spec(key string) (ModuleSpec, bool) {
|
||||
if r == nil {
|
||||
return ModuleSpec{}, false
|
||||
}
|
||||
|
||||
spec, ok := r.specs[strings.TrimSpace(key)]
|
||||
if !ok {
|
||||
return ModuleSpec{}, false
|
||||
module := strings.TrimSpace(key)
|
||||
for variant, entry := range r.typedEntries {
|
||||
if variant.module == module {
|
||||
return cloneModuleSpec(entry.spec), true
|
||||
}
|
||||
}
|
||||
return cloneModuleSpec(spec), true
|
||||
return ModuleSpec{}, false
|
||||
}
|
||||
|
||||
func (r *NormalizerRegistry) typedEntry(key string, kind contracts.ArtifactKind) (typedNormalizerEntry, bool) {
|
||||
if r == nil {
|
||||
return typedNormalizerEntry{}, false
|
||||
}
|
||||
entry, ok := r.typedEntries[artifactVariantKey{module: strings.TrimSpace(key), kind: normalizeArtifactKind(kind)}]
|
||||
return entry, ok
|
||||
}
|
||||
|
||||
func (r *NormalizerRegistry) registeredKinds(key string) []contracts.ArtifactKind {
|
||||
if r == nil {
|
||||
return nil
|
||||
}
|
||||
module := strings.TrimSpace(key)
|
||||
kinds := make([]contracts.ArtifactKind, 0)
|
||||
for variant := range r.typedEntries {
|
||||
if variant.module == module {
|
||||
kinds = append(kinds, variant.kind)
|
||||
}
|
||||
}
|
||||
sortArtifactKinds(kinds)
|
||||
return kinds
|
||||
}
|
||||
|
||||
func (r *NormalizerRegistry) RegisteredKeys() []string {
|
||||
if r == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
return sortedRegistryKeys(r.constructors)
|
||||
keys := make(map[string]struct{}, len(r.typedEntries))
|
||||
for key := range r.typedEntries {
|
||||
keys[key.module] = struct{}{}
|
||||
}
|
||||
return sortedRegistryKeys(keys)
|
||||
}
|
||||
|
||||
@@ -1,58 +0,0 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func TestNormalizerRegistryBehavior(t *testing.T) {
|
||||
runRegistryBehaviorTests(t, registryBehaviorCase[contracts.Normalizer]{
|
||||
name: "NormalizerRegistry",
|
||||
key: "generic-normalizer",
|
||||
stage: StageNormalize,
|
||||
wrongStage: StageExtract,
|
||||
newRegistry: func() any {
|
||||
return NewNormalizerRegistry()
|
||||
},
|
||||
register: func(registry any, key string, constructor func() (contracts.Normalizer, error)) error {
|
||||
return registry.(*NormalizerRegistry).Register(key, constructor)
|
||||
},
|
||||
registerWithSpec: func(registry any, spec ModuleSpec, constructor func() (contracts.Normalizer, error)) error {
|
||||
return registry.(*NormalizerRegistry).RegisterWithSpec(spec, constructor)
|
||||
},
|
||||
build: func(registry any, key string) (contracts.Normalizer, error) {
|
||||
return registry.(*NormalizerRegistry).Build(key)
|
||||
},
|
||||
spec: func(registry any, key string) (ModuleSpec, bool) {
|
||||
return registry.(*NormalizerRegistry).Spec(key)
|
||||
},
|
||||
registeredKeys: func(registry any) []string {
|
||||
return registry.(*NormalizerRegistry).RegisteredKeys()
|
||||
},
|
||||
nilRegister: func(key string, constructor func() (contracts.Normalizer, error)) error {
|
||||
var registry *NormalizerRegistry
|
||||
return registry.Register(key, constructor)
|
||||
},
|
||||
nilBuild: func(key string) (contracts.Normalizer, error) {
|
||||
var registry *NormalizerRegistry
|
||||
return registry.Build(key)
|
||||
},
|
||||
nilSpec: func(key string) (ModuleSpec, bool) {
|
||||
var registry *NormalizerRegistry
|
||||
return registry.Spec(key)
|
||||
},
|
||||
nilRegisteredKey: func() []string {
|
||||
var registry *NormalizerRegistry
|
||||
return registry.RegisteredKeys()
|
||||
},
|
||||
constructor: func(key string) func() (contracts.Normalizer, error) {
|
||||
return func() (contracts.Normalizer, error) {
|
||||
return registryNormalizer{key: key}, nil
|
||||
}
|
||||
},
|
||||
moduleKey: func(module contracts.Normalizer) string {
|
||||
return module.Key()
|
||||
},
|
||||
})
|
||||
}
|
||||
65
internal/framework/pipeline/options.go
Normal file
65
internal/framework/pipeline/options.go
Normal file
@@ -0,0 +1,65 @@
|
||||
package pipeline
|
||||
|
||||
import "fmt"
|
||||
|
||||
func validateResolvedOptions(resolved ResolvedPipeline, catalog ModuleCatalog) error {
|
||||
if err := catalog.Inputs.ValidateOptions(resolved.Input.Module, resolved.Input.Options); err != nil {
|
||||
return moduleOptionsError(resolved.ID, "", StageInput, resolved.Input.Module, err)
|
||||
}
|
||||
if err := catalog.Chunkers.ValidateOptions(resolved.Chunk.Module, resolved.Chunk.Options); err != nil {
|
||||
return moduleOptionsError(resolved.ID, "", StageChunk, resolved.Chunk.Module, err)
|
||||
}
|
||||
if err := validateChainOptions(resolved, catalog, StageChunk, "", resolved.Chunk.Module); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, lane := range resolved.ArtifactLanes {
|
||||
if err := catalog.Extractors.validateOptions(lane.Extract.Module, lane.Extract.Options); err != nil {
|
||||
return moduleOptionsError(resolved.ID, lane.ID, StageExtract, lane.Extract.Module, err)
|
||||
}
|
||||
if err := validateChainOptions(resolved, catalog, StageExtract, lane.ID, lane.Extract.Module); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := catalog.Mergers.validateOptions(lane.Merge.Module, lane.ArtifactKind, lane.Merge.Options); err != nil {
|
||||
return moduleOptionsError(resolved.ID, lane.ID, StageMerge, lane.Merge.Module, err)
|
||||
}
|
||||
if err := validateChainOptions(resolved, catalog, StageMerge, lane.ID, lane.Merge.Module); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := catalog.Normalizers.validateOptions(lane.Normalize.Module, lane.ArtifactKind, lane.Normalize.Options); err != nil {
|
||||
return moduleOptionsError(resolved.ID, lane.ID, StageNormalize, lane.Normalize.Module, err)
|
||||
}
|
||||
if err := validateChainOptions(resolved, catalog, StageNormalize, lane.ID, lane.Normalize.Module); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if err := catalog.Outputs.ValidateOptions(resolved.Output.Module, resolved.Output.Options); err != nil {
|
||||
return moduleOptionsError(resolved.ID, "", StageOutput, resolved.Output.Module, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateChainOptions(resolved ResolvedPipeline, catalog ModuleCatalog, stage ModuleStage, laneID, moduleKey string) error {
|
||||
chain := resolvedValidatorChain(stage, laneID, moduleKey, resolved.ValidatorChains)
|
||||
for _, validator := range chain.Validators {
|
||||
if err := catalog.Validators.validateOptions(validator); err != nil {
|
||||
return validatorOptionsError(resolved.ID, laneID, stage, moduleKey, validator.Binding.Module, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func moduleOptionsError(pipelineID, laneID string, stage ModuleStage, moduleKey string, cause error) error {
|
||||
if laneID == "" {
|
||||
return fmt.Errorf("pipeline %q %s module %q options: %w", pipelineID, stage, moduleKey, cause)
|
||||
}
|
||||
return fmt.Errorf("pipeline %q lane %q %s module %q options: %w", pipelineID, laneID, stage, moduleKey, cause)
|
||||
}
|
||||
|
||||
func validatorOptionsError(pipelineID, laneID string, stage ModuleStage, moduleKey, validatorKey string, cause error) error {
|
||||
if laneID == "" {
|
||||
return fmt.Errorf("pipeline %q %s module %q validator %q options: %w", pipelineID, stage, moduleKey, validatorKey, cause)
|
||||
}
|
||||
return fmt.Errorf("pipeline %q lane %q %s module %q validator %q options: %w", pipelineID, laneID, stage, moduleKey, validatorKey, cause)
|
||||
}
|
||||
@@ -8,16 +8,19 @@ import (
|
||||
)
|
||||
|
||||
type OutputEncoderConstructor func() (contracts.OutputEncoder, error)
|
||||
type OutputEncoderBuilder func(BuildRequest) (contracts.OutputEncoder, error)
|
||||
|
||||
type OutputEncoderRegistry struct {
|
||||
constructors map[string]OutputEncoderConstructor
|
||||
specs map[string]ModuleSpec
|
||||
builders map[string]OutputEncoderBuilder
|
||||
optionValidators map[string]OptionValidator
|
||||
specs map[string]ModuleSpec
|
||||
}
|
||||
|
||||
func NewOutputEncoderRegistry() *OutputEncoderRegistry {
|
||||
return &OutputEncoderRegistry{
|
||||
constructors: make(map[string]OutputEncoderConstructor),
|
||||
specs: make(map[string]ModuleSpec),
|
||||
builders: make(map[string]OutputEncoderBuilder),
|
||||
optionValidators: make(map[string]OptionValidator),
|
||||
specs: make(map[string]ModuleSpec),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -26,6 +29,15 @@ func (r *OutputEncoderRegistry) Register(key string, constructor OutputEncoderCo
|
||||
}
|
||||
|
||||
func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor OutputEncoderConstructor) error {
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("output encoder constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||
}
|
||||
return r.RegisterBuilderWithSpec(spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.OutputEncoder, error) {
|
||||
return constructor()
|
||||
})
|
||||
}
|
||||
|
||||
func (r *OutputEncoderRegistry) RegisterBuilderWithSpec(spec ModuleSpec, validateOptions OptionValidator, builder OutputEncoderBuilder) error {
|
||||
if r == nil {
|
||||
return fmt.Errorf("output encoder registry must not be nil")
|
||||
}
|
||||
@@ -34,25 +46,36 @@ func (r *OutputEncoderRegistry) RegisterWithSpec(spec ModuleSpec, constructor Ou
|
||||
if err := validateModuleSpec("output encoder", StageOutput, normalizedSpec); err != nil {
|
||||
return err
|
||||
}
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("output encoder constructor for %q must not be nil", normalizedSpec.Key)
|
||||
if validateOptions == nil {
|
||||
return fmt.Errorf("output encoder option validator for %q must not be nil", normalizedSpec.Key)
|
||||
}
|
||||
if _, ok := r.constructors[normalizedSpec.Key]; ok {
|
||||
if builder == nil {
|
||||
return fmt.Errorf("output encoder builder for %q must not be nil", normalizedSpec.Key)
|
||||
}
|
||||
if _, ok := r.builders[normalizedSpec.Key]; ok {
|
||||
return fmt.Errorf("output encoder %q is already registered", normalizedSpec.Key)
|
||||
}
|
||||
|
||||
if r.constructors == nil {
|
||||
r.constructors = make(map[string]OutputEncoderConstructor)
|
||||
if r.builders == nil {
|
||||
r.builders = make(map[string]OutputEncoderBuilder)
|
||||
}
|
||||
if r.optionValidators == nil {
|
||||
r.optionValidators = make(map[string]OptionValidator)
|
||||
}
|
||||
if r.specs == nil {
|
||||
r.specs = make(map[string]ModuleSpec)
|
||||
}
|
||||
r.constructors[normalizedSpec.Key] = constructor
|
||||
r.builders[normalizedSpec.Key] = builder
|
||||
r.optionValidators[normalizedSpec.Key] = validateOptions
|
||||
r.specs[normalizedSpec.Key] = cloneModuleSpec(normalizedSpec)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *OutputEncoderRegistry) Build(key string) (contracts.OutputEncoder, error) {
|
||||
return r.BuildWithRequest(key, BuildRequest{})
|
||||
}
|
||||
|
||||
func (r *OutputEncoderRegistry) BuildWithRequest(key string, request BuildRequest) (contracts.OutputEncoder, error) {
|
||||
if r == nil {
|
||||
return nil, fmt.Errorf("output encoder registry must not be nil")
|
||||
}
|
||||
@@ -62,12 +85,12 @@ func (r *OutputEncoderRegistry) Build(key string) (contracts.OutputEncoder, erro
|
||||
return nil, fmt.Errorf("output encoder key must not be empty")
|
||||
}
|
||||
|
||||
constructor, ok := r.constructors[normalizedKey]
|
||||
builder, ok := r.builders[normalizedKey]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("output encoder %q is not registered", normalizedKey)
|
||||
}
|
||||
|
||||
encoder, err := constructor()
|
||||
encoder, err := builder(cloneBuildRequest(request))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("build output encoder %q: %w", normalizedKey, err)
|
||||
}
|
||||
@@ -81,6 +104,18 @@ func (r *OutputEncoderRegistry) Build(key string) (contracts.OutputEncoder, erro
|
||||
return encoder, nil
|
||||
}
|
||||
|
||||
func (r *OutputEncoderRegistry) ValidateOptions(key string, options map[string]any) error {
|
||||
if r == nil {
|
||||
return fmt.Errorf("output encoder registry must not be nil")
|
||||
}
|
||||
normalizedKey := strings.TrimSpace(key)
|
||||
validator, ok := r.optionValidators[normalizedKey]
|
||||
if !ok {
|
||||
return fmt.Errorf("output encoder %q is not registered", normalizedKey)
|
||||
}
|
||||
return validateRegisteredOptions(validator, options)
|
||||
}
|
||||
|
||||
func (r *OutputEncoderRegistry) Spec(key string) (ModuleSpec, bool) {
|
||||
if r == nil {
|
||||
return ModuleSpec{}, false
|
||||
@@ -98,5 +133,5 @@ func (r *OutputEncoderRegistry) RegisteredKeys() []string {
|
||||
return nil
|
||||
}
|
||||
|
||||
return sortedRegistryKeys(r.constructors)
|
||||
return sortedRegistryKeys(r.builders)
|
||||
}
|
||||
|
||||
240
internal/framework/pipeline/preparation_test.go
Normal file
240
internal/framework/pipeline/preparation_test.go
Normal file
@@ -0,0 +1,240 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func TestResolvePipelineValidatesModuleAndValidatorOptions(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
mutate func(*PipelineProfile)
|
||||
want []string
|
||||
}{
|
||||
{
|
||||
name: "module",
|
||||
mutate: func(profile *PipelineProfile) {
|
||||
profile.Input.Options = map[string]any{"surprise": true}
|
||||
},
|
||||
want: []string{`pipeline "construction" input module "input" options`, `unknown option "surprise"`},
|
||||
},
|
||||
{
|
||||
name: "validator",
|
||||
mutate: func(profile *PipelineProfile) {
|
||||
profile.Chunk.Validators = ValidatorOverride{Set: true, Validators: []ModuleBinding{{Module: "configured", Options: map[string]any{"surprise": true}}}}
|
||||
},
|
||||
want: []string{`pipeline "construction" chunk module "chunk" validator "configured" options`, `unknown option "surprise"`},
|
||||
},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
registries, _ := constructionRegistries(t, nil, nil)
|
||||
profile := constructionProfile()
|
||||
test.mutate(&profile)
|
||||
_, err := ResolvePipeline(profile, ResolveOptions{}, registries.catalog())
|
||||
if err == nil {
|
||||
t.Fatal("ResolvePipeline() error = nil, want option validation error")
|
||||
}
|
||||
for _, want := range test.want {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Fatalf("ResolvePipeline() error = %q, want substring %q", err, want)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrepareConstructsEverythingInStableOrder(t *testing.T) {
|
||||
var built []string
|
||||
registries, _ := constructionRegistries(t, &built, nil)
|
||||
resolved, err := ResolvePipeline(constructionProfile(), ResolveOptions{}, registries.catalog())
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
prepared, err := Prepare(resolved, registries, ModuleDependencies{})
|
||||
if err != nil {
|
||||
t.Fatalf("Prepare() error = %v, want nil", err)
|
||||
}
|
||||
want := []string{"input", "chunk", "validator", "extract", "validator", "merge", "validator", "normalize", "validator", "output"}
|
||||
if !reflect.DeepEqual(built, want) {
|
||||
t.Fatalf("construction order = %#v, want %#v", built, want)
|
||||
}
|
||||
if prepared.Input.Module != "input" || prepared.Chunk.Module != "chunk" || prepared.Output.Module != "output" || len(prepared.ArtifactLanes) != 1 {
|
||||
t.Fatalf("PreparedPipeline = %#v, want explicit resolved components", prepared)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrepareFailuresOccurBeforeInputParse(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
deps ModuleDependencies
|
||||
configure func(*constructionFailure)
|
||||
want string
|
||||
wantBuilt []string
|
||||
}{
|
||||
{
|
||||
name: "missing required llm dependency",
|
||||
configure: func(failure *constructionFailure) {
|
||||
failure.requireExtractorLLM = true
|
||||
},
|
||||
want: `lane "artifact" extract module "extract"`,
|
||||
wantBuilt: []string{"input", "chunk", "validator", "extract"},
|
||||
},
|
||||
{
|
||||
name: "late output construction",
|
||||
configure: func(failure *constructionFailure) {
|
||||
failure.output = errors.New("output unavailable")
|
||||
},
|
||||
want: `output module "output"`,
|
||||
wantBuilt: []string{"input", "chunk", "validator", "extract", "validator", "merge", "validator", "normalize", "validator", "output"},
|
||||
},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
failure := &constructionFailure{}
|
||||
test.configure(failure)
|
||||
var built []string
|
||||
registries, input := constructionRegistries(t, &built, failure)
|
||||
resolved, err := ResolvePipeline(constructionProfile(), ResolveOptions{}, registries.catalog())
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
_, err = Prepare(resolved, registries, test.deps)
|
||||
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("Prepare() error = %v, want substring %q", err, test.want)
|
||||
}
|
||||
if len(input.requests) != 0 {
|
||||
t.Fatalf("input Parse calls = %d, want zero", len(input.requests))
|
||||
}
|
||||
if !reflect.DeepEqual(built, test.wantBuilt) {
|
||||
t.Fatalf("construction order = %#v, want %#v", built, test.wantBuilt)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type constructionFailure struct {
|
||||
requireExtractorLLM bool
|
||||
output error
|
||||
}
|
||||
|
||||
func constructionProfile() PipelineProfile {
|
||||
validators := ValidatorOverride{Set: true, Validators: []ModuleBinding{{Module: "configured"}}}
|
||||
return PipelineProfile{
|
||||
ID: "construction",
|
||||
Input: Binding("input"),
|
||||
Chunk: ModuleBinding{Module: "chunk", Validators: validators},
|
||||
Artifacts: map[string]ArtifactLaneProfile{
|
||||
"artifact": {
|
||||
Extract: ModuleBinding{Module: "extract", Validators: validators},
|
||||
Merge: ModuleBinding{Module: "merge", Validators: validators},
|
||||
Normalize: ModuleBinding{Module: "normalize", Validators: validators},
|
||||
},
|
||||
},
|
||||
Output: Binding("output"),
|
||||
}
|
||||
}
|
||||
|
||||
func constructionRegistries(t *testing.T, built *[]string, failure *constructionFailure) (Registries, *constructionInput) {
|
||||
t.Helper()
|
||||
if built == nil {
|
||||
built = &[]string{}
|
||||
}
|
||||
if failure == nil {
|
||||
failure = &constructionFailure{}
|
||||
}
|
||||
record := func(name string) { *built = append(*built, name) }
|
||||
strict := func(options map[string]any) error { return RejectUnknownOptions(options, "known") }
|
||||
input := &constructionInput{key: "input"}
|
||||
registries := Registries{
|
||||
Inputs: NewInputAdapterRegistry(), Chunkers: NewChunkerRegistry(), ArtifactCodecs: NewArtifactCodecRegistry(),
|
||||
Extractors: NewExtractorRegistry(), Mergers: NewMergerRegistry(), Normalizers: NewNormalizerRegistry(),
|
||||
Validators: NewValidatorRegistry(), ValidatorChains: NewValidatorChainRegistry(), Outputs: NewOutputEncoderRegistry(),
|
||||
}
|
||||
if err := RegisterArtifactCodec(registries.ArtifactCodecs, notesCodec()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := registries.Inputs.RegisterBuilderWithSpec(defaultModuleSpec("input", StageInput), strict, func(BuildRequest) (contracts.InputAdapter, error) {
|
||||
record("input")
|
||||
return input, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := registries.Chunkers.RegisterBuilderWithSpec(defaultModuleSpec("chunk", StageChunk), strict, func(BuildRequest) (contracts.Chunker, error) {
|
||||
record("chunk")
|
||||
return &typedTestChunker{key: "chunk"}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
extractSpec := defaultModuleSpec("extract", StageExtract)
|
||||
extractSpec.ArtifactKind = "test/notes"
|
||||
if err := RegisterExtractorBuilder(registries.Extractors, extractSpec, strict, func(request BuildRequest) (contracts.Extractor[codecNotes], error) {
|
||||
record("extract")
|
||||
if failure.requireExtractorLLM && request.Dependencies.LLM == nil {
|
||||
return nil, errors.New("structured LLM client is required")
|
||||
}
|
||||
return typedTestExtractor[codecNotes]{key: "extract"}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mergeSpec := defaultModuleSpec("merge", StageMerge)
|
||||
mergeSpec.ArtifactKind = "test/notes"
|
||||
if err := RegisterMergerBuilder(registries.Mergers, mergeSpec, strict, func(BuildRequest) (contracts.Merger[codecNotes], error) {
|
||||
record("merge")
|
||||
return typedTestMerger[codecNotes]{key: "merge"}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
normalizeSpec := defaultModuleSpec("normalize", StageNormalize)
|
||||
normalizeSpec.ArtifactKind = "test/notes"
|
||||
if err := RegisterNormalizerBuilder(registries.Normalizers, normalizeSpec, strict, func(BuildRequest) (contracts.Normalizer[codecNotes], error) {
|
||||
record("normalize")
|
||||
return typedTestNormalizer[codecNotes]{key: "normalize"}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
validatorSpec := ValidatorSpec{Key: "configured", ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||
if err := RegisterChunkValidatorBuilder(registries.Validators, validatorSpec, strict, func(BuildRequest) (contracts.ChunkValidator, error) {
|
||||
record("validator")
|
||||
return typedTestChunkValidator{key: "configured"}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := RegisterTypedValidatorBuilder(registries.Validators, "test/notes", validatorSpec, strict, func(BuildRequest) (contracts.TypedValidator[codecNotes], error) {
|
||||
record("validator")
|
||||
return typedTestValidator[codecNotes]{key: "configured"}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := registries.Outputs.RegisterBuilderWithSpec(defaultModuleSpec("output", StageOutput), strict, func(BuildRequest) (contracts.OutputEncoder, error) {
|
||||
record("output")
|
||||
if failure.output != nil {
|
||||
return nil, failure.output
|
||||
}
|
||||
return &typedTestOutput{key: "output"}, nil
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return registries, input
|
||||
}
|
||||
|
||||
type constructionInput struct {
|
||||
key string
|
||||
requests []contracts.ParseRequest
|
||||
}
|
||||
|
||||
func (input *constructionInput) Key() string { return input.key }
|
||||
func (input *constructionInput) Parse(_ context.Context, request contracts.ParseRequest) (*source.SourceDocument, error) {
|
||||
input.requests = append(input.requests, request)
|
||||
return typedTestDocument(), nil
|
||||
}
|
||||
348
internal/framework/pipeline/prepare.go
Normal file
348
internal/framework/pipeline/prepare.go
Normal file
@@ -0,0 +1,348 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"reflect"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
// PreparedPipeline owns the constructed, run-local implementation set for one
|
||||
// resolved pipeline. Its implementation values are private so execution cannot
|
||||
// replace or reconfigure them after preparation.
|
||||
type PreparedPipeline struct {
|
||||
Input ModuleBinding
|
||||
Chunk ModuleBinding
|
||||
ArtifactLanes []PreparedArtifactLane
|
||||
Output ModuleBinding
|
||||
|
||||
resolved ResolvedPipeline
|
||||
dependencies ModuleDependencies
|
||||
input contracts.InputAdapter
|
||||
chunker contracts.Chunker
|
||||
chunkValidators preparedValidatorChain
|
||||
lanes []preparedLaneExecutor
|
||||
output contracts.OutputEncoder
|
||||
}
|
||||
|
||||
type PreparedArtifactLane struct {
|
||||
Resolved ResolvedArtifactLane
|
||||
}
|
||||
|
||||
type preparedLaneExecutor struct {
|
||||
resolved ResolvedArtifactLane
|
||||
typed *preparedTypedLane
|
||||
extractValidators preparedValidatorChain
|
||||
mergeValidators preparedValidatorChain
|
||||
normalizeValidators preparedValidatorChain
|
||||
}
|
||||
|
||||
type preparedTypedLane struct {
|
||||
extractor any
|
||||
merger any
|
||||
normalizer any
|
||||
extract typedExtractOperation
|
||||
merge typedMergeOperation
|
||||
normalize typedNormalizeOperation
|
||||
codec artifactCodecEntry
|
||||
}
|
||||
|
||||
type preparedValidatorChain struct {
|
||||
resolved ResolvedValidatorChain
|
||||
validators []preparedValidator
|
||||
}
|
||||
|
||||
type preparedValidator struct {
|
||||
resolved ResolvedValidator
|
||||
typed any
|
||||
typedValidate typedValidateOperation
|
||||
chunk contracts.ChunkValidator
|
||||
serialized contracts.SerializedValidator
|
||||
}
|
||||
|
||||
// Prepare validates all configured options and constructs every selected
|
||||
// module and validator before any operation method can run.
|
||||
func Prepare(resolved ResolvedPipeline, registries Registries, deps ModuleDependencies) (*PreparedPipeline, error) {
|
||||
if err := validateResolvedPipeline(resolved); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := validateRegistrySet(resolved, registries); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
stable := cloneResolvedPipeline(resolved)
|
||||
prepared := &PreparedPipeline{
|
||||
Input: cloneModuleBinding(stable.Input),
|
||||
Chunk: cloneModuleBinding(stable.Chunk),
|
||||
Output: cloneModuleBinding(stable.Output),
|
||||
resolved: stable,
|
||||
dependencies: deps,
|
||||
}
|
||||
request := func(binding ModuleBinding) BuildRequest {
|
||||
return BuildRequest{Dependencies: deps, Options: cloneOptions(binding.Options)}
|
||||
}
|
||||
|
||||
input, err := registries.Inputs.BuildWithRequest(stable.Input.Module, request(stable.Input))
|
||||
if err != nil {
|
||||
return nil, constructionError(stable.ID, "", StageInput, stable.Input.Module, "", err)
|
||||
}
|
||||
prepared.input = input
|
||||
|
||||
chunker, err := registries.Chunkers.BuildWithRequest(stable.Chunk.Module, request(stable.Chunk))
|
||||
if err != nil {
|
||||
return nil, constructionError(stable.ID, "", StageChunk, stable.Chunk.Module, "", err)
|
||||
}
|
||||
prepared.chunker = chunker
|
||||
prepared.chunkValidators, err = prepareValidatorChain(stable, registries, deps, StageChunk, "", stable.Chunk.Module)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
prepared.ArtifactLanes = make([]PreparedArtifactLane, 0, len(stable.ArtifactLanes))
|
||||
prepared.lanes = make([]preparedLaneExecutor, 0, len(stable.ArtifactLanes))
|
||||
for _, lane := range stable.ArtifactLanes {
|
||||
executor, err := prepareLane(stable, lane, registries, deps)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
prepared.ArtifactLanes = append(prepared.ArtifactLanes, PreparedArtifactLane{Resolved: cloneResolvedArtifactLane(lane)})
|
||||
prepared.lanes = append(prepared.lanes, executor)
|
||||
}
|
||||
|
||||
output, err := registries.Outputs.BuildWithRequest(stable.Output.Module, request(stable.Output))
|
||||
if err != nil {
|
||||
return nil, constructionError(stable.ID, "", StageOutput, stable.Output.Module, "", err)
|
||||
}
|
||||
prepared.output = output
|
||||
return prepared, nil
|
||||
}
|
||||
|
||||
func prepareLane(pipeline ResolvedPipeline, lane ResolvedArtifactLane, registries Registries, deps ModuleDependencies) (preparedLaneExecutor, error) {
|
||||
executor := preparedLaneExecutor{resolved: cloneResolvedArtifactLane(lane)}
|
||||
request := func(binding ModuleBinding) BuildRequest {
|
||||
return BuildRequest{Dependencies: deps, Options: cloneOptions(binding.Options)}
|
||||
}
|
||||
extractEntry, ok := registries.Extractors.typedEntry(lane.Extract.Module)
|
||||
if !ok {
|
||||
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageExtract, lane.Extract.Module, "", fmt.Errorf("typed construction entry is not registered"))
|
||||
}
|
||||
module, err := buildErasedModule(extractEntry.builder, request(lane.Extract), lane.Extract.Module, "extractor")
|
||||
if err != nil {
|
||||
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageExtract, lane.Extract.Module, "", err)
|
||||
}
|
||||
codec, _, codecErr := registries.ArtifactCodecs.entry(lane.ArtifactKind)
|
||||
if codecErr != nil {
|
||||
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageExtract, lane.Extract.Module, "", codecErr)
|
||||
}
|
||||
executor.typed = &preparedTypedLane{extractor: module, extract: extractEntry.extract, codec: codec}
|
||||
|
||||
executor.extractValidators, err = prepareValidatorChain(pipeline, registries, deps, StageExtract, lane.ID, lane.Extract.Module)
|
||||
if err != nil {
|
||||
return preparedLaneExecutor{}, err
|
||||
}
|
||||
|
||||
mergeEntry, ok := registries.Mergers.typedEntry(lane.Merge.Module, lane.ArtifactKind)
|
||||
if !ok {
|
||||
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageMerge, lane.Merge.Module, "", fmt.Errorf("typed construction entry is not registered"))
|
||||
}
|
||||
module, err = buildErasedModule(mergeEntry.builder, request(lane.Merge), lane.Merge.Module, "merger")
|
||||
if err != nil {
|
||||
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageMerge, lane.Merge.Module, "", err)
|
||||
}
|
||||
executor.typed.merger = module
|
||||
executor.typed.merge = mergeEntry.merge
|
||||
executor.mergeValidators, err = prepareValidatorChain(pipeline, registries, deps, StageMerge, lane.ID, lane.Merge.Module)
|
||||
if err != nil {
|
||||
return preparedLaneExecutor{}, err
|
||||
}
|
||||
|
||||
normalizeEntry, ok := registries.Normalizers.typedEntry(lane.Normalize.Module, lane.ArtifactKind)
|
||||
if !ok {
|
||||
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageNormalize, lane.Normalize.Module, "", fmt.Errorf("typed construction entry is not registered"))
|
||||
}
|
||||
module, err = buildErasedModule(normalizeEntry.builder, request(lane.Normalize), lane.Normalize.Module, "normalizer")
|
||||
if err != nil {
|
||||
return preparedLaneExecutor{}, constructionError(pipeline.ID, lane.ID, StageNormalize, lane.Normalize.Module, "", err)
|
||||
}
|
||||
executor.typed.normalizer = module
|
||||
executor.typed.normalize = normalizeEntry.normalize
|
||||
executor.normalizeValidators, err = prepareValidatorChain(pipeline, registries, deps, StageNormalize, lane.ID, lane.Normalize.Module)
|
||||
if err != nil {
|
||||
return preparedLaneExecutor{}, err
|
||||
}
|
||||
return executor, nil
|
||||
}
|
||||
|
||||
func prepareValidatorChain(pipeline ResolvedPipeline, registries Registries, deps ModuleDependencies, stage ModuleStage, laneID, moduleKey string) (preparedValidatorChain, error) {
|
||||
resolved := resolvedValidatorChain(stage, laneID, moduleKey, pipeline.ValidatorChains)
|
||||
prepared := preparedValidatorChain{resolved: resolved}
|
||||
for _, validator := range resolved.Validators {
|
||||
request := BuildRequest{Dependencies: deps, Options: cloneOptions(validator.Binding.Options)}
|
||||
built, err := buildPreparedValidator(registries.Validators, validator, request)
|
||||
if err != nil {
|
||||
return preparedValidatorChain{}, constructionError(pipeline.ID, laneID, stage, moduleKey, validator.Binding.Module, err)
|
||||
}
|
||||
prepared.validators = append(prepared.validators, built)
|
||||
}
|
||||
return prepared, nil
|
||||
}
|
||||
|
||||
func buildPreparedValidator(registry *ValidatorRegistry, resolved ResolvedValidator, request BuildRequest) (preparedValidator, error) {
|
||||
prepared := preparedValidator{resolved: resolved}
|
||||
key := resolved.Binding.Module
|
||||
var implementation any
|
||||
var err error
|
||||
switch resolved.Target {
|
||||
case ValidatorTargetTyped:
|
||||
entry, ok := registry.typedEntry(key, resolved.ArtifactKind)
|
||||
if !ok {
|
||||
return preparedValidator{}, fmt.Errorf("typed construction entry is not registered")
|
||||
}
|
||||
implementation, err = entry.builder(cloneBuildRequest(request))
|
||||
prepared.typed = implementation
|
||||
prepared.typedValidate = entry.validate
|
||||
case ValidatorTargetChunk:
|
||||
entry, ok := registry.chunkEntry(key)
|
||||
if !ok {
|
||||
return preparedValidator{}, fmt.Errorf("chunk construction entry is not registered")
|
||||
}
|
||||
prepared.chunk, err = entry.builder(cloneBuildRequest(request))
|
||||
implementation = prepared.chunk
|
||||
case ValidatorTargetSerialized:
|
||||
entry, ok := registry.serializedEntry(key)
|
||||
if !ok {
|
||||
return preparedValidator{}, fmt.Errorf("serialized construction entry is not registered")
|
||||
}
|
||||
prepared.serialized, err = entry.builder(cloneBuildRequest(request))
|
||||
implementation = prepared.serialized
|
||||
default:
|
||||
return preparedValidator{}, fmt.Errorf("validator construction target %q is not supported", resolved.Target)
|
||||
}
|
||||
if err != nil {
|
||||
return preparedValidator{}, err
|
||||
}
|
||||
if isNilImplementation(implementation) {
|
||||
return preparedValidator{}, fmt.Errorf("validator %q builder returned nil", key)
|
||||
}
|
||||
identity, ok := implementation.(interface {
|
||||
Name() string
|
||||
ExecutionClass() contracts.ExecutionClass
|
||||
})
|
||||
if !ok {
|
||||
return preparedValidator{}, fmt.Errorf("validator %q builder returned incompatible implementation %T", key, implementation)
|
||||
}
|
||||
if identity.Name() != key {
|
||||
return preparedValidator{}, fmt.Errorf("validator %q returned name %q", key, identity.Name())
|
||||
}
|
||||
if identity.ExecutionClass() != resolved.ExecutionClass {
|
||||
return preparedValidator{}, fmt.Errorf("validator %q returned execution class %q, want %q", key, identity.ExecutionClass(), resolved.ExecutionClass)
|
||||
}
|
||||
return prepared, nil
|
||||
}
|
||||
|
||||
func buildErasedModule(builder func(BuildRequest) (any, error), request BuildRequest, key, kind string) (any, error) {
|
||||
implementation, err := builder(cloneBuildRequest(request))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if isNilImplementation(implementation) {
|
||||
return nil, fmt.Errorf("%s %q builder returned nil", kind, key)
|
||||
}
|
||||
identity, ok := implementation.(interface{ Key() string })
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("%s %q builder returned incompatible implementation %T", kind, key, implementation)
|
||||
}
|
||||
if identity.Key() != key {
|
||||
return nil, fmt.Errorf("%s %q returned key %q", kind, key, identity.Key())
|
||||
}
|
||||
return implementation, nil
|
||||
}
|
||||
|
||||
func isNilImplementation(value any) bool {
|
||||
if value == nil {
|
||||
return true
|
||||
}
|
||||
reflected := reflect.ValueOf(value)
|
||||
switch reflected.Kind() {
|
||||
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:
|
||||
return reflected.IsNil()
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
func constructionError(pipelineID, laneID string, stage ModuleStage, moduleKey, validatorKey string, cause error) error {
|
||||
scope := fmt.Sprintf("pipeline %q %s module %q", pipelineID, stage, moduleKey)
|
||||
if laneID != "" {
|
||||
scope = fmt.Sprintf("pipeline %q lane %q %s module %q", pipelineID, laneID, stage, moduleKey)
|
||||
}
|
||||
if strings.TrimSpace(validatorKey) != "" {
|
||||
scope += fmt.Sprintf(" validator %q", validatorKey)
|
||||
}
|
||||
return fmt.Errorf("prepare %s: %w", scope, cause)
|
||||
}
|
||||
|
||||
func (registries Registries) catalog() ModuleCatalog {
|
||||
return ModuleCatalog{
|
||||
Inputs: registries.Inputs, Chunkers: registries.Chunkers, ArtifactCodecs: registries.ArtifactCodecs,
|
||||
Extractors: registries.Extractors, Mergers: registries.Mergers, Normalizers: registries.Normalizers,
|
||||
Validators: registries.Validators, ValidatorChains: registries.ValidatorChains, Outputs: registries.Outputs,
|
||||
}
|
||||
}
|
||||
|
||||
func validateRegistrySet(resolved ResolvedPipeline, registries Registries) error {
|
||||
if registries.Inputs == nil {
|
||||
return fmt.Errorf("input registry must not be nil")
|
||||
}
|
||||
if registries.Chunkers == nil {
|
||||
return fmt.Errorf("chunker registry must not be nil")
|
||||
}
|
||||
if registries.Extractors == nil {
|
||||
return fmt.Errorf("extractor registry must not be nil")
|
||||
}
|
||||
if registries.Mergers == nil {
|
||||
return fmt.Errorf("merger registry must not be nil")
|
||||
}
|
||||
if registries.Normalizers == nil {
|
||||
return fmt.Errorf("normalizer registry must not be nil")
|
||||
}
|
||||
if registries.Validators == nil {
|
||||
for _, chain := range resolved.ValidatorChains {
|
||||
if len(chain.Validators) > 0 {
|
||||
return fmt.Errorf("validator registry must not be nil")
|
||||
}
|
||||
}
|
||||
}
|
||||
if registries.Outputs == nil {
|
||||
return fmt.Errorf("output encoder registry must not be nil")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func cloneResolvedPipeline(in ResolvedPipeline) ResolvedPipeline {
|
||||
out := in
|
||||
out.Input = cloneModuleBinding(in.Input)
|
||||
out.Chunk = cloneModuleBinding(in.Chunk)
|
||||
out.Output = cloneModuleBinding(in.Output)
|
||||
out.ChunkReferences = CloneReferenceTarget(in.ChunkReferences)
|
||||
out.ValidatorChains = cloneResolvedValidatorChains(in.ValidatorChains)
|
||||
if len(in.ArtifactLanes) > 0 {
|
||||
out.ArtifactLanes = make([]ResolvedArtifactLane, len(in.ArtifactLanes))
|
||||
for i, lane := range in.ArtifactLanes {
|
||||
out.ArtifactLanes[i] = cloneResolvedArtifactLane(lane)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func cloneResolvedArtifactLane(in ResolvedArtifactLane) ResolvedArtifactLane {
|
||||
out := in
|
||||
out.Extract = cloneModuleBinding(in.Extract)
|
||||
out.Merge = cloneModuleBinding(in.Merge)
|
||||
out.Normalize = cloneModuleBinding(in.Normalize)
|
||||
out.Validators = cloneModuleBindings(in.Validators)
|
||||
out.ExtractReferences = CloneReferenceTarget(in.ExtractReferences)
|
||||
out.MergeReferences = CloneReferenceTarget(in.MergeReferences)
|
||||
out.NormalizeReferences = CloneReferenceTarget(in.NormalizeReferences)
|
||||
return out
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
@@ -102,14 +103,19 @@ type ResolvedReferenceTarget struct {
|
||||
}
|
||||
|
||||
type ResolvedArtifactLane struct {
|
||||
ID string
|
||||
Extract ModuleBinding
|
||||
Merge ModuleBinding
|
||||
Normalize ModuleBinding
|
||||
Validators []ModuleBinding
|
||||
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
|
||||
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
|
||||
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
|
||||
ID string
|
||||
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
|
||||
ArtifactSchemaID string `json:"artifact_schema_id,omitempty"`
|
||||
ArtifactSchemaName string `json:"artifact_schema_name,omitempty"`
|
||||
ArtifactSchemaVersion string `json:"artifact_schema_version,omitempty"`
|
||||
ArtifactSchemaDigest string `json:"artifact_schema_digest,omitempty"`
|
||||
Extract ModuleBinding
|
||||
Merge ModuleBinding
|
||||
Normalize ModuleBinding
|
||||
Validators []ModuleBinding
|
||||
ExtractReferences ResolvedReferenceTarget `json:"extract_references"`
|
||||
MergeReferences ResolvedReferenceTarget `json:"merge_references"`
|
||||
NormalizeReferences ResolvedReferenceTarget `json:"normalize_references"`
|
||||
}
|
||||
|
||||
type ResolvedValidatorChain struct {
|
||||
@@ -122,6 +128,8 @@ type ResolvedValidatorChain struct {
|
||||
type ResolvedValidator struct {
|
||||
Binding ModuleBinding `json:"binding"`
|
||||
ExecutionClass contracts.ExecutionClass `json:"execution_class"`
|
||||
Target ValidatorTarget `json:"target,omitempty"`
|
||||
ArtifactKind contracts.ArtifactKind `json:"artifact_kind,omitempty"`
|
||||
}
|
||||
|
||||
type ResolvedPipeline struct {
|
||||
@@ -138,6 +146,7 @@ type ResolvedPipeline struct {
|
||||
type ModuleCatalog struct {
|
||||
Inputs *InputAdapterRegistry
|
||||
Chunkers *ChunkerRegistry
|
||||
ArtifactCodecs *ArtifactCodecRegistry
|
||||
Extractors *ExtractorRegistry
|
||||
Mergers *MergerRegistry
|
||||
Normalizers *NormalizerRegistry
|
||||
@@ -215,7 +224,7 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
||||
ChunkReferences: referenceTarget(StageChunk, "", chunk.Module, chunkReferences),
|
||||
Output: resolveBinding(profile.Output, DefaultOutputModule),
|
||||
}
|
||||
chunkValidatorChain, err := resolveValidatorChain(pipelineID, "", StageChunk, chunk.Module, chunk.Validators, catalog)
|
||||
chunkValidatorChain, err := resolveValidatorChain(pipelineID, "", StageChunk, chunk.Module, chunk.Validators, "", nil, catalog)
|
||||
if err != nil {
|
||||
return ResolvedPipeline{}, err
|
||||
}
|
||||
@@ -240,6 +249,9 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
|
||||
if missing, ok := outputCapabilities.missing(outputSpec.Requires); ok {
|
||||
return ResolvedPipeline{}, capabilityError(pipelineID, "", StageOutput, resolved.Output.Module, missing)
|
||||
}
|
||||
if err := validateResolvedOptions(resolved, catalog); err != nil {
|
||||
return ResolvedPipeline{}, err
|
||||
}
|
||||
|
||||
digest, err := resolvedPipelineDigest(resolved)
|
||||
if err != nil {
|
||||
@@ -278,6 +290,10 @@ func resolveArtifactLane(
|
||||
if missing, ok := capabilities.missing(extractSpec.Requires); ok {
|
||||
return ResolvedArtifactLane{}, nil, nil, capabilityError(pipelineID, laneID, StageExtract, lane.Extract.Module, missing)
|
||||
}
|
||||
artifactType, err := resolveArtifactIdentity(pipelineID, laneID, &lane, extractSpec, catalog)
|
||||
if err != nil {
|
||||
return ResolvedArtifactLane{}, nil, nil, err
|
||||
}
|
||||
extractReferences := mergeReferenceMaps(profile.References, lane.Extract.References)
|
||||
references, err := resolveReferenceTargetBindings(referenceResolutionTarget{
|
||||
PipelineID: pipelineID,
|
||||
@@ -295,7 +311,7 @@ func resolveArtifactLane(
|
||||
lane.ExtractReferences = referenceTarget(StageExtract, laneID, lane.Extract.Module, references)
|
||||
capabilities.add(extractSpec.Provides...)
|
||||
|
||||
mergeSpec, err := mergerSpec(catalog, lane.Merge.Module)
|
||||
mergeSpec, err := mergerSpecForArtifact(catalog, lane.Merge.Module, lane.ArtifactKind, artifactType)
|
||||
if err != nil {
|
||||
return ResolvedArtifactLane{}, nil, nil, moduleLookupError(pipelineID, laneID, StageMerge, lane.Merge.Module, err)
|
||||
}
|
||||
@@ -318,7 +334,7 @@ func resolveArtifactLane(
|
||||
lane.MergeReferences = referenceTarget(StageMerge, laneID, lane.Merge.Module, mergeReferences)
|
||||
capabilities.add(mergeSpec.Provides...)
|
||||
|
||||
normalizeSpec, err := normalizerSpec(catalog, lane.Normalize.Module)
|
||||
normalizeSpec, err := normalizerSpecForArtifact(catalog, lane.Normalize.Module, lane.ArtifactKind, artifactType)
|
||||
if err != nil {
|
||||
return ResolvedArtifactLane{}, nil, nil, moduleLookupError(pipelineID, laneID, StageNormalize, lane.Normalize.Module, err)
|
||||
}
|
||||
@@ -345,15 +361,15 @@ func resolveArtifactLane(
|
||||
return ResolvedArtifactLane{}, nil, nil, configuredValidatorsError(pipelineID, laneID)
|
||||
}
|
||||
|
||||
extractValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageExtract, lane.Extract.Module, lane.Extract.Validators, catalog)
|
||||
extractValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageExtract, lane.Extract.Module, lane.Extract.Validators, lane.ArtifactKind, artifactType, catalog)
|
||||
if err != nil {
|
||||
return ResolvedArtifactLane{}, nil, nil, err
|
||||
}
|
||||
mergeValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageMerge, lane.Merge.Module, lane.Merge.Validators, catalog)
|
||||
mergeValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageMerge, lane.Merge.Module, lane.Merge.Validators, lane.ArtifactKind, artifactType, catalog)
|
||||
if err != nil {
|
||||
return ResolvedArtifactLane{}, nil, nil, err
|
||||
}
|
||||
normalizeValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageNormalize, lane.Normalize.Module, lane.Normalize.Validators, catalog)
|
||||
normalizeValidatorChain, err := resolveValidatorChain(pipelineID, laneID, StageNormalize, lane.Normalize.Module, lane.Normalize.Validators, lane.ArtifactKind, artifactType, catalog)
|
||||
if err != nil {
|
||||
return ResolvedArtifactLane{}, nil, nil, err
|
||||
}
|
||||
@@ -366,7 +382,131 @@ func configuredValidatorsError(pipelineID string, laneID string) error {
|
||||
return fmt.Errorf("pipeline %q lane %q validators are not supported at artifact lane level; use extract.validators, merge.validators, or normalize.validators", pipelineID, laneID)
|
||||
}
|
||||
|
||||
func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage, module string, override ValidatorOverride, catalog ModuleCatalog) (ResolvedValidatorChain, error) {
|
||||
func resolveArtifactIdentity(pipelineID, laneID string, lane *ResolvedArtifactLane, extractSpec ModuleSpec, catalog ModuleCatalog) (reflect.Type, error) {
|
||||
if extractSpec.ArtifactKind == "" {
|
||||
return nil, fmt.Errorf("pipeline %q lane %q extract module %q does not declare an artifact kind", pipelineID, laneID, lane.Extract.Module)
|
||||
}
|
||||
if catalog.Extractors == nil {
|
||||
return nil, fmt.Errorf("pipeline %q lane %q extractor registry must not be nil", pipelineID, laneID)
|
||||
}
|
||||
extractor, ok := catalog.Extractors.typedEntry(lane.Extract.Module)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("pipeline %q lane %q extract module %q declares artifact kind %q without a typed registration", pipelineID, laneID, lane.Extract.Module, extractSpec.ArtifactKind)
|
||||
}
|
||||
if catalog.ArtifactCodecs == nil {
|
||||
return nil, fmt.Errorf("pipeline %q lane %q artifact codec registry must not be nil for kind %q", pipelineID, laneID, extractSpec.ArtifactKind)
|
||||
}
|
||||
codecSpec, ok := catalog.ArtifactCodecs.Spec(extractSpec.ArtifactKind)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("pipeline %q lane %q artifact codec %q is not registered", pipelineID, laneID, extractSpec.ArtifactKind)
|
||||
}
|
||||
codecType, ok := catalog.ArtifactCodecs.valueType(extractSpec.ArtifactKind)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("pipeline %q lane %q artifact codec %q has no Go type", pipelineID, laneID, extractSpec.ArtifactKind)
|
||||
}
|
||||
if extractor.valueType != codecType {
|
||||
return nil, artifactTypeMismatchError(pipelineID, laneID, StageExtract, lane.Extract.Module, extractSpec.ArtifactKind, codecType, extractor.valueType)
|
||||
}
|
||||
lane.ArtifactKind = codecSpec.Kind
|
||||
lane.ArtifactSchemaID = codecSpec.Schema.ID
|
||||
lane.ArtifactSchemaName = codecSpec.Schema.Name
|
||||
lane.ArtifactSchemaVersion = codecSpec.Schema.Version
|
||||
lane.ArtifactSchemaDigest = codecSpec.SchemaDigest
|
||||
return codecType, nil
|
||||
}
|
||||
|
||||
func mergerSpecForArtifact(catalog ModuleCatalog, key string, kind contracts.ArtifactKind, expectedType reflect.Type) (ModuleSpec, error) {
|
||||
if kind == "" {
|
||||
return mergerSpec(catalog, key)
|
||||
}
|
||||
if catalog.Mergers == nil {
|
||||
return ModuleSpec{}, fmt.Errorf("module %q is not registered", key)
|
||||
}
|
||||
entry, ok := catalog.Mergers.typedEntry(key, kind)
|
||||
if !ok {
|
||||
return ModuleSpec{}, missingArtifactVariantError("merger", key, kind, catalog.Mergers.registeredKinds(key))
|
||||
}
|
||||
if entry.valueType != expectedType {
|
||||
return ModuleSpec{}, fmt.Errorf("artifact kind %q requires Go type %s, but merger %q variant uses %s", kind, typeName(expectedType), key, typeName(entry.valueType))
|
||||
}
|
||||
return cloneModuleSpec(entry.spec), nil
|
||||
}
|
||||
|
||||
func normalizerSpecForArtifact(catalog ModuleCatalog, key string, kind contracts.ArtifactKind, expectedType reflect.Type) (ModuleSpec, error) {
|
||||
if kind == "" {
|
||||
return normalizerSpec(catalog, key)
|
||||
}
|
||||
if catalog.Normalizers == nil {
|
||||
return ModuleSpec{}, fmt.Errorf("module %q is not registered", key)
|
||||
}
|
||||
entry, ok := catalog.Normalizers.typedEntry(key, kind)
|
||||
if !ok {
|
||||
return ModuleSpec{}, missingArtifactVariantError("normalizer", key, kind, catalog.Normalizers.registeredKinds(key))
|
||||
}
|
||||
if entry.valueType != expectedType {
|
||||
return ModuleSpec{}, fmt.Errorf("artifact kind %q requires Go type %s, but normalizer %q variant uses %s", kind, typeName(expectedType), key, typeName(entry.valueType))
|
||||
}
|
||||
return cloneModuleSpec(entry.spec), nil
|
||||
}
|
||||
|
||||
func validatorSpecForTarget(registry *ValidatorRegistry, stage ModuleStage, key string, kind contracts.ArtifactKind, expectedType reflect.Type) (ValidatorSpec, ValidatorTarget, error) {
|
||||
key = strings.TrimSpace(key)
|
||||
if stage == StageChunk {
|
||||
if entry, ok := registry.chunkEntry(key); ok {
|
||||
return entry.spec, ValidatorTargetChunk, nil
|
||||
}
|
||||
if entry, ok := registry.serializedEntry(key); ok && entry.spec.SupportsChunks {
|
||||
return entry.spec.ValidatorSpec, ValidatorTargetSerialized, nil
|
||||
}
|
||||
if spec, ok := registry.Spec(key); ok {
|
||||
return spec, "", nil
|
||||
}
|
||||
return ValidatorSpec{}, "", fmt.Errorf("references unknown validator %q for chunk target", key)
|
||||
}
|
||||
if kind == "" {
|
||||
if spec, ok := registry.Spec(key); ok {
|
||||
return spec, "", nil
|
||||
}
|
||||
return ValidatorSpec{}, "", fmt.Errorf("references unknown validator %q without an artifact kind", key)
|
||||
}
|
||||
if entry, ok := registry.typedEntry(key, kind); ok {
|
||||
if entry.valueType != expectedType {
|
||||
return ValidatorSpec{}, "", fmt.Errorf("artifact kind %q requires Go type %s, but validator %q variant uses %s", kind, typeName(expectedType), key, typeName(entry.valueType))
|
||||
}
|
||||
return entry.spec, ValidatorTargetTyped, nil
|
||||
}
|
||||
if entry, ok := registry.serializedEntry(key); ok && entry.spec.SupportsArtifacts {
|
||||
return entry.spec.ValidatorSpec, ValidatorTargetSerialized, nil
|
||||
}
|
||||
if _, ok := registry.Spec(key); !ok {
|
||||
return ValidatorSpec{}, "", fmt.Errorf("references unknown validator %q for artifact kind %q", key, kind)
|
||||
}
|
||||
return ValidatorSpec{}, "", missingArtifactVariantError("validator", key, kind, registry.registeredTypedKinds(key))
|
||||
}
|
||||
|
||||
func missingArtifactVariantError(moduleType, key string, kind contracts.ArtifactKind, registered []contracts.ArtifactKind) error {
|
||||
if len(registered) == 0 {
|
||||
return fmt.Errorf("%s %q has no typed variant for artifact kind %q", moduleType, key, kind)
|
||||
}
|
||||
values := make([]string, len(registered))
|
||||
for i, value := range registered {
|
||||
values[i] = string(value)
|
||||
}
|
||||
return fmt.Errorf("%s %q has no typed variant for artifact kind %q; registered kinds: %s", moduleType, key, kind, strings.Join(values, ", "))
|
||||
}
|
||||
|
||||
func artifactTypeMismatchError(pipelineID, laneID string, stage ModuleStage, module string, kind contracts.ArtifactKind, expected, actual reflect.Type) error {
|
||||
return fmt.Errorf("pipeline %q lane %q %s module %q artifact kind %q requires Go type %s, got %s", pipelineID, laneID, stage, module, kind, typeName(expected), typeName(actual))
|
||||
}
|
||||
|
||||
func typeName(value reflect.Type) string {
|
||||
if value == nil {
|
||||
return "<nil>"
|
||||
}
|
||||
return value.String()
|
||||
}
|
||||
|
||||
func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage, module string, override ValidatorOverride, artifactKind contracts.ArtifactKind, artifactType reflect.Type, catalog ModuleCatalog) (ResolvedValidatorChain, error) {
|
||||
chain := ResolvedValidatorChain{
|
||||
Stage: stage,
|
||||
LaneID: strings.TrimSpace(laneID),
|
||||
@@ -395,9 +535,9 @@ func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage,
|
||||
}
|
||||
chain.Validators = make([]ResolvedValidator, 0, len(bindings))
|
||||
for _, validator := range bindings {
|
||||
spec, ok := catalog.Validators.Spec(validator.Module)
|
||||
if !ok {
|
||||
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q references unknown validator %q", pipelineID, stage, chain.ModuleKey, validator.Module)
|
||||
spec, target, err := validatorSpecForTarget(catalog.Validators, stage, validator.Module, artifactKind, artifactType)
|
||||
if err != nil {
|
||||
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q: %w", pipelineID, stage, chain.ModuleKey, err)
|
||||
}
|
||||
if strings.TrimSpace(validator.LLMProfile) != "" && spec.ExecutionClass != contracts.ExecutionClassLLMBacked {
|
||||
return ResolvedValidatorChain{}, fmt.Errorf("pipeline %q %s validator chain for module %q assigns llm_profile to deterministic validator %q", pipelineID, stage, chain.ModuleKey, validator.Module)
|
||||
@@ -405,6 +545,8 @@ func resolveValidatorChain(pipelineID string, laneID string, stage ModuleStage,
|
||||
chain.Validators = append(chain.Validators, ResolvedValidator{
|
||||
Binding: cloneModuleBinding(validator),
|
||||
ExecutionClass: spec.ExecutionClass,
|
||||
Target: target,
|
||||
ArtifactKind: artifactKind,
|
||||
})
|
||||
}
|
||||
return chain, nil
|
||||
@@ -435,6 +577,8 @@ func cloneResolvedValidators(validators []ResolvedValidator) []ResolvedValidator
|
||||
out[i] = ResolvedValidator{
|
||||
Binding: cloneModuleBinding(validator.Binding),
|
||||
ExecutionClass: validator.ExecutionClass,
|
||||
Target: validator.Target,
|
||||
ArtifactKind: validator.ArtifactKind,
|
||||
}
|
||||
}
|
||||
return out
|
||||
@@ -497,7 +641,14 @@ func validatePipelineReferenceDefaults(
|
||||
}
|
||||
|
||||
merge := resolveBinding(laneProfile.Merge, DefaultMergeModule)
|
||||
mergeSpec, err := mergerSpec(catalog, merge.Module)
|
||||
var artifactType reflect.Type
|
||||
artifactKind := extractSpec.ArtifactKind
|
||||
if artifactKind != "" && catalog.Extractors != nil {
|
||||
if entry, ok := catalog.Extractors.typedEntry(extract.Module); ok {
|
||||
artifactType = entry.valueType
|
||||
}
|
||||
}
|
||||
mergeSpec, err := mergerSpecForArtifact(catalog, merge.Module, artifactKind, artifactType)
|
||||
if err != nil {
|
||||
return moduleLookupError(pipelineID, laneID, StageMerge, merge.Module, err)
|
||||
}
|
||||
@@ -506,7 +657,7 @@ func validatePipelineReferenceDefaults(
|
||||
}
|
||||
|
||||
normalize := resolveBinding(laneProfile.Normalize, DefaultNormalizeModule)
|
||||
normalizeSpec, err := normalizerSpec(catalog, normalize.Module)
|
||||
normalizeSpec, err := normalizerSpecForArtifact(catalog, normalize.Module, artifactKind, artifactType)
|
||||
if err != nil {
|
||||
return moduleLookupError(pipelineID, laneID, StageNormalize, normalize.Module, err)
|
||||
}
|
||||
|
||||
@@ -766,20 +766,22 @@ func TestResolvePipelineUsesReferenceSlotsFromSpecWithoutConstructingExtractor(t
|
||||
profile := baselineProfile()
|
||||
profile.References = map[string]string{"roster": "./roster.yml"}
|
||||
catalog := emptyProfileCatalog()
|
||||
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, notesCodec())
|
||||
for _, spec := range defaultProfileSpecs() {
|
||||
if spec.Key != "event-extractor" {
|
||||
registerProfileSpecs(t, catalog, spec)
|
||||
}
|
||||
}
|
||||
if err := catalog.Extractors.RegisterWithSpec(ModuleSpec{
|
||||
Key: "event-extractor",
|
||||
Stage: StageExtract,
|
||||
Requires: []string{"chunk"},
|
||||
Provides: []string{"candidate"},
|
||||
if err := RegisterExtractor[codecNotes](catalog.Extractors, ModuleSpec{
|
||||
Key: "event-extractor",
|
||||
Stage: StageExtract,
|
||||
ArtifactKind: "test/notes",
|
||||
Requires: []string{"chunk"},
|
||||
Provides: []string{"candidate"},
|
||||
ReferenceSlots: []contracts.ReferenceSlot{
|
||||
{Name: "roster", Required: true},
|
||||
},
|
||||
}, func() (contracts.Extractor, error) {
|
||||
}, func() (contracts.Extractor[codecNotes], error) {
|
||||
return nil, errors.New("constructor should not run")
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
||||
@@ -1177,6 +1179,7 @@ func newProfileCatalog(t *testing.T) ModuleCatalog {
|
||||
t.Helper()
|
||||
|
||||
catalog := emptyProfileCatalog()
|
||||
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, notesCodec())
|
||||
registerProfileSpecs(t, catalog, defaultProfileSpecs()...)
|
||||
return catalog
|
||||
}
|
||||
@@ -1192,6 +1195,9 @@ func newProfileCatalogWithOverrides(t *testing.T, overrides ...ModuleSpec) Modul
|
||||
|
||||
specs := defaultProfileSpecs()
|
||||
for _, override := range overrides {
|
||||
if override.ArtifactKind == "" && (override.Stage == StageExtract || override.Stage == StageMerge || override.Stage == StageNormalize) {
|
||||
override.ArtifactKind = "test/notes"
|
||||
}
|
||||
replaced := false
|
||||
for index, spec := range specs {
|
||||
if spec.Stage == override.Stage && spec.Key == override.Key {
|
||||
@@ -1206,6 +1212,7 @@ func newProfileCatalogWithOverrides(t *testing.T, overrides ...ModuleSpec) Modul
|
||||
}
|
||||
|
||||
catalog := emptyProfileCatalog()
|
||||
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, notesCodec())
|
||||
registerProfileSpecs(t, catalog, specs...)
|
||||
return catalog
|
||||
}
|
||||
@@ -1214,6 +1221,7 @@ func emptyProfileCatalog() ModuleCatalog {
|
||||
return ModuleCatalog{
|
||||
Inputs: NewInputAdapterRegistry(),
|
||||
Chunkers: NewChunkerRegistry(),
|
||||
ArtifactCodecs: NewArtifactCodecRegistry(),
|
||||
Extractors: NewExtractorRegistry(),
|
||||
Mergers: NewMergerRegistry(),
|
||||
Normalizers: NewNormalizerRegistry(),
|
||||
@@ -1227,10 +1235,10 @@ func defaultProfileSpecs() []ModuleSpec {
|
||||
return []ModuleSpec{
|
||||
ModuleSpec{Key: "text", Stage: StageInput, Provides: []string{"source"}},
|
||||
ModuleSpec{Key: "generic", Stage: StageChunk, Requires: []string{"source"}, Provides: []string{"chunk"}},
|
||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||
ModuleSpec{Key: "note-extractor", Stage: StageExtract, Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||
ModuleSpec{Key: "appendorder", Stage: StageMerge, Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||
ModuleSpec{Key: "noop", Stage: StageNormalize, Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||
ModuleSpec{Key: "event-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||
ModuleSpec{Key: "note-extractor", Stage: StageExtract, ArtifactKind: "test/notes", Requires: []string{"chunk"}, Provides: []string{"candidate"}},
|
||||
ModuleSpec{Key: "appendorder", Stage: StageMerge, ArtifactKind: "test/notes", Requires: []string{"candidate"}, Provides: []string{"merged"}},
|
||||
ModuleSpec{Key: "noop", Stage: StageNormalize, ArtifactKind: "test/notes", Requires: []string{"merged"}, Provides: []string{"normalized"}},
|
||||
ModuleSpec{Key: "grounded", Stage: StageValidate, Requires: []string{"normalized"}, Provides: []string{"validated"}},
|
||||
ModuleSpec{Key: "json", Stage: StageOutput, Requires: []string{"normalized"}, Provides: []string{"encoded"}},
|
||||
}
|
||||
@@ -1240,30 +1248,40 @@ func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSp
|
||||
t.Helper()
|
||||
|
||||
for _, spec := range specs {
|
||||
if spec.ArtifactKind == "" && (spec.Stage == StageExtract || spec.Stage == StageMerge || spec.Stage == StageNormalize) {
|
||||
spec.ArtifactKind = "test/notes"
|
||||
}
|
||||
switch spec.Stage {
|
||||
case StageInput:
|
||||
if err := catalog.Inputs.RegisterWithSpec(spec, profileInputConstructor(spec.Key)); err != nil {
|
||||
t.Fatalf("register input spec %#v: %v", spec, err)
|
||||
}
|
||||
case StageChunk:
|
||||
if err := catalog.Chunkers.RegisterWithSpec(spec, profileChunkerConstructor(spec.Key)); err != nil {
|
||||
validateOptions := func(options map[string]any) error { return RejectUnknownOptions(options, "a", "b", "size") }
|
||||
if err := catalog.Chunkers.RegisterBuilderWithSpec(spec, validateOptions, func(BuildRequest) (contracts.Chunker, error) { return &typedTestChunker{key: spec.Key}, nil }); err != nil {
|
||||
t.Fatalf("register chunk spec %#v: %v", spec, err)
|
||||
}
|
||||
case StageExtract:
|
||||
if err := catalog.Extractors.RegisterWithSpec(spec, profileExtractorConstructor(spec.Key)); err != nil {
|
||||
if err := RegisterExtractor(catalog.Extractors, spec, func() (contracts.Extractor[codecNotes], error) {
|
||||
return typedTestExtractor[codecNotes]{key: spec.Key}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register extractor spec %#v: %v", spec, err)
|
||||
}
|
||||
case StageMerge:
|
||||
if err := catalog.Mergers.RegisterWithSpec(spec, profileMergerConstructor(spec.Key)); err != nil {
|
||||
if err := RegisterMerger(catalog.Mergers, spec, func() (contracts.Merger[codecNotes], error) { return typedTestMerger[codecNotes]{key: spec.Key}, nil }); err != nil {
|
||||
t.Fatalf("register merger spec %#v: %v", spec, err)
|
||||
}
|
||||
case StageNormalize:
|
||||
if err := catalog.Normalizers.RegisterWithSpec(spec, profileNormalizerConstructor(spec.Key)); err != nil {
|
||||
if err := RegisterNormalizer(catalog.Normalizers, spec, func() (contracts.Normalizer[codecNotes], error) {
|
||||
return typedTestNormalizer[codecNotes]{key: spec.Key}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register normalizer spec %#v: %v", spec, err)
|
||||
}
|
||||
case StageValidate:
|
||||
validatorSpec := ValidatorSpec{Key: spec.Key, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||
if err := catalog.Validators.RegisterWithSpec(validatorSpec, profileValidatorConstructor(spec.Key)); err != nil {
|
||||
if err := RegisterTypedValidator(catalog.Validators, "test/notes", validatorSpec, func() (contracts.TypedValidator[codecNotes], error) {
|
||||
return typedTestValidator[codecNotes]{key: spec.Key}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register validator spec %#v: %v", spec, err)
|
||||
}
|
||||
case StageOutput:
|
||||
@@ -1278,7 +1296,9 @@ func registerProfileSpecs(t *testing.T, catalog ModuleCatalog, specs ...ModuleSp
|
||||
|
||||
func registerProfileValidatorSpec(t *testing.T, catalog ModuleCatalog, spec ValidatorSpec) {
|
||||
t.Helper()
|
||||
if err := catalog.Validators.RegisterWithSpec(spec, profileValidatorConstructor(spec.Key)); err != nil {
|
||||
if err := RegisterTypedValidator(catalog.Validators, "test/notes", spec, func() (contracts.TypedValidator[codecNotes], error) {
|
||||
return typedTestValidator[codecNotes]{key: spec.Key}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register validator spec %#v: %v", spec, err)
|
||||
}
|
||||
}
|
||||
@@ -1301,36 +1321,6 @@ func (adapter profileInputAdapter) Parse(ctx context.Context, req contracts.Pars
|
||||
return &source.SourceDocument{}, nil
|
||||
}
|
||||
|
||||
func profileChunkerConstructor(key string) ChunkerConstructor {
|
||||
return func() (contracts.Chunker, error) {
|
||||
return registryChunker{key: key}, nil
|
||||
}
|
||||
}
|
||||
|
||||
func profileExtractorConstructor(key string) ExtractorConstructor {
|
||||
return func() (contracts.Extractor, error) {
|
||||
return registryFakeExtractor{key: key}, nil
|
||||
}
|
||||
}
|
||||
|
||||
func profileMergerConstructor(key string) MergerConstructor {
|
||||
return func() (contracts.Merger, error) {
|
||||
return registryMerger{key: key}, nil
|
||||
}
|
||||
}
|
||||
|
||||
func profileNormalizerConstructor(key string) NormalizerConstructor {
|
||||
return func() (contracts.Normalizer, error) {
|
||||
return registryNormalizer{key: key}, nil
|
||||
}
|
||||
}
|
||||
|
||||
func profileValidatorConstructor(key string) ValidatorConstructor {
|
||||
return func() (contracts.Validator, error) {
|
||||
return registryValidator{name: key}, nil
|
||||
}
|
||||
}
|
||||
|
||||
func profileOutputConstructor(key string) OutputEncoderConstructor {
|
||||
return func() (contracts.OutputEncoder, error) {
|
||||
return registryOutputEncoder{key: key}, nil
|
||||
|
||||
14
internal/framework/pipeline/reference_test_helpers_test.go
Normal file
14
internal/framework/pipeline/reference_test_helpers_test.go
Normal file
@@ -0,0 +1,14 @@
|
||||
package pipeline
|
||||
|
||||
import "gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
|
||||
func validSourceDocument() *source.SourceDocument {
|
||||
return &source.SourceDocument{
|
||||
ID: "source-1", Kind: "document", Format: "text/plain", Digest: "sha256:source",
|
||||
Units: []source.SourceUnit{
|
||||
{ID: 1, Kind: "unit", Text: "Source unit.", Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 1, EndUnitID: 1}},
|
||||
{ID: 2, Kind: "unit", Text: "Second source unit.", Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 2, EndUnitID: 2}},
|
||||
{ID: 3, Kind: "unit", Text: "Third source unit.", Ref: source.SourceRef{SourceID: "source-1", StartUnitID: 3, EndUnitID: 3}},
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -1,261 +0,0 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func TestRunnerUsesRegistries(t *testing.T) {
|
||||
var built []string
|
||||
var executed []string
|
||||
registries := integrationRegistries(t, &built, &executed)
|
||||
|
||||
output, err := New(registries).Run(context.Background(), RunInput{
|
||||
Pipeline: integrationPipeline(),
|
||||
SourceID: "source-1",
|
||||
RawInput: []byte("source text"),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
wantBuilt := []string{"input", "chunk", "extract-first", "merge", "normalize", "extract-second", "merge", "normalize", "output"}
|
||||
if !reflect.DeepEqual(built, wantBuilt) {
|
||||
t.Fatalf("built = %#v, want %#v", built, wantBuilt)
|
||||
}
|
||||
if !reflect.DeepEqual(executed, []string{"extract-first:chunk-0", "extract-second:chunk-0"}) {
|
||||
t.Fatalf("executed = %#v, want extractor chunk execution", executed)
|
||||
}
|
||||
if got := normalizeOutputKeys(output.NormalizeOutputs); !reflect.DeepEqual(got, []string{"normalize", "normalize"}) {
|
||||
t.Fatalf("normalize output keys = %#v, want one output from each lane", got)
|
||||
}
|
||||
if len(output.Rejected) != 0 {
|
||||
t.Fatalf("len(Rejected) = %d, want none", len(output.Rejected))
|
||||
}
|
||||
}
|
||||
|
||||
func integrationRegistries(t *testing.T, built, executed *[]string) Registries {
|
||||
t.Helper()
|
||||
|
||||
registries := Registries{
|
||||
Inputs: NewInputAdapterRegistry(),
|
||||
Chunkers: NewChunkerRegistry(),
|
||||
Extractors: NewExtractorRegistry(),
|
||||
Mergers: NewMergerRegistry(),
|
||||
Normalizers: NewNormalizerRegistry(),
|
||||
Outputs: NewOutputEncoderRegistry(),
|
||||
}
|
||||
if err := registries.Inputs.Register("input", func() (contracts.InputAdapter, error) {
|
||||
*built = append(*built, "input")
|
||||
return integrationInput{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register input: %v", err)
|
||||
}
|
||||
if err := registries.Chunkers.Register("chunk", func() (contracts.Chunker, error) {
|
||||
*built = append(*built, "chunk")
|
||||
return integrationChunker{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register chunker: %v", err)
|
||||
}
|
||||
registerIntegrationExtractor(t, registries.Extractors, "extract-first", built, executed)
|
||||
registerIntegrationExtractor(t, registries.Extractors, "extract-second", built, executed)
|
||||
if err := registries.Mergers.Register("merge", func() (contracts.Merger, error) {
|
||||
*built = append(*built, "merge")
|
||||
return integrationMerger{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register merger: %v", err)
|
||||
}
|
||||
if err := registries.Normalizers.Register("normalize", func() (contracts.Normalizer, error) {
|
||||
*built = append(*built, "normalize")
|
||||
return integrationNormalizer{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register normalizer: %v", err)
|
||||
}
|
||||
if err := registries.Outputs.Register("output", func() (contracts.OutputEncoder, error) {
|
||||
*built = append(*built, "output")
|
||||
return integrationOutput{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register output: %v", err)
|
||||
}
|
||||
return registries
|
||||
}
|
||||
|
||||
func registerIntegrationExtractor(t *testing.T, registry *ExtractorRegistry, key string, built, executed *[]string) {
|
||||
t.Helper()
|
||||
|
||||
if err := registry.Register(key, func() (contracts.Extractor, error) {
|
||||
*built = append(*built, key)
|
||||
return integrationExtractor{key: key, executed: executed}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("Register(%q) error = %v, want nil", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
type integrationInput struct{}
|
||||
|
||||
func (input integrationInput) Key() string {
|
||||
return "input"
|
||||
}
|
||||
|
||||
func (input integrationInput) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
|
||||
return integrationSourceDocument(), nil
|
||||
}
|
||||
|
||||
type integrationChunker struct{}
|
||||
|
||||
func (chunker integrationChunker) Key() string {
|
||||
return "chunk"
|
||||
}
|
||||
|
||||
func (chunker integrationChunker) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (chunker integrationChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
|
||||
return contracts.ChunkResult{
|
||||
Chunks: []contracts.SourceChunk{
|
||||
{
|
||||
ID: "chunk-0",
|
||||
SourceID: req.Source.ID,
|
||||
Index: 0,
|
||||
StartUnitID: req.Source.Units[0].ID,
|
||||
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
|
||||
Content: []byte(`{"units":[1]}`),
|
||||
MediaType: "application/json",
|
||||
Units: req.Source.Units,
|
||||
},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type integrationExtractor struct {
|
||||
key string
|
||||
executed *[]string
|
||||
}
|
||||
|
||||
func (extractor integrationExtractor) Key() string {
|
||||
return extractor.key
|
||||
}
|
||||
|
||||
func (extractor integrationExtractor) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (extractor integrationExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
|
||||
*extractor.executed = append(*extractor.executed, extractor.key+":"+req.Chunk.ID)
|
||||
return contracts.ExtractionResult{
|
||||
Output: contracts.ExtractOutput{
|
||||
Schema: contracts.ResponseSchema{ID: "integration", Name: "integration", Version: "v1"},
|
||||
Payload: contracts.RawPayload{
|
||||
Content: []byte(`{"value":true}`),
|
||||
MediaType: "application/json",
|
||||
},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type integrationNormalizer struct{}
|
||||
|
||||
type integrationMerger struct{}
|
||||
|
||||
func (merger integrationMerger) Key() string {
|
||||
return "merge"
|
||||
}
|
||||
|
||||
func (merger integrationMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
|
||||
output := contracts.MergeOutput{
|
||||
LaneID: req.LaneID,
|
||||
Schema: contracts.ResponseSchema{ID: "integration", Name: "integration", Version: "v1"},
|
||||
Payload: contracts.RawPayload{
|
||||
Content: []byte(`{"merged":true}`),
|
||||
MediaType: "application/json",
|
||||
},
|
||||
}
|
||||
if len(req.ExtractOutputs) > 0 {
|
||||
output.SourceID = req.ExtractOutputs[0].SourceID
|
||||
output.Schema = req.ExtractOutputs[0].Schema
|
||||
output.Payload = req.ExtractOutputs[0].Payload
|
||||
}
|
||||
return contracts.MergeResult{Output: output}, nil
|
||||
}
|
||||
|
||||
func (normalizer integrationNormalizer) Key() string {
|
||||
return "normalize"
|
||||
}
|
||||
|
||||
func (normalizer integrationNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (normalizer integrationNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
|
||||
return contracts.NormalizeResult{
|
||||
Output: contracts.NormalizeOutput{
|
||||
LaneID: req.LaneID,
|
||||
SourceID: req.MergeOutput.SourceID,
|
||||
Schema: req.MergeOutput.Schema,
|
||||
Payload: req.MergeOutput.Payload,
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type integrationOutput struct{}
|
||||
|
||||
func (output integrationOutput) Key() string {
|
||||
return "output"
|
||||
}
|
||||
|
||||
func (output integrationOutput) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
|
||||
return contracts.OutputResult{
|
||||
Files: []contracts.OutputFile{
|
||||
{Name: "output.json", ContentType: "application/json", Bytes: []byte(`{}`)},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func integrationPipeline() ResolvedPipeline {
|
||||
return ResolvedPipeline{
|
||||
ID: "pipeline-1",
|
||||
Digest: "sha256:pipeline",
|
||||
Input: Binding("input"),
|
||||
Chunk: Binding("chunk"),
|
||||
ArtifactLanes: []ResolvedArtifactLane{
|
||||
{
|
||||
ID: "first",
|
||||
Extract: Binding("extract-first"),
|
||||
Merge: Binding("merge"),
|
||||
Normalize: Binding("normalize"),
|
||||
},
|
||||
{
|
||||
ID: "second",
|
||||
Extract: Binding("extract-second"),
|
||||
Merge: Binding("merge"),
|
||||
Normalize: Binding("normalize"),
|
||||
},
|
||||
},
|
||||
Output: Binding("output"),
|
||||
}
|
||||
}
|
||||
|
||||
func integrationSourceDocument() *source.SourceDocument {
|
||||
return &source.SourceDocument{
|
||||
ID: "source-1",
|
||||
Kind: "document",
|
||||
Format: "text/plain",
|
||||
Digest: "sha256:abc123",
|
||||
Units: []source.SourceUnit{
|
||||
{ID: 1, Kind: "unit", Text: "Source unit."},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func normalizeOutputKeys(outputs []contracts.NormalizeOutput) []string {
|
||||
keys := make([]string, 0, len(outputs))
|
||||
for _, output := range outputs {
|
||||
keys = append(keys, output.NormalizerKey)
|
||||
}
|
||||
return keys
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
408
internal/framework/pipeline/runner_concurrency_test.go
Normal file
408
internal/framework/pipeline/runner_concurrency_test.go
Normal file
@@ -0,0 +1,408 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
frameworkllm "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
)
|
||||
|
||||
type countingProvider struct {
|
||||
active atomic.Int32
|
||||
maximum atomic.Int32
|
||||
calls atomic.Int32
|
||||
}
|
||||
|
||||
func (p *countingProvider) CompleteStructured(ctx context.Context, _ contracts.StructuredCompletionRequest, _ any) (contracts.StructuredCompletionResponse, error) {
|
||||
p.calls.Add(1)
|
||||
current := p.active.Add(1)
|
||||
defer p.active.Add(-1)
|
||||
for {
|
||||
seen := p.maximum.Load()
|
||||
if current <= seen || p.maximum.CompareAndSwap(seen, current) {
|
||||
break
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-time.After(5 * time.Millisecond):
|
||||
return contracts.StructuredCompletionResponse{}, nil
|
||||
case <-ctx.Done():
|
||||
return contracts.StructuredCompletionResponse{}, ctx.Err()
|
||||
}
|
||||
}
|
||||
|
||||
func preparedConcurrentPipeline(t *testing.T, chunkCount int) *PreparedPipeline {
|
||||
t.Helper()
|
||||
catalog := typedResolutionCatalog(t, completeTypedCatalogOptions())
|
||||
resolved, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, catalog)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v", err)
|
||||
}
|
||||
prepared, err := Prepare(resolved, registriesFromModuleCatalog(catalog), ModuleDependencies{})
|
||||
if err != nil {
|
||||
t.Fatalf("Prepare() error = %v", err)
|
||||
}
|
||||
doc := typedTestDocument()
|
||||
chunks := make([]source.Chunk, chunkCount)
|
||||
for i := range chunks {
|
||||
chunks[i] = source.Chunk{ID: fmt.Sprintf("chunk-%d", i+1), SourceID: doc.ID, Index: i, Ref: doc.Units[0].Ref, Content: []byte(fmt.Sprintf(`{"chunk":%d}`, i+1)), MediaType: "application/json", Units: []source.SourceUnit{doc.Units[0]}}
|
||||
}
|
||||
prepared.chunker = &typedTestChunker{key: "typed/chunk", chunks: chunks}
|
||||
return prepared
|
||||
}
|
||||
|
||||
func installExtractOperation(prepared *PreparedPipeline, laneIndex int, operation func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error)) {
|
||||
prepared.lanes[laneIndex].typed.extract = func(ctx context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return operation(ctx, request)
|
||||
}
|
||||
}
|
||||
|
||||
func typedValueForLane(laneIndex int, chunkIndex int) any {
|
||||
if laneIndex == 0 {
|
||||
return codecNotes{Items: []string{fmt.Sprintf("chunk-%d", chunkIndex)}}
|
||||
}
|
||||
return codecScore{Value: chunkIndex}
|
||||
}
|
||||
|
||||
func TestRunnerBoundsExtractJobsAndStabilizesReverseCompletion(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 3)
|
||||
started := make(chan int, 6)
|
||||
release := make([]chan struct{}, 6)
|
||||
for i := range release {
|
||||
release[i] = make(chan struct{})
|
||||
}
|
||||
var active atomic.Int32
|
||||
var maximum atomic.Int32
|
||||
for laneIndex := range prepared.lanes {
|
||||
lane := laneIndex
|
||||
installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
jobIndex := request.Chunk.Index*2 + lane
|
||||
current := active.Add(1)
|
||||
defer active.Add(-1)
|
||||
for {
|
||||
seen := maximum.Load()
|
||||
if current <= seen || maximum.CompareAndSwap(seen, current) {
|
||||
break
|
||||
}
|
||||
}
|
||||
started <- jobIndex
|
||||
select {
|
||||
case <-release[jobIndex]:
|
||||
case <-ctx.Done():
|
||||
return erasedTypedResult{}, ctx.Err()
|
||||
}
|
||||
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index), Warnings: []contracts.Warning{{Scope: fmt.Sprintf("lane-%d/chunk-%d", lane, request.Chunk.Index), ReasonCode: "observed", Message: "ordered"}}}, nil
|
||||
})
|
||||
}
|
||||
|
||||
type outcome struct {
|
||||
output RunOutput
|
||||
err error
|
||||
}
|
||||
done := make(chan outcome, 1)
|
||||
go func() {
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||
done <- outcome{output: output, err: err}
|
||||
}()
|
||||
for pair := 0; pair < 3; pair++ {
|
||||
first, second := <-started, <-started
|
||||
want := map[int]bool{pair * 2: true, pair*2 + 1: true}
|
||||
if !want[first] || !want[second] || first == second {
|
||||
t.Fatalf("started jobs = %d, %d; want round-robin pair %d", first, second, pair)
|
||||
}
|
||||
close(release[second])
|
||||
close(release[first])
|
||||
}
|
||||
result := <-done
|
||||
if result.err != nil {
|
||||
t.Fatalf("Run() error = %v", result.err)
|
||||
}
|
||||
if got := maximum.Load(); got != 2 {
|
||||
t.Fatalf("maximum concurrent extract jobs = %d, want 2", got)
|
||||
}
|
||||
wantScopes := []string{"lane-0/chunk-0", "lane-0/chunk-1", "lane-0/chunk-2", "lane-1/chunk-0", "lane-1/chunk-1", "lane-1/chunk-2"}
|
||||
gotScopes := make([]string, len(result.output.Warnings))
|
||||
for i := range result.output.Warnings {
|
||||
gotScopes[i] = result.output.Warnings[i].Scope
|
||||
}
|
||||
if !reflect.DeepEqual(gotScopes, wantScopes) {
|
||||
t.Fatalf("warning order = %#v, want %#v", gotScopes, wantScopes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerStartsLaneContinuationWhileOtherLaneExtractsRemain(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 2)
|
||||
releaseOtherLane := make(chan struct{})
|
||||
mergeStarted := make(chan struct{}, 1)
|
||||
installExtractOperation(prepared, 0, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: typedValueForLane(0, request.Chunk.Index)}, nil
|
||||
})
|
||||
installExtractOperation(prepared, 1, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
select {
|
||||
case <-releaseOtherLane:
|
||||
return erasedTypedResult{Value: typedValueForLane(1, request.Chunk.Index)}, nil
|
||||
case <-ctx.Done():
|
||||
return erasedTypedResult{}, ctx.Err()
|
||||
}
|
||||
})
|
||||
originalMerge := prepared.lanes[0].typed.merge
|
||||
prepared.lanes[0].typed.merge = func(ctx context.Context, implementation any, request contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
select {
|
||||
case mergeStarted <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
return originalMerge(ctx, implementation, request)
|
||||
}
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
_, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||
done <- err
|
||||
}()
|
||||
select {
|
||||
case <-mergeStarted:
|
||||
close(releaseOtherLane)
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("first lane merge did not start while the second lane remained in extract")
|
||||
}
|
||||
if err := <-done; err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerBoundsConcurrentLaneContinuations(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 1)
|
||||
base := prepared.lanes[0]
|
||||
lanes := make([]preparedLaneExecutor, 8)
|
||||
resolvedLanes := make([]ResolvedArtifactLane, len(lanes))
|
||||
publicLanes := make([]PreparedArtifactLane, len(lanes))
|
||||
var active atomic.Int32
|
||||
var maximum atomic.Int32
|
||||
for i := range lanes {
|
||||
lane := base
|
||||
lane.resolved.ID = fmt.Sprintf("lane-%02d", i)
|
||||
lane.typed = &preparedTypedLane{
|
||||
extractor: base.typed.extractor,
|
||||
merger: base.typed.merger,
|
||||
normalizer: base.typed.normalizer,
|
||||
extract: func(_ context.Context, _ any, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{request.Chunk.ID}}}, nil
|
||||
},
|
||||
merge: func(_ context.Context, _ any, _ contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
current := active.Add(1)
|
||||
defer active.Add(-1)
|
||||
for {
|
||||
seen := maximum.Load()
|
||||
if current <= seen || maximum.CompareAndSwap(seen, current) {
|
||||
break
|
||||
}
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
return erasedTypedResult{Value: codecNotes{}}, nil
|
||||
},
|
||||
normalize: base.typed.normalize,
|
||||
codec: base.typed.codec,
|
||||
}
|
||||
lanes[i] = lane
|
||||
resolvedLanes[i] = lane.resolved
|
||||
publicLanes[i] = PreparedArtifactLane{Resolved: lane.resolved}
|
||||
}
|
||||
prepared.lanes = lanes
|
||||
prepared.resolved.ArtifactLanes = resolvedLanes
|
||||
prepared.ArtifactLanes = publicLanes
|
||||
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(output.NormalizeOutputs) != len(lanes) {
|
||||
t.Fatalf("normalize outputs = %d, want %d", len(output.NormalizeOutputs), len(lanes))
|
||||
}
|
||||
if got := maximum.Load(); got != 2 {
|
||||
t.Fatalf("maximum concurrent lane continuations = %d, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerSelectsFrameworkErrorByStableLaneOrder(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 1)
|
||||
ready := make(chan struct{}, 2)
|
||||
release := make(chan struct{})
|
||||
for laneIndex := range prepared.lanes {
|
||||
lane := laneIndex
|
||||
installExtractOperation(prepared, lane, func(context.Context, contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
ready <- struct{}{}
|
||||
<-release
|
||||
return erasedTypedResult{}, fmt.Errorf("lane-%d failure", lane)
|
||||
})
|
||||
}
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
_, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||
done <- err
|
||||
}()
|
||||
<-ready
|
||||
<-ready
|
||||
close(release)
|
||||
if err := <-done; err == nil || !strings.Contains(err.Error(), "lane-0 failure") {
|
||||
t.Fatalf("Run() error = %v, want first resolved lane failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerSelectsMergeErrorBeforeEarlierLaneNormalizeError(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 1)
|
||||
for laneIndex := range prepared.lanes {
|
||||
lane := laneIndex
|
||||
installExtractOperation(prepared, lane, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil
|
||||
})
|
||||
}
|
||||
ready := make(chan struct{}, 2)
|
||||
release := make(chan struct{})
|
||||
prepared.lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
ready <- struct{}{}
|
||||
<-release
|
||||
return erasedTypedResult{}, errors.New("earlier lane normalize failure")
|
||||
}
|
||||
prepared.lanes[1].typed.merge = func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error) {
|
||||
ready <- struct{}{}
|
||||
<-release
|
||||
return erasedTypedResult{}, errors.New("later lane merge failure")
|
||||
}
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
_, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||
done <- err
|
||||
}()
|
||||
<-ready
|
||||
<-ready
|
||||
close(release)
|
||||
if err := <-done; err == nil || !strings.Contains(err.Error(), "later lane merge failure") {
|
||||
t.Fatalf("Run() error = %v, want merge failure before normalize failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerKeepsSuccessfulChunksWhenAnotherExtractIsRejected(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 2)
|
||||
for laneIndex := range prepared.lanes {
|
||||
lane := laneIndex
|
||||
installExtractOperation(prepared, lane, func(_ context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil
|
||||
})
|
||||
}
|
||||
validator := &prepared.lanes[0].extractValidators.validators[0]
|
||||
validator.typedValidate = func(_ context.Context, _ any, target typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
if target.chunk != nil && target.chunk.Index == 0 {
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "expected_rejection", Message: "rejected by test"}, nil
|
||||
}
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(output.Rejected) != 1 || output.Rejected[0].LaneID != prepared.lanes[0].resolved.ID || output.Rejected[0].ChunkIndex != 0 {
|
||||
t.Fatalf("rejections = %#v, want the first lane's first chunk", output.Rejected)
|
||||
}
|
||||
if len(output.NormalizeOutputs) != 2 {
|
||||
t.Fatalf("normalize outputs = %d, want both lanes to continue", len(output.NormalizeOutputs))
|
||||
}
|
||||
if output.Manifest.ValidationStatus != "rejected" {
|
||||
t.Fatalf("validation status = %q, want rejected", output.Manifest.ValidationStatus)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerSeparatelyBoundsExtractJobsAndSharedProviderCalls(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 3)
|
||||
provider := &countingProvider{}
|
||||
scheduler, err := frameworkllm.NewScheduler(2)
|
||||
if err != nil {
|
||||
t.Fatalf("NewScheduler() error = %v", err)
|
||||
}
|
||||
client := frameworkllm.NewScheduledClient(provider, scheduler)
|
||||
var jobActive atomic.Int32
|
||||
var jobMaximum atomic.Int32
|
||||
var attempts sync.Map
|
||||
for laneIndex := range prepared.lanes {
|
||||
lane := laneIndex
|
||||
prepared.lanes[lane].resolved.Extract.Retries = 1
|
||||
installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
current := jobActive.Add(1)
|
||||
defer jobActive.Add(-1)
|
||||
for {
|
||||
seen := jobMaximum.Load()
|
||||
if current <= seen || jobMaximum.CompareAndSwap(seen, current) {
|
||||
break
|
||||
}
|
||||
}
|
||||
if _, callErr := client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{StageName: "extract"}, nil); callErr != nil {
|
||||
return erasedTypedResult{}, callErr
|
||||
}
|
||||
key := fmt.Sprintf("%d/%d", lane, request.Chunk.Index)
|
||||
countValue, _ := attempts.LoadOrStore(key, &atomic.Int32{})
|
||||
count := countValue.(*atomic.Int32).Add(1)
|
||||
if lane == 0 && request.Chunk.Index == 0 && count == 1 {
|
||||
return erasedTypedResult{}, errors.New("retry extract")
|
||||
}
|
||||
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, nil
|
||||
})
|
||||
validator := &prepared.lanes[lane].extractValidators.validators[0]
|
||||
validator.typedValidate = func(ctx context.Context, _ any, _ typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
if _, callErr := client.CompleteStructured(ctx, contracts.StructuredCompletionRequest{StageName: "validate"}, nil); callErr != nil {
|
||||
return contracts.ValidationResult{}, callErr
|
||||
}
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}
|
||||
}
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 3})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(output.NormalizeOutputs) != 2 {
|
||||
t.Fatalf("normalize outputs = %d, want 2", len(output.NormalizeOutputs))
|
||||
}
|
||||
if got := jobMaximum.Load(); got != 3 {
|
||||
t.Fatalf("maximum extract jobs = %d, want 3", got)
|
||||
}
|
||||
if got := provider.maximum.Load(); got != 2 {
|
||||
t.Fatalf("maximum provider calls = %d, want 2", got)
|
||||
}
|
||||
if got := provider.calls.Load(); got != 13 {
|
||||
t.Fatalf("provider calls = %d, want 13 including retry and validators", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerReturnsParentCancellationAndStopsQueuedExtracts(t *testing.T) {
|
||||
prepared := preparedConcurrentPipeline(t, 4)
|
||||
started := make(chan struct{}, 8)
|
||||
for laneIndex := range prepared.lanes {
|
||||
lane := laneIndex
|
||||
installExtractOperation(prepared, lane, func(ctx context.Context, request contracts.TypedExtractionRequest) (erasedTypedResult, error) {
|
||||
started <- struct{}{}
|
||||
<-ctx.Done()
|
||||
return erasedTypedResult{Value: typedValueForLane(lane, request.Chunk.Index)}, ctx.Err()
|
||||
})
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
_, err := New().Run(ctx, RunInput{Prepared: prepared, RawInput: []byte("input"), ExtractWorkers: 2})
|
||||
done <- err
|
||||
}()
|
||||
<-started
|
||||
<-started
|
||||
cancel()
|
||||
if err := <-done; !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("Run() error = %v, want context.Canceled", err)
|
||||
}
|
||||
if got := len(started); got != 0 {
|
||||
t.Fatalf("additional started jobs = %d, want none after cancellation", got)
|
||||
}
|
||||
}
|
||||
399
internal/framework/pipeline/runner_concurrent.go
Normal file
399
internal/framework/pipeline/runner_concurrent.go
Normal file
@@ -0,0 +1,399 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type laneExtractState struct {
|
||||
index int
|
||||
prepared preparedLaneExecutor
|
||||
deps []CheckpointFingerprint
|
||||
decision CheckpointDecision
|
||||
values []erasedExtractArtifact
|
||||
serialized []CheckpointArtifact
|
||||
warnings []contracts.Warning
|
||||
rejected []contracts.RejectedOutput
|
||||
results map[int]extractJobResult
|
||||
remaining int
|
||||
failed bool
|
||||
}
|
||||
|
||||
type extractJob struct {
|
||||
lane *laneExtractState
|
||||
chunk source.Chunk
|
||||
}
|
||||
|
||||
type extractJobResult struct {
|
||||
laneIndex int
|
||||
chunkIndex int
|
||||
value erasedExtractArtifact
|
||||
serialized CheckpointArtifact
|
||||
warnings []contracts.Warning
|
||||
rejected *contracts.RejectedOutput
|
||||
err error
|
||||
}
|
||||
|
||||
type laneCompletion struct {
|
||||
index int
|
||||
output RunOutput
|
||||
err error
|
||||
}
|
||||
|
||||
type orderedRunError struct {
|
||||
stage int
|
||||
lane int
|
||||
chunk int
|
||||
err error
|
||||
}
|
||||
|
||||
type completedExtractLoader struct {
|
||||
CheckpointLoader
|
||||
laneID string
|
||||
checkpoint ExtractCheckpoint
|
||||
}
|
||||
|
||||
func (l completedExtractLoader) Extract(laneID, _ string, _ []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
|
||||
if laneID == l.laneID {
|
||||
return l.checkpoint, CheckpointDecision{Reused: true, Reason: "coordinated extract result"}
|
||||
}
|
||||
return ExtractCheckpoint{}, CheckpointDecision{Reason: "extract result unavailable"}
|
||||
}
|
||||
|
||||
func (r *Runner) runLanes(parent context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk) (RunOutput, error) {
|
||||
output := RunOutput{Manifest: manifestFromPipeline(input)}
|
||||
states := make([]*laneExtractState, len(input.Prepared.lanes))
|
||||
for i, prepared := range input.Prepared.lanes {
|
||||
if prepared.typed == nil {
|
||||
return output, fmt.Errorf("typed lane %q executor is not prepared", prepared.resolved.ID)
|
||||
}
|
||||
setTypedLaneManifestMetadata(&output, prepared.resolved.ID, prepared.typed.extractor, prepared.typed.merger, prepared.typed.normalizer)
|
||||
state, err := prepareLaneExtract(input, loader, doc, chunks, i, prepared)
|
||||
if err != nil {
|
||||
return output, err
|
||||
}
|
||||
if !state.decision.Reused {
|
||||
if err := checkpoints.ExtractRunning(prepared.resolved.ID, prepared.resolved.Extract.Module, state.deps); err != nil {
|
||||
return output, fmt.Errorf("write extract checkpoint for lane %q: %w", prepared.resolved.ID, err)
|
||||
}
|
||||
}
|
||||
states[i] = state
|
||||
}
|
||||
|
||||
workerCount := input.ExtractWorkers
|
||||
if workerCount < 1 {
|
||||
workerCount = 1
|
||||
}
|
||||
ctx, cancel := context.WithCancel(parent)
|
||||
defer cancel()
|
||||
jobs := make(chan extractJob, workerCount)
|
||||
results := make(chan extractJobResult, workerCount)
|
||||
completions := make(chan laneCompletion, len(states))
|
||||
continuations := make(chan *laneExtractState, workerCount)
|
||||
|
||||
var workers sync.WaitGroup
|
||||
for i := 0; i < workerCount; i++ {
|
||||
workers.Add(1)
|
||||
go func() {
|
||||
defer workers.Done()
|
||||
for job := range jobs {
|
||||
if ctx.Err() != nil {
|
||||
continue
|
||||
}
|
||||
result := r.runExtractJob(ctx, input, doc, sourceInput, sessionID, job)
|
||||
results <- result
|
||||
}
|
||||
}()
|
||||
}
|
||||
go func() {
|
||||
defer close(jobs)
|
||||
for chunkIndex := range chunks {
|
||||
for laneIndex := range states {
|
||||
state := states[laneIndex]
|
||||
if state.decision.Reused {
|
||||
continue
|
||||
}
|
||||
select {
|
||||
case jobs <- extractJob{lane: state, chunk: chunks[chunkIndex]}:
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
go func() { workers.Wait(); close(results) }()
|
||||
var continuationWorkers sync.WaitGroup
|
||||
for i := 0; i < workerCount; i++ {
|
||||
continuationWorkers.Add(1)
|
||||
go func() {
|
||||
defer continuationWorkers.Done()
|
||||
for state := range continuations {
|
||||
if err := ctx.Err(); err != nil {
|
||||
completions <- laneCompletion{index: state.index, err: err}
|
||||
continue
|
||||
}
|
||||
laneOutput, err := r.continueLane(ctx, input, checkpoints, loader, doc, sourceInput, sessionID, chunks, state)
|
||||
completions <- laneCompletion{index: state.index, output: laneOutput, err: err}
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
completedOutputs := make([]RunOutput, len(states))
|
||||
var runErrors []orderedRunError
|
||||
var pendingContinuations []*laneExtractState
|
||||
launched, completed := 0, 0
|
||||
for _, state := range states {
|
||||
if state.decision.Reused {
|
||||
pendingContinuations = append(pendingContinuations, state)
|
||||
}
|
||||
}
|
||||
|
||||
resultChannel := results
|
||||
for resultChannel != nil || len(pendingContinuations) > 0 || completed < launched {
|
||||
var continuationChannel chan<- *laneExtractState
|
||||
var nextContinuation *laneExtractState
|
||||
if len(pendingContinuations) > 0 && ctx.Err() == nil {
|
||||
continuationChannel = continuations
|
||||
nextContinuation = pendingContinuations[0]
|
||||
} else if ctx.Err() != nil {
|
||||
pendingContinuations = nil
|
||||
}
|
||||
select {
|
||||
case continuationChannel <- nextContinuation:
|
||||
pendingContinuations = pendingContinuations[1:]
|
||||
launched++
|
||||
case result, ok := <-resultChannel:
|
||||
if !ok {
|
||||
resultChannel = nil
|
||||
continue
|
||||
}
|
||||
state := states[result.laneIndex]
|
||||
state.remaining--
|
||||
if result.err != nil {
|
||||
state.failed = true
|
||||
runErrors = append(runErrors, orderedRunError{stage: 0, lane: state.index, chunk: result.chunkIndex, err: result.err})
|
||||
_ = checkpoints.ExtractFailed(state.prepared.resolved.ID, state.prepared.resolved.Extract.Module, state.deps, result.err)
|
||||
cancel()
|
||||
} else {
|
||||
state.results[result.chunkIndex] = result
|
||||
}
|
||||
if state.remaining == 0 && !state.failed && ctx.Err() == nil {
|
||||
if err := finalizeLaneExtract(checkpoints, state); err != nil {
|
||||
state.failed = true
|
||||
runErrors = append(runErrors, orderedRunError{stage: 0, lane: state.index, chunk: len(chunks), err: err})
|
||||
cancel()
|
||||
} else {
|
||||
pendingContinuations = append(pendingContinuations, state)
|
||||
}
|
||||
}
|
||||
case completion := <-completions:
|
||||
completed++
|
||||
completedOutputs[completion.index] = completion.output
|
||||
if completion.err != nil {
|
||||
runErrors = append(runErrors, classifyLaneError(completion.index, len(chunks), completion.err))
|
||||
cancel()
|
||||
}
|
||||
}
|
||||
}
|
||||
close(continuations)
|
||||
continuationWorkers.Wait()
|
||||
for i := range completedOutputs {
|
||||
mergeLaneOutput(&output, completedOutputs[i])
|
||||
}
|
||||
if err := selectRunError(parent, runErrors); err != nil {
|
||||
return output, err
|
||||
}
|
||||
return output, nil
|
||||
}
|
||||
|
||||
func prepareLaneExtract(input RunInput, loader CheckpointLoader, doc *source.SourceDocument, chunks []source.Chunk, index int, prepared preparedLaneExecutor) (*laneExtractState, error) {
|
||||
lane, typed := prepared.resolved, prepared.typed
|
||||
digest, err := joinedChunkDigest(chunks)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("digest chunks for lane %q: %w", lane.ID, err)
|
||||
}
|
||||
state := &laneExtractState{index: index, prepared: prepared, deps: digestFingerprints("chunks", digest), remaining: len(chunks), results: make(map[int]extractJobResult, len(chunks))}
|
||||
cp, decision := loadExtract(loader, lane.ID, lane.Extract.Module, state.deps)
|
||||
if decision.Reused {
|
||||
for _, stored := range cp.Outputs {
|
||||
if _, decodeErr := decodeCheckpointArtifact(typed.codec, stored); decodeErr != nil {
|
||||
decision = CheckpointDecision{Reason: "extract artifact checkpoint codec is incompatible: " + decodeErr.Error()}
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
state.decision = decision
|
||||
if decision.Reused {
|
||||
state.remaining = 0
|
||||
for _, stored := range cp.Outputs {
|
||||
value, decodeErr := decodeCheckpointArtifact(typed.codec, stored)
|
||||
if decodeErr != nil {
|
||||
return nil, fmt.Errorf("decode extract checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||
}
|
||||
stored = hydrateCheckpointArtifact(typed.codec, stored, value)
|
||||
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: stored.ChunkID, ChunkIndex: stored.ChunkIndex, ChunkRef: stored.ChunkRef, Value: value}
|
||||
if stored.ChunkIndex >= 0 && stored.ChunkIndex < len(chunks) && artifact.ChunkRef == (source.SourceRef{}) {
|
||||
artifact.ChunkRef = chunks[stored.ChunkIndex].Ref
|
||||
}
|
||||
state.values = append(state.values, artifact)
|
||||
state.serialized = append(state.serialized, cloneCheckpointArtifact(stored))
|
||||
}
|
||||
state.warnings, state.rejected = cloneWarnings(cp.Warnings), cloneRejectedOutputs(cp.Rejected)
|
||||
}
|
||||
return state, nil
|
||||
}
|
||||
|
||||
func (r *Runner) runExtractJob(ctx context.Context, input RunInput, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, job extractJob) extractJobResult {
|
||||
state, chunk := job.lane, job.chunk
|
||||
lane, typed := state.prepared.resolved, state.prepared.typed
|
||||
result := extractJobResult{laneIndex: state.index, chunkIndex: chunk.Index}
|
||||
var accepted erasedExtractArtifact
|
||||
var serialized CheckpointArtifact
|
||||
var acceptedWarnings []contracts.Warning
|
||||
ok, rejection, err := runWithRetry(ctx, lane.Extract.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
|
||||
started := time.Now().UTC()
|
||||
attemptPath := path.Join("extract", debugPathComponent(lane.ID), fmt.Sprintf("chunk-%06d", chunk.Index+1), fmt.Sprintf("attempt-%02d", attempt))
|
||||
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
|
||||
extracted, callErr := typed.extract(attemptCtx, typed.extractor, contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk, SourceInput: chunkInputMaterial(sourceInput, chunk), SessionID: sessionID, References: CloneReferenceSet(lane.ExtractReferences.ReferenceSet), LLMProfile: lane.Extract.LLMProfile, Metadata: cloneMetadata(input.Metadata)})
|
||||
if callErr != nil {
|
||||
_ = writeDebugTimed(input.Debug, attemptPath+".json", debugEnvelopeWithLLMCalls(debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started, Error: callErr.Error()}, llmScope))
|
||||
return false, nil, fmt.Errorf("extract lane %q chunk %q with extractor %q: %w", lane.ID, chunk.ID, lane.Extract.Module, callErr)
|
||||
}
|
||||
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: chunk.ID, ChunkIndex: chunk.Index, ChunkRef: chunk.Ref, Value: extracted.Value}
|
||||
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageExtract, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: lane.ExtractReferences.ReferenceSet, metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: extracted.Value}, state.prepared.extractValidators, attempt, input.Debug)
|
||||
if validateErr != nil || rejected != nil {
|
||||
return false, rejected, validateErr
|
||||
}
|
||||
stored, encodeErr := checkpointArtifact(typed.codec, artifact.LaneID, artifact.ExtractorKey, artifact.SourceID, artifact.Value)
|
||||
if encodeErr != nil {
|
||||
return false, nil, encodeErr
|
||||
}
|
||||
stored.ChunkID, stored.ChunkIndex, stored.ChunkRef = artifact.ChunkID, artifact.ChunkIndex, artifact.ChunkRef
|
||||
accepted, serialized = artifact, stored
|
||||
acceptedWarnings = append(cloneWarnings(extracted.Warnings), warnings...)
|
||||
if debugErr := writeDebugTimed(input.Debug, attemptPath+".json", debugEnvelopeWithLLMCalls(debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started, Payload: map[string]any{"output": debugCheckpointArtifact(stored), "warnings": debugWarningEnvelopes(acceptedWarnings)}}, llmScope)); debugErr != nil {
|
||||
return false, nil, debugErr
|
||||
}
|
||||
return true, nil, nil
|
||||
})
|
||||
result.err = err
|
||||
if err == nil && !ok {
|
||||
result.rejected = rejection
|
||||
return result
|
||||
}
|
||||
result.value, result.serialized, result.warnings = accepted, serialized, acceptedWarnings
|
||||
return result
|
||||
}
|
||||
|
||||
func finalizeLaneExtract(checkpoints CheckpointRecorder, state *laneExtractState) error {
|
||||
lane := state.prepared.resolved
|
||||
indexes := make([]int, 0, len(state.results))
|
||||
for index := range state.results {
|
||||
indexes = append(indexes, index)
|
||||
}
|
||||
sort.Ints(indexes)
|
||||
for _, index := range indexes {
|
||||
result := state.results[index]
|
||||
if result.rejected != nil {
|
||||
state.rejected = append(state.rejected, *result.rejected)
|
||||
continue
|
||||
}
|
||||
state.values = append(state.values, result.value)
|
||||
state.serialized = append(state.serialized, result.serialized)
|
||||
state.warnings = append(state.warnings, result.warnings...)
|
||||
}
|
||||
sort.SliceStable(state.values, func(i, j int) bool { return state.values[i].ChunkIndex < state.values[j].ChunkIndex })
|
||||
sort.SliceStable(state.serialized, func(i, j int) bool { return state.serialized[i].ChunkIndex < state.serialized[j].ChunkIndex })
|
||||
sort.SliceStable(state.rejected, func(i, j int) bool { return state.rejected[i].ChunkIndex < state.rejected[j].ChunkIndex })
|
||||
if !state.decision.Reused {
|
||||
if err := recordExtract(checkpoints, lane.ID, lane.Extract.Module, state.deps, state.serialized, state.rejected, state.warnings); err != nil {
|
||||
return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *Runner) continueLane(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk, state *laneExtractState) (RunOutput, error) {
|
||||
lane := state.prepared.resolved
|
||||
local := RunOutput{Manifest: manifestFromPipeline(input)}
|
||||
checkpoint := ExtractCheckpoint{Outputs: state.serialized, Rejected: state.rejected, Warnings: state.warnings}
|
||||
coordinatedLoader := completedExtractLoader{CheckpointLoader: loader, laneID: lane.ID, checkpoint: checkpoint}
|
||||
input.extractDecision = &state.decision
|
||||
err := r.runTypedLane(ctx, input, checkpoints, coordinatedLoader, doc, sourceInput, sessionID, chunks, state.prepared, &local)
|
||||
return local, err
|
||||
}
|
||||
|
||||
func classifyLaneError(lane, sentinel int, err error) orderedRunError {
|
||||
stage := 1
|
||||
var laneErr *laneRunError
|
||||
if errors.As(err, &laneErr) {
|
||||
switch laneErr.stage {
|
||||
case StageExtract:
|
||||
stage = 0
|
||||
case StageNormalize:
|
||||
stage = 2
|
||||
}
|
||||
} else if strings.Contains(strings.ToLower(err.Error()), "normalize") {
|
||||
stage = 2
|
||||
}
|
||||
return orderedRunError{stage: stage, lane: lane, chunk: sentinel, err: err}
|
||||
}
|
||||
|
||||
func selectRunError(parent context.Context, values []orderedRunError) error {
|
||||
if err := parent.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(values) == 0 {
|
||||
return nil
|
||||
}
|
||||
hasReal := false
|
||||
for _, value := range values {
|
||||
if !errors.Is(value.err, context.Canceled) && !errors.Is(value.err, context.DeadlineExceeded) {
|
||||
hasReal = true
|
||||
break
|
||||
}
|
||||
}
|
||||
filtered := values[:0]
|
||||
for _, value := range values {
|
||||
if hasReal && (errors.Is(value.err, context.Canceled) || errors.Is(value.err, context.DeadlineExceeded)) {
|
||||
continue
|
||||
}
|
||||
filtered = append(filtered, value)
|
||||
}
|
||||
sort.SliceStable(filtered, func(i, j int) bool {
|
||||
if filtered[i].stage != filtered[j].stage {
|
||||
return filtered[i].stage < filtered[j].stage
|
||||
}
|
||||
if filtered[i].lane != filtered[j].lane {
|
||||
return filtered[i].lane < filtered[j].lane
|
||||
}
|
||||
return filtered[i].chunk < filtered[j].chunk
|
||||
})
|
||||
return filtered[0].err
|
||||
}
|
||||
|
||||
func mergeLaneOutput(dst *RunOutput, src RunOutput) {
|
||||
if dst == nil {
|
||||
return
|
||||
}
|
||||
dst.NormalizeOutputs = append(dst.NormalizeOutputs, cloneSerializedOutputs(src.NormalizeOutputs)...)
|
||||
dst.Rejected = append(dst.Rejected, cloneRejectedOutputs(src.Rejected)...)
|
||||
dst.Warnings = append(dst.Warnings, cloneWarnings(src.Warnings)...)
|
||||
dst.CheckpointEvents = append(dst.CheckpointEvents, src.CheckpointEvents...)
|
||||
for i := range dst.Manifest.ArtifactLanes {
|
||||
for j := range src.Manifest.ArtifactLanes {
|
||||
if dst.Manifest.ArtifactLanes[i].ID == src.Manifest.ArtifactLanes[j].ID && src.Manifest.ArtifactLanes[j].Metadata != nil {
|
||||
dst.Manifest.ArtifactLanes[i].Metadata = cloneMetadata(src.Manifest.ArtifactLanes[j].Metadata)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
463
internal/framework/pipeline/runner_typed.go
Normal file
463
internal/framework/pipeline/runner_typed.go
Normal file
@@ -0,0 +1,463 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"path"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func loadExtract(loader CheckpointLoader, laneID, moduleKey string, deps []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
|
||||
return loader.Extract(laneID, moduleKey, deps)
|
||||
}
|
||||
func loadMerge(loader CheckpointLoader, laneID, moduleKey string, deps []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
|
||||
return loader.Merge(laneID, moduleKey, deps)
|
||||
}
|
||||
func loadNormalize(loader CheckpointLoader, laneID, moduleKey string, deps []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
|
||||
return loader.Normalize(laneID, moduleKey, deps)
|
||||
}
|
||||
func recordExtract(recorder CheckpointRecorder, laneID, moduleKey string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
||||
return recorder.ExtractSucceeded(laneID, moduleKey, deps, outputs, rejected, warnings)
|
||||
}
|
||||
func recordMerge(recorder CheckpointRecorder, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
return recorder.MergeSucceeded(laneID, moduleKey, deps, output, warnings)
|
||||
}
|
||||
func recordNormalize(recorder CheckpointRecorder, laneID, moduleKey string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
return recorder.NormalizeSucceeded(laneID, moduleKey, deps, output, warnings)
|
||||
}
|
||||
func cloneCheckpointArtifact(output CheckpointArtifact) CheckpointArtifact {
|
||||
output.Artifact = contracts.CloneSerializedArtifact(output.Artifact)
|
||||
return output
|
||||
}
|
||||
|
||||
func hydrateCheckpointArtifact(codec artifactCodecEntry, output CheckpointArtifact, value any) CheckpointArtifact {
|
||||
output.Artifact.Schema = contracts.CloneArtifactSchema(codec.spec.Schema)
|
||||
if codec.metadata != nil {
|
||||
output.Artifact.Metadata = cloneMetadata(codec.metadata(value))
|
||||
} else {
|
||||
output.Artifact.Metadata = nil
|
||||
}
|
||||
return output
|
||||
}
|
||||
func artifactCheckpointDigests(outputs []CheckpointArtifact) []CheckpointFingerprint {
|
||||
values := make([]CheckpointFingerprint, 0, len(outputs))
|
||||
for i, output := range outputs {
|
||||
sum := sha256.Sum256(output.Artifact.Content)
|
||||
values = append(values, CheckpointFingerprint{Name: fmt.Sprintf("artifact[%d]", i), Value: "sha256:" + hex.EncodeToString(sum[:])})
|
||||
}
|
||||
return normalizeCheckpointFingerprints(values)
|
||||
}
|
||||
func debugCheckpointArtifact(output CheckpointArtifact) map[string]any {
|
||||
artifact := output.Artifact
|
||||
schema := contracts.CloneArtifactSchema(artifact.Schema)
|
||||
digest := output.SchemaDigest
|
||||
if digest == "" {
|
||||
digest = contracts.DigestArtifactSchema(schema)
|
||||
}
|
||||
schema.JSONSchema = nil
|
||||
content := debugContentEnvelope(artifact.Content, artifact.MediaType, artifact.Metadata, nil)
|
||||
content.ContentDigest = debugContentDigest(artifact.Content)
|
||||
return map[string]any{"lane_id": output.LaneID, "module_key": output.ModuleKey, "source_id": output.SourceID, "chunk_id": output.ChunkID, "chunk_index": output.ChunkIndex, "chunk_ref": output.ChunkRef, "artifact_kind": artifact.Kind, "schema": schema, "schema_digest": digest, "content": content}
|
||||
}
|
||||
func debugCheckpointArtifacts(outputs []CheckpointArtifact) []map[string]any {
|
||||
if len(outputs) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]map[string]any, 0, len(outputs))
|
||||
for _, output := range outputs {
|
||||
out = append(out, debugCheckpointArtifact(output))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func serializeArtifact(codec artifactCodecEntry, value any, candidate bool) (contracts.SerializedArtifact, error) {
|
||||
encode := codec.encode
|
||||
if candidate {
|
||||
encode = codec.encodeCandidate
|
||||
}
|
||||
content, err := encode(value)
|
||||
if err != nil {
|
||||
return contracts.SerializedArtifact{}, err
|
||||
}
|
||||
schema := codec.spec.Schema
|
||||
metadata := map[string]any(nil)
|
||||
if codec.metadata != nil {
|
||||
metadata = codec.metadata(value)
|
||||
}
|
||||
return contracts.SerializedArtifact{Kind: codec.spec.Kind, Schema: contracts.CloneArtifactSchema(schema), MediaType: codec.spec.MediaType, Content: append([]byte(nil), content...), Metadata: cloneMetadata(metadata)}, nil
|
||||
}
|
||||
|
||||
func decodeCheckpointArtifact(codec artifactCodecEntry, artifact CheckpointArtifact) (any, error) {
|
||||
expectedDigest := contracts.DigestArtifactSchema(codec.spec.Schema)
|
||||
if artifact.Artifact.Kind != codec.spec.Kind {
|
||||
return nil, fmt.Errorf("artifact kind %q does not match codec %q", artifact.Artifact.Kind, codec.spec.Kind)
|
||||
}
|
||||
if artifact.Artifact.Schema.ID != codec.spec.Schema.ID || artifact.Artifact.Schema.Version != codec.spec.Schema.Version {
|
||||
return nil, fmt.Errorf("artifact schema %q version %q does not match codec schema %q version %q", artifact.Artifact.Schema.ID, artifact.Artifact.Schema.Version, codec.spec.Schema.ID, codec.spec.Schema.Version)
|
||||
}
|
||||
if artifact.SchemaDigest != expectedDigest {
|
||||
return nil, fmt.Errorf("artifact schema digest %q does not match codec schema digest %q", artifact.SchemaDigest, expectedDigest)
|
||||
}
|
||||
if artifact.Artifact.MediaType != codec.spec.MediaType {
|
||||
return nil, fmt.Errorf("artifact media type %q does not match codec media type %q", artifact.Artifact.MediaType, codec.spec.MediaType)
|
||||
}
|
||||
return codec.decode(append([]byte(nil), artifact.Artifact.Content...))
|
||||
}
|
||||
|
||||
func checkpointArtifact(codec artifactCodecEntry, laneID, moduleKey, sourceID string, value any) (CheckpointArtifact, error) {
|
||||
serialized, err := serializeArtifact(codec, value, false)
|
||||
if err != nil {
|
||||
return CheckpointArtifact{}, err
|
||||
}
|
||||
return CheckpointArtifact{LaneID: laneID, ModuleKey: moduleKey, SourceID: sourceID, Artifact: serialized, SchemaDigest: contracts.DigestArtifactSchema(serialized.Schema)}, nil
|
||||
}
|
||||
|
||||
type laneRunError struct {
|
||||
stage ModuleStage
|
||||
err error
|
||||
}
|
||||
|
||||
func (e *laneRunError) Error() string { return e.err.Error() }
|
||||
func (e *laneRunError) Unwrap() error { return e.err }
|
||||
|
||||
func (r *Runner) runTypedLane(ctx context.Context, input RunInput, checkpoints CheckpointRecorder, loader CheckpointLoader, doc *source.SourceDocument, sourceInput contracts.LLMInputMaterial, sessionID string, chunks []source.Chunk, prepared preparedLaneExecutor, output *RunOutput) (err error) {
|
||||
activeStage := StageExtract
|
||||
defer func() {
|
||||
if err != nil {
|
||||
err = &laneRunError{stage: activeStage, err: err}
|
||||
}
|
||||
}()
|
||||
lane, typed := prepared.resolved, prepared.typed
|
||||
if typed == nil {
|
||||
return fmt.Errorf("typed lane %q executor is not prepared", lane.ID)
|
||||
}
|
||||
setTypedLaneManifestMetadata(output, lane.ID, typed.extractor, typed.merger, typed.normalizer)
|
||||
|
||||
values := make([]erasedExtractArtifact, 0, len(chunks))
|
||||
serializedExtracts := make([]CheckpointArtifact, 0, len(chunks))
|
||||
extractWarnings := []contracts.Warning{}
|
||||
rejectedStart := len(output.Rejected)
|
||||
chunksDigest, err := joinedChunkDigest(chunks)
|
||||
if err != nil {
|
||||
return fmt.Errorf("digest chunks for lane %q: %w", lane.ID, err)
|
||||
}
|
||||
extractDeps := digestFingerprints("chunks", chunksDigest)
|
||||
cp, decision := loadExtract(loader, lane.ID, lane.Extract.Module, extractDeps)
|
||||
if decision.Reused {
|
||||
for _, stored := range cp.Outputs {
|
||||
if _, decodeErr := decodeCheckpointArtifact(typed.codec, stored); decodeErr != nil {
|
||||
decision = CheckpointDecision{Reason: "extract artifact checkpoint codec is incompatible: " + decodeErr.Error()}
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
reportedDecision := decision
|
||||
if input.extractDecision != nil {
|
||||
reportedDecision = *input.extractDecision
|
||||
}
|
||||
recordCheckpointEvent(output, loader, string(StageExtract), lane.ID, lane.Extract.Module, reportedDecision)
|
||||
if err := writeDebugTimed(input.Debug, path.Join("extract", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": reportedDecision.Reused, "decision": reportedDecision, "source": debugSourceDocumentEnvelope(doc), "chunks": debugSourceChunkEnvelopes(chunks), "options": redactSensitiveMap(lane.Extract.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
|
||||
return err
|
||||
}
|
||||
if decision.Reused {
|
||||
for _, stored := range cp.Outputs {
|
||||
value, decodeErr := decodeCheckpointArtifact(typed.codec, stored)
|
||||
if decodeErr != nil {
|
||||
return fmt.Errorf("decode extract checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||
}
|
||||
stored = hydrateCheckpointArtifact(typed.codec, stored, value)
|
||||
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: stored.ChunkID, ChunkIndex: stored.ChunkIndex, ChunkRef: stored.ChunkRef, Value: value}
|
||||
if stored.ChunkIndex >= 0 && stored.ChunkIndex < len(chunks) && artifact.ChunkRef == (source.SourceRef{}) {
|
||||
artifact.ChunkRef = chunks[stored.ChunkIndex].Ref
|
||||
}
|
||||
values = append(values, artifact)
|
||||
serializedExtracts = append(serializedExtracts, cloneCheckpointArtifact(stored))
|
||||
}
|
||||
extractWarnings = cloneWarnings(cp.Warnings)
|
||||
output.Warnings = append(output.Warnings, extractWarnings...)
|
||||
output.Rejected = append(output.Rejected, cloneRejectedOutputs(cp.Rejected)...)
|
||||
} else {
|
||||
if err := checkpoints.ExtractRunning(lane.ID, lane.Extract.Module, extractDeps); err != nil {
|
||||
return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err)
|
||||
}
|
||||
for i := range chunks {
|
||||
chunk := chunks[i]
|
||||
var accepted erasedExtractArtifact
|
||||
var serializedAccepted CheckpointArtifact
|
||||
var acceptedWarnings []contracts.Warning
|
||||
ok, rejection, runErr := runWithRetry(ctx, lane.Extract.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
|
||||
started := time.Now().UTC()
|
||||
attemptPath := path.Join("extract", debugPathComponent(lane.ID), fmt.Sprintf("chunk-%06d", chunk.Index+1), fmt.Sprintf("attempt-%02d", attempt))
|
||||
attemptCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
|
||||
result, callErr := typed.extract(attemptCtx, typed.extractor, contracts.TypedExtractionRequest{Source: doc, Chunk: &chunk, SourceInput: chunkInputMaterial(sourceInput, chunk), SessionID: sessionID, References: CloneReferenceSet(lane.ExtractReferences.ReferenceSet), LLMProfile: lane.Extract.LLMProfile, Metadata: cloneMetadata(input.Metadata)})
|
||||
if callErr != nil {
|
||||
_ = writeDebugTimed(input.Debug, attemptPath+".json", debugEnvelopeWithLLMCalls(debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started, Error: callErr.Error()}, llmScope))
|
||||
return false, nil, fmt.Errorf("extract lane %q chunk %q with extractor %q: %w", lane.ID, chunk.ID, lane.Extract.Module, callErr)
|
||||
}
|
||||
artifact := erasedExtractArtifact{LaneID: lane.ID, ExtractorKey: lane.Extract.Module, SourceID: doc.ID, ChunkID: chunk.ID, ChunkIndex: chunk.Index, ChunkRef: chunk.Ref, Value: result.Value}
|
||||
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageExtract, laneID: lane.ID, moduleKey: lane.Extract.Module, source: doc, sourceID: doc.ID, sourceInput: chunkInputMaterial(sourceInput, chunk), sessionID: sessionID, references: lane.ExtractReferences.ReferenceSet, metadata: input.Metadata, chunk: &chunk, ref: chunk.Ref, value: result.Value}, prepared.extractValidators, attempt, input.Debug)
|
||||
if validateErr != nil || rejected != nil {
|
||||
return false, rejected, validateErr
|
||||
}
|
||||
stored, encodeErr := checkpointArtifact(typed.codec, artifact.LaneID, artifact.ExtractorKey, artifact.SourceID, artifact.Value)
|
||||
if encodeErr != nil {
|
||||
return false, nil, encodeErr
|
||||
}
|
||||
stored.ChunkID, stored.ChunkIndex, stored.ChunkRef = artifact.ChunkID, artifact.ChunkIndex, artifact.ChunkRef
|
||||
accepted, serializedAccepted = artifact, stored
|
||||
acceptedWarnings = append(cloneWarnings(result.Warnings), warnings...)
|
||||
if debugErr := writeDebugTimed(input.Debug, attemptPath+".json", debugEnvelopeWithLLMCalls(debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, Attempt: attempt, StartedAt: started, Payload: map[string]any{"output": debugCheckpointArtifact(stored), "warnings": debugWarningEnvelopes(acceptedWarnings)}}, llmScope)); debugErr != nil {
|
||||
return false, nil, debugErr
|
||||
}
|
||||
return true, nil, nil
|
||||
})
|
||||
if runErr != nil {
|
||||
_ = checkpoints.ExtractFailed(lane.ID, lane.Extract.Module, extractDeps, runErr)
|
||||
return runErr
|
||||
}
|
||||
if !ok {
|
||||
output.Rejected = append(output.Rejected, *rejection)
|
||||
continue
|
||||
}
|
||||
values = append(values, accepted)
|
||||
serializedExtracts = append(serializedExtracts, serializedAccepted)
|
||||
extractWarnings = append(extractWarnings, acceptedWarnings...)
|
||||
output.Warnings = append(output.Warnings, acceptedWarnings...)
|
||||
}
|
||||
if err := recordExtract(checkpoints, lane.ID, lane.Extract.Module, extractDeps, serializedExtracts, cloneRejectedOutputs(output.Rejected[rejectedStart:]), extractWarnings); err != nil {
|
||||
return fmt.Errorf("write extract checkpoint for lane %q: %w", lane.ID, err)
|
||||
}
|
||||
}
|
||||
sort.SliceStable(values, func(i, j int) bool { return values[i].ChunkIndex < values[j].ChunkIndex })
|
||||
sort.SliceStable(serializedExtracts, func(i, j int) bool { return serializedExtracts[i].ChunkIndex < serializedExtracts[j].ChunkIndex })
|
||||
if err := writeDebugTimed(input.Debug, path.Join("extract", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageExtract), LaneID: lane.ID, ModuleKey: lane.Extract.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": reportedDecision.Reused, "outputs": debugCheckpointArtifacts(serializedExtracts), "rejected": debugRejectedOutputEnvelopes(output.Rejected[rejectedStart:]), "warnings": debugWarningEnvelopes(extractWarnings)}}); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(values) == 0 {
|
||||
return nil
|
||||
}
|
||||
activeStage = StageMerge
|
||||
|
||||
mergeInputs := make([]contracts.ExtractArtifact[any], len(values))
|
||||
for i, value := range values {
|
||||
mergeInputs[i] = contracts.ExtractArtifact[any]{LaneID: value.LaneID, ExtractorKey: value.ExtractorKey, SourceID: value.SourceID, ChunkID: value.ChunkID, ChunkIndex: value.ChunkIndex, ChunkRef: value.ChunkRef, Value: value.Value}
|
||||
}
|
||||
mergeDeps := artifactCheckpointDigests(serializedExtracts)
|
||||
mergeCP, mergeDecision := loadMerge(loader, lane.ID, lane.Merge.Module, mergeDeps)
|
||||
if mergeDecision.Reused {
|
||||
if _, decodeErr := decodeCheckpointArtifact(typed.codec, mergeCP.Output); decodeErr != nil {
|
||||
mergeDecision = CheckpointDecision{Reason: "merge artifact checkpoint codec is incompatible: " + decodeErr.Error()}
|
||||
}
|
||||
}
|
||||
recordCheckpointEvent(output, loader, string(StageMerge), lane.ID, lane.Merge.Module, mergeDecision)
|
||||
if err := writeDebugTimed(input.Debug, path.Join("merge", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageMerge), LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "decision": mergeDecision, "source": debugSourceDocumentEnvelope(doc), "extract_outputs": debugCheckpointArtifacts(serializedExtracts), "options": redactSensitiveMap(lane.Merge.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
|
||||
return err
|
||||
}
|
||||
var merged erasedMergeArtifact
|
||||
var serializedMerge CheckpointArtifact
|
||||
var mergeWarnings []contracts.Warning
|
||||
if mergeDecision.Reused {
|
||||
value, decodeErr := decodeCheckpointArtifact(typed.codec, mergeCP.Output)
|
||||
if decodeErr != nil {
|
||||
return fmt.Errorf("decode merge checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||
}
|
||||
merged = erasedMergeArtifact{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: value}
|
||||
serializedMerge = hydrateCheckpointArtifact(typed.codec, cloneCheckpointArtifact(mergeCP.Output), value)
|
||||
mergeWarnings = cloneWarnings(mergeCP.Warnings)
|
||||
output.Warnings = append(output.Warnings, mergeWarnings...)
|
||||
} else {
|
||||
if err := checkpoints.MergeRunning(lane.ID, lane.Merge.Module, mergeDeps); err != nil {
|
||||
return err
|
||||
}
|
||||
ok, rejection, runErr := runWithRetry(ctx, lane.Merge.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
|
||||
result, callErr := typed.merge(ctx, typed.merger, contracts.TypedMergeRequest[any]{Source: doc, LaneID: lane.ID, ExtractOutputs: mergeInputs, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(lane.MergeReferences.ReferenceSet), LLMProfile: lane.Merge.LLMProfile, Metadata: cloneMetadata(input.Metadata)})
|
||||
if callErr != nil {
|
||||
return false, nil, fmt.Errorf("merge lane %q with merger %q: %w", lane.ID, lane.Merge.Module, callErr)
|
||||
}
|
||||
candidate := erasedMergeArtifact{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: result.Value}
|
||||
warnings, rejected, validateErr := r.validateTypedArtifact(ctx, typed.codec, typedValidationTarget{stage: StageMerge, laneID: lane.ID, moduleKey: lane.Merge.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: lane.MergeReferences.ReferenceSet, metadata: input.Metadata, value: result.Value}, prepared.mergeValidators, attempt, input.Debug)
|
||||
if validateErr != nil || rejected != nil {
|
||||
return false, rejected, validateErr
|
||||
}
|
||||
stored, encodeErr := checkpointArtifact(typed.codec, candidate.LaneID, candidate.MergerKey, candidate.SourceID, candidate.Value)
|
||||
if encodeErr != nil {
|
||||
return false, nil, encodeErr
|
||||
}
|
||||
merged, serializedMerge = candidate, stored
|
||||
mergeWarnings = append(cloneWarnings(result.Warnings), warnings...)
|
||||
return true, nil, nil
|
||||
})
|
||||
if runErr != nil {
|
||||
_ = checkpoints.MergeFailed(lane.ID, lane.Merge.Module, mergeDeps, runErr)
|
||||
return runErr
|
||||
}
|
||||
if !ok {
|
||||
output.Rejected = append(output.Rejected, *rejection)
|
||||
if err := checkpoints.MergeRejected(lane.ID, lane.Merge.Module, mergeDeps, *rejection); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
output.Warnings = append(output.Warnings, mergeWarnings...)
|
||||
if err := recordMerge(checkpoints, lane.ID, lane.Merge.Module, mergeDeps, serializedMerge, mergeWarnings); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := writeDebugTimed(input.Debug, path.Join("merge", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageMerge), LaneID: lane.ID, ModuleKey: lane.Merge.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": mergeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedMerge), "warnings": debugWarningEnvelopes(mergeWarnings)}}); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
activeStage = StageNormalize
|
||||
normalizeDeps := artifactCheckpointDigests([]CheckpointArtifact{serializedMerge})
|
||||
normalizeCP, normalizeDecision := loadNormalize(loader, lane.ID, lane.Normalize.Module, normalizeDeps)
|
||||
if normalizeDecision.Reused {
|
||||
if _, decodeErr := decodeCheckpointArtifact(typed.codec, normalizeCP.Output); decodeErr != nil {
|
||||
normalizeDecision = CheckpointDecision{Reason: "normalize artifact checkpoint codec is incompatible: " + decodeErr.Error()}
|
||||
}
|
||||
}
|
||||
recordCheckpointEvent(output, loader, string(StageNormalize), lane.ID, lane.Normalize.Module, normalizeDecision)
|
||||
if err := writeDebugTimed(input.Debug, path.Join("normalize", debugPathComponent(lane.ID), "input.json"), debugTimedEnvelope{Stage: string(StageNormalize), LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "decision": normalizeDecision, "source": debugSourceDocumentEnvelope(doc), "merge_output": debugCheckpointArtifact(serializedMerge), "options": redactSensitiveMap(lane.Normalize.Options), "metadata": redactSensitiveMap(input.Metadata)}}); err != nil {
|
||||
return err
|
||||
}
|
||||
var serializedNormalize CheckpointArtifact
|
||||
var normalizeWarnings []contracts.Warning
|
||||
if normalizeDecision.Reused {
|
||||
value, decodeErr := decodeCheckpointArtifact(typed.codec, normalizeCP.Output)
|
||||
if decodeErr != nil {
|
||||
return fmt.Errorf("decode normalize checkpoint for lane %q: %w", lane.ID, decodeErr)
|
||||
}
|
||||
serializedNormalize, normalizeWarnings = hydrateCheckpointArtifact(typed.codec, cloneCheckpointArtifact(normalizeCP.Output), value), cloneWarnings(normalizeCP.Warnings)
|
||||
output.Warnings = append(output.Warnings, normalizeWarnings...)
|
||||
} else {
|
||||
if err := checkpoints.NormalizeRunning(lane.ID, lane.Normalize.Module, normalizeDeps); err != nil {
|
||||
return err
|
||||
}
|
||||
ok, rejection, runErr := runWithRetry(ctx, lane.Normalize.Retries, func(attempt int) (bool, *contracts.RejectedOutput, error) {
|
||||
result, callErr := typed.normalize(ctx, typed.normalizer, contracts.TypedNormalizeRequest[any]{Source: doc, LaneID: lane.ID, MergeOutput: contracts.MergeArtifact[any]{LaneID: lane.ID, MergerKey: lane.Merge.Module, SourceID: doc.ID, Value: merged.Value}, SourceInput: sourceInput.Clone(), SessionID: sessionID, References: CloneReferenceSet(lane.NormalizeReferences.ReferenceSet), LLMProfile: lane.Normalize.LLMProfile, Metadata: cloneMetadata(input.Metadata)})
|
||||
if callErr != nil {
|
||||
return false, nil, fmt.Errorf("normalize lane %q with normalizer %q: %w", lane.ID, lane.Normalize.Module, callErr)
|
||||
}
|
||||
warnings, rejected, validateErr := r.validateTypedArtifact(ctx, typed.codec, typedValidationTarget{stage: StageNormalize, laneID: lane.ID, moduleKey: lane.Normalize.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: lane.NormalizeReferences.ReferenceSet, metadata: input.Metadata, value: result.Value}, prepared.normalizeValidators, attempt, input.Debug)
|
||||
if validateErr != nil || rejected != nil {
|
||||
return false, rejected, validateErr
|
||||
}
|
||||
stored, encodeErr := checkpointArtifact(typed.codec, lane.ID, lane.Normalize.Module, doc.ID, result.Value)
|
||||
if encodeErr != nil {
|
||||
return false, nil, encodeErr
|
||||
}
|
||||
serializedNormalize = stored
|
||||
normalizeWarnings = append(cloneWarnings(result.Warnings), warnings...)
|
||||
return true, nil, nil
|
||||
})
|
||||
if runErr != nil {
|
||||
_ = checkpoints.NormalizeFailed(lane.ID, lane.Normalize.Module, normalizeDeps, runErr)
|
||||
return runErr
|
||||
}
|
||||
if !ok {
|
||||
output.Rejected = append(output.Rejected, *rejection)
|
||||
if err := checkpoints.NormalizeRejected(lane.ID, lane.Normalize.Module, normalizeDeps, *rejection); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
output.Warnings = append(output.Warnings, normalizeWarnings...)
|
||||
if err := recordNormalize(checkpoints, lane.ID, lane.Normalize.Module, normalizeDeps, serializedNormalize, normalizeWarnings); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := writeDebugTimed(input.Debug, path.Join("normalize", debugPathComponent(lane.ID), "output.json"), debugTimedEnvelope{Stage: string(StageNormalize), LaneID: lane.ID, ModuleKey: lane.Normalize.Module, StartedAt: time.Now().UTC(), Payload: map[string]any{"reused": normalizeDecision.Reused, "accepted": true, "output": debugCheckpointArtifact(serializedNormalize), "warnings": debugWarningEnvelopes(normalizeWarnings)}}); err != nil {
|
||||
return err
|
||||
}
|
||||
output.NormalizeOutputs = append(output.NormalizeOutputs, contracts.SerializedOutput{LaneID: lane.ID, NormalizerKey: lane.Normalize.Module, SourceID: doc.ID, Artifact: contracts.CloneSerializedArtifact(serializedNormalize.Artifact)})
|
||||
return nil
|
||||
}
|
||||
|
||||
func setTypedLaneManifestMetadata(output *RunOutput, laneID string, extractor, merger, normalizer any) {
|
||||
if output == nil {
|
||||
return
|
||||
}
|
||||
for i := range output.Manifest.ArtifactLanes {
|
||||
if output.Manifest.ArtifactLanes[i].ID != laneID {
|
||||
continue
|
||||
}
|
||||
metadata := make(map[string]any)
|
||||
for _, item := range []struct {
|
||||
name string
|
||||
module any
|
||||
}{{"extractor", extractor}, {"merger", merger}, {"normalizer", normalizer}} {
|
||||
if value, ok := moduleManifestMetadata(item.module); ok {
|
||||
metadata[item.name] = value
|
||||
}
|
||||
}
|
||||
if len(metadata) > 0 {
|
||||
output.Manifest.ArtifactLanes[i].Metadata = metadata
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Runner) validateTypedArtifact(ctx context.Context, codec artifactCodecEntry, target typedValidationTarget, chain preparedValidatorChain, attempt int, debug DebugRecorder) ([]contracts.Warning, *contracts.RejectedOutput, error) {
|
||||
var warnings []contracts.Warning
|
||||
for index, item := range chain.validators {
|
||||
binding := item.resolved.Binding
|
||||
var result contracts.ValidationResult
|
||||
var err error
|
||||
started := time.Now().UTC()
|
||||
attemptPath := path.Join("validate", debugPathComponent(string(target.stage)), debugPathComponent(target.laneID), debugPathComponent(target.moduleKey), fmt.Sprintf("%02d-%s-attempt-%02d", index+1, debugPathComponent(binding.Module), attempt))
|
||||
validatorCtx, llmScope := withDebugLLMScope(ctx, attemptPath)
|
||||
switch item.resolved.Target {
|
||||
case ValidatorTargetTyped:
|
||||
target.llmProfile = binding.LLMProfile
|
||||
result, err = item.typedValidate(validatorCtx, item.typed, target)
|
||||
case ValidatorTargetSerialized:
|
||||
artifact, encodeErr := serializeArtifact(codec, target.value, true)
|
||||
if encodeErr != nil {
|
||||
err = encodeErr
|
||||
break
|
||||
}
|
||||
result, err = item.serialized.Validate(validatorCtx, contracts.SerializedValidationRequest{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, Source: target.source, SourceID: target.sourceID, SourceInput: target.sourceInput.Clone(), SessionID: target.sessionID, References: CloneReferenceSet(target.references), LLMProfile: binding.LLMProfile, Metadata: cloneMetadata(target.metadata), Chunk: cloneSourceChunkPtr(target.chunk), Chunks: cloneSourceChunks(target.chunks), Schema: contracts.CloneArtifactSchema(artifact.Schema), MediaType: artifact.MediaType, Content: append([]byte(nil), artifact.Content...)})
|
||||
default:
|
||||
return nil, nil, fmt.Errorf("validator %q is incompatible with typed artifact validation", binding.Module)
|
||||
}
|
||||
artifact, _ := serializeArtifact(codec, target.value, true)
|
||||
debugCall := debugValidationCall{ValidatorName: binding.Module, Request: map[string]any{"stage": string(target.stage), "lane_id": target.laneID, "module_key": target.moduleKey, "source_id": target.sourceID, "artifact": debugCheckpointArtifact(CheckpointArtifact{Artifact: artifact, SchemaDigest: contracts.DigestArtifactSchema(artifact.Schema)}), "metadata": redactSensitiveMap(target.metadata)}, Result: debugValidationResultEnvelope(result)}
|
||||
if err != nil {
|
||||
debugCall.Error = err.Error()
|
||||
}
|
||||
if debugErr := writeDebugTimed(debug, attemptPath+".json", debugEnvelopeWithLLMCalls(debugTimedEnvelope{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, Attempt: attempt, StartedAt: started, Payload: debugCall, Error: debugCall.Error}, llmScope)); debugErr != nil {
|
||||
return nil, nil, fmt.Errorf("write validation debug artifact: %w", debugErr)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("validate typed %s output with validator %q: %w", target.stage, binding.Module, err)
|
||||
}
|
||||
if !result.Approved {
|
||||
reason := result.ReasonCode
|
||||
if reason == "" {
|
||||
reason = "artifact_rejected"
|
||||
}
|
||||
message := result.Message
|
||||
if message == "" {
|
||||
message = "artifact rejected"
|
||||
}
|
||||
return nil, &contracts.RejectedOutput{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, ChunkID: func() string {
|
||||
if target.chunk != nil {
|
||||
return target.chunk.ID
|
||||
}
|
||||
return ""
|
||||
}(), ChunkIndex: func() int {
|
||||
if target.chunk != nil {
|
||||
return target.chunk.Index
|
||||
}
|
||||
return 0
|
||||
}(), ValidatorName: binding.Module, ReasonCode: reason, Message: message, AttemptCount: attempt, DiagnosticArtifactPath: result.DiagnosticArtifactPath}, nil
|
||||
}
|
||||
warnings = append(warnings, result.Warnings...)
|
||||
}
|
||||
return warnings, nil, nil
|
||||
}
|
||||
45
internal/framework/pipeline/runner_typed_checkpoint_test.go
Normal file
45
internal/framework/pipeline/runner_typed_checkpoint_test.go
Normal file
@@ -0,0 +1,45 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func TestDecodeCheckpointArtifactRejectsIncompatibleCodecIdentityAndBytes(t *testing.T) {
|
||||
registry := NewArtifactCodecRegistry()
|
||||
if err := RegisterArtifactCodec(registry, notesCodec()); err != nil {
|
||||
t.Fatalf("RegisterArtifactCodec: %v", err)
|
||||
}
|
||||
codec, _, err := registry.entry("test/notes")
|
||||
if err != nil {
|
||||
t.Fatalf("entry: %v", err)
|
||||
}
|
||||
artifact, err := serializeArtifact(codec, codecNotes{Items: []string{"one"}}, false)
|
||||
if err != nil {
|
||||
t.Fatalf("serializeArtifact: %v", err)
|
||||
}
|
||||
base := CheckpointArtifact{Artifact: artifact, SchemaDigest: contracts.DigestArtifactSchema(artifact.Schema)}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
mutate func(*CheckpointArtifact)
|
||||
want string
|
||||
}{
|
||||
{name: "missing kind", mutate: func(v *CheckpointArtifact) { v.Artifact.Kind = "" }, want: "artifact kind"},
|
||||
{name: "schema version", mutate: func(v *CheckpointArtifact) { v.Artifact.Schema.Version = "v999" }, want: "does not match codec schema"},
|
||||
{name: "schema digest", mutate: func(v *CheckpointArtifact) { v.SchemaDigest = "sha256:different" }, want: "schema digest"},
|
||||
{name: "media type", mutate: func(v *CheckpointArtifact) { v.Artifact.MediaType = "text/plain" }, want: "media type"},
|
||||
{name: "decode failure", mutate: func(v *CheckpointArtifact) { v.Artifact.Content = []byte(`{"items":[`) }, want: "unexpected EOF"},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
stored := cloneCheckpointArtifact(base)
|
||||
test.mutate(&stored)
|
||||
if _, err := decodeCheckpointArtifact(codec, stored); err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("decode error = %v, want %q", err, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
154
internal/framework/pipeline/synchronized_collaborators.go
Normal file
154
internal/framework/pipeline/synchronized_collaborators.go
Normal file
@@ -0,0 +1,154 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"sync"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type lockedDebugRecorder struct {
|
||||
inner DebugRecorder
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
func synchronizedDebugRecorder(inner DebugRecorder) DebugRecorder {
|
||||
if _, ok := inner.(*lockedDebugRecorder); ok {
|
||||
return inner
|
||||
}
|
||||
return &lockedDebugRecorder{inner: inner}
|
||||
}
|
||||
|
||||
// SynchronizedDebugRecorder serializes access when a recorder is shared by
|
||||
// pipeline operations and construction-injected LLM clients.
|
||||
func SynchronizedDebugRecorder(inner DebugRecorder) DebugRecorder {
|
||||
return synchronizedDebugRecorder(inner)
|
||||
}
|
||||
func (r *lockedDebugRecorder) Enabled() bool {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return r.inner.Enabled()
|
||||
}
|
||||
func (r *lockedDebugRecorder) WriteJSON(name string, payload any) error {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return r.inner.WriteJSON(name, payload)
|
||||
}
|
||||
func (r *lockedDebugRecorder) WriteBytes(name string, data []byte) error {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return r.inner.WriteBytes(name, data)
|
||||
}
|
||||
|
||||
type lockedCheckpointLoader struct {
|
||||
inner CheckpointLoader
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
func synchronizedCheckpointLoader(inner CheckpointLoader) CheckpointLoader {
|
||||
if _, ok := inner.(*lockedCheckpointLoader); ok {
|
||||
return inner
|
||||
}
|
||||
return &lockedCheckpointLoader{inner: inner}
|
||||
}
|
||||
func (l *lockedCheckpointLoader) Enabled() bool {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
return l.inner.Enabled()
|
||||
}
|
||||
func (l *lockedCheckpointLoader) Source(key string) (SourceCheckpoint, CheckpointDecision) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
return l.inner.Source(key)
|
||||
}
|
||||
func (l *lockedCheckpointLoader) Chunk(key, digest string) (ChunkCheckpoint, CheckpointDecision) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
return l.inner.Chunk(key, digest)
|
||||
}
|
||||
func (l *lockedCheckpointLoader) Extract(lane, key string, deps []CheckpointFingerprint) (ExtractCheckpoint, CheckpointDecision) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
return l.inner.Extract(lane, key, deps)
|
||||
}
|
||||
func (l *lockedCheckpointLoader) Merge(lane, key string, deps []CheckpointFingerprint) (MergeCheckpoint, CheckpointDecision) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
return l.inner.Merge(lane, key, deps)
|
||||
}
|
||||
func (l *lockedCheckpointLoader) Normalize(lane, key string, deps []CheckpointFingerprint) (NormalizeCheckpoint, CheckpointDecision) {
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
return l.inner.Normalize(lane, key, deps)
|
||||
}
|
||||
|
||||
type lockedCheckpointRecorder struct {
|
||||
inner CheckpointRecorder
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
func synchronizedCheckpointRecorder(inner CheckpointRecorder) CheckpointRecorder {
|
||||
if _, ok := inner.(*lockedCheckpointRecorder); ok {
|
||||
return inner
|
||||
}
|
||||
return &lockedCheckpointRecorder{inner: inner}
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) call(fn func() error) error {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
return fn()
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) SourceRunning(key string) error {
|
||||
return r.call(func() error { return r.inner.SourceRunning(key) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) SourceSucceeded(key string, doc *source.SourceDocument) error {
|
||||
return r.call(func() error { return r.inner.SourceSucceeded(key, doc) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) SourceFailed(key string, err error) error {
|
||||
return r.call(func() error { return r.inner.SourceFailed(key, err) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) ChunkRunning(key, digest string) error {
|
||||
return r.call(func() error { return r.inner.ChunkRunning(key, digest) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) ChunkSucceeded(key, digest string, chunks []source.Chunk, warnings []contracts.Warning) error {
|
||||
return r.call(func() error { return r.inner.ChunkSucceeded(key, digest, chunks, warnings) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) ChunkRejected(key, digest string, rejected contracts.RejectedOutput) error {
|
||||
return r.call(func() error { return r.inner.ChunkRejected(key, digest, rejected) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) ChunkFailed(key, digest string, err error) error {
|
||||
return r.call(func() error { return r.inner.ChunkFailed(key, digest, err) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) ExtractRunning(lane, key string, deps []CheckpointFingerprint) error {
|
||||
return r.call(func() error { return r.inner.ExtractRunning(lane, key, deps) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) ExtractSucceeded(lane, key string, deps []CheckpointFingerprint, outputs []CheckpointArtifact, rejected []contracts.RejectedOutput, warnings []contracts.Warning) error {
|
||||
return r.call(func() error { return r.inner.ExtractSucceeded(lane, key, deps, outputs, rejected, warnings) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) ExtractFailed(lane, key string, deps []CheckpointFingerprint, err error) error {
|
||||
return r.call(func() error { return r.inner.ExtractFailed(lane, key, deps, err) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) MergeRunning(lane, key string, deps []CheckpointFingerprint) error {
|
||||
return r.call(func() error { return r.inner.MergeRunning(lane, key, deps) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) MergeSucceeded(lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
return r.call(func() error { return r.inner.MergeSucceeded(lane, key, deps, output, warnings) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) MergeRejected(lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||
return r.call(func() error { return r.inner.MergeRejected(lane, key, deps, rejected) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) MergeFailed(lane, key string, deps []CheckpointFingerprint, err error) error {
|
||||
return r.call(func() error { return r.inner.MergeFailed(lane, key, deps, err) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) NormalizeRunning(lane, key string, deps []CheckpointFingerprint) error {
|
||||
return r.call(func() error { return r.inner.NormalizeRunning(lane, key, deps) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) NormalizeSucceeded(lane, key string, deps []CheckpointFingerprint, output CheckpointArtifact, warnings []contracts.Warning) error {
|
||||
return r.call(func() error { return r.inner.NormalizeSucceeded(lane, key, deps, output, warnings) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) NormalizeRejected(lane, key string, deps []CheckpointFingerprint, rejected contracts.RejectedOutput) error {
|
||||
return r.call(func() error { return r.inner.NormalizeRejected(lane, key, deps, rejected) })
|
||||
}
|
||||
func (r *lockedCheckpointRecorder) NormalizeFailed(lane, key string, deps []CheckpointFingerprint, err error) error {
|
||||
return r.call(func() error { return r.inner.NormalizeFailed(lane, key, deps, err) })
|
||||
}
|
||||
63
internal/framework/pipeline/typed_execution.go
Normal file
63
internal/framework/pipeline/typed_execution.go
Normal file
@@ -0,0 +1,63 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"reflect"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type erasedExtractArtifact struct {
|
||||
LaneID, ExtractorKey, SourceID, ChunkID string
|
||||
ChunkIndex int
|
||||
ChunkRef source.SourceRef
|
||||
Value any
|
||||
}
|
||||
|
||||
type erasedMergeArtifact struct {
|
||||
LaneID, MergerKey, SourceID string
|
||||
Value any
|
||||
}
|
||||
|
||||
type erasedTypedResult struct {
|
||||
Value any
|
||||
Warnings []contracts.Warning
|
||||
}
|
||||
|
||||
type typedValidationTarget struct {
|
||||
stage ModuleStage
|
||||
laneID string
|
||||
moduleKey string
|
||||
source *source.SourceDocument
|
||||
sourceID string
|
||||
sourceInput contracts.LLMInputMaterial
|
||||
sessionID string
|
||||
references contracts.ReferenceSet
|
||||
llmProfile string
|
||||
metadata map[string]any
|
||||
chunk *source.Chunk
|
||||
chunks []source.Chunk
|
||||
ref source.SourceRef
|
||||
value any
|
||||
}
|
||||
|
||||
func exactTypedValue[T any](operation string, value any) (T, error) {
|
||||
want := reflect.TypeFor[T]()
|
||||
if reflect.TypeOf(value) != want {
|
||||
var zero T
|
||||
return zero, fmt.Errorf("%s: expected exact Go type %s, got %T", operation, want, value)
|
||||
}
|
||||
typed, ok := value.(T)
|
||||
if !ok {
|
||||
var zero T
|
||||
return zero, fmt.Errorf("%s: expected exact Go type %s, got %T", operation, want, value)
|
||||
}
|
||||
return typed, nil
|
||||
}
|
||||
|
||||
type typedExtractOperation func(context.Context, any, contracts.TypedExtractionRequest) (erasedTypedResult, error)
|
||||
type typedMergeOperation func(context.Context, any, contracts.TypedMergeRequest[any]) (erasedTypedResult, error)
|
||||
type typedNormalizeOperation func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error)
|
||||
type typedValidateOperation func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error)
|
||||
477
internal/framework/pipeline/typed_resolution_test.go
Normal file
477
internal/framework/pipeline/typed_resolution_test.go
Normal file
@@ -0,0 +1,477 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type typedTestExtractor[T any] struct{ key string }
|
||||
|
||||
func (e typedTestExtractor[T]) Key() string { return e.key }
|
||||
func (typedTestExtractor[T]) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||
func (typedTestExtractor[T]) Extract(context.Context, contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[T], error) {
|
||||
return contracts.TypedExtractionResult[T]{}, nil
|
||||
}
|
||||
|
||||
type typedTestMerger[T any] struct{ key string }
|
||||
|
||||
func (m typedTestMerger[T]) Key() string { return m.key }
|
||||
func (typedTestMerger[T]) Merge(context.Context, contracts.TypedMergeRequest[T]) (contracts.TypedMergeResult[T], error) {
|
||||
return contracts.TypedMergeResult[T]{}, nil
|
||||
}
|
||||
|
||||
type typedTestNormalizer[T any] struct{ key string }
|
||||
|
||||
func (n typedTestNormalizer[T]) Key() string { return n.key }
|
||||
func (typedTestNormalizer[T]) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||
func (typedTestNormalizer[T]) Normalize(context.Context, contracts.TypedNormalizeRequest[T]) (contracts.TypedNormalizeResult[T], error) {
|
||||
return contracts.TypedNormalizeResult[T]{}, nil
|
||||
}
|
||||
|
||||
type typedTestValidator[T any] struct{ key string }
|
||||
|
||||
func (v typedTestValidator[T]) Name() string { return v.key }
|
||||
func (typedTestValidator[T]) ExecutionClass() contracts.ExecutionClass {
|
||||
return contracts.ExecutionClassDeterministic
|
||||
}
|
||||
func (typedTestValidator[T]) Validate(context.Context, contracts.TypedValidationRequest[T]) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}
|
||||
|
||||
type typedTestChunkValidator struct{ key string }
|
||||
|
||||
func (v typedTestChunkValidator) Name() string { return v.key }
|
||||
func (typedTestChunkValidator) ExecutionClass() contracts.ExecutionClass {
|
||||
return contracts.ExecutionClassDeterministic
|
||||
}
|
||||
func (typedTestChunkValidator) Validate(context.Context, contracts.ChunkValidationRequest) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}
|
||||
|
||||
type typedTestSerializedValidator struct{ key string }
|
||||
|
||||
type typedTestInput struct {
|
||||
key string
|
||||
doc *source.SourceDocument
|
||||
}
|
||||
|
||||
func (v *typedTestInput) Key() string { return v.key }
|
||||
func (v *typedTestInput) Parse(context.Context, contracts.ParseRequest) (*source.SourceDocument, error) {
|
||||
return v.doc, nil
|
||||
}
|
||||
|
||||
type typedTestChunker struct {
|
||||
key string
|
||||
chunks []source.Chunk
|
||||
}
|
||||
|
||||
func (v *typedTestChunker) Key() string { return v.key }
|
||||
func (v *typedTestChunker) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||
func (v *typedTestChunker) Chunk(context.Context, contracts.ChunkRequest) (contracts.ChunkResult, error) {
|
||||
return contracts.ChunkResult{Chunks: v.chunks}, nil
|
||||
}
|
||||
|
||||
type typedTestOutput struct{ key string }
|
||||
|
||||
func (v *typedTestOutput) Key() string { return v.key }
|
||||
func (v *typedTestOutput) Encode(context.Context, contracts.OutputRequest) (contracts.OutputResult, error) {
|
||||
return contracts.OutputResult{}, nil
|
||||
}
|
||||
func typedTestDocument() *source.SourceDocument {
|
||||
ref := source.SourceRef{SourceID: "source", StartUnitID: 1, EndUnitID: 1}
|
||||
doc := &source.SourceDocument{ID: "source", Kind: "document", Format: "text/plain", Units: []source.SourceUnit{{ID: 1, Kind: "line", Text: "text", Ref: ref}}}
|
||||
doc.Digest, _ = source.DigestDocument(doc)
|
||||
return doc
|
||||
}
|
||||
|
||||
func (v typedTestSerializedValidator) Name() string { return v.key }
|
||||
func (typedTestSerializedValidator) ExecutionClass() contracts.ExecutionClass {
|
||||
return contracts.ExecutionClassDeterministic
|
||||
}
|
||||
func (typedTestSerializedValidator) Validate(context.Context, contracts.SerializedValidationRequest) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}
|
||||
|
||||
type typedCatalogOptions struct {
|
||||
registerNotesCodec bool
|
||||
registerScoreCodec bool
|
||||
registerNotesMerger bool
|
||||
registerScoreMerger bool
|
||||
registerNotesNormalizer bool
|
||||
registerScoreNormalizer bool
|
||||
registerScoreValidator bool
|
||||
scoreExtractorUsesNotes bool
|
||||
notesCodec testArtifactCodec[codecNotes]
|
||||
scoreCodec testArtifactCodec[codecScore]
|
||||
}
|
||||
|
||||
func completeTypedCatalogOptions() typedCatalogOptions {
|
||||
return typedCatalogOptions{
|
||||
registerNotesCodec: true,
|
||||
registerScoreCodec: true,
|
||||
registerNotesMerger: true,
|
||||
registerScoreMerger: true,
|
||||
registerNotesNormalizer: true,
|
||||
registerScoreNormalizer: true,
|
||||
registerScoreValidator: true,
|
||||
notesCodec: notesCodec(),
|
||||
scoreCodec: scoreCodec(),
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveTypedHeterogeneousLanes(t *testing.T) {
|
||||
catalog := typedResolutionCatalog(t, completeTypedCatalogOptions())
|
||||
resolved, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, catalog)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
if got, want := resolvedLaneIDs(resolved.ArtifactLanes), []string{"notes", "score"}; !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("lane order = %#v, want %#v", got, want)
|
||||
}
|
||||
assertResolvedArtifactIdentity(t, resolved.ArtifactLanes[0], "test/notes", "notes.v1")
|
||||
assertResolvedArtifactIdentity(t, resolved.ArtifactLanes[1], "test/score", "score.v1")
|
||||
|
||||
chunkChain := resolved.ValidatorChains[0]
|
||||
if got := resolvedValidatorTargets(chunkChain.Validators); !reflect.DeepEqual(got, []ValidatorTarget{ValidatorTargetChunk, ValidatorTargetSerialized}) {
|
||||
t.Fatalf("chunk validator targets = %#v, want chunk then serialized", got)
|
||||
}
|
||||
notesExtract := resolved.ValidatorChains[1]
|
||||
if got := resolvedValidatorTargets(notesExtract.Validators); !reflect.DeepEqual(got, []ValidatorTarget{ValidatorTargetTyped, ValidatorTargetSerialized}) {
|
||||
t.Fatalf("notes extract validator targets = %#v, want typed then serialized", got)
|
||||
}
|
||||
if notesExtract.Validators[0].ArtifactKind != "test/notes" || resolved.ValidatorChains[4].Validators[0].ArtifactKind != "test/score" {
|
||||
t.Fatalf("resolved validator kinds = %#v, want lane kinds", resolved.ValidatorChains)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrepareConstructsHeterogeneousTypedLanes(t *testing.T) {
|
||||
catalog := typedResolutionCatalog(t, completeTypedCatalogOptions())
|
||||
resolved, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, catalog)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
prepared, err := Prepare(resolved, registriesFromModuleCatalog(catalog), ModuleDependencies{})
|
||||
if err != nil {
|
||||
t.Fatalf("Prepare() error = %v, want nil", err)
|
||||
}
|
||||
if len(prepared.ArtifactLanes) != 2 || prepared.lanes[0].typed == nil || prepared.lanes[1].typed == nil {
|
||||
t.Fatalf("PreparedPipeline lanes = %#v, want two typed executors", prepared.ArtifactLanes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunExecutesHeterogeneousTypedLanesWithCheckpointsDisabled(t *testing.T) {
|
||||
catalog := typedResolutionCatalog(t, completeTypedCatalogOptions())
|
||||
resolved, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, catalog)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v", err)
|
||||
}
|
||||
prepared, err := Prepare(resolved, registriesFromModuleCatalog(catalog), ModuleDependencies{})
|
||||
if err != nil {
|
||||
t.Fatalf("Prepare() error = %v", err)
|
||||
}
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte(`{"source":true}`), RunID: "typed-run"})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if len(output.NormalizeOutputs) != 2 {
|
||||
t.Fatalf("normalize outputs = %#v, want two typed lane results", output.NormalizeOutputs)
|
||||
}
|
||||
if len(output.CheckpointEvents) != 0 {
|
||||
t.Fatalf("checkpoint events = %#v, want none with checkpoint loading disabled", output.CheckpointEvents)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveTypedLaneRejectsIncompatibleComposition(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
mutate func(*typedCatalogOptions)
|
||||
want string
|
||||
}{
|
||||
{name: "missing codec", mutate: func(options *typedCatalogOptions) { options.registerScoreCodec = false }, want: `artifact codec "test/score" is not registered`},
|
||||
{name: "missing merger variant", mutate: func(options *typedCatalogOptions) { options.registerScoreMerger = false }, want: `merger "typed/merge" has no typed variant for artifact kind "test/score"`},
|
||||
{name: "missing normalizer variant", mutate: func(options *typedCatalogOptions) { options.registerScoreNormalizer = false }, want: `normalizer "typed/normalize" has no typed variant for artifact kind "test/score"`},
|
||||
{name: "extractor Go type mismatch", mutate: func(options *typedCatalogOptions) { options.scoreExtractorUsesNotes = true }, want: `artifact kind "test/score" requires Go type pipeline.codecScore, got pipeline.codecNotes`},
|
||||
{name: "wrong validator kind", mutate: func(options *typedCatalogOptions) { options.registerScoreValidator = false }, want: `validator "typed/check" has no typed variant for artifact kind "test/score"; registered kinds: test/notes`},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
options := completeTypedCatalogOptions()
|
||||
test.mutate(&options)
|
||||
_, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, typedResolutionCatalog(t, options))
|
||||
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want %q", err, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestTypedVariantRegistrationRejectsDuplicates(t *testing.T) {
|
||||
registry := NewMergerRegistry()
|
||||
spec := ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"}
|
||||
constructor := func() (contracts.Merger[codecNotes], error) {
|
||||
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
|
||||
}
|
||||
if err := RegisterMerger(registry, spec, constructor); err != nil {
|
||||
t.Fatalf("RegisterMerger() error = %v, want nil", err)
|
||||
}
|
||||
if err := RegisterMerger(registry, spec, constructor); err == nil || !strings.Contains(err.Error(), "already registered") {
|
||||
t.Fatalf("duplicate RegisterMerger() error = %v, want duplicate variant error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConstructorRegistrationsRejectUnconfiguredOptions(t *testing.T) {
|
||||
extractors := NewExtractorRegistry()
|
||||
if err := RegisterExtractor(extractors, ModuleSpec{Key: "typed/extract", Stage: StageExtract, ArtifactKind: "test/notes"}, func() (contracts.Extractor[codecNotes], error) {
|
||||
return typedTestExtractor[codecNotes]{key: "typed/extract"}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterExtractor() error = %v", err)
|
||||
}
|
||||
if err := extractors.validateOptions("typed/extract", map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||
t.Fatalf("extractor option validation error = %v, want unknown option", err)
|
||||
}
|
||||
|
||||
mergers := NewMergerRegistry()
|
||||
if err := RegisterMerger(mergers, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: "test/notes"}, func() (contracts.Merger[codecNotes], error) {
|
||||
return typedTestMerger[codecNotes]{key: "typed/merge"}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterMerger() error = %v", err)
|
||||
}
|
||||
if err := mergers.validateOptions("typed/merge", "test/notes", map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||
t.Fatalf("merger option validation error = %v, want unknown option", err)
|
||||
}
|
||||
|
||||
normalizers := NewNormalizerRegistry()
|
||||
if err := RegisterNormalizer(normalizers, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
||||
return typedTestNormalizer[codecNotes]{key: "typed/normalize"}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||
}
|
||||
if err := normalizers.validateOptions("typed/normalize", "test/notes", map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||
t.Fatalf("normalizer option validation error = %v, want unknown option", err)
|
||||
}
|
||||
|
||||
validators := NewValidatorRegistry()
|
||||
if err := RegisterTypedValidator(validators, "test/notes", ValidatorSpec{Key: "typed/check", ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.TypedValidator[codecNotes], error) {
|
||||
return typedTestValidator[codecNotes]{key: "typed/check"}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterTypedValidator() error = %v", err)
|
||||
}
|
||||
if err := validators.validateOptions(ResolvedValidator{Binding: ModuleBinding{Module: "typed/check", Options: map[string]any{"unexpected": true}}, Target: ValidatorTargetTyped, ArtifactKind: "test/notes"}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||
t.Fatalf("validator option validation error = %v, want unknown option", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvedPipelineDigestIncludesArtifactSchemaIdentity(t *testing.T) {
|
||||
baseOptions := completeTypedCatalogOptions()
|
||||
base, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, typedResolutionCatalog(t, baseOptions))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline(base) error = %v, want nil", err)
|
||||
}
|
||||
|
||||
identityOptions := completeTypedCatalogOptions()
|
||||
identityOptions.notesCodec.schema.ID = "notes-renamed.v1"
|
||||
identity, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, typedResolutionCatalog(t, identityOptions))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline(identity) error = %v, want nil", err)
|
||||
}
|
||||
if base.Digest == identity.Digest {
|
||||
t.Fatalf("pipeline digest = %q after schema identity change, want different digest", identity.Digest)
|
||||
}
|
||||
|
||||
digestOptions := completeTypedCatalogOptions()
|
||||
digestOptions.notesCodec.schema.JSONSchema = []byte(`{"additionalProperties":false,"description":"changed","properties":{"items":{"items":{"type":"string"},"type":"array"}},"required":["items"],"type":"object"}`)
|
||||
changedSchema, err := ResolvePipeline(typedResolutionProfile(), ResolveOptions{}, typedResolutionCatalog(t, digestOptions))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline(schema bytes) error = %v, want nil", err)
|
||||
}
|
||||
if base.Digest == changedSchema.Digest {
|
||||
t.Fatalf("pipeline digest = %q after schema digest change, want different digest", changedSchema.Digest)
|
||||
}
|
||||
}
|
||||
|
||||
func typedResolutionCatalog(t *testing.T, options typedCatalogOptions) ModuleCatalog {
|
||||
t.Helper()
|
||||
catalog := ModuleCatalog{
|
||||
Inputs: NewInputAdapterRegistry(),
|
||||
Chunkers: NewChunkerRegistry(),
|
||||
ArtifactCodecs: NewArtifactCodecRegistry(),
|
||||
Extractors: NewExtractorRegistry(),
|
||||
Mergers: NewMergerRegistry(),
|
||||
Normalizers: NewNormalizerRegistry(),
|
||||
Validators: NewValidatorRegistry(),
|
||||
ValidatorChains: NewValidatorChainRegistry(),
|
||||
Outputs: NewOutputEncoderRegistry(),
|
||||
}
|
||||
mustRegisterTypedTestBase(t, catalog)
|
||||
if options.registerNotesCodec {
|
||||
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, options.notesCodec)
|
||||
}
|
||||
if options.registerScoreCodec {
|
||||
mustRegisterArtifactCodec(t, catalog.ArtifactCodecs, options.scoreCodec)
|
||||
}
|
||||
mustRegisterTypedExtractor(t, catalog.Extractors, "typed/extract-notes", "test/notes", typedTestExtractor[codecNotes]{key: "typed/extract-notes"})
|
||||
if options.scoreExtractorUsesNotes {
|
||||
mustRegisterTypedExtractor(t, catalog.Extractors, "typed/extract-score", "test/score", typedTestExtractor[codecNotes]{key: "typed/extract-score"})
|
||||
} else {
|
||||
mustRegisterTypedExtractor(t, catalog.Extractors, "typed/extract-score", "test/score", typedTestExtractor[codecScore]{key: "typed/extract-score"})
|
||||
}
|
||||
if options.registerNotesMerger {
|
||||
mustRegisterTypedMerger(t, catalog.Mergers, "test/notes", typedTestMerger[codecNotes]{key: "typed/merge"})
|
||||
}
|
||||
if options.registerScoreMerger {
|
||||
mustRegisterTypedMerger(t, catalog.Mergers, "test/score", typedTestMerger[codecScore]{key: "typed/merge"})
|
||||
}
|
||||
if options.registerNotesNormalizer {
|
||||
mustRegisterTypedNormalizer(t, catalog.Normalizers, "test/notes", typedTestNormalizer[codecNotes]{key: "typed/normalize"})
|
||||
}
|
||||
if options.registerScoreNormalizer {
|
||||
mustRegisterTypedNormalizer(t, catalog.Normalizers, "test/score", typedTestNormalizer[codecScore]{key: "typed/normalize"})
|
||||
}
|
||||
mustRegisterTypedValidator(t, catalog.Validators, "test/notes", typedTestValidator[codecNotes]{key: "typed/check"})
|
||||
if options.registerScoreValidator {
|
||||
mustRegisterTypedValidator(t, catalog.Validators, "test/score", typedTestValidator[codecScore]{key: "typed/check"})
|
||||
}
|
||||
if err := RegisterChunkValidator(catalog.Validators, ValidatorSpec{Key: "chunk/check", ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.ChunkValidator, error) {
|
||||
return typedTestChunkValidator{key: "chunk/check"}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterChunkValidator() error = %v", err)
|
||||
}
|
||||
if err := RegisterSerializedValidator(catalog.Validators, SerializedValidatorSpec{
|
||||
ValidatorSpec: ValidatorSpec{Key: "serialized/check", ExecutionClass: contracts.ExecutionClassDeterministic},
|
||||
SupportsArtifacts: true,
|
||||
}, func() (contracts.SerializedValidator, error) {
|
||||
return typedTestSerializedValidator{key: "serialized/check"}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterSerializedValidator() error = %v", err)
|
||||
}
|
||||
if err := RegisterSerializedValidator(catalog.Validators, SerializedValidatorSpec{
|
||||
ValidatorSpec: ValidatorSpec{Key: "serialized/chunks", ExecutionClass: contracts.ExecutionClassDeterministic},
|
||||
SupportsChunks: true,
|
||||
}, func() (contracts.SerializedValidator, error) {
|
||||
return typedTestSerializedValidator{key: "serialized/chunks"}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterSerializedValidator(chunks) error = %v", err)
|
||||
}
|
||||
return catalog
|
||||
}
|
||||
|
||||
func mustRegisterTypedTestBase(t *testing.T, catalog ModuleCatalog) {
|
||||
t.Helper()
|
||||
if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{Key: "typed/input", Stage: StageInput}, func() (contracts.InputAdapter, error) {
|
||||
return &typedTestInput{key: "typed/input", doc: typedTestDocument()}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register input: %v", err)
|
||||
}
|
||||
if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{Key: "typed/chunk", Stage: StageChunk}, func() (contracts.Chunker, error) {
|
||||
doc := typedTestDocument()
|
||||
return &typedTestChunker{key: "typed/chunk", chunks: []source.Chunk{{ID: "chunk-1", SourceID: doc.ID, Index: 0, Ref: doc.Units[0].Ref, Content: []byte(`{"chunk":1}`), MediaType: "application/json", Units: []source.SourceUnit{doc.Units[0]}}}}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register chunker: %v", err)
|
||||
}
|
||||
if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{Key: "typed/output", Stage: StageOutput}, func() (contracts.OutputEncoder, error) {
|
||||
return &typedTestOutput{key: "typed/output"}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register output: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func registriesFromModuleCatalog(catalog ModuleCatalog) Registries {
|
||||
return Registries{
|
||||
Inputs: catalog.Inputs, Chunkers: catalog.Chunkers, ArtifactCodecs: catalog.ArtifactCodecs,
|
||||
Extractors: catalog.Extractors, Mergers: catalog.Mergers, Normalizers: catalog.Normalizers,
|
||||
Validators: catalog.Validators, ValidatorChains: catalog.ValidatorChains, Outputs: catalog.Outputs,
|
||||
}
|
||||
}
|
||||
|
||||
func mustRegisterArtifactCodec[T any](t *testing.T, registry *ArtifactCodecRegistry, codec contracts.ArtifactCodec[T]) {
|
||||
t.Helper()
|
||||
if err := RegisterArtifactCodec(registry, codec); err != nil {
|
||||
t.Fatalf("RegisterArtifactCodec() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustRegisterTypedExtractor[T any](t *testing.T, registry *ExtractorRegistry, key string, kind contracts.ArtifactKind, extractor contracts.Extractor[T]) {
|
||||
t.Helper()
|
||||
if err := RegisterExtractor(registry, ModuleSpec{Key: key, Stage: StageExtract, ArtifactKind: kind}, func() (contracts.Extractor[T], error) { return extractor, nil }); err != nil {
|
||||
t.Fatalf("RegisterExtractor() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustRegisterTypedMerger[T any](t *testing.T, registry *MergerRegistry, kind contracts.ArtifactKind, merger contracts.Merger[T]) {
|
||||
t.Helper()
|
||||
if err := RegisterMerger(registry, ModuleSpec{Key: "typed/merge", Stage: StageMerge, ArtifactKind: kind}, func() (contracts.Merger[T], error) { return merger, nil }); err != nil {
|
||||
t.Fatalf("RegisterMerger() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustRegisterTypedNormalizer[T any](t *testing.T, registry *NormalizerRegistry, kind contracts.ArtifactKind, normalizer contracts.Normalizer[T]) {
|
||||
t.Helper()
|
||||
if err := RegisterNormalizer(registry, ModuleSpec{Key: "typed/normalize", Stage: StageNormalize, ArtifactKind: kind}, func() (contracts.Normalizer[T], error) { return normalizer, nil }); err != nil {
|
||||
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustRegisterTypedValidator[T any](t *testing.T, registry *ValidatorRegistry, kind contracts.ArtifactKind, validator contracts.TypedValidator[T]) {
|
||||
t.Helper()
|
||||
if err := RegisterTypedValidator(registry, kind, ValidatorSpec{Key: "typed/check", ExecutionClass: contracts.ExecutionClassDeterministic}, func() (contracts.TypedValidator[T], error) { return validator, nil }); err != nil {
|
||||
t.Fatalf("RegisterTypedValidator() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func typedResolutionProfile() PipelineProfile {
|
||||
validatorOverride := ValidatorOverride{Set: true, Validators: []ModuleBinding{Binding("typed/check"), Binding("serialized/check")}}
|
||||
return PipelineProfile{
|
||||
ID: "typed-pipeline",
|
||||
Input: Binding("typed/input"),
|
||||
Chunk: ModuleBinding{Module: "typed/chunk", Validators: ValidatorOverride{Set: true, Validators: []ModuleBinding{Binding("chunk/check"), Binding("serialized/chunks")}}},
|
||||
Artifacts: map[string]ArtifactLaneProfile{
|
||||
"score": typedLaneProfile("typed/extract-score", validatorOverride),
|
||||
"notes": typedLaneProfile("typed/extract-notes", validatorOverride),
|
||||
},
|
||||
Output: Binding("typed/output"),
|
||||
}
|
||||
}
|
||||
|
||||
func typedLaneProfile(extractor string, validators ValidatorOverride) ArtifactLaneProfile {
|
||||
return ArtifactLaneProfile{
|
||||
Extract: ModuleBinding{Module: extractor, Validators: validators},
|
||||
Merge: Binding("typed/merge"),
|
||||
Normalize: Binding("typed/normalize"),
|
||||
}
|
||||
}
|
||||
|
||||
func assertResolvedArtifactIdentity(t *testing.T, lane ResolvedArtifactLane, kind contracts.ArtifactKind, schemaID string) {
|
||||
t.Helper()
|
||||
if lane.ArtifactKind != kind || lane.ArtifactSchemaID != schemaID || lane.ArtifactSchemaName == "" || lane.ArtifactSchemaVersion == "" || lane.ArtifactSchemaDigest == "" {
|
||||
t.Fatalf("resolved lane identity = %#v, want kind %q schema %q with complete metadata", lane, kind, schemaID)
|
||||
}
|
||||
}
|
||||
|
||||
func resolvedLaneIDs(lanes []ResolvedArtifactLane) []string {
|
||||
ids := make([]string, len(lanes))
|
||||
for i, lane := range lanes {
|
||||
ids[i] = lane.ID
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
func resolvedValidatorTargets(validators []ResolvedValidator) []ValidatorTarget {
|
||||
targets := make([]ValidatorTarget, len(validators))
|
||||
for i, validator := range validators {
|
||||
targets[i] = validator.Target
|
||||
}
|
||||
return targets
|
||||
}
|
||||
|
||||
var _ contracts.Extractor[codecNotes] = typedTestExtractor[codecNotes]{}
|
||||
var _ contracts.Merger[codecNotes] = typedTestMerger[codecNotes]{}
|
||||
var _ contracts.Normalizer[codecNotes] = typedTestNormalizer[codecNotes]{}
|
||||
var _ contracts.TypedValidator[codecNotes] = typedTestValidator[codecNotes]{}
|
||||
var _ contracts.ChunkValidator = typedTestChunkValidator{}
|
||||
var _ contracts.SerializedValidator = typedTestSerializedValidator{}
|
||||
@@ -1,125 +1,292 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type ValidatorConstructor func() (contracts.Validator, error)
|
||||
|
||||
type ValidatorSpec struct {
|
||||
Key string `json:"key"`
|
||||
ExecutionClass contracts.ExecutionClass `json:"execution_class"`
|
||||
}
|
||||
|
||||
type SerializedValidatorSpec struct {
|
||||
ValidatorSpec
|
||||
SupportsChunks bool `json:"supports_chunks,omitempty"`
|
||||
SupportsArtifacts bool `json:"supports_artifacts,omitempty"`
|
||||
}
|
||||
|
||||
type ValidatorTarget string
|
||||
|
||||
const (
|
||||
ValidatorTargetChunk ValidatorTarget = "chunk"
|
||||
ValidatorTargetSerialized ValidatorTarget = "serialized"
|
||||
ValidatorTargetTyped ValidatorTarget = "typed"
|
||||
)
|
||||
|
||||
type ValidatorRegistry struct {
|
||||
constructors map[string]ValidatorConstructor
|
||||
specs map[string]ValidatorSpec
|
||||
typedEntries map[artifactVariantKey]typedValidatorEntry
|
||||
chunkEntries map[string]chunkValidatorEntry
|
||||
serializedEntries map[string]serializedValidatorEntry
|
||||
}
|
||||
|
||||
type typedValidatorEntry struct {
|
||||
spec ValidatorSpec
|
||||
kind contracts.ArtifactKind
|
||||
valueType reflect.Type
|
||||
validateOptions OptionValidator
|
||||
builder func(BuildRequest) (any, error)
|
||||
validate typedValidateOperation
|
||||
}
|
||||
|
||||
type chunkValidatorEntry struct {
|
||||
spec ValidatorSpec
|
||||
validateOptions OptionValidator
|
||||
builder func(BuildRequest) (contracts.ChunkValidator, error)
|
||||
}
|
||||
|
||||
type serializedValidatorEntry struct {
|
||||
spec SerializedValidatorSpec
|
||||
validateOptions OptionValidator
|
||||
builder func(BuildRequest) (contracts.SerializedValidator, error)
|
||||
}
|
||||
|
||||
func NewValidatorRegistry() *ValidatorRegistry {
|
||||
return &ValidatorRegistry{
|
||||
constructors: make(map[string]ValidatorConstructor),
|
||||
specs: make(map[string]ValidatorSpec),
|
||||
typedEntries: make(map[artifactVariantKey]typedValidatorEntry),
|
||||
chunkEntries: make(map[string]chunkValidatorEntry),
|
||||
serializedEntries: make(map[string]serializedValidatorEntry),
|
||||
}
|
||||
}
|
||||
|
||||
func (r *ValidatorRegistry) Register(key string, constructor ValidatorConstructor) error {
|
||||
return r.RegisterWithSpec(ValidatorSpec{Key: key, ExecutionClass: contracts.ExecutionClassDeterministic}, constructor)
|
||||
func RegisterTypedValidator[T any](registry *ValidatorRegistry, kind contracts.ArtifactKind, spec ValidatorSpec, constructor func() (contracts.TypedValidator[T], error)) error {
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("validator constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||
}
|
||||
return RegisterTypedValidatorBuilder(registry, kind, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.TypedValidator[T], error) {
|
||||
return constructor()
|
||||
})
|
||||
}
|
||||
|
||||
func (r *ValidatorRegistry) RegisterWithSpec(spec ValidatorSpec, constructor ValidatorConstructor) error {
|
||||
if r == nil {
|
||||
func RegisterTypedValidatorBuilder[T any](registry *ValidatorRegistry, kind contracts.ArtifactKind, spec ValidatorSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.TypedValidator[T], error)) error {
|
||||
if registry == nil {
|
||||
return fmt.Errorf("validator registry must not be nil")
|
||||
}
|
||||
|
||||
normalizedSpec, err := normalizeValidatorSpec(spec)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("validator constructor for %q must not be nil", normalizedSpec.Key)
|
||||
kind = normalizeArtifactKind(kind)
|
||||
if kind == "" {
|
||||
return fmt.Errorf("typed validator %q artifact kind must not be empty", normalizedSpec.Key)
|
||||
}
|
||||
if _, ok := r.constructors[normalizedSpec.Key]; ok {
|
||||
return fmt.Errorf("validator %q is already registered", normalizedSpec.Key)
|
||||
if validateOptions == nil {
|
||||
return fmt.Errorf("validator option validator for %q must not be nil", normalizedSpec.Key)
|
||||
}
|
||||
|
||||
if r.constructors == nil {
|
||||
r.constructors = make(map[string]ValidatorConstructor)
|
||||
if builder == nil {
|
||||
return fmt.Errorf("validator builder for %q must not be nil", normalizedSpec.Key)
|
||||
}
|
||||
if r.specs == nil {
|
||||
r.specs = make(map[string]ValidatorSpec)
|
||||
key := artifactVariantKey{module: normalizedSpec.Key, kind: kind}
|
||||
if _, ok := registry.typedEntries[key]; ok {
|
||||
return fmt.Errorf("validator %q variant for artifact kind %q is already registered", key.module, key.kind)
|
||||
}
|
||||
if registry.typedEntries == nil {
|
||||
registry.typedEntries = make(map[artifactVariantKey]typedValidatorEntry)
|
||||
}
|
||||
registry.typedEntries[key] = typedValidatorEntry{
|
||||
spec: normalizedSpec,
|
||||
kind: kind,
|
||||
valueType: reflect.TypeFor[T](),
|
||||
validateOptions: validateOptions,
|
||||
builder: func(request BuildRequest) (any, error) {
|
||||
return builder(cloneBuildRequest(request))
|
||||
},
|
||||
validate: func(ctx context.Context, implementation any, target typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
validator, ok := implementation.(contracts.TypedValidator[T])
|
||||
if !ok {
|
||||
return contracts.ValidationResult{}, fmt.Errorf("validator %q has incompatible implementation %T", normalizedSpec.Key, implementation)
|
||||
}
|
||||
value, err := exactTypedValue[T]("validate artifact value", target.value)
|
||||
if err != nil {
|
||||
return contracts.ValidationResult{}, err
|
||||
}
|
||||
return validator.Validate(ctx, contracts.TypedValidationRequest[T]{Stage: string(target.stage), LaneID: target.laneID, ModuleKey: target.moduleKey, Source: target.source, SourceID: target.sourceID, SourceInput: target.sourceInput, SessionID: target.sessionID, References: target.references, LLMProfile: target.llmProfile, Metadata: target.metadata, Chunk: target.chunk, Chunks: target.chunks, Ref: target.ref, Value: value})
|
||||
},
|
||||
}
|
||||
r.constructors[normalizedSpec.Key] = constructor
|
||||
r.specs[normalizedSpec.Key] = normalizedSpec
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *ValidatorRegistry) Build(key string) (contracts.Validator, error) {
|
||||
if r == nil {
|
||||
return nil, fmt.Errorf("validator registry must not be nil")
|
||||
func RegisterChunkValidator(registry *ValidatorRegistry, spec ValidatorSpec, constructor func() (contracts.ChunkValidator, error)) error {
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("validator constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||
}
|
||||
return RegisterChunkValidatorBuilder(registry, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.ChunkValidator, error) {
|
||||
return constructor()
|
||||
})
|
||||
}
|
||||
|
||||
normalizedKey := strings.TrimSpace(key)
|
||||
if normalizedKey == "" {
|
||||
return nil, fmt.Errorf("validator key must not be empty")
|
||||
func RegisterChunkValidatorBuilder(registry *ValidatorRegistry, spec ValidatorSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.ChunkValidator, error)) error {
|
||||
if registry == nil {
|
||||
return fmt.Errorf("validator registry must not be nil")
|
||||
}
|
||||
|
||||
constructor, ok := r.constructors[normalizedKey]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("validator %q is not registered", normalizedKey)
|
||||
}
|
||||
|
||||
validator, err := constructor()
|
||||
normalizedSpec, err := normalizeValidatorSpec(spec)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("build validator %q: %w", normalizedKey, err)
|
||||
return err
|
||||
}
|
||||
if validateOptions == nil {
|
||||
return fmt.Errorf("validator option validator for %q must not be nil", normalizedSpec.Key)
|
||||
}
|
||||
if builder == nil {
|
||||
return fmt.Errorf("validator builder for %q must not be nil", normalizedSpec.Key)
|
||||
}
|
||||
if _, ok := registry.chunkEntries[normalizedSpec.Key]; ok {
|
||||
return fmt.Errorf("chunk validator %q is already registered", normalizedSpec.Key)
|
||||
}
|
||||
if registry.chunkEntries == nil {
|
||||
registry.chunkEntries = make(map[string]chunkValidatorEntry)
|
||||
}
|
||||
registry.chunkEntries[normalizedSpec.Key] = chunkValidatorEntry{spec: normalizedSpec, validateOptions: validateOptions, builder: builder}
|
||||
return nil
|
||||
}
|
||||
|
||||
func RegisterSerializedValidator(registry *ValidatorRegistry, spec SerializedValidatorSpec, constructor func() (contracts.SerializedValidator, error)) error {
|
||||
if constructor == nil {
|
||||
return fmt.Errorf("validator constructor for %q must not be nil", strings.TrimSpace(spec.Key))
|
||||
}
|
||||
return RegisterSerializedValidatorBuilder(registry, spec, rejectUnconfiguredOptions, func(BuildRequest) (contracts.SerializedValidator, error) {
|
||||
return constructor()
|
||||
})
|
||||
}
|
||||
|
||||
func RegisterSerializedValidatorBuilder(registry *ValidatorRegistry, spec SerializedValidatorSpec, validateOptions OptionValidator, builder func(BuildRequest) (contracts.SerializedValidator, error)) error {
|
||||
if registry == nil {
|
||||
return fmt.Errorf("validator registry must not be nil")
|
||||
}
|
||||
normalizedValidatorSpec, err := normalizeValidatorSpec(spec.ValidatorSpec)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
spec.ValidatorSpec = normalizedValidatorSpec
|
||||
if !spec.SupportsChunks && !spec.SupportsArtifacts {
|
||||
return fmt.Errorf("serialized validator %q must support chunks, artifacts, or both", spec.Key)
|
||||
}
|
||||
if validateOptions == nil {
|
||||
return fmt.Errorf("validator option validator for %q must not be nil", spec.Key)
|
||||
}
|
||||
if builder == nil {
|
||||
return fmt.Errorf("validator builder for %q must not be nil", spec.Key)
|
||||
}
|
||||
if _, ok := registry.serializedEntries[spec.Key]; ok {
|
||||
return fmt.Errorf("serialized validator %q is already registered", spec.Key)
|
||||
}
|
||||
if registry.serializedEntries == nil {
|
||||
registry.serializedEntries = make(map[string]serializedValidatorEntry)
|
||||
}
|
||||
registry.serializedEntries[spec.Key] = serializedValidatorEntry{spec: spec, validateOptions: validateOptions, builder: builder}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *ValidatorRegistry) validateOptions(resolved ResolvedValidator) error {
|
||||
if r == nil {
|
||||
return fmt.Errorf("validator registry must not be nil")
|
||||
}
|
||||
key := strings.TrimSpace(resolved.Binding.Module)
|
||||
var validator OptionValidator
|
||||
switch resolved.Target {
|
||||
case ValidatorTargetTyped:
|
||||
entry, ok := r.typedEntry(key, resolved.ArtifactKind)
|
||||
if ok {
|
||||
validator = entry.validateOptions
|
||||
}
|
||||
case ValidatorTargetChunk:
|
||||
entry, ok := r.chunkEntry(key)
|
||||
if ok {
|
||||
validator = entry.validateOptions
|
||||
}
|
||||
case ValidatorTargetSerialized:
|
||||
entry, ok := r.serializedEntry(key)
|
||||
if ok {
|
||||
validator = entry.validateOptions
|
||||
}
|
||||
}
|
||||
if validator == nil {
|
||||
return nil, fmt.Errorf("validator %q constructor returned nil", normalizedKey)
|
||||
return fmt.Errorf("validator %q construction entry is not registered", key)
|
||||
}
|
||||
if validator.Name() != normalizedKey {
|
||||
return nil, fmt.Errorf("validator %q returned name %q", normalizedKey, validator.Name())
|
||||
}
|
||||
spec, ok := r.specs[normalizedKey]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("validator %q spec is not registered", normalizedKey)
|
||||
}
|
||||
if validator.ExecutionClass() != spec.ExecutionClass {
|
||||
return nil, fmt.Errorf("validator %q returned execution class %q, want %q", normalizedKey, validator.ExecutionClass(), spec.ExecutionClass)
|
||||
}
|
||||
|
||||
return validator, nil
|
||||
return validateRegisteredOptions(validator, resolved.Binding.Options)
|
||||
}
|
||||
|
||||
func (r *ValidatorRegistry) Spec(key string) (ValidatorSpec, bool) {
|
||||
if r == nil {
|
||||
return ValidatorSpec{}, false
|
||||
}
|
||||
|
||||
spec, ok := r.specs[strings.TrimSpace(key)]
|
||||
if !ok {
|
||||
return ValidatorSpec{}, false
|
||||
normalized := strings.TrimSpace(key)
|
||||
if entry, found := r.chunkEntries[normalized]; found {
|
||||
return entry.spec, true
|
||||
}
|
||||
return spec, true
|
||||
if entry, found := r.serializedEntries[normalized]; found {
|
||||
return entry.spec.ValidatorSpec, true
|
||||
}
|
||||
if kinds := r.registeredTypedKinds(normalized); len(kinds) > 0 {
|
||||
entry, found := r.typedEntry(normalized, kinds[0])
|
||||
return entry.spec, found
|
||||
}
|
||||
return ValidatorSpec{}, false
|
||||
}
|
||||
|
||||
func (r *ValidatorRegistry) typedEntry(key string, kind contracts.ArtifactKind) (typedValidatorEntry, bool) {
|
||||
if r == nil {
|
||||
return typedValidatorEntry{}, false
|
||||
}
|
||||
entry, ok := r.typedEntries[artifactVariantKey{module: strings.TrimSpace(key), kind: normalizeArtifactKind(kind)}]
|
||||
return entry, ok
|
||||
}
|
||||
|
||||
func (r *ValidatorRegistry) chunkEntry(key string) (chunkValidatorEntry, bool) {
|
||||
if r == nil {
|
||||
return chunkValidatorEntry{}, false
|
||||
}
|
||||
entry, ok := r.chunkEntries[strings.TrimSpace(key)]
|
||||
return entry, ok
|
||||
}
|
||||
|
||||
func (r *ValidatorRegistry) serializedEntry(key string) (serializedValidatorEntry, bool) {
|
||||
if r == nil {
|
||||
return serializedValidatorEntry{}, false
|
||||
}
|
||||
entry, ok := r.serializedEntries[strings.TrimSpace(key)]
|
||||
return entry, ok
|
||||
}
|
||||
|
||||
func (r *ValidatorRegistry) registeredTypedKinds(key string) []contracts.ArtifactKind {
|
||||
if r == nil {
|
||||
return nil
|
||||
}
|
||||
module := strings.TrimSpace(key)
|
||||
kinds := make([]contracts.ArtifactKind, 0)
|
||||
for variant := range r.typedEntries {
|
||||
if variant.module == module {
|
||||
kinds = append(kinds, variant.kind)
|
||||
}
|
||||
}
|
||||
sortArtifactKinds(kinds)
|
||||
return kinds
|
||||
}
|
||||
|
||||
func (r *ValidatorRegistry) RegisteredSpecs() []ValidatorSpec {
|
||||
if r == nil || len(r.specs) == 0 {
|
||||
if r == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
keys := make([]string, 0, len(r.specs))
|
||||
for key := range r.specs {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
|
||||
keys := r.RegisteredKeys()
|
||||
specs := make([]ValidatorSpec, 0, len(keys))
|
||||
for _, key := range keys {
|
||||
specs = append(specs, r.specs[key])
|
||||
if spec, ok := r.Spec(key); ok {
|
||||
specs = append(specs, spec)
|
||||
}
|
||||
}
|
||||
return specs
|
||||
}
|
||||
@@ -128,15 +295,21 @@ func (r *ValidatorRegistry) RegisteredKeys() []string {
|
||||
if r == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
return sortedRegistryKeys(r.constructors)
|
||||
keys := make(map[string]struct{})
|
||||
for key := range r.typedEntries {
|
||||
keys[key.module] = struct{}{}
|
||||
}
|
||||
for key := range r.chunkEntries {
|
||||
keys[key] = struct{}{}
|
||||
}
|
||||
for key := range r.serializedEntries {
|
||||
keys[key] = struct{}{}
|
||||
}
|
||||
return sortedRegistryKeys(keys)
|
||||
}
|
||||
|
||||
func normalizeValidatorSpec(spec ValidatorSpec) (ValidatorSpec, error) {
|
||||
normalized := ValidatorSpec{
|
||||
Key: strings.TrimSpace(spec.Key),
|
||||
ExecutionClass: spec.ExecutionClass,
|
||||
}
|
||||
normalized := ValidatorSpec{Key: strings.TrimSpace(spec.Key), ExecutionClass: spec.ExecutionClass}
|
||||
if normalized.Key == "" {
|
||||
return ValidatorSpec{}, fmt.Errorf("validator key must not be empty")
|
||||
}
|
||||
@@ -147,3 +320,7 @@ func normalizeValidatorSpec(spec ValidatorSpec) (ValidatorSpec, error) {
|
||||
}
|
||||
return normalized, nil
|
||||
}
|
||||
|
||||
func sortValidatorSpecs(specs []ValidatorSpec) {
|
||||
sort.Slice(specs, func(i, j int) bool { return specs[i].Key < specs[j].Key })
|
||||
}
|
||||
|
||||
@@ -1,117 +0,0 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func TestValidatorRegistryBehavior(t *testing.T) {
|
||||
registry := NewValidatorRegistry()
|
||||
if err := registry.Register(" generic-validator ", validatorConstructor("generic-validator", contracts.ExecutionClassDeterministic)); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
validator, err := registry.Build("generic-validator")
|
||||
if err != nil {
|
||||
t.Fatalf("Build() error = %v, want nil", err)
|
||||
}
|
||||
if validator.Name() != "generic-validator" {
|
||||
t.Fatalf("validator name = %q, want generic-validator", validator.Name())
|
||||
}
|
||||
|
||||
spec, ok := registry.Spec(" generic-validator ")
|
||||
if !ok {
|
||||
t.Fatal("Spec() ok = false, want true")
|
||||
}
|
||||
want := ValidatorSpec{Key: "generic-validator", ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||
if !reflect.DeepEqual(spec, want) {
|
||||
t.Fatalf("Spec() = %#v, want %#v", spec, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidatorRegistryRegistersSpecs(t *testing.T) {
|
||||
registry := NewValidatorRegistry()
|
||||
spec := ValidatorSpec{Key: " llm-validator ", ExecutionClass: contracts.ExecutionClassLLMBacked}
|
||||
if err := registry.RegisterWithSpec(spec, validatorConstructor("llm-validator", contracts.ExecutionClassLLMBacked)); err != nil {
|
||||
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
got, ok := registry.Spec("llm-validator")
|
||||
if !ok {
|
||||
t.Fatal("Spec() ok = false, want true")
|
||||
}
|
||||
want := ValidatorSpec{Key: "llm-validator", ExecutionClass: contracts.ExecutionClassLLMBacked}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidatorRegistryRegisteredSpecsAreSorted(t *testing.T) {
|
||||
registry := NewValidatorRegistry()
|
||||
for _, key := range []string{"zeta", "alpha"} {
|
||||
if err := registry.Register(key, validatorConstructor(key, contracts.ExecutionClassDeterministic)); err != nil {
|
||||
t.Fatalf("Register(%q) error = %v", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
specs := registry.RegisteredSpecs()
|
||||
if len(specs) != 2 || specs[0].Key != "alpha" || specs[1].Key != "zeta" {
|
||||
t.Fatalf("RegisteredSpecs() = %#v, want sorted specs", specs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidatorRegistryRejectsUnsupportedExecutionClass(t *testing.T) {
|
||||
registry := NewValidatorRegistry()
|
||||
err := registry.RegisterWithSpec(
|
||||
ValidatorSpec{Key: "invalid-validator", ExecutionClass: contracts.ExecutionClass("unsupported")},
|
||||
validatorConstructor("invalid-validator", contracts.ExecutionClass("unsupported")),
|
||||
)
|
||||
if err == nil {
|
||||
t.Fatal("RegisterWithSpec() error = nil, want unsupported execution class error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidatorRegistryRejectsConstructorExecutionClassMismatch(t *testing.T) {
|
||||
registry := NewValidatorRegistry()
|
||||
if err := registry.RegisterWithSpec(
|
||||
ValidatorSpec{Key: "validator", ExecutionClass: contracts.ExecutionClassDeterministic},
|
||||
validatorConstructor("validator", contracts.ExecutionClassLLMBacked),
|
||||
); err != nil {
|
||||
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
_, err := registry.Build("validator")
|
||||
if err == nil {
|
||||
t.Fatal("Build() error = nil, want execution class mismatch")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "execution class") {
|
||||
t.Fatalf("Build() error = %q, want execution class context", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
type testValidator struct {
|
||||
name string
|
||||
executionClass contracts.ExecutionClass
|
||||
}
|
||||
|
||||
func validatorConstructor(name string, executionClass contracts.ExecutionClass) ValidatorConstructor {
|
||||
return func() (contracts.Validator, error) {
|
||||
return testValidator{name: name, executionClass: executionClass}, nil
|
||||
}
|
||||
}
|
||||
|
||||
func (validator testValidator) Name() string {
|
||||
return validator.name
|
||||
}
|
||||
|
||||
func (validator testValidator) ExecutionClass() contracts.ExecutionClass {
|
||||
return validator.executionClass
|
||||
}
|
||||
|
||||
func (validator testValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: true}, nil
|
||||
}
|
||||
@@ -1,462 +0,0 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func TestWalkingSkeletonFixture(t *testing.T) {
|
||||
inputBytes := readTestFixture(t, "testdata/walking_skeleton_input.json")
|
||||
expectedBytes := readTestFixture(t, "testdata/walking_skeleton_output.json")
|
||||
llmClient := &walkingSkeletonLLMClient{}
|
||||
|
||||
resolved, err := ResolvePipeline(walkingSkeletonProfile(), ResolveOptions{}, walkingSkeletonCatalog(t))
|
||||
if err != nil {
|
||||
t.Fatalf("ResolvePipeline() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
output, err := New(walkingSkeletonRegistries(t)).Run(context.Background(), RunInput{
|
||||
Pipeline: resolved,
|
||||
SourceID: "fixture-source",
|
||||
Path: "walking_skeleton_input.json",
|
||||
RawInput: inputBytes,
|
||||
LLMClient: llmClient,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v, want nil", err)
|
||||
}
|
||||
if len(output.OutputFiles) != 1 {
|
||||
t.Fatalf("len(OutputFiles) = %d, want 1", len(output.OutputFiles))
|
||||
}
|
||||
if output.OutputFiles[0].ContentType != "application/json" {
|
||||
t.Fatalf("ContentType = %q, want application/json", output.OutputFiles[0].ContentType)
|
||||
}
|
||||
assertStructuralJSONEqual(t, output.OutputFiles[0].Bytes, expectedBytes)
|
||||
if llmClient.calls != 3 {
|
||||
t.Fatalf("LLM calls = %d, want extractor calls plus normalizer call", llmClient.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWalkingSkeletonResolutionRejectsMissingCapability(t *testing.T) {
|
||||
catalog := walkingSkeletonCatalog(t)
|
||||
catalog.Extractors = NewExtractorRegistry()
|
||||
if err := catalog.Extractors.RegisterWithSpec(ModuleSpec{
|
||||
Key: "fake/extract",
|
||||
Stage: StageExtract,
|
||||
Requires: []string{"missing"},
|
||||
Provides: []string{"fake_artifacts"},
|
||||
}, func() (contracts.Extractor, error) {
|
||||
return walkingSkeletonExtractor{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterWithSpec() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
_, err := ResolvePipeline(walkingSkeletonProfile(), ResolveOptions{}, catalog)
|
||||
if err == nil {
|
||||
t.Fatal("ResolvePipeline() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "missing") {
|
||||
t.Fatalf("ResolvePipeline() error = %q, want missing capability", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func TestWalkingSkeletonResolutionRejectsUnknownOnlyLane(t *testing.T) {
|
||||
_, err := ResolvePipeline(walkingSkeletonProfile(), ResolveOptions{Only: []string{"missing"}}, walkingSkeletonCatalog(t))
|
||||
if err == nil {
|
||||
t.Fatal("ResolvePipeline() error = nil, want error")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "missing") || !strings.Contains(err.Error(), "not declared") {
|
||||
t.Fatalf("ResolvePipeline() error = %q, want unknown lane error", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func walkingSkeletonProfile() PipelineProfile {
|
||||
return PipelineProfile{
|
||||
ID: "walking-skeleton",
|
||||
Input: Binding("fake/input"),
|
||||
Chunk: Binding("fake/chunk"),
|
||||
Output: Binding("json"),
|
||||
Artifacts: map[string]ArtifactLaneProfile{
|
||||
"events": {Extract: Binding("fake/extract")},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func walkingSkeletonCatalog(t *testing.T) ModuleCatalog {
|
||||
t.Helper()
|
||||
|
||||
catalog := ModuleCatalog{
|
||||
Inputs: NewInputAdapterRegistry(),
|
||||
Chunkers: NewChunkerRegistry(),
|
||||
Extractors: NewExtractorRegistry(),
|
||||
Mergers: NewMergerRegistry(),
|
||||
Normalizers: NewNormalizerRegistry(),
|
||||
ValidatorChains: NewValidatorChainRegistry(),
|
||||
Outputs: NewOutputEncoderRegistry(),
|
||||
}
|
||||
if err := catalog.Inputs.RegisterWithSpec(ModuleSpec{
|
||||
Key: "fake/input",
|
||||
Stage: StageInput,
|
||||
Provides: []string{"plain_text"},
|
||||
}, func() (contracts.InputAdapter, error) {
|
||||
return walkingSkeletonInput{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register fake input: %v", err)
|
||||
}
|
||||
if err := catalog.Chunkers.RegisterWithSpec(ModuleSpec{
|
||||
Key: "fake/chunk",
|
||||
Stage: StageChunk,
|
||||
Requires: []string{"plain_text"},
|
||||
Provides: []string{"chunks"},
|
||||
}, func() (contracts.Chunker, error) {
|
||||
return walkingSkeletonChunker{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register fake chunker: %v", err)
|
||||
}
|
||||
if err := catalog.Extractors.RegisterWithSpec(ModuleSpec{
|
||||
Key: "fake/extract",
|
||||
Stage: StageExtract,
|
||||
Requires: []string{"chunks"},
|
||||
Provides: []string{"fake_artifacts"},
|
||||
}, func() (contracts.Extractor, error) {
|
||||
return walkingSkeletonExtractor{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register fake extractor: %v", err)
|
||||
}
|
||||
if err := catalog.Mergers.RegisterWithSpec(ModuleSpec{
|
||||
Key: DefaultMergeModule,
|
||||
Stage: StageMerge,
|
||||
Requires: []string{"fake_artifacts"},
|
||||
}, func() (contracts.Merger, error) {
|
||||
return walkingSkeletonMerger{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register append-order merger: %v", err)
|
||||
}
|
||||
if err := catalog.Normalizers.RegisterWithSpec(ModuleSpec{
|
||||
Key: DefaultNormalizeModule,
|
||||
Stage: StageNormalize,
|
||||
}, func() (contracts.Normalizer, error) {
|
||||
return walkingSkeletonNormalizer{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register no-op normalizer: %v", err)
|
||||
}
|
||||
if err := catalog.Outputs.RegisterWithSpec(ModuleSpec{
|
||||
Key: "json",
|
||||
Stage: StageOutput,
|
||||
}, func() (contracts.OutputEncoder, error) {
|
||||
return walkingSkeletonOutput{}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("register fake output: %v", err)
|
||||
}
|
||||
return catalog
|
||||
}
|
||||
|
||||
func walkingSkeletonRegistries(t *testing.T) Registries {
|
||||
t.Helper()
|
||||
|
||||
catalog := walkingSkeletonCatalog(t)
|
||||
return Registries{
|
||||
Inputs: catalog.Inputs,
|
||||
Chunkers: catalog.Chunkers,
|
||||
Extractors: catalog.Extractors,
|
||||
Mergers: catalog.Mergers,
|
||||
Normalizers: catalog.Normalizers,
|
||||
Outputs: catalog.Outputs,
|
||||
}
|
||||
}
|
||||
|
||||
type walkingSkeletonInput struct{}
|
||||
|
||||
func (input walkingSkeletonInput) Key() string {
|
||||
return "fake/input"
|
||||
}
|
||||
|
||||
func (input walkingSkeletonInput) Parse(ctx context.Context, req contracts.ParseRequest) (*source.SourceDocument, error) {
|
||||
var fixture struct {
|
||||
ID string `json:"id"`
|
||||
Units []struct {
|
||||
ID int `json:"id"`
|
||||
Text string `json:"text"`
|
||||
} `json:"units"`
|
||||
}
|
||||
if err := json.Unmarshal(req.Raw, &fixture); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
units := make([]source.SourceUnit, 0, len(fixture.Units))
|
||||
for _, unit := range fixture.Units {
|
||||
units = append(units, source.SourceUnit{
|
||||
ID: unit.ID,
|
||||
Kind: "unit",
|
||||
Text: unit.Text,
|
||||
})
|
||||
}
|
||||
return &source.SourceDocument{
|
||||
ID: fixture.ID,
|
||||
Kind: "fixture",
|
||||
Format: "application/json",
|
||||
Digest: rawDigest(req.Raw),
|
||||
Units: units,
|
||||
}, nil
|
||||
}
|
||||
|
||||
type walkingSkeletonChunker struct{}
|
||||
|
||||
func (chunker walkingSkeletonChunker) Key() string {
|
||||
return "fake/chunk"
|
||||
}
|
||||
|
||||
func (chunker walkingSkeletonChunker) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (chunker walkingSkeletonChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
|
||||
if len(req.Source.Units) < 3 {
|
||||
return contracts.ChunkResult{}, fmt.Errorf("fixture source must contain at least three units")
|
||||
}
|
||||
return contracts.ChunkResult{
|
||||
Chunks: []contracts.SourceChunk{
|
||||
{
|
||||
ID: req.Source.ID + ":chunk:0",
|
||||
SourceID: req.Source.ID,
|
||||
Index: 0,
|
||||
StartUnitID: req.Source.Units[0].ID,
|
||||
EndUnitID: req.Source.Units[1].ID,
|
||||
Content: []byte(`{"units":[1,2]}`),
|
||||
MediaType: "application/json",
|
||||
Units: append([]source.SourceUnit(nil), req.Source.Units[:2]...),
|
||||
},
|
||||
{
|
||||
ID: req.Source.ID + ":chunk:1",
|
||||
SourceID: req.Source.ID,
|
||||
Index: 1,
|
||||
StartUnitID: req.Source.Units[2].ID,
|
||||
EndUnitID: req.Source.Units[len(req.Source.Units)-1].ID,
|
||||
Content: []byte(`{"units":[3]}`),
|
||||
MediaType: "application/json",
|
||||
Units: append([]source.SourceUnit(nil), req.Source.Units[2:]...),
|
||||
},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type walkingSkeletonExtractor struct{}
|
||||
|
||||
func (extractor walkingSkeletonExtractor) Key() string {
|
||||
return "fake/extract"
|
||||
}
|
||||
|
||||
func (extractor walkingSkeletonExtractor) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (extractor walkingSkeletonExtractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
|
||||
var response struct {
|
||||
Call int `json:"call"`
|
||||
}
|
||||
if _, err := req.LLMClient.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: "fake/extract",
|
||||
PromptID: "fake.event",
|
||||
PromptVersion: "v1",
|
||||
}, &response); err != nil {
|
||||
return contracts.ExtractionResult{}, err
|
||||
}
|
||||
|
||||
payload, err := json.Marshal(map[string]any{
|
||||
"chunk_id": req.Chunk.ID,
|
||||
"llm_call": response.Call,
|
||||
"text": chunkText(req.Chunk.Units),
|
||||
})
|
||||
if err != nil {
|
||||
return contracts.ExtractionResult{}, err
|
||||
}
|
||||
|
||||
return contracts.ExtractionResult{
|
||||
Output: contracts.ExtractOutput{
|
||||
Schema: contracts.ResponseSchema{ID: "fake_event", Name: "fake_event", Version: "v1"},
|
||||
Payload: contracts.RawPayload{
|
||||
Content: payload,
|
||||
MediaType: "application/json",
|
||||
},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type walkingSkeletonLLMClient struct {
|
||||
calls int
|
||||
}
|
||||
|
||||
func (client *walkingSkeletonLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
client.calls++
|
||||
if response, ok := out.(*struct {
|
||||
Call int `json:"call"`
|
||||
}); ok {
|
||||
response.Call = client.calls
|
||||
}
|
||||
content, err := json.Marshal(map[string]any{"call": client.calls})
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{
|
||||
Content: content,
|
||||
}, nil
|
||||
}
|
||||
|
||||
type walkingSkeletonMerger struct{}
|
||||
|
||||
func (merger walkingSkeletonMerger) Key() string {
|
||||
return DefaultMergeModule
|
||||
}
|
||||
|
||||
func (merger walkingSkeletonMerger) Merge(ctx context.Context, req contracts.MergeRequest) (contracts.MergeResult, error) {
|
||||
outputs := make([]json.RawMessage, 0, len(req.ExtractOutputs))
|
||||
for _, output := range req.ExtractOutputs {
|
||||
outputs = append(outputs, json.RawMessage(output.Payload.Content))
|
||||
}
|
||||
content, err := json.Marshal(map[string]any{"outputs": outputs})
|
||||
if err != nil {
|
||||
return contracts.MergeResult{}, err
|
||||
}
|
||||
return contracts.MergeResult{
|
||||
Output: contracts.MergeOutput{
|
||||
LaneID: req.LaneID,
|
||||
SourceID: req.Source.ID,
|
||||
Schema: contracts.ResponseSchema{ID: "fake_event", Name: "fake_event", Version: "v1"},
|
||||
Payload: contracts.RawPayload{
|
||||
Content: content,
|
||||
MediaType: "application/json",
|
||||
},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type walkingSkeletonNormalizer struct{}
|
||||
|
||||
func (normalizer walkingSkeletonNormalizer) Key() string {
|
||||
return DefaultNormalizeModule
|
||||
}
|
||||
|
||||
func (normalizer walkingSkeletonNormalizer) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (normalizer walkingSkeletonNormalizer) Normalize(ctx context.Context, req contracts.NormalizeRequest) (contracts.NormalizeResult, error) {
|
||||
var response struct {
|
||||
Call int `json:"call"`
|
||||
}
|
||||
if _, err := req.LLMClient.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: "fake/normalize",
|
||||
PromptID: "fake.normalize",
|
||||
PromptVersion: "v1",
|
||||
}, &response); err != nil {
|
||||
return contracts.NormalizeResult{}, err
|
||||
}
|
||||
return contracts.NormalizeResult{
|
||||
Output: contracts.NormalizeOutput{
|
||||
LaneID: req.LaneID,
|
||||
SourceID: req.MergeOutput.SourceID,
|
||||
Schema: req.MergeOutput.Schema,
|
||||
Payload: req.MergeOutput.Payload,
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
type walkingSkeletonOutput struct{}
|
||||
|
||||
func (output walkingSkeletonOutput) Key() string {
|
||||
return "json"
|
||||
}
|
||||
|
||||
func (output walkingSkeletonOutput) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
|
||||
type rawOutput struct {
|
||||
LaneID string `json:"lane_id"`
|
||||
NormalizerKey string `json:"normalizer_key"`
|
||||
SourceID string `json:"source_id"`
|
||||
Schema contracts.ResponseSchema `json:"schema"`
|
||||
MediaType string `json:"media_type"`
|
||||
Content json.RawMessage `json:"content"`
|
||||
}
|
||||
rawOutputs := make([]rawOutput, 0, len(req.NormalizeOutputs))
|
||||
for _, output := range req.NormalizeOutputs {
|
||||
rawOutputs = append(rawOutputs, rawOutput{
|
||||
LaneID: output.LaneID,
|
||||
NormalizerKey: output.NormalizerKey,
|
||||
SourceID: output.SourceID,
|
||||
Schema: output.Schema,
|
||||
MediaType: output.Payload.MediaType,
|
||||
Content: json.RawMessage(output.Payload.Content),
|
||||
})
|
||||
}
|
||||
encoded, err := json.Marshal(struct {
|
||||
Manifest artifacts.RunManifest `json:"manifest"`
|
||||
NormalizeOutputs []rawOutput `json:"normalize_outputs"`
|
||||
}{
|
||||
Manifest: artifacts.RunManifest{
|
||||
PipelineID: req.Manifest.PipelineID,
|
||||
PipelineDigest: req.Manifest.PipelineDigest,
|
||||
ArtifactLanes: req.Manifest.ArtifactLanes,
|
||||
ValidationStatus: req.Manifest.ValidationStatus,
|
||||
},
|
||||
NormalizeOutputs: rawOutputs,
|
||||
})
|
||||
if err != nil {
|
||||
return contracts.OutputResult{}, err
|
||||
}
|
||||
return contracts.OutputResult{
|
||||
Files: []contracts.OutputFile{
|
||||
{Name: "output.json", ContentType: "application/json", Bytes: encoded},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func readTestFixture(t *testing.T, path string) []byte {
|
||||
t.Helper()
|
||||
|
||||
bytes, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("read fixture %q: %v", path, err)
|
||||
}
|
||||
return bytes
|
||||
}
|
||||
|
||||
func assertStructuralJSONEqual(t *testing.T, gotBytes, wantBytes []byte) {
|
||||
t.Helper()
|
||||
|
||||
var got any
|
||||
if err := json.Unmarshal(gotBytes, &got); err != nil {
|
||||
t.Fatalf("unmarshal actual JSON: %v\n%s", err, gotBytes)
|
||||
}
|
||||
var want any
|
||||
if err := json.Unmarshal(wantBytes, &want); err != nil {
|
||||
t.Fatalf("unmarshal expected JSON: %v\n%s", err, wantBytes)
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
gotFormatted, _ := json.MarshalIndent(got, "", " ")
|
||||
wantFormatted, _ := json.MarshalIndent(want, "", " ")
|
||||
t.Fatalf("actual JSON:\n%s\nwant:\n%s", gotFormatted, wantFormatted)
|
||||
}
|
||||
}
|
||||
|
||||
func chunkText(units []source.SourceUnit) string {
|
||||
parts := make([]string, 0, len(units))
|
||||
for _, unit := range units {
|
||||
parts = append(parts, unit.Text)
|
||||
}
|
||||
return strings.Join(parts, " ")
|
||||
}
|
||||
|
||||
func rawDigest(raw []byte) string {
|
||||
sum := sha256.Sum256(raw)
|
||||
return "sha256:" + hex.EncodeToString(sum[:])
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package sharedassets
|
||||
package promptfs
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
@@ -1,4 +1,4 @@
|
||||
package sharedassets
|
||||
package promptfs
|
||||
|
||||
import (
|
||||
"io/fs"
|
||||
@@ -1,30 +0,0 @@
|
||||
package scenes
|
||||
|
||||
import "gitea.maximumdirect.net/eric/notarius/internal/modules/sharedassets/dnd"
|
||||
|
||||
type chunkResponse struct {
|
||||
Scenes []sceneResponse `json:"scenes"`
|
||||
BoundaryCaveats []string `json:"boundary_caveats"`
|
||||
}
|
||||
|
||||
type sceneResponse struct {
|
||||
StartUnitID dnd.UnitRef `json:"start_unit_id"`
|
||||
EndUnitID dnd.UnitRef `json:"end_unit_id"`
|
||||
ShortTitle string `json:"short_title"`
|
||||
PrimaryMode string `json:"primary_mode"`
|
||||
MainParticipants []string `json:"main_participants"`
|
||||
Summary string `json:"summary"`
|
||||
BoundaryNote string `json:"boundary_note"`
|
||||
BoundaryConfidence string `json:"boundary_confidence"`
|
||||
}
|
||||
|
||||
type normalizedScene struct {
|
||||
StartUnitID int
|
||||
EndUnitID int
|
||||
ShortTitle string
|
||||
PrimaryMode string
|
||||
MainParticipants []string
|
||||
Summary string
|
||||
BoundaryNote string
|
||||
BoundaryConfidence string
|
||||
}
|
||||
@@ -9,7 +9,7 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/sharedassets/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
)
|
||||
|
||||
const Key = "dnd/scenes"
|
||||
@@ -23,7 +23,7 @@ var providedCapabilities = []string{
|
||||
"chunks.scenes",
|
||||
}
|
||||
|
||||
var referenceSlotDescriptions = dnd.ReferenceSlotDescriptions{
|
||||
var referenceSlotDescriptions = shared.ReferenceSlotDescriptions{
|
||||
Glossary: "Optional campaign glossary reference material used only for scene disambiguation.",
|
||||
Party: "Optional party roster reference material used only for scene disambiguation.",
|
||||
Players: "Optional player list reference material used only for scene disambiguation.",
|
||||
@@ -33,10 +33,17 @@ var referenceSlotDescriptions = dnd.ReferenceSlotDescriptions{
|
||||
var _ contracts.Chunker = (*Chunker)(nil)
|
||||
var _ contracts.ManifestMetadataProvider = (*Chunker)(nil)
|
||||
|
||||
type Chunker struct{}
|
||||
type Options struct{}
|
||||
|
||||
func New() *Chunker {
|
||||
return &Chunker{}
|
||||
type Chunker struct {
|
||||
llm contracts.StructuredLLMClient
|
||||
}
|
||||
|
||||
func New(llmClient contracts.StructuredLLMClient, _ Options) (*Chunker, error) {
|
||||
if llmClient == nil {
|
||||
return nil, chunkerErrorf("LLM client must not be nil")
|
||||
}
|
||||
return &Chunker{llm: llmClient}, nil
|
||||
}
|
||||
|
||||
func (c *Chunker) Key() string {
|
||||
@@ -44,7 +51,7 @@ func (c *Chunker) Key() string {
|
||||
}
|
||||
|
||||
func (c *Chunker) ReferenceSlots() []contracts.ReferenceSlot {
|
||||
return dnd.ReferenceSlots(referenceSlotDescriptions)
|
||||
return shared.ReferenceSlots(referenceSlotDescriptions)
|
||||
}
|
||||
|
||||
func (c *Chunker) ManifestMetadata() map[string]any {
|
||||
@@ -71,6 +78,9 @@ func (c *Chunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contra
|
||||
if c == nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("chunker must not be nil")
|
||||
}
|
||||
if c.llm == nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("LLM client must not be nil")
|
||||
}
|
||||
if ctx == nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("context must not be nil")
|
||||
}
|
||||
@@ -86,21 +96,14 @@ func (c *Chunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contra
|
||||
if err := source.ValidateDocument(req.Source); err != nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("validate source document: %w", err)
|
||||
}
|
||||
if req.LLMClient == nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("LLM client must not be nil")
|
||||
}
|
||||
if len(req.Options) > 0 {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("options are not supported")
|
||||
}
|
||||
|
||||
var response chunkResponse
|
||||
if _, err := req.LLMClient.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
if _, err := c.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: Key,
|
||||
PromptID: PromptID,
|
||||
PromptVersion: ResponseSchemaVersion,
|
||||
ProfileID: req.LLMProfile,
|
||||
SessionID: req.SessionID,
|
||||
Inputs: dnd.PromptInputs(req.SourceInput, req.References),
|
||||
Inputs: shared.PromptInputs(req.SourceInput, req.References),
|
||||
}, &response); err != nil {
|
||||
return contracts.ChunkResult{}, chunkerErrorf("complete structured output: %w", err)
|
||||
}
|
||||
@@ -125,17 +128,33 @@ func ModuleSpec() pipeline.ModuleSpec {
|
||||
Stage: pipeline.StageChunk,
|
||||
Requires: append([]string(nil), requiredCapabilities...),
|
||||
Provides: append([]string(nil), providedCapabilities...),
|
||||
ReferenceSlots: dnd.ReferenceSlots(referenceSlotDescriptions),
|
||||
ReferenceSlots: shared.ReferenceSlots(referenceSlotDescriptions),
|
||||
}
|
||||
}
|
||||
|
||||
func Register(registry *pipeline.ChunkerRegistry) error {
|
||||
return registry.RegisterWithSpec(ModuleSpec(), func() (contracts.Chunker, error) {
|
||||
return New(), nil
|
||||
return registry.RegisterBuilderWithSpec(ModuleSpec(), validateOptions, func(request pipeline.BuildRequest) (contracts.Chunker, error) {
|
||||
options, err := DecodeOptions(request.Options)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return New(request.Dependencies.LLM, options)
|
||||
})
|
||||
}
|
||||
|
||||
func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]contracts.SourceChunk, error) {
|
||||
func validateOptions(options map[string]any) error {
|
||||
_, err := DecodeOptions(options)
|
||||
return err
|
||||
}
|
||||
|
||||
func DecodeOptions(options map[string]any) (Options, error) {
|
||||
if err := pipeline.RejectUnknownOptions(options); err != nil {
|
||||
return Options{}, chunkerErrorf("%w", err)
|
||||
}
|
||||
return Options{}, nil
|
||||
}
|
||||
|
||||
func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]source.Chunk, error) {
|
||||
if response.Scenes == nil {
|
||||
return nil, fmt.Errorf("scenes must be present")
|
||||
}
|
||||
@@ -148,7 +167,7 @@ func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]c
|
||||
unitIndexes[unit.ID] = i
|
||||
}
|
||||
|
||||
chunks := make([]contracts.SourceChunk, 0, len(response.Scenes))
|
||||
chunks := make([]source.Chunk, 0, len(response.Scenes))
|
||||
previousEnd := -1
|
||||
for i, scene := range response.Scenes {
|
||||
normalized, err := normalizeScene(doc, i, scene)
|
||||
@@ -186,15 +205,18 @@ func chunksFromResponse(doc *source.SourceDocument, response chunkResponse) ([]c
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
chunks = append(chunks, contracts.SourceChunk{
|
||||
ID: fmt.Sprintf("scene-%06d", i+1),
|
||||
SourceID: doc.ID,
|
||||
Index: i,
|
||||
StartUnitID: units[0].ID,
|
||||
EndUnitID: units[len(units)-1].ID,
|
||||
Content: content,
|
||||
MediaType: "application/json",
|
||||
Units: units,
|
||||
chunks = append(chunks, source.Chunk{
|
||||
ID: fmt.Sprintf("scene-%06d", i+1),
|
||||
SourceID: doc.ID,
|
||||
Index: i,
|
||||
Ref: source.SourceRef{
|
||||
SourceID: doc.ID,
|
||||
StartUnitID: units[0].Ref.StartUnitID,
|
||||
EndUnitID: units[len(units)-1].Ref.EndUnitID,
|
||||
},
|
||||
Content: content,
|
||||
MediaType: "application/json",
|
||||
Units: units,
|
||||
Metadata: map[string]any{
|
||||
"scene_title": normalized.ShortTitle,
|
||||
"primary_mode": normalized.PrimaryMode,
|
||||
@@ -228,11 +250,11 @@ func chunkContent(units []source.SourceUnit) ([]byte, error) {
|
||||
}
|
||||
|
||||
func normalizeScene(doc *source.SourceDocument, index int, scene sceneResponse) (normalizedScene, error) {
|
||||
startUnitID, err := dnd.ResolveUnitID(doc, "start_unit_id", scene.StartUnitID)
|
||||
startUnitID, err := shared.ResolveUnitID(doc, "start_unit_id", scene.StartUnitID)
|
||||
if err != nil {
|
||||
return normalizedScene{}, fmt.Errorf("scene[%d] %w", index, err)
|
||||
}
|
||||
endUnitID, err := dnd.ResolveUnitID(doc, "end_unit_id", scene.EndUnitID)
|
||||
endUnitID, err := shared.ResolveUnitID(doc, "end_unit_id", scene.EndUnitID)
|
||||
if err != nil {
|
||||
return normalizedScene{}, fmt.Errorf("scene[%d] %w", index, err)
|
||||
}
|
||||
@@ -332,6 +354,7 @@ func cloneUnits(units []source.SourceUnit) []source.SourceUnit {
|
||||
ID: unit.ID,
|
||||
Kind: unit.Kind,
|
||||
Text: unit.Text,
|
||||
Ref: unit.Ref,
|
||||
Metadata: cloneMetadata(unit.Metadata),
|
||||
})
|
||||
}
|
||||
@@ -11,14 +11,12 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/sharedassets/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
)
|
||||
|
||||
func TestNewModuleSpecAndRegister(t *testing.T) {
|
||||
chunker := New()
|
||||
if chunker == nil {
|
||||
t.Fatal("New() = nil, want chunker")
|
||||
}
|
||||
client := &fakeScenesLLMClient{}
|
||||
chunker := newChunker(t, client)
|
||||
if chunker.Key() != Key {
|
||||
t.Fatalf("Key() = %q, want %q", chunker.Key(), Key)
|
||||
}
|
||||
@@ -52,7 +50,7 @@ func TestNewModuleSpecAndRegister(t *testing.T) {
|
||||
if !reflect.DeepEqual(registered, want) {
|
||||
t.Fatalf("registered spec = %#v, want %#v", registered, want)
|
||||
}
|
||||
built, err := registry.Build(Key)
|
||||
built, err := registry.BuildWithRequest(Key, pipeline.BuildRequest{Dependencies: pipeline.ModuleDependencies{LLM: client}})
|
||||
if err != nil {
|
||||
t.Fatalf("Build(%q) error = %v, want nil", Key, err)
|
||||
}
|
||||
@@ -62,6 +60,18 @@ func TestNewModuleSpecAndRegister(t *testing.T) {
|
||||
if slots := built.ReferenceSlots(); !reflect.DeepEqual(slots, want.ReferenceSlots) {
|
||||
t.Fatalf("ReferenceSlots() = %#v, want %#v", slots, want.ReferenceSlots)
|
||||
}
|
||||
if err := registry.ValidateOptions(Key, map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||
t.Fatalf("ValidateOptions() error = %v, want unknown option error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConstructionRejectsMissingDependencyAndUnknownOptions(t *testing.T) {
|
||||
if _, err := New(nil, Options{}); err == nil || !strings.Contains(err.Error(), "LLM client") {
|
||||
t.Fatalf("New(nil) error = %v, want LLM client error", err)
|
||||
}
|
||||
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||
t.Fatalf("DecodeOptions() error = %v, want unknown option error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegisterNilRegistryReturnsError(t *testing.T) {
|
||||
@@ -105,8 +115,8 @@ func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
|
||||
response: chunkResponse{
|
||||
Scenes: []sceneResponse{
|
||||
{
|
||||
StartUnitID: dnd.UnitRefFromInt(1),
|
||||
EndUnitID: dnd.UnitRefFromInt(2),
|
||||
StartUnitID: shared.UnitRefFromInt(1),
|
||||
EndUnitID: shared.UnitRefFromInt(2),
|
||||
ShortTitle: " Goblin parley ",
|
||||
PrimaryMode: "Discussion",
|
||||
MainParticipants: []string{" Aria ", "Goblin scout"},
|
||||
@@ -115,8 +125,8 @@ func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
|
||||
BoundaryConfidence: "High",
|
||||
},
|
||||
{
|
||||
StartUnitID: dnd.UnitRefFromInt(3),
|
||||
EndUnitID: dnd.UnitRefFromInt(4),
|
||||
StartUnitID: shared.UnitRefFromInt(3),
|
||||
EndUnitID: shared.UnitRefFromInt(4),
|
||||
ShortTitle: "Ambush at the gate",
|
||||
PrimaryMode: "Combat",
|
||||
MainParticipants: []string{"Aria", "Goblin ambushers"},
|
||||
@@ -129,7 +139,7 @@ func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
|
||||
},
|
||||
}
|
||||
|
||||
result, err := New().Chunk(context.Background(), chunkRequestWithClient(client))
|
||||
result, err := newChunker(t, client).Chunk(context.Background(), chunkRequest())
|
||||
if err != nil {
|
||||
t.Fatalf("Chunk() error = %v, want nil", err)
|
||||
}
|
||||
@@ -179,8 +189,8 @@ func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
|
||||
if first.SourceID != "session-alpha" || first.Index != 0 {
|
||||
t.Fatalf("first chunk = %#v, want source and index fields", first)
|
||||
}
|
||||
if first.StartUnitID != 1 || first.EndUnitID != 2 {
|
||||
t.Fatalf("first boundaries = %d-%d, want 1-2", first.StartUnitID, first.EndUnitID)
|
||||
if first.Ref != (source.SourceRef{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 2}) {
|
||||
t.Fatalf("first ref = %#v, want session-alpha:1-2", first.Ref)
|
||||
}
|
||||
if first.MediaType != "application/json" || len(first.Content) == 0 {
|
||||
t.Fatalf("first payload = media type %q length %d, want JSON content", first.MediaType, len(first.Content))
|
||||
@@ -210,8 +220,8 @@ func TestChunkPassesReferencesAsPromptInputs(t *testing.T) {
|
||||
client := &fakeScenesLLMClient{response: chunkResponse{
|
||||
Scenes: []sceneResponse{
|
||||
{
|
||||
StartUnitID: dnd.UnitRefFromInt(1),
|
||||
EndUnitID: dnd.UnitRefFromInt(4),
|
||||
StartUnitID: shared.UnitRefFromInt(1),
|
||||
EndUnitID: shared.UnitRefFromInt(4),
|
||||
ShortTitle: "Ambush",
|
||||
PrimaryMode: "Combat",
|
||||
MainParticipants: []string{"Aria"},
|
||||
@@ -221,7 +231,7 @@ func TestChunkPassesReferencesAsPromptInputs(t *testing.T) {
|
||||
},
|
||||
},
|
||||
}}
|
||||
req := chunkRequestWithClient(client)
|
||||
req := chunkRequest()
|
||||
req.References = contracts.ReferenceSet{
|
||||
Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||
"players": {
|
||||
@@ -245,7 +255,7 @@ func TestChunkPassesReferencesAsPromptInputs(t *testing.T) {
|
||||
},
|
||||
}
|
||||
|
||||
if _, err := New().Chunk(context.Background(), req); err != nil {
|
||||
if _, err := newChunker(t, client).Chunk(context.Background(), req); err != nil {
|
||||
t.Fatalf("Chunk() error = %v, want nil", err)
|
||||
}
|
||||
request := client.requests[0]
|
||||
@@ -264,7 +274,7 @@ func TestChunkPassesReferencesAsPromptInputs(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestPromptInputsMapLegacyRosterReferenceToParty(t *testing.T) {
|
||||
inputs := dnd.PromptInputs(sceneSourceInput(), contracts.ReferenceSet{
|
||||
inputs := shared.PromptInputs(sceneSourceInput(), contracts.ReferenceSet{
|
||||
Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||
"roster": {
|
||||
Slot: contracts.ReferenceSlot{Name: "roster"},
|
||||
@@ -293,7 +303,7 @@ func TestChunkRejectsWhitespaceOnlyBoundaryCaveats(t *testing.T) {
|
||||
},
|
||||
}
|
||||
|
||||
_, err := New().Chunk(context.Background(), chunkRequestWithClient(client))
|
||||
_, err := newChunker(t, client).Chunk(context.Background(), chunkRequest())
|
||||
if err == nil {
|
||||
t.Fatal("Chunk() error = nil, want malformed structured output error")
|
||||
}
|
||||
@@ -306,24 +316,27 @@ func TestChunkDefensivelyCopiesSourceUnitsAndMetadata(t *testing.T) {
|
||||
doc := sceneSourceDocument()
|
||||
client := &fakeScenesLLMClient{response: validSceneResponse()}
|
||||
|
||||
result, err := New().Chunk(context.Background(), contracts.ChunkRequest{
|
||||
result, err := newChunker(t, client).Chunk(context.Background(), contracts.ChunkRequest{
|
||||
Source: doc,
|
||||
SourceInput: sceneSourceInput(),
|
||||
SessionID: "session-123",
|
||||
LLMProfile: "profile-scenes",
|
||||
LLMClient: client,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Chunk() error = %v, want nil", err)
|
||||
}
|
||||
|
||||
doc.Units[0].ID = 99
|
||||
doc.Units[0].Ref.SourceID = "mutated"
|
||||
doc.Units[0].Metadata["speaker"] = "mutated"
|
||||
client.response.Scenes[0].MainParticipants[0] = "mutated"
|
||||
|
||||
if result.Chunks[0].Units[0].ID != 1 {
|
||||
t.Fatalf("chunk unit ID changed after source mutation: %#v", result.Chunks[0].Units[0])
|
||||
}
|
||||
if got := result.Chunks[0].Units[0].Ref.SourceID; got != "session-alpha" {
|
||||
t.Fatalf("chunk unit ref changed after source mutation: %#v", result.Chunks[0].Units[0].Ref)
|
||||
}
|
||||
if result.Chunks[0].Units[0].Metadata["speaker"] != "Alice" {
|
||||
t.Fatalf("chunk unit metadata changed after source mutation: %#v", result.Chunks[0].Units[0].Metadata)
|
||||
}
|
||||
@@ -334,7 +347,7 @@ func TestChunkDefensivelyCopiesSourceUnitsAndMetadata(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestChunkerManifestMetadataIncludesPromptAndSchemaProvenance(t *testing.T) {
|
||||
metadata := New().ManifestMetadata()
|
||||
metadata := newChunker(t, &fakeScenesLLMClient{}).ManifestMetadata()
|
||||
|
||||
tests := map[string]string{
|
||||
"prompt_id": PromptID,
|
||||
@@ -364,7 +377,7 @@ func TestChunkerManifestMetadataIncludesPromptAndSchemaProvenance(t *testing.T)
|
||||
|
||||
func TestChunkRejectsInvalidRequests(t *testing.T) {
|
||||
validClient := &fakeScenesLLMClient{response: validSceneResponse()}
|
||||
validReq := chunkRequestWithClient(validClient)
|
||||
validReq := chunkRequest()
|
||||
canceledCtx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
invalidDoc := sceneSourceDocument()
|
||||
@@ -380,13 +393,11 @@ func TestChunkRejectsInvalidRequests(t *testing.T) {
|
||||
want string
|
||||
}{
|
||||
{name: "nil chunker", chunker: nil, ctx: context.Background(), req: validReq, want: "chunker"},
|
||||
{name: "nil context", chunker: New(), ctx: nil, req: validReq, want: "context"},
|
||||
{name: "canceled context", chunker: New(), ctx: canceledCtx, req: validReq, want: "context"},
|
||||
{name: "nil source", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{LLMClient: validClient}, want: "source"},
|
||||
{name: "empty source units", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{Source: emptyDoc, LLMClient: validClient}, want: "units"},
|
||||
{name: "invalid source", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{Source: invalidDoc, LLMClient: validClient}, want: "validate source document"},
|
||||
{name: "nil LLM client", chunker: New(), ctx: context.Background(), req: contracts.ChunkRequest{Source: sceneSourceDocument()}, want: "LLM client"},
|
||||
{name: "unsupported options", chunker: New(), ctx: context.Background(), req: requestWithOptions(validReq), want: "options"},
|
||||
{name: "nil context", chunker: newChunker(t, validClient), ctx: nil, req: validReq, want: "context"},
|
||||
{name: "canceled context", chunker: newChunker(t, validClient), ctx: canceledCtx, req: validReq, want: "context"},
|
||||
{name: "nil source", chunker: newChunker(t, validClient), ctx: context.Background(), req: contracts.ChunkRequest{}, want: "source"},
|
||||
{name: "empty source units", chunker: newChunker(t, validClient), ctx: context.Background(), req: contracts.ChunkRequest{Source: emptyDoc}, want: "units"},
|
||||
{name: "invalid source", chunker: newChunker(t, validClient), ctx: context.Background(), req: contracts.ChunkRequest{Source: invalidDoc}, want: "validate source document"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
@@ -451,8 +462,8 @@ func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
|
||||
name: "empty metadata field",
|
||||
response: replaceScenes(validSceneResponse(), []sceneResponse{
|
||||
{
|
||||
StartUnitID: dnd.UnitRefFromInt(1),
|
||||
EndUnitID: dnd.UnitRefFromInt(4),
|
||||
StartUnitID: shared.UnitRefFromInt(1),
|
||||
EndUnitID: shared.UnitRefFromInt(4),
|
||||
ShortTitle: " ",
|
||||
PrimaryMode: "Narrative",
|
||||
MainParticipants: []string{"Aria"},
|
||||
@@ -467,8 +478,8 @@ func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
|
||||
name: "empty participant",
|
||||
response: replaceScenes(validSceneResponse(), []sceneResponse{
|
||||
{
|
||||
StartUnitID: dnd.UnitRefFromInt(1),
|
||||
EndUnitID: dnd.UnitRefFromInt(4),
|
||||
StartUnitID: shared.UnitRefFromInt(1),
|
||||
EndUnitID: shared.UnitRefFromInt(4),
|
||||
ShortTitle: "Title",
|
||||
PrimaryMode: "Narrative",
|
||||
MainParticipants: []string{"Aria", " "},
|
||||
@@ -484,7 +495,7 @@ func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
client := &fakeScenesLLMClient{response: tt.response}
|
||||
_, err := New().Chunk(context.Background(), chunkRequestWithClient(client))
|
||||
_, err := newChunker(t, client).Chunk(context.Background(), chunkRequest())
|
||||
if err == nil {
|
||||
t.Fatal("Chunk() error = nil, want error")
|
||||
}
|
||||
@@ -498,7 +509,7 @@ func TestChunkRejectsMalformedStructuredOutput(t *testing.T) {
|
||||
func TestChunkWrapsLLMClientError(t *testing.T) {
|
||||
client := &fakeScenesLLMClient{err: errors.New("provider unavailable")}
|
||||
|
||||
_, err := New().Chunk(context.Background(), chunkRequestWithClient(client))
|
||||
_, err := newChunker(t, client).Chunk(context.Background(), chunkRequest())
|
||||
if err == nil {
|
||||
t.Fatal("Chunk() error = nil, want LLM error")
|
||||
}
|
||||
@@ -507,13 +518,12 @@ func TestChunkWrapsLLMClientError(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func chunkRequestWithClient(client contracts.StructuredLLMClient) contracts.ChunkRequest {
|
||||
func chunkRequest() contracts.ChunkRequest {
|
||||
return contracts.ChunkRequest{
|
||||
Source: sceneSourceDocument(),
|
||||
SourceInput: sceneSourceInput(),
|
||||
SessionID: "session-123",
|
||||
LLMProfile: "profile-scenes",
|
||||
LLMClient: client,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -523,9 +533,13 @@ func sceneSourceInput() contracts.LLMInputMaterial {
|
||||
return contracts.NewLLMInputMaterial("source", "application/json", []byte(sceneTranscriptJSON), "sha256:transcript", "file:///session-alpha.json")
|
||||
}
|
||||
|
||||
func requestWithOptions(req contracts.ChunkRequest) contracts.ChunkRequest {
|
||||
req.Options = map[string]any{"max_units": 2}
|
||||
return req
|
||||
func newChunker(t *testing.T, client contracts.StructuredLLMClient) *Chunker {
|
||||
t.Helper()
|
||||
chunker, err := New(client, Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v, want nil", err)
|
||||
}
|
||||
return chunker
|
||||
}
|
||||
|
||||
func sceneSourceDocument() *source.SourceDocument {
|
||||
@@ -535,10 +549,10 @@ func sceneSourceDocument() *source.SourceDocument {
|
||||
Format: "application/vnd.seriatim.minimal+json",
|
||||
Digest: "sha256:source",
|
||||
Units: []source.SourceUnit{
|
||||
{ID: 1, Kind: "transcript_segment", Text: "Aria asks whether the goblin will parley.", Metadata: map[string]any{"speaker": "Alice"}},
|
||||
{ID: 2, Kind: "transcript_segment", Text: "The goblin scout describes the gate guards."},
|
||||
{ID: 3, Kind: "transcript_segment", Text: "The guards rush out with blades drawn."},
|
||||
{ID: 4, Kind: "transcript_segment", Text: "The party defeats the ambushers."},
|
||||
{ID: 1, Kind: "transcript_segment", Text: "Aria asks whether the goblin will parley.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 1}, Metadata: map[string]any{"speaker": "Alice"}},
|
||||
{ID: 2, Kind: "transcript_segment", Text: "The goblin scout describes the gate guards.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 2}},
|
||||
{ID: 3, Kind: "transcript_segment", Text: "The guards rush out with blades drawn.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 3, EndUnitID: 3}},
|
||||
{ID: 4, Kind: "transcript_segment", Text: "The party defeats the ambushers.", Ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: 4, EndUnitID: 4}},
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -559,8 +573,8 @@ func replaceScenes(response chunkResponse, scenes []sceneResponse) chunkResponse
|
||||
|
||||
func scene(startUnitID int, endUnitID int) sceneResponse {
|
||||
return sceneResponse{
|
||||
StartUnitID: dnd.UnitRefFromInt(startUnitID),
|
||||
EndUnitID: dnd.UnitRefFromInt(endUnitID),
|
||||
StartUnitID: shared.UnitRefFromInt(startUnitID),
|
||||
EndUnitID: shared.UnitRefFromInt(endUnitID),
|
||||
ShortTitle: "Scene title",
|
||||
PrimaryMode: "Narrative",
|
||||
MainParticipants: []string{"Aria"},
|
||||
@@ -570,7 +584,7 @@ func scene(startUnitID int, endUnitID int) sceneResponse {
|
||||
}
|
||||
}
|
||||
|
||||
func chunkIDs(chunks []contracts.SourceChunk) []string {
|
||||
func chunkIDs(chunks []source.Chunk) []string {
|
||||
ids := make([]string, 0, len(chunks))
|
||||
for _, chunk := range chunks {
|
||||
ids = append(ids, chunk.ID)
|
||||
30
internal/modules/dnd/chunk/scenes/model.go
Normal file
30
internal/modules/dnd/chunk/scenes/model.go
Normal file
@@ -0,0 +1,30 @@
|
||||
package scenes
|
||||
|
||||
import "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
|
||||
type chunkResponse struct {
|
||||
Scenes []sceneResponse `json:"scenes"`
|
||||
BoundaryCaveats []string `json:"boundary_caveats"`
|
||||
}
|
||||
|
||||
type sceneResponse struct {
|
||||
StartUnitID shared.UnitRef `json:"start_unit_id"`
|
||||
EndUnitID shared.UnitRef `json:"end_unit_id"`
|
||||
ShortTitle string `json:"short_title"`
|
||||
PrimaryMode string `json:"primary_mode"`
|
||||
MainParticipants []string `json:"main_participants"`
|
||||
Summary string `json:"summary"`
|
||||
BoundaryNote string `json:"boundary_note"`
|
||||
BoundaryConfidence string `json:"boundary_confidence"`
|
||||
}
|
||||
|
||||
type normalizedScene struct {
|
||||
StartUnitID int
|
||||
EndUnitID int
|
||||
ShortTitle string
|
||||
PrimaryMode string
|
||||
MainParticipants []string
|
||||
Summary string
|
||||
BoundaryNote string
|
||||
BoundaryConfidence string
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user